MATLAB Answers

How can I import multiple .CSV files in MATLAB and SISTEMATICALLY process the data of each file?

7 views (last 30 days)
Christian Scalia
Christian Scalia on 22 Jan 2021
Answered: Jeremy Hughes on 25 Jan 2021
Hi all. I have a couple of csv files I need to analyse (1488 files, each having 3 columns and 2304 rows containing only numerical values). I would like my script to open these csv files one by one and extrapolate the informations I need. For example, the script opens the first file, does its calculations, stores these 2 relevant values (that are unique to that file) and then moves on to the next file repeating the same calculations. The script I have right now does this but just for one file. How do I change the script so that it proceeds file by file and stores the obtained values?
1:All the files are contained in the same foulder
2: The first row of the file is Text, this is why i use :
vector=csvread(filename,1);
3: The relevant values I want the script to save before it moves to the next file are: Hav and Tz (at the end of the script).
4: What I'm struggling with here is to give Matlab the right order to do things: ie: open file-->do the calculations--->store the values----> open new file-->repeat the same calculations etc.
Thanks.
filename='2016\May\2016-05-31T23h36.csv';
vector=csvread(filename,1);
eta = vector(:,1)/100; % heave disp in metres
north = vector(:,2)/100; % northing in metres
west = vector(:,3)/100; % westing in metres
%------ TIME-SERIES ANALYSIS ----------------------------------------------------
% de-mean the time series
eta = eta - mean(eta);
sr=length(eta)/(30*60); %sampling frequency estimate based on a full 30minute record
dt=1/sr;
for i=1:length(eta) % time vector for data
t(i)=(i-1)*dt;
end
% find the zero up-crossings
[ind,t0]=crossing(eta,t); % this finds all crossings (up and down)
n_up = 0;
for i = 1:length(ind)
if eta(ind(i)+1) - eta(ind(i)) > 0
n_up = n_up+1;
ind_up(n_up) = ind(i);
t0_up(n_up) = t0(i);
end
end
fprintf(1,'no. of crossings = %g\n',length(ind))
fprintf(1,'no. of up_crossings = %g\n',n_up)
nwaves = n_up-1;
for i = 1:nwaves
wv = eta(ind_up(i):ind_up(i+1));
crest(i)=max(wv); % determine crest of that wave
trough(i)=abs(min(wv)); % determine trough of that wave
ht(i)=crest(i)+trough(i); % determine height of that wave
end
Hav=mean(ht)
Tz=mean(diff(t0_up))

Answers (2)

dpb
dpb on 22 Jan 2021
See <Importing-all-files-from-a-specific-folder?> from yesterday. There's a section in the doc with examples on processing multiple files as well.
In addition, at a higher level there's a function FILEFUNCTION on the File Exchange that lets you simply use your function as an argument and will call it with all files in the argument list that can let you avoid actually writing the looping construct yourself.
I forget the author but I've used it successfully several times to save just such effort.
  2 Comments
dpb
dpb on 24 Jan 2021
datapath=uigetdir([],'Select Data Directory');
d=dir(fullfile(datapath,'*.txt');
for i=1:numel(d)
txt_file = fullfile(datapath,d(i).name);
...
end
is the code in the Answer linked to above...just insert your code inside the loop changing to use your variable instead of that poster's:
datapath=uigetdir([],'Select Data Directory');
d=dir(fullfile(datapath,'*.txt');
for i=1:numel(d)
% txt_file = fullfile(datapath,d(i).name); % Other poster's variable
vector=csvread(datapath,d(i).name,1); % Your variable instead...
...
% rest of your code goes here...
end
If you already know the directory containing the files you can dispense with retrieving it if want, of course.

Sign in to comment.


Jeremy Hughes
Jeremy Hughes on 25 Jan 2021
You could also look at tabularTextDatastore with transform, and writeall functions. This is the basic way that's done, but you may need to tweak the inputs for these depending on what you need to do
ttds = tabularTextDatastore(fileOrFolderName)
tds = transform(ttds,@myProcessFunction)
writeall(tds,outputLocation)
where
myProcessFunction takes the output of tabularTextDatastore's read and outputs the data you want to write.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!