MATLAB Answers

0

Reading portions of several CSV files

Asked by Aaron Carpenter on 14 Aug 2019
Latest activity Answered by Rick Amos on 15 Aug 2019
Hi.
I am using a software that exports CSV files of strain data from selected points throughout a test. (for example: aug6_6_point1.csv, aug6_6_point2.csv,... etc). The first 6 rows of each file are useless header text, so what I want to end up with is a 3D array (timesteps X straindata X #ofpoints). Then I'd like to be able to use the data from each column to do calculations.
Right now I have matlab reading one csv just fine, I would just like to figure out how to do it in a loop to create one large array. (Using num2string or sprintf?)
data = csvread('aug6_6_point5.csv',6,0);
.
.
I dont mind entering the number of files every time, I just dont want to manually select the file each time.
Any help would be awesome.
Thanks in advance.

  0 Comments

Sign in to comment.

3 Answers

Answer by Neuropragmatist on 14 Aug 2019
 Accepted Answer

Is the problem that the loaded data are not in a format you like and you can't concatenate it easily or is it that you just don't know how to implement a loop?
If the former is the problem uploading one of your .csv files would let us see the problem better.
Otherwise would something like this not work:
fnames = {'filename1.csv','filename2.csv','filenameX.csv'}; % cell array of the filenames you want
all_data = []; % you can preallocate this if you know what size to expect
for ff = 1:length(fnames)
data = csvread(fnames{ff},6,0);
all_data = [all_data; data];
end
Thanks,
M.

  1 Comment

I was actually able to make it work even better using the cell array method combined with what you suggested. Not sure why I couldn't understand the link I read earlier today.
Because each CSV is potentially different lengths, I do have to choose a time to stop reading (basically the shortest CSV I want to read), but that is no big deal since I am not interested in the end of the tests.
Lines 7-10 are inputs for file properties
Lines 13-18 create a cell array of file names I want to look at without having to manually type in every sequential file name.
Lines 24-27 create a 3D array of all my data for each point
Line 20 is just a sanity check and it helps me preallocate my alldata array.working2.JPG

Sign in to comment.


Answer by Rick Amos on 15 Aug 2019

You might want to take a look at tabularTextDatastore and/or tall arrays. These are geared up to do exactly this kind of thing:-
fnames = {'filename1.csv','filename2.csv','filenameX.csv'};
% This is similar to csvread, but allows you to specify multiple files
ds = tabularTextDatastore(fnames, 'NumHeaderLines', 6, 'ReadVariableNames', false);
% If you just want to read all the data into a matrix
data = readall(ds);
data = data.Variables;
% Or, if you want to work with the data without reading all of it into memory in one go
data = tall(ds);
data = data.Variables;

  0 Comments

Sign in to comment.


Answer by Aaron Carpenter on 14 Aug 2019

Thank you,
I wouldnt say its perfect, since I need to manually plug in file names instead of running them using num2str or sprintf. This shouldnt be a big deal since I should never have more than 15- 20 files and they are easy to copy and paste.
I did find a link using cells, but after reading through it, it didnt look like it woud work for csv files, or skipping rows and columns to read.
The code below gives me a 245x9x6 array, where the 3rd dimension is the number of csv files I have.working.JPG

  0 Comments

Sign in to comment.