Reading portions of several CSV files

4 views (last 30 days)
I am using a software that exports CSV files of strain data from selected points throughout a test. (for example: aug6_6_point1.csv, aug6_6_point2.csv,... etc). The first 6 rows of each file are useless header text, so what I want to end up with is a 3D array (timesteps X straindata X #ofpoints). Then I'd like to be able to use the data from each column to do calculations.
Right now I have matlab reading one csv just fine, I would just like to figure out how to do it in a loop to create one large array. (Using num2string or sprintf?)
data = csvread('aug6_6_point5.csv',6,0);
I dont mind entering the number of files every time, I just dont want to manually select the file each time.
Any help would be awesome.
Thanks in advance.

Accepted Answer

Neuropragmatist on 14 Aug 2019
Is the problem that the loaded data are not in a format you like and you can't concatenate it easily or is it that you just don't know how to implement a loop?
If the former is the problem uploading one of your .csv files would let us see the problem better.
Otherwise would something like this not work:
fnames = {'filename1.csv','filename2.csv','filenameX.csv'}; % cell array of the filenames you want
all_data = []; % you can preallocate this if you know what size to expect
for ff = 1:length(fnames)
data = csvread(fnames{ff},6,0);
all_data = [all_data; data];
  1 Comment
Aaron Carpenter
Aaron Carpenter on 14 Aug 2019
I was actually able to make it work even better using the cell array method combined with what you suggested. Not sure why I couldn't understand the link I read earlier today.
Because each CSV is potentially different lengths, I do have to choose a time to stop reading (basically the shortest CSV I want to read), but that is no big deal since I am not interested in the end of the tests.
Lines 7-10 are inputs for file properties
Lines 13-18 create a cell array of file names I want to look at without having to manually type in every sequential file name.
Lines 24-27 create a 3D array of all my data for each point
Line 20 is just a sanity check and it helps me preallocate my alldata array.working2.JPG

Sign in to comment.

More Answers (2)

Rick Amos
Rick Amos on 15 Aug 2019
You might want to take a look at tabularTextDatastore and/or tall arrays. These are geared up to do exactly this kind of thing:-
fnames = {'filename1.csv','filename2.csv','filenameX.csv'};
% This is similar to csvread, but allows you to specify multiple files
ds = tabularTextDatastore(fnames, 'NumHeaderLines', 6, 'ReadVariableNames', false);
% If you just want to read all the data into a matrix
data = readall(ds);
data = data.Variables;
% Or, if you want to work with the data without reading all of it into memory in one go
data = tall(ds);
data = data.Variables;

Aaron Carpenter
Aaron Carpenter on 14 Aug 2019
Thank you,
I wouldnt say its perfect, since I need to manually plug in file names instead of running them using num2str or sprintf. This shouldnt be a big deal since I should never have more than 15- 20 files and they are easy to copy and paste.
I did find a link using cells, but after reading through it, it didnt look like it woud work for csv files, or skipping rows and columns to read.
The code below gives me a 245x9x6 array, where the 3rd dimension is the number of csv files I have.working.JPG


Find more on Large Files and Big Data in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!