Reading portions of several CSV files

Hi.

I am using a software that exports CSV files of strain data from selected points throughout a test. (for example: aug6_6_point1.csv, aug6_6_point2.csv,... etc). The first 6 rows of each file are useless header text, so what I want to end up with is a 3D array (timesteps X straindata X #ofpoints). Then I'd like to be able to use the data from each column to do calculations.

Right now I have matlab reading one csv just fine, I would just like to figure out how to do it in a loop to create one large array. (Using num2string or sprintf?)

data = csvread('aug6_6_point5.csv',6,0);

.

.

I dont mind entering the number of files every time, I just dont want to manually select the file each time.

Any help would be awesome.

Thanks in advance.

Answer 1

[Neuropragmatist]

Is the problem that the loaded data are not in a format you like and you can't concatenate it easily or is it that you just don't know how to implement a loop?

If the former is the problem uploading one of your .csv files would let us see the problem better.

Otherwise would something like this not work:

fnames = {'filename1.csv','filename2.csv','filenameX.csv'}; % cell array of the filenames you want
all_data = []; % you can preallocate this if you know what size to expect
for ff = 1:length(fnames)
    data = csvread(fnames{ff},6,0);
    all_data = [all_data; data];
end

Thanks,

M.

[Aaron Carpenter]

I was actually able to make it work even better using the cell array method combined with what you suggested. Not sure why I couldn't understand the link I read earlier today.

Because each CSV is potentially different lengths, I do have to choose a time to stop reading (basically the shortest CSV I want to read), but that is no big deal since I am not interested in the end of the tests.

Lines 7-10 are inputs for file properties

Lines 13-18 create a cell array of file names I want to look at without having to manually type in every sequential file name.

Lines 24-27 create a 3D array of all my data for each point

Line 20 is just a sanity check and it helps me preallocate my alldata array.

Answer 2

[Rick Amos]

You might want to take a look at tabularTextDatastore and/or tall arrays. These are geared up to do exactly this kind of thing:-

fnames = {'filename1.csv','filename2.csv','filenameX.csv'};

% This is similar to csvread, but allows you to specify multiple files
ds = tabularTextDatastore(fnames, 'NumHeaderLines', 6, 'ReadVariableNames', false);

% If you just want to read all the data into a matrix
data = readall(ds);
data = data.Variables;

% Or, if you want to work with the data without reading all of it into memory in one go
data = tall(ds);
data = data.Variables;

Answer 3

[Aaron Carpenter]

Thank you,

I wouldnt say its perfect, since I need to manually plug in file names instead of running them using num2str or sprintf. This shouldnt be a big deal since I should never have more than 15- 20 files and they are easy to copy and paste.

I did find a link using cells, but after reading through it, it didnt look like it woud work for csv files, or skipping rows and columns to read.

The code below gives me a 245x9x6 array, where the 3rd dimension is the number of csv files I have.