Is it possible to fill a structure in a parfor?

19 views (last 30 days)
Hi everyone,
I need to open a large number of CSV files. The files are indexed according to some variables "ENF" (that goes from i=1:I) and "Theta" (that goes from j=1:J). Each file contains a large number of rows and columns. The number of columns and rows differ across files.
I would like load the data into a structure like this:
dataStruct.(ENFi).(Thetaj) = FileName_i_j.csv
This is my code to load the data:
I = 50;
J = 100;
DirData = ('/Users/MyDir');
NameData1 = ('Name1');
NameData2 = ('Name2');
for i=1:I % I would like to parallel process this
fprintf('ENF: %4.0f \n', i)
for j=1:J
FileName = [DirData NameData1 num2str(i) NameData2 num2str(j)];
Dat1 = ['ENF' num2str(i)];
Dat2 = ['Theta' num2str(j)];
dataStruct.(Dat1).(Dat2) = csvread([FileName '.csv']);
end
end
This loop takes a while. I would like to parallel process the first "for". I thought it should be straightforward. However, I cannot figure out how to do it. Matlab complains about the data structure within the parfor.
Any suggestions or ideas?
Thanks. Javier

Accepted Answer

Guillaume
Guillaume on 24 Nov 2014
I haven't got the parallel processing toolbox but if matlab won't let you parallelise the dynamic field assignment the way I would solve it is by storing the content of these fields into a temporary cell array:
sfields = cell(1, I);
parfor i=1:I
s = struct;
%...
sfields{i} = struct(Dat2, csvread(sprintf('%s.csv', FileName)));
%...
sfields{i} = s;
end
for i=1:I
datastruct.(sprintf('ENF%d', i)) = sfields{i}
end
The extra reassignment at the end should take no time and no memory at all due to the copy on write nature of matlab.

More Answers (1)

Javier
Javier on 24 Nov 2014
Edited: Javier on 24 Nov 2014
Hi Guillaume,
This is my original code modified with your suggestion:
sfields = cell(1, I);
parfor i=1:I
s = struct;
fprintf('ENF: %4.0f \n', i)
for j=1:stepJ:J
FileName = [DirData NameData1 num2str(i) NameData2 num2str(j)];
Dat2 = ['Theta' num2str(j)];
s.(Dat2) = csvread([FileName '.csv']);
sfields{i} = s;
end
end
for i=1:I
dataStruct.(sprintf('ENF%d', i)) = sfields{i};
end
Indeed the parfor works fine and the final loop is innocuous. Fantastic!
Compared to my original code this is 50 times faster (using 12 workers).
Thank you! Javier

Categories

Find more on Loops and Conditional Statements in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!