Is it possible to fill a structure in a parfor?
19 views (last 30 days)
Show older comments
Hi everyone,
I need to open a large number of CSV files. The files are indexed according to some variables "ENF" (that goes from i=1:I) and "Theta" (that goes from j=1:J). Each file contains a large number of rows and columns. The number of columns and rows differ across files.
I would like load the data into a structure like this:
dataStruct.(ENFi).(Thetaj) = FileName_i_j.csv
This is my code to load the data:
I = 50;
J = 100;
DirData = ('/Users/MyDir');
NameData1 = ('Name1');
NameData2 = ('Name2');
for i=1:I % I would like to parallel process this
fprintf('ENF: %4.0f \n', i)
for j=1:J
FileName = [DirData NameData1 num2str(i) NameData2 num2str(j)];
Dat1 = ['ENF' num2str(i)];
Dat2 = ['Theta' num2str(j)];
dataStruct.(Dat1).(Dat2) = csvread([FileName '.csv']);
end
end
This loop takes a while. I would like to parallel process the first "for". I thought it should be straightforward. However, I cannot figure out how to do it. Matlab complains about the data structure within the parfor.
Any suggestions or ideas?
Thanks. Javier
0 Comments
Accepted Answer
Guillaume
on 24 Nov 2014
I haven't got the parallel processing toolbox but if matlab won't let you parallelise the dynamic field assignment the way I would solve it is by storing the content of these fields into a temporary cell array:
sfields = cell(1, I);
parfor i=1:I
s = struct;
%...
sfields{i} = struct(Dat2, csvread(sprintf('%s.csv', FileName)));
%...
sfields{i} = s;
end
for i=1:I
datastruct.(sprintf('ENF%d', i)) = sfields{i}
end
The extra reassignment at the end should take no time and no memory at all due to the copy on write nature of matlab.
0 Comments
More Answers (1)
See Also
Categories
Find more on Loops and Conditional Statements in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!