Save inside parfor loop at a specific iteration step
5 views (last 30 days)
Show older comments
Good day, I'm having problems saving inside a parfor loop. What I want to do is to save the results (vector results) from 5000 -> 5000 iterations (parfor loop takes 1 week to complete and I don't want to lose all the temporary result in case that something bad happens with the machine, i.e. reboot, power off..) A simplified version of my code is:
parfor ii = 1:100000
[output1, output2] = MyFunction(~,~,ii)
% from 5000 to 5000 iterations I want to save the vector results
if ii == 1:5000:100000
parsave('results.mat',output1, output2)
% now I want to save the final results
elseif ii == 100000
parsave('results.mat', output1, output2)
end
end
Maybe someone can provide me with an useful parfor.m function, or a way to save it using matfile objects (I don't know how to use them)
2 Comments
Sean de Wolski
on 23 Feb 2015
First, the if ii==vector will never return true because all values need to be true. You would need an any() around it; compare:
if 1 == 1:5,disp('hi'),end
if any(1 == 1:5),disp('hi'),end
Matt J
on 23 Feb 2015
@CalinV
Saving every 5000th iteration will not help you because parfor loop iterations runs in parallel and in arbitrary order. When iteration #5000 gets executed, it does not imply that iterations 1-4999 have also been executed.
Accepted Answer
Matt J
on 23 Feb 2015
Edited: Matt J
on 23 Feb 2015
Since MyFunction takes as long as it does (at least 6 seconds), I might just save every iteration within a for-drange loop. It takes no time to save a 1e5x1 vector, and the local part on each worker will be smaller still.
output1=distributed.NaN(1,1e5); %pre-allocate
output2=distributed.NaN(1,1e5);
spmd
for i=drange(1:1e5)
[output1(ii), output2(ii)] = MyFunction(~,~,ii);
parsave(['TEMP', num2str(labindex)],...
getLocalPart(output1),getLocalPart(output2));
end
end
1 Comment
More Answers (1)
Sean de Wolski
on 23 Feb 2015
Edited: Sean de Wolski
on 23 Feb 2015
First, the if ii==vector will never return true because all values need to be true. You would need an any() around it; compare:
if 1 == 1:5,disp('hi'),end
if any(1 == 1:5),disp('hi'),end
Second, since two workers might try to write at the same time, you'll need to save to different MAT files and concatenate after:
save(['Results' num2str(ii,'%05i') '.mat'],etc)
Other than that, looks like it should work...
2 Comments
Matt J
on 23 Feb 2015
I can't see how the OP's initial proposal will work, or at least be beneficial. The iterations of a parfor loop are independent, so saving every 5000th iteration will discard everything else. You also don't know in what order the iterations of a parfor loop will be executed. For all one knows, the iterations 1:5000:100000 could be the final 20 that get executed at the end of the week. A power failure mid-week will lose everything.
Sean de Wolski
on 23 Feb 2015
Maybe I didn't read closely enough - I thought he just wanted those iterations saved.
If he cares about order, use parfeval.
See Also
Categories
Find more on Parallel for-Loops (parfor) in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!