Periodically saving workspace within a PARFOR loop

Hi all,

these days I was trying to take advantage of my new i7 desktop PC implementing PARFOR in some old m-files, but there is a problem I'm not able to address by myself.

One of these scripts is supposed to run for many hours (eventually, several days), so I thought it was the case to automatically save the workspace once in while, in partial remedy to temporary black-out, software crashes or nuclear attacks: I would just load the most recent saved workspace, set the FOR to start from the last known iteration and keep going again. The (very simple) idea was to have

for i=1:10000;
 ...instructions...
 storage(i) = someOutput;
 if mod(i,100)==0
   save complete.mat;
 end
end

of course this is not possible anymore with PARFOR, because several instances of the same iteration run contemporaneously and while a certain core could be working on iteration 200, another one could be stuck at 187 and a third one already at 209, therefore periodically saving this way would result in a lot of "holes" uneasy to fill in following instances.

A wild idea could be to keep this framework, load the file, clear all the iteration around missing data and start again some iterations back (in the previous example, say, from iteration 100), but this is not very efficient and would require a careful manual check.

So.. are out there some smart workarounds for that problem? :)

Thank you

Answer 1

[Walter Roberson]

dir() to figure out which iterations are already done
iters_to_do = setdiff(1:10000, list of iterations already done)
parfor against iters_to_do

[Andrea Giannuzzi]

Thank you for your answer.

It seems you are (implicitly) suggesting to set up the loop such that it generates a small file named after each iteration number just before it terminates.

In fact, however, I think that you're providing the very answer in the second part of your example, suggesting for a "targeted" FOR (which availability was admittedly unknown to me), because I could easily retrieve the informati on about completed iterations using a generic variable to be updated (for instance) just this way:

iter_done = [];
  parfor ith_iter=1:10000
     ...
     iter_done(ii) = ii;
  end

The only thing I need to make sure is whether it is more parallel-efficient to make workers updating the client variable iter_done or to actually save a small file.

[Edric Ellis]

Yes, you need to save a file each iteration around the PARFOR loop. Also, note that PARFOR only supports monotonically increasing loop indices, so you actually need

iters_to_do = setdiff(...);
parfor ii = 1:10000
  if iters_to_do(ii)
    ... loop body ...
  end
end