Correct fetchNext index after clearing FevalFuture object

Asked by Nicola Dalla Pozza on 14 Oct 2019

Latest activity Answered by Edric Ellis on 15 Oct 2019

3 views (last 30 days)

I am using parfeval to run code asynchronously on workers. The running time varies a lot between workers depending on some parameters that in the following code I represent with 'm'.

When I collect the results with fetchNext, I then process them with idx, in particular retrieving m(idx). At the same time, I need to clear the last retrieved FevalFuture object to save memory.

In Parfeval - Memory consumption piling up - Clear output data? it is suggest to simply assign [ ] to the FevalFuture object. However, it seems to me that doing this mix up the indices idx of the objects, which I need to further process the data. See for instance this mve

N = 10;
m = 1:10;
for idx = N:-1:1
    f(idx) = parfeval(@(m,N) m*rand(N), 1, m(idx), N);
end
total = 0;
for idx = 1:N
    [idx, data] = fetchNext(f);
    idx
    f(idx) = []; % This line here
    
    % process data with idx, m(idx) ..
end

If you comment the line indicated the code will work fine, but if I clear f(idx) the just-used idx could be reused and I would process the data with the wrong m(idx).

How is it possible to get the correct index idx with which f(idx) was called in the fist loop?

Thanks in advantage

0 Comments

Products

Release

R2018b

1 Answer

Link

Answer by Edric Ellis on 15 Oct 2019

One way you can deal with this is by making the same modification to m as you proceed. Something like this perhaps:

N = 10;
m = 1:10;
for idx = N:-1:1
    f(idx) = parfeval(@(m,N) m*rand(N), 1, m(idx), N);
end
for idx = 1:N
    [fidx, data] = fetchNext(f);
    % Get the value out of "m" corresponding to the completed future
    mval = m(fidx);
    % Delete corresponding elements of both "f" and "m"
    f(fidx) = [];
    m(fidx) = [];
    % Work with the particular value of "m" and the result
    disp([mval, sum(data(:))]);
end

0 Comments

Products

Release

R2018b

You are now following this question

Nicola Dalla Pozza (view profile)