Clear worker process memory without deleting pool

Let's say I am in the following situation:

• Recreating worker pools is slow
• Memory pressure is high

For the following piece of code I would like to minimize the memory usage of workers after they have finished the first parfor/parfeval, but not by deleting the worker pool:

% Create worker pool to parallelize dostuff
workers = parpool(..)
parfor 1:alot % can also be a parfeval construct
    dataout{i} = dostuff(datain{i})
end

%delete(workers); % works but leads to slow recreating of pool later

memoryConsumingThing(); % single threaded

% Try to reuse worker pool here because instantiating them is slow
parfor 1:alottoo % can also be a parfeval construct
    dataout2{i} = dostuff2(datain2{i})
end

If I leave this code as is, about 1.1GB per worker/process is kept in memory which I cannot afford to leave there. When using Google to find a solution I almost always come across people suffering from memory leaks in dostuff(), but that is not the case in my setup. Rerunning this code or increasing iteration counts does not steadily increase memory consumption.

After parfor #1, the workers all contain active non-finished infinite NOP loops (@distcomp.nop) as their tasks. This is fine because I need to reuse them, but I would like to free as much of their working memory as possible without accidentally disrupting them and destroying the Pool in the process.

With parfeval I tried this approach:

for i = 1:alot
    jobstorage{i} = parfeval(..);
end
for i = 1:alot
    wait(jobstorage{i});
    [~,datastorage{i}] = fetchOutputs(jobstorage{i}); % Keep local copy in datastorage
    delete(jobstorage{i}); % Delete copy in worker process
end

with only limited success. Accessing jobstorage in this way is also not available with parfor :(

What would be the proper way of resetting worker processes to their base state?

[Johan Mes]

Noone? :(

[Kristian Dixon]

@Mathworks staff - any comments here? I have the same issue and thus far haven't found a solution/workaround online or otherwise. How can we release worker memory without shutting down the parpool?

[Mike Crabtree]

This seems to be a frequent issue experienced by non-expert MATLAB users who nevertheless need to use MATLAB's parallel computing ability to solve engineering problems in a timely manner. Surprising, there doesn't seem to be a straightforward answer around.

[Thomas Falch]

Since R2020a, MATLAB supports parpools using threads, which have significantly reduced memory overhead (but does not support all MATLAB functionality) https://www.mathworks.com/help/parallel-computing/choose-between-thread-based-and-process-based-environments.html

[Sinan Islam]

I am having similar problem. I am using parfor loop and inside this loop, there is a matrix that changes its size on every iteration. Unfortunately, this fills up the memory of threads very fast, and MATLAB becomes so slow. I need to find a way to clear the memory of threads after the parfor loop complete. Perhaps, MATLAB has a command to explicitly invoke garbage collection on every thread? If so, please let me know. Thank you!

[Walter Roberson]

parfevalOnAll(@clearvars, 0)

perhaps?

Answer 1

[max_hid]

What I did was:

jobstorage(numel(alot), 1) = parallel.FevalFuture;
for i = 1:alot
    jobstorage{i} = parfeval(..);
end
for i = 1:alot
    wait(jobstorage{1});
    [~,datastorage{i}] = fetchOutputs(jobstorage{1}); % Keep local copy in datastorage
    jobstorage{1} = []
end

This worked quite well for me.