MATLAB Answers

How do I specify the number of workers in parfeval without deleting and recreating my pool

12 views (last 30 days)
I have a large, highly parallelized, and pretty efficient process I run for a while using parfor with all my cores (physical and logical) and a few gpus.
Once that process is finished, I need to perform a different process that is memory intensive, and I cannot use all my cores without hitting memory problems. But I can do the process just fine using half my cores.
Is there a way to take an existing pool (about 40 workers) and say "hey for this bit to code (pareval, not parfor) I only want to use 20 of my 40 workers that I have available in my pool without resetting my pool". Resetting the pool takes time that I'd like to avoid if possible. Sort of like parfor (ii = start:stop, nWorkers), but for parfeval.
To clarify, I know I can create a new pool with specified number of works, and the process in question is suited for parfeval, but not parfor. Lets assume for now that my problem is not memory efficiency and reducing the amount of memory, Ill deal with that seperately. This question is just about specifying number of workers in parfeval without deleting my pool and creating a new one.
Suggestions?

Accepted Answer

Edric Ellis
Edric Ellis on 29 Jan 2020
Unfortunately, there's no API to constrain the number of workers from the pool eligible to be used by parfeval. What you could consider is the following hack: basically, put half the workers into an infinite pause until you're done doing the rest of your work. I.e.
% Block half the workers
numToBlock = 20;
for idx = numToBlock:-1:1
blockers(idx) = parfeval(@pause, 0, Inf);
end
stopBlocking = onCleanup(@() cancel(blockers));
% Do stuff
f = parfeval(@myMemoryIntensiveThing,<args>);
... % more stuff
... % more stuff
% Eventually, clean up.
stopBlocking = []; % releases the blocked workers.
You could confine the ugliness by writing a simple class, something like
classdef Blocker < handle
properties (SetAccess = immutable)
Futures
end
methods
function obj = Blocker(pool, numToBlock)
assert(numToBlock <= pool.NumWorkers && numToBlock > 0);
for idx = numToBlock:-1:1
fut(idx) = parfeval(pool, @pause, 0, Inf);
end
obj.Futures = fut;
end
function delete(obj)
cancel(obj.Futures);
end
end
end

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!