Why do I see an increase in memory usage on my workers as I keep running jobs successively when using the Distributed Computing Toolbox?

4 views (last 30 days)
I am using the Distributed Computing Toolbox to run my simulations on multiple servers and workers. I run jobs successively and kill each job after it has completed using the DESTROY command.
I see that the memory usage on my workers gradually increases with the number of jobs finished. Eventually, I get 'out of memory' exceptions.

Accepted Answer

MathWorks Support Team
MathWorks Support Team on 21 Nov 2016
The DESTROY command in the Distributed Computing Toolbox removes the job object reference or task object reference from the local session, and removes the object from the job manager memory.
destroy(obj)
However, when a job is finished, it remains in the job manager even if you clear all the objects from the client session. The job manager keeps all the jobs it has executed until you restart the job manager in a clean state.
Additionally, tasks running on the workers may have allocated memory that is not released: global or persistent variables, locked functions, memory leaks or persistent data in MEX files, etc.
After destroying the job or task, you can restart a worker using the RestartWorker property. The memory is then returned to the worker.You can set RestartWorker to true (or logical 1) if you want the job to restart the MATLAB session on any workers before they evaluate their first task for that job.
For example, submitting the following job after your main jobs have finished will force all the workers to restart their MATLAB instances:
 
function forceRestartAllWorkers(sched)
%After this function completes, the MDCE will have
% restarted each worker. The argument 'sched' should
% be a scheduler object, that is, the result of
% a findResource operation.
memFlushJob = createParallelJob(sched, 'RestartWorker', true);
t = createTask(memFlushJob, @() true, 0, {});
submit(memFlushJob);
waitForState(memFlushJob);
destroy(memFlushJob);
end
More information about this can be found at the following URLs:

More Answers (0)

Categories

Find more on MATLAB Parallel Server in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!