I have a memory leak issue that I would like some advice on.
I've been running a parametric study in MATLAB using the Parallel Computing Toolbox. The script I'm using calls a function that is solved on one Tesla C2050 GPU (3GB max memory). This function is called ~130 times and the total (estimated) solution time is ~2 days. Most of the variables in the function are of gpuArray type and the rest is initialized at start, i.e. every time the function is called I would expect all variables to be cleared and reset to zero. The function saves a *.mat result file and a *.txt file for logging purposes.
After 12 hours of solving I noticed that the memory used by MATLAB has increased from 1GB to approximately 23GB while the GPU memory is being cleared every time that function is called.
It seems that the CPU memory is being consumed at a rate of 500kb/sec, whereas the GPU function is being recalled every 25minutes (that is when variables are reset).
Does anyone know what is consuming this memory? Interestingly enough, when I stopped the program and called (clear all) at the command line, nothing was cleared. I had to kill MATLAB from the task manager.
I'm solving the problem on a DELL T7500 with 8-cores, 1 Tesla GPU and 24GB of RAM running windows 7 64-bit.
Your help is greatly appreciated, Laith
I'll work on that. My Tesla is currently running a week long simulation so I cannot stop it to test the self contained example. I did however manage to work around the issue by using the (reset) command:
clear all clc numruns=10; g=gpuDevice(1);
for i=1:numruns out = gpuFunc(i,other_input); reset(g); wait(g); end
I'm using MATLAB 18.104.22.1684 (R2013a) with PCT version 6.2
Hi Laith, I believe this should be fixed by the following workaround:
Please can you try it (being careful about the backing up step!) and accept this answer if the problem is resolved. If not, let me know and we can try to work out what else might be amiss.