Mistery: Hist command wastes a LOT of memory. Why?

I have a memory problem with the "hist" command. It uses a lot of memory, and I have no idea how to clear it. I need to clear this memory since I am going Out of memory.

You can see this example:

memory
for i=1:300
[a]=hist(1);
clear a;
end
clear i;
memory

The "Memory used by Matlab" after doing hist 300 times is around 6 MB more, even if I am clearing "a" every time. Where is this memory going? How can I clear it?

[Oleg Komarov]

I don't see much problem with 6 MB when you should have at least 1 GB. You won't be able to clear that memory without restarting the MATLAB. Anyways, why are you using hist to count, use histc.

[Paulo Silva]

I also don't see the problem, please provide a better example because I don't see my MATLAB 2008b wasting any memory on that code. Also 6MB is nothing these days, if you are getting out of memory because of 6MB that's very strange, maybe there's other code wasting large amounts of memory or your system doesn't have enough memory available for MATLAB to run properly.

[Cristina]

Thanks for your answers. The 6 MB are not a problem. But Many 6 MB are a problem at the end. I need to use hist to calculate a histogram of different vectors, not for counting. hist(1) is just a simplied example to show how the memory is being consumed.

[Cristina]

@Paulo, I have also Matlab2008b, if you write exactly the above example in your Matlab, you do not see any difference in the "Memory used by Matlab" before and after the comands????? I do see a difference of around 6 MB.

[Paulo Silva]

I don't see any difference in memory use with your code, you don't even need to clear a because on each iteration the variable a gets new value.

My MATLAB keeps using 628MB before and after your code is run.

[Cristina]

Thank Paulo, this is very strange, I will have to contact with the Matlab support. I am getting exactly this:

>> memory

Maximum possible array: 349 MB (3.658e+008 bytes) *

Memory available for all arrays: 1085 MB (1.138e+009 bytes) **

Memory used by MATLAB: 604 MB (6.333e+008 bytes)

Physical Memory (RAM): 3024 MB (3.171e+009 bytes)

* Limited by contiguous virtual address space available.

** Limited by virtual address space available.

>> for i=1:300

[a]=hist(1);

clear a;

end

>> clear i

>> memory

Maximum possible array: 349 MB (3.658e+008 bytes) *

Memory available for all arrays: 1085 MB (1.138e+009 bytes) **

Memory used by MATLAB: 610 MB (6.393e+008 bytes)

Physical Memory (RAM): 3024 MB (3.171e+009 bytes)

* Limited by contiguous virtual address space available.

** Limited by virtual address space available.

[Walter Roberson]

If you only do (say) 1 iteration, how does the Memory Used by MATLAB change? Is it possible that the memory used by MATLAB reflects the internal pcoding of hist and the routines it calls?

Answer 1

[Jan]

I do net get any differences in the memory usage under MATLAB 2009a and 6.5.

Does the problem concern HISTC also? Does running the loop 600 times change the lost memory - perhaps to 12MB? Then I'd assume, that the underlying Mex function histc has a memory leak. You can look into the source of hist.c, which is fortunately shipped with MATLAB.

But a comparison of hist.c between the versions 6.5 and 2009a shows, that only the int->mwSize/mwIndex/mwSignedIndex and some catching of unexpected cases has been modified. But perhaps 2008b contained further lines, which could potentially leak memory.

[Cristina]

With histc the problem is solved. Thank you!

Answer 2

[Walter Roberson]

Memory fragmentation, perhaps. It is a problem with systems that do dynamic memory allocation. I do not know what techniques MATLAB has adopted to reduce it.

Answer 3

[Cristina]

The code I wrote above is just an example. I am not going Out of memory because of 6 MB. I have a code with thousend of loops, inside these loops I need to calcualte several histograms of different vectors. So 6 MB many many times are Gigabytes, and Out of memory. I just simplified the problem to show how hist is consuming a lof of memory, and I am unable to clear this memory, at the end I am running out of memory.

Does anyone know how to clear the memory used by the hist command?

[Paulo Silva]

Cristina the problem must be with your code and unless you provide it to us we can't suggest any improvements, if you are using the hist command the same way as your example code you don't need to clear the memory, the hist command returns a value (vector in your example) and that's it, it does use extra memory while running but that memory has it's own scope and is cleared when the hist command finishes processing.

[Cristina]

Dear Paulo, thanks for your answer. I am just clearing all variables to be sure that there shouldnt be any variable left in the workspace. This is exactly what I am getting if I do run the code in my Matlab:

>> memory

Maximum possible array: 349 MB (3.658e+008 bytes) *

Memory available for all arrays: 1085 MB (1.138e+009 bytes) **

Memory used by MATLAB: 604 MB (6.333e+008 bytes)

Physical Memory (RAM): 3024 MB (3.171e+009 bytes)

* Limited by contiguous virtual address space available.

** Limited by virtual address space available.

>> for i=1:300

[a]=hist(1);

clear a;

end

>> clear i

>> memory

Maximum possible array: 349 MB (3.658e+008 bytes) *

Memory available for all arrays: 1085 MB (1.138e+009 bytes) **

Memory used by MATLAB: 610 MB (6.393e+008 bytes)

Physical Memory (RAM): 3024 MB (3.171e+009 bytes)

* Limited by contiguous virtual address space available.

** Limited by virtual address space available.

[Oleg Komarov]

Note that:

a = hist(1);

b = histc(1,-4:5);

Are the same, but hist calls histc and calculates the edges. Maybe you could avoid the loss if you call histc directly.

My terminology may have not been clear, counting/binning in hist/histc context is the same.

[Cristina]

Dear Oleg,

At the end I solved the problem sustituting hist by histc. The comand histc does not consume so much memory. Thank you for your hint!