Cell array not clearing properly on Unix based system

1 view (last 30 days)
Recently I've been helping to try and improve an old MATLAB data handling package for a group I work with and appear to have stumbled into an odd memory management behavior with large sized cell arrays.
We use large amounts of data and use a RAM buffer to store commonly used data or masks. This buffer is simply a global that is declared upon initialization of the data package and then updated or called when we need some part of it.
The old-style mask buffers were stored as a NxM array of uint8 quantities (techncially just using bit array to compress the logicals...this was an old decision but explains why they are not stored as logicals). The same is true for the buffer of commonly used data, except it is stored as doubles.
The issue is that if we access these many many times in some sort of loop, MATLAB can be slow because the buffer is readout as
m = buffer(:,2);
which takes time (still faster than reading the data from disk however, which is why we have these buffers in the first place). So we thought "why not just use cell arrays" since they are just arrays of pointers to the memory location, they are a lot faster to just return the data, rather than copy it to another location (and then if we do modify it in some way MATLAB then decides to make a copy). However there appears to be some sort of leak that occurs even if one just initializes them.
So below is some example code.
function status = buff_test_init(varargin)
clear global buff_test
global nbCol nbEvents buff_test
buff_size = 5*1024^3; %5GiB buffer
if isempty(nbEvents)
nbEvents = 1.7e6;
end
if isempty(nbCol)
nbCol = floor(buff_size/(1*nbEvents));
end
nCol = nbCol(1:end);
nEvents = nbEvents(1:end);
%Create buffer
buff_test = cell(nCol,1);
for cdx = 1:nCol
tempZero = uint8(zeros(nEvents,1));
rbuff_test{cdx} = tempZero(1:end);
end
status = 1;
end
Now a few things probably stand out here. First is the seemingly pointless nCol from nbCol style arguments. I found that for some reason making arrays with global variable arguments would often result in only virtual memory allocation initially, which is not behavior we want. Similar behavior was seen by using a zeros(nEvents,1,'uint8') style allocation in the cell array, so for debugging purposes I did the copy-style allocation with the tempZeros.
If I call this buff_test_init function many times in a for loop, and alter the global variables nbCol and nbEvents in each loop (by using some randomly chosen integers), eventually my memory stops clearing back to default and I wind up with several GB of memory used even after a 'clear all'. Note that I have experimented with using persistent variables instead of globals, but get the same memory behavior. Using the old-style arrays does not show such behavior. Is this a known issue (and if so is there some work-around?) or is it perhaps a leak?
This is tested on R2012b on a 64-bit Scientific Linux 5 system, with ~42GB of RAM (hence the ability to comfortably use a large buffer).
Thanks for any insight.

Answers (0)

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!