Thread Subject: Memory leak using large amount of data files

Hey Matlab developers,

I'm working on image analysis of 10000s of images. Matlab is R2007b on
32bit Windows XP. The images are *sequentially* opened, analyzed, stored
as M-Script, closed. However, memory runs full up to 3GB. Even if I try
to free memory aggressively on the command line, 1.5GB remain blocked
ultimately:

   close all
   clear all
   clear java
   clear classes
   clear functions
   pack

What is wrong???

Thanks!
Armin

Armin Mueller wrote:
> Hey Matlab developers,
>
> I'm working on image analysis of 10000s of images. Matlab is R2007b on
> 32bit Windows XP. The images are *sequentially* opened, analyzed, stored
> as M-Script, closed. However, memory runs full up to 3GB. Even if I try
> to free memory aggressively on the command line, 1.5GB remain blocked
> ultimately:
>
> close all
> clear all
> clear java
> clear classes
> clear functions
> pack
>
> What is wrong???

This is impossible for us to tell without seeing any code.
Some general tips would be:

Can you process 5000 images? Can you process 7000 images? If
yes, then step through your code in debugger to see where you start to have
problems. This is assuming that each image is reasonably small
i.e. you can process each image on a fresh start. If this is the
case then your code is not efficient enough for your system. Fix
your code (or find a bug in ML). Otherwise (=images are too big
individually) split the problem into smaller parts or buy more RAM.

marky <no@email.com> wrote in message
<flfpam$ru4$1@news.oulu.fi>...
> Armin Mueller wrote:
> > Hey Matlab developers,
> >
> > I'm working on image analysis of 10000s of images. Matlab
is R2007b on
> > 32bit Windows XP. The images are *sequentially* opened,
analyzed, stored
> > as M-Script, closed. However, memory runs full up to 3GB.
Even if I try
> > to free memory aggressively on the command line, 1.5GB
remain blocked
> > ultimately:
> >
> > close all
> > clear all
> > clear java
> > clear classes
> > clear functions
> > pack
> >
> > What is wrong???
>
> This is impossible for us to tell without seeing any code.
> Some general tips would be:
>
> Can you process 5000 images? Can you process 7000 images? If
> yes, then step through your code in debugger to see where you
start to have
> problems. This is assuming that each image is reasonably small
> i.e. you can process each image on a fresh start. If this is
the
> case then your code is not efficient enough for your system.
Fix
> your code (or find a bug in ML). Otherwise (=images are too
big
> individually) split the problem into smaller parts or buy
more RAM.

In my experience, you could debug and monitor your MATLAB
memory by the MATLAB monitoring tool (Windows only) which is
available at MATLAB File Exchange

http://www.mathworks.com/matlabcentral/fileexchange/
loadFile.do?objectId=7127&objectType=file

and activate the MATLAB memory information by starting MATLAB
with this command

      matlab -check_malloc

Moreover, some methods to avoid “Out of Memory Errors” cause of
large data are presented at this website

http://www.mathworks.com/support/tech-notes/1100/1107.html

Anh Huy Phan
RIKEN - BSI

marky wrote:

> Fix your code (or find a bug in ML).

I assume it is a bug in Matlab, to be more precise the JIT compiler.
When I process i.e. 10.000 images and then say "clear all" on the
command line, memory footprint should be substantially smaller than
1.5GB afterwards!

> Otherwise (=images are too big individually) split the problem into
> smaller parts or buy more RAM.

More than 3GB on a 32bit system? Very funny!

One image is about in memory and 1280x1024x16bit gray level. Even in
"double" representation that is 10MB. No big deal.

:-/
Armin

This is a multi-part message in MIME format.
--------------000605070703060804090708
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Huy wrote:

> In my experience, you could debug and monitor your MATLAB
> memory by the MATLAB monitoring tool (Windows only) which is
> available at MATLAB File Exchange [...]

Thanks for all the hints about debugging memory usage in MATLAB. I'll
try that!

Meanwhile, I have created a small and obvious example to show how the
memory leak occurs. All results in my program are written as protocol in
an M-file. Later they are processed to get some statistics (see
attachment). There are two flavors, one protocol is written in M-Script
format and the other one in M-Function format.

If you i.e. run the "leak_demo2s" on 32bit Windows XP, virtual size
(measured using Process Explorer) will grow from 500MB to 1GB. There is
no chance to clear up. When you do 10.000 loops instead of 1.000, the
3GB barrier will be hit.

Using M-Function makes no difference.

Cheers,
Armin

--------------000605070703060804090708
Content-Type: text/plain;
 name="leak_demo2s.m"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="leak_demo2s.m"

function leak_demo2s

feature memstats;
tic

for nfile=1:1000

% Save als M-Script

filename = ['data' num2str(nfile, '%04d') 's'];

fid = fopen(['D:\dump\' filename '.m'], 'wt');

if fid<0
warning('leak_demo2:save', 'Could not save results.')
else
fprintf(fid, '%% Ergebnis der Auswertung der Testreihe\n');
fprintf(fid, '%% %s\n', filename);
fprintf(fid, '%% Auswertung vom %s\n\n', datestr(now));

fprintf(fid, '%% cogx\tcogy\tdiam\tcogx\tcogy\tdiam\texcent\tgradient\n');
fprintf(fid, '%% [pix]\t[pix]\t[pix]\t[mm]\t[mm]\t[mm]\t[1]\t[?]\n');

fprintf(fid, 'data\t= [\n\n');

fprintf(fid, '%g\t%g\t%g\t%g\t%g\t%g\t%g\t%g\n', rand(2048, 8));

fprintf(fid, '];\n');
fclose(fid);
end

% Load M-Script

run(['D:\dump\' filename '.m'])
xyz = mean(data);

end

toc
feature memstats;

--------------000605070703060804090708
Content-Type: text/plain;
 name="leak_demo2f.m"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="leak_demo2f.m"

function leak_demo2f

feature memstats;
tic

for nfile=1:1000

% Save als M-Function

filename = ['data' num2str(nfile, '%04d') 'f'];

fid = fopen(['D:\dump\' filename '.m'], 'wt');

if fid<0
warning('leak_demo2:save', 'Could not save results.')
else
fprintf(fid, '%% Ergebnis der Auswertung der Testreihe\n');
fprintf(fid, '%% %s\n', filename);
fprintf(fid, '%% Auswertung vom %s\n\n', datestr(now));

fprintf(fid, '%% cogx\tcogy\tdiam\tcogx\tcogy\tdiam\texcent\tgradient\n');
fprintf(fid, '%% [pix]\t[pix]\t[pix]\t[mm]\t[mm]\t[mm]\t[1]\t[?]\n');

fprintf(fid, ['function data = ' filename '\n\n']);
fprintf(fid, 'data\t= [\n\n');

fprintf(fid, '%g\t%g\t%g\t%g\t%g\t%g\t%g\t%g\n', rand(2048, 8));

fprintf(fid, '];\n');
fclose(fid);
end

% Load M-Function

cd 'D:\dump\'
data = eval(filename);
xyz = mean(data);

end

toc
feature memstats;

--------------000605070703060804090708--

Armin Mueller <arm.in@web.de> writes:

> Huy wrote:
>
>> In my experience, you could debug and monitor your MATLAB memory by
>> the MATLAB monitoring tool (Windows only) which is available at
>> MATLAB File Exchange [...]
>
> Thanks for all the hints about debugging memory usage in MATLAB. I'll
> try that!
>
> Meanwhile, I have created a small and obvious example to show how the
> memory leak occurs. All results in my program are written as protocol
> in an M-file. Later they are processed to get some statistics (see
> attachment). There are two flavors, one protocol is written in
> M-Script format and the other one in M-Function format.

I haven't tried your samples, but if confirmed, I agree this is a bug.

All the same, saving data as an M-file is not a good idea, and I'm not
surprised that dynamically generated M-files cause some sort of
problem. Why not save your data into a data file (either ASCII, or
binary, or native MAT) and load it when you need it?

-Peter

You are simultaneously running into two 'features' not exactly a memory leak.

1. Matlab stores all mfile that have been run in memory until cleared.
2. Windows never gives back memory from the standard heap used for objects less then around 512Kb.

The solution to your problem is to clear each m-file from memory after running it add
clear(filename);
after the eval statement.

Now the heap memory is reused on each iteration and never grows in the first place.


"Armin Mueller" <arm.in@web.de> wrote in message news:fllf6d$j0t$1@news2.rz.uni-karlsruhe.de...
> Huy wrote:
>
>> In my experience, you could debug and monitor your MATLAB
>> memory by the MATLAB monitoring tool (Windows only) which is
>> available at MATLAB File Exchange [...]
>
> Thanks for all the hints about debugging memory usage in MATLAB. I'll
> try that!
>
> Meanwhile, I have created a small and obvious example to show how the
> memory leak occurs. All results in my program are written as protocol in
> an M-file. Later they are processed to get some statistics (see
> attachment). There are two flavors, one protocol is written in M-Script
> format and the other one in M-Function format.
>
> If you i.e. run the "leak_demo2s" on 32bit Windows XP, virtual size
> (measured using Process Explorer) will grow from 500MB to 1GB. There is
> no chance to clear up. When you do 10.000 loops instead of 1.000, the
> 3GB barrier will be hit.
>
> Using M-Function makes no difference.
>
> Cheers,
> Armin
>


--------------------------------------------------------------------------------


> function leak_demo2s
>
> feature memstats;
> tic
>
> for nfile=1:1000
>
> % Save als M-Script
>
> filename = ['data' num2str(nfile, '%04d') 's'];
>
> fid = fopen(['D:\dump\' filename '.m'], 'wt');
>
> if fid<0
> warning('leak_demo2:save', 'Could not save results.')
> else
> fprintf(fid, '%% Ergebnis der Auswertung der Testreihe\n');
> fprintf(fid, '%% %s\n', filename);
> fprintf(fid, '%% Auswertung vom %s\n\n', datestr(now));
>
> fprintf(fid, '%% cogx\tcogy\tdiam\tcogx\tcogy\tdiam\texcent\tgradient\n');
> fprintf(fid, '%% [pix]\t[pix]\t[pix]\t[mm]\t[mm]\t[mm]\t[1]\t[?]\n');
>
> fprintf(fid, 'data\t= [\n\n');
>
> fprintf(fid, '%g\t%g\t%g\t%g\t%g\t%g\t%g\t%g\n', rand(2048, 8));
>
> fprintf(fid, '];\n');
> fclose(fid);
> end
>
> % Load M-Script
>
> run(['D:\dump\' filename '.m'])
> xyz = mean(data);
>
> end
>
> toc
> feature memstats;
>


--------------------------------------------------------------------------------


> function leak_demo2f
>
> feature memstats;
> tic
>
> for nfile=1:1000
>
> % Save als M-Function
>
> filename = ['data' num2str(nfile, '%04d') 'f'];
>
> fid = fopen(['D:\dump\' filename '.m'], 'wt');
>
> if fid<0
> warning('leak_demo2:save', 'Could not save results.')
> else
> fprintf(fid, '%% Ergebnis der Auswertung der Testreihe\n');
> fprintf(fid, '%% %s\n', filename);
> fprintf(fid, '%% Auswertung vom %s\n\n', datestr(now));
>
> fprintf(fid, '%% cogx\tcogy\tdiam\tcogx\tcogy\tdiam\texcent\tgradient\n');
> fprintf(fid, '%% [pix]\t[pix]\t[pix]\t[mm]\t[mm]\t[mm]\t[1]\t[?]\n');
>
> fprintf(fid, ['function data = ' filename '\n\n']);
> fprintf(fid, 'data\t= [\n\n');
>
> fprintf(fid, '%g\t%g\t%g\t%g\t%g\t%g\t%g\t%g\n', rand(2048, 8));
>
> fprintf(fid, '];\n');
> fclose(fid);
> end
>
> % Load M-Function
>
> cd 'D:\dump\'
> data = eval(filename);
> xyz = mean(data);
>
> end
>
> toc
> feature memstats;
>

Armin Mueller <arm.in@web.de> wrote in message
<fllf6d$j0t$1@news2.rz.uni-karlsruhe.de>...
> This is a multi-part message in MIME format.
> --------------000605070703060804090708
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> Content-Transfer-Encoding: 7bit
>
> Huy wrote:
>
> > In my experience, you could debug and monitor your MATLAB
> > memory by the MATLAB monitoring tool (Windows only) which
is
> > available at MATLAB File Exchange [...]
>
> Thanks for all the hints about debugging memory usage in
MATLAB. I'll
> try that!
>
> Meanwhile, I have created a small and obvious example to show
how the
> memory leak occurs. All results in my program are written as
protocol in
> an M-file. Later they are processed to get some statistics
(see
> attachment). There are two flavors, one protocol is written
in M-Script
> format and the other one in M-Function format.
>
> If you i.e. run the "leak_demo2s" on 32bit Windows XP,
virtual size
> (measured using Process Explorer) will grow from 500MB to
1GB. There is
> no chance to clear up. When you do 10.000 loops instead of
1.000, the
> 3GB barrier will be hit.
>
> Using M-Function makes no difference.
>
> Cheers,
> Armin
>


Armin,

I don't think there are any memory leak.
When MATLAB run any .m file, this file will be loaded into
memory. We can use INMEM function to check.

In each loop in your program, you run different .m files

data0001s.m, data0002s.m, ... Each m file contains data, and
the mimimum memory for each data file is 2048 * 8 * 8 = 131072
bytes.

Therefore, if the number of iteration nfile = 1000, there are
at least 1000 files loaded into memory which consume at least
          131072 * 1000 = 131.072.000 bytes

And we can check by the following code

usedmem1 = system_dependent('CheckMallocMemoryUsage');
run(['C:\' filename '.m'])
usedmem2 = system_dependent('CheckMallocMemoryUsage');

usedmem2 - usedmem1

To avoid this error, it is better to clear the loaded functions
which are unnecessary at end of each loop.

fullname = fullfile(''D:\dump\', filename ,'.m');
clear(fullname)


Anh Huy Phan
RIKEN - BSI

Armin

> If you i.e. run the "leak_demo2s" on 32bit Windows XP,
> virtual size (measured using Process Explorer) will grow
> from 500MB to 1GB.
> There is no chance to clear up.
 
It is possible to clear up

    clear functions

Anh Huy Phan
RIKEN - BSI

Peter Boettcher wrote:

> [...] Why not save your data into a data file (either ASCII, or
> binary, or native MAT) and load it when you need it?

Yes, I guess MAT-files would be the best choice for working within
Matlab. Loading M-files and ASCII-files takes some time anyway.

I chose M-files because they can be loaded with one line of code, and
because I can *read* them by myself :-)

Cheers,
Armin

Philip Borghesani wrote:

> 1. Matlab stores all mfile that have been run in memory until
> cleared.

Correct.

> 2. Windows never gives back memory from the standard heap used for
> objects less then around 512Kb.

Wow, didn't know that!

> The solution to your problem is to clear each m-file from memory
> after running it add
> clear(filename);
> after the eval statement.

No. That's the point. It worked in older versions, but with R2007b I
suddenly have trouble. As I've written before

   clear functions

doesn't work. And as Huy proposed,

   fullname = fullfile(''D:\dump\', filename ,'.m');
   clear(fullname)

doesn't work either. With your suggestion

   clear(filename);

memory is freed again. Surprise! But where is the difference? All three
solutions should be more or less equivalent.

:-/
Armin

>
> clear functions
>

Should not be used anywhere but the command line becuase it will cause a large performance loss if it works.

>
> fullname = fullfile(''D:\dump\', filename ,'.m');
> clear(fullname)

This will not work because clear does not take full paths to things to clear.

> clear(filename);
>

This works even though there is a .m on the file name.


**

Issuing a clear functions from the command line after running the code will not work because of the windows issue mentioned earlier.