Thread Subject: Memory overflow

Occasionally, I have run into a situation where I
accidentally tried to create a large array X when other
large arrays B, C,D,... are already in memory. This causes
a memory overflow error

 "Out of memory. Type HELP MEMORY for your options."

However, when I do "clear all" to free up memory,
my attempt to create X continues to fail with the same
error.

It's clearly a case of something getting stuck, because
when I restart MATLAB I have no further problems.


So, my question is, why might this be happening and what
can I do to avoid the hassle of restarting MATLAB to fix it?

This won't get you unstuck directly, but it is a useful
diagnostic if you are on Windows.

If you are using MATLAB R2008a, try using the function
MEMORY, which gives information about avaialable memory.

If you are using an earlier release, try using the undocumented:
>> feature MEMSTATS

It is quite a bit harder to parse through than MEMORY, but
can be useful nonetheless.

A few helpful resources:
http://www.mathworks.com/support/tech-notes/1100/1106.html
http://www.mathworks.com/support/tech-notes/1100/1107.html

-scott

Matt --

If you have not already done so, check out these two Tech Notes:

http://www.mathworks.com/support/tech-notes/1100/1106.html
http://www.mathworks.com/support/tech-notes/1100/1107.html

Sometimes, memory fragmentation can remains even after a
"clear all". This can be especially true on Windows 32-bit
platforms.

Let us know if these notes are helpful. If not, you'll need
to describe the behavior you are seeing in more detail.

Regards,
KEN

"Ken Atwell" <ken.REMOVEatwell@mathworksREMOVE.com> wrote
in message <g8ce91$obt$1@fred.mathworks.com>...
> Matt --
>
> If you have not already done so, check out these two Tech
Notes:
>
> http://www.mathworks.com/support/tech-notes/1100/1106.html
> http://www.mathworks.com/support/tech-notes/1100/1107.html
>
> Sometimes, memory fragmentation can remains even after a
> "clear all". This can be especially true on Windows 32-
bit
> platforms.
>
> Let us know if these notes are helpful. If not, you'll
need
> to describe the behavior you are seeing in more detail.
>
> Regards,
> KEN


OK. Well, I thank you both for the responses.

If it happens again, I'll try "pack". That's the only thing
in the technotes that I haven't tried yet.

I'm curious if clear then pack would help?

Gabe
http://wikis.ControlTheoryPro.com

"Matt " <mjacobson.removethis@xorantech.com> wrote in message <g8cini$aqm$1@fred.mathworks.com>...
> "Ken Atwell" <ken.REMOVEatwell@mathworksREMOVE.com> wrote
> in message <g8ce91$obt$1@fred.mathworks.com>...
> > Matt --
> >
> > If you have not already done so, check out these two Tech
> Notes:
> >
> > http://www.mathworks.com/support/tech-notes/1100/1106.html
> > http://www.mathworks.com/support/tech-notes/1100/1107.html
> >
> > Sometimes, memory fragmentation can remains even after a
> > "clear all". This can be especially true on Windows 32-
> bit
> > platforms.
> >
> > Let us know if these notes are helpful. If not, you'll
> need
> > to describe the behavior you are seeing in more detail.
> >
> > Regards,
> > KEN
>
>
> OK. Well, I thank you both for the responses.
>
> If it happens again, I'll try "pack". That's the only thing
> in the technotes that I haven't tried yet.
>


From Document 1107:
"Manage workspace fragmentation as a result of long term MATLAB usage (32-bit Windows only)

On 32-bit Windows, the workspace of MATLAB can become fragmented over time due to the fact that the Windows memory manager does not return blocks of certain types and sizes to the operating system. Clearing the MATLAB workspace does not fix this problem. The problem can be minimized by allocating the largest variables you need first. This can not address, however, the eventual fragmentation of the workspace that can occur over continual usage of MATLAB (over many days and weeks for example). The only solution to this is to save your work and restart MATLAB.

The pack command, which saves all variables to disk and loads them back, does not help with this situation."

Seems like this is a recognized problem and I am surprised that nothing has been done to correct this. It is frustrating as I wish to automate my simulation and let it run at least overnight. However, due to the memory fragmentation problem, I can't do that.

Why should I have to physically/manually do a quit and restart when perhaps a soft reload of Matlab command can solve the problem?

I am not well versed in memory management but it seems to me like a trivial problem that has been left alone even after numerous updates.

Do I have to write an external script to restart Matlab myself?

"Freddy " <nqueesan@dso.org.sg> wrote:
> Seems like this is a recognized problem and I am surprised that nothing has been done to correct this. It is frustrating as I wish to automate my simulation and let it run at least overnight. However, due to the memory fragmentation problem, I can't do that.
>
> Why should I have to physically/manually do a quit and restart when perhaps a soft reload of Matlab command can solve the problem?

Freddy --

My understanding is that this fragmentation cannot be "corrected" within the lifetime of the process itself. The MATLAB process needs to end and then be restarted. We can take another look, but I believe a solution that simultaneously works automatically, is robust, and minimally impacts overall performance is not a trivial undertaking.

The particular problem we speak of is of concern on 32-bit Windows, so I hope the problem will attenuate over time as the world slowly... slowly... migrates to 64-bit OS's.

I've experimented with a simple C program that exhibits the same behavior; drop me a line if you'd like to play with it.

KEN

Ken Atwell wrote:
 
> My understanding is that this fragmentation cannot be "corrected" within the lifetime of
> the process itself. The MATLAB process needs to end and then be restarted.

There have been numerous replacements for malloc() written . You just need to use one
of the many that automatically coalesce adjacent free slots.

I also have the same problem, although I am running matlab on 32-bit linux.
Aaron

"Ken Atwell" <ken.REMOVEatwell@mathworksREMOVE.com> wrote in message <gdahae$bcv$1@fred.mathworks.com>...
> "Freddy " <nqueesan@dso.org.sg> wrote:
> > Seems like this is a recognized problem and I am surprised that nothing has been done to correct this. It is frustrating as I wish to automate my simulation and let it run at least overnight. However, due to the memory fragmentation problem, I can't do that.
> >
> > Why should I have to physically/manually do a quit and restart when perhaps a soft reload of Matlab command can solve the problem?
>
> Freddy --
>
> My understanding is that this fragmentation cannot be "corrected" within the lifetime of the process itself. The MATLAB process needs to end and then be restarted. We can take another look, but I believe a solution that simultaneously works automatically, is robust, and minimally impacts overall performance is not a trivial undertaking.
>
> The particular problem we speak of is of concern on 32-bit Windows, so I hope the problem will attenuate over time as the world slowly... slowly... migrates to 64-bit OS's.
>
> I've experimented with a simple C program that exhibits the same behavior; drop me a line if you'd like to play with it.
>
> KEN