Thread Subject: System freeze usually after hours of heavy MATLAB load

I experience freezing of my computer, when running MATLAB, in case of matrix operations, for example large LU runs.
Sometimes the code freezes the PC immediatley, sometimes after hours.

I checked the RAM in the machine having the whole kit installed, and also module by module, found anyhow no error. Any other kind of stress tests (Prime95, Stability tests, burn-in tests) were running successfully overnights without freezes.

I have XP 64 bit with 8 GB of memory, R2009a 64 bit MATLAB, and AMD Phenom ii x4 quad-core processor. The operations which cause freezing, run of course on all cores, multithreaded. I dont know, if this can be some instability of the MKL used on AMD CPUs, or the machine itself.

Temperatures, and voltages of the system are stable, not overreacting.

Any hints, what to check ?

> Any hints, what to check ?

You did not overclock your CPU in the winter and are experiencing problems in the hot season, did you? ;)

Can you reproduce the crash, i.e.:
1) Boot your computer
2) Start Maltab
3) Run program
Does it always crash at the same point?

How is your memory usage right before the crash? Could it be that your program is eating all available physical memory and swap space?

I have experienced complete system freezes (on Linux though) when doing heavy computations. Limiting the amount of memory Matlab is allowed to use (only physical memory, no swap) seem to have solved the problem. Now Matlab stops with 'Out of memory' errors, in stead of feezing the system.

Matlab's memory management is not as good as I expected: freed variables do not always result in free system memory, not even free memory within Matlab. When doing a lot of computations involving a lot of memory, a 'clear all' will not return you the same amount of memory you had before starting Matlab. (looks like a memory-leak).

So, if your computations can be separated in sessions, it is better to close-and-restart Maltab.

Sebastiaan

> You did not overclock your CPU in the winter and are experiencing problems in the hot season, did you? ;)

Hi,

nothing is overclocked, and has never been either.
I thought also about issues of MATLAB reserving or allocating the memory, but the amount of memory, that this examples use, is not so much. One of the code samples even ran on my laptop with 4Gb of RAM.
It is hard to judge, if it is always the same point.
But I believe, that crashes at LU operations (x=A\b), when using BLAS routines.
Thats why I was curious about issues with the MKL.

Anyway, I also know about memory allocation problems in MATLAB, and restart indeed helps in many cases.

> But I believe, that crashes at LU operations (x=A\b), when using BLAS routines.
> Thats why I was curious about issues with the MKL.
>
> Anyway, I also know about memory allocation problems in MATLAB, and restart indeed helps in many cases.

There is 1 case I had strange issues with the BLAS/LAPACK libraries. I ran optimizations on different cases for days, which makes heavy use of the Cholesky decomposition.

After a while (thousands, ten-thousands of decompositions on as much different matrices), the results differed from a fresh run. I.e.:

% after some days
U = chol(A);
save A U

% open new matlab
load A U
W = chol(A);
compare W and U


W and U were different (this A is well-conditioned). I traced it back to ill-conditioned matrices. When trying to decompose ill-conditioned matrices, 'noise' is introduced somewhere (yes, this is VERY vague).

Doing a check:
if cond(A)<1e20 % have a look at rcond for faster condition checking
  U = chol(A);
 ...
solved the problem. It prevents ill-conditioned matrices to be decomposed and introduce noise, which resulted in well-conditioned matrices to be decomposed slightly different.

Sounds like a ridiculous story, but after the change, the long-run and single-run were identical.

However, it never crashed my Maltab or system (Matlab R2008, dual quadcore X5460 Xeon, RHEL5).

Hope it helps.

Sebastiaan

Yes, such things can happen because of the floating point arithmetics, and floating point speculations (if they are not completly safe) used in the libraries. That is normal.
Especially, when things are calculated multithreaded, and one can never know how the operations, and the memory access is distributed between the cores. We have to live with the fact, that our codes, running on nowadays' highly parallellized and pipelined CPUs, are not 100% deterministic.

Well, it seems, i have still have to hunt for the problem, or just simply live with this instability.

"Tamas " <tammo@freemail.hu> wrote in message <h9fmqf$6rl$1@fred.mathworks.com>...
> Yes, such things can happen because of the floating point arithmetics, and floating point speculations (if they are not completly safe) used in the libraries. That is normal.
> Especially, when things are calculated multithreaded, and one can never know how the operations, and the memory access is distributed between the cores. We have to live with the fact, that our codes, running on nowadays' highly parallellized and pipelined CPUs, are not 100% deterministic.
>
Still, it is strange that a fresh run gives different results than one in a long run. And yes, the oddities were very reproducable.

> Well, it seems, i have still have to hunt for the problem, or just simply live with this instability.
Are you sure your matrices are well conditioned?

In the case of my everyday numerical work, the matrices, which I deal with, are ill-conditioned, but effectively-well conditioned. It means, that usually the condition number is very high, and matlab gives a warning about nearly-singularity, but using LU i can still get good results. I wouldnt go into explanation on this, but this really works, and I havent had problems up to know with any computers.

But there were also test cases with well-conditioned matrices, in which case the system froze too.

"Tamas " <tammo@freemail.hu> wrote in message
news:h9fha9$8s8$1@fred.mathworks.com...
>
>> You did not overclock your CPU in the winter and are experiencing
>> problems in the hot season, did you? ;)
>
> Hi,
>
> nothing is overclocked, and has never been either.
> I thought also about issues of MATLAB reserving or allocating the memory,
> but the amount of memory, that this examples use, is not so much. One of
> the code samples even ran on my laptop with 4Gb of RAM.
> It is hard to judge, if it is always the same point.
> But I believe, that crashes at LU operations (x=A\b), when using BLAS
> routines.
> Thats why I was curious about issues with the MKL.
>
> Anyway, I also know about memory allocation problems in MATLAB, and
> restart indeed helps in many cases.

If you have a (small) section of code with which you can reproduce this
behavior more or less consistently, please send that code to Technical
Support along with the output of the VER function and as much information as
you can about your machine (OS, processor, memory, etc.) so we can
investigate the cause of this behavior.

--
Steve Lord
slord@mathworks.com
comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ

Ok,thanks for offer.
As soon as it appears again, I can try to reproduce it, and report it to the support center.


"Steven Lord" <slord@mathworks.com> wrote in message <h9ftnt$dpj$1@fred.mathworks.com>...
>
> "Tamas " <tammo@freemail.hu> wrote in message
> news:h9fha9$8s8$1@fred.mathworks.com...
> >
> >> You did not overclock your CPU in the winter and are experiencing
> >> problems in the hot season, did you? ;)
> >
> > Hi,
> >
> > nothing is overclocked, and has never been either.
> > I thought also about issues of MATLAB reserving or allocating the memory,
> > but the amount of memory, that this examples use, is not so much. One of
> > the code samples even ran on my laptop with 4Gb of RAM.
> > It is hard to judge, if it is always the same point.
> > But I believe, that crashes at LU operations (x=A\b), when using BLAS
> > routines.
> > Thats why I was curious about issues with the MKL.
> >
> > Anyway, I also know about memory allocation problems in MATLAB, and
> > restart indeed helps in many cases.
>
> If you have a (small) section of code with which you can reproduce this
> behavior more or less consistently, please send that code to Technical
> Support along with the output of the VER function and as much information as
> you can about your machine (OS, processor, memory, etc.) so we can
> investigate the cause of this behavior.
>
> --
> Steve Lord
> slord@mathworks.com
> comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ
>