script runs out of memory on a new server

Hey,

I've got a question you might be able to help me with.

we have a script that we are running to analyze logs that where taken from measurements. the logfile is about 100 megabytes of text, space separated.(about 1 milion lines).

we recently changed the matlab server from a:

• 32-bit windows xp
• dual-core
• 4gb RAM (3 available)

to a server with:

• 64-bit windows 7 professional
• 8x Xeon processors (2.4GHz each)
• 16Gb RAM (swap file was 24gb at the time of the error, now set to 36gb)

irony is that the script ran fine, if a bit slow, on the 32-bit machine as well as on a laptop of one of our employees who has a private licence (also 32-bit) but when we did a testrun on the 64-bit server, the script stopped before completion with an out-of-memory error.

it had used all of the RAM (16GB) and part of the swap, for a total of 27GB used.

entering 'memory' in matlab at that point showed that matlab had a max allowed memory of 24Gb (which it obviously exceeded)

my question to you:

• do you have any idea what could cause this?
• is there any setting we can set to avoid this?

tonight we're going to run the script again, this time with the increased swap size, and hopefully it will run to completion. I'm just surprised that a more powerful server with more memory, wouldn't be able to run something a less powerfull computer with less memory can.

kind regards,

Ken

[Cedric]

Could you copy/paste your script here or is it too long?

The first thing that you could do is to open the task manager under performance, set a few break points in your code, execute it and locate the block where the memory usage increases dramatically. Using this approach iteratively should allow you to locate precisely.quickly which operation leads to the increase.

It's a cheap guess, but I would say that the increase will not happen linearly with the number of lines processed, but as a one shot event on a specific faulty operation.

[Friedrich]

Your statement "entering 'memory' in matlab at that point showed that matlab had a max allowed memory of 24Gb (which it obviously exceeded)"

cant be correct. This is the maximum possible array size and not the maximum allowed memory in total. The maximum allowed total memory is in theory 8TB on a 64bit machine running a 64bit MATLAB.

Since you are on Windows you can use:

feature memstats

to get a better overview about the memory situation (look at the Virtual Memory)

In addition an out of memory doesnt mean that there is no memory at all available. It simple means that the request amount of memory is not available in one piece. For that take a look at the Largest Contiguous Free Blocks.

[Ken Van Rafelghem]

hey unfortunatly, we can't post the script here since it's restricted.

we'll try the suggested solutions and let you know what caused it.

thanks

Answer 1

[per isakson]

Are you aware of the memory option of the profiler?

    profile('-memory','on')

See: Undocumented profiler options

It might be worthwhile to analyze the script with a sample of the input data.