MATLAB Central Newsreader - Out of memory when allocating

Out of memory when allocating

Ida Haggstrom — Thu, 25 Sep 2008 10:43:01 -0400

Hi!
I'm trying to allocate some empty matrices that I will use in a for loop later on. They're quite big, and I keep getting the "Out of Memory" error message all the time, before I even started with my actual calculations!
Aside from the matrix I already have (matrix1, same size as empty ones), I want to allocate at first one matrix:

matrix2 = zeros(128,128,63,32);

After some work on matrix2 and 1, I then clear the original matrix1 and allocate a new one:

matrix3 = zeros(128,128,63,32);

Any this is where I get out of memory! I should only have 2 matrices in memory I guess. I changed my RAM from 2046 to to 4096. Can anyone give me some tips how to get around this problem?? Thanks!
/Ida

Re: Out of memory when allocating

Rune Allnor — Thu, 25 Sep 2008 11:09:13 -0400

On 25 Sep, 12:43, "Ida Haggstrom" <ida_haggst...@yahoo.se> wrote:
> Hi!
> I'm trying to allocate some empty matrices that I will use in a for loop later on. They're quite big, and I keep getting the "Out of Memory" error message all the time, before I even started with my actual calculations!

First, in matlab an 'empty matrice' is one which size is 0x0. From
the
context I udnderstand your matrices have large sizes, but are
probably
filled with zeros.

Initializing variables will allocatre memory, even if that memory has
not
yest been initialized by a program. So it is perfectly possible that
you
run out of memory in the allocation stage.

> Aside from the matrix I already have (matrix1, same size as empty ones), I want to allocate at first one matrix:
>
> matrix2 = zeros(128,128,63,32);

That's some 264 MBytes, assuming double-precision floating
point number format. Doesn't seem too bad.

> After some work on matrix2 and 1, I then clear the original matrix1 and allocate a new one:
>
> matrix3 = zeros(128,128,63,32);
>
> Any this is where I get out of memory! I should only have 2 matrices in memory I guess. I changed my RAM from 2046 to to 4096. Can anyone give me some tips how to get around this problem??

Several suggestions come to mind. One is to re-use the space
of matrix1 for whatever you use matrix3 for. They take up the
same space, so you can re-use that space once you are finished
with matrix1. Doing that is a bit risky, though, as you are balancing
on the RAM limit of your system. It doesn't take more than one
extra function call before the system blows up with another 'out of
memory' error.

The other suggestion is to break down the data flow. A 4D data
structure indicates to me that it is possible to structure the
data flow differently, to spend less space.

Utilizing the available space has always been a bottleneck
in large-volume data processing. The trick is to read the
data in in manageable chunks, so that each chunk fit
comprtably into available RAM, at the same time one
reduces the disc accesses in order to redeuce I/O overhead.

So I would try and restructure the computations into at
least one, maybe two, for-loops and do the computations
on a 2D or 3D array. That would significantly reduce RAM
requirements, safety (as I would save results to file in each
iteration) at the expense of a little bit longer run-time.

Rune

Re: Out of memory when allocating

Thomas Clark — Thu, 25 Sep 2008 11:28:02 -0400

Ida,

IDEA NUMBER 1

Lets do some basic maths. MATLAB uses the double precision data type - that means 8 bytes of memory are taken up for every number you use - thats 3 matrices x 128 x 128 x 63 x 32 = 99090432 numbers.

8 bytes x 99090432 numbers =>>> 756 MB

So actually all three of your matrices should fit well within your 4Gb of ram, even with other system processes running (I presume you're tried shutting down other programs etc that might be using memory)

This indicates that there might be a problem with the program. You can try using the whos command to determine what variables are taking up what amount of memory. It may be that you're preallocating in the wrong place - or multiple times, or you have a memory 'leak' (i.e. there's something in a loop that keeps growing... use 'whos' to identify variables that are likely candidates... then use size() command to print out the size of these variables - check its what its supposed to be.

IDEA NUMBER TWO

If you've heard of 'passing by reference' then be aware that MATLAB mostly but not always does this. So, say you allocate a matrix in the main program, then pass it to a subroutine:

% Define main function
function [] = IdasMain()

% Preallocate
matrix1 = zeros(128,128,63,32);

% Use a subfunction
result = IdasSub(matrix1)

end % END MAIN FUNCTION

If your subfunction uses matrix1 but does not alter its content in any way, you're fine. However, if your subfunction looks like this:

% Define a subfunction, or other m-file
function [result] = IdasSub(matrix1)

matrix1 = someOperation(matrix1);

end % END SUB FUNCTION

... Then it performs an operation on matrix1, but does not pass it back to the main function. As the operation is performed, a copy of matrix1 is made and the operation performed on that copy, instead of the operation being made on the existing variable. In that case, you end up with TWO variables stored in memory - both called matrix1 - one of these variables is known to the subfunction, the other known to the main function.

If this is your problem, you'll need to restructure.

I'm not too hot on this topic, but it's a possibility that if you write the subfunction as
[results, matrix1] = IdasSub(matrix1)
that its entirely passed by reference. Of course, you then end up with the altered version of matrix1 in your function.

IDEA NUMBER 3

If numerical precision isn't critical... then try altering the data type to 'single', with the command single(matrix1). You'd need to change the other variables that operate on matrix1 too. Doing this, you revert to single-precision accuracy, so use 4 bytes per number instead of 8 - thus your matrices take up only 378MB.

This could be ill advised - be sure to read up on the rules pertaining to data types found in the help documentation so you're sure you know what type of variable is what.

Beware that some (many!) functions can't work with single data types.

IDEA NUMBER FOUR

If a high proportion (at least half, preferably more) of the values in your matrix are zeros, then try using the sparse() format - check it out in the help documentation.

I hope this helps

Tom Clark

Re: Out of memory when allocating

Ida Haggstrom — Mon, 29 Sep 2008 10:05:04 -0400

Thanks both Tomas and Rune for all your help!
The tips are very good and I will surely check them out! For sure I'm running on the verge of memory breakdown... =)
I did find some documentation (Matlab Help under "resolving 'Out of memory' errors") on how to increase memory available to Matlab, by adding a line in the system startup document for windows. This actually worked, so now I can run my functions without errors. I suspect I'm still playing with fire though, and will probably have to follow your advice to do some for-loops instead. Thanks for your time guys! Cheers,
Ida

"Thomas Clark" <t.clarkremove_spam@cantab.net> wrote in message <gbfsk2$9k3$1@fred.mathworks.com>...
> Ida,
>
> IDEA NUMBER 1
>
> Lets do some basic maths. MATLAB uses the double precision data type - that means 8 bytes of memory are taken up for every number you use - thats 3 matrices x 128 x 128 x 63 x 32 = 99090432 numbers.
>
> 8 bytes x 99090432 numbers =>>> 756 MB
>
> So actually all three of your matrices should fit well within your 4Gb of ram, even with other system processes running (I presume you're tried shutting down other programs etc that might be using memory)
>
> This indicates that there might be a problem with the program. You can try using the whos command to determine what variables are taking up what amount of memory. It may be that you're preallocating in the wrong place - or multiple times, or you have a memory 'leak' (i.e. there's something in a loop that keeps growing... use 'whos' to identify variables that are likely candidates... then use size() command to print out the size of these variables - check its what its supposed to be.
>
>
> IDEA NUMBER TWO
>
> If you've heard of 'passing by reference' then be aware that MATLAB mostly but not always does this. So, say you allocate a matrix in the main program, then pass it to a subroutine:
>
> % Define main function
> function [] = IdasMain()
>
> % Preallocate
> matrix1 = zeros(128,128,63,32);
>
> % Use a subfunction
> result = IdasSub(matrix1)
>
> end % END MAIN FUNCTION
>
> If your subfunction uses matrix1 but does not alter its content in any way, you're fine. However, if your subfunction looks like this:
>
>
> % Define a subfunction, or other m-file
> function [result] = IdasSub(matrix1)
>
> matrix1 = someOperation(matrix1);
>
> end % END SUB FUNCTION
>
> ... Then it performs an operation on matrix1, but does not pass it back to the main function. As the operation is performed, a copy of matrix1 is made and the operation performed on that copy, instead of the operation being made on the existing variable. In that case, you end up with TWO variables stored in memory - both called matrix1 - one of these variables is known to the subfunction, the other known to the main function.
>
> If this is your problem, you'll need to restructure.
>
> I'm not too hot on this topic, but it's a possibility that if you write the subfunction as
> [results, matrix1] = IdasSub(matrix1)
> that its entirely passed by reference. Of course, you then end up with the altered version of matrix1 in your function.
>
> IDEA NUMBER 3
>
> If numerical precision isn't critical... then try altering the data type to 'single', with the command single(matrix1). You'd need to change the other variables that operate on matrix1 too. Doing this, you revert to single-precision accuracy, so use 4 bytes per number instead of 8 - thus your matrices take up only 378MB.
>
> This could be ill advised - be sure to read up on the rules pertaining to data types found in the help documentation so you're sure you know what type of variable is what.
>
> Beware that some (many!) functions can't work with single data types.
>
>
> IDEA NUMBER FOUR
>
> If a high proportion (at least half, preferably more) of the values in your matrix are zeros, then try using the sparse() format - check it out in the help documentation.
>
>
>
> I hope this helps
>
> Tom Clark
>
>
>