Path: news.mathworks.com!newsfeed-00.mathworks.com!newsfeed2.dallas1.level3.net!news.level3.com!postnews.google.com!p25g2000hsf.googlegroups.com!not-for-mail
From: Rune Allnor <allnor@tele.ntnu.no>
Newsgroups: comp.soft-sys.matlab
Subject: Re: Out of memory when allocating
Date: Thu, 25 Sep 2008 04:09:13 -0700 (PDT)
Organization: http://groups.google.com
Lines: 54
Message-ID: <0e301dea-325d-428b-8caa-64cf5e90086d@p25g2000hsf.googlegroups.com>
References: <gbfpvl$99v$1@fred.mathworks.com>
NNTP-Posting-Host: 212.17.141.54
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Trace: posting.google.com 1222340953 20902 127.0.0.1 (25 Sep 2008 11:09:13 GMT)
X-Complaints-To: groups-abuse@google.com
NNTP-Posting-Date: Thu, 25 Sep 2008 11:09:13 +0000 (UTC)
Complaints-To: groups-abuse@google.com
Injection-Info: p25g2000hsf.googlegroups.com; posting-host=212.17.141.54; 
	posting-account=VAp5gAkAAAAmkCze5hvZtMeedpZWNthI
User-Agent: G2/1.0
X-HTTP-UserAgent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET 
	CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022),gzip(gfe),gzip(gfe)
Xref: news.mathworks.com comp.soft-sys.matlab:491937


On 25 Sep, 12:43, "Ida Haggstrom" <ida_haggst...@yahoo.se> wrote:
> Hi!
> I'm trying to allocate some empty matrices that I will use in a for loop later on. They're quite big, and I keep getting the "Out of Memory" error message all the time, before I even started with my actual calculations!

First, in matlab an 'empty matrice' is one which size is 0x0. From
the
context I udnderstand your matrices have large sizes, but are
probably
filled with zeros.

Initializing variables will allocatre memory, even if that memory has
not
yest been initialized by  a program. So it is perfectly possible that
you
run out of memory in the allocation stage.

> Aside from the matrix I already have (matrix1, same size as empty ones), I want to allocate at first one matrix:
>
> matrix2 = zeros(128,128,63,32);

That's some 264 MBytes, assuming double-precision floating
point number format. Doesn't seem too bad.

> After some work on matrix2 and 1, I then clear the original matrix1 and allocate a new one:
>
> matrix3 = zeros(128,128,63,32);
>
> Any this is where I get out of memory! I should only have 2 matrices in memory I guess. I changed my RAM from 2046 to to 4096. Can anyone give me some tips how to get around this problem??

Several suggestions come to mind. One is to re-use the space
of matrix1 for whatever you use matrix3 for. They take up the
same space, so you can re-use that space once you are finished
with matrix1. Doing that is a bit risky, though, as you are balancing
on the RAM limit of your system. It doesn't take more than one
extra function call before the system blows up with another 'out of
memory' error.

The other suggestion is to break down the data flow. A 4D data
structure indicates to me that it is possible to structure the
data flow differently, to spend less space.

Utilizing the available space has always been a bottleneck
in large-volume data processing. The trick is to read the
data in in manageable chunks, so that each chunk fit
comprtably into available RAM, at the same time one
reduces the disc accesses in order to redeuce I/O overhead.

So I would try and restructure the computations into at
least one, maybe two, for-loops and do the computations
on a 2D or 3D array. That would significantly reduce RAM
requirements, safety (as I would save results to file in each
iteration) at the expense of a little bit longer run-time.

Rune