Path: news.mathworks.com!not-for-mail
From: <HIDDEN>
Newsgroups: comp.soft-sys.matlab
Subject: Re: Out of Memory
Date: Sat, 4 Apr 2009 01:16:02 +0000 (UTC)
Organization: University College Dublin
Lines: 106
Message-ID: <gr6cci$k50$1@fred.mathworks.com>
References: <gr5ega$t7f$1@fred.mathworks.com> <gr5k49$akq$1@fred.mathworks.com>
Reply-To: <HIDDEN>
NNTP-Posting-Host: webapp-03-blr.mathworks.com
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 8bit
X-Trace: fred.mathworks.com 1238807762 20640 172.30.248.38 (4 Apr 2009 01:16:02 GMT)
X-Complaints-To: news@mathworks.com
NNTP-Posting-Date: Sat, 4 Apr 2009 01:16:02 +0000 (UTC)
X-Newsreader: MATLAB Central Newsreader 87230
Xref: news.mathworks.com comp.soft-sys.matlab:530223


"Romeo " <romeo_aristogatto@yahoo.com> wrote in message <gr5k49$akq$1@fred.mathworks.com>...
> "Derek O'Connor" <derekroconnor@eircom.net> wrote in message <gr5ega$t7f$1@fred.mathworks.com>...
> > Can anyone explain the following behavior on a 16GB machine. Notice that the memory error occurs in two different places : `plus' and 'mtimes'.
> > 
> > >> clear all;
> > 
> > >> n = 2.5*10^4;
> > 
> > % --- GBytes = 8*n^2/10^9;
> > 
> > % --- Memory needed for each nxn matrix = 5GB;
> > 
> > >> C = rand(n,n);  D = rand(n,n);
> > 
> > % ---  WTM records total memory in use 12.6GB.  2.6 (OS + ?) +(5+5) matrices: checks
> > 
> > >> whos
> >   Name          Size                    Bytes  Class     Attributes
> > 
> >   C         25000x25000            5000000000  double              
> >   D         25000x25000            5000000000  double              
> >   n             1x1                         8  double              
> > 
> > >> memory
> > Maximum possible array:               3498 MB (3.667e+009 bytes) *
> > Memory available for all arrays:      3498 MB (3.667e+009 bytes) *
> > Memory used by MATLAB:                9929 MB (1.041e+010 bytes)
> > Physical Memory (RAM):               16381 MB (1.718e+010 bytes)
> > 
> > 
> > >> C = 2*C;
> > ??? Error using ==> mtimes
> > Out of memory. Type HELP MEMORY for your options
> > 
> > >> C = C+C;
> > ??? Error using ==> plus
> > Out of memory. Type HELP MEMORY for your options.
> > 
> > for i = 1:n, for j = 1:n, C(i,j) = 2*C(i,j);end;end;  % --- works ok ---
> > >> % --- Interrupted at keyboard after 5mins ---
> > 
> > >>for i = 1:n, for j = 1:n, C(i,j) = C(i,j)+C(i,j);end;end;  % --- works ok ---
> > >> % --- Interrupted at keyboard after 5mins ---
> > 
> > >> ver
> > -------------------------------------------------------------------------------------
> > MATLAB Version 7.6.0.324 (R2008a)
> > MATLAB License Number: 
> > Operating System: Microsoft Windows Vista Version 6.0 (Build 6001: Service Pack 1)
> > Java VM Version: Java 1.6.0 with Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM mixed mode
> > -------------------------------------------------------------------------------------
> > 
> > Derek O'Connor
> 
> Well, even if you would not compute those operations you are near the limit of your phisical ram just with matrix storage ... and that's not the best for ram performance. Second, there are large possibilities that you don't need all that amount of data, because those that really need don't have these problems =))) 
> Whatever, in the C= operations(C) istruction you need a copy of C, so your extimated memory usage would be 15 gb at least, and crumbs waste memory too...  
> Third, if you really want to work with all those data try to load only some data in memory and keep the whole matrix in a file, 
> and last, you really need a double random number? Try to recast to something less expensive.

Dear Romero,

Are you familiar with Shakespeare's phrase in Henry IV, Part 2, "Don't shoot the messenger"? --- Anyway, on to your 5 points :

1. 

"Well, even if you would not compute those operations you are near the
limit of your phisical ram just with matrix storage ... and that's not the best for ram performance."

The whole point of these simple tests was to see how much of the 16GB Matlab could use. Why pay for a machine with 16GB if the software cannot use it? Perhaps that should be: Why pay for the software if it cannot use your machine? Scilab 5.1 64-bit is even worse than Matlab -- it gives a memory allocation error at 5.6 GB.

I do not understand what you mean by  "... not the best for ram performance."

2. 

"Second, there are large possibilities that you don't need all that amount of data, because those that really need don't have these problems =)))" 

Who says I don't need all that data?  See these sites:

Tim Davis's site which has huge (sparse) matrices      http://www.cise.ufl.edu/research/sparse/matrices/

One of the test problems for the 9th Dimacs Challenge on the implementation of Shortest Path algorithms is the full USA map which has 23,947,347 nodes and 58,333,344 arcs.  That is a 24x10^6 X 24*10^6 adjacency matrix, but is sparse with about 2 to 3 non-zeros per row. This is typical of road networks. http://www.dis.uniroma1.it/~challenge9/download.shtml

I realise that this may sidetrack the discussion into sparse vs dense but the question is "What is the largest dense (or sparse) matrix that can be loaded into 16GB under Matlab",  with room left over for a couple of n-vectors. Remember: most iterative linear equation solvers are of the form  xnew = C*xold + d, which requires the storage of one matrix and three vectors and the only significant operation is an O(n^2) Blas 2 matrix-vector multiplication.

3.  You say "in the C = operations(C) instruction you need a copy of C".  Are you sure of that? Can anyone verify that statement? Can you explain these  lines of code?

>> C = 2*C; % --- out-of-memory failure ---
>> C = C+C; % --- out-of-memory failure ---
>>for i = 1:n, for j = 1:n, C(i,j) = 2*C(i,j);end;end;       % --- works ok ---
>>for i = 1:n, for j = 1:n, C(i,j) = C(i,j)+C(i,j);end;end;  % --- works ok ---

4. "Third, if you really want to work with all those data try to load only some data in memory and keep the whole matrix in a file,"

Yes indeed. This is really what I was trying to do. You see the actual matrix I'm working on is 500GB and sits on a 1TB Maxtor. I was trying to load into memory a small 10GB (10/500 = 2%) part of it. 

5. "... and last, you really need a double random number? Try to recast to something less expensive."

Oh dear! I've been here before. See James Tursa's and my posts on this subject here:
http://www.mathworks.com/matlabcentral/newsreader/view_thread/239186#637344


Of course, this whole question could be answered in a trice by a competent Mathworkser.

Regards,

Derek O'Connor.