Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
help cumsum

Subject: help cumsum

From: francesco

Date: 20 Dec, 2010 17:22:07

Message: 1 of 12

Hello everyone,
I'have a proble with the function "cumsum". The problem is:
I have a large matrix,, and would like to apply the command cumSum. for example:

matt=rand(65536,10302)
x=cumsum(matt);
Out of memory. Type HELP MEMORY for your options.

there is an alternative method of cumsum?
 thanks !
 

Subject: help cumsum

From: Sean de

Date: 20 Dec, 2010 17:35:08

Message: 2 of 12

"francesco " <francesco_corrado@hotmail.it> wrote in message <ieo3bv$oqt$1@fred.mathworks.com>...
> Hello everyone,
> I'have a proble with the function "cumsum". The problem is:
> I have a large matrix,, and would like to apply the command cumSum. for example:
>
> matt=rand(65536,10302)
> x=cumsum(matt);
> Out of memory. Type HELP MEMORY for your options.
>
> there is an alternative method of cumsum?
> thanks !
>

-Buying more memory and updating to a 64bit processor

-Read it in a few columns at a time from a binary file, process it, and store it.

-The best alternative is to show us what you really want and see if there's a way to bypass the creation of a matrix that big.

Subject: help cumsum

From: ImageAnalyst

Date: 20 Dec, 2010 20:04:53

Message: 3 of 12

francesco :
That's not the error I get. I can't even make an array that big:

??? Error using ==> rand
Maximum variable size allowed by the program is exceeded.

Error in ==> test at 1
matt=rand(65536,10302)

That would be a 5.4 GB array of doubles. Are you sure your code even
makes it to the cumsum line?

Subject: help cumsum

From: Sean de

Date: 20 Dec, 2010 20:35:24

Message: 4 of 12

ImageAnalyst <imageanalyst@mailinator.com> wrote in message <0934c2ee-ae4d-43fc-ae1f-51235914c46f@l24g2000vby.googlegroups.com>...
> francesco :
> That's not the error I get. I can't even make an array that big:
>
> ??? Error using ==> rand
> Maximum variable size allowed by the program is exceeded.
>
> Error in ==> test at 1
> matt=rand(65536,10302)
>
> That would be a 5.4 GB array of doubles. Are you sure your code even
> makes it to the cumsum line?

You must be using a 32 bit system. This isn't a problem on 64bit with lots of RAM. I made the array and did the CUMSUM. It only took a few seconds...

Subject: help cumsum

From: someone

Date: 20 Dec, 2010 20:51:25

Message: 5 of 12

"Sean de " <sean.dewolski@nospamplease.umit.maine.edu> wrote in message <ieo44c$epu$1@fred.mathworks.com>...
> "francesco " <francesco_corrado@hotmail.it> wrote in message <ieo3bv$oqt$1@fred.mathworks.com>...
> > Hello everyone,
> > I'have a proble with the function "cumsum". The problem is:
> > I have a large matrix,, and would like to apply the command cumSum. for example:
> >
> > matt=rand(65536,10302)
> > x=cumsum(matt);
> > Out of memory. Type HELP MEMORY for your options.
> >
> > there is an alternative method of cumsum?
> > thanks !
> >
>
> -Buying more memory and updating to a 64bit processor
>
> -Read it in a few columns at a time from a binary file, process it, and store it.
>
> -The best alternative is to show us what you really want and see if there's a way to bypass the creation of a matrix that big.

% Or use a for loop perhaps.
% Something like:

summingBucket = 0;
for index = 1:maxIndex
   summingBucket = summinmgBucket + matt(index);
end

% Modify above for your particular case.

Subject: help cumsum

From: francesco

Date: 21 Dec, 2010 09:05:39

Message: 6 of 12

working 64-bit with 6GB of ram ..... I would like to calculate the CDF.

Subject: help cumsum

From: francesco

Date: 21 Dec, 2010 09:06:36

Message: 7 of 12

I'm sure you use huge data ... it's satellite images...

Subject: help cumsum

From: francesco

Date: 21 Dec, 2010 09:14:38

Message: 8 of 12

seems strange to me .... I work in 64 bit with 6GB of RAM .... is that possible?

Subject: help cumsum

From: Jan Simon

Date: 21 Dec, 2010 09:53:06

Message: 9 of 12

Dear Francesco,

> seems strange to me .... I work in 64 bit with 6GB of RAM .... is that possible?

I cannot understand, what might be strange.
You have 6 GB of RAM. Then you allocate 5.4 GB for your rand matrix. Finally you try to reserve another 5.4 GB for the reply of CUMSUM. Your OS needs some memory also. And I assume there are some other variables defined in Matlab.

To store the two 5.4 GB arrays, you need free contiguous(!) memory blocks of that size. So I assume you need about > 20 GB of RAM to solve your problem by this straight naive approach.

Kind regards, Jan

Subject: help cumsum

From: francesco

Date: 21 Dec, 2010 10:24:07

Message: 10 of 12

"Jan Simon" <matlab.THIS_YEAR@nMINUSsimon.de> wrote in message <iepte1$3k4$1@fred.mathworks.com>...
> Dear Francesco,
>
> > seems strange to me .... I work in 64 bit with 6GB of RAM .... is that possible?
>
> I cannot understand, what might be strange.
> You have 6 GB of RAM. Then you allocate 5.4 GB for your rand matrix. Finally you try to reserve another 5.4 GB for the reply of CUMSUM. Your OS needs some memory also. And I assume there are some other variables defined in Matlab.
>
> To store the two 5.4 GB arrays, you need free contiguous(!) memory blocks of that size. So I assume you need about > 20 GB of RAM to solve your problem by this straight naive approach.
>
> Kind regards, Jan

dear Jan, You can suggest a more intelligent?
thanks very much!

Subject: help cumsum

From: Jan Simon

Date: 21 Dec, 2010 12:45:32

Message: 11 of 12

Dear Francesco,

> dear Jan, You can suggest a more intelligent?

A more intelligent what?

You did not tell us what the CUMSUM(RAND) should be used for. Just tell us the purpose of your computations and perhaps there is a leaner method than a full block of summed random DOUBLEs.

But of course you can create this matrix without using twice the memory:
Create the CUMSUM(RAND) column by column!
But you will still need 5.4 GB of 6.0 GB installed RAM. There is not much space for any other calculations!

Kind regards, Jan

Subject: help cumsum

From: Sean de

Date: 21 Dec, 2010 15:25:22

Message: 12 of 12

"francesco " <francesco_corrado@hotmail.it> wrote in message <iepv87$15l$1@fred.mathworks.com>...
> "Jan Simon" <matlab.THIS_YEAR@nMINUSsimon.de> wrote in message <iepte1$3k4$1@fred.mathworks.com>...
> > Dear Francesco,
> >
> > > seems strange to me .... I work in 64 bit with 6GB of RAM .... is that possible?
> >
> > I cannot understand, what might be strange.
> > You have 6 GB of RAM. Then you allocate 5.4 GB for your rand matrix. Finally you try to reserve another 5.4 GB for the reply of CUMSUM. Your OS needs some memory also. And I assume there are some other variables defined in Matlab.
> >
> > To store the two 5.4 GB arrays, you need free contiguous(!) memory blocks of that size. So I assume you need about > 20 GB of RAM to solve your problem by this straight naive approach.
> >
> > Kind regards, Jan
>
> dear Jan, You can suggest a more intelligent?
> thanks very much

With 18GB or RAM on Mac 64bit, r2009b
tic,M = cumsum(rand(65536,10302));toc
20.68 seconds.

Here's the piece of information that I would like to know: why do you need the cumulative sum of a random matrix of this size? Statistically for a perfectly random number scheme, the mean value will be approximately 0.5 with this high a sampling rate (65536). Why can't you just multiply that by .5, and add a little noise? If I run
[n] = histc(M(end,:),30000:1000:45000);

%n = [0 0 10288 14 zeros(1,12)]
Just as we easily could predict.

Good Luck!
-Sean

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us