Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Large Arrays - Memory Problems

Subject: Large Arrays - Memory Problems

From: Thomas Britton

Date: 29 Apr, 2010 11:30:25

Message: 1 of 7

Dear All,

I'm running into serious memory problems with Matlab 2010a and large arrays. I have 32bit WinXP and have tweaked all the memory options as much as I can. My data sets are quite big and thus I'm running into memory problems still.

In particular I have an 4d double array which I'd like to have as (230,130,33,120) elements. I would like to work out an efficient way of splitting the array up/dealing with it that maintains both the first and second dimensions (as they are in effect 'images' of the data structure).

Unfortunately the program I am writing may change the '33' size dimension and I'd like to treat that more efficiently than I am now. Splitting the array into (230,130,5,120) blocks seems to make it manageable, as I can then perform subsequent operations but it requires me to explicitly call the array name and is both inelegant and prone to me making an indexing typo.

Are there any suggestions on how to deal with this sort of problem?

Thanks,
Ben

Subject: Large Arrays - Memory Problems

From: Rune Allnor

Date: 29 Apr, 2010 11:32:05

Message: 2 of 7

On 29 apr, 13:30, "Thomas Britton"
<benjamin.brit...@materials.ox.ac.remove.uk> wrote:

> Are there any suggestions on how to deal with this sort of problem?

Use some other programming language than matlab, that allows
you to use a data structure that is suited to your problem.

Rune

Subject: Large Arrays - Memory Problems

From: Thomas Britton

Date: 29 Apr, 2010 11:40:19

Message: 3 of 7

Rune Allnor <allnor@tele.ntnu.no> wrote in message <1aeac9c9-9d8b-45cd-aac9-c1d888b52148@y36g2000yqm.googlegroups.com>...
> On 29 apr, 13:30, "Thomas Britton"
> <benjamin.brit...@materials.ox.ac.remove.uk> wrote:
>
> > Are there any suggestions on how to deal with this sort of problem?
>
> Use some other programming language than matlab, that allows
> you to use a data structure that is suited to your problem.
>
> Rune

Unfortunatly, given the amount of time it would take to translate my program and implement a lot of linear programming solutions implemented in matlab this is infeasible.

Are there any solutions that more efficiently use the matlab memory or variable naming conventions?

Subject: Large Arrays - Memory Problems

From: Bruno Luong

Date: 29 Apr, 2010 11:48:03

Message: 4 of 7

"Thomas Britton" <benjamin.britton@materials.ox.ac.remove.uk> wrote in message <hrbqkh$g5q$1@fred.mathworks.com>...
> Dear All,
>
> I'm running into serious memory problems with Matlab 2010a and large arrays. I have 32bit WinXP and have tweaked all the memory options as much as I can. My data sets are quite big and thus I'm running into memory problems still.
>
> In particular I have an 4d double array which I'd like to have as (230,130,33,120) elements. I would like to work out an efficient way of splitting the array up/dealing with it that maintains both the first and second dimensions (as they are in effect 'images' of the data structure).

You might keep 3 dimensions together in 3D-arrays, and split the dimension (e.g. the third) in *cell* array.

For example instead of the array

M = rand(230,130,33,120);

Use rather

M = cell(1,33);
for k=1:33
  M{k} = rand(230,130,120);
end

The difference is the first need a big contiguous block, whereas splitting in cell will require a chunk of smaller blocks.

Bruno

Subject: Large Arrays - Memory Problems

From: Sean

Date: 29 Apr, 2010 11:50:19

Message: 5 of 7

"Thomas Britton" <benjamin.britton@materials.ox.ac.remove.uk> wrote in message <hrbqkh$g5q$1@fred.mathworks.com>...
> Dear All,
>
> I'm running into serious memory problems with Matlab 2010a and large arrays. I have 32bit WinXP and have tweaked all the memory options as much as I can. My data sets are quite big and thus I'm running into memory problems still.
>
> In particular I have an 4d double array which I'd like to have as (230,130,33,120) elements. I would like to work out an efficient way of splitting the array up/dealing with it that maintains both the first and second dimensions (as they are in effect 'images' of the data structure).
>
> Unfortunately the program I am writing may change the '33' size dimension and I'd like to treat that more efficiently than I am now. Splitting the array into (230,130,5,120) blocks seems to make it manageable, as I can then perform subsequent operations but it requires me to explicitly call the array name and is both inelegant and prone to me making an indexing typo.
>
> Are there any suggestions on how to deal with this sort of problem?
>
> Thanks,
> Ben

How much information would you lose by conversion to single?

Subject: Large Arrays - Memory Problems

From: Mark Shore

Date: 29 Apr, 2010 11:58:04

Message: 6 of 7

"Thomas Britton" <benjamin.britton@materials.ox.ac.remove.uk> wrote in message <hrbr73$np6$1@fred.mathworks.com>...
> Rune Allnor <allnor@tele.ntnu.no> wrote in message <1aeac9c9-9d8b-45cd-aac9-c1d888b52148@y36g2000yqm.googlegroups.com>...
> > On 29 apr, 13:30, "Thomas Britton"
> > <benjamin.brit...@materials.ox.ac.remove.uk> wrote:
> >
> > > Are there any suggestions on how to deal with this sort of problem?
> >
> > Use some other programming language than matlab, that allows
> > you to use a data structure that is suited to your problem.
> >
> > Rune
>
> Unfortunatly, given the amount of time it would take to translate my program and implement a lot of linear programming solutions implemented in matlab this is infeasible.
>
> Are there any solutions that more efficiently use the matlab memory or variable naming conventions?

What about the simple and relatively inexpensive option of changing to 64-bit Windows? Your arrays are comfortably within the size that can be easily handled with 8 GB RAM.

Subject: Large Arrays - Memory Problems

From: Thomas Britton

Date: 29 Apr, 2010 12:06:04

Message: 7 of 7

Mark - I have a 64bit computer next to my desk but it is overused for the bigger computations, this code reflects a post processor/visualisation which I'd like my fellow researches to implement on their local machines!

Bruno - I didn't know that cell arrays were did not require contiguous memory, this is an easy, simple and flexible solution to code, thanks!

Ben

"Mark Shore" <mshore@magmageosciences.ca> wrote in message <hrbs8c$2e0$1@fred.mathworks.com>...
> "Thomas Britton" <benjamin.britton@materials.ox.ac.remove.uk> wrote in message <hrbr73$np6$1@fred.mathworks.com>...
> > Rune Allnor <allnor@tele.ntnu.no> wrote in message <1aeac9c9-9d8b-45cd-aac9-c1d888b52148@y36g2000yqm.googlegroups.com>...
> > > On 29 apr, 13:30, "Thomas Britton"
> > > <benjamin.brit...@materials.ox.ac.remove.uk> wrote:
> > >
> > > > Are there any suggestions on how to deal with this sort of problem?
> > >
> > > Use some other programming language than matlab, that allows
> > > you to use a data structure that is suited to your problem.
> > >
> > > Rune
> >
> > Unfortunatly, given the amount of time it would take to translate my program and implement a lot of linear programming solutions implemented in matlab this is infeasible.
> >
> > Are there any solutions that more efficiently use the matlab memory or variable naming conventions?
>
> What about the simple and relatively inexpensive option of changing to 64-bit Windows? Your arrays are comfortably within the size that can be easily handled with 8 GB RAM.

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us