Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
best way to handle a large matrix

Subject: best way to handle a large matrix

From: Mathew

Date: 28 Aug, 2011 02:32:12

Message: 1 of 10

Hello All,

I have a data set of size (2000 X 2000 X 2000). As you can see the data is huge. The files are in (.hdf) format, i.e., 2000 data files of size/resolution (2000 X 2000). I am stacking them to get a 3d view. Of course, I run out of memory if I try to stack up the whole 2d data file at once. I have heard about how software's like Paraview etc can handle huge amounts of data compared to MATLAB. I have tried the same data sets on both MATLAB and Paraview, and both seem to be handling approximately the same amount of data.

My aim is to try and visualize the whole data (I can already visualize the partial data). I am sure others have tried several other software's to be able to visualize data of this volume. Just curious if anyone is aware of how to deal with this is MATLAB or using some other software's.

Thank you,
Mathew

Subject: best way to handle a large matrix

From: Matt J

Date: 28 Aug, 2011 05:18:11

Message: 2 of 10

"Mathew " <mathew99@gmail.com> wrote in message <j3c9bc$8j4$1@newscl01ah.mathworks.com>...
> Hello All,
>
> I have a data set of size (2000 X 2000 X 2000). As you can see the data is huge. The files are in (.hdf) format, i.e., 2000 data files of size/resolution (2000 X 2000). I am stacking them to get a 3d view. Of course, I run out of memory if I try to stack up the whole 2d data file at once. I have heard about how software's like Paraview etc can handle huge amounts of data compared to MATLAB.
================

I doubt it's a question of the software. It would take at least 60GB of RAM to hold a
2000x2000x2000 array and that's as uint8. I'm not aware of any commercial machine that offers that much RAM, but if there are, they will be expensive.

Subject: best way to handle a large matrix

From: Bruno Luong

Date: 28 Aug, 2011 06:35:30

Message: 3 of 10

"Matt J" wrote in message <j3cj2j$4oc$1@newscl01ah.mathworks.com>...

> I doubt it's a question of the software. It would take at least 60GB of RAM to hold a
> 2000x2000x2000 array and that's as uint8. I'm not aware of any commercial machine that offers that much RAM, but if there are, they will be expensive.

Some software might be able store the data on disc when it becomes big and work from there.

Possible but not an easy task with Matlab.

Bruno

Subject: best way to handle a large matrix

From: Matt J

Date: 28 Aug, 2011 22:37:11

Message: 4 of 10

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message <j3cnjh$gjn$1@newscl01ah.mathworks.com>...
>
> Some software might be able store the data on disc when it becomes big and work from there.
>
> Possible but not an easy task with Matlab.
===============

Well, there's MEMMAPFILE...

Subject: best way to handle a large matrix

From: Bruno Luong

Date: 28 Aug, 2011 22:53:08

Message: 5 of 10

"Matt J" wrote in message <j3efum$hl5$1@newscl01ah.mathworks.com>...
> "Bruno Luong" <b.luong@fogale.findmycountry> wrote in message
> Well, there's MEMMAPFILE...

In my experience, memmapfile is not a good method to scan a very large file by jumping with step (for example take a slice across the third dimension). Better come back to the basic fseek/fread...

Bruno

Subject: best way to handle a large matrix

From: elyo

Date: 29 Aug, 2011 01:19:11

Message: 6 of 10

Hi Mathew,

1- You need to have the 64-bit version of MATLAB.
2- You need to have as much RAM as possible.
3- You need to configure your Windows swap file, in such a way that:
   RAM SIZE + SWAP FILE SIZE > SIZE OF YOUR MATRIX
4- You can verify the memory available to MATLAB using the "feature memstats" command. What you see as VIRTUAL MEMORY is the sum of ram memory + swap file.
5- MATLAB uses column-major order to store matrices in memory. If you have a matrix M[i j k], make sure i is always in the innermost loop, k is always in the outermost loop, i.e., you must design your loops in such a way that left-most indices toggle faster. Otherwise the operating system must deal with a lot of memory swaps and the program will run slower.

That's the best you can do using a brute force approach... But I would suggest you to find creative ways to reduce memory requirements. For example, if you halve the size of each dimension and you reduce the color depth from 32-bits to 8-bits, you need 32 times less storage (I don't say you should do so, that's just an example). And if you can find an algorithm that can achieve your needs without loading everything at once, that's even better...

I hope I have been able to give you some ideas....

Best regards,
Elyo Ravuna

"Mathew " <mathew99@gmail.com> wrote in message <j3c9bc$8j4$1@newscl01ah.mathworks.com>...
> Hello All,
>
> I have a data set of size (2000 X 2000 X 2000). As you can see the data is huge. The files are in (.hdf) format, i.e., 2000 data files of size/resolution (2000 X 2000). I am stacking them to get a 3d view. Of course, I run out of memory if I try to stack up the whole 2d data file at once. I have heard about how software's like Paraview etc can handle huge amounts of data compared to MATLAB. I have tried the same data sets on both MATLAB and Paraview, and both seem to be handling approximately the same amount of data.
>
> My aim is to try and visualize the whole data (I can already visualize the partial data). I am sure others have tried several other software's to be able to visualize data of this volume. Just curious if anyone is aware of how to deal with this is MATLAB or using some other software's.
>
> Thank you,
> Mathew

Subject: best way to handle a large matrix

From: Bruno Luong

Date: 10 Sep, 2011 08:10:26

Message: 7 of 10

"Bruno Luong" <b.luong@fogale.findmycountry> wrote in message
>
> Some software might be able store the data on disc when it becomes big and work from there.
>
> Possible but not an easy task with Matlab.

New in release 2011B, Large Matrix can be stored on file and easily access partly with MATFILE object.

Bruno

Subject: best way to handle a large matrix

From: Mathew Thomas

Date: 11 Sep, 2011 18:41:28

Message: 8 of 10

That is great. I just read about the function MATFILE. Also, when you display a large matrix, it takes time for rotating the image in a 3d view or to go through slices of the image. I am curious what is the exact reason for this delay. I would imagine when you visualize slices, the data for the next slices have to be pulled up, so it takes a bit of time. But for a 3d view rotation, it looks like the GPU plays more of a role in making it slow. Or is it a combination of GPU and Virtual memory for both cases ??

Thanks,
Mathew

Subject: best way to handle a large matrix

From: John Melonakos

Date: 12 Sep, 2011 19:15:32

Message: 9 of 10

"Mathew Thomas" wrote in message <j4ivco$kij$1@newscl01ah.mathworks.com>...
> But for a 3d view rotation, it looks like the GPU plays more of a role in making it slow. Or is it a combination of GPU and Virtual memory for both cases ??

There is some Open-GL acceleration in MATLAB, but it's all CPU-based. If you want native GPU-based renderings, you can try the Graphics Library in Jacket. Here's a video of some of those demos: http://blog.accelereyes.com/blog/2011/09/01/jacket_demo/

Subject: best way to handle a large matrix

From: Malcolm Lidierth

Date: 14 Nov, 2011 14:02:11

Message: 10 of 10

The MAT-file utilities form the FEX have been updated to provided support for fread and for memmapfile and matfile objects. Example
y=nmatrix('myfile.mat', 'x');
memory maps variable x from file and allows partial i/ousing standard MATLAB matrix syntax: y(1:10), y(:,:,1:5,:) etc.

See http://sourceforge.net/projects/waterloo/files/

The PDF compares speeds using load, fread and memmapfile in different circumstances.

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us