Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Storing a complicated data structure

Subject: Storing a complicated data structure

From: Toan Cao

Date: 15 Nov, 2013 03:54:06

Message: 1 of 4

Hello,

I have an 2D array A consisting of M rows and N columns (A =[MxN]). In turn, each element of A stores a 2D coordinate (ex: A(i,j) =[x, y]).
In Matlab, what is the optimal way for storing this data structure ?
In my case, A is a huge 2D array (M>30000, N>30000).
Looking forwards to your help.
Thanks in advance!

Toan

Subject: Storing a complicated data structure

From: Qixing

Date: 15 Nov, 2013 11:36:07

Message: 2 of 4

"Toan Cao" <toancv3010@gmail.com> wrote in message <l645su$8pl$1@newscl01ah.mathworks.com>...
> Hello,
>
> I have an 2D array A consisting of M rows and N columns (A =[MxN]). In turn, each element of A stores a 2D coordinate (ex: A(i,j) =[x, y]).
> In Matlab, what is the optimal way for storing this data structure ?
> In my case, A is a huge 2D array (M>30000, N>30000).
> Looking forwards to your help.
> Thanks in advance!
>
> Toan

you can save to .mat format and load it into memory when u need it. i guess u want to save in ascii format. maybe u can save each [x,y] to a row, and the first row will be the value of row/col number of A (ie. M, N).

Subject: Storing a complicated data structure

From: Toan Cao

Date: 15 Nov, 2013 14:16:06

Message: 3 of 4

"Qixing" wrote in message <l650v6$lo1$1@newscl01ah.mathworks.com>...
> "Toan Cao" <toancv3010@gmail.com> wrote in message <l645su$8pl$1@newscl01ah.mathworks.com>...
> > Hello,
> >
> > I have an 2D array A consisting of M rows and N columns (A =[MxN]). In turn, each element of A stores a 2D coordinate (ex: A(i,j) =[x, y]).
> > In Matlab, what is the optimal way for storing this data structure ?
> > In my case, A is a huge 2D array (M>30000, N>30000).
> > Looking forwards to your help.
> > Thanks in advance!
> >
> > Toan
>
> you can save to .mat format and load it into memory when u need it. i guess u want to save in ascii format. maybe u can save each [x,y] to a row, and the first row will be the value of row/col number of A (ie. M, N).

Sorry, i mis-explain what i mean.
My array A is used to run in my algorithm. Because it is so big, so i want to have optimal way to store this data (not save it) to reduce computation time. I already tried with structure of Matlab, but computing time when i act with this data is very large.
Thanks!

Subject: Storing a complicated data structure

From: Steven Lord

Date: 15 Nov, 2013 15:15:53

Message: 4 of 4


"Toan Cao" <toancv3010@gmail.com> wrote in message
news:l65ab6$puh$1@newscl01ah.mathworks.com...
> "Qixing" wrote in message <l650v6$lo1$1@newscl01ah.mathworks.com>...
>> "Toan Cao" <toancv3010@gmail.com> wrote in message
>> <l645su$8pl$1@newscl01ah.mathworks.com>...
>> > Hello,
>> >
>> > I have an 2D array A consisting of M rows and N columns (A =[MxN]). In
>> > turn, each element of A stores a 2D coordinate (ex: A(i,j) =[x, y]). In
>> > Matlab, what is the optimal way for storing this data structure ?
>> > In my case, A is a huge 2D array (M>30000, N>30000).
>> > Looking forwards to your help.
>> > Thanks in advance!
>> >
>> > Toan
>>
>> you can save to .mat format and load it into memory when u need it. i
>> guess u want to save in ascii format. maybe u can save each [x,y] to a
>> row, and the first row will be the value of row/col number of A (ie. M,
>> N).
>
> Sorry, i mis-explain what i mean.
> My array A is used to run in my algorithm. Because it is so big, so i want
> to have optimal way to store this data (not save it) to reduce computation
> time. I already tried with structure of Matlab, but computing time when i
> act with this data is very large.
> Thanks!

I have the same question Nasser did: "It depends also if you need all the
data in RAM at same time or not. You did not say." I would strongly suggest,
if possible, that you use some type of block processing: operate on say
30000-by-1000 chunks at a time.

Another important question is whether A is full or sparse. If only a few
elements of A are nonzero, consider storing it as two sparse matrices.

http://www.mathworks.com/help/matlab/sparse-matrices.html

If you do need to store a full data matrix, you could make things a _little_
better than Nasser's suggestion by storing the coordinates as complex
variables. That will require two contiguous blocks of 7 GB each (one for
real part, one for imaginary) rather than one contiguous block of 14 GB.
That's still a LOT of memory.

You may also be interested in parallel computing with your data distributed
among the machines in a cluster.

http://www.mathworks.com/help/distcomp/distribute-arrays.html

--
Steve Lord
slord@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us