Got Questions? Get Answers.
Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
load part of .mat (fast)

Subject: load part of .mat (fast)

From: Andreas Lobinger

Date: 22 Jan, 2008 16:06:59

Message: 1 of 4

Aloha,

in my field of work (simulation) you can get pretty fast
pretty large amounts of data.

In one of our tools therefore we use the append option of
save to save consecutive buffers to a .mat file.

The file then contains something like
buff_0
buff_1
buff_2
buff_3
etc.

with a structure told by whos
 >> pre = whos('-file','/homes/lobingera/Channel.SINR.mat')

pre =

57x1 struct array with fields:
     name
     size
     bytes
     class
     global
     sparse
     complex
     nesting

 >> pre(1)

ans =

        name: 'buff_0'
        size: [1260 50 100]
       bytes: 50400000
       class: 'double'
      global: 0
      sparse: 0
     complex: 0
     nesting: [1x1 struct]

We have a function to read it back again to memory by concatenating
the buffers.

At the moment i'm extending this facility to read only some of the
buffers.

The problem is:
load('filename.mat','buff_0','buff_1)
has somehow linear search time.
If i read n variables, it will need time n. If i read the file
completely it will need only time ~1.5.

Is there a 'fast' method to read consecutive parts of a .mat file?

Hoping for an answer and wishing a happy day,
LOBI

Subject: load part of .mat (fast)

From: Peter Boettcher

Date: 22 Jan, 2008 18:04:08

Message: 2 of 4

Andreas Lobinger <newsreturns@biszumknie.de> writes:

> Aloha,
>
> in my field of work (simulation) you can get pretty fast
> pretty large amounts of data.
>
> In one of our tools therefore we use the append option of
> save to save consecutive buffers to a .mat file.
>
> The file then contains something like
> buff_0
> buff_1
> buff_2
> buff_3
> etc.

> The problem is:
> load('filename.mat','buff_0','buff_1)
> has somehow linear search time.
> If i read n variables, it will need time n. If i read the file
> completely it will need only time ~1.5.
>
> Is there a 'fast' method to read consecutive parts of a .mat file?

I recommend instead switching to a simple binary format. You will find
it much faster, and you can implement your own random reads with almost
no overhead. This is especially true since each buffer appears to be a
3D double-precision array.

fopen the file at the beginning of the simulation, then successively
fwrite each array. fclose at the end

When it's time to read, fopen the file. Then fseek to the offset of the
buffer you want.

-Peter

Subject: load part of .mat (fast)

From: Malcolm Lidierth

Date: 22 Jan, 2008 18:24:01

Message: 3 of 4

Or use the "where" function available at
"http://www.mathworks.com/matlabcentral/fileexchange/loadFil
e.do?objectId=12250&objectType=FILE"
to get the data offsets for your variables.

You can then fread from a mat file or memory map the
variables with memmapfile.

Where requires files to be saved with the -v6 option (which
is faster anyway with large files- your speed problems
probably arise because load is uncompressing the first n-1
variables to locate the nth: save compresses the variable
header and the data in one go)

Subject: load part of .mat (fast)

From: per isakson

Date: 22 Jan, 2008 18:56:01

Message: 4 of 4

Andreas Lobinger <newsreturns@biszumknie.de> wrote in
message <fn54b3$13p$1@daniel-new.mch.sbs.de>...
> Aloha,
>
> in my field of work (simulation) you can get pretty fast
> pretty large amounts of data.
>
> In one of our tools therefore we use the append option of
> save to save consecutive buffers to a .mat file.
>
> The file then contains something like
> buff_0
> buff_1
> buff_2
> buff_3
> etc.
>
> with a structure told by whos
> >> pre = whos('-
file','/homes/lobingera/Channel.SINR.mat')
>
> pre =
>
> 57x1 struct array with fields:
> name
> size
> bytes
> class
> global
> sparse
> complex
> nesting
>
> >> pre(1)
>
> ans =
>
> name: 'buff_0'
> size: [1260 50 100]
> bytes: 50400000
> class: 'double'
> global: 0
> sparse: 0
> complex: 0
> nesting: [1x1 struct]
>
> We have a function to read it back again to memory by
concatenating
> the buffers.
>
> At the moment i'm extending this facility to read only
some of the
> buffers.
>
> The problem is:
> load('filename.mat','buff_0','buff_1)
> has somehow linear search time.
> If i read n variables, it will need time n. If i read the
file
> completely it will need only time ~1.5.
>
> Is there a 'fast' method to read consecutive parts of
a .mat file?
>
> Hoping for an answer and wishing a happy day,
> LOBI

save with the '-v6' option is faster since it doesn't
compress data. / per

Tags for this Thread

No tags are associated with this thread.

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us