MATLAB Central Newsreader - load part of .mat (fast)

load part of .mat (fast)

Andreas Lobinger — Tue, 22 Jan 2008 16:06:59 -0500

Aloha,

in my field of work (simulation) you can get pretty fast
pretty large amounts of data.

In one of our tools therefore we use the append option of
save to save consecutive buffers to a .mat file.

The file then contains something like
buff_0
buff_1
buff_2
buff_3
etc.

with a structure told by whos
>> pre = whos('-file','/homes/lobingera/Channel.SINR.mat')

pre =

57x1 struct array with fields:
     name
     size
     bytes
     class
     global
     sparse
     complex
     nesting

>> pre(1)

ans =

        name: 'buff_0'
        size: [1260 50 100]
       bytes: 50400000
       class: 'double'
      global: 0
      sparse: 0
     complex: 0
     nesting: [1x1 struct]

We have a function to read it back again to memory by concatenating
the buffers.

At the moment i'm extending this facility to read only some of the
buffers.

The problem is:
load('filename.mat','buff_0','buff_1)
has somehow linear search time.
If i read n variables, it will need time n. If i read the file
completely it will need only time ~1.5.

Is there a 'fast' method to read consecutive parts of a .mat file?

Hoping for an answer and wishing a happy day,
LOBI

Re: load part of .mat (fast)

Peter Boettcher — Tue, 22 Jan 2008 18:04:08 -0500

Andreas Lobinger <newsreturns@biszumknie.de> writes:

> Aloha,
>
> in my field of work (simulation) you can get pretty fast
> pretty large amounts of data.
>
> In one of our tools therefore we use the append option of
> save to save consecutive buffers to a .mat file.
>
> The file then contains something like
> buff_0
> buff_1
> buff_2
> buff_3
> etc.

> The problem is:
> load('filename.mat','buff_0','buff_1)
> has somehow linear search time.
> If i read n variables, it will need time n. If i read the file
> completely it will need only time ~1.5.
>
> Is there a 'fast' method to read consecutive parts of a .mat file?

I recommend instead switching to a simple binary format. You will find
it much faster, and you can implement your own random reads with almost
no overhead. This is especially true since each buffer appears to be a
3D double-precision array.

fopen the file at the beginning of the simulation, then successively
fwrite each array. fclose at the end

When it's time to read, fopen the file. Then fseek to the offset of the
buffer you want.

-Peter

Re: load part of .mat (fast)

Malcolm Lidierth — Tue, 22 Jan 2008 18:24:01 -0500

Or use the "where" function available at
"http://www.mathworks.com/matlabcentral/fileexchange/loadFil
e.do?objectId=12250&objectType=FILE"
to get the data offsets for your variables.

You can then fread from a mat file or memory map the
variables with memmapfile.

Where requires files to be saved with the -v6 option (which
is faster anyway with large files- your speed problems
probably arise because load is uncompressing the first n-1
variables to locate the nth: save compresses the variable
header and the data in one go)

Re: load part of .mat (fast)

per isakson — Tue, 22 Jan 2008 18:56:01 -0500

Andreas Lobinger <newsreturns@biszumknie.de> wrote in
message <fn54b3$13p$1@daniel-new.mch.sbs.de>...
> Aloha,
>
> in my field of work (simulation) you can get pretty fast
> pretty large amounts of data.
>
> In one of our tools therefore we use the append option of
> save to save consecutive buffers to a .mat file.
>
> The file then contains something like
> buff_0
> buff_1
> buff_2
> buff_3
> etc.
>
> with a structure told by whos
> >> pre = whos('-
file','/homes/lobingera/Channel.SINR.mat')
>
> pre =
>
> 57x1 struct array with fields:
> name
> size
> bytes
> class
> global
> sparse
> complex
> nesting
>
> >> pre(1)
>
> ans =
>
> name: 'buff_0'
> size: [1260 50 100]
> bytes: 50400000
> class: 'double'
> global: 0
> sparse: 0
> complex: 0
> nesting: [1x1 struct]
>
> We have a function to read it back again to memory by
concatenating
> the buffers.
>
> At the moment i'm extending this facility to read only
some of the
> buffers.
>
> The problem is:
> load('filename.mat','buff_0','buff_1)
> has somehow linear search time.
> If i read n variables, it will need time n. If i read the
file
> completely it will need only time ~1.5.
>
> Is there a 'fast' method to read consecutive parts of
a .mat file?
>
> Hoping for an answer and wishing a happy day,
> LOBI

save with the '-v6' option is faster since it doesn't
compress data. / per