Path: news.mathworks.com!newsfeed-00.mathworks.com!panix!bloom-beacon.mit.edu!llnews!53ab2750!not-for-mail
From: Peter Boettcher <boettcher@ll.mit.edu>
Newsgroups: comp.soft-sys.matlab
Subject: Re: load part of .mat (fast)
References: <fn54b3$13p$1@daniel-new.mch.sbs.de>
Message-ID: <muysl0p68kn.fsf@G99-Boettcher.llan.ll.mit.edu>
Organization: MIT Lincoln Laboratory
User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/23.0.0 (gnu/linux)
Cancel-Lock: sha1:5EOTaL861qkr2ikZYZDt5jfpOIs=
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Lines: 37
Date: Tue, 22 Jan 2008 13:04:08 -0500
NNTP-Posting-Host: 155.34.163.114
X-Complaints-To: news@ll.mit.edu
X-Trace: llnews 1201024549 155.34.163.114 (Tue, 22 Jan 2008 12:55:49 EST)
NNTP-Posting-Date: Tue, 22 Jan 2008 12:55:49 EST
Xref: news.mathworks.com comp.soft-sys.matlab:447025



Andreas Lobinger <newsreturns@biszumknie.de> writes:

> Aloha,
>
> in my field of work (simulation) you can get pretty fast
> pretty large amounts of data.
>
> In one of our tools therefore we use the append option of
> save to save consecutive buffers to a .mat file.
>
> The file then contains something like
> buff_0
> buff_1
> buff_2
> buff_3
> etc.

> The problem is:
> load('filename.mat','buff_0','buff_1)
> has somehow linear search time.
> If i read n variables, it will need time n. If i read the file
> completely it will need only time ~1.5.
>
> Is there a 'fast' method to read consecutive parts of a .mat file?

I recommend instead switching to a simple binary format.  You will find
it much faster, and you can implement your own random reads with almost
no overhead.  This is especially true since each buffer appears to be a
3D double-precision array.

fopen the file at the beginning of the simulation, then successively
fwrite each array.  fclose at the end

When it's time to read, fopen the file.  Then fseek to the offset of the
buffer you want.

-Peter