Path: news.mathworks.com!newsfeed-00.mathworks.com!newsfeed2.dallas1.level3.net!news.level3.com!postnews.google.com!j22g2000hsf.googlegroups.com!not-for-mail
From: Rune Allnor <allnor@tele.ntnu.no>
Newsgroups: comp.soft-sys.matlab
Subject: Re: how to save time loading data from file
Date: Tue, 5 Aug 2008 08:41:18 -0700 (PDT)
Organization: http://groups.google.com
Lines: 40
Message-ID: <9c1e7196-8752-4047-8d3d-1ff6cf8689b8@j22g2000hsf.googlegroups.com>
References: <g75abr$40d$1@fred.mathworks.com> <4b983eef-8f6c-4eb8-87fd-961d5f24413d@w7g2000hsa.googlegroups.com> 
NNTP-Posting-Host: 212.17.141.54
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Trace: posting.google.com 1217950879 14180 127.0.0.1 (5 Aug 2008 15:41:19 GMT)
X-Complaints-To: groups-abuse@google.com
NNTP-Posting-Date: Tue, 5 Aug 2008 15:41:19 +0000 (UTC)
Complaints-To: groups-abuse@google.com
Injection-Info: j22g2000hsf.googlegroups.com; posting-host=212.17.141.54; 
User-Agent: G2/1.0
X-HTTP-UserAgent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET 
Xref: news.mathworks.com comp.soft-sys.matlab:483747



On 5 Aug, 00:10, "ggk " <ggkm...@comcast.net> wrote:
> > The only thing that will speed things *significantly*
> > is to store the data on binary format. Not too long ago
> > I sped up the loading of 90 MB of ASCII files from ~45 s
> > to ~0.2 s by changing the storage format to binary. Not
> > only were the loading some 250x faster, I also saved some
> > 20% disk space by storing the data on binary format.
>
> > Rune
>
> Fortunately the one thing I know exactly for my
> application is the type of data, formatting and total #
> rows per ascii file. My file size is 1.3GB containing one
> double-precision number per row and 100M rows. Because I
> don't have enough memory to handle the 1.3GB of data, I've
> split the ascii file up into 10 files of 10M rows each.
>
> Wow, loading binary files goes 250x faster!? How can I
> convert my ascii file to binary? Any routines to do this
> that can be called from Matlab?

Well, if you can't configure the oscilloscope to store
directly to binary format, the damage has already been
done - once the data are stored to ASCII format one
necessarily needs to convert them back to binary.

In that case, do as was already suggested by somebody
else and read the files *once* from ASCII and store
the data e.g. to .mat files. You can do that from
within matlab. Remember, you don't have to sit idle
in front of the computer while it does its work.
Set up a script which converts the files and start it
before you go home and leave the computer to work
overnight.

If you can get your oscilloscope to store the data to
some 'naive'  binary format, FREAD is the function you
will want to use.

Rune