Path: news.mathworks.com!newsfeed-00.mathworks.com!newsfeed2.dallas1.level3.net!news.level3.com!postnews.google.com!d45g2000hsc.googlegroups.com!not-for-mail
From: Rune Allnor <allnor@tele.ntnu.no>
Newsgroups: comp.soft-sys.matlab
Subject: Re: how to save time loading data from file
Date: Tue, 5 Aug 2008 10:24:56 -0700 (PDT)
Organization: http://groups.google.com
Lines: 53
Message-ID: <f8a9f4dc-700e-4ebb-9258-97f836ded138@d45g2000hsc.googlegroups.com>
References: <g75abr$40d$1@fred.mathworks.com> <4b983eef-8f6c-4eb8-87fd-961d5f24413d@w7g2000hsa.googlegroups.com> 
NNTP-Posting-Host: 212.17.141.54
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Trace: posting.google.com 1217957096 1365 127.0.0.1 (5 Aug 2008 17:24:56 GMT)
X-Complaints-To: groups-abuse@google.com
NNTP-Posting-Date: Tue, 5 Aug 2008 17:24:56 +0000 (UTC)
Complaints-To: groups-abuse@google.com
Injection-Info: d45g2000hsc.googlegroups.com; posting-host=212.17.141.54; 
User-Agent: G2/1.0
X-HTTP-UserAgent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET 
Xref: news.mathworks.com comp.soft-sys.matlab:483776



On 5 Aug, 00:10, "ggk " <ggkm...@comcast.net> wrote:
> > The only thing that will speed things *significantly*
> > is to store the data on binary format. Not too long ago
> > I sped up the loading of 90 MB of ASCII files from ~45 s
> > to ~0.2 s by changing the storage format to binary. Not
> > only were the loading some 250x faster, I also saved some
> > 20% disk space by storing the data on binary format. ...

> Wow, loading binary files goes 250x faster!? How can I
> convert my ascii file to binary? Any routines to do this
> that can be called from Matlab?

Below is a script to demonstrates the timing differences
between ASCII and binary data. Output on my screen (R2006a):

Wrote ASCII data in 2.2344 seconds
Read ASCII data in 4.1719 seconds
Wrote binary data in 0.03125 seconds
Read binary data in 0.03125 seconds

File sizes:

test.txt (ASCII)  17579 kB
test.raw (Binary)  7813 kB

Rune

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
N = 1000000;
d1=randn(N,1);
t1=cputime;
save test.txt d1 -ascii
t2=cputime-t1;
disp(['Wrote ASCII data in ',num2str(t2),' seconds'])

t3=cputime;
d2=load('test.txt','-ascii');
t4=cputime-t3;
disp(['Read ASCII data in ',num2str(t4),' seconds'])

t5=cputime;
fid=fopen('test.raw','w');
fwrite(fid,d1,'double');
fclose(fid);
t6=cputime-t5;
disp(['Wrote binary data in ',num2str(t6),' seconds'])

t7=cputime;
fid=fopen('test.raw','r');
d3=fread(fid,'double');
fclose(fid);
t8=cputime-t7;
disp(['Wrote binary data in ',num2str(t8),' seconds'])