Path: news.mathworks.com!newsfeed-00.mathworks.com!newsfeed2.dallas1.level3.net!news.level3.com!postnews.google.com!p8g2000yqb.googlegroups.com!not-for-mail
From: Rune Allnor <allnor@tele.ntnu.no>
Newsgroups: comp.soft-sys.matlab
Subject: Re: really big data files
Date: Sun, 8 Nov 2009 11:31:35 -0800 (PST)
Organization: http://groups.google.com
Lines: 18
Message-ID: <dfca570a-9e21-4622-bdea-69768c9d26b4@p8g2000yqb.googlegroups.com>
References: <hd75si$m75$1@fred.mathworks.com>
NNTP-Posting-Host: 77.17.87.103
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Trace: posting.google.com 1257708695 14700 127.0.0.1 (8 Nov 2009 19:31:35 GMT)
X-Complaints-To: groups-abuse@google.com
NNTP-Posting-Date: Sun, 8 Nov 2009 19:31:35 +0000 (UTC)
Complaints-To: groups-abuse@google.com
Injection-Info: p8g2000yqb.googlegroups.com; posting-host=77.17.87.103; 
	posting-account=VAp5gAkAAAAmkCze5hvZtMeedpZWNthI
User-Agent: G2/1.0
X-HTTP-UserAgent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; 
	Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; 
	.NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 1.1.4322),gzip(gfe),gzip(gfe)
Xref: news.mathworks.com comp.soft-sys.matlab:583395


On 8 Nov, 20:24, "Jon Shultz" <jjddshu...@yahoo.com> wrote:
> I'm trying to read in a datafile that's really big (>2GB) in sections that are a couple hundred thousand lines long each.  I need to know how many lines are in the parent file first.  
>
> I have a routine now that does it like this:
> totlines=0;
> while ~feof(fid)
>     line=fgetl(fid);
>     totlines=totlines+1;
> end
>
> This does well with the memory part, but takes forever.  There has got to be a more efficient way to do this, but I'm stuck.

Read the file in larger batches than a single line.

Rune