Path: news.mathworks.com!not-for-mail
From: <HIDDEN>
Newsgroups: comp.soft-sys.matlab
Subject: really big data files
Date: Sun, 8 Nov 2009 19:24:02 +0000 (UTC)
Organization: Macaulay Brown, Inc
Lines: 12
Message-ID: <hd75si$m75$1@fred.mathworks.com>
Reply-To: <HIDDEN>
NNTP-Posting-Host: webapp-02-blr.mathworks.com
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 8bit
X-Trace: fred.mathworks.com 1257708242 22757 172.30.248.37 (8 Nov 2009 19:24:02 GMT)
X-Complaints-To: news@mathworks.com
NNTP-Posting-Date: Sun, 8 Nov 2009 19:24:02 +0000 (UTC)
X-Newsreader: MATLAB Central Newsreader 1953538
Xref: news.mathworks.com comp.soft-sys.matlab:583394


I'm trying to read in a datafile that's really big (>2GB) in sections that are a couple hundred thousand lines long each.  I need to know how many lines are in the parent file first.  

I have a routine now that does it like this:
totlines=0;
while ~feof(fid)
    line=fgetl(fid);
    totlines=totlines+1;
end

This does well with the memory part, but takes forever.  There has got to be a more efficient way to do this, but I'm stuck.

Thanks!