Path: news.mathworks.com!not-for-mail
From: <HIDDEN>
Newsgroups: comp.soft-sys.matlab
Subject: Re: processing extremely long data file sequentially?
Date: Sat, 1 Mar 2008 03:30:20 +0000 (UTC)
Organization: University of Memphis
Lines: 61
Message-ID: <fqaikc$22k$1@fred.mathworks.com>
References: <fqacv8$et8$1@news.Stanford.EDU>
Reply-To: <HIDDEN>
NNTP-Posting-Host: webapp-02-blr.mathworks.com
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 8bit
X-Trace: fred.mathworks.com 1204342220 2132 172.30.248.37 (1 Mar 2008 03:30:20 GMT)
X-Complaints-To: news@mathworks.com
NNTP-Posting-Date: Sat, 1 Mar 2008 03:30:20 +0000 (UTC)
X-Newsreader: MATLAB Central Newsreader 233092
Xref: news.mathworks.com comp.soft-sys.matlab:454723



"huhua" <lunamoonmoon@gmail.com> wrote in message
<fqacv8$et8$1@news.Stanford.EDU>...
> Hi all,
> 
> Let's say a CSV file has tens of millions lines and each
line has many 
> columns.
> 
> I actually wanted to browse through it line by line
(except the first line, 
> which is the headline),
> 
> and I need to cut most of the lines and columns out, and
only use a few 
> lines and columns.
> 
> I am estimating that out of these tens of millions of
lines, I only need to 
> retain tens of thousands of lines.
> 
> But I need to process them and cut the non-useful lines out.
> 
> Even Excel 2007 refused to load the file. Matlab crashed
several times when 
> I tried to load.
> 
> What do I do?
> 
> Is there a "textread", "textscan", "csvread" file that can
read it line by 
> line and sequentially?
> 
> I think it is important for the program to keep a relative
pointer in the 
> CSV file so that after each line is read and processed, we
can move to the 
> next line.
> 
> And I just need to sequentially write out another output
file to take the 
> filtered lines.
> 
> Of course the benefit of "textread", "textscan", "csvread"
is that they can 
> parse formated strings, including both text and numbers...
that's 
> important...
> 
> Any ideas?
> 
> Thanks
> 
> 
> 
> 
> 
> 
> 

help fgetl