MATLAB Central Newsreader - processing extremely long data file sequentially?

processing extremely long data file sequentially?

huhua — Sat, 01 Mar 2008 01:54:05 -0500

Hi all,

Let's say a CSV file has tens of millions lines and each line has many
columns.

I actually wanted to browse through it line by line (except the first line,
which is the headline),

and I need to cut most of the lines and columns out, and only use a few
lines and columns.

I am estimating that out of these tens of millions of lines, I only need to
retain tens of thousands of lines.

But I need to process them and cut the non-useful lines out.

Even Excel 2007 refused to load the file. Matlab crashed several times when
I tried to load.

What do I do?

Is there a "textread", "textscan", "csvread" file that can read it line by
line and sequentially?

I think it is important for the program to keep a relative pointer in the
CSV file so that after each line is read and processed, we can move to the
next line.

And I just need to sequentially write out another output file to take the
filtered lines.

Of course the benefit of "textread", "textscan", "csvread" is that they can
parse formated strings, including both text and numbers... that's
important...

Any ideas?

Thanks

Re: processing extremely long data file sequentially?

Paul — Sat, 01 Mar 2008 03:30:20 -0500

"huhua" <lunamoonmoon@gmail.com> wrote in message
<fqacv8$et8$1@news.Stanford.EDU>...
> Hi all,
>
> Let's say a CSV file has tens of millions lines and each
line has many
> columns.
>
> I actually wanted to browse through it line by line
(except the first line,
> which is the headline),
>
> and I need to cut most of the lines and columns out, and
only use a few
> lines and columns.
>
> I am estimating that out of these tens of millions of
lines, I only need to
> retain tens of thousands of lines.
>
> But I need to process them and cut the non-useful lines out.
>
> Even Excel 2007 refused to load the file. Matlab crashed
several times when
> I tried to load.
>
> What do I do?
>
> Is there a "textread", "textscan", "csvread" file that can
read it line by
> line and sequentially?
>
> I think it is important for the program to keep a relative
pointer in the
> CSV file so that after each line is read and processed, we
can move to the
> next line.
>
> And I just need to sequentially write out another output
file to take the
> filtered lines.
>
> Of course the benefit of "textread", "textscan", "csvread"
is that they can
> parse formated strings, including both text and numbers...
that's
> important...
>
> Any ideas?
>
> Thanks
>
>
>
>
>
>
>

help fgetl

Re: processing extremely long data file sequentially?

Andres Toennesmann — Sun, 09 Mar 2008 15:11:03 -0400

Re: processing extremely long data file sequentially?

NZTideMan — Mon, 10 Mar 2008 04:43:40 -0400

On Mar 10, 4:11=A0am, "Andres Toennesmann" <rant...@werb.de> wrote:
> "huhua" <lunamoonm...@gmail.com> wrote in message
>
> <fqacv8$et...@news.Stanford.EDU>...> Hi all,
>
> > Let's say a CSV file has tens of millions lines and each
> line has many
> > columns.
>
> > I actually wanted to browse through it line by line
>
> (except the first line,
>
> > which is the headline),
>
> > and I need to cut most of the lines and columns out, and
> only use a few
> > lines and columns.
> > []
>
> If the csv contains mainly numeric data below the header
> line, you may try txt2mat from the file exchange with its
> 'RowRange' and 'FilePos' arguments (see Help, esp. Example
> 5). This should be vastly quicker than fgetl.
> Hth
> Andres

I'd use Fortran, not Matlab for this job.
Fortran was developed back in the days of Hollerith cards, in which
you loaded one card of data at a time, so it can handle such a problem
easily and very fast.