Thread Subject: processing extremely long data file sequentially?

Hi all,

Let's say a CSV file has tens of millions lines and each line has many
columns.

I actually wanted to browse through it line by line (except the first line,
which is the headline),

and I need to cut most of the lines and columns out, and only use a few
lines and columns.

I am estimating that out of these tens of millions of lines, I only need to
retain tens of thousands of lines.

But I need to process them and cut the non-useful lines out.

Even Excel 2007 refused to load the file. Matlab crashed several times when
I tried to load.

What do I do?

Is there a "textread", "textscan", "csvread" file that can read it line by
line and sequentially?

I think it is important for the program to keep a relative pointer in the
CSV file so that after each line is read and processed, we can move to the
next line.

And I just need to sequentially write out another output file to take the
filtered lines.

Of course the benefit of "textread", "textscan", "csvread" is that they can
parse formated strings, including both text and numbers... that's
important...

Any ideas?

Thanks

"huhua" <lunamoonmoon@gmail.com> wrote in message
<fqacv8$et8$1@news.Stanford.EDU>...
> Hi all,
>
> Let's say a CSV file has tens of millions lines and each
line has many
> columns.
>
> I actually wanted to browse through it line by line
(except the first line,
> which is the headline),
>
> and I need to cut most of the lines and columns out, and
only use a few
> lines and columns.
>
> I am estimating that out of these tens of millions of
lines, I only need to
> retain tens of thousands of lines.
>
> But I need to process them and cut the non-useful lines out.
>
> Even Excel 2007 refused to load the file. Matlab crashed
several times when
> I tried to load.
>
> What do I do?
>
> Is there a "textread", "textscan", "csvread" file that can
read it line by
> line and sequentially?
>
> I think it is important for the program to keep a relative
pointer in the
> CSV file so that after each line is read and processed, we
can move to the
> next line.
>
> And I just need to sequentially write out another output
file to take the
> filtered lines.
>
> Of course the benefit of "textread", "textscan", "csvread"
is that they can
> parse formated strings, including both text and numbers...
that's
> important...
>
> Any ideas?
>
> Thanks
>
>
>
>
>
>
>

help fgetl

"huhua" <lunamoonmoon@gmail.com> wrote in message
<fqacv8$et8$1@news.Stanford.EDU>...
> Hi all,
>
> Let's say a CSV file has tens of millions lines and each
line has many
> columns.
>
> I actually wanted to browse through it line by line
(except the first line,
> which is the headline),
>
> and I need to cut most of the lines and columns out, and
only use a few
> lines and columns.

> []

If the csv contains mainly numeric data below the header
line, you may try txt2mat from the file exchange with its
'RowRange' and 'FilePos' arguments (see Help, esp. Example
5). This should be vastly quicker than fgetl.
Hth
Andres

On Mar 10, 4:11=A0am, "Andres Toennesmann" <rant...@werb.de> wrote:
> "huhua" <lunamoonm...@gmail.com> wrote in message
>
> <fqacv8$et...@news.Stanford.EDU>...> Hi all,
>
> > Let's say a CSV file has tens of millions lines and each
> line has many
> > columns.
>
> > I actually wanted to browse through it line by line
>
> (except the first line,
>
> > which is the headline),
>
> > and I need to cut most of the lines and columns out, and
> only use a few
> > lines and columns.
> > []
>
> If the csv contains mainly numeric data below the header
> line, you may try txt2mat from the file exchange with its
> 'RowRange' and 'FilePos' arguments (see Help, esp. Example
> 5). This should be vastly quicker than fgetl.
> Hth
> Andres

I'd use Fortran, not Matlab for this job.
Fortran was developed back in the days of Hollerith cards, in which
you loaded one card of data at a time, so it can handle such a problem
easily and very fast.