MATLAB Central Newsreader - read large text files

read large text files

Anandhi — Wed, 28 Oct 2009 01:14:04 -0400

Hi ,

I have text files having 6 columns of data, but the number of rows is greater than 100000. I do not know the exact row number.

When I use this prog I am able to get upto 100000 rows. How to get the rows beyond this till the end of file?

block_size = 100000;
format = '%f %f %f %f %f %f';
file_id = fopen(fno{i});
cnt=0;
segarray = textscan(file_id, format, block_size);

thanks in advance for the support

anandhi

Re: read large text files

dpb — Wed, 28 Oct 2009 01:54:03 -0400

Anandhi wrote:
...
> When I use this prog I am able to get upto 100000 rows. How to get
> the rows beyond this till the end of file?
>
> block_size = 100000;
> format = '%f %f %f %f %f %f';
> file_id = fopen(fno{i});
> cnt=0;
> segarray = textscan(file_id, format, block_size);
...

Don't specify N and textscan() should read to EOF

Alternatively, see

doc textscan

and note one can call textscan repeatedly on the same fid and continue
from where left off.

Doc doesn't indicate it, but N=-1 in textread() is a flag for "read to
end of file"; one would presume that would have been implemented in
textscan() as well. Also, I'd presume inf would have the same effect.
I can't test these hypotheses as my version predates textscan().

--

Re: read large text files

anandhi — Wed, 28 Oct 2009 03:39:18 -0400

On Oct 27, 9:54 pm, dpb <n...@non.net> wrote:
> Anandhi wrote:
>
> ...> When I use this prog I am able to get upto 100000 rows. How to get
> > the rows beyond this till the end of file?
>
> > block_size = 100000;
> > format = '%f %f %f %f %f %f';
> > file_id = fopen(fno{i});
> > cnt=0;
> > segarray = textscan(file_id, format, block_size);
>
> ...
>
> Don't specify N and textscan() should read to EOF
>
> Alternatively, see
>
> doc textscan
>
> and note one can call textscan repeatedly on the same fid and continue
> from where left off.
>
> Doc doesn't indicate it, but N=-1 in textread() is a flag for "read to
> end of file"; one would presume that would have been implemented in
> textscan() as well. Also, I'd presume inf would have the same effect.
> I can't test these hypotheses as my version predates textscan().
>
> --

Thanks for the response, however

when i call textscan repeatedly on the same fid and continue
it does continue upto 100000 lines only after which it does not
continue.

eg the file has 1179919 lines

segarray = textscan(file_id, format);
segarray1 = textscan(file_id, format);

I still get the size of segarray1 empty

Re: read large text files

Praetorian — Wed, 28 Oct 2009 04:04:56 -0400

On Oct 27, 9:39 pm, anandhi <anandhi.san...@gmail.com> wrote:
> On Oct 27, 9:54 pm, dpb <n...@non.net> wrote:
>
>
>
> > Anandhi wrote:
>
> > ...> When I use this prog I am able to get upto 100000 rows. How to get
> > > the rows beyond this till the end of file?
>
> > > block_size = 100000;
> > > format = '%f %f %f %f %f %f';
> > > file_id = fopen(fno{i});
> > > cnt=0;
> > > segarray = textscan(file_id, format, block_size);
>
> > ...
>
> > Don't specify N and textscan() should read to EOF
>
> > Alternatively, see
>
> > doc textscan
>
> > and note one can call textscan repeatedly on the same fid and continue
> > from where left off.
>
> > Doc doesn't indicate it, but N=-1 in textread() is a flag for "read to
> > end of file"; one would presume that would have been implemented in
> > textscan() as well. Also, I'd presume inf would have the same effect.
> > I can't test these hypotheses as my version predates textscan().
>
> > --
>
> Thanks for the response, however
>
> when i call textscan repeatedly on the same fid and continue
> it does continue upto 100000 lines only after which it does not
> continue.
>
> eg the file has 1179919 lines
>
> segarray = textscan(file_id, format);
> segarray1 = textscan(file_id, format);
>
> I still get the size of segarray1 empty

You could try using my CSVIMPORT submission from FEX (http://
tinyurl.com/yjctr57).

HTH,
Ashish.

Re: read large text files

Rune Allnor — Wed, 28 Oct 2009 06:32:03 -0400

On 28 Okt, 02:14, "Anandhi " <anan...@mathworks.com> wrote:
> Hi ,
>
> I have text files having 6 columns of data, but the number of rows is greater than 100000. I do not know the exact row number.
>
> When I use this prog I am able to get upto 100000 rows. How to get the rows beyond this till the end of file?

This is a trivial programming exercise on buffered I/O:

1) Decide on a buffer size
2) Clear block, set block ponter to star of buffer
3) Read till buffer is full or EOF is found
4) Process data
5) If EOF not yet found, repeat from 2)

Rune

Re: read large text files

dpb — Wed, 28 Oct 2009 13:45:32 -0400

anandhi wrote:
...
> when i call textscan repeatedly on the same fid and continue
> it does continue upto 100000 lines only after which it does not
> continue.
>
> eg the file has 1179919 lines
>
> segarray = textscan(file_id, format);
> segarray1 = textscan(file_id, format);
>
> I still get the size of segarray1 empty

I'd suspect there's a problem in the file at that point then. From
Remarks section in documentation--

"When textscan reads a specified file or string, it attempts to match
the data to the format string. If textscan fails to convert a data
field, it stops reading and returns all fields read before the failure."

Perhaps during your experimenting you accidentally wrote an EOF or some
other data to the file???

I'd suggest using a text-listing/viewing tool to verify the file is,
indeed, still pristine (my hunch is you'll find it isn't).

--