Thread Subject: import text file

Hi,

I have some problems regarding to import the text file to matlab.

My text file which i am importing has 4,000,000 points with 7 columns. When i try to import that in Matlab, it just gave me an error and basically slow down the whole computer. So, what is the best way to import this data?

Please let me know, if anyone has done this before.

thanks

On Thu, 14 May 2009 11:36:01 -0400, Tyng <juzmyluck06@aim.com> wrote:

> Hi,
>
> I have some problems regarding to import the text file to matlab.
>
> My text file which i am importing has 4,000,000 points with 7 columns.
> When i try to import that in Matlab, it just gave me an error and
> basically slow down the whole computer. So, what is the best way to
> import this data?
>
> Please let me know, if anyone has done this before.
>
> thanks

It usually helps to post the error message, I expect its the 'out of
memory' error in this case.

Do you need the entire data in the file for your processing? Could you
perhaps do the processing in parts?
For example, if you wanted to compute the MIN of a column, you do that by
reading in the data in chunks.

You might also be interested in looking at MEMMAPFILE:

Please find more details online at:
     http://www.mathworks.com/access/helpdesk/help/techdoc/ref/memmapfile.html
Or please paste this in the MATLAB command window:
     web([docroot,'/techdoc/ref/memmapfile.html'])

Hi

I work with text files of this size (containing only numeric data) and Matlab handles them nicely without any special efforts when your computer's memory is big enough :-). It may also make a difference what OS you use. Vista eats lots of memory so it allows you to import signifficantly less data than the same machine with XP on board. But even with very modest configuration you should be able to read such files but this means workin a bit more than using Import Wizard (I guess your computer stops when you use Import Wizard).


"Ashish Uthama" <first.last@mathworks.com> wrote in message

> Do you need the entire data in the file for your processing? Could you
> perhaps do the processing in parts?
> For example, if you wanted to compute the MIN of a column, you do that by
> reading in the data in chunks.

Hi Ashish, I've got one question related to what you wrote above. To do this processing in parts what function should I use? textscan? Do I have to know the number of rows in the file or can it be detected?
And one more thing. Matlab offers a few functions that can deal with numeric data stored in text files (textscan, fscanf, dlmread, load...). Does it make a big difference which of them I use? In other words are they equally efficient?
Many questions :-) I'd be very grateful for answers on any of it. Thx in advance

"Michal G" <Michal.Grotowski@uek.krakow.pl> wrote in message <h23j8t$jm1$1@fred.mathworks.com>...
[..]
> And one more thing. Matlab offers a few functions that can deal with numeric data stored in text files (textscan, fscanf, dlmread, load...). Does it make a big difference which of them I use? In other words are they equally efficient?

to my experience, textscan is much more efficient than load or dlmread on numeric ascii data. It is also the most powerful import function, all in all.
For numeric data I'd also recommend my file exchange contribution txt2mat, which can even be a bit (!) faster (in 'block' mode), and it offers quite easy options to do processing in parts and some other features. It is based on sscanf, so I assume fscanf will be the fastest solution unless textscan did not improve during the last few releases.

On Fri, 26 Jun 2009 18:49:01 -0400, Michal G
<Michal.Grotowski@uek.krakow.pl> wrote:

> Hi
>
> I work with text files of this size (containing only numeric data) and
> Matlab handles them nicely without any special efforts when your
> computer's memory is big enough :-). It may also make a difference what
> OS you use. Vista eats lots of memory so it allows you to import
> signifficantly less data than the same machine with XP on board. But
> even with very modest configuration you should be able to read such
> files but this means workin a bit more than using Import Wizard (I guess
> your computer stops when you use Import Wizard).
>
>
> "Ashish Uthama" <first.last@mathworks.com> wrote in message
>
>> Do you need the entire data in the file for your processing? Could you
>> perhaps do the processing in parts?
>> For example, if you wanted to compute the MIN of a column, you do that
>> by
>> reading in the data in chunks.
>
> Hi Ashish, I've got one question related to what you wrote above. To do
> this processing in parts what function should I use? textscan? Do I have
> to know the number of rows in the file or can it be detected?
> And one more thing. Matlab offers a few functions that can deal with
> numeric data stored in text files (textscan, fscanf, dlmread, load...).
> Does it make a big difference which of them I use? In other words are
> they equally efficient?
> Many questions :-) I'd be very grateful for answers on any of it. Thx in
> advance

Yes, I would recommend TEXTSCAN.
However, if you are comfortable using FSCANF and FREAD and it makes for
simpler code, use them instead. I would expect both of these to be faster
than dlmread/load, though the exact factor would depend on the
data/size/machine etc.

No, you would not have to know the number of rows. You can loop over the
file using "textscan(fid, 'format', N)" where N is the number of rows you
want to process at any given time (the 'chunk size'). Take care to process
the data returned by TEXTSCAN since the last call might yield < N rows.

Thanks