MATLAB Central Newsreader - about textread

about textread

Wu Zhiyong — Wed, 16 Apr 2008 13:11:02 -0400

I have a lot of data to read into matlab.They are printed
in the textfile as followed:

airpressure staions bias MAE RMSE HR
  200.0 44 1.82511 2.34681 2.78835 22
airpressure staions bias MAE RMSE HR
300.0 38 1.55661 1.83753 2.44242 26
……
I need to ignore the characters. I did it in two ways.

1.delete the characters manually.And write the code as
followed:

[airpress sta bias mae rmse hr]=textread(filename,'%f %f %
f %f %f %f');

2.add the symbol "%" in front of the characters manually:

%airpressure staions bias MAE RMSE HR
  200.0 44 1.82511 2.34681 2.78835 22
%airpressure staions bias MAE RMSE HR
300.0 38 1.55661 1.83753 2.44242 26
……

And write the code as followed:

[airpress sta bias mae rmse hr]=textread(filename,'%f %f %
f %f %f %f','commentstyle','matlab');

Both ways worked.But there are a large number of data
files.And it costs too much time.
  Do you have any other way to ignore the characters?
  Any suggestion is appreciated!
  Thank you!

Re: about textread

roberson@ibd.nrc-cnrc.gc.ca (Walter Roberson) — Thu, 17 Apr 2008 04:58:40 -0400

In article <fu4tt6$rjq$1@fred.mathworks.com>,
Wu Zhiyong <wuzhiyong_163@163.com> wrote:
>I have a lot of data to read into matlab.They are printed
>in the textfile as followed:

>airpressure staions bias MAE RMSE HR
> 200.0 44 1.82511 2.34681 2.78835 22
>airpressure staions bias MAE RMSE HR
> 300.0 38 1.55661 1.83753 2.44242 26
> ……
>I need to ignore the characters. I did it in two ways.

Do *all* the lines containing text start with 'airpressure'?
If so, then you can set the textscan() comment character to 'a'.
--
"When we all think alike no one is thinking very much."
-- Walter Lippmann

Re: about textread

Barry Williams — Thu, 17 Apr 2008 12:51:03 -0400

textread is going away. Try textscan instead. You can
skip fields.
Barry

"Wu Zhiyong" <wuzhiyong_163@163.com> wrote in message
<fu4tt6$rjq$1@fred.mathworks.com>...
> I have a lot of data to read into matlab.They are
printed
> in the textfile as followed:
>
> airpressure staions bias MAE RMSE
HR
> 200.0 44 1.82511 2.34681 2.78835 22
> airpressure staions bias MAE RMSE HR
> 300.0 38 1.55661 1.83753 2.44242 26
> ……
> I need to ignore the characters. I did it in two ways.
>
> 1.delete the characters manually.And write the code as
> followed:
>
> [airpress sta bias mae rmse hr]=textread(filename,'%f %f
%
> f %f %f %f');
>
> 2.add the symbol "%" in front of the characters
manually:
>
> %airpressure staions bias MAE RMSE
HR
> 200.0 44 1.82511 2.34681 2.78835 22
> %airpressure staions bias MAE RMSE HR
> 300.0 38 1.55661 1.83753 2.44242 26
> ……
>
> And write the code as followed:
>
> [airpress sta bias mae rmse hr]=textread(filename,'%f %f
%
> f %f %f %f','commentstyle','matlab');
>
> Both ways worked.But there are a large number of data
> files.And it costs too much time.
> Do you have any other way to ignore the characters?
> Any suggestion is appreciated!
> Thank you!
>

Re: about textread

Andres Toennesmann — Thu, 17 Apr 2008 20:05:06 -0400

Re: about textread

William Dampier — Thu, 17 Apr 2008 20:19:03 -0400

"Wu Zhiyong" <wuzhiyong_163@163.com> wrote in message
<fu4tt6$rjq$1@fred.mathworks.com>...
> I have a lot of data to read into matlab.They are printed
> in the textfile as followed:
>
> airpressure staions bias MAE RMSE HR
> 200.0 44 1.82511 2.34681 2.78835 22
> airpressure staions bias MAE RMSE HR
> 300.0 38 1.55661 1.83753 2.44242 26
> ……
> I need to ignore the characters. I did it in two ways.
>
> 1.delete the characters manually.And write the code as
> followed:
>
> [airpress sta bias mae rmse hr]=textread(filename,'%f %f %
> f %f %f %f');
>
> 2.add the symbol "%" in front of the characters manually:
>
> %airpressure staions bias MAE RMSE HR
> 200.0 44 1.82511 2.34681 2.78835 22
> %airpressure staions bias MAE RMSE HR
> 300.0 38 1.55661 1.83753 2.44242 26
> ……
>
> And write the code as followed:
>
> [airpress sta bias mae rmse hr]=textread(filename,'%f %f %
> f %f %f %f','commentstyle','matlab');
>
> Both ways worked.But there are a large number of data
> files.And it costs too much time.
> Do you have any other way to ignore the characters?
> Any suggestion is appreciated!
> Thank you!
>

If you switch to textscan, which is virtually a drop-in
replacement, then you can use fgetl(fid) and this will
remove the entire first line. If you then use textscan on
the same fid then you won't have to make any changes to the
original text-files.

Re: about textread

Wu Zhiyong — Sun, 01 Jun 2008 13:59:02 -0400

"Andres Toennesmann" <rantore@werb.de> wrote in message
<fu8ahh$c8m$1@fred.mathworks.com>...
> "Wu Zhiyong" <wuzhiyong_163@163.com> wrote in message
> <fu4tt6$rjq$1@fred.mathworks.com>...
> > I have a lot of data to read into matlab.They are
printed
> > in the textfile as followed:
> >
> > airpressure staions bias MAE RMSE
HR
> > 200.0 44 1.82511 2.34681 2.78835 22
> > airpressure staions bias MAE RMSE HR
> > 300.0 38 1.55661 1.83753 2.44242 26
> > ……
> > I need to ignore the characters. [...]
>
> You may try txt2mat from the file exchange with
> A = txt2mat('Filename',0,6,'BadLineString','air');
> This should result in the matrix
> A = [200.0 44 1.82511 2.34681 2.78835 22
> 300.0 38 1.55661 1.83753 2.44242 26
> ...]
> It would be interesting to know for me if this works
> reasonably fast.
> Regards
> Andres
>
Thank you for your great work!
I'm going on with my work after a wonderful vacation.
yeah, Your code works fast.It really saves my time!
But one more question:
How can I ignore the blank line?
for example:
airpressure staions bias MAE RMSE HR
300.0 38 1.55661 1.83753 2.44242 26

airpressure staions bias MAE RMSE HR
200.0 44 1.82511 2.34681 2.78835 22
........
Thanks very much!

Re: about textread

Andres — Mon, 02 Jun 2008 09:14:03 -0400

"Wu Zhiyong" <wuzhiyong_163@163.com> wrote in message
<g1u9v6$2i4$1@fred.mathworks.com>...
> But one more question:
> How can I ignore the blank line?
> for example:
> airpressure staions bias MAE RMSE HR
> 300.0 38 1.55661 1.83753 2.44242 26
>
> airpressure staions bias MAE RMSE HR
> 200.0 44 1.82511 2.34681 2.78835 22
> ........

I'm currently thinking about an enhancement of txt2mat
which would involve allowing to completely ignore empty
lines (this would even imply speed benefits). With the
current version on the fex, you'll need some kind of trick
to remove them (see below).

Just for the sake of completeness: The standard method
would be to directly switch to ReadMode 'line' and sort out
the NaN rows afterwards:

fn = 'c:\myfile.txt'

A = txt2mat(fn,0,6,'BadLineString', ...
{'air'},'ReadMode','line');
A = A(isfinite(A(:,1)), :);

Use the 'InfoLevel' parameter to suppress the command line
information about the different row lengths, if you like
(see help).

Now for the 'trick':
To remove the empty lines in the text, you can also delete
consecutive line break characters by the help of
the 'ReplaceExpr'. On a windows system (!), the character
pair char([13 10]) forms a single line break. If your empty
lines do not contain any whitespace, the occurence of
char([10 13]) indicates a double line break. Hence, use

B = txt2mat(fn,0,6,'BadLineString',{'air'},...
'ReplaceExpr',{{char([10, 13]),''}});

Usually, txt2mat expects anything to be manipulated with
the exception of line breaks. As txt2mat cycles throught
the text by sections containing 200'000 lines by default,
the above will only work safely if you have less data lines
+ empty lines than that. Of course, you could get around
that as well (see 'MemPar' argument). If you need further
information, feel free to contact me via the fex form.
Please check which solution is most convenient in terms of
usage and speed on your system.

Hth,
regards
Andres

Re: about textread

Wu Zhiyong — Mon, 02 Jun 2008 13:25:04 -0400

"Andres " <rantore@werb.de> wrote in message
<g20dkr$aof$1@fred.mathworks.com>...
> "Wu Zhiyong" <wuzhiyong_163@163.com> wrote in message
> <g1u9v6$2i4$1@fred.mathworks.com>...
> > But one more question:
> > How can I ignore the blank line?
> > for example:
> > airpressure staions bias MAE RMSE HR
> > 300.0 38 1.55661 1.83753 2.44242 26
> >
> > airpressure staions bias MAE RMSE HR
> > 200.0 44 1.82511 2.34681 2.78835 22
> > ........
>
> I'm currently thinking about an enhancement of txt2mat
> which would involve allowing to completely ignore empty
> lines (this would even imply speed benefits). With the
> current version on the fex, you'll need some kind of
trick
> to remove them (see below).
>
> Just for the sake of completeness: The standard method
> would be to directly switch to ReadMode 'line' and sort
out
> the NaN rows afterwards:
>
> fn = 'c:\myfile.txt'
>
> A = txt2mat(fn,0,6,'BadLineString', ...
> {'air'},'ReadMode','line');
> A = A(isfinite(A(:,1)), :);
>
> Use the 'InfoLevel' parameter to suppress the command
line
> information about the different row lengths, if you like
> (see help).
>
> Now for the 'trick':
> To remove the empty lines in the text, you can also
delete
> consecutive line break characters by the help of
> the 'ReplaceExpr'. On a windows system (!), the
character
> pair char([13 10]) forms a single line break. If your
empty
> lines do not contain any whitespace, the occurence of
> char([10 13]) indicates a double line break. Hence, use
>
> B = txt2mat(fn,0,6,'BadLineString',{'air'},...
> 'ReplaceExpr',{{char([10, 13]),''}});
>
> Usually, txt2mat expects anything to be manipulated with
> the exception of line breaks. As txt2mat cycles throught
> the text by sections containing 200'000 lines by
default,
> the above will only work safely if you have less data
lines
> + empty lines than that. Of course, you could get around
> that as well (see 'MemPar' argument). If you need
further
> information, feel free to contact me via the fex form.
> Please check which solution is most convenient in terms
of
> usage and speed on your system.
>
> Hth,
> regards
> Andres
>

I writed a simple Function(10 lines or less) to remove the
NaN lines from the output matrix A.
but absolutely,"isfinite" is conciser.
The progress speeds up with 'InfoLevel' set to be 0.
I'm expecting your enhancement!