Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
read numerical text file by fixed column width, no delimiters

Subject: read numerical text file by fixed column width, no delimiters

From: Kristin

Date: 17 Jun, 2013 23:15:07

Message: 1 of 12

I have a large text file containing 8 columns of numbers and each column has a fixed width of 10 characters (so the total length of each row is 80 characters). The first character in each group of 10 characters is either a space or a negative sign.
So for example a typical line is:
-1.6999E-3-1.3266E-3 1.2916E-3 3.5491E-4 1.1022E-3-9.5555E-4-3.9528E-4 3.5200E-4
I was trying various approaches with textread or textscan and was trying to use %c because %s requires whitespace separated values which is not always the case here.
Is there an easy way to read such a file in Matlab? I either need eight vectors of numbers (ideally) or I could also handle a vector of strings where each string is 80 characters long including spaces. When nothing was working I even tried
[N1 N2 .... N80]=textread('file.txt','%c%c.....%c','headerlines',0) indicating that it needed 80 character inputs by 80 %c's but it still was not reading the space as a separate character. However I'm sure there is a much simpler way than trying to do something like that.
Please let me know if an easy solution already exists for reading files with fixed column widths but not space delimited. Thanks

Kristin

Subject: read numerical text file by fixed column width, no delimiters

From: dpb

Date: 18 Jun, 2013 00:02:01

Message: 2 of 12

On 6/17/2013 6:15 PM, Kristin wrote:
> I have a large text file containing 8 columns of numbers and each column
> has a fixed width of 10 characters (so the total length of each row is
> 80 characters). The first character in each group of 10 characters is
> either a space or a negative sign.
> So for example a typical line is:
> -1.6999E-3-1.3266E-3 1.2916E-3 3.5491E-4 1.1022E-3-9.5555E-4-3.9528E-4
> 3.5200E-4
> I was trying various approaches with textread or textscan and was trying
> to use %c because %s requires whitespace separated values which is not
> always the case here. Is there an easy way to read such a file in
> Matlab? I either need eight vectors of numbers (ideally) ...
...
> ... Please let me know if
> an easy solution already exists for reading files with fixed column
> widths but not space delimited. Thanks

Unfortunately, it appears a near fatal flaw in C formatted input. I've
never found a scan string that will work, unfortunately.

You can use '%f10.4' and it'll read the first value just fine and
fscanf() will return an index of 11 which seems reasonable but it will
error on some internal problem that it can't then parse the next 10
characters. textscan(), textread() all have similar issues.

I've b...ed at TMW about this for 20+ yrs and it continues to fall on
deaf ears. :(

Best I know to do is to do something like read the whole file into
memory as unformatted character array, do a global insertion of a
delimiter in the proper columns and then use textscan or sscanf or the
like from the memory image. Sounds feeble and it is, but there ya' are
-- it comes from using the C i/o routines instead of Fortran FORMAT model.

Alternatively, your back to fgetl() on a line at a time then a loop to
break up each line into 10 character segments and convert them. It's a
pita, agreed.

Years ago I wrote a mex file that took essentially the Fortran source
for a READ (the format string and the corresponding variable list) and
returned the values by using Fortran i/o to do the READ. Unfortunately,
I have seemed to have lost that in the post-retirement move or I'd post
it at FEX.

I don't know why can't get TMW to fix some of the simplest things for
the longest times... :(

--

Subject: read numerical text file by fixed column width, no delimiters

From: dpb

Date: 18 Jun, 2013 01:30:29

Message: 3 of 12

On 6/17/2013 7:02 PM, dpb wrote:
...

> Best I know to do is to do something like read the whole file into
> memory as unformatted character array, do a global insertion of a
> delimiter in the proper columns and then use textscan or sscanf or the
> like from the memory image. Sounds feeble and it is, but there ya' are
> -- it comes from using the C i/o routines instead of Fortran FORMAT model.
>
> Alternatively, your back to fgetl() on a line at a time then a loop to
> break up each line into 10 character segments and convert them. It's a
> pita, agreed.
>
...

Or, if you can't change the original source, write a quick'n dirty
Fortran routine to read it then rewrite it as F11.4. Shouldn't have to
do it but might be simpler in the end.

I suppose a regexp'er could probably work something up; how it would do
performance wise I don't know...

--

Subject: read numerical text file by fixed column width, no delimiters

From: dpb

Date: 18 Jun, 2013 18:04:38

Message: 4 of 12

On 6/17/2013 6:15 PM, Kristin wrote:
> I have a large text file containing 8 columns of numbers and each column
> has a fixed width of 10 characters (so the total length of each row is
> 80 characters). The first character in each group of 10 characters is
> either a space or a negative sign.
> So for example a typical line is:
> -1.6999E-3-1.3266E-3 1.2916E-3 3.5491E-4 1.1022E-3-9.5555E-4-3.9528E-4
> 3.5200E-4
> I was trying various approaches with textread or textscan and was trying
> to use %c because %s requires whitespace separated values which is not
> always the case here. Is there an easy way to read such a file in
> Matlab? I either need eight vectors of numbers (ideally)...

Oh, forgot another "trick" I've used on occasion. How well it works
depends on the size of the file...if can hold in memory works pretty
good; if have to can do in "chunks" that do fit...

Given you have a line as yours above I make it two to demonstrate for
the general case...

 >> m=[l;l];
 >> v=reshape(str2num(reshape(m',10,[])'),8,[])'
v =
-0.0017 -0.0013 0.0013 0.0004 0.0011 -0.0010 -0.0004 0.0004
-0.0017 -0.0013 0.0013 0.0004 0.0011 -0.0010 -0.0004 0.0004
 >>

Just have to know the number per line and the width of the field for the
two reshape() operations. The first casts the input character array
into a column of the right number of characters/line one field/row; the
second recasts into the known number of entries/line.

Again and still a lot of machinations for what _should_ be doable
directly by the i/o runtime formatting but C is "just broke" in this regard.

--

Subject: read numerical text file by fixed column width, no delimiters

From: Kristin

Date: 19 Jun, 2013 19:36:11

Message: 5 of 12

dpb <none@non.net> wrote in message <kpq7fb$i95$1@speranza.aioe.org>...
> On 6/17/2013 6:15 PM, Kristin wrote:
> > I have a large text file containing 8 columns of numbers and each column
> > has a fixed width of 10 characters (so the total length of each row is
> > 80 characters). The first character in each group of 10 characters is
> > either a space or a negative sign.
> > So for example a typical line is:
> > -1.6999E-3-1.3266E-3 1.2916E-3 3.5491E-4 1.1022E-3-9.5555E-4-3.9528E-4
> > 3.5200E-4
> > I was trying various approaches with textread or textscan and was trying
> > to use %c because %s requires whitespace separated values which is not
> > always the case here. Is there an easy way to read such a file in
> > Matlab? I either need eight vectors of numbers (ideally)...
>
> Oh, forgot another "trick" I've used on occasion. How well it works
> depends on the size of the file...if can hold in memory works pretty
> good; if have to can do in "chunks" that do fit...
>
> Given you have a line as yours above I make it two to demonstrate for
> the general case...
>
> >> m=[l;l];
> >> v=reshape(str2num(reshape(m',10,[])'),8,[])'
> v =
> -0.0017 -0.0013 0.0013 0.0004 0.0011 -0.0010 -0.0004 0.0004
> -0.0017 -0.0013 0.0013 0.0004 0.0011 -0.0010 -0.0004 0.0004
> >>
>
> Just have to know the number per line and the width of the field for the
> two reshape() operations. The first casts the input character array
> into a column of the right number of characters/line one field/row; the
> second recasts into the known number of entries/line.
>
> Again and still a lot of machinations for what _should_ be doable
> directly by the i/o runtime formatting but C is "just broke" in this regard.
>
> --

Thanks for the suggestions. I will try one of those methods (reshape or fgetl to do it a line at a time). Otherwise I'll see if I can just make a quick fortran script although I haven't previously used fortran for my coding.

Kristin

Subject: read numerical text file by fixed column width, no delimiters

From: dpb

Date: 19 Jun, 2013 20:06:42

Message: 6 of 12

On 6/19/2013 2:36 PM, Kristin wrote:
...

OK, here's a very rough, not terribly user=friendly outline, but should
get the job done...

program tentoeleven
   character(len=132):: fold, fnew ! A filename for old, new file
   real :: v(8) ! Data array of 8 floats

   read(*,*) fold ! It'll wait for you...
   open(11,file=fold,action='read',status='old')

   read(*,*) fnew ! It'll wait for you...
   open(12,file=fnew,action='write',status='new')

   ios=0
   do while(ios.eq.0)
     read(11,'(F10.4)',iostat=ios) v ! read the old file F10.4
     write(12,'(F11.4)') v ! F11.4 will insert space delimiter
   end do

   close(11)
   close(12)
end

Will continue 'til EOF condition on the old file at which time the
IOSTATUS variable will become <0. Of course, other errors will also
abort, but one presumes the files are otherwise, clean.

--

Subject: read numerical text file by fixed column width, no delimiters

From: TideMan

Date: 19 Jun, 2013 21:44:44

Message: 7 of 12

On Thursday, June 20, 2013 8:06:42 AM UTC+12, dpb wrote:
> On 6/19/2013 2:36 PM, Kristin wrote:
>
> ...
>
>
>
> OK, here's a very rough, not terribly user=friendly outline, but should
>
> get the job done...
>
>
>
> program tentoeleven
>
> character(len=132):: fold, fnew ! A filename for old, new file
>
> real :: v(8) ! Data array of 8 floats
>
>
>
> read(*,*) fold ! It'll wait for you...
>
> open(11,file=fold,action='read',status='old')
>
>
>
> read(*,*) fnew ! It'll wait for you...
>
> open(12,file=fnew,action='write',status='new')
>
>
>
> ios=0
>
> do while(ios.eq.0)
>
> read(11,'(F10.4)',iostat=ios) v ! read the old file F10.4
>
> write(12,'(F11.4)') v ! F11.4 will insert space delimiter
>
> end do
>
>
>
> close(11)
>
> close(12)
>
> end
>
>
>
> Will continue 'til EOF condition on the old file at which time the
>
> IOSTATUS variable will become <0. Of course, other errors will also
>
> abort, but one presumes the files are otherwise, clean.
>
>
>
> --

Here's an alternative that can be run within Matlab using:
dos(['tentoeleven ' infile ' ' outfile])
where infile and outfile are strings with the filenames.

program tentoeleven
   character(len=132):: fold, fnew ! A filename for old, new file
   real :: v(8) ! Data array of 8 floats
   integer :: na ! No of args on command line
   integer :: ios ! Error code for read

   na=iargc()
   if(na < 2)stop 'Must specify infile and outfile on command line'
   call getarg(1,fold)
   call getarg(2,fnew)

   open(11,file=fold,action='read',status='old')
   open(12,file=fnew,action='write',status='new')

   ios=0
   do while(ios.eq.0)
     read(11,'(F10.4)',iostat=ios) v ! read the old file F10.4
     write(12,'(F11.4)') v ! F11.4 will insert space delimiter
   end do

   close(11)
   close(12)
end

Subject: read numerical text file by fixed column width, no delimiters

From: dpb

Date: 20 Jun, 2013 00:22:09

Message: 8 of 12

On 6/19/2013 4:44 PM, TideMan wrote:
> On Thursday, June 20, 2013 8:06:42 AM UTC+12, dpb wrote:
>> On 6/19/2013 2:36 PM, Kristin wrote:
>>
>> ...
>>
>>
>>
>> OK, here's a very rough, not terribly user=friendly outline, but should
>>
>> get the job done...
>>
>>
>>
>> program tentoeleven
>>
>> character(len=132):: fold, fnew ! A filename for old, new file
>>
>> real :: v(8) ! Data array of 8 floats
>>
>>
>>
>> read(*,*) fold ! It'll wait for you...
>>
>> open(11,file=fold,action='read',status='old')
>>
>>
>>
>> read(*,*) fnew ! It'll wait for you...
>>
>> open(12,file=fnew,action='write',status='new')
>>
>>
>>
>> ios=0
>>
>> do while(ios.eq.0)
>>
>> read(11,'(F10.4)',iostat=ios) v ! read the old file F10.4
>>
>> write(12,'(F11.4)') v ! F11.4 will insert space delimiter
>>
>> end do
>>
>>
>>
>> close(11)
>>
>> close(12)
>>
>> end
>>
>>
>>
>> Will continue 'til EOF condition on the old file at which time the
>>
>> IOSTATUS variable will become<0. Of course, other errors will also
>>
>> abort, but one presumes the files are otherwise, clean.
>>
>>
>>
>> --
>
> Here's an alternative that can be run within Matlab using:
> dos(['tentoeleven ' infile ' ' outfile])
> where infile and outfile are strings with the filenames.
>
> program tentoeleven
> character(len=132):: fold, fnew ! A filename for old, new file
> real :: v(8) ! Data array of 8 floats
> integer :: na ! No of args on command line
> integer :: ios ! Error code for read
>
> na=iargc()
> if(na< 2)stop 'Must specify infile and outfile on command line'
> call getarg(1,fold)
> call getarg(2,fnew)
>
> open(11,file=fold,action='read',status='old')
> open(12,file=fnew,action='write',status='new')
>
> ios=0
> do while(ios.eq.0)
> read(11,'(F10.4)',iostat=ios) v ! read the old file F10.4
> write(12,'(F11.4)') v ! F11.4 will insert space delimiter
> end do
>
> close(11)
> close(12)
> end

Man, Tide, next you'll be giving a GUI file dialog... :) Gotta' leave
_something_ as "exercise for the student" <vbg>

_WHY_ can't TMW see this is a need to fix? I've asked in C groups and
get the expected response there that one shouldn't write files that
aren't human-readable (namely, delimited) which is, of course, less than
helpful.

To best of my recollection I filed bug reports/enhancement requests
clear back w/ the first Windows version I had along about V4.0 or
thereabouts and a few later times over the years when had support in
place but never got a workaround suggestion (or, as best I can recall, a
response even). It just seems incomprehensible it can be so and that's
considered "ok". Along with it that could possibly get fixed as part of
the same is the case of fixed-width input w/ blank as input being
significant.

--

Subject: read numerical text file by fixed column width, no delimiters

From: TideMan

Date: 20 Jun, 2013 03:11:27

Message: 9 of 12

On Thursday, June 20, 2013 12:22:09 PM UTC+12, dpb wrote:
> On 6/19/2013 4:44 PM, TideMan wrote:
>
> > On Thursday, June 20, 2013 8:06:42 AM UTC+12, dpb wrote:
>
> >> On 6/19/2013 2:36 PM, Kristin wrote:
>
> >>
>
> >> ...
>
> >>
>
> >>
>
> >>
>
> >> OK, here's a very rough, not terribly user=friendly outline, but should
>
> >>
>
> >> get the job done...
>
> >>
>
> >>
>
> >>
>
> >> program tentoeleven
>
> >>
>
> >> character(len=132):: fold, fnew ! A filename for old, new file
>
> >>
>
> >> real :: v(8) ! Data array of 8 floats
>
> >>
>
> >>
>
> >>
>
> >> read(*,*) fold ! It'll wait for you...
>
> >>
>
> >> open(11,file=fold,action='read',status='old')
>
> >>
>
> >>
>
> >>
>
> >> read(*,*) fnew ! It'll wait for you...
>
> >>
>
> >> open(12,file=fnew,action='write',status='new')
>
> >>
>
> >>
>
> >>
>
> >> ios=0
>
> >>
>
> >> do while(ios.eq.0)
>
> >>
>
> >> read(11,'(F10.4)',iostat=ios) v ! read the old file F10.4
>
> >>
>
> >> write(12,'(F11.4)') v ! F11.4 will insert space delimiter
>
> >>
>
> >> end do
>
> >>
>
> >>
>
> >>
>
> >> close(11)
>
> >>
>
> >> close(12)
>
> >>
>
> >> end
>
> >>
>
> >>
>
> >>
>
> >> Will continue 'til EOF condition on the old file at which time the
>
> >>
>
> >> IOSTATUS variable will become<0. Of course, other errors will also
>
> >>
>
> >> abort, but one presumes the files are otherwise, clean.
>
> >>
>
> >>
>
> >>
>
> >> --
>
> >
>
> > Here's an alternative that can be run within Matlab using:
>
> > dos(['tentoeleven ' infile ' ' outfile])
>
> > where infile and outfile are strings with the filenames.
>
> >
>
> > program tentoeleven
>
> > character(len=132):: fold, fnew ! A filename for old, new file
>
> > real :: v(8) ! Data array of 8 floats
>
> > integer :: na ! No of args on command line
>
> > integer :: ios ! Error code for read
>
> >
>
> > na=iargc()
>
> > if(na< 2)stop 'Must specify infile and outfile on command line'
>
> > call getarg(1,fold)
>
> > call getarg(2,fnew)
>
> >
>
> > open(11,file=fold,action='read',status='old')
>
> > open(12,file=fnew,action='write',status='new')
>
> >
>
> > ios=0
>
> > do while(ios.eq.0)
>
> > read(11,'(F10.4)',iostat=ios) v ! read the old file F10.4
>
> > write(12,'(F11.4)') v ! F11.4 will insert space delimiter
>
> > end do
>
> >
>
> > close(11)
>
> > close(12)
>
> > end
>
>
>
> Man, Tide, next you'll be giving a GUI file dialog... :) Gotta' leave
>
> _something_ as "exercise for the student" <vbg>
>
>
>
> _WHY_ can't TMW see this is a need to fix? I've asked in C groups and
>
> get the expected response there that one shouldn't write files that
>
> aren't human-readable (namely, delimited) which is, of course, less than
>
> helpful.
>
>
>
> To best of my recollection I filed bug reports/enhancement requests
>
> clear back w/ the first Windows version I had along about V4.0 or
>
> thereabouts and a few later times over the years when had support in
>
> place but never got a workaround suggestion (or, as best I can recall, a
>
> response even). It just seems incomprehensible it can be so and that's
>
> considered "ok". Along with it that could possibly get fixed as part of
>
> the same is the case of fixed-width input w/ blank as input being
>
> significant.
>
>
>
> --

Oh, dpb

You, like me, date back to the days when we communicated with computers via Hollerith cards. The cards were 80 columns wide and that's why we often used 10f8.0 or 8f10.0 for formatting. Who cared if the numbers ran in together? You could hardly read them on the top of the card anyway.

But C was not invented until much later and the johnny-come-latelies who developed it had no appreciation of (nor cared) about heritage code.

And so it is with TMW.
I believe Matlab was originally Fortran code.
Why did Moler et al not do it properly way back then?
There's no hope now.
And it's pointless fighting battles that have already been lost.
Adapt and live on..............

Subject: read numerical text file by fixed column width, no delimiters

From: dpb

Date: 20 Jun, 2013 04:13:06

Message: 10 of 12

On 6/19/2013 10:11 PM, TideMan wrote:
> On Thursday, June 20, 2013 12:22:09 PM UTC+12, dpb wrote:
...

>> _WHY_ can't TMW see this is a need to fix? I've asked in C groups and
>> get the expected response there that one shouldn't write files that
>> aren't human-readable (namely, delimited) which is, of course, less than
>> helpful.
>>
>>
>>
...

>
> Oh, dpb
>
> You, like me, date back to the days when we communicated with
> computers via Hollerith cards. The cards were 80 columns wide and that's
> why we often used 10f8.0 or 8f10.0 for formatting. Who cared if the
> numbers ran in together? You could hardly read them on the top of the
> card anyway.
>
> But C was not invented until much later and the johnny-come-latelies
> who developed it had no appreciation of (nor cared) about heritage
> code.
>
> And so it is with TMW.
> I believe Matlab was originally Fortran code.
> Why did Moler et al not do it properly way back then?

I really find it hard to believe that Cleve didn't appreciate (even
then) legacy code, he's too ingrained into the development of methods at
the time to not have been.

I didn't use Matlab until the Windows version was introduced; I think,
unfortunately, the translation to C won out w/ the "easier to implement"
decision by using the default existing i/o runtime rather than
rebuilding a FORMAT facility. Unfortunate, decision, indeed...

> There's no hope now.
> And it's pointless fighting battles that have already been lost.
> Adapt and live on..............

Never!!!! Fight to the death! :)

--

Subject: read numerical text file by fixed column width, no delimiters

From: Kristin

Date: 20 Jun, 2013 19:40:19

Message: 11 of 12

TideMan <mulgor@gmail.com> wrote in message <87f787b0-17ab-4f75-9c0a-9f5dc71ca781@googlegroups.com>...
> On Thursday, June 20, 2013 8:06:42 AM UTC+12, dpb wrote:
> > On 6/19/2013 2:36 PM, Kristin wrote:
> >
> > ...
> >
> >
> >
> > OK, here's a very rough, not terribly user=friendly outline, but should
> >
> > get the job done...
> >
> >
> >
> > program tentoeleven
> >
> > character(len=132):: fold, fnew ! A filename for old, new file
> >
> > real :: v(8) ! Data array of 8 floats
> >
> >
> >
> > read(*,*) fold ! It'll wait for you...
> >
> > open(11,file=fold,action='read',status='old')
> >
> >
> >
> > read(*,*) fnew ! It'll wait for you...
> >
> > open(12,file=fnew,action='write',status='new')
> >
> >
> >
> > ios=0
> >
> > do while(ios.eq.0)
> >
> > read(11,'(F10.4)',iostat=ios) v ! read the old file F10.4
> >
> > write(12,'(F11.4)') v ! F11.4 will insert space delimiter
> >
> > end do
> >
> >
> >
> > close(11)
> >
> > close(12)
> >
> > end
> >
> >
> >
> > Will continue 'til EOF condition on the old file at which time the
> >
> > IOSTATUS variable will become <0. Of course, other errors will also
> >
> > abort, but one presumes the files are otherwise, clean.
> >
> >
> >
> > --
>
> Here's an alternative that can be run within Matlab using:
> dos(['tentoeleven ' infile ' ' outfile])
> where infile and outfile are strings with the filenames.
>
> program tentoeleven
> character(len=132):: fold, fnew ! A filename for old, new file
> real :: v(8) ! Data array of 8 floats
> integer :: na ! No of args on command line
> integer :: ios ! Error code for read
>
> na=iargc()
> if(na < 2)stop 'Must specify infile and outfile on command line'
> call getarg(1,fold)
> call getarg(2,fnew)
>
> open(11,file=fold,action='read',status='old')
> open(12,file=fnew,action='write',status='new')
>
> ios=0
> do while(ios.eq.0)
> read(11,'(F10.4)',iostat=ios) v ! read the old file F10.4
> write(12,'(F11.4)') v ! F11.4 will insert space delimiter
> end do
>
> close(11)
> close(12)
> end

I was able to get the file formatted properly using a fortran code (not written by me) and avoided use of Matlab altogether. I did try the tentoeleven fortran code (above) but for some reason it only outputted the first column and the other seven columns of data were not present. So it seems something might have gone wrong in the execution. In any case, problem was solved using fortran... thanks everyone for the good ideas and thoughts above

Subject: read numerical text file by fixed column width, no delimiters

From: dpb

Date: 20 Jun, 2013 20:50:55

Message: 12 of 12

On 6/20/2013 2:40 PM, Kristin wrote:
...

> I was able to get the file formatted properly using a fortran code (not
> written by me) and avoided use of Matlab altogether. I did try the
> tentoeleven fortran code (above) but for some reason it only outputted
> the first column and the other seven columns of data were not present.
> So it seems something might have gone wrong in the execution. In any
> case, problem was solved using fortran... thanks everyone for the good
> ideas and thoughts above

My bad, sorry...obvious oversight from just typing...for the record the
fix is

ios=0
do while(ios.eq.0)
   read( 11,'(8F10.4)',iostat=ios) v ! read the old file F10.4
   write(12,'(8F11.4)') v ! F11.4 will insert space delimiter
end do

NB: the '8' repeat specifier wasn't included so each READ read eight
records to satisfy the size of v.

Again, sorry...shoulda' seen that to begin with.

--

Tags for this Thread

No tags are associated with this thread.

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us