MATLAB Central Newsreader - Need advice on File I/O

Need advice on File I/O

G.A.M. — Sat, 01 Sep 2007 00:02:26 -0400

I am working with delimited text files (mostly, but not
always, comma separated or CSV).

Reading and creating entire files isn't a problem. My
problems start when I need to alter data in existing files.
I could use some advice.

Sometimes I will need to remove an entire line from a file.
Sometimes I need to replace a single value (at any row,
column location) without changing the rest of the file.

After searching and reading everything I can find in the
help, I am still undecided on how to go about this. I would
appreciate some advice from experienced Matlab users.

I simply can't see an easy way to modify my file data with
the Matlab file I/O functions I see listed in help.

(I'm working on Windows and my data is all ASCII. The
delimiters and data organization in the files is not always
uniform, so I often have to write code to analyze each
character I read from a file and figure out where the data
is that I want to change. However, I don't think there is
anything really unusual about what I need to do.)

Re: Need advice on File I/O

Rune Allnor — Sat, 01 Sep 2007 06:49:39 -0400

On 1 Sep, 02:02, "G.A.M. " <x0z...@gmail.com> wrote:
> I am working with delimited text files (mostly, but not
> always, comma separated or CSV).
>
> Reading and creating entire files isn't a problem. My
> problems start when I need to alter data in existing files.
> I could use some advice.
>
> Sometimes I will need to remove an entire line from a file.
> Sometimes I need to replace a single value (at any row,
> column location) without changing the rest of the file.
>
> After searching and reading everything I can find in the
> help, I am still undecided on how to go about this. I would
> appreciate some advice from experienced Matlab users.
>
> I simply can't see an easy way to modify my file data with
> the Matlab file I/O functions I see listed in help.
>
> (I'm working on Windows and my data is all ASCII. The
> delimiters and data organization in the files is not always
> uniform, so I often have to write code to analyze each
> character I read from a file and figure out where the data
> is that I want to change. However, I don't think there is
> anything really unusual about what I need to do.)

Use regular expressions to parse your files to find the data
you want. Use Walter's method #1, read the data from the source
file and write the altered data set to the destination file.

Rune

Re: Need advice on File I/O

G.A.M. — Sat, 01 Sep 2007 20:19:05 -0400

Rune Allnor <allnor@tele.ntnu.no> wrote in message
<1188629379.871780.250890@w3g2000hsg.googlegroups.com>...
> On 1 Sep, 02:02, "G.A.M. " <x0z...@gmail.com> wrote:
> > I am working with delimited text files (mostly, but not
> > always, comma separated or CSV).
> >
> > Reading and creating entire files isn't a problem. My
> > problems start when I need to alter data in existing files.
> > I could use some advice.
> >
> > Sometimes I will need to remove an entire line from a file.
> > Sometimes I need to replace a single value (at any row,
> > column location) without changing the rest of the file.
> >
> > After searching and reading everything I can find in the
> > help, I am still undecided on how to go about this. I would
> > appreciate some advice from experienced Matlab users.
> >
> > I simply can't see an easy way to modify my file data with
> > the Matlab file I/O functions I see listed in help.
> >
> > (I'm working on Windows and my data is all ASCII. The
> > delimiters and data organization in the files is not always
> > uniform, so I often have to write code to analyze each
> > character I read from a file and figure out where the data
> > is that I want to change. However, I don't think there is
> > anything really unusual about what I need to do.)
>
> Use regular expressions to parse your files to find the data
> you want. Use Walter's method #1, read the data from the
source
> file and write the altered data set to the destination file.
>
> Rune
>

Thanks for your reply. I am trying to avoid rewriting an
entire file when a single data item changes. (And it would
be great is I don't have to read to the end of the file
either, once I find the data I need to alter.)

It looks like I might be able to use the xlswrite to achieve
this end. However, I haven't figured out if it will indeed
allow me to significantly alter one cell and just write that
changed value back to disk. Help doesn't address that.

It would be even better if the csvwrite would do what I need.

Unfortunately, Matlab help isn't very clear on this. I've
asked my question on the newsgroup in about three different
ways and so far I still don't know much more than I did
about a week ago... if anyone else wants to jump in here and
give me some suggestions, I would greatly appreciate it.

(BTW, searching in here for "Walter's method #1" doesn't
return any results. I don't know what that method is.
However, maybe it isn't relevant because the suggestion
still looks like it requires reading and writing the entire
file.)

Re: Need advice on File I/O

Bill — Sat, 01 Sep 2007 22:34:21 -0400

Looks to me like you ought to consider using an Access
database table to hold the data. You can then employ
select and update sqls to modify your 'datafiles'.

I've become very enthused about using Access as a
transportable data closet. With the database toolbox, you
don't even need to have Access.

Re: Need advice on File I/O

G.A.M. — Sun, 02 Sep 2007 01:02:39 -0400

"Bill " <william.nospam.a.cobb@gm.com> wrote in message
<fbcpdd$b7$1@fred.mathworks.com>...
> Looks to me like you ought to consider using an Access
> database table to hold the data. You can then employ
> select and update sqls to modify your 'datafiles'.
>
> I've become very enthused about using Access as a
> transportable data closet. With the database toolbox, you
> don't even need to have Access.

Thanks for the suggestion. I do have the database toolbox
and I have been thinking about this as an option. I'd
probably use Sql Server Express (or MSDE) rather than
Access, but I think your suggestion is certainly an option I
should consider.

It seems like my options are coming down to either Excel or
a database. But I was hoping for a solution that involved
CSV/ASCII files. My past experience tells me that using a
database will probably be slower than writing to CSV files.
(I don't know how using one of the options to write to
Excel-formatted files will compare in speed.) I also like
the fact that ASCII text files, besides being generally fast
than a db, are really easy to deal with.

Re: Need advice on File I/O

Peter Boettcher — Tue, 04 Sep 2007 15:06:37 -0400

"G.A.M. " <x0zero@gmail.com> writes:

> "Bill " <william.nospam.a.cobb@gm.com> wrote in message
> <fbcpdd$b7$1@fred.mathworks.com>...
>
> It seems like my options are coming down to either Excel or
> a database. But I was hoping for a solution that involved
> CSV/ASCII files. My past experience tells me that using a
> database will probably be slower than writing to CSV files.

Unless you are talking about bulk saving or loading, I very much doubt
that!

> (I don't know how using one of the options to write to
> Excel-formatted files will compare in speed.) I also like
> the fact that ASCII text files, besides being generally fast
> than a db, are really easy to deal with.

Well, you are discovering precisely the things that make an ASCII file
difficult to deal with. They are very bad at random access, and
nearly impossible to do single-entry modifications. Higher-level
wrappers like csvwrite, etc, will not change this. It may hide some
of the nastiness, but it will still probably rewrite the entire file.
When I say bad, I mean slow and complex to implement.

Databases are very good at randomly accessing elements, and at
modifying, inserting, etc.

A third option not mentioned here is a binary file. If your data
items are numeric, then random access is easy (via fseek, or even via
memmapfile), and any item may be easily changed. You can also easily
append. You still cannot delete or insert items, or shorten the file.

-Peter

Re: Need advice on File I/O

G.A.M. — Wed, 05 Sep 2007 15:39:15 -0400

Peter Boettcher <boettcher@ll.mit.edu> wrote in message
<muy7in6qxky.fsf@G99-Boettcher.llan.ll.mit.edu>...
> "G.A.M. " <x0zero@gmail.com> writes:
>
> > "Bill " <william.nospam.a.cobb@gm.com> wrote in message
> > <fbcpdd$b7$1@fred.mathworks.com>...
> >
> > It seems like my options are coming down to either Excel or
> > a database. But I was hoping for a solution that involved
> > CSV/ASCII files. My past experience tells me that using a
> > database will probably be slower than writing to CSV files.
>
> Unless you are talking about bulk saving or loading, I
very much doubt
> that!

I was on a data-intensive application development project
where we collected extensive performance data on various
persistence options and we were all surprised how fast the
ASCII file I/O option was in a variety of situations. We
ended up going with the ASCII file I/O option because it was
several orders of magnitude faster for every I/O operation
we used. That said, another application might have produced
very different results.

>
> > (I don't know how using one of the options to write to
> > Excel-formatted files will compare in speed.) I also like
> > the fact that ASCII text files, besides being generally fast
> > than a db, are really easy to deal with.
>
> Well, you are discovering precisely the things that make
an ASCII file
> difficult to deal with. They are very bad at random
access, and
> nearly impossible to do single-entry modifications.
Higher-level
> wrappers like csvwrite, etc, will not change this. It may
hide some
> of the nastiness, but it will still probably rewrite the
entire file.
> When I say bad, I mean slow and complex to implement.

These points are all good reminders for me. Thanks for
bringing them up.

>
> Databases are very good at randomly accessing elements, and at
> modifying, inserting, etc.

I will seriously consider a database now.

>
> A third option not mentioned here is a binary file.

I think I'll stay away from this option. If I'm not going to
enjoy the advantages of a database, I at least want to have
the advantages of ASCII.

Re: Need advice on File I/O

G.A.M. — Wed, 05 Sep 2007 15:42:38 -0400

"G.A.M. " <x0zero@gmail.com> wrote in message
<fbchfp$g1g$1@fred.mathworks.com>...
> > > I could use some advice.

Rune Allnor <allnor@tele.ntnu.no> wrote in message
> > Use Walter's method #1, read the data from the
> source
> > file and write the altered data set to the destination file.
> >
> > Rune
> >
> "G.A.M. " <x0zero@gmail.com> wrote
> Thanks for your reply.
>
> (BTW, searching in here for "Walter's method #1" doesn't
> return any results. I don't know what that method is.
> However, maybe it isn't relevant because the suggestion
> still looks like it requires reading and writing the entire
> file.)

I never did find Walter's reply on the Matlab Newsreader web
page, but I did find his helpful email in my inbox just
today. Here is Walter's method #1:

1) open the input file and open an output file, copy from
input to
output until you find what you need to change, write the changed
data to the output file, then write the rest of the input to the
output. This will work provided there is enough disk space for
the second copy of the file, and is much safer than the other
options because if something goes wrong in the middle of the
process, you would still have the input file unchanged.

Re: Need advice on File I/O

"G.A.M. — Wed, 05 Sep 2007 15:55:56 -0400

On Sep 5, 11:42 am, "G.A.M. " <x0z...@gmail.com> wrote:
> "G.A.M. " <x0z...@gmail.com> wrote in message
>
> <fbchfp$g1...@fred.mathworks.com>...
>
>
>
> > > > I could use some advice.
> Rune Allnor <all...@tele.ntnu.no> wrote in message
> > > Use Walter's method #1, read the data from the
> > source
> > > file and write the altered data set to the destination file.
>
> > > Rune
>
> > "G.A.M. " <x0z...@gmail.com> wrote
> > Thanks for your reply.
>
> > (BTW, searching in here for "Walter's method #1" doesn't
> > return any results. I don't know what that method is.
> > However, maybe it isn't relevant because the suggestion
> > still looks like it requires reading and writing the entire
> > file.)
>
> I never did find Walter's reply on the Matlab Newsreader web
> page, but I did find his helpful email in my inbox just
> today. Here is Walter's method #1:
>
> 1) open the input file and open an output file, copy from
> input to
> output until you find what you need to change, write the changed
> data to the output file, then write the rest of the input to the
> output. This will work provided there is enough disk space for
> the second copy of the file, and is much safer than the other
> options because if something goes wrong in the middle of the
> process, you would still have the input file unchanged.

I am a new Matlab user. I had no idea this newsgroup was primarily a
USENET group. In fact, at first I thought it was accessible only via
TMW's web page. Now that I'm viewing this content via Google Groups I
see how silly it must have looked when I said "here" in my various
messages while thinking that everyone was using the same Matlab
Central web interface I was using. I probably won't access the content
"there" (Matlab's web ui) in the future because it seems to drop
messages and it seems to encourage top-posting.

I guess that for most users there was no need for me to repost
Walter's method #1. Sorry about that. When viewing the content via the
Matlab web ui, I thought I was doing everyone a favor.