Thread Subject: very slow write to text file

I am using a quite simple m-file to process numerical data
from a 200Mb+ (3M lines) text file. Values (text) are being
read from a source file, some calculations are done, and the
results written to a target text file. I have been
experimenting with different alternatives for reading the
in-data, and found a quite comfortable and speedy (20000
lines/sec) solution using the textscan function.
Unfortunately, writing the out-data is utterly slow
regardless of what function I am using (fprintf, dlmwrite,
etc). Process Monitor (SysInternals) reveals that MatLab is
writing chunks that are only between 9 and 13 bytes long,
which is disastrous for the performance of my script. The
infile is however read in 512 bytes chunks. Does anyone know
a way of boosting performance for file i/o? I am using
MatLab 7.1

"Fredrik Kronhamn" <kronhamn.removethis@ioq.uni-
jena.de.removethis> wrote in message
<g0s16v$m1o$1@fred.mathworks.com>...
> I am using a quite simple m-file to process numerical data
> from a 200Mb+ (3M lines) text file. Values (text) are
being
> read from a source file, some calculations are done, and
the
> results written to a target text file. I have been
> experimenting with different alternatives for reading the
> in-data, and found a quite comfortable and speedy (20000
> lines/sec) solution using the textscan function.
> Unfortunately, writing the out-data is utterly slow
> regardless of what function I am using (fprintf, dlmwrite,
> etc). Process Monitor (SysInternals) reveals that MatLab
is
> writing chunks that are only between 9 and 13 bytes long,
> which is disastrous for the performance of my script. The
> infile is however read in 512 bytes chunks. Does anyone
know
> a way of boosting performance for file i/o? I am using
> MatLab 7.1

I probably won't be of much help but can I try to clarify
the problem. Are you trying to write the entire output file
in one go, or are you continually appending to it as you
work through the input file?

Ken

I have elaborated different solutions, i.e. dumping an
entire array with dlmwrite, linewise output with fprintf,
and buffering the output with fprintf. Matlab is however
refusing to write larger chunks of data independent of what
approach I am using.

Maybe it is possible to do some low-level hacking (i.e.
registry settings) to change this behavior?

"Fredrik Kronhamn"
<kronhamn.removethis@ioq.uni-jena.de.removethis> wrote in
message <g0ul9i$nlf$1@fred.mathworks.com>...
> I have elaborated different solutions, i.e. dumping an
> entire array with dlmwrite, linewise output with fprintf,
> and buffering the output with fprintf. Matlab is however
> refusing to write larger chunks of data independent of what
> approach I am using.
>
> Maybe it is possible to do some low-level hacking (i.e.
> registry settings) to change this behavior?

I had the same problem. Its ridiculous, but my solution was
to write a mex file, which turned writing a 3x900000 matrix
from taking 30 seconds to 2 seconds. The time hog in the
matlab fprintf seems to be in the string formatting, since
sprintf is about as slow as fprintf.

I'll include my mex below, but it is not very general. You
supply the filename, the way to open it ('a' or 'w'), the
data, and the number of elements per text line. It will
write each element in '%g' format. Revise as needed (and if
you make a more general one, I'd be happy to get it:
d+hoiem+at+uiuc+.edu (without the +)).

- Derek

----------

void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const
mxArray *prhs[]){
 
  int strlen;
  char* filename;
  char* opentype;
  double* data;
  int ndata;
  int n;
  FILE* fid;
  int linelen;
  int k;
  int status;
  
  /* check for the proper no. of input and outputs */
  if (nrhs != 4)
    mexErrMsgTxt("4 input arguments are required: filname
fopen_mode data linelen");
  if (nlhs>0)
    mexErrMsgTxt("Too many outputs");
  
  // Read Inputs
  strlen = mxGetN(prhs[0])*sizeof(mxChar)+1;
  filename = (char*)mxMalloc(strlen);
  mxGetString(prhs[0], filename, strlen);
  
  strlen = mxGetN(prhs[1])*sizeof(mxChar)+1;
  opentype = (char*)mxMalloc(strlen);
  mxGetString(prhs[1], opentype, strlen);
  
  ndata = mxGetNumberOfElements(prhs[2]);
  data = (double*)mxGetData(prhs[2]);
    
  linelen = (int)mxGetScalar(prhs[3]);
  fid = fopen(filename, opentype);
  
  if (fid==0)
    printf("Failed to open %s\n", filename, fid);
  else {

      for (n=0; n<ndata; n+=linelen) {
          for (k=0; k<linelen; k++) {
          /*printf("%d\n", n);*/
              fprintf(fid, "%g ", (double)data[n+k]);
          }
          fprintf(fid, "\n");
      }
      fclose(fid);
  }
  
  mxFree(filename);
  mxFree(opentype);
  
}