MATLAB Answers

Matt J
0

Pre-determining the number of lines in a text file

Asked by Matt J
on 4 Jul 2013
Latest activity Commented on by Walter Roberson
on 9 Oct 2017

Is there any programmatic way of determining in advance the number of lines in a text file, for use with dlmread, textscan, etc...? I mean other than some brute force way like reading line by line in a while loop until EOF is hit.

  5 Comments

Yes, but there are still independent reasons why it would be beneficial to know the number of lines in the file. DLMREAD would provide a very easy way to read column-by-column or blocks of columns if one knows the number of lines in advance.

Suppose I want to use the syntax

 M = dlmread(filename, delimiter, range)

where I want the "range" to designate all rows in the file, but only a subset of the columns.

Honestly, that type of behavior is exactly what TEXTSCAN should be used for instead of DLMREAD. The documentation of TEXTSCAN shows nice examples of ignoring various columns, and by default it will read all rows of the file.

Well, okay, maybe that was a bad example. But surely, in general, it helps to know in advance how much data there is to read so you can plan, pre-allocate, etc...

Sign in to comment.

Products

9 Answers

Answer by Walter Roberson
on 4 Jul 2013
 Accepted Answer

The only operating system that MATLAB has ever run on that supported that ability was DEC's VMS, and for technical reasons VMS's facility for that could not be used with MATLAB.

The modern treatment of "lines" as being delimited by a particular character or character pair (e.g., LF or CR+LF) does not offer any way to count the lines short of reading through the file and counting the delimiters.

  3 Comments

I see. But I still wonder why MATLAB doesn't provide a single command that will do that. It seems like it could be a useful bit of info to extract.

Well on that note, it isn't hard for you to write a simple function that can do that...

Or, I think it should be possible to allow dlmread to specify Infs in its range argument. That could trigger the file reading to stop when the limits of the file were reached.

Sign in to comment.


Answer by Guru
on 4 Jul 2013
Edited by Guru
on 4 Jul 2013

Just out of boredom, here's a function:

function n = linecount(fid)
n = 0;
tline = fgetl(fid);
while ischar(tline)
  tline = fgetl(fid);
  n = n+1;
end

Edited: Thanks for comment Walter

  8 Comments

In the text file you used to test with, does the last line end with the line terminator, or does it just end with no terminator?

Yes, feof() has overhead.

I cannot scroll any further than the final line of text. I guess that means it ends with no terminator? Both my version and Guru's correctly count the number of lines of actual text, though.

Earlier I wrote that feof() never predicts end-of-file. That is true, but I was missing some information about the operation of fgetl and fgets that I just noticed today:

https://www.mathworks.com/help/matlab/import_export/import-text-data-files-with-low-level-io.html#br4ssin

"After each read operation, fgetl and fgets check the next character in the file for the end-of-file marker. Therefore, these functions sometimes set the end-of-file indicator before they return a value of -1.

[...]

This behavior does not conform to the ANSI specifications for the related C language functions." (emphasis added)

Sigh.

Sign in to comment.


Answer by Informaton on 29 Oct 2014

Another approach is to use the underlying operating system's functionality. Specifically, UNIX/Linux (i.e. also Mac) include a command line method 'wc -l [filename]' to get the line count of [filename].

To implement in MATLAB you could do something like this

if (~ispc) 
  [status, cmdout]= system('wc -l filenameOfInterest.txt');
  if(status~=1)
      scanCell = textscan(cmdout,'%u %s');
      lineCount = scanCell{1}; 
  else
      fprintf(1,'Failed to find line count of %s\n',filenameOfInterest.txt);
      lineCount = -1;
  end
else
  fprintf(1,'Sorry, I don''t know what the equivalent is for a windows system\n');
  lineCount = -1;
end

  1 Comment

There is actually an equivalent command for Windows-based systems using the command line. It is discussed in some length here: https://blogs.msdn.microsoft.com/oldnewthing/20110825-00/?p=9803/

The command to run in the command prompt is:

find /c /v "" filename.txt

Which can then be used in the else condition in your if-check.

 else
    % For Windows-based systems
    [status, cmdout] = system(['find /c /v "" ', filename]);
    if(status~=1)
        scanCell = textscan(cmdout,'%s %s %u');
        lineCount = scanCell{3};
        disp(['Found ', num2str(lineCount), ' lines in the file']);
    else
        disp('Unable to determine number of lines in the file');
    end
end

Sign in to comment.


Answer by Walter Roberson
on 10 Jan 2017

function n = linecount(filename)
  [fid, msg] = fopen(filename);
  if fid < 0
    error('Failed to open file "%s" because "%s"', filename, msg);
  end
    n = 0;
    while true
        t = fgetl(fid);
        if ~ischar(t)
            break;
        else
            n = n + 1;
        end
    end
    fclose(fid);

I have tested this with files that end with newline and with files that do not end with newline.

  6 Comments

I don't know that I've ever met an ASCII file that doesn't end lines with either \n or \r\n. I can understand if you don't want to publish it on the file exchange without exception handling, maybe you could email me the c file as-is and I can compile it on my machine?

EDIT: I found a much faster solution - read the entire file into memory and count the \n newline characters (uinicode 10)

>> tic; fptr = fopen(fName); s = fread(fptr); numLine = sum(s==10); fclose(fptr); toc
Elapsed time is 2.996891 seconds.
>> tic; linecount(fName); toc
Elapsed time is 16.268462 seconds.

@Peter: fread(fptr) does read the complete file and stores each byte in a double. Prefer: fread(fptr, Inf, '*uint8'), which uses less memory.

Just counting the \n can give an off-by-one error. You need to know if the final \n has any characters following it or not.

123\n456\n

has two lines.

123\n456\n7

has three lines

123\n456\n7\n

has three lines.

Sign in to comment.


Answer by Boris
on 10 Jan 2017

I came across this code a while ago which is reasonably fast and works well on large files:

   fid = fopen(strFileName, 'rt');
   chunksize = 1e6; % read chuncks of 1MB at a time
   numRows = 1;
   while ~feof(fid)
       ch = fread(fid, chunksize, '*uchar');
       if isempty(ch)
           break
       end
       numRows = numRows + sum(ch == sprintf('\n'));
   end
  fclose(fid);

strFileName is the file name for the ascii file

numRows has the total number of lines

Now, the only problem remains efficiently testing for blank lines before using csvread \ dlmread to read (sizeable) chunks of the file (ie my code is thrown if the csv file ends in a blank line so it would be nice if I could test and count the number of blank lines at the end of my files...

  2 Comments

This code has an off-by-one error for files that end in linefeeds.

If you are using OS-X or Linux, try:

! (echo line 1; echo line 2; echo line 3) > file_ends_nl.txt
! (echo line 1; echo line 2; echo -n line 3) > /file_ends_no_nl.txt

You can use !ls file*.txt to verify that they are different sizes, the one with no_nl being one byte shorter. You can use !od -cx on each file to verify that one ends in newline and the other does not.

The code will report numRows = 3 for the file with no newline at the end, but will report 4 for the file that ends in newline.

Or used the code above and check if the file ends in 0A:

    if ch(end)==10
        numRows=numRows-1;
    end

Sign in to comment.


Answer by Jan
on 10 Jan 2017

The determination of the number of lines require to read the file and interprete the line breaks. This means some real work and the disk access is the bottleneck in this case. Therefore e.g. importing the file to a cell string is not remarkably faster if you determine the number of lines at first. If the number of lines is determined initially, the main work would still be to "guess" a suiting buffer size for importing the lines. This requires either a copy of each line from the buffer to the Matlab string, or to realloc the imported string and allocate a new input buffer for each line.

I find it disappointing, that Matlab does not have a simple tool to import a text file to a cell string. Even the way to split a string (e.g. imported by fileread) to a cell considering the DOS/Linux/Mac linebreaks needed tools like strread, dataread, textread, textscan and regexp('split') which are not available in all Matlab versions and frequently outdated. Therefore I tried to write an efficient C-Mex again for the FileExchange. But the results have been grim: The best approaches have been only some percent faster than fread, replacing the different linebreaks by char(10) and calling a "Str2Cell" C-Mex. Neither counting the lines nor smart prediction techniques for a dynamic buffer allocation for the single lines accelerated the code sufficiently. The bottleneck of the disk access rules everything, even if the data are available in the cache already. For real file access, when the data are not read seconds before already and cached, all smart tricks are useless.

I think this is the reason why Matlab and many other tools do not contain a function for determine the number of lines in a text file.

If I find the time, I will try to write a LineCount.mex function, but I do not expect it to be much faster than Walter's Matlab approach.

  0 Comments

Sign in to comment.


Answer by Ken Atwell
on 30 Oct 2014

If we can make two assumptions:

  • ASCII #10 is a reliable end-of-line marker
  • The entire file will fit into memory (that is, we're not talking about Big Data)

I would do the following (using the help for the plot command in this example):

 txt=fileread(fullfile(matlabroot, 'toolbox', 'matlab', 'graph2d', 'plot.m'));
 sum(txt==10)+1

This will be fast... certainly faster than "fgetl" approach, but maybe not as fast as the "wc" approach Hyatt put forth above (assuming you can live without Windows platform support).

  1 Comment

Files are not required to end with a line terminator, but they might. So a file with 3 lines might have either 2 linefeeds (separating line 1 from line 2, separating line 2 from line 3, nothing at end of file), or 3 linefeeds (one at the end of each line.) The above code would count 4 if this hypothetical file ended with linefeed (as is more common than not.)

Sign in to comment.


Answer by Dr. Erol Kalkan, P.E. on 19 May 2016
Edited by Matt J
on 19 May 2016

Here is a short and fast way: Say file name to be read is apk.txt

fid = fopen('apk.txt','r');
Nrows = numel(textread('apk.txt','%1c%*[^\n]'));

  1 Comment

textread is deprecated.

How is your routine going to treat empty lines? I think the result is going to depend upon whether the file is CR/LF or LF delimited: in the CR/LF case the %1c is going to read the CR, leaving the LF to be matched by the %*[^\n], but in the LF case, the %1c is going to read the LF, moving the next line into position to be matched by the %*[^\n]

Sign in to comment.


Answer by John BG
on 10 Jan 2017

hi Matt

the command importdata returns a cell with all lines that are not empty of a text file.

The amount of elements of this cell is equal to the amount of lines of the text file.

file_name='filename.txt'
numel(importdata(fname))

if you find these lines useful would you please mark my answer as Accepted Answer?

To any other reader, if you find this answer of any help please click on the thumbs-up vote link,

thanks in advance for time and attention

John BG

jgb2012@sky.com

  6 Comments

>> fname = 'somedata.txt';
>> r = zeros(1,100);for K = 1 : 100; r(K) = timeit(@() size(importdata(fname),1), 0); end
>> min(r)
ans =
           0.0148051934005
>> mean(r)
ans =
           0.0156547813305
>> r2 = zeros(1,100);for K = 1 : 100; r2(K) = timeit(@() linecount(fname), 0); end
>> min(r2)
ans =
      0.000143144709071429
>> mean(r2)
ans =
       0.00015156221620381
>> min(r)/min(r2)
ans =
          103.428156699192
>> mean(r)/mean(r2)
ans =
          103.289472288058

timeit is https://www.mathworks.com/help/matlab/ref/timeit.html which is based upon https://www.mathworks.com/matlabcentral/fileexchange/18798-timeit-benchmarking-function

People other than Matt J read this, so before they implement the importdata() approach they need to know about its limitations. It is a nice compact expression that works well (if perhaps less efficient than it could be) under the circumstance of a file containing a single column of (pure) numeric values; unfortunately it turns out to be fragile if that condition is not met.

@John BG,

I'm afraid I see no special advantage to what you propose over other proposals. Walter's comments about speed and reliabilty aside, you will recall from my original post that I was asking if there was a way to determine the number of lines in the file without scanning through all the data in the file. Walter has already answered that this is impossible.

Worse, though, your solution not only reads through all the data in the file, but allocates storage for all the data simultaneously in MATLAB, something you will see I was also trying to avoid as discussed under Guru's answer.

Sign in to comment.