Reading a text file containing strings and numeric data

Hi,

I have a .dat file like this:

Time 0.000000400

154 -1 1.77472 1.34813 0.14000 -0.00418 -0.00188 0.00042

155 -1 0.32897 1.41804 0.14000 0.00034 -0.00181 -0.00068

Time 0.000000500

389 -1 1.28893 1.39899 0.12000 0.00304 -0.00191 -0.00033

Time 0.000000600

512 -1 1.86454 1.13224 0.12000 0.00433 -0.00175 -0.00084

520 -1 0.21448 1.44465 0.12000 -0.00033 -0.00190 0.00037

Time 0.000000800

677 -1 1.14717 1.54507 0.12000 -0.00272 -0.00194 0.00005

692 -1 0.64696 1.33344 0.12000 -0.00145 -0.00188 0.00048

697 -1 0.46609 1.33894 0.12000 -0.00097 -0.00187 0.00050

I only need store the numbers and the digital time into two seperate array.

The number of rows of data at each time is not the same, but the number of columns is the same.

Final arrays should look like this:

For numerical data:

154 -1 1.77472 1.34813 0.14000 -0.00418 -0.00188 0.00042

155 -1 0.32897 1.41804 0.14000 0.00034 -0.00181 -0.00068

389 -1 1.28893 1.39899 0.12000 0.00304 -0.00191 -0.00033

512 -1 1.86454 1.13224 0.12000 0.00433 -0.00175 -0.00084

520 -1 0.21448 1.44465 0.12000 -0.00033 -0.00190 0.00037

677 -1 1.14717 1.54507 0.12000 -0.00272 -0.00194 0.00005

692 -1 0.64696 1.33344 0.12000 -0.00145 -0.00188 0.00048

697 -1 0.46609 1.33894 0.12000 -0.00097 -0.00187 0.00050

For digital time:

0.000000400

0.000000500

0.000000600

0.000000800

Total data exceeds 5 million rows, I tried to use xlsread function, but this function is limited to around 1 million rows.

Any help would be appreciated.

[Jan]

Does the 2nd column contain the value -1 only?

[ZHU QIXUAN]

Hi Jan, thanks for reply.

2nd column may also contains other numbers.

Answer 1

[Jan]

fid = fopen(File, 'r');
if fid < 0, error('Cannot open file'); end

TimeLen   = 1e6;
TimeIndex = 0;
Time      = zeros(1, TimeLen);

DataLen   = 5e6
DataIndex = 0;
Data      = zeros(8, DataLen);

while 1
    C = fgetl(fid);
    if ~ischar(C)  % or: feof(fid)
        break;
    end
    
    if strncmpi(C, 'Time', 4)
        TimeIndex = TimeIndex + 1;
        if TimeIndex > TimeLen
            TimeLen = TimeLen + 1000;
            Time(TimeLen) = 0;    % Append zeros
        end
        Time(TimeIndex) = sscanf(C, 'Time %g', 1);
    else
        DataIndex = DataIndex + 1;
        if DataIndex > DataLen
            DataLen = DataLen + 10000;
            Data(:, DataLen) = 0;   % Append zeros
        end
        Data(:, DataIndex) = sscanf(C, '%g', [8, 1]);
    end
end
fclose(fid);

Time = Time(1:TimeIndex);      % Crop unused memory
Data = Data(:, 1:DataIndex);

An alternativ is reading the text at once:

S = fileread(FileName);
C = strsplit(S, '\n');

isTime = strncmp(C, 'Time', 4);
STime  = sprintf('%s ', C{isTime});
Time   = sscanf(STime, 'Time %g ', [1, inf]);

C(isTime) = [];
SData     = sprintf('%s ', C{:});
Data      = sscanf(SData, '%g', [8, inf]).';

This will take more RAM, but I assume it is faster.

By the way: Text files are useful, if the are read or edited by human. In your case a binary file with a simple structure would have been more efficient.

[ZHU QIXUAN]

Hi Jan,

That really helps a lot! I like your second method, which can be finished in 30s. Thank you so much.