Improve speed reading in a .dat file

Im attempting to process some rather heafty .dat files with 500k+ lines. The bulk of my code runtime is being taken up by two functions

fgetl and str2double

Does anyone know a faster way to read in a dat file or functions that run faster

[Stephen Cobeldick]

If at all possible, the fastest would be

https://www.mathworks.com/help/matlab/ref/fscanf.html

but whether you can use it depends on the file format, which you have not told us anything about.

If you want further help with this, upload a representative** file by clicking the paperclip button.

** This means exactly the same file format, EOL characters, delimiter, number formats, etc., but small enough so that you can upload it on this forum. Probably you achieve this by keeping only the first few hundred lines.

[Aaron Crawford]

This is essentially what the file looks like.

The example file is included as a .txt however they are actually .dat files.

TITLE="Solution flow field"

VARIABLES="X" "Y" "Z" "RHO" "RHOU" "RHOV" "RHOW" "RHOE" "RHOK" "RHOOMEGA" "MUT"

ZONE T="10005_16", I= 1 , J= 195 , K= 265 ,

DATAPACKING=BLOCK

VARLOCATION=([4-11]=CELLCENTERED)

0.42820862965225337

0.42820862965225337

0.42820862965225337

0.42820862965225337

0.42820862965225343

0.42820862965225337

It continues on with a new number every line untiil the file ends. After the first 5 lines of text there is no more text. I honestly dont even need the test at the top.

Thank you for the input. I will definitley look into the fscanf function.

Answer 1

[Image Analyst]

You could try fileread() to read in the whole file into one variable in one shot, then go through it parsing it.

Answer 2

[Stephen Cobeldick]

Some version of this is probably the fastest you can get:

[fid,msg] = fopen('example.txt','rt');
assert(fid>=3,msg)
for k = 1:5 % 5 header lines to ignore
    fgetl(fid);
end
vec = fscanf(fid,'%f'); % fast!
fclose(fid);

fprintf('%.17f\n',vec)

0.42820862965225337
0.42820862965225337
0.42820862965225337
0.42820862965225337
0.42820862965225343