MATLAB Answers

0

importing large 4GB+ tab-delimited ascii file makes MATLAB crash

Asked by Sean Phillips on 17 May 2019
Latest activity Answered by Sean Phillips on 18 May 2019
i didnt have a problem until now the data files have been scaled up in complexity
i usually use
IMPORT = importdata('components.dat')
in this case components.dat is 4 GB+
it is a table of numerical values consisting of 12 columns and millions of rows
for smaller models this has worked but now MATLAB crashes when i try to import at this size.
I have 96GB of RAM and dual Xeons.
Do i need to change setting in MATLAB or is there better ways to import this data
this is an exmaple of the file
1 0.01 -50.00 0.00 -3950.00 -2.0000E+04 0.000 -4000. -1.409E-15 -2.138E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9900E+04 0.000 -4000. -1.410E-15 -2.193E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9800E+04 0.000 -4000. -1.411E-15 -2.249E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9700E+04 0.000 -4000. -1.412E-15 -2.306E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9600E+04 0.000 -4000. -1.411E-15 -2.364E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9500E+04 0.000 -4000. -1.411E-15 -2.424E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9400E+04 0.000 -4000. -1.409E-15 -2.485E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9300E+04 0.000 -4000. -1.407E-15 -2.547E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9200E+04 0.000 -4000. -1.405E-15 -2.611E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9100E+04 0.000 -4000. -1.401E-15 -2.676E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9000E+04 0.000 -4000. -1.397E-15 -2.742E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.8900E+04 0.000 -4000. -1.392E-15 -2.810E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.8800E+04 0.000 -4000. -1.386E-15 -2.879E-15 0.00E+00 0.00E+00
it is tab delimited

  0 Comments

Sign in to comment.

3 Answers

Answer by Sean Phillips on 18 May 2019
 Accepted Answer

thank you for your answers. I found a very simple solution
using
M = dlmread('filename.dat')
as my data is asscii-delimited file of numerical data

  0 Comments

Sign in to comment.


Answer by Steven Lord
on 17 May 2019

Consider making a tall array or perhaps using memmapfile. These tools are linked from this documentation page.
Please also send the crash log file to Technical Support so we can investigate why MATLAB crashed.

  1 Comment

Hi Steven Thank you for your answer,
I have tried this so far using memmapfile
datamem = memmapfile('components.dat', 'Format', 'double', 'Writable', true);
D = datamem.data;
how then do i use mapped data for regualr MATLAB operations?

Sign in to comment.


Answer by per isakson
on 18 May 2019
Edited by per isakson
on 18 May 2019

That's strange. I use Win10 and R2018b (64bit). btw: What do you mean by "[...] up in complexity" ?
Anyhow try
>> num = cssm( 'components.dat' );
>> whos num
Name Size Bytes Class Attributes
num 13x12 1248 double
where
function num = cssm( ffs )
fid = fopen( ffs );
cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f' ...
, 'Delimiter','\t', 'CollectOutput',true );
fclose( fid );
num = cac{1};
end
If that doesn't work, I guess there is some strange stuff in the middle of your text file.
I added
, 'ReturnOnError',false
to the textscan call and the string, "an error", to the text file an run
>> tic, num = cssm( 'cssm.txt' );toc
Error using textscan
Mismatch between file and format character vector.
Trouble reading 'Numeric' field from file (row number 1005147, field number 1) ==> an error\n
Error in cssm (line 4)
cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f' ...
>>
textscan() found my string "an error" on the row 1005147.

  0 Comments

Sign in to comment.