Textscan Returning An Array of Squares

Hello all...somewhat novice user here. I am trying to import some very large text files in blocks, as I max out 16 gigs of RAM trying to use the wizard.

The file has 23 lines of header followed by about 43 million lines of a time value (floating point) and a pressure value (floating point). Tab delineated....created by LabVIEW.

I am using the following code:

fid = fopen('k3_south.txt');

i = 24;
k = 0;
formatSpec = '%f %f';

while ~feof(fid)
    i = i+1;
    k = k+1;
    L = textscan(fid, '%s %s', 'Delimiter', '|');
    Data(i, :) = L;
end

fclose(fid)

Right now the script will run indefinitely until terminated and creates an array that is far, far longer than the original file. Additionally, as stated, the array is nothing but empty little square characters.

I've done a bunch of reading on this topic and can't seem to figure out what the problem is. Any help is appreciated!

Answer 1

[Star Strider]

I suspect that the ‘empty little square characters’ mean that it may not be an ASCII file. I would check to be sure that LabVIEW is writing it in the format you expect, and with the delimiter you want. (Also, you may want to break it up into several shorter files to be sure it will fit in the available memory in your computer.)

Alternatively, LabVIEW could be writing a binary file that could be read with MATLAB’s load function. I would first see if load would work before going further to troubleshoot the problem.

And also see if the ‘Import Data’ Wizard UI can read the file.

Answer 2

[Jim Browne]

Import Data can read the file, it just ends up using up all of the memory on the machine (16Gb) and locking it up.

I haven't had trouble with files of this format before, just with the size.