Read one column only of a large file, line by line?

I have a large table e.g. 7836x72001. I am interested in extracting the first column from the table which is a vector of datenumbers. Would there be a way of doing this quickly rather than waiting to load in the entire file?

I have considered using fopen and fgetl but this is taking a long time.

folder=('Y:\SoundTrap\Boats\PSD Output\Duty cycle data\');
files = dir(fullfile(folder,'*.csv'));

nf = length(files);
i=1;
a=1;

for i = 1:nf
    filename = files(i).name;
    disp(filename);
    
    try
        fid=fopen(fullfile(folder,filename));
        tline=fgetl(fid); %read first line (column headers)
        a=1; %relevant row of new table we are adding data to
        
    while ~feof(fid) %read every row until last row reached
        
        tline=fgetl(fid); %read next line
        t=tline(1); 
        
        output(a)=t;
        a=a+1;
    end
 
    writetable(a,fullfile('L:\PSD output\Boats\Duty cycle data\TVEC',strcat('TVEC_', filename)));    
    
end

Answer 1

[Walter Roberson]

I would suggest textscan with a format that is a %T (or %D as appropriate), followed by %*[\n] to consume the rest of the line.

textscan is much faster than looping yourself. Although you know that you could "do less work" by not processing the rest of the line, your knowledge would be implemented at the m-script level, with the threaded interpreter and Just In Time engine, whereas textscan is compiled into machine code, much lower overhead. And you need to parse the characters for end of line anyways, so textscan in this form is more efficient than one might think.

[Louise Wilson]

Thanks Walter! I was not aware of textscan. I am using the format %t as the format is datenum and all looks to be good. Thank you! Would it also be faster to read the full file this way? In the past I have been using readtable and I am now wondering if the whole time I could have used textscan instead to read these large files... the first column is datenum and the first row is frequency (1:72000), and the contents of each cell is a decibel value like 82.34.

[Louise Wilson]

(it can take up to 20 mins to read a file using readtable)

[Walter Roberson]

My understanding is that readtable on text files calls textscan, except possibly for fixed-width text... I don't know how fixed-width text is handled.

Answer 2

[Seth Furman]

You might also want to consider using a tall table.

https://www.mathworks.com/help/matlab/import_export/tall-arrays.html