How to import and read several large csv in matlab

Dear all,

I have a huge amont of large csv files (about 170Mb each).

I need to open them and use the information in two of the columns to plot, let's say collumn D and E (see the attachment).

This is what I did so far, however I found that it is not very efficient.

%Load directory
directorio=uigetdir;
x=dir([directorio,'/','*.csv']); 
K=length(x); 

Atotal=[];
tabletotal=[];
fprintf('Leitura: \n')
tic;
for k=1:K 
    FILENAME =  strcat(x(k).folder,'/', x(k).name);
    
    TotalData=readtable(FILENAME);
    table=TotalData(:,[4 7 18 19 27]); %4-vessel name; %7-vessel type; 18-longitude; 19-latitude
    fprintf('File_%s: %s \n', num2str(k), FILENAME);
    
    tabletotal=[table; tabletotal];
    
    %Coluna 2= longitude
    %Coluna 3= latitude

end
loadAIS=toc;
fprintf('Tempo de carregamento do AIS: %s \n', num2str(loadAIS) );
clear table;
clear TotalData;
A = table2array(tabletotal(:,2:5));

Thank you in advance

[M. A. Hopcroft]

Which part specifically is not very efficient?

Answer 1

[Sulaymon Eshkabilov]

If your data file names are sequential, then you may consider using this type of approach here:

FILE = fullfile('C:\Users\...', '*.csv');  % Directory where the files are residing. 
LIST = dir(FILE);
N = length(LIST);                          % N = number of files 
D = zeros(???, ???, N);                    % Memory allocation to speed up the process of data storing
for ii = 1 : N   
    FFName = fullfile(LIST(ii).folder, LIST(ii).name);
    DATA = readmatrix(FFName);                
    D(:, :, ii) = DATA;                    % NOW D contains all data from 10000 files
end
...  % Select and process the part of D that is necessary
    

An alternative way might be this one:

P=pwd;   % If your files are located in the current directory
P = fullfile('C:\...\...', '*.csv'); % Specify: directory where the files are residing. 
S = dir(fullfile(P, '*.csv'));  % Select the file extension to suit your data files
M = 0;
for k = 1:numel(S) 
    F = fullfile(P, S(k).name);
    D = D + readmatrix(F);                
end
...

[Ricardo Duarte]

Thank you @Sulaymon Eshkabilov, but is there any equivalent function of readmatrix?

[Sulaymon Eshkabilov]

readmatrix() and readtable() are recommended ones due to their efficiency instead of xlsread() or csvread(), etc.