How to read all the files in a folder based on file name?

Dear MATLAB Experts,

I have hundreds of files in a folder and I need toread all the files using last 5 digits of file name.

RSS_SSMIS_FCDR_V07R00_F17_D20220701_S0134_E0326_R80783

Here problem is this information is random in each file S0134_E0326.

However R80783 is increasing for next file, as shown in image.

%% code for readin file is

clear;clc; close;

ncfile_1 = 'RSS_SSMIS_FCDR_V07R00_F17_D20220701_S0134_E0326_R80783.nc' ;

ncinfo(ncfile_1)

ncdisp(ncfile_1)

Answer 1

[Mathieu NOE]

hello

you can do a for loop to load all your files

I am not sure what your issue is, but if it's how to make sure to load files according to the last 5 digits, you can check that dir combined with natsortfiles guarantees to load your files in ascending order , whatever the file name is before the 5 digits

natsortfiles is available on the Fex :

Natural-Order Filename Sort - File Exchange - MATLAB Central (mathworks.com)

try it !

this is my result (I created the first 3 files of your list) - you can see the files are sorted out correctly

filename = 'RSS_SSMIS_FCDR_V07R00_F17_D20220701_S0134_E0326_R80783.nc'

filename_last5digits = 80783

filename = 'RSS_SSMIS_FCDR_V07R00_F17_D20220701_S0316_E0508_R80784.nc'

filename_last5digits = 80784

filename = 'RSS_SSMIS_FCDR_V07R00_F17_D20220701_S0458_E0650_R80785.nc'

filename_last5digits = 80785

                     
fileDir = pwd; % current directory (or specify which one is the working directory)

S = dir(fullfile(fileDir,'*.nc')); % get list of data files in directory
S = natsortfiles(S); % sort file names into natural order , see : 
           %(https://fr.mathworks.com/matlabcentral/fileexchange/47434-natural-order-filename-sort)
N = numel(S);

for k = 1:N
    filename = S(k).name % plot the file name in command window (on purpose)
    filename_last5digits = str2double(extractBetween(filename,'_R','.nc'))  % plot the file name last 5 digits in command window (on purpose)
%     % Load in a few selected variables
%     u = ncread(filename,'XXX');

end

[Amjad Iqbal]

@Mathieu NOE

Thank you for the solution you provided. I'm able to read all the files stored in the structure. However, I need the data to be arranged in matrices. For example, if the size of each file's data is 180x3542, then for the given 8 files, I should obtain a 3D matrix with dimensions 180x3542x8. Could you please advise on how to organize the data into matrices accordingly?

[Mathieu NOE]

hello again

so you have 8 nc files in your folder if I understand correctly

you can generate a 3D array this way :

(as I don't have the real nc files I replace the output from ncread with dummy random 2D array of size (180,3542))

fileDir = pwd; % current directory (or specify which one is the working directory)

S = dir(fullfile(fileDir,'*.nc')); % get list of data files in directory
S = natsortfiles(S); % sort file names into natural order , see : 
           %(https://fr.mathworks.com/matlabcentral/fileexchange/47434-natural-order-filename-sort)
N = numel(S);

for k = 1:N
    filename = S(k).name % plot the file name in command window (on purpose)
    filename_last5digits = str2double(extractBetween(filename,'_R','.nc'))  % plot the file name last 5 digits in command window (on purpose)
    % Load data
%     u = ncread(filename,'XXX');
    u = rand(180,3542); % dummy data 
    
    data(:,:,k) = u; % your 3D data 

end

[Mathieu NOE]

On your side , you can avoid to create the intermedaite variable u

you can directly perform :

data(:,:,k) = ncread(filename,'XXX');

[Amjad Iqbal]

@Mathieu NOE

Thank you. Indeed, this script reads and provides a 3D array. Just one query: what should 'XXX' be for my given data files? I have uploaded 8 files for your reference. Could you check if this script loads the files?

https://drive.google.com/drive/folders/16CNnvsf6jlQdUpitTyLmIFOVBg8Qv9hy?usp=sharing

Much appreciated for your time.

[Amjad Iqbal]

@Mathieu NOE @Cris LaPierre @Adam Danz @Stephen23

Dear All MATLAB Expert,

With your guidelines and suggestions, I managed to read all the 438 fiels as 3D array. This is what I needed and I'm glad to be guided by you guys. Much appreciated for your precious time and efforts.

% Define the directory where your files are located
directory = 'PWD';
% List all files in the directory
files = dir(fullfile(directory, '*.nc'));
% Check if there are any .nc files in the directory
if isempty(files)
    error('No .nc files found in the directory.');
end
% Read the brightness image from the first file to get dimensions
ncfile_first = fullfile(directory, files(1).name);
B_first = ncread(ncfile_first, 'fcdr_brightness_temperature_92H');
[rows, cols] = size(B_first);
max_rows = rows;
max_cols = cols;
% Initialize arrays to store images
all_B_images = [];
% Read each file in the directory
for i = 1:numel(files)
    % Generate the filename dynamically
    ncfile = fullfile(directory, files(i).name);
    % Read the brightness image
    B = ncread(ncfile, 'fcdr_brightness_temperature_92H');
    % Resize or crop B image if necessary to match max dimensions
    [rows, cols] = size(B);
    if rows < max_rows || cols < max_cols
        B = imresize(B, [max_rows, max_cols]);
    elseif rows > max_rows || cols > max_cols
        B = B(1:max_rows, 1:max_cols);
    end
    % Store the images
    all_B_images = cat(3, all_B_images, B);
end

[Mathieu NOE]

hello again

seems the size varies depending on which file is loaded

here the code will display the filename and what data / size (2D arrays only) it contains

I can find variables of size either 180 x 3541 or 180 x 3542

Of course , you cannot concatenate arrays if they have diferent sizes

either we keep ony data from files which data have all same size (? ) or we truncate the data that are bigger (??)

code

fileDir = pwd; % current directory (or specify which one is the working directory)

S = dir(fullfile(fileDir,'*.nc')); % get list of data files in directory
S = natsortfiles(S); % sort file names into natural order , see : 
           %(https://fr.mathworks.com/matlabcentral/fileexchange/47434-natural-order-filename-sort)
N = numel(S);

for k = 1:N
    filename = S(k).name % plot the file name in command window (on purpose)
    
%     % Info about the file content 
%     % Display global information about a netcdf file
%     ncdisp(filename);

    % Display included Variables and  Size (2D arrays only are displayed)
    meta = ncinfo(filename);
    disp('Variable List');
    VN = {meta.Variables.Name}';
    VNsize = {meta.Variables.Size}';
    for k = 1:numel(VNsize)
        s(k)= size(VNsize{k},2);
    end
    
    ind = s>1; % (2D arrays only are displayed)
    VN = VN(ind);
    VNsize = VNsize(ind);
    
    for k = 1:numel(VNsize)
        tmp{k}= num2str(VNsize{k});
    end    
    disp([VN tmp']);
    
    
%     % Load data
%     tmp = ncread(filename,'fcdr_brightness_temperature_92H'); % for example
%     size(tmp)

end

results (command window) :

filename = 'RSS_SSMIS_FCDR_V07R00_F17_D20230101_S0222_E0414_R83384.nc'

Variable List

{'iscn_flag' } {'3541 11'}

{'ical_flag_lores' } {'3541 4'}

{'ical_flag_hires' } {'3541 4'}

{'latitude_hires' } {'180 3541' }

{'longitude_hires' } {'180 3541' }

{'earth_incidence_angle_hires' } {'180 3541' }

{'earth_azimuth_angle_hires' } {'180 3541' }

{'sun_glitter_angle_hires' } {'180 3541' }

{'land_flag_hires' } {'180 3541' }

{'ice_flag_hires' } {'180 3541' }

{'fcdr_brightness_temperature…'} {'180 3541' }

{'fcdr_brightness_temperature…'} {'180 3541' }

{'latitude_lores' } {'90 3541' }

{'longitude_lores' } {'90 3541' }

{'earth_incidence_angle_lores' } {'90 3541' }

{'earth_azimuth_angle_lores' } {'90 3541' }

{'sun_glitter_angle_lores' } {'90 3541' }

{'land_flag_lores' } {'90 3541' }

{'ice_flag_lores' } {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

filename = 'RSS_SSMIS_FCDR_V07R00_F17_D20230101_S0404_E0556_R83385.nc'

Variable List

{'iscn_flag' } {'3542 11'}

{'ical_flag_lores' } {'3542 4'}

{'ical_flag_hires' } {'3542 4'}

{'latitude_hires' } {'180 3542' }

{'longitude_hires' } {'180 3542' }

{'earth_incidence_angle_hires' } {'180 3542' }

{'earth_azimuth_angle_hires' } {'180 3542' }

{'sun_glitter_angle_hires' } {'180 3542' }

{'land_flag_hires' } {'180 3542' }

{'ice_flag_hires' } {'180 3542' }

{'fcdr_brightness_temperature…'} {'180 3542' }

{'fcdr_brightness_temperature…'} {'180 3542' }

{'latitude_lores' } {'90 3542' }

{'longitude_lores' } {'90 3542' }

{'earth_incidence_angle_lores' } {'90 3542' }

{'earth_azimuth_angle_lores' } {'90 3542' }

{'sun_glitter_angle_lores' } {'90 3542' }

{'land_flag_lores' } {'90 3542' }

{'ice_flag_lores' } {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

filename = 'RSS_SSMIS_FCDR_V07R00_F17_D20230101_S0546_E0738_R83386.nc'

Variable List

{'iscn_flag' } {'3541 11'}

{'ical_flag_lores' } {'3541 4'}

{'ical_flag_hires' } {'3541 4'}

{'latitude_hires' } {'180 3541' }

{'longitude_hires' } {'180 3541' }

{'earth_incidence_angle_hires' } {'180 3541' }

{'earth_azimuth_angle_hires' } {'180 3541' }

{'sun_glitter_angle_hires' } {'180 3541' }

{'land_flag_hires' } {'180 3541' }

{'ice_flag_hires' } {'180 3541' }

{'fcdr_brightness_temperature…'} {'180 3541' }

{'fcdr_brightness_temperature…'} {'180 3541' }

{'latitude_lores' } {'90 3541' }

{'longitude_lores' } {'90 3541' }

{'earth_incidence_angle_lores' } {'90 3541' }

{'earth_azimuth_angle_lores' } {'90 3541' }

{'sun_glitter_angle_lores' } {'90 3541' }

{'land_flag_lores' } {'90 3541' }

{'ice_flag_lores' } {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

filename = 'RSS_SSMIS_FCDR_V07R00_F17_D20230101_S0728_E0920_R83387.nc'

Variable List

{'iscn_flag' } {'3541 11'}

{'ical_flag_lores' } {'3541 4'}

{'ical_flag_hires' } {'3541 4'}

{'latitude_hires' } {'180 3541' }

{'longitude_hires' } {'180 3541' }

{'earth_incidence_angle_hires' } {'180 3541' }

{'earth_azimuth_angle_hires' } {'180 3541' }

{'sun_glitter_angle_hires' } {'180 3541' }

{'land_flag_hires' } {'180 3541' }

{'ice_flag_hires' } {'180 3541' }

{'fcdr_brightness_temperature…'} {'180 3541' }

{'fcdr_brightness_temperature…'} {'180 3541' }

{'latitude_lores' } {'90 3541' }

{'longitude_lores' } {'90 3541' }

{'earth_incidence_angle_lores' } {'90 3541' }

{'earth_azimuth_angle_lores' } {'90 3541' }

{'sun_glitter_angle_lores' } {'90 3541' }

{'land_flag_lores' } {'90 3541' }

{'ice_flag_lores' } {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

filename = 'RSS_SSMIS_FCDR_V07R00_F17_D20230101_S0910_E1102_R83388.nc'

Variable List

{'iscn_flag' } {'3542 11'}

{'ical_flag_lores' } {'3542 4'}

{'ical_flag_hires' } {'3542 4'}

{'latitude_hires' } {'180 3542' }

{'longitude_hires' } {'180 3542' }

{'earth_incidence_angle_hires' } {'180 3542' }

{'earth_azimuth_angle_hires' } {'180 3542' }

{'sun_glitter_angle_hires' } {'180 3542' }

{'land_flag_hires' } {'180 3542' }

{'ice_flag_hires' } {'180 3542' }

{'fcdr_brightness_temperature…'} {'180 3542' }

{'fcdr_brightness_temperature…'} {'180 3542' }

{'latitude_lores' } {'90 3542' }

{'longitude_lores' } {'90 3542' }

{'earth_incidence_angle_lores' } {'90 3542' }

{'earth_azimuth_angle_lores' } {'90 3542' }

{'sun_glitter_angle_lores' } {'90 3542' }

{'land_flag_lores' } {'90 3542' }

{'ice_flag_lores' } {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

filename = 'RSS_SSMIS_FCDR_V07R00_F17_D20230101_S1052_E1244_R83389.nc'

Variable List

{'iscn_flag' } {'3541 11'}

{'ical_flag_lores' } {'3541 4'}

{'ical_flag_hires' } {'3541 4'}

{'latitude_hires' } {'180 3541' }

{'longitude_hires' } {'180 3541' }

{'earth_incidence_angle_hires' } {'180 3541' }

{'earth_azimuth_angle_hires' } {'180 3541' }

{'sun_glitter_angle_hires' } {'180 3541' }

{'land_flag_hires' } {'180 3541' }

{'ice_flag_hires' } {'180 3541' }

{'fcdr_brightness_temperature…'} {'180 3541' }

{'fcdr_brightness_temperature…'} {'180 3541' }

{'latitude_lores' } {'90 3541' }

{'longitude_lores' } {'90 3541' }

{'earth_incidence_angle_lores' } {'90 3541' }

{'earth_azimuth_angle_lores' } {'90 3541' }

{'sun_glitter_angle_lores' } {'90 3541' }

{'land_flag_lores' } {'90 3541' }

{'ice_flag_lores' } {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

{'fcdr_brightness_temperature…'} {'90 3541' }

filename = 'RSS_SSMIS_FCDR_V07R00_F17_D20230101_S1234_E1426_R83390.nc'

Variable List

{'iscn_flag' } {'3542 11'}

{'ical_flag_lores' } {'3542 4'}

{'ical_flag_hires' } {'3542 4'}

{'latitude_hires' } {'180 3542' }

{'longitude_hires' } {'180 3542' }

{'earth_incidence_angle_hires' } {'180 3542' }

{'earth_azimuth_angle_hires' } {'180 3542' }

{'sun_glitter_angle_hires' } {'180 3542' }

{'land_flag_hires' } {'180 3542' }

{'ice_flag_hires' } {'180 3542' }

{'fcdr_brightness_temperature…'} {'180 3542' }

{'fcdr_brightness_temperature…'} {'180 3542' }

{'latitude_lores' } {'90 3542' }

{'longitude_lores' } {'90 3542' }

{'earth_incidence_angle_lores' } {'90 3542' }

{'earth_azimuth_angle_lores' } {'90 3542' }

{'sun_glitter_angle_lores' } {'90 3542' }

{'land_flag_lores' } {'90 3542' }

{'ice_flag_lores' } {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

filename = 'RSS_SSMIS_FCDR_V07R00_F17_D20230101_S1416_E1608_R83391.nc'

Variable List

{'iscn_flag' } {'3542 11'}

{'ical_flag_lores' } {'3542 4'}

{'ical_flag_hires' } {'3542 4'}

{'latitude_hires' } {'180 3542' }

{'longitude_hires' } {'180 3542' }

{'earth_incidence_angle_hires' } {'180 3542' }

{'earth_azimuth_angle_hires' } {'180 3542' }

{'sun_glitter_angle_hires' } {'180 3542' }

{'land_flag_hires' } {'180 3542' }

{'ice_flag_hires' } {'180 3542' }

{'fcdr_brightness_temperature…'} {'180 3542' }

{'fcdr_brightness_temperature…'} {'180 3542' }

{'latitude_lores' } {'90 3542' }

{'longitude_lores' } {'90 3542' }

{'earth_incidence_angle_lores' } {'90 3542' }

{'earth_azimuth_angle_lores' } {'90 3542' }

{'sun_glitter_angle_lores' } {'90 3542' }

{'land_flag_lores' } {'90 3542' }

{'ice_flag_lores' } {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

{'fcdr_brightness_temperature…'} {'90 3542' }

[Mathieu NOE]

ok, I see you managed the problem in your final code - good point !!

Glad we could help you and all the best for the future :)

[Adam Danz]

Great collaborative work!