Concatenate variables from multiple files to just one single matlab file

Hi everyone, I have a folder that contains Matlab files in the format 'station001.mat', 'station002.mat' to lets say 'station035.mat' It has the following variabes 'stationI' and 'stationV' . all the files have the variable name as the same but with different number of values i.e 'stationI' and 'stationV' of 'station001.mat' may have 200 vales each but 'stationI' and 'stationV' of 'station002.mat' may have 300 values each. I need a single Matlab file named station.mat having variables 'stationI' and 'stationV'which contains all the variable values concatenated altogether. Could anyone help me in this, please?

Many thanks

Answer 1

[Stephen23]

This assumes that the files are returned by the OS in the required order (i.e. the filenames use leading zeros), otherwise download my FEX submission natsortfiles.

D = 'directory where the files are stored';
S = dir(fullfile(D,'station*.mat'));
S = natsortfiles(S); % (optional) alphanumeric sort by filename
C = cell(1,numel(S));
Z = load(fullfile(D,S(1).name));
F = fieldnames(Z);
for k = 2:numel(S)
    T = load(fullfile(D,S(k).name));
    for n = 1:numel(F)
        Z.(F{n}) = cat(1, Z.(F{n}), T.(F{n})); % pick a suitable dimension
    end
end
save(fullfile(D,'station.mat'),'-struct','Z')

[Samy Alkhayat]

Hello Stephen,

Trying to execute the code above, what should be "T" in the "F = fieldnames(T);" line?

Thank you in advance.

[Stephen23]

@Samy Alkhayat: it should be Z, I fixed my answer.

[Samy Alkhayat]

@Stephen23 Greatly appreciated, it now works!

[MarkusR]

Hi Stephen,

thanks for your code, it works very well!

My files are each about 2-3 GB of Data, i want to contacenate about ten (or even more) of them. At the end of the process Matlab says "not saved, use Matlab 7.3 or higher", even my version should be high enough. Any ideas?

[Stephen23]

"Matlab says "not saved, use Matlab 7.3 or higher""

The message actually states "For variables larger than 2GB use MAT-file version 7.3 or later". Exactly as the message advises, you need to specify the MAT-file version when calling SAVE:

https://www.mathworks.com/help/matlab/ref/save.html#btox10b-1-version

Note the table column giving the maximum array sizes.

[Samy Alkhayat]

Hello Stephen,

I have successfuly used the code above before, now I need for to concatenate specific vectors from those .mat files since not all 80k files I have include the same number of vectors so it thows the error below, any thoughts will be appreciated!

@Stephen23

Error using cat
Number of fields in structure arrays being concatenated do not match. Concatenation of structure arrays
requires that these arrays have the same set of fields.

[Stephen23]

"now I need for to concatenate specific vectors from those .mat files since not all 80k files I have include the same number of vectors so it thows the error below, any thoughts will be appreciated!"

If all files have the same subset of "specific" vectors (i.e. intersection), then you could define that subset yourself:

F = {'names','of','vectors','that','exist','in','all','files'};

And/or you could also add ISFIELD within the innermost FOR-loop, to check if the fields exist:

if isfield(T,F{n})
    % concatenate
end

Answer 2

[dpb]

d=dir('station*.mat');                   % return the list of files
n=length(d);                             % number files found
c=cell(n,1);                             % allocate a cell array for the returned data
for i=1:n                                % iterate over the list
  c(i)={struct2array(load(d(i).name))};  % put each set of data in the cell array

end
save station cell2mat(c)                 % put the data into station.mat

The above does save the results as a 2-column array instead of as two separate variables; if it is mandatory to keep those two names just recast the array to two variables before save.

In general, it's better practice in Matlab to use arrays and indexing over individually-named variables with metadata encoded in the variable names, however.

[Stephen23]

Note that there is no guarantee about the order of the fields returned by a .mat file, so this method loses any information related to the variable names' meta-data. One easy solution is to apply orderfields.

The command struct2cell might be of interest too.