read a large number of hdf5 files with for loop and store in new variables

Hello everyone,

I have about 1400 files in a folder in hdf5 format. A large number of data is stored in these files and I want to extract those numbers and and store in some variables I want. The name of these hdf5 files follows a specific order you can see in image below.

what I want is a "for loop" to choose each file, read it, and store its data to a new variable with a new index. To do so, I would like to do something like following, however, I know it doesn't work as "i" doesn't change in "ids", "fn", and "ft" names to generate a new variable with each loop. Please let me know what should I do. Thanks.

num_files = 700;
for i = 1 : num_files
    ids_i = h5read('D:\University\Mcmaster University\Ph.D. Thesis\Data\3D Rigid Walls\Same Stress Path\Result\MATLAB\_triax_micro100kpa\1.h5','/interactions/ids');
    fn_i = h5read('D:\University\Mcmaster University\Ph.D. Thesis\Data\3D Rigid Walls\Same Stress Path\Result\MATLAB\_triax_micro100kpa\1.h5','/interactions/normalForce');
    ft_i = h5read('D:\University\Mcmaster University\Ph.D. Thesis\Data\3D Rigid Walls\Same Stress Path\Result\MATLAB\_triax_micro100kpa\1.h5','/interactions/shearForce');
end

[Stephen23]

"Please let me know what should I do"

Use a cell array, just as the MATLAB documentation shows:

https://www.mathworks.com/help/matlab/import_export/process-a-sequence-of-files.html

Do NOT generate new variables on each loop iteration, unless you really want to force yourself into writing slow, inefficient, complex, obfuscated, buggy code that is hard to debug. Indexing is neat, simple, and much more efficient.

[Ehsan Ben]

I want to make new variables as I have some post proccesing manipulation to do on these data. To do so, my code is like the following by now, however, it still doesn't work:

file1_format = h5info('D:\University\Mcmaster University\Ph.D. Thesis\Data\3D Rigid Walls\Same Stress Path\Result\MATLAB\_triax_micro100kpa\1.h5');
file1bn_format = h5info('D:\University\Mcmaster University\Ph.D. Thesis\Data\3D Rigid Walls\Same Stress Path\Result\MATLAB\_triax_micro100kpa\1_branch_normal.h5');
path = 'D:\University\Mcmaster University\Ph.D. Thesis\Data\3D Rigid Walls\Same Stress Path\Result\MATLAB\_triax_micro100kpa\';
numFiles = 648;
mydata = cell(1, numFiles);
for i = 1 : numFiles*2
    j = mod(i,2); % j=1 >> odd , j=0 >> even
    if j == 1
        FileName = sprintf('%d.h5',i);
    else
        FileName = sprintf('%d_branch_normal.h5',i);
    end
    ids_{i} = h5read('D:\University\Mcmaster University\Ph.D. Thesis\Data\3D Rigid Walls\Same Stress Path\Result\MATLAB\_triax_micro100kpa\FileName','/interactions/ids');
    fn_{i} = h5read('D:\University\Mcmaster University\Ph.D. Thesis\Data\3D Rigid Walls\Same Stress Path\Result\MATLAB\_triax_micro100kpa\FileName','/interactions/normalForce');
    ft_{i} = h5read('D:\University\Mcmaster University\Ph.D. Thesis\Data\3D Rigid Walls\Same Stress Path\Result\MATLAB\_triax_micro100kpa\FileName','/interactions/shearForce');
end

Answer 1

[Stephen23]

"I want to make new variables as I have some post proccesing manipulation to do on these data."

You might want to do that, but using indexing is simpler, easier, more reliable, and more efficient. Using indexing makes looping over multiple files and storing their data very easy (unlike what you are trying to do). The MATLAB documentation recommends against creating new variables on the fly, as do all experienced MATLAB users.

Your code has several bugs and "features", e.g. you did not pass the generated filename as a variable (instead you wrote the variable name as part of some character vectors), you preallocate a cell array which you then ignore completely, and shadowing the inbuilt PATH function.

I recommend that you specify the path once and use FULLFILE instead of repeating the same hard-coded data over and over again (not a very nice code smell). This should get you started:

N = 648: % highest file number.
P = 'D:\University\Mcmaster University\Ph.D. Thesis\Data\3D Rigid Walls\Same Stress Path\Result\MATLAB\_triax_micro100kpa';
fmt_1   = h5info(fullfile(P,'1.h5'));
fmt_1bn = h5info(fullfile(P,'1_branch_normal.h5'));
ids = cell(N,2);
fn  = cell(N,2);
ft  = cell(N,2);
for k = 1:N
    F = fullfile(P,sprintf('%d.h5',k));
    ids{k,1} = h5read(F,'/interactions/ids');
    fn{k,1}  = h5read(F,'/interactions/normalForce');
    ft{k,i}  = h5read(F,'/interactions/shearForce');
    F = fullfile(P,sprintf('%d_branch_normal.h5',k));
    ids{k,2} = h5read(F,'/interactions/ids');
    fn{k,2}  = h5read(F,'/interactions/normalForce');
    ft{k,2}  = h5read(F,'/interactions/shearForce');
end