How to save .mat files on the disk after each iteration?

21 views (last 30 days)
I am facing difficulties how to save the data as struct after each loop.
I have huge image matrix and its respective labels matrix ,I want to save them after each iteration in a folder, and then delete them after each iteration from workspace, any suggestion?
Please note that I want to save the data in the disk and clear them after each iteration and save

Accepted Answer

Dyuman Joshi
Dyuman Joshi on 14 Nov 2023
The values will be over-written in each iteration, so you don't need to clear them.
for k = 1:number_of_iterations
%get images and labels for this iteration
%save them as MAT files, the variable names should be provided as a string or a char vector
%As you have mentioned that there is a large data for each iteration
%use the (MAT File) version 7.3
save("images"+k+".mat", "images", "-v7.3")
save("labales"+k+".mat", "labels", "-v7.3")
Stephen23 on 15 Nov 2023
Edited: Stephen23 on 15 Nov 2023
P = 'absolute or relative path to the parent directory';
S = dir(fullfile(P,'*Data','*.mat'));
If you want the folder and filenames sorted into alphanumeric order then either 1) add sufficient lrading zeros to the folder/filenames or 2) sort them yourself, e.g. by downloading NATSORTFILES:
S = natsortfiles(S); % DOWNLOAD
S contains all filenames together with their folders and other file meta-data. You could loop over all files:
for k = 1:numel(S)
F = fullfile(S(k).folder,S(k).name);
D = load(F);
.. do whatever with D (except concatenate them into a 64 GB array)
Of course you could also filter S for specific file/folder names. Or modify the DIR match expression to only match "labels" or "TRO" at the start of the filenames... whatever.
"I cant find clear method to deal with this data properly , I want to use this data in SVD and NN but before that I want to do some computations as I mentioned in the previous thread"
If this is the same data discussed in your other threads then you (probably) cannot fit all of that data into memory. So there is likely no point in trying to concatenate all of those arrays together. But you could supply those filenames to DATASTORE (of some kind) and use tall arrays to process it.... if you have sufficient time to spare (perhaps days... or weeks... or more... who knows).
Before you decide to attempt SVD on a 64GB array you should probably do an internet search on SVD of large arrays.
I suspect that you do not really appreciate that computers do not have infinite memory and infinite speed.
Stephen23 on 15 Nov 2023
Edited: Stephen23 on 15 Nov 2023
"I cant do that since the name inside each .mat file is the same "
How exactly do the variable names prevent you from processing your data?
Having exactly the same variable names in every MAT file makes your data easier to work with. It also makes your code simpler, more efficient, and more robust:
What exactly is stopping you from LOADing into an output variable and accessing its fields?
Show us the exact code that you tried.
"I want to pick up from each folder 15 files of images (4D double) and their respective files of labels and combine the images and combine the labels"
In this thread you wanted to combine around 400 "images" into a 64GB array:
Now you only want to combine "15 files of images".
Which is correct? When your explanations keep changing then it makes it hard to understand what you want.

Sign in to comment.

More Answers (1)

Catalytic on 14 Nov 2023
Edited: Catalytic on 14 Nov 2023
Often, you would put all images with the same label in the same folder -
for j=1:numel(Folders)
for i=1:10
save( fullfile( Folders{j}, "Image_"+num2str(i,'%3d') ) , 'image')
M on 14 Nov 2023
Edited: M on 14 Nov 2023
@Catalytic I don't want to put all the images of the same label together for now
I just want to save the two .mat files as it is with their iterations name in the same of the code directory to not overwrite.
As well I got the following error after trying your code:
Error using save
Argument must be a text scalar.
Catalytic on 14 Nov 2023
We are not here to refine code for you down to the last detail. The example clearly shows how you can create file names in a loop to use with the save() command. Adapt it as you see fit.

Sign in to comment.


Find more on Large Files and Big Data in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!