How to solve problem with a memory when processing a large dataset?

Consider this situation. I have 10 folders with 1000 png images in each (one file is ~4MB) and I want to extract some simple information from each image (for example sum from all pixel values) and store it for each folder separately. With the code seen below the procedure stops due to memory issue after about 2500 files (this number will be PC dependent). I don't understand why the memory issue if the loaded data are rewritten in each step. Is there any way how to get through this (except the option to process each folder separately = manually)? Thx.

path2load = 'mypath2folders';
path2save = 'mypath2savedfiles';
folders = dir(path2load);
idx = [folders.isdir];
folders = {folders(idx == 1).name};
for i = 1:length(folders)
   files = dir([path2load,'\',folders{i},'\*.png']);
   L = length(files);
   data = zeros(L,2);
   for j = 1:L
      I = single(imread([path2load,'\',folders{i},'\',files(j).name]));
      data(j,1) = files(j).datenum;
      data(j,2) = sum(I(:));
   end
   save([path2save,'\',folders{i},'.mat'],'data')
end

[Uladzimir]

Hi. I'm not sure, maybe try "clear data" in every i cicle.

[Cedric]

Which version of MATLAB are you using and on which OS? There is a priori no reason for this code to consume a lot of memory, as variable I will be a few MB large the whole time, and variable J will not be larger than 1000 * 2 * 4 = 8KB (the conversion to single should not even be necessary). Try to look what happens with the task manager "Performance" panel, and see if the memory usage increases steadily or if something happens at one point. Another thing that you can do is to run the loops once,

 for i = 1:1 %length(folders)
    ...
    for j = 1 : 1 %L

and execute WHOS to see which variables are present and their size in memory. Then execute the internal loop twice, run WHOS again, etc, to see if a variable is abnormally large.

As a side note, you can use FULLFILE instead of creating paths manually by concatenation, i.e.

 [path2load,'\',folders{i},'\',files(j).name]

can be replaced by

 fullfile(path2load, folders{i}, files(j).name)