Loading mat files within parfor takes long time

Hi,

I am writing the code below to load/process 2627 files under parfor with the size of each file is 100MB, makes it total of 272GB. It is taking very long time to process. I actually use a progress bar for parfor, not shown in this simplified code, and the progress bar even hasn't started yet. I think it is still doing some overhead work. When I use for-loop, I think it is not slower than parfor. Can someone please give me a suggestion? My computer: 40 cores and RAM of 383GB

Merry Xmas,

clear all;
folder_mat_file = '.\Mat_files_full\';
mat_filename_valid = dir([folder_mat_file '*.mat']);
no_mat_files = numel(mat_filename_valid);
if isempty(gcp) 
     parpool;
end 
poolobj  = gcp;
for n=1:no_mat_files
    filename_cell{n} = [folder_mat_file mat_filename_valid(n).name];
end
addAttachedFiles(poolobj,filename_cell)
parfor n=1:no_mat_files
    % data_struct is temporary variable
    data_struct = parload([folder_mat_file mat_filename_valid(n).name]);
    %% Here we do processing
end
%% Below is the function parload use to load file within parfor
function x = parload(filename)
    x = load(filename);
end

[Walter Roberson]

See https://www.mathworks.com/matlabcentral/answers/436854-how-can-i-load-a-mat-file-faster#comment_653337 and https://www.mathworks.com/matlabcentral/answers/425954-2-usb-port-synchronization-problem#answer_348132

[Dave Lee]

Hi Walter,

Can you please elaborate more on your idea? The time for loading one file of mine is 1.8 second.

I think addAttachedFiles is the reason of long time of running. I am attaching a large number of large files. With saving implemented similarly as parsave, I don't have to use addAttachedFiles so it was fine.

[Walter Roberson]

There should be no need to addAttachedFiles of all of the files. Provided that you are passing in the full file name, you should be be to load(). You should not even need to put the load() inside a function

thisdir = pwd();
folder_mat_file = fullfile(thisdir, 'Mat_files_full');
mat_filename_valid = dir( fullfile(folder_mat_file, '*.mat') );
no_mat_files = numel(mat_filename_valid);
filename_cell = fullfile(folder_mat_file, {mat_filename_valid.name});

if isempty(gcp) 
     parpool;
end 
poolobj  = gcp;
parfor n=1:no_mat_files
    % data_struct is temporary variable
    data_struct = load(filename_cell{n});
    %% Here we do processing
end

You will probably get some controller contention for loading the files, so do not expect speedup proportional to the number of workers. I suspect that the performance will get worse after around four cores -- unless, that is, that the "here we do processing" takes longer than loading the files. Depending on the processing you are doing, you might find it worthwhile to parpool a fraction of the available cores, but configure the pool to give you several cores per worker; see e.g., https://www.mathworks.com/matlabcentral/answers/395333-using-parfor-on-cluster-changing-number-of-cores-per-worker#answer_315541

[Dave Lee]

Awesome, yes, I can use load directly without addAttachedFiles. I don't recall why I had to do so and haven't re-tried those options but some running-errors with some types of implementation and the fact that I had to write parsave led me to the implementation in the original code.

Thanks!

[Walter Roberson]

You would want to save in a separate function if you are using save without specifying which variables to save, which is not the most recommended form.

[Dave Lee]

I am not sure if I understand you, using save within parfor as below?

[Walter Roberson]

Ah, they used to permit it, but it would give a transparency error if you did not name all of the variables.

Continue on with your parsave routine then.