How to create data file and copying data on it for parallel programing in MATLAB?

11 views (last 30 days)
Deeptajyoti Sen
Deeptajyoti Sen on 4 Apr 2022
Commented: Edric Ellis on 5 Apr 2022
I want to run an ODE system by varying a suitable parameter, suppose \alpha. Now I have divided the job into the corresponding worker and created a data file so that for each parameter \alpha, I need to copy a certain measure of the system into the data file. When I googled it and went through some documentation, I realized that using a single file only creates a route to a corrupt file. It is not a way to do that, and I need to create multiple files. I need to ask whether there is any way to create files with some exact name (must be in parallel programing) and scan data from all the files to create another data file that contains all the data.

Answers (1)

Edric Ellis
Edric Ellis on 5 Apr 2022
You are quite right that trying to write to the same file from multiple workers will cause problems. Here's one way that you could get the workers to write to unique files, and then get the client to amalgamate those files:
alphaValues = 0:.01:1;
numValues = numel(alphaValues);
parfor ii = 1:numValues
% generate a unique file name to write to
fname = sprintf('data_%d', ii);
% Run a simulation for this value of alpha
data = runMySimulation(alphaValues(ii));
% save the data - note workaround for SAVE inside PARFOR
iSaveData(fname, data)
end
% At this point, files data_1 .. data_(numValues) exist. We can read them
% at the client
for ii = 1:numValues
fname = sprintf('data_%d', ii);
s = load(fname);
allData{ii} = s.data;
end
% Save all of the data in one big data file
save('oneBigDataFile', 'allData')
% Workaround for SAVE inside PARFOR
function iSaveData(fname, data)
save(fname, 'data')
end
  4 Comments
Edric Ellis
Edric Ellis on 5 Apr 2022
There's a lot going on there, and I haven't worked through it all in detail, but I'd definitely recommend at the very least setting up the arrays that you want to save ahead of time, rather than calling iSaveData multiple times in inner nested for loops. You should be able to end up with a single call to iSaveData per parfor loop iteration. Your code will probably be quite a bit faster when it doesn't have to call save so many times.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!