memmapfile in parfor loops

7 views (last 30 days)
Lorenzo
Lorenzo on 10 Nov 2011
Commented: Walter Roberson on 10 Feb 2024
Hello, I'm trying to use mem mapped files for input and output of data from the body of a parloop.
If I understand correctly, this seems impossible, as the way memmapfile data is accessed (struct) violates the 'first indexing' rule for sliced variables in parallel loops.
As the memmapfile can be accessed by concurrent processes (as in the demo), is there a workaround to use it in parfor loops?
Thanks. L

Answers (1)

Sarah Wait Zaranek
Sarah Wait Zaranek on 11 Nov 2011
You should be able to use memmapfile in a parfor loop. The trick is to pass m into a function within the parfor loop. You can get around the restriction that way. See simple example below:
function parMemMap
% randData = gallery('uniformdata', [100, 1], 0, 'double');
% fid = fopen('myfile.dat');
% fwrite(fid, randData, 'double');
% fclose(fid);
m = memmapfile('myfile.dat','Format', 'double','Writable',true);
parfor ii=1:length(m.Data)
output(ii,1) = myTestFunc(m,ii);
end
data = m.Data(:);
disp(output)
isequal(output,data)
end
function test=myTestFunc(m,ii)
m.Data(ii) = m.Data(ii)*rand;
test = m.Data(ii);
end
  3 Comments
Keith
Keith on 10 Feb 2024
11/11/11! Suppose there's a need to share data not only inside a parfor but between ML instances? Will a memmapfile, specifiying the desired disk file, in one thread facilitate reading that disk file as written by another(cooperating) thread?
Walter Roberson
Walter Roberson on 10 Feb 2024
Will a memmapfile, specifiying the desired disk file, in one thread facilitate reading that disk file as written by another(cooperating) thread?
No.
However, a memmapfile in one thread could obtain data that you exchange with the client using parallel.pool.DataQueue and your client could then make that data available to another thread using a different set of parallel data queues. It is a somewhat ackward process.

Sign in to comment.

Categories

Find more on MATLAB Parallel Server in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!