MATLAB Answers

How can I save matrices in par-for-loop?

7 views (last 30 days)
Hi all,
I'm using MATLAB's par-for-loop to process some experimental data, which is large in size. The result of each iteration is a matrix, but the sizes of the matrices generated by different iterations are different. I'm using a cell to collect the data, and yet it turns out to be inefficient. What I've implemented now is as follows,
load('dataset.mat', 'data');
load('matsize.mat', 'matsize');
data = parallel.pool.Constant(data);
matsize = parallel.pool.Constant(matsize);
N = 24;
results = cell(N,1);
parfor L = 0:N-1
cursize = matsize.Value(L+1);
matrix(1:cursize, 1:cursize) = calculateSomething(data.Value, L);
% the matrix is very large, can be up to size of 2^N, whose sizes are differnt for each L
results{L+1} = matrix(1:cursize, 1:cursize);
end
save('results.mat', 'results');
% MATLAB version: 2019a
I use ticBytes and tocBytes to test the data passing process and find that massive data is transferred from workers to client. Is there any way to avoid this and allow data to be directly written in the same block of memory as pre-allocated, and thus improve the efficiency of the code?
Thanks a lot!

  0 Comments

Sign in to comment.

Accepted Answer

Walter Roberson
Walter Roberson on 23 Nov 2019
No, the memory lives in different processes. There is no Mathworks support for shared memory.
There is the FEX contribution https://www.mathworks.com/matlabcentral/fileexchange/28572-sharedmatrix that can take advantage of operating system facilities for shared memory.
The closest official support would possibly be https://www.mathworks.com/help/matlab/ref/memmapfile.html memmapfile() -- that is you would memory map the same file on worker and client. Note however that memory mapping a file usually implies writing to disc. Do you just happen to have enough RAM to create a RAMDISK filesystem to share? Or at least an SSD that will not run out of write cycles for a reasonable duration ?

  1 Comment

Wenjun Zhang
Wenjun Zhang on 25 Nov 2019
Thanks a lot. I'm trying FEX. This is the first time I deal with hugh matrices and parallel computation.

Sign in to comment.

More Answers (0)

Sign in to answer this question.

Products


Release

R2019a