How can I save matrices in par-for-loop?
5 views (last 30 days)
Show older comments
Wenjun Zhang
on 23 Nov 2019
Commented: Wenjun Zhang
on 25 Nov 2019
Hi all,
I'm using MATLAB's par-for-loop to process some experimental data, which is large in size. The result of each iteration is a matrix, but the sizes of the matrices generated by different iterations are different. I'm using a cell to collect the data, and yet it turns out to be inefficient. What I've implemented now is as follows,
load('dataset.mat', 'data');
load('matsize.mat', 'matsize');
data = parallel.pool.Constant(data);
matsize = parallel.pool.Constant(matsize);
N = 24;
results = cell(N,1);
parfor L = 0:N-1
cursize = matsize.Value(L+1);
matrix(1:cursize, 1:cursize) = calculateSomething(data.Value, L);
% the matrix is very large, can be up to size of 2^N, whose sizes are differnt for each L
results{L+1} = matrix(1:cursize, 1:cursize);
end
save('results.mat', 'results');
% MATLAB version: 2019a
I use ticBytes and tocBytes to test the data passing process and find that massive data is transferred from workers to client. Is there any way to avoid this and allow data to be directly written in the same block of memory as pre-allocated, and thus improve the efficiency of the code?
Thanks a lot!
0 Comments
Accepted Answer
Walter Roberson
on 23 Nov 2019
No, the memory lives in different processes. There is no Mathworks support for shared memory.
There is the FEX contribution https://www.mathworks.com/matlabcentral/fileexchange/28572-sharedmatrix that can take advantage of operating system facilities for shared memory.
The closest official support would possibly be https://www.mathworks.com/help/matlab/ref/memmapfile.html memmapfile() -- that is you would memory map the same file on worker and client. Note however that memory mapping a file usually implies writing to disc. Do you just happen to have enough RAM to create a RAMDISK filesystem to share? Or at least an SSD that will not run out of write cycles for a reasonable duration ?
More Answers (0)
See Also
Categories
Find more on Parallel for-Loops (parfor) in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!