MATLAB Answers

parfor - Attempt to serialize data which is too large

17 views (last 30 days)
Germán
Germán on 2 Apr 2012
Commented: Edric Ellis on 8 Nov 2013
I'm working on a large data set that is is stored as a cell array X, where each cell contains an array of about 2MB, totaling 1GB. I'm running it in parallel on 50 cores that are each on separate nodes. I get the error "Attempt to serialize data which is too large" when I try to run the whole array . When I run a small array X I don't get the error. I check on the nodes and there's plenty of memory left, it seems the bottleneck is data transmission
I'm using the following code
Y1= cell(1,N1);
Y2 = cell(1,N1);
parfor i = 1:N1
XX = X{i};
Np = size(XX,1);
Y1_temp = zeros(400, Np-1);
Y2_temp = zeros(400, Np-1);
for j= 1:(Np-1)
Y1_temp(:,j) = someFunction;
Y2_temp(:,j) = someOtherFunction;
end
Y1{i} = Y1_temp;
Y2{i} = Y2_temp;
end
I don't understand why this is, has anyone ever got this error? I couln't find anything online. I also tried writing it as an spmd block where I operate on the local worker data but I get the same error. Any help will be very appreciated.
Regards Germán

Accepted Answer

Edric Ellis
Edric Ellis on 8 Nov 2013
The 2GB PARFOR transmission limitation was removed from Parallel Computing Toolbox in R2013a.

More Answers (2)

Konrad Malkowski
Konrad Malkowski on 2 Apr 2012
There is a 2GB transmission limit between client and workers. Could you provide me with additional values for Np, N1, Nt, etc so that I have a better picture of which variables are causing the problem?
  1 Comment
Germán
Germán on 2 Apr 2012
sure, the values are usually Np=100000, N1=Nt=512.
My understanding is that only the sliced portions of X, Y1, Y2 would be transmitted in each iteration, is that correct? If so the the size of a single cell in those variables is never above 2 MB.
Thanks for your help!

Sign in to comment.


James
James on 4 Nov 2013
I have the same problem. I think it is a common problem, but I couldn't find a solution for this. If it is the 2GB transmission limit, then how can we get around this then?
  2 Comments
Edric Ellis
Edric Ellis on 8 Nov 2013
The 2GB PARFOR transmission limitation was removed from Parallel Computing Toolbox in R2013a.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!