Parfor problem with large structure

Hey Guys,

I am trying to analyze data about large movies that are ~120,000 frames long. Below is the error I am getting, followed by some of my code and a brief explanation of what I've tried so far to get around the issue.

The error: Warning: Error caught during construction of remote parfor code. The parfor construct will now be run locally rather than on the remote matlabpool. The most likely cause of this is an inability to send over input arguments because of a serialization error. The error report from the caught error is: Error using distcompserialize Error during serialization Error in distcomp.remoteparfor (line 36) obj.SerializedInitData = distcompMakeByteBufferHandle(distcompserialize(varargin)); Error in parallel_function (line 437) P = distcomp.remoteparfor(W, @make_channel, parfor_C); Error in mousehead_4bt_NJ (line 158) parfor j = iFiles %j = 1:numframes > In parallel_function at 450 In mousehead_4bt_NJ at 158

And here's my code up to the beginning of a parfor loop:

numChunks = 4;
framesPerChunk = ceil(length(STATS)/numChunks);
count = 1;
STATS_FULL = STATS;
for g = 1:numChunks
    if g == 1
        STATS = STATS_FULL(1:framesPerChunk);
    else
        start = framesPerChunk*(count-1)+1;
        STATS = STATS_FULL(start:start+framesPerChunk);
        count = count+1;
    end

pool_sz = matlabpool('size');
nBlocks = ceil(numframes/pool_sz);
for iBlock = 1:nBlocks
    iFiles = intersect((iBlock-1)*pool_sz + [1:pool_sz], 1:numframes);

      parfor j = iFiles %j = 1:numframes 
.
.
.

So, apparently you run into this issue when feeding large arrays/structures/etc. into parfor loops. A coworker first suggested to break my data into chunks equal to the number of cores I am pooling. In other words, instead of pafor looping through j = 1:120000 or something like that, I loop through my STATS structure (see above), break it into chunks of 12, and then feed that into the parfor loop, which didn't help. On top of that I tried breaking the large STATS struct into chunks, where each chunk is 1/4 the size of the whole thing, and feeding them in one at a time, which also did not work.

I'm sure there are some things about how parfor loops work that I am not understanding so far. I know that this code will work using a for loop. Any suggestions would be very much appreciated. Thanks!

[Edric Ellis]

Which version of MATLAB are you using? There was a restriction on the amount of data that could be sent into a PARFOR loop, but that was removed in R2013a.

[Pavlov]

I am using 2012b...

[Edric Ellis]

Are you able to upgrade to R2013a or later? If so, that would probably be the simplest thing. Otherwise, some sort of chunking would be required - it's not quite clear from the code you posted what changes you'd need to make to make that work correctly.