Why does "parfor" freeze when working with very large files?

I have a MATLAB script with a "parfor" loop with the following structure:

1. A struct array variable is loaded into MATLAB with large amounts of data.
2. "parfor" loop iterates from 1 to N, where N is the length of the array. The data is sent from MATLAB to the workers (communication overhead).
3. Some function is performed on the array element inside the "parfor" loop, the result is saved into a pre-allocated cell array, using data queue to send data from worker to the client.
4. "parfor" loop finishes.
5. Cell array is converted into a struct array.

When my "parfor" reaches around 75%, it is stuck for about 2 minutes without progress. Then it continues to work as normal. I am using around 1 GB of data, which after the "parfor" is about 13 GB in size. My RAM is 32 GB, so it should not be an issue.

If I rerun my script, it freezes at the same time. Why is "parfor" hanging at that point?

Answer 1

[MathWorks Support Team]

The pause issue may be due to the workers sending big messages (e.g. 1 GB+) to the MATLAB client.

The recommendation for this workload is to continue to use "parforOptions" to set the interval size to a fixed number. A fixed range 100 may behave better than a setting of "auto". We believe that you can see better performance if the range is started at 100 and you iteratively test larger ranges until performance stops increasing.

Your code should look approximately like the following code.

opts = parforOptions(p,'RangePartitionMethod','fixed','SubrangeSize',100);
parfor (ii=1:1:t_n_1,opts)
    CellArray{ii} = OurFunction();
    send(q, ii);
end