Is it possible to call parfeval from within a parfor loop

I have a time consuming analysis task I'm trying to speed up, on a beefy workstation with lots of cores.

Currently, code constists of an outer parfor loop, which let's say is loading an input file, then calling several functions which each do some independant processing, and output a file.

Obviously, the easiest approach is just to parfor the outer loop -- but since there are often only a small # of input files to analyze (say, 3), this doesn't make good use of the cores.

Alternatively, I could parfor the time consuming functions, but since there are small # of these functions, it also ends up using a small # of cores. (The functions could be non-communicating, fire and forget, since they can just output files, results are not used within the loop ).

Ideally, it seems like I'd want to be able to have an outer parfor, and then have each parfor kick off some separate tasks or parfeval's for the work going on inside the loop. However, in Matlab 2021, "Workers cannot submit jobs to a local cluster." , and "Workers cannot execute parfeval or parfevalOnAll."

Is there any way to run parfeval from within parfor? Or any sort of equivalent construct using the lower level job/task controls?

Answer 1

[Walter Roberson]

No, that is not possible.

See https://www.mathworks.com/matlabcentral/answers/1904941-dual-parfor-from-different-scripts#answer_1161776

What you can do:

Create a backgroundPool

loop parfeval() reading the files, recording the futures.

afterEach a function that fetches the file content and parfeval() a series of function handles on the file content, one for each of the independent tasks to be done for each file.

You would end up with a bunch of parfeval() futures that might be working on any combination of task and file; as soon as one finishes the next would start, so you should keep your CPUs busy.