Manual scheduling of parfeval jobs

I have a script which consists of a main loop and subfunctions. In order to speed up the calculation in the main loop, I use parpool('threads') and parfeval to calculate the output of the subfunctions ahead of time, so that when it reaches the right iteration, the subfunctions already completed and I can just retrieve the output. This is working well, however, I have a machine with limited memory, so I could run into out-of-memory kill if I'm not careful.

I would like to be able to manually control when to start a job, is it possible? For example, I set up parpool with 3 workers using maxNumCompThreads, and then create 10 jobs. Job 1-3 will start running and 4-10 will be in queue. The moment job 1 finishes, job 4 will start. At this moment, let's say I'm not ready to read the output of job 1, I will then need the memory to store the output of job 1-4 potentially job 5 too, which can become overwhelming for my machine. So I'd like to be able to start job 4 only after I do fetchOutputs(job1) and delete(job1).

Thanks.

Answer 1

[Chern Hui Lee]

I ended up doing this and it works so far. The idea is:

1. Start parpool('threads')
2. create only 3 jobs
3. wait for job1 to finish
4. retrieve data
5. delete job1
6. create job4
7. wait for job2 to finish (usually it's already done)
8. retrieve data
9. delete job2
10. create job5

etc etc..

Please advise if there is a smarter method. I'm looking into afterEach, but it might need a lot of work to make it work with my existing script structure.