Communicating between tasks and submitting client in parallel jobs

2 views (last 30 days)
emarch
emarch on 10 Dec 2018
Answered: Prashanth Ramesh on 14 Dec 2018
I've got a long job I've broken into several tasks to run on MS HPC and I am trying to figure out a way to send progress back to the submitting client computer. I'm using createJob/createTask to run the job. Right now I'm polling for the individual task states and this works, but I'd like to get some insight into where each task is in it's progression (e.g., 60% complete). Some ideas I'm condsidering or have tried include:
Using the task objects UserData property, but this appears to only be updated on the matlab worker (for example, one can print it to the Diary just fine from within the function the task is running).
The DataQueue object looks promising but it looks like it's only available when using parallel pools.
I've thought of writing to the diary and then on the calling/client machine having something read these every couple of minutes across the network and present a progress bar for each task. My principle hesitation being this seems a little kludgy.
Any other ideas or suggestions?
Thanks

Answers (1)

Prashanth Ramesh
Prashanth Ramesh on 14 Dec 2018
While I admit there does not exist a clean documented way of achieving your workflow, I believe implementing parallel pool with a single worker may be a workaround for this issue. Please note that MATLAB
does not add additional overheads between CPU and parallel pool, if the worker size is 1. The only caveat however is that your existing code for the jobFunction should be compatible with "parfeval" .
parpool(1); % Only 1 worker required
Q= parallel.pool.DataQueue;
Q.afterEach(@updatWaitBar);
future = parfeval(@jobFunction, 0, queue);
You could also consider using a similar workflow as mentioned in the below link, although it is implemented for "parfor", the same can be achieved for jobs:

Products


Release

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!