Run scripts in parallel, one core each

Dear all:

I’m looking for a way to run scripts simultaneously, one core each, on my local matlabpool. This is similar to the question “Run scripts in parallel on multiple workers (distributed job)” asked by “Igor Braverman” on 31 Oct 2012 but he did not get an answer to his question.

Say I have a script0.m that generates large outputs. Then script1.m, script2.m and script3.m are going to accept script0.m’s outputs as input, independently of one another, and perform their own functions. These scripts do not lend themselves well to parallelism, so I would just rather execute them in parallel.

Igor Braverman suggested using batch, say:

j1=batch(‘script1’);
j2=batch(‘script2’);
j3=batch(‘script3’);

with each batch using one core each but he found that MATLAB wants to wait for the first batch to finish before starting the next one, and I am finding the same problem. What’s the best way to go about this problem?

Thank you, cpicke

[Edric Ellis]

This should work and run those scripts in parallel. Just after you've issued the 3 batch commands, what is the output of:

disp([j1, j2, j3])

and

disp(j3.Parent)

[cpicke]

Thanks for the comment! From my observation of Windows Task Manager, it is seen that the scripts are indeed running in parallel albeit at a much slower rate than I would expect. The output is

 1x3 Job array:

         ID           Type        State       FinishTime             Username  Tasks
       -----------------------------------------------------------------------------
    1    47    independent      running                            My_Username    1
    2    48    independent      running                            My_Username    1
    3    49    independent       queued                            My_Username    1

and

   Local Cluster

    Properties: 

                          Profile: local
                         Modified: false
                             Host: My_Hostname
                       NumWorkers: 12

               JobStorageLocation: C:\Users\My_Username\AppData\Roaming\MathWorks\MATLAB\local_cluster_jobs\R2015a
 RequiresMathWorksHostedLicensing: false

    Associated Jobs: 

                   Number Pending: 1
                    Number Queued: 9
                   Number Running: 37
                  Number Finished: 0

I think it's taking about the same amount of time as the parfor approach below.

Thank you, cpicke

Answer 1

[Walter Roberson]

Either spmd or parfor should work for that provided that you use functions instead of scripts.

funs = {@script1, @script2, @script3}
parfor K = 1 : length(funs)
  outputs{K} = funs{K}(inputs);
end

[cpicke]

Thanks for the suggestion! Based on this, I tried the code

s={'opt0','opt1','opt2'};
parfor K=1:length(s)
   My_Function(s{K});
end

In reality, s would contain a few hundred cells, but I am just trying this for verification. When run sequentially (simply changing parfor to for), it takes under 45s in total. I don't know how long parfor takes [edit: parfor takes 730s].

The My_Function is completely self-contained; it reads numbers from many text files; processes them and then draws them to a graph.

Also, I am able to verify from Windows Task Manager that three cores are in use in parallel (as expected). I am not able to explain why using parfor is so much slower than for in this instance?

[Walter Roberson]

When you use "for" then if you have calculations with sufficiently large matrices, the calculations might be handed over to the BLAS or LAPACK libraries, which are highly optimized and multithreaded. The calculations might thus use all of your cores. However, when you use parfor, again those optimized libraries might be called, but they would be restricted to one core. Running in explicit parallel does not necessarily mean faster.

Also, parfor workers do not have access to the graphics hardware, so they would need to do all of their drawing in software.

Doing I/O on multiple large files in parallel can be slower than sequentially due to contention for the drive heads and drive caches. However, if the files are smaller, then typically it can be more efficient to read multiple files in multiple threads and return the data for processing.

It is possible that your process might benefit from using parfeval() to run processes that do the I/O and return ready data, with your master thread grabbing the next available set of data and doing the processing sequentially on the sets of data. Also, depending on what you are doing, a mapreduce approach might work for you.