Code covered by the BSD License  

Highlights from
Distributed Batch Job

Be the first to rate this file! 21 Downloads (last 30 days) File Size: 7.6 KB File ID: #44128
image thumbnail

Distributed Batch Job

by

 

29 Oct 2013 (Updated )

Spread batch work over multiple MATLAB instances on multiple networked PCs

| Watch this File

File Information
Description

If you have a for loop which can be written as:

   for a = 1:size(input, 2)
       output(:,a) = func(input(:,a), global_data);
   end

where input is a numeric array and output is a numeric or cell array, then batch_job_distrib() can parallelize the work across multiple worker MATLAB instances on multiple (unlimited) networked worker PCs as follows:

   output = batch_job_distrib(func, input, workers, global_data);

This is a replacement for parfor in this use case, if you don't have the Parallel Computing Toolbox and/or Distributed Computing Server.

There is also an asynchronous mode, which returns immediately, passing back a handle to a function which can load the output data later:

   output = batch_job_distrib(..., '-async'); % Start an asynchronous computation
   ... % Do other stuff here
   output = output(); % Get the results here

The function can always spread the work across multiple MATLABs on the local PC, but the requirements for it to run on OTHER PCs are that:
  - There is an ssh executable on the system path of the local PC.
  - All worker PCs can be ssh'd into non-interactively (i.e. without manually entering a password).
  - MATLAB is on the system path of all worker PCs.
  - Every worker has a valid license for MATLAB and all required toolboxes for the user ssh'ing in.
  - The current directory can be accessed on all worker PCs via the SAME path.
  - The user can write to the current directory from all worker PCs.
  - All the required functions are on the MATLAB paths of every worker PC.
  - The networked filesystem supports file locks (not crucial, but safer).
  - All the worker PCs honour the networked filesystem file locks (again, not crucial, but safer).

Note that the workers need not all run the same operating system, but they must all have working versions of the required functions, including where these are platform-dependent, e.g. mex files.

This function is in beta testing. If you have any bug reports or fixes, please send them to me via the 'Contact Author' link on my author page.

Acknowledgements

Multicore Parallel Processing On Multiple Cores and Batch Job inspired this file.

Required Products MATLAB
MATLAB release MATLAB 8.1 (R2013a)
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Updates
30 Oct 2013

Better error handling and support for operating across different OSs

11 Dec 2013

Add optional progress bar

12 Dec 2013

Bug fixes and improvements. Add keep flag.

13 Dec 2013

Bug fixes. Exits quickly and gracefully now, when an error occurs.

Contact us