Using a Job Manager :: Programming Distributed Jobs (Parallel Computing Toolbox™)

Learn more about Parallel Computing Toolbox

Using a Job Manager

On this page…
Creating and Running Jobs with a Job Manager Sharing Code Managing Objects in the Job Manager

On this page…

Creating and Running Jobs with a Job Manager

For jobs that are more complex or require more control than the functionality offered by dfeval, you have to program all the steps for creating and running of the job.

This section details the steps of a typical programming session with Parallel Computing Toolbox software using a MathWorks job manager:

Note that the objects that the client session uses to interact with the job manager are only references to data that is actually contained in the job manager process, not in the client session. After jobs and tasks are created, you can close your client session and restart it, and your job is still stored in the job manager. You can find existing jobs using the findJob function or the Jobs property of the job manager object.

Find a Job Manager

You use the findResource function to identify available job managers and to create an object representing a job manager in your local MATLAB session.

To find a specific job manager, use parameter-value pairs for matching. In this example, MyJobManager is the name of the job manager, while MyJMhost is the hostname of the machine running the job manager lookup service.

jm = findResource('scheduler','type','jobmanager', ...
                    'Name','MyJobManager','LookupURL','MyJMhost');
get(jm)
          Configuration: ''
                   Name: 'MyJobManager'
               Hostname: 'bonanza'
            HostAddress: {'123.123.123.123'}
                   Type: 'jobmanager'
          ClusterOsType: 'pc'
                   Jobs: [0x1 double]
                  State: 'running'
               UserData: []
            ClusterSize: 2
    NumberOfBusyWorkers: 0
            BusyWorkers: [0x1 double]
    NumberOfIdleWorkers: 2
            IdleWorkers: [2x1 distcomp.worker]

If your network supports multicast, you can omit property values to search on, and findResource returns all available job managers.

all_managers = findResource('scheduler','type','jobmanager')

You can then examine the properties of each job manager to identify which one you want to use.

for i = 1:length(all_managers)
  get(all_managers(i))
end

When you have identified the job manager you want to use, you can isolate it and create a single object.

jm = all_managers(3)

Create a Job

You create a job with the createJob function. Although you execute this command in the client session, the job is actually created on the job manager.

job1 = createJob(jm)

This statement creates a job on the job manager jm, and creates the job object job1 in the client session. Use get to see the properties of this job object.

get(job1)
             Configuration: ''
                      Name: 'job_3'
                        ID: 3
                  UserName: 'eng864'
                       Tag: ''
                     State: 'pending'
             RestartWorker: 0
                   Timeout: Inf
    MaximumNumberOfWorkers: 2.1475e+009
    MinimumNumberOfWorkers: 1
                CreateTime: 'Thu Oct 21 19:38:08 EDT 2004'
                SubmitTime: ''
                 StartTime: ''
                FinishTime: ''
                     Tasks: [0x1 double]
          FileDependencies: {0x1 cell}
          PathDependencies: {0x1 cell}
                   JobData: []
                    Parent: [1x1 distcomp.jobmanager]
                  UserData: []
                 QueuedFcn: []
                RunningFcn: []
               FinishedFcn: []

Note that the job's State property is pending. This means the job has not been queued for running yet, so you can now add tasks to it.

The job manager's Jobs property is now a 1-by-1 array of distcomp.job objects, indicating the existence of your job.

get(jm)
          Configuration: ''
                   Name: 'MyJobManager'
               Hostname: 'bonanza'
            HostAddress: {'123.123.123.123'}
                   Type: 'jobmanager'
          ClusterOsType: 'pc'
                   Jobs: [1x1 distcomp.job]
                  State: 'running'
               UserData: []
            ClusterSize: 2
    NumberOfBusyWorkers: 0
            BusyWorkers: [0x1 double]
    NumberOfIdleWorkers: 2
            IdleWorkers: [2x1 distcomp.worker]

You can transfer files to the worker by using the FileDependencies property of the job object. For details, see the FileDependencies reference page and Sharing Code.

Create Tasks

After you have created your job, you can create tasks for the job using the createTask function. Tasks define the functions to be evaluated by the workers during the running of the job. Often, the tasks of a job are all identical. In this example, each task will generate a 3-by-3 matrix of random numbers.

createTask(job1, @rand, 1, {3,3});
createTask(job1, @rand, 1, {3,3});
createTask(job1, @rand, 1, {3,3});
createTask(job1, @rand, 1, {3,3});
createTask(job1, @rand, 1, {3,3});

The Tasks property of job1 is now a 5-by-1 matrix of task objects.

get(job1,'Tasks')
ans =
    distcomp.task: 5-by-1

Alternatively, you can create the five tasks with one call to createTask by providing a cell array of five cell arrays defining the input arguments to each task.

T = createTask(job1, @rand, 1, {{3,3} {3,3} {3,3} {3,3} {3,3}});

In this case, T is a 5-by-1 matrix of task objects.

Submit a Job to the Job Queue

To run your job and have its tasks evaluated, you submit the job to the job queue with the submit function.

submit(job1)

The job manager distributes the tasks of job1 to its registered workers for evaluation.

Retrieve the Job's Results

The results of each task's evaluation are stored in that task object's OutputArguments property as a cell array. Use the function getAllOutputArguments to retrieve the results from all the tasks in the job.

results = getAllOutputArguments(job1);

Display the results from each task.

results{1:5}

    0.9501    0.4860    0.4565
    0.2311    0.8913    0.0185
    0.6068    0.7621    0.8214

    0.4447    0.9218    0.4057
    0.6154    0.7382    0.9355
    0.7919    0.1763    0.9169

    0.4103    0.3529    0.1389
    0.8936    0.8132    0.2028
    0.0579    0.0099    0.1987

    0.6038    0.0153    0.9318
    0.2722    0.7468    0.4660
    0.1988    0.4451    0.4186

    0.8462    0.6721    0.6813
    0.5252    0.8381    0.3795
    0.2026    0.0196    0.8318

Sharing Code

Because the tasks of a job are evaluated on different machines, each machine must have access to all the files needed to evaluate its tasks. The basic mechanisms for sharing code are explained in the following sections:

Directly Accessing Files

If the workers all have access to the same drives on the network, they can access needed files that reside on these shared resources. This is the preferred method for sharing data, as it minimizes network traffic.

You must define each worker session's path so that it looks for files in the right places. You can define the path

By using the job's PathDependencies property. This is the preferred method for setting the path, because it is specific to the job.
By putting the path command in any of the appropriate startup files for the worker:
- matlabroot\toolbox\local\startup.m
- matlabroot\toolbox\distcomp\user\jobStartup.m
- matlabroot\toolbox\distcomp\user\taskStartup.m
These files can be passed to the worker by the job's FileDependencies or PathDependencies property. Otherwise, the version of each of these files that is used is the one highest on the worker's path.

Access to files among shared resources can depend upon permissions based on the user name. You can set the user name with which the job manager and worker services of MATLAB Distributed Computing Server software run by setting the MDCEUSER value in the mdce_def file before starting the services. For Microsoft Windows operating systems, there is also MDCEPASS for providing the account password for the specified user. For an explanation of service default settings and the mdce_def file, see Defining the Script Defaults in the MATLAB Distributed Computing Server System Administrator's Guide.

Passing Data Between Sessions

A number of properties on task and job objects are designed for passing code or data from client to job manager to worker, and back. This information could include M-code necessary for task evaluation, or the input data for processing or output data resulting from task evaluation. All these properties are described in detail in their own reference pages:

InputArguments — This property of each task contains the input data provided to the task constructor. This data gets passed into the function when the worker performs its evaluation.
OutputArguments — This property of each task contains the results of the function's evaluation.
JobData — This property of the job object contains data that gets sent to every worker that evaluates tasks for that job. This property works efficiently because the data is passed to a worker only once per job, saving time if that worker is evaluating more than one task for the job.
FileDependencies — This property of the job object lists all the directories and files that get zipped and sent to the workers. At the worker, the data is unzipped, and the entries defined in the property are added to the path of the MATLAB worker session.
PathDependencies — This property of the job object provides pathnames that are added to the MATLAB workers' path, reducing the need for data transfers in a shared file system.

There is a default maximum amount of data that can be sent in a single call for setting properties. This limit applies to the OutputArguments property as well as to data passed into a job as input arguments or FileDependencies. If the limit is exceeded, you get an error message. For more information about this data transfer size limit, see Object Data Size Limitations.

Passing M-Code for Startup and Finish

As a session of MATLAB, a worker session executes its startup.m file each time it starts. You can place the startup.m file in any directory on the worker's MATLAB path, such as toolbox/distcomp/user.

Three additional M-files can initialize and clean up a worker session as it begins or completes evaluations of tasks for a job:

jobStartup.m automatically executes on a worker when the worker runs its first task of a job.
taskStartup.m automatically executes on a worker each time the worker begins evaluation of a task.
taskFinish.m automatically executes on a worker each time the worker completes evaluation of a task.

Empty versions of these files are provided in the directory

matlabroot/toolbox/distcomp/user

You can edit these files to include whatever M-code you want the worker to execute at the indicated times.

Alternatively, you can create your own versions of these M-files and pass them to the job as part of the FileDependencies property, or include the path names to their locations in the PathDependencies property.

The worker gives precedence to the versions provided in the FileDependencies property, then to those pointed to in the PathDependencies property. If any of these files is not included in these properties, the worker uses the version of the file in the toolbox/distcomp/user directory of the worker's MATLAB installation.

For further details on these M-files, see the jobStartup, taskStartup, and taskFinish reference pages.

Managing Objects in the Job Manager

Because all the data of jobs and tasks resides in the job manager, these objects continue to exist even if the client session that created them has ended. The following sections describe how to access these objects and how to permanently remove them:

What Happens When the Client Session Ends

When you close the client session of Parallel Computing Toolbox software, all of the objects in the workspace are cleared. However, the objects in MATLAB Distributed Computing Server software remain in place. Job objects and task objects reside on the job manager. Local objects in the client session can refer to job managers, jobs, tasks, and workers. When the client session ends, only these local reference objects are lost, not the actual objects in the engine.

Therefore, if you have submitted your job to the job queue for execution, you can quit your client session of MATLAB, and the job will be executed by the job manager. The job manager maintains its job and task objects. You can retrieve the job results later in another client session.

Recovering Objects

A client session of Parallel Computing Toolbox software can access any of the objects in MATLAB Distributed Computing Server software, whether the current client session or another client session created these objects.

You create job manager and worker objects in the client session by using the findResource function. These client objects refer to sessions running in the engine.

jm = findResource('scheduler','type','jobmanager', ...
             'Name','Job_Mgr_123','LookupURL','JobMgrHost')

If your network supports multicast, you can find all available job managers by omitting any specific property information.

jm_set = findResource('scheduler','type','jobmanager')

The array jm_set contains all the job managers accessible from the client session. You can index through this array to determine which job manager is of interest to you.

jm = jm_set(2)

When you have access to the job manager by the object jm, you can create objects that reference all those objects contained in that job manager. All the jobs contained in the job manager are accessible in its Jobs property, which is an array of job objects.

all_jobs = get(jm,'Jobs')

You can index through the array all_jobs to locate a specific job.

Alternatively, you can use the findJob function to search in a job manager for particular job identified by any of its properties, such as its State.

finished_jobs = findJob(jm,'State','finished')

This command returns an array of job objects that reference all finished jobs on the job manager jm.

Resetting Callback Properties

When restarting a client session, you lose the settings of any callback properties (for example, the FinishedFcn property) on jobs or tasks. These properties are commonly used to get notifications in the client session of state changes in their objects. When you create objects in a new client session that reference existing jobs or tasks, you must reset these callback properties if you intend to use them.

Permanently Removing Objects

Jobs in the job manager continue to exist even after they are finished, and after the job manager is stopped and restarted. The ways to permanently remove jobs from the job manager are explained in the following sections:

Destroying Selected Objects. From the command line in the MATLAB client session, you can call the destroy function for any job or task object. If you destroy a job, you destroy all tasks contained in that job.

For example, find and destroy all finished jobs in your job manager that belong to the user joep.

jm = findResource('scheduler','type','jobmanager', ...
           'Name','MyJobManager','LookupURL','JobMgrHost')
finished_jobs = findJob(jm,'State','finished','UserName','joep')
destroy(finished_jobs)
clear finished_jobs

The destroy function permanently removes these jobs from the job manager. The clear function removes the object references from the local MATLAB workspace.

Starting a Job Manager from a Clean State. When a job manager starts, by default it starts so that it resumes its former session with all jobs intact. Alternatively, a job manager can start from a clean state with all its former history deleted. Starting from a clean state permanently removes all job and task data from the job manager of the specified name on a particular host.

As a network administration feature, the -clean flag of the job manager startup script is described in Starting in a Clean State in the MATLAB Distributed Computing Server System Administrator's Guide.

Using a Local Scheduler

Using a Fully Supported Third-Party Scheduler

Recommended Products

Get MATLAB trial software

Includes the most popular MATLAB recorded presentations with Q&A sessions led by MATLAB experts.

Get the Interactive Kit