When/how should I swap batch jobs for independent jobs?

1 view (last 30 days)
I'm wanting to send simulations to a remote cluster with 32 workers. However, I'm running into problems sending batch jobs in that when I retrieve my data, I can't bring in more than 2GB of data. I've been trying to come up with a way to send individual jobs to each worker and retrieve each worker's data separately. Here's an example of what I've been doing:
% set up variables
for parameters
batch
Here's what I'm thinking about doing:
% set up variables
parfor the_simulations
createJob
createTask main_sim_data
createTask secondary_data_based_on_sim_data
submit job
Is this an appropriate change, to go from a batch job to independent jobs using the createJob & createTask parallel functions? When does it make more sense to do batch instead of independent jobs, and vice versa?

Answers (0)

Categories

Find more on MATLAB Parallel Server in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!