Is it possible to identify workers when using parallel computing in optimization routines?

Hi there,

I'm using an optimization routine with the UseParallel functionality. Within my objective function, I am writing to a number of files and evaluating simulation results with the "system" command. For clarity, I am editing a number of external files and then using outside-Matlab simulation software to evaluate those files. Unfortunately, this process can't really be modified. I want to use the optimizer to help determine parameter values best suited for my simulation in a way that is faster than serial optimization.

The problem is that I need to isolate which files the workers operate on so that they aren't conflicting with each others work. For this, I was hoping to use dedicated worker folders much like the top answer here. I'm not sure how to identify the cluster's ProcessID from within the optimization function, though. If I could, I would use the ProcessID to store files for each worker and ensure that their work was not affecting files that other workers are using.

Is there a way to isolate worker file access from within an optimization routine?

Any help or information is appreciated!

Answer 1

[Walter Roberson]

I might have missed something in the code, but it looks to me as if during the initial population creation phase for ga, if UseParallel is set, then parfor will be used.

Other than that, it looks as if the global optimization routines share a common set of utilities, and that the utilities will use parfeval() for parallel evaluations.

Both parfor and parfeval use the same ID mechanism; see https://www.mathworks.com/matlabcentral/answers/30294-identifying-lab-id-within-a-parfor#answer_38822

[HiWave]

I'm finding that the linked answer does not work. Can you provide a clear solution?

[Walter Roberson]

I created some demo code just to test it out. The only difficulty I had was that the first call to the objective function is done outside of parallel control -- it is part of an initialization phase that is checking to see that the function basically works. So do not expect a task ID for the first run.

On my system, this code produces a single warning about no task, and then runs without further warning, producing 4 record_* files.

In production code I would be more efficient about the file operations if I were writing to file.

!rm record_*.txt
options = optimoptions('ga', 'Useparallel', true);
nvars = 1;
A = []; b = [];
Aeq = []; beq = [];
lb = []; ub = [];
nonlcon = [];
[bestx, fval] = ga(@testit, nvars, A, b, Aeq, beq, lb, ub, nonlcon, options);
disp(bestx)
disp(fval);

function y = testit(x)
    t = getCurrentTask();
    if isempty(t)
       id = 1;
       warning('no task')
    else
      id = t.ID;
    end
    filename = sprintf('record_%d.txt', id);
    save(filename, 'x', '-ascii', '-append');
    y = x-sum(factor(id).^2);
end

[Carlos M. Velez S.]

parallel = gcp('nocreate');
if isempty(parallel)
    S.parallel = 'off';
else
    S.parallel = ['on (' num2str(parallel.NumWorkers) ' workers, Cluster: ' parallel.Cluster.Profile ')'];
end