Function handle as input argument -> conflict with parfeval queue

Hello, after spending some thoughts on the problem I currently have I have decided to turn to you guys.

I have created a simulation in matlab. A single run usually takes 1 to 120 minutes and I use parfeval to start several Jobs so they run in the background. In this way I can scan a large parameter field while simultaneously working on other problems. This works very well - until I change the working directory.

As soon as I leave the directory where the simulation function and the subfunctions (refers to a function that is called inside the simulation function) are, all jobs that are queued up end up with an error. The reason seems to be certain that subfunctions cannot be found. These are the subfunctions that are given to the main function as an input argument (in the form of a function handle).

The problem essentially occurs in this scenario: I have a directory with these four functions:

simulation.m
subfunction1.m
subfunction2.m
subfunction3.m

The main function "simulation" essentially looks like this:

function [ result ] = simulation( input )
  subfunction1(...)
  result = input(...)
end

Input is a function handle. It can either be @subfunction2 or @subfunction3. When I now start more than 8 Jobs (my local cluster has 8 workers) some jobs will end up in the queue. They will run fine until the evaluation has completed. Once a job is finished, the next job is started. If I am then not in directory where subfunction2.m or subfunction3.m are located, the respective function cannot be found. Execution is naturally halted. (Note that subfunction1 is found). The fact that this happens surprises me, because (Quote from documentation):

Once you create a handle for a function, you can invoke the function by means of the handle instead of using the function name. Because the handle contains the absolute path to its function , you can invoke the function from any location that MATLAB is able to reach, as long as the program file for the function still exists at this location

The only direct way to solve this problem would be not to use function handles as an input argument of the main function. This could easily be solved by using an integer as the input variable to specify different cases:

function [ result ] = simulation( input )
  subfunction1(...)
  if input==2
    result = subfunction2(...)
  elseif input==3
    result = subfunction3(...)
  end
end

However, this solution would not be very dynamic. In my actual program I have up to 10 and subfunctions (still increasing). I would like to prevent this solution to prevent going through the code every time I add a new subfunction. I would be glad if you could help me by somehow fixing my initial approach or finding a better one ;) .

Btw: addpath(pwd) before sending the jobs is not a good solution. This would eventually create conflicts if I am working on other problems.

Answer 1

[Edric Ellis]

There are several ways you could work around this.

Perhaps the simplest way is to use batch jobs rather than parfeval. This has the advantage that the batch job running your code does not change working directory when the client does, and the disadvantage that the overhead of running a batch job is higher, and (depending on whether you launch a single batch job corresponding to each parfeval call or a batch job with a 'Pool' argument containing multiple parfeval calls) loses the interactive nature of parfeval.

The next possibility is to pass in the appropriate working directory as an argument when invoking parfeval, and then have each worker cd to that directory before invoking the function handle.

Answer 2

[Fabian Single]

Thank you for the answer! I would like to stick to parfeval because I already created a number of scripts to start and evaluate the jobs that are queued up and running.

So I guess I will try to pass the working directory as an input argument and change it before a calling a subfunction that would cause the crash.

Meanwhile I found out that I can also add 'AdditionalPats' to a cluster in parallel preferences. I cant find a way to do this for an individual job though. This kind of solves the problem, can however lead to conflicts in the future.

Another Solution I just found would be to attach the function files to the parallel pool (add 'AttachedFiles',{'myFun.m'} when creating the pool). But again, there would be possible conflics, e.g. if I run simulations from different versions.