Best practice for passing large number of constants to function that runs in parfor

Hi everybody,

I have the following setup and I've been asking myself for a while how to handle this "the best way":

- I set/calculate a large number of constants sequentially and define a high-dimensional grid

- I then want to use those constants and the grid in a parfor loop

- Each parfor worker optimizes a function stored in an external .m file for some point on the grid

- That function uses most constants and the grid

Now what would be the best practice to use that large number of variables within the function file on each of the workers? None of the variables change within the function or the parfor loop. My first hunch would be to use globals (that's what they are for, I guess), but MATLAB doesn't allow me to do that. The two solutions I could think of is to pass all variables to the function as parameters or to all put them in a struct and pass the struct. Both of those solutions work - However, both make my code much less readable and annoy me everytime I have to work with the code. Are there any better ways of doing that?

On a sidenote/feedback, I really can't understand why MATLAB doesn't allow me to access global variables on the workers without changing them. I am a big boy and don't need MATLAB restricting my possibilities in unneccessary ways.

Thank you for suggestions,

Tobias

[Matt J]

On a sidenote/feedback, I really can't understand why MATLAB doesn't allow me to access global variables on the workers without changing them.

How does MATLAB know you are not changing them? And which of the parallel versions of the variables should it tie the version on the client to?

In any case, maybe the answer is that the MATLAB developers don't expect anyone to use global variables as a way of passing extra data to functions. It's a practice that has been discouraged for years, even in serial MATLAB programming, see Passing Extra Parameters. Although, when using the Parallel Computing Toolbox, I do recommend that you explicitly broadcast your variables to the workers first to avoid hazards like in this thread,

http://www.mathworks.com/matlabcentral/answers/115254-parfor-behavior-sensitive-to-comments

[Tobias]

If globals are to have any use, it is exactly sharing constants between functions. What else do you suggest they should be used for if not this?

[Matt J]

I suggest they be avoided altogether - as does the MathWorks link I gave you - in favor of the newer methods mentioned in that link. I don't know anyone who uses globals anymore.

Answer 1

[Matt J]

Either method you describe would work, and is preferable to using global variables. I don't know what you find annoying about bundling your variables into a struct. It's always good to collect related variables together in a common object, whether a matrix for homogenous data or a struct/cell for heterogeneous data, and that's true regardless of whether it's to serve parfor or something else. It just makes the data easier to carry around and manipulate simultaneously.

If the annoyance is from the need to access your variables using dot-indexing syntax, you could unpack them into individual variables inside the workspace of your mfunction, using this FEX file for instance,

http://www.mathworks.com/matlabcentral/fileexchange/26216-structure-fields-to-variables

[Tobias]

Yes, the reason I don't like the struct solution is the dot-indexing. If I give the struct an informative name it makes my code a lot longer and less readable. The variables are not really related to each other except that they are constant in the parfor loop. I would prefer to just declare which variables I want to be shared between the workspaces and be done with it.

Packing and unpacking variables is much worse than using globals in potential for mistakes. You need to decide at which point in the code you should pack variables into the struct and when to unpack them. Every time you add a new variable to the struct, you need to change the synchronization routine and think about the timing again. In addition I would have to add like 30 lines of useless code to each of my files.

[Matt J]

You need to decide at which point in the code you should pack variables into the struct and when to unpack them.

Why does the timing matter, exactly? Why not just pack the variables into the struct at the very end of the function that created them -- the point when you would need to transport them elsewhere -- and unpack them all at the beginning of the function that uses them as input? It can't be an issue of memory consumption, as far as I can see. You've already allocated memory to the variables as globals in the parent function that created them.

As for number of lines, the structvars command that I linked you to can spread multiple packing/unpacking statements across a single line so that the total number of lines is reduced, e.g.,

    >> s.a=1; s.b=2; s.c=3; s.d=4;
    >> structvars(2,s)

    ans =

    a = s.a;     c = s.c;     
    b = s.b;     d = s.d;

Answer 2

[Sean de Wolski]

What about using WorkerObjWrapper to broadcast them to the workers once, and then reusing them?