Error when running several Matlab scripts concurrently

Hello,

I have a single-threaded script that when run, works perfectly as expected. However, in a bid to get my work done faster on my multi-core machine, I want to run several instances of the script simultaneously. This works usually, each running on a different core and giving me results.

However, very frequently one of the scripts will simply die with a strange syntax error if there are multiple scripts running. The error I got from the last such incident was:

    [user@host ~/script_dir] ???    end
           |
    Error: Illegal use of reserved keyword "end".

    ??? end                  
        |
    Error: Illegal use of reserved keyword "end".

    [1]-  Done

This is complaining about some part - unfortunately impossible to identify - failing. The line it complains about changes each time, sometimes appearing fragmented (for example, it might say something like unknown var 'ormance' which seems to be a truncation of an actual variable performance). The error, however, isn't the code - it's something to do with the fact there are multiple scripts running at once, as if they are somehow interfering. Note that if a script does succeed, its output is correct.

Any ideas?

The machine is a RHEL5.6 box with 8 cores. Matlab is 7.11.0.584. The script is made up of multiple classes and files.

Thanks!

Answer 1

[Jason Ross]

Are you sharing a directory to store intermediate results, data files, temp files, etc? If you have filenames that are the same in a single directory, this could confuse your application. You might look into using separate (working) directories for each case. If you are putting temp files in /tmp, make sure they are uniquely named per invocation to avoid collisions and confusion.

Are you opening the same file with the multiple instances and trying to modify it? Check any statements that open files and make sure that you are indeed opening any shared files as readonly, and not inadvertently modifying things with multiple processes.

I would suggest that you start creating some log files -- separate ones for each process -- that you write into to get an idea of what's going on when you get the error. I know you mentioned that it's in different places, but perhaps with more logging you could possibly identify some common thing that's getting confused.

[Alexander]

Hi Jason, thanks for your reply.

The scripts do indeed generate output files in the same directory. These files are uniquely named and I see no way that they might be trying to read/write/access the same file. I have no input files, other than the Matlab scripts themselves.

I do actually have some crude logging going on, and it seems the places that the failure occur are pretty random; they don't seem to happen near a file I/O event or particular function call. I don't see any other pattern in there yet.

As far as I'm aware, the only interaction between the processes is the common output file directory and the common input scripts/function/classdefs..

[Jason Ross]

Is the directory local or on a network? There are plenty of cases I've encountered where where latency/filesystem caching has caused odd errors that inspire hair pulling. You could possibly try moving things local as part of a diagnostic.

You might also try changing the permissions on your input scripts/functions/classdef files to read only (or read/execute, as appropriate) and see if something funky is going on where you least expect it to happen.

Good luck, a lot of times these concurrency issues can be very difficult to pin down.

Also, as a shameless plug -- have you considered using Parallel Computing Toolbox? I know there are problems it can't tackle for one reason or another but just wanted to suggest it.