Thread Subject:

Safe file mutex without race condition

Dear Newsgroup!

Is there a method to create a safe file mutex in Matlab?

My problem:
  1. Several Matlab sessions are working from different computers on the same file pool on a network drive.
  2. All programs add a short comment to a single log file.
I want to avoid conflicts between the sessions, e.g. let a session wait until another has finished processing the data in a folder or writing its line to the log file.

Tried solution:
  ComputerName = 'PC0815'; % dummy here
  ProcessID = GetProcessIdFromOS; % dummy here
  LockFile = fullfile(FolderName, '.lock');
  if exist(LockFile, 'file') == 0 % Lock the folder
     % *** race condition here
     FID = fopen(LockFile, 'wb');
     % ... check valid FID
     fprintf(FID, '%s %s %s\n', datestr(now, 0), ComputerName, ProcessID);
     fclose(FID);
  else % Folder is locked
     ...
  end

But exactly at *** there is a race condition: another session can open the same file for writing also.
If 2 Matlab sessions are writing a line in the same file, the file contains both lines, but the order seems to depend on which session *closes* the file at first, if the line length is less than 1023 characters. Therefore I could check, if the first line of the file equals the identifier of the current Matlab session after closing the file. But I'm really unhappy to check the contents of a file, which may be open for writing by another (or even several other) program(s).

The time between EXIST and FOPEN is tiny, but it still exists and it is not worth to spend time in programming an *almost* mutual exclusion method...

The easiest method would be a FOPEN permission which does:
  create file for writing, fail if existing already
This is *not* FOPEN(r+)! The Windows API allows the wanted behaviour with CreateFile(GENERIC_WRITE, CREATE_NEW), but it would be nicer to keep this independent from the platform.
I do not have the parallel programming toolbox - does it contain a secure file mutex?

Any ideas? Thanks, Jan

On Thu, 15 Apr 2010 10:10:22 -0300, Jan Simon
<matlab.THIS_YEAR@nminussimon.de> wrote:

> Dear Newsgroup!
>
> Is there a method to create a safe file mutex in Matlab?
>
> My problem:
> 1. Several Matlab sessions are working from different computers on the
> same file pool on a network drive.
> 2. All programs add a short comment to a single log file.
> I want to avoid conflicts between the sessions, e.g. let a session wait
> until another has finished processing the data in a folder or writing
> its line to the log file.
>
> Tried solution:
> ComputerName = 'PC0815'; % dummy here
> ProcessID = GetProcessIdFromOS; % dummy here
> LockFile = fullfile(FolderName, '.lock');
> if exist(LockFile, 'file') == 0 % Lock the folder
> % *** race condition here
> FID = fopen(LockFile, 'wb');
> % ... check valid FID
> fprintf(FID, '%s %s %s\n', datestr(now, 0), ComputerName,
> ProcessID);
> fclose(FID);
> else % Folder is locked
> ...
> end
>
> But exactly at *** there is a race condition: another session can open
> the same file for writing also.
> If 2 Matlab sessions are writing a line in the same file, the file
> contains both lines, but the order seems to depend on which session
> *closes* the file at first, if the line length is less than 1023
> characters. Therefore I could check, if the first line of the file
> equals the identifier of the current Matlab session after closing the
> file. But I'm really unhappy to check the contents of a file, which may
> be open for writing by another (or even several other) program(s).
>
> The time between EXIST and FOPEN is tiny, but it still exists and it is
> not worth to spend time in programming an *almost* mutual exclusion
> method...
>
> The easiest method would be a FOPEN permission which does:
> create file for writing, fail if existing already
> This is *not* FOPEN(r+)! The Windows API allows the wanted behaviour
> with CreateFile(GENERIC_WRITE, CREATE_NEW), but it would be nicer to
> keep this independent from the platform.
> I do not have the parallel programming toolbox - does it contain a
> secure file mutex?
>
> Any ideas? Thanks, Jan

http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileLock.html ?

Dear Ashish!

> http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileLock.html ?

Good idea!
Unfortunately I have be program backward compatible with Matlab 6.5, which uses Java 1.3, which does not include the nio/channels (or at least I cannot find them).

Thanks, Jan

Jan Simon wrote:
> Dear Ashish!
>
>> http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileLock.html ?
>
> Good idea!

Your original posting specified,

 > Is there a method to create a safe file mutex in Matlab?

 > My problem:
 > 1. Several Matlab sessions are working from different computers on
 > the same file pool on a network drive.

If you read the System Dependencies of the above URL, you will find that those
Java locks are only guaranteed at the single virtual machine level, not even
for multiple threads within the same virtual machine, and not for multiple
processes within the same machine, let alone different processes operating on
different machines with a shared filesystem.

If your shared filesystem is based upon NFS (any version) or SMB for unix-type
systems, or is running on any unix-type system (that is currently supported by
Mathworks) other than Trusted AIX, then the warnings about "advisory locks"
and "extreme care" for network file systems apply: in such cases, a networked
Mutex might "usually" work, but is not guaranteed, even if you are using lockd
or are using NFSv3 with integrated lockd: in particular lockd (or its
integrated equivalent) break down when nodes with locks are rebooted. And if
you aren't using lockd (or its integrated equivalent) then networked NFS locks
are not robust at all.

The POSIX function open() with O_CREAT|O_EXCL option is defined to be atomic
and "shall fail if the file exists" -- but NFSv2 even with lockd assistance
cannot guarantee these POSIX semantics without race conditions. NFSv3 with its
integrated lockd are _intended_ to follow these semantics, but as noted above,
can break with reboots (or network problems.)


I have read that real mandatory networked locking is available under XP (might
need SP2) and later MS operating systems (but I wouldn't guarantee it for
their "home" editions!), but I have never investigated that. Based upon what I
did chase down about XP and POSIX, I would be uncomfortable in relying upon
real networked locking with XP Home even with SP2.

Another poor-man's solution: try using the fileattrib function to test & set a specific file attribute (for example: archive or execute). It's not guarantied to be "thread-safe", but it should work in practice unless you have extremely close race conditions.

Dear Yair!

> Another poor-man's solution: try using the fileattrib function to test & set a specific file attribute (for example: archive or execute). It's not guarantied to be "thread-safe", but it should work in practice unless you have extremely close race conditions.

This is the first poor-man's solution from you, that I've seen.
But you are right: FILEATTRIB is better than EXIST('file') + FOPEN. Thanks!

A horrible side-effect of FILEATTRIB in Matlab 6.5 is the speed: it takes 0.2 sec even for accessing a file on the local harddrive.

I'll ask TMW for implementing a thread & network safe file mutex. I think, my problem should be quite common.

Jan

Jan Simon wrote:

> I'll ask TMW for implementing a thread & network safe file mutex. I
> think, my problem should be quite common.

I've only heard of it being successfully implemented on Token Ring networked
based filesystems, such as the old RFS (Apollo, 1983 time frame.) Even then I
don't know how they handled systems going down.

SGI's CXFS filesystem _claimed_ to have solved the problem for SANs (circa
1994), but we used to run CXFS here and I can assure you that it didn't solve
it :(

Some of the solutions I have heard of over time were more robust than others,
but they all had problems with network partitioning. The systems I've heard of
that "self-healed" after network partitioning required journaled updates (not
a problem in itself), and required that programs be "checkpointed" as of the
beginning of the first update whose complete distribution could not be
_proved_ -- it wasn't that the programs couldn't continue to run, but you had
to be able to "roll back" the execution of all shared accessors to the same
point so that you could resume the shared accesses from where you left off
(since the result of running shared sometimes *must* be different than the
result of running all the updates in sequence. Think, for example, bank
deposits and withdrawls.

Dear Walter!

> Some of the solutions I have heard of over time were more robust than others,
> but they all had problems with network partitioning.

It sounds like I should stop working on a file system and use an ACID proof database.

Thanks, Jan

Jan Simon wrote:
> Dear Walter!

>> Some of the solutions I have heard of over time were more robust than
>> others, but they all had problems with network partitioning.

> It sounds like I should stop working on a file system and use an ACID
> proof database.

Yes, if you have multiple networked updaters to a single file and the
updaters have not been assigned specific offsets in the file (to avoid
overwriting each other), then you _are_ better off using a client/server
arrangement such as a database that uses a transaction file. Multiple
servers and fail-over is still an issue, but the reason to pay the big
bucks for database systems is that they have worked out the details.

Safe multi-client updates to a shared resource is indeed a common
request that sounds simple, but it is still open to a lot of rummination
by dining philosophers.

> Is there a method to create a safe file mutex in Matlab?

See this workaround:
atomically creating a file lock in MATLAB (file mutex)
http://stackoverflow.com/questions/3451343/automically-writing-a-file-in-matlab/3452143#3452143

David Portabella wrote:
>> Is there a method to create a safe file mutex in Matlab?
>
> See this workaround:
> atomically creating a file lock in MATLAB (file mutex)
> http://stackoverflow.com/questions/3451343/automically-writing-a-file-in-matlab/3452143#3452143
>

That's not safe either.

I have done this before, and had a similar problem. The solution that worked for me (which is probably not the best, but did fix my race condition) was to write a temporary file and then delete it when I was done writing to the log file. If the temporary file existed, then my script would know that another instance of Matlab was currently writing to the log, and that it should pause for a moment and try again.

Again, this is kind of a hackish solution, but it worked.

Sean

Dear Sean,

> The solution that worked for me (which is probably not the best, but did fix my race condition) was to write a temporary file and then delete it when I was done writing to the log file. If the temporary file existed, then my script would know that another instance of Matlab was currently writing to the log, and that it should pause for a moment and try again.

The problem of this method is, as described above:
Two instances test at the same time, if the temporary file is existing. Then both instances decide, that there is none and create it. This *is* a racing condition.

My Professor told me:
1. A program is a deterministic algorithm, which stops after a finite time.
2. Write programs, if you want to use a computer.

Although from the view point of the universe, each evaluation in a computer is deterministic, for practical usage a racing condition is not. If you can proove, that the frequency of collisions is smaller than the inverse of the age of the universe (or the age of computers), I'd be satisfied.

Kind regards, Jan