Thread Subject: parallel processing trouble

Hello all,

I am having a bit of trouble with parallel processing on windows machine. I am running parallel jobs on both unix and windows machine. on the unix machine the parallel job is submitted to a cluster (and it works just fine), whereas on my windows comp I use the local scheduler. In my parallel job I have to access a number of programs and generate/move/delete files and folders. I have made sure that all files and folders are indexed. For the majority of the time, this process works ok however every so often I receive an error that says one of the directories cannot be deleted. There is no conflicts with the folder because once the parallel job shuts down I can delete it manually from the command window. Has anybody encountered an error like this before..? and if so has anyone happend to find a solution?

Thanks,
Rob

"rcarrese " <carrese7@hotmail.com> writes:

> I am having a bit of trouble with parallel processing on windows machine. I am
> running parallel jobs on both unix and windows machine. on the unix machine the
> parallel job is submitted to a cluster (and it works just fine), whereas on my
> windows comp I use the local scheduler. In my parallel job I have to access a
> number of programs and generate/move/delete files and folders. I have made sure
> that all files and folders are indexed. For the majority of the time, this
> process works ok however every so often I receive an error that says one of the
> directories cannot be deleted. There is no conflicts with the folder because
> once the parallel job shuts down I can delete it manually from the command
> window. Has anybody encountered an error like this before..? and if so has
> anyone happend to find a solution?

Windows is much more likely than UNIX to refuse to delete a file or directory if
some other process is accessing it.

Within MATLAB, I think if one of your workers has a given directory as its PWD
or on the path then one of the other workers may well not be able to delete
it. If that is the case, it's not surprising that when the workers exit you are
able to delete things afterwards.

One tool that I use to debug stuff like this is the excellent "Process Explorer"
from sysinternals, at:

http://technet.microsoft.com/en-us/sysinternals/default.aspx

Enable the "handle view", and you can see a list of files and directories that a
process has open.

Cheers,

Edric.

Edric M Ellis <eellis@mathworks.com> wrote in message <ytwmy29d6v8.fsf@uk-eellis-deb5-64.mathworks.co.uk>...
> "rcarrese " <carrese7@hotmail.com> writes:
>
> > I am having a bit of trouble with parallel processing on windows machine. I am
> > running parallel jobs on both unix and windows machine. on the unix machine the
> > parallel job is submitted to a cluster (and it works just fine), whereas on my
> > windows comp I use the local scheduler. In my parallel job I have to access a
> > number of programs and generate/move/delete files and folders. I have made sure
> > that all files and folders are indexed. For the majority of the time, this
> > process works ok however every so often I receive an error that says one of the
> > directories cannot be deleted. There is no conflicts with the folder because
> > once the parallel job shuts down I can delete it manually from the command
> > window. Has anybody encountered an error like this before..? and if so has
> > anyone happend to find a solution?
>
> Windows is much more likely than UNIX to refuse to delete a file or directory if
> some other process is accessing it.
>
> Within MATLAB, I think if one of your workers has a given directory as its PWD
> or on the path then one of the other workers may well not be able to delete
> it. If that is the case, it's not surprising that when the workers exit you are
> able to delete things afterwards.
>
> One tool that I use to debug stuff like this is the excellent "Process Explorer"
> from sysinternals, at:
>
> http://technet.microsoft.com/en-us/sysinternals/default.aspx
>
> Enable the "handle view", and you can see a list of files and directories that a
> process has open.
>
> Cheers,
>
> Edric.

I second Edric's suggestion regarding the excellent Process Explorer. Actually, many SysInternals utilities are excellent, but ProcExplorer is indispensable. I use it all the time to kill zombie or wild processes, set process priorities, see process memory/CPU/IO usage etc.

Another free utility you can use to debug this problem is Unlocker - http://ccollomb.free.fr/unlocker/ - this utility alerts you with a popup message whenever there is a file-locking conflict, tells you which process is locking, and enables you to ignore or kill or erase etc.

One final idea: maybe the folder cannot be deleted because it is not empty?

Yair Altman
http://UndocumentedMatlab.com
 

Hello,

Thank you both for your suggestions. I will take a look at the process explorer to see if I can diagnose what is preventing Matlab from deleting the folder.
 
> One final idea: maybe the folder cannot be deleted because it is not empty?
 
Each time Matlab is asked to delete the folder it is empty and I reinforce that by using the 's' options i.e. rmdir(sprintf('folder%s',folder_name),'s')

Cheers,
Rob

Hey Rob, I'm having that exact same issue (with my windows machine using local configuration)...did you ever find out why that was happening?

Thanks in advance!

Best,
Jared

"rcarrese " <carrese7@hotmail.com> wrote in message <hemqjr$rbu$1@fred.mathworks.com>...
> Hello,
>
> Thank you both for your suggestions. I will take a look at the process explorer to see if I can diagnose what is preventing Matlab from deleting the folder.
>
> > One final idea: maybe the folder cannot be deleted because it is not empty?
>
> Each time Matlab is asked to delete the folder it is empty and I reinforce that by using the 's' options i.e. rmdir(sprintf('folder%s',folder_name),'s')
>
> Cheers,
> Rob

"Jared Goodner" <jared.goodner@gmail.com> wrote in message <idmffo$acc$1@fred.mathworks.com>...
> Hey Rob, I'm having that exact same issue (with my windows machine using local configuration)...did you ever find out why that was happening?
>
> Thanks in advance!
>
> Best,
> Jared

Hi Jared,

No, I was never able to find the problem... That Process Explorer program is really good, but unfortunately whilst you are running a script it can be difficult to monitor. Sometimes, when I run scripts that require deleteing and creating/copying/re-naming files I just allow for Windows (and any background programs) to catch up by putting in a >>pause(number of seconds) command right before I delete the entire folder. By putting in the pause command I have been able to overcome the problem 90% of the time - although there are some occasions where there is still an error.

Cheers,
Rob