Thread Subject: Parallel computing toolbox: help with parfor!

Hi everyone,
I have recently started using the parallel computing toolbox to take full advantage of my machine's 4 processors in my computations.

My problem involves running simulations for different frequencies (25 in total). The iterations are independent from each other but I need to call a fortran executable to calculate what I need. I use parfor to loop over the different frequencies passing them to a matlab function which:
1) writes the input file for the fortran routine containing the specific frequency
2) calls the fortran executable
3) uses the movefile command to change the name of the fortran output file, which is what I'm ultimately interested in.

Here is the code that I wrote for that:
matlabpool local 4
parfor j=1:25
     callbellhop(freq(j),zmax,zmax1,ndz,ndr,rmax,rotl);
 end
matlabpool close

function callbellhop(freq,zmax,zmax1,ndz,ndr,rmax,rotl)
write_envfl2(freq,zmax,zmax1,ndz,ndr,rmax,rotl); % writes the file AUTEC.env
! bellhop.exe < AUTEC.env
movefile('SHDFIL',(['SHDFIL' num2str(freq)]))

When I first attempted to run it, it seemed everything was working fine. All the processors were being used, and the output files were being created for each frequency. However, when the iteration reached j" I got an error:

??? Error using ==> parallel_function at 594
Error in ==> callbellhop at 4
mv: cannot stat `/home/.../Trans1/SHDFIL': No such file or directory


Error in ==> bw_248bdive2_input at 99
    parfor j=1:25

I repeated it more than once and I get the same problem. Does anyone have any clues why it would stop before it finished? Am I overseeing any issues regarding calling the fortran executable and renaming it's output inside a parfor loop?
Thank you in advance for any input you could give me!

Cheers,
Elizabeth

"Elizabeth Kusel" <elizabeth.kusel@oregonstate.com> wrote in message <gtnsts$no9$1@fred.mathworks.com>...
> Hi everyone,
> I have recently started using the parallel computing toolbox to take full advantage of my machine's 4 processors in my computations.
>
> My problem involves running simulations for different frequencies (25 in total). The iterations are independent from each other but I need to call a fortran executable to calculate what I need. I use parfor to loop over the different frequencies passing them to a matlab function which:
> 1) writes the input file for the fortran routine containing the specific frequency
> 2) calls the fortran executable
> 3) uses the movefile command to change the name of the fortran output file, which is what I'm ultimately interested in.
>
> Here is the code that I wrote for that:
> matlabpool local 4
> parfor j=1:25
> callbellhop(freq(j),zmax,zmax1,ndz,ndr,rmax,rotl);
> end
> matlabpool close
>
> function callbellhop(freq,zmax,zmax1,ndz,ndr,rmax,rotl)
> write_envfl2(freq,zmax,zmax1,ndz,ndr,rmax,rotl); % writes the file AUTEC.env
> ! bellhop.exe < AUTEC.env
> movefile('SHDFIL',(['SHDFIL' num2str(freq)]))
>
> When I first attempted to run it, it seemed everything was working fine. All the processors were being used, and the output files were being created for each frequency. However, when the iteration reached j" I got an error:
>
> ??? Error using ==> parallel_function at 594
> Error in ==> callbellhop at 4
> mv: cannot stat `/home/.../Trans1/SHDFIL': No such file or directory
>
>
> Error in ==> bw_248bdive2_input at 99
> parfor j=1:25
>
> I repeated it more than once and I get the same problem. Does anyone have any clues why it would stop before it finished? Am I overseeing any issues regarding calling the fortran executable and renaming it's output inside a parfor loop?
> Thank you in advance for any input you could give me!
>
> Cheers,
> Elizabeth

Elizabeth,

I think the problem is that your fortran routine in each iteration is writing out to the same file "SHDFIL". Since PARFOR loops are running in parallel, there are a lot of overwriting of the same file and different loops grabbing results from other loops. You should find a way to have your fortran routine write out to a different file name, or make sure that other iterations cannot touch the file from the current iteration.

Jiro,
Thanks for your reply. I just figured that out!

I'm trying not to change the fortran routine if I can, since I don't have admin privileges to recompile the code myself. Also the code is part of a bigger fortran toolbox and compiling is not an easy process...
May be I could try splitting the matlab calculations into 4 different temporary directories?

Back to thinking of what and how to do it!




"Jiro Doke" <jiro.doke@mathworks.com> wrote in message <gtnve9$6sl$1@fred.mathworks.com>...
> "Elizabeth Kusel" <elizabeth.kusel@oregonstate.com> wrote in message <gtnsts$no9$1@fred.mathworks.com>...
> > Hi everyone,
> > I have recently started using the parallel computing toolbox to take full advantage of my machine's 4 processors in my computations.
> >
> > My problem involves running simulations for different frequencies (25 in total). The iterations are independent from each other but I need to call a fortran executable to calculate what I need. I use parfor to loop over the different frequencies passing them to a matlab function which:
> > 1) writes the input file for the fortran routine containing the specific frequency
> > 2) calls the fortran executable
> > 3) uses the movefile command to change the name of the fortran output file, which is what I'm ultimately interested in.
> >
> > Here is the code that I wrote for that:
> > matlabpool local 4
> > parfor j=1:25
> > callbellhop(freq(j),zmax,zmax1,ndz,ndr,rmax,rotl);
> > end
> > matlabpool close
> >
> > function callbellhop(freq,zmax,zmax1,ndz,ndr,rmax,rotl)
> > write_envfl2(freq,zmax,zmax1,ndz,ndr,rmax,rotl); % writes the file AUTEC.env
> > ! bellhop.exe < AUTEC.env
> > movefile('SHDFIL',(['SHDFIL' num2str(freq)]))
> >
> > When I first attempted to run it, it seemed everything was working fine. All the processors were being used, and the output files were being created for each frequency. However, when the iteration reached j" I got an error:
> >
> > ??? Error using ==> parallel_function at 594
> > Error in ==> callbellhop at 4
> > mv: cannot stat `/home/.../Trans1/SHDFIL': No such file or directory
> >
> >
> > Error in ==> bw_248bdive2_input at 99
> > parfor j=1:25
> >
> > I repeated it more than once and I get the same problem. Does anyone have any clues why it would stop before it finished? Am I overseeing any issues regarding calling the fortran executable and renaming it's output inside a parfor loop?
> > Thank you in advance for any input you could give me!
> >
> > Cheers,
> > Elizabeth
>
> Elizabeth,
>
> I think the problem is that your fortran routine in each iteration is writing out to the same file "SHDFIL". Since PARFOR loops are running in parallel, there are a lot of overwriting of the same file and different loops grabbing results from other loops. You should find a way to have your fortran routine write out to a different file name, or make sure that other iterations cannot touch the file from the current iteration.

"Elizabeth Kusel" <elizabeth.kusel@oregonstate.com> writes:

> [...]
> I'm trying not to change the fortran routine if I can, since I don't have
> admin privileges to recompile the code myself. Also the code is part of a
> bigger fortran toolbox and compiling is not an easy process... May be I could
> try splitting the matlab calculations into 4 different temporary directories?
>
> Back to thinking of what and how to do it!

One thing you need to be aware of is that MATLABPOOL attempts to synchronise the
path and current directory between client and workers - usually this is very
convenient, but it will get in the way for you. Therefore, to make this work,
you need to be somewhat careful. Something like this should work:

parfor ii=1:N
  % Concoct a temp dir for this worker that remains constant
  mytmpdir = fullfile( tempdir, sprintf( 'mytmp.%d', system_dependent( 'getpid' ) ) );
  if ~exist( mytmpdir, 'dir' )
      mkdir( mytmpdir );
  end
  previousDir = cd( mytmpdir );

  % Call your code here...

  cd( previousDir );
end

Cheers,

Edric.

Edric,
Thank you so much for your suggestion! I had thought of doing that but didn't quite know how to translate into matlab! I was already on my way to talk to the sys admin to change the fortran code when I saw your reply :) The code is now working and the results are as they should be (that is, the output files are not being overwritten).

Cheers,
Elizabeth



Edric M Ellis <eellis@mathworks.com> wrote in message <ytwprenpthg.fsf@uk-eellis-deb4-64.mathworks.co.uk>...
> "Elizabeth Kusel" <elizabeth.kusel@oregonstate.com> writes:
>
> > [...]
> > I'm trying not to change the fortran routine if I can, since I don't have
> > admin privileges to recompile the code myself. Also the code is part of a
> > bigger fortran toolbox and compiling is not an easy process... May be I could
> > try splitting the matlab calculations into 4 different temporary directories?
> >
> > Back to thinking of what and how to do it!
>
> One thing you need to be aware of is that MATLABPOOL attempts to synchronise the
> path and current directory between client and workers - usually this is very
> convenient, but it will get in the way for you. Therefore, to make this work,
> you need to be somewhat careful. Something like this should work:
>
> parfor ii=1:N
> % Concoct a temp dir for this worker that remains constant
> mytmpdir = fullfile( tempdir, sprintf( 'mytmp.%d', system_dependent( 'getpid' ) ) );
> if ~exist( mytmpdir, 'dir' )
> mkdir( mytmpdir );
> end
> previousDir = cd( mytmpdir );
>
> % Call your code here...
>
> cd( previousDir );
> end
>
> Cheers,
>
> Edric.