MATLAB Central Newsreader - using new mpich2 library problems

using new mpich2 library problems

Brock — Fri, 06 Nov 2009 19:53:02 -0500

Hello,
We are trying to have matlab use a build of mpich2 that does not use smpd so we may use OSU's mpiexec in our PBS batch environment.

I built version 1.0.8p1 of mpich2 to match 1.0.x that comes with 2009a, I then made mpiLibConf.m to point to '/nobackup/brockp/matlab/mpich/lib/libmpich.so', and added its location to MATLABPATH,

I then made a configuration using the mpiexec type, and put in the custom mpiexec location, and argument --comm mpich2-pmi,

Problem is no matter what when I run
matlabpool open configname 4

The library that is dlopened()' is the one that comes with matlab (lsof | grep mpich), not my new one.
I am confused what needs to be done, or if there is anyway to make matlab give me more information about how it is picking what libmpich.so to load.

Any input would be helpful,

Re: using new mpich2 library problems

Edric M Ellis — Mon, 09 Nov 2009 09:03:37 -0500

"Brock " <brockp@umich.edu> writes:

> Hello, We are trying to have matlab use a build of mpich2 that does not use
> smpd so we may use OSU's mpiexec in our PBS batch environment.
>
> I built version 1.0.8p1 of mpich2 to match 1.0.x that comes with 2009a, I then
> made mpiLibConf.m to point to '/nobackup/brockp/matlab/mpich/lib/libmpich.so',
> and added its location to MATLABPATH,
>
> I then made a configuration using the mpiexec type, and put in the custom
> mpiexec location, and argument --comm mpich2-pmi,
>
> Problem is no matter what when I run matlabpool open configname 4
>
> The library that is dlopened()' is the one that comes with matlab (lsof | grep
> mpich), not my new one. I am confused what needs to be done, or if there is
> anyway to make matlab give me more information about how it is picking what
> libmpich.so to load.

Once you've run "matlabpool open ...", what happens when you run

spmd
which mpiLibConf
[a, b] = mpiLibConf
end

?

How did you add the path to your updated mpiLibConf to MATLAB's path?

Cheers,

Edric.

Re: using new mpich2 library problems

Brock — Tue, 10 Nov 2009 21:30:36 -0500

>
> Once you've run "matlabpool open ...", what happens when you run
When I do:
matlabpool open native-mpiexec 4
It says:
Starting matlabpool using the 'native-mpiexec' configuration ... connected to 1 labs.

Which is odd,
ps aux | grep mpiexec
/home/software/rhel5/mpiexec/0.83/bin/mpiexec --comm mpich2 -n 4 worker -parallel

Which is right, but:
lsof | grep mpich
MATLAB 31757 brockp mem REG 0,26 1353360 404096010 /afs/engin.umich.edu/caen/rhel_5/matlab-2009a/bin/glnxa64/libmpich.so

whcih mpiLibConf
/nobackup/brockp/matlab/mpiLibConf.m

>> spmd
which mpiLibConf
[a,b]=mpiLibConf
end
  1 /nobackup/brockp/matlab/mpiLibConf.m
    a =
    /nobackup/brockp/matlab/mpich/lib/libmpich.so
    b =
         {}

spmd
>
> spmd
> which mpiLibConf
> [a, b] = mpiLibConf
> end
>
> ?
>
> How did you add the path to your updated mpiLibConf to MATLAB's path?

export MATLABPATH=/nobackup/brockp/matlab:$MATLABPATH

>
> Cheers,
>
> Edric.

Re: using new mpich2 library problems

Edric M Ellis — Wed, 11 Nov 2009 08:23:20 -0500

"Brock " <brockp@umich.edu> writes:

> When I do:
> matlabpool open native-mpiexec 4
> It says:
> Starting matlabpool using the 'native-mpiexec' configuration ... connected to 1 labs.
>
> Which is odd,

That's probably to do with the mismatch between the MPICH library you're picking
up and the mpiexec you're using. Typically, if something goes wrong there, each
worker process ends up thinking it's got labindex==1.

>>> spmd
> which mpiLibConf
> [a,b]=mpiLibConf
> end
> 1 /nobackup/brockp/matlab/mpiLibConf.m
> a =
> /nobackup/brockp/matlab/mpich/lib/libmpich.so
> b =
> {}
>

> [...]
>> How did you add the path to your updated mpiLibConf to MATLAB's path?
>
> export MATLABPATH=/nobackup/brockp/matlab:$MATLABPATH
>

Hmm, it does appear that the mpiLibConf.m being picked up by the workers is OK
when you've checked it there. But thinking about it, the worker path gets
modified after pool startup, so it may well not reflect the value that was
used. I'm not 100% sure if your workers will see the changes you've made via
setting the environment variable. Any chance you could try putting mpiLibConf.m
into ~/matlab ?

Cheers,

Edric.

Re: using new mpich2 library problems

Brock — Wed, 11 Nov 2009 16:14:06 -0500

>
> Hmm, it does appear that the mpiLibConf.m being picked up by the workers is OK
> when you've checked it there. But thinking about it, the worker path gets
> modified after pool startup, so it may well not reflect the value that was
> used. I'm not 100% sure if your workers will see the changes you've made via
> setting the environment variable. Any chance you could try putting mpiLibConf.m
> into ~/matlab ?

That did it, things work correctly now, and torque starts matlab workers on the correct remote nodes, under the TM job manager, so nolonger use ssh, startup and shutdown is much cleaner and faster! Also nolonger have to leave a matlab running on a login node, which was the real goal.

Thank you!
Maybe make a note some place in the docs on mpiLibConf ?

>
> Cheers,
>
> Edric.