Thread Subject: omp_get_num_threads() (again)

The omp_get_num_threads() does not work properly, as well as
similar functions for getting the maximum number of threads etc.

The problem is that omp_get_num_threads() always returns 1,
no matter to what maxNumCompThreads() is set in Matlab.

This was previously reported in this newsgroup:
http://www.mathworks.com/matlabcentral/newsreader/view_thread/157701#416259

The solution given there is to:
>> setenv('OMP_NUM_THREADS','2')
>> maxNumCompThreads(2)

However, this does not work for me in 2008a and 2007b
(x86_64 Linux). The observation by CPU usage is however,
that the loops are parallellized properly.

I would like to use omp_get_num_threads() to default to a
(faster) sequential code when only 1 thread is used.

This is a test listing I used, compiled with Intel C
compiler version 10.

Thanks,
Sebastiaan

#include "mex.h"
#include <omp.h>


void mexFunction(int nlhs, mxArray* plhs[], int nrhs, const
mxArray* prhs[])
{
    mexPrintf("Num threads %d.\n", omp_get_num_threads());
}

Update:

I have tried gcc-4.3.0. Still, omp_get_num_threads() returns
1, but the recognition of the number of threads differ
between icc and gcc.

Using icc-10, I can use maxNumCompThreads(2) to specify the
number of threads, and the mex routine uses them (despite
that omp_get_num_threads() returns 1).

For gcc-4.3.0, things are different. It does not respond to
maxNumCompThreads(2), nor to setenv('OMP_NUM_THREADS','2').
It only uses 1 thread when Matlab is started by default
(OMP_NUM_THREADS not initialized). The only way to specify
the number of threads is to set OMP_NUM_THREADS prior to
starting Matlab:
OMP_NUM_THREADS=2 matlab

I do not think Matlab already officialy supports openMP in
mex files, but is there a way to make this work?

On May 30, 10:14 am, "Sebastiaan "
<s.breedv...@erasmusmc.REMOVE.BOO.BOO.nl> wrote:
> Update:
>
> I have tried gcc-4.3.0. Still, omp_get_num_threads() returns
> 1, but the recognition of the number of threads differ
> between icc and gcc.
>
> Using icc-10, I can use maxNumCompThreads(2) to specify the
> number of threads, and the mex routine uses them (despite
> that omp_get_num_threads() returns 1).
>
> For gcc-4.3.0, things are different. It does not respond to
> maxNumCompThreads(2), nor to setenv('OMP_NUM_THREADS','2').
> It only uses 1 thread when Matlab is started by default
> (OMP_NUM_THREADS not initialized). The only way to specify
> the number of threads is to set OMP_NUM_THREADS prior to
> starting Matlab:
> OMP_NUM_THREADS=2 matlab
>
> I do not think Matlab already officialy supports openMP in
> mex files, but is there a way to make this work?

Are you calling omp_get_num_threads() functions within #pragma omp
parallel block?

rych <rychphd@gmail.com> wrote in message
<c985c966-110c-42a7-8ea1-0b1714508996@f63g2000hsf.googlegroups.com>...

> Are you calling omp_get_num_threads() functions within
#pragma omp
> parallel block?

No, I wasn't... Looking back at the manual,
omp_get_num_threads() only works within a parallel region:

"If this call is made from a serial portion of the program,
or a nested parallel region that is serialized, it will
return 1."

I think I got confused by the plural form (_threads()).
omp_get_max_threads() works outside the #pramga omp region
(I have not idea why it did not work yesterday - must have
overlooked something).


This solves the first problem, but not the difference
between icc and gcc. The code I test with is now:

    mexPrintf("Max threads %d.\n", omp_get_max_threads());
/* omp_set_num_threads((int) 4);*/
    #pragma omp parallel
    {
      mexPrintf("Thread num %d.\n", omp_get_num_threads());
    }

Compiling with icc and running gives:
Max threads 4.
Thread num 4.
Thread num 4.
Thread num 4.
Thread num 4.

However, with gcc:
Max threads 4.
Thread num 1.

I have no clue why. When I compile the code in the shell
(outside Matlab, without mex) and run, it acts the same as icc.

I have checked omp_get_dynamic(), and they are both disabled
in icc and gcc. I have no idea what to look for.

I tried setting the threads manually with
omp_set_num_threads(4), but gcc crashes Matlab (icc works
correct). Same problem with gcc-4.2.4. Top of the stack trace:

Stack Trace:
  [0] libguide.so:omp_set_dynamic~(0x407fbac0,
0x2aab4e4847eb, 0, 0x2aab55900ce0) + 62 bytes
  [1] test.mexa64:mexFunction~(0, 0x2aab55900ce0,
0x127f00000000, 0xffffffff) + 56 bytes
  [2] libmex.so:mexRunMexFile(0, 0x407fc320, 0, 0x407fc260)
+ 75 bytes

which is actually interesting, since the lib is linked
against libgomp (libguide belongs to icc):

$ ldd test.mexa64
        libgomp.so.1 =>
/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.0/libgomp.so.1
(0x00002b6e57565000)

I guess OpenMP in gcc is still young. Any more ideas on this?

Thanks!
Sebastiaan

 
> $ ldd test.mexa64
> libgomp.so.1 =>
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.0/libgomp.so.1
> (0x00002b6e57565000)
>

Ok, got it. Matlab comes with libguide.so, and has all the
omp_* commands. libgomp.so is too new, so Matlab does not
know about it, and dynamically links everything to its
libguide.so.

Removing libguide.so from bin/glnxa64/, putting libgomp.so
there and make a symlink libguide.so->libgomp.so actually
starts Matlab. Compiling with gcc and running even works.

Still, something funny. Setting omp_set_num_threads(4) sets
the number of threads to 4, and works accordingly. However,
omp_get_max_threads() stays 4, even when quitting and
rebooting Matlab sessions, until omp_set_num_threads is used
again. maxNumCompThreads() has no influence on this.

Could off course be that maxNumCompThreads() only works with
libguide.so. I have no idea what filetype
toolbox/matlab/general/private is.

The simplest workaround seems to make a custom
maxNumCompThreads() which calls omp_set_num_threads().

Is it possible to static link libgomp.so to the mexfile,
without statically linking all other libraries? (and without
recompiling gcc with -fPIC).

Thanks,
Sebastiaan

Sebastiaan

I am also trying to get omp working with mex files on RHEL
5. I also have the latest icc available. Could you send
me (or post) a copy of your mexopts file and an example of
a mex file and compile line that you used successfully?

Thanks
Ben


"Sebastiaan " <s.breedveld@erasmusmc.REMOVE.BOO.BOO.nl>
wrote in message <g1pi8r$8dk$1@fred.mathworks.com>...
>
> > $ ldd test.mexa64
> > libgomp.so.1 =>
> > /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.0/libgomp.so.1
> > (0x00002b6e57565000)
> >
>
> Ok, got it. Matlab comes with libguide.so, and has all
the
> omp_* commands. libgomp.so is too new, so Matlab does not
> know about it, and dynamically links everything to its
> libguide.so.
>
> Removing libguide.so from bin/glnxa64/, putting
libgomp.so
> there and make a symlink libguide.so->libgomp.so actually
> starts Matlab. Compiling with gcc and running even works.
>
> Still, something funny. Setting omp_set_num_threads(4)
sets
> the number of threads to 4, and works accordingly.
However,
> omp_get_max_threads() stays 4, even when quitting and
> rebooting Matlab sessions, until omp_set_num_threads is
used
> again. maxNumCompThreads() has no influence on this.
>
> Could off course be that maxNumCompThreads() only works
with
> libguide.so. I have no idea what filetype
> toolbox/matlab/general/private is.
>
> The simplest workaround seems to make a custom
> maxNumCompThreads() which calls omp_set_num_threads().
>
> Is it possible to static link libgomp.so to the mexfile,
> without statically linking all other libraries? (and
without
> recompiling gcc with -fPIC).
>
> Thanks,
> Sebastiaan

"David " <dabenpb@yahoo.com> wrote in message
<g3nlue$nul$1@fred.mathworks.com>...
> Sebastiaan
>
> I am also trying to get omp working with mex files on RHEL
> 5. I also have the latest icc available. Could you send
> me (or post) a copy of your mexopts file and an example of
> a mex file and compile line that you used successfully?
>
> Thanks
> Ben
>

The program:
 
#include "mex.h"
#ifdef _OPENMP
    #include <omp.h>
#else
    #warning "OpenMP not enabled. Use -fopenmp (>gcc-4.2) or
-openmp (icc) for speed enhancements on SMP machines."
#endif

void mexFunction(int nlhs, mxArray* plhs[], int nrhs, const
mxArray* prhs[])
{
    int j;
    
    /* Some basic stuff */
    mexPrintf("Max threads %d.\n", omp_get_max_threads());
    
    #pragma omp parallel
    {
      mexPrintf("Hello from thread num %d.\n",
omp_get_thread_num());
    }
    
    /* OMP loop */
    #pragma omp for schedule(dynamic,10) nowait
    for (j=0; j<100; j++)
        mexPrintf("%d ", j);
    
    mexPrintf("\n");
    
}

Modify these lines in mexopts.sh (add -openmp to CFLAGS):
 CC='icc'
 CFLAGS='-ansi -D_GNU_SOURCE -fexceptions -openmp'

And mex it with:
mex omptest.c -lguide

This works with Maltab 2007b, icc-10.0 and gcc-4.1.2 on the
system. However, using gcc-4.3.0 breaks compilations with
icc, complaining about not able to find limits.h.

For compilation with gcc, you need to add '-fopenmp' to
CFLAGS and -lgomp in the mex command.

It would be nicer to be able to set/append to CC and CFLAGS
from Matlab, without editing mexopts.sh.

Good luck,
Sebastiaan

Thanks. I will give it a try tonight.

Ben

"Sebastiaan " <s.breedveld@erasmusmc.REMOVE.BOO.BOO.nl>
wrote in message <g3oc53$sr1$1@fred.mathworks.com>...
> "David " <dabenpb@yahoo.com> wrote in message
> <g3nlue$nul$1@fred.mathworks.com>...
> > Sebastiaan
> >
> > I am also trying to get omp working with mex files on
RHEL
> > 5. I also have the latest icc available. Could you
send
> > me (or post) a copy of your mexopts file and an
example of
> > a mex file and compile line that you used successfully?
> >
> > Thanks
> > Ben
> >
>
> The program:
>
> #include "mex.h"
> #ifdef _OPENMP
> #include <omp.h>
> #else
> #warning "OpenMP not enabled. Use -fopenmp (>gcc-
4.2) or
> -openmp (icc) for speed enhancements on SMP machines."
> #endif
>
> void mexFunction(int nlhs, mxArray* plhs[], int nrhs,
const
> mxArray* prhs[])
> {
> int j;
>
> /* Some basic stuff */
> mexPrintf("Max threads %d.\n", omp_get_max_threads
());
>
> #pragma omp parallel
> {
> mexPrintf("Hello from thread num %d.\n",
> omp_get_thread_num());
> }
>
> /* OMP loop */
> #pragma omp for schedule(dynamic,10) nowait
> for (j=0; j<100; j++)
> mexPrintf("%d ", j);
>
> mexPrintf("\n");
>
> }
>
> Modify these lines in mexopts.sh (add -openmp to CFLAGS):
> CC='icc'
> CFLAGS='-ansi -D_GNU_SOURCE -fexceptions -openmp'
>
> And mex it with:
> mex omptest.c -lguide
>
> This works with Maltab 2007b, icc-10.0 and gcc-4.1.2 on
the
> system. However, using gcc-4.3.0 breaks compilations with
> icc, complaining about not able to find limits.h.
>
> For compilation with gcc, you need to add '-fopenmp' to
> CFLAGS and -lgomp in the mex command.
>
> It would be nicer to be able to set/append to CC and
CFLAGS
> from Matlab, without editing mexopts.sh.
>
> Good luck,
> Sebastiaan
>

"Sebastiaan " <s.breedveld@erasmusmc.REMOVE.BOO.BOO.nl> wrote in message <g3oc53$sr1$1@fred.mathworks.com>...
> "David " <dabenpb@yahoo.com> wrote in message
> <g3nlue$nul$1@fred.mathworks.com>...
> > Sebastiaan
> >
> > I am also trying to get omp working with mex files on RHEL
> > 5. I also have the latest icc available. Could you send
> > me (or post) a copy of your mexopts file and an example of
> > a mex file and compile line that you used successfully?
> >
> > Thanks
> > Ben
> >
>
> The program:
>
> #include "mex.h"
> #ifdef _OPENMP
> #include <omp.h>
> #else
> #warning "OpenMP not enabled. Use -fopenmp (>gcc-4.2) or
> -openmp (icc) for speed enhancements on SMP machines."
> #endif
>
> void mexFunction(int nlhs, mxArray* plhs[], int nrhs, const
> mxArray* prhs[])
> {
> int j;
>
> /* Some basic stuff */
> mexPrintf("Max threads %d.\n", omp_get_max_threads());
>
> #pragma omp parallel
> {
> mexPrintf("Hello from thread num %d.\n",
> omp_get_thread_num());
> }
>
> /* OMP loop */
> #pragma omp for schedule(dynamic,10) nowait
> for (j=0; j<100; j++)
> mexPrintf("%d ", j);
>
> mexPrintf("\n");
>
> }
>
> Modify these lines in mexopts.sh (add -openmp to CFLAGS):
> CC='icc'
> CFLAGS='-ansi -D_GNU_SOURCE -fexceptions -openmp'
>
> And mex it with:
> mex omptest.c -lguide
>
> This works with Maltab 2007b, icc-10.0 and gcc-4.1.2 on the
> system. However, using gcc-4.3.0 breaks compilations with
> icc, complaining about not able to find limits.h.
>
> For compilation with gcc, you need to add '-fopenmp' to
> CFLAGS and -lgomp in the mex command.
>
> It would be nicer to be able to set/append to CC and CFLAGS
> from Matlab, without editing mexopts.sh.
>
> Good luck,
> Sebastiaan
>

Sebastiaan,

I have been trying to modify MATLAB so it recognizes OpenMP commands as suggested here. However, no luck with that so far, and I am a little confused as far as what I need to do.

I want to use gcc compiler in linux, and I understand that the following should work:
1) removing libguide.so from bin/glnxa64/, putting libgomp.so there and making a symlink libguide.so -> libgomp.so.
2) adding -fexceptions -openmp (and possibly -fPIC?) to compiler command line in mexopts.sh.
3) and/or compiling C/OpenMP code with -L libgomp.so option

If i make modification 1), then MATLAB does not start and gives a runtime error saying that it cannot open libguide.so. If I omit 1) and make 2) and/or 3), then programs do not compile, giving errors for OpenMP comands which are not recognized.

Could you clarify the above and tell me if I did the right thing? I would appreciate any help or a more detailed description of what I need to do to get it to work.

Lukasz

"Lukasz " wrote in message
> I have been trying to modify MATLAB so it recognizes OpenMP commands as suggested here. However, no luck with that so far, and I am a little confused as far as what I need to do.
>
> I want to use gcc compiler in linux, and I understand that the following should work:
> 1) removing libguide.so from bin/glnxa64/, putting libgomp.so there and making a symlink libguide.so -> libgomp.so.
> 2) adding -fexceptions -openmp (and possibly -fPIC?) to compiler command line in mexopts.sh.
> 3) and/or compiling C/OpenMP code with -L libgomp.so option
>
> If i make modification 1), then MATLAB does not start and gives a runtime error saying that it cannot open libguide.so. If I omit 1) and make 2) and/or 3), then programs do not compile, giving errors for OpenMP comands which are not recognized.
>
> Could you clarify the above and tell me if I did the right thing? I would appreciate any help or a more detailed description of what I need to do to get it to work.
>
> Lukasz

It worked in my case with Matlab 2008a. I am not sure if these are typo's, but:
in 1) where did you find libgomp.so? /usr/lib64/libgomp.so ? Make sure it is the same as the one used to compile your programs.
in 2) -fopenmp in stead of -openmp
in 3) -lgomp in stead of -L libgomp.so

I had this working with 2008a and gcc-4.3.0. Are you also using these versions?

"Sebastiaan " <s.breedveld@erasmusmc.REMOVE.BOO.BOO.nl> wrote in message <ga8a3u$i14$1@fred.mathworks.com>...
> "Lukasz " wrote in message
> > I have been trying to modify MATLAB so it recognizes OpenMP commands as suggested here. However, no luck with that so far, and I am a little confused as far as what I need to do.
> >
> > I want to use gcc compiler in linux, and I understand that the following should work:
> > 1) removing libguide.so from bin/glnxa64/, putting libgomp.so there and making a symlink libguide.so -> libgomp.so.
> > 2) adding -fexceptions -openmp (and possibly -fPIC?) to compiler command line in mexopts.sh.
> > 3) and/or compiling C/OpenMP code with -L libgomp.so option
> >
> > If i make modification 1), then MATLAB does not start and gives a runtime error saying that it cannot open libguide.so. If I omit 1) and make 2) and/or 3), then programs do not compile, giving errors for OpenMP comands which are not recognized.
> >
> > Could you clarify the above and tell me if I did the right thing? I would appreciate any help or a more detailed description of what I need to do to get it to work.
> >
> > Lukasz
>
> It worked in my case with Matlab 2008a. I am not sure if these are typo's, but:
> in 1) where did you find libgomp.so? /usr/lib64/libgomp.so ? Make sure it is the same as the one used to compile your programs.
> in 2) -fopenmp in stead of -openmp
> in 3) -lgomp in stead of -L libgomp.so
>
> I had this working with 2008a and gcc-4.3.0. Are you also using these versions?
>

Here is what I did with Matlab2008b...

    cd bin
    mv libguide.so libguide.so.ORIG
    cp /usr/lib64/libgomp.so.1 libgomp.so
    chmod 755 libgomp.so
    ln -s libgomp.so libguide.so


Alas... When calling Matlab I now get:

hpcu: Command? matlab
libmwblas: load error: /uhpc/Applications/u/Mathworks/MatlabR2008bomp/
bin/glnxa64/../../bin/glnxa64/../../bin/glnxa64/mkl.so: undefined
symbol: __kmpc_for_static_fini
terminate called after throwing an instance of 'std::runtime_error'
   what(): /uhpc/Applications/u/Mathworks/MatlabR2008bomp/bin/
glnxa64/../../bin/glnxa64/../../bin/glnxa64/mkl.so: undefined symbol:
__kmpc_for_static_fini
Abort

Could you please help me here??

Adam,

Things seem to have changed between Maltab versions, but the replacement of libguide with libgomp was never more than a hot-fix, because it 'cries' for instability.

I have not paid attention to OpenMP+GCC+Matlab anymore, and used only ICC.

Revisiting this issue learns that it is still not solved in R2009a, so I tend to go back to my old suggestion: making a custom maxNumCompThreads:

#include "mex.h"
#ifdef _OPENMP
    #include <omp.h>
#else
    #warning "OpenMP not enabled. Use -fopenmp (>gcc-4.2) or -openmp (icc) for speed enhancements on SMP machines."
#endif

void mexFunction(int nlhs, mxArray* plhs[], int nrhs, const mxArray* prhs[])
{
    double *res;
    /* Return current NumThreads as with maxNumCompThreads command */
    plhs[0] = mxCreateDoubleMatrix(1,1,mxREAL);
    res = mxGetPr(plhs[0]);
    res[0] = omp_get_max_threads();
    
    if ( (nrhs==1) && (mxIsDouble(prhs[0])) )
        omp_set_num_threads(mxGetPr(prhs[0])[0]);
}

Compile this with (or add -fopenmp -fPIC to mexopts.sh):
mex CC='gcc' CLFAGS='-fopenmp -fPIC' -lgomp maxNumCompThreadsGCC.c

Then edit maxNumCompThreads.m and add this line:
lastn(2) = maxNumCompThreadsGCC(varargin{:});

This makes it able to use Matlab's maxNumCompThreads command for also setting threads for the GCC openMP files (note that the 'automatic' option does not work for this simple maxNumCompThreadsGCC.c).

Compiling the example program I gave before now works and runs in Matlab and respects the number of desired threads.


I only accounted an oddity: when I run the program, sometimes not all threads are used, and sometimes some other text is displayed:
>> multithreadtest
Max threads 8.
Hello from thread num 0.
-script,/opt/matlab/2009aHello from thread num 5.
Hello from thread num 2.
Hello from thread num 1.
Hello from thread num 7.
Hello from thread num 4.
Hello from thread num 6.

Is this because of using GCC, or is there a bug in my program?

Ok, things have changed again in 2009b. maxNumCompThreads works now for gcc as well. The libguide.so has been replaced with libiomp5.so, so multithread-capable MEX files should be linked to this. (You also need this if you are linking to libmwblas or libmwlapack).

Take the program and compile it in Matlab with:

mex CC="gcc" CFLAGS="\$CFLAGS -fopenmp -funroll-all-loops" -v -liomp5

Using maxNumCompThreads will now also change the number of threads in programs compiled with GCC.

However, I still get strange text in between:
>> multithreadtest
Max threads 8.
Hello from thread num 0.
Hello from thread num 3.
Hello from thread num 4.
Hello from thread num 2.
ined -o "multithreadtestHello from thread num 1.
Hello from thread num 5.


Another issue is the removal of maxNumCompThreads in future releases:
"Why remove maxNumCompThreads in future versions?"
http://www.mathworks.com/matlabcentral/newsreader/view_thread/260966