Thread Subject:

spmd overhead and sharing small amounts of data

I think my processing time is being chewed up by spmd overhead. I have a simulation that looks like this:

for x = 1:N
   spmd
      D = expensiveFunction(D)
   end
   <gather small bits from workers>
   <fast simple function>
   spmd
     D = expensiveFunction2(D)
   end

   <repeat the above basic idea 3-4 times>

end

Watching my processors, they are seldom above 20% utilization, so it makes me think that I'm suffering from going in and out of spmd and maybe gathering even small pieces of data is a problem. Any alternatives? For one, is there a way to gather directly worker to worker without exiting spmd and bringing it all to the master? The data is small and functions cheap, so maybe I'd be ahead to let everyone do the same computations just to keep things flowing.

Thanks.

"Chuck37 " <chuck3737@yahooremovethis.com> writes:

> I think my processing time is being chewed up by spmd overhead. I
> have a simulation that looks like this:
>
> for x = 1:N
> spmd
> D = expensiveFunction(D)
> end
> <gather small bits from workers>
> <fast simple function>
> spmd
> D = expensiveFunction2(D)
> end
>
> <repeat the above basic idea 3-4 times>
>
> end
>
> Watching my processors, they are seldom above 20% utilization, so it
> makes me think that I'm suffering from going in and out of spmd and
> maybe gathering even small pieces of data is a problem.

What size MATLABPOOL did you open? One thing to note: MATLAB doesn't
consider hyperthreaded cores when choosing how many processes to make
the default for the local scheduler - so even if all the workers were
completely busy, you might not see more than 50% utilisation. (That's
what happens here on my system - the OS thinks it has 12 cores, but
really it has 6 hyperthreaded cores, so the default is for MATLAB to
launch 6 local workers).

> Any alternatives? For one, is there a way to gather directly worker
> to worker without exiting spmd and bringing it all to the master?

You can use functions like either labBroadcast to send data from one
worker to all the others, and gcat or gop to perform 'reduction'
operations. Very briefly, here's how they work

% assume matlabpool size 4
spmd
  data = rand();

  % broadcast from lab 1, x1 gets the value from 'data' on lab 1.
  x1 = labBroadcast(1, data);

  % concatenation - each lab gets x2 = [1 2 3 4];
  x2 = gcat(labindex);

  % general reduction - in this case, each lab gets
  % x3 = 1 + 2 + 3 + 4
  x3 = gop(@plus, labindex);
end

> The data is small and functions cheap, so maybe I'd be ahead to let
> everyone do the same computations just to keep things flowing.

That sounds like a very good idea.

Cheers,

Edric.

Thanks, that will help I think.

I'm using a dual hex core machine. It has 12 real processors but shows as 24 on Linux due to hyperthreading. I'm using 10 workers so I can have them to myself while other people use the others for small stuff. I do see them go to 100% sometimes, just not during most of my loop going in and out of spmd.

By the way, is it possible to plot while still in the spmd? I like to put up a plot as I go to monitor progress.

Edric M Ellis <eellis@mathworks.com> wrote in message <ytw623xcglg.fsf@uk-eellis0l.dhcp.mathworks.com>...

>
> What size MATLABPOOL did you open? One thing to note: MATLAB doesn't
> consider hyperthreaded cores when choosing how many processes to make
> the default for the local scheduler - so even if all the workers were
> completely busy, you might not see more than 50% utilisation. (That's
> what happens here on my system - the OS thinks it has 12 cores, but
> really it has 6 hyperthreaded cores, so the default is for MATLAB to
> launch 6 local workers).
>
> > Any alternatives? For one, is there a way to gather directly worker
> > to worker without exiting spmd and bringing it all to the master?
>
> You can use functions like either labBroadcast to send data from one
> worker to all the others, and gcat or gop to perform 'reduction'
> operations. Very briefly, here's how they work
>
> % assume matlabpool size 4
> spmd
> data = rand();
>
> % broadcast from lab 1, x1 gets the value from 'data' on lab 1.
> x1 = labBroadcast(1, data);
>
> % concatenation - each lab gets x2 = [1 2 3 4];
> x2 = gcat(labindex);
>
> % general reduction - in this case, each lab gets
> % x3 = 1 + 2 + 3 + 4
> x3 = gop(@plus, labindex);
> end
>
> > The data is small and functions cheap, so maybe I'd be ahead to let
> > everyone do the same computations just to keep things flowing.
>
> That sounds like a very good idea.
>
> Cheers,
>
> Edric.

"Chuck37 " <chuck3737@yahooremovethis.com> writes:

> By the way, is it possible to plot while still in the spmd? I like to
> put up a plot as I go to monitor progress.

Unfortunately it is not - the workers have no ability to show graphics,
and the client is busy looking after the workers. The best you can do
currently is to display status to the command window.

Cheers,

Edric.