Thread Subject: parfor error: No current session exists to connect a parfor controller to

Hi,

I'm starting to use parfor, so wrote a small piece of test code to see how it would work. Unfortunately, although matlab computes the answer, this is very slow and gives me a lot of identical errors (I think one for each iteration of the loop):

> In parallel_function at 447
  In timingTestM>contains at 17
  In timingTestM>(parfor body) at 7
  In parallel_function>make_general_channel/channel_general at 886
  In remoteParallelFunction at 31
Warning: Error caught during construction of remote parfor code.
The parfor construct will now be run locally rather than
on the remote matlabpool. The most likely cause of this is
an inability to send over input arguments because of a
serialization error. The error report from the caught error is:
Error using ==> distcomp.remoteparfor.remoteparfor at 12
No current session exists to connect a parfor controller to

Error in ==> parallel_function at 433
        P = distcomp.remoteparfor(W, @make_channel, parfor_C);

Error in ==> timingTestM>contains at 17
parfor k = 1:numel(vect)

Error in ==> timingTestM>(parfor body) at 7
    d = contains(r, 5);

Error in ==> parallel_function>make_general_channel/channel_general at 886
                    F(C{:});

Error in ==> remoteParallelFunction at 31
        out = distcompMakeByteBufferHandle(distcompserialize(feval(channel,
        channelArgs{:})));


Does anyone know what this means? I get this every time I try to run this particular function, but never with a different piece of test code run in the same session (I can post both if necessary).
I don't get any static errors when I load the code, so I don't think it breaks any of the parfor restrictions, and I've tried a lot of variations, but they all give the same error. The matlabpool I'm using is started with 'local', so I'm not sure why its complaint relates to remote rather than local running.

Any ideas welcomed!

Tara

Hi Tara,

What can you tell us about the size of the matrices being used in your parfor loop? If the variables are too large, parfor deems it beneficial to just run the code locally. Here the term locally simple means in serial on the host machine, not using the local scheduler.

One way to test this out is to restart your local pool and bring up the Task Manager or top. When you run your parfor, you should see the Workers increasing in memory as well. If they're not, then MATLAB is running everything serially -- as the warning mentions.

To work around the issue of marshalling large data sets around consider replacing broadcast variables with temporary variables if possible. Let me give a simple example:

a = rand(10000);
parfor idx = 1:10
   b(idx) = max(a(idx,:));
end

In the above example, a is broadcast. To make it temporary, try

parfor idx = 1:10
   a = rand(10000);
   b(idx) = max(a(idx,:));
end

The difference is that in the first example, a is created in the client and then pushed out to each of the workers. In the second, the creation of a is delayed onto Workers. It's a bit obscure for straight MATLAB and not efficient if a is required after the parfor, but it might circumvent your problem.

I haven't seen the error message before.

Raymond

"t g" <tg@remove.cs.york.ac.uk> wrote in message <gvmdhl$t0j$1@fred.mathworks.com>...
> Hi,
>
> I'm starting to use parfor, so wrote a small piece of test code to see how it would work. Unfortunately, although matlab computes the answer, this is very slow and gives me a lot of identical errors (I think one for each iteration of the loop):
>
> > In parallel_function at 447
> In timingTestM>contains at 17
> In timingTestM>(parfor body) at 7
> In parallel_function>make_general_channel/channel_general at 886
> In remoteParallelFunction at 31
> Warning: Error caught during construction of remote parfor code.
> The parfor construct will now be run locally rather than
> on the remote matlabpool. The most likely cause of this is
> an inability to send over input arguments because of a
> serialization error. The error report from the caught error is:
> Error using ==> distcomp.remoteparfor.remoteparfor at 12
> No current session exists to connect a parfor controller to
>
> Error in ==> parallel_function at 433
> P = distcomp.remoteparfor(W, @make_channel, parfor_C);
>
> Error in ==> timingTestM>contains at 17
> parfor k = 1:numel(vect)
>
> Error in ==> timingTestM>(parfor body) at 7
> d = contains(r, 5);
>
> Error in ==> parallel_function>make_general_channel/channel_general at 886
> F(C{:});
>
> Error in ==> remoteParallelFunction at 31
> out = distcompMakeByteBufferHandle(distcompserialize(feval(channel,
> channelArgs{:})));
>
>
> Does anyone know what this means? I get this every time I try to run this particular function, but never with a different piece of test code run in the same session (I can post both if necessary).
> I don't get any static errors when I load the code, so I don't think it breaks any of the parfor restrictions, and I've tried a lot of variations, but they all give the same error. The matlabpool I'm using is started with 'local', so I'm not sure why its complaint relates to remote rather than local running.
>
> Any ideas welcomed!
>
> Tara

"Raymond Norris" <raymond.norris@mathworks.com> wrote in message <gvmudl$2kf$1@fred.mathworks.com>...
> Hi Tara,

Hi Raymond,

> What can you tell us about the size of the matrices being used in your parfor loop? If the variables are too large, parfor deems it beneficial to just run the code locally.

Nothing is really passed into the loop, and a 500x500 matrix of 3s is created inside and then flattened. I've also tried this with just a 50x50 matrix, as well as just a single trial rather than 1000, but I still get the error.

The full code is (in a single file):
function time = timingTestM()
  tic;
  ntrials = 1000;
  parfor k = 1:ntrials
    r = repmat(3, 500);
    r = r(:)';
    d = contains(r, 5);
  end
  time = toc;
end

% searches all elements regardless of finding a match
function found = contains(vect, elem)
found = 0;
parfor k = 1:numel(vect)
if elem == vect(k)
found = 1;
end
end
end

> Here the term locally simple means in serial on the host machine, not using the local scheduler.

This is good to know, thanks!
 
> One way to test this out is to restart your local pool and bring up the Task Manager or top. When you run your parfor, you should see the Workers increasing in memory as well. If they're not, then MATLAB is running everything serially -- as the warning mentions.

Strangely enough, it seems like it is running it in parallel anyway - the time taken drops from about 14s to about 9s when running with 8 workers. top (with toggle 1) shows that a lot of cpus are indeed kicking in, and running
ps aux | grep -i matlab | wc -l
before and after starting the pool shows a jump from 3 entries for matlab to 19, followed by a drop back to 3 once the pool is closed.

> To work around the issue of marshalling large data sets around consider replacing ...snip...
might circumvent your problem.

Thanks for this tip as well, it definitely sounds like something I'll have to look out for in my project code, as I'm working with large numbers of images, albeit small ones.

> I haven't seen the error message before.

I wasn't able to turn anything up searching for it either, which suggests either it's a new message, or this doesn't happen very often! I did wonder if it might be a configuration problem on our system, except that another example worked fine without the error. Not really sure where to go from here! Do you think it's possible that the errors are insignificant? Is there any way of suppressing particular errors so that they don't fill up my logs?

Thanks for your help,

Tara


> "t g" <tg@remove.cs.york.ac.uk> wrote in message <gvmdhl$t0j$1@fred.mathworks.com>...
> > Hi,
> >
> > I'm starting to use parfor, so wrote a small piece of test code to see how it would work. Unfortunately, although matlab computes the answer, this is very slow and gives me a lot of identical errors (I think one for each iteration of the loop):
> >
> > > In parallel_function at 447
> > In timingTestM>contains at 17
> > In timingTestM>(parfor body) at 7
> > In parallel_function>make_general_channel/channel_general at 886
> > In remoteParallelFunction at 31
> > Warning: Error caught during construction of remote parfor code.
> > The parfor construct will now be run locally rather than
> > on the remote matlabpool. The most likely cause of this is
> > an inability to send over input arguments because of a
> > serialization error. The error report from the caught error is:
> > Error using ==> distcomp.remoteparfor.remoteparfor at 12
> > No current session exists to connect a parfor controller to
> >
> > Error in ==> parallel_function at 433
> > P = distcomp.remoteparfor(W, @make_channel, parfor_C);
> >
> > Error in ==> timingTestM>contains at 17
> > parfor k = 1:numel(vect)
> >
> > Error in ==> timingTestM>(parfor body) at 7
> > d = contains(r, 5);
> >
> > Error in ==> parallel_function>make_general_channel/channel_general at 886
> > F(C{:});
> >
> > Error in ==> remoteParallelFunction at 31
> > out = distcompMakeByteBufferHandle(distcompserialize(feval(channel,
> > channelArgs{:})));
> >
> >
> > Does anyone know what this means? I get this every time I try to run this particular function, but never with a different piece of test code run in the same session (I can post both if necessary).
> > I don't get any static errors when I load the code, so I don't think it breaks any of the parfor restrictions, and I've tried a lot of variations, but they all give the same error. The matlabpool I'm using is started with 'local', so I'm not sure why its complaint relates to remote rather than local running.
> >
> > Any ideas welcomed!
> >
> > Tara

"t g" <tg@remove.cs.york.ac.uk> writes:

> I'm starting to use parfor, so wrote a small piece of test code to see how it
> would work. Unfortunately, although matlab computes the answer, this is very
> slow and gives me a lot of identical errors (I think one for each iteration of
> the loop):
>
>> In parallel_function at 447
> In timingTestM>contains at 17
> In timingTestM>(parfor body) at 7
> In parallel_function>make_general_channel/channel_general at 886
> In remoteParallelFunction at 31
> Warning: Error caught during construction of remote parfor code.
> The parfor construct will now be run locally rather than
> on the remote matlabpool. The most likely cause of this is
> an inability to send over input arguments because of a
> serialization error. The error report from the caught error is:
> Error using ==> distcomp.remoteparfor.remoteparfor at 12
> No current session exists to connect a parfor controller to

Hi,

This is a cosmetic bug in Parallel Computing Toolbox in R2009a (it should be
fixed in the next release). The warning comes when executing the inner parfor
loop - it's warning you that it cannot run that remotely, however you're already
running remotely, so that warning is pointless (and we shouldn't even be trying
to do that).

You can silence the warning for now by running something like this:

parfor ii=1:matlabpool('size')
    warning( 'off', 'distcomp:remoteparfor:ParforRunningLocally' );
end

You only need to do this once after opening the matlabpool.

Also note that you do not get any additional parallelism by having nested parfor
loops. I would suggest that if you know that your inner loops are always going
to be executed within an outer parfor loop, then revert them to simple "for"
loops as this will avoid going through the parfor machinery which will add a
small amount of overhead.

Cheers,

Edric.

> Hi,

Hi Edric,

> This is a cosmetic bug in Parallel Computing Toolbox in R2009a (it should be
> fixed in the next release). The warning comes when executing the inner parfor
> loop - it's warning you that it cannot run that remotely, however you're already
> running remotely, so that warning is pointless (and we shouldn't even be trying
> to do that).
>
> You can silence the warning for now by running something like this:
>
> parfor ii=1:matlabpool('size')
> warning( 'off', 'distcomp:remoteparfor:ParforRunningLocally' );
> end
>
> You only need to do this once after opening the matlabpool.

Thanks for this information and workaround, the code now runs as expected both with and without matlabpool.

> Also note that you do not get any additional parallelism by having nested parfor
> loops. I would suggest that if you know that your inner loops are always going
> to be executed within an outer parfor loop, then revert them to simple "for"
> loops as this will avoid going through the parfor machinery which will add a
> small amount of overhead.

This is very useful to know, thanks! Trying it with and without the inner parfor on my test code drops the runtime from about 6s to ~1.2s on the machine I'm using, so that's a real help.

Thanks again,
Tara

Hi Edric,

I have the same error message showing up although the circumstances are different. Look below.

>
> > I'm starting to use parfor, so wrote a small piece of test code to see how it
> > would work. Unfortunately, although matlab computes the answer, this is very
> > slow and gives me a lot of identical errors (I think one for each iteration of
> > the loop):
> >
> >> In parallel_function at 447
> > In timingTestM>contains at 17
> > In timingTestM>(parfor body) at 7
> > In parallel_function>make_general_channel/channel_general at 886
> > In remoteParallelFunction at 31
> > Warning: Error caught during construction of remote parfor code.
> > The parfor construct will now be run locally rather than
> > on the remote matlabpool. The most likely cause of this is
> > an inability to send over input arguments because of a
> > serialization error. The error report from the caught error is:
> > Error using ==> distcomp.remoteparfor.remoteparfor at 12
> > No current session exists to connect a parfor controller to
>
> Hi,
>
> This is a cosmetic bug in Parallel Computing Toolbox in R2009a (it should be
> fixed in the next release). The warning comes when executing the inner parfor
> loop - it's warning you that it cannot run that remotely, however you're already
> running remotely, so that warning is pointless (and we shouldn't even be trying
> to do that).

I get this problem and I don't have any inner parfor loop. In fact I have only one parfor loop in my code. The program can run without any problem for several hours and suddenly this error message appears.
 
> You can silence the warning for now by running something like this:
>
> parfor ii=1:matlabpool('size')
> warning( 'off', 'distcomp:remoteparfor:ParforRunningLocally' );
> end
>
> You only need to do this once after opening the matlabpool.
>

Thanks for sharing it with us. However, in my case, after issuing the error message several times (can be a lot), Matlab automatically issues an matlabpool close command and terminates itself. Thus I am slightly skeptical as to the benefits of hiding the error message.


> Also note that you do not get any additional parallelism by having nested parfor
> loops. I would suggest that if you know that your inner loops are always going
> to be executed within an outer parfor loop, then revert them to simple "for"
> loops as this will avoid going through the parfor machinery which will add a
> small amount of overhead.
>
> Cheers,
>
> Edric.


Do you have any tips in this case?

Thank you so much in advance.

All the best,
José

"José Blasques" <jpbl@mek.dtu.dk> writes:

> I have the same error message showing up although the circumstances
> are different. Look below.
>
>>
>> > I'm starting to use parfor, so wrote a small piece of test code to see how it
>> > would work. Unfortunately, although matlab computes the answer, this is very
>> > slow and gives me a lot of identical errors (I think one for each iteration of
>> > the loop):
>> >
>> >> In parallel_function at 447
>> > In timingTestM>contains at 17
>> > In timingTestM>(parfor body) at 7
>> > In parallel_function>make_general_channel/channel_general at 886
>> > In remoteParallelFunction at 31
>> > Warning: Error caught during construction of remote parfor code.
>> > The parfor construct will now be run locally rather than
>> > on the remote matlabpool. The most likely cause of this is
>> > an inability to send over input arguments because of a
>> > serialization error. The error report from the caught error is:
>> > Error using ==> distcomp.remoteparfor.remoteparfor at 12
>> > No current session exists to connect a parfor controller to
>>
>> Hi,
>>
>> This is a cosmetic bug in Parallel Computing Toolbox in R2009a (it should be
>> fixed in the next release). The warning comes when executing the inner parfor
>> loop - it's warning you that it cannot run that remotely, however you're already
>> running remotely, so that warning is pointless (and we shouldn't even be trying
>> to do that).
>
> I get this problem and I don't have any inner parfor loop. In fact I
> have only one parfor loop in my code. The program can run without any
> problem for several hours and suddenly this error message appears.

You're right that if you don't have any inner PARFOR loops, then you're
in a somewhat different situation, and silencing the warning isn't
useful. As the error message states, the size of the data to be sent to
the workers can sometimes be a problem. Do you know how much data you're
sending across?

Cheers,

Edric.

Edric M Ellis <eellis@mathworks.com> wrote in message <ytwbox0aof9.fsf@uk-eellis-deb5-64.dhcp.mathworks.com>...
> "José Blasques" <jpbl@mek.dtu.dk> writes:
>
> > I have the same error message showing up although the circumstances
> > are different. Look below.
> >
> >>
> >> > I'm starting to use parfor, so wrote a small piece of test code to see how it
> >> > would work. Unfortunately, although matlab computes the answer, this is very
> >> > slow and gives me a lot of identical errors (I think one for each iteration of
> >> > the loop):
> >> >
> >> >> In parallel_function at 447
> >> > In timingTestM>contains at 17
> >> > In timingTestM>(parfor body) at 7
> >> > In parallel_function>make_general_channel/channel_general at 886
> >> > In remoteParallelFunction at 31
> >> > Warning: Error caught during construction of remote parfor code.
> >> > The parfor construct will now be run locally rather than
> >> > on the remote matlabpool. The most likely cause of this is
> >> > an inability to send over input arguments because of a
> >> > serialization error. The error report from the caught error is:
> >> > Error using ==> distcomp.remoteparfor.remoteparfor at 12
> >> > No current session exists to connect a parfor controller to
> >>
> >> Hi,
> >>
> >> This is a cosmetic bug in Parallel Computing Toolbox in R2009a (it should be
> >> fixed in the next release). The warning comes when executing the inner parfor
> >> loop - it's warning you that it cannot run that remotely, however you're already
> >> running remotely, so that warning is pointless (and we shouldn't even be trying
> >> to do that).
> >
> > I get this problem and I don't have any inner parfor loop. In fact I
> > have only one parfor loop in my code. The program can run without any
> > problem for several hours and suddenly this error message appears.
>
> You're right that if you don't have any inner PARFOR loops, then you're
> in a somewhat different situation, and silencing the warning isn't
> useful. As the error message states, the size of the data to

How can I find out how much data I am sending across? And if it is too much, how can I reduce it?

"José Blasques" <jpbl@mek.dtu.dk> writes:

>> You're right that if you don't have any inner PARFOR loops, then you're
>> in a somewhat different situation, and silencing the warning isn't
>> useful. As the error message states, the size of the data to
>
> How can I find out how much data I am sending across? And if it is too
> much, how can I reduce it?

Any variables in the workspace before the PARFOR loop will be broadcast
into the loop, check there first to see if there's anything
large. Obviously, all your sliced inputs/outputs must be transferred
too.

Cheers,

Edric.

Edric M Ellis <eellis@mathworks.com> wrote in message <ytw7h7m9z3l.fsf@uk-eellis-deb5-64.dhcp.mathworks.com>...
> "José Blasques" <jpbl@mek.dtu.dk> writes:
>
> >> You're right that if you don't have any inner PARFOR loops, then you're
> >> in a somewhat different situation, and silencing the warning isn't
> >> useful. As the error message states, the size of the data to
> >
> > How can I find out how much data I am sending across? And if it is too
> > much, how can I reduce it?
>
> Any variables in the workspace before the PARFOR loop will be broadcast
> into the loop, check there first to see if there's anything
> large. Obviously, all your sliced inputs/outputs must be transferred
> too.
>
> Cheers,
>
> Edric.


Edric,

I've checked it and the there are at least 500mb of data in the workspace before the parfor loop. I am not sure what a lot is but it seems like quite a few mb at least.

The fact is I do need all that data inside the parfor loop so nothing to do there. I do have other for loops inside so the alternative is to remove the parfor loop from where I have it now and use at some other inner for loop instead. This brings me to my next question. I will use some numbers from the numerical experiments I am doing now so you can have an idea of what I am up against.

Which one of the next alternatives you think will give the best performance?

1. Use the parfor at an outer loop (like I am doing now). It is called around 2000 times, it loops from 1:20000 each time, and requires the transference of 500mb.

2. Use the parfor at an inner loop which is called minimum 1000000 times, loops from 1:9 each time, and requires the transference of a lot less information (say a few mb).

Please feel free to expand on the topic as I am trying to learn as much as possible about this. I am running the code in a cluster and we've been having some problems with some nodes slowing down. I am wondering if my code has anything to do with that.

Thanks in advance,
José