Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Parfor structure transparency ( cannot be classified ) issue

Subject: Parfor structure transparency ( cannot be classified ) issue

From: Brendan

Date: 2 Dec, 2010 23:59:07

Message: 1 of 8

The cluster I am running my code on is sometimes unstable, and so I am trying to figure out a way to save the contents of a structure array in a .mat file in the middle of a parfor run. I am getting the error "The variable indRunStruct in a parfor cannot be classified".

Here is effectively what I would like to do:

for jj=1:10
    parfor ii=1:10
        x(ii,jj)=test2getAstruct;
        testparforsaveroutine('testparfor',x)
    end
end

function out=test2getAstruct
out.x=1;
out.y=rand;
out.z=rand;
end

function testparforsaveroutine(filename,var)
save(filename,'-append','var')
end


I sort of assume that "cannot be classified" has to do with the transparency issue, but I don't understand why I am breaking that here. If I remove the indices make it x=test2getAstruct, the code works fine. I would then like to know if there is a workaround to get it to do what I want to do.

Thanks!

Subject: Parfor structure transparency ( cannot be classified ) issue

From: Brendan

Date: 3 Dec, 2010 01:14:07

Message: 2 of 8

Figured out a workaround, will post it when I have a chance.

Subject: Parfor structure transparency ( cannot be classified ) issue

From: Edric M Ellis

Date: 3 Dec, 2010 08:12:08

Message: 3 of 8

"Brendan " <btracey@stanford.edu> writes:

> The cluster I am running my code on is sometimes unstable, and so I am
> trying to figure out a way to save the contents of a structure array
> in a .mat file in the middle of a parfor run. I am getting the error
> "The variable indRunStruct in a parfor cannot be classified".
>
> Here is effectively what I would like to do:
>
> for jj=1:10
> parfor ii=1:10
> x(ii,jj)=test2getAstruct;
> testparforsaveroutine('testparfor',x)
> end
> end
> [...]

The problem here is that 'x' looks like a "sliced" variable - i.e. one
which each worker can fill in their own pieces of as they're going
along. However, you're attempting to save the whole value of 'x' by
passing it to your 'testparforsaveroutine' function. The whole value
isn't available on the workers for a sliced variable, that's where the
problem is. You should be able to do something more like this:

for jj=1:10
   parfor ii=1:10
       tmp = test2getAstruct;
       testparforsaveroutine(sprintf('testparfor.%d.%d',ii,jj),tmp);
       x(ii,jj) = tmp;
   end
end

This saves each value into a separate file - which is probably a good
thing since if you have each worker simultaneously trying to append to
the same file, you will eventually hit problems when they clash. (This
can be particularly problematic with Windows workers).

By the way, if possible, it would probably be more efficient to have the
PARFOR as the outer loop.

Cheers,

Edric.

Subject: Parfor structure transparency ( cannot be classified ) issue

From: Brendan

Date: 3 Dec, 2010 16:21:05

Message: 4 of 8

 > The problem here is that 'x' looks like a "sliced" variable - i.e. one
> which each worker can fill in their own pieces of as they're going
> along. However, you're attempting to save the whole value of 'x' by
> passing it to your 'testparforsaveroutine' function. The whole value
> isn't available on the workers for a sliced variable, that's where the
> problem is. You should be able to do something more like this:
>
> for jj=1:10
> parfor ii=1:10
> tmp = test2getAstruct;
> testparforsaveroutine(sprintf('testparfor.%d.%d',ii,jj),tmp);
> x(ii,jj) = tmp;
> end
> end
>
> This saves each value into a separate file - which is probably a good
> thing since if you have each worker simultaneously trying to append to
> the same file, you will eventually hit problems when they clash. (This
> can be particularly problematic with Windows workers).
>
> By the way, if possible, it would probably be more efficient to have the
> PARFOR as the outer loop.
>
> Cheers,
>
> Edric.


Yea, I was hoping to avoid having a few thousand files to deal with. However, we did come up with the following:

for jj=1:10
    parfor ii=1:10
        x(ii,jj)=test2getAstruct;
        testparforsaveroutine('testparfor',x(ii,jj),ii,jj)
    end
end

function testparforsaveroutine(filename,var,i,j)
persistent mat
mat(i,j).field = var;
save(filename,'mat')
end

It gets around having all of the different files. You make a good point about workers simultaneously appending the same file. I haven't yet tried the code with a few workers to see how unstable it is.



> By the way, if possible, it would probably be more efficient to have the
> PARFOR as the outer loop.

So, I don't think that's actually true in my case because my case is really more like:
for jj=1:6
    for ii=1:2000
    end
end

and the loops around j vary significantly in time output (j=1 the inner loop takes 10 seconds, j=6 the inner loop takes 200 seconds).

But, for future reference, why is it better to parfor the outer loop?

Subject: Parfor structure transparency ( cannot be classified ) issue

From: Edric M Ellis

Date: 6 Dec, 2010 07:42:49

Message: 5 of 8

"Brendan " <btracey@stanford.edu> writes:
> [...]
> It gets around having all of the different files. You make a good
> point about workers simultaneously appending the same file. I haven't
> yet tried the code with a few workers to see how unstable it is.

One thing you might try is to have each worker process write to a
separate file. You can use the task ID on the workers from
getCurrentTask

<http://www.mathworks.com/help/toolbox/distcomp/getcurrenttask.html>

to generate a unique filename.

>> By the way, if possible, it would probably be more efficient to have the
>> PARFOR as the outer loop.
>
> So, I don't think that's actually true in my case because my case is really more like:
> for jj=1:6
> for ii=1:2000
> end
> end
>
> and the loops around j vary significantly in time output (j=1 the
> inner loop takes 10 seconds, j=6 the inner loop takes 200 seconds).
>
> But, for future reference, why is it better to parfor the outer loop?

Ah, OK, the key thing is really to try and minimise the amount of
overhead (PARFOR inevitably introduces some overhead) whilst ensuring
that you've got sufficient parallelism.

Cheers,

Edric.

Subject: Parfor structure transparency ( cannot be classified ) issue

From: Brendan

Date: 7 Dec, 2010 22:24:06

Message: 6 of 8


So I did what you suggested (creating a lot of different files), for the reasons you suggested, and it seems to be working. The remaining issue is that while on my laptop things seem to be going fast, on the cluster I'm running on saving is slow.

I have the following code which goes before my double loop which includes the parfor. I realize that my code could be more efficient (for example getting rid of the eval command before the save). Running profiler, the load command is still fast, taking about 2 seconds through both of the for loops, but the save command is slow, taking about 40 seconds. For clarification, for the save and load commands, I mean that line specifically ( save(varName,varName) ) not the double for loop on the whole. Do you have any comments on why save might be slow on one or the other, or how I might go about making it faster?

Thanks
  
%Preallocate allRunOutputs to be all empty
allRunOutputs(MCParameters.numRuns,length(MCParameters.numSampArray)).field=[];
% Load in restarted data
if MCParameters.useRestartFile==0
elseif MCParameters.useRestartFile==1
    cd('RestartFileSaves');
    cd(MCParameters.restartFileName);
    for jj=1:length(MCParameters.numSampArray);
        for ii=1:MCParameters.numRuns
            varName=['outputsave_' num2str(ii) '_' num2str(jj)];
            load(varName,varName);
            eval(['allRunOutputs(ii,jj).field=' varName ';']);
        end
    end
    cd('..');
    cd('..');
end

if MCParameters.saveWorkspaceIntermediate==1
    % Preps the workspace save
    % ParforSaveWorkspace saves the whole workspace (that exists in the
    % outer loop)
    ParforSaveWorkspace(MCParameters)
    cd('RestartFileSaves');
    cd(MCParameters.saveWorkspaceFilename);
    for jj=1:length(MCParameters.numSampArray);
        for ii=1:MCParameters.numRuns
            varName=['outputsave_' num2str(ii) '_' num2str(jj)];
            eval([varName '=allRunOutputs(ii,jj).field;']);
            save(varName,varName);
        end
    end
    cd('..')
    cd('..')
end

Subject: Parfor structure transparency ( cannot be classified ) issue

From: Brendan

Date: 7 Dec, 2010 22:36:05

Message: 7 of 8

Nevermind about different on different machines, I must have been running something differently. Any comments on speedup, or do I just have to deal with the fact that writing to disk is slow.

Subject: Parfor structure transparency ( cannot be classified ) issue

From: Brendan

Date: 7 Dec, 2010 23:03:05

Message: 8 of 8

One way I can make it better is by saving the ones that are finished and not having the placeholder files by using the "exist" command. Is that probably my best bet, or are there ways I can vectorize it which would make it more efficient.

Thanks!

New code:
%Preallocate allRunOutputs to be all empty
allRunOutputs(MCParameters.numRuns,length(MCParameters.numSampArray)).field=[];
% Load in restarted data
if MCParameters.useRestartFile==0
elseif MCParameters.useRestartFile==1
    cd('RestartFileSaves');
    cd(MCParameters.restartFileName);
    for jj=1:length(MCParameters.numSampArray);
        for ii=1:MCParameters.numRuns
            varName=['outputsave_' num2str(ii) '_' num2str(jj)];
            out=exist([varName '.mat'],'file');
            if out==2
                load(varName,varName);
            end
            eval(['allRunOutputs(ii,jj).field=' varName ';']);
        end
    end
    cd('..');
    cd('..');
end

if MCParameters.saveWorkspaceIntermediate==1
    % Preps the workspace save
    % ParforSaveWorkspace saves the whole workspace (that exists in the
    % outer loop)
    ParforSaveWorkspace(MCParameters)
    % ParforSaveOutput saves the outputs from the original run
    cd('RestartFileSaves');
    cd(MCParameters.saveWorkspaceFilename);
    for jj=1:length(MCParameters.numSampArray);
        for ii=1:MCParameters.numRuns
            if(~isempty(allRunOutputs(ii,jj).field))
                varName=['outputsave_' num2str(ii) '_' num2str(jj)];
                eval([varName '=allRunOutputs(ii,jj).field;']);
                save(varName,varName);
            end
        end
    end
    cd('..')
    cd('..')
end

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us