Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
parfor: keep outputs with same output filename

Subject: parfor: keep outputs with same output filename

From: Mike

Date: 21 Jul, 2012 03:56:23

Message: 1 of 15

Hi,

If I run an exection file which output the same filename for many times using parfor, the output files will be overwritten. At last, I have only one output file. How to keep the output file in every run?

Thanks.

Mike

Subject: parfor: keep outputs with same output filename

From: dpb

Date: 21 Jul, 2012 18:20:58

Message: 2 of 15

On 7/20/2012 10:56 PM, Mike wrote:
> Hi,
>
> If I run an exection file which output the same filename for many
> times using parfor, the output files will be overwritten. At last, I
> have only one output file. How to keep the output file in every run?
...

I presume you're looking for the 'append' option for fopen???

doc fopen

--

Subject: parfor: keep outputs with same output filename

From: Mike

Date: 23 Jul, 2012 09:45:42

Message: 3 of 15

Hi, dpb

thank you for your info.
To be more clear, I simply my question with follows:
1) I create a fortran file with following:

program main
integer:: i
open(10,file='inputpar.dat',status='old')
read(10,*) i
write(30,*) i
write(*,*) 'fortran main :', i
end

I compile it within the project and have Console2.exe in 'D:\temp\Console2\Console2\Debug'.

2) in matlab, I have following:
fid2=fopen('test.dat','a'); % this file should be run parallel'
matlabpool open local 3
parfor i=1:100
    fid1=fopen('inputpar.dat','w'); % be read by Console2.exe
    fprintf(fid1,'%2d\n',i);
    fclose(fid1);
    dos('Console2.exe<inputpar.dat');
    out=importdata('fort.30')
    fprintf(fid2,'%2d\n',out);
end
matlabpool close
s=load('test.dat')

Then I run this cell (step 2), I have the results:
Starting matlabpool using the 'local' configuration ... connected to 3 labs.
 fortran main : 64
 fortran main : 64
 fortran main : 64
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Invalid file identifier. Use fopen to generate a valid file identifier.
 
I've checked the content in inputpar.dat is 64.
I've also checked the content in fort.30 is also 64.
I also check
>> out=importdata('fort.30')

out =

    64

I don't understand the error message shown above.
May you explain for me?

Thank you in advance.

Mike
On Sunday, July 22, 2012 2:20:58 AM UTC+8, dpb wrote:
> On 7/20/2012 10:56 PM, Mike wrote:
> > Hi,
> >
> > If I run an exection file which output the same filename for many
> > times using parfor, the output files will be overwritten. At last, I
> > have only one output file. How to keep the output file in every run?
> ...
>
> I presume you're looking for the 'append' option for fopen???
>
> doc fopen
>
> --

Subject: parfor: keep outputs with same output filename

From: Torsten

Date: 23 Jul, 2012 10:05:32

Message: 4 of 15

On 23 Jul., 11:45, Mike <Sulfate...@gmail.com> wrote:
> Hi, dpb
>
> thank you for your info.
> To be more clear, I simply my question with follows:
> 1) I create a fortran file with following:
>
> program main
> integer:: i
> open(10,file='inputpar.dat',status='old')
> read(10,*) i
> write(30,*) i
> write(*,*) 'fortran main :', i
> end
>
> I compile it within the project and have Console2.exe in 'D:\temp\Console2\Console2\Debug'.
>
> 2) in matlab, I have following:
> fid2=fopen('test.dat','a'); % this file should be run parallel'
> matlabpool open local 3
> parfor i=1:100
> fid1=fopen('inputpar.dat','w'); % be read by Console2.exe
> fprintf(fid1,'%2d\n',i);
> fclose(fid1);
> dos('Console2.exe<inputpar.dat');
> out=importdata('fort.30')
> fprintf(fid2,'%2d\n',out);
> end
> matlabpool close
> s=load('test.dat')
>
> Then I run this cell (step 2), I have the results:
> Starting matlabpool using the 'local' configuration ... connected to 3 labs.
> fortran main : 64
> fortran main : 64
> fortran main : 64
> ??? Error using ==> parallel_function at 598
> Error in ==> parallel_function>make_general_channel/channel_general at 894
> Invalid file identifier. Use fopen to generate a valid file identifier.
>
> I've checked the content in inputpar.dat is 64.
> I've also checked the content in fort.30 is also 64.
> I also check
>
> >> out=importdata('fort.30')
>
> out =
>
> 64
>
> I don't understand the error message shown above.
> May you explain for me?
>
> Thank you in advance.
>
> Mike
>
>
>
> On Sunday, July 22, 2012 2:20:58 AM UTC+8, dpb wrote:
> > On 7/20/2012 10:56 PM, Mike wrote:
> > > Hi,
> > >
> > > If I run an exection file which output the same filename for many
> > > times using parfor, the output files will be overwritten. At last, I
> > > have only one output file. How to keep the output file in every run?
> > ...
>
> > I presume you're looking for the 'append' option for fopen???
>
> > doc fopen
>
> > --- Zitierten Text ausblenden -
>
> - Zitierten Text anzeigen -

Filenames in MATLAB have to start with a letter.
Look at "Naming Other Files" under
http://www.mathworks.de/help/techdoc/matlab_prog/f10-61281.html

Best wishes
Torsten.

Subject: parfor: keep outputs with same output filename

From: Steven_Lord

Date: 23 Jul, 2012 14:09:50

Message: 5 of 15



"Mike" <SulfateIon@gmail.com> wrote in message
news:f23e389b-3e97-4bcd-8657-096415228bcb@googlegroups.com...
> Hi, dpb
>
> thank you for your info.
> To be more clear, I simply my question with follows:
> 1) I create a fortran file with following:
>
> program main
> integer:: i
> open(10,file='inputpar.dat',status='old')
> read(10,*) i
> write(30,*) i
> write(*,*) 'fortran main :', i
> end
>
> I compile it within the project and have Console2.exe in
> 'D:\temp\Console2\Console2\Debug'.
>
> 2) in matlab, I have following:
> fid2=fopen('test.dat','a'); % this file should be run parallel'
> matlabpool open local 3
> parfor i=1:100

The workers have no idea that fid2 is a file identifier; they didn't open
the file so they don't have access to it.

> fid1=fopen('inputpar.dat','w'); % be read by Console2.exe
> fprintf(fid1,'%2d\n',i);
> fclose(fid1);
> dos('Console2.exe<inputpar.dat');
> out=importdata('fort.30')
> fprintf(fid2,'%2d\n',out);
> end
> matlabpool close
> s=load('test.dat')
>
> Then I run this cell (step 2), I have the results:
> Starting matlabpool using the 'local' configuration ... connected to 3
> labs.
> fortran main : 64
> fortran main : 64
> fortran main : 64
> ??? Error using ==> parallel_function at 598
> Error in ==> parallel_function>make_general_channel/channel_general at 894
> Invalid file identifier. Use fopen to generate a valid file identifier.

See above.

Note that the iterations of a PARFOR loop are NOT repeat NOT guaranteed to
be executed in any particular order. Therefore if you were to do this, at
best your output would be in arbitrary order in the file; at worst each
worker may "scribble" over the results another worker wrote to the file at
the same time.

http://en.wikipedia.org/wiki/Race_condition

I would either assemble all your data in a _variable_ rather than a file
inside the PARFOR loop [different workers can write to different elements of
a variable simultaneously without a problem] or have each worker write its
output to a separate file and combine them all together _after_ the PARFOR
loop is finished.

--
Steve Lord
slord@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

Subject: parfor: keep outputs with same output filename

From: Edric M Ellis

Date: 23 Jul, 2012 14:10:25

Message: 6 of 15

Mike <SulfateIon@gmail.com> writes:

> thank you for your info.
> To be more clear, I simply my question with follows:
> 1) I create a fortran file with following:
>
> program main
> integer:: i
> open(10,file='inputpar.dat',status='old')
> read(10,*) i
> write(30,*) i
> write(*,*) 'fortran main :', i
> end
>
> I compile it within the project and have Console2.exe in 'D:\temp\Console2\Console2\Debug'.
>
> 2) in matlab, I have following:
> fid2=fopen('test.dat','a'); % this file should be run parallel'
> matlabpool open local 3
> parfor i=1:100
> fid1=fopen('inputpar.dat','w'); % be read by Console2.exe
> fprintf(fid1,'%2d\n',i);
> fclose(fid1);
> dos('Console2.exe<inputpar.dat');
> out=importdata('fort.30')
> fprintf(fid2,'%2d\n',out);
> end
> matlabpool close
> s=load('test.dat')

I'm not 100% sure I understand the FORTRAN code, but the MATLAB code is
definitely trying to have each worker write to the same 'inputpar.dat'
file. This is bound to fail. One option would be to make the PARFOR loop
choose a filename based on the loop index, like this:

parfor i=1:100
    fname = sprintf('inputpar_%d.dat', i);
    fid1 = fopen(fname, 'w');
    if fid == -1, error('Could not write to: %s', fname); end
    ...
end

I suspect you'll also need to modify your FORTRAN code to do something
similar.

Cheers,

Edric.

Subject: parfor: keep outputs with same output filename

From: Steven_Lord

Date: 23 Jul, 2012 14:15:02

Message: 7 of 15



"Torsten" <Torsten.Hennig@umsicht.fraunhofer.de> wrote in message
news:1f1093fd-7d4b-4559-882c-5882aac33074@h20g2000yqe.googlegroups.com...

*snip*

> Filenames in MATLAB have to start with a letter.
> Look at "Naming Other Files" under
> http://www.mathworks.de/help/techdoc/matlab_prog/f10-61281.html

This is true for MATLAB program files, but MATLAB can write to a data file
whose name starts with a number or other symbol perfectly well.

fid = fopen('1234567.txt', 'wt');
for k = 1:10
    fprintf(fid, '%d\t%d\n', k, k.^2);
end
fclose(fid);
type 1234567.txt

1 1
2 4
3 9
4 16
5 25
6 36
7 49
8 64
9 81
10 100

That documentation section to which you linked appears to be incorrect.
MEX-file names do need to satisfy the requirements other MATLAB program
files need to satisfy (ISVARNAME) but MAT-files don't necessarily need to do
so. I'm not certain about Simulink model files, though I'd guess they fall
in the program file category. I'll note that to the documentation staff.

--
Steve Lord
slord@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

Subject: parfor: keep outputs with same output filename

From: dpb

Date: 23 Jul, 2012 14:27:16

Message: 8 of 15

On 7/23/2012 5:05 AM, Torsten wrote:
...


> Filenames in MATLAB have to start with a letter.
> Look at "Naming Other Files" under
> http://www.mathworks.de/help/techdoc/matlab_prog/f10-61281.html
>
...

Not OP's problem...

 >> type testit.30

109 0.42
110 0.73
111 0.69
...
131 0.43

 >> fid=fopen('testit.30','rt')
fid =
      3
 >> fgetl(fid)
ans =
109 0.42
 >> fid=fclose(fid);

% copy testit.30 30test.dat

 >> type 30test.dat

109 0.42
110 0.73
...

 >> fid=fopen('30test.dat','rt')
fid =
      3
 >> fgetl(fid)
ans =
109 0.42
 >> fgetl(fid)
ans =
110 0.73
 >> fid=fclose(fid);
 >>

I didn't pursue all the links from the initial above, but if it actually
says/implies that filenames (or extensions) can't start w/ a numeral,
it's simply in error. Filenames must follow the naming conventions of
the underlying OS at a minimum; whether there are any other restrictions
TMW has introduced inside Matlab I'm unaware, but certainly this isn't one.

--

Subject: parfor: keep outputs with same output filename

From: dpb

Date: 23 Jul, 2012 14:34:43

Message: 9 of 15

On 7/23/2012 4:45 AM, Mike wrote:

...[top posting repaired--please don't do that; hard conversation follow
makes]...

> On Sunday, July 22, 2012 2:20:58 AM UTC+8, dpb wrote:
...

>> I presume you're looking for the'append' option for fopen???
...
 > To be more clear, I simply my question with follows:
...
 > 2) in matlab, I have following:
 > fid2=fopen('test.dat','a'); % this file should be run parallel'
 > matlabpool open local 3
 > parfor i=1:100
 > fid1=fopen('inputpar.dat','w'); % be read by Console2.exe
 > fprintf(fid1,'%2d\n',i);
 > fclose(fid1);
 > dos('Console2.exe<inputpar.dat');
...
 >
 > Then I run this cell (step 2), I have the results:
 > Starting matlabpool using the 'local' configuration ... connected to
3 labs.
 > fortran main : 64
 > fortran main : 64
 > fortran main : 64
 > ??? Error using ==> parallel_function at 598
 > Error in ==> parallel_function>make_general_channel/channel_general
at 894
 > Invalid file identifier. Use fopen to generate a valid file
 > identifier.
 > I've checked the content in inputpar.dat is 64.
 > I've also checked the content in fort.30 is also 64.
 > I also check
 >>> out=importdata('fort.30')
 >
 > out =
 >
 > 64
 >
 > I don't understand the error message shown above.
 > May you explain for me?
...

My guess is the problem is one of timing--the file is locked by another
process when you're trying to open it in all likelihood. This is a very
difficult thing to try to get to work as you've written it.

Start by

[fid1,msg]=fopen('inputpar.dat','w'); % be read by Console2.exe
if fid1==-1
   error(msg)
end

And at least you'll know fid1 isn't valid and stop and have the system
error message.

--

Subject: parfor: keep outputs with same output filename

From: mike23liu@gmail.com

Date: 24 Jul, 2012 03:30:11

Message: 10 of 15

Hi,

Thank you all.
I've modified the section to
    [fid1,msg]=fopen('inputpar.dat','wt'); % be read by Console2.exe
    if fid1==-1
       error(msg)
    end
    fprintf(fid1,'%2d\n',i);
after parfor and re-run it, then also have the same message:

Starting matlabpool using the 'local' configuration ... connected to 3 labs.
 fortran main : 46
 fortran main : 46
 fortran main : 46
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Invalid file identifier. Use fopen to generate a valid file identifier.
 
>> matlabpool close
Sending a stop signal to all the labs ... stopped.

I run it again

Starting matlabpool using the 'local' configuration ... connected to 3 labs.
forrtl: severe (24): end-of-file during read, unit 10, file D:\temp\Console2\Console2\Debug\inputpar.dat
Image PC Routine Line Source
Console2.exe 004A511A Unknown Unknown Unknown
Console2.exe 00414390 Unknown Unknown Unknown
Console2.exe 0041356A Unknown Unknown Unknown
Console2.exe 00406A06 Unknown Unknown Unknown
Console2.exe 004010B9 _MAIN__ 4 Source1.f90
Console2.exe 004ABAD3 Unknown Unknown Unknown
Console2.exe 0044FC67 Unknown Unknown Unknown
Console2.exe 0044FB3F Unknown Unknown Unknown
kernel32.dll 766D339A Unknown Unknown Unknown
ntdll.dll 77489EF2 Unknown Unknown Unknown
ntdll.dll 77489EC5 Unknown Unknown Unknown
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Invalid file identifier. Use fopen to generate a valid file identifier.
 
>> matlabpool close
Sending a stop signal to all the labs ... stopped.

I run again many times. Sometimes I have

Starting matlabpool using the 'local' configuration ... connected to 3 labs.
forrtl: severe (47): write to READONLY file, unit 30, file D:\temp\Console2\Console2\Debug\fort.30
Image PC Routine Line Source
Console2.exe 004A511A Unknown Unknown Unknown
Console2.exe 00414390 Unknown Unknown Unknown
Console2.exe 0041356A Unknown Unknown Unknown
Console2.exe 0040A06A Unknown Unknown Unknown
Console2.exe 004010FE _MAIN__ 5 Source1.f90
Console2.exe 004ABAD3 Unknown Unknown Unknown
Console2.exe 0044FC67 Unknown Unknown Unknown
Console2.exe 0044FB3F Unknown Unknown Unknown
kernel32.dll 766D339A Unknown Unknown Unknown
ntdll.dll 77489EF2 Unknown Unknown Unknown
ntdll.dll 77489EC5 Unknown Unknown Unknown
 fortran main : 64
 fortran main : 64
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Invalid file identifier. Use fopen to generate a valid file identifier.
 
>> matlabpool close
Sending a stop signal to all the labs ... stopped.

There is no message from "error(msg)".

Also, I kind of choose fixed filename: "inputpar.dat", since I wouldn't like to change that filename in FORTRAN program. My whole idea is to run FORTRAN program with input parameter file created by matlab. And run many times.

Mike

On Monday, July 23, 2012 10:34:43 PM UTC+8, dpb wrote:
> On 7/23/2012 4:45 AM, Mike wrote:
>
> ...[top posting repaired--please don't do that; hard conversation follow
> makes]...
>
> > On Sunday, July 22, 2012 2:20:58 AM UTC+8, dpb wrote:
> ...
>
> >> I presume you&#39;re looking for the&#39;append&#39; option for fopen???
> ...
> > To be more clear, I simply my question with follows:
> ...
> > 2) in matlab, I have following:
> > fid2=fopen('test.dat','a'); % this file should be run parallel'
> > matlabpool open local 3
> > parfor i=1:100
> > fid1=fopen('inputpar.dat','w'); % be read by Console2.exe
> > fprintf(fid1,'%2d\n',i);
> > fclose(fid1);
> > dos('Console2.exe<inputpar.dat');
> ...
> >
> > Then I run this cell (step 2), I have the results:
> > Starting matlabpool using the 'local' configuration ... connected to
> 3 labs.
> > fortran main : 64
> > fortran main : 64
> > fortran main : 64
> > ??? Error using ==> parallel_function at 598
> > Error in ==> parallel_function>make_general_channel/channel_general
> at 894
> > Invalid file identifier. Use fopen to generate a valid file
> > identifier.
> > I've checked the content in inputpar.dat is 64.
> > I've also checked the content in fort.30 is also 64.
> > I also check
> >>> out=importdata('fort.30')
> >
> > out =
> >
> > 64
> >
> > I don't understand the error message shown above.
> > May you explain for me?
> ...
>
> My guess is the problem is one of timing--the file is locked by another
> process when you're trying to open it in all likelihood. This is a very
> difficult thing to try to get to work as you've written it.
>
> Start by
>
> [fid1,msg]=fopen('inputpar.dat','w'); % be read by Console2.exe
> if fid1==-1
> error(msg)
> end
>
> And at least you'll know fid1 isn't valid and stop and have the system
> error message.
>
> --

Subject: parfor: keep outputs with same output filename

From: Mike

Date: 24 Jul, 2012 03:48:51

Message: 11 of 15

Hi,

Thank you all.
I've modified the section to
    [fid1,msg]=fopen('inputpar.dat','wt'); % be read by Console2.exe
    if fid1==-1
       error(msg)
    end
    fprintf(fid1,'%2d\n',i);
after parfor and re-run it, then also have the same message:

Starting matlabpool using the 'local' configuration ... connected to 3 labs.
 fortran main : 46
 fortran main : 46
 fortran main : 46
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Invalid file identifier. Use fopen to generate a valid file identifier.
  
>> matlabpool close
Sending a stop signal to all the labs ... stopped.

I run it again

Starting matlabpool using the 'local' configuration ... connected to 3 labs.
forrtl: severe (24): end-of-file during read, unit 10, file D:\temp\Console2\Console2\Debug\inputpar.dat
Image PC Routine Line Source
Console2.exe 004A511A Unknown Unknown Unknown
Console2.exe 00414390 Unknown Unknown Unknown
Console2.exe 0041356A Unknown Unknown Unknown
Console2.exe 00406A06 Unknown Unknown Unknown
Console2.exe 004010B9 _MAIN__ 4 Source1.f90
Console2.exe 004ABAD3 Unknown Unknown Unknown
Console2.exe 0044FC67 Unknown Unknown Unknown
Console2.exe 0044FB3F Unknown Unknown Unknown
kernel32.dll 766D339A Unknown Unknown Unknown
ntdll.dll 77489EF2 Unknown Unknown Unknown
ntdll.dll 77489EC5 Unknown Unknown Unknown
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Invalid file identifier. Use fopen to generate a valid file identifier.
  
>> matlabpool close
Sending a stop signal to all the labs ... stopped.

I run again many times. Sometimes I have

Starting matlabpool using the 'local' configuration ... connected to 3 labs.
forrtl: severe (47): write to READONLY file, unit 30, file D:\temp\Console2\Console2\Debug\fort.30
Image PC Routine Line Source
Console2.exe 004A511A Unknown Unknown Unknown
Console2.exe 00414390 Unknown Unknown Unknown
Console2.exe 0041356A Unknown Unknown Unknown
Console2.exe 0040A06A Unknown Unknown Unknown
Console2.exe 004010FE _MAIN__ 5 Source1.f90
Console2.exe 004ABAD3 Unknown Unknown Unknown
Console2.exe 0044FC67 Unknown Unknown Unknown
Console2.exe 0044FB3F Unknown Unknown Unknown
kernel32.dll 766D339A Unknown Unknown Unknown
ntdll.dll 77489EF2 Unknown Unknown Unknown
ntdll.dll 77489EC5 Unknown Unknown Unknown
 fortran main : 64
 fortran main : 64
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Invalid file identifier. Use fopen to generate a valid file identifier.
  
>> matlabpool close
Sending a stop signal to all the labs ... stopped.

There is no message from "error(msg)".

Also, I kind of choose fixed filename: "inputpar.dat", since I wouldn't like to change that filename in FORTRAN program. My whole idea is to run FORTRAN program with input parameter file created by matlab. And run many times.

Mike

On Monday, July 23, 2012 10:34:43 PM UTC+8, dpb wrote:
> On 7/23/2012 4:45 AM, Mike wrote:
>
> ...[top posting repaired--please don't do that; hard conversation follow
> makes]...
>
> > On Sunday, July 22, 2012 2:20:58 AM UTC+8, dpb wrote:
> ...
>
> >> I presume you&#39;re looking for the&#39;append&#39; option for fopen???
> ...
> > To be more clear, I simply my question with follows:
> ...
> > 2) in matlab, I have following:
> > fid2=fopen('test.dat','a'); % this file should be run parallel'
> > matlabpool open local 3
> > parfor i=1:100
> > fid1=fopen('inputpar.dat','w'); % be read by Console2.exe
> > fprintf(fid1,'%2d\n',i);
> > fclose(fid1);
> > dos('Console2.exe<inputpar.dat');
> ...
> >
> > Then I run this cell (step 2), I have the results:
> > Starting matlabpool using the 'local' configuration ... connected to
> 3 labs.
> > fortran main : 64
> > fortran main : 64
> > fortran main : 64
> > ??? Error using ==> parallel_function at 598
> > Error in ==> parallel_function>make_general_channel/channel_general
> at 894
> > Invalid file identifier. Use fopen to generate a valid file
> > identifier.
> > I've checked the content in inputpar.dat is 64.
> > I've also checked the content in fort.30 is also 64.
> > I also check
> >>> out=importdata('fort.30')
> >
> > out =
> >
> > 64
> >
> > I don't understand the error message shown above.
> > May you explain for me?
> ...
>
> My guess is the problem is one of timing--the file is locked by another
> process when you're trying to open it in all likelihood. This is a very
> difficult thing to try to get to work as you've written it.
>
> Start by
>
> [fid1,msg]=fopen('inputpar.dat','w'); % be read by Console2.exe
> if fid1==-1
> error(msg)
> end
>
> And at least you'll know fid1 isn't valid and stop and have the system
> error message.
>
> --

Subject: parfor: keep outputs with same output filename

From: Mike

Date: 24 Jul, 2012 03:42:13

Message: 12 of 15

Hi,

Thank you all.
I've modified the section to
    [fid1,msg]=fopen('inputpar.dat','wt'); % be read by Console2.exe
    if fid1==-1
       error(msg)
    end
    fprintf(fid1,'%2d\n',i);
after parfor and re-run it, then also have the same message:

Starting matlabpool using the 'local' configuration ... connected to 3 labs.
 fortran main : 46
 fortran main : 46
 fortran main : 46
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Invalid file identifier. Use fopen to generate a valid file identifier.
  
>> matlabpool close
Sending a stop signal to all the labs ... stopped.

I run it again

Starting matlabpool using the 'local' configuration ... connected to 3 labs.
forrtl: severe (24): end-of-file during read, unit 10, file D:\temp\Console2\Console2\Debug\inputpar.dat
Image PC Routine Line Source
Console2.exe 004A511A Unknown Unknown Unknown
Console2.exe 00414390 Unknown Unknown Unknown
Console2.exe 0041356A Unknown Unknown Unknown
Console2.exe 00406A06 Unknown Unknown Unknown
Console2.exe 004010B9 _MAIN__ 4 Source1.f90
Console2.exe 004ABAD3 Unknown Unknown Unknown
Console2.exe 0044FC67 Unknown Unknown Unknown
Console2.exe 0044FB3F Unknown Unknown Unknown
kernel32.dll 766D339A Unknown Unknown Unknown
ntdll.dll 77489EF2 Unknown Unknown Unknown
ntdll.dll 77489EC5 Unknown Unknown Unknown
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Invalid file identifier. Use fopen to generate a valid file identifier.
  
>> matlabpool close
Sending a stop signal to all the labs ... stopped.

I run again many times. Sometimes I have

Starting matlabpool using the 'local' configuration ... connected to 3 labs.
forrtl: severe (47): write to READONLY file, unit 30, file D:\temp\Console2\Console2\Debug\fort.30
Image PC Routine Line Source
Console2.exe 004A511A Unknown Unknown Unknown
Console2.exe 00414390 Unknown Unknown Unknown
Console2.exe 0041356A Unknown Unknown Unknown
Console2.exe 0040A06A Unknown Unknown Unknown
Console2.exe 004010FE _MAIN__ 5 Source1.f90
Console2.exe 004ABAD3 Unknown Unknown Unknown
Console2.exe 0044FC67 Unknown Unknown Unknown
Console2.exe 0044FB3F Unknown Unknown Unknown
kernel32.dll 766D339A Unknown Unknown Unknown
ntdll.dll 77489EF2 Unknown Unknown Unknown
ntdll.dll 77489EC5 Unknown Unknown Unknown
 fortran main : 64
 fortran main : 64
??? Error using ==> parallel_function at 598
Error in ==> parallel_function>make_general_channel/channel_general at 894
Invalid file identifier. Use fopen to generate a valid file identifier.
  
>> matlabpool close
Sending a stop signal to all the labs ... stopped.

There is no message from "error(msg)".

Also, I kind of choose fixed filename: "inputpar.dat", since I wouldn't like to change that filename in FORTRAN program. My whole idea is to run FORTRAN program with input parameter file created by matlab. And run many times.

Mike

On Monday, July 23, 2012 10:34:43 PM UTC+8, dpb wrote:
> On 7/23/2012 4:45 AM, Mike wrote:
>
> ...[top posting repaired--please don't do that; hard conversation follow
> makes]...
>
> > On Sunday, July 22, 2012 2:20:58 AM UTC+8, dpb wrote:
> ...
>
> >> I presume you&#39;re looking for the&#39;append&#39; option for fopen???
> ...
> > To be more clear, I simply my question with follows:
> ...
> > 2) in matlab, I have following:
> > fid2=fopen('test.dat','a'); % this file should be run parallel'
> > matlabpool open local 3
> > parfor i=1:100
> > fid1=fopen('inputpar.dat','w'); % be read by Console2.exe
> > fprintf(fid1,'%2d\n',i);
> > fclose(fid1);
> > dos('Console2.exe<inputpar.dat');
> ...
> >
> > Then I run this cell (step 2), I have the results:
> > Starting matlabpool using the 'local' configuration ... connected to
> 3 labs.
> > fortran main : 64
> > fortran main : 64
> > fortran main : 64
> > ??? Error using ==> parallel_function at 598
> > Error in ==> parallel_function>make_general_channel/channel_general
> at 894
> > Invalid file identifier. Use fopen to generate a valid file
> > identifier.
> > I've checked the content in inputpar.dat is 64.
> > I've also checked the content in fort.30 is also 64.
> > I also check
> >>> out=importdata('fort.30')
> >
> > out =
> >
> > 64
> >
> > I don't understand the error message shown above.
> > May you explain for me?
> ...
>
> My guess is the problem is one of timing--the file is locked by another
> process when you're trying to open it in all likelihood. This is a very
> difficult thing to try to get to work as you've written it.
>
> Start by
>
> [fid1,msg]=fopen('inputpar.dat','w'); % be read by Console2.exe
> if fid1==-1
> error(msg)
> end
>
> And at least you'll know fid1 isn't valid and stop and have the system
> error message.
>
> --

On Monday, July 23, 2012 10:34:43 PM UTC+8, dpb wrote:
> On 7/23/2012 4:45 AM, Mike wrote:
>
> ...[top posting repaired--please don't do that; hard conversation follow
> makes]...
>
> > On Sunday, July 22, 2012 2:20:58 AM UTC+8, dpb wrote:
> ...
>
> >> I presume you&#39;re looking for the&#39;append&#39; option for fopen???
> ...
> > To be more clear, I simply my question with follows:
> ...
> > 2) in matlab, I have following:
> > fid2=fopen('test.dat','a'); % this file should be run parallel'
> > matlabpool open local 3
> > parfor i=1:100
> > fid1=fopen('inputpar.dat','w'); % be read by Console2.exe
> > fprintf(fid1,'%2d\n',i);
> > fclose(fid1);
> > dos('Console2.exe<inputpar.dat');
> ...
> >
> > Then I run this cell (step 2), I have the results:
> > Starting matlabpool using the 'local' configuration ... connected to
> 3 labs.
> > fortran main : 64
> > fortran main : 64
> > fortran main : 64
> > ??? Error using ==> parallel_function at 598
> > Error in ==> parallel_function>make_general_channel/channel_general
> at 894
> > Invalid file identifier. Use fopen to generate a valid file
> > identifier.
> > I've checked the content in inputpar.dat is 64.
> > I've also checked the content in fort.30 is also 64.
> > I also check
> >>> out=importdata('fort.30')
> >
> > out =
> >
> > 64
> >
> > I don't understand the error message shown above.
> > May you explain for me?
> ...
>
> My guess is the problem is one of timing--the file is locked by another
> process when you're trying to open it in all likelihood. This is a very
> difficult thing to try to get to work as you've written it.
>
> Start by
>
> [fid1,msg]=fopen('inputpar.dat','w'); % be read by Console2.exe
> if fid1==-1
> error(msg)
> end
>
> And at least you'll know fid1 isn't valid and stop and have the system
> error message.
>
> --

Subject: parfor: keep outputs with same output filename

From: Steven_Lord

Date: 24 Jul, 2012 13:24:33

Message: 13 of 15



<mike23liu@gmail.com> wrote in message
news:34476433-9ce5-4251-92cc-f786f966d8b7@googlegroups.com...
> Hi,
>
> Thank you all.
> I've modified the section to
> [fid1,msg]=fopen('inputpar.dat','wt'); % be read by Console2.exe
> if fid1==-1
> error(msg)
> end
> fprintf(fid1,'%2d\n',i);
> after parfor and re-run it, then also have the same message:

If you run this code BEFORE you enter the PARFOR loop, then when fid1 gets
passed to the workers it's just a double scalar. They didn't open the file
themselves, so they don't treat it as a file identifier. It's like you SAVEd
the identifier to a MAT-file, restarted MATLAB, then LOADed the MAT-file.

If you run this code INSIDE your PARFOR loop, you're in a race condition
situation. Don't do this.

As I said, I think your best options are to have each iteration write to a
file with a unique, iteration dependent name and either store the values
from that file into a variable inside the PARFOR loop or combine them into
one file AFTER the PARFOR loop is done.

*snip*

--
Steve Lord
slord@mathworks.com
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

Subject: parfor: keep outputs with same output filename

From: Mike

Date: 1 Aug, 2013 06:35:22

Message: 14 of 15

Hi

I encounter this problem again and review your suggestion. Allow me to reiterate the problem.
What I want to do is to run a fortran .exe file by using matlab-created input parameter file. And I'd like to run it thousand times and be fast.
Since that fortran .exe read and write both fixed filenames:'inputpar.dat' and 'fort.30' (this fortran program causes me headache so I don't want to compile/link it again), I can't create various input filenames and also expect have various output filenames. Hence, I decide to do it by creating various directory, and creating input file and output file in that directory. At last, collect all output files (fort.30). Although it works, it tends to be lengthy. I am wondering if this can be improve or better thought?

Here is:
% run FORTRAN program 'Console2.exe' with input parameter file 'inputpar.dat'
% created by matlab. And run many times parallel. Console2.exe will always read
% 'inputpar.dat' and write 'fort.30'
nimg=100;
outArr=zeros(nimg,1);
matlabpool open local 3
t = cputime;
parfor i=1:nimg
% make several dir
    dirname=['a' num2str(i)];
    mkdir(dirname);
% cd every dir and create input parameter file
    oldfolder=cd(dirname);
% input parameter file to be run by 'Console2.exe'
    [fid1,msg]=fopen('inputpar.dat','w'); % be read by Console2.exe
    fprintf(fid1,'%2d\n',i);
    fclose(fid1);
% cd dir and run fortran .exe file and save in a matrix
    % run FORTRAN program 'Console2.exe' with the above file
    dos('D:\temp\Console2\Console2\Debug\Console2.exe<inputpar.dat');
    % import output file 'fort.30' of 'Console2.exe'
    out=importdata('fort.30');
    outArr(i)=out;
    cd(oldfolder);
% remove dir
    dos(['rmdir/s/q ' dirname]);
end
matlabpool close
e = cputime-t;
fprintf('cputime: %f \n',e);
save('z.mat','outArr');

Subject: parfor: keep outputs with same output filename

From: Edric M Ellis

Date: 6 Aug, 2013 13:18:27

Message: 15 of 15

Mike <SulfateIon@gmail.com> writes:

> I encounter this problem again and review your suggestion. Allow me to
> reiterate the problem. What I want to do is to run a fortran .exe
> file by using matlab-created input parameter file. And I'd like to run
> it thousand times and be fast. Since that fortran .exe read and write
> both fixed filenames:'inputpar.dat' and 'fort.30' (this fortran
> program causes me headache so I don't want to compile/link it again),
> I can't create various input filenames and also expect have various
> output filenames. Hence, I decide to do it by creating various
> directory, and creating input file and output file in that
> directory. At last, collect all output files (fort.30). Although it
> works, it tends to be lengthy. I am wondering if this can be improve
> or better thought?

<... snip ...>

Do you know why it doesn't perform as well as it should? Have you
compared the performance when running serially vs. in parallel? Some
data-intensive operations will never obtain "perfect" linear speedup as
you add workers if the memory bandwidth is the bottleneck. The only
suggestion that I have, but it may well be irrelevant, is that you could
avoid making one directory per iteration of the PARFOR loop by doing
something like this:

baseDir = 'c:\some\path';
parfor idx = 1:N
  myTask = getCurrentTask();
  myDir = fullfile(baseDir, sprintf('workdir_%d', myTask.ID));
  ...
end

This would guarantee that each worker re-uses the same directory, and
that no workers clash with each other.

Cheers,

Edric.

Tags for this Thread

No tags are associated with this thread.

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us