View License

Download apps, toolboxes, and other File Exchange content using Add-On Explorer in MATLAB.

» Watch video

Highlights from
Progress monitor (progress bar) that works with parfor

4.87097
4.9 | 32 ratings Rate this file 113 Downloads (last 30 days) File Size: 2.21 KB File ID: #32101 Version: 1.2

Progress monitor (progress bar) that works with parfor

by

Jeremy (view profile)

  • 1 file
  • 113 downloads
  • 4.87097

 

06 Jul 2011 (Updated )

Displays the percentage of parfor iterations which have completed.

| Watch this File

File Information
Description

Monitoring the progress of a long-running parfor loop is not straightforward because the workers cannot communicate. This function gets around that limitation by storing the progress in a text file.

Example usage:

N = 100;
parfor_progress(N); % Initialize
parfor i=1:N
    pause(rand); % Replace with real code
    parfor_progress; % Count
end
parfor_progress(0); % Clean up

Acknowledgements

This file inspired Progress Bar For Matlab Loops (Incl. Parfor), Timed Progress Bar, Parfor Progress Class, Parfor Progressbar, and Parfor Progressbar V1.

Required Products Parallel Computing Toolbox
MATLAB release MATLAB 7.10 (R2010a)
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Comments and Ratings (44)
09 Dec 2016 Marcin Konowalczyk

10 Oct 2016 Xinxin Liu

03 Aug 2016 Kouichi

Kouichi (view profile)

Useful tool! Needed to replace nargchk with narinchk in order to use this for R2016b.

narginchk(0,1)

23 Jun 2016 Nalini Singh

31 May 2016 Tao Yu

Tao Yu (view profile)

07 Apr 2016 Phlip

Phlip (view profile)

Nice contribution. However, as Timo already found out, while fitting a huge amount of data I noticed that the progress monitor gets extremely slow for very long loops. In the beginning it is not so bad but with every loop iteration the call to 'fscanf' gets slower and slower.

I fixed this by reading & writing to a simple binary file (i am writing the expected loop length to the first 4 bytes and the current progress to the next 4 bytes - if you think that uint32 is not enough, feel free to use uint64).
In addition, the filename is now stored in the folder determined by matlab's 'tempname'-function (less filesystem clutter if the delete fails & e.g. tempdir on an ssd is much faster than working dir somewhere on the network).

if nargin < 1
N = -1;
end

percent = 0;
w = 50; % Width of progress bar

if N > 0
f = fopen(fullfile(tempdir,'parfor_progress.bin'), 'w');
if f<0
error('Do you have write permissions for %s?', pwd);
end
fwrite(f,N,'uint32');
fwrite(f,0,'uint32');
fclose(f);

if nargout == 0
disp([' 0%[>', repmat(' ', 1, w), ']']);
end
elseif N == 0
delete(fullfile(tempdir,'parfor_progress.bin'));
percent = 100;

if nargout == 0
disp([repmat(char(8), 1, (w+9)), char(10), '100%[', repmat('=', 1, w+1), ']']);
end
else
fname = fullfile(tempdir,'parfor_progress.bin');
if ~exist(fname, 'file')
error('parfor_progress.bin not found. Run PARFOR_PROGRESS(N) before PARFOR_PROGRESS to initialize parfor_progress.bin.');
end

f = fopen(fname, 'r+');
A = fread(f,2,'uint32');
todo = A(1);
progress = A(2) + 1;
fseek(f, 4, 'bof');
fwrite(f,progress,'uint32');
fclose(f);

percent = progress/todo * 100;

if nargout == 0
perc = sprintf('%3.0f%%', percent); % 4 characters wide, percentage
disp([repmat(char(8), 1, (w+9)), char(10), perc, '[', repmat('=', 1, round(percent*w/100)), '>', repmat(' ', 1, w - round(percent*w/100)), ']']);
end
end

Comment only
07 Apr 2016 Phlip

Phlip (view profile)

22 Mar 2016 dong yun

Jeremy,Good idea!But I have a probelm in runing a cluster paralle job,there is no display on the command line window.Due to the different workers created in the different computers,how let each work excute the disp function and print the content on the client which is the host of the cluster.

15 Mar 2016 zzzhhh

zzzhhh (view profile)

Two problems: 1) some matlab programs change working directory while running. The Progress monitor will report error because it only check the "current" working directory. A solution is to use some absolute path such as ~/parfor_progress.txt. 2) Once parfor_progress is called it will output, but sometimes the progress is not large enough to update the drawing of the progress bar. The thing is: the user has no idea of the "last progress" to tell if the current progress is large enough to make a new drawing. This should be maintained by the Progress monitor code, using, say, the same trick of file opening.

Comment only
17 Jan 2016 Asdf asdf

18 Nov 2015 Sharol Schmidt

10 Oct 2015 Carlos Vladimir Rodríguez Caballero

In Spanish we would say "está chulísimo".... i.e. it is so cool =)

09 Oct 2015 Dong-Gyu Jang

Dong-Gyu Jang (view profile)

It is not really like waitbar in GUI but progress is shown in Matlab command window.

Comment only
26 Sep 2015 Christophe Steiger

Simple, efficient, thanks!

One slight problem: when 100% of the CPU is used, the refresh of the progress bar is not perfectly smooth :)

08 Sep 2015 Jeremy

Jeremy (view profile)

  • 1 file
  • 113 downloads
  • 4.87097

Timo - if each iteration of the loop is really fast, then the overhead of parfor_progress can be a lot. If each iteration is slow, then it's not. If it's slowing you down a lot, either use another solution or restructure your loop so each iteration does more work.

Comment only
07 Sep 2015 Timo Korthals

Nice work, but I mentioned that if I am using your script, the parfor loop is as slow as the common for loop.
for-loop: 256s
parfor-loop: 46s
parfor-loop with parfor_progress_: 301
I am using r2015a.

Does anyone agree?

Comment only
16 Jul 2015 RN

RN (view profile)

Jeremy: Thanks. I ended up just doing something similar to http://stackoverflow.com/a/27378993/3490931
and if over 100 sims just partition it and print out every n iterations. Gets the job done.

Comment only
15 Jul 2015 Jeremy

Jeremy (view profile)

  • 1 file
  • 113 downloads
  • 4.87097

RN: It would probably work on a distributed system if you edited it so that parfor_progress.txt was on a shared drive, but I haven't tried it.

Comment only
15 Jul 2015 RN

RN (view profile)

Any way to do something like this with a distributed computing server like when using MathWorks Cloud Center?

Comment only
29 Jun 2015 Nils

Nils (view profile)

Neat helper!

23 Apr 2015 Mike Adams

very Useful. Thank you for this!

11 Mar 2015 Weibo

Weibo (view profile)

05 Feb 2015 Adrien

Adrien (view profile)

Very efficient, works easily and perfectly. Thanks!

20 Jan 2015 Reza Farrahi Moghaddam

02 Nov 2014 Bauyrzhan Aubakir

I would like to say: Thank you very much for such a nice script

02 Nov 2014 AJ Rivera

31 Oct 2014 Florian

31 Oct 2014 Vicky P

Jeremy: First, thanks for writing this handy script. It is very useful.

I also found display issues in the linux terminal, which Ashin and Alessandro describe. However, I found a solution pretty quickly though I'm not an expert user of MATLAB at all.

I changed only 3 lines, changing all disp() calls to fprintf().

The char(8) trick works for fprintf() as well and is actually better for linux terminals.

Hope it is useful.

10 Oct 2014 Alessandro Corbetta

28 May 2014 James McIntosh

16 May 2014 Jeremy

Jeremy (view profile)

  • 1 file
  • 113 downloads
  • 4.87097

Erik: Depends how slow each iteration of your loop is. If each iteration takes a while to run, this won't impact execution time. If each iteration is very fast, it could. You'll have to benchmark for more specifics, I've only used it on slow problems. If necessary, a quick fix to speed this up would be to change the path of parfor_progress.txt to be on a ramdisk.

Comment only
15 May 2014 Erik

Erik (view profile)

Nice work. Just wondering, what impact does this have on the execution speed of parfor?

14 Oct 2013 Hassan Naseri

06 Oct 2013 Jesse Coombs

Would it be possible to get a version that instead prints the %complete to a txt file for use in monitoring jobs on remote/server pc's?

Comment only
02 Jul 2013 Alessandro Corbetta

Excellent work!
Although it works badly on linux -nodesktop mode. In this case, the older (and less complete) bars remain showed in the console.
As a consequence, the screen is full of more or less complete version of the bar in few seconds.

10 Jun 2013 Antoine Liutkus

Antoine Liutkus (view profile)

02 Jun 2013 German

German (view profile)

Thanks for the file. It works!
As Timo suggested, if there are several instances of MATLAB running (e.g., multi-core server), it won't work properly.
It can be easily solved by adding a randon sequence to the file name.

function percent = parfor_progress(N)
%PARFOR_PROGRESS Progress monitor (progress bar) that works with parfor.
% PARFOR_PROGRESS works by creating a file called parfor_progress.txt in
% your working directory, and then keeping track of the parfor loop's
% progress within that file. This workaround is necessary because parfor
% workers cannot communicate with one another so there is no simple way
% to know which iterations have finished and which haven't.
%
% PARFOR_PROGRESS(N) initializes the progress monitor for a set of N
% upcoming calculations.
%
% PARFOR_PROGRESS updates the progress inside your parfor loop and
% displays an updated progress bar.
%
% PARFOR_PROGRESS(0) deletes parfor_progress.txt and finalizes progress
% bar.
%
% To suppress output from any of these functions, just ask for a return
% variable from the function calls, like PERCENT = PARFOR_PROGRESS which
% returns the percentage of completion.
%
% Example:
%
% N = 100;
% parfor_progress(N);
% parfor i=1:N
% pause(rand); % Replace with real code
% parfor_progress;
% end
% parfor_progress(0);
%
% See also PARFOR.

% By Jeremy Scheff - jdscheff@gmail.com - http://www.jeremyscheff.com/
error(nargchk(0, 1, nargin, 'struct'));

persistent fname;

if nargin < 1
N = -1;
end

percent = 0;
w = 50; % Width of progress bar

if N > 0
rndseq = round(10^6*rand(1));
fname = strcat('parfor_progress_',num2str(rndseq),'.txt');

f = fopen(fname, 'w');
if f<0
error('Do you have write permissions for %s?', pwd);
end
fprintf(f, '%d\n', N); % Save N at the top of progress.txt
fclose(f);

if nargout == 0
disp([' 0%[>', repmat(' ', 1, w), ']']);
end
elseif N == 0
delete(fname);
percent = 100;

if nargout == 0
disp([repmat(char(8), 1, (w+9)), char(10), '100%[', repmat('=', 1, w+1), ']']);
end
else
if ~exist(fname, 'file')
error([fname,' not found. Run PARFOR_PROGRESS(N) before PARFOR_PROGRESS to initialize ',fname]);
end

f = fopen(fname, 'a');
fprintf(f, '1\n');
fclose(f);

f = fopen(fname, 'r');
progress = fscanf(f, '%d');
fclose(f);
percent = (length(progress)-1)/progress(1)*100;

if nargout == 0
perc = sprintf('%3.0f%%', percent); % 4 characters wide, percentage
disp([repmat(char(8), 1, (w+9)), char(10), perc, '[', repmat('=', 1, round(percent*w/100)), '>', repmat(' ', 1, w - round(percent*w/100)), ']']);
end
end

14 May 2013 Timo

Timo (view profile)

This is a great function you've implemented, but there is one point you could advance.

Here is the problem:
If you execute any MATLAB script more than once in the same folder, and the script contains your function, they will end up in an error.
The point is, that the filename is always the same which is created by your function.
So you'll have more than one function which tries to work on the parfor_progress file.

Maybe you should choose an arbitrary filename-id attached to every parfor_progress file.

But anyway, great work!

Comment only
10 Mar 2013 Michael

08 Aug 2012 Donald Bucci

This is real wonderful, however I had some issues running this on a nodisplay ubuntu server terminal running matlab (ascii 8 for backspace doesn't quite produce the right output).

If I were you, I'd rewrite everything in here from DISP() functions into FPRINTF() functions. And instead of using char(8) and char(10) for your backspaces and carriage returns, use the breakout switches \b and \n respectively. I did that and it fixed any issues I've ever had with the progress bar being duplicated across multiple lines (as opposed to being overwritten when updated).

13 Jul 2012 Pantelis

05 Mar 2012 Jeremy

Jeremy (view profile)

  • 1 file
  • 113 downloads
  • 4.87097

Ashin,

I develop on Linux and I've never seen this problem. So it's likely that it is not a Linux problem in general, but it's specific to your particular setup. Maybe something to do with the character encoding on your system?

As a test, run this and see what it outputs:

disp(['aaa', char(8)])

The output should be just aa, because char(8) is a backspace. This is the technique I used in parfor_progress to delete the previous line, so maybe the backspace isn't working on your Linux machine for some reason?

Or does anyone know of a more universally portable way of doing this?

Comment only
04 Mar 2012 Ashin Mukherjee

I'm trying to use this in a linux environment. The resulting progress bar looks something like this

0% [ ]
2% [= ]
4% [== ]
.
.
.

That is it creates a new-line every time which is slightly annoying. But works smoothly in windows. Was wondering whether there is an easy fix to this problem.

Thanks
Ashin

04 Mar 2012 Mike

Mike (view profile)

Thanks - does exactly what it's supposed to, stops you scratching your head and wondering if it's nearly finished...

Updates
07 Jul 2011 1.1

Slightly simplified the function.

25 Oct 2011 1.2

It now displays a real (and pretty) progress bar.

Contact us