File Exchange

image thumbnail

TIMEIT Benchmarking Function

version 1.4.0.1 (4.54 KB) by

TIMEIT.M measures the time required to call a user-specified function

18 Downloads

Updated

View License

Editor's Note: This file was selected as MATLAB Central Pick of the Week

T = TIMEIT(F) measures the time (in seconds) required to run F, which is a function handle.
TIMEIT handles automatically the usual benchmarking procedures of "warming up" F, figuring out how many times to repeat F in a timing loop, etc. TIMEIT uses a median to form a reasonably robust time estimate.

UPDATED 16-May-2010: New option to specify number of output arguments to call F with. Overhead measurements now cached so most timings run faster.

UPDATED 31-Dec-2008: More accurate when timing very fast functions; warns you when the reported time might be affected by time-measurement overhead; calls F fewer times when F takes more than a few seconds to run.

Comments and Ratings (14)

qingpeng luo

Geert, thanks for your feedback. This function is intended for simple benchmarking. Also, I am skeptical of the value of reporting the standard deviation, given the nature of the outliers that crop up during timing experiments on typical computer systems.

Steve Eddins

Steve Eddins (view profile)

Geert, thanks for your feedback. This function is intended for simple benchmarking. Also, I am skeptical of the value of reporting the standard deviation, given the nature of the outliers that crop up during timing experiments on typical computer systems.

Thanks for the excellent functionality. However, I am also of the opinion the standard deviation is crucial for anything that goes beyond simple benchmarking.

Jan Motl

Jan Motl (view profile)

It would be great if the function returned also confidence interval.

Steve Eddins

Steve Eddins (view profile)

Andy, I don't fully understand your question. What do you mean by "warm in the time," for example? But I would guess that you're looking for something like this:

x = zeros(1024, 1024);
timeit(@() fft2(x))

James

James (view profile)

How do i use timeit to calculate iterartion times to run a for loop instead of using tic toc so i don't have the warm in the time? E.g.

x = zeros(1024,1024);
tic
for i = 1:200;
y = fft2(x)
end
toc

Steve Hoelzer

Steve Hoelzer (view profile)

Joao Henriques

This is extremely useful, but since it requires a function definition, you must either use lambdas, which don't allow procedural code (ie, a list of statements), define a local function, which means you have to turn your script into a function file (sometimes not desirable when dealing with workspace variables), or create a new file (lots of throwaway files with simple routines).

I wonder if it would be possible to add the following functionality (either in the same function or through a different one) :

while timeit %returns true when a new measurement is needed
  ... my procedural code ...
  ... multiple lines, will run as many times as timeit() sees fit ...
end
result = timeit; %next call: return result. reset state.

As you can see, this sort of usage would be tremendously useful to just profile a piece of code with no extra details -- just wrap it in a loop. Of course, it's only fair if you conclude it's not worth the additional effort :)

Steve Eddins

Steve Eddins (view profile)

Paolo, thanks for the suggestion. I'll look into it.

Paolo de Leva

Paolo de Leva (view profile)

A suggestion to improve accuracy when you measure small execution times. I believe it would be useful to subtract from median(times) the time needed for the inner iterations (“offset”). Here is the code:

[…omissis…]
offsets = zeros(num_outer_iterations, 1);
times = offsets;
for k = 1:num_outer_iterations
    t1 = tic;
    for p = 1:num_inner_iterations
    end
    offsets(k) = toc(t1);
end

for k = 1:num_outer_iterations
    t1 = tic;
    for p = 1:num_inner_iterations
        [outputs{:}] = f();
    end
    times(k) = toc(t1);
end
 
t = (median(times-offsets)) / num_inner_iterations;
[…omissis…]

This offset is not always negligible, especially when you want to time the execution of very quick commands or functions. Notice that the value of each element of variable “offsets” also include the time needed to execute

t1 = tic; times(k) = toc(t1);

On my system, this is very small (1.7321e-005 s), but not negligible when testing the execution of very quick commands, such as RESHAPE.

Paolo de Leva

Steve Eddins

Steve Eddins (view profile)

Anna - thanks. I just updated the file to fix the nargout problem.

Anna Nowak

Small remark - nargout parameter is sometimes less than 0. I had -1 result when there was no output arguments from function.

In case of problems with error "Too many output argument" juz change line :
num_outputs = ~(nargout(f) == 0);
to
num_outputs = ~(nargout(f) <= 0);

Still excellent file!

John D'Errico

Ok, this does nothing exotic. But what it does it does well. Its a simple way to time your code without worrying about some of the extra bookkeeping, like warming up the codes to make sure you get an accurate assessment of the time required, plus taking the median of the time over multiple runs.

The help and the code is as good as you would expect from this source.

Updates

1.4.0.1

Updated license

1.4

New option to specify number of output arguments to call F with. Overhead measurements now cached so most timings run faster.

1.2

More accurate when timing very fast functions; warns you when the reported time might be affected by time-measurement overhead; calls F fewer times when F takes more than a few seconds to run.

1.1

Fixed nargout problem for timing anonymous function handles.

MATLAB Release
MATLAB 7.5 (R2007b)

Download apps, toolboxes, and other File Exchange content using Add-On Explorer in MATLAB.

» Watch video