Thread Subject:

First (puzzling) impressions on R2012b regarding computational performances

Hi all,
following a previous comparison on performances of new Matlab releases vs. old matlab releases (http://www.mathworks.it/matlabcentral/newsreader/view_thread/305954#833788), I have carried out a few tests regarding the new matlab release R2012b.

The results are quite interesting because they show, in some cases, a significant decrease of performances, while in other cases a significant increase of performances.

Test code 1:

%----------------
N=10;
tcpu=NaN*zeros(1,N);
for jjj=1:N,
    tic;
    [t,y]=ode23(@vdp1,[0 2000],[2 0]);
    tcpu(jjj)=toc;
end;
mean(tcpu)
%----------------

Result from mean(tcpu):
R2009bSP1: 0.8550s
R2011a: 0.9647s
R2012b: 1.1480s
Ratio R2012b/R2009bSP1: 1.3427

i.e., with respect to ode23, R2012b seems to be more than 30% slower than R2009bSP1

Let's try something else.

Test code 2:

%----------------
N=100;
tcpu=NaN*zeros(1,N);
for jjj=1:N,
    tic;
    A=rand(1000,1000);
    B=rand(1000,1);
    x=A\B;
    tcpu(jjj)=toc;
end;
mean(tcpu)
%----------------

Result from mean(tcpu):
R2009bSP1: 0.1151s
R2011a: 0.1010s
R2012b: 0.1000s
Ratio R2012b/R2009bSP1: 0.8688

i.e., in this case, R2012b seems to be almost 15% faster than R2009bSP1, but it has more or less the same performance as R2011a

Test code 3:

%----------------
N=1000000;
tic
for jjj=1:N,
    p=rand(1,10);
    x=rand(1);
    polyval(p,x);
end;
toc
%----------------

Result from total calculation time:
R2009bSP1: 13.266702s
R2011a: 18.244994s
R2012b: 20.665393s
Ratio R2012b/R2009bSP1: 1.5577

i.e. R2012b seems to be more than 55% slower than R2009bSP1 (and about 13% slower than R2011a) !!!!!!

Test code 4: a complex simulation code with many functions calls
R2009bSP1: 44.6090s
R2011a: 48.9370s
R2012b: 26.9690s
Ratio R2012b/R2009bSP1: 0.6046

For this complex simulation tool R2012b is 40% faster than R2009bSP1 !

In general, from the initial experience, I have noticed R2012b to be less responsive than previous versions, particularly regading the time it takes to plot graphics.

Moreover, the fact that now R2012b is case sensitive also with respect to the functions is already creating me several troubles with old codes...it was something anticipated, but it remains a trouble.

Do you find something similar?

Bye,
Gabriele

Ups, actually case sensitiveness for functions was there since R2011b...but I did not upgrade after 2011a ...

"Gabriele " <ruga.ANTI@SPAM.libero.it> wrote in message <k32t8a$972$1@newscl01ah.mathworks.com>...
> Hi all,
> following a previous comparison on performances of new Matlab releases vs. old matlab releases (http://www.mathworks.it/matlabcentral/newsreader/view_thread/305954#833788), I have carried out a few tests regarding the new matlab release R2012b.
>
> The results are quite interesting because they show, in some cases, a significant decrease of performances, while in other cases a significant increase of performances.
>
> Test code 1:
>
> %----------------
> N=10;
> tcpu=NaN*zeros(1,N);
> for jjj=1:N,
> tic;
> [t,y]=ode23(@vdp1,[0 2000],[2 0]);
> tcpu(jjj)=toc;
> end;
> mean(tcpu)
> %----------------
>
> Result from mean(tcpu):
> R2009bSP1: 0.8550s
> R2011a: 0.9647s
> R2012b: 1.1480s
> Ratio R2012b/R2009bSP1: 1.3427
>
> i.e., with respect to ode23, R2012b seems to be more than 30% slower than R2009bSP1
>
> Let's try something else.
>
> Test code 2:
>
> %----------------
> N=100;
> tcpu=NaN*zeros(1,N);
> for jjj=1:N,
> tic;
> A=rand(1000,1000);
> B=rand(1000,1);
> x=A\B;
> tcpu(jjj)=toc;
> end;
> mean(tcpu)
> %----------------
>
> Result from mean(tcpu):
> R2009bSP1: 0.1151s
> R2011a: 0.1010s
> R2012b: 0.1000s
> Ratio R2012b/R2009bSP1: 0.8688
>
> i.e., in this case, R2012b seems to be almost 15% faster than R2009bSP1, but it has more or less the same performance as R2011a
>
> Test code 3:
>
> %----------------
> N=1000000;
> tic
> for jjj=1:N,
> p=rand(1,10);
> x=rand(1);
> polyval(p,x);
> end;
> toc
> %----------------
>
> Result from total calculation time:
> R2009bSP1: 13.266702s
> R2011a: 18.244994s
> R2012b: 20.665393s
> Ratio R2012b/R2009bSP1: 1.5577
>
> i.e. R2012b seems to be more than 55% slower than R2009bSP1 (and about 13% slower than R2011a) !!!!!!
>
> Test code 4: a complex simulation code with many functions calls
> R2009bSP1: 44.6090s
> R2011a: 48.9370s
> R2012b: 26.9690s
> Ratio R2012b/R2009bSP1: 0.6046
>
> For this complex simulation tool R2012b is 40% faster than R2009bSP1 !
>
> In general, from the initial experience, I have noticed R2012b to be less responsive than previous versions, particularly regading the time it takes to plot graphics.
>
> Moreover, the fact that now R2012b is case sensitive also with respect to the functions is already creating me several troubles with old codes...it was something anticipated, but it remains a trouble.
>
> Do you find something similar?
>
> Bye,
> Gabriele

"Gabriele " <ruga.ANTI@SPAM.libero.it> wrote in message <k32t8a$972$1@newscl01ah.mathworks.com>...
> Hi all,
> following a previous comparison on performances of new Matlab releases vs. old matlab releases (http://www.mathworks.it/matlabcentral/newsreader/view_thread/305954#833788), I have carried out a few tests regarding the new matlab release R2012b.
>
> The results are quite interesting because they show, in some cases, a significant decrease of performances, while in other cases a significant increase of performances.
==============

And you executed these test codes inside a function file? A script file? From the command line? It would affect whether the JIT was in play.

In any case, these kinds of slow downs are not uncommon. If a new release puts in more error checking into certain functions, things will obviously slow down.

"Matt J" wrote in message <k33fug$8lt$1@newscl01ah.mathworks.com>...
> "Gabriele " <ruga.ANTI@SPAM.libero.it> wrote in message <k32t8a$972$1@newscl01ah.mathworks.com>...
> > Hi all,
> > following a previous comparison on performances of new Matlab releases vs. old matlab releases (http://www.mathworks.it/matlabcentral/newsreader/view_thread/305954#833788), I have carried out a few tests regarding the new matlab release R2012b.
> >
> > The results are quite interesting because they show, in some cases, a significant decrease of performances, while in other cases a significant increase of performances.
> ==============
>
> And you executed these test codes inside a function file? A script file? From the command line? It would affect whether the JIT was in play.
>
> In any case, these kinds of slow downs are not uncommon. If a new release puts in more error checking into certain functions, things will obviously slow down.

Hello Matt,
yes, some additional error checking (if actually any) could reduce performances, but it is important to keep a reasonable balance.

Anyway, regarding the details.
The tests above have been carried out by running a cell from a script, and this I suppose should be equivalent to running from the command prompt. Following your suggestion I have run some test again, but embedding the code into a function.

I considered only a test where R2012b was significantly slower than R2009bSP1.

Test code 3 - function version:
%-------------------
function dummy_performance
N=1000000;
tic
for jjj=1:N,
    p=rand(1,10);
    x=rand(1);
    polyval(p,x);
end;
toc
%-------------------
Result from total calculation time:
R2009bSP1: 11.655280s (13.266702s running a cell)
R2011a: 14.672747s (18.244994s running a cell)
R2012b: 16.878261s (20.665393s running a cell)
Ratio R2012b/R2009bSP1: 1.4481 (1.5577 running a cell)

It seems the overall results are basically the same: R2012b being significantly slower.

You mentioned possible error checks as a reason.
However, comparing polyval in R2012b ad in R2009bSP1 I see some differences in the error throwing sections (which in any case do not affect the present test) and a small difference at the end of the code:
delta = repmat(Inf,size(e));
has become
delta = Inf(size(e));

but also this is not affecting the results in the present case...
it seems therefore there could be something else.

Bye,
Gabriele

"Gabriele " <ruga.ANTI@SPAM.libero.it> wrote in message <k34830$q1l$1@newscl01ah.mathworks.com>...
>
> Hello Matt,
> yes, some additional error checking (if actually any) could reduce performances, but it is important to keep a reasonable balance.
=================

Yes, but how can we ever know, simply based on speed differences, whether the balance is unreasonable? A 50% change may be an inconvenient change in performance, but on the other hand, who knows what kind of problems the missing error checking was found to cause?


> I considered only a test where R2012b was significantly slower than R2009bSP1.
==============

Personally, I think your test of ode23 was the more meaningful test. I can't see polyval being speed-critical in most applications and can see the MATLAB developers thinking the same way when evaluating compromises for a new release. If I really do need to evaluate 100000 polynomials, I would do it in a vectorized way, which your for-loop test doesn't capture. If on the other hand, I need to evaluate only 1 polynomial, I of course don't care about a 50% difference.

 

> but also this is not affecting the results in the present case...
> it seems therefore there could be something else.
================

Also, before we conclude that it is actually POLYVAL that has slowed down, I think you should redo the test with only polyval contributing to the measured time. In other words, I would remove the statements

    p=rand(1,10);
    x=rand(1);

from the loop and the tic...toc code block and just call them once initially.
Finally, most of the work in POLYVAL is being done by FILTER (for this combination of input types). It might be even better to test that in isolation, rather than polyval.

Hello Matt,
my post is not intended to be a specific test of polyval performances (indeed I did several, random actually, test). Moreover it is not intended to target a specific function. Following a past post, where I saw some deterioration of performances moving from old to new realeases, I found this deterioration, although partial, also in the case of R2012b. So my post is general in nature. For this reason I'll not go into details o whether to use or not use rand in the test, etc. etc.

To make another test, and to show things are in the same direction, I used ode45 inside a function:

%----------
function dummy_performance_ode
N=10;
tic
for jjj=1:N,
    [t,y]=ode45(@vdp1,[0 2000],[jjj/2 jjj]);
end;
toc
%-----------

R2009bSP1: 5.863999s
R2011a: 6.872794s
R2012b: 8.544259s
Ratio R2012b/R2009bSP1: 1.4571

The trend is pretty clear. So, we are not speaking here of a few percent. If a control check is eating 46% of computational time, this is clearly unreasonable. But my feeling is far from being a mater of control check. Looking at profiling, the whole computational time is basically due to the function vdp1, and its single line:

dydt = [y(2); (1-y(1)^2)*y(2)-y(1)];

This means that it is not error checking, it is a core matter...

It would be nice to have some feedback/explaation directly from MW....

Bye,
Gabriele

"Matt J" wrote in message <k34h5k$oqi$1@newscl01ah.mathworks.com>...
> "Gabriele " <ruga.ANTI@SPAM.libero.it> wrote in message <k34830$q1l$1@newscl01ah.mathworks.com>...
> >
> > Hello Matt,
> > yes, some additional error checking (if actually any) could reduce performances, but it is important to keep a reasonable balance.
> =================
>
> Yes, but how can we ever know, simply based on speed differences, whether the balance is unreasonable? A 50% change may be an inconvenient change in performance, but on the other hand, who knows what kind of problems the missing error checking was found to cause?
>
>
> > I considered only a test where R2012b was significantly slower than R2009bSP1.
> ==============
>
> Personally, I think your test of ode23 was the more meaningful test. I can't see polyval being speed-critical in most applications and can see the MATLAB developers thinking the same way when evaluating compromises for a new release. If I really do need to evaluate 100000 polynomials, I would do it in a vectorized way, which your for-loop test doesn't capture. If on the other hand, I need to evaluate only 1 polynomial, I of course don't care about a 50% difference.
>
>
>
> > but also this is not affecting the results in the present case...
> > it seems therefore there could be something else.
> ================
>
> Also, before we conclude that it is actually POLYVAL that has slowed down, I think you should redo the test with only polyval contributing to the measured time. In other words, I would remove the statements
>
> p=rand(1,10);
> x=rand(1);
>
> from the loop and the tic...toc code block and just call them once initially.
> Finally, most of the work in POLYVAL is being done by FILTER (for this combination of input types). It might be even better to test that in isolation, rather than polyval.

"Gabriele " <ruga.ANTI@SPAM.libero.it> wrote in message <k34pjk$m6u$1@newscl01ah.mathworks.com>...
>
> But my feeling is far from being a mater of control check. Looking at profiling, the whole computational time is basically due to the function vdp1, and its single line:
>
> dydt = [y(2); (1-y(1)^2)*y(2)-y(1)];
>
> This means that it is not error checking, it is a core matter...
>
> It would be nice to have some feedback/explaation directly from MW....
==============

That is weird. Why don't you send your test result to Tech Support?

Please do post any new developments here (e.g., any non-trivial MathWorks answer) - a lot of people are interested in this topic, not just Gabriele and Matt...

Yair Altman
http://UndocumentedMatlab.com
 

Hello Yair,
I hope someone from TMW will reply in the newsgroup giving some futher information.

Did you try some test yourself? Did you observe a similar trend?

Bye,
Gabriele

"Yair Altman" wrote in message <k3ai09$6ml$1@newscl01ah.mathworks.com>...
> Please do post any new developments here (e.g., any non-trivial MathWorks answer) - a lot of people are interested in this topic, not just Gabriele and Matt...
>
> Yair Altman
> http://UndocumentedMatlab.com
>

"Gabriele " <ruga.ANTI@SPAM.libero.it> wrote in message <k34pjk$m6u$1@newscl01ah.mathworks.com>...
>
> %----------
> function dummy_performance_ode
> N=10;
> tic
> for jjj=1:N,
> [t,y]=ode45(@vdp1,[0 2000],[jjj/2 jjj]);
> end;
> toc
> %-----------
>
> R2009bSP1: 5.863999s
> R2011a: 6.872794s
> R2012b: 8.544259s
> Ratio R2012b/R2009bSP1: 1.4571

After trying R2012b for a few days, I decided to move one step backward, so I installed R2012a (waiting for R2013a, maybe...)

Anyway, with R2012a I have the following result:
R2012a: 7.050889s
i.e. between 2011a and 2012b.

I have tried again running the test on R2012b, without any other matlab session open, and I get:
R2012b: 7.758444s
better than before (windows is magic!) but still the worst result.

It seems the downard trend is confirmed, at least on my M4300 with WinXP-SP3 32bit.

Do you find similar results? (maybe it is just a problem of my specific hardware/software configuation).

In ay case it would be interesting and useful if TMW would issue, together with the new release, some "standard" simple speed checks like the above to compare the improvement (or decrease) of performance between releases...just to be iformed and decide whether it is worth to move to a new version or not...

The usual bench(n) is nice for machine-to-machine comparison, but, as far as I know, not for release-to-release comparison

Bye,
Gabriele

"Gabriele " <ruga.ANTI@SPAM.libero.it> wrote in message <k3cmrb$2f7$1@newscl01ah.mathworks.com>...
>
> After trying R2012b for a few days, I decided to move one step backward, so I installed R2012a (waiting for R2013a, maybe...)
>
> Anyway, with R2012a I have the following result:
> R2012a: 7.050889s
> i.e. between 2011a and 2012b.
>
> I have tried again running the test on R2012b, without any other matlab session open, and I get:
> R2012b: 7.758444s
> better than before (windows is magic!) but still the worst result.
>
> It seems the downard trend is confirmed, at least on my M4300 with WinXP-SP3 32bit.
>
> Do you find similar results? (maybe it is just a problem of my specific hardware/software configuation).
==============

I have tested with R2011a, R2011b, and R2012a and I get virtually identical timing results for all three. This differs from your findings. You seem to report a drop from R2011a to R2012a. So, it could indeed be a platform dependent thing.

I'm in the process of installing R2011b.

"Matt J" wrote in message <k3co4k$7i5$1@newscl01ah.mathworks.com>...
> "Gabriele " <ruga.ANTI@SPAM.libero.it> wrote in message <k3cmrb$2f7$1@newscl01ah.mathworks.com>...
> >
> > After trying R2012b for a few days, I decided to move one step backward, so I installed R2012a (waiting for R2013a, maybe...)
> >
> > Anyway, with R2012a I have the following result:
> > R2012a: 7.050889s
> > i.e. between 2011a and 2012b.
> >
> > I have tried again running the test on R2012b, without any other matlab session open, and I get:
> > R2012b: 7.758444s
> > better than before (windows is magic!) but still the worst result.
> >
> > It seems the downard trend is confirmed, at least on my M4300 with WinXP-SP3 32bit.
> >
> > Do you find similar results? (maybe it is just a problem of my specific hardware/software configuation).
> ==============
>
> I have tested with R2011a, R2011b, and R2012a and I get virtually identical timing results for all three. This differs from your findings. You seem to report a drop from R2011a to R2012a. So, it could indeed be a platform dependent thing.
>
> I'm in the process of installing R2011b.

Hi Matt,
this is an interesting outcome! Are you working on a 32bit or 64bit OS?
Did you check exactly the same function as I did?

Bye,
Gabriele

"Gabriele " <ruga.ANTI@SPAM.libero.it> wrote in message <k3cpn8$dnf$1@newscl01ah.mathworks.com>...
>
> Hi Matt,
> this is an interesting outcome! Are you working on a 32bit or 64bit OS?
> Did you check exactly the same function as I did?
==============

I ran the ode45 test.

I'm running on a 64-bit OS, Dell Precision T7500, Intel Xeon X5680@3.3GHz dual hexacore.
It might be worth mentioning that my version of R2012a, though, is 32-bit while both my R2011a,b versions were 64-bit. All results were virtually identical, so I'm doubtful that was a factor in the comparison.

"Matt J" wrote in message <k3co4k$7i5$1@newscl01ah.mathworks.com>...
>
>
> I'm in the process of installing R2011b.
===========

I meant R2012b, obviously. I do see a performance drop from R2012a to R2012b. The ode45 test runs approximately as follows

R2012a (32-bit): 4.46 sec
R2012b (64-bit): 5.10 sec

Not huge, but definitely detectable.

I tried the ode23 and polyval cases on a Xeon under Windows 7. My 2009bsp1 results are anomalous with respect to the originally posted numbers, but the 2011a/2011b results have similar trends:

ode23:
posted 2009bsp1: 0.8550
my 2009bsp1: 1.3310
posted 2011a: 0.9647
my 2011a: 0.8532
posted 2012b: 1.1480
my 2012b: 0.9466

posted 2012b/2011a: 1.19
my 2012b/2011a: 1.11

polyval:
posted 2009bsp1: 13.27
my 2009bsp1: 17.53
posted 2011a: 18.25
my 2011a: 10.94
posted 2012b: 20.67
my 2012b: 15.40

posted 2012b/2011a: 1.13
my 2012b/2011a: 1.41

In the most recent postings to this thread you were looking at ode45 results, so I will go ahead and see if I can duplicate those.

"Gabriele " <ruga.ANTI@SPAM.libero.it> wrote in message <k32t8a$972$1@newscl01ah.mathworks.com>...
> Hi all,
> following a previous comparison on performances of new Matlab releases vs. old matlab releases (http://www.mathworks.it/matlabcentral/newsreader/view_thread/305954#833788), I have carried out a few tests regarding the new matlab release R2012b.
>
> The results are quite interesting because they show, in some cases, a significant decrease of performances, while in other cases a significant increase of performances.
>
> Test code 1:
>
> %----------------
> N=10;
> tcpu=NaN*zeros(1,N);
> for jjj=1:N,
> tic;
> [t,y]=ode23(@vdp1,[0 2000],[2 0]);
> tcpu(jjj)=toc;
> end;
> mean(tcpu)
> %----------------
>
> Result from mean(tcpu):
> R2009bSP1: 0.8550s
> R2011a: 0.9647s
> R2012b: 1.1480s
> Ratio R2012b/R2009bSP1: 1.3427
>
> i.e., with respect to ode23, R2012b seems to be more than 30% slower than R2009bSP1
>
> Let's try something else.
>
> Test code 2:
>
> %----------------
> N=100;
> tcpu=NaN*zeros(1,N);
> for jjj=1:N,
> tic;
> A=rand(1000,1000);
> B=rand(1000,1);
> x=A\B;
> tcpu(jjj)=toc;
> end;
> mean(tcpu)
> %----------------
>
> Result from mean(tcpu):
> R2009bSP1: 0.1151s
> R2011a: 0.1010s
> R2012b: 0.1000s
> Ratio R2012b/R2009bSP1: 0.8688
>
> i.e., in this case, R2012b seems to be almost 15% faster than R2009bSP1, but it has more or less the same performance as R2011a
>
> Test code 3:
>
> %----------------
> N=1000000;
> tic
> for jjj=1:N,
> p=rand(1,10);
> x=rand(1);
> polyval(p,x);
> end;
> toc
> %----------------
>
> Result from total calculation time:
> R2009bSP1: 13.266702s
> R2011a: 18.244994s
> R2012b: 20.665393s
> Ratio R2012b/R2009bSP1: 1.5577
>
> i.e. R2012b seems to be more than 55% slower than R2009bSP1 (and about 13% slower than R2011a) !!!!!!
>
> Test code 4: a complex simulation code with many functions calls
> R2009bSP1: 44.6090s
> R2011a: 48.9370s
> R2012b: 26.9690s
> Ratio R2012b/R2009bSP1: 0.6046
>
> For this complex simulation tool R2012b is 40% faster than R2009bSP1 !
>
> In general, from the initial experience, I have noticed R2012b to be less responsive than previous versions, particularly regading the time it takes to plot graphics.
>
> Moreover, the fact that now R2012b is case sensitive also with respect to the functions is already creating me several troubles with old codes...it was something anticipated, but it remains a trouble.
>
> Do you find something similar?
>
> Bye,
> Gabriele

I also find that R2012b's GUI is incredible sluggish compared to any previous version. It is unusable because it takes seconds for anything in the window to respond. And the PC I have is a Dell R5400 Workstation with Dual Quad Core Xeon 2Ghz and 4GB Ram