View License

Download apps, toolboxes, and other File Exchange content using Add-On Explorer in MATLAB.

» Watch video

Highlights from
Optimization Tips and Tricks

4.7 | 58 ratings Rate this file 92 Downloads (last 30 days) File Size: 629 KB File ID: #8553 Version: 1.2

Optimization Tips and Tricks


John D'Errico (view profile)


26 Sep 2005 (Updated )

Tips and tricks for use of the optimization toolbox, linear and nonlinear regression.

Editor's Notes:

This file was a File Exchange Pick of the Week

| Watch this File

File Information

New users and old of optimization in MATLAB will find useful tips and tricks in this document, as well as examples one can use as templates for their own problems.
Use this tool by editing the file optimtips.m, then execute blocks of code in cell mode from the editor, or best, publish the file to HTML. Copy and paste also works of course.

Some readers may find this tool valuable if only for the function pleas - a partitioned least squares solver based on lsqnonlin.

This is a work in progress, as I fully expect to add new topics as I think of them or as suggestions are made. Suggestions for topics I've missed are welcome, as are corrections of my probable numerous errors. The topics currently covered are listed below.

1. Linear regression basics in matlab
2. Polynomial regression models
3. Weighted regression models
4. Robust estimation
5. Ridge regression
6. Transforming a nonlinear problem to linearity
7. Sums of exponentials
8. Poor starting values
9. Before you have a problem
10. Tolerances & stopping criteria
11. Common optimization problems & mistakes
12. Partitioned least squares estimation
13. Errors in variables regression
14. Passing extra information/variables into an optimization
15. Minimizing the sum of absolute deviations
16. Minimize the maximum absolute deviation
17. Batching small problems into large problems
18. Global solutions & domains of attraction
19. Bound constrained problems
20. Inclusive versus exclusive bound constraints
21. Mixed integer/discrete problems
22. Understanding how they work
23. Wrapping an optimizer around quad
24. Graphical tools for understanding sets of nonlinear equations
25. Optimizing non-smooth or stochastic functions
26. Linear equality constraints
27. Sums of squares surfaces and the geometry of a regression
28. Confidence limits on a regression model
29. Confidence limits on the parameters in a nonlinear regression
30. Quadprog example, unrounding a curve
31. R^2
32. Estimation of the parameters of an implicit function
33. Robust fitting schemes
34. Homotopies
35. Orthogonal polynomial regression
36. Potential topics to be added or expanded in the (near) future


This file inspired Fminspleas, Rmsearch, and Sumatoria De Elementos Diagonales De Matriz.

Required Products Optimization Toolbox
MATLAB release MATLAB 7.0.1 (R14SP1)
MATLAB Search Path
Other requirements Users of older releases of matlab may still find this document useful to read although they will not be able to execute much of the code because of the heavy use of anonymous functions.
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Comments and Ratings (72)
30 Nov 2016 Penning Yu

so helpful

Comment only
13 Oct 2016 Stefan

Stefan (view profile)

Yes, this Zip file seems broken (7-Zip). Opening it in Windows 7 Explorer worked for me...

Comment only
08 May 2016 littleblack knifeking

The zip file seems wrong~ Please check it out!

27 Sep 2015 arnold

arnold (view profile)

Hi John,

maybe I've missed it but do you explicitly show how to handle errors in x AND y for fitting/regression?
I've been looking for a solution for days, lost the knowledge how to do it properly.


Comment only
23 Nov 2014 Jason Nicholson

Thanks for the examples. I have a suggestion on orthogonal polynomial fitting. Forsythe suggests a way to solve for both the correct orthogonal polynomials to use and the coefficients. The advantage is the full normal equation never has to be solved. Both the orthogonal polynomials and there coefficients are solved. It seems like an excellent method of data fitting.

Forsythe, George E. "Generation and use of orthogonal polynomials for data-fitting with a digital computer." Journal of the Society for Industrial & Applied Mathematics 5.2 (1957): 74-88.

Comment only
16 May 2014 John Booker

16 May 2014 John Booker

15 Jul 2013 John D'Errico

John D'Errico (view profile)

dav - It is too bad you don't like what I've supplied. It is also too bad that you did not read it, as then you might have learned the answer to your question.

Comment only
15 Jul 2013 dav

dav (view profile)

15 Jul 2013 dav

dav (view profile)

how can I get the code on least absolute deviation please?

15 Jul 2013 dav

dav (view profile)

17 Mar 2013 Raymundo Marcos-Martinez

05 Mar 2013 Max

Max (view profile)

An incredible resource. Very thoughtful.

26 Sep 2012 Weiyang Zhao

I just finished a Java implementation of weighted least square regression with equality constraints based on the knowledge shared by you. Thank you, John.

14 Aug 2012 Fang

Fang (view profile)

very good

28 Mar 2012 Jia

Jia (view profile)

05 Oct 2011 Khairul Shafie

great work John. Thanks a lot.

Comment only
24 Aug 2011 Liam Mescall

Superb ! Thanks for the work

Comment only
09 Aug 2011 AMVR

AMVR (view profile)

02 Aug 2011 Eric Diaz

21 Jul 2011 Hydroman S

Great submission. Thanks John.

I have a question on transforming a nonlinear problem to linearity:

If the nonlinear problem is of this form:

y = a* x^3

can we linearlize it as follows:

log(y) = log(a) + 3 log (x)



11 May 2011 John D'Errico

John D'Errico (view profile)

I've just submitted a new version, with many repairs done.

Comment only
11 May 2011 Jack Young

On running optimtips_21_36.m I get the following error:

??? In an assignment A(:) = B, the number of elements in A and B
must be the same.

Error in ==> callAllOptimOutputFcns at 12
stop(i) = feval(OutputFcn{i},xOutputfcn,optimValues,state,varargin{:});

Error in ==> fminsearch>callOutputAndPlotFcns at 468
stop = callAllOptimOutputFcns(outputfcn,xOutputfcn,optimValues,state,varargin{:}) || stop;

Error in ==> fminsearch at 203
[xOutputfcn, optimValues, stop] = callOutputAndPlotFcns(outputfcn,plotfcns,v(:,1),xOutputfcn,'init',itercount, ...

Error in ==> optimtips_21_36 at 103
Xfinal = fminsearch(rosen,[-6,4],opts);

Does anyone has an idea where is comes from?

05 Apr 2011 Jonas

Jonas (view profile)

Should have rated this as 5 a long time ago. This is a most excellent resource, and pleas.m has helped me tremendously.

24 Jan 2011 michael scheinfeild

01 Apr 2010 Nitin

Nitin (view profile)

20 Mar 2010 Eric Diaz

Hasn't updated since 2006, despite having people tell him that there are bugs in the code.

One bug which I have reported to him by email is in the pleas wrapper function, when using multiple exponentials.

Other than that, it is a great code with great examples and explanations.

28 Jan 2010 MOHD

MOHD (view profile)

2 thumbs up!

14 Oct 2009 Danila

Danila (view profile)

Very nice and thorough compilation of tips and tricks.

23 Sep 2009 Shaun

Shaun (view profile)

Hi John,

As pointed out by Eric, I guess, for newer versions, you need an update.


function stop = optimplot(x, optimValues, state)
% plots the current point of a 2-d otimization
stop = false;
hold on;

Comment only
17 Mar 2009 Jan Gläscher

Jan Gläscher (view profile)

Excellent resource. So very useful.

03 Feb 2009 Eric

Eric (view profile)

One small bug that prevents optimtips.m from running completely, e.g., when publishing optimtips.m

Change line 3 inj optimplot.m from
stop = [];
stop = false;

Comment only
22 Jan 2009 Ben Steiner

Echoing the other posts here. this is an excellent intro to optimization in general and matlab capabilities in particular. Thanks John

22 Jan 2009 Ben Steiner

23 Sep 2008 Ida Westerberg

Super! Just the help that what I was looking for.

21 Sep 2008 A B


Comment only
01 Jun 2008 jugmendra singh

14 May 2008 Adnèn Troudi

Bravo Merci beaucoup

26 Mar 2008 pravin katre



28 Jan 2008 Björn Wurst

I need robust regression methods in my diploma thesis and this work gives a verry good first impression of regression in matlab.

06 Dec 2007 Annamnaidu S

20 Nov 2007 b q

10 Oct 2007 zuduo zheng

Good Job!!!

07 Sep 2007 Sergei Koulayev


I loved this the most:
"% Likewise, reducing the value of TolFun need not reduce the error
% of the fit. If an optimizer has converged to its global optimum,
% reducing these tolerances cannot produce a better fit. Blood cannot
% be obtained from a rock, no matter how hard one squeezes. The rock
% may become bloody, but the blood came from your own hand."

09 Aug 2007 hippo man

I am thai,who love Matlab.thank a lot.

Comment only
12 Jul 2007 Varun Sakalkar

Nice work!!

02 Jul 2007 Hua Yang

Thanks very much!

Comment only
12 Jun 2007 ponthep veng

Good thank you
From thailand

07 Apr 2007 Nair SUBRA

13 Mar 2007 felix prasad

thank you

08 Feb 2007 Jorge Martinez


10 Jan 2007 John D'Errico

I'll see if I can do something with stochastic optimizers. It is a topic I apparently forgot to cover. Of course, the GADS toolbox is available for genetic algorithms. Please check back in a week or two.

Comment only
10 Jan 2007 thank you

John, could you talk more about simulated annealing and other similiar optimization techniques? Or write a general function as you have done for gridfit. Thank you. I always learnt very much from you.

03 Jan 2007 Vishnuvenkatesh Dhage

very useful

28 Dec 2006 Garrett Barter

Great work! I found the linprog examples for L1 and L_infty regression quite helpful.

05 Dec 2006 kimi raikkonen

20 Nov 2006 wilmer salazar trujillo

Deben promoverla con mayor intensidad en centros educativos desde primeros niveles

31 Jul 2006 Abdimaged Mussa

not perfet though, it has usefull informations, but not many to explore,
overall its a good website

28 Jul 2006 Dar Madi

It is very helpful for me to solve my work

27 Jul 2006 Tie Ling

This package is very useful for me. It is excellent. Thank you for your help!

03 Jun 2006 Suman Banerjee

Sir, it's a Excellent package. You should publish a book and please make sure that general students like me from India can buy it. Thnks for helping.

27 May 2006 thank you

extremely useful

14 May 2006 Sung SOo Kim

This is an excellent package.
Thank you so much.

08 May 2006 sione palu

Excellent package.

03 May 2006 Wang Qiwen

Very good

21 Feb 2006 Taghi Miri

Thank you, i found it very useful

06 Dec 2005 Peter Krug

A must read to beginners like myself. Great work that really helps - not like to on-line help of Matlab.

03 Nov 2005 21st Jocobi

31 Oct 2005 Anthony Clark

An excellent reource. You should publish this as a book, it would be a valuable resource for post graduates and carrer professionals! Really improved my routines by awnsering a lot of technical questions about using the optim toolbox (generally not covered in help or other books more general to the subject area). THANKYOU!

05 Oct 2005 Kaushik b

13 Dec 2005

Six new topics have been added, some existing
topics expanded. Added titles and axis labels for all
plots, etc.

20 May 2011 1.2


25 Apr 2016 1.2

BSD License

Contact us