No BSD License  

Highlights from
Optimization Tips and Tricks

4.76471

4.8 | 57 ratings Rate this file 169 Downloads (last 30 days) File Size: 629 KB File ID: #8553

Optimization Tips and Tricks

by

 

26 Sep 2005 (Updated )

Tips and tricks for use of the optimization toolbox, linear and nonlinear regression.

Editor's Notes:

This file was a File Exchange Pick of the Week

| Watch this File

File Information
Description

New users and old of optimization in MATLAB will find useful tips and tricks in this document, as well as examples one can use as templates for their own problems.

Use this tool by editing the file optimtips.m, then execute blocks of code in cell mode from the editor, or best, publish the file to HTML. Copy and paste also works of course.

Some readers may find this tool valuable if only for the function pleas - a partitioned least squares solver based on lsqnonlin.

This is a work in progress, as I fully expect to add new topics as I think of them or as suggestions are made. Suggestions for topics I've missed are welcome, as are corrections of my probable numerous errors. The topics currently covered are listed below.

Contents
1. Linear regression basics in matlab
2. Polynomial regression models
3. Weighted regression models
4. Robust estimation
5. Ridge regression
6. Transforming a nonlinear problem to linearity
7. Sums of exponentials
8. Poor starting values
9. Before you have a problem
10. Tolerances & stopping criteria
11. Common optimization problems & mistakes
12. Partitioned least squares estimation
13. Errors in variables regression
14. Passing extra information/variables into an optimization
15. Minimizing the sum of absolute deviations
16. Minimize the maximum absolute deviation
17. Batching small problems into large problems
18. Global solutions & domains of attraction
19. Bound constrained problems
20. Inclusive versus exclusive bound constraints
21. Mixed integer/discrete problems
22. Understanding how they work
23. Wrapping an optimizer around quad
24. Graphical tools for understanding sets of nonlinear equations
25. Optimizing non-smooth or stochastic functions
26. Linear equality constraints
27. Sums of squares surfaces and the geometry of a regression
28. Confidence limits on a regression model
29. Confidence limits on the parameters in a nonlinear regression
30. Quadprog example, unrounding a curve
31. R^2
32. Estimation of the parameters of an implicit function
33. Robust fitting schemes
34. Homotopies
35. Orthogonal polynomial regression
36. Potential topics to be added or expanded in the (near) future

Acknowledgements

This file inspired Rmsearch and Fminspleas.

Required Products Optimization Toolbox
MATLAB release MATLAB 7.0.1 (R14SP1)
Other requirements Users of older releases of matlab may still find this document useful to read although they will not be able to execute much of the code because of the heavy use of anonymous functions.
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Comments and Ratings (67)
16 May 2014 John Booker  
16 May 2014 John Booker  
15 Jul 2013 John D'Errico

dav - It is too bad you don't like what I've supplied. It is also too bad that you did not read it, as then you might have learned the answer to your question.

15 Jul 2013 dav  
15 Jul 2013 dav

how can I get the code on least absolute deviation please?

15 Jul 2013 dav  
17 Mar 2013 Raymundo Marcos-Martinez  
05 Mar 2013 Max

An incredible resource. Very thoughtful.

26 Sep 2012 Weiyang Zhao

I just finished a Java implementation of weighted least square regression with equality constraints based on the knowledge shared by you. Thank you, John.

14 Aug 2012 Fang

very good

28 Mar 2012 Jia  
05 Oct 2011 Khairul Shafie

great work John. Thanks a lot.

24 Aug 2011 Liam Mescall

Superb ! Thanks for the work

09 Aug 2011 AMVR  
02 Aug 2011 Eric Diaz  
21 Jul 2011 Hydroman S

Great submission. Thanks John.

I have a question on transforming a nonlinear problem to linearity:

If the nonlinear problem is of this form:

y = a* x^3

can we linearlize it as follows:

log(y) = log(a) + 3 log (x)

26 May 2011 HARI KRISHNAN

f

11 May 2011 John D'Errico

I've just submitted a new version, with many repairs done.

11 May 2011 Jack Young

On running optimtips_21_36.m I get the following error:

??? In an assignment A(:) = B, the number of elements in A and B
must be the same.

Error in ==> callAllOptimOutputFcns at 12
stop(i) = feval(OutputFcn{i},xOutputfcn,optimValues,state,varargin{:});

Error in ==> fminsearch>callOutputAndPlotFcns at 468
stop = callAllOptimOutputFcns(outputfcn,xOutputfcn,optimValues,state,varargin{:}) || stop;

Error in ==> fminsearch at 203
[xOutputfcn, optimValues, stop] = callOutputAndPlotFcns(outputfcn,plotfcns,v(:,1),xOutputfcn,'init',itercount, ...

Error in ==> optimtips_21_36 at 103
Xfinal = fminsearch(rosen,[-6,4],opts);

Does anyone has an idea where is comes from?

05 Apr 2011 Jonas

Should have rated this as 5 a long time ago. This is a most excellent resource, and pleas.m has helped me tremendously.

24 Jan 2011 michael scheinfeild  
01 Apr 2010 Nitin  
20 Mar 2010 Eric Diaz

Hasn't updated since 2006, despite having people tell him that there are bugs in the code.

One bug which I have reported to him by email is in the pleas wrapper function, when using multiple exponentials.

Other than that, it is a great code with great examples and explanations.

28 Jan 2010 MOHD

2 thumbs up!

14 Oct 2009 Danila

Very nice and thorough compilation of tips and tricks.

23 Sep 2009 Shaun

Hi John,

As pointed out by Eric, I guess, for newer versions, you need an update.

Shaun

function stop = optimplot(x, optimValues, state)
% plots the current point of a 2-d otimization
stop = false;
hold on;
plot(x(1),x(2),'.');
drawnow

17 Mar 2009 Jan Gläscher

Excellent resource. So very useful.

03 Feb 2009 Eric

One small bug that prevents optimtips.m from running completely, e.g., when publishing optimtips.m

Change line 3 inj optimplot.m from
stop = [];
to
stop = false;

22 Jan 2009 Ben Steiner

Echoing the other posts here. this is an excellent intro to optimization in general and matlab capabilities in particular. Thanks John

22 Jan 2009 Ben Steiner  
23 Sep 2008 Ida Westerberg

Super! Just the help that what I was looking for.

21 Sep 2008 A B

5

01 Jun 2008 jugmendra singh  
14 May 2008 Adnèn Troudi

Bravo Merci beaucoup

26 Mar 2008 pravin katre

good

12 Feb 2008 TULISHETTI SRINIVAS  
28 Jan 2008 Björn Wurst

I need robust regression methods in my diploma thesis and this work gives a verry good first impression of regression in matlab.

06 Dec 2007 Annamnaidu S  
20 Nov 2007 b q  
10 Oct 2007 zuduo zheng

Good Job!!!

07 Sep 2007 Sergei Koulayev

John,

I loved this the most:
"% Likewise, reducing the value of TolFun need not reduce the error
% of the fit. If an optimizer has converged to its global optimum,
% reducing these tolerances cannot produce a better fit. Blood cannot
% be obtained from a rock, no matter how hard one squeezes. The rock
% may become bloody, but the blood came from your own hand."

09 Aug 2007 hippo man

I am thai,who love Matlab.thank a lot.

12 Jul 2007 Varun Sakalkar

Nice work!!

02 Jul 2007 Hua Yang

Thanks very much!

12 Jun 2007 ponthep veng

Good thank you
From thailand

07 Apr 2007 Nair SUBRA  
13 Mar 2007 felix prasad

thank you

08 Feb 2007 Jorge Martinez

Ok.

10 Jan 2007 John D'Errico

I'll see if I can do something with stochastic optimizers. It is a topic I apparently forgot to cover. Of course, the GADS toolbox is available for genetic algorithms. Please check back in a week or two.
John

10 Jan 2007 thank you

John, could you talk more about simulated annealing and other similiar optimization techniques? Or write a general function as you have done for gridfit. Thank you. I always learnt very much from you.

03 Jan 2007 Vishnuvenkatesh Dhage

very useful

28 Dec 2006 Garrett Barter

Great work! I found the linprog examples for L1 and L_infty regression quite helpful.

05 Dec 2006 kimi raikkonen  
20 Nov 2006 wilmer salazar trujillo

Deben promoverla con mayor intensidad en centros educativos desde primeros niveles

31 Jul 2006 Abdimaged Mussa

not perfet though, it has usefull informations, but not many to explore,
overall its a good website

28 Jul 2006 Dar Madi

It is very helpful for me to solve my work

27 Jul 2006 Tie Ling

This package is very useful for me. It is excellent. Thank you for your help!

03 Jun 2006 Suman Banerjee

Sir, it's a Excellent package. You should publish a book and please make sure that general students like me from India can buy it. Thnks for helping.

27 May 2006 thank you

extremely useful

14 May 2006 Sung SOo Kim

This is an excellent package.
Thank you so much.

08 May 2006 sione palu

Excellent package.

03 May 2006 Wang Qiwen

Very good

21 Feb 2006 Taghi Miri

Thank you, i found it very useful

06 Dec 2005 Peter Krug

A must read to beginners like myself. Great work that really helps - not like to on-line help of Matlab.

03 Nov 2005 21st Jocobi  
31 Oct 2005 Anthony Clark

An excellent reource. You should publish this as a book, it would be a valuable resource for post graduates and carrer professionals! Really improved my routines by awnsering a lot of technical questions about using the optim toolbox (generally not covered in help or other books more general to the subject area). THANKYOU!

05 Oct 2005 Kaushik b  
Updates
13 Dec 2005

Six new topics have been added, some existing
topics expanded. Added titles and axis labels for all
plots, etc.

20 May 2011

Contact us