Optimization Tips and Tricks

Tips and tricks for use of the optimization toolbox, linear and nonlinear regression.

Description

New users and old of optimization in MATLAB will find useful tips and tricks in this document, as well as examples one can use as templates for their own problems.

Use this tool by editing the file optimtips.m, then execute blocks of code in cell mode from the editor, or best, publish the file to HTML. Copy and paste also works of course.

Some readers may find this tool valuable if only for the function pleas - a partitioned least squares solver based on lsqnonlin.

This is a work in progress, as I fully expect to add new topics as I think of them or as suggestions are made. Suggestions for topics I've missed are welcome, as are corrections of my probable numerous errors. The topics currently covered are listed below.

Contents
1. Linear regression basics in matlab
2. Polynomial regression models
3. Weighted regression models
4. Robust estimation
5. Ridge regression
6. Transforming a nonlinear problem to linearity
7. Sums of exponentials
8. Poor starting values
9. Before you have a problem
10. Tolerances & stopping criteria
11. Common optimization problems & mistakes
12. Partitioned least squares estimation
13. Errors in variables regression
14. Passing extra information/variables into an optimization
15. Minimizing the sum of absolute deviations
16. Minimize the maximum absolute deviation
17. Batching small problems into large problems
18. Global solutions & domains of attraction
19. Bound constrained problems
20. Inclusive versus exclusive bound constraints
21. Mixed integer/discrete problems
22. Understanding how they work
23. Wrapping an optimizer around quad
24. Graphical tools for understanding sets of nonlinear equations
25. Optimizing non-smooth or stochastic functions
26. Linear equality constraints
27. Sums of squares surfaces and the geometry of a regression
28. Confidence limits on a regression model
29. Confidence limits on the parameters in a nonlinear regression
30. Quadprog example, unrounding a curve
31. R^2
32. Estimation of the parameters of an implicit function
33. Robust fitting schemes
34. Homotopies
35. Orthogonal polynomial regression
36. Potential topics to be added or expanded in the (near) future

• Acknowledgements: This submission has inspired the following:; RMSEARCH, Fminspleas
• Required Products: Optimization Toolbox
• MATLAB release: MATLAB 7.0.1 (R14SP1)
• Other requirements: Users of older releases of matlab may still find this document useful to read although they will not be able to execute much of the code because of the heavy use of anonymous functions.

Other Files

license.txt,
opt_reg_tips/.DS_Store,
opt_reg_tips/consolidator.m,
opt_reg_tips/expfitfun3.m,
opt_reg_tips/expfitfun4.m,
opt_reg_tips/fminsearchbnd.m,
opt_reg_tips/implicit_obj.m,
opt_reg_tips/optimplot.m,
opt_reg_tips/optimtips.m,
opt_reg_tips/optimtips_0_20.m,
opt_reg_tips/optimtips_21_36.m,
opt_reg_tips/pleas.m,
opt_reg_tips/ReadMe.m,
opt_reg_tips/testnestfun.m

[Kaushik b]

 

[Anthony Clark]

An excellent reource. You should publish this as a book, it would be a valuable resource for post graduates and carrer professionals! Really improved my routines by awnsering a lot of technical questions about using the optim toolbox (generally not covered in help or other books more general to the subject area). THANKYOU!

[21st Jocobi]

 

[Peter Krug]

A must read to beginners like myself. Great work that really helps - not like to on-line help of Matlab.

[Taghi Miri]

Thank you, i found it very useful

[Wang Qiwen]

Very good

[sione palu]

Excellent package.

[Sung SOo Kim]

This is an excellent package.
Thank you so much.

[thank you]

extremely useful

[Suman Banerjee]

Sir, it's a Excellent package. You should publish a book and please make sure that general students like me from India can buy it. Thnks for helping.

[Tie Ling]

This package is very useful for me. It is excellent. Thank you for your help!

[Dar Madi]

It is very helpful for me to solve my work

[Abdimaged Mussa]

not perfet though, it has usefull informations, but not many to explore,
overall its a good website

[wilmer salazar trujillo]

Deben promoverla con mayor intensidad en centros educativos desde primeros niveles

[kimi raikkonen]

 

[Garrett Barter]

Great work! I found the linprog examples for L1 and L_infty regression quite helpful.

[Vishnuvenkatesh Dhage]

very useful

[thank you]

John, could you talk more about simulated annealing and other similiar optimization techniques? Or write a general function as you have done for gridfit. Thank you. I always learnt very much from you.

[John D'Errico]

I'll see if I can do something with stochastic optimizers. It is a topic I apparently forgot to cover. Of course, the GADS toolbox is available for genetic algorithms. Please check back in a week or two.
John

[Jorge Martinez]

Ok.

[felix prasad]

thank you

[Nair SUBRA]

 

[ponthep veng]

Good thank you
From thailand

[Hua Yang]

Thanks very much!

[Varun Sakalkar]

Nice work!!

[hippo man]

I am thai,who love Matlab.thank a lot.

[Sergei Koulayev]

John,

I loved this the most:
"% Likewise, reducing the value of TolFun need not reduce the error
% of the fit. If an optimizer has converged to its global optimum,
% reducing these tolerances cannot produce a better fit. Blood cannot
% be obtained from a rock, no matter how hard one squeezes. The rock
% may become bloody, but the blood came from your own hand."

[zuduo zheng]

Good Job!!!

[b q]

 

[Annamnaidu S]

 

[Björn Wurst]

I need robust regression methods in my diploma thesis and this work gives a verry good first impression of regression in matlab.

[TULISHETTI SRINIVAS]

 

[pravin katre]

good

[Adnèn Troudi]

Bravo Merci beaucoup

[jugmendra singh]

 

[A B]

5

[Ida Westerberg]

Super! Just the help that what I was looking for.

[Ben Steiner]

 

[Ben Steiner]

Echoing the other posts here. this is an excellent intro to optimization in general and matlab capabilities in particular. Thanks John

[Eric]

One small bug that prevents optimtips.m from running completely, e.g., when publishing optimtips.m

Change line 3 inj optimplot.m from
    stop = [];
to
     stop = false;

[Jan Gläscher]

Excellent resource. So very useful.

[Shaun]

Hi John,

As pointed out by Eric, I guess, for newer versions, you need an update.

Shaun

function stop = optimplot(x, optimValues, state)
% plots the current point of a 2-d otimization
stop = false;
hold on;
plot(x(1),x(2),'.');
drawnow

[Danila]

Very nice and thorough compilation of tips and tricks.