Code covered by the BSD License  

Highlights from
fminsearchbnd, fminsearchcon

4.94828

4.9 | 61 ratings Rate this file 334 Downloads (last 30 days) File Size: 20.4 KB File ID: #8277

fminsearchbnd, fminsearchcon

by

 

11 Aug 2005 (Updated )

Bound constrained optimization using fminsearch

Editor's Notes:


This is a File Exchange Select file.



Select files are submissions that have been peer-reviewed and approved as meeting a high standard of utility and quality.

| Watch this File

File Information
Description

Fminsearch does not admit bound constraints.
However simple transformation methods exist to
convert a bound constrained problem into an
unconstrained problem.

Fminsearchbnd is used exactly like fminsearch,
except that bounds are applied to the variables.
The bounds are applied internally, using a
transformation of the variables. (Quadratic for
single bounds, sin(x) for dual bounds.)

The bounds are inclusive inequalities, which admit
the boundary values themselves, but will not permit
ANY function evaluations outside the bounds.

Note that fminsearchbnd allows the user to exactly fix a variable at some given value, by setting both bounds to the exact same value.

Example usage:
rosen = @(x) (1-x(1)).^2 + 105*(x(2)-x(1).^2).^2;

% unconstrained fminsearch solution
fminsearch(rosen,[3 3])
ans =
    1.0000 1.0000

% Lower bounds, no upper bounds
fminsearchbnd(rosen,[2.5 2.5],[2 2],[])
ans =
    2.0000 4.0000

Lower bounds on both vars, upper bound on x(2)
fminsearchbnd(rosen,[2.5 2.5],[2 2],[inf 3])
ans =
    2.0000 3.0000

I've now included fminsearchcon in the package, a tool that also allows nonlinear constraints.

Acknowledgements

This file inspired Zfit Gui(Varargin) Fits And Simulates Impedance Data, Easyfit Gui, Fminspleas, Fminsearchbnd New, Zfit, Variogramfit, Accelerated Failure Time (Aft) Models, Fit Distributions To Censored Data, Total Least Squares Method, and Minimize.

MATLAB release MATLAB 7.0.1 (R14SP1)
Other requirements This code should work on any release of matlab which allows the use of sub-functions.
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Comments and Ratings (102)
21 Oct 2014 Alexandre Laurin  
23 Sep 2014 Simon Mæng  
22 Sep 2014 Dmitry  
26 Aug 2014 John D'Errico

It uses fminsearch, which is part of MATLAB, not in the optimization toolbox. If you already had the optimization toolbox, I'd just tell you to use fmincon.

26 Aug 2014 Seb Biass

Hey, thanks for that. I was just wondering… does it work without the optimization toolbox?
Thanks

12 Aug 2014 Sani  
21 Jun 2014 John D'Errico

I have no idea why you are having a problem installing it, after apparently many days of trying to do so. I can't tell you much more than what I've already said.

1. Click on the button that says DOWNLOAD SUBMISSION. Use your mouse to do so.

2. Unzip the file. There are MANY utilities that will unzip a file. In fact, MATLAB itself contains an unzip tool. Use it.

3. Add the resulting top level directory to your search path. Use either pathtool or addpath to do this.

I don't know which of the above things has caused you a problem, but they all seem pretty basic.

I have NO idea what you are asking about how to use an if statement in this context.

Finally, I'm sorry, but I won't go into depth about the differences between fminsearch and fmincon. That would take far too much time to do on my part. There are many places online where Nelder/Mead tools are explained for you to learn about FMINSEARCH. As far as fmincon goes, start with reading the doc for it. If you bother to look at the bottom, you will find references.

20 Jun 2014 Nathalie

Hi, i'm struggling how to install it?

I'll be thankful if you help me with that.

And also, can i instead write some loop with 'if..' , imposing constraints/conditions on the values my parameters can take, so to double check my constrained problem. Also, why 'fmincon' may work worse than 'fminsearch'. I can't find the mechanics of how these two commands work? thank you in advance.

11 Jun 2014 John D'Errico

1. Download it.
2. Install it as directed.
3. Read the help, look at the examples.

What else can I possibly say?

11 Jun 2014 Nathalie

hi, how to use this command?

02 Jun 2014 Matthew  
24 May 2014 Seb Biass

Always good to see Mathworks relying on users to provide such functions...

08 Apr 2014 Kees de Kapper

Dear John,

Thank you for your contribution to the community.
I’ve got a question regarding the initial step. Probably it is related to the Q/A by Nguyen, but I’m not completely sure.

If the initial value is at the boundary, the initial step is very small and therefore the optimization is stuck at the boundary value (in case of my optimization function). If the initial value is middle of the boundaries, the initial step is reasonable, and the optimization can walk to the “global” minimum (which can be close to one of the boundaries). Apparently there is scaling of the step size between the boundaries.
Is it possible to avoid this scaling?

Thanks in advance,
Kees

14 Mar 2014 Nguyen

Dear John,
Thank you very much for your last help.
May I have a further question about algorithm?
Regarding Trust-Region-Reflective algorithm, minimization for example, basically its four steps are repeated until convergence.
How about the relationship between this convergence and TolFun or TolX?
Thank you again!

11 Mar 2014 Jakob Sievers

Thanks John! This resolved a problem I had been dealing with, very nicely!

20 Feb 2014 Christophe

Dear John,
No problem, thank you so much for taking the time to answer my question. So very much appreciated. I am using fminsearchbnd a lot and have also included rmsearch in my list of tools for optimization. Thank you so much for making these tools available!

20 Feb 2014 John D'Errico

Hi Christophe,

The problem is, these tools are really just wrappers around fminsearch, which uses only a restricted set of parameters. From the help for fminsearch, I see only:

fminsearch uses
these options: Display, TolX, TolFun, MaxFunEvals, MaxIter, FunValCheck,
PlotFcns, and OutputFcn.

So I could never be able to control the problem as you would desire. Sorry. Even that limited set of variables becomes corrupted, since fminsearchbnd uses a transformation, which will prevent you from controlling things using TolX as it is designed.

John

20 Feb 2014 Christophe

Dear John,
So many thanks for this program. I was wondering if there was an easy way to modify the fminsearchbnd to be able to handle the input of an argument like ( 'DiffMinChange',1e-2 )? My variables are in millimetres and I would like to have the variables changing by at least 0.01mm.
Once again, thank you for fminsearchbnd!

17 Feb 2014 Nguyen

Thank you so much for your support.
I have total 7 variables.
With a support from my friend, we tried to devide constrained range of each variable into n value.
We carried out n^7 local optimization and chose the best.
We tried to globalize the local minimization function.
However, we hope to find other ways/tools.

17 Feb 2014 John D'Errico

How clear can I make this? These are search routines, just like fminsearch, fmincon, fsolve, etc. In fact, the tools I have supplied are based on fminsearch. All of those tools search for a local minimizer, based on their starting point.

If you choose a good starting point, then you get a good solution. Your knowledge of the system you will optimize will always help.

NO such tool can EVER be assured of finding a global optimum on a completely general function. Period.

If you will insist on positively finding a globally optimal solution, regardless of your starting point, then these are the wrong tools for you. Of course, unless you can be assured that your function has some nice properties, global optimization can be a terribly tricky problem. Good luck in that.

At best, a scheme that has some decent properties is to start with many random starting points, then choose the best of the solutions that result. This is implemented in my rmsearch tool, also found on the file exchange, although it is fairly trivial to do on your own. A virtue of such a stochastic scheme is that if you start in the basin of attraction of the global optimizer, then it should converge to that solution. So as long as that basin is not too small that it is never visited, then you can succeed here. And because of the Monte Carlo starts, you can even assign a probability that the global solution was found. But certainty? Sorry, no. Again, good luck.

Finally, you say only that you have MORE than 4 variables. How many more? My advice has always been that too many variables make tools like fminsearch work poorly. I have suggested that 6 variables is a reasonable upper limit, but I have heard of people solving larger problems.

17 Feb 2014 Nguyen

Thank you very much for your file.
But I still have an unclear issue regarding fminsearchbnd.
When I change the starting value x0, the fval result will be changed (a lot).
So I have an initial conclusion that your function will find out local minimum, depends on starting guess.
Is it right?
In such that case, please help me how to find global minimum between lower and upper boundaries of variables (more than 4 variables)?

Again, thank you very much.

22 Jan 2014 Adam Auton  
22 Jan 2014 Blake Milner  
04 Dec 2013 Lorenz  
25 Sep 2013 Jacob Kirkensgaard  
22 Jul 2013 Poorya  
17 May 2013 Alvaro Campos Duque

Yeah, it works =) Thank u so much!!

16 May 2013 John D'Errico

Fminsearchcon uses your objective function, which for 5 parameter models, must take vector input. Thus it will pass in a vector x to the objective function that has 5 elements here. The constraint function must then also accept a vector of length 5. So your call to fminsearchcon might look like this:

[x,fval]=fminsearchcon(@fit,x0,lb,ub,[],[],@(x) 2.3*(x(4)^2+x(3))^0.5-x(5)-x(2));

16 May 2013 Alvaro Campos Duque

Congrats again for the useful function John. I just have a qestion: i'm trying to use fminsearchcon in a 5 parameter function i've created:

function error=fit(parameters)
x1=parameters(1);
x2=parameters(2);
x3=parameters(3);
x4=parameters(4);
x5=parameters(5);
(...) Here there is a long script but, at the end, the important thing are the input and outputs (...)

What I wonder is to find the parameters that optimize the error function. These parameters are interelated by the non-linear constrain:

-2.3*(x4^2+x3)^0.5+x5+x2>0

The problem is that I don't know how to implement this constrain, the examples I have seen are just with one parameter. I have tried this:

x0=[0.5,0.05,0.05,1,0.5];
lb=[0,0,0,0,0];
ub=[2,1,1,10,5];
[x,fval]=fminsearchcon(@fit,x0,lb,ub,[],[],@(x2,x3,x4,x5) 2.3*(x4^2+x3)^0.5-x5-x2)

But, of course, it doesn't recognise what is x2, x3, x4, or x5 because it is defined nowhere.

Any clue?? Thanks in advance! :)

16 May 2013 Alvaro Campos Duque

Congratulations John, thank you so much for such a good job!

27 Mar 2013 Jeff Warner

Congratulations on a great job creating this function. This is something I have been searching for. I have been using the fminsearch in the past but it is not a very robust fitting routine because you cannot constrain the fitting parameters so I would not always obtain a good fit.
Now that I am using your fminsearchbnd, every time I run the program on the same file the fitting result is close to the previous fits.

12 Mar 2013 John D'Errico

Alfonso - sorry, but I strenuously refuse to answer questions that are not directly related to a submission. The comments field really is not for consulting, which by the way, is something I now avoid anyway. Your question is related only peripherally to this tool.

11 Mar 2013 Alfonso

Hi again,

thanks for your "non-answer".

My previous comment was just a general question on box constrained optimization using variable transformation, and not a critics on your code (which, btw, I judge 4 stars).

Thought it is just a wrapper..., I have not doubts on the quality of your work....

However, in a real world, multidimensional problems are very common and they need to be solved without waiting years to converge to the minimum.

In this context, gradient-based methods (e.g. L-BFGS, SD or CG) can be used togheter with variable transformation.

Then, I ask you again, how can we avoid to be stuck on the bonduary of the constraints (where the transformation gradient is 0)?

This is not a trivial problem, and it can lead to early stopping the optimization iterations on a sub-optimal saddle point.

Regards, Alfonso

10 Jan 2013 Alessandro Magnani  
28 Aug 2012 John D'Errico

(Part 3, continued from my previous responses.)

So, has fminsearch succeeded in this task? Well, yes, a qualified yes. It was fairly slow. For example, lets try fminunc from the optimization toolbox.

>> [xfinal,fval,exitflag] = fminunc(fofx,x0,optimset('display','iter'))

First-order
Iteration Func-count f(x) Step-size optimality
0 26 2500 20
1 52 2025 0.05 18
2 78 0 1 1.49e-08

Local minimum found.

Optimization completed because the size of the gradient is less than
the default value of the function tolerance.

<stopping criteria details>

xfinal =
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

fval =
0

exitflag =
1

Fminunc took 0.014 seconds to run, requiring 2 iterations and 78 function evaluations. When done, the solution was composed of exact zeros.

>> format long g
>> max(abs(xfinal))
ans =
0

As I said, this was a really really REALLY simple problem. There is no curved valley to follow, as many optimization problems will have. There are no bounds or constraints here to bounce away from. Did fminsearch succeed on that problem? Well, yes, but it took some effort. There are better tools to be found, but fminsearch based tools will manage to stumble along. I generally advise that fminsearch is perfectly fine for small problems. For 2 or 3 variables, fminsearch is often my "go to" algorithm. (I'll even use it on 1 variable problems at times.) There is no reason to pull out a big gun there, and fminsearch has some very nicely robust qualities about it for small problems. It is insensitive to some irregularities in your function, and can even survive some things like derivative discontinuities that may make gradient based optimizers unhappy campers.

Up to about 6 variables it is still ok. I tend to be careful above 6 variables or so, and that limit is pretty problem dependent, pretty soft. Even 10 or 12 variables might be ok.

But really, by the time a problem has more than 10 variables, I'll be looking at a tool from the optimization toolbox. fminunc, fmincon will be common choices here. Even better is lsqlin for the problems that really are "linear" in nature. As such, you can almost think of lsqlin (or quadprog) as almost not being iterative solvers at all, in the sense that you don't need to provide starting values.

So fminsearch and the derived tools I provide here are good, workable tools. These are NOT toy functions, but they have their limits. They work splendidly on smaller problems, giving you an easy way to implement simple constraints. They MAY do a reasonable job for your problem, but I won't push them too far. Even as the author of fminsearchbnd, the very first toolbox I would recommend buying is the optimization toolbox if you do at all many optimizations of any size.

28 Aug 2012 John D'Errico

(Part 2, continued from the last response.)

Lets try being more explicit here. Vary the MaxFunEvals parameter.

>> [xfinal,fval,exitflag] = fminsearch(fofx,x0,optimset('maxfunevals',1000))

Exiting: Maximum number of function evaluations has been exceeded
- increase MaxFunEvals option.
Current function value: 887.434654

xfinal =
Columns 1 through 12
-2.923 12.909 3.1561 3.0168 -1.2408 3.0691 4.9753 2.4416 7.3143 -2.3687 3.5243 5.8719
Columns 13 through 24
2.311 -10.707 6.0148 0.45126 11.192 4.6186 3.3215 5.484 3.3361 4.2973 1.9503 12.542
Column 25
3.2774

fval =
887.43

>> [xfinal,fval,exitflag] = fminsearch(fofx,x0,optimset('maxfunevals',2000))

Exiting: Maximum number of function evaluations has been exceeded
- increase MaxFunEvals option.
Current function value: 64.925968

xfinal =
Columns 1 through 12
1.1789 -1.8439 1.2661 -1.4893 0.20456 3.6405 -0.75783 0.82694 -0.63308 -0.96542 1.5783 1.6521
Columns 13 through 24
-0.07538 -2.409 0.76939 1.9824 -1.9948 0.87846 -0.94908 2.7612 0.351 -0.74091 0.32458 3.1321
Column 25
1.0073

fval =
64.926

>> [xfinal,fval,exitflag] = fminsearch(fofx,x0,optimset('maxfunevals',4000))

Exiting: Maximum number of function evaluations has been exceeded
- increase MaxFunEvals option.
Current function value: 5.887733

xfinal =
Columns 1 through 12
-0.039379 -0.62272 0.3608 0.033666 -0.3333 -0.43402 -0.30185 0.16978 0.106 0.41576 0.3982 -0.15116
Columns 13 through 24
0.48606 -0.83993 0.66158 0.95052 0.20017 0.77219 -0.80237 0.57812 0.10016 0.62498 -0.36942 -0.37967
Column 25
0.06037

fval =
5.8877

>> [xfinal,fval,exitflag] = fminsearch(fofx,x0,optimset('maxfunevals',8000))

Exiting: Maximum number of iterations has been exceeded
- increase MaxIter option.
Current function value: 1.364211

xfinal =
Columns 1 through 12
0.0085952 -0.12505 0.034718 -0.21946 -0.16216 0.020216 0.21074 -0.054082 0.25911 0.16582 0.41984 0.15747
Columns 13 through 24
-0.049798 -0.12582 0.45415 0.4152 -0.093939 0.38964 -0.40392 -0.12336 0.055191 -0.077377 -0.39227 -0.14665
Column 25
-0.094889

fval =
1.3642

Ok, so after 8000 function evaluations, now it is hitting the default iteration limit.

mean(abs(xfinal))
ans =
0.18638

I can really let it go wild. For example, this next example took under 2 seconds to solve on my machine:

>> [xfinal,fval,exitflag] = fminsearch(fofx,x0,optimset('maxfunevals',100000,'maxiter',100000))

xfinal =
Columns 1 through 12
0.00012307 2.9632e-05 -5.5942e-05 -6.4526e-05 -0.00013126 -0.00019728 -8.2417e-05 3.2675e-05 8.4082e-05 -5.6463e-06 -0.00017336 -0.00015729
Columns 13 through 24
-4.7605e-05 0.00023687 -1.0958e-05 0.00019857 -9.4663e-05 -3.9436e-05 6.2949e-05 -5.8151e-05 4.9917e-05 -8.9443e-05 1.8443e-06 0.00011523
Column 25
3.8857e-05

fval =
2.9028e-07

exitflag =
1

>> mean(abs(xfinal))
ans =
8.7266e-05

All of the other examples had exitflag==0 by the way. It finally went far enough that fminsearch is tripping on its convergence tolerances, rather than on the function or iteration limits. It thinks it may have converged!

(End of part 2. Part 3 to follow.)

28 Aug 2012 John D'Errico

Hi Jessica,

The problem is, fminsearch tools are simply not very efficient at searching spaces that are at all high dimensional. The polytope flops around, expanding, contracting, etc., but the human mind simply does not appreciate how large a 25 dimensional space is. That makes sense, because we are built to visualize things in 3 dimensions, so we tend to think of everything in terms of our experience.

I'll give a simple example that illustrates the problem. Consider a perfectly circular well in 25 dimensions. I don't need to worry about bounds or anything, because the characteristics won't change in any material way.

The function is a paraboloid of revolution, centered at the 25 dimensional origin, and I'll start out at repmat(10,1,25) so it has to go a little ways, but this is a really simple problem. How does fminsearch handle it?

>> fofx = @(x) sum(x.^2,2);
>> x0 = repmat(10,1,25);
>> [xfinal,fval] = fminsearch(fofx,x0)

Exiting: Maximum number of function evaluations has been exceeded
- increase MaxFunEvals option.
Current function value: 3.762363

xfinal =
Columns 1 through 12
0.15208 -0.37597 0.47583 -0.4156 -0.47475 -0.22822 0.30263 0.15141 0.25204 0.45515 0.48452 -0.16627
Columns 13 through 24
0.18958 -0.65985 0.55836 0.54621 -0.012447 0.32472 -0.55204 0.29942 0.13633 0.28518 -0.69201 -0.25536
Column 25
0.21961

fval =
3.7624

As you can see, starting at a vector of all 10's, it got down to something that averaged roughly 0.35, so it was an improvement.

mean(abs(xfinal))
ans =
0.34662

So by the time it had run out of iterations, it had gone 96.5% of the way it had to go along a straight line, so some might say it has done pretty well. On the other hand, we can also think of what it has done is to improve the starting estimate by only enough to get the first decimal digit correct.

(End of part 1, of avery long response.)

27 Aug 2012 Jessica Piper

Hi John, love fminsearchbnd, thank you! Just a quick question. I've been using fminsearchbnd on a problem with 26 variables, and I've been getting good results. But based on your comments below, I worry I'm just getting lucky! What algorithm would you suggest for larger problems? (I'm fitting data from an optics experiment, but the objective is nonlinear, and the problem isn't even quasi-convex.)

24 Aug 2012 John D'Errico

Alfonso, It appears that you totally misunderstand how these tools work. fminsearchbnd and its cousin are based on fminsearch, a Nelder-Mead (polytope) optimizer. In fact, they call fminsearch to do the actual work. Fminsearch is NOT a gradient based tool. No gradient is EVER computed or even approximated. As such, they tend not to get stuck at such a point unless that is where the optimization drives them. You might want to do some reading about how the class of Nelder-Mead style tools work before you judge the algorithm.

Any optimization is of course subject to problem specific issues. Singularities, ill-conditioning, and multiple local solutions are all problems. But that point at the boundary is not as much an issue as you seem to fear.

24 Aug 2012 Alfonso

Hi John,

One simple question for you.

When you apply a quadratic transformation x=y^2 for x>=0 (btw, the same question holds for sin(x)); how you prevent the case that the optimizer is stuck on the bonduary (y=0)? Indeed, in the case that the actual solution is not on the bonduary, but during optimization iterations your gradient-based optimizer arrives in a point that lies on the bonduary, it cannot improve towards the minimum since the transformation gradient is 0 (dx/dy=2y=0) there.

I hope I was clear enough,

thanks in advance for your kind answer.

Alfonso

28 Jun 2012 John D'Errico

Kirk - when you change the bounds, you change the implicit transformation. It will still find a minimizer, but the problem has been changed, so there is no absolute guarantee it will converge to the same solution. When you have a problem with multiple local minimizers, any optimization tool can suffer from this issue.

20 Jun 2012 Kirk Smith

I found some odd behavior. I had my bounds on 7 variables [0,1]. My last 2 came out to be below 0.01. Next I changed the bounds for these last two variables to [0,0.4] and I got a new set of values.... shouldn't these be the same seeing as how in the first run they never even got close to 0.4?
Anyways I just added something like e*(max(x-1,0)+max(-x,0)) for my ub and lb with original fminsearch and it worked great. (Choose e large enough)

10 Jun 2012 John D'Errico

I'm sorry Pietro, but if you are using a Nelder-Mead solver to solve a 20 variable problem, you are wasting your time and I'm afraid mine too. I will not try to debug that mass of code for a problem that should never be thrown at this tool anyway.

10 Jun 2012 pietro

john here can find my code....

http://www.mathworks.se/matlabcentral/newsreader/view_thread/320630#879608

05 Jun 2012 John D'Errico

Pietro - without a specific example that shows there is a problem, I cannot know what is happening. You may be mistaken about whether the start point truly does satisfy the constraints. You may be supplying the constraints or the objective itself improperly. There may be a bug. Or, perhaps your starting point is right on a constraint boundary, or even epsilon over that edge. How can I guess? Again, fmincon does NOT require a feasible solution to start, although that surely helps. So that fmincon succeeds is irrelevant as these are different algorithms.

05 Jun 2012 pietro

the starting point provided satisfies all the constrains.

05 Jun 2012 John D'Errico

Fmincon uses a different algorithm, that can better handle infeasible starting values. The style of penalty used by fminsearchcon fails there, so you need to provide a starting point that at least satisfies your constraints.

05 Jun 2012 pietro

thanks John! I got this message:

Infeasible starting values (nonlinear inequalities failed).

But with the same starting point, with fmincon the optimization works. Do you have any suggestion to give me?

thanks.

Best regards

Pietro

05 Jun 2012 John D'Errico

Pietro - function handles make it easy to pass parameters in. For example, suppose you wish to find the minimum of the function (x-a)^2 for some fixed value of a. (Yes, you and I know that the min happens at a, but the computer does not, and I'm feeling too lazy to be more creative.)

myfun = @(x,a) (x-a).^2;
x0 = 1;
% Solve for the unconstrained min, with a = 12.
[xmin,fmin] = fminsearchbnd(@(x) myfun(x,12),x0)

xmin =
12
fmin =
6.18466949693273e-28

The above example, really just passes the call directly into fminsearch, but the point is how to pass in a.

% Solve for a constrained min, with a = 12.
ub = 5;
[xmin,fmin] = fminsearchbnd(@(x) myfun(x,12),x0,[],ub)

xmin =
5
fmin =
49

And of course, the min was found at the upper bound, at the point nearest to a that was allowed.

05 Jun 2012 pietro

Great function. How can I pass extra parameters to the objective function using the function handle?

11 May 2012 Davide

Thanks John! I fixed it.

Thanks again.

11 May 2012 John D'Errico

I should point out that these tools use fminsearch as the optimizer, just providing an overlay on top to implement constraints. That means that any parameters that fminsearch ignores are ignored here too. So diffminchange and diffmaxchange are completely irrelevant.

I can't really test out what is happening since I don't have any data. But I looked at the code, and usually when you get a nan as a result, it comes from some parameter going into a bad place.

So I'd put some debugging effort into the code, letting you drop in with the debugger as soon as a nan gets generated. Do one of these before you execute the code:

dbstop if naninf

I'm a bit confused why you are calling det on a diagonal matrix too. That would be far more efficiently done as simply the product of the elements. And since you are then computing the log of the determinant, do it as the sum of the logs of the elements, avoiding potential underflows or overflows.

Anyway, before you do any optimizations, always test your objective function. Make sure that it does what you expect, that it yields valid results, and that the results change when you perturb the parameters. Next, when you run an optimizer and you see strange things happening, set the "Display" option in the optimizer to 'iter'. In fact, I always do this on every optimization the first time I run one for any problem. Verify that it is doing something intelligent. Are the parameters changing? (Note that fminsearchbnd transforms the parameters, so that fminsearch is working with different numbers than you expect.)

And, finally, 9 parameters is at the very upper limit of what I'd ever recommend for a Nelder-Mead based tool, but it should run, albeit a bit slowly.

11 May 2012 Davide

Dear John, very nice function! It worked very well. I have used it in my optimization problems. However, in my last code I get a problem. The output is NaN and the function parameters seem not optimized. Perhaps you you may help me... Here is my code. I get the necessary data from an excel table.

I cannot realize where the problem is. Thanks in advance! Cheers

function [H,par,output]=mygarch()
format short
data=xlsread('select3stocks.xlsx',2,'A1:D249');
data1=data(1:149,:);
data2=data(150:199,:);
[L1, L2]=size(data2);

C0=[0.01,0.02,0.03]';
A0=[0.04,0.02,0.06]';
B0=[0.05,0.09,0.03]';
matr=[C0,A0,B0];
par0=reshape(matr,1,9);

X=zeros(L1-1,L2);
m=mean(data2);
X(1,:)=data2(1,:)-m;

s=0;

H=zeros(L1-1,L2);
H(1,:)=(std(data1)).^2;

lb=zeros(1,9);
ub=ones(1,9);

options =optimset('MaxIter',50000,'TolX',1e-10,'DiffMaxChange',0.00005,'DiffMinChange',0.00001,'MaxFunEvals',10000000);
[par,output]=fminsearchbnd(@maxlikelyhood,par0,lb,ub,options);

function y=maxlikelyhood(par)
C=par(1:3); A=par(4:6); B=par(7:9);
for i=2:L1
for j=1:L2
H(i,j)=C(j)+A(j)*X(i-1,j)^2+B(j)*H(i-1,j);
end
X(i,:)=data2(i,:)-m;
D=diag(H(i,:));
s=s+3*log(2*pi)+log(det(D))+X(i,:)*D.^(-2)*X(i,:)';
end
y=s;
end

end

06 Feb 2012 John D'Errico

(I've fixed the bug about the outputfcn problem and re-posted it. The new version should appear sometime this morning.)

As far as Stefan's problem goes, anytime you throw problems with variables that vary by 20 to 30 powers of 10 at ANY numerical optimizer, expect serious things to go wrong. This is a no-no for literally ANY numerical tool. And setting a convergence tolerance (TolX) to eps is just as silly, especially when your parameters vary by that many orders of magnitude.

Essentially, you are asking that one variable, which can apparently vary somewhere between 1 and 1e30, must be computed to an absolute accuracy of roughly 2e-16.

A common solution is to normalize your variables, so that both are at least of similar orders of magnitude. If you were trying to solve a problem where one variable had bounds of 1e20 to 5e20, and a second variable was bounded in the interval 1e-6 to 1e-5, then normalizing the variables to both be of order 1 would make complete sense. But your variables each vary by many powers of 10!

One thing all novices need to learn about computing, is that when things vary by that many powers of 10, is to use logs! Let fminsearch vary the log10 of your variables, so they now have bound vectors that look like [0, -6] for the lower bounds, and [30, 0] for the upper bound. Now, inside your function, take the parameters and raise 10 to that power BEFORE they are used. Do the same with the output when it is returned. As far as fminsearch is concerned, your numbers are perfectly normal looking, but your objective sees the numbers in their proper scales.

You will find that any optimizer runs much more happily when you do this, as now the variables are very nicely normalized. All of the mathematics works more simply when you work in the log space. In fact, this is surely the natural way to work with variables that vary by many powers of 10. A nice consequence is the tolerances in TolX become relative tolerances on the variables, something that surely makes much more sense.

And DON'T use eps as a convergence tolerance. Fminsearch will never be able to give you even a reasonable shot at convergence in even 10000 function evals with that fine of a tolerance. So what happens is fminsearch will keep on iterating until it runs out of function evaluations. Use a meaningful, realistic tolerance. You were probably only setting that fine of a tolerance because of the ridiculous variable scaling anyway.

06 Feb 2012 Stefan

The following test shows strange results:
----------------------------------
G0bnd 2.22e+014 L0bnd 2.707e-006

G0 2e+014 L0 3e-006

----------------------------------

function test()
options=optimset('Display','on', ...
'MaxFunEvals',1e4, ...
'MaxIter',1e4, ...
'TolFun', eps, ...
'TolX', eps, ...
'FunValCheck', 'on' ...
);

w= 150e-6;
xData = 1e-6:1e-6:w;
G0 = 2e14;
L0 = 3e-6;
yData = G0*exp(-xData./L0);

start_params = [G0, L0];
min_params = [ 1, 1e-6];
max_params = [1e30, 1];


plot(xData,yData,'-r');
hold('on');


result_params=fminsearchbnd(@fitG_error_function,start_params,min_params,max_params,options,xData,yData);
['G0bnd ' num2str(result_params(1),4) ' L0bnd ' num2str(result_params(2),4)]
plot(xData, result_params(1)*exp(-xData./result_params(2)),'-g');

result_params=fminsearch(@fitG_error_function,start_params,options,xData,yData);
['G0 ' num2str(result_params(1),4) ' L0 ' num2str(result_params(2),4)]
plot(xData, result_params(1)*exp(-xData./result_params(2)),'-b');


function fiterror = fitG_error_function(start_params,xData,yData)
Fitted_Curve= start_params(1) * exp(-xData./start_params(2));
Error_Vector=Fitted_Curve - yData;
fiterror=sum(Error_Vector.^2);
results.fiterror=fiterror;
end
end

06 Feb 2012 Romain

Great function, very handy.

I have noticed a small error, though. If fminsearch is not called, the function returns output.funcount, whereas it returns output.funcCount otherwise.

01 Feb 2012 John D'Errico

Sorry. I'd been playing with various alternatives sometime ago. The name on the header will be correct now when the latest update comes alive.

01 Feb 2012 Martin Richard

John,
I've been using that function for years now. Love it. Perhaps you may want to change the function name so that it fits the file name in the new version (fminsearchbnd3 in new file).

23 Nov 2011 John D'Errico

HI Kathleen - I've been asked about citing many of my submissions before. WE came up with a few options, detailed here:

http://blogs.mathworks.com/desktop/2010/12/13/citing-file-exchange-submissions/

23 Nov 2011 Kathleen

Hi John

I use fminsearchbnd a lot and I'd like to cite it in my work. Do you have a reference you would like used?
Thanks
Kathleen

23 Nov 2011 Kathleen

Hi John

I use fminsearchbnd a lot and I'd like to cite it in my work. Do you have a reference you would like used?
Thanks
Kathleen

11 Oct 2011 John D'Errico

Nick - fminsearchbnd is a simple optimizer, a close cousin to fminsearch. All it cares about is finding an optimum value, and it has no idea that your objective is based on a least squares estimation problem of some sort. If you need confidence bounds, a simple solution is to use a linear approximation to your function at the optimum, computing the Jacobian matrix at that point. Then it is a simple problem to compute approximate confidence intervals on the parameters. You can find this procedure explained, with an example, in my Optimization Tips & Tricks, also on the File Exchange.

11 Oct 2011 Brennan Smith

Thanks! Very useful

10 Oct 2011 Nick M.

Hi John

This is a great work and I actually used it and worked for me. I have a question though. I used fminsearchbnd and it turned it parameters, how can I compute uncertainty for those parameters?(covariance matrix?)
Thanks!

27 Jul 2011 Christophe

Dear John,

Many thanks for this very useful program. I have a question if I may:

I deal with the optimization of "physical" problems, keeping in mind the manufacturing-accuracies available to me. I know, for example, that I cannot have my parameters (defining the geometry I optimize) accurately machined. To me, a parameter set at 12.000, 12.001 or even 12.01 would give pretty much the same value for the output function.
So, I would like to be able to ensure that the parameters are moved by at least a minimum "delta" in the optimization, say 1e-2 for example.
Is this something that fminsearchbnd can handle?

12 Jul 2011 Yankun

Very useful.

23 Jun 2011 ks

Thank you very much. It works Great! :)

03 May 2011 Oleg

This is great! Thank you!

20 Apr 2011 Prabuddha Mukherjee

Thank you very much...very useful

27 Apr 2010 nico

Very useful...thank you !

30 Nov 2009 Sahin Aktas

Excellent Code...

16 Sep 2009 John D'Errico

Ryan,

This is really not a bound constrained problem. Your constraint is a general one, either a linear or a nonlinear inequality constraint.

The simple solution is to use a code that allows explicit linear or nonlinear constraints. There are a few on the FEX. In fact my fminsearchcon on the FEX does that, by applying a penalty to the problem when a constraint is violated. You might also look at optimize, by Rody Oldenhuis. This code allows explicit constraints of all forms.

However, there is a simple way to solve this class of problem with fminsearchbnd. Use another class of transformation. For example, suppose one wishes to minimize the function f(x,y), subject to the "bound" constraint that x <= erf(y).

Transform your problem such that

u = x + erf(y)
v = x - erf(y)

Clearly, v must be bounded above by 0. So use fminsearchbnd to optimize over the two dimensional domain (u,v). Inside your objective function, for any value of (u,v) you will compute the parameters (x,y) as

x = (u + v)/2;
y = erfinv(u - x);

Now you can finish evaluating the function f(x,y).

The only problem here happens if you also had other fixed bounds on x and y. But many other transformations would also have worked. For example, this transformation:

u = x
v = x - erf(y)

will still allow simple bound constraints on the parameter x, as well as allowing the nonlinear constraint x <= erf(y) as a bound constraint.

John

16 Sep 2009 Ryan Webb

Great Function. Used it mane times.

However, now I have a trickier problem where one of the boundaries is a function of one of the variables. Making UB a function of xtrans is possible, but how would fminsearchbnd determine this intent given that the cases for transformation (k) are determined solely by the initial numerical values of the constraints? Thoughts?

07 Sep 2009 Will

Excellent modification to create a very useful algorithm. Thanks!

23 Jul 2009 Srinivasa Chemudupati

Cannot Thank you enough John!!

21 May 2009 Chris Men

the beauty is yours.

19 Nov 2008 Umesh Rudrapatna

Thanks a lot! Helped me a lot.

08 Nov 2008 leo nidas

Thanks John!

10 Sep 2008 Chirackel Yoonus

Very useful

25 Apr 2008 M. P.  
25 Apr 2008 M. P.

Worked beautifully for my own optimization problem. Thanks John.

29 Nov 2007 Ken Purchase

Notice: I have submitted a slightly modified version that includes output and plot functions as well as slightly improved handling of varargin. Search for fminsearch.

The original is excellent and very useful- I hope the changes didn't break anything.

15 Nov 2007 Mert Sabuncu

excellent code - gets the job done!

11 Jun 2007 Yossef Ehrlichman

It's really working. Helped alot! Matlab should incoprate it into its libraries.

05 Jun 2007 Alex Chirokov

I really like this code: it is very well written and useful.

31 May 2007 John D'Errico

See fminsearchcon for linear and nonlinear inequality constraints. Equality constraints are more difficult to implement when coupled with bound constraints as implemented using transformations - John.

31 May 2007 Yang Zhang

I like this program and do you think it is possible to expand it to deal with linear constraints?

05 May 2007 Lauren Cooney

thank you for this great program! it saved me a lot of time and frustration!

20 Mar 2007 Eli Tom

Nice indeed.

15 Mar 2007 jimmy tsai

It's great. I couldn't solve my problem with fmincon but did it with this file. I really appreciate it.

13 Feb 2007 Natis Angarita  
28 Nov 2006 Dmitrey Kroshko

I connected John D'Errico file to the OpenOpt project
http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=13115&objectType=file
& informed him (I hope my letter passed antispam filter OK)
if any pretensions will be received, I promise to exclude the one
btw now default inner solver is Shor ralg with AST, which is better than current implementation of fminsearch, at least, for those task I tried with. Also, it can handle (sub)gradient info provided by user, and corresponding changes in John code were made.

11 Aug 2006 Umberto .  
11 Jul 2006 Yes its Good

works very well!

14 Jun 2006 CC Gomez

Nicely written.

07 Apr 2006 G.H. Rao

excellent program. on these lines I applied bounds successfully to the genetic algorithm program 'ga' of Matlab release 14

18 Jan 2006 cedric penard

Works very well, exellent ! Thanks.

20 Dec 2005 Rui Miguel

I was really needing it! thanks a lot.
Has worked flawlessly for me.

07 Oct 2005 Kaushik b

Thanks a lot. It really useful and works beautifully.

23 Sep 2005 Evan Palmer

Works really well and is very handy! Nice job!

23 Aug 2005 Vijit Nair

This is a great funcn. Thanks John.

12 Aug 2005 Ken Campbell

Nice trick solving a common problem

Updates
11 Oct 2005

Fixed a problem when a variable with dual bounds
is started at exactly the mid-point of the bounds.

09 Nov 2005

Allow the user to fix a variable by setting the lower
and upper bounds equal.

09 Nov 2005

Version 3: also solves unbounded problems.

12 Jun 2006

Release 2.1: Fixed a bug when some variables are fixed and other variables are bounded from both above and below.

24 Jul 2006

Update for FX Select nomination structure - added demo, test and doc files

06 Feb 2012

Fixed outputfcn bug when fminsearch is never actually called

Contact us