Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Which optimization method to use?

Subject: Which optimization method to use?

From: Aino

Date: 27 Mar, 2013 17:09:19

Message: 1 of 6

Hello all.

I have a function that includes a simulation. The input of the simulation is seven parameters. I need to vary five of the seven parameters to find the combination that would minimize the final output (error that is calculated between a measured signal and the simulated signal). I have somewhat a good initial guess for the five parameters and I know that they should all be positive. I need to do this optimization several times, so computational cost is an issue too.

I tried the optimization with "fminsearch". It worked quite ok, but two problems still occurred. First, one of the five parameters had negative values. This particular parameter is possible to just leave out. Second, it would seem that there are several local minimum that are all feasible when considering the parameter values :( and "fminsearch" didn't always find the global minimum. Also, I think that the output error includes "noise", or maybe one/several parameters are not always relevant.

I am now trying out "fminsearchbnd" (File Exchange) without the fifth parameter and with lower bounds [0,0,0,0]. However, I was wondering if "fmincon" would be more appropriate for me? I am not sure how to implement it, though. Is there possibly even better (more robust, not too time-consuming) optimization method for me?

Thanks,
Aino

Subject: Which optimization method to use?

From: Alan_Weiss

Date: 28 Mar, 2013 12:34:29

Message: 2 of 6

On 3/27/2013 1:09 PM, Aino wrote:
> Hello all.
>
> I have a function that includes a simulation. The input of the
> simulation is seven parameters. I need to vary five of the seven
> parameters to find the combination that would minimize the final
> output (error that is calculated between a measured signal and the
> simulated signal). I have somewhat a good initial guess for the five
> parameters and I know that they should all be positive. I need to do
> this optimization several times, so computational cost is an issue too.
>
> I tried the optimization with "fminsearch". It worked quite ok, but
> two problems still occurred. First, one of the five parameters had
> negative values. This particular parameter is possible to just leave
> out. Second, it would seem that there are several local minimum that
> are all feasible when considering the parameter values :( and
> "fminsearch" didn't always find the global minimum. Also, I think that
> the output error includes "noise", or maybe one/several parameters are
> not always relevant.
> I am now trying out "fminsearchbnd" (File Exchange) without the fifth
> parameter and with lower bounds [0,0,0,0]. However, I was wondering if
> "fmincon" would be more appropriate for me? I am not sure how to
> implement it, though. Is there possibly even better (more robust, not
> too time-consuming) optimization method for me?
>
> Thanks,
> Aino

You might be interested in the documentation on optimizing simulations:
http://www.mathworks.com/help/optim/ug/optimizing-a-simulation-or-ordinary-differential-equation.html

fmincon is indeed an appropriate solver for your problem. It is part of
Optimization Toolbox. You should use the interior-point or sqp
algorithms; see
http://www.mathworks.com/help/optim/ug/writing-constraints.html#br9p_ry

Alan Weiss
MATLAB mathematical toolbox documentation

Subject: Which optimization method to use?

From: Aino

Date: 28 Mar, 2013 15:23:05

Message: 3 of 6

Alan_Weiss <aweiss@mathworks.com> wrote in message <kj1dck$hh5$1@newscl01ah.mathworks.com>...
> On 3/27/2013 1:09 PM, Aino wrote:
> > Hello all.
> >
> > I have a function that includes a simulation. The input of the
> > simulation is seven parameters. I need to vary five of the seven
> > parameters to find the combination that would minimize the final
> > output (error that is calculated between a measured signal and the
> > simulated signal). I have somewhat a good initial guess for the five
> > parameters and I know that they should all be positive. I need to do
> > this optimization several times, so computational cost is an issue too.
> >
> > I tried the optimization with "fminsearch". It worked quite ok, but
> > two problems still occurred. First, one of the five parameters had
> > negative values. This particular parameter is possible to just leave
> > out. Second, it would seem that there are several local minimum that
> > are all feasible when considering the parameter values :( and
> > "fminsearch" didn't always find the global minimum. Also, I think that
> > the output error includes "noise", or maybe one/several parameters are
> > not always relevant.
> > I am now trying out "fminsearchbnd" (File Exchange) without the fifth
> > parameter and with lower bounds [0,0,0,0]. However, I was wondering if
> > "fmincon" would be more appropriate for me? I am not sure how to
> > implement it, though. Is there possibly even better (more robust, not
> > too time-consuming) optimization method for me?
> >
> > Thanks,
> > Aino
>
> You might be interested in the documentation on optimizing simulations:
> http://www.mathworks.com/help/optim/ug/optimizing-a-simulation-or-ordinary-differential-equation.html
>
> fmincon is indeed an appropriate solver for your problem. It is part of
> Optimization Toolbox. You should use the interior-point or sqp
> algorithms; see
> http://www.mathworks.com/help/optim/ug/writing-constraints.html#br9p_ry
>
> Alan Weiss
> MATLAB mathematical toolbox documentation

Thank you for your answer!

I am now testing the "fmincon" with "sqp". I still have some questions about the optimization.. First, I don't know exactly the whole parameter space (therefore I use only the "0" lower bounds and "Inf" higher bounds) and with bad parameters the simulation stops and gives something rather arbitrary for the error. I tried to put "NaN" as the error when the simulation stops, but "sqp" algorithm doesn't seem to know how to deal with the "NaN". Did I misunderstand the last part of your first link, about returning "NaN"? I get the following error statement:

"Objective function is undefined at initial point. Fmincon cannot continue."

Also, I would be happy with an error smaller than 0.05, is that what "TolFun" is for? I didn't see any difference between TolFun=0.1 and TolFun=0.05. Also, to speed the optimization, I would like to have a minimum step size for the input parameters. "DiffMinChange"? Setting DiffMinChange=0.01 lead to slower computing but also smaller error. Also, the parameters have a very different magnitude, is it possible to tweak the step size separately for each parameter? I have to say I'm a bit lost with this optimization.

-Aino

Subject: Which optimization method to use?

From: Alan_Weiss

Date: 28 Mar, 2013 16:02:45

Message: 4 of 6

On 3/28/2013 11:23 AM, Aino wrote:
> Alan_Weiss <aweiss@mathworks.com> wrote in message
> <kj1dck$hh5$1@newscl01ah.mathworks.com>...
>> On 3/27/2013 1:09 PM, Aino wrote:
>> > Hello all.
>> >
>> > I have a function that includes a simulation. The input of the >
>> simulation is seven parameters. I need to vary five of the seven >
>> parameters to find the combination that would minimize the final >
>> output (error that is calculated between a measured signal and the >
>> simulated signal). I have somewhat a good initial guess for the five
>> > parameters and I know that they should all be positive. I need to
>> do > this optimization several times, so computational cost is an
>> issue too.
>> >
>> > I tried the optimization with "fminsearch". It worked quite ok, but
>> > two problems still occurred. First, one of the five parameters had
>> > negative values. This particular parameter is possible to just
>> leave > out. Second, it would seem that there are several local
>> minimum that > are all feasible when considering the parameter values
>> :( and > "fminsearch" didn't always find the global minimum. Also, I
>> think that > the output error includes "noise", or maybe one/several
>> parameters are > not always relevant.
>> > I am now trying out "fminsearchbnd" (File Exchange) without the
>> fifth > parameter and with lower bounds [0,0,0,0]. However, I was
>> wondering if > "fmincon" would be more appropriate for me? I am not
>> sure how to > implement it, though. Is there possibly even better
>> (more robust, not > too time-consuming) optimization method for me?
>> >
>> > Thanks,
>> > Aino
>>
>> You might be interested in the documentation on optimizing simulations:
>> http://www.mathworks.com/help/optim/ug/optimizing-a-simulation-or-ordinary-differential-equation.html
>>
>>
>> fmincon is indeed an appropriate solver for your problem. It is part
>> of Optimization Toolbox. You should use the interior-point or sqp
>> algorithms; see
>> http://www.mathworks.com/help/optim/ug/writing-constraints.html#br9p_ry
>>
>> Alan Weiss
>> MATLAB mathematical toolbox documentation
>
> Thank you for your answer!
>
> I am now testing the "fmincon" with "sqp". I still have some questions
> about the optimization.. First, I don't know exactly the whole
> parameter space (therefore I use only the "0" lower bounds and "Inf"
> higher bounds) and with bad parameters the simulation stops and gives
> something rather arbitrary for the error. I tried to put "NaN" as the
> error when the simulation stops, but "sqp" algorithm doesn't seem to
> know how to deal with the "NaN". Did I misunderstand the last part of
> your first link, about returning "NaN"? I get the following error
> statement:
>
> "Objective function is undefined at initial point. Fmincon cannot
> continue."
>
> Also, I would be happy with an error smaller than 0.05, is that what
> "TolFun" is for? I didn't see any difference between TolFun=0.1 and
> TolFun=0.05. Also, to speed the optimization, I would like to have a
> minimum step size for the input parameters. "DiffMinChange"? Setting
> DiffMinChange=0.01 lead to slower computing but also smaller error.
> Also, the parameters have a very different magnitude, is it possible
> to tweak the step size separately for each parameter? I have to say
> I'm a bit lost with this optimization.
>
> -Aino

If the simulation fails at the initial point x0, then fmincon fails, and
there is nothing you can do about it except ensure that the initial
point leads to a sensible result.

To find out what the various tolerances such as TolFun mean, see
http://www.mathworks.com/help/optim/ug/tolerances-and-stopping-criteria.html

To see what the DiffMinChange option means, read carefully the section
http://www.mathworks.com/help/optim/ug/optimizing-a-simulation-or-ordinary-differential-equation.html

You might also want to look at the options description:
http://www.mathworks.com/help/optim/ug/optimization-options-reference.html

The "step size" might not mean what you think it means. You seem to
think it is the change in parameters that fmincon takes. But it is, in
fact, the size of a step for estimating gradients by finite differences.

Good luck,

Alan Weiss
MATLAB mathematical toolbox documentation

Subject: Which optimization method to use?

From: Aino

Date: 28 Nov, 2013 17:00:13

Message: 5 of 6

Alan_Weiss <aweiss@mathworks.com> wrote in message <kj1pj5$178$1@newscl01ah.mathworks.com>...
> On 3/28/2013 11:23 AM, Aino wrote:
> > Alan_Weiss <aweiss@mathworks.com> wrote in message
> > <kj1dck$hh5$1@newscl01ah.mathworks.com>...
> >> On 3/27/2013 1:09 PM, Aino wrote:
> >> > Hello all.
> >> >
> >> > I have a function that includes a simulation. The input of the >
> >> simulation is seven parameters. I need to vary five of the seven >
> >> parameters to find the combination that would minimize the final >
> >> output (error that is calculated between a measured signal and the >
> >> simulated signal). I have somewhat a good initial guess for the five
> >> > parameters and I know that they should all be positive. I need to
> >> do > this optimization several times, so computational cost is an
> >> issue too.
> >> >
> >> > I tried the optimization with "fminsearch". It worked quite ok, but
> >> > two problems still occurred. First, one of the five parameters had
> >> > negative values. This particular parameter is possible to just
> >> leave > out. Second, it would seem that there are several local
> >> minimum that > are all feasible when considering the parameter values
> >> :( and > "fminsearch" didn't always find the global minimum. Also, I
> >> think that > the output error includes "noise", or maybe one/several
> >> parameters are > not always relevant.
> >> > I am now trying out "fminsearchbnd" (File Exchange) without the
> >> fifth > parameter and with lower bounds [0,0,0,0]. However, I was
> >> wondering if > "fmincon" would be more appropriate for me? I am not
> >> sure how to > implement it, though. Is there possibly even better
> >> (more robust, not > too time-consuming) optimization method for me?
> >> >
> >> > Thanks,
> >> > Aino
> >>
> >> You might be interested in the documentation on optimizing simulations:
> >> http://www.mathworks.com/help/optim/ug/optimizing-a-simulation-or-ordinary-differential-equation.html
> >>
> >>
> >> fmincon is indeed an appropriate solver for your problem. It is part
> >> of Optimization Toolbox. You should use the interior-point or sqp
> >> algorithms; see
> >> http://www.mathworks.com/help/optim/ug/writing-constraints.html#br9p_ry
> >>
> >> Alan Weiss
> >> MATLAB mathematical toolbox documentation
> >
> > Thank you for your answer!
> >
> > I am now testing the "fmincon" with "sqp". I still have some questions
> > about the optimization.. First, I don't know exactly the whole
> > parameter space (therefore I use only the "0" lower bounds and "Inf"
> > higher bounds) and with bad parameters the simulation stops and gives
> > something rather arbitrary for the error. I tried to put "NaN" as the
> > error when the simulation stops, but "sqp" algorithm doesn't seem to
> > know how to deal with the "NaN". Did I misunderstand the last part of
> > your first link, about returning "NaN"? I get the following error
> > statement:
> >
> > "Objective function is undefined at initial point. Fmincon cannot
> > continue."
> >
> > Also, I would be happy with an error smaller than 0.05, is that what
> > "TolFun" is for? I didn't see any difference between TolFun=0.1 and
> > TolFun=0.05. Also, to speed the optimization, I would like to have a
> > minimum step size for the input parameters. "DiffMinChange"? Setting
> > DiffMinChange=0.01 lead to slower computing but also smaller error.
> > Also, the parameters have a very different magnitude, is it possible
> > to tweak the step size separately for each parameter? I have to say
> > I'm a bit lost with this optimization.
> >
> > -Aino
>
> If the simulation fails at the initial point x0, then fmincon fails, and
> there is nothing you can do about it except ensure that the initial
> point leads to a sensible result.
>
> To find out what the various tolerances such as TolFun mean, see
> http://www.mathworks.com/help/optim/ug/tolerances-and-stopping-criteria.html
>
> To see what the DiffMinChange option means, read carefully the section
> http://www.mathworks.com/help/optim/ug/optimizing-a-simulation-or-ordinary-differential-equation.html
>
> You might also want to look at the options description:
> http://www.mathworks.com/help/optim/ug/optimization-options-reference.html
>
> The "step size" might not mean what you think it means. You seem to
> think it is the change in parameters that fmincon takes. But it is, in
> fact, the size of a step for estimating gradients by finite differences.
>
> Good luck,
>
> Alan Weiss
> MATLAB mathematical toolbox documentation

Hello again. :)

I'm still working with this optimization problem, though the simulation is updated a bit. I realized another possible problem. Changing the input parameters e.g. by 0.05 (5%), some parameters have larger effect to the outcome measure (the variable that is being minimized) than others. Could this have an effect to my results (optimized parameters)?

-Aino

Subject: Which optimization method to use?

From: Alan_Weiss

Date: 2 Dec, 2013 14:52:31

Message: 6 of 6

On 11/28/2013 12:00 PM, Aino wrote:
> Alan_Weiss <aweiss@mathworks.com> wrote in message
> <kj1pj5$178$1@newscl01ah.mathworks.com>...
>> On 3/28/2013 11:23 AM, Aino wrote:
>> > Alan_Weiss <aweiss@mathworks.com> wrote in message >
>> <kj1dck$hh5$1@newscl01ah.mathworks.com>...
>> >> On 3/27/2013 1:09 PM, Aino wrote:
>> >> > Hello all.
>> >> >
>> >> > I have a function that includes a simulation. The input of the >
>> >> simulation is seven parameters. I need to vary five of the seven >
>> >> parameters to find the combination that would minimize the final >
>> >> output (error that is calculated between a measured signal and the
>> > >> simulated signal). I have somewhat a good initial guess for the
>> five >> > parameters and I know that they should all be positive. I
>> need to >> do > this optimization several times, so computational
>> cost is an >> issue too.
>> >> >
>> >> > I tried the optimization with "fminsearch". It worked quite ok,
>> but >> > two problems still occurred. First, one of the five
>> parameters had >> > negative values. This particular parameter is
>> possible to just >> leave > out. Second, it would seem that there are
>> several local >> minimum that > are all feasible when considering the
>> parameter values >> :( and > "fminsearch" didn't always find the
>> global minimum. Also, I >> think that > the output error includes
>> "noise", or maybe one/several >> parameters are > not always relevant.
>> >> > I am now trying out "fminsearchbnd" (File Exchange) without the
>> >> fifth > parameter and with lower bounds [0,0,0,0]. However, I was
>> >> wondering if > "fmincon" would be more appropriate for me? I am
>> not >> sure how to > implement it, though. Is there possibly even
>> better >> (more robust, not > too time-consuming) optimization method
>> for me?
>> >> >
>> >> > Thanks,
>> >> > Aino
>> >>
>> >> You might be interested in the documentation on optimizing
>> simulations:
>> >>
>> http://www.mathworks.com/help/optim/ug/optimizing-a-simulation-or-ordinary-differential-equation.html
>> >>
>> >>
>> >> fmincon is indeed an appropriate solver for your problem. It is
>> part >> of Optimization Toolbox. You should use the interior-point or
>> sqp >> algorithms; see
>> >>
>> http://www.mathworks.com/help/optim/ug/writing-constraints.html#br9p_ry
>> >>
>> >> Alan Weiss
>> >> MATLAB mathematical toolbox documentation
>> >
>> > Thank you for your answer!
>> >
>> > I am now testing the "fmincon" with "sqp". I still have some
>> questions > about the optimization.. First, I don't know exactly the
>> whole > parameter space (therefore I use only the "0" lower bounds
>> and "Inf" > higher bounds) and with bad parameters the simulation
>> stops and gives > something rather arbitrary for the error. I tried
>> to put "NaN" as the > error when the simulation stops, but "sqp"
>> algorithm doesn't seem to > know how to deal with the "NaN". Did I
>> misunderstand the last part of > your first link, about returning
>> "NaN"? I get the following error > statement:
>> >
>> > "Objective function is undefined at initial point. Fmincon cannot >
>> continue."
>> >
>> > Also, I would be happy with an error smaller than 0.05, is that
>> what > "TolFun" is for? I didn't see any difference between
>> TolFun=0.1 and > TolFun=0.05. Also, to speed the optimization, I
>> would like to have a > minimum step size for the input parameters.
>> "DiffMinChange"? Setting > DiffMinChange=0.01 lead to slower
>> computing but also smaller error. > Also, the parameters have a very
>> different magnitude, is it possible > to tweak the step size
>> separately for each parameter? I have to say > I'm a bit lost with
>> this optimization.
>> >
>> > -Aino
>> If the simulation fails at the initial point x0, then fmincon fails,
>> and there is nothing you can do about it except ensure that the
>> initial point leads to a sensible result.
>>
>> To find out what the various tolerances such as TolFun mean, see
>> http://www.mathworks.com/help/optim/ug/tolerances-and-stopping-criteria.html
>>
>>
>> To see what the DiffMinChange option means, read carefully the section
>> http://www.mathworks.com/help/optim/ug/optimizing-a-simulation-or-ordinary-differential-equation.html
>>
>> You might also want to look at the options description:
>> http://www.mathworks.com/help/optim/ug/optimization-options-reference.html
>>
>>
>> The "step size" might not mean what you think it means. You seem to
>> think it is the change in parameters that fmincon takes. But it is,
>> in fact, the size of a step for estimating gradients by finite
>> differences.
>>
>> Good luck,
>>
>> Alan Weiss
>> MATLAB mathematical toolbox documentation
>
> Hello again. :)
>
> I'm still working with this optimization problem, though the
> simulation is updated a bit. I realized another possible problem.
> Changing the input parameters e.g. by 0.05 (5%), some parameters have
> larger effect to the outcome measure (the variable that is being
> minimized) than others. Could this have an effect to my results
> (optimized parameters)?
> -Aino

Certainly, scaling can affect your answers, though fmincon tries hard to
estimate any such scaling and take it into account. You can read a very
generic, overly simplified description of scaling problems here:
http://www.mathworks.com/help/optim/ug/when-the-solver-fails.html#br53duy-1

Alan Weiss
MATLAB mathematical toolbox documentation

Tags for this Thread

No tags are associated with this thread.

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us