Thread Subject: details in optimization documentation

Dear all

There are some issues in optimization with user - supplied derivatives I couldn't clarify from the documentation and I am wondering if some of you can shed some light.

Q1: From what I red, the trast region reflective algorithm can accept a user supplied gradient or both gradient and hessian. When only gradient is provided the algorithm can accept bound constraints and linear inequalities but when both gradient and hessian are supplied the algorithm can accept only bound constraints or linear inequalities, but not both. Is this correct or am I missing something?

Q2: In the interior point algorithm the user can supply the hessian as a separate function (HessFcn). However in the documentation it is not clear if this hessian function can accept more arguments than x and λ. Concequently the way the user can pass these extra parameters to the hessian (if possible) is also unclear.

Q3: The λ input in the user supplied hessian is supposed to be calculated by the fmincon. So how can I input something to a function that is not calculated yet?

To be more precise, this is my problem:

[fval, grad, hessian] = myfunc(x, data, flag) % function to be optimized
gradient = mygrad(x, data, flag) % user supplied gradient
hessian = myhess(x, data, flag) % user supplied hessian

The problem has bound constraints and inequality constraints therefore interior point is my only choice, if I want to use the hessian (Q1) but I do not know how to pass the extra parameters of the hessian, to optimset (Q2 & Q3)

Can anyboby help? I work with R2008a but I have access to R2009b also.

Thanks in advance

Manthos Vogiatzoglou wrote:
> Dear all
>
> There are some issues in optimization with user - supplied derivatives I
> couldn't clarify from the documentation and I am wondering if some of
> you can shed some light.
>
> Q1: From what I red, the trast region reflective algorithm can accept a
> user supplied gradient or both gradient and hessian. When only gradient
> is provided the algorithm can accept bound constraints and linear
> inequalities but when both gradient and hessian are supplied the
> algorithm can accept only bound constraints or linear inequalities, but
> not both. Is this correct or am I missing something?
>
> Q2: In the interior point algorithm the user can supply the hessian as a
> separate function (HessFcn). However in the documentation it is not
> clear if this hessian function can accept more arguments than x and
> λ. Concequently the way the user can pass these extra parameters to
> the hessian (if possible) is also unclear.
> Q3: The λ input in the user supplied hessian is supposed to be
> calculated by the fmincon. So how can I input something to a function
> that is not calculated yet?
> To be more precise, this is my problem:
>
> [fval, grad, hessian] = myfunc(x, data, flag) % function to be optimized
> gradient = mygrad(x, data, flag) % user supplied gradient
> hessian = myhess(x, data, flag) % user supplied hessian
>
> The problem has bound constraints and inequality constraints therefore
> interior point is my only choice, if I want to use the hessian (Q1) but
> I do not know how to pass the extra parameters of the hessian, to
> optimset (Q2 & Q3)
>
> Can anyboby help? I work with R2008a but I have access to R2009b also.
>
> Thanks in advance
I already replied to you in another thread about how to pass extra
parameters. For an example showing how to use an analytic gradient with
the interior-point algorithm, see
http://www.mathworks.com/access/helpdesk/help/toolbox/optim/ug/brn4nh7.html#bri8026
To use symbolic math functions, too, see
http://www.mathworks.com/access/helpdesk/help/toolbox/optim/ug/brn4nh7.html#brv_i_1

I tried to be very clear in the documentation about what the
trust-region-reflective algorithm allows:
http://www.mathworks.com/access/helpdesk/help/toolbox/optim/ug/f12471.html#bsbwxm7
There it states "'trust-region-reflective' requires you to provide a
gradient, and allows only bounds or linear equality constraints, but not
both." Whether or not you supply a Hessian is irrelevant to what type of
problem the algorithm can solve.

Alan Weiss
MATLAB mathematical toolbox documentation

Dear Alan

Thanks for the reply, sorry for missing your other post. Q1 and Q3 are clear but I am still note sure about Q2 (declaring the hessian function to optimset).

I have changed my hessian to add the lagrange parameter lambda:

hessian = myhess(x, lambda, data, flag)

If I understood correctly I should use the following command:

options = optimset(options,'Algorithm','interior-point',...
        'Display','iter','GradObj','on','GradConstr','on',...
        'Hessian','user-supplied','HessFcn',@(x,lambda)myhess(x, lambda, data, flag));

Am I right?

and another one question. In my problem there are no nonlinear constraints therefore the hessian of the function and the hessian of the lagrangian are the same. However the gradient of the function is not the same as the gradient of the lagrangian. When I input the gradient, this should be the gradient of the function or the gradient of the lagrangian?

Manthos Vogiatzoglou wrote:
> Dear Alan
>
> Thanks for the reply, sorry for missing your other post. Q1 and Q3 are
> clear but I am still note sure about Q2 (declaring the hessian function
> to optimset).
>
> I have changed my hessian to add the lagrange parameter lambda:
>
> hessian = myhess(x, lambda, data, flag)
>
> If I understood correctly I should use the following command:
>
> options = optimset(options,'Algorithm','interior-point',...
> 'Display','iter','GradObj','on','GradConstr','on',...
> 'Hessian','user-supplied','HessFcn',@(x,lambda)myhess(x, lambda,
> data, flag));
>
> Am I right?
>
> and another one question. In my problem there are no nonlinear
> constraints therefore the hessian of the function and the hessian of the
> lagrangian are the same. However the gradient of the function is not the
> same as the gradient of the lagrangian. When I input the gradient, this
> should be the gradient of the function or the gradient of the lagrangian?

Your options look OK to me, although I wonder why you have an instance
of options after optimset and before 'Algorithm'.

As stated in the documentation
http://www.mathworks.com/access/helpdesk/help/toolbox/optim/ug/brhkghv-3.html
the gradient you supply in the objective function should be the gradient
of the objective function.

You might want to have the toolbox check to see if your gradient is correct:
http://www.mathworks.com/access/helpdesk/help/toolbox/optim/ug/f12471.html#br5u_mf

Alan Weiss
MATLAB mathematical toolbox documentation

unfortunately my options don't work...

the objective function is: [logl, gradient, hessian, out] = AR_GARCHlogl(theta, data, distr, lags)

the hessian is the third output argument (for trust region reflective )

the hessian is: hessian = AR_GARCHhess(params, data, distr, lags) (for interior point)

I do not have non - linear constraints therefore lambda is not necessary (?)

the function that computes bounds and constraints and optimizes the objective function is:

[parameters, logl, grad, hessian] = fit_AR_GARCH(data, x0, distr, lags)

my options are:

options = optimset('Display','iter','MaxFunEvals',3000,'MaxIter',100,'TolCon',10^-16,'TolFun',10^-6,'TolX',10^-6);

options = optimset(options,'Algorithm','interior-point','GradObj','on','DerivativeCheck','on','Hessian','user-supplied');

options = optimset(options,'HessFcn',@AR_GARCHhess);

I experimented with the last line of my options. I also tried the following:

options = optimset(options,'HessFcn',@(x)AR_GARCHhess(x,data, distr, lags));

options = optimset(options,'HessFcn',@AR_GARCHhess(x,data, distr, lags));

but I always get the same error:

??? Error using ==> AR_GARCHhess
Too many input arguments.

Error in ==> C:\Program
Files\MATLAB\R2008b\toolbox\optim\optim\private\computeHessian.p>computeHessian at 23


Error in ==> C:\Program Files\MATLAB\R2008b\toolbox\optim\optim\barrier.p>barrier at 206


Error in ==> fmincon at 766
    [X,FVAL,EXITFLAG,OUTPUT,LAMBDA,GRAD,HESSIAN] =
    barrier(funfcn,X,A,B,Aeq,Beq,l,u,confcn,options_ip.HessFcn, ...

Error in ==> fit_AR_GARCH at 30
        [parameters,logl,exitflag,output,lambda,grad,hessian]=
        fmincon('AR_GARCHlogl',x0,AA,bb,[],[],lb,ub,[],options,data,distr,lags);

any ideas?? it is like the 'hessFcn' function should only have two input arguments and I am trying to pass aditional arguments to it.

I don't know what is going on here, but I suspect that your method of
passing additional parameters might be at fault. I suggest you use
nested or anonymous functions for including additional parameters:
http://www.mathworks.com/access/helpdesk/help/toolbox/optim/ug/brhkghv-7.html

I also suggest you check that your objective function is
conditionalized, since the interior-point algorithm does not accept a
Hessian as an output from the objective function:
http://www.mathworks.com/access/helpdesk/help/toolbox/optim/ug/brhkghv-3.html
"Some solvers do not use gradient or Hessian information. You should
"conditionalize" a function so that it returns just what is needed:
* f(x) alone
* Both f(x) and ∇f(x)
* All three of f(x), ∇f(x), and H(x)"

Alan Weiss
MATLAB mathematical toolbox documentation

Manthos Vogiatzoglou wrote:
> unfortunately my options don't work...
>
> the objective function is: [logl, gradient, hessian, out] =
> AR_GARCHlogl(theta, data, distr, lags)
>
> the hessian is the third output argument (for trust region reflective )
>
> the hessian is: hessian = AR_GARCHhess(params, data, distr, lags) (for
> interior point)
>
> I do not have non - linear constraints therefore lambda is not necessary
> (?)
>
> the function that computes bounds and constraints and optimizes the
> objective function is:
>
> [parameters, logl, grad, hessian] = fit_AR_GARCH(data, x0, distr, lags)
>
> my options are:
>
> options =
> optimset('Display','iter','MaxFunEvals',3000,'MaxIter',100,'TolCon',10^-16,'TolFun',10^-6,'TolX',10^-6);
>
>
> options =
> optimset(options,'Algorithm','interior-point','GradObj','on','DerivativeCheck','on','Hessian','user-supplied');
>
>
> options = optimset(options,'HessFcn',@AR_GARCHhess);
>
> I experimented with the last line of my options. I also tried the
> following:
>
> options = optimset(options,'HessFcn',@(x)AR_GARCHhess(x,data, distr,
> lags));
>
> options = optimset(options,'HessFcn',@AR_GARCHhess(x,data, distr, lags));
>
> but I always get the same error:
>
> ??? Error using ==> AR_GARCHhess
> Too many input arguments.
>
> Error in ==> C:\Program
> Files\MATLAB\R2008b\toolbox\optim\optim\private\computeHessian.p>computeHessian
> at 23
>
>
> Error in ==> C:\Program
> Files\MATLAB\R2008b\toolbox\optim\optim\barrier.p>barrier at 206
>
>
> Error in ==> fmincon at 766
> [X,FVAL,EXITFLAG,OUTPUT,LAMBDA,GRAD,HESSIAN] =
> barrier(funfcn,X,A,B,Aeq,Beq,l,u,confcn,options_ip.HessFcn, ...
>
> Error in ==> fit_AR_GARCH at 30
> [parameters,logl,exitflag,output,lambda,grad,hessian]=
>
> fmincon('AR_GARCHlogl',x0,AA,bb,[],[],lb,ub,[],options,data,distr,lags);
>
> any ideas?? it is like the 'hessFcn' function should only have two input
> arguments and I am trying to pass aditional arguments to it.

Dear Alan

Thanx for keep trying to help me. The only reason I tried this... function handle option is because you and I thing Loren suggested me to do. I do not like function handles and I do not use them unless I have to.
To tell you the truth I am not yet convinced that the HessFcn can accept more arguments than x and lambda without causing problems to optimset. I am only trying because you believe it can and obniously you know better than me.
I have red the documentation, there is no instanse of a user supplied hessian that needs more than two arguments and concequently the is nothing on how you pass these extra arguments via optimset. Therefore what I would like to see here is some applied suggestions about how my optimset should be constructed, with paradigms, if possible.

For your other suggestion I will create a second objective which will actually be identical to the first but will produce only one output argument, the objective value, without grad or hessian. The new one will be used with interior point, if we manage it to work and the old one with trust region reflective.

Best

Manthos

Manthos Vogiatzoglou wrote:
> Dear Alan
>
> Thanx for keep trying to help me. The only reason I tried this...
> function handle option is because you and I thing Loren suggested me to
> do. I do not like function handles and I do not use them unless I have
> to. To tell you the truth I am not yet convinced that the HessFcn can
> accept more arguments than x and lambda without causing problems to
> optimset. I am only trying because you believe it can and obniously you
> know better than me.
> I have red the documentation, there is no instanse of a user supplied
> hessian that needs more than two arguments and concequently the is
> nothing on how you pass these extra arguments via optimset. Therefore
> what I would like to see here is some applied suggestions about how my
> optimset should be constructed, with paradigms, if possible.
>
> For your other suggestion I will create a second objective which will
> actually be identical to the first but will produce only one output
> argument, the objective value, without grad or hessian. The new one will
> be used with interior point, if we manage it to work and the old one
> with trust region reflective.
>
> Best
>
> Manthos
Please write your objective function with a gradient as well for
interior point! It is not hard to conditionalize your objective
function, I don't think you should have two different ones.

the only thing that concerns me Alan is the issue with passing the extra hessian parameters to optimset. The issue on the objective is ruther trivial since there are 100 ways to solve it