How to regularize the Levenberg-Marquardt algorithm in lsqnonlin?

I need to solve a non-linear least square problem using Levenberg-Marquardt (LM) method but with some additional terms in the objective function (see below)

lsqnonlin function in MATLAB optimization toolbox only takes ( as input and updates the parameters using the normal equations for the least-square problem [1]. Is there any way to solve such problems using lsqnonlin?

[1] Moré, J.J., 1978. The Levenberg-Marquardt algorithm: implementation and theory. In Numerical analysis (pp. 105-116). Springer, Berlin, Heidelberg.

Answer 1

[Matt J]

You can append the square root of the regularization term to the vector . However, your objective function does not look differentiable, which Levenberg-Marquardt assumes, so you might have to use a differentiable approximation to the TV term.

[Abhilash Awasthi]

Thank you for the reply.

The regularization terms can be differentiated with respect to the parameters (θ) to give:

I can approximate the second term on RHS and represent the same in the form of a vector. The user defined jacobian can also be specified.

Could you please elaborate more on appending the () with the regularization term?

[Matt J]

The regularization terms can be differentiated with respect to the parameters (θ) to give:

As you can see though, you have a potential division by zero, wherever =0.

Could you please elaborate more on appending the () with the regularization term?

Since you Accepted the answer, I assume you figured it out?

[Torsten]

@Abhilash Awasthi

What is Omega and how does the integrand depend on an element omega in Omega ?

[Abhilash Awasthi]

@Matt J sorry I'm new here so accepted the answer by mistake.

Since I am modlling hetergeneous material so it is highly unlikely that I would be getting . The regularization term can be approximated to a finite value. Could you please elaborate using lsqnonlin here?

@Torsten I am using Finite Element Method to solve the above problem. Ω denotes the whole domain while dΩ is a smaller element (subset) of the whole domain.

Thank you.

[Torsten]

@Abhilash Awasthi

And you want to determine the solution "theta" in a [(2d) (3d) ?] domain Omega that minimizes the above functional Pi ?

[Abhilash Awasthi]

@Torsten the domain could be 2D or 3D. The aim is to find a set of parameters θ that minimize the difference with determined by solving a partial differential equation parametrized by θ.

[Torsten]

And in the end, PI is one scalar value - some sort of cumulated approximation error over complete Omega ?

In this case, fmincon is better suited to solve your problem compared to lsqnonlin, I guess, because PI already performs summation of the errors squared in the discretization points whereas lsqnonlin would work with the errors in the discretization points separately.

[Matt J]

@Abhilash Awasthi If you construct the vector-valued function,

F(theta)=[u(theta)-um;
           sqrt(2*regularization(theta))]

and give this F(theta) to lsqnonlin as the objective function, then lsqnonlin will minimize the regularized objective function H(theta) that you have posted.