Optimization options parameter 'findiffrelstep'

I want to know exactly what this parameter is. i have two variables one is in the range 10 to 25 and other in the range 1000 to 2000... I am wondering whether setting findiffrelstep will improve my solution performance. Thanks in advance. Pappu murthy

Answer 1

[Matt J]

The description at the FMINUNC webpage is pretty detailed.

[Matt J]

Pappu Murthy Commented

I don't think the explanation given is good enough for me. Because I am not an expert in this field. For. e.g. the description says "delta = v.*sign(x).*max(abs(x),TypicalX);" Delta I am assuming the step length DX that is normally used to compute function values and then divide the difference by DX. However, I am not clear on "abs(X)" does it mean of the two parameters I take abs value of both the parameters? Also "TypicalX" what is that for my example is that my parameter 1 which is in range 10 to 20 or parameter 2 which varies in the range 1000 to 2000. See so much confusion and I tried some values based on my limited understanding but I got nothing . No improvement. So in my case of say if I gave [10, 1400] as my starting point, what should I use "v" for 'findiffrelstep' parameter?

[Matt J]

does it mean of the two parameters I take abs value of both the parameters?

Not you. You supply the vector v and fminunc will evaluate the expression

delta = v.*sign(x).*max(abs(x),TypicalX)

to compute the vector of step lengths, delta. Here, x is a vector and the expression is executed exactly as shown, I imagine. TypicalX is another optimization option settable by optimoptions.

If you think the finite differences are giving a bad approximation of the derivatives, then just set v smaller.

[Pappu Murthy]

I set [0.05 0.05] as FindiffRelStep. My variables are like [10,1100] so my step should be [.5 55] however, for each iteration i noticed the first variable moves pretty good but the second one hardly moves at all .. like for example after some 6 iterations the variable changes from 1100 to 1100.8 or so.. and this is always the same no matter where I start. For example if I started [15, 1400] then my second variable changes from 1400 to 1400.6 or so. The first one changes pretty good and settles around 25.. Always it is trying to move the first one but hardly changes the second one.Not sure what kind of TypicalX and FinDiffRelStep I am supposed to use in order to give both variables equal importance. I hope this calrifies my question quite a bit.

[Matt J]

I think it is best if you show your objective function and your code and start talking in more detail about why you think you need to play with FinDiffRelStep.

I set [0.05 0.05] as FindiffRelStep. My variables are like [10,1100] so my step should be [.5 55]

I don't really follow that logic. The magnitude needed for delta is not related to the size of your variables. It is related to the size of the derivatives of your objective function. For example, if you're approximating the derivative of a 1D function f(x), the second order term in the Taylor remainder theorem gives you the error,

(f(x+delta)-f(x))/delta=f'(x)+f''(y)*delta

So, here the error term in the finite difference approximation of f'(x) is f''(y)*delta, where y is a point on the line between x and x+delta. You need to choose delta small enough to make the error term negligible. This requires some pre-analysis and some knowledge of a global bound on the second derivative magnitudes |f''(x)|.

[Pappu Murthy]

Ok I understand what you are saying. However, my objective function is not simple to quote here. It requires lots of analysis using a DOS based program that is run from Matlab script. I need to fit a nonlinear fatigue law with two parameters in least square error sense. The two variables are the two parameters of the fatigue law. And since I noticed that second parameter hardly changing at all no matter where I started my initial point, I thought the parameter findiffrelstep will do the trick. I must be barking on the wrong tree clearly according to your explanation. I don't know what else I can say about my problem really. However, scaling of the parameters is another optioin, and playing with the parameter "TypicalX" might be another option. But i really dont have much experience or any idea of those two.

[Matt J]

Well, I can venture guesses. Are you sure the function is differentiable? Failure to move far from the initial point is often linked to non-smoothness in the objective functions. Common mistakes are putting quantizing operations like floor, fix,ceil, round in your objective function. Nearest-neighbor interpolation is similarly bad, since it contains rounding. You must also avoid common non-differentiable operators like sqrt, abs, etc... People doing least squares minimization often make the mistake of putting in sqrt() operations unnecessarily instead of working purely with the squares.

Another suggestion is to try fminsearch, which should work well enough since you only have 2 parameters. fminsearch doesn't use derivatives, so you don't have to fuss with finite difference parameters. However, quantizing operations will still cause problems, if you have them.

[Pappu Murthy]

Thanks for the valueable insights you are giving. I am sure my function is not differentiable in closed form. It involves a lot of computations which are done using a complicated fortran program as I mentioned earlie. That said, I have no reasons to believe there are 'undifferentiable' aspects to the compuations. I do have actually 5 parameters. The two parameter case is a sub case for testing purposes. I can try "fminsearch" if that can do the job. However, by scaling I am able to move properly all the 5 parameters now. I just divied one parameter by 100 and that seemed to do that job. Not sure though, whether I am doing this correctly. Now the question is i dod have sqrt in my least squares operation. But why should it matter? It is not doing a closed form differentiation anyway. So for finite difference it should not matter at all. Right?

[Matt J]

It is not doing a closed form differentiation anyway. So for finite difference it should not matter at all. Right?

No, the theory of the algorithms assumes that the function is twice continuously differentiable. I'm not sure what distinction you're drawing between "closed-form differentiable" and other kinds. Either the function is continuously differentiable, or it's not. If it is not, the assumptions of the algorithm are violated, and that's the end of the story.

Finite difference operations are just a way of approximating the true derivative. The algorithm performs better if you provide an analytical gradient calculation (whether or not it is closed form). If the true derivative doesn't exist, the finite difference will not produce anything more useful. It will produce some unstable value greatly depending on the delta step size you use.