Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Biased Solution to least sqaures optimization?

Subject: Biased Solution to least sqaures optimization?

From: Eric Diaz

Date: 6 Mar, 2010 02:25:23

Message: 1 of 13

Hello mathworks community,

I am hoping that you might be able to help a bit with a problem.

My broad understanding is that many of the least squares functions, linear and nonlinear, act to essentialy minimize the norm of the residuals with the assumption that the noise is gaussian.

Now, if I know, a priori, that my noise is not gaussian, then what alternatives do I have to introducing a known bias into my solution.

Eric Diaz

Subject: Biased Solution to least sqaures optimization?

From: John D'Errico

Date: 6 Mar, 2010 02:38:22

Message: 2 of 13

"Eric Diaz" <eric.diaz@gmail.com> wrote in message <hmseej$o73$1@fred.mathworks.com>...
> Hello mathworks community,
>
> I am hoping that you might be able to help a bit with a problem.
>
> My broad understanding is that many of the least squares functions, linear and nonlinear, act to essentialy minimize the norm of the residuals with the assumption that the noise is gaussian.
>
> Now, if I know, a priori, that my noise is not gaussian, then what alternatives do I have to introducing a known bias into my solution.
>
> Eric Diaz

If it is NOT? Sorry, but that is not how it works. IF
you know what the actual underlying distribution is,
THEN you can do something. But merely saying it is
not Gaussian is a meaningless statement.

If you do have some knowledge of the "true" noise
distribution, then you could use maximum likelihood
estimation to solve the problem.

John

Subject: Biased Solution to least sqaures optimization?

From: Eric Diaz

Date: 6 Mar, 2010 04:15:09

Message: 3 of 13

Hi John,

Thanks for your response. I'm glad to have gotten one so quickly in fact. BTW, I noticed that you still haven't updated the error in your pleas wrapper for optimtips that I emailed you about.

Anyway, so, in fact, I do know, a priori, that my noise is NOT gaussian. I am not sure, however, what you mean by the statement below.

> But merely saying it is not Gaussian is a meaningless statement.

And so, I thought that MLE would be the answer, since I have already investigated the issue slightly. Is there any MLE alogorithm that I could use in matlab that will allow me to choose what my "true" noise distribution? I am not very familiar with them.

> If you do have some knowledge of the "true" noise distribution, then you could use
> maximum likelihood estimation to solve the problem.
>
> John

Subject: Biased Solution to least sqaures optimization?

From: Matt J

Date: 6 Mar, 2010 14:39:06

Message: 4 of 13

"Eric Diaz" <eric.diaz@gmail.com> wrote in message <hmseej$o73$1@fred.mathworks.com>...
 
> Now, if I know, a priori, that my noise is not gaussian, then what alternatives do I have to introducing a known bias into my solution.
=========================

If the bias introduced is known, why not just subtract it off?

Apart from that, I'll just point out that if you stick to linear least squares, there will be no bias in the estimate (just sub-optimal variance). That's assuming your noise is zero mean, of course.

Subject: Biased Solution to least sqaures optimization?

From: Eric Diaz

Date: 7 Mar, 2010 00:03:06

Message: 5 of 13

Hi Matt,

Well, if I may try to clarify what I meant by the bias introduced into the solution. I meant that least squares algorithm, in my case the nonlinear least square curve fit (but the argument also applies to the linear case), would be finding the "best" solution, however that "best" solution would be biased by the fact that it was trying to minimize the residual norm of noise with a gaussian distribution and zero mean.

In my case, however, I know that my noise distribution is not gaussian and also not zero mean. In addition, to add to the complexity, I know that my noise distribution is dependent on the signal itself, i.e., it changes as a function of when there is less or more signal. So, there is a distribution of noise distributions depending on how strong the signal is.

I am not sure if I am able to just subtract it off nor am I sure of exactly what you mean by that. Do you mean subtract information from the data or do you mean subtract the bias after the algorithm is finished? If the prior, I typically don't like doing that. However, if the latter, I am not sure if I would be able to, since the noise distribution is dependent on the signal strength, which would require subtracting a family of noise distributions from the solutions. Too complicated, I think.

At any rate, thanks for your thoughts Matt!

Eric Diaz

"Matt J " <mattjacREMOVE@THISieee.spam> wrote in message <hmtpea$e7e$1@fred.mathworks.com>...
> "Eric Diaz" <eric.diaz@gmail.com> wrote in message <hmseej$o73$1@fred.mathworks.com>...
>
> > Now, if I know, a priori, that my noise is not gaussian, then what alternatives do I have to introducing a known bias into my solution.
> =========================
>
> If the bias introduced is known, why not just subtract it off?
>
> Apart from that, I'll just point out that if you stick to linear least squares, there will be no bias in the estimate (just sub-optimal variance). That's assuming your noise is zero mean, of course.

Subject: Biased Solution to least sqaures optimization?

From: Eric Diaz

Date: 7 Mar, 2010 19:35:22

Message: 6 of 13

Bumpity bump bump...wow, this forum sure is active! Any input is greatly appreciated!

Thanks in advance,

Eric Diaz

Subject: Biased Solution to least sqaures optimization?

From: Matt J

Date: 8 Mar, 2010 02:40:21

Message: 7 of 13

"Eric Diaz" <eric.diaz@gmail.com> wrote in message <hn0v5q$4mc$1@fred.mathworks.com>...




I am not sure if I am able to just subtract it off nor am I sure of exactly what you mean by that.
=================

In your original post, you said your bias was a known quantity (for some unclear reason). It is now clear that it is unknown. If it were a known quantity, you could obviously subtract it off the estimate to obtain an unbiased result.



> Bumpity bump bump...wow, this forum sure is active! Any input is greatly appreciated!
=================

It's not clear what challenges you're having with the MLE approach that John suggested. You need to write down the distribution of your signal measurements as a function of the parameters you are trying to estimate and maximize this function.

Subject: Biased Solution to least sqaures optimization?

From: Eric Diaz

Date: 8 Mar, 2010 03:23:05

Message: 8 of 13

My apologies,

In my original post, what I meant to say was not that my noise was a "known quantity". In fact, what I said was that I know that my noise is Not Gaussian. Which by that, I was trying to communicate that I know what my noise is not rather than what it is.

I am unclear with the MLE approach, simply because I have never used it. I understood from the literature that it was probably the route I should take, however, considering how dependent I am on the lsqnonlin algorithm, I thought I would ask for some help or perhaps compare/contrast between that and an MLE approach.

Thanks again in advance,

Eric Diaz


"Matt J " <mattjacREMOVE@THISieee.spam> wrote in message <hn1o2l$rtg$1@fred.mathworks.com>...
> "Eric Diaz" <eric.diaz@gmail.com> wrote in message <hn0v5q$4mc$1@fred.mathworks.com>...
>
>
>
>
> I am not sure if I am able to just subtract it off nor am I sure of exactly what you mean by that.
> =================
>
> In your original post, you said your bias was a known quantity (for some unclear reason). It is now clear that it is unknown. If it were a known quantity, you could obviously subtract it off the estimate to obtain an unbiased result.
>
>
>
> > Bumpity bump bump...wow, this forum sure is active! Any input is greatly appreciated!
> =================
>
> It's not clear what challenges you're having with the MLE approach that John suggested. You need to write down the distribution of your signal measurements as a function of the parameters you are trying to estimate and maximize this function.

Subject: Biased Solution to least sqaures optimization?

From: Eric Diaz

Date: 8 Mar, 2010 07:30:15

Message: 9 of 13

Hi mathworks community,

I guess this post has become confusing starting off with my first post and followed subsequently with perplexing posts on the meaninglessness of my question, and statements about what I did or didn't say or know.

So, to set the record straight, my question is...

Would I need to use a maximum likelihood estimation approach (rather than non-linear least squares) to extract the "true" parameters from a signal from which I know the noise distribution is not Gaussian, if I can determine what the actual distribution of my noise is?

If so, can anyone point me in the right direction, as I have never used an MLE approach and don't know how it compares with least squares in terms of accuracy?

Thanks in advance,

Eric

"Eric Diaz" <eric.diaz@gmail.com> wrote in message <hn1qip$4m4$1@fred.mathworks.com>...
> My apologies,
>
> In my original post, what I meant to say was not that my noise was a "known quantity". In fact, what I said was that I know that my noise is Not Gaussian. Which by that, I was trying to communicate that I know what my noise is not rather than what it is.
>
> I am unclear with the MLE approach, simply because I have never used it. I understood from the literature that it was probably the route I should take, however, considering how dependent I am on the lsqnonlin algorithm, I thought I would ask for some help or perhaps compare/contrast between that and an MLE approach.
>
> Thanks again in advance,
>
> Eric Diaz
>
>
> "Matt J " <mattjacREMOVE@THISieee.spam> wrote in message <hn1o2l$rtg$1@fred.mathworks.com>...
> > "Eric Diaz" <eric.diaz@gmail.com> wrote in message <hn0v5q$4mc$1@fred.mathworks.com>...
> >
> >
> >
> >
> > I am not sure if I am able to just subtract it off nor am I sure of exactly what you mean by that.
> > =================
> >
> > In your original post, you said your bias was a known quantity (for some unclear reason). It is now clear that it is unknown. If it were a known quantity, you could obviously subtract it off the estimate to obtain an unbiased result.
> >
> >
> >
> > > Bumpity bump bump...wow, this forum sure is active! Any input is greatly appreciated!
> > =================
> >
> > It's not clear what challenges you're having with the MLE approach that John suggested. You need to write down the distribution of your signal measurements as a function of the parameters you are trying to estimate and maximize this function.

Subject: Biased Solution to least sqaures optimization?

From: Matt J

Date: 8 Mar, 2010 08:47:07

Message: 10 of 13

"Eric Diaz" <eric.diaz@gmail.com> wrote in message <hn2927$cqn$1@fred.mathworks.com>...

> Would I need to use a maximum likelihood estimation approach (rather than non-linear least squares) to extract the "true" parameters from a signal from which I know the noise distribution is not Gaussian, if I can determine what the actual distribution of my noise is?
======================

Using MLE is likely to reduce the variance of your parameter estimate, and likely also to reduce bias if your measurements consist of many i.i.d. samples.
 
> If so, can anyone point me in the right direction, as I have never used an MLE approach and don't know how it compares with least squares in terms of accuracy?
==================

http://en.wikipedia.org/wiki/Maximum_likelihood

Subject: Biased Solution to least sqaures optimization?

From: Eric Diaz

Date: 9 Mar, 2010 20:30:23

Message: 11 of 13

Hi Matt,

Thanks for your input. That's what I was thinking. Now, if only I can figure out how to perform MLE in matlab.

Eric

"Matt J " <mattjacREMOVE@THISieee.spam> wrote in message <hn2dib$qrq$1@fred.mathworks.com>...
> "Eric Diaz" <eric.diaz@gmail.com> wrote in message <hn2927$cqn$1@fred.mathworks.com>...
>
> > Would I need to use a maximum likelihood estimation approach (rather than non-linear least squares) to extract the "true" parameters from a signal from which I know the noise distribution is not Gaussian, if I can determine what the actual distribution of my noise is?
> ======================
>
> Using MLE is likely to reduce the variance of your parameter estimate, and likely also to reduce bias if your measurements consist of many i.i.d. samples.
>
> > If so, can anyone point me in the right direction, as I have never used an MLE approach and don't know how it compares with least squares in terms of accuracy?
> ==================
>
> http://en.wikipedia.org/wiki/Maximum_likelihood

Subject: Biased Solution to least sqaures optimization?

From: Matt J

Date: 9 Mar, 2010 20:59:07

Message: 12 of 13

"Eric Diaz" <eric.diaz@gmail.com> wrote in message <hn6b4v$2al$1@fred.mathworks.com>...
> Hi Matt,
>
> Thanks for your input. That's what I was thinking. Now, if only I can figure out how to perform MLE in matlab.
====

It's basically a function minimization problem. If you have the Optimization Toolbox, you can use fmincon(), fminunc(), or similar. I believe there are also MLE tools in the Statistics Toolbox as well, but I'm less familiar with those.

Subject: Biased Solution to least sqaures optimization?

From: Matt J

Date: 9 Mar, 2010 21:04:06

Message: 13 of 13

"Eric Diaz" <eric.diaz@gmail.com> wrote in message <hn6b4v$2al$1@fred.mathworks.com>...
> Hi Matt,
>
> Thanks for your input. That's what I was thinking. Now, if only I can figure out how to perform MLE in matlab.
====

It's basically a function minimization problem. If you have the Optimization Toolbox, you can use fmincon(), fminunc(), or similar. I believe there are also MLE tools in the Statistics Toolbox as well, but I'm less familiar with those.

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us