On May 24, 11:28 pm, "Shiguo " <jiangshi...@gmail.com> wrote:
> Nice input John. I totally agree with you. I've read many of your posts and downloaded your files.
>
> John D'Errico <?derr...@flare.n?e?t> wrote in message <?derrico440D83.10305324122...@news.newsguy.com>...
> > In article <eecd15d...@webx.raydaftYaTP>,
> > Maurice <p12tpe1.nos...@yahoo.co.uk> wrote:
>
> > > Hi,
>
> > > I've noticed that MATLAB is keen on using the "norm of residuals" as
> > > a measure of how well data fits a polynomial. But I (and more
> > > importantly my boss...) have learned to use the Rsquared value as a
> > > measure of "goodness of fit".
>
> > > While the Rsquared value indicates a better fit as it's value
> > > approaches 1, the norm(V,2) indicates a better fit as its value
> > > approaches 0.
>
> > > Is there a relation between the two? For instance can I say that
> > > Rsquared=1norm(V,2)?
>
> > > Anyway, is there a function that lets me calculate the Rsquared
> > > value for a given set of data and residuals?
>
> > Interesting. I tend to ignore R^2. It has dangers
> > that many do not realize. For example, R^2 is
> > invalid when there is no constant term in your
> > model. (One can modify R^2 for models with no
> > constant term, but it is inconvenient.)
>
> > Suppose I told you a model for some data had an R^2
> > of 0.975. Is the model adequate for my purposes?
> > What if the R^2 is 0.999?
>
> > Your response should be that you have no idea.
> > My data may well be very accurate, so an R^2 of
> > only 0.975 may well be a relatively poor fit.
> > I needed to tell you what my goals were in fitting
> > this specific model before you have any idea of
> > what the fit quality should be. I also needed to
> > tell you how good the data is, etc. Since R^2 is
> > a dimensionless number, you really cannot say
> > anything.
>
> > I'll argue that only by looking at the residuals,
> > plotting the curve fit, etc., can you really
> > assess how good a fit is. Mononumerosis, i.e.,
> > the use of a single statistic to gauge the quality
> > of a model, in any fitting scheme is dangerous.
>
> > Having climbed on top of that soapbox, I'll step
> > down for a moment. What is the definition of R^2?
> > How would you compute it? Don't tell me that you
> > have no idea. If you do not know then you have
> > no business in using R^2 in the first place.
> > The question you ask above tells me you do not
> > know. UNDERSTAND YOUR TOOLS! Otherwise it is all
> > a complete waste of time. So, I repeat. What is
> > R^2?
>
> > R^2 = 1  ss_regression/ss_total
>
> > The idea is you divide the regression sum of
> > squares by the total sum of squares, then subtract
> > it from 1. What is the regression sum of squares?
> > Its the square of the norm of your residuals.
> > How about the total sum of squares? Does it change
> > due to your regression model? No. So, minimizing
> > the norm of the residuals is the mathematical
> > equivalent of maximizing R^2. Except...
>
> > Practically, you can see subtractive cancellation
> > problems in computing R^2. This can be crucially
> > important in a nonlinear optimization near the
> > solution. (If your model is nonlinear. If it is
> > a polynomial model, then no iterative scheme is
> > required.)
>
> > The point of all this is to suggest that you
> > should not look at any single measure of fit.
> > Learn data analysis and statistical modeling
> > so that you know when and how to apply the proper
> > measures of fit. Learn to understand your model.
> > Learn about your data and its sources of error.
> > If you do all this, then R^2 will seem trivial
> > to you too.
>
> > You never know. You might even end up impressing
> > that boss of yours one day.
>
> > HTH,
> > John D'Errico
I have been designing neural networks for ~40 yrs and have been
contributing to comp.ai.neuralnets and comp.softsys.matlab since
~1997. For those last 15 years I have seen every imaginable
perturbation of misuse and abuse associated with designing NNs for
classification and regression.
For example:
1. No regard for the physical and/or mathematical source of the data.
2. No attempt to visualize the data re clustering and/or I/O variable
plots.
3. No consideration of multiple input scales, multiple output scales
and outliers.
4. No consideration of simpler models; e.g.,
a. Constant (output = mean(target data))
b. Linear
5. No attempt to simplify the model by reducing the number of hidden
nodes.
6. No attempt to simplify the model by removing insignificant and/or
redundant inputs.
7. No degreeoffreedom adjustments when characterizing the model via
performance on training data.
Quite often the reason is obvious:
"I'm in a hurry to get an answer and hand in the assignment before the
deadline."
However, I get the feeling that some of the blame rests on the
shoulders of teachers and/or advisors.
End of rant.
I often interpret R^2 in terms of the normalized meansquared error
NMSE = MSE/MSE00 = 1  R^2
where MSE00 is the mse obtained from the abovementioned Constant
Model. For nontraining data, the MSE is proportional to the sumsquare
error, SSE, via
MSE = SSE/Neq and MSE00 = SSE00/Neq
where
Neq = N*O
is the total number of scalar training equations (N = number of data
points and O = number of output variables). Consequently, R^2 can
also be interpreted in terms of the normalized sumsquarederror
NSSE = SSE/SSE00 = 1  R^2.
However, if the training data is used to obtain the measures, the
corresponding optimistic bias should be mitigated via degreeof
freedomadjustments. For example,
MSEa = SSE/(NeqNw) and MSE00a = SSE00/(NeqNw00)
Where Nw represents the loss in degreesoffreedom by using the same
data to estimate both Nw weights and corresponding model errors. It
should be noted that
1. MSEa > inf as Nw > Neq and is undefined for Nw >= Neq
2. Nw00 = O and MSE00a = mean(training data output variances).
The resulting adjustments yield
NMSEa = MSEa/MSE00a = 1  Ra^2.
I'll let the reader contemplate the direct relationship between R^2
and Ra^2.
I can't find anything interesting to say about it.
Questions:
1. Is R considered to be the multivariate correlation coefficient?
2. If so how is Ra interpreted?
Greg
