Thread Subject: Give me a Regression Problem

On Jul 16, 5:37 pm, paulvbi...@gmail.com wrote:
> > Although our methods are not optimal, neither Baldrick nor me has
> > found grounds for eliminating x3.
>
> **********************************************************
>
> Dear Greg
>
> why not drop x3 ONLY and see if you get an improvement with your NN
>
> in a way I do not like this "choosing" with linear methods

Linear? ...

Linear in coefficients so the canned MATLAB stagewise selection
function can be used. However, NONLINEAR in variables by
including cross-products and higher powers ... very quick and
useful. Furthermore, since the simpler models have fewer degrees
of freedom, they tend to pinpoint the best variables for
consideration.

For the concrete data set R^2 ~ 0.6, 0.8 and 0.9 were obtained
from linear, quadratic and NN models.

That's why I always compare Linear and Quadratic polynomial
models before designing NNs. It is very quick and is part of my
pretraining data familiarization ritual that considers ranks of input
and output data matrices, x-x and y-x correlations, plotting, PCA
and clustering.

> and I want to look at bit more closely at Phil's suggestion of using
> the NN for picking and choosing the variable's importance

Certainly...the NN always has the final say.

However, searching through the different combinations from scratch
can get expensive with backward search algorithms; even when they
are stepwise greedy instead of stagewise.

This has lead to quicker algorithms based on path-weight products,
sensitivity, and other concepts as well as the longer algorithms based
on GA.

I have never used any of these so I cannot summarize their pros
and cons.

However, regardless of which NN subset selection method is used,
the Linear/Quadratic warmup can be used as a consistency check
and may be useful for reducing the number of variable combinations
considered.

> the NN has more power and should be the best "spokesperson" for who
> should be in and who should be out and not our favourite linear method
> or thereto

As Bill Clinton said:

 "It depends on what you mean by "linear"

... sums of certain nonlinear combinations are known to be universal
approximators.

Hope this helps.

Greg