Michelle-
These results are entirely consistent with how classification trees work. Simply rescaling each of the inputs by multiplying them with different coefficients should have no effect on the tree.
For exactly why this is, I'd recommend Breiman's book (which is referenced in the doc), but the short answer is that trees sort each predictor's observations and try a candidate split within each of the gaps. The tree will then select the split that gives the "best" splitting criterion (and that's an entirely different discussion). Scaling the predictor only serves to scale this process, but it doesn't fundamentally change the results.
As an example: suppose we have a simple set of obervations where the predictor has been measured at 1, 2, 4, and 10. The tree will try splits at 1.5, 3, and 7. Let's say that the "best" split is at 7.
Now we go ahead and rescale this input-- mulitply it by 100 or some other coefficient. Now, the tree tries splits at 150, 300, and 700, and it will still select the split at 700. Rescaling doesn't change anything.
Now, if we were to cleverly create _new_ predictors out of a well-chosen combination (linear or otherwise) of our existing predictors, then that certainly would change the tree's performance. For instance, make a 6th predictor in your X from Altman's coefficient's times your original X-- then you might get some interesting results.
Hi Michael,
I've noticed a very curious result of the treebagger, and was wondering if you have had experience with this.
I send a matrix X which has 5 columns of a variety of accounting data. I also send Y, which is a vector of credit ratings. I have about 3000 rows of data.
I understand that the bankruptcy academic community have done extensive research to determine optimal coefficients to predict bankruptcy. I thought it would be interesting to see what the impacts of the coefficients are on the bagging results. So I created a vector, coeff, and multiplied each row of X by the parameters in coeff.
for example: Altman's coefficients are {0.717 0.847 3.107 0.42 0.998}
Curiously, varying the coefficients has NO effect on the oobErrors. I've run exhaustive loops to vary all of the coefficients to track this down.
It seems like in the case of the limit where the coefficient goes to zero, there should be an effect.
thanks for any insights.
Thank you Michael for the clarification on the use of the predict vs oobPredict functions. I am aware of the benefits of the treebagger in automatically splitting the data, I just wanted to be able to see the individual bits.
Your point about the histogram is well taken, but I do think histograms have some value in seeing the data distribution before and after, even if it doesn't provide insight into the performance of the bagger. The various methods you highlighted in your code are great for this.
I am working with real credit data, so being a visual person, I like to see what the ratings distribution looks like at the beginning/end of the process. I should think it would be curious if the classified data had a completely different distribution.
Thanks again for your demo and webinar, I'm finding it incredibly helpful.
Michelle -
I'm afriad that I don't understand what histograms will do for you in this case. One typically matches the "actual" outputs to the model's "predicted" outputs and compares the difference between them in some way to assess the model's performance. Confusion matrices, ROC curves, and other techniques are commonly used to do this. Histograms are not used because they can hide a lot of information. Consider this simple case of classifying data that can take values of either "1" or "2":
% The "actual" data:
Y = [1 2 1 2 2 1];
% The "predicted" data, arrived at through some model:
Y_Pred = [2 1 2 1 1 2];
Most would argue that this "model" is terrible: it has a 100% misclassification rate! In spite of this, hist(Y) and hist(Y_Pred) give the exact same plot.
That visualization concern aside, I think there is some confusion about how Bagging works. In this example, the "actual" ratings are Y, and every observation is used for training in some way. So, Y also represents the training ratings. One of the strengths of ensemble methods like Bagging is that it's not necessary to manually split the data into training and validation sets: you can have your proverbial cake and eat it, too. The out-of-bag errors in this case, though, have a special significance that is too much to explain here-- you should check the doc or (better yet) Breiman's original article for more details on that. Suffice to say, you should not use the OOB errors in the way that you seem to be using them here. If you're looking for the ensemble's predicted ratings, they are simply found by
Y_Pred = predict(b,X);
Comment only