Got Questions? Get Answers.
Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Stochastic gradient boosting of decision tree ensemble?

Subject: Stochastic gradient boosting of decision tree ensemble?

From: Evan Ruzanski

Date: 1 Nov, 2012 02:17:09

Message: 1 of 2

Hello,

Does anyone know of a method to implement stochastic gradient boosting (Friedman, 1999) for regularization of an 'LSBoost' decision tree ensemble for regression? Regularization via 'regularize' and 'shrink' is apparent, but it appears that either 'LSBoost' or 'Bag' can be selected for a regression ensemble created with the Statistics Toolbox, not both as the stochastic gradient boosting method suggests.

Many thanks!

Subject: Stochastic gradient boosting of decision tree ensemble?

From: Ilya Narsky

Date: 1 Nov, 2012 12:53:11

Message: 2 of 2

"Evan Ruzanski" <ruzanski@alumni.colostate.edu> wrote in message
news:k6sm35$fuf$1@newscl01ah.mathworks.com...
> Hello,
>
> Does anyone know of a method to implement stochastic gradient boosting
> (Friedman, 1999) for regularization of an 'LSBoost' decision tree ensemble
> for regression? Regularization via 'regularize' and 'shrink' is apparent,
> but it appears that either 'LSBoost' or 'Bag' can be selected for a
> regression ensemble created with the Statistics Toolbox, not both as the
> stochastic gradient boosting method suggests.
>
> Many thanks!

Stochastic gradient boosting is already provided in fitensemble under
different names. The LS_Boost algorithm described by Friedman is the LSBoost
algorithm you get from fitensemble. To apply shrinkage described in the
regularization section of Friedman's paper, set the 'LearnRate' parameter
passed to fitensemble to something less than 1. This shrinkage is applied in
the process of ensemble construction. Bagging provided by 'Bag' in
fitensemble does not fall in the gradient boosting paradigm, and this type
of shrinkage does not apply to ensembles of bagged trees.

Methods regularize() and shrink() provide post-fitting, that is,
regularization by lasso after the ensemble is constructed. As I recall,
lasso for ensembles is not described in that particular Friedman's paper (I
could be wrong though). -Ilya

Tags for this Thread

No tags are associated with this thread.

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us