fitcensemble
Fit ensemble of learners for classification
Syntax
Description
returns the trained classification ensemble model object
(Mdl
= fitcensemble(Tbl
,ResponseVarName
)Mdl
) that contains the results of boosting 100
classification trees and the predictor and response data in the table
Tbl
. ResponseVarName
is the name of
the response variable in Tbl
. By default,
fitcensemble
uses LogitBoost for binary classification
and AdaBoostM2 for multiclass classification.
applies Mdl
= fitcensemble(Tbl
,formula
)formula
to fit the model to the predictor and
response data in the table Tbl
. formula
is
an explanatory model of the response and a subset of predictor variables in
Tbl
used to fit Mdl
. For example,
'Y~X1+X2+X3'
fits the response variable
Tbl.Y
as a function of the predictor variables
Tbl.X1
, Tbl.X2
, and
Tbl.X3
.
uses additional options specified by one or more Mdl
= fitcensemble(___,Name,Value
)Name,Value
pair arguments and any of the input arguments in the previous syntaxes. For
example, you can specify the number of learning cycles, the ensemble aggregation
method, or to implement 10fold crossvalidation.
Examples
Train Classification Ensemble
Create a predictive classification ensemble using all available predictor variables in the data. Then, train another ensemble using fewer predictors. Compare the insample predictive accuracies of the ensembles.
Load the census1994
data set.
load census1994
Train an ensemble of classification models using the entire data set and default options.
Mdl1 = fitcensemble(adultdata,'salary')
Mdl1 = ClassificationEnsemble PredictorNames: {1x14 cell} ResponseName: 'salary' CategoricalPredictors: [2 4 6 7 8 9 10 14] ClassNames: [<=50K >50K] ScoreTransform: 'none' NumObservations: 32561 NumTrained: 100 Method: 'LogitBoost' LearnerNames: {'Tree'} ReasonForTermination: 'Terminated normally after completing the requested number of training cycles.' FitInfo: [100x1 double] FitInfoDescription: {2x1 cell} Properties, Methods
Mdl
is a ClassificationEnsemble
model. Some notable characteristics of Mdl
are:
Because two classes are represented in the data, LogitBoost is the ensemble aggregation algorithm.
Because the ensemble aggregation method is a boosting algorithm, classification trees that allow a maximum of 10 splits compose the ensemble.
One hundred trees compose the ensemble.
Use the classification ensemble to predict the labels of a random set of five observations from the data. Compare the predicted labels with their true values.
rng(1) % For reproducibility [pX,pIdx] = datasample(adultdata,5); label = predict(Mdl1,pX); table(label,adultdata.salary(pIdx),'VariableNames',{'Predicted','Truth'})
ans=5×2 table
Predicted Truth
_________ _____
<=50K <=50K
<=50K <=50K
<=50K <=50K
<=50K <=50K
<=50K <=50K
Train a new ensemble using age
and education
only.
Mdl2 = fitcensemble(adultdata,'salary ~ age + education');
Compare the resubstitution losses between Mdl1
and Mdl2
.
rsLoss1 = resubLoss(Mdl1)
rsLoss1 = 0.1058
rsLoss2 = resubLoss(Mdl2)
rsLoss2 = 0.2037
The insample misclassification rate for the ensemble that uses all predictors is lower.
Speed Up Training by Binning Numeric Predictor Values
Train an ensemble of boosted classification trees by using fitcensemble
. Reduce training time by specifying the 'NumBins'
namevalue pair argument to bin numeric predictors. This argument is valid only when fitcensemble
uses a tree learner. After training, you can reproduce binned predictor data by using the BinEdges
property of the trained model and the discretize
function.
Generate a sample data set.
rng('default') % For reproducibility N = 1e6; X = [mvnrnd([1 1],eye(2),N); mvnrnd([1 1],eye(2),N)]; y = [zeros(N,1); ones(N,1)];
Visualize the data set.
figure scatter(X(1:N,1),X(1:N,2),'Marker','.','MarkerEdgeAlpha',0.01) hold on scatter(X(N+1:2*N,1),X(N+1:2*N,2),'Marker','.','MarkerEdgeAlpha',0.01)
Train an ensemble of boosted classification trees using adaptive logistic regression (LogitBoost
, the default for binary classification). Time the function for comparison purposes.
tic Mdl1 = fitcensemble(X,y); toc
Elapsed time is 478.988422 seconds.
Speed up training by using the 'NumBins'
namevalue pair argument. If you specify the 'NumBins'
value as a positive integer scalar, then the software bins every numeric predictor into a specified number of equiprobable bins, and then grows trees on the bin indices instead of the original data. The software does not bin categorical predictors.
tic
Mdl2 = fitcensemble(X,y,'NumBins',50);
toc
Elapsed time is 165.598434 seconds.
The process is about three times faster when you use binned data instead of the original data. Note that the elapsed time can vary depending on your operating system.
Compare the classification errors by resubstitution.
rsLoss1 = resubLoss(Mdl1)
rsLoss1 = 0.0788
rsLoss2 = resubLoss(Mdl2)
rsLoss2 = 0.0788
In this example, binning predictor values reduces training time without loss of accuracy. In general, when you have a large data set like the one in this example, using the binning option speeds up training but causes a potential decrease in accuracy. If you want to reduce training time further, specify a smaller number of bins.
Reproduce binned predictor data by using the BinEdges
property of the trained model and the discretize
function.
X = Mdl2.X; % Predictor data Xbinned = zeros(size(X)); edges = Mdl2.BinEdges; % Find indices of binned predictors. idxNumeric = find(~cellfun(@isempty,edges)); if iscolumn(idxNumeric) idxNumeric = idxNumeric'; end for j = idxNumeric x = X(:,j); % Convert x to array if x is a table. if istable(x) x = table2array(x); end % Group x into bins by using the discretize function. xbinned = discretize(x,[inf; edges{j}; inf]); Xbinned(:,j) = xbinned; end
Xbinned
contains the bin indices, ranging from 1 to the number of bins, for numeric predictors. Xbinned
values are 0
for categorical predictors. If X
contains NaN
s, then the corresponding Xbinned
values are NaN
s.
Estimate Generalization Error of Boosting Ensemble
Estimate the generalization error of ensemble of boosted classification trees.
Load the ionosphere
data set.
load ionosphere
Crossvalidate an ensemble of classification trees using AdaBoostM1 and 10fold crossvalidation. Specify that each tree should be split a maximum of five times using a decision tree template.
rng(5); % For reproducibility t = templateTree('MaxNumSplits',5); Mdl = fitcensemble(X,Y,'Method','AdaBoostM1','Learners',t,'CrossVal','on');
Mdl
is a ClassificationPartitionedEnsemble
model.
Plot the cumulative, 10fold crossvalidated, misclassification rate. Display the estimated generalization error of the ensemble.
kflc = kfoldLoss(Mdl,'Mode','cumulative'); figure; plot(kflc); ylabel('10fold Misclassification rate'); xlabel('Learning cycle');
estGenError = kflc(end)
estGenError = 0.0769
kfoldLoss
returns the generalization error by default. However, plotting the cumulative loss allows you to monitor how the loss changes as weak learners accumulate in the ensemble.
The ensemble achieves a misclassification rate of around 0.06 after accumulating about 50 weak learners. Then, the misclassification rate increase slightly as more weak learners enter the ensemble.
If you are satisfied with the generalization error of the ensemble, then, to create a predictive model, train the ensemble again using all of the settings except crossvalidation. However, it is good practice to tune hyperparameters, such as the maximum number of decision splits per tree and the number of learning cycles.
Optimize Classification Ensemble
Optimize hyperparameters automatically using fitcensemble
.
Load the ionosphere
data set.
load ionosphere
You can find hyperparameters that minimize fivefold crossvalidation loss by using automatic hyperparameter optimization.
Mdl = fitcensemble(X,Y,'OptimizeHyperparameters','auto')
In this example, for reproducibility, set the random seed and use the 'expectedimprovementplus'
acquisition function. Also, for reproducibility of random forest algorithm, specify the 'Reproducible'
namevalue pair argument as true
for tree learners.
rng('default') t = templateTree('Reproducible',true); Mdl = fitcensemble(X,Y,'OptimizeHyperparameters','auto','Learners',t, ... 'HyperparameterOptimizationOptions',struct('AcquisitionFunctionName','expectedimprovementplus'))
===================================================================================================================================  Iter  Eval  Objective  Objective  BestSoFar  BestSoFar  Method  NumLearningC LearnRate  MinLeafSize    result   runtime  (observed)  (estim.)   ycles    ===================================================================================================================================  1  Best  0.10256  2.8201  0.10256  0.10256  RUSBoost  11  0.010199  17   2  Best  0.082621  6.3089  0.082621  0.083414  LogitBoost  206  0.96537  33   3  Accept  0.099715  4.0004  0.082621  0.082624  AdaBoostM1  130  0.0072814  2   4  Best  0.068376  1.5887  0.068376  0.068395  Bag  25    5   5  Best  0.059829  1.7618  0.059829  0.062829  LogitBoost  58  0.19016  5   6  Accept  0.068376  1.6662  0.059829  0.065561  LogitBoost  58  0.10005  5   7  Accept  0.088319  13.07  0.059829  0.065786  LogitBoost  494  0.014474  3   8  Accept  0.065527  0.79673  0.059829  0.065894  LogitBoost  26  0.75515  8   9  Accept  0.15385  0.93354  0.059829  0.061156  LogitBoost  32  0.0010037  59   10  Accept  0.059829  3.8828  0.059829  0.059731  LogitBoost  143  0.44428  1   11  Accept  0.35897  2.3272  0.059829  0.059826  Bag  54    175   12  Accept  0.068376  0.53634  0.059829  0.059825  Bag  10    1   13  Accept  0.12251  9.5155  0.059829  0.059826  AdaBoostM1  442  0.57897  102   14  Accept  0.11966  4.9323  0.059829  0.059827  RUSBoost  95  0.80822  1   15  Accept  0.062678  4.2429  0.059829  0.059826  GentleBoost  156  0.99502  1   16  Accept  0.065527  3.0688  0.059829  0.059824  GentleBoost  115  0.99693  13   17  Best  0.05698  1.659  0.05698  0.056997  GentleBoost  60  0.0010045  3   18  Accept  0.13675  2.0647  0.05698  0.057002  GentleBoost  86  0.0010263  108   19  Accept  0.062678  2.4037  0.05698  0.05703  GentleBoost  88  0.6344  4   20  Accept  0.065527  1.029  0.05698  0.057228  GentleBoost  35  0.0010155  1  ===================================================================================================================================  Iter  Eval  Objective  Objective  BestSoFar  BestSoFar  Method  NumLearningC LearnRate  MinLeafSize    result   runtime  (observed)  (estim.)   ycles    ===================================================================================================================================  21  Accept  0.079772  0.44308  0.05698  0.057214  LogitBoost  11  0.9796  2   22  Accept  0.065527  21.191  0.05698  0.057523  Bag  499    1   23  Accept  0.068376  20.294  0.05698  0.057671  Bag  494    2   24  Accept  0.64103  1.2793  0.05698  0.057468  RUSBoost  30  0.088421  174   25  Accept  0.088319  0.53606  0.05698  0.057456  RUSBoost  10  0.010292  5   26  Accept  0.074074  0.36802  0.05698  0.05753  AdaBoostM1  11  0.14192  13   27  Accept  0.099715  12.133  0.05698  0.057646  AdaBoostM1  498  0.0010096  6   28  Accept  0.079772  10.877  0.05698  0.057886  AdaBoostM1  474  0.030547  31   29  Accept  0.068376  12.326  0.05698  0.061326  GentleBoost  493  0.36142  2   30  Accept  0.065527  0.3945  0.05698  0.061165  LogitBoost  11  0.71408  16 
__________________________________________________________ Optimization completed. MaxObjectiveEvaluations of 30 reached. Total function evaluations: 30 Total elapsed time: 165.9329 seconds Total objective function evaluation time: 148.4504 Best observed feasible point: Method NumLearningCycles LearnRate MinLeafSize ___________ _________________ _________ ___________ GentleBoost 60 0.0010045 3 Observed objective function value = 0.05698 Estimated objective function value = 0.061165 Function evaluation time = 1.659 Best estimated feasible point (according to models): Method NumLearningCycles LearnRate MinLeafSize ___________ _________________ _________ ___________ GentleBoost 60 0.0010045 3 Estimated objective function value = 0.061165 Estimated function evaluation time = 1.6503
Mdl = ClassificationEnsemble ResponseName: 'Y' CategoricalPredictors: [] ClassNames: {'b' 'g'} ScoreTransform: 'none' NumObservations: 351 HyperparameterOptimizationResults: [1×1 BayesianOptimization] NumTrained: 60 Method: 'GentleBoost' LearnerNames: {'Tree'} ReasonForTermination: 'Terminated normally after completing the requested number of training cycles.' FitInfo: [60×1 double] FitInfoDescription: {2×1 cell} Properties, Methods
The optimization searched over the ensemble aggregation methods for binary classification, over NumLearningCycles
, over the LearnRate
for applicable methods, and over the tree learner MinLeafSize
. The output is the ensemble classifier with the minimum estimated crossvalidation loss.
Optimize Classification Ensemble Using CrossValidation
One way to create an ensemble of boosted classification trees that has satisfactory predictive performance is by tuning the decision tree complexity level using crossvalidation. While searching for an optimal complexity level, tune the learning rate to minimize the number of learning cycles.
This example manually finds optimal parameters by using the crossvalidation option (the 'KFold'
namevalue pair argument) and the kfoldLoss
function. Alternatively, you can use the 'OptimizeHyperparameters'
namevalue pair argument to optimize hyperparameters automatically. See Optimize Classification Ensemble.
Load the ionosphere
data set.
load ionosphere
To search for the optimal treecomplexity level:
Crossvalidate a set of ensembles. Exponentially increase the treecomplexity level for subsequent ensembles from decision stump (one split) to at most n  1 splits. n is the sample size. Also, vary the learning rate for each ensemble between 0.1 to 1.
Estimate the crossvalidated misclassification rate of each ensemble.
For treecomplexity level $$j$$, $$j=1...J$$, compare the cumulative, crossvalidated misclassification rate of the ensembles by plotting them against number of learning cycles. Plot separate curves for each learning rate on the same figure.
Choose the curve that achieves the minimal misclassification rate, and note the corresponding learning cycle and learning rate.
Crossvalidate a deep classification tree and a stump. These classification trees serve as benchmarks.
rng(1) % For reproducibility MdlDeep = fitctree(X,Y,'CrossVal','on','MergeLeaves','off', ... 'MinParentSize',1); MdlStump = fitctree(X,Y,'MaxNumSplits',1,'CrossVal','on');
Crossvalidate an ensemble of 150 boosted classification trees using 5fold crossvalidation. Using a tree template, vary the maximum number of splits using the values in the sequence $$\{{3}^{0},{3}^{1},...,{3}^{m}\}$$. m is such that $${3}^{m}$$ is no greater than n  1. For each variant, adjust the learning rate using each value in the set {0.1, 0.25, 0.5, 1};
n = size(X,1); m = floor(log(n  1)/log(3)); learnRate = [0.1 0.25 0.5 1]; numLR = numel(learnRate); maxNumSplits = 3.^(0:m); numMNS = numel(maxNumSplits); numTrees = 150; Mdl = cell(numMNS,numLR); for k = 1:numLR for j = 1:numMNS t = templateTree('MaxNumSplits',maxNumSplits(j)); Mdl{j,k} = fitcensemble(X,Y,'NumLearningCycles',numTrees,... 'Learners',t,'KFold',5,'LearnRate',learnRate(k)); end end
Estimate the cumulative, crossvalidated misclassification rate for each ensemble and the classification trees serving as benchmarks.
kflAll = @(x)kfoldLoss(x,'Mode','cumulative'); errorCell = cellfun(kflAll,Mdl,'Uniform',false); error = reshape(cell2mat(errorCell),[numTrees numel(maxNumSplits) numel(learnRate)]); errorDeep = kfoldLoss(MdlDeep); errorStump = kfoldLoss(MdlStump);
Plot how the crossvalidated misclassification rate behaves as the number of trees in the ensemble increases. Plot the curves with respect to learning rate on the same plot, and plot separate plots for varying treecomplexity levels. Choose a subset of tree complexity levels to plot.
mnsPlot = [1 round(numel(maxNumSplits)/2) numel(maxNumSplits)]; figure for k = 1:3 subplot(2,2,k) plot(squeeze(error(:,mnsPlot(k),:)),'LineWidth',2) axis tight hold on h = gca; plot(h.XLim,[errorDeep errorDeep],'.b','LineWidth',2) plot(h.XLim,[errorStump errorStump],'.r','LineWidth',2) plot(h.XLim,min(min(error(:,mnsPlot(k),:))).*[1 1],'k') h.YLim = [0 0.2]; xlabel('Number of trees') ylabel('Crossvalidated misclass. rate') title(sprintf('MaxNumSplits = %0.3g', maxNumSplits(mnsPlot(k)))) hold off end hL = legend([cellstr(num2str(learnRate','Learning Rate = %0.2f')); ... 'Deep Tree';'Stump';'Min. misclass. rate']); hL.Position(1) = 0.6;
Each curve contains a minimum crossvalidated misclassification rate occurring at the optimal number of trees in the ensemble.
Identify the maximum number of splits, number of trees, and learning rate that yields the lowest misclassification rate overall.
[minErr,minErrIdxLin] = min(error(:));
[idxNumTrees,idxMNS,idxLR] = ind2sub(size(error),minErrIdxLin);
fprintf('\nMin. misclass. rate = %0.5f',minErr)
Min. misclass. rate = 0.05128
fprintf('\nOptimal Parameter Values:\nNum. Trees = %d',idxNumTrees);
Optimal Parameter Values: Num. Trees = 130
fprintf('\nMaxNumSplits = %d\nLearning Rate = %0.2f\n',... maxNumSplits(idxMNS),learnRate(idxLR))
MaxNumSplits = 9 Learning Rate = 1.00
Create a predictive ensemble based on the optimal hyperparameters and the entire training set.
tFinal = templateTree('MaxNumSplits',maxNumSplits(idxMNS)); MdlFinal = fitcensemble(X,Y,'NumLearningCycles',idxNumTrees,... 'Learners',tFinal,'LearnRate',learnRate(idxLR))
MdlFinal = ClassificationEnsemble ResponseName: 'Y' CategoricalPredictors: [] ClassNames: {'b' 'g'} ScoreTransform: 'none' NumObservations: 351 NumTrained: 130 Method: 'LogitBoost' LearnerNames: {'Tree'} ReasonForTermination: 'Terminated normally after completing the requested number of training cycles.' FitInfo: [130×1 double] FitInfoDescription: {2×1 cell} Properties, Methods
MdlFinal
is a ClassificationEnsemble
. To predict whether a radar return is good given predictor data, you can pass the predictor data and MdlFinal
to predict
.
Instead of searching optimal values manually by using the crossvalidation option ('KFold'
) and the kfoldLoss
function, you can use the 'OptimizeHyperparameters'
namevalue pair argument. When you specify 'OptimizeHyperparameters'
, the software finds optimal parameters automatically using Bayesian optimization. The optimal values obtained by using 'OptimizeHyperparameters'
can be different from those obtained using manual search.
mdl = fitcensemble(X,Y,'OptimizeHyperparameters',{'NumLearningCycles','LearnRate','MaxNumSplits'})
====================================================================================================================  Iter  Eval  Objective  Objective  BestSoFar  BestSoFar  NumLearningC LearnRate  MaxNumSplits    result   runtime  (observed)  (estim.)  ycles    ====================================================================================================================  1  Best  0.094017  3.7194  0.094017  0.094017  137  0.001364  3   2  Accept  0.12251  0.66511  0.094017  0.095735  15  0.013089  144 
 3  Best  0.065527  0.90035  0.065527  0.067815  31  0.47201  2   4  Accept  0.19943  8.6107  0.065527  0.070015  340  0.92167  7   5  Accept  0.071225  0.90081  0.065527  0.065583  32  0.14422  2   6  Accept  0.099715  0.688  0.065527  0.065573  23  0.0010566  2   7  Accept  0.11681  0.90799  0.065527  0.065565  28  0.0010156  259   8  Accept  0.17379  0.82143  0.065527  0.065559  29  0.0013435  1   9  Best  0.059829  0.59677  0.059829  0.059844  18  0.87865  3   10  Accept  0.11111  0.40132  0.059829  0.059843  10  0.0012112  48   11  Accept  0.08547  0.41121  0.059829  0.059842  10  0.62108  25   12  Accept  0.11681  0.41538  0.059829  0.059841  10  0.0012154  20   13  Accept  0.082621  0.46504  0.059829  0.059842  10  0.55351  35   14  Accept  0.079772  0.46297  0.059829  0.05984  11  0.74109  74   15  Accept  0.088319  0.69297  0.059829  0.05984  19  0.91106  347   16  Accept  0.062678  0.3637  0.059829  0.059886  10  0.97239  3   17  Accept  0.065527  1.9404  0.059829  0.059887  78  0.97069  3   18  Accept  0.065527  0.39816  0.059829  0.062228  11  0.75051  2   19  Best  0.054131  0.36381  0.054131  0.059083  10  0.69072  3   20  Accept  0.065527  0.38429  0.054131  0.060938  10  0.64403  3  ====================================================================================================================  Iter  Eval  Objective  Objective  BestSoFar  BestSoFar  NumLearningC LearnRate  MaxNumSplits    result   runtime  (observed)  (estim.)  ycles    ====================================================================================================================  21  Accept  0.079772  0.40405  0.054131  0.060161  10  0.80548  13   22  Accept  0.05698  0.37983  0.054131  0.059658  10  0.56949  5   23  Accept  0.10826  0.36128  0.054131  0.059244  10  0.0055133  5   24  Accept  0.074074  0.38056  0.054131  0.05933  10  0.92056  6   25  Accept  0.11966  0.35336  0.054131  0.059132  10  0.27254  1   26  Accept  0.065527  0.77041  0.054131  0.059859  26  0.97412  3   27  Accept  0.068376  0.38116  0.054131  0.060205  10  0.82146  4   28  Accept  0.062678  0.47015  0.054131  0.060713  14  0.99445  3   29  Accept  0.11966  0.41033  0.054131  0.060826  10  0.0012621  344   30  Accept  0.08547  0.45352  0.054131  0.060771  10  0.93676  187 
__________________________________________________________ Optimization completed. MaxObjectiveEvaluations of 30 reached. Total function evaluations: 30 Total elapsed time: 41.5854 seconds Total objective function evaluation time: 28.4744 Best observed feasible point: NumLearningCycles LearnRate MaxNumSplits _________________ _________ ____________ 10 0.69072 3 Observed objective function value = 0.054131 Estimated objective function value = 0.061741 Function evaluation time = 0.36381 Best estimated feasible point (according to models): NumLearningCycles LearnRate MaxNumSplits _________________ _________ ____________ 14 0.99445 3 Estimated objective function value = 0.060771 Estimated function evaluation time = 0.48009
mdl = ClassificationEnsemble ResponseName: 'Y' CategoricalPredictors: [] ClassNames: {'b' 'g'} ScoreTransform: 'none' NumObservations: 351 HyperparameterOptimizationResults: [1×1 BayesianOptimization] NumTrained: 14 Method: 'LogitBoost' LearnerNames: {'Tree'} ReasonForTermination: 'Terminated normally after completing the requested number of training cycles.' FitInfo: [14×1 double] FitInfoDescription: {2×1 cell} Properties, Methods
Input Arguments
Tbl
— Sample data
table
Sample data used to train the model, specified as a table. Each
row of Tbl
corresponds to one observation, and
each column corresponds to one predictor variable. Tbl
can
contain one additional column for the response variable. Multicolumn
variables and cell arrays other than cell arrays of character vectors
are not allowed.
If
Tbl
contains the response variable and you want to use all remaining variables as predictors, then specify the response variable usingResponseVarName
.If
Tbl
contains the response variable, and you want to use a subset of the remaining variables only as predictors, then specify a formula usingformula
.If
Tbl
does not contain the response variable, then specify the response data usingY
. The length of response variable and the number of rows ofTbl
must be equal.
Note
To save memory and execution time, supply X
and Y
instead
of Tbl
.
Data Types: table
ResponseVarName
— Response variable name
name of response variable in Tbl
Response variable name, specified as the name of the response variable in
Tbl
.
You must specify ResponseVarName
as a character
vector or string scalar. For example, if Tbl.Y
is the
response variable, then specify ResponseVarName
as
'Y'
. Otherwise, fitcensemble
treats all columns of Tbl
as predictor
variables.
The response variable must be a categorical, character, or string array, logical or numeric vector, or cell array of character vectors. If the response variable is a character array, then each element must correspond to one row of the array.
For classification, you can specify the order of the classes using the
ClassNames
namevalue pair argument. Otherwise,
fitcensemble
determines the class order, and stores
it in the Mdl.ClassNames
.
Data Types: char
 string
formula
— Explanatory model of response variable and subset of predictor variables
character vector  string scalar
Explanatory model of the response variable and a subset of the predictor variables,
specified as a character vector or string scalar in the form
"Y~x1+x2+x3"
. In this form, Y
represents the
response variable, and x1
, x2
, and
x3
represent the predictor variables.
To specify a subset of variables in Tbl
as predictors for
training the model, use a formula. If you specify a formula, then the software does not
use any variables in Tbl
that do not appear in
formula
.
The variable names in the formula must be both variable names in Tbl
(Tbl.Properties.VariableNames
) and valid MATLAB^{®} identifiers. You can verify the variable names in Tbl
by
using the isvarname
function. If the variable names
are not valid, then you can convert them by using the matlab.lang.makeValidName
function.
Data Types: char
 string
X
— Predictor data
numeric matrix
Predictor data, specified as numeric matrix.
Each row corresponds to one observation, and each column corresponds to one predictor variable.
The length of Y
and the number of rows
of X
must be equal.
To specify the names of the predictors in the order of their
appearance in X
, use the PredictorNames
namevalue
pair argument.
Data Types: single
 double
Y
— Response data
categorical array  character array  string array  logical vector  numeric vector  cell array of character vectors
Response data, specified as a categorical, character, or string array,
logical or numeric vector, or cell array of character vectors. Each entry in
Y
is the response to or label for the observation in
the corresponding row of X
or Tbl
.
The length of Y
and the number of rows of
X
or Tbl
must be equal. If the
response variable is a character array, then each element must correspond to
one row of the array.
You can specify the order of the classes using the
ClassNames
namevalue pair argument. Otherwise,
fitcensemble
determines the class order, and stores
it in the Mdl.ClassNames
.
Data Types: categorical
 char
 string
 logical
 single
 double
 cell
NameValue Arguments
Specify optional pairs of arguments as
Name1=Value1,...,NameN=ValueN
, where Name
is
the argument name and Value
is the corresponding value.
Namevalue arguments must appear after other arguments, but the order of the
pairs does not matter.
Before R2021a, use commas to separate each name and value, and enclose
Name
in quotes.
Example: 'CrossVal','on','LearnRate',0.05
specifies to implement
10fold crossvalidation and to use 0.05
as the learning
rate.
Note
You cannot use any crossvalidation namevalue argument together with the
'OptimizeHyperparameters'
namevalue argument. You can modify the
crossvalidation for 'OptimizeHyperparameters'
only by using the
'HyperparameterOptimizationOptions'
namevalue argument.
Method
— Ensemble aggregation method
'Bag'
 'Subspace'
 'AdaBoostM1'
 'AdaBoostM2'
 'GentleBoost'
 'LogitBoost'
 'LPBoost'
 'RobustBoost'
 'RUSBoost'
 'TotalBoost'
Ensemble aggregation method, specified as the commaseparated pair
consisting of 'Method'
and one of the following
values.
Value  Method  Classification Problem Support  Related NameValue Pair Arguments 

'Bag'  Bootstrap aggregation (bagging, for example,
random forest[2]) — If
'Method' is
'Bag' , then
fitcensemble uses bagging
with random predictor selections at each split
(random forest) by default. To use bagging without
the random selections, use tree learners whose
'NumVariablesToSample' value is
'all' or use discriminant
analysis learners.  Binary and multiclass  N/A 
'Subspace'  Random subspace  Binary and multiclass  NPredToSample 
'AdaBoostM1'  Adaptive boosting  Binary only  LearnRate 
'AdaBoostM2'  Adaptive boosting  Multiclass only  LearnRate 
'GentleBoost'  Gentle adaptive boosting  Binary only  LearnRate 
'LogitBoost'  Adaptive logistic regression  Binary only  LearnRate 
'LPBoost'  Linear programming boosting — Requires Optimization Toolbox™  Binary and multiclass  MarginPrecision 
'RobustBoost'  Robust boosting — Requires Optimization Toolbox  Binary only  RobustErrorGoal ,
RobustMarginSigma ,
RobustMaxMargin 
'RUSBoost'  Random undersampling boosting  Binary and multiclass  LearnRate ,
RatioToSmallest 
'TotalBoost'  Totally corrective boosting — Requires Optimization Toolbox  Binary and multiclass  MarginPrecision 
You can specify sampling options
(FResample
, Replace
,
Resample
) for training data when you use
bagging ('Bag'
) or boosting
('TotalBoost'
, 'RUSBoost'
,
'AdaBoostM1'
, 'AdaBoostM2'
,
'GentleBoost'
, 'LogitBoost'
,
'RobustBoost'
, or
'LPBoost'
).
The defaults are:
'LogitBoost'
for binary problems and'AdaBoostM2'
for multiclass problems if'Learners'
includes only tree learners'AdaBoostM1'
for binary problems and'AdaBoostM2'
for multiclass problems if'Learners'
includes both tree and discriminant analysis learners'Subspace'
if'Learners'
does not include tree learners
For details about ensemble aggregation algorithms and examples, see Algorithms, Tips, Ensemble Algorithms, and Choose an Applicable Ensemble Aggregation Method.
Example: 'Method','Bag'
NumLearningCycles
— Number of ensemble learning cycles
100
(default)  positive integer  'AllPredictorCombinations'
Number of ensemble learning cycles, specified as the commaseparated
pair consisting of 'NumLearningCycles'
and a positive
integer or 'AllPredictorCombinations'
.
If you specify a positive integer, then, at every learning cycle, the software trains one weak learner for every template object in
Learners
. Consequently, the software trainsNumLearningCycles*numel(Learners)
learners.If you specify
'AllPredictorCombinations'
, then setMethod
to'Subspace'
and specify one learner only forLearners
. With these settings, the software trains learners for all possible combinations of predictors takenNPredToSample
at a time. Consequently, the software trainsnchoosek
(size(X,2),NPredToSample)
learners.
The software composes the ensemble using all trained learners and
stores them in Mdl.Trained
.
For more details, see Tips.
Example: 'NumLearningCycles',500
Data Types: single
 double
 char
 string
Learners
— Weak learners to use in ensemble
'discriminant'
 'knn'
 'tree'
 weaklearner template object  cell vector of weaklearner template objects
Weak learners to use in the ensemble, specified as the commaseparated
pair consisting of 'Learners'
and a weaklearner
name, weaklearner template object, or cell vector of weaklearner
template objects.
Weak Learner  WeakLearner Name  Template Object Creation Function  Method
Setting 

Discriminant analysis  'discriminant'  templateDiscriminant  Recommended for 'Subspace' 
knearest neighbors  'knn'  templateKNN  For 'Subspace' only 
Decision tree  'tree'  templateTree  All methods except
'Subspace' 
Weaklearner name (
'discriminant'
,'knn'
, or'tree'
) —fitcensemble
uses weak learners created by a template object creation function with default settings. For example, specifying'Learners','discriminant'
is the same as specifying'Learners',templateDiscriminant()
. See the template object creation function pages for the default settings of a weak learner.Weaklearner template object —
fitcensemble
uses the weak learners created by a template object creation function. Use the namevalue pair arguments of the template object creation function to specify the settings of the weak learners.Cell vector of m weaklearner template objects —
fitcensemble
grows m learners per learning cycle (seeNumLearningCycles
). For example, for an ensemble composed of two types of classification trees, supply{t1 t2}
, wheret1
andt2
are classification tree template objects returned bytemplateTree
.
The default 'Learners'
value is
'knn'
if 'Method'
is
'Subspace'
.
The default 'Learners'
value is
'tree'
if 'Method'
is
'Bag'
or any boosting method. The default values
of templateTree()
depend on the value of
'Method'
.
For bagged decision trees, the maximum number of decision splits (
'MaxNumSplits'
) isn–1
, wheren
is the number of observations. The number of predictors to select at random for each split ('NumVariablesToSample'
) is the square root of the number of predictors. Therefore,fitcensemble
grows deep decision trees. You can grow shallower trees to reduce model complexity or computation time.For boosted decision trees,
'MaxNumSplits'
is 10 and'NumVariablesToSample'
is'all'
. Therefore,fitcensemble
grows shallow decision trees. You can grow deeper trees for better accuracy.
See templateTree
for the
default settings of a weak learner. To obtain reproducible results, you
must specify the 'Reproducible'
namevalue pair argument of
templateTree
as true
if
'NumVariablesToSample'
is not
'all'
.
For details on the number of learners to train, see
NumLearningCycles
and Tips.
Example: 'Learners',templateTree('MaxNumSplits',5)
NPrint
— Printout frequency
'off'
(default)  positive integer
Printout frequency, specified as the commaseparated pair consisting
of 'NPrint'
and a positive integer or 'off'
.
To track the number of weak learners or folds that
fitcensemble
trained so far, specify a positive integer. That
is, if you specify the positive integer m:
Without also specifying any crossvalidation option (for example,
CrossVal
), thenfitcensemble
displays a message to the command line every time it completes training m weak learners.And a crossvalidation option, then
fitcensemble
displays a message to the command line every time it finishes training m folds.
If you specify 'off'
, then fitcensemble
does
not display a message when it completes training weak learners.
Tip
For fastest training of some boosted decision trees, set NPrint
to the
default value 'off'
. This tip holds when the classification
Method
is 'AdaBoostM1'
,
'AdaBoostM2'
, 'GentleBoost'
, or
'LogitBoost'
, or when the regression Method
is
'LSBoost'
.
Example: 'NPrint',5
Data Types: single
 double
 char
 string
NumBins
— Number of bins for numeric predictors
[]
(empty) (default)  positive integer scalar
Number of bins for numeric predictors, specified as the commaseparated pair
consisting of 'NumBins'
and a positive integer scalar. This argument
is valid only when fitcensemble
uses a tree learner, that is,
'Learners'
is either 'tree'
or a template
object created by using templateTree
.
If the
'NumBins'
value is empty (default), thenfitcensemble
does not bin any predictors.If you specify the
'NumBins'
value as a positive integer scalar (numBins
), thenfitcensemble
bins every numeric predictor into at mostnumBins
equiprobable bins, and then grows trees on the bin indices instead of the original data.The number of bins can be less than
numBins
if a predictor has fewer thannumBins
unique values.fitcensemble
does not bin categorical predictors.
When you use a large training data set, this binning option speeds up training but might cause
a potential decrease in accuracy. You can try 'NumBins',50
first, and
then change the value depending on the accuracy and training speed.
A trained model stores the bin edges in the BinEdges
property.
Example: 'NumBins',50
Data Types: single
 double
CategoricalPredictors
— Categorical predictors list
vector of positive integers  logical vector  character matrix  string array  cell array of character vectors  'all'
Categorical predictors list, specified as one of the values in this table.
Value  Description 

Vector of positive integers 
Each entry in the vector is an index value indicating that the corresponding predictor is
categorical. The index values are between 1 and If 
Logical vector 
A 
Character matrix  Each row of the matrix is the name of a predictor variable. The names must match the entries in PredictorNames . Pad the names with extra blanks so each row of the character matrix has the same length. 
String array or cell array of character vectors  Each element in the array is the name of a predictor variable. The names must match the entries in PredictorNames . 
"all"  All predictors are categorical. 
Specification of 'CategoricalPredictors'
is appropriate if:
'Learners'
specifies tree learners.'Learners'
specifies knearest learners where all predictors are categorical.
Each learner identifies and treats categorical predictors in the same way as
the fitting function corresponding to the learner. See 'CategoricalPredictors'
of fitcknn
for knearest learners and 'CategoricalPredictors'
of fitctree
for tree learners.
Example: 'CategoricalPredictors','all'
Data Types: single
 double
 logical
 char
 string
 cell
PredictorNames
— Predictor variable names
string array of unique names  cell array of unique character vectors
Predictor variable names, specified as a string array of unique names or cell array of unique
character vectors. The functionality of PredictorNames
depends on the
way you supply the training data.
If you supply
X
andY
, then you can usePredictorNames
to assign names to the predictor variables inX
.The order of the names in
PredictorNames
must correspond to the column order ofX
. That is,PredictorNames{1}
is the name ofX(:,1)
,PredictorNames{2}
is the name ofX(:,2)
, and so on. Also,size(X,2)
andnumel(PredictorNames)
must be equal.By default,
PredictorNames
is{'x1','x2',...}
.
If you supply
Tbl
, then you can usePredictorNames
to choose which predictor variables to use in training. That is,fitcensemble
uses only the predictor variables inPredictorNames
and the response variable during training.PredictorNames
must be a subset ofTbl.Properties.VariableNames
and cannot include the name of the response variable.By default,
PredictorNames
contains the names of all predictor variables.A good practice is to specify the predictors for training using either
PredictorNames
orformula
, but not both.
Example: "PredictorNames",["SepalLength","SepalWidth","PetalLength","PetalWidth"]
Data Types: string
 cell
ResponseName
— Response variable name
"Y"
(default)  character vector  string scalar
Response variable name, specified as a character vector or string scalar.
If you supply
Y
, then you can useResponseName
to specify a name for the response variable.If you supply
ResponseVarName
orformula
, then you cannot useResponseName
.
Example: "ResponseName","response"
Data Types: char
 string
Options
— Options for computing in parallel and setting random numbers
structure
Options for computing in parallel and setting random numbers, specified as a structure. Create
the Options
structure with statset
.
Note
You need Parallel Computing Toolbox™ to compute in parallel.
This table lists the option fields and their values.
Field Name  Value  Default 

UseParallel  Set this value to  false 
UseSubstreams  Set this value to To compute reproducibly, set
 false 
Streams  Specify this value as a RandStream object or cell array of such objects. Use a single object except when the UseParallel value is true and the UseSubstreams value is false . In that case, use a cell array that has the same size as the parallel pool.  If you do not specify Streams , then fitcensemble uses the default stream or streams. 
For an example using reproducible parallel training, see Train Classification Ensemble in Parallel.
For dualcore systems and above, fitcensemble
parallelizes
training using Intel^{®} Threading Building Blocks (TBB). Therefore, specifying the
UseParallel
option as true
might not provide a
significant speedup on a single computer. For details on Intel TBB, see https://www.intel.com/content/www/us/en/developer/tools/oneapi/onetbb.html.
Example: 'Options',statset('UseParallel',true)
Data Types: struct
CrossVal
— Crossvalidation flag
'off'
(default)  'on'
Crossvalidation flag, specified as the commaseparated pair
consisting of 'Crossval'
and 'on'
or 'off'
.
If you specify 'on'
, then the software implements
10fold crossvalidation.
To override this crossvalidation setting, use one of these
namevalue pair arguments: CVPartition
, Holdout
, KFold
,
or Leaveout
. To create a crossvalidated model,
you can use one crossvalidation namevalue pair argument at a time
only.
Alternatively, crossvalidate later by passing Mdl
to crossval
or crossval
.
Example: 'Crossval','on'
CVPartition
— Crossvalidation partition
[]
(default)  cvpartition
partition object
Crossvalidation partition, specified as a cvpartition
partition object
created by cvpartition
. The partition object
specifies the type of crossvalidation and the indexing for the training and validation
sets.
To create a crossvalidated model, you can specify only one of these four namevalue
arguments: CVPartition
, Holdout
,
KFold
, or Leaveout
.
Example: Suppose you create a random partition for 5fold crossvalidation on 500
observations by using cvp = cvpartition(500,'KFold',5)
. Then, you can
specify the crossvalidated model by using
'CVPartition',cvp
.
Holdout
— Fraction of data for holdout validation
scalar value in the range (0,1)
Fraction of the data used for holdout validation, specified as a scalar value in the range
(0,1). If you specify 'Holdout',p
, then the software completes these
steps:
Randomly select and reserve
p*100
% of the data as validation data, and train the model using the rest of the data.Store the compact, trained model in the
Trained
property of the crossvalidated model.
To create a crossvalidated model, you can specify only one of these four namevalue
arguments: CVPartition
, Holdout
,
KFold
, or Leaveout
.
Example: 'Holdout',0.1
Data Types: double
 single
KFold
— Number of folds
10
(default)  positive integer value greater than 1
Number of folds to use in a crossvalidated model, specified as a positive integer value
greater than 1. If you specify 'KFold',k
, then the software completes
these steps:
Randomly partition the data into
k
sets.For each set, reserve the set as validation data, and train the model using the other
k
– 1 sets.Store the
k
compact, trained models in ak
by1 cell vector in theTrained
property of the crossvalidated model.
To create a crossvalidated model, you can specify only one of these four namevalue
arguments: CVPartition
, Holdout
,
KFold
, or Leaveout
.
Example: 'KFold',5
Data Types: single
 double
Leaveout
— Leaveoneout crossvalidation flag
'off'
(default)  'on'
Leaveoneout crossvalidation flag, specified as 'on'
or
'off'
. If you specify 'Leaveout','on'
, then
for each of the n observations (where n is the
number of observations, excluding missing observations, specified in the
NumObservations
property of the model), the software completes
these steps:
Reserve the one observation as validation data, and train the model using the other n – 1 observations.
Store the n compact, trained models in an nby1 cell vector in the
Trained
property of the crossvalidated model.
To create a crossvalidated model, you can specify only one of these four namevalue
arguments: CVPartition
, Holdout
,
KFold
, or Leaveout
.
Example: 'Leaveout','on'
ClassNames
— Names of classes to use for training
categorical array  character array  string array  logical vector  numeric vector  cell array of character vectors
Names of classes to use for training, specified as a categorical, character, or string
array; a logical or numeric vector; or a cell array of character vectors.
ClassNames
must have the same data type as the response variable
in Tbl
or Y
.
If ClassNames
is a character array, then each element must correspond to one row of the array.
Use ClassNames
to:
Specify the order of the classes during training.
Specify the order of any input or output argument dimension that corresponds to the class order. For example, use
ClassNames
to specify the order of the dimensions ofCost
or the column order of classification scores returned bypredict
.Select a subset of classes for training. For example, suppose that the set of all distinct class names in
Y
is["a","b","c"]
. To train the model using observations from classes"a"
and"c"
only, specify"ClassNames",["a","c"]
.
The default value for ClassNames
is the set of all distinct class names in the response variable in Tbl
or Y
.
Example: "ClassNames",["b","g"]
Data Types: categorical
 char
 string
 logical
 single
 double
 cell
Cost
— Misclassification cost
square matrix  structure array
Misclassification cost, specified as the commaseparated pair
consisting of 'Cost'
and a square matrix or structure.
If you specify:
The square matrix
Cost
, thenCost(i,j)
is the cost of classifying a point into classj
if its true class isi
. That is, the rows correspond to the true class and the columns correspond to the predicted class. To specify the class order for the corresponding rows and columns ofCost
, also specify theClassNames
namevalue pair argument.The structure
S
, then it must have two fields:S.ClassNames
, which contains the class names as a variable of the same data type asY
S.ClassificationCosts
, which contains the cost matrix with rows and columns ordered as inS.ClassNames
The default is ones(
, where K
) 
eye(K
)K
is
the number of distinct classes.
fitcensemble
uses Cost
to adjust the prior
class probabilities specified in Prior
. Then,
fitcensemble
uses the adjusted prior probabilities for
training.
Example: 'Cost',[0 1 2 ; 1 0 2; 2 2 0]
Data Types: double
 single
 struct
Prior
— Prior probabilities
'empirical'
(default)  'uniform'
 numeric vector  structure array
Prior probabilities for each class, specified as the commaseparated
pair consisting of 'Prior'
and a value in this
table.
Value  Description 

'empirical'  The class prior probabilities are the class relative frequencies
in Y . 
'uniform'  All class prior probabilities are equal to 1/K, where K is the number of classes. 
numeric vector  Each element is a class prior probability. Order the elements according to
Mdl.ClassNames or specify the order using the
ClassNames namevalue pair argument. The
software normalizes the elements such that they sum to
1 . 
structure array  A structure

fitcensemble
normalizes
the prior probabilities in Prior
to sum to 1.
Example: struct('ClassNames',{{'setosa','versicolor','virginica'}},'ClassProbs',1:3)
Data Types: char
 string
 double
 single
 struct
ScoreTransform
— Score transformation
"none"
(default)  "doublelogit"
 "invlogit"
 "ismax"
 "logit"
 function handle  ...
Score transformation, specified as a character vector, string scalar, or function handle.
This table summarizes the available character vectors and string scalars.
Value  Description 

"doublelogit"  1/(1 + e^{–2x}) 
"invlogit"  log(x / (1 – x)) 
"ismax"  Sets the score for the class with the largest score to 1, and sets the scores for all other classes to 0 
"logit"  1/(1 + e^{–x}) 
"none" or "identity"  x (no transformation) 
"sign"  –1 for x < 0 0 for x = 0 1 for x > 0 
"symmetric"  2x – 1 
"symmetricismax"  Sets the score for the class with the largest score to 1, and sets the scores for all other classes to –1 
"symmetriclogit"  2/(1 + e^{–x}) – 1 
For a MATLAB function or a function you define, use its function handle for the score transform. The function handle must accept a matrix (the original scores) and return a matrix of the same size (the transformed scores).
Example: "ScoreTransform","logit"
Data Types: char
 string
 function_handle
Weights
— Observation weights
numeric vector of positive values  name of variable in Tbl
Observation weights, specified as the commaseparated pair consisting
of 'Weights'
and a numeric vector of positive values
or name of a variable in Tbl
. The software weighs
the observations in each row of X
or Tbl
with
the corresponding value in Weights
. The size of Weights
must
equal the number of rows of X
or Tbl
.
If you specify the input data as a table Tbl
, then
Weights
can be the name of a variable in Tbl
that contains a numeric vector. In this case, you must specify
Weights
as a character vector or string scalar. For example, if
the weights vector W
is stored as Tbl.W
, then
specify it as 'W'
. Otherwise, the software treats all columns of
Tbl
, including W
, as predictors or the
response when training the model.
The software normalizes Weights
to sum up
to the value of the prior probability in the respective class.
By default, Weights
is ones(
,
where n
,1)n
is the number of observations in X
or Tbl
.
Data Types: double
 single
 char
 string
FResample
— Fraction of training set to resample
1
(default)  positive scalar in (0,1]
Fraction of the training set to resample for every weak learner, specified as a positive
scalar in (0,1]. To use 'FResample'
, set
Resample
to 'on'
.
Example: 'FResample',0.75
Data Types: single
 double
Replace
— Flag indicating to sample with replacement
'on'
(default)  'off'
Flag indicating sampling with replacement, specified as the
commaseparated pair consisting of 'Replace'
and 'off'
or 'on'
.
For
'on'
, the software samples the training observations with replacement.For
'off'
, the software samples the training observations without replacement. If you setResample
to'on'
, then the software samples training observations assuming uniform weights. If you also specify a boosting method, then the software boosts by reweighting observations.
Unless you set Method
to 'bag'
or
set Resample
to 'on'
, Replace
has
no effect.
Example: 'Replace','off'
Resample
— Flag indicating to resample
'off'
 'on'
Flag indicating to resample, specified as the commaseparated
pair consisting of 'Resample'
and 'off'
or 'on'
.
If
Method
is a boosting method, then:'Resample','on'
specifies to sample training observations using updated weights as the multinomial sampling probabilities.'Resample','off'
(default) specifies to reweight observations at every learning iteration.
If
Method
is'bag'
, then'Resample'
must be'on'
. The software resamples a fraction of the training observations (seeFResample
) with or without replacement (seeReplace
).
If you specify to resample using Resample
, then it is good
practice to resample to entire data set. That is, use the default setting of 1 for
FResample
.
LearnRate
— Learning rate for shrinkage
1
(default)  numeric scalar in (0,1]
Learning rate for shrinkage, specified as the commaseparated pair consisting of
'LearnRate'
and a numeric scalar in the interval (0,1].
To train an ensemble using shrinkage, set LearnRate
to a value less than 1
, for example, 0.1
is a popular choice. Training an ensemble using shrinkage requires more learning iterations, but often achieves better accuracy.
Example: 'LearnRate',0.1
Data Types: single
 double
LearnRate
— Learning rate for shrinkage
1
(default)  numeric scalar in (0,1]
Learning rate for shrinkage, specified as the commaseparated pair consisting of
'LearnRate'
and a numeric scalar in the interval (0,1].
To train an ensemble using shrinkage, set LearnRate
to a value less than 1
, for example, 0.1
is a popular choice. Training an ensemble using shrinkage requires more learning iterations, but often achieves better accuracy.
Example: 'LearnRate',0.1
Data Types: single
 double
RatioToSmallest
— Sampling proportion with respect to lowestrepresented class
positive numeric scalar  numeric vector of positive values
Sampling proportion with respect to the lowestrepresented class,
specified as the commaseparated pair consisting of 'RatioToSmallest'
and
a numeric scalar or numeric vector of positive values with length
equal to the number of distinct classes in the training data.
Suppose that there are K
classes
in the training data and the lowestrepresented class has m
observations
in the training data.
If you specify the positive numeric scalar
s
, thenfitcensemble
samples
observations from each class, that is, it uses the same sampling proportion for each class. For more details, see Algorithms.s
*m
If you specify the numeric vector
[
, thens1
,s2
,...,sK
]fitcensemble
samples
observations from classsi
*m
i
,i
= 1,...,K. The elements ofRatioToSmallest
correspond to the order of the class names specified usingClassNames
(see Tips).
The default value is ones(
,
which specifies to sample K
,1)m
observations
from each class.
Example: 'RatioToSmallest',[2,1]
Data Types: single
 double
MarginPrecision
— Margin precision to control convergence speed
0.1
(default)  numeric scalar in [0,1]
Margin precision to control convergence speed, specified as
the commaseparated pair consisting of 'MarginPrecision'
and
a numeric scalar in the interval [0,1]. MarginPrecision
affects
the number of boosting iterations required for convergence.
Tip
To train an ensemble using many learners, specify a small value
for MarginPrecision
. For training using a few learners,
specify a large value.
Example: 'MarginPrecision',0.5
Data Types: single
 double
RobustErrorGoal
— Target classification error
0.1
(default)  nonnegative numeric scalar
Target classification error, specified as the commaseparated
pair consisting of 'RobustErrorGoal'
and a nonnegative
numeric scalar. The upper bound on possible values depends on the
values of RobustMarginSigma
and RobustMaxMargin
.
However, the upper bound cannot exceed 1
.
Tip
For a particular training set, usually there is an optimal range
for RobustErrorGoal
. If you set it too low or too
high, then the software can produce a model with poor classification
accuracy. Try crossvalidating to search for the appropriate value.
Example: 'RobustErrorGoal',0.05
Data Types: single
 double
RobustMarginSigma
— Classification margin distribution spread
0.1
(default)  positive numeric scalar
Classification margin distribution spread over the training
data, specified as the commaseparated pair consisting of 'RobustMarginSigma'
and
a positive numeric scalar. Before specifying RobustMarginSigma
,
consult the literature on RobustBoost
, for example, [19].
Example: 'RobustMarginSigma',0.5
Data Types: single
 double
RobustMaxMargin
— Maximal classification margin
0
(default)  nonnegative numeric scalar
Maximal classification margin in the training data, specified
as the commaseparated pair consisting of 'RobustMaxMargin'
and
a nonnegative numeric scalar. The software minimizes the number of
observations in the training data having classification margins below RobustMaxMargin
.
Example: 'RobustMaxMargin',1
Data Types: single
 double
NPredToSample
— Number of predictors to sample
1
(default)  positive integer
Number of predictors to sample for each random subspace learner,
specified as the commaseparated pair consisting of 'NPredToSample'
and
a positive integer in the interval 1,...,p, where p is
the number of predictor variables (size(X,2)
or size(Tbl,2)
).
Data Types: single
 double
OptimizeHyperparameters
— Parameters to optimize
'none'
(default)  'auto'
 'all'
 string array or cell array of eligible parameter names  vector of optimizableVariable
objects
Parameters to optimize, specified as the commaseparated pair
consisting of 'OptimizeHyperparameters'
and one of
the following:
'none'
— Do not optimize.'auto'
— Use{'Method','NumLearningCycles','LearnRate'}
along with the default parameters for the specifiedLearners
:Learners
='tree'
(default) —{'MinLeafSize'}
Learners
='discriminant'
—{'Delta','Gamma'}
Learners
='knn'
—{'Distance','NumNeighbors'}
Note
For hyperparameter optimization,
Learners
must be a single argument, not a string array or cell array.'all'
— Optimize all eligible parameters.String array or cell array of eligible parameter names
Vector of
optimizableVariable
objects, typically the output ofhyperparameters
The optimization attempts to minimize the crossvalidation loss
(error) for fitcensemble
by varying the parameters.
For information about crossvalidation loss (albeit in a different
context), see Classification Loss. To control the
crossvalidation type and other aspects of the optimization, use the
HyperparameterOptimizationOptions
namevalue
pair.
Note
The values of 'OptimizeHyperparameters'
override any values you specify
using other namevalue arguments. For example, setting
'OptimizeHyperparameters'
to 'auto'
causes
fitcensemble
to optimize hyperparameters corresponding to the
'auto'
option and to ignore any specified values for the
hyperparameters.
The eligible parameters for fitcensemble
are:
Method
— Depends on the number of classes.Two classes — Eligible methods are
'Bag'
,'GentleBoost'
,'LogitBoost'
,'AdaBoostM1'
, and'RUSBoost'
.Three or more classes — Eligible methods are
'Bag'
,'AdaBoostM2'
, and'RUSBoost'
.
NumLearningCycles
—fitcensemble
searches among positive integers, by default logscaled with range[10,500]
.LearnRate
—fitcensemble
searches among positive reals, by default logscaled with range[1e3,1]
.The eligible hyperparameters for the chosen
Learners
:Learners Eligible Hyperparameters
Bold = Used By DefaultDefault Range 'discriminant'
Delta
Logscaled in the range [1e6,1e3]
DiscrimType
'linear'
,'quadratic'
,'diagLinear'
,'diagQuadratic'
,'pseudoLinear'
, and'pseudoQuadratic'
Gamma
Real values in [0,1]
'knn'
Distance
'cityblock'
,'chebychev'
,'correlation'
,'cosine'
,'euclidean'
,'hamming'
,'jaccard'
,'mahalanobis'
,'minkowski'
,'seuclidean'
, and'spearman'
DistanceWeight
'equal'
,'inverse'
, and'squaredinverse'
Exponent
Positive values in [0.5,3]
NumNeighbors
Positive integer values logscaled in the range [1, max(2,round(NumObservations/2))]
Standardize
'true'
and'false'
'tree'
MaxNumSplits
Integers logscaled in the range [1,max(2,NumObservations1)]
MinLeafSize
Integers logscaled in the range [1,max(2,floor(NumObservations/2))]
NumVariablesToSample
Integers in the range [1,max(2,NumPredictors)]
SplitCriterion
'gdi'
,'deviance'
, and'twoing'
Alternatively, use
hyperparameters
with your chosenLearners
. Note that you must specify the predictor data and response when creating anoptimizableVariable
object.load fisheriris params = hyperparameters('fitcensemble',meas,species,'Tree');
To see the eligible and default hyperparameters, examine
params
.
Set nondefault parameters by passing a vector of
optimizableVariable
objects that have nondefault
values. For example,
load fisheriris params = hyperparameters('fitcensemble',meas,species,'Tree'); params(4).Range = [1,30];
Pass params
as the value of
OptimizeHyperparameters
.
By default, the iterative display appears at the command line,
and plots appear according to the number of hyperparameters in the optimization. For the
optimization and plots, the objective function is the misclassification rate. To control the
iterative display, set the Verbose
field of the
'HyperparameterOptimizationOptions'
namevalue argument. To control the
plots, set the ShowPlots
field of the
'HyperparameterOptimizationOptions'
namevalue argument.
For an example, see Optimize Classification Ensemble.
Example: 'OptimizeHyperparameters',{'Method','NumLearningCycles','LearnRate','MinLeafSize','MaxNumSplits'}
HyperparameterOptimizationOptions
— Options for optimization
structure
Options for optimization, specified as a structure. This argument modifies the effect of the
OptimizeHyperparameters
namevalue argument. All fields in the
structure are optional.
Field Name  Values  Default 

Optimizer 
 'bayesopt' 
AcquisitionFunctionName 
Acquisition functions whose names include
 'expectedimprovementpersecondplus' 
MaxObjectiveEvaluations  Maximum number of objective function evaluations.  30 for 'bayesopt' and
'randomsearch' , and the entire grid for
'gridsearch' 
MaxTime  Time limit, specified as a positive real scalar. The time limit is in seconds, as
measured by  Inf 
NumGridDivisions  For 'gridsearch' , the number of values in each dimension. The value can be
a vector of positive integers giving the number of
values for each dimension, or a scalar that
applies to all dimensions. This field is ignored
for categorical variables.  10 
ShowPlots  Logical value indicating whether to show plots. If true , this field plots
the best observed objective function value against the iteration number. If you
use Bayesian optimization (Optimizer is
'bayesopt' ), then this field also plots the best
estimated objective function value. The best observed objective function values
and best estimated objective function values correspond to the values in the
BestSoFar (observed) and BestSoFar
(estim.) columns of the iterative display, respectively. You can
find these values in the properties ObjectiveMinimumTrace and EstimatedObjectiveMinimumTrace of
Mdl.HyperparameterOptimizationResults . If the problem
includes one or two optimization parameters for Bayesian optimization, then
ShowPlots also plots a model of the objective function
against the parameters.  true 
SaveIntermediateResults  Logical value indicating whether to save results when Optimizer is
'bayesopt' . If
true , this field overwrites a
workspace variable named
'BayesoptResults' at each
iteration. The variable is a BayesianOptimization object.  false 
Verbose  Display at the command line:
For details, see the  1 
UseParallel  Logical value indicating whether to run Bayesian optimization in parallel, which requires Parallel Computing Toolbox. Due to the nonreproducibility of parallel timing, parallel Bayesian optimization does not necessarily yield reproducible results. For details, see Parallel Bayesian Optimization.  false 
Repartition  Logical value indicating whether to repartition the crossvalidation at every
iteration. If this field is The setting
 false 
Use no more than one of the following three options.  
CVPartition  A cvpartition object, as created by cvpartition  'Kfold',5 if you do not specify a crossvalidation
field 
Holdout  A scalar in the range (0,1) representing the holdout fraction  
Kfold  An integer greater than 1 
Example: 'HyperparameterOptimizationOptions',struct('MaxObjectiveEvaluations',60)
Data Types: struct
Output Arguments
Mdl
— Trained classification ensemble model
ClassificationBaggedEnsemble
model object  ClassificationEnsemble
model object  ClassificationPartitionedEnsemble
crossvalidated
model object
Trained ensemble model, returned as one of the model objects in this table.
Model Object  Specify Any CrossValidation Options?  Method
Setting  Resample
Setting 

ClassificationBaggedEnsemble  No  'Bag'  'on' 
ClassificationEnsemble  No  Any ensemble aggregation method for classification  'off' 
ClassificationPartitionedEnsemble  Yes  Any ensemble aggregation method for classification  'off' or
'on' 
The namevalue pair arguments that control crossvalidation
are CrossVal
, Holdout
,
KFold
, Leaveout
, and
CVPartition
.
To reference properties of Mdl
, use dot notation. For
example, to access or display the cell vector of weak learner model objects
for an ensemble that has not been crossvalidated, enter
Mdl.Trained
at the command line.
Tips
NumLearningCycles
can vary from a few dozen to a few thousand. Usually, an ensemble with good predictive power requires from a few hundred to a few thousand weak learners. However, you do not have to train an ensemble for that many cycles at once. You can start by growing a few dozen learners, inspect the ensemble performance and then, if necessary, train more weak learners usingresume
for classification problems.Ensemble performance depends on the ensemble setting and the setting of the weak learners. That is, if you specify weak learners with default parameters, then the ensemble can perform poorly. Therefore, like ensemble settings, it is good practice to adjust the parameters of the weak learners using templates, and to choose values that minimize generalization error.
If you specify to resample using
Resample
, then it is good practice to resample to entire data set. That is, use the default setting of1
forFResample
.If the ensemble aggregation method (
Method
) is'bag'
and:The misclassification cost (
Cost
) is highly imbalanced, then, for inbag samples, the software oversamples unique observations from the class that has a large penalty.The class prior probabilities (
Prior
) are highly skewed, the software oversamples unique observations from the class that has a large prior probability.
For smaller sample sizes, these combinations can result in a low relative frequency of outofbag observations from the class that has a large penalty or prior probability. Consequently, the estimated outofbag error is highly variable and it can be difficult to interpret. To avoid large estimated outofbag error variances, particularly for small sample sizes, set a more balanced misclassification cost matrix using
Cost
or a less skewed prior probability vector usingPrior
.Because the order of some input and output arguments correspond to the distinct classes in the training data, it is good practice to specify the class order using the
ClassNames
namevalue pair argument.To determine the class order quickly, remove all observations from the training data that are unclassified (that is, have a missing label), obtain and display an array of all the distinct classes, and then specify the array for
ClassNames
. For example, suppose the response variable (Y
) is a cell array of labels. This code specifies the class order in the variableclassNames
.Ycat = categorical(Y); classNames = categories(Ycat)
categorical
assigns<undefined>
to unclassified observations andcategories
excludes<undefined>
from its output. Therefore, if you use this code for cell arrays of labels or similar code for categorical arrays, then you do not have to remove observations with missing labels to obtain a list of the distinct classes.To specify that the class order from lowestrepresented label to mostrepresented, then quickly determine the class order (as in the previous bullet), but arrange the classes in the list by frequency before passing the list to
ClassNames
. Following from the previous example, this code specifies the class order from lowest to mostrepresented inclassNamesLH
.Ycat = categorical(Y); classNames = categories(Ycat); freq = countcats(Ycat); [~,idx] = sort(freq); classNamesLH = classNames(idx);
After training a model, you can generate C/C++ code that predicts labels for new data. Generating C/C++ code requires MATLAB Coder™. For details, see Introduction to Code Generation.
Algorithms
For details of ensemble aggregation algorithms, see Ensemble Algorithms.
If you set
Method
to be a boosting algorithm andLearners
to be decision trees, then the software grows shallow decision trees by default. You can adjust tree depth by specifying theMaxNumSplits
,MinLeafSize
, andMinParentSize
namevalue pair arguments usingtemplateTree
.If you specify the
Cost
,Prior
, andWeights
namevalue arguments, the output model object stores the specified values in theCost
,Prior
, andW
properties, respectively. TheCost
property stores the userspecified cost matrix (C) without modification. ThePrior
andW
properties store the prior probabilities and observation weights, respectively, after normalization. For model training, the software updates the prior probabilities and observation weights to incorporate the penalties described in the cost matrix. For details, see Misclassification Cost Matrix, Prior Probabilities, and Observation Weights.For bagging (
'Method','Bag'
),fitcensemble
generates inbag samples by oversampling classes with large misclassification costs and undersampling classes with small misclassification costs. Consequently, outofbag samples have fewer observations from classes with large misclassification costs and more observations from classes with small misclassification costs. If you train a classification ensemble using a small data set and a highly skewed cost matrix, then the number of outofbag observations per class can be low. Therefore, the estimated outofbag error can have a large variance and can be difficult to interpret. The same phenomenon can occur for classes with large prior probabilities.For the RUSBoost ensemble aggregation method (
'Method','RUSBoost'
), the namevalue pair argumentRatioToSmallest
specifies the sampling proportion for each class with respect to the lowestrepresented class. For example, suppose that there are two classes in the training data: A and B. A has 100 observations and B has 10 observations. Suppose also that the lowestrepresented class hasm
observations in the training data.If you set
'RatioToSmallest',2
, then
=s
*m
2*10
=20
. Consequently,fitcensemble
trains every learner using 20 observations from class A and 20 observations from class B. If you set'RatioToSmallest',[2 2]
, then you obtain the same result.If you set
'RatioToSmallest',[2,1]
, then
=s1
*m
2*10
=20
and
=s2
*m
1*10
=10
. Consequently,fitcensemble
trains every learner using 20 observations from class A and 10 observations from class B.
For dualcore systems and above,
fitcensemble
parallelizes training using Intel Threading Building Blocks (TBB). For details on Intel TBB, see https://www.intel.com/content/www/us/en/developer/tools/oneapi/onetbb.html.
References
[1] Breiman, L. “Bagging Predictors.” Machine Learning. Vol. 26, pp. 123–140, 1996.
[2] Breiman, L. “Random Forests.” Machine Learning. Vol. 45, pp. 5–32, 2001.
[3] Freund, Y. “A more robust boosting algorithm.” arXiv:0905.2138v1, 2009.
[4] Freund, Y. and R. E. Schapire. “A DecisionTheoretic Generalization of OnLine Learning and an Application to Boosting.” J. of Computer and System Sciences, Vol. 55, pp. 119–139, 1997.
[5] Friedman, J. “Greedy function approximation: A gradient boosting machine.” Annals of Statistics, Vol. 29, No. 5, pp. 1189–1232, 2001.
[6] Friedman, J., T. Hastie, and R. Tibshirani. “Additive logistic regression: A statistical view of boosting.” Annals of Statistics, Vol. 28, No. 2, pp. 337–407, 2000.
[7] Hastie, T., R. Tibshirani, and J. Friedman. The Elements of Statistical Learning section edition, Springer, New York, 2008.
[8] Ho, T. K. “The random subspace method for constructing decision forests.” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 8, pp. 832–844, 1998.
[9] Schapire, R. E., Y. Freund, P. Bartlett, and W.S. Lee. “Boosting the margin: A new explanation for the effectiveness of voting methods.” Annals of Statistics, Vol. 26, No. 5, pp. 1651–1686, 1998.
[10] Seiffert, C., T. Khoshgoftaar, J. Hulse, and A. Napolitano. “RUSBoost: Improving classification performance when training data is skewed.” 19th International Conference on Pattern Recognition, pp. 1–4, 2008.
[11] Warmuth, M., J. Liao, and G. Ratsch. “Totally corrective boosting algorithms that maximize the margin.” Proc. 23rd Int’l. Conf. on Machine Learning, ACM, New York, pp. 1001–1008, 2006.
Extended Capabilities
Automatic Parallel Support
Accelerate code by automatically running computation in parallel using Parallel Computing Toolbox™.
fitcensemble
supports parallel training
using the 'Options'
namevalue argument. Create options using statset
, such as options = statset('UseParallel',true)
.
Parallel ensemble training requires you to set the 'Method'
namevalue
argument to 'Bag'
. Parallel training is available only for tree learners, the
default type for 'Bag'
.
To perform parallel hyperparameter optimization, use the
'HyperparameterOptimizationOptions', struct('UseParallel',true)
namevalue argument in the call to the fitcensemble
function.
For more information on parallel hyperparameter optimization, see Parallel Bayesian Optimization.
For general information about parallel computing, see Run MATLAB Functions with Automatic Parallel Support (Parallel Computing Toolbox).
GPU Arrays
Accelerate code by running on a graphics processing unit (GPU) using Parallel Computing Toolbox™.
Usage notes and limitations:
fitcensemble
supports only decision tree learners. You can specify the namevalue argumentLearners
only as"tree"
, a learner template object or cell vector of learner template objects created bytemplateTree
. If you usetemplateTree
, you can specify the namevalue argumentsSurrogate
andPredictorSelection
only as"off"
and"allsplits"
, respectively.You can specify the namevalue argument
Method
only as"AdaBoostM1"
,"AdaBoostM2"
,"GentleBoost"
,"LogitBoost"
, or"RUSBoost"
.You cannot specify the namevalue argument
NPredToSample
.If you use
templateTree
and the data contains categorical predictors, the following apply:For multiclass classification,
fitcensemble
supports only theOVAbyClass
algorithm for finding the best split.You can specify the namevalue argument
NumVariablesToSample
only as"all"
.
fitcensemble
fits the model on a GPU if either of the following apply:The input argument
X
is agpuArray
object.The input argument
Tbl
containsgpuArray
predictor variables.
If you use
templateTree
to specifyMaxNumSplits
, note thatfitcensemble
might not execute faster on a GPU than a CPU for deeper decision trees.
Version History
Introduced in R2016b
Open Example
You have a modified version of this example. Do you want to open this example with your edits?
MATLAB Command
You clicked a link that corresponds to this MATLAB command:
Run the command by entering it in the MATLAB Command Window. Web browsers do not support MATLAB commands.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
You can also select a web site from the following list:
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
 América Latina (Español)
 Canada (English)
 United States (English)
Europe
 Belgium (English)
 Denmark (English)
 Deutschland (Deutsch)
 España (Español)
 Finland (English)
 France (Français)
 Ireland (English)
 Italia (Italiano)
 Luxembourg (English)
 Netherlands (English)
 Norway (English)
 Österreich (Deutsch)
 Portugal (English)
 Sweden (English)
 Switzerland
 United Kingdom (English)