Documentation

This is machine translation

Translated by
Mouseover text to see original. Click the button below to return to the English verison of the page.

fitrtree

Fit binary regression decision tree

Syntax

``tree = fitrtree(Tbl,ResponseVarName)``
``tree = fitrtree(Tbl,formula)``
``tree = fitrtree(Tbl,Y)``
``tree = fitrtree(X,Y)``
``tree = fitrtree(___,Name,Value)``

Description

````tree = fitrtree(Tbl,ResponseVarName)` returns a regression tree based on the input variables (also known as predictors, features, or attributes) in the table `Tbl` and output (response) contained in `Tbl.ResponseVarName`. `tree` is a binary tree where each branching node is split based on the values of a column of `Tbl`. ```
````tree = fitrtree(Tbl,formula)` returns a regression tree based on the input variables contained in the table `Tbl`. `formula` is an explanatory model of the response and a subset of predictor variables in `Tbl` used to fit `tree`.```
````tree = fitrtree(Tbl,Y)` returns a regression tree based on the input variables contained in the table `Tbl` and output in vector `Y`.```

example

````tree = fitrtree(X,Y)` returns a regression tree based on the input variables `X` and output `Y`. `tree` is a binary tree where each branching node is split based on the values of a column of `X`. ```

example

````tree = fitrtree(___,Name,Value)` fits a tree with additional options specified by one or more `Name,Value` pair arguments. For example, you can specify observation weights or train a cross-validated model.```

Examples

collapse all

```load carsmall; ```

Construct a regression tree using the sample data.

```tree = fitrtree([Weight, Cylinders],MPG,... 'categoricalpredictors',2,'MinParentSize',20,... 'PredictorNames',{'W','C'}) ```
```tree = RegressionTree PredictorNames: {'W' 'C'} ResponseName: 'Y' CategoricalPredictors: 2 ResponseTransform: 'none' NumObservations: 94 ```

Predict the mileage of 4,000-pound cars with 4, 6, and 8 cylinders.

```mileage4K = predict(tree,[4000 4; 4000 6; 4000 8]) ```
```mileage4K = 19.2778 19.2778 14.3889 ```

You can control the depth of trees using the `MaxNumSplits`, `MinLeafSize`, or `MinParentSize` name-value pair parameters. `fitrtree` grows deep decision trees by default. You can grow shallower trees to reduce model complexity or computation time.

Load the `carsmall` data set. Consider `Displacement`, `Horsepower`, and `Weight` as predictors of the response `MPG`.

```load carsmall X = [Displacement Horsepower Weight]; ```

The default values of the tree-depth controllers for growing regression trees are:

• `n - 1` for `MaxNumSplits`. `n` is the training sample size.

• `1` for `MinLeafSize`.

• `10` for `MinParentSize`.

These default values tend to grow deep trees for large training sample sizes.

Train a regression tree using the default values for tree-depth control. Cross validate the model using 10-fold cross validation.

```rng(1); % For reproducibility MdlDefault = fitrtree(X,MPG,'CrossVal','on'); ```

Draw a histogram of the number of imposed splits on the trees. The number of imposed splits is one less than the number of leaves. Also, view one of the trees.

```numBranches = @(x)sum(x.IsBranch); mdlDefaultNumSplits = cellfun(numBranches, MdlDefault.Trained); figure; histogram(mdlDefaultNumSplits) view(MdlDefault.Trained{1},'Mode','graph') ```

The average number of splits is between 14 and 15.

Suppose that you want a regression tree that is not as complex (deep) as the ones trained using the default number of splits. Train another regression tree, but set the maximum number of splits at 7, which is about half the mean number of splits from the default regression tree. Cross validate the model using 10-fold cross validation.

```Mdl7 = fitrtree(X,MPG,'MaxNumSplits',7,'CrossVal','on'); view(Mdl7.Trained{1},'Mode','graph') ```

Compare the cross validation MSEs of the models.

```mseDefault = kfoldLoss(MdlDefault) mse7 = kfoldLoss(Mdl7) ```
```mseDefault = 27.7277 mse7 = 28.3833 ```

`Mdl7` is much less complex and performs only slightly worse than `MdlDefault`.

This example shows how to optimize hyperparameters automatically using `fitrtree`. The example uses the `carsmall` data.

Load the `carsmall` data.

```load carsmall ```

Use `Weight` and `Horsepower` as predictors for `MPG`. Find hyperparameters that minimize five-fold cross-validation loss by using automatic hyperparameter optimization.

For reproducibility, set the random seed and use the `'expected-improvement-plus'` acquisition function.

```X = [Weight,Horsepower]; Y = MPG; rng default Mdl = fitrtree(X,Y,'OptimizeHyperparameters','auto',... 'HyperparameterOptimizationOptions',struct('AcquisitionFunctionName',... 'expected-improvement-plus')) ```
```|======================================================================================| | Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | MinLeafSize | | | result | | runtime | (observed) | (estim.) | | |======================================================================================| | 1 | Best | 3.2818 | 1.3647 | 3.2818 | 3.2818 | 28 | | 2 | Accept | 3.4183 | 0.66091 | 3.2818 | 3.2888 | 1 | | 3 | Best | 3.1491 | 0.23495 | 3.1491 | 3.166 | 4 | | 4 | Best | 2.9885 | 0.3322 | 2.9885 | 2.9885 | 9 | | 5 | Accept | 2.9978 | 0.28918 | 2.9885 | 2.9885 | 7 | | 6 | Accept | 3.0203 | 0.12269 | 2.9885 | 3.0013 | 8 | | 7 | Accept | 2.9885 | 0.12976 | 2.9885 | 2.9981 | 9 | | 8 | Best | 2.9589 | 0.1058 | 2.9589 | 2.985 | 10 | | 9 | Accept | 3.0459 | 0.083948 | 2.9589 | 2.9895 | 12 | | 10 | Accept | 4.1881 | 0.15664 | 2.9589 | 2.9594 | 50 | | 11 | Accept | 3.4182 | 0.13842 | 2.9589 | 2.9594 | 2 | | 12 | Accept | 3.0376 | 0.2142 | 2.9589 | 2.9592 | 6 | | 13 | Accept | 3.1453 | 0.18239 | 2.9589 | 2.9856 | 19 | | 14 | Accept | 2.9589 | 0.19389 | 2.9589 | 2.9591 | 10 | | 15 | Accept | 2.9589 | 0.12531 | 2.9589 | 2.959 | 10 | | 16 | Accept | 2.9589 | 0.099919 | 2.9589 | 2.959 | 10 | | 17 | Accept | 3.3055 | 0.12093 | 2.9589 | 2.959 | 3 | | 18 | Accept | 3.4577 | 0.10651 | 2.9589 | 2.9589 | 37 | | 19 | Accept | 3.1584 | 0.25313 | 2.9589 | 2.9589 | 15 | | 20 | Accept | 3.107 | 0.1615 | 2.9589 | 2.9589 | 5 | |======================================================================================| | Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | MinLeafSize | | | result | | runtime | (observed) | (estim.) | | |======================================================================================| | 21 | Accept | 3.0398 | 0.29153 | 2.9589 | 2.9589 | 23 | | 22 | Accept | 3.3226 | 0.3691 | 2.9589 | 2.9589 | 32 | | 23 | Accept | 3.1883 | 0.12918 | 2.9589 | 2.9589 | 17 | | 24 | Accept | 4.1881 | 0.15126 | 2.9589 | 2.9589 | 43 | | 25 | Accept | 3.0123 | 0.22355 | 2.9589 | 2.9589 | 11 | | 26 | Accept | 3.0932 | 0.19774 | 2.9589 | 2.9589 | 21 | | 27 | Accept | 3.078 | 0.15353 | 2.9589 | 2.9589 | 13 | | 28 | Accept | 3.2818 | 0.19153 | 2.9589 | 2.9589 | 25 | | 29 | Accept | 3.0992 | 0.097133 | 2.9589 | 2.9589 | 14 | | 30 | Accept | 3.4361 | 0.096905 | 2.9589 | 2.9589 | 34 | __________________________________________________________ Optimization completed. MaxObjectiveEvaluations of 30 reached. Total function evaluations: 30 Total elapsed time: 131.221 seconds. Total objective function evaluation time: 6.9785 Best observed feasible point: MinLeafSize ___________ 10 Observed objective function value = 2.9589 Estimated objective function value = 2.9589 Function evaluation time = 0.1058 Best estimated feasible point (according to models): MinLeafSize ___________ 10 Estimated objective function value = 2.9589 Estimated function evaluation time = 0.18343 Mdl = RegressionTree ResponseName: 'Y' CategoricalPredictors: [] ResponseTransform: 'none' NumObservations: 94 HyperparameterOptimizationResults: [1x1 BayesianOptimization] ```

Load the `carsmall` data set. Consider a model that predicts the mean fuel economy of a car given its acceleration, number of cylinders, engine displacement, horsepower, manufacturer, model year, and weight. Consider `Cylinders`, `Mfg`, and `Model_Year` as categorical variables.

```load carsmall Cylinders = categorical(Cylinders); Mfg = categorical(cellstr(Mfg)); Model_Year = categorical(Model_Year); X = table(Acceleration,Cylinders,Displacement,Horsepower,Mfg,... Model_Year,Weight,MPG); ```

Display the number of categories represented in the categorical variables.

```numCylinders = numel(categories(Cylinders)) numMfg = numel(categories(Mfg)) numModelYear = numel(categories(Model_Year)) ```
```numCylinders = 3 numMfg = 28 numModelYear = 3 ```

Because there are 3 categories only in `Cylinders` and `Model_Year`, the standard CART, predictor-splitting algorithm prefers splitting a continuous predictor over these two variables.

Train a regression tree using the entire data set. To grow unbiased trees, specify usage of the curvature test for splitting predictors. Because there are missing values in the data, specify usage of surrogate splits.

```Mdl = fitrtree(X,'MPG','PredictorSelection','curvature','Surrogate','on'); ```

Estimate predictor importance values by summing changes in the risk due to splits on every predictor and dividing the sum by the number of branch nodes. Compare the estimates using a bar graph.

```imp = predictorImportance(Mdl); figure; bar(imp); title('Predictor Importance Estimates'); ylabel('Estimates'); xlabel('Predictors'); h = gca; h.XTickLabel = Mdl.PredictorNames; h.XTickLabelRotation = 45; h.TickLabelInterpreter = 'none'; ```

In this case, `Displacement` is the most important predictor, followed by `Horsepower`.

Input Arguments

collapse all

Sample data used to train the model, specified as a table. Each row of `Tbl` corresponds to one observation, and each column corresponds to one predictor variable. Optionally, `Tbl` can contain one additional column for the response variable. Multi-column variables and cell arrays other than cell arrays of character vectors are not allowed.

If `Tbl` contains the response variable, and you want to use all remaining variables in `Tbl` as predictors, then specify the response variable using `ResponseVarName`.

If `Tbl` contains the response variable, and you want to use only a subset of the remaining variables in `Tbl` as predictors, then specify a formula using `formula`.

If `Tbl` does not contain the response variable, then specify a response variable using `Y`. The length of response variable and the number of rows of `Tbl` must be equal.

Data Types: `table`

Response variable name, specified as the name of a variable in `Tbl`. The response variable must be a numeric vector.

You must specify `ResponseVarName` as a character vector. For example, if `Tbl` stores the response variable `Y` as `Tbl.Y`, then specify it as `'Y'`. Otherwise, the software treats all columns of `Tbl`, including `Y`, as predictors when training the model.

Explanatory model of the response and a subset of the predictor variables, specified as a character vector in the form of `'Y~X1+X2+X3'`. In this form, `Y` represents the response variable, and `X1`, `X2`, and `X3` represent the predictor variables. The variables must be variable names in `Tbl` (`Tbl.Properties.VariableNames`).

To specify a subset of variables in `Tbl` as predictors for training the model, use a formula. If you specify a formula, then the software does not use any variables in `Tbl` that do not appear in `formula`.

Data Types: `char`

Response data, specified as a numeric column vector with the same number of rows as `X`. Each entry in `Y` is the response to the data in the corresponding row of `X`.

The software considers `NaN` values in `Y` to be missing values. `fitrtree` does not use observations with missing values for `Y` in the fit.

Data Types: `single` | `double`

Predictor data, specified as numeric matrix. Each column of `X` represents one variable, and each row represents one observation.

`fitrtree` considers `NaN` values in `X` as missing values. `fitrtree` does not use observations with all missing values for `X` the fit. `fitrtree` uses observations with some missing values for `X` to find splits on variables for which these observations have valid values.

Data Types: `single` | `double`

Name-Value Pair Arguments

Specify optional comma-separated pairs of `Name,Value` arguments. `Name` is the argument name and `Value` is the corresponding value. `Name` must appear inside single quotes (`' '`). You can specify several name and value pair arguments in any order as `Name1,Value1,...,NameN,ValueN`.

Example: `'CrossVal','on','MinParentSize',30` specifies a cross-validated regression tree with a minimum of 30 observations per branch node.

Note

You cannot use any cross-validation name-value pair along with `OptimizeHyperparameters`. You can modify the cross-validation for `OptimizeHyperparameters` only by using the `HyperparameterOptimizationOptions` name-value pair.

Model Parameters

collapse all

Categorical predictors list, specified as the comma-separated pair consisting of `'CategoricalPredictors'` and one of these values:

ValueDescription
Vector of positive integersAn entry in the vector is the index value corresponding to the column of the predictor data (`X` or `Tbl`) that contains a categorical variable.
Logical vectorA `true` entry means that the corresponding column of predictor data (`X` or `Tbl`) is a categorical variable.
Character matrixEach row of the matrix is the name of a predictor variable. The names must match the entries in `PredictorNames`. Pad the names with extra blanks so each row of the character matrix has the same length.
Cell array of character vectorsEach element in the array is the name of a predictor variable. The names must match the entries in `PredictorNames`.
'all'All predictors are categorical.

By default, if the predictor data is in a table (`Tbl`), `fitrtree` assumes that a variable is categorical if it contains logical values, categorical values, or a cell array of character vectors. If the predictor data is a matrix (`X`), `fitrtree` assumes all predictors are continuous. To identify any categorical predictors when the data is a matrix, use the `'CategoricalPredictors'` name-value pair argument.

Example: `'CategoricalPredictors','all'`

Data Types: `single` | `double` | `logical` | `char` | `cell`

Leaf merge flag, specified as the comma-separated pair consisting of `'MergeLeaves'` and `'on'` or `'off'`.

If `MergeLeaves` is `'on'`, then `fitrtree`:

• Merges leaves that originate from the same parent node, and that yields a sum of risk values greater or equal to the risk associated with the parent node

• Estimates the optimal sequence of pruned subtrees, but does not prune the regression tree

Otherwise, `fitrtree` does not merge leaves.

Example: `'MergeLeaves','off'`

Minimum number of branch node observations, specified as the comma-separated pair consisting of `'MinParentSize'` and a positive integer value. Each branch node in the tree has at least `MinParentSize` observations. If you supply both `MinParentSize` and `MinLeafSize`, `fitrtree` uses the setting that gives larger leaves: ```MinParentSize = max(MinParentSize,2*MinLeafSize)```.

Example: `'MinParentSize',8`

Data Types: `single` | `double`

Predictor variable names, specified as the comma-separated pair consisting of `'PredictorNames'` and a cell array of unique character vectors. The functionality of `'PredictorNames'` depends on the way you supply the training data.

• If you supply `X` and `Y`, then you can use `'PredictorNames'` to give the predictor variables in `X` names.

• The order of the names in `PredictorNames` must correspond to the column order of `X`. That is, `PredictorNames{1}` is the name of `X(:,1)`, `PredictorNames{2}` is the name of `X(:,2)`, and so on. Also, `size(X,2)` and `numel(PredictorNames)` must be equal.

• By default, `PredictorNames` is `{'x1','x2',...}`.

• If you supply `Tbl`, then you can use `'PredictorNames'` to choose which predictor variables to use in training. That is, `fitrtree` uses the predictor variables in `PredictorNames` and the response only in training.

• `PredictorNames` must be a subset of `Tbl.Properties.VariableNames` and cannot include the name of the response variable.

• By default, `PredictorNames` contains the names of all predictor variables.

• It good practice to specify the predictors for training using one of `'PredictorNames'` or `formula` only.

Example: `'PredictorNames',{'SepalLength','SepalWidth','PedalLength','PedalWidth'}`

Data Types: `cell`

Algorithm used to select the best split predictor at each node, specified as the comma-separated pair consisting of `'PredictorSelection'` and a value in this table.

ValueDescription
`'allsplits'`

Standard CART — Selects the split predictor that maximizes the split-criterion gain over all possible splits of all predictors [1].

`'curvature'`Curvature test — Selects the split predictor that minimizes the p-value of chi-square tests of independence between each predictor and the response [2]. Training speed is similar to standard CART.
`'interaction-curvature'`Interaction test — Chooses the split predictor that minimizes the p-value of chi-square tests of independence between each predictor and the response (that is, conducts curvature tests), and that minimizes the p-value of a chi-square test of independence between each pair of predictors and response [2]. Training speed can be slower than standard CART.

For `'curvature'` and `'interaction-curvature'`, if all tests yield p-values greater than 0.05, then `fitrtree` stops splitting nodes.

Tip

• Standard CART tends to select split predictors containing many distinct values, e.g., continuous variables, over those containing few distinct values, e.g., categorical variables [3]. Consider specifying the curvature or interaction test if any of the following are true:

• If there are predictors that have relatively fewer distinct values than other predictors, for example, if the predictor data set is heterogeneous.

• If an analysis of predictor importance is your goal. For more on predictor importance estimation, see `predictorImportance`.

• Trees grown using standard CART are not sensitive to predictor variable interactions. Also, such trees are less likely to identify important variables in the presence of many irrelevant predictors than the application of the interaction test. Therefore, to account for predictor interactions and identify importance variables in the presence of many irrelevant variables, specify the interaction test .

• Prediction speed is unaffected by the value of `'PredictorSelection'`.

For details on how `fitrtree` selects split predictors, see Node Splitting Rules.

Example: `'PredictorSelection','curvature'`

Data Types: `char`

Flag to estimate the optimal sequence of pruned subtrees, specified as the comma-separated pair consisting of `'Prune'` and `'on'` or `'off'`.

If `Prune` is `'on'`, then `fitrtree` grows the regression tree and estimates the optimal sequence of pruned subtrees, but does not prune the regression tree. Otherwise, `fitrtree` grows the regression tree without estimating the optimal sequence of pruned subtrees.

To prune a trained regression tree, pass the regression tree to `prune`.

Example: `'Prune','off'`

Pruning criterion, specified as the comma-separated pair consisting of `'PruneCriterion'` and `'mse'`.

Quadratic error tolerance per node, specified as the comma-separated pair consisting of `'QuadraticErrorTolerance'` and a positive scalar value. Splitting nodes stops when quadratic error per node drops below `QuadraticErrorTolerance*QED`, where `QED` is the quadratic error for the entire data computed before the decision tree is grown.

Example: `'QuadraticErrorTolerance',1e-4`

Response variable name, specified as the comma-separated pair consisting of `'ResponseName'` and a character vector.

Example: `'ResponseName','response'`

Data Types: `char`

Response transformation, specified as the comma-separated pair consisting of `'ResponseTransform'` and either `'none'` or a function handle. The default is `'none'`, which means `@(x)x`, or no transformation. For a MATLAB® function or a function you define, use its function handle. The function handle must accept a vector (the original response values) and return a vector of the same size (the transformed response values).

Example: `'ResponseTransform','none'`

Data Types: `char` | `function_handle`

Split criterion, specified as the comma-separated pair consisting of `'SplitCriterion'` and `'MSE'`, meaning mean squared error.

Example: `'SplitCriterion','MSE'`

Surrogate decision splits flag, specified as the comma-separated pair consisting of `'Surrogate'` and `'on'`, `'off'`, `'all'`, or a positive integer.

• When `'on'`, `fitrtree` finds at most 10 surrogate splits at each branch node.

• When set to a positive integer, `fitrtree` finds at most the specified number of surrogate splits at each branch node.

• When set to `'all'`, `fitrtree` finds all surrogate splits at each branch node. The `'all'` setting can use much time and memory.

Use surrogate splits to improve the accuracy of predictions for data with missing values. The setting also enables you to compute measures of predictive association between predictors.

Example: `'Surrogate','on'`

Data Types: `single` | `double` | `char`

Observation weights, specified as the comma-separated pair consisting of `'Weights'` and a vector of scalar values. The software weights the observations in each row of `X` or `Tbl` with the corresponding value in `Weights`. The size of `Weights` must equal the number of rows in `X` or `Tbl`.

If you specify the input data as a table `Tbl`, then `Weights` can be the name of a variable in `Tbl` that contains a numeric vector. In this case, you must specify `Weights` as a character vector. For example, if weights vector `W` is stored as `Tbl.W`, then specify it as `'W'`. Otherwise, the software treats all columns of `Tbl`, including `W`, as predictors when training the model.

`fitrtree` normalizes the weights in each class to add up to 1.

Data Types: `single` | `double`

Cross Validation

collapse all

Cross-validation flag, specified as the comma-separated pair consisting of `'CrossVal'` and either `'on'` or `'off'`.

If `'on'`, `fitrtree` grows a cross-validated decision tree with 10 folds. You can override this cross-validation setting using one of the `'KFold'`, `'Holdout'`, `'Leaveout'`, or `'CVPartition'` name-value pair arguments. You can only use one of these four options (`'KFold'`, `'Holdout'`, `'Leaveout'`, or `'CVPartition'`) at a time when creating a cross-validated tree.

Alternatively, cross-validate `tree` later using the `crossval` method.

Example: `'CrossVal','on'`

Partition for cross-validated tree, specified as the comma-separated pair consisting of `'CVPartition'` and an object created using `cvpartition`.

If you use `'CVPartition'`, you cannot use any of the `'KFold'`, `'Holdout'`, or `'Leaveout'` name-value pair arguments.

Fraction of data used for holdout validation, specified as the comma-separated pair consisting of `'Holdout'` and a scalar value in the range `[0,1]`. Holdout validation tests the specified fraction of the data, and uses the rest of the data for training.

If you use `'Holdout'`, you cannot use any of the `'CVPartition'`, `'KFold'`, or `'Leaveout'` name-value pair arguments.

Example: `'Holdout',0.1`

Data Types: `single` | `double`

Number of folds to use in a cross-validated tree, specified as the comma-separated pair consisting of `'KFold'` and a positive integer value greater than 1.

If you use `'KFold'`, you cannot use any of the `'CVPartition'`, `'Holdout'`, or `'Leaveout'` name-value pair arguments.

Example: `'KFold',8`

Data Types: `single` | `double`

Leave-one-out cross-validation flag, specified as the comma-separated pair consisting of `'Leaveout'` and either `'on'` or `'off`. Use leave-one-out cross validation by setting to `'on'`.

If you use `'Leaveout'`, you cannot use any of the `'CVPartition'`, `'Holdout'`, or `'KFold'` name-value pair arguments.

Example: `'Leaveout','on'`

Hyperparameters

collapse all

Maximal number of decision splits (or branch nodes), specified as the comma-separated pair consisting of `'MaxNumSplits'` and a positive integer. `fitrtree` splits `MaxNumSplits` or fewer branch nodes. For more details on splitting behavior, see Tree Depth Control.

Example: `'MaxNumSplits',5`

Data Types: `single` | `double`

Minimum number of leaf node observations, specified as the comma-separated pair consisting of `'MinLeafSize'` and a positive integer value. Each leaf has at least `MinLeafSize` observations per tree leaf. If you supply both `MinParentSize` and `MinLeafSize`, `fitrtree` uses the setting that gives larger leaves: ```MinParentSize = max(MinParentSize,2*MinLeafSize)```.

Example: `'MinLeafSize',3`

Data Types: `single` | `double`

Number of predictors to select at random for each split, specified as the comma-separated pair consisting of `'NumVariablesToSample'` and a positive integer value. You can also specify `'all'` to use all available predictors.

Example: `'NumVariablesToSample',3`

Data Types: `single` | `double`

Hyperparameter Optimization

collapse all

Parameters to optimize, specified as:

• `'none'` — Do not optimize.

• `'auto'` — Use `{'MinLeafSize'}`.

• `'all'` — Optimize all eligible parameters.

• Cell array of eligible parameter names

• Vector of `optimizableVariable` objects, typically the output of `hyperparameters`

The optimization attempts to minimize the cross-validation loss (error) for `fitrtree` by varying the parameters. To control the cross-validation type and other aspects of the optimization, use the `HyperparameterOptimizationOptions` name-value pair.

Note

`OptimizeHyperparameters` values override any values you set using other name-value pairs. For example, setting `OptimizeHyperparameters` to `'auto'` causes the `'auto'` values to apply.

The eligible parameters for `fitrtree` are:

• `MaxNumSplits``fitrtree` searches among integers, by default log-scaled in the range `[1,max(2,NumObservations-1)]`.

• `MinLeafSize``fitrtree` searches among integers, by default log-scaled in the range `[1,max(2,floor(NumObservations/2))]`.

• `NumVariablesToSample``fitrtree` does not optimize over this hyperparameter. If you pass `NumVariablesToSample` as a parameter name, `fitrtree` simply uses the full number of predictors. However, `fitrensemble` does optimize over this hyperparameter.

Set nondefault parameters by passing a vector of `optimizableVariable` objects that have nondefault values. For example,

```load carsmall params = hyperparameters('fitrtree',[Horsepower,Weight],MPG); params(1).Range = [1,30];```

Pass `params` as the value of `OptimizeHyperparameters`.

By default, iterative display appears at the command line, and plots appear according to the number of hyperparameters in the optimization. For the optimization and plots, the objective function is log(1 + cross-validation loss) for regression, and the misclassification rate for classification. To control the iterative display, set the `HyperparameterOptimizationOptions` name-value pair, `Verbose` field. To control the plots, set the `HyperparameterOptimizationOptions` name-value pair, `ShowPlots` field.

For an example, see Optimize Regression Tree.

Example: `'auto'`

Data Types: `char` | `cell`

Options for optimization, specified as a structure. Modifies the effect of the `OptimizeHyperparameters` name-value pair. All fields in the structure are optional.

Field NameValuesDefault
`Optimizer`
• `'bayesopt'` — Use Bayesian optimization. Internally, this setting calls `bayesopt`.

• `'gridsearch'` — Use grid search with `NumGridDivisions` values per dimension.

• `'randomsearch'` — Search at random among `MaxObjectiveEvaluations` points.

`'gridsearch'` searches in a random order, using uniform sampling without replacement from the grid. After optimization, you can get a table in grid order by using the command `sortrows(Mdl.ParameterOptimizationResults)`.

`'bayesopt'`
`AcquisitionFunctionName`
• `'expected-improvement-per-second-plus'`

• `'expected-improvement'`

• `'expected-improvement-plus'`

• `'expected-improvement-per-second'`

• `'lower-confidence-bound'`

• `'probability-of-improvement'`

For details, see the `bayesopt ``AcquisitionFunctionName` name-value pair, or Acquisition Function Types.
`'expected-improvement-per-second-plus'`
`MaxObjectiveEvaluations`Maximum number of objective function evaluations.`30` for `'bayesopt'` or `'randomsearch'`, and the entire grid for `'gridsearch'`
`MaxTime`

Time limit, specified as a positive real. The time limit is in seconds, as measured by `tic` and `toc`. Run time can exceed `MaxTime` because `MaxTime` does not interrupt function evaluations.

`Inf`
`NumGridDivisions`For `'gridsearch'`, the number of values in each dimension. Can be a vector of positive integers giving the number of values for each dimension, or a scalar that applies to all dimensions. Ignored for categorical variables.`10`
`ShowPlots`Logical value indicating whether to show plots. If `true`, plots the best objective function value against iteration number. If there are one or two optimization parameters, and if `Optimizer` is `'bayesopt'`, then `ShowPlots` also plots a model of the objective function against the parameters.`true`
`SaveIntermediateResults`Logical value indicating whether to save results when `Optimizer` is `'bayesopt'`. If `true`, overwrites a workspace variable named `'BayesoptResults'` at each iteration. The variable is a `BayesianOptimization` object.`false`
`Verbose`Display to the command line.
• `0` — No iterative display

• `1` — Iterative display

• `2` — Iterative display with extra information

For details, see the `bayesopt` `Verbose` name-value pair.
`1`
`UseParallel`Logical value indicating whether to run Bayesian optimization in parallel, which requires Parallel Computing Toolbox™. For details, see Parallel Bayesian Optimization.`false`
`Repartition`

Logical value indicating whether to repartition the cross-validation at every iteration. If `false`, the optimizer uses a single partition for the optimization.

`true` usually gives the most robust results because this setting takes partitioning noise into account. However, for good results, `true` requires at least twice as many function evaluations.

`false`
Use no more than one of the following three field names.
`CVPartition`A `cvpartition` object, as created by `cvpartition``Kfold` = `5`
`Holdout`A scalar in the range `(0,1)` representing the holdout fraction.
`Kfold`An integer greater than 1.

Example: `struct('MaxObjectiveEvaluations',60)`

Data Types: `struct`

Output Arguments

collapse all

Regression tree, returned as a regression tree object. Using the `'Crossval'`, `'KFold'`, `'Holdout'`, `'Leaveout'`, or `'CVPartition'` options results in a tree of class `RegressionPartitionedModel`. You cannot use a partitioned tree for prediction, so this kind of tree does not have a `predict` method.

Otherwise, `tree` is of class `RegressionTree`, and you can use the `predict` method to make predictions.

collapse all

Curvature Test

The curvature test is a statistical test assessing the null hypothesis that two variables are unassociated.

The curvature test between predictor variable x and y is conducted using this process.

1. If x is continuous, then partition it into its quartiles. Create a nominal variable that bins observations according to which section of the partition they occupy. If there are missing values, then create an extra bin for them.

2. For each level in the partitioned predictor j = 1...J and class in the response k = 1,...,K, compute the weighted proportion of observations in class k

`${\stackrel{^}{\pi }}_{jk}=\sum _{i=1}^{n}I\left\{{y}_{i}=k\right\}{w}_{i}.$`
wi is the weight of observation i, $\sum {w}_{i}=1$, I is the indicator function, and n is the sample size. If all observations have the same weight, then ${\stackrel{^}{\pi }}_{jk}=\frac{{n}_{jk}}{n}$, where njk is the number of observations in level j of the predictor that are in class k.

3. Compute the test statistic

`$t=n\sum _{k=1}^{K}\sum _{j=1}^{J}\frac{{\left({\stackrel{^}{\pi }}_{jk}-{\stackrel{^}{\pi }}_{j+}{\stackrel{^}{\pi }}_{+k}\right)}^{2}}{{\stackrel{^}{\pi }}_{j+}{\stackrel{^}{\pi }}_{+k}}$`
${\stackrel{^}{\pi }}_{j+}=\sum _{k}{\stackrel{^}{\pi }}_{jk}$, that is, the marginal probability of observing the predictor at level j. ${\stackrel{^}{\pi }}_{+k}=\sum _{j}{\stackrel{^}{\pi }}_{jk}$, that is the marginal probability of observing class k. If n is large enough, then t is distributed as a χ2 with (K – 1)(J – 1) degrees of freedom.

4. If the p-value for the test is less than 0.05, then reject the null hypothesis that there is no association between x and y.

When determining the best split predictor at each node, the standard CART algorithm prefers to select continuous predictors that have many levels. Sometimes, such a selection can be spurious and can also mask more important predictors that have fewer levels, such as categorical predictors.

The curvature test can be applied instead of standard CART to determine the best split predictor at each node. In that case, the best split predictor variable is the one that minimizes the significant p-values (those less than 0.05) of curvature tests between each predictor and the response variable. Such a selection is robust to the number of levels in individual predictors.

For more details on how the curvature test applies to growing regression trees, see Node Splitting Rules and [3].

Interaction Test

The interaction test is a statistical test that assesses the null hypothesis that there is no interaction between a pair of predictor variables and the response variable.

The interaction test assessing the association between predictor variables x1 and x2 with respect to y is conducted using this process.

1. If x1 or x2 is continuous, then partition that variable into its quartiles. Create a nominal variable that bins observations according to which section of the partition they occupy. If there are missing values, then create an extra bin for them.

2. Create the nominal variable z with J = J1J2 levels that assigns an index to observation i according to which levels of x1 and x2 it belongs. Remove any levels of z that do not correspond to any observations.

3. Conduct a curvature test between z and y.

When growing decision trees, if there are important interactions between pairs of predictors, but there are also many other less important predictors in the data, then standard CART tends to miss the important interactions. However, conducting curvature and interaction tests for predictor selection instead can improve detection of important interactions, which can yield more accurate decision trees.

For more details on how the interaction test applies to growing decision trees, see Curvature Test, Node Splitting Rules and [2].

Predictive Measure of Association

The predictive measure of association is a value that indicates the similarity between decision rules that split observations. Among all possible decision splits that are compared to the optimal split (found by growing the tree), the best surrogate decision split yields the maximum predictive measure of association. The second-best surrogate split has the second-largest predictive measure of association.

Suppose xj and xk are predictor variables j and k, respectively, and jk. At node t, the predictive measure of association between the optimal split xj < u and a surrogate split xk < v is

`${\lambda }_{jk}=\frac{\text{min}\left({P}_{L},{P}_{R}\right)-\left(1-{P}_{{L}_{j}{L}_{k}}-{P}_{{R}_{j}{R}_{k}}\right)}{\text{min}\left({P}_{L},{P}_{R}\right)}.$`
• PL is the proportion of observations in node t, such that xj < u. The subscript L stands for the left child of node t.

• PR is the proportion of observations in node t, such that xju. The subscript R stands for the right child of node t.

• ${P}_{{L}_{j}{L}_{k}}$ is the proportion of observations at node t, such that xj < u and xk < v.

• ${P}_{{R}_{j}{R}_{k}}$ is the proportion of observations at node t, such that xju and xkv.

• Observations with missing values for xj or xk do not contribute to the proportion calculations.

λjk is a value in (–∞,1]. If λjk > 0, then xk < v is a worthwhile surrogate split for xj < u.

Surrogate Decision Splits

A surrogate decision split is an alternative to the optimal decision split at a given node in a decision tree. The optimal split is found by growing the tree; the surrogate split uses a similar or correlated predictor variable and split criterion.

When the value of the optimal split predictor for an observation is missing, the observation is sent to the left or right child node using the best surrogate predictor. When the value of the best surrogate split predictor for the observation is also missing, the observation is sent to the left or right child node using the second-best surrogate predictor, and so on. Candidate splits are sorted in descending order by their predictive measure of association.

Algorithms

collapse all

Node Splitting Rules

`fitrtree` uses these processes to determine how to split node t.

• For standard CART (that is, if `PredictorSelection` is `'allpairs'`) and for all predictors xi, i = 1,...,p:

1. `fitrtree` computes the weighted, mean-square error (MSE) of the responses in node t using

`${\epsilon }_{t}=\sum _{j\in T}{w}_{j}{\left({y}_{j}-{\overline{y}}_{t}\right)}^{2}.$`
wj is the weight of observation j, and T is the set of all observation indices in node t. If you do not specify `Weights`, then wj = 1/n, where n is the sample size.

2. `fitrtree` estimates the probability that an observation is in node t using

`$P\left(T\right)=\sum _{j\in T}{w}_{j}.$`

3. `fitrtree` sorts xi in ascending order. Each element of the sorted predictor is a splitting candidate or cut point. `fitrtree` records any indices corresponding to missing values in the set TU, which is the unsplit set.

4. `fitrtree` determines the best way to split node t using xi by maximizing the reduction in MSE (ΔI) over all splitting candidates. That is, for all splitting candidates in xi:

1. `fitrtree` splits the observations in node t into left and right child nodes (tL and tR, respectively).

2. `fitrtree` computes ΔI. Suppose that for a particular splitting candidate, tL and tR contain observation indices in the sets TL and TR, respectively.

• If xi does not contain any missing values, then the reduction in MSE for the current splitting candidate is

`$\Delta I=P\left(T\right){\epsilon }_{t}-P\left({T}_{L}\right){\epsilon }_{{t}_{L}}-P\left({T}_{R}\right){\epsilon }_{{t}_{R}}.$`

• If xi contains missing values, then, assuming that the observations are missing at random, the reduction in MSE is

`$\Delta {I}_{U}=P\left(T-{T}_{U}\right){\epsilon }_{t}-P\left({T}_{L}\right){\epsilon }_{{t}_{L}}-P\left({T}_{R}\right){\epsilon }_{{t}_{R}}.$`
TTU is the set of all observation indices in node t that are not missing.

• If you use surrogate decision splits, then:

1. `fitrtree` computes the predictive measures of association between the decision split xj < u and all possible decision splits xk < v, jk.

2. `fitrtree` sorts the possible alternative decision splits in descending order by their predictive measure of association with the optimal split. The surrogate split is the decision split yielding the largest measure.

3. `fitrtree` decides the child node assignments for observations with a missing value for xi using the surrogate split. If the surrogate predictor also contains a missing value, then `fitrtree` uses the decision split with the second largest measure, and so on, until there are no other surrogates. It is possible for `fitrtree` to split two different observations at node t using two different surrogate splits. For example, suppose the predictors x1 and x2 are the best and second best surrogates, respectively, for the predictor xi, i ∉ {1,2}, at node t. If observation m of predictor xi is missing (i.e., xmi is missing), but xm1 is not missing, then x1 is the surrogate predictor for observation xmi. If observations x(m + 1),i and x(m + 1),1 are missing, but x(m + 1),2 is not missing, then x2 is the surrogate predictor for observation m + 1.

4. `fitrtree` uses the appropriate MSE reduction formula. That is, if `fitrtree` fails to assign all missing observations in node t to children nodes using surrogate splits, then the MSE reduction is ΔIU. Otherwise, `fitrtree` uses ΔI for the MSE reduction.

3. `fitrtree` chooses the candidate that yields the largest MSE reduction.

`fitrtree` splits the predictor variable at the cut point that maximizes the MSE reduction.

• For the curvature test (that is, if `PredictorSelection` is `'curvature'`):

1. `fitrtree` computes the residuals ${r}_{ti}={y}_{ti}-{\overline{y}}_{t}$ for all observations in node t. ${\overline{y}}_{t}=\frac{1}{{\sum }_{i}{w}_{i}}{\sum }_{i}{w}_{i}{y}_{ti}$, which is the weighted average of the responses in node t. The weights are the observation weights in `Weights`.

2. `fitrtree` assigns observations to one of two bins according to the sign of the corresponding residuals. Let zt be a nominal variable that contains the bin assignments for the observations in node t.

3. `fitrtree` conducts curvature tests between each predictor and zt. For regression trees, K = 2.

• If all p-values are at least 0.05, then `fitrtree` does not split node t.

• If there is a minimal p-value, then `fitrtree` chooses the corresponding predictor to split node t.

• If more than one p-value is zero due to underflow, then `fitrtree` applies standard CART to the corresponding predictors to choose the split predictor.

4. If `fitrtree` chooses a split predictor, then it uses standard CART to choose the cut point (see step 4 in the standard CART process).

For the interaction test (that is, if `PredictorSelection` is `'interaction-curvature'` ):

1. For observations in node t, `fitrtree` conducts curvature tests between each predictor and the response and interaction tests between each pair of predictors and the response.

• If all p-values are at least 0.05, then `fitrtree` does not split node t.

• If there is a minimal p-value and it is the result of a curvature test, then `fitrtree` chooses the corresponding predictor to split node t.

• If there is a minimal p-value and it is the result of an interaction test, then `fitrtree` chooses the split predictor using standard CART on the corresponding pair of predictors.

• If more than one p-value is zero due to underflow, then `fitrtree` applies standard CART to the corresponding predictors to choose the split predictor.

2. If `fitrtree` chooses a split predictor, then it uses standard CART to choose the cut point (see step 4 in the standard CART process).

Tree Depth Control

• If `MergeLeaves` is `'on'` and `PruneCriterion` is `'mse'` (which are the default values for these name-value pair arguments), then the software applies pruning only to the leaves and by using MSE. This specification amounts to merging leaves coming from the same parent node whose MSE is at most the sum of the MSE of its two leaves.

• To accommodate `MaxNumSplits`, `fitrtree` splits all nodes in the current layer, and then counts the number of branch nodes. A layer is the set of nodes that are equidistant from the root node. If the number of branch nodes exceeds `MaxNumSplits`, `fitrtree` follows this procedure:

1. Determine how many branch nodes in the current layer must be unsplit so that there are at most `MaxNumSplits` branch nodes.

2. Sort the branch nodes by their impurity gains.

3. Unsplit the number of least successful branches.

4. Return the decision tree grown so far.

This procedure produces maximally balanced trees.

• The software splits branch nodes layer by layer until at least one of these events occurs:

• There are `MaxNumSplits` branch nodes.

• A proposed split causes the number of observations in at least one branch node to be fewer than `MinParentSize`.

• A proposed split causes the number of observations in at least one leaf node to be fewer than `MinLeafSize`.

• The algorithm cannot find a good split within a layer (i.e., the pruning criterion (see `PruneCriterion`), does not improve for all proposed splits in a layer). A special case is when all nodes are pure (i.e., all observations in the node have the same class).

• For values `'curvature'` or `'interaction-curvature'` of `PredictorSelection`, all tests yield p-values greater than 0.05.

`MaxNumSplits` and `MinLeafSize` do not affect splitting at their default values. Therefore, if you set `'MaxNumSplits'`, splitting might stop due to the value of `MinParentSize`, before `MaxNumSplits` splits occur.

Parallelization

For dual-core systems and above, `fitrtree` parallelizes training decision trees using Intel® Threading Building Blocks (TBB). For details on Intel TBB, see https://software.intel.com/en-us/intel-tbb.

References

[1] Breiman, L., J. Friedman, R. Olshen, and C. Stone. Classification and Regression Trees. Boca Raton, FL: CRC Press, 1984.

[2] Loh, W.Y. “Regression Trees with Unbiased Variable Selection and Interaction Detection.” Statistica Sinica, Vol. 12, 2002, pp. 361–386.

[3] Loh, W.Y. and Y.S. Shih. “Split Selection Methods for Classification Trees.” Statistica Sinica, Vol. 7, 1997, pp. 815–840.