Documentation |
Create generalized linear regression model by stepwise regression
mdl = stepwiseglm(tbl,modelspec) creates a generalized linear model of a table or dataset array tbl, using stepwise regression to add or remove predictors. modelspec is the starting model for the stepwise procedure.
mdl = stepwiseglm(X,y,modelspec) creates a generalized linear model of the responses y to a data matrix X, using stepwise regression to add or remove predictors.
mdl = stepwiseglm(...,modelspec,Name,Value) creates a generalized linear model with additional options specified by one or more Name,Value pair arguments.
Create response data using just three of 20 predictors, and create a generalized linear model using stepwise algorithm to see if it uses just the correct predictors.
Create data with 20 predictors, and Poisson response using just three of the predictors, plus a constant.
rng('default') % for reproducibility X = randn(100,20); mu = exp(X(:,[5 10 15])*[.4;.2;.3] + 1); y = poissrnd(mu);
Fit a generalized linear model using the Poisson distribution.
mdl = stepwiseglm(X,y,... 'constant','upper','linear','Distribution','poisson')
1. Adding x5, Deviance = 134.439, Chi2Stat = 52.24814, PValue = 4.891229e-13 2. Adding x15, Deviance = 106.285, Chi2Stat = 28.15393, PValue = 1.1204e-07 3. Adding x10, Deviance = 95.0207, Chi2Stat = 11.2644, PValue = 0.000790094 mdl = Generalized Linear regression model: log(y) ~ 1 + x5 + x10 + x15 Distribution = Poisson Estimated Coefficients: Estimate SE tStat pValue (Intercept) 1.0115 0.064275 15.737 8.4217e-56 x5 0.39508 0.066665 5.9263 3.0977e-09 x10 0.18863 0.05534 3.4085 0.0006532 x15 0.29295 0.053269 5.4995 3.8089e-08 100 observations, 96 error degrees of freedom Dispersion: 1 Chi^2-statistic vs. constant model: 91.7, p-value = 9.61e-20
The starting model is the constant model. stepwiseglm by default uses deviance of the model as the criterion. It first adds x5 into the model, as the p-value for the test statistic, deviance (the differences in the deviances of the two models), is less than the default threshold value 0.05. Then, it adds x15 because given x5 is in the model, when x15 is added, the p-value for chi-squared test is smaller than 0.05. It then adds x10 because given x5 and x15 are in the model, when x10 is added, the p-value for the chi-square test statistic is again less than 0.05.
Input data, specified as a table or dataset array. When modelspec is a formula, it specifies the variables to be used as the predictors and response. Otherwise, if you do not specify the predictor and response variables, the last variable is the response variable and the others are the predictor variables by default.
Predictor variables can be numeric, or any grouping variable type, such as logical or categorical (see Grouping Variables). The response must be numeric or logical.
To set a different column as the response variable, use the ResponseVar name-value pair argument. To use a subset of the columns as predictors, use the PredictorVars name-value pair argument.
Data Types: single | double | logical
Predictor variables, specified as an n-by-p matrix, where n is the number of observations and p is the number of predictor variables. Each column of X represents one variable, and each row represents one observation.
By default, there is a constant term in the model, unless you explicitly remove it, so do not include a column of 1s in X.
Data Types: single | double | logical
Response variable, specified as an n-by-1 vector, where n is the number of observations. Each entry in y is the response for the corresponding row of X.
Data Types: single | double
Starting model for stepwiseglm, specified as one of the following:
String specifying the type of model.
String | Model Type |
---|---|
'constant' | Model contains only a constant (intercept) term. |
'linear' | Model contains an intercept and linear terms for each predictor. |
'interactions' | Model contains an intercept, linear terms, and all products of pairs of distinct predictors (no squared terms). |
'purequadratic' | Model contains an intercept, linear terms, and squared terms. |
'quadratic' | Model contains an intercept, linear terms, interactions, and squared terms. |
'polyijk' | Model is a polynomial with all terms up to degree i in the first predictor, degree j in the second predictor, etc. Use numerals 0 through 9. For example, 'poly2111' has a constant plus all linear and product terms, and also contains terms with predictor 1 squared. |
t-by-(p+1) matrix, namely terms matrix, specifying terms to include in model, where t is the number of terms and p is the number of predictor variables, and plus one is for the response variable.
String representing a formula in the form
'Y ~ terms',
where the terms are in Wilkinson Notation.
Specify optional comma-separated pairs of Name,Value arguments. Name is the argument name and Value is the corresponding value. Name must appear inside single quotes (' '). You can specify several name and value pair arguments in any order as Name1,Value1,...,NameN,ValueN.
Example: 'Criterion','aic','Distribution','poisson','Upper','interactions' specifies Akaike Information Criterion as the criterion to add or remove variables to the model, Poisson distribution as the distribution of the response variable, and a model with all possible interactions as the largest model to consider as the fit.Number of trials for binomial distribution, that is the sample size, specified as the comma-separated pair consisting of a scalar value or a vector of the same length as the response. This is the parameter n for the fitted binomial distribution. BinomialSize applies only when the Distribution parameter is 'binomial'.
If BinomialSize is a scalar value, that means all observations have the same number of trials.
As an alternative to BinomialSize, you can specify the response as a two-column vector with counts in column 1 and BinomialSize in column 2.
Data Types: single | double
Categorical variables in the fit, specified as the comma-separated pair consisting of 'CategoricalVars' and either a cell array of strings of the names of the categorical variables in the table or dataset array tbl, or a logical or numeric index vector indicating which columns are categorical.
If data is in a table or dataset array tbl, then the default is to treat all categorical or logical variables, character arrays, or cell arrays of strings as categorical variables.
If data is in matrix X, then the default value of this name-value pair argument is an empty matrix []. That is, no variable is categorical unless you specify it.
For example, you can specify the observations 2 and 3 out of 6 as categorical using either of the following examples.
Example: 'CategoricalVars',[2,3]
Example: 'CategoricalVars',logical([0 1 1 0 0 0])
Data Types: single | double | logical
Criterion to add or remove terms, specified as the comma-separated pair consisting of 'Criterion' and one of the following:
'Deviance' — Default for stepwiseglm. p-value for F or chi-squared test of the change in the deviance by adding or removing the term. F-test is for testing a single model. Chi-squared test is for comparing two different models. This option is not valid for stepwiselm.
'sse' — Default for stepwiselm. p-value for an F-test of the change in the sum of squared error by adding or removing the term.
'aic' — Change in the value of Akaike information criterion (AIC).
'bic' — Change in the value of Bayesian information criterion (BIC).
'rsquared' — Increase in the value of R^{2}.
'adjrsquared' — Increase in the value of adjusted R^{2}.
Example: 'Criterion','bic'
Indicator to compute dispersion parameter for 'binomial' and 'poisson' distributions, specified as the comma-separated pair consisting of 'DispersionFlag' and one of the following.
true | Estimate a dispersion parameter when computing standard errors |
false | Default. Use the theoretical value when computing standard errors |
The fitting function always estimates the dispersion for other distributions.
Example: 'DispersionFlag',true
Distribution of the response variable, specified as the comma-separated pair consisting of 'Distribution' and one of the following.
'normal' | Normal distribution |
'binomial' | Binomial distribution |
'poisson' | Poisson distribution |
'gamma' | Gamma distribution |
'inverse gaussian' | Inverse Gaussian distribution |
Example: 'Distribution','gamma'
Observations to exclude from the fit, specified as the comma-separated pair consisting of 'Exclude' and a logical or numeric index vector indicating which observations to exclude from the fit.
For example, you can exclude observations 2 and 3 out of 6 using either of the following examples.
Example: 'Exclude',[2,3]
Example: 'Exclude',logical([0 1 1 0 0 0])
Data Types: single | double | logical
Indicator the for constant term (intercept) in the fit, specified as the comma-separated pair consisting of 'Intercept' and either true to include or false to remove the constant term from the model.
Use 'Intercept' only when specifying the model using a string, not a formula or matrix.
Example: 'Intercept',false
Link function to use in place of the canonical link function, specified as the comma-separated pair consisting of 'Link' and one of the following.
Link Function Name | Link Function | Mean (Inverse) Function |
---|---|---|
'identity' | f(μ) = μ | μ = Xb |
'log' | f(μ) = log(μ) | μ = exp(Xb) |
'logit' | f(μ) = log(μ/(1–μ)) | μ = exp(Xb) / (1 + exp(Xb)) |
'probit' | f(μ) = Φ^{–1}(μ) | μ = Φ(Xb) |
'comploglog' | f(μ) = log(–log(1 – μ)) | μ = 1 – exp(–exp(Xb)) |
'reciprocal' | f(μ) = 1/μ | μ = 1/(Xb) |
p (a number) | f(μ) = μ^{p} | μ = Xb^{1/p} |
S (a structure) with three fields. Each field holds a function handle that accepts a vector of inputs and returns a vector of the same size:
| f(μ) = S.Link(μ) | μ = S.Inverse(Xb) |
The link function defines the relationship f(μ) = X*b between the mean response μ and the linear combination of predictors X*b.
For more information on the canonical link functions, see Definitions.
Example: 'Link','probit'
Model specification describing terms that cannot be removed from the model, specified as the comma-separated pair consisting of 'Lower' and one of the string options for modelspec naming the model.
Example: 'Lower','linear'
Offset variable in the fit, specified as the comma-separated pair consisting of 'Offset' and a vector or name of a variable with the same length as the response.
fitglm and stepwiseglm use Offset as an additional predictor, with a coefficient value fixed at 1.0. In other words, the formula for fitting is
μ ~ Offset + (terms involving real predictors)
with the Offset predictor having coefficient 1.
For example, consider a Poisson regression model. Suppose the number of counts is known for theoretical reasons to be proportional to a predictor A. By using the log link function and by specifying log(A) as an offset, you can force the model to satisfy this theoretical constraint.
Data Types: single | double | char
Improvement measure for adding a term, specified as the comma-separated pair consisting of 'PEnter' and a scalar value. The default values are below.
Criterion | Default value | Decision |
---|---|---|
'Deviance' | 0.05 | If the p-value of F or chi-squared statistic is smaller than PEnter, add the term to the model. |
'SSE' | 0.05 | If the SSE of the model is smaller than PEnter, add the term to the model. |
'AIC' | 0 | If the change in the AIC of the model is smaller than PEnter, add the term to the model. |
'BIC' | 0 | If the change in the BIC of the model is smaller than PEnter, add the term to the model. |
'Rsquared' | 0.1 | If the increase in the R-squared of the model is larger than PEnter, add the term to the model. |
'AdjRsquared' | 0 | If the increase in the adjusted R-squared of the model is larger than PEnter, add the term to the model. |
For more information on the criteria, see Criterion name-value pair argument.
Example: 'PEnter',0.075
Predictor variables to use in the fit, specified as the comma-separated pair consisting of 'PredictorVars' and either a cell array of strings of the variable names in the table or dataset array tbl, or a logical or numeric index vector indicating which columns are predictor variables.
The strings should be among the names in tbl, or the names you specify using the 'VarNames' name-value pair argument.
The default is all variables in X, or all variables in tbl except for ResponseVar.
For example, you can specify the second and third variables as the predictor variables using either of the following examples.
Example: 'PredictorVars',[2,3]
Example: 'PredictorVars',logical([0 1 1 0 0 0])
Data Types: single | double | logical | cell
Improvement measure for removing a term, specified as the comma-separated pair consisting of 'PRemove' and a scalar value.
Criterion | Default value | Decision |
---|---|---|
'Deviance' | 0.10 | If the p-value of F or chi-squared statistic is larger than PRemove, remove the term from the model. |
'SSE' | 0.10 | If the p-value of the F statistic is larger than PRemove, remove the term from the model. |
'AIC' | 0.01 | If the change in the AIC of the model is larger than PRemove, remove the term from the model. |
'BIC' | 0.01 | If the change in the BIC of the model is larger than PRemove, remove the term from the model. |
'Rsquared' | 0.05 | If the increase in the R-squared value of the model is smaller than PRemove, remove the term from the model. |
'AdjRsquared' | -0.05 | If the increase in the adjusted R-squared value of the model is smaller than PRemove, remove the term from the model. |
At each step, stepwise algorithm also checks whether any term is redundant (linearly dependent) with other terms in the current model. When any term is linearly dependent with other terms in the current model, it is removed, regardless of the criterion value.
For more information on the criteria, see Criterion name-value pair argument.
Example: 'PRemove',0.05
Response variable to use in the fit, specified as the comma-separated pair consisting of 'ResponseVar' and either a string of the variable name in the table or dataset array tbl, or a logical or numeric index vector indicating which column is the response variable. You typically need to use 'ResponseVar' when fitting a table or dataset array tbl.
For example, you can specify the fourth variable, say yield, as the response out of six variables, in one of the following ways.
Example: 'ResponseVar','yield'
Example: 'ResponseVar',[4]
Example: 'ResponseVar',logical([0 0 0 1 0 0])
Data Types: single | double | logical | char
Model specification describing the largest set of terms in the fit, specified as the comma-separated pair consisting of 'Upper' and one of the string options for modelspec naming the model.
Example: 'Upper','quadratic'
Names of variables in fit, specified as the comma-separated pair consisting of 'VarNames' and a cell array of strings including the names for the columns of X first, and the name for the response variable y last.
'VarNames' is not applicable to variables in a table or dataset array, because those variables already have names.
For example, if in your data, horsepower, acceleration, and model year of the cars are the predictor variables, and miles per gallon (MPG) is the response variable, then you can name the variables as follows.
Example: 'VarNames',{'Horsepower','Acceleration','Model_Year','MPG'}
Data Types: cell
Observation weights, specified as the comma-separated pair consisting of 'Weights' and an n-by-1 vector of nonnegative scalar values, where n is the number of observations.
Data Types: single | double
Generalized linear model representing a least-squares fit of the link of the response to the data, returned as a GeneralizedLinearModel object.
For properties and methods of the generalized linear model object, mdl, see the GeneralizedLinearModel class page.
Use fitglm to create a model with a fixed specification. Use step, addTerms, or removeTerms to adjust a fitted model.
A terms matrix is a t-by-(p + 1) matrix specifying terms in a model, where t is the number of terms, p is the number of predictor variables, and plus one is for the response variable.
The value of T(i,j) is the exponent of variable j in term i. Suppose there are three predictor variables A, B, and C:
[0 0 0 0] % Constant term or intercept [0 1 0 0] % B; equivalently, A^0 * B^1 * C^0 [1 0 1 0] % A*C [2 0 0 0] % A^2 [0 1 2 0] % B*(C^2)
The 0 at the end of each term represents the response variable. In general,
If you have the variables in a table or dataset array, then 0 must represent the response variable depending on the position of the response variable. The following example illustrates this.
Load the sample data and define the dataset array.
load hospital ds = dataset(hospital.Sex,hospital.BloodPressure(:,1),hospital.Age,... hospital.Smoker,'VarNames',{'Sex','BloodPressure','Age','Smoker'});
Represent the linear model 'BloodPressure ~ 1 + Sex + Age + Smoker' in a terms matrix. The response variable is in the second column of the dataset array, so there must be a column of 0s for the response variable in the second column of the terms matrix.
T = [0 0 0 0;1 0 0 0;0 0 1 0;0 0 0 1]
T = 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1
Redefine the dataset array.
ds = dataset(hospital.BloodPressure(:,1),hospital.Sex,hospital.Age,... hospital.Smoker,'VarNames',{'BloodPressure','Sex','Age','Smoker'});
Now, the response variable is the first term in the dataset array. Specify the same linear model, 'BloodPressure ~ 1 + Sex + Age + Smoker', using a terms matrix.
T = [0 0 0 0;0 1 0 0;0 0 1 0;0 0 0 1]
T = 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1
If you have the predictor and response variables in a matrix and column vector, then you must include 0 for the response variable at the end of each term. The following example illustrates this.
Load the sample data and define the matrix of predictors.
load carsmall
X = [Acceleration,Weight];
Specify the model 'MPG ~ Acceleration + Weight + Acceleration:Weight + Weight^2' using a term matrix and fit the model to the data. This model includes the main effect and two-way interaction terms for the variables, Acceleration and Weight, and a second-order term for the variable, Weight.
T = [0 0 0;1 0 0;0 1 0;1 1 0;0 2 0]
T = 0 0 0 1 0 0 0 1 0 1 1 0 0 2 0
Fit a linear model.
mdl = fitlm(X,MPG,T)
mdl = Linear regression model: y ~ 1 + x1*x2 + x2^2 Estimated Coefficients: Estimate SE tStat pValue (Intercept) 48.906 12.589 3.8847 0.00019665 x1 0.54418 0.57125 0.95261 0.34337 x2 -0.012781 0.0060312 -2.1192 0.036857 x1:x2 -0.00010892 0.00017925 -0.6076 0.545 x2^2 9.7518e-07 7.5389e-07 1.2935 0.19917 Number of observations: 94, Error degrees of freedom: 89 Root Mean Squared Error: 4.1 R-squared: 0.751, Adjusted R-Squared 0.739 F-statistic vs. constant model: 67, p-value = 4.99e-26
Only the intercept and x2 term, which correspond to the Weight variable, are significant at the 5% significance level.
Now, perform a stepwise regression with a constant model as the starting model and a linear model with interactions as the upper model.
T = [0 0 0;1 0 0;0 1 0;1 1 0];
mdl = stepwiselm(X,MPG,[0 0 0],'upper',T)
1. Adding x2, FStat = 259.3087, pValue = 1.643351e-28 mdl = Linear regression model: y ~ 1 + x2 Estimated Coefficients: Estimate SE tStat pValue (Intercept) 49.238 1.6411 30.002 2.7015e-49 x2 -0.0086119 0.0005348 -16.103 1.6434e-28 Number of observations: 94, Error degrees of freedom: 92 Root Mean Squared Error: 4.13 R-squared: 0.738, Adjusted R-Squared 0.735 F-statistic vs. constant model: 259, p-value = 1.64e-28
The results of the stepwise regression are consistent with the results of fitlm in the previous step.
A formula for model specification is a string of the form 'Y ~ terms'
where
Y is the response name.
terms contains
Variable names
+ means include the next variable
- means do not include the next variable
: defines an interaction, a product of terms
* defines an interaction and all lower-order terms
^ raises the predictor to a power, exactly as in * repeated, so ^ includes lower order terms as well
() groups terms
Note: Formulas include a constant (intercept) term by default. To exclude a constant term from the model, include -1 in the formula. |
For example,
'Y ~ A + B + C' means a three-variable
linear model with intercept.
'Y ~ A + B +
C - 1' is a three-variable linear model without intercept.
'Y ~ A + B + C + B^2' is a three-variable
model with intercept and a B^2 term.
'Y
~ A + B^2 + C' is the same as the previous example because B^2 includes
a B term.
'Y ~ A + B +
C + A:B' includes an A*B term.
'Y
~ A*B + C' is the same as the previous example because A*B
= A + B + A:B.
'Y ~ A*B*C - A:B:C' has
all interactions among A, B,
and C, except the three-way interaction.
'Y
~ A*(B + C + D)' has all linear terms, plus products of A with
each of the other variables.
Wilkinson notation describes the factors present in models. The notation relates to factors present in models, not to the multipliers (coefficients) of those factors.
Wilkinson Notation | Factors in Standard Notation |
---|---|
1 | Constant (intercept) term |
A^k, where k is a positive integer | A, A^{2}, ..., A^{k} |
A + B | A, B |
A*B | A, B, A*B |
A:B | A*B only |
-B | Do not include B |
A*B + C | A, B, C, A*B |
A + B + C + A:B | A, B, C, A*B |
A*B*C - A:B:C | A, B, C, A*B, A*C, B*C |
A*(B + C) | A, B, C, A*B, A*C |
Statistics Toolbox™ notation always includes a constant term unless you explicitly remove the term using -1.
The default link function for a generalized linear model is the canonical link function.
Canonical Link Functions for Generalized Linear Models
Distribution | Link Function Name | Link Function | Mean (Inverse) Function |
---|---|---|---|
'normal' | 'identity' | f(μ) = μ | μ = Xb |
'binomial' | 'logit' | f(μ) = log(μ/(1–μ)) | μ = exp(Xb) / (1 + exp(Xb)) |
'poisson' | 'log' | f(μ) = log(μ) | μ = exp(Xb) |
'gamma' | -1 | f(μ) = 1/μ | μ = 1/(Xb) |
'inverse gaussian' | -2 | f(μ) = 1/μ^{2} | μ = (Xb)^{–1/2} |
The generalized linear model mdl is a standard linear model unless you specify otherwise with the Distribution name-value pair.
For other methods such as devianceTest, or properties of the GeneralizedLinearModel object, see GeneralizedLinearModel.
Stepwise regression is a systematic method for adding and removing terms from a linear or generalized linear model based on their statistical significance in explaining the response variable. The method begins with an initial model, specified using modelspec, and then compares the explanatory power of incrementally larger and smaller models.
MATLAB^{®} uses forward and backward stepwise regression to determine a final model. At each step, the method searches for terms to add to or remove from the model based on the value of the 'Criterion' argument. The default value of 'Criterion' is 'sse', and in this case, stepwiselm uses the p-value of an F-statistic to test models with and without a potential term at each step. If a term is not currently in the model, the null hypothesis is that the term would have a zero coefficient if added to the model. If there is sufficient evidence to reject the null hypothesis, the term is added to the model. Conversely, if a term is currently in the model, the null hypothesis is that the term has a zero coefficient. If there is insufficient evidence to reject the null hypothesis, the term is removed from the model.
Here is how stepwise proceeds when 'Criterion' is 'sse':
Fit the initial model.
If any terms not in the model have p-values less than an entrance tolerance (that is, if it is unlikely that they would have zero coefficient if added to the model), add the one with the smallest p-value and repeat this step; otherwise, go to step 3.
If any terms in the model have p-values greater than an exit tolerance (that is, the hypothesis of a zero coefficient can be rejected), remove the one with the largest p-value and go to step 2; otherwise, end.
The default for stepwiseglm is 'Deviance' and it follows a similar procedure for adding or removing terms.
There are several other criteria available, which you can specify using the 'Criterion' argument. You can use the change in the value of the Akaike information criterion, Bayesian information criterion, R-squared, adjusted R-squared as a criterion to add or remove terms.
Depending on the terms included in the initial model and the order in which terms are moved in and out, the method might build different models from the same set of potential terms. The method terminates when no single step improves the model. There is no guarantee, however, that a different initial model or a different sequence of steps will not lead to a better fit. In this sense, stepwise models are locally optimal, but might not be globally optimal.
[1] Collett, D. Modeling Binary Data. New York: Chapman & Hall, 2002.
[2] Dobson, A. J. An Introduction to Generalized Linear Models. New York: Chapman & Hall, 1990.
[3] McCullagh, P., and J. A. Nelder. Generalized Linear Models. New York: Chapman & Hall, 1990.