Prediction Error Variance Viewer :: Designs (Model-Based Calibration Toolbox™)

Learn more about Model-Based Calibration

Prediction Error Variance Viewer

On this page…
Introducing the Prediction Error Variance Viewer Display Options Prediction Error Variance Prediction Error Variance for Two-Stage Models

On this page…

Introducing the Prediction Error Variance Viewer

Prediction Error Variance for Two-Stage Models

Introducing the Prediction Error Variance Viewer

You can use the Prediction Error Variance (PEV) viewer to examine the quality of the model predictions. You can examine the properties of designs or global models. When you open it from the Design Editor, you can see how well the underlying model predicts over the design region. When you open it from a global model, you can view how well the current global model predicts. A low PEV (tending to zero) means that good predictions are obtained at that point.

The Prediction Error Variance Viewer is only available for linear models and radial basis functions.

When designs are rank deficient, the Prediction Error Variance Viewer appears but is empty; that is, the PEV values cannot be evaluated because there are not enough points to fit the model.

From the Design Editor, select Tools > Prediction Error Variance Viewer.
From the global level of the Model Browser, if the selected global model is linear or a radial basis function,
- Click the toolbar button to open the Prediction Error Variance Viewer.
- Alternatively, select Model > Utilities > Prediction Error Variance Viewer.
  If a model has child nodes you can only select the Prediction Error Variance Viewer from the child models.

The default view is a 3-D plot of the PEV surface.

The plot shows where the model predictions are best. The model predicts well where the PEV values are lowest.

If you have transformed the output data (eg using a Box-Cox transform), then the Prediction Error Variance Viewer displays the predicted variance of the transformed model.

Display Options

The View menu has many options to change the look of the plots.
You can change the factors displayed in the 2-D and 3-D plots. The drop-down menus below the plot select the factors, while the unselected factors are held constant. You can change the values of the unselected factors using the buttons or edit boxes in the frame, top left.
The Movie option shows a sequence of surface plots as a third input factor's value is changed. You can change the factors, replay, and change the frame rate.
You can change the number, position, and color of the contours on the contour plot with the Contours button. See the contour plot section (in Response Surface View) for a description of the controls.
You can select the Clip Plot check box, as shown in the preceding example. Areas that move above the PEV value in the Clipping envelope edit box are removed. You can enter the value for the clipping envelope. If you do not select Clip Plot, a white contour line is shown on the plot where the PEV values pass through the clipping value.
You can also clip with the boundary model or design constraints if available. Select the check boxes Apply constraint or Apply boundary model to clip the plot.

When you use the Prediction Error Variance Viewer to see design properties, optimality values for the design appear in the Optimality criteria frame.

Note that you can choose Prediction Error shading in the Response Feature view (in Model Selection or Model Evaluation). This shades the model surface according to Prediction Error values (sqrt(PEV)). This is not the same as the Prediction Error Variance Viewer, which shows the shape of a surface defined by the PEV values. See Response Surface View.

Optimality Criteria

No optimality values appear in the Optimality criteria frame until you click Calculate. Clicking Calculate opens the Optimality Calculations dialog box. Here iterations of the optimization process are displayed.

In the Optimality criteria frame in the Prediction Error Variance Viewer are listed the D, V, G and A optimality criteria values, and the values of the input factors at the point of maximum PEV (Location of G value). This is the point where the model has its maximum PEV value, which is the G-optimality criteria. The D, V and A values are functions of the entire design space and do not have a corresponding point.

For statistical information about how PEV is calculated, see the next section Prediction Error Variance.

Prediction Error Variance

Prediction Error Variance (PEV) is a very useful way to investigate the predictive capability of your model. It gives a measure of the precision of a model's predictions.

You can examine PEV for designs and for models. It is useful to remember that:

PEV (model) = PEV (design) * MSE

So the accuracy of your model's predictions is dependent on the design PEV and the mean square errors in the data. You should try to make PEV for your design as low as possible, as it is multiplied by the error on your model to give the overall PEV for your model. A low PEV (close to zero) means that good predictions are obtained at that point.

You can think of the design PEV as multiplying the errors in the data. If the design PEV < 1, then the errors are reduced by the model fitting process. If design PEV >1, then any errors in the data measurements are multiplied. Overall the predictive power of the model will be more accurate if PEV is closer to zero.

You start with the regression (or design) matrix, for example, for a quadratic in N (engine speed) and L (load or relative air charge):

If you knew the actual model, you would know the actual model coefficients . In this case the observations would be:

where is the measurement error with variance

However you can only ever know the predicted coefficients:

which have variance

Let x be the regression matrix for some new point where you want to evaluate the model, for example:

Then the model prediction for this point is:

Now you can calculate PEV as follows:

Note the only dependence on the observed values is in the variance (MSE) of the measurement error. You can look at the PEV(x) for a design (without MSE, as you don't yet have any observations) and see what effect it will have on the measurement error - if it is greater than 1 it will magnify the error, and the closer it is to 0 the more it will reduce the error.

You can examine PEV for designs or global models using the Prediction Error Variance viewer. When you open it from the Design Editor, you can see how well the underlying model predicts over the design region. When you open it from a global model, you can view how well the current global model predicts. A low PEV (tending to zero) means that good predictions are obtained at that point. See Prediction Error Variance Viewer.

For information on the calculation of PEV for two-stage models, see Prediction Error Variance for Two-Stage Models.

Prediction Error Variance for Two-Stage Models

It is very useful to evaluate a measure of the precision of the model's predictions. You can do this by looking at Prediction Error Variance (PEV). Prediction error variance will tend to grow rapidly in areas outside the original design space. The following section describes how PEV is calculated for two-stage models.

For linear global models applying the variance operator to Equation 6-15 yields:

(3-1)

since Var(P) = W. Assume that it is required to calculate both the response features and their associated prediction error variance for the i^thtest. the predicted response features are given by:

(3-2)

where is an appropriate global covariate matrix. Applying the variance operator to Equation 3-2 yields:

(3-3)

In general, the response features are non-linear functions of the local fit coefficients. Let denote the non-linear function mapping onto . Similarly let denote the inverse mapping.

(3-4)

Approximating using a first order Taylor series expanded about (the true and unknown fixed population value) and after applying the variance operator to the result:

(3-5)

where the dot notation denotes the Jacobian matrix with respect to the response features, . This implies that is of dimension (pxp). Finally the predicted response values are calculated from: