Selecting Global and Two-Stage Models

On this page…
Inspect the Global Models Create Multiple Models to Compare Create a Two-Stage Model Adding New Response Models

Inspect the Global Models

When you are satisfied with the local fits, inspect the global models in turn. You should check trends of global models. Do the trends go in the right direction? Previous engineering knowledge can be applied. The following steps suggest useful plots for investigating trends.

1. Expand the local model node (PS22) in the model tree and click knot.

   

   

2. Right-click outliers (or any point) to see a plot of the test. You can inspect the shape of the torque/spark curve and see the values of the global variables. This can help you identify problem tests, perhaps on the edge of the stable operating region.

   Also you can use the global model view scatter plots to plot predicted values against variables.

   Note   After identifying problem tests at the global level, return to the local level to decide whether to remove outliers or the whole test.

3. Select Model > Evaluate > Fit Data to open the Model Evaluation window.

   • Select View > Cross Section to see the trends of your current model e.g., maximum torque (max model), or MBT (knot model). Try the Response Surface view. Apply engineering knowledge to check trends.

   • Be aware that Bhigh2 and Blow2 must be negative (to ensure maximum torque occurs at MBT). To check for this, select View > Response Surface and click the Table Display type — here cells outside the boundary model are highlighted yellow to help you focus on the area of interest.

4. Also you can use the global model view scatter plots to plot Predicted Bhigh2 or Predicted Blow2 against Speed to look for suspicious positive values.

Back to Top

Create Multiple Models to Compare

1. Click the Build Models toolbar button.

2. Click Browse, locate the mbctraining directory and click OK.

3. Select the DIVCP template and click OK.

   Choose PRESS RMSE in the Model Selection dialog box and click OK, and a selection of child model types are built for knot. PRESS RMSE is the criterion used for automatically selecting the best model out of the child nodes.

   

   You can save your own template of any collection of child models. From the parent model node (e.g., knot) select Model > Make Template.

4. Now knot has several child models. Inspect each model in turn.If you remove outliers that have an RBF center (marked with a star) be sure to refit the model (click the toolbar button Update Model Fit) to reselect widths and centers.

5. Return to the knot model node and look at the statistics reported in the list of child models at the bottom. From any parent model node you can see a list of statistical comparisons for all the child nodes in the lower list pane, along with information such as the number of parameters. Use this information to help you decide which model is best.

   

   • Look for lower RMSE values to indicate better fits.

   • Look for lower PRESS RMSE values to indicate better fits without overfitting.

     PRESS RMSE is a measure of the predictive power of your models. It is useful to compare PRESS RMSE with RMSE as this may indicate problems with overfitting. RMSE is minimized when the model gets close to each data point; 'chasing' the data will therefore improve RMSE. However chasing the data can sometimes lead to strong oscillations in the model between the data points; this behavior can give good values of RMSE but is not representative of the data and will not give reliable prediction values where you do not already have data. The PRESS RMSE statistic guards against this by testing how well the current model would predict each of the points in the data set (in turn) if they were not included in the regression. To get a small PRESS RMSE usually indicates that the model is not overly sensitive to any single data point.

     If the value of PRESS RMSE is much bigger than the RMSE then you are overfitting — the model is unnecessarily complex.

     PRESS RMSE can be the most helpful single statistic you can use to search for the best fit relative to the number of terms in the model. However you should not rely on any single statistic, but use a variety of criteria and especially the graphical tools available for comparison of models in the Model Evaluation tool when you click Select. You can also use other diagnostic statistics to help you select models. For detailed guidance on how to understand the selection tools see Creating Multiple Models to Compare and the Model Selection Guide in the Model Browser documentation. (In the Help Browser you can right-click and select Back to return to previous pages.)

   • You can add other statistics here. Select Model > Summary Statistics. In the dialog, select the check box for AICc and click OK. A column of AICc values appears in the model list. This can be a useful statistic for comparison of models as it penalizes over-parameterized models. Over-fitting models can later cause problems with optimization when you use the models to create calibrations. See Using Information Criteria to Compare Models.

6. From the knot model node, click Select in the Models pane at the bottom to open the Model Selection window. Try the different views and compare the child models by selecting them in the Model List at the bottom.

   

7. Select one of the child models as best (click the button Assign Best) and close the Model Selection window to return to the Model Browser. Try other model types if you are not satisfied with the quality of the fit. You could work through the modeling tutorial for more guided examples of how to select models, see Tutorial: Model Quickstart .

8. When you have selected a best child model, select File > Clean-up Tree to discard all but the model you chose as best. The process of searching for good model fits often results in a large selection of models, and cleaning up the tree reduces file size. It can also save time when saving the file, especially if you have made a change such as removing an outlier at local level, that causes all response feature models to be refitted.

9. Select another global model such as Bhigh_2 and click Build Models in the toolbar. Select the DIVCP template to automatically build the same selection of child model types for Bhigh_2. This can help you quickly build multiple models to compare. The Model Selection dialog appears, where you can choose a criterion such as PRESS RMSE or AICc for automatically selecting the best model out of the child nodes.

10. Repeat this process of searching for good global model fits for the other three global models.

Back to Top

Create a Two-Stage Model

1. When you have selected best models for each global model, return to the local model node (PS22) in the model tree, and click Select in the Response Features model list pane at the bottom to calculate the two-stage model and open the Model Selection window.

   

   Look through the plots of the new two-stage model against the local fits and the data.

   When you close the Model Selection window and accept the new model as best, the two-stage model is copied to the BTQ response node in the model tree.

2. You can choose to calculate maximum likelihood estimation (MLE) at this point. This process refits, taking proper account of the correlation between different response features. Try calculating MLE, then select the BTQ node and return to the Model Selection window to compare the MLE model with the univariate model (click Select All in the Model List pane).

For more guidance on creating multiple local, global and two-stage models to search for the best fit, work through the step-by-step examples in Tutorial: Model Quickstart , especially the section Creating Multiple Models to Compare.

Back to Top

Adding New Response Models

To complete the model building you should follow the same process as for modeling torque to create and select good models of the other responses: exhaust temperature and residual fraction (instructions below). These models are required for the optimization problems. Remember you can look at the example finished project, Gasoline_project.mat, in the mbctraining directory, to see how the example models have been constructed. You will use these example models in CAGE for the optimization section.

Exhaust Temperature Model

1. Select the test plan node in the model tree to return to the test plan view.

2. Double-click the Responses icon in the block diagram.

3. The Response Model Setup dialog appears.

   1. Select EXTEMP from the list of signals.

   2. Click the Local Model Set Up button. Select Polynomial from the Model class drop-down menu and click OK.

   3. Select BTQ datum from the Datum drop-down menu. It can be useful to plot the position of MBT on other models.

      

   4. Click OK.

A new set of local and global models is calculated for the exhaust temperature response.

The view switches to the new local node (POLY2).

1. You can copy the outliers you selected for the torque models. Select Outliers > Copy Outliers From. The Copy Outliers dialog box appears. Select the BTQ local model node in the tree (PS22, under BTQ), and click OK to copy the outlier selections to the EXTEMP local node.

2. Examine the fit in the same way as you did the torque fits.

   • Try different local models. Click New at the EXTEMP response node.

   • Try different global models as you did for the torque response features. Use Build Models and the DIVCP model template, or click New to add child nodes and try different model types one at a time. If you use Build Models at the local level you can apply a template to all response feature models at once.

   • Try a new response feature. Click New at one of the local nodes you have created and enter -10 for the value to use MBT minus 10 degrees of spark as a new response feature.

     See Tutorial: Model Quickstart for simple worked examples of adding new local, global, response feature and two-stage models.

3. When you are satisfied with the fits, return to the local model node and click Select to calculate the two-stage model. If you have added new response features, there will be more than one two-stage model to choose from in the Model Selection window.

4. Try calculating MLE and return to the Model Selection window to compare the MLE model with the univariate model and select the best.

Residual Fraction Model

1. Repeat the steps to create a model of residual fraction (RESIDFRAC).

   Make sure you use the BTQ datum datum link model so you can view MBT. As a result of this, one of the response features, FX_0, models residual fraction at MBT. You will use this in your optimizations in CAGE.

2. The process of searching for good model fits often results in a large selection of models. Remember to discard all but the models you chose as best, by selecting File > Clean-up Tree.

Look at the example finished project, Gasoline_project.mat, in the mbctraining directory, to see how the example models have been constructed. You will use these example models in CAGE for the optimization section.

Back to Top

Creating Boundary Models

Using Validation Data

© 1984-2012- The MathWorks, Inc.    -   Site Help   -   Patents   -   Trademarks   -   Privacy Policy   -   Preventing Piracy   -   RSS