|
|
|
| R2012a Documentation → Model-Based Calibration Toolbox | |
Learn more about Model-Based Calibration Toolbox |
|
|
|
|
| R2012a Documentation → Model-Based Calibration Toolbox | |
Learn more about Model-Based Calibration Toolbox |
|
| Contents | Index |
| On this page… |
|---|
Once you have fitted and examined a single model, you will normally want to create more models to search for the best fit. You can create individual new models, use the Build Models function to create a selection of models at once, or create a template to save a variety of model settings for reuse.
You can create new child nodes by clicking the New button from any modeling node. The Model Setup dialog appears where you can change the type and settings, and when you close the dialog the view switches to the new child node on the tree. You can do this for multiple child nodes to create a selection of different model types fitted to the same data. You can also use the Build Models dialog to quickly create a selection of different child nodes to compare. The following exercises show you examples of these processes. Note that you need to complete the previous tutorial sections to have a complete two-stage model as a starting point.
As an example, select the tq response node and click New in the Local Models list pane.
The Local Model Setup dialog appears.
Select a Polynomial Spline with a spline order of 3 below the knot and 2 above. Click OK.
A new set of local models (and associated response feature models) is calculated.
Return to the parent tq response node , and click New again, in the Local Models list pane.
Select a Polynomial with an order of 2 in the Local Model Setup dialog. Click OK.
A new set of local models and response feature models is calculated.
Now you have three alternative local models to compare: two polynomial splines (order 3,2 and order 2,2) and a polynomial (order 2), as shown.
You can select the alternative local models in turn and compare their statistics. For an example, follow these steps:
In the Diagnostic statistics pane, select Local diagnostics from the drop-down menu. Observe the value of s_i in this pane. This is the value of RMSE (root mean squared error) for the current (ith) test.
The RMSE value is our basic measure of how closely a model fits some data, which measures the average mismatch between each data point and the model. This is why you should look at the RMSE values as your first tool to inspect the quality of the fit — high RMSE values can indicate problems.
Now select the local model node POLY2 and see how the value of s_i changes.
Observe that the shape of the torque/spark sweep for this test is better suited to a polynomial spline model than a polynomial model. The curve is not symmetrical because curvature differs above and below the maximum (marked by the red cross at the datum). This explains why the value of s_i is much lower for PS32 (the polynomial spline) than for the POLY2 (polynomial) for this test. The polynomial spline is a better fit for the current test.
Look through some other tests and compare the values of s_i for the different local models. To choose the most suitable local model you must decide which fits the majority of tests better, as there are likely to be differences among best fit for different tests.
To help you quickly identify which local
models have the highest RMSE, indicating problems with the model fit,
click RMSE Plots (
) in
the toolbar (or select View > RMSE Plots) to open the RMSE
Explorer dialog.
Use the plot to help you identify problem tests. Use the drop-down menus to change the display. For example, select s_knot to investigate the error values for knot (MBT), or s_e to look at overall error.
You can navigate to a test of interest from the RMSE Explorer by double-clicking a point in the plot to select the test in the Model Browser local model view.
Look at the value of Local RMSE reported in the Pooled Statistics pane on the right (this is pooled between all tests). Now switch between the POLY2 and the PS32 local models again and observe how this value changes.
You can compare these values directly by selecting the parent tq response node, when the Local RMSE is reported for each child local model in the list at the bottom.
When all child models have a two-stage model calculated, you can also compare two-stage values of RMSE here. Remember, you can always see statistics for the list of child models of the currently selected node in this bottom list pane.
When comparing models, look for lower RMSE values to indicate better fits. However, remember that a model that interpolates between all the points can have an RMSE of zero but be useless for predicting between points. Always use the graphical displays to visually examine model fits and beware of "overfitting" — chasing points at the expense of prediction quality. You will return to the problem of overfitting in a later section when you have two-stage models to compare.
Recall that two-stage models are made up of local models and global models. The global models are fitted to the response features of the local models. The response features available are specific to the type of local model. You can add different response features to see which combination of response features makes the best two-stage model as follows:
Click the New button under the list of response features.
A dialog appears with a list of available response features.
Select f(x+datum) from the list and enter -10 in the Value edit box. Click OK.
A new response feature called FX_less10 is added under the PS32 local model. Recall that the datum marks the maximum, in this case maximum torque. The spark angle at maximum torque is referred to as maximum brake torque (MBT). You have defined this response feature (f(x+datum)) to measure the value of the model (torque) at (-10 + MBT) for each test. It can be useful to use a response feature like this to track a value such as maximum brake torque (MBT) minus 10 degrees of spark angle. This response feature is not an abstract property of a curve, so engineering knowledge can then be applied to increase confidence in the models.
Select the local node PS32, and click Select. Notice that there are four possible two-stage models this time. This is because you added a sixth response feature. Only five (which must include knot) are required for the two-stage model, so you can see the combinations available and compare them. Note that not all combinations of five response features can completely describe the shape of the curve for the two-stage model, so only the possible alternatives are shown.
Close the Model Selection window and click OK to accept one of the models as best. Click Cancel to avoid calculating MLE.
Notice that the four response features chosen to calculate the two-stage model are highlighted in blue, and the unused response feature is not highlighted, as shown.
Select the tq response node to see a comparison of the statistics of both two-stage models (your original PS22 and the new PS32).
Remember that the POLY2 local model has no two-stage model yet; no two-stage statistics are reported for POLY2 in the bottom list pane. You also cannot use the Model Selection window to fully compare the two-stage models until every local model in the test plan has a two-stage model calculated.
To calculate the two-stage model for POLY2, click Select at the POLY2 node. Either double-click to assign a model as best or close the Model Selection window and click OK to accept the best model. Click Cancel to avoid calculating MLE, then the two-stage model is calculated.
Now you have three two-stage models. Select the tq response node and look at the statistics, particularly Local RMSE, Two-Stage RMSE, and PRESS RMSE, reported in the list of child models at the bottom.
Look for lower RMSE values to indicate better fits.
Look for lower PRESS RMSE values to indicate better fits without overfitting. PRESS RMSE is a measure of the predictive power of your models.
It is useful to compare PRESS RMSE with RMSE as this may indicate problems with overfitting. RMSE is minimized when the model gets close to each data point; "chasing" the data will therefore improve RMSE. However, chasing the data can sometimes lead to strong oscillations in the model between the data points; this behavior can give good values of RMSE but is not representative of the data and will not give reliable prediction values where you do not already have data. The PRESS RMSE statistic guards against this by testing how well the current model would predict each of the points in the data set (in turn) if they were not included in the regression. To get a small PRESS RMSE usually indicates that the model is not overly sensitive to any single data point.
If the value of PRESS RMSE is much bigger than the RMSE, then you are overfitting - the model is unnecessarily complex. For a fuller description of the meaning of overfitting, and how RMSE and PRESS can help you select good models, see Model Selection Guide. As a rule of thumb, if you have about 100 data points, you should aim for a PRESS RMSE no more than 5% larger than the RMSE (remember here you have only 27 tests).
Notice that your first two-stage model (PS22) does not have a PRESS RMSE value. This is because it cannot be displayed for MLE models. You need non-MLE models to be able to use PRESS for direct comparison.
Look for lower T^2 values. A large T^2 value indicates that there is a problem with the response feature models.
Look for large negative log likelihood values to indicate better fits.
See Pooled Statistics for more on T^2 and log likelihood.
Now click Select to open Model Selection to compare all three two-stage models simultaneously. Here you can see the same statistics to compare the models in the bottom list, but you can also make use of a variety of views to look for the best fit:
You can plot the models simultaneously on the Tests, Residuals and Cross Section views (Shift- or Ctrl-click to select models in the list)
You can view each model in the Response Surface view as a surface; movie, contour or multiline plot, and as a table
You can select a model and click Assign Best in the Model Selection window, or double-click a model to assign it as best.
When you close the Model Selection window and return to the Model Browser, the model you selected as best is copied to the parent response node, tq.
In this example, you have not yet searched for the best global model types. You would normally do this before creating and comparing two-stage models. For the purpose of this tutorial, you have already created two-stage models and used the Model Selection tool to introduce the use of RMSE and PRESS to help you identify better models. The principle is the same at each level in the model tree: add new child models and use the Model Selection window to choose the best.
Select one of the response feature nodes under the PS32 node, such as knot.
Click New. Click OK in the Model Setup dialog without changing any settings, to create a copy of the parent model. Return to the parent model and repeat.
Two new global model child nodes now appear underneath knot, as shown. Both are labeled Quadratic, as they are currently copies of the parent model. You can create any number of child nodes to search for the best global model fit for each response feature in your tree. When you choose the best, it is copied to the parent node, in this case knot, including any outliers you decide to exclude.
A good technique for creating multiple models can be to leave the first child node unchanged, then you always have a copy of the original model for comparison.
Select one of the new Quadratic nodes, then select the menu item Model > Set Up.
The Global Model Setup dialog appears. Here you can change the type and settings of the model to see if you can find a better fit to the data with a different model type.
Use the drop-down menu to change the Model class to Hybrid RBF and click OK.
The new model fit is calculated, and the Quadratic node's name changes to Linear-RBF.
Select the remaining Quadratic node, then select Model > Set Up.
Use the drop-down menu to change the Model class to Radial Basis Function, and click OK. There are many other settings you can alter for both these model types, but for a quick exploration of the trends in the data it is worth trying the default model settings.
The new model fit is calculated and the Quadratic node's name changes to RBF-multiquadric.
To compare the two child node models, select the parent node knot and click Select. Whichever model you assign as best is copied to the knot node when you close the Model Selection window and click OK.
Notice that the child node model assigned as best is highlighted in blue, and the local node has changed from the two-stage icon back to the local model icon (a red house) as shown. This is because you have changed one of the response feature models, and so you need to recalculate the two-stage model using the new global model for this response feature. First you need to select best global models for every response feature.
Add two more child nodes to the knot global model (select knot, then click New, and repeat).
Notice that now the new nodes are copies of Linear-RBF, because that model was selected as best.
Select the two new nodes in turn and change their model types. Try a cubic and quadratic polynomial model type.
Select the menu item Model > Set Up.
Choose Linear model from the Model class drop-down menu and set the polynomial order for each factor to 3 for one model, then 2 for the other. Click OK.
To compare all four child node models, select the parent node knot and click Select. Linear-RBF still performs the best for PRESS RMSE. Whichever model you assign as best is copied to the knot node.
Select the knot model node, then select Model > Make Template. Browse to a suitable work directory and enter the name Mytemplate. Click OK.
The quickest way to create multiple different models to compare is to use the Build Models function. You can use this to select a template and build a selection of models as child nodes of the current node. The best model of this selection of child nodes is automatically selected (it will have a blue icon), based on the selection criteria you choose (such as PRESS RMSE, RMSE, Box-Cox, and so on).
Before calculating MLE, select a global model such as max.
You cannot reach the Build Models dialog from an MLE global model. Note that calculating MLE is not irreversible — to go back you can always go to Model Selection (from the local node) and select the Univariate model as best.
Click Build Models in the toolbar.
The Build Models dialog appears. Here you can choose a template for the type of models you want to build. There are predefined templates for polynomials, RBFs, hybrid RBFs, free knot splines (for single input models), or you can select a suitable parent node in the current project to use as a template.
You can also create templates of whatever models you choose by selecting the New template in the Build Models dialog box, or using the Model > Make Template menu item at a model node, as you did in the previous section. Your user-defined templates can then be found via the Build Models dialog. You can use the Browse button to find stored templates that are not in the default directory.
Click Browse and select the directory containing the template you created earlier, named Mytemplate. Click OK.
Your new template (called Mytemplate) now appears in the Build Models dialog along with the defaults. Note that you can set the default directory where the toolbox looks for templates (and models, data, and projects) using File > Preferences.
Select Mytemplate. Notice the four model types you saved in this template appear in the Information pane. Click OK.
The Model Selection dialog appears, where you can choose a criterion for automatically selecting the best model out of the child nodes.
Use the drop-down menu to choose PRESS RMSE as the selection criteria for the best model, and click OK.
Four child nodes appear: Linear-RBF, RBF-multiquadric, Cubic, and Quadratic. These are the model types you selected when you built the template and are now fitted to the data for max.
The most favorable child node model, based on PRESS RMSE, is selected as best (highlighted in blue) as shown in the following figure. This model is also copied to the parent node max, in the same way as if you had used the Model Selection window to assign a best model.
Try one of the default templates in the Build Models dialog box as follows:
Select RBF and click OK. Click Build in the following Model Building Options dialog to build a selection of child nodes.
Similarly, you can use the Build Models dialog to automatically build a selection of polynomial or hybrid RBF models, or your own selection of model types, to search for the best fit.
Click OK in the Model Selection dialog to accept the default, PRESS RMSE, as the selection criteria for the best model.
Look at the statistics in the lower list pane to quickly compare all the different RBF kernel child models. If one model performs significantly better in terms of PRESS RMSE and RMSE you might choose not to click Select to compare all the child node models. However, it is usually useful to visually inspect the models to see how they compare.
When you have chosen a best model, it can be useful to select some (or all) of the rejected models in the bottom list pane and press Delete. You can also select File > Clean Up Tree. This deletes all rejected child models where best models have been chosen; only the child nodes selected as best remain.
Creating a template containing a list of all the models you want is a very efficient way to quickly build a selection of alternative model child nodes for many global models. Use these techniques to find models well suited to the data for each of your global models.
When you have chosen best global models for all your response features, you need to recalculate the two-stage model. Click Select at the local model (PS32) node to calculate the two-stage model.
 | Exporting the Model | Tutorial: Design of Experiment |  |
Includes the most popular MATLAB recorded presentations with Q&A sessions led by MATLAB experts.
| © 1984-2012- The MathWorks, Inc. - Site Help - Patents - Trademarks - Privacy Policy - Preventing Piracy - RSS |
