|
|
|
| R2011b Documentation → Model-Based Calibration Toolbox | |
Learn more about Model-Based Calibration Toolbox |
|
|
|
|
| R2011b Documentation → Model-Based Calibration Toolbox | |
Learn more about Model-Based Calibration Toolbox |
|
| Contents | Index |
| On this page… |
|---|
Viewing Local Model Statistics Using the RMSE Explorer with Local Models |
When you select a local node (with the
icon) in the model tree, the local level view appears.
At the local level you can:
View local model plots and statistics, and scroll through all local models test by test. See Using Local Model Plots, and Viewing Local Model Statistics.
Look for problem tests with the RMSE Explorer. See Using the RMSE Explorer with Local Models.
Remove and restore outliers and update fits. See Removing Outliers and Updating Fits.
Calculate two-stage models, and add or remove response features. After calculating the two-stage model, you can compare the local fit and the two-stage fit on the local level plots. See Calculating Two-Stage Models and Response Features.
Note that after the two-stage model is calculated the local
node icon changes to a two-stage icon (
) to reflect this. See the model tree for clarification. The response
node also has a two-stage icon, but produces the response level view instead.
The following example shows a local model of torque/spark curves. In this example the two-stage model has been calculated so you can compare the local fit and the two-stage fit on the plots.
The default view is the Model tab, with plots described below. You can click the Data tab to view plots of other variables. See Data Tab.
You can scroll through all the local models by using the up and down test buttons, type directly in the edit box or go directly to test numbers by clicking Select Test.
The lower plots are referred to as special plots as they can be different for different models.
The lower plot at the local level shows the local model fit to the data for the current test only, with the datum point if there is a datum model. If there are multiple inputs to the local model, a predicted/observed plot is displayed. In this case to examine the model surface in more detail you can use Model > Evaluate. See Model Evaluation Window.
To examine the local fit in more detail, double-click the arrows (indicated in the preceding figure) to hide the scatter plot and expand the lower plot. You can zoom in on parts of the plot by Shift-click-dragging or middle-click-dragging on the place of interest on the plot. Return to full size by double-clicking.
You can select the plot type from the drop-down menu at the top of the plot. Choices here can be:
Local Response
Normal Plot Normal plots are a useful graph for assessing whether data comes from a normal distribution. For more information, see Normal Probability Plots in the Statistics Toolbox documentation.
Validation Residuals If you are using validation data, the plot shows the local model validation residuals if there is validation data for the current test (the global variables must match). If there is a two-stage model, the two-stage validation residuals are also shown. Validation data must be attached at the Test Plan Level. See Using Validation Data.
Surface and Contour are only available for models with two inputs. You can view a surface or contour plot with data points.
Above are the right-click context menus for both plots. On both plots you can manipulate outliers with all the same commands available in the Outliers menu. See Outliers Menu (Local Level) for details.
The Print to Figure command opens a MATLAB figure plot showing the current plot. On the special plots you can switch the confidence intervals and legend on and off, and hide or show removed data. For both plots you can switch the display of Record Numbers on and off. This is similar to test number for global models but relates to individual records within tests.
The upper plots are referred to as scatter plots. They can show various scatter plots of statistics for assessing goodness-of-fit for the current local model shown. Upper scatter plots are replaced by an icon if you resize the Browser too small.
The statistics available for plotting are model dependent.
The preceding is an example drop-down menu on the scatter plot for changing x and y factors. In this case spark is the local input factor and torque is the response. The local inputs, the response, and the predicted response are always available in these menus. The observation number is also always available.
The other options are statistics that are model dependent, and can include residuals, weighted residuals, studentized residuals, and leverage. At local level these are internally studentized residuals.
When you click the Data tab at the local level, or select View > Data Plots, you can view plots of the data for the current test.
Use the right-click menu item Set Up Plot Variables to open the Plot Variables Setup dialog box.
The dialog box appears automatically if you open the tab using the View menu.
In the Plot Variables Setup dialog box, you can choose to view any of the data signals in the data set for the current test (including signals not being used in modeling).
Choose variables from the list on the left and use the buttons to move them into the Y Variable(s) list or the X variable edit box.
You can use the No X Data button to plot a variable against record number only.
Note Remember you can also display values of global variables in the Diagnostic Statistics pane if you want to see these values at the same time as the Model tab. |
You can use the Test Notes pane to record information on particular tests. Each test has its own notes pane. Data points with notes recorded against them are colored in the global model plots. You choose the color using the Set Color button in the Test Notes pane.
The Diagnostic Statistics pane drop-down menu is shown, where you can select the information to be displayed in the pane.
If there is not enough room there are scroll bars.
Local Parameters — Shows the values and standard errors of the parameters in the local model for the current test selected.
Local Correlations — Shows a table of the correlations between the parameters.
Response Features — Shows the values and standard errors of the response features defined for this local model, from the current test (often some or all of them are the same as the parameters; others are derived from the parameters).
Global Variables — Shows the values and standard errors of the global variables at the position of the current test.
Local Diagnostics — s_i (the standard error for the current (ith) test), number of observations, degrees of freedom on the error, R squared, Cond(J) and Cond(Sigma): the condition indices for the Jacobian matrix and the covariance matrix.
Note Check for high values of Cond(J) (e.g., > 108). High values of this condition indicator can be a sign of numerical instability. |
Validation RMSE for the current test appears here if there is validation data for the current test.
Global Covariance — For MLE models, shows a covariance matrix for the response features at the global level.
These are seen at the local node (when two-stage modeling) in the Pooled Statistics table, and at the response node in the list of local models. If you have a selection of local or two-stage models, use these statistics to help you choose which model is best.
Local RMSE | Root mean squared error between the local model and the data for all tests. The divisor used for RMSE is the number of observations minus the number of parameters. |
Two-Stage RMSE | Root mean squared error between the two-stage model and the data for all tests. You want this error to be small for a good model fit. |
PRESS RMSE | Root mean squared error of predicted errors, useful for indicating overfitting; see PRESS statistic. The divisor used for PRESS RMSE is the number of observations. Not displayed for MLE models because the simple univariate formula cannot be used. |
Two-Stage T^2 | T^2 is a normalized sum of squared errors for all the response features models. You can see the basic formula on the Likelihood view of the Model Selection window.
Where
A large T^2 value indicates that there is a problem with the response feature models. |
-log L | Log-likelihood function: the probability of a set of observations given the value of some parameters. You want the likelihood to be large, tending towards -infinity, so large negative is good. For n observations x1,x2,..xn,
with probability distribution
This is the basis of MLE.
which is the same as:
This assumes a normal distribution. You can view plots of -log L in the Model Selection window, see Likelihood View. |
Validation RMSE | Root mean squared error between the two-stage model and the validation data for all tests. |
To explain blockdiag as it appears under T^2 in the Pooled statistics
table:
, where Ci is
the local covariance for test i, is calculated
as shown below.
You can open the RMSE Explorer to view plots of the standard errors of all the tests, both overall and by response feature. This tool can help you quickly identify problem tests. You can navigate to a test of interest from the RMSE Explorer by double-clicking a point in the plot to select the test in the Model Browser local model view.
The plot displays one value of standard error per test, overall and for each response feature. As a best practice, first plot plain s_e against test number to get an idea of how the error is distributed and locate any tests with much higher errors. Right-click to toggle display of test numbers. Ideally, all the standard errors should be roughly the same value to satisfy the statistical assumptions for two-stage models. If these assumptions are not satisfied, error estimates for two-stage models may not be valid.
You can also use the X- and Y-axis factor drop-down lists to plot these standard errors against the global variables to examine the global distribution of error.
You can use the right-click context menus on all plots or the Outliers menu to remove and restore outliers.
All the commands except Remove All Data and Copy Outliers are also available in the right-click context menus on all plots.
Select Multiple Outliers — Use this item to draw a selection box around as many data points as required to select them all as outliers. This is useful for removing many data points at once.
Clear Outliers — Returns all data points to the unselected state (that is, no points outlined in red) as possible outliers.
Remove Outliers — Removes red-outlined data points from the fit and refits the current local fit only. Use the Update Fit toolbar button or Model > Update Fit to refit all the global models also. You can choose to update or defer when another node is selected. See Updating Fits.
Restore Removed Data — Opens the Restore Removed Data dialog box, where you can choose the points to restore from the list by record number, or restore all available points. You can also press Ctrl+Z. Select points in the left list and use the buttons to move points between the lists. When you click OK this refits the local model, including all data points previously removed as outliers. Use Update Fit once more to refit the global models, or you are prompted when a new node is selected. See Updating Fits.
Copy Outliers — Opens the Copy Outliers dialog box, where you can choose which model's outliers to copy. Select a model (of the same type — local or global) in the tree and click OK, and the current model (and other models affected) are refitted using the outlier selections for that model.
Selection Criteria — Opens the Outlier Selection Criteria dialog box where you can set the criteria for the automatic selection of outliers. This is disabled for MLE models.
Remove All Data — Leaves the current local model with no data, so entirely removes the current test. This test is removed from all the global models.
When you remove an outlier from your local model, it refits immediately. Other dependent fits also need updates. You can choose when to update the other fits. Removing an outlier can affect several other models. Removing an outlier from a best local model changes all the response features for that two-stage model. The global models all change; therefore the two-stage model must be recalculated. For this reason the local model node returns to the local (house) icon and the response node becomes blank again. If the two-stage model has a datum model defined, and other models within the test plan are using a datum link model, they are similarly affected.
To update fits, either:
In the local view, use the Update Fit toolbar button to update all the dependent fits.
When you select another model node, you are prompted to update or defer updates.
When leaving the local node, a dialog box asks if you want to update all dependent fits. Click Yes to update all global models, or No to delay lengthy updates of dependent fits. Delaying updates can be useful when you want to examine only a particular global model after removing an outlier at the local level. With the defer option, you can avoid waiting while updating all other dependent fits.
If you defer updating fits and you go to a response feature node, the toolbox refits only that node, so you can inspect that global model fit. Other response features do not update unless you click them. When you return to the local node again the Update Fit button is enabled. Until you update fits, a status message at the bottom of the browser tells you that you have deferred updates.
You can select outliers as those satisfying a condition on the value of some statistic (for example, residual > 3), or by selecting those points that fall in a region of the distribution of values of that statistic.
For example, assume that residuals are normally distributed and select those with p-value > 0.9. You can also select outliers using the values of model input factors.
The drop-down menu labeled Select using contains all the available criteria, shown in the following example.
The options available in this menu change depending on the type of model currently selected. The options are exactly the same as those found in the drop-down menus for the x- and y-axis factors of the scatter plots in the Model Browser (local level and global level views).
In the preceding example, the model selected is the knot response feature, so knot and Predicted knot appear in the criteria list, plus the global input factors; and it is a linear non-MLE model, so Cook's Distance and Leverage are also available.
The range of the selected criteria (for the current data) is indicated above the Value edit box, to give an indication of suitable values. You can type directly in the edit box. You can also use the up/down buttons on this box to change the value (incrementing by about 10% of the range).
Distribution. You can use the Distribution drop-down menu to remove a proportion of the tail ends of the normal or t distribution. For example, to select residuals found in the tails of the distribution making up 10% of the total area:
Select Normal in the Distribution drop-down menu.
Select the operator >.
Enter 10 as the
% value in the edit box.
Residuals found in the tails of the distribution that make up 10% of the total area are selected. If you had a vast data set, approximately 10% of the residuals would be selected as outliers.
As shown, residuals found beyond the value
of
in the distribution are selected as outliers.
is a measure
of significance; that is, the probability of finding residuals beyond
is less than
10%. Absolute value is used (the modulus) so outliers are selected
in both tails of the distribution.
The t distribution is used for limited degrees of freedom.
If you select None in the Distribution drop-down menu, you can choose whether or not to use the absolute value. That is, you are selecting outliers using the actual values rather than a distribution. Using absolute value allows you to select using magnitude only without taking sign into account (for example, both plus and minus ranges). You can select No here if you are only interested in one direction: positive or negative values, above or below the value entered. For example, selecting only values of speed below 2000 rpm.
The Select using custom MATLAB file check box enables the adjacent edit box. Here you can choose a function file that selects outliers. Type the name of the file and path into the edit box, or use the browse button.
In this file you define a MATLAB function of the form:
function outIndices = funcname (Model, Data, Names)
Model is the current MBC model.
Data is the data used in the scatter plots. For example, if there are currently 10 items in the drop-down menus on the scatter plot and 70 data points, the data make up a 70 x 10 array.
Names is a cell array containing the strings from the drop-down menus on the scatter plot. These label the columns in the data (for example, spark, residuals, leverage, and so on).
The output, outIndices, must be an array of logical indices, the same size as one column in the input Data, so that it contains one index for each data point. Those points where index = 1 in outIndices are highlighted as outliers; the remainder are not highlighted.
This toolbar appears when a local node is selected in the model tree.
The eight left icons remain constant throughout the levels. They are for project and node management, and the help button, and here the print icon is enabled as there are plots in this view. See Project Level: Toolbar for details on these buttons. In the example shown the slider bar has been dragged to hide the Help button.
View Model — Opens a dialog box displaying the terms in the current model. See Model Definition Dialog Box.
Update Fit — This button is only enabled when data has been excluded from the plot using the Remove Outliers command (in the right-click context menu or the Outliers menu). At this point the local fit in the view is updated to fit only the remaining data, but this change also affects a point in all the global models. You can make this update to all the global models by using this toolbar button, or you are prompted when another node is selected.
See Updating Fits.
Calculate MLE — Calculates the two-stage model using maximum likelihood estimation. This takes correlations between response features into account. See MLE for details.
RMSE Plots — Opens the RMSE Explorer dialog box, where you can view plots of the standard errors of all the tests. See Using the RMSE Explorer with Local Models.
View Local Fit Data — Opens the Data Editor, where you can view a read-only version of the inputs, predicted and actual responses. This allows you to examine and export your modeling data, using all the powerful display features of the Data Editor. Also in the View menu.
File Menu. Only the New (child node) and Delete (current node) functions change according to the node level currently selected. Otherwise the File menu remains constant.
See File Menu.
Window and Help Menus. The Window and Help menus remain throughout the Model Browser, offering access to different windows, general help and context help.
See Window Menu and Help Menu.
Set Up — Opens the Local Model Setup dialog box where you can change the model type. See Local Model Setup.
Fit Local — Opens the Local Model Fit Tool dialog box. Without covariance modeling, you see the following controls. This example shows the results after clicking Fit once. The optimization process can be stopped early by clicking Stop or you can wait until it finishes. The Ordinary Least Squares (OLS) parameters are displayed. You can click Fit to run the process again as many times as required, or Close to exit the dialog box. You can enter a different change in parameters in the edit box.
For covariance models this offers three different algorithms: REML (Restricted Maximum Likelihood - the default), Pseudo-likelihood, and Absolute residuals. The following example shows that there is also an additional button, One Step. Using the Fit button might take several steps to converge, but if you use the One Step button only one step in the optimization process is taken.
Every time you run a process, the initial and final Generalized Least Squares parameter values are displayed for each iteration.
Update Fit — Only enabled when outliers have been removed. The Update Fit option updates the global models to take these changes into account. You can also choose to update or defer updates when you select a different node. Duplicated in the toolbar. Update Fit can affect many models; see the explanation in Updating Fits.
Calculate MLE — Calculates the two-stage model using maximum likelihood estimation. This calculation takes interactions between response features into account. Duplicated in the toolbar. See MLE for details.
Evaluate — Select from the submenu Fit Data (also Ctrl+E), Validation Data, No Data, or Other Data. Opens the Model Evaluation window displaying the fit relative to the selected data (or no data). You must attach validation data to the test plan before you can use it. If you select Other Data a wizard appears to select the data. See Using Validation Data.
Utilities — Only enabled for local multiple models. See Analyzing Point-by-Point Models.
Build Models — Opens the Build Models dialog box, where you can choose a selection of model types to build for each response feature. See Build Models Dialog Box.
Select — Available whenever the Select button is also enabled in the lower right pane (when it is titled Local Models, Response Features, or Models). This item opens the Model Selection window to allow you to choose the best model. See Select Button.
Assign Best — Selects the current model as best. If it is one of several child node models of a response model, selecting it as best means that this local model (and associated response features) is used for the two-stage model. Note that this option is only enabled if the local node selected has a two-stage model calculated; that is, if the local node still has a local icon (a house) you cannot use Assign Best. See Calculating Two-Stage Models and Response Features.
See also
Model Definition — Opens the Model Definition dialog box, showing the terms in the model. See Model Definition Dialog Box.
Data Plots — Opens the Data tab on the display and the Plot Variables Setup dialog box, where you can choose to view any of the data signals in the data set for the current test (including signals not being used in modeling). Choose variables from the list on the left and use the buttons to move them into the Y Variable(s) list or the X variable edit box. You can use the No X Data button to plot a variable against record number only.
RMSE Plots — Opens the RMSE Explorer where you can view plots of the standard errors of all the tests. See Using the RMSE Explorer with Local Models.
Local Fit Data — Opens the Data Editor, where you can view a read-only version of the inputs and predicted values. This allows you to examine your modeling data and inputs using all the powerful display features of the Data Editor. You can also export your modeling data to the workspace or Excel. Also in the toolbar.
Response Feature Data — Opens the Data Editor, where you can view a read-only version of the values of all the response features for the local models. You can also use the Data Editor export facilities.
Record numbers — Toggles the display of record numbers on plots.
Next Test, Previous Test, and Select Test — Duplicate the buttons for changing tests above the plots in the top left of the Local Model display pane.
The Response Features List at the bottom of the local view shows a list of all the response features calculated for the local model.
To calculate a two-stage model, click the Select button in the Response Features List pane. The Model Selection window opens. This step is required before two-stage models can be calculated. A two-stage model using the local and global models is formed by using Select. After calculating the two-stage model, you can compare the local fit and the two-stage fit on the local level plots.
The list view displays the number of parameters and observations, the value of any Box-Cox transformation (1 indicates no transform), and the values of RMSE and PRESS RMSE (linear models only) for each response feature model. For definitions of RMSE and PRESS RMSE, see Summary Table. For information on Box-Cox transforms, see Box-Cox Transformation.
Click New to add a new response feature model (or Delete to remove one). For more information see the Test Plans List Pane. The contents of this pane change in different views; it always contains the child nodes of the node selected in the Model Tree (and the New, Delete, and Select buttons). At the local level it contains a list of response features.
 | Selecting Models | Global Models |  |
Includes the most popular MATLAB recorded presentations with Q&A sessions led by MATLAB experts.
| © 1984-2012- The MathWorks, Inc. - Site Help - Patents - Trademarks - Privacy Policy - Preventing Piracy - RSS |
