Learn more about Simulink Design Optimization

Preprocessing Data in the GUI

On this page…
Ways to Preprocess Data Using the Data Preprocessing Tool Opening the Data Preprocessing Tool Handling Missing Data Handling Outliers Detrending Data Filtering Data Selecting Data Adding Preprocessed Data Sets to an Estimation Project Exporting Prepared Data to the MATLAB Workspace

Ways to Preprocess Data Using the Data Preprocessing Tool

After you import the estimation data, as described in Importing Data into the GUI, you can perform the following preprocessing operations using the Data Preprocessing Tool in Simulink Design Optimization software:

• Exclusion — Exclude a portion of the data from the estimation process. You can exclude data by:

  • Selecting it with your mouse.

  • Graphically by selecting regions on a plot.

  • Using rules, such as upper or lower bounds.

• Handle missing data –– Remove missing data, or compute missing data using interpolation.

• Handle outliers –– Remove outliers.

• Detrend — Remove mean values or a straight line trend.

• Filter — Smooth data using a first-order filter, an arbitrary transfer function, or an ideal filter.

Back to Top

Opening the Data Preprocessing Tool

To open the Data Preprocessing Tool:

1. In the Control and Estimation Tools Manager GUI, select the Transient Data node under the Estimation Task node, and then choose the data you want to preprocess either in the Input Data, or Output Data tab. This enables the Pre-process button.

   

2. Click Pre-process to open the Data Preprocessing Tool.

   

Tip   When you have multiple data sets, select the data set that you want to preprocess from the Modify data from drop-down list in the Data Preprocessing Tool.

In this section, the sample data set imported for preprocessing is the same as used in the engine_idle_speed Simulink model. For an overview of creating estimation projects and importing data sets, see Configuring a Model for Importing Data, and Creating an Estimation Project.

Back to Top

Handling Missing Data

• Removing Missing Data

• Interpolating Missing Data

Removing Missing Data

Rows of missing or excluded data are represented by NaNs. To remove the rows containing missing or excluded data, select the Remove rows where check box in the Missing Data Handling area of the Data Preprocessing Tool GUI.

When the data set contains multiple columns of data, select all to remove rows in which all the data is excluded. Select any to remove any excluded cell. In the case of one-column data, any and all are equivalent.

Tip   You can view the modified data in the Modified data tab of the Data Preprocessing Tool GUI.

Interpolating Missing Data

The interpolation operation computes the missing data values using known data values. When you select the Interpolate missing values using interpolation method check box in the Missing Data Handling area of the Data Preprocessing Tool GUI, the software interpolates the missing data values.

You can compute the missing data values using one of the following interpolation methods:

• Zero-order hold (zoh) — Fills the missing data sample with the data value immediately preceding it.

• Linear interpolation (Linear) — Fills the missing data sample with the average of the data values immediately preceding and following it.

By default, the interpolation method is set to zoh. You can select the Linear interpolation method from the Interpolate missing values using interpolation method drop-down list.

Tip   You can view the results of interpolation in the Modified data tab of the Data Preprocessing Tool GUI.

Back to Top

Handling Outliers

Outliers are data values that deviate from the mean by more than three standard deviations. When estimating parameters from data containing outliers, the results may not be accurate.

To remove outliers, select the Outliers check box to activate outlier exclusion. You can set the Window length to any positive integer, and use confidence limits from 0 to 100%. The window length specifies the number of data points used when calculating outliers.

Removing outliers replaces the data samples containing outliers with NaNs, which you can interpolate in a subsequent operation. To learn more, see Interpolating Missing Data.

Back to Top

Detrending Data

To detrend, select the Detrending check box. You can choose constant or straight line detrending. Constant detrending removes the mean of the data to create zero-mean data. Straight line detrending finds linear trends (in the least-squares sense) and then removes them.

Back to Top

Filtering Data

• Types of Filters

• How to Filter Data

Types of Filters

You have these choices for filtering your data:

• First order — A filter of the type

  where is the time constant that you specify in the associated field.

• Transfer function — A filter of the type

  

  where you specify the coefficients as vectors in the associated A coefficients and B coefficients fields.

• Ideal — An idealized (noncausal) filter, either stop or pass band. Specify either filter as a two-element vector in the Range (Hz) field. These filters are ideal in the sense that there is no finite rolloff or ripple; the ends of the ranges are perfectly horizontal in the frequency domain.

How to Filter Data

To filter the data to remove noise, select the Detrend/Filtering tab in the Data Preprocessing Tool GUI. Select the Filtering check box, and choose the type of filter from the Select filter type drop-down list.

Back to Top

Selecting Data

• Techniques for Excluding Data in the Data Preprocessing Tool

• Graphically Selecting Data

• Using Rules to Select Data Samples

• Using the Data Table to Select Data Samples

Techniques for Excluding Data in the Data Preprocessing Tool

You can use the Data Preprocessing Tool to select a portion of the data to be excluded from the estimation process. You can choose one of the following techniques:

• Selecting data from the Data Editing Table.

• Selecting data from a plot of the data.

• Specifying a rule.

You accomplish the first two manually, and for the last you specify a rule. When you exclude data using manual selection, the excluded data is shown as red. When you exclude data using a rule, the background color of the cell becomes gray. When a portion of the data is excluded both manually and by a rule, the data is red, and the background is gray.

Note   Changes in data are visible everywhere. When you use the Data Editing table, you can view the results in the data plot.

Graphically Selecting Data

You can exclude data graphically. Click Exclude Graphically to open the Select Points for Preprocessing Rule window.

The way you exclude data is similar to the way you select a region for zooming: place your cursor in the Input Data plot and drag the mouse to draw a region of exclusion.

This figure shows an example of resulting data exclusion in the input data.

In the Output Data plot, the excluded input data produces a blank area by default. This corresponds to the NaNs that now represent excluded data. If you choose to interpolate or remove the excluded data, the output data shows the interpolated points.

When you make changes in the Select Points for Preprocessing Rule window, they immediately appear in the Data Editing pane, and vice versa.

Selection Pane.   By default, any box that you draw with your mouse selects data for exclusion, but you can toggle between exclusion and inclusion using the Selection pane on the left side of the Select Points for Preprocessing Rule window.

Using Rules to Select Data Samples

A more precise way to exclude data is to use mathematical rules. The Exclusion Rules pane in the Data Preprocessing Tool allows you to enter customized rules for excluding data.

These are the rules you can use to exclude data:

• Upper and Lower Bounds

• MATLAB Expressions

• Flatlines

Upper and Lower Bounds.   Select the Bounds check box to activate upper and lower bound exclusion. Enter numbers in the Exclude X and Exclude Y fields for upper and lower bound exclusion. By default, the exclusion rule is to include the boundary values, but you can use the menu to exclude the boundaries as well.

MATLAB Expressions.   Use the MATLAB expression field to enter any mathematical expression using MATLAB code. Use x as the variable name in your expression for the data being tested.

Flatlines.   If you have areas of your data set where the data is constant, providing no new information, then you can choose to exclude those data points as flatlines. The Window length field sets the minimum number of constant data points required to define the area as a flatline.

Example of Rule Exclusion.   This figure shows data with a region of the x-axis excluded.

Using the Data Table to Select Data Samples

The Data Editing table lists both the raw data set and the modified data that you create.

There are two tabs in the Data Editing pane: Raw data and Modified data. The Raw Data pane shows the working copy of the data. For example, if you exclude rows of data in the Raw data pane, the corresponding rows of numbers become red in this table. By default the Modified data pane represents the rows you removed by inserting NaNs.

In the Modified data pane, you can choose to remove the excluded data completely or interpolate it. See Handling Missing Data for more information.

After you select data for exclusion, you can view it graphically by clicking Exclude Graphically.

As you make changes in the Data Editing pane, they immediately appear in the Select Points for Preprocessing Rule window, and vice versa.

Back to Top

Adding Preprocessed Data Sets to an Estimation Project

After you preprocess the data using the techniques described in Ways to Preprocess Data Using the Data Preprocessing Tool, you can add the data set to an estimation project either by overwriting an existing data set or creating a new data set.

• Overwriting an Existing Data Set

• Creating a New Data Set

Overwriting an Existing Data Set

To overwrite an existing data set with the preprocessed data:

1. In the Write results to area of the Data Preprocessing Tool GUI, select the existing dataset option.

2. Choose the data set you want to overwrite from the drop-down list.

   

3. Click Add.

   This action overwrites the selected data set with the modified data in the Control and Estimation Tools Manager GUI.

   

Tip   You can export the preprocessed data to the MATLAB Workspace, as described in Exporting Prepared Data to the MATLAB Workspace.

Creating a New Data Set

If you do not want to overwrite an existing data set with the preprocessed data, as described in Overwriting an Existing Data Set, you can create a new data set for the preprocessed data:

1. In the Write results to area of the Data Preprocessing Tool GUI, select the new dataset option.

2. Specify the name of the data set in the adjacent field.

   

3. Click Add.

   This action adds a new data node in the Control and Estimation Tools Manager GUI containing the modified data.

   

Tip   You can export the preprocessed data to the MATLAB Workspace, as described in Exporting Prepared Data to the MATLAB Workspace.

Back to Top

Exporting Prepared Data to the MATLAB Workspace

After you add the preprocessed data to an estimation project, as described in Adding Preprocessed Data Sets to an Estimation Project, you can export the data set to the MATLAB Workspace. You can use the data to further prepare it or estimate parameters using the data.

1. In the Transient Data node of the Control and Estimation Tools Manager GUI, select the node containing the prepared data set.

2. Right-click the table Data cell containing the data that you want to export, and select Export.

   The Export to Workspace dialog box opens.

3. Specify the MATLAB variable names for the prepared data and the corresponding time vector in the Data and Time fields, respectively.

   

4. Click OK.

   The resulting MATLAB variables data and time4 appear in the MATLAB Workspace browser.

Back to Top

Plotting and Analyzing Data in the GUI

Estimating Model Parameters

© 1984-2009- The MathWorks, Inc.    -   Site Help   -   Patents   -   Trademarks   -   Privacy Policy   -   Preventing Piracy   -   RSS