|
|
|
| R2011b Documentation → SystemTest | |
Learn more about SystemTest |
|
| Contents | Index |
| On this page… |
|---|
This simple example illustrates the four primary stages of testing: planning, building, running the test, and viewing test results.
The example uses a simple MATLAB expression to emulate a scalar measurement during each iteration of the test. The example uses an arbitrary formula dependent on the test vector named signal to generate the Y data. The example tests each measurement to determine if it falls within certain specified limits. If a measurement exceeds these limits, that particular iteration of the test fails. By default, the test fails if any iteration fails, but you can configure other pass/fail criteria.
The following sections provide more information about each stage, building the example test along the way. If you prefer, instead of working through the following sections to build the example, you can load it into the SystemTest software by running the Getting Started with SystemTest demo from the Demos page in the MATLAB Help browser (under MATLAB > SystemTest > MATLAB) or by entering systemtest Simple_Demo at the MATLAB command prompt.
In this first stage, you must identify what it is you want to test. The SystemTest software lets you specify input data, such as measurements from a model or device, and compare this input data to some predefined limits. Based on this comparison, the SystemTest software can declare whether a test passes or fails.
Keep the following in mind as you plan tests:
Identify your test data and test vectors.
Specify test limits and determine if these limits can be expressed as scalar or matrix values. (The Limit Check element supports both scalar and matrix data.)
Determine what operations your test must perform. Must certain operations happen before others?
Determine pass/fail criteria for your test.
Decide which test variables you want to save as test results.
After this planning, you can begin to construct your test, which is described in Building Your Test.
The SystemTest interface provides a graphical integrated environment that you can use to create and edit tests. Tests consist of elements, test vectors, and test variables. You can use each of these entities to create a variety of test scenarios ranging from a simple test that runs a series of elements once to a full parameter sweep that iterates over the values of test vectors that you define.
The following sections show how to construct a test:
Start by opening the SystemTest desktop using the MATLAB Start button. To open the SystemTest software, select Start > MATLAB > SystemTest > SystemTest Desktop.
Alternatively, you can execute the systemtest command from the MATLAB command line.
The SystemTest software displays the desktop on your screen. See Quick Tour of the SystemTest Software for an overview.
The SystemTest software divides tests into three sections.
Pre Test — This section is used to execute test elements in order to perform any test set-up operations, such as initializing variables, loading data from a file, and initializing system resources. Using Pre Test variables, you can assign an initial value to a test variable that persists between Main Test section iterations (unless another element in Main Test modifies the value). Pre Test is not mandatory, but it can be used if your test requires set-up operations to be performed.
Main Test — Main Test defines the test elements that need to be performed across the parameter space defined by your test vectors. In this section Main Test variables are initialized before each Main Test iteration, which lets you assign an initial value to a test variable each time the Main Test runs. This is useful if your test variable has a derived value such as being indexed by a test vector or is the result of a MATLAB expression.
The number of iterations performed in the Main Test is indicated in the Test Browser in parentheses after Main Test. Iterations specifies the number of times the Main Test section will be run. This is determined from the test vectors you define. The SystemTest desktop also offers a Save Results area for you to specify which test variables you want to save as test results at the end of each Main Test iteration.
Post Test — In this section you can perform any cleanup work necessary at the completion of the Main Test section, such as clearing workspace variables, closing a file, or freeing system resources.
For details about the sections of the test, see Working with the Sections of a Test.
The following figure illustrates the structure of a test.
The SystemTest software has its own internal workspace that it uses to manage test variables and test vectors independently. However it does leverage the MATLAB workspace during test execution, and when using a MATLAB element.
During test execution, SystemTest test variables and test vectors are evaluated in the MATLAB base workspace. Then at the end of test execution, they are cleared out and the MATLAB base workspace is restored to what it was before the test execution.
When using a MATLAB element in the SystemTest software, you can reference a variable in the base workspace without having to create a test vector or test variable in the SystemTest software. However the SystemTest software will not be aware of this data, so you could not make use of it in any other element type or in saved results. You can only access it from a MATLAB element. If you need to use it in other elements, you can create test variables or test vectors in the SystemTest software.
Test vectors are composed of values derived from a MATLAB expression. You can use any MATLAB expression that evaluates to a 1-by-n matrix or cell array to define your test vector. Using test vectors, you can iterate through a range of values to see how a system performs. Test vectors constitute parameterized testing in the SystemTest software. They are the test cases for your test.
For tests with multiple test vectors, the product of the lengths of the test vectors defines the number of iterations the test performs. For example, if you define the test vector [10 20 30], the test runs three times, using a value of 10 for the first run, 20 for the second, and 30 for the final run. If you add a second test vector with three other values, the total number of test runs would be nine. The SystemTest software iterates through each vector in combination with the other vector as though the test were a group of nested FOR loops—the outermost loop being the first test vector in your table and the innermost loop being the last test vector. The Main Test section in the Test Browser shows the total number of test iterations defined by your test vectors.
For the example, use the vector [pi/15:pi/15:4*pi] which defines 60 values for our test vector ranging from pi/15 to 4*pi in pi/15 increments. To specify this test vector:
Click the New Vector button in the Test Vectors pane.
The Insert Test Vector dialog box opens.
Keep the default test vector type of MATLAB Expression. Assign a name to the test vector by clicking the Name field. For this example, name the test vector signal.
Assign a value to the test vector by clicking the Expression field. Enter the test vector specified above for the pi values. Click OK.
After you create the test vector, in the Test Browser pane, the Main Test section label updates to include the number of iterations defined by the test vector. It should say Main Test (60 Iterations).
Note Grouping test vectors determines how they will be iterated through when the test runs. For information on grouping vectors, see Creating Grouped Test Vectors. |
Note You can also use probability distributions when you create a test vector. For information, see Creating Randomized Test Vectors with Probability Distributions. |
The SystemTest software uses test variables to define temporary storage variables that a test acts on or generates. You assign test variables in the Pre Test or Main Test sections of your test.
You can define Pre Test variables or Main Test variables. Using Pre Test variables, you can assign an initial value to a test variable that persists between Main Test section iterations (unless another element in Main Test modifies the value). Pre Test is not mandatory, but it can be used if your test requires set-up operations to be performed.
Main Test defines the test elements that need to be performed across the parameter space defined by your test vectors. Main Test variables are initialized before each Main Test iteration, which allows you to assign an initial value to a test variable each time the Main Test runs. This is useful if your test variable has a derived value such as being indexed by a test vector or is the result of a MATLAB expression. You add elements in this section.
The example test requires three test variables:
Y — Contains a value that will be calculated from the signal test vector at each iteration.
HiLimit — Contains the upper limit for Y that you do not want the signal to exceed.
LowLimit — Contains the lower limit for Y that you do not want the signal to go below.
To create these test variables:
Click the Test Variables tab in the middle pane of the SystemTest desktop.
Click the New button to create a Pre Test or Main Test variable. The Insert Test Variable dialog box opens. Leave the default value of Main Test in the Assigned in field, to create a new Main Test variable.
Assign a name to the test variable by clicking the Name field and entering the test variable name. For this example, enter Y.
Set the test variable's initial value by clicking the Initial Value field and entering a value. For the example test variable Y, enter 0. Click OK.
Note If you do not provide an initial value, it will default to empty, that is, Var1 = []; in MATLAB code. |
Note Test variables are re-initialized at the start of each test iteration. The Initial value field is blank by default when you create a test variable. If you leave it blank, it will initialize to []. If you enter an initial value (which can be any valid MATLAB expression), that value gets assigned in every iteration. |
Repeat steps 2 to 4 to create the remaining two test variables, using the settings listed in the following table:
| Variable Name | Initial Value | Assign in |
|---|---|---|
| HiLimit | 1 | Main Test |
| LowLimit | -1 | Main Test |
Elements are the actions that a test performs. The SystemTest software includes the following set of elements, listed in alphabetical order.
General Plot — Used to plot any type of data over multiple iterations.
IF — Implements a logic control operator.
Limit Check — Specifies the comparison to be performed of the value(s) under test and their expected value(s), or limit(s).
MATLAB — Executes any MATLAB statements.
Simulink — Runs a Simulink model. Note that you need to have a license for Simulink to use this element.
Stop — Implements a logic control operator.
Subsection — Creates a new section in a test that you can use to group elements within.
Note Some MathWorks® products, such as the Image Acquisition Toolbox™ software, the Data Acquisition Toolbox™ software, and the Instrument Control Toolbox™ software, provide their own elements that integrate those products' capabilities within the SystemTest software. If you have licenses for those products, those elements will also appear in the elements list. |
For more information about using the basic elements, see Working with the Basic Elements.
You add elements to a section in your test; however, not all elements can be added to all sections. For example, you can use a MATLAB element anywhere within a test, but you can only use the Limit Check element in the Main Test section.
To illustrate using elements, let's continue with this example. This test uses three elements in the Main Test section.
| Element | Description |
|---|---|
| MATLAB | Use a MATLAB expression to assign data to Y that is dependent on the test vector signal. |
| Limit Check | Compare the value generated in the MATLAB element to the specified limit and see if the Y test variable exceeds the upper or lower limit you defined in your HiLimit and LowLimit test variables. |
| General Plot | Plot the current test variable values and see whether the test variable exceeds the upper and lower limits. |
To add these elements:
Select the section of the test in which you want to add the element. For this example, click Main Test in the Test Browser.
Specify the element you want to add to the test section. For this example, click the New > Test Element button and select MATLAB. A MATLAB element appears in the Main Test section of your test and the MATLAB element property page opens in the Properties pane of the SystemTest desktop.
In the Properties pane, type the following code in the MATLAB Script edit box. This MATLAB code calculates a value for Y that is dependent on the test vector signal.
Y = sin(signal)+ rand -.5;
During each iteration, the SystemTest software evaluates the MATLAB expression and assigns a value to Y.
Add the Limit Check element to the Main Test section of the test. With the MATLAB element selected, click the New > Test Element button, and click Limit Check. A Limit Check element appears in the Main Test section of the test and the Limit Check properties page opens in the Properties pane. For this example, the Limit Check element must follow the MATLAB element in the test.
Note You can reposition an element in a test by selecting the element and then clicking the up and down arrows in the Test Browser toolbar. You can also drag and drop elements within Main Test. You cannot move elements between test sections. |
In the General Check tab, click the New button to add a limit check. Notice that the Limit Check element icon in the Test Browser shows a red x, which indicates that information is missing. The corresponding red outlining in the Properties pane highlights any fields that require configuration. A test cannot run unless everything is properly configured.
Specify the limit comparison operations in the Limit Check element.
In the Test Variable column, click the drop-down list and select a test variable you created in step 4. For this example, select Y.
In the Operator column, click the drop-down list and select the comparison you want to perform. For this example, pick the less-than-or-equal-to operator, <=.
In the Limit column, click the drop-down list and select the test variable you want to compare to. For this example, select HiLimit, which is the test variable you created earlier.
The following figure shows the configuration of this limit.
To add another limit comparison operation, click the New button again in the General Check Properties pane. A new row appears below the last limit you specified. In this new row, set Test Variable to Y, set Operator to >=, and set Limit to LowLimit.
The following figure shows the configuration of this second limit.
For each iteration of the Main Test, the MATLAB element's expression is evaluated and a new value assigned to Y. When the Limit Check element runs, it determines whether the value of Y falls between the HiLimit and LowLimit values. If Y is outside this range, the test iteration fails. The default pass/fail criteria for the overall test passes the test only if both expressions in the limit check evaluate to true.
To view the test variables as the test runs, plot the data. To add a Plot element to the test, click the New > Test Element button, and select General Plot. A General Plot element appears in the Main Test section, and the properties page for the element opens in the Properties pane.
With each Main Test iteration of the test, the General Plot element updates a figure window with data you selected.
Click the Add Plot button, and choose plot from the list to create a line plot.
We will set up three axes. For the first axes, use the one automatically created. Configure it as follows:
| Click the arrow in Y Data Source and select Y. |
| Keep the default Line color of blue, and keep the default Line style of solid. |
| Change Line marker to point (the first selection in the list that shows one dot). |
| On the Options tab, select Keep any existing data on the figure. |
Add the second and third axes by clicking the Add Plot button again twice and choosing plot from the list to create a line plot.
Configure the second and third axes to match the following table, and using <Iteration Number> as the X Data Source for each one and selecting the Keep any existing data on the figure option for each one. The configured element looks like the figure following the table.
| Y Data Sorce | Line Color | Line Style | Line Marker |
|---|---|---|---|
| Y | Blue | Solid | Point |
| HiLimit | Red | Dashed | No Marker |
| LowLimit | Black | Dashed | No Marker |
To see the resulting plot, see Tracking Output.
You can define whether your test passes or fails by monitoring the outcome of any or all Limit Check elements during any or all Main Test iterations. Your test's threshold of success can range from the passing of any Limit Check in any single test iteration to the passing of all Limit Check elements in all test iterations. If your test contains no Limit Check elements, there is no notion of pass/fail and no pass/fail information is displayed. (Testing of this type is useful for experimenting with a system or to explore its behavior rather than validate its performance.)
You can set any of the following conditions to define when your test passes:
All Limit Check elements pass in all test iterations.
All Limit Check elements pass in any test iteration.
Any Limit Check element passes in all test iterations.
Any Limit Check element passes in any test iteration.
You can configure this behavior within the test's Properties pane. Click the test name in the Test Browser (named Untitled by default) to open the test's properties and look for the section labeled This Test Passes If.
Using the signal test example that you constructed in this section, set the test to pass if all Limit Check elements pass in all test iterations.
You can save the results from the iterations of your test in a MAT-file. You must explicitly specify which test variables to save as test results.
Note Test variables that are not saved as a test result will be lost at the end of the test execution. |
The SystemTest software lets you save results at the end of each iteration. Before running your test, select the Save Results section in your test and specify which test variables to save as test results. Click the New Mapping button and then select from the drop-down list the name of the test variable to map to a result. You can optionally specify a name for the results that you want to save. By default, the name of the saved result is the same as the test variable. The following figure shows the mapping of test variables to test results.
After specifying which test variables to save as test results, specify the name of the MAT-file to use. Using this MAT-file you can reload the test results into the base workspace. By default for a test that is untitled, the SystemTest software names the file Untitled_results.mat and puts the file in the same folder as the TEST-File. To change the name or location of the MAT-file, click the test name in the Test Browser, then click the Output Files tab in the Properties pane.
Use the Select File Names section to name the results file. Use the Output Folder Numbering section to select overwrite behavior. By default, each time you run the test you overwrite this file unless you select the Create new folder for each run option. If you select that option, a new folder is created for each run and a new results file is created and put inside the folder. In this case, the Outputs subfolder name is appended by successive numbers for each test run. For example, if the folder name is MyResults, the first test run will create MyResults\Outputs and the next test run results folder will be called MyResults\Outputs(1), followed by MyResults\Outputs(2), etc.
Use the Select Output Folder section to designate the location to save the results file, as follows:
Same folder as TEST-file – This default setting saves any output files to the same location as the TEST-File. In this case your test and any output files it generates will be in the same location. You set this location when you select File > Save to save your test, or if prompted to save when you close the SystemTest software.
Same folder as the MATLAB current folder (PWD) – Save any output files to the current working directory in MATLAB. You can see the current working directory when you open SystemTest from MATLAB.
Browse – Select the third option and then click the Browse button to choose an absolute directory location for the output files. This location should be stable and not read-only.
Note that the location you select here is also where the Test Report will be saved if you generate one by selecting the Generate report check box.
Note If a file or folder location is read-only, you will get an error when the test runs. For results files and Test Reports to be generated, the files and folder locations must be writable. |
When you run your test, the test status appears in the Run Status pane. This display contains basic information about your test:
Time elapsed since your test started running.
Which section your test is in.
How many test iterations have passed or failed as defined by any limit checks.
Whether your test completed successfully.
Any errors that cause your test to stop.
You can generate and save more detail about the running test by enabling the Test Report, which is a test execution log file in HTML format. This report is useful when you use limit checks in your test and you want to see specific test iterations that passed or failed. For example, instead of just finding that a test iteration failed, the report helps you determine how far a test variable varied from the upper or lower limit you defined in a Limit Check element. It also displays any plots that were generated. This report is also useful for documenting and sharing your test results.
To enable the Test Report:
Select the test name in the Test Browser, then click the Output Files tab on the Properties pane.
In the Select File Names section, select the Generate report check box.
Use the default name or type a new name in the edit field next to the check box.
Use the Select Output Folder section to designate the location to save the Test Report:
Same folder as TEST-file – This default setting saves any output files to the same location as the TEST-File. In this case your test and any output files it generates will be in the same location. You set this location when you select File > Save to save your test, or if prompted to save when you close the SystemTest software.
Same folder as the MATLAB current folder (PWD) – Save any output files to the current working directory in MATLAB. You can see the current working directory when you open SystemTest from MATLAB.
Browse – Select the third option and then click the Browse button to choose an absolute directory location for the output files. This location should be stable and not read-only.
The Test Report is stored in an Outputs subfolder in this folder, along with all dependent files, such as plot or Simulink model snapshots. The overwrite options you set for your test results MAT-file also apply to the file name and folder of your report file. To learn how to change these options, see Saving Test Results .
Note that the location you select here is also where the test results will be saved.
Note If a file or folder location is read-only, you will get an error when the test runs. For results files and Test Reports to be generated, the files and folder locations must be writable. |
The Test Report contains the following information about the test run, organized by iteration in the report:
The test description, if you entered one in the Description field of the Properties pane of the test.
A test summary, including start and stop times, number of iterations completed, number of iterations that passed and failed, and final status of the test.
Pass/fail results of Limit Check elements, by iteration.
Values for any saved results you captured by setting up mappings in Saved Results, by iteration.
Test vector values, by iteration.
A snapshot of your model if you use a Simulink element in the test.
A snapshot of your plot if you use a Vector Plot, Scalar Plot, or General Plot element in your test, by iteration.
A summary of generated files, with links to them. These can include a Simulink model coverage report and test results.
Note Because the Test Report generates while the test is running, this option results in the test taking longer to execute. |
To see what information the report generates, see Viewing the Test Report.
You can save tests so that you can reuse them later. For example, to save the signal test:
Select File > Save As to open the Save file as dialog box.
Select a directory location and enter mySavedTest in the File name field.
Click Save.
The SystemTest software saves the test as mySavedTest.test and renames your test as it appears in the Test Browser. This does not rename the test results MAT-file or the Test Report file. Their names are controlled separately from the name of the test, as explained in Saving Test Results.
After you build a test, you are ready to run it. At run time, the SystemTest software assigns values to test vectors and test variables in the order they appear in the Test Vectors and Test Variables panes. Each test section runs elements in the order that they appear in the Test Browser.
To execute your test, do one of the following:
Click the Run button.
Select Run > Run.
Press the F5 key.
Note While a test is running, you can stop its execution by pressing Ctrl+C or clicking the Stop button on the toolbar. |
While the test runs, the Run Status pane shows summary test output, including start and stop times, number of iterations completed, number of iterations that passed and failed, and final status of the test. It will also display any error messages if the test has an error.
If your test includes a Plot element, the SystemTest software creates the plot and updates the plot during each iteration. Since Limit Check elements evaluate whether an iteration passed or failed, they directly affect the data that appears in the Test Report and the Run Status pane.
In the example test, the plot includes the high and low limits defined in the Limit Check element, to show which test iterations exceed the limits.
When the test is done running, the Run Status pane provides links to generated output. The Generated Files section contains a summary of generated files, with links to them, such as the Test Report, saved test results, and the Simulink model coverage report, if your test uses the model coverage feature.
After the SystemTest software runs your test, you can explore the results that are generated. This section shows how to:
View and interpret the Test Report.
Inspect your test results.
When you enable the Test Report, the SystemTest software saves information about each test iteration in an HTML file. To enable the Test Report, check the Generate report option on the Output Files tab of the Properties pane before running your test. The report contains summary information about the test run, snapshots of any plots you used, snapshots of any models you used, pass/fail results of Limit Check elements, and other information. See Test Report for a full description of what the report contains.
After a test runs, you can see the contents of this file by clicking Tools > Test Report or using the Test Report link in the Run Status pane. The generated output resembles the following.
The Main Test section of the report shows each iteration. You see the value of the test vector signal and determine the values the Limit Check element used in evaluating whether the test passed. For the first several iterations, the value of Y did not exceed either the high or low limits so the iterations passed. You can also see this in the scalar plot drawn while the test ran. For other iterations that failed, you can scroll through the report to find the values of Y.
The SystemTest software allows you to view the results you have chosen to save for your test using a workspace variable called stresults. It provides access to the test results object, which is useful for comparing the results of separate test runs and for postprocessing test results.
For more information, see Accessing Test Results from the MATLAB Command Line.
To continue the example of Simple Demo, after you run the test, return to MATLAB and type stresults. The following summary of the results is returned.
You can see that the test contains 60 iterations, that it contains a test vector signal, and the names of the three saved results are HiLimit, LowLimit, and Y.
To see a more complete list of properties, type:
get(stresults)
That displays the following.
The ResultsDataSet property contains the test results data in the form of a dataset array. This is what you set up using the Saved Results node in the Test Browser. See Saving Test Results for more information on setting up saved results.
To access the ResultsDataSet property, type:
stresults.ResultsDataSet
This returns the test results data in the form of a dataset array.
In the Simple Demo example, a portion of the test results data looks like this:
In the dataset array, each row represents a test iteration, labeled using the convention of ['I' + Iteration_Number]. This example shows the first 10 iterations. Test vector values are listed first, in alphabetical order, followed by test results, listed in alphabetical order, as shown in the above figure. This is a simple way to view the results you set up in Saved Results. The test results for all iterations are displayed at the command line, even though only the first ten are only shown here.
You can now plot the results. See Plotting Results Data to see plots created from these results.
 | Running Tests from the MATLAB Command Line | Examples |  |
Includes the most popular MATLAB recorded presentations with Q&A sessions led by MATLAB experts.
| © 1984-2012- The MathWorks, Inc. - Site Help - Patents - Trademarks - Privacy Policy - Preventing Piracy - RSS |
