Documentation Center

  • Trial Software
  • Product Updates

Construct a Test

The SystemTest™ interface provides a graphical integrated environment that you can use to create and edit tests. Tests consist of elements, test vectors, and test variables. You can use each of these entities to create a variety of test scenarios ranging from a simple test that runs a series of elements once to a full parameter sweep that iterates over the values of test vectors that you define.

The following sections show how to construct a test:

Start the SystemTest Software

Start by opening the SystemTest desktop by typing systemtest at the MATLAB® command line.

The SystemTest software displays the desktop on your screen. See Overview of the SystemTest Software for an overview.

Structure a Test

The SystemTest software divides tests into three sections.

  • Pre Test — This section is used to execute test elements in order to perform any test set-up operations, such as initializing variables, loading data from a file, and initializing system resources. Using Pre Test variables, you can assign an initial value to a test variable that persists between Main Test section iterations (unless another element in Main Test modifies the value). Pre Test is not mandatory, but it can be used if your test requires set-up operations to be performed.

  • Main Test — Main Test defines the test elements that need to be performed across the parameter space defined by your test vectors. In this section Main Test variables are initialized before each Main Test iteration, which lets you assign an initial value to a test variable each time the Main Test runs. This is useful if your test variable has a derived value such as being indexed by a test vector or is the result of a MATLAB expression.

    The number of iterations performed in the Main Test is indicated in the Test Browser in parentheses after Main Test. Iterations specifies the number of times the Main Test section will be run. This is determined from the test vectors you define. The SystemTest desktop also offers a Save Results area for you to specify which test variables you want to save as test results at the end of each Main Test iteration.

  • Post Test — In this section you can perform any cleanup work necessary at the completion of the Main Test section, such as clearing workspace variables, closing a file, or freeing system resources.

The following figure illustrates the structure of a test.

How Test Vectors and Test Variables Relate to the MATLAB Workspace

The SystemTest software has its own internal workspace that it uses to manage test variables and test vectors independently. However it does leverage the MATLAB workspace during test execution, and when using a MATLAB element.

During test execution, SystemTest test variables and test vectors are evaluated in the MATLAB base workspace. Then at the end of test execution, they are cleared out and the MATLAB base workspace is restored to what it was before the test execution.

When using a MATLAB element in the SystemTest software, you can reference a variable in the base workspace without having to create a test vector or test variable in the SystemTest software. However the SystemTest software will not be aware of this data, so you could not make use of it in any other element type or in saved results. You can only access it from a MATLAB element. If you need to use it in other elements, you can create test variables or test vectors in the SystemTest software.

Create a Test Vector

Test vectors are composed of values derived from a MATLAB expression. You can use any MATLAB expression that evaluates to a 1-by-n matrix or cell array to define your test vector. Using test vectors, you can iterate through a range of values to see how a system performs. Test vectors constitute parameterized testing in the SystemTest software. They are the test cases for your test.

For tests with multiple test vectors, the product of the lengths of the test vectors defines the number of iterations the test performs. For example, if you define the test vector [10 20 30], the test runs three times, using a value of 10 for the first run, 20 for the second, and 30 for the final run. If you add a second test vector with three other values, the total number of test runs would be nine. The SystemTest software iterates through each vector in combination with the other vector as though the test were a group of nested FOR loops—the outermost loop being the first test vector in your table and the innermost loop being the last test vector. The Main Test section in the Test Browser shows the total number of test iterations defined by your test vectors.

For the example, use the vector [pi/15:pi/15:4*pi] which defines 60 values for our test vector ranging from pi/15 to 4*pi in pi/15 increments. To specify this test vector:

  1. Click the New Vector button in the Test Vectors pane.

    The Insert Test Vector dialog box opens.

  2. Keep the default test vector type of MATLAB Expression. Assign a name to the test vector by clicking the Name field. For this example, name the test vector signal.

  3. Assign a value to the test vector by clicking the Expression field. Enter the test vector specified above for the pi values. Click OK.

After you create the test vector, in the Test Browser pane, the Main Test section label updates to include the number of iterations defined by the test vector. It should say Main Test (60 Iterations).

    Note:   Grouping test vectors determines how they will be iterated through when the test runs. For information on grouping vectors, see Grouped Test Vectors.

Define Test Variables

The SystemTest software uses test variables to define temporary storage variables that a test acts on or generates. You assign test variables in the Pre Test or Main Test sections of your test.

You can define Pre Test variables or Main Test variables. Using Pre Test variables, you can assign an initial value to a test variable that persists between Main Test section iterations (unless another element in Main Test modifies the value). Pre Test is not mandatory, but it can be used if your test requires set-up operations to be performed.

Main Test defines the test elements that need to be performed across the parameter space defined by your test vectors. Main Test variables are initialized before each Main Test iteration, which allows you to assign an initial value to a test variable each time the Main Test runs. This is useful if your test variable has a derived value such as being indexed by a test vector or is the result of a MATLAB expression. You add elements in this section.

The example test requires three test variables:

  • Y — Contains a value that will be calculated from the signal test vector at each iteration.

  • HiLimit — Contains the upper limit for Y that you do not want the signal to exceed.

  • LowLimit — Contains the lower limit for Y that you do not want the signal to go below.

To create these test variables:

  1. Click the Test Variables tab in the middle pane of the SystemTest desktop.

  2. Click the New button to create a Pre Test or Main Test variable. The Insert Test Variable dialog box opens. Leave the default value of Main Test in the Assigned in field, to create a new Main Test variable.

  3. Assign a name to the test variable by clicking the Name field and entering the test variable name. For this example, enter Y.

  4. Set the test variable's initial value by clicking the Initial Value field and entering a value. For the example test variable Y, enter 0. Click OK.

      Note:   If you do not provide an initial value, it will default to empty, that is, Var1 = []; in MATLAB code.

      Note:   Test variables are re-initialized at the start of each test iteration. The Initial value field is blank by default when you create a test variable. If you leave it blank, it will initialize to []. If you enter an initial value (which can be any valid MATLAB expression), that value gets assigned in every iteration.

  5. Repeat steps 2 to 4 to create the remaining two test variables, using the settings listed in the following table:

    Variable NameInitial ValueAssign in
    HiLimit1Main Test
    LowLimit-1Main Test

Add Elements

Elements are the actions that a test performs. The SystemTest software includes the following set of elements, listed in alphabetical order.

  • General Plot — Used to plot any type of data over multiple iterations.

  • IF — Implements a logic control operator.

  • Limit Check — Specifies the comparison to be performed of the value(s) under test and their expected value(s), or limit(s).

  • MATLAB — Executes any MATLAB statements.

  • Simulink — Runs a Simulink® model. Note that you need to have a license for Simulink to use this element.

  • Stop — Implements a logic control operator.

  • Subsection — Creates a new section in a test that you can use to group elements within.

    Note:   Some MathWorks® products, such as the Image Acquisition Toolbox™, the Data Acquisition Toolbox™, and the Instrument Control Toolbox™, provide their own elements that integrate those products' capabilities within the SystemTest software. If you have licenses for those products, those elements will also appear in the elements list.

You add elements to a section in your test; however, not all elements can be added to all sections. For example, you can use a MATLAB element anywhere within a test, but you can only use the Limit Check element in the Main Test section.

To illustrate using elements, let's continue with this example. This test uses three elements in the Main Test section.

ElementDescription
MATLABUse a MATLAB expression to assign data to Y that is dependent on the test vector signal.
Limit CheckCompare the value generated in the MATLAB element to the specified limit and see if the Y test variable exceeds the upper or lower limit you defined in your HiLimit and LowLimit test variables.
General PlotPlot the current test variable values and see whether the test variable exceeds the upper and lower limits.

To add these elements:

  1. Select the section of the test in which you want to add the element. For this example, click Main Test in the Test Browser.

  2. Specify the element you want to add to the test section. For this example, click the New > Test Element button and select MATLAB. A MATLAB element appears in the Main Test section of your test and the MATLAB element property page opens in the Properties pane of the SystemTest desktop.

  3. In the Properties pane, type the following code in the MATLAB Script edit box. This MATLAB code calculates a value for Y that is dependent on the test vector signal.

    Y = sin(signal)+ rand -.5;
    

    During each iteration, the SystemTest software evaluates the MATLAB expression and assigns a value to Y.

  4. Add the Limit Check element to the Main Test section of the test. With the MATLAB element selected, click the New > Test Element button, and click Limit Check. A Limit Check element appears in the Main Test section of the test and the Limit Check properties page opens in the Properties pane. For this example, the Limit Check element must follow the MATLAB element in the test.

      Note:   You can reposition an element in a test by selecting the element and then clicking the up and down arrows in the Test Browser toolbar. You can also drag and drop elements within Main Test. You cannot move elements between test sections.

    In the General Check tab, click the New button to add a limit check. Notice that the Limit Check element icon in the Test Browser shows a red x, which indicates that information is missing. The corresponding red outlining in the Properties pane highlights any fields that require configuration. A test cannot run unless everything is properly configured.

  5. Specify the limit comparison operations in the Limit Check element.

    1. In the Test Variable column, click the drop-down list and select a test variable you created in step 4. For this example, select Y.

    2. In the Operator column, click the drop-down list and select the comparison you want to perform. For this example, pick the less-than-or-equal-to operator, <=.

    3. In the Limit column, click the drop-down list and select the test variable you want to compare to. For this example, select HiLimit, which is the test variable you created earlier.

    The following figure shows the configuration of this limit.

  6. To add another limit comparison operation, click the New button again in the General Check Properties pane. A new row appears below the last limit you specified. In this new row, set Test Variable to Y, set Operator to >=, and set Limit to LowLimit.

    The following figure shows the configuration of this second limit.

    For each iteration of the Main Test, the MATLAB element's expression is evaluated and a new value assigned to Y. When the Limit Check element runs, it determines whether the value of Y falls between the HiLimit and LowLimit values. If Y is outside this range, the test iteration fails. The default pass/fail criteria for the overall test passes the test only if both expressions in the limit check evaluate to true.

  7. To view the test variables as the test runs, plot the data. To add a Plot element to the test, click the New > Test Element button, and select General Plot. A General Plot element appears in the Main Test section, and the properties page for the element opens in the Properties pane.

    With each Main Test iteration of the test, the General Plot element updates a figure window with data you selected.

  8. Click the Add Plot button, and choose plot from the list to create a line plot.

  9. We will set up three axes. For the first axes, use the one automatically created. Configure it as follows:

    Click the arrow in Y Data Source and select Y.
    Keep the default Line color of blue, and keep the default Line style of solid.
    Change Line marker to point (the first selection in the list that shows one dot).
    On the Options tab, select Keep any existing data on the figure.
  10. Add the second and third axes by clicking the Add Plot button again twice and choosing plot from the list to create a line plot.

  11. Configure the second and third axes to match the following table, and using <Iteration Number> as the X Data Source for each one and selecting the Keep any existing data on the figure option for each one. The configured element looks like the figure following the table.

    Y Data SorceLine ColorLine StyleLine Marker
    YBlueSolidPoint
    HiLimitRedDashedNo Marker
    LowLimitBlackDashedNo Marker

To see the resulting plot, see Track Output.

Define Pass/Fail Criteria

You can define whether your test passes or fails by monitoring the outcome of any or all Limit Check elements during any or all Main Test iterations. Your test's threshold of success can range from the passing of any Limit Check in any single test iteration to the passing of all Limit Check elements in all test iterations. If your test contains no Limit Check elements, there is no notion of pass/fail and no pass/fail information is displayed. (Testing of this type is useful for experimenting with a system or to explore its behavior rather than validate its performance.)

You can set any of the following conditions to define when your test passes:

  • All Limit Check elements pass in all test iterations.

  • All Limit Check elements pass in any test iteration.

  • Any Limit Check element passes in all test iterations.

  • Any Limit Check element passes in any test iteration.

You can configure this behavior within the test's Properties pane. Click the test name in the Test Browser (named Untitled by default) to open the test's properties and look for the section labeled This Test Passes If.

Using the signal test example that you constructed in this section, set the test to pass if all Limit Check elements pass in all test iterations.

Save Test Results

You can save the results from the iterations of your test in a MAT-file. You must explicitly specify which test variables to save as test results.

    Note:   Test variables that are not saved as a test result will be lost at the end of the test execution.

The SystemTest software lets you save results at the end of each iteration. Before running your test, select the Save Results section in your test and specify which test variables to save as test results. Click the New Mapping button and then select from the drop-down list the name of the test variable to map to a result. You can optionally specify a name for the results that you want to save. By default, the name of the saved result is the same as the test variable. The following figure shows the mapping of test variables to test results.

After specifying which test variables to save as test results, specify the name of the MAT-file to use. Using this MAT-file you can reload the test results into the base workspace. By default for a test that is untitled, the SystemTest software names the file Untitled_results.mat and puts the file in the same folder as the TEST-File. To change the name or location of the MAT-file, click the test name in the Test Browser, then click the Output Files tab in the Properties pane.

Use the Select File Names section to name the results file. Use the Output Folder Numbering section to select overwrite behavior. By default, each time you run the test you overwrite this file unless you select the Create new folder for each run option. If you select that option, a new folder is created for each run and a new results file is created and put inside the folder. In this case, the Outputs subfolder name is appended by successive numbers for each test run. For example, if the folder name is MyResults, the first test run will create MyResults\Outputs and the next test run results folder will be called MyResults\Outputs(1), followed by MyResults\Outputs(2), etc.

Use the Select Output Folder section to designate the location to save the results file, as follows:

  • Same folder as TEST-file – This default setting saves any output files to the same location as the TEST-File. In this case your test and any output files it generates will be in the same location. You set this location when you select File > Save to save your test, or if prompted to save when you close the SystemTest software.

  • Same folder as the MATLAB current folder (PWD) – Save any output files to the current working directory in MATLAB. You can see the current working directory when you open SystemTest from MATLAB.

  • Browse – Select the third option and then click the Browse button to choose an absolute directory location for the output files. This location should be stable and not read-only.

Note that the location you select here is also where the Test Report will be saved if you generate one by selecting the Generate report check box.

    Note:   If a file or folder location is read-only, you will get an error when the test runs. For results files and Test Reports to be generated, the files and folder locations must be writable.

Generate a Test Report

When you run your test, the test status appears in the Run Status pane. This display contains basic information about your test:

  • Time elapsed since your test started running.

  • Which section your test is in.

  • How many test iterations have passed or failed as defined by any limit checks.

  • Whether your test completed successfully.

  • Any errors that cause your test to stop.

You can generate and save more detail about the running test by enabling the Test Report, which is a test execution log file in HTML format. This report is useful when you use limit checks in your test and you want to see specific test iterations that passed or failed. For example, instead of just finding that a test iteration failed, the report helps you determine how far a test variable varied from the upper or lower limit you defined in a Limit Check element. It also displays any plots that were generated. This report is also useful for documenting and sharing your test results.

To enable the Test Report:

  1. Select the test name in the Test Browser, then click the Output Files tab on the Properties pane.

  2. In the Select File Names section, select the Generate report check box.

  3. Use the default name or type a new name in the edit field next to the check box.

  4. Use the Select Output Folder section to designate the location to save the Test Report:

    • Same folder as TEST-file – This default setting saves any output files to the same location as the TEST-File. In this case your test and any output files it generates will be in the same location. You set this location when you select File > Save to save your test, or if prompted to save when you close the SystemTest software.

    • Same folder as the MATLAB current folder (PWD) – Save any output files to the current working directory in MATLAB. You can see the current working directory when you open SystemTest from MATLAB.

    • Browse – Select the third option and then click the Browse button to choose an absolute directory location for the output files. This location should be stable and not read-only.

The Test Report is stored in an Outputs subfolder in this folder, along with all dependent files, such as plot or Simulink model snapshots. The overwrite options you set for your test results MAT-file also apply to the file name and folder of your report file. To learn how to change these options, see Save Test Results .

Note that the location you select here is also where the test results will be saved.

    Note:   If a file or folder location is read-only, you will get an error when the test runs. For results files and Test Reports to be generated, the files and folder locations must be writable.

The Test Report contains the following information about the test run, organized by iteration in the report:

  • The test description, if you entered one in the Description field of the Properties pane of the test.

  • A test summary, including start and stop times, number of iterations completed, number of iterations that passed and failed, and final status of the test.

  • Pass/fail results of Limit Check elements, by iteration.

  • Values for any saved results you captured by setting up mappings in Saved Results, by iteration.

  • Test vector values, by iteration.

  • A snapshot of your model if you use a Simulink element in the test.

  • A snapshot of your plot if you use a Vector Plot, Scalar Plot, or General Plot element in your test, by iteration.

  • A summary of generated files, with links to them. These can include a Simulink model coverage report and test results.

    Note:   Because the Test Report generates while the test is running, this option results in the test taking longer to execute.

To see what information the report generates, see View the Test Report.

Save a Test

You can save tests so that you can reuse them later. For example, to save the signal test:

  1. Select File > Save As to open the Save file as dialog box.

  2. Select a directory location and enter mySavedTest in the File name field.

  3. Click Save.

The SystemTest software saves the test as mySavedTest.test and renames your test as it appears in the Test Browser. This does not rename the test results MAT-file or the Test Report file. Their names are controlled separately from the name of the test, as explained in Save Test Results.

Was this topic helpful?