Documentation

Run Combinations of Tests Using Iterations

Test Manager iterations help you create test cases for multiple sets of data. Use iterations to test multiple combinations of parameter sets, external inputs, configuration sets, Signal Builder groups, or baselines. The Iterations section of a test case enables you to have many iterations in one centralized location.

There are two ways to set up iterations: tabled and scripted. You can use one or both ways to create iterations in a test case. If you use iterations in a test case where you have specified coverage settings, then the same coverage settings are applied to all iterations in the test case.

Create Table Iterations

The Table Iterations section is a quick way to add iterations. The table makes the set of iterations easy to view at a glance.

  1. Add parameter sets, external inputs, configuration sets, Signal Builder groups, or baselines to a test case if they are applicable to your tests.

  2. To add an iteration to the table manually, click Add.

  3. By default, the Parameter Set and External Input columns are visible in the table. To add or remove columns, click the button, and select a column from the list.

  4. In the iteration row, select the column cell you want to use to change the test setting. For example, if you want to have an iteration with a parameter set, click the cell below Parameter Set, and select the parameter set from the list.

Autogenerated iteration combinations are ordered in lockstep. Lockstep means that each iteration is formed using sequential pairings of test case settings. For example, the model sldemo_autotrans has a Signal Builder block with four signal groups, labeled in the figure as S1, S2, S3, and S4. If you use this model in a test case with three parameter sets, labeled as P1, P2, and P3, then the Test Manager generates three iterations. Generated iterations are limited to the minimum number settings between Signal Builder groups and parameter sets, which is three. Each iteration, labeled as I1, I2, and I3, contains one Signal Builder group with the corresponding parameter set. The Signal Builder group and parameter set are matched in the order that they are listed in the Signal Builder block or parameter set section, respectively.

In the table iterations, Default [None] means that the iteration does not change the test case setting. The test iteration setting is the same as what is specified in the test case.

View Table Iterations

To see a list of iterations from the Table Iterations section, click Show Iterations. The list includes table iterations and scripted iterations.

Generate Table Iterations

If you have test case settings that you want to transform into test iterations, then you can use the Auto Generate button. You can select the test case sections to use to generate iterations. If you select multiple sections, then the Test Manager combines iterations, and lockstep ordering applies.

When you autogenerate iterations, you can specify the naming rule for each iteration. Click Auto Generate and select the settings to use as the basis for the iterations. You can then use a rule to name the iterations to help to identify them. In the Iteration naming rule box, enter the rule using:

  • The name of each setting you want to use in the name, with spaces removed

  • An underscore or space to separate each setting

For example, if you want to include the name of the parameter set, configuration set, and baseline file name, enter ParameterSet_ConfigurationSet_Baseline. When you generate the iterations, the corresponding names of the test case replace each part of the rule.

Section OptionPurpose

Signal Builder Group

Applies to the Inputs section of a simulation, baseline, or equivalence test case, for the specified Signal Builder Group. Each Signal Builder group is used to generate an iteration.

Parameter Set

Applies to the Parameter Overrides section of a simulation, baseline, or equivalence test case. Each parameter override set is used to generate an iteration.

External Input

Applies to the Inputs section of a simulation, baseline, or equivalence test case, for the specified External Inputs sets. Each external input set is used to generate an iteration.

Configuration Set

Applies to the Configuration Setting Overrides section of a simulation, baseline, or equivalence test case. Each iteration uses the configuration setting specified.

Baseline

Applies only to baseline test case types, specifically to the Baseline Criteria section of a baseline test case. Each baseline criteria set is used to generate an iteration.

Simulation 1 or 2

Applies only to equivalence test case types. At the top of the Auto Generate Reports dialog box, there is a menu for Simulation 1 or Simulation 2. These sections correspond to the two simulation sections within the equivalence test case.

Create Scripted Iterations

In the scripted iterations section of the test case, you can customize your own set of iterations using a programmatic workflow. You can define your own parameter sets, customize the order of the iterations, create your own Monte Carlo script, and more. Scripted iterations are generated at run time when a test executes. Enter the script into the Scripted Iterations section text box.

Iteration Script Components

An iteration script must have certain components to execute the tests. The basic iteration script contains three elements: an iteration object, an iteration setting, and the iteration registration. This script iterates over a single signal builder groups. This example is not practical, but it is meant to illustrate the anatomy of an iteration script.

%% Iterate Using a Signal Builder Group
 
% Set up a new iteration object
testItr = sltestiteration;

% Set iteration setting using Signal Builder group
setTestParam(testItr,'SignalBuilderGroup',sltest_signalBuilderGroups{1});

% Add the iteration to run in this test case
% The predefined sltest_testCase variable is used here
addIteration(sltest_testCase,testItr);

For more information about the test iteration class, see sltest.testmanager.TestIteration. In practice, you iterate over numerous settings, such as multiple Signal Builder groups. If you take the stripped-down iteration script and put it into a loop, you can iterate over all Signal Builder groups in the test case.

%% Iterate Over All Signal Builder Groups
 
% Determine the number of possible iterations
numSteps = length(sltest_signalBuilderGroups);
 
% Create each iteration
for k = 1 : numSteps
    % Set up a new iteration object
    testItr = sltestiteration;
 
    % Set iteration settings
    setTestParam(testItr,'SignalBuilderGroup',sltest_signalBuilderGroups{k});
 
    % Add the iteration to run in this test case
    % You can pass in an optional iteration name
    addIteration(sltest_testCase,testItr);
end

Predefined Variables

You can use predefined variables to write iterations scripts. To see the list of predefined variables in the Test Manager, expand the Help on creating test iterations section. You write the iterations script in the script box within the Scripted Iterations section. The script box is a functional workspace, which means the MATLAB® base workspace cannot access information from the script box. If you define variables in the script box, then other workspaces cannot use the variable.

The predefined variables are:

  • sltest_bdroot — Model simulated by the test case, defined as a string

  • sltest_sut — The System Under Test, defined as a string

  • sltest_isharnesstrue if sltest_bdroot is a harness model, defined as a logical

  • sltest_externalInputs — Name of external inputs, defined as a cell array of strings

  • sltest_parameterSets — Name of parameter override sets, defined as a cell array of strings

  • sltest_configSets — Name of configuration settings, defined as a cell array of strings

  • sltest_tableIterations — Iteration objects created in the iterations table, defined as a cell array of sltest.testmanager.TestIteration objects

  • sltest_testCase — Current test case object, defined as an sltest.testmanager.TestCase object

Scripted Iteration Templates

You can quickly generate iterations for your test case using templates for Signal Builder groups, parameter sets, external inputs, configuration sets, and baseline sets, if you are using a baseline test case. Scripted iteration templates follow lockstep ordering and pairing of test settings. For more information about lockstep ordering, see Create Table Iterations.

For example, if you want to run all signal builder groups in a scripted iteration:

  1. Click Iteration Templates.

  2. Select the test case settings you want to iterate through. Click OK.

    The script is generated and added to the script box below any existing scripts.

  3. To generate a table that gives a preview of the iterations that execute when you run the test case, click Show Iterations.

Capture Baseline Data from Iterations

This example shows how to create a baseline test by capturing data from a test case with table iterations. You create the iterations from Signal Builder groups in the model. Before running the example, navigate to a writable folder on the MATLAB® path.

1. Open the model. At the command line, enter

Model = 'sltestCar';
open_system(fullfile(matlabroot,'examples','simulinktest',Model));

2. Create a test file that contains iterations, and open the Test Manager. At the command line, enter

tf = sltest.testmanager.TestFile('IterationBaselineTest');
sltest.testmanager.load(tf.Name);
sltest.testmanager.view;

3. In the Test Manager, right-click the test case and select Rename. Rename the test case Baseline Test.

4. In the System Under Test section, for Model, enter sltestCar.

5. Select the signals for the baseline data:

  1. In the Simulation Outputs section, click Add. The Signal Selection dialog box appears.

  2. In the model canvas, select the output torque and vehicle speed signals. The signals appear in the Signal Selection dialog box.

  3. In the dialog box, select both signals and click Add.

  4. The signals appear in the Logged Signals table.

6. Add iterations for the test case:

  1. Expand the Iterations section of the test case.

  2. Expand the Table Iterations section and click Auto Generate.

  3. In the dialog box, select Signal Builder Group. Click OK.

  4. The table lists the iterations corresponding to the four Signal Builder groups.

7. Capture baseline data for the iterations:

  1. In the Baseline Criteria section, click the arrow next to Capture, and select Capture for Iterations.

  2. Specify a location for the baseline data files.

  3. Click Create.

The model simulates for all Signal Builder groups. The baseline data for output_torque and vehicle_speed are captured in four MAT files. Also, each baseline data set is added to its corresponding iterations in the table.

Sweep Through a Set of Parameters

Scripted iterations can be used to test a model by sweeping through a set of parameters. In this example of a parameter sweep, the number of Signal Builder groups and parameter values is the same. Each iteration has one Signal Builder group and one parameter value for a total of four iterations.

%% Iterate over all Signal Builder Groups and Parameters
 
% Determine the number of possible iterations
numSteps = length(sltest_signalBuilderGroups);

% Set up the parameter values to sweep over
IeiValues = [0.021,0.022,0.022,0.023];

% Create each iteration
for k = 1 : numSteps
    % Set up a new iteration object
    testItr = sltestiteration;
 
    % Set Signal Builder iteration setting
    setTestParam(testItr,'SignalBuilderGroup',sltest_signalBuilderGroups{k});
    
    % Set value of lei (parameter in model workspace)
    setVariable(testItr,'Name','Iei','Source','model workspace',...
                'Value',IeiValues(k));
 
    % Add the iteration to run in this test case
    addIteration(sltest_testCase,testItr); 
end

See Also

Related Topics

Was this topic helpful?