How do I Perform Requirement Based SIL testing and Autogenerate tests for the missing coverage?

I have developed requirement-based tests for my functionality and have performed requirement-based SIL testing. However, I am not getting 100% structural coverage after executing requirements-based tests. How can I autogenerate test cases for missing coverage to achieve 100% structural coverage?

Answer 1

[MathWorks Support Team]

This answer demonstrates the workflow to perform Requirement-Based SIL testing and to add test cases for missing coverage

 

Following MathWorks products are required for this workflow

• Embedded Coder: To generate production code for the model
• Simulink Test: To Author, Create, and Execute tests
• Simulink Coverage: To analyze Code Coverage 
• Simulink Design Verifier: To automatically add tests for missing code coverage using Simulink Design Verifier

The following steps are demonstrated in this workflow

1. Requirement Details
2. Requirement Implementation in Simulink
3. Requirement based tests
4. Test setup in Simulink Test
5. Test execution, Results, and Code Coverage analysis
6. Autogenerating tests for missing code coverage using Simulink Design Verifier

The Simulink Test file and Model shown in the steps below are attached with this answer.

Let us take a simple model and the requirements associated with it.

 

• Inputs and Parameters: gearIn, gearFirst = 1, gearTop = 5
• REQ 1.1: If 'gearIn' is between 1-5 set 'fwdGearEngaged' as TRUE
• REQ 1.2: If 'gearIn' is outside 1-5 set 'fwdGearEngaged' as FALSE

To verify the requirement user has written with the requirement-based tests (RBT) in test harnesses. These tests are written based on the defined system requirements. To learn more about creating harnesses, refer to this documentation: https://www.mathworks.com/help/sltest/ug/select-test-harness-properties-for-your-task.html#mw_30a56f31-c842-46f1-8f76-c811075ac747

For this example, create 2 harnesses as specified below

• Verify_Requirement_REQ1_1: This harness verifies REQ1.1 and the requirement-based test case is present in the signal builder block
  • If 'gearIn' is between 1-5 set 'fwdGearEngaged' as TRUE
• Verify_Requirement_REQ1_2: This harness verifies REQ1.2 and the requirement-based test case is present in the signal builder block
  • If 'gearIn' is outside 1-5 set 'fwdGearEngaged' as FALSE

The output verification from the model (black box) is done using the 'Test Assessment' block. The following snapshot is taken from test harness 'Verify_Requirement_REQ1_1'.

Simulink Test is leveraged to, author and execute the requirement-based test case, the test setup is done as shown below.

• Open Simulink Test Manager (>>sltestmgr)
• Create a new 'Test Suite' and name it as  'Requirement Based Test Suite' 
• Add 2 new Test cases in this 'Test Suite', name these 2 test cases as follows
  • Verify_Requirement_REQ1_1
  • Verify_Requirement_REQ1_2

• Navigate to test case 'Verify_Requirement_REQ1_1' and perform the test case setup as shown below
• In the System Under Test section we need to set the following properties:
• Model: Model Under Test (Simple_RequirementBasedTest.slx)
• Harness: The existing harness with requirements-based tests, set the following harness as 'RBT_Harness_Verify_Requirement_REQ1_1'
  • Note: For test case 'Verify_Requirement_REQ1_2' set harness as 'RBT_Harness_Verify_Requirement_REQ1_2'
• Simulation Mode: As we are performing code-based testing, set it as 'Software-in-the-loop (SIL )'

• In the Inputs section, the Signal builder group is selected. This signal builder group tests associated requirements
• Note: Following is a snapshot from the Verify_Requirement_REQ1_1  test case, the same operation needs to be done for the Verify_Requirement_REQ1_2 test case

• In the Iterations section,  click on autogenerate and add descriptions associated with Iterations (optional). This operation maps Iterations to the Signal builder test cases.
• Note: Following is a snapshot from the Verify_Requirement_REQ1_1  test case, the same operation needs to be done for the Verify_Requirement_REQ1_2 test case

Now run the test and analyze the results. Run the test case in Simulink Test manager by clicking on the Run button or right-clicking the test suite (Requirement Based Test Suite) and selecting run.

Once test execution is done, we can observe that verifications in both of the requirement-based tests are passed.

• Verify_Requirement_REQ1_1 : Iteration 1,  verifies REQ1.1 where we verify fwdGearEngaged==TRUE
• Verify_Requirement_REQ1_2 : Iteration 1,  verifies REQ1.2 where we verify fwdGearEngaged==FALSE

However, we see that the code coverage is not 100% though we have written the requirement-based tests. This implies that the requirement-based tests are not sufficient enough to achieve 100% code coverage.

 

To understand the missing coverage generate the coverage report from Simulink Test results (by clicking the arrow button, as shown in the image above). In the generated coverage report it is observed that:

• Code associated with the relational operator block (>=) is having insufficient condition coverage  because
  • In the existing requirement-based tests, the Input 'gearIn' is never set to a value that is less than 1
  • Hence False outcome of this condition (gearIn>=1) is not covered.
  • This also results in insufficient condition and MCDC coverage for downstream logical operator (AND) block
• Refer to this link to understand more about Simulink Code Coverage Report: https://www.mathworks.com/help/slcoverage/ug/top-level-code-coverage-report.html

As we have figured out the reason behind the insufficient coverage, we can now auto-generate tests for missing coverage using Simulink Design Verifier.  To do so, click on 'Add Tests for Missing Coverage' from Simulink Test Results. 

This operation opens a window, wherein we can select the destination harness and test case to add additional test cases for missing coverage.  In this case, the following selections are done:

• Harness <Create a new harness>: To create a new test harness to cater tests for missing coverage generated by Simulink Design Verifier
• Source <Inport>: Inport harness,  based upon the use case it can be selected to 'Signal builder' or 'Signal Editor' 
• Test File: Use the same Simulink Test file 'Requirement Based Test File' to add a new test case

The following window shows Simulink Design Verifier in action and generating tests for the missing coverage. The following snippet shows Simulink Design Verifier generates code-based tests for 4 coverage objectives that were not covered with the existing requirement-based tests. To view the test cases you can navigate to the 'sldv_output' folder created in the current working directory. Refer to this link for more details on Automatic Code Coverage Test Generation using Simulink Design Verifier: https://www.mathworks.com/help/sldv/ug/code-coverage-test-generation.html

Once SLDV generates tests for missing code coverage, we can observe that the 'Test file' is automatically updated with the new test suite, the harness and other settings are automatically done while importing autogenerated test cases from SLDV.

With this affordance, we can have different test suites for Requirement based and coverage tests.

Now, rerun the test case in Simulink Test Manager and analyze the results. Run the test case in the Simulink Test manager by clicking on the Run button or right-clicking the test file (Requirement Based Test File) and selecting run. As we can see the test is passed and we have achieved 100% code coverage on the model.

• Requirement-Based Test Suite: Has coverage for existing requirement-based test cases
• New Test Suite 1: Has coverage data for missing coverage (autogenerated using SLDV)
• Clicking on the top-level results shows (Requirement Based Test File) shows 100% coverage
• Expanding and clicking of respective test suites (Requirement Based Test Suite, New Test Suite 1) show the contribution of the respective suite in overall aggregated coverage.

The following code snippet shows all covered lines of code.