Code Coverage Tool
The Code Coverage Tool (CCT) measures C statement coverage in code built
as a Real-Time Workshop S-Function. It operates in conjunction with
lcc (the C-Compiler that ships with the standard MATLAB installation).
CCT inserts instrumentation code into the assembly source generated
by lcc during a normal compile. This additional code logs when a
line is executed in memory on the PC. When a simulation completes,
the line execution counts are sent to the MATLAB base workspace.
The Code Coverage Tool is designed to be used as part of a production
code generation workflow to help measure the completeness of tests
for generated code. It is a common requirement in production applications
that every line of code must be tested. CCT allows software engineers
to measure the effectiveness of tests at the earliest possible point
in the development process. CCT is most powerful when used in conjunction
with the Model Coverage Tool, part of Simulink Verification and Validation.
CCT is intended for use by engineers developing tests for embedded software
generated from Simulink models using Real-Time Workshop Embedded Coder.
An article describing CCT and how it can be used with the Model Coverage
tool for model based design is available in the March 2007 issue of
MATLAB Digest, LINK.
This version of the Code Coverage Tool has been tested with R2006a-R2009a.
CCT requires a Windows 32-bit platform and has only been tested against
Windows XP.
Overview
CCT sits between the mex command and the compiler driver (lcc). A new
compiler driver function (lcccov.exe) is supplied as part of the tool.
This driver has been written in C and compiles using lcc. When cct_setup is
first run, gmake will be called to build the executables lcccov.exe and lcccovlnk.exe.
The C source for the executables is also provided with the tool. A conventional
M-script cannot be used because lcc is normally called from DOS (gmake) or
mex (a perl script). When the compiler driver is called by the mex build
process, it first builds the source code and keeps the intermediate assembly
(.asm). The compiler driver then inserts the necessary instrumentation at
the executable lines in the assembly file and rebuilds the code from the
.asm file. A new linker driver function (lcccovlnk.exe) operates in a
similar way to lcccov. The linker driver itself generates and compiles a new
source file, responsible for collecting all CCT data from the build and
inserting the data into the MATLAB workspace. The output from the simulation
run is written when the model Terminate function executes. All CCT data
will be in the form cct_sourcefile for the count data and
cct_sourcefile_l for the array of line numbers.
CCT provides two command line utilities: cct_setup and html_dump.
For CCT to operate correctly, certain build options must be set.
This script checks the code generation settings on the test harness model
and applies the appropriate settings for CCT. html_dump is a script
that inserts CCT data into a Real-Time Workshop html code generation
report. This allows you to visualise the coverage results alongside
the generated code.
The main sections of this document are:
CCT Commands
The commands described in this section allow you to configure models, generate
code and analyze results:
-
testing that CCT is installed correctly
-
configuring a model for code generation with CCT
-
building the S-function
-
measuring the coverage
-
working with CCT data
Testing that CCT is Installed Correctly
The simplest way to check that CCT has been installed correctly is to type
help cct_setup. If this returns the help text for the setup function
the MATLAB path is correctly configured to find the tool.
Configuring a Model for Code Generation with CCT
The function cct_setup('model_name') is used to apply
the necessary configuration settings to a test harness model called
model_name. This test harness model will usually contain
the component under test and this arrangement is used in the CCT
demo model. When the script applies a configuration setting to a
model, it will echo a message to the command window to document
which settings have been changed. The script will save the model
automatically once it has completed.
The constraints imposed on the test harness by cct_setup
are as follows:
| CCT Constraint |
Reason |
| mex compiler must be lcc |
lcc is only supported compiler |
| System target file must be ert.tlc |
CCT depends on Embedded Coder |
| Must generate ERT S-Function |
This is how the built code is executed on the PC |
| Generate Code Only option must be off |
This would prevent CCT building anything |
| Solver must be fixed step discrete |
Recommended setting for safety related applications |
| Template makefile must be ert_lcc.tmf |
Default (ert_default_tmf) could pick up non-lcc makefile |
| Model must have a terminate function |
CCT runs code here to put data into base workspace |
| Code must zero initialize parameters at start |
States in model would not reset between CCT runs |
| Custom make command |
This hook is used by CCT to configure the build |
| Code will be built as an lcc debug image |
This ensures no C lines are optimised out |
Building the S-function
Once cct_setup as completed, the CCT S-function can be
built in one of three ways:
- Right-Click, Real-Time Workshop -> Subsystem Build
-
Since
cct_setup has imposed the setting "Generate ERT S-Function",
right-clicking on the subsystem under test and selecting "Real-Time Workshop"
then "Build Subsystem" will build the coverage S-Function. This can be reused
in the test harness model.
- Right-Click, Real-Time Workshop -> Generate S-Function
-
This has the same effect as selection "Build Subsystem", but you must check
the box "Use Embedded Coder" for the coverage information to be included.
- Tools -> Real-Time Workshop -> Build Model
-
Since
cct_setup has imposed the setting "Generate ERT S-Function",
this will result in the whole model being wrapped up as an S-Function. Note it
is often undesirable to include test vectors in the built coverage S-function, since
you will have to rebuild or re-parameterize the S-Function each time a new
test case is required.
Measuring the Coverage
When a model that contains a CCT S-Function block is simulated, the coverage
data is sent to the MATLAB base workspace at the end of the simulation.
For every C-source file (sourcefile) in the build directory,
a pair of vectors are created in the MATLAB base workspace. One vector
contains the executable lines from the C source file and is named
cct_sourcefile_l. The second vector contains the execution
counts for each line specified in cct_sourcefile_l. The
execution count variable is named cct_sourcefile.
Note If
the C-source file used in the build #includes other C
source files, CCT will treat all included sourrce as a single file.
This means CCT may present line indexes that appear to be higher than
the number of source lines in a file.
|
Working with CCT Data
A utility called html_dump is provided with CCT to assist
with viewing coverage results. Real-Time Workshop Embedded Coder
provides the facility to generate an html report containing the
generated C source code. html_dump will take coverage
results from the MATLAB base workspace and insert them into the
Real-Time Workshop html report. The syntax for html_dump is as follows:
>> html_dump('sfname','cname')
This follows the path ./SFNAME_ert_rtw/html to find an html
report generated from the file named CNAME.c. The coverage data generated
by cct in the base workspace (cct_CNAME and cct_CNAME_l)
will be inserted into the file CNAME_c.html. The file with
coverage measurement inserted is finally saved as CNAME_cov.html.
The coverage file will be saved alongside the other html report files.
Note CCT inserts
a function call into the model's Terminate Function to write data to the
MATLAB base workspace (__cct_dump();). There is a single
executable line after this function call that will not normally show up
as being covered. This line is the closing brace (})
associated with the model's Terminate Function. html_dump
contains code to correct this count and insert a 1 for the line associated
with the closing brace. html_dump makes this correction in
the MATLAB base workspace. If html_dump is not being used,
the analysis should disregard a count of zero in the last element of
cct_sourcename.
|
The CCT demo model uses M-Script in the model's StopFcn to
run html_dump and launch the results in a web browser:
html_dump('controller_model','controller_model');
web([pwd '\controller_model_ert_rtw\html\controller_model_cov.html']);
CCT can also be used programatically to find gaps in coverage. The following
command returns all source lines that did not execute during a simulation
run:
>> cct_controller_model_l(find(cct_controller_model==0))
ans =
56 58 66
CCT Demo
The CCT demo model can be started by typing
cov_example at the command
prompt. This model is a test harness for an example controller and it
tests the S-function built using CCT alongside the original controller.
The MATLAB Digest article, LINK,
that discusses CCT illustrates a typical workflow using this example model.
Limitations and Restrictions
-
CCT only works on 32 bit Windows platforms. Since the source for
executables is provided, the user could port to another platform
if desired.
-
Code is built as a Real-Time Workshop S-function. No other forms of
build are supported.
-
The path to the tool installation location is inserted in the test
harness model. If using CCT across two different machines with different
install locations for CCT, it will be necessary to re-run cct_setup on
the new machine.
-
The line count variables are unsigned 16 bit integers and will wrap when
they reach 65535.
-
Source file length cannot be more than 65535 lines (also a 16 bit counter).
-
If the user has specified custom terminate code in the RTW build settings,
CCT will overwrite these with the instrumentation functions for model coverage.
-
If the build includes hand-written C files, CCT will measure coverage for
them so long as they are in the same directory as the model.
-
Since hand-written C files do not appear in RTW Code Generation Reports,
it will not be possible to use html_dump to view coverage results. The
coverage data is available in the MATLAB base workspace.
