Store 
SystemTest 2.4
Analyzing Test Results for a Fault-Tolerant Fuel Control System
This demo illustrates how test results produced by SystemTest™ can be accessed and analyzed from the MATLAB® command line. The test results used are from a test that validates a fault-tolerant fuel control system modeled in Simulink®.
Contents
Loading Test Results
Results for a test are generated by running a SystemTest TEST-file. After running a test, test results are immediately available in the MATLAB workspace as a variable named stresults. If the variable has been cleared, test results can be accessed by loading the test results MAT-file.
load demosystest_fuelctrl_results
By loading the test results MAT-file, the stresults variable is made available in the MATLAB workspace.
Test Results Object Overview
The stresults variable is a test results object providing a high-level summary of the test, its results, and any artifacts or reports that may have been associated with the test.
stresults
stresults =
Test Results Object Summary for 'demosystest_fuelctrl':
NumberOfIterations: 96
TestVectorNames: EGOSensor, EngineSpeed, MAPSensor, SpeedSens
or,
ThrottleSensor
SavedResultNames: AvgAirFuel, AvgFuelRate, NSensorsActive,
SimAFRatio, SimFuelRate
ResultsDataSet: [96x10 dataset]
There are no Test Vector Groups associated with this test result object
.
There are no artifacts associated with this test result object.
Information such as the test vector and saved result names, as well as the number of test iterations executed, are provided via properties.
stresults.TestVectorNames
ans =
Columns 1 through 4
'EGOSensor' 'EngineSpeed' 'MAPSensor' 'SpeedSensor'
Column 5
'ThrottleSensor'
stresults.SavedResultNames
ans =
Columns 1 through 4
'AvgAirFuel' 'AvgFuelRate' 'NSensorsActive' 'SimAFRatio'
Column 5
'SimFuelRate'
stresults.NumberOfIterations
ans =
96
Working with the Test Results Dataset Object
The actual test result data is managed as a dataset object stored in the ResultsDataSet property. Accessing the results data set will display the test vector and saved result values for each test iteration executed.
Each row prefixed with an "I" represents a test iteration, while each column represents the test vectors, followed by the saved results.
ds = stresults.ResultsDataSet;
Displaying all the test results may not always be manageable if there are too many test iterations. You can limit the view into the test results by specifying the specific test iterations you wish to see.
% View the first 5 iterations for all test vectors and saved results. % Index into the results using the iteration names. ds({'I1' 'I2' 'I3' 'I4' 'I5'}, :)
ans =
EGOSensor EngineSpeed MAPSensor SpeedSensor
I1 [1] [300] [1] [1]
I2 [0] [300] [1] [1]
I3 [1] [400] [1] [1]
I4 [0] [400] [1] [1]
I5 [1] [500] [1] [1]
ThrottleSensor AvgAirFuel AvgFuelRate NSensorsActive
I1 [1] [14.4466] [1.3302] [4]
I2 [1] [11.8858] [1.6251] [3]
I3 [1] [14.4283] [1.5517] [4]
I4 [1] [11.7511] [1.9158] [3]
I5 [1] [14.4281] [1.6298] [4]
SimAFRatio SimFuelRate
I1 [4098x1 Simulink.Timeseries] [4098x1 double]
I2 [4098x1 Simulink.Timeseries] [4098x1 double]
I3 [4084x1 Simulink.Timeseries] [4084x1 double]
I4 [4084x1 Simulink.Timeseries] [4084x1 double]
I5 [4067x1 Simulink.Timeseries] [4067x1 double]
% View the last 10 iterations for all test vectors and saved results. % Index into the results using regular MATLAB indexing. ds(86:end, :)
ans =
EGOSensor EngineSpeed MAPSensor SpeedSensor
I86 [0] [300] [0] [0]
I87 [1] [400] [0] [0]
I88 [0] [400] [0] [0]
I89 [1] [500] [0] [0]
I90 [0] [500] [0] [0]
I91 [1] [600] [0] [0]
I92 [0] [600] [0] [0]
I93 [1] [700] [0] [0]
I94 [0] [700] [0] [0]
I95 [1] [800] [0] [0]
I96 [0] [800] [0] [0]
ThrottleSensor AvgAirFuel AvgFuelRate NSensorsActive
I86 [0] [] [0.0012] [0]
I87 [0] [] [0.0012] [1]
I88 [0] [] [0.0012] [0]
I89 [0] [] [0.0012] [1]
I90 [0] [] [0.0012] [0]
I91 [0] [] [0.0020] [1]
I92 [0] [] [0.0020] [0]
I93 [0] [] [0.0020] [1]
I94 [0] [] [0.0020] [0]
I95 [0] [] [0.0024] [1]
I96 [0] [] [0.0024] [0]
SimAFRatio SimFuelRate
I86 [4098x1 Simulink.Timeseries] [4098x1 double]
I87 [4084x1 Simulink.Timeseries] [4084x1 double]
I88 [4084x1 Simulink.Timeseries] [4084x1 double]
I89 [4067x1 Simulink.Timeseries] [4067x1 double]
I90 [4067x1 Simulink.Timeseries] [4067x1 double]
I91 [4007x1 Simulink.Timeseries] [4007x1 double]
I92 [4007x1 Simulink.Timeseries] [4007x1 double]
I93 [4009x1 Simulink.Timeseries] [4009x1 double]
I94 [4009x1 Simulink.Timeseries] [4009x1 double]
I95 [4013x1 Simulink.Timeseries] [4013x1 double]
I96 [4013x1 Simulink.Timeseries] [4013x1 double]
Accessing the Test Results Data
In the previous examples, the results returned were in the form of a dataset object. To access the test results in a non-object form, a dot indexing syntax is provided.
% Access the engine speed test vector values for each iteration.
engSpd = ds.EngineSpeed'
engSpd =
Columns 1 through 8
[300] [300] [400] [400] [500] [500] [600] [600]
Columns 9 through 16
[700] [700] [800] [800] [300] [300] [400] [400]
Columns 17 through 24
[500] [500] [600] [600] [700] [700] [800] [800]
Columns 25 through 32
[300] [300] [400] [400] [500] [500] [600] [600]
Columns 33 through 40
[700] [700] [800] [800] [300] [300] [400] [400]
Columns 41 through 48
[500] [500] [600] [600] [700] [700] [800] [800]
Columns 49 through 56
[300] [300] [400] [400] [500] [500] [600] [600]
Columns 57 through 64
[700] [700] [800] [800] [300] [300] [400] [400]
Columns 65 through 72
[500] [500] [600] [600] [700] [700] [800] [800]
Columns 73 through 80
[300] [300] [400] [400] [500] [500] [600] [600]
Columns 81 through 88
[700] [700] [800] [800] [300] [300] [400] [400]
Columns 89 through 96
[500] [500] [600] [600] [700] [700] [800] [800]
% Note, the data returned is in the form of a cell array. whos engSpd
Name Size Bytes Class Attributes engSpd 1x96 11520 cell
Providing test results as a cell array allows for "holes" in the data set, highlighting the case where a test may not have assigned a value to a result.
For example, in this test, the average air/fuel ratio is only measured if the system under test has not entered a shutdown mode, which is represented in the TEST-file by a series of IF and STOP elements. Viewing the average air/fuel ratio results, you will notice "holes" for a number of test iterations.
avgAirFuel = ds.AvgAirFuel'
avgAirFuel =
Columns 1 through 5
[14.4466] [11.8858] [14.4283] [11.7511] [14.4281]
Columns 6 through 12
[11.6776] [14.4196] [11.6281] [] [] [] []
Columns 13 through 19
[11.7683] [] [11.5788] [] [11.4980] [] [11.4705]
Columns 20 through 28
[] [] [] [] [] [11.5422] [] [11.5138] []
Columns 29 through 37
[11.5033] [] [11.4782] [] [] [] [] [] []
Columns 38 through 48
[] [] [] [] [] [] [] [] [] [] [
]
Columns 49 through 55
[11.8829] [] [11.7481] [] [11.6745] [] [11.6248]
Columns 56 through 66
[] [] [] [] [] [] [] [] [] [] [
]
Columns 67 through 77
[] [] [] [] [] [] [] [] [] [] [
]
Columns 78 through 88
[] [] [] [] [] [] [] [] [] [] [
]
Columns 89 through 96
[] [] [] [] [] [] [] []
Visualizing Test Results
In order to work with the test results, you may need to convert the test results data to their native form. This can be easily done using the cell2mat function in MATLAB.
engSpd = cell2mat(ds.EngineSpeed)';
whos engSpd
Name Size Bytes Class Attributes engSpd 1x96 768 double
If you suspect or know there are "holes" in the data for a given test result, determining the iterations where these gaps occurred may be helpful before converting the data into its native form.
% If managing "holes" in the data is not of concern, this step is not % necessary. emptyIters = cellfun(@isempty, ds.AvgAirFuel); % Note the length of the variable after CELL2MAT is called. The "holes" % in the data are automatically removed. avgAirFuel = cell2mat(ds.AvgAirFuel)'; whos avgAirFuel
Name Size Bytes Class Attributes avgAirFuel 1x20 160 double
To plot the engine speed versus the average air/fuel ratio, you can use the scatter function in MATLAB. Note, in order to properly plot test results with their corresponding test vector values, we should only plot the engine speed values for the iterations where we have an average air/fuel measurement.
scatter(engSpd(~emptyIters), avgAirFuel) xlabel('Engine Speed (rad/sec)') ylabel('Avg. Air/Fuel')
Verifying System Requirements
One of the requirements for our system under test is that it must shut down if one of these scenarios occur:
- 1 or more of the 4 sensors become disabled
- The engine is operating at high speeds (~700 rad/sec)
This requirement can be verified by plotting the fuel rate consumption against engine speed and the number of active sensors.
% First extract some additional test results.
numSensorsActive = cell2mat(ds.NSensorsActive);
avgFuelRate = cell2mat(ds.AvgFuelRate);
% For each sensor scenario, plot the engine speed vs. average fuel rate. for nSensorsActive = 1:4 indices = find(numSensorsActive==nSensorsActive); subplot(2, 2, nSensorsActive); scatter(engSpd(indices), avgFuelRate(indices)) title(sprintf('%d Active Sensor(s)', nSensorsActive)) xlabel('Engine Speed (rad/sec)') ylabel('Fuel Rate') end
Note in the case where 2 or 3 sensors are active, there is visible fuel rate consumption by the system at high engine speeds. This violates the requirement that the system shut down at high engine speeds, highlighting a flaw in the design of our system.
