Contents

Analyzing Test Results for a Fault-Tolerant Fuel Control System

This example shows how to access and analyze test results produced by SystemTest™ from the MATLAB® command line. The test results are from a test that validates a fault-tolerant fuel control system modeled in Simulink®.

Loading Test Results

Running a SystemTest TEST-file generates the results for a test. After running a test, test results are immediately available in the MATLAB workspace as a variable named stresults. If the variable has been cleared, test results can be accessed by loading the test results MAT-file.

load demosystest_fuelctrl_results

By loading the test results MAT-file, the stresults variable is made available in the MATLAB workspace.

Test Results Object Overview

The stresults variable is a test results object providing a high-level summary of the test, its results, and any artifacts or reports that may have been associated with the test.

stresults
stresults =

     Test Results Object Summary for 'S:\sandbox\matlab\toolbox\systemtest\systemtestdemos\demosystest_fuelctrl':

            NumberOfIterations: 96
               TestVectorNames: EGOSensor, EngineSpeed, MAPSensor, SpeedSensor, 
                                ThrottleSensor
              SavedResultNames: AvgAirFuel, AvgFuelRate, NSensorsActive, 
                                SimAFRatio, SimFuelRate
                ResultsDataSet: [96x10 dataset]

     There are no Test Vector Groups associated with this test result object.

     There are no artifacts associated with this test result object.

Information such as the test vector and saved result names, as well as the number of test iterations executed, are provided via properties.

stresults.TestVectorNames
ans = 

  Columns 1 through 4

    'EGOSensor'    'EngineSpeed'    'MAPSensor'    'SpeedSensor'

  Column 5

    'ThrottleSensor'

stresults.SavedResultNames
ans = 

  Columns 1 through 4

    'AvgAirFuel'    'AvgFuelRate'    'NSensorsActive'    'SimAFRatio'

  Column 5

    'SimFuelRate'

stresults.NumberOfIterations
ans =

    96

Working with the Test Results Dataset Object

The actual test result data is managed as a dataset object stored in the ResultsDataSet property. Accessing the results data set displays the test vector and saved result values for each test iteration executed.

Each row prefixed with an "I" represents a test iteration, while each column represents the test vectors, followed by the saved results.

ds = stresults.ResultsDataSet;

If there are too many test iterations, displaying all of the test results may not always be manageable. You can limit the view into the test results by specifying the specific test iterations you wish to see.

% View the first 5 iterations for all test vectors and saved results.
% Index into the results using the iteration names.
ds({'I1' 'I2' 'I3' 'I4' 'I5'}, :)
ans = 

          EGOSensor    EngineSpeed    MAPSensor    SpeedSensor    ThrottleSensor
    I1    [1]          [300]          [1]          [1]            [1]           
    I2    [0]          [300]          [1]          [1]            [1]           
    I3    [1]          [400]          [1]          [1]            [1]           
    I4    [0]          [400]          [1]          [1]            [1]           
    I5    [1]          [500]          [1]          [1]            [1]           


          AvgAirFuel       AvgFuelRate     NSensorsActive
    I1    [14.4466]        [1.3302]        [4]           
    I2    [11.8858]        [1.6251]        [3]           
    I3    [14.4283]        [1.5517]        [4]           
    I4    [11.7511]        [1.9158]        [3]           
    I5    [14.4281]        [1.6298]        [4]           


          SimAFRatio                          SimFuelRate        
    I1    [4098x1 Simulink.Timeseries]        [4098x1 double]    
    I2    [4098x1 Simulink.Timeseries]        [4098x1 double]    
    I3    [4084x1 Simulink.Timeseries]        [4084x1 double]    
    I4    [4084x1 Simulink.Timeseries]        [4084x1 double]    
    I5    [4067x1 Simulink.Timeseries]        [4067x1 double]    

% View the last 10 iterations for all test vectors and saved results.
% Index into the results using regular MATLAB indexing.
ds(86:end, :)
ans = 

           EGOSensor    EngineSpeed    MAPSensor    SpeedSensor
    I86    [0]          [300]          [0]          [0]        
    I87    [1]          [400]          [0]          [0]        
    I88    [0]          [400]          [0]          [0]        
    I89    [1]          [500]          [0]          [0]        
    I90    [0]          [500]          [0]          [0]        
    I91    [1]          [600]          [0]          [0]        
    I92    [0]          [600]          [0]          [0]        
    I93    [1]          [700]          [0]          [0]        
    I94    [0]          [700]          [0]          [0]        
    I95    [1]          [800]          [0]          [0]        
    I96    [0]          [800]          [0]          [0]        


           ThrottleSensor    AvgAirFuel    AvgFuelRate     NSensorsActive
    I86    [0]               []            [0.0012]        [0]           
    I87    [0]               []            [0.0012]        [1]           
    I88    [0]               []            [0.0012]        [0]           
    I89    [0]               []            [0.0012]        [1]           
    I90    [0]               []            [0.0012]        [0]           
    I91    [0]               []            [0.0020]        [1]           
    I92    [0]               []            [0.0020]        [0]           
    I93    [0]               []            [0.0020]        [1]           
    I94    [0]               []            [0.0020]        [0]           
    I95    [0]               []            [0.0024]        [1]           
    I96    [0]               []            [0.0024]        [0]           


           SimAFRatio                          SimFuelRate        
    I86    [4098x1 Simulink.Timeseries]        [4098x1 double]    
    I87    [4084x1 Simulink.Timeseries]        [4084x1 double]    
    I88    [4084x1 Simulink.Timeseries]        [4084x1 double]    
    I89    [4067x1 Simulink.Timeseries]        [4067x1 double]    
    I90    [4067x1 Simulink.Timeseries]        [4067x1 double]    
    I91    [4007x1 Simulink.Timeseries]        [4007x1 double]    
    I92    [4007x1 Simulink.Timeseries]        [4007x1 double]    
    I93    [4009x1 Simulink.Timeseries]        [4009x1 double]    
    I94    [4009x1 Simulink.Timeseries]        [4009x1 double]    
    I95    [4013x1 Simulink.Timeseries]        [4013x1 double]    
    I96    [4013x1 Simulink.Timeseries]        [4013x1 double]    

Accessing the Test Results Data

In the previous examples, the results were in the form of a dataset object. To access the test results in a non-object form, a dot indexing syntax is provided.

% Access the engine speed test vector values for each iteration.
engSpd = ds.EngineSpeed'
engSpd = 

  Columns 1 through 8

    [300]    [300]    [400]    [400]    [500]    [500]    [600]    [600]

  Columns 9 through 16

    [700]    [700]    [800]    [800]    [300]    [300]    [400]    [400]

  Columns 17 through 24

    [500]    [500]    [600]    [600]    [700]    [700]    [800]    [800]

  Columns 25 through 32

    [300]    [300]    [400]    [400]    [500]    [500]    [600]    [600]

  Columns 33 through 40

    [700]    [700]    [800]    [800]    [300]    [300]    [400]    [400]

  Columns 41 through 48

    [500]    [500]    [600]    [600]    [700]    [700]    [800]    [800]

  Columns 49 through 56

    [300]    [300]    [400]    [400]    [500]    [500]    [600]    [600]

  Columns 57 through 64

    [700]    [700]    [800]    [800]    [300]    [300]    [400]    [400]

  Columns 65 through 72

    [500]    [500]    [600]    [600]    [700]    [700]    [800]    [800]

  Columns 73 through 80

    [300]    [300]    [400]    [400]    [500]    [500]    [600]    [600]

  Columns 81 through 88

    [700]    [700]    [800]    [800]    [300]    [300]    [400]    [400]

  Columns 89 through 96

    [500]    [500]    [600]    [600]    [700]    [700]    [800]    [800]

% Note, the data returned is in the form of a cell array.
whos engSpd
  Name        Size            Bytes  Class    Attributes

  engSpd      1x96            11520  cell               

Providing test results as a cell array allows for "holes" in the data set, which highlights the case where a test may not have assigned a value to a result.

For example, in this test, the average air/fuel ratio is only measured if the system under test has not entered a shutdown mode, which is represented in the TEST-file by a series of IF and STOP elements. Viewing the average air/fuel ratio results, notice "holes" for a number of test iterations.

avgAirFuel = ds.AvgAirFuel'
avgAirFuel = 

  Columns 1 through 6

    [14.4466]    [11.8858]    [14.4283]    [11.7511]    [14.4281]    [11.6776]

  Columns 7 through 14

    [14.4196]    [11.6281]    []    []    []    []    [11.7683]    []

  Columns 15 through 23

    [11.5788]    []    [11.4980]    []    [11.4705]    []    []    []    []

  Columns 24 through 31

    []    [11.5422]    []    [11.5138]    []    [11.5033]    []    [11.4782]

  Columns 32 through 44

    []    []    []    []    []    []    []    []    []    []    []    []    []

  Columns 45 through 53

    []    []    []    []    [11.8829]    []    [11.7481]    []    [11.6745]

  Columns 54 through 65

    []    [11.6248]    []    []    []    []    []    []    []    []    []    []

  Columns 66 through 78

    []    []    []    []    []    []    []    []    []    []    []    []    []

  Columns 79 through 91

    []    []    []    []    []    []    []    []    []    []    []    []    []

  Columns 92 through 96

    []    []    []    []    []

Visualizing Test Results

In order to work with the test results, you may need to convert the test results data to their native form. You can do this using the cell2mat function in MATLAB.

engSpd = cell2mat(ds.EngineSpeed)';
whos engSpd
  Name        Size            Bytes  Class     Attributes

  engSpd      1x96              768  double              

If you suspect or know there are "holes" in the data for a given test result, it is helpful to determine the iterations where these gaps occurred before converting the data into its native form.

% If managing "holes" in the data is not a concern, this step is not
% necessary.
emptyIters = cellfun(@isempty, ds.AvgAirFuel);

% Note the length of the variable after CELL2MAT is called. The "holes"
% in the data are automatically removed.
avgAirFuel = cell2mat(ds.AvgAirFuel)';
whos avgAirFuel
  Name            Size            Bytes  Class     Attributes

  avgAirFuel      1x20              160  double              

To plot the engine speed versus the average air/fuel ratio, you can use the scatter function in MATLAB. In order to properly plot test results with their corresponding test vector values, we should only plot the engine speed values for the iterations where we have an average air/fuel measurement.

scatter(engSpd(~emptyIters), avgAirFuel)
xlabel('Engine Speed (rad/sec)')
ylabel('Avg. Air/Fuel')

Verifying System Requirements

One of the requirements for our system under test is that it must shut down if one of these scenarios occur:

  • 1 or more of the 4 sensors become disabled

  • The engine is operating at high speeds (~700 rad/sec)

This requirement can be verified by plotting the fuel rate consumption against engine speed and the number of active sensors.

% First extract some additional test results.
numSensorsActive = cell2mat(ds.NSensorsActive);
avgFuelRate = cell2mat(ds.AvgFuelRate);
% For each sensor scenario, plot the engine speed vs. average fuel rate.
for nSensorsActive = 1:4
    indices = find(numSensorsActive==nSensorsActive);
    subplot(2, 2, nSensorsActive);
    scatter(engSpd(indices), avgFuelRate(indices))
    title(sprintf('%d Active Sensor(s)', nSensorsActive))
    xlabel('Engine Speed (rad/sec)')
    ylabel('Fuel Rate')
end

Note in the case where 2 or 3 sensors are active, there is visible fuel rate consumption by the system at high engine speeds. This violates the requirement that the system shut down at high engine speeds, highlighting a flaw in the design of our system.

Was this topic helpful?