A Benchmark Problem for Model Based Control System Tests - 001

A benchmark problem for model based control system testing

Today safety critical flight control systems are tested using model based approach. The model blocks are proprietary and seldom shared in the open. A benchmark problem was designed as part of a research activity to test out certain test case generation techniques. This model was also used as a problem for the test case generation methodology training classes. Trainees, normally fresh graduates from colleges, were asked to design manual test cases to find out the errors embedded in the model. The control system blocks are typical of the ones used in a flight control system or an automobile control system. It is a combination of linear filters, integrators, non-linear blocks like rate limiters and lookup table. There is a combination of logic and time dependency in terms of persistence blocks. These are however more complex for generating test cases as explained below. The blocks have been selected and placed to ensure that the students exercise some thought process and understand the underlying functionality of the control system blocks.

This problem is being provided as open source to the Control System test community. This we believe is a first of its kind and we hope to provide more benchmark problems as we go along comparing Taguchi method of testing. Users are free to try out the Matlab or other commercial test tools against this model. We would like to hear about your experience in using this benchmark problem.

Benchmark Problem

The benchmark model is called compete_2010.mdl. This is a Simulink model with a test harness, which takes in 10 inputs from the workspace named Inp1 to Inp10. It has 7 outputs, which come out as a vector in a variable simout (refer Figure 1). There is a Matlab code, which is an exact replica of the model in code form (file model00001.m). This has undergone more than 20,000 test runs and both model and code match very well. There is a variation of this with instrumentation for coverage called model00001_c.m. This file has a variable COVERAGE, which collects the line, condition and logic coverage. There is a Simulink model with 17 mutants injected into the model subsystems called compete.mdl. This has the model and the mutants in the same file. Each mutant block has a single mutant embedded into it. The output of the mutant block and the original model is compared and is available as error in the scope. The model output is Output in workspace and the mutant outputs are available as Output1 to Output17 in the workspace.

Automated mutant generation

The model was used as a benchmark problem to verify random test case generation techniques. A novel method of test case generation using Taguchi was also used to generate test cases. An automated mutant generator was developed for the Simulink and Matlab code. These Matlab scripts generate all combinations of mutants for the Simulink and Matlab code. An OR gate, as an example, would be replaced by an AND gate, XOR gate etc in each mutant file. Each mutant file will have only one mutant. The mutant description is provided as a text file for the Simulink mutant models. The mutant Matlab code has the change description as the first line in the mutant. The script could generate 414 Simulink mutant and 7592 mutants for the Matlab code.