|
|
|
| R2012a Documentation → Polyspace | |
Learn more about Polyspace |
|
| Contents | Index |
The Polyspace Automatic Orange Tester performs dynamic stress tests on unproven code (orange checks) to help you identify run-time errors.
Performing an exhaustive orange review manually can be time consuming. The Automatic Orange Tester saves time by automatically creating test cases for all input variables in orange code, and then dynamically testing the code to find actual run-time errors.
Before you run a verification, select the Automatic Orange Tester through the option -automatic-orange-tester. See Selecting the Automatic Orange Tester. When the software runs the Automatic Orange Tester at the end of static verification, the software might categorize some orange checks as potential run-time errors. See Reviewing Results at Level 0. The verification log indicates whether the Automatic Orange Tester has identified potential run-time errors. For example, the following log states that no orange checks have been selected for Level 0 review, which indicates that the Automatic Tester has not identified potential run-time errors.
... Automatic Orange Tester (AOT) statistics: - Execution status: * Number of executions: 50 ** Successful: 3 ** Failed: 47 - No orange checks selected for Level 0 review. - Execution times: * Fastest run: 00:00:00.2 * Slowest run: 00:00:00.19 ...
The Automatic Orange Tester is only one of a few ways by which the software identifies potential run-time errors.
You can also run the Automatic Orange Tester manually. See Starting the Automatic Orange Tester Manually.
Polyspace verification mathematically analyzes the operations in the code to derive its dynamic properties without actually executing it (see What is Static Verification). Although this verification can identify almost all run-time errors, some operations cannot be proved either true or false because the input values are unknown. The software reports these operations as orange checks in the Results Manager perspective (see What is an Orange Check?).
If you select the Automatic Orange Tester , at the end of the verification Polyspace generates an instrumented version of the source code. For each orange check that could lead to a run-time error, the software generates instrumented code around the orange check. The software compiles the instrumented code and generates binary code. In addition, the software generates randomized test cases based on the input variables. For each test case, the Automatic Orange Tester executes the binary code and records whether the test is a failure. Consider the following example.
int x; x = f(); x = 1 / x; // orange ZDV: division by zero
During static verification, Polyspace determines that the function f() can return values between -10 and 10. Therefore, for each test, the Automatic Orange Tester assigns x to be a random number between -10 and 10. If the number is 0, division by zero occurs, and the Automatic Orange Tester records the failure.
As the Automatic Orange Tester uses a finite number of test cases to analyze the code, there is no guarantee that it will identify a problem in any particular run. Consider an example where a specific variable value causes an error. If no test case uses this value, then the Automatic Orange Tester does not record a failure.
The Automatic Orange Tester creates new randomized test cases for each run. Therefore, there is no guarantee that the results from two separate runs will be the same.
Running more tests increases the chances of finding run-time errors. However, the tests take more time to complete.
You must run a verification with the -automatic-orange-tester option selected, if you want:
The software to run the Automatic Orange Tester at the end of the verification
To manually run the Automatic Orange Tester after the verification
To enable the Automatic Orange Tester:
In the Project Manager perspective, select the Configuration > All Settings tab.
Expand the Polyspace inner settings node.
Select the Automatic Orange Tester check box.
Specify values for the following options:
Number of automatic tests — Total number of test cases that you want to run. Running more tests increases the chances of finding a run-time error, but takes more time to complete. The default is 500. The maximum value that the software supports is 100,000.
Maximum loop iterations — Maximum number of iterations allowed before a loop is considered to be an infinite loop. A larger number of iterations decreases the chances of incorrectly identifying an infinite loop, but takes more time to complete. The default is 1000, which is also the maximum value that the software supports.
Maximum test time — Maximum time (in seconds) allowed for a test before Automatic Orange Tester moves on to the next test. Increasing test time reduces the number of tests that time out, but increases total verification time. The default is 5 seconds. The maximum value that the software supports is 60.
If you ran a verification with the -automatic-orange-tester option selected, you can run the Automatic Orange Tester manually. See Selecting the Automatic Orange Tester.
To start the Automatic Orange Tester:
Open your results in the Results Manager perspective.
On the Results Manager toolbar, click
(Launch the Polyspace
Automatic Orange Tester) to open the Automatic Orange Tester.
In the Test Campaign Configuration window, specify the following parameters:
Number of tests – Specifies the total number of test cases that you want to run. Running more tests increases the chances of finding a run-time error, but also takes more time to complete.
Number of iterations for infinite loops – Specifies the maximum number of loop iterations to perform before the Automatic Orange Tester identifies an infinite loop. A larger number of iterations decreases the chances of incorrectly identifying an infinite loop, but also might take more time to complete.
Per test timeout – Specifies the maximum time that an individual test can run (in seconds) before the Automatic Orange Tester moves on to the next test. Increasing the time limit reduces the number of tests that time out, but can also increase the total verification time.
Click Start to begin testing.
The Automatic Orange Tester generates test cases and runs the dynamic tests.
If you want to stop the testing before it is complete:
Click Stop Current to stop the current test and move on to the next one.
Click Stop All to immediately stop all tests.
When testing is complete, the Automatic Orange Tester displays an overview of the testing results, along with detailed information about each failed test.
The Test Campaign Results window displays overview information about the results of your dynamic tests, including:
Completed tests – The total number of tests completed.
No Polyspace runtime errors detected – The number of tests that did not produce a run-time error.
Total failed – The number of tests that produced a run-time error.
Number of checks/Tests with errors – The number of Polyspace checks that produced at least one failed test, as well as the total number of tests that produced a run-time error.
Timeout – The number of tests that exceeded the specified Per test timeout limit.
Stopped tests – The number of tests that were stopped manually.
Use the Test Campaign Results Window to see an overall assessment of your test results, as well as to decide if you need to increase the Per test timeout value.
The Results table displays detailed information about each failed test to help you identify the cause of the run-time error. This information includes:
The file name, line number, and column in which the error was found.
The type of error that occurred.
The number of test cases in which the error occurred.
You can view more details about any failed test by clicking the appropriate row in the Results table. The Test Case Detail dialog box opens.
The Test Case Detail dialog box displays the portion of the code in which the error occurred, and gives detailed information about why each test case failed. Because the Automatic Orange Tester performs run-time tests, this information includes the actual values that caused the error.
You can use this information to quickly identify the cause of the error, and to determine if there is an actual bug in the code.
The Log window displays a complete list of all the tests which failed, as well as summary information.
You can copy information from the log window to paste into other applications, such as Microsoft Excel.
The log file is also saved in the Polyspace-Instrumented folder
with the following file name:
TestGenerator_day_month_year-time.out
The Automatic Orange Tester allows you to specify ranges for external variables. You can perform run-time tests using real-world values for your variables, rather than randomly selected values.
Setting ranges for your variables reduces the number of tests that fail due to unrealistic data values, allowing you to focus on actual problems, rather than purely theoretical problems. Once you set data ranges, you can export them to a DRS file for use in future verifications, reducing the number of orange checks in your results (see Exporting Data Ranges for Polyspace Verification).
To refine your data ranges:
In the Variables section at the top of the Automatic Orange Tester, identify the variable for which you want to set a data range.
Select Advanced.
In the Edit Values dialog box, set the appropriate values for the variable:
Single Value – Specifies a constant value for the variable.
Range of values, – Specifies a minimum and maximum value for the variable.
Click Next to edit the values for the next variable.
When you have finished setting values, click OK to save your changes and close the Edit Values dialog box.
Click Start to retest the code.
The Automatic Orange Tester generates test cases, runs the tests, and displays the updated results.
The updated results show fewer failed tests, allowing you to focus on any actual code problems.
You can save your Automatic Orange Tester preferences and variable ranges for use in future dynamic testing.
To save your configuration:
Select File > Save.
Enter an appropriate name and click Save.
Your configuration is saved in a .tgf file.
To open a configuration from a previous verification:
Select File > Open.
Select the appropriate .tgf file, then click Open.
When you open a previously saved configuration, the Log window displays any differences in the configuration files. For example:
If a variable does not exist in the new configuration, a warning is displayed.
If the ranges for a variable are no longer valid (if the variable type changes, for example), a warning is displayed and the range is changed to the largest valid range for the new data type (if possible).
Once you have set the data ranges for your variables, you can export them to a Data Range Specifications (DRS) file for use in future Polyspace verifications. Using these data ranges allows you to reduce the number of orange checks identified in the Results Manager perspective.
To export your data ranges:
Set the appropriate values for each variable that you want to specify.
Select File > Export DRS.
Enter an appropriate name and click Save.
The DRS file is saved.
For information on using a DRS file for Polyspace verifications, see Specifying Data Ranges for Variables and Functions (Contextual Verification).
Before using the Automatic Orange Tester, on UNIX, Solaris, or Linux systems, you must configure your compiler and linker options .
Note On Windows systems, the compiler options cannot be modified. You can configure only the library dependencies. |
To set compiler and linker options:
Open the Automatic Orange Tester.
Select Options > Configure.
In the Preferences dialog box, set the appropriate parameters for your compiler.
The Automatic Orange Tester has the following limitations:
The software does not support the following options with -automatic-orange-tester.
-div-round-down
-char-is-16its
-short-is-8bits
In addition, the software does not support global asserts in the code of the form Pst_Global_Assert(A,B) .
Do not specify the following with -automatic-orange-tester:
-target [c18 | tms320c3c | x86_64 | sharc21x61]
-data-range-specification (in global assert mode)
You must use the -target mcpu option together with -pointer-is-32bits.
The software does not support verification of C code that contains calls to the following routines:
va_start
va_arg
va_end
va_copy
setjmp
sigsetjmp
longjmp
siglongjmp
signal
sigset
sighold
sigrelse
sigpause
sigignore
sigaction
sigpending
sigsuspend
sigvec
sigblock
sigsetmask
sigprocmask
siginterrupt
srand
srandom
initstate
setstate
 | Reviewing Orange Checks | Day to Day Use |  |
| © 1984-2012- The MathWorks, Inc. - Site Help - Patents - Trademarks - Privacy Policy - Preventing Piracy - RSS |