polyspace.test.TestResults Class

Namespace: polyspace.test

(Python) Review test results

Since R2024b

Description

This Python^® class contains test results that you obtain after running C/C++ tests on a Polyspace^® Platform project or reading directly from a C/C++ test result file (.pstestr). Use an object of this class to package results as necessary, or generate reports.

Creation

Syntax

testResults = polyspace.test.run(proj)

testResults = polyspace.test.TestResults(testResultsPSTESTRFile)

testResults = polyspace.test.TestResults.fromMRF(testResultsMRFFile)

testResults = polyspace.test.TestResults.fromXML(testResultsXMLFile)

Description

testResults = polyspace.test.run(proj) runs the test executable generated by building the source code and test files in the project proj and stores the results in the results object testResults. For extensions of this syntax, see polyspace.test.run.

testResults = polyspace.test.TestResults(testResultsPSTESTRFile) loads the results stored in the .pstestr file testResultsPSTESTRFile into the results object testResults.

testResults = polyspace.test.TestResults.fromMRF(testResultsMRFFile) loads the results stored in the .mrf file testResultsMRFFile into the results object testResults.

testResults = polyspace.test.TestResults.fromXML(testResultsXMLFile) loads the results stored in the .xml file testResultsXMLFile into the results object testResults.

Input Arguments

proj — Polyspace Platform project

polyspace.project.Project object

Polyspace Platform project, specified as a polyspace.project.Project object.

testResultsPSTESTRFile — Test results as .pstestr file

string

Path to test results in the form of a .pstestr file, specified as a string. This is the default format in which you generate test results:

• When you run tests in the Polyspace Platform user interface. See Open Polyspace Results in Polyspace Platform User Interface.

• When you run tests using the command polyspace-test -run.

Example: "C:\Polyspace_Results\testResults.pstestr"

testResultsMRFFile — Test results as .mrf file

string

Path to test results in the form of a .mrf file, specified as a string. For more information on how to generate results in MRF format, see Machine-Readable Results for Tests Written Using Polyspace Test API.

Example: "C:\Polyspace_Results\testResults.mrf"

testResultsXMLFile — Test results as .xml file

string

Path to test results in the form of a .xml file, specified as a string. You can use the fromXML() method to import XML results of tests authored using one of these frameworks:

• Polyspace Test™ xUnit API. You generate results in XML format using the generateXMLReport() method or using the command polyspace-test -report.

• GoogleTest

Example: "C:\Polyspace_Results\testResults.xml"

Properties

RunLog — Run log information

polyspace.ExecutionLog object

Since R2026a

Run log information, returned as a polyspace.ExecutionLog object. If you have not built the project earlier, the run log contains the results of both project build and test execution. Note that the test execution results can be retrieved using other properties of a test results object and do not need to be extracted from the run log.

This polyspace.ExecutionLog object has the following property:

Property	Description
Content	Run log content, specified as a single string. To display the run log string from a polyspace.test.TestResults object testResults in a human readable format, enter: print(testResults.RunLog.Content)

To save the run log content to a file, use the save() method. For instance, to save the error string from a polyspace.test.TestResults object testResults to a file logs.txt, enter:

testResults.RunLog.save("logs.txt")

Note

This property contains the logs from a successful test execution (irrespective of whether the tests pass, fail or return incomplete results). If you have not built the project prior to test execution, the polyspace.test.run() function first builds the project prior to test execution. If an error occurs during this build, the function throws an exception of type polyspace.ErrorWithLog. Catch this exception in an except block to see the error log content. For more information, see polyspace.ErrorWithLog.

Status — Execution status of tests

polyspace.test.StatusInfo object

Overall execution status of all tests in the results object, returned as a polyspace.test.StatusInfo object with the following properties:

Property	Values	Description
Failed	True or False	The value of this property is True if at least one test fails or encounters a runt-time error or exception. The value can also be True if a test, which is expected to fail, passes instead. For more information on expected failures, see Write C/C++ Tests with Known Errors or Failures to Support Test-Driven Development. In all other cases, the value is False.
Incomplete	True or False	The value of this property is True if at least one test does not run to completion because of an assertion or assumption failure. For more information on assertions and assumptions, see Assessment Macros in Polyspace Test API for C/C++ Code. In all other cases, the value is False.

SuitesFailed — Number of failing test suites

integer

Number of failing test suites. A suite is failing if it contains at least one failed test.

SuitesIncomplete — Number of incomplete test suites

integer

Number of incomplete test suites. A suite is incomplete if it contains at least one incomplete test.

SuitesPassed — Number of passing test suites

integer

Number of passing test suites. A suite is passing if it contains no failed or incomplete test.

SuitesRequested — Total number of test suites

integer

Number of test suites that run during test execution. This number includes passed, failed, and incomplete suites.

TestsFailed — Number of failing tests

integer

Number of failing tests.

TestsIncomplete — Number of incomplete tests

integer

Number of incomplete tests.

TestsPassed — Number of passing tests

integer

Number of passing tests.

TestsRequested — Total number of tests

integer

Number of tests that run during test execution. This number includes passed, failed, and incomplete tests.

Timing — Time of execution and timestamp

polyspace.test.TimingInfo object

Timing of suite or test execution and timestamp of execution, returned as a polyspace.test.TimingInfo object with the following properties:

Property	Value	Description
Timestamp	datetime.datetime value	Time stamp of the beginning of suite or test execution, for example, 2024-Jul-04 10:38:21.
ExecutionTime	Floating-point value	Time taken for suite or test execution in seconds (up to millisecond precision).

TestSuiteResults — Result details per test suite

list of polyspace.test.TestSuiteResult objects

Result details for test suites, returned as a list of polyspace.test.TestSuiteResult objects with one object per test suite. This property is the primary way to drill down into details of test results.

For more information on this object, see polyspace.test.TestSuiteResult.

Messages — Messages emitted during test execution

list of polyspace.test.Message objects

Messages emitted during test execution that are not related to assessments. For instance, if you run a parameterized test with a parameter index that is outside the range of allowed parameter indices, a warning message is emitted as a polyspace.test.Message object with the following properties:

Property	Value
Phase	TestPhase.NONE
Type	MessageType.WARNING The other possible values are: MessageType.NONE MessageType.ERROR MessageType.FATAL_ERROR MessageType.INFO
Msg	Actual content of the message, for instance: Cannot find and/or run test 'pst_default_suite/test_saturate_add_uint', test parameter index number 1 is out of bounds: actual value: 100, maximum value: 9.

Profiling — Code profiling results

polyspace.test.ProfilingResults object

Code profiling results, if you enabled code profiling before building your project and running tests.

For more information on this object, see polyspace.test.ProfilingResults.

Methods

Public Methods

Save results to a .pstestr file, view profiling results if applicable, and generate reports.

Method	Description
getResults()	Get test results for a test suite, a test, or test steps. Example:step_res = testResults.getResults("pst_default_suite", "testName", "stepName")
save()	Save results to a .pstestr file. Example: testResults.save("myResults.pstestr")
generateHTMLReport()	Save the results to HTML reports. The method takes the following arguments: Path to folder where report HTML-s are generated. For instance, to generate reports in the folder path/to/reports, enter: testResults.generateHTMLReport( "path/to/reports" ) To review the HTML reports, start from the file index.html in this folder. (Optional) Whether test specifications must also be reported along with test results using the keyword argument GenerateSpecificationReport. For instance, to generate reports of test specifications, enter: testResults.generateHTMLReport( "path/to/reports", GenerateSpecificationReport=True ) If you omit this argument, only the test results are reported. (Optional) Whether passing assessments must be omitted from the report using the keyword argument ReportPassedAssessments. For instance, to omit passing assessments from reports, enter: testResults.generateHTMLReport( "path/to/reports", ReportPassedAssessments=False ) If you omit this argument, passing assessments are also shown in the test reports. (Optional) Report options file in JSON format. For instance, to customize the layout of the report using a file reportOptions.json, enter: testResults.generateHTMLReport( "path/to/reports", OptionsFile="reportOptions.json" ) For more information on report options files, see Customize Content and Style of Polyspace Test Reports. For an example, see Generate Reports of C/C++ Test Results Using Python API for Polyspace.
generateXMLReport()	Save the results to an XML report. The method takes the following arguments: Full path to the XML report. For instance, to generate a report testreport.xml in the folder path/to/reports, enter: testResults.generateXMLReport( "path/to/reports/testreport.xml" ) (Optional) Whether passing assessments must be omitted from the report using the keyword argument ReportPassedAssessments. For instance, to omit passing assessments from reports, enter: testResults.generateXMLReport( "path/to/reports", ReportPassedAssessments=False ) If you omit this argument, passing assessments are also shown in the test reports.

Examples

Check Overall Status of Tests in Project

This example shows how to check the overall status of test execution after running tests in a project.

In general, you generate and manage Polyspace Test results by using classes from the polyspace.project and polyspace.test modules. Before starting, make sure you can import these modules on a Python shell or in a Python script without errors. For more information, see Set Up Python API for Polyspace.

1. Import the required modules:

   import polyspace.project
   import polyspace.test
   import os

2. Add source files and xUnit test files to a project. This example uses some example source and test files available with a Polyspace Test installation. Instead, you can use your own sources and tests.

   examples_path = os.path.join(polyspace.__install_path__, "polyspace", 
                               "examples", "pstest", "Getting_Started_Example")
   polyspaceProject = polyspace.project.Project("newProject")
   polyspaceProject.Code.Files.add(os.path.join(examples_path, "sources", "utils.c"))
   polyspaceProject.IncludePaths.add(os.path.join(examples_path, "includes"))
   polyspaceProject.Tests.Files.add(os.path.join(examples_path, "tests", "test.c"))

3. Run the tests added to the project.

   res = polyspace.test.run(polyspaceProject)

4. Print the overall status of test execution.

   if not res.Status.Failed and not res.Status.Incomplete: 
       print("All tests have passed.")
   elif res.Status.Failed:
       print("At least one test failed.")
   else:
       print("At least one test did not run to completion.")

Version History

Introduced in R2024b

R2026a: New property that supports reading run logs

Starting in R2026a, a polyspace.test.TestResults object contains a RunLog property with a Content subproperty that contains logs from the run as a string.