TAP results do not show up during execution time, if a TestClassSetup is present

Hi there

I have a problem. We use the Matlab testing framework to analyze our codebase. To track the results in our CI system TeamCity we use the TAP-format. Here we have the following problem:

If a test includes a TestClassSetup section, the TAP results show up only at the end, and not during the exection. This results in a few issues for us: - Timestamps might not be correct - If informative output is given per test-case, it is not shown together with the assertion statment.

We use the following (simplified) snippet to identify out TestSuite and execute it:

 testSuite = matlab.unittest.TestSuite.fromFolder('.');
 runner = matlab.unittest.TestRunner.withNoPlugins();
 runner.addPlugin(matlab.unittest.plugins.TAPPlugin.producingOriginalFormat());
 results = runner.run(testSuite);

With the following two classes the issue is reproducible (the content is of course made up & meaningless...):

 classdef SomeTest < matlab.unittest.TestCase
    properties (TestParameter)
        param = {1, 2};
        param2 = {1, 2};
    end        
    methods (TestClassSetup)
        function someSetup(testCase)
            pause(0.1);
        end
    end    
    methods (Test)
        function testMethod(self, param, param2)
           fprintf('I''m here, with the params: %f/%f\n', param, param2);
           pause(0.1);
           self.assertGreaterThan(param, param2); 
        end
    end    
 end

 classdef SomeOtherTest < matlab.unittest.TestCase
    properties (TestParameter)
        param = {1, 2};
        param2 = {1, 2};
    end    

    methods (Test)
        function testMethod(self, param, param2)
           fprintf('I''m here, with the params: %f/%f\n', param, param2);
           pause(0.1);
           self.assertGreaterThan(param, param2); 
        end
    end

 end

If you copy all three files into one folder, and execute the runner, you'll see the output (assertions are simplified):

 1..8
 I'm here, with the params: 1.000000/1.000000
 not ok 1 - SomeOtherTest/testMethod(param=1,param2=1)
 # ================================================================================
 # Assertion failed in SomeOtherTest/testMethod(param=1,param2=1) and it did not run to completion.
 # ================================================================================
 # 
 I'm here, with the params: 1.000000/2.000000
 not ok 2 - SomeOtherTest/testMethod(param=1,param2=2)
 # ================================================================================
 # Assertion failed in SomeOtherTest/testMethod(param=1,param2=2) and it did not run to completion.
 # ================================================================================
 # 
 I'm here, with the params: 2.000000/1.000000
 ok 3 - SomeOtherTest/testMethod(param=2,param2=1)
 I'm here, with the params: 2.000000/2.000000
 not ok 4 - SomeOtherTest/testMethod(param=2,param2=2)
 # ================================================================================
 # Assertion failed in SomeOtherTest/testMethod(param=2,param2=2) and it did not run to completion.
 # ================================================================================
 # 
 I'm here, with the params: 1.000000/1.000000
 I'm here, with the params: 1.000000/2.000000
 I'm here, with the params: 2.000000/1.000000
 I'm here, with the params: 2.000000/2.000000
 not ok 5 - SomeTest/testMethod(param=1,param2=1)
 # ================================================================================
 # Assertion failed in SomeTest/testMethod(param=1,param2=1) and it did not run to completion.
 # ================================================================================
 # 
 not ok 6 - SomeTest/testMethod(param=1,param2=2)
 # ================================================================================
 # Assertion failed in SomeTest/testMethod(param=1,param2=2) and it did not run to completion.
 # ================================================================================
 # 
 ok 7 - SomeTest/testMethod(param=2,param2=1)
 not ok 8 - SomeTest/testMethod(param=2,param2=2)
 # ================================================================================
 # Assertion failed in SomeTest/testMethod(param=2,param2=2) and it did not run to completion.
 # ================================================================================

What I would expect is that also in the second case the Assertion statements (and the ok / not ok TAP flags) are aligned with the fprintf-statements.

Has anyone an idea?

Answer 1

[Andy Campbell]

This was also asked and answered on stack overflow:

https://stackoverflow.com/questions/51760637/tap-results-do-not-show-up-during-execution-time-if-a-testclasssetup-is-present/51775630#51775630