MATLAB Answers

Execution rules of SharedTestFixtures for complex dependencies

1 view (last 30 days)
Ilya
Ilya on 13 Jan 2016
Commented: Ilya on 18 Jan 2016
I'm a bit confused about how exactly SharedTestFixtures are executed. Given 3 test files, some of which use SharedTestFixtures, with the file test1.m requesting SharedTestFixtures '1' and '2', test2.m - '2', test3.m - '3', how the fixtures '1', '2' and '3' will be executed for a test suite created from the folder with the three test files?
It seems like all the SharedTestFixtures should be launched at the beginning of the overall testing, if the goal is to avoid repeated setups and teardowns. However, it may happen that, for instance, test1.m and test2.m both need fixture '2', but test2.m doesn't need (or even should avoid) fixture '1'.
The reference shows a very simplistic example in which the advantage is clear, but what are the guidelines in more complex situations?

  0 Comments

Sign in to comment.

Accepted Answer

Andy Campbell
Andy Campbell on 14 Jan 2016
Edited: Andy Campbell on 14 Jan 2016
Hi Ilya,
Great question! First the guiding principle is that shared test fixtures first value robustness/correctness, and second they value sharing as much as possible. Another principle is that tests explicitly declare the shared test fixtures that they need, and the test framework ensures that they get exactly that list of test fixtures set up for the context. This means, for example, that if a test does not have a shared test fixture "declared" but that fixture is setup and running because the previous test required it, then it will be torn down.
Once a fixture needs to be torn down, it needs to be torn down with all the other active fixture in LIFO order in order to ensure that the fixture restoration happens robustly. Otherwise, fixture state can be incorrectly restored. An example of this is shown in the "A House of Order" section of this blog post.
So thus we want to provide the ability to shared fixtures across test files, folders, packages, and indeed entire suites to allow setup that is very expensive to be optimized. However, incorrect software is worse than inefficient software, so we must respect precisely the fixtures that each test states they require, and we must respect that the teardown execute in the reverse order of all shared fixture setup. This means that there are times when fixtures are setup and torn down only to be setup once again. For your example, assuming these three test classes have just one test method each, then you have
  • test1 {sf1, sf2}
  • test2 {sf2}
  • test3 {sf3}
and if you run them in that order then you will
  • setup sf1 & sf2 for test1
  • You'll need to tear down sf1 for test 2, but since sf2 was setup last and we need to teardown in the reverse order of setup, we will need to teardown sf2 in order to be able to teardown sf1. Then we need to setup sf2 again
  • For test3 we will teardown sf2 and setup sf3.
If the test were run in this order:
  • test2
  • test1
  • test3
then the result would be:
  • setup sf2 for test2
  • setup sf1 for test1 (which now has both sf1 and two running as it requires)
  • teardown sf1, then sf2 for test3 which specifies neither, and setup sf3 (and subsequently tear it down when done.
You can see with this approach that the order of the elements in the Test array matters a whole lot, and you can imagine someone doing the following if they want to be really inefficient:
suite = suite(randperm(length(suite)));
This is inefficient because this splits all the test elements, including the various methods of each test class and distributes them randomly around. The result will be lots of expensive setup and teardown code executing. While expensive, however, this approach even in the randperm case is still robust and each test element is provide with precisely the correct environment it requires. We have considered providing a feature to "reorder" the suite to be as efficient as we can make it. Does that sound interesting to you? In your example such a suite reordering would have preferred choosing test2->test1->test3 over test1-test2-test3.

  1 Comment

Ilya
Ilya on 18 Jan 2016
Yes, this is good to optimize the order of tests for the given SharedFixtures, as the users are often unaware of such subtle points and even if they are, optimizing manually can be often tedious

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!