MATLAB Answers

Unit tests fail on "even" runs, but pass on "odd" runs (alternating pass/fail)

6 views (last 30 days)
matquest
matquest on 22 Apr 2020
Commented: matquest on 27 Apr 2020
I have some unit tests that I wrote for code that is doing some very minor image manipulation (combining several small images into a larger image). When I first run the unit tests, I noticed that three out of four of them failed on the line where they are reading an image from a directory (fails with an index out of bounds error). However, if I run it a second time, they all pass. When I was writing the code as well, I noticed that whenever I set a breakpoint in my code, I'd have to run the unit tests twice because the first time it would run through the tests without hitting any breakpoints.
I have my repo organized like this
src/
/* source code .m files are in here */
unit_tests/
images/
squares/
- img1.png
- img2.png
...
- imgn.png
- unit_tests.m
And I have a line in my setup (inside the unit tests) to add paths that I generate:
function tests = unit_tests()
addpath(genpath('..'));
tests = functiontests(localfunctions);
end
At first, I thought that perhaps the unit tests were executing too quickly and it was running the unit tests before the addpath/genpath functions had finished, so I added a pause statement and re-ran the tests, but I had the same issue only this time it would wait for the requisite number of seconds before going ahead and failing. If I run it again, no problem - all my tests pass.
Here's the output of running the unit tests twice (I started on an "odd" run):
>> runtests('unit_tests\unit_tests.m')
Running unit_tests
.......
Done unit_tests
__________
ans =
1×7 TestResult array with properties:
Name
Passed
Failed
Incomplete
Duration
Details
Totals:
7 Passed, 0 Failed, 0 Incomplete.
0.39979 seconds testing time.
>> runtests('unit_tests')
Running unit_tests
..
================================================================================
Error occurred in unit_tests/testCompositeImage_2x2 and it did not run to completion.
---------
Error ID:
---------
'MATLAB:badsubscript'
--------------
Error Details:
--------------
Index exceeds array bounds.
Error in unit_tests>testCompositeImage_2x2 (line 47)
img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
.
================================================================================
Error occurred in unit_tests/testCompositeImage_2x3 and it did not run to completion.
---------
Error ID:
---------
'MATLAB:badsubscript'
--------------
Error Details:
--------------
Index exceeds array bounds.
Error in unit_tests>testCompositeImage_2x3 (line 71)
img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
.
================================================================================
Error occurred in unit_tests/testCompositeImage_3x2 and it did not run to completion.
---------
Error ID:
---------
'MATLAB:badsubscript'
--------------
Error Details:
--------------
Index exceeds array bounds.
Error in unit_tests>testCompositeImage_3x2 (line 95)
img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
...
Done unit_tests
__________
Failure Summary:
Name Failed Incomplete Reason(s)
==================================================================
unit_tests/testCompositeImage_2x2 X X Errored.
------------------------------------------------------------------
unit_tests/testCompositeImage_2x3 X X Errored.
------------------------------------------------------------------
unit_tests/testCompositeImage_3x2 X X Errored.
ans =
1×7 TestResult array with properties:
Name
Passed
Failed
Incomplete
Duration
Details
Totals:
4 Passed, 3 Failed (rerun), 3 Incomplete.
0.0040965 seconds testing time.
The fact that it's failing on basically the first line because it's failing to read anything from the folder leads me to suspect that even when the other 4 tests supposedly pass, they are in fact not running at all. I am completely at a loss as to why this is happening; I am using vanilla matlab (R2018a) and don't have anything fancy going on.

  2 Comments

Geoff Hayes
Geoff Hayes on 23 Apr 2020
matquest - why (in the above output) do you call
runtests('unit_tests\unit_tests.m')
first and then on the second run call as
runtests('unit_tests')
? How is this input used by the runtests function?
matquest
matquest on 23 Apr 2020
I actually just noticed this and it appears to explain the bizarre output. I am just pressing the "Run Tests" button in the matlab editor tab. I have no idea why, but when I press it, it alternates between running the command:
runtests('unit_tests\unit_tests.m')
and
runtests('unit_tests')
Do you have any ideas as to why it might be doing that?

Sign in to comment.

Answers (3)

matquest
matquest on 24 Apr 2020
Edited: matquest on 24 Apr 2020
Leaving this here in case someone else runs into the same issue.
Turns out something about the line:
addpath(genpath('..'));
causes the matlab GUI to go into a weird state where pressing the "Run Tests" button alternates between calling runtests('unit_tests\unit_tests.m'); and runtests('unit_tests'); which in turn cases the tests to alternately pass and fail (run/not run). If anyone knows why this is the case or has any insights into using relative paths in matlab, I would be very interested to learn more.
Replacing that line with the following seems to resolve the issue:
pth = fullfile(fileparts(fileparts(mfilename('fullpath'))),'src'); % goes up one dir and into 'src' => ../src
paths = regexp(genpath(pth), ';', 'split');
for idx = 1:length(paths) - 1 % last element is empty
addpath(paths{idx});
end
Restart matlab and the changes should take effect.

  0 Comments

Sign in to comment.


Steven Lord
Steven Lord on 24 Apr 2020
You add a directory (or multiple directories) to your path but you never restore the path to the state it was in prior to the test running. Since your test is a function-based test you should create setup and teardown functions. The setup function will run before your test does to store the path and add directories to it. The teardown function will run after your test does and restore the path to its original value afterwards. See this documentation page for a description of how to do this.
When or if you decide to convert your tests (or write new tests) as class-based tests you could use a fixture to handle the setup and teardown once it has been applied (with applyFixture) to your test object.

  5 Comments

Show 2 older comments
matquest
matquest on 24 Apr 2020
Is there anything besides the path variable which could explain this behavior? Or has a similar "global" attribute? I just tried resetting the path variable manually between runs (as theoretically this should force it back into the "initial" state) and I am still seeing this strange alternating effect.
Sean de Wolski
Sean de Wolski on 25 Apr 2020
I'm almost certainly willing to bet it's a path issue. I'm also almost certainly willing to bet that using a project to manage the path will fix it.
matquest
matquest on 27 Apr 2020
@Sean de Wolski - Projects appear to have been added in R2019a based on the release notes, so I'm not sure if that's an option for me.

Sign in to comment.


Sean de Wolski
Sean de Wolski on 24 Apr 2020
You should convert your code files and tests into a MATLAB Project which manages the path for you. Then as long as the project is open (and there's a ProjectFixture in the test framework to ensure this) then everything you need is on the path. Additionally, projects make it easy to run tests because you can run all tests in a project and the test framework makes sure everything is set up.

  0 Comments

Sign in to comment.

Products


Release

R2018a