Simulink Test Iterations Issues when running in Parallel Mode

CONTEXT:
I'm running some Simulink Test harnesses programmatically for control engineering purposes. First, each test use the pre-load call back to load variables and set cache folder to manage simulation artifact. Then it run each set of test either in parallel or in serials.
ACTION:
I tried to add parameter iterations to 1 test in the group.(See #1) It run well in the simulink test manager GUI sequentially. it ran well sequentially programmatically, but the unexpected happen when I tried to run it parallel.
EXPECTED:
It split all test in group and run in parallel pool without any issues like #3 and and show in test results the iterations #4.
RESULT:
It split all test in group and run in parallel pool but it had a fetch next issue like in #2.
Any idea?
Can simulink test iteration be run in parallel mode?
INFO:
1 . To not have a data concurrency issue per test, I do create 1 artifact cache folder per test using simulink test callbacks . but I didn't do so per iteration, I hope it didn't matter and I was a bit lost on how to.But, I think that could be related. Ref:
https://stackoverflow.com/questions/64824205/how-to-fix-conflicting-simulink-simulation-accelerated-artificats-issue-when-run
https://www.mathworks.com/help/simulink/ug/model-reference-simulation-targets-1.html
2 . Here's some literature I read on the subjects
https://www.mathworks.com/help/sltest/ug/run-multiple-combinations-of-tests-using-iterations.html
https://www.mathworks.com/help/sltest/ref/sltest.testmanager.testcase.additeration.html
3 . I'm using the runInParallel command from the matlab.unittest.TestRunner pluggin. I'm also using matlab.unittest.plugins.DiagnosticsRecordingPlugin.
CODE & LOGS:
#1 Here I try to use Parameter Override and Iterations to split 1 test in 2 iteration. All other test are individual test.
for i = 1:6
if regexp(requirement{i},'FOO' ,'once')
for j = 1:2%7
%% Generate Variant Scenarios
Foo= TS_Foo(j);
%% Parameter Set
currParamName = char(strcat("Parameter Set ",string(j)));
ps{i,j} = addParameterSet(tcase{i},'Name',currParamName);
po{i,j} = addParameterOverride(ps{i,j},'A',Foo.A);
po{i,j} = addParameterOverride(ps{i,j},'B',Foo.B);
po{i,j} = addParameterOverride(ps{i,j},'C',Foo.C);
po{i,j} = addParameterOverride(ps{i,j},'D',Foo.D);
%% Iterations
% Create test iteration object
testItr = sltest.testmanager.TestIteration();
% Use the parameter set in this iteration
testItr.setTestParam('ParameterSet', currParamName);
% Add the iteration object to the test case
addIteration(tcase{i},testItr,currParamName);
end
#2 Here I run 6 test with 1 test having 2 iterations in parallel. so it's 7 test total split.
Split tests into 7 groups and running them on 6 workers.
Error using parallel.FevalFuture/fetchNext (line 255)
The function evaluation completed with an error.
Error in
matlab.unittest.internal.TestRunnerExtension/runInParallel>printFinishedOutput
(line 85)
groupIdx = fetchNext(groups);
Error in matlab.unittest.internal.TestRunnerExtension/runInParallel (line 58)
printFinishedOutput(groups, stream);
Error in runTest (line 141)
results = runInParallel(runner,suite);
Caused by:
Error using
matlab.unittest.plugins.testrunprogress.ConciseProgressPlugin/runTestClass
(line 59)
In 'MATLAB:unittest:TestRunProgressPlugin:Running', parameter {0} must be
a scalar.
#3 Here I ran 35 test in parallel with no iterations, without any issue.
I also ran 6 test in parallel, without iteration with no issue.
2021-08-26T18:39:18.9495254Z Split tests into 22 groups and running them on 8 workers.
2021-08-26T19:37:12.5841596Z ans =
2021-08-26T19:37:12.5842363Z
2021-08-26T19:37:12.5842546Z 35x6 table
2021-08-26T19:37:12.5842637Z
2021-08-26T19:37:12.6891297Z Name Passed Failed Incomplete Duration Details
2021-08-26T19:37:12.6913253Z ________________________________________________________________ ______ ______ __________ ________ ____________
2021-08-26T19:37:12.6913549Z
2021-08-26T19:37:12.6919980Z 'API_Test_Files28 > API_Test_Suites/MUT_Harness_****#1' true false false 1824.4 [1x1 struct]
...
...
2021-08-26T19:37:12.6943686Z 'API_Test_Files28 > API_Test_Suites/MUT_Harness_****#35' true false false 323.12 [1x1 struct]
#4 Here I ran the the 6 test plus the iteration in sequential mode.
7×6 table
Name Passed Failed Incomplete Duration Details
_____________________________________________________________________________________________ ______ ______ __________ ________ ____________
'API_Test_Files49 > API_Test_Suites/MUT_Harness_****' true false false 1002.2 [1×1 struct]
'API_Test_Files49 > API_Test_Suites/MUT_Harness_***(TableIteration=ParameterSet1)' true false false 502.1 [1×1 struct]
'API_Test_Files49 > API_Test_Suites/MUT_Harness_***(TableIteration=ParameterSet2)' true false false 277.4 [1×1 struct]

Answers (0)

Products

Release

R2017b

Asked:

on 31 Aug 2021

Edited:

on 31 Aug 2021

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!