Simulink Test Iterations Issues when running in Parallel Mode
Show older comments
CONTEXT:
I'm running some Simulink Test harnesses programmatically for control engineering purposes. First, each test use the pre-load call back to load variables and set cache folder to manage simulation artifact. Then it run each set of test either in parallel or in serials.
ACTION:
I tried to add parameter iterations to 1 test in the group.(See #1) It run well in the simulink test manager GUI sequentially. it ran well sequentially programmatically, but the unexpected happen when I tried to run it parallel.
EXPECTED:
It split all test in group and run in parallel pool without any issues like #3 and and show in test results the iterations #4.
RESULT:
It split all test in group and run in parallel pool but it had a fetch next issue like in #2.
Any idea?
Can simulink test iteration be run in parallel mode?
INFO:
1 . To not have a data concurrency issue per test, I do create 1 artifact cache folder per test using simulink test callbacks . but I didn't do so per iteration, I hope it didn't matter and I was a bit lost on how to.But, I think that could be related. Ref:
https://stackoverflow.com/questions/64824205/how-to-fix-conflicting-simulink-simulation-accelerated-artificats-issue-when-run
https://www.mathworks.com/help/simulink/ug/model-reference-simulation-targets-1.html
2 . Here's some literature I read on the subjects
https://www.mathworks.com/help/sltest/ug/run-multiple-combinations-of-tests-using-iterations.html
https://www.mathworks.com/help/sltest/ref/sltest.testmanager.testcase.additeration.html
3 . I'm using the runInParallel command from the matlab.unittest.TestRunner pluggin. I'm also using matlab.unittest.plugins.DiagnosticsRecordingPlugin.
CODE & LOGS:
#1 Here I try to use Parameter Override and Iterations to split 1 test in 2 iteration. All other test are individual test.
for i = 1:6
if regexp(requirement{i},'FOO' ,'once')
for j = 1:2%7
%% Generate Variant Scenarios
Foo= TS_Foo(j);
%% Parameter Set
currParamName = char(strcat("Parameter Set ",string(j)));
ps{i,j} = addParameterSet(tcase{i},'Name',currParamName);
po{i,j} = addParameterOverride(ps{i,j},'A',Foo.A);
po{i,j} = addParameterOverride(ps{i,j},'B',Foo.B);
po{i,j} = addParameterOverride(ps{i,j},'C',Foo.C);
po{i,j} = addParameterOverride(ps{i,j},'D',Foo.D);
%% Iterations
% Create test iteration object
testItr = sltest.testmanager.TestIteration();
% Use the parameter set in this iteration
testItr.setTestParam('ParameterSet', currParamName);
% Add the iteration object to the test case
addIteration(tcase{i},testItr,currParamName);
end
#2 Here I run 6 test with 1 test having 2 iterations in parallel. so it's 7 test total split.
Split tests into 7 groups and running them on 6 workers.
Error using parallel.FevalFuture/fetchNext (line 255)
The function evaluation completed with an error.
Error in
matlab.unittest.internal.TestRunnerExtension/runInParallel>printFinishedOutput
(line 85)
groupIdx = fetchNext(groups);
Error in matlab.unittest.internal.TestRunnerExtension/runInParallel (line 58)
printFinishedOutput(groups, stream);
Error in runTest (line 141)
results = runInParallel(runner,suite);
Caused by:
Error using
matlab.unittest.plugins.testrunprogress.ConciseProgressPlugin/runTestClass
(line 59)
In 'MATLAB:unittest:TestRunProgressPlugin:Running', parameter {0} must be
a scalar.
#3 Here I ran 35 test in parallel with no iterations, without any issue.
I also ran 6 test in parallel, without iteration with no issue.
2021-08-26T18:39:18.9495254Z Split tests into 22 groups and running them on 8 workers.
2021-08-26T19:37:12.5841596Z ans =
2021-08-26T19:37:12.5842363Z
2021-08-26T19:37:12.5842546Z 35x6 table
2021-08-26T19:37:12.5842637Z
2021-08-26T19:37:12.6891297Z Name Passed Failed Incomplete Duration Details
2021-08-26T19:37:12.6913253Z ________________________________________________________________ ______ ______ __________ ________ ____________
2021-08-26T19:37:12.6913549Z
2021-08-26T19:37:12.6919980Z 'API_Test_Files28 > API_Test_Suites/MUT_Harness_****#1' true false false 1824.4 [1x1 struct]
...
...
2021-08-26T19:37:12.6943686Z 'API_Test_Files28 > API_Test_Suites/MUT_Harness_****#35' true false false 323.12 [1x1 struct]
#4 Here I ran the the 6 test plus the iteration in sequential mode.
7×6 table
Name Passed Failed Incomplete Duration Details
_____________________________________________________________________________________________ ______ ______ __________ ________ ____________
'API_Test_Files49 > API_Test_Suites/MUT_Harness_****' true false false 1002.2 [1×1 struct]
'API_Test_Files49 > API_Test_Suites/MUT_Harness_***(TableIteration=ParameterSet1)' true false false 502.1 [1×1 struct]
'API_Test_Files49 > API_Test_Suites/MUT_Harness_***(TableIteration=ParameterSet2)' true false false 277.4 [1×1 struct]
Answers (0)
Categories
Find more on Inputs in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!