Reinforcement Learning multiple agent validation: Can I have a Simulink model host TWO agents and test them

Hi,

I am conducting research to see how PPO performs versus DDPG - for non-linear plants. I have trained two agents.

Can I have a Simulink circuit host TWO agents and test them? Basically trying to create a unified validation bench. Please see image below.

I did go through the documentation and tried this implementation but am getting errors.

Code:

% Set Simulink model pointers
PLANT_SIMULATION_MODEL = 'sm_Experimental_Setup'; % Simulink experimentation circuit
DDPG_AGENT = '/DDPG Sub-System/DDPG_Agent';
PPO_AGENT = '/PPO Sub-System/PPO_Agent';

% Load experiences from pre-trained agent    
DDPG_agent = load(DDPG_MODEL_FILE,'agent');
PPO_agent = load(PPO_MODEL_FILE,'agent');

% Code here for setting (1) obsInfo and (2) actionInfo_DDPG and (3) actionInfo_PPO
% .... ... 

% Intialise the environment with the serialised agent and run the test
env = rlSimulinkEnv(VALVE_SIMULATION_MODEL, [DDPG_AGENT PPO_AGENT], [obsInfo obsInfo], [actionInfo_DDPG actionInfo_PPO]);
simOpts = rlSimulationOptions('MaxSteps', 2000);
xpr = sim(env,[DDPG_agent.agent, PPO_agent.agent]);

ERROR message:

Error using rlSimulinkEnv (line 108)
No block diagram name specified.

Error in code_DDPG_PPO_Experimental_Setup (line 97)
env = rlSimulinkEnv(VALVE_SIMULATION_MODEL, [DDPG_AGENT PPO_AGENT], [obsInfo obsInfo], [actionInfo_DDPG actionInfo_PPO]);

Screen capture of Simulink model:

Answer 1

[Emmanouil Tzorakoleftherakis]

That should be possible. Did you follow the multi-agent examples? Since the agents are trained already you may want to check the last part in the links below where agents are simulated.

https://www.mathworks.com/help/reinforcement-learning/ug/train-agents-for-path-following.html

https://www.mathworks.com/help/reinforcement-learning/ug/train-3-agents-for-area-coverage.html

By the way, the error you are getting makes me think the the path you provide below is not complete

DDPG_AGENT = '/DDPG Sub-System/DDPG_Agent';
PPO_AGENT = '/PPO Sub-System/PPO_Agent';

Try adding the model name too.

Hope this helps

[Rajesh Siraskar]

Thanks Emmanouil.

Those links helped a lot. There were some learnings like - observation and action information arrays need to be "cell arrays" for it to work - so I had to use num2cell(ObservationInfo, 1) for it to work.

However: I am still not able to get it to work - the path names are still something that I am not able to figure out. I tried the combinations below. But I still get the error. Do you have any suggestions?

VALVE_SIMULATION_MODEL = "sm_DDPG_PPO_Experimental_Setup";

% OPTIONs tried
DDPG_AGENT = "/DDPG_Sub_System/DDPG_Agent/";
DDPG_AGENT = "/DDPG_Sub_System/DDPG_Agent/DDPG_agent";
DDPG_AGENT = "/DDPG_Sub_System/DDPG_Agent/DDPG_agent.agent";

[Emmanouil Tzorakoleftherakis]

Based on what I see on the screenshot it should be something along the lines of 'sm_DDPG_PPO_Experimental_Setup/DDPG_Sub_System/DDPG_Agent'

I suggest taking a look at any of the Simulink examples in Reinforcement Learning Toolbox to see how the path is defined. Even the links above should work. The first thing in the path should be the model name.

[Rajesh Siraskar]

Thank you Emmanouil - that worked.