Reinforcement learning deployment in real-time system

Hello everyone,

I trained an agent on my problem, and it works really well in simulation in Simulink. I'm trying to implement the same agent in the real-time target; for that reason, I need to build my Simulink file to generate code and then upload it to my hardware. Unfortunately, I got an error in building my model. I was wondering if anyone can help me with this.

### Generating code into build folder: G:\My Drive\00Research\RL\QUBE_DC_control\Deployment\DC_motor_RL_v02_exp_grt_rtw

Dot indexing is not supported for variables of this type. Function 'evaluatePolicy.m' (#38.286.341), line 13, column 11: "coder.loadDeepLearningNetwork('agentData.mat','policy')" Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Persistent variable 'policy' must be assigned before it is used. The only exception is a check using 'isempty(policy)' that can be performed prior to assignment. Function 'evaluatePolicy.m' (#38.365.371), line 15, column 19: "policy" Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Function call failed. Function 'evaluatePolicy.m' (#38.140.167), line 7, column 11: "localEvaluate(observation1)" Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Function call failed. Function 'MATLAB Function1' (#24.45.62), line 3, column 16: "evaluatePolicy(u)" Launch diagnostic report.

Component:MATLAB Function | Category:Coder error

Errors occurred during parsing of MATLAB function 'DC_motor_RL_v02_exp/MATLAB Function1'

Component:MATLAB Function | Category:Coder error

### Build procedure for model: 'DC_motor_RL_v02_exp' aborted due to an error.

Errors occurred during parsing of MATLAB function 'DC_motor_RL_v02_exp/MATLAB Function1'

Answer 1

[Emmanouil Tzorakoleftherakis]

Hello,

To generate code from a trained policy, you should follow the process shown here. Note that this is a MATLAB-based workflow, i.e. you cannot generate code directly from Simulink prior to 20b, unless you import the network into your Simulink model through an S-function. Here is a video and an example that works through the workflow with GPU Coder.

In 20b, you can bring a neural network in Simulink with the deep learning block library (see video) and generate code directly from Simulink (see example).

[Armin Norouzi]

Thank you for your response. I created the code using the posted link. This resulted in "evaluatePolicy_mex.mexw64", but I am not sure how I can use it. I tried the legacy code tool to create S-function, but unfortunately, there isn't any good example/documentation to do that.

[Emmanouil Tzorakoleftherakis]

Updated my answer with some links.

[Justus Nwoke]

I have a similar problem: https://www.mathworks.com/matlabcentral/answers/1774210-deploying-trained-agent-to-run-in-simulink-hardware-in-the-loop-self-balancing-robot