i have unstable second order system and trying to make it follow square wave using reinforcement learning but the agent doesn't converge

1 view (last 30 days)
i have unstable second oreder system with 2 poles at 2 ,3 and i have square wave as refrence and trying to make the system follow the ref. the observation are x1,x2 and error and the reward function is -rms(error) and the error signal is (ref-x2) as x2 is the output
the A matrix [0 -6;1 5] B=[10;1] C[0 1] D[0]
  6 Comments
Sam Chak
Sam Chak on 27 Jul 2022
Edited: Sam Chak on 27 Jul 2022
Hi @farouk, @farouk, Thinking from the mathematical perspective, what are the manipulated variables in RL that you think can be tuned (maybe with GA)?
Does the difficulty have something to do with Learning Rate of RL (which I read about in some documentation)?
What knowledge and theorem are required so that we can deterministically set the values to make the agent converge?

Sign in to comment.

Answers (1)

Sam Chak
Sam Chak on 27 Jul 2022
I don't know how many iterations your RL will take to stabilize the system and track the square wave. But you can probably at least use the generated data for performance comparison purposes.
% 2nd-order system
A = [0 -6; 1 5]; B = [10; 1]; C = [0 1]; D = [0];
sys = ss(A,B,C,D);
Gs = tf(sys)
Gs = s + 10 ------------- s^2 - 5 s + 6 Continuous-time transfer function.
% Manipulated Variable (MV)
s = tf('s');
Gv = 20*(s^2 - 5*s + 6)/(s*(s + 10))
Gv = 20 s^2 - 100 s + 120 -------------------- s^2 + 10 s Continuous-time transfer function.
% Closed-loop system
Gcl = minreal(feedback(Gv*Gs, 1))
Gcl = 20 ------ s + 20 Continuous-time transfer function.
% Tracking a square wave reference signal
tau = 2;
[u, t] = gensig("square", tau, 4);
lsim(Gcl, u, t)
ylim([-1 2]), grid on
% Output signal of MV
Gu = minreal(feedback(Gv, Gs))
Gu = 20 s^4 - 200 s^3 + 740 s^2 - 1200 s + 720 ----------------------------------------- s^4 + 25 s^3 + 56 s^2 - 820 s + 1200 Continuous-time transfer function.
lsim(Gu, u, t), grid on
The system output can track the square wave reference input. It takes approximately 0.25 s to converge to the reference signal without overshoot.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!