some of the saved agents in DQN reinforcement learning algorithm do not reproduce the training rewards

6 views (last 30 days)
Hello everyone,
I am training a DQN agent in my reinforcement learning problem in Simulink Matlab 2019a. I have set a save criteria of agents with episode rewards equal to 6. My problem is that after a number of episodes, some agents will save based on my save criteria, but when I use them, some of the agents do not reproduce rewards equal to 6 or more but the amount of rewards is less than 6. If anyone could help me I will appriciate.
masoud k
masoud k on 24 Sep 2020
Hi Madhav,
The criteria for saving the candidate agents is getting reward of 6 or more of an episode.
I use epsilon-greedy policy to train agents with epsilon equal to 0.6 and the decay rate of 0.001.
Some of saved agents come up with an episode reward around 1 and some other with 6 or more.

Sign in to comment.

Accepted Answer

Emmanouil Tzorakoleftherakis
This is a common misconception. When you train an agent and you get a certain reward out of a specific episode, if you were to stop training and run the same episode again with the same agent you would most likely get a different response from the agent and thus a different episode reward.
There are a few reasons for that:
1) During training, the agent explores different options based on some probabilistic exploration strategy. For DQN this strategy is related to the epsilon parameter in DQN options. After training, the agent does not explore anymore and only relies on the underlying neural network for inference, so it makes sense that results are different.
2) It is best practice to randomize various elements of the environment during training to get a more robust policy. In that case there will also be differences in how the environment behaves during and after training, so agent will respond differently as well.
3) Some agents are stochastic. This by itself implies different decisions/behavior even if everything else in the environment remains deterministic.
Hope that helps

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!