Reasons for bad training performance using prioritized experience replay compared to uniform experience replay using DDPG agent
7 views (last 30 days)
Show older comments
I am currently trying to use prioritized experience replay while training DDPG agent on Quadruped robot (Quadruped Robot Locomotion Using DDPG Agent - MATLAB & Simulink (mathworks.com) ) instead of uniform experience replay to have faster training time. But while training with prioritized experience replay i notice considerable variance and instability while training compared to uniform replay buffer. This can be seen with frequent spikes and drops in the training monitor. The image below is when i use prioritized experience replay with following parameters
agentOptions = rlDDPGAgentOptions();
agentOptions.SampleTime = Ts;
agentOptions.DiscountFactor = 0.99;
agentOptions.MiniBatchSize = 256;
% agentOptions.ExperienceBufferLength = 1e6;
agentOptions.TargetSmoothFactor = 1e-3;
agentOptions.MaxMiniBatchPerEpoch = 200;
agentOptions.NoiseOptions.StandardDeviation = 0.1;
agentOptions.NoiseOptions.MeanAttractionConstant = 1.0;
agentOptions.ActorOptimizerOptions.Algorithm = "adam";
agentOptions.ActorOptimizerOptions.LearnRate = 1e-3;
agentOptions.ActorOptimizerOptions.GradientThreshold = 1;
agentOptions.CriticOptimizerOptions.Algorithm = "adam";
agentOptions.CriticOptimizerOptions.LearnRate = 1e-3;
agentOptions.CriticOptimizerOptions.GradientThreshold = 1;
initOpts = rlAgentInitializationOptions(NumHiddenUnit=256);
agent = rlDDPGAgent(obsInfo,actInfo,initOpts,agentOptions);
agent.ExperienceBuffer = rlPrioritizedReplayMemory(obsInfo,actInfo);
resize(agent.ExperienceBuffer,1e6);
agent.ExperienceBuffer.NumAnnealingSteps = 1e4;
agent.ExperienceBuffer.PriorityExponent = 0.6;
agent.ExperienceBuffer.InitialImportanceSamplingExponent = 0.4;
And the image below is when i use uniform experience replay for your comaparision.
Therefore im not sure why this is happening. I even tried with different hyperparameters for the prioritized experience replay but observe similar training results. Any posible solution would be really helpful.
0 Comments
Answers (1)
Kaustab Pal
on 8 Aug 2024
Hi @Gaurav
Prioritized Experience Replay (PER) tends to outperform uniform experience replay in environments where rewards are sparse, delayed, or where the dynamics are non-stationary. However, in the case of quadruped robot locomotion, the rewards are consistent. We receive positive rewards at every time-step to avoid early termination and additional rewards for positive forward velocity.
Since rewards are given consistently at every time-step, the variance in the importance of different experiences is lower. This means that each experience contributes relatively equally to the learning process, making uniform sampling sufficient for effective learning. The continuous nature of the rewards also ensures that the agent receives regular feedback about its performance, reducing the need for prioritizing specific experiences.
Because of these reasons, you can observe that the reward plot with uniform experience replay is smoother compared to the reward plot using prioritized experience replay.
Hope this clears your doubt.
With regards,
Kaustab Pal
2 Comments
Pavl M.
on 13 Oct 2024
Can you explain consistently what next parameters means:
agent.ExperienceBuffer.PriorityExponent = 0.6;
agent.ExperienceBuffer.InitialImportanceSamplingExponent = 0.4;
?
See Also
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!