Boosting Reinforcement Learning Algorithms in Continuous Robotic Reaching Tasks Using Adaptive Potential Functions

In reinforcement learning, reward shaping is an efficient way to augment the reward signal, so to guide the learning process of an agent. A well-known reward shaping framework is the potential-based reward shaping (PBRS) framework, which uses a so-called potential function to guarantee the policy invariance after reward shaping, to prevent undesirable behavior. Different from using a predefined potential function in many works, [3] proposed a novel adaptive potential function (APF) method to learn the potential function concurrently with the RL training from the agent’s training history. However, the APF method was only deployed and evaluated in small discrete environments. This paper bridges the gap by adapting the APF method in robotics, a typical continuous scenario. We apply the APF method with the Deep Deterministic Policy Gradient (DDPG) algorithm to form a new APF-DDPG algorithm. To evaluate our method, we deploy the APF-DDPG to control a Baxtor robot for a series of reaching tasks in both simulations and the real world. The experimental results show that the APF-DDPG algorithm significantly outperforms the baseline DDPG algorithm. The code is available at https://github.com/yfchenShirley/APF_DDPG.