Deep Reinforcement Learning with Python: With PyTorch, TensorFlow and OpenAI Gym

© Nimish Sanghi 2021

N. SanghiDeep Reinforcement Learning with Pythonhttps://doi.org/10.1007/978-1-4842-6809-4_8

8. Combining Policy Gradient and Q-Learning

Nimish Sanghi¹  

(1)

Bangalore, India

 

So far in this book, in the context of deep learning combined with reinforcement learning, we have looked at deep Q-learning with its variants in Chapter 6 and at policy gradients in Chapter 7. Neural network training requires multiple iterations, and Q-learning, an off-policy approach, enables us to use transitions multiple times, giving us sample efficiency. However, Q-learning can be unstable at times. Further, it is an indirect way of learning. Instead of learning an optimal policy directly, we first learn q-values and then use these action values to learn ...