Hands-On Q-Learning with Python

Double-deep Q-networks

The problem of maximization bias occurs when a network consistently overestimates the Q-values leading to the states it might end up in as a result of each potential action. This happens often at the beginning of training, where the DQN is still working with very noisy estimates of Q-values.

One alternative implementation of a DQN called a double-deep Q-network splits the process of choosing actions using two networks. One network is responsible for choosing the next action, which as usual will be the action with the highest Q-value. The other network, called the target network, carries out the process of evaluating and updating the Q-value of taking that action.

Decoupling these two functions cuts down on maximization ...