Hands-On Q-Learning with Python

Calculating DQN loss

Calculating the prediction loss in a DQN (also called the TD or Temporal Difference error) is a matter of finding the difference between the true Q-value of a state-action pair and the value estimated by the network. We then backpropagate the loss to the earlier nodes in the network to update their weights.

The issue we run into is that we don't actually know the true Q-value of a state. We are estimating it using some version of the following Bellman equation:

• Q(s,a): Q target
• r(s,a): Reward of taking that action at that state
• γmax_aQ(s^',a): Discounted max q value among all possible actions from next state

Until our ...