Python Deep Learning - Second Edition

Policy evaluation

TD methods rely on their experience for policy evaluation. But unlike MC, they don't have to wait until the end of an episode. Instead, they can update the action-value function after each step of the episode. In its most basic form, a TD algorithm uses the following formula to perform a state-value update:

Where α is called step size (learning rate) and it's in the range of [0, 1]. Let's analyze this equation. We're going to update the value of the s_t state and we're following a policy, π, that has led the agent to transition from the s_t state to the s_t+1 state. During the transition, the agent has received a reward, r