Numerical Computing with Python

TD prediction

Both TD and MC use experience to solve z prediction problem. Given some policy π, both methods update their estimate v of v_π  for the non-terminal states S_t occurring in that experience. Monte Carlo methods wait until the return following the visit is known, then use that return as a target for V(S_t).

The preceding method can be called as a constant - α MC, where MC must wait until the end of the episode to determine the increment to V(S_t) (only then is G_t known).

TD methods need to wait only until the next timestep. At time t+1, they immediately form a target and make a useful update using the observed reward R_t+1 and the estimate ...