Hands-On Reinforcement Learning for Games

Bootstrapping and backup diagrams

TDL can learn during an episode by approximating the updated value function given previous experience. This allows the algorithm to learn while it is in an episode and hence make corrections as needed. To understand the differences further, let's review a composite of the backup diagrams for DP, MC, and TD in the following diagram:

Backup diagrams for DP, MC, and TDL

The diagram was taken from An Introduction to Reinforcement Learning by Barto and Sutton (2018). In the diagram, you can see our previous two methods, DP and MC, as well as TDL. The shaded area (red or black) denotes the algorithm's learning space. ...