Hands-On Reinforcement Learning for Games

Backward views and eligibility traces

Do you recall the last time you found a coin on the floor or street? After you picked up the coin, did you think to yourself: a) "I knew looking down all that time would pay off," or b) "Wow, I found a coin, how did I do that?" In fact, in most cases, it would be option b, that is, we learned something was good and then thought back to how we discovered it. The moment of brilliance in option a is akin to believing in quantum particles, atoms, and bacteria. This is no different in RL, and what we find is that it is often more useful to look back at what happened in the past; however, not so far back as to be a past event as in MC.

We can use TDL to take a backward look at the returns for several steps. ...