Hands-On Intelligent Agents with OpenAI Gym

Experience replay 

In most environments, the information received by the agent is not independent and identically distributed (i.i.d). What this means is that the observation that the agent receives is strongly correlated with the previous observation it had received and the next observation it will receive. This is understandable because typically, the problems that the agent solves in typical reinforcement learning environments are sequential. Neural networks are shown to converge better if the samples are i.i.d.

Experience replay also enables the reuse of the past experience of the agent. Neural network updates, especially with lower learning rates, require several back-propagation and optimization steps to converge to good values. Reusing ...