Deep Learning with Keras

Experience replay, or the value of experience

Based on the equations that represent the Q-value for a state action pair (s_t, a_t) in terms of the current reward r_t and the discounted maximum Q-value for the next time step (s_t+1, a_t+1), our strategy would logically be to train the network to predict the best next state s' given the current state (s, a, r). It turns out that this tends to drive the network into a local minimum. The reason for this is that consecutive training samples tend to be very similar.

To counter this, during game play, we collect all the previous moves (s, a, r, s') into a large fixed size queue called the replay memory. The replay memory represents the experience of the network. When training the network, we generate ...