Artificial Intelligence By Example

RNN, LSTM, and vanishing gradients

To simulate sequences and memory, an RNN and an LSTM will use backpropagation algorithms.

An RNN often has problems of gradients when calculating them over deeper and deeper layers in the network. Sometimes, it vanishes (too close to 0) due to the sequence property, just like us when a memory sequence becomes too long. 

The backpropagation (just like us in a sequence) becomes less efficient. The are many backpropagation algorithms such as vanilla backpropagation, which is commonly used. This algorithm performs efficient backpropagation because it updates the weights after every training pattern. 

One way to force the gradient not to vanish is to use a ReLU activation function, f(x)=max(0,x), forcing values ...