Deep Learning Essentials

Training RNN is tough

In this section, we will look at some of the existing limitations of training RNN. We will also dive deeper and understand why it is tough to train RNN.

The traditional method of training neural networks is through the backpropagation algorithm. In the case of RNN, we need to perform a backpropagation of gradient through time, often referred to as backpropagation through time (BPTT). Although it's numerically possible to compute backward propagation of gradients through time, it often results in poor results due to the classic vanishing (or exploding) gradient problem as shown in figure vanishing gradient problem in RNN with multiplicative gradients:

Vanishing gradient problem in RNN with multiplicative gradients