Deep Learning with R Cookbook

There's more...

By now, you are aware of how BPTT works in an RNN. We traverse the network backward, calculating gradients of errors with respect to the weights in each iteration. During backpropagation, as we move closer to the early layers of the network, these gradients become too small, thus making the neurons in these layers learn very slowly. For an accurate model, it is crucial for the early layers to get trained accurately since these layers are responsible for learning simple patterns from the input and passing the relevant information to the following layers accordingly. RNNs often face this challenge when we train huge networks with more dependencies within the layers. This challenge is referred to as the vanishing gradient problem ...