Neural Networks with Keras Cookbook

There's more...

A traditional RNN that takes multiple time steps into account for giving predictions can be visualized as follows:

Notice that, as time step increases, the impact of input at a much earlier layer would be lower. An intuition of that can be seen here (for a moment, let's ignore the bias terms):

h₅ = WX₅ + Uh₄ = WX5 + UWX₄ + U²WX₃ + U³WX₂ + U⁴WX₁

You can see that, as the time step increases, the value of hidden layer is highly dependent on X₁ if U>1, or much less dependent on X₁ if U<1.

The dependency on U matrix can also result in vanishing gradient when the value of U is very small and can result in exploding gradient when ...