Machine Learning for Developers

Main problems of the traditional RNNs — exploding and vanishing gradients

However, RNNs have turned out to be difficult to train, especially on problems with complicated long-range temporal structures – precisely the setting where RNNs ought to be most useful. Since their potential has not been realized, methods that address the difficulty of training RNNs are of great importance.

The most widely used algorithms for learning what to put in short-term memory, however, take too much time or do not work well at all, especially when minimal time lags between inputs and corresponding teacher signals are long. Although theoretically fascinating, the existing methods did not provide clear practical advantages over traditional feedforward networks. ...