Hands-On Deep Learning with Go

Gated Recurrent Units

An alternative to LSTM units are GRUs. These were first described by a team that was led by another significant figure in the history of deep learning, Yoshua Bengio. Their initial paper, Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014), offers an interesting way of thinking about these ways of augmenting the effectiveness of our RNNs.

Specifically, they draw an equivalence between the Tanh activation function in a vanilla RNN and LSTM/GRU units, also describing them as activations. The difference in the nature of their activation is whether information is retained, unchanged, or updated in the units themselves. In effect, the use of the Tanh function means that your ...