Hands-On Deep Learning with Go

Bias initialization of gates

Recently, at an ML conference, the International Conference on Learning Representations, a paper was delivered by a team from Facebook AI Research that described the progress of RNNs. This paper was concerned with the effectiveness of RNNs that had been augmented with GRU/LSTM units. Though a deep dive into the paper is outside the scope of this book, you can read more about it in the Further reading section, at the end of this chapter. An interesting hypothesis fell out of their research: that these units could have their bias vector initialized in a certain way, and that this would improve the network's ability to learn very long-term dependencies. They published their results, and it was shown that there seems ...