Deep Learning with PyTorch

Long-term dependency

RNNs, in theory, should learn all the dependency required from the historical data to build a context of what happens next. Say, for example, we are trying to predict the last word in the sentence the clouds are in the sky. RNN would be able to predict it, as the information (clouds) is just a few words behind. Let's take another long paragraph where the dependency need not be that close, and we want to predict the last word in it. The sentence looks like I am born in Chennai a city in Tamilnadu. Did schooling in different states of India and I speak... . The vanilla version of RNN, in practice, finds it difficult to remember the contexts that happened in the earlier parts of sequences. LSTMs and other different variants ...