Deep Learning with PyTorch Quick Start Guide

Long short-term memory networks

Long short-term memory networks (LSTMS), are a special type of RNN capable of learning long-term dependencies. While standard RNNs can remember previous states to some extent, they did this on a fairly basic level by updating a hidden state on each time step. This enabled the network to remember short-term dependencies. The hidden state, being a function of previous states, retains information about these previous states. However, the more time steps there are between the current state and a previous state, it diminishes the effect that this earlier state will have on the current state. Far less information is retained on a state that is say 10 time steps before the time step immediately preceding the current ...