Advanced Deep Learning with Python

Introducing memory-augmented NNs

We've already seen the concept of memory (albeit in a strange form) in NNs—for example, the LSTM cell can add or delete information on its hidden cell state with the help of the input and the forget gates. Another example is the attention mechanism, where the set of vectors that represent the encoded source sequence can be viewed as external memory that is written to by the encoder and read from by the decoder. But this ability comes with some limitations. For one, the encoder can only write to a single memory location, which is the current element of the sequence. It also cannot update previously written vectors. On the other hand, the decoder can only read from the database, but cannot write to it.