Hands-On Mathematics for Deep Learning

Writing

The writing operation takes inspiration from the input and forget gates of LSTMs, where some information is erased and then replaced (or added).

The memory is then updated using two formulas, the first of which erases the memory and the second of which adds memory:

• 
• 

Here, is the erase vector, a_t is the add vector, and 1 is a vector containing only ones. From these equations, we can see that the memory at a specific location is only ...