Hands-On Transfer Learning with Python

The attention-based neural network model

We have discussed attention-based models for machine translation. The good part about attention-based models is that they give a way to interpret the model and understand how it works. The attention mechanism is a form of memorizing previous internal states. It's like internal memory. Unlike typical memory, the memory access mechanism here is soft, which means that the network retrieves a weighted combination of all memory locations, not a value from a single discrete location. Soft memory access makes it feasible to train the network by backpropagation. Attention-based architectures are used beyond machine translation in automatically generating captions for images.

This work was published in 2016 ...