Advanced Deep Learning with Python

General attention

Although we've discussed the attention mechanism in the context of NMT, it is a general deep-learning technique that can be applied to any seq2seq task. Let's assume that we are working with hard attention. In this case, we can think of the vector s_t-1 as a query executed against a database of key-value pairs, where the keys are vectors and the hidden states h_i are the values. These are often abbreviated as Q, K, and V, and you can think of them as matrices of vectors. The keys Q and the values V of Luong and Bahdanau attention are the same vector—that is, these attention models are more like Q/V, rather than Q/K/V. The general attention mechanism uses all three components.

The following diagram illustrates this new general ...