Advanced Deep Learning with Python

Luong attention

Luong attention (see Effective Approaches to Attention-based Neural Machine Translation at https://arxiv.org/abs/1508.04025) introduces several improvements over Bahdanau attention. Most notably, the alignment scores e_t depend on the decoder hidden state s_t, as opposed to s_t-1 in Bahdanau attention. To better understand this, let's compare the two algorithms:

Left: Bahdanau attention; right: Luong attention

Let's go through a step-by-step execution of Luong attention:

1. Feed the encoder with the input sequence and compute the set of encoder hidden states .
2. Compute the decoder hidden state based on the previous decoder hidden ...