Hands-On Natural Language Processing with Python

The attention mechanism

The attention context is obtained by first concatenating the output vector generated by the bidirectional GRU layer of the CBHG-based encoder with the output of the attention RNN. Then, the resulting vector feeds a tanh activated dense layer, followed by another dense layer. A softmax layer allows for getting the activation weights that give the attention context, through a dot product with the encoder output vector:

def get_attention_context(encoder_output,attention_rnn_output):    attention_input=Concatenate(axis=-1)([encoder_output,                                           attention_rnn_output])    e=Dense(10, activation = "tanh")(attention_input)    energies=Dense(1, activation = "relu")(e)    attention_weights=Activation('softmax')(energies) context=Dot(axes = ...