O'Reilly logo

Hands-On Natural Language Processing with Python by Rajalingappaa Shanmugamani, Rajesh Arumugam

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

The attention mechanism

The attention context is obtained by first concatenating the output vector generated by the bidirectional GRU layer of the CBHG-based encoder with the output of the attention RNN. Then, the resulting vector feeds a tanh activated dense layer, followed by another dense layer. A softmax layer allows for getting the activation weights that give the attention context, through a dot product with the encoder output vector:

def get_attention_context(encoder_output,attention_rnn_output):    attention_input=Concatenate(axis=-1)([encoder_output,                                           attention_rnn_output])    e=Dense(10, activation = "tanh")(attention_input)    energies=Dense(1, activation = "relu")(e)    attention_weights=Activation('softmax')(energies) context=Dot(axes = ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required