Python Deep Learning Projects

Attention

Now we will define the attention mechanism popularly known as Bahdanau attention (https://arxiv.org/pdf/1409.0473.pdf). We will need the features from the CNN encoder of a shape of (batch_size, 64, embedding_dim). This attention mechanism will return the context vector and the attention weights over the time axis:

class BahdanauAttention(tf.keras.Model):  def __init__(self, units):    super(BahdanauAttention, self).__init__()    self.W1 = tf.keras.layers.Dense(units)    self.W2 = tf.keras.layers.Dense(units)    self.V = tf.keras.layers.Dense(1)    def call(self, features, hidden):    # hidden_with_time_axis shape == (batch_size, 1, hidden_size)    hidden_with_time_axis = tf.expand_dims(hidden, 1)        # score shape == (batch_size, 64, hidden_size) score ...