Python Deep Learning Projects

Defining the captioning model

The model architecture we are using to build the auto captioning is inspired by the Show, Attend and Tell paper (https://arxiv.org/pdf/1502.03044.pdf). The features that we extracted from the lower convolutional layer of Inception-v3 gave us a vector of a shape of (8, 8, 2048). Then, we squash that to a shape of (64, 2048).

This vector is then passed through the CNN encoder, which consists of a single fully connected layer. The RNN (GRU in our case) attends over the image to predict the next word:

def gru(units):  if tf.test.is_gpu_available():    return tf.keras.layers.CuDNNGRU(units,                                     return_sequences=True,                                     return_state=True,                                     recurrent_initializer='glorot_uniform')  else:    return tf.keras.layers.GRU(units,  return_sequences=True, ...