Deep Learning for Computer Vision

Generator

The generator has an encoder and decoder. The encoder takes an image, question, and history as inputs. The encoder first attends to the history from LSTM and attends to the output from the image. The flow is shown here:

Reproduced from Lu et al.

The entire history is available and LSTM attends to the history of the chat. The output is attended by the image producing the embedding. The embedding generated by the encoder is used by the decoder to create an answer. The decoder is made of RNN. The encoder and decoder together form the generator producing the possible answers. Next, we will see how the discriminator works.