Deep Learning for Computer Vision

Using RNN on CNN features to generate captions

Vinyals et al., in the paper https://arxiv.org/pdf/1411.4555.pdf, proposed an end to end trainable deep learning for image captioning, which has CNN and RNN stacked back to back. This is an end to end trainable model. The structure is shown here:

Reproduced from Vinyals et al. (2015)

This model could generate a sentence that is completed in natural language. The expanded view of the CNN and LSTM is shown here:

Figure illustrating the CNN AND LSTM architecture [Reproduced from Vinyals et al.]