Deep Learning for Computer Vision

Using multimodal metric space

Mao et al., in the paper https://arxiv.org/pdf/1412.6632.pdf, proposed a method that uses multimodal embedding space to generate the captions. The following figure illustrates this approach:

Reproduced from Mao et al.

Kiros et al., in the paper https://arxiv.org/pdf/1411.2539.pdf, proposed another multimodal approach to generate captions, which can embed both image and text into the same multimodal space. The following figure illustrates this approach:

Reproduced from Kiros et al.

Both of the multimodal approaches ...