Vision Language Models

Chapter 2. Vision Language Model Applications

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Vision Language Models (VLMs) are models that can interpret both image and text. VLMs, typically trained on massive vision, vision-language, and language datasets, can accomplish a wide variety of vision tasks like identifying objects, actions, and scenes, as well as understand text for multimodal tasks.

In this chapter, take a look at multimodal tasks and models, how they are employed practically, and how to evaluate them.

Image Captioning

Image captioning is the task of describing the visual content of an image in natural language​, see Figure 2-1 to quickly get it. Image captioning sits at the intersection of computer vision and natural language processing. It requires a model to understand an image (identify objects, attributes, and their relationships) and generate a coherent sentence describing the image. This task is a fundamental problem in artificial intelligence that connects vision and language​.

Figure 2-1. Image captioning input, ...