Vision Language Models

Chapter 2. Vision Language Model Applications

We have seen how vision and language is bridged, so let’s see how modern vision language models (VLMs) work. Modern VLMs—typically trained on massive vision, vision language, and language datasets—can accomplish a wide variety of vision tasks like identifying objects, actions, and scenes, as well as understand text for multimodal tasks. The possibilities of bridging images and text make the use cases endless, but we will focus on the most common use cases, early approaches used to solve them, how to accomplish these tasks, and how to evaluate a model on a given task.

The most natural place to start is the simplest form of connecting vision and language: describing what we see. Given an image, can a model produce a sentence that captures its content? This is image captioning, and it is the foundation on which many of the tasks in this chapter build.

Learning Objective

In this chapter, we will learn all the tasks we can accomplish with modern VLMs. In each task, we will see what the task looks like, offer model recommendations, consider previous approaches to solve the problem, evaluate a good model, and more!

Image Captioning

The first task we will see is image captioning. This task is about describing the visual content of an image in natural language​. Image captioning sits at the intersection of computer vision and natural language processing. It requires a model to understand an image (identify objects, attributes, and their ...