Vision Language Models

Chapter 6. Core Architectures of Vision Language Models

In the previous chapters, you explored the history of vision language models (VLMs), looked at the different tasks they can perform, and learned how to benchmark their capabilities. You also trained them from scratch on large-scale datasets and post-trained them to steer specific behaviors. Now it’s time to open them up and see how they actually work inside.

In this chapter, we will explore the inner workings of VLMs. We will go past general ideas and look at how these models are built. How are visual and textual information combined? What common designs do people use when building these models? How do these design choices affect how well and how quickly a VLM can work? We will answer these questions and look at the main parts and strategies used in modern VLM designs.

Learning Objective

By the end of this chapter, you will clearly understand how vision and language parts work together in VLMs and know the common design patterns that make these models powerful.

The Key to Combining Information: Multimodal Attention

The core of many successful VLMs is the attention mechanism. If you’ve heard about transformers, you’ve likely heard about attention. Attention first became popular in language processing because it helps models understand how different words interact. In VLMs, attention helps the model find and connect important parts of visual and text data. Understanding attention is important because it’s a big reason why ...