Vision Language Models

Chapter 8. Video-Language Models

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 8th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Deep learning has significantly advanced our ability to understand images. Videos, on the other hand, contain a sequence of images over time, showing motion and unfolding events. Extending image models to handle video is challenging because treating each frame individually ignores the important connections between them. For example, an image model might easily identify objects or people in each frame, but without understanding the sequence, it can’t tell if a person is entering a room or leaving it.

Figure 8-1. We provide some frames from a video and their timestamps to a video language model, and we get answers from it.

As you will learn through this chapter, a simple “frame-by-frame” approach would fail to distinguish actions that rely on the order of events, such as opening vs closing a door. Additionally, because videos contain so much data spread across ...