Video Generation with AI

Chapter 6. Vision-Language Models for Video Understanding

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 6th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Learning objective: In this chapter, you will learn the architecture and implementation of vision-language models (VLMs) for video understanding, including the three-component design (vision encoder, projector, language model), 3D positional encoding for spatiotemporal reasoning, and building practical video question-answering systems with fine-tuning capabilities.

Building on Chapter 5’s fine-tuning techniques, we now tackle a complementary challenge: given a video, how can a model understand and reason about its contents? VLMs provide the answer, transforming video understanding into a language modeling problem. As discussed in Chapter 2, fully semantic understanding is an additional benefit to leveraging VLMs.

From Generation to Understanding

If video generation is like painting from a description, video understanding is like art criticism that analyzes what’s already there. The previous chapters taught you to create video from ...