Vision Language Models

Chapter 10. Any to Any Models

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 10th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Multimodal models are great because they ground language in the world we actually live in. VLMs can see images and answer questions in natural language. Video models understand how actions unfold, cause-and-effect over time. Audio models can hear speech and sound. All of them bring AI closer to human-like perception.

What if you want to build an assistant that can see, hear and speak? With what we have seen until now, you would need a vision model, an audio model, a speech synthesis model and careful orchestration among them.

The next frontier is any-to-any models: unified architectures that handle understanding and generation across text, images, audio and video within a single system (Figure 10-1).

Figure 10-1. Overview of a unified multimodal model supporting understanding and generation across text, images, video, and audio.

To be truly any-to-any, a model ...