Fine-Tuning AI

Chapter 11. Vision Language Models

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 11th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

For the past ten chapters, your work has lived in a world of language and text. Prompts in, completions out. Questions in, answers out. Tokens, like turtles, all the way down. But the real world is more than just language. It’s also visual: pixels, X-rays, dashboards, and factory floors. Vision language models (VLMs) begin to bridge that gap, being able to take an image and a question and produce an answer that demonstrates genuine visual understanding. In this chapter, we’ll explore how they work, using Gemma 4 as our guide. We’ll also explore how to fine-tune a VLM to be a better expert at understanding your specific images. By the end of the chapter, you’ll have a grasp of the two-tower architecture that powers modern VLMs and technology like SigLIP and the Vision Transformer (ViT) architecture that encode images into the same spaces as language tokens. You’ll see a complete vision-tuning pipeline that uses similar technology to what you’ve been using, ...