Vision Language Models

Chapter 6. Post Training Vision Language Models

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 6th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

So far until the previous chapter we have been living the full “from scratch” fantasy. We took a tiny VLM, wired images into text, fought with padding and packing, watched the loss wobble its way down, and even made it answer a few questions about bears and mountains. That is roughly what pre-training and basic supervised training look like in miniature.

In practice though, most people do not spin up a VLM from nothing. You usually start from a strong base model that already knows a lot about language and images, and then you nudge it into the behavior you actually want. That second phase is what people call post-training. It has two big ingredients:

• Supervised fine-tuning, where you show the model lots of “here is a prompt, here is a good answer” pairs so it can follow instructions and handle the tasks you care about.

• Alignment, where you teach the model what humans actually prefer, using techniques like RLHF, DPO, MPO, and GRPO so that the ...