Vision Language Models

Chapter 5. Post-Training Vision Language Models

Until now, we have been living the full “from scratch” fantasy. We took a tiny vision-language model (VLM), wired images into text, fought with padding and packing, watched the loss wobble its way down, and even made it answer a few questions about bears and mountains. That is roughly what pretraining and basic supervised training look like in miniature.

In practice, though, most people do not spin up a VLM from nothing. You usually start from a strong base model that already knows a lot about language and images, and then you nudge it into the behavior you actually want. That second phase is what people call post-training. It has three big ingredients:

Supervised fine-tuning (SFT)

This is where you show the model lots of “here is a prompt, here is a good answer” pairs so it can follow instructions and handle the tasks you care about.

Alignment

This is where you teach the model what humans actually prefer, using techniques like reinforcement learning from human feedback, or RLHF (direct preference optimization, mixed preference optimization, and group relative policy optimization) so that the answers are not just correct but also helpful and safe.

Reinforcement learning with verifiable rewards (RLVR)

This replaces human preferences with verifiable reward functions (like math or code results) for increased performance on tasks that can be verified.

In this chapter, we will look at both parts. We will start with parameter efficient ...