Vision Language Models

Chapter 3. Vision Language Model Training

In Chapter 2, we explored vision language model (VLM) applications like a child running through a field of fallen leaves. We were delighted, but we did not quite know what we were doing. Now, let us find some deeper meaning. Let’s grab all that motivation and use it for good. Let’s train some models.

In this chapter, we will demystify the process of model training and walk step by step through how a VLM actually learns.

By the end of this chapter, you will be able to:

• Train a baby model using single-sample updates.

• Scale it up to batches and see how the training dynamics change.

• Understand how images and instructions are packed together for efficient large-scale training.

We will touch the full stack: VLM architecture, pretraining, data packing, inference, and a handful of practical training tricks. Buckle up, you are about to start a lifelong obsession with loss curves going down.

Learning Objective

Develop best practices for designing and training VLMs.

A Bird’s-Eye View of Training Vision Language Models

Before we start training VLMs, let’s establish a common vocabulary. You might know many of these concepts individually, but seeing them all together can be overwhelming. The terminology often mixes stages of training (pretraining, midtraining, post-training) with training methods (contrastive learning, alignment, RLHF).

To make things even worse, these definitions shift over time. For example, supervised fine-tuning and ...