Vision Language Models

Chapter 4. Training Data and Preprocessing for VLMs

In Chapter 3, you trained your first vision language model (VLM). You watched loss curves go down, debugged loops caused by padding tokens, and learned that batch size 1 is a good start but batch size 4 is where things get real. You got excited when the model finally started producing decent results. You also discovered something slightly terrifying: training even a baby VLM on 0.5% of FineVision (120,000 samples) required careful token packing, placeholder tokens for images, and frequent vibe checking to catch bugs early.

Now imagine scaling that up. Not 120,000 samples: try 3.2 million. Then 15.3 million. Not 0.5% of one dataset but mixtures of dozens of datasets, each with different visual domains, different annotation quality, different proportions, and different purposes. Welcome to the data challenges of real VLM training.

VLMs are not really trained from scratch, as you saw in Chapter 3 when we loaded pretrained SigLIP and SmolLM. Modern VLMs build on top of pretrained vision encoders and language backbones before any multimodal training begins. What follows is babysitting (yes, literally, day and night, my friend!) the model through multiple training stages, each demanding different data qualities and volumes.

Let’s look at the Jina-VLM training data details in Figure 4-1.

The numbers tell a story that is hard to ignore, especially if you (like me) are a dinosaur who trained convolutional neural networks (CNNs) some years ...