Vision Language Models

Chapter 4. Training Data and Preprocessing for VLMs

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 4th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

A shocking difference between VLMs and traditional machine learning based computer vision is the sheer size, scale, and diversity of the data needed to train modern vision-language models.

If you examine SmolLM3’s pretraining recipe in Figure 4-1 and add the durations of its three training phases, you can see that it required 11.1 trillion tokens. To put this in perspective: assuming 1 token is roughly 4 characters, then 11.1 trillion tokens equals about 44 trillion characters of text. That’s on the same order of magnitude as the entire publicly readable Internet.

Figure 4-1. SmolLM3 data mixture and training stage

When moving on to training VLMs the shear variability in data becomes an added complication. While even early works in LLMs recognized the importance of data mixtures (e.g., what percent of the data is code, math etc), the variability ...