Vision Language Models

Chapter 5. Model Training and Optimization

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 5th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

In the last chapter, we explored the world of data like a child running through a field of fallen leaves. We were delighted, but did not quite know why. Now, let us find some deeper meaning. Let us grab all that data knowledge and use it for good. Let us train some models.

In this chapter, we will demystify the process of model training and walk step by step through how a vision language model actually learns.

By the end, you will be able to

• train a baby model using single-sample updates

• scale it up to batches and see how the training dynamics change

• understand how images and instructions are packed together for efficient large-scale training

We will touch the full stack: VLM architecture, pre-training, data packing, inference, and a handful of practical training tricks. Buckle up, you are about to start a life long obsession with lines going down.

Learning Objective:

Develop best practices for designing and training vision language models. ...