Fine-Tuning AI

Chapter 5. Synthetic Data

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 5th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Chapter 4 covered the strategies for collecting and cleaning real-world data. But what happens when you’ve exhausted those strategies and still don’t have enough coverage to train a model effectively? Maybe your domain is so specialized that public datasets don’t exist for it, or perhaps the data you need is locked behind privacy regulations (such as real patient questions or financial transactions), and you can’t use it for training even if you do have it!

You’ve tried with the data you have, and you’ve found that it just doesn’t work. Exploring different architecture might work, but if you’re honest with yourself, if you don’t have enough data, what happens next?

This is where synthetic data enters the picture.

The core idea is simple: use a large, capable model (the “teacher”) to generate training examples that a smaller, cheaper model (the “student”) will learn from. The teacher already knows how to answer medical questions, write legal analyses, or debug code. ...