Fine-Tuning AI

Chapter 13. Fine-Tuning Diffusion Models

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 13th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

In Chapter 12, you saw how a diffusion model works from the inside: a frozen VAE, a frozen text encoder, a UNet that predicts noise, and a scheduler that turns predictions into pixels. You also saw, at the conceptual level, where the trainable knobs live. This chapter is where you actually turn and tune those knobs.

You’re going to fine-tune diffusion models for two completely different outcomes. These will be to teach the model a specific subject and a specific style. You’ll use the same model architecture and training loop for both, switching between them by changing only the training data.

The first (subject) teaches the model to draw one specific thing: your dog, your product, your face. The second (style) teaches it a visual vocabulary: your brand’s photography, your illustrator’s hand, an anime line-art look.

By the end of the chapter, you’ll have walked through both recipes end-to-end on a single consumer GPU, you’ll know how to compose multiple ...