Chapter 6. Fine-tuning Stable Diffusion

In the previous chapter, we introduced how fine-tuning can teach language models to write in a particular style or to learn concepts for a specific domain. We can apply the same principles to text-to-image models, allowing us to customize the models even with access to a single GPU (versus the multi-GPU nodes required to pre-train a model like Stable Diffusion).

In this chapter, we will use the base pre-trained Stable Diffusion model we learned in Chapter 5 and extend it to learn styles and concepts that it might not know about, such as the concept of "your pet" or a particular painting style. We will also learn how to give it new capabilities, such as super-resolution, inpainting, or new conditions as inputs.

Rather than writing code from scratch here, we will look into understanding and running existing scripts created for fine-tuning the models in this section. For that, we recommend ...

Get Hands-On Generative AI with Transformers and Diffusion Models now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.