AI and ML for Coders in PyTorch

Chapter 20. Tuning Generative Image Models with LoRA and Diffusers

In Chapter 19, you explored the idea of diffusers and how models trained with diffusion techniques can generate images based on prompts. Like text-based models (as we explored in Chapter 16), text-to-image models can be fine-tuned for specific tasks. The architecture of diffusion models and how to fine-tune them is enough for a full book in its own right, so in this chapter, you’ll just explore these concepts at a high level. There are several techniques for doing this, including DreamBooth, textual inversion, and the more recent low-ranking adaptation (LoRA), which you’ll go through step-by step in this chapter. This last technique allows you to customize models for a specific subject or style with very little data.

As with transformers, the diffusers Hugging Face library is designed to make using diffusers, as well as fine-tuning them, as easy as possible. To that end, it includes pre-built scripts that you can use.

We’ll go through a full sample of creating a dataset of a fictitious digital influencer called Misato, using LoRA and diffusers to fine-tune a text-to-image model called Stable Diffusion 2 for her. Then, we’ll perform text-to-image inference to demonstrate how to create new images of Misato (see Figure 20-1).

Figure 20-1. LoRA-tuned Stable Diffusion 2 images

Training a LoRA with Diffusers

To train ...