Video Generation with AI

Chapter 5. Fine-Tuning for Specific Video Tasks

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 5th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Learning objective: In this chapter you’ll look into fine-tuning methods leveraging the CogVideo model from Chapter 1. CogVideo provides an accessible and comprehensive fine-tuning framework developed by Tsinghua University, supporting both LoRA and supervised fine-tuning (SFT) with the necessary tooling required to specialize our model.

In Chapter 4 you focused on gaining intuition via training the Latte model; you configured experiments, implemented training loops, and distributed training across GPUs. This chapter bridges theory and practice. You’ll explore low-rank adaptation (LoRA) for parameter-efficient fine-tuning (PEFT) that requires nominal 16–24 GB VRAM, and supervised fine-tuning (SFT) with DeepSpeed integration. Each technique provided includes discussion of environmental requirements, optimization strategies, and issues you may encounter.

Note

Fine-tuning requirements vary dramatically by method and model type:

LoRA

16 GB (RTX ...