Video Generation with AI

Chapter 4. Training Video Generation Models

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 4th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Learning objective: In this chapter, you’ll build on your implementation of the Latte architecture and learn how to create progressive training pipelines that transform your model into a production-ready video generation system.

In Chapter 3, you implemented the complete Latte architecture, from VAE compression and patch embedding through spatial and temporal attention blocks to the diffusion process and text conditioning. Now we’ll transform that architecture into a functioning video generation model through a systematic training loop.

Training video generation models requires more than just feeding data through your architecture. As mentioned in previous chapters, the temporal dimension introduces unique challenges: datasets are massive, memory constraints are severe, and optimization is delicate. Building on Chapter 3, and leveraging Latte’s training approach as our guide, we’ll build a progressive pipeline that handles these challenges elegantly. ...