MLOps with Databricks

Chapter 7. LLMOps

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 7th chapter of the final book. Please note that the GitHub repo is available at https://github.com/mvechtomova/oreilly-mlops-with-databricks.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at [e-mail].

The number of Generative AI (GenAI) use cases has grown rapidly over the past few years, moving well beyond experimentation into applications that deliver tangible business value.

For example, Airbnb completed a large-scale LLM-driven code migration, updating nearly 3,500 React component test files. What was initially estimated to require 1.5 years of manual engineering effort was completed in just six weeks by combining frontier models with robust automation.

Similarly, DoorDash built AutoEval, an LLM-powered, human-in-the-loop system for evaluating the quality of search result pages. Instead of relying on slow and inconsistent human labeling, the system increased evaluation speed by 98% and expanded capacity by a factor of nine. Beyond scalability, it improved alignment with expert judgment and enabled continuous quality monitoring, allowing human experts ...