Practical MLflow for Generative AI on Databricks

Preface

It is remarkably easy to build an AI prototype that feels impressive. You can take a foundation model, add a prompt, maybe add retrieval, and get something that answers in a way that seems human. The demo lands. The stakeholders smile. Someone asks, “Can we ship this?”

Then reality shows up.

We wrote this book to close that gap. In production, a GenAI application is judged by the average experience, the worst failure, and the operational burden it creates over time. Users ask vague questions, combine multiple intents, and omit critical details. The application responds with confidence even when it should slow down and clarify. Retrieval sometimes returns irrelevant context or no context at all. Agent tools time out. Policies change. A prompt tweak fixes one behavior and quietly breaks another. The model upgrade improves fluency but increases cost and latency. Safety rules that looked fine in testing fail under a new traffic pattern. And when something goes wrong, it is rarely obvious where in the system it went awry.

Our goal is to help you move from “We built something that works” to “We run something we can trust.” Trust comes from evidence. Evidence comes from a disciplined lifecycle where you can answer basic questions without guessing:

• What is the GenAI application doing for each request, step by step?

• What changed between two versions, and why did quality shift?

• What does “good” mean for your domain, and how do you consistently measure it?

• How do you promote ...