book

Practical MLflow for Generative AI on Databricks

by Nuwan Ganganath, Julie Nguyen, Chang Shi Lim

August 2026

Intermediate to advanced

336 pages

9h 5m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Foreword
Preface
Why We Wrote This BookWho This Book Is ForWhat You Will LearnUsing Code ExamplesConventions Used in This BookO’Reilly Online LearningHow to Contact Us
1. Introduction to MLflow for GenAI on Databricks
What Is MLflow?Experiment and Model Tracking for ReproducibilityMLflow EvaluationCollaborationModel Registry and VersioningCore MLflow Components: Experiments, Runs, and Model RegistryFrom MLflow 2.x to MLflow 3.xOpen Source Versus Databricks ImplementationsWhy MLflow for GenAI?Management of Iterative Prompts and ModelsGenAI Application End-to-End ObservabilityAutomated Quality Evaluation with LLM-As-a-JudgeContinuous Improvement from Production FeedbackIntroducing the Book’s Core Use CaseConclusion and Key Takeaways
2. End-to-End GenAI Application Lifecycle with MLflow
Overview of the GenAI Application Lifecycle with MLflowThe Five-Phase GenAI LifecycleWhat This Lifecycle SolvesHallmarks of Effective GenAI WorkflowsApplying the Lifecycle to the Unity Airways Use CaseDevelop with MLflowEvidence-Driven Development WorkflowBuilding the Minimum Evaluable ProductPracticing Trace-First DevelopmentVersioning Development ProcessBalancing Quality, Cost, Latency, Safety, and Product FeelThe Disciplined Iteration LoopEvaluate with MLflowEvaluation ObjectivesEvaluation WorkflowBuilding and Maintaining Evaluation DatasetsInterpreting Scores and Traces TogetherHuman-in-the-Loop FeedbackDiagnosing and Resolving IssuesDeploy with MLflowVersioning Applications, Prompts, and ModelsDeployment and Serving ArchitectureReproducible Deployments and Controlled ReleasesEvidence-Based Traffic Promotion and RollbackMonitor with MLflowMonitoring GoalsObservability at Scale with TracingGovernance and Compliance in MonitoringOperational Reliability and Evidence-Based MonitoringImprove with MLflowImprovement ObjectivesWhat Improvement Looks Like in PracticeBridging Offline and Online SignalsThe Improve Loop As a Disciplined PracticeGrounding Improvement in the Unity Airways Use CaseComplete Workflow IntegrationConclusion and Key Takeaways
3. Prompt Engineering with MLflow
Fundamentals of Prompt EngineeringCore Prompting TechniquesPrompt Engineering Best PracticesAntipatterns and Debugging HabitsManaging Prompts with the MLflow Prompt RegistryThe “Why” Behind the RegistryPrompt Registry SDK OverviewA Tour of Prompt Lifecycle with Prompt RegistryUsing Prompts in CodeLoading a PromptRendering Template Variables with format()Making the Model Call: A Minimal Databricks PatternError-Handling Patterns and GotchasEvaluating and Comparing Prompt VersionsBuilding a Prompt Evaluation DatasetCreating Prompt Versions to CompareRunning Comparative EvaluationComparing Results and Choosing a WinnerPrompt OptimizationWhen Optimization HelpsMultiple Techniques Through a Single MLflow InterfaceA Unity Airways Optimization WorkflowHow Optimization Works Under the HoodKeeping Optimization PracticalIntegrating Prompts into ApplicationsPublishing Versions and Assigning AliasesReferencing Aliases in Your ApplicationEnd-to-End “Thin Slice”Promoting to Production and Rolling Back SafelyConclusion and Key Takeaways
4. Building and Versioning a Tool-Calling Agent
What Is a Tool-Calling Agent?Unstructured Data Preparation for RetrievalLangChain and Databricks IntegrationChatDatabricksDatabricks Vector SearchBuilding a LangChain Agent and Versioning in MLflowLLM-Only ApplicationFull Tool-Calling Agent ApplicationMLflow App VersioningPackaging and Logging the Agent and ArtifactsModel Signatures and Dependent ResourcesLogging with LangChain FlavorLogging Model As CodeConclusion and Key Takeaways
5. MLflow Tracing for GenAI Application Observability
Understanding MLflow TracingTraceSpanInstrumenting TracesAutomated TracingManual TracingQuerying TracesConclusion and Key Takeaways
6. Evaluating GenAI Applications with MLflow
The MLflow Evaluation FrameworkEvaluation HarnessEvaluation ModesCreating and Managing Evaluation DatasetsWhat a GenAI Evaluation Dataset Looks LikeHow to Build an Evaluation DatasetDesigning and Managing ScorersWhat Makes a Good ScorerCode-Based ScorersLLM JudgesScorer Lifecycle ManagementRunning and Comparing Evaluations in MLflowEvaluation Run StructureEvaluating the Unity Airways AssistantComparing Evaluation RunsDiagnosing Regressions and Troubleshooting with TracesMultiturn EvaluationCapturing Human FeedbackData Model OverviewCollecting Human FeedbackAligning LLM Judges with Human FeedbackConclusion and Key Takeaways
7. Advanced Agents and Tools
Developing Agents with MLflowWhat Are Agents, AI Agents, and Agentic AI?Integration on MLflowMLflow AI Gateway Interface for LLM IntegrationCreating ToolsVector Retriever ToolStructured Data Lookup ToolAPI-Calling ToolExtending the Unity Airways Chatbot with ToolsTesting ToolsPackaging and Deploying Advanced GenAI Applications with ResponsesAgentWhy and When to Use ResponsesAgentPackaging the Agent with the MLflow ResponsesAgentLogging the ResponsesAgent with MLflowRegister the ResponsesAgent with MLflow in Unity CatalogIntegrating Advanced CapabilitiesContext EngineeringMCP Servers As ToolsMultiagents Orchestration and Agents As ToolsConclusion and Key Takeaways
8. Deploying a GenAI Application with MLflow
Deploying to Databricks Model ServingModel Serving EndpointReview AppMLflow AI GatewayUsing Unity AI GatewayUnity AI Gateway CapabilitiesFallbacksAI GuardrailsAI Guardrails ApplicationSupported GuardrailsCustom GuardrailsGuardrails ExamplesLLM Operations and Agent OperationsEnvironment ManagementChallenger Versus ChampionDeployment RolloutAuthentication for Agent EndpointBest PracticesChallenges of ProductionizationComponents of a Productionized AI ApplicationDecoupling Production Agent Systems from RuntimeMLflow Agent ServerConclusion and Key Takeaways

9. Production Monitoring with MLflow
Types of Metrics to MonitorOperational MetricsQuality MetricsBusiness Impact MetricsOnline Monitoring WorkflowInstrumenting TracesReal-Time Capturing of TracesCollecting End-User FeedbackRegister and Start Scorers for Online MonitoringAdvanced Monitoring on Production TracesNatural Language Processing Analysis on TracesCustom Dashboard with Databricks AI/BICreating Metrics AlertsInteractively Gather Insights from TracesExpanding the Evaluation DatasetConclusion and Key Takeaways
10. Unifying GenAI Systems with MLflow
MLflow Traces As the Shared Integration ArtifactOpenTelemetry for Cross-Stack TracingMLflow Agent Server As a Stable Serving LayerMLflow MCP As the Bridge to Assistants and IDEsMLflow Skills for Coding AgentsIntegrating the Ecosystem Through Third-Party JudgesThe Road Ahead
Index
About the Authors

Overview

Bridge the gap between cutting edge research and the practical operationalization of generative AI. This handy guide demystifies the complexities of managing GenAI applications using leading open source platform MLflow on Databricks.

What sets this book apart is its focus on the requirements of managing and deploying GenAI applications. ML and AI specialists from Databricks, authors Nuwan Ganganath, Julie Nguyen, and Chang Shi Lim take you through each stage of the development of an agentic chatbot, sharing best practices, innovative techniques, and common challenges along the way. By combining deep technical insights with interactive exercises and actionable real-world code examples, they ensure that you're equipped to implement robust GenAI applications in modern enterprise environments.

Understand MLflow fundamentals, including core components and the tool's application to GenAI on the Databricks platform
Build, track, and manage GenAI applications using MLflow, from prompts to the overall agentic chain
Optimize and evaluate GenAI applications through a combination of automated LLM-assisted judges and human-in-the-loop review
Deploy production-ready GenAI applications, leveraging Databricks' scalable infrastructure and monitoring capability

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9798341652743Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills