book

Practical MLflow for Generative AI on Databricks

by Nuwan Ganganath, Julie Nguyen, Chang Shi Lim

August 2026

Intermediate to advanced

336 pages

9h 5m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Foreword
Preface
Why We Wrote This BookWho This Book Is ForWhat You Will LearnUsing Code ExamplesConventions Used in This BookO’Reilly Online LearningHow to Contact Us
1. Introduction to MLflow for GenAI on Databricks
What Is MLflow?Experiment and Model Tracking for ReproducibilityMLflow EvaluationCollaborationModel Registry and VersioningCore MLflow Components: Experiments, Runs, and Model RegistryFrom MLflow 2.x to MLflow 3.xOpen Source Versus Databricks ImplementationsWhy MLflow for GenAI?Management of Iterative Prompts and ModelsGenAI Application End-to-End ObservabilityAutomated Quality Evaluation with LLM-As-a-JudgeContinuous Improvement from Production FeedbackIntroducing the Book’s Core Use CaseConclusion and Key Takeaways
2. End-to-End GenAI Application Lifecycle with MLflow
Overview of the GenAI Application Lifecycle with MLflowThe Five-Phase GenAI LifecycleWhat This Lifecycle SolvesHallmarks of Effective GenAI WorkflowsApplying the Lifecycle to the Unity Airways Use CaseDevelop with MLflowEvidence-Driven Development WorkflowBuilding the Minimum Evaluable ProductPracticing Trace-First DevelopmentVersioning Development ProcessBalancing Quality, Cost, Latency, Safety, and Product FeelThe Disciplined Iteration LoopEvaluate with MLflowEvaluation ObjectivesEvaluation WorkflowBuilding and Maintaining Evaluation DatasetsInterpreting Scores and Traces TogetherHuman-in-the-Loop FeedbackDiagnosing and Resolving IssuesDeploy with MLflowVersioning Applications, Prompts, and ModelsDeployment and Serving ArchitectureReproducible Deployments and Controlled ReleasesEvidence-Based Traffic Promotion and RollbackMonitor with MLflowMonitoring GoalsObservability at Scale with TracingGovernance and Compliance in MonitoringOperational Reliability and Evidence-Based MonitoringImprove with MLflowImprovement ObjectivesWhat Improvement Looks Like in PracticeBridging Offline and Online SignalsThe Improve Loop As a Disciplined PracticeGrounding Improvement in the Unity Airways Use CaseComplete Workflow IntegrationConclusion and Key Takeaways
3. Prompt Engineering with MLflow
Fundamentals of Prompt EngineeringCore Prompting TechniquesPrompt Engineering Best PracticesAntipatterns and Debugging HabitsManaging Prompts with the MLflow Prompt RegistryThe “Why” Behind the RegistryPrompt Registry SDK OverviewA Tour of Prompt Lifecycle with Prompt RegistryUsing Prompts in CodeLoading a PromptRendering Template Variables with format()Making the Model Call: A Minimal Databricks PatternError-Handling Patterns and GotchasEvaluating and Comparing Prompt VersionsBuilding a Prompt Evaluation DatasetCreating Prompt Versions to CompareRunning Comparative EvaluationComparing Results and Choosing a WinnerPrompt OptimizationWhen Optimization HelpsMultiple Techniques Through a Single MLflow InterfaceA Unity Airways Optimization WorkflowHow Optimization Works Under the HoodKeeping Optimization PracticalIntegrating Prompts into ApplicationsPublishing Versions and Assigning AliasesReferencing Aliases in Your ApplicationEnd-to-End “Thin Slice”Promoting to Production and Rolling Back SafelyConclusion and Key Takeaways
4. Building and Versioning a Tool-Calling Agent
What Is a Tool-Calling Agent?Unstructured Data Preparation for RetrievalLangChain and Databricks IntegrationChatDatabricksDatabricks Vector SearchBuilding a LangChain Agent and Versioning in MLflowLLM-Only ApplicationFull Tool-Calling Agent ApplicationMLflow App VersioningPackaging and Logging the Agent and ArtifactsModel Signatures and Dependent ResourcesLogging with LangChain FlavorLogging Model As CodeConclusion and Key Takeaways
5. MLflow Tracing for GenAI Application Observability
Understanding MLflow TracingTraceSpanInstrumenting TracesAutomated TracingManual TracingQuerying TracesConclusion and Key Takeaways
6. Evaluating GenAI Applications with MLflow
The MLflow Evaluation FrameworkEvaluation HarnessEvaluation ModesCreating and Managing Evaluation DatasetsWhat a GenAI Evaluation Dataset Looks LikeHow to Build an Evaluation DatasetDesigning and Managing ScorersWhat Makes a Good ScorerCode-Based ScorersLLM JudgesScorer Lifecycle ManagementRunning and Comparing Evaluations in MLflowEvaluation Run StructureEvaluating the Unity Airways AssistantComparing Evaluation RunsDiagnosing Regressions and Troubleshooting with TracesMultiturn EvaluationCapturing Human FeedbackData Model OverviewCollecting Human FeedbackAligning LLM Judges with Human FeedbackConclusion and Key Takeaways
7. Advanced Agents and Tools
Developing Agents with MLflowWhat Are Agents, AI Agents, and Agentic AI?Integration on MLflowMLflow AI Gateway Interface for LLM IntegrationCreating ToolsVector Retriever ToolStructured Data Lookup ToolAPI-Calling ToolExtending the Unity Airways Chatbot with ToolsTesting ToolsPackaging and Deploying Advanced GenAI Applications with ResponsesAgentWhy and When to Use ResponsesAgentPackaging the Agent with the MLflow ResponsesAgentLogging the ResponsesAgent with MLflowRegister the ResponsesAgent with MLflow in Unity CatalogIntegrating Advanced CapabilitiesContext EngineeringMCP Servers As ToolsMultiagents Orchestration and Agents As ToolsConclusion and Key Takeaways
8. Deploying a GenAI Application with MLflow
Deploying to Databricks Model ServingModel Serving EndpointReview AppMLflow AI GatewayUsing Unity AI GatewayUnity AI Gateway CapabilitiesFallbacksAI GuardrailsAI Guardrails ApplicationSupported GuardrailsCustom GuardrailsGuardrails ExamplesLLM Operations and Agent OperationsEnvironment ManagementChallenger Versus ChampionDeployment RolloutAuthentication for Agent EndpointBest PracticesChallenges of ProductionizationComponents of a Productionized AI ApplicationDecoupling Production Agent Systems from RuntimeMLflow Agent ServerConclusion and Key Takeaways

9. Production Monitoring with MLflow
Types of Metrics to MonitorOperational MetricsQuality MetricsBusiness Impact MetricsOnline Monitoring WorkflowInstrumenting TracesReal-Time Capturing of TracesCollecting End-User FeedbackRegister and Start Scorers for Online MonitoringAdvanced Monitoring on Production TracesNatural Language Processing Analysis on TracesCustom Dashboard with Databricks AI/BICreating Metrics AlertsInteractively Gather Insights from TracesExpanding the Evaluation DatasetConclusion and Key Takeaways
10. Unifying GenAI Systems with MLflow
MLflow Traces As the Shared Integration ArtifactOpenTelemetry for Cross-Stack TracingMLflow Agent Server As a Stable Serving LayerMLflow MCP As the Bridge to Assistants and IDEsMLflow Skills for Coding AgentsIntegrating the Ecosystem Through Third-Party JudgesThe Road Ahead
Index
About the Authors

Content preview from Practical MLflow for Generative AI on Databricks

Chapter 8. Deploying a GenAI Application with MLflow

In the previous chapter, we upgraded our Unity Airways assistant from a basic tool-calling agent into a more capable agent and registered it in Unity Catalog. Now that our agent is built and safely stored, it’s time to get it out into the real world where it can actually start helping users.

This chapter focuses on that next step: deployment. We will explore how MLflow manages the lifecycle of your AI application. This is where the specific components MLflow uses to deploy, govern, and version your developed agent on Databricks will shine, and help you reliably transition your application from development to a production environment.

As we operationalize our Unity Airways assistant, we’ll walk through these different steps needed for proper agent productionization (see Figure 8-1):

Deploying our agent to an endpoint for downstream usage
Setting Unity AI Gateway for endpoint access control and rate limitations
Adding AI guardrails to watch out for inquiries relevant to our Unity Airways assistant
Setting authentication methods to agents, tools, and data for production best practices
Using LLMOps best practices to address any challenges in production

These steps will ensure you make the right architectural decisions for your projects so that your deployments are effective, secure, and compliant.

Diagram illustrating the steps for deploying an AI application, highlighting the inclusion of an AI gateway and guardrails for security before generating a response from a traveler query.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9798341652743Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Practical MLflow for Generative AI on Databricks

by Nuwan Ganganath, Julie Nguyen, Chang Shi Lim

Chapter 8. Deploying a GenAI Application with MLflow

Figure 8-1. Productionizing ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.