Practical MLflow for Generative AI on Databricks

Chapter 5. MLflow Tracing for GenAI Application Observability

In Chapter 4, we built a tool-calling agent to answer commonly asked questions about Unity Airways. This is a relatively simple setup consisting of two components: a vector database and an LLM. But even with this simple setup, our experience when working with practitioners showed that many of them struggled to debug quality issues. They are aware of the typical pitfalls like poor document chunking, weak retrieval, or ineffective prompting. What we’ve found, though, is that the real challenge often lies in the lack of tools to help them systematically diagnose and fix these quality issues.

This is where MLflow Tracing can make a real difference. Built on open standards, MLflow Tracing captures detailed information for every step in your GenAI applications. This enables you to:

• Check each step and behavior of an agent.

• Pinpoint source of bugs and unexpected behavior.

• Evaluate and improve application quality.

• Monitor performance and optimize costs.

Tracing is becoming increasingly essential as the industry starts to build more complex agentic applications (further explored in Chapter 7) composed of multiple components, as compared to the simple example in Chapter 4.

In this chapter, we’ll cover the fundamentals of MLflow Tracing, which will be essential for observability in your GenAI applications.

Understanding MLflow Tracing

The purpose of MLflow Tracing is to provide a tool for detailed tracking and visualization ...