Practical MLflow for Generative AI on Databricks

Chapter 9. Production Monitoring with MLflow

In Chapter 8, we successfully deployed our GenAI application as a Databricks model serving endpoint. This was a major milestone, moving the application from a development or testing environment into a production-ready state where it can be accessed and utilized by other services and users. Now that the application is deployed and serving requests, does that mean you’re finished and ready to move on to the next project? Not quite.

From our experience, we found that many teams focus so heavily on achieving a successful initial launch that they fail to plan for what happens after the application is released into production. Launching a GenAI application into production is only the beginning. What comes next is often even more critical as the application is now operating in a higher-stakes, more dynamic environment, with more users (almost a total opposite of the controlled environment during development and testing). A GenAI application’s journey does not end at production launch; it is an iterative process that requires continuous monitoring, evaluation, and improvement.

The long-term success depends on strong production observability. How does the application’s quality evolve over time? How can errors be detected before users notice them? What feedback are end users providing? What is the application’s token consumption? How can the application adapt to evolving user needs? Answering these questions is essential, and MLflow provides ...