MLOps with Databricks

Chapter 9. Deploying in Monitoring LLM-Based Systems

In this chapter, we explore how to deploy an agent and the infrastructure around it. There is no single “correct” architecture as the right approach depends on the use case, latency requirements, and governance constraints.

Many agent use cases are naturally batch oriented. Agents can enrich newly ingested documents with summaries or metadata, classify support tickets, and generate offline reports. In these scenarios, running the agent as part of a scheduled batch job, much like how we tested it earlier, is often sufficient, without requiring a real-time endpoint.

Another pattern is embedding the agent directly into an application codebase. While convenient at first, this can create separation-of-concerns issues by tightly coupling application logic with agent orchestration and configuration. Over time, this makes versioning, monitoring, and governance more difficult, especially when different teams own the agent and application layers.

A more robust, production-ready approach is to deploy the agent behind a Model Serving endpoint. This introduces a clear separation between the agent and the consuming application, which makes it easier to evolve the agent independently, manage versions, and support multiple clients.

In this chapter, we follow this approach and deploy the agent using a serving endpoint.

Deploying a Databricks Model Serving Endpoint

In Chapter 4, we deployed several Model Serving endpoints using Databricks SDK. ...