MLOps with Databricks

Chapter 5. Machine Learning Model Deployment

So far, we’ve explored the individual stages of building and deploying a ML model: data preprocessing, feature engineering, model training, evaluation, and serving. In this chapter, we’ll shift our focus to orchestration and look at how to connect these steps into an automated pipeline that runs reliably in production.

We’ll walk through how to structure end-to-end ML workflows on Databricks, including platform components, workspace design, permission management, and deployment strategies. By the end of the chapter, you’ll understand how to design your ML pipeline using best practices that are suited to the Databricks environment.

Let’s begin with the orchestration of our machine learning pipelines. In Chapter 2, we introduced DABs and examined the structure of the databricks.yml file. What we have not yet looked at are the resources that get deployed when the databricks bundle deploy command runs.

The resources folder is referenced in the bundle definition and includes the following files:

ml_pipeline.yml

This is a Lakeflow job that defines the whole ML pipeline.

ml_monitoring.yml

This is a Lakeflow job that updates the monitoring table.

alert.yml

This file contains alert-related resources.

In this chapter, we focus on the ML pipeline, and in Chapter 6, we’ll turn to monitoring and alerting resources.

Machine Learning Pipeline

Before we examine the structure of the ml_pipeline.yml file, let’s clarify the objective. The ...