ML and Generative AI in the Data Lakehouse

Chapter 4. End-to-End ML with MLflow

Building an ML model is often the easy part. The harder challenges, such as tracking experiments, managing model versions, governing artifacts, and deploying to production, determine whether ML delivers real business value or remains a proof-of-concept.

This chapter walks through a complete ML workflow on the Databricks Lakehouse platform, using hotel booking cancellation prediction as a representative example. While the specific use case involves forecasting no-shows to help hotels optimize occupancy, the patterns you’ll learn apply broadly to any classification problem: customer churn, fraud detection, equipment failure prediction, and beyond.

You’ll work through the entire ML lifecycle, from exploratory analysis through production deployment, while learning how Delta Lake, MLflow, and Unity Catalog combine to make ML reproducible, governable, and operationally ready. By the end of this chapter, you’ll have a template for building production ML pipelines in the Lakehouse.

ML in Context

ML encompasses a broad spectrum of techniques, each suited to different types of business problems. Before diving into the hands-on implementation that forms the core of this chapter, it’s worth briefly orienting ourselves on when different approaches apply.

Matching ML Approaches to Business Problems

At the highest level, ML problems fall into several categories:

Classification

Predicts categorical outcomes, which are discrete labels or classes. Binary ...