ML and Generative AI in the Data Lakehouse

Preface

AI is no longer a research curiosity confined to academic papers and proof-of-concept demos. It is reshaping how organizations derive value from data, how software systems are built, and how decisions are made at scale. At the center of this transformation sits a powerful and rapidly maturing platform: the data lakehouse. Combining the openness and flexibility of a data lake with the governance and performance of a data warehouse, the lakehouse has become the natural home for machine learning (ML) and, more recently, for the generative AI (GenAI) and agentic systems that are defining the next era of intelligent applications.

What makes this moment so exciting is the pace of convergence. Traditional machine learning, large language models, feature engineering, model serving, AI agents, and the protocols that connect them are all maturing simultaneously, and the Databricks Data Intelligence Platform sits at the intersection of every one of these disciplines. This book is about that intersection. It is about understanding not just what each piece does in isolation, but how they fit together to form a coherent, production-grade AI and ML practice built on the lakehouse.

Whether you are training your first scikit-learn model on Databricks, building a Retrieval-Augmented Generation (RAG) pipeline, orchestrating multiagent workflows, or designing the integration layer that connects your language models to enterprise data systems through the Model Context Protocol (MCP), this ...