ML and Generative AI in the Data Lakehouse

Chapter 9. AI Agents in the Lakehouse

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 9th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

In the previous chapters, you worked with models, prompts, and retrieval pipelines as individual building blocks. This chapter shifts the unit of design from a single model call to a larger, stateful system. In an agentic system, the model is only one component. Around it sit memory, tool interfaces, execution logic, and control policies that let the system pursue a goal over multiple steps.

This distinction matters in the Lakehouse. Many enterprise tasks cannot be completed with a single prompt, no matter how well written. Answering a business question may require retrieving documentation, querying governed data, running calculations, checking results, and assembling a final explanation. Those tasks benefit from a system that can decompose work, choose actions, and adapt as new information arrives.

An AI agent is therefore best understood as an orchestrated application built around a model, not as a model in isolation. The Lakehouse provides the ingredients ...