book

ML and Generative AI in the Data Lakehouse

Name: ML and Generative AI in the Data Lakehouse
Author: Bennie Haelen
ISBN: 9781098178499

by Bennie Haelen

June 2026

Intermediate to advanced

430 pages

12h 4m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Preface
Who Should Read This BookWhy I Wrote This BookNavigating This BookConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
1. An Overview of Machine Learning, AI, and GenAI
What Is AI?The Evolution of AIThe Subdomains of MLSupervised LearningUnsupervised LearningSemi-Supervised LearningReinforcement LearningAn Overview of GenAITypes of Generative ModelsGenAI Applications and Use CasesResponsible AIData PrivacyAccountabilityTransparencyExplainabilityPrivacy RisksSocietal ImpactContent Integrity and AccuracySummary
2. The Databricks Data Intelligence Platform
Why the Databricks Platform?Unified Lakehouse ArchitectureCollaborative Tools and Managed InfrastructureComprehensive ML EcosystemTraining, Inference, and the GenAI Paradigm ShiftEnterprise Security and GovernanceDatabricks Platform OverviewData Storage and ReliabilityData Governance and SecurityData Engineering and PipelinesAnalytics and Business IntelligenceML and AIThe Databricks RuntimeOptimized Spark EngineIntegrated ML LibrariesLanguage SupportAdvanced Analytics and AI CapabilitiesRuntime SummaryClusters, Workspaces, and NotebooksUnderstanding Databricks ClustersNavigating Databricks WorkspacesLeveraging Databricks NotebooksIntegrating Clusters, Workspaces, and NotebooksSetting Up an ML EnvironmentUnderstanding the Databricks ML WorkspacePrerequisitesCreating an Account and a WorkspaceSetting Up Clusters for MLRuntime SelectionDriver and Worker NodesExtending the ML Runtime with Additional LibrariesSummary
3. An Introduction to ML on Databricks
The End-to-End ML EnvironmentData PreparationModel TrainingModel Creation and ValidationModel ProductionalizationManaging the ML Lifecycle with MLflowChallenge 1: Experiment Tracking and ReproducibilityChallenge 2: Model Versioning and LineageChallenge 3: Collaboration Across TeamsChallenge 4: Model Packaging and DeploymentMLflow in the MLOps LandscapeMLflow ComponentsExample WorkflowSummary
4. End-to-End ML with MLflow
ML in ContextMatching ML Approaches to Business ProblemsFocus on Implementation, Not Model SelectionWhat You Will BuildExploratory Data AnalysisLoading the DatasetCreating a VolumeUploading the Dataset to a VolumeExploring the DatasetFeature EngineeringHandling Missing ValuesData CleansingSaving to Unity CatalogModel TrainingCreating an MLflow ExperimentModel BuildingModel Training with MLflow TrackingModel PredictionsPreparing the Test DataLoading the Model from the Unity Catalog RegistryLoading as a Native scikit-learn ModelInspecting Feature ImportancesLoading the Model as a Spark UDFLoading the Model as a Generic PyFunc ModelWhen to Use Each Loading MethodModel DeploymentVerifying the Registered ModelReviewing Model Metadata and LineageManaging the Model Lifecycle with AliasesDeploying to a Model Serving EndpointQuerying the EndpointUpdating the Endpoint with a New VersionModel InferencingPreparing the Test DataRetrieving Workspace CredentialsDefining the Scoring FunctionLoading the Champion Model for Local ComparisonCalling the Endpoint and Comparing PredictionsVerifying Prediction ConsistencyEvaluating Accuracy and Prediction DistributionSummary
5. Feature Engineering in the Unity Catalog
The Role of Feature Engineering in MLDelta Tables: The Foundation of Feature EngineeringFeature Tables: The Core of the Feature StoreIntegration with Unity CatalogFeature Engineering in the LakehouseOffline and Online StoresFeature Consistency and ReusabilityML Workflow with the Feature StoreStep 1: Creating and Managing Feature TablesStep 2: Training with Features from the Feature StoreStep 3: Real-Time InferenceExample: Lending Club Feature StoreIntroduction to the Lending Club DatasetApproachStep 1: Loading and Preprocessing the Lending Club DatasetStep 2: Build Your Feature TableStep 3: Building and Training Dataset Using FeatureEngineeringClientStep 4: Building the Model with Features from the Feature TableStep 5: Model VisualizationsStep 6: Logging and Registering the Model in Unity CatalogSummary
6. ML at Scale
The Role of Spark in MLScaling ML with SparkDistributed Data Processingpandas API on SparkUsing Spark MLlibDistributed Deep Learning: TorchDistributor, DeepSpeed, and RayParallelized Model TrainingSpark Structured Streaming for Real-Time MLHyperparameter Tuning with Optuna, Hyperopt, and Ray TuneThe Role of Delta Lake in Scalable MLACID Transactions and Schema EnforcementTime Travel and Data VersioningEfficient Incremental Data ProcessingSynergy with Spark and DatabricksPlatform Capabilities for ML at ScaleThe Shift Away from AutoMLThe Broader Mosaic AI DirectionAccelerating ML Workloads with PhotonMLflow Integration at ScaleServerless Compute for ML WorkloadsUnity Catalog Integration for ML at ScaleCost Management at ScaleML at Scale Use CaseStep 1: Data Loading and Initial Type ConversionsStep 2: Data PreprocessingStep 3: Feature EngineeringStep 4: Feature Transformations and VectorizationStep 5: Efficient Storage with Delta LakeStep 6: Model Training at ScaleStep 7: Hyperparameter Tuning with TrainValidationSplitStep 8: Distributed Hyperparameter Tuning with OptunaStep 9: Log and Register the Best Model in Unity CatalogUse Case ResultsSummary
7. GenAI in the Lakehouse: Foundations and Architecture
From Predictive to Generative IntelligenceGenAI ArchitectureThe Mathematics of Scale: Why Bigger Models Work BetterCompute-Optimal Training: The Chinchilla InsightPractical Implications for the LakehouseThe Transformer ArchitectureWhy Transformers Succeeded: Parallelism at ScaleThree Transformer VariantsCore Components of a Transformer BlockAutoregressive Generation: From Parallel Training to Sequential OutputImplications for the LakehouseModel Serving Optimization in DatabricksUnity Catalog Integration: Governance for Foundation ModelsInfrastructure Trade-Offs and RAGTokenization, Embeddings, and Context WindowsTokenization: Breaking Text into Learnable UnitsEmbeddings: From Tokens to GeometryContext Windows: The Memory BottleneckFrom Probabilities to Text: Decoding StrategiesFoundation Models as the New Transfer Learning ParadigmEmergent Capabilities at ScaleFrom Foundation Models to Production SystemsFoundation Models in the LakehouseGenAI in the LakehouseSummary
8. GenAI in a Databricks Lakehouse Environment
Prompt EngineeringUnderstanding Prompts and Prompt EngineeringAnatomy of a High-Quality PromptPrompt Engineering TechniquesPractical Tips for Effective PromptingThe AI Playground in DatabricksExploring the PlaygroundModel Exploration and Prompt DesignRetrieval-Augmented GenerationRAG DefinitionHow Do Language Models Learn Knowledge?Factual Recall with ContextVector StoresDatabricks AI Vector StorePractical RAG ImplementationPreparing Data for RAGExecuting Vector Retrieval via Databricks SQL or PySparkSummary
9. AI Agents in the Lakehouse
Fundamentals of AI AgentsWhat Is an AI Agent?Benefits of AgentsChoosing the Right Execution ModelCore Components of a Lakehouse AgentA Lakehouse Agent Reference ArchitectureAgent LifecycleMemory Approaches for Lakehouse AgentsAgent Memory ShapesMemory Design PatternsAgent PatternsReflectionTool UseReActPlanningMultiagent CollaborationBringing the Patterns TogetherFailure Modes and Controls for Lakehouse AgentsEvidence-Based Diagnostic Analytics with ReActThe Scenario: Asking the Agent “Why?”Overview of the Lakehouse ArchitectureReAct Logic and a Custom-Built ToolkitCode WalkthroughReAct Agent SummarySummary

10. The Model Context Protocol
What Is MCP?LLM Tool Standardization with MCPREST Versus MCP: Static Contracts Versus Dynamic DiscoveryThe REST Model: Static ContractsThe MCP Model: Dynamic DiscoveryKey DifferencesMCP ArchitectureCore ComponentsThe Three Primary Server PrimitivesWhy This Architecture Matters in the LakehouseDeployment Modes and Runtime BehaviorLocal ModeRemote ModeScalability and FlexibilityClient/Server CommunicationPhase 1: Initialization and Capability NegotiationPhase 2: Discovery and InvocationExample: Weather API ServiceProgress Notifications and Long-Running OperationsWhy the Lifecycle MattersDatabricks Unity Catalog MCP ServerKey CapabilitiesServer Architecture and InitializationRun It Now!Main Solution ComponentsProject StructureDesign PrinciplesImplementation WalkthroughTesting and Interacting with the MCP ServerSummary
11. Agent-to-Agent Communication and the DSPy Framework
A2A ProtocolOverview and FoundationCore ArchitectureIntegration with Data LakehousesRelationship with MCPEnterprise Adoption and EcosystemDeclarative Self-Improving PythonCore ConceptsDSPy in the Data LakehouseDSPy Integration with MLOps and Model ManagementAdvanced DSPy Capabilities for Lakehouse WorkloadsA2A and DSPy Together: Multiagent Intelligence on the LakehouseExample: Retail Analytics Assistant on a Databricks LakehouseProblem OverviewWhy A2A + DSPy?Architecture OverviewExample InteractionRunning the ExampleSummary
About the Author

Content preview from ML and Generative AI in the Data Lakehouse

Chapter 11. Agent-to-Agent Communication and the DSPy Framework

As organizations modernize data lakehouse platforms to support GenAI, decision intelligence, and autonomous workflows, two technologies are increasingly shaping how we design and operationalize intelligent systems.

Agent2Agent (A2A) is an open protocol for inter-agent discovery, task coordination, and artifact exchange across heterogeneous agent implementations. Google announced A2A in April 2025, and the Linux Foundation launched the A2A project in June 2025 to provide neutral governance. As of this writing (early 2026), the protocol is still evolving, with new capabilities being added regularly.

DSPy (Declarative Self-Improving Python) is a framework for declarative, optimizable language-model programming that reduces reliance on brittle prompt engineering. DSPy emerged from Stanford NLP research, with the foundational concepts appearing in 2022, and the framework itself released in 2023. It has since become a leading approach ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781098178482Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

ML and Generative AI in the Data Lakehouse

by Bennie Haelen

Chapter 11. Agent-to-Agent Communication and the DSPy Framework

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.