book

Generative AI Solutions with Databricks

by Alan L. Dennis, Mahesh Das, Vipul Choudhary, Thomas Hart

March 2027

Intermediate to advanced

450 pages

4h 26m

English

O'Reilly Media, Inc.

Read now

Unlock full access

1. Databricks Data Intelligence Platform Components for AI Solutions
Need for a Data Intelligence Platform in the Age of Generative AIChallenges of Traditional Data Architectures for AIHow Generative AI Amplifies These ChallengesThe “Lakehouse” Concept Revisited in the Context of AIDatabricks as the Foundation for Generative AI SolutionsFoundation of Core Data Management & Governance: Unity Catalog & Delta LakeDelta Lake: The Reliable Data Foundation for the LakehouseProcessing and Compute Power: Photon and Databricks SQLPhoton: The High-Performance Query EngineMachine Learning Operations (MLOps) and AI Development: Databricks Machine LearningOperationalizing and Monitoring AI Solutions: Databricks Lakehouse MonitoringAutomated Data and Model Quality MonitoringPerformance MonitoringExplainability and Bias DetectionCustomizable Alerts and DashboardsRole in Generative AI SolutionsThe Databricks Data Intelligence Platform in Action: A Generative AI WorkflowIllustrative Example: Building a RAG (Retrieval Augmented Generation) SystemHow Components Interoperate for Seamless AI DevelopmentConclusionRecap of Key Components and Their ValueThe Power of the Unified Data Intelligence PlatformLooking Ahead
2. Human-Computer Interaction Considerations, Requirements, Design, and Data Interactions
Human-Computer Interaction (HCI) and AIBrief Overview of HCIHCI and GenAIKey HCI TermsSolution FoundationsBusiness ProblemScope, Approach, and ObjectivesUsers and PersonasSuccess CriteriaAI’s Impact on HCIRequirementsHow to Gather RequirementsDocumentation Approach and ProcessGenerated DocumentationIdentify Required Source DataEvaluationFunctional and Nonfunctional RequirementsSystem OutputsPrepare for Analysis and DesignAnalysis and DesignAnalysisDesignTypes of Human and AI InteractionContext is CriticalInterfacesData and Tool InventorySecurity, Performance, and ScalabilityStakeholders Documentation ReviewPlanning, Designing, and Building for ChangeStrategic and Organizational ConsiderationsArchitectural and Technical FlexibilityData, Models, and AgentsOperations, Monitoring, and ScalabilityConclusion
3. Agents, Tools and Function Calling
Background/TheoryLLM as mini Human BrainHow It Works:Overview of Agentic FrameworkCore Parts of Agentic FrameworkToolsCommon Tools in DatabricksDefining a Tool in PythonGuided vs. Unguided AgentsGuided AgentsUnguided AgentsExample: Data Deletion RequestArchitecture1. The Input Prompt2. The Central Agent3. Memory: Short-Term vs. Long-Term4. The Planner5. Tools - Execution Layer6. Response Generation7. Governance and SecurityPutting It All TogetherRouter AgentsTool Calling Agents - ReAct ArchitectureDatabricks Implementation:Custom Agent Architecture (human in the loop)Frameworks like - Mosaic AI Agent FrameworkConclusionReferences
4. Other GenAI patterns on Databricks: Code generation, testing, and AI/BI data products
Introduction: Expanding the scope of GenAIBeyond the chatbot: Integrating GenAI into core workflowsThe platform advantage: Bridging the “Context Gap”Pattern 1: Intelligent code generation and co-programming1. The Databricks Assistant: A schema-aware sidekick2. Automated pipeline generation: From intent to DLT3. Legacy code migration: Modernizing the core4. Documentation-as-code: Illuminating the “dark data”Why this matters for the CobaltWorks executivePattern 2: AI-driven testing and quality assurance1. Synthetic data generation: Mirroring the factory floor2. Automated unit test generation: Scaling reliability3. AI-suggested data quality rules (DLT Expectations)Technical deep dive: The CobaltWorks synthetic data generatorWhy: De-risking InnovationPattern 3: AI/BI – The Evolution of Data Products1. From Dashboards to Dialogue: The end of “Request Queues”2. Curating the Semantic Layer: Teaching the AI “Cobalt-speak”3. AI-augmented insights: Root cause analysisTechnical Implementation: Building the “Genie Space”The executive “why”: The Speed of DecisionThe Human-in-the-loop (HITL): Governance for CobaltWorksThe reviewer workflow:Practical case study: The “supplier onboarding” sprintStep 1: Intelligent ingestion (Pattern 1)Step 2: AI-generated quality testing (Pattern 2)Step 3: Enabling the business (Pattern 3)Case study: The Apex Haulers integration sprintThe scenarioStep 1: GenAI-powered ingestionStep 2: Automated data quality suiteStep 3: Deploying the Genie spaceExercise wrap-upSummary: The new standard for data products
5. Governance of AI and Multi-Modal Systems
IntroductionBrief review of medallion architecture for AIThe bronze layer: Landing raw multi-modal assetsThe silver layer: The governance filterAdvanced multi-modal processing: Acoustic telemetryRow and column level security for AI training dataThe gold layer: Curated business intelligence for AIDeepSeek, PII, and other reasons for sleepless nightsThe importance of governance in an intelligent data platformThreat modeling the multi-modal pipelineInfrastructure as Code (IaC) for Unity Catalog GovernanceDefining the security perimeter via TerraformAdvanced governance of models using Unity CatalogCentralized discovery and namespaceUnified access control via SQLAttribute-Based Access Control (ABAC) for AIModel aliases and automated webhooks: The modern lifecycleLineage, data, and AI solutionsThe end-to-end AI lineage chainProgrammatic lineage extraction for compliance auditsImpact analysis: Incident response for AIPlatform security and governanceCustomer-Managed Keys (CMK) and EncryptionNetwork security: Private Link and IP access listsDisaster recovery and multi-region AI governanceObservability via inference tables and system tablesAI gateways and shadow ITThe Mosaic AI Gateway solutionSemantic caching for latency and cost reductionRoute optimization and fallback modelsImplementing AI guardrails and custom evaluators (LLM-as-a-Judge)Deploying AI via CI/CD: Declarative Automation Bundles (DABs)Managing multiple environments with DABsIncident Response Playbook: A CobaltWorks Case StudyThe IncidentThe InvestigationRoot Cause Analysis via LineageRemediationEthics, legality, and licensing concernsEthical AI and evaluating vision embeddings for driftGDPR and the “Right to be Forgotten” in AIRegulatory compliance and the EU AI ActNavigating licensing concerns: The “open source” vs. “open weights” dilemmaEnterprise AI Production Readiness ChecklistSecurity and GovernanceReliability and High AvailabilityObservability and PerformanceD. Cost Management and LifecycleTroubleshooting and Operational RunbooksResolving Out-Of-Memory (OOM) errors during multi-modal ingestionInvestigating a Lakehouse Monitor drift alertDebugging an overactive AI GuardrailConclusion
6. Assembling and Deploying
Importance of Automate DeploymentGetting Things Deployed to Production Without Violating Data PoliciesEnvironmentsWhy Some Solutions Only Work in Non-Production EnvironmentsPromote Code or ModelsUse-Case Logic: Where the Product LivesPackaging and Deployment ToolsAutomated CI/CDGit FoldersDevOps Pipelines and GitHub ActionsDeclarative Automation BundlesDeploying Databricks ServicesMLflowDatabricks Model ServingAI GatewayVector Search IndexesDatabricks AppsData Ingestion and RefinementConclusion

Content preview from Generative AI Solutions with Databricks

Chapter 2. Human-Computer Interaction Considerations, Requirements, Design, and Data Interactions

You might be eager to delve into the code and quickly add GenAI features to your projects. However, in our experience, technology is a secondary concern for building what people want. Before we log in to Databricks and start building GenAI solutions, in this chapter we will take a step back and review the basics. Our goal is to make sure real user needs lead the design, and to prevent the project from falling into “Proof of Concept (PoC) purgatory,” from which few make it to production. We start with an overview of human-computer interaction and the impact AI has had on the field. Then we discuss requirements, analysis, and design that can help organizations move from reports, dashboards, and notifications to generative and agentic artificial intelligence solutions.

Upon completing this chapter, you will be able to:

Learn the relationship between Human Computer Interaction (HCI) and AI. You will ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9798341639881Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Generative AI Solutions with Databricks

by Alan L. Dennis, Mahesh Das, Vipul Choudhary, Thomas Hart

Chapter 2. Human-Computer Interaction Considerations, Requirements, Design, and Data Interactions

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.