book

Building Modern Data Applications Using Databricks Lakehouse

Name: Building Modern Data Applications Using Databricks Lakehouse
Author: Will Girten
ISBN: 9781801073233

by Will Girten

October 2024

Intermediate to advanced

246 pages

6h 22m

English

Packt Publishing

Read now

Unlock full access

Building Modern Data Applications Using Databricks Lakehouse
Contributors
About the author
About the reviewer
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesConventions usedGet in touchShare Your ThoughtsDownload a free PDF copy of this book
Part 1:Near-Real-Time Data Pipelines for the Lakehouse
Chapter 1: An Introduction to Delta Live Tables
Technical requirementsThe emergence of the lakehouseThe Lambda architectural patternIntroducing the medallion architectureThe Databricks lakehouseThe maintenance predicament of a streaming applicationWhat is the DLT framework?How is DLT related to Delta Lake?Introducing DLT conceptsStreaming tablesMaterialized viewsViewsPipelinePipeline triggersWorkflowTypes of Databricks computeDatabricks RuntimeUnity CatalogA quick Delta Lake primerThe architecture of a Delta tableThe contents of a transaction commitSupporting concurrent table reads and writesTombstoned data filesCalculating Delta table stateTime travelTracking table changes using change data feedA hands-on example – creating your first Delta Live Tables pipelineSummary
Chapter 2: Applying Data Transformations Using Delta Live Tables
Technical requirementsIngesting data from input sourcesIngesting data using Databricks Auto LoaderScalability challenge in structured streamingUsing Auto Loader with DLTApplying changes to downstream tablesAPPLY CHANGES commandThe DLT reconciliation processPublishing datasets to Unity CatalogWhy store datasets in Unity Catalog?Creating a new catalogAssigning catalog permissionsData pipeline settingsThe DLT product editionPipeline execution modeDatabricks runtimePipeline cluster typesA serverless compute versus a traditional computeLoading external dependenciesData pipeline processing modesHands-on exercise – applying SCD Type 2 changesSummary
Chapter 3: Managing Data Quality Using Delta Live Tables
Technical requirementsDefining data constraints in Delta LakeUsing temporary datasets to validate data processingAn introduction to expectationsExpectation compositionHands-on exercise – writing your first data quality expectationActing on failed expectationsHands-on example – failing a pipeline run due to poor data qualityApplying multiple data quality expectationsDecoupling expectations from a DLT pipelineHands-on exercise – quarantining bad data for correctionSummary
Chapter 4: Scaling DLT Pipelines
Technical requirementsScaling compute to handle demandHands-on example – setting autoscaling properties using the Databricks REST APIAutomated table maintenance tasksWhy auto compaction is importantVacuuming obsolete table filesMoving compute closer to the dataOptimizing table layouts for faster table updatesRewriting table files during updatesData skipping using table partitioningDelta Lake Z-ordering on MERGE columnsImproving write performance using deletion vectorsServerless DLT pipelinesIntroducing Enzyme, a performance optimization layerSummary
Part 2:Securing the Lakehouse Using the Unity Catalog

Chapter 5: Mastering Data Governance in the Lakehouse with Unity Catalog
Technical requirementsUnderstanding data governance in a lakehouseIntroducing the Databricks Unity CatalogA problem worth solvingAn overview of the Unity Catalog architectureUnity Catalog-enabled cluster typesUnity Catalog object modelEnabling Unity Catalog on an existing Databricks workspaceIdentity federation in Unity CatalogData discovery and catalogingTracking dataset relationships using lineageObservability with system tablesTracing the lineage of other assetsFine-grained data accessHands-on example – data masking healthcare datasetsSummary
Chapter 6: Managing Data Locations in Unity Catalog
Technical requirementsCreating and managing data catalogs in Unity CatalogManaged data versus external dataSaving data to storage volumes in Unity CatalogSetting default locations for data within Unity CatalogIsolating catalogs to specific workspacesCreating and managing external storage locations in Unity CatalogStoring cloud service authentication using storage credentialsQuerying external systems using Lakehouse FederationHands-on lab – extracting document text for a generative AI pipelineGenerating mock documentsDefining helper functionsChoosing a file format randomlyCreating/assembling the DLT pipelineSummary
Chapter 7: Viewing Data Lineage Using Unity Catalog
Technical requirementsIntroducing data lineage in Unity CatalogTracing data origins using the Data Lineage REST APIVisualizing upstream and downstream transformationsIdentifying dependencies and impactsHands-on lab – documenting data lineage across an organizationSummary
Part 3:Continuous Integration, Continuous Deployment, and Continuous Monitoring
Chapter 8: Deploying, Maintaining, and Administrating DLT Pipelines Using Terraform
Technical requirementsIntroducing the Databricks provider for TerraformSetting up a local Terraform environmentImporting the Databricks Terraform providerConfiguring workspace authenticationDefining a DLT pipeline source notebookApplying workspace changesConfiguring DLT pipelines using TerraformnamenotificationchanneldevelopmentcontinuouseditionphotonconfigurationlibraryclustercatalogtargetstorageAutomating DLT pipeline deploymentHands-on exercise – deploying a DLT pipeline using VS CodeSetting up VS CodeCreating a new Terraform projectDefining the Terraform resourcesDeploying the Terraform projectSummary
Chapter 9: Leveraging Databricks Asset Bundles to Streamline Data Pipeline Deployment
Technical requirementsIntroduction to Databricks Asset BundlesElements of a DAB configuration fileSpecifying a deployment modeDatabricks Asset Bundles in actionUser-to-machine authenticationMachine-to-machine authenticationInitializing an asset bundle using templatesHands-on exercise – deploying your first DABHands-on exercise – simplifying cross-team collaboration with GitHub ActionsSetting up the environmentConfiguring the GitHub ActionTesting the workflowVersioning and maintenanceSummary
Chapter 10: Monitoring Data Pipelines in Production
Technical requirementsIntroduction to data pipeline monitoringExploring ways to monitor data pipelinesUsing DBSQL Alerts to notify data validityPipeline health and performance monitoringHands-on exercise – querying data quality events for a datasetData quality monitoringIntroducing Lakehouse MonitoringHands-on exercise – creating a lakehouse monitorBest practices for production failure resolutionHandling pipeline update failuresRecovering from table transaction failureHands-on exercise – setting up a webhook alert when a job runs longer than expectedSummary
Index
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your ThoughtsDownload a free PDF copy of this book

Overview

This book, "Building Modern Data Applications Using Databricks Lakehouse," provides a comprehensive guide for data professionals to master the Databricks platform. You'll learn to effectively build, deploy, and monitor robust data pipelines with Databricks' Delta Live Tables, empowering you to manage and optimize cloud-based data operations effortlessly.

What this Book will help me do

Understand the foundations and concepts of Delta Live Tables and its role in data pipeline development.
Learn workflows to process and transform real-time and batch data efficiently using the Databricks lakehouse architecture.
Master the implementation of Unity Catalog for governance and secure data access in modern data applications.
Deploy and automate data pipeline changes using CI/CD, leveraging tools like Terraform and Databricks Asset Bundles.
Gain advanced insights in monitoring data quality and performance, optimizing cloud costs, and managing DataOps tasks effectively.

Author(s)

Will Girten, the author, is a seasoned Solutions Architect at Databricks with over a decade of experience in data and AI systems. With a deep expertise in modern data architectures, Will is adept at simplifying complex topics and translating them into actionable knowledge. His books emphasize real-time application and offer clear, hands-on examples, making learning engaging and impactful.

Who is it for?

This book is geared towards data engineers, analysts, and DataOps professionals seeking efficient strategies to implement and maintain robust data pipelines. If you have a basic understanding of Python and Apache Spark and wish to delve deeper into the Databricks platform for streamlining workflows, this book is tailored for you.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Building AI Agents with LangGraph: Creating Agentic Applications with Large Language Models and LangGraph

Publisher Resources

ISBN: 9781801073233

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills