book

Building Modern Data Applications Using Databricks Lakehouse

Name: Building Modern Data Applications Using Databricks Lakehouse
Author: Will Girten
ISBN: 9781801073233

by Will Girten

October 2024

Intermediate to advanced

246 pages

6h 22m

English

Packt Publishing

Read now

Unlock full access

Building Modern Data Applications Using Databricks Lakehouse
Contributors
About the author
About the reviewer
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesConventions usedGet in touchShare Your ThoughtsDownload a free PDF copy of this book
Part 1:Near-Real-Time Data Pipelines for the Lakehouse
Chapter 1: An Introduction to Delta Live Tables
Technical requirementsThe emergence of the lakehouseThe Lambda architectural patternIntroducing the medallion architectureThe Databricks lakehouseThe maintenance predicament of a streaming applicationWhat is the DLT framework?How is DLT related to Delta Lake?Introducing DLT conceptsStreaming tablesMaterialized viewsViewsPipelinePipeline triggersWorkflowTypes of Databricks computeDatabricks RuntimeUnity CatalogA quick Delta Lake primerThe architecture of a Delta tableThe contents of a transaction commitSupporting concurrent table reads and writesTombstoned data filesCalculating Delta table stateTime travelTracking table changes using change data feedA hands-on example – creating your first Delta Live Tables pipelineSummary
Chapter 2: Applying Data Transformations Using Delta Live Tables
Technical requirementsIngesting data from input sourcesIngesting data using Databricks Auto LoaderScalability challenge in structured streamingUsing Auto Loader with DLTApplying changes to downstream tablesAPPLY CHANGES commandThe DLT reconciliation processPublishing datasets to Unity CatalogWhy store datasets in Unity Catalog?Creating a new catalogAssigning catalog permissionsData pipeline settingsThe DLT product editionPipeline execution modeDatabricks runtimePipeline cluster typesA serverless compute versus a traditional computeLoading external dependenciesData pipeline processing modesHands-on exercise – applying SCD Type 2 changesSummary
Chapter 3: Managing Data Quality Using Delta Live Tables
Technical requirementsDefining data constraints in Delta LakeUsing temporary datasets to validate data processingAn introduction to expectationsExpectation compositionHands-on exercise – writing your first data quality expectationActing on failed expectationsHands-on example – failing a pipeline run due to poor data qualityApplying multiple data quality expectationsDecoupling expectations from a DLT pipelineHands-on exercise – quarantining bad data for correctionSummary
Chapter 4: Scaling DLT Pipelines
Technical requirementsScaling compute to handle demandHands-on example – setting autoscaling properties using the Databricks REST APIAutomated table maintenance tasksWhy auto compaction is importantVacuuming obsolete table filesMoving compute closer to the dataOptimizing table layouts for faster table updatesRewriting table files during updatesData skipping using table partitioningDelta Lake Z-ordering on MERGE columnsImproving write performance using deletion vectorsServerless DLT pipelinesIntroducing Enzyme, a performance optimization layerSummary
Part 2:Securing the Lakehouse Using the Unity Catalog

Chapter 5: Mastering Data Governance in the Lakehouse with Unity Catalog
Technical requirementsUnderstanding data governance in a lakehouseIntroducing the Databricks Unity CatalogA problem worth solvingAn overview of the Unity Catalog architectureUnity Catalog-enabled cluster typesUnity Catalog object modelEnabling Unity Catalog on an existing Databricks workspaceIdentity federation in Unity CatalogData discovery and catalogingTracking dataset relationships using lineageObservability with system tablesTracing the lineage of other assetsFine-grained data accessHands-on example – data masking healthcare datasetsSummary
Chapter 6: Managing Data Locations in Unity Catalog
Technical requirementsCreating and managing data catalogs in Unity CatalogManaged data versus external dataSaving data to storage volumes in Unity CatalogSetting default locations for data within Unity CatalogIsolating catalogs to specific workspacesCreating and managing external storage locations in Unity CatalogStoring cloud service authentication using storage credentialsQuerying external systems using Lakehouse FederationHands-on lab – extracting document text for a generative AI pipelineGenerating mock documentsDefining helper functionsChoosing a file format randomlyCreating/assembling the DLT pipelineSummary
Chapter 7: Viewing Data Lineage Using Unity Catalog
Technical requirementsIntroducing data lineage in Unity CatalogTracing data origins using the Data Lineage REST APIVisualizing upstream and downstream transformationsIdentifying dependencies and impactsHands-on lab – documenting data lineage across an organizationSummary
Part 3:Continuous Integration, Continuous Deployment, and Continuous Monitoring
Chapter 8: Deploying, Maintaining, and Administrating DLT Pipelines Using Terraform
Technical requirementsIntroducing the Databricks provider for TerraformSetting up a local Terraform environmentImporting the Databricks Terraform providerConfiguring workspace authenticationDefining a DLT pipeline source notebookApplying workspace changesConfiguring DLT pipelines using TerraformnamenotificationchanneldevelopmentcontinuouseditionphotonconfigurationlibraryclustercatalogtargetstorageAutomating DLT pipeline deploymentHands-on exercise – deploying a DLT pipeline using VS CodeSetting up VS CodeCreating a new Terraform projectDefining the Terraform resourcesDeploying the Terraform projectSummary
Chapter 9: Leveraging Databricks Asset Bundles to Streamline Data Pipeline Deployment
Technical requirementsIntroduction to Databricks Asset BundlesElements of a DAB configuration fileSpecifying a deployment modeDatabricks Asset Bundles in actionUser-to-machine authenticationMachine-to-machine authenticationInitializing an asset bundle using templatesHands-on exercise – deploying your first DABHands-on exercise – simplifying cross-team collaboration with GitHub ActionsSetting up the environmentConfiguring the GitHub ActionTesting the workflowVersioning and maintenanceSummary
Chapter 10: Monitoring Data Pipelines in Production
Technical requirementsIntroduction to data pipeline monitoringExploring ways to monitor data pipelinesUsing DBSQL Alerts to notify data validityPipeline health and performance monitoringHands-on exercise – querying data quality events for a datasetData quality monitoringIntroducing Lakehouse MonitoringHands-on exercise – creating a lakehouse monitorBest practices for production failure resolutionHandling pipeline update failuresRecovering from table transaction failureHands-on exercise – setting up a webhook alert when a job runs longer than expectedSummary
Index
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your ThoughtsDownload a free PDF copy of this book

Content preview from Building Modern Data Applications Using Databricks Lakehouse

4 Scaling DLT Pipelines

In this chapter, we’re going to look at several methods for scaling your Delta Live Tables (DLT) pipelines to handle the processing demands of a typical production environment. We’ll cover several aspects of tuning your DLT pipelines, from optimizing the DLT cluster settings so that your pipelines can quickly scale to handle the spikes of heavy processing demand to looking at ways we can optimize the data layout of the underlying tables in cloud storage. By the end of this chapter, you should have mastered how DLT clusters can automatically scale out to handle demand. You should also have a good understanding of the impact that table maintenance tasks, which are automatically run in the background by the DLT system, have ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Building AI Agents with LangGraph: Creating Agentic Applications with Large Language Models and LangGraph

Publisher Resources

ISBN: 9781801073233

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Building Modern Data Applications Using Databricks Lakehouse

by Will Girten

4

Scaling DLT Pipelines

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.