book

Data Engineering with Azure Databricks

Name: Data Engineering with Azure Databricks
ISBN: 9781806106370

by Dmitry Foshin, Dmitry Anoshin, Tonya Chernyshova, Sergii Volodarskyi

April 2026

Intermediate to advanced

412 pages

10h 17m

English

Packt Publishing

Read now

Unlock full access

Preface
Free benefits with your book
Chapter 1: The Role of Azure Databricks in Modern Data Engineering
The evolution of data engineeringRise of Apache SparkMeet databricksAzure Databricks key conceptsDatabricks and competitorsReference architectures and use casesSummary
Chapter 2: Setting up an End-To-End Azure Databricks Environment
Governance foundation: Unity catalogWhy unity catalog was the need of the hourMetastoreAdministrative access and workspace provisioningCreate an admin userSet up a workspaceContinue building the platform: Introducing use casesConnecting to cloud storageUnity catalog storage integration overviewCreate the storage accountCreate containers in the storage accountCreate an access connector for Azure DatabricksCreate a Storage Credential Using a Managed IdentityCreate an external locationCatalog design considerations and creationCreating a volume: Staging location for raw filesCreate an external volume for stagingBronze, silver, gold tablesData qualityIdentity managementIdentity provisioningGroupsCreate groups and assign to workspacesCreate groups at the account level (UI)Assign groups to workspaces (UI)Entitlements, privileges and access controlManage entitlementsManage access control listsPrivilegesPermission matrixComputeServerlessClassic compute access modesStandard (shared) access modeDedicated (Single-User) access modeNo Isolation Shared Access ModePhotonSQL WarehousesSystem TablesDeleting resourcesSummary
Chapter 3: Data Ingestion Strategies for Azure Databricks
Technical requirementsUnderstanding batch ingestionCharacteristics of batch ingestionCommon Batch Ingestion ScenariosIngesting data from Azure Storage (ADLS and blob storage)Authentication methodsService principal with OAuth 2.0Reading different file formatsCSV filesJSON FilesParquet filesIncremental loading patternWorking example: ADLS batch ingestionUsing Azure Data Factory for ingestionCopy activity patternADF to Databricks patternsConnecting to relational databasesJDBC connectivity basicsSetting up sample data in Azure SQLFull table load exampleIncremental Loading PatternPerformance optimization tipsAzure Synapse AnalyticsIngesting data from REST APIsAuthentication methodsAPI keysHandling paginationRate limiting and error handlingWorking example: JSONPlaceholder API ingestionOther data sourcesAzure cosmos DBSFTP/FTP sourcesSummary
Chapter 4: Data Engineering with Apache Spark
Apache Spark architecture and execution modelCore components and cluster architectureApplications of spark architectureSpark driverExecutorsCluster managerExecution model and job schedulingMemory management and storageMemory configuration and optimizationAzure Databricks runtime enhancementsOptimizing spark jobs with partitioning and cachingData partitioning strategiesSelecting optimal partitioning strategiesCaching and persistence mechanismsCaching and persistence mechanismsAdvanced optimization techniquesWriting efficient PySpark and scala codeDataFrame API best practicesSQL optimization techniquesUser-Defined functions and performanceHandling Large-Scale data processing with sparkScaling strategies and cluster managementData skew and resource optimizationProduction deployment patternsSummary
Chapter 5: Building Real-Time Data Pipelines
What is Data Streaming?Streaming vs.Batch ProcessingCommon Streaming Use CasesTypes of Streaming: Real-Time vs.Near Real-TimeReal-Time StreamingNear Real-Time StreamingChoosing the Right TypeAzure Databricks and Spark Streaming CapabilitiesWhat is Apache Spark Structured Streaming?Azure Databricks Streaming FeaturesCommon Azure Streaming PatternsBuilding Your First Streaming PipelineSetting Up Your Azure AccountPrerequisitesCreating an Azure Event HubSetting Up Azure Blob StorageWriting Your First Streaming JobUsing Autoloader for File StreamingAutoloader vs.Event HubMonitoring and DebuggingUsing the Databricks UICommon Problems and SolutionsPerformance TipsAdvanced Streaming ConceptsManaging Stateful StreamingWhat is State in Streaming?When to Use Stateful StreamingStateful Operations in Spark Structured StreamingHandling Late Data and Fault ToleranceWatermarks: Managing Late-Arriving DataCheckpointing: Ensuring Fault ToleranceExactly-Once ProcessingPerformance OptimizationPartitioning for PerformanceCaching for PerformanceResource Tuning for StreamingAutoloader OptimizationSummary
Chapter 6: Working with Delta Lake: ACID Transactions and Schema Evolution
Technical requirementsIntroduction to delta lake: Why it mattersEnabling ACID transactions for reliable data processingIntroduction to delta logWhat do ACID guarantees mean?What role does the _delta_log play?When are transactions written to the Delta Log?Are failed operations recorded?How does Delta Lake handle rollbacks?Why is delta log the single source of truth?How does reading the Delta Log scale?How Delta Lake implements consistencyHow Delta Lake implements isolationHow Delta Lake implements durabilitySchema enforcement and evolutionWhy schema management mattersDelta Lake's approach to schema enforcementDelta Lake's approach to schema evolutionSchema evolution: Overwrite schema approachSchema evolution: Automated approachSchema evolution best practicesDelta versioning and time travelWhy time travel mattersHow does the Delta Lake version dataQuerying historical data with the SQL APICreating multiple table versionsQuerying by version numberQuerying by timestampRestore the table stateRetention and data lifecycleChange data capture in Delta LakeWhy CDC matters for analytical use casesDelta table as a target for CDCApproach using MERGE INTOApproach using AUTO CDCDelta table as a source of CDCEnabling change data feedHow to read changesStreaming queriesHow VACUUM affects change data feedSummary
Chapter 7: Automating Data Systems with Lakeflow Spark Declarative Pipelines
Technical requirementsIntroduction to LakeFlow Spark Declarative PipelinesThe Challenge with Traditional PipelinesKey benefits of Lakeflow SDPWhen to use Lakeflow SDCore conceptsBuilding your first Lakeflow pipelineThe example scenarioSetup: Preparing your dataThe complete pipeline codePipeline setup: Imports and configurationUnderstanding the bronze layerUnderstanding the silver layerUnderstanding the gold layerTables vs.materialized views in LakeFlowCreating and running the pipelineStep 1: Create the pipelineStep 2: Configure the pipelineStep 3: Start the pipelineStep 4: Monitor executionVerifying pipeline resultsViewing data quality metrics in the pipeline UIManaging data quality with expectationsTypes of expectationsImplementing effective quality rulesMonitoring quality metricsBest practices for data qualityOptimizing Lakeflow pipeline performanceAutomatic optimizationsPipeline configurationClustering strategiesOptimizing joins and aggregationsMonitoring Pipeline PerformanceSummary
Chapter 8: Orchestrating Data Workflows: From Notebooks to Production
Technical RequirementsUsing Lakeflow Jobs for Task SchedulingUnderstanding Lakeflow Jobs ArchitectureHands-on Job Creation and ConfigurationAdvanced Job Patterns and OptimizationIntegrating Databricks with Azure Data Factory and AirflowAzure Data Factory Integration ArchitectureHands-on ADF Pipeline DevelopmentApache Airflow Integration PatternsOrchestration Tool Comparison and SelectionBest Practices for Modularizing NotebooksNotebook Modularization PrinciplesHands-on Notebook Refactoring TechniquesProduction Notebook Patterns and StandardsComprehensive Monitoring InfrastructureHands-on Debugging Techniques and ToolsOperational Excellence and Continuous ImprovementSummary
Chapter 9: CI/CD and DevOps for Azure Databricks
Technical RequirementsDevOps Practices in DatabricksSetting Up Git Integration for DatabricksBranching StrategyOur Approach: Trunk-based Development with Environment SeparationEnvironment SeparationOur ApproachInfrastructure as Code with TerraformPrerequisitesWhat the Terraform Project CreatesProject StructureConfigurationDeployingDeclarative Automation BundlesPrerequisitesDatabricks CLI commandsProject StructureThe Bundle Configuration: databricks.ymlResource DefinitionsSource CodeTestsDeployingImplementing CI/CD Pipelines with Azure DevOpsPrerequisitesRequesting Free Parallel JobsCI/CD pipeline creation guideStep 1: Copy project files from the book repoStep 2: Generate Databricks Personal Access TokensStep 3: Create Variable Groups in Azure DevOpsStep 4: Create the Dev PipelineStep 5: Create the Staging and Production PipelineStep 6: Create Environments with Approval GatesStep 7: Set Up Branch PoliciesObservabilitySummary

Chapter 10: Optimizing Query Performance and Cost Management
Technical RequirementsPerformance Tuning for Spark and Delta LakeThe Philosophy of OptimizationAdvanced Storage Layer Optimization with Delta LakeFile Compaction and the Small File ProblemData SkippingZ-OrderingPartitioning StrategiesGenerated Columns for PartitioningLiquid Clustering: The Modern ApproachAdvanced Processing Layer Tuning with SparkJoin and Shuffle OptimizationTuning Shuffle PartitionsThe Perils of User-Defined Functions (UDFs)Adaptive Query Execution and Caching TechniquesThe Power of PhotonDynamic Re-optimization with Adaptive Query Execution (AQE)A Deeper Look at AQE FeaturesAccelerating Reads with CachingSpark Cache Storage LevelsWhen Not to CacheManaging Cluster Costs and Autoscaling StrategiesCluster Configuration and SizingInstance FamiliesEnhanced AutoscalingAutoscaling for StreamingUsing Spot Instances for Cost SavingsSpot Instance Best PracticesCost Control with Cluster Policies and TaggingMonitoring and Debugging Performance BottlenecksThe Spark UI: A Deeper DiveNavigating the Spark UIIdentifying Common BottlenecksOther Monitoring ToolsReal-World Example: Debugging a Skewed JoinExample: Salting a Skewed Join KeySummary
Chapter 11: Security, Compliance, and Data Governance
Technical requirementsSecuring the Databricks environmentNetwork security with VNet injectionEncryption: At rest and in transitWorkspace-Level security settingsFine-Grained access control with Unity CatalogColumn maskingRow-Level security with row filtersDynamic views for complex access patternsWorking example: Securing a customer analytics tableAuditing and monitoring data accessUnity catalog system tablesBuilding an audit dashboardMonitoring with Azure diagnostic settingsSetting up alertsEnsuring compliance with regulatory standardsGDPR: Right to erasure with Delta LakeHIPAA: Protecting health informationBuilding a compliance frameworkData lineage and impact analysisHow the Unity Catalog captures lineageQuerying lineage programmaticallyImpact analysis of schema changesLineage for complianceSecure data sharing with delta sharingInternal sharing across workspacesExternal sharing with delta sharingSummary
Chapter 12: Machine Learning and AI on Databricks
Technical RequirementsIntroduction to Databricks AI/ML innovation and capabilitiesThe Databricks AI/ML StackML and AI Capabilities OverviewTraditional Machine LearningGenerative AI Platform: Mosaic AIIntegration and GovernanceLineage Flow ExampleKey Integration PointsMLflow: ML Experiment Tracking and Model ManagementWhat is MLflow?Core ConceptsExperiment TrackingModel RegistryTry It: Tracking an ExperimentModel Registry and DeploymentRegistering a ModelManaging Model Versions with AliasesDeploying to Model ServingBusiness Use Case IdeasFeature Store: Centralized Feature ManagementFeature Store Core ConceptsTry It: Create and Use FeaturesCore ConceptsTry It: Forecasting with AutoMLUsing AutoML ResultsWhen to Use AutoMLGenie: Conversational Analytics for Business UsersWhat is Genie?Example and Use CasesMosaic AI Platform OverviewCommon Use CasesWhat is Vector Search?Creating a Vector IndexSearching the IndexWhen to Use Vector SearchWhat is RAG?RAG vs Fine-Tuning: Which Should You Use?Building a Simple RAG FunctionAdding Conversation MemoryEvaluating RAG QualityAI Gateway: Governance and Cost ControlWhat AI Gateway DoesSetting Up AI GatewayUsing the GatewayDeployment and MonitoringDeploying AI AgentsSummaryNext Steps
Chapter 13: Unlock Access to the Code Bundle and the PDF Version
Unlock this book's free benefits in three easy steps
Index

Content preview from Data Engineering with Azure Databricks

4 Data Engineering with Apache Spark

Organizations today face enormous challenges when processing and analyzing large-scale datasets. The sheer volume, velocity, and variety of data can overwhelm traditional data processing systems, leading to issues with complexity, performance, scalability, and operational management. Apache Spark has emerged as a leading solution to these challenges, providing a powerful, unified platform for batch processing, real-time streaming, machine learning, and interactive analytics. Azure Databricks enhances Spark's capabilities by offering an enterprise-grade, fully managed cloud platform with advanced features for security, cluster management, and team collaboration.

This chapter provides a comprehensive exploration ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Building Modern Data Applications Using Databricks Lakehouse

Publisher Resources

ISBN: 9781806106370

Cloud Computing