book

Azure Data Engineering Cookbook - Second Edition

by Luca Zanna, Nagaraj Venkatesan, Ahmad Osama

September 2022

Beginner to intermediate

608 pages

12h 30m

English

Packt Publishing

Read now

Unlock full access

Second EditionContributorsAbout the authorsAbout the reviewers
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedSectionsGetting readyHow to do it…How it works…There’s more…See alsoGet in touchShare your thoughts
Technical requirementsProvisioning an Azure storage account using the Azure portalGetting readyHow to do it…How it works…Provisioning an Azure storage account using PowerShellGetting readyHow to do it…How it works…Creating containers and uploading files to Azure Blob storage using PowerShellGetting readyHow to do it…How it works…Managing blobs in Azure Storage using PowerShellGetting readyHow to do it…How it works…Configuring blob lifecycle management for blob objects using the Azure portalGetting readyHow to do it…How it works…
Configuring a firewall for an Azure Data Lake account using the Azure portalGetting readyHow to do it…How it works…Configuring virtual networks for an Azure Data Lake account using the Azure portalGetting readyHow to do it…How it works…Configuring private links for an Azure Data Lake accountGetting readyHow to do it…How it works…Configuring encryption using Azure Key Vault for Azure Data LakeGetting readyHow to do it…How it works…Accessing Blob storage accounts using managed identitiesGetting readyHow to do it…How it works…Creating an alert to monitor an Azure storage accountGetting readyHow to do it…How it works…Securing an Azure storage account with SAS using PowerShellGetting readyHow to do it…How it works…
Technical requirementsProvisioning Azure Data FactoryHow to do it…How it works…Copying files to a database from a data lake using a control flow and copy activityGetting readyHow to do it…How it works…Triggering a pipeline in Azure Data FactoryGetting readyHow to do it…How it works…Copying data from a SQL Server virtual machine to a data lake using the Copy data wizardGetting readyHow to do it…How it works…
Technical requirementsConfiguring a self-hosted IRGetting readyHow to do it…How it works…Configuring a shared self-hosted IRGetting readyHow to do it…Configuring high availability for a self-hosted IRGetting readyHow to do it…How it works…Patching a self-hosted IRGetting readyHow to do it…How it works…Migrating an SSIS package to Azure Data FactoryGetting readyHow to do it…How it works...
Technical requirementsProvisioning and connecting to an Azure SQL database using PowerShellGetting readyHow to do it…How it works…Implementing an Azure SQL Database elastic pool using PowerShellGetting readyHow to do it...How it works…Configuring a virtual network and private endpoints for Azure SQL DatabaseGetting readyHow to do it…How it works…Configuring Azure Key Vault for Azure SQL Database Getting readyHow to do it…How it works…Provisioning and configuring a wake-up script for a serverless SQL databaseGetting readyHow to do it…How it works…Configuring the Hyperscale tier of Azure SQL DatabaseGetting readyHow to do it…
Implementing active geo-replication for an Azure SQL database using PowerShellGetting readyHow to do it…How it works…Implementing an auto-failover group for an Azure SQL database using PowerShellGetting readyHow to do it…How it works…Configuring high availability to the Hyperscale tier of Azure SQL Database Getting readyHow to do it…How it works…Implementing vertical scaling for an Azure SQL database using PowerShellGetting readyHow to do it…How it works…Monitoring an Azure SQL database using the Azure portalGetting readyHow to do it…Configuring auditing for Azure SQL DatabaseGetting readyHow to do it…How it works…
Technical requirementsConfiguring the Azure Databricks environmentGetting readyHow to do it…Integrating Databricks with Azure Key VaultGetting readyHow to do it…How it works…Mounting an Azure Data Lake container in DatabricksGetting readyHow to do it…How it works…Processing data using notebooksGetting readyHow to do it…How it works…Scheduling notebooks using job clustersGetting readyHow to do it…How it works…Working with Delta Lake tablesGetting readyHow to do it…How it works…Connecting a Databricks Delta Lake table to Power BIGetting readyHow to do it…How it works…
Technical requirementsProvisioning an Azure Synapse Analytics workspaceGetting readyHow to do it…Analyzing data using serverless SQL pool Getting readyHow to do it…How it works…Provisioning and configuring Spark poolsGetting readyHow to do it…How it works…Processing data using Spark pools and a lake databaseGetting readyHow to do it…How it works…Querying the data in a lake database from serverless SQL poolGetting readyHow to do it…How it works…Scheduling notebooks to process data incrementallyGetting readyHow to do it…How it works…Visualizing data using Power BI by connecting to serverless SQL pool Getting readyHow to do it…How it works…

Technical requirementsCopying data using a Synapse data flowGetting readyHow to do it…How it works…Performing data transformation using activities such as join, sort, and filterGetting readyHow to do it…How it works…Monitoring data flows and pipelinesGetting readyHow to do it…How it works…Configuring partitions to optimize data flowsGetting readyHow to do it…How it works…Parameterizing Synapse data flowsGetting readyHow to do it…How it works…Handling schema changes dynamically in data flows using schema driftGetting readyHow to do it…How it works…
Technical requirementsLoading data into dedicated SQL pools using PolyBase and T-SQL Getting readyHow to do it…How it works...Loading data into a dedicated SQL pool using COPY INTOGetting readyHow to do it...How it works...Creating distributed tables and modifying table distributionGetting readyHow to do it…How it works...Creating statistics and automating the update of statisticsGetting readyHow to do it…How it works…Creating partitions and archiving data using partitioned tables Getting readyHow to do it…How it works...Implementing workload management in an Azure Synapse dedicated SQL poolGetting readyHow to do it…How it works…Creating workload groups for advanced workload managementGetting readyHow to do it…How it works...
Technical requirementsConfiguring a Log Analytics workspace for Synapse SQL poolsGetting readyHow to do it…How it works…Configuring a Log Analytics workspace for Synapse Spark pools Getting readyHow to do it…How it works…Using Kusto queries to monitor SQL and Spark pools Getting readyHow to do it…How it works…Creating workbooks in a Log Analytics workspace to visualize monitoring dataGetting readyHow to do it…How it works…Monitoring table distribution, data skew, and index health using Synapse DMVsGetting readyHow to do it…Building monitoring dashboards for Synapse with Azure MonitorGetting readyHow to do it…How it works…
Technical requirementsAnalyzing a query plan and fixing table distribution Getting readyHow to do it…How it works…Monitoring and rebuilding a replication table cacheGetting readyHow to do it…How it works…Configuring result set caching in Azure Synapse dedicated SQL pool Getting readyHow to do it…How it works…Configuring longer backup retention for a Synapse SQL database Getting readyHow to do it…How it works…Auto pausing Synapse dedicated SQL poolGetting readyHow to do it…How it works…Optimizing Delta tables in a Synapse Spark pool lake databaseGetting readyHow to do it…How it works…Optimizing query performance in Synapse Spark poolsGetting readyHow to do it…How it works…
Technical requirementsMonitoring Synapse integration pipelines using Log Analytics and workbooksGetting readyHow to do it…How it works…Tracing SQL queries for dedicated SQL pool to Synapse integration pipelines Getting readyHow to do it…How it works…Provisioning a Microsoft Purview account and creating a data catalog Getting readyHow to do it…How it works…Integrating a Synapse workspace with Microsoft Purview and tracking data lineage Getting readyHow to do it…How it works…Applying Azure tags using PowerShell to multiple Azure resourcesGetting readyHow to do it…How it works…
Other Books You May EnjoyPackt is searching for authors like youShare your thoughts

Overview

Azure Data Engineering Cookbook is your ultimate guide to mastering data engineering on Microsoft's Azure platform. Through an engaging collection of recipes, this book breaks down procedures to build sophisticated data pipelines, leveraging tools like Azure Data Factory, Data Lake, Databricks, and Synapse Analytics.

What this Book will help me do

Efficiently process large datasets using Azure Synapse analytics and Azure Databricks pipelines.
Transform and shape data within systems by leveraging Azure Synapse data flows.
Implement and manage relational databases in Azure with performance tuning and administration.
Configure data pipeline solutions integrated with Power BI for insightful reporting.
Monitor, optimize, and ensure lineage tracking for your data systems efficiently with Purview and Log analytics.

Author(s)

Nagaraj Venkatesan is an experienced cloud architect specializing in Microsoft Azure, with years of hands-on data engineering expertise. Ahmad Osama is a seasoned data professional and author's shared emphasis is on practical learning and bridging this with actionable skills effectively.

Who is it for?

This book is essential for data engineers seeking expertise in Azure's rich engineering capabilities. It's tailored for professionals with a foundational knowledge of cloud services, looking to achieve advanced proficiency in Azure data engineering pipelines.