book

Azure Data Engineering Cookbook

by Ahmad Osama, Nagaraj Venkatesan

April 2021

Beginner to intermediate

454 pages

7h 53m

English

Packt Publishing

Read now

Unlock full access

ContributorsAbout the authorAbout the reviewers
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedSectionsGetting readyHow to do it…How it works…There's more…See alsoGet in touchReviews
Technical requirementsProvisioning an Azure storage account using the Azure portalGetting readyHow to do it…How it works…Provisioning an Azure storage account using PowerShellGetting readyHow to do it…How it works…Creating containers and uploading files to Azure Blob storage using PowerShellGetting readyHow to do it…How it works…Managing blobs in Azure Storage using PowerShellGetting readyHow to do it…How it works…Managing an Azure blob snapshot in Azure Storage using PowerShell Getting readyHow to do it…How it works…Configuring blob life cycle management for blob objects using the Azure portalGetting readyHow to do it…How it works…Configuring a firewall for an Azure storage account using the Azure portalGetting readyHow to do it…How it works…Configuring virtual networks for an Azure storage account using the Azure portalGetting readyHow to do it…How it works…Configuring a firewall for an Azure storage account using PowerShellGetting readyHow to do it…How it works…Configuring virtual networks for an Azure storage account using PowerShellGetting readyHow to do it…How it works…Creating an alert to monitor an Azure storage accountGetting readyHow to do it…How it works…Securing an Azure storage account with SAS using PowerShellGetting readyHow to do it…How it works…
Provisioning and connecting to an Azure SQL database using PowerShellGetting readyHow to do it…How it works…Provisioning and connecting to an Azure PostgreSQL database using the Azure CLIGetting readyHow to do it…How it works…Provisioning and connecting to an Azure MySQL database using the Azure CLIGetting readyHow it works…How it works…Implementing active geo-replication for an Azure SQL database using PowerShellGetting readyHow to do it…How it works…Implementing an auto-failover group for an Azure SQL database using PowerShellGetting readyHow to do it…How it works…Implementing vertical scaling for an Azure SQL database using PowerShellGetting readyHow to do it…How it works…Implementing an Azure SQL database elastic pool using PowerShellGetting readyHow it works…How it works…Monitoring an Azure SQL database using the Azure portalGetting readyHow to do it…
Technical requirementsProvisioning and connecting to an Azure Synapse SQL pool using PowerShellGetting readyHow to do it…How it works…Pausing or resuming a Synapse SQL pool using PowerShellGetting readyHow to do it…How it works…Scaling an Azure Synapse SQL pool instance using PowerShellGetting readyHow to do it…How it works…Loading data into a SQL pool using PolyBase with T-SQLGetting readyHow to do it…How it works…Loading data into a SQL pool using the COPY INTO statementGetting readyHow to do it…How it works…Implementing workload management in an Azure Synapse SQL poolGetting readyHow to do it…How it works…Optimizing queries using materialized views in Azure Synapse AnalyticsGetting readyHow to do it…How it works…
Technical requirementsImplementing control flow activitiesGetting readyHow to do it…Implementing control flow activities – Lookup and If activitiesGetting readyHow to do it…Triggering a pipeline in Azure Data FactoryGetting readyHow to do it…
Technical requirementsImplementing HDInsight Hive and Pig activitiesGetting readyHow to do it…Implementing an Azure Functions activityGetting readyHow to do it…Implementing a Data Lake Analytics U-SQL activityGetting readyHow to do it…Copying data from Azure Data Lake Gen2 to an Azure Synapse SQL pool using the copy activityGetting readyHow to do it…Copying data from Azure Data Lake Gen2 to Azure Cosmos DB using the copy activityGetting readyHow to do it…
Technical requirementsImplementing incremental data loading with a mapping data flowGetting readyHow to do it…Implementing a wrangling data flowGetting readyHow to do it…
Technical requirementsConfiguring a self-hosted IRGetting readyHow to do it…Configuring a shared self-hosted IRGetting readyHow to do it…Migrating an SSIS package to Azure Data FactoryGetting readyHow to do it…Executing an SSIS package with an on-premises data storeGetting readyHow to do it…
Technical requirementsConfiguring the development, test, and production environmentsGetting readyHow to do it…Deploying Azure Data Factory pipelines using the Azure portal and ARM templatesGetting readyHow to do it…Automating Azure Data Factory pipeline deployment using Azure DevOpsGetting readyHow to do it…

Technical requirementsConfiguring the Azure Databricks environmentGetting readyHow to do it…Transforming data using PythonGetting readyHow to do it…Transforming data using ScalaGetting readyHow to do it…Working with Delta LakeGetting readyHow to do it…Processing structured streaming data with Azure DatabricksGetting readyHow to do it…
Other Books You May EnjoyPackt is searching for authors like youLeave a review - let other readers know what you think

Overview

Dive into the world of data engineering with 'Azure Data Engineering Cookbook' to master building efficient ETL workflows using Microsoft Azure Data services. Whether you're working on batch processing solutions or real-time analytics, this book is your guide to implementing effective, scalable data operations.

What this Book will help me do

Design and implement efficient ETL pipelines for batch and real-time processing on MS Azure.
Understand the use of Azure Blob storage for managing large data sets.
Ingest, process, and analyze data using tools like Azure Synapse and Databricks.
Develop and secure automation pipelines using Azure Data Factory.
Leverage Azure Stream Analytics for real-time data processing workflows.

Author(s)

Ahmad Osama and Nagaraj Venkatesan bring years of expertise in cloud solutions and data engineering. Renowned for their practical teaching approach, they have helped countless professionals master the intricacies of Azure. Their focus is on equipping readers with actionable skills for real-world data challenges.

Who is it for?

This book is ideal for data engineers and database professionals aiming to hone their expertise in advanced Azure data engineering tasks. Readers should have a working knowledge of Azure fundamentals and basic data engineering concepts. If you're a technical architect or ETL developer seeking to transition or enhance your skills in Azure's ecosystem, you'll find immense value here.