book

Azure Data Factory Cookbook

Name: Azure Data Factory Cookbook
ISBN: 9781800565296

by Dmitry Anoshin, Dmitry Foshin, Roman Storchak, Xenia Ireton

December 2020

Beginner to intermediate

382 pages

7h 19m

English

Packt Publishing

Read now

Unlock full access

Azure Data Factory Cookbook
Why subscribe?ContributorsAbout the authors About the reviewersPackt is searching for authors like you
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedSectionsGetting readyHow to do it…How it works…There's more…See alsoGet in touchReviews
Chapter 1: Getting Started with ADF
Introduction to the Azure data platformGetting readyHow to do it...How it works...Creating and executing our first job in ADFGetting readyHow to do it...How it works...There's more...See alsoCreating an ADF pipeline by using the Copy Data toolGetting readyHow to do it...How it works...There's more...Creating an ADF pipeline using PythonGetting readyHow to do it...How it works...There's more...See alsoCreating a data factory using PowerShellGetting readyHow to do it…How it works...There's more...See alsoUsing templates to create ADF pipelinesGetting readyHow to do it...How it works...See also
Chapter 2: Orchestration and Control Flow
Technical requirementsUsing parameters and built-in functionsGetting readyHow to do it…How it works…There's more…See alsoUsing Metadata and Stored Procedure activitiesGetting readyHow to do it…How it works…There's more…Using the ForEach and Filter activities Getting readyHow to do it…How it works…Chaining and branching activities within a pipelineGetting readyHow to do it…There's more…Using the Lookup, Web, and Execute Pipeline activitiesGetting readyHow to do it…How it works…There's more…See alsoCreating event-based triggers Getting readyHow to do it…How it works…There's more…See also
Chapter 3: Setting Up a Cloud Data Warehouse
Technical requirementsConnecting to Azure Synapse AnalyticsGetting readyHow to do it…How it works…There's more…Loading data to Azure Synapse Analytics using SSMSGetting readyHow to do it…How it works…There's more…Loading data to Azure Synapse Analytics using Azure Data FactoryGetting readyHow to do it…How it works…There's more…Pausing/resuming an Azure SQL pool from Azure Data FactoryGetting readyHow to do it…How it works…There's more…Creating an Azure Synapse workspaceGetting readyHow to do it…There's more…Loading data to Azure Synapse Analytics using bulk loadGetting readyHow to do it…How it works…Copying data in Azure Synapse OrchestrateGetting readyHow to do it…How it works…Using SQL on-demandGetting readyHow to do it…How it works…
Chapter 4: Working with Azure Data Lake
Technical requirementsSetting up Azure Data Lake Storage Gen2Getting readyHow to do it...Connecting Azure Data Lake to Azure Data Factory and loading dataGetting readyHow to do it...How it works...Creating big data pipelines using Azure Data Lake and Azure Data FactoryGetting readyHow to do it...How it works
Chapter 5: Working with Big Data – HDInsight and Databricks
Technical requirementsSetting up an HDInsight clusterGetting readyHow to do it…How it works…Processing data from Azure Data Lake with HDInsight and HiveGetting readyHow to do it…How it works…Processing big data with Apache Spark Getting readyHow to do it…How it works…Building a machine learning app with Databricks and Azure Data Lake StorageGetting readyHow to do it…How it works…
Chapter 6: Integration with MS SSIS
Technical requirementsCreating a SQL Server databaseGetting readyHow to do it…How it works…Building an SSIS package Getting readyHow to do it…How it works… Running SSIS packages from ADFGetting readyHow to do it…How it works…
Chapter 7: Data Migration – Azure Data Factory and Other Cloud Services
Technical requirementsCopying data from Amazon S3 to Azure Blob storageGetting readyHow to do it…How it works…Copying large datasets from S3 to ADLSGetting readyHow to do it…How it works…See alsoCopying data from Google Cloud Storage to Azure Data Lake Getting readyHow to do it…How it works…See alsoCopying data from Google BigQuery to Azure Data Lake StoreGetting readyHow to do it…Migrating data from Google BigQuery to Azure SynapseGetting readyHow to do it…See alsoMoving data to DropboxGetting readyHow to do it…How it works…There's more…See also
Chapter 8: Working with Azure Services Integration
Technical requirementsTriggering your data processing with Logic AppsGetting readyHow to do it…How it works…There's more…Using the web activity to call an Azure logic appGetting readyHow to do it…How it works…There's more…Adding flexibility to your pipelines with Azure FunctionsGetting ready…How to do it…How it works…There's more…Automatically building ML models with speed and scaleGetting readyHow to do it...How it works…There's more...Transforming and preparing your data via Azure DatabricksGetting readyHow to do it…How it works…There's more…

Chapter 9: Managing Deployment Processes with Azure DevOps
Technical requirementsSetting up Azure DevOpsGetting readyHow to do it...How it works...Publishing changes to ADFGetting readyHow to do it...How it works...Deploying your features into the master branchGetting readyHow to do it...How it works...Getting ready for the CI/CD of ADFGetting readyHow to do it...How it works...Creating an Azure pipeline for CDGetting readyHow to do it...How to do it...There's more...
Chapter 10: Monitoring and Troubleshooting Data Pipelines
Technical requirementsMonitoring pipeline runs and integration runtimesGetting readyHow to do it…How it works…Investigating failures – running in debug modeGetting readyHow to do it…How it works…There's more…See alsoRerunning activitiesGetting readyHow to do it…How it works…Configuring alerts for your Data Factory runsGetting readyHow to do it…How it works…There's more…See also
Other Books You May Enjoy
Leave a review - let other readers know what you think

Overview

Azure Data Factory is a powerful tool for modern data integration on the Microsoft Azure platform. In the "Azure Data Factory Cookbook," you'll learn how to create and manage ETL and ELT pipelines using Azure's visual tools, integrate with other Azure services, and handle data from on-premises and cloud environments. This practical guide is packed with recipes to help you implement real-world data solutions.

What this Book will help me do

Understand the use of Azure Data Factory for constructing ETL and ELT pipelines.
Learn to integrate Azure Data Factory with Azure Synapse, Azure Data Lake, and other services.
Master data migration techniques between on-premise and cloud systems using built-in connectors.
Explore building data pipelines that incorporate advanced Azure services like Databricks and Azure ML.
Gain expertise in troubleshooting and monitoring data pipelines effectively using Azure tools.

Author(s)

Dmitry Anoshin, Dmitry Foshin, None Storchak, and Xenia Ireton are experienced professionals and authors, known for their deep expertise in data engineering and Microsoft Azure. Their collaborative work in producing this book draws from hands-on experience with real-world projects, ensuring practical and applicable content.

Who is it for?

This book is tailored for data engineers, ETL developers, and architects interested in mastering Azure Data Factory. If you're aiming to learn or enhance your data orchestration and pipeline modeling skills with Azure services, this book is for you. A foundational knowledge of ETL processes and basic familiarity with cloud services would let you maximize its utility.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Azure Data Factory Cookbook - Second Edition

Publisher Resources

ISBN: 9781800565296

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills