book

Data Engineering on Azure

Name: Data Engineering on Azure
Author: Vlad Riscutia
ISBN: 9781617298929

by Vlad Riscutia

August 2021

Intermediate to advanced

336 pages

9h 35m

English

Manning Publications

Read now

Unlock full access

inside front cover
Data Platform Architecture
Data Engineering on Azure
Copyright
dedication
brief contents
contents
front matter
prefaceacknowledgmentsabout this bookabout the authorabout the cover illustration
1 Introduction
1.1 What is data engineering?1.2 Who this book is for1.3 What is a data platform?1.3.1 Anatomy of a data platform1.3.2 Infrastructure as code, codeless infrastructure1.4 Building in the cloud1.4.1 IaaS, PaaS, SaaS1.4.2 Network, storage, compute1.4.3 Getting started with Azure1.4.4 Interacting with Azure1.5 Implementing an Azure data platformSummary
Part 1 Infrastructure
2 Storage
2.1 Storing data in a data platform2.1.1 Storing data across multiple data fabrics2.1.2 Having a single source of truth2.2 Introducing Azure Data Explorer2.2.1 Deploying an Azure Data Explorer cluster2.2.2 Using Azure Data Explorer2.2.3 Working around query limits2.3 Introducing Azure Data Lake Storage2.3.1 Creating an Azure Data Lake Storage account2.3.2 Using Azure Data Lake Storage2.3.3 Integrating with Azure Data Explorer2.4 Ingesting data2.4.1 Ingestion frequency2.4.2 Load type2.4.3 Restatements and reloadsSummary

3 DevOps
3.1 What is DevOps?3.1.1 DevOps in data engineering3.2 Introducing Azure DevOps3.2.1 Using the az azure-devops extension3.3 Deploying infrastructure3.3.1 Exporting an Azure Resource Manager template3.3.2 Creating Azure DevOps service connections3.3.3 Deploying Azure Resource Manager templates3.3.4 Understanding Azure Pipelines3.4 Deploying analytics3.4.1 Using Azure DevOps marketplace extensions3.4.2 Storing everything in Git; deploying everything automaticallySummary
4 Orchestration
4.1 Ingesting the Bing COVID-19 open dataset4.2 Introducing Azure Data Factory4.2.1 Setting up the data source4.2.2 Setting up the data sink4.2.3 Setting up the pipeline4.2.4 Setting up a trigger4.2.5 Orchestrating with Azure Data Factory4.3 DevOps for Azure Data Factory4.3.1 Deploying Azure Data Factory from Git4.3.2 Setting up access control4.3.3 Deploying the production data factory4.3.4 DevOps for the Azure Data Factory recap4.4 Monitoring with Azure MonitorSummary
Part 2 Workloads
5 Processing
5.1 Data modeling techniques5.1.1 Normalization and denormalization5.1.2 Data warehousing5.1.3 Semistructured data5.1.4 Data modeling recap5.2 Identity keyrings5.2.1 Building an identity keyring5.2.2 Understanding keyrings5.3 Timelines5.3.1 Building a timeline view5.3.2 Using timelines5.4 Continuous data processing5.4.1 Tracking processing functions in Git5.4.2 Keyring building in Azure Data Factory5.4.3 Scaling outSummary
6 Analytics
6.1 Structuring storage6.1.1 Providing development data6.1.2 Replicating production data6.1.3 Providing read-only access to the production data6.1.4 Storage structure recap6.2 Analytics workflow6.2.1 Prototyping6.2.2 Development and user acceptance testing6.2.3 Production6.2.4 Analytics workflow recap6.3 Self-serve data movement6.3.1 Support model6.3.2 Data contracts6.3.3 Pipeline validation6.3.4 Postmortems6.3.5 Self-serve data movement recapSummary
7 Machine learning
7.1 Training a machine learning model7.1.1 Training a model using scikit-learn7.1.2 High spender model implementation7.2 Introducing Azure Machine Learning7.2.1 Creating a workspace7.2.2 Creating an Azure Machine Learning compute target7.2.3 Setting up Azure Machine Learning storage7.2.4 Running ML in the cloud7.2.5 Azure Machine Learning recap7.3 MLOps7.3.1 Deploying from Git7.3.2 Storing pipeline IDs7.3.3 DevOps for Azure Machine Learning recap7.4 Orchestrating machine learning7.4.1 Connecting Azure Data Factory with Azure Machine Learning7.4.2 Machine learning orchestration7.4.3 Orchestrating recapSummary
Part 3 Governance
8 Metadata
8.1 Making sense of the data8.2 Introducing Azure Purview8.3 Maintaining a data inventory8.3.1 Setting up a scan8.3.2 Browsing the data dictionary8.3.3 Data dictionary recap8.4 Managing a data glossary8.4.1 Adding a new glossary term8.4.2 Curating terms8.4.3 Custom templates and bulk import8.4.4 Data glossary recap8.5 Understanding Azure Purview's advanced features8.5.1 Tracking lineage8.5.2 Classification rules8.5.3 REST API8.5.4 Advanced features recapSummary
9 Data quality
9.1 Testing data9.1.1 Availability tests9.1.2 Correctness tests9.1.3 Completeness tests9.1.4 Detecting anomalies9.1.5 Testing data recap9.2 Running data quality checks9.2.1 Testing using Azure Data Factory9.2.2 Executing tests9.2.3 Creating and using a template9.2.4 Running data quality checks recap9.3 Scaling out data testing9.3.1 Supporting multiple data fabrics9.3.2 Testing at rest and during movement9.3.3 Authoring tests9.3.4 Storing tests and resultsSummary
10 Compliance
10.1 Data classification10.1.1 Feature data10.1.2 Telemetry10.1.3 User data10.1.4 User-owned data10.1.5 Business data10.1.6 Data classification recap10.2 Changing classification through processing10.2.1 Aggregation10.2.2 Anonymization10.2.3 Pseudonymization10.2.4 Masking10.2.5 Processing classification changes recap10.3 Implementing an access model10.3.1 Security groups10.3.2 Securing Azure Data Explorer10.3.3 Access model recap10.4 Complying with GDPR and other considerations10.4.1 Data handling10.4.2 Data subject requests10.4.3 Other considerationsSummary
11 Distributing data
11.1 Data distribution overview11.2 Building a data API11.2.1 Introducing Azure Cosmos DB11.2.2 Populating the Cosmos DB collection11.2.3 Retrieving data11.2.4 Data API recap11.3 Serving machine learning11.4 Sharing data for bulk copy11.4.1 Separating compute resources11.4.2 Introducing Azure Data Share11.4.3 Sharing data for bulk copy recap11.5 Data sharing best practicesSummary
Appendix A. Azure services
Azure StorageAzure SQLAzure Synapse AnalyticsAzure Data ExplorerAzure DatabricksAzure Cosmos DB
Appendix B. KQL quick reference
Common query referenceSQL to KQL
Appendix C. Running code samples
index
inside back cover
MLOps

Content preview from Data Engineering on Azure

9 Data quality

This chapter covers

Testing data to ensure quality
Different types of data quality checks
Executing data tests
Considerations for scaling out data testing

The insights generated by a data platform are only as good as the quality of the underlying data. A good data platform needs to provide some guarantees around data quality. In this chapter, we will focus on data quality.

At the time of writing, data quality testing isn’t yet offered “as a service” by all major cloud providers. Unlike some of the previous topics we’ve covered in this book—such as storage, data processing, or machine learning (ML)—we don’t have an out-of-the-box PaaS (platform as a service) solution, so we’ll have to stitch something together ourselves.

We’ll ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617298929Publisher Support Publisher Website

Cloud Computing