book

Azure Storage, Streaming, and Batch Analytics

by Richard Nuckolls

October 2020

Beginner to intermediate

448 pages

13h 6m

English

Manning Publications

Read now

Unlock full access

prefaceacknowledgementsabout this bookWho should read this bookHow this book is organized: a roadmapAbout the codeAuthor onlineabout the authorabout the cover illustration
1.1 What is data engineering?1.2 What do data engineers do?1.3 How does Microsoft define data engineering?1.3.1 Data acquisition1.3.2 Data storage1.3.3 Data processing1.3.4 Data queries1.3.5 Orchestration1.3.6 Data retrieval1.4 What tools does Azure provide for data engineering?1.5 Azure Data Engineers1.6 Example applicationSummary
2.1 Fundamentals of Azure architecture2.1.1 Azure subscriptions2.1.2 Azure regions2.1.3 Azure naming conventions2.1.4 Resource groups2.1.5 Finding resources2.2 Lambda architecture2.3 Azure cloud services2.3.1 Azure analytics system architecture2.3.2 Event Hubs2.3.3 Stream Analytics2.3.4 Data Lake Storage2.3.5 Data Lake Analytics2.3.6 SQL Database2.3.7 Data Factory2.3.8 Azure PowerShell2.4 Walk-through of processing a series of event data records2.4.1 Hot path2.4.2 Cold path2.4.3 Choosing abstract Azure services2.5 Calculating cloud hosting costs2.5.1 Event Hubs2.5.2 Stream Analytics2.5.3 Data Lake Storage2.5.4 Data Lake Analytics2.5.5 SQL Database2.5.6 Data FactorySummary
3.1 Cloud storage services3.1.1 Before you begin3.2 Creating an Azure Storage account3.2.1 Using Azure portal3.2.2 Using Azure PowerShell3.2.3 Azure Storage replication3.3 Storage account services3.3.1 Blob storage3.3.2 Creating a Blobs service container3.3.3 Blob tiering3.3.4 Copy tools3.3.5 Queues3.3.6 Creating a queue3.3.7 Azure Storage queue options3.4 Storage account access3.4.1 Blob container security3.4.2 Designing Storage account access3.5 Exercises3.5.1 Exercise 13.5.2 Exercise 2Summary
4.1 Create an Azure Data Lake store4.1.1 Using Azure Portal4.1.2 Using Azure PowerShell4.2 Data Lake store access4.2.1 Access schemes4.2.2 Configuring access4.2.3 Hierarchy structure in the Data Lake store4.3 Storage folder structure and data drift4.3.1 Hierarchy structure revisited4.3.2 Data drift4.4 Copy tools for Data Lake stores4.4.1 Data Explorer4.4.2 ADLCopy tool4.4.3 Azure Storage Explorer tool4.5 Exercises4.5.1 Exercise 14.5.2 Exercise 2Summary

5.1 How does an Event Hub work?5.2 Collecting data in Azure5.3 Create an Event Hubs namespace5.3.1 Using Azure PowerShell5.3.2 Throughput units5.3.3 Event Hub geo-disaster recovery5.3.4 Failover with geo-disaster recovery5.4 Creating an Event Hub5.4.1 Using Azure portal5.4.2 Using Azure PowerShell5.4.3 Shared access policy5.5 Event Hub partitions5.5.1 Multiple consumers5.5.2 Why specify a partition?5.5.3 Why not specify a partition?5.5.4 Event Hubs message journal5.5.5 Partitions and throughput units5.6 Configuring Capture5.6.1 File name formats5.6.2 Secure access for Capture5.6.3 Enabling Capture5.6.4 The importance of time5.7 Securing access to Event Hubs5.7.1 Shared Access Signature policies5.7.2 Writing to Event Hubs5.8 Exercises5.8.1 Exercise 15.8.2 Exercise 25.8.3 Exercise 3Summary
6.1 Creating a Stream Analytics service6.1.1 Elements of a Stream Analytics job6.1.2 Create an ASA job using the Azure portal6.1.3 Create an ASA job using Azure PowerShell6.2 Configuring inputs and outputs6.2.1 Event Hub job input6.2.2 ASA job outputs6.3 Creating a job query6.3.1 Starting the ASA job6.3.2 Failure to start6.3.3 Output exceptions6.4 Writing job queries6.4.1 Window functions6.4.2 Machine learning functions6.5 Managing performance6.5.1 Streaming units6.5.2 Event ordering6.6 Exercises6.6.1 Exercise 16.6.2 Exercise 2Summary
7.1 U-SQL language7.1.1 Extractors7.1.2 Outputters7.1.3 File selectors7.1.4 Expressions7.2 U-SQL jobs7.2.1 Selecting the biometric data files7.2.2 Schema extraction7.2.3 Aggregation7.2.4 Writing files7.3 Creating a Data Lake Analytics service7.3.1 Using Azure portal7.3.2 Using Azure PowerShell7.4 Submitting jobs to ADLA7.4.1 Using Azure portal7.4.2 Using Azure PowerShell7.5 Efficient U-SQL job executions7.5.1 Monitoring a U-SQL job7.5.2 Analytics units7.5.3 Vertexes7.5.4 Scaling the job execution7.6 Using Blob Storage7.6.1 Constructing Blob file selectors7.6.2 Adding a new data source7.6.3 Filtering rowsets7.7 Exercises7.7.1 Exercise 17.7.2 Exercise 2Summary
8.1 Data Lake Analytics Catalog8.1.1 Simplifying U-SQL queries8.1.2 Simplifying data access8.1.3 Loading data for reuse8.2 Window functions8.3 Local C# functions8.4 Exercises8.4.1 Exercise 18.4.2 Exercise 2Summary
9.1 Processing unstructured data9.1.1 Azure Cognitive Services9.1.2 Managing assemblies in the Data Lake9.1.3 Image data extraction with Advanced Analytics9.2 Reading different file types9.2.1 Adding custom libraries with a Catalog9.2.2 Creating a catalog database9.2.3 Building the U-SQL DataFormats solution9.2.4 Code folders9.2.5 Using custom assemblies9.3 Connecting to remote sources9.3.1 External databases9.3.2 Credentials9.3.3 Data Source9.3.4 Tables and views9.4 Exercises9.4.1 Exercise 19.4.2 Exercise 2Summary
10.1 Creating an Azure Data Factory service10.2 Secure authentication10.2.1 Azure Active Directory integration10.2.2 Azure Key Vault10.3 Copying files with ADF10.3.1 Creating a Files storage container10.3.2 Adding secrets to AKV10.3.3 Creating a Files storage linkedservice10.3.4 Creating an ADLS linkedservice10.3.5 Creating a pipeline and activity10.3.6 Creating a scheduled trigger10.4 Running an ADLA job10.4.1 Creating an ADLA linkedservice10.4.2 Creating a pipeline and activity10.5 Exercises10.5.1 Exercise 110.5.2 Exercise 2Summary
11.1 Creating an Azure SQL Database11.1.1 Create a SQL Server and SQLDB11.2 Securing SQLDB11.3 Availability and recovery11.3.1 Restoring and moving SQLDB11.3.2 Database safeguards11.3.3 Creating alerts for SQLDB11.4 Optimizing costs for SQLDB11.4.1 Pricing structure11.4.2 Scaling SQLDB11.4.3 Serverless11.4.4 Elastic Pools11.5 Exercises11.5.1 Exercise 111.5.2 Exercise 211.5.3 Exercise 311.5.4 Exercise 4Summary
12.1 Before you begin12.2 Importing data with external data sources12.2.1 Creating a database scoped credential12.2.2 Creating an external data source12.2.3 Creating an external table12.2.4 Importing Blob files12.3 Importing file data with ADF12.3.1 Authenticating between ADF and SQLDB12.3.2 Creating SQL Database linkedservice12.3.3 Creating datasets12.3.4 Creating a copy activity and pipeline12.4 Exercises12.4.1 Exercise 112.4.2 Exercise 212.4.3 Exercise 3Summary
13.1 Data catalog13.1.1 Data Catalog as a service13.1.2 Data locations13.1.3 Data definitions13.1.4 Data frequency13.1.5 Business drivers13.2 Version control and backups13.2.1 Blob Storage13.2.2 Data Lake Storage13.2.3 Stream Analytics13.2.4 Data Lake Analytics13.2.5 Data Factory configuration files13.2.6 SQL Database13.3 Microsoft certifications13.4 Signing offSummary
A.1 Setting up Azure PowerShellA.2 Create a subscriptionA.3 Azure naming conventionsA.4 Setting up common Azure resources using PowerShellA.4.1 Creating a new resource groupA.4.2 Creating a new Azure Active Directory userA.4.3 Creating a new Azure Active Directory groupA.5 Setting up Azure services using PowerShellA.5.1 Creating a new Storage accountA.5.2 Creating a new Data Lake storeA.5.3 Create new Event HubA.5.4 Create new Stream Analytics jobA.5.5 Create new Data Lake Analytics accountA.5.6 Create new SQL Server and DatabaseA.5.7 Create a new Data Factory serviceA.5.8 Creating a new App registrationA.5.9 Creating a new key vaultA.5.10 Create new SQL Server and Database with lookup data
B.1 Solution designB.1.1 Hot pathB.1.2 Cold pathB.2 Naming conventionB.3 Creation scriptB.4 Configure Azure services using PowerShellB.4.1 Stream Analytics Managed IdentityB.4.2 Data Lake storeB.4.3 Stream Analytics job configurationB.4.4 SQL DatabaseB.4.5 Data FactoryB.5 Load event dataB.6 Output of batch and stream processingB.7 Removing services