book

Azure Storage, Streaming, and Batch Analytics

Name: Azure Storage, Streaming, and Batch Analytics
Author: Richard Nuckolls
ISBN: 9781617296307

by Richard Nuckolls

October 2020

Beginner to intermediate

448 pages

13h 6m

English

Manning Publications

Audiobook available

Read now

Unlock full access

Azure Storage, Streaming, and Batch Analytic
Copyright
dedication
brief contents
contents
front matter
prefaceacknowledgementsabout this bookWho should read this bookHow this book is organized: a roadmapAbout the codeAuthor onlineabout the authorabout the cover illustration
1 What is data engineering?
1.1 What is data engineering?1.2 What do data engineers do?1.3 How does Microsoft define data engineering?1.3.1 Data acquisition1.3.2 Data storage1.3.3 Data processing1.3.4 Data queries1.3.5 Orchestration1.3.6 Data retrieval1.4 What tools does Azure provide for data engineering?1.5 Azure Data Engineers1.6 Example applicationSummary
2 Building an analytics system in Azure
2.1 Fundamentals of Azure architecture2.1.1 Azure subscriptions2.1.2 Azure regions2.1.3 Azure naming conventions2.1.4 Resource groups2.1.5 Finding resources2.2 Lambda architecture2.3 Azure cloud services2.3.1 Azure analytics system architecture2.3.2 Event Hubs2.3.3 Stream Analytics2.3.4 Data Lake Storage2.3.5 Data Lake Analytics2.3.6 SQL Database2.3.7 Data Factory2.3.8 Azure PowerShell2.4 Walk-through of processing a series of event data records2.4.1 Hot path2.4.2 Cold path2.4.3 Choosing abstract Azure services2.5 Calculating cloud hosting costs2.5.1 Event Hubs2.5.2 Stream Analytics2.5.3 Data Lake Storage2.5.4 Data Lake Analytics2.5.5 SQL Database2.5.6 Data FactorySummary
3 General storage with Azure Storage accounts
3.1 Cloud storage services3.1.1 Before you begin3.2 Creating an Azure Storage account3.2.1 Using Azure portal3.2.2 Using Azure PowerShell3.2.3 Azure Storage replication3.3 Storage account services3.3.1 Blob storage3.3.2 Creating a Blobs service container3.3.3 Blob tiering3.3.4 Copy tools3.3.5 Queues3.3.6 Creating a queue3.3.7 Azure Storage queue options3.4 Storage account access3.4.1 Blob container security3.4.2 Designing Storage account access3.5 Exercises3.5.1 Exercise 13.5.2 Exercise 2Summary
4 Azure Data Lake Storage
4.1 Create an Azure Data Lake store4.1.1 Using Azure Portal4.1.2 Using Azure PowerShell4.2 Data Lake store access4.2.1 Access schemes4.2.2 Configuring access4.2.3 Hierarchy structure in the Data Lake store4.3 Storage folder structure and data drift4.3.1 Hierarchy structure revisited4.3.2 Data drift4.4 Copy tools for Data Lake stores4.4.1 Data Explorer4.4.2 ADLCopy tool4.4.3 Azure Storage Explorer tool4.5 Exercises4.5.1 Exercise 14.5.2 Exercise 2Summary

5 Message handling with Event Hubs
5.1 How does an Event Hub work?5.2 Collecting data in Azure5.3 Create an Event Hubs namespace5.3.1 Using Azure PowerShell5.3.2 Throughput units5.3.3 Event Hub geo-disaster recovery5.3.4 Failover with geo-disaster recovery5.4 Creating an Event Hub5.4.1 Using Azure portal5.4.2 Using Azure PowerShell5.4.3 Shared access policy5.5 Event Hub partitions5.5.1 Multiple consumers5.5.2 Why specify a partition?5.5.3 Why not specify a partition?5.5.4 Event Hubs message journal5.5.5 Partitions and throughput units5.6 Configuring Capture5.6.1 File name formats5.6.2 Secure access for Capture5.6.3 Enabling Capture5.6.4 The importance of time5.7 Securing access to Event Hubs5.7.1 Shared Access Signature policies5.7.2 Writing to Event Hubs5.8 Exercises5.8.1 Exercise 15.8.2 Exercise 25.8.3 Exercise 3Summary
6 Real-time queries with Azure Stream Analytics
6.1 Creating a Stream Analytics service6.1.1 Elements of a Stream Analytics job6.1.2 Create an ASA job using the Azure portal6.1.3 Create an ASA job using Azure PowerShell6.2 Configuring inputs and outputs6.2.1 Event Hub job input6.2.2 ASA job outputs6.3 Creating a job query6.3.1 Starting the ASA job6.3.2 Failure to start6.3.3 Output exceptions6.4 Writing job queries6.4.1 Window functions6.4.2 Machine learning functions6.5 Managing performance6.5.1 Streaming units6.5.2 Event ordering6.6 Exercises6.6.1 Exercise 16.6.2 Exercise 2Summary
7 Batch queries with Azure Data Lake Analytics
7.1 U-SQL language7.1.1 Extractors7.1.2 Outputters7.1.3 File selectors7.1.4 Expressions7.2 U-SQL jobs7.2.1 Selecting the biometric data files7.2.2 Schema extraction7.2.3 Aggregation7.2.4 Writing files7.3 Creating a Data Lake Analytics service7.3.1 Using Azure portal7.3.2 Using Azure PowerShell7.4 Submitting jobs to ADLA7.4.1 Using Azure portal7.4.2 Using Azure PowerShell7.5 Efficient U-SQL job executions7.5.1 Monitoring a U-SQL job7.5.2 Analytics units7.5.3 Vertexes7.5.4 Scaling the job execution7.6 Using Blob Storage7.6.1 Constructing Blob file selectors7.6.2 Adding a new data source7.6.3 Filtering rowsets7.7 Exercises7.7.1 Exercise 17.7.2 Exercise 2Summary
8 U-SQL for complex analytics
8.1 Data Lake Analytics Catalog8.1.1 Simplifying U-SQL queries8.1.2 Simplifying data access8.1.3 Loading data for reuse8.2 Window functions8.3 Local C# functions8.4 Exercises8.4.1 Exercise 18.4.2 Exercise 2Summary
9 Integrating with Azure Data Lake Analytics
9.1 Processing unstructured data9.1.1 Azure Cognitive Services9.1.2 Managing assemblies in the Data Lake9.1.3 Image data extraction with Advanced Analytics9.2 Reading different file types9.2.1 Adding custom libraries with a Catalog9.2.2 Creating a catalog database9.2.3 Building the U-SQL DataFormats solution9.2.4 Code folders9.2.5 Using custom assemblies9.3 Connecting to remote sources9.3.1 External databases9.3.2 Credentials9.3.3 Data Source9.3.4 Tables and views9.4 Exercises9.4.1 Exercise 19.4.2 Exercise 2Summary
10 Service integration with Azure Data Factory
10.1 Creating an Azure Data Factory service10.2 Secure authentication10.2.1 Azure Active Directory integration10.2.2 Azure Key Vault10.3 Copying files with ADF10.3.1 Creating a Files storage container10.3.2 Adding secrets to AKV10.3.3 Creating a Files storage linkedservice10.3.4 Creating an ADLS linkedservice10.3.5 Creating a pipeline and activity10.3.6 Creating a scheduled trigger10.4 Running an ADLA job10.4.1 Creating an ADLA linkedservice10.4.2 Creating a pipeline and activity10.5 Exercises10.5.1 Exercise 110.5.2 Exercise 2Summary
11 Managed SQL with Azure SQL Database
11.1 Creating an Azure SQL Database11.1.1 Create a SQL Server and SQLDB11.2 Securing SQLDB11.3 Availability and recovery11.3.1 Restoring and moving SQLDB11.3.2 Database safeguards11.3.3 Creating alerts for SQLDB11.4 Optimizing costs for SQLDB11.4.1 Pricing structure11.4.2 Scaling SQLDB11.4.3 Serverless11.4.4 Elastic Pools11.5 Exercises11.5.1 Exercise 111.5.2 Exercise 211.5.3 Exercise 311.5.4 Exercise 4Summary
12 Integrating Data Factory with SQL Database
12.1 Before you begin12.2 Importing data with external data sources12.2.1 Creating a database scoped credential12.2.2 Creating an external data source12.2.3 Creating an external table12.2.4 Importing Blob files12.3 Importing file data with ADF12.3.1 Authenticating between ADF and SQLDB12.3.2 Creating SQL Database linkedservice12.3.3 Creating datasets12.3.4 Creating a copy activity and pipeline12.4 Exercises12.4.1 Exercise 112.4.2 Exercise 212.4.3 Exercise 3Summary
13 Where to go next
13.1 Data catalog13.1.1 Data Catalog as a service13.1.2 Data locations13.1.3 Data definitions13.1.4 Data frequency13.1.5 Business drivers13.2 Version control and backups13.2.1 Blob Storage13.2.2 Data Lake Storage13.2.3 Stream Analytics13.2.4 Data Lake Analytics13.2.5 Data Factory configuration files13.2.6 SQL Database13.3 Microsoft certifications13.4 Signing offSummary
appendix A. Setting up Azure services through PowerShell
A.1 Setting up Azure PowerShellA.2 Create a subscriptionA.3 Azure naming conventionsA.4 Setting up common Azure resources using PowerShellA.4.1 Creating a new resource groupA.4.2 Creating a new Azure Active Directory userA.4.3 Creating a new Azure Active Directory groupA.5 Setting up Azure services using PowerShellA.5.1 Creating a new Storage accountA.5.2 Creating a new Data Lake storeA.5.3 Create new Event HubA.5.4 Create new Stream Analytics jobA.5.5 Create new Data Lake Analytics accountA.5.6 Create new SQL Server and DatabaseA.5.7 Create a new Data Factory serviceA.5.8 Creating a new App registrationA.5.9 Creating a new key vaultA.5.10 Create new SQL Server and Database with lookup data
appendix B. Configuring the Jonestown Sluggers analytics system
B.1 Solution designB.1.1 Hot pathB.1.2 Cold pathB.2 Naming conventionB.3 Creation scriptB.4 Configure Azure services using PowerShellB.4.1 Stream Analytics Managed IdentityB.4.2 Data Lake storeB.4.3 Stream Analytics job configurationB.4.4 SQL DatabaseB.4.5 Data FactoryB.5 Load event dataB.6 Output of batch and stream processingB.7 Removing services
index

Content preview from Azure Storage, Streaming, and Batch Analytics

7 Batch queries with Azure Data Lake Analytics

This chapter covers

Writing job queries using U-SQL
Creating U-SQL jobs
Creating a Data Lake Analytics service
Estimating appropriate parallelization for U-SQL jobs

In the last chapter, you used Azure Stream Analytics as a source for raw data, using a passthrough query. The passthrough query takes incoming data and passes it to the output, in this case files in Azure Data Lake Storage (ADLS). Figure 7.1 shows this use of Stream Analytics in parallel with the serving layer.

Figure 7.1 Lambda architecture with Azure PaaS Speed layer

This is the latest example of prep work for batch processing ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Writing and Running Azure Data Lake Analytics Jobs

Publisher Resources

ISBN: 9781617296307Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Azure Storage, Streaming, and Batch Analytics

by Richard Nuckolls

7 Batch queries with Azure Data Lake Analytics

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.