book

Designing Cloud Data Platforms

Name: Designing Cloud Data Platforms
ISBN: 9781617296444

by Lynda Partner, Danil Zburivsky

May 2021

Beginner to intermediate

336 pages

11h

English

Manning Publications

Audiobook available

Read now

Unlock full access

inside front cover
Designing Cloud Data Platforms
Copyright
brief contents
contents
front matter
prefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A roadmapAbout the codeliveBook discussion forumabout the authorsabout the cover illustration
1 Introducing the data platform
1.1 The trends behind the change from data warehouses to data platforms1.2 Data warehouses struggle with data variety, volume, and velocity1.2.1 Variety1.2.2 Volume1.2.3 Velocity1.2.4 All the V’s at once1.3 Data lakes to the rescue?1.4 Along came the cloud1.5 Cloud, data lakes, and data warehouses: The emergence of cloud data platforms1.6 Building blocks of a cloud data platform1.6.1 Ingestion layer1.6.2 Storage layer1.6.3 Processing layer1.6.4 Serving layer1.7 How the cloud data platform deals with the three V’s1.7.1 Variety1.7.2 Volume1.7.3 Velocity1.7.4 Two more V’s1.8 Common use casesSummary
2 Why a data platform and not just a data warehouse
2.1 Cloud data platforms and cloud data warehouses: The practical aspects2.1.1 A closer look at the data sources2.1.2 An example cloud data warehouse–only architecture2.1.3 An example cloud data platform architecture2.2 Ingesting data2.2.1 Ingesting data directly into Azure Synapse2.2.2 Ingesting data into an Azure data platform2.2.3 Managing changes in upstream data sources2.3 Processing data2.3.1 Processing data in the warehouse2.3.2 Processing data in the data platform2.4 Accessing data2.5 Cloud cost considerationsSummary2.6 Exercise answers
3 Getting bigger and leveraging the Big 3: Amazon, Microsoft Azure, and Google
3.1 Cloud data platform layered architecture3.1.1 Data ingestion layer3.1.2 Fast and slow storage3.1.3 Processing layer3.1.4 Technical metadata layer3.1.5 The serving layer and data consumers3.1.6 Orchestration and ETL overlay layers3.2 The importance of layers in a data platform architecture3.3 Mapping cloud data platform layers to specific tools3.3.1 AWS3.3.2 Google Cloud3.3.3 Azure3.4 Open source and commercial alternatives3.4.1 Batch data ingestion3.4.2 Streaming data ingestion and real-time analytics3.4.3 Orchestration layerSummary3.5 Exercise answers
4 Getting data into the platform
4.1 Databases, files, APIs, and streams4.1.1 Relational databases4.1.2 Files4.1.3 SaaS data via API4.1.4 Streams4.2 Ingesting data from relational databases4.2.1 Ingesting data from RDBMSs using a SQL interface4.2.2 Full-table ingestion4.2.3 Incremental table ingestion4.2.4 Change data capture (CDC)4.2.5 CDC vendors overview4.2.6 Data type conversion4.2.7 Ingesting data from NoSQL databases4.2.8 Capturing important metadata for RDBMS or NoSQL ingestion pipelines4.3 Ingesting data from files4.3.1 Tracking ingested files4.3.2 Capturing file ingestion metadata4.4 Ingesting data from streams4.4.1 Differences between batch and streaming ingestion4.4.2 Capturing streaming pipeline metadata4.5 Ingesting data from SaaS applications4.5.1 No standard approach to API design4.5.2 No standard way to deal with full vs. incremental data exports4.5.3 Resulting data is typically highly nested JSON4.6 Network and security considerations for data ingestion into the cloud4.6.1 Connecting other networks to your cloud data platformSummary4.7 Exercise answers

5 Organizing and processing data
5.1 Processing as a separate layer in the data platform5.2 Data processing stages5.3 Organizing your cloud storage5.3.1 Cloud storage containers and folders5.4 Common data processing steps5.4.1 File format conversion5.4.2 Data deduplication5.4.3 Data quality checks5.5 Configurable pipelinesSummary5.6 Exercise answers
6 Real-time data processing and analytics
6.1 Real-time ingestion vs. real-time processing6.2 Use cases for real-time data processing6.2.1 Retail use case: Real-time ingestion6.2.2 Online gaming use case: Real-time ingestion and real-time processing6.2.3 Summary of real-time ingestion vs. real-time processing6.3 When should you use real-time ingestion and/or real-time processing?6.4 Organizing data for real-time use6.4.1 The anatomy of fast storage6.4.2 How does fast storage scale?6.4.3 Organizing data in the real-time storage6.5 Common data transformations in real time6.5.1 Causes of duplicates in real-time systems6.5.2 Deduplicating data in real-time systems6.5.3 Converting message formats in real-time pipelines6.5.4 Real-time data quality checks6.5.5 Combining batch and real-time data6.6 Cloud services for real-time data processing6.6.1 AWS real-time processing services6.6.2 Google Cloud real-time processing services6.6.3 Azure real-time processing servicesSummary6.7 Exercise answers
7 Metadata layer architecture
7.1 What we mean by metadata7.1.1 Business metadata7.1.2 Data platform internal metadata or “pipeline metadata”7.2 Taking advantage of pipeline metadata7.3 Metadata model7.3.1 Metadata domains7.4 Metadata layer implementation options7.4.1 Metadata layer as a collection of configuration files7.4.2 Metadata database7.4.3 Metadata API7.5 Overview of existing solutions7.5.1 Cloud metadata services7.5.2 Open source metadata layer implementationsSummary7.6 Exercise answers
8 Schema management
8.1 Why schema management8.1.1 Schema changes in a traditional data warehouse architecture8.1.2 Schema-on-read approach8.2 Schema-management approaches8.2.1 Schema as a contract8.2.2 Schema management in the data platform8.2.3 Monitoring schema changes8.3 Schema Registry Implementation8.3.1 Apache Avro schemas8.3.2 Existing Schema Registry implementations8.3.3 Schema Registry as part of a Metadata layer8.4 Schema evolution scenarios8.4.1 Schema compatibility rules8.4.2 Schema evolution and data transformation pipelines8.5 Schema evolution and data warehouses8.5.1 Schema-management features of cloud data warehousesSummary8.6 Exercise answers
9 Data access and security
9.1 Different types of data consumers9.2 Cloud data warehouses9.2.1 AWS Redshift9.2.2 Azure Synapse9.2.3 Google BigQuery9.2.4 Choosing the right data warehouse9.3 Application data access9.3.1 Cloud relational databases9.3.2 Cloud key/value data stores9.3.3 Full-text search services9.3.4 In-memory cache9.4 Machine learning on the data platform9.4.1 Machine learning model lifecycle on a cloud data platform9.4.2 ML cloud collaboration tools9.5 Business intelligence and reporting tools9.5.1 Traditional BI tools and cloud data platform integration9.5.2 Using Excel as a BI tool9.5.3 BI tools that are external to the cloud provider9.6 Data security9.6.1 Users, groups, and roles9.6.2 Credentials and configuration management9.6.3 Data encryption9.6.4 Network boundariesSummary9.7 Exercise Answers
10 Fueling business value with data platforms
10.1 Why you need a data strategy10.2 The analytics maturity journey10.2.1 SEE: Getting insights from data10.2.2 PREDICT: Using data to predict what to do10.2.3 DO: Making your analytics actionable10.2.4 CREATE: Going beyond analytics into products10.3 The data platform: The engine that powers analytics maturity10.4 Platform project stoppers10.4.1 Time does indeed kill10.4.2 User adoption10.4.3 User trust and the need for data governance10.4.4 Operating in a platform silo10.4.5 The dollar danceSummary
index
inside back cover

Content preview from Designing Cloud Data Platforms

5 Organizing and processing data

This chapter covers

Organizing and processing data in your cloud data platform
Understanding the different stages of data processing
Discussing the rationale for separating storage from compute
Organizing data in cloud storage and designing a data flow
Implementing common data processing patterns
Choosing the right file formats for archive, staging, and production
Creating a single parameter-driven pipeline with common data transformations

We will introduce a number of concepts, such as the difference between common data processing steps (such as file format conversion, deduplication, and schema management) versus custom business logic (such as the rules each company chooses to apply to transform their data ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617296444Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Designing Cloud Data Platforms

by Lynda Partner, Danil Zburivsky

5 Organizing and processing data

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.