book

Designing Cloud Data Platforms

by Lynda Partner, Danil Zburivsky

May 2021

Beginner to intermediate

336 pages

11h

English

Manning Publications

Read now

Unlock full access

prefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A roadmapAbout the codeliveBook discussion forumabout the authorsabout the cover illustration
1.1 The trends behind the change from data warehouses to data platforms1.2 Data warehouses struggle with data variety, volume, and velocity1.2.1 Variety1.2.2 Volume1.2.3 Velocity1.2.4 All the V’s at once1.3 Data lakes to the rescue?1.4 Along came the cloud1.5 Cloud, data lakes, and data warehouses: The emergence of cloud data platforms1.6 Building blocks of a cloud data platform1.6.1 Ingestion layer1.6.2 Storage layer1.6.3 Processing layer1.6.4 Serving layer1.7 How the cloud data platform deals with the three V’s1.7.1 Variety1.7.2 Volume1.7.3 Velocity1.7.4 Two more V’s1.8 Common use casesSummary
2.1 Cloud data platforms and cloud data warehouses: The practical aspects2.1.1 A closer look at the data sources2.1.2 An example cloud data warehouse–only architecture2.1.3 An example cloud data platform architecture2.2 Ingesting data2.2.1 Ingesting data directly into Azure Synapse2.2.2 Ingesting data into an Azure data platform2.2.3 Managing changes in upstream data sources2.3 Processing data2.3.1 Processing data in the warehouse2.3.2 Processing data in the data platform2.4 Accessing data2.5 Cloud cost considerationsSummary2.6 Exercise answers
3.1 Cloud data platform layered architecture3.1.1 Data ingestion layer3.1.2 Fast and slow storage3.1.3 Processing layer3.1.4 Technical metadata layer3.1.5 The serving layer and data consumers3.1.6 Orchestration and ETL overlay layers3.2 The importance of layers in a data platform architecture3.3 Mapping cloud data platform layers to specific tools3.3.1 AWS3.3.2 Google Cloud3.3.3 Azure3.4 Open source and commercial alternatives3.4.1 Batch data ingestion3.4.2 Streaming data ingestion and real-time analytics3.4.3 Orchestration layerSummary3.5 Exercise answers
4.1 Databases, files, APIs, and streams4.1.1 Relational databases4.1.2 Files4.1.3 SaaS data via API4.1.4 Streams4.2 Ingesting data from relational databases4.2.1 Ingesting data from RDBMSs using a SQL interface4.2.2 Full-table ingestion4.2.3 Incremental table ingestion4.2.4 Change data capture (CDC)4.2.5 CDC vendors overview4.2.6 Data type conversion4.2.7 Ingesting data from NoSQL databases4.2.8 Capturing important metadata for RDBMS or NoSQL ingestion pipelines4.3 Ingesting data from files4.3.1 Tracking ingested files4.3.2 Capturing file ingestion metadata4.4 Ingesting data from streams4.4.1 Differences between batch and streaming ingestion4.4.2 Capturing streaming pipeline metadata4.5 Ingesting data from SaaS applications4.5.1 No standard approach to API design4.5.2 No standard way to deal with full vs. incremental data exports4.5.3 Resulting data is typically highly nested JSON4.6 Network and security considerations for data ingestion into the cloud4.6.1 Connecting other networks to your cloud data platformSummary4.7 Exercise answers

5.1 Processing as a separate layer in the data platform5.2 Data processing stages5.3 Organizing your cloud storage5.3.1 Cloud storage containers and folders5.4 Common data processing steps5.4.1 File format conversion5.4.2 Data deduplication5.4.3 Data quality checks5.5 Configurable pipelinesSummary5.6 Exercise answers
6.1 Real-time ingestion vs. real-time processing6.2 Use cases for real-time data processing6.2.1 Retail use case: Real-time ingestion6.2.2 Online gaming use case: Real-time ingestion and real-time processing6.2.3 Summary of real-time ingestion vs. real-time processing6.3 When should you use real-time ingestion and/or real-time processing?6.4 Organizing data for real-time use6.4.1 The anatomy of fast storage6.4.2 How does fast storage scale?6.4.3 Organizing data in the real-time storage6.5 Common data transformations in real time6.5.1 Causes of duplicates in real-time systems6.5.2 Deduplicating data in real-time systems6.5.3 Converting message formats in real-time pipelines6.5.4 Real-time data quality checks6.5.5 Combining batch and real-time data6.6 Cloud services for real-time data processing6.6.1 AWS real-time processing services6.6.2 Google Cloud real-time processing services6.6.3 Azure real-time processing servicesSummary6.7 Exercise answers
7.1 What we mean by metadata7.1.1 Business metadata7.1.2 Data platform internal metadata or “pipeline metadata”7.2 Taking advantage of pipeline metadata7.3 Metadata model7.3.1 Metadata domains7.4 Metadata layer implementation options7.4.1 Metadata layer as a collection of configuration files7.4.2 Metadata database7.4.3 Metadata API7.5 Overview of existing solutions7.5.1 Cloud metadata services7.5.2 Open source metadata layer implementationsSummary7.6 Exercise answers
8.1 Why schema management8.1.1 Schema changes in a traditional data warehouse architecture8.1.2 Schema-on-read approach8.2 Schema-management approaches8.2.1 Schema as a contract8.2.2 Schema management in the data platform8.2.3 Monitoring schema changes8.3 Schema Registry Implementation8.3.1 Apache Avro schemas8.3.2 Existing Schema Registry implementations8.3.3 Schema Registry as part of a Metadata layer8.4 Schema evolution scenarios8.4.1 Schema compatibility rules8.4.2 Schema evolution and data transformation pipelines8.5 Schema evolution and data warehouses8.5.1 Schema-management features of cloud data warehousesSummary8.6 Exercise answers
9.1 Different types of data consumers9.2 Cloud data warehouses9.2.1 AWS Redshift9.2.2 Azure Synapse9.2.3 Google BigQuery9.2.4 Choosing the right data warehouse9.3 Application data access9.3.1 Cloud relational databases9.3.2 Cloud key/value data stores9.3.3 Full-text search services9.3.4 In-memory cache9.4 Machine learning on the data platform9.4.1 Machine learning model lifecycle on a cloud data platform9.4.2 ML cloud collaboration tools9.5 Business intelligence and reporting tools9.5.1 Traditional BI tools and cloud data platform integration9.5.2 Using Excel as a BI tool9.5.3 BI tools that are external to the cloud provider9.6 Data security9.6.1 Users, groups, and roles9.6.2 Credentials and configuration management9.6.3 Data encryption9.6.4 Network boundariesSummary9.7 Exercise Answers
10.1 Why you need a data strategy10.2 The analytics maturity journey10.2.1 SEE: Getting insights from data10.2.2 PREDICT: Using data to predict what to do10.2.3 DO: Making your analytics actionable10.2.4 CREATE: Going beyond analytics into products10.3 The data platform: The engine that powers analytics maturity10.4 Platform project stoppers10.4.1 Time does indeed kill10.4.2 User adoption10.4.3 User trust and the need for data governance10.4.4 Operating in a platform silo10.4.5 The dollar danceSummary

Content preview from Designing Cloud Data Platforms

4 Getting data into the platform

This chapter covers

Understanding databases, files, APIs, and streams
Ingesting data from RDBMSs using SQL versus change data capture
Parsing and ingesting data from various file formats
Developing strategies to deal with source schema changes
Designing an ingestion pipeline to handle the challenges of data streams
Building an ingestion pipeline for SaaS data
Implementing quality control and monitoring in your ingestion pipeline
Discussing network and security considerations for cloud data ingestion

If you’ve read the chapters up to this point, you’re able to architect a good, layered data lake. Now it’s time to start diving into a few of these layers in much greater detail.

In this chapter, we’ll focus on ...