book

Designing Big Data Platforms

by Yusuf Aytas

July 2021

Beginner to intermediate

336 pages

9h 22m

English

Wiley

Read now

Unlock full access

1.1 Defining Modern Big Data Platform1.2 Fundamentals of a Modern Big Data Platform
2.1 A Bit of History2.2 What Makes Big Data2.3 Components of Big Data Architecture2.4 Making Use of Big Data

3.1 Problem Definition3.2 Processing Large Data with Linux Commands3.3 Processing Large Data with PostgreSQL3.4 Cost of Big Data
4.1 Big Data Storage Patterns4.2 On‐Premise Storage Solutions4.3 Cloud Storage Solutions4.4 Hybrid Storage Solutions
5.1 Defining Offline Data Processing5.2 MapReduce Technologies5.3 Apache Spark5.4 Apache Flink5.5 Presto
6.1 The Need for Stream Processing6.2 Defining Stream Data Processing6.3 Streams via Message Brokers6.4 Streams via Stream Engines
7.1 Log Collection7.2 Transferring Big Data Sets7.3 Aggregating Big Data Sets7.4 Data Pipeline Scheduler7.5 Patterns and Practices7.6 Exploring Data Visually
8.1 Data Science Applications8.2 Data Science Life Cycle8.3 Data Science Toolbox8.4 Productionalizing Data Science
9.1 Need for Data Discovery9.2 Data Governance9.3 Data Discovery Tools
10.1 Infrastructure Security10.2 Data Privacy10.3 Law Enforcement10.4 Data Security Tools
11.1 Platforms11.2 Big Data Systems and Tools11.3 Challenges
12.1 Event Sourcing12.2 Kappa Architecture12.3 Data Mesh12.4 Data Reservoirs12.5 Data Catalog12.6 Self‐service Platform12.7 Abstraction12.8 Data Guild12.9 Trade‐offs12.10 Data Ethics
A.1 Lambda ArchitectureA.2 Apache CassandraA.3 Apache Beam
B.1 Activity Tracking RecipeB.2 Data Quality AssuranceB.3 Estimating Time to DeliveryB.4 Incident Response RecipeB.5 Leveraging Spark SQL MetricsB.6 Airbnb Price Prediction

Content preview from Designing Big Data Platforms

Index

a

access control list (ACL) 200, 221
access permission 6, 16, 206
activeAPI 105
advanced message queuing protocol (AMQP) 92, 105
alerting 25, 234, 259
amundsen 191–193
ansible 49, 53, 221
Apache airflow 139–143
- airflow backfill 140
- airflow dag run 140–141
- airflow pool 140
- airflow queue 140
- airflow scheduler 141–142
- airflow sensor 140
- airflow worker 142
Apache Ambari 48, 233
Apache Atlas 193–197
- atlas bridge 196
- atlas hook 196
- atlas term 194
- atlas term category 194
- atlas term hierarchy 194
- atlas type system 193
Apache Avro 122, 129
Apache beam 228
Apache bookkeeper 101–102
- bookkeeper bookie 102
- bookkeeper ledger 102
Apache Cassandra 106, 112, 119, 225
Apache Flink 81–82, 106–111
- Flink alignment 109–110
- Flink barrier 109
- Flink DataSet 81
- Flink DataStream 81
- Flink Gelly 108
- Flink ML 108
- Flink sink 107
- Flink source 107
- Flink table API 82
- Flink TableEnvironment 82
- Flink task manager 110
Apache Flume 122
- Flume agent 122
- Flume channel 122
- Flume even 122
- Flume sink 122
Apache Gobblin 130–132
- Gobblin converter 131–132
- Gobblin extractor 131–132
- Gobblin publisher 131–132
- Gobblin quality checker 131–132
- Gobblin runtime 132
- Gobblin source 131–132
- Gobblin writer 131–132
Apache Hadoop 15, 44–53, 229, 244
Apache HBase 174, 196, 225
Apache Heron 116–118
- Heron instance 118
- Heron metrics manager 118
- Heron stream manager 118
- Heron topology master 117–118
- Heron tracker 118
- Heron UI 118
- Heron VIZ 118
Apache ...