book

Designing Big Data Platforms

by Yusuf Aytas

July 2021

Beginner to intermediate

336 pages

9h 22m

English

Wiley

Read now

Unlock full access

1.1 Defining Modern Big Data Platform1.2 Fundamentals of a Modern Big Data Platform
2.1 A Bit of History2.2 What Makes Big Data2.3 Components of Big Data Architecture2.4 Making Use of Big Data

3.1 Problem Definition3.2 Processing Large Data with Linux Commands3.3 Processing Large Data with PostgreSQL3.4 Cost of Big Data
4.1 Big Data Storage Patterns4.2 On‐Premise Storage Solutions4.3 Cloud Storage Solutions4.4 Hybrid Storage Solutions
5.1 Defining Offline Data Processing5.2 MapReduce Technologies5.3 Apache Spark5.4 Apache Flink5.5 Presto
6.1 The Need for Stream Processing6.2 Defining Stream Data Processing6.3 Streams via Message Brokers6.4 Streams via Stream Engines
7.1 Log Collection7.2 Transferring Big Data Sets7.3 Aggregating Big Data Sets7.4 Data Pipeline Scheduler7.5 Patterns and Practices7.6 Exploring Data Visually
8.1 Data Science Applications8.2 Data Science Life Cycle8.3 Data Science Toolbox8.4 Productionalizing Data Science
9.1 Need for Data Discovery9.2 Data Governance9.3 Data Discovery Tools
10.1 Infrastructure Security10.2 Data Privacy10.3 Law Enforcement10.4 Data Security Tools
11.1 Platforms11.2 Big Data Systems and Tools11.3 Challenges
12.1 Event Sourcing12.2 Kappa Architecture12.3 Data Mesh12.4 Data Reservoirs12.5 Data Catalog12.6 Self‐service Platform12.7 Abstraction12.8 Data Guild12.9 Trade‐offs12.10 Data Ethics
A.1 Lambda ArchitectureA.2 Apache CassandraA.3 Apache Beam
B.1 Activity Tracking RecipeB.2 Data Quality AssuranceB.3 Estimating Time to DeliveryB.4 Incident Response RecipeB.5 Leveraging Spark SQL MetricsB.6 Airbnb Price Prediction

Content preview from Designing Big Data Platforms

4Big Data Storage

After reading this chapter, you should be able to:

Identify the differences between different Big Data storage patterns

Set up on premise data storage

Use cloud data warehousing solutions

Come up with a hybrid data warehouse

Explain pros and cons of different solutions

In Chapter 3, few ways to deal with data were addressed. When the business requires more than simple solutions, we have to be ready. Before the emergence of big data, businesses already used solutions for analytics and business intelligence. However, these solutions are extremely expensive. Fortunately, the vast amount of data and advancement in distributed computing boosted the big data storage solutions. We will go through types of data storage and then discuss various technologies for different setups.

4.1 Big Data Storage Patterns

Over the years, storage patterns have changed drastically with regard to data processing. With the pace of big data, many concepts are presently obtained that are very close to each other but with subtle differences. The big data platform should effectively use relevant patterns to make the best out of the data. In this section, we will explain storage patterns: data lakes, data warehouses, and data marts.

4.1.1 Data Lakes

The appearance of the data lake concept is quite new. Dixon (2010) first referred to the term as a large body of water in a natural state. The catch here is that data lakes may contain raw data without any processing and need further processing ...