book

Building Big Data Pipelines with Apache Beam

Name: Building Big Data Pipelines with Apache Beam
Author: Jan Lukavský
ISBN: 9781800564930

by Jan Lukavský

January 2022

Beginner to intermediate

342 pages

7h 19m

English

Packt Publishing

Read now

Unlock full access

Building Big Data Pipelines with Apache Beam
ContributorsAbout the authorAbout the reviewer
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
Section 1 Apache Beam: Essentials
Chapter 1: Introduction to Data Processing with Apache Beam
Technical requirementsWhy Apache Beam?Writing your first pipelineRunning our pipeline against streaming dataExploring the key properties of unbounded dataMeasuring event time progress inside data streamsStates and triggersTimersAssigning data to windowsDefining the life cycle of a state in terms of windowsPane accumulationUnifying batch and streaming data processingSummary
Chapter 2: Implementing, Testing, and Deploying Basic Pipelines
Technical requirementsSetting up the environment for this bookInstalling Apache KafkaMaking our code accessible from minikubeInstalling Apache FlinkReinstalling the complete environmentTask 1 – Calculating the K most frequent words in a stream of lines of textDefining the problemDiscussing the problem decompositionImplementing the solutionTesting our solutionDeploying our solutionTask 2 – Calculating the maximal length of a word in a streamDefining the problemDiscussing the problem decompositionImplementing the solutionTesting our solutionDeploying our solutionSpecifying the PCollection Coder object and the TypeDescriptor objectUnderstanding default triggers, on time, and closing behaviorIntroducing the primitive PTransform object – CombineTask 3 – Calculating the average length of words in a streamDefining the problemDiscussing the problem decompositionImplementing the solutionTesting our solutionDeploying our solutionTask 4 – Calculating the average length of words in a stream with fixed lookbackDefining the problemDiscussing the problem decompositionImplementing the solutionTesting our solutionDeploying our solutionEnsuring pipeline upgradabilityTask 5 – Calculating performance statistics for a sport activity tracking applicationDefining the problemDiscussing the problem decompositionSolution implementationTesting our solutionDeploying our solutionIntroducing the primitive PTransform object – GroupByKeyIntroducing the primitive PTransform object – PartitionSummary
Chapter 3: Implementing Pipelines Using Stateful Processing
Technical requirementsTask 6 – Using an external service for data augmentationDefining the problemDiscussing the problem decompositionImplementing the solutionTesting our solutionDeploying our solutionIntroducing the primitive PTransform object – stateless ParDoTask 7 – Batching queries to an external RPC serviceDefining the problemDiscussing the problem decompositionImplementing the solutionTask 8 – Batching queries to an external RPC service with defined batch sizesDefining the problemDiscussing the problem decompositionImplementing the solutionIntroducing the primitive PTransform object – stateful ParDoDescribing the theoretical properties of the stateful ParDo objectApplying the theoretical properties of the stateful ParDo object to the API of DoFnUsing side outputsAs an example, let's imagine we are processing data coming in as JSON values. We need to parse these messages into an internal object. But what should we do with the values that cannot be parsed because they contain a syntax error? If we do not do any validation before we store them in the stream (topic), then it is certainly possible that we will encounter such a situation. We can silently drop those records, but that is obviously not a great idea, as that could cause hard-to-debug problems. A much better option would be to store these values on the side to be able to investigate and fix them. Therefore, we should aim to do the following:Defining droppable data in BeamTask 9 – Separating droppable data from the rest of the data processingDefining the problemDiscussing the problem decompositionImplementing the solutionTesting our solutionDeploying our solutionTask 10 – Separating droppable data from the rest of the data processing, part 2Defining the problemDiscussing the problem decompositionImplementing the solutionTesting our solutionDeploying our solutionUsing side inputsSummary
Section 2 Apache Beam: Toward Improving Usability
Chapter 4: Structuring Code for Reusability
Technical requirementsExplaining PTransform expansionTask 11 – Enhancing SportTracker by runner motivation using side inputsProblem definitionProblem decomposition discussionSolution implementationTesting our solutionDeploying our solutionIntroducing composite transform – CoGroupByKeyTask 12 – enhancing SportTracker by runner motivation using CoGroupByKeyProblem definitionProblem decomposition discussionSolution implementationIntroducing the Join library DSLStream-to-stream joins explainedTask 13 – Writing a reusable PTransform – StreamingInnerJoinProblem definitionProblem decomposition discussionSolution implementationTesting our solutionDeploying our solutionTable-stream dualitySummary
Chapter 5: Using SQL for Pipeline Implementation
Technical requirementsUnderstanding schemasAttaching a schema to a PCollectionTransforms for PCollections with schemasImplementing our first streaming pipeline using SQLTask 14 – Implementing SQLMaxWordLengthProblem definitionProblem decomposition discussionSolution implementationTask 15 – Implementing SchemaSportTrackerProblem definitionProblem decomposition discussionSolution implementationTask 16 – Implementing SQLSportTrackerMotivationProblem definitionProblem decomposition discussionSolution implementationFurther development of Apache Beam SQLSummary
Chapter 6: Using Your Preferred Language with Portability
Technical requirementsIntroducing the portability layerPortable representation of the pipelineJob ServiceSDK harnessImplementing our first pipelines in the Python SDKImplementing our first Python pipelineImplementing our first streaming Python pipelineTask 17 – Implementing MaxWordLength in the Python SDKProblem definitionProblem decomposition discussionSolution implementationTesting our solutionDeploying our solutionPython SDK type hints and codersTask 18 – Implementing SportTracker in the Python SDKProblem definitionSolution implementationTesting our solutionDeploying our solutionTask 19 – Implementing RPCParDo in the Python SDKProblem definitionSolution implementationTesting our solutionDeploying our solutionTask 20 – Implementing SportTrackerMotivation in the Python SDKProblem definitionSolution implementationDeploying our solutionUsing the DataFrame APIInteractive programming using InteractiveRunnerIntroducing and using cross-language pipelinesSummary

Section 3 Apache Beam: Advanced Concepts
Chapter 7: Extending Apache Beam's I/O Connectors
Technical requirementsDefining splittable DoFn as a unification for bounded and unbounded sourcesTask 21 – Implementing our own splittable DoFn – a streaming file sourceThe problem definitionDiscussing the problem decompositionImplementing the solutionTesting our solutionDeploying our solutionTask 22 – A non-I/O application of splittable DoFn – PiSamplerThe problem definitionDiscussing the problem decompositionImplementing the solutionTesting our solutionDeploying our solutionThe legacy Source API and the Read transformWriting a custom data sinkThe inherent non-determinism of Apache Beam pipelinesSummary
Chapter 8: Understanding How Runners Execute Pipelines
Describing the anatomy of an Apache Beam runnerIdentifying which transforms should be overriddenExplaining the differences between classic and portable runnersClassic runnersPortable pipeline representationsThe executable stage concept and the pipeline fusion processUnderstanding how a runner handles stateEnsuring fault toleranceLocal state with periodic checkpointsRemote stateExploring the Apache Beam capability matrixUnderstanding windowing semantics in depthMerging and non-merging windowsDebugging pipelines and using Apache Beam metrics for observabilityUsing metrics in the Java SDKSummary
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Building Big Data Pipelines with Apache Beam

Chapter 7: Extending Apache Beam's I/O Connectors

In previous chapters, we focused on how to write data transformations after reading the data from data sources. There are two types of sources: bounded and unbounded. The difference between these is obvious – the size of the bounded type is limited (and this limitation is known in advance), while the size of the unbounded type is (possibly) infinite. A classic example of a bounded source is a file (or a set of immutable files), while an unbounded source is typically a streaming source such as Apache Kafka. Note that we can always convert an unbounded source to a bounded one by defining a bounding constraint. This could be, for example, the number of records that we want to read or the (processing ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781800564930

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Building Big Data Pipelines with Apache Beam

by Jan Lukavský

Chapter 7: Extending Apache Beam's I/O Connectors

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.