Chapter 8: Understanding How Runners Execute Pipelines
So far in this book, we have focused on Apache Beam from the user's perspective. We have seen how to code pipelines in the Java Software Development Kit (SDK), how to use Domain-Specific Languages (DSLs) such as SQL, and how to use portability with the Python SDK. In this chapter, we will focus on how the runner executes the pipeline. This will help us if we want to develop a runner for a new technology, debug our code, or improve performance issues.
We will not try to implement our own runner in this chapter. Instead, we will focus on the theoretical concepts that underpin runners. We will explore the building blocks of a typical runner, and this will help us understand how a runner executes ...
Get Building Big Data Pipelines with Apache Beam now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
