High Performance Spark, 2nd Edition

Chapter 1. Introduction to High Performance Spark

This chapter provides an overview of what we hope you will be able to learn from this book and does its best to convince you to learn to read some Scala. Feel free to skip ahead to Chapter 2 if you already know what you’re looking for.

What Is Spark and Why Performance Matters

ASF (currently) stands for Apache Software Foundation, although there are calls to rename the foundation.¹ Spark is a high performance, general-purpose data-parallel distributed computing system that has become the most active ASF open source project, with more than 2,000 active contributors. Spark enables us to process large quantities of data, beyond what can fit on a single machine, with a high-level, relatively easy-to-use API. Spark’s design and interface are unique, and it is one of the fastest systems of its kind. Uniquely, Spark allows us to write the logic of data transformations and machine learning algorithms in a parallelizable way while being relatively system agnostic.²

However, despite its many advantages and the excitement around Spark, the simplest implementation of many common data science routines in Spark can be much slower and less robust than the best version. Since the computations we are concerned with may involve data at a very large scale, the time and resource gains from tuning code for performance are enormous. Performance does not just mean running faster; often, at this scale, it means getting something to run at all. It is ...