Chapter 14. Scalable Algorithms

In this chapter, we discuss the challenges associated with writing efficient and scalable analytics running on Spark. We will start by introducing the reader to the general concepts of distributed parallelization and scalability and how they relate to Spark. We will recap over Spark's distributed architecture giving the reader an understanding of its underlying principles and how this supports the parallel processing paradigm. We will learn about the characteristics of scalable analytics and the elements of Spark that underpin these (for example, RDD, combineByKey, and GraphX).

Next, we will learn about why sometimes even basic algorithms, despite working at small scale, will often fail in big data. We'll see how ...

Get Mastering Spark for Data Science now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.