Skip to Main Content
Data Algorithms with Spark
book

Data Algorithms with Spark

by Mahmoud Parsian
April 2022
Intermediate to advanced content levelIntermediate to advanced
435 pages
9h 44m
English
O'Reilly Media, Inc.
Book available
Content preview from Data Algorithms with Spark

Chapter 9. Classic Data Design Patterns

This chapter discusses some of the most fundamental and classic data design patterns used in the vast majority of big data solutions. Even though these are simple design patterns, they are useful in solving many common data problems, and I’ve used many of them in examples in this book. In this chapter, I will present PySpark implementations of the following design patterns:

  1. Input-Map-Output

  2. Input-Filter-Output

  3. Input-Map-Reduce-Output

  4. Input-Multiple-Maps-Reduce-Output

  5. Input-Map-Combiner-Reduce-Output

  6. Input-MapPartitions-Reduce-Output

  7. Input-Inverted-Index-Pattern-Output

Before we get started, however, I’d like to address the question of what I mean by “design patterns.” In computer science and software engineering, given a commonly occurring problem, a design pattern is a reusable solution to that problem. It’s a template or best practice for how to solve a problem, not a finished design that can be transformed directly into code. The patterns presented in this chapter will equip you to handle a wide range of data analysis tasks.

Note

The data design patterns discussed in this chapter are basic patterns. You can create your own, depending on your requirements. For additional examples, see “MapReduce: Simplified Data Processing on Large Clusters” by Jeffrey Dean and Sanjay Ghemawat.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Data Algorithms

Data Algorithms

Mahmoud Parsian
Algorithms and Data Structures for Massive Datasets

Algorithms and Data Structures for Massive Datasets

Dzejla Medjedovic, Emin Tahirovic, Ines Schweigert

Publisher Resources

ISBN: 9781492082378Errata PageSupplemental Content