Skip to Main Content
Data Algorithms with Spark
book

Data Algorithms with Spark

by Mahmoud Parsian
April 2022
Intermediate to advanced content levelIntermediate to advanced
435 pages
9h 44m
English
O'Reilly Media, Inc.
Book available
Content preview from Data Algorithms with Spark

Chapter 10. Practical Data Design Patterns

The goal of this chapter is to introduce some practical data design patterns that are useful in solving common data problems. We will focus on actual design patterns that are used in big data solutions and deployed in production environments.

As in the previous chapter, I’ll provide simple examples to illustrate the use of each one and show you how to use Spark’s transformations to implement them. I’ll also talk more about the concept of monoids, to help you better understand reduction transformations.

The best design patterns book available is the iconic computer science book Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides (known as The “Gang of Four”). Rather than present data design patterns similar to those in the “Gang of Four” book, I will focus on practical, informal data design patterns that have been used in production environments.

The data design patterns that we’ll cover in this chapter can help us to write scalable solutions to be deployed on Spark clusters. However, be aware that when it comes to adopting and using design patterns, there is no silver bullet. Every pattern should be tested for performance and scalability using real data, in an environment similar to your production environment.

Note

For a general introduction to design patterns in software engineering, see the aforementioned Design Patterns: Elements of Reusable Object-Oriented ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Data Algorithms

Data Algorithms

Mahmoud Parsian
Algorithms and Data Structures for Massive Datasets

Algorithms and Data Structures for Massive Datasets

Dzejla Medjedovic, Emin Tahirovic, Ines Schweigert

Publisher Resources

ISBN: 9781492082378Errata PageSupplemental Content