book

Mastering Large Datasets with Python

by John Wolohan

January 2020

Intermediate to advanced

312 pages

10h 22m

English

Manning Publications

Read now

Unlock full access

1.1. What you’ll learn in this book1.2. Why large datasets?1.3. What is parallel computing?1.4. The map and reduce style1.5. Distributed computing for speed and scale1.6. Hadoop: A distributed framework for map and reduce1.7. Spark for high-powered map, reduce, and more1.8. AWS Elastic MapReduce—Large datasets in the cloudSummary

2.1. An introduction to map2.2. Parallel processing2.3. Putting it all together: Scraping a Wikipedia network2.4. ExercisesSummary
3.1. Helper functions and function chains3.2. Unmasking hacker communications3.3. Twitter demographic projections3.4. ExercisesSummary
4.1. What is laziness?4.2. Some lazy functions to know4.3. Understanding iterators: The magic behind lazy Python4.4. The poetry puzzle: Lazily processing a large dataset4.5. Lazy simulations: Simulating fishing villages4.6. ExercisesSummary
5.1. N-to-X with reduce5.2. The three parts of reduce5.3. Reductions you’re familiar with5.4. Using map and reduce together5.5. Analyzing car trends with reduce5.6. Speeding up map and reduce5.7. ExercisesSummary
6.1. Getting the most out of parallel map6.2. Solving the parallel map and reduce paradoxSummary
7.1. Distributed computing7.2. Hadoop for batch processing7.3. Using Hadoop to find high-scoring words7.4. Spark for interactive workflows7.5. Document word scores in Spark7.6. ExercisesSummary
8.1. Unstructured data: Logs and documents8.2. Tennis analytics with Hadoop8.3. mrjob for Pythonic Hadoop streaming8.4. Tennis match analysis with mrjob8.5. ExercisesSummary
9.1. A closer look at PySpark9.2. Tennis rankings with Elo and PageRank in PySpark9.3. ExercisesSummary
10.1. What is machine learning?10.2. Machine learning basics with decision tree classifiers10.3. Fast random forest classifications in PySparkSummary
11.1. AWS Simple Storage Service—A solution for large datasets11.2. Storing data in the cloud with S311.3. ExercisesSummary
12.1. Running Hadoop on EMR with mrjob12.2. Machine learning in the cloud with Spark on EMR12.3. ExercisesSummary

Content preview from Mastering Large Datasets with Python

Chapter 5. Accumulation operations with reduce

This chapter covers

Recognizing the reduce pattern for N-to-X data transformations
Writing helper functions for reductions
Writing lambda functions for simple reductions
Using reduce to summarize data

In chapter 2, we learned about the first part of the map and reduce style of programming: map. In this chapter, we introduce the second part: reduce. As we noted in chapter 2, map performs N-to-N transformations. That is, if we have a situation where we want to take a sequence and get a same-sized sequence back, map is our go-to function. Among the examples of this that we’ve reviewed are file processing (we have a list of files and we want to do something to all of them; discussed in chapter ...