book

Mastering Large Datasets with Python

by John Wolohan

January 2020

Intermediate to advanced

312 pages

10h 22m

English

Manning Publications

Read now

Unlock full access

1.1. What you’ll learn in this book1.2. Why large datasets?1.3. What is parallel computing?1.4. The map and reduce style1.5. Distributed computing for speed and scale1.6. Hadoop: A distributed framework for map and reduce1.7. Spark for high-powered map, reduce, and more1.8. AWS Elastic MapReduce—Large datasets in the cloudSummary

2.1. An introduction to map2.2. Parallel processing2.3. Putting it all together: Scraping a Wikipedia network2.4. ExercisesSummary
3.1. Helper functions and function chains3.2. Unmasking hacker communications3.3. Twitter demographic projections3.4. ExercisesSummary
4.1. What is laziness?4.2. Some lazy functions to know4.3. Understanding iterators: The magic behind lazy Python4.4. The poetry puzzle: Lazily processing a large dataset4.5. Lazy simulations: Simulating fishing villages4.6. ExercisesSummary
5.1. N-to-X with reduce5.2. The three parts of reduce5.3. Reductions you’re familiar with5.4. Using map and reduce together5.5. Analyzing car trends with reduce5.6. Speeding up map and reduce5.7. ExercisesSummary
6.1. Getting the most out of parallel map6.2. Solving the parallel map and reduce paradoxSummary
7.1. Distributed computing7.2. Hadoop for batch processing7.3. Using Hadoop to find high-scoring words7.4. Spark for interactive workflows7.5. Document word scores in Spark7.6. ExercisesSummary
8.1. Unstructured data: Logs and documents8.2. Tennis analytics with Hadoop8.3. mrjob for Pythonic Hadoop streaming8.4. Tennis match analysis with mrjob8.5. ExercisesSummary
9.1. A closer look at PySpark9.2. Tennis rankings with Elo and PageRank in PySpark9.3. ExercisesSummary
10.1. What is machine learning?10.2. Machine learning basics with decision tree classifiers10.3. Fast random forest classifications in PySparkSummary
11.1. AWS Simple Storage Service—A solution for large datasets11.2. Storing data in the cloud with S311.3. ExercisesSummary
12.1. Running Hadoop on EMR with mrjob12.2. Machine learning in the cloud with Spark on EMR12.3. ExercisesSummary

Content preview from Mastering Large Datasets with Python

Chapter 12. MapReduce in the cloud with Amazon’s Elastic MapReduce

This chapter covers

Launching and configuring cloud compute clusters with Elastic MapReduce
Running Hadoop jobs in the cloud with mrjob
Distributed cloud machine learning with Spark

Throughout this book, we’ve been talking about the ability to scale code up. We started by looking at how to parallelize code locally; then we moved on to distributed computing frameworks; and finally, in chapter 11, we introduced cloud computing technologies. In this chapter, we’ll look at techniques we can use to work with data of any scale. We’ll see how to take the Hadoop and Spark frameworks we covered in the middle of the book (chapters 7 and 8 for Hadoop; chapters 7, 9, and 10 for Spark) ...