book

Mastering Large Datasets with Python

by John Wolohan

January 2020

Intermediate to advanced

312 pages

10h 22m

English

Manning Publications

Read now

Unlock full access

1.1. What you’ll learn in this book1.2. Why large datasets?1.3. What is parallel computing?1.4. The map and reduce style1.5. Distributed computing for speed and scale1.6. Hadoop: A distributed framework for map and reduce1.7. Spark for high-powered map, reduce, and more1.8. AWS Elastic MapReduce—Large datasets in the cloudSummary

2.1. An introduction to map2.2. Parallel processing2.3. Putting it all together: Scraping a Wikipedia network2.4. ExercisesSummary
3.1. Helper functions and function chains3.2. Unmasking hacker communications3.3. Twitter demographic projections3.4. ExercisesSummary
4.1. What is laziness?4.2. Some lazy functions to know4.3. Understanding iterators: The magic behind lazy Python4.4. The poetry puzzle: Lazily processing a large dataset4.5. Lazy simulations: Simulating fishing villages4.6. ExercisesSummary
5.1. N-to-X with reduce5.2. The three parts of reduce5.3. Reductions you’re familiar with5.4. Using map and reduce together5.5. Analyzing car trends with reduce5.6. Speeding up map and reduce5.7. ExercisesSummary
6.1. Getting the most out of parallel map6.2. Solving the parallel map and reduce paradoxSummary
7.1. Distributed computing7.2. Hadoop for batch processing7.3. Using Hadoop to find high-scoring words7.4. Spark for interactive workflows7.5. Document word scores in Spark7.6. ExercisesSummary
8.1. Unstructured data: Logs and documents8.2. Tennis analytics with Hadoop8.3. mrjob for Pythonic Hadoop streaming8.4. Tennis match analysis with mrjob8.5. ExercisesSummary
9.1. A closer look at PySpark9.2. Tennis rankings with Elo and PageRank in PySpark9.3. ExercisesSummary
10.1. What is machine learning?10.2. Machine learning basics with decision tree classifiers10.3. Fast random forest classifications in PySparkSummary
11.1. AWS Simple Storage Service—A solution for large datasets11.2. Storing data in the cloud with S311.3. ExercisesSummary
12.1. Running Hadoop on EMR with mrjob12.2. Machine learning in the cloud with Spark on EMR12.3. ExercisesSummary

Content preview from Mastering Large Datasets with Python

Part 2.

Part 2 teaches how to use two popular open source distributed computing frameworks: Hadoop and Spark. Hadoop is the originator and foundation of contemporary distributed computing. We’ll explore how to use Hadoop streaming and how to write Hadoop jobs with the mrjob library. We’ll also learn Spark, a modern distributed computing framework that can take full advantage of the latest, high-memory compute resources. You can use the tools and techniques in this part for large data in categories 2 and 3: tasks that needs parallelization to finish in a reasonable amount of time.