book

Mastering Large Datasets with Python

by John Wolohan

January 2020

Intermediate to advanced

312 pages

10h 22m

English

Manning Publications

Read now

Unlock full access

1.1. What you’ll learn in this book1.2. Why large datasets?1.3. What is parallel computing?1.4. The map and reduce style1.5. Distributed computing for speed and scale1.6. Hadoop: A distributed framework for map and reduce1.7. Spark for high-powered map, reduce, and more1.8. AWS Elastic MapReduce—Large datasets in the cloudSummary

2.1. An introduction to map2.2. Parallel processing2.3. Putting it all together: Scraping a Wikipedia network2.4. ExercisesSummary
3.1. Helper functions and function chains3.2. Unmasking hacker communications3.3. Twitter demographic projections3.4. ExercisesSummary
4.1. What is laziness?4.2. Some lazy functions to know4.3. Understanding iterators: The magic behind lazy Python4.4. The poetry puzzle: Lazily processing a large dataset4.5. Lazy simulations: Simulating fishing villages4.6. ExercisesSummary
5.1. N-to-X with reduce5.2. The three parts of reduce5.3. Reductions you’re familiar with5.4. Using map and reduce together5.5. Analyzing car trends with reduce5.6. Speeding up map and reduce5.7. ExercisesSummary
6.1. Getting the most out of parallel map6.2. Solving the parallel map and reduce paradoxSummary
7.1. Distributed computing7.2. Hadoop for batch processing7.3. Using Hadoop to find high-scoring words7.4. Spark for interactive workflows7.5. Document word scores in Spark7.6. ExercisesSummary
8.1. Unstructured data: Logs and documents8.2. Tennis analytics with Hadoop8.3. mrjob for Pythonic Hadoop streaming8.4. Tennis match analysis with mrjob8.5. ExercisesSummary
9.1. A closer look at PySpark9.2. Tennis rankings with Elo and PageRank in PySpark9.3. ExercisesSummary
10.1. What is machine learning?10.2. Machine learning basics with decision tree classifiers10.3. Fast random forest classifications in PySparkSummary
11.1. AWS Simple Storage Service—A solution for large datasets11.2. Storing data in the cloud with S311.3. ExercisesSummary
12.1. Running Hadoop on EMR with mrjob12.2. Machine learning in the cloud with Spark on EMR12.3. ExercisesSummary

Content preview from Mastering Large Datasets with Python

About this book

Who should read this book

The goal of this book is to teach a scalable style of programming. To do that, we’ll cover a wider range of material than you might be familiar with from other programming or technology books. Where other books might cover a single library, this book covers many libraries—both built-in modules, such as functools and itertools, as well as third-party libraries, such as toolz, pathos, and mrjob. Where other books cover just one technology, this book covers many, including Hadoop, Spark, and Amazon Web Services (AWS). The choice to cover a broad range of technologies is admitting the fact that to scale your code, you need to be able to adapt to new situations. Across all the technologies, however, I emphasize ...