book

Mastering Large Datasets with Python

by John Wolohan

January 2020

Intermediate to advanced

312 pages

10h 22m

English

Manning Publications

Read now

Unlock full access

1.1. What you’ll learn in this book1.2. Why large datasets?1.3. What is parallel computing?1.4. The map and reduce style1.5. Distributed computing for speed and scale1.6. Hadoop: A distributed framework for map and reduce1.7. Spark for high-powered map, reduce, and more1.8. AWS Elastic MapReduce—Large datasets in the cloudSummary

2.1. An introduction to map2.2. Parallel processing2.3. Putting it all together: Scraping a Wikipedia network2.4. ExercisesSummary
3.1. Helper functions and function chains3.2. Unmasking hacker communications3.3. Twitter demographic projections3.4. ExercisesSummary
4.1. What is laziness?4.2. Some lazy functions to know4.3. Understanding iterators: The magic behind lazy Python4.4. The poetry puzzle: Lazily processing a large dataset4.5. Lazy simulations: Simulating fishing villages4.6. ExercisesSummary
5.1. N-to-X with reduce5.2. The three parts of reduce5.3. Reductions you’re familiar with5.4. Using map and reduce together5.5. Analyzing car trends with reduce5.6. Speeding up map and reduce5.7. ExercisesSummary
6.1. Getting the most out of parallel map6.2. Solving the parallel map and reduce paradoxSummary
7.1. Distributed computing7.2. Hadoop for batch processing7.3. Using Hadoop to find high-scoring words7.4. Spark for interactive workflows7.5. Document word scores in Spark7.6. ExercisesSummary
8.1. Unstructured data: Logs and documents8.2. Tennis analytics with Hadoop8.3. mrjob for Pythonic Hadoop streaming8.4. Tennis match analysis with mrjob8.5. ExercisesSummary
9.1. A closer look at PySpark9.2. Tennis rankings with Elo and PageRank in PySpark9.3. ExercisesSummary
10.1. What is machine learning?10.2. Machine learning basics with decision tree classifiers10.3. Fast random forest classifications in PySparkSummary
11.1. AWS Simple Storage Service—A solution for large datasets11.2. Storing data in the cloud with S311.3. ExercisesSummary
12.1. Running Hadoop on EMR with mrjob12.2. Machine learning in the cloud with Spark on EMR12.3. ExercisesSummary

Content preview from Mastering Large Datasets with Python

Brief Table of Contents

Copyright

Brief Table of Contents

Table of Contents

Preface

Acknowledgments

About this book

About the author

About the cover illustration

1.

Chapter 1. Introduction

Chapter 2. Accelerating large dataset work: Map and parallel computing

Chapter 3. Function pipelines for mapping complex transformations

Chapter 4. Processing large datasets with lazy workflows

Chapter 5. Accumulation operations with reduce

Chapter 6. Speeding up map and reduce with advanced parallelization

2.

Chapter 7. Processing truly big datasets with Hadoop and Spark

Chapter 8. Best practices for large data with Apache Streaming and mrjob

Chapter 9. PageRank with map and reduce in PySpark

Chapter 10. Faster decision-making with machine learning ...