Introduction to PySpark

Table of contents

1. Introduction
   1. Introduction And Course Overview
   2. About The Author
   3. Installing Python
   4. Installing iPython And Using Notebooks
2. Installing Spark
   1. Download And Setup
   2. Running The Spark Shell
   3. Running The Spark Shell With iPython
3. Spark Fundamentals
   1. What Is A Resilient Distributed Dataset - RDD?
   2. Reading A Text File
   3. Actions
   4. Transformations
   5. Persisting Data
4. Transformations
   1. Map
   2. Filter
   3. Flatmap
   4. MapPartitions
   5. MapPartitionsWithIndex
   6. Sample
   7. Union
   8. Intersection
   9. Distinct
   10. Cartesian
   11. Pipe
   12. Coalesce
   13. Repartition
   14. RepartitionAndSortWithinPartitions
5. Actions
   1. Reduce
   2. Collect
   3. Count
   4. First
   5. Take
   6. TakeSample
   7. TakeOrdered
   8. SaveAsTextFile
   9. CountByKey
   10. ForEach
6. Key-Value Pair RDDs
   1. GroupByKey
   2. ReduceByKey
   3. AggregateByKey
   4. SortByKey
   5. Join
   6. CoGroup
7. Input And Output
   1. WholeTextFile
   2. Pickle Files
   3. HadoopInputFormat
   4. HadoopOutputFormat
8. Performance
   1. Broadcast Variables
   2. Accumulators
   3. Using A Custom Accumulator
   4. Partitioning
9. Running On A Cluster
   1. Spark Standalone Cluster
   2. Mesos
   3. Yarn
   4. Client Versus Cluster Mode
10. Advanced Spark
    1. Spark Streaming
    2. Dataframes And SQL
    3. MLlib
11. Conclusion
    1. Resources And Where To Go From Here
    2. Wrap Up