PySpark Recipes: A Problem-Solution Approach with PySpark2

©  Raju Kumar Mishra 2018

Raju Kumar MishraPySpark Recipeshttps://doi.org/10.1007/978-1-4842-3141-8_4

4. Spark Architecture and the Resilient Distributed Dataset

Raju Kumar Mishra¹ 

(1)

Bangalore, Karnataka, India

 

You learned Python in the preceding chapter. Now it is time to learn PySpark and utilize the power of a distributed system to solve problems related to big data. We generally distribute large amounts of data on a cluster and perform processing on that distributed data.

• This chapter covers the following recipes:

• Recipe 4-1. Create an RDD

• Recipe 4-2. Convert temperature data

• Recipe 4-3. Perform basic data manipulation

• Recipe 4-4. Run set operations

• Recipe 4-5. Calculate summary statistics

• Recipe 4-6. Start PySpark shell on Standalone cluster manager ...