Frank Kane's Taming Big Data with Apache Spark and Python

Frank Kane's Taming Big Data with Apache Spark and Python

by Frank Kane
Released June 2017
Publisher(s): Packt Publishing
ISBN: 9781787287945

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Frank Kane’s hands-on Spark training course, based on his bestselling Taming Big Data with Apache Spark and Python video, now available in a book. Understand and analyze large data sets using Spark on a single system or on a cluster.

About This Book

  • Understand how Spark can be distributed across computing clusters

  • Develop and run Spark jobs efficiently using Python

  • A hands-on tutorial by Frank Kane with over 15 real-world examples teaching you Big Data processing with Spark

    • Who This Book Is For

    If you are a data scientist or data analyst who wants to learn Big Data processing using Apache Spark and Python, this book is for you. If you have some programming experience in Python, and want to learn how to process large amounts of data using Apache Spark, Frank Kane’s Taming Big Data with Apache Spark and Python will also help you.

    What You Will Learn

  • Find out how you can identify Big Data problems as Spark problems

  • Install and run Apache Spark on your computer or on a cluster

  • Analyze large data sets across many CPUs using Spark’s Resilient Distributed Datasets

  • Implement machine learning on Spark using the MLlib library

  • Process continuous streams of data in real time using the Spark streaming module

  • Perform complex network analysis using Spark’s GraphX library

  • Use Amazon's Elastic MapReduce service to run your Spark jobs on a cluster

    • In Detail

    Frank Kane’s Taming Big Data with Apache Spark and Python is your companion to learning Apache Spark in a hands-on manner. Frank will start you off by teaching you how to set up Spark on a single system or on a cluster, and you’ll soon move on to analyzing large data sets using Spark RDD, and developing and running effective Spark jobs quickly using Python.

    Apache Spark has emerged as the next big thing in the Big Data domain – quickly rising from an ascending technology to an established superstar in just a matter of years. Spark allows you to quickly extract actionable insights from large amounts of data, on a real-time basis, making it an essential tool in many modern businesses.

    Frank has packed this book with over 15 interactive, fun-filled examples relevant to the real world, and he will empower you to understand the Spark ecosystem and implement production-grade real-time Spark projects with ease.

    Style and approach

    Frank Kane’s Taming Big Data with Apache Spark and Python is a hands-on tutorial with over 15 real-world examples carefully explained by Frank in a step-by-step manner. The examples vary in complexity, and you can move through them at your own pace.

    Table of contents

    1. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Downloading the example code
        2. Downloading the color images of this book
        3. Errata
        4. Piracy
        5. Questions
    2. Getting Started with Spark
      1. Getting set up - installing Python, a JDK, and Spark and its dependencies
        1. Installing Enthought Canopy
        2. Installing the Java Development Kit
        3. Installing Spark
        4. Running Spark code
      2. Installing the MovieLens movie rating dataset
      3. Run your first Spark program - the ratings histogram example
        1. Examining the ratings counter script
        2. Running the ratings counter script
      4. Summary
    3. Spark Basics and Spark Examples
      1. What is Spark?
        1. Spark is scalable
        2. Spark is fast
        3. Spark is hot
        4. Spark is not that hard
        5. Components of Spark
        6. Using Python with Spark
      2. The Resilient Distributed Dataset (RDD)
        1. What is the RDD?
          1. The SparkContext object
          2. Creating RDDs
          3. Transforming RDDs
          4. Map example
          5. RDD actions
      3. Ratings histogram walk-through
        1. Understanding the code
          1. Setting up the SparkContext object
          2. Loading the data
          3. Extract (MAP) the data we care about
          4. Perform an action - count by value
          5. Sort and display the results
        2. Looking at the ratings-counter script in Canopy
      4. Key/value RDDs and the average friends by age example
        1. Key/value concepts - RDDs can hold key/value pairs
          1. Creating a key/value RDD
          2. What Spark can do with key/value data?
          3. Mapping the values of a key/value RDD
        2. The friends by age example
          1. Parsing (mapping) the input data
          2. Counting up the sum of friends and number of entries per age
          3. Compute averages
          4. Collect and display the results
      5. Running the average friends by age example
        1. Examining the script
        2. Running the code
      6. Filtering RDDs and the minimum temperature by location example
        1. What is filter()
          1. The source data for the minimum temperature by location example
          2. Parse (map) the input data
          3. Filter out all but the TMIN entries
          4. Create (station ID, temperature) key/value pairs
          5. Find minimum temperature by station ID
          6. Collect and print results
      7. Running the minimum temperature example and modifying it for maximums
        1. Examining the min-temperatures script
        2. Running the script
      8. Running the maximum temperature by location example
      9. Counting word occurrences using flatmap()
        1. Map versus flatmap
          1. Map ()
          2. Flatmap ()
        2. Code sample - count the words in a book
      10. Improving the word-count script with regular expressions
        1. Text normalization
        2. Examining the use of regular expressions in the word-count script
        3. Running the code
      11. Sorting the word count results
        1. Step 1 - Implement countByValue() the hard way to create a new RDD
        2. Step 2 - Sort the new RDD
        3. Examining the script
        4. Running the code
      12. Find the total amount spent by customer
        1. Introducing the problem
        2. Strategy for solving the problem
        3. Useful snippets of code
      13. Check your results and sort them by the total amount spent
      14. Check your sorted implementation and results against mine
      15. Summary
    4. Advanced Examples of Spark Programs
      1. Finding the most popular movie
        1. Examining the popular-movies script
        2. Getting results
      2. Using broadcast variables to display movie names instead of ID numbers
        1. Introducing broadcast variables
          1. Examining the popular-movies-nicer.py script
        2. Getting results
      3. Finding the most popular superhero in a social graph
        1. Superhero social networks
          1. Input data format
        2. Strategy
      4. Running the script - discover who the most popular superhero is
        1. Mapping input data to (hero ID, number of co-occurrences) per line
        2. Adding up co-occurrence by hero ID
        3. Flipping the (map) RDD to (number, hero ID)
        4. Using max() and looking up the name of the winner
        5. Getting results
      5. Superhero degrees of separation - introducing the breadth-first search algorithm
        1. Degrees of separation
          1. How the breadth-first search algorithm works?
          2. The initial condition of our social graph
          3. First pass through the graph
          4. Second pass through the graph
          5. Third pass through the graph
          6. Final pass through the graph
      6. Accumulators and implementing BFS in Spark
        1. Convert the input file into structured data
          1. Writing code to convert Marvel-Graph.txt to BFS nodes
        2. Iteratively process the RDD
          1. Using a mapper and a reducer
          2. How do we know when we're done?
      7. Superhero degrees of separation - review the code and run it
        1. Setting up an accumulator and using the convert to BFS function
          1. Calling flatMap()
          2. Calling an action
          3. Calling reduceByKey
        2. Getting results
      8. Item-based collaborative filtering in Spark, cache(), and persist()
        1. How does item-based collaborative filtering work?
        2. Making item-based collaborative filtering a Spark problem
        3. It's getting real
          1. Caching RDDs
      9. Running the similar-movies script using Spark's cluster manager
        1. Examining the script
        2. Getting results
      10. Improving the quality of the similar movies example
      11. Summary
    5. Running Spark on a Cluster
      1. Introducing Elastic MapReduce
        1. Why use Elastic MapReduce?
        2. Warning - Spark on EMR is not cheap
      2. Setting up our Amazon Web Services / Elastic MapReduce account and PuTTY
      3. Partitioning
        1. Using .partitionBy()
          1. Choosing a partition size
      4. Creating similar movies from one million ratings - part 1
        1. Changes to the script
      5. Creating similar movies from one million ratings - part 2
        1. Our strategy
          1. Specifying memory per executor
          2. Specifying a cluster manager
          3. Running on a cluster
        2. Setting up to run the movie-similarities-1m.py script on a cluster
          1. Preparing the script
          2. Creating a cluster
          3. Connecting to the master node using SSH
          4. Running the code
      6. Creating similar movies from one million ratings – part 3
        1. Assessing the results
        2. Terminating the cluster
      7. Troubleshooting Spark on a cluster
      8. More troubleshooting and managing dependencies
        1. Troubleshooting
          1. Managing dependencies
      9. Summary
    6. SparkSQL, DataFrames, and DataSets
      1. Introducing SparkSQL
        1. Using SparkSQL in Python
          1. More things you can do with DataFrames
        2. Differences between DataFrames and DataSets
        3. Shell access in SparkSQL
        4. User-defined functions (UDFs)
      2. Executing SQL commands and SQL-style functions on a DataFrame
        1. Using SQL-style functions instead of queries
      3. Using DataFrames instead of RDDs
      4. Summary
    7. Other Spark Technologies and Libraries
      1. Introducing MLlib
        1. MLlib capabilities
          1. Special MLlib data types
          2. For more information on machine learning
        2. Making movie recommendations
      2. Using MLlib to produce movie recommendations
        1. Examining the movie-recommendations-als.py script
      3. Analyzing the ALS recommendations results
        1. Why did we get bad results?
      4. Using DataFrames with MLlib
        1. Examining the spark-linear-regression.py script
        2. Getting results
      5. Spark Streaming and GraphX
        1. What is Spark Streaming?
        2. GraphX
      6. Summary
    8. Where to Go From Here? – Learning More About Spark and Data Science

    Product information

    • Title: Frank Kane's Taming Big Data with Apache Spark and Python
    • Author(s): Frank Kane
    • Release date: June 2017
    • Publisher(s): Packt Publishing
    • ISBN: 9781787287945