Apache Spark with Scala – Hands-On with Big Data!

Apache Spark with Scala – Hands-On with Big Data!

by Frank Kane
Released October 2022
Publisher(s): Packt Publishing
ISBN: 9781787129849

Watch it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Video description

“Big data” analysis is a hot and highly valuable skill—and this course will teach you the hottest technology in big data: Apache Spark. Employers including Amazon, eBay, NASA JPL, and Yahoo all use Spark to quickly extract meaning from massive datasets across a fault-tolerant Hadoop cluster. You will learn those same techniques using your own Windows system right at home. It is easier than you think, and you will learn from an ex-engineer and senior manager from Amazon and IMDb.

In this course, you will learn the concepts of Spark’s Resilient Distributed Datasets, DataFrames, and datasets. We will also cover a crash course in the Scala programming language that will help you with the course. You will learn how to develop and run Spark jobs quickly using Scala, IntelliJ, and SBT. You will learn how to translate complex analysis problems into iterative or multi-stage Spark scripts. You will learn how to scale up to larger datasets using Amazon’s Elastic MapReduce service and understand how Hadoop YARN distributes Spark across computing clusters. We will also be practicing using other Spark technologies, such as Spark SQL, DataFrames, DataSets, Spark Streaming, Machine Learning, and GraphX.

By the end of this course, you will be running code that analyzes gigabytes worth of information—in the cloud—in a matter of minutes.

what You Will Learn

  • Learn the concepts of Spark’s RDD, DataFrames, and Datasets
  • Get a crash course in the Scala programming language
  • Develop and run Spark jobs quickly using Scala, IntelliJ, and SBT
  • Translate complex analysis problems into iterative or multi-stage Spark scripts
  • Scale up to larger datasets using Amazon’s Elastic MapReduce service
  • Understand how Hadoop YARN distributes Spark across computing clusters

Audience

This course is designed for software engineers who want to expand their skills into the world of big data processing on a cluster. It is necessary to have some prior programming or scripting knowledge.

About The Author

Frank Kane: Frank Kane has spent nine years at Amazon and IMDb, developing and managing the technology that automatically delivers product and movie recommendations to hundreds of millions of customers all the time. He holds 17 issued patents in the fields of distributed computing, data mining, and machine learning. In 2012, Frank left to start his own successful company, Sundog Software, which focuses on virtual reality environment technology and teaches others about big data analysis.

Table of contents

  1. Chapter 1 : Getting Started
    1. Introduction and Installing the Course Materials, IntelliJ, and Scala
    2. Introduction to Apache Spark
  2. Chapter 2 : Scala Crash Course (Optional)
    1. (Activity) Scala Basics
    2. (Exercise) Flow Control in Scala
    3. (Exercise) Functions in Scala
    4. (Exercise) Data Structures in Scala
  3. Chapter 3 : Using Resilient Distributed Datasets (RDDs)
    1. The Resilient Distributed Dataset
    2. Ratings Histogram Example
    3. Spark Internals
    4. Key / Value RDDs, and the Average Friends by Age Example
    5. (Activity) Running the Average Friends by Age Example
    6. Filtering RDDs, and the Minimum Temperature by Location Example
    7. (Activity) Running the Minimum Temperature Example, and Modifying It for Maximum
    8. (Activity) Counting Word Occurrences Using Flatmap()
    9. (Activity) Improving the Word Count Script with Regular Expressions
    10. (Activity) Sorting the Word Count Results
    11. (Exercise) Find the Total Amount Spent by Customer
    12. (Exercise) Check Your Results and Sort Them by Total Amount Spent
    13. Check Your Results and Implementation Against Mine
  4. Chapter 4 : SparkSQL, DataFrames, and DataSets
    1. Introduction to SparkSQL
    2. (Activity) Using SparkSQL
    3. (Activity) Using DataSets
    4. (Exercise) Implement the Friends by Age Example Using DataSets
    5. Exercise Solution: Friends by Age, with DataSets
    6. (Activity) Word Count Example Using DataSets
    7. (Activity) Revisiting the Minimum Temperature Example, with DataSets
    8. (Exercise) Implement the Total Spent by Customer Problem with DataSets
    9. Exercise Solution: Total Spent by Customer with DataSets
  5. Chapter 5 : Advanced Examples of Spark Programs
    1. (Activity) Find the Most Popular Movie
    2. (Activity) Use Broadcast Variables to Display Movie Names
    3. (Activity) Find the Most Popular Superhero in a Social Graph
    4. (Exercise) Find the Most Obscure Superheroes
    5. Exercise Solution: Find the Most Obscure Superheroes
    6. Superhero Degrees of Separation: Introducing Breadth-First Search
    7. Superhero Degrees of Separation: Accumulators, and Implementing BFS in Spark
    8. (Activity) Superhero Degrees of Separation: Review the Code and Run It!
    9. Item-Based Collaborative Filtering in Spark, cache(), and persist()
    10. (Activity) Running the Similar Movies Script Using Spark's Cluster Manager
    11. (Exercise) Improve the Quality of Similar Movies
  6. Chapter 6 : Running Spark on a Cluster
    1. (Activity) Using spark-submit to Run Spark Driver Scripts
    2. (Activity) Packaging Driver Scripts with SBT
    3. (Exercise) Package a Script with SBT and Run It Locally with spark-submit
    4. Exercise Solution: Using SBT and spark-submit
    5. Introducing Amazon Elastic MapReduce
    6. Creating Similar Movies from One Million Ratings on EMR
    7. Partitioning
    8. Best Practices for Running on a Cluster
    9. Troubleshooting and Managing Dependencies
  7. Chapter 7 : Machine Learning with Spark ML
    1. Introducing MLLib
    2. (Activity) Using MLLib to Produce Movie Recommendations
    3. Linear Regression with MLLib
    4. (Activity) Running a Linear Regression with Spark
    5. (Exercise) Predict Real Estate Values with Decision Trees in Spark
    6. Exercise Solution: Predicting Real Estate with Decision Trees in Spark
  8. Chapter 8 : Introduction to Spark Streaming
    1. The DStream API for Spark Streaming
    2. (Activity) Real-Time Monitoring of the Most Popular Hashtags on Twitter
    3. Structured Streaming
    4. (Activity) Using Structured Streaming for Real-Time Log Analysis
    5. (Exercise) Windowed Operations with Structured Streaming
    6. Exercise Solution: Top URLs in a 30-Second Window
  9. Chapter 9 : Introduction to GraphX
    1. GraphX, Pregel, and Breadth-First Search with Pregel
    2. Using the Pregel API with Spark GraphX
    3. (Activity) Superhero Degrees of Separation Using GraphX
  10. Chapter 10 : You Made It! Where to Go from Here
    1. Learning More, and Career Tips

Product information

  • Title: Apache Spark with Scala – Hands-On with Big Data!
  • Author(s): Frank Kane
  • Release date: October 2022
  • Publisher(s): Packt Publishing
  • ISBN: 9781787129849