Video Description
Efficiently tackle large data sets and big data analysis challenges using Spark and Python
About This Video
This course will allow the learner to:
- Get up and running with Apache Spark and Python
- Integrate Spark with AWS for real-time analytics
- Apply processed data streams to machine learning APIs of Apache Spark
In Detail
Processing big data in real time is challenging due to scalability, information consistency, and fault-tolerance. Big Data Processing with Apache Spark teaches you how to use Spark to make your overall analytical workflow faster and more efficient. You'll explore all core concepts and tools within the Spark ecosystem, such as Spark Streaming, the Spark Streaming API, machine learning extension, and structured streaming.
You'll begin by learning data processing fundamentals using Resilient Distributed Datasets (RDDs), SQL, Datasets, and Dataframes APIs. After grasping these fundamentals, you'll move on to using Spark Streaming APIs to consume data in real time from TCP sockets, and integrate Amazon Web Services (AWS) for stream consumption.
By the end of this course, you’ll not only have understood how to use machine learning extensions and structured streams but you’ll also be able to apply Spark in your own upcoming big data projects.
Downloading the example code for this course: You can download the example code files for all Packt video courses you have purchased from your account at http://www.PacktPub.com. If you purchased this course elsewhere, you can visit http://www.PacktPub.com/support and register to have the files e-mailed directly to you.
Table of Contents
-
Chapter 1 : Introduction to Spark Distributed Processing
- Course Overview 00:02:34
- Installation and Setup 00:04:50
- Lesson Overview 00:03:35
- Introduction to Spark and Resilient Distributed Datasets 00:16:14
- Operations Supported by the RDD API 00:15:42
- Map Reduce Operations 00:07:29
- Self-Contained Python Spark Programs 00:10:48
- Nested Functions and Standalone Python Programs 00:10:11
- Introduction to SQL, Datasets, and DataFrames 00:14:16
- Lesson Summary 00:00:44
-
Chapter 2 : Introduction to Spark Streaming
- Lesson Overview 00:01:18
- Introduction to Streaming Architectures 00:02:29
- Introduction to Discretized Streams (Dstreams) 00:12:27
- Operations Supported by the Spark Streaming API 00:17:56
- Windowing Operations 00:13:00
- Structured Streaming 00:11:08
- Lesson Summary 00:00:41
-
Chapter 3 : Spark Streaming Integration with AWS
- Lesson Overview 00:01:04
- Spark Integration with AWS Services 00:09:56
- Integrating AWS Kinesis and Python 00:14:46
- AWS S3 Basic Functionality 00:08:38
- Kinesis Streams and Spark Streams 00:01:54
- Lesson Summary 00:00:40
-
Chapter 4 : Spark Streaming, ML, and Windowing Operations
- Lesson Overview 00:01:12
- Spark Integration with Machine Learning 00:17:45
- Spark Streaming Windowing Operations 00:07:25
- Lesson Summary 00:01:32