Video Description
Apache Spark 2.0 has become the gold standard for processing large datasets. This course, designed for learners with basic Python programming experience, takes you on an introductory journey into the world of big data analysis using Spark 2.0, Python, and the Spark DataFrame API.
Beginning with an overview of Spark 2.0 and Python, and then moving into a detailed examination of DataFrames, you'll learn about using SQL with DataFrames, DataFrame dates and timestamps, DataFrame aggregate operations, and about DataFrames and missing data. The course includes a hands-on data analysis exercise using real stock data. Learners should have Python and Spark installed on their computers before starting the class.
- Gain a core understanding of Spark 2.0 and Spark DataFrames
- Learn how to use Python with Spark DataFrames
- Gain big data experience analyzing stock data with Python and Spark DataFrames
Table of Contents
-
Introduction
- Welcome To The Course 00:00:45
- About The Author 00:00:28
- Course Curriculum Overview 00:00:33
-
Overview Of Spark 2.0
- What Is Spark 00:01:08
- Why Spark 2.0 DataFrames 00:00:29
-
DataFrame Basics
- Jupyter Notebook Overview 00:02:57
- Python Review Part One 00:08:11
- Python Review Part Two 00:08:09
- Creating A DataFrame 00:02:29
- Data Input 00:03:01
- Data Output 00:03:00
- Getting DataFrame Information 00:02:17
- Selecting Columns And Rows 00:03:00
- Creating and Renaming Columns 00:03:57
- Using SQL With DataFrames 00:02:27
- Filtering The Data 00:04:52
-
Spark DataFrame Dates And Timestamps
- Introduction To Date And Timestamps 00:00:15
- Working With Dates 00:04:22
- Working With Timestamps 00:04:03
-
Spark DataFrame Aggregate Operations
- Introduction To Aggregate And GroupBy Concepts 00:00:24
- Spark GroupBy Method 00:03:08
- Spark Built In Aggregate Methods 00:03:25
- Sorting And Ordering 00:01:34
-
Spark DataFrame Working With Missing Data
- Introduction To Missing Data 00:00:21
- Dropping Data 00:04:25
- Filling Missing Data 00:02:46
-
Spark DataFrame Exercises
- Introduction To Exercises 00:00:34
- Exercise Solutions 00:04:05
- Thank You