Video description
Processing big data in real time is challenging due to scalability, information consistency, and fault-tolerance. Big Data Processing with Apache Spark teaches you how to use Spark to make your overall analytical workflow faster and more efficient. You'll explore all core concepts and tools within the Spark ecosystem, such as Spark Streaming, the Spark Streaming API, machine learning extension, and structured streaming.
You'll begin by learning data processing fundamentals using Resilient Distributed Datasets (RDDs), SQL, Datasets, and Dataframes APIs. After grasping these fundamentals, you'll move on to using Spark Streaming APIs to consume data in real time from TCP sockets, and integrate Amazon Web Services (AWS) for stream consumption.
By the end of this course, you’ll not only have understood how to use machine learning extensions and structured streams but you’ll also be able to apply Spark in your own upcoming big data projects.
What You Will Learn
With this course, you will:
- Write your own Python programs that can interact with Spark
- Recognize common operations in Spark to process known data streams
- Integrate Spark streaming with Amazon Web Services
- Create a collaborative filtering model with Python and the movielens dataset
- Apply processed data streams to Spark machine learning APIs
Audience
This course is for you if you are a software engineer, architect, or IT professional who wants to explore distributed systems and big data analytics. Although you don‘t need any knowledge of Spark, prior experience of working with Python is recommended.
About The Author
john Bura: John Bura has been programming games since 1997 and teaching since 2002. He is the owner of the game development studio Mammoth Interactive. This company produces games for Xbox 360, iPhone, iPad, Android, HTML5, ad-games, and others. Mammoth Interactive recently sold a game to Nickelodeon! He has been contracted by many companies to provide game design, audio, programming, level design, and project management. To this day, he has contributed to 40 commercial games. Several of the games he has produced have risen to number one in Apple's App Store. In his spare time, he likes playing ultimate frisbee, cycling, and working out.
Publisher resources
Table of contents
-
Chapter 1 : Introduction to Spark Distributed Processing
- Course Overview
- Installation and Setup
- Lesson Overview
- Introduction to Spark and Resilient Distributed Datasets
- Operations Supported by the RDD API
- Map Reduce Operations
- Self-Contained Python Spark Programs
- Nested Functions and Standalone Python Programs
- Introduction to SQL, Datasets, and DataFrames
- Lesson Summary
- Chapter 2 : Introduction to Spark Streaming
- Chapter 3 : Spark Streaming Integration with AWS
- Chapter 4 : Spark Streaming, ML, and Windowing Operations
Product information
- Title: Big Data Processing with Apache Spark
- Author(s):
- Release date: January 2019
- Publisher(s): Packt Publishing
- ISBN: 9781789953688
You might also like
book
Designing Large Language Model Applications
Transformer-based language models are powerful tools for solving a variety of language tasks and represent a …
book
Learning Web Design, 5th Edition
Do you want to build web pages but have no prior experience? This friendly guide is …
book
Fluent Python, 2nd Edition
Don't waste time bending Python to fit patterns you've learned in other languages. Python's simplicity lets …
book
AWS Certified Database - Specialty (DBS-C01) Certification Guide
Pass the AWS Certified Database- Specialty Certification exam with the help of practice tests Key Features …