Translating a Spark application from running in a local environment to running on a production cluster in the cloud requires several critical steps, including publishing artifacts, installing dependencies, and defining the steps in a pipeline. This video is a hands-on guide through the process of deploying your Spark ML pipelines in production. You’ll learn how to create a pipeline that supports model reproducibility—making your machine learning models more reliable—and how to update your pipeline incrementally as the underlying data change. Learners should have basic familiarity with the following: Scala or Python; Hadoop, Spark, or Pandas; SBT or Maven; Amazon Web Services such as S3, EMR, and EC2; Bash, Docker, and REST.
- Understand how various cloud ecosystem components interact (i.e., Amazon S3, EMR, EC2, and so on)
- Learn how to architect the components of a cloud ecosystem into an end-to-end model pipeline
- Explore the capabilities and limitations of Spark in building an end-to-end model pipeline
- Learn to write, publish, deploy, and schedule an ETL process using Spark on AWS using EMR
- Understand how to create a pipeline that supports model reproducibility and reliability
Jason Slepicka is a senior data engineer with Los Angeles based DataScience, where he builds pipelines and data science platform infrastructure. He has a decade of experience integrating data to support efforts like fighting human trafficking for DARPA, exploring the evolution of evolvability in yeast, and tracking intruders in computer networks. Jason has both a Bachelor's and Master’s in Computer Science from the University of Arizona and is working on his PhD in Computer Science at the University of Southern California Information Sciences Institute.
Table of contents
- Title: Deploying Spark ML Pipelines in Production on AWS
- Release date: December 2017
- Publisher(s): O'Reilly Media, Inc.
- ISBN: 9781491988862
You might also like
Software Engineering at Google
Today, software engineers need to know not only how to program effectively but also how to …
Head First Design Patterns, 2nd Edition
You know you don’t want to reinvent the wheel, so you look to design patterns—the lessons …
Analyzing Big Data with Spark and Amazon EMR
You're a software developer somewhat familiar with Apache Spark and how it's used to analyze Big …
Data Engineering with Python and AWS Lambda LiveLessons
7 Hours of Video Instruction Data Engineering with Python and AWS Lambda LiveLessons shows users how …