Data Engineering with Python and AWS Lambda LiveLessons

Data Engineering with Python and AWS Lambda LiveLessons

by Noah Gift, Robert Jordan, Kennedy Behrman
Released August 2019
Publisher(s): Pearson
ISBN: 0135964334

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Video description

7 Hours of Video Instruction

Data Engineering with Python and AWS Lambda LiveLessons shows users how to build complete and powerful data engineering pipelines in the same language that Data Scientists use to build Machine Learning models. By embracing serverless data engineering in Python, you can build highly scalable distributed systems on the back of the AWS backplane. Users learn to think in the new paradigm of serverless, which means to embrace events and event-driven programs that replace expensive and complicated servers.

Description

Some of the many benefits of programming with AWS Lambda in Python include no servers to manage, continuous scaling, and subsecond metering. Several use cases include data processing, stream processing, IoT backends, mobile, and web applications. Learn to take advantage of a new paradigm in software architecture that will make your code easier to write, maintain, and deploy.

AWS Lambda functions are the building blocks for creating sophisticated applications and services on AWS. In this LiveLesson, you learn to use Python to develop Lambda functions that communicate with key AWS services: API Gateway, SQS, and CloudWatch functions. You also learn how a new cloud-based development environment, Cloud9, can streamline writing, debugging, and deploying AWS Lambda functions.

About the Instructors

Noah Gift is a lecturer and consultant at both the UC Davis Graduate School of Management MSBA program and the Graduate Data Science program, MSDS, at Northwestern. He is teaching and designing graduate Machine Learning, AI, and Data Science courses, and consulting on Machine Learning and Cloud Architecture for students and faculty, including leading a multi-cloud certification initiative for students. Noah is a Python Software Foundation Fellow, AWS Subject Matter Expert (SME) on Machine Learning, AWS Certified Solutions Architect and AWS Academy Accredited Instructor, Google Certified Professional Cloud Architect, and Microsoft MTA on Python. Noah has published close to 100 technical publications, including two books on subjects ranging from Cloud Machine Learning to DevOps. Gift received an MBA from UC Davis, an M.S. in Computer Information Systems from Cal State Los Angeles, and a B.S. in Nutritional Science from Cal Poly San Luis Obispo. Currently, he is consulting startups and other companies on Machine Learning, Cloud Architecture, and CTO level consulting as the founder of Pragmatic AI Labs. His most recent book is Pragmatic AI: An Introduction to Cloud-Based Machine Learning (Pearson, 2018).

Robert Jordan is a visionary architect with more than 20 years of experience designing, implementing, and deploying production applications for some of the world’s largest media and scientific customers. He has successfully led projects on all major cloud platforms and is currently certified on both AWS and GCP platforms.

Kennedy Behrman is a veteran consultant specializing in architecting and implementing cloud solutions for early-stage startups. He is experienced in data engineering, data science, AWS solutions, and engineering management, and has acted as a technical editor on a number of Python and data science-related publications. He has experience developing a training curriculum used in international economic development and more than a decade of hands-on Python experience. Kennedy has recently acted as both a content specialist for AWS Machine Learning certification development and as a technical editor for the book Pragmatic AI: An introduction to Cloud-Based Machine Learning (Pearson, 2018). He is also a founder of Pragmatic AI Labs.

What You Will Learn

  • Performing Data Engineering tasks on AWS
  • Developing with Cloud9
  • Writing AWS Lambda functions in Python
  • Implementing cloud-native Data Engineering patterns, i.e. serverless
  • Architecting event-driven architectures on the AWS platform using SQS, Python Lambda, and other AWS technologies

Download the project files: GitHub Repo

Who Should Take This Course

  • You are an aspiring data engineer using Python
  • You work with data and want to learn cloud-native data engineering patterns
  • You are new to the AWS Cloud and want to write functions in Python that do not require servers
  • You are a data scientist who needs a simpler way to get data engineering results
  • You want to learn about serverless technology and how to accomplish it in Python

Course Requirements

  • Can write functions in Python and execute statements
  • Have a basic understanding of AWS

About Pearson Video Training

Pearson publishes expert-led video tutorials covering a wide selection of technology topics designed to teach you the skills you need to succeed. These professional and personal technology videos feature world-leading author instructors published by your trusted technology brands: Addison-Wesley, Cisco Press, Pearson IT Certification, Prentice Hall, Sams, and Que. Topics include IT Certification, Network Security, Cisco Technology, Programming, Web Development, Mobile Development, and more. Learn more about Pearson Video training at http://www.informit.com/video.

Table of contents

  1. Introduction
    1. Data Engineering with Python and AWS Lambda LiveLessons: Introduction
  2. Lesson 1: Get Started with AWS Lambda
    1. 1.1 Create a Hello World AWS Lambda function in the console
    2. 1.2 Learn basic Lambda patterns
    3. 1.3 Learn Lambda Management console
    4. 1.4 Upload external code to AWS Lambda
  3. Lesson 2: Use Cloud9 to Develop Python Lambda Functions
    1. 2.1 Set up Cloud9
    2. 2.2 Develop with Cloud9
    3. 2.3 Launch Cloud9 and workspace configuration
    4. 2.4 Import Lambda functions
    5. 2.5 Invoke Lambda functions
    6. 2.6 Invoke Lambda functions inside API Gateway
    7. 2.7 Deploy a Lambda function
  4. Lesson 3: Create Timed Lambda Functions
    1. 3.1 Use AWS Lambda with Cloudwatch events
    2. 3.2 Use AWS Lambda to populate AWS SQS
    3. 3.3 Use AWS Cloudwatch logging with AWS Lambda
  5. Lesson 4: Create Event-Driven Lambdas
    1. 4.1 Create a Producer Lambda function
    2. 4.2 Enable SQS Trigger
    3. 4.3 Serverless data engineering architecture
  6. Lesson 5: Learn SAM Local
    1. 5.1 Install SAM Local
    2. 5.2 Use SAM Local to invoke functions locally
    3. 5.3 Use SAM to package and deploy Lambda
    4. 5.4 Use SAM with IAM
    5. 5.5 Use SAM Lambda environment variables
  7. Lesson 6: Learn AWS Glue
    1. 6.1 What is AWS Glue?
    2. 6.2 Use AWS Glue
  8. Lesson 7: Create State Machines with Step Functions
    1. 7.1 Learn step functions
    2. 7.2 Use Amazon States Language
    3. 7.3 Step functions demo
  9. Lesson 8: Use Step Functions with AWS Services
    1. 8.1 Learn integration with other AWS products
    2. 8.2 Use DynamoDB with step functions
    3. 8.3 Use AWS ECS/Fargate with step functions
    4. 8.4 Use AWS Callback Pattern
  10. Lesson 9: Serverless Relational Databases
    1. 9.1 Serverless relational databases
    2. 9.2 Use Aurora Serverless
    3. 9.3 Use Data API for Aurora Serverless
    4. 9.4 Use stored procedures to invoke Lambda
  11. Lesson 10: Build APIs with API Gateway
    1. 10.1 Use API Gateway
    2. 10.2 Integrate Lambda and API Gateway best practices
  12. Lesson 11: Authenticate APIs with AWS Cognito
    1. 11.1 Begin Cognito authentication
    2. 11.2 Use Cognito User Pools
    3. 11.3 Use Cognito authentication with API Gateway
    4. 11.4 Use Federated Identity
  13. Lesson 12: Use Serverless Datastores
    1. 12.1 Use DynamoDB for data engineering
    2. 12.2 Use Amazon Athena for data engineering
    3. 12.3 Use Amazon EMR for data engineering
    4. 12.4 Use Amazon EFS for data engineering
  14. Lesson 13: Create Serverless Business Intelligence and AutoML
    1. 13.1 Integrate Amazon Quicksite
    2. 13.2 Integrate Lambda with AI APIs
    3. 13.3 Integrate Lambda with Sagemaker
  15. Lesson 14: Create Serverless Data Streaming
    1. 14.1 Use Kinesis Streams
    2. 14.2 Use Computer Vision Streams
  16. Lesson 15: Case Studies
    1. 15.1 Compare AWS Lambda with Google Cloud Functions
    2. 15.2 Use GCP Cloud Functions with Pub Sub + Cloud Scheduler
    3. 15.3 Use Chalice framework
    4. 15.4 Push versus Pull Architecture
    5. 15.5 Principles of DevOps
    6. 15.6 Principles of cloud computing
    7. 15.7 Summary of serverless computing
    8. 15.8 Managing Packages in AWS Lambda
    9. 15.9 Multi-cloud solutions
  17. Lesson 16: Course Summary
    1. 16.1 Course summary

Product information

  • Title: Data Engineering with Python and AWS Lambda LiveLessons
  • Author(s): Noah Gift, Robert Jordan, Kennedy Behrman
  • Release date: August 2019
  • Publisher(s): Pearson
  • ISBN: 0135964334