O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Data Engineering with Python and AWS Lambda LiveLessons

Video Description

7 Hours of Video Instruction

Data Engineering with Python and AWS Lambda LiveLessons shows users how to build complete and powerful data engineering pipelines in the same language that Data Scientists use to build Machine Learning models. By embracing serverless data engineering in Python, you can build highly scalable distributed systems on the back of the AWS backplane. Users learn to think in the new paradigm of serverless, which means to embrace events and event-driven programs that replace expensive and complicated servers.


Some of the many benefits of programming with AWS Lambda in Python include no servers to manage, continuous scaling, and subsecond metering. Several use cases include data processing, stream processing, IoT backends, mobile, and web applications. Learn to take advantage of a new paradigm in software architecture that will make your code easier to write, maintain, and deploy.

AWS Lambda functions are the building blocks for creating sophisticated applications and services on AWS. In this LiveLesson, you learn to use Python to develop Lambda functions that communicate with key AWS services: API Gateway, SQS, and CloudWatch functions. You also learn how a new cloud-based development environment, Cloud9, can streamline writing, debugging, and deploying AWS Lambda functions.

About the Instructors

Noah Gift is a lecturer and consultant at both the UC Davis Graduate School of Management MSBA program and the Graduate Data Science program, MSDS, at Northwestern. He is teaching and designing graduate Machine Learning, AI, and Data Science courses, and consulting on Machine Learning and Cloud Architecture for students and faculty, including leading a multi-cloud certification initiative for students. Noah is a Python Software Foundation Fellow, AWS Subject Matter Expert (SME) on Machine Learning, AWS Certified Solutions Architect and AWS Academy Accredited Instructor, Google Certified Professional Cloud Architect, and Microsoft MTA on Python. Noah has published close to 100 technical publications, including two books on subjects ranging from Cloud Machine Learning to DevOps. Gift received an MBA from UC Davis, an M.S. in Computer Information Systems from Cal State Los Angeles, and a B.S. in Nutritional Science from Cal Poly San Luis Obispo. Currently, he is consulting startups and other companies on Machine Learning, Cloud Architecture, and CTO level consulting as the founder of Pragmatic AI Labs. His most recent book is Pragmatic AI: An Introduction to Cloud-Based Machine Learning (Pearson, 2018).

Robert Jordan is a visionary architect with more than 20 years of experience designing, implementing, and deploying production applications for some of the world’s largest media and scientific customers. He has successfully led projects on all major cloud platforms and is currently certified on both AWS and GCP platforms.

Kennedy Behrman is a veteran consultant specializing in architecting and implementing cloud solutions for early-stage startups. He is experienced in data engineering, data science, AWS solutions, and engineering management, and has acted as a technical editor on a number of Python and data science-related publications. He has experience developing a training curriculum used in international economic development and more than a decade of hands-on Python experience. Kennedy has recently acted as both a content specialist for AWS Machine Learning certification development and as a technical editor for the book Pragmatic AI: An introduction to Cloud-Based Machine Learning (Pearson, 2018). He is also a founder of Pragmatic AI Labs.

What You Will Learn

  • Performing Data Engineering tasks on AWS
  • Developing with Cloud9
  • Writing AWS Lambda functions in Python
  • Implementing cloud-native Data Engineering patterns, i.e. serverless
  • Architecting event-driven architectures on the AWS platform using SQS, Python Lambda, and other AWS technologies

Who Should Take This Course

  • You are an aspiring data engineer using Python
  • You work with data and want to learn cloud-native data engineering patterns
  • You are new to the AWS Cloud and want to write functions in Python that do not require servers
  • You are a data scientist who needs a simpler way to get data engineering results
  • You want to learn about serverless technology and how to accomplish it in Python

Course Requirements

  • Can write functions in Python and execute statements
  • Have a basic understanding of AWS

About Pearson Video Training

Pearson publishes expert-led video tutorials covering a wide selection of technology topics designed to teach you the skills you need to succeed. These professional and personal technology videos feature world-leading author instructors published by your trusted technology brands: Addison-Wesley, Cisco Press, Pearson IT Certification, Prentice Hall, Sams, and Que. Topics include IT Certification, Network Security, Cisco Technology, Programming, Web Development, Mobile Development, and more. Learn more about Pearson Video training at http://www.informit.com/video.

Table of Contents

  1. Lesson 1: Get Started with AWS Lambda
    1. Learning objectives 00:00:28
    2. 1.1 Create a Hello World AWS Lambda function in the console 00:06:32
    3. 1.2 Learn basic Lambda patterns 00:06:03
    4. 1.3 Learn Lambda Management console 00:08:13
    5. 1.4 Upload external code to AWS Lambda 00:07:03
  2. Lesson 2: Use Cloud9 to Develop Python Lambda Functions
    1. Learning objectives 00:00:30
    2. 2.1 Set up Cloud9 00:05:11
    3. 2.2 Develop with Cloud9 00:04:07
    4. 2.3 Launch Cloud9 and workspace configuration 00:03:55
    5. 2.4 Import Lambda functions 00:03:32
    6. 2.5 Invoke Lambda functions 00:04:18
    7. 2.6 Invoke Lambda functions inside API Gateway 00:06:47
    8. 2.7 Deploy a Lambda function 00:03:40
  3. Lesson 3: Create Timed Lambda Functions
    1. Learning objectives 00:00:26
    2. 3.1 Use AWS Lambda with Cloudwatch events 00:05:00
    3. 3.2 Use AWS Lambda to populate AWS SQS 00:15:56
    4. 3.3 Use AWS Cloudwatch logging with AWS Lambda 00:04:22
  4. Lesson 4: Create Event-Driven Lambdas
    1. Learning objectives 00:00:25
    2. 4.1 Create a Producer Lambda function 00:09:10
    3. 4.2 Enable SQS Trigger 00:04:22
    4. 4.3 Serverless data engineering architecture 00:04:29
  5. Lesson 5: Learn SAM Local
    1. Learning objectives 00:00:32
    2. 5.1 Install SAM Local 00:04:37
    3. 5.2 Use SAM Local to invoke functions locally 00:04:59
    4. 5.3 Use SAM to package and deploy Lambda 00:02:59
    5. 5.4 Use SAM with IAM 00:06:28
    6. 5.5 Use SAM Lambda environment variables 00:06:07
  6. Lesson 6: Learn AWS Glue
    1. Learning objectives 00:00:17
    2. 6.1 What is AWS Glue? 00:04:10
    3. 6.2 Use AWS Glue 00:03:46
  7. Lesson 7: Create State Machines with Step Functions
    1. Learning objectives 00:00:35
    2. 7.1 Learn step functions 00:09:43
    3. 7.2 Use Amazon States Language 00:08:26
    4. 7.3 Step functions demo, part I 00:18:52
  8. Lesson 8: Use Step Functions with AWS Services
    1. Learning objectives 00:00:41
    2. 8.1 Learn integration with other AWS products 00:04:23
    3. 8.2 Use DynamoDB with step functions 00:05:07
    4. 8.3 Use AWS ECS/Fargate with step functions 00:08:05
    5. 8.4 Use AWS Callback Pattern 00:04:39
  9. Lesson 9: Serverless Relational Databases
    1. Learning objectives 00:00:47
    2. 9.1 Serverless relational databases 00:06:06
    3. 9.2 Use Aurora Serverless 00:04:15
    4. 9.3 Use Data API for Aurora Serverless 00:04:50
    5. 9.4 Use stored procedures to invoke Lambda 00:12:21
  10. Lesson 10: Build APIs with API Gateway
    1. Learning objectives 00:00:29
    2. 10.1 Use API Gateway 00:17:21
    3. 10.2 Integrate Lambda and API Gateway best practices 00:23:44
  11. Lesson 11: Authenticate APIs with AWS Cognito
    1. Learning objectives 00:00:45
    2. 11.1 Begin Cognito authentication 00:00:37
    3. 11.2 Use Cognito User Pools 00:10:48
    4. 11.3 Use Cognito authentication with API Gateway 00:06:22
    5. 11.4 Use Federated Identity 00:05:21
  12. Lesson 12: Use Serverless Datastores
    1. Learning objectives 00:00:19
    2. 12.1 Use DynamoDB for data engineering 00:05:48
    3. 12.2 Use Amazon Athena for data engineering 00:04:20
    4. 12.3 Use Amazon EMR for data engineering 00:03:43
    5. 12.4 Use Amazon EFS for data engineering 00:03:20
  13. Lesson 13: Create Serverless Business Intelligence and AutoML
    1. Learning objectives 00:00:32
    2. 13.1 Integrate Amazon Quicksite 00:04:06
    3. 13.2 Integrate Lambda with AI APIs 00:02:56
    4. 13.3 Integrate Lambda with Sagemaker 00:05:45
  14. Lesson 14: Create Serverless Data Streaming
    1. Learning objectives 00:00:21
    2. 14.1 Use Kinesis Streams 00:08:33
    3. 14.2 Use Computer Vision Streams 00:02:56
  15. Lesson 15: Case Studies
    1. Learning objectives 00:00:33
    2. 15.1 Compare AWS Lambda with Google Cloud Functions 00:02:37
    3. 15.2 Use GCP Cloud Functions with Pub Sub + Cloud Scheduler 00:09:30
    4. 15.3 Use Chalice framework 00:04:25
    5. 15.4 Push versus Pull Architecture 00:04:59
    6. 15.5 Principles of DevOps 00:05:51
    7. 15.6 Principles of cloud computing 00:08:03