Skip to Main Content
Data Engineering with AWS
book

Data Engineering with AWS

by Gareth Eagar
December 2021
Beginner to intermediate content levelBeginner to intermediate
482 pages
11h 27m
English
Packt Publishing
Content preview from Data Engineering with AWS

Chapter 10: Orchestrating the Data Pipeline

Throughout this book, we have been discussing various services that can be used by data engineers to ingest and transform data, as well as make it available for consumers. We looked at how we could ingest data via Amazon Kinesis Data Firehose and Amazon Database Migration Service, and how we could run AWS Lambda and AWS Glue functions to transform our data. We also discussed the importance of updating a data catalog as new datasets are added to a data lake, and how we can load subsets of data into a data mart for specific use cases.

For the hands-on exercises, we made use of various services, but for the most part, we triggered these services manually. However, in a real production environment, it ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Data Engineering with AWS - Second Edition

Data Engineering with AWS - Second Edition

Gareth Eagar
Data Science on AWS

Data Science on AWS

Chris Fregly, Antje Barth
Data Engineering with Python and AWS Lambda LiveLessons

Data Engineering with Python and AWS Lambda LiveLessons

Noah Gift, Robert Jordan, Kennedy Behrman

Publisher Resources

ISBN: 9781800560413Supplemental Content