August 2025
Intermediate to advanced
476 pages
12h 52m
English
Content preview from AWS Certified Data Engineer Associate Study Guide
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Chapter 4. Data Ingestion and Transformation
The ability to efficiently collect, ingest, and transform data from diverse sources is crucial for driving valuable insights and well-informed decisions. Organizations constantly strive to unlock insights from their data by analyzing their data assets and making better business decisions through data analytics.
For data engineers, optimizing this end-to-end data ingestion and transformation process is a core responsibility. They are often tasked with building reliable pipelines that power an organization’s data-driven initiatives. However, selecting the appropriate data ingestion and transformation solutions is critical, as different varieties of data sources with varying volumes, velocities, and transformation needs may necessitate the use of various AWS services and approaches.
In this chapter, you will learn how to do the following:
Design and implement efficient data ingestion pipelines using appropriate AWS services
Choose the right AWS services for real-time/near-real-time and batch data processing
Build reliable data pipelines for both batch and real-time scenarios
Implement efficient data transformation strategies
Apply best practices for data ingestion and transformation
Orchestrate complex data workflows
By the end of this chapter, you will have a comprehensive understanding of how to build data pipelines with data ingestion and transformation capabilities in AWS, and how to orchestrate them. You will learn how to carefully ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.