AWS Certified Data Engineer Associate Study Guide

Chapter 1. Data Ingestion and Transformation

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the fourth chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

The ability to efficiently collect, ingest, and transform data from diverse sources is crucial for driving valuable insights and well-informed decisions. Organizations must constantly expand their data horizons by ingesting streaming data, data from databases, data in the form of files, integrating SaaS applications, handling on-premises files, bringing third party datasets etc - and then applying the necessary transformations to make this data useful for downstream analytics.

For data engineers, optimizing this end-to-end data ingestion and transformation process is a core responsibility. They are often tasked with building reliable pipelines that power an organization’s data-driven initiatives. However, selecting the appropriate data ingestion and transformation solutions is critical, as different data sources, volumes, velocities, and transformation needs may necessitate the use of various AWS services and approaches.

In this chapter, you will ...