Fundamentals of Data Engineering

Chapter 7. Ingestion

You’ve learned about the various source systems you’ll likely encounter as a data engineer and about ways to store data. Let’s now turn our attention to the patterns and choices that apply to ingesting data from various source systems. In this chapter, we discuss data ingestion (see Figure 7-1), the key engineering considerations for the ingestion phase, the major patterns for batch and streaming ingestion, technologies you’ll encounter, whom you’ll work with as you develop your data ingestion pipeline, and how the undercurrents feature in the ingestion phase.

Figure 7-1. To begin processing data, we must ingest it

What Is Data Ingestion?

Data ingestion is the process of moving data from one place to another. Data ingestion implies data movement from source systems into storage in the data engineering lifecycle, with ingestion as an intermediate step (Figure 7-2).

Figure 7-2. Data from system 1 is ingested into system 2

It’s worth quickly contrasting data ingestion with data integration. Whereas data ingestion is data movement from point A to B, data integration combines data from disparate sources into a new dataset. For example, you can use data integration to combine data from a CRM system, advertising analytics data, and web analytics to create a user ...