Data Engineering Design Patterns

Chapter 2. Data ingestion design patterns

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

Data engineering systems are rarely data generators. More often their first stage is data acquisition from various data producers. Working with these producers is not easy; they can be different pipelines inside your team, different teams within your company, or even completely different organizations. Because each producer has dedicated constraints inherited from the technical and business environments, the interaction with them might be challenging for you.

But you have no choice. You have to adapt. Otherwise, you won’t get any data and as a result, you won’t feed your data analytics or data science workloads. Or even worse, you will get some data, share it with your downstream consumers, but a few days later, you’ll get some complaints in return. The complaint may be about an incomplete dataset, an inefficient data organization, or a completely broken data requiring internal restore process and backfilling.

As you can already see, bringing data to ...