Data Engineering Design Patterns

Chapter 3. Error management design patterns

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 3rd chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

Our new Constitution is now established, and has an appearance that promises permanency; but in this world nothing can be said to be certain, except death and taxes.                                                                                                 -- Benjamin Franklin

That’s what Benjamin Franklin, who is one of the Founding Fathers of the United States, wrote in a letter to French physicist, Jean-Baptiste Le Roy in 1789. If Franklin was a data engineer in our days, he would probably write something like:

In this world nothing can be said to be certain, except errors and data quality issues.

Sad but true but your data engineering life will rarely be all roses. Remember that data is dynamic and your expectations from today will not remain the same for the whole lifecycle. That’s why you will need to expect the worst and adapt accordingly.

Besides, keep in mind that you’re processing data generated by others. You directly inherit their ...