97 Things Every Data Engineer Should Know

Chapter 78. The Three Rs of Data Engineering

Tobias Macey

In school we are told that the three Rs we need to know are reading, writing, and arithmetic. (I never did understand that, since only one of those words starts with R.) But once you grow up and get a job as a data engineer, you’ll learn that there really are three Rs that you need to know, and in this case, they all actually start with that letter!

Reliability

At the end of the day, the most important characteristic of your data is that it needs to be reliable. All the fancy machine learning and sophisticated algorithms in the world won’t do you any good if you are feeding them messy and inconsistent data. Rather than the healthy, beautiful, and valuable systems that you know they could be, they will grow up to be hideous monsters that will consume your time and ambition.

Reliability can mean a lot of things, but in this context we are talking about characteristics of your data that contribute to a high degree of confidence that the analyses you are performing can be assumed correct. Some of the elements of a data platform that is reliable are as follows:

• Consistent and validated schemas. All records follow the same pattern and can be treated the same way by your analytical processes.

• High availability of your systems to ensure read and write operations succeed. This requires systems that provide redundancy (there’s ...