97 Things Every Data Engineer Should Know

Chapter 25. Demystify the Source and Illuminate the Data Pipeline

Meghan Kwartler

You’ve been assigned to a new project, new team, or new company. You want to dig in and make an impact to add business value. It can be tempting to start writing code immediately, but if you resist the inclination to make initial assumptions and instead give your attention to setting up a solid foundation, it will pay dividends moving forward.

First, discover where and how the data originates. When your data is initiated from users, it is useful to get their perspective on their entry experience. Each time I walk the floor of a manufacturing plant or talk to a machine operator about how they use a system, I gain valuable knowledge. Often I discover ways users are entering data that are inconsistent with the original system design, or the valid reasons why they are omitting data. If you don’t have access to the people entering data, study their training documentation and talk to the business analysts associated with that function.

Get to know the specifications when your data originates from sensors, equipment, or hardware. Dig into the manuals and documentation to clarify how the data is generated. You will then have a clear understanding when you encounter this data later in your analysis. Knowing the expected values also helps you identify possible malfunctions in the source equipment.

Now examine ...