97 Things Every Data Engineer Should Know

Chapter 44. Listen to Your Users—but Not Too Much

Amanda Tomlinson

We’ve all seen the cool infographics, but no one needs to tell a data engineer about the immense and ever-increasing amount of data being generated every day. We’re all living it.

We’re extracting it, transforming it, landing it (now considering whether we should maybe have started landing before transforming), cleaning it (what, you’re saying we should stop cleaning it?), deciding where and for how long to store it, standing up new infrastructure to handle the sheer volume of it, filtering it, joining it, building KPIs and models from it, creating workflows for it, exposing it, cataloging it, monitoring it (easy enough when we started, but increasingly difficult a few years in). With so much to do and so much demand from our stakeholders, it’s not at all surprising that so many data teams, especially those serving internal customers, get so caught up in the technical aspects of data engineering that they forget to consider who their users are and what they actually need. Data is data, right?

Before too long, this will lead to frustration, wasted effort, and a lack of confidence in the data team. Yep, all of the stuff I’ve listed is hugely important, but it’s just as important to realize that not all of your users are the same, and to take a step back and consider what they need from the data you produce and what ...