97 Things Every Data Engineer Should Know

Chapter 15. Cultivate Good Working Relationships with Data Consumers

Ido Shlomo

The relationship between data engineers and data consumers—whether they’re data scientists, business intelligence (BI) teams, or one of a multitude of analytics teams—is always complex. All of these functions exist to serve the overall data-driven goals of the organization and are expected to integrate seamlessly. So there is clear motivation to cooperate, but more often than not, the division of labor is far from balanced—a situation that can develop into real tension between the teams.

There is no recipe for creating a perfect symbiotic relationship, and this is doubly true given the great amount of variation in the structure of data teams across organizations. That said, the following points can help data engineers avoid major pitfalls and aim for a better direction.

Don’t Let Consumers Solve Engineering Problems

Avoid the temptation of letting data consumers solve data-engineering problems. Many types of data consumers exist, and the core competencies of each individual vary across multiple spectrums—coding skills, statistical knowledge, visualization abilities, and more. In many cases, the more technically capable data consumers will attempt to close infrastructure gaps themselves by applying ad hoc fixes. This can take the form of applying additional data transformations to a pipeline that isn’t ...