97 Things Every Data Engineer Should Know

Chapter 94. When to Talk and When to Listen

Steven Finkelstein

As we approach the one-year anniversary on my current digitization project at work, I look back to see what can be learned from the experience. Project SWIFT was aimed at providing information about my employer’s global manufacturing plants to the product developers earlier in the development process. The output would be a data-collection tool and a business intelligence report in Tableau. The objective was quite simple, but we encountered constant obstacles at the micro and macro level.

At the macro level, huge shifts were occurring in our organization. Our department was undergoing a strategic shift to the cloud and becoming more data driven. Significant organizational restructures happened, including the creation of a new data analytics team that absorbed my position. For new projects such as SWIFT, my team took the lead on the technical side, as opposed to IT in prior years. The organizational changes left a gap in leadership for critical design decisions. I believe this macro-level change was the foundation for the many challenges this project faced.

At the micro level, I was at the center of all requirements discussions. The data required for the final report did not exist, so early conversations were about what data needed to be captured, and the relationships between the entities being modeled. The initial ...