97 Things Every Data Engineer Should Know

Chapter 8. Beware of Silver-Bullet Syndrome

Thomas Nield

Technology job postings often describe the need for passionate people who work hard and strongly advocate their ideas. After all, Steve Jobs was passionate, so maybe that is a trait of successful people!

But in the data engineering and analytics world, passionate people often have a strong opinion about using a certain platform. We all have encountered this person: the one who zealously promotes Apache Spark or pushes to have all data wrangling work done in Alteryx Designer. There is a strong emphasis on the tool and not so much on the problem/solution pairing.

Sometimes this behavior is driven by a desire for standardization and to simplify hiring, which is certainly valid. But more often than not, I have seen people zealously advocate a tool simply because they were passionate about it or, even worse, built their professional identity around it.

To put it simply, it is never a good idea to build your professional identity around a tool. Tools and applications come and go, and what is hot today may be deprecated tomorrow. This alone should give you reason to pause before advocating a technology too quickly. Ask anyone who has done JavaScript.

Another reason that over-advocating a tool is a bad idea is best described by the expression “When all you have is a hammer, everything starts to look like a nail.” More than once ...