97 Things Every Data Engineer Should Know

Chapter 84. Tools Don’t Matter, Patterns and Practices Do

Bas Geerdink

If you’re taking a dive into the data-engineering world, it’s easy to be overwhelmed by technical terms and frameworks. Most articles and books about big data start with extensive explanations about Apache Hadoop, Spark, and Kafka. Diving a bit further into the sister-realms of software development and data science, the list of programming languages and tools becomes a never-ending pile of strange-sounding terms, of which you tell yourself, “I should investigate this!”

I know plenty of engineers who keep track of lists of frameworks and tools that they have to dive into. However, that is not the way to go if you’re just starting. Rather than learning the tools (frameworks, products, languages, engines), you should focus on the concepts (common patterns, best practices, techniques). If after studying you come across a new gimmick, doing some quick research should be enough to give you an idea of where to place it in the landscape.

But all these abstract concepts can also feel academic and high level. The key is to find learning resources with good examples. Search for books, blogs, and articles that don’t jump straight to the source code, but explain general concepts and architectures.

When you come across an interesting piece of technology or a term that you’re not familiar with, a trick is to always ask yourself ...