97 Things Every Data Engineer Should Know

Chapter 52. Moving from Software Engineering to Data Engineering

John Salinas

The move from software engineering to data engineering is rewarding and exciting. Data engineering is everything software engineering is: passion and principles for solving technical challenges efficiently and elegantly. But you expand your craft to include analytical and data-related problems. This involves solving problems at an even larger scale, and also helping people find insights from the problems you are solving.

When I first started in data engineering, I felt lost because there was so much to learn. But like you, I love learning. I was thrilled by the challenge to learn new technologies and new patterns. It’s been comforting to realize that all the same principles still apply—things like using abstraction, keeping things simple, and building applications so they are scalable and easy to maintain. Also, all my experience is still relevant, and even more important, so are skills like troubleshooting, scaling enterprise applications, API development, networking, programming, and scripting.

With data engineering, you solve problems similar to those in software engineering, but at a larger scale. I’ve worked on code that processes billions of events. I wore it as a badge of honor that traditional technologies and platforms would break with the amount of information we were trying to process. Early ...