97 Things Every Data Engineer Should Know

Chapter 46. Maintain Your Mechanical Sympathy

Tobias Macey

As a new technologist, you have an endless variety of lessons to learn—everything from the fundamentals of logic, through syntax and software design, and on to building and integrating large distributed systems. After years of practice, ceaseless learning, and inscrutable errors, you will internalize an understanding of more computational errata than you could possibly foresee. In this torrent of information, your perpetual bafflement will be washed away, along with your beginner’s mind.

One thing that will serve you well on every step of your journey is a healthy dose of mechanical sympathy. It is possible to write useful software, build data platforms, or produce valuable analytics without ever understanding all of the underlying infrastructure and computation. However, when something breaks or you run into confusing performance issues, that understanding will prove invaluable.

Regardless of your chosen specialization, whether it is data engineering, data science, web development, or any other of an infinite gradation, you will be navigating an endless sea of abstractions. Each layer of indirection brings you further from the hardware that is diligently churning through your demanding instructions and winging your data across the world at light speed.

By considering the physical implementations of how CPUs execute instructions, ...