The Evolving Role of the Data Engineer

Every organization can benefit from data, used adeptly in coordination with the organization’s goals. Today’s process of accumulating enormous amounts of information from different sources—big data in common parlance—is like a powerful telescope that lets you see further into the universe and learn about trends you couldn’t track before.

The term data engineer is relatively new, and the role appears only sporadically in technical literature. It is the data scientists who are exalted as today’s heroes: the high-flying fighter pilots of big data who create machine learning models for predictions and other analytics. But no airplane could fly without a small army of mechanics and other trained staff to prepare the planes. In this analogy, the data engineer is like the airplane mechanic, preparing the data that enables the data scientist to carry out organizational goals. When you add up all the tasks that fit under this role—getting the data, cleaning it, creating enhanced versions—observers often claim that data engineering comprises 80 to 90% of the work organizations do with data.

Why did data engineering become so important? During the 1990s, the world underwent a momentous transformation that reached more and more deeply into our lives and brought cascading impacts upon just about every career. The transformation was driven by a hodgepodge of trends:

• New roles for data throughout society, such as the growth of the web and social media ...