97 Things Every Data Engineer Should Know

Chapter 87. What Is a Data Engineer? Clue: We’re Data Science Enablers

Lewis Gavin

The job title data engineer doesn’t always come with the same sexy connotations as something like data scientist. Understandably, topics like machine learning and AI are always going to win the popularity contest. However, a good chunk of the work that sits behind these concepts stems from data engineering. I’ve seen a sea of articles recently talking about this exact point: that 80% of a data scientist’s work is data preparation and cleansing.

AI and Machine Learning Models Require Data

Talk to any data scientist, and they’ll tell you that obtaining data, especially from a source that has absolutely everything they require for their model, is a distant dream. This is where data engineers thrive.

The benefit of a data engineer is just that: the engineering. A data engineer can not only provide you with data from disparate sources, but also do it in a way that’s repeatable, current, and even in real time.

Clean Data == Better Model

A 2016 survey states that 80% of data science work is data prep and that 75% of data scientists find this to be the most boring aspect of the job.

Guess what—this is where the data engineer also thrives. They’re great at joining, cleaning, manipulating, and aggregating data. All of this will be done in a repeatable way, providing a consistent source of fresh, clean ...