Chapter 1. Data Engineers vs. Data Scientists

It’s important to understand the differences between a data engineer and a data scientist. Misunderstanding or not knowing these differences are making teams fail or underperform with big data.

A key misunderstanding is the strengths and weaknesses of each position. I think some of these misconceptions come from the diagrams that are used to describe data scientists and data engineers.

venn diagram data engineer vs data scientist
Figure 1-1. Overly simplistic venn diagram with data scientists and data engineers. Illustration by Jesse Anderson.

Venn diagrams like Figure 1 oversimplify the complex positions and how they’re different. It makes the two positions seem interchangeable. Yes, both positions work on big data. However, what each position does to create value or data pipelines with big data is very different. This difference comes from the base skills of each position.

What Are Data Scientists and Data Engineers?

When I work with organizations on their team structures, I don’t use a Venn diagram to illustrate the relationship between a data engineer and a data scientist. I draw the diagram as shown in Figure 2.

core competencies of data scientists and data engineers and their overlapping skills
Figure 1-2. Diagram showing the core competencies of data scientists and data engineers and their overlapping skills. Illustration by Jesse Anderson and the Big Data ...

Get Data Engineers vs. Data Scientists now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.