The Data Science Handbook, 2nd Edition

13Big Data

Historically, there has been a lot of overlap between the terms “data science” and “big data,” since data science as a distinct job function emerged to deal with Big Data technologies. Today though, the relationship is not as strong. On the one hand, Big Data technologies have matured and no longer require the level of software engineering skills they once did. On the other hand, data scientists nowadays usually do most of their work without the aid of Big Data tools. However, Big Data is still an important tool in the data scientist’s toolbox.

Big Data refers to several trends in data storage and processing, which have posed new challenges, provided new opportunities, and demanded new solutions. Especially in the early days, these Big Data problems required a level of software engineering expertise that normal statisticians and data analysts weren’t able to handle. It also raised a lot of difficult, ill‐posed questions such as how best to segment users based on raw click‐stream data. This demand is what turned “data scientist” into a new, distinct job title.

Big Data is an area where low‐level software engineering concerns become especially important for data scientists. It’s always important that they think hard about the logic of their code, but performance concerns are generally of secondary importance. In Big Data though, it’s easy to accidentally add several hours to your code’s runtime, or even have the code fail several hours in due to a memory error, if ...