Productive and Efficient Data Science with Python: With Modularizing, Memory profiles, and Parallel/GPU Processing

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

T. SarkarProductive and Efficient Data Science with Pythonhttps://doi.org/10.1007/978-1-4842-8121-5_9

9. Scalable Data Science

Tirthajyoti Sarkar¹  

(1)

Fremont, CA, USA

 

Data science tasks may encounter a wide variety of dataset sizes, ranging from kilobytes to petabytes. Some business spreadsheets will only have a few hundred rows while a whole factory may send a deluge of sensor data to a single dataset, resulting in billions of rows per day or even per hour. Some datasets can have many rows and a small number of columns, while others may consist of a few rows but millions of columns as feature dimensions. Even within the same organization or a data science ...