Productive and Efficient Data Science with Python: With Modularizing, Memory profiles, and Parallel/GPU Processing

© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022

T. SarkarProductive and Efficient Data Science with Pythonhttps://doi.org/10.1007/978-1-4842-8121-5_8

8. Memory and Timing Profile

Tirthajyoti Sarkar¹  

(1)

Fremont, CA, USA

 

Data science tasks come with a wide variety of computational costs of both space and time. Data wrangling jobs may need the support of large storage, while advanced ML algorithms need high intensity computing speed. Some ML algorithms work better with the support of large local memory (RAM) and cannot perform well with data situated far from the CPU on a hard disk, while others are optimized to perform well with distributed data storage.

Furthermore, the nature of the data may change slowly ...