Python Machine Learning By Example - Second Edition

Best practice 5 – storing large-scale data

With the ever-growing size of data, oftentimes we can't simply fit the data in our single local machine and need to store it on the cloud or distributed filesystems. As this is mainly a book on machine learning with Python, we will just touch on some basic areas that you can look into. The two main strategies of storing big data are scale-up and scale-out:

• A scale-up approach increases storage capacity if data exceeds the current system capacity, such as by adding more disks. This is useful in fast-access platforms.
• In a scale-out approach, storage capacity grows incrementally with additional nodes in a storage cluster. Apache Hadoop (https://hadoop.apache.org/) is used to store and process big ...