Mastering Machine Learning with Spark 2.x

Data caching

Many machine learning algorithms are iterative in nature and thus require multiple passes over the data. However, all data stored in Spark RDD are by default transient, since RDD just stores the transformation to be executed and not the actual data. That means each action would recompute data again and again by executing the transformation stored in RDD.

Hence, Spark provides a way to persist the data in case we need to iterate over it. Spark also publishes several StorageLevels to allow storing data with various options:

• NONE: No caching at all
• MEMORY_ONLY: Caches RDD data only in memory
• DISK_ONLY: Write cached RDD data to a disk and releases from memory
• MEMORY_AND_DISK: Caches RDD in memory, if it's not possible to offload ...