Apache Spark Quick Start Guide

Storage 

The way you store your data decides how fast you are going to retrieve it. It is always better to choose some partition or bucketing strategies that can help you process your data faster. Spark provides both partition and bucketing while storing the DataFrames. In the case of partitioning, the files are created under directories based on a key field. For example, you can choose a data field to partition your data. One thing to keep in mind when working with partitioning is that the key field should have low cardinality; that is, the field should have lesser possible values. If the cardinality were too high, you would end up creating a large number of partitions that can become a bottleneck, as so many tasks are launched to process ...