To write a DataFrame or RDD to disk, we can use the write method. We have a selection of formats we can use; in this case, we will save it as a CSV file on the local machine:
In: (df.na.drop().write .save("file:////home//jovyan//complete_users.csv", format='csv'))
Checking the output on the local filesystem, we immediately see that something is different from what we expected: this operation creates multiple files (part-r-…). Each of them contains some rows serialized as JSON objects, and merging them together will create the comprehensive output. As Spark is made to process large and distributed files, the write operation is tuned for that, and each node writes part of the full RDD:
In: ...
