O'Reilly logo

Practical Predictive Analytics by Ralph Winters

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Saving your work

Now that we have produced our final Spark data frame, we can write it to disk. Then, from the next chapter onwards, we will read it back into the workspace rather than have to recreate it from scratch. If you are proceeding directly to the next chapter, you can skip this step for now:

  • We will save in Parquet file format, which is a very efficient format for Spark and SQL. The %fs (file system) directive allows you to issue a directory (or file listing) command using the ls operating system command.
  • Once the file is saved, you can validate the integrity of the file by reading it back in and assigning it to the out_sd dataframe (again).
  • Use the head command to verify that the data was read back in:
 saveAsParquetFile(out_sd, ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required