Now let's work out a few interesting examples, starting out with a simple one and then moving on to progressively complex operations.
The code files are in
fdps-v3/code, and the data files are in
fdps-v3/data. You can run the code either from a Scala IDE or just from the Spark Shell.
Start Spark Shell from the bin directory where you have installed the spark:
Inside the shell, the following command will load the source:
As we saw earlier,
SparkSession.read.* gives us a rich set of features to read different types of data with flexible control over the options.
Dataset.write.* does the same for writing ...