Apache Spark 2.x Cookbook

How to do it...

Let's start with the example of Shakespeare's "to be or not to be":

1. Create the words directory by using the following command:

$ mkdir words

2. Get into the words directory:

$ cd words

3. Create the sh.txt text file and enter "to be or not to be" in it:

$ echo "to be or not to be" > sh.txt

4. Start the Spark shell:

$ spark-shell

5. Load the words directory as a DataFrame:

scala> val wdf = spark.read.text("file:///home/hduser/words")

6. Convert wdf  dataset into a dataset of strings wds :

scala> val wds = wdf.as[String]

It is very important to convert a DataFrame into a dataset before doing operations, which were done by RDDs in the past. DataFrame, essentially, is a collection of rows, and converting it into a more ...