Apache Spark 2.x Cookbook

How to do it...

Hadoop MapReduce's word count, which takes at least three class files and one configuration file, namely project object model (POM), becomes very simple with the Spark shell. In this recipe, we are going to create a simple one-line text file, upload it to the Hadoop distributed file system (HDFS), and use Spark to count the occurrences of words. Let's see how:

1. Create the words directory using the following command:

        $ mkdir words

2. Get into the words directory:

        $ cd words

3. Create a sh.txt text file and enter "to be or not to be" in it:

        $ echo "to be or not to be" > sh.txt

4. Start the Spark shell:

        $ spark-shell

5. Load the words directory as RDD:

        scala> val words =  sc.textFile("hdfs://localhost:9000/user/hduser/words") ...