Apache Spark 2.x Cookbook

How to do it...

Let's do the word count, which counts the number of occurrences of each word. In this recipe, we will load data from HDFS:

1. Create the words directory by using the following command:

$ mkdir words

2. Create the sh.txt  text file and enter "to be or not to be" in it:

$ echo "to be or not to be" > words/sh.txt

3. Upload words: 

$ hdfs dfs -put words

4. Start the Spark shell:

$ spark-shell

5. Load the words directory as a dataset:

scala> val words = spark.read.textFile("hdfs://localhost:9000/user/hduser/words")

The spark.read method also supports passing an additional option for the number of partitions, such as spark.read.repartition(10).textFile(...) . By default, Spark creates one partition for each InputSplit class, ...