Apache Spark 2.x Cookbook

How to do it...

1. Start the Spark shell:

$ spark-shell  

2. Create a DataFrame to read what's coming on port 8585:

scala> val lines = spark.readStream.format("socket").option("host","localhost").option("port",8585).load

3. Cast the lines from DataFrame to Dataset with the String datatype and then flatten it:

scala> val words = lines.as[String].flatMap(_.split(" "))

4.  Do the word count:

scala> val wordCounts = words.groupBy("value").count()

5. Start the netcat server in a separate window:

$ nc -lk 8585

6. Come back to the previous terminal and print the complete set of counts to the console every time it is updated:

scala> val query = wordCounts.writeStream.outputMode("complete").format("console").start()

7. Now go back to the terminal ...