Mastering Machine Learning with Spark 2.x

Stream creation

There are several ways to create a stream, described in the Spark documentation (https://spark.apache.org/docs/2.1.1/structured-streaming-programming-guide.html), including socket-based, Kafka, or file-based streams. In this chapter, we will use file-based streams, streams that are pointed to a directory and deliver all the new files that appear in the directory.

Moreover, our application will read CSV files; thus, we will connect the stream input with the Spark CSV parser. We also need to configure the parser with the input data schema, which we exported from the mode-training application. Let's load the schema first:

def loadSchema(srcFile: File): StructType = {import org.apache.spark.sql.types.DataTypeStructType( DataType.fromJson(scala.io.Source.fromFile(srcFile).mkString).asInstanceOf[StructType].map ...