O'Reilly logo

Machine Learning with Spark - Second Edition by Nick Pentreath, Manpreet Singh Ghotra, Rajdeep Dua

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Creating a basic streaming application

Next, we will create our first streaming program. We will simply connect to the producer and print out the contents of each batch. Our streaming code looks like this:

/**   * A simple Spark Streaming app in Scala **/ object SimpleStreamingApp {   def main(args: Array[String]) {     val ssc = new StreamingContext("local[2]", "First Streaming       App", Seconds(10))     val stream = ssc.socketTextStream("localhost", 9999)     // here we simply print out the first few elements of each batch     stream.print()     ssc.start()     ssc.awaitTermination()   } }

It looks fairly simple, and it is mostly due to the fact that Spark Streaming takes care of all the complexity for us. First, we initialized a StreamingContext (which is the ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required