Scala and Spark for Big Data Analytics

Driver failure recovery

Driver failure recovery can be accomplished by using StreamingContext.getOrCreate() to either initialize StreamingContext from an existing checkpoint or to create a new StreamingContext.

The two conditions for a streaming application when started are as follows:

• When the program is being started for the first time, it needs to create a new StreamingContext, set up all the streams, and then call start()
• When the program is being restarted after failure, it needs to initialize a StreamingContext from the checkpoint data in the checkpoint directory and then call start()

We will implement a function createStreamContext(), which creates the StreamingContext and sets up the various DStreams to parse the tweets and generate ...