Frank Kane's Taming Big Data with Apache Spark and Python

What is Spark Streaming?

Spark Streaming is useful for very specific tasks. If you have a continual stream of data that needs constant analysis, then Spark Streaming is for you. A common example of using it is if you have a fleet of web servers running and you need to process log data coming into it continuously. Who knows what you want to do, maybe you want to keep track of the most frequent occurrences, or keep an eye out for a certain kind of error; Spark Streaming can do that. The way it works is that, at some given interval that you define, it will aggregate the data that's coming in and analyze it however you say. So you can reduce things or map things for example, at some given time frame after a certain batch of data has been taken ...