Practical Real-time Data Processing and Analytics

When to avoid using Spark

We have said so much about Spark and its distinct features, I really don't want our readers to believe that it's a one solution fits all kind of tool and it comes with a magic wand to solve all our woes. Well, there are situations when spark should be avoided and a few common ones are listed as follows:

• No real-time and event Windowing: Spark doesn't do real-time processing, in fact it plays on micro batching that essentially gives an illusion of NRT. So what it actually does is it creates micro batches out of the streaming events and instead of executing the processing logic on one big batch, it executes it on a micro batch and thus results arrive very close to realtime. There is a big trade off to this approach, ...