Designing Big Data Platforms

6Stream Big Data Processing

After reading this chapter, you should be able to:

• Explain the need for stream processing
• Understand stream processing via message brokers
• Understand stream processing via stream processors

The first stop for Big Data processing was offline processing. In this chapter, we will examine stream data processing technologies in depth.

6.1 The Need for Stream Processing

As the number of devices connected to the network increases, so does the amount of streaming data. The streaming data comes from phones, tablets, and small computational devices. What is more, many systems produce data continuously. Some of the examples are the internet of things, social media, online transactions, and more. While data is coming in streams, the fast feedback loop is becoming increasingly crucial for applications and businesses.

Stream Big Data processing is a vital part of a modern Big Data platform. Stream processing can help a modern Big Data platform from many different perspectives. Streaming data gives the platform to cleanse, filter, categorize, and analyze the data while it is in motion. Thus, we don't have to store irrelevant and fruitless data to disk. With stream Big Data processing, we get a chance to respond to user interactions or events swiftly rather than waiting for more significant periods. Having fast loops of discovery and acting can introduce a competitive advantage to the businesses. Streaming solutions bring additional agility with added risk. ...