Streaming Architecture

Chapter 8. Putting It All Together

Where do you go from here?

Try reexamining your goals for different projects and see what advantages you might gain from transitioning to a universal stream-based approach in addition to the specific benefits for your real-time analytics applications.

The fact is, there’s a revolution in what you can do with streaming data for a wide variety of use cases, from IoT sensor data to financial services, telecommunications, web-based business, retail, healthcare, and more. New technologies that efficiently handle continuous event data with speed at scale are part of why this revolution is possible. Another key ingredient is a new way to design architecture that exploits these emerging technologies. The big change is to see the power in a universal stream-based design. This does not mean that streaming data is used for everything, but it does mean that streaming becomes a common approach rather than something considered only for specialized, real-time projects.

Note

There are great benefits to be gained when stream-based designs for big data architectures become a habit.

At the heart of effective stream-based architecture is the message passing itself. A big difference between stream-based and traditional design (or even people’s preconception of streaming) is that the messaging layer plays a much more prominent role. It can and should be used for more than just a step to precede real-time analytics, although it is essential for processing streaming ...