Streaming Architecture

Chapter 2. Stream-based Architecture

In the previous chapter, we looked at some of the reasons why so many people are getting interested in using streaming data. Now we explain the how—the ways to build a streaming system to best advantage.

Emerging technologies for message passing now make it possible to use streaming almost everywhere. This innovation is the biggest idea in this chapter.

Note

Stream-based architecture provides great benefits when employed across any or all of the data activities for your enterprise.

The new designs we have in mind rely on a large-scale shift in the overall design approach you use to build systems. This transition is not just about acquiring a particular technology or the skill to use a certain fast algorithm—it is about change on a much broader and more fundamental level. It is also unusual among advances in system architecture in that it can be introduced incrementally with accelerating benefits as you convert more and more services.

A Limited View: Single Real-Time Application

The need for that level of overall change toward a streaming architecture may not be apparent to everyone right away. The initial lure to use streaming data may be a particular project or goal that requires real-time analytics. For example, suppose your organization is interested in building a dashboard for real-time updates. You might initially identify the streaming data source of interest and look for a powerful stream-processing software, such as Apache Spark Streaming. ...