Flow Architectures

Chapter 4. Evaluating the Current Streaming Market

In previous chapters, we established what flow is, why businesses will be motivated to adopt it, and used two modeling techniques, Wardley Mapping and Promise Theory, to understand what the core technical components of flow are likely to be. In this chapter, we will evaluate the current state of those components by looking at many of the specific technologies that can fill those needs.

We are going to start by briefly reviewing some of the workloads that are driving the growth of streaming architectures. Having some context for why things have evolved the way they have is important for understanding how they work. As I describe these architectures, I will map the architectural components involved to the components we defined in our Wardley Map, demonstrating the common structures emerging in flow.

I won’t be doing an in-depth breakdown of these architectures, however. If you are interested in studying streaming architectures in more depth, I suggest Tyler Akidau, Slava Chernyak, and Reuven Lax’s Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing (O’Reilly), Fabian Hueske and Vasiliki Kalavri’s Stream Processing with Apache Flink (O’Reilly), or Gerard Maas and Francois Garillot’s Stream Processing with Apache Spark (O’Reilly). Each demonstrates a different approach to stream processing using an appropriate open source platform for the purpose.

We will follow our brief survey with an overview of four ...