Fast Data Architectures for Streaming Applications, 2nd Edition

Chapter 5. Real-World Systems

Fast data architectures raise the bar for the “ilities” of distributed data processing. Whereas batch jobs seldom last more than a few hours, a streaming pipeline is designed to run for weeks, months, even years. If you wait long enough, even the most obscure problem is likely to happen.

The umbrella term reactive systems embodies the qualities that real-world systems must meet. These systems must be:

Responsive

The system can always respond in a timely manner, even when it’s necessary to respond that full service isn’t available due to some failure.

Resilient

The system is resilient against failure of any one component, such as server crashes, hard drive failures, or network partitions. Replication prevents data loss and enables a service to keep going using the remaining instances. Isolation prevents cascading failures.

Elastic

You can expect the load to vary considerably over the lifetime of a service. Dynamic, automatic scalability, both up and down, allows you to handle heavy loads while avoiding underutilized resources in less busy times.

Message driven

While fast data architectures are obviously focused on data, here we mean that all services respond to directed commands and queries. Furthermore, they use messages to send commands and queries to other services as well.

Classic big data systems, focused on batch and offline interactive workloads, have had less need to meet these qualities. Fast data architectures are just like other ...