Fast Data Architectures for Streaming Applications, 2nd Edition

Chapter 3. Logs and Message Queues

“Everything is a file” is the powerful, unifying abstraction at the heart of *nix systems. It’s proved surprisingly flexible and effective as a metaphor for over 40 years. In a similar way, “everything is a log” is the powerful abstraction for streaming architectures.

Message queues provide ideal semantics for managing producers that write messages to queues and consumers that read them, thereby joining subsystems together with a level of indirection that provides decoupling. Implementations can provide durable message storage with tunable persistence characteristics and other benefits.

Let’s explore these two concepts, how they are different, their relative strengths and weaknesses, and a merger that provides the best of both worlds.

The Log Is the Core Abstraction

Logs have been used for a long time as a mechanism for services to output information about what they are doing, including implementation details and problems encountered, as well as application state transitions. Log entries may include a timestamp, a notion of “urgency” (e.g., error, warning, or informational), information about the process and/or machine, and an ad hoc text message with more details. The log entries may be written in space-separated text, JSON, or a binary format (useful for efficient transport and storage). Well-structured log entries at appropriate execution points are proxies for system events. The order of entries is significant, as it indicates event sequencing ...