Fast Data Architectures for Streaming Applications

Chapter 3. Event Logs and Message Queues

“Everything is a file” is the core, unifying abstraction at the heart of *nix systems. It’s proved surprisingly flexible and effective as a metaphor for over 40 years. In a similar way, “everything is an event log” is the powerful, core abstraction for streaming architectures.

Message queues provide ideal semantics for managing producers writing messages and consumers reading them, thereby joining subsystems together. Implementations can provide durable storage of messages with tunable persistence characteristics and other benefits.

Let’s explore these two concepts.

The Event Log Is the Core Abstraction

Logs have been used for a long time as a mechanism for services to output information about what they are doing, including problems they encounter. Log entries usually include a timestamp, a notion of “urgency” (e.g., error, warning, or informational), information about the process and/or machine, and an ad hoc text message with more details. Well-structured log messages at appropriate execution points are proxies for significant events.

The metaphor of a log generalizes to a wide class of data streams, such as these examples:

Database CRUD transactions

Each insert, update, and delete that changes state is an event. Many databases use a WAL (write-ahead log) internally to append such events durably and quickly to a file before acknowledging the change to clients, after which in-memory data structures and other files with the actual records ...