O'Reilly logo

Storm Blueprints: Patterns for Distributed Real-time Computation by Brian O'Neill, P. Taylor Goetz

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Introducing the log analysis topology

With the means to write our log data to Kafka, we're ready to turn our attention to the implementation of a Trident topology to perform the analytical computation. The topology will perform the following operations:

  1. Receive and parse the raw JSON log event data.
  2. Extract and emit necessary fields.
  3. Update an exponentially-weighted moving average function.
  4. Determine if the moving average has crossed a specified threshold.
  5. Filter out events that do not represent a state change (for example, rate moved above/below threshold).
  6. Send an instant message (XMPP) notification.

The topology is depicted in the following diagram with the Trident stream operations at the top and stream processing components at the bottom:

Kafka spout ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required