book

Making Sense of Stream Processing

by Martin Kleppmann

May 2016

Intermediate to advanced

181 pages

4h 3m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Implementing Google Analytics: A Case StudyAggregated SummariesEvent Sourcing: From the DDD CommunityBringing Together Event Sourcing and Stream ProcessingTwitterFacebookImmutable Facts and the Source of TruthWikipediaLinkedInUsing Append-Only Streams of Immutable EventsTools: Putting Ideas into PracticeCEP, Actors, Reactive, and More
Case Study: Web Application Developers Driven to InsanityDual WritesMaking Sure Data Ends Up in the Right PlacesThe Ubiquitous LogHow Logs Are Used in Practice1) Database Storage Engines2) Database Replication3) Distributed Consensus4) KafkaSolving the Data Integration ProblemTransactions and Integrity ConstraintsConclusion: Use Logs to Make Your Infrastructure SolidFurther Reading
Introducing Change Data CaptureDatabase = Log of ChangesImplementing the Snapshot and the Change StreamBottled Water: Change Data Capture with PostgreSQL and KafkaWhy Kafka?Why Avro?The Logical Decoding Output Plug-InThe Client DaemonConcurrencyStatus of Bottled Water
Simple Log Analysis with Unix ToolsPipes and ComposabilityUnix Architecture versus Database ArchitectureComposability Requires a Uniform InterfaceBringing the Unix Philosophy to the Twenty-First Century
How Databases Are Used1. Replication2. Secondary Indexes3. Caching4. Materialized ViewsSummary: Four Database-Related IdeasMaterialized Views: Self-Updating CachesExample: Implementing TwitterThe Unbundled DatabaseStreaming All the Way to the User InterfaceConclusion

Overview

How can event streams help make your application more scalable, reliable, and maintainable? In this report, O’Reilly author Martin Kleppmann shows you how stream processing can make your data storage and processing systems more flexible and less complex. Structuring data as a stream of events isn’t new, but with the advent of open source projects such as Apache Kafka and Apache Samza, stream processing is finally coming of age.

Using several case studies, Kleppmann explains how these projects can help you reorient your database architecture around streams and materialized views. The benefits of this approach include better data quality, faster queries through precomputed caches, and real-time user interfaces. Learn how to open up your data for richer analysis and make your applications more scalable and robust in the face of failures.

Understand stream processing fundamentals and their similarities to event sourcing, CQRS, and complex event processing
Learn how logs can make search indexes and caches easier to maintain
Explore the integration of databases with event streams, using the new Bottled Water open source tool
Turn your database architecture inside out by orienting it around streams and materialized views