book

Fast Data: Smart and at Scale

by Ryan Betts, John Hugg

October 2015

Intermediate to advanced

50 pages

1h 2m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Looking Beyond Streaming
Applications of Fast DataIngestionStreaming AnalyticsPer-Event TransactionsUses of Fast DataFront End for HadoopEnriching Streaming DataQueryable Cache
What Is ACID?What Does ACID Stand For?What Is CAP?What Does CAP Stand For?How Is CAP Consistency Different from ACID Consistency?What Does “Eventual Consistency” Mean in This Context?
Idea in BriefPattern: Reject Requests Past a ThresholdPattern: Alerting on Variations from Predicted TrendsWhen to Avoid This PatternRelated Concepts
Idea in BriefPattern: Use Streaming Transformations to Avoid ETLPattern: Connect Big Data Analytics to Real-Time Stream ProcessingPattern: Use Loose Coupling to Improve ReliabilityWhen to Avoid Pipelines
Idea in BriefPattern: At-Most-Once DeliveryPattern: At-Least-Once DeliveryPattern: Exactly-Once Delivery
Idea in BriefPattern: Use Upserts Over InsertsPattern: Tag Data with Unique IdentifiersSubpattern: Fine-Grained TimestampsSubpattern: Unique IDs at the Event SourcePattern: Use Kafka Offsets as Unique IdentifiersExample: Call Center ProcessingVersion 1: Events Are OrderedVersion 2: Events Are Not OrderedWhen to Avoid This PatternRelated Concepts and Techniques

Overview

The need for fast data applications is growing rapidly, driven by the IoT, the surge in machine-to-machine (M2M) data, global mobile device proliferation, and the monetization of SaaS platforms. So how do you combine real-time, streaming analytics with real-time decisions in an architecture that’s reliable, scalable, and simple?

In this O’Reilly report, Ryan Betts and John Hugg from VoltDB examine ways to develop apps for fast data, using pre-defined patterns. These patterns are general enough to suit both the do-it-yourself, hybrid batch/streaming approach, as well as the simpler, proven in-memory approach available with certain fast database offerings.

Their goal is to create a collection of fast data app development recipes. We welcome your contributions, which will be tested and included in future editions of this report.