book

Streaming Architecture

by Ted Dunning, Ellen Friedman

May 2016

Beginner to intermediate

120 pages

2h 41m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Who Should Use This BookWhat Is CoveredConventions Used in This BookSafari® Books OnlineHow to Contact Us
Planes, Trains, and Automobiles: Connected Vehicles and the IoTStreaming Data: Life As It HappensWhere Streaming MattersBeyond Real Time: More Benefits of Streaming ArchitectureEmerging Best Practices for Streaming ArchitecturesHealthcare Example with Data StreamsStreaming Data as a Central Aspect of Architectural Design
A Limited View: Single Real-Time ApplicationKey Aspects of a Universal Stream-based ArchitectureImportance of the Messaging TechnologyChoices for Real-Time AnalyticsApache StormApache Spark StreamingApache FlinkApache ApexComparison of Capabilities for Streaming AnalyticsSummary
Why Microservices MatterWhat Is Needed to Support MicroservicesMicroservices in More DetailDesigning a Streaming Architecture: Online Video Service ExampleA New Design: Infrastructure to Support MessagingImportance of a Universal MicroarchitectureWhat’s in a Name?Why Use Distributed Files and NoSQL Databases?New Design for the Video ServiceSummary: The Converged Platform View
Motivations for KafkaKafka InnovationsKafka Basic ConceptsOrderingPersistenceThe Kafka APIsKafkaProducer APIKafkaConsumer APILegacy APIsKafka Utility ProgramsLoad BalancingMirroringKafka GotchasKafka in Production SettingsLimited Number of Topics and PartitionsManual Balancing of Partitions and LoadNo Inherent Serialization MechanismMirroring DeficienciesSummary
Innovations in MapR StreamsHistory and Context of MapR’s Streaming SystemHow MapR Streams WorksHow to Configure MapR StreamsGeo-Distributed ReplicationMapR Streams Gotchas
Card VelocityFast Response Decision to the Question: “Is It Fraud?”Multiuse Streaming DataScaling Up the Fraud DetectorSummary
StakeholdersDesign GoalsDesign ChoicesOur DesignFollow the DataControl Who Has Access to Stream DataAdvantages of Streams-based Geo-Replication
Benefits of Stream-based ArchitecturesMaking the Transition to Streaming ArchitectureConclusion
Streaming Data TopicsSelected O’Reilly Publications by the Authors

Content preview from Streaming Architecture

Chapter 5. MapR Streams

A second option for a messaging system that supports the requirements of a stream-based architecture is MapR Streams. Developed as a ground-up reimplementation of the Apache Kafka API, MapR Streams provides the same basic functions of Kafka but also some additional capabilities, as we’ll discuss in this chapter. MapR Streams is integrated into the MapR converged data platform, and it is compatible with the Kafka 0.9 API. Most programs written to run on that API will also run efficiently, without change, on MapR Streams. If you know how to use Kafka, you’ll have a head start on understanding how to use Streams. If you are not familiar with Apache Kafka, you may want to review the previous chapter.

Innovations in MapR Streams

Although similar to Kafka, MapR Streams enables you to do some very different things. At a high level, the differences include running a much larger number of topics and applying policies such as time-to-live or controlled access to many topics as a group. (Such a group of topics in MapR is called a stream, as described later.) The ability to set up a very large number of topics in MapR Streams lets you build topics that reflect business goals rather than infrastructural limitations. This capability allows a good fit between architecture and the business problem being addressed.

Integration of the MapR messaging system into the MapR converged data platform means less administration than is required when the messaging technology is run ...