book

Streaming Databases

by Hubert Dulay, Ralph Matthias Debusmann

August 2024

Intermediate to advanced

260 pages

6h 44m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Conventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsHubert’s AcknowledgementsRalph’s Acknowledgments
Turning the Database Inside OutExternalizing Database FeaturesWrite-Ahead LogStreaming PlatformsMaterialized ViewsUse Case: Clickstream AnalysisUnderstanding Transactions and EventsDomain-Driven DesignContext EnrichmentChange Data CaptureConnectorsConnector MiddlewareEmbeddedCustom-BuiltSummary
Stateful TransformationsData PipelinesELT LimitationsStream Processing with ELTStream ProcessorsPopular Stream ProcessorsNewer Stream ProcessorsEmulating Materialized Views in Apache SparkTwo Types of StreamsAppend StreamDebezium Change DataMaterialized ViewsSummary
Real-Time ExpectationsChoosing an Analytical Data StoreSourcing from a TopicIngestion TransformationsOLTP Versus OLAPACIDRow- Versus Column-Based OptimizationQueries Per Second and ConcurrencyIndexingServing Analytical ResultsSynchronous QueriesAsynchronous QueriesPush Versus Pull QueriesSummary
Views, Materialized Views, and Incremental UpdatesChange Data CapturePush Versus Pull QueriesCDC and UpsertJoining StreamsApache CalciteClickstream Use CaseSummary
Identifying the Streaming DatabaseColumn-Based Streaming DatabaseRow-Based Streaming DatabaseEdge Streaming-Like DatabasesSQL ExpressivityStreaming DebuggabilityAdvantages of Debugging in Streaming DatabasesSQL Is Not a Silver BulletStreaming Database ImplementationsStreaming Database ArchitectureELT with Streaming DatabasesSummary
A Toy ExampleTransactionsAnalyzing the TransactionsComparing Consistency Across Stream Processing SystemsFlink SQLksqlDBProton (Timeplus)RisingWaveMaterializePathwayOut-of-Order MessagesGoing Beyond Eventual ConsistencyWhy Do Eventually Consistent Stream Processors Fail the Toy Example?How Do Internally Consistent Stream Processing Systems Pass the Toy Example?How Can We Fix Eventually Consistent Stream Processing Systems to Pass the Toy Example?Consistency Versus LatencySummary
Data PlanesHybrid Transactional/Analytical DatabaseOther Hybrid DatabasesMotivations for Hybrid SystemsThe Influence of PostgreSQL on Hybrid DatabasesNear-Edge AnalyticsNext-Generation Hybrid DatabasesNext-Generation Streaming OLTP DatabasesNext-Generation Streaming RTOLAP DatabasesNext-Generation HTAP DatabasesSummary
ETL ModelZero-ETLNear-Zero-ETLPeerDBProtonEmbedded OLAPData Gravity and ReplicationAnalytical Data ReductionLambda ArchitectureApache Pinot Hybrid TablesPipeline ConfigurationsSummary

Data GravityComponents of the Streaming PlaneStreaming Plane InfrastructureOperational AnalyticsData MeshPillars of a Data MeshChallenge of a Data MeshStreaming Data Mesh with Streaming Plane and Streaming DatabasesData LocalityData ReplicationSummary
Consistent Streaming DatabaseConsistent Streaming Processor and RTOLAPEventually Consistent OLAP Streaming DatabaseEventually Consistent Stream Processor and RTOLAPEventually Consistent Stream Processor and HTAPksqlDBIncremental View MaintenancePostgres Multicorn Foreign Data WrapperWhen to Use Code-Based Stream ProcessorsWhen to Use Lakehouse/Streamhouse TechnologiesCaching TechnologiesWhere to Do Processing and Querying in General?The Four “Where” QuestionsAn Analytical Use CaseConsequencesSummary
The Convergence of the Data PlanesGraph DatabasesMemgraphthatDot/QuineVector DatabasesMilvus 2.x: Streaming as the Central BackboneRTOLAP Databases: Adding Vector SearchIncremental View Maintenancepg_ivmHydraEpsioFelderaPeerDBData Wrapping and Postgres MulticornClassical DatabasesData WarehousesBigQueryRedshiftSnowflakeLakehouseDelta LakeApache PaimonApache IcebergApache HudiOneTable or XTableThe Relationship of Streaming and LakehousesConclusion

Overview

Real-time applications are becoming the norm today. But building a model that works properly requires real-time data from the source, in-flight stream processing, and low latency serving of its analytics. With this practical book, data engineers, data architects, and data analysts will learn how to use streaming databases to build real-time solutions.

Authors Hubert Dulay and Ralph M. Debusmann take you through streaming database fundamentals, including how these databases reduce infrastructure for real-time solutions. You'll learn the difference between streaming databases, stream processing, and real-time online analytical processing (OLAP) databases. And you'll discover when to use push queries versus pull queries, and how to serve synchronous and asynchronous data emanating from streaming databases.

This guide helps you: