book

Building Real-Time Analytics Systems

by Mark Needham

September 2023

Beginner to intermediate

220 pages

4h 36m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

Conventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
What Is an Event Stream?Making Sense of Streaming DataWhat Is Real-Time Analytics?Benefits of Real-Time AnalyticsNew Revenue StreamsTimely Access to InsightsReduced Infrastructure CostImproved Overall Customer ExperienceReal-Time Analytics Use CasesUser-Facing AnalyticsPersonalizationMetricsAnomaly Detection and Root Cause AnalysisVisualizationAd Hoc AnalyticsLog Analytics/Text SearchClassifying Real-Time Analytics ApplicationsInternal Versus External FacingMachine Versus Human FacingSummary
Defining the Real-Time Analytics EcosystemThe Classic Streaming StackComplex Event ProcessingThe Big Data EraThe Modern Streaming StackEvent ProducersStreaming Data PlatformStream Processing LayerServing LayerFrontendSummary
Existing ArchitectureSetupMySQLApache KafkaZooKeeperOrders ServiceSpinning Up the ComponentsInspecting the DataApplications of Real-Time AnalyticsSummary
What Is Kafka Streams?What Is Quarkus?Quarkus ApplicationInstalling the Quarkus CLICreating a Quarkus ApplicationCreating a TopologyQuerying the Key-Value StoreCreating an HTTP EndpointRunning the ApplicationQuerying the HTTP EndpointLimitations of Kafka StreamsSummary
Why Can’t We Use Another Stream Processor?Why Can’t We Use a Data Warehouse?What Is Apache Pinot?How Does Pinot Model and Store Data?SchemaTableSetupData IngestionPinot Data ExplorerIndexesUpdating the Web AppSummary
Dashboard ArchitectureWhat Is Streamlit?SetupBuilding the DashboardSummary
Capturing Changes from Operational DatabasesChange Data CaptureWhy Do We Need CDC?What Is CDC?What Are the Strategies for Implementing CDC?Log-Based Data CaptureRequirements for a CDC SystemDebeziumApplying CDC to AATDSetupConnecting Debezium to MySQLQuerying the Products StreamUpdating ProductsSummary
Enriching Orders with Kafka StreamsAdding Order Items to PinotUpdating the Orders ServiceRefreshing the Streamlit DashboardSummary

Order StatusesEnriched Orders StreamUpserts in Apache PinotUpdating the Orders ServiceCreating UsersResourceAdding an allUsers EndpointAdding an Orders for User EndpointAdding an Individual Order EndpointConfiguring Cross-Origin Resource SharingFrontend AppOrder Statuses on the DashboardTime Spent in Each Order StatusOrders That Might Be StuckSummary
Delivery StatusesUpdating Apache PinotOrdersDelivery StatusesUpdating the Orders ServiceIndividual OrdersDelayed Orders by AreaConsuming the New API EndpointsSummary
PreproductionCapacity PlanningData PartitioningThroughputData RetentionData GranularityTotal Data SizeReplication FactorDeployment PlatformIn-House SkillsData Privacy and SecurityCostControlPostproductionMonitoring and AlertingData GovernanceSummary
Content Recommendation (Professional Social Network)The ProblemThe SolutionBenefitsOperational Analytics (Streaming Service)The ProblemThe SolutionBenefitsReal-Time Ad Analytics (Online Marketplace)The ProblemThe SolutionBenefitsUser-Facing Analytics (Collaboration Platform)The ProblemThe SolutionBenefitsSummary
Edge AnalyticsCompute-Storage SeparationData LakehousesReal-Time Data VisualizationStreaming DatabasesStreaming Data Platform as a ServiceReverse ETLSummary

Content preview from Building Real-Time Analytics Systems

About the Author

Mark Needham is an Apache Pinot advocate and developer advocate at StarTree. Mark helps users learn how to use Apache Pinot to build their real-time, user-facing analytics applications. He also does developer experience, simplifying the getting-started experience by making product tweaks and improvements to the documentation. Mark writes about his experiences working with Pinot at markhneedham.com. He tweets as @markhneedham.