book

Apache Pulsar in Action

by David Kjerrumgaard

December 2021

Intermediate to advanced

400 pages

12h 27m

English

Manning Publications

Read now

Unlock full access

forewordprefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A roadmapAbout the codeOther online resourcesliveBook discussion forumabout the authorabout the cover illustration
1.1 Enterprise messaging systems1.1.1 Key capabilities1.2 Message consumption patterns1.2.1 Publish-subscribe messaging1.2.2 Message queuing1.3 The evolution of messaging systems1.3.1 Generic messaging systems1.3.2 Message-oriented middleware1.3.3 Enterprise service bus1.3.4 Distributed messaging systems1.4 Comparison to Apache Kafka1.4.1 Multilayered architecture1.4.2 Message consumption1.4.3 Data durability1.4.4 Message acknowledgment1.4.5 Message retention1.5 Why do I need Pulsar?1.5.1 Guaranteed message delivery1.5.2 Infinite scalability1.5.3 Resilient to failure1.5.4 Support for millions of topics1.5.5 Geo-replication and active failover1.6 Real-world use cases1.6.1 Unified messaging systems1.6.2 Microservices platforms1.6.3 Connected cars1.6.4 Fraud detectionAdditional resourcesSummary
2.1 Pulsar’s physical architecture2.1.1 Pulsar’s layered architecture2.1.2 Stateless serving layer2.1.3 Stream storage layer2.1.4 Metadata storage2.2 Pulsar’s logical architecture2.2.1 Tenants, namespaces, and topics2.2.2 Addressing topics in Pulsar2.2.3 Producers, consumers, and subscriptions2.2.4 Subscription types2.3 Message retention and expiration2.3.1 Data retention2.3.2 Backlog quotas2.3.3 Message expiration2.3.4 Message backlog vs. message expiration2.4 Tiered storageSummary
3.1 Getting started with Pulsar3.2 Administering Pulsar3.2.1 Creating a tenant, namespace, and topic3.2.2 Java Admin API3.3 Pulsar clients3.3.1 The Pulsar Java client3.3.2 The Pulsar Python client3.3.3 The Pulsar Go client3.4 Advanced administration3.4.1 Persistent topic metrics3.4.2 Message inspectionSummary

4.1 Stream processing4.1.1 Traditional batching4.1.2 Micro-batching4.1.3 Stream native processing4.2 What is Pulsar Functions?4.2.1. Programming model4.3 Developing Pulsar functions4.3.1 Language native functions4.3.2 The Pulsar SDK4.3.3 Stateful functions4.4 Testing Pulsar functions4.4.1 Unit testing4.4.2 Integration testing4.5 Deploying Pulsar functions4.5.1 Generating a deployment artifact4.5.2 Function configuration4.5.3 Function deployment4.5.4 The function deployment life cycle4.5.5 Deployment modes4.5.6 Pulsar function data flowSummary
5.1 What are Pulsar IO connectors?5.1.1 Sink connectors5.1.2 Source connectors5.1.3 PushSource connectors5.2 Developing Pulsar IO connectors5.2.1 Developing a sink connector5.2.2 Developing a PushSource connector5.3 Testing Pulsar IO connectors5.3.1 Unit testing5.3.2 Integration testing5.3.3 Packaging Pulsar IO connectors5.4 Deploying Pulsar IO connectors5.4.1 Creating and deleting connectors5.4.2 Debugging deployed connectors5.5 Pulsar’s built-in connectors5.5.1 Launching the MongoDB cluster5.5.2 Link the Pulsar and MongoDB containers5.5.3 Configure and create the MongoDB sink5.6 Administering Pulsar IO connectors5.6.1 Listing connectors5.6.2 Monitoring connectorsSummary
6.1 Transport encryption6.2 Authentication6.2.1 TLS authentication6.2.2 JSON Web Token authentication6.3 Authorization6.3.1 Roles6.3.2 An example scenario6.4 Message encryptionSummary
7.1 Microservice communication7.1.1 Microservice APIs7.1.2 The need for a schema registry7.2 The Pulsar schema registry7.2.1 Architecture7.2.2 Schema versioning7.2.3 Schema compatibility7.2.4 Schema compatibility check strategies7.3 Using the schema registry7.3.1 Modelling the food order event in Avro7.3.2 Producing food order events7.3.3 Consuming the food order events7.3.4 Complete example7.4 Evolving the schemaSummary
8.1 Data pipelines8.1.1 Procedural programming8.1.2 DataFlow programming8.2 Message routing patterns8.2.1 Splitter pattern8.2.2 Dynamic router pattern8.2.3 Content-based router pattern8.3 Message transformation patterns8.3.1 Message translator pattern8.3.2 Content enricher pattern8.3.3 Content filter patternSummary
9.1 Pulsar Functions resiliency9.1.1 Adverse events9.1.2 Fault detection9.2 Resiliency design patterns9.2.1 Retry pattern9.2.2 Circuit breaker pattern9.2.3 Rate limiter pattern9.2.4 Time limiter pattern9.2.5 Cache pattern9.2.6 Fallback pattern9.2.7 Credential refresh pattern9.3 Multiple layers of resiliencySummary
10.1 Data sources10.2 Data access use cases10.2.1 Device validation10.2.2 Driver location dataSummary
11.1 Deploying ML models11.1.1 Batch processing11.1.2 Near real-time11.2 Near real-time model deployment11.3 Feature vectors11.3.1 Feature stores11.3.2 Feature calculation11.4 Delivery time estimation11.4.1 ML model export11.4.2 Feature vector mapping11.4.3 Model deployment11.5 Neural nets11.5.1 Neural net training11.5.2 Neural net deployment in JavaSummary
12.1 IIoT architecture12.1.1 The perception and reaction layer12.1.2 The transportation layer12.1.3 The data processing layer12.2 A Pulsar-based processing layer12.3 Edge analytics12.3.1 Telemetric data12.3.2 Univariate and multivariate12.4 Univariate analysis12.4.1 Noise reduction12.4.2 Statistical analysis12.4.3 Approximation12.5 Multivariate analysis12.5.1 Creating a bidirectional messaging mesh12.5.2 Multivariate dataset construction12.6 Beyond the bookSummary
A.1 Create a Kubernetes clusterA.1.1 Install prerequisitesA.1.2 MinikubeA.2 The Pulsar Helm chartA.2.1 What is Helm?A.2.2 The Pulsar Helm chartA.3 Using the Pulsar Helm chartA.3.1 Administering Pulsar on KubernetesA.3.2 Configuring clients
B.1 Synchronous geo-replicationB.2 Asynchronous geo-replicationB.2.1 Configuring asynchronous geo-replicationB.3 Asynchronous geo-replication patternsB.3.1 Multi-active geo-replicationB.3.2 Active-standby geo-replicationB.3.3 Aggregation geo-replication

Content preview from Apache Pulsar in Action

5 Pulsar IO connectors

This chapter covers

An introduction to the Pulsar IO framework
Configuring, deploying, and monitoring Pulsar IO connectors
Writing your own Pulsar IO connector in Java

Messaging systems are much more useful when you can easily use them to move data into and out of other external systems, such as databases, local and distributed filesystems, or other messaging systems. Consider the scenario where you want to ingest log data from external sources, such as applications, platforms, and cloud-based services, and publish it to a search engine for analysis. This could easily be accomplished with a pair of Pulsar IO connectors; the first would be a Pulsar source that collects the application logs, and the second would be ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Publisher Resources

ISBN: 9781617296888Supplemental Content Publisher Support Other Publisher Website Supplemental Content Purchase Link

Apache Pulsar in Action

by David Kjerrumgaard

5 Pulsar IO connectors

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

You might also like

Mastering Apache Pulsar

Apache Pulsar Versus Apache Kafka

Cloud-Native Microservices with Apache Pulsar: Build Distributed Messaging Microservices

Event Streams in Action

Publisher Resources

5 Pulsar IO connectors

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,and much more.

You might also like

Mastering Apache Pulsar

Apache Pulsar Versus Apache Kafka

Cloud-Native Microservices with Apache Pulsar: Build Distributed Messaging Microservices

Event Streams in Action

Publisher Resources

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.