Mastering Apache Pulsar

Chapter 7. Pulsar IO

You’ve learned about Pulsar’s philosophy and architecture, use cases for Pulsar, and the producer and consumer model. Now it’s time to work through some examples of using Pulsar. The most accessible place to start on this journey is with Pulsar IO, a framework that lets you easily create, deploy, and manage Pulsar connectors that interact with third-party systems. I’ll start by covering the Pulsar IO architecture and then move on to practical examples. At the end of the chapter, we’ll write our own Pulsar IO connector.

Pulsar IO Architecture

Before jumping in, I want to justify my reasons for starting with Pulsar IO as the best way to learn about producers and consumers in Pulsar:

• Pulsar IO provides a framework for producing and consuming messages with Apache Pulsar.

• Pulsar IO connectors can work after configuration without additional code by the end user.

• Pulsar IO is simple to understand.

In Chapters 5 and 6, we discussed producers and consumers in Pulsar and how configuration decisions on one side impact the other side. The best way to understand what it takes to write an application that utilizes Pulsar and the implications of configuration decisions is to write an end-to-end application and break down the pieces. Using Pulsar IO, we can use a Pulsar IO connector (or multiple connectors) to achieve our end goal of understanding the message lifecycle and message topologies in Pulsar.

Pulsar has first-party client library support in Java, Python, ...