Apache Pulsar Versus Apache Kafka

Apache Kafka is a widely used publish–subscribe (pub–sub) messaging system. It originated from LinkedIn and became a top-level Apache Software Foundation (ASF) project in 2011. In recent years, Apache Pulsar has emerged as a serious alternative to Kafka and is being adopted by an increasing number of enterprises in use cases where Kafka has long ruled. In this report, we go over the key differences between Kafka and Pulsar and give some insight into why Pulsar is gaining momentum.

What Is Apache Pulsar?

Like Kafka, Apache Pulsar was developed inside an internet-scale company to solve its own problems. In 2015, engineers at Yahoo! needed a pub–sub messaging system that could deliver low-consistency latency on commodity hardware. It also needed to scale to millions of topics and provide strong durability guarantees for all messages that it handled.

The Yahoo! engineers evaluated the solutions that were available at the time, but couldn’t find one that met all their requirements. So, they set off to build a new pub–sub messaging system that would be able to support their global applications such as Mail, Finance, Sports, and Gemini Ads. Their solution, which became Apache Pulsar, has been running in production in Yahoo! since 2016.

Architecture

Let’s begin our comparison of Kafka and Pulsar by looking at the architecture of the two systems. Since Kafka was well known at the time, the creators of Pulsar were well aware of its architecture. As you ...