Hands-On Software Architecture with Golang

Publishing messages

Sarama (https://github.com/Shopify/sarama) is the most widely-used Go client library for Apache Kafka, written purely in Go. We will be using it to demonstrate various Kafka interactions.

Producing a message for Kafka involves multiple steps, including figuring out what partition of a topic the message must go to, finding the Leader for that topic partition, and transporting the message to that broker (leader) instance. In Sarama, the producer is based on the pipeline-concurrency pattern, where each message is processed by four structs. Each struct has a goroutine that implements a stage in the message-publish process, and these stages are connected by channels.

There are two options for the producer, as described in the ...