Kafka Connect workers standalone versus distributed mode

Users can run Kafka Connect in two ways: standalone mode or distributed mode.

In standalone mode, a single process runs all the connectors. It is not fault tolerant. Since it uses only a single process, it is not scalable. Generally, it is useful for users for development and testing purposes.

In distributed mode, multiple workers run Kafka Connect. In this mode, Kafka Connect is scalable and fault tolerant, so it is used in production deployment.

Let's learn more about Kafka and Kafka Connect (standalone mode). In this example, we will do the following:

  1. Install Kafka
  2. Create a topic
  3. Generate a few messages to verify the producer and consumer
  4. Kafka Connect-File-source and file-sink ...

Get Modern Big Data Processing with Hadoop now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.