Apache Kafka for Absolute Beginners

Video description

This course is designed to get you up and running with the fundamentals and the working of Apache Kafka with the help of practical examples. You will also delve into client API programming in Java. As you progress, you will not only cover the Apache Kafka stack, but also the architecture of Kafka components and Kafka client APIs (Producers and Consumers). The course will even guide you on how to apply the knowledge you’ve gained through the course to create efficient Kafka programs in Java.

This course is based on Apache Kafka 2.x. All the source code and examples used in this course on Apache Kafka 2.3 open-source distribution have been tested. Several examples in this course also make use of the Confluent Community Version of Kafka. The course will focus on the Confluent Community Version to explain and demonstrate functionalities that are only available in the Confluent Platform, such as Schema Registry and Avro Serdes.

This course will be making extensive use of IntelliJ IDEA as the preferred development IDE. However, based on your prior experience, you should be able to work with any other IDE designed for Java application development.

Although the course will be using Apache Maven as the preferred build tool, based on your prior experience, you should be able to use any other build tool designed for Java applications.

In addition to this, the course will use Log4j2 to teach you industry-standard log implementation in your applications.

This course is fully example-driven and several examples will be created in the class.

By the end of this course, you will have gained the skills you need to confidently build Kafka programs in Java.

What You Will Learn

  • Delve into the multi-node Confluent Community of Kafka
  • Discover the Kafka Storage architecture
  • Gain insights into the Kafka Cluster architecture
  • Understand Kafka Consumer API programming in Java
  • Get up to speed with JSON and AVRO Serialization


This course is designed for software engineers, solution architects, and managers looking to implement Kafka and solve real-time stream processing problems. Architects who are not directly involved in Kafka implementation or anyone who is looking to implement Kafka Streams will also find this course helpful. This course is designed for both beginner and intermediate-level Apache Kafka users.

About The Author

Prashant Kumar Pandey: Prashant Kumar Pandey is passionate about helping people learn and grow in their careers by bridging the gap between their existing and required skills. In his journey to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. He is also the founder, lead author, and chief editor of the Learning Journal portal that has been providing various skill development courses, training sessions, and technical articles since 2018.

Publisher resources

Download Example Code

Table of contents

  1. Chapter 1 : Kafka Ecosystem - The Big Picture
    1. What is Apache Kafka?
    2. Apache Kafka Core Concepts
    3. Kafka Connect Core Concepts
    4. Kafka Streams Core Concepts
    5. Kafka SQL Core Concepts
    6. When to use what?
  2. Chapter 2 : Apache Kafka- Getting Started
    1. Kafka Quick Start
    2. Installing Single Node Kafka Cluster
    3. Using Command-Line Producer and Consumer
    4. Installing a Multi-Node Kafka Cluster
    5. Using Consumer Groups
    6. Configuring your development IDE
  3. Chapter 3 : Apache Kafka- Storage Architecture
    1. Understanding Kafka Storage Architecture
    2. Kafka Topics and Partitions
    3. Kafka Topic Replication
    4. Partition Leaders and Followers
    5. Kafka Log Segments
    6. Kafka Message Offsets
    7. Kafka Message Index
  4. Chapter 4 : Apache Kafka- Cluster Architecture
    1. Understanding Kafka Cluster
    2. Zookeeper in Kafka
    3. Kafka Cluster Controller
    4. Partition Allocation and Fault Tolerance
    5. Partition Leader Vs Partition Follower
    6. The ISR List - In Sync Replica
    7. Committed Vs Un-Committed Records
    8. Minimum ISR List
  5. Chapter 5 : Apache Producer Internals
    1. Introducing Kafka Producers
    2. Creating your first Kafka Producer
    3. Producer Record
    4. Producer Serializer
    5. Producer Partitioner
    6. Message Timestamp
    7. Producer Message Buffer
    8. Producer IO Thread and Retires
    9. Summarizing Producer Internals
  6. Chapter 6 : Advanced Kafka Producers
    1. Horizontal Vs. Vertical Scalability
    2. Producer Multi-Threading Scenario
    3. Creating Multi-Threaded Kafka Producer
    4. At Least Once Vs. At Most Once
    5. Exactly Once - Producer Idempotence
    6. Transactions in Kafka Producer
  7. Chapter 7 : Types and Serialization
    1. Working with Types and Serialization
    2. Using JSON Schema
    3. Using AVRO Schema
  8. Chapter 8 : Micro Project- Summing up the Producers
    1. POS Simulator using JSON Serialized Invoices
    2. POS Simulator using AVRO Serialized Invoices
  9. Chapter 9 : Kafka Consumers
    1. Introducing Kafka Consumers
    2. Creating Kafka Consume - Transform - Produce Pipeline
    3. Consumer Group and Scalability
    4. Consumer Positions - Current Offset Vs. Committed Offset
  10. Chapter 10 : Where to go Next
    1. Challenges with Kafka Consumers?

Product information

  • Title: Apache Kafka for Absolute Beginners
  • Author(s): Prashant Kumar Pandey
  • Release date: April 2020
  • Publisher(s): Packt Publishing
  • ISBN: 9781800202054