Apache Kafka for Beginners - Learn Kafka by Hands-On

Apache Kafka for Beginners - Learn Kafka by Hands-On

by Dilip Sundarraj
Released November 2021
Publisher(s): Packt Publishing
ISBN: 9781803247090

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Video description

This course will walk you through the Kafka concepts step by step. We will explore the Kafka internals from a theoretical perspective followed by hands-on exploring the Kafka internals. Then we will learn about building the Kafka Producer to produce messages into Kafka using the producer API. This is followed by learning the different configurations that guarantee message delivery into Kafka and building Kafka Consumer to read messages from Kafka using the consumer API.

In the next section, we will learn about consumer groups and consumer rebalance, consumer offsets, and the different strategies that are available for the consumer to commit the offsets.

Next, we will learn about consumer rebalance listeners to perform some clean-up before partition revocation or some initialization tasks during partition assignment. Then learn about the different techniques from the consumer to seek a different offset.

And finally, we will code and learn how to build custom Serializer/Deserializer and integrate it into our Kafka Producer/Consumer.

By the end of the course, you will have a complete understanding of Apache Kafka and will be able to build Kafka Consumer applications using Java.

What You Will Learn

  • Understand all the Kafka concepts and Kafka core internals
  • Learn Kafka internals through practice
  • Build Kafka Producer applications using Java
  • Learn the Kafka Producer techniques for reliable data delivery
  • Build Kafka Consumer applications using Java
  • Learn to build custom Serializer and Deserializer in Kafka

Audience

This course is designed for Kafka beginners and advanced level. Anyone interested in building Java applications using producer and consumer API or interested in learning advanced Kafka Producer and Consumer operations, then this course is for you.

About The Author

Dilip Sundarraj: Dilip Sundarraj is a software engineer who has experience with building software since 2008. He is passionate about learning modern technologies and staying up to date with all the modern technologies, tools, frameworks, and more. He loves to share his knowledge with the world and that is one of the key reasons for him to be in the online teaching industry.

He loves interacting with other software developers and believes that this helps him share knowledge and learn from them. During his leisure time, he loves to play cricket, watch movies, and work out in the gym to have a balance between physical and mental strength.

Dilip has a YouTube channel named Code with Dilip, where he has been sharing a lot of technical content related to languages, frameworks, best practices, and more.

Table of contents

  1. Chapter 1 : Getting Started with the Course
    1. Course Introduction and Objectives
    2. Prerequisites
  2. Chapter 2 : Getting Started with Kafka
    1. Introduction to Kafka
    2. Kafka Terminologies and Client APIs
  3. Chapter 3 : Download and Install Kafka
    1. Download and Install Kafka
  4. Chapter 4 : Understanding Kafka Components and its Internals - (Theory + Hands-On)
    1. Kafka Topics and Partitions - Theory
    2. Set Up a ZooKeeper/Kafka Broker in Local
    3. Create Topic, Produce and Consume Messages Using the CLI
    4. Produce and Consume Messages with Key
    5. Consumer Offsets
    6. Consumer Groups
    7. Commit Log and Retention Policy
    8. Kafka as a Distributed Streaming System
    9. Setting up a Kafka Cluster in Local with 3 Kafka Brokers
    10. How Kafka Cluster Distributes the Client Requests? Leader/Follower
    11. How Kafka Handles Data Loss? Replication and In-Sync-Replica(ISR)
    12. Fault Tolerance and Robustness in Kafka
  5. Chapter 5 : Kafka Producer API - Hands-On
    1. Project Setup
    2. Sending ProducerRecord Using Producer API
    3. Build Kafka Producer - Send Messages Synchronously
    4. Add Logging Using LogBack
    5. Sending Messages Asynchronously
    6. Send Messages with Key
    7. Programmatic Interface to Produce Messages
    8. Kafka Producer API - Behind the Scenes
  6. Chapter 6 : Kafka Producer API - Guaranteed Message Delivery Configurations
    1. Kafka Producer API Configurations
    2. Configuring Acks and min.insync.replicas – Hands-On
    3. Configuring retries and retry.backoff.ms – Hands-On
  7. Chapter 7 : Kafka Consumer API – Hands-On
    1. Project Setup
    2. Consuming Messages using Consumer API - poll() loop
    3. Build Kafka Consumer Using the Consumer API
    4. auto.offset.reset - Earliest Versus Latest
  8. Chapter 8 : Consumer Groups and Consumer Rebalance – Hands-On
    1. Scaling the Consumer Using Consumer Groups
    2. Consumer Rebalance
    3. Consumer Rebalance and max.poll.interval.ms
  9. Chapter 9 : Consumer Offsets - Default and Manual Offset Management – Hands-On
    1. Committing Offsets - Default Offset Commit Strategy
    2. Manually Committing Offsets
    3. Committing Offsets Synchronously - Using commitSync()
    4. Committing Offsets Asynchronously - Using commitAsync()
    5. Committing Specific Offsets - using CommitSync()
  10. Chapter 10 : Consumer Rebalance Listeners – Hands-On
    1. Introduction to Rebalance Listeners
    2. Implement ConsumerRebalanceListener in Kafka Consumer
    3. Committing Offsets Using ConsumerRebalanceListener
  11. Chapter 11 : Kafka Consumer - seekToBeginning(), seekToEnd(), and seek() – Hands-On
    1. Introduction to seekToBeginning() or seekToEnd()
    2. seekToBeginning() and seekToEnd() – Hands-On
    3. Introduction to seek()
    4. Seek to a Specific Offset Using seek() – Hands-On
  12. Chapter 12 : Custom Serializer and Deserializers in Kafka – Hands-On
    1. Why Custom Kafka Serializer/Deserializer?
    2. Build a Custom Serializer in Kafka Producer Using Jackson ObjectMapper
    3. Build Kafka Producer Using Custom Serializer
    4. Build a Custom DeSerializer in Kafka Producer Using Jackson ObjectMapper
    5. Build Kafka Consumer Using Custom Serializer
    6. Publish Custom Objects Using StringSerializer and Jackson ObjectMapper
    7. Consume Custom Objects using StringDeserializer and Jackson ObjectMapper

Product information

  • Title: Apache Kafka for Beginners - Learn Kafka by Hands-On
  • Author(s): Dilip Sundarraj
  • Release date: November 2021
  • Publisher(s): Packt Publishing
  • ISBN: 9781803247090