Apache Kafka A-Z with Hands-On Learning

Video description

Do you want to learn Apache Kafka to store and process multiple, nonstop streams of information faster? Or do you want to prepare for the Apache Kafka certification exam? This course is an ideal resource to learn Apache Kafka and to reach your goals.

The course starts with an introduction to big data and Apache Kafka. Next, you will get into the details of a Kafka producer and consumer and grasp the techniques to tune the performance of Kafka operations. Moving along, you will explore the Kafka architecture and learn how to administer Kafka effectively. Next, you will learn to monitor server, infrastructure, and application and understand how the Schema Registry works in Kafka. Later, you will get to grips with Kafka Streams and Connectors and learn the process of integrating Kafka with Apache Storm, Spark, and Flume. Towards the end, you will learn how to perform various admin tasks using the admin clients and uncover useful tips to secure Kafka.

By the end of this course, you will have developed the Kafka skills needed to handle real-time data feed and to pass the Apache Kafka certification exam.

What You Will Learn

  • Create a Kafka producer and a Kafka consumer
  • Discover Kafka reliability methods
  • Become familiar with MirrorMaker deployment and tuning
  • Build a simple stand-alone application using Kafka and Storm
  • Create a Flume agent for sending data from Kafka to HDFS
  • Configure SSL certificate in Kafka


If you are into analytics or research or are a software developer, test engineer, or a project manager, who is looking to learn Apache Kafka and achieve an Apache Kafka certification, this course is for you. A basic understanding of big data is needed to get started with this course.

About The Author

Learnkart Technology Private Limited: Learnkart understands the importance of upskilling and its impact on you. They pride themselves in creating specialized e-learning courses that will not only prepare you for certification exams but also help you gain hands-on knowledge for real-world applications. Thousands of students have developed their skills with Learnkart through a variety of courses, such as PgMP, Risk Management Professional (RMP), speech recognition, Python, Amazon Web Services (AWS), and more. Learnkart regularly adds courses to its portfolio to lend more value to your learning experience.

Table of contents

  1. Chapter 1 : Course Introduction
    1. Introduction
  2. Chapter 2 : Getting Started with Big data and Apache Kafka
    1. Learning Objectives
    2. Data Evolution over Time
    3. Big Data Overview
    4. Big Data Analytics
    5. Messaging System
    6. Introduction and Features of Kafka
    7. Kafka Use Cases
    8. Kafka Terminologies
    9. Kafka Components
    10. Kafka Architecture
    11. Kafka Clusters
    12. Kafka Industry Use Cases
    13. Demo - Installing Zookeeper
    14. Demo - Installing Kafka
    15. Demo – Setting up a Single Node and Single Broker Cluster
    16. Demo – Setting up a Multi-node Single Broker Cluster
    17. Key Takeaways
  3. Chapter 3 : Kafka Producers
    1. Learning Objective
    2. Overview of a Kafka Producer and Its Architecture
    3. Kafka Producer Configuration
    4. Kafka Producer Optional Configuration
    5. Kafka Producer Configuration Objects
    6. Demo - Creating a Kafka Producer
    7. Sending Messages
    8. Serializers and Custom Serializers
    9. Demo - Creating a Custom Serializer
    10. Serializers Challenges and Serializing Using Apache Avro
    11. Demo - Serializing Using Apache Avro
    12. Partitions
    13. Demo – Setting up a Custom Partition
    14. Key Takeaway
  4. Chapter 4 : Kafka Consumers
    1. Learning Objective
    2. Overview of Kafka Consumers
    3. Consumer Groups
    4. Partition Rebalance and Creating a Kafka Consumer
    5. Poll Loop and Its Functioning
    6. Configuring a Kafka Consumer - Part 1
    7. Configuring a Kafka Consumer - Part 2
    8. Demo - Creating a Kafka Consumer
    9. Commit and Offset
    10. Ways of Committing Offset - Automatic Offset
    11. Ways of Committing Offset - Commit Current Offset
    12. Ways of Committing Offset - Asynchronous Commit
    13. Ways of Committing Offset - Combining Synchronous and Asynchronous Commits
    14. Ways of Committing Offset - Commit Specified Offset
    15. Rebalance Listeners
    16. Consuming Records with Specific Offset
    17. Deserializers
    18. Demo - Creating and Using Custom Deserializer
    19. Key Takeaways
  5. Chapter 5 : Kafka Operations and Performance Tuning
    1. Learning Objective
    2. Kafka Internals Overview
    3. Replication and Replica Types
    4. Preferred Ladder, Request, and Request Processing
    5. Types of Requests
    6. Partition Allocation, File Management, and Segments
    7. File Format, Index, and Compaction
    8. Kafka Reliability and Reliability Methods
    9. Broker Configuration for Replication
    10. Producer in a Reliable System
    11. Consumer in a Reliable System
    12. Key Takeaways
  6. Chapter 6 : Kafka Cluster Architectures and Administering Kafka
    1. Learning Objective
    2. Cluster Mirroring
    3. Multi-cluster Architecture - Hub, Spokes, and Active Architecture – Part 1
    4. Multi-cluster Architecture – Active/Stand-by Architecture and Stretch Clusters Part – 2
    5. Configuring MirrorMaker – Part 1
    6. Configuring MirrorMaker – Part 2
    7. Demo - Setting up MirrorMaker
    8. Administering Kafka - Topic Operations
    9. Administering Kafka - Consumer Group Operations
    10. Dynamic Configuration Changes
    11. Partition Management
    12. Console Producer Tool
    13. Console Consumer Tool
    14. Key Takeaways
  7. Chapter 7 : Monitoring Kafka and Schema Registry
    1. Learning Objective
    2. Monitoring and Its Importance
    3. Server or Infrastructure Monitoring and Application Monitoring
    4. Monitoring Kafka
    5. Kafka Broker Metrics – Under-replicated Partitions – Part 1
    6. Kafka Broker Metrics – Others– Part 2
    7. Topic and Partition Specific Metrics
    8. Logging and Client Monitoring
    9. Producer and Consumer Metrics
    10. Quotas and Lag Monitoring
    11. Monitoring Dashboard
    12. Demo - Setting up an Open Source Health Monitor
    13. Kafka Schema Registry
    14. Kafka Components and Its Architecture
    15. Kafka Schema Registry - Internal working and Use cases
    16. Working of the Kafka Schema Registry
    17. Demo – Using the Kafka Schema Registry with Kafka
    18. Key Takeaway
  8. Chapter 8 : Kafka Streams and Kafka Connectors
    1. Learning Objective
    2. Kafka Streams Overview
    3. Kafka Streams
    4. Kafka Streams Architecture and Working
    5. Kafka Streams Components
    6. Kafka Streams Architecture Tasks, Threading Model and Local State Store
    7. Kafka Streams Architecture - Record Buffer
    8. Memory Management and Streaming Data Pipeline
    9. Kafka Streams Domain Specific Language (DSL)
    10. KStream Operations
    11. KTable Operations
    12. KTable Operations
    13. Aggregation and Windowing
    14. Processor Topology and Stream Processor
    15. Stream and Processor Application Programming Interfaces (APIs)
    16. Processor Application Programming Interfaces (APIs) and Creating Topology
    17. Kafka Connectors
    18. Stand-alone and Sink Connector Configuration
    19. Running Kafka Connect
    20. Kafka Connector Distributed Mode
    21. Hypertext Transfer Protocol (HTTP) Representational State Transfer (REST) Interface
    22. Demo - Kafka Connector
    23. Demo - Creating an Application Using Kafka Streams
    24. Key Takeaway
  9. Chapter 9 : Integration of Kafka with Storm
    1. Learning Objective
    2. Apache Storm
    3. Real-time Analytics
    4. Apache Storm Architecture
    5. Apache Storm Components
    6. Apache Storm Topology Part - 1
    7. Apache Storm Topology Part - 2
    8. Kafka Spout
    9. Integration of Apache Storm and Kafka
    10. Demo - Simple Stand-alone Application Using Kafka and Storm
    11. Key Takeaway
  10. Chapter 10 : Kafka Integration with Spark and Flume
    1. Learning Objective
    2. Introduction to Spark
    3. Spark Components
    4. Resilient Distributed Dataset
    5. Resilient Distributed Dataset (RDD) Operations - Transformation - Map, FlatMap, and Filter
    6. RDD Operations - Transformation - Join, Distinct, First, and Take
    7. Resilient Distributed Dataset (RDD) Operations - Actions
    8. Data Sets and SparkSession
    9. Data Sets and Spark Session Operations
    10. Spark Stream
    11. Integrating Spark with Kafka
    12. Demo - Running a Small Stand-alone Application in Spark with Kafka
    13. Flume
    14. Flume Connectors
    15. Configuring Flume Kafka to Hadoop Distributed File System (HDFS)
    16. Demo - Creating a Flume Agent and Sending Data from Kafka to Hadoop Distributed File System (HDFS)
    17. Key Takeaways
  11. Chapter 11 : Admin Client and Securing Kafka
    1. Learning Objective
    2. AdminClient
    3. Demo - Performing Various Admin Tasks Using AdminClient
    4. Kafka Security
    5. Kafka Security Components
    6. Simple Authentication and Security Layer (SASL)
    7. Configuring Simple Authentication and Security Layer (SASL) in Kafka
    8. Securing Using Access Control Lists (ACLs)
    9. Key Takeaway

Product information

  • Title: Apache Kafka A-Z with Hands-On Learning
  • Author(s): Learnkart Technology Private Limited
  • Release date: January 2021
  • Publisher(s): Packt Publishing
  • ISBN: 9781801077569