Apache Kafka A-Z with Hands-On Learning

Table of contents

1. Chapter 1 : Course Introduction
   1. Introduction
2. Chapter 2 : Getting Started with Big data and Apache Kafka
   1. Learning Objectives
   2. Data Evolution over Time
   3. Big Data Overview
   4. Big Data Analytics
   5. Messaging System
   6. Introduction and Features of Kafka
   7. Kafka Use Cases
   8. Kafka Terminologies
   9. Kafka Components
   10. Kafka Architecture
   11. Kafka Clusters
   12. Kafka Industry Use Cases
   13. Demo - Installing Zookeeper
   14. Demo - Installing Kafka
   15. Demo – Setting up a Single Node and Single Broker Cluster
   16. Demo – Setting up a Multi-node Single Broker Cluster
   17. Key Takeaways
3. Chapter 3 : Kafka Producers
   1. Learning Objective
   2. Overview of a Kafka Producer and Its Architecture
   3. Kafka Producer Configuration
   4. Kafka Producer Optional Configuration
   5. Kafka Producer Configuration Objects
   6. Demo - Creating a Kafka Producer
   7. Sending Messages
   8. Serializers and Custom Serializers
   9. Demo - Creating a Custom Serializer
   10. Serializers Challenges and Serializing Using Apache Avro
   11. Demo - Serializing Using Apache Avro
   12. Partitions
   13. Demo – Setting up a Custom Partition
   14. Key Takeaway
4. Chapter 4 : Kafka Consumers
   1. Learning Objective
   2. Overview of Kafka Consumers
   3. Consumer Groups
   4. Partition Rebalance and Creating a Kafka Consumer
   5. Poll Loop and Its Functioning
   6. Configuring a Kafka Consumer - Part 1
   7. Configuring a Kafka Consumer - Part 2
   8. Demo - Creating a Kafka Consumer
   9. Commit and Offset
   10. Ways of Committing Offset - Automatic Offset
   11. Ways of Committing Offset - Commit Current Offset
   12. Ways of Committing Offset - Asynchronous Commit
   13. Ways of Committing Offset - Combining Synchronous and Asynchronous Commits
   14. Ways of Committing Offset - Commit Specified Offset
   15. Rebalance Listeners
   16. Consuming Records with Specific Offset
   17. Deserializers
   18. Demo - Creating and Using Custom Deserializer
   19. Key Takeaways
5. Chapter 5 : Kafka Operations and Performance Tuning
   1. Learning Objective
   2. Kafka Internals Overview
   3. Replication and Replica Types
   4. Preferred Ladder, Request, and Request Processing
   5. Types of Requests
   6. Partition Allocation, File Management, and Segments
   7. File Format, Index, and Compaction
   8. Kafka Reliability and Reliability Methods
   9. Broker Configuration for Replication
   10. Producer in a Reliable System
   11. Consumer in a Reliable System
   12. Key Takeaways
6. Chapter 6 : Kafka Cluster Architectures and Administering Kafka
   1. Learning Objective
   2. Cluster Mirroring
   3. Multi-cluster Architecture - Hub, Spokes, and Active Architecture – Part 1
   4. Multi-cluster Architecture – Active/Stand-by Architecture and Stretch Clusters Part – 2
   5. Configuring MirrorMaker – Part 1
   6. Configuring MirrorMaker – Part 2
   7. Demo - Setting up MirrorMaker
   8. Administering Kafka - Topic Operations
   9. Administering Kafka - Consumer Group Operations
   10. Dynamic Configuration Changes
   11. Partition Management
   12. Console Producer Tool
   13. Console Consumer Tool
   14. Key Takeaways
7. Chapter 7 : Monitoring Kafka and Schema Registry
   1. Learning Objective
   2. Monitoring and Its Importance
   3. Server or Infrastructure Monitoring and Application Monitoring
   4. Monitoring Kafka
   5. Kafka Broker Metrics – Under-replicated Partitions – Part 1
   6. Kafka Broker Metrics – Others– Part 2
   7. Topic and Partition Specific Metrics
   8. Logging and Client Monitoring
   9. Producer and Consumer Metrics
   10. Quotas and Lag Monitoring
   11. Monitoring Dashboard
   12. Demo - Setting up an Open Source Health Monitor
   13. Kafka Schema Registry
   14. Kafka Components and Its Architecture
   15. Kafka Schema Registry - Internal working and Use cases
   16. Working of the Kafka Schema Registry
   17. Demo – Using the Kafka Schema Registry with Kafka
   18. Key Takeaway
8. Chapter 8 : Kafka Streams and Kafka Connectors
   1. Learning Objective
   2. Kafka Streams Overview
   3. Kafka Streams
   4. Kafka Streams Architecture and Working
   5. Kafka Streams Components
   6. Kafka Streams Architecture Tasks, Threading Model and Local State Store
   7. Kafka Streams Architecture - Record Buffer
   8. Memory Management and Streaming Data Pipeline
   9. Kafka Streams Domain Specific Language (DSL)
   10. KStream Operations
   11. KTable Operations
   12. KTable Operations
   13. Aggregation and Windowing
   14. Processor Topology and Stream Processor
   15. Stream and Processor Application Programming Interfaces (APIs)
   16. Processor Application Programming Interfaces (APIs) and Creating Topology
   17. Kafka Connectors
   18. Stand-alone and Sink Connector Configuration
   19. Running Kafka Connect
   20. Kafka Connector Distributed Mode
   21. Hypertext Transfer Protocol (HTTP) Representational State Transfer (REST) Interface
   22. Demo - Kafka Connector
   23. Demo - Creating an Application Using Kafka Streams
   24. Key Takeaway
9. Chapter 9 : Integration of Kafka with Storm
   1. Learning Objective
   2. Apache Storm
   3. Real-time Analytics
   4. Apache Storm Architecture
   5. Apache Storm Components
   6. Apache Storm Topology Part - 1
   7. Apache Storm Topology Part - 2
   8. Kafka Spout
   9. Integration of Apache Storm and Kafka
   10. Demo - Simple Stand-alone Application Using Kafka and Storm
   11. Key Takeaway
10. Chapter 10 : Kafka Integration with Spark and Flume
    1. Learning Objective
    2. Introduction to Spark
    3. Spark Components
    4. Resilient Distributed Dataset
    5. Resilient Distributed Dataset (RDD) Operations - Transformation - Map, FlatMap, and Filter
    6. RDD Operations - Transformation - Join, Distinct, First, and Take
    7. Resilient Distributed Dataset (RDD) Operations - Actions
    8. Data Sets and SparkSession
    9. Data Sets and Spark Session Operations
    10. Spark Stream
    11. Integrating Spark with Kafka
    12. Demo - Running a Small Stand-alone Application in Spark with Kafka
    13. Flume
    14. Flume Connectors
    15. Configuring Flume Kafka to Hadoop Distributed File System (HDFS)
    16. Demo - Creating a Flume Agent and Sending Data from Kafka to Hadoop Distributed File System (HDFS)
    17. Key Takeaways
11. Chapter 11 : Admin Client and Securing Kafka
    1. Learning Objective
    2. AdminClient
    3. Demo - Performing Various Admin Tasks Using AdminClient
    4. Kafka Security
    5. Kafka Security Components
    6. Simple Authentication and Security Layer (SASL)
    7. Configuring Simple Authentication and Security Layer (SASL) in Kafka
    8. Securing Using Access Control Lists (ACLs)
    9. Key Takeaway