Apache Kafka Series - Learn Apache Kafka for Beginners v3

Video description

Understand Apache Kafka 3.0 and its configuration and apply the knowledge in real-time use cases.

About This Video

  • Find out why Apache Kafka is an excellent choice of distributed data-streaming platform
  • Learn major CLIs: kafka-topics, kafka-console-producer, kafka-console-consumer, kafka-consumer-groups, kafka-configs
  • Get a solid grip on Apache Kafka by understanding various real-world examples

In Detail

The high throughput and low latency of Apache Kafka have made it one of the leading distributed data-streaming enterprise technologies. It is used by many Fortune 500 companies such as Netflix, Airbnb, Uber, Walmart, and LinkedIn. If you want to develop the Apache Kafka skills to stream data easily, then this course is for you.

The course starts by explaining the architecture of the Apache Kafka ecosystem, before going on to cover the core concepts of Kafka such as topics, partitions, brokers, replicas, producers, consumers, and more. Next, you will use native Kafka binaries to launch your own Kafka cluster on Windows, Mac OS X, and Linux.

As you advance, you will get hands-on experience with the Kafka command-line interface (CLI), along with understanding how to create producers and consumers in Java to interact with Kafka. Next, a real-world project using Wikimedia as a source of data for a producer and OpenSearch as a sink for our consumers. Moving on, you will get to grips with advanced APIs such as Kafka Connect and Kafka Streams and perform a case study on the real-world applications of Kafka. Finally, you will get an overview of advanced Kafka for administrators and understand advanced topic configurations.

By the end of this course, you will be well-versed with how Apache Kafka 3.0 plays an important role in data-streaming applications.

Audience

This course is for developers who want to learn the fundamentals of Apache Kafka, software architects who want to understand how Kafka fits into their solution architecture, or anyone who is looking to understand how Apache Kafka works as a distributed system. Basic knowledge of Java and the Linux command line will be beneficial to effectively understanding the concepts covered in the course.

Publisher resources

Download Example Code

Table of contents

  1. Chapter 1 : Kafka Introduction
    1. Course Introduction
    2. Apache Kafka in Five Minutes
    3. Course Objectives
  2. Chapter 2 : Kafka Theory
    1. Topics, Partitions, and Offsets
    2. Producers and Message Keys
    3. Consumers and Deserialization
    4. Consumer Groups and Consumer Offsets
    5. Brokers and Topics
    6. Topic Replication
    7. Producer Acknowledgments and Topic Durability
    8. Zookeeper
    9. Kafka Kraft – Removing Zookeeper
    10. Theory Roundup
  3. Chapter 3 : Starting Kafka
    1. Important: Starting Kafka
    2. Starting Kafka with Conduktor – Multi Platform
    3. Mac OS X - Download Kafka and Set Up PATH
    4. Mac OS X - Start Zookeeper and Kafka
    5. Mac OS X - Using Brew
    6. Linux - Download and Set Up Kafka in PATH
    7. Linux - Start Zookeeper and Kafka
    8. Windows WSL2 - Download Kafka and PATH Setup
    9. Windows WSL2 - Start Zookeeper and Kafka
    10. Windows Non-WSL2 - Start Zookeeper and Kafka
  4. Chapter 4 : Starting Kafka without Zookeeper
    1. Mac OS X - Start Kafka in Kraft Mode
    2. Linux - Start Kafka in Kraft Mode
    3. Windows WSL2 - Start Kafka in Kraft Mode
  5. Chapter 5 : Command Line Interface (CLI) 101
    1. Command Line Interface (CLI) Introduction
    2. Kafka Topics Command Line Interface (CLI)
    3. Kafka Console Producer Command Line Interface (CLI)
    4. Kafka Console Consumer Command Line Interface (CLI)
    5. Kafka Consumers in a Group
    6. Kafka Consumer Groups Command Line Interface (CLI)
    7. Resetting Offsets
  6. Chapter 6 : Kafka UI
    1. Conduktor - Demo
  7. Chapter 7 : Kafka Java Programming 101
    1. Kafka SDK List
    2. Creating Kafka Project
    3. Java Producer
    4. Java Producer Callback
    5. Java Producer with Keys
    6. Java Consumer
    7. Java Consumer – Graceful Shutdown
    8. Java Consumer Inside a Consumer Group
    9. Java Consumer Incremental Cooperative Rebalance and Static Group Membership
    10. Java Consumer Incremental Cooperative Rebalance - Practice
    11. Java Consumer Auto Offset Commit Behavior
    12. Programming - Advanced Tutorials
  8. Chapter 8 : Kafka Real-World Project
    1. Real-World Project Overview
  9. Chapter 9 : Kafka Wikimedia Producer and Advanced Configurations
    1. Wikimedia Producer Project Setup
    2. Wikimedia Producer Implementation
    3. Wikimedia Producer Run
    4. Wikimedia Producer - Producer Config Intros
    5. Producer Acknowledgements Deep Dive
    6. Producer Retries
    7. Idempotent Producer
    8. Safe Kafka Producer Settings
    9. Wikimedia Producer Safe Producer Implementation
    10. Kafka Message Compression
    11. linger.ms and batch.size Producer Settings
    12. Wikimedia Producer High Throughput Implementation
    13. Producer Default Partitioner and Sticky Partitioner
    14. [Advanced] max.block.ms and buffer.memory
  10. Chapter 10 : OpenSearch Consumer and Advanced Consumer Configurations
    1. OpenSearch Consumer - Project Overview
    2. OpenSearch Consumer - Project Setup
    3. Setting Up OpenSearch on Docker
    4. Setting Up OpenSearch on the Cloud
    5. OpenSearch 101
    6. OpenSearch Consumer Implementation - Part 1
    7. OpenSearch Consumer Implementation Part 2
    8. Consumer Delivery Semantics
    9. OpenSearch Consumer Implementation Part 3 - Idempotence
    10. Consumer Offsets Commit Strategies
    11. OpenSearch Consumer Implementation Part 4 - Delivery Semantics
    12. OpenSearch Consumer Implementation Part 5 - Batching Data
    13. Consumer Offset Reset Behavior
    14. OpenSearch Consumer Implementation Part 6 - Replaying Data
    15. Consumer Internal Threads
    16. Consumer Replica Fetching - Rack Awareness
  11. Chapter 11 : Kafka Extended APIs for Developers
    1. Kafka Extended APIs - Overview
    2. Kafka Connect Introduction
    3. Kafka Connect Wikimedia and ElasticSearch Hands-On
    4. Kafka Streams Introduction
    5. Kafka Streams Hands-On
    6. Kafka Schema Registry Introduction
    7. Kafka Schema Registry Hands-On
    8. Which Kafka API Should I Use?
  12. Chapter 12 : Real-World Insights and Case Studies (Big Data / Fast Data)
    1. Choosing Partition Count and Replication Factor
    2. Kafka Topics Naming Convention
    3. Case Study – MovieFlix
    4. Case Study – GetTaxi
    5. Case Study – MySocialMedia
    6. Case Study – MyBank
    7. Case Study - Big Data Ingestion
    8. Case Study - Logging and Metrics Aggregation
  13. Chapter 13 : Kafka in the Enterprise for Admins
    1. Kafka Cluster Setup High-Level Architecture Overview
    2. Kafka Monitoring and Operations
    3. Kafka Security
    4. Kafka Multi-Cluster and MirrorMaker
    5. Advertised Listeners: Kafka Client and Server Communication Protocol
  14. Chapter 14 : Advanced Topics Configurations
    1. Changing a Topic Configuration
    2. Segment and Indexes
    3. Log Cleanup Policies
    4. Log Cleanup Delete
    5. Log Compaction Theory
    6. Log Compaction Practice
    7. Unclean Leader Election
    8. Large Messages in Kafka
  15. Chapter 15 : Next Steps
    1. What's Next?

Product information

  • Title: Apache Kafka Series - Learn Apache Kafka for Beginners v3
  • Author(s): Stéphane Maarek
  • Release date: May 2022
  • Publisher(s): Packt Publishing
  • ISBN: 9781789342604