Apache Kafka Series - Learn Apache Kafka for Beginners v3

Apache Kafka Series - Learn Apache Kafka for Beginners v3

by Stéphane Maarek
Released May 2022
Publisher(s): Packt Publishing
ISBN: 9781789342604

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Video description

The high throughput and low latency of Apache Kafka have made it one of the leading distributed data-streaming enterprise technologies. It is used by many Fortune 500 companies such as Netflix, Airbnb, Uber, Walmart, and LinkedIn. If you want to develop Apache Kafka skills to stream data easily, then this course is for you.

The course starts by explaining the architecture of the Apache Kafka ecosystem, before going on to cover the core concepts of Kafka such as topics, partitions, brokers, replicas, producers, consumers, and more. Next, you will use native Kafka binaries to launch your own Kafka cluster on Windows, Mac OS X, and Linux.

As you advance, you will get hands-on experience with the Kafka command-line interface (CLI), along with an understanding of how to create producers and consumers in Java to interact with Kafka. Next, a real-world project using Wikimedia as a source of data for a producer and OpenSearch as a sink for our consumers. Moving on, you will get to grips with advanced APIs such as Kafka Connect and Kafka Streams and perform a case study on the real-world applications of Kafka. Finally, you will get an overview of advanced Kafka for administrators and understand advanced topic configurations.

By the end of this course, you will be well-versed with how Apache Kafka 3.0 plays an important role in data-streaming applications.

What You Will Learn

  • Understand the Apache Kafka ecosystem, architecture, core concepts, and operations
  • Start a personal Kafka development environment
  • Create producers and consumers in Java to interact with Kafka
  • Master concepts such as topics, partitions, brokers, producers, consumers
  • Discover how to change a topic configuration
  • Practice and understand log compaction

Audience

This course is for developers who want to learn the fundamentals of Apache Kafka, software architects who want to understand how Kafka fits into their solution architecture or anyone who is looking to understand how Apache Kafka works as a distributed system. Basic knowledge of Java and the Linux command line will be beneficial to effectively understand the concepts covered in the course.

About The Author

Stéphane Maarek: Stéphane Maarek is a solutions architect, consultant, and software developer who has a particular interest in all things related to big data and analytics. He is also a bestseller instructor on Udemy for his courses on Apache Kafka, Apache NiFi, and AWS Lambda. He loves Apache Kafka and regularly contributes to the Apache Kafka project.

Stéphane has also written a guest blog post that was featured on the Confluent website, the company behind Apache Kafka. He is also an AWS Certified Solutions Architect and has many years of experience with technologies such as Apache Kafka, Apache NiFi, Apache Spark, Hadoop, PostgreSQL, Tableau, Spotfire, Docker, Ansible, and more.

Table of contents

  1. Chapter 1 : Kafka Introduction
    1. Course Introduction
    2. Apache Kafka in Five Minutes
    3. Course Objectives
  2. Chapter 2 : Kafka Theory
    1. Topics, Partitions, and Offsets
    2. Producers and Message Keys
    3. Consumers and Deserialization
    4. Consumer Groups and Consumer Offsets
    5. Brokers and Topics
    6. Topic Replication
    7. Producer Acknowledgments and Topic Durability
    8. Zookeeper
    9. Kafka KRaft – Removing Zookeeper
    10. Theory Roundup
  3. Chapter 3 : Starting Kafka
    1. Important: Starting Kafka
    2. Starting Kafka with Conduktor – Multi-Platform
    3. Mac OS X - Download Kafka and Set Up PATH
    4. Mac OS X - Start Zookeeper and Kafka
    5. Mac OS X - Using Brew
    6. Linux - Download and Set Up Kafka in PATH
    7. Linux - Start Zookeeper and Kafka
    8. Windows WSL2 - Download Kafka and PATH Setup
    9. Windows WSL2 - Start Zookeeper and Kafka
    10. Windows WSL2 - How to Fix Problems
    11. Windows Non-WSL2 - Start Zookeeper and Kafka
  4. Chapter 4 : Starting Kafka without Zookeeper
    1. Mac OS X - Start Kafka in KRaft Mode
    2. Linux - Start Kafka in KRaft Mode
    3. Windows WSL2 - Start Kafka in KRaft Mode
  5. Chapter 5 : Command Line Interface (CLI) 101
    1. Command Line Interface (CLI) Introduction
    2. Kafka Topics Command Line Interface (CLI)
    3. Kafka Console Producer Command Line Interface (CLI)
    4. Kafka Console Consumer Command Line Interface (CLI)
    5. Kafka Consumers in a Group
    6. Kafka Consumer Groups Command Line Interface (CLI)
    7. Resetting Offsets
  6. Chapter 6 : Kafka UI
    1. Conduktor - Demo
  7. Chapter 7 : Kafka Java Programming 101
    1. Kafka SDK List
    2. Creating Kafka Project
    3. Java Producer
    4. Java Producer Callback
    5. Java Producer with Keys
    6. Java Consumer
    7. Java Consumer – Graceful Shutdown
    8. Java Consumer Inside a Consumer Group
    9. Java Consumer Incremental Cooperative Rebalance and Static Group Membership
    10. Java Consumer Incremental Cooperative Rebalance - Practice
    11. Java Consumer Auto Offset Commit Behavior
    12. Programming - Advanced Tutorials
  8. Chapter 8 : Kafka Real-World Project
    1. Real-World Project Overview
  9. Chapter 9 : Kafka Wikimedia Producer and Advanced Configurations
    1. IMPORTANT: Start Local Kafka with Conduktor Using Docker
    2. Wikimedia Producer Project Setup
    3. Wikimedia Producer Implementation
    4. Wikimedia Producer Run
    5. Wikimedia Producer - Producer Config Intros
    6. Producer Acknowledgements Deep Dive
    7. Producer Retries
    8. Idempotent Producer
    9. Safe Kafka Producer Settings
    10. Wikimedia Producer Safe Producer Implementation
    11. Kafka Message Compression
    12. linger.ms and batch.size Producer Settings
    13. Wikimedia Producer High Throughput Implementation
    14. Producer Default Partitioner and Sticky Partitioner
    15. [Advanced] max.block.ms and buffer.memory
  10. Chapter 10 : OpenSearch Consumer and Advanced Consumer Configurations
    1. OpenSearch Consumer - Project Overview
    2. OpenSearch Consumer - Project Setup
    3. Setting Up OpenSearch on Docker
    4. Setting Up OpenSearch on the Cloud
    5. OpenSearch 101
    6. OpenSearch Consumer Implementation - Part 1
    7. OpenSearch Consumer Implementation - Part 2
    8. Consumer Delivery Semantics
    9. OpenSearch Consumer Implementation Part 3 - Idempotence
    10. Consumer Offsets Commit Strategies
    11. OpenSearch Consumer Implementation Part 4 - Delivery Semantics
    12. OpenSearch Consumer Implementation Part 5 - Batching Data
    13. Consumer Offset Reset Behavior
    14. OpenSearch Consumer Implementation Part 6 - Replaying Data
    15. Consumer Internal Threads
    16. Consumer Replica Fetching - Rack Awareness
  11. Chapter 11 : Kafka Extended APIs for Developers
    1. Kafka Extended APIs - Overview
    2. Kafka Connect Introduction
    3. Kafka Connect Wikimedia and ElasticSearch Hands-On
    4. Kafka Streams Introduction
    5. Kafka Streams Hands-On
    6. Kafka Schema Registry Introduction
    7. Kafka Schema Registry Hands-On
    8. Which Kafka API Should I Use?
  12. Chapter 12 : Real-World Insights and Case Studies (Big Data / Fast Data)
    1. Choosing Partition Count and Replication Factor
    2. Kafka Topics Naming Convention
    3. Case Study – MovieFlix
    4. Case Study – GetTaxi
    5. Case Study – MySocialMedia
    6. Case Study – MyBank
    7. Case Study - Big Data Ingestion
    8. Case Study - Logging and Metrics Aggregation
  13. Chapter 13 : Kafka in the Enterprise for Admins
    1. Kafka Cluster Setup High-Level Architecture Overview
    2. Kafka Monitoring and Operations
    3. Kafka Security
    4. Kafka Multi-Cluster and MirrorMaker
    5. Advertised Listeners: Kafka Client and Server Communication Protocol
  14. Chapter 14 : Advanced Topics Configurations
    1. Changing a Topic Configuration
    2. Segment and Indexes
    3. Log Cleanup Policies
    4. Log Cleanup Delete
    5. Log Compaction Theory
    6. Log Compaction Practice
    7. Unclean Leader Election
    8. Large Messages in Kafka
  15. Chapter 15 : Next Steps
    1. What's Next?

Product information

  • Title: Apache Kafka Series - Learn Apache Kafka for Beginners v3
  • Author(s): Stéphane Maarek
  • Release date: May 2022
  • Publisher(s): Packt Publishing
  • ISBN: 9781789342604