Apache Kafka Complete Developer's Guide

Apache Kafka Complete Developer's Guide

by Bogdan Stashchuk
Released April 2020
Publisher(s): Packt Publishing
ISBN: 9781800208247

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Video description

Learn core Apache Kafka features along with creating Java, Node.js and Python producers and consumers

About This Video

  • Understand how Apache Kafka works and its core features
  • Build custom Apache Kafka producers and consumers using native Java API
  • Learn and practice how to run multiple brokers on the same computer

In Detail

Explore an interactive and easy way to learn Apache Kafka with this practical guide. Featuring interesting activities to help you build projects using APIs for programming languages such as Node.js and Python, this course will ensure you get hands-on experience.

You will learn how Apache Kafka works and also understand its core features. After delving into the architecture of Kafka, you will focus on how to build custom Apache Kafka producers and consumers using native Java API. The course will gradually assist you in working through a variety of activities related to Apache Kafka, from creating a cluster with multiple brokers and developing topics with replication factors, through to launching the performance monitor for testing the performance of consumers and producers. The course will also demonstrate how to use Apache Kafka API to create your own consumers and producers using Python and Node.js.

By the end of this course, you'll be well-versed with Apache Kafka and have the skills and confidence you need to build projects using it.

Table of contents

  1. Chapter 1 : Apache Kafka Installation Overview
    1. Apache Kafka Installation Overview 00:01:07
    2. Installing Apache Kafka on the Mac and Unix-like systems 00:00:52
    3. Installing Apache Kafka on the Mac 00:05:25
    4. Installing Ubuntu on MacOS using VirtualBox 00:12:41
  2. Chapter 2 : Installing Apache Kafka on the remote Ubuntu server
    1. SECTION 2 Introduction 00:00:59
    2. Creating remote Ubuntu Virtual Private Server 00:06:15
    3. Installing Apache Kafka on Virtual Private Server 00:07:04
  3. Chapter 3 : Installing Apache Kafka on Windows
    1. SECTION 3 Introduction 00:01:02
    2. Installing Apache Kafka on Windows 00:08:25
    3. Starting Zookeeper and Kafka server on Windows 00:06:31
    4. Installing Ubuntu on Windows using VirtualBox 00:11:38
    5. Installing Apache Kafka on Ubuntu using GUI 00:06:15
  4. Chapter 4 : Starting Apache Zookeeper and Kafka Broker
    1. SECTION 4 Introduction 00:01:02
    2. Observing contents of the Kafka folder 00:05:55
    3. Reading and editing Kafka files using VisualStudio Code 00:06:20
    4. Trying to start Kafka Server 00:04:10
    5. Observing Kafka Server logs 00:01:42
    6. Starting Zookeeper 00:03:12
    7. Starting Kafka Server while Zookeeper is up and running 00:06:12
    8. Observing logs folder and current kafka server setup 00:04:43
  5. Chapter 5 : Creating and exploring Kafka Topic
    1. SECTION 5 Introduction 00:00:30
    2. How to connect to Kafka cluster 00:02:18
    3. Create new Kafka topic 00:05:04
    4. What happened after creation of the new topic 00:03:36
    5. Read details about topic 00:05:21
  6. Chapter 6 : Producing and consuming Messages
    1. SECTION 6 Introduction 00:00:43
    2. Send some messages using Kafka Console Producer 00:03:13
    3. Consuming messages using Kafka Console Consumer 00:03:19
    4. Consuming messages from the beginning 00:01:35
    5. Running multiple consumers 00:01:30
    6. Running multiple producers 00:03:13
    7. What was changed in the Kafka logs 00:09:32
  7. Chapter 7 : What is Apache Kafka and how it works
    1. SECTION 7 Introduction 00:01:35
    2. What is Apache Kafka 00:04:01
    3. Broker 00:02:27
    4. Broker cluster 00:01:54
    5. Zookeeper 00:01:59
    6. Zookeeper ensemble 00:03:30
    7. Multiple Kafka clusters 00:02:19
    8. Default ports of Zookeeper and Broker 00:03:49
    9. Kafka Topic 00:02:32
    10. Message structure 00:03:33
    11. Topics and Partitions 00:04:33
    12. Spreading messages across partitions 00:05:28
    13. Partition Leader and Followers 00:06:23
    14. Controller and it's responsibilities 00:05:16
    15. How Producers write messages to the topic 00:02:15
    16. How Consumers read messages from the topic 00:03:33
  8. Chapter 8 : GitHub Repository and Diagrams for the course
    1. SECTION 8 Introduction 00:00:45
    2. GitHub repository and list of basic Kafka commands 00:07:32
    3. Diagrams for the course 00:03:43
  9. Chapter 9 : EXAMPLE 1 - Topic with Multiple Partitions
    1. SECTION 9 Introduction 00:00:53
    2. Cleaning up existing kafka installation 00:02:13
    3. Creating topic with multiple partitions 00:06:00
    4. How messages were spread across different partitions 00:06:19
    5. Reading messages from specific partition 00:02:45
    6. Reading messages from specific offset in specific partition 00:05:44
    7. Reading details about topic and __consumer_offsets topic 00:06:46
    8. Summary for multiple partitions example 00:01:57
  10. Chapter 10 : EXAMPLE 2 - Kafka Cluster with Multiple Brokers
    1. SECTION 10 Introduction 00:00:57
    2. Example overview - run multiple brokers 00:01:26
    3. Creating separate configuration files for brokers 00:05:34
    4. Launching three brokers 00:02:40
    5. Getting cluster information and broker details from Zookeeper 00:02:48
    6. Creating multiple-partition topic in the Kafka cluster 00:03:16
    7. Looking at logs folders of every broker 00:02:14
    8. Producing and consuming messages in the cluster 00:03:33
    9. Details about topic in the cluster 00:03:28
    10. Simulating broker failure in the cluster 00:05:55
    11. Summary for broker cluster and topic without replication 00:01:46
  11. Chapter 11 : EXAMPLE 3 - Multiple Brokers and Topic with Replication
    1. SECTION 11 Introduction 00:01:23
    2. Preparing for the next example with replication 00:02:42
    3. Launching brokers and creating topic with replication 00:04:41
    4. Observing logs folder and details of the topic 00:06:05
    5. Producing and consuming messages in the topic with replication 00:06:10
    6. Observing how messages were stored in the partitions on different brokers 00:03:05
    7. Bringing down one of three brokers and observing changes 00:03:33
    8. Bringing down another broker in the cluster 00:03:44
    9. Bringing back both brokers 00:01:57
    10. Summary for replication 00:01:36
  12. Chapter 12 : EXAMPLE 4 - Kafka Consumer Groups
    1. SECTION 12 Introduction 00:00:52
    2. Example with consumer groups overview 00:01:01
    3. Exploring default consumer groups 00:10:00
    4. Starting consumer in the custom consumer group 00:08:12
    5. Starting second consumer in the same consumer group 00:04:40
    6. Launching one more consumer in the same group 00:01:59
    7. Idle consumers in the group 00:05:52
    8. Summary for consumer groups 00:01:48
  13. Chapter 13 : EXAMPLE 5 - Performance Testing
    1. SECTION 13 Introduction 00:00:49
    2. Overview of the performance testing example 00:01:18
    3. Starting cluster and launching basic performance test 00:04:23
    4. Increasing performance test parameters 00:02:52
    5. Testing consumer performance 00:03:18
    6. Getting non-zero LAG values for consumers 00:05:18
    7. Performance test example summary 00:00:59
  14. Chapter 14 : PROJECT 1 - Java
    1. SECTION 14 Introduction 00:01:46
    2. Project Files for the Java section 00:01:06
    3. Installing IntelliJ editor 00:03:23
    4. Creating and configuring Maven project 00:05:41
    5. Starting Kafka Cluster 00:01:38
    6. Creating Java Producer 00:07:04
    7. Continue Creating Java Producer 00:07:07
    8. Launching Java Producer 00:05:05
    9. Explaining most common Producer parameters 00:12:04
    10. Modifying Serializer type 00:05:27
    11. Producing meaningful messages with delay 00:06:06
    12. Refactoring Producer by adding previous example 00:02:52
    13. Creating consumer with autocommitting - PART 1 00:07:48
    14. Creating consumer with autocommitting - PART 2 00:08:31
    15. Consumer parameters overview 00:06:24
    16. Consumer with Manual Committing 00:10:33
    17. Consumer with Partitions Assignment 00:05:36
    18. Launching multiple consumers in the same consumer group 00:07:41
    19. CHALLENGE - Subscribe vs Assign with consumer groups 00:01:54
  15. Chapter 15 : PROJECT 2 - Node.js
    1. SECTION 15 Introduction 00:00:55
    2. Installing Node.js with NPM 00:01:56
    3. Starting up Kafka cluster with 3 brokers 00:02:21
    4. Initializing Node.js project 00:01:37
    5. Final Node.js project filesv 00:01:15
    6. Creating basic Node.js producer 00:07:03
    7. Producing random animal names 00:06:42
    8. Creating Node.js consumer 00:04:54
  16. Chapter 16 : PROJECT 3 - Python
    1. SECTION 16 Introduction 00:00:44
    2. Installing Python 00:01:40
    3. Final Python project files 00:00:44
    4. Launching basic Python producer 00:05:36
    5. Launching consumer and receiving messages 00:03:36
    6. Generating fake names in the messages by producer 00:06:01
  17. Chapter 17 : Course Summary
    1. Course Summary 00:00:57

Product information

  • Title: Apache Kafka Complete Developer's Guide
  • Author(s): Bogdan Stashchuk
  • Release date: April 2020
  • Publisher(s): Packt Publishing
  • ISBN: 9781800208247