Apache Kafka Series - Kafka Monitoring and Operations

Apache Kafka Series - Kafka Monitoring and Operations

by Stephane Maarek
Released January 2018
Publisher(s): Packt Publishing
ISBN: 9781838558765

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to live online training experiences, plus books, videos, and digital content from O’Reilly and nearly 200 trusted publishing partners.

Start your free trial

Video description

Get hands-on with Kafka monitoring setup with Prometheus and Grafana, Kafka operations and Kafka cluster upgrades Setup in AWS.

About This Video

  • Set up a multi-broker Kafka cluster in no time in AWS (using CloudFormation)
  • Set up administration tools such as Kafka Manager, ZooNavigator, and LinkedIn's Kafka Monitor
  • Set up monitoring using Grafana and Prometheus

In Detail

This course helps you learn Kafka administration, Kafka monitoring, Kafka operations, and Kafka upgrades. It covers the following topics:

  • Administration for Apache Kafka
  • Setup UI tools such as Kafka Manager, Zoo Navigator, and Kafka Monitor to get a full view of your cluster
  • Understand basic operations you can perform with these tools Monitoring for Apache Kafka
  • A Kafka that is not monitored is a ticking time-bomb.
  • Set up proper monitoring for Kafka and Zookeeper
  • Store your metrics in Prometheus and visualize them in Grafana
  • Operations for Apache Kafka
  • Perform the most common and difficult operations in your Apache Kafka cluster
  • Roll-restart your brokers in only one command
  • Update Kafka configurations safely with no downtime
  • Rebalance partitions and change the replication factor of your topics
  • Add, replace, and remove brokers.
  • Upgrades for Apache Kafka
  • Understand the subtlety of upgrading a Kafka cluster
  • Step-by-step explanations on upgrading a cluster with no downtime

All the code files are placed at https://github.com/PacktPublishing/Apache-Kafka-Series---Kafka-Monitoring-and-Operations

Downloading the example code for this course: You can download the example code files for all Packt video courses you have purchased from your account at http://www.PacktPub.com. If you purchased this course elsewhere, you can visit http://www.PacktPub.com/support and register to have the files e-mailed directly to you.

Table of contents

  1. Chapter 1 : Course Overview and Introduction
    1. Course Overview 00:07:10
  2. Chapter 2 : Kafka Quick Setup in AWS
    1. What is AWS CloudFormation 00:01:32
    2. Target Architecture and Setup 00:04:19
    3. [Hands-On] Deploy Zookeeper Using CloudFormation 00:08:16
    4. [Hands-On] Deploy Kafka Using CloudFormation 00:07:38
    5. [Hands-On] Overview of our Zookeeper Cluster 00:09:26
    6. [Hands-On] Overview of our Kafka Cluster 00:05:03
    7. [Hands-On] Stopping the Kafka Cluster 00:02:50
  3. Chapter 3 : Kafka Administration Setup
    1. [Hands-On] Overview and Setup of Monitoring Machine 00:08:10
    2. [Hands-On] Setup Zoonavigator 00:07:52
    3. [Hands-On] Setup Kafka Manager 00:08:08
    4. [Hands-On] Setup Kafka Monitor 00:09:28
  4. Chapter 4 : Kafka Monitoring Setup: Grafana + Prometheus
    1. Overview of Monitoring 00:05:06
    2. [Hands-On] Setting up Prometheus + Kafka Broker 1 00:17:31
    3. [Solution] Broker 2 & 3 Prometheus Setup 00:05:21
    4. [Solution] Zookeeper & Prometheus Setup 00:09:40
    5. [Hands-On] Setup Grafana on Administration Server 00:06:49
    6. [Hands-On] Setup Kafka Dashboard on Grafana 00:05:44
    7. Important Metrics to monitor 00:06:05
    8. [Hands-On] Adding Graphs to Grafana 00:10:16
    9. [Solution] Adding more graphs to Grafana 00:03:01
    10. [Hands-On] Observing Grafana when we kill a broker 00:05:05
    11. Considerations to make the setup production-ready 00:02:00
  5. Chapter 5 : Kafka Operations
    1. [Hands-On] Install Jolokia Agent on Broker 1 00:07:54
    2. [[Solution] Install Jolokia on Broker 2 & 3 00:03:38
    3. [Hands-On] Install Rolling Restart Script 00:14:04
    4. [Hands-On] Run Roll Restart Script 00:07:21
    5. [Hands-On] Updating Configurations 1st method (rolling restart) 00:03:45
    6. [Hands-On] Updating Configurations 2nd method (kafka-configs command) 00:11:19
    7. [Hands-On] Rebalancing Partitions using plain commands and JSON 00:12:10
    8. [Hands-On] Rebalancing Partitions using Kafka Manager 00:02:41
    9. [Hands-On] Rebalancing Partitions using LinkedIn tools 00:08:11
    10. [Hands-On] Increasing Replication Factor using plain commands and JSON 00:06:38
    11. [Hands-On] Increasing Replication Factor using LinkedIn tools 00:04:47
    12. [Hands-On] Adding Broker 4 using CloudFormation 00:09:55
    13. [Solution] Moving partitions over to Broker 4 using the CLI tools 00:04:30
    14. [Hands-On] Replacing a Broker keeping EBS Volume 00:07:03
    15. [Hands-On] Replacing a Broker losing an EBS Volume 00:03:41
    16. Removing a Broker Overview 00:01:01
    17. [Solution] Remove Broker 4 00:08:01
  6. Chapter 6 : Kafka Cluster Upgrade
    1. Upgrading a Kafka Cluster Overview 00:02:20
    2. [Advanced] Zero Copy, Up and Down Conversions 00:08:06
    3. [Hands-On] Kafka Broker Upgrade Step 1 00:04:02
    4. [Hands-On] Kafka Broker Upgrade Step 2 00:02:58
    5. [Hands-On] Kafka Broker Upgrade Step 3 00:03:03
    6. [Hands-On] Kafka Broker Upgrade Step 4 00:01:40
    7. [Hands-On] Kafka Broker Upgrade Step 5 00:03:15
  7. Chapter 7 : Next Steps
    1. Cleaning up AWS 00:01:17
    2. Next Steps & Congratulations! 00:02:02

Product information

  • Title: Apache Kafka Series - Kafka Monitoring and Operations
  • Author(s): Stephane Maarek
  • Release date: January 2018
  • Publisher(s): Packt Publishing
  • ISBN: 9781838558765