Apache Kafka Series - Kafka Monitoring and Operations

Apache Kafka Series - Kafka Monitoring and Operations

by Stephane Maarek
Released January 2018
Publisher(s): Packt Publishing
ISBN: 9781838558765

Explore a preview version of Apache Kafka Series - Kafka Monitoring and Operations right now.

O’Reilly members get unlimited access to live online training experiences, plus books, videos, and digital content from 200+ publishers.

Start your free trial

Video description

Get hands-on with Kafka monitoring setup with Prometheus and Grafana, Kafka operations and Kafka cluster upgrades Setup in AWS.

About This Video

  • Set up a multi-broker Kafka cluster in no time in AWS (using CloudFormation)
  • Set up administration tools such as Kafka Manager, ZooNavigator, and LinkedIn's Kafka Monitor
  • Set up monitoring using Grafana and Prometheus

In Detail

This course helps you learn Kafka administration, Kafka monitoring, Kafka operations, and Kafka upgrades. It covers the following topics:

  • Administration for Apache Kafka
  • Setup UI tools such as Kafka Manager, Zoo Navigator, and Kafka Monitor to get a full view of your cluster
  • Understand basic operations you can perform with these tools Monitoring for Apache Kafka
  • A Kafka that is not monitored is a ticking time-bomb.
  • Set up proper monitoring for Kafka and Zookeeper
  • Store your metrics in Prometheus and visualize them in Grafana
  • Operations for Apache Kafka
  • Perform the most common and difficult operations in your Apache Kafka cluster
  • Roll-restart your brokers in only one command
  • Update Kafka configurations safely with no downtime
  • Rebalance partitions and change the replication factor of your topics
  • Add, replace, and remove brokers.
  • Upgrades for Apache Kafka
  • Understand the subtlety of upgrading a Kafka cluster
  • Step-by-step explanations on upgrading a cluster with no downtime

All the code files are placed at https://github.com/PacktPublishing/Apache-Kafka-Series---Kafka-Monitoring-and-Operations

Downloading the example code for this course: You can download the example code files for all Packt video courses you have purchased from your account at http://www.PacktPub.com. If you purchased this course elsewhere, you can visit http://www.PacktPub.com/support and register to have the files e-mailed directly to you.

Table of contents

  1. Chapter 1 : Course Overview and Introduction
    1. Course Overview 00:07:10
  2. Chapter 2 : Kafka Quick Setup in AWS
    1. What is AWS CloudFormation 00:01:32
    2. Target Architecture and Setup 00:04:19
    3. [Hands-On] Deploy Zookeeper Using CloudFormation 00:08:16
    4. [Hands-On] Deploy Kafka Using CloudFormation 00:07:38
    5. [Hands-On] Overview of our Zookeeper Cluster 00:09:26
    6. [Hands-On] Overview of our Kafka Cluster 00:05:03
    7. [Hands-On] Stopping the Kafka Cluster 00:02:50
  3. Chapter 3 : Kafka Administration Setup
    1. [Hands-On] Overview and Setup of Monitoring Machine 00:08:10
    2. [Hands-On] Setup Zoonavigator 00:07:52
    3. [Hands-On] Setup Kafka Manager 00:08:08
    4. [Hands-On] Setup Kafka Monitor 00:09:28
  4. Chapter 4 : Kafka Monitoring Setup: Grafana + Prometheus
    1. Overview of Monitoring 00:05:06
    2. [Hands-On] Setting up Prometheus + Kafka Broker 1 00:17:31
    3. [Solution] Broker 2 & 3 Prometheus Setup 00:05:21
    4. [Solution] Zookeeper & Prometheus Setup 00:09:40
    5. [Hands-On] Setup Grafana on Administration Server 00:06:49
    6. [Hands-On] Setup Kafka Dashboard on Grafana 00:05:44
    7. Important Metrics to monitor 00:06:05
    8. [Hands-On] Adding Graphs to Grafana 00:10:16
    9. [Solution] Adding more graphs to Grafana 00:03:01
    10. [Hands-On] Observing Grafana when we kill a broker 00:05:05
    11. Considerations to make the setup production-ready 00:02:00
  5. Chapter 5 : Kafka Operations
    1. [Hands-On] Install Jolokia Agent on Broker 1 00:07:54
    2. [[Solution] Install Jolokia on Broker 2 & 3 00:03:38
    3. [Hands-On] Install Rolling Restart Script 00:14:04
    4. [Hands-On] Run Roll Restart Script 00:07:21
    5. [Hands-On] Updating Configurations 1st method (rolling restart) 00:03:45
    6. [Hands-On] Updating Configurations 2nd method (kafka-configs command) 00:11:19
    7. [Hands-On] Rebalancing Partitions using plain commands and JSON 00:12:10
    8. [Hands-On] Rebalancing Partitions using Kafka Manager 00:02:41
    9. [Hands-On] Rebalancing Partitions using LinkedIn tools 00:08:11
    10. [Hands-On] Increasing Replication Factor using plain commands and JSON 00:06:38
    11. [Hands-On] Increasing Replication Factor using LinkedIn tools 00:04:47
    12. [Hands-On] Adding Broker 4 using CloudFormation 00:09:55
    13. [Solution] Moving partitions over to Broker 4 using the CLI tools 00:04:30
    14. [Hands-On] Replacing a Broker keeping EBS Volume 00:07:03
    15. [Hands-On] Replacing a Broker losing an EBS Volume 00:03:41
    16. Removing a Broker Overview 00:01:01
    17. [Solution] Remove Broker 4 00:08:01
  6. Chapter 6 : Kafka Cluster Upgrade
    1. Upgrading a Kafka Cluster Overview 00:02:20
    2. [Advanced] Zero Copy, Up and Down Conversions 00:08:06
    3. [Hands-On] Kafka Broker Upgrade Step 1 00:04:02
    4. [Hands-On] Kafka Broker Upgrade Step 2 00:02:58
    5. [Hands-On] Kafka Broker Upgrade Step 3 00:03:03
    6. [Hands-On] Kafka Broker Upgrade Step 4 00:01:40
    7. [Hands-On] Kafka Broker Upgrade Step 5 00:03:15
  7. Chapter 7 : Next Steps
    1. Cleaning up AWS 00:01:17
    2. Next Steps & Congratulations! 00:02:02

Product information

  • Title: Apache Kafka Series - Kafka Monitoring and Operations
  • Author(s): Stephane Maarek
  • Release date: January 2018
  • Publisher(s): Packt Publishing
  • ISBN: 9781838558765