Apache Kafka Series - Kafka Monitoring and Operations

Apache Kafka Series - Kafka Monitoring and Operations

by Stephane Maarek
Released January 2018
Publisher(s): Packt Publishing
ISBN: 9781838558765

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Video description

This course helps you learn Kafka administration, Kafka monitoring, Kafka operations, and Kafka upgrades. It covers the following topics:

Administration for Apache Kafka

o Setup UI tools such as Kafka Manager, Zoo Navigator, and Kafka Monitor to get a full view of your cluster

o Understand basic operations you can perform with these tools

Monitoring for Apache Kafka

o A Kafka that is not monitored is a ticking time-bomb.

o Set up proper monitoring for Kafka and Zookeeper

o Store your metrics in Prometheus and visualize them in Grafana

Operations for Apache Kafka

o Perform the most common and difficult operations in your Apache Kafka cluster

o Roll-restart your brokers in only one command

o Update Kafka configurations safely with no downtime

o Rebalance partitions and change the replication factor of your topics

o Add, replace, and remove brokers.

Upgrades for Apache Kafka

o Understand the subtlety of upgrading a Kafka cluster

o Step-by-step explanations on upgrading a cluster with no downtime

What You Will Learn

  • Learn how to perform a safe and automated roll-restart of Kafka brokers
  • Update broker configurations in a safe way
  • Rebalance partitions in a Kafka cluster
  • Increase and decrease the replication factor of topics
  • Add a broker to a Kafka cluster
  • Service and replace a broker in a Kafka cluster
  • Remove a broker in a Kafka cluster
  • Install Command Line Interface (CLI) tools to automate workflows
  • Upgrade a Kafka cluster with no downtime

Audience

Kafka administrators who want to learn how to set up monitoring, perform Kafka operations, and upgrade a Kafka cluster.

About The Author

Stéphane Maarek: Stéphane Maarek is a solutions architect, consultant, and software developer who has a particular interest in all things related to big data and analytics. He is also a bestseller instructor on Udemy for his courses on Apache Kafka, Apache NiFi, and AWS Lambda. He loves Apache Kafka and regularly contributes to the Apache Kafka project.

Stéphane has also written a guest blog post that was featured on the Confluent website, the company behind Apache Kafka. He is also an AWS Certified Solutions Architect and has many years of experience with technologies such as Apache Kafka, Apache NiFi, Apache Spark, Hadoop, PostgreSQL, Tableau, Spotfire, Docker, Ansible, and more.

Table of contents

  1. Chapter 1 : Course Overview and Introduction
    1. Course Overview
  2. Chapter 2 : Kafka Quick Setup in AWS
    1. What is AWS CloudFormation
    2. Target Architecture and Setup
    3. [Hands-On] Deploy Zookeeper Using CloudFormation
    4. [Hands-On] Deploy Kafka Using CloudFormation
    5. [Hands-On] Overview of our Zookeeper Cluster
    6. [Hands-On] Overview of our Kafka Cluster
    7. [Hands-On] Stopping the Kafka Cluster
  3. Chapter 3 : Kafka Administration Setup
    1. [Hands-On] Overview and Setup of Monitoring Machine
    2. [Hands-On] Setup Zoonavigator
    3. [Hands-On] Setup Kafka Manager
    4. [Hands-On] Setup Kafka Monitor
  4. Chapter 4 : Kafka Monitoring Setup: Grafana + Prometheus
    1. Overview of Monitoring
    2. [Hands-On] Setting up Prometheus + Kafka Broker 1
    3. [Solution] Broker 2 3 Prometheus Setup
    4. [Solution] Zookeeper Prometheus Setup
    5. [Hands-On] Setup Grafana on Administration Server
    6. [Hands-On] Setup Kafka Dashboard on Grafana
    7. Important Metrics to monitor
    8. [Hands-On] Adding Graphs to Grafana
    9. [Solution] Adding more graphs to Grafana
    10. [Hands-On] Observing Grafana when we kill a broker
    11. Considerations to make the setup production-ready
  5. Chapter 5 : Kafka Operations
    1. [Hands-On] Install Jolokia Agent on Broker 1
    2. [[Solution] Install Jolokia on Broker 2 3
    3. [Hands-On] Install Rolling Restart Script
    4. [Hands-On] Run Roll Restart Script
    5. [Hands-On] Updating Configurations 1st method (rolling restart)
    6. [Hands-On] Updating Configurations 2nd method (kafka-configs command)
    7. [Hands-On] Rebalancing Partitions using plain commands and JSON
    8. [Hands-On] Rebalancing Partitions using Kafka Manager
    9. [Hands-On] Rebalancing Partitions using LinkedIn tools
    10. [Hands-On] Increasing Replication Factor using plain commands and JSON
    11. [Hands-On] Increasing Replication Factor using LinkedIn tools
    12. [Hands-On] Adding Broker 4 using CloudFormation
    13. [Solution] Moving partitions over to Broker 4 using the CLI tools
    14. [Hands-On] Replacing a Broker keeping EBS Volume
    15. [Hands-On] Replacing a Broker losing an EBS Volume
    16. Removing a Broker Overview
    17. [Solution] Remove Broker 4
  6. Chapter 6 : Kafka Cluster Upgrade
    1. Upgrading a Kafka Cluster Overview
    2. [Advanced] Zero Copy, Up and Down Conversions
    3. [Hands-On] Kafka Broker Upgrade Step 1
    4. [Hands-On] Kafka Broker Upgrade Step 2
    5. [Hands-On] Kafka Broker Upgrade Step 3
    6. [Hands-On] Kafka Broker Upgrade Step 4
    7. [Hands-On] Kafka Broker Upgrade Step 5
  7. Chapter 7 : Next Steps
    1. Cleaning up AWS
    2. Next Steps Congratulations!

Product information

  • Title: Apache Kafka Series - Kafka Monitoring and Operations
  • Author(s): Stephane Maarek
  • Release date: January 2018
  • Publisher(s): Packt Publishing
  • ISBN: 9781838558765