Video description
Get hands-on with Kafka monitoring setup with Prometheus and Grafana, Kafka operations and Kafka cluster upgrades Setup in AWS.
About This Video
- Set up a multi-broker Kafka cluster in no time in AWS (using CloudFormation)
- Set up administration tools such as Kafka Manager, ZooNavigator, and LinkedIn's Kafka Monitor
- Set up monitoring using Grafana and Prometheus
In Detail
This course helps you learn Kafka administration, Kafka monitoring, Kafka operations, and Kafka upgrades. It covers the following topics:
- Administration for Apache Kafka
- Setup UI tools such as Kafka Manager, Zoo Navigator, and Kafka Monitor to get a full view of your cluster
- Understand basic operations you can perform with these tools Monitoring for Apache Kafka
- A Kafka that is not monitored is a ticking time-bomb.
- Set up proper monitoring for Kafka and Zookeeper
- Store your metrics in Prometheus and visualize them in Grafana
- Operations for Apache Kafka
- Perform the most common and difficult operations in your Apache Kafka cluster
- Roll-restart your brokers in only one command
- Update Kafka configurations safely with no downtime
- Rebalance partitions and change the replication factor of your topics
- Add, replace, and remove brokers.
- Upgrades for Apache Kafka
- Understand the subtlety of upgrading a Kafka cluster
- Step-by-step explanations on upgrading a cluster with no downtime
All the code files are placed at https://github.com/PacktPublishing/Apache-Kafka-Series---Kafka-Monitoring-and-Operations
Downloading the example code for this course: You can download the example code files for all Packt video courses you have purchased from your account at http://www.PacktPub.com. If you purchased this course elsewhere, you can visit http://www.PacktPub.com/support and register to have the files e-mailed directly to you.
Table of contents
-
Chapter 1 : Course Overview and Introduction
- Course Overview 00:07:10
-
Chapter 2 : Kafka Quick Setup in AWS
- What is AWS CloudFormation 00:01:32
- Target Architecture and Setup 00:04:19
- [Hands-On] Deploy Zookeeper Using CloudFormation 00:08:16
- [Hands-On] Deploy Kafka Using CloudFormation 00:07:38
- [Hands-On] Overview of our Zookeeper Cluster 00:09:26
- [Hands-On] Overview of our Kafka Cluster 00:05:03
- [Hands-On] Stopping the Kafka Cluster 00:02:50
-
Chapter 3 : Kafka Administration Setup
- [Hands-On] Overview and Setup of Monitoring Machine 00:08:10
- [Hands-On] Setup Zoonavigator 00:07:52
- [Hands-On] Setup Kafka Manager 00:08:08
- [Hands-On] Setup Kafka Monitor 00:09:28
-
Chapter 4 : Kafka Monitoring Setup: Grafana + Prometheus
- Overview of Monitoring 00:05:06
- [Hands-On] Setting up Prometheus + Kafka Broker 1 00:17:31
- [Solution] Broker 2 & 3 Prometheus Setup 00:05:21
- [Solution] Zookeeper & Prometheus Setup 00:09:40
- [Hands-On] Setup Grafana on Administration Server 00:06:49
- [Hands-On] Setup Kafka Dashboard on Grafana 00:05:44
- Important Metrics to monitor 00:06:05
- [Hands-On] Adding Graphs to Grafana 00:10:16
- [Solution] Adding more graphs to Grafana 00:03:01
- [Hands-On] Observing Grafana when we kill a broker 00:05:05
- Considerations to make the setup production-ready 00:02:00
-
Chapter 5 : Kafka Operations
- [Hands-On] Install Jolokia Agent on Broker 1 00:07:54
- [[Solution] Install Jolokia on Broker 2 & 3 00:03:38
- [Hands-On] Install Rolling Restart Script 00:14:04
- [Hands-On] Run Roll Restart Script 00:07:21
- [Hands-On] Updating Configurations 1st method (rolling restart) 00:03:45
- [Hands-On] Updating Configurations 2nd method (kafka-configs command) 00:11:19
- [Hands-On] Rebalancing Partitions using plain commands and JSON 00:12:10
- [Hands-On] Rebalancing Partitions using Kafka Manager 00:02:41
- [Hands-On] Rebalancing Partitions using LinkedIn tools 00:08:11
- [Hands-On] Increasing Replication Factor using plain commands and JSON 00:06:38
- [Hands-On] Increasing Replication Factor using LinkedIn tools 00:04:47
- [Hands-On] Adding Broker 4 using CloudFormation 00:09:55
- [Solution] Moving partitions over to Broker 4 using the CLI tools 00:04:30
- [Hands-On] Replacing a Broker keeping EBS Volume 00:07:03
- [Hands-On] Replacing a Broker losing an EBS Volume 00:03:41
- Removing a Broker Overview 00:01:01
- [Solution] Remove Broker 4 00:08:01
-
Chapter 6 : Kafka Cluster Upgrade
- Upgrading a Kafka Cluster Overview 00:02:20
- [Advanced] Zero Copy, Up and Down Conversions 00:08:06
- [Hands-On] Kafka Broker Upgrade Step 1 00:04:02
- [Hands-On] Kafka Broker Upgrade Step 2 00:02:58
- [Hands-On] Kafka Broker Upgrade Step 3 00:03:03
- [Hands-On] Kafka Broker Upgrade Step 4 00:01:40
- [Hands-On] Kafka Broker Upgrade Step 5 00:03:15
-
Chapter 7 : Next Steps
- Cleaning up AWS 00:01:17
- Next Steps & Congratulations! 00:02:02
Product information
- Title: Apache Kafka Series - Kafka Monitoring and Operations
- Author(s):
- Release date: January 2018
- Publisher(s): Packt Publishing
- ISBN: 9781838558765
You might also like
video
Kubernetes Microservices
Microservices designs require you to change how you build and deploy applications. Instead of creating a …
video
Clean Code
Expanded Edition (August 2018) Updated with Design Patterns episodes from the Clean Code series from Clean …
video
Apache Kafka Fundamentals LiveLessons
Almost 4 hours of video instruction to help you learn the key concepts and features behind …
video
Amazon Web Services AWS LiveLessons 2nd Edition
More Than 17 Hours of Video Instruction More than 17 hours of video instruction on Amazon …