Apache Kafka Series - Learn Apache Kafka for Beginners

Video description

Understand Apache Kafka and its configuration and learn how to apply the knowledge in real-time use cases

About This Video

  • Find out why Apache Kafka is an excellent choice as a distributed data-streaming platform
  • Explore Kafka concepts such as topics, partitions, brokers, producers, and consumers
  • Get to grips with Apache Kafka by programming a real-world Twitter producer and Elasticsearch consumer

In Detail

The high throughput and low latency of Apache Kafka have made it one of the leading distributed data-streaming enterprise technologies. It is used by many Fortune 500 companies such as Netflix, Airbnb, Uber, Walmart, and LinkedIn. If you want to develop the Apache Kafka skills to stream data easily, then this course is for you.

The course starts by explaining the architecture of the Apache Kafka ecosystem, before going on to cover the core concepts of Kafka such as topics, partitions, brokers, replicas, producers, consumers, and more. Next, you'll use native Kafka binaries to launch your own Kafka cluster on Windows, Mac OS X, and Linux. As you advance, you'll get hands-on experience with the Kafka command-line interface (CLI), along with understanding how to create producers and consumers in Java to interact with Kafka. Next, you'll build a project using Twitter as a source of data for a producer and set up Elasticsearch as a sink for the consumers. Moving on, you'll get to grips with advanced APIs such as Kafka Connect and Kafka Streams, and perform a case study on the real-world applications of Kafka. Finally, you'll get an overview of advanced Kafka for administrators and understand advanced topic configurations.

By the end of this course, you'll be well-versed with how Apache Kafka plays an important role in data-streaming applications.

Publisher resources

Download Example Code

Table of contents

  1. Chapter 1 : Kafka Introduction
    1. Apache Kafka in Five Minutes
    2. Course Objectives
  2. Chapter 2 : Kafka Theory
    1. Kafka Theory Overview
    2. Topics, Partitions, and Offsets
    3. Brokers and Topics
    4. Topic Replication
    5. Producer and Message Keys
    6. Consumers and Consumer Groups
    7. Consumer Offsets and Delivery Semantics
    8. Kafka Broker Discovery
    9. Zookeeper
    10. Kafka Guarantees
    11. Theory Roundup
  3. Chapter 3 : Starting Kafka
    1. Important: Starting Kafka
    2. Mac OS X - Download Kafka and Setup PATH
    3. Mac OS X - Using Brew
    4. Mac OS X - Start Zookeeper and Kafka
    5. Linux - Download Kafka and Setup PATH
    6. Linux - Start Zookeeper and Kafka
    7. Windows - Download Kafka and Setup PATH
    8. Windows - Start Zookeeper and Kafka
  4. Chapter 4 : Command Line Interface (CLI) 101
    1. Command Line Interface (CLI) Introduction
    2. Kafka Topics Command Line Interface (CLI)
    3. Kafka Console Producer Command Line Interface (CLI)
    4. Kafka Console Consumer Command Line Interface (CLI)
    5. Kafka Consumers in a Group
    6. Kafka Consumer Group Command Line Interface (CLI)
    7. Resetting Offsets
  5. Chapter 5 : Kafka Java Programming 101
    1. Intro to Kafka Programming
    2. Creating Kafka Project
    3. Java Producer
    4. Java Producer Callback
    5. Java Producer with Keys
    6. Java Consumer
    7. Java Consumer Inside a Consumer Group
    8. Java Consumer with Threads
    9. Java Consumer Seek and Assign
    10. Client Bidirectional Compatibility
  6. Chapter 6 : Kafka Real-World Project
    1. Real-World Project Overview
  7. Chapter 7 : Kafka Twitter Producer and Advanced Configurations
    1. Producer and Advanced Configurations Overview
    2. Twitter Setup
    3. Producer Part 1 - Writing Twitter Client
    4. Producer Part 2 - Writing the Kafka Producer
    5. Producer Configurations Introduction
    6. Acks and Min.insync.replicas
    7. Retries and Max.in.flight.requests.per.connection
    8. Idempotent Producer
    9. Producer Part 3 - Safe Producer
    10. Producer Compression
    11. Producer Batching
    12. Producer Part 4 - High Throughput Producer
    13. Producer Default Partitions and Key Hashing
    14. Max.block.ms and Buffer.memory
    15. Refactoring the Project
  8. Chapter 8 : Kafka Elasticsearch Consumer and Advanced Configurations
    1. Consumer and Advanced Configuration Overview
    2. Setting up Elasticsearch in the Cloud
    3. Elasticsearch 101
    4. Consumer Part 1 - Setup Project
    5. Consumer Part 2 - Write the Consumer and Send to Elasticsearch
    6. Delivery Semantics for Consumers
    7. Consumer Part 3 – Idempotence
    8. Consumer Poll Behaviour
    9. Consumer Offset Commit Strategies
    10. Consumer Part 4 - Manual Commit of Offsets
    11. Consumer Part 5 - Performance Improvement using Batching
    12. Consumer Offsets Reset Behaviour
    13. Consumer Part 6 - Replaying Data
    14. Consumer Internal Threads
  9. Chapter 9 : Kafka Ecosystem and Real-World Architectures
    1. Kafka in the Real World
  10. Chapter 10 : Kafka Extended APIs for Developers
    1. Kafka Connect Introduction
    2. Kafka Connect Twitter Hands-On
    3. Kafka Streams Introduction
    4. Kafka Streams Hands-On
    5. Kafka Schema Registry Introduction
  11. Chapter 11 : Real World Insights and Case Studies (Big Data / Fast Data)
    1. Choosing Partition Count and Replication Factor
    2. Case Study – MovieFlix
    3. Case Study – GetTaxi
    4. Case Study – MySocialMedia
    5. Case Study – MyBank
    6. Case Study - Big Data Ingestion
    7. Case Study - Logging and Metrics Aggregation
  12. Chapter 12 : Kafka in the Enterprise for Admins
    1. Kafka Cluster Setup High-Level Architecture Overview
    2. Kafka Monitoring and Operations
    3. Kafka Security
    4. Kafka Multi-Cluster and MirrorMaker
  13. Chapter 13 : Advanced Topics Configurations
    1. Changing a Topic Configuration
    2. Segment and Indexes
    3. Log Cleanup Policies
    4. Log Cleanup Delete
    5. Log Compaction Theory
    6. Log Compaction Practice
    7. Min.insync.replicas Reminder
    8. Unclean Leader Election
  14. Chapter 14 : Annexes
    1. What are Annexes?
  15. Chapter 15 : Annex - Starting Kafka Differently
    1. Annex 1 – Overview
    2. Starting Kafka with the Confluent Command Line Interface (CLI)
    3. Starting a Multi-broker Kafka Cluster Using Binaries
    4. Start Kafka Development Environment Using Docker
    5. Starting a Multi-broker Kafka Cluster Using Docker
    6. Kafka Advertised Host Setting
    7. Starting Kafka on a Remote Machine
  16. Chapter 16 : Next Steps
    1. What's Next?

Product information

  • Title: Apache Kafka Series - Learn Apache Kafka for Beginners
  • Author(s): Stéphane Maarek
  • Release date: May 2018
  • Publisher(s): Packt Publishing
  • ISBN: 9781789342604