Apache Kafka 1.0 Cookbook

Apache Kafka 1.0 Cookbook

by Raul Estrada
Released December 2017
Publisher(s): Packt Publishing
ISBN: 9781787286849

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Simplify real-time data processing by leveraging the power of Apache Kafka 1.0

About This Book

  • Use Kafka 1.0 features such as Confluent platforms and Kafka streams to build efficient streaming data applications to handle and process your data
  • Integrate Kafka with other Big Data tools such as Apache Hadoop, Apache Spark, and more
  • Hands-on recipes to help you design, operate, maintain, and secure your Apache Kafka cluster with ease

Who This Book Is For

This book is for developers and Kafka administrators who are looking for quick, practical solutions to problems encountered while operating, managing or monitoring Apache Kafka. If you are a developer, some knowledge of Scala or Java will help, while for administrators, some working knowledge of Kafka will be useful.

What You Will Learn

  • Install and configure Apache Kafka 1.0 to get optimal performance
  • Create and configure Kafka Producers and Consumers
  • Operate your Kafka clusters efficiently by implementing the mirroring technique
  • Work with the new Confluent platform and Kafka streams, and achieve high availability with Kafka
  • Monitor Kafka using tools such as Graphite and Ganglia
  • Integrate Kafka with third-party tools such as Elasticsearch, Logstash, Apache Hadoop, Apache Spark, and more

In Detail

Apache Kafka provides a unified, high-throughput, low-latency platform to handle real-time data feeds. This book will show you how to use Kafka efficiently, and contains practical solutions to the common problems that developers and administrators usually face while working with it.

This practical guide contains easy-to-follow recipes to help you set up, configure, and use Apache Kafka in the best possible manner. You will use Apache Kafka Consumers and Producers to build effective real-time streaming applications. The book covers the recently released Kafka version 1.0, the Confluent Platform and Kafka Streams. The programming aspect covered in the book will teach you how to perform important tasks such as message validation, enrichment and composition.Recipes focusing on optimizing the performance of your Kafka cluster, and integrate Kafka with a variety of third-party tools such as Apache Hadoop, Apache Spark, and Elasticsearch will help ease your day to day collaboration with Kafka greatly. Finally, we cover tasks related to monitoring and securing your Apache Kafka cluster using tools such as Ganglia and Graphite.

If you're looking to become the go-to person in your organization when it comes to working with Apache Kafka, this book is the only resource you need to have.

Style and approach

Following a cookbook recipe-based approach, we'll teach you how to solve everyday difficulties and struggles you encounter using Kafka through hands-on examples.

Table of contents

  1. Preface
    1. What this book covers
    2. What you need for this book
    3. Who this book is for
    4. Sections
      1. Getting ready
      2. How to do it…
      3. How it works…
      4. There's more…
      5. See also
    5. Conventions
    6. Reader feedback
    7. Customer support
      1. Downloading the example code
      2. Downloading the color images of this book
      3. Errata
      4. Piracy
      5. Questions
  2. Configuring Kafka
    1. Introduction
    2. Installing Kafka
      1. Getting ready
      2. How to do it...
        1. Installing Java in Linux
        2.  Installing Scala in Linux
        3. Installing Kafka in Linux
      3. There's more...
      4. See also
    3. Running Kafka
      1. Getting ready
      2. How to do it...
      3. There's more...
      4. See also
    4. Configuring Kafka brokers
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    5. Configuring Kafka topics
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more…
    6. Creating a message console producer
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more…
    7. Creating a message console consumer
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
    8. Configuring the broker settings
      1. Getting ready
      2. How to do it...
      3. How it works…
      4. There's more…
    9. Configuring threads and performance
      1. Getting ready
      2. How to do it...
      3. How it works…
      4. There's more...
    10. Configuring the log settings
      1. Getting ready
      2. How to do it...
      3. How it works…
      4. There's more…
      5. See also
    11. Configuring the replica settings
      1. Getting ready
      2. How to do it...
      3. How it works…
      4. There's more...
    12. Configuring the ZooKeeper settings
      1. Getting ready
      2. How to do it…
      3. How it works…
      4. See also
    13. Configuring other miscellaneous parameters
      1. Getting ready
      2. How to do it...
      3. How it works…
      4. See also
  3. Kafka Clusters
    1. Introduction
    2. Configuring a single-node single-broker cluster – SNSB
      1. Getting ready
      2. How to do it...
        1. Starting ZooKeeper
        2. Starting the broker
      3. How it works...
      4. There's more...
      5. See also
    3. SNSB – creating a topic, producer, and consumer
      1. Getting ready
      2. How to do it...
        1. Creating a topic
        2. Starting the producer
        3. Starting the consumer
      3. How it works...
      4. There's more...
    4. Configuring a single-node multiple-broker cluster – SNMB
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    5. SNMB – creating a topic, producer, and consumer
      1. Getting ready
      2. How to do it...
        1. Creating a topic
        2. Starting a producer
        3. Starting a consumer
      3. How it works...
      4. There's more...
      5. See also
    6. Configuring a multiple-node multiple-broker cluster – MNMB
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
  4. Message Validation
    1. Introduction
    2. Modeling the events
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    3. Setting up the project
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    4. Reading from Kafka
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    5. Writing to Kafka
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    6. Running ProcessingApp
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    7. Coding the validator
      1. Getting ready
      2. How to do it...
      3. There's more...
      4. See also
    8. Running the validator
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
  5. Message Enrichment
    1. Introduction
    2. Geolocation extractor
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    3. Geolocation enricher
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    4. Currency price extractor
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    5. Currency price enricher
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    6. Running the currency price enricher
      1. Getting ready
      2. How to do it...
      3. How it works...
    7. Modeling the events
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    8. Setting up the project
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    9. Open weather extractor
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    10. Location temperature enricher
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    11. Running the location temperature enricher
      1. Getting ready
      2. How to do it...
      3. How it works...
  6. The Confluent Platform
    1. Introduction
    2. Installing the Confluent Platform
      1. Getting ready
      2. How to do it...
      3. There's more...
      4. See also
    3. Using Kafka operations
      1. Getting ready
      2. How to do it...
      3. There's more...
      4. See also
    4. Monitoring with the Confluent Control Center
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
    5. Using the Schema Registry
      1. Getting ready
      2. How to do it...
      3. See also
    6. Using the Kafka REST Proxy
      1. Getting ready
      2. How to do it...
      3. There's more...
      4. See also
    7. Using Kafka Connect
      1. Getting ready
      2. How to do it...
      3. There's more...
      4. See also
  7. Kafka Streams
    1. Introduction
    2. Setting up the project
      1. Getting ready
      2. How to do it...
      3. How it works...
    3. Running the streaming application
      1. Getting ready
      2. How to do it...
  8. Managing Kafka
    1. Introduction
    2. Managing consumer groups
      1. Getting ready
      2. How to do it...
      3. How it works...
    3. Dumping log segments
      1. Getting ready
      2. How to do it...
      3. How it works...
    4. Importing ZooKeeper offsets
      1. Getting ready
      2. How to do it...
      3. How it works...
    5. Using the GetOffsetShell
      1. Getting ready
      2. How to do it...
      3. How it works...
    6. Using the JMX tool
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
    7. Using the MirrorMaker tool
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    8. Replaying log producer
      1. Getting ready
      2. How to do it...
      3. How it works...
    9. Using state change log merger
      1. Getting ready
      2. How to do it...
      3. How it works...
  9. Operating Kafka
    1. Introduction
    2. Adding or removing topics
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    3. Modifying message topics
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    4. Implementing a graceful shutdown
      1. Getting ready
      2. How to do it...
      3. How it works...
    5. Balancing leadership
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
    6. Expanding clusters
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
    7. Increasing the replication factor
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
    8. Decommissioning brokers
      1. Getting ready
      2. How to do it...
      3. How it works...
    9. Checking the consumer position
      1. Getting ready
      2. How to do it...
      3. How it works...
  10. Monitoring and Security
    1. Introduction
    2. Monitoring server statistics
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    3. Monitoring producer statistics
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    4. Monitoring consumer statistics
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    5. Connecting with the help of Graphite
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    6. Monitoring with the help of Ganglia
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    7. Implementing authentication using SSL
      1. How to do it...
      2. See also
    8. Implementing authentication using SASL/Kerberos
      1. How to do it...
      2. See also
  11. Third-Party Tool Integration
    1. Introduction
    2. Moving data between Kafka nodes with Flume
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    3. Writing to an HDFS cluster with Gobblin
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    4. Moving data from Kafka to Elastic with Logstash
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    5. Connecting Spark streams and Kafka
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
    6. Ingesting data from Kafka to Storm
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    7. Pushing data from Kafka to Elastic
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    8. Inserting data from Kafka to SolrCloud
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    9. Building a Kafka producer with Akka
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
    10. Building a Kafka consumer with Akka
      1. Getting ready
      2. How to do it...
    11. Storing data in Cassandra
      1. Getting ready
      2. How to do it...
      3. How it works...
    12. Running Kafka on Mesos
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
    13. Reading Kafka with Apache Beam
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    14. Writing to Kafka from Apache Beam
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also

Product information

  • Title: Apache Kafka 1.0 Cookbook
  • Author(s): Raul Estrada
  • Release date: December 2017
  • Publisher(s): Packt Publishing
  • ISBN: 9781787286849