Apache Kafka Series - Confluent Schema Registry and REST Proxy

Apache Kafka Series - Confluent Schema Registry and REST Proxy

by Stéphane Maarek
Released May 2018
Publisher(s): Packt Publishing
ISBN: 9781789344806

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Video description

This is the new volume in the Apache Kafka Series! Learn Apache Avro, the confluent schema registry for Apache Kafka and the confluent REST proxy for Apache Kafka. Join hundreds of knowledge savvy students in learning some of the most important components in a typical Apache Kafka stack. Apache Avro is one of the most powerful and most popular fast data serialisation mechanism with Apache Kafka: Schema is a first class citizen of the data; data serialisation format that is lightweight; documentation is embedded in the schema; easy to read and write using many languages; fast and big data de-facto data serialisation format. Confluent schema registry for Apache Kafka is the de-facto standard way of storing Avro schemas for your Apache Kafka topics: Stores a versioned history of all your schemas in Apache Kafka; supports and enforces conditions for schema evolution (backward, forward, full compatibility): Kafka Avro serialiser and deserialiser automatically integrate with Apache Kafka and confluent schema registry; make your messages much smaller and lighter! Perfect for increasing Apache Kafka throughput and performance. Confluent REST Proxy is the perfect way to communicate for sending Avro data using non Java languages to Apache Kafka: Write and read binary, JSON and Avro data to Apache Kafka using an HTTP REST API; interact with Apache Kafka using any programming language (not just Java); consult topic list and topic metadata in Apache Kafka

What You Will Learn

  • Write simple and complex Avro schemas
  • Create, write and read Avro objects in Java
  • Write a Java producer and consumer leveraging Avro data and the schema registry
  • Learn about schema evolution
  • Perform schema evolution using the command line and in Java
  • Utilize the REST proxy using a REST client

Audience

Developers who would like to learn how to write and run an application that leverages Avro and the schema registry. Architects who want to understand the role of the schema registry and REST proxy in an enterprise pipeline. This course is intended for people who fully understand the basics of Kafka. People who want to learn the confluent components: schema registry and REST proxy

About The Author

Stéphane Maarek: Stéphane Maarek is a solutions architect, consultant, and software developer who has a particular interest in all things related to big data and analytics. He is also a bestseller instructor on Udemy for his courses on Apache Kafka, Apache NiFi, and AWS Lambda. He loves Apache Kafka and regularly contributes to the Apache Kafka project.

Stéphane has also written a guest blog post that was featured on the Confluent website, the company behind Apache Kafka. He is also an AWS Certified Solutions Architect and has many years of experience with technologies such as Apache Kafka, Apache NiFi, Apache Spark, Hadoop, PostgreSQL, Tableau, Spotfire, Docker, Ansible, and more.

Table of contents

  1. Chapter 1 : Course Introduction
    1. The need of schemas in Kafka
    2. Course Structure Objectives
    3. About your instructor
    4. Architecture for Kafka with the Schema Registry and REST Proxy
  2. Chapter 2 : Avro Schemas
    1. What is Avro?
    2. Avro Primitive Types
    3. Avro Record Schema Definition
    4. Avro Complex Types
    5. Practice Exercise: Customer CustomerAddress
    6. Avro Logical Types
    7. The complex case of Decimals
    8. Avro Section Summary
  3. Chapter 3 : Avro in Java
    1. Section Objective
    2. Generic Record in Avro - Hands On
    3. Specific Record in Avro - Hands On
    4. Check-in on now vs later in Kafka
    5. Avro Tools - Hands On
    6. Reflection in Avro - Hands On
    7. Schema Evolution - Theory
    8. Schema Evolution - Hands On
    9. Avro Java Section Summary
  4. Chapter 4 : Setup and Launch Kafka
    1. Docker on Mac (recent versions)
    2. Docker Toolbox on Mac (older versions)
    3. Docker on Linux (Ubuntu as an example)
    4. Docker on Windows 10 64bit
    5. Docker Toolbox on Windows (older versions)
    6. Starting Kafka using Docker Compose
  5. Chapter 5 : Confluent Schema Registry and Kafka
    1. Confluent Schema Registry
    2. Kafka Avro Console Producer Consumer
    3. Writing a Kafka Avro Producer in Java
    4. Writing a Kafka Avro Consumer in Java
    5. Reminder on Schema Evolution
    6. Writing a V2 Kafka Producer
    7. Writing a V2 Kafka Consumer
    8. Summary on Compatibility Changes
    9. Kafka Schema Registry Deep Dive
    10. Managing Schemas Efficiently Section Summary
  6. Chapter 6 : Confluent REST Proxy
    1. Kafka REST Proxy Introduction and Purpose
    2. V1 vs V2 APIs
    3. Insomnia Setup (REST Client)
    4. Topic Operations
    5. Producing in Binary with the Kafka REST Proxy
    6. Consuming in Binary with the Kafka REST Proxy
    7. Producing in JSON with the Kafka REST Proxy
    8. Consuming in JSON with the Kafka REST Proxy
    9. Producing in Avro with the Kafka REST Proxy
    10. Consuming in Avro with the Kafka REST Proxy
  7. Chapter 7 : Annexes
    1. Full Avro End to End: Kafka Producer + Kafka Connect + Kafka Streams
    2. Kafka REST Proxy Installation and Scaling - Overview

Product information

  • Title: Apache Kafka Series - Confluent Schema Registry and REST Proxy
  • Author(s): Stéphane Maarek
  • Release date: May 2018
  • Publisher(s): Packt Publishing
  • ISBN: 9781789344806