O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Apache Kafka Series - Kafka Connect Hands-on Learning

Apache Kafka Series - Kafka Connect Hands-on Learning

by Stéphane Maarek
Publisher: Packt Publishing
Release Date: May 2018
ISBN: 9781789344738
View table of contents
Start watching

Video Description

Kafka Connect - Learn How to Source Twitter Data, Store in Apache Kafka Topics and Sink in ElasticSearch and PostgreSQL

About This Video

  • In this course, you will learn what is Kafka Connect, Kafka Connect architecture, how to deploy an Apache Kafka Connector in standalone and in distributed modes.
  • How to set up and launch a Kafka Connect cluster using Docker Compose.
  • You will be able to deploy Kafka Connect source connectors and Kafka Connect sinks connectors at the end of this course.

In Detail

A comprehensive and new course for learning the Apache Kafka Connect framework with hands-on Training.Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems. Apache Kafka Connect is a common framework for Apache Kafka producers and consumers.Apache Kafka Connect offers an API, runtime, and REST service to enable developers to define connectors that move large data sets into and out of Apache Kafka in real time. It inherits strong concepts such as fault-tolerance and elasticity thanks to being an extension of Apache Kafka. Kafka Connect can ingest entire databases, collect metrics, and gather logs from all your application servers into Apache Kafka topics, making the data available for stream processing with low latency. Kafka Connect standardises the integration of other data systems with Apache Kafka, simplifying connector development, deployment, and management. In this course, we are going to learn Kafka connector deployment, configuration, and management with hands-on exercises. We are also going to see the distributed and standalone modes to scale up to a large, centrally-managed service supporting an entire organisation or scale down to development, testing, and small production deployments. The REST interface is used to submit and manage connectors to your Kafka Connect cluster via easy to use REST API’s.

Table of Contents

  1. Chapter 1 : Course Introduction
    1. Important Pre-Requisites 00:01:03
    2. Course Objectives 00:01:20
    3. Course Structure 00:01:55
    4. About Your Instructor 00:00:55
  2. Chapter 2 : Kafka Connect Concepts
    1. What is Kafka Connect? 00:04:44
    2. Kafka Connect Architecture Design 00:03:21
    3. Connectors, Configuration, Tasks, Workers 00:03:04
    4. Standalone vs Distributed Mode 00:02:11
    5. Distributed Architecture in Details 00:02:58
  3. Chapter 3 : Setup and Launch Kafka Connect Cluster
    1. Docker on Mac (recent versions) 00:03:54
    2. Docker Toolbox on Mac (older versions) 00:05:17
    3. Docker on Linux (Ubuntu as an example) 00:04:56
    4. Docker on Windows 10 64bit 00:03:43
    5. Docker Toolbox on Windows (older versions) 00:05:37
    6. Starting Kafka Connect Cluster using Docker Compose 00:05:21
  4. Chapter 4 : Troubleshooting Kafka Connect
    1. It's not working! What to do? 00:01:33
    2. Where to view logs? 00:04:06
  5. Chapter 5 : Kafka Connect Source - Hands On
    1. Kafka Connect Source Architecture Design 00:01:14
    2. FileStream Source Connector - Standalone Mode - Part 1 00:09:06
    3. FileStream Source Connector - Standalone Mode - Part 2 00:07:57
    4. FileStream Source Connector - Distributed Mode 00:11:29
    5. List of Available Connectors 00:03:39
    6. Twitter Source Connector - Distributed Mode - Part 1 00:06:54
    7. Twitter Source Connector - Distributed Mode - Part 2 00:06:12
    8. Section Summary 00:01:37
  6. Chapter 6 : Kafka Connect Sink - Hands On
    1. Kafka Connect Sink Architecture Design 00:01:43
    2. ElasticSearch Sink Connector - Distributed Mode - Part 1 00:08:58
    3. ElasticSearch Sink Connector - Distributed Mode - Part 2 00:07:31
    4. Kafka Connect REST API 00:11:08
    5. JDBC Sink Connector - Distributed Mode 00:13:07
  7. Chapter 7 : Writing your own Kafka Connector
    1. Goal of the section: GitHubSourceConnector 00:01:29
    2. Finding the code and installing required software 00:03:10
    3. Description of the GitHub Issues API 00:03:56
    4. Using the Maven Archetype to get started 00:05:43
    5. Config Definitions 00:12:31
    6. Connector Class 00:06:57
    7. Writing a schema 00:07:32
    8. Data Model for our Objects 00:06:10
    9. Writing our GitHub API HTTP Client 00:07:04
    10. Source Partition & Source Offsets 00:04:23
    11. Source Task 00:13:40
    12. Building and running a Connector in Standalone Mode 00:04:59
    13. Deploying our Connector on the Landoop cluster 00:06:31
    14. More Resources for Developers 00:01:58
  8. Chapter 8 : Advanced Concepts
    1. Setting up Kafka Connect in Production (1/2) 00:14:20
    2. Setting up Kafka Connect in Production (2/2) 00:06:15
    3. What's next? 00:01:46