Video description
Kafka Connect - Learn How to Source Twitter Data, Store in Apache Kafka Topics and Sink in ElasticSearch and PostgreSQL
About This Video
- In this course, you will learn what is Kafka Connect, Kafka Connect architecture, how to deploy an Apache Kafka Connector in standalone and in distributed modes.
- How to set up and launch a Kafka Connect cluster using Docker Compose.
- You will be able to deploy Kafka Connect source connectors and Kafka Connect sinks connectors at the end of this course.
In Detail
A comprehensive and new course for learning the Apache Kafka Connect framework with hands-on Training.Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems. Apache Kafka Connect is a common framework for Apache Kafka producers and consumers.Apache Kafka Connect offers an API, runtime, and REST service to enable developers to define connectors that move large data sets into and out of Apache Kafka in real time. It inherits strong concepts such as fault-tolerance and elasticity thanks to being an extension of Apache Kafka. Kafka Connect can ingest entire databases, collect metrics, and gather logs from all your application servers into Apache Kafka topics, making the data available for stream processing with low latency. Kafka Connect standardises the integration of other data systems with Apache Kafka, simplifying connector development, deployment, and management. In this course, we are going to learn Kafka connector deployment, configuration, and management with hands-on exercises. We are also going to see the distributed and standalone modes to scale up to a large, centrally-managed service supporting an entire organisation or scale down to development, testing, and small production deployments. The REST interface is used to submit and manage connectors to your Kafka Connect cluster via easy to use REST API’s.
Audience
Developers who wants to learn the Apache Kafka Connect Framework and get hands-on with it.
Professionals who are good at the Apache Kafka ecosystem and basic core concepts. Architects who want to understand how Kafka Connect fits into their solution architecture.
Publisher resources
Table of contents
- Chapter 1 : Course Introduction
- Chapter 2 : Kafka Connect Concepts
- Chapter 3 : Setup and Launch Kafka Connect Cluster
- Chapter 4 : Troubleshooting Kafka Connect
-
Chapter 5 : Kafka Connect Source - Hands On
- Kafka Connect Source Architecture Design
- FileStream Source Connector - Standalone Mode - Part 1
- FileStream Source Connector - Standalone Mode - Part 2
- FileStream Source Connector - Distributed Mode
- List of Available Connectors
- Twitter Source Connector - Distributed Mode - Part 1
- Twitter Source Connector - Distributed Mode - Part 2
- Section Summary
- Chapter 6 : Kafka Connect Sink - Hands On
-
Chapter 7 : Writing your own Kafka Connector
- Goal of the section: GitHubSourceConnector
- Finding the code and installing required software
- Description of the GitHub Issues API
- Using the Maven Archetype to get started
- Config Definitions
- Connector Class
- Writing a schema
- Data Model for our Objects
- Writing our GitHub API HTTP Client
- Source Partition Source Offsets
- Source Task
- Building and running a Connector in Standalone Mode
- Deploying our Connector on the Landoop cluster
- More Resources for Developers
- Chapter 8 : Advanced Concepts
Product information
- Title: Apache Kafka Series - Kafka Connect Hands-on Learning
- Author(s):
- Release date: May 2018
- Publisher(s): Packt Publishing
- ISBN: 9781789344738
You might also like
video
Apache Kafka Series - Kafka Cluster Setup and Administration
Hands-On Training on ZooKeeper Quorum Setup, Kafka Cluster Setup, and Administration in AWS About This Video …
video
Apache Kafka Series - Confluent Schema Registry and REST Proxy
Kafka - Master Avro, the Confluent Schema Registry and Kafka REST Proxy. Build Avro Producers/Consumers, Evolve …
video
Amazon Web Services AWS LiveLessons 2nd Edition
More Than 17 Hours of Video Instruction More than 17 hours of video instruction on Amazon …
video
Introduction to Apache Kafka
Currently one of the hottest projects across the Hadoop ecosystem, Apache Kafka is a distributed, real-time …