Apache Kafka - Real-time Stream Processing (Master Class)

Apache Kafka - Real-time Stream Processing (Master Class)

by Prashant Kumar Pandey
Released April 2020
Publisher(s): Packt Publishing
ISBN: 9781800209343

Watch it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Video description

If you want to understand the concept of stream processing, this course is for you. Using Kafka, the course will help you get to grips with real-time stream processing and enable you to apply that knowledge to learn Kafka programming techniques.

This course uses the Kafka Streams library available in Apache Kafka 2.x. All the source code and examples on Apache Kafka 2.3 open-source distribution have been tested. You'll understand and explore Confluent Platform functionalities such as Schema Registry and Avro Serdes using the Confluent Community Version. This course makes extensive use of IntelliJ IDEA and Apache Maven as the preferred development IDE. You'll leverage Log4J2 and JUnit5 for industry-standard log implementation in your application and implementing unit test cases, respectively.

What You Will Learn

  • Learn Apache Kafka fundamentals and Kafka Architecture
  • Understand how to create streams using the Kafka Producer APIs
  • Design, develop, and test real-time stream processing applications using the Kafka Streams library
  • Get well-versed with the Kafka Streams architecture, Streams DSL, and the Processor API
  • Support microservice architecture and implement interactive queries in Kafka Streams
  • Explore serializing and deserializing and work on JSON and AVRO messages

Audience

The Kafka Streams - Real-time Stream Processing course is for software engineers looking to develop a stream processing application using the Kafka Streams library. This course will also help Java programmers who want to learn everything necessary to start implementing real-time streaming applications using Apache Kafka. Data architects and data engineers responsible for designing and building the organization’s data-centric infrastructure will benefit from this course.

About The Author

Prashant Kumar Pandey: Prashant Kumar Pandey is passionate about helping people learn and grow in their careers by bridging the gap between their existing and required skills. In his journey to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. He is also the founder, lead author, and chief editor of the Learning Journal portal that has been providing various skill development courses, training sessions, and technical articles since 2018.

Table of contents

  1. Chapter 1 : Before you start
    1. Introduction
    2. About the Course
    3. What do you Need for this Course?
    4. Debugging Problems
  2. Chapter 2 : Introduction to Real-time Streams
    1. Emergence of Bigdata - A Quick Recap
    2. Conception of Event Streams
    3. Real-time Streaming - Use Cases
    4. Real-time Streaming Challenges
    5. Real-time Streaming Design Consideration
    6. Section Summary
  3. Chapter 3 : Enter the world of Apache Kafka
    1. What is Apache Kafka?
    2. Kafka Storage Architecture
    3. Kafka Cluster Architecture
    4. Kafka Work Distribution Architecture - Part 1
    5. Kafka Work Distribution Architecture - Part 2
    6. Section Summary
  4. Chapter 4 : Creating Real-time Streams
    1. Streaming into Kafka
    2. Kafka Producers - Quick Start
    3. Kafka Producer Internals
    4. Scaling Kafka Producer
    5. Advanced Kafka Producers (Exactly Once)
    6. Advanced Kafka Producer (Implementing Transaction)
    7. Kafka Producer - Micro Project
    8. Kafka Producer - Final Note and References
  5. Chapter 5 : Enter the Stream Processing
    1. Stream Processing in Apache Kafka
    2. Kafka Consumer - Practical Introduction
    3. Kafka Consumer - Scalability, Fault tolerance and Missing Features
    4. Kafka Streams API - Quick Start
    5. Creating Streams Topology
    6. Implementing Streams Topology
    7. Kafka Streams Architecture
    8. Section Summary and References
  6. Chapter 6 : Foundation for Real Life Implementations
    1. Introduction to Types and Serialization in Kafka
    2. JSON Schema to POJO for JSON Serdes
    3. Creating and Using JSON Serdes
    4. AVRO Schema to POJO for AVRO Serdes
    5. Creating and using AVRO schema in Producers
    6. Creating and using AVRO schema in Kafka Streams
    7. Section Summary and References
  7. Chapter 7 : States and Stores
    1. Understanding States and State Stores
    2. Creating your First State Store
    3. Caution with States
    4. State Store Fault Tolerance
    5. Section Summary and References
  8. Chapter 8 : KTable - An Update Stream
    1. Introducing KTable
    2. Creating your First Update Stream - Ktable
    3. Table Caching and Emit Rates
    4. Introducing GlobalKTable
  9. Chapter 9 : Real-time Aggregates
    1. Computing Your First Aggregate - Real-time Streaming Word Count
    2. Streaming Aggregates - Core Concept
    3. KStream Aggregation using Reduce()
    4. KStream Aggregation using Aggregate()
    5. Common Mistakes in Aggregation
    6. Count on KTable
    7. KTable Aggregation using Aggregate()
  10. Chapter 10 : Timestamps and Windows
    1. Timestamps and Timestamp Extractors
    2. Creating Tumbling Windows
    3. Stream Time and Grace Period
    4. Supressing Intermediate Results
    5. Creating Hopping Windows
    6. Creating Session Windows
  11. Chapter 11 : Joining Streams and Tables
    1. Streaming Joins
    2. Joining a KStrem to another KStream
    3. Joining a KTable to another KTable
    4. Joining a KStream to a KTable and GlobalKTable
    5. Mixing Joins with Aggregates - Computing Top 3
    6. Mixing Joins with Aggregates - Advert CTR
  12. Chapter 12 : Testing Streams Application
    1. How to test a Stream Processing Application
    2. Unit Testing Your Topology
  13. Chapter 13 : Interactive Query and Micro-Service Responses
    1. Introducing Micro-services Requirement
    2. Understanding Local Vs Remote State Store
    3. Implementing Interactive Query Micro-service
  14. Chapter 14 : Appendix
    1. Setting up Apache Kafka Development Environment

Product information

  • Title: Apache Kafka - Real-time Stream Processing (Master Class)
  • Author(s): Prashant Kumar Pandey
  • Release date: April 2020
  • Publisher(s): Packt Publishing
  • ISBN: 9781800209343