Microservices Deployment Cookbook

Chapter 7. Building Asynchronous Streaming Systems with Kafka and Spark

In this chapter, you will learn the following recipes:

• Setting up Kafka using Docker
• Creating Kafka topics to stream data
• Writing a streaming program using Kafka Streams
• Improving the performance of the Kafka Streams program
• Writing a streaming program using Apache Spark
• Improving the performance of the Spark job
• Aggregating logs into Kafka using Log4J
• Integrating Kafka with log management systems

Introduction

Streaming has been picking up traction lately. It is one way of processing your data. In fact, there are two modes in which your data processing application can operate: batching and streaming. In batching, you work on batches of datasets at frequent intervals. However, in streaming, ...