Data Engineering with Python

Chapter 13: Streaming Data with Apache Kafka

Apache Kafka opens up the world of real-time data streams. While there are fundamental differences in stream processing and batch processing, how you build data pipelines will be very similar. Understanding the differences between streaming data and batch processing will allow you to build data pipelines that take these differences into account.

In this chapter, we're going to cover the following main topics:

• Understanding logs
• Understanding how Kafka uses logs
• Building data pipelines with Kafka and NiFi
• Differentiating stream processing from batch processing
• Producing and consuming with Python

Understanding logs

If you have written code, you may be familiar with software logs. Software developers ...