Using messaging systems in big data streaming applications

In this section, we will talk about how messaging systems play important role in a big data application.

Let's understand the different layers in a big data application:

  • Ingestion layer: The input data required for the processing gets ingested in some storage system. There can be many sources of data for which the same or different processing needs to be done.
  • Processing layer: This contains the business logic that processes the data received in the ingestion layer and applies some transformation to make it into a usable form. You can call it converting raw data to information. There can be multiple processing applications for the same or different data. Each application may have ...

Get Building Data Streaming Applications with Apache Kafka now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.