Building Data Streaming Applications with Apache Kafka

Direct approach

In receiver-based approach, we saw issues of data loss, costing less throughput using write-ahead logs and difficulty in achieving exactly one semantic of data processing. To overcome all these problems, Spark introduced the direct stream approach of integrating Spark with Kafka.

Spark periodically queries messages from Kafka with a range of offsets, which in short we call batch. Spark uses a low level consumer API and fetches messages directly from Kafka with a defined range of offsets. Parallelism is defined by a partition in Kafka and the Spark direct approach takes advantage of partitions.

The following illustration gives a little detail about parallelism:

Direct approach 

Let's look at a few features of the direct ...