Input sources and output stores
Spark Streaming supports three kinds of input sources:
- Basic sources: Sources directly available in the StreamingContext API. Examples: file systems, socket connections, and Akka actors.
- Advanced sources: Sources like Kafka, Flume, Kinesis, Twitter, and so on, which are available through extra utility classes.
- Custom sources: Requires implementing a user-defined receiver.
Multiple receivers can be created in the same application to receive data from different sources. It is important to allocate enough resources (cores and memory) for enabling receivers and tasks to execute simultaneously. For example, if you start your application with one core, it will be taken by the receiver and no tasks will be executed because ...
Get Big Data Analytics now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
