Practical Real-time Data Processing and Analytics

Transactional spout

Let's have a look at how trident spout processes tuples and what the characteristics are:

• Trident processes tuples in small batches.
• Each batch has its own unique transaction ID
• Trident ensures every tuple is in batch so no tuples are skipped
• Batches with a given transaction id are always the same. If a batch is reprocessed, even then the batch would have the same transaction ID and the same set of tuples
• Tuples can't be part of more than one batch. Every batch would have a unique set of tuples

Using the following statement in code, we can define a transactional spout with Kafka:

TransactionalTridentKafkaSpout tr = new TransactionalTridentKafkaSpout(new TridentKafkaConfig(new ZkHosts("localhost:9091"), "test"));

The ...