Batching Messages
Using processors is great for many tasks, but sometimes you will need another layer of stages. Your business logic in handle_message/3 could become a bottleneck and slow down the pipeline, especially under heavy workloads. You may also want to improve efficiency by grouping and running some operations together. This is where batching comes in to help.
Using batching in Broadway is an easy way to add another step in your data-ingestion pipeline for further processing. It allows you to leverage concurrency, and group relevant messages together to perform operations in bulk. In this section, we’re going to configure batching for BookingsPipeline to improve how tickets are being created. Rather than inserting tickets in the database ...
Get Concurrent Data Processing in Elixir now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
