Adding Concurrency with ConsumerSupervisor

Using multiple consumers to process events one by one, concurrently, is a very useful technique. To make this task even easier, GenStage comes with a special type of supervisor called ConsumerSupervisor. It works similarly to a consumer and can subscribe to one or more producers. It can also monitor, start, and restart child processes, just like a supervisor.

What makes ConsumerSupervisor special is that when it receives a list of events from a producer, it automatically starts a process for each event and passes the event as an argument to that process. When a child process exits successfully, new demand will be issued by ConsumerSupervisor and the cycle repeats. This figure illustrates how child processes ...

Get Concurrent Data Processing in Elixir now with the O’Reilly learning platform.

O’Reilly members experience live online training, plus books, videos, and digital content from nearly 200 publishers.