Concurrent Data Processing in Elixir

Adding Concurrency with ConsumerSupervisor

Using multiple consumers to process events one by one, concurrently, is a very useful technique. To make this task even easier, GenStage comes with a special type of supervisor called ConsumerSupervisor. It works similarly to a consumer and can subscribe to one or more producers. It can also monitor, start, and restart child processes, just like a supervisor.

What makes ConsumerSupervisor special is that when it receives a list of events from a producer, it automatically starts a process for each event and passes the event as an argument to that process. When a child process exits successfully, new demand will be issued by ConsumerSupervisor and the cycle repeats. This figure illustrates how child processes ...