Concurrent Data Processing in Elixir

Creating Multi-Stage Data Pipelines

We already demonstrated how :producer and :consumer stages work in practice. The only type of stage that we haven’t seen in action yet is the :producer_consumer. Producer-consumer stages are the key to building infinitely complex data-processing pipelines. The good news is that if you understand how producers and consumers work, you already know producer-consumers.

When you learn how to add stages and extend your data pipelines, you may be tempted to start organizing your business logic using stages, rather than plain Elixir modules and functions. As the GenStage documentation warns us, this is an anti-pattern:

If your domain has to process the data in multiple steps, you should write that logic in separate ...