Concurrent Data Processing in Elixir

Managing Series of Tasks

Let’s imagine we have one million users and we want to send an email to all of them. We can use Enum.map/2 and Task.async/1 like we did before, but starting one million processes will put sudden pressure on our system resources. It can degrade the system’s performance and potentially make other services unresponsive. Our email service provider will not be happy either, because we also put a lot of pressure on their email infrastructure.

On the other hand, we don’t want to send emails one by one, because it is slow and inefficient. It seems that we are at a crossroad, and whichever way we take, we end up in peril. We don’t want to choose between performance and reliability—we want to be able to run Task processes to leverage ...