GenStage

GenStage is an Elixir behaviour that was in the making for a while. The plan was to include it in the Elixir standard library but it grew sufficiently enough to justify its own :gen_stage library.

Elixir lets you lazily process collections with the Stream module, and we previously saw how we can process a collection in parallel by spawning tasks for each element in the collection. However, if the process through which we are passing our collection has many steps, the usage of Task.async_stream isn't the best solution because, for each step, it will wait for all tasks to return or time out, before calling the next wave of Task.async_stream.

What we want is something that can spawn the needed processes to consume the collection in ...

Get Mastering Elixir now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.