Programming Ecto

Fetching Large Datasets with Streams

Streams are a core part of Elixir. We use streams for lazy processing, to avoid loading lots of data into memory at once, and even for processing infinite data streams. Many of Elixir’s concurrency constructs build on top of streams, such as the Task.async_stream function and the GenStage and Flow packages. It’s important to try to utilize concurrency when working with databases because much of the time spent during query execution is waiting for network I/O, during which the CPU is free to do other work.

Throughout much of the book, we’ve used Repo.all to fetch data from the database. Repo.stream is its stream-based counterpart. It returns a lazy stream that can work with a database as its source.

Like ...