Chapter 6. Advanced Task Scheduling: Futures and Friends
So far in Dask, you have mostly seen lazy operations, where Dask doesn’t do the work until something “forces” the computation. This pattern has a number of benefits, including allowing Dask’s optimizer to combine steps when it makes sense. However, not all tasks are well suited to lazy evaluation. One common pattern not well suited to lazy evaluation is “fire-and-forget”, where we call a function for its side effect1 and necessarily care about the output. Trying to express this with lazy evaluation (e.g. dask.delayed) results in unnecessary blocking to force computation. When lazy evaluation is not what you need, you can explore Dask’s futures. Futures can be used for much more than just “fire-and-forget” and you can return results from them. This chapter will explore a number of common use cases for futures.
You may already be familiar with futures from Python. Dask’s ...
Get Scaling Python with Dask now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.