Scaling Python with Dask

Chapter 6. Advanced Task Scheduling: Futures and Friends

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the sixth chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

So far in Dask, you have mostly seen lazy operations, where Dask doesn’t do the work until something “forces” the computation. This pattern has a number of benefits, including allowing Dask’s optimizer to combine steps when it makes sense. However, not all tasks are well suited to lazy evaluation. One common pattern not well suited to lazy evaluation is “fire-and-forget”, where we call a function for its side effect¹ and necessarily care about the output. Trying to express this with lazy evaluation (e.g. dask.delayed) results in unnecessary blocking to force computation. When lazy evaluation is not what you need, you can explore Dask’s futures. Futures can be used for much more than just “fire-and-forget” and you can return results from them. This chapter will explore a number of common use cases for futures.

Note

You may already be familiar with futures from Python. Dask’s ...