Scaling Python with Dask

Chapter 7. Adding Changeable/Mutable State with Dask Actors

Dask is focused on scaling analytic use cases, but you can use it to scale many other types of problems. So far, most of the tools you have used in Dask are functional. Functional programming means that previous calls do not impact future calls. Stateless functions are common in distributed systems like Dask, as they can safely be re-executed multiple times on failure. Updating the weights of a model during training is an example of state common in data science. One of the most common ways of handling state in a distributed system is with the actor model. This chapter will introduce both the general actor model and Dask’s specific implementation.

Dask futures offer a non-mutable distributed state, where values are stored on the workers. However, this doesn’t work well for situations in which you want to update the state, like changing a bank account balance (an alternative solution is illustrated in Example 7-1), or updating machine learning model weights during training.

Tip

Dask actors have a number of limitations, and we believe that in many cases the right answer is to keep mutable state outside of Dask (like in a database).

Of course, you don’t have to use distributed mutable state. In some cases, you may choose to not use distributed state and instead put it all in your main program. This can quickly lead to bottlenecks on the node responsible for your main program. Other options include storing your state outside ...