Scaling Python with Dask

Chapter 3. How Dask Works: The Basics

Now that you’ve run your first few tasks with Dask, it’s time to learn a little bit about what’s happening behind the scenes. Depending on whether you are using Dask locally or in a distributed fashion, the behavior can be a little different. While Dask does a good job of abstracting away many of the details of running on multiple threads or servers, having a solid grasp of how Dask is working will help you better decide both how and when to use it.

To be familiar with Dask, you need to understand:

• The deployment framework that Dask is able to run on, and its strengths and weaknesses

• The types of data that Dask is able to read, and how you can interact with the data types in Dask

• The computational pattern of Dask, and how to turn your ideas into Dask code

• How to monitor and troubleshoot

In this chapter, we will introduce each of these concepts, and we will expand upon them in the rest of the book.

Execution Backends

Dask has many different execution backends, but we find it easiest to think about them in two groups, local and distributed. With local backends, you are limited in scale to what a single computer can handle. Local backends also have advantages like the avoidance of network overhead, simpler library management, and a lower dollar cost.¹ Dask’s distributed backend has many options for deployment, from cluster managers such as Kubernetes to job queue–like systems.

Local Backends

Dask’s three local backends are single-process, ...