Scaling Python with Dask

Chapter 12. Productionizing Dask: Notebooks, Deployment, Tuning, and Monitoring

We have bundled most of the things we believe are going to be critical for you to move from your laptop into production in this chapter. Notebooks and deployments go together, as Dask’s notebook interface greatly simplifies many aspects of using its distributed deployments. While you don’t need to use notebooks to access Dask, and in many cases notebooks have serious drawbacks, for interactive use cases it’s often hard to beat the trade-offs. Interactive/exploratory work has a way of becoming permanent mission-critical workflows, and we cover the steps necessary to turn exploratory work into production deployments.

You can deploy Dask in many fashions, from running it on top of other distributed compute engines like Ray to deploying it on YARN or a raw collection of machines. Once you’ve got your Dask job deployed, you’ll likely need to tune it so you don’t use your company’s entire AWS budget on one job. And then, finally, before you can walk away from a job, you’ll need to set up monitoring—so you know when it’s broken.

Note

If you’re just here to learn how to use Dask with notebooks, feel free to skip ahead to that section. If you want to learn more about deploying Dask, congratulations and condolences on exceeding the scale you can handle on a single computer.

In this chapter, we will cover some (but not all) of the deployment options for Dask and their trade-offs. You will learn how to integrate ...