Scaling Python with Dask

Appendix C. Debugging Dask

Depending on your debugging techniques, moving to distributed systems could require a new set of techniques. While you can use debuggers in remote mode, it often requires more setup work. You can also run Dask locally to use your existing debugging tools in many other situations, although—take it from us—a surprising number of difficult-to-debug errors don’t show up in local mode. Dask has a special hybrid approach. Some errors happen outside Python, making them more difficult to debug, like container out-of-memory (OOM) errors, segmentation faults, and other native errors.

Note

Some of this advice is common across distributed systems, including Ray and Apache Spark. As such, some elements of this chapter are shared with High Performance Spark, second edition, and Scaling Python with Ray.

Using Debuggers

There are a few different options for using debuggers in Dask. PyCharm and PDB both support connecting to remote debugger processes, but figuring out where your task is running and also setting up the remote debugger can be a challenge. For details on PyCharm remote debugging, see the JetBrains article “Remote Debugging with PyCharm”. One option is to use epdb and run import epdb; epdb.serve() inside of an actor. The easiest option, which is not perfect, is to have Dask re-run failed tasks locally by running client.recreate_error_locally on the future that failed.

General Debugging Tips with Dask

You likely have your own standard debugging techniques ...