Scaling Python with Dask

Chapter 5. Dask’s Collections

So far you’ve seen the basics of how Dask is built as well as how Dask uses these building blocks to support data science with DataFrames. This chapter explores where Dask’s bag and array interfaces—often overlooked, relative to DataFrames—are more appropriate. As mentioned in “Hello Worlds”, Dask bags implement common functional APIs, and Dask arrays implement a subset of NumPy arrays.

Tip

Understanding partitioning is important for understanding collections. If you skipped “Partitioning/Chunking Collections”, now is a good time to head back and take a look.

Dask Arrays

Dask arrays implement a subset of the NumPy ndarray interface, making them ideal for porting code that uses NumPy to run on Dask. Much of your understanding from the previous chapter with DataFrames carries over to Dask arrays, as well as much of your understanding of ndarrays.

Common Use Cases

Some common use cases for Dask arrays include:

• Large-scale imaging and astronomy data

• Weather data

• Multi-dimensional data

Similar to Dask DataFrames and pandas, if you wouldn’t use an nparray for the problem at a smaller scale, a Dask array may not be the right solution.

When Not to Use Dask Arrays

If your data fits in memory on a single computer, using Dask arrays is unlikely to give you many benefits over nparrays, especially compared to local accelerators like Numba. Numba is well suited to vectorizing and parallelizing local tasks with and without Graphics Processing Units ...