Python High Performance - Second Edition

Dask arrays

One of the main use-cases of Dask is the automatic generation of parallel array operations, which greatly simplifies the handling of arrays that don't fit into memory. The strategy employed by Dask is to split the array into a number of subunits that, in Dask array terminology, are called chunks.

Dask implement a NumPy-like interface for arrays in the dask.array module (which we will abbreviate as da). An array can be created from a NumPy-like array using the da.from_array function, which requires the specification of a chunk size. The da.from_array function will return a da.array object that will handle the splitting of the original array into subunits of the specified chunk size. In the following example, we create an array ...