High Performance Python, 3rd Edition

Chapter 7. Pandas, Dask, and Polars

Questions You’ll Be Able to Answer After This Chapter

• What’s the fastest way to apply a function to a Pandas DataFrame?

• What’s the quickest way to build a DataFrame from partial results?

• Can we use Numba to compile for more performance inside Pandas?

• Can Dask be used for distributed CPU computation?

• How can Polars execute similar queries faster than Pandas?

• How can I parallelize Pandas using Dask and Swifter?

Many scientific and data science projects use tabular-shaped data that fits into a dataframe. A dataframe typically collects a heterogeneous (i.e., mixed) collection of datatypes where the datatypes are assigned to columns. Each entry in the dataframe is a row—generally they look like a spreadsheet that you might see in Excel.

Pandas was released in 2008 and quickly became the main dataframe library in the Python ecosystem. As it evolved, a lot of shortcomings were discussed, as later documented in Wes McKinney’s infamous 2017 blog post “Apache Arrow and the ‘10 Things I Hate About pandas’”. At the time of this writing, eight years after that blog post, Pandas is still the most popular Python dataframe library. In the following sections we discuss some of the common ways of writing slow—or fast—Pandas solutions.

Dask, first introduced in 2014, is more than a dataframe library. It is a distributed computing framework for scientific applications—the most famous component is the distributed dataframe, which wraps the Pandas ...