Python Polars: The Definitive Guide

Chapter 17. Extending Polars

As you’ve seen in previous chapters, Polars’ API is already quite extensive and covers a wide range of functionality. However, there might be cases where you want to extend Polars with your own custom functionality. This could be because you have a specific use case that isn’t covered by the built-in functions, or because you want to optimize the performance of your code.

In this chapter, you’ll learn how to:

• Apply a custom Python function to Polars data

• Register a Polars namespace

• Write Rust plugins and run them on the Polars engine for maximum performance

• Use Rust crates in those plugins

The instructions to get any files you might need are in Chapter 2. We assume that you have the files in the data subdirectory.

User-Defined Functions in Python

Polars has an extensive set of expressions that allow you to perform a wide range of operations. However, sometimes you need to perform an operation that isn’t covered by the available expressions, or is performed by an external package. To leave you this option, Polars allows for user-defined functions (UDFs). The Polars methods that allow you to do this are:

Expr.map_elements()

Apply a Python function to each element of a Series

Expr.map_batches()

Apply a Python function to a Series or sequence of Series

Expr.map_groups()

Apply a Python function to each group in the GroupBy context

Expr.pipe()

Apply a Python function to an Expression

df.pipe()

Apply a Python function to an entire DataFrame ...