Python Polars: The Definitive Guide

Foreword

It was never meant to be this serious.

In December 2019, I became a dad, and then the pandemic hit. This left me dazed; juggling life with a newborn and trying to hold on to a sense of “me.” In a burst of new-dad sleep deprivation, I figured I’d take on a new project. Honestly, it started as nothing more than a hobby.

At work, I had to join two CSV files while programming in Rust, and it felt like way more hassle than it should have been. I wondered if there were easier ways to get it done than setting up SQLite. So, as one often does in software development, I decided to try my hand at creating my own join algorithm.

At the time, I was pretty new to Rust and didn’t know much about optimizing performance. So as a proud writer of my first join algorithm, I learned that my implementation was much slower than pandas. This unsatisfying result planted the seed of what would later become Polars.

This led me down a path of researching database engines, learning Rust, and a lot of trial and error over the next year and a half. As I learned more about databases, algorithms, performance, memory, unsafe code, etc., my goals shifted. I went from just wanting to make a faster join than pandas to building a DataFrame package for Rust, and eventually, a high-performance query engine that could rival the state of the art in the Python landscape.

I drew inspiration from pandas’ strengths and weaknesses, the declarative approaches of SQL and PySpark, functional programming principles, ...