Python for Data Analysis, 2nd Edition

Chapter 10. Data Aggregation and Group Operations

Categorizing a dataset and applying a function to each group, whether an aggregation or transformation, is often a critical component of a data analysis workflow. After loading, merging, and preparing a dataset, you may need to compute group statistics or possibly pivot tables for reporting or visualization purposes. pandas provides a flexible groupby interface, enabling you to slice, dice, and summarize datasets in a natural way.

One reason for the popularity of relational databases and SQL (which stands for “structured query language”) is the ease with which data can be joined, filtered, transformed, and aggregated. However, query languages like SQL are somewhat constrained in the kinds of group operations that can be performed. As you will see, with the expressiveness of Python and pandas, we can perform quite complex group operations by utilizing any function that accepts a pandas object or NumPy array. In this chapter, you will learn how to:

• Split a pandas object into pieces using one or more keys (in the form of functions, arrays, or DataFrame column names)

• Calculate group summary statistics, like count, mean, or standard deviation, or a user-defined function

• Apply within-group transformations or other manipulations, like normalization, linear regression, rank, or subset selection

• Compute pivot tables and cross-tabulations

• Perform quantile analysis and other statistical group analyses

Note

Aggregation of time series data, a special ...