Chapter 10. Data Aggregation and Group Operations
Categorizing a dataset and applying a function to each group, whether an
aggregation or transformation, can be a critical component of a data
analysis workflow. After loading, merging, and preparing a dataset, you may
need to compute group statistics or possibly pivot tables for reporting or
visualization purposes. pandas provides a versatile
groupby interface, enabling you to slice, dice,
and summarize datasets in a natural way.
One reason for the popularity of relational databases and SQL (which stands for “structured query language”) is the ease with which data can be joined, filtered, transformed, and aggregated. However, query languages like SQL impose certain limitations on the kinds of group operations that can be performed. As you will see, with the expressiveness of Python and pandas, we can perform quite complex group operations by expressing them as custom Python functions that manipulate the data associated with each group. In this chapter, you will learn how to:
Split a pandas object into pieces using one or more keys (in the form of functions, arrays, or DataFrame column names)
Calculate group summary statistics, like count, mean, or standard deviation, or a user-defined function
Apply within-group transformations or other manipulations, like normalization, linear regression, rank, or subset selection
Compute pivot tables and cross-tabulations
Perform quantile analysis and other statistical group analyses
Time-based aggregation ...
Get Python for Data Analysis, 3rd Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.