Python Data Cleaning Cookbook

Chapter 7: Fixing Messy Data when Aggregating

Earlier chapters of this book introduced techniques for generating summary statistics on a whole DataFrame. We used methods such as describe, mean, and quantile to do that. This chapter covers more complicated aggregation tasks: aggregating by categorical variables, and using aggregation to change the structure of DataFrames.

After the initial stages of data cleaning, analysts spend a substantial amount of their time doing what Hadley Wickham has called splitting-applying-combining. That is, we subset data by groups, apply some operation to those subsets, and then draw conclusions about a dataset as a whole. In slightly more specific terms, this involves generating descriptive statistics by key categorical ...