Pandas Cookbook

How it works...

There are multiple ways to accomplish the same thing in step 1. Here, we show the versatility of the read_csv function. The usecols parameter accepts either a list of the columns that we would like to import or a function that dynamically determines them. We use an anonymous function that checks whether the column name contains UGDS_ or is equal to INSTNM. The function is passed each column name as a string and must return a boolean. A huge amount of memory can be saved in this manner.

The stack method in step 2 puts all column names into the innermost index level and returns a Series. In step 3, the unstack method inverts this operation by taking all the values in the innermost index level converting them to column names. ...