book

Pandas in Action

by Boris Paskhaver

September 2021

Intermediate to advanced

440 pages

11h 39m

English

Manning Publications

Read now

Unlock full access

prefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A road mapAbout the codeliveBook discussion forumOther online resourcesabout the authorabout the cover illustration
1.1 Data in the 21st century1.2 Introducing pandas1.2.1 Pandas vs. graphical spreadsheet applications1.2.2 Pandas vs. its competitors1.3 A tour of pandas1.3.1 Importing a data set1.3.2 Manipulating a DataFrame1.3.3 Counting values in a Series1.3.4 Filtering a column by one or more criteria1.3.5 Grouping dataSummary
2.1 Overview of a Series2.1.1 Classes and instances2.1.2 Populating the Series with values2.1.3 Customizing the Series index2.1.4 Creating a Series with missing values2.2 Creating a Series from Python objects2.3 Series attributes2.4 Retrieving the first and last rows2.5 Mathematical operations2.5.1 Statistical operations2.5.2 Arithmetic operations2.5.3 Broadcasting2.6 Passing the Series to Python’s built-in functions2.7 Coding challenge2.7.1 Problems2.7.2 SolutionsSummary
3.1 Importing a data set with the read_csv function3.2 Sorting a Series3.2.1 Sorting by values with the sort_values method3.2.2 Sorting by index with the sort_index method3.2.3 Retrieving the smallest and largest values with the nsmallest and nlargest methods3.3 Overwriting a Series with the inplace parameter3.4 Counting values with the value_counts method3.5 Invoking a function on every Series value with the apply method3.6 Coding challenge3.6.1 Problems3.6.2 SolutionsSummary
4.1 Overview of a DataFrame4.1.1 Creating a DataFrame from a dictionary4.1.2 Creating a DataFrame from a NumPy ndarray4.2 Similarities between Series and DataFrames4.2.1 Importing a DataFrame with the read_csv function4.2.2 Shared and exclusive attributes of Series and DataFrames4.2.3 Shared methods of Series and DataFrames4.3 Sorting a DataFrame4.3.1 Sorting by a single column4.3.2 Sorting by multiple columns4.4 Sorting by index4.4.1 Sorting by row index4.4.2 Sorting by column index4.5 Setting a new index4.6 Selecting columns and rows from a DataFrame4.6.1 Selecting a single column from a DataFrame4.6.2 Selecting multiple columns from a DataFrame4.7 Selecting rows from a DataFrame4.7.1 Extracting rows by index label4.7.2 Extracting rows by index position4.7.3 Extracting values from specific columns4.8 Extracting values from Series4.9 Renaming columns or rows4.10 Resetting an index4.11 Coding challenge4.11.1 Problems4.11.2 SolutionsSummary

5.1 Optimizing a data set for memory use5.1.1 Converting data types with the astype method5.2 Filtering by a single condition5.3 Filtering by multiple conditions5.3.1 The AND condition5.3.2 The OR condition5.3.3 Inversion with ~5.3.4 Methods for Booleans5.4 Filtering by condition5.4.1 The isin method5.4.2 The between method5.4.3 The isnull and notnull methods5.4.4 Dealing with null values5.5 Dealing with duplicates5.5.1 The duplicated method5.5.2 The drop_duplicates method5.6 Coding challenge5.6.1 Problems5.6.2 SolutionsSummary
6.1 Letter casing and whitespace6.2 String slicing6.3 String slicing and character replacement6.4 Boolean methods6.5 Splitting strings6.6 Coding challenge6.6.1 Problems6.6.2 Solutions6.7 A note on regular expressionsSummary
7.1 The MultiIndex object7.2 MultiIndex DataFrames7.3 Sorting a MultiIndex7.4 Selecting with a MultiIndex7.4.1 Extracting one or more columns7.4.2 Extracting one or more rows with loc7.4.3 Extracting one or more rows with iloc7.5 Cross-sections7.6 Manipulating the Index7.6.1 Resetting the index7.6.2 Setting the index7.7 Coding challenge7.7.1 Problems7.7.2 SolutionsSummary
8.1 Wide vs. narrow data8.2 Creating a pivot table from a DataFrame8.2.1 The pivot_table method8.2.2 Additional options for pivot tables8.3 Stacking and unstacking index levels8.4 Melting a data set8.5 Exploding a list of values8.6 Coding challenge8.6.1 Problems8.6.2 SolutionsSummary
9.1 Creating a GroupBy object from scratch9.2 Creating a GroupBy object from a data set9.3 Attributes and methods of a GroupBy object9.4 Aggregate operations9.5 Applying a custom operation to all groups9.6 Grouping by multiple columns9.7 Coding challenge9.7.1 Problems9.7.2 SolutionsSummary
10.1 Introducing the data sets10.2 Concatenating data sets10.3 Missing values in concatenated DataFrames10.4 Left joins10.5 Inner joins10.6 Outer joins10.7 Merging on index labels10.8 Coding challenge10.8.1 Problems10.8.2 SolutionsSummary
11.1 Introducing the Timestamp object11.1.1 How Python works with datetimes11.1.2 How pandas works with datetimes11.2 Storing multiple timestamps in a DatetimeIndex11.3 Converting column or index values to datetimes11.4 Using the DatetimeProperties object11.5 Adding and subtracting durations of time11.6 Date offsets11.7 The Timedelta object11.8 Coding challenge11.8.1 Problems11.8.2 SolutionsSummary
12.1 Reading from and writing to JSON files12.1.1 Loading a JSON file Into a DataFrame12.1.2 Exporting a DataFrame to a JSON file12.2 Reading from and writing to CSV files12.3 Reading from and writing to Excel workbooks12.3.1 Installing the xlrd and openpyxl libraries in an Anaconda environment12.3.2 Importing Excel workbooks12.3.3 Exporting Excel workbooks12.4 Coding challenge12.4.1 Problems12.4.2 SolutionsSummary
13.1 Getting and setting pandas options13.2 Precision13.3 Maximum column width13.4 Chop threshold13.5 Option contextSummary
14.1 Installing matplotlib14.2 Line charts14.3 Bar graphs14.4 Pie chartsSummary
A.1 The Anaconda distributionA.2 The macOS setup processA.2.1 Installing Anaconda in macOSA.2.2 Launching TerminalA.2.3 Common Terminal commandsA.3 The Windows setup processA.3.1 Installing Anaconda in WindowsA.3.2 Launching Anaconda PromptA.3.3 Common Anaconda Prompt commandsA.4 Creating a new Anaconda environmentA.5 Anaconda NavigatorA.6 The basics of Jupyter Notebook
B.1 Simple data typesB.1.1 NumbersB.1.2 StringsB.1.3 BooleansB.1.4 The None objectB.2 OperatorsB.2.1 Mathematical operatorsB.2.2 Equality and inequality operatorsB.3 VariablesB.4 FunctionsB.4.1 Arguments and return valuesB.4.2 Custom functionsB.5 ModulesB.6 Classes and objectsB.7 Attributes and methodsB.8 String methodsB.9 ListsB.9.1 List iterationB.9.2 List comprehensionB.9.3 Converting a string to a list and vice versaB.10 TuplesB.11 DictionariesB.11.1 Dictionary IterationB.12 Sets
C.1 DimensionsC.2 The ndarray objectC.2.1 Generating a numeric range with the arange methodC.2.2 Attributes on a ndarray objectC.2.3 The reshape methodC.2.4 The randint functionC.2.5 The randn functionC.3 The nan object
D.1 Installing FakerD.2 Getting started with FakerD.3 Populating a DataFrame with fake values
E.1 Introduction to Python’s re moduleE.2 MetacharactersE.3 Advanced search patternsE.4 Regular expressions and pandas

Content preview from Pandas in Action

front matter

preface

Truth be told, I discovered pandas entirely by luck.

In 2015, I interviewed for a data operations analyst position at Indeed.com, the world’s largest jobs site. For my final technical challenge, I was asked to derive insights from an internal data set, using the Microsoft Excel spreadsheet software. Eager to impress, I pulled out as many tricks as I could from my data analysis toolbox: column sorts, text manipulations, pivot tables, and of course the iconic VLOOKUP function. (OK, maybe iconic is a bit of an exaggeration.)

Strange as it may sound, at the time I didn’t realize that there were any tools for data analysis besides Excel. Excel was ubiquitous: my parents used it, my teachers used it, and my colleagues used it. ...