Pandas for Everyone: Python Data Analysis, 2nd Edition

17

Regularization

In Chapter 16, we considered various ways to measure model performance. Section 16.3 described k-fold cross-validation, a technique that tries to measure model performance by looking at how it predicts on test data. This chapter explores regularization, one technique to improve performance on test data. Specifically, this method aims to prevent overfitting.

17.1 Why Regularize?

Let’s begin with a base case of linear regression. We will be using the ACS data.

Click here to view code image

import pandas as pd
acs = pd.read_csv('data/acs_ny.csv')
print(acs.columns)

Index(['Acres', 'FamilyIncome', 'FamilyType', 'NumBedrooms', 'NumChildren', 'NumPeople', 'NumRooms', 'NumUnits', 'NumVehicles', 'NumWorkers', 'OwnRent', 'YearBuilt', ...