Feature Engineering Made Easy

Removing harmful rows of data

Probably the most common and easiest of our two options for dealing with missing data is to simply remove the observations that have any missing values. By doing so, we will be left with only the complete data points with all data filled in. We can obtain a new DataFrame by invoking the dropna method in pandas, as shown in the following code:

# drop the rows with missing valuespima_dropped = pima.dropna()

Now, of course, the obvious problem here is that we lost a few rows. To check how many exactly, use the following code:

num_rows_lost = round(100*(pima.shape[0] - pima_dropped.shape[0])/float(pima.shape[0]))print "retained {}% of rows".format(num_rows_lost)# lost over half of the rows!retained 49.0% of rows ...