7.1 INTRODUCTION TO MODEL EVALUATION
So far in Data Science Using Python and R, we have covered the first five phases of the Data Science Methodology:
- Data Understanding Phase
- Data Preparation Phase
- Exploratory Data Analysis Phase
- Setup Phase
- Modeling Phase (at least a little bit)
But, so far we have not examined whether our models are any good. That is, we have not evaluated their usefulness in making predictions. Note the difference between evaluation and validation. Model validation simply makes sure that our model results are consistent between the training and test data sets. But, model validation does not tell us how accurate our models are, or what their error rate is. For measures like these, we need to turn to model evaluation. Since the only models we have learned so far are decision trees for classification, we shall restrict our discussion to evaluative measures for classification models.
7.2 CLASSIFICATION EVALUATION MEASURES
We will develop classification evaluation measures for the case where we have a binary target variable. In order to apply the measures we will learn in this chapter, we will need to denote (arbitrarily, if desired) one of the two target outcomes as positive and one as negative. For example, suppose we are trying to predict income, a binary variable with values high income and low income. We could denote high income as positive and low income as negative.1
Now, the classification model evaluation measures we will ...