R Statistics Cookbook

Classification in caret and ROC curves

Classification models are harder to evaluate than regression models, because when we are classifying labels, we might have a severe imbalance. If, for example, we were predicting whether people are going to finish their university degrees or not, and 50% of people finish their degrees, the accuracy would be an ideal metric. But what happens when we have 95% of people finishing their degrees? In that case, the accuracy will be a very bad metric (maybe the model explains most of that 95% well, but doesn't work for the other 5% of the data).

There are several ways of assessing how well a classification model works that consider class imbalance. Apart from all these metrics, we can work with either ROC and ...