Building Machine Learning Systems with Python - Third Edition

How to measure and compare classifiers

How do we decide which classifier is best? Rarely do we find the perfect solution, the model that never makes any mistakes, so we need to decide which one to use. We used accuracy before, but sometimes it will be better to optimize so that the model makes fewer errors of a specific kind. For example, in spam filtering, it may be worse to delete a good email than to erroneously let a bad email through. In that case, we may want to choose a model that is conservative in throwing out emails rather than the one that just makes the fewest mistakes overall. We can discuss these issues in terms of gain (which we want to maximize) or loss (which we want to minimize). They are equivalent, but sometimes one is ...