Machine Learning, 2nd Edition

Chapter 6Combining Classifiers

For each individual among the many has a share of virtue and prudence, and when they meet together, they become in a manner one man, who has many feet, and hands, and senses; that is a figure of their mind and disposition. Hence the many are better judges than a single man of music and poetry; for some understand one part, and some another, and among them they understand the whole.

— Aristotle, Politics, 350 BCE

The methods described in Chapters 3 and 4 produce regressors or classifiers that are not naturally decomposable into collections of smaller regressors or classifiers. As such, these regressors and classifiers are called base learners or primitive learners – although perhaps a more descriptive term would be atomic learners. This chapter introduces classification methods that combine base learners in various ways. The design of specific methods is usually based on an understanding of the bias-variance trade-off described in Chapter 5.

6.1 Ensembles

If one is willing to do the work to train classifiers, of any type or types, one can get a new classifier by having the classifiers “vote” as a committee. That is, one can classify a new data point by applying each classifier to it, summing up the votes cast for each class , and predicting ...