Combining Pattern Classifiers: Methods and Algorithms, 2nd Edition

8 DIVERSITY IN CLASSIFIER ENSEMBLES

Common sense suggests that the classifiers in the ensemble should be as accurate as possible and should not make coincident errors. Ensemble-creating methods which rely on inducing diversity in an intuitive manner have proven their value. Even weakening the individual classifiers for the sake of better diversity appears to be an excellent ensemble building strategy, unequivocally demonstrated by the iconic AdaBoost. Ironically, trying to measure diversity and using it explicitly in the process of building the ensemble does not share the success of the implicit methodologies.

8.1 WHAT IS DIVERSITY?

If we have a perfect classifier which makes no errors, then we do not need an ensemble. If however, the classifier does make errors, then we seek to complement it with another classifier which makes errors on different objects. The diversity of the classifier outputs is therefore a vital requirement for the success of the ensemble. However, diversity alone is not responsible for the ensemble performance. It is intricately related with other characteristics of the ensemble. For example, individual classifier accuracy can be sacrificed in order to make the classifiers more diverse. This pays off by a more accurate ensemble compared to that employing the more accurate classifiers. Then how do we strike a compromise between diversity and individual accuracy? How far can we “shake” the ensemble members without destroying the ensemble performance? How ...