Machine Learning Algorithms - Second Edition

Cross-entropy impurity index

The cross-entropy measure is defined as follows:

This measure is based on information theory, and assumes null values when samples belonging to a single class are present in a split, while it is at its maximum when there's a uniform distribution among classes (which is one of the worst cases in Decision Trees because it means that there are still many decision steps until the final classification). This index is very similar to the Gini impurity, even though, more formally, cross-entropy allows you to select the split that minimizes uncertainty about the classification, while the Gini impurity minimizes the probability ...