Mastering Machine Learning with scikit-learn - Second Edition

Information gain

Testing for the animals that prefer cat food resulted in one subset with six cats, zero dogs, and zero bits of entropy, and another subset with two cats, six dogs, and 0.811 bits of entropy. How can we measure which of these tests reduced our uncertainty about the classification the most? Averaging the entropies of the subsets may seem to be an appropriate measure of the reduction in entropy. In this example, the subsets produced by the cat food test have the lowest average entropy. Intuitively, this test seems to be effective, as we can use it to classify almost half of the training instances.

However, selecting the test that produces the subsets with the lowest average entropy can produce a sub-optimal tree. For example, ...