Practical Predictive Analytics

Impurity

One a node is split according to the best criteria, the resulting nodes are examined for impurity. Impurity measures the separation of the classes, based upon what the expected frequencies should be at that point. The most impure case is when a node is split 50/50 for a binary class. This essentially designates a random class assignment. The least impure case is when a decision rule places all the observations completely in one class, and 0 observations are placed in the other class. This is the more desirable case, since it allows us to make a perfect prediction for that node.

Once an impurity measure is calculated, the algorithm will compute an information gain measure, which calculates how much the impurity decreases by splitting ...