Machine Learning Algorithms - Second Edition

Feature importance

When growing a Decision Tree with a multidimensional dataset, it can be useful to evaluate the importance of each feature in predicting the output values. In Chapter 3, Feature Selection and Feature Engineering, we discussed some methods to reduce the dimensionality of a dataset by selecting only the most significant features. Decision Trees offer a different approach based on the impurity reduction determined by every single feature. In particular, considering a feature, x⁽ⁱ⁾, its importance can be determined as follows:

The sum is extended to all nodes where x⁽ⁱ⁾ is used, and N_k is the number of samples reaching the node, ...