Machine Learning Algorithms

Feature importance

When growing a decision tree with a multidimensional dataset, it can be useful to evaluate the importance of each feature in predicting the output values. In Chapter 3, Feature Selection and Feature Engineering, we discussed some methods to reduce the dimensionality of a dataset by selecting only the most significant features. Decision trees offer a different approach based on the impurity reduction determined by every single feature. In particular, considering a feature x_i, its importance can be determined as:

 

The sum is extended to all nodes where x_i is used, and N_k is the number of samples reaching the node k. Therefore, ...