Now that we know how it works, let's talk at a high level about how it produces these rules. Essentially, trees produce their predictions by partitioning the feature space into rectangular regions. In the case of classification, it finds those splits that make the regions as homogeneous as possible. The method commonly used is called recursive binary splitting. Suppose we have two categories, purple and yellow, that we would like to classify. Let's say that we have two features and you would like to split one of them such that the space is divided into two regions. The goal is to make the two regions as "pure" or homogeneous as possible; you have to answer two questions:
- Which feature should you split?
- At which point should ...
