Machine Learning Algorithms - Second Edition

Gradient Tree Boosting

Gradient Tree Boosting is a technique that allows you to build a tree ensemble step by step (the method is also known as forward stage-wise additive modeling), with the goal of minimizing a target loss function. The generic output of the ensemble can be represented as follows:

Here, c_j(x) is a function representing a weak learner (in this particular case, it's always a Decision Tree that can be modeled as a single parametrized function, f(•), where the vector, θ_i, groups all the splitting tuples of the i^th tree). The algorithm is based on the concept of adding a new Decision Tree at each step to minimize a global cost ...