Hands-On Data Analysis with Pandas

Gradient boosting

Boosting looks to improve upon the mistakes of previous models. One way of doing this is to move in the direction of the steepest reduction in the loss function for the model. Since the gradient (the multi-variable generalization of the derivative) is the direction of steepest ascent, this can be done by calculating the negative gradient, which yields the direction of steepest descent, meaning the best improvement in the loss function from the current result. This technique is called gradient descent.

Although gradient descent sounds great, there are some potential issues with it. It is possible that we end up in a local minimum (a minimum in a certain region of the cost function) and the algorithm stops, thinking that we ...