Java: Data Science Made Easy

Gradient Boosting Machines and XGBoost

Gradient Boosting Machines (GBM) is an ensembling algorithm. The main idea behind GBM is to take some base model and then  fit this model, over and over, to the data, gradually improving the performance. It is different from Random Forest models because GBM tries to improve the results at each step, while random forest builds multiple independent models and takes their average.

The main idea behind GBM can be best illustrated with a Linear Regression example. To fit several linear regressions to data, we can do the following:

1. Fit the base model to the original data.
2. Take the difference between the target value and the prediction of the first model (we call it the residuals of Step 1) and use this for ...