Machine Learning and Big Data with kdb+/q

CHAPTER 23AdaBoost with Stumps

In this section, we introduce a particular machine learning technique called AdaBoost and apply it on stumps. We aim to provide a working example of the AdaBoost technique with a simple weak learner – a binary decision stump. There are numerous extensions of this technique as well as interesting theory to be covered, which are not within our scope. This chapter serves as a gentle introduction for the next section, which focuses on decision trees and related techniques.

23.1 BOOSTING

Boosting, in general, is one of the most powerful techniques for machine learning. In practice, we have the frequently available number of “weak” classifiers, each of them being able to explain a small fraction of the dependent variable. Forming a suitable committee of such weak classifiers which can appropriately combine their power will provide a considerable advantage.

In principle, the Boosting algorithm works as follows. Let us assume we have a set of weak classifiers denoted as , with and the data set. Boosting uses the following logic:

• Start with the original data set.
• Apply the first weak classifier using the original data set, adjust the data set by the outcome ...