Learning Data Mining with Python - Second Edition

Kernels

When the data cannot be separated linearly, the trick is to embed it on to a higher dimensional space. What this means, with a lot of hand-waving about the details, is to add new features to the dataset until the data is linearly separable. If you add the right kinds of features, this linear separation will always, eventually, happen.

The trick is that we often compute the inner-produce of the samples when finding the best line to separate the dataset. Given a function that uses the dot product, we effectively manufacture new features without having to actually define those new features. This is known as the kernel trick and is handy because we don't know what those features were going to be anyway. We now define a kernel as a function ...