The Mathematics of Machine Learning

Part II Supervised learning

4 Statistical learning theory

In this chapter, we take the point of view of statistics to study learning theory. In short, statistics is concerned with estimating an unknown distribution μ from its observations. Typical questions can be: how many data points should we collect to ensure that an estimator $\hat{\mathit{\mu }}$ of μ is close enough (in some sense) to μ? How does the complexity of a model relate to its empirical error and its generalization error? More generally, we are interested in deriving approximation bounds of hypothesis classes.

Remark 4.0.1.

If you are familiar with numerical analysis, then you can think of this chapter as techniques to come up with a priori error estimates (i. e., before we sample the ...