The Mathematics of Machine Learning

Part III Beyond supervised learning

10 Topics in unsupervised learning

So far, we have always assumed that we have access to a data-generating process D that provides a pair of inputs and outputs $\mathrm{(}\mathit{x}\mathrm{,}\mathit{y}\mathrm{)}$, thus providing labeled data. In this section, we assume that we no longer have access to labeled data. We have access to a set of datapoints $\mathcal{X}={\{{\mathit{x}}_{\mathit{i}}\}}_{\mathit{i}=1}^{\mathit{m}}$ and want to make sense of them. More formally, unsupervised learning tasks often correspond to minimizing a loss function of the form

$\begin{array}{r}\mathit{L}\mathrm{(}\mathit{f}\mathrm{)}={\displaystyle \frac{1}{\mathit{m}}}\sum _{\mathit{i}=1}^{\mathit{m}}\ell \mathrm{(}\mathit{f}\mathrm{(}{\mathit{x}}_{\mathit{i}}\mathrm{)}\mathrm{,}{\mathit{x}}_{\mathit{i}}\mathrm{)}\mathrm{,}\end{array}$

where $\{{\mathit{x}}_{\mathit{i}};\mathit{i}=1\mathrm{,}\dots \mathrm{,}\mathit{m}\}$ is the unlabeled dataset, and ℓ is a nonnegative map measuring the error at one datapoint. This is similar to the empirical risk minimization framework in supervised learning, that is, (1.3) with ${\mathit{x}}_{\mathit{i}}$ in place ...