Introduction to High-Dimensional Statistics, 2nd Edition

Appendix E

Reproducing Kernel Hilbert Spaces

Reproducing Kernel Hilbert Spaces (RKHS) are some functional Hilbert spaces, where the smoothness of a function is driven by its norm. RKHS also fulfill a special “reproducing property” that is crucial in practice, since it allows efficient numerical computations as in Proposition 11.9 in Chapter 11.

A function k: X × X → ℝ is said to be a positive definite kernel if it is symmetric (k(x, y) = k(y, x) for all x, y ∈ X), and if for any N ∈ ℕ, x₁,..., x_N ∈ X and a₁,..., a_N ∈ ℝ we have

${\displaystyle \sum _{i,j=1}^N{a}_i{a}_{j}k\left(x_i,x_j\right)}\ge 0.$ (E.1)

Examples of positive definite kernels in X = ℝ^d:

• linear kernel: k(x, y) = <x, y>
• Gaussian kernel: k(x,y) = e^{−‖x-y‖2/2σ2}
• histogram kernel (d = 1): k(x,y ...