Reproducing Kernel Hilbert Space

A reproducing kernel Hilbert space (RKHS) is a Hilbert space of functions where the evaluation functional is continuous. It is equipped with a reproducing kernel $K(x, y)$ that satisfies: $f(x) = \langle f, K(x, \cdot) \rangle$ for all $f \in \mathcal{H}$, where $\mathcal{H}$ is the RKHS. This property ensures that the values of the functions in the space can be represented using the inner product with the kernel. The theory of RKHSs has profound implications in machine learning, statistical inference, and signal processing, where it provides a unified framework for various algorithms and models.


In its essence, a reproducing kernel Hilbert space is a Hilbert space parametrizing some underlying space $\Omega$,1 equipped with a sensible kernel function measuring the similarity between elements of $\Omega$. Various kernels have been proposed for learning from different types of data such as vectors in $\mathbb{R}^n$, discrete classes/categories, graphs/networks, image arrays, time series, manifolds, dynamical systems, and other such structured objects.

Reproducing kernel Hilbert spaces provide a powerful framework for understanding and working with functions in infinite-dimensional spaces. By employing kernel functions, we can implicitly map data into a higher-dimensional space, allowing for more complex modeling and analysis. This approach has led to groundbreaking developments in regression, classification, and clustering among other things. The flexibility and generality of the RKHS framework have made it a central concept in modern statistical learning theory. Its applications extend beyond machine learning, impacting areas such as functional analysis, optimization, and signal processing, making study of RKHS in itself a rich and multifaceted area of study.

The concept of RKHS bridges the gap between linear and nonlinear methods, allowing for the application of linear techniques in a transformed space. The kernel trick, which leverages the reproducing property of the kernel to implicitly map data into a higher-dimensional space, is a cornerstone of this approach. By working in the RKHS, we can develop powerful models that capture complex relationships in the data without explicitly dealing with the high-dimensional space.

The analysis of distributions is fundamental in machine learning and statistics, and many algorithms in these fields rely on information theoretic approaches such as entropy, mutual information, and Kullback–Leibler divergence. However, to estimate these quantities, one must first either perform density estimation, or employ sophisticated space-partitioning and bias-correction strategies which are typically infeasible for high-dimensional data. Embedding using reproducing kernel Hilbert spaces sidesteps many of the problems when working with high-dimensional numerical data. For instance, data can be modeled without restrictive assumptions about the form of distributions and relationships between variables. Furthermore, the flexibility in the choice of the kernel allows for the incorporation of prior knowledge, and so additional Hilbert space structures can be defined on specific properties of the distribution that are most relevant for the problem at hand.


Reproducing kernel Hilbert spaces are intimately connected with kernel functions, which are symmetric, positive definite functions. These functions allow us to define an inner product space, whose completion leads to the construction of a Hilbert space. The properties of kernel functions, such as Mercer's condition, play a crucial role in the theory and application of reproducing kernel Hilbert spaces.

Kernels and Their Embeddings

Main article: Kernel function

A kernel is a function $K: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ that satisfies the property of being symmetric and positive definite. Symmetry means that $K(x, y) = K(y, x)$ for all $x, y \in \mathcal{X}$. Positive definiteness ensures that for any finite set $\{x_1, \ldots, x_n\} \subset \mathcal{X}$, the corresponding Gram matrix $G = [K(x_i, x_j)]_{i,j=1}^n$ is positive semidefinite. Kernels provide a way to measure similarity between data points and are central to the concept of reproducing kernel Hilbert spaces.

The Moore-Aronszajn Theorem states that there is a unique RKHS $\mathcal{H}_K$ associated to every kernel $K$. One way to construct this RKHS is to simply take the completion of the inner product space consisting of $\{f:f(\cdot)=\sum_{i=1}^n\alpha_i K(\cdot,y_i), n\in\mathbb{Z}^+,\alpha_i\in\mathbb{R},y_i\in\mathcal{X}\}$ with the inner product between $f(\cdot)=\sum_{i=1}^n\alpha_i K(\cdot,y_i)$ and $g(\cdot)=\sum_{j=1}^m\beta_j K(\cdot,z_j)$ defined as $\langle f,g\rangle_K = \sum_{i=1}^n\sum_{j=1}^m\alpha_i\beta_j K(y_i,z_j)$. $K$ is referred to as a reproducing kernel because function evaluation may be performed by taking the inner product with the kernel function: $f(x)= \sum_{i=1}^n\alpha_i K(x,y_i) =\langle f,K(x,\cdot)\rangle_K$. Applying Cauchy-Schwarz to this representation, we see that the evaluation functional is a linear and bounded operator, i.e. $|f(x)|\le \lVert f\rVert_K K(x,x)$, and so evaluation is continuous in the RKHS. Thus if two functions are close to each other in the RKHS, their evaluations will also be close.

Kernel Is Embedding

Theorem. A function $k : \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ is a kernel function if and only if there exists a Hilbert space and a map $\Phi : \mathcal{X} \to \mathcal{H}$ such that $k(x,y) = \langle\Phi(x),\Phi(y)\rangle$.

Mercer's Condition

Main article: Mercer's theorem

Mercer's condition is a crucial property of kernel functions that ensures the existence of an eigenfunction expansion. It states that a continuous, symmetric, and positive definite kernel $K(x, y)$ can be expressed as:
$$ K(x, y) = \sum_{n=1}^{\infty} \lambda_n \phi_n(x) \phi_n(y) $$

  • $\lambda_n$ are non-negative eigenvalues
  • $\phi_n(x)$ are corresponding eigenfunctions.

Mercer's condition guarantees that the kernel can be represented in terms of orthogonal functions, providing a solid foundation for the construction of reproducing kernel Hilbert spaces.


Machine Learning

In machine learning, the kernel embedding of distributions (also called the kernel mean or mean map) comprises a class of nonparametric methods in which a probability distribution is represented as an element of an RKHS. It serves as a generalization of the individual data-point feature mapping done in classical kernel methods, the embedding of distributions into infinite-dimensional feature spaces can preserve all of the statistical features of arbitrary distributions, while allowing one to compare and manipulate distributions using Hilbert space structure to define operations such as inner products, distances, projections, linear transformations. This learning framework is very general and permits a spectral analysis and learning of various spaces $\Omega$ on which a sensible kernel measuring similarity between elements of $\Omega$ can be defined.

See also:


Topics in Artificial Intelligence and Machine Learning


Topics in Information Theory and Control Theory

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License