A reproducing kernel Hilbert space (RKHS) is a Hilbert space of functions where the evaluation functional is continuous. It is equipped with a reproducing kernel $K(x, y)$ that satisfies: $f(x) = \langle f, K(x, \cdot) \rangle$ for all $f \in \mathcal{H}$, where $\mathcal{H}$ is the RKHS. This property ensures that the values of the functions in the space can be represented using the inner product with the kernel. The theory of RKHSs has profound implications in machine learning, statistical inference, and signal processing, where it provides a unified framework for various algorithms and models.
Motivation
In its essence, a reproducing kernel Hilbert space is a Hilbert space parametrizing some underlying space $\Omega$,^{1} equipped with a sensible kernel function measuring the similarity between elements of $\Omega$. Various kernels have been proposed for learning from different types of data such as vectors in $\mathbb{R}^n$, discrete classes/categories, graphs/networks, image arrays, time series, manifolds, dynamical systems, and other such structured objects.
Reproducing kernel Hilbert spaces provide a powerful framework for understanding and working with functions in infinitedimensional spaces. By employing kernel functions, we can implicitly map data into a higherdimensional space, allowing for more complex modeling and analysis. This approach has led to groundbreaking developments in regression, classification, and clustering among other things. The flexibility and generality of the RKHS framework have made it a central concept in modern statistical learning theory. Its applications extend beyond machine learning, impacting areas such as functional analysis, optimization, and signal processing, making study of RKHS in itself a rich and multifaceted area of study.
The concept of RKHS bridges the gap between linear and nonlinear methods, allowing for the application of linear techniques in a transformed space. The kernel trick, which leverages the reproducing property of the kernel to implicitly map data into a higherdimensional space, is a cornerstone of this approach. By working in the RKHS, we can develop powerful models that capture complex relationships in the data without explicitly dealing with the highdimensional space.
The analysis of distributions is fundamental in machine learning and statistics, and many algorithms in these fields rely on information theoretic approaches such as entropy, mutual information, and Kullback–Leibler divergence. However, to estimate these quantities, one must first either perform density estimation, or employ sophisticated spacepartitioning and biascorrection strategies which are typically infeasible for highdimensional data. Embedding using reproducing kernel Hilbert spaces sidesteps many of the problems when working with highdimensional numerical data. For instance, data can be modeled without restrictive assumptions about the form of distributions and relationships between variables. Furthermore, the flexibility in the choice of the kernel allows for the incorporation of prior knowledge, and so additional Hilbert space structures can be defined on specific properties of the distribution that are most relevant for the problem at hand.
Overview
Reproducing kernel Hilbert spaces are intimately connected with kernel functions, which are symmetric, positive definite functions. These functions allow us to define an inner product space, whose completion leads to the construction of a Hilbert space. The properties of kernel functions, such as Mercer's condition, play a crucial role in the theory and application of reproducing kernel Hilbert spaces.
Kernels and Their Embeddings
Main article: Kernel function
A kernel is a function $K: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ that satisfies the property of being symmetric and positive definite. Symmetry means that $K(x, y) = K(y, x)$ for all $x, y \in \mathcal{X}$. Positive definiteness ensures that for any finite set $\{x_1, \ldots, x_n\} \subset \mathcal{X}$, the corresponding Gram matrix $G = [K(x_i, x_j)]_{i,j=1}^n$ is positive semidefinite. Kernels provide a way to measure similarity between data points and are central to the concept of reproducing kernel Hilbert spaces.
The MooreAronszajn Theorem states that there is a unique RKHS $\mathcal{H}_K$ associated to every kernel $K$. One way to construct this RKHS is to simply take the completion of the inner product space consisting of $\{f:f(\cdot)=\sum_{i=1}^n\alpha_i K(\cdot,y_i), n\in\mathbb{Z}^+,\alpha_i\in\mathbb{R},y_i\in\mathcal{X}\}$ with the inner product between $f(\cdot)=\sum_{i=1}^n\alpha_i K(\cdot,y_i)$ and $g(\cdot)=\sum_{j=1}^m\beta_j K(\cdot,z_j)$ defined as $\langle f,g\rangle_K = \sum_{i=1}^n\sum_{j=1}^m\alpha_i\beta_j K(y_i,z_j)$. $K$ is referred to as a reproducing kernel because function evaluation may be performed by taking the inner product with the kernel function: $f(x)= \sum_{i=1}^n\alpha_i K(x,y_i) =\langle f,K(x,\cdot)\rangle_K$. Applying CauchySchwarz to this representation, we see that the evaluation functional is a linear and bounded operator, i.e. $f(x)\le \lVert f\rVert_K K(x,x)$, and so evaluation is continuous in the RKHS. Thus if two functions are close to each other in the RKHS, their evaluations will also be close.
Theorem. A function $k : \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ is a kernel function if and only if there exists a Hilbert space and a map $\Phi : \mathcal{X} \to \mathcal{H}$ such that $k(x,y) = \langle\Phi(x),\Phi(y)\rangle$. 
Mercer's Condition
Main article: Mercer's theorem
Mercer's condition is a crucial property of kernel functions that ensures the existence of an eigenfunction expansion. It states that a continuous, symmetric, and positive definite kernel $K(x, y)$ can be expressed as:
$$ K(x, y) = \sum_{n=1}^{\infty} \lambda_n \phi_n(x) \phi_n(y) $$
Where
 $\lambda_n$ are nonnegative eigenvalues
 $\phi_n(x)$ are corresponding eigenfunctions.
Mercer's condition guarantees that the kernel can be represented in terms of orthogonal functions, providing a solid foundation for the construction of reproducing kernel Hilbert spaces.
Applications
Machine Learning
In machine learning, the kernel embedding of distributions (also called the kernel mean or mean map) comprises a class of nonparametric methods in which a probability distribution is represented as an element of an RKHS. It serves as a generalization of the individual datapoint feature mapping done in classical kernel methods, the embedding of distributions into infinitedimensional feature spaces can preserve all of the statistical features of arbitrary distributions, while allowing one to compare and manipulate distributions using Hilbert space structure to define operations such as inner products, distances, projections, linear transformations. This learning framework is very general and permits a spectral analysis and learning of various spaces $\Omega$ on which a sensible kernel measuring similarity between elements of $\Omega$ can be defined.
See also:
 Hilbert space
 Kernel function
 Kernel embedding map
 Covariance function
 Mercer's Condition
 Eigenfunction Expansion
 Support Vector Machines (SVM)
 Autoencoders
 Statistical manifold
 Metric tensor
 Latent Space
 Natural Language Processing

