Log-Likelihood

The log-likelihood is an essential concept in statistical modeling and machine learning, related to probability theory and information theory. It's a logarithmic transformation of the likelihood function, playing a key role in parameter estimation, model selection, and hypothesis testing. The log-likelihood is also related to the Kullback-Leibler divergence and to Boltzmann distribution as understood in context of energy-based models in machine learning.

Overview

The log-likelihood function is given by the natural logarithm of the likelihood function, expressed as:

$$ l(\theta) = \log L(\theta) = \sum_{i=1}^n \log f(x_i; \theta) $$

Here, $L(\theta)$ is the likelihood function, $f(x_i; \theta)$ is the probability density function (pdf), and $\theta$ is the vector of parameters. The log-likelihood is easier to work with computationally because it turns products into sums.

Properties

The log-likelihood has several important properties:

• Concavity: Under regular conditions, the log-likelihood is a concave function, leading to a unique maximum.
• Invariance: The maximum likelihood estimator (MLE) is invariant under transformation, and this invariance extends to the log-likelihood.
• Asymptotic Normality: Asymptotically, the MLE follows a normal distribution, and the log-likelihood has a well-known Fisher information matrix.

Applications

Applications of the log-likelihood include:

• Information Geometry: Log-likelihood and its derivatives lead to important geometric constructs like Fisher metric in the space of probability distributions. KL divergence can also be expressed in terms of the log-likelihood, as shown in the article on EBMs.
• Model Selection: Criteria like Akaike Information Criterion (AIC) and Bayesian Information Criterion are based on the log-likelihood.
• Hypothesis Testing: Likelihood ratio tests are constructed using the log-likelihood.
• Parameter Estimation: Maximizing the log-likelihood yields the MLE of the parameters.

Statistical Mechanics

In statistical mechanics, the log-likelihood plays a role in mean-field theory, connecting with free energy and entropy.

Energy-Based Models

Main article: Energy based model

In the context of energy-based models (EBMs), the log-likelihood is connected to the Boltzmann distribution. The negative log-likelihood is often utilized as a loss function to be minimized during training. An expression for the negative log-likelihood in EBMs is:

$$ -l(\mathbf{w}) = \langle E(\mathbf{x}; \mathbf{w})\rangle_q - F_\mathbf{w} $$

Where $\langle \cdot \rangle_q$ denotes an average with respect to the data distribution $q(\mathbf{x})$, and $F_\mathbf{w} = -\log Z_\mathbf{w}$ is the Helmholtz free energy of the model distribution $p(\mathbf{x}; \mathbf{w})$. Learning corresponds to maximizing the log-likelihood or minimizing the negative log-likelihood.

See Also

• Information theory
• KL divergence
• Maximum likelihood estimation
• Energy-based models

(edit) Topics in Information Theory and Control Theory (📁) Entropy Shannon entropy • Conditional entropy • Mutual Information • Joint entropy • Relative entropy • Rényi entropy • Tsallis entropy • Hartley's measure • Transfer entropy Information Geometry Fisher information (metric) • Exponential family • Kullback–Leibler divergence • Bregman divergence • f-divergence • Statistical divergence • Amari's alpha-divergence • Information manifold • Statistical structure • Cramér-Rao bound Model Selection Maximum likelihood estimation • Parameter space • Likelihood function (log-likelihood) • Bayesian inference • Akaike Information Criterion (AIC) • Bayesian Information Criterion (BIC) • Cross-validation • Bootstrap method • Minimum description length Network Information Theory Multiuser information theory • Broadcast channels • Multiple-access channels • Interference channels • Relay channels • Network coding • Slepian-Wolf coding Algorithmic Information Theory Kolmogorov complexity • Kolmogorov invariance theorem • Algorithmic probability • Chaitin's constant • Solomonoff's theory of inductive inference • Universal Turing machine Quantum Information Theory Quantum entropy • Quantum mutual information • Quantum channel capacity • Quantum error correction • Quantum cryptography Coding Theory Source Coding Shannon's source coding theorem • Huffman coding • Arithmetic coding • LZW coding • Run-length encoding • Golomb coding • Kraft-McMillan Inequality Channel Coding Shannon's channel coding theorem • Error detection and correction • Hamming codes • Reed-Solomon codes • Turbo codes • Convolutional codes • LDPC codes • Viterbi algorithm State Spaces State variables • State-transition matrix • Controllability • Observability • Stability • Kalman filter • Luenberger observer • Covariance intersection • Sensor fusion Control Theory Classical Transfer functions • Block diagrams • PID controllers • Root locus • Bode plots • Nyquist stability criterion • Routh-Hurwitz stability criterion Optimal Pontryagin's maximum principle • Hamilton-Jacobi-Bellman equation • Linear-quadratic regulator (LQR) • Dynamic programming • Model-predictive control Robust H-infinity methods • Sliding mode control • Robust stability • Uncertainty modeling • Youla–Kucera parametrization Adaptive Model reference adaptive control (MRAC) • Self-tuning regulator • Adaptive observers • Gain scheduling Nonlinear Feedback linearization • Backstepping • Passivity-based control • Lyapunov stability • Nonlinear system identification Stochastic Wiener filter • Stochastic optimal control • Markov decision processes • Stochastic stability • Stochastic differential equations Networks Networked Control System Architectures • Time Delay Systems • Multi-Agent Networks Applications Process control (Robotics, Aerospace, Automotive) • Biological Systems