Main Page

Welcome to a free encyclopedia on complexity sciences and information physics anyone can edit!
Read our Style Guide

IJBOL THEORY WIKI

複雑科学研究所

"Research is about friendship and having fun."

(edit the sidebar) • (edit this header) • (edit top row)]

$\sigma_\mu(A) = -\log\; \mu(A)$

Did you know that…
(edit this page)

Boltzmann machine, a predecessor of modern machine learning models, is based on a spin-glass model of Sherrington-Kirkpatrick's stochastic Ising Model.
Nerves, a construct of Čech cohomology theory, can be used to track down attractors in spike train activity in computational neuroscience?
The loss function landscape pertaining to can be thought of thermodynamically as an energy function over thermal degrees of freedom? Stochastic gradient descent then acts as a dissipative process introducing self-generated disorder.
Any real function on a cotangent bundle can be interpreted to be a Hamiltonian, so real-valued cotangent bundle is a phase space.
A more general form of Arrow's impossibility theorem can be shown to be true as a consequence of homotopy.
A recent model called rapid freeze neuroplasticity, a phenomenon similar to frozen random patterns in coupled map lattices pertaining to self-organized quenched disorder, was proposed as an explanation for neurodynamics of conspiracy theories?
Philip J. Anderson's treated emergence centrally through generalized rigidity, to be understood as a consequence of broken symmetry and the order parameter.
A pair of homologous cycles in a cellular sheaf has effective resistance as defined via the minimal-norm bounding chain. Effective resistance acts as a probability on graph cycles, thus allowing for random sampling to compress a sheaf with control of the eigenspectrum of the sheaf Laplacian. Cellular sheaves thus have natural variational approximations.

Recently Added Articles:
(edit this page)

Week of 7/31/2023 to 8/06/2023

Week of 8/07/2023 to 8/13/2023

Week of 8/14/2023 to 8/20/2023

Week of 8/21/2023 to 8/27/2023

In the works:
Cognitive Effort • Order parameter • Topological Data Analysis • Statistical mechanics • Equilibrium Statistical mechanics • Non-Equilibrium Statistical mechanics • Spin glass • Kuramoto Model • Neural Mean-Field Theory • Phase Space • Discourse sheaf • Čech cohomology • Čech complex • Topological Entropy • Metric Measure Space • Gromov–Hausdorff Distance (Gromov–Wasserstein Distance) • Holonomy • Kalman Filter • Algorithmic Probability • Cognitive Effort • Salience Network • Autoencoder • Taken's theorem • Manifold Learning • Kullback-Leibler divergence • Maximum Likelihood Estimation • Mean-Field Theory (Curie-Weiss Theorem, Bethe-Peierls Approximation)

Rewrite: Hopf Decomposition

Chosen Page:
(edit this page)

Energy-based model

Energy-based probabilistic models are closely related to physics, specified as a Boltzmann distribution (with the Boltzmann factor $$kT = 1$$ ):

$p(\mathbf{x}; \mathbf{w}) = \frac{1}{Z_\mathbf{w}}e^{E(\mathbf{x}; \mathbf{w})}$

The earliest energy-based probabilistic models in machine learning were in fact called Boltzmann machines, and map directly onto Ising spin models with a learned coupling structure $\mathbf{w}$ . Inserting the Boltzmann form into the log-likelihood learning objective $l(\mathbf{w}) = \int q(\mathbf{x}) \log(\mathbf{x}; \mathbf{w}) \mathrm{d}\mathbf{x}$ yields:

$-l(\mathbf{w}) = \langle E(\mathbf{x}; \mathbf{w})\rangle_q - F_\mathbf{w}$

Where $\langle \cdot \rangle_q$ denotes an average with respect to the data distribution $q(\mathbf{x})$ and $F_\mathbf{w} = -\log Z_\mathbf{w}$ is the Helmholtz free energy of the model distribution $p(\mathbf{x}; \mathbf{w})$ . Thus, learning via maximizing the log-likelihood corresponds to minimizing the energy of observed data while increasing overall free energy of the model distribution. Maximizing $l(\mathbf{w})$ is also equivalent to minimizing the Kullback-Leibler divergence,

$D_\mathrm{KL}(q\| p) = \int q(\mathbf{x}) \log \left( \frac{q(\mathbf{x})}{p(\mathbf{x}; \mathbf{w})}\right)\mathrm{d}\mathbf{x} = G_\mathbf{w}(q) - F_\mathbf{w}$

Kullback-Leibler divergence $D_\mathrm{KL}(q \| p)$ is nonnegative measure of the divergence between two distributions $$q$$ and $$p$$ that is zero if and only if $$q=p$$ . In the special case when $$p$$ takes the Boltzmann form, the KL divergence becomes the difference between the Gibbs free energy of $$q$$ , defined as $G_\mathbf{w}(q) = \langle E(\mathbf{x}; \mathbf{w})\rangle_q - S(q)$ (where $S(q) = -\int q(\mathbf{x})\log q(\mathbf{x}) \mathrm{d}\mathbf{x}$ is the entropy of $$q$$ ) and the Helmholtz free energy $F_\mathbf{w}$ of $$p$$ .