Mean field theory for graphical models
In this chapter, mean eld theory is introduced from an information theoretic view point. The mean eld approximation is deened as the factorized distribution that is closest to the target distribution. When using the KL divergence to deene closeness, this factorized distribution must have equal marginals as the target distribution. Such marginals can be approximately computed by using a Taylor series expansion in the couplings around the factorized distribution. To lowest order in the couplings, the usual naive mean eld equations are obtained and to second order, one obtains the TAP equations. An important advantage of this procedure is that it does not require the concept of a free energy. Therefore, it can be applied to arbitrary probability distributions, such as arising in asymmetric stochastic neural networks and graphical models. 1.1 Introduction During the last few years, the use of probabilistic methods in artii-cial intelligence and machine learning has gained enormous popularity. In particular, probabilistic graphical models have become the preferred method for knowledge representation and reasoning 1]. The advantage of the probabilistic approach is that all assumptions are made explicit in the modeling process and that consequences, such as predictions on novel data, are assumption free and follow from a mechanistic computation. The drawback of the probabilistic approach is that the method is intractable. This means that the typical computation scales exponentially with the problem size. Recently, a number of authors have proposed methods for approximate inference in large graphical models. The simplest approach gives a lower bound on the probability of a subset of variables using Jenssen's inequality 2]. The method involves the minimization of the KL divergence between the target probability distribution p and some 'simple' variational distribution q. The method can be applied to a any probability model, whether directed or undirected. The Boltzmann-Gibbs distributions is widely used in physics, and mean eld theory has been known for these distributions for a long time. For instance, for the Ising model on a square lattice, it is known as the Bragg-Williams approximation 3] and it is generalized to other models in the Landau theory 4]. One can show that the above lower bound corresponds to the rst term in a Taylor series expansion of the free energy around a factorized model. This Taylor series can be continued and the second order term is known as the Thouless Anderson Palmer (TAP) correction 5, 6, 7, 8]. The second …