Tutorial on variational approximation methods
Tutorial topics • A bit of history • Examples of variational methods • A brief intro to graphical models • Variational mean field theory – Accuracy of variational mean field – Structured mean field theory • Variational methods in Bayesian estimation • Convex duality and variational factorization methods – Example: variational inference and the QMR-DT Variational methods • Classical setting: " finding the extremum of an integral involving a function and its derivatives " Example: finding the trajectory of a particle under external field • The key idea here is that the problem of interest is formulated as an optimization problem Variational methods cont'd • Variational methods have a long history in physics, statistics, control theory as well as economics. – calculus of variations (physics) – linear/non-linear moments problems (statistics) – dynamic programming (control theory) • Variational formulations appear naturally also in machine learning contexts: – regularization theory – maximum entropy estimation • Recently variational methods been used and further developed in the context of approximate inference and estimation Examples of variational methods • In classical examples the formulation itself is given but for us this is one of the key problems • We provide here a few examples that highlight 1. how to cast problems as optimization problems 2. how to find an approximate solution when the exact solution is not feasible • The examples we use involve a) finite element methods for solving differential equations b) large deviation methods (Chernoff bound)