Turbo Factor Analysis
A few years ago, Pearl’s probability propagation algorithm in graphs with cycles was shown to produce excellent results for error-correcting “turbodecoding”. Ever since, we have wondered whether iterative probability propagation could be used successfully for machine learning. As a first step in this direction, we study iterative inference and learning in the simple factor analyzer network – a two-layer densely connected network that models bottom layer sensory inputs as a linear combination of top layer factors plus independent Gaussian sensor noise. The number of bottom-up/top-down iterations needed to exactly infer the factors given a network and an input scales with the number of factors in the top layer. In online learning, this iterative procedure must be reinitialized upon each pattern presentation and so learning becomes prohibitively slow in big networks, such as those used for face recognition and for large-scale models of the cortex. We show that probability propagation in a factor analyzer usually takes just a few iterations to achieve a low inference error, even in networks with 320 sensors and 80 factors. We derive an expression for the algorithm’s fixed point and provide an eigenvalue condition for global convergence. We also show how iterative inference can be used to do online learning and give results on using this method to do online dimensionality reduction for the purpose of recognizing 560-pixel images of faces using a 40-dimensional subspace. This work suggests that iterative probability propagation in densely connected networks may lead to a broad class of useful algorithms for machine learning. UW/CS Adaptive Computation TR-99-1, Apr 10, 1999. Submitted to Neural Computation, May 25, 1999.