Automatic Derivation of the Multinomial PCA Algorithm
Machine learning has reached a point where probabilistic methods can be understood as variations, extensions, and combinations of a small set of abstract themes, e.g., as different instances of the EM algorithm, or as exponential family methods. This allows the automatic derivation of algorithms customized for different models. One interesting new model is a multinomial version of PCA which has received attention due to its ability to better model documents as having multiple topics. Here we explain how AutoBayes, an automatic synthesis system for machine learning programs, derives from a high-level statistical specification of this model code similar to the original non-negative matrix factorization algorithm of Lee and Seung. The derivation combines multiple statistical schemes to solve a new problem not originally envisaged. This demonstrates that the approach can be scaled up from text-book problems to research-level algorithm design.