Bayesian Voting Schemes as Large Margin Classifiers
It is often claimed that one of the main distinctive features of Bayesian Learning Algorithms for neural networks is that they don't simply output one hypothesis, but rather an entire distribution of probability over an hypothesis set: the Bayes posterior. An alternative perspective is that they output a linear combination of classiiers, whose coeecients are given by Bayes theorem. This can be regarded as a hyperplane in a high-dimensional feature space. We provide a novel theoretical analysis of such classiiers, based on data-dependent VC theory, proving that they can be expected to be large margin hyperplanes in a Hilbert space, and hence to have low eeective VC-dimension. This not only explains the remarkable resistance to overrtting exhibited by such classiiers, but also co-locates them in the same class as other systems, such as Support Vector Machines and Adaboost, which have a similar performance. 1.1 Introduction In recent years, a new method for training neural networks has been proposed and used, mainly due to the work of MacKay and Neal 7, 8, 9]. The systems inspired by this approach are generally know as Bayesian Learning Algorithms and have proven to be quite resistant to overrtting.