Bayesian Classifiers Are Large Margin Hyperplanes in a Hilbert Space
Bayesian algorithms for Neural Networks are known to produce classiiers which are very resistent to overrt-ting. It is often claimed that one of the main distinctive features of Bayesian Learning Algorithms is that they don't simply output one hypothesis, but rather an entire distribution of probability over an hypothesis set: the Bayes posterior. An alternative perspective is that they output a linear combination of classiiers, whose coeecients are given by Bayes theorem. One of the concepts used to deal with thresholded convex combinations is thèmargin' of the hyperplane with respect to the training sample, which is correlated to the predictive power of the hypothesis itself. We provide a novel theoretical analysis of such clas-siiers, based on Data-Dependent VC theory, proving that they can be expected to be large margin hyper-planes in a Hilbert space. We then present experimental evidence that the predictions of our model are correct , i.e. that bayesian classifers really nd hypotheses which have large margin on the training examples. This not only explains the remarkable resistance to overrtting exhibited by such classiiers, but also co-locates them in the same class of other systems, like Support Vector machines and Adaboost, which have a similar performance.