Optimal online learning : a Bayesian approachSara
A recently proposed Bayesian approach to online learning is applied to learning a rule deened as a noisy single layer perceptron. In the Bayesian online approach, the exact posterior distribution is approximated by a simple parametric posterior that is updated online as new examples are incorporated to the dataset. In the case of binary weights, the approximate posterior is chosen to be a biased binary distribution. The resulting online algorithm is shown to outperform several other online approaches to this problem. Neural networks are adaptive systems characterized by a set of parameters w, the weights and biases that specify the connectivity among the neuronal computational elements. Consider the case of multilayer architectures, which implement deterministic maps from an input space fxg onto an output space fyg. For a xed architecture, the properties of the speciic input-output map y = f w (x) implemented by the network depend only on the values selected for the network parameters w. Of particular interest is the ability of these systems to learn from examples. In the supervised learning scenario investigated here, each example is in the form of an input-output pair (x; y). A training set of size t is denoted by D t = f(x ; y); 1 tg; every example is an input-output pair labeled by the integer counting index. Traditional formulations of the learning problem are based on a dynamical prescription for the adaptation of the parameters w. The learning process 1 We acknowledge the support of CONNECT, the Computational Neural Network Center at the Niels Bohr Institute, Copenhagen, Denmark; CONNECT provided the hospitable environment where most of the discussions that led to this work took place.