Theoretical Approaches to On-line Learning in Smooth Multilayer Nets
On-line learning is one of the leading training techniques for neural networks. In this paper we will review recent theoretical approaches, based on methods from non-equilibrium statistical mechanics, for analysing on-line learning in multilayer neural network, concentrating on new approaches for defining optimal learning parameters or rules and on incorporating second order information into the learning process using natural gradient descent.