Learning in neural networks and an integrable system (Applied Mathematics of Discrete Integrable Systems)
This paper investigates the dynamics of batch learning of multilayer neural networks in the asymptotic case where the number of training data is much larger than the number of parameters. First, we present experimental results on the behavior in the steepest descent learning of multilayer perceptrons and three-layer linear neural networks. We see in these results that strong overtraining, which is the increase of generalization error in training, occurs if the model has surplus hidden units to realize the target function. Next, to analyze overtraining from the theoretical viewpoint, we analyze the steepest descent learning equation of a three-layer linear neural network, and theoretically show that a network with surplus hidden units presents overtraining. From this theoretical analysis, we can see that overtraining is not a feature observed in the nal stage of learning, but it occurs in an intermediate time interval.