Combining Regularized Neural Networks
In this paper we show that the improvement in performance which can be achieved by averaging depends critically on the degree of regularization which is used in training the individual neural networks. We compare four different averaging approaches: simple averaging, bagging, variance-based weighting and variance-based bagging. Bagging and variance-based bagging seem to be the overall best combining methods over a wide range of degrees of regularization.