MASSIVELY PARALLEL TRAINING OF MULTI LAYERPERCEPTRONS WITH IRREGULAR
|In this paper we present an approach to the training of feed forward neural networks on massively parallel SIMD-architectures. In order to cover a wide eld of applications we focus our attention on the exibility of the load balancing routines. Our approach is characterized by three important properties: 1. All four types of parallelism inherent in the training phase are used. 2. In a preprocessing step neural networks are transformed into equivalent topologies, more suited for parallel computation. 3. Each learning task can be parallelized in a number of diierent ways, the best of which is chosen according to estimations of the computing eeciency. Following these concepts we developed PINK 2 , a massively parallel simulator kernel for the MasPar MP1216. In contrast to most known approaches, ef-cient only for special topologies, it achieves good computing performance on a broad range of diiering benchmark problems. I. Introduction The vast requirements of computing power for the training of neural networks and their inherent parallelism made neural computation to an interesting and thoroughly investigated eld of application for parallel and massively parallel machines. Apart from dedicated hardware solutions for the training and recall phase of feed forward neural networks , the most promising results have been shown by SIMD-architectures with often some thousand processing elements (e.g. 1], 2], 3], 4], 5]). However, while attaining good peak values on special neu-ral networks their average performance on other network topologies is mostly comparatively poor. It depends heavily on the structure of the given learning problem, i.e. the number of networks to be trained, the number of training examples , the number and width of the network layers and the structure of interconnection between them. To put it another way: each parallelization is tacitly or explicitly tailored to a special structure of learning problems; their load balancing routines are designed for those special cases. As a result the scope of most massively parallel neural