Optimization of Entropy with Neural Networks
OF THE DISSERTATION Optimization of Entropy with Neural Networks by Nicol Norbert Schraudolph Doctor of Philosophy in Cognitive Science and Computer Science University of California, San Diego, 1995 Professor Terrence J. Sejnowski, Chair The goal of unsupervised learning algorithms is to discover concise yet informative representations of large data sets; the minimum description length principle and exploratory projection pursuit are two representative attempts to formalize this notion. When implementedwith neural networks, both suggest the minimization of entropy at the network’s output as an objective for unsupervised learning. The empirical computation of entropy or its derivative with respect to parameters of a neural network unfortunately requires explicit knowledge of the local data density; this information is typically not available when learning from data samples. This dissertation discusses and applies three methods for making density information accessible in a neural network: parametric modelling, probabilistic networks, and nonparametric estimation. By imposing their own structure on the data, parametric density models implement impoverished but tractable forms of entropy such as the log-variance. We have used this method to improve the adaptive dynamics of an anti-Hebbian learning rule which has proven successful in extracting disparity from random stereograms.