Learning to Find Independent Components in Natural Scenes
The brain may operate in an arbitrarily complex way, while its self-organizational, or learning, processes, to the extent that they can be distinguished from normal operation, may be quite simple. Therefore one research program is to define learning rules using candidate abstract principles, and apply these algorithms to natural data to see if they produce processing systems similar to those found in neurons exposed to the same data. In this chapter, we apply information-theoretic learning to noiseless sigmoidal neurons exposed to natural images. The neurons learn localized oriented receptive fields qualitatively similar to simple cells in area V l of visual cortex. The algorithm maximizes the information contained about the input while choosing a coordinate system that makes each element of the resulting code as independent as possible. To a first approximation, the recoding of visual input in early perceptual processing may follow simple informationtheoretic principles. Both the classic experiments of Hubel and Wiesel (1968) o n neurons in visual cortex and several decades of theorizing about feature detection in vision (Marc and Hildreth 1980) have left open the question most succinctly phrased by Barlow and Tolhurst (1992) "Why do we have edge detectors?" Barlow (1989) has suggested that the line and edge selectivities of neurons found in primary visual cortex of cats and monkeys should emerge from an unsupervised learning algorithm that attempts to find a factorial code of independent visual features. Along similar lines, Field (1994) has argued that a sparse, distributed representation of natural scenes should be goal of early visual representations. These hypotheses can now be tested with unsupervised learning algorithms whose goal is to either find maximally independent linear filters (Bell and ~ejnowski 1995a, 1997) or to maximize sparseness (Olshausen and Field 1997), applied to an ensemble of natural scenes. Both of these approaches produce sets of visual filters that are localized and oriented, including some filters whose associated basis. functions are Gabor-like. These results are quite different from the filters produced by other decorrelating filters produced by principal components analysis (PCA) and zero-phase components analysis (ZCA). The set of filters produced by independent component analysis (ICA) has more sparsely distributed (kurtotic) otltputs on natural scenes (Bell and Sejnowski 1997). They also resemble the receptive fields of simple cells in visual cortex, which suggests that these neurons form a natural, information-theoretic coordinate system for natural images. Most studies that have examined the statistics of natural images for the purpose of reducing redundancy, and thereby making a more efficient code, have u$kd only the second-order statistics required for decorrelating the outputs of a set of feature detectors. Although Hebbian feature-learning algorithnis for decorrelation have been proposed (Linsker.1992; Miller 1988; Oja 1989; Sanger 1989; FoldiQ 1990; Atick and Redlich 1993), in the absence of particular external constraints the solutions to the decorrelation problem are nonunique. One popular decorrelating solution is principal components analysis (PCA), but the principal components of natdal scenes amount to Anthony J , Bell and Tevvence J. Sejnowski a global spatial frequency analysis (Hancock, Baddeley and Smith 1992). Thus, second-order statistics alone do not sufi& to predict the formation of localized edge detectors. Additional constraints are required. Field (1987, 1994) has argued for the importance of sparse, or "minimum entropy," coding (Barlow 1994), in which each feature detector is activated as rarely as possible. This has led to feature-learning algorithms with a "projection pursuit" flavor (Huber 1985, Intrator 1992, Baddeley 1996, Olshausen and Field 1997). An alternative constraint is to start with an information-theoretic criterion that maximizes the joint entropy of a nonlinearly transformed output feature vector. This is the approach taken by "independent components analysis" (Comon 1994) which can achieve the blind separation of mixed sources (Jutten and HCrault 1991; Bell and Sejnowski 1995a, 1996). Finding independent components is equivalent to Barlow's redundancy reduction problem; therefore if Barlow's reasoning is correct, the independent components should produce filters which are localized and oriented, and in fact it does. In addition, when applied to natural images, the outputs of the resulting filters are more sparsely distributed than those of other decorrelating filters, thus supporting some of the arguments of Field (1994) and helping to explain the results of Olshausen and Field (1997) from an information-theoretic point of view. We will return to the issues of sparseness, noise and higher-order statistics. First, we describe more concretely the filter-learning problem. 19.1 "Causes" in Natural Images A perceptual system is exposed to a series of small image patches, drawn from one or more larger images. Imagine that each image patch, represented by the vector x, has been formed by the linear combination of N basis functions. The basis functions form the columns of a futed matrix, A. The weigfiting of this linear combination (which varies with each image) is given by a vector, s. Each component of this vector has its own associated basis function, and represents an underlying "cause" of the image. he linkar image synthesis model is therefore given by: which is the matrix version of tke set of equations where each xi represents a pixel in an image, and contains contributions from each one of a set of N image "sources," 3, linearly weighted by a coeficient, a+ The goal of a perceptual system, in this simplified framework, is to linearly transform the images, x, with a matrix of filters, W, so that the resulting vector: recovers the underlying causes, s, possibly in a different order, and rescaled. Representing, by P, an arbitrary permutation matrix (all zero except for a single "one" in each row and each column), and, by S, an arbitrary scaling matrix (nonzero entries only on the diagonal), such a system has converged when: u = WAS = PSs (19.4) The scaling and permuting of the causes are arbitrary, unknowable factors, so consider the causes to be defined such that PS = I (the identity matrix). Learning to Find Independent Components Then the basis functions (columns of A) and the filters that recover the causes (rows of W) have the simple relation: W = A-' . All that remains in defining an algorithm to learn W (and thus also A) is to decide what constitutes a "cause." We concentrate here on algorithms producing causes that are decorrelated, and those attempting to produce causes that are statistically independent. 19.2 Decorrelation and Independence The matrix, W, is a decorrelating matrix when the covariance matrix of the output vector, u, satisfies: <uuT) = diagonal matrix (19.5) In general, there will be many W matrices which decorrelate. For example, when <uuT) = I, then: which clearly leaves freedom in the choice of W. There are, however, several special solutions to Eq. (19.6). Principal components analysis (PCA) is the orthogonal solution to Eq. (19.5). The principal components come from the eigenvectors of the covariance matrix, The fdters are orthogonal. When the image statistics are stationary (Field 1994), the PCA filters are global Fourier filters, ordered according to the amplitude spectrum of the image. Example PCA filters are shown in figure 19.1a. If W is forced to be symmetrical, so that W: = Wz, then the resulting decorrelating filters are zero-phase (ZCA). ZCA is in several ways the polar opposite of PCA. It produces local (center-surroqpd type) whitening filters, which are ordered according to the phase spectrum of the image. That is, each filter whitens a given pixel in the image, preserving the spatial arrangement of the image and flattening its frequency (amplitude) spectrum (Goodall 1960; Atick and Redlich 1993). Example ZCA filters and basis functions are shown in figure 19.lb. Another way to constrain the solution is to attempt to produce outputs that are not just decorrelated but statistically independent (Jutten and HCrault 1991; Comon 1994). The values of the ui are independent when their probability distribution, f,, factorizes: &(u) = n,f ,(ui) . There are many ICA algorithms, based on different approaches (Cardoso and Laheld 1996; Karhunen et al. 1996; Amari, Cichoki, and Yang 1996; Cichocki, Unbehauen, and Rummert 1994; Pham, Garrat, and Jutten 1992; Bell and Sejnowski 1995a). ICA produces decorrelating filters that are sensitive to both phase (locality) and frequency information, just as in transforms involving oriented Gabor functions (Daugman 19 85) or wavelets. These filters are thus semilocal, depicted in figure 19.2 as partway along the path from the local (ZCA) to the global (PCA) solutions in the space of decorrelating solutions. Example ICA fdters are shown in figure 19.ld and their corresponding basis functions are shown in figure 19.le. It is important to recognize two differences between finding an ICA solution, WI, and other decorrelation methods: (1) there may be no ICA solution, and (2) a given ICA algorithm may not find the solution even if it exists, because there are approximations involved. In these senses, ICA is different from PCA and ZCA, and cannot be calculated analytically, for example, from second-order statistics (the covariance matrix), except in the Gaussian case (when second-order statistics completely characterize the signal distribution). The approach developed in Bell and Sejnowski 1995a was to maximize by stocha&ic gradient ascent the joint entropy, H[g(u)], of the linear transform