Extreme Attraction: On the Discrete Representation Preference of Attractor Networks
Attractor networks play an important role in many connectionist cognitive models. The dynamic behavior of these recurrent networks, which typically involves the convergence of activation levels to stable fixed-points over time, provides a means for modeling the time course of cognitive processes. Networks of this kind are often used as associative memories and as resolvers of soft constraint satisfaction problems. Early attractor networks used step activation functions, allowing only two activation values for each processing element. Contemporary networks, however, often use sigmoidal activation functions. Continuous outputs allow for continuous target patterns, and these have at least two apparent advantages: (1) they allow for richer representational schemes, such as mapping activations to probabilities, and (2) they allow for a larger number of attractors. Attractor network models, like those for lexical semantics (Clouse and Cottrell, 1995) and consciousness (Mathis and Mozer, 1995), would seem to benefit from continuous patterns, and other models using continuous vectors, like the CHARM memory model (Metcalfe Eich, 1982), would benefit from attractor dynamics. In this work we argue that these apparent advantages of continuous targets are illusory, at least when attractors are being learned using standard methods. Our simulation results indicate that attractor networks, even those with continuous activation functions, are best suited for use with target vectors consisting of polarized discrete elements.