Reinforcement Learning of Dimensional Attention for Categorization - eScholarship
Reinforcement Learning of Dimensional Attention for Categorization Joshua L. Phillips & David C. Noelle JOSHUA . L . PHILLIPS , DAVID . NOELLE @ VANDERBILT . EDU ) Vanderbilt University Nashville, TN 37235 USA Abstract The ability to selectively focus attention on stimulus di- mensions appears to play an important role in human cat- egory learning. This insight is embodied by learned di- mensional attention weights in the ALCOVE model (Kr- uschke, 1992). The success of this psychological model suggests its use as a foundation for efforts to under- stand the neural basis of category learning. One obsta- cle to such an effort is ALCOVE’s use of the biologi- cally implausible backpropagation of error algorithm to adapt dimensional attention weights. This obstacle may be overcome by replacing this attention mechanism with one grounded in the reinforcement learning processes of the brain’s dopamine system. In this paper, such a biologically-based mechanism for dimensional attention is proposed, and the fit of this mechanism to human per- formance is shown to be comparable to that of ALCOVE. Introduction Human category learning performance cannot be easily explained without recourse to a mechanism for selec- tive dimensional attention (Shepard et al., 1961). Di- mensional attention is the cognitive process which em- phasizes task relevant stimulus dimensions while deem- phasizing others. Thus, contemporary formal models of categorization, such as the Generalized Context Model (GCM) (Nosofsky, 1984), have incorporated adaptable dimensional attention parameters. By adjusting these pa- rameters in a category-specific fashion, the GCM has re- peatedly provided excellent fits to human data reflecting the frequency (or probability) with which each stimulus is recognized as an instance of a target category. When the GCM is applied to experimental results, dimensional attention parameters are freely varied to optimize the model fit. This means that, while the GCM provides a powerful account of learned categorization performance, it offers no explanation for how dimensional attention is adjusted over the course of learning. This shortcoming of the GCM has been addressed by a connectionist model called ALCOVE (Kruschke, 1992). ALCOVE incorporates the GCM’s formalization of cat- egory knowledge, but it also provides a precise algo- rithm for modifying the attentional “weight” assigned to each stimulus dimension, based on feedback provided to learners on their categorization judgments. In a typ- ical category learning experiment, learners are presented with stimulus objects, one at a time, and are asked to make classification judgments for each. Immediately fol- lowing each judgment, feedback is provided, typically informing the learner of the correct category label for the preceding stimulus. Once learning is complete, catego- rization judgments on transfer stimuli, for which no feed- back is provided, can provide a window into the structure of the learned category knowledge. The ALCOVE model uses the feedback provided during training to calculate an “error signal”, which is simply the difference between the category assignment made by the model and the spec- ified “true” category. A variant of the backpropagation of error learning algorithm (Rumelhart et al., 1986) is used to communicate this error signal to an early stage of stimulus encoding, and this backpropagated error sig- nal is used to adjust ALCOVE’s dimensional attention weights. Like the GCM, ALCOVE provides good fits to human performance data on learned categories. Unlike the GCM, ALCOVE provides a detailed account of how dimensional attention is shaped by experience. ALCOVE has been proposed as a model of psycho- logical processes, with virtually no aspiration to explain the neural basis of human category learning. Despite this fact, the empirical successes of ALCOVE and its con- nectionist formalization make the model a tempting can- didate for a coarse characterization of associated brain mechanisms. Perhaps ALCOVE can be refined, with each of its proposed psychological mechanisms mapped onto a corresponding detailed account of the underlying neural machinery. One feature of ALCOVE that stands in the way of such a theoretical reduction is its use of the backpropagation of error algorithm in order to learn dimensional attention weights. This powerful learning algorithm has long been criticized for its lack of biolog- ical plausibility (Crick, 1989), suggesting that the brain cannot be adapting dimensional attention based on such a gradient-based technique (c.f., O’Reilly (1996)). As a first step toward a biological model of category learning, we replaced the backpropagation-based dimen- sional attention mechanism used by ALCOVE with a re- inforcement learning mechanism intended to reflect the role of the brain’s dopamine (DA) system in learning. This role for dopamine has been formalized by other re- searchers in terms of an algorithm called temporal dif- ference (TD) learning (Sutton, 1988; Montague et al., 1996). Versions of ALCOVE which adapt dimensional attention weights using the biologically supported TD