Age of Acquistion in Connectionist Networks - eScholarship
Age of Acquisition in Connectionist Networks Karen L. Anderson (kanders@cs.ucsd.edu) Garrison W. Cottrell (gary@cs.ucsd.edu) Computer Science and Engineering Department 0114 Institute for Neural Computation University of California, San Diego La Jolla, CA 92093-0114 USA Abstract Recently, there has been a resurgence of interest in the role of the Age of Acquisition (AoA) of an item in determining subjects’ reaction time in naming words, objects, and faces. Using the number of epochs required to learn an item as a direct measure of AoA in connectionist networks, Smith, Cottrell & Anderson (in press) have shown that AoA is a stronger predictor of final Sum Squared Error than frequency. In this paper, we replicate Smith et al. using more realistic frequency distributions for the items, and examine why some patterns may be learned earlier than others. First, we have found that the same patterns tend to be learned early and late by networks differing in their initial random weights; hence, the issue is, what property of the patterns determines AoA? We have found that even very weak pattern similarity structure is a strong predictor of AoA when frequency is controlled for. Also, we have found evidence that such a similarity structure may still be an important factor in determining AoA even when pattern frequency is varied. Introduction Ever since Carroll & White (1973) reanalyzed Oldfield & Wingfield' s (1965) naming latency data and discovered that frequency was not significant when AoA was considered, controversy has surrounded discussions of the import of the two variables. Technological and methodological refinements have led to agreement that both frequency and AoA play significant roles. Hence, interest has returned to the pursuit of understanding the mechanisms underlying AoA effects. It had been proposed recently (Morrison & Ellis, 1995; Moore & Valentine, 1998) that connectionist networks would be incapable of exhibiting AoA effects because training on late patterns would cause “catastrophic interference” resulting in the unlearning of early patterns. However, this sort of interference is only found if training on early patterns ceases. Ellis & Lambon Ralph (2000) demonstrated AoA effects in a neural network by training the net on an “early” set of patterns and then simply adding a second set of “late” patterns halfway through training. Smith et al. (in press) independently demonstrated AoA effects in networks. In contrast to the staging method of Ellis & Lambon Ralph (2000), where AoA is assumed to correspond to the time at which patterns are presented to the network (early or late), all patterns were presented to the model from the outset. AoA can then be measured for each pattern individually as the time during training when the pattern is learned. Using this more natural definition, Smith et al. reported significant effects of AoA on naming latency (defined as the residual error on a pattern after training is completed, a measure of the network’s uncertainty). What we would like to know is why certain patterns are learned earlier than others, and how early learning of a pattern comes to affect the network' s performance. Ellis & Lambon Ralph’s (2000) approach cannot be used to find out why patterns are acquired in a particular order as it imposes an order by staging pattern presentation. Instead, we vary properties of the patterns and then measure AoA, as in Smith et al. Ellis & Lambon Ralph (2000), do suggest why early AoA is important for final performance – the network is more “plastic” earlier in training, so items that are learned first have the opportunity to make the biggest impression on the weights. We also want to know whether our finding that AoA is a stronger predictor of final error than frequency survives a more realistic version of Zipf’s (1935) frequency distribution than was used by Smith et al. Here we show that it does. Methods Our investigation is organized around a series of experiments in which we replicate and extend network simulations and analyses previously reported by Smith et al. (in press). We begin with one of the simplest connectionist models of lexical access, an autoencoder network. This kind of network simply reproduces its input on its output through a set of hidden units, and has seen surprisingly wide application in cognitive modeling. We then extend our simulations to more complex mappings.