Book Reviews
Reinforcement learning, as understood by Sutton and Barto, is a fusion of the trial-and-error “law-of-effect” tradition in psychology, optimal control theory in engineering, the secondary reinforcement tradition in learning, and the use of decaying stimulus traces in, for example, Hull’s (1952) concept of a goal gradient and, more recently, Wagner’s (1981) model of conditioning. This fusion has given researchers in artiacial intelligence a number of ideas for computer algorithms that learn a policy that maximizes the agent’s long term return (amount of reward) from the performance of a task. Although many of the ideas behind reinforcement learning originated in psychological theorizing, in recent years these ideas have been most extensively developed within the artiacial learning community, particularly by the authors of this important summary, their students, and colleagues. The book is intended for use in a one-semester course for students interested in machine learning and artiacial intelligence. It would probably not be suitable for a course intended for psychology and neuroscience students, because it does not present models of experimentally established behavioral or neuroscientiac phenomena, and the problems given to illustrate how reinforcement learning algorithms may be applied are not necessarily problems that animals (even human animals) are notably good at solving (e.g., efacient scheduling problems). However, reinforcement learning, incentive, and utility remain central concepts in contemporary work on the neurobiological basis of learned, goaldirected behavior (Schultz, Dayan, & Montague, 1997; Shizgal, 1997), and this book is the place to look for the latest ideas on how these concepts may be developed into effective models for the direction of action. The preface says that the only mathematical background assumed is familiarity with elementary concepts of probability, such as expectations of random variables. Most students will feel that rather more than that is in fact assumed. Nonetheless, the material is presented in an intuitively understandable form, emphasizing the basic ideas and giving helpful illustrations of their application, rather than elaborating proofs. It can be read with proat by motivated neuroscience and psychology students interested in a more rigorous development of these psychologically important ideas. The biographical and historical sections at the end of each chapter are useful for the perspective they give. The many suggested exercises are challenging, open ended, and thought provoking.