Multi-Time Models for Reinforcement Learning Doina PrecupDepartment of Computer ScienceUniversity of MassachusettsAmherst
Reinforcement learning can be used not only to predict rewards, but also to predict states, i.e. to learn a model of the world's dynamics. Models can be deened at diierent levels of temporal abstraction. Multi-time models are models that focus on predicting what will happen, rather than when a certain event will take place. Based on multi-time models, we can deene abstract actions, which enable planning (presumably in a more eecient way) at various levels of abstraction. 1 Action models Model-based reinforcement learning ooers a possible solution to the problem of integrating planning in a real-time learning agent. Models are used to make predictions about the environment. The input of a model is a state and an action. The model should output a distribution of possible future states, as well as the expected value of the reward along the way. Since models provide action-dependent predictions, they also deene policies for achieving certain states, or for achieving maximum expected rewards. Reinforcement learning algorithms have traditionally been concerned with 1-step models, which assume that the agent interacts with its environment at some discrete , lowest-level time scale. We extend this framework by deening multi-time models, which describe the environment at diierent time scales. Multi-time models are a formalism for describing abstract actions. Intuitively, abstract actions should express complex behaviors, which can take variable amounts of time. Our main goal is to achieve faster learning and cheaper planning by using abstract actions in addition to primitive actions. As a consequence , the models describing abstract actions should be suitable for planning (i.e. should allow the development of \good" policies), expressive and learnable. 2 Value functions A key step in many reinforcement learning systems is the computation of state value functions. The value function for a policy associates to each state s the expected value of the total discounted reward that the agent gathers when starting in s and acting according to : V (s) = E f 1 X t=0 t r t+1 js 0 = sg; where 2 0; 1) is the discount-rate parameter. The optimal value function V associates to each state s the maximum expected reward that can be achieved when starting in s: V (s) = max V (s): An agent that acts greedily with respect to V will maximize its expected reward. Since value functions are the basis of planning in reinforcement learning systems , action models should …