STUDENT PAPER : A Multiagent Reinforcement Learning Algorithm by Dynamically Merging Markov Decision Processes
ABSTRACT One general strategy for a elerating the learning of ooperative multiagent tasks is to reuse (good or optimal) solutions to the task when ea h agent is a ting alone. In this paper, we formalize this approa h as dynami ally merging solutions to multiple Markov de ision pro esses (MDPs), ea h representing an individual agent's solution when a ting alone, to obtain (good or optimal) solutions to the overall multiagent MDP when all the agents a t together. We present a new temporal-di eren e learning algorithm alled MAPLE (MultiAgent Poli y LEarning) that uses Q-learning and dynami merging to e iently onstru t global solutions to the overall multiagent problem from solutions to the individual MDPs. We illustrate the e ien y of MAPLE by omparing its performan e with standard Q-learning applied to the overall multiagent MDP. We also des ribe a orresponding planning algorithm that, given omplete knowledge of the underlying single agent MDPs, uses dynami merging to efiently solve the multiagent MDP. We also illustrate how the dynami merging framework an be extended to the ase when agents use temporally extended a tions, by using semiMarkov de ision pro esses (SMDPs) to represent variablelength de ision epo hs.