PolicyBlocks: An Algorithm for Creating Useful Macro-Actions in Reinforcement Learning
We present PolicyBlocks, an algorithm by which a reinforcement learning agent can extract useful macro-actions from a set of related tasks. The agent creates macroactions by finding commonalities in solutions to previous tasks. Using these macro-actions, learning to do future related tasks is accelerated. This increase in performance is illustrated in a “rooms” grid-world, in which the macro-actions found by PolicyBlocks outperform even hand designed macro-actions, and in a hydroelectric reservoir control task. We provide empirical comparisons of PolicyBlocks with the Reuse options of Bernstein (1999) and the SKILLS algorithm of Thrun and Schwartz (1995), which elucidate conditions under which each algorithm performs well.