Policy-contingent state abstraction for hierarchical MDPs
Hierarchically structured planning problems often provid e great opportunities for state abstraction: high-level planners can ig nore fine details, while low-level planners can focus only on specific tasks. Mo st previous hierarchical MDP algorithms rely on hand-designed, k nowledgeintensive state abstractions. We propose instead an automa tic, lazy algorithm which plans from the bottom of the hierarchy up and fin s good abstractions for high-level planners contingent on their s ubtasks’ optimal policies. This policy-contingent approach can often abstr act away more details than a human, specifically because it has access to mo re inf rmation (namely its subtasks’ policies) when it decides whet her o cluster states. We demonstrate that our algorithm finds better abstr actions than human designers on well-known planning tasks.