Efficient Exploration in Reinforcement Learning with Hidden State
Undoubtedly, efficient exploration is crucial for the success of a learning agent. Previous approaches to exploration in reinforcement learning exclusively address exploration in Markovian domains, i.e. domains in which the state of the environment is fully observable. If the environment is only partially observable, they cease to work because exploration statistics are confounded between aliased world states. This paper presents Fringe Exploration, a technique for efficient exploration in partially observable domains. The key idea, (applicable to many exploration techniques), is to keep statistics in the space of possible shortterm memories, instead of in the agent’s current state space. Experimental results in a partially observable maze and in a difficult driving task with visual routines show dramatic performance improvements.