Abstract
In a POMDP (Partially Observable Markov Decision Process) problem, the Reinforcement Learning agent always has a chance to unable to distinguish two different state of the world, called perceptual aliasing, due to the limitation of sensory system. To solve this problem, some researchers have incorporated memory of preceding events to distinguish perceptually-aliased states. McCallum proposed Utile Suffix Memory (USM) [7], an instance-based method using a tree to store instances and to represent states. He use of a fringe (an extension of the tree to a pre-specified depth below the real tree) provides the algorithm a limited degree of lookahead capability. However, the use of a fringe make the tree hold more node in terms of tree size. We introduce a modification of USM to solve this issue without the use of fringe by using a different criterion with USM to split a leaf node. In our experiments, we have show that our method always produces trees that contain fewer nodes than USM and the agent learns a applicable policy.