COMP-627 Project Belief State Space Compression for Bayes-Adaptive POMDPs

Partially Observable Markov Decision Processes (POMDP) provide a nice mathematical framework for sequential decision making in partially observable stochastic environments. While it is generally assumed that the POMDP model is known, this is rarely the case in practice, as the parameters of the model must be finely tuned to reflect the reality as close as possible. Hence it is of crucial importance to develop new approaches which can take the uncertainty of these parameters into account during the planning process and further refine the model of the POMDP as experience is acquired in the environment. To this end, we have proposed in a previous project the Bayes-Adaptive POMDP model which allows one to both learn the POMDP model and plan by considering the model uncertainty and the value of learning information about the POMDP model. One problem with this model is that it has a infinite state space, which makes computing solution to this problem a very challenging task. To alleviate this problem, we propose a semimetric that allows one to approximate this infinite state space by a finite set of states that can preserve the value function of the infinite dimensionnal belief state space to arbitrary precision. This will allow us to define a new finite POMDP that we could theoretically solve using standard methods, which solution will be arbitrarily close to the optimal solution of the Bayes-Adaptive POMDP.