Towards Exploiting Duality in Approximate Linear Programming for MDPs

A weakness of classical methods for solving Markov decision processes is that they scale very poorly because of the flat state space, which subjects them to the curse of dimensionality. Fortunately, many MDPs are well-structured, which makes it possible to avoid enumerating the state space. To this end, factored MDP representations have been proposed (Boutilier, Dearden, & Goldszmidt 1995; Koller & Parr 1999) that model the state space as a cross product of state features, represent the transition function as a Bayesian network, and assume the rewards can be expressed as sums of compact functions of the state features. A challenge in creating algorithms for the factored representations is that well-structured problems do not always lead to compact and well-structured solutions (Koller & Parr 1999); that is, an optimal policy does not, in general, retain the structure of the problem. Because of this, it becomes necessary to resort to approximation techniques. Approximate linear programming (ALP) has recently emerged as a very promising MDP-approximation technique (Schweitzer & Seidmann 1985; de Farias & Roy 2003). As such, ALP has received a significant amount of attention, which has led to a theoretical foundation (de Farias & Roy 2003) and efficient solution techniques (e.g., (de Farias & Roy 2004; Guestrin et al. 2003; Patrascu et al. 2002)). However, this work has focused only on approximating the primal LP, and no effort has been invested in approximating the dual LP, which is the basis for solving a wide range of constrained MDPs (e.g., (Altman 1999; Dolgov & Durfee 2004)). Unfortunately, as we demonstrate, linear approximations do not interact with the dual LP as well as they do with the primal LP, because the constraint coefficients cannot be computed efficiently (the operation does not maintain the compactness of the representation). To address this, we propose an LP formulation, which we call a composite ALP, that approximates both the primal and the dual optimization coordinates (the value function and the occupation measure), which is equivalent to approximating both the objective functions and the feasible regions of the LPs. This method provides a basis for efficient approximations of constrained MDPs and also serves as a new approach to a widely-discussed problem of dealing with exponentially many constraints in ALPs, which plagues both the primal