Discrete-time Markov control processes with discounted unbounded costs: Optimality criteria

We consider discrete-time Markov control processes with Borel state and control spaces, unbounded costs per stage and not necessarily compact control constraint sets. The basic control problem we are concerned with is to minimize the infinite-horizon, expected total discounted cost. Under easily verifiable assumptions, we provide characterizations of the optimal cost function and optimal policies, including all previously known optimality criteria, such as Bellman's Principle of Optimality, and the martingale and discrepancy function criteria. The convergence of value iteration, policy iteration and other approximation procedures is also discussed, together with criteria for asymptotic optimality.