Policy Learning for Robust Markov Decision Process with a Mismatched Generative Model

In high-stake scenarios like medical treatment and auto-piloting, it's risky or even infeasible to collect online experimental data train the agent. Simulation-based training can alleviate this issue, but may suffer from its inherent mismatches simulator real environment. It is therefore imperative utilize learn a robust policy for real-world deployment. work, we consider learning Robust Markov Decision Processes (RMDP), where agent tries seek with respect unexpected perturbations on environments. Specifically, focus setting environment be characterized as generative model constrained perturbation added during testing. Our goal identify near-optimal perturbed testing environment, which introduces additional technical difficulties need simultaneously estimate uncertainty samples find worst-case To solve propose generic method formalizes an opponent obtain two-player zero-sum game, further show that Nash Equilibrium corresponds policy. We prove that, polynomial number of model, our algorithm high probability. able deal general under some mild assumptions also extended more complex problems partial observable decision process, thanks game-theoretical formulation.