Imitation Learning as f-Divergence Minimization

We address the problem of imitation learning with multi-modal demonstrations. Instead attempting to learn all modes, we argue that in many tasks it is sufficient imitate any one them. show state-of-the-art methods such as GAIL and behavior cloning, due their choice loss function, often incorrectly interpolate between modes. Our key insight minimize right divergence learner expert state-action distributions, namely reverse KL or I-projection. propose a general framework for estimating minimizing f-Divergence. By plugging different divergences, are able recover existing algorithms Behavior Cloning (Kullback-Leibler), (Jensen Shannon) DAgger (Total Variation). Empirical results our approximate I-projection technique behaviors more reliably than cloning.