Bayesian Neural Networks with Maximum Mean Discrepancy regularization

Bayesian Neural Networks (BNNs) are trained to optimize an entire distribution over their weights instead of a single set, having significant advantages in terms of, e.g., interpretability, multi-task learning, and calibration. Because the intractability resulting optimization problem, most BNNs either sampled through Monte Carlo methods, or by minimizing suitable Evidence Lower BOund (ELBO) on variational approximation. In this paper, we propose variant latter, wherein replace Kullback-Leibler divergence ELBO term with Maximum Mean Discrepancy (MMD) estimator, inspired recent work inference. After motivating our proposal based properties MMD term, proceed show number empirical proposed formulation state-of-the-art. particular, achieve higher accuracy multiple benchmarks, including several image classification tasks. addition, they more robust selection prior weights, better calibrated. As second contribution, provide new for estimating uncertainty given prediction, showing it performs fashion against adversarial attacks injection noise inputs, compared classical criteria such as differential entropy.