Information Divergence Measures and Surrogate Loss Functions

In this extended abstract, we provide an overview of our recent work on the connection between information divergence measures and convex surrogate loss functions used in statistical machine learning. Further details can be found in the technical report [7] and conference paper [6]. The class of f -divergences, introduced independently by Csiszar [4] and Ali and Silvey [1], arise in many areas of information theory and statistics. For instance, they frequently play the role of error exponents in an asymptotic setting [3]. These connections have motivated various researchers from the 1960’s onwards—studying problems such as signal selection or quantizer design in hypothesis testing—to advocate maximizing various types of f -divergence measures as a computationally feasible alternative to the intractable problem of minimizing the probability of error directly [5], [8]. On the other hand, convex surrogates play an important role in the binary classification problem that is studied in statistical machine learning. Here the goal is to design a hypothesis testing procedure when the underlying distributions are unknown, but the learner has access to labeled samples from both classes. Any such procedure for learning a decision rule is said to be consistent if it achieves the Bayes-optimal misclassification error as the number of samples grows. A unifying theme in the recent literature on statistical learning is the notion of a surrogate loss function [2], [10]—meaning a convex upper bound on the 0-1 loss. Many practical and widely-used algorithms for learning classifiers can be formulated in terms of minimizing empirical averages of such surrogate loss functions. Our work [7], [6] establishes a general correspondence between the class of f -divergences, and the family of surrogate loss functions (see Fig. 1). This correspondence has a number of interesting consequences. First, it partitions the set of surrogate loss functions into a set of equivalence classes, defined by the relation of inducing the same f -divergence measure. Second, it allows various well-known inequalities between different f -divergences [9] to be leveraged in analyzing surrogate φ1