Theory of Optimizing Pseudolinear Performance Measures: Application to F-measure

State of the art classification algorithms are designed to minimize the misclassification error of the system, which is a linear function of the per-class false negatives and false positives. Nonetheless non-linear performance measures are widely used for the evaluation of learning algorithms. For example, F -measure is a commonly used non-linear performance measure in classification problems. We study the theoretical properties of a subset of non-linear performance measures called pseudo-linear performance measures which includes F -measure, Jaccard index, among many others. We establish that many notions of F -measures and Jaccard index are pseudo-linear functions of the per-class false negatives and false positives for binary, multiclass and multilabel classification. Based on this observation, we present a general reduction of such performance measure optimization problem to cost-sensitive classification problem with unknown costs. We then propose an algorithm with provable guarantees to obtain an approximately optimal classifier for the F -measure by solving a series of cost-sensitive classification problems. The strength of our analysis is to be valid on any dataset and any class of classifiers, extending the existing theoretical results on binary F -score, which are asymptotic in nature. Our analysis shows that thresholding cost-insensitive scores, a common technique employed to optimize F -measure, yields sub-optimal results. We also establish the multi-objective nature of the F -measure maximization problem by linking the algorithm with the weighted-sum approach used in multi-objective optimization. We present numerical experiments to illustrate the relative importance of cost asymmetry and thresholding when learning linear classifiers on various F -measure optimization tasks.