Taxonomy of Large Margin Principle Algorithms for Ordinal Regression Problems

We discuss the problem of ranking instances where an instance is associated with an integer from 1 to k. In other words, the specialization of the general multi-class learning problem when there exists an ordering among the instances — a problem known as “ordinal regression” or “ranking learning”. This problem arises in various settings both in visual recognition and other information retrieval tasks. In the context of applying a large margin principle to this learning problem, we introduce two main approaches for implementing the large margin optimization criteria for k 1 margins. The first is the “fixed margin” policy in which the margin of the closest neighboring classes is being maximized — which turns out to be a direct generalization of SVM to ranking learning. The second approach allows for k 1 different margins where the sum of margins is maximized, thus effectively having the solution biased towards the pairs of neighboring classes which are the farthest apart from each other. This approach is shown to reduce to SVM when the number of classes k = 2. Both approaches are optimal in size (of the dual functional) of 2l where l is the total number of training examples. Experiments performed on visual classification and “collaborative filtering” show that both approaches outperform existing ordinal regression algorithms applied for ranking and multi-class SVM applied to general multi-class classification.