Learning From Disagreements: Discriminative Performance Evaluation

Selecting test cases in order to evaluate computer vision methods is important, albeit has not been addressed before. If the methods are evaluated on examples on which they perform very well or very poorly then no reliable conclusions can be made regarding the superiority of one method versus the others. In this paper we put forth the idea that algorithms should be evaluated on test cases they disagree most. We present a simple method which identifies the test cases that should be taken into account when comparing two algorithms and at the same time assesses the statistical significance of the differences in performance. We employ our methodology to compare two object detection algorithms and demonstrate its usefulness in enhancing the differences between the methods.