Oracle MCG: A first peek into COCO Detection Challenges

Microsoft COCO [2] is a new annotated database in computer vision consisting of more than 200.000 images. There are currently more than one million annotated objects from 80 categories, with fully segmented masks. With respect to Pascal [1], the previous available dataset with semantic segmentation annotations, COCO has four times the number of categories and two orders of magnitude more images and annotated objects. In this context, the challenges for object detection in COCO have recently been presented. In a nutshell, competing methods should provide a list of detections on each image in the form of bounding boxes or segmentation masks. Each detection should have associated one of the 80 categories and a confidence score. These new challenges on a new dataset are unchartered territory for researchers used to work on Pascal for some years. This work’s main aim is to set a reference point in the challenges to get a first grasp of which results to expect. To do so, we provide Oracle MCG, a hypothetical detector consisting of an oracle picking the best object proposal of a state-of-the-art technique on all annotated objects. This result could be understood as the upper-bound result one could get by restricting themselves to selecting object proposals without any further refinement. In particular, we use the publicly-available pre-computed results given by MCG [3] on the validation set of COCO (also available for the test set). On average, this method produces 5075 proposals per image. We then overlap all proposals on each image against all annotated objects and pick the best one for each of them. Each of these best proposals is associated to the annotated category and assigned a constant score of 1. The results obtained, as given by the COCO official evaluation software, are shown in Table 1 for bounding boxes and in Table 2 for segmentation masks. The highlighted row is the one that will define the ranking of the competition. Below some observations about the results.