A Performance Evaluation Protocol for Graphics Recognition Systems

1 I n t r o d u c t i o n Systems which convert existing paper-based drawings into electronic format are in demand and a few have been developed. However, the performance of the prototypes and commercial systems is either unknown, or only reported in a limited way by the system developers. An evaluation for these systems, or their subsystems, would contribute to the advancement of the field. Responding to this need, a dashed-line detection competition for developers of dashed-line detection algorithms was proposed and took place during the first International Workshop on Graphics Recognition at Penn State University, in 1995. A benchmark[l] was developed and used in that competition. That benchmark includes a performance evaluator and a software tool that automatically generates dashed-line test images and the corresponding groundtruth. In this paper, we extend that protocol to evaluate the performance of graphics recognition systems on images that contain straight lines (solid or dashed), circles (solid or dashed), partial arcs of circles (solid or dashed), and text blocks. (Engineering drawings primarily use a combination of these geometric elements. Therefore, despite being restricted to these simple entity types, the evaluation protocol is applicable to a wide variety of drawings. Upgrading the evaluator to handle other types of entities is straight forward. To do this, one needs to provide the evaluator the parameters of the entity or entities and the performance evaluation criteria.) To evaluate a given recognition system, the system is tested on a set of pre-selected test images. (The groundtruth for the test images must be reliable.) The results of the recognition system are matched, using the criteria defined in this protocol, with the corresponding groundtruth of the test