On Using a Benchmark to Evaluate C++ Extractors

In this paper, we take the concept of benchmarking as used extensively in computing and apply it to evaluating C++ fact extractors. We demonstrated the efficacy of this approach by developing a prototype benchmark, CppETS 1.0 (C++ Extractor Test Suite, pronounced see-pets) and collecting feedback in a workshop setting. The CppETS benchmark characterises C++ extractors along two dimensions: Accuracy and Robustness. It consists of a series of test buckets that contain small C++ programs and related questions that pose different challenges to the extractors. As with other research areas, benchmarks are best developed through technical work and consultation with a community, so we invited researchers to apply CppETS to their extractors and report on their results in a workshop. Four teams participated in this effort, evaluating Ccia, cppx, the Rigi C++ parser, and TkSee/SN. They found that CppETS gave results that were consistent with their experience with these tools and therefore had good external validity. Workshop participants agreed that CppETS was an important contribution to fact extractor development and testing. Further efforts to make CppETS a widely-accepted benchmark will involve technical improvements and collaboration with the broader community.