Predicting Patch Correctness Based on the Similarity of Failing Test Cases

How do we know a generated patch is correct? This key challenging question that automated program repair (APR) systems struggle to address given the incompleteness of available test suites. Our intuition can triage correct patches by checking whether each implements code changes (i.e., behavior) are relevant bug it addresses. Such commonly specified failing case. Towards predicting correctness in APR, propose novel yet simple hypothesis on how link between behavior and specifications be drawn: similar cases should require . We then BATS , an unsupervised learning-based approach predict B ehavior A gainst T est S pecification. exploits deep representation learning models for patches: For case, yielded embedding used compute similarity metrics search historical identify associated applied patches, which as proxy assessing APR-generated patches. Experimentally, first validate our ground-truth developer cluster together same way their clustered. Then, after collecting large dataset 1,278 plausible (written developers or 32 APR tools), use achieves AUC 0.557 0.718 recall 0.562 0.854 identifying outperforms state-of-the-art techniques without need labeled datasets—as case with machine approaches. While constrained availability cases, show still complementary existing approaches: When combined recent relies supervised learning, improves overall detecting finally PATCH-SIM dynamic tools.