Beyond API Signatures: An Empirical Study on Behavioral Backward Incompatibilities of Java Software Libraries

To make sure that existing client software applications are not broken after a library update, backward compatibility has always been one of the most important requirements during the evolution of software libraries. However, due to various reasons, backward compatibility is seldom fully achieved in practice, so it is important to understand the status, major reasons, and impact of backward incompatibilities in real world software. Previous studies related to this topic mainly focus on API signature changes between consecutive versions of software libraries, while in this paper, we mainly consider behavioral changes of APIs. Specifically, we performed large-scale cross-version regression testing on 68 consecutive version pairs from 15 most popular Java software libraries. Furthermore, we collected and studied 144 real world software bugs caused by backward incompatibilities of software libraries. Our major findings include: (1) more than 1,000 test failures / errors and 280 groups of behavioral backward incompatibilities are detected from 52 of 68 consecutive version pairs; (2) a large portion backward incompatibilities causing real-world bugs are related to user interface, which may be difficult to be detected by current automatic regression testing techniques; (3) the majority of backward incompatibilities are not well documented; and (4) there exists a number of fix patterns for behavioral backward incompatibilities.