research-article

Open access

Understanding Breaking Changes in the Wild

Authors:

Dhanushka Jayasuriya,

Valerio Terragni,

Kelly BlincoeAuthors Info & Claims

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 1433 - 1444

https://doi.org/10.1145/3597926.3598147

Published: 13 July 2023 Publication History

Abstract

Modern software applications rely heavily on the usage of libraries, which provide reusable functionality, to accelerate the development process. As libraries evolve and release new versions, the software systems that depend on those libraries (the clients) should update their dependencies to use these new versions as the new release could, for example, include critical fixes for security vulnerabilities. However, updating is not always a smooth process, as it can result in software failures in the clients if the new version includes breaking changes. Yet, there is little research on how these breaking changes impact the client projects in the wild. To identify if changes between two library versions cause breaking changes at the client end, we perform an empirical study on Java projects built using Maven. For the analysis, we used 18,415 Maven artifacts, which declared 142,355 direct dependencies, of which 71.60% were not up-to-date. We updated these dependencies and found that 11.58% of the dependency updates contain breaking changes that impact the client. We further analyzed these changes in the library which impact the client projects and examine if libraries have adhered to the semantic versioning scheme when introducing breaking changes in their releases. Our results show that changes in transitive dependencies were a major factor in introducing breaking changes during dependency updates and almost half of the detected client impacting breaking changes violate the semantic versioning scheme by introducing breaking changes in non-Major updates.

References

[1]

Mahmoud Alfadel, Diego Elias Costa, and Emad Shihab. 2021. Empirical Analysis of Security Vulnerabilities in Python Packages. In International Conference on Software Analysis, Evolution and Reengineering (SANER ’21). IEEE, 446–457. https://doi.org/10.1109/SANER50967.2021.00048

[2]

Gabriele Bavota, Gerardo Canfora, Massimiliano Di Penta, Rocco Oliveto, and Sebastiano Panichella. 2013. The Evolution of Project Inter-Dependencies in a Software Ecosystem: The Case of Apache. ICSM ’13. IEEE, 280–289. isbn:9780769549811 https://doi.org/10.1109/ICSM.2013.39

Digital Library

[3]

Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, and Dawson Engler. 2010. A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World. Commun. ACM, 53, 2 (2010), feb, 66–75. issn:0001-0782 https://doi.org/10.1145/1646353.1646374

Digital Library

[4]

Aline Brito, Marco Tulio Valente, Laerte Xavier, and Andre Hora. 2020. You broke my code: understanding the motivations for breaking changes in APIs. Empirical Software Engineering, 25, 2 (2020), 1458–1492. https://doi.org/10.1007/s10664-019-09756-z

[5]

Aline Brito, Laerte Xavier, Andre Hora, and Marco Tulio Valente. 2018. APIDiff: Detecting API breaking changes. In 25th International Conference on Software Analysis, Evolution and Reengineering (SANER ’18). IEEE, 507–511. https://doi.org/10.1109/SANER.2018.8330249

[6]

Aline Brito, Laerte Xavier, Andre Hora, and Marco Tulio Valente. 2018. Why and how Java developers break APIs. In 25th International Conference on Software Analysis, Evolution and Reengineering (SANER ’18). 255–265. https://doi.org/10.1109/SANER.2018.8330214

[7]

Eric Bruneton, Eugene Kuleshov, Andrei Loskutov, and Rémi Forax. 2022. ASM. https://asm.ow2.io/

[8]

Joel Cox, Eric Bouwers, Marko van Eekelen, and Joost Visser. 2015. Measuring Dependency Freshness in Software Systems. In International Conference on Mobile Software Engineering and Systems (MOBILESoft ’15). IEEE, 109–118. https://doi.org/10.1109/ICSE.2015.140

[9]

Daniela S. Cruzes and Tore Dyba. 2011. Recommended Steps for Thematic Synthesis in Software Engineering. In 2011 International Symposium on Empirical Software Engineering and Measurement(ESEM). 275–284. https://doi.org/10.1109/ESEM.2011.36

Digital Library

[10]

Joe Darcy. 2021. Kinds of Compatibility. https://wiki.openjdk.org/display/csr/Kinds+of+Compatibility

[11]

Alexandre Decan and Tom Mens. 2021. What Do Package Dependencies Tell Us about Semantic Versioning? IEEE Transactions on Software Engineering, 47, 6 (2021), 6, 1226–1240. issn:19393520 https://doi.org/10.1109/TSE.2019.2918315

[12]

Jim des Rivières. 2017. Evolving Java-based APIs 2. https://wiki.eclipse.org/Evolving_Java-based_APIs_2

[13]

Jens Dietrich, Kamil Jezek, and Premek Brada. 2014. Broken promises: An empirical study into evolution problems in Java programs caused by library upgrades. In Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE 14). 64–73. https://doi.org/10.1109/CSMR-WCRE.2014.6747226

[14]

Jens Dietrich, David Pearce, Jacob Stringer, Amjed Tahir, and Kelly Blincoe. 2019. Dependency Versioning in the Wild. In 16th International Conference on Mining Software Repositories (MSR ’19). 349–359. https://doi.org/10.1109/MSR.2019.00061

Digital Library

[15]

Danny Dig and Ralph Johnson. 2006. How Do APIs Evolve? A Story of Refactoring: Research Articles. Journal of software maintenance and evolution: Research and Practice, 18, 2 (2006), 3, 83–107. issn:1532-060X https://doi.org/10.1002/smr.328

[16]

Dino Distefano, Manuel Fähndrich, Francesco Logozzo, and Peter W. O’Hearn. 2019. Scaling Static Analyses at Facebook. Commun. ACM, 62, 8 (2019), jul, 62–70. issn:0001-0782 https://doi.org/10.1145/3338112

Digital Library

[17]

Khaled El Emam. 1999. Benchmarking Kappa: Interrater agreement in software process assessments. Empirical Software Engineering, 4 (1999), 113–133.

Digital Library

[18]

Darius Foo, Hendy Chua, Jason Yeo, Ming Yi Ang, and Asankhaya Sharma. 2018. Efficient Static Checking of Library Updates. In 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2018). ACM, 791–796. isbn:9781450355735 https://doi.org/10.1145/3236024.3275535

Digital Library

[19]

The Apache Software Foundation. 2023. Apache Maven Project. https://maven.apache.org/

[20]

James Gosling, Bill Joy, Guy Steele, Gilad Bracha, Alex Buckley, Daniel Smith, and Gavin Bierman. 2021. The Java language specification. Oracle America, Inc.

[21]

Nicolas Harrand, Amine Benelallam, César Soto-Valero, François Bettega, Olivier Barais, and Benoit Baudry. 2022. API beauty is in the eye of the clients: 2.2 million Maven dependencies reveal the spectrum of client–API usages. Journal of Systems and Software, 184 (2022), 111134. issn:0164-1212 https://doi.org/10.1016/j.jss.2021.111134

Digital Library

[22]

Hao He, Runzhi He, Haiqiao Gu, and Minghui Zhou. 2021. A Large-Scale Empirical Study on Java Library Migrations: Prevalence, Trends, and Rationales. ESEC/FSE ’21. ACM, 478–490. isbn:9781450385626 https://doi.org/10.1145/3468264.3468571

Digital Library

[23]

Dhanushka Jayasuriya, Valerio Terragni, Jens Dietrich, Samuel Ou, and Kelly Blincoe. 2023. Replication Package for Understanding Breaking Changes in the Wild. https://doi.org/10.5281/zenodo.7978507

Digital Library

[24]

Kamil Jezek, Jens Dietrich, and Premek Brada. 2015. How Java APIs Break - An Empirical Study. 65, C (2015), sep, 129–146. issn:0950-5849 https://doi.org/10.1016/j.infsof.2015.02.014

Digital Library

[25]

Riivo Kikas, Georgios Gousios, Marlon Dumas, and Dietmar Pfahl. 2017. Structure and Evolution of Package Dependency Networks. In 14th International Conference on Mining Software Repositories (MSR ’17). IEEE, 102–112. isbn:9781538615447 https://doi.org/10.1109/MSR.2017.55

Digital Library

[26]

Rediana Koçi, Xavier Franch, Petar Jovanovic, and Alberto Abelló. 2019. Classification of Changes in API Evolution. In 23rd International Enterprise Distributed Object Computing Conference (EDOC ’19). IEEE, 243–249. https://doi.org/10.1109/EDOC.2019.00037

[27]

Raula Gaikovina Kula, Daniel M. German, Ali Ouni, Takashi Ishio, and Katsuro Inoue. 2018. Do Developers Update Their Library Dependencies? Empirical Software Engineering, 23, 1 (2018), 2, 384–417. issn:1382-3256 https://doi.org/10.1007/s10664-017-9521-5

Digital Library

[28]

Anders Møller, Benjamin Barslev Nielsen, and Martin Toldam Torp. 2020. Detecting Locations in JavaScript Programs Affected by Breaking Library Changes. Proc. ACM Program. Lang., 4 (2020), 11, 1–25. https://doi.org/10.1145/3428255

Digital Library

[29]

Anders Møller and Martin Toldam Torp. 2019. Model-Based Testing of Breaking Changes in Node.Js Libraries. In 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019). ACM, 409–419. isbn:9781450355728 https://doi.org/10.1145/3338906.3338940

Digital Library

[30]

Suhaib Mujahid, Rabe Abdalkareem, Emad Shihab, and Shane McIntosh. 2020. Using Others’ Tests to Identify Breaking Updates. In 17th International Conference on Mining Software Repositories (MSR ’20). ACM, 466–476. isbn:9781450375177 https://doi.org/10.1145/3379597.3387476

Digital Library

[31]

Lina Ochoa, Thomas Degueule, and Jean-Rémy Falleri. 2022. BreakBot. In ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER ’22). ACM. https://doi.org/10.1145/3510455.3512783

Digital Library

[32]

Lina Ochoa, Thomas Degueule, Jean-Rémy Falleri, and Jurgen Vinju. 2022. Breaking Bad? Semantic Versioning and Impact of Breaking Changes in Maven Central: An External and Differentiated Replication Study. Empirical Softw. Engg., 27, 3 (2022), may, 42 pages. issn:1382-3256 https://doi.org/10.1007/s10664-021-10052-y

Digital Library

[33]

Fernando Rodriguez Olivera. 2022. MVN Repository: repository stats. https://mvnrepository.com/repos

[34]

Oracle. n.d. Java Virtual Machine Specification: Chapter 5. Loading, Linking, and Initializing. https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-5.html

[35]

Cliodhna O’Connor and Helene Joffe. 2020. Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines. International Journal of Qualitative Methods, 160906919899220. https://doi.org/10.1177/1609406919899220

[36]

Ivan Pashchenko, Henrik Plate, Serena Elisa Ponta, Antonino Sabetta, and Fabio Massacci. 2018. Vulnerable Open Source Dependencies: Counting Those That Matter. In ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM ’18). ACM. isbn:9781450358231 https://doi.org/10.1145/3239235.3268920

Digital Library

[37]

Tom Preston-Werner. n.d. Semantic Versioning 2.0.0. https://semver.org/

[38]

Steven Raemaekers, Arie van Deursen, and Joost Visser. 2014. Semantic Versioning versus Breaking Changes: A Study of the Maven Repository. In 14th International Working Conference on Source Code Analysis and Manipulation (SCAM ’14). IEEE, 215–224. https://doi.org/10.1109/SCAM.2014.30

Digital Library

[39]

S. Raemaekers, A. van Deursen, and J. Visser. 2017. Semantic versioning and impact of breaking changes in the Maven repository. Journal of Systems and Software, 129 (2017), 140–158. issn:0164-1212 https://doi.org/10.1016/j.jss.2016.04.008

Digital Library

[40]

Caitlin Sadowski, Edward Aftandilian, Alex Eagle, Liam Miller-Cushon, and Ciera Jaspan. 2018. Lessons from Building Static Analysis Tools at Google. Commun. ACM, 61, 4 (2018), mar, 58–66. issn:0001-0782 https://doi.org/10.1145/3188720

Digital Library

[41]

Pasquale Salza, Fabio Palomba, Dario Di Nucci, Cosmo D’Uva, Andrea De Lucia, and Filomena Ferrucci. 2018. Do Developers Update Third-Party Libraries in Mobile Apps? ICPC ’18. Association for Computing Machinery, New York, NY, USA. 255–265. isbn:9781450357142 https://doi.org/10.1145/3196321.3196341

Digital Library

[42]

Danilo Silva and Marco Tulio Valente. 2017. RefDiff: Detecting Refactorings in Version Histories. In IEEE/ACM 14th International Conference on Mining Software Repositories (MSR ’17). 269–279. https://doi.org/10.1109/MSR.2017.14

Digital Library

[43]

Inc Tidelift. 2022. Libraries.io - The Open Source Discovery Service. https://libraries.io/data

[44]

Ying Wang, Bihuan Chen, Kaifeng Huang, Bowen Shi, Congying Xu, Xin Peng, Yijian Wu, and Yang Liu. 2020. An Empirical Study of Usages, Updates and Risks of Third-Party Libraries in Java Projects. In International Conference on Software Maintenance and Evolution (ICSME ’20). IEEE, 35–45. https://doi.org/10.1109/ICSME46990.2020.00014

[45]

Thomas H. Wonnacott and Ronald J. Wonnacott. 1991. Introductory Statistics.

[46]

Laerte Xavier, Aline Brito, Andre Hora, and Marco Tulio Valente. 2017. Historical and impact analysis of API breaking changes: A large-scale study. In 24th International Conference on Software Analysis, Evolution and Reengineering (SANER ’17). IEEE, 138–147. https://doi.org/10.1109/SANER.2017.7884616

[47]

Zach. 2021. What is Inter-rater Reliability. https://www.statology.org/inter-rater-reliability/

[48]

Lyuye Zhang, Chengwei Liu, Zhengzi Xu, Sen Chen, Lingling Fan, Bihuan Chen, and Yang Liu. 2022. Has My Release Disobeyed Semantic Versioning? Static Detection Based on Semantic Differencing. In EEE/ACM International Conference on Automated Software Engineering (ASE ’22). ACM. isbn:9781450394758 https://doi.org/10.1145/3551349.3556956

Digital Library

Cited By

Zhou ZYang YWu SHuang YChen BPeng XFilkov VRay BZhou M(2024)Magneto: A Step-Wise Approach to Exploit Vulnerabilities in Dependent Libraries via LLM-Empowered Directed FuzzingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695531(1633-1644)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695531
Jayasuriya DTerragni VDietrich JBlincoe K(2024)Understanding the Impact of APIs Behavioral Breaking Changes on Client ApplicationsProceedings of the ACM on Software Engineering10.1145/36437821:FSE(1238-1261)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643782
Reyes FGamage YSkoglund GBaudry BMonperrus M(2024)BUMP: A Benchmark of Reproducible Breaking Dependency Updates2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00024(159-170)Online publication date: 12-Mar-2024
https://doi.org/10.1109/SANER60148.2024.00024

Index Terms

Understanding Breaking Changes in the Wild
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques
      1. Reusability
    2. Software post-development issues
      1. Maintaining software
      2. Software evolution
  2. Software notations and tools
    1. Software libraries and repositories

Recommendations

Understanding the Impact of APIs Behavioral Breaking Changes on Client Applications

Libraries play a significant role in software development as they provide reusable functionality, which helps expedite the development process. As libraries evolve, they release new versions with optimisations like new functionality, bug fixes, and ...
I Depended on You and You Broke Me: An Empirical Study of Manifesting Breaking Changes in Client Packages
Complex software systems have a network of dependencies. Developers often configure package managers (e.g., npm) to automatically update dependencies with each publication of new releases containing bug fixes and new features. When a dependency release ...
Towards Automated Updates of Software Dependencies
SPLASH Companion 2022: Companion Proceedings of the 2022 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity

Modern software applications use existing software components to fast forward the development process, resulting in software dependencies. These components will evolve for various reasons and could also be updated to address critical issues existing in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2023

1554 pages

ISBN:9798400702211

DOI:10.1145/3597926

General Chair:
René Just
University of Washington, USA
,
Program Chair:
Gordon Fraser
University of Passau, Germany

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution 4.0 International License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Conference

ISSTA '23

Sponsor:

SIGSOFT

ISSTA '23: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

July 17 - 21, 2023

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
503
Total Downloads

Downloads (Last 12 months)376
Downloads (Last 6 weeks)59

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhou ZYang YWu SHuang YChen BPeng XFilkov VRay BZhou M(2024)Magneto: A Step-Wise Approach to Exploit Vulnerabilities in Dependent Libraries via LLM-Empowered Directed FuzzingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695531(1633-1644)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695531
Jayasuriya DTerragni VDietrich JBlincoe K(2024)Understanding the Impact of APIs Behavioral Breaking Changes on Client ApplicationsProceedings of the ACM on Software Engineering10.1145/36437821:FSE(1238-1261)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643782
Reyes FGamage YSkoglund GBaudry BMonperrus M(2024)BUMP: A Benchmark of Reproducible Breaking Dependency Updates2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00024(159-170)Online publication date: 12-Mar-2024
https://doi.org/10.1109/SANER60148.2024.00024

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents