article

On the reproducibility of empirical software engineering studies based on data retrieved from development repositories

Authors:

Jesús M. González-Barahona,

Gregorio RoblesAuthors Info & Claims

Empirical Software Engineering, Volume 17, Issue 1-2

Pages 75 - 89

https://doi.org/10.1007/s10664-011-9181-9

Published: 01 February 2012 Publication History

Abstract

Among empirical software engineering studies, those based on data retrieved from development repositories (such as those of source code management, issue tracking or communication systems) are specially suitable for reproduction. However their reproducibility status can vary a lot, from easy to almost impossible to reproduce. This paper explores which elements can be considered to characterize the reproducibility of a study in this area, and how they can be analyzed to better understand the type of reproduction studies they enable or obstruct. One of the main results of this exploration is the need of a systematic approach to asses the reproducibility of a study, due to the complexity of the processes usually involved, and the many details to be taken into account. To address this need, a methodology for assessing the reproducibility of studies is also presented and discussed, as a tool to help to raise awareness about research reproducibility in this field. The application of the methodology in practice has shown how, even for papers aimed to be reproducible, a systematic analysis raises important aspects that render reproduction difficult or impossible. We also show how, by identifying elements and attributes related to reproducibility, it can be better understood which kind of reproduction can be done for a specific study, given the description of datasets, methodologies and parameters it uses.

References

[1]

Barr ET, Bird C, Hyatt E, Menzies T, Robles G (2010) On the shoulders of giants. In: FoSER, pp 23-28.

[2]

Basili VR, Shull F, Lanubile F (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25(4):456-473.

Digital Library

[3]

Boetticher G, Menzies T, Ostrand T (2007) PROMISE repository of empirical software engineering data. Department of Computer Science, West Virginia University. http://promisedata.org/

[4]

de Leeuw J (2001) Reproducible research. The bottom line. Technical report, UC Los Angeles: Department of Statistics, UCLA. http://escholarship.org/uc/item/9050x4r4

[5]

Donoho DL, Maleki A, Rahman IU, Shahram M, Stodden V (2009) Reproducible research in computational harmonic analysis. Comput Sci Eng 11:8-18.

Digital Library

[6]

Fayyad U, Piatetsky-Shapiro G, Smyth P (1996) Knowledge discovery and data mining: towards a unifying framework. In: Proceedings of the 2nd international conference on knowledge discovery and data mining, KDD-96, Portland (Oregon, USA). AAAI Press, Menlo Park, pp 82-88.

[7]

Fomel S, Claerbout JF (2009) Guest editors' introduction: reproducible research. Comput Sci Eng 11:5-7.

Digital Library

[8]

Gentleman R, Lang DT (2007) Statistical analyses and reproducible research. J Comput Graph Stat 16(1):1-23.

[9]

Germán DM (2004) Mining CVS repositories, the softChange experience. In: Proceedings of the international workshop on mining software repositories, Edinburghh, UK.

[10]

Gomez OS, Juristo N, Vegas S (2010) Replication, reproduction and re-analysis: three ways for verifying experimental findings. In: Proceedings of the 1st international workshop on replication in empirical software engineering research (RESER 2010), Cape Town, South Africa.

[11]

González-Barahona JM, Robles G, Michlmayr M, Amor JJ, Germán DM (2009) Macro-level software evolution: a case study of a large software compilation. Empir Software Eng 14(3):262-285.

Digital Library

[12]

Hayes JH, Dekhtyar A, Sundaram S (2005) Text mining for software engineering: how analyst feedback impacts final results. In: Proceedings of the second international workshop on mining software repositories, St. Louis, USA.

[13]

Herraiz I, Izquierdo-Cortazar D, Rivas-Hernández F (2009) FLOSSMetrics: Free/Libre/Open Source Software Metrics. In: CSMR, pp 281-284.

Digital Library

[14]

Hothorn T, Leisch F (2011) Case studies in reproducibility. Brief Bioinform.

[15]

Howison J, Conklin M, Crowston K (2006) FLOSSmole: a collaborative repository for FLOSS research data and analyses. IJITWE 1(3):17-26.

[16]

Knutson CD, Krein JL, Prechelt L, Juristo N (2010) Report from the 1st international workshop on replication in empirical software engineering research (RESER 2010). SIGSOFT Softw Eng Notes 35:42-44.

Digital Library

[17]

Koenker R, Zeileis A (2009) On reproducible econometric research. J Appl Econ 24(5):833-847.

[18]

Miller J (2005) Replicating software engineering experiments: a poisoned chalice or the holy grail. Inf Softw Technol 47:233-244.

Digital Library

[19]

Panjer LD (2007) Predicting eclipse bug lifetimes. In: Proceedings of the fourth international workshop on mining software repositories, MSR '07, p 29.

[20]

Robles G (2010) Replicating MSR: A study of the potential replicability of papers published in the mining software repositories proceedings. In: 2010 7th IEEE working conference on mining software repositories (MSR), pp 171-180.

[21]

Robles G, Germán DM (2010) Beyond replication: an example of the potential benefits of replicability in the mining of software repositories community. In: Proceedings of the 1st international workshop on replication in empirical software engineering sesearch (RESER 2010).

[22]

Robles G, González-Barahona JM, Merelo-Guervós JJ (2006) Beyond source code: the importance of other artifacts in software development (a case study). J Syst Softw 79(9):1233-1248.

Digital Library

[23]

Shull F, Mendonça MG, Basili VR, Carver J, Maldonado JC, Fabbri SCPF, Travassos GH, de Oliveira MCF (2004) Knowledge-sharing issues in experimental software engineering. Empir Software Eng 9(1-2):111-137.

[24]

Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in empirical software engineering. Empir Software Eng 13(2):211-218.

Digital Library

[25]

Vandewalle P, Barrenexea G, Jovanovic I, Ridolfi A, Vetterli M (2007) Experiences with reproducible research in various facets of signal processing research. In: Proceedings of the international conference on acoustics, speech and signal processing. ICASSP 2007, vol 4, pp IV-1253-IV-1256.

[26]

Vegas S, Juristo N, Moreno A, Solari M, Letelier P (2006) Analysis of the influence of communication between researchers on experiment replication. In: ISESE '06: Proceedings of the 2006 ACM/IEEE international symposium on empirical software engineering, pp 28-37.

Cited By

Hassan ZTreude CNorrish MWilliams GPotanin Ad'Amorim M(2024)Reproducibility Debt: Challenges and Future PathwaysCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663778(462-466)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663778
Abualhaija SAydemir FDalpiaz FDell'Anna DFerrari AFranch XFucci D(2024)Replication in Requirements Engineering: The NLP for RE CaseACM Transactions on Software Engineering and Methodology10.1145/365866933:6(1-33)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3658669
Runeson PSoderberg EHost M(2024)A Conceptual Framework and Recommendations for Open Data and Artifacts in Empirical Software EngineeringProceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering10.1145/3643664.3648206(68-75)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1145/3643664.3648206
Show More Cited By

Recommendations

Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories
Abstract Context:
In 2012, our paper “On the reproducibility of empirical software engineering studies based on data retrieved from development repositories” was published. It proposed a method for assessing the reproducibility of studies based on mining ...
Towards Evidence-Based Software Quality Practices for Reproducibility: Preliminary Results and Research Directions
ACM REP '23: Proceedings of the 2023 ACM Conference on Reproducibility and Replicability

In the computational science and engineering (CSE) community, there is a prevailing belief that adopting better software development practices and investing in software quality will directly lead to more robust, reproducible software. There is, however, ...
How to Measure the Reproducibility of System-oriented IR Experiments
SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

Replicability and reproducibility of experimental results are primary concerns in all the areas of science and IR is not an exception. Besides the problem of moving the field towards more reproducible experimental practices and protocols, we also face a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Empirical Software Engineering

Empirical Software Engineering Volume 17, Issue 1-2

February 2012

127 pages

ISSN:1382-3256

Issue’s Table of Contents

Copyright © Copyright © 2012 Springer Science+Business Media, LLC.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 February 2012

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

42
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hassan ZTreude CNorrish MWilliams GPotanin Ad'Amorim M(2024)Reproducibility Debt: Challenges and Future PathwaysCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663778(462-466)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663778
Abualhaija SAydemir FDalpiaz FDell'Anna DFerrari AFranch XFucci D(2024)Replication in Requirements Engineering: The NLP for RE CaseACM Transactions on Software Engineering and Methodology10.1145/365866933:6(1-33)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3658669
Runeson PSoderberg EHost M(2024)A Conceptual Framework and Recommendations for Open Data and Artifacts in Empirical Software EngineeringProceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering10.1145/3643664.3648206(68-75)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1145/3643664.3648206
Gonzalez-Barahona JRobles G(2024)Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositoriesInformation and Software Technology10.1016/j.infsof.2023.107318164:COnline publication date: 10-Jan-2024
https://dl.acm.org/doi/10.1016/j.infsof.2023.107318
Kalu KSchorlemmer TChen SRobinson KKocinare EDavis JChandra SBlincoe KTonella P(2023)Reflecting on the Use of the Policy-Process-Product Theory in Empirical Software EngineeringProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613075(2112-2116)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613075
Raff EFarris A(2023)A Siren Song of Open Source Reproducibility, Examples from Machine LearningProceedings of the 2023 ACM Conference on Reproducibility and Replicability10.1145/3589806.3600042(115-120)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3589806.3600042
Nong YSharma RHamou-Lhadj ALuo XCai H(2023)Open Science in Software Engineering: A Study on Deep Learning-Based Vulnerability DetectionIEEE Transactions on Software Engineering10.1109/TSE.2022.320714949:4(1983-2005)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TSE.2022.3207149
Wang SHuang LGao AGe JZhang TFeng HSatyarth ILi MZhang HNg V(2023)Machine/Deep Learning for Software Engineering: A Systematic Literature ReviewIEEE Transactions on Software Engineering10.1109/TSE.2022.317334649:3(1188-1231)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1109/TSE.2022.3173346
Giray GBennin KKöksal ÖBabur ÖTekinerdogan B(2023)On the use of deep learning in software defect predictionJournal of Systems and Software10.1016/j.jss.2022.111537195:COnline publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.jss.2022.111537
Rani PBlasi AStulova NPanichella SGorla ANierstrasz O(2023)A decade of code comment quality assessmentJournal of Systems and Software10.1016/j.jss.2022.111515195:COnline publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.jss.2022.111515
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents