article

Review participation in modern code review

Authors:

Patanamon Thongtanunam,

Shane Mcintosh,

Ahmed E. Hassan,

Hajimu IidaAuthors Info & Claims

Empirical Software Engineering, Volume 22, Issue 2

Pages 768 - 817

https://doi.org/10.1007/s10664-016-9452-6

Published: 01 April 2017 Publication History

Abstract

Software code review is a well-established software quality practice. Recently, Modern Code Review (MCR) has been widely adopted in both open source and proprietary projects. Our prior work shows that review participation plays an important role in MCR practices, since the amount of review participation shares a relationship with software quality. However, little is known about which factors influence review participation in the MCR process. Hence, in this study, we set out to investigate the characteristics of patches that: (1) do not attract reviewers, (2) are not discussed, and (3) receive slow initial feedback. Through a case study of 196,712 reviews spread across the Android, Qt, and OpenStack open source projects, we find that the amount of review participation in the past is a significant indicator of patches that will suffer from poor review participation. Moreover, we find that the description length of a patch shares a relationship with the likelihood of receiving poor reviewer participation or discussion, while the purpose of introducing new features can increase the likelihood of receiving slow initial feedback. Our findings suggest that the patches with these characteristics should be given more attention in order to increase review participation, which will likely lead to a more responsive review process.

References

[1]

Abelein U, Paech B (2013) Understanding the influence of user participation and involvement on system success a systematic mapping study. Empir Softw Eng (EMSE) 20(1):28-31.

Digital Library

[2]

Bacchelli A, Bird C (2013) Expectations, Outcomes, and Challenges of Modern Code Review. In: Proceedings of the 35th International Conference on Software Engineering (ICSE), pp 712-721.

[3]

Bacchelli A, Lanza M, Robbes R (2010) Linking E-Mails and Source Code Artifacts. In: Proceedings of the 32nd International Conference on Software Engineering (ICSE), pp 375-384.

[4]

Balachandran V (2013) Reducing Human Effort and Improving Quality in Peer Code Reviews using Automatic Static Analysis and Reviewer Recommendation. In: Proceedings of the 35th International Conference on Software Engineering (ICSE), pp 931-940.

[5]

Bavota G, Russo B (2015) Four Eyes Are Better Than Two: On the Impact of Code Reviews on Software Quality. In: Proceedings of the 31st International Conference on Software Maintenance and Evolution (ICSME), pp 81-90.

[6]

Baysal O, Kononenko O, Holmes R, Godfrey MW (2012) The Secret Life of Patches: A Firefox Case Study. In: Proceedings of the 19th Working Conference on Reverse Engineering (WCRE), pp 447-455.

[7]

Baysal O, Kononenko O, Holmes R, Godfrey MW (2015) Investigating technical and non-technical factors influencing modern code review. Empirical Software Engineering (EMSE) 1-28.

[8]

Beller M, Bacchelli A, Zaidman A, Juergens E (2014) Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix? In: Proceedings of the 11th Working Conference on Mining Software Repositorie (MSR), pp 202-211.

[9]

Bettenburg N, Hassan AE, Adams B, German DM (2013) Management of community contributions - A case study on the Android and Linux software ecosystems. Empirical Software Engineering (EMSE) 1-38.

[10]

Bird C, Gourley A, Devanbu P, Gertz M, Swaminathan A (2006) Mining Email Social Networks. In: Proceedings of the 3rd International Workshop on Mining Software Repositories (MSR), pp 137-143.

[11]

Bird C, Gourley A, Devanbu P (2007) Detecting Patch Submission and Acceptance in OSS Projects. In: Proceedings of the 4th International Workshop on Mining Software Repositories (MSR), pp 26-29.

[12]

Bird C, Nagappan N, Murphy B, Gall H, Devanbu P (2011) Don't Touch My Code! Examining the Effects of Ownership on Software Quality. In: Proceedings of the 8th joint meeting of the European Software Engineering Conference and the Internaltional Symposium on the Foundations of Software Engineering (ESEC/FSE), pp 4-14.

[13]

Bosu A, Carver JC (2013) Impact of Peer Code Review on Peer Impression Formation: A Survey. In: Proceedings of the 7th International Symposium on Empirical Software Engineering and Measurement (ESEM), pp 133-142.

[14]

Cramer H (1999) Mathematical methods of statistics. University Press, Princeton, p 9.

[15]

Efron B (1986) How biased is the apparent error rate of a prediction Rule? J Am Stat Assoc 81(394):461-470.

[16]

Fagan ME (1999) Design and code inspections to reduce errors in program development. IBM Syst J 38(2-3):258-287.

Digital Library

[17]

Fowler M, Foemmel M (2006) Continuous integration. http://wwwthoughtworkscom/ContinuousIntegrationpdf.

[18]

Gousios G, Pinzger M, van Deursen A (2014) An Exploratory Study of the Pull-based Software Development Model. In: Proceedings of the 36th International Conference on Software Engineering (ICSE), pp 345-355.

[19]

Guzzi A, Bacchelli A, Lanza M, Pinzger M, Van Deursen A (2013) Communication in Open Source Software Development Mailing Lists. In: Proceedings of the 10th International Working Conference on Mining Software Repositories (MSR), pp 277-286.

[20]

Hamasaki K, Kula RG, Yoshida N, Erika CCA, Fujiwara K, Iida H (2013) Who does what during a Code Review? An extraction of an OSS Peer Review Repository. In: Proceedings of the 10th International Working Conference on Mining Software Repositories (MSR), pp 49-52.

[21]

Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiological Society of North America 143(1):29-36.

[22]

Harrell Jr FE (2002) Regression Modeling Strategies: With Application to Linear Models, Logistic Regression, and Survival Analysis, 1st. Springer, Berlin.

[23]

Harrell Jr FE (2015) rms: Regression Modeling Strategies. http://biostat.mc.vanderbilt.edu/rms.

[24]

Hassan AE (2008) Automated Classification of Change Messages in Open Source Projects. In: Proceedings of the 23rd Symposium on Applied Computing (SAC), pp 837-841.

[25]

Hassan AE (2009) Predicting Faults Using the Complexity of Code Changes. In: Proceedings of the 31st International Conference on Software Engineering (ICSE), pp 78-88.

[26]

Hinkle DE, Wiersma W, Jurs SG (1998) Applied statistics for the behavioral sciences, 4th edn. Houghton Mifflin, Boston.

[27]

Jiang Y, Adams B, German DM (2013) Will My Patch Make It? And How Fast? Case Study on the Linux Kernel. In: Proceeding of the 10th International Working Conference on Mining Software Repositories (MSR), pp 101-110.

[28]

Kamei Y, Shihab E, Adams B, Hassan AE, Mockus A, Sinha A, Ubayashi N (2013) A large-scale empirical study of just-in-time quality assurance. Trans Softw Eng (TSE) 39(6):757-773.

Digital Library

[29]

Kim S, Whitehead EJ Jr, Zhang Y (2008) Classifying software changes: Clean or buggy? Trans Softw Eng (TSE) 34(2):181-196.

Digital Library

[30]

Kononenko O, Baysal O, Guerrouj L, Cao Y, Godfrey MW (2015) Investigating Code Review Quality : Do People and Participation Matter ? In: Proceedings of the 31st International Conference on Software Maintenance and Evolution (ICSME), pp 111-120.

[31]

McIntosh S, Kamei Y, Adams B, Hassan AE (2014) The Impact of Code Review Coverage and Code Review Participation on Software Quality. In: Proceedings of the 11th International Working Conference on Mining Software Repositorie (MSR), pp 192-201.

[32]

McIntosh S, Kamei Y, Adams B, Hassan AE (2015) An Empirical Study of the Impact of Modern Code Review Practices on Software Quality. Empirical Software Engineering (EMSE).

[33]

Mishra R, Sureka A (2014) Mining Peer Code Review System for Computing Effort and Contribution Metrics for Patch Reviewers. In: Proceedings of the 4thWorkshop on Mining Unstructured Data (MUD), pp 11-15.

[34]

Mockus A, Votta LG (2000) Identifying Reasons for Software Changes using Historic Databases. In: Proceedings of the 16th International Conference on Software Maintainance (ICSM), pp 120-130.

[35]

Morales R, Mcintosh S, Khomh F (2015) Do Code Review Practices Impact Design Quality? A Case Study of the Qt, VTK, and ITK Projects. In: Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[36]

Mukadam M, Bird C, Rigby PC (2013) Gerrit software code review data from android. In: Proceedings of the 10th International Working Conference on Mining Software Repositories (MSR), pp 45-48.

[37]

Nagappan M, Zimmermann T, Bird C (2013) Diversity in Software Engineering Research. In: Proceedings of the 9th joint meeting of the European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE), pp 466-476.

[38]

Nagappan N, Zeller A, Zimmermann T, Herzig K, Murphy B (2010) Change bursts as defect predictors. In: Proceedings of the 21st International Symposium on Software Reliability Engineering (ISSRE), pp 309-318.

[39]

Nurolahzade M, Nasehi SM, Khandkar SH, Rawal S (2009) The Role of Patch Review in Software Evolution: An Analysis of the Mozilla Firefox. In: Proceedings of the join International and Anual ERCIM Workshops on Principles of Software Evolution and Software Evolution Workshop (IWPSE-Evol), pp 9-17.

[40]

Porter A, Siy H, Mockus A, Votta L (1998) Understanding the sources of variation in software inspections. Trans Softw Eng Methodol (TOSEM) 7(1):41-79.

Digital Library

[41]

Ratcliffe JG (2009) Moving Software Quality Upstream: The Positive Impact of Lightweight Peer Code Review. In: Pacific NW Software Quality Conference, pp 1-10.

[42]

Raymond ES (1999) The cathedral and the bazaar. Knowl, Technol Policy 12(3):23-49.

[43]

Rea LM, Parker RA (2014) Designing and conducting survey research: A comprehensive guide. John Wiley & Sons.

[44]

Rigby PC, Bird C (2013) Convergent Contemporary Software Peer Review Practices. In: Proceedings of the 9th joint meeting of the European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE), pp 202-212.

[45]

Rigby PC, Storey MA (2011) Understanding Broadcast Based Peer Review on Open Source Software Projects. In: Proceeding of the 33rd International Conference on Software Engineering (ICSE), pp 541-550.

[46]

Rigby PC, German DM, Storey MA (2008) Open Source Software Peer Review Practices: A Case Study of the Apache Server. In: Proceedings of the 30th International Conference on Software Engineering (ICSE), pp 541-550.

[47]

Rigby PC, Cleary B, Painchaud F, Storey MA, German DM (2012) Contemporary peer review in action: Lessons from open source development. IEEE Softw 29(6):56-61.

Digital Library

[48]

Rigby PC, German DM, Cowen L, Storey MA (2014) Peer review on open-source software projects: Parameters, Statistical Models, and Theory. Transactions on Software Engineering and Methodology (TOSEM) 23(4):Article No. 35.

[49]

Sarle WS (1990) The VARCLUS procedure, 4th edn. SAS Institute, Inc.

[50]

Sauer C, Jeffery DR, Land L, Yetton P (2000) The effectiveness of software development technical reviews: a behaviorally motivated program of research. Trans Softw Eng (TSE) 26(1):1-14.

Digital Library

[51]

Shihab E, Jiang ZM, Hassan AE (2009) Studying the Use of Developer IRC Meetings in Open Source Projects. In: Prceedings of the 25th International Conference on Software Maintenance (ICSM), pp 147-156.

[52]

Shull F, Basili V, Boehm B, Brown AW, Costa P, Lindvall M, Port D, Rus I, Tesoriero R, Zelkowitz M (2002) What We Have Learned About Fighting Defects. In: Proceedings of the 8th International Software Metrics Symposium (METRICS), pp 249-258.

[53]

Tantithamthavorn C, McIntosh S, Hassan AE, Ihara A, Matsumoto K (2015) The Impact of Mislabelling on the Performance and Interpretation of Defect Prediction Models. In: Proceedings of the 37th International Conference on Software Engineering (ICSE), pp 812-823.

[54]

Tao Y, Dang Y, Xie T, Zhang D, Kim S (2012) How Do Software Engineers Understand Code Changes?: An Exploratory Study in Industry. In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (FSE), pp 51:1-51:11.

[55]

Thongtanunam P, Yang X, Yoshida N, Kula RG, Ana Erika CC, Fujiwara K, Iida H (2014) ReDA: A Web-based Visualization Tool for Analyzing Modern Code Review Dataset. In: The proceeding of the 30th International Conference on Software Maintenance and Evolution (ICSME), pp 606-609.

[56]

Thongtanunam P, McIntosh S, Hassan AE, Iida H (2015a) Investigating Code Review Practices in Defective Files: An Empirical Study of the Qt System. In: Proceedings of the 12th International Working Conference on Mining Software Repositories (MSR), pp 168-179.

[57]

Thongtanunam P, Tantithamthavorn C, Kula RG, Yoshida N, Iida H, Matsumoto K (2015b) Who Should Review My Code? A File Location-Based Code-Reviewer Recommendation Approach for Modern Code Review. In: Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp 141-150.

[58]

Thongtanunam P, McIntosh S, Hassan AE, Iida H (2016) Revisiting Code Ownership and its Relationship with Software Quality in the Scope of Modern Code Review. In: Proceedings of the 38th International Conference on Software Engineering (ICSE), pp 1039-1050.

[59]

Tsay J, Dabbish L, Herbsleb J (2014) Let's Talk About It: Evaluating Contributions through Discussion in GitHub. In: Proceedings of the 22nd International Symposium on the Foundations of Software Engineering (FSE), pp 144-154.

[60]

Ukkonen E (1985) Algorithms for approximate string matching. Inf Control 64(1-3):100-118.

Digital Library

[61]

Weißgerber P, Neu D, Diehl S (2008) Small Patches Get In ! In: Proceedings of the 2008 international working conference on Mining software repositories (MSR'08), pp 67-75.

[62]

Zanjani M, Kagdi H, Bird C (2015) Automatically recommending peer reviewers in modern code review. IEEE Trans Softw Eng:1-13.

Cited By

Yang LXu JZhang HWu FLyu JLi YBacchelli AFilkov VRay BZhou M(2024)GPP: A Graph-Powered Prioritizer for Code Review RequestsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3694990(104-116)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3694990
Hong YTantithamthavorn CPasuksmit JThongtanunam PFriedman AZhao XKrasikov Ad'Amorim M(2024)Practitioners’ Challenges and Perceptions of CI Build Failure Predictions at AtlassianCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663856(370-381)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663856
Sarma AChen N(2024)Effective Teaching through Code Reviews: Patterns and Anti-patternsProceedings of the ACM on Software Engineering10.1145/36607641:FSE(1262-1283)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660764
Show More Cited By

Recommendations

Code review quality: how developers see it
ICSE '16: Proceedings of the 38th International Conference on Software Engineering

In a large, long-lived project, an effective code review process is key to ensuring the long-term quality of the code base. In this work, we study code review practices of a large, open source project, and we investigate how the developers themselves ...
A study of the quality-impacting practices of modern code review at Sony mobile
ICSE '16: Proceedings of the 38th International Conference on Software Engineering Companion

Nowadays, a flexible, lightweight variant of the code review process (i.e., the practice of having other team members critique software changes) is adopted by open source and proprietary software projects. While this flexibility is a blessing (e.g., ...
Improving Code Review with GitHub Issue Tracking
ASONAM '22: Proceedings of the 2022 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining

Software quality is an important problem for technology companies, since it substantially impacts the efficiency, usefulness, and maintainability of the final product; hence, code review is a must-do activity for software developers. During the code ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Empirical Software Engineering

Empirical Software Engineering Volume 22, Issue 2

April 2017

383 pages

ISSN:1382-3256

Issue’s Table of Contents

Copyright © Copyright © 2017 Springer Science+Business Media New York.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 April 2017

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

47
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang LXu JZhang HWu FLyu JLi YBacchelli AFilkov VRay BZhou M(2024)GPP: A Graph-Powered Prioritizer for Code Review RequestsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3694990(104-116)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3694990
Hong YTantithamthavorn CPasuksmit JThongtanunam PFriedman AZhao XKrasikov Ad'Amorim M(2024)Practitioners’ Challenges and Perceptions of CI Build Failure Predictions at AtlassianCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663856(370-381)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663856
Sarma AChen N(2024)Effective Teaching through Code Reviews: Patterns and Anti-patternsProceedings of the ACM on Software Engineering10.1145/36607641:FSE(1262-1283)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660764
Liang MCharoenwet WThongtanunam PSpinellis DConstantinou EBacchelli A(2024)Curated Email-Based Code Reviews DatasetsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644872(294-298)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644872
Xiao THata HTreude CMatsumoto K(2024)Generative AI for Pull Request Descriptions: Adoption, Impact, and Developer InterventionsProceedings of the ACM on Software Engineering10.1145/36437731:FSE(1043-1065)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643773
Khatoonabadi SAbdellatif ACosta DShihab E(2024)Predicting the First Response Latency of Maintainers and Contributors in Pull RequestsIEEE Transactions on Software Engineering10.1109/TSE.2024.344374150:10(2529-2543)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3443741
Hajari FMalmir SMirsaeedi ERigby P(2024)Factoring Expertise, Workload, and Turnover Into Code Review RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2024.336675350:4(884-899)Online publication date: 23-Feb-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3366753
Zhang BFu LLiang PYu JWang C(2024)Demystifying code snippets in code reviews: a study of the OpenStack and Qt communities and a practitioner surveyEmpirical Software Engineering10.1007/s10664-024-10484-229:4Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1007/s10664-024-10484-2
Moreno-Lumbreras DRobles GIzquierdo-Cortázar DGonzalez-Barahona J(2024)Software development metrics: to VR or not to VREmpirical Software Engineering10.1007/s10664-023-10435-329:2Online publication date: 3-Feb-2024
https://dl.acm.org/doi/10.1007/s10664-023-10435-3
Nejati MAlfadel MMcIntosh SGrundy JPollock LPenta M(2023)Code Review of Build System Specifications: Prevalence, Purposes, Patterns, and PerceptionsProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00108(1213-1224)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00108
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents