Exploring the Effects of Urgency and Reputation in Code Review: An Eye-Tracking Study
Pages 202 - 213
Abstract
The Pull-Based development model, a fundamental mechanism of collaboration in modern software engineering (SE), initiates the code review process when a contributor submits pull requests (PRs) for evaluation. Although the decision to approve or decline PRs is often perceived as grounded in their technical quality, prior research presents a more intricate narrative where both non technical factors-beyond the code itself and technical factors influence the acceptance. This study, uniquely integrating cues of urgency (represented by code priority level) and reputation (represented by the experience level of the code's author), delves into these biases, leveraging eye-tracking technology to illuminate the cognitive processes underpinning the evaluation of PRs.
In an experimentally-controlled study involving 37 participants reviewing Java code PRs, we found that perceived priority level of a PR impacted both the time spent on tasks and the associated cognitive load. Moreover, while participants' behaviors reflected the influence of the priority level and the author's experience, they remained largely unaware of these effects on their decision-making, highlighting the critical importance of understanding these implicit biases in code reviews. Interestingly, despite variations in attention when reviewing contributions from novice versus senior authors, there was no discernible difference in acceptance outcomes based on the author's experience. This study takes the next step toward a better understanding of urgency and reputation in SE and may inform future research about code review platforms and guidelines, code reuse, and automated code generation.
References
[1]
Gene M Alarcon, Rose Gamble, Sarah A Jessup, Charles Walter, Tyler J Ryan, David W Wood, and Chris S Calhoun. 2017. Application of the heuristic-systematic model to computer code trustworthiness: The influence of reputation and transparency. Cogent Psychology 4, 1 (2017), 1389640.
[2]
Gene M. Alarcon, Anthony M. Gibson, Charles Walter, Rose F. Gamble, Tyler J. Ryan, Sarah A. Jessup, Brian E. Boyd, and August Capiola. 2020. Trust Perceptions of Metadata in Open-Source Software: The Role of Performance and Reputation. Systems 8, 3 (2020).
[3]
Gene M. Alarcon and Tyler J. Ryan. 2018. Trustworthiness Perceptions of Computer Code: A Heuristic-Systematic Processing Model. In Proceedings of the 51st Hawaii International Conference on System Sciences.
[4]
Gene M. Alarcon, Charles Walter, Anthony M. Gibson, Rose F. Gamble, August Capiola, Sarah A. Jessup, and Tyler J. Ryan. 2020. Would You Fix This Code for Me? Effects of Repair Source and Commenting on Trust in Code Repair. Systems 8, 1 (2020).
[5]
Olga Baysal, Oleksii Kononenko, Reid Holmes, and Michael W. Godfrey. 2013. The influence of non-technical factors on code review. In 2013 20th Working Conference on Reverse Engineering (WCRE). 122--131.
[6]
Andrew Begel and Hana Vrzakova. 2018. Eye Movements in Code Review. In Proceedings of the Workshop on Eye Movements in Programming. Article 5, 5 pages.
[7]
Ian Bertram, Jack Hong, Yu Huang, Westley Weimer, and Zohreh Sharafi. 2020. Trustworthiness Perceptions in Code Review: An Eye-Tracking Study. In Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) (Bari, Italy) (ESEM '20). Association for Computing Machinery, New York, NY, USA, Article 31, 6 pages.
[8]
Martha E. Crosby and Jan Stelovsky. 1990. How Do We Read Algorithms? A Case Study. Computer 23, 1 (Jan. 1990), 24--35.
[9]
Denae Ford, Mahnaz Behroozi, Alexander Serebrenik, and Chris Parnin. 2019. Beyond the code itself: how programmers really look at pull requests. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS). IEEE, 51--60.
[10]
Zachary P. Fry, Bryan Landau, and Westley Weimer. 2012. A human study of patch maintainability. In International Symposium on Software Testing and Analysis, ISSTA 2012, Minneapolis, MN, USA, July 15--20, 2012, Mats Per Erik Heimdahl and Zhendong Su (Eds.). ACM, 177--187.
[11]
Claudia Geitner, Ben D Sawyer, S Birrell, P Jennings, L Skyrypchuk, Bruce Mehler, and Bryan Reimer. 2017. A link between trust in technology and glance allocation in on-road driving. (2017).
[12]
Christian Gold, Moritz Körber, Christoph Hohenberger, David Lechner, and Klaus Bengler. 2015. Trust in automation-Before and after the experience of take-over scenarios in a highly automated vehicle. Procedia Manufacturing 3 (2015), 3025--3032.
[13]
Joseph H. Goldberg and Jonathan I. Helfman. 2010. Comparing Information Graphics: A Critical Look at Eye Tracking. In Proceedings of the 3rd BEyond Time and Errors: Novel evaLuation Methods for Information Visualization Workshop (Atlanta, Georgia) (BELIV '10). ACM, New York, NY, USA, 71--78.
[14]
Georgios Gousios, Martin Pinzger, and Arie van Deursen. 2014. An Exploratory Study of the Pull-Based Software Development Model. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 345--355.
[15]
Israel Herraiz, Daniel M. German, Jesus M. Gonzalez-Barahona, and Gregorio Robles. 2008. Towards a Simplification of the Bug Report Form in Eclipse. In Proceedings of the 2008 International Working Conference on Mining Software Repositories (Leipzig, Germany) (MSR '08). Association for Computing Machinery, New York, NY, USA, 145--148.
[16]
https://www.tobiipro.com/. 2001. Online; Accessed 17-07-2020.
[17]
Yu Huang, Kevin Leach, Zohreh Sharafi, Nicholas McKay, Tyler Santander, and Westley Weimer. 2020. Biases and Differences in Code Review Using Medical Imaging and Eye-Tracking: Genders, Humans, and Machines. In International Symposium on the Foundations of Software Engineering (Virtual Event, USA) (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 456--468.
[18]
Rahul N. Iyer, S. Alex Yun, Meiyappan Nagappan, and Jesse Hoey. 2021. Effects of Personality Traits on Pull Request Acceptance. IEEE Transactions on Software Engineering 47, 11 (2021), 2632--2643.
[19]
Marcel A Just and Patricia A Carpenter. 1980. A theory of reading: from eye fixations to comprehension. Psychological review 87, 4 (1980), 329.
[20]
Dongsun Kim, Jaechang Nam, Jaewoo Song, and Sunghun Kim. 2013. Automatic patch generation learned from human-written patches. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, 802--811.
[21]
Barbara A. Kitchenham, Shari Lawrence Pfleeger, Lesley M. Pickard, Peter W. Jones, David C. Hoaglin, Khaled El Emam, and Jarrett Rosenberg. 2002. Preliminary Guidelines for Empirical Research in Software Engineering. IEEE Transactions on Software Engineering 28, 8 (Aug. 2002), 721--734.
[22]
Oleksii Kononenko, Olga Baysal, Latifa Guerrouj, Yaxin Cao, and Michael W. Godfrey. 2015. Investigating code review quality: Do people and participation matter?. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). 111--120.
[23]
John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50--80.
[24]
Y. Lu and N. Sarter. 2019. Eye Tracking: A Process-Oriented Method for Inferring Trust in Automation as a Function of Priming and System Reliability. IEEE Transactions on Human-Machine Systems 49, 6 (Dec 2019), 560--568.
[25]
A. Marginean, J. Bader, S. Chandra, M. Harman, Y. Jia, K. Mao, A. Mols, and A. Scott. 2019. SapFix: Automated End-to-End Repair at Scale. In International Conference on Software Engineering: Software Engineering in Practice. 269--278.
[26]
Jennifer Marlow, Laura Dabbish, and Jim Herbsleb. 2013. Impression Formation in Online Peer Production: Activity Traces and Personal Profiles in Github. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (San Antonio, Texas, USA) (CSCW '13). Association for Computing Machinery, New York, NY, USA, 117--128.
[27]
Matias Martinez, Thomas Durieux, Romain Sommerard, Jifeng Xuan, and Martin Monperrus. 2016. Automatic Repair of Real Bugs in Java: A Large-Scale Experiment on the Defects4J Dataset. Springer Empirical Software Engineering (2016).
[28]
Martin Monperrus, Simon Urli, Thomas Durieux, Matias Martinez, Benoit Baudry, and Lionel Seinturier. 2019. Repairnator Patches Programs Automatically. Ubiquity 2019, July, Article 2 (July 2019), 12 pages.
[29]
Rennie Naidoo. 2015. Analysing urgency and trust cues exploited in phishing scam designs. In 10th International Conference on Cyber Warfare and Security. 216.
[30]
Yannic Noller, Ridwan Shariffdeen, Xiang Gao, and Abhik Roychoudhury. 2022. Trust Enhancement Issues in Program Repair. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania) (ICSE '22). Association for Computing Machinery, New York, NY, USA, 2228--2240.
[31]
Richard E Petty, John T Cacioppo, Richard E Petty, and John T Cacioppo. 1986. The elaboration likelihood model of persuasion. Springer.
[32]
Alex Poole and Linden J. Ball. 2005. Eye Tracking in Human-Computer Interaction and Usability Research: Current Status and Future. In Prospects", Chapter in C. Ghaoui (Ed.): Encyclopedia of Human-Computer Interaction. Pennsylvania: Idea Group, Inc. Information Science Reference, Hershey, PA, 1--5.
[33]
K. Rayner. 1978. Eye movements in reading and information processing. Psychological Bulletin 85, 3 (1978), 618--660.
[34]
Tyler J Ryan, Gene M Alarcon, Charles Walter, Rose Gamble, Sarah A Jessup, August Capiola, and Marc D Pfahler. 2019. Trust in automated software repair: The effects of repair source, transparency, and programmer experience on perceived trustworthiness and trust. In HCI for Cybersecurity, Privacy and Trust: First International Conference, HCI-CPT 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26--31, 2019, Proceedings 21. Springer, 452--470.
[35]
Timothy R Shaffer, Jenna L Wise, Braden M Walters, Sebastian C Müller, Michael Falcone, and Bonita Sharif. 2015. itrace: Enabling eye tracking on software artifacts within the ide to support software engineering tasks. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. 954--957.
[36]
Jenessa R Shapiro and Steven L Neuberg. 2007. From stereotype threat to stereotype threats: Implications of a multi-threat framework for causes, moderators, mediators, consequences, and interventions. Personality and Social Psychology Review 11, 2 (2007), 107--130.
[37]
Zohreh Sharafi, Yu Huang, Kevin Leach, and Westley Weimer. 2020. Towards an objective measure of developers' cognitive activities. In Transactions on Software Engineering and Methodology (TOSEM). ACM, to appear.
[38]
Zohreh Sharafi, Timothy Shaffer, Bonita Sharif, and Yann-Gaël Guéhéneuc. 2015. Eye-tracking metrics in software engineering. In Proceeding of 2015 Asia-Pacific Software Engineering Conference (APSEC). IEEE, 96--103.
[39]
Zohreh Sharafi, Bonita Sharif, Yann-Gaël Guéhéneuc, Andrew Begel, Roman Bednarik, and Martha Crosby. 2020. A practical guide on conducting eye tracking studies in software engineering. Empirical Software Engineering (2020), 1--47.
[40]
Bonita Sharif, Michael Falcone, and Jonathan I. Maletic. 2012. An Eye-Tracking Study on the Role of Scan Time in Finding Source Code Defects. In Symposium on Eye Tracking Research and Applications.
[41]
Janet Siegmund, Christian Kästner, Jörg Liebig, Sven Apel, and Stefan Hanenberg. 2014. Measuring and modeling programming experience. Empirical Software Engineering 19, 5 (2014), 1299--1334.
[42]
Steven J Spencer, Claude M Steele, and Diane M Quinn. 1999. Stereotype threat and women's math performance. Journal of experimental social psychology 35, 1 (1999), 4--28.
[43]
Claude M Steele and Joshua Aronson. 1995. Stereotype threat and the intellectual test performance of African Americans. Journal of personality and social psychology 69, 5 (1995), 797.
[44]
Cass R Sunstein. 2005. Moral heuristics. Behavioral and brain sciences 28, 4 (2005), 531--541.
[45]
Jason Tsay, Laura Dabbish, and James Herbsleb. 2014. Influence of social and technical factors for evaluating contribution in GitHub. In Proceedings of the 36th international conference on Software engineering. 356--366.
[46]
Hidetake Uwano, Masahide Nakamura, Akito Monden, and Ken-ichi Matsumoto. 2006. Analyzing Individual Performance of Source Code Review Using Reviewers' Eye Movement. In Eye Tracking Research Applications. 133--140.
[47]
Jacob O Wobbrock, Leah Findlater, Darren Gergle, and James J Higgins. 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 143--146.
[48]
Xunhui Zhang, Yue Yu, Georgios Gousios, and Ayushi Rastogi. 2023. Pull Request Decisions Explained: An Empirical Overview. IEEE Transactions on Software Engineering 49, 2 (2023), 849--871.
Index Terms
- Exploring the Effects of Urgency and Reputation in Code Review: An Eye-Tracking Study
Recommendations
Trustworthiness Perceptions in Code Review: An Eye-tracking Study
ESEM '20: Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)Background: Automated program repair and other bug-fixing approaches are gaining attention in the software engineering community. Automation shows promise in reducing bug fixing costs. However, many developers express reluctance about accepting machine-...
Eye movements in code review
EMIP '18: Proceedings of the Workshop on Eye Movements in ProgrammingIn order to ensure sufficient quality, software engineers conduct code reviews to read over one another's code looking for errors that should be fixed before committing to their source code repositories. Many kinds of errors are spotted, from simple ...
Code review quality: how developers see it
ICSE '16: Proceedings of the 38th International Conference on Software EngineeringIn a large, long-lived project, an effective code review process is key to ensuring the long-term quality of the code base. In this work, we study code review practices of a large, open source project, and we investigate how the developers themselves ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
April 2024
487 pages
ISBN:9798400705861
DOI:10.1145/3643916
- Chair:
- Igor Steinmacher,
- Co-chair:
- Mario Linares-Vasquez,
- Program Chair:
- Kevin Patrick Moran,
- Program Co-chair:
- Olga Baysal
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ICPC '24
Sponsor:
ICPC '24: 32nd IEEE/ACM International Conference on Program Comprehension
April 15 - 16, 2024
Lisbon, Portugal
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 47Total Downloads
- Downloads (Last 12 months)47
- Downloads (Last 6 weeks)11
Reflects downloads up to 14 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in