The Role of Generative AI in Software Development Productivity: A Pilot Case Study
Authors: 
Mariana Coutinho, Lorena Marques, Anderson Santos, Marcio Dahia, Cesar França, Ronnie de Souza SantosAuthors Info & Claims
Pages 131 - 138
Abstract
With software development increasingly reliant on innovative technologies, there is a growing interest in exploring the potential of generative AI tools to streamline processes and enhance productivity. In this scenario, this paper investigates the integration of generative AI tools within software development, focusing on understanding their uses, benefits, and challenges to software professionals, in particular, looking at aspects of productivity. Through a pilot case study involving software practitioners working in different roles, we gathered valuable experiences on the integration of generative AI tools into their daily work routines. Our findings reveal a generally positive perception of these tools in individual productivity while also highlighting the need to address identified limitations. Overall, our research sets the stage for further exploration into the evolving landscape of software development practices with the integration of generative AI tools.
References
[1]
A. J. Albrecht. 1979. Measuring Application Development Productivity. In Proceedings of IBM Applications Development Symposium. Monterey. 83.
[2]
Amazon Web Services. 2023. Amazon CodeWhisperer. https://aws.amazon.com/codewhisperer/ Accessed: 2023-12-10
[3]
Barry Boehm. 2000. Software Cost Estimation with COCOMO II. Prentice Hall, Upper Saddle River.
[4]
Alexia Cambon, Brent Hecht, Benjamin Edelman, Donald Ngwe, Sonia Jaffe, Amy Heger, Mihaela Vorvoreanu, Sida Peng, Jake Hofman, and Alex Farach. 2023. Early LLM-based Tools for Enterprise Information Workers Likely Provide Meaningful Boosts to Productivity. MSFT Technical Report. https://www. microsoft. com/en-us/research ….
[5]
Kathy Charmaz. 2014. Constructing grounded theory. sage.
[6]
Daniela S Cruzes and Tore Dyba. 2011. Recommended steps for thematic synthesis in software engineering. In 2011 international symposium on empirical software engineering and measurement. 275–284.
[7]
McKinsey Digital. 2023. Unleashing developer productivity with generative AI. https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/unleashing-developer-productivity-with-generative-ai Acessed in Mar 22, 2024
[8]
Christof Ebert and Panos Louridas. 2023. Generative AI for software practitioners. IEEE Software, 40, 4 (2023), 30–38.
[9]
Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler. 2021. The SPACE of Developer Productivity: There’s more to it than you think. Queue, 19, 1 (2021), 20–48.
[10]
Github. 2021. GitHub Copilot. https://copilot.github.com Accessed on November 23, 2023
[11]
Marcela Guerrero-Calvache and Giovanni Hernández. 2022. Team productivity in agile software development: a systematic mapping study. In International Conference on Applied Informatics. 455–471.
[12]
Martin Hoegl, K Praveen Parboteeah, and Hans Georg Gemuenden. 2003. When teamwork really matters: task innovativeness as a moderator of the teamwork–performance relationship in software development projects. Journal of Engineering and Technology Management, 20, 4 (2003), 281–302.
[13]
Eirini Kalliamvakou. 2022. Research: quantifying GitHub Copilot’s impact on developer productivity and happiness. https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/ Acessed in Mar 22, 2024
[14]
Young-Ho Kim, Eun Kyoung Choe, Bongshin Lee, and Jinwook Seo. 2019. Understanding personal productivity: How knowledge workers define, evaluate, and reflect on their productivity. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
[15]
B Lakhanpal. 1993. Understanding the factors influencing the performance of software development groups: An exploratory group-level analysis. Information and Software Technology, 35, 8 (1993), 468–473. issn:0950-5849 https://doi.org/10.1016/0950-5849(93)90044-4
[16]
Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and ZHAO-XIANG ZHANG. 2024. SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models. Advances in Neural Information Processing Systems, 36 (2024).
[17]
Yngve Lindsjørn, Dag IK Sjøberg, Torgeir Dingsøyr, Gunnar R Bergersen, and Tore Dybå. 2016. Teamwork quality and project success in software development: A survey of agile development teams. Journal of Systems and Software, 122 (2016), 274–286.
[18]
Jefferson Seide Molléri, Kai Petersen, and Emilia Mendes. 2016. Survey guidelines in software engineering: An annotated review. In Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement. 1–6.
[19]
Cleviton VF Monteiro, Fabio QB da Silva, and Luiz Fernando Capretz. 2016. The innovative behaviour of software engineers: Findings from a pilot case study. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 1–10.
[20]
Mauricio Monteiro, Bruno Castelo Branco, Samuel Silvestre, Guilherme Avelino, and Marco Tulio Valente. 2023. End-to-End Software Construction using ChatGPT: An Experience Report. arXiv preprint arXiv:2310.14843.
[21]
Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an llm to help with code understanding. In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). 881–881.
[22]
Shakked Noy and Whitney Zhang. 2023. Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381, 6654 (2023), 187–192.
[23]
OpenAI. 2023. ChatGPT: Optimizing Language Models for Dialogue. https://openai.com/blog/chatgpt Acessado em 10 de dezembro de 2023
[24]
Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590.
[25]
Rafael Prikladnicki, Yvonne Dittrich, Helen Sharp, Cleidson De Souza, Marcelo Cataldo, and Rashina Hoda. 2013. Cooperative and human aspects of software engineering: Chase 2013. ACM SIGSOFT Software Engineering Notes, 38, 5 (2013), 34–37.
[26]
Daniel Rodríguez, MA Sicilia, E García, and Rachel Harrison. 2012. Empirical findings on team size and productivity in software development. Journal of Systems and Software, 85, 3 (2012), 562–570.
[27]
Per Runeson and Martin Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering, 14 (2009), 131–164.
[28]
Anastasia Ruvimova, Alexander Lill, Jan Gugler, Lauren Howe, Elaine Huang, Gail Murphy, and Thomas Fritz. 2022. An exploratory study of productivity perceptions in software teams. In Proceedings of the 44th International Conference on Software Engineering. 99–111.
[29]
Caitlin Sadowski and Thomas Zimmermann. 2019. Rethinking productivity in software engineering. Springer Nature.
[30]
Carolyn B. Seaman. 1999. Qualitative methods in empirical studies of software engineering. IEEE Transactions on software engineering, 25, 4 (1999), 557–572.
[31]
Samarth Sikand, Kanchanjot Kaur Phokela, Vibhu Saujanya Sharma, Kapil Singi, Vikrant Kaulgud, Teresa Tung, Pragya Sharma, and Adam P Burden. 2024. How much SPACE do metrics have in GenAI assisted software development? In Proceedings of the 17th Innovations in Software Engineering Conference. 1–5.
[32]
Diane Strode, Torgeir Dingsøyr, and Yngve Lindsjorn. 2022. A teamwork effectiveness model for agile software development. Empirical Software Engineering, 27, 2 (2022), 56.
[33]
Claes Wohlin and Mattias Ahlgren. 1995. Soft factors and their impact on time to market. Software Quality Journal, 4, 3 (1995), 189–205. https://doi.org/10.1007/bf01351923
[34]
Qing Xie and Atif M Memon. 2008. Using a pilot study to derive a GUI model for automated testing. ACM Transactions on Software Engineering and Methodology (TOSEM), 18, 2 (2008), 1–35.
[35]
Robert K Yin. 1994. Discovering the future of the case study. Method in evaluation research. Evaluation practice, 15, 3 (1994), 283–290.
[36]
Beiqi Zhang, Peng Liang, Xiyu Zhou, Aakash Ahmad, and Muhammad Waseem. 2023. Practices and challenges of using github copilot: An empirical study. arXiv preprint arXiv:2303.08733.
Index Terms
- The Role of Generative AI in Software Development Productivity: A Pilot Case Study
Recommendations
Impacts of the Usage of Generative Artificial Intelligence on Software Development Process
SBSI '24: Proceedings of the 20th Brazilian Symposium on Information SystemsContext: Over the years, tools have been created to improve the execution of development process activities. The emergence of generative Artificial Intelligence (AI) and, more recently, the launch and dissemination of Copilot, ChatGPT-3 and other ...
Measuring productivity in agile software development process: a scoping study
ICSSP 2015: Proceedings of the 2015 International Conference on Software and System ProcessAn agile software development process is often claimed to increase productivity. However, productivity measurement in agile software development is little researched. Measures are not explicitly defined nor commonly agreed upon. In this paper, we ...
"Leagile" software development
In recent years there has been a noticeable shift in attention from those who use agile software development toward lean software development, often labelled as a shift "from agile to lean". However, the reality may not be as simple or linear as this ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2024
182 pages
ISBN:9798400706851
DOI:10.1145/3664646
- General Chairs:
- Bram Adams,
- Thomas Zimmermann,
- Ipek Ozkaya,
- Program Chairs:
- Dayi Lin,
- Jie M. Zhang
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 10 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
AIware '24
Sponsor:
AIware '24: 1st ACM International Conference on AI-Powered Software
July 15 - 16, 2024
Porto de Galinhas, Brazil
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 138Total Downloads
- Downloads (Last 12 months)138
- Downloads (Last 6 weeks)138
Reflects downloads up to 14 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in