Characterizing high-quality test methods: a first empirical study
Authors: 
Victor Veloso, Andre HoraAuthors Info & Claims
Pages 265 - 269
Abstract
To assess the quality of a test suite, one can rely on mutation testing, which computes whether the overall test cases are adequately exercising the covered lines. However, this high level of granularity may overshadow the quality of individual test methods. In this paper, we propose an empirical study to assess the quality of test methods by relying on mutation testing at the method level. We find no major differences between high-quality and low-quality test methods in terms of size, number of asserts, and modifications. In contrast, high-quality test methods are less affected by critical test smells. Finally, we discuss practical implications for researchers and practitioners.
References
[1]
Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and David Binkley. 2012. An empirical analysis of the distribution of unit test smells and their impact on software maintenance. In International Conference on Software Maintenance. 56--65.
[2]
Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and Dave Binkley. 2015. Are test smells really harmful? An empirical study. Empirical Software Engineering 20, 4 (2015), 1052--1094.
[3]
Hudson Borges, Andre Hora, and Marco Tulio Valente. 2016. Understanding the Factors that Impact the Popularity of GitHub Repositories. In International Conference on Software Maintenance and Evolution. 334--344.
[4]
Gemma Catolino, Fabio Palomba, Andy Zaidman, and Filomena Ferrucci. 2019. How the experience of development teams relates to assertion density of test classes. In International Conference on Software Maintenance and Evolution. 223--234.
[5]
Code Coverage Best Practices. November, 2020. https://testing.googleblog.com/2020/08/code-coverage-best-practices.html.
[6]
Codecov. November, 2020. https://codecov.io.
[7]
Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. Pit: a practical mutation testing tool for java. In International Symposium on Software Testing and Analysis. 449--452.
[8]
Coverage.py. November, 2020. https://coverage.readthedocs.io.
[9]
Coveralls. November, 2020. https://coveralls.io.
[10]
Moritz Eck, Fabio Palomba, Marco Castelluccio, and Alberto Bacchelli. 2019. Understanding Flaky Tests: The Developer's Perspective. In Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 830--840.
[11]
Giovanni Grano, Cristian De Iaco, Fabio Palomba, and Harald C. Gall. 2020. Pizza versus Pinsa: On the Perception and Measurability of Unit Test Code Quality. In International Conference on Software Maintenance and Evolution. 336--347.
[12]
Michael Hilton, Jonathan Bell, and Darko Marinov. 2018. A large-scale study of test coverage evolution. In International Conference on Automated Software Engineering. 53--63.
[13]
Marko Ivanković, Goran Petrović, René Just, and Gordon Fraser. 2019. Code coverage at Google. In Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 955--963.
[14]
Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. Transactions on Software Engineering (2011), 649--678.
[15]
Ayaan M Kazerouni, Clifford A Shaffer, Stephen H Edwards, and Francisco Servant. 2019. Assessing incremental testing practices and their impact on project outcomes. In Technical Symposium on Computer Science Education. 407--413.
[16]
Thomas Laurent and Anthony Ventresque. 2019. PIT-HOM: an Extension of Pitest for Higher Order Mutation Analysis. In International Conference on Software Testing, Verification and Validation Workshops. 83--89.
[17]
Valentina Lenarduzzi, Nyyti Saarimäki, and Davide Taibi. 2019. The technical debt dataset. In International Conference on Predictive Models and Data Analytics in Software Engineering. 2--11.
[18]
Mateusz Machalica, Alex Samylkin, Meredith Porth, and Satish Chandra. 2019. Predictive Test Selection. In International Conference on Software Engineering: Software Engineering in Practice. 91--100.
[19]
Robert C Martin. 2009. Clean code: a handbook of agile software craftsmanship. Pearson Education.
[20]
Gerard Meszaros. 2007. xUnit test patterns: Refactoring test code. Pearson Education.
[21]
Rainer Niedermayr, Elmar Juergens, and Stefan Wagner. 2016. Will My Tests Tell Me If I Break This Code?. In International Workshop on Continuous Software Evolution and Delivery. 23--29.
[22]
Fabio Palomba and Andy Zaidman. 2017. Does refactoring of test smells induce fixing flaky tests?. In International Conference on Software Maintenance and Evolution. 1--12.
[23]
A. Panichella, S. Panichella, G. Fraser, A. A. Sawant, and V. J. Hellendoorn. 2020. Revisiting Test Smells in Automatically Generated Tests: Limitations, Pitfalls, and Opportunities. In International Conference on Software Maintenance and Evolution. 523--533.
[24]
Fabiano Pecorelli, Fabio Palomba, and Andrea De Lucia. 2021. The Relation of Test-Related Factors to Software Quality: A Case Study on Apache Systems. Empirical Software Engineering 26, 2 (2021), 1--42.
[25]
Anthony Peruma, Khalid Almalki, Christian D. Newman, Mohamed Wiem Mkaouer, Ali Ouni, and Fabio Palomba. 2020. TsDetect: An Open Source Test Smells Detection Tool. In Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1650--1654.
[26]
PIT Mutation Testing. November, 2020. https://pitest.org.
[27]
August Shi, Alex Gyori, Milos Gligoric, Andrey Zaytsev, and Darko Marinov. 2014. Balancing Trade-Offs in Test-Suite Reduction. In Foundations of Software Engineering. 246--256.
[28]
Hudson Silva and Marco Tulio Valente. 2018. What's in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform. Journal of Systems and Software 146 (2018), 112--129.
[29]
Davide Spadini, Maurício Aniche, and Alberto Bacchelli. 2018. PyDriller: Python framework for mining software repositories. In Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 908--911.
[30]
D. Spadini, F. Palomba, A. Zaidman, M. Bruntink, and A. Bacchelli. 2018. On the Relation of Test Smells to Software Code Quality. In International Conference on Software Maintenance and Evolution. 1--12.
[31]
Davide Spadini, Martin Schvarcbacher, Ana-Maria Oprescu, Magiel Bruntink, and Alberto Bacchelli. 2020. Investigating Severity Thresholds for Test Smells. In International Conference on Mining Software Repositories. 311--321.
[32]
Michele Tufano, Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrea De Lucia, and Denys Poshyvanyk. 2016. An Empirical Investigation into the Nature of Test Smells. In International Conference on Automated Software Engineering. 4--15.
[33]
Arie Van Deursen, Leon Moonen, Alex Van Den Bergh, and Gerard Kok. 2001. Refactoring test code. In International Conference on Extreme Programming and Flexible Processes in Software Engineering (XP2001). 92--95.
[34]
Andreas Zeller, Rahul Gopinath, Marcel Böhme, Gordon Fraser, and Christian Holler. 2019. The Fuzzing Book. Saarland University. https://www.fuzzingbook.org
[35]
J. Zhang, L. Zhang, M. Harman, D. Hao, Y. Jia, and L. Zhang. 2019. Predictive Mutation Testing. Transactions on Software Engineering 45 (2019), 898--918.
Index Terms
- Characterizing high-quality test methods: a first empirical study
Recommendations
Comparing test quality measures for assessing student-written tests
ICSE Companion 2014: Companion Proceedings of the 36th International Conference on Software EngineeringMany educators now include software testing activities in programming assignments, so there is a growing demand for appropriate methods of assessing the quality of student-written software tests. While tests can be hand-graded, some educators also use ...
Faster mutation testing inspired by test prioritization and reduction
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and AnalysisMutation testing is a well-known but costly approach for determining test adequacy. The central idea behind the approach is to generate mutants, which are small syntactic transformations of the program under test, and then to measure for a given test ...
Please Fix This Mutant: How Do Developers Resolve Mutants Surfaced during Code Review?
ICSE-SEIP '23: Proceedings of the 45th International Conference on Software Engineering: Software Engineering in PracticeMutation testing has been demonstrated to motivate developers to write more tests when presented with undetected, actionable mutants. To facilitate this effect, modern mutation systems aim to generate and surface only actionable mutants---few in ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2022
815 pages
ISBN:9781450393034
DOI:10.1145/3524842
- General Chair:
- David Lo,
- Program Chairs:
- Shane McIntosh,
- Nicole Novielli
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
In-Cooperation
- IEEE CS
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 17 October 2022
Check for updates
Author Tags
Qualifiers
- Short-paper
Funding Sources
Conference
MSR '22
Sponsor:
MSR '22: 19th International Conference on Mining Software Repositories
May 23 - 24, 2022
Pennsylvania, Pittsburgh
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 54Total Downloads
- Downloads (Last 12 months)15
- Downloads (Last 6 weeks)0
Reflects downloads up to 01 Nov 2024
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in