research-article

Faster mutation testing inspired by test prioritization and reduction

Authors:

Lingming Zhang,

Sarfraz KhurshidAuthors Info & Claims

ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and Analysis

Pages 235 - 245

https://doi.org/10.1145/2483760.2483782

Published: 15 July 2013 Publication History

Abstract

Mutation testing is a well-known but costly approach for determining test adequacy. The central idea behind the approach is to generate mutants, which are small syntactic transformations of the program under test, and then to measure for a given test suite how many mutants it kills. A test t is said to kill a mutant m of program p if the output of t on m is different from the output of t on p. The effectiveness of mutation testing in determining the quality of a test suite relies on the ability to apply it using a large number of mutants. However, running many tests against many mutants is time consuming. We present a family of techniques to reduce the cost of mutation testing by prioritizing and reducing tests to more quickly determine the sets of killed and non-killed mutants. Experimental results show the effectiveness and efficiency of our techniques.

References

[1]

PIT. http://pitest.org/.

[2]

P. Ammann and J. Offutt. Introduction to Software Testing. Cambridge University Press.

Digital Library

[3]

J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In Proc. ICSE, pages 402–411, 2005.

Digital Library

[4]

E. F. Barbosa, J. C. Maldonado, and A. M. R. Vincenzi. Toward the determination of sufficient mutant operators for C. STVR, 11(2):113–136, 2001.

[5]

J. Black, E. Melachrinoudis, and D. Kaeli. Bi-criteria models for all-uses test suite reduction. In Proc. ICSE, pages 106–115, 2004.

Digital Library

[6]

T. Chen and M. Lau. A new heuristic for test suite reduction. IST, 40(5):347–354, 1998.

[7]

R. A. DeMillo, E. W. Krauser, and A. P. Mathur. Compiler-integrated program mutation. In Proc. COMPSAC, pages 351–356, 1991.

[8]

R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: Help for the practicing programmer. Computer, 11(4):34–41, 1978.

Digital Library

[9]

H. Do and G. Rothermel. On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE TSE, 32(9):733–752, 2006.

Digital Library

[10]

S. Elbaum, A. Malishevsky, and G. Rothermel. Test case prioritization: A family of empirical studies. IEEE TSE, 28(2):159–182, 2002.

Digital Library

[11]

M. Gligoric, L. Zhang, C. Pereira, and G. Pokam. Selective mutation testing for concurrent code. In Proc. ISSTA, page to appear, 2013.

Digital Library

[12]

R. G. Hamlet. Testing programs with the aid of a compiler. IEEE TSE, pages 279–290, 1977.

Digital Library

[13]

D. Hao, L. Zhang, X. Wu, H. Mei, and G. Rothermel. On-demand test suite reduction. In Proc. ICSE, pages 738–748, 2012.

Digital Library

[14]

M. Harman, Y. Jia, and W. Langdon. Strong higher order mutation-based test data generation. In Proc. FSE, pages 212–222, 2011.

Digital Library

[15]

M. Harrold, R. Gupta, and M. Soffa. A methodology for controlling the size of a test suite. ACM TOSEM, 2(3):270–285, 1993.

Digital Library

[16]

M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings, S. Sinha, S. A. Spoon, and A. Gujarathi. Regression test selection for java software. In Proc. OOPSLA, pages 312–326, 2001.

Digital Library

[17]

W. E. Howden. Weak mutation testing and completeness of test sets. IEEE TSE, pages 371–379, 1982.

Digital Library

[18]

Y. Jia and M. Harman. Higher order mutation testing. Information and Software Technology, 51(10):1379–1393, 2009.

Digital Library

[19]

Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. IEEE TSE, 37(5):649–678, 2011.

Digital Library

[20]

R. Just, G. M. Kapfhammer, and F. Schweiggert. Using non-redundant mutation operators and test suite prioritization to achieve efficient and scalable mutation analysis. In Proc. ISSRE, pages 11–20, 2012.

Digital Library

[21]

K. N. King and A. J. Offutt. A Fortran language system for mutation-based software testing. Software: Practice and Experience, 21(7):685–718, 1991.

Digital Library

[22]

A. P. Mathur. Performance, effectiveness, and reliability issues in software testing. In Proc. COMPSAC, pages 604–605, 1991.

[23]

A. P. Mathur and E. W. Krauser. Mutant unification for improved vectorization. Technical report, Purdue University, 1988.

[24]

H. Mei, D. Hao, L. Zhang, L. Zhang, J. Zhou, and G. Rothermel. A static approach to prioritizing junit test cases. TSE, 38(6):1258–1275, 2012.

Digital Library

[25]

A. S. Namin, J. H. Andrews, and D. J. Murdoch. Sufficient mutation operators for measuring test effectiveness. In Proc. ICSE, pages 351–360, 2008.

Digital Library

[26]

A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf. An experimental determination of sufficient mutation operators. ACM TOSEM, 5(2):99–118, 1996.

Digital Library

[27]

A. Orso, N. Shi, and M. J. Harrold. Scaling regression testing to large software systems. In Proc. FSE, pages 241–252, 2004.

Digital Library

[28]

G. Rothermel and M. J. Harrold. A safe, efficient regression test selection technique. ACM TOSEM, 6(2):173–210, 1997.

Digital Library

[29]

G. Rothermel, R. Untch, C. Chu, and M. Harrold. Prioritizing test cases for regression testing. IEEE TSE, 27(10):929–948, 2001.

Digital Library

[30]

D. Schuler, V. Dallmeier, and A. Zeller. Efficient mutation testing by checking invariant violations. In Proc. ISSTA, pages 69–80, 2009.

Digital Library

[31]

D. Schuler and A. Zeller. Javalanche: Efficient mutation testing for Java. In Proc. FSE, pages 297–298, 2009.

Digital Library

[32]

R. Untch, A. J. Offutt, and M. J. Harrold. Mutation analysis using program schemata. In Proc. ISSTA, pages 139–148, 1993.

Digital Library

[33]

L. J. Williams and H. Abdi. Fisher’s least significant difference (LSD) test. Encyclopedia of Research Design, 2010.

[34]

W. E. Wong and A. P. Mathur. Reducing the cost of mutation testing: An empirical study. JSS, 31(3):185–196, 1995.

Digital Library

[35]

S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: a survey. STVR, 22(2):67–120, 2012.

Digital Library

[36]

L. Zhang, D. Hao, L. Zhang, G. Rothermel, and H. Mei. Bridging the gap between the total and additional test-case prioritization strategies. In Proc. ICSE, pages 192–201, 2013.

Digital Library

[37]

L. Zhang, S. S. Hou, J. J. Hu, T. Xie, and H. Mei. Is operator-based mutant selection superior to random mutant selection? In Proc. ICSE, pages 435–444, 2010.

Digital Library

[38]

L. Zhang, D. Marinov, L. Zhang, and S. Khurshid. An empirical study of junit test-suite reduction. In Proc. ISSRE, pages 170–179, 2011.

Digital Library

[39]

L. Zhang, D. Marinov, L. Zhang, and S. Khurshid. Regression mutation testing. In Proc. ISSTA, pages 331–341, 2012.

Digital Library

[40]

H. Zhu, P. A. V. Hall, and J. H. R. May. Software unit test coverage and adequacy. ACM Comput. Surv., 29(4):366–427, Dec. 1997.

Digital Library

Cited By

Ghanbari AChristakis MPradel M(2024)Decomposition of Deep Neural Networks into Modules via Mutation AnalysisProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680390(1669-1681)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680390
Wang HYang KWu T(2024)Optimizing Mutation-Based Fault Localization Through Contribution-Based Test Case ReductionInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402450027X34:10(1537-1564)Online publication date: 5-Jul-2024
https://doi.org/10.1142/S021819402450027X
Xiao YYang CWang BXiong Y(2024)Accelerating Patch Validation for Program Repair With Interception-Based Execution SchedulingIEEE Transactions on Software Engineering10.1109/TSE.2024.335996950:3(618-635)Online publication date: Mar-2024
https://doi.org/10.1109/TSE.2024.3359969
Show More Cited By

Index Terms

Faster mutation testing inspired by test prioritization and reduction
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Evaluating non-adequate test-case reduction
ASE '16: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

Given two test cases, one larger and one smaller, the smaller test case is preferred for many purposes. A smaller test case usually runs faster, is easier to understand, and is more convenient for debugging. However, smaller test cases also tend to ...
Mutation testing cost reduction by clustering overlapped mutants

We defined the term conditionally-overlapped (c-overlapped) mutants.C-overlapped mutants are expected to produce the same results against a test case.Clustering c-overlapped mutants effectively reduces the cost of mutation testing.Clustering c-...
Experimental Comparison of Code-Based and Model-Based Test Prioritization
ICSTW '09: Proceedings of the IEEE International Conference on Software Testing, Verification, and Validation Workshops

During regression testing, a modified system needs to beretested using the existing test suite. Since test suites may be very large, developers are interested in detecting faults in the system as early as possible. Test prioritization orders test cases ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and Analysis

July 2013

381 pages

ISBN:9781450321594

DOI:10.1145/2483760

General Chair:
Mauro Pezzè,
Program Chair:
Mark Harman

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 July 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '13

Sponsor:

SIGSOFT

ISSTA '13: Iitsnternational Symposium on Software Testing and Analysis

July 15 - 20, 2013

Lugano, Switzerland

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

73
Total Citations
View Citations
651
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)2

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ghanbari AChristakis MPradel M(2024)Decomposition of Deep Neural Networks into Modules via Mutation AnalysisProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680390(1669-1681)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680390
Wang HYang KWu T(2024)Optimizing Mutation-Based Fault Localization Through Contribution-Based Test Case ReductionInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402450027X34:10(1537-1564)Online publication date: 5-Jul-2024
https://doi.org/10.1142/S021819402450027X
Xiao YYang CWang BXiong Y(2024)Accelerating Patch Validation for Program Repair With Interception-Based Execution SchedulingIEEE Transactions on Software Engineering10.1109/TSE.2024.335996950:3(618-635)Online publication date: Mar-2024
https://doi.org/10.1109/TSE.2024.3359969
Liu HLi ZHan BLiu YChen XLiu Y(2024)Delta4Ms: Improving mutation‐based fault localization by eliminating mutant biasSoftware Testing, Verification and Reliability10.1002/stvr.187234:4Online publication date: 16-Jan-2024
https://doi.org/10.1002/stvr.1872
Garg AOjdanic MDegiovanni RChekam TPapadakis MLe Traon Y(2023)Cerebro: Static Subsuming Mutant SelectionIEEE Transactions on Software Engineering10.1109/TSE.2022.314051049:1(24-43)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3140510
Lévai ZMcMinn P(2023)Batching Non-Conflicting Mutations for Efficient, Safe, Parallel Mutation Analysis in Rust2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00014(49-59)Online publication date: Apr-2023
https://doi.org/10.1109/ICST57152.2023.00014
Xia CDutta SMisailovic SMarinov DZhang L(2023)Balancing Effectiveness and Flakiness of Non-Deterministic Machine Learning Tests2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00154(1801-1813)Online publication date: May-2023
https://doi.org/10.1109/ICSE48619.2023.00154
Xiao YYang CWang BXiong Y(2023)ExpressAPR: Efficient Patch Validation for Java Automated Program Repair Systems2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00012(2038-2041)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00012
Vercammen SDemeyer SBorg MPettersson NHedin G(2023)Mutation testing optimisations using the Clang front‐endSoftware Testing, Verification and Reliability10.1002/stvr.1865Online publication date: 17-Oct-2023
https://doi.org/10.1002/stvr.1865
Cornejo OPastore FBriand LDwyer M(2022)MASSProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings10.1145/3510454.3516840(134-138)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510454.3516840
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents