research-article

A Dissection of the Test-Driven Development Process: Does It Really Matter to Test-First or to Test-Last?

Authors:

Hakan Erdogmus,

Natalia JuristoAuthors Info & Claims

IEEE Transactions on Software Engineering, Volume 43, Issue 7

Pages 597 - 614

https://doi.org/10.1109/TSE.2016.2616877

Published: 01 July 2017 Publication History

Abstract

<italic>Background</italic>: Test-driven development (TDD) is a technique that repeats short coding cycles interleaved with testing. The developer first writes a unit test for the desired functionality, followed by the necessary production code, and refactors the code. Many empirical studies neglect unique process characteristics related to TDD iterative nature. <italic>Aim</italic>: We formulate four process characteristic: sequencing, granularity, uniformity, and refactoring effort. We investigate how these characteristics impact quality and productivity in TDD and related variations. <italic>Method</italic>: We analyzed 82 data points collected from 39 professionals, each capturing the process used while performing a specific development task. We built regression models to assess the impact of process characteristics on quality and productivity. Quality was measured by functional correctness. <italic>Result</italic>: Quality and productivity improvements were primarily positively associated with the granularity and uniformity. Sequencing, the order in which test and production code are written, had no important influence. Refactoring effort was negatively associated with both outcomes. We explain the unexpected negative correlation with quality by possible prevalence of mixed refactoring. <italic>Conclusion</italic>: The claimed benefits of TDD may not be due to its distinctive test-first dynamic, but rather due to the fact that TDD-like processes encourage fine-grained, steady steps that improve focus and flow.

References

[1]

R. Jeffries and G. Melnik, “TDD–The art of fearless programming,” IEEE Software, vol. 24, no. 3, pp. 24–30, May/Jun. 2007.

Digital Library

[2]

K. Beck, Test-Driven Development: By Example. Reading, MA, USA :Addison-Wesley, 2003.

Digital Library

[3]

R. C. Martin, Agile Software Development: Principles, Patterns, and Practices. Upper Saddle River, NJ, USA: Prentice-Hall, 2003.

Digital Library

[4]

R. Jeffries, A. Anderson, and C. Hendrickson, Extreme Programming Installed. Reading, MA, USA: Addison-Wesley, 2001.

Digital Library

[5]

K. Schwaber, “Scrum development process,” in Business Object Design and Implementation. Berlin, Germany: Springer, 1997, pp. 117–134.

[6]

H. Erdogmus, G. Melnik, and R. Jeffries, “ Test-driven development,” in Encyclopedia of Software Engineering, P. Laplante, Ed. New York, NY, USA: Taylor and Francis, 2009, pp. 1211–1229.

[7]

M. Fowler, Refactoring: Improving the Design of Existing Code. Upper Saddle River, NJ, USA: Pearson Education, 2009.

[8]

P. Wei ßgerber and S. Diehl, “Are refactorings less error-prone than other changes?” in Proc. Int. Workshop Mining Softw. Repositories, 2006, pp. 112 –118.

[9]

G. Bavota, B. De Carluccio, A. De Lucia, M. Di Penta, R. Oliveto, and O. Strollo, “When does a refactoring induce bugs? an empirical study,” in Proc. IEEE 12th Int. Working Conf. Source Code Anal. Manipulation, 2012, pp. 104–113.

[10]

A. Causevic, D. Sundmark, and S. Punnekkat, “Factors limiting industrial adoption of test-driven development: A systematic review,” in Proc. IEEE 4th Int. Conf. Softw. Testing Verification Validation, 2011, pp. 337–346.

[11]

H. Erdogmus, M. Morisio, and M. Torchiano, “On the effectiveness of the test-first approach to programming,” IEEE Trans. Softw. Eng., vol. 31, no. 3, pp. 226–237, Mar. 2005.

Digital Library

[12]

M. M. Müller and A. Höfer, “The effect of experience on the test-driven development process,” Empirical Softw. Eng., vol. 12, no. 6, pp. 593–615, 2007.

Digital Library

[13]

M. Pan čur and M. Ciglarič, “Impact of test-driven development on productivity, code and tests: A controlled experiment,” Inf. Softw. Technol., vol. 53, no. 6, pp. 557–573, 2011.

[14]

Y. Rafique and V. B. Mišić, “The effects of test-driven development on external quality and productivity: A meta-analysis,” IEEE Trans. Softw. Eng., vol. 39, no. 6, pp. 835–856, Jun. 2013 .

Digital Library

[15]

Y. Wang and H. Erdogmus, “The role of process measurement in test-driven development,” in Proc. 4th Conf. Extreme Program. Agile Methods, 2004, pp. 32 –42.

[16]

C. Mann, “Observational research methods. Research design II: Cohort, cross-sectional, and case-control studies,” Emergency Med. J., vol. 20, no. 1, pp. 54–60, 2003.

[17]

A. T. Misirli, H. Erdogmus, N. Juristo, and O. Dieste, “Topic selection in industry experiments,” in Proc. 2nd Int. Workshop, 2014, pp. 25–30.

[18]

D. Fucci and B. Turhan, “On the role of tests in test-driven development: A differentiated and partial replication,” Empirical Softw. Eng., vol. 19, pp. 1– 26, 2013.

[19]

D. Fucci, B. Turhan, and M. Oivo, “Impact of process conformance on the effects of test-driven development,” in Proc. 8th ACM/IEEE Int. Symp. Empirical Softw. Eng. Meas., 2014, Art. no. 10.

Digital Library

[20]

L. Madeyski, “The impact of pair programming and test-driven development on package dependencies in object-oriented design—an experiment,” in Proc. 7th Int. Conf. Product-Focused Softw. Process Improvement, 2006, pp. 278–289.

Digital Library

[21]

A. Hernández-López, R. Colomo-Palacios, and A. García-Crespo, “Productivity in software engineering: a study of its meanings for practitioners: Understanding the concept under their standpoint,” in Proc. 7th Iberian Conf. Inf. Syst. Technol., Jun. 2012, pp. 1–6.

[22]

K. Becker, B. Pedroso, and M. S. Pimenta, “ Besouro: A framework for exploring compliance rules in automatic TDD behavior assessment,” Inf. Softw. Syst. J., vol. 57, pp. 494–508, 2015.

[23]

H. Kou, P. M. Johnson, and H. Erdogmus, “Operational definition and automated inference of test-driven development with Zorro,” Automat. Softw. Eng. , vol. 17, pp. 57–85, 2010.

Digital Library

[24]

B. George and L. Williams, “An initial investigation of test driven development in industry,” in Proc. ACM Symp. Appl. Comput., 2003, pp. 1135– 1139.

[25]

D. Fucci, “A dissection of test-driven development process: Does it really matter to test-first or to test-last?-lab package,” 2016. [Online]. Available: https://dx.doi.org/10.6084/m9.figshare.3458996.v5

[26]

D. Fucci and B. Turhan, “A replicated experiment on the effectiveness of test-first development,” in Proc. ACM/IEEE Int. Symp. Empirical Softw. Eng. Meas., 2013, pp. 103–112.

[27]

M. Aniche and M. Gerosa, “Most common mistakes in test-driven development practice: Results from an online survey with developers,” in Proc. 3rd Int. Conf. Softw. Testing Verification Validation Workshops, Apr. 2010, pp. 469–478.

[28]

E. McCrum-Gardner, “Which is the correct statistical test to use? ” British J. Oral Maxillofacial Surgery, vol. 46, no. 1, pp. 38–41, Jan. 2008.

[29]

P. D. Ellis, The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results. Cambridge, U.K.: Cambridge Univ. Press, 2010.

[30]

K. P. Burnham and D. R. Anderson, “Multimodel inference understanding AIC and BIC in model selection,” Sociological Methods Res., vol. 33, no. 2, pp. 261 –304, 2004.

[31]

J. Cohen, “A power primer,” Psychological Bulletin, vol. 112, no. 1, 1992, Art. no. 155.

[32]

V. B. Kampenes, T. Dybå, J. E. Hannay, and D. I. Sjøberg, “A systematic review of effect size in software engineering experiments,” Inf. Softw. Technol., vol. 49, no. 11, pp. 1073–1086, 2007 .

Digital Library

[33]

D. E. Farrar and R. R. Glauber, “Multicollinearity in regression analysis: The problem revisited,” Rev. Econ. Statistics, vol. 49, pp. 92–107, 1967.

[34]

J. Jobson, Applied Multivariate Data Analysis: Regression and Experimental Design (Springer Texts in Statistics). Berlin, Germany: Springer, 1999.

[35]

H. Akaike, “A new look at the statistical model identification,” IEEE Trans. Automat. Control, vol. 19, no. 6, pp. 716–723, Dec. 1974.

[36]

K. P. Burnham and D. R. Anderson, Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach. Berlin, Germany: Springer, 2003.

[37]

Y. Yang, “Can the strengths of AIC and BIC be shared? A conflict between model indentification and regression estimation,” Biometrika, vol. 92, no. 4, pp. 937–950, 2005.

[38]

E. Murphy-Hill, C. Parnin, and A. P. Black, “How we refactor, and how we know it,” IEEE Trans. Softw. Eng., vol. 38, no. 1, pp. 5–18, Jan. 2012.

Digital Library

[39]

E. A. Pena and E. H. Slate, “Global validation of linear model assumptions,” J. Amer. Statist. Assoc., vol. 101, no. 473, pp. 341–354, 2006.

[40]

W. Cleveland and S. Devlin, “Locally weighted regression: An approach to regression analysis by local fitting,” J. Amer. Statist. Assoc., vol. 83, no. 403, pp. 596 –610, 1988.

[41]

Y. Xia, “Model checking in regression via dimension reduction,” Biometrika, vol. 96, no. 1, pp. 133–148, 2009.

[42]

C. Wohlin, Experimentation in Software Engineering: An Introduction, vol. 6. Berlin, Germany: Springer, 2000 .

Digital Library

[43]

B. Turhan, L. Layman, M. Diep, H. Erdogmus, and F. Shull, “How effective is test-driven development?” in Making Software. Sebastopol, CA, USA: O’Reilly Media, 2010.

[44]

H. Munir, M. Moayyed, and K. Petersen, “ Considering rigor and relevance when evaluating test driven development: A systematic review,” Inf. Softw. Technol., vol. 56, no. 4, pp. 375– 394, Apr. 2014.

Digital Library

[45]

A. Causevic, D. Sundmark, and S. Punnekkat, “Test case quality in test driven development: A study design and a pilot experiment,” in Proc. 16th Int. Conf. Eval. Assessment Softw. Eng., 2012, pp. 223–227.

[46]

D. Fucci, B. Turhan, and M. Oivo, “Conformance factor in test-driven development: Initial results from an enhanced replication,” in Proc. 18th Int. Conf. Eval. Assessment Softw. Eng., 2014, pp. 22:1–22:4. [Online]. Available: http://doi.acm.org/10.1145/2601248.2601272

Cited By

Tufano RMastropaolo APepe FDabic ODi Penta MBavota GSpinellis DConstantinou EBacchelli A(2024)Unveiling ChatGPT's Usage in Open Source Projects: A Mining-based StudyProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644918(571-583)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644918
Fucci D(2024)FAIR enough: a Vision for Research Objects in Empirical Software Engineering StudiesProceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering10.1145/3643664.3648201(64-67)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1145/3643664.3648201
Lemos OSilveira FFerrari FSilva TGuerra EGarcia A(2024)Unraveling the code: an in-depth empirical study on the impact of development practices in auxiliary functions implementationSoftware Quality Journal10.1007/s11219-024-09682-432:3(1137-1174)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s11219-024-09682-4
Show More Cited By

Index Terms

A Dissection of the Test-Driven Development Process: Does It Really Matter to Test-First or to Test-Last?

Index terms have been assigned to the content through auto-classification.

Recommendations

An experimental evaluation of test driven development vs. test-last development with industry professionals
EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

Test-Driven Development (TDD) is a software development approach where test cases are written before actual development of the code in iterative cycles. Context: TDD has gained attention of many software practitioners during the last decade since it has ...
Test-Driven Development: Concepts, Taxonomy, and Future Direction

Test-driven development creates software in very short iterations with minimal upfront design. Poised for widespread adoption TDD has become the focus of an increasing number of researchers and developers.
Realizing quality improvement through test driven development: results and experiences of four industrial teams

Test-driven development (TDD) is a software development practice that has been used sporadically for decades. With this practice, a software engineer cycles minute-by-minute between writing failing unit tests and writing implementation code to pass ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Software Engineering

IEEE Transactions on Software Engineering Volume 43, Issue 7

July 2017

104 pages

ISSN:0098-5589

Issue’s Table of Contents

0098-5589 © 2016 IEEE.

Publisher

IEEE Press

Publication History

Published: 01 July 2017

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tufano RMastropaolo APepe FDabic ODi Penta MBavota GSpinellis DConstantinou EBacchelli A(2024)Unveiling ChatGPT's Usage in Open Source Projects: A Mining-based StudyProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644918(571-583)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644918
Fucci D(2024)FAIR enough: a Vision for Research Objects in Empirical Software Engineering StudiesProceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering10.1145/3643664.3648201(64-67)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1145/3643664.3648201
Lemos OSilveira FFerrari FSilva TGuerra EGarcia A(2024)Unraveling the code: an in-depth empirical study on the impact of development practices in auxiliary functions implementationSoftware Quality Journal10.1007/s11219-024-09682-432:3(1137-1174)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s11219-024-09682-4
van Heugten Breurkes JGilson FGalster M(2022)Overlap between Automated Unit and Acceptance Testing – a Systematic Literature ReviewProceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering10.1145/3530019.3530028(80-89)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3530019.3530028
Papis BGrochowski KSubzda KSijko K(2022)Experimental Evaluation of Test-Driven Development With Interns Working on a Real Industrial ProjectIEEE Transactions on Software Engineering10.1109/TSE.2020.302752248:5(1644-1664)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1109/TSE.2020.3027522
Ljung KGonzalez-Huerta J(2022)“To Clean Code or Not to Clean Code” A Survey Among PractitionersProduct-Focused Software Process Improvement10.1007/978-3-031-21388-5_21(298-315)Online publication date: 21-Nov-2022
https://dl.acm.org/doi/10.1007/978-3-031-21388-5_21
Shrikanth NNichols WFahid FMenzies T(2021)Assessing practitioner beliefs about software engineeringEmpirical Software Engineering10.1007/s10664-021-09957-526:4Online publication date: 1-Jul-2021
https://dl.acm.org/doi/10.1007/s10664-021-09957-5
Santos AVegas SDieste OUyaguari FTosun AFucci DTurhan BScanniello GRomano SKarac IKuhrmann MMandić VRamač RPfahl DEngblom CKyykka JRungi KPalomeque CSpisak JOivo MJuristo N(2021)A family of experiments on test-driven developmentEmpirical Software Engineering10.1007/s10664-020-09895-826:3Online publication date: 1-May-2021
https://dl.acm.org/doi/10.1007/s10664-020-09895-8
Roman AMnich M(2020)Test-driven development with mutation testing – an experimental studySoftware Quality Journal10.1007/s11219-020-09534-x29:1(1-38)Online publication date: 18-Nov-2020
https://dl.acm.org/doi/10.1007/s11219-020-09534-x
Kazerouni AShaffer CEdwards SServant FHawthorne EPérez-Quiñones MHeckman SZhang J(2019)Assessing Incremental Testing Practices and Their Impact on Project OutcomesProceedings of the 50th ACM Technical Symposium on Computer Science Education10.1145/3287324.3287366(407-413)Online publication date: 22-Feb-2019
https://dl.acm.org/doi/10.1145/3287324.3287366
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents