research-article

On the "naturalness" of buggy code

Authors:

Vincent Hellendoorn,

Saheel Godhane,

Alberto Bacchelli,

Premkumar DevanbuAuthors Info & Claims

ICSE '16: Proceedings of the 38th International Conference on Software Engineering

Pages 428 - 439

https://doi.org/10.1145/2884781.2884848

Published: 14 May 2016 Publication History

Abstract

Real software, the kind working programmers produce by the kLOC to solve real-world problems, tends to be "natural", like speech or natural language; it tends to be highly repetitive and predictable. Researchers have captured this naturalness of software through statistical models and used them to good effect in suggestion engines, porting tools, coding standards checkers, and idiom miners. This suggests that code that appears improbable, or surprising, to a good statistical language model is "unnatural" in some sense, and thus possibly suspicious. In this paper, we investigate this hypothesis. We consider a large corpus of bug fix commits (ca. 7,139), from 10 different Java projects, and focus on its language statistics, evaluating the naturalness of buggy code and the corresponding fixes. We find that code with bugs tends to be more entropic (i.e. unnatural), becoming less so as bugs are fixed. Ordering files for inspection by their average entropy yields cost-effectiveness scores comparable to popular defect prediction methods. At a finer granularity, focusing on highly entropic lines is similar in cost-effectiveness to some well-known static bug finders (PMD, FindBugs) and ordering warnings from these bug finders using an entropy measure improves the cost-effectiveness of inspecting code implicated in warnings. This suggests that entropy may be a valid, simple way to complement the effectiveness of PMD or FindBugs, and that search-based bug-fixing methods may benefit from using entropy both for fault-localization and searching for fixes.

References

[1]

PROMISE '14: Proceedings of the 10th International Conference on Predictive Models in Software Engineering, New York, NY, USA, 2014. ACM.

[2]

M. Allamanis, E. T. Barr, C. Bird, and C. Sutton. Learning natural coding conventions. In Proceedings of the 22^nd International Symposium on the Foundations of Software Engineering (FSE'14), 2014.

Digital Library

[3]

M. Allamanis and C. Sutton. Mining idioms from source code. In SIGSOFT FSE, pages 472--483, 2014.

Digital Library

[4]

E. Arisholm, L. C. Briand, and E. B. Johannessen. A systematic and comprehensive investigation of methods to build and evaluate fault prediction models. JSS, 83(1):2--17, 2010.

Digital Library

[5]

N. Ayewah, D. Hovemeyer, J. D. Morgenthaler, J. Penix, and W. Pugh. Using static analysis to find bugs. IEEE Software, 25(5):22--29, 2008.

Digital Library

[6]

V. R. Basili, L. C. Briand, and W. L. Melo. A validation of object-oriented design metrics as quality indicators. IEEE Trans. Software Eng., 22(10):751--761, 1996.

Digital Library

[7]

J. C. Campbell, A. Hindle, and J. N. Amaral. Syntax errors just aren't natural: improving error reporting with language models. In MSR, pages 252--261, 2014.

Digital Library

[8]

C. Catal and B. Diri. A systematic review of software fault prediction studies. Expert systems with applications, 36(4):7346--7354, 2009.

Digital Library

[9]

B. Chelf, D. Engler, and S. Hallem. How to write system-specific, static checkers in metal. In Proceedings of the 2002 ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering, PASTE '02, pages 51--60, New York, NY, USA, 2002. ACM.

Digital Library

[10]

T. Copeland. PMD applied. Centennial Books San Francisco, 2005.

[11]

M. D'Ambros, M. Lanza, and R. Robbes. An extensive comparison of bug prediction approaches. In Mining Software Repositories (MSR), 2010 7th IEEE Working Conference on, pages 31--41. IEEE, 2010.

[12]

M. Dias, A. Bacchelli, G. Gousios, D. Cassou, and S. Ducasse. Untangling fine-grained code changes. In Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering, 2015.

[13]

D. Engler, D. Y. Chen, S. Hallem, A. Chou, and B. Chelf. Bugs as deviant behavior: A general approach to inferring errors in systems code. In Proceedings of the Eighteenth ACM Symposium on Operating Systems Principles, SOSP '01, pages 57--72, New York, NY, USA, 2001. ACM.

Digital Library

[14]

FindBugs. http://findbugs.sourceforge.net/. Accessed 2015/03/10.

[15]

C. Franks, Z. Tu, P. Devanbu, and V. Hellendoorn. Cacheca: A cache language model based code suggestion tool. In ICSE Demonstration Track, 2015.

Digital Library

[16]

E. Giger, M. D'Ambros, M. Pinzger, and H. C. Gall. Method-level bug prediction. In Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement, ESEM '12, pages 171--180, 2012.

Digital Library

[17]

A. E. Hassan. Predicting faults using the complexity of code changes. In Proceedings of ICSE 2009, pages 78--88, 2009.

Digital Library

[18]

S. Heckman and L. Williams. On establishing a benchmark for evaluating static analysis alert prioritization and classification techniques. In Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement, pages 41--50. ACM, 2008.

Digital Library

[19]

S. S. Heckman. Adaptively ranking alerts generated from automated static analysis. Crossroads, 14(1):7, 2007.

Digital Library

[20]

K. Herzig and A. Zeller. Untangling changes. Unpublished manuscript, September, 2011.

[21]

K. Herzig and A. Zeller. The impact of tangled code changes. In Mining Software Repositories (MSR), 2013 10th IEEE Working Conference on, pages 121--130. IEEE, 2013.

Digital Library

[22]

A. Hindle, E. Barr, M. Gabel, Z. Su, and P. Devanbu. On the naturalness of software. In ICSE, pages 837--847, 2012.

Digital Library

[23]

B. Johnson, Y. Song, E. Murphy-Hill, and R. Bowdidge. Why don't software developers use static analysis tools to find bugs? In Software Engineering (ICSE), 2013 35th International Conference on, pages 672--681. IEEE, 2013.

Digital Library

[24]

S. Karaivanov, V. Raychev, and M. Vechev. Phrase-based statistical translation of programming languages. In SPLASH, Onward!, pages 173--184, 2014.

Digital Library

[25]

S. Katz. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech and Signal Processing, 35:400--401, 1987.

[26]

S. Kim and M. D. Ernst. Which warnings should i fix first? In Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, pages 45--54. ACM, 2007.

Digital Library

[27]

S. Kim, T. Zimmermann, J. Whitehead, and A. Zeller. Predicting faults from cached history. In Proceedings of ICSE 2007, pages 489--498. IEEE CS, 2007.

Digital Library

[28]

T. Kremenek, K. Ashcraft, J. Yang, and D. Engler. Correlation exploitation in error ranking. In ACM SIGSOFT Software Engineering Notes, volume 29, pages 83--93. ACM, 2004.

Digital Library

[29]

T. Kremenek and D. Engler. Z-ranking: Using statistical analysis to counter the impact of static analysis approximations. In Static Analysis, pages 295--315. Springer, 2003.

Digital Library

[30]

T. Kremenek, P. Twohey, G. Back, A. Ng, and D. Engler. From uncertainty to belief: Inferring the specification within. In Proceedings of the 7th symposium on Operating systems design and implementation, pages 161--176. USENIX Association, 2006.

Digital Library

[31]

F. R. Kschischang, B. J. Frey, and H.-A. Loeliger. Factor graphs and the sum-product algorithm. IEEE Transactions on Information Theory, 47(2):498--519, 2001.

Digital Library

[32]

Z. Li and Y. Zhou. Pr-miner: Automatically extracting implicit programming rules and detecting violations in large software code. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, ESEC/FSE-13, pages 306--315, New York, NY, USA, 2005. ACM.

Digital Library

[33]

A. Mockus and L. G. Votta. Identifying reasons for software changes using historic databases. In ICSM, pages 120--130, 2000.

Digital Library

[34]

R. Moser, W. Pedrycz, and G. Succi. A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction. In Proceedings of ICSE 2008, pages 181--190, 2008.

Digital Library

[35]

N. Nagappan and T. Ball. Use of relative code churn measures to predict system defect density. In Software Engineering, 2005. ICSE 2005. Proceedings. 27th International Conference on, pages 284--292. IEEE, 2005.

Digital Library

[36]

N. Nagappan, T. Ball, and A. Zeller. Mining metrics to predict component failures. In Proceedings of the 28th international conference on Software engineering, pages 452--461. ACM, 2006.

Digital Library

[37]

A. T. Nguyen, H. A. Nguyen, T. T. Nguyen, and T. N. Nguyen. Statistical learning of api mappings for language migration. In ICSE Companion, pages 618--619, 2014.

Digital Library

[38]

A. T. Nguyen, T. T. Nguyen, and T. N. Nguyen. Lexical statistical machine translation for language migration. In SIGSOFT FSE, pages 651--654, 2013.

Digital Library

[39]

A. T. Nguyen, T. T. Nguyen, and T. N. Nguyen. Migrating code with statistical machine translation. In ICSE Companion, pages 544--547, 2014.

Digital Library

[40]

C. Parnin and A. Orso. Are automated debugging techniques actually helping programmers? In Proceedings of the 2011 International Symposium on Software Testing and Analysis, pages 199--209. ACM, 2011.

Digital Library

[41]

D. Posnett, V. Filkov, and P. Devanbu. Ecological inference in empirical software engineering. In Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, pages 362--371. IEEE Computer Society, 2011.

Digital Library

[42]

F. Rahman and P. Devanbu. How, and why, process metrics are better. In Proceedings of ICSE, pages 432--441, 2013.

Digital Library

[43]

F. Rahman, S. Khatri, E. T. Barr, and P. T. Devanbu. Comparing static bug finders and statistical prediction. In ICSE, pages 424--434, 2014.

Digital Library

[44]

F. Rahman, D. Posnett, and P. Devanbu. Recalling the imprecision of cross-project defect prediction. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, page 61. ACM, 2012.

Digital Library

[45]

B. Ray, M. Kim, S. Person, and N. Rungta. Detecting and characterizing semantic inconsistencies in ported code. In ASE, pages 367--377, 2013.

Digital Library

[46]

B. Ray, D. Posnett, V. Filkov, and P. Devanbu. A large scale study of programming languages and code quality in github. In SIGSOFT FSE, 2014.

Digital Library

[47]

V. Raychev, M. Vechev, and A. Krause. Predicting program properties from "big code". In POPL, pages 111--124, 2015.

Digital Library

[48]

V. Raychev, M. Vechev, and E. Yahav. Code completion with statistical language models. In PLDI, pages 419--428, 2014.

Digital Library

[49]

J. R. Ruthruff, J. Penix, J. D. Morgenthaler, S. Elbaum, and G. Rothermel. Predicting accurate and actionable static analysis warnings: an experimental approach. In Proceedings of the 30th international conference on Software engineering, pages 341--350. ACM, 2008.

Digital Library

[50]

J. Śliwerski, T. Zimmermann, and A. Zeller. When do changes induce fixes? In MSR, pages 1--5, 2005.

Digital Library

[51]

S. Thummalapenta and T. Xie. Alattin: Mining alternative patterns for detecting neglected conditions. In Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering, pages 283--294. IEEE Computer Society, 2009.

Digital Library

[52]

F. Thung, D. Lo, L. Jiang, F. Rahman, P. T. Devanbu, et al. To what extent could we detect field defects? an empirical study of false negatives in static bug finding tools. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, pages 50--59. ACM, 2012.

Digital Library

[53]

Z. Tu, Z. Su, and P. Devanbu. On the localness of software. In SIGSOFT FSE, pages 269--280, 2014.

Digital Library

[54]

A. Wasylkowski, A. Zeller, and C. Lindig. Detecting object usage anomalies. In Proceedings of the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, pages 35--44. ACM, 2007.

Digital Library

[55]

I. H. Witten and E. Frank. Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann, 2005.

Digital Library

[56]

H. Zhang and S. Cheung. A cost-effectiveness criterion for applying software defect prediction models. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pages 643--646. ACM, 2013.

Digital Library

[57]

T. Zimmermann, R. Premraj, and A. Zeller. Predicting defects for eclipse. In Predictor Models in Software Engineering, 2007. PROMISE'07: ICSE Workshops 2007. International Workshop on, pages 9--9. IEEE, 2007.

Digital Library

Cited By

Yang CChen JJiang JHuang Y(2024)Dependency-Aware Code NaturalnessProceedings of the ACM on Programming Languages10.1145/36897948:OOPSLA2(2355-2377)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689794
Xia CZhang LChristakis MPradel M(2024)Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPTProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680323(819-831)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680323
Chen XXu FHuang YZhang NZheng Z(2024)JIT-Smart: A Multi-task Learning Framework for Just-in-Time Defect Prediction and LocalizationProceedings of the ACM on Software Engineering10.1145/36437271:FSE(1-23)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643727
Show More Cited By

Recommendations

CBCD: cloned buggy code detector
ICSE '12: Proceedings of the 34th International Conference on Software Engineering

Developers often copy, or clone, code in order to reuse or modify functionality. When they do so, they also clone any bugs in the original code. Or, different developers may independently make the same mistake. As one example of a bug, multiple ...
Predicting Buggy Code Clones through Machine Learning
CASCON '22: Proceedings of the 32nd Annual International Conference on Computer Science and Software Engineering
Code clones (similar code fragments in a code-base} often have negative impacts on the maintenance and evolution of software systems. According to the existing studies, code clones may contain bugs or inconsistencies that can cause an increased ...
Scalable and systematic detection of buggy inconsistencies in source code
OOPSLA '10

Software developers often duplicate source code to replicate functionality. This practice can hinder the maintenance of a software project: bugs may arise when two identical code segments are edited inconsistently. This paper presents DejaVu, a highly ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '16: Proceedings of the 38th International Conference on Software Engineering

May 2016

1235 pages

ISBN:9781450339001

DOI:10.1145/2884781

General Chair:
Laura Dillon
Michigan State University
,
Program Chairs:
Willem Visser
Stellenbosch University, South Africa
,
Laurie Williams
North Carolina State University

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\TCSE: TC on Software Engineering
IEEE-CS\DATC: IEEE Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

1414172

Conference

ICSE '16

Sponsor:

ACM
SIGSOFT
IEEE-CS\TCSE
IEEE-CS\DATC

ICSE '16: 38th International Conference on Software Engineering

May 14 - 22, 2016

Texas, Austin

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

162
Total Citations
View Citations
1,468
Total Downloads

Downloads (Last 12 months)141
Downloads (Last 6 weeks)7

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang CChen JJiang JHuang Y(2024)Dependency-Aware Code NaturalnessProceedings of the ACM on Programming Languages10.1145/36897948:OOPSLA2(2355-2377)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689794
Xia CZhang LChristakis MPradel M(2024)Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPTProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680323(819-831)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680323
Chen XXu FHuang YZhang NZheng Z(2024)JIT-Smart: A Multi-task Learning Framework for Just-in-Time Defect Prediction and LocalizationProceedings of the ACM on Software Engineering10.1145/36437271:FSE(1-23)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643727
Chowdhury SUddin GHemmati HHolmes R(2024)Method-level Bug Prediction: Problems and PromisesACM Transactions on Software Engineering and Methodology10.1145/364033133:4(1-31)Online publication date: 13-Jan-2024
https://dl.acm.org/doi/10.1145/3640331
Pârțachi PSugiyama MRoychoudhury APaiva AAbreu RStorey M(2024)Bringing Structure to Naturalness: On the Naturalness of ASTsProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3643517(378-379)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639478.3643517
Guo HChen YChen XHuang YZheng Z(2024)Smart Contract Code Repair Recommendation based on Reinforcement Learning and Multi-metric OptimizationACM Transactions on Software Engineering and Methodology10.1145/363722933:4(1-31)Online publication date: 18-Apr-2024
https://dl.acm.org/doi/10.1145/3637229
Xu ZQiang SSong DZhou MWan HZhao XLuo PZhang HRoychoudhury APaiva AAbreu RStorey M(2024)DSFM: Enhancing Functional Code Clone Detection with Deep Subtree InteractionsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639215(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639215
Ding YSteenhoek BPei KKaiser GLe WRay BRoychoudhury APaiva AAbreu RStorey M(2024)TRACED: Execution-aware Pre-training for Source CodeProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3608140(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3608140
Nader Palacio DVelasco ACooper NRodriguez AMoran KPoshyvanyk D(2024)Toward a Theory of Causation for Interpreting Neural Code ModelsIEEE Transactions on Software Engineering10.1109/TSE.2024.337994350:5(1215-1243)Online publication date: May-2024
https://doi.org/10.1109/TSE.2024.3379943
Karmakar ARobbes R(2024)INSPECT: Intrinsic and Systematic Probing Evaluation for Code TransformersIEEE Transactions on Software Engineering10.1109/TSE.2023.334162450:2(220-238)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1109/TSE.2023.3341624
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents