Article

Using benchmarking to advance research: a challenge to software engineering

Authors:

Susan Elliott Sim,

Steve Easterbrook,

Richard C. HoltAuthors Info & Claims

ICSE '03: Proceedings of the 25th International Conference on Software Engineering

Pages 74 - 83

Published: 03 May 2003 Publication History

Publisher Site Get Access

Abstract

Benchmarks have been used in computer science to compare the performance of computer systems, information retrieval algorithms, databases, and many other technologies. The creation and widespread use of a benchmark within a research area is frequently accompanied by rapid technical progress and community building. These observations have led us to formulate a theory of benchmarking within scientific disciplines. Based on this theory, we challenge software engineering research to become more scientific and cohesive by working as a community to define benchmarks. In support of this challenge, we present a case study of the reverse engineering community, where we have successfully used benchmarks to advance the state of research.

References

[1]

M. N. Armstrong and C. Trudeau, "Evaluating Architectural Extractors," presented at Working Conference on Reverse Engineering, Honolulu, HI, pp. 30--39, October 12--14, 1998.]]

Digital Library

[2]

Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach, "The Goal Question Metric Approach," in Encyclopedia of Software Engineering, Two Volume Set, Gianluigi Caldiera and Dieter H. Rombach, Eds. New York City: John Wiley and Sons, Inc., pp. 528--532, 1994.]]

[3]

Berndt Bellay and Harald Gall, "An Evaluation of Reverse Engineering Tool Capabilities," Software Maintenance: Research and Practice, vol. 10, pp. 305--331, 1998.]]

Digital Library

[4]

Ivan T. Bowman, Michael W. Godfrey, and Ric Holt, "Connecting Architecture Reconstruction Frameworks," Journal of Information and Software Technology, vol. 42, no. 2, pp. 93--104, 1999.]]

[5]

Serge Demeyer, Tom Mens, and Michel Wermelinger, "Towards a Software Evolution Benchmark," presented at Proceedings of the International Workshop on Principles of Software Evolution, Vienna, Austria, pp. 172--175, 10--11 September 2002.]]

Digital Library

[6]

Martin S. Feather, Stephen Fickas, Anthony Finkelstein, and Axel van Lamsweerde, "Requirements and Specification Exemplars," Automated Software Engineering, vol. 4, pp. 419--438, 1997.]]

Digital Library

[7]

Rudolf Ferenc, Susan Elliott Sim, Richard C. Holt, Rainer Koschke, and Tibor Gyimóthy, "Towards a Standard Schema for C/C++," presented at Eighth Working Conference on Reverse Engineering, Stuttgart, Germany, pp. 49--58, 2--5 October 2001.]]

Digital Library

[8]

Jim Gray, "The Benchmark Handbook: For Database and Transaction Processing Systems," San Mateo, CA: Morgan Kaufman Publishers, Inc, 1991.]]

Digital Library

[9]

John L. Henning, "SPEC CPU2000: Measuring CPU Performance in the New Millennium," IEEE Computer, pp. 28--35, July, 2000.]]

Digital Library

[10]

Richard C. Holt, Andreas Winter, and Andy Schürr, "GXL: Toward a Standard Exchange Format.," presented at Seventh Working Conference on Reverse Engineering, Brisbane, Queensland, Australia, pp. 162--171, 23--25 November 2000.]]

Digital Library

[11]

Barbara Ann Kitchenham, "Evaluating software engineering methods and tools. Parts 1 to 12.," ACM SIGSOFT Software Engineering Notes, vol. 21--23, 1996--1998.]]

Digital Library

[12]

Thomas S. Kuhn, The Structure of Scientific Revolutions, Third Edition. Chicago: The University of Chicago Press, 1996.]]

[13]

William E. McUmber and Betty H. C. Cheng, "A Generic Framework for Formalizing UML," presented at Proceedings of the Twenty-third International Conference on Software Engineering, Toronto, Canada, pp. 433--442, 12--19 May 2001.]]

Digital Library

[14]

Gail C. Murphy, David Notkin, William G. Griswold, and Erica S. Lan, "An Empirical Study of Static Call Graph Extractors," ACM Transactions on Software Engineering and Methodology, vol. 7, no. 2, pp. 158--191, 1998.]]

Digital Library

[15]

Shari Lawrence Pfleeger, "Experimental Design and Analysis in Software Engineering, Parts 1--6," ACM SIGSOFT Software Engineering Notes, vol. 19--20, 1994--1995.]]

Digital Library

[16]

M. Plath and M. Ryan, "SFI: A Feature Integration Tool," in Advances in Computer Science, R. Berghammer and Y. Lakhnech, Eds. Heidelberg, Germany: Springer Verlag, pp. 201--216, 1999.]]

[17]

Susan Elliott Sim, Richard C. Holt, and Steve Easterbrook, "On Using a Benchmark to Evaluate C++ Extractors," presented at Tenth International Workshop on Program Comprehension, Paris, France, pp. 114--123,.]]

Digital Library

[18]

Susan Elliott Sim and Margaret-Anne D. Storey, "A Structured Demonstration of Program Comprehension Tools," presented at Seventh Working Conference on Reverse Engineering, Brisbane, Queensland, Australia, pp. 184--193, 23--25 November 2000.]]

Digital Library

[19]

Susan Elliott Sim, Margaret-Anne D. Storey, and Andreas Winter, "A Structured Demonstration of Five Program Comprehension Tools: Lessons Learnt," presented at Seventh Working Conference on Reverse Engineering, Brisbane, Queensland, Australia, pp. 210--212, 23--25 November 2000.]]

Digital Library

[20]

Tirumale Sreemani and Joanne M. Atlee, "Feasibility of Model Checking Software Requirements: A Case Study," presented at Proceedings of COMPAS'96, Gaithersburg, Maryland, pp. 77--88, 17--21 June 1996.]]

[21]

Walter F. Tichy, "Should Computer Scientists Experiment More?" IEEE Computer, pp. 32--40, May, 1998.]]

Digital Library

[22]

Ellen M. Voorhees and Donna Harman, "Overview of the Eighth Text REtrieval Conference (TREC-8)," presented at Text REtrieval Conference (TREC-8), Gaithersberg, Maryland, pp. 1--24, November 17--19, 2000.]]

Cited By

Strüber DMukelabai MKrüger JFischer SLinsbauer LMartinez JBerger T(2019)Facing the TruthProceedings of the 23rd International Systems and Software Product Line Conference - Volume A10.1145/3336294.3336302(177-188)Online publication date: 9-Sep-2019
https://dl.acm.org/doi/10.1145/3336294.3336302
Rueda UKifetew FDevroey XGorla ARojas J(2019)Towards automated test case generation maturityProceedings of the 12th International Workshop on Search-Based Software Testing10.1109/SBST.2019.00011(9-10)Online publication date: 27-May-2019
https://dl.acm.org/doi/10.1109/SBST.2019.00011
Babur ÖStephan MChechik MStrüber DVarró D(2019)MoCoPProceedings of the 11th International Workshop on Modelling in Software Engineerings10.1109/MiSE.2019.00019(78-81)Online publication date: 26-May-2019
https://dl.acm.org/doi/10.1109/MiSE.2019.00019
Show More Cited By

Index Terms

Using benchmarking to advance research: a challenge to software engineering

Recommendations

Application-specific benchmarking
Benchmarking as Empirical Standard in Software Engineering Research
EASE '21: Proceedings of the 25th International Conference on Evaluation and Assessment in Software Engineering

In empirical software engineering, benchmarks can be used for comparing different methods, techniques and tools. However, the recent ACM SIGSOFT Empirical Standards for Software Engineering Research do not include an explicit checklist for ...
Benchmarking data warehouses

Database benchmarks can either help users in comparing the performances of different systems, or help engineers in testing the effect of various design choices. In the field of data warehouses, the Transaction Processing Performance Council's standard ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '03: Proceedings of the 25th International Conference on Software Engineering

May 2003

841 pages

ISBN:076951877X

General Chair:
Lori Clarke
University of Massachusetts
,
Program Chairs:
Laurie Dillon
Michigan State University
,
Walter Tichy
University of Karlsruhe, Germany

Copyright © Copyright (c) 2003 Institute of Electrical and Electronics Engineers, Inc. All rights reserved.

Sponsors

IEEE Computer Society Technical Council on Software Engineering
SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

IEEE Computer Society

United States

Publication History

Published: 03 May 2003

Check for updates

Qualifiers

Article

Conference

ICSE03

Sponsor:

SIGSOFT

ICSE03: 25th International Conference on Software Engineering

May 3 - 10, 2003

Oregon, Portland

Acceptance Rates

ICSE '03 Paper Acceptance Rate 42 of 324 submissions, 13%;

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

43
Total Citations
View Citations
1,774
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Reflects downloads up to 24 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Strüber DMukelabai MKrüger JFischer SLinsbauer LMartinez JBerger T(2019)Facing the TruthProceedings of the 23rd International Systems and Software Product Line Conference - Volume A10.1145/3336294.3336302(177-188)Online publication date: 9-Sep-2019
https://dl.acm.org/doi/10.1145/3336294.3336302
Rueda UKifetew FDevroey XGorla ARojas J(2019)Towards automated test case generation maturityProceedings of the 12th International Workshop on Search-Based Software Testing10.1109/SBST.2019.00011(9-10)Online publication date: 27-May-2019
https://dl.acm.org/doi/10.1109/SBST.2019.00011
Babur ÖStephan MChechik MStrüber DVarró D(2019)MoCoPProceedings of the 11th International Workshop on Modelling in Software Engineerings10.1109/MiSE.2019.00019(78-81)Online publication date: 26-May-2019
https://dl.acm.org/doi/10.1109/MiSE.2019.00019
Vogel TAndersson JWeyns D(2018)mRUBiSProceedings of the 13th International Conference on Software Engineering for Adaptive and Self-Managing Systems10.1145/3194133.3194161(101-107)Online publication date: 28-May-2018
https://dl.acm.org/doi/10.1145/3194133.3194161
Duretec KRauber ABecker CMcDonald RWorby NJatowt AMarshall CMilligan I(2017)A text extraction software benchmark based on a synthesized datasetProceedings of the 17th ACM/IEEE Joint Conference on Digital Libraries10.5555/3200334.3200347(109-118)Online publication date: 19-Jun-2017
https://dl.acm.org/doi/10.5555/3200334.3200347
Panichella AMolina UGaleotti JPetke J(2017)Java unit testing tool competitionProceedings of the 10th International Workshop on Search-Based Software Testing10.5555/3105427.3105434(32-38)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.5555/3105427.3105434
Gerasimou SCalinescu RShevtsov SWeyns DGarlan DNuseibeh B(2017)UNDERSEAProceedings of the 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems10.1109/SEAMS.2017.19(83-89)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.1109/SEAMS.2017.19
Martin-Rodilla PGonzalez-Perez C(2016)Understanding user behavior in textual analysisProceedings of the Fourth International Conference on Technological Ecosystems for Enhancing Multiculturality10.1145/3012430.3012528(269-276)Online publication date: 2-Nov-2016
https://dl.acm.org/doi/10.1145/3012430.3012528
Simon Sheard JD'Souza DKlemperer PPorter LSorva JStegeman MZingaro DClear ACuadros-Vargas ECarter JTupac Y(2016)Benchmarking Introductory Programming ExamsProceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education10.1145/2899415.2899473(154-159)Online publication date: 11-Jul-2016
https://dl.acm.org/doi/10.1145/2899415.2899473
Martinez JZiadi TPapadakis MBissyandé TKlein JTraon Y(2016)Feature Location Benchmark for Software Families Using Eclipse Community ReleasesProceedings of the 15th International Conference on Software Reuse: Bridging with Social-Awareness - Volume 967910.1007/978-3-319-35122-3_18(267-283)Online publication date: 5-Jun-2016
https://dl.acm.org/doi/10.1007/978-3-319-35122-3_18
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents