research-article

Visual web test repair

Authors:

Rahulkrishna Yandrapally,

Ali MesbahAuthors Info & Claims

ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 503 - 514

https://doi.org/10.1145/3236024.3236063

Published: 26 October 2018 Publication History

Abstract

Web tests are prone to break frequently as the application under test evolves, causing much maintenance effort in practice. To detect the root causes of a test breakage, developers typically inspect the test's interactions with the application through the GUI. Existing automated test repair techniques focus instead on the code and entirely ignore visual aspects of the application. We propose a test repair technique that is informed by a visual analysis of the application. Our approach captures relevant visual information from tests execution and analyzes them through a fast image processing pipeline to visually validate test cases as they re-executed for regression purposes. Then, it reports the occurrences of breakages and potential fixes to the testers. Our approach is also equipped with a local crawling mechanism to handle non-trivial breakage scenarios such as the ones that require to repair the test's workflow. We implemented our approach in a tool called Vista. Our empirical evaluation on 2,672 test cases spanning 86 releases of four web applications shows that Vista is able to repair, on average, 81% of the breakages, a 41% increment with respect to existing techniques.

References

[1]

E. Alégroth, M. Nass, and H. H. Olsson. 2013. JAutomate: A Tool for System- and Acceptance-test Automation. In Proceedings of IEEE 6th International Conference on Software Testing, Verification and Validation (ICST ’13). 439–446.

Digital Library

[2]

Applitools 2018. Applitools. Visual app testing and monitoring. https://applitools. com/. (2018). Accessed: 2017-08-01.

[3]

Kartik Bajaj, Karthik Pattabiraman, and Ali Mesbah. 2015. Synthesizing Web Element Locators. In Proceedings of 30th IEEE/ACM International Conference on Automated Software Engineering (ASE ’15). IEEE Computer Society, 331–341.

Digital Library

[4]

Mohammad Bajammal, Davood Mazinanian, and Ali Mesbah. 2018. Generating Reusable Web Components from Mockups. In Proceedings of 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE ’18). IEEE Computer Society.

Digital Library

[5]

Mohammad Bajammal and Ali Mesbah. 2018. Web Canvas Testing through Visual Inference. In Proceedings of 11th International Conference on Software Testing, Verification and Validation (ICST ’18). IEEE Computer Society, 193–203.

[6]

Robert V. Binder. 1996. Testing object-oriented software: a survey. Software Testing, Verification and Reliability 6, 3-4 (1996), 125–252.

[7]

Kai Briechle and Uwe D Hanebeck. 2001. Template matching using fast normalized cross correlation. In Optical Pattern Recognition XII, Vol. 4387. International Society for Optics and Photonics, 95–103.

[8]

Roberto Brunelli. 2009. Template Matching Techniques in Computer Vision: Theory and Practice. Wiley Publishing.

Digital Library

[9]

Gilles Burel and Dominique Carel. 1994. Detection and localization of faces on digital images. Pattern Recognition Letters 15, 10 (1994), 963 – 967.

Digital Library

[10]

Tsung-Hsiang Chang, Tom Yeh, and Robert C. Miller. 2010. GUI testing using computer vision. In Proceedings of 28th ACM Conference on Human Factors in Computing Systems (CHI ’10). ACM, 1535–1544.

Digital Library

[11]

Shauvik Roy Choudhary, Dan Zhao, Husayn Versee, and Alessandro Orso. 2011. WATER: Web Application TEst Repair. In Proceedings of 1st International Workshop on End-to-End Test Script Engineering (ETSE ’11). ACM, 24–29.

Digital Library

[12]

Laurent Christophe, Reinout Stevens, Coen De Roover, and Wolfgang De Meuter. 2014. Prevalence and Maintenance of Automated Functional Tests for Web Applications. In Proceedings of 30th International Conference on Software Maintenance and Evolution (ICSME ’14). IEEE, 141–150.

Digital Library

[13]

Cloc 2018. Counts blank lines, comment lines, and physical lines of source code in many programming languages. https://github.com/AlDanial/cloc. (2018).

[14]

Brett Daniel, Danny Dig, Tihomir Gvero, Vilas Jagannath, Johnston Jiaa, Damion Mitchell, Jurand Nogiec, Shin Hwei Tan, and Darko Marinov. 2011. ReAssert: A Tool for Repairing Broken Unit Tests. In Proceedings of 33rd International Conference on Software Engineering (ICSE ’11). ACM, 1010–1012.

Digital Library

[15]

Brett Daniel, Tihomir Gvero, and Darko Marinov. 2010. On Test Repair Using Symbolic Execution. In Proceedings of 19th International Symposium on Software Testing and Analysis (ISSTA ’10). ACM, 207–218.

Digital Library

[16]

Brett Daniel, Vilas Jagannath, Danny Dig, and Darko Marinov. 2009. ReAssert: Suggesting Repairs for Broken Unit Tests. In Proceedings of 2009 IEEE/ACM International Conference on Automated Software Engineering (ASE ’09). IEEE Computer Society, 433–444.

Digital Library

[17]

Brett Daniel, Qingzhou Luo, Mehdi Mirzaaghaei, Danny Dig, Darko Marinov, and Mauro Pezzè. 2011. Automated GUI Refactoring and Test Script Repair. In Proceedings of First International Workshop on End-to-End Test Script Engineering (ETSE ’11). ACM, 38–41.

Digital Library

[18]

Yang Feng, James A. Jones, Zhenyu Chen, and Chunrong Fang. 2016. Multiobjective Test Report Prioritization Using Image Understanding. In Proceedings of 31st IEEE/ACM International Conference on Automated Software Engineering (ASE ’16). ACM, 202–213.

Digital Library

[19]

Mark Fewster and Dorothy Graham. 1999. Software Test Automation: Effective Use of Test Execution Tools. Addison-Wesley Longman Publishing Co., Inc.

Digital Library

[20]

Zebao Gao, Zhenyu Chen, Yunxiao Zou, and Atif M. Memon. 2016. SITAR: GUI Test Script Repair. IEEE Transactions on Software Engineering 42, 2 (feb 2016), 170–186.

Digital Library

[21]

Z. Gao, C. Fang, and A. M. Memon. 2015. Pushing the limits on automation in GUI regression testing. In Proceedings of IEEE 26th International Symposium on Software Reliability Engineering (ISSRE ’15). 565–575.

Digital Library

[22]

Mark Grechanik, Qing Xie, and Chen Fu. 2009. Maintaining and evolving GUIdirected test scripts. In Proceedings of 31st International Conference on Software Engineering (ICSE ’09). IEEE Computer Society, 408–418.

Digital Library

[23]

Mouna Hammoudi, Gregg Rothermel, and Andrea Stocco. 2016. WATERFALL: An Incremental Approach for Repairing Record-replay Tests of Web Applications. In Proceedings of 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE ’16). ACM, 751–762.

Digital Library

[24]

Mouna Hammoudi, Gregg Rothermel, and Paolo Tonella. 2016. Why do Record/Replay Tests of Web Applications Break?. In Proceedings of 9th International Conference on Software Testing, Verification and Validation (ICST ’16). IEEE, 180–190.

[25]

Si Huang, Myra B. Cohen, and Atif M. Memon. 2010. Repairing GUI Test Suites Using a Genetic Algorithm. In Proceedings of 3rd International Conference on Software Testing, Verification and Validation (ICST ’10). IEEE Computer Society, 245–254.

Digital Library

[26]

Ayman Issa, Jonathan Sillito, and Vahid Garousi. 2012. Visual Testing of Graphical User Interfaces: An Exploratory Study Towards Systematic Definitions and Approaches. In Proceedings of IEEE 14th International Symposium on Web Systems Evolution (WSE ’12). IEEE Computer Society, 11–15.

Digital Library

[27]

M. Furkan Kıraç, Barış Aktemur, and Hasan Sözer. 2018. VISOR: A fast image processing pipeline with scaling and translation invariance for test oracle automation of visual output systems. Journal of Systems and Software 136 (2018), 266 – 277.

Digital Library

[28]

V. Lelli, A. Blouin, and B. Baudry. 2015. Classifying and Qualifying GUI Defects. In Proceedings of 8th IEEE International Conference on Software Testing, Verification and Validation (ICST ’15). 1–10.

[29]

Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2014. Visual vs. DOM-based Web Locators: An Empirical Study. In Proceedings of 14th International Conference on Web Engineering (ICWE ’14), Vol. 8541. Springer, 322–340.

[30]

Maurizio Leotta, Diego Clerissi, Filippo Ricca, and Paolo Tonella. 2016. Approaches and Tools for Automated End-to-End Web Testing. Advances in Computers 101 (2016), 193–237.

[31]

Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2014. Reducing Web Test Cases Aging by means of Robust XPath Locators. In Proceedings of 25th International Symposium on Software Reliability Engineering Workshops (ISSREW ’14). IEEE Computer Society, 449–454.

Digital Library

[32]

Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2015. Automated Migration of DOM-based to Visual Web Tests. In Proceedings of 30th Symposium on Applied Computing (SAC ’15). ACM, 775–782.

Digital Library

[33]

Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2015. Using Multi-Locators to Increase the Robustness of Web Test Cases. In Proceedings of 8th IEEE International Conference on Software Testing, Verification and Validation (ICST ’15). IEEE, 1–10.

[34]

Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2016. ROBULA+: An Algorithm for Generating Robust XPath Locators for Web Testing. Journal of Software: Evolution and Process (2016), 28:177–204.

Digital Library

[35]

Maurizio Leotta, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2018. PESTO: Automated migration of DOM-based Web tests towards the visual approach. Software Testing, Verification And Reliability 28, 4 (2018).

[36]

D. G. Lowe. 1999. Object recognition from local scale-invariant features. In Proceedings of 7th IEEE International Conference on Computer Vision, Vol. 2. 1150– 1157.

Digital Library

[37]

David G. Lowe. 2004. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60, 2 (01 Nov 2004), 91–110.

Digital Library

[38]

S. Mahajan and W. G. J. Halfond. 2015. Detection and Localization of HTML Presentation Failures Using Computer Vision-Based Techniques. In Proceedings of 8th IEEE International Conference on Software Testing, Verification and Validation (ICST ’15). 1–10.

[39]

Tomasz Malisiewicz, Abhinav Gupta, and Alexei A. Efros. 2011. Ensemble of exemplar-SVMs for Object Detection and Beyond. In Proceedings of 2011 International Conference on Computer Vision (ICCV ’11). IEEE Computer Society, 89–96.

Digital Library

[40]

Atif M. Memon. 2008. Automatically Repairing Event Sequence-based GUI Test Suites for Regression Testing. ACM Transactions on Software Engineering and Methodologies 18, 2, Article 4 (nov 2008), 36 pages.

Digital Library

[41]

Ali Mesbah, Arie van Deursen, and Stefan Lenselink. 2012. Crawling Ajax-based Web Applications through Dynamic Analysis of User Interface State Changes. ACM Transactions on the Web 6, 1 (2012), 3:1–3:30.

Digital Library

[42]

Ali Mesbah, Arie van Deursen, and Danny Roest. 2012. Invariant-based Automatic Testing of Modern Web Applications. IEEE Transactions on Software Engineering 38, 1 (2012), 35–53.

Digital Library

[43]

Bao N. Nguyen, Bryan Robbins, Ishan Banerjee, and Atif Memon. 2014. GUITAR: an innovative tool for automated testing of GUI-driven software. Automated Software Engineering 21, 1 (2014), 65–105.

Digital Library

[44]

OpenCV 2018. Open Source Computer Vision Library. https://opencv.org. (2018).

[45]

Rudolf Ramler and Klaus Wolfmaier. 2006. Economic Perspectives in Test Automation: Balancing Automated and Manual Testing with Opportunity Cost. In Proceedings of 1st International Workshop on Automation of Software Test (AST ’06). ACM, 85–91.

Digital Library

[46]

R. Ramler and T. Ziebermayr. 2017. What You See Is What You Test - Augmenting Software Testing with Computer Vision. In Proceedings of 10th IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW 2017). 398–400.

[47]

Edward Rosten and Tom Drummond. 2005. Fusing points and lines for high performance tracking. In IEEE International Conference on Computer Vision, Vol. 2. 1508–1511.

Digital Library

[48]

Edward Rosten, Reid Porter, and Tom Drummond. 2010. FASTER and better: A machine learning approach to corner detection. IEEE Transaction on Pattern Analysis and Machine Intelligence 32 (2010), 105–119.

Digital Library

[49]

Shauvik Roy Choudhary, Mukul R. Prasad, and Alessandro Orso. 2013. X-PERT: Accurate Identification of Cross-browser Issues in Web Applications. In Proceedings of 2013 International Conference on Software Engineering (ICSE ’13). IEEE Press, 702–711. ESEC/FSE ’18, November 4–9, 2018, Lake Buena Vista, FL, USA A. Stocco et al.

Digital Library

[50]

Shauvik Roy Choudhary, Husayn Versee, and Alessandro Orso. 2010. WEBDIFF: Automated Identification of Cross-browser Issues in Web Applications. In Proceedings of 2010 IEEE International Conference on Software Maintenance (ICSM ’10). IEEE Computer Society, 1–10.

Digital Library

[51]

Andrea Stocco, Maurizio Leotta, Filippo Ricca, and Paolo Tonella. 2014. PESTO: A Tool for Migrating DOM-based to Visual Web Tests. In Proceedings of 14th International Working Conference on Source Code Analysis and Manipulation (SCAM ’14). IEEE Computer Society, 65–70.

Digital Library

[52]

Andrea Stocco, Maurizio Leotta, Filippo Ricca, and Paolo Tonella. 2015. Why Creating Web Page Objects Manually If It Can Be Done Automatically?. In Proceedings of 10th IEEE/ACM International Workshop on Automation of Software Test (AST ’15). IEEE/ACM, 70–74.

Digital Library

[53]

Andrea Stocco, Maurizio Leotta, Filippo Ricca, and Paolo Tonella. 2016. Automatic Page Object Generation with APOGEN. In Proceedings of 16th International Conference on Web Engineering (ICWE ’16 - Demo Track). Springer, 533–537.

[54]

Andrea Stocco, Maurizio Leotta, Filippo Ricca, and Paolo Tonella. 2016. Clustering-Aided Page Object Generation for Web Testing. In Proceedings of 16th International Conference on Web Engineering (ICWE ’16). Springer, 132–151.

[55]

Andrea Stocco, Maurizio Leotta, Filippo Ricca, and Paolo Tonella. 2017. APOGEN: Automatic Page Object Generator for Web Testing. Software Quality Journal 25, 3 (Sept. 2017), 1007–1039.

Digital Library

[56]

Andrea Stocco, Rahulkrishna Yandrapally, and Ali Mesbah. 2018. Web Test Repair Using Computer Vision. In Proceedings of 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2018 - Demonstration Track). ACM.

Digital Library

[57]

Paolo Tonella, Filippo Ricca, and Alessandro Marchetto. 2014. Recent Advances in Web Testing. Advances in Computers 93 (2014), 1–51.

[58]

Vista 2018. Web Test Repair using Computer Vision. https://github.com/saltlab/ vista. (2018).

[59]

Rahulkrishna Yandrapally, Suresh Thummalapenta, Saurabh Sinha, and Satish Chandra. 2014. Robust Test Automation Using Contextual Clues. In Proceedings of 2014 International Symposium on Software Testing and Analysis (ISSTA ’14). ACM, 304–314.

Digital Library

Cited By

Ji RQian ZLi ZZhu XPei YPan MZhang T(2024)Repairing Obsolete GUI Test Scripts for Android Applications with Exploration and BacktrackingProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671377(51-60)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3671377
Di Meglio SStarace L(2024)Towards Predicting Fragility in End-to-End Web TestsProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661179(387-392)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661179
Tufano RMastropaolo APepe FDabic ODi Penta MBavota GSpinellis DConstantinou EBacchelli A(2024)Unveiling ChatGPT's Usage in Open Source Projects: A Mining-based StudyProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644918(571-583)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644918
Show More Cited By

Index Terms

Visual web test repair
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Diversity-based web test generation
ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Existing web test generators derive test paths from a navigational model of the web application, completed with either manually or randomly generated input values. However, manual test data selection is costly, while random generation often results in ...
Vista: web test repair using computer vision
ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Repairing broken web element locators represents the major main- tenance cost of web test cases. To detect possible repairs, testers typically inspect the tests’ interactions with the application under test through the GUI. Existing automated test ...
WATER: Web Application TEst Repair
ETSE '11: Proceedings of the First International Workshop on End-to-End Test Script Engineering

Web applications tend to evolve quickly, resulting in errors and failures in test automation scripts that exercise them. Repairing such scripts to work on the updated application is essential for maintaining the quality of the test suite. Updating such ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

October 2018

987 pages

ISBN:9781450355735

DOI:10.1145/3236024

General Chair:
Gary T. Leavens
University of Central Florida, USA
,
Program Chairs:
Alessandro Garcia
PUC-Rio, Brazil
,
Corina S. Păsăreanu
NASA Ames Research Center, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEC/FSE '18

Sponsor:

SIGSOFT

ESEC/FSE '18: 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 4 - 9, 2018

FL, Lake Buena Vista, USA

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
571
Total Downloads

Downloads (Last 12 months)64
Downloads (Last 6 weeks)1

Reflects downloads up to 14 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ji RQian ZLi ZZhu XPei YPan MZhang T(2024)Repairing Obsolete GUI Test Scripts for Android Applications with Exploration and BacktrackingProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671377(51-60)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3671377
Di Meglio SStarace L(2024)Towards Predicting Fragility in End-to-End Web TestsProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661179(387-392)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661179
Tufano RMastropaolo APepe FDabic ODi Penta MBavota GSpinellis DConstantinou EBacchelli A(2024)Unveiling ChatGPT's Usage in Open Source Projects: A Mining-based StudyProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644918(571-583)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644918
Cao SPan MPei YYang WZhang TWang LLi XRoychoudhury APaiva AAbreu RStorey M(2024)Comprehensive Semantic Repair of Obsolete GUI Test Scripts for Mobile ApplicationsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639108(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639108
De Luca MFasolino ATramontana P(2024)Investigating the robustness of locators in template-based Web application testing using a GUI change classification modelJournal of Systems and Software10.1016/j.jss.2023.111932210:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111932
Khalili FMariani LMohebbi APezzè MTerragni V(2024)Semantic matching in GUI test reuseEmpirical Software Engineering10.1007/s10664-023-10406-829:3Online publication date: 9-May-2024
https://doi.org/10.1007/s10664-023-10406-8
Jiang PWang SLiu Y(2024)Tree-Based Synthesis of Web Test Sequences from Manual ActionsTheoretical Aspects of Software Engineering10.1007/978-3-031-64626-3_14(242-260)Online publication date: 14-Jul-2024
https://doi.org/10.1007/978-3-031-64626-3_14
Sandnes F(2024)How Order and Omission of Web Content Can Vary Unintentionally Across User Cohorts: A ReviewUniversal Access in Human-Computer Interaction10.1007/978-3-031-60881-0_6(80-99)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-60881-0_6
Zhu HWei LTerragni VLiu YCheung SWu JSheng QZhang BSong L(2023) StubCoder: Automated Generation and Repair of Stub Code for Mock ObjectsACM Transactions on Software Engineering and Methodology10.1145/361717133:1(1-31)Online publication date: 21-Aug-2023
https://dl.acm.org/doi/10.1145/3617171
Qi XQian XLi YChandra SBlincoe KTonella P(2023)Semantic Test Repair for Web ApplicationsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616324(1190-1202)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616324
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents