research-article

Open access

Test automation in open-source Android apps: a large-scale empirical study

Authors:

Navid Salehnamadi,

Sam MalekAuthors Info & Claims

ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

Pages 1078 - 1089

https://doi.org/10.1145/3324884.3416623

Published: 27 January 2021 Publication History

Abstract

Automated testing of mobile apps has received significant attention in recent years from researchers and practitioners alike. In this paper, we report on the largest empirical study to date, aimed at understanding the test automation culture prevalent among mobile app developers. We systematically examined more than 3.5 million repositories on GitHub and identified more than 12, 000 non-trivial and real-world Android apps. We then analyzed these non-trivial apps to investigate (1) the prevalence of adoption of test automation; (2) working habits of mobile app developers in regards to automated testing; and (3) the correlation between the adoption of test automation and the popularity of projects. Among others, we found that (1) only 8% of the mobile app development projects leverage automated testing practices; (2) developers tend to follow the same test automation practices across projects; and (3) popular projects, measured in terms of the number of contributors, stars, and forks on GitHub, are more likely to adopt test automation practices. To understand the rationale behind our observations, we further conducted a survey with 148 professional and experienced developers contributing to the subject apps. Our findings shed light on the current practices and future research directions pertaining to test automation for mobile app development.

References

[1]

360logica. 2020. A sneak peek into test framework and testing pyramid. Retrieved January 19, 2020 from https://www.360logica.com/blog/sneak-peek-test-framework-test-pyramid-testing-pyramid/

[2]

Christoffer Quist Adamsen, Gianluca Mezzetti, and Anders Møller. 2015. Systematic Execution of Android Test Suites in Adverse Conditions. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (Baltimore, MD, USA) (ISSTA 2015). Association for Computing Machinery, New York, NY, USA, 83--93.

Digital Library

[3]

Kevin Allix, Tegawendé F. Bissyandé, Jacques Klein, and Yves Le Traon. 2016. AndroZoo: Collecting Millions of Android Apps for the Research Community. In Proceedings of the 13th International Conference on Mining Software Repositories (Austin, Texas) (MSR '16). ACM, New York, NY, USA, 468--471.

Digital Library

[4]

AndroZoo. 2020. AndroZoo Markets. Retrieved January 19, 2020 from https://androzoo.uni.lu/markets

[5]

Larissa Azevedo, Altino Dantas, and Celso G. Camilo-Junior. 2018. DroidBugs: An Android Benchmark for Automatic Program Repair. CoRR abs/1809.07353 (2018). arXiv:1809.07353 http://arxiv.org/abs/1809.07353

[6]

F. Behrang and A. Orso. 2019. Test Migration Between Mobile Apps with Similar Functionality. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 54--65.

[7]

Moritz Beller, Georgios Gousios, Annibale Panichella, and Andy Zaidman. 2015. When, How, and Why Developers (Do Not) Test in Their IDEs. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). Association for Computing Machinery, New York, NY, USA, 179--190.

Digital Library

[8]

N. Chang, L. Wang, Y. Pei, S. K. Mondal, and X. Li. 2018. Change-Based Test Script Maintenance for Android Apps. In 2018 IEEE International Conference on Software Quality, Reliability and Security (QRS). 215--225.

[9]

W. Choi, K. Sen, G. Necul, and W. Wang. 2018. DetReduce: Minimizing Android GUI Test Suites for Regression Testing. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). 445--455.

[10]

Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated Test Input Generation for Android: Are We There Yet? (E). In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) (ASE '15). IEEE Computer Society, Washington, DC, USA, 429--440.

Digital Library

[11]

Mike Cohn. 2010. Succeeding with agile: software development using Scrum. Pearson Education.

[12]

Riccardo Coppola, Maurizio Morisio, Marco Torchiano, and Luca Ardito. 2019. Scripted GUI testing of Android open-source apps: evolution of test code and fragility causes. Empirical Software Engineering 24, 5 (01 Oct 2019), 3205--3248.

Digital Library

[13]

Luis Cruz, Rui Abreu, and David Lo. 2019. To the attention of mobile software developers: guess what, test your app! Empirical Software Engineering 24, 4 (01 Aug 2019), 2438--2468.

Digital Library

[14]

The Dataset. 2020. The Dataset. Retrieved January 19, 2020 from https://github.com/seal-hub/ASE20Empirical

[15]

Lin Deng, Jeff Offutt, Paul Ammann, and Nariman Mirzaei. 2017. Mutation operators for testing Android apps. Information and Software Technology 81 (2017), 154--168.

Digital Library

[16]

Paul M Duvall, Steve Matyas, and Andrew Glover. 2007. Continuous integration: improving software quality and reducing risk. Pearson Education.

[17]

F-Droid. 2020. F-Droid - Free and Open Source Android App Repository. Retrieved January 19, 2020 from https://f-droid.org

[18]

Martin Fowler. 2020. Test Pyramid. Retrieved January 19, 2020 from https://martinfowler.com/bliki/TestPyramid.html

[19]

G. Fraser and A. Arcuri. 2012. Sound empirical evidence in software testing. In 2012 34th International Conference on Software Engineering (ICSE). 178--188.

[20]

GHTorrent. 2020. GHTorrent Downloads. Retrieved January 19, 2020 from https://ghtorrent.org/downloads.html

[21]

Google. 2020. Advanced Android in Kotlin: Testing Basics. Retrieved January 19, 2020 from https://codelabs.developers.google.com/codelabs/advanced-android-kotlin-training-testing-basics/index.html#4

[22]

Google. 2020. Android Studio release notes. Retrieved January 19, 2020 from https://developer.android.com/studio/releases

[23]

Google. 2020. App Manifest Overview. Retrieved January 19, 2020 from https://developer.android.com/guide/topics/manifest/manifest-intro

[24]

Google. 2020. Application Fundamentals. Retrieved January 19, 2020 from https://developer.android.com/guide/components/fundamentals.html

[25]

Google. 2020. Create UI tests with Espresso Test Recorder. Retrieved January 19, 2020 from https://developer.android.com/studio/test/espresso-test-recorder

[26]

Google. 2020. Espresso. Retrieved January 19, 2020 from https://developer.android.com/training/testing/espresso

[27]

Google. 2020. Fundamentals of Testing. Retrieved January 19, 2020 from https://developer.android.com/training/testing/fundamentals

[28]

Georgios Gousios. 2013. The GHTorrent dataset and tool suite. In Proceedings of the 10th Working Conference on Mining Software Repositories (San Francisco, CA, USA) (MSR '13). IEEE Press, Piscataway, NJ, USA, 233--236. http://dl.acm.org/citation.cfm?id=2487085.2487132

Digital Library

[29]

Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (Singapore, Singapore) (ASE 2016). Association for Computing Machinery, New York, NY, USA, 426--437.

Digital Library

[30]

Reyhaneh Jabbarvand and Sam Malek. 2017. MuDroid: An Energy-Aware Mutation Testing Framework for Android. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). Association for Computing Machinery, New York, NY, USA, 208--219.

Digital Library

[31]

Reyhaneh Jabbarvand, Alireza Sadeghi, Hamid Bagheri, and Sam Malek. 2016. Energy-Aware Test-Suite Minimization for Android Apps. In Proceedings of the 25th International Symposium on Software Testing and Analysis (Saarbrücken, Germany) (ISSTA 2016). Association for Computing Machinery, New York, NY, USA, 425--436.

Digital Library

[32]

B. Jiang, Y. Wu, Y. Zhang, Z. Zhang, and W. K. Chan. 2018. ReTestDroid: Towards Safer Regression Test Selection for Android Application. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol. 01. 235--244.

[33]

Mona Erfani Joorabchi, Ali Mesbah, and Philippe Kruchten. 2013. Real Challenges in Mobile App Development. In 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, Baltimore, Maryland, USA, October 10--11, 2013. 15--24.

[34]

JUnit. 2020. JUnit. Retrieved January 19, 2020 from https://junit.org

[35]

Kostis Kapelonis. 2020. Software Testing Anti-patterns. Retrieved January 19, 2020 from http://blog.codepipes.com/testing/software-testing-antipatterns.html

[36]

P. S. Kochhar, T. F. Bissyandé, D. Lo, and L. Jiang. 2013. An Empirical Study of Adoption of Software Testing in Open Source Projects. In 2013 13th International Conference on Quality Software. 103--112.

Digital Library

[37]

Pavneet Singh Kochhar, Ferdian Thung, Nachiappan Nagappan, Thomas Zimmermann, and David Lo. 2015. Understanding the Test Automation Culture of App Developers. In Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International Conference on. IEEE, 1--10.

[38]

Erwin Kreyszig. 2009. Advanced Engineering Mathematics, 10th Eddition. Wiley.

[39]

X. Li, N. Chang, Y. Wang, H. Huang, Y. Pei, L. Wang, and X. Li. 2017. ATOM: Automatic Maintenance of GUI Test Scripts for Evolving Mobile Applications. In 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST). 161--171.

[40]

J. Lin, R. Jabbarvand, and S. Malek. 2019. Test Transfer Across Mobile Apps Through Semantic Mapping. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 42--53.

[41]

Mario Linares-Vásquez, Gabriele Bavota, Michele Tufano, Kevin Moran, Massimiliano Di Penta, Christopher Vendome, Carlos Bernal-Cárdenas, and Denys Poshyvanyk. 2017. Enabling Mutation Testing for Android Apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). Association for Computing Machinery, New York, NY, USA, 233--244.

Digital Library

[42]

M. Linares-Vásquez, C. Bernal-Cardenas, K. Moran, and D. Poshyvanyk. 2017. How do Developers Test Android Applications?. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME). 613--622.

[43]

M. Linares-Vásquez, K. Moran, and D. Poshyvanyk. 2017. Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mobile App Testing. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME). 399--410.

[44]

H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. Ann. Math. Statist. 18, 1 (03 1947), 50--60.

[45]

Atif M. Memon and Myra B. Cohen. 2013. Automated Testing of GUI Applications: Models, Tools, and Controlling Flakiness. In Proceedings of the 2013 International Conference on Software Engineering (San Francisco, CA, USA) (ICSE '13). IEEE Press, 1479--1480.

[46]

Mockito. 2020. Most popular mocking framework for Java. Retrieved January 19, 2020 from https://github.com/mockito/mockito

[47]

Duncan Nisbet. 2020. Test Automation Basics - Levels, Pyramids & Quadrants. Retrieved January 19, 2020 from http://www.duncannisbet.co.uk/test-automation-basics-levels-pyramids-quadrants

[48]

Chairat Onyaem. 2020. Separate Unit, Integration, and Functional Tests for Continuous Delivery. Retrieved January 19, 2020 from https://medium.com/pacroy/separate-unit-integration-and-functional-tests-for-continuous-delivery-f4dc240d8f2f

[49]

M. Pan, T. Xu, Y. Pei, Z. Li, T. Zhang, and X. Li. 2019. GUI-Guided Repair of Mobile Test Scripts. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). 326--327.

[50]

Fabiano Pecorelli, Gemma Catolino, Filomena Ferrucci, Andrea De Lucia, and Fabio Palomba. 2020. Testing of Mobile Applications in the Wild: A Large-Scale Empirical Study on Android Apps. In 28th IEEE/ACM International Conference on Program Comprehension (ICPC).

Digital Library

[51]

Qualtrics. 2020. Qualtrics Survey Software. Retrieved January 19, 2020 from https://www.qualtrics.com/

[52]

Robolectric. 2020. Test-drive your Android code. Retrieved January 19, 2020 from http://robolectric.org/

[53]

Shayna Rusticus and Chris Lovato. 2014. Impact of Sample Size and Variability on the Power and Type I Error Rates of Equivalence Tests: A Simulation Study. Practical Assessment, Research and Evaluation 19 (01 2014).

[54]

S. S. SHAPIRO and M. B. WILK. 1965. An analysis of variance test for normality (complete samples)&dagger;. Biometrika 52, 3--4 (12 1965), 591--611. arXiv:https://academic.oup.com/biomet/article-pdf/52/3-4/591/962907/52-3-4-591.pdf

[55]

S. H. Tan, Z. Dong, X. Gao, and A. Roychoudhury. 2018. Repairing Crashes in Android Apps. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). 187--198.

[56]

Wenyu Wang, Dengfeng Li, Wei Yang, Yurui Cao, Zhenwen Zhang, Yuetang Deng, and Tao Xie. 2018. An Empirical Study of Android Test Generation Tools in Industrial Cases. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (Montpellier, France) (ASE 2018). Association for Computing Machinery, New York, NY, USA, 738--748.

Digital Library

[57]

Xia Zeng, Dengfeng Li, Wujie Zheng, Fan Xia, Yuetang Deng, Wing Lam, Wei Yang, and Tao Xie. 2016. Automated Test Input Generation for Android: Are We Really There yet in an Industrial Case?. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (Seattle, WA, USA) (FSE 2016). Association for Computing Machinery, New York, NY, USA, 987--992.

Digital Library

Cited By

Yu LAlégroth EChatzipetrou PGorschek T(2024)A Roadmap for Using Continuous Integration EnvironmentsCommunications of the ACM10.1145/363151967:6(82-90)Online publication date: 12-Apr-2024
https://dl.acm.org/doi/10.1145/3631519
Taeb MSwearngin ASchoop ECheng RJiang YNichols J(2024)AXNav: Replaying Accessibility Tests from Natural LanguageProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642777(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642777
Salehnamadi NHe ZMalek S(2023)Assistive-Technology Aided Manual Accessibility Testing in Mobile Apps, Powered by Record-and-ReplayProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580679(1-20)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580679
Show More Cited By

Index Terms

Test automation in open-source Android apps: a large-scale empirical study
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Inter-app communication between Android apps developed in app-inventor and Android studio
MOBILESoft '16: Proceedings of the International Conference on Mobile Software Engineering and Systems

Communications between mobile apps are an important aspect of mobile platforms. Android is specifically designed with inter-app communication in mind and depends on this to provide different platform specific functionalities. Android Apps can either be ...
Comparing Apples to Androids: Discovery, Retrieval, and Matching of iOS and Android Apps for Cross-Platform Analyses
MSR '24: Proceedings of the 21st International Conference on Mining Software Repositories

For years, researchers have been analyzing mobile Android apps to investigate diverse properties such as software engineering practices, business models, security, privacy, or usability, as well as differences between marketplaces. While similar studies ...
Code smells in iOS apps: how do they compare to Android?
MOBILESoft '17: Proceedings of the 4th International Conference on Mobile Software Engineering and Systems

With billions of app downloads, the Apple App Store and Google Play Store succeeded to conquer mobile devices. However, this success also challenges app developers to publish high-quality apps to keep attracting and satisfying end-users. In particular, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

December 2020

1449 pages

ISBN:9781450367684

DOI:10.1145/3324884

General Chair:
John Grundy,
Program Chairs:
Claire Le Goues,
David Lo

Copyright © 2020 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 January 2021

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ASE '20

Sponsor:

ASE '20: 35th IEEE/ACM International Conference on Automated Software Engineering

December 21 - 25, 2020

Virtual Event, Australia

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
894
Total Downloads

Downloads (Last 12 months)295
Downloads (Last 6 weeks)24

Reflects downloads up to 14 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yu LAlégroth EChatzipetrou PGorschek T(2024)A Roadmap for Using Continuous Integration EnvironmentsCommunications of the ACM10.1145/363151967:6(82-90)Online publication date: 12-Apr-2024
https://dl.acm.org/doi/10.1145/3631519
Taeb MSwearngin ASchoop ECheng RJiang YNichols J(2024)AXNav: Replaying Accessibility Tests from Natural LanguageProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642777(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642777
Salehnamadi NHe ZMalek S(2023)Assistive-Technology Aided Manual Accessibility Testing in Mobile Apps, Powered by Record-and-ReplayProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580679(1-20)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580679
Li CXiong YLi ZYang WPan M(2023)Mobile Test Script Generation from Natural Language Descriptions2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS)10.1109/QRS60937.2023.00042(348-359)Online publication date: 22-Oct-2023
https://doi.org/10.1109/QRS60937.2023.00042
Gambi AGouni HBerreiter DTymofyeyev VFazzini M(2023)Action-Based Test Carving for Android Apps2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00032(107-116)Online publication date: Apr-2023
https://doi.org/10.1109/ICSTW58534.2023.00032
Rua RSaraiva J(2023)PyAnaDroid: A fully-customizable execution pipeline for benchmarking Android Applications2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00077(586-591)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICSME58846.2023.00077
Gambi ALi MFazzini M(2023)Artisan: An Action-Based Test Carving Tool for Android Apps2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00076(580-585)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICSME58846.2023.00076
Gu RRojas J(2023)An Empirical Study on the Adoption of Scripted GUI Testing for Android Apps2023 38th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW)10.1109/ASEW60602.2023.00030(179-182)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASEW60602.2023.00030
Rua RSaraiva J(2023)A large-scale empirical study on mobile performance: energy, run-time and memoryEmpirical Software Engineering10.1007/s10664-023-10391-y29:1Online publication date: 27-Dec-2023
https://doi.org/10.1007/s10664-023-10391-y
Qian JMa YLin CChen L(2022)Accelerating OCR-Based Widget Localization for Test Automation of GUI ApplicationsProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556966(1-13)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3556966
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents