research-article

Testing Real-World Healthcare IoT Application: Experiences and Lessons Learned

Authors:

Kjetil MobergAuthors Info & Claims

ESEC/FSE 2023: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 2044 - 2049

https://doi.org/10.1145/3611643.3613888

Published: 30 November 2023 Publication History

Abstract

Healthcare Internet of Things (IoT) applications require rigorous testing to ensure their dependability. Such applications are typically integrated with various third-party healthcare applications and medical devices through REST APIs. This integrated network of healthcare IoT applications leads to REST APIs with complicated and interdependent structures, thus creating a major challenge for automated system-level testing. We report an industrial evaluation of a state-of-the-art REST APIs testing approach (RESTest) on a real-world healthcare IoT application. We analyze the effectiveness of RESTest’s testing strategies regarding REST APIs failures, faults in the application, and REST API coverage, by experimenting with six REST APIs of 41 API endpoints of the healthcare IoT application. Results show that several failures are discovered in different REST APIs with ≈56% coverage using RESTest. Moreover, nine potential faults are identified. Using the evidence collected from the experiments, we provide our experiences and lessons learned.

References

[1]

[n. d.]. APIFuzzer — HTTP API Testing Framework. https://github.com/KissPeter/APIFuzzer [Online; accessed 20-April-2023]

[2]

[n. d.]. Dredd — HTTP API Testing Framework. https://github.com/apiaryio/dredd [Online; accessed 20-April-2023]

[3]

[n. d.]. National Welfare Technology Program. https://www.helsedirektoratet.no/tema/velferdsteknologi [Online; accessed 11-May-2023]

[4]

[n. d.]. Norwegian Health Authority. https://www.oslo.kommune.no/etater-foretak-og-ombud/helseetaten/ [Online; accessed 11-May-2023]

[5]

[n. d.]. Tcases: A Model-Based Test Case Generator. https://github.com/Cornutum/tcases [Online; accessed 20-April-2023]

[6]

Abbas Ahmad, Fabrice Bouquet, Elizabeta Fourneret, Franck Le Gall, and Bruno Legeard. 2016. Model-based testing as a service for iot platforms. In Leveraging Applications of Formal Methods, Verification and Validation: Discussion, Dissemination, Applications: 7th International Symposium, ISoLA 2016, Imperial, Corfu, Greece, October 10-14, 2016, Proceedings, Part II 7. 727–742.

[7]

Juan C Alonso, Alberto Martin-Lopez, Sergio Segura, Jose Maria Garcia, and Antonio Ruiz-Cortes. 2022. ARTE: Automated Generation of Realistic Test Inputs for Web APIs. IEEE Transactions on Software Engineering, 49, 1 (2022), 348–363.

[8]

Domenico Amalfitano, Nicola Amatucci, Vincenzo De Simone, Vincenzo Riccio, and Fasolino Anna Rita. 2017. Towards a Thing-In-the-Loop approach for the verification and validation of IoT systems. In Proceedings of the 1st ACM Workshop on the Internet of Safe Things. 57–63. https://doi.org/10.1145/3137003.3137007

Digital Library

[9]

Andrea Arcuri. 2018. Evomaster: Evolutionary multi-context automated system test generation. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). 394–397.

[10]

Andrea Arcuri. 2019. RESTful API automated test case generation with EvoMaster. ACM Transactions on Software Engineering and Methodology (TOSEM), 28, 1 (2019), 1–37. https://doi.org/10.1145/3293455

Digital Library

[11]

Andrea Arcuri, Man Zhang, Asma Belhadi, Bogdan Marculescu, Amid Golmohammadi, Juan Pablo Galeotti, and Susruthan Seran. 2023. Building an open-source system test generation tool: lessons learned and empirical analyses with EvoMaster. Software Quality Journal, 1–44.

[12]

Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. Restler: Stateful rest api fuzzing. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 748–758.

Digital Library

[13]

Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2020. Checking security properties of cloud service REST APIs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). 387–397.

[14]

Davide Corradini, Amedeo Zampieri, Michele Pasqua, and Mariano Ceccato. 2022. RestTestGen: An Extensible Framework for Automated Black-box Testing of RESTful APIs. In 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). 504–508.

[15]

Davide Corradini, Amedeo Zampieri, Michele Pasqua, Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2022. Automated black-box testing of nominal and error scenarios in RESTful APIs. Software Testing, Verification and Reliability, 32, 5 (2022), e1808. https://doi.org/10.1002/stvr.1808

[16]

João Pedro Dias, Flávio Couto, Ana CR Paiva, and Hugo Sereno Ferreira. 2018. A brief overview of existing tools for testing the internet-of-things. In 2018 IEEE international conference on software testing, verification and validation workshops (ICSTW). 104–109. https://doi.org/10.1109/ICSTW.2018.00035

[17]

Hamza Ed-Douibi, Javier Luis Cánovas Izquierdo, and Jordi Cabot. 2018. Automatic generation of test cases for REST APIs: A specification-based approach. In 2018 IEEE 22nd international enterprise distributed object computing conference (EDOC). 181–190. https://doi.org/10.1109/EDOC.2018.00031

[18]

Haya Elayan, Moayad Aloqaily, and Mohsen Guizani. 2021. Digital twin for intelligent context-aware IoT healthcare systems. IEEE Internet of Things Journal, 8, 23 (2021), 16749–16757. https://doi.org/10.1109/JIOT.2021.3051158

[19]

Duarte Felício, José Simão, and Nuno Datia. 2023. RapiTest: Continuous Black-Box Testing of RESTful Web APIs. Procedia Computer Science, 219 (2023), 537–545.

Digital Library

[20]

Roy Thomas Fielding. 2000. Architectural styles and the design of network-based software architectures. University of California, Irvine.

Digital Library

[21]

Bernhard Garn, Dominik-Philip Schreiber, Dimitris E Simos, Rick Kuhn, Jeff Voas, and Raghu Kacker. 2022. Combinatorial methods for testing internet of things smart home systems. Software Testing, Verification and Reliability, 32, 2 (2022), e1805.

[22]

Patrice Godefroid, Bo-Yuan Huang, and Marina Polishchuk. 2020. Intelligent REST API data fuzzing. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 725–736. https://doi.org/10.1145/3368089.3409719

Digital Library

[23]

Patrice Godefroid, Daniel Lehmann, and Marina Polishchuk. 2020. Differential regression testing for REST APIs. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 312–323.

Digital Library

[24]

Zac Hatfield-Dodds and Dmitry Dygalo. 2022. Deriving semantics-aware fuzzers from web API schemas. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 345–346.

Digital Library

[25]

Linghuan Hu, W Eric Wong, D Richard Kuhn, Raghu N Kacker, and Shuo Li. 2022. CT-IoT: a combinatorial testing-based path selection framework for effective IoT testing. Empirical Software Engineering, 27 (2022), 1–38.

Digital Library

[26]

Stefan Karlsson, Adnan Čaušević, and Daniel Sundmark. 2020. QuickREST: Property-based test generation of OpenAPI-described RESTful APIs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). 131–141. https://doi.org/10.1109/ICST46399.2020.00023

[27]

Myeongsoo Kim, Qi Xin, Saurabh Sinha, and Alessandro Orso. 2022. Automated test generation for rest apis: No time to rest yet. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 289–301.

Digital Library

[28]

Nuno Laranjeiro, João Agnelo, and Jorge Bernardino. 2021. A black box tool for robustness testing of REST services. IEEE Access, 9 (2021), 24738–24754.

[29]

Yi Liu, Yuekang Li, Gelei Deng, Yang Liu, Ruiyuan Wan, Runchao Wu, Dandan Ji, Shiheng Xu, and Minli Bao. 2022. Morest: model-based RESTful API testing with execution feedback. In Proceedings of the 44th International Conference on Software Engineering. 1406–1417. https://doi.org/10.1145/3510003.3510133

Digital Library

[30]

Alberto Martin-Lopez, Andrea Arcuri, Sergio Segura, and Antonio Ruiz-Cortés. 2021. Black-box and white-box test case generation for RESTful APIs: Enemies or allies? In 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE). 231–241.

[31]

Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2019. Test coverage criteria for RESTful web APIs. In Proceedings of the 10th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. 15–21. https://doi.org/10.1145/3340433.3342822

Digital Library

[32]

Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2021. RESTest: automated black-box testing of RESTful web APIs. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. 682–685. https://doi.org/10.1145/3460319.3469082

Digital Library

[33]

Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2022. Online testing of RESTful APIs: Promises and challenges. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 408–420.

Digital Library

[34]

A Giuliano Mirabella, Alberto Martin-Lopez, Sergio Segura, Luis Valencia-Cabrera, and Antonio Ruiz-Cortés. 2021. Deep learning-based prediction of test input validity for RESTful APIs. In 2021 IEEE/ACM Third International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest). 9–16. https://doi.org/10.1109/DeepTest52559.2021.00008

[35]

Francis Palma, Tobias Olsson, Anna Wingkvist, and Javier Gonzalez-Huerta. 2022. Assessing the linguistic quality of REST APIs for IoT applications. Journal of Systems and Software, 191 (2022), 111369.

Digital Library

[36]

Hassan Sartaj, Shaukat Ali, Tao Yue, and Kjetil Moberg. 2023. HITA: An Architecture for System-level Testing of Healthcare IoT Applications. In Proceedings of the 17th European Conference on Software Architecture: Companion Proceedings. To Appear

[37]

Sergio Segura, José A Parejo, Javier Troya, and Antonio Ruiz-Cortés. 2018. Metamorphic testing of RESTful web APIs. In Proceedings of the 40th International Conference on Software Engineering. 882–882.

Digital Library

[38]

Janet Siegmund, Norbert Siegmund, and Sven Apel. 2015. Views on internal and external validity in empirical software engineering. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering. 1, 9–19.

[39]

Stelios Sotiriadis, Nik Bessis, Eleana Asimakopoulou, and Navonil Mustafee. 2014. Towards simulating the internet of things. In 2014 28th International Conference on Advanced Information Networking and Applications Workshops. 444–448.

Digital Library

[40]

Dimitri Stallenberg, Mitchell Olsthoorn, and Annibale Panichella. 2021. Improving test case generation for REST APIs through hierarchical clustering. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 117–128. https://doi.org/10.1109/ASE51524.2021.9678586

Digital Library

[41]

Tao Wang, Kangkang Zhang, Wei Chen, Wensheng Dou, Jiaxin Zhu, Jun Wei, and Tao Huang. 2022. Understanding device integration bugs in smart home system. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 429–441.

Digital Library

[42]

Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. 2022. Combinatorial testing of restful apis. In Proceedings of the 44th International Conference on Software Engineering. 426–437. https://doi.org/10.1145/3510003.3510151

Digital Library

Cited By

Sartaj HAli SGjøby J(2024)Digital Twins Environment Simulation for Testing Healthcare IoT Applications2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00124(900-901)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00124
Sartaj HAli SMarie Gjøby J(2024)Uncertainty-aware environment simulation of medical devices digital twinsSoftware and Systems Modeling10.1007/s10270-024-01223-8Online publication date: 1-Nov-2024
https://doi.org/10.1007/s10270-024-01223-8
Sartaj HAli SYue TMoberg K(2024)Model‐based digital twins of medicine dispensers for healthcare IoT applicationsSoftware: Practice and Experience10.1002/spe.331154:6(1172-1192)Online publication date: 15-Jan-2024
https://doi.org/10.1002/spe.3311
Show More Cited By

Index Terms

Testing Real-World Healthcare IoT Application: Experiences and Lessons Learned
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Assessing the linguistic quality of REST APIs for IoT applications
Abstract
Internet of Things (IoT) is a growing technology that relies on connected ‘things’ that gather data from peer devices and send data to servers via APIs (Application Programming Interfaces). The design quality of those APIs has a direct ...
Highlights
- SARAv2 approach for syntactic & semantic analysis of REST APIs for IoT applications.
Generating REST API Specifications through Static Analysis
ICSE '24: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

Web Application Programming Interfaces (APIs) allow services to be accessed over the network. RESTful (or REST) APIs, which use the REpresentation State Transfer (REST) protocol, are a popular type of web API. To use or test REST APIs, developers use ...
Testing of RESTful Web APIs
Service-Oriented Computing – ICSOC 2022 Workshops
Abstract
RESTful web APIs nowadays may be considered the de facto standard for web integration, since they enable interoperability between heterogeneous software systems in a standard way, and their usage is widespread in industry. Testing these systems ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2023: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

November 2023

2215 pages

ISBN:9798400703270

DOI:10.1145/3611643

General Chair:
Satish Chandra
Google, USA
,
Program Chairs:
Kelly Blincoe
University of Auckland, New Zealand
,
Paolo Tonella
USI Lugano, Switzerland

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 November 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Research Council of Norway

Conference

ESEC/FSE '23

Sponsor:

SIGSOFT

ESEC/FSE '23: 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

December 3 - 9, 2023

CA, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
87
Total Downloads

Downloads (Last 12 months)87
Downloads (Last 6 weeks)8

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sartaj HAli SGjøby J(2024)Digital Twins Environment Simulation for Testing Healthcare IoT Applications2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00124(900-901)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00124
Sartaj HAli SMarie Gjøby J(2024)Uncertainty-aware environment simulation of medical devices digital twinsSoftware and Systems Modeling10.1007/s10270-024-01223-8Online publication date: 1-Nov-2024
https://doi.org/10.1007/s10270-024-01223-8
Sartaj HAli SYue TMoberg K(2024)Model‐based digital twins of medicine dispensers for healthcare IoT applicationsSoftware: Practice and Experience10.1002/spe.331154:6(1172-1192)Online publication date: 15-Jan-2024
https://doi.org/10.1002/spe.3311
Isaku ESartaj HLaaber CYue TAli SSchwitalla TNygård J(2023)Cost Reduction on Testing Evolving Cancer Registry System2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00065(508-518)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICSME58846.2023.00065
Sartaj HAli SYue TGjøby J(2023)HITA: An Architecture for System-level Testing of Healthcare IoT ApplicationsSoftware Architecture. ECSA 2023 Tracks, Workshops, and Doctoral Symposium10.1007/978-3-031-66326-0_28(451-468)Online publication date: 18-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-66326-0_28

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents