research-article

An exploratory study of autopilot software bugs in unmanned aerial vehicles

Authors:

Yulei SuiAuthors Info & Claims

ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 20 - 31

https://doi.org/10.1145/3468264.3468559

Published: 18 August 2021 Publication History

Abstract

Unmanned aerial vehicles (UAVs) are becoming increasingly important and widely used in modern society. Software bugs in these systems can cause severe issues, such as system crashes, hangs, and undefined behaviors. Some bugs can also be exploited by hackers to launch security attacks, resulting in catastrophic consequences. Therefore, techniques that can help detect and fix software bugs in UAVs are highly desirable. However, although there are many existing studies on bugs in various types of software, the characteristics of UAV software bugs have never been systematically studied. This impedes the development of tools for assuring the dependability of UAVs. To bridge this gap, we conducted the first large-scale empirical study on two well-known open-source autopilot software platforms for UAVs, namely PX4 and Ardupilot, to characterize bugs in UAVs. Through analyzing 569 bugs from these two projects, we observed eight types of UAV-specific bugs (i.e., limit, math, inconsistency, priority, parameter, hardware support, correction, and initialization) and learned their root causes. Based on the bug taxonomy, we summarized common bug patterns and repairing strategies. We further identified five challenges associated with detecting and fixing such UAV-specific bugs. Our study can help researchers and practitioners to better understand the threats to the dependability of UAV systems and facilitate the future development of UAV bug diagnosis tools.

References

[1]

Uttam Adhikari, Thomas H Morris, and Shengyi Pan. 2014. A cyber-physical power system test bed for intrusion detection systems. In 2014 IEEE PES General Meeting| Conference & Exposition. 1–5. https://doi.org/10.1109/PESGM.2014.6939262

[2]

Michel Albonico, Ivano Malavolta, Gustavo Pinto, Emitza Guzman, Katerina Chinnappan, and Patricia Lago. 2021. Mining Energy-Related Practices in Robotics Software. CoRR, abs/2103.13762 (2021), arxiv:2103.13762. arxiv:2103.13762

[3]

Ardupilot.org. 2020. Homepage of Ardupilot. https://ardupilot.org

[4]

Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2014. The oracle problem in software testing: A survey. IEEE transactions on software engineering, 41, 5 (2014), 507–525. https://doi.org/10.1109/TSE.2014.2372785

Digital Library

[5]

Nicolas Bettenburg, Sascha Just, Adrian Schröter, Cathrin Weiß, Rahul Premraj, and Thomas Zimmermann. 2007. Quality of bug reports in eclipse. In Proceedings of the 2007 OOPSLA workshop on eclipse technology eXchange. 21–25. https://doi.org/10.1145/1328279.1328284

Digital Library

[6]

Nicolas Bettenburg, Sascha Just, Adrian Schröter, Cathrin Weiss, Rahul Premraj, and Thomas Zimmermann. 2008. What makes a good bug report? In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering. 308–318. https://doi.org/10.1145/1453101.1453146

Digital Library

[7]

John W Creswell and Cheryl N Poth. 2016. Qualitative inquiry and research design: Choosing among five approaches. SAGE publications. https://doi.org/10.1177/1524839915580941

[8]

Rogério De Lemos, Holger Giese, Hausi A Müller, Mary Shaw, Jesper Andersson, Marin Litoiu, Bradley Schmerl, Gabriel Tamura, Norha M Villegas, and Thomas Vogel. 2013. Software engineering for self-adaptive systems: A second research roadmap. In Software Engineering for Self-Adaptive Systems II. Springer, 1–32. https://doi.org/10.1007/978-3-642-35813-5_1

[9]

Xiaoting Du, Guanping Xiao, and Yulei Sui. 2020. Fault Triggers in the TensorFlow Framework: An Experience Report. In 31st IEEE International Symposium on Software Reliability Engineering, ISSRE 2020, Coimbra, Portugal, October 12-15, 2020, Marco Vieira, Henrique Madeira, Nuno Antunes, and Zheng Zheng (Eds.). IEEE, 1–12. https://doi.org/10.1109/ISSRE5003.2020.00010

[10]

Emad Ebeid, Martin Skriver, Kristian Husum Terkildsen, Kjeld Jensen, and Ulrik Pagh Schultz. 2018. A survey of open-source UAV flight controllers and flight simulators. Microprocessors and Microsystems, 61 (2018), 11–20. https://doi.org/10.1016/j.micpro.2018.05.002

[11]

Anders Fischer-Nielsen, Zhoulai Fu, Ting Su, and Andrzej Wasowski. 2020. The forgotten case of the dependency bugs: on the example of the robot operating system. In ICSE-SEIP 2020: 42nd International Conference on Software Engineering, Software Engineering in Practice, Seoul, South Korea, 27 June - 19 July, 2020, Gregg Rothermel and Doo-Hwan Bae (Eds.). ACM, 21–30.

Digital Library

[12]

Gordon Fraser and Andreas Zeller. 2011. Mutation-driven generation of unit tests and oracles. IEEE Transactions on Software Engineering, 38, 2 (2011), 278–292. https://doi.org/10.1109/TSE.2011.93

Digital Library

[13]

Joshua Garcia, Yang Feng, Junjie Shen, Sumaya Almanee, Yuan Xia, and Chen. 2020. A comprehensive study of autonomous vehicle bugs. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 385–396. https://doi.org/10.1145/3377811.3380397

Digital Library

[14]

Gregory Gay, Matt Staats, Michael Whalen, and Mats PE Heimdahl. 2015. Automated oracle data selection support. IEEE Transactions on Software Engineering, 41, 11 (2015), 1119–1137. https://doi.org/10.1109/TSE.2015.2436920

Digital Library

[15]

Dimitra Giannakopoulou, Neha Rungta, and Michael Feary. 2011. Automated test case generation for an autopilot requirement prototype. In 2011 IEEE International Conference on Systems, Man, and Cybernetics. 1825–1830. https://doi.org/10.1109/ICSMC.2011.6083936

[16]

GitHub.com. 2020. Issue 4745 of project PX4. https://github.com/PX4/Firmware/issues/4745

[17]

GitHub.com. 2020. Issue 5035 of project PX4. https://github.com/PX4/Firmware/issues/5305

[18]

GitHub.com. 2020. Issue 5110 of project PX4. https://github.com/PX4/Firmware/issues/5110

[19]

GitHub.com. 2020. Issue 5243 of project PX4. https://github.com/PX4/Firmware/issues/5243

[20]

GitHub.com. 2020. Issue 6444 of project PX4. https://github.com/PX4/Firmware/issues/6444

[21]

GitHub.com. 2020. Issue 6651 of project PX4. https://github.com/PX4/Firmware/issues/6651

[22]

GitHub.com. 2020. Issue 6669 of project PX4. https://github.com/PX4/Firmware/issues/6669

[23]

GitHub.com. 2020. Issue 7079 of project PX4. https://github.com/PX4/Firmware/issues/7097

[24]

GitHub.com. 2020. Issue 7535 of project PX4. https://github.com/PX4/Firmware/issues/7535

[25]

GitHub.com. 2020. Issue 7737 of project PX4. https://github.com/PX4/Firmware/issues/7737

[26]

GitHub.com. 2020. Issue 7746 of project PX4. https://github.com/PX4/Firmware/issues/7746

[27]

GitHub.com. 2020. Issue 8186 of project PX4. https://github.com/PX4/Firmware/issues/8186

[28]

GitHub.com. 2020. Issue 8189 of project PX4. https://github.com/PX4/Firmware/issues/8189

[29]

GitHub.com. 2020. Issue 8628 of project PX4. https://github.com/PX4/Firmware/issues/8628

[30]

GitHub.com. 2020. Pull Request 11208 of project Ardupilot. https://github.com/ArduPilot/ardupilot/pull/11208

[31]

Gautier Hattenberger, Murat Bronz, and Michel Gorraz. 2014. Using the paparazzi UAV system for scientific research.

[32]

Zhijian He, Yao Chen, Enyan Huang, Qixin Wang, Yu Pei, and Haidong Yuan. 2019. A system identification based oracle for control-cps software fault localization. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 116–127. https://doi.org/10.1109/ICSE.2019.00029

Digital Library

[33]

Lydia M. Hilton. 2017. Personal Injury and Property Damage with Drones. https://www.bfvlaw.com/personal-injury-and-property-damage-with-drones/

[34]

Jeff Huang, Charles Zhang, and Julian Dolby. 2013. CLAP: recording local executions to reproduce concurrency failures. In ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’13, Seattle, WA, USA, June 16-19, 2013, Hans-Juergen Boehm and Cormac Flanagan (Eds.). ACM, 141–152. https://doi.org/10.1145/2499370.2462167

Digital Library

[35]

James A Jones and Mary Jean Harrold. 2005. Empirical evaluation of the tarantula automatic fault-localization technique. In Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering. 273–282. https://doi.org/10.1145/1101908.1101949

Digital Library

[36]

Aaron Kane, Thomas Fuhrman, and Philip Koopman. 2014. Monitor based oracles for cyber-physical system testing: Practical experience report. In 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks. 148–155. https://doi.org/10.1109/DSN.2014.28

Digital Library

[37]

Edward A Lee. 2008. Cyber physical systems: Design challenges. In 2008 11th IEEE international symposium on object and component-oriented real-time distributed computing (ISORC). 363–369. https://doi.org/10.1109/ISORC.2008.25

Digital Library

[38]

Ma ł gorzata Steinder and Adarshpal S Sethi. 2004. A survey of fault localization techniques in computer networks. Science of computer programming, 53, 2 (2004), 165–194. https://doi.org/10.1016/j.scico.2004.01.010

[39]

Shuqing Li, Yechang Wu, Yi Liu, Dinghua Wang, Ming Wen, Yida Tao, Yulei Sui, and Yepang Liu. 2020. An Exploratory Study of Bugs in Extended Reality Applications on the Web. In 31st IEEE International Symposium on Software Reliability Engineering, ISSRE 2020, Coimbra, Portugal, October 12-15, 2020, Marco Vieira, Henrique Madeira, Nuno Antunes, and Zheng Zheng (Eds.). IEEE, 172–183. https://doi.org/10.1109/ISSRE5003.2020.00025

[40]

Lorenz Meier, Dominik Honegger, and Marc Pollefeys. 2015. PX4: A node-based multithreaded open source robotics framework for deeply embedded platforms. In 2015 IEEE international conference on robotics and automation (ICRA). 6235–6240. https://doi.org/10.1109/ICRA.2015.7140074

[41]

Claudio Menghi, Shiva Nejati, Khouloud Gaaloul, and Lionel C Briand. 2019. Generating automated and online test oracles for simulink models with continuous and uncertain behaviors. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 27–38. https://doi.org/10.1145/3338906.3338920

Digital Library

[42]

Nataliia Neshenko, Elias Bou-Harb, Jorge Crichigno, Georges Kaddoum, and Nasir Ghani. 2019. Demystifying IoT security: an exhaustive survey on IoT vulnerabilities and a first empirical look on internet-scale IoT exploitations. IEEE Communications Surveys & Tutorials, 21, 3 (2019), 2702–2733. https://doi.org/10.1109/COMST.2019.2910750

[43]

PX4.io. 2020. Airframe Reference in the User Guide of PX4. https://docs.px4.io/master/en/airframes/airframe_reference.html

[44]

PX4.io. 2020. Flight Modes in the User Guide of PX4. https://dev.px4.io/v1.9.0/en/concept/flight_modes.html

[45]

PX4.io. 2020. Parameter Reference in the User Guide of PX4. https://docs.px4.io/master/zh/advanced/parameter_reference.html

[46]

Yi Qin, Tao Xie, Chang Xu, Angello Astorga, and Jian Lu. 2019. CoMID: Context-Based Multiinvariant Detection for Monitoring Cyber-Physical Software. IEEE Transactions on Reliability, 69, 1 (2019), 106–123. https://doi.org/10.1109/TR.2019.2933324

[47]

Ragunathan Rajkumar, Insup Lee, Lui Sha, and John Stankovic. 2010. Cyber-physical systems: the next computing revolution. In Design automation conference. 731–736. https://doi.org/10.1145/1837274.1837461

Digital Library

[48]

Lucio R Ribeiro and Neusa Maria F Oliveira. 2010. UAV autopilot controllers test platform using Matlab/Simulink and X-Plane. In 2010 IEEE Frontiers in Education Conference (FIE). S2H–1. https://doi.org/10.1109/FIE.2010.5673378

[49]

Mahadev Satyanarayanan. 2001. Pervasive computing: Vision and challenges. IEEE Personal communications, 8, 4 (2001), 10–17. https://doi.org/10.1109/98.943998

[50]

Bosch Sensortec. 2015. Digital pressure sensor. May 7th.

[51]

Thierry Sotiropoulos, Hélène Waeselynck, Jérémie Guiochet, and Félix Ingrand. 2017. Can Robot Navigation Bugs Be Found in Simulation? An Exploratory Study. In 2017 IEEE International Conference on Software Quality, Reliability and Security, QRS 2017, Prague, Czech Republic, July 25-29, 2017. IEEE, 150–159. https://doi.org/10.1109/QRS.2017.25

[52]

Lin Tan, Chen Liu, Zhenmin Li, Xuanhui Wang, Yuanyuan Zhou, and Chengxiang Zhai. 2014. Bug characteristics in open source software. Empirical software engineering, 19, 6 (2014), 1665–1705. https://doi.org/10.1007/s10664-013-9258-8

Digital Library

[53]

PX4 Dev Team. 2017. Open Source for Drones-PX4 Pro Open Source Autopilot.

[54]

N VANITHA and G PADMAVATHI. [n.d.]. Enhanced Security Mechanisms for the Selected Cyber Attacks in UAV Networks.

[55]

Mark Weiser. 1999. The Computer for the 21^st Century. SIGMOBILE Mob. Comput. Commun. Rev., 3, 3 (1999), July, 3–11. issn:1559-1662 https://doi.org/10.1145/329124.329126

Digital Library

[56]

W Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A survey on software fault localization. IEEE Transactions on Software Engineering, 42, 8 (2016), 707–740. https://doi.org/10.1109/TSE.2016.2521368

Digital Library

[57]

Guanping Xiao, Zheng Zheng, Beibei Yin, Kishor S. Trivedi, Xiaoting Du, and Kai-Yuan Cai. 2019. An Empirical Study of Fault Triggers in the Linux Operating System: An Evolutionary Perspective. IEEE Trans. Reliab., 68, 4 (2019), 1356–1383. https://doi.org/10.1109/TR.2019.2916204

[58]

Tao Xie. 2006. Augmenting automatically generated unit-test suites with regression oracle checking. In European Conference on Object-Oriented Programming. 380–403. https://doi.org/10.1007/11785477_23

Digital Library

[59]

Wenhua Yang, Yepang Liu, Chang Xu, and Shing-Chi Cheung. 2015. A survey on dependability improvement techniques for pervasive computing systems. Science China Information Sciences, 58, 5 (2015), 1–14. https://doi.org/10.1007/s11432-015-5300-3

[60]

Yanbing Yu, James Jones, and Mary Jean Harrold. 2008. An empirical study of the effects of test-suite reduction on fault localization. In 2008 ACM/IEEE 30th International Conference on Software Engineering. 201–210. https://doi.org/10.1145/1368088.1368116

Digital Library

[61]

Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, and Lu Zhang. 2018. An empirical study on TensorFlow program bugs. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. 129–140. https://doi.org/10.1145/3213846.3213866

Digital Library

[62]

Xi Zheng and Christine Julien. 2015. Verification and validation in cyber physical systems: Research challenges and a way forward. In 2015 IEEE/ACM 1st International Workshop on Software Engineering for Smart Cyber-Physical Systems. 15–18. https://doi.org/10.1109/SEsCPS.2015.11

Digital Library

Cited By

Hassan MSalvador JSantu SRahman A(2024)State Reconciliation Defects in Infrastructure as CodeProceedings of the ACM on Software Engineering10.1145/36607901:FSE(1865-1888)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660790
Birchler CMohammed TRani PNechita TKehrer TPanichella S(2024)How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?Proceedings of the ACM on Software Engineering10.1145/36437681:FSE(929-950)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643768
De Santana TNeto PDe Almeida EAhmed I(2024)Bug Analysis in Jupyter Notebook Projects: An Empirical StudyACM Transactions on Software Engineering and Methodology10.1145/364153933:4(1-34)Online publication date: 18-Apr-2024
https://dl.acm.org/doi/10.1145/3641539
Show More Cited By

Index Terms

An exploratory study of autopilot software bugs in unmanned aerial vehicles
1. General and reference
  1. Cross-computing tools and techniques
    1. Empirical studies
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

An evaluative review of the VTOL technologies for unmanned and manned aerial vehicles
Abstract
VTOL (Vertical Take-Off and Landing) capabilities are desired features of both UAVs (Unmanned Aerial Vehicles) and MAVs (Manned Aerial Vehicles) on condition that a comparable flight performance is achieved. VTOL is not only a very ...
Unmanned aerial vehicles

As unmanned aerial vehicles have become more affordable, their popularity with the general public and commercial organisations has seen significant growth in recent years. Whilst remaining a device for both the hobbyist and aircraft-enthusiast to enjoy, ...
Integral Backstepping Control of an Unconventional Dual-Fan Unmanned Aerial Vehicle

In this paper, a dynamic model of vertical take-off and landing (VTOL) aerial vehicles, having lateral and longitudinal rotor tilting mechanism, is first developed using a Newton---Euler formulation. Then an integral backstepping (IB) control technique ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 2021

1690 pages

ISBN:9781450385626

DOI:10.1145/3468264

General Chairs:
Diomidis Spinellis
Athens University of Economics and Business, Greece
,
Georgios Gousios
Facebook, Netherlands / Delft University of Technology, Netherlands
,
Program Chairs:
Marsha Chechik
University of Toronto, Canada
,
Massimiliano Di Penta
University of Sannio, Italy

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 August 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Artifacts Available / v1.1

Author Tags

Qualifiers

Research-article

Conference

ESEC/FSE '21

Sponsor:

SIGSOFT

ESEC/FSE '21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 23 - 28, 2021

Athens, Greece

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
662
Total Downloads

Downloads (Last 12 months)166
Downloads (Last 6 weeks)19

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hassan MSalvador JSantu SRahman A(2024)State Reconciliation Defects in Infrastructure as CodeProceedings of the ACM on Software Engineering10.1145/36607901:FSE(1865-1888)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660790
Birchler CMohammed TRani PNechita TKehrer TPanichella S(2024)How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?Proceedings of the ACM on Software Engineering10.1145/36437681:FSE(929-950)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643768
De Santana TNeto PDe Almeida EAhmed I(2024)Bug Analysis in Jupyter Notebook Projects: An Empirical StudyACM Transactions on Software Engineering and Methodology10.1145/364153933:4(1-34)Online publication date: 18-Apr-2024
https://dl.acm.org/doi/10.1145/3641539
Liu DJiang HGuo SChen YQiao L(2024)What's Wrong With Low-Code Development Platforms? An Empirical Study of Low-Code Development Platform BugsIEEE Transactions on Reliability10.1109/TR.2023.329500973:1(695-709)Online publication date: Mar-2024
https://doi.org/10.1109/TR.2023.3295009
Tang HBurns JStrong ALiu Y(2024)A Compiler Framework for Proactive UAV Regulation Enforcement2024 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA57147.2024.10610359(11781-11788)Online publication date: 13-May-2024
https://doi.org/10.1109/ICRA57147.2024.10610359
Lemieux-Mack CLeach KAngstadt K(2024)Sensor Data Transplantation for Redundant Hardware Switchover in Micro Autonomous Vehicles2024 ACM/IEEE 15th International Conference on Cyber-Physical Systems (ICCPS)10.1109/ICCPS61052.2024.00019(135-146)Online publication date: 13-May-2024
https://doi.org/10.1109/ICCPS61052.2024.00019
Khatiri SDi Sorbo AZampetti FVisaggio CDi Penta MPanichella S(2024)Identifying safety–critical concerns in unmanned aerial vehicle software platforms with SALIENTSoftwareX10.1016/j.softx.2024.10174827(101748)Online publication date: Sep-2024
https://doi.org/10.1016/j.softx.2024.101748
Ma CXi NLu DFeng YMa J(2024)CToMP: a cycle-task-oriented memory protection scheme for unmanned systemsScience China Information Sciences10.1007/s11432-023-3865-067:6Online publication date: 21-May-2024
https://doi.org/10.1007/s11432-023-3865-0
Timperley Cvan der Hoorn GSantos ADeshpande HWąsowski A(2024)ROBUST: 221 bugs in the Robot Operating SystemEmpirical Software Engineering10.1007/s10664-024-10440-029:3Online publication date: 23-Mar-2024
https://dl.acm.org/doi/10.1007/s10664-024-10440-0
Sartaj HMuqeet AIqbal MKhan M(2024)Automated system-level testing of unmanned aerial systemsAutomated Software Engineering10.1007/s10515-024-00462-931:2Online publication date: 1-Aug-2024
https://doi.org/10.1007/s10515-024-00462-9
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents