skip to main content
10.1145/3468264.3468534acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article
Public Access

Hazard analysis for human-on-the-loop interactions in sUAS systems

Published: 18 August 2021 Publication History

Abstract

With the rise of new AI technologies, autonomous systems are moving towards a paradigm in which increasing levels of responsibility are shifted from the human to the system, creating a transition from human-in-the-loop systems to human-on-the-loop (HoTL) systems. This has a significant impact on the safety analysis of such systems, as new types of errors occurring at the boundaries of human-machine interactions need to be taken into consideration. Traditional safety analysis typically focuses on system-level hazards with little focus on user-related or user-induced hazards that can cause critical system failures. To address this issue, we construct domain-level safety analysis assets for sUAS (small unmanned aerial systems) applications and describe the process we followed to explicitly, and systematically identify Human Interaction Points (HiPs), Hazard Factors and Mitigations from system hazards. We evaluate our approach by first investigating the extent to which recent sUAS incidents are covered by our hazard trees, and second by performing a study with six domain experts using our hazard trees to identify and document hazards for sUAS usage scenarios. Our study showed that our hazard trees provided effective coverage for a wide variety of sUAS application scenarios and were useful for stimulating safety thinking and helping users to identify and potentially mitigate human-interaction hazards.

References

[1]
2009. Interaction Design for Situation Awareness-Eyetracking and Heuristics for Control Centers.
[2]
A. Agrawal, S. Abraham, B. Burger, C. Christine, L. Fraser, J. Hoeksema, S. Hwang, E. Travnik, S. Kumar, W. Scheirer, J. Cleland-Huang, M. Vierhauser, R. Bauer, and S. Cox. 2020. The Next Generation of Human-Drone Partnerships: Co-Designing an Emergency Response System. In Proc. of the 2020 Conf. on Human Factors in Computing Systems.
[3]
A. Agrawal, S.Khoshmanesh, M. Vierhauser, M. Rahimi, J. Cleland-Huang, and R. R. Lutz. 2019. Leveraging artifact trees to evolve and reuse safety cases. In Proc. of the 41st Int’l Conf. on Software Engineering. 1222–1233.
[4]
Ankit Agrawal, Jan-Philipp Steghöfer, and Jane Cleland-Huang. 2020. Model-Driven Requirements for Humans-on-the-Loop Multi-UAV Missions. In Proc. of the 10th Int’l Model-Driven Requirements Engineering WS.
[5]
Ardupilot. 2020. Ardupilot – open source autopilot software. https://ardupilot.org [Last accessed 01-06-2021].
[6]
Ardupilot. 2020. MissionPlanner. https://ardupilot.org/planner [Last accessed 01-06-2021].
[7]
Aviation Safety Reporting System. 2021. ASRS Database Report Set: Unmanned Aerial Vehicle (UAV) Reports (ACN: 1599671). https://asrs.arc.nasa.gov/docs/rpsts/uav.pdf [Last accessed 01-06-2021].
[8]
Paul Baybutt. 2012. Process hazard analysis for phases of operation in the process life cycle. Process Safety Progress, 31, 3 (2012), 279–281.
[9]
Francesco N Biondi, Monika Lohani, Rachel Hopman, Sydney Mills, Joel M Cooper, and David L Strayer. 2018. 80 MPH and out-of-the-loop: Effects of real-world semi-automated driving on driver workload and arousal. In Proc. of the Human Factors and Ergonomics Society Annual Meeting. 62, 1878–1882.
[10]
Robin E. Bloomfield and Kateryna Netkachova. 2014. Building Blocks for Assurance Cases. In Proc. of the 25th IEEE Int’l Symp. on Software Reliability Engineering Workshops. 186–191.
[11]
CBS SF Bay Area News Outlet. 2015. Pilot Of Drone That Nearly Hit CHP Helicopter Says It Was On Autopilot. https://sanfrancisco.cbslocal.com/2015/12/17/drone-near-miss-chp-helicopter-martinez-owen-ouyang-apology-autopilot [Last accessed 01-06-2021].
[12]
Jane Cleland-Huang and Ankit Agrawal. 2020. Human-Drone Interactions with Semi-Autonomous Cohorts of Collaborating Drones. In Proc. of the Interdisciplinary WS on Human-Drone Interaction; co-located with the 2020 ACM CHI Conf. on Human Factors in Computing Systems.
[13]
Jane Cleland-Huang, Ankit Agrawal, Md Nafee Al Islam, Eric Tsai, Maxime Van Speybroeck, and Michael Vierhauser. 2020. Requirements-Driven Configuration of Emergency Response Missions with Small Aerial Vehicles. In Proc. of the 24th ACM Conf. on Systems and Software Product Lines. 1–12.
[14]
J. Cleland-Huang, A. Agrawal, M. Vierhauser, and C. Mayr-Dorn. 2021. Visualizing Change in Agile Safety-Critical Systems. IEEE Software, 38, 03 (2021), May, 43–51. issn:1937-4194
[15]
Jane Cleland-Huang, Mats Per Erik Heimdahl, Jane Huffman Hayes, Robyn R. Lutz, and Patrick Maeder. 2012. Trace Queries for Safety Requirements in High Assurance Systems. In Proc. of the Int’l Working Conf. on Requirements Engineering: Foundation for Software Quality. 179–193.
[16]
Jane Cleland-Huang, Michael Vierhauser, and Sean Bayley. 2018. Dronology: an incubator for cyber-physical systems research. In Proc. of the 40th Int’l Conf. on Software Engineering: New Ideas and Emerging Results. 109–112.
[17]
Nancy J. Cook. 2007. Stories of Modern Technology Failures and Cognitive Engineering Successes. CRC Press, 2007.
[18]
Josh Dehlinger and Robyn R. Lutz. 2006. PLFaultCAT: A Product-Line Software Fault Tree Analysis Tool. Autom. Softw. Eng., 13, 1 (2006), 169–193.
[19]
Ewen Denney and Ganesh Pai. 2014. Automating the assembly of aviation safety cases. IEEE Transactions on Reliability, 63, 4 (2014), 830–849.
[20]
Ewen Denney and Ganesh Pai. 2015. Argument-based airworthiness assurance of small UAS. In Proc. of the 2015 IEEE/AIAA 34th Digital Avionics Systems Conf. 5E4–1.
[21]
Ewen Denney and Ganesh Pai. 2016. Composition of safety argument patterns. In Proc. of the Int’l Conf. on Computer Safety, Reliability, and Security. 51–63.
[22]
Ewen Denney and Ganesh Pai. 2016. Safety considerations for UAS ground-based detect and avoid. In Proc. of the IEEE/AIAA 35th Digital Avionics Systems Conf. 1–10.
[23]
Ewen Denney, Ganesh Pai, and Josef Pohl. 2012. Heterogeneous Aviation Safety Cases: Integrating the Formal and the Non-formal. In Proc. of the 17th IEEE Int’l Conf. on Engineering of Complex Computer Systems. 199–208.
[24]
Ewen Denney, Ganesh J. Pai, and Ibrahim Habli. 2015. Dynamic Safety Cases for Through-Life Safety Assurance. In Proc. of the 37th IEEE/ACM Int’l Conf. on Software Engineering. 587–590.
[25]
Ewen Denney, Ganesh J. Pai, and Iain Whiteside. 2017. Modeling the Safety Architecture of UAS Flight Operations. In Proc. of the 2017 Int’l Conf. on Computer Safety, Reliability, and Security.
[26]
Homayoon Dezfuli, Allan Benjamin, Christopher Everett, Martin Feather, Peter Rutledge, Dev Sen, and Robert Youngblood. 2015. NASA System Safety Handbook. Volume 2: System Safety Concepts, Guidelines, and Implementation Examples.
[27]
Mica R. Endsley. 2011. Designing for Situation Awareness: An Approach to User-Centered Design, Second Edition (2nd ed.). CRC Press, Inc., Boca Raton, FL, USA. isbn:1420063553, 9781420063554
[28]
Davide Falessi, Shiva Nejati, Mehrdad Sabetzadeh, Lionel Briand, and Antonio Messina. 2011. SafeSlice: a model slicing and design safety inspection tool for SysML. In Proc. of the 19th ACM SIGSOFT Symp. and the 13th European Conf. on Foundations of Software Engineering. 460–463.
[29]
Martin S. Feather and Lawrence Z. Markosian. 2013. Architecting and generalizing a safety case for critical condition detection software: an experience report. In Proc. of the 1st Int’l WS on Assurance Cases for Software-Intensive Systems. 29–33.
[30]
K. Fellah and M. Guiatni. 2019. Tactile Display Design for Flight Envelope Protection and Situational Awareness. IEEE Transactions on Haptics, 12, 1 (2019), Jan, 87–98. issn:1939-1412
[31]
Qian Feng and Robyn R. Lutz. 2005. Bi-directional safety analysis of product lines. Journal of Systems and Software, 78, 2 (2005), 111–127.
[32]
Donald Firesmith. 2004. Engineering Safety Requirements, Safety Constraints, and Safety-Critical Requirements. Journal of Object Technology, 3, 3 (2004), 27–42.
[33]
Markus Funk. 2018. Human-drone interaction: Let’s get ready for flying user interfaces!. Interactions, 25, 3 (2018), 78–81.
[34]
Matthew C. Gombolay, Anna Bair, Cindy Huang, and Julie A. Shah. 2017. Computational design of mixed-initiative human-robot teaming that considers human factors: situational awareness, workload, and workflow preferences. I. J. Robotics Res., 36, 5-7 (2017), 597–617.
[35]
Richard Hawkins, Ibrahim Habli, and Tim Kelly. 2013. Principled Construction of Software Safety Cases. In Proc. of the Next Generation of System Assurance Approaches for Safety-Critical Systems WS of the 32nd Int’l Conf. on Computer Safety, Reliability and Security.
[36]
Andreas Holzinger. 2005. Usability engineering methods for software developers. Commun. ACM, 48, 1 (2005), 71–74.
[37]
Teton Valley News Julia Tellman. 2018. First-ever recorded drone-hot air balloon collision prompts safety conversation. https://www.postregister.com/news/local/first-ever-recorded-drone-hot-air-balloon-collision-prompts-safety/article_7cc41c24-6025-5aa6-b6dd-6d1ea5e85961.html [Last accessed 01-06-2021].
[38]
Shuanglong Kan. 2014. Traceability and model checking to support safety requirement verification. In Proc. of the 22nd ACM SIGSOFT Int’l Symp. on Foundations of Software Engineering. 783–786.
[39]
Leah Kaufman and Brad Weed. 1998. Too much of a good thing?: Identifying and resolving bloat in the user interface. In Proc. of the CHI 98 Conf. Summary on Human Factors in Computing Systems. 207–208.
[40]
Tim Kelly and Rob Weaver. 2004. The goal structuring notation–a safety argument notation. In Proc. of the dependable systems and networks 2004 workshop on assurance cases. 6.
[41]
Tim P Kelly and John A McDermid. 2001. A systematic approach to safety case maintenance. Reliability Engineering & System Safety, 71, 3 (2001), 271–284.
[42]
L.T. Kohn, J.M. Corrigan, and M.s. Donaldson. 1999. To err is human, Building a safety health system. Washington, DC: National Academy Press.
[43]
Yasuhiro Kuriki and Toru Namerikawa. 2014. Consensus-based cooperative formation control with collision avoidance for a multi-UAV system. In Proc. of the 2014 American Control Conf. 2077–2082.
[44]
Nancy G. Leveson. 1995. Safeware, System Safety and Computers. Addison Wesley. isbn:0-201-11972-2
[45]
Mai Skjott Linneberg and Steffen Korsgaard. 2019. Coding qualitative data: A synthesis guiding the novice. Qualitative Research Journal.
[46]
Yuanna Liu and Hao Lu. 2019. A strategy of multi-UAV cooperative path planning based on CCPSO. In Proc. of the 2019 IEEE Int’l Conf. on Unmanned Systems. 328–333.
[47]
Robyn R. Lutz and Robert M. Woodhouse. 1997. Requirements Analysis Using Forward and backward Search. Ann. Software Eng., 3 (1997), 459–475.
[48]
Joanna McGrenere, Ronald M. Baecker, and Kellogg S. Booth. 2002. An Evaluation of a Multiple Interface Design Solution for Bloated Software. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems. ACM, 164–170. isbn:1-58113-453-3
[49]
D.C. Nagel. 1998. Human error in aviation Operations. Human factors in Aviation, E.L.Weiner and E.C.Nagel (Eds), 263–303.
[50]
Saeid Nahavandi. 2017. Trusted autonomy between humans and robots: toward human-on-the-loop in robotics and autonomous systems. IEEE Systems, Man, and Cybernetics Magazine, 3, 1 (2017), 10–17.
[51]
Jakob Nielsen and Thomas K Landauer. 1993. A mathematical model of the finding of usability problems. In Proc. of the INTERACT’93 and CHI’93 Conf. on Human Factors in Computing Systems. 206–213.
[52]
E Onal, C Craddock, and Mica Endsley. 2013. From Theory to Practice: How Designing for Situation Awareness Can Transform Confusing, Overloaded Shovel Operator Interfaces, Reduce Costs, and Increase Safety. In Proc. of the Int’l Symp. on Automation and Robotics in Construction.
[53]
Parsif.al. 2021. Tool support for Systematic Literature Reviews. https://parsif.al [Last accessed 01-06-2021].
[54]
PX4. 2021. Open Source Flight Controller. https://px4.io [Last accessed 01-06-2021].
[55]
QGroundControl. 2021. Ground Control Station. http://qgroundcontrol.com [Last accessed 01-06-2021].
[56]
Nicolas Regis, Frédéric Dehais, Emmanuel Rachelson, Charles Thooris, Sergio Pizziol, Mickaël Causse, and Catherine Tessier. 2014. Formal Detection of Attentional Tunneling in Human Operator-Automation Interactions. IEEE Transactions Human-Machine Systems, 44, 3 (2014), 326–336.
[57]
Donald J. Reifer. 1979. Software Failure Modes and Effects Analysis. IEEE Transactions on Reliability, R-28,3 (1979), 247–249.
[58]
Mehrdad Sabetzadeh, Shiva Nejati, Lionel Briand, and Anne-Heidi Evensen Mills. 2011. Using SysML for modeling of safety-critical software-hardware interfaces: Guidelines and industry experience. In Proc. of the IEEE 13th Int’l Symp. on High-Assurance Systems Engineering. 193–201.
[59]
Neil R. Storey. 1996. Safety Critical Computer Systems. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA. isbn:0201427877
[60]
Tim Claudius Stratmann and Susanne Boll. 2016. Demon Hunt - The Role of Endsley’s Demons of Situation Awareness in Maritime Accidents. In Proc. of the Int’l Working Conf. on Human Error, Safety, and System Development. Springer, 203–212.
[61]
Kevin J. Sullivan, Joanne Bechta Dugan, and David Coppit. 1999. The Galileo Fault Tree Analysis Tool. In Digest of Papers: FTCS-29, The Twenty-Ninth Annual Int’l Symp on Fault-Tolerant Computing. IEEE Computer Society, 232–235.
[62]
Hongyu Sun, Miriam Hauptman, and Robyn R. Lutz. 2007. Integrating Product-Line Fault Tree Analysis into AADL Models. In Proc. of the 10th IEEE Int’l Symp. on High Assurance Systems Engineering. IEEE Computer Society, 15–22.
[63]
Dante Tezza and Marvin Andujar. 2019. The State-of-the-Art of Human–Drone Interaction: A Survey. IEEE Access, 7 (2019), 167438–167454. issn:2169-3536
[64]
Christopher D Wickens and Amy L Alexander. 2009. Attentional tunneling and task management in synthetic vision displays. The International Journal of Aviation Psychology, 19, 2 (2009), 182–199.
[65]
Xueyi Zou, Rob Alexander, and John McDermid. 2016. Testing method for multi-uav conflict resolution using agent-based simulation and multi-objective search. Journal of Aerospace Information Systems, 13, 5 (2016), 191–203.

Cited By

View all
  • (2024)Self-Adaptation of Loosely Coupled Systems across a System of Small Uncrewed Aerial SystemsProceedings of the 12th ACM/IEEE International Workshop on Software Engineering for Systems-of-Systems and Software Ecosystems10.1145/3643655.3643882(37-44)Online publication date: 14-Apr-2024
  • (2024)Human–machine Teaming with Small Unmanned Aerial Systems in a MAPE-K EnvironmentACM Transactions on Autonomous and Adaptive Systems10.1145/361800119:1(1-35)Online publication date: 14-Feb-2024
  • (2024)HIFuzz: Human Interaction Fuzzing for Small Unmanned Aerial VehiclesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642958(1-14)Online publication date: 11-May-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
August 2021
1690 pages
ISBN:9781450385626
DOI:10.1145/3468264
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 August 2021

Permissions

Request permissions for this article.

Check for updates

Badges

Author Tags

  1. Human-sUAS interaction
  2. hazard analysis
  3. sUAS
  4. safety analysis

Qualifiers

  • Research-article

Funding Sources

  • LIT
  • NSF

Conference

ESEC/FSE '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)195
  • Downloads (Last 6 weeks)36
Reflects downloads up to 05 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Self-Adaptation of Loosely Coupled Systems across a System of Small Uncrewed Aerial SystemsProceedings of the 12th ACM/IEEE International Workshop on Software Engineering for Systems-of-Systems and Software Ecosystems10.1145/3643655.3643882(37-44)Online publication date: 14-Apr-2024
  • (2024)Human–machine Teaming with Small Unmanned Aerial Systems in a MAPE-K EnvironmentACM Transactions on Autonomous and Adaptive Systems10.1145/361800119:1(1-35)Online publication date: 14-Feb-2024
  • (2024)HIFuzz: Human Interaction Fuzzing for Small Unmanned Aerial VehiclesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642958(1-14)Online publication date: 11-May-2024
  • (2024)Coupled Requirements-Driven Testing of CPS: From Simulation to Reality2024 IEEE 32nd International Requirements Engineering Conference (RE)10.1109/RE59067.2024.00040(337-344)Online publication date: 24-Jun-2024
  • (2023)Automated Identification and Qualitative Characterization of Safety Concerns Reported in UAV Software PlatformsACM Transactions on Software Engineering and Methodology10.1145/356482132:3(1-37)Online publication date: 26-Apr-2023
  • (2023)A Requirements-Driven Platform for Validating Field Operations of Small Uncrewed Aerial Vehicles2023 IEEE 31st International Requirements Engineering Conference (RE)10.1109/RE57278.2023.00013(29-40)Online publication date: Sep-2023
  • (2023)Engineering Challenges for AI-Supported Computer Vision in Small Uncrewed Aerial Systems2023 IEEE/ACM 2nd International Conference on AI Engineering – Software Engineering for AI (CAIN)10.1109/CAIN58948.2023.00033(158-170)Online publication date: May-2023
  • (2023)DroneReqValidator: Facilitating High Fidelity Simulation Testing for Uncrewed Aerial Systems DevelopersProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE56229.2023.00011(2082-2085)Online publication date: 11-Nov-2023
  • (2023) GRuM — A flexible model-driven runtime monitoring framework and its application to automated aerial and ground vehiclesJournal of Systems and Software10.1016/j.jss.2023.111733203:COnline publication date: 1-Sep-2023
  • (2022)Towards Real-Time Safety Analysis of Small Unmanned Aerial Systems in the National AirspaceAIAA AVIATION 2022 Forum10.2514/6.2022-3540Online publication date: 15-Jun-2022
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media