Anunnaki: A Modular Framework for Developing Trusted Artificial Intelligence

For DNNs, uncertainty can arise in both data acquisition and model construction [27]. Uncertainties with respect to the dataset can be due to variability in real-world environments or measurement error/noise. Uncertainties with respect to the model can be due to errors in the model structure, errors in the training procedure, or errors caused by unknown data. Two common trust issues to address when implementing Trusted AI with respect to uncertainty are reliability and robustness [88].

For this work, reliability relates to a DNN’s performance with routine (or known) operating conditions, where a DNN’s output is expected to be consistent with ground truth results [28]. Typical test procedures for DNNs include a cross validation step with test data that is independent and identically distributed with respect to training data. Evaluation metrics are task-dependent. For regression problems, metrics such as mean squared error measure the deviance between a DNN’s output and the ground truth. For classification problems, metrics such as cross entropy and accuracy can be used for evaluation. Furthermore, for object detection problems, where a distinction between false positives and false negatives is of interest, additional metrics such as precision and recall can be considered. Depending on the evaluation metrics chosen, a reliable AI system is expected to correctly interpolate and produce results that are consistent with known training data.

Adversarial detection techniques address uncertainty with respect to the reliability of a DNN [43]. Model inference enables the creation of behavior models to map specific run-time conditions of a software component to corresponding patterns of expected behavior [53]. When leveraged at run time, behavior models can be used to proactively mitigate failures resulting from the use of a DNN expected to fail [38]. Out-of-distribution techniques can also assess a degree of confidence by comparing a DNN’s run-time inputs to its training data distribution [27]. Run-time inputs that are found to fall outside of a DNN’s training distribution can be marked as uncertain and trigger alternative actions to prevent use of the DNN in a potentially hazardous state. Thus, adversarial detection techniques help to ensure DNNs are used only in contexts (sufficiently) comparable to those that have been previously validated.

Robustness relates to a DNN’s performance in the presence of previously unseen (or unknown) operating conditions, where the DNN’s output is expected to be consistent with sufficiently similar known conditions. Deviant conditions may cover any perturbation in input data for an AI system that is either malicious (e.g., jamming or image-spoofing) or inadvertent interference (e.g., environmental phenomena such as rainfall or fog). The discovery of adversarial examples [78] has demonstrated that DNNs can be highly sensitive to human-imperceptible noise and exploited into producing erroneous outputs by targeted data manipulation. Furthermore, research shows that the sensitivity of DNNs to latch onto superficial regularities in training data casts doubt on their ability to generalize to semantically valid abstractions [30]. Robust AI systems are expected to extrapolate and produce correct results for data that is reasonably different from training data.

To address the robustification of DNNs, techniques have been proposed to automatically generate synthetic data for retraining DNNs when real examples of adverse interference are absent (i.e., known unknown phenomena) [44, 80, 90, 94]. Typically, synthetic data of simulated interference is generated by transforming existing real-world data. Naïve techniques generate interference by adding random perturbations to given inputs (i.e., fuzzing) [57]. More sophisticated techniques use search-based methods to uncover interference patterns that maximize certain aspects of the AI system (e.g., neuron coverage, Kullback–Leiber divergence). Retraining DNNs with synthetically augmented data has been demonstrated to improve robustness [38].

A key challenge addressed by this work is how to enable AI systems to operate across distinct domains. In this work, we consider a domain to be a subset of model inputs that share common attributes or characteristics and serve as a means to categorize training samples. A domain D is typically defined by three components: a feature (input) space X, a label (output) space Y, and a probability distribution \(p(x,y)\) [22]. The concept of domain adaptation was introduced by Ben-David et al. [6] in the context of natural language processing with the goal of effectively reusing an existing language model to identify malicious e-mail for a wide range of users. Domain Adaptation is needed when a neural network trained to perform a task on a dataset \(D_s\) (the source domain) may need to perform the same task on dataset \(D_t\) (the target domain) [6, 7]. More recent efforts have adopted domain adaptation as a technique for improving object classification and detection models [82]. Domain adaptation (DA) is closely related to transfer learning [84] but rather than reusing a learned model on a different task, DA reuses a learned model in a different, but closely related, domain. We assume that labels and feature spaces remain consistent across domains. The characteristics of each domain may be described at various levels of abstraction. For an image processing model, we can describe a set of unique domains based on environmental conditions (e.g., rainy, low-light), levels of noise (e.g., peak signal-to-noise ratio metrics), or resulting model behavior. It is important to note that the proposed framework is not limited to the provided examples.

Multi-domain training tools such as Enki [38] aim at increasing model robustness for a given target domain \(D_t\) by generating synthetic data belonging to \(D_t\) and retraining an existing model in this new domain. Enki inputs environmental conditions (e.g., the rain domain) and corresponding contexts (e.g., raindrop positions, appearance) and then uses an evolutionary algorithm to generate a diverse archive of environmental contexts, where diversity is defined with respect to system behavior. Enki can then be used to (i) assess the robustness of a DNN in a given domain, and (ii) robustify the DNN by retraining on the previously generated archive of diverse synthetic dataset.

Domain detection models such as the behavior oracles generated by Enlil [36] map incoming data samples \(x_i\) to a corresponding target domain \(D_t\). Enlil takes environmental conditions, corresponding contexts, and behavior category specifications as input for an evolutionary algorithm to generate diverse archives of environmental contexts for each behavior category. The generated archives can then be used to (i) assess model robustness for a given domain and then (ii) train a behavior oracle that predicts the behavior category of an incoming data sample. The behavior oracle can inform an autonomous system of its current operating context (domain) and enable more informed adaptation decisions (e.g., ensuring the applicability [and utility] of a given LEC for a given operating context).

In order to manage heterogeneous hardware and enable software reuse in robotic applications, many developers implement the control logic of sensors and actuators as components of a robot middleware [19]. The Robot Operating System (ROS) [64] is an open source robot middleware that has been widely adopted by both academia and industry [35]. The fundamental elements of a ROS-based system are nodes, topics, and services. ROS enables the controlling algorithms for a single application to be divided into multiple independent processes (i.e., ROS nodes). ROS nodes can publish/subscribe to data unidirectionally through message buses (i.e., ROS topics) and also handle bidirectional request/reply interactions (i.e., ROS services). As a peer-to-peer network of nodes, a ROS-based system can be implemented over multiple processing units with a common registry service to facilitate communication between nodes (illustrated in Figure 1). Ultimately, ROS enables developers to abstract away from individual robotic components to focus on their software architecture.

In order to facilitate systematic development of ROS-based systems, Malavolta et al. [45] empirically identified a set of guidelines (by data mining ROS projects and surveying experts in the field to identify best practices for developing ROS-based systems) to support developers in applying good design principles to meet quality requirements and mitigate common ROS-specific software problems. Importantly, ROS-based systems that follow these guidelines align well with software engineering principles such as modularity, composability, and reusability.

Early in the development process, requirement engineers must identify and specify the needs and constraints of the target system to be built. Requirement engineers work closely with stakeholders to identify their goals and objectives in order to create a set of requirements that orient and guide the development process to ensure the system being built will meet the needs of relevant parties. Stakeholder goals are often qualitative in nature (e.g., the system operates safely) and difficult to formalize for automatic satisfaction guarantees. Furthermore, specifying the requirements becomes increasingly difficult when working with cyberphysical systems because of the inherent uncertainty present in unknown environments [12].

Goal-oriented requirements engineering techniques such as KAOS [81] have emerged in response to the aforementioned challenges to enable rigorous requirements specification. KAOS provides a goal-based approach to model system objectives and hierarchically decompose high-level goals into leaf-level requirements of the system [36], represented by a directed acyclic graph. Figure 2(a) presents a graphical depiction of objects in the KAOS goal model ecosystem. KAOS Goals are declarative statements describing objectives that the system under consideration should achieve [41]. Goals may exist at various levels of abstraction and the decomposition of high-level goals into lower-level sub-goals is depicted with refinement arrows. As shown in Figure 2(b), KAOS supports two types of refinements, AND-refinement and OR-refinement. A parent goal can only be satisfied if the boolean conjunction of its children evaluates to true in the case of AND-refinement, or if the boolean disjunction of its children evaluates to true in the case of OR-refinement. KAOS also supports obstacles, defined as any behavior or goal that prevents the satisfaction of another goal [41]. Obstacles may be resolved by including resolution goals that provide alternative ways a blocked goal may be achieved given the presence of an obstacle. Previously, we extended KAOS goal modeling to include utility functions [13] (i.e., functions that map system attributes to Boolean or real scalar values [36]) that are attached to goals to map attributes of the goal to quantifiable metrics to enable design- and run-time assessment of system behavior with respect to requirements satisfaction [65]. Finally, KAOS supports agents, represented by white hexagons, and defined as entities responsible for achieving system requirements and overcoming obstacles. Both the human and non-human components of a system can be represented as agents in KAOS. The combination of these entities into a logical structure enables developers to refine high-level goals into low-level requirements and explicitly define the intended behavior and functionality of the system to be built.

One challenge that arises when specifying the requirements of an autonomous system is how to strategically address environmental uncertainty. If requirements are too rigid, then the system may unnecessarily reconfigure and/or enter a failure mode, thereby preventing successful mission completion. To this end, requirements specification languages such as RELAX [86] and FLAGS [4] have been proposed to explicitly account for various sources of uncertainty by adding flexibility to system requirements. Developers can use the RELAX language to formally define and extend existing goal models to account for sources of uncertainty by “relaxing” system requirements through a set of RELAX operators. An overview of the RELAX language and its corresponding definitions are presented in Table 1. The semantics of the RELAX language have been specified in terms of a set of fuzzy logic propositions. Correspondingly, RELAX-ed requirements can be annotated with fuzzy-logic based utility functions in the KAOS goal model [65]. While KAOS obstacles are useful for identifying what factors may cause a goal to become violated, RELAX enables a developer to specify the tolerable impact of uncertainty on requirements satisficement, rather than identifying the specific causes/sources of uncertainty.

The concept of autonomic computing has become more commonly used with increasing system complexity in deployed software systems that must operate continuously, even under uncertain conditions [34]. Systems comprising numerous interconnected components can be difficult to configure and maintain. Autonomic computing proposes that such systems should manage themselves according to high-level objectives provided by system administrators [34]. These self-managing systems commonly use a feedback controller (i.e., autonomic manager) to observe and adapt managed components of the larger system [8]. Figure 3 illustrates a common realization of an autonomic manager called the Monitor-Analyze-Plan-Execute over a Knowledge base (MAPE-K) loop [34]. A MAPE-K loop comprises steps to monitor system components, analyze the system state, plan what adaptive actions need to be taken to maintain optimal performance, and execute the plan to realize the corresponding system reconfiguration. Adaptation tactics are methods for adaptation [14]. Each tactic has pre and post-conditions and a set of actions to realize an adaptation [14]. A shared knowledge base acts as a repository for any data that can inform each MAPE-K step (e.g., adaptation goals, tactics). For autonomous systems, a MAPE-K controller can automate system adaptations to achieve optimal performance in response to changing environments.

Anunnaki requires goal models described in the KAOS [81] format to specify the expected system requirements. This section describes the development and run-time monitoring of KAOS goal models to address the robustness and resilience of LESs using the Anunnaki framework and its aggregate services.

In this work, we consider a system to be resilient if it can mitigate different sources of uncertainty to maintain safe behavior [40]. To this end, the Anunnaki framework leverages model inference and behavior models of an LEC to support domain detection (Figure 4, Steps 1 and 2). Adverse interference can include any malicious noise or environmental phenomena that result in undesirable behavior from an LEC. Behavior models are used to predict the impact of adverse conditions absent from existing training/validation data, thus enabling the Anunnaki framework to prevent the use of LECs under conditions they would normally perform unreliably (e.g., poor lighting conditions). As an abstract service, the Domain Detection service (Figure 4, Step 1) can implement any behavior modeling technique that takes raw sensor data and detects the presence of adverse interference.

One example model inference method for domain detection is Enlil [39], that constructs behavior models of an LEC by assessing the impact of various environmental phenomena within an external simulator. Enlil generates a behavior model that can be executed, independent of the LEC as a behavior oracle. One or more behavior oracles can run in parallel to the managed AI system and subscribe to the same sensor data received by managed LECs. As sensor data is received, behavior oracles output behavior assessments (Figure 4, Step 1), which include both a perceived context for any apparent adversarial noise and an inferred behavior category to summarize the impact of the adversarial noise. As adversarial detection services, behavior oracles publish behavior assessments to any other subscribing service, thus enabling the Anunnaki framework to detect and respond to adverse run-time conditions.

To address the robustness of LECs, the Anunnaki framework can use robustified alternate learning models, created through any multi-domain training techniques, such as adversarial training, or using data synthetically augmented to include adverse phenomena (e.g., rain, fog). For example, Enki is a method proposed for robustifying LECs to known unknown adverse environmental phenomena [38]. Using Enki, robust learning models are generated by running a simulator to uncover examples of adverse phenomena that lead to a diverse array of behavior patterns for the given LEC. The diverse collection of adversarial examples are then used to retrain the default learning model [38].

At run time, a Learning Model Manager service (Figure 4, Step 2) enables the managed AI system’s LECs to swap default learning models with alternative, robustified learning models created through adversarial training (e.g., Enki). This service-oriented approach enables separate learning models to be robustified with respect to specific forms of adverse interference, and applicable learning models can be swapped in based on the behavior oracle’s assessment of run-time contexts. When no adverse interference is detected, the default learning model is activated. By decoupling the problem of robustification from a single learning model to separate, independent learning models, the Anunnaki framework enables more flexibility to the developers on what forms of adverse phenomena are addressed by any given implementation of the managed AI system. Furthermore, this approach enables developers to maintain and augment specific context-dependent models without needing to retrain and validate the base learning model. For example, if rainy environments are a concern for an LEC, an additional robustified learning model can be provided to the Anunnaki framework at run time to handle rain without needing to retrain/validate the default learning model. Furthermore, additional robustified models can be created for alternative phenomena (e.g., foggy weather, poor lighting) that are also independent from each other and the default learning model. Thus, the Anunnaki framework provides a modular and composable solution to robustifying LECs.

To monitor and control the managed AI system, the Anunnaki framework uses Utu to monitor, analyze, and reconfigure the use of LECs in response to uncertain environmental conditions (Figure 4, Step 3). In order to mitigate faults from the use of LECs in untrusted conditions, Utu assesses the run-time state of the managed AI system and issues reconfiguration requests in response to the run-time environment. Namely, Utu follows the MAPE-K model for autonomic management that comprises five separate services to support run time decision-making; see Figure 8 for a high-level description of each of these services. The remainder of this section describes how Anunnaki uses the Utu services to ensure run-time goal satisfaction.

Run-time Goal Monitoring. Utu inputs a KAOS goal model (Figure 8, Knowledge Manager Service) to analyze utility functions [16] associated with each goal and obstacle with logic propositions [70] to support run-time monitoring of goal model satisfaction (Figure 8, Monitor Service). For example, the utility function “A1.buzzer == true” is attached to goal G14. Thus, when the “buzzer” attribute of agent A1 is set to true, the goal G14 is evaluated as satisfied. Through the use of utility functions, the Anunnaki framework can interpret a KAOS goal model as a logic tree of run-time system checks to determine the satisfaction of high-level system objectives (Figure 8, Analyze Service). For example, Figure 9 shows a logic tree interpretation of the KAOS goal model in Figure 6. The Anunnaki framework also extends goal models by enabling message channels to be associated with each agent to specify which channels each respective agent publishes state data. For example, the message channel “/utu/oracle/output” is attached to agent A4, indicating that attributes for the behavior oracle can be monitored by observing the corresponding message channel. These extensions enable developers to map the same goal model to different platforms by simply redefining the associated message channels and system attributes.

Run-time Mitigation. Utu also takes a predefined set of tactics to determine what actions should be taken to mitigate faults resulting from violated goal models [14] (Figure 8, Plan Service). Because system objectives and tactics are not hard-coded into Utu, but, instead, are model-driven, the Anunnaki framework can be deployed with alternative goals and adaptation tactics by simply re-instantiating Utu at design time, with new goal models and tactics. Figure 10 shows an example adaptation tactic [14], specified in an Extensible Markup Language (XML) format. Tactics are defined with a set of preconditions, actions, and postconditions. In the given example, a “fail-safe” tactic is defined with a precondition to trigger when G3 in Figure 6 is found to be unsatisfied. For the example fail-safe tactic, the actions are to (1) request a mode-change to “manual” mode for the rover and (2) e-mail a notification to the user. Finally, a postcondition is given in the example to state that goal G3 is expected to be satisfied upon execution of the given actions. Thus, when a reconfiguration is needed due to goal violations, Utu realizes the specific actions defined by the corresponding adaptation tactic (Figure 8, Execute Service) to ensure continued goal satisfaction at run time.

This section describes how the Anunnaki framework has been implemented for the autonomous rover presented in Section 3. First, we describe the hardware and software used in the rover case study. Next, we present results obtained during the implementation, instantiation, and execution of the Anunnaki framework (see Figure 4) and how they pertain to increased robustness to environmental uncertainty. Finally, we outline the implementation of the Utu autonomic manager and how the aggregate components of Anunnaki are integrated to provide a robust and resilient learning-enabled autonomous system.

To further illustrate how Anunnaki supports modularity, reusability, and extendability when developing learning-enabled autonomous systems, we present a second empirical study implementing the Anunnaki framework for use in an unmanned aerial vehicle (UAV). The new operating domain necessitates a new set of safety requirements, a different LEC architecture, and a different goal model when compared with the autonomous rover. To assess Anunnaki in a UAV, we implemented a simulation environment and UAV controller using the Webots autonomous vehicle simulator [49]. A 3D render of the UAV model used in our demonstration is provided in Figure 16. The UAV’s sensors include a forward facing camera, altitude meter, and onboard GPS. The UAV can be controlled both manually and autonomously. A comprehensive overview of differences in Anunnaki framework instantiations and customizations for the respective applications (e.g., DNN type, hardware) is presented in Table 3, where the shaded rows indicate application differences.

When operating autonomously, the UAV relies on computer vision to perceive its environment via an onboard camera. In contrast to the autonomous rover, the UAV’s object detector is implemented as a YOLOv5 [31] DNN architecture. Previously, Zhan et al. [92] used this architecture successfully for onboard UAV image detection. YOLOv5 offers faster inference times than other single-shot detection models such as RetinaNet [79], thus making YOLOv5 a popular choice for real-time object detection on mobile devices with limited resources. The greater inference speed comes at the cost of lower overall accuracy due to localization errors. To train and validate the UAV’s object detector, we use the professional-grade VisDrone-2019 dataset [96]. This dataset consists of 6,471 training images with 343,205 object labels; 548 validation images with 38,800 labels; and 1,060 unlabeled test images. Each labeled object belongs to one of the following 10 classes: pedestrian, people, bicycle, car, van, truck, tricycle, awning-tricycle, bus, and motor. We use the pretrained weights provided by the open source YOLOv5 implementation [31] and train our model until convergence. When evaluated against the unseen validation sample, the model correctly identified labeled bounding boxes for the 10 classes with a precision of \(60.6\%\), recall of \(62.5\%\), and an F-score of \(61.3\%\).

A goal of this work was to explore how fundamental principles of software design (e.g., modularity, composability, reusability) [5, 32] can be used to develop tools/techniques to address trusted AI concerns. A key feature of modular systems is their flexibility and ability to manage complexity and uncertainty [3], both of which apply when attempting to assess and improve robustness and resilience of LESs. Namely, a modular system comprises hierarchical units that are well-defined, have high cohesion (internal interconnectedness), and have low coupling (units are independent of other units) [3]. Additionally, previous work in service-oriented architecture [52, 83] has demonstrated how decomposing a monolithic application into smaller individual services that can be developed, monitored, and reconfigured independently, has led to greater system modularity. By constructing the Anunnaki framework following key tactics outlined by Bass et al. [5], Johnson et al. [32], and Baldwin et al. [3] (e.g., low-coupling, service-oriented architecture, interoperability), the Anunnaki framework provides a core set of application-independent configurable services with support for future modifications, thereby making it well aligned with the definition of modular architecture [3]. In additional to the structure of a software system, Baldwin et al. explain that modular software should implicitly support a set of actions or operations that are unique to modular systems; these include: substitution, augmentation, and porting. We further explore how Anunnaki supports each of these modular operations to address trusted AI concerns next. Specifically, Anunnaki targets two dimensions of trust: robustness and resilience, both of which can be addressed in a variety of different ways for different sources of uncertainty. In our first demonstration, the autonomous terrestrial rover’s goal model (see Figure 6) used standard utility functions [65] to inform the autonomic manager of goal violations. However, when greater flexibility to sources of uncertainty was needed for the autonomous UAV, a new microservice was added to Anunnaki to support the RELAX language (applying modularity’s support for augmentation and substitution), thus enabling run-time management of an LES with a goal model containing RELAX-ed goals (see Figure 18). Additionally, we have shown how developers can reuse Anunnaki services to address assurance concerns for different sources of uncertainty. Namely, Anunnaki enabled run-time adaptations (e.g., swapping the active learning model, executing a fail-safe tactic) for an autonomous terrestrial rover exposed to poor lighting conditions and for an autonomous UAV exposed to fog conditions. We anticipated the autonomous terrestrial rover might encounter poor lighting conditions and leveraged the Enlil domain detection service to improve system resilience to this phenomenon. To implement a similar service for a completely different use case (i.e., autonomous UAV), we only needed to change Enlil parameters (e.g., behavior category specification, weather phenomena specifications, model and dataset addresses), thus saving a significant amount of development time (benefiting from modularity’s support for porting). This modification illustrated the low-coupling of the domain detection module as the implemented changes did not impact the majority of the framework. The Anunnaki model-driven framework and its aggregate microservices facilitated modular operations to improve LES robustness to interference and resilience with respect to known unknown sources of uncertainty (i.e., lighting variations, rain, fog conditions). Therefore, we have shown that a modular approach can be used to support the automated assessment and improvement of the robustness and resilience of LESs (RQ1).

A system that can be used for developing and run-time managing LESs for diverse applications and their respective data sets without needing to rebuild/retrain the entire system is considered data and model-agnostic [87, 93]. To this end, we have shown how the Anunnaki framework enables run-time adaptation in response to adverse environmental conditions for two managed AI systems deployed for two different applications. In the first case study, we leveraged the Enki microservice to generate specialized RetinaNet models with increased robustness to HSL lighting variations for our custom real-world dataset. When the autonomous rover encountered adverse lighting conditions, Utu’s autonomic manager used an Enlil behavior oracle to detect LEC degradation and reconfigure the system to execute a fail-safe procedure. In the second case study, we demonstrated how Enki was used to generate specialized YoloV5 models with increased robustness to fog conditions for the VisDrone dataset. We also demonstrated how Utu used an Enlil behavior oracle to detect the presence of LEC failure and then automatically reconfigured the managed UAV to apply a fail-safe procedure preventing potential mission failure. Anunnaki supported the generation of robustified models and the detection of the operating domain for each set of uncertainties through isolated configuration changes to corresponding microservices. Therefore, we have shown that Anunnaki is data and model-agnostic as each system’s LEC relied on a different set of learning models with distinct DNN architectures and different datasets (RQ2).

Many previous works propose techniques to address trustworthy AI concerns such as robustness [58, 60, 89, 95] and/or resilience [24, 53] with respect to environmental uncertainty. For example, DeepCert [58] uses formally defined environmental contexts and image perturbation levels to verify contextually-dependent DNN robustness. DeepCert further supports the selection of the best DNN from a set of developer-provided models for a given operational context. Given the large variety of tools that can support trusted AI concerns (e.g., Enki [38], Enlil [39], DeepRoad [95], DeepXplore [60], BESTEST [24]), Anunnaki is intended as a loosely-coupled collection of services rather than a fixed set of hard-coded tools. For example, to instantiate alternative services (e.g., replace Enki with DeepCert), developers need only configure the appropriate interface (e.g., published ROS messages) for the new service. Each service-type supported by Anunnaki can be interchanged with alternative techniques, to meet evolving stakeholder requirements, without requiring (potentially extensive) architecture or code changes.

Several existing works propose configurable frameworks that use AI to support adaptive systems. For example, Weyns et al. [85] propose an architecture that uses AI to model sources of environmental uncertainty, a MAPE-K loop and user-provided goals to make adaptation decisions, and control theory to realize the selected adaptations. Specifically, Weyns et al. use DNNs to build models of sources of environmental uncertainty to enable adaptation decisions that are relevant to the current operating context. Caldas et al. [9] use AI to first optimize an algorithm that searches over a system’s adaptation space and then use AI to generate controller configurations that can realize the discovered adaptations. Jamshidi et al. [29] propose a technique that uses a simulator to generate a range operating contexts and then uses transfer learning to train a performance model that predicts an adaptive system’s performance in a given context. They show how the performance model can instantiate a knowledge base in a MAPE-K loop to inform context-dependent configuration changes. The learned performance model may be considered an implicit behavior oracle, and could potentially be used for domain detection in an instantiation of Anunnaki. Table 4 provides an overview of the specific differences between the related frameworks, where rows correspond to frameworks, columns represent features of the frameworks, and checkmarks indicate if the framework supports the respective feature. First, Table 4a overviews the role of AI in each framework. Next, Table 4b overviews design-time services supported by each framework, including the use of hierarchical goal models to represent both functional and non-functional system requirements and inform system adaptations (column D4.), and the use of optimization algorithms to generate adaptation configurations (column D5.). Model Robustification (column D3.) indicates a framework’s support for generating alternative robustified learning models to address different sources of uncertainty, which is a distinguishing feature of Anunnaki. Finally, Table 4c overviews run-time services supported by each framework. Model Inference (column R1.) indicates support for a service that can assess a managed LEC’s behavior and output behavioral assessments (e.g., perceived operating context, behavior category) to inform adaptation decisions. Model Management (column R2.) indicates support for switching the active learning model (e.g., switching the default learning model to a context-specific robustified model) in response to environmental uncertainty. Online Learning (column R3.) indicates support for incremental training or updating of LECs from incoming data in an online (i.e., during run-time) setting [74]. Quantifying Functional Goals (column R4.) signifies a framework’s ability to evaluate functional goal satisficement (e.g., via utility functions). A key difference between existing approaches and this work is the role of AI. Specifically, while previous work focuses on using AI to support one or more steps of the adaptation process for self-adaptive systems, this work focuses on using modularity and other foundational software engineering principles to address assurance for self-adaptive systems that contain one or more AI components.

We consider several threats to the validity of our study as outlined in this section.

External. For the demonstration of the resource delivery UAV, we rely on several external sources. These include the open source YOLOv5 model, VisDrone dataset, and the Webot simulator and corresponding UAV model. We have reviewed the source code of the YOLOv5 model to confirm its implementation follows the theoretical architecture outlined in previous research [31, 92]. The dataset used for training has been used for validating state-of-the-art techniques and model training competitions [17], both of which make it useful as a benchmark dataset for aerial object detection. Likewise, the Webots simulator has been used for numerous robotics applications [15, 67].⁵ As such, we believe that the external software used poses a minimal risk to the validity of the obtained results.

Simulation. We note that there exists a reality gap between the simulated environments (i.e., Webots environment, Enki-generated environmental contexts and the physical world. The Webots simulator is well-regarded as a robotics testing platform enabling us to demonstrate a proof-of-concept case study. We argue that any physical inconsistencies do not directly impact the results presented in this work. Additionally, our demonstration of the autonomous rover provides an implementation of the proposed framework in the physical world. We acknowledge that simulated phenomena, such as those produced by Enki, may differ from their appearance in the physical world. While the simulated contexts for environmental phenomena may not perfectly reflect the real world, previous work has shown that they can provide significant insights to developers leading to a more targeted approach of real-world data collection [38]. Finally, we note that Enki is an instantiation of an Anunnaki service rather than a core feature of the framework. Development teams may use a different tool for multi-domain training and doing so will not impact the architecture of the proposed framework.)

Stochastic Variations. The results obtained from learning-enabled services are subject to variations due to stochastic functions present in their implementations. It is important to note that this work does not seek to promote a specific learning-based technique but draws from are well studied in previous research and implemented as-provided for our demonstrations. Any refactoring of these components was conducted through a configurable wrapper class to integrate the various framework services with platform-dependent technologies and support seamless communication while keeping the underlying architecture unaltered.

Computational Complexity. In our demonstration of the Anunnaki framework, we rely on configurable services (i.e., domain detection and multi-domain training) that generate/use context-specific models (e.g., model robustified for rain conditions) to address LES robustness and resilience for different sources of uncertainty. Naturally, developers may want to robustify an LES with respect to combinations of operating contexts (e.g., model robustified to fog and rain and lighting variations) which may lead to an exponential increase in the number of models, training/memory costs, and complexity of goal model/adaptation specifications. While Enki was used to study context composition in previous work [38], there are still concerns regarding scalability of such an approach as more contexts are considered. By using evolutionary algorithms that can effectively explore complex spaces, services such as Enki and Enlil can mitigate some concerns regarding algorithmic complexity. Moreover, the Anunnaki framework enables developers to specify how many sources of uncertainty are considered and the granularity of the search procedures. Nonetheless, as the number of sources of uncertainty increase and the granularity of information increases, computational challenges increase. We will explore techniques to address this limitation in future work.