# Runtime Architecture and Task Plan Co-Adaptation for Autonomous Robots with
Metaplan

Jeroen M. Zwanepol∗, Gustavo Rezende Silva∗ and Carlos Hernández Corbato∗ ∗
Delft University of Technology, Delft, The Netherlands
<EMAIL_ADDRESS>{g.rezendesilva, c.h.corbato}@tudelft.nl;This
work was supported by the European Union’s Horizon 2020 Framework Programme
through the MSCA network REMARO (Grant Agreement No 956200).

###### Abstract

Autonomous robots need to be able to handle uncertainties when deployed in the
real world. For the robot to be able to robustly work in such an environment,
it needs to be able to adapt both its architecture as well as its task plan.
Architecture adaptation and task plan adaptation are mutually dependent, and
therefore require the system to apply runtime architecture and task plan co-
adaptation. This work presents Metaplan, which makes use of models of the
robot and its environment, together with a PDDL planner to apply runtime
architecture and task plan co-adaptation. Metaplan is designed to be easily
reusable across different domains. Metaplan is shown to successfully perform
runtime architecture and task plan co-adaptation with a self-adaptive unmanned
underwater vehicle exemplar, and its reusability is demonstrated by applying
it to an unmanned ground vehicle.

##  I Introduction

While operating, autonomous robots are subject to uncertainties, both from
internal system factors (e.g., sensor failures) and external environment
factors (e.g., unexpected obstacles). To overcome these uncertainties, robots
can be designed as self-adaptive systems [1], enabling them to adapt at
runtime their architecture (i.e., deactivating or activating software
components and changing their parameters) and their mission execution (i.e.,
change the task being performed). However, as stated by Cámara et al. [2] the
mutual dependencies between architecture and task plan adaptation pose a
significant challenge when determining how to apply both adaptations in a
coordinated manner. Thus, this paper focuses on designing a solution for
runtime architecture and task plan co-adaptation (RATPA).

In the context of self-adaptive robotic systems, several works have addressed
either task adaptation, architecture adaptation, or both adaptations
independently [3, 4, 5, 6, 7], but only few works apply RATPA [2, 8]. Both
these works define the co-adaptation problem and propose solutions for it,
however, the solutions presented are not general and reusable for different
robotic applications.

This paper proposes Metaplan, a planning based approach that uses off-the
shelf general reasoners and knowledge about the robot’s architecture and
mission to drive runtime architecture and task plan co-adaptation. Metaplan
combines Metacontrol [3], a knowledge-based framework for architecture self-
adaptation, with a Planning Domain Definition Language (PDDL)-based planner
[9] used for simultaneous architecture and task planning, ensuring the
selection of a suitable configuration for each task. This results in a general
and modular solution that is only dependent on: (1) a PDDL formulation of the
robot’s task planning problem, and (2) an architectural model of the robot
conforming to Metacontrol’s metamodel. This explicit dependency on models
makes Metaplan reusable across different applications.

Metaplan is implemented as a ROS 2-based system leveraging MROS [10, 11] and
PlanSys2 [12, 13], a ROS implementation of Metacontrol and a package for
handling plan generation and execution in ROS 2 respectively. It was evaluated
with two different types of robots, an Unmanned Underwater Vehicle (UUV) [14],
and an Unmanned Ground Vehicle (UGV) [2]. The experiments showed that Metaplan
is able to perform RATPA in robotic systems, and that it is general and
reusable across different applications with minimal overhead.

In summary, the contributions of this paper are:

  * •

A reusable approach for runtime architecture and task plan co-adaptation in
robotic systems with minimal overhead;

  * •

Experimental results demonstrating runtime architecture and task plan co-
adaptation in the SUAVE exemplar;

##  II Related Works

In the context of self-adaptive robotic systems, there are multiple works that
have addressed either architecture adaptation or task adaptation [3, 4, 15],
independent architecture and task adaptation [6, 7], or coordinated task and
architecture adaptation [2, 8]. Table I presents an overview of the related
works.

As a solution for architecture adaptation, Hernández et al. [16] proposed
Metacontrol, a knowledge-based framework that incorporates systems with the
ability to adapt their architecture at runtime. An important aspect of
Metacontrol is that it uses knowledge of how the system is designed to reason
when and how the system should adapt. This design choice increases
Metacontrol’s reusability, since it is only required to modify the knowledge
base in order to apply it to different applications. Metacontrol was
demonstrated with different robotics applications, such as mobile base
navigation and underwater vehicles [3, 16, 17, 18, 19].

Purandare et al. [7] proposed the CICADA framework to enable Unmanned Aerial
Vehicles (UAVs) to recover from misconfiguration-related flight instability.
This is achieved by adapting the UAVs’ flight controller configuration at
runtime to a safe baseline configuration or by adapting the task it is
performing. Although CICADA is able to perform both architecture and task
adaptation, they are performed independently. In relation to reusability,
CICADA was specifically designed to handle misconfiguration-related flight
instability in UAVs, and does not generalize to other applications.

Valner et al. [6] proposed the TeMoto software architecture to address the
challenges of dynamic task and resource management. Similarly to CICADA,
TeMoto is able to handle architecture and task adaptation independently, but
not in a coordinated manner. In terms of reusability, TeMoto is task
independent, but it requires the adaptation logic to be manually implemented
for each managed resource.

RATPA was first addressed by Braberman et al. [8] with the MORPH reference
architecture. Although MORPH is proposed as a solution for architecture and
task plan co-adaptation, it is only demonstrated on a conceptual level. This
makes it difficult to evaluate the feasibility of applying MORPH to robotic
systems at runtime, and to compare it to other methods for robotics
adaptation.

Cámara et al. [2] present a framework for architecture and task co-adaptation
based on probabilistic model checking. Their approach consists of using an
explicit model of the robot with a probabilistic model checker to synthesize
all possible reconfiguration plans, including their associated cost, and to
find the most suitable task plan for the robot, including the required
reconfiguration plans. Some limitations of their approach are that task
planning does not take into account that an action might become unavailable
when there is no architectural configuration available to solve it.
Furthermore, their task planning solution is specific for navigation on a
graph-like map, thus applying it to a new task planning scenario requires
extensive changes.

In the current state of the art, as far as the authors are aware, the only
solution available for architecture and task plan co-adaptation is the one
proposed by Cámara et al. [2]. However, their solution is not general enough
to be applied to different applications without modifications. Therefore, this
work aims to fill this gap by proposing Metaplan.

TABLE I: Related adaptation frameworks

Approach | Adaptation type |  Robotics Middleware | Reusable
---|---|---|---
Architecture | Task | RATPA
Metacontrol [16] | Yes | X | X | ROS, ROS 2 | Yes
CICADA [7] | Yes | Yes | X | ROS | X
TeMoto [6] | Yes | Yes | X | ROS | Yes
MORPH [8] | Yes | Yes | Yes | N/A | X
Cámara et al. [2] | Yes | Yes | Yes | ROS | X
Metaplan | Yes | Yes | Yes | ROS 2 | Yes

##  III Metacontrol Background

Metacontrol adheres to the principle of separation of concerns for self-
adaptive systems into a managed subsystem and a managing subsystem [1]. Where
the managed subsystem is responsible for domain concerns, in this case, the
robotic application, and the managing subsystem is responsible for adapting
the managed subsystem. Its reasoning cycle follows the MAPE-K loop [20]. It
monitors the relevant parameters of the managed subsystem and the environment,
analyzes whether the managed subsystem configuration fulfills its
requirements, reconfigures when necessary. All these steps are performed in
interaction with its knowledge base (KB).

Metacontrol’s KB conforms to the Teleological and Ontological Model for
Autonomous Systems (TOMASys) metamodel [16]. In short, TOMASys captures
engineering knowledge of how the system is designed, its possible
configuration alternatives, and how attributes of the managed system or
environment relate to the selection of a configuration. A simplified version
of TOMASys with the required concepts for this work can be seen in Figure 1.
The design time elements are specified by the designer of the adaptation
logic, and they are static throughout the system operation. The runtime
elements are dynamically specified by the system during its operation.
Throughout this work, TOMASys elements will be written with _this font_.

Figure 1: TOMASys metamodel

The design time elements of TOMASys are: _function_ , _function design_ ,
_component_ , _qualitty attribute_ , and _environment attribute_. A _function_
represents a functionality of the managed subsystem, e.g., navigation. A
_function design_ is a particular solution for a _function_ , in other words,
each _function design_ represents a possible configuration that can be
selected by Metacontrol, e.g., a particular navigation algorithm. A
_component_ represents a hardware or software component of the system that is
required by a _function design_ , e.g., a sonar. A _quality attribute_
represents an attribute of the system, e.g., battery level. And a _environment
attribute_ represents an attribute of the environment, e.g., water visibility.
Specific _quality attribute_ or _environment attribute_ values can be set as
requirements for _function designs_ , e.g., a certain navigation algorithm can
only be used if the battery level is above $50\%$.

The runtime elements of TOMASys are: _objective_ and _function grounding_. An
_objective_ represents the _functions_ of the system that are currently
required, e.g., while performing a navigation task the robot has an
_objective_ of the navigation _function_ type. A _function grounding_
represents the current _function design_ that is deployed to solve an
_objective_.

In addition to the architectural model, the KB contains rules that are
reasoned over at runtime to evaluate the status of the system and decide when
adaptation is required, i.e., a rule indicating that a _function grounding_ is
in error when the measured value of a _quality attribute_ violates its
requirement.

The architectural model is implemented using the Ontology Web Language (OWL)
[21]. The rules to evaluate configurations are expressed with the Semantic Web
Rule Language (SWRL) [22], and the rules are verified with the Pellet reasoner
[23].

##  IV Metaplan

Metaplan is designed to perform the role of a managing subsystem. Metaplan
assumes that the managed subsystem is component-based and task oriented. The
former means that the manged subsystem is composed of multiple loosely coupled
software components that can be activated or deactivated at runtime. The
latter means that the managing subsystem is designed to handle tasks, which
are realized by a sequence of actions. The managed subsystem is designed to
accomplish these actions by activating the corresponding components.

The architecture of Metaplan is depicted in Figure 2. Its main components are
the Planner and the Metacontrol Reasoner. The Metacontrol Reasoner is
responsible for analyzing runtime information to decide when the robot needs
to adapt, which configurations are currently available, and executing the
reconfigurations selected by the Planner by activating and deactivating
Function nodes of the managed subsystem. The Planner is responsible for
planning the actions of the robot and its respective required configurations,
and informing Metacontrol about the selected reconfiguration plans. The
planner is based on PDDL, and its planning problem is defined in the Domain
and Problem files.

To reach its goal state, Metaplan generates a task plan consisting of a
sequence of actions, and selects the required configuration to accomplish each
action. These actions correspond to one or more _functions_ of the robot
expressed in the KB, and the configurations for each action correspond to the
selected _function designs_. When an action finishes execution, the next
action of the task plan starts execution.

Whenever a _function design_ becomes (un)available or an _objective_ is in
error, Metaplan generates a new task plan. This allows for Metaplan to adapt
the system configuration using the updated available _function designs_ , and
could adapt the task plan by generating a different plan, thereby achieving
runtime architecture and task plan co-adaptation.

The remainder of this section details Metaplan’s architecture, the design time
activities required to configure Metaplan for a robotic scenario, and its
runtime behavior.

Figure 2: System Architecture of Metaplan. The system is split into a section
containing the managed subsystem and a section containing the managing
subsystem.

###  IV-A Design time activities

To apply Metaplan to a specific application, it is necessary to create a KB
containing an architectural model of the robot, and define the robot task
planning problem with PDDL in a Domain and Problem file.

####  IV-A1 Knowledge Base

The KB captures information about the robot’s architecture and its variants,
it is modeled conforming to the TOMASys metamodel (see section III). The KB is
reusable for different applications within the same domain, that is, for the
same robot with the same architecture in a similar environment.

####  IV-A2 Domain File

The Domain File defines which actions are available, and their preconditions
and effects. The Domain File contains a generic description of the actions the
robotic system is able to perform, which conform to the _functions_ defined in
the KB. This means that the Domain File can be used for different uses within
the same domain, i.e., a Domain File containing a description of a mobile
robot used in a hospital environment can be used for multiple missions for
that same robot in the same environment.

An important design requirement for each action is that it needs a
precondition stating which _functions_ it requires, and that there need to be
_function designs_ available to solve these _functions_. The PDDL formulation
of these requirements can be seen in Listing 1. Assume action1 is one of the
actions the system can perform. The action would first need to be linked to an
action object (line 1). This action object requires a _function_ (line 2).
This _function_ is solved by a _function design_ (line 3), which is available
(line 4). This allows the PDDL planner to select an action to be executed,
together with a suitable configuration.

Listing 1: PDDL Required Action Definitions

⬇

1: (at start (action1 ?a))

2: (at start (a_req_f ?a ?f))

3: (at start (fd_solve_f ?fd ?f))

4: (at start (fd_available ?fd))

####  IV-A3 Problem File

The Problem File defines the initial state and goal state of the planning
problem. This definition is specific to the mission of the robot and only
applies within this context.

An important design requirement for Metaplan is information on which actions
require which _functions_. The PDDL formulation of this requirement can be
seen in Listing 2. The object specifying action1 (line 1) and which
_functions_ it requires (line 2) need to be created by the user.

Listing 2: PDDL Required Problem Definitions

⬇

1: (action1 a1)

2: (a_req_f a1 f1)

###  IV-B Runtime behavior

This section explains the runtime behavior of the components of Metaplan
depicted in Figure 2. Upon startup, the information from the KB, the Domain
File, and the Problem File is loaded by the corresponding nodes.

The Metaplan loop starts by monitoring the _environment_ and _quality
attributes_. This information is passed to the Metacontrol reasoner. The
Metacontrol Reasoner loop can be seen in Algorithm 1. First (line 1), the
Metacontrol Reasoner updates the Knowledge Base with information on the
_environment_ and _quality attributes_ received from the Observer nodes. Next
(line 2), the KB is analyzed with automatic reasoning using the KB rules to
find which _function designs_ are available and whether any _objectives_ are
in error. If the available _function designs_ have changed since the last
iteration of the loop or there are _objectives_ in error, then the new
_function designs_ are sent to the Planner and a new plan is requested (lines
3-5). Lastly (line 6), the old _function designs_ are updated with the new
_function designs_ for the next iteration.

A plan request triggers the planner to generate a new plan. Algorithm 1 gives
a step-by-step overview of how this is done. First (line 1), the Problem File
(PF) is updated with the _function designs_ that are available. Afterwards
(line 2), the Planner generates an action plan using the Problem File and
Domain File (DF) consisting of a list of actions in sequence. These actions
are then executed one by one, until the whole action plan is finished (lines
3-5).

An action is requested together with information on which _function designs_
(i.e., which configuration) should be used to solve the _functions_ required
by the action. At the start of execution of an action, the _objectives_ and
_function groundings_ of the Metacontrol Reasoner are updated. This, in turn,
triggers the managed subsystem to use the Function nodes corresponding to the
required _functions_ and to use the updated _function groundings_.

Algorithm 1 Metacontrol Reasoner

1:$KB\leftarrow$ Update($KB,QA,EA$)

2:$FD_{new},O_{error}\leftarrow$ Analyze($KB$)

3:if $FD_{new}\neq FD_{old}$ or $O_{error}\neq\emptyset$ then

4: RequestPlan($FD_{new}$)

5:end if

6:$FD_{old}\leftarrow FD_{new}$

Algorithm 2 Planner

1:$PF$ $\leftarrow$ Update($PF,FD_{new}$)

2:$A\leftarrow$ Plan($DF,PF$)

3:for $a$ in $A$ do

4: RequestAction($a$)

5:end for

##  V Evaluation

Two experiments were performed to demonstrate the reusability and feasibility
of Metaplan. The first experiment consisted of using Metaplan to design a
solution for the UGV scenario described by Cámara et al. [2]. The goal of this
experiment was to demonstrate the reusability and minimal overhead required
for Metaplan following the design activities presented in subsection IV-A. The
second experiment consisted of applying Metaplan to a simulated unmanned
underwater vehicle performing a pipeline inspection mission. The goal of this
experiment was to demonstrate the feasibility to use Metaplan with robots.

###  V-A UGV scenario

In this scenario, the UGV has to navigate from an initial position to a goal
position in the least amount of time and as safely and efficiently as
possible. During its operation, the robot encounters obstacles and changing
lighting conditions, to which it needs to adapt its architecture and task
plan.

Figure 3: Adapted map of the UGV scenario

The robot has three distinct algorithms for localization: AMCL, MRPT, and
Aruco. Each of these algorithms require input from different sensors, AMCL and
MRPT require input from a lidar or a kinect, and the Aruco algorithm requires
camera input. However, the camera can only be used in low light conditions in
combination with a flashlight.

The robot navigates in a graph-like environment, with vertices representing
intersections, and edges representing corridors. The robot knows the layout of
the environment, and which corridors are dark or contain obstacles. A
simplified version of the map of the environment can be seen in Figure 3. Note
that the corridor between wp6 and wp9 is dark, and the corridor between wp2
and wp5 contains obstacles. At runtime, the robot needs to decide which paths
to take and which configuration to use for each path.

The setup for the original experiment is not publicly available, thus this
work is only simulated at the task and configuration level.

####  V-A1 Design time activities

To apply Metaplan to this case, the design activities described in subsection
IV-A are followed, and they are described below.

##### Knowledge Base

The only task the UGV is able to perform is navigating from point A to B, and
the only variability available is which localization algorithm in combination
with which sensors to use. Thus, only a localization _function_ called
_f_localization_ is included in the ontology. Each combination of the
available localization algorithms with the available components is represented
as a distinct _function design_ requiring specific _components_ , as can be
seen in the first two columns of Table II.

Each _function design_ has an expected battery usage that is represented as a
_quality attribute_ named _QA_battery_usage_. Furthermore, each _function
design_ requires a minimum level of the _environment attributes_ that
represent how safe a path is and the lighting condition of the environment,
respectively, _EA_safety_ and _EA_light_. The specific values for the
_quality_ and _environment attributes_ associated to each _function design_
can be seen in the last three columns of Table II.

TABLE II: UGV Knowledge Base

Function Design |  Required Components | QA | EA
---|---|---|---
| Expected
---
Battery
Usage
| Required
---
Safety
| Required
---
Light
fd_AMCL_lidar | c_lidar | 4 | 0.4 | 0
fd_AMCL_kinect | c_kinect | 2 | 1.0 | 0
fd_MRPT_lidar | c_lidar | 6 | 0.3 | 0
fd_MRPT_kinect | c_kinect | 4 | 0.9 | 0
fd_aruco | c_camera | 7 | 0.7 | 0
fd_aruco_with_light |  | c_camera &
---
c_flashlight
10 | 0.7 | 1

##### Domain File

Three distinct move actions are defined, one for each type of corridor, e.g.,
move, move_with_obstacle, and move_dark.

##### Problem File

The Problem File defines the initial state and goal state of the robot. It
also contains information about all corridors between two waypoints and the
type of each corridor, e.g., dark corridor or corridor with obstacles.
Furthermore, the problem file also contains information on which _functions_
the move actions require, the current location of the robot, the battery level
of the robot, and the goal location of the robot.

####  V-A2 Results

The robot Metaplan updates the availability of the _function designs_ for each
corridor comparing the _EA_light_ and _EA_safety_ levels specified in the KB
(Table II) with the required levels corresponding to the environment map
(Figure 3). The measured _EA_light_ is 0 for the dark corridors and 1 for the
other corridors, and the measured _EA_safety_ is 0.8 for the corridors with
obstacles and 0 for the other corridors. Note that the measured safety
indicates the level of safety required to pass through that corridor, so
Metaplan needs to select a _function design_ with an _EA_safety_ of more than
0.8 to pass through those corridors.

For this experiment, the battery level is initially set to 100% and the robot
has to navigate from wp1 to wp9. The first plan generated by the robot is
presented on the left side of Figure 4. The robot selects _fd_amcl_kinect_ ,
since enough battery is available to select this _function design_. Afterward,
the robot encounters a corridor with obstacles and does not need to adapt its
configuration, since _fd_amcl_kinect_ meets the required safety levels.

When the robot reaches wp2, _fd_amcl_kinect_ and _fd_mrpt_kinect_ are set to
be unavailable to simulate a failure of _c_kinect_. This requires the robot to
adapt its task plan, since no _function designs_ are available for navigating
through an obstacle corridor. Furthermore, the robot is also required to adapt
its architecture, since _fd_amcl_kinect_ is unavailable. This results in the
task plan presented on the right side of Figure 4. Metaplan successfully finds
a new plan avoiding the obstacle corridor and passing through wp3 instead, and
adapts its configuration to _fd_amcl_lidar_. After this, the robot can fulfil
the rest of the mission without needing replanning.

Figure 4: Initial plan (left) and adapted plan after kinect failure at wp2
(right). Move action, route, and function design have changed as a result.

The reusability of Metaplan can be demonstrated with the following
modifications of the UGV scenario.

##### Modification 1

Suppose that the UGV’s initial position, goal position, or the map layout is
changed. In comparison to the original scenario, only the PDDL problem file
needs to be modified by changing the variables corresponding to the initial
position, goal position, or map layout.

##### Modification 2

Suppose that the move action in a corridor obstacle is altered to use more
battery. In comparison to the original scenario, only the PDDL domain file
needs to be modified by changing the values of the required battery level
precondition and the battery consumption effect in the move action for the
corridor with obstacles.

##### Modification 3

Suppose that the implementation of the MRPT localization algorithm is
improved, thus its required safety levels can be changed. In comparison to the
original scenario, only the KB needs to be modified by changing the values of
the required _EA_safety_ for _fd_mrpt_kinect_ and _fd_mrpt_lidar_.

###  V-B UUV pipeline inspection

This scenario consists of a UUV that is deployed to inspect a pipeline as fast
as possible, while ensuring it has enough battery available to complete the
inspection. The UUV has a set of actions available with which it can achieve
its mission. These actions are search pipeline, follow pipeline, and recharge.
If at any time the battery falls below a critical level, the UUV needs to go
to the charging station and recharge. Each action requires a different set of
_functions_ and therefore different configurations of the UUV.

The UUV needs to be able to handle two types of uncertainties: changing water
visibility and a sudden drop in battery level. This requires the UUV to adapt
both its architecture, to use less battery, and its task plan, to go to the
charging station before fulfilling the rest of the mission.

The UUV is able to configure its speed to low, medium, or high. A high speed
allows the UUV to reach its goal state faster, however, it consumes more
energy. The robot is also able to adjust the height at which it searches for
the pipeline. There are three different heights the UUV can search at, namely
low, medium, and high. A high search height allows the robot to detect the
pipeline faster, however the UUV will not be able to see the pipeline if the
water visibility is low.

####  V-B1 Results

At the beginning of execution of the UUV’s mission, an initial plan specifying
a sequence of actions and required configurations was generated, which can be
seen on the left side of Figure 5. The initial plan requires the UUV to first
search for the pipeline and, when found, follow the pipeline. When a simulated
sudden drop in battery level is inserted, the UUV is required to adapt and
generates a new plan, which can be seen on the right side of Figure 5. The UUV
changes its configuration by adjusting its speed to low and changes its plan
by first going to the recharge station.

This experiment was successfully executed in simulation using the SUAVE
exemplar [14], demonstrating the feasibility of applying Metplan at runtime to
a ROS 2-based robotic systems. A video of the simulation showcasing the RATPA
behavior can be seen here.

Figure 5: Initial plan (left) and adapted plan after sudden battery level drop
(right). Action and function design have changed as a result.

##  VI Conclusions and Future Works

This work presented Metaplan as a reusable solution for architecture and task
plan co-adaptation. Metaplan uses a PDDL-based planner for task and
configuration planning, and Metacontrol to capture, analyze, and manage
architectural knowledge. This resulted in a general and reusable framework
since it only depends on a PPDL formulation of the robot’s task planning
problem, and in an architectural model of the robot conforming to the TOMASys
metamodel, as detailed in subsection IV-A. The experiments performed provided
further evidence to the reusability claim by applying Metaplan to two distinct
robotic applications, and demonstrated the feasibility of applying it to
robotic systems at runtime.

Future works include: evaluating Metaplan’s time performance and how it scales
in more complex scenarios; capturing the dependency between tasks and
functionalities in TOMASys instead of in the PDDL formulation, to make it more
general and reusable.

## References

  * [1] D. Weyns, An Introduction to Self-adaptive Systems: A Contemporary Software Engineering Perspective.  John Wiley & Sons, 2020.
  * [2] J. Cámara, B. Schmerl, and D. Garlan, “Software architecture and task plan co-adaptation for mobile service robots,” in Proceedings of the IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, pp. 125–136, 2020.
  * [3] E. Aguado, Z. Milosevic, C. Hernández, R. Sanz, M. Garzon, D. Bozhinoski, and C. Rossi, “Functional self-awareness and metacontrol for underwater robot autonomy,” Sensors, vol. 21, no. 4, p. 1210, 2021.
  * [4] M. Molina and P. Santamaria, “Behavior coordination for self-adaptive robots using constraint-based configuration,” arXiv preprint arXiv:2103.13128, 2021.
  * [5] B. H. Cheng, R. J. Clark, J. E. Fleck, M. A. Langford, and P. K. McKinley, “Ac-ros: assurance case driven adaptation for the robot operating system,” in Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, pp. 102–113, 2020.
  * [6] R. Valner, V. Vunder, A. Aabloo, M. Pryor, and K. Kruusamäe, “Temoto: A software framework for adaptive and dependable robotic autonomy with dynamic resource management,” IEEE Access, 2022.
  * [7] S. Purandare, U. Sinha, M. N. A. Islam, J. Cleland-Huang, and M. B. Cohen, “Self-adaptive mechanisms for misconfigurations in small uncrewed aerial systems,” in 2023 IEEE/ACM 18th Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS), pp. 169–180, 2023.
  * [8] V. Braberman, N. D’Ippolito, J. Kramer, D. Sykes, and S. Uchitel, “Morph: A reference architecture for configuration and behaviour self-adaptation,” in Proceedings of the 1st international workshop on control theory for software engineering, pp. 9–16, 2015.
  * [9] M. Ghallab, A. Howe, C. Knoblock, D. Mcdermott, A. Ram, M. Veloso, D. Weld, and D. Wilkins, “PDDL—The Planning Domain Definition Language,” 1998.
  * [10] C. H. Corbato, D. Bozhinoski, M. G. Oviedo, G. van der Hoorn, N. H. Garcia, H. Deshpande, J. Tjerngren, and A. Wasowski, “Mros: Runtime adaptation for robot control architectures,” arXiv preprint arXiv:2010.09145, 2020.
  * [11] “Mros package.” https://github.com/meta-control/mc_mros_reasoner.  Accessed: 2023-09-13.
  * [12] F. Martín, J. G. Clavero, V. Matellán, and F. J. Rodríguez, “Plansys2: A planning system framework for ros2,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9742–9749, IEEE, 2021.
  * [13] “Plansys2 package.” https://github.com/PlanSys2/ros2_planning_system.  Accessed: 2023-09-13.
  * [14] G. R. Silva, J. Päßler, J. Zwanepol, E. Alberts, S. L. T. Tarifa, I. Gerostathopoulos, E. B. Johnsen, and C. H. Corbato, “Suave: An exemplar for self-adaptive underwater vehicles,” in 2023 IEEE/ACM 18th Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS), pp. 181–187, 2023.
  * [15] Y. Carreno, J. S. Willners, Y. R. Petillot, and R. P. Petrick, “Situation-aware task planning for robust auv exploration in extreme environments,” in Proceedings of the IJCAI Workshop on Robust and Reliable Autonomy in the Wild, 2021.
  * [16] C. Hernández, “Model-based self-awareness patterns for autonomy,” 2013.
  * [17] C. Hernández, J. Bermejo-Alonso, and R. Sanz, “A self-adaptation framework based on functional knowledge for augmented autonomy in robots,” Integrated Computer-Aided Engineering, vol. 25, no. 2, pp. 157–172, 2018.
  * [18] C. Hernandez Corbato, Z. Milosevic, C. Olivares, G. Rodriguez, and C. Rossi, “Meta-control and self-awareness for the ux-1 autonomous underwater robot,” in Iberian Robotics conference, pp. 404–415, Springer, 2019.
  * [19] D. Bozhinoski, M. G. Oviedo, N. H. Garcia, H. Deshpande, G. van der Hoorn, J. Tjerngren, A. Wasowski, and C. H. Corbato, “Mros: runtime adaptation for robot control architectures,” Advanced Robotics, vol. 36, no. 11, pp. 502–518, 2022.
  * [20] P. Horn, “Autonomic computing: Ibm’s perspective on the state of information technology,” 2001.
  * [21] G. Antoniou and F. van Harmelen, Web Ontology Language: OWL, pp. 67–92.  Berlin, Heidelberg: Springer Berlin Heidelberg, 2004.
  * [22] A. SWRL, “Semantic web rule language combining owl and ruleml,” W3C Member Submission (May 21, 2004), http://www. w3. org/Submission/SWRL/(last visited March 2011), 2004.
  * [23] E. Sirin, B. Parsia, B. C. Grau, A. Kalyanpur, and Y. Katz, “Pellet: A practical owl-dl reasoner,” Journal of Web Semantics, vol. 5, no. 2, pp. 51–53, 2007.


from infinite temperature. Furthermore, when $\delta=0$, the Hamiltonian is
particle-hole symmetric, leading to a different kind of universal behavior,
detailed in Section F.1. In Fig. 1(e), we use dephasing error
$\\{\sigma^{z}_{j}\\}$ at a total error rate of $\gamma=0.05$, in a $5\times
5$ square grid of Rydberg atoms (with total Hilbert space dimension of 55447).

##  Appendix F Limitations of benchmarking protocol: Failure cases

As discussed in the main text, our benchmarking formula can fail when: 1) the
$k$-th no resonance condition is not satisfied, 2) the effective dimension
$D_{\beta}$ is small, 3) the approach to equilibrium is slow, 4) the error is
highly non-local, or 5) the readout basis states have well-defined symmetry
quantum numbers. Below, we detail our findings for each case, and discuss
alternate benchmarking formulae.

###  F.1 Violating the no-resonance condition: Particle-hole symmetry

The second no-resonance condition (S3) is extensively violated in the presence
of a particle-hole symmetry (PHS). By this we mean a reflection symmetry in
the spectrum: for every eigenvalue $E$, there is one at $-E$. For any given
$E,E^{\prime}$, the equation $E_{1}-E_{2}+E_{3}-E_{4}=0$ not only has
solutions $(E_{1},E_{2},E_{3},E_{4})=(E,E,E^{\prime},E^{\prime})$ and
$(E,E^{\prime},E^{\prime},E)$, but also has solutions:
$(E_{1},E_{2},E_{3},E_{4})=(E,-E,E^{\prime},-E^{\prime}),\leavevmode\nobreak\
(E,-E,-E^{\prime},E^{\prime})$. This will give rise to additional terms in the
second Hamiltonian twirling identity [Eq. (S4)]:

$\displaystyle\mathbb{E}_{t}\left[U_{t}^{\otimes 2}AU_{-t}^{\otimes 2}\right]$
(S84)
$\displaystyle\approx\sum_{E,E^{\prime}}\bigg{[}\outerproduct{E,E^{\prime}}{E,E^{\prime}}A\outerproduct{E,E^{\prime}}{E,E^{\prime}}+\outerproduct{E,E^{\prime}}{E,E^{\prime}}A\outerproduct{E^{\prime},E}{E^{\prime},E}+\outerproduct{E,-E}{E,-E}A\outerproduct{E^{\prime},-E^{\prime}}{E^{\prime},-E^{\prime}}\bigg{]}\leavevmode\nobreak\
,$

suppressing the subleading collision terms. This can arise from, an anti-
unitary charge conjugation operator, for example. Here, we focus on a simpler
example. PHS is present in the PXP model with no detuning ($\delta=0$):
$H_{\text{PXP}}=\sum_{j}PX_{j}$. $H_{\text{PXP}}$ anticommutes with the
unitary operator $\mathcal{C}\equiv\prod_{j}\sigma_{j}^{z}$:
$\mathcal{C}H_{\text{PXP}}\mathcal{C}=-H_{\text{PXP}}$ [56]. This symmetry is
present in arbitrary dimensions— we will present numerical evidence for the
two dimensional PXP model.

The PXP model is particularly convenient, for a number of reasons:

  * (i)

The computational basis states $\ket{z}$ are eigenstates of $\mathcal{C}$:
$\mathcal{C}\ket{z}=\pm\ket{z}$. This implies that easily-prepared bitstring
initial states such as $\ket{\Psi_{0}}=\ket{\downarrow\dots\downarrow}$ are
also eigenstates of $\mathcal{C}$.

  * (ii)

$H_{\text{PXP}}$ is real and thus possesses a time-reversal symmetry:
$H_{\text{PXP}}=H^{*}_{\text{PXP}}$. In particular, its eigenstates are real
in the computational basis.

The breakdown of the second no-resonance condition can be seen when we compute
the time average of $P^{(2)}(t)=\sum_{z}p_{\text{avg}}(z)\tilde{p}(z,t)^{2}$:

$\mathbb{E}_{t}\left[F_{\text{id,d}}(t)\right]\approx\sum_{z}\bigg{[}2\bigg{(}\sum_{E}\absolutevalue{\innerproduct{z}{E}\innerproduct{E}{\Psi_{0}}}^{2}\bigg{)}^{2}+\big{|}\sum_{E}\innerproduct{z}{E}\innerproduct{z}{-E}\innerproduct{E}{\Psi_{0}}\innerproduct{-E}{\Psi_{0}}\big{|}^{2}\bigg{]}/p_{\text{avg}}(z)\leavevmode\nobreak\
.$ (S85)

Using properties (i) and (ii) above,
$\sum_{E}\innerproduct{z}{E}\innerproduct{z}{-E}\innerproduct{E}{\Psi_{0}}\innerproduct{-E}{\Psi_{0}}=p_{\text{avg}}(z)$
and we obtain $\mathbb{E}_{t}\left[F_{\text{id,d}}(t)\right]\approx 3$ instead
of the usual value of two. This is illustrated in Fig. S6(a). We note that
this result is contingent on conditions (i) and (ii) above — if these
conditions were not satisfied, $\mathbb{E}_{t}\left[F_{\text{id,d}}(t)\right]$
will have a non-universal value between two and three. For example, if we use
an initial state that is not particle-hole symmetric, such as a low-energy
Boltzmann state, $\mathbb{E}_{t}\left[F_{\text{id,d}}(t)\right]\approx 2$, and
$F_{d}$ remains valid.

In particular, this implies that our $F_{d}$ formula will not estimate the
fidelity. Applying our simplifying assumptions in Eq. (S35), we obtain:
$F_{d}\approx\frac{4}{3}F-\frac{1}{3}$. Instead, a modified formula should be
employed:

$F_{d}^{\text{(PH)}}=\frac{3}{2}\frac{\sum_{z}q(z)\tilde{p}(z)}{\sum_{z}p(z)\tilde{p}(z)}-\frac{1}{2}\leavevmode\nobreak\
.$ (S86)

However, a similar analysis reveals that $F_{e}$ can be used with no
modification. This is illustrated in Fig. S6(b).

####  F.1.1 Real Porter-Thomas distribution

The value of three above is the second moment of the real Porter-Thomas
distribution [39] (stated, for simplicity, in the $D\rightarrow\infty$ limit):

$P_{\text{PT},\mathbb{R}}(x)=\frac{\exp(-x/2)}{\sqrt{2\pi
x}}\leavevmode\nobreak\ .$ (S87)

This distribution can be viewed as the overlap probability of a real-valued
random vector $\ket{\Psi}$. It has $k$-th moments:
$\int_{0}^{\infty}dx\leavevmode\nobreak\
x^{k}P_{\text{PT},\mathbb{R}}(x)=(2k-1)!!\equiv(2k-1)(2k-3)\cdots 3\cdot 1$.

Intuitively, if both the initial state $\ket{\Psi_{0}}$ and $\ket{z}$ are even
under $\mathcal{C}$, the overlap will be purely real:
$\innerproduct{z}{\Psi(t)}=2\sum_{E\geq
0}\innerproduct{z}{E}\\!\innerproduct{E}{\Psi}\cos(Et)$. We intuitively expect
this to be a random variable and for
$p(z,t)=\absolutevalue{\innerproduct{z}{\Psi(t)}}^{2}$ to follow the real
Porter-Thomas distribution. This also holds if $\ket{\Psi_{0}}$ or $\ket{z}$
are odd under $\mathcal{C}$.

We can prove that $\tilde{p}(z,t)$ (both with fixed time, and fixed
configuration $z$) follows $P_{\text{PT},\mathbb{R}}$, assuming conditions (i)
and (ii) above, and a particle-hole symmetric $k$ no-resonance condition. The
proof is identical to that of Theorem 2, except that instead only considering
the permutations of $k$ elements, we consider perfect matchings of $2k$
vertices. There are $(2k-1)!!$ such perfect matchings, giving the desired
$k$-th moments. We emphasize that to satisfy both conditions (i) and (ii),
both time-reversal and particle-hole symmetry must be present in the
Hamiltonian, and the initial state and readout basis states must have definite
PHS parity.

###  F.2 Violating the no-resonance condition: Non-interacting models

The second no-resonance condition is also violated in non-interacting models.
In non-interacting models, energy eigenvalues are given in terms of single-
particle eigenvalues. For concreteness, consider a non-interacting Bose-
Hubbard model, Eq. (S79) with $U=0$. Let the eigenvalues of the single-
particle Hamiltonian be $\epsilon_{k}$. An $N$-particle eigenstate is labelled
by the occupation $\\{n_{k}\\}$ of the single-particle modes $k$, with energy
$\sum_{k}n_{k}\epsilon_{k}$. This violates the 2 no-resonance condition: For
any $i,j,k,l$, there are eigenvalues
$E_{1}=\epsilon_{i}+\epsilon_{j},E_{2}=\epsilon_{k}+\epsilon_{l}$ and
$E_{3}=\epsilon_{i}+\epsilon_{k},E_{4}=\epsilon_{j}+\epsilon_{l}$ such that
$E_{1}+E_{2}=E_{3}+E_{4}$. Therefore, our Theorem does not apply to non-
interacting models, reflected in Fig. S6(c). Consequently, neither $F_{d}$ or
$F_{e}$ estimate the many-body fidelity (Fig. S6(d)). However, integrable and
non-integrable dynamics may be quantitatively similar at short times, at which
times the integrability breaking terms have not had sufficient time to
appreciably change the dynamics. Since $F_{d}$ is designed to work for these
short quenches, $F_{d}$ appears to estimate $F$ in this case, as seen in Fig.
S6(d).

Figure S6: Failure cases of benchmarking protocol: Violation of no-resonance
conditions and low effective dimension. (a) The particle-hole symmetry in the
2d PXP model violates the $k$ no-resonance conditions (S3). Under special
conditions (satisfied here in the 2d PXP model quenched from $\ket{0\cdots
0}$), $\tilde{p}(z,t)$ follows the real Porter-Thomas distribution, and
$F_{\text{id,d}}+1$ has an equilibrium value of 3 (inset). (b) The $F_{d}$
formula (blue) breaks down, and must be modified as in Eq. (S86). In contrast,
$F_{e}$ (red) can be used without modification. (c) Non-interacting models
also violate the no-resonance conditions. Here, we simulate a quench of the
Bose-Hubbard model with no interaction $U=0$. The $\tilde{p}(z,t)$ follows
neither PT distributions (red and black), and $F_{\text{id,d}}+1$ has large
temporal fluctuations (inset). (d) As a consequence, neither $F_{d}$ nor
$F_{e}$ approximate $F$, although $F_{d}$ is a good estimate at short times.
(e) A low-temperature quench has a small effective dimension. We realize this
with a Gibbs initial state $\ket{\Psi_{0}}=\sum_{E}\exp(-\beta/2E)\ket{E}$,
with $\beta=4$ in a Bose-Hubbard model with 8 particles on 8 sites.
$\tilde{p}_{z}(t)$, for an arbitrarily chosen $z$, follows a Porter-Thomas
distribution with a small dimension (orange): here $D_{\beta}(z)\approx 3.75$.
(f) As a consequence, $F_{d}$ (blue) and $F_{e}$ (red) have large temporal
fluctuations, and $F_{d}$ has a large systematic error. $F_{e}$ is less
susceptible to a small $D_{\beta}$, and is on average close to $F$. Figure S7:
Failure cases of benchmarking protocol: Slow equilibration and non-local
errors. (a) In the 1D PXP model, a quench of the Neel state
$\ket{\downarrow\uparrow\downarrow\uparrow\dots}$ undergoes slow
thermalization due to large overlap with a “tower” of eigenstates (inset).
$p_{z}(t)$ follows a PT distribution. (b) However, both $F_{d}$ (blue) and
$F_{e}$ (red) show large oscillations and do not estimate $F$ well (c) If an
inhomogeneous state is quench evolved, it undergoes slow hydrodynamic
relaxation, evidenced by the decaying oscillations of the average density in
the left half of a Bose-Hubbard chain (blue). (d) Under this quench, both
$F_{d}$ (blue) and $F_{e}$ (not plotted) exhibit oscillations. A modified
$F_{e}$ (orange) using a moving-averaged $p_{\text{avg}}$ yields a closer
estimate of the fidelity. (e) $F_{d}$ and $F_{e}$ response to an initial
error. At short times $F_{d}$ (blue) is a better estimate of $F$ than $F_{e}$
(red). Solid lines are average values over an ensemble of disordered
Hamiltonians, with shaded areas indicating variance. (f) We apply a local
error on a backward time-evolved state $\ket{\Psi(-\tau)}$ to engineer a non-
local error on the initial state. In this case $F_{e}$ (red) estimates $F$,
while $F_{d}$ (blue) does not, consistent with Appendix. D.

###  F.3 Small effective dimension: Low temperature quench

Our protocol also breaks down when the effective dimension $D_{\beta}$ is
small. This happens in low temperature quenches, for example. In the extreme
case that $\ket{\Psi_{0}}$ is an eigenstate, $D_{\beta}=1$. When
$\ket{\Psi_{0}}$ is a low temperature state, it’s dynamics are effectively
constrained to a low-energy subspace. Following Theorem 2, in Fig. S6(e), the
variable $\tilde{p}_{z}(t)$ approximately follows a Porter-Thomas distribution
(S12) with the effective dimension of the outcome $z$, $D_{\beta}(z)$.
Meanwhile, the distributions $\tilde{p}_{t}(z)$ show large fluctuations over
time, again because of the small effective dimension.

Our protocol’s systematic error and fluctuations scale inversely with
$D_{\beta}$. Therefore, it is less applicable to low temperature quenches. In
Fig. S6(f), we see a large discrepancy between $F_{d}$ and $F$ and large
fluctuations over time. However, since the systematic error of $F_{e}$ is less
senstive to $D_{\beta}$ [its $O(D_{\beta}^{-1})$ term vanishes], $F_{e}$
estimates $F$, when averaged over a time window.

###  F.4 Slow equilibration: Quantum many-body scar states

Some many-body systems exhibit long-lived oscillations which delays
equilibration of the quenched state. In the presence of these oscillations,
normalizing the measurement outcomes $p(z)$ with the equilibrium distribution
$p_{\text{avg}}(z)$ gives incorrect estimators.

One prominent example of long-lived oscillations is the recent discovery of
quantum many-body scar states [57, 56]. Originally experimentally discovered
in a Rydberg quantum simulator, this phenomenon can be captured with the 1D
PXP model:

$\displaystyle
H_{\text{1D-PXP}}=\Omega\sum_{j=2}^{L-1}P_{j-1}^{\downarrow}X_{j}P_{j+1}^{\downarrow}+X_{0}P_{1}^{\downarrow}+P_{L-1}^{\downarrow}X_{L}+\delta\sum_{j}n_{j}\leavevmode\nobreak\
,$ (S88)

where $P^{\downarrow}_{j}\equiv\outerproduct{\downarrow}{\downarrow}_{j}$ is
the projector onto the atomic ground state and
$n_{j}\equiv\outerproduct{\uparrow}{\uparrow}_{j}$ is the projector on the
Rydberg excited state. A quench of the Neel state
$\ket{\downarrow\uparrow\downarrow\uparrow\cdots\downarrow\uparrow}$ was
observed to exhibit long-lived oscillations — in particular, persistent
revivals of observables such as the magnetization and many-body fidelity.
These are attributed to a tower of eigenstates, energy eigenstates with which
$\ket{\Psi_{0}}$ has considerable overlap [Fig. S7(a), inset]. Due to special
properties of these eigenstates — most notably low half-chain entanglement
entropy — these states are dubbed many-body scar states. While the phenomenon
is most apparent with zero detuning, it persists with non-zero detuning. In
our results in Fig. S7(a,b), we use a non-zero detuning $\delta=0.2\Omega$ to
break the particle-hole symmetry.

These revivals eventually decay in time. Therefore, $\tilde{p}_{z}(t)$ follows
a PT distribution at late times [Fig. S7(a)]. However, these long-lived
oscillations are reflected in $F_{d}$ and $F_{e}$ [Fig. S7(b)] — $F_{d}$ is a
closer estimate to $F$ than $F_{e}$.

###  F.5 Slow equilibration: Inhomogeneous initial state

If our initial state is strongly inhomogeneous, it will exhibit hydrodynamic
oscillations before equilibrating [58]. At short times, the outcome
probabilities $p(z,t)$ will be far from the relatively homogeneous
$p_{\text{avg}}(z)$. We illustrate this with the initial state
$\ket{\Psi_{0}}=\ket{2,2,2,2,0,0,0,0}$ on a Bose-Hubbard model with 8 bosons
on 8 sites. We measure its relaxation with the average density in the left
half $\langle n_{L}\rangle\equiv\langle n_{1}+n_{2}+n_{3}+n_{4}\rangle/4$,
which exhibits decaying oscillations in Fig. S7(c). These oscillations are
reflected in $F_{d}$ and $F_{e}$, which show large deviations from $F$ (Fig.
S7(d)). We obtain better performance by normalizing $p(z,t)$ and $q(z,t)$ by a
moving average $p^{\prime}_{\text{avg}}(z)\equiv\int_{t-\Delta t/2}^{t+\Delta
t/2}dt\leavevmode\nobreak\ p(z,t)$, for a short interval $\Delta t$. We find
that the modified $F_{e}$,
$F^{\prime}_{e}(t)\equiv(\sum_{z}q(z,t)\tilde{p}(z,t)-1)/(\sum_{z}p(z,t)\tilde{p}(z,t)-1)$
is a closer approximation to $F$.

###  F.6 Non-local error on low entanglement states

In Appendix D, the random MPS model predicts that when the error size $N_{A}$
is much smaller than the bond dimension $\chi$, $F_{d}$ is a better
approximation to $F$ at short times than $F_{e}$. This condition is satisfied
in typical open system dynamics, since $\chi$ grows exponentially in time will
$N_{A}$ only grows linearly in time.

To verify our claim, we apply a single error at the start of the quench
($t=0$) and compute $F_{d}$ and $F_{e}$. Averaging over a large number of
Bose-Hubbard Hamiltonians with disordered hopping strengths, we verify that
$F_{d}$ on average estimates $F$ at very short times (albeit with large
variance) (blue, Fig. S7(e)), while $F_{e}$ (red) has a longer relaxation
time.

To engineer a situation in the opposite limit, where $\chi\ll N_{A}$, we
backwards time-evolve the initial state, then apply the local error on
$\ket{\Psi(-\tau)}$. We then forward evolve the ideal and perturbed states.
When $\absolutevalue{t}$ is small, this is equivalent to a non-local error
$V(t+\tau)$ on the low-entanglement states $\ket{\Psi(t)}$. In this regime, as
predicted, $F_{d}$ (blue, Fig. S7(f)) deviates from $F$ — reaching a minimum
value of $2F-1$, consistent with Eq. (S69) — while $F_{e}$ is close to $F$
(red).

We emphasize that such a situation is unlikely to arise in practical
situations, and therefore $F_{d}$ is more appropriate for short time quench
dynamics than $F_{e}$. We illustrate this in an example of target state
benchmarking in Fig. S8(b).

###  F.7 Non-ergodicity due to symmetry and readout in symmetry-resolved basis

We lastly discuss another case where $F_{d}$ does not estimate $F$ in the
presence of symmetry and a readout basis that has well-defined symmetry
quantum numbers.

In the presence of a symmetry, quantum states ergodically explore the Hilbert
space of their symmetry sector. In generic readout bases, the presence of such
symmetries does not impede $F_{d}$. However, if every state in the readout
basis belongs to one symmetry sector, $F_{d}$ will not be able to detect
relative phase errors, or phase decoherence, between the different symmetry
sectors. Physically, we can understand this as non-ergodicity of certain
information about the state: the phase information never propagates into the
readout basis. Mathematically, this does not affect our Theorems 1 and 2.
However, our ansatz that $q(z)=Fp(z)+(1-F)p_{\perp}(z)$ is no longer valid.
Instead, consider two states, expressed as superpositions of states in
different symmetry sectors: $\ket{\Psi}=\sum_{j}c_{j}\ket{\Psi_{j}}$ and
$\ket{\Psi^{\prime}}=\sum_{j}d_{j}\ket{\Psi^{\prime}_{j}}$. Then for a
bitstring $z$ in sector $j$, the ansatz should be modified to (assuming the
quench processes for $\ket{\Psi}$ and $\ket{\Psi^{\prime}}$ are identical):

$\displaystyle q(z)=F_{j}p(z)+(1-F_{j})p_{\perp}(z)\leavevmode\nobreak\
,\leavevmode\nobreak\
F_{j}\equiv\absolutevalue{\innerproduct{\Psi_{j}}{\Psi^{\prime}_{j}}}^{2}$
(S89)

Under this ansatz, $F_{d}$ does not reproduce the true fidelity
$F=\absolutevalue{\sum_{j}c^{*}_{j}d_{j}\innerproduct{\Psi_{j}}{\Psi^{\prime}_{j}}}^{2}$.
Applying our ansatz, and recalling that
$p_{\text{avg}}(z)\equiv\mathbb{E}[p(z,t)]$,
$F_{\text{XEB,d}}\equiv\sum_{z}q(z)\tilde{p}(z)-1$ takes the value
$2\sum_{j}\absolutevalue{d_{j}}^{2}F_{j}-1$

Consider the simplest case, $\ket{\Psi_{j}}=\ket{\Psi^{\prime}_{j}}$. Then
$F_{j}=1$, and $F_{\text{XEB,d}}=2\sum_{j}\absolutevalue{d_{j}}^{2}-1=1$,
regardless of the value of $c_{j},d_{j}$. Therefore, $F_{d}$ is unable to
estimate the fidelity. The infidelity arising from unequal populations
$\absolutevalue{c_{j}}^{2}$ of the symmetry sectors may be captured by a
suitably modified formula. However, infidelity arising from different phases
of $c_{j}$ is intrinsically inaccessible from such measurements: if
$d_{j}=\exp(i\theta_{j})c_{j}$, then $q(z)=p(z)$ identically.

Many analog simulators have readout bases that belong to a given symmetry
sector. For example, itinerant particles in optical lattices readout
configuration outcomes that have well-defined particle number, and their
natural Hamiltonians also conserve particle number. However, states without
well-defined particle number are not natural in such settings.

Such states that are in multiple symmetry sectors, however, may arise in other
settings. Consider our trapped ion model $H_{\text{TI}}$ (S82). This
Hamiltonian conserves parity $\prod_{j}\sigma^{z}_{j}$, and the computational
basis states have well-defined parity. Furthermore, it is easy to prepare
states without well-defined parity: for example, one starts with the all-downs
state $\ket{\downarrow,\cdots,\downarrow}$, and applies a local rotation on a
single site. Therefore, such a family of states cannot be benchmarked with the
$F_{d}$ formula. As another example, such a scenario can arise in a spin
chain, with a quench Hamiltonian that conserves magnetization
$\sum_{j}\sigma_{j}^{z}$. Bitstrings in the computational basis have well-
defined magnetization. In both cases, however, this can be remedied by
measuring in the $X$-basis, in which outcomes do not have well-defined parity
nor magnetization. This is explicitly demonstrated in a trapped-ion spin model
in Fig. S8(e,f).

> ##  Appendix G Greedy algorithm for multiparameter estimation

###  G.1 Algorithm

In this section we detail our greedy algorithm for simultaneous estimation of
multiple Hamiltonian parameters, used in Fig. 4(c). Our algorithm is as
follows. We are given a Hamiltonian $H(\pmb{\theta})$ with $n$ unknown
parameters $\pmb{\theta}=\\{\theta_{j}\\}_{j=1}^{n}$. We want to determine the
actual experimental parameters $\\{\theta_{j}^{*}\\}$, assuming that we can
prepare a fiducial state $\ket{\Psi_{0}}$ with high fidelity.

Our procedure is as follows:

> 1\. Evolve $\ket{\Psi_{0}}$ for a fixed time.
>
> 2\. Measure the configurations to approximate the distribution
> $q(z)=\bra{z}\rho(t)\ket{z}$.
>
> 3\. For some guess $\\{\theta_{j}\\}$, compute
> $p(z,\\{\theta_{j}\\})=\absolutevalue{\bra{z}\exp(-iH(\\{\theta_{j}\\})t)\ket{\Psi_{0}}}^{2}$.
> Compute $F_{d}(\\{J_{j}\\})$.
>
> 4\. Choose a random permutation $\pi\in S_{n}$. For each $j$, maximize
> $F_{d}(\\{\theta_{j}\\})$ over $\theta_{\pi(j)}$, keeping all other
> parameters fixed.
>
> 5\. Repeat 4. for all $j=1,\cdots n$.
>
> 6\. Repeat 3-5. until $F_{d}$ converges.
>
> 7\. Optionally repeat all steps with different initial guesses of
> $\\{\theta_{j}\\}$.

This procedure is justified as long as $F_{d}$ does not vary too quickly in
the range of $\\{\theta_{j}\\}$ considered.

In Fig. 4(c), we use this procedure to determine an unknown disorder potential
atop a known systematic Hamiltonian. Our greedy algorithm works well as long
as the disorder potential is small, and is also robust to errors in the
experimental evolution.

##  Appendix H Target state benchmarking

In Fig. 4(a) of our main text, we demonstrate target state benchmarking to
determine the unknown phase of a GHZ-like state in a 2D array of Rydberg
atoms. We take the prepared state to have a phase $\phi^{*}=\pi/4$. We wish to
measure $\phi^{*}$ by measuring the fidelity of the prepared state to target
states with various values of $\phi$. Using our protocol, we estimate the
fidelity by measuring $F_{d}$ after quench evolution.

In this section, we provide further demonstrations of target state
benchmarking. We first consider the simplest setting of coherent local errors,
which we demonstrate in the three other platforms studied. We then study
benchmarking between states that are related by non-local deformations. We
consider this setting for two applications: determining the Hamiltonian
parameters corresponding to a ground state, as well as estimating the
temperature of a low-energy state by measuring its overlap with the ground
state.

###  H.1 Coherent local errors

Here, we demonstrate target state benchmarking in the Bose-Hubbard, Fermi-
Hubbard, and trapped-ion spin chain model, using the same Hamiltonian and
readout basis discussed in Section E.

We consider locally rotated initial states: for the Bose-Hubbard model we
consider a unity-filling Mott insulator state on 8 sites deformed by a local
rotation $\phi$ on the middle two sites:

$\ket{\Psi(\phi)}_{\text{BH}}=\cos{\phi}\ket{1,1,1,1,1,1,1,1}+i\sin{\phi}\ket{1,1,1,2,0,1,1,1}\leavevmode\nobreak\
.$ (S90)

For the Fermi-Hubbard model we consider the anti-ferromagnetic state with 8
particles on 8 sites, again rotated along the middle bond:

$\ket{\Psi(\phi)}_{\text{FH}}=\cos{\phi}\ket{\uparrow,\downarrow,\uparrow,\downarrow,\uparrow,\downarrow,\uparrow,\downarrow}+i\sin{\phi}\ket{\uparrow,\downarrow,\uparrow,d,0,\downarrow,\uparrow,\downarrow}\leavevmode\nobreak\
,$ (S91)

where $d=\uparrow\downarrow$ denotes a doubly-occupied site, and $0$ denotes
an empty site. Lastly, for the trapped-ion model we consider the fully
polarized state of 12 qubits, rotated by a pair of spin-flips across the
middle bond

$\ket{\Psi(\phi)}_{\text{TI}}=\cos{\phi}\ket{\downarrow,\cdots,\downarrow,\downarrow,\downarrow,\cdots,\downarrow}+i\sin{\phi}\ket{\downarrow,\cdots,\downarrow,\uparrow,\downarrow,\cdots,\downarrow}\leavevmode\nobreak\
.$ (S92)

A single bit-flip produces a coherent superposition across states with
different parity. These states do not mix under the quench dynamics and the
resulting fidelity cannot be estimated by $F_{d}$, if the readout basis is the
computational basis. In contrast, if we measured in the $X$ basis, $F_{d}$
would be accurate, as discussed in (Section F.7). If instead we had flipped
two spins, our state would remain in the even-parity sector, and $F_{d}$ would
estimate $F$ with any readout basis.

In all three cases, we compute the fidelity
$F=\absolutevalue{\innerproduct{\Psi(\phi^{*})}{\Psi(\phi)}}^{2}=\cos(\phi-\phi^{*})^{2}$.
In Fig. S8(a,c,e), we show that $F_{d}$, evaluated at a fixed point in time,
estimates the fidelity $F$ in all three cases. In Fig. S8(b,d,f), we show that
$F_{d}$ takes a short delay time to learn the value of $F$ to a target state
$\phi=\frac{\pi}{2}$ — only a relatively short quench evolution of $t=O(1/J)$
is needed for this protocol. In the trapped ion model, $F_{d}$ shows slightly
larger deviations from $F$. This is because $F_{d}$ fluctuates in time about
$F$: this is indicated by the blue band in Fig. S8(e) and larger fluctuations
Fig. S8(f).

Figure S8: Target state benchmarking of locally-perturbed states. We apply
local perturbations to initial states in a (a,b) Bose-Hubbard, (c,d) Fermi-
Hubbard, and (e,f) trapped-ion spin chain model. We consider a class of states
parametrized by an angle $\phi$. We take states with $\phi^{*}=\pi/4$ as the
simulated prepared state, and compare it to target states with variable
$\phi$. As a function of $\phi$, the fidelity between the prepared and target
states vary from 0 to 1. By quench evolving these states, we estimate their
fidelity with $F_{d}$. In the top row, we demonstrate that the estimator
$F_{d}$ at a fixed, sufficiently late time, faithfully reproduces the fidelity
$F$. Error bars denote the statistical uncertainty from a finite number
$M=1000$ of samples. In the bottom row, we demonstrate the required evolution
time for $F_{d}$ to learn the true fidelity $F$, with the target state at
$\phi=\pi/2$. In (b), we compare the different estimators discussed in the
text: $F_{d}$ (blue), $F_{e}$ (green), and $F_{\text{XEB,d}}$ (red) in a Bose-
Hubbard model. The difference between $F_{\text{XEB,d}}$ and $F_{d}$
highlights the role of the denominator
$\sum_{z}p_{\text{avg}}(z)\tilde{p}(z)^{2}$ at short times. In (e), we show
that evaluating $F_{d}$ at a fixed time shows slight deviations from $F$. This
is due to the small effective dimension $D_{\beta}\approx 11$ in our trapped
ion model: $F_{d}$ fluctuates about $F$ over time, as seen in (f) and
indicated with the blue shaded region in (e). The purple markers and shaded
regions in (e,f) indicate readout in the computational basis. Due to
constraints from symmetry, $F_{d}$ does not estimate $F$ (in fact, it is
constant in our setting). In (f), the purple curve is evaluated with a target
state at $\phi=\pi/2+\varepsilon$ for a small $\varepsilon$. This is because
the target state at $\phi=\pi/2$ occupies only one out of two symmetry
sectors, hence is a fine-tuned, pathological case.

###  H.2 Comparing non-locally deformed states

Figure S9: Target state benchmarking by verifying preparation of a ground
state in a system. We consider two tasks: (a) estimating the corresponding
parameter value and (b) estimating the temperature of the prepared state. In
(a), we estimate the fidelity of the prepared state — assumed to be the ground
state at the critical value of $\left(U/\Omega\right)_{\text{crit}}\approx
3.6$ — against the ground states at various values of $U/\Omega$. We find that
$F_{d}$ (blue markers) is close to the true fidelity $F$ (dashed line). The
blue shaded region indicates the size of fluctuations of $F_{d}$ over time. In
(b), we estimate the fidelity of the prepared state — assumed to be the
Boltzmann state at various temperatures $T$ — against the ground state at
$\left(U/\Omega\right)_{\text{crit}}$. We find that $F_{d}$ (blue) slightly
over-estimates $F$, while $F_{e}$ (red) closely estimates $F$, consistent with
our analysis in Section. C. In both plots, error bars indicate the statistical
uncertainty associated with 1000 measurement samples.

We next demonstrate verifying the preparation of a non-trivial ground state in
two ways by computing the fidelities of states related by non-local
deformations. We first demonstrate verifying the position in the phase diagram
by computing the fidelities between a prepared critical ground state and the
ground states of the same Hamiltonian at other parameter values. We then
demonstrate estimating the temperature of the prepared state by considering a
prepared low-temperature Boltzmann state of the Hamiltonian at the critical
point and estimating its fidelity (overlap) to the ground state.

We simulate the ground state of the Bose-Hubbard model (S79) tuned to the Mott
insulator-superfluid transition point at
$\left(U/\Omega\right)_{\text{crit}}\approx 3.6$ on a system of 10 bosons on
10 lattice sites [67]. We wish to benchmark the fidelity of our prepared state
against its desired ground state. We consider two purposes: estimating the
best parameter $U/\Omega$ of the prepared ground state, and estimating the
temperature of the prepared state.

To do so, we consider quench evolving our prepared state via a modified Bose-
Hubbard Hamiltonian, and comparing it to simulated data of quenched ground
states at various values of $U/\Omega$. Here, care must be taken for two
reasons. Firstly, the quench Hamiltonian should be chosen such that the ground
states under investigation do not have too low energy with respect to the
quench Hamiltonian. Secondly, the family of states we consider do not simply
differ by local errors. Rather, the family of ground states are related by
possibly non-local transformations. This is outside of the scope of our
heuristic derivation. However, we numerically observe that the $F_{d}$ formula
still approximates $F$ under judicious choices of the quench Hamiltonian.

In order for the ground states in both phases to have sufficiently ergodic
quench dynamics, we consider a quench with a disordered Hamiltonian:
$H=H_{\text{BH}}\left[(U/\Omega)_{\text{crit}}\right]+\sum_{j=1}^{8}\Delta_{j}n_{j}$,
where $\Delta_{j}\sim\text{Unif}[-3,3]$. In Fig. S9(a), $F_{d}$ approximates
the true ground state overlap in a system of 8 bosons on 8 lattice sites.

We next demonstrate estimating the temperature of an experimentally prepared
state. Assuming we have correctly determined the parameter $U/\Omega$, we
simulate the $F_{d}$ values between the quench-evolved ground state and
quench-evolved Boltzmann states $\rho=Z(T)^{-1}\exp(-H_{\text{BH}}/T)$, where
$Z(T)$ is the partition function at temperature $T$. The fidelity is simply
$Z(T)^{-1}\exp(-E_{0}/T)$. In this case, $F_{d}$ [Fig. S9(b), blue], is close
to the true fidelity. However, because of a relatively small effective
dimension $D_{\beta}\approx 5$, $F_{d}$ is larger than $F$, consistent with
the prediction in Eq. (S41). This systematic shift can be manually subtracted.
Alternatively, the estimator $F_{e}$ [Eq. (S70)] is more robust to small
values of $D_{\beta}$ and closely estimates the fidelity [Fig. S9(b), red].


rightsretained https://sites.google.com/view/ai4hci/accepted-papers

# BlackBox Toolkit: Intelligent Assistance to UI Design

Vinoth Pandian Sermuga Pandian

Sarah Suleri

Fraunhofer FIT - UCC  Sankt Augustin, Germany<EMAIL_ADDRESS>Fraunhofer FIT - UCC  Sankt Augustin, Germany<EMAIL_ADDRESS>
(2020)

###### Abstract

User Interface (UI) design is an creative process that involves considerable
reiteration and rework. Designers go through multiple iterations of different
prototyping fidelities to create a UI design. In this research, we propose to
modify the UI design process by assisting it with artificial intelligence
(AI). We propose to enable AI to perform repetitive tasks for the designer
while allowing the designer to take command of the creative process. This
approach makes the machine act as a black box that intelligently assists the
designers in creating UI design. We believe this approach would greatly
benefit designers in co-creating design solutions with AI.

###### keywords:

UI Element Dataset, Neural Networks, Deep Learning, Sketch Detection, Sketch
Recognition, Blueprints, UI Layout, Prototyping

††conference: CHI’20, Workshop on Artificial Intelligence for HCI: A Modern
Approach April 25–30, 2020, Honolulu, HI, USA

<ccs2012> <concept>
<concept_id>10003120.10003121.10003124.10010865</concept_id>
<concept_desc>Human-centered computing Graphical user
interfaces</concept_desc> <concept_significance>500</concept_significance>
</concept> <concept>
<concept_id>10003120.10003123.10010860.10010858</concept_id>
<concept_desc>Human-centered computing User interface design</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10003120.10003123.10010860.10011694</concept_id>
<concept_desc>Human-centered computing Interface design
prototyping</concept_desc> <concept_significance>500</concept_significance>
</concept> <concept>
<concept_id>10003120.10003121.10003122.10010854</concept_id>
<concept_desc>Human-centered computing Usability testing</concept_desc>
<concept_significance>300</concept_significance> </concept> <concept>
<concept_id>10003120.10003121.10003122.10010856</concept_id>
<concept_desc>Human-centered computing Walkthrough evaluations</concept_desc>
<concept_significance>300</concept_significance> </concept> <concept>
<concept_id>10003120.10003123.10010860.10010859</concept_id>
<concept_desc>Human-centered computing User centered design</concept_desc>
<concept_significance>300</concept_significance> </concept> </ccs2012>
<ccs2012> <concept>
<concept_id>10010147.10010178.10010224.10010245.10010250</concept_id>
<concept_desc>Computing methodologies Object detection</concept_desc>
<concept_significance>500</concept_significance> </concept>

[500]Computing methodologies Object detection [500]Human-centered computing
Graphical user interfaces [500]Human-centered computing User interface design
[500]Human-centered computing Interface design prototyping [300]Human-centered
computing User centered design

##  1 Introduction

The user interface (UI) acts as the bridge between a human and a machine.
††margin:  "Just as the Industrial Revolution freed up a lot of humanity from
physical drudgery I think AI has the potential to free up humanity from a lot
of the mental drudgery."
–Andrew Ng It acts as a translator mediating between two worlds: one
disorderly, irrational, but adept at noticing patterns; another structured,
analytical, however inept in pattern-finding (as of now). A UI designer is an
architect who designs this bridge between man and machine.

The most important job of a UI designer is not to find a balance between both
worlds; instead, reduce the mental load of the one with issues and emotions
and try to fit it into the confinements and restrictions placed by the other.
This task of designing such interfaces is strenuous. Among the numerous ways
of creating user interfaces to satisfy both the worlds involved, the most
commonly used technique is user-centered design.

In user-centered design, users are kept at the heart of design, and designers
attempt to satisfy their needs by analyzing the usage context, user needs, and
requirements before starting the design process. Then during the design
process, designers go through multiple fidelities of prototypes. Starting from
low-fidelity (lo-fi) freehand sketches to medium-fidelity (me-fi) digital
images and finally to high-fidelity (hi-fi) interactive screens or code. The
different fidelities have their strengths and weaknesses. For example, lo-fi
is cheap and supports ideating different designs quickly; however, it does not
do justice to the final look-and-feel of the system. Similarly, me-fi contains
the most necessary information and is quicker to create than hi-fi; however,
it is hard to create multiple design variations compared to lo-fi. Hi-fi
resembles the final product, but the workload of creating such a system is
humongous, and it is hard to create multiple design modifications to test the
system.

Several researchers attempt to solve the issues in this design process. After
the advancements of AI, one solution is to automate this whole process - the
designers sketch a UI, and then the machine analyses the sketch and generates
the hi-fi code. This interesting approach, however, has one major flaw. The
whole system acts like a transformer to the designer who enters their sketch
and gets a corresponding code with no control over tweaking the intricate
design details. As a solution, we approach the same problem domain and propose
two solutions. Our goal is to enable the machine to perform repetitive tasks
for the designer while allowing the designer to take command of the creative
process. This approach makes the machine act as a black box that intelligently
assists the designers in UI design. The machine’s role in our proposed design
process is no different from the apprentices of renaissance art maestros. The
apprentice’s task to assist the artist in preparing materials and executing
the less critical and quite tedious decorative parts of frescoes or statues.
We believe this approach would greatly benefit designers where AI and human
co-create creative solutions.

##  2 Research Focus

Our focus in this research is on how to move designers as the drivers of
creativity and let AI assist designers in the UI design process. Our primary
research question in this research is, "How can we automate the UI design
process while allowing UI designers to control the design details." We address
this question by using artificial intelligence to automate the transformation
of different fidelities of UI design. In the following sections, we expand on
each of our proposed solutions, the challenges we faced with implementation
and the benefit of that solution.

##  3 MetaMorph

MetaMorph is a UI element detector, created with a Deep Neural Network (DNN)
object detection model [6]. MetaMorph detects constituent UI element
categories and their position from a lo-fi sketch using a fine-tuned RetinaNet
object detection model trained with a dataset of 125,000 synthetically
generated lo-fi sketches.

###  3.1 Challenge & Solution

The major challenge in creating MetaMorph was the dataset. We required a large
scale lo-fi sketch dataset, which was non-existent. Therefore, we collected UI
element sketches from 350 participants using paper and digital questionnaires.
Then we processed this data and labeled it to create the UISketch
dataset111https://www.kaggle.com/vinothpandian/uisketch [5]. This dataset
contains 5,917 sketches of 19 UI elements. However, this labeled dataset is
only useful for classifying UI elements; but, a UI element detector would need
ground truth of both the identity and location of UI elements in a lo-fi
sketch.

Sample generated synthetic data

As a solution, we created Syn222https://www.kaggle.com/vinothpandian/syn-
dataset, a large-scale synthetic dataset containing 125,000 synthetically
generated lo-fi sketches [5]. To create Syn, we randomly chose UI elements
from the labeled UISketch dataset and stitched them in random locations with
random scaling (Figure 3.1). This random allocation of elements in an image is
similar to the pre-processing and data augmentation techniques used in
improving detection metrics of object detection models. We used Syn to train
the MetaMorph UI element detector.

We then collected 200 lo-fi sketches to evaluate MetaMorph. The evaluation
results indicate that MetaMorph detects UI elements from lo-fi sketches with
63.5% mAP. MetaMorph333https://metamorph.designwitheve.com/ is available as an
open web API444http://api.metamorph.designwitheve.com/. We have also open-
sourced the UISketch dataset and Syn.

Usage of MetaMorph to detect lo-fi elements and transforming them to me-fi in
Eve

###  3.2 Benefits

By detecting the UI elements present in a lo-fi sketch, the lo-fi prototype
can be converted to me-fi or hi-fi instead of fully automating the process
(Figure 3.1). MetaMorph enabled us to create Eve, a prototyping workbench [6]
where a designer can sketch or upload a lo-fi, which will be converted to me-
fi and later to hi-fi by means of UI element detection. Eve enables the
designer to control the styling of the UI in me-fi and progress it to hi-fi
android XML code.

##  4 Blu

Blu is a tool that generates UI layout information from UI screenshots. With
the detected information, it enables conversion of UI screenshots to
blueprints and editable vector graphics [4]. Herring et al. demonstrate the
benefits and role of design examples in different aspects of the UI design
process [3]. In this research, we expand on this idea, and from a UI design
example screenshot, we detect UI element categories, positions, grouping
information, and layout using deep neural networks.

###  4.1 Challenge & Solution

We faced two significant challenges in implementing Blu: dataset and layout
detection.

Fortunately, as a solution to dataset issue, Deka et al. collected a large
scale android UI screenshot dataset, RICO [2]. RICO contains 72k UI
screenshots with UI element hierarchy and semantic annotations. However, RICO
was annotated using an automated approach; therefore, the annotations are
sometimes mislabelled. If RICO is directly used as training data of a DNN,
these mistakes propagate and provide inadequate results. Therefore, we had to
re-annotate the elements and layout information for a subset of RICO to train
Blu.

Another challenge in creating Blu is identifying the layout information. A UI
designer by education and practice groups and aligns UI elements while
creating a me-fi prototype. They group UI elements mostly based on gestalt
laws. This layout information further helps the front-end developers to create
the hi-fi with the constraints placed on them by programming languages and
development environments. However, there is no algorithmic way to automate the
layout of UI elements using gestalt law yet. To solve this issue, we are
attempting to automate the alignment process algorithmically based on gestalt
laws.

UI screen (left) and its respective blueprint (middle) and UI layout tree
(right) created using Blu.

###  4.2 Benefits

Blu reduces the rework of UI designers for starting a design from scratch [4].
It also assists designers to generate blueprints of UI design and convey the
design information to developers. To demonstrate Blu, we created a web
application555https://blu.blackbox-toolkit.com/ that utilizes the annotations
from RICO and generates a blueprint (Figure 4.1). This web app helps to convey
the design and layout information of UI screen.

##  5 Summary & Future work

In this paper, we presented our research on utilizing AI to assist designers
in the UI designing process. We introduce the first two of our tools
(MetaMorph and Blu) from our proposed solution, BlackBox toolkit. This
research is an exploration of applying bleeding-edge AI research in human-
computer interaction domain. This ongoing research is at its incipient phase
with two tools. Further, we are planning to ideate and implement similar
tools, such as generating UIs and automatic detection of accessibility issues
in UIs.

By this research, we are looking forward to a future where humans and AI
collaborate in creative tasks similar to the analytical tasks.

## References

  * [1]
  * [2] B Deka, Z Huang, C Franzen, J Hibschman, D Afergan, Y Li, J Nichols, and R Kumar. 2017.  Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual ACM Symposium on UIST (UIST ’17). ACM, New York, NY, USA, 845–854.  DOI:http://dx.doi.org/10.1145/3126594.3126651
  * [3] S R. Herring, C C Chang, J Krantzler, and B P. Bailey. 2009.  Getting Inspired! Understanding How and Why Examples Are Used in Creative Design Practice. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09). ACM, New York, NY, USA, 87–96.  DOI:http://dx.doi.org/10.1145/1518701.1518717
  * [4] VPS Pandian, S Suleri, and M Jarke. 2020a.  Blu: What GUIs Are Made Of. In Proceedings of the 25th International Conference on IUI Companion (IUI ’20). ACM, New York, NY, USA, 81–82.  DOI:http://dx.doi.org/10.1145/3379336.3381497
  * [5] VPS Pandian, S Suleri, and M Jarke. 2020b.  Syn: Synthetic Dataset for Training UI Element Detector From Lo-Fi Sketches. In Proceedings of the 25th International Conference on IUI Companion (IUI ’20). ACM, New York, NY, USA, 79–80.  DOI:http://dx.doi.org/10.1145/3379336.3381498
  * [6] S Suleri, VPS Pandian, S Shishkovets, and M Jarke. 2019.  Eve: A Sketch-Based Software Prototyping Workbench. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19). ACM, New York, NY, USA, Article Paper LBW1410, 6 pages.  DOI:http://dx.doi.org/10.1145/3290607.3312994