Unpacking Human-AI interactions: From interaction primitives to a design space

Konstantinos Tsiakas
Dave Murray-Rust

Delft University of Technology
The Netherlands

This paper aims to develop a semi-formal design space for Human-AI interactions, by building a set of interaction primitives which specify the communication between users and AI systems during their interaction. We show how these primitives can be combined into a set of interaction patterns which can provide an abstract specification for exchanging messages between humans and AI/ML models to carry out purposeful interactions. The motivation behind this is twofold: firstly, to provide a compact generalisation of existing practices, that highlights the similarities and differences between systems in terms of their interaction behaviours; and secondly, to support the creation of new systems, in particular by opening the space of possibilities for interactions with models. We present a short literature review on frameworks, guidelines and taxonomies related to the design and implementation of HAI interactions, including human-in-the-loop, explainable AI, as well as hybrid intelligence and collaborative learning approaches. From the literature review, we define a vocabulary for describing information exchanges in terms of providing and requesting particular model-specific data types. Based on this vocabulary, a message passing model for interactions between humans and models is presented, which we demonstrate can account for existing systems and approaches. Finally, we build this into design patterns as mid-level constructs that capture common interactional structures. We discuss how this approach can be used towards a design space for Human-AI interactions that creates new possibilities for designs as well as keeping track of implementation issues and concerns.

<concept_desc>Human-centered computing Interaction design</concept_desc>
<concept_desc>Human-centered computing Interaction paradigms</concept_desc>
<concept_desc>Computing methodologies Artificial intelligence</concept_desc>

[500]Human-centered computing Interaction design
[300]Human-centered computing Interaction paradigms
[300]Computing methodologies Artificial intelligence

§ INTRODUCTION

Artificial Intelligence and Machine Learning (AI/ML) models are in the midst of a transition from use in back-end data science systems, operated and interacted with by experts, to end-user focused applications that prioritise ease of use and interactional fluidity <cit.>. This transition to human-centered AI approaches is driven by several factors – the increasing power of the models and systems; the need for more engagement with real-world data; the enlisting of users as participants in model development through annotation or other feedback; the desire by more organisations to make use of the possibilities of AI/ML; and the corresponding need to make sure that the models are operating correctly or adapt them for particular tasks. This has given rise to a growing set of human-AI interaction paradigms, in particular human-in-the-loop (HITL) <cit.>, explainable AI (XAI) <cit.> and hybrid intelligence (HI) and collaborative learning systems <cit.>. This expansion in interactivity, coupled with the need to understand how systems grow and change over time <cit.> and affect diverse stakeholders <cit.> leads to a need for new ways to design interactions, as we transition from Human-Computer to Human-AI interactions <cit.>.

Human-AI (HAI) interactions can involve humans in different parts of the algorithmic operation, offering possibilities for designing new types of interactions, considering the different paradigms. XAI methods are used to provide additional information about the underlying AI processes <cit.> through communicating descriptions of model functioning <cit.> even if it is not always a silver bullet <cit.>. HITL approaches aim to enhance system's decisions by incorporating human decision making, when needed, to enable human oversight <cit.>. Human feedback can be integrated to the learning mechanism of ML models through interactive ML (iML) methods to facilitate model training, as well as to explore new forms of interactivity between humans and ML algorithms <cit.>. Hybrid intelligence and collaborative learning methods can combine human capabilities with AI, allowing both parts to efficiently communicate and exchange information in order to mutually make decisions, inform and learn from each other <cit.>.

Despite the computational advances and capabilities of AI models, designing HAI interactions remains a challenging task. Designers often have a broad understanding of the possibilities of AI/ML, but lack specifics <cit.>. This is related to two AI attributes: capability uncertainty, which indicates the uncertainty of what the possibilities of an ML model are, and output complexity, which covers the general difficulty of working with multidimensional, rich outputs from large systems <cit.>. Envisioning new AI-based solutions for a given UX problem should consider that HAI interactions need to adjust to different users and evolve over time <cit.>. Going beyond the one-to-one interactions with systems, there is a also the need to investigate the effects of automated algorithmic systems on on the different groups and types of users involved in or affected by the AI decisions <cit.>, such as in organisational decision making <cit.>. Human oversight is an important aspect which enables users to maintain human agency and accountability, by participating in the decision making. From such a sociotechical perspective, hybrid (or shared) decision making can involve both decision-makers and decision-subjects towards interactions with contestable AI models <cit.>. However, designing for hybrid decision making can be challenging while considering the legal, social, technical and organizational issues <cit.>. Moreover, designing AI-based systems that facilitate meaningful human control requires the implementation of moral decision making methods in order to address responsibility gaps related to hybrid decisions <cit.>.

Responses to this complexity and the need for human-centered AI design takes several forms, from high level frameworks through to new interaction paradigms and implementation methods. High level frameworks shape the way that people approach creating HAI systems. Shneiderman's Human-Centered Artificial Intelligence (HCAI) framework looks to create reliable, safe and trustworthy HAI interactions through active participation <cit.>, with the intent to achieve a high level of human control, while maintaining a high level of computer automation. Xu's HAI framework <cit.> sets out three key components for system design: technology enhancement, ethically aligned design, and human factors design, highlighting the need to design responsible and reliable AI-based solutions. Yurrita et. al's multi-stakeholder framework looks at how human values connect to properties of AI/ML systems <cit.>. To address sociotechnical questions, new human-machine configurations use HITL techniques to augment the algorithmic system through model auditing and altering <cit.>. Such interactions enable the user to be part of the decision making process by querying, evaluating, and editing the underlying model and data. Similarly, a Human-Centered XAI approach (HCXAI) proposes to put the user and human factors in the center of technology design, taking into consideration the interplay between values, interpersonal dynamics, and the socially situated nature of AI systems <cit.>.

Design guidelines and frameworks tend to work upwards from an interactional perspective, to support practitioners through a set of design methods, practices and examples for designing with AI, e.g., Google's People + AI Guidebook[https://pair.withgoogle.com/guidebook]. Other approaches focus on specific contexts and application areas to identify the most important design aspects and challenges for a given context, e.g., design guidelines for AI-based Tutoring Robots <cit.>. Similarly, design frameworks aim  to address challenges related to Responsible AI by integrating ethical analysis into engineering practice <cit.>, or by addressing ethical challenges for specific AI application contexts, e.g., ethical AI in K-12 education <cit.>. Research through Design (RtD) has been proposed as an approach to ensure that the role of AI in a system is legible to the end users <cit.>. Sketching and prototyping are RtD activities which could support the ideation and implementation of HAI interactions. At a higher conceptual level, metaphors can affect expectations of performance <cit.>, help designers to understand key concepts <cit.>, or lead to envisioning new kinds of relations between humans and technology <cit.>.

This work attempts to address the gap of ways to specify interactions between humans and models. Our proposed design space aims to be more pragmatic than high level frameworks, as it works from concrete actions at a user-model level, providing a link from guidelines and suggestions for design practices to the actual implementation and prototyping of existing and new types of interaction. It attempts to give designers and developers more direct facility to engage with the potentials for interaction with models by providing a palette of possibilities that map understandable human concepts to specific exchanges of information. We do this through proposing a communication protocol which can describe the interactions that take place around human user(s) and AI/ML model(s). We draw inspiration from agent communication protocols <cit.> which use communicative acts to enable agents to communicate their intent for a specific service <cit.>. Working from this, the intuition behind this paper is that a common set of communicative acts can be defined to describe HAI interaction patterns as information exchanges whether by providing training examples to a model, validating a model's prediction or providing explanations. Such a specification would help to imagine richer interactions with models, and could allow a broader range of people to take part in the design of HAI interactions.

Our approach defines interactions between users and models based on the intent and type of the exchanged information. Towards this, we review existing frameworks and guidelines for HAI interactions, as well as application examples, in order to identify a common set of communication channels between users and AI models. Following our approach, we unpack existing HAI interaction into low-level communicative acts. Based on this, we define a set of HAI interaction primitives which consider both design and implementation aspects of an interaction (Figure <ref>). Our motivation for defining a design space based on HAI interaction primitives is to: (a) provide a space in which existing interaction concepts and paradigms can be represented, bridging the gap between human and machine understanding of what an interaction entails, (b) allow for the potential invention of new kinds of interactions through exploring the space of communicative actions and interaction patterns, and (c) provide a specification which carries the required information needed for the prototyping and implementation of HAI interactions. The key contributions of the paper are:

* A review of HAI interaction paradigms, design frameworks and guidelines for XAI, HITL, and HI, along with a set of use cases, focusing on the different types of HAI communication.

* A system of HAI primitives and types that can be used to encode the interactions between users and AI models in the form of actions and message passing, capturing the intent and type of the communicated information.

* A collection of interaction patterns as sequences of messages which can be used as design patterns for human-AI interactions to (a) reflect existing practice and (b) create new forms of interactions.

Proposed Approach. We review existing guidelines, taxonomies and frameworks related to the design and implementation of human-AI interactions. Our goal is to unpack HAI interactions into a set of interaction primitives and patterns, based on which more complex interactions can be defined. Taking into consideration both technical and design challenges, we describe how the proposed primitives can be used to define a design space which can inform both design and implementation choices for HAI interactions.

The paper is structured as follows:

* In order to define the proposed communicative framework, we review existing taxonomies, frameworks, and guidelines for HAI interactions, including HI, XAI, HITL, interactive ML (iML), and collaborative learning systems (Section <ref>). Considering the range of these interaction paradigms, approaches and concepts, our goal is to specify the interactions between humans and AI models, based on the intent and type of the communication.

* From this, we extract a set of communicative acts that can be combined into interaction patterns to represent a range of HAI interactions within these dominant paradigms (Section <ref>). We build these acts and patterns from a combination of simple primitives and a set of types into human-readable verbs that specify an interaction. We demonstrate our approach by describing interactions from existing use cases and frameworks (Section <ref>).

* We discuss how a design space using the proposed HAI interaction primitives and patterns can be developed to support both designers and AI practitioners to design and implement HAI interactions considering both design and technical aspects (Section <ref>).

§ BACKGROUND AND RELATED WORK

The goal of the review is to identify the ways of communication between human users and AI models and characterize them in terms of the intent and type of the exchanged information, exploring existing HAI interaction paradigms. We provide an overview of design guidelines and taxonomies which provide suggestions on how to apply design principles for HAI interaction systems, as well as frameworks for design patterns. Finally, we present a set of HAI interaction systems focusing on the design and implementation aspects, as well as approaches for communicative protocols, towards defining a set of communicative acts and patterns for HAI interactions.

§.§ Human-AI Interaction Paradigms

Human-AI interaction is defined as “the completion of a user's task with the help of AI support, which may manifest itself in non-intermittent scenarios” <cit.>. Following this definition, there are three main HAI interaction paradigms, intermittent, and continuous, and proactive, taking into consideration “how differences in initiation and control result in diverging user needs”. Intermittent HAI interactions are user-initiated and turn-taking interactions where the system provides a response for an explicit and predefined cue. Continuous HAI interactions utilize user's implicit input, as part of a continuous input stream, and provide a response which can either be accepted or ignored by the user. Finally, proactive HAI interactions are AI-initiated and triggered by predefined changes in the system. Since such interaction paradigms can exist in parallel, there is an increasing need to consider the open challenges while designing continuous and proactive interactions in terms of how users and AI systems interact with each other.

One of the key concerns while designing HAI interactions is the black box nature of AI models, where decisions and underlying operations are not visible or explained to humans. Moreover, human intentions are not always clear or they differ from the actual communicated action. This problem is known as the “two black boxes problem”, based on which both human cognition (cognitive intelligence) and AI are considered as black boxes. In order to address this problem, a symmetric and collaborative model for HAI interactions has been proposed, highlighting the need for explainability for both communication channels <cit.>. Based on this framework, Semantic Interaction (SI) is used as a design philosophy for symmetric and collaborative interactions between humans and AI. In the context of SI, XAI serves as a method to communicate information from AI to human users, while explainable cognitive intelligence (XCI) is used to provide information from humans to AI systems. A main challenge while considering both communication channels for the design of HAI is the semantic gap. Human high-level decisions and actions need to be translated to a set of model-understandable parameters, values and operations, while AI models must communicate its internal processes and decisions in a human-understandable way. HAI interactions should enable both AI systems and human users to efficiently communicate to each other and make decisions in a collaborative manner to augment both human and artificial intelligence, towards Hybrid Intelligence (HI) interactions. Akata et al. <cit.> unpack the research challenge of building HI systems into four research themes: Collaborative, Adaptive, Responsible, and Explainable HI. An open challenge towards designing interactions with HI systems is to develop methods for designing negotiation, agreements, planning, and delegation interactions in hybrid teams, considering the needs and role of the team members. Usable and Useful AI are two terms which describe how interactions with AI models can be designed to provide usable solutions that are easy to understand and apply in order to satisfy user needs <cit.>. Towards this, AI designers and practitioners should consider both the development of AI/ML models and the design of the interactions and behaviors around such models, including user-centric techniques, such as explainability and user control.

The integration of explainability and user control features creates new types of interactions between users and models. Designing interactions with (X)AI models should follow specific design principles towards the development of Responsible AI systems, related to fairness, transparency, and privacy aspects <cit.>. In terms of user control and feedback, HITL and iML techniques enable users to interact with AI/ML models in order to guide, steer and facilitate the learning process. Human users can be involved in the different stages of the ML pipeline: data extraction, integration and cleaning, iterative labeling, as well as model training and inference <cit.>. Related to how human users can intervene with models, model contestation is defined as an interaction with “humans challenging machine predictions” <cit.>. A proposed framework for contestable AI by design <cit.> highlights a set of sociotechnical features and practices for model contestation and hybrid decision making, including interactive controls, explanations, and intervention requests. Such approaches highlight the need for designing new types of interactions, considering the different interaction paradigms, leading to the development of design frameworks and guidelines for HAI interactions.

§.§ Design Guidelines, Frameworks and Taxonomies for Human-AI interactions

Design guidelines and frameworks aim to support designers on how to apply specific design principles while designing interactions with AI systems. Microsoft Research has proposed a set of guidelines for designing HAI interactions <cit.>. These guidelines provide design choices and examples for interactions between users and AI systems, considering aspects of system transparency, explainability, user feedback and control. Designing interactions which support such features requires human users to interact with the system at a user-model level. That means that designers should take into consideration the different ways that users can efficiently interact with an AI model, as well as the underlying design and technical challenges while prototyping such interactions. A process model for co-creation of AI experiences (AIX) describes how designers can be familiarized with AI models and their functionalities in order to include them in the design process as design materials <cit.>. However, there is a lack of design innovation in envisioning how ML/AI might improve and create UX value to users. Yang et al. <cit.> highlight the need to support UX practitioners by creating design patterns. They propose four channels based on which AI/ML capabilities can create UX value to users, namely self, context, optimal, and utility-capability, considering what users can infer about these aspects through their interaction with models. Such frameworks and guidelines aim to support the ideation process of designers while prototyping and designing interactions between users and an AI system.

Taxonomies and frameworks for XAI approaches and methods can support the design and implementation of interactions with explainable systems, considering different aspects of XAI-based interactions. A collection of XAI-based questions has been proposed as a design space for XAI interactions  considering diverging user characteristics, e.g., user needs, role, expertise, and experience  <cit.>. Other taxonomies focus on the different aspects of the use case, e.g., problem definition, explanator properties, and evaluation metrics <cit.> or the relations between the use case, the AI system, and the explanation algorithm <cit.>. Design frameworks can provide guidelines on how to design and evaluate XAI-based interactions by mapping design goals to evaluation methods based on the target population <cit.>. Focusing on user evaluation for XAI interactions, Chromik et al. <cit.> classify evaluation methods based on task-related, participant-related and study design-related dimensions. Such frameworks can support the selection of appropriate XAI methods for a given interaction concept. Sperrle et al. <cit.> proposed a dependency model for XAI processes  as a design space for human-XAI interactions, considering the potential bias that can be propagated during the interactions between different stakeholders and end-users. Wang et al. <cit.> followed a theory-driven approach to link explanation features to user reasoning goals, resulting to a conceptual framework which can inform the selection of appropriate explanations towards mitigating possible cognitive biases. Designing XAI interfaces should also consider specific design principles based on the interaction concept and explanatory goals <cit.>. Human users may interact with XAI interfaces for information transmission, where the goal of XAI is to help the user understand the underlying AI behavior, while XAI-interaction as dialogue refers to an iterative user-driven communication through user queries and AI responses/explanations. For both concepts, the AI behavior does not change in contrast to XAI-interaction as control, where the user provides feedback to adjust the (explainable) model until a desired AI behavior is reached, which relates to HITL/iML interactions.

HITL/iML methods enable human users to participate in the decision making and model training process. Design principles for user control and feedback interfaces include interactivity to promote rich interactions, providing explicit and clear task goals, support user understanding and engagement, and capture user's intent based on their input <cit.>. Moreover, designing interactions with HITL/iML interfaces requires appropriate learning methods, considering new forms of relationships between humans and ML algorithms <cit.>.
The design and implementation of iML/HITL interfaces depend on the type of user feedback and can affect both user experience and system/model performance. Michael et al. <cit.> discuss how the different types of cognitive feedback can be integrated to the iML mechanism, i.e., self-reporting, implicit feedback, and modeled feedback, through different feedback mechanisms. For example, domain-expert feedback can be communicated to the system and translated to model updates, i.e., modify dataset or model parameters <cit.>. Such HITL/iML approaches consider the effects of the interaction on human engagement and feedback quality. Since explanations can be used to enhance user's understanding of the model's performance, they can play an important role in ensuring a high quality of user feedback. Explanations can be combined with interactive capabilities enabling users to train ML models from scratch, resulting to a closed loop of XAI- and HITL-based interactions <cit.>.

In the domain of hybrid intelligence, a taxonomy for design knowledge organizes the design decisions for HI interactions, including XAI and HITL methods, along four dimensions: task characteristics, learning paradigm, human-AI interaction, and AI-human interaction <cit.>.

Focusing on hybrid decision making, a common interaction paradigm is backward reasoning design where the system's goal is to provide correct outputs and explain them to the user, following an AI-driven interaction. Forward reasoning can provide more agency to human users, involving both human and AI in the decision loop in order to address the issue of system uncertainty <cit.>.

Such guidelines, frameworks and taxonomies can help AI designers to select appropriate types and methods for HAI interactions, based on the system requirements and interaction goals. Designing interactions which comply with high-level guidelines can be challenging, especially when multiple requirements need to be met. Design patterns can be used to structure the interaction by defining smaller parts and combining them into more complex interactions.

§.§ Design Patterns for Human-AI Interactions

Describing an HAI interaction using design patterns is not straightforward but it can simplify the design of complex AI systems and make them transparent in terms of their implementation requirements. Focusing on co-creation applications with Generative Adversarial Networks (GANs), Grabe at al. provide a list of HAI interaction patterns which categorize  interactions based on the AI's co-creativity support level <cit.>. These interaction patterns are sequences of actions which describe an activity, e.g. initialize, create, adapt model, and can be combined to design interactions with different co-creativity support levels. A co-creative framework for interaction design analyzes the interactions between users, AI models, and the shared product <cit.>, resulting to a set of design choices. The design choices can define the behavior of the AI, as a generative, improvisational, or advisor agent. Design patterns in HAI co-learning are used to design interactions which enable humans and AI to share knowledge and experience. Learning Design Patterns (LDP) refer to sequences of interactions that aim to initiate and facilitate the co-learning process <cit.>, either by identifying knowledge gaps that team members may have or by enabling team partners to learn from other team members. In the context of AI-based game design, AI can serve different roles based on the selection of the interaction pattern <cit.>. For example, AI can act as role-model, where users need to imitate an AI agent to complete a game, or as an (AI trainee) which enables the user to teach the agent to do something in the game.

Design patterns have also been proposed considering specific interaction paradigms. Focusing on collaborative learning and hybrid intelligence systems, humans and computers can learn from each other through an iterative process, combining human-in-the-loop with computer-in-the-loop interactions <cit.>. More specifically, users and computers can communicate through a set of learning process patterns: (a) decision support, (b) exploration, and (c) integration. Each pattern refers to different types of interactions between human users and computers as an exchange of inputs, outputs, and feedback/explanations. iML methods have been used as design materials for movement interaction design <cit.>, as an approach to design interactions with models based on human movements. The interaction is defined as a closed loop between designers and software which allows designers to provide information in the form of movement examples and parameters, and receive AI test outputs and visualizations. Through this loop, both AI and designers can learn from each other; designers can reflect on how movement parameters may affect AI behavior and AI can update its internal models based on the feedback received from the designer. However, even prototyping such interactions would require a lower-level specification of how users and models communicate and exchange information during such interactions at a user-model level.

Design patterns have been used to specify and categorize the interactions with hybrid intelligence and reasoning systems at a user-model level <cit.>. A modular approach is used to specify patterns of interactions of hybrid systems, including both data-driven (ML) and knowledge-driven (symbolic AI) approaches. Based on this approach, data (numbers, texts, tensors, and streams) and symbols (labels, relation, traces) are the two types of instances that can be used for AI model operations. The authors present a list of design patterns for hybrid AI systems and demonstrate how such patterns can be used to describe interactions from existing applications. Their proposed approach specifies the interactions in terms of model operations, including training, inference, and transformation, and can be extended to capture the concept of different types of human actors included in interactions with hybrid AI systems.

§.§ Design and Implementation of Human-AI interactions

In order to explore how users and AI models communicate and exchange information during HAI interactions, we review applications of HAI systems, focusing on the interaction design and implementation aspects. Advances in computational and learning methods, as well as the plethora of human-generated data has recently led to innovative AI systems which can be used by non-expert users for complex tasks. More specifically, OpenAI has released two types of deep learning models which generate new content based on user's input (promtps); DALL-E generates images based on a textual description <cit.> and chatGPT is a Large Language Model (LMM) which generates structured textual content based on user's prompts and questions[https://openai.com/blog/chatgpt/]. Despite their complex architecture and advanced learning methods, interactions with such models are straightforward; the user provides a prompt and the model returns the generated data. However, the implementation of the training process requires methods to integrate human feedback and expertise in order to facilitate model learning. The training process utilizes an HITL method to integrate human users to the learning process. More specifically, Reinforcement Learning from Human Feedback (RLHF) is used to integrate human expertise to the learning process. Based on this approach, human users provide different types of feedback based on the learning process step. During model initialization, human users demonstrate the desired optimal behavior of the model (response to a prompt). For model optimization, a reward model was trained based on user ranking of possible model outcomes (output) for a given prompt (input). Such an approach demonstrates how different types of users can interact with different types of models, considering both the goal of such interactions, e.g., interface design, and their technical implications, e.g., selection of learning/update mechanisms.

Interactions can become complex when designing for explainability and user feedback, even with less advanced or pre-trained models. Designing an XAI-based interaction can serve as a channel to communicate additional information about the agent's performance for a given concept. For example, model transparency (e.g., visualizing model's confidence) has shown to improve user's trust during an interaction with a virtual agent with speech recognition capabilities <cit.>. More specifically, human participants interacted with a virtual agent and an underlying speech recognition system. The virtual agent was used as a visualization means for XAI-based feedback to enhance user's understanding and trust of the speech recognition system. During these interactions, the user provides a model input (utterance) and the model communicates both its output and additional feedback (prediction/explanation). HITL-based interactions can allow for user feedback and control within an interaction in various ways <cit.>. The type of human feedback (as well as amount, frequency, granularity, etc.) should be inline both with the user's characteristics (e.g., role, expertise, preferences, etc.) and the model's characteristics (model architecture, learning/update rules, etc.).

HAI interactions can also involve multiple users with different roles, expertise and intentions. In such interactions, an AI model should be able to interact with multiple users and behave in a different way based on the user's characteristics. An interactive RL-based framework has been proposed for a personalized robot-based cognitive game which involves two different types of users <cit.>. A player (primary user) interacts with a robot (RL) through a game-based interaction (robot sets a game difficulty and user plays the game) and a supervisor (secondary user) who can remotely supervise and control the interactions. In terms of communicated information, the player provides implicit feedback to the model through task performance and engagement, while a supervisor can monitor the model's decisions through a informative UI and modify the robot's decisions, when needed. Both communication channels contribute to the model's learning updates and the decision making process in a different way. Similarly, the Human Experience Transfer Model (HETM) <cit.> combines two interaction loops; one for a human expert (trainer) who guides the model learning process and one for the human learner (trainee) who can provide feedback during the learning interaction. The trainer loop aims to facilitate the model training while the learner loop aims to personalize the learner's interaction. These loops specify two different interactions with different communication channels and model update methods. Such approaches for designing and implementing HAI interactions highlight the need to specify design patterns considering the underlying implementation requirements. Our work moves towards a design space which takes into consideration the technical aspects while designing such human-model communications.

§.§ Communicative Protocols

Based on the Semantic Interaction paradigm <cit.>, both types of interacting agents, humans and models/systems, should exchange information in a mutually understandable way, to ensure meaningful communicative acts from both sides. In order to do so, user actions and intentions should be translated into an understandable model-specific format. On the other hand, model actions should be effectively communicated and understandable to human users. In the domain of communicative protocols, process calculus provides a tool for the high-level description of interactions, communications, and synchronizations between agents or processes. Agent modeling and communication languages make use of communicative acts to enable agents to communicate their intent for a specific service <cit.>. For example, the Knowledge Query and Manipulation Language (KQML) and the Foundation for Intelligent Physical Agents agent communication language (FIPA-ACL) are two major developments in message exchange interaction protocols between agents <cit.>. Such protocols aim to encode the communicative acts of an agent, as well as to model the agents' knowledge using semantics. More specifically, the FIPA-ACL defines a set of (primitive and composite) communicative acts, along with a formal definition of the underlying semantic model. The semantic language formalism allows for the specification of the mental model of the agent (e.g., belief, uncertainty) and the effects of the acts on the interacting agents. A set of primitives defines how agents communicate information, including an assertive inform act, based on which an agent provides a message (in a proposition form), and a directive request act, which describes a request of sender (for an action from the receiver). Taking into consideration multi-agent coordination and social norms, Lightweight Coordination Calculus (LCC) can be used to specify the behaviours required of agents interacting in a given social context <cit.>, as a form of `electronic institution' <cit.>. This formalisation has been extended to model coordination and communication between multiple interacting actors in social context <cit.>. Our approach is inspired by such communicative protocols; however, our aim is to encode the interactions capturing the intent and the type of the exchanged information and not the effects of the act on the underlying semantic models of the interacting agents.

§ PROPOSED APPROACH: UNPACKING HUMAN-AI INTERACTIONS INTO INTERACTION PRIMITIVES

In this section, we present our approach towards a design space for HAI interactions. Our goal is to identify common interaction patterns from HAI interactions and specify them in terms of the type and intent of the communicated information. We provide a semi-formal definition of interaction primitives and we demonstrate how they can be used to define interactions between users and models. The outcome of the proposed approach is a list of interaction patterns and the actions that define them, along with a description of the interaction in terms of design and implementation aspects. We discuss how the proposed formalization can be used towards a design space.

§.§ Unpacking Human-AI interactions to patterns and primitives

Our approach aims to unpack HAI interactions into low-level communicative acts or interaction primitives which can specify the type and intent of the communication during a single interaction step. Such interaction steps are defined as user-model interactions, e.g., user provides an input, model provides an output, etc.. Our motivation is that unpacking HAI interactions can provide us with insights about the underlying design and technical challenges for HAI interaction patterns. In order to unpack HAI interactions into interaction primitives, we identify the interaction patterns between users and models and we specify the intent and type of the communicated information. The goal of this approach is to identify a set of communicative acts from different HAI interaction scenarios, towards a formalization for HAI interaction primitives. We demonstrate our approach using an interactive robot learning system for multimodal emotion recognition as a running example <cit.>. More use cases were selected for unpacking in order to cover different types of interactions, including XAI, HITL and hybrid intelligence interactions (See Appendix section <ref>). We provide a description of the HAI interactions in the form of patterns and actions, and we highlight the design and implementation aspects of the interactions (Figure <ref>).

Interactive robot learning <cit.> using descriptions of interaction patterns and primitives. The image shows the original interaction schema as a box and arrow diagram, followed by a high level description of the HAI interactions patterns, the unpacking into actions involving the exchange of information between the human and the system, and a description of the interaction requirements.

Description of the HAI interactions. The goal of this system is to collect and annotate human-generated data for an emotion classification model, through the human-robot interaction. In terms of interaction design, the interaction starts with an emotion elicitation session (user watches a clip which invokes a specific emotion). After the session, the user is asked to select their current emotional state from a list of emotions. This emotional state is provided as a model output (class) and it is used by the robot for the interaction as the ground truth emotion. The robot asks the user to walk towards and stand in front of it, demonstrating the selected emotion. The robot uses the gait/thermal data to predict the user's emotional state. If user's response (ground truth) is different from the predicted emotion, the robot asks the user about their current emotional state, annotating the collected gait/thermal data which are used to retrain the model.

Interaction Patterns. We unpack the HAI interactions into three types of interactions between the user and the robot (model): (a) class selection: model asks user to select an emotion from a list (class) – user selects an emotion from the list, (b) new class sample: robot asks user to provide an input by demonstrating the selected emotion – user responds by walking and standing in front of the robot, and (c) annotate sample: model makes a prediction and asks the user for labeling, if prediction is different from ground truth – user provides the correct label by responding to the robot's question. During these interactions, user and robot exchange information in the form of model input (gait/thermal data) and model output (emotion from list, model prediction, robot question, user response). Model input is communicated as a set of raw data generated and captured during user's activity (walking and standing). Model output is communicated (a) through user's selection from a list, (b) implicitly through the robot's (model) prediction, and (c) as a user response to the robot's request during their interaction.

Design and implementation aspects. The robot runs an underlying emotion classification model which uses walking (gait data) and facial expressions (thermal data) as model inputs to predict the user's emotional state. The system utilizes the interaction with the user in order to dynamically improve the classification model by retraining on new (labeled) data. The design of the interactions enables the user to participate in the interactive learning process in an implicit way. The prediction model consists of two models (gait and thermal models) and the final prediction is estimated through a modified confusion matrix. After each interaction with the user, the model is retrained including the new data from the user as an input-output pair. The robot initiates the interactions to request for user's input (emotion label, input data and user response), and uses the responses to make interaction decisions; if model's prediction is different from user's input, the model implicitly requests the correct label from the user without communicating its own prediction. A possible limitation of this implementation is that it highly depends on the quality of user's input. User may provide inaccurate information both during the selection (label) and the demonstration (sample) of the emotion.

§.§ Defining Interaction Primitives and Patterns for Human-AI interactions

Our motivation is to characterize HAI interactions in terms of the communicated information. In order to describe such interactions, we need to define how users and models interact with each other at a human-model level to either provide or request information. Based on the Semantic Interaction paradigm <cit.>, both interactive agents should be able to communicate in an understandable and meaningful way. Following the taxonomy of instances for (hybrid) AI systems <cit.>, model-specific information can be in different formats, including train/test data, learning and estimated parameters, symbols, rules, labels, and others. Moreover, user feedback and explanations can be communicated in a model-specific format. Based on the model specifications, user actions are translated through the user interface into machine-readable information. Considering these different types of communication (Figure <ref>), we define a set of interaction primitives to specify user/model actions in terms of the intent (provide or ask for information) and the type (format) of the communication. We demonstrate how the proposed formalization can be used to design interactions as exchanges of messages between the interacting agents. The descriptive formalization, along with a visual representation is shown in Figure <ref>. We provide a description of the definitions, as well as examples to demonstrate our approach.

Communication types during human-model interactions. AI models provide/receive information in a model-specific format: (a) input (test/train data, relations, hyperparameters, etc.), (c) output (labels, predictions, estimated parameters, projections, etc.) and (c) feedback (explanations, validation, requests, etc.). Users send/receive information through the UI which translates user actions to a model-understandable format and vice versa. Our approach aims to specify this communications  using interaction primitives.

Definitions and visual representation for interaction primitives, types and actions. An action is defined as a primitive specified by a set of operations. Actions are communicated through messages, by specifying the interacting agents and the modifiers of the message. Actions are reusable and can be used by different messages. Sequences of messages can form patterns of interaction.

§.§.§ Interaction primitives and types

We define an interaction primitive as a low-level communicative act which specifies the type and intent of the communicated information. Considering the intent during an interaction step, the communicating agents can either provide or request information. We define a provide primitive as P: provideX:type+,Y:type* to describe the act of providing information (provide type  referencing type ). Similarly, a request primitive is defined as R: requestX:type+,Y:type* to describe the act of requesting information (request type  referencing type ). For both primitives, the optional argument () is used to reference an additional type. Considering the format of the information, human users and models can provide and request the following types of information: , , and . The definition of the types (and subtypes) can inform the design and implementation of the specific interaction, since they characterize how users and models exchange information.

We provide an overview of the types (and their possible subtypes) with examples.

*  is information to be fed into a model. The subtype depends on the model type and architecture, including numeric vectors, world state information, images, video frames or sequences, user preferences, audio, text and so on. Model-specific parameters (hyperparameters) are also used as model input, e.g., learning rate, number of clusters, model sensitivity, etc.

*  is information coming out of a model — class labels,  feature estimations, lists of recommendations, selected actions, generated images, projections of the input space, cluster labels, lower dimensional data and so on. This can also include learned parameters such as model weights, confidence values, loss etc.

*  refers to the additional information that can be provided both from users and models, including (requests for) evaluation, validation, explanations, etc. Explanations can be provided in different modalities, e.g., salient maps or natural language. Users can provide explanations to justify their decisions, as well as feedback for a model's decision. Similarly with model input/output, feedback is model-specific.

The proposed interaction primitives and types can describe different actions between users and AI models. Primitives capture the intent of the communication (provide/request) and types describe the format of the communicated information. For example, actions u1: provideX:input and u2: requestY:output, X:input can both describe a user communicating a model input type. However, the first action describes a user providing , while the second one describes a user request for  given the provided input. These actions are similar in terms of the provided input () but they differ on the type of the action (provide input vs. request output).

3|c|Examples of interaction primitives and types
1|l|provideX:input     1l|provide an input                        upload/capture an image
1|l|provideX:output,Y:input     1l|provide an output for a given input     detect a face in the image
1|l|provide[X:input,Y:output]     1l|provide an input-output pair            show an image and the detected face
1|l|requestX:output,Y:input     1l|request an output for an input                ask if there is a face in a given image
1|l|request[Y:input,X:output]     1l|request an input-output pair                  ask for an image with a detected face (if any)
1|l|provideZ:output,[Y:input,X:output]     1l|provide output for an input-output pair       modify an existing bounding box on an image
1|l|requestF:feedback,Y:output     1l|request feedback for the given output     ask for confirmation about a bounding box
1|l|requestX:[input]     1l|request a set of inputs     ask for a set of input images

Examples and descriptions of interaction primitives and types

The selection of the types (and subtypes) depends on the model specifications and can inform design and implementation choices. For example, given that provideX:input.raw_data can describe a user who communicates a model input in the form of raw data, the interaction design (e.g., interface) should enable the user to interact with raw data (e.g., images). Primitives and types can also provide information about the requirements related to the goal of the interaction. Let us consider a face recognition model which takes as an input an image (raw data or extracted features) and detects faces (if any) in the form of bounding boxes. For this case, both of the defined actions requestY:output;X:input and requestY:output, [X:input, Z:output] can describe a request for an output (detected face image). The first one is an output request for a given input image, and can describe a face detection interaction. The second action is an output request given an an input-output pair (image - bounding box) and can describe a modification request for the model's detection. Table <ref> provides examples of interaction primitives and types along with a description in the context of face detection.

§.§.§ Actions, operations and modifiers

In order to specify an interaction between a user and a model for a given interaction context, we define a message as senderreceiveractionmod* to describe the communication of an action from a sender to a receiver. An action is defined as  and specifies a primitive action by adding a description of how the arguments need to be communicated, through a set of . The operations contextualize an action by specifying the preconditions for its communication in a given interaction context. More specifically, the operations define how the argument is being created and describe the relations between multiple arguments. A list of operations includes, but is not limited to:

* : argument  is selected (from a given set  - optional argument) – this operation can describe the selection of an item from a list or set of choices, i.e., recommendations, labels, samples, etc.

* : argument  is mapped to argument  –  this operation can be used to describe a model prediction (e.g., classification, regression, clustering), human labeling, evaluation, etc.

* : modify argument  to argument  – this operation can describe a modification of a sample (modify input image), an alternate decision (change label), etc.

* : create new argument  – this operation can describe data acquisition or generation (e.g., image, sound), a human annotation  (e.g., new label), etc.

Finally, a set of  (mod: key: type, val=any) can be used to further characterize the message in terms of interaction requirements, as a free-form annotation feature. Modifiers can provide information about the interaction modality, interface elements (e.g., buttons, forms, etc), type of communication (e.g., explicit vs. implicit), etc. Based on the above, the definition of an action includes the primitive (provide/request), the type (input, output, feedback), the operations needed to communicate the types, as well as additional information for the interaction through the modifiers, and describes the communication of an action as a single message from a sender to a receiver. A sequence of actions for a given interaction context is defined as an interaction pattern.

The proposed formalization allows for a custom definition of an action and its specification as a message for a given interaction context. For example, we can define the action $\equiv$$\leftarrow$ to describe a request for a generated input in the form of raw data. Sequences or exchanges of messages can be used to define interactions between users and models for a given context. For interactions with a face recognition model, we can define a message   to describe the model's request for an image from the user. As a response to this request message, the user could respond with either:

-  ,
where: $\equiv$$\leftarrow$, or

-  ,

where:  $\equiv$

Based on the first message, the user responds by providing the requested input (create/capture image), while with the second message, a request is made to the model for face detection given the generated input (map/assign the captured image to a label). The same action can be communicated by different messages. A message specifies the communication of an action from a sender to a receiver in a given interaction context. For example, a speech recognition model can communicate its request for user-generated input (speech) using  . From this, a set of actions can be defined as a vocabulary which can be used in different HAI interactions. Such actions and messages can be used to design interaction patterns as sequences of messages between users and models. Given the message definitions above, we can define $\equiv$ as a query pattern to describe the request and communication of a model input, while $\equiv$ could describe an interaction where the user awaits for the model's decision based on a new sample. These patterns serve different interaction goals and also require different implementation. In the next section, we provide a set of action definitions and interaction patterns from existing HAI interactions and frameworks.

§.§ Designing interaction patterns using actions and messages

In this section, we describe how the proposed primitives can be used to define actions and interaction patterns. Such patterns serve a given goal during the interaction and can be applied for the design of other interactions. For the interactive robot learning for emotion recognition (Section <ref>), we define the following messages and actions (Table <ref>):

2|c|Message     1c|Action definition
1|l|     c]@l@

1|l|     c]@l@

1|l|     c]@l@

1|l|     c]@l@

1|l|     c]@l@

1|l|     c]@l@

Messages and action definitions for the interactive robot learning interactions

Description of the HAI interactions using definitions of interaction primitives and patterns. The diagram illustrates our unpacking approach and the process of describing interaction primitives and patterns in terms of the interaction requirements.

Interaction Patterns. The defined messages and actions can semi-formally specify: (a) the intent of the sender's act (primitive), (b) the type of the communicated information (types and operations), and (c) how the information is communicated (modifiers). Based on the unpacking of the interactions, we define the following interaction patterns:  ,  and  (Table <ref>). All patterns describe a query-response interaction initiated by the robot (model). The goal of the first pattern is to set the target class, without requiring any information about the model input. The class represents the user's emotional state selected from a predefined list of emotions (after an emotion elicitation activity). The goal of the second pattern is to receive a model input for the target class (emotion), resulting to a training example (input-output pair). The robot asks the user to demonstrate the selected emotion (model input - waling and standing). For the third pattern, the model captures the gait/thermal data and makes a prediction for the user's emotion. This prediction is not communicated to the user but it is used for the design of the interaction; if the prediction is inaccurate, the robot interacts with the user and uses the response to retrain its model. These patterns can be used to design the interactions between the human user and the model (Figure <ref>).

Design and implementation aspects. The selection of the patterns and the actions that define them can provide insights about the design and implementation aspects. For example, designing the  interaction requires an interface for the user to choose an emotion so it can be communicated to the model (robot). For the  pattern, the model needs to capture the gait/thermal data and process them for a model prediction (feature extraction). In terms of technical aspects, the implementation does not consider user's input uncertainty, making the assumption that the user provides the correct data (model input and output). Such choices can affect both the performance of the model and its interaction with the user. Different actions or patterns can lead to different types of interactions between users and models, e.g., negotiation, which may require different methods for implementation, e.g., shared decision making.

pattern     actions     message description
2*class-selection          model asks user for a class from list
user selects a class from a list
2*new_class_sample          model asks user to provide a sample for the given class
user provides a sample for the given class
2*sample-annotation          model asks user to annotate sample
user provides a label for the sample

Interaction patterns for the interactive robot learning interactions

Based on the unpacking process of HAI interactions to primitives and patterns on this running example, we compiled a list of three patterns, and the actions that define them, which can describe the interactions between the user and the robot. Following the proposed formalization for actions and patterns, this approach aims to (a) characterize existing interactions, and (b) modify or design new interactions. Our motivation is to compile a list of commonly used actions and patterns, towards a design space for HAI interactions.

§.§ A list of interaction patterns and actions

Based on the unpacking of HAI interaction use cases (Appendix <ref>) and review on frameworks (Section <ref>), we compiled a list of actions, messages and interaction patterns. We provide a set of action definitions with their description (Table <ref>). Such actions can be further specified as messages and used for a given interaction context. For example,  describes the annotation of a sample during a classification task, as the mapping of input sample X to output label Y. A similar action, , describes an RL policy as the mapping of input state S to output action A. This list is not exhaustive since it does not cover all possible actions and patterns, but rather provides examples from a range of HAI interaction scenarios. More actions can be defined following the proposed formalization, resulting to an extendable library of actions. While there are different ways to define these actions and messages for the selected use cases, the goal of this approach is to specify the intent and type of the communicated information considering existing interaction concepts and HAI interaction paradigms.

1|c|Action Definition     1|c|Description
3*c]@l@ $\equiv$

$\rightarrow$        3*

request the selection of a class Y from list L

c]@l@ $\equiv$

$\rightarrow$      select a class Y from a list L
c]@l@ $\equiv$

$\rightarrow$       ask for a new sample X for a given class Y
c]@l@ $\equiv$

$\rightarrow$       select a sample X of a given class Y
c]@l@ $\equiv$

$\rightarrow$      ask for the class Y of a given sample X
c]@l@ $\equiv$

$\rightarrow$      request the annotation of a generated sample
c]@l@ $\equiv$

$\rightarrow$      request the annotation of a selected sample
c]@l@ $\equiv$

$\rightarrow$      annotate a given sample
c]@l@ $\equiv$

$\rightarrow$      show selected action A for state S
c]@l@ $\equiv$

$\rightarrow$      select a reward R for the mapping from S to A
c]@l@ $\equiv$

$\rightarrow$       modify prediction of X from Y to Z
c]@l@ $\equiv$

$\rightarrow$        show a list of candidate samples CS based on sample S
c]@l@ $\equiv$

$\rightarrow$      select sample X from a list of samples CS
c]@l@ $\equiv$

$\rightarrow$      modify a sample from X to M
c]@l@ $\equiv$

$\rightarrow$      modify a sample from X to M
c]@l@ $\equiv$

$\rightarrow$      modify model parameter from P to M
c]@l@ $\equiv$

$\rightarrow$      modify a feature vector from X to M
c]@l@ $\equiv$

$\rightarrow$      ask for feedback F to evaluate the mapping of X to Y
c]@l@ $\equiv$

$\rightarrow$      select feedback F to evaluate the mapping of X to Y
c]@l@ $\equiv$

$\rightarrow$      provide explanation for the mapping of X to Y

A list of action definitions and their description, extracted by unpacking existing HAI interactions into interaction primitives.

Following the proposed definitions, we provide a collection of interaction patterns (Table <ref>). Each pattern is described as a sequence of messages based on the action definitions and the unpacking process of the selected HAI interactions. The selection of the messages and actions play an important role to the definition of an interaction pattern. For example,  and  can both describe a model's request for an output. However, the first action specifies the selection of a class from a list, while the second action requires a class given an input sample. The selection of an action for a given pattern depends on the goal of the interaction, as well as the design and technical requirements. Combining different actions can lead to patterns with different interaction goals. For example, model queries for (informative/evaluative) advice aim to improve model's performance, while XAI-based interaction patterns can be used to support the user by providing justifications about model predictions. This collection captures a range of interaction patterns in terms of the interaction concept and requirements. Combinations of interaction patterns can serve multiple goals and concepts. For example, providing explanations to users (XAI) can enhance the quality of human feedback (HITL) resulting to collaborative learning systems. The long-term goal of this research is the formalization of a new design space for HAI interactions which will support designers to explore, modify, and apply interaction patterns by providing design and implementation choices towards the prototyping of new types of interactions between human users and AI models.

Patterns     Actions     Description
2*class-selection          request for a class from a list
select a class from a list
2*new_sample          ask for a new input sample
generate an input sample
2*new_class_sample          request a new sample for a given class
generate a sample for a given class
2*sample-annotation          ask for the class of a given sample
select a class for a given sample
2*new_sample-annotation          ask for a new input sample
ask for the class of a new sample
2*candidate_samples          ask for a set of candidate input samples
show a set of candidate input samples
2*sample-modification          ask for a modified input sample
modify an input sample
2*feature-modification          ask for a modified feature vector
provide a modified feature vector
2*parameter-modification          ask for a modified model parameter
modify a model input parameter
2*prediction-modification          provide a label for a sample
modify a prediction of a given sample
policy-visualization          show selected action for current state
ask for informative advice based on state-action
2-3          modify (or not) the selected action
2*evaluative_advice          ask for evaluative feedback for state-action
evaluate the state-action pair
2*prediction-based_XAI          ask for explanations for a given input-output
show explanation for a given input-output pair
2*outcome-evaluation          request evaluative feedback for a given outcome
provide evaluative feedback for a given outcome
2*prediction_parameters          request predictions with model output parameters
show predictions with model output parameters
3*turn_taking-evaluation          request a modified sample based on a generated input
provide a modified sample based on input
provide evaluative feedback based on outcome
3*prediction-with-XAI          select a sample
show explanation for the sample prediction
modify the outcome based on the input-output pair
3*recommendations          modify a model input parameter
show recommended items for a given model input
evaluate a selected recommended item (accept/reject)

A list of interaction patterns as sequences of the defined actions/messages.

§ FROM INTERACTION PRIMITIVES TO A DESIGN SPACE FOR HAI INTERACTIONS

In this section, we discuss how the proposed formalization can be used towards the definition of a design space for HAI interactions. We demonstrate how the proposed interaction primitives and patterns can be used as design materials to prototype HAI interactions. We highlight how differences in patterns can serve different interaction goals and concepts. The goal of the proposed design space for HAI interactions is to support AI designers and practitioners by providing appropriate design and implementation choices for a given interaction concept. Based on the literature review and the unpacking of existing HAI interactions, we provide an overview of how interaction patterns can be designed for given interaction paradigms, aiming to bridge the gap between high-level guidelines and implementation requirements.

§.§ Interaction primitives and actions as design materials

Human-centered AI approaches, including explainability, transparency, interactivity, and human control, have provided opportunities to design new types of interactions, e.g., model auditing and contestation, negotiation, shared decision making, and others. HAI interactions can utilize the available information provided by the models, apart from decisions and predictions. For example, model uncertainty can be measured, communicated and used as a design material through transparency <cit.>. AI designers and practitioners need to consider interactions with AI as a design material, based on its capabilities, limitations, and the challenges that may arise while designing for transparency, unpredictability, learning, and shared control <cit.>. Our proposed formalization for interaction primitives and patterns aims to enable designers and AI practitioners to design HAI interactions by exploring appropriate patterns for a given set of design and technical requirements. The following example illustrates two alternatives of a query pattern, where the user queries the model for a prediction using a model input - , receives the model's prediction, and provides a modified prediction - . Using the same user actions, we can define two query patterns based on two different model actions (Figure <ref>):

-  , and

Query Patterns. The user asks for a model prediction for a given input. Based on pattern P1, the model provides the prediction and the user chooses to modify the prediction. Based on pattern P2, the model makes an explicit request for a modified prediction.

In the first case (P1), the model provides the user with a prediction on user's input and the user contests the prediction by providing an alternate output. The second case (P2) describes a model which queries the user to provide an alternative output given its prediction on the user's input. In terms of design and implementation choices, the first query alternative may require a mechanism to decide who makes the decision based on the quality of user/model predictions and how to update the model based on user's prediction (hybrid decision making). For the second query, the model is designed to explicitly ask the user for an alternative output and could be an example of active learning, where the model requests human annotation for (uncertain) predictions. In that case, human's decision should have a larger effect on the model updates, compared to the first version. Through these two versions of a query, we demonstrate how primitives can be used as design materials to generate different patterns given an interaction goal. Both patterns could describe the interactive control feature for the design of contestable AI interactions <cit.>, which enable the users to intervene and modify a decision made by the system.

§.§ Prototyping interactions

A key aspect of the proposed design space is the ability to create and explore alternatives of interactions by selecting different patterns or actions. We consider a set of alternative designs for the interactive robot learning scenario <cit.>, using the defined interaction primitives and patterns as design materials. We demonstrate how different sequences of patterns can result to different types of interactions. For each design alternative, we provide a description of the design and technical aspects, as well as the interaction concept (Figure <ref>).

Design alternatives for the interactive robot learning interactions <cit.>. Different sequences of patterns result to a set of design alternatives for different interaction concepts. Each alternative is related to specific design/implementation aspects and requirements. D1 describes the original design. D2 describes a robot-initiated interaction for user feedback. Based on D3, user asks the robot for its prediction based on the generated sample which can be modified. D4 adds an XAI-based interaction for model explanations for the robot's prediction.

The first design (D1) represents the original interaction design (Figure <ref>); a robot-initiated interaction where the user provides an annotated sample in a query-response manner. The second design (D2) describes an interaction where the robot communicates its prediction for the user's emotion, followed by a request for user's validation. This is achieved by replacing the last pattern  with a pattern for evaluating the model's prediction . The third design (D3) can describe a system where the user asks the robot to make a prediction which they can modify. This design introduces two patterns to allow the user to request model's prediction for the generated sample  and modify it . The fourth design (D4) includes an additional XAI-based interaction pattern, where the model provides explanations about its prediction to the user .

These alternatives can characterize different types of interactions with respect to the goal of the interaction. The original design is proposed as an interactive robot learning approach where the goal is to evaluate and improve the model's predictions through its robot-initiated interactions with the user. The second design requires the user to be more active in the interaction through an interaction pattern for model's prediction and user's feedback (evaluation). Based on the third design, the user initiates the interaction for the model's prediction. Finally, adding the XAI interaction pattern enables the user to further interact with the robot for explanations.
These designs are also related to specific design and implementation aspects. The original design assumes that user's provided sample and self-reported emotion are accurate. The second one requires an interaction where the robot communicates the prediction and asks for user's evaluation. Such an interaction requires an appropriate learning mechanism which can integrate user's feedback to the learning process. The XAI-based interaction requires the implementation of a specific explanation method. These alternatives can also be related to different roles of users involved in the interaction. The first design (D1) has been proposed for an end-user who implicitly participates in the model training process. The fourth alternative (D4) could be a design for an interaction between the model and the developer for model evaluation, where XAI is used to enhance the user's understanding about the system capabilities, e.g., identify "hard-to-predict" emotions.

Considering multi-user HAI interactions, different types of users can participate in the interaction for different goals. For example, the robot-based game interactions <cit.> are divided into two interaction loops; player-AI and supervisor-AI interaction. Each loop serves a specific interaction goal. The player-AI interaction goal is to elicit feedback from the player implicitly in order to personalize the model's decisions, while the supervisor-AI loop aims to enhance safety through the interventions of a supervisor using a transparent interface. Considering these, different interaction patterns should be used to serve each goal (Figure <ref>).

Interaction Patterns for the robot-based multi-user interaction <cit.>. The robot visualizes its policy to the supervisor () who can alter it (). The robot receives implicit feedback from user's engagement . Both types of feedback are integrated to model learning updates.

The patterns describe the interactions between the robot, the player and the supervisor. The robot  adjusts its policy (difficulty selection) based on user's performance and engagement, and the supervisor's interventions. Two main design aspects of the proposed system are (a) transparency and explainability and (b) user feedback and control. Considering the player-AI interactions, the model communicates the selected difficulty through the robot's announcement and the player provides implicit feedback through task performance and engagement. For the supervisor-AI interaction, model's transparency through the UI aims to enhance user's decision making by providing appropriate interventions. Considering both types of interactions and their patterns, a learning mechanism is required to learn from both types of users in an online way so the robot can dynamically improve its policy and select the appropriate levels of difficulty. The goal of the interaction is to include both types of users in the personalization process.

Designing multi-user interactions may also require combining interaction patterns for different goals. Considering the framework for contestable AI <cit.>, we provide a description of possible interactions during model contestation, highlighting the different interaction concepts between users and the AI model. The framework follows the paradigm of mixed-initiative interaction, based on which human controllers and decision subjects can both participate in the decision making process. Decision subjects can interact with the system to negotiate a decision which affects them. Such decisions may be the outcome of decision support interactions between the model and the human controller (semi-automated decisions). In order to support such types of interactions, the proposed framework provides the following features: interactive controls, explanations, and intervention requests. Interactive controls enable both types of users to provide feedback to the system for different purposes. Explanations are used to provide justification for the model's decisions and aim to support the semi-automated decision making process. Intervention requests enable the decision subject to initiate a model auditing process. XAI-based interaction patterns can be used to: (a) make a user aware of a decision made by the system, (b) inform the user about how to contest a model decision and (c) provide an explanation to justify the decision of the system. HITL-based interaction patterns can provide both types of users with the ability to negotiate and even override AI decisions (interactive controls). Moreover, collaborative learning interactions can enable both users and system to learn from their interactions and augment their decision making towards hybrid intelligence.

Examples of Interaction Patterns for contestable AI interactions <cit.>. We describe three examples of interactions based on the proposed framework for mixed-initiative interactions between the AI system, the decision subject and the human controller. Each interaction describes a different contestation aspect using interactive controls, explanations and intervention requests.

Considering the above, we provide examples of interaction based on our defined actions and patterns (Figure <ref>). The first example is a combination of three interaction patterns and describes the communication between AI and the decision subject. During the interaction, the user asks for the model's prediction for a sample () and for explanations for this prediction (). The subject can utilize the explanations to evaluate the outcome. if needed (). The second example is a multi-user interaction, where the decision subject needs to generate a new sample (e.g., submit a form) and ask for a decision (). The AI provides its decision to subject who makes a request to the human controller for the modification of the decision (). Finally, the third example describes a semi-automated decision making process, where the human controller can provide a decision (considering the AI's output) and modify it based on the decision subject's evaluative feedback for the model's prediction ().

§.§ Interaction Patterns and Requirements

We demonstrated how the proposed formalization can be used towards prototyping HAI interactions using patterns. In order to implement such interactions, we need to consider the interaction goals that each pattern can serve. Towards this, based on our literature review (Section <ref>) and the extracted patterns and actions (See Tables <ref> and <ref>) from the unpacking process of HAI interaction use cases (See Appendix <ref>), we provide an overview of how interaction patterns can be used in the context of different concepts for XAI-based, HITL-based, and HI-based interactions. The goal of the overview is to identify possible relations between interaction patterns and requirements.

§.§.§ XAI-based interactions

Human-XAI interactions can be designed for several interaction concepts and goals (e.g., debugging, persuasion, decision support, etc.). Designing explainable and transparent models is not trivial, especially while considering the various parameters that can affect the interaction, e.g., user's expertise, perception and understanding, cognitive load, preferences, etc. From the unpacking process, we identified the following XAI-based patterns and interaction goals:

* XAI-based interactions can manage user's expectations about the AI model's behavior. A Meeting Scheduling Assistant <cit.> (See Appendix <ref> - Figure <ref> for description) uses explanations to calibrate user's trust and expectations about its model predictions. The model predicts if an email is a meeting request to help the user make a decision. The  pattern enables the model to be transparent by communicating its accuracy rate along with the prediction. The system visualizes the model's accuracy rate, as well as a set of prediction/explanation examples with different levels of uncertainty to enhance user's understanding.

* XAI-based interactions can enhance user's perception about the model's performance. In the context of active learning for emotion recognition <cit.> (See Appendix <ref> - Figure <ref> for description), XAI-based interactions aim to facilitate the selection of appropriate samples for model refinement. Based on the unpacking process of the proposed system, we identify two patterns with different goals: the  pattern aims to support the selection of appropriate samples for annotation by visualizing the confidence of predictions, and the  pattern is used to support the annotation process by providing additional information about the model's prediction for a given sample through visual explanations.

* Model transparency can support human trainers while providing feedback to iML models. Policy visualization <cit.> (See Appendix <ref> - Figure <ref> for description) serves as a transparency method to engage the user to provide feedback to the model during task performance. The model utilizes user feedback to facilitate its learning process, i.e., faster convergence to the optimal policy. Based on our unpacking approach, the  pattern is used to communicate the model's current policy, by visualizing the current state (input) and the selected action (output).

* Explanations can justify model's prediction based on user preferences. For an explainable Music Recommendation system <cit.> (see Appendix <ref> - Figure <ref> for description), explainable user (preference) models are used to enhance user's perception about their own preferences and how these affect model's recommendations. Considering this, the  pattern is used to provide predictions and justify them through explanations based on user's preferences. In terms of design aspects, different visualizations (and explanation methods) are required considering the individual characteristics of users, e.g., need for cognition.

§.§.§ HITL-based interactions
The goal of HITL methods is to efficiently integrate the human user to the learning and decision making process of an AI system. Human users can participate in the model's development and deployment phases. The selection of appropriate iML/HITL methods and approaches depends on several aspects of the interaction, including user role and expertise. We present a set of HITL-based patterns extracted from the unpacking process, considering the different interaction goals.

* HITL methods can be used for interactive data collection and labeling. For the interactive robot learning scenario for emotion recognition (Figure <ref>), the model utilizes human-robot interaction data in order to evaluate its predictions and collect training data. Model re-trains without making the user aware of their participation in the data collection, annotation and training processes. Based on our unpacking,  defines a human labeling action given a generated sample.

* HITL methods can be used for model improvement by relabeling uncertain predictions. In the context of active learning, the NOVA system (See Appendix <ref> - Figure <ref> for description) asks the user to select uncertain samples and modify their predictions. In terms of the interaction design, the  pattern enables the user to validate the model's prediction or provide a new label. Manual corrections are used to update the model. Since the user makes the final decision (label), XAI methods are used to enhance user's perception and, thus,  the quality of feedback.

* Feedback interfaces should be user-friendly and intuitive in order to ensure human feedback quality. Training an RL agent through human advice (Appendix <ref> - Figure <ref>) requires an appropriate learning methods to integrate human feedback. The interaction supports two types of feedback. The  pattern is used to receive human advice in the form of a corrective action and the  pattern describes the evaluation of the model's decision in the form of binary feedback. These types of feedback are integrated through different feedback interfaces and learning methods (policy/reward shaping).

* Users can control model predictions and  parameters. For the Meeting Scheduling Assistant (Appendix <ref> - Figure <ref>), HITL methods enable the user to (a) provide feedback for a prediction by accepting or rejecting it, and (b) control the model's sensitivity parameters through a UI slider, affecting the model's predictions. The  action enables the user to validate (or not) a prediction and the  action allows the user to adjust the model's sensitivity parameters until they are satisfied with the model predictions.

§.§.§ Collaborative Learning and Hybrid Intelligence
The goal of collaborative learning and hybrid intelligence interactions is to enable both humans and machines (AI systems) to learn from each other in a collaborative manner. Collaborative learning interactions can be designed by combining XAI-based and HITL-based interactions, enabling both users and AI models to exchange information while solving a task. Based on our unpacking process, we identify and discuss interaction patterns used in collaborative learning and hybrid intelligence interactions.

* User can control and evaluate the collaboration with a model. In the context of a robot-based collaborative sketching interaction <cit.> (See Appendix <ref> - Figure <ref> for description), both user and robot work together during the co-ideation process in a turn-taking interaction. Both agents generate ideas building on their partner's generated sketch. The model uses the captured image to generate a variation of the user's sketch - to provide alternative ideas. During the interaction, the user can control what the robot will capture as an input by moving the robot and also provide feedback for the generated outcome. The  pattern is repeated until the user is satisfied. The model uses image classification for the user's input and generates a variation of this. The goal of such interactions is for the user to explore and identify new insights, rather than to identify a correct solution.

* Model can support the user through semi-automated decision making. An interactive sound segmentation system <cit.> (Appendix <ref> - Figure <ref>) enables the user and the model to work together in order to complete an audio segmentation and annotation task. The model supports the user by providing a list of candidate samples which can be selected, edited and annotated by the user, towards a collaborative interaction. The  pattern requires a model mechanism to identify the candidate samples and a proper visualization to highlight the segments. Based on this visualization, users provide feedback to adjust the model's predictions.

* Explanations can be provided both by users and models to justify their predictions. In a game-based scenario <cit.> (See Appendix <ref> - Figure <ref> for description), XAI and iML methods are used to enhance user's trust about the model's decisions and allow for corrections. The  pattern describes an interaction where both user and model communicate their prediction justification in the form of rule-based XAI. In terms of the design and implementation of this interaction, an appropriate visualization of the rules is needed to enable the user to understand the reasoning of the model in order to provide appropriate modifications and justification.

Based on this overview, we can observe that each pattern can serve a specific goal within the interaction. For example, considering the XAI-based patterns, the  pattern is used to calibrate end-user's trust, while  is used to enhance user's understanding about a prediction. For HITL-based patterns,  enables a user to improve a model by providing annotations, while  is used as a control pattern which enables the user to alter the model's predictions until the user is satisfied with the decision. Considering possible commonalities and differences between patterns and goals, we envision a design space which can provide suggestions for interaction patterns based on a given interaction concept. This would allow for fast prototyping of interactions using patterns, considering the interaction goals and requirements.

§ DISCUSSION

§.§ Limitations

A basic limitation of our proposed formalization is that it describes interactions without providing information about how the interacting agents are affected by the communication of a message, or their underlying processes (e.g., predict or fit/update) and the dataflow during the interaction. Moreover, the current formalization allows for multiple ways to define a given interaction. In order to support the design and prototyping of HAI interactions following our approach, we need to introduce a formalization of the primitives and actions as design materials, including instances of objects and agents (user profiles, model cards, data sheets) and their operations (model operations, human decisions, etc.). For example, the  pattern (Table <ref>) includes an action for the generation of the new sample and an action for the sample annotation. These actions are related to specific model operations; generating a sample needs a  step so it can be used by the model for a  operation. Moreover, retraining a model by altering its predictions, e.g., , can be implemented as a  operation using updated training data.
Such formalization will allow for the design of interactions between multiple agents and objects (users with different roles, models/data with different levels of access, etc.). Another limitation of the current approach is that the XAI-based interactions can be further described considering the explanation method. Our current approach considers XAI as a separate type for the defined primitives. For example, LIME-based visualization <cit.> (Figure <ref>) communicates the important features/pixels of an image which affect the prediction. With our current approach, this interaction is described using the  type and the LIME method is defined as a modifier. In order to support designing with XAI and HITL methods, we will define an in-depth description of XAI/HITL-based interaction patterns based on existing taxonomies <cit.>. A mapping from interaction patterns to a set of implementation techniques will help us explore possible commonalities between design and implementation aspects. In order to address both limitations, our proposed design space will be informed by existing guidelines and frameworks related to the design and implementation of HAI interactions.

§.§ Envisioned Applications

We propose a design space which can support designers and AI practitioners to design and implement HAI interactions based on interaction primitives and patterns. More specifically, we envision a design space which can enable users to explore and choose between existing patterns and modify them towards new types of interactions. The design space will be developed as a prototyping tool by providing suggestions about the design and implementation aspects of existing and new patterns. Building on the existing list of patterns and actions (Tables <ref>, <ref>), an extendable collection of interaction patterns and their design/implementation choices will be used as design materials for more complex HAI interactions. A mapping from interaction patterns to common implementation aspects could support fast prototyping of HAI interactions, in the form of auto-generated code for basic model operations during HAI interactions, e.g., data operations, model predict, fit, etc.. A key aspect of the proposed space is to link design aspects with implementation choices for a given interaction concept. Towards this, we provide a short description of the implementation aspects of extracted patterns (Section <ref>), aiming to identify possible commonalities between patterns and implementation issues. Finally, we discuss how the proposed space can be informed by existing frameworks and guidelines.

§.§.§ Manifesting implementation concerns

A main motivation for the proposed design space is the need to bridge the gap between design and implementation choices. This can be achieved by characterizing the defined interaction patterns in terms of interface and implementation requirements. We envisage that developing a collection of interaction patterns as well-defined, named entities, can enable us to extract a list of common implementation issues attached to a given interaction pattern. Towards this, we provide an overview of the implementation aspects for the defined patterns, considering the different HAI interaction paradigms.

XAI-based interaction patterns. Based on the literature review and the unpacking process, we identify the following implementation aspects and challenges regarding XAI-based interaction patterns: (a) the selection of appropriate explanation strategies and methods considering the interaction goals <cit.>, (b) the adaptation of explanations based on user's roles and characteristics (e.g., expertise, perception, etc.) <cit.>, and (c) the evaluation of XAI methods in terms of the interaction goal and user's behavior <cit.>. These can be related to the patterns found above:

* The  is used to help a non-technical user to understand if they can trust the model's prediction or not. This requires the model to communicate its accuracy rate along with a prediction, in an intuitive way (e.g., chart) in order to manage user's expectations about the model decisions.

* The  requires a proper explanation method considering the role of the user. For a domain expert user, it needs to provide appropriate information towards altering the model's decisions, while for a non-technical user, it considers the user's characteristics, e.g., cognitive load or preferences. The accuracy of the explanations is a key factor towards an effective interaction.

* The  is used to help the user identify the weaknesses of the model, e.g., prediction with low confidence. This requires the model to communicate its confidence values along with its predictions for a set of input samples, towards a scrutable model. The user is able to explore and alter the model decisions, providing feedback for model updates.

* The  is used to communicate the model's policy in an online fashion. In terms of implementation, the model communicates and updates its policy (input-output) during the interactions. This requires an online policy update mechanism to enable the user get an understanding of how the model's performance changes over time.

HITL-based interaction patterns. The goal of HITL methods and approaches is to efficiently integrate the human user to the learning and decision making process of an AI system. Human users can participate in the model's development and deployment phases. The selection of appropriate iML/HITL methods and approaches depends on several aspects of the interaction, e.g., goal, user role and expertise, etc. For a given context, HITL-based patterns can be used to design interactions where the user is part of the decision making and learning process. We identify the following implementation challenges for HITL/iML-based interactions: (a) the selection of teaching strategies considering user roles and expertise <cit.>, (b) the design of intuitive and user-friendly feedback/control interfaces to ensure high-quality feedback <cit.>, and (c) the integration of feedback (models) to model updates and decision making <cit.>. We can relate these to the patterns above as:

*  is used to enable the user provide a label for a sample. A key decision relates to the quality of the user-provided data. For the interactive robot learning, the model considers the human label as the ground truth for the generated sample, based on which it performs the learning update (online training). User is able to provide training data implicitly during the interaction.

*  is used to enable a user to evaluate the model's performance. Evaluation feedback requires an interface for numerical feedback, as well as a reward shaping mechanism to integrate human feedback into model updates. User needs to be able to perceive and evaluate the model's predictions (policy) in an online manner.

*  is used to enable provide an alternate decision. Informative advice requires an interface for action selection, as well as a policy shaping mechanism to integrate human feedback into model updates. User needs to be able to perceive and modify the model's predictions (policy) in an online manner.

*  is used as a control pattern which enables the user to alter the model's decisions by changing a model's parameter. In terms of implementation, the model should be able to dynamically alter its decisions based on the modified input. The model uses feedback only to alter its predictions for a new parameter and not to update its learning weights.

Hybrid Intelligence and Collaborative learning interaction patterns
The goal of collaborative learning and hybrid intelligence interactions is to enable both humans and machines (AI systems) to learn from each other in a collaborative manner. Collaborative learning interactions can be designed by combining XAI-based and HITL-based interactions, enabling both users and AI models to exchange information while solving a task. Based on our unpacking process, we identify and discuss interaction patterns used in collaborative learning and hybrid intelligence interactions. Implementation challenges for hybrid intelligence and collaborative learning systems include, but not limited to: (a) identify the appropriate levels of human control and AI automation both for model learning and decision making <cit.>, (b) design systems to facilitate the interaction between explainable artificial and cognitive intelligence <cit.>, and (c) develop methods for the adaptation of HI systems considering user needs and capabilities to enhance user's perception towards improving the model's learning process <cit.>. Considering these, we identify the following challenges for the extracted patterns:

* The  pattern is used to enable the user collaborate with a model in a turn-taking interaction. In terms of implementation, the model needs to capture the user's input during the interaction and generate a modified sketch. The user can control what the robot will capture, as well as when to provide feedback and terminate the interaction.

* The  pattern is used to support the human labeling process in a semi-automated way. The user makes the final decisions based on the model's initial decisions. The model is being updated based on user's feedback. The model needs to make online updates to improve the selection of the candidate samples, which can affect the user's and thus model's performance.

* The  pattern enables both model and user to justify their decisions using explanations. The implementation of this pattern requires an interface where the user can modify the model's prediction and explanations. The type of explanations (visualization) depends on user characteristics, e.g., preferences, cognitive load.

Focusing on the implementation aspects of these patterns, we observe that each pattern can be linked to a set of technical aspects that need to be considered. We envision a library of patterns which can characterize a pattern in terms of the possible implementation choices to manifest common concerns and issues. In order to support both design and implementation choices, our proposed design space will be informed by existing frameworks and guidelines for HAI interactions.

§.§.§ Relation to existing frameworks and guidelines.

In order to get insights about the formalization of the proposed design space, we briefly discuss how existing design guidelines and frameworks are related to our proposed approach. Our vision is a design space which can support AI designers to explore this space between design guidelines and implementation practices by enabling the collaboration of such frameworks and guidelines (Figure <ref>).

Our proposed design space as a link between design guidelines and implementation choices.

For example, the Microsoft guidelines for Human-AI interactions <cit.> provide a set of pattern examples along with descriptions of interactions. These design guidelines can be used as a set of suggestions and pattern examples for the ideation phase of an HAI interaction system. For example, a guideline suggests helping the user understands what the system can do[https://www.microsoft.com/en-us/haxtoolkit/guideline/make-clear-what-the-system-can-do/]. One of the suggested pattern examples for this given guideline is to use explanation patterns in order to enable users to gain insights into system capabilities (XAI-based interaction patterns). Based on another guideline, the design should encourage granular feedback and enable the user to indicate their preferences[https://www.microsoft.com/en-us/haxtoolkit/guideline/encourage-granular-feedback/]. A suggested pattern for this guideline is to request explicit feedback on selected system outputs in order to assess the system and help it improve over time (HITL-based interactions). Our proposed design space could be informed by the provided guidelines and pattern examples in order to develop a collection of design patterns considering these guidelines. Working at a higher-level, the Assessment List for Trustworthy AI (ALTAI)[https://altai.insight-centre.org/] was developed for the assessment of AI systems in terms of seven requirements specified in the Ethics Guidelines for Trustworthy AI <cit.>. Human Agency and Oversight is one of the requirements and refers to the ability of human users to make informed decisions and to monitor and supervise the system. HITL-based interaction patterns can be used to integrate the user to the decision making and model learning process. Another ALTAI guideline is Transparency and refers to the ability of the data, system and AI models to be transparent and explainable to the user. XAI-based interaction patterns can describe different approaches to provide explanations to the user based on the role and intent in the interaction. Considering the assessment list and requirements, our proposed design space can provide support towards selecting appropriate patterns and interactions to comply with specific assessment items.

Apart from high-level guidelines, our proposed design space will be informed by technical and implementation frameworks for AI/ML models. Considering interactions with ML systems, implementation methods are required to enable (a) the communication of information between users and models and (b) the integration of user-provided data to the model learning (and decision making) mechanism. A classification of methods and approaches for interactive ML systems <cit.> considers the development lifecycle of ML systems and provides a list of implementation choices based on a given category of methods, including interactive learning and explainability. However, designing efficient HITL-based interaction patterns requires both the selection of appropriate design choices and learning methods for feedback integration. Focusing on detecting and mitigating bias in ML pipelines, FaiPrep is an open library which extracts dataflow representations to support fairness during model development <cit.>. Such representations can be used in our context to further characterize HAI interactions in terms of dataflows and model/data operations, e.g., fit, predict, update data. Finally, our proposed design space will be informed by existing approaches for design patterns for AI systems. Design patterns have been proposed for hybrid AI and reasoning systems, combining both data-driven and knowledge-driven AI models <cit.>. These hybrid AI patterns can specify the operations that take place during the interactions (e.g., fit, update, predict) and can provide guidelines for the selection of appropriate learning methods.

§ CONCLUSION

The design space developed here is aimed at the complex space between concepts and practice, between formality and accessibility. The interaction patterns developed here act as intermediate level knowledge for the design and understanding of HAI systems, giving a formal representation of the configurations used in existing work.  Starting from a small set of interaction primitives and types to specify the communicated information between the interacting agents, we showed that the proposed primitives can be used to describe patterns of interactions from a range of systems, resulting in a collection of crisply defined actions and interaction patterns. We demonstrated that these patterns and actions are consistent with key HAI paradigms of HITL, XAI and hybrid intelligence, and that they can be used to explore and prototype a range of alternate interactions for a given situation.

This space has been built  from theoretical ideas about communication between humans and models, based on ideas from agent communication languages and the semantic interaction framework for understanding human-model communication. It has then been developed and tested with examples of systems and interaction paradigms from the literature, demonstrating that it can meaningfully describe existing work. The representation language starts with data types and primitive communicative actions of providing and requesting information, and works up to high level conceptual activities — interaction patterns — that both capture common structures and describe the intent of the computational architectures. Extracting these patterns gives people of varying technicalities a common language to talk about what a particular system is doing, by building re-usable descriptions of the interactions taking place. This level of description allows for alternative design choices to be explored, while highlighting concerns that might arise and giving a framework for implementing the interactions. This provides a way to document existing practices, re-use well tested solutions and also speculate about new interaction possibilities through an exploration of the design space. Finally, through focussing on the interactive and communicative possibilities around models, the design space helps to shift thinking from a single user, single model, single purpose viewpoint to one where various stakeholders can carry out different kinds of interaction with a single model, creating a more ecosystemic view of human-model interactions.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Adhikari et al(2022)]
authorpersonAjaya Adhikari, personEdwin
Wenink, personJasper van der Waa, personCornelis
Bouter, personIoannis Tolios, and personStephan
Raaijmakers. year2022.
Towards FAIR Explainable AI: a standardized
ontology for mapping XAI solutions to use cases, explanations, and AI
systems. In booktitleProceedings of the 15th International
Conference on PErvasive Technologies Related to Assistive Environments.

[Ahmad et al(2007)]
authorpersonRaheel Ahmad, personShahram
Rahimi, and personBidyut Gupta.
An Intelligence-Aware Process Calculus for
Multi-Agent System Modeling. In booktitle2007 International
Conference on Integration of Knowledge Intensive Multi-Agent Systems. IEEE,

[Ahmed et al(2009)]
authorpersonMoamin Ahmed,
personMohd Sharifuddin Ahmad, and personMohd
Zaliman Mohd Yusoff. year2009.
A review and development of agent communication
journalElectronic Journal of Computer Science and
Information Technology volume1, number1

[Akata et al(2020)]
authorpersonZeynep Akata, personDan
Balliet, personMaarten De Rijke, personFrank Dignum,
personVirginia Dignum, personGuszti Eiben,
personAntske Fokkens, personDavide Grossi,
personKoen Hindriks, personHolger Hoos,
et al year2020.
A research agenda for hybrid intelligence:
augmenting human intellect with collaborative, adaptive, responsible, and
explainable artificial intelligence.
journalComputer volume53,
number08 (year2020), pages18–28.

[Ala-Pietilä et al(2020)]
authorpersonPekka Ala-Pietilä,
personYann Bonnet, personUrs Bergmann,
personMaria Bielikova, personCecilia Bonefeld-Dahl,
personWilhelm Bauer, personLoubna Bouarfa,
personRaja Chatila, personMark Coeckelbergh,
personVirginia Dignum, et al
booktitleThe assessment list for trustworthy
artificial intelligence (ALTAI).
publisherEuropean Commission.

[Alfrink et al(2022)]
authorpersonKars Alfrink, personIanus
Keller, personGerd Kortuem, and personNeelke
Doorn. year2022.
Contestable AI by Design: Towards a Framework.
journalMinds and Machines (year2022),

[Amershi et al(2019)]
authorpersonSaleema Amershi, personDan
Weld, personMihaela Vorvoreanu, personAdam Fourney,
personBesmira Nushi, personPenny Collisson,
personJina Suh, personShamsi Iqbal,
personPaul N Bennett, personKori Inkpen,
et al year2019.
Guidelines for human-AI interaction. In
booktitleProceedings of the 2019 chi conference on human
factors in computing systems. pages1–13.

[Arrieta et al(2020)]
authorpersonAlejandro Barredo Arrieta,
personNatalia Díaz-Rodríguez, personJavier
Del Ser, personAdrien Bennetot, personSiham Tabik,
personAlberto Barbado, personSalvador García,
personSergio Gil-López, personDaniel Molina,
personRichard Benjamins, et al
Explainable Artificial Intelligence (XAI):
Concepts, taxonomies, opportunities and challenges toward responsible AI.
journalInformation fusion  volume58
(year2020), pages82–115.

[Balayn et al(2021)]
authorpersonAgathe Balayn, personPanagiotis
Soilis, personChristoph Lofi, personJie Yang, and
personAlessandro Bozzon. year2021.
What Do You Mean? Interpreting Image
Classification with Crowdsourced Concept Extraction and Analysis.
In booktitleProceedings of the Web Conference 2021.
publisherACM, addressLjubljana Slovenia,

[Behymer and Flach(2016)]
authorpersonKyle J Behymer and
personJohn M Flach. year2016.
From autonomous systems to sociotechnical systems:
Designing effective collaborations.
journalShe Ji: The Journal of Design, Economics, and
Innovation volume2, number2 (year2016),

[Bhatt et al(2021)]
authorpersonUmang Bhatt, personJavier
Antorán, personYunfeng Zhang, personQ Vera Liao,
personPrasanna Sattigeri, personRiccardo Fogliato,
personGabrielle Melançon, personRanganath
Krishnan, personJason Stanley, personOmesh Tickoo,
et al year2021.
Uncertainty as a form of transparency: Measuring,
communicating, and using uncertainty. In
booktitleProceedings of the 2021 AAAI/ACM Conference on AI,
Ethics, and Society. pages401–413.

[Bignold et al(2022)]
authorpersonAdam Bignold, personFrancisco
Cruz, personRichard Dazeley, personPeter Vamplew,
and personCameron Foale. year2022.
Human engagement providing evaluative and
informative advice for interactive reinforcement learning.
journalNeural Computing and Applications
(year2022), pages1–16.

authorpersonAbeba Birhane.
Algorithmic injustice: a relational ethics
journalPatterns volume2,
number2 (year2021), pages100205.

[Cavalcante Siebert et al(2022)]
authorpersonLuciano Cavalcante Siebert,
personMaria Luce Lupetti, personEvgeni Aizenberg,
personNiek Beckers, personArkady Zgonnikov,
personHerman Veluwenkamp, personDavid Abbink,
personElisa Giaccardi, personGeert-Jan Houben,
personCatholijn M Jonker, et al
Meaningful human control: Actionable properties for
AI system development.
journalAI and Ethics (year2022),

[Chai and Li(2020)]
authorpersonChengliang Chai and
personGuoliang Li. year2020.
Human-in-the-loop Techniques in Machine Learning.
journalIEEE Data Eng. Bull. volume43,
number3 (year2020), pages37–52.

[Chen et al(2022)]
authorpersonValerie Chen, personUmang
Bhatt, personHoda Heidari, personAdrian Weller,
and personAmeet Talwalkar. year2022.
Perspectives on Incorporating Expert Feedback into
Model Updates.
journalarXiv preprint arXiv:2205.06905

[Chromik and Butz(2021)]
authorpersonMichael Chromik and
personAndreas Butz. year2021.
Human-XAI interaction: a review and design
principles for explanation user interfaces. In
booktitleIFIP Conference on Human-Computer Interaction.
Springer, pages619–640.

[Chromik and Schuessler(2020)]
authorpersonMichael Chromik and
personMartin Schuessler. year2020.
A Taxonomy for Human Subject Evaluation of
Black-Box Explanations in XAI.
journalExss-atec@ iui  volume94

[Cui et al(2021)]
authorpersonYuchen Cui, personPallavi
Koppol, personHenny Admoni, personScott Niekum,
personReid G Simmons, personAaron Steinfeld, and
personTesca Fitzgerald. year2021.
Understanding the Relationship between Interactions
and Outcomes in Human-in-the-Loop Machine Learning.. In
booktitleIJCAI. pages4382–4391.

[Dellermann et al(2021)]
authorpersonDominik Dellermann, personAdrian
Calma, personNikolaus Lipusch, personThorsten Weber,
personSascha Weigel, and personPhilipp Ebel.
The future of human-AI collaboration: a taxonomy of
design knowledge for hybrid intelligence systems.
journalarXiv preprint arXiv:2105.03354

[d'Inverno et al(2012)]
authorpersonMark d'Inverno, personMichael
Luck, personPablo Noriega, personJuan A.
Rodriguez-Aguilar, and personCarles Sierra.
Communicating Open Systems.
journalArtificial Intelligence
volume186 (dateJuly year2012),

[Dove and Fayard(2020)]
authorpersonGraham Dove and
personAnne-Laure Fayard. year2020.
Monsters, Metaphors, and Machine Learning.
In booktitleProceedings of the 2020 CHI Conference on
Human Factors in Computing Systems (seriesCHI
'20). publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages1–17.

[Dove et al(2017)]
authorpersonGraham Dove, personKim Halskov,
personJodi Forlizzi, and personJohn Zimmerman.
UX design innovation: Challenges for working with
machine learning as a design material. In
booktitleProceedings of the 2017 chi conference on human
factors in computing systems. pages278–288.

[Dudley and Kristensson(2018)]
authorpersonJohn J Dudley and
personPer Ola Kristensson. year2018.
A review of user interface design for interactive
machine learning.
journalACM Transactions on Interactive Intelligent
Systems (TiiS) volume8, number2
(year2018), pages1–37.

[Edwards and Veale(2017)]
authorpersonLilian Edwards and
personMichael Veale. year2017.
booktitleSlave to the Algorithm? Why a
'right to an Explanation' Is Probably Not the Remedy You Are Looking For.
typePreprint. institutionLawArXiv.

[Ehsan and Riedl(2020)]
authorpersonUpol Ehsan and personMark O
Riedl. year2020.
Human-centered explainable ai: Towards a reflective
sociotechnical approach. In booktitleInternational
Conference on Human-Computer Interaction. Springer,

[Enarsson et al(2022)]
authorpersonTherese Enarsson, personLena
Enqvist, and personMarkus Naarttijärvi.
Approaching the human in the loop–legal
perspectives on hybrid human/algorithmic decision-making in three contexts.
journalInformation & Communications Technology
Law volume31, number1 (year2022),

[Endert et al(2012)]
authorpersonAlex Endert, personPatrick
Fiaux, and personChris North.
Semantic interaction for sensemaking: inferring
analytical reasoning for model steering.
journalIEEE Transactions on Visualization and
Computer Graphics volume18, number12
(year2012), pages2879–2888.

[Finin et al(1994)]
authorpersonTim Finin, personRichard
Fritzson, personDon McKay, and personRobin
McEntire. year1994.
KQML as an agent communication language. In
booktitleProceedings of the third international conference
on Information and knowledge management. pages456–463.

[Giaccardi and Redström(2020)]
authorpersonElisa Giaccardi and
personJohan Redström. year2020.
Technology and more-than-human design.
journalDesign Issues volume36,
number4 (year2020), pages33–44.

authorpersonMarco Gillies.
Understanding the role of interactive machine
learning in movement interaction design.
journalACM Transactions on Computer-Human
Interaction (TOCHI) volume26, number1
(year2019), pages1–34.

[Grabe et al(2022)]
authorpersonImke Grabe, personMiguel
González-Duque, personSebastian Risi, and
personJichen Zhu. year2022.
Towards a Framework for Human-AI Interaction
Patterns in Co-Creative GAN Applications.

[Grønsund and Aanestad(2020)]
authorpersonTor Grønsund and
personMargunn Aanestad. year2020.
Augmenting the algorithm: Emerging
human-in-the-loop work configurations.
journalThe Journal of Strategic Information
Systems volume29, number2 (year2020),

[Guo et al(2022)]
authorpersonLijie Guo, personElizabeth M
Daly, personOznur Alkan, personMassimiliano
Mattetti, personOwen Cornec, and personBart
Knijnenburg. year2022.
Building Trust in Interactive Machine Learning via
User Contributed Interpretable Rules. In booktitle27th
International Conference on Intelligent User Interfaces.

[Heimerl et al(2019)]
authorpersonAlexander Heimerl, personTobias
Baur, personFlorian Lingenfelser, personJohannes
Wagner, and personElisabeth André.
NOVA-a tool for eXplainable Cooperative Machine
Learning. In booktitle2019 8th International Conference on
Affective Computing and Intelligent Interaction (ACII). IEEE,

[Hirsch et al(2017)]
authorpersonTad Hirsch, personKritzia
Merced, personShrikanth Narayanan, personZac E Imel,
and personDavid C Atkins. year2017.
Designing contestability: Interaction design,
machine learning, and mental health. In
booktitleProceedings of the 2017 Conference on Designing
Interactive Systems. pages95–99.

authorpersonLars Erik Holmquist.
Intelligence on tap: artificial intelligence as a
new design material.
journalinteractions volume24,
number4 (year2017), pages28–33.

[Khadpe et al(2020)]
authorpersonPranav Khadpe, personRanjay
Krishna, personLi Fei-Fei, personJeffrey T.
Hancock, and personMichael S. Bernstein.
Conceptual Metaphors Impact Perceptions of
Human-AI Collaboration.
journalProceedings of the ACM on Human-Computer
Interaction volume4, numberCSCW2
(year2020), pages1–26.

[Kim and Pardo(2018)]
authorpersonBongjun Kim and personBryan
Pardo. year2018.
A human-in-the-loop system for sound event
detection and annotation.
journalACM Transactions on Interactive Intelligent
Systems (TiiS) volume8, number2
(year2018), pages1–23.

[Kocielnik et al(2019)]
authorpersonRafal Kocielnik, personSaleema
Amershi, and personPaul N Bennett.
Will you accept an imperfect ai? exploring designs
for adjusting end-user expectations of ai systems. In
booktitleProceedings of the 2019 CHI Conference on Human
Factors in Computing Systems. pages1–14.

[Liao et al(2020)]
authorpersonQ Vera Liao, personDaniel
Gruen, and personSarah Miller.
Questioning the AI: informing design practices for
explainable AI user experiences. In booktitleProceedings of
the 2020 CHI Conference on Human Factors in Computing Systems.

[Liao and Varshney(2021)]
authorpersonQ Vera Liao and personKush R
Varshney. year2021.
Human-centered explainable ai (xai): From
algorithms to user experiences.
journalarXiv preprint arXiv:2110.10790

[Lin et al(2020)]
authorpersonYuyu Lin, personJiahao Guo,
personYang Chen, personCheng Yao, and
personFangtian Ying. year2020.
It is your turn: collaborative ideation with a
co-creative robot through sketch. In booktitleProceedings
of the 2020 CHI conference on human factors in computing systems.

[Lindley et al(2020)]
authorpersonJoseph Lindley, personHaider Ali
Akmal, personFranziska Pilling, and personPaul
Coulton. year2020.
Researching AI legibility through design. In
booktitleProceedings of the 2020 CHI Conference on Human
Factors in Computing Systems. pages1–13.

authorpersonMichal Luria.
Designing Robot Personality Based on
Fictional Sidekick Characters. In booktitleCompanion of
the 2018 ACM/IEEE International Conference on Human-Robot
Interaction (seriesHRI '18).
publisherAssociation for Computing Machinery,
addressNew York, NY, USA, pages307–308.

[Maettig and Foot(2020)]
authorpersonBenedikt Maettig and
personHermann Foot. year2020.
Approach to improving training of human workers in
industrial applications through the use of intelligence augmentation and
human-in-the-loop. In booktitle2020 15th International
Conference on Computer Science & Education (ICCSE). IEEE,

[Martijn et al(2022)]
authorpersonMillecamp Martijn,
personCristina Conati, and personKatrien Verbert.
“Knowing me, knowing you”: personalized
explanations for a music recommender system.
journalUser Modeling and User-Adapted Interaction
volume32, number1 (year2022),

[Michael et al(2020)]
authorpersonChris J Michael, personDina
Acklin, and personJaelle Scheuerman.
On interactive machine learning and the potential
of cognitive feedback.
journalarXiv preprint arXiv:2003.10365

[Mohseni et al(2021)]
authorpersonSina Mohseni, personNiloofar
Zarei, and personEric D Ragan.
A multidisciplinary survey and framework for design
and evaluation of explainable AI systems.
journalACM Transactions on Interactive Intelligent
Systems (TiiS) volume11, number3-4
(year2021), pages1–45.

[Mosqueira-Rey et al(2022a)]
authorpersonEduardo Mosqueira-Rey,
personElena Hernández-Pereira, personDavid
Alonso-Ríos, personJosé Bobes-Bascarán, and
personÁngel Fernández-Leal.
Human-in-the-loop machine learning: a state of the
journalArtificial Intelligence Review
(year2022), pages1–50.

[Mosqueira-Rey et al(2022b)]
authorpersonEduardo Mosqueira-Rey,
personElena Hernández Pereira, personDavid
Alonso-Ríos, and personJosé Bobes-Bascarán.
A classification and review of tools for developing
and interacting with machine learning systems. In
booktitleProceedings of the 37th ACM/SIGAPP Symposium on
Applied Computing. pages1092–1101.

[Murray-Rust et al(2022)]
authorpersonDave Murray-Rust,
personIohanna Nicenboim, and personDan Lockton.
Metaphors for Designers Working with AI. In
booktitleDRS Biennial Conference Series.

[Murray-Rust et al(2015)]
authorpersonDave Murray-Rust, personPetros
Papapanagiotou, and personDave Robertson.
Softening electronic institutions to support
natural interaction.
journalHuman Computation volume2,
number2 (year2015).

[Nadj et al(2020)]
authorpersonMario Nadj, personMerlin
Knaeble, personMaximilian Xiling Li, and
personAlexander Maedche. year2020.
Power to the oracle? design principles for
interactive labeling systems in machine learning.
journalKI-Künstliche Intelligenz
volume34, number2 (year2020),

[Nunes and Jannach(2017)]
authorpersonIngrid Nunes and personDietmar
Jannach. year2017.
A systematic review and taxonomy of explanations in
decision support and recommender systems.
journalUser Modeling and User-Adapted Interaction
volume27, number3 (year2017),

[O'Brien and Nicol(1998)]
authorpersonPaul D O'Brien and
personRichard C Nicol. year1998.
FIPA—towards a standard for software agents.
journalBT Technology Journal volume16,
number3 (year1998), pages51–59.

authorpersonKieron O'Hara.
Explainable AI and the Philosophy and Practice
of Explanation.
journalComputer Law & Security Review
volume39 (dateNov. year2020),

[Peters et al(2020)]
authorpersonDorian Peters, personKarina
Vold, personDiana Robinson, and personRafael A
Calvo. year2020.
Responsible AI—two frameworks for ethical design
journalIEEE Transactions on Technology and Society
volume1, number1 (year2020),

[Ramesh et al(2021)]
authorpersonAditya Ramesh, personMikhail
Pavlov, personGabriel Goh, personScott Gray,
personChelsea Voss, personAlec Radford,
personMark Chen, and personIlya Sutskever.
Zero-shot text-to-image generation. In
booktitleInternational Conference on Machine Learning.
PMLR, pages8821–8831.

[Rezwana and Maher(2022)]
authorpersonJeba Rezwana and
personMary Lou Maher. year2022.
Designing Creative AI Partners with COFI: A
Framework for Modeling Interaction in Human-AI Co-Creative Systems.
journalACM Transactions on Computer-Human
Interaction (year2022).

authorpersonDavid Robertson.
A lightweight coordination calculus for agent
systems. In booktitleDeclarative Agent Languages and
Technologies II: Second International Workshop, DALT 2004, New York, NY, USA,
July 19, 2004, Revised Selected Papers 2. Springer,

[Schoonderwoerd et al(2022)]
authorpersonTjeerd AJ Schoonderwoerd,
personEmma M van Zoelen, personKarel van den Bosch,
and personMark A Neerincx. year2022.
Design patterns for human-AI co-learning: A
wizard-of-Oz evaluation in an urban-search-and-rescue task.
journalInternational Journal of Human-Computer
Studies  volume164 (year2022),

[Schwalbe and Finzel(2021)]
authorpersonGesina Schwalbe and
personBettina Finzel. year2021.
A Comprehensive Taxonomy for Explainable Artificial
Intelligence: A Systematic Survey of Surveys on Methods and Concepts.
journalarXiv e-prints (year2021),

authorpersonBen Shneiderman.
Human-centered artificial intelligence: Reliable,
safe & trustworthy.
journalInternational Journal of Human–Computer
Interaction volume36, number6
(year2020), pages495–504.

[Sperrle et al(2020)]
authorpersonFabian Sperrle,
personMennatallah El-Assady, personGrace Guo,
personDuen Horng Chau, personAlex Endert, and
personDaniel Keim. year2020.
Should we trust (x) AI? Design dimensions for
structured experimental evaluations.
journalarXiv preprint arXiv:2009.06433

[Subramonyam et al(2021)]
authorpersonHariharan Subramonyam,
personColleen Seifert, and personEytan Adar.
Towards a process model for co-creating AI
experiences. In booktitleDesigning Interactive Systems
Conference 2021. pages1529–1543.

[Teso et al(2023)]
authorpersonStefano Teso, personÖznur
Alkan, personWolfgang Stammer, and personElizabeth
Daly. year2023.
Leveraging explanations in interactive machine
learning: An overview.
journalFrontiers in Artificial Intelligence
volume6 (year2023).

[Treanor et al(2015)]
authorpersonMike Treanor, personAlexander
Zook, personMirjam P Eladhari, personJulian
Togelius, personGillian Smith, personMichael Cook,
personTommy Thompson, personBrian Magerko,
personJohn Levine, and personAdam Smith.
AI-based game design patterns.

[Tsiakas et al(2018)]
authorpersonKonstantinos Tsiakas,
personMaher Abujelala, and personFillia Makedon.
Task engagement as personalization feedback for
socially-assistive robots and cognitive training.
journalTechnologies volume6,
number2 (year2018), pages49.

[van Bekkum et al(2021)]
authorpersonMichael van Bekkum, personMaaike
de Boer, personFrank van Harmelen, personAndré
Meyer-Vitali, and personAnnette ten Teije.
Modular design patterns for hybrid learning and
reasoning systems.
journalApplied Intelligence volume51,
number9 (year2021), pages6528–6546.

[van Berkel et al(2021)]
authorpersonNiels van Berkel, personMikael B
Skov, and personJesper Kjeldskov.
Human-AI interaction: intermittent, continuous, and
journalInteractions volume28,
number6 (year2021), pages67–71.

[Wang et al(2019)]
authorpersonDanding Wang, personQian Yang,
personAshraf Abdul, and personBrian Y Lim.
Designing theory-driven user-centric explainable
AI. In booktitleProceedings of the 2019 CHI conference on
human factors in computing systems. pages1–15.

[Weitz et al(2021)]
authorpersonKatharina Weitz, personDominik
Schiller, personRuben Schlagowski, personTobias
Huber, and personElisabeth André.
“Let me explain!”: exploring the potential of
virtual agents in explainable AI interaction design.
journalJournal on Multimodal User Interfaces
volume15, number2 (year2021),

[Wenskovitch and North(2020)]
authorpersonJohn Wenskovitch and
personChris North. year2020.
Interactive Artificial Intelligence: Designing for
the" Two Black Boxes" Problem.
journalComputer volume53,
number8 (year2020), pages29–39.

[Wiethof and Bittner(2021)]
authorpersonChristina Wiethof and personE
Bittner. year2021.
Hybrid intelligence-combining the human in the loop
with the computer in the loop: a systematic literature review. In
booktitleForty-Second International Conference on
Information Systems, Austin.

[Wu et al(2022)]
authorpersonXingjiao Wu, personLuwei Xiao,
personYixuan Sun, personJunhang Zhang,
personTianlong Ma, and personLiang He.
A survey of human-in-the-loop for machine
journalFuture Generation Computer Systems

authorpersonWei Xu. year2019.
Toward human-centered AI: a perspective from
human-computer interaction.
journalinteractions volume26,
number4 (year2019), pages42–46.

[Xu et al(2022)]
authorpersonWei Xu, personMarvin J Dainoff,
personLiezhong Ge, and personZaifeng Gao.
Transitioning to human interaction with AI systems:
New challenges and opportunities for HCI professionals to enable
human-centered AI.
journalInternational Journal of Human–Computer
Interaction (year2022), pages1–25.

[Yang and Zhang(2019)]