# FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning,
and Emulator Assisted Tuning for Artificial Intelligence Foundation Models
with Mobile Edge Computing

Terence Jie Chua, Wenhan Yu  Graduate College
Nanyang Technological University
Singapore
{terencej001<EMAIL_ADDRESS>Jun Zhao, Kwok-Yan Lam  School of
Computer Science and Engineering
Nanyang Technological University
Singapore
{junzhao<EMAIL_ADDRESS>
###### Abstract

The emergence of foundation models including large language and computer
vision models has transformed the landscape of artificial intelligence,
offering unprecedented capabilities and versatility across a wide spectrum of
applications. However, the deployment and fine-tuning of these models present
unique challenges, particularly in the current era of foundation models. These
colossal models, such as GPT-3 and BERT, have the potential to revolutionize
industries ranging from healthcare to entertainment. Nonetheless, addressing
issues related to collaborative training, model ownership, and computational
limitations is imperative for realizing their full potential. We generalize
the offsite tuning approach to Emulator-Assisted Tuning (EAT) and combine it
with Parameter-Efficient Fine-Tuning (PEFT) to create Parameter-Efficient
Emulator-Assisted Tuning (PEAT), expanding its use into federated learning
(FL) as Federated Parameter-Efficient Emulator-Assisted Tuning (FedPEAT). Our
proposed FedPEAT framework uses adapters, emulators, and PEFT techniques for
federated model fine-tuning, offering a solution that enhances model privacy
and streamlines memory-efficient downstream fine-tuning. Adapters, featuring
trainable neural network parameters, customize pre-trained models for specific
tasks, while emulators provide compressed, fixed-parameter representations of
the original model. This combination not only addresses model privacy concerns
by obviating the need to transmit complete models to mobile edge devices but
also significantly improves memory and computational efficiency. Our approach
is adaptable to diverse neural network architectures and is complemented by an
adaptive control mechanism, utilizing deep reinforcement learning, to optimize
critical hyper-parameters. The mechanism ensures efficient resource
orchestration, making certain that devices possess the memory capacity and
computational capabilities required for effective fine-tuning. Our
experimental evaluation considers a special case of FedPEAT framework in which
the server possesses data and partakes in the collaborative federated
foundation model fine-tuning process instead of acting purely as an
aggregator. Through these experiments, we demonstrate the practical
applicability and efficacy of our proposed framework in addressing the complex
challenges associated with large language models in the current era of
foundation models.

###### Index Terms:

Foundation model, large language model, federated learning, parameter-
efficient fine-tuning, edge computing.

##  I Introduction

Foundation models. In the realm of artificial intelligence, the advent of
large language models represents a monumental leap forward in our
understanding of natural language processing. These models have transcended
traditional approaches to machine learning and have ushered in a new era, one
characterized by their vast capabilities and applicability across a plethora
of tasks. These models, pretrained on massive amounts of data, have the
ability to understand and generate images, texts, audio with remarkable
accuracy. GPT-3 [1], BERT [2], and similar models have become household names,
celebrated for their prowess in tasks ranging from natural language
understanding and generation to machine translation and summarization. Their
immense size, often involving hundreds of millions or even billions of
parameters, enables them to capture intricate linguistic nuances and exhibit
human-level performance in various tasks. The current age is characterized by
the ascendancy of these foundation models, with an ever-expanding suite of
applications across industries, from healthcare and finance to education and
entertainment. Large language models serve as versatile tools for a wide array
of downstream tasks, where they can be fine-tuned to excel in specific domains
or tasks. Fine-tuning adapts the pretrained model to a narrower context,
making it more specialized and effective in a particular application. For
instance, a large language model pretrained on general text data can be fine-
tuned to become an expert in medical diagnosis, legal document review, or
customer service chatbots. It should be noted that the pre-training of
foundation models focuses on self-supervised learning from large-scale
unlabeled data, capturing general language understanding. On the other hand,
fine-tuning involves adapting these pre-trained models to specific tasks using
task-specific labeled data, enabling specialized performance.

Figure 1: Here we illustrate the intersection of Emulator-Assisted Tuning
(EAT), Parameter-Efficient Fine-Tuning (PEFT), and Federated learning (FL).
The main contribution of our current paper is to introduce Federated
Parameter-Efficient Emulator-Assisted Tuning (FedPEAT), as a convergence of
EAT, PEFT, and FL, while EAT and Parameter-Efficient Emulator-Assisted Tuning
(PEAT) are also terms coined by our paper.

Motivation. One of the significant challenges associated with large language
models lies in the distribution of data. Many real-world applications
necessitate the utilization of data that resides on user devices, such as
smartphones, laptops, and mobile edge devices, rather than on centralized
servers. This decentralization poses logistical and privacy concerns.
Federated learning has emerged as promising solutions to these issues.
Federated learning is a decentralized machine learning approach that enables
model training without the need to centralize data. Instead of sending raw
data to a central server, federated learning trains models directly on the
user’s device. These models are then aggregated to create a global model,
preserving data privacy while achieving the desired performance. However,
large language models are often owned by research institutions or companies
that bear the responsibility of maintaining and updating them. These model
owners typically cannot directly share the entire model with external devices
due to various reasons, including privacy concerns, intellectual property
rights, and the potential for misuse. The lack of easy sharing mechanisms
hampers the democratization of large language models and their use in
applications that require continuous updates and fine-tuning. As a result,
there is a need to develop mechanisms that allow model owners to collaborate
with external parties or distribute portions of models securely. Furthermore,
fine-tuning large language models is computationally intensive. Training a
model with hundreds of millions or billions of parameters demands substantial
computational resources, often beyond the reach of individual users or small
organizations. This computational bottleneck can limit the widespread adoption
of these models and impede their deployment in resource-constrained
environments. Moreover, fine-tuning on local devices, such as smartphones or
edge devices, is often not feasible due to their limited computational
capabilities. Distributing the model fine-tuning process across devices while
ensuring data privacy and model performance adds another layer of complexity.

Figure 2:  FedPEAT overview

Proposed EAT structure. In addressing the pressing issues of model and data
privacy and ownership, as well as the imperative need for memory and
computation-efficient downstream model fine-tuning, we propose a novel
Emulator-Assisted Tuning (EAT) structure which generalizes the offsite tuning
approach introduced by Xiao et al. [3] to encompass all possible combinations
of adapter and emulator configurations for large foundation model fine-tuning.
Our proposed EAT structure offers the flexibility to adapt the adapter and
emulator to the specific requirements of a given application. The adapter and
emulator can take any form, whether encompassing layers within a transformer
architecture, multi-layer perceptron, or any other neural network structure.
The emulator, can have variable number of neural network layers, variable
number of nodes per layer, and even variable arrangements of transformer
attention units. This adaptability ensures that the model can be fine-tuned
efficiently across a wide spectrum of tasks, from simple to complex.

Expansion to PEAT architecture. In the field of model tuning, various
Parameter-Efficient Fine Tuning (PEFT) methods such as Low-rank Adapters
(LoRA) [4], prompt tuning [5], and adapters [6, 7, 8] have been explored to
make fine-tuning of pre-trained models more efficient. We combine EAT and
Parameter-Efficient Fine-Tuning (PEFT) to present Parameter-Efficient
Emulator-Assisted Tuning (PEAT).

FedPEAT framework. We extend the use of the PEAT into the domain of federated
learning (FL) and introduce a novel framework, Federated Parameter-Efficient
Emulator-Assisted Tuning (FedPEAT). This unique integration not only addresses
model and data privacy concerns by eliminating the need for the model owner to
transmit the entire model to the client and the client to send their local
data to the model owner but also substantially improves the memory and
computational efficiency of collaborative, downstream federated model fine-
tuning. We illustrate the intersection our proposed EAT, PEAT, and FedPEAT in
Fig. 1.

FedPEAT adaptive control mechanism. To optimize and streamline this adaptive
combination of adapters and emulators, we propose coupling them with an
adaptive control mechanism. This mechanism employs a deep reinforcement
learning orchestrator to control critical hyper-parameters, such as emulator
model compression ratios, adapter parameter-efficient fine-tuning parameters,
and even device selection for participation in collaborative federated
learning during each iteration. This integration facilitates the efficient
orchestration of resources, ensuring that the fine-tuning process remains both
memory and computation-efficient. This orchestration ensures that
participating devices possess the necessary computational resources to carry
out fine-tuning effectively. This contribution is essential in guaranteeing
the successful application of our model adaptation and fine-tuning technique
in real-world, resource-constrained environments.

Server-Device collaborative tuning. The FedPEAT framework is applicable to
collaborative federated learning of various contribution nature. We note two
distinct types of contribution cases. The first case involves federated
learning where all data resides on mobile edge devices (i.e., clients), with
no central server involvement. In this scenario, model tuning is entirely
performed on the client, while the server’s role is restricted to aggregating
adapter module parameters. The second case entails federated learning where
data is distributed across both client devices and a central server. Fine-
tuning occurs on both client devices and the server, presenting a more complex
but realistic setting that highlights the adaptability and versatility of our
proposed framework. In our experiments, we consider the special case of
FedPEAT framework in which the server possesses data and partakes in the
collaborative federated foundation model fine-tuning process instead of acting
purely as an aggregator. Through these experiments, we aim to demonstrate the
practical applicability and efficacy of our approach.

Our contributions can be summarized as follows:

  * •

We generalize the offsite tuning approach introduced by Xiao et al. [3] to
Emulator-Assisted Tuning (EAT), encompassing all possible combinations of
adapter and emulator configurations for large foundation model fine-tuning.

  * •

We combine EAT and Parameter-Efficient Fine-Tuning (PEFT) to present
Parameter-Efficient Emulator-Assisted Tuning (PEAT).

  * •

We expand the use of PEAT into the domain of federated learning (FL) and
introduce a novel framework, Federated Parameter-Efficient Emulator-Assisted
Tuning (FedPEAT), which forms the bedrock of privacy-aware federated
foundation model fine-tuning.

  * •

We propose an adaptive control mechanism which utilizes deep reinforcement
learning approaches that dynamically selects emulator compression parameters
and user-device participation to facilitate and empower the adoption of the
FedPEAT framework.

  * •

We considered user device memory constraints in our adaptive control mechanism
problem formulation, in light of the era of foundation models, which was
scarcely considered in previous federated learning works.

  * •

We considered and experimented with a special case of FedPEAT framework in
which the server possesses data and partakes in the collaborative federated
foundation model fine-tuning process instead of acting purely as an
aggregator.

##  II Related works

Foundation Models. Foundation models such as GPT-3 [1] and CLIP [9], which are
also known as large pre-trained models, have garnered attention for their
capacity to make zero-shot predictions and their ability to adapt to new tasks
through a transfer learning approach called fine-tuning [10, 11]. Leveraging
these models offers an advantage in terms of time and resource savings as
compared to training models from the ground up, especially for large models
like GPT3 with 175B+ parameters.

Model Tuning. In the field of model tuning, various methods have been explored
to make fine-tuning of pre-trained models more efficient. Efforts in model
tuning have extended to the realm of adapters [6, 7, 8], which encode task-
specific representations within intermediate layers while preserving pre-
training knowledge. Different techniques like Low-rank Adapters (LoRA) [4] and
Parameter-Efficient Fine Tuning (PEFT) have been proposed, encompassing
approaches such as prompt tuning [5, 12], prefix-tuning [13], adapters [8],
P-tuning V2 [14], tuning embedding layer inputs [15], tuning hidden states
[16], and more. These methods aim to update or add only a limited number of
model parameters, reducing resource requirements and allowing for the sharing
of parameters from the pre-trained model.

However, fine-tuning for downstream tasks of local devices often necessitates
knowledge of the entire model’s weights, potentially raising privacy concerns.
Furthermore, the process of fine-tuning and deploying foundation models can
pose significant resource challenges due to their substantial parameter sizes
[17, 18]. Xiao et al. [3] proposed an approach to fine-tune large foundation
models using the combination of an emulator, which is a compressed version of
a subset of the original large foundation model. Nevertheless, these authors
do not consider the federated and collaborative tuning between devices.

Federated Learning for Foundation Models. Several earlier works have proposed
federated learning as a privacy-preserving, decentralized machine learning
approach [19, 20]. In the domain of federated learning for foundation models,
researchers have acknowledged the substantial resource costs associated with
cross-device Federated Learning (FL) [21]. To address this challenge, multiple
strategies have been explored, encompassing communication efficiency
optimizations [22], model compression, quantization [23], client and data
sampling [24, 25], as well as on-device training speedup [26]. These efforts
aim to make FL more efficient and scalable across diverse devices and
datasets.

Federated learning, as a privacy-preserving technique, enables collaborative
model training without sharing user data with a central server [19]. However,
it raises concerns related to model privacy since each user maintains a local
copy of the entire model. Furthermore, federated learning assumes that users
can perform training on the complete model weights, which can be impractical
for large foundation models [3]. To tackle these abovementioned issues,
several works [27, 28, 29, 30, 31] proposed Federated-PEFT approaches.
Regarding the fine-tuning of federated models with emulators and adapters,
Ding et al. [32] introduced an approach that involves model compression and an
emulator-adapter-like strategy for collaborative tuning of large vision models
in a device-server setting. Kuang et al. [33] proposed FedOT, which is
federated version of offsite-tuning. Although Kuang et al. [33] briefly touch
upon an architecture similar to our FedPEAT, they do not provide detailed
discussions. In light of these, we propose the FedPEAT framework which
generalizes the emulator-adapter approach to all architectural configurations
for the federated learning context, and proposed an adaptive control mechanism
to support the adoption of FedPEAT.

##  III Parameter-Efficient Emulator-Assisted Tuning (PEAT) sub-units

In this section, we explore the intricacies of the PEAT sub-components,
specifically the emulator, adapter modules, and the Parameter-Efficient Fine-
Tuning (PEFT) adaptation.

###  III-A Emulator

The emulator represents a collection of neural network weights meticulously
designed to mimic the behavior of the original foundation model. Through the
compression of extensive neural network knowledge into a more compact
architecture, emulators aim to deliver performance that closely rivals their
larger counterparts while dramatically reducing computational and storage
requirements. The decision to share emulators with client devices, rather than
the original foundation model, serves a dual purpose: firstly, it safeguards
the proprietary nature of model ownership by obviating the need to divulge the
complete model to local devices; secondly, it empowers local devices to store
and undertake model fine-tuning using a significantly smaller-sized emulator.
In essence, an emulator serves as a streamlined and resource-efficient
rendition of a more extensive model, crafted through techniques such as
pruning [34], layer drop [35], or knowledge distillation [36]. Importantly,
our approach employs emulators with fixed-parameter values, without fine-
tuning, to encapsulate the bulk of knowledge and information derived from pre-
trained foundation models.

###  III-B Adapter

Adapters are modular additions to pre-existing foundation models like large
language model (LLM), designed to facilitate task-specific adaptations with
minimal modifications to the original model [3]. Essentially, adapters are a
smaller set of neural network weights with tunable parameters so as to encode
information at the user device for downstream task fine-tuning. The smaller
adapter size serves two main purpose. Firstly, the adapter is designed to be a
plug which can be conveniently placed at the end of the original foundation
model at the server and also a plug at the end of the emulator on the local
devices. Secondly, the smaller adapter size reduces adapter transmission
costs. By only tuning the parameters of these added layers, one can harness
the generalized capabilities of large models while efficiently tailoring them
for specific tasks.

###  III-C PEFT integration

PEFT methods like LoRA [4] and Adapter [8] can significantly reduce model
size, consequently save memory, while achieving comparable model performance
to a model which do not use PEFT approaches [3]. The integration of PEFT
methods is seamless and can be directly applied on the adapter module in each
federated learning iteration.

##  IV Federated Parameter-Efficient Emulator-Assisted Tuning (FedPEAT)

In this section, we will discuss our proposed innovative framework, known as
the Federated Parameter-Efficient Emulator-Assisted Tuning (FedPEAT). We offer
a thorough examination of the synergistic interaction between emulators and
adapters, illuminating their seamless integration with parameter-efficient
tuning techniques and their expansion into the domain of federated learning.

Figure 3: Emulator-Assisted Tuning generalized to three cases.

###  IV-A Emulators and Adapters

The server houses a foundation model $M_{\theta_{g}}$, while each user device
(UEs) labeled by index $i$ receives and holds:

$\displaystyle\begin{cases}\text{An adapter $A_{\phi_{i}}$ specifically tuned
for the downstream task.}\\\ \text{An emulator $E_{\theta_{i}}$, which is a
tailored version of the}\\\
\text{~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}foundational
model, represented by}\\\
\text{~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}$E_{\theta_{i}}=f(M_{\theta_{g}}-A_{\phi_{i}})$.}\end{cases}$

The adapter, denoted as $A$, aligns with the definition put forth by [3],
comprising sets of layers embedded within the foundation model’s architecture.
These layers feature tunable parameters, specifically designed to facilitate
model fine-tuning by encoding new information from downstream tasks. In
contrast, the emulators designated by $E$ encompass a (potentially) modified
rendition of the original foundation model. The adaptation of the emulator
occurs after the removal of the adapter layers and serves as a guiding
framework for tuning the adapter parameters. The parameter value of the
emulators are fixed and aims to emulate the large foundation models.

The transformation function $f()$, in this context, refers to model
compression algorithms such as layer dropping [37], model pruning [34].

Let $\omega$ represent the weights that are collaboratively trainable on both
the server and device. $\omega^{\prime}_{s}$ refers to the untrainable weights
specific to the server, excluding $\omega$. $\omega^{\prime}_{c}$ denotes the
untrainable weights specific to a device, distinct from $\omega$.

Given this, we can generalize emulator-assisted tuning to three cases:

  * •

Case 1:  $\omega^{\prime}_{s}\neq\omega^{\prime}_{c}\neq\varnothing:$ This is
our proposed, more generalized framework, in which we permit various user
devices (UEs) to employ distinct emulators, denoted as $E_{\theta_{i}}$. These
emulators correspond to the untrainable weights on a device,
$\omega^{\prime}_{c}$, which are maintained at fixed values. Similarly, the
subset of the model with untrainable parameters $E^{\prime}_{\theta_{i}}$
corresponds to $\omega^{\prime}_{s}$. This flexibility is particularly
important since UEs frequently operate with constrained storage and
computational resources. Therefore, emphasizing the efficient decompression
and adaptiveness of the foundation model becomes essential.

  * •

Case 2:  $\omega^{\prime}_{s}=\omega^{\prime}_{c}\neq\varnothing:$ This
scenario is a subset of Case 1. Here, the emulator designated for UE $i$
aligns with the static parameters of the overarching foundation model (i.e.,
$\omega^{\prime}_{s}=\omega^{\prime}_{c}$). In this setup, we synergize
Federated Learning (FL) training with Parameter-Efficient Fine-Tuning (PEFT)
techniques, reflecting strategies showcased in previous research such as [38,
39, 28, 30].

  * •

Case 3:  $\omega^{\prime}_{s}=\omega^{\prime}_{c}=\varnothing:$ This is
another specific instance within the purview of Case 1. In this scenario, all
participants utilize the adjustable parameters of the global foundation model
$\omega$. Essentially, no weights remain untrainable beyond the
collaboratively trainable ones (i.e.,
$\omega^{\prime}_{s}=\omega^{\prime}_{c}=\varnothing$). This methodology
closely mirrors the conventional federated learning (FL) paradigm, where
individual model parameters are amalgamated to shape the global model.

The details of the cases are further illustrated in Fig. 3.

###  IV-B Tuning Process

Prior to tuning. The server model, denoted as $M_{\theta_{g}}$, can be
decomposed into two primary components: the untrainable subset of weights of
the foundation model $E^{\prime}_{\theta}$, and the adapter $A_{\phi}$. After
such decomposition, the server model is expressed as $E^{\prime}_{\theta}\circ
A_{\phi}$, with the symbol “$\circ$” signifying the neural network connections
between $E^{\prime}_{\theta}$ and $A_{\phi}$. It is important to note that the
arrangement of layers $E^{\prime}_{\theta}$ and $A_{\phi}$ is flexible and can
be configured in various orders. Emulator-assisted tuning (EAT) is an approach
which generalizes all emulator-adapter based configurations, which include
those proposed by [3] and extends it to cases beyond those proposed by [3]
such as the “Vertical” splitting of the foundation model [32]. Furthermore,
the term “offsite” in their work [3] only considers a single device tuning and
does not consider a multiple device collaborative training scenario. Our
proposed EAT approach generalizes the emulator and adapter approach to
collaborative tuning between multiple devices. Furthermore, Xiao et al. [3] do
not consider a collaborative fine-tuning scenario where there are datasets
that are stored at the server and that the server is able to partake in
collaborative fine-tuning as proposed by [32]. The emulator-to-be, represented
as $E^{\prime}_{\theta}$, can be customized to create emulator
$E_{\theta_{i}}$ specific to UE $i$ taking into account UE $i$’s device
hardware configurations and conditions of its environment. Subsequently, this
tailored emulator, $E_{\theta_{i}}$, is distributed to each respective UE. In
our work, we extend our proposed PEAT approach to a collaborative, federated
model fine-tuning context and establish the Federated Parameter-Efficient
Emulator-Assisted Tuning (FedPEAT) framework.

Tuning. At the start of the first iteration, each adapter $A^{0}_{\theta_{i}}$
with randomly initialized parameter values is disseminated to UE $i$ where the
complete user device model for UE $i$ will be $E^{0}_{\theta_{i}}\circ
A^{0}_{\theta_{i}}$. Each user device UE $i$ will then carry out model
emulated-assisted fine-tuning with their local dataset $D^{t}_{i}$ and update
the parameter values of their adapter to produce $A^{1}_{\theta_{i}}$ with the
assistance of the emulator. Each user device UE $i$ will then upload their
adapter parameters to the server for adapter parameter aggregation as follows:

$\displaystyle
A^{t+1}_{\phi_{g}}=\frac{1}{\sum^{N}_{i=1}|D^{t}_{i}|}\cdot\sum^{N}_{i=1}(|D^{t}_{i}|\cdot
A^{t}_{\phi_{i}}),$ (1)

where $|D^{t}_{i}|$ is the size of the data being trained on at UE $i$. The
server will then disseminate this global adapter $A^{1}_{\phi_{g}}$. This
above mentioned tuning process proceeds for further iterations until model
convergence, or as defined by a specific criterion. This process can be
summarized in Algorithm 1.

Algorithm 1 FedPEAT with Adaptive Control

0: device set $\mathcal{U}$, iteration set $\mathcal{T}$, set of Adapters
$\mathcal{A}^{0}$, set of emulators $\mathcal{E}^{0}$

1: for iteration = $1,2...,T$ do

2: Adaptive control to set $\mathcal{U}$ vector and $\varepsilon$ parameter
based on Section V

3: Transmit $\mathcal{A}$ and $\mathcal{E}$ to devices

4: for device $i$ $\in$ $\mathcal{N}$ in parallel do

5: for epoch $\nu=1$ to $V$ do

6:
$A^{t^{\nu+1}}_{\phi_{i}}=\text{ModelTuning(}D^{t}_{i},E^{t}_{\theta_{i}},A^{t^{\nu}}_{\phi_{i}}$).

7: end for

8: Transmit local $A^{t}_{\phi_{i}}$ to server.

9: end for

10: Perform adapter parameter aggregation with equation (1) to obtain
$A^{t+1}_{\phi_{g}}$

11: end for

##  V FedPEAT Adaptive Control

The FedPEAT framework facilitates the collaborative, federated fine-tuning of
models for downstream tasks. However, successful adoption of the framework for
downstream task fine-tuning has to be achieved through adaptive control on the
hyper-parameters related to the FedPEAT framework, the PEFTs and the federated
learning process.

While the variables-of-concern, scenario, and environment will vary, we
designed a simple adaptive control system for the purpose of demonstration. To
illustrate the FedPEAT with Adaptive Control mechanism, we consider a
situation where UEs are moving within a fixed geographic space where the
channel gain between the user device and the server changes. The round-trip
transmission time taken for the transmission of adapters and downlink of
emulators between each UE $i$ and the server is
$Q^{t}_{i}=\frac{F^{t}_{i}}{r^{t}_{i}}$ where $r^{t}_{i}$ is the rate of
transfer between the user device $i$ and server, which depends on the server
transmission power $p^{t}_{i,trans}$ to UE $i$, channel gain $g^{t}_{i}$
between UE $i$ and the server, and the propagation model. For sake of
simplicity, we take $p^{t}_{i,trans}$ and $g^{t}_{i}$ to be arbitrary values
in each iteration. $F^{t}_{i}$ is the total adapter and emulator size
transmitted to and adapter size transmitted from UE $i$ at transmission step
$t$. We also omit the model fine-tuning computation time for the sake of
demonstration but the methodology would follow the works of [40] when
considered.

Therefore, we formulate the problem as follows:

$\displaystyle\min_{\varepsilon^{t},\mathcal{U}^{t}}\frac{1}{N}$
$\displaystyle\sum_{t=1}^{T}\sum_{i=1}^{N}P^{t}_{i}+f\cdot\sum_{t=1}^{T}\max_{i\in\mathcal{N}}Q^{t}_{i}+b\cdot\frac{1}{N}\cdot\sum_{t=1}^{T}\sum_{i=1}^{N}m^{t}_{i}$
$\displaystyle+s\cdot\frac{1}{N}\cdot\sum_{t=1}^{T}\sum_{i=1}^{N}\chi[E_{n}^{t}\neq
E_{n}^{t-1}],$ (2)
$\displaystyle\text{s.t.}~{}~{}m^{t}_{i}\leq\frac{1}{d}\cdot M,~{}\forall
i\in\mathcal{N},~{}\forall t\in\mathcal{T},$ (2a)
$\displaystyle\sum_{t=1}^{T}\chi[E_{i}^{t}\neq E_{i}^{t-1}]^{t}_{i}\leq
T\cdot\frac{1}{c},~{}\forall i\in\mathcal{N},~{}\forall t\in\mathcal{T}.$ (2b)

In the objective function (2) above, $P^{t}_{i}$ stands for the perplexity
score achieved by UE $i$ at iteration $t$, where it is a performance measure
for how well a language model predicts a set of data, $Q^{t}_{i}$ stands for
the total time taken for a single round of adapter and emulator transmission.
$\chi[E_{i}^{t}\neq E_{i}^{t-1}]$ represents the emulator exchange count, and
equals 1 if $E_{i}^{t}\neq E_{i}^{t-1}$, and 0 otherwise. Essentially, we
introduce a penalty for $\varepsilon$, $\mathcal{U}$, which stands for the
emulator compression parameter and UE selection vector, respectively. $t$, $T$
stands for the iteration, total number of training iterations, respectively.
$\mathcal{T}$ represents the set of iterations. $f$, $b$, and $s$ stands for
the weight balancing parameters for the perplexity term, transmission delay
term, and memory usage term, respectively. $m^{t}_{i}$ represents the memory
space taken for the model assigned to UE $i$, while $M_{i}$ represents the
total memory capacity of device $i$ at iteration $t$. $c$ and $d$ are
constants.

Essentially, the objective function (2) aims to minimize the sum of perplexity
scores across $t$ iterations which is synonymous with achieving a quicker rate
of model tuning convergence, and minimizing the maximum training time amongst
all devices for the federated fine-tuning process, via optimizing the emulator
compression parameter and device selection vector. For the sake of simplicity
in our demonstration, we use $\frac{1}{N}\sum_{i=1}^{N}P^{t}_{i}$ as an
estimate of global $P^{t}$. The motivation for a formulation as such is to
encourage quicker model convergence while ensuring that the maximum total
transmission delay among UEs, and memory consumption for all UE is minimized.
Constraint 2a ensures that the total memory consumed for any device in each
round falls well below a predefined fraction of its memory capacity.
Constraint 2b prevents excessive transmissions of emulator to reduce
transmission cost.

###  V-A Deep reinforcement learning approach

We have devised a multi-output, single-agent deep reinforcement learning
approach as our driver behind our adaptive control mechanism to tackle our
proposed problem as the problem is highly sequential and is a mixed-integer
non-linear programming problem. The ingenious configuration of the state,
action spaces, and reward function plays a crucial role in the effective
integration of reinforcement learning techniques. In the upcoming sections, we
will delve into a detailed examination of these three components in our
approach. We choose a multi-output single-agent deep reinforcement learning
approach over a multi-agent approach for simplicity of demonstration. The
choice of deep reinforcement learning agent type and structure is arbitrary.

####  V-A1 State

To effectively execute the FedPEAT approach, factors such as memory usage,
model-tuning performance and device participation is of utmost importance.
Therefore we included the following variables within the state: (1) user
device-server channel gain $g$ which is required for the computation of
$r^{t}_{i}$, (2) FedPEAT UE $i$ emulator exchange count $\chi[E_{i}^{t}\neq
E_{i}^{t-1}]$ which keeps track of the number of times UE $i$ has undergone
emulator exchange.

####  V-A2 Action

In this study, we have 2 actions to include in the agent action space: (1)
choice of emulator compression parameter $\varepsilon^{t}_{i}$ for each
device, stored in a vector
$\varepsilon^{t}=\\{\varepsilon^{t}_{i},\varepsilon^{t}_{2},...,\varepsilon^{t}_{N}\\}$,
(2) UE selection vector
$\mathcal{U}^{t}=\\{\text{UE}^{t}_{i},\text{UE}^{t}_{2},...,\text{UE}^{t}_{N}\\}$
for each device, stored in a vector.

####  V-A3 Reward

We formulate our reward function as per our objective function, where we
assign our reinforcement learning agent the reward as follows in each
iteration:

$\displaystyle\upsilon=$
$\displaystyle-\frac{1}{TN}\sum_{t=1}^{T}\sum_{i=1}^{N}P^{t}_{i}-f\cdot\frac{1}{T}\sum_{t=1}^{T}\max_{i\in\mathcal{N}}Q^{t}_{i}$
$\displaystyle-b\cdot\frac{1}{TN}\cdot\sum_{t=1}^{T}\sum_{i=1}^{N}m^{t}_{i}-s\cdot\frac{1}{TN}\cdot\sum_{t=1}^{T}\sum_{i=1}^{N}\chi[E_{n}^{t}\neq
E_{n}^{t-1}]$ (3)

In addition, we assign the agent very large penalties $\varkappa$ when (1) the
memory size of emulator $E^{t}_{i}$ and adapter $A^{t}_{i}$ exceeds an
allowable fraction of the local device $i$’s memory capacity, in accordance to
constraint 2a, and (2) the emulator exchange count $\chi[E_{i}^{t}\neq
E_{i}^{t-1}]$ exceeds a given fraction of the total iteration, in accordance
to constraint 2b.

####  V-A4 Reinforcement Learning Algorithm

We adopted the Proximal Policy Optimization (PPO) algorithm, developed by
OpenAI [41], which stands as an advancement over traditional policy gradient
algorithms. In the domain of sequential problems such as reinforcement
learning, even minor adjustments to parameters can have a profound impact on
performance, making parameter fine-tuning a challenging endeavor. PPO tackles
the issue of delicate and noisy advantage estimates by implementing a cautious
approach. It incorporates a Kullback–Leibler (KL) divergence penalty to
regulate policy adjustments. Furthermore, PPO makes use of an importance
sampling technique [42] by employing asynchronous policies for training and
data collection, enhancing overall efficiency. The loss function for the Actor
is formally defined as follows [41]:

$\displaystyle
L^{CLIP}(\varphi)=\mathbb{E}_{t}\left[\min\left(\upsilon_{\varphi}\varpi_{t},\text{clip}\left(\upsilon_{t}(\varphi),1-\epsilon,1+\epsilon\right)\varpi_{t}\right)\right].$

In this context, $\varphi$ represents the policy. $\mathbb{E}_{t}$ signifies
empirical expectations over the trajectory. $\upsilon_{t}$ represents the
ratio of the current policy to the old policy. $\varpi_{t}$ denotes the
estimated advantage at time $t$ and $\epsilon$ denotes the clip value. This
clipping mechanism acts as a safeguard, preventing significant bias and
ensuring that the policy remains within a trusted range.

(a) Memory consumption.

(b) Perplexity

(c) Transmission delay.

Figure 4: Memory consumption, Perplexity, and Transmission delay.

(a) Reward obtained.

(b) Emulator exchange count

Figure 5: RL reward and emulator exchange count.

##  VI Numerical Experiments

In this section, we will delve into our experimental configurations and offer
a comprehensive analysis of the obtained results.

###  VI-A Experiment configuration

We substantiate our study with several experiments by showing that the FedPEAT
framework with adaptive control works, and compare our proposed framework
against (1) FedPEAT without adaptive control and (2) Federated full model
fine-tuning (Fed-FT). The configuration settings are designed for simplicity
of demonstration and are arbitrary. We consider the scenario where the server
has data and partakes in UE-server collaborative federated learning.

To further simplify our workflow and for the ease of demonstration, we
utilized numerical solutions from the works by [3] to facilitate our
experiment. We utilized the GPT2-XL [9] large language model as the foundation
model, which has 1475 million parameters and is of approximately 6.5 gigabytes
(GB) in storage memory. We utilized the layer-drop approach [37] for the
emulator compression. We adopted the perplexity-layer drop retention numerical
solution from the works by [3] and established the function to be approximated
by $P=25.2\varrho^{2}-43.1\varrho+31.9$ for $0<\varrho\leq 1$, with an $R^{2}$
score of $0.97$, where $\varrho$ stands for the layer drop retention ratio and
varies from 0 to 1. We also adopted LoRA model compression numerical results
and perplexity improvements of using LoRA from the works by [3], and establish
the parameter compression ratio and perplexity improvement upon application of
LoRA to be $\frac{1}{300}$ and $-0.78$, respectively. We assume model storage
memory usage to follow a linear relationship with the number of parameters of
the model. Since the adapter layers are much smaller in size as compared to
the emulators, we take their sizes to be negligible.

We set $T$ which is the number of federated fine-tuning rounds in an episode
to be 100. We consider a scenario with 1 main server and 10 user devices. In
each round of federated fine-tuning, our adaptive control orchestrator selects
5 devices for fine-tuning. We set our large penalty $\varkappa$ to be -50.

We assign the bandwidth $B$ to be $100$ MHz and noise $\sigma^{2}$ to be
$-100$ dBm. We initialize and constrain main server power output and user-
device channel gain to $(0.0,10.0)$ Watt and $(-60,-120)$ dB, respectively.
$f$, $b$, and $s$ are set to 1, 100, and 1, respectively, and these numbers
are empirically derived with the aim of balancing the variables in the
objective function. We adopt the ADAM optimizer [43] for the algorithms
implemented in our study. The models are trained for 1,000,000 steps and
evaluated at every 200,000 steps.

###  VI-B Result analysis

Overall, our FedPEAT framework with adaptive control works relatively well as
shown in figure 5(a), where the average reward obtained by the adaptive
control orchestrator increases with training. This improvement is reflected in
the decrease in average model memory usage in local devices as training
proceeds, decrement in average perplexity (which is taken as a proxy for the
global perplexity) score, and emulator transmission delay (shown in fig
4(a),4(b),4(c)). Furthermore, the improvement in the adaptive control
orchestrator with training is noted by a reduction in average emulator
exchange count (shown in figure 5(b)), which signifies that the orchestrator
learns to not redundantly exchange and transmit new emulator weights if the
costs of transmission outweigh the improvements in perplexity or memory
consumption.

Nevertheless, we document the baseline results, along with our proposed
FedPEAT framework in Table I. Note that the steps displayed are in thousands
and the memory usage is measured in gigabytes. We can observe that FedPEAT
with adaptive control obtains the highest eventual reward. FedPEAT with
adaptive control performs better than the full model federated tuning (Fed-FT)
as the large foundation model consumes alot of memory storage space in local
devices and have high transmission cost. Furthermore, We observe that FedPEAT
works much better with adaptive control than without as the adaptive control
mechanism allows the FedPEAT algorithm to obtain more efficient use of memory
space, obtain better model performance, achieving a lower emulator
transmission delay while reducing the emulator exchange count.

TABLE I: Metric comparison with Baselines

Step(k) |  Reward |  Memory(Gb) |  Perplexity |  Delay(s) |  EE count
---|---|---|---|---|---
FedPEAT
$0$ | $-158$ | $0.0369$ | $31.5$ | $22.8$ | $100$
$200$ | $-128$ | $0.0347$ | $27.4$ | $17.1$ | $80$
$400$ | $-92.1$ | $0.0323$ | $23.6$ | $13.3$ | $52$
$600$ | $-68.7$ | $0.0310$ | $21.8$ | $8.85$ | $35$
$800$ | $-53.2$ | $0.0294$ | $19.9$ | $7.36$ | $23$
$1000$ | $-47.7$ | $0.0281$ | $18.4$ | $6.49$ | $20$
Full model federated tuning (Fed-FT)
$0$ | $-1030$ | $6.5$ | $13.58$ | $367$ | NA
$200$ | $-949$ | $6.5$ | $13.58$ | $286$ | NA
$400$ | $-891$ | $6.5$ | $13.58$ | $227$ | NA
$600$ | $-859$ | $6.5$ | $13.58$ | $195$ | NA
$800$ | $-838$ | $6.5$ | $13.58$ | $174$ | NA
$1000$ | $-814$ | $6.5$ | $13.58$ | $151$ | NA
FedPEAT w/o adaptive control
$0$ | $-157$ | $0.0359$ | $29.6$ | $23.7$ | $100$
$200$ | $-151$ | $0.0337$ | $25.8$ | $22.8$ | $100$
$400$ | $-153$ | $0.0345$ | $27.15$ | $24.1$ | $100$
$600$ | $-154$ | $0.0351$ | $28.2$ | $24.6$ | $100$
$800$ | $-154$ | $0.0348$ | $27.7$ | $21.1$ | $100$
$1000$ | $-151$ | $0.0339$ | $26.1$ | $22.3$ | $100$

##  VII Conclusion

In conclusion, deploying and fine-tuning large foundation models pose unique
challenges. Collaborative training, model ownership, and computational
constraints must be addressed to unlock their full capabilities. We generalize
the offsite tuning approach to Emulator-Assisted Tuning (EAT) and combine it
with Parameter-Efficient Fine-Tuning (PEFT) to create Parameter-Efficient
Emulator-Assisted Tuning (PEAT), expanding its use into federated learning
(FL) as Federated Parameter-Efficient Emulator-Assisted Tuning (FedPEAT). Our
proposed FedPEAT framework with adaptive control, a novel fusion of adapters
and emulators, offers a path forward by enhancing model privacy and
streamlining memory-efficient downstream federated fine-tuning. Adapters,
equipped with trainable neural network parameters, tailor models for specific
tasks, while emulators provide compressed, fixed-parameter representations.
This approach not only mitigates model privacy concerns by eliminating the
need to transmit complete models to edge devices but also substantially
improves memory and computational efficiency. It is adaptable to diverse
neural network architectures and is complemented by an adaptive control
mechanism leveraging deep reinforcement learning to optimize essential hyper-
parameters, ensuring efficient resource orchestration. Our experimental
evaluation, considering a collaborative user device-central server federated
learning scenario, demonstrates the practical applicability and effectiveness
of our approach in addressing the intricate challenges associated with large
language models in the current age of foundation models. Further work demands
that we work on studying the intricacies of federated learning for foundation
models empirically by performing a more complete range of federated learning
experiments across different model types and architectures, adaptive control
variables, and server-device configurations.

## References

  * [1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 1877–1901, 2020.
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [3] G. Xiao, J. Lin, and S. Han, “Offsite-tuning: Transfer learning without full model,” _arXiv preprint arXiv:2302.04870_ , 2023.
  * [4] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” _arXiv preprint arXiv:2106.09685_ , 2021.
  * [5] G. Qin and J. Eisner, “Learning how to ask: Querying LMs with mixtures of soft prompts,” _arXiv preprint arXiv:2104.06599_ , 2021.
  * [6] S.-A. Rebuffi, H. Bilen, and A. Vedaldi, “Learning multiple visual domains with residual adapters,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [7] J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, “Towards a unified view of parameter-efficient transfer learning,” _arXiv preprint arXiv:2110.04366_ , 2021.
  * [8] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 2790–2799.
  * [9] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [10] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned language models are zero-shot learners,” _arXiv preprint arXiv:2109.01652_ , 2021.
  * [11] N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. L. Scao, M. S. Bari, S. Shen, Z.-X. Yong, H. Schoelkopf _et al._ , “Crosslingual generalization through multitask finetuning,” _arXiv preprint arXiv:2211.01786_ , 2022.
  * [12] B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” _arXiv preprint arXiv:2104.08691_ , 2021.
  * [13] X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” _arXiv preprint arXiv:2101.00190_ , 2021.
  * [14] X. Liu, K. Ji, Y. Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang, “P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,” _arXiv preprint arXiv:2110.07602_ , 2021.
  * [15] S. An, Y. Li, Z. Lin, Q. Liu, B. Chen, Q. Fu, W. Chen, N. Zheng, and J.-G. Lou, “Input-tuning: Adapting unfamiliar inputs to frozen pretrained models,” _arXiv preprint arXiv:2203.03131_ , 2022.
  * [16] H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 1950–1965, 2022.
  * [17] S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti _et al._ , “Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model,” _arXiv preprint arXiv:2201.11990_ , 2022.
  * [18] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” in _International Conference on Machine Learning_. PMLR, 2023, pp. 38 087–38 099.
  * [19] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial Intelligence and Statistics_. PMLR, 2017, pp. 1273–1282.
  * [20] J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” _arXiv preprint arXiv:1610.05492_ , 2016.
  * [21] K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konečnỳ, S. Mazzocchi, B. McMahan _et al._ , “Towards federated learning at scale: System design,” _Proceedings of Machine Learning and Systems_ , vol. 1, pp. 374–388, 2019.
  * [22] C. Yang, Q. Wang, M. Xu, Z. Chen, K. Bian, Y. Liu, and X. Liu, “Characterizing impacts of heterogeneity in federated learning upon large-scale smartphone data,” in _Proceedings of the Web Conference 2021_ , 2021, pp. 935–946.
  * [23] J. Bernstein, Y.-X. Wang, K. Azizzadenesheli, and A. Anandkumar, “signSGD: Compressed optimisation for non-convex problems,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 560–569.
  * [24] F. Lai, X. Zhu, H. V. Madhyastha, and M. Chowdhury, “Oort: Efficient federated learning via guided participant selection,” in _15th USENIX Symposium on Operating Systems Design and Implementation (OSDI)_ , 2021, pp. 19–35.
  * [25] A. Li, J. Sun, P. Li, Y. Pu, H. Li, and Y. Chen, “Hermes: An efficient federated learning framework for heterogeneous mobile clients,” in _Proceedings of the 27th Annual International Conference on Mobile Computing and Networking_ , 2021, pp. 420–437.
  * [26] Q. Wang, M. Xu, C. Jin, X. Dong, J. Yuan, X. Jin, G. Huang, Y. Liu, and X. Liu, “Melon: Breaking the memory wall for resource-efficient on-device machine learning,” in _Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services_ , 2022, pp. 450–463.
  * [27] Z. Zhang, Y. Yang, Y. Dai, Q. Wang, Y. Yu, L. Qu, and Z. Xu, “FedPETuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models,” in _Annual Meeting of the Association of Computational Linguistics 2023_. Association for Computational Linguistics (ACL), 2023, pp. 9963–9977.
  * [28] H. Zhao, W. Du, F. Li, P. Li, and G. Liu, “FedPrompt: Communication-efficient and privacy-preserving prompt tuning in federated learning,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [29] J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, G. Wang, and Y. Chen, “Towards building the federated gpt: Federated instruction tuning,” _arXiv preprint arXiv:2305.05644_ , 2023.
  * [30] T. Guo, S. Guo, J. Wang, X. Tang, and W. Xu, “PromptFL: Let federated participants cooperatively learn prompts instead of models-federated learning in age of foundation model,” _IEEE Transactions on Mobile Computing_ , 2023.
  * [31] D. Cai, Y. Wu, S. Wang, F. X. Lin, and M. Xu, “FedAdapter: Efficient federated learning for modern NLP,” in _ACM 29th Annual International Conference on Mobile Computing and Networking (MobiCom)_ , 2023.
  * [32] Y. Ding, C. Niu, F. Wu, S. Tang, C. Lyu, and G. Chen, “DC-CCL: Device-cloud collaborative controlled learning for large vision models,” _arXiv preprint arXiv:2303.10361_ , 2023.
  * [33] W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y. Xie, Y. Li, B. Ding, and J. Zhou, “FederatedScope-LLM: A comprehensive package for fine-tuning large language models in federated learning,” _arXiv preprint arXiv:2309.00363_ , 2023.
  * [34] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” _arXiv preprint arXiv:1510.00149_ , 2015.
  * [35] H. Sajjad, F. Dalvi, N. Durrani, and P. Nakov, “On the effect of dropping layers of pre-trained transformer models,” _Computer Speech & Language_, vol. 77, p. 101429, 2023.
  * [36] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [37] M. Zhang and Y. He, “Accelerating training of transformer-based language models with progressive layer dropping,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 14 011–14 023, 2020.
  * [38] Z. Zhang, Y. Yang, Y. Dai, Q. Wang, Y. Yu, L. Qu, and Z. Xu, “FedPETuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models,” in _Findings of the Association for Computational Linguistics: ACL 2023_. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 9963–9977. [Online]. Available: https://aclanthology.org/2023.findings-acl.632
  * [39] J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, G. Wang, and Y. Chen, “Towards building the federated GPT: Federated instruction tuning,” _arXiv preprint arXiv:2305.05644_ , 2023.
  * [40] Y. Mao, J. Zhang, and K. B. Letaief, “Dynamic computation offloading for mobile-edge computing with energy harvesting devices,” _IEEE Journal on Selected Areas in Communications_ , vol. 34, no. 12, pp. 3590–3605, 2016.
  * [41] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017.
  * [42] H. Kahn and T. E. Harris, “Estimation of particle transmission by random sampling,” _National Bureau of Standards Applied Mathematics Series_ , vol. 12, pp. 27–30, 1951.
  * [43] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.


# Tell Me Something That Will Help Me Trust You:
A Survey of Trust Calibration in Human-Agent Interaction

George J. Cancro1,2 Shimei Pan1&James Foulds1 1University of Maryland,
Baltimore County, Information Systems Department
2Johns Hopkins University Applied Physics Laboratory, Laurel, MD
{gcancro1, shimei<EMAIL_ADDRESS>
###### Abstract

When a human receives a prediction or recommended course of action from an
intelligent agent, what additional information, beyond the prediction or
recommendation itself, does the human require from the agent to decide whether
to trust or reject the prediction or recommendation? In this paper we survey
literature in the area of trust between a single human supervisor and a single
agent subordinate to determine the nature and extent of this additional
information and to characterize it into a taxonomy that can be leveraged by
future researchers and intelligent agent practitioners. By examining this
question from a human-centered, information-focused point of view, we can
begin to compare and contrast different implementations and also provide
insight and directions for future work.

##  1 Introduction

Imagine you are a Hearing Examiner conducting a parole hearing and it is your
job to decide whether an inmate should be released from prison. To increase
consistency and improve decision-making, you have been provided a partner; an
intelligent agent who, upon reviewing the inmate’s record and other
attributes, provides a recommendation to either approve or refuse parole. In
one specific case, you recommend parole, but the intelligent agent recommends
parole be refused. Now what do you do?

The situation you are in is difficult because even though the intelligent
agent was presented as your partner, you have a legal and ethical
responsibility to get this decision right. Therefore, you are acting as the
agent’s supervisor; faced with a decision to either override or accept your
subordinate’s (the agent’s) recommendation.

This decision is called a “ _trust calibration_ ” decision. We define trust
calibration as a process by which a human supervisor determines whether an
individual agent’s prediction or recommended course of action should be
trusted on a case-by-case basis. The original work on trust calibration was
performed by Muir (1987) who stated: the human supervisor “ _must learn to
calibrate his trust, that is to set his trust to a level corresponding to [an
agent’s] trustworthiness, and then use the [agent] accordingly._ ”

Unfortunately, not all human supervisors ‘use the agent accordingly’. Some
humans just side with the intelligent agent (i.e., “automation bias” or
“missed intervention”) (Lyell and Coiera, 2017; Lee, 2018). Others immediately
override the recommendation and go with their own decision (i.e., “algorithm
aversion” or “bad intervention”) (Dietvorst et al., 2015; Awad et al., 2018).
In a benign situation (e.g., movie recommendation), being automation biased or
algorithm adverse may have little consequence. This is not the case in high-
stakes situations (e.g., military, security, health-care, criminal justice),
where inability to calibrate trust correctly can result in severe effects to
human life. For example, for algorithm aversion, if judges had agreed with an
intelligent agent in ’failure to appear’ decisions, pre-trial detention rates
would have been reduced by 41.9% with no increase in crime rate (Lowens,
2020). In another example, for algorithm bias, if operators had not trusted
the Patriot missile targeting system course of action, a Royal Air Force jet
would not have been shot down by friendly fire (Bode and Watts, 2021).

The ideal partnership lies between these two extremes by achieving the goal of
complementary performance; where the human supervisor agrees with the agent
when the agent’s performance is likely to exceed the supervisor, and the
supervisor rejects the agent when the agent is likely to under-perform.

To determine when the agent is likely to exceed or under-perform the human
supervisor, you will need more information than simply the main output of the
agent. We call this additional information “ _trust calibration meta-
information_.” We define trust calibration meta-information as information
flowing from the agent subordinate to the human supervisor that provides
context to the main output of the agent; thereby facilitating a better
understanding of whether the main output should be accepted.

The concept of trust calibration appears in many different research fields:
automation, decision-making, recommender systems, human-robot interaction, and
explainable AI (XAI). In some fields, such as automation (Lee and See, 2004),
decision-making (Wang and Yin, 2021) and recommender systems (Kunkel et al.,
2019), trust calibration has been apart of the literature for some time. In
human-robot interaction, trust includes multiple factors related to
performance, appearance, and interaction behavior (Khavas, 2021), but trust
calibration decisions still exist to determine whether the human supervisor
should intervene in a robot’s course of action (Wang et al., 2016). In XAI,
much focus is given to understandability, comprehensibility, interpretability,
and transparency (Arrieta et al., 2020) but a “central task for XAI is
calibrating trust” (Sperrle et al., 2020). To ensure coverage across this wide
set of research, we structured our survey as a trace through these various
fields.

###  1.1 Test Set Accuracy vs. Meta-Information

When confronted with the decision to accept or intervene in an agent main
output, one might consider the test set accuracy of the intelligent agent as a
useful guide. Test set accuracy (i.e., the performance of the agent on a set
of data not used during agent training) can be considered trust calibration
meta-information, but in most cases is not sufficient for this decision. For
example, knowing an agent achieved 99% accuracy during testing, does not help
determine if the current case is the statistical 1 out of 100 where the agent
is incorrect. And moreover, test set accuracy does not inform the user if the
current case is even in the same distribution as the cases used during the
agent’s training or testing. Therefore, we need to consider what additional
types of meta-information can be leveraged.

###  1.2 Why is a Taxonomy Needed?

As this survey will show, multiple works have considered trust calibration and
have identified, indirectly, the meta-information associated with their
proposed methods. Unfortunately, it is difficult to cross-compare these papers
since terminology used is overlapping in some senses and conflicting in
others. For example, Zhang et al. (2020) states “unlike confidence,
explanation did not seem to affect participants’ trust.” In another paper,
McGuirl and Sarter (2006) state “confidence information can improve trust
calibration.” Upon closer examination, even though these authors appear to
agree, these authors have used the same term, “confidence” with slightly
different meanings. Zhang uses “confidence” to refer to historical performance
values, while McGuirl uses “confidence” to refer to an uncertainty of how
close the current case matches the training data set.

Before making judgements on what types or modalities of trust calibration
meta-information best drives human-agent partnerships toward the goal of
complementary performance, we believe the community needs a taxonomy to
specifically characterize the meta-information leveraged in the different
literature on trust calibration.

Therefore, the goal of this paper is to survey trust calibration from a meta-
information point of view and create a taxonomy of trust calibration meta-
information; thereby providing a more rigorous basis for describing and
contrasting different approaches to trust calibration. This taxonomy should
improve the discussion in the field of trust calibration, lead to more nuanced
and effective comparison studies across different trust calibration
approaches, and potentially open new areas of trust calibration research.

In summary, the novel contributions of our work include:

  * •

A taxonomy for describing trust calibration meta-information, in terms of
content, modality, interactivity, and timing.

  * •

An overview of trust calibration literature from a human-computer interaction
point of view

  * •

Identification of future research opportunities in the area of trust
calibration meta-information.

The remainder of the paper is structured as follows: Section 2 reviews the
work done in performing the survey. Section 3 details the taxonomy and
demonstrates how surveyed literature is characterized within the taxonomy. In
Section 4 we provide a discussion of initial findings generated by leveraging
our taxonomy. Finally, we present limitations and conclusions in Sections 5
and 6 respectively.

##  2 Survey Methodology

Our survey on trust calibration leveraged a bootstrapping method. Starting
with a set of initial seed papers retrieved using keywords ’trust,’ ’trust
calibration,’ and ’calibrate trust’ on Google Scholar, we surveyed papers
across the automation, XAI, decision-aids, and robotics fields and then
expanded our survey by adding relevant papers through paper references. To
maintain scope within our survey, we focused on literature that examined trust
on an agent’s instance-level decisions. We also limited our survey to papers
where trust calibration is performed within an asymmetric partnership between
a single human and a single agent where the human is the supervisor, and the
intelligent agent is the subordinate (or assistant). Finally, instead of
focusing on methods of trust calibration, we focused on literature that
directly or indirectly discussed the meta-information supporting the trust
calibration. This resulted in a more human-computer interaction (HCI) focused
perspective that we believe will facilitate our ability to group research
similarities across diverse fields and diverse agent implementations.

##  3 Meta-Information Taxonomy

After performing the survey, we attempted to classify the meta-information
into major categories. In this paper, we characterize trust calibration meta-
information along four dimensions: content, modality, interactivity, and
timing. Content refers to the different types of information content that can
be present in the meta-information. Modality refers to the different ways in
which the meta-information content can be presented. Interactivity refers to
whether the human supervisor can interact with the meta-information. Timing
refers to the different times in which the flow of meta-information can be
initiated.

Within each major category, we used results from the survey to build up sub-
categories. The names of the sub-categories were generated by selecting terms
that were common in multiple papers or that we felt best represented the
entire type.

The combination of the major categories (content, modality, interactivity,
timing) with the various sub-categories forms our proposed taxonomy for trust
calibration meta-information (Table 1).

Using this taxonomy, any trust calibration approach can be characterized by
selecting a sub-category from each of the four major categories. For example,
if an agent provides a numeric confidence as trust calibration meta-
information with its main output, the associated meta-information would be
classified in the above taxonomy as: Confidence/Risk Content, Values Modality,
No Interactivity, and Concurrent Timing.

The following four subsections describe each of the major categories:
_content_ , _modality_ , _interactivity_ , and _timing_.

Major Category | Sub-Category
---|---
Content | Case-Based
| Sensitivity
| Influence
| Confidence/Risk
| Historic Performance
| Uncertainty
| Logic Rationale
Modality | Narriative
| Values
| Graphical
Interactivity | Yes/No
Timing | Pro-active
| Concurrent
| Post-hoc
Table 1: Trust calibration meta-information taxonomy structure of major
categories and sub-categories

###  3.1 Meta-Information Content

Although there is a huge diversity of papers in trust calibration, when we
examine the content of meta-information, we observed 7 meta-information
content sub-categories, as shown in Table 2, that are common across previous
works. We summarize the multiple sub-categories of meta-information content
below.

####  3.1.1 Case-based Content

_Case-based content_ uses examples (positive or negative) from the agent
training, validation, or operations that is pertinent to the current decision
or situation in order to provide the human supervisor with context of how
previous decisions were made or situations were handled. By understanding the
relationship between the current situation and the past examples, the human
supervisor has more information by which to judge the agent’s ability.

Case-based content typically is presented by showing the prior case that is
most similar to the current situation (Lai and Tan, 2019). If more than one
case contains the same input as the current situation, the percentage of exact
matches for each output class can be shown (Dodge et al., 2019). If no exact
matches are found, a nearest neighbor process (e.g., a KNN approach) can be
run to determine the closest match. More than one closest match can also be
provided. For example, Lai and Tan (2019); Cai et al. (2019); Wang and Yin
(2021) provide multiple cases: the closest match from the training data that
has the same output as the current situation and the closest match that has
the outputs that differ from the current situation. Case-based content can
also take the form of a comparison between the current situation and a more
accepted case (Gregor and Benbasat, 1999; Bussone et al., 2015; Ehrlich et
al., 2011). For example, Bussone et al. (2015) suggested that comparison of
current symptoms to accepted symptoms from a standard medical diagnosis can
provide the context for agent’s diagnosis. The comparison to the accepted case
can be side-by-side or through the calculation of a similarity metric (Ehrlich
et al., 2011).

####  3.1.2 Sensitivity Content

_Sensitivity content_ presents the human supervisor with how much one or more
inputs of the task would have to differ to change the output. Sensitivity
content attempts to refine the human supervisor’s mental model of the agent by
providing insight into the agent’s decision boundaries. Decision boundaries
represent a more compact representation of the details of the agent’s
decision-making process and enables the human to compare the agent’s decision
boundaries to their own.

Sensitivity content is typically presented as what is the input change
necessary to change the output (Dodge et al., 2019; Wang and Yin, 2021) (e.g.,
“If the defendant’s age had been 29 instead of 26, she would have predicted as
NOT likely to re-offend”) (Wang and Yin, 2021).

####  3.1.3 Influence Content

_Influence content_ presents the human supervisor with how much inputs of the
task influence the output. Influence content attempts to assist the human
supervisor by provide insight into the agent’s decision-making process. The
weights each input/feature are a pseudo-representation of the decision-making
process that enable the human supervisor to make contrasting examinations of
the weights by comparing them to the supervisor’s own mental model of what is
most important about the task.

Influence content can be presented as: the contribution or importance of an
attribute in determining the outcome (Lai and Tan, 2019; Dodge et al., 2019;
Cheng et al., 2019; Gregor and Benbasat, 1999; Wang and Yin, 2021) (e.g.,
showing the level of importance of each word in a product review to determine
if review is genuine (Lai and Tan, 2019)), or as an indication of which inputs
are impacting the outcome or course of action (Wintersberger et al., 2021;
Ehsan et al., 2021a; Wang et al., 2016; Bussone et al., 2015; Svrcek et al.,
2019) (e.g., “these objects in the environment are being considered in an
automated vehicles driving decisions” (Wintersberger et al., 2021).)

####  3.1.4 Confidence/Risk Content

_Confidence/Risk content_ is the estimate, by the agent, of the likelihood of
an event occurring (task success, correct answer, loss of life, etc).
Confidence/Risk content is a higher-level content than others since this
content offers a summarization of the outcome of the task. While the easiest
for human supervisors to relate to and make decisions based upon, it may not
be the easiest to understand unless the meaning of the confidence number
and/or how it was derived are clearly communicated (Bussone et al., 2015).

Confidence is presented typically as a risk or confidence about a specific
instance of an output (Ashoori and Weisz, 2019; Wang et al., 2016; Bussone et
al., 2015)(e.g., ”I am 67% confident about this assessment” (Wang et al.,
2016)).

####  3.1.5 Historic Performance Content

_Historic Performance content_ is success statistics on prior training,
validation or operations that is pertinent to the current situation. Similar
to Case-based content in terms of using prior examples, Historic Performance
differs from Case-based in that Historic Performance is an aggregate measure
of the past rather than providing key cases. In addition, Historic Performance
content differs from Case-based in terms of purpose. Where the purpose of
Case-based is to provide details on agent decision making under similar
circumstances, Historic Performance focuses on whether the agent should be
used in this situation or not based on aggregated statistics.

Historic Performance is typically presented as performance during validation
or past operational experience (e.g., “Agent has an accuracy of approximately
87%”) (Lai and Tan, 2019; Wang et al., 2016), but also can be presented as
performance against similar inputs in the past (e.g., “The agent is correct N
times out of 10 in cases similar to this one”) (Zhang et al., 2020; Dodge et
al., 2019), performance against other peers in operation (e.g., 1 out of 3
peers sold at the recommended price) (Ehsan et al., 2021a), or performance for
attribute-based groups (e.g., “58% of people in 18-29 age group re-offended”)
(Dodge et al., 2019).

####  3.1.6 Uncertainty Content

_Uncertainty content_ is an estimate, by the agent, of the lack of knowledge
about the outcome. Uncertainty attempts to assist the human supervisor by
providing insight how well the agent understands the problems represented by
the task. For example, if an agent’s uncertainty is perceived by the human
supervisor as too large, the supervisor may be less likely to trust the
outcome of the agent (Bhatt et al., 2021). Uncertainty can be broken down into
aleatoric and epistemic uncertainty. Aleatoric uncertainty (doubt) represents
uncertainty inherent in the system being modeled because of stochastic
behavior. Epistemic uncertainty (ambiguity) is the uncertainty due to limited
data or knowledge (Tomsett et al., 2020). Uncertainty can be presented as the
error bars on an outcome, probability distribution summary, or spaghetti plots
(Bhatt et al., 2021).

Another important area of uncertainty is the concept of out-of-distribution.
Out-of-distribution refers to difference in inputs that occur when an agent is
trained and validated with a closed-world assumption (all data from the same
distribution) and when the agent is deployed into an open-world scenario where
inputs can now be out-of-distribution from the training and validation data
sets (Yang et al., 2021). A form of epistemic uncertainty, out-of-distribution
detection attempts to determine how far new inputs are from the inputs that
the agent was trained on (Tomsett et al., 2020). By providing an estimate if
the current data is out-of-distribution, an agent provides a powerful estimate
of uncertainty. This type of uncertainty content can be presented as a
percentage of how close the current input matched the training set (McGuirl
and Sarter, 2006) and is a great counterbalance to Confidence/Risk content as
it provides a better understanding of how much weight to put on agent-
generated confidence (e.g., a high confidence should be viewed as useless if
the agent also provides high probability that the assessment was performed on
inputs out-of-distribution with training inputs).

####  3.1.7 Logic Rationale Content

_Logic Rationale_ content provides insight into the inner workings of agent
logic that led to the outcome. The benefit of Logic Rationale content is that
it assists the human supervisor to understand the “why” behind agent actions.

Logic Rationale is typically presented in the form of limited logic statements
(Ehsan et al., 2021b; Zhu and Williams, 2020; Svrcek et al., 2019) (e.g., “I
am going to collect red resources because you are low on red resources” (Zhu
and Williams, 2020), or “This was recommended because it contains features of
movies you positively rated in the past” (Svrcek et al., 2019)). Logic
Rationale can also be presented in the form of actual internal logic, if the
agents method allows (e.g., decision tree, rules) (Huysmans et al., 2011) or
if the understandable logic can be extracted from a black box agent (Guidotti
et al., 2019).

Sub-Category |  Description |  Explanation Examples |  Relevant Papers
---|---|---|---
Case-Based |  Examples (positive or negative) from the agent training, testing or operations that is pertinent to the current situation |  “The closest match to this situation in my training set is **this** and the outcome of that training example was a success.” |  (Wang and Yin, 2021; Dodge et al., 2019; Cai et al., 2019; Lai and Tan, 2019; Bussone et al., 2015; Ehrlich et al., 2011; Gregor and Benbasat, 1999)
Sensitivity |  Show how much one or more inputs of the task would have to differ to change the outcome |  “The weight of the object would have to grow by 250 lbs. before I would not be able to carry it.” |  (Wang and Yin, 2021; Dodge et al., 2019)
Influence |  Show how much inputs of the task influence the outcome |  “The inputs most driving this decision was prior convictions and age.” |  (Wintersberger et al., 2021; Zhang et al., 2020; Wang and Yin, 2021; Dodge et al., 2019; Lai and Tan, 2019; Cheng et al., 2019; Svrcek et al., 2019; Wang et al., 2016; Bussone et al., 2015; Gregor and Benbasat, 1999)
Confidence/ Risk |  Estimate by the agent of the likelihood of a correct output or an event occurring |  “I estimate a 57% risk of damage to my equipment in the process of performing this task.” |  (Zhu and Williams, 2020; Ashoori and Weisz, 2019; Wang et al., 2016; Bussone et al., 2015)
Historic
Performance |  Success statistics on prior situations that is pertinent to the current situation |  “I have performed this mission in similar conditions before, and did so successfully 80% of the time.” |  (Zhang et al., 2020; Dodge et al., 2019; Ehsan et al., 2021a; Lai and Tan, 2019; Wang et al., 2016)
Uncertainty |  Estimate by the agent of the lack of knowledge about the outcome |  “The object in this image has a 20% chance of representing an object I have not been trained on.” |  (Tomsett et al., 2020; Bhatt et al., 2021; McGuirl and Sarter, 2006)
Logic
Rationale |  Provide insights into the inner workings of agent logic that led to the outcome |  “I am going to collect red resources because you are low on red resources.” |  (Ehsan et al., 2021b; Zhu and Williams, 2020; Svrcek et al., 2019)
Table 2: Summary of meta-information content sub-categories with description
and explanation examples

###  3.2 Meta-Information Modality

Previous work in trust calibration combined aspects of meta-information flow
content and modality. We offer a different analysis where we separate content
and modality. The sections below summarize the multiple sub-categories of
meta-information modalities.

####  3.2.1 Narrative Modality

The _Narrative modality_ is a text-based presentation of meta-information flow
content. This modality is typically represented as one or more sentences of
text (Dodge et al., 2019; Lai and Tan, 2019; Zhang et al., 2020; Svrcek et
al., 2019). The narrative modality allows the human supervisor to read about
behavior or outcomes in a manner similar to how a human supervisor would
interact a human subordinate.

####  3.2.2 Values Modality

The _Values modality_ is a numeric presentation of meta-information flow
content. This modality can be presented as a percentage (e.g., “70%”) (Lai and
Tan, 2019; Wang et al., 2016), or as a frequency (e.g., “2 out of 3”) (Zhang
et al., 2020). The values modality provides a straightforward mechanism to
communicate meta-information amounts. As Miller (2019) states, however, solely
using values may be “unsatisfying unless accompanied by an underlying causal
explanation.”

####  3.2.3 Graphical Modality

The _Graphical modality_ is a visual presentation of meta-information flow
content. This modality is typically presented as highlights on top of existing
content (e.g., highlights of important objects in the field of view),
(Wintersberger et al., 2021), plots (Bhatt et al., 2021; McGuirl and Sarter,
2006) (e.g., cone-of-uncertainty of a predicted path) (Bhatt et al., 2021), or
charts (Schaffer et al., 2019; Wang and Yin, 2021) (e.g., bar chart of long-
term reward of various outcomes (Schaffer et al., 2019)). The graphical
modality provides a quickly digestible form of meta-information.

###  3.3 Meta-Information Interactivity

When describing meta-information flows, we need to consider whether
interactivity exists or not within the meta-information flow.

Interactivity enables a back-and-forth interaction with the human supervisor
to reveal aspects and trends in meta-information flow content. The interactive
modality allows the human supervisor to explore the agent’s behavior through
adjusting inputs or various internal parameters of the agent. For example,
Cheng et al. (2019) proposed an interactive interface where the human can
modify inputs to determine which inputs are driving the agent’s decision
making. In another example, Schaffer et al. (2019) enabled users to alter
estimates of the external environment (i.e., other player tendencies), to
understand the relationship between outcome and input.

###  3.4 Meta-Information Timing

In order to completely describe meta-information flows, we need to also
consider the timing of the flow (i.e. when is the flow initiated). The
sections below summarize the multiple sub-categories of meta-information flow
timings.

####  3.4.1 Proactive Timing

In _Proactive timing_ , the meta-information flow is initiated prior to the
agent preforming a task. The benefit of Proactive timing is that by informing
the human supervisor what is to occur, the supervisor is better prepared to
understand agent behavior instead of reacting to behavior that may not be
understood in the supervisor’s mental model without the proactive information.
An example of this is described by Zhu and Williams (2020), where they
identify proactive explanations as a method of communicating logic rationale
for an agent’s actions prior to the agent performing these actions.

####  3.4.2 Concurrent Timing

In _Concurrent timing_ , the meta-information flow occurs at the same time as
the agent provides the recommendation or course of action. The benefit of
Concurrent timing is that the supervisor is informed at the point of having to
react to the agent and therefore best leverages the supervisor’s focus. The
majority of the literature provides meta-information concurrently as a one-
time flow but can also be dynamically updating as shown by McGuirl and Sarter
(2006).

####  3.4.3 Post-Hoc Timing

In _Post-Hoc timing_ , the meta-information flow is provided after an agent
provides a result or performs a course of action. Post-hoc meta-information is
provided in response to a request from the human supervisor to explain a
previous behavior (e.g., “Why did/didn’t you do that?,” “Why can’t you do
that?”) (de Graaf and Malle, 2017; Fox et al., 2017) or in the case of a
specific trigger, such as anomalous agent behavior (Gregor and Benbasat,
1999).

##  4 Discussion

By performing this survey and developing this taxonomy, we have gained insight
into the breadth and direction of research in trust calibration from a human-
centered point of view. This survey also provided us a unique view into the
difficulty of trust calibration and into future directions for our taxonomy.

###  4.1 Difficulty of Trust Calibration

The reason why trust calibration is difficult is that intelligent agents
suffer from a form of the Dunning–Kruger Effect (Dunning, 2011): Intelligent
agents are too ignorant to know how ignorant they are. More precisely:
performing trust calibration forces the intelligent agent to provide cues to a
human supervisor indicating the agent’s own output is wrong even when the
agent does not know that it is wrong.

####  4.1.1 Meta-Information Recommendations

So what makes an intelligent agent wrong during operations? Over-fitting, not
enough training, training not being an accurate representation of operations,
poor assumptions or abstractions of the operational environment, the
complexity of the operations environment, and unforeseen operations situations
are some examples of issues driving an intelligent agent to under-perform in
operations. Since we believe these issues drive an agent away from the as-
developed performance (e.g., test set accuracy), we recommend providing meta-
information content that helps identify a mismatch between the training set
and the current instance, or helps identify incorrect modeling assumptions.
These include Uncertainty Content (especially for determining out-of-
distribution) and Sensitivity Content in the context of Graphical Modalities
(especially for highlighting salient features which drive decisions) in
Proactive and Concurrent Timings for operations.

####  4.1.2 Trust-Based Trust Calibration

Not all operations are the same. In low-stakes, ’everyday’ operations (e.g.,
purchase recommenders), according to Bunt et al. (2012), some human
supervisors may be wary of any meta-information that requires time and
cognitive effort unless it would enhance the speed or accuracy of the system.
In high-stakes operations (e.g., military, security, health care, criminal
justice), more meta-information may be required to ensure compliance with
legal and ethical frameworks. Even though these types of operations are vastly
different, we believe both should adapt to human supervisors’ level of trust.
For example, Wintersberger et al. (2021) suggest that trust calibration
information should be provided to distrusting supervisors, while not burdening
supervisors with trust values that are already high. Therefore we suggest that
supervisor trust level be an input to intelligent agents so that the agent can
vary the content, modality, timing, interactivity, and amount of trust
calibration meta-information accordingly. This construct allows for
flexibility in low-cost and high-cost operations as well as providing for
cross over cases when a low-cost user wants to understand more or when a high-
cost user wants communications brevity (e.g., time-critical military
operations).

###  4.2 Future Directions

Now that meta-information content, modalities, interactivity, and timing are
formalized in the taxonomy presented in this paper, future studies can
determine, for example: which information flow content achieves the best
human-agent partnership performance; whether information flow content or
modality is what drives this performance; and which information flow content,
modalities, interactivity, timings, and amounts work best with which human
supervisor characteristics or groups.

One interesting follow on to formalizing meta-information is to standardize
this information. Standardization would involve formally defining each
content, modality, interactivity, and timing sub-category including the
information formats within each. Even though this may be a daunting task, the
benefits could be substantial. For example, having standardized trust
calibration meta-information could improve real-world adoption of intelligent
agents, reduce the cost of infrastructure necessary to test and field agents,
reduce supervisor training, and increase understanding during operations.

Another follow on area deals with meta-information timing. The majority of
literature we surveyed on trust calibration and meta-information focused on
information flow before, during, or after performing a task, as we described
in Section 3.4. Achieving complementary performance, however, may require
additional timing phases outside of the operational task. For example,
military supervisor-subordinate partnerships develop a ‘shared confidence’ in
team members’ abilities over time during training exercises where each side of
the partnership gains understanding of how the other will react and respond in
different situations (USArmy, 2019). Initial studies have shown that human-
agent partnerships also function better during operations due to training
exercises (Johnson et al., 2021) and pre-operations team building(Walliser et
al., 2019). Therefore, an important area of research would be to determine:
what meta-information is needed to support these pre-operations phases and how
does the training exercise and rehearsal meta-information differ from what is
outlined in this paper.

##  5 Limitations

We chose to limit our survey to trust calibration between a single human and a
single agent. Due to this limitation, trust calibration between humans and
multi-agent systems (Setter et al., 2017) and trust calibration between
multiple agents (Liu et al., 2016) are not discussed.

We also limited our survey to trust calibration meta information in support of
individual decisions. Any additional meta-information necessary to support
longitudinal trust calibration (de Visser et al., 2020), such as reliability
(Lyons et al., 2021), was not covered.

We structured our survey based on information flowing from agent to human but
acknowledge that there are other very important information flows present in
trust calibration that were not covered since they do not originate from the
agent. This includes information flows about task risk (Devitt, 2018), task
complexity (Hancock et al., 2011), and task consequences (Ashoori and Weisz,
2019) which change the threshold for trust.

Finally, since this paper focuses on information, other factors such as
agents’ physical presence and supervisors’ subjective feelings toward
intelligent agents (Khavas, 2021; Hancock et al., 2011) are not discussed.

##  6 Conclusions

We have presented a taxonomy based on a review of trust calibration literature
for single human to single agent interactions. Through this survey we have
distilled the content, modality, interactivity, and timing of meta-information
necessary to inform a trust calibration decision for intelligent agents. By
focusing on information flows, we have been able to consider several
directions to guide future research.

## References

  * Arrieta et al. [2020] Alejandro Barredo Arrieta, Natalia D’iaz-Rodr’iguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, A. Barbado, Salvador Garc’ia, Sergio Gil-L’opez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  ArXiv, abs/1910.10045, 2020.
  * Ashoori and Weisz [2019] Maryam Ashoori and Justin D. Weisz.  In ai we trust? factors that influence trustworthiness of ai-infused decision-making processes.  ArXiv, abs/1912.02675, 2019.
  * Awad et al. [2018] Edmond Awad, Sydney Levine, Max Kleiman-Weiner, Sohan Dsouza, Joshua B. Tenenbaum, Azim F. Shariff, Jean‐François Bonnefon, and Iyad Rahwan.  Blaming humans in autonomous vehicle accidents: Shared responsibility across levels of automation.  ArXiv, abs/1803.07170, 2018.
  * Bhatt et al. [2021] Umang Bhatt, Yunfeng Zhang, Javier Antorán, Qingzi Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Gauthier Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, Lama Nachman, Rumi Chunara, Adrian Weller, and Alice Xiang.  Uncertainty as a form of transparency: Measuring, communicating, and using uncertainty.  Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 2021.
  * Bode and Watts [2021] Ingvild Bode and Tom Watts.  Worried about the autonomous weapons of the future? look at what’s already gone wrong.  https://thebulletin.org/2021/04/worried-about-the-autonomous-weapons-of-the-future-
look-at-whats-already-gone-wrong/, 2021\.  Last Accessed: 2022-02-14.

  * Bunt et al. [2012] Andrea Bunt, Matthew Lount, and Catherine Lauzon.  Are explanations always important?: a study of deployed, low-cost intelligent interactive systems.  In IUI ’12, 2012.
  * Bussone et al. [2015] Adrian Bussone, Simone Stumpf, and Dympna M. O’Sullivan.  The role of explanations on trust and reliance in clinical decision support systems.  2015 International Conference on Healthcare Informatics, pages 160–169, 2015.
  * Cai et al. [2019] Carrie J. Cai, Jonas Jongejan, and Jess Holbrook.  The effects of example-based explanations in a machine learning interface.  Proceedings of the 24th International Conference on Intelligent User Interfaces, 2019.
  * Cheng et al. [2019] Hao Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O’Connell, Terrance Gray, F. Maxwell Harper, and Haiyi Zhu.  Explaining decision-making algorithms through ui: Strategies to help non-expert stakeholders.  Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019.
  * de Graaf and Malle [2017] Maartje de Graaf and Bertram F. Malle.  How people explain action (and autonomous intelligent systems should too).  In AAAI Fall Symposia, 2017.
  * de Visser et al. [2020] Ewart de Visser, Marieke M. M. Peeters, Malte F. Jung, Spencer Kohn, Tyler H. Shaw, Richard Pak, and Mark A. Neerincx.  Towards a theory of longitudinal trust calibration in human–robot teams.  International Journal of Social Robotics, 12:459–478, 2020.
  * Devitt [2018] Susannah Kate Devitt.  Trustworthiness of autonomous systems.  In CDC 2018, 2018.
  * Dietvorst et al. [2015] Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey.  Algorithm aversion: people erroneously avoid algorithms after seeing them err.  Journal of experimental psychology. General, 144 1:114–26, 2015\.
  * Dodge et al. [2019] Jonathan Dodge, Qingzi Vera Liao, Yunfeng Zhang, Rachel K. E. Bellamy, and Casey Dugan.  Explaining models: an empirical study of how explanations impact fairness judgment.  Proceedings of the 24th International Conference on Intelligent User Interfaces, 2019.
  * Dunning [2011] David Dunning.  The dunning–kruger effect: On being ignorant of one’s own ignorance.  In Advances in Experimental Social Psychology, Volume 44, Academic Press, 2011.
  * Ehrlich et al. [2011] Kate Ehrlich, Susanna E. Kirk, John F. Patterson, Jamie C. Rasmussen, Steven I. Ross, and Dan Gruen.  Taking advice from intelligent systems: the double-edged sword of explanations.  In IUI ’11, 2011.
  * Ehsan et al. [2021a] Upol Ehsan, Qingzi Vera Liao, Michael J. Muller, Mark O. Riedl, and Justin D. Weisz.  Expanding explainability: Towards social transparency in ai systems.  Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021.
  * Ehsan et al. [2021b] Upol Ehsan, Samir Passi, Qingzi Vera Liao, Larry Chan, I-Hsiang Lee, Michael J. Muller, and Mark O. Riedl.  The who in explainable ai: How ai background shapes perceptions of ai explanations.  ArXiv, abs/2107.13509, 2021.
  * Fox et al. [2017] Maria Fox, Derek Long, and Daniele Magazzeni.  Explainable planning.  ArXiv, abs/1709.10256, 2017.
  * Gregor and Benbasat [1999] Shirley D Gregor and Izak Benbasat.  Explanations from intelligent systems: Theoretical foundations and implications for practice.  MIS Q., 23:497–530, 1999.
  * Guidotti et al. [2019] Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and Fosca Giannotti.  A survey of methods for explaining black box models.  ACM Computing Surveys (CSUR), 51:1 – 42, 2019.
  * Hancock et al. [2011] Peter A. Hancock, Deborah R. Billings, Kristin E. Schaefer, Jessie Y.C. Chen, Ewart de Visser, and Raja Parasuraman.  A meta-analysis of factors affecting trust in human-robot interaction.  Human Factors: The Journal of Human Factors and Ergonomics Society, 53:517 – 527, 2011.
  * Huysmans et al. [2011] Johan Huysmans, Karel Dejaeger, Christophe Mues, Jan Vanthienen, and Bart Baesens.  An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models.  Decis. Support Syst., 51:141–154, 2011.
  * Johnson et al. [2021] Craig J. Johnson, Mustafa Demir, Nathan J. Mcneese, Jamie C. Gorman, Alexandra T. Wolff, and Nancy J. Cooke.  The impact of training on human-autonomy team communications and trust calibration.  Human factors, page 187208211047323, 2021.
  * Khavas [2021] Zahra Rezaei Khavas.  A review on trust in human-robot interaction.  ArXiv, abs/2105.10045, 2021.
  * Kunkel et al. [2019] Johannes Kunkel, Tim Donkers, Lisa Michael, Catalin-Mihai Barbu, and Jürgen Ziegler.  Let me explain: Impact of personal and impersonal explanations on trust in recommender systems.  Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019.
  * Lai and Tan [2019] Vivian Lai and Chenhao Tan.  On human predictions with explanations and predictions of machine learning models: A case study on deception detection.  Proceedings of the Conference on Fairness, Accountability, and Transparency, 2019.
  * Lee and See [2004] John D. Lee and Katrina A. See.  Trust in automation: Designing for appropriate reliance.  Human Factors: The Journal of Human Factors and Ergonomics Society, 46:50 – 80, 2004.
  * Lee [2018] Timothy B. Lee.  Police: Uber driver was streaming hulu just before fatal self-driving car crash.  https://arstechnica.com/cars/2018/06/police-uber-driver-was-streaming-hulu-just-before-fatal
self-driving-car-crash/, 2018\.  Last Accessed: 2022-02-07.

  * Liu et al. [2016] Ling Liu, Margaret L. Loper, Yusuf Özkaya, Abdurrahman Yaşar, and Emre Yigitoglu.  Machine to machine trust in the iot era.  In TRUST@AAMAS, 2016.
  * Lowens [2020] Ethan Lowens.  Accuracy is not enough: The task mismatch explanation of algorithm aversion and its policy implications.  Harvard Journal of Law & Technology, Vol. 34, No. 1, 2020.
  * Lyell and Coiera [2017] David Lyell and Enrico W. Coiera.  Automation bias and verification complexity: a systematic review.  Journal of the American Medical Informatics Association, 24:423–431, 2017.
  * Lyons et al. [2021] Joseph B. Lyons, Thy Vo, Kevin T. Wynne, Sean Mahoney, Chang Soo Nam, and Darci Gallimore.  Trusting autonomous security robots: The role of reliability and stated social intent.  Human Factors: The Journal of Human Factors and Ergonomics Society, 63:603 – 618, 2021.
  * McGuirl and Sarter [2006] John M. McGuirl and Nadine B. Sarter.  Supporting trust calibration and the effective use of decision aids by presenting dynamic system confidence information.  Human Factors: The Journal of Human Factors and Ergonomics Society, 48:656 – 665, 2006.
  * Miller [2019] Tim Miller.  Explanation in artificial intelligence: Insights from the social sciences.  Artif. Intell., 267:1–38, 2019.
  * Muir [1987] Bonnie M. Muir.  Trust between humans and machines, and the design of decision aids.  Int. J. Man Mach. Stud., 27:527–539, 1987.
  * Schaffer et al. [2019] James Austin Schaffer, John O’Donovan, James R. Michaelis, Adrienne Jeanisha Raglin, and Tobias Höllerer.  I can do better than your ai: expertise and explanations.  Proceedings of the 24th International Conference on Intelligent User Interfaces, 2019.
  * Setter et al. [2017] Tina Setter, Andrea Gasparri, and Magnus Egerstedt.  Trust in multi-agent networks: From self-centered to team-oriented.  2017 American Control Conference (ACC), pages 997–1002, 2017.
  * Sperrle et al. [2020] Fabian Sperrle, Mennatallah El-Assady, Grace Guo, Duen Horng Chau, Alex Endert, and Daniel A. Keim.  Should we trust (x)ai? design dimensions for structured experimental evaluations.  ArXiv, abs/2009.06433, 2020.
  * Svrcek et al. [2019] Martin Svrcek, Michal Kompan, and Mária Bieliková.  Towards understandable personalized recommendations: Hybrid explanations.  Comput. Sci. Inf. Syst., 16:179–203, 2019.
  * Tomsett et al. [2020] Richard J. Tomsett, Alun David Preece, Dave Braines, Federico Cerutti, Supriyo Chakraborty, Mani B. Srivastava, Gavin Pearson, and Lance M. Kaplan.  Rapid trust calibration through interpretable and uncertainty-aware ai.  Patterns, 1, 2020.
  * USArmy [2019] USArmy.  Army doctrine publication (adp) 6-0, mission command.  https://armypubs.army.mil/ProductMaps/PubForm/Details.aspx?PUB_ID=1007502, 2019\.  Last Accessed: 2022-02-09.
  * Walliser et al. [2019] James Walliser, Ewart J. de Visser, Eva Wiese, and Tyler H. Shaw.  Team structure and team building improve human–machine teaming with autonomous agents.  Journal of Cognitive Engineering and Decision Making, 13:258 – 278, 2019.
  * Wang and Yin [2021] Xinru Wang and Ming Yin.  Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making.  26th International Conference on Intelligent User Interfaces, 2021\.
  * Wang et al. [2016] Ning Wang, David V. Pynadath, and Susan G. Hill.  Trust calibration within a human-robot team: Comparing automatically generated explanations.  2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 109–116, 2016.
  * Wintersberger et al. [2021] Philipp Wintersberger, Frederica Janotta, Jakob Peintner, Andreas Löcken, and Andreas Riener.  Evaluating feedback requirements for trust calibration in automated vehicles.  it - Information Technology, 63:111 – 122, 2021.
  * Yang et al. [2021] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu.  Generalized out-of-distribution detection: A survey.  ArXiv, abs/2110.11334, 2021.
  * Zhang et al. [2020] Yunfeng Zhang, Qingzi Vera Liao, and Rachel K. E. Bellamy.  Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making.  Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 2020.
  * Zhu and Williams [2020] Lixiao Zhu and Thomas Emrys Williams.  Effects of proactive explanations by robots on human-robot trust.  In ICSR, 2020.