# Transitions in echo index and dependence on input repetitions

Peter Ashwin<EMAIL_ADDRESS>Andrea Ceni<EMAIL_ADDRESS>Department of Mathematics, University of Exeter, Exeter EX4 4QF, UK
Department of Computer Science, University of Pisa, Largo Bruno Pontecorvo, 3
- 56127, IT

###### Abstract

The _echo index_ counts the number of simultaneously stable asymptotic
responses of a nonautonomous (i.e. input-driven) dynamical system. It
generalizes the well-known _echo state property_ for recurrent neural networks
- this corresponds to the echo index being equal to one. In this paper, we
investigate how the echo index depends on parameters that govern typical
responses to a finite-state ergodic external input that forces the dynamics.
We consider the echo index for a nonautonomous system that switches between a
finite set of maps, where we assume that each map possesses a finite set of
hyperbolic equilibrium attractors. We find the minimum and maximum repetitions
of each map are crucial for the resulting echo index. Casting our theoretical
findings in the RNN computing framework, we obtain that for small amplitude
forcing the echo index corresponds to the number of attractors for the input-
free system, while for large amplitude forcing, the echo index reduces to one.
The intermediate regime is the most interesting; in this region the echo index
depends not just on the amplitude of forcing but also on more subtle
properties of the input.

###### keywords:

Nonautonomous dynamical system , Input-driven system , Multistability ,
Recurrent neural network , Echo state property.

###### Contents

  1. 1 Introduction
    1. 1.1 Input-driven dynamics and shift dynamics
    2. 1.2 Local attractors and the echo index
  2. 2 Sufficient conditions to determine minimum echo index
    1. 2.1 Subshifts with min-max repetitions of input sequences
    2. 2.2 Minimum echo for multiple attractor maps
    3. 2.3 Obstructions to conditions for echo consistency
  3. 3 Echo index dependence for RNNs
    1. 3.1 Switching dynamics for RNNs
    2. 3.2 An example of input-driven RNN with multiple attractors
    3. 3.3 Bifurcations of echo index
  4. 4 Conclusions

##  1 Introduction

One of the most pressing questions for artificial intelligence systems, is
whether one can understand and query the reasons behind a decision made by
such a system. The difficulty of answering this explainability problem [18] is
reflected in the fact that a trained neural network is commonly referred to as
a black box. This suggests it is important to try and understand the
functioning of neural networks in decision making under input - it is
important to open the black box [19] and nonlinear dynamics gives tools that
can be used for this. As an example, [8] show that excitable network
attractors can be used to understand function and malfunction of trained RNNs
for certain tasks.

Recurrent neural networks (RNN) such as echo state networks [10, 15] can
retain memory of internal states. In this case, it is important to view the
system in an input-driven context [17]. A useful criterion for successful
computation is the Echo State Property (ESP) [10], which holds if there is
asymptotic loss of information about the internal state of the system and only
the input is important to determine the output. However, as discussed in [16],
the presence of the ESP depends not just on system but on the particular input
considered. As the input streams to an input-driven RNN will never be fully
deterministic, this means there is no guarantee that ESP will be satisfied in
practise. Recent work has highlighted that RNNs that do not have ESP may still
be a useful model for understanding errors in neural networks [9], or in order
to design multifunction RNNs that switch between different tasks [7, 4].

In this paper we develop ideas in [9] in more depth by considering
parametrizations of input by repetition properties of the inputs. The
remainder of this section discusses shift dynamics and introduces the echo
index for discrete time dynamical systems. We discuss parametrization of
inputs by symbol repetition. Section 2 presents some conditions to give bounds
on echo index and the ESP. Section 3 turns to a numerical example of an echo
state RNN where we show how the echo index changes depending on min-max block-
length and parameters of the input to the RNN. Section 4 concludes with a
discussion, including barriers to strengthening the theoretic results in
Section 3.

###  1.1 Input-driven dynamics and shift dynamics

We consider properties of a discrete time input-driven dynamical system [17]
of the form

$x[k+1]=f(x[k],u[k]),$ (1)

where time is indexed by $k\in\mathbb{Z}$, states are $x[k]\in
X\subset\mathbb{R}^{n}$ and $u[k]\in U\subset\mathbb{R}^{p}$ is an input
sequence. We denote sequences in bold font, i.e.

$\mathbf{x}=\\{x[k]\\}_{k\in\mathbb{Z}},~{}~{}\mathbf{u}=\\{u[k]\\}_{k\in\mathbb{Z}},$

and space of sequences in calligraphic font, i.e.
$\mathcal{X}=X^{\mathbb{Z}}$, $\mathcal{U}=U^{\mathbb{Z}}$, so that
$\mathbf{x}\in\mathcal{X}$, $\mathbf{u}\in\mathcal{U}$. We assume that $X$ and
$U$ are compact sets and assume that $f:X\times U\rightarrow X$ is an update
function that is continuous in both arguments, so that a forward orbit is
defined. Note that for a given input sequence $u[k]$, (1) can be thought of as
a nonautonomous dynamical system [12]

$x[k+1]=F(x[k],k),$ (2)

where $F:X\times\mathbb{R}\rightarrow X$ is defined by $F(x,k)=f(x,u[k])$. The
system (1) can also be viewed as a skew product dynamical system [12] over
shift dynamics on the input sequence $\mathbf{u}\in\mathcal{U}$.

We consider the special case where $U$ is taken from a finite set of $M$ input
values. Without loss of generality we denote $U=\\{0,\ldots,M-1\\}$ and call
elements of $U$ symbols. We recall some standard concepts from the symbolic
dynamics of shifts; see for example [1, 13, 6] for background and more
details. We define the shift operator $\sigma$ by

$[\sigma({\bf u})][k]=u[k+1].$

such that $\sigma:\mathcal{U}\rightarrow\mathcal{U}$. Consider a given subset
of input sequences $\mathcal{V}\subseteq\mathcal{U}$. We say $\mathcal{V}$ is
_shift-invariant_ if $\sigma(\mathcal{V})=\mathcal{V}$ and call
$(\sigma,\mathcal{U})$ the full shift on $U$. We consider the product topology
on $\mathcal{V}$ induced by the metric

$d(\mathbf{u},\mathbf{v})=\sum_{k\in\mathbb{Z}}\frac{d_{U}(u[k],v[k])}{2^{|k|}}$
(3)

where $d_{U}(u,v)$ is a metric on $U$. Note that $\sigma$ acts continuously
under such a topology.

If we assume that $\mathcal{V}$ is shift-invariant and closed then one can
lift the nonautonomous system (1) to a continuous _autonomous_ dynamical
system on an extended space $\mathcal{F}:X\times\mathcal{V}\rightarrow
X\times\mathcal{V}$. This is given by

$[x[k+1],\mathbf{u}[k]]=\mathcal{F}(x[k],{\bf
u}[k])=\left(f(x,u[0]),\sigma({\bf u}[k])\right)$ (4)

where we write $\mathbf{u}[k]=\sigma^{k}\mathbf{u}$. The skew product nature
of (4) means that composition can be written

$\mathcal{F}^{n}(x,{\bf u})=\left(\Phi_{n,{\bf
u}}(x),\sigma^{n}(\mathbf{u})\right)$ (5)

for all $n\in\mathbb{Z}_{0}^{+}$, where $\Phi$ is a cocycle, namely
$\Phi_{0,{\mathbf{u}}}$ is the identity map on $X$ and
$\Phi_{n+k,\mathbf{u}}=\Phi_{n,\sigma^{k}(\mathbf{u})}\circ\Phi_{k,\mathbf{u}}$
for any $n,k\in\mathbb{Z}^{+}_{0}$. We write the forward orbit of $x[0]$
driven by the input sequence $\mathbf{u}$ in terms of this cocycle, as

$x[k]=\Phi_{k,\mathbf{u}}(x[0]).$ (6)

for any $k\in\mathbb{Z}_{0}^{+}$. This can be extended to negative $k$ if $f$
is invertible.

There are many choices of $\mathcal{V}\subset\mathcal{U}$ that may be used to
characterise a set of possible input sequences, for convenience we only
consider closed and shift-invariant subsets. Given any
$\mathbf{u}\in\mathcal{U}$ one can define a closed invariant subset in terms
of its orbit closure

$\mathcal{U}(\mathbf{u})=\overline{\\{\sigma^{k}(\mathbf{u})~{}:~{}k\in\mathbb{Z}\\}}$

where $\overline{\mathcal{V}}$ denotes the closure of $\mathcal{V}$ in the
topology of (3).

An important set of closed and shift-invariant subsets are subshifts of finite
type defined as follows: For any $m\geq k$ we write
$u[k,m]=(u[k],u[k+1],\ldots,u[m])$ to denote a finite string or word of
symbols. Given a finite set $\mathcal{B}$ of words we define

$\mathcal{U}_{\mathcal{B}}=\\{\mathbf{u}\in\mathcal{U}~{}:~{}u[k,m]\cap\mathcal{B}=\emptyset\mbox{
for all }k,m\in\mathbb{Z}\mbox{ with }k\leq m\\},$

namely the set of sequences that contain no word in the set $\mathcal{B}$.
This is called a _subshift of finite type_ with _forbidden words_
$\mathcal{B}$. Without loss of generality we can assume that $\mathcal{B}$ is
_minimal_ in the sense there is no proper subset
$\mathcal{B}^{\prime}\subset\mathcal{B}$ such that
$\mathcal{U}_{\mathcal{B}}=\mathcal{U}_{\mathcal{B}^{\prime}}$.

If $\mathcal{U}_{\mathcal{B}}$ has a set of forbidden words of length at most
$k+1$ then we say it is a _$k$ -step subshift_: knowing $k$ consecutive
symbols provides the only constraints on the next symbol. Note that by
considering a higher block shift on overlapping words of length $m$ it is
possible to express such a $\mathcal{U}$ as a 1-step subshift of finite type
[6].

We write system (1) in the form of an iterated function system

$x[k+1]=f_{u[k]}(x[k]),$ (7)

for $k\in\mathbb{Z}$, where (7) describes the dynamics of switching between
one of $M$ of autonomous dynamical systems.

###  1.2 Local attractors and the echo index

We recall a nonautonomous notion of local attractor of (1) from [9] for fixed
input sequence; this was used to proposed a generalization of the notion of
ESP for input-driven systems with finitely many local attractors. For a given
input sequence $\mathbf{v}\in\mathcal{U}$, we say
$\mathbf{x}=\\{x[k]\\}_{k\in\mathbb{Z}}$ is an entire solution if it is a
trajectory for the input $\mathbf{v}$ in forwards and backwards time; i.e. it
satisfies (1) for all $k\in\mathbb{Z}$.

Note that we do not require $f_{i}$ to be invertible or surjective, and hence
given some $x[0]\in X$ there may be (a) multiple choices for $x[-1]$ or (b) no
choice for $x[-1]$ that lies within $X$. Note also that whether (a) or (b)
hold will typically depend on $v[k]$ for $k<0$.

Note that there will always be a point $x[0]$ that is on an entire solution
$\mathbf{x}$. In particular, the set

$X_{-n,0}(\mathbf{v}):=\Phi_{n,\sigma^{-n}(\mathbf{v})}(X)$

is well-defined, compact and non-empty as $f(X,v)\subset X$ is a continuous
image of a compact set. Moreover

$X_{-n-1,0}(\mathbf{v})=\Phi_{n,\sigma^{-n}(\mathbf{v})}\circ
f_{v[-n-1]}(X)\subset X_{-n,0}(\mathbf{v})$

and hence the set

$X_{0}(\mathbf{v}):=\bigcap_{n>0}X_{-n,0}(\mathbf{v})$

consists of points that lie on entire solutions.

An entire solution $\mathbf{x}$ is globally (pullback) attracting for input
$\mathbf{v}$ if

$\lim_{n\rightarrow\infty}h(X_{-n,0}(\mathbf{v}),x[0])=0$

where $h(A,B)=\sup_{y\in A}\inf_{z\in B}d(y,z)$. Following [16] say (1) for a
given input $\mathbf{v}$ has the ESP if it has a unique entire solution
$\mathbf{x}$ that is globally attracting for this input. An equivalent
condition is that there is single point in $X_{0}(\mathbf{v})$.

However, just as an autonomous dynamical system may be multistable, there may
be more than one locally attracting entire solution. Moreover, the system (1)
may have the echo state property for some inputs but not for others. This is
explored in [9] where we define a notion of echo index as the smallest number
of uniform attraction entire solutions (UAES) that attract almost all initial
states of the system.

If there are a number $m$ of UAESs
$\\{\mathbf{x}_{1},\ldots,\mathbf{x}_{m}\\}$ such that they decompose111apart
from a subset of zero Lebesgue measure. the whole phase space in sets that are
uniformly attracted to them, then we say the system has echo index $m$ and
write $\mathcal{I}(\mathbf{v})=m$; see [9, Definitions 3.3 and 3.4] for formal
definitions. Note that the echo index is shift-invariant, that is
$\mathcal{I}(\sigma({\bf v}))=\mathcal{I}({\bf v})$. We note that the echo
index $\mathcal{I}$ may take a number of values on a given closed shift-
invariant set $\mathcal{V}\subset\mathcal{U}$. In particular, we say
$\mathcal{V}$ has _consistent echo_ $n$ if $\mathcal{I}(\mathbf{v})=n$ for all
$\mathbf{v}\in\mathcal{V}$.

##  2 Sufficient conditions to determine minimum echo index

We show that certain assumptions on the behaviour of individual maps and input
sequences can be used to guarantee minimum echo index in terms of symbol
repetitions for the input.

###  2.1 Subshifts with min-max repetitions of input sequences

Suppose we have a finite set of symbols in $U$ and consider a subshift
$U_{\mathcal{B}}$ of finite type defined by a set of prohibited words
$\mathcal{B}$. We consider a particular class of subshifts
$\mathcal{U}_{\mathcal{B}}$ that are characterised by minimum and maximum
numbers of a repetition.

We say there is min repetition $m^{-}_{i}$ of symbol $i$ if $\mathcal{B}$
contains all words of the form

$j\,i^{m}\,k$

for all $m<m^{-}_{i}$ and all $j,k\neq i$. Similarly, we say there is max
repetition $m^{+}_{i}$ of symbol $i$ if $\mathcal{B}$ contains the word

$i^{m^{+}_{i}+1}.$

We say $m_{i}^{+}=\infty$ if there is no such word in $\mathcal{B}$. We say
the subshift of finite type $\mathcal{U}_{\mathcal{B}}$ has min-max
repetitions $m^{\pm}_{i}$ in this case.

For example, for the two symbols $0$ and $1$ consider the subshift with
prohibited words

$\mathcal{B}=\\{010,0110,01110,1111111,101,1001\\}.$

In this case $\mathcal{U}_{\mathcal{B}}$ has minimum $3$ and maximum $\infty$
repetitions of symbol $0$, and minimum $4$ and maximum $6$ repetitions of
symbol $1$. Note that this subshift can be expressed as a 1-step subshift on
blocks of $7$ consecutive symbols, which in its most general form will have
$2^{7}$ states and at most two arrows from each state. One can represent
$\mathcal{U}_{\mathcal{B}}$ using a smaller number of states using multiple
representations of the same state, this is shown in Figure 1.

Figure 1: Graphical representation of a subshift that has minimum $3$ repeats
of $0$ and between $4$ and $6$ repeats of $1$. For this case we have min-max
repetitions $m_{0}^{-}=3$, $m_{0}^{+}=\infty$, $m_{1}^{-}=4$ and
$m_{1}^{+}=6$.

If the subshift $\mathcal{V}$ supports an ergodic shift-invariant measure then
one can apply tools from ergodic theory [14, 1] or random dynamical systems
[2]. In particular, if sequences are chosen with respect to an ergodic shift-
invariant probability measure $\mu$ on $\mathcal{U}$ then one can use this to
ignore annoying sequences in $\mathcal{U}$ as long as they lie on some set the
can be shown to be zero-measure.

For example, given any set of non-zero probabilities $P$ assigned to symbols
$U$, the Bernoulli product measure $\mu=P^{\mathbb{Z}}$ assumes the
probability of a symbol is given, independent of location, by the same
distribution $P$. This can be shown to be an ergodic measure that is zero for
any set of sequences where the frequency of each symbol does not match the
probability $P$.

Note that there are many ergodic invariant measures corresponding to the
topological subshift shown in Figure 1. By allocating transition probabilities
one can define a Markov process that explores this subshift according to this
measure. A useful example of this is the family of ergodic subshifts on two
symbols

$\mathcal{U}(m_{0}^{\pm},m_{1}^{\pm},p_{0},p_{1})$

with minimum and maximum repeats $m_{i}^{\pm}$ and repeat probability
$0<p_{i}<1$, for each additional repeat of $i$ after the minimum number.
Figure 2 illustrates an example of a family of measures that correspond to the
topological subshift in Figure 1 and depends on the parameters $p_{0}$ and
$p_{1}$.

Figure 2: A Markov process for the topological subshift in Figure 1. For any
choice of parameters $0<p_{0}<1$ and $0<p_{1}<1$ the transition probabilities
shown generate a Markov process with min-max repetitions $m_{0}^{-}=3$,
$m_{0}^{+}=\infty$, $m_{1}^{-}=4$ and $m_{1}^{+}=6$. For example, after in
initial 3 repeats, symbol $0$ will be repeated with probability $p_{0}$ for
each further repeat.

###  2.2 Minimum echo for multiple attractor maps

We now discuss some testable (but restrictive) assumptions on the maps (7)
that can be used to give bounds on echo index. Suppose
$X\subset\mathbb{R}^{n}$ is a compact $n$-dimensional manifold, write
$\ell(\cdot)$ to denote Lebesgue measure on this and consider the finite set
of $M$ maps $f_{i}:X\rightarrow X$ for $i=0,\ldots,M-1$.

###### Assumption 2.1

Suppose that for each $i=0,\ldots,M-1$ we have:

  * (i)

$f_{i}$ is a continuously differentiable map that is almost everywhere a local
diffeomorphism.

  * (ii)

$f_{i}$ has a finite number of hyperbolic stable fixed points
$x_{i}^{0},\ldots,x_{i}^{L(i)-1}$.

  * (iii)

The basins of attraction $\mathcal{B}_{i}^{j}$ of $x_{i}^{j}$ exhaust the
measure of state space in measure, i.e.

$\ell(X\setminus\bigcup_{j}\mathcal{B}_{i}^{j})=0.$

  * (iv)

There is a $P(i,j,k)\in\\{1,\ldots,L(k)\\}$ such that

$x_{i}^{j}\in\mathcal{B}_{k}^{P(i,j,k)}.$

for all $k=1,\ldots,M-1$ and $j=0,\ldots,L(i)-1$.

###### Remark 2.1

Condition (i) implies in particular that the pre-image of any set of zero
Lebesgue measure under $f_{i}$ also has zero measure. Condition (ii) implies
that near each $x_{i}^{j}$ some iterate of $f_{i}$ is locally a contraction in
some neighbourhood; we characterise this in Lemma 2.1. Condition (iii) means
that $\\{\mathcal{B}_{i}^{j}\\}_{j=1}^{L(i)}$ is a full measure partition for
each $i$. Condition (iv) means that $x_{i}^{j}$ lies within the basin of an
attractor for $f_{k}$ but moreover it is a non-degeneracy assumption that
means that no attractor for $f_{i}$ is on the basin boundary for $f_{k}$.

We characterise the local contraction property more precisely:

###### Lemma 2.1

Suppose that Assumption 2.1 is satisfied. Then for each $i,j$ and any choice
of $0<\rho<1$ there is a neighbourhood $N_{i}^{j}$ of $x_{i}^{j}$ and
$n_{i}^{j}\geq 1$ such that if

$F:=(f_{i}|_{N_{i}^{j}})^{k}$

for any $k\geq n_{i}^{j}$ then $F:N_{i}^{j}\rightarrow N_{i}^{j}$ has a unique
fixed point at $x_{i}^{j}$ and moreover $F$ contracts by $\rho$

$\|F(x)-F(y)\|<\rho\|x-y\|.$

Proof. The assumption of a hyperbolic attraction fixed point $x_{i}^{j}$
implies linear stability, and hence that in any neighbourhood contained in
$\mathcal{B}_{i}^{j}$ some iterate of $f_{i}$ will be a contraction.

Now consider a specific input sequence
$\mathbf{v}=\\{v[k]\\}_{k\in\mathbb{Z}}$. Given some choice of
$A[0]\in\\{1,\ldots,N_{v[0]}\\}$ and applying Assumption 2.1(iv) we have

$x_{v[0]}^{A[0]}\in\mathcal{B}_{v[1]}^{P(v[0],A[0],v[1])}.$

Hence, associated with a sequence $\mathbf{v}$ and initial choice of attractor
$x_{v[0]}^{A[0]}$ there will be a unique sequence

$\left\\{x_{v[k]}^{A[k]}\right\\}_{k\geq 0}$

such that $x_{v[k]}^{A[k]}$ is an attractor for $f_{v[k]}$ contained in the
basin of $x_{v[k+1]}^{A[k+1]}$ for the map $f_{v[k+1]}$, namely

$A[k]=P(v[k-1],A[k-1],v[k]).$ (8)

We call such a sequence $A[k]$ a forward attractor sequence for $\mathbf{v}$
starting at $A[0]$. Since $A[k]$ is determined by (8) there will be only
finitely many forward attractor sequences and the number of these is bounded
above by $\min_{i}L(i)$. An entire attractor sequence is a sequence $A[k]$
satisfying (8) that is associated with a bi-infinite $\mathbf{v}$; this
depends not only on the $v[k]$ for $k\geq 0$ but also on $k<0$.

###### Lemma 2.2

Suppose that Assumption 2.1 holds for the system (7). Then there is a
$m_{\min}$ such that for any $\mathbf{v}$ with $m_{i}^{-}\geq m_{\min}$ for
all $i$ and any entire attractor sequence $A[k]$ for this $\mathbf{v}$, there
is a pullback attracting entire solution $x^{A[k]}_{v[k]}$ such that
$x^{A[k]}_{v[k]}\in\mathcal{B}_{v[k+1]}^{A[k+1]}$.

Proof. Choose any $0<\rho<1$; by applying Lemma 2.1 for all choices of $i,j$
we can find an $\epsilon>0$ and an $m_{\min}$ such that $m_{\min}>n_{i}^{j}$
and $B_{\epsilon}(x_{i}^{j})\subset N_{i}^{j}$. Pick any attractor sequence
$A[k]$ for a $\mathbf{v}$ that satisfies $m_{i}^{-}\geq m_{\min}$ and define

$x^{[A]}[k,n]:=\Phi_{n,\sigma^{k-n}(\mathbf{v})}B_{\epsilon}(x_{v[k-n]}^{A[k-n]}).$

This is a nested sequence in increasing $n$ for fixed $k$ and there is
contraction by $\rho$ over blocks of the same symbol. Hence the set is non-
empty and has diameter that shrinks to zero as $n\rightarrow\infty$. This
means that

$x^{[A]}[k]:=\bigcap_{n<k}x^{[A]}[k,n]$

consists of a single entire solution that pullback attracts an
$\epsilon$-neighbourhood of itself.

A consequence of this is that, for long enough minimum block-lengths we can
get a lower bound for echo index from the number of distinct entire attractor
sequences.

###### Theorem 2.3

Suppose that Assumption 2.1 holds for (7) and choose $m_{\min}$ and
$\mathbf{v}$ such that the conclusion of Lemma 2.2 holds. Suppose that there
are $E$ distinct entire attractor sequences for $\mathbf{v}$. Then the echo
index is at least $E$ for this sequence.

Proof. By Lemma 2.2 each entire attractor sequence $A[k]$ has a pullback
attractor $x^{[A]}[k]$. These are distinct as long as the entire attractor
sequences are distinct.

Under additional assumptions, one can ensure echo index, in particular for the
following case where we assume there is a uniform bound on how long it takes
points to enter a given neighbourhood of a single attractor.

###### Theorem 2.4

Suppose that Assumption 2.1 holds for (7) and in addition assume that there is
an $i$ such that $f_{i}$ has a single attracting fixed point $x_{i}^{0}$ such
that $h(f_{i}^{m}(X),x_{i}^{0})\rightarrow 0$ as $m\rightarrow\infty$. Then
there is a single attractor sequence and an $m_{\min}$ such that for every
$\mathbf{v}$ with $m_{i}^{-}\geq m_{\min}$ the system has echo index one.

Proof. In this case note that whenever $v[k]=i$ we have $A[k]=0$. Hence there
is only one entire attractor sequence. Moreover, given any $\epsilon$ there is
an $m$ such that all points must enter the neighbourhood
$B_{\epsilon}(x_{i}^{0})$ after at most $m$ iterates and hence the entire
solution pullback attracts all points in $X$.

###  2.3 Obstructions to conditions for echo consistency

Theorem 2.3 gives sufficient conditions to guarantee a minimum echo index for
the forced system in terms of $E$ the number of distinct attractor sequences
for the system with input $\mathbf{v}$. Conversely, Theorem 2.4 shows under
stronger conditions that the echo index is precisely one.

One might naively expect that an even stronger result than Theorem 2.3 may
follow, namely that the echo index is in fact $E$ for cases where $E\geq 2$.
This is not the case because under composition new attractors may appear near
a basin boundary.

As an example, define

$F(x)=x+x^{2}\sin\frac{\pi}{x},~{}~{}F(0)=0$

and $f(x)$ is the continuous function such that $f(x)=F(x)$ for $|F(x)|<1$ and
$f(x)$ is a linear function with constant slope $0.5$ for $|F(x)|>1$. Now
consider the maps

$f_{0}(x)=f(x)+1,~{}~{}~{}f_{1}(x)=f(x-1).$ (9)

One can verify that $f_{1}(x)$ and $f_{2}(x)$ each has a unique attracting
fixed point, namely $x_{0}^{0}=3$, $x_{1}^{0}=2$. However, $f_{0}\circ
f_{1}=f^{2}$ has infinitely many attracting fixed points separated by
repelling fixed points; it also has neutrally stable and linearly unstable
fixed points; see Figure 3. In summary, this system satisfies Assumption 2.1
and for any input there is only one attractor sequence $A[k]=0$ for all $k$,
but for an input $\mathbf{v}$ that alternates between $0$ and $1$ there are
infinitely many attracting entire solutions.

Hence for this input there is infinite echo index even though each.
Nonetheless, applying Theorem 2.4 there is a minimum block-length such that
there is only one attracting entire solution. In this case, numerical
simulations suggest this minimum is $2$, which would suggest that the system
has echo index one for all input sequences except the periodic sequence that
is an infinite repeat of $01$.

Figure 3: Maps $f_{0}$ and $f_{1}$ (9) such that each individual map has a
unique globally attracting linearly stable fixed point. However the
composition of $f_{1}\circ f_{0}$ has infinitely many stable fixed points.

##  3 Echo index dependence for RNNs

One of the main motivations for this study is the need to understand how many
responses there are for RNNs such as (a) trained discrete-time RNNs of the
form [20, 5]

$\displaystyle x[k+1]$ $\displaystyle=\phi(W_{r}x[k]+W_{i}u[k+1]+W_{f}z[k])$
(10)

driven by inputs $u[k]$ or (b) trained ESNs of leaky-integrator neurons:

$x[k+1]=G(x[k],u[k],z[k]),~{}~{}G(x,u,z)=(1-\alpha)x+\alpha\phi(W_{r}x+W_{i}u+W_{f}z),$
(11)

where $\alpha\in(0,1)$ quantifies the leakage, $\phi$ is a nonlinear function;
in both case the input sequence is given by $u[k]$ and $z[k]$ represents the
output feedback

$z[k]=\psi(x[k]).$ (12)

The nonlinear function $\phi$ is called an _activation function_ , we assume
it is bounded, monotonically increasing, and differentiable, e.g. a scaling of
$\tanh$. By contrast, $\psi$ is usually the identity function or a softmax
function.

###  3.1 Switching dynamics for RNNs

We consider an ESN with leaky-integrator neurons [11] and no output
feedback222As discussed in [8] whenever the readout is linear the feedback
term can be formally incorporated in the reservoir term. Therefore the absence
of the feedback term in an ESN state-update rule can represent an ESN state-
update rule after the training session where $W_{r}$ represents the “effective
reservoir” after training. for implementing the input-driven state update rule
(7). We consider a map of the form (11) but with

$G_{\alpha}(x,u):=(1-\alpha)x+\alpha\tanh(W_{r}x+W_{in}u).$ (13)

A finite set of input values $U=\\{u_{0},u_{1},\ldots,u_{M-1}\\}$ defines a
number $M$ of autonomous maps $f_{i}:X\rightarrow X$, where $X=[-1,1]^{r}$ and
$r$ the dimension of the internal state of the RNN, i.e. the number of
neurons. In this RNN’s framework, we can see that for small enough input
values, the echo index of the nonautonomous switching system is the number of
attractors of the input-free ESN. On the other hand, Theorem 2.4 has an
interpretation in terms of large amplitude forcing for RNN-like systems. In
fact, we know that large amplitude inputs drives the system in the saturating
tails regime of $\tanh$ characterised by a single attracting fixed point [8].
Thus Theorem 2.4 implies that on forcing an RNN with a large amplitude input
for long enough, the resulting nonautonomous RNN switching dynamics are
characterised by echo index one.

###  3.2 An example of input-driven RNN with multiple attractors

We provide here an example with a two dimensional ESN to better illustrate the
concepts. We choose,

$W_{r}=\begin{bmatrix}\frac{1}{2}&0\\\ 0&\frac{7}{4}\end{bmatrix},\quad
W_{in}=I_{2},$ (14)

with $I_{2}$ the identity matrix. We consider input sequences
$\mathcal{U}=\\{u_{0},u_{1}\\}^{\mathbb{Z}}$ where

$u_{0}:=\begin{pmatrix}\frac{1}{4}\\\ \frac{1}{20}\end{pmatrix}~{}\mbox{ and
}~{}u_{1}:=\begin{pmatrix}-\frac{1}{4}\\\ -\frac{1}{2}\end{pmatrix}.$ (15)

What follows can be observed for any value of $\alpha\in(0,1]$. We chose
$\alpha=\dfrac{1}{4}$ again because a small leak rate highlights the transient
dynamics.

The nonautonomous dynamics driven by some input sequence
$\mathbf{u}\in\mathcal{U}$ consists of sequence of applications of the two
maps:

$f_{0}(x):=G(x,u_{0}),~{}~{}f_{1}(x):=G(x,u_{1}).$

One can verify that the autonomous system $x[k+1]=f_{0}(x[k])$ has two
asymptotically stable points with a saddle between them along the vertical
line of $x_{1}\approx 0.45$ (see Figure 4) while the autonomous system
$x[k+1]=f_{1}(x[k])$ has only one (asymptotically stable) fixed point lying in
the quadrant where both variables are negatives. Note that [9, Theorem 4.1]
can be applied in this example to prove the existence of a local point
attractor lying in a strip of negative values of the $x_{2}$ variable.
Nevertheless, it is not straightforward to prove the existence of additional
local point attractors.

(a) (b)

Figure 4:  (a) Phase portrait of the map $f_{0}$. Red line represents the
stable manifold of the saddle. Some initial conditions have been evolved and
plotted (as black points) in order to visualise the vector field. (b) phase
space portrait of the map $f_{1}$. Note that the purple point is not a fixed
point but a _slow point_ [19, 8]. In fact, the map $f_{1}$ is close to a
saddle-node bifurcation which occurs nearby the position of such a slow point.

The stable manifold of the saddle of the autonomous map $f_{0}$ is a
horizontal line dividing the phase space in two sets. Let us denote with
$x^{*}$ the upper stable node of $f_{0}$ lying on the quadrant where both
variables are positive. Thanks to [9, Proposition B.1], we can consider the
phase space to be $X=[-1,1]^{2}$. Let us call $X^{up}$ the upper half where
$x^{*}$ lies (including the stable manifold line) and $X^{down}$ the remaining
part. On the other hand, the global attractor of the autonomous map $f_{1}$
consists of only an asymptotically stable fixed point lying in $X^{down}$.
Lemma 2.1 can be applied to show there exists a (minimum) positive integer
$m_{\min}$ for which $f_{1}^{m_{\min}}(X^{up})$ is mapped into the interior of
$X^{down}$. For the particular choice of parameters it turns out that
$m_{\min}\approx 30$.

###  3.3 Bifurcations of echo index

In this section we perform numerical simulations to compute the echo index of
the ESN’s example of section 3.2. For each choice of parameters determining
the sequence $\mathbf{v}$, we choose $50$ uniformly distributed initial
conditions and iterate $T$ steps before using a clustering algorithm to
numerically estimate the number of clusters in the final state - this is an
estimate of the echo index. Random sequences of length $T$ with varying
$m_{0}^{-}$ and $m_{1}^{+}$ are chosen, and we fix $m_{0}^{+}=40$ and
$m_{1}^{-}=1$. Figure 5 highlights that for $m_{0}^{+}$ large enough there
will be echo index $2$ for small enough $m_{1}^{+}$. Panel (a) shows $T=100$,
$p_{0}=0.9$ and $p_{1}=0.95$. Note that the short length of the timeseries is
not enough to collapse the initial conditions down to one of two values. Panel
(b) is as for (a) but with $T=1000$; in this case we have index one or two,
though some cases where $m_{1}^{+}$ is large have not been sampled for long
enough giving some spurious echo index $2$. Panel (c) is as for (a) but with
$T=2000$; we find what is presumably a clear boundary emerging between
different values of the echo index. Finally (d) shows the deterministic case
where $p_{0}=0$ and $p_{1}=1$ which corresponds to the periodic orbit where
there are $m_{0}^{-}$ repeats of $0$ and $m_{1}^{+}$ repeats of $1$ \- in this
case there a clear boundary between regions with echo index one and two; this
presumably corresponds to a bifurcation of the periodically forced map where a
second attractor appears. Note that for $m_{1}^{+}>30$ and large enough $T$,
we always find echo index one as suggested by the discussion at the end of
Section 3.2, but for $m_{0}^{-}$ small it is possible to find echo index one
for some values of $m_{1}^{+}<30$.

(a) (b)

(c) (d)

Figure 5: Estimates of echo index for the system (13) with parameters as in
the text for a range of random input sequences. Random sequences of length $T$
with varying $m_{0}^{-}$ and $m_{1}^{+}$ are chosen, fixing $m_{0}^{+}=40$ and
$m_{1}^{-}=1$. Note that the echo index is apparently 2 or more for small
enough $m_{1}^{+}$, or for small $T$. (a-c) show estimates of the echo index
for randomly generate inputs with the given min-max block-lengths and
probabilities of repetition $p_{0}$ and $p_{1}$ as in Figure 2. Examining
longer timeseries when estimating the echo index leads to a clear boundary
between regions of different echo index. (d) shows a special case where there
are only periodic inputs. We conjecture that the cases (a-c) will limit to (d)
for arbitrarily large $T$.

##  4 Conclusions

In this paper we have gone beyond work in [9] to highlight specific ways in
which the echo index varies with input signal (and hence whether the echo
state property holds for a given input). We present this for an iterated
function system on a compact space with an example application to an echo
state network.

We have so far only considered min-max block-length and repetition
probabilities on determining bounds on echo index but the actual value may
depend on much more subtle properties of words appearing in the input and
properties of the individual maps. It remains a challenge to better understand
this relationship between input set, system properties and echo index, even if
we restrict only to functions where the only attractors are fixed points.
Clearly, responses for cases where the autonomous dynamics of the maps include
more complex attractors (such as chaotic, quasiperiodic or period) will be
more challenging, not least because the a simple generalization of Assumption
2.1(iv) is unreasonable - a single attractor of one map will can stably
straddle several basins of attractions for attractors of another map.

One interpretation of our results is that minimum block-lengths are a proxy
for the input rate to the system - a long enough minimum block-length
corresponds to a slow rate of input. Our results Theorems 2.3 and 2.4 imply
that the expected behaviour can be characterised by attractors and basins of
the individual maps for slow enough rates of input. For shorter min block-
lengths the picture becomes more complex and the transient or nonautonomous
behaviour of each map features more strongly in the response to input - there
will be an analogy to rate-induced critical transitions [3] in the response
for such cases.

A future direction that seems worthy of study is the role of transients in
determining the echo index. The computations in Figure 5 show that even for
quite long (but finite) computations the number of responses may apparently
exceed the echo index which is attained asymptotically. A thorough
understanding of responses will be needed to explain such transient behaviour
of the echo index. This clearly depends not only on transients in the map
dynamics but also on waiting times to see certain words within the inputs.

### Acknowledgements

We thank EPSRC for support via EP/W52265X/1. We thank Lorenzo Livi, Muhammed
Fadera and Claire Postlethwaite for very informative discussions in relation
to this work.

### Data Access

The Matlab code for Figure 5 is available from
https://github.com/peterashwin/ashwin-ceni-2023.

## References

  * Adler et al. [1992] Roy L Adler, Peter Walters, et al.  _Symbolic dynamics and its applications_ , volume 135.  American Mathematical Soc., 1992.
  * Arnold [1998] Ludwig Arnold.  _Random Dynamical Systems_.  Springer, 1998.
  * Ashwin et al. [2012] Peter Ashwin, Sebastian Wieczorek, Renato Vitolo, and Peter Cox.  Tipping points in open systems: bifurcation, noise-induced and rate-dependent examples in the climate system.  _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ , 370(1962):1166–1184, 2012.
  * Baranwal et al. [2022] Mayank Baranwal, Ryan L Clark, Jaron Thompson, Zeyu Sun, Alfred O Hero, and Ophelia S Venturelli.  Recurrent neural networks enable design of multifunctional synthetic human gut microbiome dynamics.  _eLife_ , 11:e73870, jun 2022.  ISSN 2050-084X.  doi: 10.7554/eLife.73870.
  * Bianchi et al. [2017] F M Bianchi, Enrico Maiorino, M C Kampffmeyer, Antonello Rizzi, and Robert Jenssen.  _Recurrent Neural Networks for Short-Term Load Forecasting: An Overview and Comparative Analysis_.  Springer, 2017.  doi: 10.1007/978-3-319-70338-1.
  * Bruin [2022] Henk Bruin.  _Topological and Ergodic Theory Of Symbolic Dynamics_.  American Mathematical Society, 2022.
  * Ceni [2021] Andrea Ceni.  _Interpreting multi-stable behaviou in input-driven recurrent neural networks_.  PhD thesis, University of Exeter, 2021.  URL http://hdl.handle.net/10871/126978.
  * Ceni et al. [2020a] Andrea Ceni, Peter Ashwin, and Lorenzo Livi.  Interpreting recurrent neural networks behaviour via excitable network attractors.  _Cognitive Computation_ , 12:330–356, 2020a.
  * Ceni et al. [2020b] Andrea Ceni, Peter Ashwin, Lorenzo Livi, and Claire Postlethwaite.  The echo index and multistability in input-driven recurrent neural networks.  _Physica D: Nonlinear Phenomena_ , 412:132609, 2020b.
  * Jaeger [2001] Herbert Jaeger.  The “echo state” approach to analysing and training recurrent neural networks-with an erratum note.  _Bonn, Germany: German National Research Center for Information Technology GMD Technical Report_ , 148(34):13, 2001.
  * Jaeger et al. [2007] Herbert Jaeger, Mantas Lukoševičius, Dan Popovici, and Udo Siewert.  Optimization and applications of echo state networks with leaky-integrator neurons.  _Neural Networks_ , 20(3):335–352, 2007.  doi: 10.1016/j.neunet.2007.04.016.
  * Kloeden and Rasmussen [2011] Peter E Kloeden and Martin Rasmussen.  _Nonautonomous Dynamical Systems_ , volume 176 of _Mathematical Surveys and Monographs_.  American Mathematical Soc., 2011.
  * Lind and Marcus [1995] Douglas Lind and Brian Marcus.  _Introduction to Symbolic Dynamics and Coding_.  Cambridge University Press, 1995.
  * Lind and Marcus [2021] Douglas Lind and Brian Marcus.  _An introduction to symbolic dynamics and coding_.  Cambridge university press, 2021.
  * Lukoševičius and Jaeger [2009] Mantas Lukoševičius and Herbert Jaeger.  Reservoir computing approaches to recurrent neural network training.  _Computer Science Review_ , 3(3):127–149, 2009.  doi: 10.1016/j.cosrev.2009.03.005.
  * Manjunath and Jaeger [2013] G Manjunath and Herbert Jaeger.  Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks.  _Neural Computation_ , 25(3):671–696, 2013.  doi: 10.1162/NECO˙a˙00411.
  * Manjunath et al. [2012] G Manjunath, P Tino, and H Jaeger.  Theory of input driven dynamical systems.  In _ESANN 2012 proceedings, European symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning_ , pages 25–27. i6doc.com, 2012.  URL www.i6doc.com/en/livre/?GCOI=28001100967420.
  * Saleem et al. [2022] Rabia Saleem, Bo Yuan, Fatih Kurugollu, Ashiq Anjum, and Lu Liu.  Explaining deep neural networks: A survey on the global interpretation methods.  _Neurocomputing_ , 513:165–180, 2022.  ISSN 0925-2312.  doi: https://doi.org/10.1016/j.neucom.2022.09.129.
  * Sussillo and Barak [2013] David Sussillo and Omri Barak.  Opening the black box: Low-dimensional dynamics in high-dimensional recurrent neural networks.  _Neural Computation_ , 25(3):626–649, 2013.  doi: 10.1162/NECO˙a˙00409.
  * Tallec and Ollivier [2018] C. Tallec and Y. Ollivier.  Can recurrent neural networks warp time?  In _International Conference on Learning Representations_ , page , 2018.  URL https://openreview.net/forum?id=SJcKhk-Ab.


[1]Divya Srivastava

[1]Mechanical Engineering, Georgia Institute of Technology, North Ave,
Atlanta, 30308, Georgia, USA

2]Computer Science, Georgia Institute of Technology, North Ave, Atlanta,
30308, Georgia, USA

3]Aerospace Engineering, Georgia Institute of Technology, North Ave, Atlanta,
30308, Georgia, USA

# Improving Operator Situation Awareness when Working with AI Recommender
Systems

<EMAIL_ADDRESS>J. Mason Lilly<EMAIL_ADDRESS>Karen M.
Feigh<EMAIL_ADDRESS>*  [  [

###### Abstract

AI recommender systems are sought for decision support by providing
suggestions to operators responsible for making final decisions. However,
these systems are typically considered black boxes, and are often presented
without any context or insight into the underlying algorithm. As a result,
recommender systems can lead to miscalibrated user reliance and decreased
situation awareness. Recent work has focused on improving the transparency of
recommender systems in various ways such as improving the recommender’s
analysis and visualization of the figures of merit, providing explanations for
the recommender’s decision, as well as improving user training or calibrating
user trust. In this paper, we introduce an alternative transparency technique
of structuring the order in which contextual information and the recommender’s
decision are shown to the human operator. This technique is designed to
improve the operator’s situation awareness and therefore the shared situation
awareness between the operator and the recommender system.

This paper presents the results of a two-phase between-subjects study in which
participants and a recommender system jointly make a high-stakes decision. We
varied the amount of contextual information the participant had, the
assessment technique of the figures of merit, and the reliability of the
recommender system. We found that providing contextual information upfront
improves the team’s shared situation awareness by improving the human decision
maker’s initial and final judgment, as well as their ability to discern the
recommender’s error boundary. Additionally, this technique accurately
calibrated the human operator’s trust in the recommender. This work proposes
and validates a way to provide model-agnostic transparency into AI systems
that can support the human decision maker and lead to improved team
performance.

###### keywords:

shared mental models, shared situation awareness, human autonomy teaming, AI-
advised decision making

##  1 Introduction

In human-AI teaming, humans are paired with artificially intelligent
“partners” with the goal of leveraging the complementary strengths of each to
improve task performance.

However, important problems arise when partnering with so-called “black box”
systems - AI systems whose decision making processes are either not available
to or not easily interpretable by humans [1] \- namely, that it is difficult
or impossible to recognize erroneous output, or to know how to fix such
errors. Black box systems are particularly common in recommender systems where
the model or algorithms underlying the artificial intelligence (AI) is often
hidden from the human teammate [2, 3]. Utilizing a black box AI can lead to
satisfactory results if the decision problem is well-characterized and simple
or well aligned to the decision environment [4], or if the human is an expert
in the task. However, black box approaches are more prone to failure in
complex settings and those in which the decision environment is challenging.
When the result of the AI’s decision is unsatisfactory – or even catastrophic
– there are typically no mechanisms in place to help users understand why that
decision was suggested [5].

Recommender systems have great potential to support humans in high-stakes
domains. Significant work has sought to improve the quality of recommender
systems and ameliorate the challenges associated with the complex and opaque
algorithms inside. Efforts have included superior training for human operators
[6], improved visualizations for proposed solutions [7, 8], ranking candidate
solutions for the user to compare between [9], and attempting to explain the
underlying reasoning for the suggested solution [10]. Each of these approaches
have addressed challenges in niche classes of recommender systems, however
identifying best human-AI interaction practices for recommender systems
broadly remains an open problem.

This paper investigates an alternative method inspired by a combination of
naturalistic decision making (NDM) and the growing literature showing the
importance of having a shared mental model (SMM) even between humans and
artificial agents. Here we propose an approach designed to enhance the SMM
between humans and recommender systems by focusing on improving the team’s
shared situation awareness (SSA) which can be defined as “a shared
understanding of that subset of information that is necessary for [every
teammate’s] goals” [11]. We wish to ground the human in the decision context
and allow their judgment process to help supplement the joint decision. We
hypothesize that providing contextual information, specifically designed to
support the human’s judgment and improve SSA between the teammates, will
positively impact overall performance, decrease over reliance, and help to
calibrate trust. It will also provide one more tool to help improve the design
of recommender systems.

##  2 Background

###  2.1 Shared Mental Models

Mental models [12] are internal representations of how and why a phenomenon
occurs. People create mental models of complex systems whenever they interact
with one to facilitate their use of it [13]. An extension of mental model
theory is the concept of shared mental models [14], i.e. mental models that
are held in common across multiple individuals. The key finding from this area
regarding human-human teams is that if team members have similar mental models
of their shared task and of each other, then they are able to accurately
predict their teammates’ needs and behaviors. SMMs enable high-performing
teams in which everyone understands and anticipates the work of others in the
team. Teammates make decisions based on a common understanding of the state of
the world which positively affects team performance.

Related work has applied this concept analogously to human-AI teams, seeking
to foster mutual task-and-teammate understanding between a human and an AI
[15]. Work to improve the human’s mental model of the AI [16] and vice versa
[17, 18] is ongoing, however, few empirical studies have attempted to quantify
the development and effects of SMMs on human-autonomy teams.

Hanna and Richards’ study found a meaningful correlation between the effects
of trust and commitment to SMM development in teams of humans and intelligent
virtual agents (IVAs) [19]. Better shared mental models were found to
positively correlate with human trust in their artificial teammate. In
addition, teammate trust was found to significantly correlate with task
commitment, which is found to significantly correlate with improved team
performance. However, these measurements were made via subjective self-
assessments of SMM quality, and little research has attempted to establish a
quantitative link on this topic. This gap in literature can be attributed to
limitations such as defining and implementing the AI’s mental model and the
methods used to elicit and measure the human-AI SMM.

###  2.2 Recommender Systems & NDM

Decision Support Systems (DSS) are a common application of AI in various
industries and have been around for decades, long before AI methods were used
to drive their internal algorithms. They are designed to assist operators in
decision making tasks with a goal of helping people to make better decisions
faster. They do so by either simplifying the decision space or generating
potential solutions to reduce the burden on human decision-makers. A core
benefit of decision support systems is their ability to encode expert-level
information to support non-expert users. AI-driven recommender systems are a
subset of DSS in which an AI-based algorithm is utilized to recommend a course
of action. Specific examples of AI algorithms include machine learning,
reinforcement learning, and deep neural networks. The process of decision
making has been modeled in several ways [20, 21, 22], but for the purposes of
this paper, we will utilize the language of the well-established OODA loop
[23] which models the following cognitive processes:

  * •

Observation: the collection of data through sensory perception

  * •

Orientation: the analysis and synthesis of data to form one’s current mental
perspective

  * •

Decision: the determination of a course of action based on one’s current
mental perspective

  * •

Action: the physical playing-out of the decision

The field of NDM focuses on understanding, modeling, and improving how people
make decisions and perform cognitively complex functions (such as observing
and orienting) in demanding, real-world situations. Several works within NDM
have shown that context and spending sufficient time and energy on the process
of orienting or judging relevant information is influential to the decision
part of the decision making process [24, 25, 26].

In AI-advised decision making tasks, the AI is responsible for the Observing
and Orienting parts of the loop, and can also be used in the Decision part:
The AI gathers relevant information (Observe), and uses that information to
generate possible solutions (Orient) or a judgment of the decision that needs
to be made (Decide). The human’s role is mainly that of a safety checkpoint;
the final decision (Decide) and implementation (Act) is the operator’s
responsibility. However, the black box nature of many systems limits
appropriate understanding of what information the AI is observing, and how the
AI orients that information to produce its outputs. Because the human is not
present in the Observation and Orientation parts of the decision making loop,
this prevents the development of a team SSA because often the human
intentionally has little to no situation awareness.

The absence of a judgment phase for humans in the decision making process also
leads to a common issue with recommender systems, which is the human’s
propensity to overrely on the system’s suggestion. Overreliance has been
observed in a variety of domains [27], with both embodied [28, 29] and
nonembodied automated systems [30, 31]. This can be attributed to a variety of
explanations, though Wagner et al conclude that “people might assume
[automation] has knowledge [they] do not possess, viewing [automation]’s
actions, at times mistakenly, as a direct reflection of the intentions of the
developer, when in fact the [automation] may be malfunctioning or users may be
misinterpreting its abilities. Even when presented with evidence of a system’s
bad behavior or failure” [27]. Whatever the underlying cause, overreliance can
be mitigated when the human and the AI have an accurate understanding of each
other’s tasks, roles, and responsibilities, and can make accurate predictions
of each other’s decision making capabilities (that is, if they had an accurate
SMM).

###  2.3 Known Contributions to Mitigate Challenges

The field of explainable AI (XAI) aims to address challenges brought upon by
the black box barrier. Some approaches attempt to “open” the black box by
providing algorithmic transparency - insight into how an AI agent processes
information. Much work in this area focuses on providing explanations for
singular decision points [32, 33, 10]. The most common types of explanation
are global explanations, which explain the behavior of the model as a whole,
and local explanations, which offer a specific explanation for that specific
set of decision parameters. An example of a global explanation would be a full
set of heuristics or rules used to classify inputs into outputs, while a local
explanation might offer specific rules to map inputs to outputs.

While common, this approach has limitations. Explanations can take many forms
[33], and there is no consensus on what qualifies as a good or effective
explanation. For some AI algorithms, explanations can be straightforward, such
as displaying a decision tree or providing the features that triggered the
decision. For other algorithms, such as deep neural networks, the sheer volume
of parameters and nodes used by the algorithm makes extracting a
straightforward human-like explanation from them very difficult. Additionally,
the explanation given to the human is only useful if it is relevant to their
existing mental models, and only if it succeeds at increasing the human’s
understanding. The given explanation needs to be communicated at a level of
abstraction that makes sense to the individual user.

If successfully implemented, algorithmic transparency can provide post-hoc
context for an output to the decision maker. However, even if this is the
case, the decision making process is still truncated for the decision maker
because they do not Observe any information and have to work backwards to
Orient the explanation with the AI’s suggestion.

Other work has attempted to assist the human in evaluating the AI’s suggestion
by providing more interaction with the suggestion, or visualizing the
suggestion differently [7, 8]. In simple algorithms, these techniques can
increase understanding of the AI model, but for complex decision making tasks,
this alone may not be enough to support the human’s judgment.

All of these techniques make valid contributions to the push to make human-AI
decision making teams better, but as stand-alone techniques, they all truncate
the decision making process which necessarily creates gaps in the decision
maker’s knowledge, and therefore, judgment.

More recent work has attempted to implement model-agnostic methods to
interpret sophisticated machine learning models. Methods such as Local
Interpretable Model-agnostic Explanations (LIME) [34] and Shapley Additive
Explanations (SHAP) [35] have made significant advances in increasing
interpretability of complex algorithms. However, these methods are not very
robust– small changes in input features can lead to dramatic changes in
explanations [36]. Additionally, SHAP requires exponential computational time
[37]. While useful for fine-tuning algorithms, these methods are not suited to
real-time, high-stakes decision making tasks.

Given that the transparency techniques truncate the human’s decision making
process and existing model-agnostic methods are limited in real-time
capabilities, there is a need to support the human’s real time decision making
process in a way that does not rely on understanding the specific algorithms
used by AI recommender systems. To this end, we propose our technique of
providing contextual information of the task environment to better support the
human’s Orientation process when making a high-stakes decision.

##  3 Methodology

We conducted a two-phase between-subjects experiment in which participants
were asked to work with an automated recommender system to complete a decision
making task. We manipulated the amount of contextual information the
participants were provided, and we measured the effects on final task
performance, trust in the recommender system, user workload, and the shared
situation awareness between participants and the recommender. Participants
were recruited through an online recruitment platform (Prolific,
www.prolific.co). Individuals who were under 18, located outside of the USA,
not proficient in English, and/or did not have normal or corrected-to-normal
vision were excluded from the studies. The experiments were conducted online
and collected data from 90 participants in each study. Participants were
assigned randomly to a treatment group, and the order of scenarios shown to
participants was balanced.

###  3.1 Task Domain

Participants played the role of the commander of a craft in Mars orbit, tasked
with finalizing the entry, descent, and landing (EDL) trajectory of a probe to
a landing site. Participants were told that they were aided by an AI Mission
Computer that made a parallel evaluation of the proposed trajectory and
offered agreement or disagreement with the participant’s decision. After
seeing the AI’s recommendation, the participant had the final call on whether
to execute or abort the mission. We imposed no time constraints to the task.
For control and reproducibility, though the system was presented to
participants as “intelligent”, its responses in each scenario were
predetermined and fixed.

Figure 1: Task Outline and Metrics for EDL Trajectory Planning

The task is outlined in Fig. 1. First, the two agents in the team (human
participant and AI Mission Computer) independently assessed the state of the
world, considering three factors: the orbital positions of GPS satellites, the
locations of dust storms, and the atmospheric entry angle required by the
craft’s current orbital state. Using this information, each evaluated whether
the present world state permits a suitable landing (Initial Shared Situation
Awareness Check represented by the yellow diamond #1).

Then, the AI Mission Computer used the world state conditions to generate a
possible trajectory for the probe’s landing. The team was shown a set of six
figures of merit (FoM) that assessed the proposed trajectory: velocity vs.
altitude; heating rate, heat load, and acceleration vs time; latitude vs.
longitude; and landing confidence. The FoM charts were shaded to indicate
safe, risky, and dangerous thresholds, and the participants were instructed on
how to interpret each. Using the FoMs, the agents individually evaluated
whether or not to execute the landing trajectory (Final Shared Situation
Awareness Check represented by yellow diamond #2). The AI Mission Computer
recommendation was made known to the participant following registration of
their own. The participant made a final decision on whether to execute or
abort the mission in light of the AI’s recommendation (Final Decision
Agreement Point represented by yellow diamond #3).

###  3.2 Experiment Design and Task Procedure

Table 1: Design of Experiments Phase | IV | IV | DV | DV
---|---|---|---|---
(Group) | World State | Trajectory | World State | Trajectory
| Awareness | Awareness | Agreement | Agreement
1(1) | None | Obs |  | X
1(2) | None | Int |  | X
1(3) | Obs | Obs | X | X
1(4) | Obs | Int | X | X
1(5) | Int | Obs | X | X
1(6) | Int | Int | X | X
2(1) | None | Obs |  | X
2(2) | None | Int |  | X
2(3) | Obs | Obs | X | X
2(4) | Obs | Int | X | X
2(5) | Int | Obs | X | X
2(6) | Int | Int | X | X

Both experimental phases included two independent variables: World State
Awareness (3 levels) and Trajectory Awareness (2 levels). The second phase
additionally manipulated the accuracy of the AI. In the first phase, the AI is
correct in its evaluation of the trajectory (yellow diamond #2 in Fig. 1) in
100% of the scenarios. In the second phase, this accuracy drops to 60%, and
the AI “fails” in its suggested course of action when faced with a specific
condition, which is unknown to the participant. Each phase was designed as a
2x3 between-subjects design. We then compared between phases (making a 2x3x2
design, see Table 1) to evaluate how the change in AI’s reliability (100%
accuracy vs 60% accuracy) affected the human’s performance, team performance,
workload, and trust. Participants were assigned to a single treatment in a
single Phase, which was one of six combinations of the World State Awareness
and Trajectory Awareness levels. Participants completed 10 tasks within their
treatment group.

####  3.2.1 World State Assessment

Figure 2: World State Information from Left to Right: GPS, Atmosphere/Weather,
Anticipated Entry Angle

In the first part of the task, the agents separately evaluated how risky the
world state was (risky conditions vs safe conditions) in that scenario. The
World State Awareness independent variable had 3 levels: Observation-Only,
Interactive, or Absent altogether. The Absent mode was our control group, in
which participants were not given any information about the world state (World
State Awareness = None in Table 1). This lack of contextual information mimics
current black box systems, and participants in this group started directly at
the Trajectory Evaluation part of the experiment. In the Observation-Only
mode, participants viewed three information screens about the world state
(Fig. 2). They were not prompted for any further engagement with the
information. In the Interactive mode, participants viewed the world state
information screens and were asked a multiple-choice question about each
component, such as “Which satellite is closest to Mars?” and “How are the
weather conditions near the landing zone?”. These questions were intended to
increase the participant’s situation awareness by forcing them to actively
process each information source to a minimum degree.

After viewing and/or answering the question on each tab, participants were
asked if the world state conditions were risky or safe enough to attempt a
landing. The AI Mission Computer prepared its own answer to the same question.
For implementation purposes, for all phases, these responses were always
correct. This was the first assessment of the participant and AI having a
shared understanding, in this case of the world state. The participant was
then informed of the AI’s judgment of the world state conditions (and thus
whether they are in agreement). Then the experiment proceeded to the next
step.

####  3.2.2 Trajectory Evaluation

Figure 3: AI-generated trajectory with interactive questions

Following the World State Awareness assessment (Treatment Groups 3-6 in both
phases), a landing trajectory was presented, and the agents evaluated whether
or not to execute it. Participants were shown a spread of six figures of merit
that characterized the proposed flight plan. The Trajectory Awareness variable
had two levels: Observation-Only or Interactive. In the Observation-Only mode,
participants reviewed the six figures of merit for the suggested trajectory
and made an execute/abort decision. In Interactive mode, participants were
additionally asked to mark each of the six figures of merit as “Good”, “Bad”,
or “Maybe” according to that chart’s specific risk factors (Fig. 3). After
observing (Groups 1, 3, 5) or interacting with (Groups 2, 4, 6) each of the
trajectory charts, participants were asked to decide whether to execute the
trajectory. This allowed us to get our second measure of shared understanding
between the agents.

####  3.2.3 Final Assessment

As in the World State Assessment, the AI Mission Computer shared its own
evaluation after participants gave their assessment. Then participants were
asked to make a final decision. This allowed us to understand how much the
AI’s own assessment influenced that of the participant.

###  3.3 Experiment Procedure

Participants were asked to complete a consent form and given a pre-experiment
questionnaire to establish their baseline dispositional trust[38]. In both
Phases, participants were assigned to one of six experimental treatments.
Instructional videos, tailored to the specific treatment, were used to train
participants on the task and the AI Mission Computer. No task-relevant
experience was assumed. Participants then completed 6 practice rounds of the
mission planning task. Those meeting a specific performance standard were
allowed to proceed past the practice rounds. Passing participants then
proceeded to complete 10 trials of the mission planning task. At the end of
the experiment, participants completed two final questionnaires: 1) TLX
workload [39] and 2) i-THAU trust assessment [38].

###  3.4 Experiment Considerations and Limitations

The “AI” behavior utilized in these studies is fully predetermined, with
certain inputs (world state conditions) mapped to outputs (trajectory charts).
To simplify the analysis, the decision to execute or abort is decided
beforehand, which means we can manipulate the accuracy of the AI’s suggestion.
In Phase 1, the AI’s suggested course of action (whether to execute or abort a
trajectory) is correct 100% of the time. Designing the AI this way conflates
team agreement with task performance, and as these two metrics are not
realistically equivalent and 100% correct performance from AI is unlikely, we
manipulate the AI’s reliability in Phase 2. In experiment 2, the AI is correct
60% of the time, and fails in specific weather conditions. The decision to
have the AI fail in a predictable way mimics real world situations in which
algorithms that are trained on clearly labeled data and well-defined problems
are successful when encountering similar situations, but fail when faced with
noisy, degraded, or unfamiliar data. Additionally, by having the AI fail in a
predictable manner, we were able to discern if providing transparency by way
of contextual information helps the human understand the AI’s limitations
holistically (does the human pick up on the fact that the AI is giving
inaccurate suggestions?) and specifically (under what circumstances does the
AI fail?). These answers will inform the human’s mental model of their AI
partner, and thus their collaboration with it.

###  3.5 Measures

In each experimental phase, we first recorded the agreement between the
participant’s and the AI’s initial judgment of the world state conditions
(yellow diamond #1 in Fig. 1) and between their initial decisions of whether
or not to execute the proposed trajectory (yellow diamond #2 in Fig. 1). This
served as a crude measure of the final shared situation awareness (SSA)
between the participant and the AI. We computed this on a per participant
basis across all 10 scenarios they experienced. Second, we recorded the final
agreement between the human and the AI after the AI’s recommendation to
execute/abort was revealed. Third, we recorded the number of decisions per
participant that changed from initial to final decisions and from there
computed the percentage of initial disagreements that were resolved. Fourth,
for the interactive treatments, we were able to verify that the understanding
of the human matched the understanding of the AI.

Additionally, responses were recorded from the three subjective questionnaires
given to participants. The pre-experiment questionnaire was an i-THAu trust
assessment in which participants were asked to answer a series of statements
about their Faith in Persons and Faith in Technology on a seven-point Likert
scale from [-3:3]. A composite average of their answers informs their overall
dispositional trust in these two categories. The first post-experiment
questionnaire was the NASA TLX workload assessment which measured overall
cognitive workload. The second was the i-THAu trust assessment in which
participants responded to a series of statements about their experience of
working with the automated system on a seven-point Likert scale from [-3:3].
For both i-THAu assessments, a rating of -3 indicated a lack of trust – the
subject didn’t trust people/technology or depend on the AI to help them with
the EDL task, or they didn’t understand the role of the AI. Conversely, a
rating of 3 indicated a high level of trust in people/technology, as well as
high trust in and understanding of the AI.

##  4 Results

The results of both experimental phases obtained through objective and
subjective measures are presented here. For all objective metrics, we
performed a linear mixed effects analysis to determine the significance of the
relationship between the metric and our independent variables. The fixed
effects were the World State Awareness, Trajectory Awareness, the average
Dispositional Trust in People and the average Dispositional Trust in
Technology. We also included an interaction effect between World State
Awareness and Trajectory Awareness, and intercepts for subjects as a random
effect. We performed an ANOVA for each fitted linear mixed-effects model.

###  4.1 Experimental Phase 1: 100% Accurate AI

####  4.1.1 Objective Metrics

We first assessed the final shared situation awareness of the two teammates
(yellow diamond #2 in Fig. 1). Figure 4 shows the percentage of the initial
agreement between the participant and the AI during the World State Assessment
and Trajectory Evaluation phases. Those who did not receive world state
information were in agreement with the AI Mission Computer 60% - 85% of the
time. When world state information was available, the participant and AI
Mission Computer tended to agree 80% - 95% of the time, and the variation
reduced. This indicates that being aware of the decision environment (the
world state conditions) increased the shared situation awareness about the
specific decision task, i.e. trajectory evaluation. And, as the AI was correct
in this phase, the initial human performance also increased.

Figure 4: Final Shared Situation Awareness [%] Between Human and 100% Accurate AI During Evaluation Phases Table 2: ANOVA for Final SSA | df, error | F | P
---|---|---|---
WSAwareness | 2, 82 | 10.425 | 0.0001
TrajAwareness | 1, 82 | 2.960 | 0.0891
Faith in Tech | 1, 82 | 0.250 | 0.6181
Faith in Persons | 1, 82 | 0.596 | 0.4424
WS-A:Traj-A | 2, 82 | 0.650 | 0.5248
|  |  |

Table 2 presents the ANOVA for the fitted linear mixed-effects model of the
initial decision agreement between the participant and the AI Mission computer
before the AI’s decision was revealed (yellow diamond #2 in Fig. 1). The
results indicate that the World State Awareness significantly impacts the
initial agreement between the participant and AI. No other effects were
significant.

Figure 5: Final Agreement [%] Between Human and 100% Accurate AI

We also analyzed how the overall team performed by assessing the team’s final
agreement (after the AI reveals its decision, shown as yellow diamond #3 in
Fig 1), seen in Figures 5 and 6. The average final agreement was around 90%
for all levels of World State Awareness but the variance decreased as World
State Awareness increased. Here it seemed an accurate AI helped the
participants who did not have World State information perform as well as those
who did. A linear mixed effects analysis for this metric revealed that none of
the fixed effects were statistically significant in predicting the final
agreement between the team (Table 3).

Table 3:  ANOVA for Final Agreement | df, error | F | P
---|---|---|---
WSAwareness | 2, 82 | 0.076 | 0.9265
TrajAwareness | 1, 82 | 1.844 | 0.1782
Faith in Tech | 1, 82 | 0.357 | 0.5518
Faith in Persons | 1, 82 | 0.483 | 0.4888
WS-A:Traj-A | 2, 82 | 0.030 | 0.9706
|  |  |
Table 4:  ANOVA for Resolved Team Disagreements | df, error | F | P
---|---|---|---
WSAwareness | 2, 82 | 18.95 |  $<$0.0001
TrajAwareness | 1, 82 | 0.408 | 0.5244
Faith in Tech | 1, 82 | 0.004 | 0.9508
Faith in Persons | 1, 82 | 0.933 | 0.8565
WS-A:Traj-A | 2, 82 | 1.183 | 0.3115
|  |  |
Figure 6: Resolved Team Disagreements [%] Post-AI Suggestion

A well known challenge in human-automation/AI teams is that humans can often
overrely on their computational teammates. Figure 6 depicts the percentage of
scenarios in which the participant reconsidered their initial decision to
align with the AI after seeing what the AI suggested during the evaluation
phases. The results from the ANOVA for this metric (Table 4) indicate that the
World State Awareness is statistically significant in predicting when the
participant will change their decision to align with the AI’s suggestion. No
other effects were significant. Based on Figures 5 and 6, we can see that
those who did not have access to the World State conditions had a tendency to
be swayed by their AI partner to make the correct decision, while those who
did observe or interact with the world state conditions had higher initial
agreement with the AI. In this phase, agreement with the AI is equivalent to
correct assessment, and so the tendency to agree with the AI may be seen as
positive behavior.

####  4.1.2 Subjective Metrics

Figure 7: Composite Results of NASA TLX Questionnaire

To determine if adding the extra tasks of allowing or requiring participants
to be aware of the World State and Trajectory Assessments impacted the user’s
mental workload, we conducted a subjective mental workload assessment (NASA
TLX) at the end of the experiment. Figure 7 depicts the results of 4 of the 6
the NASA TLX subscales. Overall, increasing the participant’s situation
awareness required more effort of them but ultimately was less frustrating and
resulted in increased perceived performance. Results also show that those who
interacted with the figures of merit (as opposed to merely observing them)
during the Trajectory Evaluation phase tended to experience more mental demand
and frustration, and perceived themselves as less successful in their
performance.

The i-THAu assessment measures different aspects of trust between humans and
automation[38]. We used it for both pre- and post- trust assessments. The pre-
experiment i-THAu survey measures dispositional trust, that is an individuals’
propensity to trust other people or technology. Our analysis revealed no
impact of these trust factors on any objective measures such as SSA (Table 2),
final team agreement (Table 3), etc., nor were they found to influence
subjective measures such as cognitive workload or learned trust. The post-
experiment i-THAu assessment measures situational and learned trust. Figure 8
shows the human’s trust in the AI’s capabilities to complete the task, which
is measured following all 10 data collection scenarios. Positive values (+3)
indicate more trust in the AI’s capabilities, while negative values (-3)
indicate less trust in the AI’s capabilities. While the presence of and
interaction with world state information did not influence people’s trust in
the AI’s capabilities, there was (non-statistically significant) reduced level
of trust from those who interacted with the trajectory’s figures of merit vs
those who merely observed them.

Figure 8: Post-Experiment i-THAu Metric of Human’s Trust in AI’s Capabilities

###  4.2 Experimental Phase 2: 60% Accurate AI

In this phase of the experiment, the accuracy of the AI’s assessment of the
trajectory was reduced from 100% to 60%. 60% was chosen as it is below the
standard 70% threshold at which human participants begin to be impacted by
poorly performing automation [40].

####  4.2.1 Objective Metrics

As before, we assessed the final shared situation awareness of the two
teammates (yellow diamond #2 in Fig 1). Ideally the human would only agree
with their AI partner 60% of the time. This would indicate that the human can
recognize a limitation in their AI partner and adjust their mental model and
actions accordingly to ensure successful team performance. Figure 9 shows the
percentage of the final SSA between the participant and the AI during the
World State Assessment and Trajectory Evaluation phases. Those who did not
receive world state information tended to be in agreement with the AI Mission
Computer 60% - 75% of the time. Once world state information was introduced,
the participant and AI Mission Computer tended to agree 55% - 70% of the time.
Additionally, those who interacted with the trajectory figures of merit
trended closer to the ideal 60% agreement than those who merely observed, and
the variation was reduced. This indicates that being aware of the decision
environment (the world state conditions) helped to calibrate the accuracy of
the team’s shared situation awareness about the specific decision task, i.e.
trajectory evaluation.

Figure 9: Final Shared Situation Awareness [%] Between Human and 60% Accurate AI During Evaluation Phases Table 5:  ANOVA for Final SSA (Phase 2) | df, error | F | P
---|---|---|---
WSAwareness | 2, 82 | 1.577 | 0.2127
TrajAwareness | 1, 82 | 2.465 | 0.1203
Faith in Tech | 1, 82 | 0.020 | 0.8871
Faith in Persons | 1, 82 | 1.181 | 0.2804
WS-A:Traj-A | 2, 82 | 0.004 | 0.9962
|  |  |

Table 5 presents the ANOVA for the fitted linear mixed-effects model for final
SSA. The results indicate that no factors were statistically significant in
predicting the final SSA between the participant and AI when the AI was 60%
accurate.

Figure 10: Final Decision Agreement [%] Between Human and 60% Accurate AI for both Trajectory State Awareness Levels Table 6:  ANOVA for Final Agreement (Phase 2) | df, error | F | P
---|---|---|---
WSAwareness | 2, 82 | 2.597 | 0.0806
TrajAwareness | 1, 82 | 0.937 | 0.3358
Faith in Tech | 1, 82 | 2.888 | 0.0931
Faith in Persons | 1, 82 | 0.270 | 0.6046
WS-A:Traj-A | 2, 82 | 0.167 | 0.8462
|  |  |

As the final SSA could also be viewed as the initial decision agreement
between the AI and the Human, we went further to understand how the trends for
agreement were influenced by the reliability of the AI. Figure 10 depicts the
team’s final decision agreement (yellow diamond #3 in Fig. 1). In this figure,
we collapsed across trajectory awareness for clarity. Ideally, when the AI is
correct, the human would agree with it 100% of the time; conversely when the
AI is incorrect, that agreement would drop to 0%. The average final agreement
between the participant and the AI when the AI was correct (purple line in
Fig. 10) trended upward toward 100% when the participant had access to world
state information. We also see that the average final agreement between the
participant and the AI when the AI was incorrect (brown line in Fig. 10)
trended downward toward 40% as the participant’s world state awareness
increased. This indicates that providing contextual information to the human
improves their mental model of their AI partner (understanding that their AI
teammate is limited in some capacity). In turn, this improvement in the
human’s mental model helps them determine when (and when not) to align with
the AI’s suggestion for the final decision. Further this indicates a bias
toward AI agreement, particularly in the incorrect cases, that can be reduced,
but not eliminated, with the introduction of world state.

None of the fixed effects in the linear-mixed effects model were statistically
significant in predicting the final agreement between the team (Table 6).
However, when we performed a linear mixed effects analysis and ANOVA of the
same metric specifically in scenarios in which the AI is incorrect, we found
that World State Awareness was statistically significant in predicting final
agreement between the team (Table 7).

Table 7:  ANOVA for Final Agreement in Scenarios in which the AI is Incorrect (Phase 2) | df, error | F | P
---|---|---|---
WSAwareness | 2, 82 | 8.115 | 0.0006
TrajAwareness | 1, 82 | 2.471 | 0.1198
Faith in Tech | 1, 82 | 2.193 | 0.1425
Faith in Persons | 1, 82 | 0.300 | 0.5852
WS-A:Traj-A | 2, 82 | 1.167 | 0.3165
|  |  |

####  4.2.2 Subjective Metrics

Figure 11: Composite Results of NASA TLX Questionnaire

As in the first experimental phase, we recorded the subjective mental workload
of each participant using the NASA TLX assessment at the end of the
experiment. Figure 11 depicts the results of the NASA TLX questionnaire for
Experimental Phase 2. Similar to results from the first experimental phase,
participants who had access to world state information reported more effort (F
value = 3.5694) and mental demand (F value = 6.5426) required of them to
complete the task. Both frustration (F value = 0.7146) and perceived
performance (F value = 0.5273) were not significantly impacted by the world
state variable. Results also show that those who interacted with the figures
of merit (Trajectory Awareness Interaction) during the Trajectory Evaluation
phase tended to experience more mental demand, effort, and frustration, but
still perceived themselves as having high success in their performance.

Similar to Phase 1, the measured dispositional trust from the pre-experiment
i-THAu assessment was not statistically significant in predicting any
objective measures (Tables 5-7), nor subjective measures such as cognitive
workload or learned trust. Figure 12 shows the human’s trust in the AI’s
capabilities to complete the task measured in the post-experiment i-THAu
assessment. World State Awareness was not a statistically significant
(p=0.4764) predictor of Capability Trust. People were slightly more trusting
of the AI’s capabilities when they interacted with the trajectory charts.

Figure 12: Post-Experiment i-THAu Metric of Human’s Trust in AI’s Capabilities

##  5 Discussion

The underlying premise of this study was to understand the importance of
supporting human judgment as part of a more holistic cognitive process when
designing recommender systems. We tested this by evaluating the impact of
providing contextual information about the task environment, which we believed
would increase the situation awareness of the human decision maker. Further,
we aligned the contextual information specifically to match the information
used by an AI recommender system in this task domain, so that we could remove
any situation awareness discrepancies between the two agents. We compared this
judgment support technique and shared situation awareness to the more commonly
applied suggestion evaluation support technique of added interaction to
determine its relative impact on the overall mission performance. We also
added a final opportunity for the human to change their assessment to align
with that of the AI recommender, to understand how providing judgment support
and shared situation awareness might combat automation bias. Finally, we
further assessed the impact of the additional tasks on mental workload and
situational & learned trust.

The combined results from both phases indicate that providing contextual
information of the task environment to humans improves their team performance
with their AI partner, regardless of the accuracy of that partner. In the
first experimental phase, the AI Mission Computer was always correct, so
agreeing with the AI was equivalent to performing correctly. The results from
Phase 1 indicate that providing world state information increased shared
situation awareness (Fig. 4) and improved team performance. However, we see
from Experimental Phase 2, (when the AI Mission Computer is 60% accurate),
that when the AI is incorrect in its suggestion, providing world state
information helps to adjust the human’s situation awareness, thereby aligning
the team’s SSA (Fig. 9). This indicates that providing contextual information
is an effective technique in calibrating team’s shared situation awareness,
regardless of the autonomous agent’s reliability.

World state awareness also appeared to protect against over reliance on
automation/their AI teammate. The results of our baseline conditions show that
those who do not have access to the world state conditions have a tendency to
over-rely on their AI partner to make the correct decision (Figs. 10 and 6).
This finding is crucial because, as seen from objective metrics (Figs. 5 and
10) and subjective metrics (Figs. 7 and 11), not seeing the world state
information may result in acceptable final performance and less effort, mental
demand, and frustration on the human’s part, but ultimately leads to humans
who are less confident in their own performance and a blind deference to an AI
model that the human cannot understand, which as established, can lead to
catastrophic results. Correspondingly, providing contextual information about
the task environment aids the human in better understanding when their AI
teammate is incorrect (Figs. 9-10, Table 7), what to do about it, and improves
their overall confidence in their own task performance (Fig. 11).

Additionally, including the human in the Observation part of the decision
making process by giving them contextual information increases their cognitive
workload. The TLX data from both experimental phases (Figs. 7 and 11) indicate
that more effort and mental demand is required from humans when given
contextual information. When working with a reliable AI partner, having
contextual information ultimately makes the task less frustrating whereas
frustration increases if the human is working with an unreliable AI partner.
However, in both experimental phases, accessing the world state information
raised the human’s own confidence in their ability to perform the task.
Holistically, these workload results indicate that, although more is required
of the human when given contextual information, the technique ultimately
improves their objective performance as well as their own self-assessment in
completing the task with their AI partner.

The human’s trust in the AI is not affected by the presence of contextual
information, but is influenced by the reliability of the AI partner. In
Experimental Phase 1, there was little variation in the trust ratings of the
AI’s capabilities amongst the different levels of world state awareness (Fig.
8). However, in Experimental Phase 2, trust in the AI’s capabilities decreased
as world state awareness increased (Fig. 8), although this downward trend was
not statistically significant. This finding, that humans are less
trusting/forgiving of AI once they pick up on errors, is in line with current
literature.

Based on overall results, we can see that providing contextual information to
the human and including them in the Observation stage of the decision making
process can accurately calibrate team shared situation awareness, improve the
human’s overall task performance and their mental model of their AI teammate,
and decrease over-reliance on automation without much added workload.

In an effort to support the evaluation of a suggested course of action, we
introduced an interaction component during the Trajectory Evaluation phase.
The added interaction is meant to force the human to actively process each
information source. In Experimental Phase 1, we found that making sure humans
understood the “goodness” of the trajectory suggestion did not improve
performance of the human significantly nor did it improve shared situation
awareness between the teammates, which may contradict existing literature.
However, in Experimental Phase 2, those who interacted with the suggested
trajectory’s figures of merit had more accurate team SSA and final performance
than those who merely observed the AI’s suggestion. Additionally, from both
experimental phases, the TLX data trends overall indicate that the interaction
component tended to lead to a worse user experience than merely observing the
trajectory characteristics. Since no real-world automation will be flawless
and 100% accurate in its outputs, this important finding indicates that adding
interaction to AI outputs/suggestions is an effective technique for increasing
SSA in real-world human-AI teams with unreliable AI team members, despite the
poorer human experience. However, it is important to note that this technique
is not statistically significant in improving team SSA and final performance,
so supporting the evaluation of a suggested course of action is not enough on
its own to counteract the pitfalls of the black box phenomenon.

This study provides some evidence of the importance of situating a human
decision maker, aligning the human’s understanding of the world to match that
of its teammates (in this case AI ones), and providing support for not just
decision making, but also judgment elements of the cognitive process. The
study however is limited by the simplistic nature of the task assessed. It
also is limited in that the errors that the AI made were all risky and were
designed to be detectable by the novice human decision makers. The
participants here were also all novices and only worked in the team for a very
short period of time - this may have limited the trust metric findings.
Importantly, although we designed the experiments such that SSA should
increase by providing contextual information, we cannot explicitly verify this
because we do not measure the same metrics across treatments (e.g. we did not
have a way to explicitly measure the situation awareness of those who did not
see world state nor interact with figures of merit). But based on the final
decisions made, we know that as world state awareness increases, the team’s
performance increases. Since we know from SMM literature that high performing
teams utilize SMM, we can attribute this increase in performance to an
increase in SSA, which we achieved through the technique of adding contextual
information. Finally, we cannot separate out the impact of SSA from our best
guess as to the human’s mental model.

##  6 Conclusions

Results indicate that introducing transparency of the information used to
generate trajectories accurately calibrates the shared situation awareness
between the human and the AI (Figs. 4 and 9). While increasing transparency
into the mission planning system by including the human in the Observation
part of the process resulted in increased cognitive workload on the human’s
part, it also improved human robustness to overreliance (Figs. 6 and 10),
final team performance when the AI teammate was unreliable (Fig. 10), and the
participants’ confidence in their own performance (Fig. 7).

Added interaction with the suggested solution led to poorer SSA (Fig. 4),
increased workload (Fig. 7), and less trust in the AI teammate (Fig. 8) when
it was 100% reliable. While added interaction did seem to help calibrate SSA
and trust in the AI when the AI was unreliable (Figs. 9 and 12), it was not
statistically impactful in these metrics, and therefore we cannot conclude
that this technique is useful holistically.

Our proposed technique of displaying contextual information about the task
environment is broadly applicable to high-stakes decision support systems, in
particular to improve human-AI shared situational awareness in systems built
around black box recommender systems. This work can be extended by testing if
these findings hold when contextual information is relayed differently, in a
different decision making domain, or how this technique affects human-AI team
performance in conjunction with other transparency methods.

Acknowledgments

This work was funded by Sandia National Laboratory (SNL) with Dr. Paul Schutte
serving as Program Manager. This work is solely that of the authors and does
not represent an official SNL position.

## References

  *   * Lipton [2017] Lipton, Z.C.: The Mythos of Model Interpretability. arXiv:1606.03490 [cs, stat] (2017). arXiv: 1606.03490
  * Baker et al. [1995] Baker, J.A., Kornguth, P.J., Lo, J.Y., Williford, M.E., Floyd, C.E.: Breast cancer: prediction with artificial neural network based on bi-rads standardized lexicon. Radiology 196 3, 817–22 (1995)
  * Parmar et al. [2021] Parmar, S., Illingworth, D.A., Thomas, R.P.: Model blindness: A framework for understanding how model-based decision support systems can lead to performance degradation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting (2021)
  * Martignon and Hoffrage [2002] Martignon, L., Hoffrage, U.: Fast, frugal, and fit: Simple heuristics for paired comparison. Theory and Decision 52, 29–71 (2002)
  * Board [2014] Board, U.S.N.T.S.: Descent Below Visual Glidepath and Impact with Seawall, Asiana Airlines Flight 214, Boeing 777-200ER, HL7742, San Francisco, California, July 6, 2013. Aircraft accident report. National Transportation Safety Board, ??? (2014). https://books.google.com/books?id=b4zrjgEACAAJ
  * Yang et al. [2019] Yang, Y., Kandogan, E., Li, Y., Sen, P., Lasecki, W.S.: A study on interaction in human-in-the-loop machine learning for text analytics. In: IUI Workshops (2019)
  * Becker et al. [2001] Becker, B., Kohavi, R., Sommerfield, D.: Visualizing the simple baysian classifier. In: Information Visualization in Data Mining and Knowledge Discovery, pp. 237–249 (2001)
  * Erra et al. [2011] Erra, U., Frola, B., Scarano, V.: An interactive bio-inspired approach to clustering and visualizing datasets. In: 2011 15th International Conference on Information Visualisation, pp. 440–447 (2011). IEEE
  * Monroe [2019] Monroe, C.C.: Optimizing military planners course of action decision making. Master’s thesis, Atlanta GA (2019)
  * Dazeley et al. [2021] Dazeley, R., Vamplew, P., Foale, C., Young, C., Aryal, S., Cruz, F.: Levels of explainable artificial intelligence for human-aligned conversational explanations. Artificial Intelligence 299 (2021)
  * Endsley et al. [2003] Endsley, M.R., Bolte, B., Jones, D.G.: Designing for Situation Awareness: An Approach to User-Centered Design. Taylor & Francis, ??? (2003)
  * Johnson-Laird [1983] Johnson-Laird, P.N.: Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Harvard University Press, ??? (1983)
  * Norman [1983] Norman, D.A.: Some Observations on Mental Models. In: Gentner, D., Stevens, A.L. (eds.) Mental Models, (1983)
  * Converse et al. [1993] Converse, S., Cannon-Bowers, J., Salas, E.: Shared mental models in expert team decision making. Individual and group decision making: Current issues 221, 221–46 (1993). Publisher: Hillsdale
  * Andrews et al. [2022] Andrews, R.W., Lilly, J.M., Srivastava, D., Feigh, K.M.: The role of shared mental models in human-ai teams: a theoretical review. Theoretical Issues in Ergonomics Science, 1–47 (2022)
  * Bansal et al. [2019] Bansal, G., Nushi, B., Kamar, E., Lasecki, W., Weld, D., Horvitz, E.: Beyond accuracy: The role of mental models in human-ai team performance. In: HCOMP (2019). AAAI
  * Fan and Yen [2011] Fan, X., Yen, J.: Modeling Cognitive Loads for Evolving Shared Mental Models in Human–Agent Collaboration. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 41(2), 354–367 (2011) https://doi.org/10.1109/TSMCB.2010.2053705
  * Walsh and Feigh [2022] Walsh, S.E., Feigh, K.M.: Understanding human decision processes: Inferring decision strategies from behavioral data. Journal of Cognitive Engineering and Decision Making 16(4), 301–325 (2022)
  * Hanna and Richards [2018] Hanna, N., Richards, D.: The Impact of Multimodal Communication on a Shared Mental Model, Trust, and Commitment in Human–Intelligent Virtual Agent Teams. Multimodal Technologies and Interaction 2, 48 (2018). Publisher: Multidisciplinary Digital Publishing Institute
  * Dewey [1910] Dewey, J.: How We Think. D.C. Heath, New York, NY, USA (1910)
  * Simon and Barnard [1976] Simon, H.A., Barnard, C.I.: Administrative Behavior: A Study of Decision-making Processes in Administrative Organization. Free Press paperback. Free Press, ??? (1976)
  * Kersten and Szpakowicz [1994] Kersten, G.E., Szpakowicz, S.: Decision making and decision aiding. defining the process, its representations, and support. Group Decision and Negotiation, 237–261 (1994)
  * Hendrick [2009] Hendrick, D.: Complexity theory and conflict transformation: An exploration of potential and implications (2009)
  * Harrington and Ottenbacher [2009] Harrington, R.J., Ottenbacher, M.C.: Decision-making tactics and contextual features: Strategic, tactical and operational implications. International Journal of Hospitality & Tourism Administration (2009)
  * Ben-Akiva et al. [2012] Ben-Akiva, M., Palma, A., Mcfadden, D., Abou-Zeid, M., Chiappori, P., Lapparent, M., Durlauf, S., Fosgerau, M., Fukuda, D., Hess, S., Manski, C., Pakes, A., Picard, N., Walker, J.: Process and context in choice models. Marketing Letters 23, 439–456 (2012) https://doi.org/10.1007/s11002-012-9180-7
  * Vazquez-Diz et al. [2019] Vazquez-Diz, J., Morillo Baro, J., Reigal Garrido, R., Morales Sanchez, V., Hernández Mendo, A.: Contextual factors and decision-making in the behavior of finalization in the positional attack in beach handball: Differences by gender through polar coordinates analysis. Frontiers in Psychology 10 (2019) https://doi.org/10.3389/fpsyg.2019.01386
  * Wagner et al. [2018] Wagner, A.R., Borenstein, J., Howard, A.: Overtrust in the robotic age. Commun. ACM 61(9), 22–24 (2018) https://doi.org/10.1145/3241365
  * Robinette et al. [2016] Robinette, P., Li, W., Allen, R., Howard, A.M., Wagner, A.R.: Overtrust of robots in emergency evacuation scenarios. In: 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 101–108 (2016). https://doi.org/10.1109/HRI.2016.7451740
  * Booth et al. [2017] Booth, S., Tomokin, J., Pfister, H., Waldo, J., Gajos, K., Nagpal, R.: Piggybacking robots: Human-robot overtrust in university dormitory security. In: 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI, pp. 426–434 (2017)
  * Lee and See [2004] Lee, J.D., See, K.A.: Trust in automation: Designing for appropriate reliance. Human Factors 46(1), 50–80 (2004) https://doi.org/10.1518/hfes.46.1.50{_}30 https://doi.org/10.1518/hfes.46.1.50_30392. PMID: 15151155
  * Bussone et al. [2015] Bussone, A., Stumpf, S., O’Sullivan, D.: The role of explanations on trust and reliance in clinical decision support systems. In: 2015 International Conference on Healthcare Informatics, pp. 160–169 (2015). https://doi.org/10.1109/ICHI.2015.26
  * Lakkaraju et al. [2017] Lakkaraju, H., Kamar, E., Caruana, R., Leskovec, J.: Interpretable & explorable approximations of black box models. CoRR abs/1707.01154 (2017) 1707.01154
  * Guidotti et al. [2018] Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5) (2018)
  * Ribeiro et al. [2016] Ribeiro, M.T., Singh, S., Guestrin, C.: ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier. arXiv (2016). https://doi.org/10.48550/ARXIV.1602.04938 . https://arxiv.org/abs/1602.04938
  * Lundberg and Lee [2017] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc., ??? (2017)
  * Alvarez-Melis and Jaakkola [2018] Alvarez-Melis, D., Jaakkola, T.S.: On the Robustness of Interpretability Methods. arXiv (2018). https://doi.org/10.48550/ARXIV.1806.08049 . https://arxiv.org/abs/1806.08049
  * Aas et al. [2019] Aas, K., Jullum, M., Løland, A.: Explaining individual predictions when features are dependent: More accurate approximations to Shapley values. arXiv (2019). https://doi.org/10.48550/ARXIV.1903.10464 . https://arxiv.org/abs/1903.10464
  * [38] Razin, Y.: Interdependent Trust for Humans and Automation Survey. Available at https://sites.gatech.edu/feigh-lab/publications/ (2022)
  * Hart [2006] Hart, S.G.: Nasa-task load index (nasa-tlx); 20 years later. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, pp. 904–908 (2006). Human Factors and Ergonomics Society
  * Wickens and Dixon [2007] Wickens, C.D., Dixon, S.R.: The benefits of imperfect diagnostic automation: a synthesis of the literature. Theoretical Issues in Ergonomics Science 8(3), 201–212 (2007) https://doi.org/10.1080/14639220500370105