11institutetext: Department of Computer Science, Università di Pisa
22institutetext: Institute for Informatics and Telematics (IIT), National
Research Council (CNR) 22email<EMAIL_ADDRESS>
# Optimizing Delegation Between Human and AI Collaborative Agents††thanks:
This work was partially supported by the CHIST-ERA-19-XAI010 SAI project. M.
Conti’s and A. Passarella’s work was partly funded by the PNRR - M4C2 -
Investimento 1.3, Partenariato Esteso PE00000013 - "FAIR" funded by the
European Commission under the NextGeneration EU programme.

Andrew Fuchs 1122 0000-0001-7191-8781 Andrea Passarella 22 0000-0002-1694-612X
Marco Conti 22 0000-0003-4097-4064

###### Abstract

In the context of humans operating with artificial or autonomous agents in a
hybrid team, it is essential to accurately identify when to authorize those
team members to perform actions. Given past examples where humans and
autonomous systems can either succeed or fail at tasks, we seek to train a
delegating manager agent to make delegation decisions with respect to these
potential performance deficiencies. Additionally, we cannot always expect the
various agents to operate within the same underlying model of the environment.
It is possible to encounter cases where the actions and transitions would vary
between agents. Therefore, our framework provides a manager model which learns
through observations of team performance without restricting agents to
matching dynamics. Our results show our manager learns to perform delegation
decisions with teams of agents operating under differing representations of
the environment, significantly outperforming alternative methods to manage the
team.

###### Keywords:

Reinforcement Learning Markov Decision Process Delegation Learning to defer
Hybrid Decision-making

##  1 Introduction

Assuming a context with humans working directly in collaboration with
autonomous/artificial agents, it is essential to enable a team dynamic
eliciting the best combined performance or reduce agent-specific costs. For
example, think of an autonomous car, where a human driver or an AI agent can
take decisions on the next driving action. It is well known [1, 19] that
neither the human nor the agent is always the best choice, as either can make
mistakes depending on the driving context. Therefore, it is of utmost
importance to design a _delegation policy_ deciding, at any point in time, who
between the human driver and the AI agent should operate the car. More
specifically, a reasonable goal is to identify how and when to enable agent
actions to maximize overall team performance while also considering any costs
when an agent is operating. Of course, the definition of performance will vary
with each application, but the need for performance will not. As such, we
investigate a method for guiding delegation decisions from the perspective of
a managing authority which selects the agent who chooses the action(s) the
team will take in their shared current state.

With the emergence of a hybrid team comprised of human and artificial agents,
it is essential we provide oversight or imbue the agents with an ability to
identify the key performer in a given context. To this end, we define and
train the model for a Reinforcement Learning (RL) based manager which
identifies desirable delegation decisions given context and knowledge
regarding agent performance. To ensure a more realistic team dynamic, we
assume the manager is not directly observing the individual actions of the
agents, but rather that the manager must learn to delegate strictly by
learning an association between agents, states, delegations, and outcomes.

In this paradigm, such a manager will allow for behavior learning through
indirect observations of agents, with those observations being without any
unfair access to private or domain-specific knowledge. In addition, this
approach reduces dependencies between the model through which the _agents_
learn how to behave, and that through which the _manager_ learns how to
delegate. With this approach, we will show that our manager can learn a
suitable policy. Further, the results demonstrate how the learned policies
properly align with the reward function, and how the manager’s performance
exceeds that of relevant reference alternatives.

In the paper we define a learning model for the manager that can be cast to a
well-formulated Reinforcement Learning instance based on a Markov Decision
Problem (MDP, see Section 2). Without loss of generality, we assume that both
the human and the AI agent can be modeled through their own MDP, whose
specific actions and policies for transition are independent from each other,
and not under the control of the manager. The manager is thus modeled through
a separate MDP, whose policy we show how to optimize, based on the underlying
policies of the individual agents. This allows us to analytically define
optimal policies for the manager, that can be learned according to standard RL
machinery. This is quite important, as we demonstrate the possibility of
formal optimization of the problem addressed.

In addition, we highlight two further contributions of this work, which
distinguish our work from prior literature (analyzed in Section 3). First, our
inclusion of a manager delegating between agents moving through the _same_
state space with _different_ actions and transitions is a novel
conceptualization. The closest alignment to this view is the use of different
options (defined in Section 2) for moving inside a unique MDP, and their
corresponding option policies. In this case, a unique MDP is split into
several sub-problems which are managed through a separate learning problem.
However, this does not consider managing _multiple_ MDPs for solving a given
problem, as in our case, where each MDP represents a specific managed agent,
as explained in Section 4.

Another clear distinction is our consideration of distinct and single agent
actions at a given time. This means we are not allowing concurrent actions for
the various agents of the team. Rather, the task hinges on the manager’s
ability to learn an association between contexts and individual agent
performance. Therefore, the concepts in the multi-agent cases do not apply.

To summarize, the allowance of distinct and compatible models of the
individual agents’ MDPs is a worthwhile and novel concept. Under this
paradigm, we can remove the common assumptions of many domains while allowing
for compatibility with prior RL and related theory. These methods allow us to
show analytical results as well as test agent performance.

Before presenting our approach to model and optimize the manager (Section 4)
and the performance achieved in a relevant case (Section 6), in Section 2 we
summarize the main background we exploit in terms of analytical approach, and
in Section 3 we analyze in more details the relations and advancements with
respect to the state of the art.

##  2 Background

###  2.1 Markov Decision Process (MDP)

Given the use of RL agents to define both our underlying agents and the
manager, we assume that the acting agents, those performing actions in the
environment, operate with respect to a Markov Decision Process (MDP). We
assume a standard definition of the MDP $M=\langle S,A,R,T,\gamma\rangle$,
where:

  * •

$S$, the state space, is the set of states the agent can traverse

  * •

$A$, the action space, is the set of actions an agent is allowed to take

  * •

$R:S\times A\rightarrow\mathbb{R}$, is the reward function denoting action
utility

  * •

$T:S\times A\times S\rightarrow[0,1]$, the transition function, denotes the
probability of transitioning between two states when an action is performed

  * •

$\gamma$, the discount parameter, dictates how strongly values decay according
to their temporal distance

With the assumption of an MDP, we know the actions of the agents depend
specifically on the current state/context.

###  2.2 Semi-Markov Decision Process (SMDP)

For our manager, we encounter parallels with concepts investigated with
regards to Semi-Markov Decision Processes (SMDP), which allows for splitting
the problem into a hierarchy of representations. The base level includes a
model of discrete states and actions, with the addition of a policy which
guides the action selection at this bottom level. Higher levels abstract state
and/or action representations into options. The use of options enables
treating a sequence of states and actions as a singular item at the higher
level of abstraction, which enables behavior learning to at an abstracted
level and at the lower level.

In the case of a SMDP, the Markov assumption is relaxed. In one sense, this
can be a result of temporally extended actions at the lower abstraction level,
where the transitions depend on more than just current state and the selected
action. In the other case, SMDPs allow actions conditioned on the history of
states and actions. This history is commonly a subset of the episode history
and only includes information since the start of a particular time window. The
window depends on the use case, but a common one is that of options in which a
hierarchy can be formed with a collection of actions treated as a single
abstract action (i.e., an option). In this case, the option could take
multiple time steps to complete, with the total time being stochastic. As
such, the estimates of transitions and rewards need to accommodate the
variance in times and actions. A common representation is options
$\langle\mathcal{I},\pi,\beta\rangle$ where:

  * •

$\mathcal{I}\subseteq S$ is the set of states where option $o$ is available
for selection/activation

  * •

$\pi:S\times A\rightarrow[0,1]$ is the policy for behavior used for the option

  * •

$\beta:S^{+}\rightarrow[0,1]$ is the likelihood an option terminates in a
state

with $S^{+}$ referring to the original state space with an included optional
terminating/trap state. As will be exemplified in Section 5, such a trap state
allows to easily model terminating conditions once an agent has reached a
certain goal and should stop any further actions afterwards.

In the SMDP model, the notion of rewards and transitions are updated to
consider the new form of interaction agents have with the environment. Rewards
are now defined as

$r^{o}_{s}=\mathbb{E}\left[r_{t+1}+\gamma
r_{t+2}+\cdots+\gamma^{k-1}r_{t+k}|\mathcal{E}(o,s,t)\right]$ (1)

where $t+k$ represents the random option termination times according to $\pi$
and $\beta$ and $\mathcal{E}(o,s,t)$ is an indicator denoting option $o$ was
initiated in state $s$ at time $t$. In this representation, we see the length
of an option and the observed rewards dictating the overall option’s reward.
This accounts for both the stochastic time and the behavior policy underlying
agent interaction with the environment. As for state predictions, the common
form is a decaying sum of joint probabilities

$p^{o}_{ss^{\prime}}=\sum^{\infty}_{k=1}p(s^{\prime},k)\gamma^{k}$ (2)

where $p(s^{\prime},k)$ is the probability that the option terminates in state
$s^{\prime}$ after $k$ time steps. Note that this does not define a valid
distribution but ensures estimates at longer horizons offer much less
importance than the more accurate and shorter horizon estimates. These rewards
and visitation estimates can then be combined to define a value function

$V^{\mu}(s)=\mathbb{E}\left[r_{t+1}+\gamma
r_{t+2}+\cdots+\gamma^{k-1}r_{t+k}+\gamma^{k}V^{\mu}(s_{t+k})|\mathcal{E}(\mu,s,t)\right]$
(3)

for Markov policy $\mu$. The policy $\mu$ selects the option to use in state
$s$ while $k$ is the option duration. The state-action value can be estimated
similarly. Further, this enables temporal difference methods for policy
learning. Specifically, we can define a standard $Q()$ function of a
Reinforcement Learning problem [23]

$Q(s,o)\leftarrow Q(s,o)+\alpha\left[r+\gamma^{k}\max_{o^{\prime}\in
O_{s^{\prime}}}Q(s^{\prime},o^{\prime})-Q(s,o)\right]$ (4)

where the use of $\gamma^{k}$ to discount the future value follows [2].

##  3 Related Work

###  3.1 Hybrid Reinforcement Learning for Single Agent

In [9], the goal is to define a model and training method for option learning
in a single agent setting. More specifically, the approach focuses on learned
options which are “reusable”, where agents are only expected to add new
options if they improve the performance. Another relevant topic is that of
[5], where the authors generate Cooperative Consecutive Policies (CCP). This
is intended to generate multiple policies which can be utilized sequentially
to accomplish a larger task. The goal with CCP is to reduce the complexity of
any one task while improving the compatibility/performance with respect to the
consecutive policies.

Generally, approaches found in the single-agent Hierarchical Reinforcement
Learning (HRL) literature are more closely aligned to our approach but are
attempting to learn policies at both levels of the hierarchy. Instead, we
assume our option policies are controlled by existing behavior policies for
our available delegation agents. This means the manager agent is not learning
the option policies but is instead learning a delegation policy to select
“options”, which in our case are represented by our agents. This means we can
instead focus on more direct behavior learning for the manager, but we will
need to still ensure that the value function reflects the dynamics the manager
will observe when selecting between existing agents.

###  3.2 Hybrid Reinforcement Learning in Multi-Agent Reinforcement Learning
Contexts

There are numerous approaches attempting to support multiple agents in a
Reinforcement Learning setting, including cases regarding the representation
of agent behaviors via “options” in fully or partially observable environments
[3, 4, 13, 14, 16, 20, 21, 24]. Similar to these approaches of temporal
abstraction, [10] demonstrates state abstraction in a pursuit-evasion game. As
a relevant multi-agent HRL example, consider the approach outlined in [24]
which learns to optimally assign agents to sub-tasks in a cooperative multi-
agent task. In [24], similar agents are grouped so they can share knowledge
and be selected for compatible tasks. Selection is performed based on agent
ability regarding the task. In this case, the ability measure compares agent
experience with the sub-task to compare the relation of its knowledge to the
features of the proposed task. This gives a likelihood of a sub-task being
selected for an agent which is based on a notion of compatibility.

As we are in fact utilizing multiple agents, it would seem at first like there
could be significant relevance to our scenario. In our case, as we are not
including cases of simultaneous/concurrent control, we focus on what could
equate to a single agent HRL scenario. The manager is only responsible for
identifying an agent to delegate and then observing the resulting state
changes. Therefore, the manager is learning a policy more closely aligned to
the single agent version of this topic. We could conceivably extend our
approach in the future to include more agents, but the use of single agent
control seems more immediately relevant and far less explored.

###  3.3 Delegation

Previous work in delegation, such as [6, 7, 12, 17, 22], demonstrate relevant
concepts and approaches. However, it is worth noting that all of these are
assuming the traditional use of single MDP models for all agents. This means
that generally the actions, transitions, etc. are consistent for the different
agents. In other words, the decision to change delegation comes down strictly
to a measure of performance/cost per time step within a common model. This
means the variation will come from a difference of agent cost, policy skill
levels, observation abilities, etc. In this case, the delegation is occurring
strictly between agents operating in entirely equivalent situations but with
variations in agent performance. Again, this means that the current common
methods focus on a measure of cost or agent performance in determining the
best choice.

##  4 Agents and Manager Models

###  4.1 Team Agent Models

To demonstrate our model and assumptions, we will describe key aspects from
the perspective of a team of two underlying agents. Extending our assumption
of operating in an MDP, we assume the agents are not required to be operating
in the same MDP but rather in compatible MDPs (see Section 4.2). We assume
agents exist in the same state space, but our approach allows for differences
in actions, transitions, and rewards. As demonstrated in Figure 1, two agents
could exist which operate in the same state space, but transition differently
between states. In Figure 1, these transitions are represented by green arrows
for the first agent and black arrows for the second. Both agents can reach the
same states (i.e., equivalent state reachability), but the trajectories would
vary depending on which agent you observe. This ensures the agents are never
given acting authority in a state which does not exist in their MDP, or in a
state which could not be reached in their learning phase. Further, this
ensures that mixed trajectories utilizing decisions from one or more of the
agents remain valid.

Figure 1: Sample MDPs based on two transition functions (Agent #1: black
arrows, Agent #2: green arrows)

The key distinction in our MDP assumptions comes from our allowance of
mismatched actions, transitions, and rewards. In this paradigm, the remaining
key characteristic is ensuring the transitions maintain the
reachability/compatibility assumption. In other words, following the example
in Figure 1, the transitions of each agent only need to ensure that a state
reachable by one agent can also be reached by the other. For example, the
black arrows allow moving from $1\rightarrow 2$, but the agent following the
green arrows would need an additional transition to reach the same state
(e.g., $1\rightarrow 4\rightarrow 2$). With the defined requirements on the
transitions, we ensure agents are not restricted when learning policies. This
ensures we do not restrict the supported agents in our framework, which
enables more generalized support for team compositions.

###  4.2 Manager MDP & Model

Following the previous assumption of two agents, we assume there exist two
compatible MDPs $M_{d_{1}}=\langle S_{1},A_{1},R_{1},T_{1},\gamma_{1}\rangle$
and $M_{d_{2}}=\langle S_{2},A_{2},R_{2},T_{2},\gamma_{2}\rangle$ where
$S_{1}=S_{2}$. Further, assume $A_{1}\neq A_{2}\textrm{ and }T_{1}\neq T_{2}$.
Finally, letting $S_{M}=S_{1}=S_{2}$, our reachability assumption noted in
Section 4.1 requires that a state $s^{\prime}\in S$ reachable by agent
$\mathop{d}_{1}$ from some state $s_{1}\in S$ must be reachable from some
other state $s_{2}\in S$ for agent $\mathop{d}_{2}$. Again, this does not
require that $s_{1}=s_{2}$, but we do require that each agent can form
trajectories that start and terminate in the same states. With these
assumptions, it is clear a manager $m$ delegating between agents
$\mathop{d}_{1}$ and $\mathop{d}_{2}$ for $M_{d_{1}}$ and $M_{d_{2}}$,
respectively, can treat the team as navigating a combined Managed MDP (MMDP)
$M$. We will define MMDP $M=\langle D,S,A_{M},R_{M},T_{M},\gamma_{M}\rangle$
with:

  * •

$D=\\{\mathop{d}_{1},\mathop{d}_{2}\\}$, the set of agents available for
delegation

  * •

$A_{M}=\bigcup_{d\in D}\\{(a,d):a\in A_{d}\\}$, the action space for MDP
$M_{i},i\in D$

  * •

$R_{M}(s,d,s^{\prime})$, the manager’s reward function

  * •

$T_{M}$, the manager’s transition function

With our definition of the MMDP, we can define our relevant manager-specific
RL model. As our scenario relates to the options framework from the SMDP
model, our manager model is similarly inspired. First, we will define the
manager transition, which will utilize the underlying dynamics of the agents
in the team:

$T_{M}(s^{\prime}|s,d)=\sum_{a\in
A_{d}}\pi_{d}(a|s)T_{d}\left(s^{\prime}|s,a\right)$ (5)

As is indicated, the likelihood of transitions relies directly on the agent
policy $\pi_{d}$ and transitions $T_{d}$ for the delegated agent. This allows
us to model the single action transitions the manager could encounter given a
team. With our transition model, we can then define a value function for our
manager given a manager policy $\pi_{m}$:

$\displaystyle V^{\pi_{m}}(s)$
$\displaystyle=\mathbb{E}_{d\sim\pi_{m}}\left[\sum^{\infty}_{i=0}\gamma^{i}r_{t+i}|s_{t}=s,\pi_{d}\right]$
(6) $\displaystyle=\mathbb{E}_{d\sim\pi_{m}}\left[r_{t}+\gamma
V^{\pi_{m}}(s_{t+1})|s_{t}=s,\pi_{d}\right]$ $\displaystyle=\sum_{d\in
D}\pi_{m}(d|s)\sum_{a\in A_{d}}\pi_{d}(a|s)\sum_{s^{\prime}\in
S}T_{d}(s^{\prime}|s,a)\left[R_{M}(s,d,s^{\prime})+\gamma
V^{\pi_{m}}(s^{\prime})\right]$

Again, the key changes over a standard RL model are the need to account for
the policy and transitions of the delegated agents. This gives the definition
of the optimal value function $V^{*}$ as

$V^{*}(s)=\max_{d}\sum_{a\in A_{d}}\sum_{s^{\prime}\in
S}\pi_{d}(a|s)T_{d}(s^{\prime}|s,a)\left[R_{M}(s,d,s^{\prime})+\gamma
V^{*}(s^{\prime})\right]$ (7)

with the corresponding state-action value function $Q^{*}$

$Q^{*}(s,d)=\sum_{a\in A_{d}}\sum_{s^{\prime}\in
S}\pi_{d}(a|s)T_{d}(s^{\prime}|s,a)\left[R_{M}(s,d,s^{\prime})+\gamma
V^{*}(s^{\prime})\right]$ (8)

It can be shown that the above definitions ensure the model conforms to the
requirements for convergence in Reinforcement Learning (see Appendix I).
Therefore, we can use standard RL techniques to estimate the optimal manager
policy $\pi_{m}$, i.e., we can optimize the choice of the delegation agent $d$
that maximizes Equation 6 to maximize the expected reward achieved by the
manager (which clearly means optimizing the behavior of the entire system).

##  5 Application in Case of Single Action Delegations

To demonstrate our approach, we will test the manager model in a gridworld
scenario (see Figure 2). This allows us to test our approach in a scenario
which removes possible ambiguous characteristics while still being non-trivial
and used in recent literature (e.g., [8, 18]). In a gridworld environment,
agents commonly transition through single step actions
$a\in\\{\uparrow,\rightarrow,\downarrow,\leftarrow\\}$. Agents are then
trained to accomplish a task, such as safe navigation from a start state to a
goal state.

The gridworld also ensures compatible states since agent distinction comes
from specific action spaces and transitions for the different agent types.
This is done via action spaces $A_{i}\subset
A=\times_{k}\\{\uparrow,\rightarrow,\downarrow,\leftarrow\\}$, for agent step
size $k$ where we assume that $k$ is a constant parameter, which may be
different for each managed agent. In our scenario, the actions for an agent
are assumed to take a single time step and are comprised of one or more atomic
actions. Therefore, different agents can navigate an environment at different
rates, and the manager observes that, when it delegates to a certain agent,
that agent will operate for $k$ single step actions in a time step without
possibilities for further interventions.

Figure 2: Gridworld environment for training and testing of agents and teams.

Note, the level of restriction of the action space will determine how likely
an agent is to encounter a wall collision. For instance, if we only allow a
3-step agent to move in repeated sequences of atomic actions (e.g.,
$\rightarrow\rightarrow\rightarrow$), then gridworlds with numerous hallways
and turns will increase the frequency of collisions. It is essential the
manager identifies where these differences matter and how best to utilize
them.

The actions for our $k$-step agents are demonstrated in Figure 3. In the
figure, each green arrow indicates an action available to the agent starting
from the center point, and the direction/termination of the arrow indicates
the movement completed in an unobstructed scenario. Specifically, starting
from the center, the agent can terminate in any of the squares containing the
end of an arrow, respectively after a 1-, 2- or 3-step action. If the agent
encounters a wall along the arrow’s path, the action terminates in the cell
the agent reached along the arrow’s path prior to the collision.

a

b

c

Figure 3: Agent actions spaces by step size. Arrows indicate actions starting
from center cell. (a) 1-Step agent, (b) 2-Step agent, (c) 3-Step agent with
sample action destinations X, Y, and Z.

Generally, we can treat any goal state as a trap state with value zero. This
will ensure the probability of a trajectory which moves beyond the goal state
is zero and that the value of any trajectory with additional steps in the goal
state will not accumulate further value. Additionally, a trajectory of step-
size $k$ which reaches a goal state after $\hat{k}<k$ atomic actions can be
treated as equivalent to any longer sequence with the same sub-sequence. This
is due to the probability of all transitions while at the goal state being
zero aside from those which loop to stay in the same state with probability
equal to one. Hence, early termination of a delegation should have no impact
on the manager’s transition and value function.

With the above framework, we update our model to include a representation of
the $k$-step actions the agents will utilize. This ensures the manager will be
observing accurate transitions, which will further ensure the transitions and
value function are accurate with respect to the current team. First, we derive
the following manager transition function:

$T_{M}(s^{\prime}|s,d)=\sum_{a_{d}(t),s_{d}(t)}\prod^{t+k-1}_{i=t}\pi_{d}(a_{i}|s_{i})T_{d}(s_{i+1}|s_{i},a_{i})\mathds{1}\left[s_{t+k}=s^{\prime}\right]$
(9)

where $a_{d}(t)=\\{a_{t},\dots,a_{t+k-1}\\}$ and
$s_{d}(t)=\\{s_{t+1},\dots,s_{t+k}|s_{t}=s\\}$ represent the agent’s sequence
of atomic actions and states during their delegation window. Note that the
transition function depends only on the starting and ending states
respectively ($s$ and $s^{\prime}$) and not on the intermediate states, as the
latter are not under the control of the manager. This transition function is
derived starting from the general formulation of $T_{M}$ provided in Section
4, but we have extended the probability to include the multi-step nature of
the agent actions. This ensures the transitions of the manager still reflect
the underlying mechanics of the agent MDPs.

Given a definition of manager transitions, we can derive a manager value
function. Remember that we assume that the manager has no access to the
internal mechanisms through which the managed agents learn their own policies.
Therefore, in this context, we will assume that the manager observes no reward
derived from the subsequent states and actions of the delegated agents.
Therefore, the manager’s value function will treat each estimate as
independent of these underlying dynamics as well. However, we have included
the use of a discount parameter based on the delegated agent’s step size. This
is inspired by the options framework in SMDPs. In our case, the intermediate
rewards that would be discounted in the options framework are unobserved
(i.e., are zero) by the manager, so only the $\gamma^{k_{d}}$ discount is
applied to the observed value. As such, the value function will follow a
similar pattern to Equation 6. In this case, the delegation agent specific
actions and transitions are encompassed by $T_{M}$, resulting in value
function.

$V(s)=\sum_{d\in
D}\pi_{M}(d|s)\sum_{s^{\prime}}T_{M}(s^{\prime}|s,d)[R_{M}(s,d)+\gamma^{k_{d}}V(s^{\prime})]$
(10)

where $k_{d}$ is the step-size of the delegated agent $d$, while
$[R_{M}(s,d)+\gamma^{k_{d}}V(s^{\prime})]$ denotes the expected rewards of the
manager when choosing delegation agent $d$ with expected transition in state
$s^{\prime}$. The use of $\gamma^{k_{d}}$ aligns with the SMDP framework and
enables indicating a discount dependent on the number of atomic actions a
delegated agent will perform.

For the reward function, we will use a simple reward which does not rely on
any knowledge specific to the underlying agents. For our manager, we will
provide rewards using

$R=\begin{cases}100-c_{d}\quad&\,\text{goal is reached}\\\
-1-c_{d}\quad&\,\text{goal not reached, but valid step}\\\
-10-c_{d}\quad&\,\text{delegated agent caused wall collision}\\\
-100-c_{d}\quad&\,\text{episode terminated without reaching goal}\\\
\end{cases}$ (11)

where $c_{d}$ denotes the costs of agent $d$. In other words, we penalize the
manager for agent costs, and for episode length with the step penalty in the
reward function. Note that the specific values of the reward are not that
important. What is key is to have a form of reward that can help drive the
manager in the desired direction. With the given reward, the manager should
learn to minimize episode lengths while accounting for agent costs. In our
paradigm, cost can represent relative preference, operating costs, etc. We
should note that this simpler reward does not specifically prevent the manager
from allowing a wall collision if the overall reward is higher. In this
scenario, we wanted to demonstrate the manager’s ability to find a team which
can reach the goal fastest and lowest cost, but with the option to accept a
collision if it means higher overall reward. Regarding the manager reward,
safety-critical cases could be considered. In this case, wall collisions could
terminate an episode with a penalty. Alternatively, the penalty for collisions
could be increased to further discourage these outcomes.

##  6 Results

In this section, we will demonstrate the performance of our manager in
multiple team configurations. Further, our results will include cases where
the manager delegations are subject to agent-specific costs. With the
inclusion of cost, the manager must make delegations which both match the
desire for good team performance while also accounting for agent cost.

The x-axis labels correspond to the various team compositions. Our label
convention is "AB-CD", where A/C denote step size and B/D denote error
likelihood – L: low, M: medium, H: high, N: none. For example, 1H-2L
corresponds to a team with a 1-step high error likelihood agent and 2-step low
error likelihood agent. Likelihood indicates how frequently an agent makes an
error and likely severity of change. The actions selected can change with our
error mapping via a decaying exponential function (truncated in $[0,\pi]$),
which determines how large a change is made. An error occurs when the action
performed differs from the originally intended action. The severity of the
error indicates how much of a divergence is observed between the error and the
intended action.

Following Figure 3, each $k$-step action corresponds to an arrow leading from
the center cell to another cell. Errors occur when an agent follows a
different arrow than intended. As seen in Figure 3b and Figure 3c, different
actions/arrows can lead to the same state but different outcomes. For
instance, the costs for actions could vary or walls could exist along one
arrow’s path and not another. In these cases, the rewards or trajectories
could vary significantly. As an example of errors, starting from
$\uparrow\uparrow\uparrow$ leading to Y, $\uparrow\uparrow\rightarrow$ is a
clockwise shift of one arrow/action leading to X and
$\leftarrow\leftarrow\downarrow$ is an anticlockwise shift by eight
actions/arrows leading to Z (Note: $\downarrow\downarrow\leftarrow$ is a nine-
error shift also leading to Z). Errors are determined by sampling according to
our function and binning to determine how many arrows away to select the error
action, with clockwise/anticlockwise chosen randomly.

Regarding team compositions, we consider cases of agents with step sizes of 1
to 3. This was intended to represent agents with clear distinction in action
spaces while maintaining a reasonable level of “speed” for the agents (i.e.,
limiting the advantages of any one agent). Further, the step sizes enable
teams with agents demonstrating strong performance in particular grid
regions/types, but they also allow for variation between agents. This ensures
the manager will have opportunities to operate under multiple team
configurations and varying ideal utilization.

###  6.1 Low Cost for Shorter Steps

For the following case, the manager was given a cost of 1, 4, and 7 for the
1-step, 2-step, and 3-step agents, respectively. With this, the manager is
given a reward function which counteracts some benefits of choosing agents
with higher step sizes. This impact is in conjunction with the fact that the
manager reward allows for some wall collisions with penalty. Therefore, we see
an increasing per-step cost as the step size increases (e.g., cost of 6 for
six 1-step actions vs. a cost of 14 for two 3-step actions). In the following
plots, we illustrate the rewards observed for both trained and random
managers, i.e., managers that delegate between the two agents according to a
uniform probability of 0.5 at any decision point.

a Low-Cost: 1-Step/2-Step team

b High-Cost: 1-Step/2-Step team

c Low-Cost: 1-Step/3-Step team

d High-Cost: 1-Step/3-Step team

e Low-Cost: 2-Step/3-Step team

f High-Cost: 2-Step/3-Step team

Figure 4: Manager performance compared to random selection. Agents subject to
varying levels of error likelihood – L: Low, M: Medium, H: High, N: None

As is apparent in our results, cases with the least error-prone agents offer
the manager the best chance of success. For example, the 1N-2N (Figure 4a),
1L-3N (Figure 4c), and 2N-3L (Figure 4e) cases all demonstrate the manager
utilizing the low error rates of the agents, resulting in higher overall
rewards. Further, the lower error rates allow the manager to take advantage of
the higher step sizes to maintain strong team performance. For cases where the
manager encounters one or more team members with high error likelihoods, we
continue to see cases of strong performance. As seen for 1H-2L (Figure 4a),
1H-3N (Figure 4c), and 2L-3H (Figure 4e), our manager finds agent utilization
achieving high scores despite the high variance in agent performances.

###  6.2 Higher Cost for Shorter Steps

In a cost reversal, we demonstrate the case where the manager now observes a
much higher cost for the short step agents and reduced for the longer ones.
The manager is given agent costs of 7, 4, and 1 for the 1-step, 2-step, and
3-step agents, respectively. While this change should not impact the manager’s
ability to select better performing agents, we do anticipate seeing
modifications to the relative performance and overall manager results (i.e.,
maximum reward and agent utilization).

We again see our manager learns desirable agent utilization. Clearly, the
1L-2N (Figure 4b), 1L-3N (Figure 4d), and 2L-3N (Figure 4f) cases all
demonstrate the manager utilizing low error agents while accounting for their
corresponding costs to find optimal results. In the cases of one or more team
members with high error likelihoods, we continue to see cases of strong
performance. As seen in the 1H-2L (Figure 4b), 1H-3N (Figure 4d), and 2M-3M
(Figure 4f) cases, our manager again finds agent utilization achieving high
scores despite the high variance in agent performances.

##  7 Conclusion

In Figure 4, we see the manager has consistent and stronger performance with
less error-prone agents for both agent cost scenarios. In both cases, as
expected, increased variance of agent performance coincides with increased
variance in manager rewards. Still, given the performance indicated by our
method, we see that our manager was successful in learning a delegation
policy. Despite the manager operating without direct observation of agent
actions, the manager learned identify desirable delegations to generate good
trajectories. Consequently, we demonstrated a method which could train a
manager to successfully delegate between agents without assuming unfair access
to additional knowledge. Further, we demonstrated the manager can accommodate
for changes in agent costs to learn a delegation policy.

#### Acknowledgments

This work was partially supported by the CHIST-ERA-19-XAI010 SAI project. M.
Conti’s and A. Passarella’s work was partly funded by the PNRR - M4C2 -
Investimento 1.3, Partenariato Esteso PE00000013 - "FAIR" funded by the
European Commission under the NextGeneration EU programme.

## References

  * [1] Fatality and injury reporting system tool (first) (Nov 2022), https://cdan.dot.gov/query
  * [2] Bradtke, S., Duff, M.: Reinforcement learning methods for continuous-time markov decision problems. Advances in neural information processing systems 7 (1994)
  * [3] Chakravorty, J., Ward, N., Roy, J., Chevalier-Boisvert, M., Basu, S., Lupu, A., Precup, D.: Option-critic in cooperative multi-agent systems. arXiv preprint arXiv:1911.12825 (2019)
  * [4] Chen, J., Haliem, M., Lan, T., Aggarwal, V.: Multi-agent deep covering option discovery. arXiv preprint arXiv:2210.03269 (2022)
  * [5] Erskine, J., Lehnert, C.: Developing cooperative policies for multi-stage reinforcement learning tasks. IEEE Robotics and Automation Letters 7(3), 6590–6597 (2022)
  * [6] Fuchs, A., Passarella, A., Conti, M.: A cognitive framework for delegation between error-prone ai and human agents. In: 2022 IEEE International Conference on Smart Computing (SMARTCOMP). pp. 317–322 (2022). https://doi.org/10.1109/SMARTCOMP55677.2022.00074
  * [7] Fuchs, A., Passarella, A., Conti, M.: Compensating for sensing failures via delegation in human–ai hybrid systems. Sensors 23(7) (2023). https://doi.org/10.3390/s23073409, https://www.mdpi.com/1424-8220/23/7/3409
  * [8] Gabor, T., Sedlmeier, A., Kiermeier, M., Phan, T., Henrich, M., Pichlmair, M., Kempter, B., Klein, C., Sauer, H., AG, R.S., et al.: Scenario co-evolution for reinforcement learning on a grid world smart factory domain. In: Proceedings of the Genetic and Evolutionary Computation Conference. pp. 898–906 (2019)
  * [9] Garcia, F.M., Nota, C., Thomas, P.S.: Learning reusable options for multi-task reinforcement learning. arXiv preprint arXiv:2001.01577 (2020)
  * [10] Guan, Y., Afshari, M., Zhang, Q., Tsiotras, P.: Hierarchical decompositions of stochastic pursuit-evasion games. In: 2022 IEEE 61st Conference on Decision and Control (CDC). pp. 5062–5067. IEEE (2022)
  * [11] Jaakkola, T., Jordan, M., Singh, S.: Convergence of stochastic iterative dynamic programming algorithms. Advances in neural information processing systems 6 (1993)
  * [12] Jacq, A., Ferret, J., Pietquin, O., Geist, M.: Lazy-mdps: Towards interpretable rl by learning when to act. In: Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. pp. 669–677 (2022)
  * [13] Kurzer, K., Zhou, C., Zöllner, J.M.: Decentralized cooperative planning for automated vehicles with hierarchical monte carlo tree search. In: 2018 IEEE intelligent vehicles symposium (IV). pp. 529–536. IEEE (2018)
  * [14] Lau, Q.P., Lee, M.L., Hsu, W.: Coordination guided reinforcement learning. In: AAMAS. pp. 215–222 (2012)
  * [15] Melo, F.S.: Convergence of q-learning: A simple proof. Institute Of Systems and Robotics, Tech. Rep pp. 1–4 (2001)
  * [16] Menda, K., Chen, Y.C., Grana, J., Bono, J.W., Tracey, B.D., Kochenderfer, M.J., Wolpert, D.: Deep reinforcement learning for event-driven multi-agent decision processes. IEEE Transactions on Intelligent Transportation Systems 20(4), 1259–1268 (2018)
  * [17] Meresht, V.B., De, A., Singla, A., Gomez-Rodriguez, M.: Learning to switch between machines and humans. CoRR abs/2002.04258 (2020), https://arxiv.org/abs/2002.04258
  * [18] Moy, G., Shekh, S.: Evolution strategies for sparse reward gridworld environments. In: AI 2022: Advances in Artificial Intelligence: 35th Australasian Joint Conference, AI 2022, Perth, WA, Australia, December 5–8, 2022, Proceedings. pp. 266–278. Springer (2022)
  * [19] Rahman, Q.M., Sünderhauf, N., Dayoub, F.: Did you miss the sign? a false negative alarm system for traffic sign detectors. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 3748–3753. IEEE (2019)
  * [20] Rohanimanesh, K., Mahadevan, S.: Learning to take concurrent actions. Advances in neural information processing systems 15 (2002)
  * [21] Singh, A.J., Kumar, A., Lau, H.C.: Hierarchical multiagent reinforcement learning for maritime traffic management (2020)
  * [22] Straitouri, E., Singla, A., Meresht, V.B., Gomez-Rodriguez, M.: Reinforcement learning under algorithmic triage. CoRR abs/2109.11328 (2021), https://arxiv.org/abs/2109.11328
  * [23] Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. MIT press (2018)
  * [24] Yang, M., Zhao, J., Hu, X., Zhou, W., Li, H.: Ldsa: Learning dynamic subtask assignment in cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2205.02561 (2022)

##  8 Appendices

##  Appendix I Convergence of Q-Learning in Mismatched MDPs

Following the proof of convergence for Q-Learning on MDPs in [11, 15], we will
show that the learning algorithm converges to the optimal $Q^{*}$.

_Proof._ As with traditional Q-Learning, we will start by showing that the
optimal $Q$ is a fixed point of a contraction operator $\mathbf{H}$. First,
recall that we are considering MDPs $M_{d_{1}}=\langle
S_{1},A_{1},R_{1},T_{1},\gamma_{1}\rangle$ and $M_{d_{2}}=\langle
S_{2},A_{2},R_{2},T_{2},\gamma_{2}\rangle$, with $S_{1}=S_{2}$. Therefore,
define

$(\mathbf{H}q)(x,d)=\sum_{s^{\prime}\in S}\sum_{a\in
A_{d}}\pi_{d}(a|s)T_{d}(s^{\prime}|s,a)\left[R_{M}(s,d,s^{\prime})+\gamma\max_{d^{\prime}}q_{(}s^{\prime},d^{\prime})\right]$
(12)

We show $\mathbf{H}$ is a contraction in the sup-norm by showing that
$\left\lVert\mathbf{H}q_{1}-\mathbf{H}q_{2}\right\rVert_{\infty}\leq\gamma\left\lVert
q_{1}-q_{2}\right\rVert_{\infty}$. From the definition of $\mathbf{H}$, we see
that

$\displaystyle\left\lVert\mathbf{H}q_{1}-\mathbf{H}q_{2}\right\rVert_{\infty}$
$\displaystyle=\max_{s,d}\left|\sum_{s^{\prime}\in S}\sum_{a\in
A_{d}}\pi_{d}(a|s)T_{d}(s^{\prime}|s,a)\left[R_{M}(s,d,s^{\prime})+\gamma\max_{d^{\prime}}q_{1}(s^{\prime},d^{\prime})\right.\right.$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\quad\left.\left.-\,R_{M}(s,d,s^{\prime})-\gamma\max_{d^{\prime}}q_{2}(s^{\prime},d^{\prime})\right]\right|$
$\displaystyle=\max_{s,d}\gamma\left|\sum_{s^{\prime}\in S}\sum_{a\in
A_{d}}\pi_{d}(a|s)T_{d}(s^{\prime}|s,a)\left[\max_{d^{\prime}}q_{1}(s^{\prime},d^{\prime})-\max_{d^{\prime}}q_{2}(s^{\prime},d^{\prime})\right]\right|$
$\displaystyle\leq\max_{s,d}\gamma\sum_{s^{\prime}\in S}\sum_{a\in
A_{d}}\pi_{d}(a|s)T_{d}(s^{\prime}|s,a)\left|\max_{d^{\prime}}q_{1}(s^{\prime},d^{\prime})-\max_{d^{\prime}}q_{2}(s^{\prime},d^{\prime})\right|$
$\displaystyle\leq\max_{s,d}\gamma\sum_{s^{\prime}\in S}\sum_{a\in
A_{d}}\pi_{d}(a|s)T_{d}(s^{\prime}|s,a)\max_{d^{\prime}}\left|q_{1}(s^{\prime},d^{\prime})-q_{2}(s^{\prime},d^{\prime})\right|$
$\displaystyle=\max_{s,d}\gamma\sum_{s^{\prime}\in S}\sum_{a\in
A_{d}}\pi_{d}(a|s)T_{d}(s^{\prime}|s,a)\left\lVert
q_{1}-q_{2}\right\rVert_{\infty}$ $\displaystyle=\gamma\left\lVert
q_{1}-q_{2}\right\rVert_{\infty}$

Given $\mathbf{H}$ is a contraction, we will then show that $Q_{t}$ converges
to $Q^{*}$ as $t\rightarrow\infty$. To start, we will assume that for $\forall
t$, we have $0\leq\alpha_{t}\leq
1,\sum_{t}\alpha_{t}=\infty,\mathrm{and}\sum_{t}\alpha_{t}<\infty$. Let

$F_{t}(s,d)=R_{M}(s,d,X(s,d))+\gamma\max_{d^{\prime}}Q_{t}(X(s,d),d^{\prime})-Q^{*}(s,d)$

for next state distribution $X(s,d)$. Then, we will first show that
$\left\lVert\mathbb{E}\left[F_{t}(s,d)|\mathcal{F}_{t}\right]\right\rVert_{\infty}\leq\gamma\left\lVert\Delta_{t}\right\rVert_{\infty}$,
with $\gamma<1$. Let

$\Delta_{t}=Q_{t}(s,d)-Q^{*}(s,d)$

then

$\Delta_{t+1}=(1-\alpha_{t})\Delta_{t}+\alpha_{t}F_{t}(s,d)$

Therefore,

$\displaystyle\mathbb{E}\left[F_{t}(s,d)|\mathcal{F}_{t}\right]$
$\displaystyle=\sum_{s^{\prime}\in S}\sum_{a\in
A_{d}}\pi_{d}(a|s)T_{d}(s^{\prime}|s,a)\left[R_{M}(s,d,s^{\prime})+\gamma\max_{d^{\prime}}Q_{t}(s^{\prime},d^{\prime})-Q*(s,d)\right]$
$\displaystyle=(\mathbf{H}Q_{t})(s,d)-Q^{*}(s,d)$
$\displaystyle=(\mathbf{H}Q_{t})(s,d)-(\mathbf{H}Q^{*})(s,d)$
$\displaystyle\overset{(a)}{\Rightarrow}\left\lVert\mathbb{E}\left[F_{t}(s,d)|\mathcal{F}_{t}\right]\right\rVert_{\infty}\leq\gamma\left\lVert
Q_{t}-Q^{*}\right\rVert_{\infty}$
$\displaystyle=\gamma\left\lVert\Delta_{t}\right\rVert_{\infty}$

with $(a)$ resulting from the contraction proof. Next, we will show that
$\mathrm{\textbf{var}}\left[F_{t}(s)|\mathcal{F}_{t}\right]\leq
C(1+\left\lVert\Delta_{t}\right\rVert_{\infty})^{2}$ for $C>0$.

$\displaystyle\mathrm{\textbf{var}}\left[F_{t}(s)|\mathcal{F}_{t}\right]$
$\displaystyle=\mathbb{E}\left[\left(R_{M}(s,d,X(s,d))+\gamma\max_{d^{\prime}}Q_{t}(s^{\prime},d^{\prime})-Q^{*}(s,d)-(\mathbf{H}Q_{t})(s,d)+Q^{*}(s,d)\right)^{2}\right]$
$\displaystyle=\mathbb{E}\left[\left(R_{M}(s,d,X(s,d))+\gamma\max_{d^{\prime}}Q_{t}(s^{\prime},d^{\prime})-(\mathbf{H}Q_{t})(s,d)\right)^{2}\right]$
$\displaystyle\overset{(a)}{=}\mathrm{\textbf{var}}\left[R_{M}(s,d,X(s,d))+\gamma\max_{d^{\prime}}Q_{t}(s^{\prime},d^{\prime})|\mathcal{F}_{t}\right]$
$\displaystyle\leq C(1+\left\lVert\Delta_{t}\right\rVert_{\infty})^{2}$

with the final inequality resulting from [15]. Further, note that $(a)$ comes
from the fact that $(\mathbf{H}Q_{t})(s,d)=\mathbb{E}\left[Q_{t}(s,d)\right]$.
Therefore, we see that $Q^{t}$ converges to $Q^{*}$ with probability $1$.

$\blacksquare$


# ARPES in strongly disordered systems – theory of electronic band’s melting

Piotr Chudzinski  School of Mathematics and Physics, Queen’s University of
Belfast  Institute of Fundamental Technological Research, Polish Academy of
Sciences, Pawinskiego 5B, 02-105 Warsaw, Poland  Lenart Dudy  TEMPO Beamline,
Synchrotron SOLEIL, L’Orme des Merisiers Saint-Aubin, B.P.48, 91192 Gif-sur-
Yvette, France  Physikalisches Institut and Würzburg-Dresden Cluster of
Excellence ct.qmat, Universität Würzburg, D-97074 Würzburg, Germany

###### Abstract

It is well known that translational symmetry-breaking disorder will disrupt
ARPES spectra up to the point where they become invisible. However, a
theoretical framework to capture this phenomenon has been largely missing.
Here, based on a rigorous theory of the ARPES process, we provide this much-
needed framework. In particular, we show how the frequently used sudden
electron approximation has to be modified in this situation. Our main result
is an argument that links the photoemission line broadening with an operator
content of a disorder operator and so with the criticality of the
corresponding order-disorder phase transition. For concreteness, we focus here
on the frustrated 2D trigonal (pseudo-)spin model, with Ising order-disorder
operators behind the transition. Still, our formalism is general and can be
applied in a much broader context.

##  I Introduction

Angle-Resolved Photoemission Spectroscopy (ARPES) is one of the most important
methods to study the electronic properties of materials. Its ability to probe
at the same time the energy and momentum of electrons has granted the method a
primary role in contemporary experimental solid-state physics. Indeed, over
the last few decades, it has led us to some of the most important discoveries
in the field [1, 2, 3, 4, 5, 6, 7, 8]. The method is defined in reciprocal
momentum space. Although it can cope with broken translational invariance in
the direction perpendicular to the surface, its ability to deliver information
diminishes when in-plane translational invariance is severely broken, for
instance, in disordered systems. This naturally leads to a question: can ARPES
be useful at all in the case of a strongly disordered system? This is a very
pertinent question, especially considering recent developments in the fields
such as many-body localization (MBL)[9, 10] or dirty d-wave
superconductors[11]. The MBL plays a particularly important role in
contemporary solid-state physics. Recently given exact proof of its
existence[12, 13] inspired many numerical and experimental papers including
experimental studies that showed both its realization[14, 15] and the slowdown
of electron propagation in its vicinity[16]. The disordered systems thus
entered the forefront of experimental materials research. In this work, we
would like to provide the theoretical framework to describe an outcome of
ARPES experiment in the situation where the band structure is gradually
diminished with the increasingly strong disorder.

For simplicity, we will consider a situation of two coexisting phases on a
surface of a material. In one of these phases, the carriers can easily
propagate. This is our dominant “parent” state, possibly the system’s ground
state. The second phase provides us with a strong disorder. Within this phase,
the mobile carriers are trapped. The two phases are nearly orthogonal, i.e.,
the tunneling between them is the slowest process during carriers propagation
through the sample. Moreover, this implies that the growth/recombination of
one phase into another is also prolonged. One should note that this is a
realization of a strong disorder, contrary to weak scattering on impurities
which may be tackled employing an additional self-energy[17] e.g., in Born
approximation. Here the large disorder strength invalidates these perturbative
approximations. Since the two phases are drastically different and
impenetrable one also cannot use any method of effective averaging medium[18,
19]. The aim of this work is to derive formalism that will capture diminishing
coherent band amplitude as the disorder strength increases.

The outline of this work is as follows. First, in Sec. II we introduce a
specific model where our phenomenology can be realized. In Sec. III, we
critically analyze the canonical theory of ARPES to extract elements that need
to be re-defined. In Sec. IV, we derive formulas for ARPES spectra in the
presence of disorder. Finally, in Sec. V, we provide a broader context of our
study and give examples of other possible realizations.

##  II Model

Figure 1: Possible realization of our model for strong disorder. At low
temperature there is a ”clean” system which host a well defined bands –
dispersive energy states of pseudo-spins. At higher temperature (bottom panel)
the coherence along $a$-axis is gradually lost and 1D regions that appears are
able to trap photo-holes impeding their immediate propagation after the photo-
emission event.

###  II.1 The idea of dimensional cross-over

As mentioned in the introduction the idea is to introduce some external
control parameter, that will govern the strength of the disorder, such that
one can gradually ramp it up. The most natural candidates would be either
surface irradiation or mechanically induced plasticity. The issue is whether,
in an arbitrarily chosen material, the disorder induced by these two tools
fulfills other criteria mentioned in the introduction. We then decided to
focus on a dimensional cross-over between two insulating phases.

To convince the reader that the peculiar situation of two impenetrable,
topologically incompatible phases, can be realized, we shall focus on a
strongly anisotropic 2D triangular anti-ferromagnet. In this system, there are
two possible phases[20]: 1) the 2D phase, later called the uud phase, where
natural excitation are vertexes on each triangle[21], and 2) the quasi-1D
phase where natural excitations are solitons[22]. Since the natural
excitations are topologically incompatible, it is impossible that one phase
will easily recombine into another. Moreover, from the theory of dimensional
crossover, we know that, when perpendicular coupling $V_{\perp}$ depends on an
internal degree of freedom, by increasing temperature we should reduce the
perpendicular coherence and increase the amount of the secondary 1D phase.
Hence, by creating a thermal mixed state, one increases the amount of disorder
in the system. We see that this offers one practical platform to realize our
theoretical model, and so, for concreteness, we shall stick to it in this
work.

###  II.2 The Hamiltonian

ARPES measures the propagation of fermions and so we need a model where a full
fermion is present. We consider a 2D surface described by Hubbard-type $U-V$
model on a 2D triangular lattice:

${{\cal
H}_{eff}}=\sum\limits_{k\in\textrm{1${}^{st.}$BZ}}\widetilde{\varepsilon\left(k\right)}\widetilde{c_{k}^{+}}\widetilde{c_{k}}+\sum\limits_{i}U{n_{m,i}}{n_{m,i}}+\sum\limits_{i,j\in\,\textrm{chain}\;m}V_{\parallel}\left(j\right){n_{m,i}}{n_{m,i+j}}+\sum\limits_{i,m}{V_{\bot}}{n_{m,i}}{n_{m\pm
1,i\pm 1/2}}+\ldots$ (1)

Instead of working with a single particle description
$\widetilde{\varepsilon\left(k\right)}$ we now consider a strongly correlated
system dominated by interactions. When $U$ is by far the largest energy scale
in the problem, so we can exclude double occupancies and map the problem onto
a pseudo-spin system where ${V_{\bot}}$ and ${V_{||}}$ take the role of an
anisotropic Ising-interaction. The inter-site hybridization takes the role of
an equally anisotropic Heisenberg term. A self-localized hole’s occupancy (or
lack of it) translates into a pseudo-spin up (or down) state. Therefore, by
taking the limit $U\to\infty$ we arrive at an effective description in terms
of pseudo-spins. The 2D pseudo-spin $XXZ$ model reads:

${\cal
H}_{XXZ}=\sum\limits_{i}{J_{xy}}(S_{i}^{+}S_{i+1}^{-}+h.c.)+\sum\limits_{i,j\in\,\textrm{in
- chain}}J_{z}^{\parallel}S_{z,i}S_{z,i+j}+\sum\limits_{i\in\textrm{in-
chain},n\in\textrm{n.n\;chain}}J_{z}^{\bot}S_{z,i}(n)S_{z,i}(n+1)$ (2)

This 2D $XXZ$ model on the triangular lattice is the simplest model example of
frustrated magnetism and, as such, has been a subject of many, primarily
numerical studies (see Ref. 20 and references therein, also Ref. 23 and Ref.
24) . It is known that for a magnetic field (chemical potential) close to 1/3
the model has two phases depending on the anisotropy. One of them is the 2D
uud phase and another quasi-1D phase that can be distinguished by the presence
of $\times$2$\times$4 solitons. We are interested in a situation where ARPES
probes a mixed state, e.g. thermal state, where both these phases coexist.

The $J_{z}^{\bot}$ or to be more precise ${{\rm{\Delta}}_{1}}\sim
J_{z}^{\bot}$, the energy of screened (effective) perpendicular coupling, sets
the energy scale at which perpendicular coherence is gradually lost. Then
there exists a low-temperature phase where the system is in a purely uud state
and at $T_{1}=\Delta_{1}$ the system is in a mixed state of the two phases.

##  III The theory of the ARPES process

A canonical way of obtaining the photo-electron intensity is to assume that it
is proportional to the single-particle propagator inside the sample and
compute its self-energy from the momentum-conserving interactions. Here we
show that when retardation effects are present in the coupling between
electronic liquid and light, then the spectral characteristics of this
coupling will affect the photo-electron intensity. In this section we shall
first revisit the general ARPES theory to tackle this situation, particularly
the sudden approximation (Sec.III.1). Then, we can give a precise statement of
the disorder problem (Sec.III.2). In the following, we perform calculations
for LT and HT regimes. In the first case (Sec.III.3), the translational
invariance holds, enabling us to link to standard formalism. In the second
case (Sec.IV), we employ our findings to incorporate the effect of N-1
particles on the top of the single-particle propagator.

###  III.1 ARPES theory: the principle of sudden approximation

The general photoemission process can be described by a triangular Feynman-
diagram, see Fig. 2 (upper panel) and compare with e.g. Ref. 25, 26. It
consists of the photoemission corner, the detection corner, and the
recombination corner at the sample exit. There are three creation and three
annihilation operators and the electromagnetic field with vector potential
$\vec{\cal A}\left(r\right)$ acts at the photoemission and recombination
corners. Usually, one splits the six-fermion correlator into three propagators
between these corners requiring altogether three Green functions (symbolized
by the ${\cal G}$’s in Fig. 1) to describe the complete process. In the most
general case (e.g., to account for intrinsic and extrinsic losses), instead of
the empty PES triangle (bottom panel), one has to deal with a filled triangle
where there is a correlation between the propagators; For example, as the
photo-electron leaves the sample, it keeps interacting with the photo-hole
inside the sample. These vertex corrections are symbolized in the upper Fig.1
by the dark blue hexagon. The PES current intensity $I\left({k,\omega}\right)$
is proportional to a quantum superposition of all these processes. Hence, a
large $I\left({k,\omega}\right)$ is equivalent to a constructive quantum
interference established on the entire diagram. In a simplified manner, one
can think of it as Einstein’s “spooky interaction on a distance” between free
carriers going into the detector and a photo-hole left in the sample. Only one
specific photo-hole is selected.

Figure 2: (Upper Panel) Triangular Feynman-diagram in order to explain the
photoemission process. The electron is emitted at
$\left({{r_{1}},t=0}\right)$, then it goes into detector located at $R$ and,
finally at ${r_{2}}$ and $t={t_{d}}+\tau$, it recombines with the photo-hole
that was propagating through the many-body system. The overlap of these
processes contributes to the signal visible at detection. (Lower Panel)
Reduction of the upper diagram according to the sudden approximation. Both
vertices at the detection corner are simplified by spherical harmonics with
well-defined $k$ and $\omega$. Any long-distance electron-electron vertex
corrections are neglected. (adapted from Ref. 25)

To simplify and reduce the description to a single correlation function, one
uses the so-called sudden approximation, see the lower panel of Fig. 1. It is
based on the assumption that the photoemission process is so fast that the
rest of the many-body system is unaltered when exactly one electron is excited
and removed. This is equivalent to neglecting any long-distance electron-
electron vertex corrections at the photoemission corner. If one neglects the
same vertex corrections at the recombination corner, then the time evolution
in the sample decouples; and we arrive at an expression for the photocurrent
that is close to Fermi’s-golden-rule expression:

$I=\int\int
d{r_{1}}d{r_{2}}\;\left\langle{\Psi_{f}^{N-1}(r_{2})}\right|{\vec{\cal
A}(r)}\nabla_{r=r_{2}}\left|{\Psi_{f}^{N-1}(r_{2})}\right\rangle\;\left\langle{\Psi_{i}^{N-1}(r_{1})}\right|{\vec{\cal
A}(r)\nabla_{r={r_{1}}}}\left|{\Psi_{i}^{N-1}(r_{1})}\right\rangle$ (3)

where the wave-functions above are many-body wave functions, and we integrate
over all possible locations of photoemission-recombination events. In the
sudden approximation, one furthermore declares a simple tensor product:

${\rm{\Psi}}_{i}^{N-1}\left({{r_{1}}}\right)=\left|{\psi_{i}(r_{1})}\right\rangle\otimes\left|{N-1}\right\rangle$
(4)

which effectively decouples the propagation of the $N-1$ many-body system from
the photo-electrons propagation. It is important to note that this single-
particle wave-function ${\psi_{i}}\left({{r_{1}}}\right)$ can only be strictly
speaking defined near the space point of the instantaneous event. The
${\psi_{i}}\left({{r_{1}}}\right)$ can be obtained either in the three-step
approximation or (more accurate) in the single-step approximation.

Our description in the sample has been real space. However, we apply a
monochromatic beam of photons given by the vector-potential $\vec{\cal
A}\left({\omega,k}\right)$ with a well-defined angle of impact, with real-
space description, e.g., $\vec{\cal
A}({r_{1}})={\rm{exp}}\left({i{r_{1}}k}\right)\vec{\cal
A}\left({\omega,k}\right)$. The detection counts electrons with a well-defined
$\omega$ in the basis of spherical harmonics. When we further assume no
interactions (no vertex corrections) on the detector and free-electron
propagators out of the sample
${G_{2}}\left({\omega,k}\right)=\;{G_{3}}\left({\omega,k}\right){\rm{\;}}={G_{0}}\left({\omega,k}\right)$
then from the ortho-normality relation of spherical harmonics

$\int dR\,\exp{(-ik^{\prime}R)}\exp{(ikR)}=\delta(k-k^{\prime})$ (5)

it follows that only that electron with the specific, well-defined momentum
$k$ can be selected to give a signal at detection. This also implies that the
final state at the recombination point has to be a simple tensor product

$\left|{\rm{\Psi}}_{f}^{N-1}\left({{r_{2}}}\right)\right\rangle=\left|{\psi_{f}(r_{2};k)}\right\rangle\otimes\left|{N-1}\right\rangle$
(6)

So the final state, after propagating through detection and recombination
corners, has been identified as a simple tensor product of a single particle
in a state ${\psi_{f}}$ with a well-defined momentum k. In a system with where
the translational invariance is obeyed, one can also express the initial
state:

${\psi_{i}}\left({{r_{1}}}\right)=\int dk\;{\psi_{k}}\left\langle
k{\rm{|}}{r_{1}}\right\rangle\quad$ (7)

And since the action of the dipole operators $\vec{\cal A}(r)\nabla_{r_{1}}$
and $\vec{\cal A}\left(r\right)\nabla_{r_{2}}$ is the same at every point of a
translationally invariant system, one arrives at Fermi’s golden rule:

$I=\left|\left\langle{\psi_{f}(k)}\right|\vec{\cal
A}(r)\nabla\left|{\psi_{i}(r)}\right\rangle\right|^{2}$ (8)

This formula is commonly used as a starting point for ARPES analysis. By
substituting Eq.(7) into Eq.(3), we see that only one combination of
phases/amplitudes of events at various points ${r_{1}}$ gives the desired
constructive interference on the entire diagram, the one with a well-defined
momentum k. Hence we conclude that ARPES measures the single-particle spectral
function $Im\left[{{G_{1}}\left({\omega,k}\right)}\right]$ with a well-defined
$k$ and $\omega$.

At this point, it is convenient to think about one selected, distinguishable
particle in the k-mode that enjoys a coherent propagation through the sample,
sometimes[27] called the “blue particle”, as it does not suffer any red-
shifting incoherent losses. An example of the projection procedure on the
plane wave final state is performed in Sec.III.3.

###  III.2 The validity of the sudden approximation in a disordered system

The translational invariance is clearly broken in a strongly disordered
sample, and the reasoning described above, e.g., Eq.(7), does not hold
anymore. Rooted in the general aspects described above, the problems of a
description of ARPES in the disordered medium of uud and $\times$2$\times$4
regions are twofold. The first is to assure the validity of the sudden
approximation as much too frequently[26] all the steps leading to Eq.(8) are
assumed to be straightforward implications of the sudden approximation. The
second one is to find a way to compute ${G_{1}}$ in the situation where the
system’s response to incoming photons depends on the exact place where the
emission event took place. In particular, since the assumption still holds, we
need to find a way to form a constructive interference with the spherical-
harmonically single-electron wave-functions
$\left|{\psi_{f}}\left({{r_{2}};k}\right)\right\rangle$. There are two
equivalent ways of tackling our problem:

(1) We take the basis where the ARPES triangle-diagram in Fig.2 is empty. The
photo-electron is suddenly removed from the sample, and we can safely assume
that it does not interact (no long-distance retardation effects) with a photo-
hole left inside the point ${r_{1}}$ of the sample (so Eq.(4) holds). Photo-
hole acquires non-local self energy, an attempt to compute this quantity has
been given in Ref.28 but in a simpler translationally invariant case. Within
the disordered sample, we need to consider a time evolution of an object that
depends strongly on the position where the photoemission event took place.
While Eq. (3) holds, we do not know how to perform the real-space integral,
and then, Eq. (8) does not hold. The $G_{1}(r_{1},r_{2})$ becomes a real-
space-only object that would be extremely tedious to time-evolve along all
possible paths, all possible realizations of disorder. The analytical progress
of such a theory is hard, and here we will not follow this route. However,
this picture proves that it is possible to construct the so-called “blue
particle” of the sudden approximation, albeit very complicated and dressed up
with a time-evolving disorder.

The way (1) above would be the only way to proceed for a massively disordered
system with no extra information at hand. However, in our case, we know that
in the uud zones, which percolate (or are close to percolation) at least at
the beginning of the HT-phase, the momentum remains a good quantum number.
This brings us to the second way

(2) We can then divide $G_{1}({r_{1}},{r_{2}})$ into two components, an
effective $\widetilde{{G_{1}}}$ inside a uud zone and the rest of the time
evolution in the $\times$2$\times$4 regions. To apply Eq.(7) (valid for the
uud part of evolution), the vertex corrections (that describe a time-evolution
inside $\times$2$\times$4 region, immediately after the photoelectron escape)
have to be located in either the photoemission or the recombination corner of
the triangle PES diagram. As shown below, these corrections capture the
contribution in the photo-electron intensity describing the path a photo-hole
has to endure in the disordered medium of the $\times$2$\times$4 regions.
Hence, we work with a particle that has lost some of its energy to propagate
out of the disordered zone. This slightly red-shifted particle still follows a
coherent time-propagation, and, thereby, it is distinguishable from the
incoherent background[27, 29] \- by analogy, one can call it a “green
particle”.

Adding the above vertex corrections is similar to incorporating an extra
“fourth-step” in the three-step model. This extra step involves all N-1
particles in the Hilbert space undergoing a coherent rotation. In this Dirac
picture, the $\widetilde{{G_{1}}}$ and vertex corrections are separable as
Heisenberg and Schrodinger time evolutions. The procedure is applied in Sec.
IV.

###  III.3 Description in the translationally invariant medium (LT phase)

For illustrative purposes, let us begin with a description at $T=0$. The state
of the system before the transition is translationally invariant as it
consists entirely of the $\left|{uud}\right\rangle$ sites on all sites:

${\left|{{\psi_{0}}}\right\rangle^{T=0}}=\mathop{\prod}\limits_{i}^{N}{\left|{uud}\right\rangle^{\left(i\right)}}$
(9)

Now, we include the action of a photon (an electromagnetic field
$A\left(t\right)$), which leads to a final state where a photo-electron has
been suddenly removed. The coherent band we consider, called $S_{coh}$ in the
following, is a surface spectral feature. Since carriers $S_{coh}$ exist on
the surface therefore, there are no issues with escape length or penetration
depth. The final state at $t=0$, intermediately after the photoemission event,
can be expressed as a sum of Slater determinants in a real-space basis:

${\left|{{\psi_{im}}}\right\rangle^{T=0}}=\mathop{\sum}\limits_{x_{0}}^{N}{\left|{uud}\right\rangle_{x_{0}}^{*}}\otimes\widetilde{\prod\limits_{x\neq{x_{0}}}}{\left|{uud}\right\rangle^{\left(x\right)}}$
(10)

Where we have singled out one particular site as the rapid photoemission event
takes place at a given site ${x_{0}}$ containing suddenly one photo-hole,
written as $\left|{\cdots}\right\rangle_{{x_{0}}}^{*}$, $\otimes$ is a tensor
product, and $\widetilde{\prod\limits_{x\neq{x_{0}}}}$ is an anti-symmetrized
product (i.e., a Slater determinant over all other occupied sites). The photo-
hole now propagates through the sample (to be precise, in the strongly
correlated model under consideration a full fermion is a fusion of fractional
particles with charge e/3 as derived in Ref.[21]), and, upon exiting the
sample, it re-combines with a photo-electron on the detector. We now make a
key assumption of this reasoning: Since at $T=0$ the sample is translationally
invariant (along the edge direction), hence the momentum is a good quantum
number and only states with well-defined momentum emerge out of the
${x_{0}}$-summation. Therefore, the initial state can be written as a simple
tensor product

$\left|{\psi_{i}}\right\rangle^{T=0}\sim
c_{k}^{+}(0)\left|{\psi_{f-im}}\right\rangle^{T=0}\sim
c_{k}^{+}(0)\left|{\psi_{k}}\right\rangle\left|{N-1}\right\rangle_{uud}^{(0)}$
(11)

where the $(0)$ subscript indicates that the $N-1$ particles stay in their
unaffected quantum state. It is identical to the form of the final state that
can be deduced from the propagation on the triangle diagram.

For the final state, by invoking momentum conservation (which we assume to be
obeyed), we use the statement derived at the beginning of this section that
only a particular combination of real space Slater determinants survives the
quantum interference:

${\left|{{\psi_{f}}\left(t\right)}\right.^{T=0}}=\sum\limits_{s}c_{k}^{+}\left(t\right)\left|{N-1}\right\rangle_{uud}^{s}$
(12)

where the desired combination has been generated by applying the $c_{k}^{+}$
operator acting on the $N-1$ other fermions in the uud state. The summation
over $s$ is over all possible trajectories that will produce the desired state
with momentum $k$, e.g., on its way, the photo-hole may emit/absorb a phonon,
plasmon, etc. … (when the self-energy is zero, there shall be only one term
left in this sum). The expression for the photo-electron intensity simplifies
remarkably, especially if evaluated in the Heisenberg picture:

$\displaystyle I$
$\displaystyle=\;^{T=0}\left\langle{\psi_{f}}|{\psi_{i}}\right\rangle^{T=0}$
$\displaystyle=\sum\limits_{s}{}_{uud}^{s}\left\langle{N-1}\right|\otimes\left\langle{\psi_{k}}\right|c_{k}(t)c_{k}^{+}(0)\left|{\psi_{k}}\right\rangle\otimes\left|{N-1}\right\rangle_{uud}^{(0)}$
$\displaystyle=A(k,\omega)\sum\limits_{s}{}_{uud}^{s}\left\langle{N-1}\right|1\left|{N-1}\right\rangle_{uud}^{(0)}.$
(13)

Where
$A\left({k,\omega}\right)={\psi_{k}}|c_{k}^{+}\left(t\right)c_{k}\left(0\right)|{\psi_{k}}$
is the spectral function, a correlator whose expectation value can be computed
on a single disentangled state of the Hilbert space since we assumed that the
“blue particle” is disentangled from the remaining $N-1$ particles. The time
evolution of the single hole part of the final state (${{\cal
U}^{+}(t)\;c_{k}^{+}{\cal U}(t)}$) combined with the initial state
$c_{k}^{+}(t=0)$ gives (an imaginary part of) a propagator whose temporal
Fourier transform is the single-particle spectral function with a given
momentum $k$. Based on this, it is usually found that ARPES measures the
spectral function for carriers with given momentum as expressed by the 2nd
equality on the r.h.s. As a result, our ARPES measures the excitation of the
$S_{coh}$ carriers with well-defined energy and momentum. Please note that we
are making here a hidden assumption that the remaining bracket is unitary: the
$c_{k}^{+}(t)$ electron has not changed any other states such that the overlap
between the $N-1$ states is perfect.

##  IV Solution for the disordered medium (HT phase)

Based on the reasoning in Sec.III we are now ready to tackle the photoemission
process from the disordered system.

###  IV.1 Carrier propagation through the mixed state

Now we move to finite temperatures where the initial state consists of a
mixture of $\left|{uud}\right\rangle$\- sites and single chain
$\left|{sc}\right\rangle$-sites (NOTE: $\times$2$\times$4 region is equivalent
to the $\left|{sc}\right\rangle$). The latter ones presumably form a fractal
structure – a Brownian tree[30]. The initial state reads:

$\left|{\psi_{0}}\right\rangle^{T>{T^{*}}}=\prod\limits_{i}^{n}\left|{uud}\right\rangle^{(i)}\otimes\prod\limits_{j}^{m}\left|{sc}\right\rangle^{(j)}$
(14)

Removing a photo-electron from this state is a non-trivial operation as the
state is intrinsically non-homogeneous. The translational invariance is not
obeyed; we cannot assume that the momentum is a good quantum number. For the
intermediate state at $t=0$, intermediately after the photoemission event, we
need to write a double sum to account for the fact that an empty site can be
either within $\left|{uud}\right\rangle$ or $\left|{sc}\right\rangle$:

${\left|{{\psi_{im}}}\right\rangle^{T>{T^{*}}}}=\mathop{\sum}\limits_{{x_{0}}}^{n}\left|{uud}\right._{{x_{0}}}^{*}\otimes\widetilde{\prod\limits_{\begin{array}[]{*{20}{c}}{x\in
n}\\\
{x\neq{x_{0}}}\end{array}}}\left|{uud}\right\rangle^{(x)}+\sum\limits_{{x_{0}}}^{m}\left|{sc}\right\rangle_{{x_{0}}}^{*}\otimes\widetilde{\prod\limits_{\begin{array}[]{*{20}{c}}{x\in
m}\\\ {x\neq{x_{0}}}\end{array}}}\left|{sc}\right\rangle^{(x)}$ (15)

If the disordered state would be infinitely rigid (i.e., the $sc$ and $uud$
would be described by two independent quantum states), then the photo-hole on
${\left|{sc}\right\rangle^{*}}$-site would be forever locked and would not
contribute to a $uud$ photocurrent. More precisely: a different ansatz for the
$N$-body wave function should then be used. Namely, each observable would have
been described by a simple sum of signals from $\left|{uud}\right\rangle$ and
disordered zones that contain the $\left|{sc}\right\rangle$-sites. At
temperatures in the transition range, we would then expect a double peak
structure (a narrow coherent feature with decreasing amplitude, stemming from
the $\left|{uud}\right\rangle$ -sites, and a much broader increasing
incoherent feature stemming from the $\;\left|{sc}\right\rangle$-sites.

In our problem however, the photo-hole in the
${\left|{sc}\right\rangle^{*}}$-site can be extracted by applying the Ising-
disorder operator
$\mathop{\prod}\limits_{i}^{sd}{\hat{\mu}^{\left(i\right)}}\left(t\right)$
sufficiently many times. This represents an additional time-evolution[29] of
the $\left|{N-1}\right\rangle$-system compared to the case at $T=0$ discussed
above. This time evolution affects all the other states. We can incorporate in
the triangle diagram as a vertex correction at photoemission corner and
include this extra time evolution in the Dirac picture by taking an $sd$-long
sequence of Ising-disorder operators
$\mathop{\prod}\limits_{i}^{sd}{\hat{\mu}^{\left(i\right)}}\left(t\right)$ for
producing the additional time-dependent interaction and find the final form of
the initial state:

${\left|{{\psi_{i}}}\right\rangle^{T>{T^{*}}}}=\mathop{\sum}\limits_{s}\left|{\psi_{k}}\right\rangle\;\otimes{\left({\mathop{\sum}\limits_{sd}\mathop{\prod}\limits_{i}^{sd}{{\hat{\mu}}^{\left(i\right)}}\left(t\right)\left|{N-1}\right\rangle}\right)_{uud}}$
(16)

Where the summation over $sd$ indicates various trajectories over which the
$\prod\limits_{i}^{sd}{\hat{\mu}^{(i)}}(t)$ have been applied. Once the photo-
hole is in the domain of the uud states, its further propagator towards the
relaxation is known, the momentum is now conserved, and we can follow the same
lines as those computed above for the $T=0$ case. By our reasoning, the time
evolution consists now of two stages. Hence, its Fourier transform will be a
convolution of two propagators: the $\left|{uud}\right\rangle$ propagator,
constituting the $S_{coh}$ dispersion, and an additional propagator of the
$\left|{N-1}\right\rangle$-carriers due to the application of the Ising
operators.

###  IV.2 Temperature dependence of ARPES in the disordered medium

We then arrive at our main result that the photo-electron current in momentum-
frequency space, analogous Eq.(13), is given by a convolution:

$I=A(k,\omega)\otimes\left\langle(N-1)_{f_{t=t_{d}(T)}}|(N–1)_{i}\right\rangle$
(17)

where we see that standard spectral function of the coherent part
$A(k,\omega)$ is convoluted with a term that describes the Dirac time
evolution of remaining $N-1$ fermions. The subscript indicates that we
consider a final state after a given time $t_{d}$, see below for its
definition.

In our case the $sc\rightarrow uud$ tunneling is local, hence the momentum
dependence of Dirac part drops out. Furthermore, for our immobile $sc$ units,
in 1D regime it is known that even for interacting case the effect of
scattering can be absorbed as a purely phase factor[31]. In a language of
Ref.32 we can thus safely neglect the dynamics of amplitudes and focus on
phase evolution (as in Ref.33) of the disorder operators $\hat{\mu}$. In this
case the convolution can be reduced to a simple multiplication.

To complete our analysis, we can now compute this overlap between the initial
and final $\left|{N-1}\right\rangle$ states at finite temperature, which we
shall call Dirac evolution amplitude:

${A^{D}}=\left\langle{\left({N-1}\right)_{f_{t={t_{d}}(T)}}|({N-1})_{i}}\right\rangle$
(18)

This overlap enters into the total overlap between
$\left\langle{\psi_{f}}{\rm{|}}{\psi_{i}}\right\rangle$ as an extra, non-
unitary factor. We assume that there exist a typical delay time ${t_{d}}$
which is needed to remove the photo-hole from the $\left|{sc}\right\rangle$
zone. Taking a constant inter-site tunneling rate, this delay is merely
proportional to a typical escape length ${x_{d}}$ from inside the aggregate of
$sc$-zones which is a fractal. From the diffusion-limited aggregation[34] we
know that the fractal is a Brown tree[30]. By increasing the temperature $T$
we increase the number of $\times$2$\times$4 units (as there are more
available many-body states in an energy window ${k_{B}}T$), the volume of the
fractal ${V_{d}}$ (Brown tree) scales with the number of states like
${N^{{d_{H}}}}$ (where ${d_{H}}$ is the Hausdorff dimension) while the
$X_{d}\sim{V_{d}}^{1/d}$. In our case ${d_{H}}=2$ and $d=2$. Hence, we deduce
that $t_{d}\sim T$. Having this information, we assume in the following that
the convolution of the Dirac evolution amplitude with the uud spectral
function results in a Lorentzian-like function showing a continuous change of
maximum intensity and width.

For instance, let us consider the strength of the maximum intensity of photo-
electrons. When we increase the temperature above
${{\rm{T}}_{1}}={\rm{\;}}{{\rm{\Delta}}_{1}}$ the amplitude of the maximum
shall decrease as the overlap between initial and final Ising operators
configuration degrades. This is because close to the transition, due to
$\prod\limits_{i}^{sd}{\hat{\mu}^{\left(i\right)}}\left(t\right)$
fluctuations, the Ising order is known[35] to decay like $1/{t^{1/4}}$. Hence,
we expect $1/{T^{1/4}}$ decay of the photocurrent amplitude that is:

$A_{D}=1/T^{1/4}$ (19)

simple relation that can be readily probed by experiments. It should be
emphasized that the temperature dependence in Eq.19 is not due to any internal
phononic-type dynamics[32] on each site, but it is a many body effect of a
coherence loss within the $sc$ conglomerate. In other words there is a loss of
coherence between two Ising $\hat{\mu}$ operators applied at a certain
interval of time $t_{d}$.

##  V Discussion and Conclusions

Firstly, within our example, we would like to point out, that the formalism
proposed here is much more general than for a transition between the uud and
sc phases only induced by temperature. One can think of other external
parameters inducing the transition. For the specific case of screened Coulomb
interaction mediated inter-chain coupling, these can be, i.e., local chemical
doping, local tension or a local screening by an external inhomogeneous
electrode. This richness of possible means of control comes from the fact that
the screening has an internal degrees of freedom and is a great advantage of
the proposed platform based on a dimensional-cross over.

In our example, we had certain operators that induced the disorder. These were
the Ising operators because these are the ones that diagonalize the strongly-
correlated Hamiltonian. An advantage of our example was that, by construction
through topological protection, it created _localized and stable_ disordered
units. Generally, one can consider other operators that induce disorder. We
could consider a variation of the chemical potential for introducing disorder.
However, this approach will only succeed in cases where the Hamiltonian
results in a single particle description (hence: disorder leading to Anderson
localization). Then, the critical behaviour of the chemical-potential operator
is known; it is a marginal operator. One can also consider a model where
electrons are coupled with a local boson, e.g. an optical phonon mode. If the
disorder is then proportional to the local density of phonons ($\equiv$
displacement), the criticality is proportional to the scaling operator of the
phononic density.

The main outcome of this work is that it provides a formalism to quantify how
an ARPES band is disappearing in the presence of disorder. Remarkably, from
the energy scale of its disappearance as well as from the profile of the
broadening curve, we can gather a lot of information about the nature of a
phase transition induced by that disorder. The underlying reason, why the band
broadens, is the breaking of the translational symmetry. This implies that the
momentum is not any longer a good quantum number and that the spectral weight
is broadly distributed within the reciprocal space. We have taken here a
particular model which allows for only one selected band to be affected. It
should be emphasized that, while Eq.(17) describes how the spectral weight of
the _coherent_ part of the band disappears, it does not invalidate the ARPES-
sum-rule. The ARPES sum-rule can be traced back to the anti-commutation rules
for the underlying fermionic fields[36] which, as such, still hold. It just
requires to include all scattering channels of fermions, i.e., all the ways in
which the _incoherent_ part of the propagator can be generated. In the
specific case of the Ising model proposed here, that would mean including all
the possible order-disorder operators’ correlation functions.

From the experimental perspective, our theory can also be useful in other
ways, albeit more limited. Consider a system with an unknown amount of
embedded disorder. The non-linear dependence of the amplitudes’ suppression
(and the coherent band’s broadening) allows to extract, i.e. by exploring the
derivative, small changes in disorder strength. This, in turn, allows to
detremine the amplitude of the disorder. With this information, the entire
ARPES spectra can be fitted using an additional correction that fully accounts
for non-perturbative effects of the disorder.

In _conclusion_ we derived here a formalism that extends the standard _sudden-
electron_ approximation to cases where carriers are localized for a finite
time during the photo-emission event. Then, the ARPES spectrum acquires an
extra broadening factor which can be interpreted as a vertex correction. If
there is any operator that links the ordered and disordered phases, we then
showed that the broadening of the ARPES spectrum mimics the dynamics of this
operator.

## References

  * Ding et al. [1996] H. Ding, T. Yokoya, J. C. Campuzano, T. Takahashi, M. Randeria, M. Norman, T. Mochiku, K. Kadowaki, and J. Giapintzakis, Nature 382, 51 (1996).
  * Loeser et al. [1996] A. Loeser, Z.-X. Shen, D. Dessau, D. Marshall, C. Park, P. Fournier, and A. Kapitulnik, Science 273, 325 (1996).
  * Im et al. [2008] H. J. Im, T. Ito, H.-D. Kim, S. Kimura, K. E. Lee, J. B. Hong, Y. S. Kwon, A. Yasui, and H. Yamagami, Phys. Rev. Lett. 100, 176402 (2008), URL https://link.aps.org/doi/10.1103/PhysRevLett.100.176402.
  * Xia et al. [2009] Y. Xia, D. Qian, D. Hsieh, L. Wray, A. Pal, H. Lin, A. Bansil, D. Grauer, Y. S. Hor, R. J. Cava, et al., Nature Physics 5, 398 (2009), URL https://doi.org/10.1038/nphys1274.
  * Zhang et al. [2014] Y. Zhang, T.-R. Chang, B. Zhou, Y.-T. Cui, H. Yan, Z. Liu, F. Schmitt, J. Lee, R. Moore, Y. Chen, et al., Nature Nanotechnology 9, 111 (2014), URL https://doi.org/10.1038/nnano.2013.277.
  * Kim et al. [2008] B. J. Kim, H. Jin, S. J. Moon, J.-Y. Kim, B.-G. Park, C. S. Leem, J. Yu, T. W. Noh, C. Kim, S.-J. Oh, et al., Phys. Rev. Lett. 101, 076402 (2008), URL https://link.aps.org/doi/10.1103/PhysRevLett.101.076402.
  * Min et al. [2019] C.-H. Min, H. Bentmann, J. N. Neu, P. Eck, S. Moser, T. Figgemeier, M. Ünzelmann, K. Kissner, P. Lutz, R. J. Koch, et al., Phys. Rev. Lett. 122, 116402 (2019), URL https://link.aps.org/doi/10.1103/PhysRevLett.122.116402.
  * Denlinger et al. [2022] J. D. Denlinger, J.-S. Kang, L. Dudy, J. W. Allen, K. Kim, J.-H. Shim, K. Haule, J. L. Sarrao, N. P. Butch, and M. B. Maple, Electronic Structure 4, 013001 (2022), URL https://doi.org/10.1088/2516-1075/ac4315.
  * Alet and Laflorencie [2018] F. Alet and N. Laflorencie, Comptes Rendus Physique 19, 498 (2018), ISSN 1631-0705, quantum simulation / Simulation quantique, URL https://www.sciencedirect.com/science/article/pii/S163107051830032X.
  * Abanin et al. [2019] D. A. Abanin, E. Altman, I. Bloch, and M. Serbyn, Rev. Mod. Phys. 91, 021001 (2019), URL https://link.aps.org/doi/10.1103/RevModPhys.91.021001.
  * Chamon and Mudry [2001] C. Chamon and C. Mudry, Phys. Rev. B 63, 100503 (2001), URL https://link.aps.org/doi/10.1103/PhysRevB.63.100503.
  * Basko et al. [2006] D. Basko, I. Aleiner, and B. Altshuler, Annals of Physics 321, 1126 (2006), ISSN 0003-4916, URL https://www.sciencedirect.com/science/article/pii/S0003491605002630.
  * Gornyi et al. [2005] I. V. Gornyi, A. D. Mirlin, and D. G. Polyakov, Phys. Rev. Lett. 95, 206603 (2005), URL https://link.aps.org/doi/10.1103/PhysRevLett.95.206603.
  * Schreiber et al. [2015] M. Schreiber, S. S. Hodgman, P. Bordia, H. P. Lüschen, M. H. Fischer, R. Vosk, E. Altman, U. Schneider, and I. Bloch, Science 349, 842 (2015), eprint https://www.science.org/doi/pdf/10.1126/science.aaa7432, URL https://www.science.org/doi/abs/10.1126/science.aaa7432.
  * yoon Choi et al. [2016] J. yoon Choi, S. Hild, J. Zeiher, P. Schauß, A. Rubio-Abadal, T. Yefsah, V. Khemani, D. A. Huse, I. Bloch, and C. Gross, Science 352, 1547 (2016), eprint https://www.science.org/doi/pdf/10.1126/science.aaf8834, URL https://www.science.org/doi/abs/10.1126/science.aaf8834.
  * Lüschen et al. [2017] H. P. Lüschen, P. Bordia, S. Scherg, F. Alet, E. Altman, U. Schneider, and I. Bloch, Phys. Rev. Lett. 119, 260401 (2017), URL https://link.aps.org/doi/10.1103/PhysRevLett.119.260401.
  * Hwang and Das Sarma [2008] E. H. Hwang and S. Das Sarma, Phys. Rev. B 77, 081412 (2008), URL https://link.aps.org/doi/10.1103/PhysRevB.77.081412.
  * Durham [1981] P. J. Durham, Journal of Physics F: Metal Physics 11, 2475 (1981), URL https://doi.org/10.1088/0305-4608/11/11/027.
  * Bansil and Pessa [1983] A. Bansil and M. Pessa, Physica Scripta T4, 52 (1983), URL https://doi.org/10.1088/0031-8949/1983/t4/009.
  * Starykh [2015] O. A. Starykh, Reports on Progress in Physics 78, 052502 (2015), URL https://doi.org/10.1088/0034-4885/78/5/052502.
  * Lecheminant and Orignac [2004] P. Lecheminant and E. Orignac, Phys. Rev. B 69, 174409 (2004), URL https://link.aps.org/doi/10.1103/PhysRevB.69.174409.
  * Essler and Tsvelik [2002] F. H. L. Essler and A. M. Tsvelik, Phys. Rev. B 65, 115117 (2002), URL https://link.aps.org/doi/10.1103/PhysRevB.65.115117.
  * Starykh et al. [2010] O. A. Starykh, H. Katsura, and L. Balents, Phys. Rev. B 82, 014421 (2010), URL https://link.aps.org/doi/10.1103/PhysRevB.82.014421.
  * Starykh and Balents [2007] O. A. Starykh and L. Balents, Phys. Rev. Lett. 98, 077205 (2007), URL https://link.aps.org/doi/10.1103/PhysRevLett.98.077205.
  * Berthod [2018] C. Berthod, _Spectroscopic Probes of Quantum Matter_ , 2053-2563 (IOP Publishing, 2018), ISBN 978-0-7503-1741-2, URL https://dx.doi.org/10.1088/978-0-7503-1741-2.
  * Almbladh [2006] C.-O. Almbladh, Journal of Physics: Conference Series 35, 127 (2006), URL https://doi.org/10.1088/1742-6596/35/1/011.
  * Bardyszewski and Hedin [1985] W. Bardyszewski and L. Hedin, Physica Scripta 32, 439 (1985), URL https://doi.org/10.1088/0031-8949/32/4/033.
  * Yasuhara et al. [1999] H. Yasuhara, S. Yoshinaga, and M. Higuchi, Phys. Rev. Lett. 83, 3250 (1999), URL https://link.aps.org/doi/10.1103/PhysRevLett.83.3250.
  * McMullen et al. [1976] T. McMullen, B. Bergersen, and P. Jena, Journal of Physics C: Solid State Physics 9, 975 (1976), URL https://doi.org/10.1088/0022-3719/9/6/016.
  * Witten and Sander [1981a] T. A. Witten and L. M. Sander, Phys. Rev. Lett. 47, 1400 (1981a), URL https://link.aps.org/doi/10.1103/PhysRevLett.47.1400.
  * Eggert and Affleck [1992] S. Eggert and I. Affleck, Phys. Rev. B 46, 10866 (1992), URL https://link.aps.org/doi/10.1103/PhysRevB.46.10866.
  * Braun et al. [2013] J. Braun, J. Minár, S. Mankovsky, V. N. Strocov, N. B. Brookes, L. Plucinski, C. M. Schneider, C. S. Fadley, and H. Ebert, Phys. Rev. B 88, 205409 (2013), URL https://link.aps.org/doi/10.1103/PhysRevB.88.205409.
  * Feder [1981] R. Feder, Journal of Physics C: Solid State Physics 14, 2049 (1981), URL https://doi.org/10.1088/0022-3719/14/15/006.
  * Witten and Sander [1981b] T. A. Witten and L. M. Sander, Phys. Rev. Lett. 47, 1400 (1981b), URL https://link.aps.org/doi/10.1103/PhysRevLett.47.1400.
  * Wu et al. [1976] T. T. Wu, B. M. McCoy, C. A. Tracy, and E. Barouch, Phys. Rev. B 13, 316 (1976), URL https://link.aps.org/doi/10.1103/PhysRevB.13.316.
  * Gursoy et al. [2012] U. Gursoy, E. Plauschinn, H. Stoof, and S. Vandoren, Journal of High Energy Physics 2012, 18 (2012), ISSN 1029-8479, URL https://doi.org/10.1007/JHEP05(2012)018.