# On the Sensitivity of Reward Inference to
Misspecified Human Models

Joey Hong
UC Berkeley
<EMAIL_ADDRESS>&Kush Bhatia
Stanford
<EMAIL_ADDRESS>&Anca Dragan
UC Berkeley Work conducted as student at UC Berkeley

###### Abstract

Inferring reward functions from human behavior is at the center of value
alignment – aligning AI objectives with what we, humans, actually want. But
doing so relies on models of how humans behave given their objectives. After
decades of research in cognitive science, neuroscience, and behavioral
economics, obtaining accurate human models remains an open research topic.
This begs the question: _how accurate do these models need to be in order for
the reward inference to be accurate?_ On the one hand, if small errors in the
model can lead to catastrophic error in inference, the entire framework of
reward learning seems ill-fated, as we will never have perfect models of human
behavior. On the other hand, if as our models improve, we can have a guarantee
that reward accuracy also improves, this would show the benefit of more work
on the modeling side. We study this question both theoretically and
empirically. We do show that it is unfortunately possible to construct small
adversarial biases in behavior that lead to arbitrarily large errors in the
inferred reward. However, and arguably more importantly, we are also able to
identify reasonable assumptions under which the reward inference error can be
bounded linearly in the error in the human model. Finally, we verify our
theoretical insights in discrete and continuous control tasks with simulated
and human data.

##  1 Introduction

The expanding interest in the area of reward learning stems from the concern
that it is difficult (or even impossible) to specify what we actually want AI
agents to optimize, when it comes to increasingly complex, real-world tasks
(Ziebart et al., 2009; Muelling et al., 2017). At the core of reward learning
is the idea that human behavior serves as evidence about the underlying
desired objective. Research on inferring rewards typically uses noisy-
rationality as a model for human behavior: the human will take higher value
actions with higher probability. It has enjoyed great success in a variety of
reward inference applications (Ziebart et al., 2008; Vasquez et al., 2014;
Wulfmeier et al., 2015), but researchers have also started to come up against
its limitations (Reddy et al., 2018). This is not surprising, given decades of
research in _behavioral_ economics that has identified a deluge of systematic
biases people have when making decisions on how to act, like myopia/hyperbolic
discounting (Grüne-Yanoff, 2015), optimism bias (Sharot et al., 2007),
prospect theory (Kahneman & Tversky, 2013), and many more (Thompson, 1999; Do
et al., 2008). Hence, the noisy-rationality model has become a complication in
many reward learning tasks AI researchers are interested in. For instance, in
shared autonomy (Javdani et al., 2015), a human operating a robotic arm may
behave suboptimally due to being unfamiliar with the control interface or the
robot’s dynamics, leading to the robot inferring the wrong goal (Reddy et al.,
2014; Chan et al., 2021).

Recent work in reward learning attempts to go beyond noisy rationality and
consider more accurate models of human behavior, by for instance looking at
biases as variations on the Bellman update (Chan et al., 2021), modeling the
human’s false beliefs (Reddy et al., 2018), or learning their suboptimal
perception process (Reddy et al., 2020). And while we might be getting closer,
we will realistically never have a _perfect_ model of human behavior. This
raises an obvious question: _Does the human model need to be perfect in order
for reward inference to be successful?_ On the one hand, if small errors in
the model can lead to catastrophic error in inference, the entire framework of
reward learning seems ill-fated, especially as it applies to value alignment:
we will never have perfect models, and we will therefore never have guarantees
that the agent does not do something catastrophically bad with respect to what
people actually value. On the other hand, if we can show that as our models
improve, we have a guarantee that reward accuracy also improves, then there is
hope: though modeling human behavior is difficult, we know that improving such
models will make AI agents more aligned with us.

The main goal of this work is to study whether we can bound the error in
inferred reward parameters by some function of the distance between the
assumed and true human model, specifically the KL divergence between the two
models. We study this question both theoretically and empirically. Our first
result is a negative answer: we show that given a finite dataset of
demonstrations, it is possible to hypothesize a true human model, under which
the dataset is most likely to be generated, that is "close" to the assumed
model, but results in arbitrarily large error in the reward we would infer via
maximum likelihood estimation (MLE). However, we argue that though this
negative scenario can arise, it is unlikely to occur in practice. This is
because the result relies on an adversarial construction of the human model,
and though the dataset is most likely, it is not necessarily representative of
datasets sampled by the model. Given this, our main result is thus a reason
for hope: we identify mild assumptions on the true human behavior, under which
we can actually bound the error in inferred reward parameters linearly by the
error of the human model. Thus, if these assumptions hold, refining the human
model will monotonically improve the accuracy of the learned reward. We also
show how this bound simplifies for particular biases like false internal
dynamics or myopia.

Empirically, we also show a similar, optimistic message about reward learning,
using both diagnostic gridworld domains (Fu et al., 2019b), as well as the
Lunar Lander game, which involves continuous control over a continuous state
space. First, we verify that under various simulated biases, when the
conditions on the human model are likely to be satisfied, small divergences in
human models do not lead to large reward errors. Second, using real human
demonstration data, we derive a natural human bias and demonstrate that the
same finding holds even with real humans. Overall, our results suggest an
optimistic perspective on the framework of reward learning, and that efforts
in improving human models will further enhance the quality of the inferred
rewards.

##  2 Related Work

Inverse reinforcement learning (IRL) aims to use expert demonstrations, often
from a human, to infer a reward function (Ng & Russel, 2000; Ziebart et al.,
2008). Maximum-entropy (MaxEnt) IRL is a popular IRL framework that models the
demonstrator as noisily optimal, maximizing reward while also randomising
actions as much as possible (Ziebart et al., 2008; 2010). This is equivalent
to modeling humans as Boltzmann rational. MaxEnt IRL is preferred in practice
over Bayesian IRL (Ramachandran & Amir, 2007), which learns a posterior over
reward functions rather than a point estimate, due to better scaling in high-
dimensional environments (Wulfmeier et al., 2015). More recently, Guided Cost
Learning (Finn et al., 2016) and Adversarial IRL (Fu et al., 2018) learn
reward functions more robust to environment changes, but build off similar
modeling assumptions as MaxEnt IRL. Gleave & Toyer (2022) connected MaxEnt IRL
to maximum likelihood estimation (MLE), which is the framework that we
consider in this work. One of the challenges with IRL is that rewards are not
always uniquely identified from expert demonstrations (Cao et al., 2021; Kim
et al., 2021). Since identifiability is orthogonal to the main message of our
work–sensitivity to misspecified human models–we assume that the dataset
avoids this ambiguity.

Recent IRL algorithms attempt to account for possible irrationalities in the
expert (Evans et al., 2016; Reddy et al., 2018; Shah et al., 2019). Reddy et
al. (2018; 2020) consider when experts behave according to an internal
physical and belief dynamics, and show that explicitly learning these dynamics
improves accuracy of the learned reward. Singh et al. (2018) account for human
risk sensitivity when learning the reward. Shah et al. (2019) propose learning
general biases using demonstrations across similar tasks, but conclude that
doing so without prior knowledge is difficult. Finally, Chan et al. (2021)
show that knowing the type of irrationality the expert exhibits can improve
reward inference over even an optimal expert. In this work, we do not assume
the bias can be uncovered, but rather analyze how sensitive reward inference
is to such biases.

More generally, reward learning is a specific instantiation of an inverse
problem, which is well-studied in existing literature. In the framework of
Bayesian inverse problems, prior work has analyzed how misspecified likelihood
models affect the accuracy of the inferred quantity when performing Bayesian
inference. Owhadi et al. (2015) showed that two similar models can lead to
completely opposite inference of the desired quantity. Meanwhile, Sprungk
(2020) showed inference is stable under a different measure of distance
between models. In this work, we also derive both instability and stability
results, but consider the different problem of reward learning using MLE.

##  3 Problem Setup

Reward parameters. We consider Markov decision processes (MDP), which are
defined by a tuple $(\mathcal{S},\mathcal{A},P,r,\gamma)$. Here,
$\mathcal{S},\mathcal{A}$ represent state and action spaces,
$P(s^{\prime}|s,a)$ and $r(s,a)$ represent the dynamics and reward function,
and $\gamma\in(0,1)$ represents the discount factor. In this work, we are
interested in the setting where the reward function $r$ is unknown and needs
to be inferred by a learner. We assume rewards are bounded
$\left|r(s,a)\right|\leq R_{\max}$. We assume that the reward can be
parameterized by _reward function parameters_ $\theta\in\Theta$. We denote by
$r(\cdot;\theta)$ the reward function with $\theta$ as parameters.

True vs. assumed human policy. Instead of having access to the reward, we
observe the behavior of an “expert" demonstrator. Let
$\pi^{*}:\Theta\times\mathcal{S}\to\Delta(\mathcal{A})$ be the reward-
conditioned _demonstrator policy_ , and
$\mathcal{D}=\\{(s_{t},a_{t})\\}_{t=1}^{n}$ be a _dataset_ of demonstrations
provided to the learner, sampled from $\pi^{*}$. We use $(s,a)\sim w^{\pi}$ to
denote observations generated by policy $\pi$, where $w^{\pi}$ denotes the
discounted stationary distribution. We shorthand $w^{\pi^{*}}$ as $w^{*}$.
Finally, let $\widetilde{\pi}:\Theta\times\mathcal{S}\to\Delta(\mathcal{A})$
be the _model_ that the learner assumes generated the dataset. In practice,
$\widetilde{\pi}$ is often the Boltzmann rational policy (Ziebart et al.,
2008), while $\pi^{*}$ is an irrational policy based on human biases.

Reward inference using the assumed policy. Many popular algorithms in inverse
reinforcement learning (IRL) (Ziebart et al., 2008; 2010) infer the reward
function parameters via maximum-likelihood estimation (MLE). This is because
unlike Bayesian IRL methods that learn a posterior over rewards, such MLE
methods are shown to scale to high-dimensional environments (Wulfmeier et al.,
2015). Using a dataset $\mathcal{D}$, the learner would estimate parameters

$\displaystyle\widetilde{\theta}$
$\displaystyle=\arg\min_{\theta}\frac{1}{n}\sum_{t=1}^{n}-\log\widetilde{\pi}(a_{t}\mid
s_{t};\theta):=\arg\min_{\theta}L(\theta;\widetilde{\pi},\mathcal{D})\,.$ (1)

Let $\theta^{*}$ be the true reward function parameters. Though $\theta^{*}$
cannot always be uniquely determined (Cao et al., 2021; Kim et al., 2021), for
simplicity of analysis, we assume that $\theta^{*}$ is identifiable:

######  Assumption 1.

There exists a unique $\theta^{*}$ satisfying
$\theta^{*}=\arg\min_{\theta}L(\theta;\pi^{*},\mathcal{D})$.

Though Assumption 1 is rather strong, we make it only because we view
identifiability as orthogonal to the subject of our work – sensitivity to
misspecified models.

Goal: effect of error in the model on the error in the inferred reward. The
goal of our paper is to answer whether we can bound the distance between the
inferred reward and the true reward,
$d_{\theta}(\theta^{*},\widetilde{\theta})$, as a function of the distance
between the assumed human model and the true human policy,
$d_{\pi}(\pi^{*},\widetilde{\pi})$, for some useful notions of distance. If
so, then we know that more accurate policies will monotonically improve the
fidelity of the learned rewards. We discuss our choice of distances below.

Reward inference error. The inferred reward is typically used to optimize a
policy in a test environment, and ultimately evaluated using the policy’s
performance in the new environment (Ng & Russel, 2000; Ziebart et al., 2008).
However, the reason we infer reward (instead of simply cloning the human
policy) is because we do not necessarily know the test environment – having
the reward means we can optimize it in environments with different dynamics or
initial state distributions. And unfortunately, adversarial environments exist
whose dynamics amplify small disagreements between the learned and true reward
functions. Hence, without prior knowledge about the test environment, we
cannot derive any bound on a performance-based distance metric. We thus focus
on analysis directly on a distance between the rewards themselves, via their
parameters,
$d_{\theta}(\theta^{*},\widetilde{\theta})=\|\widetilde{\theta}-\theta^{*}\|_{2}^{2}$.
We choose squared distance as a natural and general distance metric, but admit
that rewards may not be smooth in their parameters. As future work, we can
improve our work using more robust measures of reward similarity (Gleave et
al., 2021).

Human model error. Since policies are probability distributions, we can
measure error in the human model as the KL-divergence between the model and
demonstrator policies. We consider two different instatiations of policy
divergence. The first is a _worst-case policy divergence_ that takes the
supremum over all reward parameters _and_ states:

$\displaystyle
d_{\pi}^{\mathsf{wc}}(\pi^{*},\widetilde{\pi})=\sup_{\theta\in\Theta}\sup_{s\in\mathcal{S}}D_{\text{KL}}(\pi^{*}(\cdot\mid
s;\theta)||\widetilde{\pi}(\cdot\mid s;\theta))\,.$ (2)

We use the forward direction of KL-divergence because it contains an
expectation over the demonstrator policy, which aligns with the sampling
distribution of the dataset. Also note that this direction implies that the
human model must cover actions of the true human to have small divergence.
Finally, note that a small model error entails having small error across all
states. This is thus a strong metric for a lower bound. It is not, however, a
strong metric for an upper bound: when used in an upper bound, it would allow
the model to be wrong on states and rewards without paying any more penalty
than a model that is only wrong on one state and reward. We thus also consider
an average error that makes for a stronger upper bound – it considers the
model only on the true reward parameters $\theta^{*}$ and takes an
_expectation over states_. We term this the _weighted policy divergence_ :

$\displaystyle d_{\pi}^{\mathsf{w}}(\pi^{*},\widetilde{\pi})=\mathbb{E}_{s\sim
w^{*}}\left[D_{\text{KL}}(\pi^{*}(\cdot\mid s;\theta^{*})\ ||\
\widetilde{\pi}(\cdot\mid s;\theta^{*}))\right]\,.$ (3)

The weighted policy divergence only looks at the states visited under the true
human behavioral policy $\pi^{*}$ as compared to the worst case divergence,
which compares against all states and rewards.

##  4 The Bad News: Instability Results

We begin our theoretical analysis with a negative result. We prove that even
under the worst-case policy divergence from eq. (2), a small difference
between the model and the true human policy
$d_{\pi}^{\mathsf{wc}}(\pi^{*},\widetilde{\pi})<\varepsilon$ can lead to a
large inference error $d_{\theta}(\theta^{*},\widetilde{\theta})$. Since a
trivial way for this to happen is to get “unlucky" with the data set
$\mathcal{D}$ drawn from $\pi^{*}$, and contain actions that are unlikely even
if $\pi^{*}=\widetilde{\pi}$, we strengthen the result by excluding tail
events and imposing the strong requirement that $\mathcal{D}$ contains the
most likely actions under the demonstrator policy and the true reward:

######  Definition 1.

A policy $\pi$ “likely generates" $\mathcal{D}$ if for every
$(s_{t},a_{t})\in\mathcal{D}$, the observed action is the most likely one
under the true reward, _i.e._ , $a_{t}=\arg\max\pi(a_{t}\mid
s_{t};\theta^{*})$.

This means that actions that appear in $\mathcal{D}$ are always at the modes
of the policy $\pi^{*}$. Our theorem shows that despite the two policies being
close on each state and reward pair, under the stronger notion of worst-case
policy divergence, the inference procedure can lead to large errors.

######  Theorem 2.

For any MDP $\mathcal{M}$ with continuous actions, policy error
$\varepsilon>0$, assumed model $\widetilde{\pi}$, and dataset $\mathcal{D}$,
there exists a demonstrator policy $\pi^{*}$ that likely generates
$\mathcal{D}$ such that the worst-case policy divergence satisfies
$d_{\pi}^{\mathsf{wc}}(\pi^{*},\widetilde{\pi})<\varepsilon$, but the reward
inference error satisfies

$\displaystyle\|\widetilde{\theta}-\theta^{*}\|_{2}^{2}>\frac{1}{2}\sup_{\theta,\theta^{\prime}\in\Theta}\|\theta-\theta^{\prime}\|_{2}^{2}\,.$

The theorem shows that for any continuous-action MDP, and any observed dataset
$\mathcal{D}$, even a small perturbation in the assumed human model can lead
to large inference error. We are able to prove such a strong result by
perturbing the policy $\pi^{*}$ on only the _observed_ state-action pairs in
the dataset. We defer the proof of the theorem to Appendix C, and provide an
illustrative example in Appendix A. Though the theorem presents a pessimistic
worse-case scenario, there are aspects of the proof of it that make it
impractical and potentially unlikely to occur in practice. First, the
construction of $\pi^{*}$ is adversarial, by biasing the demonstrator at
exactly the actions in the dataset. Second, though Definition 1 means the
dataset is most likely to be generated by the demonstrator, it may contain
actions that are not representative of what they would take. Since actions are
continuous, this means that actions that are similar to the observed one (via
some distance metric) are in fact very unlikely.

##  5 The Good News: A Stability Result

Theorem 2 paints a pessimistic picture on the feasibility of reward inference
from human demonstrations, but as discussed towards the end of Section 4, the
mechanisms used to derive Theorem 2 may be impractical and unlikely to occur.
In this section, we show that under reasonable assumptions on the true policy
we can indeed obtain a positive stability result wherein we can upper bound
the reward inference error by a linear function of the weighted policy error.
We identify the following assumption that enables such a result:

######  Assumption 2.

The true and model policies $\pi^{*},\widetilde{\pi}$ are strongly log-concave
with respect to reward parameters $\theta\in\Theta$. Formally, there exists
constant $c>0$ such that for any $s\in\mathcal{S},a\in\mathcal{A}$, $\pi^{*}$
satisfies

$\displaystyle\log\pi^{*}(a\mid s;\theta^{\prime})\leq\log\pi^{*}(a\mid
s;\theta)+\nabla_{\theta}\log\pi^{*}(a\mid
s;\theta)^{\top}(\theta-\theta^{\prime})-\frac{c}{2}\|\theta-\theta^{\prime}\|_{2}^{2}\,,$

and analogously for $\widetilde{\pi}$.

The adversarial construction of demonstrator policy $\pi^{*}$ in deriving
Theorem 2 violate the above log-concavity assumption as they involve drastic
perturbations of the probabilities of just the observed actions, creating
policies that are not smooth in their parameters.

Intuition. We know that log-concavity is violated by unnatural, adversarial
constructions, but, we aim to answer: _does log-concavity always hold outside
of such contrived examples?_ Intuitively, we notice that log-concavity holds
only if, as the reward parameter increases, an action that has become less
preferred cannot become more preferred in the future. This appears to be a
natural property of many policies. For example, if someone already prefers
chocolate over vanilla, increasing the reward of chocolate will never cause
the person to prefer vanilla over chocolate. However, it turns out there are
simple problems where this is violated. In Appendix B, we present a simple
navigation example where Assumption 2 is violated. Though counter examples
exist, we show in our experiments that many natural biases still result in
human models that satisfy Assumption 2.

Under Assumption 2, we can show that the reward inference error can be bounded
linearly by weighted policy divergence. We state the formal result below, and
defer its proof to Appendix C.

######  Theorem 3.

Under Assumption 2 with parameter $c>0$, for any policies
$\pi^{*},\widetilde{\pi}$ with corresponding MLE reward parameters
$\widetilde{\theta},\theta^{*}$, the reward inference error
$d_{\theta}(\theta^{*},\widetilde{\theta})$ is bounded as

$\displaystyle\mathbb{E}_{\mathcal{D}\sim\pi^{*}}\left[\|\widetilde{\theta}-\theta^{*}\|_{2}^{2}\right]\leq\frac{2}{c}\mathbb{E}_{s\sim
w^{*}}\left[D_{\text{KL}}(\pi^{*}(\cdot\mid s;\theta^{*})\ ||\
\widetilde{\pi}(\cdot\mid s;\theta^{*}))\right]\,.$

Theorem 3 differs from Theorem 2 in two important ways: (1) the reward
inference error is in expectation over sampled datasets, and (2) the policy
divergence is the weighted policy divergence. Both these properties are
desirable, as we are agnostic to tail events due to randomness in dataset
sampling, and as discussed in Section 3, we use the smaller of the two notions
of divergence in the upper bound.

###  5.1 Instantiating the upper bound for specific biases

Theorem 3 shows that the reward inference error can be bounded by the weighted
policy divergence between the assumed and true policies. To understand the
result in more detail, we now consider different systematic biases that could
appear in human behavior, and show how they affect the weighted policy
divergence and thus the upper bound.

Without loss of generality, we parameterize both the true and assumed policies
as acting noisily optimal with respect to their own “Q-functions”, _i.e._ ,
$\pi^{*}(a\mid s;\theta)\propto\exp(Q^{*}(s,a;\theta))$ and
$\widetilde{\pi}(a\mid s;\theta)\propto\exp(\widetilde{Q}(s,a;\theta))$.
Importantly, note that even though this parameterization is used in MaxEnt IRL
with the soft Q-values (Ziebart et al., 2008; 2010), neither $Q^{*}$ nor
$\widetilde{Q}$ need necessarily be optimal – in this analysis, we will use
$\widetilde{Q}$, the human model, as the soft Q-value function, and show what
happens when the true model coming from $Q^{*}$ suffers from certain biases.
Following prior work (Reddy et al., 2018; Chan et al., 2021), we examine
biases that can be modelled as deviations from the Bellman update. For a
tabular MDP $M$ with
$\left|\mathcal{S}\right|,\left|\mathcal{A}\right|<\infty$, the soft Bellman
update satisfies:

$\displaystyle
Q(s,a;\theta):=r(s,a;\theta)+\gamma\sum_{s^{\prime}}P(s^{\prime}\mid
s,a)V(s;\theta)\,,\
V(s;\theta):=\log\left(\sum_{a\in\mathcal{A}}\exp(Q(s,a;\theta)\right)\,.$ (4)

Formally, we study examples under which the human demonstrator’s Q-values
$Q^{*}(s,a;\theta)$ satisfy (4) but under a biased MDP $M^{*}$. We consider
two specific sources of bias in the MDP: (1) the transition model $P$ and (2)
the discounting factor $\gamma$. By parameterizing the biases in this way, we
now have an intuitive notion of the degree of bias, and can study how the
magnitude of the bias affects the policy divergence in (3). For brevity, we
simply state the results as corollaries and defer proofs to Appendix C.

Internal dynamics. We first consider irrationalities that result from human
demonstrators having an _internal dynamics model_ $P^{*}$ that is
misspecified. For example, studies in cognitive science have shown that humans
tend to underestimate the effects of inertia in projectile motion (Caramazza
et al., 1981). Similar studies have also shown that humans overestimate their
control over randomness in the environment (Thompson, 1999), dubbed _illusion
of control_. The latter irrationality can be formalized in our
parameterization by assuming that $P^{*}(\cdot\mid s,a)\propto(P(\cdot\mid
s,a))^{n}$, where as $n\to\infty$, the human will believe the dynamics of the
MDP are increasingly more deterministic. In Corollary 4, we show that the
policy distance can be bounded linearly by the bias in transition dynamics:

######  Corollary 4.

Let $\Delta_{P}=\sup_{s,a}\|P^{*}(\cdot\mid s,a)-\widetilde{P}(\cdot\mid
s,a)\|_{1}$. Also, let $\pi^{*},\widetilde{\pi}$ be the policies that result
from from value iteration using (4) with dynamics models
$P^{*},\widetilde{P}$, respectively. Then, their weighted policy divergence is
bounded as

$\displaystyle\mathbb{E}_{s\sim d^{*}}\left[D_{\text{KL}}(\pi^{*}(\cdot\mid
s;\theta^{*})\ ||\ \widetilde{\pi}(\cdot\mid s;\theta^{*}))\right]\leq\
\frac{2|\mathcal{A}|R_{\max}}{(1-\gamma)^{2}}\,\Delta_{P}\,.$

Myopia Bias. The other irrationality we study is when humans overvalue near-
term rewards, dubbed _myopia_ (Grüne-Yanoff, 2015). Such bias can be captured
in our parameterization through a biased discount factor $\gamma^{*}$, where
as $\gamma^{*}\to 0$, the human will act more greedily and prioritize
immediate reward. In Corollary 5, we bound the distance between policies by
the absolute difference in their internal discount factor.

######  Corollary 5.

Let $\pi^{*},\widetilde{\pi}$ be the policies that result from value iteration
using (4) with discount factors $\gamma^{*},\widetilde{\gamma}$, respectively.
Then, their weighted policy divergence is bounded as

$\displaystyle\mathbb{E}_{s\sim d^{*}}\left[D_{\text{KL}}(\pi^{*}(\cdot\mid
s;\theta^{*})\ ||\ \widetilde{\pi}(\cdot\mid s;\theta^{*}))\right]\leq\
\frac{2|\mathcal{A}|R_{\max}}{(1-\widetilde{\gamma})(1-\gamma^{*})}\,\left|\widetilde{\gamma}-\gamma^{*}\right|\,.$

The above result shows that the degree of bias linearly upper-bounds the
weighted policy divergence and hence, from Theorem 3, the expected reward
inference error.

##  6 Empirical Analysis

Our theoretical results predict that in the worst-case, reward inference error
can be arbitrarily bad relative to the human model error, but on average,
under some assumptions, the reward error should be small. Our empirical
results aim to complement our theory by answering the following question under
natural biases in human models in various environments: _do we find a stable
relationship between policy divergence and reward error?_

We tackle this in three ways: (1) simulating the specific biases we analyzed
in Section 5.1, (2) simulating a non-Bellman-update structured kind of bias (a
demonstrator that is still learning about the environment), and (3) collecting
_real human data_. We consider both tabular navigation tasks on gridworld, as
well as more challenging continuous control tasks on the Lunar Lander game
(Brockman et al., 2016b).

Experiment design. Each experiment has a bias we study and an environment
(gridworld or LunarLander). When considering simulated biases, we manipulate
$\pi^{*}$ by manipulating the _magnitude of the bias_ starting at
$\pi^{*}=\widetilde{\pi}$ the Boltzmann optimal policy. This helps us simulate
different hypothetical humans, and see what degree of deviation from
optimality ends up negatively impacting reward inference. When modeling bias
with real human data, we instead fix $\pi^{*}$ as the real human policy, and
manipulate $\widetilde{\pi}$ by interpolating between the Boltzmann optimal
policy and the real human policy – this emulates a practical process where
human models get increasingly more accurate.

###  6.1 Tabular experiments with structured biases

Figure 1: Effect of transition error (measured as the degree of
underestimation of unintended transitions) on (a) weighted policy divergence
and (b) reward inference error on three Gridworld environments (A,B,C). In
(c), we show a scatter plot of the policy and reward errors for different
biased transition model. Note that small policy divergence results in small
reward inference error.

Figure 2: Effect of discount error in the three Gridworld environments. Like
Figure 1, we see a strong correlation between policy divergence and reward
inference error. Figure 3: Gridworld environments.

First, we consider tabular navigation in gridworld domains (Fu et al., 2019a),
where the task is the reach the goal state and earn a reward of $\theta>1$,
which is not known to the agent, while avoiding getting trapped at lava
states. To further complicate the task, the agent can also get stuck at
“waypoint" states that yield a reward of $1$. Depending on the environment, it
can be better for the agent to stop at the waypoint state, to circumvent
taking the longer, more treacherous path to the goal state. The agent is able
to move in either of the four directions, or choose to stay still. To
introduce stochasticity in the transition dynamics, there is a $30\%$ chance
that the agent travels in a different direction than commanded. We consider
three different gridworlds (which we simply call environments A, B, and C)
where we vary in the location of the waypoint state (shown in Figure 3).

In each environment, we want to the learn the underlying reward parameter
$\theta$ from demonstrations; however, the model $\widetilde{\pi}$ is noisily
optimal, whereas the demonstrator policy $\pi^{*}$ is irrational by suffering
from false internal dynamics, or myopia. We model these irrationalities by
either modifying the transition matrix or discount factor, respectively, in
the soft Bellman update in Equation 4. Note that the stationary distribution
$w^{\pi}$ for a policy $\pi$ can be exactly computed; hence, instead of
sampling data from $\pi^{*}$, we use $w^{*}$ to compute exact quantities (see
Section D.1 for technical details).

Internal dynamics. The first irrationality we consider is illusion of control,
where the demonstrator policy significantly underestimates the stochasticity
in the environment. Such biased policies $\pi^{*}$ are obtained via value
iteration on a biased transition matrix $P^{*}$, where the human wrongly
believes the probability $p$ of unintended transitions is smaller than the
true value. As $p\to 0$, the demonstrator becomes more confident that they can
reach the goal state, and will prefer reaching the goal over the waypoint
state, even when the latter is much closer and safely reachable (see Section
D.1 for visualizations of the biased policies). In Figure 1, we show the
effect of the transition bias (error in $p$) on both the weighted policy
divergence, and the reward inference error. The sub-linear trend in Figure 1a
agrees with Corollary 4. Figure 1b and c show a sub-linear dependence of the
reward inference error on the policy divergence, as predicted by Theorem 3.
For environment A, the reward error goes up most quickly with the dynamics
error, but so does the weighted policy divergence, making this divergence a
better indicator of reward error than simply the dynamics error.

Myopia. The next irrationality we look at is myopia, where the demonstrator
policy assumes a biased discounting factor $\gamma^{*}$ that underestimates
the true one. As $\gamma^{*}\to 0$, the biased agent will much more strongly
prefer the closer waypoint state over the goal state. In Figure 2, we see
analogous results to the internal dynamics bias. Namely, Figure 2a agrees with
Corollary 5, and Figure 2b and c shows a sub-linear correlation between policy
and reward error, as predicted by Theorem 3.

###  6.2 Continuous control experiments

Figure 4: Effect of transition error (measured as error in $p$) in the
continuous Lunar Lander environments. The results are consistent with earlier
Gridworld results.

Figure 5: Effect of the simulated human learning bias in the continuous Lunar
Lander environments.

Next, we consider a more challenging domain of navigation with continuous
states and actions. The exact navigation environment is a modification the
Lunar Lander game with continuous actions (Brockman et al., 2016a), where the
agent receives a reward for landing safely on the landing pad. The agent is
able to take a continuous action in $[-1,1]^{2}$ that encodes the directions
it wants to move (left, right, up) via its sign, as well as how much power it
wants to use in each direction via its cardinality.

Figure 6: Lunar Lander environments.

In constrast to the classic version of the game where the landing pad is
always in the middle, we vary its location. The unknown reward parameter
$\theta\in(0,1)$ is the location of the landing pad (as a horizontal
displacement normalized by the total width of the environment). We consider
three different environments that differ in the location of the landing pad
(see Figure 6). In each environment, the human model $\widetilde{\pi}$ is the
near-optimal one obtained by soft actor-critic (Haarnoja et al., 2018). We
provide details on the training procedure in Section D.2. In these
experiments, we simulate the internal dynamics bias as well as a new one based
on the notion of a demonstrator that is still themselves learning.

Internal dynamics. We first study of the effect of demonstrator policies with
biased dynamics models. We bias the dynamics model by varying a parameter $p$
that describes how much one unit of power will increase acceleration in the
corresponding direction. This is a plausibly natural human bias as people tend
to underestimate the effects of inertia in projectile motion (Caramazza et
al., 1981). For each false setting of $p$, we learn a biased policy that is
near-optimal for that $p$. As $p$ increase, the biased policy tends to
underestimate the amount of power required to move the lander enough to the
right to reach the landing pad (see Section D.2 for visualizations). In Figure
4, we show the effect of the transition bias (error in $p$) on both the
weighted policy divergence and the reward inference error. We see that even in
a challenging continuous control domain, Theorem 3 still holds.

Demonstrators that are learning. We next simulate a bias that might arise from
humans that are learning how to do the task (as would be the case, for
instance, in our Lunar Lander task). We do so by varying the amount of
training iterations in learning the policy. The degree of such bias is
captured in a parameter $\rho\in[0,1]$, that denotes the number of training
iterations, normalized by the amount used to learn the near-optimal believed
policy. In Figure 5, we show the effect of $\rho$ on both the weighted policy
divergence and the reward inference error. Reassuringly, we again notice a
sub-linear correlation in line with Theorem 3.

###  6.3 Analysis of real human policies

Figure 7: Effect of modeling human bias (measured by probability of acting
according to human policy) on (a) weighted policy divergence and (b) reward
inference error on discrete Lunar Lander environments. In (c), we show a
scatter plot of the policy and reward errors for different probabilities. We
see that more accurate human models correspond to lower reward inference
error. Figure 8: Visualization of trajectories under the human policy.

The previous experiments have considered natural but simulated biases to
construct demonstrator policies modeling biased humans. However, it remains to
be seen whether Theorem 3 remains predictive even when biases are from real
humans. We consider the same Lunar Lander game in Section 6.2 but discretize
the action space so that people can intuitively interact in it.Using this
environment, we create a demonstrator policy grounded in real human
demonstrations. We do this by collecting trajectories from $10$ human
demonstrators, then learning a policy that imitates human behavior by running
behavior cloning (BC) on the aggregated trajectories. We visualize
trajectories from this policy in Figure 8. We observe that in general, humans
tend to be unable to properly account for the effect of gravity, causing them
to crash the lander before it has moved enough horizontally, particularly in
environments B and C where the landing pad is horizontally displaced from the
middle.

Then, we emulate a process through which the learner’s model of the human,
$\widetilde{\pi}$, would evolve to align more and more with the true human
policy. Specifically, we vary $\widetilde{\pi}$ to interpolate between near-
optimal and the true human policy, while keeping $\pi^{*}$ fixed to the
latter. To do so, we vary a parameter $\alpha$ that controls the probability
of sampling from the human policy (vs. the near-optimal one). In Figure 7, we
show the effect of $\alpha$ on the weighted policy divergence and reward
inference error, and conclude that larger $\alpha$ result in smaller policy
divergence as well as reward error. Importantly, this suggest that as the
model gets closer and closer to the true human policy, the reward estimate
gets increasingly better, with no sign that a small model error leads to a
terrible reward. In addition, we also match the simulation experiments by
keeping $\widetilde{\pi}$ fixed as the optimal policy, and interpolate between
the optimal policy and the real human policy for $\pi^{*}$ (see Section D.3).
This gives us hope that even in real-world problems, better human models
$\widetilde{\pi}$ can translate to better reward inference.

##  7 Discussion

Summary. In this paper, we conduct a theoretical and empirical study of how
sensitive reward learning from human demonstrations is to misspecification of
the human model. First, we provide an ominous result that arbitrarily small
divergences in the assumed human model can result in large reward inference
error. While this is in theory possible, it requires a rather adversarial
construction that makes it unlikely to occur in practice. In light of this, we
identify assumptions under which the reward error can actually be upper-
bounded linearly by the model error. Experiments with multiple biases in
different environments, as well as an analysis of the true human policy (which
potentially suffers from unknown, yet to be characterized suboptimalities),
reassuringly show remarkably consistent results: over and over again, we see
that as the human model and the true human behavior are more and more aligned,
the reward error decreases. Overall, our results convey the optimistic message
that reward learning improves as we obtain better human models, and motivate
further research into improved models.

Limitations and future work. Our upper-bound relies on Assumption 2 of log-
concavity of the human policy. However, we hypothesize that weaker assumptions
exist from which we can derive similar bounds as Theorem 3. In addition, via
Assumption 1, we ignore ambiguity in reward identification. It may be
important in the future to consider reward error and identifiability jointly,
potentially through equivalence classes of reward functions. Finally, as
alluded to in Section 3, more robust measures of reward similarity exist that
avoid needing to implicitly assume smoothness of the reward function in their
parameters (Gleave et al., 2021). Interesting directions of further
investigation include: (1) how to better model human biases, or orthogonally,
(2) how to modify existing reward inference algorithms to be more robust to
misspecification. A negative side-effect of our work could occur when we
mistakenly rely on the upper-bound in Theorem 3 when its conditions are not
met, _i.e._ , Assumption 2 does not hold, resulting in catastrophically bad
inference without knowing it. More broadly, reward learning in general has the
issue that it does not specify _whose_ reward to learn, and how to combine
different people’s values.

## Acknowledgements

We thank the members of the InterACT lab, and BAIR as a whole, at UC Berkeley
for their helpful discussions, as well as providing human demonstrations for
our experiments. We also thank anonymous reviewers for feedback on an early
version of this paper. This research is funded in part by the Office of Naval
Research Young Investigator Program, and UCSF Weill Institute for
Neurosciences. Additionally, KB was supported in part by a grant from Long-
Term Future Fund (LTFF).

## References

  * Brockman et al. (2016a) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym.  _arXiv preprint arXiv:1606.01540_ , 2016a.
  * Brockman et al. (2016b) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym, 2016b.
  * Cao et al. (2021) Haoyang Cao, Samuel N. Cohen, and Lukasz Szpruch.  Identifiability in inverse reinforcement learning.  _CoRR_ , abs/2106.03498, 2021.
  * Caramazza et al. (1981) Alfonso Caramazza, Michael McCloskey, and Bert Green.  Naive beliefs in “sophisticated” subjects: Misconceptions about trajectories of objects.  _Cognition_ , 9, 1981.
  * Chan et al. (2021) Lawrence Chan, Andrew Critch, and Anca Dragan.  Human irrationality: both bad and good for reward inference.  _preprint arXiv:2111.06956_ , 2021.
  * Do et al. (2008) Amy M Do, Alexander V Rupert, and George Wolford.  Evaluations of pleasurable experiences: The peak-end rule.  _Psychonomic Bulletin & Review_, 2008.
  * Evans et al. (2016) Owain Evans, Andreas Stuhlmueller, and Noah D. Goodman.  Learning the preferences of ignorant, inconsistent agents.  In _AAAI Conference on Artificial intelligence_ , 2016.
  * Finn et al. (2016) Chelsea Finn, Sergey Levine, and Pieter Abbeel.  Guided cost learning: Deep inverse optimal control via policy optimization.  In _International Conference on Machine Learning (ICML)_ , 2016.
  * Fu et al. (2018) J. Fu, K. Luo, and S. Levine.  Learning robust rewards with adversarial inverse reinforcement learning.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Fu et al. (2019a) Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine.  Diagnosing bottlenecks in deep q-learning algorithms.  In _Proceedings of the 36th International Conference on Machine Learning_. PMLR, 2019a.
  * Fu et al. (2019b) Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine.  Diagnosing bottlenecks in deep Q-learning algorithms.  _arXiv preprint arXiv:1902.10250_ , 2019b.
  * Gleave & Toyer (2022) Adam Gleave and Sam Toyer.  A primer on maximum causal entropy inverse reinforcement learning.  _preprint arXiv:2203.11409_ , 2022.
  * Gleave et al. (2021) Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike.  Quantifying differences in reward functions.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Grüne-Yanoff (2015) Till Grüne-Yanoff.  Models of temporal discounting 1937–2000: An interdisciplinary exchange between economics and psychology.  _Science in context_ , 2015.
  * Haarnoja et al. (2018) T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _arXiv_ , 2018.  URL https://arxiv.org/pdf/1801.01290.pdf.
  * Javdani et al. (2015) S. Javdani, S. Srinivasa, and J. A. Bagnell.  Shared autonomy via hindsight optimization.  In _Robotics: Science and Systems (RSS)_ , 2015.
  * Kahneman & Tversky (2013) Daniel Kahneman and Amos Tversky.  Prospect theory: An analysis of decision under risk.  _Handbook of the fundamentals of financial decision making: Part I_ , 2013.
  * Kim et al. (2021) Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon.  Reward identification in inverse reinforcement learning.  In _Proceedings of the 38th International Conference on Machine Learning_ , 2021.
  * Muelling et al. (2017) Katharina Muelling, Arun Venkatraman, Jean-Sebastien Valois, John E Downey, Jeffrey Weiss, Shervin Javdani, Martial Hebert, Andrew B Schwartz, Jennifer L Collinger, and J Andrew Bagnell.  Autonomy infused teleoperation with application to brain computer interface controlled manipulation.  _Autonomous Robots_ , 2017.
  * Ng & Russel (2000) Andrew Y. Ng and Stuart Russel.  Algorithms for inverse reinforcement learning.  In _International Conference on Machine Learning_ , 2000.
  * Owhadi et al. (2015) Houman Owhadi, Clint Scovel, and Tim Sullivan.  On the brittleness of bayesian inference.  _SIAM Review_ , 2015.
  * Ramachandran & Amir (2007) Deepak Ramachandran and Eyal Amir.  Bayesian inverse reinforcement learning.  In _IJCAI_ , 2007.
  * Reddy et al. (2014) Siddharth Reddy, Anca Dragan, and Sergey Levine.  Shared autonomy via deep reinforcement learning.  In _Robotics: Science and Systems (RSS)_ , 2014.
  * Reddy et al. (2018) Siddharth Reddy, Anca D. Dragan, and Sergey Levine.  Where do you think you’re going?: Inferring beliefs about dynamics from behavior.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Reddy et al. (2020) Siddharth Reddy, Sergey Levine, and Anca D. Dragan.  Assisted perception: Optimizing observations to communicate state.  _CoRR_ , abs/2008.02840, 2020.
  * Shah et al. (2019) Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca D. Dragan.  On the feasibility of learning, rather than assuming, human biases for reward inference.  In _International Conference on Machine Learning_ , 2019.
  * Sharot et al. (2007) Tali Sharot, Alison M Riccardi, Candace M Raio, and Elizabeth A Phelps.  Neural mechanisms mediating optimism bias.  _Nature_ , 2007.
  * Singh et al. (2018) Sumeet Singh, Jonathan Lacotte, Anirudha Majumdar, and Marco Pavone.  Risk-sensitive inverse reinforcement learning via semi- and non-parametric methods.  _The International Journal of Robotics Research_ , 2018.
  * Sprungk (2020) Björn Sprungk.  On the local lipschitz stability of bayesian inverse problems.  _arxiv preprint arxiv:1906.07120_ , 2020.
  * Thompson (1999) Suzanne C. Thompson.  Illusions of control: How we overestimate our personal influence.  _Current Directions in Psychological Science_ , 1999.
  * Vasquez et al. (2014) Dizan Vasquez, Billy Okal, and Kai O. Arras.  Inverse reinforcement learning algorithms and features for robot navigation in crowds: An experimental comparison.  In _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2014.
  * Wulfmeier et al. (2015) M. Wulfmeier, P. Ondruska, and I. Posner.  Maximum entropy deep inverse reinforcement learning.  In _Neural Information Processing Systems Conference, Deep Reinforcement Learning Workshop_ , 2015.
  * Ziebart et al. (2008) B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey.  Maximum entropy inverse reinforcement learning.  In _International Conference on Artificial Intelligence (AAAI)_ , 2008\.
  * Ziebart et al. (2010) B. D. Ziebart, J. A. Bagnell, and A. K. Dey.  Modeling interaction via the principle of maximum causal entropy.  In _International Conference on Machine Learning (ICML)_ , 2010.
  * Ziebart et al. (2009) Brian D Ziebart, Nathan Ratliff, Garratt Gallagher, Christoph Mertz, Kevin Peterson, J Andrew Bagnell, Martial Hebert, Anind K Dey, and Siddhartha Srinivasa.  Planning-based prediction for pedestrians.  In _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2009.

##  Appendix A Example where Lower-Bound Occurs

Let us consider a stochastic bandit with continuous actions
$\mathcal{A}\in[0,1]$. Since bandits consist of a single, stationary state, we
drop dependence on state in all quantities. The reward for choosing action
$a\in\mathcal{A}$ is $r(a;\theta)=a^{\theta}(1-a)^{1-\theta}\,,$ for some
parameter $\theta\in(0,1)$. When $\theta$ is close to $0$, the reward is
higher for actions close to $0$, and vice-versa when $\theta$ is close to $1$.

For simplicity, let us only consider a dataset of a single action $a_{1}$. Let
us consider a Boltzmann rational policy as the assumed model, namely
$\widetilde{\pi}(a;\theta)\propto\exp(r(a;\theta))$ and have the demonstrator
policy $\pi^{*}$ be an adversarial perturbation of $\widetilde{\pi}$ that
overestimates the reward of $a_{1}$:

$\displaystyle\pi^{*}(a;\theta)\propto\begin{cases}\exp(r(a;\theta))\left(\mathds{1}\\!\left\\{a\not\in(a_{1}-\frac{\delta}{2},a_{1}+\frac{\delta}{2})\right\\}+10^{9}\
\mathds{1}\\!\left\\{a\in(a_{1}-\frac{\delta}{2},a_{1}+\frac{\delta}{2})\right\\}\right)&\text{if
$\theta<0.001$}\\\ \exp(r(a;\theta))&\text{otherwise}\,,\end{cases}$

for some $\delta\in(0,1)$. The interpretation of this is that the human is
believed to be noisily optimal; however, the human actually overestimates the
value of an infinitesimal region centered at action $a_{1}$ only if $\theta$
is close to $0$. Note that
$d_{\pi}^{\mathsf{wc}}(\pi^{*},\widetilde{\pi})<c\delta$ for some constant
$c$, so we can choose $\delta$ such that the two policies are “close" to each
other. When $a_{1}=1$, we will infer $\widetilde{\theta}\approx 1$; however,
$\theta^{*}\approx 0$, leading to reward inference error equal to the range of
reward parameters.

##  Appendix B Example Where Log-Concavity is Violated

The environment is a $3\times 3$ gridworld with deterministic transitions and
discount $\gamma=1$. Let $s$ be the center cell, and $a$ be going up. In
Figure 9, we show that a natural policy that chooses “up" according to
$\pi(a\mid s;\theta)\propto\exp(\max(\theta,10-\theta))$ violates log-
concavity. The reason is that “up" is optimal for $\theta\in[0,4]\cup[6,10]$
but not in between.

Figure 9: Simple navigation environment where a near optimal policy violates
Assumption 2.

##  Appendix C Proofs

###  C.1 Proof of Theorem 2

Without loss of generality, let $\widetilde{\pi}$ satisfy
$\widetilde{\pi}(a\mid s;\theta)=\exp(\Phi(s,a;\theta))/Z(s;\theta)$. Note
that this parameterization can be used to express any probability
distribution. Also, for any $\delta>0$, let us define

$B_{\delta}(\mathcal{D})=\bigcup_{t=1}^{n}\left(a_{t}-\frac{\delta}{2},a_{t}+\frac{\delta}{2}\right)$

as a union of $\frac{\delta}{2}$-balls around the actions that appear in
dataset $\mathcal{D}$. Then, for any $\theta^{*}\in\Theta$, let $\pi^{*}$
satisfy

$\displaystyle\pi^{*}(a\mid
s;\theta)=\begin{cases}\frac{1}{Z^{\prime}(s;\theta)}\left(\exp(\Phi(s,a;\theta))\mathds{1}\\!\left\\{a\not\in
B_{\delta}(\mathcal{D})\right\\}+C\ \mathds{1}\\!\left\\{a\in
B_{\delta}(\mathcal{D})\right\\}\right)&\text{if $\theta=\theta^{*}$}\,,\\\
\frac{1}{Z(s;\theta)}\exp(\Phi(s,a;\theta))&\text{otherwise}\,,\end{cases}$

where $C$ is the supremum $C=\sup_{s,a,\theta}\exp(\Phi(s,a;\theta))$. By
construction, it is clear that MLE on $\mathcal{D}$ using $\pi^{*}$ would
yield reward parameter $\theta^{*}$. Since such $\pi^{*}$ can be constructed
for any $\theta^{*}$, we can choose $\theta^{*}$ that satisfies
$\|\widetilde{\theta}-\theta^{*}\|_{2}^{2}>\sup_{\theta,\theta^{\prime}}\|\theta-\theta^{\prime}\|_{2}/2$.
What remains is showing that there exists $\delta$ such that
$d_{\pi}^{\mathsf{wc}}(\pi^{*},\widetilde{\pi})<\varepsilon$.

#### Bounding the worst-case policy divergence.

Note that the worst-case divergence is necessarily satisfied at $\theta^{*}$.
Fix any state $s\in\mathcal{S}$. We have,

$\displaystyle D_{\text{KL}}(\pi^{*}(\cdot\mid
s;\theta^{*})||\widetilde{\pi}(\cdot\mid
s;\theta^{*}))=\int_{\mathcal{A}}\pi^{*}(a\mid
s;\theta^{*})\log\frac{\pi^{*}(a\mid s;\theta^{*})}{\widetilde{\pi}(a\mid
s;\theta^{*})}da$ $\displaystyle\qquad=\int_{\mathcal{A}\setminus
B_{\delta}(\mathcal{D})}\pi^{*}(a\mid s;\theta^{*})\log\frac{\pi^{*}(a\mid
s;\theta^{*})}{\widetilde{\pi}(a\mid
s;\theta^{*})}da+\int_{B_{\delta}(\mathcal{D})}\pi^{*}(a\mid
s;\theta^{*})\log\frac{\pi^{*}(a\mid s;\theta^{*})}{\widetilde{\pi}(a\mid
s;\theta^{*})}da\,.$

We consider each term individually. Starting with the first term, we have

$\displaystyle\int_{\mathcal{A}\setminus B_{\delta}(\mathcal{D})}\pi^{*}(a\mid
s;\theta^{*})\log\frac{\pi^{*}(a\mid s;\theta^{*})}{\widetilde{\pi}(a\mid
s;\theta^{*})}da$
$\displaystyle\leq\log\frac{Z(s;\theta^{*})}{Z^{\prime}(s;\theta^{*})}$
$\displaystyle\leq\frac{1}{Z(s;\theta^{*})}\left|Z(s;\theta^{*})-Z^{\prime}(s;\theta^{*})\right|\,,$

where we use that the policies only differ in their normalizers in the first
inequality, and that
$\log(t)-\log(s)\leq\frac{1}{\min\\{t,s\\}}\left|t-s\right|$ in the second.
Now, using that

$Z^{\prime}(s;\theta^{*})=\int_{\mathcal{A}\setminus
B_{\delta}(\mathcal{D})}\exp(\Phi(s,a,;\theta^{*})da+C\left|B_{\delta}(\mathcal{D})\right|\,,$

we have

$\displaystyle\int_{\mathcal{A}\setminus B_{\delta}(\mathcal{D})}\pi^{*}(a\mid
s;\theta^{*})\log\frac{\pi^{*}(a\mid s;\theta^{*})}{\widetilde{\pi}(a\mid
s;\theta^{*})}da\leq\frac{C}{Z(s;\theta^{*})}\
\left|B_{\delta}(\mathcal{D})\right|\,.$

Now, let us consider the second term. we have

$\displaystyle\int_{B_{\delta}(\mathcal{D})}\pi^{*}(a\mid
s;\theta^{*})\log\frac{\pi^{*}(a\mid s;\theta^{*})}{\widetilde{\pi}(a\mid
s;\theta^{*})}da$
$\displaystyle\leq\int_{B_{\delta}(\mathcal{D})}\pi^{*}(a\mid
s;\theta^{*})\left|\log{C}-\Phi(s,a;\theta^{*})\right|da+\log\frac{Z(s;\theta^{*})}{Z^{\prime}(s;\theta^{*})}$
$\displaystyle\leq
2\log{C}\left|B_{\delta}(\mathcal{D})\right|+\frac{C}{Z(s;\theta^{*})}\left|B_{\delta}(\mathcal{D})\right|\,,$

where we reuse the bound for the first term, and use that
$\left|\phi(s,a;\theta^{*})\right|\leq\log{C}$. Combining the two bounds
yields

$\displaystyle D_{\text{KL}}(\pi^{*}(\cdot\mid
s;\theta^{*})||\widetilde{\pi}(\cdot\mid s;\theta^{*}))\leq
2\log{C}\left|B_{\delta}(\mathcal{D})\right|+\frac{2C}{Z(s;\theta^{*})}\left|B_{\delta}(\mathcal{D})\right|\,.$

Using that $\left|B_{\delta}(\mathcal{D})\right|\leq n\delta$ by construction,
we can solve for $\delta=\mathcal{O}(\varepsilon/n)$ such that
$d_{\pi}^{\mathsf{wc}}(\pi^{*},\widetilde{\pi})<\varepsilon$, as desired. This
completes the proof.

###  C.2 Proof of Theorem 3

Recall that $L(\theta;\pi,\mathcal{D})$ is the negative log-likelihood of
demonstrations $\mathcal{D}$ under policy $\pi$ and reward parameters
$\theta$. Note that we can write

$\displaystyle
L(\theta;\widetilde{\pi},\mathcal{D})=\frac{1}{n}\sum_{t=1}^{n}-\log\widetilde{\pi}(a_{t}\mid
s_{t};\theta)$ $\displaystyle=\frac{1}{n}\sum_{t=1}^{n}-\log\pi^{*}(a_{t}\mid
s_{t};\theta)+\log\frac{\pi^{*}(a_{t}\mid
s_{t};\theta)}{\widetilde{\pi}(a_{t}\mid s_{t};\theta)}$
$\displaystyle=L(\theta;\pi^{*},\mathcal{D})+\frac{1}{n}\sum_{t=1}^{n}\log\frac{\pi^{*}(a_{t}\mid
s_{t};\theta)}{\widetilde{\pi}(a_{t}\mid s_{t};\theta)}\,.$

By the law of large numbers, we have that under expectation over dataset
$\mathcal{D}$,

$\displaystyle\mathbb{E}_{\mathcal{D}\sim\pi^{*}}\left[\frac{1}{n}\sum_{t=1}^{n}\log\frac{\pi^{*}(a_{t}\mid
s_{t};\theta)}{\widetilde{\pi}(a_{t}\mid
s_{t};\theta)}\right]=\mathbb{E}_{s\sim
d^{*}}\left[D_{\text{KL}}(\pi^{*}(\cdot\mid s;\theta^{*})\ ||\
\widetilde{\pi}(\cdot\mid s;\theta^{*}))\right]$

Using Assumption 2 on $\pi^{*}$, for any $\theta\in\Theta$, we also have

$\displaystyle L(\theta;\pi^{*},\mathcal{D})$ $\displaystyle\geq
L(\theta^{*};\pi^{*},\mathcal{D})+\nabla_{\theta}L(\theta^{*};\pi^{*},\mathcal{D})^{\top}(\theta^{*}-\theta)+\frac{cn}{2}\|\theta-\theta^{*}\|_{2}^{2}\,.$

By definition of $\theta^{*}$ and Assumption 1, we know that
$\nabla_{\theta}L(\theta^{*};\pi^{*},\mathcal{D})=\mathbf{0}$. Substituting
$\theta=\widetilde{\theta}$ and rearranging yields

$\displaystyle\|\widetilde{\theta}-\theta^{*}\|_{2}^{2}\leq\frac{2}{c}\,\left(L(\widetilde{\theta};\pi^{*},\mathcal{D})-L(\theta^{*};\pi^{*},\mathcal{D})\right)\,.$

Analogously, using Assumption 2 on $\widetilde{\pi}$ 111In the statement of
Assumption 2 in the main paper, we only assume log-concavity for $\pi^{*}$.
This will be corrected in a future revision to include both
$\pi^{*},\widetilde{\pi}$, we have that

$\displaystyle\|\widetilde{\theta}-\theta^{*}\|_{2}^{2}\leq\frac{2}{c}\,\left(L(\theta^{*};\widetilde{\pi},\mathcal{D})-L(\widetilde{\theta};\widetilde{\pi},\mathcal{D})\right)\,.$

Combining the two bounds yields,

$\displaystyle\|\widetilde{\theta}-\theta^{*}\|_{2}^{2}$
$\displaystyle\leq\frac{2}{c}\,\left(L(\theta^{*};\widetilde{\pi},\mathcal{D})-L(\theta^{*};\pi^{*},\mathcal{D})+L(\widetilde{\theta};\pi^{*},\mathcal{D})-L(\widetilde{\theta};\widetilde{\pi},\mathcal{D})\right)$
$\displaystyle\leq\frac{2}{c}\,\left(\frac{1}{n}\sum_{t=1}^{n}\log\frac{\pi^{*}(a_{t}\mid
s_{t};\theta^{*})}{\widetilde{\pi}(a_{t}\mid
s_{t};\theta^{*})}-\frac{1}{n}\sum_{t=1}^{n}\log\frac{\pi^{*}(a_{t}\mid
s_{t};\widetilde{\theta})}{\widetilde{\pi}(a_{t}\mid
s_{t};\widetilde{\theta})}\right)\,$

Taking an expectation over dataset $\mathcal{D}$ yields the desired result

$\displaystyle\mathbb{E}\left[\|\widetilde{\theta}-\theta^{*}\|_{2}^{2}\right]$
$\displaystyle\leq\frac{2}{c}\mathbb{E}_{s\sim
d^{*}}\left[D_{\text{KL}}(\pi^{*}(\cdot\mid s;\theta^{*})\ ||\
\widetilde{\pi}(\cdot\mid s;\theta^{*}))-D_{\text{KL}}(\pi^{*}(\cdot\mid
s;\widetilde{\theta})\ ||\ \widetilde{\pi}(\cdot\mid
s;\widetilde{\theta}))\right]$ $\displaystyle\leq\frac{2}{c}\mathbb{E}_{s\sim
d^{*}}\left[D_{\text{KL}}(\pi^{*}(\cdot\mid s;\theta^{*})\ ||\
\widetilde{\pi}(\cdot\mid s;\theta^{*}))\right]\,,$

which is the desired result.

###  C.3 Proof of Corollary 4

Recall that $\widetilde{\pi},\pi^{*}$ are parameterized by Q-values
$\widetilde{Q},Q^{*}$ that satisfy the soft Bellman update in (4). Fix state
$s$. We have

$\displaystyle D_{\text{KL}}(\pi^{*}(\cdot\mid
s;\theta^{*})||\widetilde{\pi}(\cdot\mid
s;\theta^{*}))=\mathbb{E}_{a\sim\pi^{*}(\cdot\mid
s;\theta^{*})}\left[Q^{*}(s,a;\theta^{*})-\widetilde{Q}(s,a;\theta^{*})\right]+\log\frac{\sum_{a^{\prime}}\exp(\widetilde{Q}(s,a^{\prime};\theta^{*}))}{\sum_{a^{\prime}}\exp(Q^{*}(s,a^{\prime};\theta^{*}))}\,.$

For any action $a$, we have

$\displaystyle Q^{*}(s,a;\theta^{*})-\widetilde{Q}(s,a;\theta^{*})$
$\displaystyle=\gamma\sum_{s^{\prime}}P^{*}(s^{\prime}\mid
s,a)V^{*}(s^{\prime};\theta^{*})-\gamma\sum_{s^{\prime}}\widetilde{P}(s^{\prime}\mid
s,a)\widetilde{V}(s^{\prime};\theta^{*})$
$\displaystyle=\gamma\sum_{s^{\prime}}P^{*}(s^{\prime}\mid
s,a)V^{*}(s^{\prime};\theta^{*})+\gamma\sum_{s^{\prime}}P^{*}(s^{\prime}\mid
s,a)\widetilde{V}(s^{\prime};\theta^{*})$
$\displaystyle\qquad-\gamma\sum_{s^{\prime}}P^{*}(s^{\prime}\mid
s,a)\widetilde{V}(s^{\prime};\theta^{*})-\gamma\sum_{s^{\prime}}\widetilde{P}(s^{\prime}\mid
s,a)\widetilde{V}(s^{\prime};\theta^{*})$
$\displaystyle=\gamma\|P^{*}(\cdot\mid s,a)-\widetilde{P}(\cdot\mid
s,a)\|_{1}\widetilde{V}(s^{\prime};\theta^{*})+\gamma\sum_{s^{\prime}}P^{*}(s^{\prime}\mid
s,a)(V^{*}(s^{\prime};\theta^{*})-\widetilde{V}(s^{\prime};\theta^{*}))$
$\displaystyle\leq\frac{R_{\max}}{1-\gamma}\Delta_{P}+\gamma\sum_{s^{\prime}}P^{*}(s^{\prime}\mid
s,a)\max_{a^{\prime}}\\{Q^{*}(s^{\prime},a^{\prime};\theta^{*})-\widetilde{Q}(s^{\prime},a^{\prime};\theta^{*})\\}$
$\displaystyle\leq\ldots$
$\displaystyle\leq\frac{R_{\max}}{(1-\gamma)^{2}}\Delta_{P}\,.$

Now, let us consider the normalization term. We have

$\displaystyle\log\frac{\sum_{a^{\prime}}\exp(\widetilde{Q}(s,a^{\prime};\theta^{*}))}{\sum_{a^{\prime}}\exp(Q^{*}(s,a^{\prime};\theta^{*}))}$
$\displaystyle\leq\frac{1}{\sum_{a^{\prime}}\exp(\widetilde{Q}(s,a^{\prime};\theta^{*}))}\
\sum_{a^{\prime}}\exp(\widetilde{Q}(s,a^{\prime};\theta^{*}))\log\frac{\exp(\widetilde{Q}(s,a^{\prime};\theta^{*}))}{\exp(Q^{*}(s,a^{\prime};\theta^{*}))}$
$\displaystyle\leq\sum_{a^{\prime}}(\widetilde{Q}(s,a^{\prime};\theta^{*})-Q^{*}(s,a^{\prime};\theta^{*}))$
$\displaystyle\leq\frac{|\mathcal{A}|R_{\max}}{(1-\gamma)^{2}}\Delta_{P}\,.$

Combining the two bounds and taking an expectation over $s$ yields the desired
result.

###  C.4 Proof of Corollary 5

The proof follows the format of the proof for Corollary 4. Fix state $s$. We
have

$\displaystyle D_{\text{KL}}(\pi^{*}(\cdot\mid
s;\theta^{*})||\widetilde{\pi}(\cdot\mid
s;\theta^{*}))=\mathbb{E}_{a\sim\pi^{*}(\cdot\mid
s;\theta^{*})}\left[Q^{*}(s,a;\theta^{*})-\widetilde{Q}(s,a;\theta^{*})\right]+\log\frac{\sum_{a^{\prime}}\exp(\widetilde{Q}(s,a^{\prime};\theta^{*}))}{\sum_{a^{\prime}}\exp(Q^{*}(s,a^{\prime};\theta^{*}))}\,.$

For any action $a$, we have

$\displaystyle Q^{*}(s,a;\theta^{*})-\widetilde{Q}(s,a;\theta^{*})$
$\displaystyle=\gamma^{*}\sum_{s^{\prime}}P(s^{\prime}\mid
s,a)V^{*}(s^{\prime};\theta^{*})-\widetilde{\gamma}\sum_{s^{\prime}}P(s^{\prime}\mid
s,a)\widetilde{V}(s^{\prime};\theta^{*})$
$\displaystyle=\gamma^{*}\sum_{s^{\prime}}P(s^{\prime}\mid
s,a)V^{*}(s^{\prime};\theta^{*})+\gamma^{*}\sum_{s^{\prime}}P(s^{\prime}\mid
s,a)\widetilde{V}(s^{\prime};\theta^{*})$
$\displaystyle\qquad-\gamma^{*}\sum_{s^{\prime}}P(s^{\prime}\mid
s,a)\widetilde{V}(s^{\prime};\theta^{*})-\widetilde{\gamma}\sum_{s^{\prime}}P(s^{\prime}\mid
s,a)\widetilde{V}(s^{\prime};\theta^{*})$
$\displaystyle=(\gamma^{*}-\widetilde{\gamma})\sum_{s^{\prime}}P(s^{\prime}\mid
s,a)\widetilde{V}(s^{\prime};\theta^{*})+\gamma^{*}\sum_{s^{\prime}}P(s^{\prime}\mid
s,a)(V^{*}(s^{\prime};\theta^{*})-\widetilde{V}(s^{\prime};\theta^{*}))$
$\displaystyle\leq\frac{R_{\max}}{1-\widetilde{\gamma}}\left|\gamma^{*}-\widetilde{\gamma}\right|+\gamma^{*}\sum_{s^{\prime}}P(s^{\prime}\mid
s,a)\max_{a^{\prime}}\\{Q^{*}(s^{\prime},a^{\prime};\theta^{*})-\widetilde{Q}(s^{\prime},a^{\prime};\theta^{*})\\}$
$\displaystyle\leq\ldots$
$\displaystyle\leq\frac{R_{\max}}{(1-\widetilde{\gamma})(1-\gamma^{*})}\left|\gamma^{*}-\widetilde{\gamma}\right|\,.$

Now, let us consider the normalization term. We have

$\displaystyle\log\frac{\sum_{a^{\prime}}\exp(\widetilde{Q}(s,a^{\prime};\theta^{*}))}{\sum_{a^{\prime}}\exp(Q^{*}(s,a^{\prime};\theta^{*}))}$
$\displaystyle\leq\frac{1}{\sum_{a^{\prime}}\exp(\widetilde{Q}(s,a^{\prime};\theta^{*}))}\
\sum_{a^{\prime}}\exp(\widetilde{Q}(s,a^{\prime};\theta^{*}))\log\frac{\exp(\widetilde{Q}(s,a^{\prime};\theta^{*}))}{\exp(Q^{*}(s,a^{\prime};\theta^{*}))}$
$\displaystyle\leq\sum_{a^{\prime}}(\widetilde{Q}(s,a^{\prime};\theta^{*})-Q^{*}(s,a^{\prime};\theta^{*}))$
$\displaystyle\leq\frac{|\mathcal{A}|R_{\max}}{(1-\widetilde{\gamma})(1-\gamma)}\left|\widetilde{\gamma}-\gamma^{*}\right|\,.$

Combining the two bounds yields the desired result.

##  Appendix D Experiment Details

###  D.1 Tabular experiments

Figure 10: Visualization of gridworld policies (as state-visitation
distributions) with (a) different transition biases (probability $p$ of
unintended transitions), and (b) different discount factors.

#### Environment and training details

Recall that the gridworld environments we consider are described by $8\times
4$ grids, with a start and goal state, and walls, lava, and exactly one
waypoint state placed in between. We consider a sparse reward where the agent
earns a reward of $\theta=3$ upon reaching the goal state. Alternatively, if
the agent reaches a lava or waypoint state, then its reward is $0$ or $1$,
respectively, for the rest of the trajectory. The agent is able to move in
either of the four direction (or choose to stay still), and there is a
$p=30\%$ chance that the agent travels in a different direction than
commanded. We choose $\gamma^{*}=0.98$ high enough that the goal state is
preferred over the closer waypoint state under the optimal policy.

A reward-conditioned policy (model or demonstrator) under each environment is
given by $\pi(a\mid s;\theta)\propto\exp(Q(s,a;\theta))$, where
$Q(s,a;\theta)$ were derived by value iteration using the an MDP model (can be
the true underlying MDP or a biased one) of the environment. During reward
inference, we discretize the reward parameter space $\Theta=[1,4]$ with
resolution $64$. Because the environment is tabular, instead of sampling
demonstrations $\mathcal{D}$ from $\pi$, we can instead compute $w^{\pi}$ the
discounted stationary distribution. Specifically, let $\rho$ be the
distribution of the starting state (which in our case, is an indicator vector
at the start state of each environment), then $w^{\pi}$ satisfies:

$\displaystyle
w^{\pi}(s)=(1-\gamma)\rho(s)+\gamma^{*}\sum_{s^{\prime}\in\mathcal{S}}\sum_{a^{\prime}\in\mathcal{A}}w^{\pi}(s^{\prime})\pi(a^{\prime}\mid
s^{\prime})P(s\mid s^{\prime},a^{\prime})\,.$

We can use this to solve for the true state visitations $w^{*}$ for any
demonstrator policy $\pi^{*}$, which can be used to compute the weighted
policy divergence as in (3) without explicitly sampling a dataset
$\mathcal{D}$ of demonstrations.

#### Visualization of biased policies

In Figure 10, we visualize the demonstrator policies $\pi^{*}$ under the
systematic biases considered. We see that in Figure 10(a), when the
demonstrator underestimates the probability of unintended transitions, it
heavily prefers the goal state, which has higher reward but is much more
dangerous to reach, over the waypoint state Conversely, in Figure 10(b), when
the demonstrator underestimates the discount factor, they strongly prefer the
waypoint state that yields lower reward but is much closer.

###  D.2 Continuous control experiments

Figure 11: Visualization of Lunar Lander trajectories for policies with (a)
biased internal dynamics that underestimate left-right acceleration and (b)
correct internal dynamics.

#### Environment and training details

Recall that the domain we consider is the Lunar Lander game, where an agent
needs to navigate a lander onto the landing pad. The reward function yields a
large reward for landing on the pad, and a penalty for crashing or going out
of bounds. The magnitude of the reward depends on the speed and tilt of the
lander upon reaching the landing pad. The physics of the game are
deterministic. The reward parameter $\theta\in[0,1]$ we try to infer is the
location of the landing pad (expressed as normalized horizontal displacement).

In this domain, we train a reward-conditioned policy $\pi(a\mid s;\theta)$ by
folding the reward parameter $\theta$ into the state representation, which is
an $8$-dimensional vector capturing the lander’s current location, velocity,
and tilt. The policy is parameterized as a $3$-layer fully-connected neural
network with hidden dimension of $128$, and outputs a squashed Gaussian
distribution over actions. Because the state and action space are continuous,
we use soft-actor-critic (SAC) (Haarnoja et al., 2018) with fixed entropy
regularization $\alpha=1$. We train the policy for $600$ episodes of length at
most $1,000$, with a batch size of $264$, until the policy was able to land on
the landing pad with a high success rate. During reward inference, we
discretize the reward parameter space $\Theta=[0,1]$ with resolution $32$. We
sample datasets $\mathcal{D}$ consisting of $10,000$ observations, and report
the mean and standard error of policy and reward error across $10$ independent
samples of datasets.

#### Visualization of biased policies

In Figure 11, we visualize the demonstrator policies $\pi^{*}$ under different
degrees of internal dynamics bias. Recall that parameter $p$ describes how
much one unit of power will increase acceleration in the left-right
directions. When $p$ is underestimated, the policy will not move right enough
to reach the landing pad; in contrast, when $p$ is properly estimated, the
policy will reach the landing pad with a high success rate.

###  D.3 Additional experiment with human policies

In line with the experiments with simulated biases, we run an additional
experiment similar to the one in Section 6.3, where we instead keep
$\widetilde{\pi}$ fixed as the optimal policy, and interpolate between the
optimal policy and the real human policy for demonstrator policy $\pi^{*}$. We
show the effect of the interpolation proportion on policy divergence and
reward inference error in Figure 12. Again, we notice the consistent message
that policy error bounds reward error.

Figure 12: Effect of human bias (measured by probability of acting according
to human policy) on (a) weighted policy divergence and (b) reward inference
error on discrete Lunar Lander environments. In (c), we show a scatter plot of
the policy and reward errors for fixed probabilities.


aainstitutetext: Department of Physics, Syracuse University, Syracuse, NY
13244, USA bbinstitutetext: Department of Physics and Astronomy, RPI, Troy, NY
12180, USA

# Holography from lattice ${\cal N}=4$ super Yang-Mills

Simon Catterall b Joel Giedt a Goksu Can Toga

###### Abstract

In this paper we use lattice simulation to study four dimensional ${\cal N}=4$
super Yang-Mills (SYM) theory. We have focused on the three color theory on
lattices of size $12^{4}$ and for ’t Hooft couplings up to $\lambda=40.0$. Our
lattice action is based on a discretization of the Marcus or GL twist of
${\cal N}=4$ SYM and retains one exact supersymmetry for non-zero lattice
spacing. We show that lattice theory exists in a single non-Abelian Coulomb
phase for all ’t Hooft couplings. Furthermore the static potential we obtain
from correlators of Polyakov lines is in good agreement with that obtained
from holography - specifically the potential has a Coulombic form with a
coefficent that varies as the square root of the ’t Hooft coupling.

††preprint: preprint

##  1 Introduction

${\cal N}=4$ super Yang-Mills is both a fascinating and non-trivial quantum
field theory. It possesses a line of conformal fixed points, is conjectured to
be invariant under a strong-weak coupling duality, and most importantly,
furnishes the original example of holographic duality by providing a
description of type IIb string theory on five dimensional anti-de Sitter
space. The holographic description is most easily understood in the planar
$N_{c}\to\infty$ strong coupling limit where the five dimensional theory
reduces to classical supergravity. However, string loop corrections arise at
$O(\frac{1}{N_{c}})$ and are difficult to access analytically. This provides
motivation to study the theory using numerical simulation.

Naive approaches to constructing a lattice theory break supersymmetry
completely and lead to a large number of relevant supersymmetry breaking
counterterms whose couplings would need to be tuned to take a continuum limit.
This stymied progress for many years until models were constructed that
preserved one or more supercharges at non-zero lattice spacing - see the
review Catterall:2009it  and references therein. The key idea underlying these
constructions is to find linear combinations of the continuum supercharges
that are nilpotent and hence compatible with finite lattice translations. This
may be accomplished either by discretization of a topologically twisted
version of the supersymmetric theory Catterall:2007kn  or by building a
lattice theory from a matrix model using orbifolding and deconstruction
techniques Cohen:2003xe ; Cohen:2003qw ; Kaplan:2005ta .

While these developments have led to many numerical studies providing evidence
supporting holography for dimensional reductions of ${\cal N}=4$ Yang-Mills
Hanada:2016zxj ; Anagnostopoulos:2007fw ; Hanada:2008gy ; Catterall:2008yz ;
Catterall:2009xn ; Catterall:2010fx ; Berkowitz:2016jlq ; Catterall:2017lub ;
Rinaldi:2017mjl ; Catterall:2020nmn  it has proven difficult until recently to
test holography directly in four dimensions. The original supersymmetric
construction in four dimensions produced lattice artifacts in the form of
$U(1)$ monopoles that condense and lead to a chirally broken phase for ’t
Hooft couplings $\lambda>4$ Catterall:2012yq ; Catterall:2014vka . Recently we
have constructed a new lattice action that appears to avoid these problems
Catterall:2020lsi . It differs from both the original and improved
Catterall:2015ira  actions for ${\cal N}=4$ SYM by the addition of a new
supersymmetric term that breaks the gauge symmetry from $U(N)$ to $SU(N)$.
This removes the monopoles completely and, as we will show in this paper,
yields a single non-Abelian Coulomb phase.

##  2 Lattice action

We use the supersymmetric lattice action appearing in Catterall:2020lsi .

$S=\frac{N}{4\lambda}{\cal Q}\sum_{x}{\rm Tr\;}\left[\chi_{ab}{\cal
F}_{ab}+\eta\left({\overline{\cal D}}_{a}{\cal U}_{a}+\kappa\left({\rm
Re\,det\,}{\cal U}_{a}(x)-1\right)I_{N}\right)+\frac{1}{2}\eta d\right]+S_{\rm
closed}$ (1)

where the lattice field strength

${\cal F}_{ab}(x)={\cal U}_{a}(x){\cal U}_{b}(x+\hat{a})-{\cal U}_{b}(x){\cal
U}_{a}(x+\hat{b})$ (2)

where ${\cal U}_{a}(x)$ denotes the complexified gauge field living on the
lattice link running from $x\to x+\hat{a}$ and $\hat{a}$ denotes one of the
five basis vectors of an underlying $A_{4}^{*}$ lattice. $I_{N}$ denotes the
$N\times N$ unit matrix in color space. Similarly

${\overline{\cal D}}_{a}{\cal U}_{a}={\cal U}_{a}(x){\overline{\cal
U}}_{a}(x)-{\overline{\cal U}}_{a}(x-\hat{a}){\cal U}_{a}(x-\hat{a}).$ (3)

The five fermion fields $\psi_{a}$, being superpartners of the (complex) gauge
fields, live on the corresponding links, while the ten fermion fields
$\chi_{ab}(x)$ are associated with new face links running from
$x+\hat{a}+\hat{b}\to x$. The scalar fermion $\eta(x)$ lives on the lattice
site $x$ and is associated with a conserved supercharge ${\cal Q}$ which acts
on the fields in the following way

$\displaystyle{\cal Q}\,{\cal U}_{a}$ $\displaystyle\to\psi_{a}$
$\displaystyle{\cal Q}\,\psi_{a}$ $\displaystyle\to 0$ $\displaystyle{\cal
Q}\,\eta$ $\displaystyle\to d$ $\displaystyle{\cal Q}\,d$ $\displaystyle\to 0$
$\displaystyle{\cal Q}\,\chi_{ab}$ $\displaystyle\to{\overline{\cal F}}_{ab}$
$\displaystyle{\cal Q}\,{\overline{\cal U}}_{a}$ $\displaystyle\to 0$ (4)

Notice that ${\cal Q}^{2}=0$ which guarantees the supersymmetric invariance of
the first part of the lattice action. The auxiliary site field $d(x)$ is
needed for nilpotency of ${\cal Q}$ offshell. The second term $S_{\rm closed}$
is given by

$S_{\rm closed}=-\frac{N}{16\lambda}\sum_{x}{\rm
Tr\;}\epsilon_{abcde}\chi_{ab}{\overline{\cal D}}_{c}\chi_{de}$ (5)

where the covariant difference operator acting on the fermion field
$\chi_{de}$ takes the form

${\overline{\cal D}}_{c}\chi_{de}(x)={\overline{\cal
U}}_{c}(x-\hat{c})\chi_{de}(x+\hat{a}+\hat{b})-\chi_{de}(x-\hat{d}-\hat{e}){\overline{\cal
U}}_{c}(x+\hat{a}+\hat{b})$ (6)

The latter term can be shown to be supersymmetric via an exact lattice Bianchi
identity $\epsilon_{abcde}{\overline{\cal D}}_{c}{\overline{\cal F}}_{de}=0$.
Carrying out the ${\cal Q}$ variation and integrating out the auxiliary field
$d$ we obtain the final supersymmetric lattice action $S=S_{b}+S_{f}$ where

$S_{b}=\frac{N}{4\lambda}\sum_{x}{\rm Tr\;}\left({\cal F}_{ab}{\overline{\cal
F}}_{ab}\right)+\frac{1}{2}{\rm Tr\;}({\overline{\cal D}}_{a}{\cal
U}_{a}+\kappa({\rm Re\,det\,}{\cal U}_{a}-1)I_{N})^{2}$ (7)

and

$\displaystyle S_{f}$ $\displaystyle=-\frac{N}{4\lambda}\sum_{x}\left({\rm
Tr\;}\chi_{ab}{\cal D}_{\left[a\right.}\psi_{\left.b\right]}+{\rm
Tr\;}\eta{\overline{\cal D}}_{a}\psi_{a}+\frac{1}{4}{\rm
Tr\;}\epsilon_{abcde}\chi_{ab}{\overline{\cal D}}_{c}\chi_{de}\right)-$ (8)
$\displaystyle\frac{\kappa N}{8\lambda}\sum_{x,a}{\rm Tr\;}(\eta)\,{\rm
det\,}{\cal U}_{a}(x){\rm Tr\;}({\cal U}_{a}^{-1}(x)\psi_{a}(x))$ (9)

Setting $\kappa=0$ and taking the naive continuum limit ${\cal U}_{a}=I+{\cal
A}_{a}+\ldots$ one can show that this action can be obtained by discretization
of the Marcus or GL twist of ${\cal N}=4$ Yang-Mills in flat space
Marcus:1995mq ; Kapustin:2006pk . In the continuum this twisted formulation is
used to construct a topological field theory but here we use the twisted
construction merely as a change of variables that allows for discretization
while preserving the single exact supersymmetry ${\cal Q}$.

As described above, the discrete theory is defined on a somewhat exotic
lattice - $A_{4}^{*}$. This admits a larger set of discrete rotational
symmetries corresponding to the group $S^{5}$ in comparison to those of a
hypercubic lattice and this fact plays a role in controlling the
renormalization of the theory Catterall:2013roa . In fact one can show that
gauge symmetry, ${\cal Q}$ supersymmetry and $S^{5}$ invariance ensure that
the only relevant counterterms correspond to operators present in the
classical lattice action together with a single new marginal operator of the
form $\alpha\sum_{x,a}{\rm Tr}\,(\eta{\cal U}_{a}{\overline{\cal U}}_{a})$.
However, the calculation in Catterall:2011pd  shows that even this term is
absent at $\kappa=0$ to all orders in perturbation theory.

Finally, to retain exact supersymmetry, all fields reside in the algebra of
the gauge group – taking their values in the adjoint representation of $U(N)$:
$f(x)=\sum_{A=1}^{N^{2}}T^{A}f^{A}(x)$ with ${\rm
Tr\;}(T^{A}T^{B})=-\delta^{AB}$. Ordinarily this would be incompatible with
lattice gauge invariance because the measure would not be gauge invariant for
link based fields. However, in this ${\cal N}=4$ construction the problem is
evaded since the fields are complexified which ensures that the Jacobians that
arise after gauge transformation of ${\cal U}$ and ${\overline{\cal U}}$
cancel.

The term involving the coupling $\kappa$ suppresses the troublesome $U(1)$
modes while leaving the important $SU(N)$ gauge symmetry intact. It also has
another advantage; by selecting out gauge fields with unit determinant it
ensures that the gauge field has the expansion ${\cal U}_{a}(x)=I+{\cal
A}_{a}(x)+\ldots$ where ${\cal A}_{a}$ are traceless complex fields. This
ensures the correct naive continuum limit.

The resultant action still possesses a set of flat directions corresponding to
constant gauge fields that are valued in the Cartan subalgebra. To regulate
these flat directions we additionally add to the action a term of the form

$S_{\rm mass}=\mu^{2}\sum_{x}{\rm Tr\;}\left({\overline{\cal U}}_{a}(x){\cal
U}_{a}(x)-I\right)^{2}$ (10)

While this breaks the exact supersymmetry softly all counter terms induced by
this breaking will have couplings that are multiplicative in $\mu^{2}$ and
hence vanishing as $\mu^{2}\to 0$.

##  3 Results

Our simulations utilize the rational hybrid Monte Carlo (HMC) algorithm where
the Pfaffian resulting from the fermion integration is replaced by

${\rm Pf}\,(M)={\rm det}\left[\left(M^{\dagger}M\right)^{\frac{1}{4}}\right]$
(11)

where $M$ is the fermion operator and the fractional power is replaced by a
rational fraction approximation Clark:2006wq . In principle this throws away
any phase in the fermion operator. In the appendix (fig. 6) we show that this
phase is always small even at strong coupling for sufficiently small
$\mu^{2}$. Typical ensembles used in our analysis consist of $3500-4000$ HMC
trajectories with $750-800$ discarded for thermalization. Errors are assessed
using a jackknife procedure using $20-40$ bins. We also fix $\kappa=1$ for all
our simulations.

Figure 1:  Bosonic Action vs $\lambda$ for $8^{4}$ lattice at $\kappa=1.0$ for
various values of $\mu$

One of the simplest observables that can be measured is the expectation value
of the bosonic action. This can be calculated exactly at any coupling by
exploiting the exact lattice supersymmetry. We start by rescaling the fermions
to remove the dependence of the ${\cal Q}$-closed term on
$\beta=\frac{N}{4\lambda}$ so that the partition function $Z$ is given by

$Z=\beta^{\frac{-6N^{2}V}{2}}\int D\eta D\psi D\chi DUD{\overline{\cal
U}}Dd\;e^{-\beta{\cal Q}\Lambda-\chi{\overline{\cal D}}\chi}$ (12)

where $\Lambda$ generates the ${\cal Q}$-exact terms in the action and $V$ is
the number of lattice points. After integrating over the auxiliary field $d$
one finds that the coefficient $-\frac{6}{2}N^{2}V$ is shifted to
$-\frac{7}{2}N^{2}V$. But

$-\frac{\partial\ln{Z}}{\partial\beta}=<S_{B}>-<S_{F}>=-\frac{7N^{2}V}{2\beta}+<{\cal
Q}\Lambda>$ (13)

Using the ${\cal Q}$ Ward identity $<{\cal Q}\Lambda>=0$ and the fact that the
expectation value of the fermion action can be trivially found by a scaling
argument since the fermion fields appear only quadratically one finds the
final result

$\frac{1}{V}\beta S_{B}=\frac{9N^{2}}{2}$ (14)

Fig.1 shows the bosonic action density as a function of $\lambda$ for several
values of the susy breaking mass $\mu$. It should be clear that the measured
values approach the exact result $<\beta\frac{S_{B}}{V}>=40.5$ independent of
$\lambda$ as $\mu\to 0$.

To check that we have indeed suppressed the $U(1)$ modes we plot the
expectation value of the link determinant in fig. 2. Clearly the observed
value of the link determinant lies close to unity for all ’t Hooft couplings
provided that $\mu$ is small enough. Notice that both the bosonic action and
the link determinant show no sign of a phase transition over the range
$0<\lambda<40$. This is consistent with the continuum expectation that the
${\cal N}=4$ SYM theory exists in a single phase out to arbitrarily strong
coupling.

Figure 2:  Link Determinant vs $\lambda$ for $8^{4}$ lattice and $\kappa=1.0$

The conclusion is strengthened further by examining a variety of
supersymmetric Wilson loop corresponding to the product of complexified
lattice gauge fields around a closed loop on the lattice. We have plotted the
logarithm of such Wilson loops as a function of $\sqrt{\lambda}$ in fig. 3.
Clearly the behavior is smooth as $\lambda$ varies and again there is no sign
of any phase transition. Furthermore the linear dependence of $W$ seen for
large $\lambda$ is consistent with holography. Indeed, both square and
circular Wilson loops can be computed in the strong coupling planar limit and
show a $\sqrt{\lambda}$ dependence on the ’t Hooft coupling Maldacena:1998im ;
Drukker:2000rr .

Figure 3:  $R\times R$ Wilson Loops vs $\sqrt{\lambda}$ for $8^{4}$ lattices
at $\mu=0.025$ and $\kappa=1.0$

It should be noted that this $\lambda$ dependence cannot be seen in
perturbation theory and constitutes a non-trivial test that the lattice model
is able to reproduce the non-perturbative physics of the continuum theory.

However fig. 3 makes it clear that there is also a perimeter dependence to the
Wilson loop. This is not unexpected and arises also in the continuum
calculations as a regulator term associated with a bare quark mass. In general
the Wilson loop arises as the amplitude for the propagation of heavy
fundamental sources that interact via a static potential of the form

$V(r)=-\frac{\alpha}{r}+M$ (15)

where $M$ represents the static quark mass. To subtract the perimeter term
from our analysis and look for the presence of an underlying non-abelian
Coulomb term in the static potential we have turned to another related
observable – the correlation function of two Polyakov lines. These are just
Wilson lines that close via the toroidal boundary conditions. The measured
correlator is defined by

$P(r)=\sum_{x,y}\left[<P(x)P^{\dagger}(y)>-<P(x)><P^{\dagger}(y)>\delta(r,|x-y|)\right]$
(16)

where $|x-y|$ is the distance in the $A_{4}^{*}$ lattice. As for the Wilson
loop it is expected to vary like

$P(r)\sim e^{-V(r)t}$ (17)

with $V(r)$ the static potential.

Figure 4:  $-\ln{<P(x)P(y)>}$ vs $R$ for $12^{4}$ lattices at $\mu=0.05$ and
$\kappa=1.0$

In practice we have computed this correlator on ensembles of smeared Polyakov
lines. Smearing the lattice gauge fields is a procedure for replacing each of
the lattice gauge links by an average over neighboring link paths or staples.
This smearing procedure has the effect of reducing U.V effects and increasing
the signal to noise ratio in observables that depend on the gauge field. It
also suppresses the contribution of any bare quark mass. We have used the APE
smearing procedure which replaces the link fields ${\cal U}$ in the following
way

${\cal U}_{a}(x)^{(N_{smear})}=(1-\alpha){\cal
U}_{a}(x)^{(N_{smear}-1)}+\frac{\alpha}{8(1-\alpha)}{\cal
S}_{a}(x)^{(N_{smear}-1)}$ (18)

where ${\cal S}$ denotes the sum over directional staples, $N_{smear}$ denotes
the number of iterative smearing steps and $\alpha$ is the smearing
coefficient albanese1987glueball .

We have fitted the correlator for a range of smearing parameters at each value
of $\lambda$ assuming a non-abelian Coulomb form for $V(r)$ – see the tables
1-3 in the appendix. In practice we find that $N_{smear}=5$ and
$0.45\leq\alpha\leq 0.55$ yields good, robust fits to the data over the range
$2<r<5$ on a $12^{4}$ lattice. Fig. 4 shows the logarithm of this correlator
for a lattice of size $L=12^{4}$ with $\mu=0.05$ with
$N_{smear}=5,\alpha=0.50$.

Taking the coefficients from these Coulomb fits and plotting them as a
function of $\sqrt{\lambda}$ we again see a linear dependence on
$\sqrt{\lambda}$ which is consistent with the holographic expectation
Chu:2009qt  . Indeed, even the numerical coefficient in the fit lies within
$10\%$ or so of the holographic prediction which can be seen in fig.5. Note
that the holographic prediction has been expressed in terms of the lattice
coupling $\lambda$ and not the continuum coupling. In the appendix B we
include equivalent fits (figs.10 and 10) for a range of different smearing
parameters thereby verifying that the agreement with the holographic
prediction is robust.

Figure 5:  Coefficient of the $1/r$ vs $\sqrt{\lambda}$ for $12^{4}$ lattices
at $\mu=0.05$

##  4 Conclusions

We have studied a new supersymmetric lattice action for ${\rm N}=4$ super
Yang-Mills in four dimensions at strong ’t Hooft coupling. We have focused on
the case of three colors $N=3$ and utilized lattices as large as $12^{4}$.
Correlators of (smeared) Polyakov lines show that the static potential
exhibits a non-Abelian Coulomb form $V(r)=\frac{\alpha\sqrt{\lambda}}{r}$
where the value of $\alpha$ and the square root dependence on the ’t Hooft
coupling match expectations from holography.

###### Acknowledgements.

This work was supported by the US Department of Energy (DOE), Office of
Science, Office of High Energy Physics, under Award Numbers DE-SC0009998
(SC,GT) and DE-SC0013496 (JG). Numerical calculations were carried out on the
DOE-funded USQCD facilities at Fermilab and NSF-funded ACCESS SDSC-Expanse
facilities under Award number PHY170035.

## References

  * (1) S. Catterall, D. B. Kaplan and M. Unsal, _Exact lattice supersymmetry_ , _Phys. Rept._ 484 (2009) 71 [0903.4881].
  * (2) S. Catterall, _From Twisted Supersymmetry to Orbifold Lattices_ , _JHEP_ 0801 (2008) 048 [0712.2532].
  * (3) A. G. Cohen, D. B. Kaplan, E. Katz and M. Unsal, _Supersymmetry on a Euclidean space-time lattice. 1. A Target theory with four supercharges_ , _JHEP_ 08 (2003) 024 [hep-lat/0302017].
  * (4) A. G. Cohen, D. B. Kaplan, E. Katz and M. Unsal, _Supersymmetry on a Euclidean space-time lattice. 2. Target theories with eight supercharges_ , _JHEP_ 12 (2003) 031 [hep-lat/0307012].
  * (5) D. B. Kaplan and M. Unsal, _A Euclidean lattice construction of supersymmetric Yang-Mills theories with sixteen supercharges_ , _JHEP_ 09 (2005) 042 [hep-lat/0503039].
  * (6) M. Hanada, Y. Hyakutake, G. Ishiki and J. Nishimura, _Numerical tests of the gauge/gravity duality conjecture for D0-branes at finite temperature and finite N_ , _Phys. Rev. D_ 94 (2016) 086010 [1603.00538].
  * (7) K. N. Anagnostopoulos, M. Hanada, J. Nishimura and S. Takeuchi, _Monte Carlo studies of supersymmetric matrix quantum mechanics with sixteen supercharges at finite temperature_ , _Phys. Rev. Lett._ 100 (2008) 021601 [0707.4454].
  * (8) M. Hanada, A. Miwa, J. Nishimura and S. Takeuchi, _Schwarzschild radius from Monte Carlo calculation of the Wilson loop in supersymmetric matrix quantum mechanics_ , _Phys. Rev. Lett._ 102 (2009) 181602 [0811.2081].
  * (9) S. Catterall and T. Wiseman, _Black hole thermodynamics from simulations of lattice Yang-Mills theory_ , _Phys. Rev. D_ 78 (2008) 041502 [0803.4273].
  * (10) S. Catterall and T. Wiseman, _Extracting black hole physics from the lattice_ , _JHEP_ 04 (2010) 077 [0909.4947].
  * (11) S. Catterall, A. Joseph and T. Wiseman, _Thermal phases of D1-branes on a circle from lattice super Yang-Mills_ , _JHEP_ 12 (2010) 022 [1008.4964].
  * (12) E. Berkowitz, E. Rinaldi, M. Hanada, G. Ishiki, S. Shimasaki and P. Vranas, _Precision lattice test of the gauge/gravity duality at large- $N$_, _Phys. Rev. D_ 94 (2016) 094501 [1606.04951].
  * (13) S. Catterall, R. G. Jha, D. Schaich and T. Wiseman, _Testing holography using lattice super-Yang-Mills theory on a 2-torus_ , _Phys. Rev. D_ 97 (2018) 086020 [1709.07025].
  * (14) E. Rinaldi, E. Berkowitz, M. Hanada, J. Maltz and P. Vranas, _Toward Holographic Reconstruction of Bulk Geometry from Lattice Simulations_ , _JHEP_ 02 (2018) 042 [1709.01932].
  * (15) S. Catterall, J. Giedt, R. G. Jha, D. Schaich and T. Wiseman, _Three-dimensional super-Yang–Mills theory on the lattice and dual black branes_ , _Phys. Rev. D_ 102 (2020) 106009 [2010.00026].
  * (16) S. Catterall, P. H. Damgaard, T. Degrand, R. Galvez and D. Mehta, _Phase Structure of Lattice N=4 Super Yang-Mills_ , _JHEP_ 1211 (2012) 072 [1209.5285].
  * (17) S. Catterall, D. Schaich, P. H. Damgaard, T. DeGrand and J. Giedt, _N=4 Supersymmetry on a Space-Time Lattice_ , _Phys. Rev. D_ 90 (2014) 065013 [1405.0644].
  * (18) S. Catterall, J. Giedt and G. C. Toga, _Lattice $\mathcal{N}$ = 4 super Yang-Mills at strong coupling_, _JHEP_ 12 (2020) 140 [2009.07334].
  * (19) S. Catterall and D. Schaich, _Lifting flat directions in lattice supersymmetry_ , _JHEP_ 07 (2015) 057 [1505.03135].
  * (20) N. Marcus, _The Other topological twisting of N=4 Yang-Mills_ , _Nucl. Phys._ B452 (1995) 331 [hep-th/9506002].
  * (21) A. Kapustin and E. Witten, _Electric-Magnetic Duality And The Geometric Langlands Program_ , _Commun. Num. Theor. Phys._ 1 (2007) 1 [hep-th/0604151].
  * (22) S. Catterall, J. Giedt and A. Joseph, _Twisted supersymmetries in lattice ${\cal N}=4$ super Yang-Mills theory_, _JHEP_ 1310 (2013) 166 [1306.3891].
  * (23) S. Catterall, E. Dzienkowski, J. Giedt, A. Joseph and R. Wells, _Perturbative renormalization of lattice N=4 super Yang-Mills theory_ , _JHEP_ 1104 (2011) 074 [1102.1725].
  * (24) M. A. Clark, _The Rational Hybrid Monte Carlo Algorithm_ , _PoS_ LAT2006 (2006) 004 [hep-lat/0610048].
  * (25) J. M. Maldacena, _Wilson loops in large N field theories_ , _Phys. Rev. Lett._ 80 (1998) 4859 [hep-th/9803002].
  * (26) N. Drukker and D. J. Gross, _An Exact prediction of N=4 SUSYM theory for string theory_ , _J. Math. Phys._ 42 (2001) 2896 [hep-th/0010274].
  * (27) M. Albanese, F. Costantini, G. Fiorentini, F. Flore, M. Lombardo, R. Tripiccione et al., _Glueball masses and string tension in lattice qcd_ , _Physics Letters B_ 192 (1987) 163.
  * (28) S.-x. Chu, D. Hou and H.-c. Ren, _The Subleading Term of the Strong Coupling Expansion of the Heavy-Quark Potential in a N=4 Super Yang-Mills Vacuum_ , _JHEP_ 08 (2009) 004 [0905.1874].

##  Appendix A Appendix - Phase of the Pfaffian

Figure 6:  $1-cos(\alpha)$ vs $\mu$ for $L=2^{4},3^{3}\times 4,3^{2}\times
4^{2},4^{3}\times 3$

##  Appendix B Appendix – Dependence of the fits on smearing parameters

Figure 7: $-\ln{<P(x)P(y)>}$ for $\alpha=0.45$

Figure 8:  $-\ln{<P(x)P(y)>}$ for $\alpha=0.55$

Figure 9:  Coefficient of the $1/r$ vs $\sqrt{\lambda}$ for $\alpha=0.45$

Figure 10:  Coefficient of the $1/r$ vs $\sqrt{\lambda}$ for $\alpha=0.55$

$\lambda$ | $a+b/R$ | Reduced-$\chi^{2}$
---|---|---
$5$ | $6.51(1)+0.76(3)/R$ | 0.18
$10$ | $6.29(1)+0.78(5)/R$ | 0.085
$15$ | $6.03(4)+0.79(1)/R$ | 1.6
$20$ | $6.06(1)+0.80(5)/R$ | 0.13
$25$ | $5.92(1)+0.81(2)/R$ | 1.4
$30$ | $5.91(1)+0.92(1)/R$ | 1.8
$35$ | $5.96(1)+0.93(2)/R$ | 1.2
$40$ |  $6.22(2)+0.97(1)/R$ ($3.5<R<5.0$) | 2.3
Table 1:  $1/R$ Fitting results for $L=12^{4},\mu=0.05,N_{smear}=5,\alpha=0.45$ $\lambda$ | $a+b/R$ | Reduced-$\chi^{2}$
---|---|---
$5$ | $6.67(1)+0.79(4)/R$ | 0.16
$10$ | $6.44(2)+0.83(5)/R$ | 0.07
$15$ | $6.17(1)+0.82(1)/R$ | 1.50
$20$ | $6.51(3)+0.81(8)/R$ | 0.04
$25$ | $6.09(1)+0.86(2)/R$ | 2.10
$30$ | $6.09(1)+0.937(1)/R$ | 1.34
$35$ | $6.05(1)+0.934(2)/R$ | 0.92
$40$ |  $6.25(1)+1.07(2)/R$ ($3.0<R<5.0$) | 3.03
Table 2:  $1/R$ Fitting results for $L=12^{4},\mu=0.05,N_{smear}=5,\alpha=0.50$ $\lambda$ | $a+b/R$ | Reduced-$\chi^{2}$
---|---|---
$5$ | $6.76(1)+0.85(3)/R$ | 0.17
$10$ | $6.54(1)+0.87(5)/R$ | 0.09
$15$ | $6.28(1)+0.87(1)/R$ | 1.23
$20$ | $6.47(2)+0.88(6)/R$ | 0.06
$25$ | $6.09(1)+0.88(1)/R$ | 1.53
$30$ | $6.13(1)+0.96(1)/R$ | 1.47
$35$ | $6.14(1)+0.97(2)/R$ | 0.90
$40$ |  $6.28(1)+1.07(2)/R$ ($2.5<R<5.0$) | 2.48
Table 3:  $1/R$ Fitting results for
$L=12^{4},\mu=0.05,N_{smear}=5,\alpha=0.55$