# On Faking a Nash Equilibrium

Young Wu
University of Wisconsin-Madison
<EMAIL_ADDRESS>&Jeremy McMahan
University of Wisconsin-Madison
<EMAIL_ADDRESS>&Xiaojin Zhu
University of Wisconsin-Madison
<EMAIL_ADDRESS>&Qiaomin Xie
University of Wisconsin-Madison
<EMAIL_ADDRESS>

###### Abstract

We characterize offline data poisoning attacks on Multi-Agent Reinforcement
Learning (MARL), where an attacker may change a data set in an attempt to
install a (potentially fictitious) unique Markov-perfect Nash equilibrium. We
propose the unique Nash set, namely the set of games, specified by their Q
functions, with a specific joint policy being the unique Nash equilibrium. The
unique Nash set is central to poisoning attacks because the attack is
successful if and only if data poisoning pushes all plausible games inside it.
The unique Nash set generalizes the reward polytope commonly used in inverse
reinforcement learning to MARL. For zero-sum Markov games, both the inverse
Nash set and the set of plausible games induced by data are polytopes in the Q
function space. We exhibit a linear program to efficiently compute the optimal
poisoning attack. Our work sheds light on the structure of data poisoning
attacks on offline MARL, a necessary step before one can design more robust
MARL algorithms.

##  1 Introduction

Data poisoning attacks are well-known in supervised learning (intentionally
forcing the learner to train a wrong classifier) and reinforcement learning
(wrong policy) [1, 5, 6, 10, 11, 9, 13, 17, 8, 12, 15, 16]. Can data poisoning
attacks be a threat to Markov Games, too? This paper answers this question in
the affirmative: Under mild conditions, an attacker can force two game-playing
agents to adopt any fictitious Nash Equilibrium (NE), which does not need to
be a true NE of the original Markov Game. Furthermore, the attacker can
achieve this goal while minimizing its attack cost, which we define below.
Obviously, such power poses a threat to the security of Multi-Agent
Reinforcement Learning (MARL).

Formally, we study two-player zero-sum offline MARL. Let $D$ be a dataset
$\\{(s^{(k)},\textbf{a}^{(k)},r^{(k)})\\}_{k=1}^{K}$ with $K$ tuples of state
$s$, joint action $\textbf{a}=(a_{1},a_{2})$, rewards $(r,-r)$. The attacker’s
target NE is an arbitrary pure strategy pair
$\pi^{\dagger}:=(\pi^{\dagger}_{1},\pi^{\dagger}_{2})$. The attacker can
poison $D$ into another dataset $D^{\dagger}$ by paying cost
$C(D,D^{\dagger})$. Two MARL agents then receive $D^{\dagger}$ instead of $D$.
The attacker wants to ensure that the agents learn the target NE
$\pi^{\dagger}$ while minimizing $C$.

This problem is not well studied in the literature. Naive approaches – such as
modifying all the actions in the dataset to those specified by the target
policy $(\pi^{\dagger}_{1},\pi^{\dagger}_{2})$ – might not achieve the goal
for MARL learners who assign penalties due to the lack of data coverage.
Modifying all the rewards in the dataset that coincides with the target policy
to the reward upper bound might be feasible, but would not be optimal in terms
of attack cost $C$. Results on data poisoning against single-agent
reinforcement learning also cannot be directly applied to the multi-agent
case. In particular, there are no optimal policies in MARL, and equilibrium
policies are computed instead. There could be multiple equilibria that are
significantly different, and as a result, installing a target policy as the
unique equilibrium is difficult.

Adversarial attacks on MARL have been studied in [7, 3, 4], but we are only
aware of one previous work [14] on offline reward poisoning against MARL.
Nonetheless, they made the strong assumption that the learners compute the
Dominant Strategy Markov Perfect Equilibrium (DSMPE). In contrast, we assume a
weaker solution concept, Markov Perfect Equilibrium (MPE). Our general attack
framework also accommodates other forms of data poisoning.

Our framework can be summarized by the mnemonic “ToM moves to the UN”. (i) UN
stands for the Unique Nash set, which is the set of Q functions that make the
target $\pi^{\dagger}$ the unique NE. Uniqueness is crucial for the attacker
to ensure that MARL agents choose the target NE with certainty, and not
breaking ties arbitrarily among multiple NEs. (ii) ToM stands for the
attacker’s Theory of Mind of the MARL agents, namely the plausible set of Q
functions that the attacker believes the agents will entertain upon receiving
the poisoned dataset $D^{\dagger}$. (iii) The attack is successful if, by
controlling $D^{\dagger}$, the attacker can move the Tom set inside the UN
set. A successful attack with the smallest cost $C(D,D^{\dagger})$ is optimal.

Summary of Contributions:

  * •

We show that the set of zero-sum Markov games for which a deterministic policy
is the unique MPE is equivalent to the set of games for which the policy is a
strict MPE, and can be characterized by a polytope in the Q function space.

  * •

We describe a class of MARL learners that compute equilibrium policies based
on games within confidence regions around a point estimate of the Q function
of the Markov game. With appropriate parameters, an attack on these learners
would work on most of the model-based and model-free offline MARL learners
proposed in the literature.

  * •

We convert a version of the reward poisoning problem to a linear program that
can be solved efficiently, and we provide an attack that is always feasible as
long as the sizes of the attacker’s confidence regions are sufficiently small.

  * •

We provide a unified framework for offline data poisoning attacks on MARL
agents. Our results highlight a security threat to multi-agent reinforcement
learning agents, a necessary step before one can design novel MARL algorithms
robust to adversarial attacks.

##  2 Faking a Nash Equilibrium

###  2.1 The Unique Nash Set (UN) of a Normal-form Game

We present the main components of our approach with a normal-form game, in
particular, a two-player zero-sum game is a tuple
$\left(\mathcal{A},R\right)$, where
$\mathcal{A}=\mathcal{A}_{1}\times\mathcal{A}_{2}$ is the joint action space
and $R:\mathcal{A}\to\left[-b,b\right]$ is the mean reward function. We use
$b=\infty$ in the case of unbounded rewards. Given $\mathcal{A}$, we denote
the set of reward functions by
$\mathcal{R}=\left\\{R:\mathcal{A}\to\mathbb{R}\right\\}$.

A pure strategy profile $\pi=\left(\pi_{1},\pi_{2}\right)$ is a pair of
actions, where $\pi_{i}\in\mathcal{A}_{i}$ specifies the action for agent
$i\in\left\\{1,2\right\\}$. We focus on pure strategies, but we allow mixed
strategies in which case we use the notation $\pi_{i}\left(a_{i}\right)$ to
represent the probability of $i$ using the action $a_{i}\in\mathcal{A}_{i}$,
and $R$ computes the expected reward
$R\left(\pi\right)\coloneqq\displaystyle\sum_{a_{1}\in\mathcal{A}_{1},a_{2}\in\mathcal{A}_{2}}\pi_{1}\left(a_{1}\right)\pi_{2}\left(a_{2}\right)R\left(\left(a_{1},a_{2}\right)\right)$.

######  Definition 1 (Nash Equilibrium).

A Nash equilibrium (NE) of a normal-form game $\left(\mathcal{A},R\right)$ is
a mixed strategy profile $\pi$ that satisfies,

$\displaystyle R\left(\left(\pi_{1},a_{2}\right)\right)$
$\displaystyle=R\left(\pi\right)=R\left(\left(a_{1},\pi_{2}\right)\right),\forall\;a_{1}:\pi_{1}\left(a_{1}\right)>0,a_{2}:\pi_{2}\left(a_{2}\right)>0,$
$\displaystyle R\left(\left(\pi_{1},a_{2}\right)\right)$ $\displaystyle\leq
R\left(\pi\right)\leq
R\left(\left(a_{1},\pi_{2}\right)\right),\forall\;a_{1}:\pi_{2}\left(a_{1}\right)=0,a_{2}:\pi_{2}\left(a_{1}\right)=0,$

in particular, for a pure strategy profile $\pi$, it is a Nash equilibrium if,

$\displaystyle R\left(\left(\pi_{1},a_{2}\right)\right)$ $\displaystyle\leq
R\left(\pi\right)\leq
R\left(\left(a_{1},\pi_{2}\right)\right),\forall\;a_{1}\neq\pi_{1},a_{2}\neq\pi_{2}.$
(1)

We define $\mathcal{N}\left(R\right)\coloneqq\left\\{\pi:\pi\text{\;is an NE
of\;}\left(\mathcal{A},R\right)\right\\}$ to be the set of all Nash equilibria
of a normal-form game $\left(\mathcal{A},R\right)$.

Now, we define the inverse image of $\mathcal{N}$ from a single pure strategy
profile $\pi$ back to the space of reward functions to be the unique Nash set.

######  Definition 2 (Unique Nash).

The unique Nash set of a pure strategy profile $\pi$ is the set of reward
functions $R$ such that $\left(\mathcal{A},R\right)$ has a unique Nash
equilibrium $\pi$,

$\displaystyle\mathcal{U}\left(\pi\right)$
$\displaystyle\coloneqq\mathcal{N}^{-1}\left(\left\\{\pi\right\\}\right)=\left\\{R\in\mathcal{R}:\mathcal{N}\left(R\right)=\left\\{\pi\right\\}\right\\}.$
(2)

To characterize $\mathcal{U}\left(\pi\right)$, we note that for normal-form
games, a pure strategy profile $\pi$ is the unique Nash equilibrium of a game
if and only if it is a strict Nash equilibrium, which is defined as a policy
$\pi$ that satisfies (1) with strict inequalities.

######  Proposition 1 (Unique Nash Polytope).

For any pure strategy profile $\pi$,

$\displaystyle\mathcal{U}\left(\pi\right)$
$\displaystyle=\left\\{R\in\mathcal{R}:\pi\text{\;is a strict NE
of\;}\left(\mathcal{A},R\right)\right\\}$
$\displaystyle=\left\\{R\in\mathcal{R}:R\left(\left(\pi_{1},a_{2}\right)\right)<R\left(\pi\right)<R\left(\left(a_{1},\pi_{2}\right)\right),\forall\;a_{1}\neq\pi_{1},a_{2}\neq\pi_{2}\right\\}.$
(3)

Here, the uniqueness is among all Nash equilibria including mixed-strategy
Nash equilibria. The proof of the equivalence between (2) and (3) is in the
appendix. We restrict our attention to pure-strategy equilibria and defer the
discussion of mixed strategy profiles to the last section.

To avoid working with strict inequalities, we define a closed subset of
$\mathcal{U}\left(\pi\right)$ of reward functions that lead to strict Nash
equilibria with an $\iota$ reward gap, which means all strict inequalities in
(3) are satisfied with a gap of at least $\iota$, for some $\iota>0$.

######  Definition 3 (Iota Strict Unique Nash).

For $\iota>0$, the $\iota$ strict unique Nash set of a pure strategy profile
$\pi$ is,

$\displaystyle\underline{\mathcal{U}}\left(\pi;\iota\right)$
$\displaystyle\coloneqq\left\\{R\in\mathcal{R}:R\left(\left(\pi_{1},a_{2}\right)\right)+\iota\leq
R\left(\pi\right)\leq
R\left(\left(a_{1},\pi_{2}\right)\right)-\iota,\forall\;a_{1}\neq\pi_{1},a_{2}\neq\pi_{2}\right\\}.$
(4)

For every pure strategy profile $\pi$ and $\iota>0$, we have
$\underline{\mathcal{U}}\left(\pi;\iota\right)\subset\mathcal{U}\left(\pi\right)$,
and the set is a polytope in $\mathcal{R}$.

###  2.2 The Attacker’s Theory of Mind (ToM) for Offline Normal-form Game
Learners

We provide a model of the attacker’s theory of mind of the victim. We assume
that the victims compute the Nash equilibria based on the reward functions
estimated from a dataset $D\in\mathcal{D}$, where $\mathcal{D}$ is the set of
possible datasets with $K$ episodes in the form
$\left\\{\left(\textbf{a}^{(k)},r^{(k)}\right)\right\\}_{k=1}^{K}$ , with
$\textbf{a}^{(k)}\in\mathcal{A}$ and $r^{(k)}\in\left[-b,b\right]$ for every
$k\in\left[K\right]$.

######  Definition 4 (Theory of Mind).

Given a dataset $D\in\mathcal{D}$, the theory-of-mind set
$\mathcal{T}\left(D\right)\subseteq\mathcal{R}$ is the set of plausible reward
functions that the victims estimate based on $D$ to compute their equilibria.
In particular, if the victims learn an action profile $\pi$, then
$\pi\in\displaystyle\bigcup_{R\in\mathcal{T}\left(D\right)}\mathcal{N}\left(R\right)$.

The theory-of-mind sets can be arbitrary and could be difficult to work with.
We define an outer approximation the set that is a hypercube in $\mathcal{R}$.

######  Definition 5 (Outer Approximation of Theory of Mind).

An outer approximation of $\mathcal{T}\left(D\right)$ is a set denoted by
$\overline{\mathcal{T}}\left(D\right)$ that satisfies
$\mathcal{T}\left(D\right)\subseteq\overline{\mathcal{T}}\left(D\right)$ for
every $D\in\mathcal{D}$, and can be written in the form,

$\displaystyle\overline{\mathcal{T}}\left(D\right)$
$\displaystyle=\left\\{R\in\mathcal{R}:\left|R\left(\textbf{a}\right)-\hat{R}\left(\textbf{a}\right)\right|\leq\rho^{(R)}\left(\textbf{a}\right),\forall\;\textbf{a}\in\mathcal{A}\right\\},$
(5)

for some point estimate $\hat{R}$ and radius $\rho^{(R)}$.
We call $\overline{\mathcal{T}}\left(D\right)\text{\;a\;}$ linear outer
approximation if $\hat{R}$ is linear in $\left\\{r^{(k)}\right\\}_{k=1}^{K.}$

We present a few examples of the theory-of-mind sets as follows.

######  Example 1 (Theory of Mind for Maximum Likelihood Victims).

Given a dataset $D\in\mathcal{D}$, if the attacker believes the victims are
maximum likelihood learners, then $\mathcal{T}\left(D\right)$ is a singleton
$R^{\text{\;MLE\;}}$, where, for every $\textbf{a}\in\mathcal{A}$,

$\displaystyle R^{\text{\;MLE\;}}\left(\textbf{a}\right)$
$\displaystyle\coloneqq\begin{cases}\dfrac{1}{N\left(\textbf{a}\right)}\displaystyle\sum_{k=1}^{K}r^{(k)}\mathbbm{1}_{\left\\{\textbf{a}^{(k)}=\textbf{a}\right\\}}&\text{\;if\;}N\left(\textbf{a}\right)>0\\\
0&\text{\;if\;}N\left(\textbf{a}\right)=0\\\
\end{cases};N\left(\textbf{a}\right)\coloneqq\displaystyle\sum_{k=1}^{K}\mathbbm{1}_{\left\\{\textbf{a}^{(k)}=\textbf{a}\right\\}}.$
(6)

The smallest outer approximation $\overline{\mathcal{T}}\left(D\right)$ can be
specified using $\hat{R}=R^{\text{\;MLE\;}}$ and $\rho^{(R)}=0$, and
$\overline{\mathcal{T}}$ is linear since (6) is linear in
$\left\\{r^{(k)}\right\\}_{k=1}^{K}$ .

######  Example 2 (Theory of Mind for Pessimistic Optimistic Victims).

Given a dataset $D\in\mathcal{D}$, if the attacker believes the victims are
learners that use pessimism and optimism by adding and subtracting bonus terms
and estimating one or two games, as in [2], then $\mathcal{T}\left(D\right)$
may contain two reward functions $\underline{R}$ and $\overline{R}$, where for
every $\textbf{a}\in\mathcal{A}$,

$\displaystyle\underline{R}\left(\textbf{a}\right)$ $\displaystyle\coloneqq
R^{\text{\;MLE\;}}\left(\textbf{a}\right)-\beta\left(\textbf{a}\right);\overline{R}\left(\textbf{a}\right)\coloneqq
R^{\text{\;MLE\;}}\left(\textbf{a}\right)+\beta\left(\textbf{a}\right),$ (7)

with $\beta\left(\textbf{a}\right)=\dfrac{c}{\sqrt{N\left(\textbf{a}\right)}}$
being the bonus term, for some constant $c$.
The smallest outer approximation $\overline{\mathcal{T}}\left(D\right)$ can be
specified using $\hat{R}=R^{\text{\;MLE\;}}$ and
$\rho^{(R)}\left(\textbf{a}\right)=\beta\left(\textbf{a}\right)$ for every
$\textbf{a}\in\mathcal{A}$, and $\overline{\mathcal{T}}$ is linear since (6)
and (7) are both linear in $\left\\{r^{(k)}\right\\}_{k=1}^{K}$ .

######  Example 3 (Theory of Mind for Data Splitting Victims).

Given a dataset $D\in\mathcal{D}$, if the attacker believes the victims use
maximum likelihood estimates on a subsample of the $D$, similar to the data-
splitting procedure in [2], then $\overline{\mathcal{T}}\left(D\right)$ could
be viewed as a high-probability set of rewards that the victims are estimating
and $\rho^{(R)}$ would be half of the confidence interval width for the mean
of the subsample around the mean of the complete dataset $R^{\text{\;MLE\;}}$.

###  2.3 The Cheapest Way to Move ToM into UN for Normal-form Games

The goal of the attacker is to install a specific action profile as the unique
Nash equilibrium of the game learned by the victim while minimally modifying
the training data. We consider a general attacker’s cost as a function
$C:\mathcal{D}\times\mathcal{D}\to\mathbb{R}^{+}$ where
$C\left(D,D^{\dagger}\right)$ is the cost of modifying the dataset from $D$ to
$D^{\dagger}$. Given the original data set $D\in\mathcal{D}$, the attacker’s
attack modality $\mathcal{D}\left(D\right)$ is the set of datasets the
attacker is allowed to modify the original dataset to. For the reward
poisoning problem, where $\mathcal{D}^{\left(R\right)}\left(D\right)$ is all
possible datasets in which only rewards are modified from $r^{(k)}$ to
$r^{\dagger,(k)}$, we consider the following cost function.

######  Example 4 ($L_{1}$ Cost Function).

For reward poisoning problems, we define the $L_{1}$ cost of modifying the
dataset from
$D=\left\\{\left(\textbf{a}^{(k)},r^{(k)}\right)\right\\}_{k=1}^{K}$ to
$D^{\dagger}=\left\\{\left(\textbf{a}^{(k)},r^{\dagger,(k)}\right)\right\\}_{k=1}^{K}$
by
$C^{\left(1\right)}\left(D,D^{\dagger}\right)\coloneqq\displaystyle\sum_{k=1}^{K}\left|r^{(k)}-r^{\dagger,(k)}\right|$.

Now, given the original dataset $D$ and the attacker’s target action profile
$\pi^{\dagger}$, we formally state the attacker’s problem as finding the
cheapest way to move $\mathcal{T}\left(D\right)$ into
$\mathcal{U}\left(\pi^{\dagger}\right)$.

######  Definition 6 (Attacker’s Problem).

The attacker’s problem with the target action profile $\pi^{\dagger}$ is,

$\displaystyle\displaystyle\inf_{D^{\dagger}\in\mathcal{D}\left(D\right)}$
$\displaystyle C\left(D,D^{\dagger}\right)$ (8) $\displaystyle\;s.t.\;$
$\displaystyle\mathcal{T}\left(D^{\dagger}\right)\subseteq\mathcal{U}\left(\pi^{\dagger}\right).$

$\mathcal{D}$ : Space of Datasets$\mathcal{R}$ : Space of
Rewards$\underline{\mathcal{U}}\left(\pi^{\dagger}\right)$Unique
Nash$\mathcal{U}\left(\pi^{\dagger}\right)$$D$$D^{\dagger}$Theory of
MindTheory of Mind$\displaystyle\min
C\left(D,D^{\dagger}\right)$$\overline{\mathcal{T}}\left(D\right)$$\mathcal{T}\left(D\right)$ToM
moves to the
UN$\overline{\mathcal{T}}\left(D^{\dagger}\right)$$\mathcal{T}\left(D^{\dagger}\right)$
Figure 1: Attacker’s Problem

In general, (8) cannot be solved efficiently, but for reward poisoning
problems with $L_{1}$ cost objective, we can relax the attacker’s problem
using $\iota$ strict unique Nash sets, which is a polytope described by (4),
and a linear outer approximation of the theory-of-mind set, a hypercube
described by (5), which can be converted into a linear program and solved
efficiently. We state this observation as the following proposition and depict
the relationship between the sets in Figure 1.

######  Proposition 2 (Reward Poisoning Linear Program).

Given $\iota>0$ and a linear $\overline{\mathcal{T}}$, the following problem
is a relaxation of the attacker’s reward poisoning problem and can be
converted into a linear program,

$\displaystyle\displaystyle\min_{D^{\dagger}\in\mathcal{D}^{\left(R\right)}\left(D\right)}$
$\displaystyle C^{\left(1\right)}\left(D,D^{\dagger}\right)$ (9)
$\displaystyle\;s.t.\;$
$\displaystyle\overline{\mathcal{T}}\left(D^{\dagger}\right)\subseteq\underline{\mathcal{U}}\left(\pi^{\dagger};\iota\right).$

In Figure 1, given a dataset $D$, the general attacker’s problem (8) of moving
$\mathcal{T}\left(D\right)$ (light green) to
$\mathcal{T}\left(D^{\dagger}\right)$ (light red) such that it is inside
$\mathcal{U}\left(\pi^{\dagger}\right)$ (light blue) while minimizing the
distance from $D$ to $D^{\dagger}$ is often intractable. We construct a
relaxed problem (9) of moving $\overline{\mathcal{T}}\left(D\right)$ (green)
to $\overline{\mathcal{T}}\left(D^{\dagger}\right)$ (red) such that it is
inside $\underline{\mathcal{U}}\left(\pi^{\dagger}\right)$ (blue), in which
all sets are polytopes and thus can be converted to a linear program for
linear costs and linear theory-of-mind mappings.

In the appendix, we provide the complete linear program and show that the
solution of (9) is feasible for (8). The optimality of the linear program
solution depends on how close the outer approximation of the theory-of-mind
set is, and in the case when the theory-of-mind set is already a hypercube,
the infimum in (8) can be achieved by taking the limit as $\iota\to 0$. The
following is an example illustrating the conversion of (9) into a linear
program.

######  Example 5 (Maximum Likelihood Centered Linear Program).

In the case $\hat{R}=R^{\text{\;MLE\;}}$ in the theory-of-mind set, (9) is
given by,

$\displaystyle\displaystyle\min_{r^{\dagger}\in\left[-b,b\right]^{K}}$
$\displaystyle\displaystyle\sum_{k=1}^{K}\left|r^{(k)}-r^{\dagger,(k)}\right|$
(10) $\displaystyle\;s.t.\;$ $\displaystyle R^{\text{\;MLE\;}}\text{\;is a
linear function of\;}r^{\dagger}\text{\;satisfying\;}~{}\eqref{eq:emle}$
$\displaystyle\overline{R}\text{\;and\;}\underline{R}\text{\;are upper and
lower bounds
of\;}\overline{\mathcal{T}}\left(r^{\dagger};R^{\text{\;MLE\;}}\right)\text{\;satisfying\;}~{}\eqref{eq:eoatom}$
$\displaystyle\left(\overline{R},\underline{R}\right)\text{\;is
in\;}\underline{\mathcal{U}}\left(\pi^{\dagger}\right)\text{\;satisfying\;}~{}\eqref{eq:eisun}$

Since $\overline{\mathcal{T}}\left(r^{\dagger};R^{\text{\;MLE\;}}\right)$ is a
hypercube and $\underline{\mathcal{U}}\left(\pi^{\dagger}\right)$ is a
polytope, the fact that the corners of the hypercube are inside the unique
Nash set if and only if every element in the hypercube is in the unique Nash
set implies that the constraint in (9) is satisfied. Technically, we only
require one corner of the hypercube to be inside the unique Nash polytope, as
shown in Figure 1, and we leave the details to the proof of Proposition 2 in
the appendix. Then, because the objective and all of the constraints in (10)
are linear in $r^{\dagger},\overline{R},\underline{R}$ and
$R^{\text{\;MLE\;}}$, this problem is a linear program.

##  3 Faking a Markov Perfect Equilibrium

###  3.1 The Unique Nash Set (UN) of a Markov Game

We now consider the attacker’s problem for Markov games. A finite-horizon two-
player zero-sum Markov game $G$ is a tuple
$\left(\mathcal{S},\mathcal{A},P,R,H\right)$, where $\mathcal{S}$ is the
finite state space; $\mathcal{A}=\mathcal{A}_{1}\times\mathcal{A}_{2}$ is the
joint action space;
$P=\left\\{P_{h}:\mathcal{S}\times\mathcal{A}\to\Delta\mathcal{S}\right\\}_{h=1}^{H}$
is the transition function with the initial state distribution
$P_{0}\in\Delta\mathcal{S}$; and
$R=\left\\{R_{h}:\mathcal{S}\times\mathcal{A}\to\left[-b,b\right]\right\\}_{h=1}^{H}$
is the mean reward function; and $H$ is the finite time horizon.

A deterministic Markovian policy $\pi=\left(\pi_{1},\pi_{2}\right)$ is a pair
of policies, where
$\pi_{i}=\left\\{\pi_{i,h}:\mathcal{S}\to\mathcal{A}_{i}\right\\}_{h=1}^{H}$
for $i\in\left\\{1,2\right\\}$, and $\pi_{i,h}\left(s\right)$ specifies the
action used in period $h$ and state $s$. Again, we focus on deterministic
policies, but we allow stochastic policies in which case we use the notation
$\pi_{i}=\left\\{\pi_{i,h}:\mathcal{S}\to\Delta\mathcal{A}_{i}\right\\}_{h=1}^{H}$
for $i\in\left\\{1,2\right\\}$, and
$\pi_{i,h}\left(s\right)\left(a_{i}\right)$ represent the probability of $i$
using the action $a_{i}\in\mathcal{A}_{i}$ in period $h$ state $s$.

The Q function is defined as, for every
$h\in\left[H\right],s\in\mathcal{S},\textbf{a}\in\mathcal{A}$,

$\displaystyle
Q_{h}\left(s,\textbf{a}\right)=R_{h}\left(s,\textbf{a}\right)+\displaystyle\sum_{s^{\prime}\in\mathcal{S}}P_{h}\left(s^{\prime}|s,\textbf{a}\right)\displaystyle\max_{\pi_{1}\in\Delta\mathcal{A}_{1}}\displaystyle\min_{\pi_{2}\in\Delta\mathcal{A}_{2}}Q_{h+1}\left(s^{\prime},\pi\right),$
(11)

with the convention $Q_{H+1}\left(s,\textbf{a}\right)=0$, and in the case
$\pi$ is stochastic, we write,

$\displaystyle
Q_{h}\left(s,\pi_{h}\left(s\right)\right)=\displaystyle\sum_{a_{1}\in\mathcal{A}_{1}}\displaystyle\sum_{a_{2}\in\mathcal{A}_{2}}\pi_{1,h}\left(s\right)\left(a_{1}\right)\pi_{2,h}\left(s\right)\left(a_{2}\right)Q_{h}\left(s,\left(a_{1},a_{2}\right)\right).$

Given $\mathcal{S},\mathcal{A},H$, we denote the set of Q functions by
$\mathcal{Q}=\left\\{\left\\{Q_{h}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}\right\\}_{h=1}^{H}\right\\}$.
Technically, $\mathcal{Q}$ is not the set of proper Q functions of Markov
games since both the reward functions and the transition functions do not have
to be proper, and given $Q\in\mathcal{Q}$, we may not be able to construct a
Markov game that induces $Q$. This choice is made to accommodate both model-
based and model-free victims who may or may not estimate the rewards and
transitions explicitly from the dataset.

A stage game of a Markov game $G$ in period $h\in\left[H\right]$, state
$s\in\mathcal{S}$ under policy $\pi$ is a normal form game
$\left(\mathcal{A},Q_{h}\left(s\right)\right)$, where $\mathcal{A}$ is the
joint action space of $G$; and $Q_{h}\left(s\right)$ is the mean reward
function, meaning the reward from action profile $\textbf{a}\in\mathcal{A}$ is
$Q_{h}\left(s,\textbf{a}\right)$. We define Markov perfect equilibria as
policies in which the action profile used in every stage game is a Nash
equilibrium.

######  Definition 7 (Markov Perfect Equilibrium).

A Markov perfect equilibrium (MPE) policy $\pi$ is a policy such that
$\pi_{h}\left(s\right)$ is a Nash equilibrium in the stage game
$\left(\mathcal{A},Q_{h}\left(s\right)\right).$
We define the set of all Markov perfect equilibria policies of a Markov game
that induces $Q\in\mathcal{Q}$ by
$\mathcal{M}\left(Q\right)=\left\\{\pi:\pi\text{\;is an MPE of a Markov game
with Q function\;}Q\right\\}.$

We note that Nash equilibria for Markov games can also be defined by
converting the Markov game into a single normal-form game, but we only
consider Markov perfect equilibria since Nash equilibria that are not Markov
perfect require coordination and commitment to policies in stage games that
are not visited along equilibrium paths, which is not realistic in the multi-
agent reinforcement learning setting.

We define the unique Nash set for Markov games as follows.

######  Definition 8 (Unique Nash).

The unique Nash set of a deterministic Markovian policy $\pi$ for a Markov
game $G$ is the set of Q functions such that $\pi$ is the unique Markov
perfect equilibrium under policy $\pi$,

$\displaystyle\mathcal{U}\left(\pi\right)$
$\displaystyle\coloneqq\mathcal{M}^{-1}\left(\left\\{\pi\right\\}\right)=\left\\{Q\in\mathcal{Q}:\mathcal{M}\left(Q\right)=\left\\{\pi\right\\}\right\\}.$
(12)

Next, we extend the characterization of the unique Nash set for normal-form
games to the Markov game setting.

######  Theorem 1 (Unique Nash Polytope).

For any deterministic policy $\pi$,

$\displaystyle\mathcal{U}\left(\pi\right)$
$\displaystyle=\left\\{Q\in\mathcal{Q}:\pi_{h}\left(s\right)\text{\;is a
strict NE
of\;}\left(\mathcal{A},Q_{h}\left(s\right)\right),\forall\;h\in\left[H\right],s\in\mathcal{S}\right\\}$
$\displaystyle=\left\\{Q\in\mathcal{Q}:\begin{cases}Q_{h}\left(s,\left(\pi_{1,h}\left(s\right),a_{2}\right)\right)<Q_{h}\left(s,\pi\left(s\right)\right)<Q_{h}\left(s,\left(a_{1},\pi_{2,h}\left(s\right)\right)\right),\\\
\hskip
20.00003pt\forall\;a_{1}\neq\pi_{1,h}\left(s\right),a_{2}\neq\pi_{2,h}\left(s\right),h\in\left[H\right],s\in\mathcal{S}\\\
\end{cases}\right\\},$ (13)

We show the equivalence between (12) and (13) in the proof of Theorem 1 in the
appendix. To avoid working with strict inequalities in (13), we again define
the $\iota$ strict version of the unique Nash polytope.

######  Definition 9 (Iota Strict Unique Nash).

For $\iota>0$, the $\iota$ strict unique Nash set of a deterministic policy
$\pi$ is,

$\displaystyle\underline{\mathcal{U}}\left(\pi;\iota\right)\coloneqq\left\\{Q\in\mathcal{Q}:\begin{cases}Q_{h}\left(s,\left(\pi_{1,h}\left(s\right),a_{2}\right)\right)+\iota\leq
Q_{h}\left(s,\pi\left(s\right)\right),\\\
Q_{h}\left(s,\pi\left(s\right)\right)\leq
Q_{h}\left(s,\left(a_{1},\pi_{2,h}\left(s\right)\right)\right)-\iota,\\\
\hskip
20.00003pt\forall\;a_{1}\neq\pi_{1,h}\left(s\right),a_{2}\neq\pi_{2,h}\left(s\right),h\in\left[H\right],s\in\mathcal{S}\\\
\end{cases}\right\\}.$ (14)

For every deterministic policy $\pi$ and $\iota>0$, we have
$\underline{\mathcal{U}}\left(\pi;\iota\right)\subset\mathcal{U}\left(\pi\right)$,
and the set is a polytope in $\mathcal{Q}$.

###  3.2 The Attacker’s Theory of Mind (ToM) for Offline Multi-Agent
Reinforcement Learners

Similar to the theory-of-mind set for normal-form game learners, we define the
set for Markov game learners in the $\mathcal{Q}$ space. Here, $\mathcal{D}$
is the set of datasets with $K$ episodes in the form
$\left\\{\left\\{\left(s^{(k)}_{h},\textbf{a}^{(k)}_{h},r^{(k)}_{h}\right)\right\\}_{h=1}^{H}\right\\}_{k=1}^{K}$
with $s^{(k)}_{h}\in\mathcal{S},\textbf{a}^{(k)}_{h}\in\mathcal{A}$ and
$r^{(k)}_{h}\in\left[-b,b\right]$ for every $k\in\left[K\right]$, and the
victims compute the Markov perfect equilibria based on the Q functions
estimated from such datasets.

######  Definition 10 (Theory of Mind).

Given a dataset $D\in\mathcal{D}$, the theory-of-mind set
$\mathcal{T}\left(D\right)\subseteq\mathcal{Q}$ is the set of Q functions that
the victims estimate based on $D$ to compute their equilibria. In particular,
if the victims learn a policy $\pi$, then
$\pi\in\displaystyle\bigcup_{Q\in\mathcal{T}\left(D\right)}\mathcal{M}\left(Q\right).$

######  Example 6 (Theory of Mind for Maximum Likelihood Victims).

To extend Example 1 in the Markov game setting, we define $R^{\text{\;MLE\;}}$
the same way and $P^{\text{\;MLE\;}}$ as follows,

$\displaystyle R_{h}^{\text{\;MLE\;}}\left(s,\textbf{a}\right)$
$\displaystyle\coloneqq\begin{cases}\dfrac{1}{N_{h}\left(s,\textbf{a}\right)}\displaystyle\sum_{k=1}^{K}r^{(k)}_{h}\mathbbm{1}_{\left\\{s^{(k)}_{h}=s,\textbf{a}^{(k)}_{h}=\textbf{a}\right\\}}&\text{\;if\;}N_{h}\left(s,\textbf{a}\right)>0\\\
0&\text{\;if\;}N_{h}\left(s,\textbf{a}\right)=0\\\ \end{cases},$ (15)
$\displaystyle N_{h}\left(s,\textbf{a}\right)$
$\displaystyle\coloneqq\displaystyle\sum_{k=1}^{K}\mathbbm{1}_{\left\\{s^{(k)}_{h}=s,\textbf{a}^{(k)}_{h}=\textbf{a}\right\\}},$
$\displaystyle P_{h}^{\text{\;MLE\;}}\left(s^{\prime}|s,\textbf{a}\right)$
$\displaystyle\coloneqq\begin{cases}\dfrac{\displaystyle\sum_{k=1}^{K}r^{(k)}_{h}\mathbbm{1}_{\left\\{s^{(k)}_{h+1}=s^{\prime},s^{(k)}_{h}=s,\textbf{a}^{(k)}_{h}=\textbf{a}\right\\}}}{N_{h}\left(s,\textbf{a}\right)}&\text{\;if\;}N_{h}\left(s,\textbf{a}\right)>0\\\
\dfrac{1}{\left|\mathcal{S}\right|}&\text{\;if\;}N_{h}\left(s,\textbf{a}\right)=0\\\
\end{cases},$ (16) $\displaystyle P_{0}^{\text{\;MLE\;}}\left(s\right)$
$\displaystyle\coloneqq\dfrac{1}{K}\displaystyle\sum_{k=1}^{K}\mathbbm{1}_{\left\\{s^{\left(k\right)}_{1}=s\right\\}}.$

We can construct $Q^{\text{\;MLE\;}}$ based on $R^{\text{\;MLE\;}}$ and
$P^{\text{\;MLE\;}}$ according to (11), and since all Nash equilibria have the
same value for zero-sum games, $Q^{\text{\;MLE\;}}$ is unique for every Markov
perfect equilibrium of the Markov game with rewards $R^{\text{\;MLE\;}}$ and
transitions $P^{\text{\;MLE\;}}$. Then we have that
$\mathcal{T}\left(D\right)$ is a singleton $Q^{\text{\;MLE\;}}$.

######  Example 7 (Theory of Mind for Confidence Bound Victims).

Given a dataset $D\in\mathcal{D}$, if the attacker believes the victims
estimate the Markov game by estimating the rewards and transitions within some
confidence region around some point estimates such as the maximum likelihood
estimates, as described in [14], then $\mathcal{T}\left(D\right)$ would be a
polytope with Q functions induced by the Markov games
$\left(\mathcal{S},\mathcal{A},P,R,H\right)$ with $P$ and $R$ satisfying, for
every $h\in\left[H\right],s\in\mathcal{S},\textbf{a}\in\mathcal{A}$,

$\displaystyle R_{h}\left(s,\textbf{a}\right)$
$\displaystyle\in\mathcal{C}^{(R)}_{h}\left(s,\textbf{a}\right)\coloneqq\left\\{R\in\mathbb{R}:\left|R-\hat{R}_{h}\left(s,\textbf{a}\right)\right|\leq\rho^{(R)}_{h}\left(s,\textbf{a}\right)\right\\},$
(17) $\displaystyle P_{h}\left(s,\textbf{a}\right)$
$\displaystyle\in\mathcal{C}^{(P)}_{h}\left(s,\textbf{a}\right)\coloneqq\left\\{P\in\Delta\mathcal{S}:\left\|P-\hat{P}_{h}\left(s,\textbf{a}\right)\right\|_{1}\leq\rho^{(P)}_{h}\left(s,\textbf{a}\right)\right\\},$
(18)

for some point estimates $\hat{P},\hat{R}$, and radii $\rho^{(R)}$ and
$\rho^{(P)}$. We note that $\mathcal{T}\left(D\right)$ is a polytope in
$\mathcal{Q}$, but it has an exponential number of vertices. We can construct
a tight hypercube around this polytope and call it the outer approximation of
$\mathcal{T}\left(D\right)$. It contains all the Q functions in the following
set, for every $h\in\left[H\right],s\in\mathcal{S},\textbf{a}\in\mathcal{A}$,

$\displaystyle Q_{h}\left(s,\textbf{a}\right)$
$\displaystyle\in\left[\underline{Q}_{h}\left(s,\textbf{a}\right),\overline{Q}_{h}\left(s,\textbf{a}\right)\right],$
(19) $\displaystyle\underline{Q}_{h}\left(s,\textbf{a}\right)$
$\displaystyle\coloneqq\displaystyle\min_{R\in\mathcal{C}^{(R)}_{h}\left(s,\textbf{a}\right)}R+\displaystyle\min_{P\in\mathcal{C}^{(P)}_{h}\left(s,\textbf{a}\right)}\displaystyle\sum_{s^{\prime}\in\mathcal{S}}P\left(s^{\prime}\right)\displaystyle\max_{\pi_{1}\in\Delta\mathcal{A}_{1}}\displaystyle\min_{\pi_{2}\in\Delta\mathcal{A}_{2}}\underline{Q}_{h+1}\left(s^{\prime},\pi\right),$
$\displaystyle\overline{Q}_{h}\left(s,\textbf{a}\right)$
$\displaystyle\coloneqq\displaystyle\max_{R\in\mathcal{C}^{(R)}_{h}\left(s,\textbf{a}\right)}R+\displaystyle\max_{P\in\mathcal{C}^{(P)}_{h}\left(s,\textbf{a}\right)}\displaystyle\sum_{s^{\prime}\in\mathcal{S}}P\left(s^{\prime}\right)\displaystyle\max_{\pi_{1}\in\Delta\mathcal{A}_{1}}\displaystyle\min_{\pi_{2}\in\Delta\mathcal{A}_{2}}\overline{Q}_{h+1}\left(s^{\prime},\pi\right).$

We omit Example 2 and Example 3 for Markov games since the constructions are
identical, except it is done for every stage game. As described in Example 7,
we formally define the outer approximation of the theory-of-mind set for
Markov games as follows.

######  Definition 11 (Outer Approximation of Theory of Mind).

An outer approximation of $\mathcal{T}\left(D\right)$ is a set denoted by
$\overline{\mathcal{T}}\left(D\right)$ that satisfies
$\mathcal{T}\left(D\right)\subseteq\overline{\mathcal{T}}\left(D\right)$ for
every $D\in\mathcal{D}$, and can be written in the form,

$\displaystyle\overline{\mathcal{T}}\left(D\right)$
$\displaystyle=\left\\{Q\in\mathcal{Q}:\left|Q_{h}\left(s,\textbf{a}\right)-\hat{Q}_{h}\left(s,\textbf{a}\right)\right|\leq\rho^{(Q)}_{h}\left(s,\textbf{a}\right),\forall\;\textbf{a}\in\mathcal{A},h\in\left[H\right],s\in\mathcal{S}\right\\},$
(20)

for some point estimate $\hat{Q}$ and radius $\rho^{(Q)}$.
We call $\overline{\mathcal{T}}\left(D\right)\text{\;a\;}$ linear outer
approximation if $\hat{Q}$ is linear in
$\left\\{\left\\{r^{(k)}_{h}\right\\}_{h=1}^{H}\right\\}_{k=1}^{K}$ .

###  3.3 The Cheapest Way to Move ToM into UN for Markov Games

In this subsection, we restate the attacker’s problem for multi-agent
reinforcement learners.

######  Definition 12 (Attacker’s Problem).

The attacker’s problem with target policy $\pi^{\dagger}$ is,

$\displaystyle\displaystyle\inf_{D^{\dagger}\in\mathcal{D}\left(D\right)}$
$\displaystyle C\left(D,D^{\dagger}\right)$ (21) $\displaystyle\;s.t.\;$
$\displaystyle\mathcal{T}\left(D^{\dagger}\right)\subseteq\mathcal{U}\left(\pi^{\dagger}\right).$

For reward poisoning problems, we consider the following $L_{1}$ cost.

######  Example 8 ($L_{1}$ Cost Function).

For reward poisoning problem, where
$\mathcal{D}^{\left(R\right)}\left(D\right)$ is all possible datasets in the
form
$D^{\dagger}=\left\\{\left\\{\left(s^{(k)}_{h},\textbf{a}^{(k)}_{h},r^{\dagger,(k)}_{h}\right)\right\\}_{h=1}^{H}\right\\}_{k=1}^{K}$
that are modified from
$D=\left\\{\left\\{\left(s^{(k)}_{h},\textbf{a}^{(k)}_{h},r^{(k)}_{h}\right)\right\\}_{h=1}^{H}\right\\}_{k=1}^{K}$
, we define the $L_{1}$ cost by
$C^{\left(1\right)}\left(D,D^{\dagger}\right)=\displaystyle\sum_{k=1}^{K}\displaystyle\sum_{h=1}^{H}\left|r^{(k)}_{h}-r^{\dagger,(k)}_{h}\right|.$

We use the same $\iota$ strictness relaxation of the unique Nash set and the
linear outer approximation of the theory-of-mind set to convert (21) into a
linear program, which can be solved efficiently. We state this observation as
the following theorem.

######  Theorem 2 (Reward Poisoning Linear Program).

Given $\iota>0$ and a linear $\overline{\mathcal{T}}$, the following problem
is a relaxation of the attacker’s reward poisoning problem and can be
converted into a linear program,

$\displaystyle\displaystyle\min_{D^{\dagger}\in\mathcal{D}^{\left(R\right)}\left(D\right)}$
$\displaystyle C^{\left(1\right)}\left(D,D^{\dagger}\right)$ (22)
$\displaystyle\;s.t.\;$
$\displaystyle\overline{\mathcal{T}}\left(D^{\dagger}\right)\subseteq\underline{\mathcal{U}}\left(\pi^{\dagger};\iota\right).$

######  Example 9 (Maximum Likelihood Centered Linear Program).

In the case $\hat{R}=R^{\text{\;MLE\;}}$ and $\hat{P}=P^{\text{\;MLE\;}}$, and
we construct $\overline{\mathcal{T}}\left(D\right)$ as described in Example 7,
(22) can be converted into a linear program even without explicitly
constructing the $\overline{\mathcal{T}}\left(D\right)$ set. We provide an
intuition here and the formal construction in the proof of Theorem 2,

$\displaystyle\displaystyle\min_{r^{\dagger}\in\left[-b,b\right]^{K}}$
$\displaystyle\displaystyle\sum_{k=1}^{K}\displaystyle\sum_{h=1}^{H}\left|r^{(k)}_{h}-r^{\dagger,(k)}_{h}\right|$
(23) $\displaystyle\;s.t.\;$ $\displaystyle R^{\text{\;MLE\;}}\text{\;is a
linear function of\;}r^{\dagger}\text{\;satisfying\;}~{}\eqref{eq:emler}$
$\displaystyle P^{\text{\;MLE\;}}\text{\;is independent
of\;}r^{\dagger}\text{\;satisfying\;}~{}\eqref{eq:emlep}$ $\displaystyle
Q^{\text{\;MLE\;}}\text{\;is a linear function
of\;}R^{\text{\;MLE\;}}\text{\;thus\;}r^{\dagger}\text{\;satisfying\;}~{}\eqref{eq:eq}$
$\displaystyle\overline{Q}\text{\;and\;}\underline{Q}\text{\;are upper and
lower bounds
of\;}\overline{\mathcal{T}}\left(r^{\dagger};Q^{\text{\;MLE\;}}\right)\text{\;satisfying\;}~{}\eqref{eq:ecblq}$
$\displaystyle\left(\overline{Q},\underline{Q}\right)\text{\;is
in\;}\underline{\mathcal{U}}\left(\pi^{\dagger}\right)\text{\;satisfying\;}~{}\eqref{eq:eisunm}$

Similar to Example 5, we move the hypercube
$\overline{\mathcal{T}}\left(r^{\dagger};Q^{\text{\;MLE\;}}\right)$ into the
polytope $\underline{\mathcal{U}}\left(\pi^{\dagger}\right)$ by moving one of
the corners into the polytope. Note that if $\overline{Q}$ and $\underline{Q}$
are not constructed directly as linear functions of $r^{\dagger}$, and are
computed by (19), then these constraints are not linear in $r^{\dagger}$. We
avoid this problem by using the dual linear program of (19). We present the
details in the appendix in the proof of Theorem 2. All other constraints are
linear in $r^{\dagger}$, and as a result, (23) is a linear program.

In the end, we present a sufficient but not necessary condition for the
feasibility of (22) and (21). This condition applies directly to normal-form
games with $H=1$.

######  Theorem 3 (Reward Poisoning Linear Program Feasibility).

For $\iota>0$, $\mathcal{T}\left(D\right)$ with $\hat{Q}=Q^{\text{\;MLE\;}}$,
and $N_{h}(s,\textbf{a})>0$ for every
$h\in\left[H\right],s\in\mathcal{S},\textbf{a}\in\mathcal{A}$ where either
$a_{1}=\pi^{\dagger}_{1,h}\left(s\right)$ or
$a_{2}=\pi^{\dagger}_{2,h}\left(s\right)$, the attacker’s reward poisoning
problem is feasible if for every
$h\in\left[H\right],s\in\mathcal{S},\textbf{a}\in\mathcal{A}$,

$\displaystyle\rho^{(Q)}_{h}\left(s,\textbf{a}\right)$
$\displaystyle\leq\dfrac{b-\iota}{4H}.$ (24)

$\mathcal{A}_{1}\setminus\mathcal{A}_{2}$ | $1^{\dagger}$ | $2$ | $3$
---|---|---|---
$1^{\dagger}$ | $0$ | $-b$ | $-b$
$2$ | $b$ | no change | no change
$3$ | $b$ | no change | no change
Table 1: A Feasible Attack

$\displaystyle r^{\dagger,(k)}_{h}$
$\displaystyle=\begin{cases}0&\text{\;if\;}\textbf{a}^{(k)}_{h}=\pi^{\dagger}\left(s^{(k)}_{h}\right)\\\
-b&\text{\;if\;}a^{(k)}_{1,h}=\pi^{\dagger}_{1,h}\left(s^{(k)}_{h}\right)\\\
b&\text{\;if\;}a^{(k)}_{2,h}=\pi^{\dagger}_{2,h}\left(s^{(k)}_{h}\right)\\\
r^{(k)}_{h}&\text{\;otherwise\;}\\\ \end{cases}$ (25)

To construct a feasible attack under (24), we use the poisoned rewards in
(25). An example where each agent has three actions and the target action
profile being action $\left(1,1\right)$ is shown in Table 1. With this
$r^{\dagger}$, the maximum likelihood estimate of the game has a unique Nash
equilibrium $\pi^{\dagger}_{h}\left(s\right)$ with a value of $0$ in every
stage $\left(h,s\right)$. Furthermore, if either the radius of rewards or the
radius of Q functions for the theory-of-mind set is less than
$\dfrac{b-\iota}{4H}$, we can show inductively that
$\pi^{\dagger}_{h}\left(s\right)$ remains the unique Nash equilibrium in every
stage $\left(h,s\right)$, thus showing that every Q function in the theory-of-
mind set is also in the unique Nash set, which means the attack is feasible.
The complete proof is in the appendix.

##  4 Discussions

We discuss a few extensions.

  * •

Faking a Unique Mixed Strategy Nash Equilibrium: due to the sensitivity of
mixing probabilities from small perturbations of the reward function, as long
as the theory-of-mind set has non-zero volume, it is impossible to install a
mixed strategy profile (or stochastic policy for Markov games) as the unique
equilibrium in general. However, this could be possible when the theory-of-
mind set is a singleton. To characterize the unique Nash set for a mixed
strategy profile, we need to extend Proposition 1 to include an additional
invertibility condition on the reward function, but it is difficult to convert
this condition into a linear constraint. We leave the technical details for
future work.

  * •

Faking an Optimal Policy for Single-Agent Reinforcement Learners: to attack a
single-agent Markov decision process, we observe that a policy $\pi$ is the
unique optimal policy if and only if $\pi$ is deterministic and is the strict
optimal policy. As a result, the unique optimal policy set is also a polytope
and can be viewed as a special case of the unique Nash set for a one-player
game. In the case of reward poisoning, the attacker’s problem can be
formulated as a linear program similar to (22).

  * •

Faking a Unique Coarse Correlated Equilibrium in Every Stage Game: for two-
player zero-sum Markov games, $\pi$ is the unique Markov Perfect Coarse
Correlated Equilibrium if and only if $\pi$ is the unique Markov Perfect
Equilibrium. Therefore, the results in the previous section apply directly.

  * •

Faking a Unique Markov Perfect Dominant Strategy Equilibria for General-Sum
Games: for $n$-player general-sum Markov games, if $\pi$ is a deterministic
policy and it is a Markov Perfect Strict Dominant Strategy Equilibrium, then
$\pi$ is the unique Markov Perfect Equilibrium. The attacker’s formulation in
[14] can be viewed as a special case of our results when Nash equilibria are
replaced by dominant strategy equilibria.

## References

  * [1] Kiarash Banihashem, Adish Singla, Jiarui Gan, and Goran Radanovic.  Admissible policy teaching through reward design.  arXiv preprint arXiv:2201.02185, 2022.
  * [2] Qiwen Cui and Simon S Du.  When is offline two-player zero-sum markov game solvable?  arXiv preprint arXiv:2201.03522, 2022.
  * [3] Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell.  Adversarial policies: Attacking deep reinforcement learning.  arXiv preprint arXiv:1905.10615, 2019.
  * [4] Wenbo Guo, Xian Wu, Sui Huang, and Xinyu Xing.  Adversarial policy learning in two-player competitive games.  In International Conference on Machine Learning, pages 3910–3919. PMLR, 2021.
  * [5] Yunhan Huang and Quanyan Zhu.  Deceptive reinforcement learning under adversarial manipulations on cost signals.  In International Conference on Decision and Game Theory for Security, pages 217–237. Springer, 2019.
  * [6] Guanlin Liu and Lifeng Lai.  Provably efficient black-box action poisoning attacks against reinforcement learning.  Advances in Neural Information Processing Systems, 34, 2021.
  * [7] Yuzhe Ma, Young Wu, and Xiaojin Zhu.  Game redesign in no-regret game playing.  arXiv preprint arXiv:2110.11763, 2021.
  * [8] Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Jerry Zhu.  Policy poisoning in batch reinforcement learning and control.  Advances in Neural Information Processing Systems, 32:14570–14580, 2019.
  * [9] Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla.  Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning.  In International Conference on Machine Learning, pages 7974–7984. PMLR, 2020.
  * [10] Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla.  Policy teaching in reinforcement learning via environment poisoning attacks.  Journal of Machine Learning Research, 22(210):1–45, 2021.
  * [11] Amin Rakhsha, Xuezhou Zhang, Xiaojin Zhu, and Adish Singla.  Reward poisoning in reinforcement learning: Attacks against unknown learners in unknown environments.  arXiv preprint arXiv:2102.08492, 2021.
  * [12] Anshuka Rangi, Haifeng Xu, Long Tran-Thanh, and Massimo Franceschetti.  Understanding the limits of poisoning attacks in episodic reinforcement learning.  In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 3394–3400. International Joint Conferences on Artificial Intelligence Organization, 7 2022.  Main Track.
  * [13] Yanchao Sun, Da Huo, and Furong Huang.  Vulnerability-aware poisoning mechanism for online rl with unknown dynamics.  arXiv preprint arXiv:2009.00774, 2020.
  * [14] Young Wu, Jeremy McMahan, Xiaojin Zhu, and Qiaomin Xie.  Reward poisoning attacks on offline multi-agent reinforcement learning.  In The Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI), 2023.
  * [15] Haoqi Zhang and David C Parkes.  Value-based policy teaching with active indirect elicitation.  In AAAI, volume 8, pages 208–214, 2008.
  * [16] Haoqi Zhang, David C Parkes, and Yiling Chen.  Policy teaching through reward function learning.  In Proceedings of the 10th ACM conference on Electronic commerce, pages 295–304, 2009.
  * [17] Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu.  Adaptive reward-poisoning attacks against reinforcement learning.  In International Conference on Machine Learning, pages 11225–11234. PMLR, 2020.

##  5 Supplementary Material

###  5.1 Proof of Proposition 1 and Theorem 1

We show that for zero-sum games, strict MPEs are MPEs and they are unique. We
use the following definition of MPE and strict MPE for zero-sum games
rewritten in terms of Q functions. Proposition 1 is a special case of Theorem
1 with $H=\left|\mathcal{S}\right|=1$.

######  Definition 13.

(Markov Perfect Equilibrium for Zero-sum Games$)\pi^{\dagger}$ is a MPE if for
each $h\in\left[H\right],s\in\mathcal{S}$,

$\displaystyle
Q_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)$
$\displaystyle\geq
Q_{h}^{\pi^{\dagger}}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right),\forall\;a_{1}\neq\pi^{\dagger}_{1,h}\left(s\right),$
(26) $\displaystyle
Q_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)$
$\displaystyle\leq
Q_{h}^{\pi^{\dagger}}\left(s,\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right),\forall\;a_{2}\neq\pi^{\dagger}_{2,h}\left(s\right).$
(27)

######  Definition 14.

(Strict Markov Perfect Equilibrium for Zero-sum Games$)\pi^{\dagger}$ is a
strict MPE if for each $h\in\left[H\right],s\in\mathcal{S}$,

$\displaystyle
Q_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)$
$\displaystyle>Q_{h}^{\pi^{\dagger}}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right),\forall\;a_{1}\neq\pi^{\dagger}_{1,h}\left(s\right),$
(28) $\displaystyle
Q_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)$
$\displaystyle<Q_{h}^{\pi^{\dagger}}\left(s,\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right),\forall\;a_{2}\neq\pi^{\dagger}_{2,h}\left(s\right).$
(29)

###### Proof.

Fix a period $h\in\left[H\right]$, and assume in periods
$h+1,h+2,...,H,\pi^{\dagger}$ is the unique NE in every state
$s\in\mathcal{S}$. This is vacuously true in period $H$.
First, $\pi^{\dagger}_{h}\left(s\right)$ is a NE since (28) implies (26) and
(29) implies (27).
Now, for a contradiction, assume
$\left(a^{{}^{\prime}}_{1},a^{{}^{\prime}}_{2}\right)\neq\pi^{\dagger}_{h}\left(s\right)$
is another NE in the stage game in period $h$ in some state $s\in\mathcal{S}$,
then,

$\displaystyle
Q_{h}^{\pi^{\dagger}}\left(s,\left(a^{{}^{\prime}}_{1},a^{{}^{\prime}}_{2}\right)\right)$
$\displaystyle\geq
Q_{h}^{\pi^{\dagger}}\left(s,\left(\pi^{\dagger}_{1,h}\left(s\right),a^{{}^{\prime}}_{2}\right)\right),$
(30) $\displaystyle
Q_{h}^{\pi^{\dagger}}\left(s,\left(a^{{}^{\prime}}_{1},a^{{}^{\prime}}_{2}\right)\right)$
$\displaystyle\leq
Q_{h}^{\pi^{\dagger}}\left(s,\left(a^{{}^{\prime}}_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right).$
(31)

From the strict MPE conditions,

$\displaystyle
Q_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:abr}}}{{>}}Q_{h}^{\pi^{\dagger}}\left(s,\left(\pi^{\dagger}_{1,h}\left(s\right),a^{{}^{\prime}}_{2}\right)\right),$
(32) $\displaystyle
Q_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:bbr}}}{{<}}Q_{h}^{\pi^{\dagger}}\left(s,\left(a^{{}^{\prime}}_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right).$
(33)

Combine the above inequalities, we get,

$\displaystyle
Q_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:qbp},~{}\eqref{eq:nqbp}}}{{>}}Q_{h}^{\pi^{\dagger}}\left(s,\left(a^{{}^{\prime}}_{1},a^{{}^{\prime}}_{2}\right)\right),$
(34) $\displaystyle
Q_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:qap},~{}\eqref{eq:nqap}}}{{<}}Q_{h}^{\pi^{\dagger}}\left(s,\left(a^{{}^{\prime}}_{1},a^{{}^{\prime}}_{2}\right)\right),$
(35)

which is a contradiction.
Therefore, $\pi^{\dagger}$ is the unique NE in period $h$, state $s$. Since
$h$ and $s$ are arbitrary, $\pi^{\dagger}$ is the unique MPE.

∎

###  5.2 Proof of Proposition 2 and Theorem 2

We first write out the complete optimization problem for (23) in Example 9,
then we show that the optimization is a relaxation by showing for any
$Q^{\pi^{\dagger}}\in\left[\underline{Q}^{\pi^{\dagger}},\overline{Q}^{\pi^{\dagger}}\right]$
elememtwise, $\pi^{\dagger}$ is a strict MPE, and as a result Theorem 1
implies its uniqueness. The proof that the problem can be converted into a
linear program is similar to LP conversions in [14]. We do not write out the
complete LP, and instead we show that each constraint can be converted into a
linear constraint. Theorem 2 is a special case of (23) with given
$\underline{Q}^{\pi^{\dagger}}$ and $\overline{Q}^{\pi^{\dagger}}$ that are
not derived from the rewards and transitions, and Proposition 2 is a special
case of Theorem 2 when $H=\left|\mathcal{S}\right|=1$.

$\displaystyle\displaystyle\min_{r^{\dagger}\in\left[0,1\right]^{HK}}$
$\displaystyle\displaystyle\sum_{k=1}^{K}\displaystyle\sum_{h=1}^{H}\left|r^{\dagger,(k)}_{h}-r^{(k)}_{h}\right|$
subject to $\displaystyle
R_{h}\left(s,\textbf{a}\right)=\dfrac{\displaystyle\sum_{k=1}^{K}\displaystyle\sum_{h=1}^{H}r^{(k)}_{h}\mathbbm{1}_{\left\\{s^{(k)}_{h}=s,\textbf{a}^{(k)}_{h}=\textbf{a}\right\\}}}{\displaystyle\max\left\\{N_{h}\left(s,\textbf{a}\right),1\right\\}},$
(36) $\displaystyle\hskip
20.00003pt\forall\;h\in\left[H\right],s\in\mathcal{S},\textbf{a}\in\mathcal{A},$
$\displaystyle\underline{Q}_{h}^{\pi^{\dagger}}\left(s,\textbf{a}\right)=\displaystyle\min_{R\in\mathcal{C}^{(R)}_{i,h}\left(s,\textbf{a}\right)}R+\displaystyle\min_{P\in\mathcal{C}^{(P)}_{h}\left(s,\textbf{a}\right)}\displaystyle\sum_{s^{\prime}\in\mathcal{S}}P\left(s^{\prime}\right)\underline{Q}_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\pi^{\dagger}_{h+1}\left(s^{\prime}\right)\right),$
(37) $\displaystyle\hskip
20.00003pt\forall\;h\in\left[H\right],s\in\mathcal{S},\textbf{a}\in\mathcal{A},$
$\displaystyle\overline{Q}_{h}^{\pi^{\dagger}}\left(s,\textbf{a}\right)=\displaystyle\max_{R\in\mathcal{C}^{(R)}_{i,h}\left(s,\textbf{a}\right)}R+\displaystyle\max_{P\in\mathcal{C}^{(P)}_{h}\left(s,\textbf{a}\right)}\displaystyle\sum_{s^{\prime}\in\mathcal{S}}P\left(s^{\prime}\right)\overline{Q}_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\pi^{\dagger}_{h+1}\left(s^{\prime}\right)\right),$
(38) $\displaystyle\hskip
20.00003pt\forall\;h\in\left[H\right],s\in\mathcal{S},\textbf{a}\in\mathcal{A},$
$\displaystyle\underline{Q}_{H+1}^{\pi^{\dagger}}\left(s,\textbf{a}\right)=\overline{Q}_{H+1}^{\pi^{\dagger}}\left(s,\textbf{a}\right)=0,\hskip
20.00003pt\forall\;s\in\mathcal{S},\textbf{a}\in\mathcal{A},$
$\displaystyle\underline{Q}_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)\geq\overline{Q}_{h}^{\pi^{\dagger}}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right)+\iota,$
(39) $\displaystyle\hskip
20.00003pt\forall\;h\in\left[H\right],s\in\mathcal{S},a_{1}\neq\pi^{\dagger}_{1,h}\left(s\right),$
$\displaystyle\overline{Q}_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)\leq\underline{Q}_{h}^{\pi^{\dagger}}\left(s,\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right)-\iota,$
(40) $\displaystyle\hskip
20.00003pt\forall\;h\in\left[H\right],s\in\mathcal{S},a_{2}\neq\pi^{\dagger}_{2,h}\left(s\right).$

Since we evaluate the $\underline{Q}$ and $\overline{Q}$ functions on the
policy $\pi^{\dagger}$, we add superscript $\pi^{\dagger}$ on $\underline{Q}$
and $\overline{Q}$ inside the optimization for clarity.

###### Proof.

Take any $R\in\mathcal{C}^{(R)}$ and $P\in\mathcal{C}^{(P)}$, due to the
definition of $\underline{Q}^{\pi^{\dagger}}$ and
$\overline{Q}^{\pi^{\dagger}}$, which are replicated in (37) and (38), we know
that, for each $h\in\left[H\right],s\in\mathcal{S},\textbf{a}\in\mathcal{A}$,

$\displaystyle\underline{Q}_{h}^{\pi^{\dagger}}\left(s,\textbf{a}\right)$
$\displaystyle\leq
Q_{h}^{\pi^{\dagger}}\left(s,\textbf{a}\right)\leq\overline{Q}_{h}^{\pi^{\dagger}}\left(s,\textbf{a}\right).$
(41)

Fix period $h\in\left[H\right]$, and assume in periods
$h+1,h+2,...,H,\pi^{\dagger}$ is the Nash equilibrium in every state
$s\in\mathcal{S}$. This is vacuously true in period $H$.
For a fixed $s\in\mathcal{S}$, for any
$a_{1}\neq\pi^{\dagger}_{1,h}\left(s\right)$,

$\displaystyle
Q_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:qbtw}}}{{\geq}}\underline{Q}_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:lpqa}}}{{\geq}}\overline{Q}_{h}^{\pi^{\dagger}}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right)+\iota$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:qbtw}}}{{\geq}}Q_{h}^{\pi^{\dagger}}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right)+\iota,$
(42)

and for any $a_{2}\neq\pi^{\dagger}_{2,h}\left(s\right)$,

$\displaystyle
Q_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:qbtw}}}{{\leq}}\overline{Q}_{h}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:lpqb}}}{{\leq}}\underline{Q}_{h}^{\pi^{\dagger}}\left(s,\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right)-\iota$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:qbtw}}}{{\geq}}Q_{h}^{\pi^{\dagger}}\left(s,\left(\pi^{\dagger}_{2,h}\left(s\right),a_{2}\right)\right)-\iota,$
(43)

(42) and (43) imply that $\pi^{\dagger}_{h}\left(s\right)$ is the Nash
equilibrium in period $h$ state $s$.
Therefore,
$Q^{\pi^{\dagger}}\in\underline{\mathcal{U}}\left(\pi^{\dagger};\iota\right)$,
and by Theorem 1, $\pi^{\dagger}$ is the unique MPE.
Now, to show that the problem can be converted into an LP, we note that (36),
(39) and (40) are linear constraints. We only have to convert (37) and (38)
into linear constraints, in particular, we convert the following linear
program, for some
$h\in\left[H\right],s\in\mathcal{S},\textbf{a}\in\mathcal{A}$,

$\displaystyle\displaystyle\min_{P}\displaystyle\sum_{s^{\prime}\in\mathcal{S}}$
$\displaystyle
P\left(s^{\prime}\right)\underline{Q}_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\pi^{\dagger}_{h+1}\left(s^{\prime}\right)\right)$
subject to $\displaystyle
P\left(s^{\prime}\right)\leq\hat{P}_{h}\left(s^{\prime}|s,\textbf{a}\right)+\rho^{(P)}_{h}\left(s,\textbf{a}\right),\forall\;s^{\prime}\in\mathcal{S},$
$\displaystyle
P\left(s^{\prime}\right)\geq\hat{P}_{h}\left(s^{\prime}|s,\textbf{a}\right)-\rho^{(P)}\left(s,\textbf{a}\right),\forall\;s^{\prime}\in\mathcal{S},$
$\displaystyle\displaystyle\sum_{s^{\prime}\in\mathcal{S}}P\left(s^{\prime}\right)=1,$
$\displaystyle P\left(s^{\prime}\right)\geq
0,\forall\;s^{\prime}\in\mathcal{S},$

into its dual problem,

$\displaystyle\displaystyle\max_{\underline{u}\in\mathbb{R}^{\mathcal{S}},\underline{v}\in\mathbb{R}^{\mathcal{S}},\underline{w}\in\mathbb{R}}$
$\displaystyle\displaystyle\sum_{s^{\prime}\in\mathcal{S}}\hat{P}_{h}\left(s^{\prime}|s,\textbf{a}\right)\left(\underline{u}_{s^{\prime}}-\underline{v}_{s^{\prime}}\right)+\rho^{(P)}_{h}\left(s,\textbf{a}\right)\left(\underline{u}_{s^{\prime}}+\underline{v}_{s^{\prime}}\right)+\underline{w}$
subject to
$\displaystyle\underline{u}_{s^{\prime}}-\underline{v}_{s^{\prime}}+\underline{w}\geq-\underline{Q}_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\pi^{\dagger}_{h+1}\left(s^{\prime}\right)\right),\forall\;s^{\prime}\in\mathcal{S},$
$\displaystyle\underline{u}_{s^{\prime}}\geq 0,\underline{v}_{s^{\prime}}\geq
0,\forall\;s^{\prime}\in\mathcal{S}.$

Therefore, (37) can be rewritten as the following linear constraints,

$\displaystyle\underline{Q}_{h}^{\pi^{\dagger}}\left(s,\textbf{a}\right)$
$\displaystyle=R_{h}\left(s,\textbf{a}\right)-\rho^{(R)}_{h}\left(s,\textbf{a}\right)$
$\displaystyle\hskip
10.00002pt+\displaystyle\sum_{s^{\prime}\in\mathcal{S}}\hat{P}_{h}\left(s^{\prime}|s,\textbf{a}\right)\left(\underline{u}_{s^{\prime}}-\underline{v}_{s^{\prime}}\right)+\rho^{(P)}_{h}\left(s,\textbf{a}\right)\left(\underline{u}_{s^{\prime}}+\underline{v}_{s^{\prime}}\right)+\underline{w},$
$\displaystyle\underline{u}_{s^{\prime}}-\underline{v}_{s^{\prime}}+\underline{w}$
$\displaystyle\geq-\underline{Q}_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\pi^{\dagger}_{h+1}\left(s^{\prime}\right)\right),\forall\;s^{\prime}\in\mathcal{S},$
$\displaystyle\underline{u}_{s^{\prime}}$ $\displaystyle\geq
0,\underline{v}_{s^{\prime}}\geq 0,\forall\;s^{\prime}\in\mathcal{S}.$

The similar dual problem can be written out for the $\overline{Q}$ to replace
(38),

$\displaystyle\overline{Q}_{h}^{\pi^{\dagger}}\left(s,\textbf{a}\right)$
$\displaystyle=R_{h}\left(s,\textbf{a}\right)+\rho^{(R)}_{h}\left(s,\textbf{a}\right)$
$\displaystyle\hskip
10.00002pt+\displaystyle\sum_{s^{\prime}\in\mathcal{S}}\hat{P}_{h}\left(s^{\prime}|s,\textbf{a}\right)\left(\overline{u}_{s^{\prime}}-\overline{v}_{s^{\prime}}\right)+\rho^{(P)}_{h}\left(s,\textbf{a}\right)\left(\overline{u}_{s^{\prime}}+\overline{v}_{s^{\prime}}\right)+\overline{w},$
$\displaystyle\overline{u}_{s^{\prime}}-\overline{v}_{s^{\prime}}+\overline{w}$
$\displaystyle\geq\overline{Q}_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\pi^{\dagger}_{h+1}\left(s^{\prime}\right)\right),\forall\;s^{\prime}\in\mathcal{S},$
$\displaystyle\overline{u}_{s^{\prime}}$ $\displaystyle\geq
0,\overline{v}_{s^{\prime}}\geq 0,\forall\;s^{\prime}\in\mathcal{S}.$

The linearization of the other $\underline{Q}$ and $\overline{Q}$ constraints
are similar.

∎

###  5.3 Proof of Theorem 3

Again, we write the proof for (23) in Example 9, and Theorem 3 is a special
case with given $\underline{Q}^{\pi^{\dagger}}$ and
$\overline{Q}^{\pi^{\dagger}}$ that are not derived from the rewards and
transitions. In particular, setting $\rho^{(Q)}=\rho^{(R)}$ and $\rho^{(P)}=0$
would like to the result stated in Theorem 3. We first provide the intuition
behind the proofs. The proof is at the end of this subsection.

Suppose the target action profile is $\left(1,1\right)$ in some state $s$ in
period $h$, we show that the target action profile $\left(1,1\right)$ is the
unique NE for any
$Q_{h}\left(s,\cdot\right)\in\left[\underline{Q}_{h}\left(s,\cdot\right),\overline{Q}_{h}\left(s,\cdot\right)\right]$
under the following attack,

$\displaystyle r^{\dagger,(k)}_{h}$
$\displaystyle=\begin{cases}-b&\text{\;if\;}a^{(k)}_{1,h}\neq\pi^{\dagger}_{1,h}\left(s^{(k)}_{h}\right),a^{(k)}_{2,h}=\pi^{\dagger}_{2,h}\left(s^{(k)}_{h}\right)\\\
0&\text{\;if\;}a^{(k)}_{1,h}=\pi^{\dagger}_{1,h}\left(s^{(k)}_{h}\right),a^{(k)}_{2,h}=\pi^{\dagger}_{2,h}\left(s^{(k)}_{h}\right)\\\
b&\text{\;if\;}a^{(k)}_{1,h}=\pi^{\dagger}_{1,h}\left(s^{(k)}_{h}\right),a^{(k)}_{2,h}\neq\pi^{\dagger}_{2,h}\left(s^{(k)}_{h}\right)\\\
r^{(k)}_{h}&\text{\;otherwise\;}\\\ \end{cases}.$ (44)

To simplify the notations, we define the bounds on the cumulative $Q$ value in
period $h+1,h+2,...,H$ as,

$\displaystyle\underline{S}_{h}$
$\displaystyle=\displaystyle\sum_{h^{\prime}=h+1}^{H}\displaystyle\min_{s^{\prime}\in\mathcal{S}}\underline{Q}_{h^{\prime}}\left(s^{\prime},\pi^{\dagger}_{h^{\prime}}\left(s^{\prime}\right)\right)$
$\displaystyle\overline{S}_{h}$
$\displaystyle=\displaystyle\sum_{h^{\prime}=h+1}^{H}\displaystyle\max_{s^{\prime}\in\mathcal{S}}\overline{Q}_{h^{\prime}}\left(s^{\prime},\pi^{\dagger}_{h^{\prime}}\left(s^{\prime}\right)\right)$

$\underline{Q}_{h}\left(s\right)$ is lower bounded by,

$\mathcal{A}_{1}\setminus\mathcal{A}_{2}$ | $1$ | $2$ | $...$ | $\left|\mathcal{A}_{2}\right|$
---|---|---|---|---
$1$ | $0-\rho^{(R)}_{h}\left(s,\left(1,1\right)\right)+\underline{S}_{h}$ | $b-\rho^{(R)}_{h}\left(s,\left(1,2\right)\right)+\underline{S}_{h}$ | $...$ | $b-\rho^{(R)}_{h}\left(s,\left(1,\left|\mathcal{A}_{2}\right|\right)\right)+\underline{S}_{h}$
$2$ | $-b-\rho^{(R)}_{h}\left(s,\left(2,1\right)\right)+\underline{S}_{h}$ | ? | $...$ | ?
$...$ | $...$ | $...$ | $...$ | $...$
$\left|\mathcal{A}_{1}\right|$ | $-b-\rho^{(R)}_{h}\left(s,\left(\left|\mathcal{A}_{1}\right|,1\right)\right)+\underline{S}_{h}$ | ? | $...$ | ?

$\overline{Q}_{h}\left(s\right)$ is upper bounded by,

$\mathcal{A}_{1}\setminus\mathcal{A}_{2}$ | $1$ | $2$ | $...$ | $\left|\mathcal{A}_{2}\right|$
---|---|---|---|---
$1$ | $0+\rho^{(R)}_{h}\left(s,\left(1,1\right)\right)+\overline{S}_{h}$ | $b+\rho^{(R)}_{h}\left(s,\left(1,2\right)\right)+\overline{S}_{h}$ | $...$ | $b+\rho^{(R)}_{h}\left(s,\left(1,\left|\mathcal{A}_{2}\right|\right)\right)+\overline{S}_{h}$
$2$ | $-b+\rho^{(R)}_{h}\left(s,\left(2,1\right)\right)+\overline{S}_{h}$ | ? | $...$ | ?
$...$ | $...$ | $...$ | $...$ | $...$
$\left|\mathcal{A}_{1}\right|$ | $-b+\rho^{(R)}_{h}\left(s,\left(\left|\mathcal{A}_{1}\right|,1\right)\right)+\overline{S}_{h}$ | ? | $...$ | ?

For $\left(1,1\right)$ to be the $\iota$ strict, thus unique, Nash equilibrium
for all $Q\in\left[\underline{Q},\overline{Q}\right]$, sufficient conditions
are, for $a_{1}\neq 1$ and $a_{2}\neq 1$,

$\displaystyle-\rho^{(R)}_{h}\left(s,\left(1,1\right)\right)+\underline{S}_{h}-\dfrac{\iota}{2}$
$\displaystyle\geq-\dfrac{b}{2H}\left(H-h+1\right)\geq-b+\rho^{(R)}_{h}\left(s,\left(a_{1},1\right)\right)+\overline{S}_{h}+\dfrac{\iota}{2},$
$\displaystyle\rho^{(R)}_{h}\left(s,\left(1,1\right)\right)+\overline{S}_{h}+\dfrac{\iota}{2}$
$\displaystyle\leq\dfrac{b}{2H}\left(H-h+1\right)\leq
b-\rho^{(R)}_{h}\left(s,\left(1,a_{2}\right)\right)+\underline{S}_{h}-\dfrac{\iota}{2},$

which would be true in period $1$ if the following is satisfied for a such
that either $a_{1}=\pi^{\dagger}_{1,h}\left(s\right)$ or
$a_{2}=\pi^{\dagger}_{2,h}\left(s\right)$,

$\displaystyle\rho^{(R)}_{h}\left(s,\textbf{a}\right)$
$\displaystyle\leq\dfrac{b-\iota}{4H}\leq\dfrac{b}{2H}-\dfrac{\iota}{2,}$

which in turn implies,

$\displaystyle\underline{S}_{h}$
$\displaystyle\geq-\dfrac{b}{2H}\left(H-h+1\right)+\dfrac{b}{4H},$
$\displaystyle\overline{S}_{h}$
$\displaystyle\leq\dfrac{b}{2H}\left(H-h+1\right)-\dfrac{b}{4H}.$

We provide the formal proof below.

###### Proof.

We assume is satisfied, meaning, for each
$h\in\left[H\right],s\in\mathcal{S},\textbf{a}\in\mathcal{A}$,

$\displaystyle\rho^{(R)}_{h}\left(s,\textbf{a}\right)$
$\displaystyle\leq\dfrac{b-\iota}{4H}\leq\dfrac{b}{2H}-\dfrac{\iota}{2}.$ (45)

In addition, take $R\in\mathcal{C}^{(R)}$, based on (44), we can compute
$\hat{R}$ using (36), and for each $h\in\left[H\right],s\in\mathcal{S}$,

$\displaystyle-\rho^{(R)}_{h}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)$
$\displaystyle\leq
R_{h}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)\leq\rho^{(R)}_{h}\left(s,\pi^{\dagger}_{h}\left(s\right)\right),$
(46)
$\displaystyle-b-\rho^{(R)}_{h}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right)$
$\displaystyle\leq
R_{h}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right)$
$\displaystyle\leq-b+\rho^{(R)}_{h}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right),$
(47) $\displaystyle
b-\rho^{(R)}_{h}\left(s,\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right)$
$\displaystyle\leq
R_{h}\left(s,\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right)$
$\displaystyle\leq
b+\rho^{(R)}_{h}\left(s,\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right).$
(48)

We proceed by induction. In period $H$, for
$a_{1}\neq\pi^{\dagger}_{1,H}\left(s\right)$,

$\displaystyle
Q_{H}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{H}\left(s\right)\right)-\dfrac{\iota}{2}$
$\displaystyle=R_{H}\left(s,\pi^{\dagger}_{H}\left(s\right)\right)-\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:rooul}}}{{\geq}}-\rho^{(R)}_{h}\left(s,\pi^{\dagger}_{H}\left(s\right)\right)-\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:ccc}}}{{\geq}}-\dfrac{b}{2H}$
(49) $\displaystyle\geq-b+\dfrac{b}{2H}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:ccc}}}{{\geq}}-b+\rho^{(R)}_{h}\left(s,\left(a_{1},\pi^{\dagger}_{2,H}\left(s\right)\right)\right)+\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:raoul}}}{{\geq}}R_{H}\left(s,\left(a_{1},\pi^{\dagger}_{2,H}\left(s\right)\right)\right)+\dfrac{\iota}{2}$
$\displaystyle=Q_{h}^{\pi^{\dagger}}\left(s,\left(a_{1},\pi^{\dagger}_{2,H}\left(s\right)\right)\right)+\dfrac{\iota}{2},$

and for $a_{2}\neq\pi^{\dagger}_{2,H}\left(s\right)$,

$\displaystyle
Q_{H}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{H}\left(s\right)\right)+\dfrac{\iota}{2}$
$\displaystyle=R_{H}\left(s,\pi^{\dagger}_{H}\left(s\right)\right)+\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:rooul}}}{{\leq}}\rho^{(R)}_{h}\left(s,\pi^{\dagger}_{H}\left(s\right)\right)+\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:ccc}}}{{\leq}}\dfrac{b}{2H}$
(50) $\displaystyle\leq b-\dfrac{b}{2H}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:ccc}}}{{\leq}}b-\rho^{(R)}_{h}\left(s,\left(a_{1},\pi^{\dagger}_{2,H}\left(s\right)\right)\right)-\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:raoul}}}{{\leq}}R_{H}\left(s,\left(\pi^{\dagger}_{1,H}\left(s\right),a_{2}\right)\right)-\dfrac{\iota}{2}$
$\displaystyle=Q_{H}^{\pi^{\dagger}}\left(s,\left(\pi^{\dagger}_{1,H}\left(s\right),a_{2}\right)\right)-\dfrac{\iota}{2}.$

Now, fix a period $h<H$, we assume in periods
$h^{\prime}\in\left\\{h+1,h+2,...,H\right\\}$, in every state
$s\in\mathcal{S},\pi^{\dagger}$ is the Nash equilibrium, and,

$\displaystyle-\dfrac{b}{2}\left(H-h^{\prime}+1\right)$ $\displaystyle\leq
Q_{h^{\prime}}^{\pi^{\dagger}}\left(s,\pi^{\dagger}_{h^{\prime}}\left(s\right)\right)\leq\dfrac{b}{2}\left(H-h^{\prime}+1\right).$
(51)

This is true in period $H$ due to (49) and (50).
Now in period $h$, for a fixed $s\in\mathcal{S}$, for any
$a_{1}\neq\pi^{\dagger}_{1,h}\left(s\right)$,

$\displaystyle Q_{h}^{\pi^{\dagger}}$
$\displaystyle\left(s,\pi^{\dagger}_{h}\left(s\right)\right)-\dfrac{\iota}{2}$
$\displaystyle=R_{h}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)+\displaystyle\sum_{s^{\prime}\in\mathcal{S}}P_{h}\left(s^{\prime}|s,\pi^{\dagger}_{h}\left(s\right)\right)Q_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\pi^{\dagger}_{h+1}\left(s^{\prime}\right)\right)-\dfrac{\iota}{2}$
$\displaystyle\geq
R_{h}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)+\displaystyle\min_{s^{\prime}\in\mathcal{S}}Q_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\pi^{\dagger}_{h+1}\left(s^{\prime}\right)\right)-\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:qhhul}}}{{\geq}}R_{h}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)-\dfrac{b}{2}\left(H-h\right)-\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:rooul}}}{{\geq}}-\rho^{(R)}_{h}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)-\dfrac{b}{2H}\left(H-h\right)-\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:ccc}}}{{\geq}}-\dfrac{b}{2H}-\dfrac{b}{2H}\left(H-h\right)$
$\displaystyle\geq-\dfrac{b}{2H}\left(H-h+1\right)$ (52)
$\displaystyle\geq-b+\dfrac{b}{2H}+\dfrac{b}{2H}\left(H-h\right)$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:ccc}}}{{\geq}}-b+\rho^{(R)}_{h}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right)+\dfrac{b}{2H}\left(H-h\right)+\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:raoul}}}{{\geq}}R_{h}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right)+\dfrac{b}{2H}\left(H-h\right)+\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:qhhul}}}{{\geq}}R_{h}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right)+\displaystyle\max_{s^{\prime}\in\mathcal{S}}Q_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\left(a_{1},\pi^{\dagger}_{h+1}\left(s^{\prime}\right)\right)\right)+\dfrac{\iota}{2}$
$\displaystyle\geq
R_{h}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right)+\displaystyle\sum_{s^{\prime}\in\mathcal{S}}P_{h}\left(s^{\prime}|s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right)Q_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\left(a_{1},\pi^{\dagger}_{h+1}\left(s^{\prime}\right)\right)\right)+\dfrac{\iota}{2}$
$\displaystyle=Q_{h}^{\pi^{\dagger}}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right)+\dfrac{\iota}{2},$

and for $a_{2}\neq\pi^{\dagger}_{2,h}\left(s\right)$,

$\displaystyle Q_{h}^{\pi^{\dagger}}$
$\displaystyle\left(s,\pi^{\dagger}_{h}\left(s\right)\right)+\dfrac{\iota}{2}$
$\displaystyle=R_{h}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)+\displaystyle\sum_{s^{\prime}\in\mathcal{S}}P_{h}\left(s^{\prime}|s,\pi^{\dagger}_{h}\left(s\right)\right)Q_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\pi^{\dagger}_{h+1}\left(s^{\prime}\right)\right)+\dfrac{\iota}{2}$
$\displaystyle\leq
R_{h}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)+\displaystyle\max_{s^{\prime}\in\mathcal{S}}Q_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\pi^{\dagger}_{h+1}\left(s^{\prime}\right)\right)+\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:qhhul}}}{{\leq}}R_{h}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)+\dfrac{b}{2H}\left(H-h\right)+\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:rooul}}}{{\leq}}\rho^{(R)}_{h}\left(s,\pi^{\dagger}_{h}\left(s\right)\right)+\dfrac{b}{2H}\left(H-H\right)+\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:ccc}}}{{\leq}}\dfrac{b}{2H}+\dfrac{b}{2H}\left(H-h\right)$
$\displaystyle=\dfrac{b}{2H}\left(H-h+1\right)$ (53) $\displaystyle\leq
b-\dfrac{b}{2H}-\dfrac{b}{2H}\left(H-h\right)$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:ccc}}}{{\leq}}b+\rho^{(R)}_{h}\left(s,\left(a_{1},\pi^{\dagger}_{2,h}\left(s\right)\right)\right)-\dfrac{b}{2H}\left(H-h\right)-\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:raoul}}}{{\leq}}R_{h}\left(s,\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right)-\dfrac{b}{2H}\left(H-h\right)-\dfrac{\iota}{2}$
$\displaystyle\stackrel{{\scriptstyle~{}\eqref{eq:qhhul}}}{{\leq}}R_{h}\left(s,\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right)+\displaystyle\min_{s^{\prime}\in\mathcal{S}}Q_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right)-\dfrac{\iota}{2}$
$\displaystyle\leq
R_{h}\left(s,\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right)+\displaystyle\sum_{s^{\prime}\in\mathcal{S}}P_{h}\left(s^{\prime}|s,\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right)Q_{h+1}^{\pi^{\dagger}}\left(s^{\prime},\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right)-\dfrac{\iota}{2}$
$\displaystyle=Q_{h}^{\pi^{\dagger}}\left(s,\left(\pi^{\dagger}_{1,h}\left(s\right),a_{2}\right)\right)-\dfrac{\iota}{2}.$

Therefore, $\pi^{\dagger}$ is the Nash equilibrium in period $h$ state $s$,
and (52) and (53) are consistent (51). By induction, $\pi^{\dagger}$ is a
strict, thus unique, Nash equilibrium in every stage game, making
$\pi^{\dagger}$ the unique MPE.

∎


Canonical coordinates
for Yang-Mills-Chern-Simons theory

Måns Henningson

Department of Physics

University of Gothenburg

Göteborg, Sweden

<EMAIL_ADDRESS>
Abstract:

We consider the classical field theory of 2+1-dimensional Yang-Mills-Chern-
Simons theory on an arbitrary spatial manifold. We first define a gauge
covariant transverse electric field strength, which together with the gauge
covariant scalar magnetic field strength can be taken as coordinates on the
classical phase space. We then determine the Poisson-Dirac bracket and find
that these coordinates are canonically conjugate to each other. The
Hamiltonian is non-polynomial when expressed in terms of these coordinates,
but can be expanded in a power series in the coupling constant with polynomial
coefficients.

##  1 Introduction

Three-dimensional Yang-Mills-Chern-Simons theory [1][2][3] with coupling
constant111of dimension $(\mathrm{energy})^{1/2}$. $\lambda$ and integer level
$k$ is governed by the action222Our conventions for the gauge covariant
derivate $D_{\mu}$ of a field $X$ in the adjoint representation and the
curvature $F_{\mu\nu}$ of the Yang-Mills connection $A_{\mu}$ are
$D_{\mu}X=\nabla_{\mu}X+i[X,A_{\mu}]$ and
$F_{\mu\nu}=\nabla_{\mu}A_{\nu}-\nabla_{\nu}A_{\mu}-i[A_{\mu},A_{\nu}]$, with
$\nabla_{\mu}$ being the covariant derivative constructed out of the spin-
connection on the space-time manifold. This gives the commutator of gauge
covariant derivatives $[D_{\mu},D_{\nu}]X=i[X,F_{\mu\nu}]$. These conventions
are compatible with $X$, $A_{\mu}$ and $F_{\mu\nu}$ all being Hermitian.
333With an $SU(N)$ gauge group, $\,\mathrm{Tr}$ is the trace in the
fundamental $N$-dimensional representation.

$S=\frac{1}{8\lambda^{2}}\int
d^{3}x\sqrt{G}\,\mathrm{Tr}\left(F^{\mu\nu}F_{\mu\nu}\right)+\frac{k}{4\pi}\int
d^{3}x\epsilon^{\mu\nu\rho}\,\mathrm{Tr}\left(A_{\mu}\partial_{\nu}A_{\rho}-\frac{2i}{3}A_{\mu}A_{\nu}A_{\rho}\right).$
(1)

The Bianchi identity and the equations of motion read

$\displaystyle D_{\mu}F_{\nu\rho}+D_{\nu}F_{\rho\mu}+D_{\rho}F_{\mu\nu}$
$\displaystyle=$ $\displaystyle 0$ (2)
$\displaystyle\frac{1}{2\lambda^{2}}D_{\nu}F^{\mu\nu}+\frac{k}{4\pi}\epsilon^{\mu\nu\rho}F_{\nu\rho}$
$\displaystyle=$ $\displaystyle 0.$ (3)

It follows from these equations that a current of the form
$J^{\mu}=T^{\mu\nu}K_{\nu}$, where

$T^{\mu\nu}=\frac{1}{2\lambda^{2}}\,\mathrm{Tr}\left(F^{\mu\rho}F^{\nu}{}_{\rho}-\frac{1}{4}G^{\mu\nu}F^{\rho\sigma}F_{\rho\sigma}\right)$
(4)

is the energy momentum tensor and the vector field $K_{\mu}$ obeys the Killing
condition $\nabla_{\mu}K_{\nu}+\nabla_{\nu}K_{\mu}=0$, is conserved, i.e.
$\nabla_{\mu}J^{\mu}=0$.

In this note, we will be concerned with the case where the space-time metric
$G_{\mu\nu}$ admits a time-like Killing vector field, which by a choice of
space-time coordinates $x^{\mu}$, $\mu=0,1,2$ can be taken as $K_{0}=1$,
$K_{i}=0$, $i=1,2$.444The spatial metric $g_{ij}$ may admits further Killing
vector fields, that would give rise to additional conserved currents. The
space-time metric then takes the form
$G_{\mu\nu}dx^{\mu}dx^{\nu}=-(dt)^{2}+g_{ij}dx^{i}dx^{j}$, and the equations
(2) can be rewritten in a $2+1$-dimensional notation:

$\displaystyle D_{0}B-\hat{\epsilon}^{ij}D_{i}E_{j}$ $\displaystyle=$
$\displaystyle 0$ (5)
$\displaystyle\frac{1}{2\lambda^{2}}(D_{0}E^{i}+\hat{\epsilon}^{ij}D_{j}B)-\frac{k}{2\pi}\hat{\epsilon}^{ij}E_{j}$
$\displaystyle=$ $\displaystyle 0$ (6)

and

$\displaystyle-\frac{1}{2\lambda^{2}}D_{i}E^{i}+\frac{k}{2\pi}B$
$\displaystyle=$ $\displaystyle 0,$ (7)

where

$\displaystyle B$ $\displaystyle=$
$\displaystyle\frac{1}{2}\hat{\epsilon}^{ij}F_{ij}$ (8) $\displaystyle E_{i}$
$\displaystyle=$ $\displaystyle F_{0i}$ (9)

are the magnetic and electric field strengths respectively.555We define the
tensor $\hat{\epsilon}^{ij}=\frac{1}{\sqrt{g}}\epsilon^{ij}$ in terms of the
Levi-Civita tensor density $\epsilon^{ij}$. The equations (5) determine the
time development of $B$ and $E^{i}$ respectively, while equation (7) is a
constraint that needs to be imposed on the data at some initial time. The time
component of the current constructed from the time-like Killing vector is the
Hamiltonian density

$\displaystyle{\cal H}$ $\displaystyle=$ $\displaystyle T^{00}$ (10)
$\displaystyle=$
$\displaystyle\frac{1}{2\lambda^{2}}\,\mathrm{Tr}(E_{i}E^{i}+BB),$ (11)

the spatial integral of which is the Hamiltonian

$H=\int d^{2}x\sqrt{g}{\cal H}.$ (12)

The phase space of the theory is given by the space of solutions to the
equations (5) and (7) modulo (infinitesimal) gauge transformations acting as

$\displaystyle\delta A_{i}$ $\displaystyle=$ $\displaystyle D_{i}\Lambda$ (13)
$\displaystyle\delta B$ $\displaystyle=$ $\displaystyle i[B,\Lambda]$ (14)
$\displaystyle\delta E_{i}$ $\displaystyle=$ $\displaystyle i[E_{i},\Lambda]$
(15)

with a Lie algebra-valued parameter $\Lambda$. The phase space is endowed with
a symplectic structure, i.e. a closed, non-degenerate two-form given by a
Poisson-Dirac bracket $\\{.,.\\}$. The equations of motion (5) can then be
cast in Hamiltonian form:

$\displaystyle\\{H,B\\}$ $\displaystyle=$ $\displaystyle D_{0}B+i[B,\Lambda]$
(16) $\displaystyle=$
$\displaystyle\hat{\epsilon}^{ij}D_{i}E_{j}+i[B,\Lambda]$ (17)
$\displaystyle\\{H,E^{i}\\}$ $\displaystyle=$ $\displaystyle
D_{0}E^{i}+i[E^{i},\Lambda]$ (18) $\displaystyle=$
$\displaystyle-\hat{\epsilon}^{ij}D_{j}B+\frac{\lambda^{2}k}{\pi}\hat{\epsilon}^{ij}E_{j}+i[E^{i},\Lambda],$
(19)

where we have included a possible infinitesimal gauge transformation with
parameter $\Lambda$.

The aim of this note is to describe a convenient set of coordinates to
parametrise this phase space. We will then compute the symplectic structure by
giving the Poisson-Dirac bracket between these coordinates. We will see that
our coordinates are canonically conjugate to each other, i.e. they are Darboux
coordinates. Finally, we will express the Hamiltonian in terms of these
coordinates. The Hamiltonian is non-polynomial, but can be expanded in a
power-series in the coupling constant $\lambda$ with coefficients that are
polynomial in the (rescaled) coordinates.

For another, somewhat unrelated, approach to a Hamilton treatment of Yang-
Mills-Chern-Simons theory, see [4].

We hope that these observations and constructions may prove useful for the
purpose of canonical quantization of Yang-Mills-Chern-Simons theory.

##  2 The coordinates

The electric field strength $E_{i}$ may, as any vector field in two
dimensions, be decomposed into its longitudinal and transverse scalar
components $E_{\parallel}$ and $E_{\perp}$:666In this note, we will not be
concerned with any issues connected to possible zero-modes.

$E_{i}=D_{i}E_{\parallel}+\hat{\epsilon}_{i}{}^{j}D_{j}E_{\perp}.$ (20)

In principle, this equation may be inverted to express $E_{\parallel}$ and
$E_{\perp}$ in terms of the vector components $E_{i}$, but not in closed form.

The constraint (7) can now be written as777We define $D^{2}=D_{i}D^{i}$ with
inverse $D^{-2}$, and similarly $\nabla^{2}=\nabla_{i}\nabla^{i}$ with inverse
$\nabla^{-2}$, again not worrying about possible zero-modes. 888By the
identity $\hat{\epsilon}^{ij}D_{i}D_{j}E_{\perp}=i[E_{\perp},B]$.

$-\frac{1}{2\lambda^{2}}\left(D^{2}E_{\parallel}+i[E_{\perp},B]\right)+\frac{k}{2\pi}B=0,$
(21)

and can be solved by expressing $E_{\parallel}$ in terms of $E_{\perp}$ and
the connection $A_{i}$:

$E_{\parallel}=D^{-2}\left(-i[E_{\perp},B]+\frac{\lambda^{2}k}{\pi}B\right).$
(22)

We will take the transverse electric field $E_{\perp}$ together with the
magnetic field $B$ as our phase space coordinates. They are gauge covariant,
but of course not gauge invariant, so to determine a point in the phase space,
they must be supplemented by a gauge condition

$\phi_{A}=0.$ (23)

Here the Lie algebra-valued scalar field $\phi_{A}$ is some fairly arbitrary
(linear) functional of the spatial gauge connection $A_{i}$ chosen so that
this equation determines a single point on each orbit of the group of gauge
transformations. This allows us to express $A_{i}$ in terms of $B$. For many
purposes, it will e.g. be convenient to use the Coulomb gauge condition
$\phi_{A}=\nabla_{i}A^{i}=0$. The general solution is then
$A_{i}=\epsilon_{i}{}^{j}\nabla_{j}A$, where $A$ is a Lie algebra-valued
scalar field. This gives
$B=-\nabla^{2}A-\frac{i}{2}\hat{\epsilon}^{ij}[\nabla_{i}A,\nabla_{j}A]$,
which may be inverted to express $A$, and thus $A_{i}$, as a power series in
$B$:

$A_{i}=\epsilon_{i}{}^{j}\nabla_{j}\left(-\nabla^{-2}B-\frac{i}{2}\hat{\epsilon}^{kl}\nabla^{-2}[\nabla_{k}\nabla^{-2}B,\nabla_{l}\nabla^{-2}B]+\ldots\right)$
(24)

Finally, we note that the time component $A_{0}$ of the gauge connection is
not an independent variable: The constraint (7) can be written as

$\frac{1}{2\lambda^{2}}(-D^{2}A_{0}+\nabla_{i}\nabla_{0}A^{i}+i[\nabla_{0}A_{i},A^{i}])+\frac{k}{2\pi}B=0$
(25)

and solved for $A_{0}$ in terms of the spatial components $A_{i}$:

$A_{0}=D^{-2}\left(\nabla_{i}\nabla_{0}A^{i}+i[\nabla_{0}A_{i},A^{i}]\right)+\frac{\lambda^{2}k}{\pi}D^{-2}B.$
(26)

To summarize, once a choice of gauge has been made, the phase space can be
parametrized by the scalar fields $E_{\perp}$ and $B$. The quantities $A_{i}$,
$E_{i}$ and $E_{\parallel}$ are then not independent variables, but are
defined in terms of $E_{\perp}$ and $B$ by the gauge condition (23), the
decomposition (20) and equation (22) respectively. Also the temporal component
$A_{0}$ of the gauge connection is a dependent variable given by the
expression (26).

##  3 The symplectic structure

The most familiar route to determining the symplectic structure of a theory
with constraints, like the Yang-Mills-Chern-Simons theory, is to identify a
set of configuration space coordinates like $A_{\mu}$ and determine the
corresponding canonical momenta $\Pi^{\mu}=\frac{\partial{\cal
L}}{\partial\dot{A_{\mu}}}$ from the Lagrangian ${\cal L}$. However, the
canonical Poisson bracket
$\\{\Pi^{\mu}(x),A_{\nu}(x^{\prime})\\}_{\mathrm{PB}}=\delta^{\mu}_{\nu}\delta(x-x^{\prime})$
between the coordinates and the momenta would not be consistent because of the
constraints on the latter. This can be dealt with by imposing additional
constraints (i.e. a gauge condition $\phi_{A}=0$), chosen so that the Poisson
bracket is non-degenerate on the space spanned by the constraints. There is
then an explicit procedure for constructing the Dirac modification
$\\{.\,,.\\}$ of the Poisson bracket $\\{.\,,.\\}_{\mathrm{PB}}$, so that it
will be consistent with the constraints and fullfil all the other requirements
on the symplectic structure on the phase space [5].

This procedure is not obvious to implement in the present case because of the
implicit definition (20) of our coordinate $E_{\perp}$. We will therefore
follow a different strategy and determine the Dirac bracket $\\{.\,,.\\}$
‘directly’ by requiring that it reproduces the correct Hamiltonian equations
of motion (16).

Clearly

$\left\\{\int d^{2}x\sqrt{g}\,\mathrm{Tr}(SB),\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(S^{\prime}B)\right\\}=0$ (27)

for any c-number Lie algebra-valued scalar fields $S$ and $S^{\prime}$.999This
would come out of the Dirac procedure since the Poisson bracket between
$A_{i}$ and a gauge condition formulated purely in terms of the connection
vanishes. The Dirac bracket thus agrees with the Poisson bracket and vanishes
when evaluted between functionals of $A_{i}$ only. The rest of the symplectic
structure can now be described by giving the bilinear functionals $f$ and $g$
defined by

$\displaystyle f[S;S^{\prime}]$ $\displaystyle=$ $\displaystyle\left\\{\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(SE_{\perp}),\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(S^{\prime}B)\right\\}$ (28) $\displaystyle
g[S;S^{\prime}]$ $\displaystyle=$ $\displaystyle\left\\{\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(SE_{\perp}),\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(S^{\prime}E_{\perp})\right\\}.$ (29)

To determine $f$, we start by rewriting

$\displaystyle\int d^{2}x\sqrt{g}\,\mathrm{Tr}(V^{i}E_{i})$ $\displaystyle=$
$\displaystyle\int d^{2}x\sqrt{g}\,\mathrm{Tr}({\cal W}E_{\perp})$ (31)
$\displaystyle-\frac{\lambda^{2}k}{\pi}\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(D_{i}V^{i}D^{-2}B),$

where

${\cal W}=\hat{\epsilon}^{ij}D_{i}V_{j}-i[D^{-2}D_{i}V^{i},B]$ (32)

for any c-number Lie algebra-valued vector field $V^{i}$. Thus

$\left\\{\int d^{2}x\sqrt{g}\,\mathrm{Tr}(V^{i}E_{i}),\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(S^{\prime}B)\right\\}=f[{\cal W};S^{\prime}].$
(33)

If we now apply this with $V_{i}=E_{i}$ (although this is not a c-number
field), we get ${\cal W}={\cal W}_{E}$, where

$\displaystyle{\cal W}_{E}$ $\displaystyle=$
$\displaystyle\hat{\epsilon}^{ij}D_{i}E_{j}-i[D^{-2}D_{i}E^{i},B]$ (34)
$\displaystyle=$
$\displaystyle\hat{\epsilon}^{ij}D_{i}E_{j}+\frac{\lambda^{2}k}{\pi}i[B,D^{-2}B].$
(35)

Here we have used the constraint (7) in the last step. This gives

$\left\\{\frac{1}{2}\int d^{2}x\sqrt{g}\,\mathrm{Tr}(E^{i}E_{i}),\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(S^{\prime}B)\right\\}=f\left[{\cal
W}_{E};S^{\prime}\right],$ (36)

where the prefactor $\frac{1}{2}$ in the first argument of the Dirac bracket
compensates for the integrand being of order $2$ in $E_{i}$. Since

$\left\\{\frac{1}{2}\int d^{2}x\sqrt{g}\,\mathrm{Tr}(BB),\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(S^{\prime}B)\right\\}=0$ (37)

we see that

$\left\\{H,\int d^{2}x\sqrt{g}\,\mathrm{Tr}(S^{\prime}B)\right\\}=\int
d^{2}x\sqrt{g}\,\mathrm{Tr}\left((\hat{\epsilon}^{ij}D_{i}E_{j}+i[B,\Lambda])S^{\prime}\right)$
(38)

as required provided that the functional $f$ is given by

$\displaystyle f[S;S^{\prime}]$ $\displaystyle=$ $\displaystyle\left\\{\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(SE_{\perp},\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(S^{\prime}B)\right\\}$ (39) $\displaystyle=$
$\displaystyle\lambda^{2}\int d^{2}x\sqrt{g}\,\mathrm{Tr}(SS^{\prime})$ (40)

and

$\Lambda=\frac{\lambda^{2}k}{\pi}D^{-2}B.$ (41)

The functional $g$ is now constrained by the requirement that the symplectic
structure obey the Jacobi identity, i.e. that it is given by a closed two-form
on the phase space. This forces $g$ to vanish identically so that

$\left\\{\int d^{2}x\sqrt{g}\,\mathrm{Tr}(SE_{\perp}),\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(S^{\prime}E_{\perp})\right\\}=0.$ (42)

The equations (27), (39) and (42) define the symplectic structure on the phase
space. The Poisson-Dirac bracket for quantities involving covariant
derivatives can be found by choosing a gauge and expressing the connection
$A_{i}$ in terms of the magnetic field strength $B$. An alternative gauge
invariant argument is to use the relation

$\delta B=\hat{\epsilon}^{ij}D_{i}\delta A_{j}$ (43)

for the variation of $B$ induced by an arbitrary variation $\delta A_{j}$. It
follows that the equations (27) and (39) are consistent with

$\displaystyle\left\\{\int d^{2}x\sqrt{g}\,\mathrm{Tr}(SE_{\perp}),\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(V^{\prime j}A_{j})\right\\}$ $\displaystyle=$
$\displaystyle\int
d^{2}x\sqrt{g}\hat{\epsilon}^{ij}\,\mathrm{Tr}(V^{\prime}_{j}D_{i}D^{-2}S)$
(44) $\displaystyle\left\\{\int d^{2}x\sqrt{g}\,\mathrm{Tr}(V^{i}A_{i}),\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(V^{\prime j}A_{j})\right\\}$ $\displaystyle=$
$\displaystyle 0$ (45)

for any c-number Lie algebra-valued scalar and vector fields $S$, $V^{i}$ and
$V^{\prime i}$.

As a check of the above results for the symplectic structure, it would be
gratifying to verify that also the second equation in (16) comes out
correctly. It follows from equation (33) that

$\displaystyle\Bigl{\\{}\frac{1}{2}\int d^{2}x\sqrt{g}\,\mathrm{Tr}(BB)$
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!,\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(V^{\prime i}E_{i})\Bigr{\\}}$ (46)
$\displaystyle=$ $\displaystyle-f[{\cal W};B]$ (47) $\displaystyle=$
$\displaystyle-\lambda^{2}\int
d^{2}x\sqrt{g}\,\mathrm{Tr}\left((\hat{\epsilon}^{ij}D_{i}V^{\prime}_{j}-i[D^{-2}D_{i}V^{\prime
i},B])B\right)$ (48) $\displaystyle=$ $\displaystyle-\lambda^{2}\int
d^{2}x\sqrt{g}\hat{\epsilon}^{ij}\,\mathrm{Tr}(V^{\prime}_{i}D_{j}B).$ (49)

If we now define the auxiliary functional $c$ as

$c[V;V^{\prime}]=\left\\{\int d^{2}x\sqrt{g}\,\mathrm{Tr}(V^{i}E_{i}),\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(V^{\prime j}E_{j})\right\\}$ (50)

for arbitrary c-number Lie algebra-valued vector fields $V$ and $V^{\prime}$
and apply this with $V^{i}=E^{i}$ (although this again is not a c-number
field) we have

$\left\\{\frac{1}{2}\int d^{2}x\sqrt{g}\,\mathrm{Tr}(E^{i}E_{i}),\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(V^{\prime j}E_{j})\right\\}=c[E;V^{\prime}]$ (51)

and thus

$\left\\{H,\int d^{2}x\sqrt{g}\,\mathrm{Tr}(V^{\prime j}E_{j})\right\\}=-\int
d^{2}x\sqrt{g}\hat{\epsilon}^{ij}\,\mathrm{Tr}(V^{\prime}_{i}D_{j}B)+\frac{1}{\lambda^{2}}c[E;V^{\prime}].$
(52)

So to reproduce the second equation in (16), we must have

$c[E;V^{\prime}]=-\frac{\lambda^{4}k}{\pi}\int
d^{2}x\sqrt{g}\,\mathrm{Tr}\left(\hat{\epsilon}^{ij}E_{i}V^{\prime}_{j}\right)+i\lambda^{2}\int
d^{2}x\sqrt{g}\,\mathrm{Tr}\left([\Lambda,E^{i}V^{\prime}_{i}]\right),$ (53)

with $\Lambda$ given by equation (41).

Unfortunately, while in principle determined by our previous results, a direct
computation of the functional $c$ appears tedious. We will therefore content
ourselves with the abelian case. Here

$E_{i}=\hat{\epsilon}_{i}{}^{j}\nabla_{j}E_{\perp}+\frac{\lambda^{2}k}{\pi}\nabla^{-2}\nabla_{i}B$
(54)

which gives

$c[V;V^{\prime}]=-\frac{\lambda^{4}k}{\pi}\int
d^{2}x\sqrt{g}\hat{\epsilon}^{ij}\,\mathrm{Tr}(V_{i}V^{\prime}_{j})$ (55)

so that

$c[E;V^{\prime}]=-\frac{\lambda^{4}k}{\pi}\int
d^{2}x\sqrt{g}\hat{\epsilon}^{ij}\,\mathrm{Tr}(E_{i}V^{\prime}_{j})$ (56)

in agreement with the abelian version of equation (53).

##  4 The Hamiltonian

By using equations (20), (21) and (22) and partial integrations, the
Hamiltonian $H$ can be written in terms of the variables $E_{\perp}$ and $B$
as

$\displaystyle H$ $\displaystyle=$ $\displaystyle\frac{1}{2\lambda^{2}}\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(BB+E_{i}E^{i})$ (57) $\displaystyle=$
$\displaystyle\frac{1}{2\lambda^{2}}\int
d^{2}x\sqrt{g}\,\mathrm{Tr}\Bigl{(}BB-E_{\perp}D^{2}E_{\perp}$ (59)
$\displaystyle\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;-[E_{\perp},B]D^{-2}[E_{\perp},B]-\frac{\lambda^{4}k^{2}}{\pi^{2}}BD^{-2}B\Bigr{)}.$

This is non-polynomial in $B$, which enters implicitly through the connection
$A_{i}$ in the covariant derivative $D_{i}$. To expand the Hamiltonian as a
power series in the coupling constant $\lambda$, we introduce the canonically
normalized variables $b$ and $e$ through

$\displaystyle B$ $\displaystyle=$ $\displaystyle\lambda b$ (60)
$\displaystyle E$ $\displaystyle=$ $\displaystyle\lambda e.$ (61)

Just like $B$ and $E_{\perp}$, the rescaled variables $b$ and $e$ are Darboux
coordinates on the phase space:

$\displaystyle\left\\{\int d^{2}x\sqrt{g}\,\mathrm{Tr}(Se),\int
d^{2}x\sqrt{g}(S^{\prime}b)\right\\}$ $\displaystyle=$ $\displaystyle\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(SS^{\prime})$ (62) $\displaystyle\left\\{\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(Sb),\int d^{2}x\sqrt{g}(S^{\prime}b)\right\\}$
$\displaystyle=$ $\displaystyle 0$ (63) $\displaystyle\left\\{\int
d^{2}x\sqrt{g}\,\mathrm{Tr}(Se),\int d^{2}x\sqrt{g}(S^{\prime}e)\right\\}$
$\displaystyle=$ $\displaystyle 0$ (64)

for any c-number Lie algebra-valued scalar fields $S$ and $S^{\prime}$.

We can now express the Hamiltonian as a power series in $\lambda$. Choosing
e.g. Coulomb gauge we have from equation (24) that

$A_{i}=\hat{\epsilon}_{i}{}^{j}\nabla_{j}\left(-\lambda\nabla^{-2}b-\frac{i\lambda^{2}}{2}\hat{\epsilon}^{kl}\nabla^{-2}[\nabla_{k}\nabla^{-2}b,\nabla_{l}\nabla^{-2}b]+{\cal
O}(\lambda^{3})\right).$ (65)

This gives

$\displaystyle H$ $\displaystyle=$ $\displaystyle\frac{1}{2}\int
d^{2}x\sqrt{g}\,\mathrm{Tr}\Bigl{(}bb-e\nabla^{2}e\Bigr{)}$ (70)
$\displaystyle+i\lambda\int
d^{2}x\sqrt{g}\hat{\epsilon}^{ij}\,\mathrm{Tr}\Bigl{(}[\nabla_{i}e,\nabla_{j}e]\nabla^{-2}b\Bigr{)}$
$\displaystyle+\frac{\lambda^{2}}{2}\int
d^{2}x\sqrt{g}\,\mathrm{Tr}\Bigl{(}-[e,b]\nabla^{-2}[e,b]-[e,\nabla_{i}\nabla^{-2}b][e,\nabla^{i}\nabla^{-2}b]$
$\displaystyle\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;-2[\nabla_{i}e,\nabla_{j}e]\nabla^{-2}[\nabla^{i}\nabla^{-2}b,\nabla^{j}\nabla^{-2}b]\Bigr{)}$
$\displaystyle+{\cal O}(\lambda^{3}).$

The Chern-Simons level $k$ will only affect terms of order $\lambda^{4}$ or
higher.

The abelian theory is of course free and has the bilinear harmonic oscillator
Hamiltonian

$H=\frac{1}{2}\int
d^{2}x\sqrt{g}\,\mathrm{Tr}\left(b(1-\frac{\lambda^{4}k^{2}}{\pi^{2}}\nabla^{-2})b+e(-\nabla^{2})e\right).$
(71)

Excitations which are eigenfunctions to the spatial Laplacian $-\nabla^{2}$
with eigenvalue $p^{2}$ then have energy

$\displaystyle E$ $\displaystyle=$
$\displaystyle\sqrt{(1-\frac{\lambda^{4}k^{2}}{\pi^{2}}\nabla^{-2})(-\nabla^{2})}$
(72) $\displaystyle=$ $\displaystyle\sqrt{p^{2}+m^{2}}$ (73)

with the mass gap

$m=\frac{\lambda^{2}k}{\pi}.$ (74)

as expected.

## References

  * [1] J. F. Schonfeld, “A Mass Term for Three-Dimensional Gauge Fields,” Nucl. Phys. B 185 (1981), 157-171.
  * [2] R. Jackiw and S. Templeton, “How Superrenormalizable Interactions Cure their Infrared Divergences,” Phys. Rev. D 23 (1981), 2291.
  * [3] S. Deser, R. Jackiw and S. Templeton, “Three-Dimensional Massive Gauge Theories,” Phys. Rev. Lett. 48 (1982) 975.
S. Deser, R. Jackiw and S. Templeton, “Topologically Massive Gauge Theories,”
Annals Phys. 140 (1982) 372.

  * [4] D. Karabali, V. P. Nair and A. Yelnikov, “The Hamiltonian Approach to Yang-Mills (2+1): An Expansion Scheme and Corrections to String Tension,” Nucl. Phys. B 824 (2010), 387-414.
  * [5] P. A. M. Dirac, “Lectures on Quantum Mechanics,” Belfer Graduate School of Science, Yeshiva University (1964).