# A variational characterization of Einstein–Brillouin–Keller quantization

Kai Cieliebak and Urs Frauenfelder

##  1 Introduction

In 1917, Albert Einstein presented an intrinsic formulation of the
Bohr–Sommerfeld quantization conditions for arbitrary integrable systems [7].
In 1926, Léon Brillouin derived Einstein’s conditions from the recently found
Schrödinger equation [4]. In 1958, Joseph Keller rediscovered Einstein’s
conditions, enhanced by a Maslov correction term involving half-integer
quantum numbers [10, 11]. These enhanced conditions became known as
Einstein–Brillouin–Keller (EBK) quantization.

EBK quantization expresses the spectrum by quantizing the actions of all
invariant tori (see §2.1) and is defined only for integrable systems. On the
other hand, the Gutzwiller trace formula [8] expresses the spectrum in terms
of periodic orbits, which also makes sense for some non-integrable systems.
This raises the question of how to describe, in the integrable case, the EBK
spectrum in terms of periodic orbits. This question is addressed e.g. in [2,
3] via complex periodic orbits.

In this paper we address this question via an object from symplectic geometry,
the marked action spectrum of a hypersurface $S$ in an exact symplectic
manifold (see §2.2). This encodes the actions of periodic orbits on $S$
together with their free homotopy classes. An old question in symplectic
geometry asks to which extent a hypersurface is determined by its marked
action spectrum, generalizing the classical question to which extent a
Riemannian manifold is determined by its marked length spectrum. In [5] a
positive answer is given if $S$ is a level set of an integrable Hamiltonian
system. Using this, we deduce expressions of the EBK spectrum in terms of the
marked action spectrum. We will ignore the Maslov corrections for most of this
paper and indicate the required adjustments in §9.

To state the results, consider a toric Hamiltonian, i.e. a Hamiltonian
$H:\mathbb{C}^{n}\to\mathbb{R}$ of the form

$H(z)=f\Bigl{(}\frac{1}{2}|z_{1}|^{2},\dots,\frac{1}{2}|z_{n}|^{2}\Bigr{)}$

for a smooth function $f:\mathbb{R}_{+}^{n}\to\mathbb{R}_{+}$. Assume in
addition that $f$ is homogeneous of some degree $d>0$, so that it is uniquely
determined by its regular level set $N=f^{-1}(1)\subset\mathbb{R}_{+}^{n}$.
Our first result is

######  Theorem 1.1.

In the setup above, assume in addition that the points with nonvanishing Gauss
curvature are dense in $N$. Then the EBK spectrum of $H$ can be constructed
from the marked action spectrum of the hypersurface $H^{-1}(1)$.

The proof of this theorem gives explicit formulas for reconstructing $N$ from
the marked action spectrum (Proposition 4.1), and for constructing the EBK
spectrum from $N$ (Corollary 4.2).

If $N$ is strictly convex or concave we get more explicit expressions.

######  Theorem 1.2.

In the setup above, assume in addition that $N$ is strictly convex. Let
$\mathcal{A}\subset\mathbb{Z}^{n}\times\mathbb{R}_{+}$ be the marked action
spectrum of $H^{-1}(1)$. Then the EBK spectrum of $H$ consists of the values

$E_{m}=\Bigl{(}\sup_{(k,a)\in\mathcal{A}}\frac{\hbar\langle
m,k\rangle}{a}\Bigr{)}^{d},\qquad m\in\mathbb{N}_{0}^{n}.$

For $N$ strictly concave an analogous formula holds with $\inf$ instead of
$\sup$.

An interesting limiting case of this theorem is a system of $n$ uncoupled
harmonic oscillators described by a linear function
$f(p)=\omega_{1}p_{1}+\cdots+\omega_{n}p_{n}$. We explain in Example 4.4 how
the correct spectrum can be recovered from Theorem 1.2 in this case.

Disk billiard.  The second half of this paper studies in more detail a
particular integrable system, the billiard on the round disk
$D_{R}\subset\mathbb{R}^{2}$ of radius $R$. A direct computation (see [14] or
§5) shows that its EBK spectrum consists of the values

$E_{m,n}=\frac{\hbar^{2}F_{m,n}^{2}}{2R^{2}},\qquad m\in\mathbb{Z},\
n\in\mathbb{N}_{0},$

where $F=F_{m,n}$ is the unique solution of the equation

$\sqrt{F^{2}-m^{2}}-m\arccos(m/F)=\pi n.$

On the other hand, V. Ramos has shown that the disk billiard can be described
by concave toric Hamiltonian and above [13]. We verify in §6 that applying
Theorem 1.2 to this Hamiltonian recovers the energies $E_{m,n}$ above.

Our final result concerns the number theoretic properties of the EBK spectrum
of the disk billiard, or equivalently of the numbers $F_{m,n}$.

######  Theorem 1.3.

All the numbers $F_{m,n}$ are transcendental. Moreover, if Schanuel’s
conjecture (see §7) is true, then for each $n\in\mathbb{N}$ the set
$\\{F_{m,n}\mid m\in\mathbb{N}_{0}\\}$ is algebraically independent.

##  2 EBK quantization of toric domains

###  2.1 EBK quantization

Let us start by recalling EBK quantization, ignoring Maslov corrections which
will be discussed in §9.

Consider an exact symplectic manifold $(V,\omega=d\lambda)$ of dimension $2n$
and an autonomous Hamilton function $H:V\to\mathbb{R}$. It gives rise to the
Hamiltonian vector field $X_{H}$ defined by $\omega(\cdot,X_{H})=dH$, the
Poisson bracket $\\{F,G\\}=\omega(X_{F},X_{G})$, and the Hamiltonian system
$\dot{x}=X_{H}(x)$. We assume that this system is integrable. This means that
there exist $n$ integrals of motions $P_{1},\dots,P_{n}$ such that
$\\{P_{i},P_{j}\\}=0$ for all $i,j$ and the differentials
$dP_{1},\dots,dP_{n}$ are linearly independent on an open dense set $U\subset
V$. We assume further that the function

$P=(P_{1},\dots,P_{n}):V\to\mathbb{R}^{n}$

is proper (i.e. preimages of compact sets are compact). By the
Arnold–Liouville theorem [1], $U$ is then fibered by invariant Lagrangian
$n$-tori on which the Hamiltonian flow is linear. Moreover, the Hamiltonian
$H$ has the form

$H=f(P_{1},\dots,P_{n})=f\circ P$

for a function $f:P(V)\to\mathbb{R}$. We call an invariant torus $L\subset V$
quantized if

$\int_{\gamma}\lambda\in 2\pi\hbar\mathbb{Z}\quad\text{for each loop $\gamma$
on $L$,}$ (1)

where as usual $\hbar$ denotes Planck’s constant divided by $2\pi$. Since
$\lambda|_{L}$ is closed, is suffices to check this condition for $n$ loops
forming a $\mathbb{Z}$-basis of $H_{1}(L)$. The EBK approximation to the
energy spectrum of the quantized system, or simply the EBK spectrum, is then
defined as

$\displaystyle\sigma_{\text{EBK}}(H)$ $\displaystyle:=\\{E\in\mathbb{R}\mid
H^{-1}(E)\text{ contains a quantized invariant torus}\\}$
$\displaystyle=\\{f(p)\mid\text{ the invariant torus $L_{p}=\\{P=p\\}$ is
quantized}\\}.$

The EBK spectrum has the following obvious properties:

(Scaling)

For a smooth function $\phi:\mathbb{R}\to\mathbb{R}$,

$\sigma_{\text{EBK}}(\phi\circ H)=\phi\bigl{(}\sigma_{\text{EBK}}(H)\bigr{)}.$

(Invariance)

For a diffeomorphism $\psi:V\to V$ such that $\psi^{*}\lambda-\lambda$ is
exact,

$\sigma_{\text{EBK}}(H\circ\psi)=\sigma_{\text{EBK}}(H).$

(Continuity)

The map $H=f\circ P\mapsto\sigma_{\text{EBK}}(H)$ is continuous with respect
to the strong $C^{0}$-distance on functions $f:P(V)\to\mathbb{R}$ and the
Hausdorff distance on subsets of $\mathbb{R}$.

###  2.2 Toric Hamiltonians

Now we specialize to our main class of examples. Consider $\mathbb{C}^{n}$
with coordinates $z_{j}=x_{j}+iy_{j}$ and the standard Liouville form
$\lambda=\sum_{j}x_{j}dy_{j}$. A toric Hamiltonian is a Hamiltonian
$H:\mathbb{C}^{n}\to\mathbb{R}$ of the form

$H(z)=f\Bigl{(}\frac{1}{2}|z_{1}|^{2},\dots,\frac{1}{2}|z_{n}|^{2}\Bigr{)}$

for a smooth function $f:\mathbb{R}_{+}^{n}\to\mathbb{R}_{+}$. The sublevel
set $\\{H\leq 1\\}\subset\mathbb{C}^{n}$ is known as a toric domain. The
corresponding Hamiltonian system $\dot{z}=i\nabla H(z)$ has the solutions

$z_{j}(t)=z_{j}(0)e^{i\partial_{j}f(p)t},$

where $\partial_{j}f(p)$ denotes the $j$-th partial derivative of $f$ at the
point $p=(\frac{1}{2}|z_{1}|^{2},\dots,\frac{1}{2}|z_{n}|^{2})$. So the system
is integrable with integrals $P_{j}=\frac{1}{2}|z_{j}|^{2}$. Note that
$(P_{j},\phi_{j})$, where $\phi_{j}$ is the argument of $z_{j}$, are action-
angle coordinates. Each $p=(p_{1},\dots,p_{n})\in\mathbb{R}_{+}^{n}$ defines
an invariant torus

$L_{p}=\Bigl{\\{}z\in\mathbb{C}^{n}\;\Bigl{|}\
\frac{1}{2}|z_{j}|^{2}=p_{j}\text{ for }j=1,\dots,n\Bigr{\\}}.$

The loops

$\gamma_{j}(t)=(\sqrt{2p_{1}},\dots,\sqrt{2p_{j}}e^{it},\dots,\sqrt{2p_{n}}),\qquad
t\in[0,2\pi]$

generate $H_{1}(L_{p})$ and have action $\int_{\gamma_{j}}\lambda=2\pi p_{j}$,
so $L_{p}$ is quantized iff

$p_{j}=\hbar m_{j}\quad\text{with }m_{j}\in\mathbb{N}_{0},\quad j=1,\dots,n.$

Hence, the EBK spectrum is given by

$\sigma_{\text{EBK}}=\\{f(\hbar m)\mid m\in\mathbb{N}_{0}^{n}\\}.$ (2)

Note that a solution $z$ of the Hamiltonian system is $T$-periodic iff
$\partial_{j}f(p)T=2\pi k_{j}$ for integers $k_{1},\dots,k_{n}$, so the flow
on a torus $L_{p}$ is $T$-periodic iff

$\frac{T}{2\pi}\nabla f(p)=k\in\mathbb{Z}^{n}.$

The $T$-periodic solutions $z$ on such a rational torus have action

$\int_{z}\lambda=2\pi\langle p,k\rangle.$

We define the marked action spectrum of the energy hypersurface $H^{-1}(1)$ by

$\mathcal{A}:=\\{(k,\langle p,k\rangle)\mid
k\in\mathbb{Z}^{n},\,f(p)=1,\,\nabla f(p)\sim k\\}.$

It records the actions (up to the factor $2\pi$) of orbits on rational tori
together with their homology classes in $H_{1}(T^{n})=\mathbb{Z}^{n}$. Using
the unit normal vector $n(p)=\nabla f(p)/|\nabla f(p)|$, we can write it in
terms of the level set $N=f^{-1}(p)$ as

$\mathcal{A}=\\{(k,\langle p,k\rangle)\mid k\in\mathbb{Z}^{n},\,p\in
N,\,n(p)\sim k\\}.$

######  Example 2.1 (Uncoupled harmonic oscillators).

A system of $n$ uncoupled harmonic oscillators of angular frequencies
$\omega_{1},\dots,\omega_{n}$ is described by the Hamilton function

$H(z)=\sum_{j=1}^{n}\frac{\omega_{j}}{2}|z_{j}|^{2}.$

This is a toric domain as above with the linear function
$f(p)=\omega_{1}p_{1}+\dots+\omega_{n}p_{n}$, so its EBK energy spectrum is

$\sigma_{\text{EBK}}=\Bigl{\\{}\sum_{j=1}^{n}\hbar\omega_{j}m_{j}\mid
m\in\mathbb{N}_{0}^{n}\Bigr{\\}}.$

This agrees with the quantum mechanical spectrum up to replacing $m_{j}$ by
$m_{j}+\frac{1}{2}$. Note that for $n\geq 2$ and rationally independent
frequencies $\omega_{1},\dots,\omega_{n}$ the flow on each $n$-torus $L_{p}$
is irrational (i.e. its orbits are dense), so the formula for the EBK energy
spectrum arises entirely from irrational tori. The only periodic orbits of
energy $E$ in this case are the $n$ orbits where all but one coordinate is
zero. We will see in Example 4.4 how the EBK spectrum can nonetheless be
derived from the marked action spectrum by an approximation process. A direct
expression of the EBK spectrum in terms of periodic orbits is given in [6]
using Tate Rabinowitz Floer homology.

##  3 Legendre transformations

In this section we describe Legendre-type transformations in three different
settings and discuss their properties and relationships.

###  3.1 Strictly convex functions

The Legendre transform of a strictly convex continuous function
$f:\mathbb{R}^{n}\to\mathbb{R}$ is the function
$\mathcal{L}f:\mathbb{R}^{n}\to\mathbb{R}$ defined by

$\mathcal{L}f(q):=\sup_{p\in\mathbb{R}^{n}}\Bigl{(}\langle
p,q\rangle-f(p)\Bigr{)}.$

It is well-known (see e.g. [1]) that $\mathcal{L}f$ is again strictly convex
and $\mathcal{L}(\mathcal{L}f)=f$. If $f$ is of class $C^{2}$ and the matrix
$d^{2}f(p)$ of second derivatives is positive definite for each $p$, then

$\mathcal{L}f(q)=\langle p_{0},q\rangle-f(p_{0})=\langle(\nabla
f)^{-1}(q),q\rangle-f\bigl{(}(\nabla f)^{-1}(q)\bigr{)}$

for the unique $p_{0}\in\mathbb{R}^{n}$ with $\nabla f(p_{0})=q$.

###  3.2 Homogeneous convex functions

Suppose now that $f:\mathbb{R}^{n}\to\mathbb{R}$ is smooth and homogeneous of
degree $1$,

$f(tp)=tf(p)\quad\text{for all }t>0,$

and its level set $N:=f^{-1}(1)$ is regular and strictly convex (i.e. all its
normal curvatures are positive, where $N$ is cooriented by $\nabla f$). Then
$f$ is only weakly convex and the Legendre transform from §3.1 is not
applicable. Instead, we define its homogeneous Legendre transform
$Lf:\mathbb{R}^{n}\to\mathbb{R}$ by

$Lf(q):=\sup_{p\in N}\Bigl{(}\langle p,q\rangle-f(p)\Bigr{)}+1=\sup_{p\in
N}\langle p,q\rangle.$

It follows that

$Lf(q)=\langle p_{0},q\rangle=\langle n^{-1}(q/|q|),q\rangle$

for the unique $p_{0}\in N$ with $n(p_{0})\sim q$, where $\sim$ denotes
positive proportionality and $n$ is the Gauss map

$n:N\to S^{n-1},\qquad p\mapsto\frac{\nabla f(p)}{|\nabla f(p)|}.$

###  3.3 Hypersurfaces

Since the homogeneous function $f$ in §3.2 and its level set $N=f^{-1}(1)$
determine each other, one can reformulate the homogeneous Legendre transform
in terms of the hypersurface $N$. This transformation has been introduced in
greater generality in [5] and we recall here the relevant definitions and
results.

Consider a compact smooth hypersurface $N\subset\mathbb{R}^{n}$ with Gauss map
$n:N\to S^{n-1}$ associating to $p\in N$ its outward pointing unit normal
vector $n(p)$. The Gauss curvature at $p\in N$ is given by $K(p):=\det Dn(p)$.
We do not require $N$ to be convex. For $p\in N$ with $\langle
p,n(p)\rangle\neq 0$ we define

$L(p):=\frac{n(p)}{\langle p,n(p)}.$ (3)

######  Lemma 3.1 ([5]).

Assume that the points with nonvanishing Gauss curvature are dense in $N$.
Then the set of nice points

$N^{\prime}:=\bigl{\\{}p\in N\mid K(p)\neq 0,\,\langle p,n(p)\rangle\neq
0,\,L^{-1}(L(p))=\\{p\\}\bigr{\\}}$

is dense in $N$ and the map (3) defines an embedding
$L:N^{\prime}\hookrightarrow\mathbb{R}^{n}$. Its image $L(N^{\prime})$
consists again of nice points and $L(L(p))=p$ for each $p\in N^{\prime}$. Thus
$L(L(N^{\prime}))=N^{\prime}$, so the hypersurface $N$ can be reconstructed
from its Legendre transform $L(N^{\prime})$ by

$N=\overline{L(L(N^{\prime}))}.$

######  Lemma 3.2 ([5]).

Let $N\subset\mathbb{R}^{n}$ be a smooth, compact, strictly convex
hypersurface enclosing the origin. Then $N=N^{\prime}$ consists of nice
points, $L(N)$ is again a smooth, compact, strictly convex hypersurface
enclosing the origin, and

$L(N)=\bigl{\\{}q\in\mathbb{R}^{n}\mid\sup_{p\in N}\langle p,q\rangle=1\\}.$

###### Proof.

Everything except the last assertion is proved in [5]. The last assertion
holds because $\sup_{p\in N}\langle p,q\rangle=\langle p_{0},q\rangle$ for the
unique $p_{0}\in N$ with $n(p_{0})\sim q$, thus $\sup_{p\in N}\langle
p,q\rangle=\langle p_{0},q\rangle=1$ iff $q=\frac{n(p_{0})}{\langle
p_{0},n(p_{0})\rangle}\in L(N)$. ∎

Lemmas 3.1 and 3.2 immediately imply

######  Corollary 3.3.

Consider a $1$-homogeneous function $f:\mathbb{R}^{n}\to\mathbb{R}$ with
strictly convex compact level set $N=f^{-1}(1)$ and its homogeneous Legendre
transform $Lf$ as in §3.2. Then

$L(N)=(Lf)^{-1}(1),$

$Lf:\mathbb{R}^{n}\to\mathbb{R}$ is again $1$-homogeneous with strictly convex
compact level sets, and

$L(Lf)=f.$

######  Remark 3.4.

It immediately follows from the proofs that Lemma 3.2 and Corollary 3.3 remain
true with “convex” replaced by “concave” and “sup” replaced by “inf” in the
definitions of $L(N)$ and $Lf$.

##  4 Recovering the EBK spectrum from the marked action spectrum

###  4.1 General toric domains

Consider a hypersurface $N\subset\mathbb{R}^{n}$. Recall from §2.2 its marked
action spectrum

$\mathcal{A}=\\{(k,\langle p,k\rangle)\mid k\in\mathbb{Z}^{n},\,p\in
N,\,n(p)\sim k\\}.$

######  Proposition 4.1 ([5]).

Assume that the points with nonvanishing Gauss curvature are dense in $N$.
Then $N$ can be recovered from its marked action spectrum as the closure of
the Legendre transform

$N=\overline{L(M^{\prime})},$

where $M^{\prime}$ is the set of nice points in

$M:=\overline{\Bigl{\\{}\frac{k}{a}\;\Bigl{|}\;(k,a)\in\mathcal{A},\,a\neq
0\Bigr{\\}}}.$

###### Proof.

For the reader’s convenience let us recall the proof from [5]. For $p\in N$
with $n(p)\sim k\in\mathbb{Z}^{n}$ and $a=\langle p,k\rangle\neq 0$ we have
$\frac{k}{a}=\frac{n(p)}{\langle p,n(p)\rangle}=L(p)$. Thus

$M=\overline{\\{L(p)\mid p\in N^{\prime}\text{ rational
}\\}}=\overline{L(N^{\prime})},$

where $p\in N^{\prime}$ is called rational if $Tn(p)\in\mathbb{Z}^{n}$ for
some $T>0$, and the last equality follows because the rational points are
dense in the set $N^{\prime}$ of nice points. Let $M^{\prime}\subset M$ be the
set near which $M$ is a nice hypersurface. Then $M^{\prime}=L(N^{\prime})$ and
therefore $N=\overline{L(M^{\prime})}$ by Lemma 3.1. ∎

Let $f:\mathbb{R}^{n}\to\mathbb{R}$ be the $1$-homogeneous function with level
set $N=f^{-1}(1)$, and $\sigma_{\text{EBK}}$ the EBK spectrum of the
associated Hamilton function $H(z)=f(|z_{1}|^{2}/2,\dots,|z_{n}|^{2}/2)$.
According to (2) we have $E\in\sigma_{\text{EBK}}$ iff $E=f(\hbar m)$ for some
$m\in\mathbb{N}_{0}^{n}$, or equivalently iff

$\frac{\hbar m}{E}=\frac{\hbar m}{f(\hbar m)}\in N.$

Thus Proposition 4.1 implies

######  Corollary 4.2.

For $N$ as in Proposition 4.1, the EBK spectrum of the associated
$1$-homogeneous Hamiltonian can be recovered from its marked action spectrum
by recovering $N$ as described in Proposition 4.1 and writing

$\sigma_{\text{EBK}}=\Bigl{\\{}E\in\mathbb{R}\;\Bigl{|}\;\text{there exists
}m\in\mathbb{N}_{0}^{n}\text{ such that }\frac{\hbar m}{E}\in N\Bigr{\\}}.$

###  4.2 Convex and concave toric domains

In the case that the hypersurface $N\subset\mathbb{R}^{n}$ is strictly convex
or concave the EBK spectrum can be given by an explicit formula in terms of
the marked action spectrum. For $\ell\in\mathbb{N}_{0}$ we denote

$\mathbb{N}_{\ell}^{n}:=\\{k\in\mathbb{Z}^{n}\mid k_{j}\geq\ell\text{ for all
$j$ and there exists $p\in N$ with $n(p)\sim k$}\\}.$

For $k\in\mathbb{N}_{\ell}^{n}$ we set

$a(k):=\langle p,k\rangle\text{ for the unique $p\in N$ with $n(p)\sim k$}.$

######  Proposition 4.3.

Let $N\subset\mathbb{R}^{n}$ be a strictly convex compact hypersurface and
$\mathcal{A}\subset\mathbb{Z}^{n}\times\mathbb{R}_{+}$ its marked action
spectrum. Then the EBK spectrum $\sigma_{\text{EBK}}$ of the associated
$1$-homogeneous Hamiltonian consists of the values

$E_{m}=\sup_{(k,a)\in\mathcal{A}}\frac{\hbar\langle
m,k\rangle}{a}=\sup_{k\in\mathbb{N}_{0}^{n}}\frac{\hbar\langle
m,k\rangle}{a(k)},\qquad m\in\mathbb{N}_{0}^{n}.$ (4)

Equivalently, $E_{m}$ can be characterized as the unique number such that

$\sup_{\ell\in\mathbb{N}}\inf_{k\in\mathbb{N}_{\ell}^{n}}\Bigl{(}Ea(k)-\hbar\langle
m,k\rangle\Bigr{)}=\begin{cases}+\infty&\text{for
}E>E_{m},\cr-\infty&\text{for }E<E_{m}.\end{cases}$ (5)

For $N$ strictly concave analogous formulas hold with $\sup$ and $\inf$
exchanged.

###### Proof.

The EBK spectrum $\sigma_{\text{EBK}}$ consists of the energies $E_{m}$,
$m\in\mathbb{N}_{0}^{n}$, that are uniquely characterized by the following
equivalent conditions:

$\frac{\hbar m}{E_{m}}\in
N=\overline{L(M^{\prime})}\Longleftrightarrow\sup_{q\in
M}\Bigl{\langle}\frac{\hbar
m}{E_{m}},q\Bigr{\rangle}=1\Longleftrightarrow\sup_{(k,a)\in\mathcal{A}}\Bigl{\langle}\frac{\hbar
m}{E_{m}},\frac{k}{a}\Bigr{\rangle}=1.$

Here the first characterization follows from Proposition 4.1 and Corollary
4.2, the first equivalence from Lemma 3.2, and the second equivalence from the
definition on $M$ in Proposition 4.1. Solving the last condition for $E_{m}$
yields formula (4).

To prove (5), note that for $k\in\mathbb{N}_{\ell}^{n}$ we have $a(k)=\langle
p,k\rangle\geq\max_{j}p_{j}k_{j}\geq c\ell$ with $c:=\min_{p\in
N}\max_{j}p_{j}>0$. Consider now $E>E_{m}$. Then for each
$\ell\in\mathbb{N}_{0}$ we have

$\inf_{k\in\mathbb{N}_{\ell}^{n}}\Bigl{(}E-\frac{\hbar\langle
m,k\rangle}{a(k)}\Bigr{)}=E-\sup_{k\in\mathbb{N}_{\ell}^{n}}\frac{\hbar\langle
m,k\rangle}{a(k)}=E-E_{m}>0,$

where the last equality follows by replacing $k$ by large integer multiples
and making all its components nonzero. Combining these we get

$\inf_{k\in\mathbb{N}_{\ell}^{n}}\Bigl{(}Ea(k)-\hbar\langle
m,k\rangle\Bigr{)}=\inf_{k\in\mathbb{N}_{\ell}^{n}}\Bigl{(}a(k)\bigl{(}E-\frac{\hbar\langle
m,k\rangle}{a(k)}\bigr{)}\Bigr{)}\geq c\ell(E-E_{m}),$

which becomes $+\infty$ in the supremum over $\ell\in\mathbb{N}$. This proves
the case $E>E_{m}$ in (5), and the case $E<E_{m}$ is analogous. ∎

######  Example 4.4 (Uncoupled harmonic oscillators (continued)).

In the notation of Example 2.1, suppose that the frequency vector
$\omega=(\omega_{1},\dots,\omega_{n})$ is rational, i.e.
$T\omega\in\mathbb{N}_{0}^{n}$ for some $T>0$. Then all orbits are periodic
and Proposition 4.3 is still applicable. Thus $\sigma_{\text{EBK}}$ consists
of the values $E_{m}=\sup_{k\in\mathbb{N}_{0}^{n}}\frac{\hbar\langle
m,k\rangle}{a(k)}$ where $k=\ell T\omega$ for some $\ell\in\mathbb{N}$ and
$a(k)=\langle p,k\rangle=\ell T\langle p,\omega\rangle=\ell T$ for some (hence
any) $p\in N$. This implies $E_{m}=\hbar\langle m,\omega\rangle$ and we
recover the EBK spectrum from Example 2.1. Continuity of the spectrum now
implies that this formula for $\sigma_{\text{EBK}}$ continues to hold for all
(not necessarily rational) frequency vectors, although Proposition 4.3 is not
applicable to such $\omega$ due to the lack of rational invariant tori.

So far in this section we have assumed that $f:\mathbb{R}^{n}\to\mathbb{R}$ is
$1$-homogeneous. If $f$ is instead $d$-homogeneous for some $d>0$, then
$\bar{f}=f^{1/d}$ is $1$-homogeneous with the same level set
$\bar{f}^{-1}(1)=f^{-1}=N$. By the (Scaling) property of the EBK spectrum,
Corollary 4.2 and Proposition 4.3 thus carry over to the $d$-homogeneous case
by replacing the EBK energies given there by their $d$-th powers. In
particular, we obtain Theorems 1.1 and 1.2 from the Introduction.

##  5 Billiard on the disk

A well studied integrable system is the billiard on the closed disk
$D_{R}\subset\mathbb{R}^{2}$ of radius $R>0$; see e.g. [14] and the references
therein for its classical and quantum mechanical treatment. In this section we
indicate how to derive its EBK spectrum and verify Proposition 4.3 on this
example.

Without loss of generality we set the mass of the billiard ball to $1$, so
that the system is described by the free Hamiltonian

$H(q,p)=\frac{1}{2}|p|^{2}.$

An invariant torus $L$ in an energy hypersurface $H^{-1}(E)$ consists of all
billiard trajectories of speed $\sqrt{2E}$ with a given reflection angle
$\alpha\in(0,\pi)$ with the boundary. These trajectories are tangent to the
circle of radius $S=R|\cos\alpha|$. They are periodic if

$\alpha=\frac{k\pi}{\ell}\qquad\text{with
}k\in\mathbb{Z},\,\ell\in\mathbb{N},$ (6)

and otherwise the are dense in the annulus $A=\\{S\leq|q|\leq R\\}$. The
projection $(q,p)\to q$ defines a map $L\to A$ which is 2-1 over the interior
of $A$, corresponding to the two possible values of $p$ for trajectories of
$L$ through a point $q$ (cf. [7]).

Let us first assume that $\alpha\leq\pi/2$. The first homology of $L$ is
generated by two loops: the loop $\gamma$ running once around the inner circle
given by

$q(t)=R\cos\alpha e^{it},\ p(t)=i\sqrt{2E}e^{it},\qquad t\in[0,2\pi],$

and the loop $\delta$ running along a straight line from the inner circle to
the outer one with one lift and back with the other lift whose first half is
given by

$q(t)=t,\ p(t)=\sqrt{2E}e^{i\arcsin(R\cos\alpha/t)},\qquad t\in[S,R].$

Short computations with $\lambda=p\,dq$ yield the action integrals

$\int_{\gamma}\lambda=2\pi
R\sqrt{2E}\cos\alpha,\qquad\int_{\delta}\lambda=2R\sqrt{2E}(\sin\alpha-\alpha\cos\alpha).$

The EBK quantization conditions are

$\int_{\gamma}\lambda=2\pi\hbar
m,\quad\text{and}\quad\int_{\delta}\lambda=2\pi\hbar n,\qquad
m,n\in\mathbb{N}_{0},$

where $m,n\geq 0$ because $\cos\alpha\geq 0$ and
$\sin\alpha-\alpha\cos\alpha\geq 0$ for all $\alpha\in[0,\pi/2]$. Abbreviating

$F:=R\sqrt{2E}/\hbar,$

these conditions become

$F\cos\alpha=m\quad\text{and}\quad F(\sin\alpha-\alpha\cos\alpha)=\pi n,\qquad
m,n\in\mathbb{N}_{0}$

and they give rise to the equation

$\sqrt{F^{2}-m^{2}}-m\arccos(m/F)=\pi n.$ (7)

An angle $\alpha\in(\pi/2,\pi)$ leads to the same quantization condition as
the angle $\pi-\alpha\in(0,\pi/2)$. The EBK spectrum thus consists of the
values

$E_{m,n}=\frac{\hbar^{2}F_{m,n}^{2}}{2R^{2}},$ (8)

where $F_{m,n}$ is the unique111See §8 for the proof of uniqueness. solution
of (7) for given $m,n\in\mathbb{N}_{0}$ and each energy value with $m>0$
appears with multiplicity $2$. Writing $F=kR$, this agrees with formula (4.19)
in [14] up to replacing $n$ by $n+3/4$.

A short computation shows that the periodic orbits of energy $E$ with angle
$\alpha$ satisfying (6) have action

$a(k,\ell)=2R\sqrt{2E}\ell\sin\Bigl{(}\frac{\pi k}{\ell}\Bigr{)}.$ (9)

We will see in the next section how the EBK spectrum can be derived from these
action values.

##  6 Disk billiard as a toric domain

In the beautiful paper [13] V. Ramos explains how the billiard on the disk can
be interpreted as a concave toric domain. For this, he first describes the
billiard on the unit disk $D_{1}\subset\mathbb{R}^{2}$ as the flow on the
boundary of the Lagrangian bidisk $D_{1}\times D_{1}\subset
T^{*}\mathbb{R}^{2}=\mathbb{R}^{4}$. Then he proves ([13, Theorem 3]) that the
interior of the Lagrangian bidisk $D_{1}\times D_{1}$ is symplectomorphic to
the concave toric domain

$X_{\Omega}=\Bigl{\\{}(z_{1},z_{2})\in\mathbb{C}^{2}\;\Bigl{|}\;\Bigl{(}\frac{\pi}{2}|z_{1}|^{2},\frac{\pi}{2}|z_{2}|^{2}\Bigr{)}\in\Omega\Bigr{\\}},$
(10)

where $\Omega\subset[0,\pi]^{2}$ is the domain bounded by the coordinate axes
and the curve $N\subset[0,\pi]^{2}$ parametrized by222 To simplify the
formulas we have replaced the angle $\alpha$ in [13] by $2\alpha$ and denoted
the domain $\Omega_{0}$ in [13] by $2\Omega$.

$\rho(\alpha)=\Big{(}\sin(\alpha)-\alpha\cos(\alpha),\sin(\alpha)+(\pi-\alpha)\cos(\alpha)\Big{)},\qquad\alpha\in[0,\pi].$

To determine the action spectrum, we compute

$\displaystyle\rho^{\prime}(\alpha)$ $\displaystyle=$
$\displaystyle\Big{(}\alpha\sin(\alpha),(\alpha-\pi)\sin(\alpha)\Big{)}.$

So the normal vector $n(p)$ at a point $p=(p_{1},p_{2})=\rho(\alpha)\in N$
satisfies

$n(p)\sim\Big{(}(\pi-\alpha)\sin(\alpha),\alpha\sin(\alpha)\Big{)}\sim\Big{(}\pi-\alpha,\alpha\Big{)}.$

The correponding orbits are periodic iff
$n(p)\sim(k_{1},k_{2})\in\mathbb{N}_{0}^{2}$, which can be solved for

$\alpha=\frac{k_{2}\pi}{k_{1}+k_{2}}\qquad\text{and}\qquad\pi-\alpha=\frac{k_{1}\pi}{k_{1}+k_{2}}.$

The corresponding action therefore becomes

$\displaystyle a(k_{1},k_{2})$ $\displaystyle=$ $\displaystyle
k_{1}p_{1}+k_{2}p_{2}$ $\displaystyle=$ $\displaystyle
k_{1}\Big{(}\sin(\alpha)-\alpha\cos(\alpha)\Big{)}+k_{2}\Big{(}\sin(\alpha)+(\pi-\alpha)\cos(\alpha)\Big{)}$
$\displaystyle=$ $\displaystyle(k_{1}+k_{2})\sin\bigg{(}\frac{\pi
k_{2}}{k_{1}+k_{2}}\bigg{)}$ $\displaystyle=$
$\displaystyle(k_{1}+k_{2})\sin\bigg{(}\frac{\pi}{2}\bigg{(}1+\frac{k_{2}-k_{1}}{k_{1}+k_{2}}\bigg{)}\bigg{)}$
$\displaystyle=$
$\displaystyle(k_{1}+k_{2})\cos\bigg{(}\frac{\pi(k_{2}-k_{1})}{2(k_{1}+k_{2}}\bigg{)}.$

Note that these actions agree up to a scaling factor with those for the disk
billiard given by (9) with $k=k_{2}$ and $\ell=k_{1}+k_{2}$.

The EBK spectrum can now be determined from these action values via
Proposition 4.3. We leave the straightforward computation to the reader and
instead determine the EBK spectrum via Corollary 4.2 (which we know yields the
same result). Thus $E\in\sigma_{\text{EBK}}$ iff there exists
$(m_{1},m_{2})\in\mathbb{N}_{0}^{2}$ such that
$\frac{\hbar}{E}(m_{1},m_{2})\in N$, i.e.

$m_{1}=\frac{E}{\hbar}(\sin\alpha-\alpha\cos\alpha),\qquad
m_{2}=\frac{E}{\hbar}\bigl{(}\sin\alpha+(\pi-\alpha)\cos\alpha\bigr{)}$

for some $\alpha\in[0,\pi]$. Setting

$F:=\frac{E\pi}{\hbar},$

the difference of the two equations yields $m_{2}-m_{1}=F\cos\alpha$, hence

$\alpha=\arccos\bigg{(}\frac{m_{2}-m_{1}}{F}\bigg{)}.$

Inserting this into the equation for $m_{1}$ gives

$\displaystyle\pi m_{1}$ $\displaystyle=$ $\displaystyle
F(\sin\alpha-\alpha\cos\alpha)$ $\displaystyle=$ $\displaystyle
F\biggl{(}\sqrt{1-\frac{(m_{2}-m_{1})^{2}}{F^{2}}}-\frac{m_{2}-m_{1}}{F}\arccos\Bigl{(}\frac{m_{2}-m_{1}}{F}\Bigr{)}\biggr{)}$
$\displaystyle=$
$\displaystyle\sqrt{F^{2}-(m_{2}-m_{1})^{2}}-(m_{2}-m_{1})\arccos\Bigl{(}\frac{m_{2}-m_{1}}{F}\Bigr{)}.$

With $n=m_{1}$ and $m=m_{2}-m_{1}$ this agrees with equation (7) for the EBK
spectrum of the disk billiard. Note that the actual energies $E=E_{m,n}$ of
the billiard on the unit disk are obtained from the unique solutions
$F=F_{m,n}$ of (7) using the relation (8) with $R=1$, i.e.
$E=\frac{\hbar^{2}F^{2}}{2}$, rather than the relation $E=\frac{\hbar F}{\pi}$
above. The additional factor $\pi$ arises because in (10) we have defined the
toric domain $X_{\Omega}$ using $\frac{\pi}{2}|z_{j}|^{2}$ rather than the
action coordinates $\frac{1}{2}|z_{j}|^{2}$. The remaining discrepancy is due
to the billiard Hamiltonian $\frac{1}{2}|p|^{2}$ and the $1$-homogeneous
Hamiltonian $|p|$ which we need to use in Corollary 4.2 and Proposition 4.3.

##  7 Transcendental numbers and Schanuel’s conjecture

In this section we recall some definitions and facts about transcendental
field extensions and Schanuel’s conjecture, see e.g. [9, Chapter 23] and [12,
Chapter 21].

Consider a field extension $K\subset L$. A finite subset
$\\{a_{1},\dots,a_{N}\\}\subset L$ is called algebraically independent over
$K$ if there exists no nontrivial polynomial $p\in K[x_{1},\ldots x_{N}]$ in
$N$ variables with coefficients in $K$ such that

$p(a_{1},\ldots,a_{N})=0.$

An arbitrary subset $A\subset L$ is called algebraically independent if each
finite subset of $A$ is algebraically independent. A transcendence basis of
$L$ over $K$ is a maximal algebraically independent subset of $L$.
Transcendence bases exist and all have the same cardinality, which is called
the _transcendence degree_ of $L$ over $K$ and denoted

$\mathrm{trdeg}(L/K).$

Thus the field extension is algebraic iff $\mathrm{trdeg}(L/K)=0$, and
$\\{a_{1},\dots,a_{N}\\}\subset L$ is algebraically independent iff
$\mathrm{trdeg}(K(a_{1},\dots,a_{N})/K)=N$. Here $K(A)\subset L$ denotes the
smallest subfield containing $K$ and a subset $A\subset L$. A basic property
of the transcendence degree is its additivity for field extensions $K\subset
L\subset M$,

$\mathrm{trdeg}(M/K)=\mathrm{trdeg}(M/L)+\mathrm{trdeg}(L/K).$ (11)

We will need the following easy consequence of this formula.

######  Lemma 7.1.

Let $K\subset L$ be a field extension and $A,B\subset L$ be subsets such that
each element of $A$ is algebraic over $K(B)$ and vice versa. Then

$\mathrm{trdeg}(K(A)/K)=\mathrm{trdeg}(K(B)/K).$

###### Proof.

Apply (11) to the field extensions $K\subset K(A)\subset K(A\cup B)$ and
$K\subset K(B)\subset K(A\cup B)$, noting that $K(A)\subset K(A\cup B)$ and
$K(B)\subset K(A\cup B)$ are algebraic and thus have transcendence degree
zero. ∎

From now on we will specialize to field extensions $K=\mathbb{Q}\subset
L\subset\mathbb{C}$ and drop the specification “over $\mathbb{Q}$”, writing
$\mathrm{trdeg}(L)$ for $\mathrm{trdeg}(L/\mathbb{Q})$ etc. We will need the
following special case of the Lindemann–Weierstrass Theorem (see e.g. [12,
Corollary 4.2]).

######  Theorem 7.2.

If $0\neq\alpha\in\mathbb{C}$ is algebraic, then $e^{a}$ is transcendental.

This corresponds to the case $N=1$ of the following conjecture.

Schanuel’s Conjecture.  _If $\alpha_{1},\ldots\alpha_{N}\in\mathbb{C}$ are
linearly independent over $\mathbb{Q}$, then_

$\mathrm{trdeg}\,\mathbb{Q}\big{(}\alpha_{1},\ldots\alpha_{N},e^{\alpha_{1}},\ldots,e^{\alpha_{N}}\big{)}\geq
N.$

We quote from the book of Ram Murty and Rath [12, Chapter 21]:

> “This conjecture is believed to include all known transcendence results as
> well as all reasonable transcendence conjectures on the values of the
> exponential function.”

##  8 Transcendence of the EBK spectrum of the disk billiard

We fix a positive rational number $n\in\mathbb{Q}\cap(0,\infty)$. We denote by

$\arccos\colon[-1,1]\to[0,\pi]$

the inverse function of $\cos|_{[0,\pi]}$. For $m\in\mathbb{N}_{0}$ we
consider the real numbers

$F_{m}:=F_{m,n}\in\mathbb{R}$

defined implicitly by equation (7),

$\sqrt{F^{2}_{m}-m^{2}}-m\arccos\bigg{(}\frac{m}{F_{m}}\bigg{)}=n\pi.$ (12)

That this equation has a unique solution can be seen as follows. For $m\geq 0$
consider the function

$\mathfrak{f}_{m}\colon[m,\infty)\to\mathbb{R},\qquad
x\,\mapsto\sqrt{x^{2}-m^{2}}-m\arccos\bigg{(}\frac{m}{x}\bigg{)}.$

Its derivative is given by

$\mathfrak{f}_{m}^{\prime}(x)=\frac{x}{\sqrt{x^{2}-m^{2}}}-\frac{m^{2}}{x^{2}\sqrt{1-\frac{{}^{2}}{x^{2}}}}=\frac{x^{2}-m^{2}}{x\sqrt{x^{2}-m^{2}}}=\frac{\sqrt{x^{2}-m^{2}}}{x},$

so $\mathfrak{f}_{m}$ is strictly increasing. Moreover, it satisfies

$\mathfrak{f}_{m}\big{(}m)=0,\qquad\lim_{x\to\infty}\mathfrak{f}_{m}(x)=\infty$

and thus gives rise to a bijection

$\mathfrak{f}_{m}\colon[m,\infty)\to[0,\infty).$

Therefore, we can set

$F_{m}=\mathfrak{f}_{m}^{-1}(n).$

From the above discussion we infer that

$F_{m}>m.$ (13)

Note that

$F_{0}=n\pi$ (14)

is a transcendental number, since $\pi$ is transcendental and $n$ is rational.
More generally, we have

######  Proposition 8.1.

For every $m\in\mathbb{N}_{0}$ the number $F_{m}$ is transcendental.

###### Proof.

In view of (14) we can assume without loss of generality that $m\geq 1$. We
argue by contradiction and assume that $F_{m}$ is algebraic. From (12) we
obtain

$\frac{m}{F_{m}}=\cos\bigg{(}\sqrt{\frac{F_{m}^{2}}{m^{2}}-1}-\frac{n\pi}{m}\bigg{)}.$
(15)

Since $F_{m}>m$ by (13), the assumption that $F_{m}$ is algebraic implies that
$i\sqrt{\tfrac{F_{m}^{2}}{m^{2}}-1}$ is an algebraic number different from
zero. From Theorem 7.2 we conclude that
$e^{i\sqrt{\frac{F_{m}^{2}}{m^{2}}-1}}$ is transcendental. Since $n$ is
rational and $m$ is an integer, this implies that

$\cos\bigg{(}\sqrt{\frac{F_{m}^{2}}{m^{2}}-1}-\frac{n\pi}{m}\bigg{)}=\frac{1}{2}\Bigg{(}\frac{e^{i\sqrt{\tfrac{F_{m}^{2}}{m^{2}}-1}}}{e^{i\frac{n\pi}{m}}}+\frac{e^{i\frac{n\pi}{m}}}{e^{i\sqrt{\tfrac{F_{m}^{2}}{m^{2}}-1}}}\Bigg{)}$

is transcendental as well. But then by (15) the number $F_{m}$ is
transcendental, contradicting our assumption. ∎

If we assume Schanuel’s conjecture we can say much more about these numbers.

######  Proposition 8.2.

Under the assumption that Schanuel’s conjecture is true it follows that the
set $\\{F_{m}\mid m\in\mathbb{N}_{0}\\}$ is algebraically independent.

###### Proof.

Under the assumption that Schanuel’s conjecture holds true we will show by
induction that for every $N\in\mathbb{N}_{0}$ the set

$\mathfrak{F}_{N}:=\\{F_{m}\mid 0\leq m\leq N\\}$

is algebraically independent. For $N=0$ the set $\mathfrak{F}_{0}$ consists of
the single element $F_{0}=n\pi$ which is transcendental, and therefore
$\mathfrak{F}_{0}$ is algebraically independent. It remains to carry out the
induction step. For this purpose, we assume that $\mathfrak{F}_{N-1}$ is
algebraically independent and we want to conclude that $\mathfrak{F}_{N}$ is
algebraically independent. The strategy to prove this is the following. We
abbreviate

$G_{m}:=\sqrt{\frac{F_{m}^{2}}{m^{2}}-1}$

and consider the set

$\mathfrak{G}_{N}:=\\{G_{m}\ \Big{|}\ 0\leq m\leq N\\}.$

We will first use the induction hypothesis to show that the set
$\mathfrak{G}_{N}$ is linearly independent over $\mathbb{Q}$, and then
conclude the induction step from Schanuel’s conjecture in view of the defining
equation (12).

Let us first show that $\mathfrak{G}_{N}$ is linearly independent over
$\mathbb{Q}$. We argue by contradiction and assume that this is not the case.
By induction hypothesis, $\mathfrak{F}_{N-1}$ is algebraically independent.
Since $G_{m}$ is an algebraic function of $F_{m}$ and vice versa, it follows
from Lemma 7.1 that the set $\mathfrak{G}_{N-1}$ is algebraically independent
as well. In particular, $\mathfrak{G}_{N-1}$ is linearly independent over
$\mathbb{Q}$. Hence, if $\mathfrak{G}_{N}$ is not linearly independent over
$\mathbb{Q}$, then there exist rational numbers $r_{0},\ldots
r_{N-1}\in\mathbb{Q}$ such that

$G_{N}=\sum_{m=0}^{N-1}r_{m}G_{m}.$ (16)

Since $\frac{m}{F_{m}}\in(0,1)$, we infer from (15) that

$G_{m}-\frac{n\pi}{m}\in\Big{(}0,\frac{\pi}{2}\Big{)}$

and therefore

$\sin\Big{(}G_{m}-\frac{n\pi}{m}\Big{)}\in(0,1).$

Hence we obtain from (15) the formula

$e^{iG_{m}-\frac{in\pi}{m}}=\frac{m}{F_{m}}+i\sqrt{1-\frac{m^{2}}{F_{m}^{2}}}.$
(17)

From this we infer

$\displaystyle\frac{2}{\sqrt{\bigg{(}\sum_{m=0}^{N-1}r_{m}\sqrt{\frac{F_{m}^{2}}{m^{2}}-1}\bigg{)}^{2}+1}}$
$\displaystyle=$ $\displaystyle\frac{2N}{F_{N}}$ $\displaystyle=$
$\displaystyle 2\cos\Big{(}G_{N}-\frac{n\pi}{N}\Big{)}$ $\displaystyle=$
$\displaystyle\frac{e^{\frac{in\pi}{N}}}{e^{i\sum_{m=0}^{N-1}r_{m}G_{m}}}+\frac{e^{i\sum_{m=0}^{N-1}r_{m}G_{m}}}{e^{\frac{in\pi}{N}}}$
$\displaystyle=$
$\displaystyle\frac{e^{in\pi\big{(}\frac{1}{N}-\sum_{m=0}^{N-1}\frac{r_{m}}{m}\big{)}}}{\prod_{m=0}^{N-1}\bigg{(}\frac{m}{F_{m}}+i\sqrt{1-\frac{m^{2}}{F_{m}^{2}}}\bigg{)}^{r_{m}}}$
$\displaystyle+\frac{\prod_{m=0}^{N-1}\bigg{(}\frac{m}{F_{m}}+i\sqrt{1-\frac{m^{2}}{F_{m}^{2}}}\bigg{)}^{r_{m}}}{e^{in\pi\big{(}\frac{1}{N}-\sum_{m=0}^{N-1}\frac{r_{m}}{m}\big{)}}}$

Here the first equality follows from (16) and the definition of $G_{m}$, the
second one from (15) for $m=N$, the third one from Euler’s formular and (16),
and the fourth one from (17). This shows that the set $\mathfrak{F}_{N-1}$ is
algebraically dependent, in contradiction to the induction hypothesis. Hence
the set $\mathfrak{G}_{N}$ is linearly independent over $\mathbb{Q}$.

It follows that the set

$i\mathfrak{G}_{N}=\\{iG_{m}\mid 0\leq m\leq N\\}$

is linearly independent over $\mathbb{Q}$ as well. By (17), for each $0\leq
m\leq N$ the number $e^{iG_{m}}$ is algebraic over
$\mathbb{Q}(\mathfrak{F}_{N})$ and thus over $\mathbb{Q}(i\mathfrak{G}_{N})$.
Therefore, from Lemma 7.1 we conclude that

$\mathrm{trdeg}\,\mathbb{Q}\big{(}iG_{0},\ldots iG_{N},e^{iG_{0}},\ldots
e^{iG_{N}}\big{)}=\mathrm{trdeg}\,\mathbb{Q}\big{(}iG_{0},\ldots,iG_{N}\big{)}.$

Since we assume that Schanuel’s conjecture holds true, this implies that

$\mathrm{trdeg}\,\mathbb{Q}\big{(}iG_{0},\ldots,iG_{N}\big{)}\geq N+1.$

Hence the set $i\mathfrak{G}_{N}$, and therefore also the set
$\mathfrak{F}_{N}$, is algebraically independent. This proves the proposition.
∎

Propositions 8.1 and 8.2 together give Theorem 1.3 from the Introduction.

######  Remark 8.3.

It is interesting to observe that the action values (9) of the disk billiard
are all algebraic multiples of $R\sqrt{E}$, and the determine via Proposition
4.3 the EBK spectral values that are algebraically independent (assuming
Schanuel’s conjecture). This is possible because the formulas in Proposition
4.3 involve a limiting process (sup or inf).

######  Remark 8.4.

The quantum mechanical spectrum of the disk billiard is given by zeroes of
higher order Bessel functions [14]. In view of Proposition 8.2, one may wonder
whether these numbers are also algebraically independent.

##  9 Maslov shifts

In this section we discuss the effect of Maslov shifts on the various spectra.
For this, we replace the quantization condition (1) on a torus $L$ by

$\int_{\gamma_{j}}\lambda\in 2\pi\hbar(\mathbb{Z}+\mu_{j})\quad\text{for
}j=1,\dots,n.$

Here $\gamma_{1},\dots,\gamma_{n}$ is a basis of $H_{1}(L)$ and we are fixing
a vector (representing $1/4$ times the Maslov shifts)

$\mu=(\mu_{1},\dots,\mu_{n})\in\mathbb{R}^{n}.$

The shifted EBK spectrum is defined as before, using this quantization
condition. For a toric Hamiltonian as in §2.2 the quantization condition
becomes $p_{j}=\hbar(m_{j}+\mu_{j})$ and the EBK spectrum in (2) becomes

$\sigma_{\text{EBK}}^{\mu}=\\{f(\hbar(m+\mu))\mid m\in\mathbb{N}_{0}^{n}\\}.$

The shifted EKB spectrum of the uncoupled harmonic oscillators in Example 2.1
becomes

$\sigma_{\text{EBK}}^{\mu}=\Bigl{\\{}\sum_{j=1}^{n}\hbar\omega_{j}(m_{j}+\mu_{j})\mid
m\in\mathbb{N}_{0}^{n}\Bigr{\\}}.$

which for $\mu_{j}=1/2$ agrees with the quantum mechanical spectrum.

We shift the marked action spectrum in §2.2 to

$\mathcal{A}^{\mu}=\\{(k,\langle p+\mu,k\rangle)\mid k\in\mathbb{Z}^{n},\,p\in
N,\,n(p)\sim k\\}$

and use this to define

$M_{\mu}:=\overline{\Bigl{\\{}\frac{k}{a}\;\Bigl{|}\;(k,a)\in\mathcal{A}^{\mu},\,a\neq
0\Bigr{\\}}}.$

as in Proposition4.1. Since

$\displaystyle\Bigl{\\{}\frac{k}{a}\;\Bigl{|}\;(k,a)\in\mathcal{A}^{\mu}\Bigr{\\}}$
$\displaystyle=\Bigl{\\{}\frac{k}{\langle
p+\mu,k\rangle}\;\Bigl{|}\;k\in\mathbb{Z}^{n},\,p\in N,\,n(p)\sim k\Bigr{\\}}$
$\displaystyle=\Bigl{\\{}\frac{k}{\langle
p,k\rangle}\;\Bigl{|}\;k\in\mathbb{Z}^{n},\,p\in N+\mu,\,n(p)\sim
k\Bigr{\\}},$

the formula for recovering $N$ from the marked action spectrum in
Proposition4.1 becomes

$N+\mu=\overline{L(M_{\mu}^{\prime})}.$

Corollary 4.2 remains true with the shifted EBK spectrum

$\sigma_{\text{EBK}}^{\mu}=\Bigl{\\{}E\in\mathbb{R}\;\Bigl{|}\;\text{there
exists }m\in\mathbb{N}_{0}^{n}\text{ such that }\frac{\hbar(m+\mu)}{E}\in
N\Bigr{\\}}.$

Formula (4) in Proposition 4.3 becomes

$E_{m}^{\mu}=\sup_{(k,a)\in\mathcal{A}}\frac{\hbar\langle
m+\mu,k\rangle}{a}=\sup_{k\in\mathbb{N}_{0}^{n}}\frac{\hbar\langle
m+\mu,k\rangle}{a(k)},\qquad m\in\mathbb{N}_{0}^{n}.$

For the disk billiard in §5 we choose the shifts $0$ and $3/4$ for the loops
$\gamma$ and $\delta$, respectively, so the EBK quantization conditions become

$\int_{\gamma}\lambda=2\pi\hbar
m,\quad\text{and}\quad\int_{\delta}\lambda=2\pi\hbar\Bigl{(}n+\frac{3}{4}\Bigr{)},\qquad
m,n\in\mathbb{N}_{0}.$

Then equation (7) gets replaced by

$\sqrt{F^{2}-m^{2}}-m\arccos(m/F)=\pi(n+3/4),$

which for $F=kR$ agrees with formula (4.19) in [14]. This formula is
reproduced by the approach in §6 with the shifts $\mu_{1}=\mu_{2}=3/4$, i.e.
replacing $m_{j}$ by $m_{j}+3/4$ for $j=1,2$.

## References

  * [1] V.I. Arnold, _Mathematical methods of classical mechanics_ , second edition, Graduate Texts in Mathematics 60, Springer (1989).
  * [2] R. Balian, C. Bloch, _Distribution of eigenfrequencies for the wave equation in a finite domain: III. Eigenfrequency density oscillations_ Ann. Physics 69, 76–160 (1972).
  * [3] M. Berry, M. Tabor, _Closed orbits and the regular bound spectrum_ , Proc. Roy. Soc. London Ser. A 349, no. 1656, 101–123 (1976).
  * [4] L. Brillouin, _Remarques sur la mécanique ondulatoire_ , Journal de Physique et le Radium 7 (12), 353–368 (1926).
  * [5] K. Cieliebak, Symplectic boundaries: Closed characteristics and action spectra, PhD thesis ETH Zürich (1996).
  * [6] K. Cieliebak, U. Frauenfelder, _Spectral numbers in Tate Rabinowitz Floer homology_ , in preparation.
  * [7] A. Einstein, _Zum Quantensatz von Sommerfeld und Epstein (German) [On the quantum theory of Sommerfeld and Epstein]_ , Verhandlungen der Deutschen Physikalischen Gesellschaft 19 (9–10), 82–92, (1917).
  * [8] M. Gutzwiller, _Chaos in classical and quantum mechanics_ , Springer-Verlag, New York (1990).
  * [9] C. Karpfinger, K. Meyberg, _Algebra_ , 5th edition, Springer (2021).
  * [10] J. Keller, _Corrected Bohr–Sommerfeld quantum conditions for nonseparable systems_ , Annals of Physics 4, 180–188 (1958).
  * [11] J. Keller, S. Rubinow, _Asymptotic Solution of Eigenvalue Problems_ , Annals of Physics 9, 24–75 (1960).
  * [12] M. Ram Murty, P. Rath, _Transcendental Numbers_ , Springer (2014).
  * [13] V. Ramos, _Symplectic embeddings and the Lagrangian bidisk_ , Duke Math. J. 166, no. 9, 1703–1738 (2017).
  * [14] K. Weibert, Semiclassical quantization of integrable and chaotic billiard systems by harmonic inversion, PhD thesis Universität Stuttgart (2001).


# Mitigation of Adversarial Policy Imitation via Constrained Randomization of
Policy (CRoP)

Nancirose Piazza
SAIL Lab
University of New Haven
West Haven, CT, USA
<EMAIL_ADDRESS>&Vahid Behzadan
SAIL Lab
University of New Haven
West Haven, CT, USA
<EMAIL_ADDRESS>

###### Abstract

Deep reinforcement learning (DRL) policies are vulnerable to unauthorized
replication attacks, where an adversary exploits imitation learning to
reproduce target policies from observed behavior. In this paper, we propose
Constrained Randomization of Policy (CRoP) as a mitigation technique against
such attacks. CRoP induces the execution of sub-optimal actions at random
under performance loss constraints. We present a parametric analysis of CRoP,
address the optimality of CRoP, and establish theoretical bounds on the
adversarial budget and the expectation of loss. Furthermore, we report the
experimental evaluation of CRoP in Atari environments under adversarial
imitation, which demonstrate the efficacy and feasibility of our proposed
method against policy replication attacks.

##  1 Introduction

Deep Reinforcement Learning (DRL) is a learning framework for stochastic,
discrete-time decision-making leveraging neural networks for generalization
and function approximation. With the growing interest in DRL and its
integration in commercial and critical systems, the security of such
algorithms have become of paramount importance [1].

In tandem with DRL, similar advancements have been made in Imitation Learning
(IL) techniques that utilize expert demonstrations to learn and replicate the
expert’s behavior in sequential decision making tasks. Deep Q-Learning from
Demonstration (DQfD)[2] is an IL variant that has enabled DRL agents to
converge quicker to an optimal policy. However, recent work in [3] and [4]
demonstrate that IL can also be exploited by adversaries to replicate
protected policies from passive observation of the target’s behavior,
resulting in risks concerning intellectual property and adversarial
information gain for more effective active attacks. Current state of the art
in countering such attacks include watermarking [5][6], which enables the
post-attack identification of replicated policies. In this paper, we propose
an active mitigation technique against policy imitation attacks, named
Constrained Randomization of Policy (CRoP). The proposed technique is based on
intermittent randomization of a trained policy, constrained on a threshold for
maximum amount of acceptable loss in the expected return. The goal is to
increase the adversary’s imitation training cost, measured as the minimum
number of training iterations and observed demonstrations required for
training a replica that matches the target policy’s performance.

The main contributions of this paper are: (1) We propose and formulate CRoP as
a mitigation technique against adversarial policy imitation, (2) We present a
formal analysis of the bounds on expected loss of optimality under CRoP, (3)
We formally establish bounds on the adversary’s imitation cost induced by
CRoP. (3) We report the results of empirical evaulation of adversarial
imitation via DQfD against CRoP agents in classical DRL benchmarks, and
demonstrate the efficacy and feasibility of CRoP in those settings.

The remainder of this paper is organized as follow: Section (2) details
Constraint Randomization of Policy (CRoP) which analyzes the optimality of a
CRoP policy in relation to an optimal policy and describes CRoP’s impact upon
minimizing divergence objectives, and presents the minimal adversarial budget
induced by CRoP and analysis on expectation of loss. Section 3 provides
demonstrations of CRoP in three Atari benchmark environments with training and
test-time performance of adversarial imitation learning agents trained by an
expert policy induced by CRoP through DQfD, and Section 4 concludes the paper
with a summary of findings.

##  2 Constrained Randomization of Policy

In the remainder of this paper, we assume the target policy aims to solve a
Markov Decision Process (MDP) denoted by the tuple $<S,A,R,T,\gamma>$ where
$S$ is a finite state space, $A$ is a finite action space, $T$ defines the
environment’s transition probabilities, a discount value $\gamma\in[0,1)$, and
a reward function $R:S\times A\rightarrow[0,1]$. The solution to this MDP is a
policy $\pi:S\rightarrow A$ that maps states to actions. An agent implementing
a policy $\pi$ can measure the value of a state
$V(s)=\underset{a}{\max}(r_{s,a}+\gamma V(s^{\prime})$), where $s^{\prime}$ is
the next state. Similarly, the value of a state-action pair is given by
$Q(s,a)=\underset{a}{\max}(r_{s,a}+\gamma Q(s^{\prime},a^{\prime}))$ where
$s^{\prime}$ is the next state and $a^{\prime}$ is the next action.

Constrained Randomization of Policy (CRoP) is an action diversion strategy
from an optimal policy under constrained performance deviation from optimal.
Let $\hat{a}\in\hat{A}$ where $\hat{a}$ are candidate actions that satisfy
$Q(s,\pi(s))-Q(s,\hat{a}_{i})<\rho$ and $\hat{A}$ be the space of all
candidate actions for $s\in S$ excluding the optimal action $\pi(s)$. We
define CRoP as the function below:

$\small f(s)=\begin{cases}\pi(s)\quad Pr\text{ ($\delta$) or $\not\exists$
$\text{ }\hat{a}\in\hat{A}$}\\\ \hat{A}_{\hat{a}\sim U(\hat{A})}\quad
Pr(1-\delta)\\\ \end{cases}$ (1)

Where $U(\hat{A})$ is the uniform distribution over $\hat{A}$. This definition
of $\rho$ threshold is the difference of Q-values. We have three variations of
$\rho$ for CRoP: Q-value difference (Q-diff) as described in Equation 1, and
two measures inspired by the advantage function: advantage-inspired difference
(A-diff), and positive advantage-inspired difference (A+-diff). A-diff CRoP is
thus defined as:

$\small\tilde{A}(s_{t},a_{t})=Q(s_{t},a_{t})-V(s_{t-1})>-\rho$ (2)

A+-diff’s $\rho$ has the condition $\hat{A}(s_{t},a_{t})\geq 0$. A-diff and
A+-diff’s $\rho$ are interpreted as 1-step hindsight estimation which is
relevant to the trajectory taken instead of only pure future estimate as with
Q-diff, eg. played badly, now play safe vs. plan to feint ahead. However, the
selection of $\rho$ should consider estimation error due to either finite
training or function approximation.

Figure 1: Visualization that $\pi^{\prime}$ is an
$(\epsilon+\epsilon^{\prime}$)-optimal policy to $\pi^{*}$

We define $\epsilon$-optimal policies that are within $\epsilon$ neighborhood
of $V^{*}$, specifically $V^{*}-V^{\pi}<\epsilon$ for all $a\in A$ and $s\in
S$ at probability $(1-\delta)$. As illustrated in Figure 1, $\pi^{*}$ is the
optimal and greedy policy extracted from $V^{*}$ where $\pi$ is the extracted
policy from $V^{\pi}$ and $\pi^{\prime}$ is the extracted policy from
$V^{\pi^{\prime}}$, we see that $\pi{{}^{\prime}}$ may be expressed as an
$(\epsilon+\epsilon^{\prime})$-optimal policy. Since we do not assume $\pi$ to
be an optimal policy, it is possible for $\pi^{\prime}$ to be more optimal
than $\pi$. However, it is noteworthy that an evaluation of optimality based
on a (euclidean) measure to the value function does not imply extracted
policies with small error to $V^{*}$ resemble the optimal policy when assessed
on behavioral differences. Theorem1 establishes that CRoP policy $f$ is at
worst $(\epsilon$ \+ $\epsilon^{\prime})$-optimal to $Q^{*}$ at probability
$(1-\delta)$.

###### Theorem 1

Given $Q^{*}(s_{t},a_{t})-Q^{\pi}(s_{t},a_{t})<\epsilon^{\prime}$ at
probability $(1-\delta)$ and
$|Q^{\pi}(s_{t},a_{t})-Q^{\pi^{\prime}}(s_{t},a_{t})|\leq\epsilon$ for all
$s\in S$ and $a\in A$, then
$Q^{*}(s_{t},a_{t})-Q^{\pi^{\prime}}(s_{t},a_{t})\leq\epsilon+\epsilon^{\prime}$
at probability $(1-\delta)$. $\pi^{\prime}$ is an
$(\epsilon+\epsilon^{\prime})$-optimal policy at probability $(1-\delta)$.
[proof in supplement (0.1.1)]

IL has two common approaches: Behavioral Clones (BC) which are supervised
learners and inverse RL which finds a reward function to match the
demonstration. Work by [7] shows that: BC minimizes the KL divergence,
Generative Adversarial Imitation Learning (GAIL) [8] minimize the Jensen
Shannon divergence and DAgger [9] minimizes total variance. For BC, CRoP
affects the maximum likelihood in a similar manner to data poisoning attacks
like label flipping [10] or class imbalance. In regard to GAIL, the
discriminator from a GAN prioritizes expert experiences so unless modified for
decay when out-performed, additional penalty is given to the training policy.
Furthermore, when CRoP lowers the action distribution for $a^{*}$ according to
$\delta$ probability and increases the distribution for candidate actions, it
results in smaller maximal difference for DAgger.

###  2.1 Budget Analysis for Perfect Information Adversary

We measure the adversary’s budget in the sample quantity or trajectories that
it can acquire through a passive attack. Nair and Doshi-Velez [11] derive
upper and lower bounds on the sample complexity of direct policy learning and
model-based imitation learning in relaxed problem spaces. This follows the
research of RL sample efficiency and Offline RL[12]. However, in this work we
divert from a direct treatment of sample efficiency to consider information
optimality from observed target demonstration without environment interaction.
Consider the set $\mathcal{T}$ where $\tau_{i}$
$(\forall,\tau_{i}\in\mathcal{T})$ which is composed of a $T$-length chain of
$(s,a)$-pairs. Assume each $(s,a)$-pair has two possible outcomes, optimal at
$P(\delta)$ or sub-optimal at $P(1-\delta)$. Assume pair and trajectory
uniqueness, this would contain $2^{T}$ trajectories where $T$ is the length of
the horizon. To obtain optimal target $\pi$, we would require all trajectories
except the event of a complete sub-optimal trajectory $(1-\delta)^{T}$. Let an
adversary pull from $\mathcal{T}$. Group the desired $2^{T}-1$ trajectories in
set $\alpha$ and the worst event trajectory in set $\beta$. As an adversary
samples from $\mathcal{T}$, if they obtain an unseen desired trajectory
$\tau$, it is from $\alpha$ and is moved to their adversarial set
$\hat{\mathcal{T}}$. $\tau$ is then replaced in $\mathcal{T}$ but is no longer
unseen so if encountered again, it would be from $\beta$. Let $\tau_{w}$ be
the worst-case trajectory and $\hat{m}$ be the sum of the expected number of
trajectories for each sequential pull from $\mathcal{T}$. It follows that:

$\small\mathbb{E}[\hat{m}]=\overset{2^{T}-1}{\underset{n=1}{\sum}}\mathbb{E}[m_{n}]=\underset{1}{\overset{2^{T}-1}{{\sum}}}1/(1-P(\tau_{w})+\underset{\tau_{i}\in\hat{\mathcal{T}}}{\sum}-P(\tau_{i}))$
(3)

Intuitively, we see in the denominator the probability of pulling unseen
trajectories given the trajectories in $\hat{\mathcal{T}}$ and known
probability for all $\tau_{i}\in\hat{\mathcal{T}}$. We know an expectation on
expensive to obtain informative trajectories from $\pi$. However, typically an
adversary has a fixed budget and therefore we would want to know what to
expect given their budget $\mathbb{B}$, here we calculate for a budget
measured in optimal state-action pairs. To calculated an expected number of
optimal state-action pairs, we find a $t<T$ such that:

$\mathbb{B}\approx\underset{i=1}{\overset{t}{\sum}}\mathbb{E}[m_{i}]=\overset{t}{\underset{i=1}{\sum}}\frac{1}{\delta}$
(4)

Given we can reset to the previous state and resample until we obtain an
optimal state-action pair. This would give an expectation for the adversary to
obtain $t$ optimal state-action pairs with $\mathbb{B}$ budget. This can be
extended to the expectation of number of trajectories by approximating
$\mathbb{B}$, similar to Equation 4 where we find a $t<T$, but with Equation
3.

We can consider re-visitation as an expectation. Let $k$ = $\mathbb{E}[n]$
where $n$ is the number of state-action pair without re-visitation of maximum
length $T$ for a trajectory. Consider using $k$ as the new horizon, rounding
$k$ up to the nearest integer. We would expect that the expected number of
trajectories to obtain $\pi$ decrease because of shorter horizon. Using the
Markov Property, for some $\hat{X}$ non-negative, bounded random variable for
$N$ iterations, for any $t>0$

$P(\tau_{i})=(\delta)^{N}(1-\delta)^{k-N}\quad P(\hat{X}\geq
t)\leq\mathbb{E}[X]/t$

Like before let $\mathcal{T}$ be the set of all trajectories $\tau_{i}$ with
maximum length $T$, $\hat{\mathcal{T}}$ randomly sample from $\mathcal{T}$,
and $\hat{\tau}$ be the fragmented trajectory of all unique
$(s_{i},a_{i})\in\tau$, Assume for the instance below that $|\circ|$ refers to
cardinality and $k$ still refers to $\mathbb{E}[n]$, then the Markov
inequality and reverse Markov inequality for $0<t<k$ with $T$ as the maximum
trajectory length:

$\small P\big{(}|\hat{\tau}_{i}|<t\big{)}\geq 1-k/t\quad
P\big{(}|\hat{\tau}_{i}|\leq t\big{)}\leq(T-k)/(T-t)$ (5)

For interpretation, we can say we have an expectation on the number of
trajectories $\mathbb{E}[\hat{m}]$ with probability between $(1-k/t)$ to
$(T-k)/(T-t)$ given a fixed $t$ where $0<t<k$, which is a weak bound with lack
of information on variance.

###  2.2 Policy Evaulation and Expectation of Loss

We see that the Q-value under $f$ will be either equivalent or less than the
Q-value under target policy $\pi$ which dictates selected $a^{\prime}$.
Furthermore, the expected return $G^{f}_{t}$ for stochastic policy $f$ with
uniform sampling from $\hat{A}$ is expressed as the following:

$\small
G^{f}_{t}=\delta\underset{t=0,1,2...}{\overset{N}{\sum}}\gamma^{t}\bigg{[}r_{s_{t},a_{t}^{*}}\bigg{]}+\frac{1-\delta}{|\hat{A}|}\underset{t=0,1,2...}{\overset{N}{\sum}}\gamma^{t}\bigg{[}\underset{\hat{a_{t}}}{\sum}r_{s_{t},\hat{a}_{t}}\bigg{]}$
(6)

With Equation 6, $G^{f}_{t}$ is the weighted sum of an optimal expected return
at probability $\delta$ and the expected return across all rewards given by
candidate actions at probability $(1-\delta)$. Given $G^{*}_{t}$ and
$G^{f}_{t}$, the difference between the expected return in $Q$-value form is
exactly:

$\small
G^{*}_{t}-G^{f}_{t}=(1-\delta)\bigg{[}Q^{\pi}(s_{t},a_{t})-\mathbb{E}[Q^{f}(s_{t},\hat{a}_{t})]\bigg{]}$
(7)

Since $Q^{\pi}(s_{t},a_{t})-\mathbb{E}[Q^{f}(s_{t},\hat{a}_{t})]<\rho$, then
the expectation loss $G^{*}_{t}-G^{f}_{t}\leq(1-\delta)\rho\leq\rho$. This
expectation of loss is calculated from the current state’s forward estimation
of future reward. We see there exists an upperbound, call it $\mathbb{E}[L]$:

$\small\underset{t=0}{\overset{N}{\sum}}|Q^{\pi}(s_{t},a_{t})-\mathbb{E}[Q^{f}(s_{t},\hat{a}_{t})]|\leq
N\times(1-\delta)\rho\leq N\times\rho=\mathbb{E}[L]$ (8)

##  3 Experimental Evaluation

We investigate DQfD as our adversarial IL method and evaluate test-time and
training time performance across three Atari environments: Breakout, Cartpole,
and Space Invaders. We train DQfD agents under default parameters (supplied in
supplements) with CRoP induced demonstrations, a control DQfD agent, and a
default, double DQN (DDQN) agent which provided the expert demonstrations. The
results of a parameter search on trained DDQN policies from Stable-Baseline
Zoo [13] are in supplementary section 0.2.1. As expected, higher $\delta$
allows for higher values of $\rho$. The trade-off on $\delta$ and $\rho$ is
similar to an allowance of high or low variance in Q-value. The results,
illustrated in Figure 3, demonstrate that the performance of imitated policies
generally remain below their control DQfD agents for earlier spans of training
episodes. CRoP may induce variance similar to optimistic initialization, for
example, work by [14] and [15]. Figure 3 depicts the comparison of test-time
performance among agents trained with various values of $\delta$ and $\rho$.
We emphasize the constrains in CRoP are expected loss which are not true
performance loss. The table for test-time evaluation timestep counts and
timesteps with successful action diversion counts in the supplementary
material section 0.3.1. Many of the environments resulted in different
behaviors when induced by different variants of $\rho$.

(a) Cartpole

(b) Breakout

(c) SpaceInvaders

Figure 2: DQfD agents trained on CRoP-induced demonstration

(d) Breakout

(e) Cartpole

(f) SpaceInvaders

Figure 3: Test-time evaluation of imitated agents and target DDQN agent across
10 episodes

##  4 Conclusion

This study investigated the threat emanating from passive policy replication
attacks. We proposed CRoP as a mitigation technique against such attacks, and
analyzed its performance with regards to $\epsilon$-optimality, estimated
affect on adversarial cost, and the expectation of loss. Furthermore, we
empirically evaluated CRoP across 3 Atari game benchmarks, and verified the
efficacy and efficiency of CRoP against DQfD-based policy replication attacks.

## References

  * [1] V. Behzadan and A. Munir, “The faults in our pi stars: Security issues and open challenges in deep reinforcement learning,” _arXiv preprint arXiv:1810.10369_ , 2018.
  * [2] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, G. Dulac-Arnold, I. Osband, J. Agapiou, J. Z. Leibo, and A. Gruslys, “Deep q-learning from demonstrations,” 2017.
  * [3] V. Behzadan and W. Hsu, “Adversarial exploitation of policy imitation,” 2019.
  * [4] K. Chen, S. Guo, T. Zhang, X. Xie, and Y. Liu, “Stealing deep reinforcement learning models for fun and profit,” 2020.
  * [5] V. Behzadan and W. Hsu, “Sequential triggers for watermarking of deep reinforcement learning policies,” _arXiv preprint arXiv:1906.01126_ , 2019\.
  * [6] K. Chen, S. Guo, T. Zhang, S. Li, and Y. Liu, “Temporal watermarks for deep reinforcement learning models,” in _Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems_ , 2021, pp. 314–322.
  * [7] L. Ke, S. Choudhury, M. Barnes, W. Sun, G. Lee, and S. Srinivasa, “Imitation learning as $f$-divergence minimization,” 2020.
  * [8] J. Ho and S. Ermon, “Generative adversarial imitation learning,” 2016.
  * [9] S. Ross, G. J. Gordon, and J. A. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” 2011.
  * [10] H. Xiao, H. Xiao, and C. Eckert, “Adversarial label flips attack on support vector machines,” in _ECAI_ , 2012.
  * [11] Y. Nair and F. Doshi-Velez, “Pac bounds for imitation and model-based batch learning of contextual markov decision processes,” 2020.
  * [12] S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” 2020.
  * [13] A. Raffin, “Rl baselines zoo,” https://github.com/araffin/rl-baselines-zoo, 2018.
  * [14] M. Kamiura and K. Sano, “Optimism in the face of uncertainty supported by a statistically-designed multi-armed bandit algorithm,” _Biosystems_ , vol. 160, 08 2017.
  * [15] I. Szita and A. Lörincz, “Optimistic initialization and greediness lead to polynomial time learning in factored mdps,” in _Proceedings of the 26th International Conference On Machine Learning, ICML 2009_ , vol. 382, 06 2009, p. 126.

##  Appendix A Theorems

###  A.1 Theorem 1

In Equation 9 and 10, we state that $Q^{f}$ is $\epsilon^{\prime}$-optimal to
$Q^{*}$ at $(1-\delta)$ probability and $Q^{\pi^{\prime}}$ is
$\epsilon$-optimal to $Q^{f}$.

$0<Q^{*}(s_{t},a_{t})-Q^{f}(s_{t},a_{t})<\epsilon^{\prime}$ (9)

$|Q^{f}(s_{t},a_{t})-Q^{\pi^{\prime}}(s_{t},a_{t})|\leq\epsilon$ (10)

at a probability of $(1-\delta)$. Let

$Q_{diff}=Q^{*}(s_{t},a_{t})-Q^{f}(s_{t},a_{t})+|Q^{f}(s_{t},a_{t})-Q^{\pi^{\prime}}(s_{t},a_{t})|$

Given that $Q(s,a)\in(0,\frac{1}{1-\gamma})$, at $(1-\delta)$ probability:

$\small Q^{*}(s_{t},a_{t})-Q^{\pi^{\prime}}(s_{t},a_{t})\leq
Q_{diff}\leq\epsilon+\epsilon^{\prime}$ (11)

##  Appendix B Figures

###  B.1 Experimental Evaluation Figure - parameter search

Figure 4: Parameter search performance 5000 timesteps

##  Appendix C Tables

###  C.1 Experimental Evaluation Table - test-time timestep count

- | Q-value difference $\rho$ | Positive advantage-inspired $\rho$
---|---|---
env | $\delta$ | $\rho$ | succ. |  $\delta$ $\times$ T | T | $\delta$ | succ. |  $\delta$ $\times$ T | T
Breakout-v4 | 0.0 | 0.1 | 7812 | 8450 | 8450 | 0.0 | 9857 | 15412 | 14512
Breakout-v4 | 0.5 | 0.02 | 12056 | 25761 | 51686 | 0.4 | 12402 | 33658 | 56336
Cartpole-v0 | 0.7 | 0.01 | 1345 | 1979 | 2000 | 0.0 | 505 | 2000 | 2000
Cartpole-v0 | 0.7 | 0.01 | 1345 | 1979 | 2000 | 0.1 | 430 | 1746 | 1938
SpaceInvaders-v4 | 0.0 | 0.1 | 18963 | 18968 | 26038 | 0.0 | 10111 | 21190 | 21190
SpaceInvaders-v4 | 0.6 | 0.02 | 10281 | 10358 | 26038 |  |  |  |
- | Advantage-inspired $\rho$ |  |  |  |
env | $\delta$ | $\rho$ | succ. |  $\delta$ $\times$ T | T |  |  |  |
Breakout-v4 | 0.0 | 0.1 | 3238 | 3464 | 3464 |  |  |  |
Breakout-v4 | 0.0 | 0.1 | 3238 | 3464 | 3464 |  |  |  |
Cartpole-v0 | 0.0 | 0.02 | 279 | 2000 | 2000 |  |  |  |
Cartpole-v0 | 0.0 | 0.1 | 946 | 2000 | 2000 |  |  |  |
SpaceInvaders-v4 | 0.0 | 0.1 | 21706 | 21706 | 21706 |  |  |  |
SpaceInvaders-v4 | 0.7 | 0.15 | 7117 | 7117 | 23730 |  |  |  |
Table 1: Test-time evaluation timestep count over 10 episodes