Nicholas H. Barbara, Ruigang Wang, and Ian R. Manchester 11institutetext:
Australian Centre for Robotics, School of Aerospace, Mechanical and
Mechatronic Engineering, The University of Sydney, Sydney, NSW 2006,
Australia.
11email<EMAIL_ADDRESS>
# On Robust Reinforcement Learning with Lipschitz-Bounded Policy Networks

Nicholas H. Barbara  Ruigang Wang  Ian R. Manchester

###### Abstract

This paper presents a study of robust policy networks in deep reinforcement
learning. We investigate the benefits of policy parameterizations that
naturally satisfy constraints on their Lipschitz bound, analyzing their
empirical performance and robustness on two representative problems: pendulum
swing-up and Atari Pong. We illustrate that policy networks with small
Lipschitz bounds are significantly more robust to disturbances, random noise,
and targeted adversarial attacks than unconstrained policies composed of
vanilla multi-layer perceptrons or convolutional neural networks. Moreover, we
find that choosing a policy parameterization with a non-conservative Lipschitz
bound and an expressive, nonlinear layer architecture gives the user much
finer control over the performance-robustness trade-off than existing state-
of-the-art methods based on spectral normalization.

###### keywords:

reinforcement learning, robustness, Lipschitz networks

##  1 Introduction

Deep reinforcement learning (Deep RL) has been the driving force behind many
recent successes in learning-based control, including in discrete game-like
problems [17], robotic manipulation [12], and locomotion [22]. However, the
applicability of Deep RL to performance- and safety-critical systems is
currently limited by questions of its robustness [11]. It is well known that
neural networks can be highly sensitive to small input perturbations, making
policy networks learned via Deep RL potentially unrobust to disturbances,
noise, and targeted adversarial attacks [25, 6]. Despite sharing similar
sensitivity issues to neural classifiers, for which many robust neural
networks have recently been developed [3, 28, 30], the use of robust policy
architectures in Deep RL has not been widely studied.

Most common approaches to improving policy robustness in Deep RL are based on
adversarial training [20], where adversarial attacks on a policy’s inputs are
optimized during training to encourage the network to perform well under
perturbations. While this works well in applications where the structure of
the perturbations is always similar to those seen during training, adversarial
training only certifies a lower bound on a policy’s robustness. It is
therefore possible to find new, out-of-sample attacks that cause the policy to
fail [23]. An alternative strategy is to learn control policies with a
certified upper bound on their sensitivity to perturbations using methods like
randomized smoothing [13, 32] and loss-function regularization [19, 18]. These
methods bound the sensitivity of a learned policy during the training process.
It is interesting to ask whether, via careful choice of a policy’s
architecture and parameterisation, we can directly bound the sensitivity of a
policy independently of how it is trained.

Figure 1: Reinforcement learning and adversarial attacks.

One promising approach is to use neural network policies that are certifiably
robust to perturbations by construction. This can be achieved by constraining
the Lipschitz bound of the network. A neural network
$f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ is said to have an $\ell_{2}$
Lipschitz bound of $\gamma$ if

$\|f(x_{1})-f(x_{2})\|_{2}\leq\gamma\|x_{1}-x_{2}\|_{2},\quad\forall
x_{1},x_{2}\in\mathbb{R}^{n}$ (1)

The true $\ell_{2}$ Lipschitz constant, denoted $\mathrm{Lip}(f)$, is the
smallest $\gamma$ satisfying (1). Lipschitz-bounded networks are then
“smoother” and less sensitive to input perturbations since small variations to
their inputs will induce only small variations in their outputs. Despite the
wealth of recent work on constructing Lipschitz-bounded deep networks [3, 28,
30], to the best of our knowledge, the only method tested so far in Deep RL is
spectral normalization [4, 26] which is known to give conservative bounds on
the true Lipschitz constant of the network.

This paper therefore investigates the following questions:

  1. 1.

Can Lipschitz-bounded deep networks improve the empirically-observed
robustness of control policies in Deep RL?

  2. 2.

If so, does the policy network architecture matter? That is, do more
sophisticated policy parameterizations with less-conservative Lipschitz bounds
give finer control over the performance-robustness trade-off?

We provide an initial empirical study of Lipschitz-bounded policy networks in
Deep RL on two representative problems: pendulum swing-up, a classical
benchmark problem in control and RL; and Atari Pong, a simple proxy for
vision-based autonomous decision-making tasks. To the best of our knowledge,
our comparison of Lipschitz-bounded policy architectures in Deep RL is the
first of its kind.

##  2 Background and prior work

In this section, we first introduce the concepts of deep reinforcement
learning and adversarial attacks. We then provide an overview of recent
developments in robust neural networks with certifiable Lipschitz bounds.

###  2.1 Deep reinforcement learning

We introduce RL from a control-theoretic perspective (Fig. 1). Consider a
discrete-time nonlinear dynamical system

$\displaystyle x_{t+1}$ $\displaystyle=f(x_{t},u_{t},w_{t})$ (2)

with state vector $x_{t}\in\mathbb{R}^{n}$, controlled inputs
$u_{t}\in\mathbb{R}^{m}$, and disturbances $w_{t}\in\mathbb{R}^{p}$. The task
is to learn feedback control policies of the form $u_{t}=\kappa(x_{t};\theta)$
parameterized by $\theta\in\mathbb{R}^{q}$ which (locally and approximately)
solve

$\displaystyle\max_{\theta}\
R:=\mathop{\mathbb{E}}\left[\sum_{t=0}^{\infty}\rho^{t}r_{t}(x_{t},u_{t})\right]\;\text{s.t.}\;\
x_{t+1}=f(x_{t},u_{t},w_{t}),\ u_{t}=\kappa(x_{t};\theta),$ (3)

where $r_{t}$ is the instantaneous reward, $\rho\in(0,1)$ is a “discount
factor,” $R$ is the total expected reward, and the expectation
$\mathbb{E}[\cdot]$ is taken over some known distributions of disturbances
$w_{t}$ and initial conditions $x_{0}$. In Deep RL, the controller $\kappa$ is
a deep neural network (DNN) parameterized by $\theta$. In this paper we take
Proximal Policy Optimization (PPO) [24] as our training method due to its
simplicity, speed, and performance.

###  2.2 Adversarial attacks for RL

Given a learned policy $\kappa(\cdot\,;\theta)$, an adversarial attack is an
input sequence $v=(v_{0},v_{1},\ldots)$ with some restricted “attack size”
$\epsilon>0$ that is designed to reduce the expected cumulative reward of the
policy as much as possible. While attacks can be designed to change the
closed-loop behavior in many ways, the most common structure is an additive
perturbation to the policy input (i.e., perturbations to the state
measurements, Fig. 1) [11, 20]. The problem can be formulated as

$\displaystyle\min_{v}$ $\displaystyle R\quad\text{s.t.}\quad
x_{t+1}=f(x_{t},\kappa(x_{t}+v_{t};\theta),w_{t}),\quad\norm{v_{t}}\leq\epsilon,\;\forall
t,$ (4)

where $\norm{\cdot}$ can be any $p$-norm with $1\leq p\leq\infty$. If the
reward function $r_{t}$ and dynamic model $f$ are both differentiable and
known to the attacker, (4) can be solved by gradient descent methods. If part
of the dynamical system is unknown or not differentiable, solving (4) is
difficult or impossible and it is common instead to solve the simplified
problem:

$\max_{v_{t}}\quad\|\kappa(x_{t}+v_{t};\theta)-\kappa(x_{t};\theta)\|\quad\mathrm{s.t.}\quad\|v_{t}\|\leq\epsilon.$
(5)

That is, find an admissible attack $v_{t}$ which leads to large policy output
perturbation at each time step, regardless of its effect on the final reward
[11]. Problem (5) can be solved by attack methods like Projected Gradient
Descent (PGD) [14].

###  2.3 Lipschitz-bounded deep networks

The adversarial attack problem in (5) is exactly the calculation of the
(local) Lipschitz constant of the policy network $\kappa$. We can therefore
control the effect of adversarial attacks everywhere in state space by
bounding the global Lipschitz constant $\mathrm{Lip}(\kappa)\leq\gamma$.
Policy networks with smaller $\gamma$ are then “smoother” and are likely to be
more robust to small attacks than those with a large $\gamma$.

In deep RL, the policy network $\kappa$ is often parameterized by a multi-
layer perceptron (MLP) or a convolutional neural network (CNN) of the form of

$\kappa=g_{L}\circ\sigma\circ g_{L-1}\circ\cdots\circ\sigma\circ g_{1}$ (6)

where $\sigma$ is a fixed monotone and 1-Lipschitz scalar activation (e.g.,
ReLU, tanh, sigmoid, etc.) and $g_{k}$ is a linear layer
$g_{k}(x)=W_{k}x+b_{k}$ with $W_{k},b_{k}$ as the weights and biases,
respectively. The Lipschitz constraint for (6) is then

$\mathrm{Lip}(\kappa)=\sup_{J_{2},\ldots,J_{L}}\|W_{L}J_{L}W_{L-1}\cdots
J_{2}W_{1}\|_{2}\leq\gamma$ (7)

where $J_{k}$ is a diagonal matrix with $0\leq J_{k,ii}\leq 1$. Since it is
NP-hard to compute the exact Lipschitz constant [29], a practical approach is
to find an upper bound $\overline{\gamma}$ for $\mathrm{Lip}(\kappa)$ and then
impose the constraint $\overline{\gamma}=\gamma$. There are many bound
estimation methods resulting in different constructions of Lipschitz networks.
A metric to measure the expressive power of Lipschitz networks is _tightness_
, which is defined as $\underline{\gamma}/\gamma$ [30] with
$\underline{\gamma}$ as the empirical lower Lipschitz bound:

$\underline{\gamma}:=\max_{x\in\mathbb{X},\|v\|_{2}\leq\epsilon}\|\kappa(x+v)-\kappa(x)\|_{2}/\|v\|_{2}$
(8)

where $v\in\mathbb{R}^{n}$ and $\mathbb{X}\subset\mathbb{R}^{n}$ is some
compact region. Note that
$\mathrm{Lip}(\kappa)\in[\underline{\gamma},\gamma]$.

A simple Lipschitz bound estimation is the layer-wise spectral norm bound
$\overline{\gamma}_{s}=\prod_{k=1}^{L}\|W_{k}\|_{2}$. This approach utilizes
the fact that common activation functions $\sigma$ are 1-Lipschitz. Thus, we
can construct Lipschitz policy networks via scaling factors and 1-Lipschitz
linear layers, where three representative examples are given as follows.

  * $\bullet$

(SN) The spectral normalization (SN) layer [16] is a linear layer with weight
$W=(1/\rho)A$ where $A$ is a learnable weight, and $\rho$ is its maximal
singular value (i.e., $\|W\|_{2}=1$). Lipschitz networks composed of SN layers
often have quite loose Lipschitz bounds such that $\underline{\gamma}/\gamma$
is very small.

  * $\bullet$

(AOL) The almost orthogonal Lipschitz (AOL) layer [3] is a linear layer with
weight $W=AD$ where $A$ is a learnable matrix and $D$ is a diagonal matrix
with $D_{ii}=\sqrt{\sum_{j}|A^{\top}A|_{ij}}.$ The experimental results in [3]
show that the learned weight $W$ tends to have singular values close to 1,
which helps to improve the model tightness.

  * $\bullet$

(Cayley) To obtain an even tighter Lipschitz bound, an orthogonal layer was
proposed in [28] by leveraging the Cayley transform. Specifically, given a
free learnable weight $P\in\mathbb{R}^{n\times n}$, one first obtains a skew
symmetric matrix $A=P-P^{\top}$ and then constructs the weight matrix by
$W=(I-A)(I+A)^{-1}.$ It is easy to verify that $W^{\top}W=I$ and so all
singular values of $W$ are 1.

A much tighter bound estimation method was recently proposed in [8], which
explores both the monotonicity and Lipschitz properties of the activation
$\sigma$ by leveraging the integral quadratic constraint (IQC) framework [15].
Direct (i.e. unconstrained) parameterizations based on IQC were proposed in
[21] for deep equilibrium networks, in [2] for residual networks, and in [30]
for deep MLPs and CNNs. In particular, [30] shows that the IQC-based approach
can lead to a 1-Lipschitz nonlinear layer as follows.

  * $\bullet$

(Sandwich) The Sandwich layer is a 1-Lipschitz nonlinear layer of the form

$g(x)=\sqrt{2}A^{\top}\Psi\sigma\bigl{(}\sqrt{2}\Psi^{-1}Bx+b\bigr{)}$ (9)

where $Q=\begin{bmatrix}A&B\end{bmatrix}$ is a semi-orthogonal matrix
($QQ^{\top}=I$) parameterized by the Cayley transformation, and $\Psi$ is a
positive diagonal matrix parameterized via an exponential mapping. Based on
the above 1-Lipschitz layer, we can construct the $\gamma$-Lipschitz policy
network $\kappa$ via

$\kappa(x)=\sqrt{\gamma}g_{L}\circ g_{L-1}\circ\cdots\circ
g_{1}(\sqrt{\gamma}x).$ (10)

Note that (10) can be transformed back to the MLP form (6). In [30],
experimental results on image datasets show that (10) can achieve better
performance than 1-Lipschitz linear layers such as AOL [3] and Cayley [28]. It
is natural to ask whether similar results hold true for Deep RL.

##  3 Experimental Setup

We study two classic RL problems — pendulum swing-up and Atari Pong — to
investigate the research questions outlined in Section 1. Our code and
training details are available on GitHub111https://github.com/nic-
barbara/Lipschitz-RL-MJX${}^{\text{,}}$222https://github.com/nic-
barbara/Lipschitz-RL-Atari and in Appendix A, respectively.

Pendulum swing-up aims to swing a physical pendulum to its upright equilibrium
based on the quadratic reward
$r_{t}=-(\alpha_{t}^{2}+0.1\dot{\alpha}_{t}^{2}+0.001u_{t}^{2})$, where
$\alpha_{t}$ is the pendulum angle (wrapped to $[-\pi,\pi]$) and $u_{t}$ is
the pendulum torque. The optimal policy is well-known to have sharp decision
boundaries which make it susceptible to chattering and instability under small
measurement perturbations, delays, or uncertainty. We trained MLP policies (6)
without any bounds on the network Lipschitz constant, and Lipschitz-bounded
policies using the Sandwich parameterization (9). We investigated the
robustness of each policy to two sources of perturbations: a) sample delays;
and b) adversarial attacks with constrained $\ell_{2}$ norm. Adversarial
attacks were computed by solving (4) over a sequence of four 50-sample windows
with gradient descent.

Atari Pong is a video game in which two players each control a paddle that can
move up and down and try to deflect a puck into their opponent’s goal (Fig.
6). The reward is the net game score, with a maximum score of 21 goals to nil.
An automated “computer” player controls the left paddle and the RL policy
controls the right paddle. It is a commonly-studied benchmark in Deep RL [17,
23]. The game can be written as an RL problem (3) where the states $x_{t}$ are
grayscale gameplay images and the control actions $u_{t}$ are discrete paddle
movements. We trained a classic, unconstrained CNN model (6) and four
different Lipschitz-bounded policy architectures (SN, AOL, Sandwich, Cayley)
with various Lipschitz bounds. We compared the robustness of each policy
network to: a) uniform random noise; b) PGD attacks (5) with constrained
$\ell_{2}$ norm; and c) PGD attacks (5) with constrained $\ell_{\infty}$ norm.

##  4 Results and Discussion

We first study the advantages of Lipschitz-bounded policy networks in terms of
robustness to perturbations and adversarial attacks, using pendulum swing-up
as an illustrative example (Sec. 4.1). We then extend this study to vision-
based feedback control in Pong (Sec. 4.2) and compare the benefits of
different Lipschitz-bounded policy architectures. In Figures 4, 5, and Table
1, Lipschitz lower bounds $\underline{\gamma}$ were computed for each policy
by performing gradient ascent on the model inputs to maximize the local
Lipschitz constant.

###  4.1 Illustrative example — Pendulum swing-up

(a) Unconstrained policy landscape.

(b) Lipschitz-bounded policy landscape.

(c) Local Lipschitz (unconstrained).

(d) Local Lipschitz (Lipschitz-bounded).

Figure 2: Contours of control actions (a,b) and local Lipschitz bounds (c,d)
for an unconstrained (MLP) and a Lipschitz-bounded (Sandwich, $\gamma=4$)
policy show how Lipschitz bounds control a policy’s smoothness.

Let us first consider the effect of small Lipschitz bounds on unperturbed
policy networks. Figure 2 compares the policy landscape and (empirically-
estimated) local Lipschitz constant in phase space for an unconstrained MLP
policy and a Lipschitz-bounded policy composed of Sandwich layers with
$\gamma=4$. The unconstrained policy has sharp decision boundaries, either
side of which is flips the sign of the control torque (limited to $\pm 1$
N.m). While these sharp changes are optimal in the unperturbed case, it is
clear that any small uncertainty in the pendulum’s position or velocity will
cause the network to apply a drastically different control action, potentially
driving the system to instability. In contrast, imposing a bound on the
Lipschitz constant with Sandwich layers visibly smooths the decision
boundaries with negligible penalty to the final test reward ($-153$ for
unconstrained and $-157$ for Lipschitz-bounded).

(a) Unconstrained.

(b) Lipschitz-bounded.

(c) Unconstrained (0.1 s delay).

(d) Lipschitz-bounded (0.1 s delay).

(e) Unconstrained ($\ell_{2}$ attack, $\epsilon=0.11$).

(f) Lipschitz-bounded ($\ell_{2}$ attack, $\epsilon=0.11$).

Figure 3: Pendulum trajectories generated by unconstrained (MLP) and
Lipschitz-bounded (Sandwich, $\gamma=4$) policies in nominal operation (a,b),
with sample delays (c,d), and with $\ell_{2}$ adversarial attacks (e,f). Red
lines indicate the target.

Figure 3 illustrates the effect of this smoothing on each policy’s robustness
to sample delays and $\ell_{2}$-optimal adversarial attacks. The simulated
trajectories in Figures 3(a) and 3(b) indicate that both policy networks
perform similarly in nominal operation. However, when introducing a small
sample delay (2 time samples or 0.1 s, Figs. 3(c) & 3(d)) or a small
adversarial attack ($\epsilon=0.11$, Figs. 3(e) & 3(f)), the unconstrained
policy is unable to hold the pendulum stable and upright, whereas the
Lipschitz-bounded policy is successful and only exhibits minor oscillations
about the equilibrium under adversarial attacks.

(a) Rewards under sample delay.

(b) Rewards at 2-step delay (0.1 s).

(c) Reward under $\ell_{2}$-optimal attacks.

(d) Rewards at $\epsilon=0.11$.

Figure 4: Robust performance of unconstrained (MLP) and Lipschitz-bounded
(Sandwich) policies on pendulum swing-up under sample delays and
$\ell_{2}$-optimal adversarial attacks. Panels (b,d) show cross-sections of
(a,c) as a function of each model’s empirically-estimated Lipschitz lower
bound. Bands and error bars show one standard deviation over 10 random model
initializations.

We find that this improvement in robustness is highly correlated with the
policy’s Lipschitz bound. Comparing unconstrained policies to Lipschitz-
bounded policies with various $\gamma$ in Figure 4, there is a smooth
transition from high to low robustness to sample delays and $\ell_{2}$-optimal
adversarial attacks as the policy’s Lipschitz bound increases. Interestingly,
it appears that there is a “best choice” for $\gamma$, and that restricting it
to very small values ($\gamma=3$) harms the closed-loop performance. This is
to be expected, since the optimal policy for pendulum swing-up is known to be
non-smooth, hence with excessive regularization it is likely that the
network’s parameter space does not contain any high-performing policies. Aside
from the policies with $\gamma=3$, all models were successfully trained to
approximately the same final reward (reward curves omitted for brevity). It is
therefore clear from Figures 2 to 4 that, at least in the context of pendulum
swing-up, Lipschitz-bounded policy networks significantly improve robustness
to disturbances and adversarial attacks over standard, unconstrained networks.

###  4.2 Comparing architectures — Atari Pong

In the game of Pong, we expect small amounts of image noise to make very
little difference to the state of the game and the optimal action. We would
therefore hope that smooth, Lipschitz-bounded policies can improve robustness
to perturbations like random noise and adversarial attacks. We see this
immediately in Figure 5, where we compare the robustness of unconstrained CNN
policies and Lipschitz-bounded Sandwich policies to uniform random noise,
$\ell_{2}$ PGD attacks, and $\ell_{\infty}$ PGD attacks. The same qualitative
results observed for the pendulum in Figure 4 can be seen in Figure 5: there
is a smooth transition from high to low robustness as $\gamma$ decreases;
robustness is improved not just for $\ell_{2}$-constrained attacks (which we
expect for policies with a small $\ell_{2}$ Lipschitz bound), but also for
random noise and $\ell_{\infty}$-constrained attacks; and if $\gamma$ is too
small (e.g., $\gamma=5$ here), the policy’s nominal performance and robustness
to perturbations is degraded. In this case, it is possible that the $\gamma=5$
models have not finished training and could perform better if trained over
more epochs (Fig. 7(d)).

(a) Uniform random noise.

(b) Cross-section at $\epsilon=25$.

(c) $\ell_{2}$ PGD attacks.

(d) Cross-section at $\epsilon=70$.

(e) $\ell_{\infty}$ PGD attacks.

(f) Cross-section at $\epsilon=1.05$.

Figure 5: Robust performance of unconstrained (CNN) and Lipschitz-bounded
(Sandwich) policies for Atari Pong. Bands and error bars show one standard
deviation over 4 random model initializations. Figure 6: Adversarial examples
showing the smallest attacks required to make an unconstrained (CNN) and a
Lipschitz-bounded (Sandwich, $\gamma=10$) policy lose the game. In each image,
the “computer” controls the paddle on the left, while the policy controls the
paddle on the right. All attacked frames show scenarios where the policy is
about to concede a goal (puck moving to the right).

It is interesting to look deeper into the effect of adversarial attacks on
these models. Figure 6 shows just how much of an improvement Lipschitz-bounded
policy networks provide in Pong over a standard, unconstrained CNN. The CNN
loses the game when subject to very small amounts of random noise and almost
imperceptible adversarial attacks. In contrast, the Lipschitz-bounded policy
is only beaten with a level of random noise that would even make the game
difficult for a human. Moreover, highly-structured $\ell_{2}$-constrained
attacks are required to beat the Lipschitz-bounded policy. Looking closely at
Figure 6, successful $\ell_{2}$ PGD attacks try to trick the policy into
thinking the opponent’s paddle is higher than it actually is while also trying
to hide the exact location of the puck. The $\ell_{2}$ attacks also seem to
focus on the white boundary walls of the game. It is less clear why the
policies should be sensitive to these features, but we hypothesize that the
straight lines of the walls may appear similar to a paddle in feature space
after passing through convolutional layers. There is no clear structure to the
$\ell_{\infty}$ PGD attacks and they remain rather small, since the Lipschitz-
bounded policies have a constrained $\ell_{2}$ Lipschitz bound, and the
$\ell_{2}$ norm is only a loose upper bound of the $\ell_{\infty}$ norm in
high-dimensional spaces. The additional robustness to $\ell_{\infty}$ PGD
attacks over CNN policies is a nice bonus.

(a) SN.

(b) AOL.

(c) Cayley.

(d) Sandwich.

Figure 7: Test reward during training on Atari Pong for all layer
architectures. Bands show one standard deviation over 4 random model
initializations.

So far, we have compared Lipschitz-bounded policy networks constructed from
Sandwich layers with unconstrained networks to illustrate that smoother
policies can improve robustness in Deep RL. It turns out that in addition to
tuning its upper bound $\gamma$, the layer architecture we use to bound the
Lipschitz constant $\mathrm{Lip}(\kappa)$ of a policy $\kappa$ is extremely
important. Figure 7 compares the training curves of all four Lipschitz-bounded
policy architectures introduced in Section 2.3 with unconstrained CNN
policies, while Table 1 summarizes their nominal and robust performance. It is
immediately clear from Figures 7(a) and 7(b) that the two layer architectures
known to have conservative bounds on $\mathrm{Lip}(\kappa)$, SN and AOL,
perform poorly when $\gamma$ is small. When $\gamma\leq 20$, neither
architecture ever produces a winning policy. For larger $\gamma$, the SN and
AOL policies can learn to win the game, but the training dynamics are
extremely slow in comparison to the CNN (this is a known problem for AOL, see
[3, Sec.7]). Interestingly, Table 1 shows that the estimated lower bound
$\underline{\gamma}$ on $\mathrm{Lip}(\kappa)$ for these policies is very
small when $\gamma$ is small. This suggests that the conservative
parameterization of SN and AOL layers restricts their parameter spaces to a
small set of very smooth models which does not include high-performing policy
networks. In contrast, Figures 7(c) and 7(d) show that the two layer
architectures with much tighter bounds on $\mathrm{Lip}(\kappa)$, Cayley and
Sandwich, perform quite well even for small choices of $\gamma$. The choice of
$\gamma$ still seems to have a strong impact on the training dynamics for
these layers — as $\gamma$ increases, so too does the speed at which the
policies converge on a winning strategy. This raises interesting questions
about the coupling between a policy’s Lipschitz constant and the exploration
of its parameter space, which we leave for future work.

Policy | Lipschitz |  Reward | Smallest winning attack size $\epsilon$ ($\uparrow$)
---|---|---|---
|  $\gamma$ |  $\underline{\gamma}$ |  Unperturbed |  Uniform |  $\ell_{2}$ PGD |  $\ell_{\infty}$ PGD
CNN |  - |  738 |  20.1 |  18.7 |  19.5 |  0.38
SN |  5 |  1.14 |  -20.2 |  - |  - |  -
|  10 |  2.78 |  -19.4 |  - |  - |  -
|  20 |  6.05 |  -18.7 |  - |  - |  -
|  60 |  24.7 |  1.21 |  15.0 |  9.80 |  0.21
|  100 |  42.4 |  14.8 |  21.5 |  21.3 |  0.49
AOL |  5 |  1.40 |  -18.9 |  - |  - |  -
|  10 |  2.95 |  -16.9 |  - |  - |  -
|  20 |  7.62 |  -9.28 |  - |  - |  -
|  60 |  15.3 |  1.31 |  12.4 |  19.2 |  0.52
|  100 |  17.4 |  8.85 |  23.6∗ |  42.3 |  1.20∗
Cayley |  5 |  4.56 |  13.6 |  23.7 |  106 |  1.52
|  10 |  8.99 |  17.8 |  28.7 |  154 |  1.59
|  20 |  16.6 |  20.0 |  23.1 |  129∗ |  1.36
|  60 |  34.8 |  20.6∗ |  20.9 |  67.4∗ |  0.98∗
|  100 |  47.0 |  20.4∗ |  17.1 |  54.0∗ |  0.76
Sandwich |  5 |  4.16 |  17.5∗ |  33.4∗ |  183∗ |  2.01∗
|  10 |  6.90 |  19.5∗ |  35.0∗ |  $\mathbf{>200}$∗ |  2.08∗
|  20 |  12.4 |  20.2∗ |  30.8∗ |  99.2 |  1.53∗
|  60 |  28.0 |  20.6∗ |  23.9∗ |  58.1 |  0.94
|  100 |  42.5 |  20.2 |  20.1 |  43.3 |  0.63
Table 1: Averaged results (4 random model initialisations and 20 random game
and attack seeds) on performance and robustness of policy networks trained on
Atari Pong. $\gamma$ is the certified Lipschitz upper-bound for a network and
$\underline{\gamma}$ is its empirically-estimated lower-bound. The unperturbed
reward is the final mean test reward achieved during training. Results for
each attack strategy are the smallest average attack size $\epsilon$ required
to beat the policy (i.e., reward $<0$). Attack results are not provided for
policies that did not learn a positive reward. Bold values indicate the
overall best performing models in each column. Values with an * indicate the
best performing models for each $\gamma$.

Looking closer at Table 1 reveals that simply having a tight Lipschitz bound
(i.e., close lower and upper bounds $\underline{\gamma}$ and $\gamma$) is not
the only factor contributing to a policy’s performance and robustness.
Instead, there appears to be an advantage to using “expressive” policy
networks, particularly when high robustness (small $\gamma$) is required.
Table 1 indicates that both the Sandwich and Cayley policies exhibit strong
performance and robustness for $\gamma\geq 20$. Either layer architecture is
therefore a suitable choice for reasonable improvement over existing methods
in robust RL like SN. For smaller $\gamma<20$, however, Sandwich policies are
far superior, and with $\gamma=10$ they are the most robust of any policy
across all three input perturbations while still achieving a strong
unperturbed reward of 19.6. This is despite the fact that Cayley policies
often have a tighter Lipschitz bound than the Sandwich policies (take
$\gamma=10$ as an example). We suggest that this is due to the less
conservative parameterization of the Sandwich layers. Each of the SN, AOL, and
Cayley policies are composed of linear layers with a spectral norm of
approximately 1 (exactly 1 for Cayley). In contrast, Sandwich layers are
nonlinear, have no direct restriction on their spectral norm [30, Fig. 4], and
instead constrain $\mathrm{Lip}(\kappa)$ via a composition of nonlinear layers
which are a complete parameterization of the tightest known bounds on the
Lipschitz constant of DNNs [8]. This allows the Sandwich models to converge on
policy networks that are both performant and robust even when their parameter
space is restricted by a small value of $\gamma$, allowing finer control over
the performance-robustness trade-off in Deep RL.

##  5 Conclusions

This paper has studied the robustness benefits of Lipschitz-bounded policy
networks in Deep RL. We have found that policy networks with small Lipschitz
bounds are significantly more robust to perturbations such as disturbances,
random noise, and targeted adversarial attacks. Moreover, we have observed
that choosing a policy with non-conservative Lipschitz bounds and an
expressive, nonlinear layer architecture gives the user finer control over the
performance-robustness trade-off than existing methods based on spectral
normalization. This raises interesting questions for future study, such as
whether Lipschitz-bounded policy networks can augment or alleviate the need
for adversarial training, and whether the observed benefits can be transferred
to real-world robotic systems.

## Acknowledgments

This work was supported in part by Google LLC, the Australian Research Council
(DP230101014), and the Centre for Advanced Defence Research in Robotics and
Autonomous Systems (CADR-RAS). The authors would also like to thank Jack
Naylor for his assistance with computational resources.

## References

  * [1] Andrychowicz, M., Raichuk, A., Stańczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., Gelly, S., Bachem, O.: What matters for on-policy deep actor-critic methods? a large-scale study.  In: International Conference on Learning Representations (2021)
  * [2] Araujo, A., Havens, A.J., Delattre, B., Allauzen, A., Hu, B.: A unified algebraic perspective on Lipschitz neural networks.  In: International Conference on Learning Representations (2023)
  * [3] Bernd, C.H.P., Lampert: Almost-orthogonal layers for efficient general-purpose lipschitz networks.  Computer Vision – ECCV 2022 pp. 350–365 (2022)
  * [4] Bjorck, J., Gomes, C.P., Weinberger, K.Q.: Towards deeper deep reinforcement learning.  Advances in Neural Information Processing Systems (2021)
  * [5] Bradbury, J., Frostig, R., Hawkins, P., Johnson, M.J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., Zhang, Q.: JAX: composable transformations of Python+NumPy programs (2018).  URL http://github.com/google/jax
  * [6] Chakraborty, A., Alam, M., Dey, V., Chattopadhyay, A., Mukhopadhyay, D.: Adversarial attacks and defences: A survey.  arXiv preprint arXiv:1810.00069 (2018)
  * [7] Ding, G.W., Wang, L., Jin, X.: AdverTorch v0.1: An adversarial robustness toolbox based on pytorch.  arXiv preprint arXiv:1902.07623 (2019)
  * [8] Fazlyab, M., Robey, A., Hassani, H., Morari, M., Pappas, G.J.: Efficient and accurate estimation of lipschitz constants for deep neural networks.  Advances in Neural Information Processing Systems (NeurIPS) (2019)
  * [9] Freeman, C.D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., Bachem, O.: Brax – a differentiable physics engine for large scale rigid body simulation.  arXiv preprint arXiv:2106.13281 (2021)
  * [10] Huang, S., Dossa, R.F.J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., Araújo, J.G.: Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.  Journal of Machine Learning Research 23(274), 1–18 (2022)
  * [11] Huang, S., Papernot, N., Goodfellow, I., Duan, Y., Abbeel, P.: Adversarial attacks on neural network policies.  International Conference on Learning Representations (2017)
  * [12] Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., Levine, S.: Scalable deep reinforcement learning for vision-based robotic manipulation.  Proceedings of The 2nd Conference on Robot Learning pp. 651–673 (2018)
  * [13] Kumar, A., Levine, A., Feizi, S.: Policy smoothing for provably robust reinforcement learning.  International Conference on Learning Representations (2022)
  * [14] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks.  International Conference on Learning Representations (2018)
  * [15] Megretski, A., Rantzer, A.: System analysis via integral quadratic constraints.  IEEE Transactions on Automatic Control 42(6), 819–830 (1997)
  * [16] Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks.  In: International Conference on Learning Representations (2018)
  * [17] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning.  Nature 518, 529–533 (2015)
  * [18] Nie, B., Ji, J., Fu, Y., Gao, Y.: Improve robustness of reinforcement learning against observation perturbations via $\ell_{\infty}$ lipschitz policy networks.  Proceedings of the AAAI Conference on Artificial Intelligence 38, 14,457–14,465 (2024)
  * [19] Oikarinen, T., Zhang, W., Meche, M., Megretski, A., Daniel, L., Weng, T.W.: Robust deep reinforcement learning through adversarial loss.  Advances in Neural Information Processing Systems 34, 26,156–26,167 (2021)
  * [20] Pattanaik, A., Tang, Z., Liu, S., Bommannan, G., Chowdhary, G.: Robust deep reinforcement learning with adversarial attacks.  International Joint Conference on Autonomous Agents and Multiagent Systems 3, 2040–2042 (2017)
  * [21] Revay, M., Wang, R., Manchester, I.R.: Lipschitz bounded equilibrium networks.  arXiv:2010.01732 (2020)
  * [22] Rudin, N., Hoeller, D., Reist, P., Hutter, M.: Learning to walk in minutes using massively parallel deep reinforcement learning.  Proceedings of Machine Learning Research 164, 91–100 (2021)
  * [23] Russo, A., Proutiere, A.: Towards optimal attacks on reinforcement learning policies.  American Control Conference pp. 4561–4567 (2021)
  * [24] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms.  arXiv:1707.06347 (2017)
  * [25] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks.  International Conference on Learning Representations (2013)
  * [26] Takase, R., Yoshikawa, N., Mariyama, T., Tsuchiya, T.: Stability-certified reinforcement learning control via spectral normalization.  Machine Learning with Applications 10, 100,409 (2022)
  * [27] Todorov, E., Erez, T., Tassa, Y.: Mujoco: A physics engine for model-based control.  International Conference on Intelligent Robots and Systems pp. 5026–5033 (2012)
  * [28] Trockman, A., Kolter, J.Z.: Orthogonalizing convolutional layers with the cayley transform.  International Conference on Learning Representations (2021)
  * [29] Virmaux, A., Scaman, K.: Lipschitz regularity of deep neural networks: analysis and efficient estimation.  Advances in Neural Information Processing Systems 31 (2018)
  * [30] Wang, R., Manchester, I.: Direct parameterization of lipschitz-bounded deep networks.  International Conference on Machine Learning (2023)
  * [31] Weng, J., Lin, M., Huang, S., Liu, B., Makoviichuk, D., Makoviychuk, V., Liu, Z., Song, Y., Luo, T., Jiang, Y., Xu, Z., Yan, S.: EnvPool: A highly parallel reinforcement learning environment execution engine.  Advances in Neural Information Processing Systems (2022)
  * [32] Wu, F., Li, L., Huang, Z., Vorobeychik, Y., Zhao, D., Li, B.: Crop: Certifying robust policies for reinforcement learning through functional smoothing.  International Conference on Learning Representations (2022)

##  Appendix A Training details

###  A.1 Pendulum swing-up experiments

The pendulum was modeled in MuJoCo XLA (MJX), a fully-differentiable
implementation of the MuJoCo physics simulator [27] written in JAX [5] that
allows users to take gradients through the entire closed-loop system. We
trained unconstrained MLP and Lipschitz-bounded (Sandwich) policies using the
PPO implementation in [9], which leverages the scalability of JAX to massively
multi-thread parallel physics simulations on a single workstation GPU.
Hyperparameters were tuned by varying each parameter one at a time and
choosing the best-performing parameters for an unconstrained MLP. The same
hyperparameters were used to train Lipschitz-bounded policies without any
further tuning. Our chosen hyperparameters can be found on our GitHub
repository333https://github.com/nic-barbara/Lipschitz-RL-
MJX/blob/main/scripts/pendulum/train˙2˙train˙models.py. We trained 10 policies
for each model architecture and choice of $\gamma$, each with a different
random seed for model initialization. Results showing robust performance to
sample delays and adversarial attacks in Figure 4 were averaged over the 10
policies and 1024 pendulum environments starting from random initial states.
MLP networks were composed of 4 linear layers of 32 hidden nodes. Lipschitz-
bounded policies were composed of 4 Sandwich layers of 21 hidden nodes to
ensure the two model architectures had a similar number of trainable
parameters. We chose tanh activations for all policies in accordance with [1].

###  A.2 Atari Pong experiments

We trained unconstrained CNN models and the Lipschitz-bounded policy
architectures (SN, AOL, Sandwich, Cayley) across 4 random model
initialisations using the PPO implementation in [10] and the ALE/Pong-v5
environment from [31]. We used the default hyperparameters chosen for CNN
policies in [10] for all policy architectures. We could not use the reward
gradient to directly optimize (4) as the Pong environment is not
differentiable, hence we implemented attacks with the PGD method [7]. Robust
performance to noise and adversarial attacks in Figure 5 and Table 1 were
averaged over the 4 policies and 20 games of Pong, each with a different
random seed. Further details on the network architecture for each layer type
can be found on our GitHub repository444https://github.com/nic-
barbara/Lipschitz-RL-Atari/blob/main/liprl/atari˙agent.py.