# Real-time Adversarial Perturbations against Deep Reinforcement Learning
Policies: Attacks and Defenses

Buse G. A. Tekgul  Aalto UniversityEspooFinland<EMAIL_ADDRESS>,  Shelly
Wang  University of WaterlooWaterlooCanada<EMAIL_ADDRESS>,  Samuel
Marchal  WithSecure Corporation & Aalto University EspooFinland
<EMAIL_ADDRESS>and  N. Asokan  University of Waterloo & Aalto
UniversityWaterlooCanada<EMAIL_ADDRESS>
###### Abstract.

Deep reinforcement learning (DRL) is vulnerable to adversarial perturbations.
Adversaries can mislead the policies of DRL agents by perturbing the state of
the environment observed by the agents. Existing attacks are feasible in
principle, but face challenges in practice, either by being too slow to fool
DRL policies in real time or by modifying past observations stored in the
agent’s memory. We show that Universal Adversarial Perturbations (UAP),
independent of the individual inputs to which they are applied, can fool DRL
policies effectively and in _real time_. We introduce three attack variants
leveraging UAP. Via an extensive evaluation using three Atari 2600 games, we
show that our attacks are effective, as they fully degrade the performance of
three different DRL agents (up to 100%, even when the $l_{\infty}$ bound on
the perturbation is as small as 0.01). It is faster than the frame rate (60
Hz) of image capture and considerably faster than prior attacks ($\approx
1.8$ms). Our attack technique is also efficient, incurring an online
computational cost of $\approx 0.027$ms. Using two tasks involving robotic
movement, we confirm that our results generalize to complex DRL tasks.
Furthermore, we demonstrate that the effectiveness of known defenses
diminishes against universal perturbations. We introduce an effective
technique that detects all known adversarial perturbations against DRL
policies, including all universal perturbations presented in this paper.111The
code reproducing our work is available in https://github.com/ssg-
research/ad3-action-distribution-divergence-detector.

††copyright: none

##  1\. Introduction

Machine learning models are vulnerable to _adversarial examples_ : maliciously
crafted inputs generated by adding small perturbations to the original input
to force a model into generating wrong predictions (goodfellow2014explaining,
; szegedy2014intriguing, ). Prior work (huang2017adversarial, ;
kos2017delving, ; lin2017tactics, ) has also shown that adversarial examples
can fool deep reinforcement learning (DRL) agents using deep neural networks
(DNNs) to approximate their decision-making strategy. If this vulnerability is
exploited in safety-critical DRL applications such as robotic surgery and
autonomous driving, the impact can be disastrous.

A DRL agent can partially or fully observe the _state_ of the environment by
capturing complex, high-dimensional observations. For example, a DRL agent
playing an Atari 2600 game observes pixels from each image frame of the game
to construct states by combining a number of observations. DRL agents use the
current state as an input to their _policy_ that outputs an optimal action for
that state. Consequently, adversaries can modify the environment to mislead
the agent’s policy. Various state-of-the-art attack methods assume _white-box_
knowledge, where adversaries have access to the parameters of the agent’s
policy model and the reinforcement learning algorithm. In _untargeted_
attacks, the adversary aims to fool the agent’s policy so that the agent 1)
cannot complete its task or 2) finishes its task with unacceptably poor
performance. Prior work has shown that white-box attacks can successfully
destroy agents’ performance using one-step gradient-based approaches
(goodfellow2014explaining, ), optimization-based methods (carlini2017towards,
), or adversarial saliency maps (papernot2016limitations, ). Previous work has
also proposed different attack strategies where the adversary generates the
perturbation for 1) each state (behzadan2017vulnerability, ;
huang2017adversarial, ), 2) _critical states_ where the agent prefers one
action with high confidence (kos2017delving, ; lin2017tactics, ;
sun2020stealthy, ), or 3) periodically, at every $N^{th}$ state
(kos2017delving, ).

[¡short description¿]¡long description¿

Figure 1.  In the Breakout game, when a universal adversarial perturbation is
added into different states, it causes the policy of a victim DQN agent to
take sub-optimal actions instead of optimal actions, i.e., fail to catch the
ball and lose the game.

Although prior white-box attacks using adversarial perturbations are effective
in principle, they are not realistic in practice. First, some attack
strategies are based on computing the perturbation by solving an optimization
problem (lin2017tactics, ). This is computationally expensive, even if it is
done for every $N^{th}$ state. DRL agents must respond to new states very
quickly to carry out the task effectively _on-the-fly_. Therefore, attacks
that take longer than the average time between two consecutive observations
are too slow to be realized in real-time. Second, in realistic scenarios, the
adversary cannot have full control over the environment. However, iterative
attacks (lin2017tactics, ; xiao2019characterizing, ) require querying agents
with multiple perturbed versions of the current state and resetting the
environment to find the optimal perturbation. Therefore, iterative attacks
cannot be applied in real-life scenarios, such as autonomous agents
interacting with a dynamic environment. Finally, the aforementioned state-of-
the-art attacks require seeing all observations to generate and apply
perturbations to the state containing multiple observations. However, the
agent can store clean observations that are part of the current state in its
memory before the adversary can generate perturbations that need to be applied
to _all of those observations_.

Contributions: We propose an effective, _real-time_ attack strategy to fool
DRL policies by computing state-agnostic _universal_ perturbations _offline_.
Once this perturbation is generated, it can be added into any state to force
the victim agent to choose sub-optimal actions.(Figure 1) Similarly to
previous work (goodfellow2014explaining, ; lin2017tactics, ;
xiao2019characterizing, ), we focus on untargeted attacks in a white-box
setting. Our contributions are as follows:

  1. (1)

We design two new real-time white-box attacks, UAP-S and UAP-O, using
Universal Adversarial Perturbation (UAP) (moosavi2017universal, ) to generate
state-agnostic adversarial examples. We also design a third real-time attack,
OSFW(U), by extending Xiao et al.’s (xiao2019characterizing, ) attack so that
it generates a universal perturbation once, applicable to _any subsequent
episode_ (Section 3). An empirical evaluation of these three attacks using
three different DRL agents playing three different Atari 2600 games (Breakout,
Freeway, and Pong) demonstrates that our attacks are comparable to prior work
in their effectiveness (100% drop in return), while being _significantly
faster_ (0.027 ms on average, compared to 1.8 ms) and _less visible_ than
prior adversarial perturbations (huang2017adversarial, ;
xiao2019characterizing, ) (Section 4.2). Using two additional tasks that
involve the MuJoCo robotics simulator (todorov2012mujoco, ), which requires
continuous control, we show that our results generalize to more complex tasks
(Section 4.3).

  2. (2)

We demonstrate the limitations of prior defenses. We show that agents trained
with the state-of-the-art robust policy regularization technique
(zhang2020robust, ) exhibit reduced effectiveness against adversarial
perturbations at higher perturbation bounds $(\geq 0.05)$. In some tasks
(Pong), universal perturbations completely destroy agents’ performance (Table
2). Visual Foresight (lin2017detecting, ), which is another defense method
that can restore an agent’s performance in the presence of prior adversarial
perturbations (huang2017adversarial, ), fails to do so when faced with
universal perturbations (Section 5.2).

  3. (3)

We propose an efficient method, AD3, to detect adversarial perturbations. AD3
can be combined with other defenses to provide stronger resistance for DRL
agents against untargeted adversarial perturbations (Section 5.3).

##  2\. Background and Related Work

###  2.1. Deep Reinforcement Learning

####  2.1.1. Reinforcement learning

Reinforcement learning involves settings where an agent continuously interacts
with a non-stationary environment to decide which action to take in response
to a given state. At time step $t$, the environment is characterized by its
state $\bm{s}\in\mathcal{S}$ consisting of $N$ past observations $o$ pre-
processed by some function $f_{pre}$, i.e.,
$\bm{s}=\\{f_{pre}(o_{t-N+1}),\cdots,f_{pre}(o_{t})\\}$. At each $\bm{s}$, the
agent takes an action $a\in\mathcal{A}$, which moves the environment to the
next $\bm{s{{}^{\prime}}}\in\mathcal{S}$ and receives a reward $r$ from the
environment. The agent uses this information to optimize its policy $\pi$, a
probability distribution that maps states into actions (Sutton1998, ). During
training, the agent improves the estimate of a value function $V(\bm{s})$ or
an action-value function $Q(\bm{s},a)$. $V(\bm{s})$ measures how valuable it
is to be in a state $\bm{s}$ by calculating the _expected discounted return_ :
the discounted cumulative sum of future rewards while following a specific
$\pi$. Similarly, $Q(\bm{s},a)$ estimates the value of taking action $a$ in
the current $\bm{s}$ while following $\pi$. During evaluation, the optimized
$\pi$ is used for decision making, and the _performance_ of the agent is
measured by the return. In this work, we focus on _episodic_
(bellemare2013arcade, ) and _finite-horizon_ (tassa2018deepmind, ) tasks. In
episodic tasks such as Atari games, each episode ends with a terminal state
(e.g., winning/losing a game, arriving at a goal state), and the return for
one episode is computed by the total score in single-player games (e.g.,
Breakout, Freeway) or the relative score when it is played against a computer
(e.g., Pong). Finite-horizon tasks include continuous control, where (e.g.,
Humanoid, Hopper) the return is measured for a fixed length of the episode.

####  2.1.2. DNN

DNNs are parameterized functions $f(\bm{x},\theta)$ consisting of neural
network layers. For an input $\bm{x}\in\mathbb{R}^{n}$ with $n$ features, the
parameter vector $\theta$ is optimized by training $f$ over a labeled training
set. $f(\bm{x},\theta)$ outputs a vector $\bm{y}\in\mathbb{R}^{m}$ with $m$
different classes. In classification problems, the predicted class is denoted
as $\hat{f}(\bm{x})=\operatorname*{argmax}_{m}f(\bm{x},\theta)$. For
simplicity, we will use $f(\bm{x})$ to denote $f(\bm{x},\theta)$.

####  2.1.3. DRL

DNNs are useful for approximating $\pi$ when $\mathcal{S}$ or $\mathcal{A}$ is
too large, or largely unexplored. Deep Q Networks (DQN) is one of the well-
known _value-based_ DRL algorithms (mnih2015human, ) that uses DNNs to
approximate $Q(\bm{s},a)$. During training, DQN aims to find the optimal
$Q(\bm{s},a)$, and defines the optimal $\pi$ implicitly using the optimal
$Q(\bm{s},a)$. Despite its effectiveness, DQN cannot be used in continuous
control tasks, where $a$ is a real-valued vector sampled from a range, instead
of a finite set. Continuous control tasks require _policy-based_ DRL
algorithms (mnih2016asynchronous, ; schulman2017proximal, ). They use two
different DNNs that usually share a number of lower layers to approximate both
$\pi$ and $V(\bm{s})$ (or $Q(\bm{s},a)$), and update $\pi$ directly. For
example, in _actor-critic_ methods (A2C) (mnih2016asynchronous, ), the critic
estimates $V(\bm{s})$ or $Q(\bm{s},a)$ for the current $\pi$, and the actor
updates the parameter vector $\theta$ of $\pi$ by using _advantage_ , which
refers to the critic’s evaluation of the action decision using the estimated
$V(\bm{s})$ or $Q(\bm{s},a)$, while the actor follows the current $\pi$.
Proximal Policy Optimization (PPO) (schulman2017proximal, ) is another on-
policy method that updates the current $\pi$ by ensuring that the updated
$\pi$ is close to the old one.

###  2.2. Adversarial Examples

An adversarial example $\bm{x}^{*}$ against a classifier $f$ is a deliberately
modified version of an input $\bm{x}$ such that $\bm{x}$ and $\bm{x}^{*}$ are
similar, but $\bm{x}^{*}$ is misclassified by $f$, i.e.,
$\hat{f}(\bm{x})\neq\hat{f}(\bm{x}^{*})$. An untargeted adversarial example is
found by solving

(1)
$\displaystyle\operatorname*{argmax}_{\bm{x}^{*}}\ell(f(\bm{x}^{*}),\hat{f}(\bm{x}))\;\text{\small
s.t.:
}\lVert{\bm{x}^{*}-\bm{x}}\rVert_{p}=\lVert{\bm{r}}\rVert_{p}\leq\epsilon,$

where $\epsilon$ is the $l_{p}$ norm bound and $\ell$ is the loss between
$f(\bm{x}^{*})$ and the predicted label $\hat{f}(\bm{x})$. In this work, we
use the $l_{\infty}$ norm bound (i.e., any element in the perturbation must
not exceed a specified threshold $\epsilon$) as in the state-of-the-art Fast
Gradient Sign Method (FGSM) (goodfellow2014explaining, ). FGSM calculates
$\bm{r}$ by

(2)
$\bm{r}=\epsilon\cdot\text{sign}({\nabla_{\bm{x}}\ell(f(\bm{x}),\hat{f}(\bm{x})})).$

Adversarial examples are usually computed for each $\bm{x}$. An alternative is
to generate input-agnostic _universal perturbations_. For instance, Moosavi et
al. propose _Universal Adversarial Perturbation_ (UAP) (moosavi2017universal,
) that searches for a sufficiently small $\bm{r}$ that can be added to the
_arbitrary_ inputs to yield adversarial examples against $f$. UAP iteratively
computes a unique $\bm{r}$ that fools $f$ for almost all inputs $\bm{x}$
belonging to a training set $\mathcal{D}_{train}$. UAP utilizes DeepFool
(moosavi2016deepfool, ) to update $\bm{r}$ at each iteration. UAP aims to
achieve the desired _fooling rate_ $\delta$: the proportion of successful
adversarial examples against $f$ with respect to the total number of perturbed
samples $|\mathcal{D}_{train}|$.

Following the first work (moosavi2017universal, ) introducing UAP, many
different strategies (co2019procedural, ; hayes2018learning, ; mopuri2017fast,
; mopuri2018nag, ) have been proposed to generate universal adversarial
perturbations to fool image classifiers. For example, Hayes et al.
(hayes2018learning, ) and Mopuri et al. (mopuri2018nag, ) use generative
models to compute universal adversarial perturbations. Mopuri et al.
(mopuri2017fast, ) introduce the Fast Feature Fool algorithm, which does not
require a training set to fool the classifier, but optimize a random noise to
change activations of the DNN in different layers. Co et al.
(co2019procedural, ) design black-box, untargeted universal adversarial
perturbations using procedural noise functions.

####  2.2.1. Adversarial Examples in DRL

In discrete action spaces with finite actions, adversarial examples against
$\pi$ are found by modifying Equation 1: $\bm{x}$ is changed to $\bm{s}$ at
time step $t$, $f$ is replaced with $Q(\bm{s},a)$, and $\hat{Q}(\bm{s})$
refers to the decided action. We also denote $Q(\bm{s},a_{m})$ as the state-
action value of $m^{th}$ action at $\bm{s}$. In this setup, adversarial
examples are computed to decrease $Q(\bm{s},a)$ for the optimal action at
$\bm{s}$, resulting in a sub-optimal decision.

Since Huang et al. (huang2017adversarial, ) showed the vulnerability of DRL
policies to adversarial perturbations, several untargeted attack methods
(behzadan2017vulnerability, ; kos2017delving, ; lin2017tactics, ;
sun2020stealthy, ; xiao2019characterizing, ) have been proposed that
manipulate the environment. Recent work (baluja2018learning, ;
carlini2017towards, ; hussenot2020copycat, ; lin2017tactics, ;
tretschk2018sequential, ) has also developed targeted attacks, where the
adversary’s goal is to lure the victim agent into a specific state or to force
the victim policy to follow a specific path. Most of these methods implement
well-known adversarial example generation methods such as FGSM
(huang2017adversarial, ; kos2017delving, ), JSMA (behzadan2017vulnerability, )
and Carlini&Wagner (lin2017tactics, ). Therefore, even though they effectively
decrease the return of the agent, these methods cannot be implemented in real-
time and have a temporal dependency: they need to compute a different $\bm{r}$
for every $\bm{s}$.

Similarly to our work, Xiao et al. propose (“obs-seq-fgsm-wb”, OSFW) to
generate universal perturbations. OSFW computes a single $\bm{r}$ by applying
FGSM to the averaged gradients over the $k$ states and adds $\bm{r}$ to the
remaining states in the current episode. However, OSFW has limitations, such
as the need to compute $\bm{r}$ for every new episode. Moreover, OSFW has to
freeze the task to calculate $\bm{r}$, and its performance depends on the
particular agent-environment interaction for each episode. Hussenot et al.
(hussenot2020copycat, ) design targeted attacks that use different universal
perturbations for each action to force the victim policy to follow the
adversary’s policy.

In addition to white-box attacks, multiple black-box attack methods based on
finite-difference methods (xiao2019characterizing, ), and proxy methods that
approximate the victim policy (inkawhich2019snooping, ; zhao2020blackbox, )
were proposed, but they cannot be mounted in real time, as they require
querying the agent multiple times. Recent work (gleave2019adversarial, ;
wu2021adversarial, ) also shows that adversaries can fool DRL policies in
multi-agent, competitive games by training an adversarial policy for the
opponent agent that exploits vulnerabilities of the victim agent. These
attacks rely on creating natural observations with adversarial effects,
instead of manipulating the environment by adding adversarial perturbations.
In this paper, we focus on single-player games, where adversaries can only
modify the environment to fool the DRL policies.

##  3\. State- and Observation-Agnostic Perturbations

###  3.1. Adversary Model

The goal of the adversary $\mathit{Adv}$ is to degrade the performance of the
victim DRL agent $\mathit{v}$ by adding perturbations $\bm{r}$ to the state
$\bm{s}$ observed by $\mathit{v}$. $\mathit{Adv}$ is successful when the
attack:

  1. (1)

is _effective_ , i.e., limits $\mathit{v}$ to a low return,

  2. (2)

is _efficient_ , i.e., can be realized in real-time, and

  3. (3)

_evades_ known detection mechanisms.

$\mathit{Adv}$ has a white-box access to $\mathit{v}$; therefore, it knows
$\mathit{v}$’s action value function $Q_{\mathit{v}}$, or the policy
$\pi_{\mathit{v}}$ and the value function $V_{\mathit{v}}$, depending on the
DRL algorithm used by $\mathit{v}$. However, $\mathit{Adv}$ is constrained to
using $\bm{r}$ with a small norm to evade possible detection, either by
specific anomaly detection mechanisms or via human observation. We assume that
$\mathit{Adv}$ cannot reset the environment or return to an earlier state. In
other words, we rule out trivial attacks (e.g., swapping one video frame with
another or changing observations with random noise) as ineffective because
they can be easily detected. We also assume that $\mathit{Adv}$ has only read-
access to $\mathit{v}$’s memory. Modifying the agent’s inner workings is an
assumption that is too strong in realistic adversarial settings because it
forces $\mathit{Adv}$ to modify both the environment and $\mathit{v}$. If
$\mathit{Adv}$ is able to modify or rewrite $\mathit{v}$’s memory, then it
does not need to compute adversarial perturbations and simply rewrites
$\mathit{v}$’s memory to destroy its performance.

###  3.2. Attack Design

####  3.2.1. Training data collection and sanitization

$\mathit{Adv}$ collects a training set $\mathcal{D}_{train}$ by monitoring
$\mathit{v}$’s interaction with the environment for one episode, and saving
each $\bm{s}$ into $\mathcal{D}_{train}$. Simultaneously, $\mathit{Adv}$
clones $Q_{\mathit{v}}$ or $V_{\mathit{v}}$ into a _proxy agent_ , $adv$.
Specifically, in value-based methods, $\mathit{Adv}$ copies the weights of
$Q_{\mathit{v}}$ into $Q_{adv}$. In policy-based methods, $\mathit{Adv}$
copies the weights of the critic network into $V_{adv}$. In the latter case,
$\mathit{Adv}$ can obtain $Q_{adv}(\bm{s},a)$ by calculating $V_{adv}(\bm{s})$
for each _discrete_ action $a\in\mathcal{A}$.

After collecting $\mathcal{D}_{train}$, $\mathit{Adv}$ sanitizes it by
choosing only the _critical states_. Following (lin2017tactics, ), we define
critical states as those that can have a significant influence on the course
of the episode. We identify critical states using the relative action
preference function

(3)
$\displaystyle\text{Var}_{a\in\mathcal{A}}\left[\text{Softmax}(Q_{adv}(\bm{s},a))\right]\geq\beta$
$\displaystyle\beta$
$\displaystyle=1/|\mathcal{D}_{train}|\;\sum_{\quad\mathclap{\bm{s}\in\mathcal{D}_{train}}}{\text{Var}_{a\in\mathcal{A}}\left[\text{Softmax}(Q_{adv}(\bm{s},a))\right]},$

modified from (lin2017tactics, ), where $\mathrm{Var}$ is the variance of the
normalized $Q_{adv}(\bm{s},a)$ values computed for $\forall a\in\mathcal{A}$.
This ensures that both UAP-S and UAP-O are optimized to fool $Q_{\mathit{v}}$
in critical states, and achieves the first attack criterion.

####  3.2.2. Computation of perturbation

For both UAP-S and UAP-O, we assume that $s\in\mathcal{D}_{train}$ and
$\mathcal{D}_{train}\subset\mathcal{S}$. $\mathit{Adv}$ searches for an
optimal $\bm{r}$ that satisfies the constraints in Equation 1, while achieving
a high fooling rate $\delta$ on $\mathcal{D}_{train}$. For implementing UAP-S
and UAP-O, we modify Universal Adversarial Perturbation (moosavi2017universal,
) (see Section 2.2). The goal of both UAP-S and UAP-O is to find a
sufficiently small $\bm{r}$ such that
$\hat{Q}(\bm{s}+\bm{r})\neq\hat{Q}(\bm{s})$, leading $\mathit{v}$ to choose
sub-optimal actions. Algorithm 1 summarizes the method for generating UAP-S
and UAP-O.

input : sanitized $\mathcal{D}_{train},Q_{adv},\text{desired fooling rate
}\delta_{th},$

$\text{max. number of iterations }it_{max},\text{ perturbation constraint
}\epsilon$

output : universal $\bm{r}$

1 Initialize $\bm{r}\leftarrow 0,it\leftarrow 0$;

2 while  _$\delta <\delta_{max}$ and  $it<it_{max}$_ do

3 for  _$\bm{s}\in\mathcal{D}_{train}$_ do

4 if  _$\hat{Q}(\bm{s}+\bm{r})=\hat{Q}(\bm{s})$ _ then

5 Find the extra, minimal $\Delta\bm{r}$:
$\Delta\bm{r}\leftarrow\text{argmin}_{\Delta\bm{r}}{\lVert\Delta\bm{r}\rVert}_{2}\text{
s.t.: }\hat{Q}(\bm{s}+\bm{r}+\Delta\bm{r})\neq\hat{Q}(\bm{s})$;

6
$\bm{r}\leftarrow\text{sign}(\text{min}(\text{abs}(\bm{r}+\Delta\bm{r}),\epsilon))$;

7

8 Calculate $\delta$ with updated $\bm{r}$ on $\mathcal{D}_{train}$;

9 $it\leftarrow(it+1)$;

10

Algorithm 1 Computation of UAP-S and UAP-O

In lines 5-6 of Algorithm 1, UAP-S and UAP-O utilize DeepFool to compute
$\Delta\bm{r}$ by iteratively updating the perturbed
$\bm{s}^{*}_{i}=\bm{s}+\bm{r}+\Delta\bm{r}_{i}$ until $\hat{Q}_{adv}$ outputs
a wrong action (see Algorithm 2 in (moosavi2016deepfool, )). At each iteration
$i$, DeepFool finds the closest hyperplane $\hat{l}(\bm{s}^{*}_{i})$ and
$\Delta\bm{r}_{i}$ that projects $\bm{s}^{*}_{i}$ on the hyperplane. It
recomputes $\Delta\bm{r}_{i}$ as

$\displaystyle Q^{{}^{\prime}}(\bm{s}^{*}_{i},a_{\hat{l}})$
$\displaystyle\leftarrow
Q_{adv}(\bm{s}^{*}_{i},a_{\hat{l}})-Q_{adv}(\bm{s}^{*}_{i},a_{m}),$
$\displaystyle\bm{w}^{{}^{\prime}}_{\hat{l}}$ $\displaystyle\leftarrow\nabla
Q_{adv}(\bm{s}^{*}_{i},a_{\hat{l}})-\nabla Q_{adv}(\bm{s}^{*}_{i},a_{m}),$ (4)
$\displaystyle\Delta\bm{r}_{i}$
$\displaystyle\leftarrow\frac{|Q^{{}^{\prime}}(\bm{s}^{*}_{i},a_{\hat{l}})|}{{\lVert{\bm{w}^{{}^{\prime}}_{\hat{l}}}\rVert}^{2}_{2}}\bm{w}^{{}^{\prime}}_{\hat{l}},$

where $\nabla$ is the gradient of $Q_{adv}$ w.r.t. $\bm{s}_{i}$ and
$Q_{adv}(\bm{s}^{*}_{i},a_{m})$ is the value of the $m$-th action chosen for
the state $\bm{s}+\bm{r}$.

UAP-S computes a different perturbation for each observation $o_{j}$ in
$\bm{s}$, i.e., $\bm{r}=\\{r_{t-N+1},\cdots,r_{t}\\},\,r_{j}\neq
r_{k},\,\forall j,k\in\\{t-N+1,\cdots,t\\},\,j\neq k$. In contrast, UAP-O
applies the same perturbation to all observations in $\bm{s}$. Therefore, it
can be considered as an observation-agnostic, completely universal attack.
UAP-O aims to find a modified version $\tilde{\bm{r}}$ of $\bm{r}$ by solving

(5)
$\displaystyle\operatorname*{min}({\lVert\bm{r}-\tilde{\bm{r}}\rVert}^{2}_{2})$
$\displaystyle\begin{aligned} \text{\small s.t.:
}\tilde{r}_{j}=\tilde{r}_{k},\,\forall
j,k\in\\{t-N+1,\cdots,t\\}\;\text{\small and
}\lVert{\tilde{\bm{r}}}\rVert_{\infty}\leq\epsilon.\\\ \end{aligned}$

In UAP-O, we modify lines 5-6 of Algorithm 1 to find $\Delta\bm{r}$. The
closest $\Delta\tilde{\bm{r}}_{i}$ to $\Delta\bm{r}_{i}$ satisfying the
conditions of Equation 5 is found by averaging
$\bm{w}^{{}^{\prime}}_{\hat{l}}$ over observations:

(6)
$\Delta\tilde{r}_{i_{j}}\leftarrow\frac{|Q^{{}^{\prime}}(\bm{s}^{*}_{i},a_{\hat{l}})|}{N{\lVert{\bm{w}^{{}^{\prime}}_{\hat{l}}}\rVert}^{2}_{2}}\quad\smashoperator[]{\sum_{k=(t-N+1)}^{t}}\;\bm{w}^{{}^{\prime}}_{\hat{l}_{k}},\,\forall
j\in\\{t-N+1,\cdots,t\\}.$

In UAP-O, DeepFool returns $\Delta\tilde{\bm{r}}_{i}=\Delta\tilde{r}_{i_{j}}$
as the optimal additional perturbation. UAP-O adds the same $\tilde{r}_{j}$ to
every $o_{j}$ in $\bm{s}$. If $\bm{s}$ consists of only one observation, then
UAP-S will simply reduce to UAP-O. The proof of Equation 6 can be found in
Appendix A.

####  3.2.3. Extending OSFW to OSFW(U)

As explained in Section 2.2.1, OSFW calculates $\bm{r}$ by averaging gradients
of $Q_{\mathit{v}}$ using the first k states in an episode and then adds
$\bm{r}$ to the remaining states. This requires 1) generating a different
$\bm{r}$ for each episode, and 2) suspending the task (e.g., freezing or
delaying the environment) and $\mathit{v}$ to perform backward propagation.
Moreover, the effectiveness of OSFW varies when $\mathit{v}$ behaves
differently in individual episodes. We extend OSFW to a completely universal
adversarial perturbation by using the proxy agent’s DNN, and calculate
averaged gradients with first k samples from the same, un-sanitized
$\mathcal{D}_{train}$. The formula for calculating $\bm{r}$ in OSFW(U) is

(7)
$\bm{r}=\epsilon\cdot\text{sign}(1/k\sum_{i=0}^{i<k}\nabla_{\bm{s}_{i}}(-\log(Q_{adv}(\bm{s}_{i},\hat{a})))),$

where $\hat{a}$ denotes the action chosen and
$\bm{s}_{i}\in\mathcal{D}_{train}$.

###  3.3. Attacks in Continuous Control Settings

In continuous control tasks, the optimal action is a real-valued array that is
selected from a range. These tasks have complex environments that involve
physical system control such as Humanoid robots with multi-joint dynamics
(todorov2012mujoco, ). Agents trained for continuous control tasks have no $Q$
that can be utilized for generating perturbations to decrease the value of the
optimal action. Nevertheless, $\mathit{Adv}$ can find a perturbed state
$\bm{s}+\bm{r}$ that has the worst $V(\bm{s})$, so that $\pi_{\mathit{v}}$
might produce a sub-optimal action (zhang2020robust, ). In continuous control,
OSFW and OSFW(U) can be simply modified by changing $Q_{adv}$ with $V_{adv}$.
However, in Algorithm 1, the lines 4-6 need to be adjusted to handle these
tasks. $\mathit{Adv}$ can only use the copied network parameters of $V_{adv}$
to find $\bm{r}$. The corrected computation for UAP-S and UAP-O is given in
Algorithm 2. In each iteration $it$, $\Delta\bm{r}$ is computed by DeepFool
(moosavi2016deepfool, ) (lines 5 and 6).

input : sanitized $\mathcal{D}_{train},V_{adv}$, hyper-parameter  $\alpha$

$\text{max. number of iterations }it_{max},\text{ pert. constraint }\epsilon$

output : universal $\bm{r}$

1 Initialize $\bm{r}\leftarrow 0,it\leftarrow 0$;

2

3while  _$\delta <\delta_{max}$ and  $it<it_{max}$_ do

4 for  _$\bm{s}\in\mathcal{D}_{train}$_ do

5 if  _$V_{adv}(\bm{s}+\bm{r})+\alpha <V_{adv}(\bm{s})$ _ then

6 Find the extra, minimal $\Delta\bm{r}$:

7
$\Delta\bm{r}\leftarrow\text{argmin}_{\Delta\bm{r}}{\lVert\Delta\bm{r}\rVert}_{2}$

8 s.t.  $V_{adv}(\bm{s}+\bm{r}+\Delta\bm{r})+\alpha<V_{adv}(\bm{s})$;

9
$\bm{r}\leftarrow\text{sign}(\text{min}(\text{abs}(\bm{r}+\Delta\bm{r}),\epsilon))$;

10

11 Calculate $\delta$ with updated $\bm{r}$ on $\mathcal{D}_{train}$;

12 $it\leftarrow(it+1)$;

13

Algorithm 2 Computation of UAP in continuous control

##  4\. Attack evaluation

###  4.1. Experimental Setup

We compared the effectiveness and efficiency of our attacks (UAP-S, UAP-O and
OSFW(U)) with prior attacks (FGSM and OSFW) on discrete tasks using three
Atari 2600 games (Pong, Breakout, Freeway) in the Arcade Learning Environment
(bellemare2013arcade, ). We further extended our experimental setup with the
MuJoCo robotics simulator (todorov2012mujoco, ) and compared these attacks in
continuous control tasks.

[¡short description¿]¡long description¿

Figure 2. Comparison of attacks against DQN, PPO and A2C agents trained for
Pong, Breakout and Freeway. The graph shows how the return (averaged over 10
games) changes with different $\epsilon$ values for six different attack
strategies.

To provide an extensive evaluation, for every Atari game, we trained three
agents each using a different DRL algorithm: value-based DQN (mnih2015human,
), policy-based PPO (schulman2017proximal, ) and actor-critic method A2C
(mnih2016asynchronous, ). We used the same DNN architecture proposed in
(mnih2015human, ) for approximating $Q_{\mathit{v}}$ in DQN and
$V_{\mathit{v}}$ in other algorithms. To facilitate comparison, we used the
same setup to implement all DRL policies as well as the different attacks:
PyTorch (version 1.2.0), NumPy (version 1.18.1), Gym (a toolkit for developing
reinforcement learning algorithms, version 0.15.7) and MuJoCo 1.5 libraries.
All experiments were done on a computer with 2x12 core Intel(R) Xeon(R) CPUs
(32GB RAM) and NVIDIA Quadro P5000 with 16GB memory.

Our implementations of DQN, PPO and A2C are based on OpenAI
baselines111https://github.com/DLR-RM/rl-baselines3-zoo, and our
implementations achieve similar returns as in OpenAI Baselines. For pre-
processing, we converted each RGB frame to gray-scale, re-sized those from
$210\times 160$ to $84\times 84$, and normalized the pixel range from
$[0,255]$ to $[0,1]$. We used the frame-skipping technique (mnih2015human, ),
where the victim $\mathit{v}$ constructs $\bm{s}$ at every $N{th}$
observation, then selects and action and repeats it until the next state
$\bm{s}^{\prime}$. WE set $N=4$, so the input state size is $4\times 84\times
84$. The frame rate of each game is 60 Hz by default (bellemare2013arcade, );
thus, the time interval between two consecutive frames is $1/60=0.017$
seconds.

We implemented UAP-S and UAP-O by setting the desired $\delta_{max}$ to
$95\%$, so that they stop searching for another $\bm{r}$ when $\delta\geq
95\%$. As baselines, we used random noise addition, FGSM
(huang2017adversarial, ), and OSFW (xiao2019characterizing, ). We chose FGSM
as the baseline since it is the fastest adversarial perturbation generation
method from the previous work and is effective in degrading the performance of
DRL agents. We measured attack effectiveness where $0\leq\epsilon\leq 0.01$.
We reported the average return over 10 episodes and used different seeds
during training and evaluation.

###  4.2. Attack Performance

####  4.2.1. Performance degradation

Figure 2 compares UAP-S, UAP-O, and OSFW(U) with two baseline attacks and
random noise addition. Random noise addition cannot cause a significant drop
in $\mathit{v}$’s performance, and FGSM is the most effective attack, reducing
the return up to 100% even with a very small $\epsilon$ value. UAP-S is the
second most effective attack in almost every setup, reducing the return by
more than $50\%$ in all experiments when $\epsilon\geq 0.004$. All attacks
completely destroy $\mathit{v}$’s performance at $\epsilon=0.01$, except the
PPO agent playing Freeway. The effectiveness of UAP-O, OSFW, and OSFW(U) is
comparable in all setups. We also observe that the effectiveness of OSFW
fluctuates heavily (Breakout-A2C) or has a high variance (Freeway-PPO). This
phenomenon is the result of the different behaviors of $\mathit{v}$ in
individual episodes (Breakout-A2C), and OSFW’s inability to collect enough
knowledge (Freeway-PPO) to generalize $\bm{r}$ to the rest of the episode.

####  4.2.2. Timing comparison

Table 4.2.2 shows the computational cost to generate $\bm{r}$, and the upper
bound on the online computational cost to mount the attack in real-time. This
upper bound is measured as $T_{max}=1/\text{(frame rate)}-\text{(response
time)}$, where the response time is the time spent feeding $\bm{s}$ forward
through $\pi_{\mathit{v}}$ or $Q_{\mathit{v}}$ and executing the corresponding
action. If the online cost of injecting $\bm{r}$ during deployment (and the
online cost of generating $\bm{r}$ during deployment in the case of FGSM and
OSFW) is greater than $T_{max}$, then $\mathit{Adv}$ must stop or delay the
environment, which is infeasible in practice. Table 4.2.2 confirms that the
online cost of OSFW is higher than that of all other attacks and $T_{max}$ due
to its online perturbation generation approach. OSFW has to stop the
environment to inject $\bm{r}$ into the current $\bm{s}$ or has to wait 102
states on average $(\text{online cost}/(T_{max}\cdot N))$ to correctly inject
$\bm{r}$, which can decrease the attack effectiveness. UAP-S and UAP-O have a
higher offline cost than OSFW(U), but the offline generation of $\bm{r}$ does
not interfere with the task, as it does not require interrupting or pausing
$\mathit{v}$. The online cost of FGSM, UAP-S, UAP-O and OSFW(U) is lower than
$T_{max}$.

Experiment | Attack | Offline cost$\pm$ std | Online cost $\pm$ std
---|---|---|---
method | (seconds) | (seconds)
Pong, DQN, $T_{max}=0.0163\pm 10^{-6}$ seconds | FGSM | - | $13\times 10^{-4}\pm 10^{-5}$
OSFW | - | $5.3\pm 0.1$
UAP-S | $36.4\pm 21.1$ | $2.7\times 10^{-5}\pm 10^{-6}$
UAP-O | $138.3\pm 25.1$ | $2.7\times 10^{-5}\pm 10^{-6}$
OSFW(U) | $5.3\pm 0.1$ | $2.7\times 10^{-5}(\pm 10^{-6})$
Pong, PPO,  $T_{max}=0.0157\pm 10^{-5}$ seconds | FGSM | - | $21\times 10^{-4}\pm 10^{-5}$
OSFW | - | $7.02\pm 0.6$
UAP-S | $41.9\pm 16.7$ | $2.7\times 10^{-5}\pm 10^{-6}$
UAP-O | $138.3\pm 25.1$ | $2.7\times 10^{-5}\pm 10^{-6}$
OSFW(U) | $7.02\pm 0.6$ | $2.7\times 10^{-5}\pm 10^{-6}$
Pong, A2C  $T_{max}=0.0157\pm 10^{-5}$ seconds | FGSM | - | $21\times 10^{-4}\pm 10^{-5}$
OSFW | - | $7.2\pm 1.1$
UAP-S | $11.4\pm 4.3$ | $2.7\times 10^{-5}\pm 10^{-6}$
UAP-O | $55.5\pm 29.3$ | $2.7\times 10^{-5}\pm 10^{-6}$
OSFW(U) | $7.2\pm 1.1$ | $2.7\times 10^{-5}\pm 10^{-6}$

Table 1. Offline and online cost of attacks where victim agents are DQN, PPO,
A2C trained for Pong and $\epsilon=0.01$. Attacks that cannot be implemented
in real-time are in highlighted in red.

####  4.2.3. Difference in perturbation sizes

Figure 3 shows that $\bm{r}$ obtained via UAP-S and UAP-O are smaller than
other adversarial perturbations for the same $\epsilon$, since UAP-S and UAP-O
try to find a minimal $\bm{r}$ that sends all $\bm{x}\in\mathcal{D}_{train}$
outside the decision boundary (moosavi2017universal, ). We conclude that UAP-S
and UAP-O are likely to be less detectable based on the amount of the
perturbation (e.g., via visual observation). Appendix B also presents
additional results for other agents and other games, as well as the perturbed
version of the observations.

As summarized in Table 2, FGSM computes a new $\bm{r}$ for each $\bm{s}$ after
observing the complete $\bm{s}$. Therefore, it requires rewriting
$\mathit{v}$’s memory to change all previously stored observations $o_{j}$ of
the current $\bm{s}$, in which $\mathit{v}$ is attacked. Unlike FGSM, UAP-S
and UAP-O add $r_{j}$ to the incoming $o_{j}$, and $\mathit{v}$ stores
adversarially perturbed $o_{j}$ into its memory. The online cost of OSFW is
too high, and it cannot be mounted without interfering with the environment.
UAP-S, UAP-O and OSFW(U) are real-time attacks that do not require stopping
the agent or the environment while adding $\bm{r}$. UAP-S and OSFW(U) generate
$\bm{r}$ that is independent of $\bm{s}$, but the $r_{j}$ for each observation
$o_{j}\in\bm{s}$ is different. On the other hand, UAP-O adds the same $r_{j}$
to all observations in any $\bm{s}$, which makes the perturbation generation
independent of the size of $\bm{s}$. UAP-O leads to an efficient and effective
attack when $\epsilon\geq 0.006$, and does not have temporal and observational
dependency. UAP-S is the optimal attack considering both effectiveness and
efficiency.

Attack | Online | State | Observation
---|---|---|---
Method | cost | dependency | dependency
FGSM (huang2017adversarial, ) | Low | Dependent | Dependent
OSFW (xiao2019characterizing, ) | High | Independent | Dependent
UAP-S | Low | Independent | Dependent
UAP-O | Low | Independent | Independent
OSFW(U) | Low | Independent | Dependent

Table 2. Summary of five attacks based on the characteristics that makes an
attack plausible in a real deployment scenario. New attacks proposed in this
paper are highlighted in blue and underlined.

[¡short description¿]¡long description¿

Clean, RGB

FGSM

OSFW

UAP-S

UAP-O

OSFW(U)

Figure 3. Comparison of the perturbation size added into the same clean (RGB)
observation in different attacks against the DQN agent playing Pong and
$\epsilon=0.01$. In perturbations, black pixels: $-0.01$, white pixels:
$+0.01$, gray pixels: $0.0$.

In simple environments like Atari 2600 games, sequential states are not i.i.d.
Moreover, Atari 2600 games are _controlled_ environments, where future states
are predictable and episodes do not deviate much from one another for the same
task. OSFW and OSFW(U) leverage this non-i.i.d property. However, their
effectiveness might decrease in uncontrolled environments and the physical
world due to the uncertainty of the future states. In contrast, UAP-S and
UAP-O are independent of the correlation between sequential states. To confirm
our conjecture, we implemented OSFW(U) against VGG-16 image classifers
(zhang2015accelerating, ) pre-trained on ImageNet (russakovsky2015imagenet, ),
where ImageNet can be viewed as a non i.i.d., uncontrolled environment. We
measured that OSFW(U) achieves a fooling rate of up to $30\%$ on the ImageNet
validation set with $\epsilon=10$, while UAP-S has a fooling rate of $78\%$
(moosavi2017universal, ), where the $l_{\infty}$ norm of an image in the
validation set is around 250. Therefore, we conclude that UAP-S can mislead
DRL policies more than OSFW(U) in complex, uncontrolled environments.

###  4.3. Attack Performance in Continuous Control

[¡short description¿]¡long description¿

Figure 4. Comparison of attacks against PPO agents trained for Humanoid and
Walker-2d tasks. The graph shows how the return (averaged over 50 games)
changes with different $\epsilon$ values for five different attack strategies.

Experiment | Attack | Offline cost$\pm$ std | Online cost $\pm$ std
---|---|---|---
method | (seconds) | (seconds)
Walker2d, PPO, $T_{max}=0.0079\pm 10^{-5}$ seconds | FGSM | - | $31\times 10^{-5}\pm 10^{-5}$
OSFW | - | $0.02\pm 0.001$
UAP-S (O) | $8.75\pm 0.024$ | $2.9\times 10^{-5}\pm 10^{-6}$
OSFW(U) | $0.02\pm 0.001$ | $2.9\times 10^{-5}\pm 10^{-6}$
Humanoid PPO, $T_{max}=0.0079\pm 10^{-6}$ seconds | FGSM | - | $35\times 10^{-5}\pm 10^{-5}$
OSFW | - | $0.02\pm 0.001$
UAP-S (O) | $35.86\pm 0.466$ | $2.4\times 10^{-5}\pm 10^{-6}$
OSFW(U) | $0.02\pm 0.001$ | $2.4\times 10^{-5}\pm 10^{-6}$

Table 3. Offline and online cost of attacks where victim agents are PPO
trained for Walker2d and Humanoid at $\epsilon=0.02$. Attacks that cannot be
implemented in real-time are highlighted in red.

In continuous control, we used PPO agents in (zhang2020robust, ) pre-trained
for two different MuJoCo tasks (Walker2d and Humanoid) as $\mathit{v}$
222Agents are downloaded from https://github.com/huanzhang12/SA_PPO and frame
rates are set to default values as in https://github.com/openai/gym. We used
the original experimental setup to compare our attacks with baseline attacks
and random noise addition. In our experiments, PPO agents show performance
similar to those reported in the original paper (zhang2020robust, ). We
implemented UAP-S by copying the parameters of $V_{\mathit{v}}$ into
$V_{\mathit{adv}}$, and set $\delta_{max}=95\%$. FGSM, OSFW and OSFW(U) also
use $V_{\mathit{v}}$ to minimize the value in a perturbed state. Additionally,
in both tasks, $\bm{s}$ contains only one observation, which reduces UAP-S to
UAP-O.

Figure 4 shows the attack effectiveness when $0.0\leq\epsilon\leq 0.2$. FGSM
is the most effective attack in Humanoid, while UAP-S decreases the return
more than FGSM in Walker2d when $\epsilon\geq 0.12$. Overall, all attacks
behave similarly in both discrete and continuous action spaces, and our
conclusions regarding the effectiveness of universal adversarial perturbations
generalize to continuous control tasks, where $Q_{\mathit{v}}$ is not
available. However, in these tasks, $\mathit{Adv}$ only decreases the critic’s
evaluation of $\bm{s}$ when taking an action $a$. Even if $\mathit{Adv}$
decreases the value of $V_{\mathit{adv}}(\bm{s})$, it does not necessarily
lead to $\pi_{\mathit{v}}$ choosing a sub-optimal action over the optimal one.
Therefore, attacks using $V$ require $\epsilon\geq 0.2$ to fool
$\pi_{\mathit{v}}$ effectively. For a more efficient attack, $\mathit{Adv}$
can copy $\pi_{\mathit{v}}$ into a proxy agent as $\pi_{\mathit{adv}}$, and
try to maximize the total variation distance (zhang2020robust, ) in
$\pi_{\mathit{adv}}$ for states perturbed by universal perturbations. UAP-S,
UAP-O and OSFW(U) need to be modified to compute the total variation distance,
and we leave this as a future work.

Table 3 presents the online and offline computational cost for the
perturbation generation. The results are comparable with Table 4.2.2, and
confirm that FGSM, UAP-S, UAP-O and OSFW(U) can be mounted in real-time,
although FGSM needs write-access to the agent’s memory. OSFW cannot inject the
generated noise immediately into subsequent states, since it has a higher
online cost than the maximum upper bound.

|  | Average return $\pm$ std in the presence of adversarial perturbation attacks
---|---|---
epsilon | Defense | No attack | FGSM | OSFW | UAP-S | UAP-O | OSFW(U)
$0.01$ | No defense | $21.0\pm 0.0$ | $\bm{-21.0\pm 0.0}$ | $-20.0\pm 3.0$ | $\bm{-21.0\pm 0.0}$ | $-19.8\pm 0.4$ | $\bm{-21.0\pm 0.0}$
VF (lin2017detecting, ) | $21.0\pm 0.0$ | $21.0\pm 0.0$ | $-19.7\pm 0.5$ | $0.7\pm 1.7$ | $0.4\pm 2.7$ | $\bm{-21.0\pm 0.0}$
SA-MDP (zhang2020robust, ) | $21.0\pm 0.0$ | $21.0\pm 0.0$ | $21.0\pm 0.0$ | $21.0\pm 0.0$ | $21.0\pm 0.0$ | $21.0\pm 0.0$
$0.02$ | No defense | $21.0\pm 0.0$ | $-19.9\pm 1.3$ | $\bm{-21.0\pm 0.0}$ | $-20.8\pm 0.6$ | $-20.0\pm 0.0$ | $\bm{-21.0\pm 0.0}$
VF (lin2017detecting, ) | $21.0\pm 0.0$ | $21\pm 0.0$ | $-19.7\pm 0.6$ | $9.4\pm 0.8$ | $5.3\pm 3.9$ | $-20.5\pm 0.5$
SA-MDP (zhang2020robust, ) | $21.0\pm 0.0$ | $-14.6\pm 8.8$ | $-20.5\pm 0.5$ | $-20.6\pm 0.5$ | $-20.6\pm 0.5$ | $\bm{-21.0\pm 0.0}$
$0.05$ | No defense | $21.0\pm 0.0$ | $-20.5\pm 0.7$ | $\bm{-21.0\pm 0.0}$ | $-20.6\pm 0.8$ | $-20.0\pm 0.0$ | $\bm{-21.0\pm 0.0}$
VF (lin2017detecting, ) | $21.0\pm 0.0$ | $21.0\pm 0.0$ | $-20.0\pm 0.0$ | $7.6\pm 4.7$ | $-14.1\pm 1.1$ | $\bm{-21.0\pm 0.0}$
SA-MDP (zhang2020robust, ) | $21.0\pm 0.0$ | $\bm{-21.0\pm 0.0}$ | $\bm{-21.0\pm 0.0}$ | $-20.6\pm 0.5$ | $-20.6\pm 0.5$ | $\bm{-21.0\pm 0.0}$

(a) DQN agent playing Pong

|  | Average return $\pm$ std in the presence of adversarial perturbation attacks
---|---|---
epsilon | Defense | No attack | FGSM | OSFW | UAP-S | UAP-O | OSFW(U)
$0.01$ | No defense | $34.0\pm 0.0$ | $\bm{0.0\pm 0.0}$ | $2.0\pm 1.1$ | $2.1\pm 0.8$ | $4.0\pm 0.6$ | $0.5\pm 0.5$
VF (lin2017detecting, ) | $32.0\pm 1.5$ | $32.6\pm 1.7$ | $24.1\pm 1.0$ | $22.9\pm 0.9$ | $25.8\pm 1.1$ | $\bm{20.9\pm 1.2}$
SA-MDP (zhang2020robust, ) | $30.0\pm 0.0$ | $30.0\pm 0.0$ | $30.0\pm 0.0$ | $30.0\pm 0.0$ | $30.0\pm 0.0$ | $30.0\pm 0.0$
$0.02$ | No defense | $34.0\pm 0.0$ | $\bm{0.0\pm 0.0}$ | $1.0\pm 0.0$ | $0.1\pm 0.3$ | $0.8\pm 0.6$ | $\bm{0.0\pm 0.0}$
VF (lin2017detecting, ) | $32.0\pm 1.5$ | $32.6\pm 1.7$ | $\bm{1.1\pm 0.3}$ | $24.0\pm 2.0$ | $25.6\pm 1.0$ | $4.4\pm 1.1$
SA-MDP (zhang2020robust, ) | $30.0\pm 0.0$ | $29.8\pm 0.6$ | $29.9\pm 0.3$ | $\bm{29.4\pm 1.2}$ | $\bm{29.4\pm 1.2}$ | $30.0\pm 0.0$
$0.05$ | No defense | $34.0\pm 0.0$ | $\bm{0.0\pm 0.0}$ | $1.2\pm 0.0$ | $2.2\pm 1.7$ | $2.2\pm 1.4$ | $\bm{0.0\pm 0.0}$
VF (lin2017detecting, ) | $32.0\pm 1.4$ | $32.6\pm 1.6$ | $1.0\pm 0.0$ | $29.0\pm 1.1$ | $23.9\pm 0.3$ | $\bm{0.0\pm 0.0}$
SA-MDP (zhang2020robust, ) | $30.0\pm 0.0$ | $21.1\pm 1.3$ | $\bm{20.9\pm 0.8}$ | $21.1\pm 1.7$ | $21.1\pm 1.7$ | $21.1\pm 1.7$

(b) DQN agent playing Freeway

Table 4. Average returns (10 episodes) for the DQN agent playing Pong in the
presence of different adversarial perturbations, and agents are equipped with
different defenses. In each row, the best attack (lowest return) is in bold.
In each column, for a given $\epsilon$ value, the most robust defense (highest
return) for that particular attack is shaded green if the defense can fully
recover the victim’s return, and blue if the victim’s return is not fully
recovered.

##  5\. Detection and Mitigation of Adversarial Perturbations

A defender’s goal is the opposite of the adversary $\mathit{Adv}$’s goal.
Section 3.1 outlines three criteria for a successful attack. A good defense
mechanism should thwart one or more of these criteria. We begin by surveying
previously proposed defenses (Section 5.1), followed by an evaluation of the
effectiveness of two of the most relevant defenses (Section 5.2) that aim to
address the first (lowering the victim $\mathit{v}$’s return) and third attack
criterion (evasion of detection). We then present a new approach to detect the
presence of adversarial perturbations in state observations (Section 5.3),
aimed at thwarting third attack criterion on evading defenses.

###  5.1. Defenses against Adversarial Examples

####  5.1.1. Defenses in image classification

Many prior studies proposed different methods to differentiate between normal
images and adversarial examples. Meng and Chen (meng2017magnet, ) or Rouhani
et al. (rouhani2018deepfense, ) model the underlying data manifolds of the
normal images to differentiate clean images from adversarial examples. Xu et
al. (xu2017feature, ) propose feature squeezing to detect adversarial
examples. However, these detection methods are found to be inefficient
(carlini2017adversarial, ) and not robust against adaptive adversaries that
can specifically tailor adversarial perturbation generation methods for a
given defense (tramer2020adaptive, ).

####  5.1.2. Defenses in DRL

Previous work on adversarial training (behzadan2017whatever, ; kos2017delving,
) presents promising results as a defense. However, Zhang et al.
(zhang2020robust, ) show that adversarial training leads to unstable training,
performance degradation, and is not robust against strong attacks. Moreover,
Moosavi et al. (moosavi2017universal, ) prove that despite a slight decrease
in the fooling rate $\delta$ in the test set, $\mathit{Adv}$ can easily
compute another universal perturbation against retrained agents. To overcome
the challenges of adversarial training, Zhang et al. (zhang2020robust, )
propose state-adversarial Markov decision process (SA-MDP), which aims to find
an optimal $\pi$ under the strongest $\mathit{Adv}$ using policy
regularization. This regularization technique helps DRL agents _maintain_
their performance even against adversarially perturbed inputs. Similarly,
Oikarinen et al. (oikarinen2021robust, ) use adversarial loss functions during
training to improve the robustness of agents.

Visual Foresight (VF) (lin2017detecting, ) is another defense that _recovers_
the performance of an agent in the presence of $\mathit{Adv}$. VF predicts the
current observation $\hat{o}_{t}$ at time $t$ using $k$ previous observations
$o_{t-k}:o_{t-1}$ and corresponding actions $a_{t-k}:a_{t-1}$. It also
predicts the possible action $\hat{a}_{t}$ for the partially predicted
$\hat{\bm{s}}$ using $Q(\hat{\bm{s}},a)$. The difference between
$Q(\hat{\bm{s}},a)$ and $Q(\hat{\bm{s}},\hat{a})$ determines whether $\bm{s}$
is perturbed or not. In the case of detection, $\hat{a}_{t}$ is selected to
recover the performance.

###  5.2. Effectiveness of Existing Defenses

To investigate the limitations of previously proposed defenses for DRL, we
implemented two defense methods that aim to retain the average return of
$\mathit{v}$ when it is under attack: VF (lin2017detecting, ) and SA-MDP
(zhang2020robust, ), both of which seek to prevent the first attack objective
that limits $\mathit{v}$ to a low return. VF also prevents the third attack
criterion and detects adversarial perturbations. Since we want to evaluate the
effectiveness of VF and SA-MDP, we focus on the DQN agents for Pong and
Freeway as these are the ones that are common between our experiments (Section
4) and these defenses ((lin2017detecting, ; zhang2020robust, )).

We implemented VF from scratch for our DQN models following the original
experimental setup in (lin2017detecting, ) by setting $k=3$ to predict every
$4^{th}$ observation. We also set the pre-defined threshold value to $0.01$,
which is used to detect adversarial perturbations to achieve the highest
detection rate and performance recovery. We downloaded state adversarial DQN
agents, which are trained using SA-MDP, from their reference
implementation333https://github.com/chenhongge/SA_DQN Downloaded SA-MDP agents
use $\epsilon=1/255$ in training as in the original work (zhang2020robust, ).
Using higher $\epsilon$ values in training led to poor performance.. SA-MDP
agents only use one observation per $\bm{s}$; therefore, UAP-S would reduce to
UAP-O for SA-MDP in this setup. Table 4 shows the average return for each
agent under a different attack, and while equipped with different defenses. In
Section 4, we established that when the perturbation bound $\epsilon$ is
$0.01$, all attacks are devastatingly effective. In the interest of evaluating
the robustness of the defense, we also consider two higher $\epsilon$ values,
$0.02$ and $0.05$.

VF is an effective defense against FGSM and UAP-S as it can recover
$\mathit{v}$’s average return. However, it is not very effective against OSFW,
OSFW(U), and UAP-O. SA-MDP is better than VF against OSFW and OSFW(U) when
$\epsilon=0.01$, but it fails to defend any attack in Pong when $\epsilon\geq
0.02$. Notably, VF’s detection performance depends on the accuracy of the
action-conditioned frame prediction module used in its algorithm
(lin2017detecting, ), and the pre-defined threshold value used for detecting
adversarial perturbations. SA-MDP is able to partially retain the performance
of $\mathit{v}$ when $\epsilon=0.01$ and $0.02$ in the case of Freeway.
Notably, $\mathit{v}$’s average return when it is under attack is still
impacted, and SA-MDP cannot retain the same average return as when there is no
attack. Furthermore, SA-MDP’s effectiveness decreases drastically when the
perturbation bound $\epsilon$ is increased to $0.05$.

###  5.3. Action Distribution Divergence Detector(AD3)

####  5.3.1. Methodology

In a typical DRL episode, sequential actions exhibit some degree of _temporal
coherence_ : the likelihood of the agent selecting a specific current action
given a specific last action is consistent across different episodes. We also
observed that the temporal coherence is disrupted when the episode is
subjected to an attack. We leverage this knowledge to propose a detection
method, Action Distribution Divergence Detector (AD3), which calculates the
statistical distance between the _conditional action probability distribution_
(CAPD) of the current episode to the learned CAPD in order to detect whether
the agent is under attack. Unlike prior work on detecting adversarial examples
in the image domain (meng2017magnet, ; rouhani2018deepfense, ; xu2017feature,
), AD3 does not analyze the input image or tries to detect adversarial
examples. Instead, it observes the distribution of the actions triggered by
the inputs and detects unusual action sequences. To train AD3, $\mathit{v}$
first runs $k_{1}$ episodes in a controlled environment before deployment. AD3
saves all actions taken during that time and approximates the conditional
probability of the next action given the current one using the bigram
model444We tested different ngrams and selected the bigram as the best
option.. We call the conditional probability of actions approximated by
$k_{1}$ episodes the _learned_ CAPD. Second, to differentiate between the CAPD
of a normal game versus a game that is under attack with high confidence, $v$
runs another $k_{2}$ episodes in a safe environment. AD3 decides a threshold
value $th$, where the statistical distance between the CAPD of the normal game
and the learned CAPD falls mostly below this threshold. We use Kullback-
Leibler (KL) divergence (kl-div, ) as the statistical distance measure. The KL
divergence between the learned CAPD and the CAPD of the current episode is
calculated at each time-step starting after the first $t_{1}$ steps. We skip
the first $t_{1}$ steps because the CAPD of the current episode is initially
unstable and the KL divergence is naturally high at the beginning of every
episode. We set the threshold $th$ as the $p^{th}$ percentile of all KL-
divergence values calculated for $k_{2}$ episodes. During deployment, AD3
continuously updates the CAPD of the current episode, and after $t_{1}$ steps,
it calculates the KL-divergence between the CAPD of the current episode and
the learned CAPD. If the KL-divergence exceeds the threshold $th$ by $r$% or
more during a time window $t_{2}$, then AD3 raises an alarm that the agent is
under attack.

####  5.3.2. Evaluation

We evaluate the precision and recall of AD3 in three tasks with discrete
action spaces (Pong, Freeway and Breakout) against all proposed attacks. AD3
can detect all five attacks in Pong with perfect precision and recall scores
in all configurations. In Freeway, AD3 has perfect precision and recall scores
for 12 out of 15 different setups (FGSM against the DQN agent, OSFW against
the PPO agent, and OSFW(U) against the PPO agent). In Freeway, we found that
attacks lead to a lower action change rate (as low as $20\%-30\%$ in some
episodes) than other tasks, thus negatively affecting the precision and/or
recall of AD3. AD3 is also less effective in Breakout, as this task often
terminates too quickly when it is under attack, and AD3 cannot store enough
actions for CAPD to converge in such a short time. The optimal parameters used
for training AD3, a more detailed discussion of the performance of AD3, and
the full result of the evaluation can be found in Appendix C.

####  5.3.3. Limitations

AD3 is designed for attacks where $\mathit{Adv}$ injects adversarial
perturbations consistently throughout the episode. $\mathit{Adv}$ with the
knowledge of the detection strategy could apply their adversarial perturbation
at a lower frequency to avoid detection. However, lowering the attack
frequency also decreases the attack effectiveness. Another way for
$\mathit{Adv}$ to evade AD3 is to perform targeted attacks to lure
$\mathit{v}$ into a specific state, where the adversarial perturbation is
applied to a limited number of states in an episode. This type of attack is
outside the scope of our paper, and defense strategies against it will be
explored in future work.

####  5.3.4. Adversary vs. defender strategy with negative returns

|  | Losing Rate
---|---|---
ϵ | Method | No attack | FGSM | OSFW | UAP-S | UAP-O | OSFW(U)
$0.01$ | No defense | 0.0 | $1.0$ | $1.0$ | $1.0$ | $1.0$ | $1.0$
VF (lin2017detecting, ) | 0.0 | 0.0 | $1.0$ | 0.0 | 0.2 | $1.0$
SA-MDP (zhang2020robust, ) | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
| AD3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
$0.02$ | No defense | 0.0 | $1.0$ | $1.0$ | $1.0$ | $1.0$ | $1.0$
VF (lin2017detecting, ) | 0.0 | 0.0 | $1.0$ | 0.0 | 0.3 | $1.0$
SA-MDP (zhang2020robust, ) | 0.0 | $0.9$ | $1.0$ | $1.0$ | $1.0$ | $1.0$
| AD3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0

Table 5. Losing rate (10 episodes) of DQN agents playing Pong with or without
additional defense. Losing rate is calculated by counting the number of games
where the computer gains 21 points first in an episode. If AD3 raises an alarm
before an episode ends, then $\mathit{v}$ does not lose the game. In each row,
the best attack with the highest losing rate is in bold, and given an
$\epsilon$ value, the defense with the highest losing rate for that particular
attack is shaded red.

In any DRL task where there is a clear negative result for an episode (e.g.,
losing a game) or the possibility of negative return, a reasonable choice for
$\mathit{v}$ is to suspend an episode when $\mathit{Adv}$’s presence is
detected. For example, in Pong, a negative result would be when the computer
(as the opponent) reaches the score of 21 before $\mathit{v}$ does. Suspending
an episode prevents $\mathit{v}$ from losing the game. Defense mechanisms such
as VF and SA-MDP are useful in retaining or recovering $\mathit{v}$’s return;
however, they may not always prevent $\mathit{v}$ from falling into a negative
result, e.g., losing the game in Pong. Combining a recovery/retention
mechanism with suspension on attack detection can reduce the number of losses
for $\mathit{v}$.

To illustrate the effectiveness of combining AD3 with a retention/recovery
mechanism, we designed an experiment using a DQN agent playing Pong to compare
_losing rate_ of $\mathit{v}$ when it is under attack. We used Pong for this
experiment, as Pong has a clear negative result. In Pong, an episode ends with
loss when (a) the computer reaches 21 points before $\mathit{v}$, or (b) AD3
did not raise an alarm. The result of this experiment can be found in Table
5.3.4. As shown in this table, VF is not effective in reducing the losing rate
of $\mathit{v}$ for OSFW and OSFW(U) for all $\epsilon$. SA-MDP is effective
in avoiding losses when $\epsilon=0.01$; however, it fails against all
universal perturbations when $\epsilon=0.02$. In contrast, AD3 can detect the
presence of adversarial perturbations in all games. Although
retention/recovery and detection are two orthogonal aspects of defense, our
results above suggest that they can be combined in tasks with negative returns
or results in order to more effectively thwart $\mathit{Adv}$ from achieving
its first goal. In the scenario where the defense mechanism fails to recover
enough reward for the $\mathit{v}$ to avoid a negative end state, the
detection module can be used as a forensic method to understand where and why
the system fails and to forfeit negative results when $\mathit{Adv}$ is
present.

##  6\. Conclusion

We showed that white-box universal perturbation attacks are effective in
fooling DRL policies in real-time. Our evaluation of the three different
attacks (UAP-S, UAP-O, and OSFW(U)) demonstrates that universal perturbations
are effective in tasks with discrete action spaces. Universal perturbation
attacks are also able to generalize to continuous control tasks with the same
efficiency. We confirmed that the effectiveness of prior defenses depends on
the perturbation bound, and fail to completely recover the agent performance
when they are confronted with universal perturbations of larger bounds. We
proposed a detection mechanism, AD3, that detects all five attacks evaluated
in the paper. AD3 can be combined with other defense techniques to protect
agents in tasks with negative returns or results to stop the adversary from
achieving its goal. We plan to extend our attacks to the black-box case by
first mounting a model extraction attack and then applying our current
techniques to find transferable universal perturbations.

#### Acknowledgements

This research was funded in part by the EU H2020 project SPATIAL (Grant No.
101021808) and Intel Private-AI Consortium.

## References

  * (1) Baluja, S., and Fischer, I. Learning to attack: Adversarial transformation networks.  In Proceedings of AAAI-2018 (2018).
  * (2) Behzadan, V., and Munir, A. Vulnerability of deep reinforcement learning to policy induction attacks.  In International Conference on Machine Learning and Data Mining in Pattern Recognition (2017), Springer, pp. 262–275.
  * (3) Behzadan, V., and Munir, A. Whatever does not kill deep reinforcement learning, makes it stronger.  arXiv preprint arXiv:1712.09344 (2017).
  * (4) Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents.  Journal of Artificial Intelligence Research 47 (2013), 253–279.
  * (5) Carlini, N., and Wagner, D. Adversarial examples are not easily detected: Bypassing ten detection methods.  In Proceedings of the 10th ACM workshop on artificial intelligence and security (2017), pp. 3–14.
  * (6) Carlini, N., and Wagner, D. Towards evaluating the robustness of neural networks.  In 2017 IEEE Symposium on Security and Privacy (SP) (2017), pp. 39–57.
  * (7) Co, K. T., Muñoz-González, L., de Maupeou, S., and Lupu, E. C. Procedural noise adversarial examples for black-box attacks on deep convolutional networks.  In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security (2019), pp. 275–289.
  * (8) Gleave, A., Dennis, M., Kant, N., Wild, C., Levine, S., and Russell, S. Adversarial policies: Attacking deep reinforcement learning.  arXiv preprint arXiv:1905.10615 (2019).
  * (9) Goodfellow, I., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples.  In International Conference on Learning Representations (2015).
  * (10) Hayes, J., and Danezis, G. Learning universal adversarial perturbations with generative models.  In 2018 IEEE Security and Privacy Workshops (SPW) (2018), IEEE, pp. 43–49.
  * (11) Huang, S., Papernot, N., Goodfellow, I., Duan, Y., and Abbeel, P. Adversarial attacks on neural network policies.  arXiv (2017).
  * (12) Hussenot, L., Geist, M., and Pietquin, O. Copycat: Taking control of neural policies with constant attacks.  In International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS) (2020).
  * (13) Inkawhich, M., Chen, Y., and Li, H. Snooping attacks on deep reinforcement learning.  In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (Richland, SC, 2020), AAMAS ’20, p. 557–565.
  * (14) Kos, J., and Song, D. Delving into adversarial attacks on deep policies.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings (2017), OpenReview.net.
  * (15) Kullback, S., and Leibler, R. A. On information and sufficiency.  The annals of mathematical statistics 22, 1 (1951), 79–86.
  * (16) Lin, Y., Hong, Z., Liao, Y., Shih, M., Liu, M., and Sun, M. Tactics of adversarial attack on deep reinforcement learning agents.  In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017 (2017), ijcai.org, pp. 3756–3762.
  * (17) Lin, Y., Liu, M., Sun, M., and Huang, J. Detecting adversarial attacks on neural network policies with visual foresight.  CoRR abs/1710.00814 (2017).
  * (18) Meng, D., and Chen, H. MagNet: a two-pronged defense against adversarial examples.  In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (2017), pp. 135–147.
  * (19) Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning.  In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 (2016), vol. 48, pp. 1928–1937.
  * (20) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M. A., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning.  Nature 518, 7540 (2015), 529–533.
  * (21) Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., and Frossard, P. Universal adversarial perturbations.  In Proceedings of the IEEE conference on computer vision and pattern recognition (2017), pp. 1765–1773.
  * (22) Moosavi-Dezfooli, S.-M., Fawzi, A., and Frossard, P. DeepFool: a simple and accurate method to fool deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition (2016), pp. 2574–2582.
  * (23) Mopuri, K. R., Garg, U., and Babu, R. V. Fast feature fool: A data independent approach to universal adversarial perturbations.  arXiv preprint arXiv:1707.05572 (2017).
  * (24) Mopuri, K. R., Ojha, U., Garg, U., and Babu, R. V. NAG: network for adversary generation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), pp. 742–751.
  * (25) Oikarinen, T., Zhang, W., Megretski, A., Daniel, L., and Weng, T.-W. Robust deep reinforcement learning through adversarial loss.  In Advances in Neural Information Processing Systems (2021), M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34, Curran Associates, Inc., pp. 26156–26167.
  * (26) Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., and Swami, A. The limitations of deep learning in adversarial settings.  In 2016 IEEE European symposium on security and privacy (EuroS&P) (2016), IEEE, pp. 372–387.
  * (27) Rouhani, B. D., Samragh, M., Javaheripi, M., Javidi, T., and Koushanfar, F. DeepFense: Online accelerated defense against adversarial deep learning.  IEEE, pp. 1–8.
  * (28) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. ImageNet large scale visual recognition challenge.  International journal of computer vision 115, 3 (2015), 211–252.
  * (29) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms.  CoRR abs/1707.06347 (2017).
  * (30) Sun, J., Zhang, T., Xie, X., Ma, L., Zheng, Y., Chen, K., and Liu, Y. Stealthy and efficient adversarial attacks against deep reinforcement learning.  In Proceedings of the AAAI Conference on Artificial Intelligence (2020), AAAI Press, pp. 5883–5891.
  * (31) Sutton, R. S., and Barto, A. G. Reinforcement learning: An introduction.  MIT press, 2018.
  * (32) Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks.  In International Conference on Learning Representations (2014).
  * (33) Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. DeepMind control suite.  arXiv preprint arXiv:1801.00690 (2018).
  * (34) Todorov, E., Erez, T., and Tassa, Y. MuJoCo: A physics engine for model-based control.  In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (2012), IEEE, pp. 5026–5033.
  * (35) Tramer, F., Carlini, N., Brendel, W., and Madry, A. On adaptive attacks to adversarial example defenses.  In Advances in Neural Information Processing Systems (2020), H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33, Curran Associates, Inc., pp. 1633–1645.
  * (36) Tretschk, E., Oh, S. J., and Fritz, M. Sequential attacks on agents for long-term adversarial goals.  CoRR abs/1805.12487 (2018).
  * (37) Wu, X., Guo, W., Wei, H., and Xing, X. Adversarial policy training against deep reinforcement learning.  In 30th USENIX Security Symposium (USENIX Security 21) (Aug. 2021), USENIX Association, pp. 1883–1900.
  * (38) Xiao, C., Pan, X., He, W., Peng, J., Sun, M., Yi, J., Li, B., and Song, D. Characterizing attacks on deep reinforcement learning.  arXiv preprint arXiv:1907.09470 (2019).
  * (39) Xu, W., Evans, D., and Qi, Y. Feature squeezing: Detecting adversarial examples in deep neural networks.  In 25th Annual Network and Distributed System Security Symposium, NDSS (2018), The Internet Society.
  * (40) Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D., and Hsieh, C.-J. Robust deep reinforcement learning against adversarial perturbations on state observations.  In Advances in Neural Information Processing Systems (2020), vol. 33, Curran Associates, Inc., pp. 21024–21037.
  * (41) Zhang, X., Zou, J., He, K., and Sun, J. Accelerating very deep convolutional networks for classification and detection.  IEEE transactions on pattern analysis and machine intelligence 38, 10 (2015), 1943–1955.
  * (42) Zhao, Y., Shumailov, I., Cui, H., Gao, X., Mullins, R., and Anderson, R. Blackbox attacks on reinforcement learning agents using approximated temporal information.  In 2020 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W) (Los Alamitos, CA, USA, 2020), IEEE, IEEE Computer Society, pp. 16–24.

##  Appendix A Proof of Equation 6

In this section, we provide a formal proof of Equation 6 that was used to find
the perturbation in UAP-O.

In UAP-O, we defined an additional constraint for the perturbation as
$\tilde{r}_{j}=\tilde{r}_{k},\;\forall j,k\in\\{t-N+1,\cdots,t\\}$, so that
the same $\tilde{r}_{j}$ is applied into all observations $o_{j}$. As pointed
out in Section 3.2, at $i$-th iteration, the DeepFool algorithm computes the
$i$-th perturbation $\bm{r}_{i}$ for $i$-th state $\bm{s}_{i}$ as

(8)
$\Delta\bm{r}_{i}\leftarrow\frac{|Q^{{}^{\prime}}(\bm{s}^{*}_{i},a_{\hat{l}})|}{{\lVert{\bm{w}^{{}^{\prime}}_{\hat{l}}}\rVert}^{2}_{2}}\bm{w}^{{}^{\prime}}_{\hat{l}}.$

We can keep the term
$|Q^{{}^{\prime}}(\bm{s}^{*}_{i},a_{\hat{l}})|/{{\lVert{\bm{w}^{{}^{\prime}}_{\hat{l}}}\rVert}^{2}_{2}}$
as constant, since it gives the magnitude of the perturbation update
$\Delta\bm{r}_{i}$, and we can maintain the same magnitude of
$\Delta\bm{r}_{i}$ in $\Delta\tilde{\bm{r}}_{i}$. Then, in UAP-O, DeepFool
finds the closest $\Delta\tilde{\bm{r}}_{i}$ to $\Delta\bm{r}_{i}$ by
minimizing the direction of the perturbation as
${\lVert{\bm{w}^{{}^{\prime}}_{\hat{l}}-\Delta\tilde{\bm{r}}_{i}}\rVert}^{2}_{2}$.

We can find the optimal $\Delta\tilde{\bm{r}}_{i}$ by setting
$\partial{\lVert{\bm{w}^{{}^{\prime}}_{\hat{l}}-\Delta\tilde{\bm{r}}_{i}}\rVert}_{2}/{\partial\Delta\tilde{r}_{i_{j}}}$
to zero for all $j\in\\{t-N+1,\cdots,t\\}$, where $N$ denotes the total number
of observations inside state $\bm{s}$. We also set all
$\Delta\tilde{r}_{i_{j}}$ into the same variable $\tilde{r}_{i_{t}}$.
Therefore, we simply obtain $\Delta\tilde{r}_{i_{t}}$ by finding the point
where the partial derivative is equal to zero as in Equation 9.

(9)
$\begin{split}\frac{\partial{\lVert{\bm{w}^{{}^{\prime}}_{\hat{l}}-\Delta\tilde{r}_{i_{t}}}\rVert}_{2}}{{\partial\Delta\tilde{r}_{i_{t}}}}&=\frac{\partial((\bm{w}^{{}^{\prime}}_{\hat{l}_{t}}-\Delta\tilde{r}_{i_{t}})^{2}+\cdots+(\bm{w}^{{}^{\prime}}_{\hat{l}_{t-N}}-\Delta\tilde{r}_{i_{t}})^{2})^{1/2}}{{\partial\Delta\tilde{r}_{i_{t}}}}\\\
0&=\frac{\partial
f(\Delta\tilde{r}_{i_{t}})^{1/2}}{{\partial\Delta\tilde{r}_{i_{t}}}}\\\
0&=0.5f(\Delta\tilde{r}_{i_{t}})^{-3/2}\Big{(}2(N+1)\Delta\tilde{r}_{i_{t}}-2\;\smashoperator[]{\sum_{k=t-N+1}^{t}}\bm{w}^{{}^{\prime}}_{\hat{l}_{k}}\Big{)}\\\
\end{split}$

The squared difference function $f(\Delta\tilde{r}_{i_{t}})^{-3/2}\geq 0$, and
only equal to zero when $\Delta\tilde{r}_{i_{t}}=0$, which does not give any
useful perturbation. Therefore, Equation 9 is solved when
$\bm{w}^{{}^{\prime}}_{\hat{l}_{k}}=\Delta\tilde{r}_{i_{t}},\;\forall
k\in\\{t-N+1,\cdots,t\\}$. Therefore, by putting back the constant, DeepFool
computes the perturbation at $i^{th}$ iteration as

(10)
$\Delta\tilde{r}_{i_{j}}\leftarrow\frac{|Q^{{}^{\prime}}(\bm{s}^{*}_{i},a_{\hat{l}})|}{N{\lVert{\bm{w}^{{}^{\prime}}_{\hat{l}}}\rVert}^{2}_{2}}\quad\smashoperator[]{\sum_{k=(t-N+1)}^{t}}\;\bm{w}^{{}^{\prime}}_{\hat{l}_{k}}.$

##  Appendix B Adversarial Perturbations

On top of comparing the perturbation size in Pong (Figure 3, Section 4.2), we
also include results comparing the perturbation size for other games. As shown
in Figure 5, FGSM generates a perturbation that contains grey patches where
the sign of the gradient of the loss is zero against the DQN agent. It has
also completely black and white pixels due to the $l_{\infty}$ constraint,
which makes the perturbation more visible. In addition, Figure 6 compares the
clean and perturbed frames generated by different attacks. Note that we inject
the perturbation into pre-processed frames (i.e., resized and gray-scale
frames), so we show both RGB and gray-scale clean frames in Figure 5 and
Figure 6. In Pong, the perturbation size in UAP-S is the smallest when
compared to other attacks.

[¡short description¿]¡long description¿

(a) Clean (RGB)

(b) Clean (Gray)

(c) FGSM

(d) OSFW

(e) UAP-S

(f) UAP-O

(g) OSFW(U)

Figure 5. Comparison of perturbation size added into the same clean (in both
RGB and gray-scale) observation in different attacks when $\epsilon=0.01$.
UAP-S and UAP-O generate smaller perturbations. Top row: DQN agent playing
Pong, Middle row: PPO agent playing Breakout, Bottom row: A2C agent playing
Freeway. In perturbation images, black pixels: $-0.01$, white pixels: $+0.01$,
gray pixels: $0.0$.

[¡short description¿]¡long description¿

(a) Clean (RGB)

(b) Clean (Gray)

(c) FGSM

(d) OSFW

(e) UAP-S

(f) UAP-O

(g) OSFW(U)

Figure 6. Comparison of the clean (in both RGB and gray-scale) and perturbed
observations in different attacks when $\epsilon=0.01$. UAP-S and UAP-O
generate smaller perturbations that are less visible. Top row: DQN agent
playing Pong, Middle row: PPO agent playing Breakout, Bottom row: A2C agent
playing Freeway.

##  Appendix C Optimal Parameters for AD3

Game | Agent | Parameters | Attack | Precision/Recall
---|---|---|---|---
Pong | DQN |  $k_{1}=12$, $k_{2}=24$, $p=100$, $r=0.9$, $t_{1}=400$, $t_{2}=200$ | FGSM | 1.0/1.0
OSFW(U) | 1.0/1.0
UAP-S | 1.0/1.0
UAP-O | 1.0/1.0
OSFW | 1.0/1.0
A2C |  $k_{1}=12$, $k_{2}=24$, $p=100$, $r=0.9$, $t_{1}=400$, $t_{2}=200$ | FGSM | 1.0/1.0
OSFW | 1.0/1.0
UAP-S | 1.0/1.0
UAP-O | 1.0/1.0
OSFW(U) | 1.0/1.0
PPO |  $k_{1}=12$, $k_{2}=24$, $p=100$, $r=0.9$, $t_{1}=400$, $t_{2}=200$ | FGSM | 1.0/1.0
OSFW | 1.0/1.0
UAP-S | 1.0/1.0
UAP-O | 1.0/1.0
OSFW(U) | 1.0/1.0
Freeway | DQN |  $k_{1}=12$, $k_{2}=24$, $p=95$ , $r=0.8$, $t_{1}=400$, $t_{2}=200$ | FGSM | 1.0/0.8
OSFW | 1.0/1.0
UAP-S | 1.0/1.0
UAP-O | 1.0/1.0
OSFW(U) | 1.0/1.0
a2c |  $k_{1}=12$, $k_{2}=24$, $p=100$, $r=1.0$, $t_{1}=400$, $t_{2}=200$ | FGSM | 1.0/1.0
OSFW | 1.0/1.0
UAP-S | 1.0/1.0
UAP-O | 1.0/1.0
OSFW(U) | 1.0/1.0
PPO |  $k_{1}=12$, $k_{2}=24$, $p=90$, $r=0.9$, $t_{1}=200$, $t_{2}=200$ | FGSM | 1.0/1.0
OSFW | 1.0/0.5
UAP-S | 1.0/1.0
UAP-O | 1.0/1.0
OSFW(U) | 1.0/0.6
Breakout | DQN |  $k_{1}=12$, $k_{2}=24$, $p=97$, $r=0.6$, $t_{1}=60$, $t_{2}=40$ | FGSM | 0.7/1.0
OSFW | 0.7/1.0
UAP-S | 0.7/1.0
UAP-O | 0.7/1.0
OSFW(U) | 0.2/0.2
A2C |  $k_{1}=12$, $k_{2}=24$, $p=100$, $r=0.8$, $t_{1}=60$, $t_{2}=40$ | FGSM | 1.0/1.0
OSFW | 1.0/1.0
UAP-S | 1.0/0.8
UAP-O | 1.0/1.0
OSFW(U) | 1.0/1.0
PPO |  $k_{1}=12$, $k_{2}=24$, $p=97$, $r=0.8$, $t_{1}=60$, $t_{2}=40$ | FGSM | 1.0/1.0
OSFW | 0.8/1.0
UAP-S | 0.8/1.0
UAP-O | 0.8/1.0
OSFW(U) | 0.8/1.0

Table 6. Optimal values of AD3 to detect five different attacks: FGSM, OSFW,
UAP-S, UAP-O, and OSFW(U) at $\epsilon=0.01$ for all games and agents over ten
episodes.

We performed a grid search to find optimal parameters for AD3 in Breakout,
Pong, and Freeway using three different agents (DQN, A2C, and PPO) and choose
the parameters with the best F1-score. The optimal parameter values can be
found in Table 6. The precision and recall scores of AD3 against five
different attacks at $\epsilon=0.01$ can also be found in this table. AD3 is
able to detect the presence of the adversary $\mathit{Adv}$ with perfect
precision and recall for most environments and agents. However, there are some
combinations of environment and agent, where AD3 has a low recall and/or
precision.

Attacks such as OSFW(U) in Freeway against PPO agents have a low action change
rate (as low as $20\%-30\%$ in some episodes), and the resulting average
return is still high. This means that $\mathit{Adv}$ does not change many
actions in an episode. This low action change rate is consistent with all of
the combinations of the game, agent, and attack that have a low recall score.
As AD3 relies on modeling the distribution of the victim $\mathit{v}$’s action
distribution during an episode, it cannot detect the presence of
$\mathit{Adv}$ if there are not enough actions influenced by the attack in an
episode. This means that a low action change rate from $\mathit{Adv}$ would
make it difficult for AD3 to detect. The precision of AD3 in Breakout is lower
than other environments, since episodes in Breakout terminate very quickly (as
fast as in $112$ time steps) when $\mathit{v}$ is under attack. This means
that to detect the presence of $\mathit{Adv}$, AD3 has to raise an alarm early
in Breakout. However, AD3 relies on modeling CAPD of the current episode, and
it takes time to converge. Before CAPD converges, the anomaly score is going
to be high for any episode, including episodes with no attack presence. As an
episode can terminate quickly in Breakout, AD3 cannot store enough actions for
CAPD to converge, and this is reflected in the reported low precision values.