11institutetext: Washington University in St. Louis, MO 63130, USA
Department of Computer Science & Engineering
11email<EMAIL_ADDRESS>
# Reward Delay Attacks on Deep Reinforcement Learning

Anindya Sarkar  Jiarui Feng  Yevgeniy Vorobeychik  Christopher Gill  Ning
Zhang

###### Abstract

Most reinforcement learning algorithms implicitly assume strong synchrony. We
present novel attacks targeting Q-learning that exploit a vulnerability
entailed by this assumption by delaying the reward signal for a limited time
period. We consider two types of attack goals: _targeted attacks_ , which aim
to cause a target policy to be learned, and _untargeted attacks_ , which
simply aim to induce a policy with a low reward. We evaluate the efficacy of
the proposed attacks through a series of experiments. Our first observation is
that reward-delay attacks are extremely effective when the goal is simply to
minimize reward. Indeed, we find that even naive baseline reward-delay attacks
are also highly successful in minimizing the reward. Targeted attacks, on the
other hand, are more challenging, although we nevertheless demonstrate that
the proposed approaches remain highly effective at achieving the attacker’s
targets. In addition, we introduce a second threat model that captures a
minimal mitigation that ensures that rewards cannot be used out of sequence.
We find that this mitigation remains insufficient to ensure robustness to
attacks that delay, but preserve the order, of rewards.

###### Keywords:

Deep Reinforcement Learning Adversarial Attack Reward Delay Attack.

##  1 Introduction

In recent years, deep reinforcement learning (DRL) has achieved super-human
level performance in a number of applications including game playing [28],
clinical decision support [24], and autonomous driving [18]. However, as we
aspire to bring DRL to safety-critical settings, such as autonomous driving,
it is important to ensure that we can reliably train policies in realistic
scenarios, such as on autonomous vehicle testing tracks [1, 13, 29]. In such
settings, reward signals are often not given exogenously, but derived from
sensory information. For example, in lane following, the reward may be a
function of vehicle orientation and position relative to the center of the
lane, and these features are obtained from perception [14, 32]. Since many
such settings are also safety-critical, any adversarial tampering with the
training process—particularly, with the integrity of the reward stream derived
from perceptual information—can have disastrous consequences.

A number of recent efforts demonstrated vulnerability of deep reinforcement
learning algorithms to adversarial manipulation of the reward stream [3, 15,
20, 22]. We consider an orthogonal attack vector which presumes that the
adversary has compromised the _scheduler_ and is thereby able to manipulate
reward timing, but cannot modify rewards directly. For example, ROS 2.0
features modular design with few security checks and the ability to substitute
different _executors_ [8, 30, 6, 9, 5]. This means that once adversaries gain
access to the ROS software stack, they can replace its scheduling policy
readily, and as long as the executor behavior is not overtly malicious it can
be a long time before the compromise has been discovered. Additionally, we
assume that the adversary can infer (but not modify) memory contents using
side channel attacks. This is a realistic assumption, since it has been
demonstrated that it is feasible to leverage different types of system or
architectural side channels, such as cache-based or proc-fs based side
channels to infer secret information in other ROS modules [25, 10, 19].
However, write access to memory is often a lot more difficult to obtain due to
existing process isolation [12, 11].

Our attack exploits a common assumption of _synchrony_ in reinforcement
learning algorithms. Specifically, we assume that the adversary can delay
rewards a bounded number of time steps (for example, by scheduling tasks
computing a reward at time $t$ after the task computing a reward at time $t+k$
for some integer $k\geq 0$). We consider two variations of such _reward delay_
attacks. In the first, we allow the adversary to arbitrarily shuffle or drop
rewards, assuming effectively that no security mechanisms are in place at all.
Our second model evaluates the efficacy of the most basic security mechanism
in which we can detect any rewards computed out of their arrival sequence, for
example, through secure time stamping. Consequently, we propose the _reward
shifting_ attacks, where in order to remain undetected, the adversary can only
drop rewards, or _shift_ these a bounded number of steps into the future.
Efficacy comparison between these two threat models will then exhibit the
extent to which this simple security solution reduces vulnerability to reward
delay attacks. In both attack variants, we consider two adversarial goals:
untargeted attacks, which aim to minimize total reward accumulated at
prediction time (essentially, eroding the efficacy of training), and targeted
attacks, the goal of which is to cause the RL algorithm to learn a policy that
takes target actions in specific target states.

We specifically study attacks on deep Q-learning algorithms. The adversarial
model we introduce is a complex discrete dynamic optimization problem, even in
this more narrow class of DRL algorithms. We propose an algorithmic framework
for attacks that is itself based on deep Q learning, leveraging the fact that
the current Q function, along with the recent sequence of states, actions, and
rewards observed at training, provide sufficient information about system
state from the attacker’s perspective. The key practical challenge is how to
design an appropriate reward function, given that the “true” reward is a
property of the final policy resulting from long training, and only truly
possible to evaluate at test time. We address this problem by designing proxy
reward functions for both untargeted and targeted attacks that make use of
only the information immediately available at training time.

We evaluate the proposed approach experimentally using two Atari games in
OpenAI gym: Pong and Breakout. Our experiments demonstrate that the proposed
attacks are highly effective, and remain nearly as effective even with the
simple mitigation that ensures that rewards are not encountered out of order.
Altogether, our results demonstrate the importance of implementing sound
security practices, such as hardware and software-level synchrony assurance
[21, 27], in safety-critical applications of reinforcement learning.

##  2 Related Work

There are two closely related literature strands in attacks on reinforcement
learning and multiarmed bandits: attacks that take place at decision time, and
poisoning attacks.

#### Decision-Time attacks on Reinforcement Learning

Prior literature on adversarial attacks against RL has focused mainly on
inference time attacks [3, 15, 20, 22], where the RL policy $\pi$ is pre-
trained and fixed, and the attacker manipulates the perceived state $s_{t}$ of
the learner to $s^{\prime}_{t}$ in order to induce undesired actions, while
restricting $s^{\prime}_{t}$ and $s_{t}$ to be very similar to (or human-
indistinguishable from) each other. For example, in video games the attacker
can make small pixel perturbations to a frame to induce an action
$\pi(s^{\prime}_{t})\neq\pi(s_{t})$. [15] developed the uniform attack
mechanism, which generates adversarial examples by perturbing each image the
agent observes to attack a deep RL agent at every time step in an episode in
order to reduce the agent’s reward. [15] also introduced a decision time
targeted attack strategy, i.e. enchanting attack tactic, which is a planning-
based adversarial attack to mislead the agent towards a target state. [22]
proposed a strategically timed attack, which can reach the same effect of the
uniform attack by attacking the agent four times less often on average. [20]
leverages the policy’s value function as a guide for when to inject
adversarial perturbations and shows that with guided injection, the attacker
can inject perturbations in a fraction of the frames, and this is more
successful than injecting perturbations with the same frequency but no
guidance. [3] proposes an attack mechanism that exploits the transferability
of adversarial examples to implement policy induction attacks on deep Q
networks. Although test-time attacks can severely affect the performance of a
fixed policy $\pi$ during deployment, they do not modify the policy $\pi$
itself.

#### Reward Poisoning Attacks on Reinforcement Learning

Reward poisoning has been studied in bandits [2, 17, 26, 23], where the
authors show that an adversarially perturbed reward can mislead standard
bandit algorithms to suffer large regret. Reward poisoning has also been
studied in batch RL where rewards are stored in a pre-collected batch data set
by some behavior policy, and the attacker modifies the batch data [16, 35, 34,
36]. [16] provides a set of threat models for RL and establishes a framework
for studying strategic manipulation of cost signals in RL. [16] also provides
results to understand how manipulations of cost signals can affect Q-factors
and hence the policies learned by RL agents. [36] proposed an adaptive reward
poisoning attack against RL, where the perturbation at time t not only depends
on $(s_{t},a_{t},s_{t+1})$ but also relies on the RL agent’s Q-value at time
t. [35] presents a solution to the problem of finding limited incentives to
induce a particular target policy, and provides tractable methods to elicit
the desired policy after a few interactions. Note that all these previous
works directly modify the value of the reward signal itself, by adding a
quantity $\delta_{t}$ to the true reward $r(t)$, i.e.
$r^{\prime}(t)=r_{t}+\delta_{t}$. In contrast, we focus on delaying the reward
signal with the aim to mislead a learner RL agent, but cannot directly modify
the rewards. As such, our key contribution is the novel threat model that
effectively exploits the common synchrony assumption in reinforcement
learning.

##  3 Model

Consider a discounted Markov Decision Process (MDP) with a set of states $S$,
set of actions $A$, expected reward function $r(s,a)$, transition function
$P_{ss^{\prime}}^{\alpha}=\Pr\\{s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a\\}$,
discount factor $\gamma\in[0,1)$, and initial state distribution
$D(s)=\Pr\\{s_{0}=s\\}$. Suppose that we only know $S$ and $A$, but must learn
an optimal policy $\pi(s)$ from experience using reinforcement learning. To
this end, we consider a Deep $Q$-Network (DQN) reinforcement learning
framework; it is straightforward to extend our approach to other variants of
deep $Q$ learning. Let $Q(s,a;\theta)$ be a neural network with parameters
$\theta$ representing the $Q$-function, and let $Q_{t}(s,a;\theta_{t})$ denote
an approximation of the $Q$-function at iteration $t$ of RL (which we also
denote by $Q_{t}$ when the input is clear, and $Q_{t}(s,a;\theta)$ when we
treat $\theta$ as a variable). In the DQN, parameters $\theta$ are updated
after each iteration using the loss function
$L(\theta)=(r_{t}+\gamma\max_{a^{\prime}}Q_{t}(s_{t+1},a^{\prime};\theta)-Q_{t}(s_{t},a_{t};\theta))^{2}.$
Below, we omit the explicit dependence on $\theta$. As this update rule makes
evident, without experience replay the DQN training process itself is a Markov
decision process (MDP) in which
$\sigma_{t}=(Q_{t},(s_{t},a_{t},r_{t},s_{t+1}))$ constitutes state. Let $T$ be
the total number of DQN update iterations. This observation will be useful
below. Given a Q function obtained at the end of $T$ learning iterations,
$Q_{T}(s,a)$, we assume that the learner will follow a deterministic policy
that is optimal with respect to this function, i.e.,
$\pi(s)=\arg\max_{a}Q_{T}(s,a)$. For convenience, we abuse this notation
slightly, using $\pi(s,a)$ as an indicator which is 1 if action $a$ is played
in state $s$, and 0 otherwise.

Suppose that the attacker has compromised the scheduler, which can delay a
reward computed at any time step by a bounded number of time steps $\delta$
(for example, to prevent attacks from appearing too obvious). Attacks of this
kind take advantage of the settings in which reward needs to be computed based
on perceptual information. For example, the goal may be to learn a lane-
following policy, with rewards computed based on vehicle location relative to
lane markers inferred from camera and GPS/IMU data. A compromised scheduler
can delay the computation associated with a reward, but cannot directly modify
rewards (contrasting our attacks from prior research on reward poisoning [16,
35, 34, 36]). As a useful construct, we endow the attacker with a
$\delta$-sized disk $\mathcal{D}$ in which the past rewards are stored, and
the attacker can utilize $\mathcal{D}$ to replace the original reward used to
update the $Q$-function parameters at time $t$.

We consider two common attack goals: 1) _untargeted attacks_ which simply aim
to minimize the reward obtained by the learned policy, and 2) _targeted
attacks_ , which attempt to cause the learner to learn a policy that takes
particular target actions. To formalize these attack goals, let
$\mathbf{r}^{\alpha}=\\{r_{0}^{\alpha},\ldots,r_{T-1}^{\alpha}\\}$ be an
adversarial reward stream induced by our attack, resulting in the learned Q
function $\tilde{Q}_{T}(s,a;\mathbf{r}^{\alpha})$. On the other hand, let
$Q_{T}(s,a)$ be the Q function learned without adversarial reward
perturbations. Let $\tilde{\pi}(s;\mathbf{r}^{\alpha})$ be the policy induced
by $\tilde{Q}_{T}$, with $\tilde{\pi}(s,a;\mathbf{r}^{\alpha})$ a binary
indicator of which action is taken in which state. The goal of an untargeted
attack is

$\min_{\mathbf{r}^{\alpha}}\sum_{a}\tilde{\pi}(s,a;\mathbf{r}^{\alpha})Q_{T}(s,a).$
(1)

For a targeted attack, we define a _target policy_ as a set-value function
$f(s)$ which maps each state to a set of target actions, i.e., $f:S\rightarrow
2^{A}$, where $f(s)\subseteq A$ for each state $s\in S$. That is, the attacker
aims to cause the learned policy $\tilde{\pi}$ to take one of a target actions
$f(s)$, that is, $\tilde{\pi}(s;\mathbf{r}^{\alpha})\subseteq f(s)$. A natural
special case is when $f(s)$ is a singleton for each state $s$. This objective
can be equivalently expressed in terms of the learned Q function
$\tilde{Q}_{T}$ as the following condition in each state $s$:

$\underset{a\in
f(s)}{\max}\tilde{Q}_{T}(s,a;\mathbf{r}^{\alpha})>\underset{a\notin
f(s)}{\max}\tilde{Q}_{T}(s,a;\mathbf{r}^{\alpha}),\ \ \forall
s:f(s)\neq\pi(s).$ (2)

We assume that the current state $\sigma_{t}$ of the DQN algorithm is observed
by the adversary at each time $t$. This assumption amounts to the compromised
scheduler being able to read memory. This can be done either because the
adversary has gained access to kernel space, or through a side-channel attack
that recovers memory contents [25, 10, 19].

Our key observation is that whichever of the above goals the attacker chooses,
since learning itself can be modeled as an MDP, the attacker’s problem also
becomes an MDP in which the state is $o_{t}=(\sigma_{t},\mathcal{D}_{t})$,
where $\mathcal{D}_{t}$ is the state of the attacker disk (i.e., rewards
saved, and their current delay length). The reward in this MDP is defined as
above for both untargeted and targeted attacks. Let $O$ denote the set of
possible states in the attacker MDP. Next, we define two types of reward-delay
attacks, which then determine the action space.

Figure 1: Reward Delay Attack Model.

We consider two variants of the reward delay attack. The first is a general
_reward delay_ variant in which rewards can be swapped arbitrarily or dropped,
with the only constraint that if a reward is delayed, it is by at most
$\delta$ time steps. The attacker also has the option of waiting at time t. In
particular, at time $t$, the attacker can publish (e.g., by prioritizing the
scheduling of) a reward $r_{t}^{\alpha}\in\mathcal{D}_{t}$ selected from disk
at time $t$, $\mathcal{D}_{t}$; the implied set of attacker actions at time
$t$ is denoted by $A_{t}^{\alpha}=A^{\alpha}(o_{t})$ (since it depends on the
current state $o_{t}$). As a result, the learner receives the transition tuple
$(s_{t},a_{t},r_{t}^{\alpha},s_{t+1})$ in place of
$(s_{t},a_{t},r_{t},s_{t+1})$, where the current reward $r_{t}$ is added to
disk, obtaining the disk $\mathcal{D}_{t+1}$ for the next training step. We
present an illustrative visualization of the _reward delay_ attack model in
Figure 1.

A minimal level of security that a learner can easily implement is to securely
time stamp incoming sensor data. Our goal is to evaluate how much impact this
minimal level of protection has on the attack efficacy. To this end, we
introduce a second significantly more constrained attack variant that only
allows _reward shifting_ : rewards can only be shifted forward (effectively,
dropping some of these), but not arbitrarily swapped. In reward shifting
attacks, since the sequence must be preserved, any time the attacker selects a
reward $r_{t}^{\alpha}\in\mathcal{D}_{t}$ to publish, it must be the case that
the time stamp on this reward exceeds that of the reward published at time
$t-1$. Consequently, the disc $\mathcal{D}_{t+1}$ is updated with the actual
reward $r_{t}$, but all rewards in $\mathcal{D}_{t}$ with time stamp earlier
than $r_{t}^{\alpha}$ are also removed (effectively, dropped). Additionally,
the attacker has the option of waiting at time $t$, publishing a reward (or a
sequence of rewards) at a later time point from the disk, which are then
aligned in the corresponding temporal sequence with states and actions used
for DQN updates. We present an illustrative visualization of the _reward
shifting_ attack model in Figure 2. We denote the implied action space for the
attacker in reward shifting attacks by
$A_{t}^{\alpha,\text{shift}}=A^{\alpha,\text{shift}}(o_{t})$.

Figure 2: Reward Shifting Attack Model.

Next, we present our algorithmic approaches for implementing the attack
variants above.

##  4 Algorithmic Approaches for Reward-Delay Attacks

Figure 3: Learner’s and Attacker’s Q-network.

Recall that general reward-delay attacks can be represented by an attacker MDP
with state space $O$ and action sets $A^{\alpha}(o)$ for the general reward-
delay attacks, and $A^{\alpha,\text{shift}}(o)$ for reward shifting attacks,
where $o\in O$ is the current state. Since the state space is in general high-
dimensional, it is natural to apply deep Q-learning attacks to learn an
effective attack policy $\pi^{\alpha}(o)$. In Figure 3 we present both the
learner’s and attacker’s Q-network architecture (as the latter is partly
derived from the former). An important practical challenge, however, is that
delay reward signal so long considerably reduces efficacy of learning.
Consequently, our algorithmic approaches to the different types of attacks
involve designing effective proxy-reward signals that can be computed in each
time step $t$ of the learning process.

Since the reward shifting attack involves a considerably stronger constraint
on what the attacker can do (which we model by modifying what information can
be stored on disk $\mathcal{D}_{t}$ at time $t$ above), we further enhance our
ability to effectively learn an attack policy in two ways. First, we heavily
leverage the _wait_ option by delaying attack choice until the disk
$\mathcal{D}_{t}$ is full, in the sense that we can no longer wait without
having to drop one of the rewards in the disk (which would otherwise exceed
the delay time constraint $\delta$). We then significantly simplify the attack
strategy by selecting a _drop index_ $i$ (where $0\leq i\leq K$) and dropping
all the rewards in the disk $\mathcal{D}_{t}$ with index $\leq i$ from
consideration. Note that, $K<d$, where $d$ is the size of the attacker’s disk.
The residual temporal sequence of rewards on disk is then published (as
depicted in Figure 2) and used to train the learner’s Q-network, and finally
the attacker’s disk is emptied. This reduces the policy consideration space to
only the choice of an index of the reward to drop given a full disk.

At this point, the only remaining piece of the attack approach is the design
of the proxy reward function, which we turn to next. Armed with appropriate
proxy reward functions, we can apply any deep Q-network based algorithm for
learning an attack policy for any of the attacks discussed above.

###  4.1 Proxy Reward Design for Untargeted Attacks

For untargeted attacks, suppose that $Q_{t}$ is the _learner’s_ Q function in
iteration $t$. Since DQN updates are deterministic, the learner’s Q function
$Q_{t+1}$ can be precomputed for any reward $r_{t}^{\alpha}$ published by the
attacker. Let $s_{t}$ be the state observed by the learner at time $t$. We
propose the following proxy reward for the attacker which is used for the
_attacker’s_ DQN update:

$\begin{split}\tilde{R}^{\text{att}}_{t}=\sum_{a\in
A}-(\tilde{\pi}_{t+1}(s_{t},a)\cdot Q_{t}(s_{t},a)),\\\
\text{where}\>\>\tilde{\pi}_{t+1}(s_{t},a)=\frac{\exp(\tilde{Q}_{t+1}(s_{t},a))}{\sum_{a^{\prime}\in
A}\exp(\tilde{Q}_{t+1}(s_{t},a^{\prime}))}\end{split}$ (3)

where $Q_{t}(s_{t})$ is the _true_ Q function vector and
$\tilde{Q}_{t+1}(s_{t})$ is the _proxy_ Q function vector corresponding to all
the learner’s actions in step $t$ and $t+1$, respectively. Note that the
learner’s _true_ $Q_{t+1}(s_{t})$ is obtained by updating $Q_{t}(s_{t})$ using
randomly sampled batch data stored in the learner’s replay buffer, whereas,
the learner’s _proxy_ $\tilde{Q}_{t+1}(s_{t})$ is obtained by updating
$Q_{t}(s_{t})$ using the recent transition tuples published by the attacker.
The intuition for this proxy reward is that it accomplishes two things at
once: first, by minimizing correlation between successive Q functions, the
attacker minimizes the marginal impact of learning updates, thereby causing
learning to fail, and second, if the learner happens to obtain a good estimate
of the true Q function in iteration $t$, the quality of this function is
actively reduced in iteration $t+1$.

###  4.2 Proxy Reward Design for Targeted Attacks

The intuition for our proposed proxy reward function in the case of targeted
attacks is to maximize similarity between the target policy $f(s)$ and the
policy induced by the current Q function $Q_{t}$. However, since the policy
induced by $Q_{t}$ is not differential, we replace it with a stochastic policy
$\pi_{t}(s_{t})=\text{softmax}({Q_{t}(s_{t})})$, where

$\pi_{t}(s_{t},a)=\frac{\exp(Q_{t}(s_{t},a))}{\sum_{a^{\prime}\in
A}\exp(Q_{t}(s_{t},a^{\prime}))}.$

Further, we represent $f(s)$ as a vector $\hat{f}(s,a)$

$\hat{f}(s,a)\xleftarrow{}\left\\{\begin{array}[]{r@{\quad\quad}l}\text{1,}&\text{if}\
a\in f(s,a)\\\ \text{0,}&\text{o.w.}\end{array}\right.$ (4)

$\hat{f}(s)$ then denotes the binary vector corresponding to $f(s)$. We then
define the proxy reward for a targeted attack as follows:

$\displaystyle\tilde{R}^{\text{att}}_{t}=\mathrm{sign}\\{\mathcal{L}_{CE}(\pi_{t}(s_{t}),\hat{f}(s_{t}))-\mathcal{L}_{CE}(\tilde{\pi}_{t+1}(s_{t}),\hat{f}(s_{t}))\\},$
(5)

where $\mathcal{L}_{CE}$ is the cross-entropy loss. The reward function in (5)
suggests that, an attacker receives a positive reward, only if the performed
actions of the attacker (i.e. choosing $r^{\alpha}_{t}$ from the disk
$\mathcal{D}_{t}$ ) steers the learner’s updated proxy Q-value distribution
$\tilde{\pi}_{t+1}(s_{t})$ to be more aligned with the target Q-table
distribution corresponds to target policy $f(s)$ compared to the learner’s
previous true Q-value distribution $\pi_{t}(s_{t})$.

###  4.3 Rule Based Targeted Reward Delay Attack

In addition to a targeted reward delay attack strategy that requires a proxy
reward computation of the attacker as defined in equation (5), we also propose
a simple rule based strategy in order to feed the attacker a reward at a given
state as defined in equation (6). Note that the attacker’s disk configuration
at current time step (t) is represented as $\mathcal{D}_{t}$. According to
(6), the attacker feeds back a high reward to the learner, if the learner acts
in a way that is preferred by the attacker at any given target state.

$\tilde{R}^{\text{att}}_{t}\xleftarrow{}\left\\{\begin{array}[]{r@{\quad\quad}l}\text{\emph{Maximum}
Reward in $\mathcal{D}_{t}$,}&\text{if}\ \arg\max_{a}(Q_{t}(S_{t},a))\in
a^{T}\>\text{and}\>S_{t}\in S^{\prime}\\\ \text{\emph{Minimum} Reward in
$\mathcal{D}_{t}$,}&\text{if}\ \arg\max_{a}(Q_{t}(S_{t},a))\notin
a^{T}\>\text{and}\>S_{t}\in S^{\prime}\\\ \text{\emph{Random} Reward in
$\mathcal{D}_{t}$,}&\text{if}\ S_{t}\notin S^{\prime}\\\ \end{array}\right.$
(6)

##  5 Results

We evaluate the effectiveness of the proposed attack approaches on the Pong
and Breakout Atari-2600 environments in OpenAI Gym [7] on top of the Arcade
Learning Environment [4]. The states in those environments are high
dimensional RGB images with dimensions (210 * 160 * 3) and discrete actions
that control the agent to accomplish certain tasks. Specifically, we leverage
the _NoFrameskip-v4_ version for all our experiments, where the randomness
that influences the environment dynamics can be fully controlled by fixing the
random seed of the environment at the beginning of an episode. Please note
that we used a standard computing server with 3 GeForce GTX 1080 Ti GPUs each
of 12GB for all the experiments in our work. We choose the Double DQN
algorithm [31] with the Duelling style architecture [33] as the reference Q
network for both the learner and attacker agents. Note that our proposed
reward-delay attack and reward-shifting attack strategies can be easily
applied to other DQN based learning algorithms without requiring further
modifications. Unless noted otherwise, we set $\delta$ (i.e. the maximum
number of time steps a reward can be delayed) to be 8. In the case of a
_reward shifting_ attack, we choose the maximum value of “drop-index”($K$) to
be 4, and attacker’s maximum disk size (effectively, maximum wait time before
implementing the attack) to be 8. We also perform experiments to show the
impact of the reward delay attack for different choices of attack hyper-
parameters.

We compare our approaches to two baselines: _random attack_ and _fixed-delay
attack_. In the random attack, an attacker chooses to publish a reward
randomly from the disk at every time-step. In the fixed-delay attack, an
attacker delays the reward signal by $\delta$ time steps. In addition, in the
reward shifting attack setting, we use a _random reward Shift_ baseline, where
in place of an attacker agent, we randomly select a value for “drop-index”
(ranging from index 0 to $K$) to drop the reward. Apart from this step, the
random baseline attack operates exactly as the _reward shifting_ attack.

For untargeted attacks, our measure of effectiveness is the expected total
reward at test time. In the case of targeted attacks, we evaluate the _success
rate_ of the attacks, measured as follows:

$SuccessRate(SR)=\frac{\sum_{s\in S^{\prime}}\mathbb{I}[a_{t}\in
f(s)]}{\textit{No. of times agent visits target states}},$ (7)

where $\mathbb{I}[.]$ is an indicator function and $S^{\prime}$ is a set of
_target states_ in which the attacker has a non-trivial preference over which
action is played. Equation (7) yields the fraction of times the policy learned
by the targeted RL agent chooses an action in the attacker’s target set $f(s)$
in target states $s\in S^{\prime}$.

In order to generate target policies $f(s)$, and in particular which states
constitute target states $S^{\prime}$, we leverage an (approximately) optimal
Q-network (learned without attacks), denoted by $Q^{*}$, which gives us a way
to decide a subset of states that we target given the target action sets
preferred by the attacker; for the remaining states, $f(s)$ allows any action,
that is, the attacker is indifferent. Next, we define a state-independent set
of target actions $a^{T}$; these will be target actions for a subset of target
states, which we choose dynamically using the following rule:

$s_{t}\xleftarrow{}\left\\{\begin{array}[]{r@{\quad\quad}l}\text{Is a Target
State,}&\text{if}\ \arg\max_{a}(Q^{\textbf{*}}(S_{t},a))\notin a^{T}.\\\
\text{Not a Target State,}&\text{otherwise.}\end{array}\right.$ (8)

We describe our choice of the set $a^{T}$ in the concrete experiment domains
below. Note that at every time step in an episode, the learner interacts with
the environment, but instead of receiving the true reward from the
environment, the learner receives a poisoned reward published by the attacker.
We train the learner’s Q network with the modified reward sequence published
by the attacker and also update the learner’s Q network parameter following
the Double DQN update rule. In parallel, we train and update the attacker’s Q
network following the Double DQN update rule as described above (see also
Algorithms 1 and 2 in the Supplement). After the completion of each episode,
we evaluate the learner’s performance on test episodes. We report our results
(cumulative rewards, etc) obtained on these test episodes.

###  5.1 Untargeted Reward-Delay Attacks

We begin by evaluating the efficacy of the proposed approaches for
accomplishing the goals of untargeted attacks. Figure 4 presents the total
reward obtained during evaluation as a function of the number of training
episodes in Atari Pong and Breakout, respectively. We observe that essentially
all baselines perform nearly the same as each other and as our attack, with
reward nearly zero in the case of Atari Breakout and nearly -21 in the case of
Atari Pong. This clearly contrasts with normal training, which is highly
effective. Consequently, in the untargeted setting, even if we limit
considerably by how much rewards can be delayed, essentially any reshuffling
of rewards entirely prevents effective learning.

(a) Reward Comparison on Atari Pong.

(b) Reward Comparison on Atari Breakout.

Figure 4: Effect of Reward Delay Attack on Minimizing Reward.

We further investigate the efficacy of the untargeted reward delay attacks as
we change $\delta$, the maximum delay we can add to a reward (i.e., the
maximum we can shift reward back in time relative to the rest of DQN update
information), from 8 (the default value in experiments) to 16. As Figure 5
shows, we see an improvement in the attack efficacy as would be expected
intuitively; what is surprising, however, is that this improvement is
extremely slight, even though we doubled the amount of time the reward can be
delayed. Our results thus suggest that even a relatively short delay in the
reward signal can lead DQN learning to be entirely ineffective.

(a) Effect of $\delta$ on Untargeted Reward Delay Attack Strategy on Atari
Pong environment.

(b) Effect of $\delta$ on Untargeted Reward Delay Attack Strategy on Breakout
environment.

Figure 5: Effect of $\delta$.

We also examine the effectiveness of the untargeted reward delay attacks on a
pre-trained Q network. Figure 6 depicts the net reward obtained by the pre-
trained Q network during evaluation as a function of the number of training
episodes in Atari Pong and Breakout, respectively. Our experimental finding
indicates that, the reward decays exponentially as training progresses. We
also notice that even naive baseline untargeted attacks are as effective as
the untargeted reward delay attack in reducing the reward of the pretrained
network. Such observations shows the importance of reward synchrony at every
phase of deep reinforcement learning training.

(a) Effect of Untargeted Reward Delay Attack Strategy on a Pre-trained Q
network on Atari Pong environment.

(b) Effect of Untargeted Reward Delay Attack Strategy on a Pre-trained Q
network on Breakout environment.

Figure 6: Effect of Reward Delay Attack on a Pre-trained Q network.

Next, we evaluate the efficacy of our approach in the context of the far more
challenging targeted attacks.

###  5.2 Targeted Reward-Delay Attacks

In targeted attacks, we aim to achieve a particular target action (or one of a
set of actions) in a subset of _target states_ , with the attacker indifferent
about which action is taken in the remaining states. In both Pong and Breakout
environments, we chose _do not move_ as the target action for this evaluation.
Target states were defined using the condition in Equation (8).

In Figure 7(b), we compare the efficacy of our approach for targeted attacks
compared to our baseline approach. Here we can see that the proposed targeted
attack approaches are considerably more effective than the baseline, with
success rate significantly higher than the best baseline in both Pong and
Breakout. Interestingly, we can also see that while the proposed attack
improves in efficacy with the number of training episodes, the baselines
either have a constant success rate (essentially due entirely to chance), or
the success rate of these may even decrease (we can see a mild decrease in the
case of Pong, in particular). We also observe that the rule based reward delay
targeted attack strategy is highly effective in achieving the targeted attack
objective.

We further look into the effectiveness of targeted reward delay attacks as we
vary $\delta$, the maximum delay we can add to a reward. We present those
results in figure 8. We observe that the _success rate_ stays the same as we
change the $\delta$ in both Atari Pong and Breakout environments.

Next, we assess the effectiveness of reward-shifting attacks, which are more
constrained than reward delay attacks.

(a) Success Rate comparison on Atari Pong.

(b) Success Rate comparison on Atari Breakout.

Figure 7: Success Rate Comparison of Reward Delay Attack.

(a) Effect of $\delta$ on Targeted Reward Delay Attack Strategy on Atari Pong
environment.

(b) Effect of $\delta$ on Targeted Reward Delay Attack Strategy on Breakout
environment.

Figure 8: Effect of $\delta$.

###  5.3 Reward Shifting Attacks

We now turn to evaluating the effectiveness of a simple defense in which we
ensure that rewards cannot be shuffled out of sequence. To this end, we
evaluate the efficacy of the proposed reward shifting attacks, and compare
that to our observations of the efficacy of reward delay attacks above.

(a) Reward Comparison on Atari Pong.

(b) Reward Comparison on Atari Breakout.

Figure 9: Effect of Reward Shifting Attack on Minimizing Reward.

In Figure 9, we present the results of the _reward shifting_ attack in the
Pong and Breakout environments. First, we observe that both the baseline and
our untargeted reward shifting attacks are as effective as any of the attacks
without the sequence-preserving constraint. Moreover, the proposed untargeted
attack is now tangibly better than its baseline (random) counterpart, with the
gap increasing with the number of episodes. In addition, Figure 10 shows that
the net episodic reward of a pre-trained Q network drops exponentially when
trained with the untargeted reward shifting attack strategy. We also found
that even a random reward shifting attack is highly capable of reducing the
reward of a pre-trained Q network. Such findings indicate the adverse impact
of incorrect reward timing on deep reinforcement learning. So, not only the
_ordering of the reward sequence_ , but the _precise timings of the reward
sequence_ are also very important for efficient deep reinforcement learning.

(a) Effect of Untargeted Reward Shifting Attack on a Pre-trained Q network on
Pong environment.

(b) Effect of Untargeted Reward Shifting Attack on a Pre-trained Q network on
Breakout env.

Figure 10: Effect of Reward Shifting Attack on a Pre-trained Q network.

Nevertheless, the defense is clearly not effective in mitigating the
untargeted reward shifting attacks. In the Pong environment, our proposed
untargeted reward shifting attack yields reward near -20—that is almost what
was achieved without any mitigation at all, but a far cry from the result of
nominal training. Similarly, the reward after our attack in the Breakout
environment is still far below what is achievable without attack.

Finally, we investigate the potency of the reward shifting attack in the
context of the targeted attack. We report those results in Figure 11. We
observe that the targeted reward shifting attack actually yields a higher
success rate compared to the baseline random reward shifting attack in both
Atari Pong and Breakout environments, with the success rate gap increasing
with the number of episodes.

(a) Success Rate Comparison on Atari Pong.

(b) Success Rate Comparison on Atari Breakout.

Figure 11: Success Rate of Reward Shifting Attack.

##  6 Conclusions

We study the problem of reward delay attacks against reinforcement learning
agents. Our empirical findings suggest that it is possible to induce a sub-
optimal policy, or even a specific target policy by strategically reshuffling
the true reward sequence. Indeed, we find that even randomly shuffling rewards
within relatively short time intervals is already sufficient to cause learning
failure. This raises a potentially serious security threat to different
downstream applications that rely on RL. Moreover, we showed that reward
shifting attacks that assure that reward signals are not observed out of
order, also have a disastrous effect on DQN learning. Our finding shows that
current deep RL training strategy falls far short of ensuring adequate
robustness to these in many cases.

A natural subject for future work is to develop mitigation techniques that can
assure adequate synchrony in reinforcement learning when it is necessary.
Often, mitigations of this kind must involve hardware support that enables us
to assure synchrony of state and reward information. A natural second open
question is whether it is possible to avoid strong reliance on such hardware
support by developing reinforcement learning approaches that have weaker
synchrony requirements.

#### Acknowledgments

This research was supported in part by the National Science Foundation (grants
IIS-1905558, IIS-2214141, ECCS-2020289), Army Research Office (grant
W911NF1910241), and NVIDIA.

## References

  * ACM [2022] ACM: American center for mobility (2022), https://www.acmwillowrun.org/
  * Altschuler et al. [2019] Altschuler, J., Brunel, V.E., Malek, A.: Best arm identification for contaminated bandits.  J. Mach. Learn. Res. 20(91), 1–39 (2019)
  * Behzadan and Munir [2017] Behzadan, V., Munir, A.: Vulnerability of deep reinforcement learning to policy induction attacks.  In: International Conference on Machine Learning and Data Mining in Pattern Recognition. pp. 262–275. Springer (2017)
  * Bellemare et al. [2013] Bellemare, M.G., Naddaf, Y., Veness, J., Bowling, M.: The arcade learning environment: An evaluation platform for general agents.  Journal of Artificial Intelligence Research 47, 253–279 (2013)
  * Blaß et al. [2021a] Blaß, T., Casini, D., Bozhko, S., Brandenburg, B.: A ros 2 response-time analysis exploiting starvation freedom and execution-time variance.  In: Proceedings of the 42nd IEEE Real-Time Systems Symposium (RTSS). pp. 41––53 (2021a)
  * Blaß et al. [2021b] Blaß, T., Hamann, A., Lange, R., Ziegenbein, D., Brandenburg, B.: Automatic latency management for ros 2: Benefits, challenges, and open problems.  In: Proceedings of the 27th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS). pp. 264––277 (2021b)
  * Brockman et al. [2016] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym.  arXiv preprint arXiv:1606.01540 (2016)
  * Casini et al. [2019] Casini, D., Blaß, T., Lütkebohle, I., Brandenburg, B.: Response-time analysis of ros 2 processing chains under reservation-based scheduling.  In: Proceedings of the 31st Euromicro Conference on Real-Time Systems (ECRTS). pp. 6:1–6:23 (2019)
  * Choi et al. [2021] Choi, H., Xiang, Y., Kim, H.: Picas: New design of priority- driven chain-aware scheduling for ros2.  In: Proceedings of the 27th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS) (2021)
  * Chung et al. [2019] Chung, K., Li, X., Tang, P., Zhu, Z., Kalbarczyk, Z.T., Iyer, R.K., Kesavadas, T.: Smart malware that uses leaked control data of robotic applications: The case of Raven-II surgical robots.  In: 22nd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2019). pp. 337–351. USENIX Association, Chaoyang District, Beijing (Sep 2019), https://www.usenix.org/conference/raid2019/presentation/chung
  * DeMarinis et al. [2019] DeMarinis, N., Tellex, S., Kemerlis, V.P., Konidaris, G., Fonseca, R.: Scanning the internet for ros: A view of security in robotics research.  In: 2019 International Conference on Robotics and Automation (ICRA). pp. 8514–8521. IEEE (2019)
  * Dieber et al. [2016] Dieber, B., Kacianka, S., Rass, S., Schartner, P.: Application-level security for ros-based applications.  In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 4477–4482. IEEE (2016)
  * Dong et al. [2019] Dong, Y., Zhong, Y., Yu, W., Zhu, M., Lu, P., Fang, Y., Hong, J., Peng, H.: Mcity data collection for automated vehicles study.  arXiv preprint arXiv:1912.06258 (2019)
  * Garnett et al. [2019] Garnett, N., Cohen, R., Pe’er, T., Lahav, R., Levi, D.: 3d-lanenet: end-to-end 3d multiple lane detection.  In: IEEE/CVF International Conference on Computer Vision. pp. 2921–2930 (2019)
  * Huang et al. [2017] Huang, S., Papernot, N., Goodfellow, I., Duan, Y., Abbeel, P.: Adversarial attacks on neural network policies.  arXiv preprint arXiv:1702.02284 (2017)
  * Huang and Zhu [2019] Huang, Y., Zhu, Q.: Deceptive reinforcement learning under adversarial manipulations on cost signals.  In: International Conference on Decision and Game Theory for Security. pp. 217–237. Springer (2019)
  * Jun et al. [2018] Jun, K.S., Li, L., Ma, Y., Zhu, J.: Adversarial attacks on stochastic bandits.  Advances in Neural Information Processing Systems 31 (2018)
  * Kiran et al. [2021] Kiran, B.R., Sobh, I., Talpaert, V., Mannion, P., Al Sallab, A.A., Yogamani, S., Pérez, P.: Deep reinforcement learning for autonomous driving: A survey.  IEEE Transactions on Intelligent Transportation Systems (2021)
  * Kocher et al. [2019] Kocher, P., Horn, J., Fogh, A., Genkin, D., Gruss, D., Haas, W., Hamburg, M., Lipp, M., Mangard, S., Prescher, T., et al.: Spectre attacks: Exploiting speculative execution.  In: 2019 IEEE Symposium on Security and Privacy (SP). pp. 1–19. IEEE (2019)
  * Kos and Song [2017] Kos, J., Song, D.: Delving into adversarial attacks on deep policies.  arXiv preprint arXiv:1705.06452 (2017)
  * Li et al. [2021] Li, A., Wang, J., Zhang, N.: Chronos: Timing interference as a new attack vector on autonomous cyber-physical systems.  In: Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security. pp. 2426–2428 (2021)
  * Lin et al. [2017] Lin, Y.C., Hong, Z.W., Liao, Y.H., Shih, M.L., Liu, M.Y., Sun, M.: Tactics of adversarial attack on deep reinforcement learning agents.  arXiv preprint arXiv:1703.06748 (2017)
  * Liu and Shroff [2019] Liu, F., Shroff, N.: Data poisoning attacks on stochastic bandits.  In: International Conference on Machine Learning. pp. 4042–4050. PMLR (2019)
  * Liu et al. [2020] Liu, S., See, K.C., Ngiam, K.Y., Celi, L.A., Sun, X., Feng, M., et al.: Reinforcement learning for clinical decision support in critical care: comprehensive review.  Journal of medical Internet research 22(7), e18477 (2020)
  * Luo et al. [2020] Luo, M., Myers, A.C., Suh, G.E.: Stealthy tracking of autonomous vehicles with cache side channels.  In: 29th USENIX Security Symposium (USENIX Security 20). pp. 859–876 (2020)
  * Ma et al. [2018] Ma, Y., Jun, K.S., Li, L., Zhu, X.: Data poisoning attacks in contextual bandits.  In: International Conference on Decision and Game Theory for Security. pp. 186–204. Springer (2018)
  * Mahfouzi et al. [2019] Mahfouzi, R., Aminifar, A., Samii, S., Payer, M., Eles, P., Peng, Z.: Butterfly attack: Adversarial manipulation of temporal properties of cyber-physical systems.  In: 2019 IEEE Real-Time Systems Symposium (RTSS). pp. 93–106. IEEE (2019)
  * Silver et al. [2018] Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al.: A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science 362(6419), 1140–1144 (2018)
  * STII [2022] STII: Illinois autonomous and connected track (2022), https://ict.illinois.edu/i-act
  * Tang et al. [2020] Tang, Y., Zhiwei, F., Guan, N., Jiang, X., Lv, M., Deng, Q., Yi, W.: Response time analysis and priority assignment of processing chains on ros2 executors.  In: Proceedings of the 41st IEEE Real-Time Systems Symposium (RTSS) (2020)
  * Van Hasselt et al. [2016] Van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning.  In: Proceedings of the AAAI conference on artificial intelligence. vol. 30 (2016)
  * Wang et al. [2018] Wang, Z., Ren, W., Qiu, Q.: Lanenet: Real-time lane detection networks for autonomous driving.  arXiv preprint arXiv:1807.01726 (2018)
  * Wang et al. [2016] Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., Freitas, N.: Dueling network architectures for deep reinforcement learning.  In: International conference on machine learning. pp. 1995–2003. PMLR (2016)
  * Zhang and Parkes [2008] Zhang, H., Parkes, D.C.: Value-based policy teaching with active indirect elicitation.  In: AAAI. vol. 8, pp. 208–214 (2008)
  * Zhang et al. [2009] Zhang, H., Parkes, D.C., Chen, Y.: Policy teaching through reward function learning.  In: Proceedings of the 10th ACM conference on Electronic commerce. pp. 295–304 (2009)
  * Zhang et al. [2020] Zhang, X., Ma, Y., Singla, A., Zhu, X.: Adaptive reward-poisoning attacks against reinforcement learning.  In: International Conference on Machine Learning. pp. 11225–11234. PMLR (2020)

##  7 Appendix

In this section, we present the end-to-end algorithmic approach of reward
delay attack strategy and reward shifting attack strategy in algorithm 1 and 2
respectively.

Algorithm 1 Reward Delay Attack Strategy

1:$Q_{t}(s_{t},a_{t},\theta)$ \- learner’s ($\mathcal{A}$) current Q-network
parameterized by $\theta$; An attacker agent $\mathcal{A}^{\prime}$ with
corresponding MDP
$\mathcal{M}^{\prime}=(\Xi^{\prime},\rho^{\prime},\tau^{\prime})$,
Q-network($Q^{\prime}$) parameterized by $\theta^{\prime}$, disk
$\mathcal{D}_{0}$ to store the rewards and attack constraint parameters
$\\{\delta,d\\}$; Action set of the $\mathcal{A}^{\prime}$, i.e.
A$\in${1,2,..,d} where $d$ is the disk size; T - No. of time-steps in an
episode; $\mathcal{B}$, $\mathcal{B}^{\prime}$ \- Replay buffer of
$\mathcal{A}$ and $\mathcal{A}^{\prime}$;

2:for t=0, 1, …, T do

3: $\mathcal{A}$ interacts with the environment by taking an action $a_{t}$
given a state $s_{t}$. Environment transits according to
$P(s_{t+1}|s_{t},a_{t})$, and feeds back a reward signal $r_{t}$ and a next
state $s_{t+1}$.

4: $\mathcal{A}^{\prime}$ extracts ($s_{t},a_{t},r_{t},s_{t+1}$) and push
$r_{t}$ into $\mathcal{D}_{t}$.

5: $\mathcal{A}^{\prime}$ leverages the following information to act:
$\emph{info}_{t}$ = $[s_{t},a_{t},r_{t},\mathcal{D}_{t}]$ and learner’s
$Q_{t}$.

6: $\mathcal{A}^{\prime}$ takes action $a^{\prime}_{t}$ according to
$\epsilon$-greedy behavior policy
$\emph{i}\xleftarrow{}\left\\{\begin{aligned} argmax_{a\in
A}\>\>Q^{\prime}_{t}(\emph{info}_{t},a,\theta^{\prime}),\ \ \ \ \ \ \
\text{If}\ 1-\epsilon\\\ \text{Randomly select from }A,\ \ \ \ \ \ \
\text{else}\end{aligned}\right\\}$

7: $r_{t}$ (the true reward) is exchanged with $r^{\mathcal{D}}_{i}$(The
reward stored at the i’th index in the disk $\mathcal{D}_{t}$). Accordingly,
t’th transitional tuple data is updated as
($s_{t},a_{t},r^{\mathcal{D}}_{i},s_{t+1}$) and stored into $\mathcal{B}$.

8: Disk $\mathcal{D}_{t}$ is updated by removing $r^{\mathcal{D}}_{i}$ from
the disk. We also _drop_ the reward from $\mathcal{D}_{t}$ whose stored-time
(i.e. delay duration) exceeds $\delta$.

9: $\mathcal{A}$ performs Q-learning given the poisoned mini-batch transition
data sampled from $\mathcal{B}$, where t’th transition tuple of the mini-batch
is represented as $(s_{t},a_{t},r^{\mathcal{D}}_{i},s_{t+1})$:
$y_{j}\xleftarrow{}\left\\{\begin{aligned} r^{\mathcal{D}}_{i},\ \ \ \ \
\text{If}\ s_{t+1}\ \text{is a Terminal State}\\\
r^{\mathcal{D}}_{i}+max_{a^{\prime}}(Q_{t}(s_{t+1},a^{\prime},\theta)),\ \ \ \
\text{else}\end{aligned}\right.$ Update the learner’s ($\mathcal{A})$
Q-network parameters from $Q_{t}$ to $Q_{t+1}$ by minimizing the loss
$\mathcal{L}(\theta)=(y_{j}-Q_{t}(s_{t},a_{t},\theta))^{2}$ following DQN
update rule.

10: $\mathcal{A}^{\prime}$ computes reward ${r}^{\textbf{attacker}}_{t}$
following the methods as described in section 4 and stores the transition
tuple
$(\emph{info}_{t},a^{\prime}_{t},r^{\textbf{attacker}}_{t},\emph{info}_{t+1})$
into its own replay buffer $\mathcal{B}^{\prime}$.

11: $\mathcal{A}^{\prime}$ performs Q-learning given the attacker’s mini-batch
transition data sampled from $\mathcal{B}^{\prime}$, where t’th transition
tuple is denoted as
$(\emph{info}_{t},a^{\prime}_{t},r^{\textbf{attacker}}_{t},\emph{info}_{t+1})$:
$y_{j}\xleftarrow{}\left\\{\begin{aligned} {r}^{\textbf{attacker}}_{t},\ \ \ \
\ \text{If}\ \emph{info}_{t+1}\ \text{is a Terminal State}\\\
{r}^{\textbf{attacker}}_{t}+max_{a^{\prime}\in
A}(Q^{\prime}_{t}(\emph{info}_{t+1},a^{\prime},\theta^{\prime})),\ \ \ \
\text{else}\end{aligned}\right.$

12: $\mathcal{A}^{\prime}$ also updates it’s Q-network parameter
$\theta^{\prime}$ by minimizing
$\mathcal{L}(\theta)=(y_{j}-Q^{\prime}_{t}(\emph{info}_{t},a^{\prime}_{t},\theta^{\prime}))^{2}$.

13:end for

14:return $\mathcal{A}$ and it’s corresponding Q-network parameter $\theta$;

Algorithm 2 Reward Shifting Attack Strategy

1:$Q_{t}(s_{t},a_{t},\theta)$ \- learner’s ($\mathcal{A}$) current Q-network
parameterized by $\theta$; An attacker ($\mathcal{A}^{\prime}$) with
corresponding MDP
$\mathcal{M}^{\prime}=(\Xi^{\prime},\rho^{\prime},\tau^{\prime})$,
Q-network($Q^{\prime}$) parameterized by $\theta^{\prime}$, disk
$\mathcal{D}_{0}$ to store the rewards and attack constraint parameters
$\\{l,d\\}$; Action set of the $\mathcal{A}^{\prime}$, i.e. A$\in${1,2,..,K};
$\mathcal{B},\mathcal{B}^{\prime}$ \- are Learner’s and attacker’s Replay
Buffer respectively; T - The no. of time-steps in an episode.

2:for t=0, 1, …, T do

3: $\mathcal{A}$ interacts with the environment which results a transition
tuple ($s_{t},a_{t},r_{t},s_{t+1}$).

4: $\mathcal{A}^{\prime}$ Extract ($s_{t},a_{t},r_{t},s_{t+1}$) and push
reward $r_{t}$ into $\mathcal{D}_{t}$.

5: $\mathcal{A}^{\prime}$ leverages the following information to act when
$\mathcal{D}_{t}$ is full: $\emph{info}_{t}$ =
$[s_{t},a_{t},r_{t},\mathcal{D}_{t}]$ and $Q_{t}$.

6: $\mathcal{A}^{\prime}$ takes action $a^{\prime}_{t}$ according to
$\epsilon$-greedy behavior policy when $\mathcal{D}_{t}$ is full.
$\emph{i}\xleftarrow{}\left\\{\begin{aligned} argmax_{a\in
A}\>\>Q^{\prime}_{t}(\emph{info}_{t},a,\theta^{\prime}),\ \ \ \ \ \ \
\text{If}\ 1-\epsilon\\\ \text{Randomly select from }A,\ \ \ \ \ \ \
\text{else}\end{aligned}\right\\}$

7: $\mathcal{A}^{\prime}$ publishes an updated reward sequence with the
following modifications: Firstly, the rewards stored from index $0$ to $i$ are
dropped, Secondly, the reward sequence ranging from index $(i+1)$ to $d$ is
published in sequence. We denote the attacker’s published reward sequence as
$\\{r^{\mathcal{D}}_{1},r^{\mathcal{D}}_{2},..,r^{\mathcal{D}}_{t},..\\}$.

8: $\mathcal{A}$ performs Q-learning given the poisoned mini-batch transition
data sampled from $\mathcal{B}$, where t’th transition tuple of the mini-batch
is represented as $(s_{t},a_{t},r^{\mathcal{D}}_{t},s_{t+1})$:
$y_{j}\xleftarrow{}\left\\{\begin{aligned} r^{\mathcal{D}}_{t},\ \ \ \ \
\text{If}\ s_{t+1}\ \text{is a Terminal State}\\\
r^{\mathcal{D}}_{t}+max_{a^{\prime}}(Q_{t}(s_{t+1},a^{\prime},\theta)),\ \ \ \
\text{else}\end{aligned}\right.$ Update the learner’s ($\mathcal{A})$
Q-network parameters from $Q_{t}$ to $Q_{t+1}$ following DQN update rule.

9: $\mathcal{A}^{\prime}$ computes reward ${r}^{\textbf{attacker}}_{t}$
following the methods as described in section 4 and stores the transition
tuple
$(\emph{info}_{t},a^{\prime}_{t},r^{\textbf{attacker}}_{t},\emph{info}_{t+1})$
into its own replay buffer $\mathcal{B}^{\prime}$.

10: $\mathcal{A}^{\prime}$ performs Q-learning given the attacker’s mini-batch
transition data sampled from $\mathcal{B}^{\prime}$, where t’th transition
tuple is denoted as
$(\emph{info}_{t},a^{\prime}_{t},r^{\textbf{attacker}}_{t},\emph{info}_{t+1})$:
$y_{j}\xleftarrow{}\left\\{\begin{aligned} {r}^{\textbf{attacker}}_{t},\ \ \ \
\ \text{If}\ \emph{info}_{t+1}\ \text{is a Terminal State}\\\
{r}^{\textbf{attacker}}_{t}+max_{a^{\prime}\in
A}(Q^{\prime}_{t}(\emph{info}_{t+1},a^{\prime},\theta^{\prime})),\ \ \ \
\text{else}\end{aligned}\right.$

11: $\mathcal{A}^{\prime}$ also updates it’s Q-network parameter
$\theta^{\prime}$ by minimizing
$\mathcal{L}(\theta)=(y_{j}-Q^{\prime}_{t}(\emph{info}_{t},a^{\prime}_{t},\theta^{\prime}))^{2}$.

12:end for

13:return $\mathcal{A}$ and it’s corresponding Q-network parameter $\theta$;