# ReRoGCRL: Representation-based Robustness in Goal-Conditioned Reinforcement
Learning

Xiangyu Yin1, Sihao Wu1, Jiaxu Liu1, Meng Fang1, Xingyu Zhao2,
Xiaowei Huang1, Wenjie Ruan1111Corresponding Author

###### Abstract

While Goal-Conditioned Reinforcement Learning (GCRL) has gained attention, its
algorithmic robustness against adversarial perturbations remains unexplored.
The attacks and robust representation training methods that are designed for
traditional RL become less effective when applied to GCRL. To address this
challenge, we first propose the Semi-Contrastive Representation attack, a
novel approach inspired by the adversarial contrastive attack. Unlike existing
attacks in RL, it only necessitates information from the policy function and
can be seamlessly implemented during deployment. Then, to mitigate the
vulnerability of existing GCRL algorithms, we introduce Adversarial
Representation Tactics, which combines Semi-Contrastive Adversarial
Augmentation with Sensitivity-Aware Regularizer to improve the adversarial
robustness of the underlying RL agent against various types of perturbations.
Extensive experiments validate the superior performance of our attack and
defence methods across multiple state-of-the-art GCRL algorithms. Our tool
ReRoGCRL is available at https://github.com/TrustAI/ReRoGCRL.

## Introduction

Goal-Conditioned Reinforcement Learning (GCRL) trains an agent to learn skills
in the form of reaching distinct goals. Unlike conventional RL, GCRL
necessitates the agent to make decisions that are aligned with goals. This
attribute allows agents to learn and accomplish a variety of tasks with shared
knowledge, better generalization, and improved exploration capabilities.
Furthermore, it is binary-rewarded, which is easier to implement compared to
hand-crafted complex reward functions. Recently, there has been a significant
surge in research related to GCRL. Exemplary works include methods based on
techniques such as hindsight experience replay (Andrychowicz et al. 2018; Fang
et al. 2018, 2019), imitation learning (Ghosh et al. 2020; Yang et al. 2021b),
or offline learning (Chebotar et al. 2021; Ma et al. 2022; Mezghani et al.
2023).

Generally, these studies tend to assume that the sensing and perception
systems of agents are devoid of uncertainties. However, this presumption is
hardly applicable in real-world scenarios because there are known gaps between
simulated and real-world environments, such as measurement errors, motor
noise, etc. The observations made by agents encompass unavoidable disturbances
that originate from unforeseeable stochastic noises or errors in sensing
(Huang et al. 2020; Zhang, Ruan, and Xu 2023; Wang et al. 2022; Huang, Jin,
and Ruan 2023). For example, due to the existence of adversarial attacks in RL
agents (Huang et al. 2017; Weng et al. 2019; Bai, Guan, and Wang 2019; Mu et
al. 2023a, b), even minor perturbations can lead to unsafe outcomes in safety-
critical GCRL control strategies (Gleave et al. 2019; Sun et al. 2020).

Recently, numerous robust techniques have been implemented in traditional RL,
which can be broadly classified into two categories: i) Adversarial Training
(AT), and ii) Robust Representation Training. Particularly, AT in RL trains
agents with adversarial states or actions (Pinto et al. 2017; Kos and Song
2017; Zhang et al. 2020b), and then enhances their adversarial robustness.
However, AT-trained RL agents cannot be utilized across downstream tasks,
which limits their transferability. Compared to AT, robust representation
training can deliver low-dimentional, collapse-resistant, and perturbation-
robust representations of observations (Gelada et al. 2019; Zhang et al.
2020a; Zang, Li, and Wang 2022), which is the focus of recent studies.
Specifically, it learns representations that capture only task-relevant
information based on the bisimulation metric of states (Ferns, Panangaden, and
Precup 2011). Technically, this is achieved by reducing the behavioural
difference between representations of similar observation pairs in the latent
space.

Although there are already many articles enhancing the robustness of RL, few
works (He and Lv 2023) consider the robustness of GCRL against adversarial
perturbations. Different from the vanilla one, GCRL employs reward functions
characterized by sequences of unshaped binary signals. For example,
(Andrychowicz et al. 2018) allocates the reward by determining whether the
distance between the achieved goal $\hat{g}$ and the desired goal $g$ is less
than a threshold $\epsilon$:
$\mathds{1}(\left\|\hat{g}-g\right\|\leq\epsilon)$. This makes the reward
sequences in GCRL notably sparser than those in traditional RL. The sparsity
of rewards in GCRL leads to inaccurate estimations of both Q-values and
actions. This presents a significant challenge for directly implementing
conventional RL attacks, particularly those that rely on pseudo labels such as
Q-values or action values.Consequently, this underscores the necessity to
develop new attack methods tailored for GCRL to evaluate its robustness.
Inspired by adversarial contrastive learning (Kim, Tack, and Hwang 2020; Jiang
et al. 2020; Ho and Nvasconcelos 2020), we propose the Semi-Contrastive
Representation (SCR) attack, with the aim of maximizing the distance between
the representations of original states and their corresponding perturbed
versions. This approach effectively bypasses the need for approximated labels.
Notably, this method can operate independently of the critic function and is
readily deployable.

Contrary to conventional RL, where state rewards approach 1.0 as the agent
nears the goal, in GCRL, rewards consistently remain at 0.0 until the goal is
reached. This distinction is highlighted in Fig. 1: for a given state $s$, the
rewards of subsequent states in Conventional RL (a) significantly differ from
those in Conventional RL (b). However, such a disparity in rewards is not
present in GCRL, where they invariably stay at 0.0. Consequently, bisimulation
metric-based representation training methods cannot capture well the
differences between state-goal tuples in GCRL. To deal with this, we propose a
universal defensive framework named Adversarial Representation Tactics (ARTs).
Firstly, ARTs enhance the robustness of vanilla GCRL algorithms with Semi-
Contrastive Adversarial Augmentation (SCAA). Secondly, ARTs mitigate the
performance degradation associated with bisimulation metric-based robust
representation techniques using the Sensitivity-Aware Regularizer (SAR).

Figure 1: Trajectories of the agent at state $s$ approaching blue and orange
goals in conventional RL and GCRL, where the designated goals vary with
different initialization. Rewards are indicated in each block along the
trajectories.

In summary, our contributions are summarized as follows:

  * •

We introduce a novel Semi-Contrastive Representation (SCR) attack. It can
operate without the critic function and is ready for direct deployment.

  * •

We propose a mixed defensive strategy, termed Adversarial Representation
Tactics (ARTs), to dynamically enhance adversarial robustness tailored to
specific GCRL algorithms.

  * •

Extensive experiments validate that our proposed attack method and defence
techniques outperform state-of-the-art algorithms in GCRL by a large margin.

## Related Work

### Goal-Conditioned Reinforcement Learning

Several methods have tackled GCRL. Hindsight experience replay (Andrychowicz
et al. 2018) is often employed to enhance the effectiveness of policies by
relabeling the goal. Goal-conditioned supervised learning approaches (Ghosh et
al. 2020; Yang et al. 2021b) solve this problem by iteratively relabeling and
imitating self-generated experiences. For long-horizon tasks, hierarchical RL
(Chane-Sane, Schmid, and Laptev 2021) learns high-level policies to
recursively estimate a sequence of intermediate subgoals. Moreover, model-
based methods (Charlesworth and Montana 2020; Yang et al. 2021a), self-
supervised learning (Mezghani et al. 2022; Eysenbach et al. 2022), and
planning-based methods (Eysenbach, Salakhutdinov, and Levine 2019; Nasiriany
et al. 2019) are also used to solve GCRL problems. Unlike these prior methods,
our paper considers the representation training and robustness performance on
GCRL.

### Adversarial Contrastive Attack

Adversarial contrastive attacks have gained significant attention in recent
studies. Particularly, (Kim, Tack, and Hwang 2020) proposes the instance-wise
adversarial attack for unlabeled data, which makes the model confuse the
instance-level identities of the perturbed data samples. (Jiang et al. 2020)
directly exploits the Normalized Temperature-scaled Cross Entropy loss to
generate PGD attacks. (Ho and Nvasconcelos 2020) introduces a new family of
adversarial examples by backpropagating the gradients of the contrastive loss
to the network input. While the above methodologies employ varied methods to
generate adversaries, they consistently rely on the cosine similarity among
the original sample, positive, and negative samples. In this paper, however,
we devise a novel contrastive attack method, which is more suitable in GCRL
and can be extended to other RL settings.

### Bisimulation Metric

Bisimulation metrics offer a framework to gauge the similarity between states
within MDPs. The traditional bisimulation metric, as defined by (Ferns,
Panangaden, and Precup 2012), considers two states to be close if their
immediate rewards and transition dynamics are similar (Larsen and Skou 1989;
Givan, Dean, and Greig 2003). This self-referential concept has been
mathematically formalized using the Kantorovich distance, leading to a unique
fixed-point definition. However, the conventional approach has been criticized
for resulting in pessimistic outcomes, as it requires consideration of all
actions, including those that may be suboptimal. To address this challenge,
(Castro et al. 2022) introduced the strategy of Matching Under Independent
Couplings (MICo). This approach focuses solely on actions induced by a
specific policy $\pi$, offering a more optimistic and tailored measure of
state equivalence. Whereas, in this work, we focus on their extended version,
Simple State Representation (SimSR).

## Background

In this section, we first present the fundamental concepts of goal-conditioned
MDPs and delve into their extensions in adversarial state-goal scenarios.
Then, we outline the architecture of our training backbone and describe SimSR,
the cutting-edge robust representation training method.

### Goal-Conditioned MDPs

In this section, we model our task as a goal-conditioned MDP. This can be
formalized as a tuple
$(\mathcal{S},\mathcal{G},\mathcal{Z},\mathcal{A},r,\gamma,\mathcal{T})$.
Here, $\mathcal{S}$ represents the state space, $\mathcal{G}$ the goal space,
$\mathcal{Z}$ the latent representation space, and $\mathcal{A}$ the action
space. The reward function is denoted by $r$, $\gamma$ stands for the discount
factor, and $\mathcal{T}$ signifies the state-goal transition probability
function.

We assume that the policy function $\pi$ can be approximated using a
combination of an encoder, $\psi(\cdot)$, and an actor-network,
$\varphi(\cdot)$. The critic function is represented by $\varrho(\cdot)$.
Specifically, for any given state $s\in\mathcal{S}$ and goal
$g\in\mathcal{G}$, the encoder $\psi(\cdot)$ transforms the concatenated tuple
$\left<s,g\right>$ into the representation space $\mathcal{Z}$. The resulting
feature, $z_{\left<s,g\right>}$, is expressed as $\psi(\left<s,g\right>)$. The
actor network, $\varphi(\cdot)$, maps this feature, $z_{\left<s,g\right>}$, to
a specific action $a_{\left<s,g\right>}\in\mathcal{A}$. The action
distribution is defined as
$a_{\left<s,g\right>}\sim\pi(\cdot|\left<s,g\right>)$. Based on the chosen
action $a_{\left<s,g\right>}$, the agent’s subsequent state, $s^{\prime}$, can
be sampled using
$s^{\prime}\sim\mathcal{T}(\cdot|\left<s,g\right>,a_{\left<s,g\right>})$ and
succinctly represented as $\mathcal{T}^{\pi}_{\left<s,g\right>}$.

In contrast to traditional RL algorithms, GCRL only provides rewards when the
agent successfully achieves a predefined goal. Specifically, we define the
reward for a state-goal tuple $\left<s,g\right>$ as
$r_{\left<s,g\right>}=\mathds{1}\left(\mathcal{D}(s,g)\leq\eta\right)$. Here,
$\eta$ is a predefined threshold, $\mathcal{D}$ acts as a distance metric,
determining if $s$ and $g$ are sufficiently close. We utilize
$\ell_{\infty}$-norm in this paper. Given an initial state $s_{0}$ and a goal
$g$, our primary objective is to optimize the expected cumulative rewards
across joint distributions following:

$\displaystyle
J_{\pi}(s_{0},g)=\mathbb{E}_{\begin{subarray}{c}a_{t}\sim\pi(\cdot|\left<s_{t},g\right>),\\\
s_{t+1}\sim\mathcal{T}^{\pi}_{\left<s_{t},g\right>}\end{subarray}}\left[\sum_{t=0}^{T-1}\gamma^{t}r_{\left<s_{t},g\right>}\right].$
(1)

In GCRL, the reward function yields only two possible outcomes: 0 or 1, which
results in a notably sparser reward sequence
$(r_{\left<s_{0},g\right>},\cdots,r_{\left<s_{T-1},g\right>})$ over a $T$-step
trajectory compared to other RL algorithms. While these binary rewards may not
offer granular insights into the agent-environment interactions, they do guide
the agent in a strategy where it initially distances itself from the goal
before converging towards it. To derive the optimal policy $\pi^{*}$, we
refine Eq. (1) using the Q-value at the initial state $s_{0}$, then
$Q_{\pi}(\left<s_{0},g\right>,a_{0})$ can be estimated through the Bellman
equations:

$r_{\left<s_{0},g\right>}+\mathbb{E}_{s_{1}\sim\mathcal{T}^{\pi}_{\left<s_{0},g\right>}}\max_{a_{1}\sim\pi(\cdot|\left<s_{1},g\right>)}Q_{\pi}(\left<s_{1},g\right>,a_{1}).$
(2)

######  Definition 1 ((Zhang et al. 2020b))

The adversarial version of a state $s$ can be defined as
$\mathcal{V}(s;\theta)$. It is considered stationary, deterministic, and
Markovian if its behaviour is solely determined by $s$ and the policy network,
which is parameterized by $\theta$. For simplicity, we denote
$\mathcal{V}(s;\theta)$ as $\mathcal{V}(s)$.

### State-Goal Adversarial Observations

Building on Def. 1, we consider states and goals separately and individually
construct the set of adversarial states $\mathcal{B}_{p}^{\epsilon}(s)$ and
goals $\mathcal{B}_{p}^{\epsilon}(g)$ using the State-Adversarial MDP (SA-MDP)
framework:

$\displaystyle\begin{split}&\mathcal{B}_{p}^{\epsilon}(s)\coloneqq\\{\mathcal{V}(s)\mid\left\|\mathcal{V}(s)-s\right\|_{p}\leq\epsilon_{s}\\},\\\
&\mathcal{B}_{p}^{\epsilon}(g)\coloneqq\\{\mathcal{V}(g)\mid\left\|\mathcal{V}(g)-g\right\|_{p}\leq\epsilon_{g}\\},\end{split}$
(3)

where $\epsilon_{s}$ and $\epsilon_{g}$ are predefined thresholds of
adversarial perturbations on states and goals respectively.
$\mathcal{V}(\cdot)$ acts as a stationary and deterministic mapping, which
transforms a clean input into its adversarial counterpart. Note that the
adversary introduces perturbations solely to the state-goal observations.
Consequently, even though the action is determined as
$a\sim\pi(\cdot|\left<\mathcal{V}(s),\mathcal{V}(g)\right>)$, transitions of
the environment are still governed by the original state-goal pair
$\left<s,g\right>$, instead of its perturbed counterpart
$\left<\mathcal{V}(s),\mathcal{V}(g)\right>$. Due to uncertainties in the
state-goal estimation, there is a potential for generating actions that are
not optimal.

### Neural Network-based Backbone

In this study, we utilize Multi-Layer Perceptrons (MLPs) of varying depths as
the foundational architectures for both the policy function, represented as
$\varphi(\psi(\cdot))$, and the critic function, $\varrho(\cdot)$.
Specifically, for an encoder $\psi(\cdot)$ comprising $L$ layers, its output,
$o_{L}$, can be articulated as
$o_{L}=\mathbf{W}_{L}\phi\left(\mathbf{W}_{L-1}\cdots\phi(\mathbf{W}_{1}o_{0})\right)$,
with the stipulation that $L\geq 2$. Here, $o_{0}$ refers to the input vector
in its flattened form, $\phi(\cdot)$ represents the ReLU-based activation
function, and $\mathbf{W}_{i}$ is the weight matrix associated with the $i$-th
layer. To be simplified, we denote the parameters and the depths of
$\psi(\cdot)$, $\varphi(\cdot)$, and $\varrho(\cdot)$ as $\theta_{\psi}$,
$\theta_{\varphi}$, $\theta_{\varrho}$, and $L_{\psi}$, $L_{\varphi}$,
$L_{\varrho}$, respectively. For clarity, we omit the bias vector present at
each layer.

### Simple State Representation (SimSR)

In this paper, SimSR functions as a metric within the representation space,
enhancing our base methods in GCRL. Unlike the bisimulation metric (Ferns,
Panangaden, and Precup 2012) or MICo (Castro et al. 2022), which employ the
Wasserstein or diffuse metric to gauge the distance between two distributions,
SimSR adopts a non-diffuse metric, offering a more relaxed approach. Take
input tuples $\left<s_{i},g_{1}\right>$ and $\left<s_{j},g_{2}\right>$ from
Fig. 1 for instance, the measurement $\mathcal{M}(\cdot,\cdot)$ between their
representations is constructed using the cosine distance (a $\ell_{2}$
normalized dot product distance) as the foundational metric, formulated as:

$\begin{aligned}
\mathcal{M}(\left<s_{i},g_{1}\right>,\left<s_{j},g_{2}\right>)=1-\frac{\psi(\left<s_{i},g_{1}\right>)^{\top}\psi(\left<s_{j},g_{2}\right>)}{\left\|\psi(\left<s_{i},g_{1}\right>)\right\|_{2}\left\|\psi(\left<s_{j},g_{2}\right>)\right\|_{2}}\end{aligned}$

(4)

Note that SimSR shares the same fixed point as MICo. Its update operator can
be defined in a manner analogous to MICo.

######  Theorem 1 ((Zang, Li, and Wang 2022))

Given a policy function $\pi$, the SimSR operator $T^{\pi}$ which defines the
state-goal tuple similarity between $\left<s_{i},g_{1}\right>$ and
$\left<s_{j},g_{2}\right>$ can be updated as:

$\displaystyle(T^{\pi}\mathcal{M})(\left<s_{i},g_{1}\right>,\left<s_{j},g_{2}\right>)=|r_{\left<s_{i},g_{1}\right>}-r_{\left<s_{j},g_{2}\right>}|+$
(5)
$\displaystyle\gamma\mathbb{E}_{s_{i+1}\sim\mathcal{T}_{\left<s_{i},g_{1}\right>}^{\pi},s_{j+1}\sim\mathcal{T}_{\left<s_{j},g_{2}\right>}^{\pi}}\left[\mathcal{M}(\left<s_{i+1},g_{1}\right>,\left<s_{j+1},g_{2}\right>)\right],$

for all
$\mathcal{M}:\left<\mathcal{S}\times\mathcal{G}\right>\times\left<\mathcal{S}\times\mathcal{G}\right>\rightarrow\mathbb{R}$.

The second term in Eq. (5) is derived through sample-based approximation.
Using the SimSR operator, one can iteratively refine the representation by
applying it to an arbitrarily initialized $\psi(\cdot)$.

Note that by using cosine distance as the foundational metric in SimSR, all
derived state features are normalized to unit length. Furthermore, in contrast
to the bisimulation metric which employs the Wasserstein distance, the SimSR
operator achieves a computational complexity on par with the MICo operator.

## Semi-Contrastive Representation Attack

We present a representation-based adversarial attack for GCRL, which is
independent of the critic function and can be seamlessly integrated in the
deployment phase.

### Limitations of Traditional Attacks in GCRL

Adversarial attacks on states lack the gradient information from labeled
examples. To navigate these constraints, existing methods lean on various
pseudo-labels, such as Q-values (Zhang et al. 2020b; Kos and Song 2017) or
actions (Gleave et al. 2019; Sun et al. 2020) to generate adversarial states.
However, as illustrated in Eq. (2), Q-value function is heavily influenced by
the reward sequence. Moreover, many of these attacks necessitate access to
value networks, making them unsuitable for direct deployment in the inference
stage. To address these challenges, we introduce a novel attack method
tailored for GCRL algorithms. This approach neither depends on specific
pseudo-labels nor requires access to the critic network. Although crafted with
GCRL’s unique characteristics, it is versatile enough for broader applications
of RL algorithms.

### Negative Tuple-Based Adversary

To craft adversaries without relying on labels or pseudo-labels, several
studies (Ho and Nvasconcelos 2020; Jiang et al. 2020; Kim, Tack, and Hwang
2020) have introduced the concept of adversarial contrastive attack in the
realm of unsupervised learning. Viewed through the lens of representation, an
adversary $\mathcal{V}(x)$ for a clean input $x$ is designed to ensure that
the feature $f(\mathcal{V}(x))$ diverges significantly from $f(x^{+})$ while
converging towards the feature of a negative sample, $f(x^{-})$. Typically,
these methods employ the NT-Xent loss to gauge the similarity among
$f(\mathcal{V}(x))$, $f(x^{+})$, and $f(x^{-})$. This loss can be articulated
as:

$\begin{aligned}
-\log\frac{\sum_{\\{x^{+}\\}}\exp\left(\mathbb{S}(\mathcal{V}(x),x^{+})\right)}{\sum_{\\{x^{+}\\}}\exp\left(\mathbb{S}(\mathcal{V}(x),x^{+})\right)+\sum_{\\{x^{-}\\}}\exp\left(\mathbb{S}(\mathcal{V}(x),x^{-})\right)},\end{aligned}$

(6)

where $\mathbb{S}(a,b)=\frac{f(a)^{\top}f(b)}{\tau}$, $\tau$ refers to the
temperature hyperparameter. Normally, the positive set $\\{x^{+}\\}$ can be
constructed using different types of data augmentation, such as rotation,
color jittering, or scaling, while $\left\\{x^{-}\right\\}$ indicates the set
of samples collected from other classes. In RL contexts, it is challenging to
directly quantify positively correlated samples. Thus, in this section, we
present an adversarial contrastive attack approach that exclusively focuses on
the negative tuple, termed the Semi-Contrastive Representation (SCR) attack in
Def. 2.

######  Definition 2 (SCR attack)

Given a feature extraction function $f(\cdot)$ and the input tuple
$\left<s,g\right>$, the semi-contrastive representation attack can be defined
as follows:

$\begin{aligned}
\mathop{\arg\sup}_{\mathcal{V}(s)\in\mathcal{B}_{p}^{\epsilon}(s),\mathcal{V}(g)\in\mathcal{B}_{p}^{\epsilon}(g)}\mathbb{E}_{\left<s,g\right>^{-}}\left[\mathcal{L}_{lg}\left(-f(\left<\mathcal{V}(s),\mathcal{V}(g)\right>)^{\top}f(\left<s,g\right>^{-})\right)\right],\end{aligned}$

(7)

where the logistic function $\mathcal{L}_{lg}(v)=\log(1+\exp(-v))$ for any
$v\in\mathbb{R}$, $\left<s,g\right>^{-}$ indicates the negative tuple.

It is easy to derive that
$\frac{\mathcal{L}_{lg}(v_{1})+\mathcal{L}_{lg}(v_{2})}{2}\geq\mathcal{L}_{lg}(\frac{v_{1}+v_{2}}{2})$,
so we can apply the sub-additivity to the supremum of the values of
$\mathcal{L}_{lg}$ by:

$\begin{aligned}
&\sup_{\mathcal{V}(s),\mathcal{V}(g)}\mathbb{E}_{\left<s,g\right>^{-}}\left[\mathcal{L}_{lg}\left(-f(\left<\mathcal{V}(s),\mathcal{V}(g)\right>)^{\top}f(\left<s,g\right>^{-})\right)\right]\\\
&\leq\mathbb{E}_{\left<s,g\right>^{-}}\sup_{\mathcal{V}(s),\mathcal{V}(g)}\left[\mathcal{L}_{lg}\left(-f(\left<\mathcal{V}(s),\mathcal{V}(g)\right>)^{\top}f(\left<s,g\right>^{-})\right)\right].\end{aligned}$

(8)

Consequently, we can reframe the challenge of computing the supremum over
expectations into determining the expectation over the supremum. This means
our goal is to optimize $\mathcal{L}_{lg}$ for a given negative tuple
$\left<s,g\right>^{-}$. Moreover, leveraging the non-increasing nature of
$\mathcal{L}_{lg}(\cdot)$, we can approximate our attack objective by directly
minimizing the similarity-based loss
$-f(\left<\mathcal{V}(s),\mathcal{V}(g)\right>)^{\top}f(\left<s,g\right>^{-})$.
In this paper, $f(\cdot)$ is equivalent to $\psi(\cdot)$. Building on this, we
present a Projected Gradient Descent (PGD)-based approximation for both
$\mathcal{V}(s)$ and $\mathcal{V}(g)$.

######  Definition 3 (PGD-based Approximation)

Given an encoder $\psi(\cdot)$, an original input tuple $\left<s,g\right>$, a
pre-defined negative tuple $\left<s,g\right>^{-}$, and the step size $\alpha$,
then the semi-contrastive representation attack $\mathcal{V}_{scr}(s)$ and
$\mathcal{V}_{scr}(g)$ at the iterative step $i+1$ can be individually defined
as:

$\mathcal{V}^{i+1}_{scr}(s)=\mathcal{V}^{i}_{scr}(s)-\alpha\nabla_{\mathcal{V}_{scr}^{i}(s)}\mathcal{L}_{sim}(s,g,i),$
(9)
$\mathcal{V}^{i+1}_{scr}(g)=\mathcal{V}^{i}_{scr}(g)-\alpha\nabla_{\mathcal{V}_{scr}^{i}(g)}\mathcal{L}_{sim}(s,g,i),$
(10)

Particularly, we define:

$\begin{aligned}
\mathcal{L}_{sim}(s,g,i)=-\psi\left(\left<\mathcal{V}_{scr}^{i}(s),\mathcal{V}^{i}_{scr}(g)\right>\right)^{\top}\psi(\left<s,g\right>^{-}).\end{aligned}$

(11)

If the number of iteration steps is $\mathcal{I}(\geq$1$)$, then the finally
generated adversarial tuple can be defined as
$\left<\mathrm{proj}(\mathcal{V}^{I}_{scr}(s)),\mathrm{proj}(\mathcal{V}^{I}_{scr}(g))\right>$,
where $\mathrm{proj}(\cdot)$ is the projection head.

In this work, We denote $proj(\cdot)$ in Def. 3 as an $\epsilon$-bounded
$\ell_{\infty}$-norm ball. In a straightforward yet effective manner, we
construct the negative tuple $\left<s,g\right>^{-}$ by performing negation
operation, including $\left<-s,-g\right>$, $\left<-s,g\right>$ and
$\left<s,-g\right>$. By incorporating the SCR attack at each timestep within
an episode of $T$ steps in GCRL, our primary aim is to divert the agent,
ensuring that it remains distant from the goal. This approach ensures that the
reward sequence $(r_{0},\cdots,r_{T-1})$ remains as sparse as possible.

To verify the efficacy of our approach, we train multiple agents using diverse
GCRL algorithms and evaluate their adversarial robustness against our
introduced attack. Comprehensive results are presented in the Experiments
section.

## Adversarial Representation Tactics

As illustrated in Tab. 1 and 2, GCRL agents trained using both base methods
and SimSR-strengthened ones are vulnerable to our SCR attack, highlighting a
significant security concern. To address this, we introduce a composite
defensive strategy termed ARTs in this section, which strategically combine
the Semi-Contrastive Adversarial Augmentation (SCAA) and the Sensitivity-Aware
Regularizer (SAR), catering to diverse GCRL algorithms.

### Semi-Contrastive Adversarial Augmentation

In Algorithm 1, we investigate the influence of data augmentations by
subjecting input tuples to the SCR attack. Specifically, during each training
epoch for the base GCRL agent, we retrieve a mini-batch of tuples from the
replay buffer and generate semi-contrastive augmented samples,
$\mathcal{V}_{scr}(s_{t}^{i})$ and $\mathcal{V}_{scr}(g^{i})$, using the
constructed negative tuples. We then utilize the augmented critic value,
denoted as $\hat{\varrho}$, and the actor value, represented as
$\hat{\varphi}$, to refine the weights of the encoder ($\theta_{\psi}$), actor
network ($\theta_{\varphi}$), and critic network ($\theta_{\varrho}$).
Notably, $\mathcal{L}_{\psi}$, $\mathcal{L}_{\varphi}$, and
$\mathcal{L}_{\varrho}$ correspond to the loss functions for the encoder,
actor network, and critic network, respectively.

### Sensitivity-Aware Regularizer

Given two pairs of observations
$\left(s_{i},g_{1},r_{\left<s_{i},g_{1}\right>},a_{i},s_{i+1}\right)$ and
$\left(s_{j},g_{2},r_{\left<s_{j},g_{2}\right>},a_{j},s_{j+1}\right)$ shown in
Fig. 1, it is common that
$r_{\left<s_{i},g_{1}\right>}=r_{\left<s_{j},g_{2}\right>}=0$, due to the
sparsity of reward sequences. Therefore, given the above pair of transitions,
we can draw from Thm. 1, and iteratively update the encoder $\psi(\cdot)$ in
the policy function using the mean square loss:

$\begin{aligned}
\left(\mathcal{M}(\left<s_{i},g_{1}\right>,\left<s_{j},g_{2}\right>)-\gamma\mathbb{E}_{s_{i+1},s_{j+1}}\left[\mathcal{M}(\left<s_{i+1},g_{1}\right>,\left<s_{j+1},g_{2}\right>)\right]\right)^{2},\end{aligned}$

(12)

where $\mathcal{M}(\left<s_{i},g_{1}\right>,\left<s_{j},g_{2}\right>)$ is
highly dependent on the next states $s_{i+1}$ and $s_{j+1}$, and rarely
captures information from the absolute reward difference
$\left|r_{\left<s_{i},g_{1}\right>}-r_{\left<s_{j},g_{2}\right>}\right|$. This
cannot provide any evaluations of the difference between state-goal tuples at
the current step, and prevents the policy and critic functions from gaining
insight from the interaction between the agent and the environment. To
compensate for this deficiency of SimSR utilized in GCRL, we construct the
Sensitivity-Aware Regularizer as a substitute for
$\left|r_{\left<s_{i},g_{1}\right>}-r_{\left<s_{j},g_{2}\right>}\right|$.

######  Definition 4 (Sensitivity-Aware Regularizer)

Given perturbations $\delta_{s_{i}}$, $\delta_{g_{1}}$, $\delta_{s_{j}}$,
$\delta_{g_{2}}$ for tuples $\left<s_{i},g_{1}\right>$ and
$\left<s_{j},g_{2}\right>$, and a trade-off factor $\beta$, the sensitivity-
aware regularizer for the encoder $\psi(\cdot)$ can be defined as:

$\begin{aligned}
&\left|\frac{\mathcal{M}(\left<s_{i},g_{1}\right>,\left<s_{i}+\delta_{s_{i}},g_{1}\right>)}{\left\|\delta_{s_{i}}\right\|_{2}}-\frac{\mathcal{M}(\left<s_{j},g_{2}\right>,\left<s_{j}+\delta_{s_{j}},g_{2}\right>)}{\left\|\delta_{s_{j}}\right\|_{2}}\right|+\\\
&\beta\left|\frac{\mathcal{M}(\left<s_{i},g_{1}\right>,\left<s_{i},g_{1}+\delta_{g_{1}}\right>)}{\left\|\delta_{g_{1}}\right\|_{2}}-\frac{\mathcal{M}(\left<s_{j},g_{2}\right>,\left<s_{j},g_{2}+\delta_{g_{2}}\right>)}{\left\|\delta_{g_{2}}\right\|_{2}}\right|,\end{aligned}$

(13)

then we can empirically optimize the encoder parameters $\theta_{\psi}$ using
the loss function combining Eq. (12) and Eq. (13).

As shown in Def. 4, we explore the Lipschitz constant-based robustness term
$\mathcal{M}(\cdot)/\left\|\delta\right\|_{2}$ to indicate the implicit
difference. Generally, on the one hand, physically close tuples should have
similar Lipschitz constants both on states and goals with a high probability,
physically distant tuples. On the other hand, physically distant tuples tend
to have dissimilar Lipschitz constants with a high probability. This reflects
the similar characteristic as the absolute reward difference utilized in Eq.
(5).

Algorithm 1 Semi-Contrastive Adversarial Augmentation

Require: Offline dataset $\mathcal{D}$, learning rate $\alpha$, training steps
$T$
Initialisation: Critic function $\varrho(\cdot)$, policy function
$\varphi(\psi(\cdot))$

1:for training timestep $1\dots T$ do

2: Sample a mini-batch from the offline dataset:
$(s_{t}^{i},a_{t}^{i},s_{t+1}^{i},g^{i})\sim\mathcal{D}$

3: Construct negative tuples: $\left<s_{t}^{i},g^{i}\right>^{-}$ and
$\left<s_{t+1}^{i},g^{i}\right>^{-}$

4: Compute the semi-contrastive augmented samples:
$\mathcal{V}_{scr}(s_{t}^{i})$ and $\mathcal{V}_{scr}(g^{i})$ by Eq. (9-10)

5: Augment critic value:
$\hat{\varrho}\leftarrow\mathbb{E}[\varrho(s_{t}^{i},g^{i},a_{t}^{i})+\varrho(\mathcal{V}_{scr}(s_{t}^{i}),\mathcal{V}_{scr}(g^{i}),a_{t}^{i})]$

6: Augment actor value:
$\scriptstyle\hat{\varphi}\leftarrow\mathbb{E}[\varphi(\psi(\left<s_{t}^{i},g^{i}\right>))+\varphi(\psi(\left<\mathcal{V}_{scr}(s_{t}^{i}),\mathcal{V}_{scr}(g^{i})\right>)]$

7: Update encoder weights:
$\scriptstyle\theta_{\psi}\leftarrow\theta_{\psi}-\alpha\nabla\mathcal{L}_{\psi}(\left<s_{t}^{i},g^{i}\right>)$

8: Update actor weights:
$\theta_{\varphi}\leftarrow\theta_{\varphi}-\alpha\nabla\mathcal{L}_{\varphi}(\hat{\varphi})$

9: Update critic weights:
$\theta_{\varrho}\leftarrow\theta_{\varrho}-\alpha\nabla\mathcal{L}_{\varrho}(\hat{\varrho})$

Task | Method |  | Attack Return
---|---|---|---
Nature | Uniform | SA-FGSM | SA-PGD | SCR-FGSM | SCR-PGD
|  |  |  | state | goal | state+goal | state | goal | state+goal
FetchPick | DDPG | 14.82$\pm$1.53 | 16.68$\pm$4.33 | 19.61$\pm$2.84 | 19.92$\pm$2.32 | 11.18$\pm$2.74 | 15.00$\pm$1.88 | 7.83$\pm$3.06 | 10.97$\pm$4.12 | 16.38$\pm$2.72 | 10.49$\pm$2.86
GCSL | 11.39$\pm$1.97 | 9.95$\pm$1.65 | 9.94$\pm$1.80 | 10.07$\pm$1.15 | 9.52$\pm$2.13 | 10.41$\pm$1.95 | 8.99$\pm$1.56 | 8.21$\pm$3.12 | 11.72$\pm$1.81 | 8.11$\pm$2.04
GoFar | 21.01$\pm$2.05 | 19.91$\pm$2.39 | 20.07$\pm$2.13 | 18.92$\pm$1.80 | 17.14$\pm$2.89 | 20.22$\pm$3.25 | 16.71$\pm$3.26 | 17.93$\pm$2.48 | 19.84$\pm$1.88 | 15.65$\pm$1.83
FetchPush | DDPG | 12.81$\pm$4.61 | 13.61$\pm$4.81 | 17.24$\pm$2.12 | 15.42$\pm$3.38 | 8.71$\pm$3.22 | 10.27$\pm$3.85 | 6.43$\pm$2.33 | 9.79$\pm$2.15 | 13.61$\pm$2.56 | 5.42$\pm$3.00
GCSL | 12.74$\pm$1.38 | 12.57$\pm$1.67 | 9.64$\pm$3.28 | 9.71$\pm$3.37 | 9.64$\pm$0.51 | 12.97$\pm$3.09 | 8.99$\pm$1.56 | 6.20$\pm$3.66 | 13.56$\pm$1.49 | 8.11$\pm$2.04
GoFar | 18.38$\pm$3.24 | 16.11$\pm$2.39 | 15.45$\pm$2.71 | 13.66$\pm$3.97 | 11.66$\pm$3.93 | 17.47$\pm$2.07 | 8.19$\pm$3.99 | 12.14$\pm$3.78 | 18.38$\pm$2.91 | 10.57$\pm$3.93
FetchReach | DDPG | 29.92$\pm$0.20 | 28.86$\pm$0.46 | 24.21$\pm$2.21 | 20.14$\pm$5.04 | 6.10$\pm$7.33 | 5.14$\pm$7.19 | 2.02$\pm$3.32 | 7.02$\pm$2.20 | 11.03$\pm$8.47 | 1.86$\pm$3.31
GCSL | 22.04$\pm$0.68 | 22.75$\pm$0.82 | 22.01$\pm$1.47 | 21.95$\pm$1.51 | 20.97$\pm$1.31 | 22.24$\pm$0.78 | 23.04$\pm$1.62 | 21.74$\pm$0.81 | 22.22$\pm$0.79 | 23.46$\pm$1.14
GoFar | 27.84$\pm$0.63 | 27.57$\pm$0.92 | 27.22$\pm$0.49 | 27.22$\pm$0.49 | 27.51$\pm$0.45 | 27.45$\pm$1.11 | 15.71$\pm$4.48 | 27.96$\pm$0.38 | 27.87$\pm$0.33 | 11.24$\pm$6.76
FetchSlide | DDPG | 0.58$\pm$1.30 | 1.89$\pm$0.89 | 0.59$\pm$1.09 | 0.50$\pm$1.12 | 0.00$\pm$0.00 | 0.36$\pm$0.72 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.38$\pm$0.60 | 0.00$\pm$0.00
GCSL | 1.55$\pm$0.84 | 1.84$\pm$1.19 | 0.60$\pm$0.42 | 0.57$\pm$0.59 | 0.28$\pm$0.63 | 0.97$\pm$0.63 | 0.37$\pm$0.51 | 0.37$\pm$0.84 | 0.96$\pm$1.26 | 0.00$\pm$0.00
GoFar | 2.55$\pm$1.37 | 1.23$\pm$1.14 |  0.77$\pm$0.92 | 1.13$\pm$0.72 | 0.09$\pm$0.21 | 1.25$\pm$0.55 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 1.86$\pm$0.73 | 0.09$\pm$0.22

Table 1: Comparison of discounted returns for DDPG, GCSL, and GoFar against
various attack methods in GCRL. Red, Purple, and Blue individually represent
the worst, the second worst, and the third worst returns in each row. The
experiments are averaged over 5 seeds.

| DDPG (SimSR)
---|---
Task | Nature | Uniform | SA-FGSM | SA-PGD | SCR-FGSM | SCR-PGD
FetchPick | 16.78 | 16.00 | 19.51 | 17.92 | 15.74 | 16.15
FetchPush | 12.63 | 15.43 | 16.24 | 14.61 | 10.70 | 10.51
FetchReach | 29.90 | 29.46 | 28.68 | 26.76 | 27.72 | 27.16
FetchSlide | 0.62 | 0.83 | 0.80 | 0.43 | 0.37 | 0.84
GoFar (SimSR)
---
Nature | Uniform | SA-FGSM | SA-PGD | SCR-FGSM | SCR-PGD
17.44 | 16.73 | 17.63 | 13.80 | 16.27 | 16.11
14.19 | 13.13 | 15.52 | 13.91 | 12.44 | 13.08
27.93 | 27.91 | 27.95 | 27.96 | 27.96 | 27.98
1.66 | 1.78 | 2.31 | 1.41 | 1.07 | 0.59

Table 2: Comparison of discounted returns for DDPG (SimSR) and GoFar (SimSR)
against various attack methods in GCRL, averaged over 5 seeds. Bold denotes
the worst return in each row.

## Experiments

Figure 2: Epoch-wise evaluations on the SCR attack of ARTs-defended DDPG, DDPG
(SimSR), GoFar, and GoFar (SimSR).

Our experimental design proceeds as follows: Initially, we identify the
optimal representation layer by executing SCR attacks across various layers.
Next, we benchmark the SCR attack against other adversarial attacks, using
state-of-the-art algorithms as base methods. Finally, we achieve enhanced
robustness of different base methods using ARTs.

### Baselines

We evaluate our method222Our code is available at github.com/TrustAI/ReRoGCRL
using four robot manipulation tasks, namely FetchPush, FetchReach, FetchSlide,
and FetchPick, as described in (Plappert et al. 2018). The offline dataset for
each task is collected through either a purely random policy or a combination
of $90\%$ random policies and $10\%$ expert policies, depending on whether the
random data sufficiently represents the desired goal distribution. We select 3
algorithms as our baselines for GCRL: Deep Deterministic Policy Gradient
(DDPG) (Andrychowicz et al. 2018), Goal-Conditioned Supervised Learning (GCSL)
(Ghosh et al. 2020), and Goal-Conditioned F-Advantage Regression (GoFar) (Ma
et al. 2022). It is worth noting that to circumvent costly and potentially
risky environment interactions, we learn general goal-reaching policies from
offline interaction datasets.

### Network Architectures

We use 3-layer MLPs as the backbones of $\varphi(\psi(\cdot))$ and
$\varrho(\cdot)$. To determine $L_{\psi}$ and $L_{\varphi}$, we conduct
preliminary experiments based on $\varphi(\psi(\cdot))$, trained via DDPG and
GoFar to ascertain the optimal layer for representation. As depicted in Fig.
3, we compute the discounted returns across three distinct layers (Layer 1
denotes the first layer of $\varphi(\psi(\cdot))$, and so forth), with each
layer acting as the representation layer. Comprehensive results indicate that
Layer 1 is the most susceptible. Therefore, all attacks and training
algorithms in this paper are computed on Layer 1.

In particular, the encoder $\psi(\cdot)$ processes the input concatenating a
state $s$ and a goal $g$, which is represented as $o_{0}^{\pi}$. The dimension
of the input tuple is $D_{sg}$. Thus, we define the latent representation
given by $\psi(o_{0}^{\pi})=\phi(\mathbf{W}_{1}o_{0}^{\pi})$,
$\mathbf{W}^{\pi}_{1}\in\mathbb{R}^{256\times D_{sg}}$. Subsequently, the
actor network is constructed as
$\varphi(\psi(o_{0}^{\pi}))=\mathbf{W}^{\pi}_{4}\phi(\mathbf{W}^{\pi}_{3}\phi(\mathbf{W}^{\pi}_{2}\psi(o_{0}^{\pi})))$,
with $\mathbf{W}^{\pi}_{2},\mathbf{W}^{\pi}_{3}\in\mathbb{R}^{256\times 256}$,
$\mathbf{W}^{\pi}_{4}\in\mathbb{R}^{D_{a}\times 256}$, where $D_{a}$ is the
dimension of the action. Similar to the policy function $\pi$, we construct
the output of critic function as
$o_{4}^{c}=\mathbf{W}^{c}_{4}\phi(\mathbf{W}^{c}_{3}\phi(\mathbf{W}^{c}_{2}\phi(\mathbf{W}^{c}_{1}o^{c}_{0})))$,
where $o_{0}^{c}$ is the concatenation of the tuple $\left<s,g\right>$ and the
action $a$, which has $(D_{sg}+D_{a})$ dimensions. Specifically,
$\mathbf{W}_{1}^{c}\in\mathbb{R}^{256\times(D_{sg}+D_{a})}$,$\mathbf{W}_{2}^{c}$,
$\mathbf{W}_{3}^{c}$ $\in\mathbb{R}^{256\times 256}$, and
$\mathbf{W}_{4}^{c}\in\mathbb{R}^{1\times 256}$.

### Comparison of SCR and Other Attacks

Following Def. 3, we evaluate the robustness of multiple GCRL algorithms using
$5$ different attack methods, including Uniform, SA-FGSM, SA-PGD, SCR-FGSM,
and SCR-PGD. Specifically, we employ a $10$-step PGD with a designated step
size of $0.01$. For both FGSM and PGD attacks, the attack radius is set to
$0.1$.

Our SCR attack introduces perturbations in the form of
$\left<\mathcal{V}(s),g\right>$, $\left<s,\mathcal{V}(g)\right>$, and
$\left<\mathcal{V}(s),\mathcal{V}(g)\right>$, respectively. Specifically, Tab.
1-2 presents the results when noise is added solely to the states. We provide
the remaining attack results in the Appendix. As outlined in Def. 3, The
columns state, goal, and state+goal denote negative tuples as
$\left<-s,g\right>$, $\left<s,-g\right>$, and $\left<-s,-g\right>$
individually.

As shown in Table 1, for DDPG-based GCRL, we achieve $47.17\%$, $57.69\%$,
$93.78\%$, and $100.00\%$ decrease of discounted returns in FetchPick,
FetchPush, FetchReach, and FetchSlide respectively, outperforming SA-based
attacks by $60.07\%$, $64.85\%$, $90.76\%$, and $100.00\%$. In detail, all of
them are derived from the negative tuple $\left<-s,-g\right>$. The results are
similar in GCSL, compared with the nature return, our method degrades the
performance in FetchPick, FetchPush, and FetchSlide each by $18.41\%$,
$35.68\%$, and $100.00\%$. Note that GCSL-based GCRL is more robust in
FetchReach than DDPG-based one, our attack can only decrease the performance
by $4.46\%$. As the state-of-the-art algorithm in GCRL, GoFar achieves better
nature returns than DDPG and GCSL, thus all attacks exhibit a degradation in
their attack capabilities. Particularly, our method achieves a reduction in
overall returns of $17.28\%$, $40.04\%$, $58.71\%$, and $100.00\%$ across the
four tasks, respectively. Similar to DDPG, the best-performing attack for each
of these 4 tasks originates from $\left<-s,-g\right>$.

We further evaluate the adversarial robustness of SimSR-enhanced algorithms as
presented in Tab. 2. All attack strategies employ the negative tuple
$\left<-s,g\right>$, notably, our attack surpasses most other attacks when
applied to DDPG. However, when it comes to GoFar, our strategy faces more
robust defensive measures.

Figure 3: Evaluation of the SCR attack across various layers in MLP-based
architectures. The upper charts use the DDPG method. The lower charts use the
GoFar method.

### Effectiveness of ARTs

To test the efficacy of ARTs, we correspondingly perform attacks on ARTs-
enhanced versions of different GCRL algorithms. Following Alg. 1, we train the
agent using the SCR attack. For SimSR-boosted representations, we adaptively
employ SAR during the optimization of $\psi(\cdot)$. Tab. 3 only showcases the
best attack results from Uniform, SA-FGSM, SA-PGD, SCR-FGSM, and SCR-PGD. All
attacks and training procedures in this section are based on
$\left<-s,g\right>$, with perturbations applied solely to the state positions.

As illustrated in Tab. 3, our defensive strategy significantly bolsters the
robust performance of DDPG. Specifically, we register performance enhancements
of $85.57\%$, $99.08\%$, and $1388.17\%$ in FetchPick, FetchPush and
FetchReach, respectively. In a similar vein, for DDPG (SimSR), the
implementation of ARTs leads to robustness improvements of $6.61\%$,
$20.17\%$, $10.13\%$, and $48.65\%$ across FetchPick, FetchPush, FetchReach,
and FetchSlide, respectively.

For GoFar, our results are comparable and even superior in FetchPush,
FetchReach, and FetchSlide. Additionally, for GoFar (SimSR), we observe a
performance boost of $7.39\%$ and $149.15\%$, respectively, in FetchPick and
FetchSlide. Complete results are available in the Appendix.

Qualitatively, Fig. 2 provides a visual representation of the epoch-wise
performance of ARTs-enhanced GCRL algorithms in comparison to several base
methods. In the FetchReach environment, both DDPG+SCAA and GoFar+SCAA exhibit
notable improvements within the initial 100 epochs. A similar trend is
observed in the FetchPush environment.

Methods | FetchPick | FetchPush | FetchReach | FetchSlide
---|---|---|---|---
DDPG | Vanilla | 7.83$\pm$3.06 | 5.42$\pm$3.00 | 1.86$\pm$3.31 | 0.00$\pm$0.00
Vanilla+SCAA | 14.53$\pm$3.08 | 10.79$\pm$4.55 | 27.68$\pm$1.30 | 0.00$\pm$0.00
SimSR | 15.74$\pm$0.65 | 10.51$\pm$5.79 | 27.16$\pm$2.75 | 0.37$\pm$0.60
SimSR+SAR | 16.78$\pm$1.39 | 12.63$\pm$3.89 | 29.91$\pm$0.20 | 0.55$\pm$0.20
GoFar | Vanilla | 15.65$\pm$1.83 | 8.19$\pm$3.99 | 11.24$\pm$6.76 | 0.00$\pm$0.00
Vanilla+SCAA | 14.72$\pm$4.40 | 8.00$\pm$3.76 | 13.99$\pm$6.46 | 0.00$\pm$0.00
SimSR | 13.80$\pm$4.40 | 12.44$\pm$3.80 | 27.91$\pm$0.34 | 0.59$\pm$0.66
SimSR+SAR | 14.82$\pm$3.04 | 12.44$\pm$3.80 | 27.84$\pm$0.32 | 1.47$\pm$0.79

Table 3: Defensive performance of ARTs on DDPG, GoFar, DDPG (SimSR), and GoFar
(SimSR) against best attacks, averaged over 5 seeds. Bold indicates the better
result in a block.

## Conclusion

Due to the sparse rewards in GCRL, we introduce the Semi-Contrastive
Representation attack to probe its vulnerability, which only requires access
to the policy function. Accordingly, we propose Adversarial Representation
Tactics to bolster the robust performance of the underlying agent. In
particular, we devise the Semi-Contrastive Adversarial Augmentation for
baseline methods in GCRL and introduce the Sensitivity-Aware Regularizer for
their robust counterparts that are guided by the bisimulation metric. We
utilize SimSR to learn invariant representations. The effectiveness of our
methods is validated by intensive empirical experiments.

## Acknowledgements

This work is supported by the UK EPSRC under project EnnCORE [EP/T026995/1],
the University of Liverpool and the China Scholarship Council.

## References

  * Andrychowicz et al. (2018) Andrychowicz, M.; Wolski, F.; Ray, A.; Schneider, J.; Fong, R.; Welinder, P.; McGrew, B.; Tobin, J.; Abbeel, P.; and Zaremba, W. 2018.  Hindsight Experience Replay.  arXiv:1707.01495.
  * Bai, Guan, and Wang (2019) Bai, X.; Guan, J.; and Wang, H. 2019.  A model-based reinforcement learning with adversarial training for online recommendation.  _Advances in Neural Information Processing Systems_ , 32.
  * Castro et al. (2022) Castro, P. S.; Kastner, T.; Panangaden, P.; and Rowland, M. 2022.  MICo: Improved representations via sampling-based state similarity for Markov decision processes.  arXiv:2106.08229.
  * Chane-Sane, Schmid, and Laptev (2021) Chane-Sane, E.; Schmid, C.; and Laptev, I. 2021.  Goal-conditioned reinforcement learning with imagined subgoals.  In _International Conference on Machine Learning_ , 1430–1440. PMLR.
  * Charlesworth and Montana (2020) Charlesworth, H.; and Montana, G. 2020.  Plangan: Model-based planning with sparse rewards and multiple goals.  _Advances in Neural Information Processing Systems_ , 33: 8532–8542.
  * Chebotar et al. (2021) Chebotar, Y.; Hausman, K.; Lu, Y.; Xiao, T.; Kalashnikov, D.; Varley, J.; Irpan, A.; Eysenbach, B.; Julian, R.; Finn, C.; et al. 2021.  Actionable models: Unsupervised offline reinforcement learning of robotic skills.  _arXiv preprint arXiv:2104.07749_.
  * Eysenbach, Salakhutdinov, and Levine (2019) Eysenbach, B.; Salakhutdinov, R. R.; and Levine, S. 2019.  Search on the replay buffer: Bridging planning and reinforcement learning.  _Advances in Neural Information Processing Systems_ , 32.
  * Eysenbach et al. (2022) Eysenbach, B.; Zhang, T.; Levine, S.; and Salakhutdinov, R. R. 2022.  Contrastive learning as goal-conditioned reinforcement learning.  _Advances in Neural Information Processing Systems_ , 35: 35603–35620.
  * Fang et al. (2018) Fang, M.; Zhou, C.; Shi, B.; Gong, B.; Xu, J.; and Zhang, T. 2018.  DHER: Hindsight experience replay for dynamic goals.  In _International Conference on Learning Representations_.
  * Fang et al. (2019) Fang, M.; Zhou, T.; Du, Y.; Han, L.; and Zhang, Z. 2019.  Curriculum-guided hindsight experience replay.  _Advances in neural information processing systems_ , 32.
  * Ferns, Panangaden, and Precup (2011) Ferns, N.; Panangaden, P.; and Precup, D. 2011.  Bisimulation metrics for continuous Markov decision processes.  _SIAM Journal on Computing_ , 40(6): 1662–1714.
  * Ferns, Panangaden, and Precup (2012) Ferns, N.; Panangaden, P.; and Precup, D. 2012.  Metrics for Finite Markov Decision Processes.  arXiv:1207.4114.
  * Gelada et al. (2019) Gelada, C.; Kumar, S.; Buckman, J.; Nachum, O.; and Bellemare, M. G. 2019.  DeepMDP: Learning Continuous Latent Space Models for Representation Learning.  arXiv:1906.02736.
  * Ghosh et al. (2020) Ghosh, D.; Gupta, A.; Reddy, A.; Fu, J.; Devin, C. M.; Eysenbach, B.; and Levine, S. 2020.  Learning to Reach Goals via Iterated Supervised Learning.  In _International Conference on Learning Representations_.
  * Givan, Dean, and Greig (2003) Givan, R.; Dean, T.; and Greig, M. 2003.  Equivalence notions and model minimization in Markov decision processes.  _Artificial Intelligence_ , 147(1-2): 163–223.
  * Gleave et al. (2019) Gleave, A.; Dennis, M.; Wild, C.; Kant, N.; Levine, S.; and Russell, S. 2019.  Adversarial policies: Attacking deep reinforcement learning.  _arXiv preprint arXiv:1905.10615_.
  * He and Lv (2023) He, X.; and Lv, C. 2023.  Robotic Control in Adversarial and Sparse Reward Environments: A Robust Goal-Conditioned Reinforcement Learning Approach.  _IEEE Transactions on Artificial Intelligence_ , 1–10.
  * Ho and Nvasconcelos (2020) Ho, C.-H.; and Nvasconcelos, N. 2020.  Contrastive learning with adversarial examples.  _Advances in Neural Information Processing Systems_ , 33: 17081–17093.
  * Huang et al. (2017) Huang, S.; Papernot, N.; Goodfellow, I.; Duan, Y.; and Abbeel, P. 2017.  Adversarial attacks on neural network policies.  _arXiv preprint arXiv:1702.02284_.
  * Huang, Jin, and Ruan (2023) Huang, X.; Jin, G.; and Ruan, W. 2023.  Deep reinforcement learning.  In _Machine Learning Safety_ , 219–235. Springer.
  * Huang et al. (2020) Huang, X.; Kroening, D.; Ruan, W.; Sharp, J.; Sun, Y.; Thamo, E.; Wu, M.; and Yi, X. 2020.  A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability.  _Computer Science Review_ , 37: 100270.
  * Jiang et al. (2020) Jiang, Z.; Chen, T.; Chen, T.; and Wang, Z. 2020.  Robust pre-training by adversarial contrastive learning.  _Advances in neural information processing systems_ , 33: 16199–16210.
  * Kim, Tack, and Hwang (2020) Kim, M.; Tack, J.; and Hwang, S. J. 2020.  Adversarial self-supervised contrastive learning.  _Advances in Neural Information Processing Systems_ , 33: 2983–2994.
  * Kos and Song (2017) Kos, J.; and Song, D. 2017.  Delving into adversarial attacks on deep policies.  _arXiv preprint arXiv:1705.06452_.
  * Larsen and Skou (1989) Larsen, K. G.; and Skou, A. 1989.  Bisimulation through probabilistic testing (preliminary report).  In _Proceedings of the 16th ACM SIGPLAN-SIGACT symposium on Principles of programming languages_ , 344–352.
  * Ma et al. (2022) Ma, J. Y.; Yan, J.; Jayaraman, D.; and Bastani, O. 2022.  Offline goal-conditioned reinforcement learning via $f$-advantage regression.  _Advances in Neural Information Processing Systems_ , 35: 310–323.
  * Mezghani et al. (2022) Mezghani, L.; Sukhbaatar, S.; Bojanowski, P.; Lazaric, A.; and Alahari, K. 2022.  Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping.  In _6th Annual Conference on Robot Learning_.
  * Mezghani et al. (2023) Mezghani, L.; Sukhbaatar, S.; Bojanowski, P.; Lazaric, A.; and Alahari, K. 2023.  Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping.  In _Conference on Robot Learning_ , 1401–1410. PMLR.
  * Mu et al. (2023a) Mu, R.; Marcolino, L. S.; Zhang, T.; Zhang, Y.; Huang, X.; and Ruan, W. 2023a.  Reward Certification for Policy Smoothed Reinforcement Learning.  _arXiv preprint arXiv:2312.06436_.
  * Mu et al. (2023b) Mu, R.; Ruan, W.; Marcolino, L. S.; Jin, G.; and Ni, Q. 2023b.  Certified Policy Smoothing for Cooperative Multi-Agent Reinforcement Learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’23)_.
  * Nasiriany et al. (2019) Nasiriany, S.; Pong, V.; Lin, S.; and Levine, S. 2019.  Planning with goal-conditioned policies.  _Advances in Neural Information Processing Systems_ , 32.
  * Pinto et al. (2017) Pinto, L.; Davidson, J.; Sukthankar, R.; and Gupta, A. 2017.  Robust adversarial reinforcement learning.  In _International Conference on Machine Learning_ , 2817–2826. PMLR.
  * Plappert et al. (2018) Plappert, M.; Andrychowicz, M.; Ray, A.; McGrew, B.; Baker, B.; Powell, G.; Schneider, J.; Tobin, J.; Chociej, M.; Welinder, P.; et al. 2018.  Multi-goal reinforcement learning: Challenging robotics environments and request for research.  _arXiv preprint arXiv:1802.09464_.
  * Sun et al. (2020) Sun, J.; Zhang, T.; Xie, X.; Ma, L.; Zheng, Y.; Chen, K.; and Liu, Y. 2020.  Stealthy and efficient adversarial attacks against deep reinforcement learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 5883–5891.
  * Wang et al. (2022) Wang, F.; Zhang, C.; Xu, P.; and Ruan, W. 2022.  Deep learning and its adversarial robustness: A brief introduction.  In _HANDBOOK ON COMPUTER LEARNING AND INTELLIGENCE: Volume 2: Deep Learning, Intelligent Control and Evolutionary Computation_ , 547–584.
  * Weng et al. (2019) Weng, T.-W.; Dvijotham, K. D.; Uesato, J.; Xiao, K.; Gowal, S.; Stanforth, R.; and Kohli, P. 2019.  Toward evaluating robustness of deep reinforcement learning with continuous control.  In _International Conference on Learning Representations_.
  * Yang et al. (2021a) Yang, R.; Fang, M.; Han, L.; Du, Y.; Luo, F.; and Li, X. 2021a.  MHER: Model-based hindsight experience replay.  _arXiv preprint arXiv:2107.00306_.
  * Yang et al. (2021b) Yang, R.; Lu, Y.; Li, W.; Sun, H.; Fang, M.; Du, Y.; Li, X.; Han, L.; and Zhang, C. 2021b.  Rethinking Goal-Conditioned Supervised Learning and Its Connection to Offline RL.  In _International Conference on Learning Representations_.
  * Zang, Li, and Wang (2022) Zang, H.; Li, X.; and Wang, M. 2022.  SimSR: Simple Distance-based State Representation for Deep Reinforcement Learning.  arXiv:2112.15303.
  * Zhang et al. (2020a) Zhang, A.; McAllister, R.; Calandra, R.; Gal, Y.; and Levine, S. 2020a.  Learning invariant representations for reinforcement learning without reconstruction.  _arXiv preprint arXiv:2006.10742_.
  * Zhang, Ruan, and Xu (2023) Zhang, C.; Ruan, W.; and Xu, P. 2023.  Reachability Analysis of Neural Network Control Systems.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’23)_.
  * Zhang et al. (2020b) Zhang, H.; Chen, H.; Xiao, C.; Li, B.; Liu, M.; Boning, D.; and Hsieh, C.-J. 2020b.  Robust deep reinforcement learning against adversarial perturbations on state observations.  _Advances in Neural Information Processing Systems_ , 33: 21024–21037.

## Appendix

In this section, we offer additional experimental details that are not covered
in the Experiments section of the main manuscript. This encompasses:

  * •

Implementation details for DDPG, GCSL, and GoFar in GCRL.

  * •

Hyperparameters of training algorithms in GCRL.

  * •

Additional SCR attack results for DDPG, GCSL, and GoFar.

  * •

Defensive performance of ARTs on DDPG, GoFar, DDPG (SimSR) and GoFar (SimSR)
against SCR-FGSM and SCR-PGD.

  * •

Other definitions for SA and SCR attacks.

### Baseline Implementations

DDPG We utilize an open-source implementation of DDPG that has been fine-tuned
for the set of Fetch tasks. The objective of the critic network is:

$\displaystyle\underset{Q}{\mathrm{min}}\mathbb{E}_{(s_{t},a_{t},s_{t+1},g)\sim\mathcal{D}}[(r(s_{t},g)+\gamma\bar{Q}(s_{t+},\pi(s_{t+1},g),g)-Q(s_{t},a_{t},g))^{2}],$
(14)

where $\bar{Q}$ is the targeted Q network. The objective of the policy network
is:

$\displaystyle\underset{\pi}{\mathrm{min}}-\mathbb{E}_{(s_{t},a_{t},s_{t+1},g)\sim\mathcal{D}}[Q(s_{t},\pi(s_{t},g),g)].$
(15)

GCSL GCSL is a straightforward supervised regression method that employs
behaviour cloning from a hindsight relabelling dataset. We execute the
implementation of GCSL by excluding the DDPG critic component and altering the
policy loss to adhere to maximum likelihood.

$\displaystyle\underset{\pi}{\mathrm{min}}-\mathbb{E}_{(s_{t},a_{t},g)\sim\mathcal{D}}[Q(s_{t},\pi(s_{t},g),g)].$
(16)

GoFar GoFAR benefits from uninterleaved optimization for its value and policy
networks, eliminating the need for hindsight relabeling. They use a
discriminator to estimate the reward as below:

$\displaystyle\mathcal{L}_{c}(\psi)=\frac{1}{N}\sum_{i=1}^{N}[\mathrm{log}(1-c_{\psi}(s_{d}^{j},g^{j}))+\frac{1}{M}\sum_{j=1}^{M}[\mathrm{log}c_{\psi}(s_{d}^{j},g^{j})]],$
(17)

The value function is trained as follows:

$\displaystyle\mathcal{L}_{V}(\theta)=\frac{1-\gamma}{M}\sum_{i=1}^{M}[V_{\theta}(s_{0}^{i},g_{0}^{i})]+\frac{1}{N}\sum_{i=1}^{N}[f_{*}(R_{t}^{i}+\gamma
V(s_{t+1}^{i},g_{t}^{i})-V(s_{t}^{i},g_{t}^{i}))],$ (18)

Once the optimal $V^{*}$ has been obtained, the policy is learned via:

$\displaystyle\mathcal{L}_{\pi}(\phi)=\frac{1}{N}[f_{*}^{\prime}(R_{t}^{i}+\gamma
V_{\theta}(s_{t+1}^{i},g_{t}^{i})-V_{\theta}(s_{t}^{i},g_{t}^{i}))\mathrm{log}\pi(a|s,g)],$
(19)

All hyperparameters are kept the same as the original work, and the detailed
algorithm can be found in Algorithm 2 in (Ma et al. 2022).

### Hyperparameters

In this work, DDPG, GCSL, and GoFar share the same network architectures. For
all experiments, We train each method for 5 seeds, and each training run uses
100 epochs, containing a minibatch with size 512. The hyperparameters of our
backbone architectures and algorithms are reported in Tab. 4.

| Hyperparameter | Value
---|---|---
Training | Optimizer | Adam
| Policy learning rate | $1\times 10^{-3}$
| Actor learning rate | $1\times 10^{-3}$
| Number of epochs | 100
| Number of cycles | 20
| Buffer size | 2$\times 10^{6}$
| Discount factor | 0.98
| Number of Tests | 10
| Critic hidden dim | 256
| Critic hidden layers | 3
| Critic activation function | ReLU
| Actor hidden dim | 256
| Actor hidden layers | 3

Table 4: Hyperparameters for base methods in GCRL.

Task | Method | Add on $s$ and $g$ | Add on $g$
---|---|---|---
SCR-FGSM | SCR-PGD | SCR-FGSM | SCR-PGD
state | goal | state+goal | state | goal | state+goal | state | goal | state+goal | state | goal | state+goal
FetchPick | DDPG | 11.18$\pm$2.74 | 14.79$\pm$1.39 | 11.39$\pm$3.41 | 10.84$\pm$3.59 | 14.97$\pm$1.56 | 10.16$\pm$2.68 | 15.70$\pm$3.87 | 16.94$\pm$2.87 | 16.76$\pm$2.65 | 15.11$\pm$2.45 | 16.38$\pm$2.72 | 13.31$\pm$2.94
GCSL | 8.06$\pm$2.64 | 11.14$\pm$1.92 | 6.10$\pm$2.43 | 8.18$\pm$3.10 | 11.30$\pm$1.93 | 6.12$\pm$3.31 | 10.85$\pm$1.23 | 11.66$\pm$1.95 | 9.73$\pm$2.09 | 11.17$\pm$2.05 | 11.72$\pm$1.81 | 9.48$\pm$2.45
GoFar | 17.14$\pm$2.89 | 20.98$\pm$2.06 | 16.23$\pm$2.40 | 17.83$\pm$2.06 | 21.01$\pm$2.05 | 14.77$\pm$2.45 | 20.63$\pm$2.19 | 19.78$\pm$1.94 | 14.79$\pm$3.59 | 21.03$\pm$2.03 | 19.84$\pm$1.88 | 15.40$\pm$2.91
FetchPush | DDPG | 6.22$\pm$2.06 | 11.50$\pm$2.50 | 4.64$\pm$2.20 | 9.79$\pm$2.15 | 12.80$\pm$4.61 | 6.08$\pm$4.04 | 12.72$\pm$4.07 | 12.57$\pm$4.19 | 13.41$\pm$2.40 | 11.72$\pm$2.56 | 13.61$\pm$2.56 | 8.10$\pm$2.67
GCSL | 5.86$\pm$1.53 | 12.63$\pm$2.02 | 5.48$\pm$2.90 | 6.20$\pm$2.11 | 12.73$\pm$1.39 | 6.39$\pm$2.71 | 12.95$\pm$2.06 | 13.37$\pm$0.90 | 11.36$\pm$3.01 | 12.90$\pm$2.50 | 13.56$\pm$1.49 | 10.09$\pm$3.29
GoFar | 11.66$\pm$3.93 | 18.32$\pm$3.77 | 10.85$\pm$4.41 | 14.55$\pm$2.78 | 18.38$\pm$3.24 | 11.81$\pm$3.67 | 16.65$\pm$1.96 | 16.33$\pm$3.17 | 10.95$\pm$2.80 | 18.34$\pm$3.99 | 18.83$\pm$1.39 | 10.62$\pm$1.60
FetchReach | DDPG | 27.42$\pm$1.32 | 28.28$\pm$0.70 | 27.33$\pm$1.43 | 27.02$\pm$2.20 | 29.91$\pm$0.20 | 24.70$\pm$4.07 | 28.48$\pm$0.55 | 10.40$\pm$8.64 | 21.28$\pm$7.45 | 29.91$\pm$0.20 | 11.03$\pm$8.47 | 6.07$\pm$5.32
GCSL | 20.97$\pm$1.31 | 22.14$\pm$0.76 | 20.60$\pm$1.26 | 21.18$\pm$0.95 | 22.04$\pm$0.68 | 20.81$\pm$0.95 | 22.94$\pm$0.75 | 22.22$\pm$0.79 | 20.60$\pm$1.26 | 22.04$\pm$0.68 | 22.22$\pm$0.79 | 20.81$\pm$0.95
GoFar | 27.51$\pm$0.45 | 27.66$\pm$0.95 | 25.91$\pm$2.02 | 27.49$\pm$0.94 | 27.84$\pm$0.63 | 26.05$\pm$1.85 | 27.65$\pm$0.80 | 27.85$\pm$0.34 | 22.57$\pm$4.15 | 27.83$\pm$0.65 | 27.87$\pm$0.33 | 24.18$\pm$2.43
FetchSlide | DDPG | 0.00$\pm$0.00 | 0.22$\pm$0.32 | 0.00$\pm$0.00 | 0.00$\pm$0.00 | 0.56$\pm$1.27 | 0.00$\pm$0.00 | 0.91$\pm$0.95 | 0.60$\pm$0.55 | 0.13$\pm$0.31 | 0.08$\pm$0.18 | 0.38$\pm$0.60 | 0.84$\pm$0.68
GCSL | 0.28$\pm$0.63 | 1.57$\pm$1.06 | 0.33$\pm$0.74 | 0.00$\pm$0.00 | 1.43$\pm$0.62 | 0.39$\pm$0.89 | 1.96$\pm$0.98 | 0.87$\pm$0.52 | 1.40$\pm$1.38 | 1.33$\pm$0.67 | 0.96$\pm$1.26 | 0.62$\pm$0.80
GoFar | 0.09$\pm$0.21 | 2.10$\pm$1.38 | 0.25$\pm$0.55 | 0.00$\pm$0.00 | 2.55$\pm$1.37 | 0.16$\pm$0.22 | 2.36$\pm$1.46 | 1.73$\pm$0.74 | 1.18$\pm$1.17 | 2.68$\pm$1.01 | 1.86$\pm$0.73 | 0.64$\pm$0.73

Table 5: Comparison of discounted returns for DDPG, GCSL, and GoFar against
SCR-FGSM and SCR-PGD. The left sub-table shows the attack results added on
both $s$ and $g$, while the right sub-table shows the attack results added on
$g$. The experiments are averaged over 5 random seeds.

### Task Details

Each of these 4 tasks has continuous state, action, and goal space. The
maximum episode horizon is defined as 50. The environments are built upon the
7-degree-of-freedom (DoF) Fetch robotics arm, equipped with a two-fingered
parallel gripper. In this section, we provide a detailed introduction to these
tasks.

FetchPick In this task, the objective is to grasp an object and transport it
to a designated target location, which is randomly selected within the space
(e.g., on the table or above it). The goal is represented by three dimensions,
specifying the desired object position.

FetchPush In this task, a block is positioned in front of the robot. The
robot’s actions are limited to pushing or rolling the object to a specified
target location. The goal is described by three dimensions, indicating the
desired position of the object. The state space encompasses 25 dimensions,
encompassing attributes such as the gripper’s position, and linear velocities,
as well as the position, rotation, and linear and angular velocities of the
box.

FetchReach Within this setting, a 7 DoF robotic arm is tasked with accurately
reaching a designated location using its two-fingered gripper. The state space
encompasses 10 dimensions, including attributes like the gripper’s position
and linear velocities. Correspondingly, the action space is composed of 4
dimensions, representing the gripper’s movements as well as its open/close
status.

FetchSlide The FetchSlide environment is similar to FetchPush, involving the
control of a 7 DoF robotic arm to slide a box towards a designated location.
The key distinction lies in the fact that the target position lies beyond the
robot’s immediate reach. Therefore, the robot is required to strike the block
to initiate its sliding motion towards the target position.

### Additional Results of the SCR Attack

Besides the previous attack results shown in Tab. 1 and Tab. 2, we conduct
extra experiments in this section. As illustrated in Tab. 5, we evaluate the
attack results derived from perturbations on both states and goals and solely
on goals.

Env | $\epsilon$ | Method | Nature | Attack Return
---|---|---|---|---
Uniform | SA-FGSM | SA-PGD | SCR-FGSM | SCR-PGD
FetchPick | 0.1 | DDPG | 14.82$\pm$1.53 | 16.68$\pm$4.33 | 19.61$\pm$2.84 | 19.92$\pm$2.32 |  7.83$\pm$3.06 | 10.49$\pm$2.86
DDPG+SCAA | 19.41$\pm$2.59 | 18.75$\pm$2.65 | 20.84$\pm$1.47 | 19.73$\pm$1.93 | 15.85$\pm$2.65 |  14.53$\pm$3.08
DDPG(SimSR) | 16.78$\pm$1.39 | 16.00$\pm$1.68 | 19.51$\pm$1.05 | 17.92$\pm$1.65 |  15.74$\pm$0.65 | 16.15$\pm$3.49
DDPG(SimSR)+SAR | 16.78$\pm$1.39 | 18.16$\pm$0.87 | 19.51$\pm$1.05 | 18.92$\pm$0.94 |  16.78$\pm$1.39 |  16.78$\pm$1.39
GoFar | 21.01$\pm$2.05 | 19.91$\pm$2.39 | 20.07$\pm$2.13 | 18.92$\pm$1.80 | 16.71$\pm$3.26 |  15.65$\pm$1.83
GoFar+SCAA | 19.83$\pm$3.94 | 18.99$\pm$3.57 | 17.89$\pm$4.13 | 16.24$\pm$3.86 | 16.82$\pm$3.46 |  14.72$\pm$4.40
GoFar(SimSR) | 17.44$\pm$4.40 | 16.73$\pm$3.81 | 17.63$\pm$3.61 |  13.80$\pm$4.40 | 16.27$\pm$2.75 | 16.11$\pm$3.49
GoFar(SimSR)+SAR | 17.44$\pm$2.95 | 17.41$\pm$3.37 | 17.54$\pm$4.23 |  14.82$\pm$3.04 | 17.44$\pm$2.95 | 17.44$\pm$2.95
FetchPush | 0.1 | DDPG | 12.81$\pm$4.61 | 13.61$\pm$4.81 | 17.24$\pm$2.12 | 15.42$\pm$3.38 | 6.43$\pm$2.33 |  5.42$\pm$3.00
DDPG+SCAA | 12.21$\pm$4.16 | 13.28$\pm$3.72 | 19.84$\pm$2.50 | 17.73$\pm$2.77 |  10.79$\pm$4.55 | 14.36$\pm$2.90
DDPG(SimSR) | 12.63$\pm$3.89 | 15.43$\pm$5.36 | 16.24$\pm$2.42 | 14.61$\pm$4.41 | 10.70$\pm$4.55 |  10.51$\pm$5.79
DDPG(SimSR)+SAR | 10.11$\pm$2.28 | 13.69$\pm$6.81 | 16.20$\pm$2.80 | 14.04$\pm$5.44 |  12.63$\pm$3.89 | 12.63$\pm$3.89
GoFar | 18.38$\pm$3.24 | 16.11$\pm$2.39 | 15.45$\pm$2.71 | 13.66$\pm$3.97 |  8.19$\pm$3.99 | 10.57$\pm$3.93
GoFar+SCAA | 15.13$\pm$3.23 | 16.03$\pm$2.44 | 16.83$\pm$2.59 | 16.37$\pm$2.87 | 8.26$\pm$3.66 |  8.00$\pm$3.76
GoFar(SimSR) | 14.19$\pm$4.49 | 13.13$\pm$4.68 | 15.52$\pm$3.88 | 13.91$\pm$4.49 |  12.44$\pm$3.80 | 13.08$\pm$4.08
GoFar(SimSR)+SAR | 14.19$\pm$4.49 |  12.44$\pm$2.39 | 16.08$\pm$2.36 | 14.78$\pm$2.23 | 14.19$\pm$4.49 | 14.19$\pm$4.49
FetchReach | 0.1 | DDPG | 29.92$\pm$0.20 | 28.86$\pm$0.46 | 24.21$\pm$2.21 | 20.14$\pm$5.04 | 2.02$\pm$3.32 |  1.86$\pm$3.31
DDPG+SCAA | 29.88$\pm$0.14 | 29.08$\pm$0.31 | 28.85$\pm$0.39 | 29.36$\pm$0.21 |  27.68$\pm$1.30 | 27.96$\pm$0.91
DDPG(SimSR) | 29.90$\pm$0.18 | 29.46$\pm$0.40 | 28.68$\pm$0.97 | 26.76$\pm$0.98 | 27.72$\pm$2.35 |  27.16$\pm$2.75
DDPG(SimSR)+SAR | 29.91$\pm$0.20 | 29.29$\pm$0.43 | 28.47$\pm$0.97 |  27.53$\pm$1.70 | 29.91$\pm$0.20 | 29.91$\pm$0.20
GoFar | 27.84$\pm$0.63 | 27.57$\pm$0.92 | 27.22$\pm$0.49 | 27.22$\pm$0.49 | 15.71$\pm$4.48 |  11.24$\pm$6.76
GoFar+SCAA | 27.91$\pm$0.36 | 27.72$\pm$0.27 | 27.93$\pm$0.35 | 27.93$\pm$0.35 | 15.70$\pm$5.03 |  13.99$\pm$6.46
GoFar(SimSR) | 27.93$\pm$0.34 |  27.91$\pm$0.34 | 27.95$\pm$0.33 | 27.96$\pm$0.32 | 27.96$\pm$0.29 | 27.98$\pm$0.33
GoFar(SimSR)+SAR | 27.93$\pm$0.34 |  27.84$\pm$0.32 | 27.96$\pm$0.37 | 27.98$\pm$0.36 | 27.93$\pm$0.34 | 27.93$\pm$0.34
FetchSlide | 0.1 | DDPG | 0.58$\pm$1.30 | 1.89$\pm$0.89 | 0.59$\pm$1.09 | 0.50$\pm$1.12 |  0.00$\pm$0.00 |  0.00$\pm$0.00
DDPG+SCAA | 0.74$\pm$1.57 | 1.54$\pm$1.21 | 1.08$\pm$1.02 | 0.94$\pm$1.36 |  0.00$\pm$0.43 | 0.18$\pm$0.85
DDPG(SimSR) | 0.62$\pm$0.54 | 0.83$\pm$0.77 | 0.80$\pm$0.52 | 0.43$\pm$0.90 |  0.37$\pm$0.60 | 0.84$\pm$0.71
DDPG(SimSR)+SAR | 0.62$\pm$0.54 | 0.61$\pm$0.73 |  0.55$\pm$0.62 | 0.63$\pm$1.02 | 0.62$\pm$0.54 | 0.62$\pm$0.54
GoFar | 2.55$\pm$1.37 | 1.23$\pm$1.14 | 0.77$\pm$0.92 | 1.13$\pm$0.72 |  0.00$\pm$0.00 |  0.00$\pm$0.00
GoFar+SCAA | 1.12$\pm$0.95 | 1.13$\pm$0.91 | 0.90$\pm$0.79 | 0.86$\pm$0.84 | 0.27$\pm$0.61 |  0.00$\pm$0.00
GoFar(SimSR) | 1.66$\pm$0.75 | 1.78$\pm$0.78 | 2.31$\pm$1.36 | 1.41$\pm$0.92 | 1.07$\pm$1.13 |  0.59$\pm$0.66
GoFar(SimSR)+SAR | 1.66$\pm$0.75 | 1.95$\pm$1.30 |  1.47$\pm$0.79 | 2.25$\pm$1.29 | 1.66$\pm$0.75 | 1.66$\pm$0.75

Table 6: Defensive performance of ARTs on DDPG, GoFar, DDPG (SimSR) and GoFar
(SimSR) against Uniform, SA-FGSM, SA-PGD, SCR-FGSM, and SCR-PGD attacks. The
experiments are averaged over 5 random seeds. Bold indicates the best attack
result in a row.

### Full Results for Defensive Performance of ARTs

Building upon the results presented in Tab. 6, we provide a comprehensive
analysis of the defensive performance of DDPG, GoFar, and their ARTs-enhanced
counterparts. The experimental setup is the same as the Experiments section.
Details are illustrated in Tab. 6.

FetchPick: Even though all algorithms undergo training with perturbations
introduced by the SCR attack, we observed a boost in defensive performance
against SA-FGSM for DDPG and DDPG (SimSR). A similar trend is evident against
SA-PGD for DDPG (SimSR) and GoFar (SimSR). When defending against SCR-FGSM,
the performance improvements are 102.43%, 6.60%, 0.66%, and 7.19% for DDPG,
DDPG (SimSR), GoFar, and GoFar (SimSR) respectively. For SCR-PGD, the
increments were 38.51%, 3.90%, and 8.26% for DDPG, DDPG (SimSR), and GoFar
(SimSR) respectively.

FetchPush: We achieve the performance enhancements when defending against SA-
FGSM for DDPG, GoFar, and GoFar (SimSR). Similarly, improvements are observed
against SA-PGD for DDPG, GoFar, and GoFar (SimSR). In the context of SCR-FGSM,
the increments are 67.81%, 18.04%, 0.85%, and 14.07% for DDPG, DDPG (SimSR),
GoFar, and GoFar (SimSR) respectively. For SCR-PGD, the gains were 164.94%,
20.17%, and 8.49% for DDPG, DDPG (SimSR), and GoFar (SimSR) respectively.

FetchReach: The defence against SA-FGSM sees improvements for DDPG, GoFar, and
GoFar (SimSR). Similarly, the performance against SA-PGD also rises for DDPG,
GoFar, and GoFar (SimSR). For SCR-FGSM, the gains are 1270.30% and 7.90% for
DDPG and DDPG (SimSR), respectively. Against SCR-PGD, the increments are
1403.23%, 10.13%, and 24.47% for DDPG, DDPG (SimSR), and GoFar, respectively.

FetchSlide: Defense improvements against SA-FGSM are observed for DDPG and
GoFar, while for SA-PGD, the gains were seen for DDPG, DDPG (SimSR), and GoFar
(SimSR). Remarkably, for SCR-FGSM, the improvements were 67.57% and 55.14% for
DDPG (SimSR) and GoFar (SimSR) respectively. Against SCR-PGD, the gains were
181.36% for GoFar (SimSR).

### Others

To streamline the main text, we reserve some definitions in this section.

######  Definition 5 (FGSM-based Approximation)

Given an input tuple $\left<s,g\right>$, attack radius $\epsilon$, and loss
function $\mathcal{L}$, the FGSM-based adversarial state and goal can be
individually denoted as:

$\displaystyle\begin{split}&\mathcal{V}(s)=s+\epsilon\nabla_{s}\mathcal{L}(\left<s,g\right>)\\\
&\mathcal{V}(g)=g+\epsilon\nabla_{g}\mathcal{L}(\left<s,g\right>)\end{split}$
(20)

Different from Def. 3, Def. 5 provides a single-step attack on
$\left<s,g\right>$, where $\mathcal{V}(s)$ generated by FGSM is equivalent to
$\mathcal{V}^{\mathcal{I}}(s)$ generated by PGD, $\mathcal{V}(g)$ generated by
FGSM is equivalent to $\mathcal{V}^{\mathcal{I}}(g)$ generated by PGD. Similar
to Def. 3, $\mathcal{I}$ is the number of iteration steps.

######  Definition 6 (SA attack)

Generally, given an input tuple $\left<s,g\right>$ and an action $a$, the SA
attack can be defined as follows:

$\mathop{\arg\sup}_{\mathcal{V}(s)\in\mathcal{B}_{p}^{\epsilon}(s),\mathcal{V}(g)\in\mathcal{B}_{p}^{\epsilon}(g)}\left[-Q(\left<s,g\right>,a)\right]$
(21)

Then PGD and FGSM variants of Def. 6 can be analogously represented following
Def. 3 and Def. 5.