# Safety Guaranteed Manipulation Based on Reinforcement Learning Planner and
Model Predictive Control Actor

Zhenshan Bing1, Aleksandr Mavrichev1, Sicong Shen1, Xiangtong Yao1, Kejia
Chen1,
Kai Huang2, and Alois Knoll1 1Z. Bing, A. Mavrichev, S. Shen, X. Yao, K. Chen,
and A. Knoll are with the Department of Informatics, Technical University of
Munich, Germany. 2K. Huang is with the School of Data and Computer Science,
Sun Yat-sen University, China.

###### Abstract

Deep reinforcement learning (RL) has been endowed with high expectations in
tackling challenging manipulation tasks in an autonomous and self-directed
fashion. Despite the significant strides made in the development of
reinforcement learning, the practical deployment of this paradigm is hindered
by at least two barriers, namely, the engineering of a reward function and
ensuring the safety guaranty of learning-based controllers. In this paper, we
address these challenging limitations by proposing a framework that merges a
reinforcement learning planner that is trained using sparse rewards with a
model predictive controller (MPC) actor, thereby offering a safe policy. On
the one hand, the RL planner learns from sparse rewards by selecting
intermediate goals that are easy to achieve in the short term and promising to
lead to target goals in the long term. On the other hand, the MPC actor takes
the suggested intermediate goals from the RL planner as the input and predicts
how the robot’s action will enable it to reach that goal while avoiding any
obstacles over a short period of time. We evaluated our method on four
challenging manipulation tasks with dynamic obstacles and the results
demonstrate that, by leveraging the complementary strengths of these two
components, the agent can solve manipulation tasks in complex, dynamic
environments safely with a $100\%$ success rate. Videos are available at
https://videoviewsite.wixsite.com/mpc-hgg.

##  I Introduction

Deep reinforcement learning (RL) has been widely used to solve complex
decision-making tasks in robotics, such as controlling robotic arms to perform
manipulation tasks [1], generating agile locomotion gaits for legged robots
[2], and planning trajectories for autonomous vehicles [3]. As there is no
modeling computation or optimization involved, RL-based methods are superior
to traditional control methods in solving long-horizon planning and dynamic
tasks in a timely manner. However, RL methods are constantly facing two major
challenges, namely, requiring handcrafted reward functions that are tailored
to individual tasks and lacking rigorous guarantee to ensure the safety of
operations.

For the first challenge, in most complex robotic tasks, where a concrete
representation of efficient or even admissible behavior is unknown, it is
extremely difficult and time-consuming to design an adequate task tailored
reward, thereby making this strategy impractical for wide robotic applications
of RL. One promising concept to address the reward engineering problem is to
use a binary reward to simply indicate the completion of the task based on its
success or failure condition. This kind of binary reward is also known as a
sparse reward and is easy to derive from task definition with minimum effort.

Learn “unsafe” long-horizon RL planner in a simulated scene with dynamic
obstacles RL planner intermediate goals action target MPC actor target
evaluation solution $U_{t}$ last iteration safe action Execute “safe” actions
in a real-world scene with dynamic obstacles Long horizon planning Short
horizon planning Figure 1: Overview of the proposed MPC-HGG framework. The RL
planner performs long horizon planning by proposing intermediate goals. The
MPC actor performs short horizon planning to ensure the safety of the action
by avoiding dynamic obstacles.

Although a sparse reward is easy to derive from task definition with minimum
effort, RL algorithms that support sparse rewards usually suffer from bad
learning efficiency. This is because the sparse reward only delivers shallow
and insufficient information during training, which can limit the performance
of the RL algorithm. Hindsight experience replay (HER) [4] is one of
fundamental methods which improves the success of off-policy RL algorithms in
multigoal RL problems with sparse rewards. The core idea behind HER is to
train an agent using handcrafted, easy-to-achieve intermediate goals, and then
gradually increase the difficulty of the goals. To achieve this, HER
constructs hindsight goals from previously achieved states, replays known
trajectories with these hindsight goals, and uses the results to train a goal-
dependent value function. An extremely useful extension to HER is hindsight
goal generation (HGG)algorithm, which can generate more meaningful hindsight
goals in the direction of the desired goal and accelerate the learning of
distant targets [5]. However, HGG shows poor performance in the presence of
obstacles since distance norm measure for the hindsight goals distribution
does not account for the occupied area.

For the second challenge, learning methods are always criticized by their
indistinct interpretability and suffering its adaptability to environments
with uncertainty. Take manipulation tasks for example, although robotic arms
are mostly operated in a relatively enclosed space, the unpredictable movement
of human operators or a sudden invasion of unknown objects will pose great
safety threat to the completion of the task or even for life and property. As
an effective control approach, the model predictive control (MPC)is
extensively used for path planning and collision avoidance [6, 7]. Model
predictive control requires a dynamic model of the environment to predict the
future states and optimise control over a finite time horizon. This means that
MPC provides online trajectory optimization with the ability to rapidly react
to changes in the environment.

To overcome these challenges, we propose a control framework that combines the
advantages of RL-based methods in long-horizon planning and traditional model-
based control methods in safety guaranteed performance (see Figure 1).
Specifically, the HGG algorithm is used as the high-level planner to propose
intermediate goals for solving a long-horizon task. While the MPC algorithm is
used as an actor to execute the sub-task that takes the intermediate goal
proposed by the planner as the input. We show that the combination of the both
methods can guarantee collision avoidance in the randomized dynamic
environments. Our results demonstrate how the MPC actor can effectively
address the issue of unreliable safety in a RL policy and how the RL policy
can guide MPC through the infinite horizon.

Our contributions to the literature are summarized as follows. First, we
propose an RL planner that utilizes the hindsight goal generation algorithm to
generate intermediate goals for long-term manipulation tasks. We enhance the
collision avoidance capability of the HGG algorithm by introducing a multi-
objective sparse reward concept. This reward function incentivizes not only
reaching the goal, but also avoiding any obstacle collision, all while
minimizing the required engineering effort. Second, we formulate an MPC actor
that can optimize the trajectory of a robotic arm to reach to the goal
proposed by the RL planner and avoid colliding with any obstacle over finite
horizon. Last, we show the proposed framework is able to solve complex
manipulation tasks effectively and safely in the simulation and the real
world. The controller solves all the tasks with a success rate of $100\%$ and
attains a real-time performance with less than $3$ ms per timestep.

##  II Related Works

This section aims to provide a brief overview of the literature that explores
the combination of model predictive control with reinforcement learning
algorithms.

A closely related work by [8] introduces the GO-MPC algorithm for autonomous
navigation in crowded scenarios. The authors propose a pre-trained on-policy
RL algorithm to provide long-term sub-goals to the local motion planner based
on MPC. Hansen et al. [9] incorporate MPC and twin delayed deep deterministic
policy gradient (TD3) to improve sample efficiency via model learning. The
authors jointly train a terminal value function and a latent dynamics model,
which are utilized for local trajectory optimization and global guidance,
respectively. By leveraging the latent space representation, the learned model
can reduce the state space by eliminating irrelevant features, such as
background noise and shading. To address the problem of high computational
costs over a long planning horizon, Negenborn et al. [10] propose to utilize
the learned value function in the cost function of a conventional MPC
controller.

Bhardwaj et al. [11] propose a novel approach to capitalize on the advantages
of both model-free and model-based methods. Specifically, they propose a Model
Predictive Control (MPC) based Q-Learning algorithm that utilizes local
optimization to improve the value function and demonstrate faster learning
speeds with fewer system interactions. This method provides a promising
alternative that combines the strengths of both model-based and model-free
methods for control tasks. Greatwood et al. [12] employ MPC to regulate a
quadrotor micro air vehicle (MAV) using guidance from a RL policy. The
approach is designed to operate under the assumption of only rectangular
obstacles and employs two distinct MPCs for each axis to effectively control
the MAV. Xue et al. [13] leverage MPC within the deep deterministic policy
gradient (DDPG) algorithm to predict the trajectory of dynamic obstacles. The
proposed approach employs a complex reward function that comprises target
attraction, obstacle repulsion, collision penalty, and reward for reaching the
target. This technique demonstrates promise in navigating dynamic environments
with obstacles.

Another interesting approach was introduced by [14], in which the authors
suggest learning a control policy directly in the real-world environment. The
approach utilizes an additional safe policy that can be triggered to return
the robot to a safe state during training. An approximated dynamics model is
then used to decide when to revert back to the learning policy to continue the
training. MPC is employed to achieve more stable operation due to the reduced
action space and to avoid direct control of motor torques. This methodology
offers a promising technique for training control policies in real-world
environments while ensuring safety.

##  III Preliminaries

###  III-A Goal-Conditioned RL

In goal-conditioned RL, an agent interacts with its environment to reach some
goals, which can be modeled as a goal-conditioned Markov decision process
(MDP) with a state space $\mathcal{S}$, an action space $\mathcal{A}$, a goal
space $\mathcal{G}$, a probabilistic transition function
$P:S\times\mathcal{A}\rightarrow\mathcal{S}$, a reward function
$r_{g}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and a discount
factor $\gamma$. The agent’s action $a_{t}$ is defined by a probabilistic
policy $\pi(s_{t}||g)$ at every time step $t$, given by the current state
$s_{t}$ and the goal $g$ (we use $||$ as a symbol for concatenation into
$\mathcal{S}\times\mathcal{G}$). The goal is to find a policy that can
maximize the expected curriculum reward starting from the initial state
sampled from the initial state distribution $s\in S_{0}$, which is defined as

$\begin{split}V^{\pi}(s||g)=\mathbb{E}_{s_{0},a_{t}\sim\pi(s_{t}||g),\,s_{t+1}\sim
P(s_{t},a_{t})}\big{[}\sum_{t=0}^{\infty}\gamma^{t}r_{g}(s_{t},a_{t})\big{]}\text{.}\end{split}$
(1)

###  III-B Hindsight Experience Replay

Hindsight Experience Replay (HER [4]) is an RL algorithm specifically
engineered for goal-oriented tasks characterized by sparse rewards, which are
often difficult for agents to learn efficiently. Despite its simplicity, HER
has been demonstrated to be a highly effective method for improving agent
performance in these challenging scenarios. HER is designed to enhance
learning efficiency by using a relabeling approach that exploits the idea that
experiences that are uninformative for a given goal may still contain valuable
information for other goals. HER assumes that, in a multi-goal RL task with
sparse rewards, each goal $g$ is associated with a predicate
$f_{g}:\mathcal{S}\rightarrow\\{0,1\\}$. Once the agent reaches a state $s$
that satisfies $f_{g}(s)=1$, it is considered that the goal has been achieved.
The reward function is defined as sparse if it satisfies
$r_{g}(s,a)=-[f_{g}(s)=0]$. This implies that until the agent reaches the
goal, it continuously receives negative rewards. In HER, each transition
$(s_{t}||g,a_{t},r_{t},s_{t+1}||g)$ is not only stored with the original
episode goal $g$, but also with a subset of hindsight goals $g^{\prime}$ as
$(s_{t}||g^{\prime},a_{t},r_{t},s_{t+1}||g^{\prime})$. As a result, when
replaying the resulting transitions
$(s_{t}||g^{\prime},a_{t},r_{t},s_{t+1}||g^{\prime})$, the agent is more
likely to encounter informative rewards. An interpretation of HER is that it
acts as an implicit curriculum, focusing initially on simpler intermediate
goals and subsequently progressing towards more challenging goals that are
nearer to the ultimate target goals.

##  IV Methodology

This section first gives an overview of our proposed algorithm MPC-HGG. Then
we explain the RL planner and the MPC actor in detail. Finally, we summarize
MPC-HGG with its pseudocode.

###  IV-A Overview

The overall architecture of the MPC-HGG algorithm is shown in Figure 1. The
algorithm is briefly explained in two phases as follows.

  1. 1.

In the first stage, we design a RL planner that can solve complex, long-
horizon planning manipulation tasks via a curriculum learning approach. This
RL controller can adapt itself to multi-goal tasks, but is not able to
guarantee the safety of the proposed action.

  2. 2.

In the second stage, we develop a MPC actor that can provide safe actions to
reach to an intermediate goal in a short planning horizon. The intermediate
goals are suggested by the RL planner from stage one.

###  IV-B RL Planner

Inspired by HER [4] and HGG [5], we train the RL planner in the following
fashion. The episode starts with sampling an initial state - goal pair
$(s_{0},g)$ from an initial state distribution $\mathcal{S}_{0}$ and a target
goal distribution $\mathcal{G}_{T}$. A state $s$ can be mapped to a goal
$g_{s}$ by $g_{s}=m(s)$. In the beginning of the learning stage, the
exploration is random since no meaningful policy has been developed yet.
Exploration naturally starts from $s_{0}\sim\mathcal{S}_{0}$, thus goals that
are close to $m(s_{0})$ are reached more easily. The agent can leverage the
generalization capabilities inherent in neural networks, enabling it to
extrapolate from past experiences and extend its ability to achieve goals
similar to those previously reached. The idea is illustrated in Figure 2.

The distribution $\mathcal{T}^{*}:\mathcal{G}\times\mathcal{S}\to\mathbb{R}$
determines how they are sampled. Instead of optimizing the value function
$V^{\pi}$ with difficult target goals, which carries the risk of being too far
from the known goals, we try to optimize with a set of intermediate goals
sampled from $\mathcal{T}$. On the one hand, the goals contained in
$\mathcal{T}$ should be easy to reach, which requires a high
$V^{\pi}(\mathcal{T})$. On the other hand, goals in $\mathcal{T}$ should be
close enough to $\mathcal{T^{*}}$ to be challenging for the agent.

Inspired by HGG [5], a guided schedule for selecting suitable intermediate
goals $g^{\prime}\in\mathcal{G}$ that will be used by the agent instead of
$g\in\mathcal{G}_{T}$. This will guide the agent to learn from easy to
difficult, so it can learn to reach the goals from $\mathcal{G}_{T}$
gradually. Therefore, it is necessary to find a substitute distribution
$\mathcal{T}:\mathcal{G}\times\mathcal{S}\to\mathbb{R}$ which chooses
appropriate intermediate goals. On the one hand, such goals must be close to
goals that the agent can already reach, and on the other hand, they still have
some distances to achieved goals so that the agent learns something new to
approach the final goal. This trade-off can be formalized as

$\max_{\mathcal{T},\pi}V^{\pi}(\mathcal{T})-L\cdot\mathcal{D}(\mathcal{T^{*}},\mathcal{T})\text{.}$
(2)

The Lipschitz constant $L$ is treated as a hyper-parameter. In practice, to
select these goals, we first approximate $\mathcal{T}^{*}$ by taking $K$
samples from $\mathcal{T}^{*}$ and storing them in $\hat{\mathcal{T}}^{*}$.
Then, for an initial state and goal
$(\hat{s}_{0}^{i},\hat{g}^{i})\in\hat{\mathcal{T}}^{*}$, we select a
trajectory $\tau=\\{s_{t}\\}_{t=1}^{T}$ that minimizes the following function:

$\displaystyle w(\hat{s}_{0}^{i},\hat{g}^{i},\tau):={}$ $\displaystyle
c\|m(\hat{s}_{0}^{i})-m(s_{0})\|$ (3)
$\displaystyle+\min_{s_{t}\in\tau}\left(\|\hat{g}^{i}-m(s_{t})\|-\frac{1}{L}V^{\pi}((s_{0}||m(s_{t}))\right)\text{.}$

$c>0$ provides a trade-off between 1) the distance between target goals and 2)
the distance between the goal representation of the initial states. Finally,
from each of the $K$ selected trajectories $\tau^{i}$, the hindsight goal
$g^{i}$ is selected from the state $s_{t}^{i}\in\tau^{i}$, that minimized (3).
More formally,

$g^{i}:=\mathop{\text{arg
min}}_{s_{t}\in\tau}\left(\|\hat{g}^{i}-m(s_{t})\|-\frac{1}{L}V^{\pi}((s_{0}\|m(s_{t}))\right)\text{.}$
(4)

final goalinitial goalintermediate goalsguided learning with intermediate
goals Figure 2: Visualization of the intermediate goal distributions generated
by the hindsight goal generation.

Through experimentation, it was observed that the agent was able to achieve
the target goal despite encountering obstacles and colliding during task
execution, which should be punished as we expect the agent to complete the
task while avoiding collisions. Prior work addresses this issue by creating a
dense reward signal linked to specific obstacle measurements to ensure
collision-free movement. However, designing an well-designed reward tailored
to a specific task is challenging and demands time and effort.

We suggest a way to balance the advantages of using sparse rewards with the
need for task-oriented behavior. Our solution is a multi-objective conditioned
binary reward function that assigns different magnitudes to each objective.
Specifically, the multi-objective sparse reward is defined as

$r_{g}(s):=\begin{cases}\eta,&\text{if collision }\\\
0,&\text{if}\,f_{g}(s)=1\\\ -1,&\text{otherwise.}\end{cases}$ (5)

The reward $\eta$, as a hyperparameter, is a constant negative value that is
customized to prevent the collision. If the agent encounters an obstacle, it
will receive a reward of $\eta<-1$. However, if the agent does not collide
with any obstacles but still fails to reach the goal, it will receive a reward
of $-1$. The agent will only receive a reward of $0$ when it successfully
reaches the goal. The pseudocode of the HGG algorithm is shown in Algorithm 1.

Algorithm 1 Hindsight Goal Generation (HGG)

1:Given:

  * •

An off-policy RL algorithm $\mathbb{A}$,$\triangleright$ e.g. DDPG

  * •

A strategy $\mathbb{S}$ for sampling goals for replay, $\triangleright$ e.g.
$\mathbb{S}(s_{0},...,s_{T})=m(s_{T})$

  * •

A set of reward functions
$r_{g}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ $\triangleright$
e.g. $r_{g}(s,a)=-\mid f_{g}(s)==0\mid$

2:Initialize $\mathbb{A}$

3:Initialize replay buffer $R$

4:for $iteration$ do

5: Construct a set of $M$ intermediate tasks
$\\{(\hat{s}_{0}^{i},g^{i})\\}_{i=1}^{M}$: $\triangleright$ HGG  • Sample
target tasks
$\\{(\hat{s}_{0}^{i},\hat{g}^{i})\\}_{i=1}^{K}\sim\mathcal{T}^{*}$ • Find $K$
distinct trajectories $\\{\tau^{i}\\}_{i=1}^{K}$ that together minimize (3)
$\triangleright$ weighted bipartite matching • Find $M$ intermediate tasks
$(\hat{s}_{0}^{i},g^{i})$ by selecting and intermediate goal $g^{i}$ from each
$\tau^{i}$

6: for $episode=1,M$ do

7: $(s_{0},g)\leftarrow(\hat{s}_{0}^{i},g^{i})$ $\triangleright$ hindsight
goal-oriented exploration

8: for $t=0,T-1$ do

9: Sample an action $a_{t}$ using the policy from $\mathbb{A}$ with noise:
$a_{t}\leftarrow\pi(s_{t}\parallel g)+\mathcal{N}_{t}$ (6)

10: Execute action $a_{t}$ and observe new state $s_{t+1}$

11: end for

12: for $t=0,T-1$ do

13: $r_{t}:=r_{g}(s_{t},a_{t})$

14: Store transition $(s_{t}\parallel g,\;a_{t},\;r_{t},\;s_{t+1}\parallel g)$
$\triangleright$ DDPG experience replay

15: Sample a set of additional goals for replay
$G:=\mathbb{S}(current\,episode)$

16: for $g^{\prime}\in G$ do

17: $r^{\prime}:=r_{g^{\prime}}(s_{t},a_{t})$

18: Store the transition $(s_{t}\parallel
g^{\prime},\;a_{t},\;r^{\prime},\;s_{t+1}\parallel g^{\prime})$ in $R$
$\triangleright$ HER

19: end for

20: end for

21: end for

22: for $t=1,N$ do

23: Sample a minibatch $B$ from the replay buffer $R$ $\triangleright$ HER or
EBP

24: Perform one step of optimization using $\mathbb{A}$ and minibatch $B$
$\triangleright$ DDPG

25: end for

26:end for

###  IV-C MPC Actor

Model predictive control, which is model-based method for trajectory planning,
uses a dynamic model of the system to predict the future position of the agent
and optimizes the sequence of the control inputs to achieve lower costs.

####  IV-C1 Definition

Our proposed model is designed based on the idea that modern industrial robots
have optimization algorithms that can quickly convert a desired location into
specific torque rates for each joint of the robot arm. This allows for precise
movement of the arm to the desired location. As the action space used in the
RL planner is the gripper’s coordinates $[x,y,z]$ of the robotic arm, we use a
simple point-mass model for the MPC actor:

$\displaystyle\dot{x}=v_{x}\text{,}\quad\dot{v}_{x}=\dfrac{F_{x}}{m}$ (7)
$\displaystyle\dot{y}=v_{y}\text{,}\quad\dot{v}_{y}=\dfrac{F_{y}}{m}$
$\displaystyle\dot{z}=v_{z}\text{,}\quad\dot{v}_{z}=\dfrac{F_{z}}{m}$

We define the state vector as
$x=[x,y,z,v_{x},v_{y},v_{z}]\in\mathcal{X}=\mathbb{R}^{6}$ and the control
input vector as $u=[F_{x},F_{y},F_{z},\xi]\in\mathcal{U}=\mathbb{R}^{4}$.
$\xi$ is used to soften the hard constraints. $m$ is the mass of the
manipulable object.

####  IV-C2 Formulation

We define the problem as non-convex, finite-time nonlinear optimal control
with horizon length $N$ and takes the following form:

minimize $\displaystyle\sum_{k=1}^{N}f_{k}(z_{k},p_{k})$ cost function (8)
subject to $\displaystyle z_{1}(\mathcal{I})=z_{\mathrm{init}}$ initial
equality $\displaystyle\underline{z}_{k}\leq z_{k}\leq\bar{z}_{k}$ upper-lower
bounds $\displaystyle\underline{h}_{k}\leq h_{k}(z_{k},p_{k})\leq\bar{h}_{k}$
nonlinear constraints

where $z_{k}\in\mathcal{U}\times\mathcal{X}\in\mathcal{Z}=\mathbb{R}^{10}$ is
a stage variable which stacks the input and differential state variables
together. $p_{k}\in\mathcal{G}\times\mathcal{O}_{i,k},\>\forall
i\in\\{1,\dots,N_{o}\\}$ contains the real-time data, such as current goal and
obstacles. $f_{k}(z_{k},p_{k})$ represents the cost function, which should be
minimized during the optimization process, and $h_{k}(z_{k},p_{k})$ represents
the nonlinear constraint function used for collision avoidance. For each of
these variables, the respective inequalities must be satisfied during each
optimization step.

Safe-area around the
obstacle$obstacle$•$(x_{0},y_{0})$$(x,y)$$object$$w_{0}+w_{r}$$h_{0}+h_{r}$
Figure 3: Hard constraint definition for the rectangle-shaped obstacles in 2D.
Algorithm 2 MPC-HGG algorithm

1:Given: RL pre-trained policy $\pi_{\theta}$, observation $s_{0}$, horizon
$N$ and goal $g$

2:$u_{0}\leftarrow(0,0,0)$

3:$m\leftarrow 1$

4:for $t\leftarrow 0$ to $T-1$ do

5: $p_{t}\leftarrow getPos(s_{t})$

6: $v_{t}\leftarrow getVel(s_{t})$

7: $a_{t}\leftarrow\pi_{\theta}(s_{t}\>|\>g)$ $\triangleright$ Get RL action

8: $g_{t}\leftarrow p_{t}+a_{t}$ $\triangleright$ Convert action to a new
target

9: if $\lVert\mathbf{p_{t},g}\rVert_{2}\leq Nv_{\max}d_{t}$ then
$\triangleright$ Use MPC directly when goal is reachable

10: $g_{t}\leftarrow g$

11: end if

12: $z_{1}\leftarrow u_{t}\times p_{t}\times v_{t}$ $\triangleright$ Setup
first MPC state to start solver from

13: $p\leftarrow\\{g_{t}\times(o_{k}^{i})_{i=1}^{N_{o}}\\}_{k=1}^{N}$
$\triangleright$ Setup MPC parameters

14: $U_{t}\leftarrow minimize\sum_{k=1}^{N}f_{k}(z_{k},p_{k})$
$\triangleright$ Solve MPC

15: if $U_{t}$ is feasible then $\triangleright$ A feasible solution found

16: $u_{t}\leftarrow U_{t,1}$ $\triangleright$ Take first MPC action

17: $a_{t}^{\prime}\leftarrow getAction(u_{t},a_{t})$ $\triangleright$ Convert
MPC action to the MuJoCo action

18: else

19: $m\leftarrow m+1$

20: if $m<N$ then

21: $u_{t}\leftarrow U_{t-1,m}$ $\triangleright$ Try next prediction from the
previous solution

22: $a_{t}^{\prime}\leftarrow getAction(u_{t},a_{t})$

23: else

24: $a_{t}^{\prime}\leftarrow noAction(a_{t})$ $\triangleright$ Decelerate
robot

25: end if

26: end if

27: Perform $a_{t}^{\prime}$, get $s_{t+1}$ $\triangleright$ Perform action in
the simulator and get a new observation

28:end for

(a) DynamicSquareObstacles.

(b) DynamicMixedObstacles.

(c) DynamicRecObstacles.

(d) DynamicLiftedObstacles.

(e) DynamicSquareObstacles.

(f) DynamicMixedObstacles.

(g) DynamicRecObstacles.

(h) DynamicLiftedObstacles.

Figure 4: Robotic manipulation environments in the simulation and the real
world.

####  IV-C3 Cost function

The cost function $f_{k}(z_{k},p_{k})$ is formulated as follows:

$\displaystyle\begin{gathered}f_{k}(z_{k},p_{k})=\left\\{\begin{array}[]{lll}w_{1}\lVert\mathbf{s,g}\rVert_{2}^{2}+w_{5}\xi^{2}&\\\
+w_{2}F_{x}^{2}+w_{3}F_{y}^{2}+w_{4}F_{z}^{2}&\mbox{for}&k<N\\\\[5.0pt]
f_{k-1}(z_{k},p_{k})&\\\
+w_{6}v_{x}^{2}+w_{7}v_{y}^{2}+w_{8}v_{z}^{2}&\mbox{for}&k=N\end{array}\right.\\\
\end{gathered}$ (9)

, where we separate it to the stage cost and terminal cost. The stage cost
includes the Euclidean distance from the current position $\bm{s}$ to the
target goal $\bm{g}$ and penalization of the control commands to achieve the
smooth trajectory. For the terminal cost, the robot should reduce the velocity
to zero when reaching the goal. We denote $w_{i}$ as weights $\forall
i\in\\{1,\dots,8\\}$ which should be fine-tuned to prioritize the penalties.

####  IV-C4 Constraints

The constraints for avoiding the obstacle is defined in the following form:

$\displaystyle\begin{gathered}h_{rect}(s_{k},s_{o},dim_{o})=\frac{1}{2}max(\dfrac{|x-x_{o}|}{w_{o}+w_{r}},\dfrac{|y-y_{o}|}{h_{o}+h_{r}},\dfrac{|z-z_{o}|}{d_{o}+d_{r}})\\\
s_{k}=[x,y,z]\in\mathbb{R}^{3}\\\ s_{o}=[x_{o},y_{o},z_{o}]\in\mathcal{P}\\\
dim_{o}=[w_{o},h_{o},d_{o}]\in\mathcal{D}\\\ \end{gathered}$ (10)

, where $w_{r}$, $h_{r}$, $d_{r}$ are width, height and depth of the rectangle
describing the manipulated box together with the gripper. $w_{0}$, $h_{0}$,
$d_{0}$ are width, height and depth of the bounding box of the obstacle.
Figure 3 illustrates the proposed hard-constraint definition. Since the $\max$
operator is not differentiable and thus not suitable for the gradient descent
algorithms used in the MPC controller, we propose to approximate it using the
smooth maximum [15]:

$\displaystyle\begin{gathered}{\mathcal{S}}_{\alpha}(x_{1},\ldots,x_{n})={\frac{\sum_{i=1}^{n}x_{i}e^{\alpha
x_{i}}}{\sum_{i=1}^{n}e^{\alpha x_{i}}}}\\\
{\mathcal{S}}_{\alpha}\to\max\>\mbox{as}\>\alpha\to\infty\\\ \end{gathered}$
(11)

###  IV-D MPC-HGG Algorithm

The overall MPC-HGG algorithm is provided as Algorithm 2. First, we extract
the proposed RL action based on the current observation and convert it to a
new intermediate goal for the MPC in line 7. If the task can be solved within
the planning horizon, we provide the main goal directly to the MPC in line 10.
Note that we still need RL action to control the gripper. Once a sub-goal is
selected by the RL planner, the MPC can be formalized in line 13. We choose
the horizon length $N$ for the MPC actor, so that the robot is able to reach
the proposed position within the planning horizon $t_{p}$ and to stop there.
It is also important to choose the right integration time step $t_{i}<t_{p}$,
so that the robot will still be able to compute the next action within the
time of the movement $t_{i}$. Then we solve the MPC problem in line 14.
Finally, we perform the optimized action in the simulation (in line 27) and
receive a new observation which is used in the next iteration.

##  V Experiments

$0$$1$$2$$0$$0.2$$0.4$$0.6$$0.8$$1$Tolerance of $N$ collisionAveraged Success
Rate (%)HGGMPCMPC-HGG (a) DynamicSquareObstacles.

$0$$1$$2$$0$$0.2$$0.4$$0.6$$0.8$$1$Tolerance of $N$ collisionAveraged Success
Rate (%)HGGMPCMPC-HGG (b) DynamicMixedObstacles.

$0$$1$$2$$0$$0.2$$0.4$$0.6$$0.8$$1$Tolerance of $N$ collisionAveraged Success
Rate (%)HGGMPCMPC-HGG (c) DynamicRecObstacles.

$0$$1$$2$$0$$0.2$$0.4$$0.6$$0.8$$1$Tolerance of $N$ collisionAveraged Success
Rate (%)HGGMPCMPC-HGG (d) DynamicLiftedObstacles.

Figure 5: Success rates of the collision avoidance testing of MPC-HGG, HGG,
and the MPC controller.

(a) Step 12

(b) Step 15

(c) Step 19

(d) Step 25

Figure 6: Example of a resolved collision in the DynamicSquareObstacles
environment. The red dot represents the goal position and the cross marks are
the intermediate goals suggested by the RL planner. The dashed blue circles
are the current position of the obstacle and the dashed green circles are the
predicted states in the near future.

In this section, we present an empirical evaluation of the performance of MPC-
HGG in comparison to HGG and vanilla MPC across four distinct MuJoCo
environments. These environments are variants of the Fetch gripper
environments introduced by [16], and are characterized by the presence of
long-horizon goal-reaching tasks with static or dynamic obstacles.

###  V-A Simulations

All our tasks are simulated in MuJoCo [17] and are performed in the real
world, in which a Panda robot with a gripper is controlled to push a puck
through environments with dynamic obstacles (see Figure 4). These tasks share
the following characteristics. First, the agent receives a state containing
the joint positions and velocities of the robotic arm. This information is
directly retrieved from the simulation or from the robotic arm. Second, the
robot is controlled by a three-dimensional vector describing the end
effector’s position. In the case of enabled gripper control, the gripper’s
opening control parameter is added as the fourth component. Third, the
accessible goal space $\mathcal{G}_{A}$ is defined by a 2D region on the
table. Fourth, we obtained the positional information of obstacles directly
from the simulation. Conversely, in the physical experiment, a side-view
camera was employed to capture the relevant positional data. Notably, we
determined the distance threshold required to ascertain the successful
attainment of a goal to be $0.05$ m.

The difficulty of the four tasks is gradually increased, and a brief
description of each environment is given as follows.

  1. 1.

DynamicSquareObstacles (see Figure 4(a): In this environment, there are two
dynamic square obstacles that can move linearly in two directions with a
velocity randomly sampled between $0.6$ m/s and $0.9$ m/s. The direction of
the movement is also randomized. The transparent red regions represent the
area that can be blocked by the obstacles.

  2. 2.

DynamicMixedObstacles (see Figure 4(b): In this environment, there is one
dynamic square obstacle and one static big rectangle obstacle. Both obstacles
are sampled randomly inside the red region. The square obstacle is moving with
a velocity sampled between $0.6$ m/s and $0.9$ m/s.

  3. 3.

DynamicRectObstacles (see Figure 4(c): In this environment, the speed of the
rectangle obstacle is chosen randomly from the interval between $0.2$m/s and
$0.6$m/s.

  4. 4.

DynamicLiftedObstacles (see Figure 4(d): In this environment, a rectangular
static obstacle is placed under one of the dynamic obstacles. Therefore, the
robot must lift the object above the static obstacle and subsequently lower it
down to the goal positions. Dynamic obstacles are sampled randomly inside the
red region.

Figure 5 shows the testing success rate of HGG, MPC, and MPC-HGG, which is
calculated by averaging the performance of the best policy from each algorithm
in $100$ episodes. These tests have a tolerance parameter $N\in\\{0,1,2\\}$
for the number of collisions that can be allowed per episode. If the number of
collisions surpasses $N$, then the episode is terminated as a failure. The
most remarkable results can be observed, in all four environments; MPC-HGG can
learn safe obstacle-avoiding behavior with a success rate of $100\%$ across
different numbers of tolerance parameters, while the other algorithms are not
able to solve tasks without collisions. In the DynamicSquareObstacles
environment (see Figure 5 (a), the HGG controller achieves a success rate
around $75\%$ when $N=0$ and slightly increases its performance when $N=1$ and
$N=2$. Since the MPC controller is a deterministic method, it demonstrates a
consistently stable performance without any associated error bars. The
proposed MPC-HGG controller combines the advantage of long-horizon planning of
the HGG controller with the short-horizon safety guarantee of the MPC
controller, leading to a highly effective solution with a remarkable success
rate of $100\%$. Similar results can also observed in the other three
scenarios (see Figure 5 (b,c,d)).

Figure 6 shows a successful obstacle avoiding behavior in the
DynamicSquareObstacles environment. At step $15$, the subgoals proposed by the
RL planner (marked by “‘$\times$”) lead to an intersection between the puck
and the second dynamic obstacle. In the effect of the MPC actor, the agent
predicts the upcoming of the obstacle and calculates a safe path by shifting
its moving direction away from the obstacle (see Figure 6(c), step 19). At
step $25$, the puck is successfully placed to the goal position (see Figure
6(d)). The simulation videos show the obstacle-avoidance movements in detail
111https://videoviewsite.wixsite.com/mpc-hgg.

###  V-B Real-world Experiments

As illustrated in Figure 4(e), the real-world experiment setup uses a Franka
Emika Panda robotic arm and an Intel RealSense camera to obtain the
coordinates of the manipulatable object. Two leading screws that carry white
blocks are controlled by stepper motors and act as the dynamic obstacles. The
control rate is consistent with the simulation, which is $20$ Hz across all
experiments. The ForcesPro [18] is used as the MPC solver for fast-speed
computation. The overall computational time is less than $3$ ms for a planning
horizon of $8$ steps.

As in the four environments in the simulation, we also create four
environments featuring the Panda robotic arm with the obstacles (see Figure
4). It should be noted that unlike the simulation that obtains the coordinates
of the obstacles directly, we use a camera to gather this information by
tracking the ArUco Maker attached to moving obstacles in the real world.

We took the policy directly from each of the tasks trained in the simulation
and deployed it in the real world without any fine tuning. Inspired by the
experiment performed by HER [4], we also add Gaussian noise to the observed
object’s position during policy training to compensate for the small errors
introduced by the camera, which can increase the success rate of these tasks.
The performances of these four tasks demonstrate that the policy can be
successfully transferred to the corresponding tasks in the real world. In each
scenario, the task is performed three times by selecting different goal
locations. As shown in the video, the robot arm can always successfully
approach the target position in all trajectories while actively avoiding the
dynamic obstacles. Once the MPC actor anticipates a collision, it will control
the robotic arm to move away from the moving direction of the obstacle and
replan its trajectory to the target goal position. The real-world experiment
videos can be found at the project’s website. As shown in the video, the robot
arm can always successfully avoid the obstacles and approach the target
position in every trial.

Figure 7: The scenario scene from the front and right (left). Right: One
example illustrates the trajectory of the robotic arm in the
DynamicSquareObstacles environment from the isometric view (right). This
animated exemplary video is available at the project’s website.

##  VI Conclusion

In this paper, we have introduced a safety guaranteed planning framework for
manipulation tasks based on a RL planner and a MPC actor. We showed that our
RL planner exhibits the capability to choose a sequence of intermediary goals
that culminate in achieving the ultimate targets, while proficiently
circumventing dynamic impediments encountered in its trajectory. We also
demonstrated that our MPC actor is able to accomplish the designated goals
proposed the RL planner while ensuring no collision with dynamic obstacles.
For future work, we will explore more advanced methods to ensure the safety of
the manipulation operation.

## Acknowledgment

This project/research has received funding from the European Union’s Horizon
2020 Framework Programme for Research and Innovation under the Specific Grant
Agreement No.945539 (Human Brain Project SGA3). The authors also acknowledge
the financial support by the Bavarian State Ministry for Economic Affairs,
Regional Development and Energy (StMWi) for the Lighthouse Initiative
KI.FABRIK, (Phase 1: Infrastructure as well as the research and development
program under, grant no. DIK0249).

## References

  * [1] Z. Bing, M. Brucker, F. O. Morin, R. Li, X. Su, K. Huang, and A. Knoll, “Complex robotic manipulation via graph-based hindsight goal generation,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 33, no. 12, pp. 7863–7876, 2022.
  * [2] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” _Science Robotics_ , vol. 4, no. 26, p. eaau5872, 2019.
  * [3] T. Kollar and N. Roy, “Trajectory optimization using reinforcement learning for map exploration,” _The International Journal of Robotics Research_ , vol. 27, no. 2, pp. 175–196, 2008.
  * [4] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, “Hindsight experience replay,” 2017. [Online]. Available: https://arxiv.org/abs/1707.01495
  * [5] Z. Ren, K. Dong, Y. Zhou, Q. Liu, and J. Peng, “Exploration via hindsight goal generation,” 2019. [Online]. Available: https://arxiv.org/abs/1906.04279
  * [6] M. Schwenzer, M. Ay, T. Bergs, and D. Abel, “Review on model predictive control: an engineering perspective,” _The International Journal of Advanced Manufacturing Technology_ , vol. 117, no. 5, pp. 1327–1349, 2021.
  * [7] M. Eckhoff, R. J. Kirschner, E. Kern, S. Abdolshah, and S. Haddadin, “An mpc framework for planning safe and trustworthy robot motions,” in _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 4737–4742.
  * [8] B. Brito, M. Everett, J. How, and J. Alonso-Mora, “Where to go next: Learning a subgoal recommendation policy for navigation among pedestrians,” _IEEE Robotics and Automation Letters_ , 2021.
  * [9] N. Hansen, X. Wang, and H. Su, “Temporal difference learning for model predictive control,” 2022. [Online]. Available: https://arxiv.org/abs/2203.04955
  * [10] R. R. Negenborn, B. De Schutter, M. A. Wiering, and H. Hellendoorn, “Learning-based model predictive control for markov decision processes,” _IFAC Proceedings Volumes_ , vol. 38, no. 1, pp. 354–359, 2005, 16th IFAC World Congress. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1474667016362929
  * [11] M. Bhardwaj, A. Handa, D. Fox, and B. Boots, “Information theoretic model predictive q-learning,” 2020. [Online]. Available: https://arxiv.org/abs/2001.02153
  * [12] C. Greatwood and A. G. Richards, “Reinforcement learning and model predictive control for robust embedded quadrotor guidance and control,” _Autonomous Robots_ , vol. 43, no. 7, pp. 1681–1693, 2019.
  * [13] J. Xue, X. Kong, B. Dong, and M. Xu, “Multi-agent path planning based on mpc and ddpg,” 2021. [Online]. Available: https://arxiv.org/abs/2102.13283
  * [14] T.-Y. Yang, T. Zhang, L. Luu, S. Ha, J. Tan, and W. Yu, “Safe reinforcement learning for legged locomotion,” 2022. [Online]. Available: https://arxiv.org/abs/2203.02638
  * [15] M. Lange, D. Zühlke, O. Holz, and T. Villmann, “Applications of lp-norms and their smooth approximations for gradient based learning vector quantization,” in _ESANN_ , 2014.
  * [16] M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. Tobin, M. Chociej, P. Welinder _et al._ , “Multi-goal reinforcement learning: Challenging robotics environments and request for research,” _arXiv:1802.09464_ , 2018.
  * [17] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2012, pp. 5026–5033.
  * [18] A. Domahidi and J. Jerez, “Forces professional,” Embotech AG website, 2014. [Online]. Available: https://embotech.com/FORCES-Pro


# MUTEN: Boosting Gradient-Based Adversarial Attacks via Mutant-Based
Ensembles

Yuejun Guo  Qiang Hu  Maxime Cordy  Michail Papadakis  Yves Le Traon
1Interdisciplinary Centre for Security, Reliability and Trust, University of
Luxembourg

###### Abstract

Deep Neural Networks (DNNs) are vulnerable to adversarial examples, which
causes serious threats to security-critical applications. This motivated much
research on providing mechanisms to make models more robust against
adversarial attacks. Unfortunately, most of these defenses, such as gradient
masking, are easily overcome through different attack means. In this paper, we
propose MUTEN, a low-cost method to improve the success rate of well-known
attacks against gradient-masking models. Our idea is to apply the attacks on
an ensemble model which is built by mutating the original model elements after
training. As we found out that mutant diversity is a key factor in improving
success rate, we design a greedy algorithm for generating diverse mutants
efficiently. Experimental results on MNIST, SVHN, and CIFAR10 show that MUTEN
can increase the success rate of four attacks by up to 0.45.

##  1 Introduction

Deep neural networks (DNNs) have achieved impressive success in a wide range
of artificial intelligence tasks, such as image classification Rawat and Wang
(2017) and speech recognition Zhang et al. (2018). However, most DNNs suffer
from being fooled by adversarial examples, which can result in serious
security issues Finlayson et al. (2019); Eykholt et al. (2018). Such an
adversarial example is typically crafted by adding a subtle (elusive)
perturbation to a benign input in a way that misleads a DNN model. The
adversarial examples are useful not only to reveal security threats in DNNs,
but also to improve their robustness, e.g., through adversarial training Madry
et al. (2018).

Many adversarial attacks have been investigated. In general, there are two
types of methods depending on what model information is available. Black-box
attacks have no knowledge of the model, while the white-box attacks have
access to model information such as its architecture, gradient, and weights.
Most adversarial attacks are _gradient-based_ , in the sense that they utilize
the gradient of the loss function with the hope to compute the perturbation
direction that will maximize the likelihood of misclassification.

The adversarial phenomena, together with the increasing use of machine
learning models in real-world applications, have motivated much research on
effective ways to generate adversarial examples and defend against them
(making the models _robust_ to adversarial attacks). Gradient masking Papernot
et al. (2017) is one such protection means which is commonly used by
researchers.111Back in ICLR 2018, 7 out of 9 papers on adversarial defense
were relying on gradient masking. Athalye et al. (2018a) It refers to the
phenomenon where the gradient cannot be used to optimize the DNN loss function
and produce a corresponding perturbation. Gradient masking can result from
intentional actions (defenses relying on gradient obfuscation) or natural
properties of deep models (where successive computations or specific
components cause gradient vanishing); see Figure 1(a) for an illustration.

Perhaps paradoxically, despite its popularity the use of gradient masking as a
defense mechanism has been fiercely dismissed Athalye et al. (2018a); Uesato
et al. (2018). The main reason is that the security it offers is superficial
and gives a _“false sense of security”_. For instance, Athalye et al. Athalye
et al. (2018a) show that although gradient masking successfully defends
against common gradient-based attacks (e.g. PGD Madry et al. (2018) and C&W
Carlini and Wagner (2017)), it is easily overcome through alternative means
(e.g., approximated derivatives, Expectation Over Transformation Athalye et
al. (2018b) and reparameterization).

In this paper, we further reduce the trust that one can have in this defense
mechanism and show that _common attacks can easily overcome gradient masking_.
We propose MUTEN, a fast and effective way to attack models with masked
gradients. Unlike previous work circumventing gradient masking Athalye et al.
(2018a); Uesato et al. (2018), MUTEN does not rely on new attack means and can
be applied jointly with any existing algorithms (we consider, more
specifically, FGSM Goodfellow et al. (2015), BIM Kurakin et al. (2017), PGD
Madry et al. (2018) and C&W Carlini and Wagner (2017)). Thereby, our work
strengthens the claims that the robustness promised by gradient masking is a
mere illusion.

The effectiveness of our approach leans on Liu et al.’s work Liu et al.
(2017), where they show that adversarial examples crafted from an ensemble of
surrogate models transfer relatively well to the original model. Unlike their
method, though, MUTEN avoids the prohibitive cost of training multiple models.
More precisely, we generate an ensemble of models containing a set of _mutant_
models, obtained by altering the original one _after training_ (e.g., by
introducing random noise into its weights). Working in white-box settings, we
also include the original model in the ensemble. Thus, the overhead of
attacking the ensemble is limited to applying the attack to each mutant and
computing the average of the perturbations.

Given that there exist many ways to mutate a model Ma et al. (2018), we
investigate which combinations of mutation are more effective in increasing
the success rate of the attacks. Our hypothesis is that an effective ensemble
should contain _a diverse set of accurate mutants_. On the one hand, we ensure
that the mutants retain the performance (test accuracy) of their original
model by controlling the proportion of modified weights, neurons, or layers.
On the other hand, we propose a greedy algorithm to produce a _diverse_
mutants where the diversity is measured by the centered kernel alignment (CKA)
metric Kornblith et al. (2019) and PageRank algorithm Moler (2011).

(a) A model

(b) An ensemble

Figure 1: An illustration of the gradient masking phenomenon and how our
approach solves it. Both figures show the gradients of a Lenet-5 model and our
produce ensemble with respect to a 28×28 pixel MNIST image (model’s output:
92.73% probability belong to class “8”). Here, we see that gradient masking
occurs in the original model but not in the ensemble.

To summarize, the main contributions of this paper are:

  1. 1.

We propose MUTEN, a novel approach to boost the effectiveness of gradient-
based attacks, overcoming gradient masking, by building an ensemble from the
original model without extra training.

  2. 2.

We plug MUTEN on top of existing adversarial attacks without requiring
modifications to their core algorithms.

  3. 3.

We design a greedy algorithm to generate diverse mutants that retain the
original model accuracy.

  4. 4.

We show the effectiveness of our approach through an extensive experimental
study involving three datasets, four models, and four attacks. Our results
reveal that the success rate can be increased by up to 0.45.

##  2 Background

###  2.1 Gradient-Based Attacks

#### FGSM

Goodfellow _et al._ Goodfellow et al. (2015) proposed one of the simplest
methods, fast gradient sign method (FGSM), to generate adversarial examples.
Let $x$ and $x^{\prime}$ be the original image and the adversarial example,
respectively, FGSM crafts $x^{\prime}$ by $x^{\prime}=x+\epsilon\cdot{\rm
sign}\left(\triangledown_{x}J\left(x,y\right)\right)$ where $\epsilon$
controls the perturbation size. $sign\left(\cdot\right)$ is the sign function.
The sign of a real number is -1 for a negative value, 1 for a positive value,
and 0 for value 0. $\triangledown_{x}J\left(x,y\right)$ computes the gradient
of the training loss $J$ given $x$ and its true class $y$.

#### BIM

Based on FGSM, Kurakin _et al._ Kurakin et al. (2017) proposed an iterative
version, basic iterative method (BIM, also called i-FGSM), which applies FGSM
multiple times with a small step size to craft $x^{\prime}$:
$x^{\prime}_{n+1}=Clip_{x,\epsilon}\left\\{x^{\prime}_{n}+\alpha\cdot{\rm
sign}\left(\triangledown_{x}J\left(x^{\prime}_{n},y\right)\right)\right\\}$
where $x^{\prime}_{0}=x$ and $Clip_{x,\epsilon}$ is a clip function applied
after each iteration to ensure that the result is still in
$\epsilon$-neighbourhood of $x$. $n$ is the iteration index and $\alpha$ is
the step size.

#### PGD

Madry _et al._ Madry et al. (2018) proposed the projected gradient descent
(PGD) to improve BIM. The difference is that BIM starts from the original
point, while PGD randomly chooses the starting point within a $\epsilon$ norm
ball.

#### C&W

The C&W attack, proposed by Carlini and Wagner Carlini and Wagner (2017), is
known as one of the strongest attacks to DNNs. Instead of using the training
loss, C&W uses a designed loss function $f$ to craft the adversarial example
$x^{\prime}$ which minimizes $D\left(x,x^{\prime}\right)+c\cdot f(x^{\prime})$
where $D$ is a distance metric and $c$ is a constant that controls the
distance and the confidence of $x^{\prime}$.

###  2.2 Mutation of Deep Learning Models

Post-training mutation of DNNs has been applied mainly for quality assurance
purposes. Due to the unique characters of DNN models, various mutation
operators have been proposed at source level (modifying the training data or
the training program) or at model level Ma et al. (2018).

In this paper, we apply model-level mutations, where a mutant is created
directly by changing the neurons, weights or, layers slightly without
training. In general, modifying the layers requires specific architectures of
the DNN models and degrades the performance (accuracy) significantly, and is
less applicable. Both the weight- and neuron-level operators work efficiently
to generate mutants and are more widely used.

Recent studies have shown the utility of mutation in different tasks. Ma _et
al._ Ma et al. (2018) propose to mutate test data class by class to figure out
the weakness in test data, which is helpful to check to bias in data. Hu _et
al._ Hu et al. (2019) points out that by a defined killing score metric, the
mutants can be used to validate how robust a DNN model is against an input
data or its segment. In Wang et al. (2019), Wang _et al._ assume that the
adversarial examples are near the decision boundary, thus, the data that
change the labels by different mutants are considered as adversarial.

##  3 Approach

We aim to improve the success rate of gradient-based adversarial attacks
applied to gradient-masking models. The main idea of MUTEN is to produce a
collection of diverse mutant models to build an ensemble, and attack the
ensemble instead of the single original model.

###  3.1 Diverse Mutant Generation

To build an ensemble without extra training, we employ the model-level
mutation. Since previous research has shown that mutating layers always
degrade the performance (test accuracy) significantly Ma et al. (2018), we
consider 5 operators working only at the weight- and neuron-level. Gaussian
fuzzing (GF) operator adds noise to the selected weights following the
Gaussian distribution. The weight shuffling (WS) rearranges the selected
weights. The neuron effect blocking (NEB) resets the connection weight of a
selected neuron to the next layer to zero. The neuron activation inverse (NAI)
operator inverts the activation status of a neuron. The neuron switch (NS)
exchanges two neurons within the same layer. The mutation ratio controls how
much percentage of weights or neurons are selected for mutation in each layer.
As applying more mutations has a higher likelihood to decrease model
performance, we randomly set it between 1% and 4% following the previous
findings of Ma et al. Ma et al. (2018).

To measure mutant diversity, we use the centered kernel alignment (CKA)
Kornblith et al. (2019) metric and the PageRank algorithm Moler (2011). More
precisely, CKA measures the similarity between DNN representations. Given the
input data X, let $H_{1}$ and $H_{2}$ be two feature matrices of $X$ by two
models, respectively. $H_{1}$ and $H_{2}$ are considered as the DNN
representations. The similarity between $H_{1}$ and $H_{2}$ is defined by

$CKA\left(K,L\right)=\frac{{\rm HSIC}\left(K,L\right)}{\sqrt{{\rm
HSIC}\left(K,K\right){\rm HSIC}\left(L,L\right)}}$ (1)

where $K$ and $L$ are the kernel matrices by passing $H_{1}$ and $H_{2}$
through kernels. $\rm HSIC$ is the Hilbert-Schmidt independence criterion.
Like Chen et al. (2020), we use the output of the last hidden layer in a DNN
as the feature.

The PageRank algorithm aims at measuring the importance/rank of website pages
where a page linked to by many pages has a high rank. Inspired by this, taking
a mutant as a website page and the similarity as the linking weight, we assume
that the mutant with a low rank is diverse within the mutant set as it is
dissimilar with the others.

Algorithm 1 shows our overall generation method. To increase the diversity of
generated mutants, we use 20 pairs of mutation operators and ratios. Given a
model $M$ and a required number of mutants $n$, the mutant set $set$ and a
similarity matrix $D$ are initialized to be empty, and a counting index
$count$ is used to control the termination of the algorithm (Line 1). In each
iteration, a pair of mutation operator and ratio is randomly selected from all
the candidates (Line 3) to generate a mutant (Line 4). After the first
iteration, the mutant set $Mu$ is updated with a mutant (Lines 5-6). When
another new mutant is generated, first, we compute the linear CKA similarity
between this mutant and the ones in $Mu$ to update the similarity matrix $D$
(Lines 7-9). If the size of mutation set, $|Mu|$, is smaller than $n$, the
procedure continues, otherwise, $D$ is fed into the $pageRank$ function to
compute the diversity and update the mutant set (Lines 11-13 ). The maximum
size of $Mu$ is $n$. The iteration terminates until it reaches a preset number
of iterations. Figure 2 shows the linear CKA similarity of 10 mutants in types
of diverse, random, and similar, respectively. Diverse mutants are more
dissimilar from each other.

Algorithm 1 Greedy mutant generation algorithm

Input: $M$: DNN model
$O=\\{$GF, WS, NEB, NAI, NS$\\}$: mutation operators
$R=\left\\{0.01,0.02,0.03,0.04\right\\}$: mutation ratios
$n$: required number of mutants
$ite$: number of iterations
Output: $Mu$: a set of mutants

1: Initialize $Mu$, $D$, $count=0$

2: while $count<ite$ do

3: $\left(o,r\right)=randomSelect\left(O,R\right)$

4: $m=mutantGenerator\left(M,o,r\right)$

5: if $|Mu|$==0 then

6: $Mu=\left\\{m\right\\}$

7: else

8: $D=computeCka\left(m,Mu\right)$

9: if $|Mu|<n$ then

10: $|Mu|=Mu\cup\left\\{m\right\\}$

11: else

12: $Mu=pageRank\left(D,Mu,m\right)$

13: end if

14: end if

15: $count++$

16: end while

17: return $Mu$

(a) Diverse

(b) Random (c) Similar

Figure 2: CKA similarity of 10 mutants (diverse, random and similar). Darker
color indicates a higher similarity between models. Dataset: MNIST; Model:
Lenet-5.

###  3.2 Ensemble Model Construction

Figure 3 illustrates how we construct an ensemble. Given the training data, a
model is trained with a specific architecture and parameters. By the greedy
algorithm mentioned in Section 3.1, multiple diverse mutants are obtained. In
the example, we show the effect of the mutation operators GF, NEB, and NS, and
the modified weights and neurons are highlighted in blue. At last, an ensemble
model is built by gathering all the original model and its mutants. When
performing an adversarial attack, we use the simple average strategy Demir
(2016). That is, the attack accesses each base model to obtain the gradient
given an input data, then the perturbation is calculated based on the average
of all the gradients.

Figure 3: Example of building an ensemble model.

##  4 Experiments

To evaluate the effectiveness of our approach in increasing adversarial attack
success rate, we conducted experiments using TensorFlow 2.3.0 and Keras 2.4.3.
We implemented MUTEN by extending the DeepMutation++ tool Hu et al. (2019) and
apply the library of CKA222https://github.com/google-research/google-
research/tree/master/representation_similarity and fast-
pagerank333https://github.com/asajadi/fast-pagerank to compute the rank of
diversity. To allow for reproducibility, our full implementation and
evaluation subjects are available on GitHub.444Link anomymized for double-
blind review.

###  4.1 Experimental Setup

#### DNN models and datasets

We conduct the experiments on 3 widely used datasets, MNIST Lecun et al.
(1998), SVHN Netzer et al. (2011), and CIFAR10 Krizhevsky (2009). MNIST is a
10-class grayscale handwritten digit dataset. SVHN is a real-world image
dataset including 10 classes of street view house numbers. CIFAR10 is a
10-class dataset with color images. For MNIST and SVHN, we employ the Lenet-5
model as Athalye et al. (2018a). For CIFAR10, we use VGG16 and ResNet20V1
because, as deep models, they are subject to natural gradient vanishing
phenomena. Table 1 summarizes the detailed information of the datasets and
models. As we generate weight- and neuron-level mutants, the number of weights
and neurons are also given in the third and fourth columns, respectively.

Dataset | Model | #Weights | #Neurons | #Tests | Accuracy(%)
---|---|---|---|---|---
MNIST | Lenet-5 | 107550 | 236 | 10000 | 98.89
SVHN | Lenet-5 | 136650 | 236 | 26032 | 88.93
CIFAR10 | ResNet20V1 | 270896 | 794 | 10000 | 90.71
| VGG16 | 2851008 | 1674 | 10000 | 91.17

Table 1: DNN models and datasets

#### Attacks and parameter setting

We evaluate MUTEN with four state-of-the-art gradient-based attacks (see
Section 2), _i.e._ , FGSM, BIM, PGD ($l_{\infty}$-norm), C&W ($l_{2}$-norm).
The attacks are implemented based on the IBM ART
framework555https://github.com/Trusted-AI/adversarial-robustness-toolbox
Nicolae et al. (2018). To avoid the influence of parameters, we set 7 values
for each attack. Table 2 presents the configurations of the attacks used for
each dataset. Besides, in FGSM, BIM, PGD, the step size of $\epsilon$ is set
as $\frac{\epsilon}{10}$, and the maximum iteration is set as 40 and 20, for
MNIST/SVHN and CIFAR10, respectively. In C&W, the learning rate is 0.1, and
the maximum iteration is 100 for all the models. For the rest of the
parameters, we use the default setting in ART.

Dataset |  | FGSM, BIM, PGD
---
$\epsilon$
| C&W
---
$c$
MNIST | $0.1,0.15,0.2,0.25,0.3,0.35,0.4$ | $7,8,9,10,11,12,13$
SVHN
CIFAR10 | $\frac{2}{255},\frac{4}{255},\frac{6}{255},\frac{8}{255},\frac{10}{255},\frac{12}{255},\frac{14}{255}$ | $0.1,0.15,0.2,0.25,0.3,0.35,0.4$

Table 2: Attack configurations. “$\epsilon$” and $c$ stand for the maximum
perturbation and initial constant, respectively.

###  4.2 Results

We conducted three series of experiments to evaluate MUTEN from different
perspectives. Since it is meaningless to attack misclassified data as being
considered as “adversarial” for the model, each attack crafts one example
based on each benign data. Each experiment was repeated five times to reduce
the influence of randomness in both the creation of mutants and the
application of the attacks. Reported results are the average of those five
runs.

#### Effectiveness

Firstly, we evaluate the effectiveness of MUTEN measured as the success rate
of the applied attacks. The success rate of the four attacks applied to the
original model is taken as the baseline. Figure 4 shows the success rate of
MUTEN with different attack configurations. As the success rate converges
mostly when the ensemble includes 5 mutants (see next section), we only
present the result with this number. Overall, compared with the four
baselines, MUTEN achieves a higher success rate, especially as the maximum
perturbation size increases. For example, in the case of PGD with VGG16, the
success rate by MUTEN reaches 0.97, while the baseline increases it by 0.62
with the maximum perturbation size. When the threshold is very small (for
instance, $\epsilon<0.2$ for MNIST and $\epsilon<\frac{8}{255}$ for ResNet20V1
by FGSM), the success rate achieved by our approach is lower than or keeps
similar to the baseline. The reason is that the ensemble is an average over
multiple models and so as the gradient, given a very small perturbation, the
attack requires more iterations to converge, which is demonstrated by FGSM and
BIM (which is an iterative version of FGSM).

(a) MNIST, FGSM (b) SVHN, FGSM (c) ResNet20V1, FGSM (d) VGG16, FGSM

(e) MNIST, BIM (f) SVHN, BIM (g) ResNet20V1, BIM (h) VGG16, BIM

(i) MNIST, PGD (j) SVHN, PGD (k) ResNet20V1, PGD (l) VGG16, PGD

(m) MNIST, C&W (n) SVHN, C&W (o) ResNet20V1, C&W (p) VGG16, C&W

Figure 4: Success rate VS. attack configuration. The shade area indicates
where MUTEN outperforms the baseline.

#### Impact of the number of mutants

Secondly, we investigate how the number of mutants in the ensemble model
impacts the effectiveness of MUTEN. We use the commonly used configurations
($\epsilon=\frac{8}{255}$, $c=0.3$) for the attacks and consider a number of
mutants ranging from 1 to 10. Figure 5 shows the results for CIFAR10 with
VGG16. In general, the effectiveness of MUTEN increases as more mutant is
integrated but tends to saturate quickly in the cases of FGSM, BIM and PGD.
Concerning the improvement of success rate, it increases from 0.20 to 0.33 in
FGSM, 0.34 to 0.47 in BIM, 0.32 to 0.43 in PGD, and -0.09 to 0.21 in C&W. In
the case of FGSM, IFGSM, and PGD, a very high improvement of success rate can
be reached with only 1 mutant, which is only the double-time of attacking the
original model. Adding more mutants will increase more the success rate but
with a slower growth rate. In the case of the strongest attack, C&W, the
success rate is lower than the baseline when the ensemble includes 1 mutant,
which also happens to the other models. The reason is that for C&W, when the
parameter $c$ is very small, the gradient loss has a small contribution to the
loss function used by the attack algorithm (see Section 2.1). By contrast,
letting the gradient loss be more important by increasing $c$, the success
rate boosts quickly. Thus, in this case, to increase the success rate of C&W,
one can either adjust $c$ to be greater or include more mutants. Note that,
attacking an ensemble requires multiple times due to that the attack computes
the average gradient over each base model. As a result, the time cost is
linearly related to the number of models in the ensemble. Figure 7 shows the
ratio of time cost of attacking an ensemble to a single model. Therefore, in
practical applications, one has to consider the trade-off between a high
success rate and a small number of mutants.

(a) FGSM (b) BIM

(c) PGD (d) C&W

Figure 5: Success rate VS. number of mutants. The shade area indicates where
MUTEN outperforms the baseline. Dataset: CIFAR10; Model: VGG16.

#### Diversity of mutants

Thirdly, we investigate if the diversity contributes to MUTEN. We produce
three types of mutants, diverse, random, and similar. The random mutants are
generated by random selection, and similar mutants are created by limiting the
test accuracy to be at least 95% reserved for the original model. Figure 6
shows the result of CIFAR10 with model VGG16. In general, the diverse ensemble
performs the best, and the random ensemble works better than the similar one.
In the case of FGSM, BIM, and PGD, when the perturbation is small (for
instance, smaller than $\frac{4}{255}$), the difference of using three types
of mutants is not big. By increasing the number of iterations of the greedy
algorithm, the mutants can be more diverse, and the difference between random
and diverse ensembles will be greater.

(a) FGSM (b) BIM

(c) PGD (d) C&W

Figure 6: Success rate VS. type of mutants. Each ensemble model includes 5
mutants. The shade area indicates where MUTEN outperforms the baseline.
Dataset: CIFAR10; Model: VGG16.

(a) FGSM (b) BIM

(c) PGD (d) C&W

Figure 7: Ratio of time cost VS. number of mutants. Dataset: CIFAR10; Model:
VGG16.

###  4.3 Discussion

#### Extensions of MUTEN

Overall, using the diverse mutant-based ensemble is helpful to improve the
effectiveness of craft adversarial examples. As it is easy and simple to
generate a large number of mutants, MUTEN can be taken as a plug-in module in
white-box attacks. Besides, since the number of mutants linearly relates to
the efficiency, the number can be adjusted to balance between the time cost
and the desired number of adversarial examples.

#### Threats to validity

The internal threat comes from our implementation. To counter this, we use
some popular libraries as mentioned in the section of experiments, and
carefully check the code. The external threat is the selected DNN models and
datasets. We choose popular datasets with well-known models, and the model
ranges from simple to complex. The construct threat is mainly about the
method. As the diversity of mutants is important to obtain a good result, the
metric to measure the similarity between mutants and the method to rank the
diversity of mutants, and the ensemble strategy Demir (2016) could be changed
to improve the performance.

##  5 Conclusion

We proposed MUTEN to effectively improve the gradient-based adversarial
attacks. The main idea is to build an ensemble by diverse mutants to modify
the gradient for the attacks to easier figure out the perturbation direction.
Besides, since no extra training is required to produce a large number of
mutants, MUTEN can be regarded as a simple plug-in module in white-box
attacks. The experiments on different datasets and models have demonstrated
that MUTEN performs promisingly to increase the success rate of four state-of-
the-art gradient-based adversarial attacks with only a few mutants.
Additionally, the diversity of mutants have been verified to be necessary to
achieve effectiveness.

## References

  * Athalye et al. [2018a] Anish Athalye, Nicholas Carlini, and David Wagner.  Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples.  In ICML, volume 80, pages 274–283, Stockholmsmässan, Stockholm Sweden, July 2018. PMLR.
  * Athalye et al. [2018b] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok.  Synthesizing robust adversarial examples.  In ICML, volume 80, pages 284–293, Stockholmsmässan, Stockholm, Sweden, July 2018. PMLR.
  * Carlini and Wagner [2017] Nicholas Carlini and David Wagner.  Towards evaluating the robustness of neural networks.  In IEEE Symposium on Security and Privacy (SP), pages 39–57, San Jose, CA, USA, May 2017.
  * Chen et al. [2020] Junjie Chen, Zhuo Wu, Zan Wang, Hanmo You, Lingming Zhang, and Ming Yan.  Practical accuracy estimation for efficient deep neural network testing.  ACM Trans. Softw. Eng. Methodol., 29(4), October 2020.
  * Demir [2016] Necati Demir.  Ensemble methods: elegant techniques to produce improved machine learning results, 2016.
  * Eykholt et al. [2018] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song.  Robust physical-world attacks on deep learning models.  In CVPR, pages 1625–1634, June 2018.
  * Finlayson et al. [2019] Samuel G. Finlayson, John D. Bowers, Joichi Ito, Jonathan L. Zittrain, Andrew L. Beam, and Isaac S. Kohane.  Adversarial attacks on medical machine learning.  Science, 363(6433):1287–1289, 2019.
  * Goodfellow et al. [2015] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples, 2015.
  * Hu et al. [2019] Qiang Hu, Lei Ma, Xiaofei Xie, Bing Yu, Yang Liu, and Jianjun Zhao.  Deepmutation++: a mutation testing framework for deep learning systems.  In IEEE/ACM International Conference on Automated Software Engineering, pages 1158–1161, San Diego, CA, USA, December 2019.
  * Kornblith et al. [2019] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton.  Similarity of neural network representations revisited.  In ICML, Long Beach, California, 2019.
  * Krizhevsky [2009] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, Toronto, 2009.
  * Kurakin et al. [2017] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio.  Adversarial examples in the physical world.  ICLR Workshop, 2017.
  * Lecun et al. [1998] Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278 – 2324, November 1998.
  * Liu et al. [2017] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song.  Delving into transferable adversarial examples and black-box attacks.  In ICLR, November 2017.
  * Ma et al. [2018] Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang.  Deepmutation: mutation testing of deep learning systems.  In International Symposium on Software Reliability Engineering, pages 100–111, Memphis, TN, USA, October 2018.
  * Madry et al. [2018] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In ICLR, Vancouver, BC, Canada, April 2018.
  * Moler [2011] Cleve B Moler.  Experiments with MATLAB.  Society for Industrial and Applied Mathematics, 2011.
  * Netzer et al. [2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng.  Reading digits in natural images with unsupervised feature learning.  NIPS, January 2011.
  * Nicolae et al. [2018] Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian Molloy, and Ben Edwards.  Adversarial robustness toolbox v1.2.0.  CoRR, 1807.01069, 2018.
  * Papernot et al. [2017] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami.  Practical black-box attacks against machine learning.  In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pages 506–519, April 2017.
  * Rawat and Wang [2017] Waseem Rawat and Zenghui Wang.  Deep convolutional neural networks for image classification: a comprehensive review.  Neural Computation, 29(9):2352–2449, 2017.
  * Uesato et al. [2018] Jonathan Uesato, Brendan O’Donoghue, Pushmeet Kohli, and Aäron van den Oord.  Adversarial risk and the dangers of evaluating against weak attacks.  In ICML, volume 80, pages 5032–5041, Stockholmsmässan, Stockholm, Sweden, July 2018. PMLR.
  * Wang et al. [2019] Jingyi Wang, Guoliang Dong, Jun Sun, Xinyu Wang, and Peixin Zhang.  Adversarial sample detection for deep neural network through model mutation testing.  In IEEE/ACM 41st International Conference on Software Engineering, pages 1245–1256, Montreal, QC, Canada, May 2019.
  * Zhang et al. [2018] Zixing Zhang, Jürgen Geiger, Jouni Pohjalainen, Amr El-Desoky Mousa, Wenyu Jin, and Björn Schuller.  Deep learning for environmentally robust speech recognition: an overview of recent developments.  ACM Transactions on Intelligent Systems and Technology, 9(5), April 2018.