# A Review of Adversarial Attack and Defense for Classification Methods

Yao Li<EMAIL_ADDRESS>Department of Statistics & Operations Research,
University of North Carolina at Chapel Hill Minhao Cheng
<EMAIL_ADDRESS>Department of Computer Science, University of
California at Los Angeles Cho-Jui Hsieh<EMAIL_ADDRESS>Department of
Computer Science, University of California at Los Angeles Thomas C. M.
<EMAIL_ADDRESS>Department of Statistics, University of California
at Davis

(August 18, 2021; revised: November 5, 2021)

###### Abstract

Despite the efficiency and scalability of machine learning systems, recent
studies have demonstrated that many classification methods, especially deep
neural networks (DNNs), are vulnerable to adversarial examples; i.e., examples
that are carefully crafted to fool a well-trained classification model while
being indistinguishable from natural data to human. This makes it potentially
unsafe to apply DNNs or related methods in security-critical areas. Since this
issue was first identified by Biggio _et al._ (2013) and Szegedy _et al._
(2014), much work has been done in this field, including the development of
attack methods to generate adversarial examples and the construction of
defense techniques to guard against such examples. This paper aims to
introduce this topic and its latest developments to the statistical community,
primarily focusing on the generation and guarding of adversarial examples.
Computing codes (in python and R) used in the numerical experiments are
publicly available for readers to explore the surveyed methods. It is the hope
of the authors that this paper will encourage more statisticians to work on
this important and exciting field of generating and defending against
adversarial examples.

Keywords: adversarial examples, adversarial training, deep neural networks,
defense robustness

##  1 Introduction

Machine learning systems achieve state-of-the-art performance in various tasks
in artificial intelligence, such as image classification, speech recognition,
machine translation and game-playing (e.g., Simonyan and Zisserman, 2014;
Silver _et al._ , 2016; Devlin _et al._ , 2019). Despite their tremendous
successes, machine learning models have been shown to be vulnerable to
adversarial examples. By adding imperceptible perturbations to the original
inputs, the attacker can produce adversarial examples to fool a learned
classifier (e.g., Szegedy _et al._ , 2014; Goodfellow _et al._ , 2015).
Adversarial examples are indistinguishable from the original inputs to human,
but are mis-classified by the classifier. For an illustration, consider the
following images in Figure 1:

Figure 1: An example of adversarial example, taken from Chen _et al._ (2017).

To humans, the two images appear to be the same – the vision system of human
will identify each image as a bagel. The image on the left side is an ordinary
image of a bagel (the original image). However, the image on the right side is
generated by adding a small and imperceptible perturbation that forces a
particular classifier to classify it as a piano.  The example in Figure 1 was
generated to fool a deep neural network (DNN) based classifier. However, an
adversarial example is not an isolated case that happens only to some neural
networks, but a general case for most of the practically used neural networks.
Besides, recent works have shown that linear models and many other general
statistical learning models, such as logistic regression and tree-based
models, are also vulnerable to adversarial examples (Chen _et al._ , 2019).
With the wide application of machine learning models, this causes serious
concerns about the safety of machine learning systems in security sensitive
areas, such as self-driving cars, flight control systems, healthcare systems
and so on. Therefore, studies on adversarial examples have attracted much
attention in recent years.

Most of the studies on adversarial examples can be generally classified into
two categories: works on how to efficiently generate adversarial examples
(i.e., attack), and works on how to guard against such adversarial examples
(i.e., defense). This paper reviews the latest research on adversarial attack
and defense, and compares the state-of-the-art attack and defense methods on
benchmark datasets. We illustrate the main concepts via an image
classification task with a continuous input space. However, we also note that
there have been research activities on adversarial robustness for other
applications with discrete input data, such as natural language processing
(NLP), discrete models and nearest neighbor classifiers (e.g., Jia and Liang,
2017; Samanta and Mehta, 2017; Gao _et al._ , 2018; Cheng _et al._ , 2020b;
Yang _et al._ , 2020b).

With this paper, it is our hope that more statisticians will be engaged in
this important and exciting field of attacking and defending classification
methods. As to be demonstrated below, many problems from this field are
statistical in nature, and statistical methodologies and principles can be
adopted to improve existing methods or even to develop new methods. R codes
that demonstrate how to attack a logistic regression model are provided for
statisticians who are interested in this problem to experiment with.  Lastly,
we note that there are other review papers in related topics that might be of
interest to the reader; e.g., Serban _et al._ (2020) reviewed adversarial
phenomenon in the field of object detection, while other authors (e.g., Qiu
_et al._ , 2019; Yuan _et al._ , 2019; Ren _et al._ , 2020; Xu _et al._ ,
2020) provided reviews of adversarial examples with emphasis in computer
vision and natural language processing from the perspective of computer
science. These authors categorized the methods in a different manner when
compared to ours.

Notation. In this paper, all the vectors are denoted as bold symbols. The
input to the classifier is represented by ${\bm{x}}$ and the label associated
with the input is represented by $y$. Thus, one observation is a pair
$({\bm{x}},y)$. The classifier is denoted as $f(\cdot)$ and $f({\bm{x}})$
represents the output vector of the classifier. The dimension of $f({\bm{x}})$
is equal to the number of classes of the dataset. Let $f({\bm{x}})_{i}$ denote
the score of predicting ${\bm{x}}$ with label $i$. We caution that, in some
works in the literature, $f({\bm{x}})_{i}$ is taken as the “probability” that
sample ${\bm{x}}$ corresponds to label $i$, but very often these
$f({\bm{x}})_{i}$’s are not probabilities. The prediction of the classifier is
denoted as
$c({\bm{x}})=\operatornamewithlimits{argmax}\limits_{i}f({\bm{x}})_{i}$; that
is, the predicted label is the one with the highest prediction score. We use
the $\ell_{\infty}$ and $\ell_{2}$ distortion metrics to measure similarity
between inputs and report the $\ell_{\infty}$ distance in the normalized
$[0,1]$ space, and the $\ell_{2}$ distance as the total root-mean-square
distortion normalized by the total number of dimensions.

The rest of this paper is organized as follows. Section 2 provides a quick
summary of DNNs. This section can be omitted if the reader has a good
understanding of the classification task, by treating a DNN as a black-box
classifer. Then Section 3 describes the latest developments for attacking
classification methods, while Section 4 presents defense methods for defending
against such attacks. Lastly, numerical comparisons are given in Section 5
while concluding remarks, including future directions for statistical
research, are offered in Section 6.

##  2 Deep Neural Networks

Deep Neural Networks (DNNs) have become one of the most prominent technologies
of our time, as they achieve state-of-the-art performance in many machine
learning tasks, including but not limited to image classification, text
mining, and speech processing. A brief introduction to DNNs is given in this
section to help readers understand attack and defense methods. More details of
DNNs can be found, for example, in Goodfellow _et al._ (2016).  To speed up
the pace, the readers may skip this section (and come back later), which will
not affect understanding the concepts behind those attack and defense methods.
It is because the DNNs described below can be viewed as black-box classifiers
and their details are not crucial for the comprehension of such concepts.
However, for readers who are interested in the robustness of DNNs and not
familiar with their architectures, this section would be helpful.

A DNN consists of a large class of models and learning methods. Here we
describe a general form of DNN classifier that is widely applied in image
classification. For a DNN model $f(\cdot)$ with $L$ layers, each layer is
parameterized by a weight matrix $\bm{\theta}_{i}$, which holds the knowledge
of the DNN model and is updated during the training process. Each neuron of
the DNN has an activation function $\phi_{i}$, which is usually used to
introduce non-linearity into the output of a neuron. Each function $\phi_{i}$
for $i\in\\{1,...,L\\}$ is modeled using a layer of neurons, where the
function $\phi_{i}$ takes the output of the previous layer (${\bm{h}}_{i-1}$)
as input and produces ${\bm{h}}_{i}$ as the output of the current layer. The
function $\phi_{i}$ is usually chosen to be ReLU (rectified linear unit)
$\phi(v)=\max(0,v)$, but there are also many other choices. In fully connected
neural networks, the output of each layer is generated by multiplying a weight
matrix with the previous layer output and adding a bias term. However, there
could be other designs for the weights. For example, in a convolutional neural
network (CNN), the weight $\bm{\theta}_{i}$ is a kernel matrix. The kernel
matrix scans over the input matrix, from left to right and from top to bottom,
and each one of the values within the kernel is multiplied by the input value
on the same position to generate the output matrix; see Figure 2 for an
example. More details of convolutional neural network design can be found in
LeCun _et al._ (1998). In general, a DNN can be expressed as:

$\displaystyle
f({\bm{x}})=\phi_{L}(\bm{\theta}_{L},\phi_{L-1}(\bm{\theta}_{L-1},...\phi_{2}(\bm{\theta}_{2},\phi_{1}(\bm{\theta}_{1},{\bm{x}})))).$

As illustrated in Figure 3, a deep classifier is a machine learning model that
uses a hierarchical composition of $L$ parametric functions to model an input
${\bm{x}}$. Depending on the loss applied to the final layer, a neural network
can easily handle both regression and classification tasks. For regression,
there is only one output unit at the end of the network. For multi-class
classification, there are $K$ output units, where $K$ is the number of
classes.

Figure 2: Convolutional operation with stride (step size) equals to 1. Figure
3: DNN Classifier: the model takes an image of a handwritten digit as input
and predicts the probability of it being in one of the $K=10$ classes for
digits $0$ to $9$ (Papernot _et al._ , 2017).

For the classification task, the cross-entropy loss is widely used:

$\displaystyle\ell({\bm{x}},y)=-\sum_{i=1}^{K}\mathbbm{1}_{\\{i=y\\}}\log(f({\bm{x}})_{i}).$

Given a dataset with many known input-label pairs $({\bm{x}},y)$, the DNN will
adjust its parameters ($\\{\bm{\theta}_{1},...,\bm{\theta}_{n}\\}$) to reduce
the prediction error between the prediction and the correct label. The generic
approach to minimize the prediction loss is gradient descent, called back-
propagation in this setting (Hecht-Nielsen, 1992). Because of the
compositional form of the model, the error gradient is derived using the chain
rule for differentiation. The gradient is computed by a two-pass algorithm,
where a forward pass is used to compute the loss and a backward pass is
carried out to derive the gradient. In backward pass, error gradients with
respect to network parameters are successively propagated from the network’s
output layer to its input layer. For large datasets, stochastic gradient
descent is usually applied, where the gradient descent is done on a random
subset of data.

At the testing stage, the DNN with fixed parameters is used to predict the
label of unseen input. The output of the DNN is a probability vector
$f({\bm{x}})$, and the prediction is
$c({\bm{x}})=\operatornamewithlimits{argmax}\limits_{i}f({\bm{x}})_{i}$.

DNNs achieve state-of-the-art performance in the task of image classification.
For example, an vision transformer network (Dosovitskiy _et al._ , 2021) can
achieve over $99\%$ accuracy when classifying natural images from the
benchmark dataset CIFAR10 (Krizhevsky and Hinton, 2009). Ideally, we expect
the deep classifier to generalize well, making accurate predictions for inputs
outside of the domain explored during training. However, recent studies on
adversarial attack show that carefully crafted adversarial examples can easily
fool the model.

##  3 Attack

Figure 4: Illustration of adversarial attacks.

Adversarial attacks are processes to generate an adversarial example based on
a given natural sample and the victim model. Figure 4 illustrates this process
of generating adversarial examples. Here ${\bm{x}}_{0}$ denotes the natural
input, and the DNN can correctly predict its label $y_{0}$. An adversarial
attack aims to find a small perturbation $\bm{\delta}$ such that the
adversarial example ${\bm{x}}^{*}={\bm{x}}_{0}+\bm{\delta}$, which looks
similar to ${\bm{x}}_{0}$ to humans, will be mis-classified by the victim
model.

Multiple attack methods have been introduced for the crafting of adversarial
examples to attack various DNNs. In general, there are two kinds of attack
goals: targeted and untargeted. For untargeted attack, the attack is
successful if the input is predicted with any wrong label. Take Figure 1 as
example. As long as the bagel image is not classified as bagel, the attack is
successful. For targeted attack, the attack is successful only when the
adversarial example is classified as the target class. In this example, if the
target class is piano, the attack is successful only when the right side image
is labeled as piano by the victim classifier.

Based on the information needed, the attack methods can be grouped into three
categories: (1) gradient-based, (2) score-based, and (3) decision-based. Most
of these methods can perform both targeted and untargeted attacks. In general,
one attack method belongs to one of the three categories, while recently it
has been shown that an ensemble of attacks from multiple categories can
potentially lead to a stronger attack (Croce and Hein, 2020).  If all the
information of the victim model, such as model structure, parameters and so
on, is revealed to the attacker, the scenario is called white box setting. If
only the predicted scores are available, the scenario is called soft-label
black box setting. If only the predicted labels are revealed, the scenario is
called hard-label black box setting. There are also gray box settings, where
part of the model information is available. Notice that the part of the model
information that is available/unavailable is problem dependent. For example,
Yang _et al._ (2020a) defined the gray box as the scenario where the attacker
has access to the classifier but not the design of the adversarial example
detector. We will thus focus on white-box and black-box attacks in this paper.
The next three subsections present representative examples and deeper
discussions of the three categories of attack methods.

###  3.1 Gradient-Based Attack

Many existing attack methods fall in this category. These methods leverage the
gradients of the loss w.r.t. the input to form adversarial examples. For
instance, the Fast Gradient Sign Method (FGSM) (Goodfellow _et al._ , 2015)
generates adversarial examples based on the sign of gradients, and uses a step
size to control the $\ell_{\infty}$ norm of perturbation. The Basic Iterative
Method (BIM) (Kurakin _et al._ , 2016) and Projected Gradient Descent Attack
(PGD) (Madry _et al._ , 2018) can be viewed as iterative versions of FGSM. PGD
crafts adversarial examples by running FGSM for a fixed number of iterations
or until mis-classification is achieved. The most effective gradient-based
adversarial attack methods to date are C$\&$W (Carlini and Wagner, 2017b) and
PGD. Both C$\&$W attack and PGD have been frequently used to benchmark the
defense algorithms due to their effectiveness (Athalye _et al._ , 2018). Since
gradient information is required to perform attack, gradient-based attack is
mainly for the white-box setting. The process of crafting adversarial examples
can be formulated as an optimization problem. Depending on the optimization
formulations of the attack methods, gradient-based attack methods can be
further divided into two sub-categories.

Constraint-optimization formulation based methods: The first sub-category
consists of constraint-optimization formulation based methods. Given a model
with a fixed parameter $\bm{\theta}$ and an input pair $({\bm{x}}_{0},y_{0})$,
the process of generating adversarial examples ${\bm{x}}^{*}$ can be described
by the following optimization problem:

${\bm{x}}^{*}={\bm{x}}_{0}+\bm{\delta}\quad\mbox{with}\quad\bm{\delta}=\arg\max_{\bm{\delta}\in\mathcal{S}}L(\bm{\theta},{\bm{x}}_{0}+\bm{\delta},y_{0}),$
(1)

where $L$ is the loss function, $\bm{\delta}$ is the adversarial perturbation,
and $\mathcal{S}\in\mathbb{R}^{d}$ is the set of allowed perturbations,
usually chosen to be
$\mathcal{S}=\\{\bm{\delta}|\|\bm{\delta}\|_{\infty}\leq\epsilon\\}$ for some
$\epsilon$.

The optimization goal is to search for the adversarial perturbation that leads
to wrong classification. Therefore, for untargeted attack, the loss function
$L$ can be the loss function used to train the classifier, while for targeted
attack, other loss functions designed for the target class should be used. By
maximizing (1), the attacker is forcing the classifier to err on the
classification task. To guarantee that the perturbation is not too large so
that ${\bm{x}}_{0}$ and ${\bm{x}}^{*}$ are indistinguishable to human, the
search space is restricted to be within the $\epsilon$-ball around the input.

The optimization problem defined in (1) can be reformulated as

$\color[rgb]{0,0,0}\arg\max_{\bm{\delta}\in\mathcal{S}}L(\bm{\theta},{\bm{x}}_{0}+\bm{\delta},y_{0})=\operatornamewithlimits{argmax}_{\|\bm{\delta}\|_{\infty}\leq\epsilon}L(\bm{\theta},{\bm{x}}_{0}+\bm{\delta},y_{0}),$
(2)

which can be solved by gradient descent. Projected-Gradient Descent Attack
(PGD) crafts adversarial examples by solving this optimization problem with
the projected gradient descent method. It is proposed by Madry _et al._
(2018), which finds adversarial examples in an $\epsilon$-ball of the input.
The PGD attack updates in the direction that decreases the probability of the
original class most, then projects the result back to the $\epsilon$-ball of
the input. The updating equation of the PGD attack is:

$\displaystyle{\bm{x}}^{t+1}=\Pi_{\epsilon}\left\\{{\bm{x}}^{t}+\alpha\cdot\text{sign}\Big{(}\nabla_{\bm{x}}L(\bm{\theta},{\bm{x}}^{t},y)\Big{)},{\bm{x}}_{0}\right\\},$
(3)

where ${\bm{x}}_{0}$ is the original input, ${\bm{x}}^{t}$ is the updated
input in step $t$, $\epsilon$ controls the maximum distortion, $\alpha$ is the
step size and $\nabla_{\bm{x}}L(\bm{\theta},{\bm{x}}^{t},y)$ represents the
gradient of classification loss $L(\cdot)$ w.r.t. input ${\bm{x}}^{t}$.
Basically, PGD attack constructs an adversarial example by adding or
subtracting a small error term, $\alpha$, to each input dimension. The
decision of adding or subtracting an error term depends on whether the sign of
the gradient for an input dimension is positive or negative. Then the result
from previous step is projected to the $\epsilon$-ball around the original
input. If (3) is run for only one step, it is equivalent to the Fast Gradient
Sign Method (FGSM) (Goodfellow _et al._ , 2015). If it is run for multiple
steps, it is the PGD attack or the Basic Iterative Method (BIM) (Kurakin _et
al._ , 2016). Recently, Croce and Hein (2020) introduced a new attack method
called Autoattack, which combines two extensions of PGD and a decision-based
attack to automatically evaluate defense methods without parameter tuning.

Regularization-optimization formulation based methods: The second sub-category
consists of regularization-optimization formulation based methods. The C$\&$W
attack proposed by Carlini and Wagner (2017a) is a representative one, and is
by far one of the strongest attack methods. It can perform both targeted and
untargeted attacks, which are formulated as the following optimization
problem:

$\displaystyle{\bm{x}}^{*}=\operatornamewithlimits{argmin}\limits_{\bm{x}}\left\\{\|{\bm{x}}-{\bm{x}}_{0}\|_{2}^{2}+cg({\bm{x}})\right\\},$
(4)

where the first term enforces small distortion of the original input and the
second term is the loss function that measures the success of the attack. The
parameter $c>0$ controls the trade-off between distortion and attack success.

If it is for untargeted attack, which means the attacker only wants the
classifier to make a mistake and does not care the predicted label of the
adversarial example, then $g({\bm{x}})$ is defined as:

$\displaystyle g({\bm{x}})=\max\\{f({\bm{x}})_{y_{0}}-\max\limits_{i\neq
y_{0}}f({\bm{x}})_{i},0\\},$

where $y_{0}$ denotes the true label of input ${\bm{x}}_{0}$,
$f({\bm{x}})_{y_{0}}$ represents the score of predicting input ${\bm{x}}$ with
label $y_{0}$, and $f({\bm{x}})_{i}$ represents the score of predicting input
${\bm{x}}$ with label $i$. Minimizing $g({\bm{x}})$ will make the prediction
score of true class $y_{0}$ smaller than the prediction scores of classes
other than $y_{0}$. Therefore, the adversarial example ${\bm{x}}^{*}$ will be
classified to a wrong class.

If the attacker wants the adversarial example to be classified into a specific
target class with label $t$, where $t\neq y_{0}$, then $g({\bm{x}})$ is
defined as:

$\displaystyle g({\bm{x}})=\max\\{\max\limits_{i\neq
t}f({\bm{x}})_{i}-f({\bm{x}})_{t},0\\}.$

The above targeted attack loss function will push the prediction score of the
target class $t$ to be higher than those of other classes. Carlini and Wagner
(2017a) showed that their attack can successfully bypass ten different defense
methods designed for detecting adversarial examples.

Inspired by an elastic-net method proposed by Zou and Hastie (2005), Chen _et
al._ (2018) proposed the elastic-net attack (EAD), which also belongs to this
sub-category of regularization-optimization formulation based methods. EAD
formulates the process of generating adversarial examples as an elastic-net
regularized optimization problem, which can be viewed as extended version of
the C&W Attack. Their formulation of elastic-net attacks to DNNs for crafting
an adversarial example with respect to a labeled sample is as follows:

$\displaystyle{\bm{x}}^{*}=\operatornamewithlimits{argmin}\limits_{\bm{x}}\left\\{\|{\bm{x}}-{\bm{x}}_{0}\|_{2}^{2}+\beta\|{\bm{x}}-{\bm{x}}_{0}\|_{1}+cg({\bm{x}})\right\\},$

where $g({\bm{x}})$ is the same as the one defined in C&W attack.  The authors
showed that EAD can improve attack transferability and complement adversarial
training.

We note that most gradient-based attack methods fall into the above two sub-
categories, but there are also other kinds of gradient-based attack methods,
such as Jacobian-based Saliency Map Attack (Papernot _et al._ , 2016),
Deepfool (Moosavi-Dezfooli _et al._ , 2016), Network for Adversary Generation
(Mopuri _et al._ , 2018) and Adversarial Transformation Networks (Baluja and
Fischer, 2018). All of these methods perform adversarial attacks based on the
gradient information.

Choice of perturbation set in adversarial attacks.  Both constraint and
regularization optimization based formulations require a distance measurement
to quantify the difference between the original example and the perturbed
example. Despite $\ell_{\infty}$ norm being used for PGD-based attacks (1) and
the $\ell_{2}$ norm being widely used in C&W based attacks (4), in general we
can use any $\ell_{p}$ norm for both constraint and regularization based
formulations. For example, $\ell_{1}$ norm and a mixed of different norms have
been used in (Chen _et al._ , 2018). Further, recently it has been recognized
that $\ell_{p}$ norm based attacks may not be realistic, so researchers have
started to explore more flexible perturbation sets. For instance, Wong _et
al._ (2019) uses Wasserstein distance to measure the perturbation strength,
and Wong and Kolter (2020) proposed learning the perturbation sets based on
human perception.

###  3.2 Score-Based Attack

In reality, detailed model information, such as the gradient, may not be
available to the attackers. The score-based attack methods do not require
access to gradients. They perform adversarial attacks based on the output
scores $f({\bm{x}})_{i}$’s of the victim classifier. For example, Chen _et
al._ (2017) proposed a method to estimate the gradient with score information
and craft adversarial examples with the estimated gradient. Ilyas _et al._
(2018) introduced a method which leverages natural evolutionary strategy to
estimate the gradient and generate adversarial examples. In general, score-
based attacks can be divided into two sub-categories.

Gradient-approximation based methods: The first sub-category are gradient-
approximation based methods. As the name suggests, these methods first
approximate the gradients or the signs of the gradients, then generate
adversarial examples using the approximated information.

Zeroth Order Optimization Based Attack (ZOO) proposed by Chen _et al._ (2017)
uses a finite difference method to approximate the gradient of the loss w.r.t.
the input. Then the C$\&$W attack is applied to generate an adversarial
example. It uses the following formula to estimate the gradient:

$\displaystyle\frac{\partial
f({\bm{x}})}{\partial{\bm{x}}_{(i)}}\approx\frac{f({\bm{x}}+h{\bm{e}}_{i})-f({\bm{x}}-h{\bm{e}}_{i})}{2h},$

where $h$ is a small constant and ${\bm{e}}_{i}$ is a standard basis vector
with only the $i$th component as $1$, and $i$ ranges from $1$ to the dimension
of input. The time used to estimate the gradients grows with the dimension.
When the dimension of the input is large, the authors introduced several
techniques to scale-up the estimation, making it possible for the method to
craft adversarial examples in reasonable time for large DNNs trained on large
datasets, such as ImageNet (Deng _et al._ , 2009).

Many other methods are introduced to efficiently approximate the gradient and
generate adversarial examples based on it, such as NES attack (Ilyas _et al._
, 2018) and Bandits Attack (Ilyas _et al._ , 2019). NES attack uses natural
evolutionary strategies (hence the acronym NES) to estimate the gradient of
loss w.r.t. the input then generates adversarial examples based on the
estimated gradient:

$\displaystyle\frac{1}{\sigma
n}\sum_{i=1}^{n}\bm{\delta}_{i}f({\bm{x}}+\sigma\bm{\delta}_{i}),$

where $n$ is the number of searches to estimate gradient, $\bm{\delta}_{i}$ is
random direction sampled from $\mathcal{N}(0,I)$ and $\sigma$ is the standard
deviation of the search step. The authors also extended the method to a
partial-information setting, where only part of the scores or top-$k$ sorted
labels are given.

In fact, the most important information for adversarial attack is the sign of
the gradient, which provides the direction of optimization. Many methods are
proposed to estimate the sign of the gradient and use that information to
craft adversarial examples; notable examples include SignSGD (Liu _et al._ ,
2018a) and Sign hunter (Al-Dujaili and O’Reilly, 2020). To further improve the
efficiency of gradient approximation or sign estimation, some researchers
propose using a substitute model, which is a model trained with the same data
as the victim model and performs similarly to the victim model. Representative
examples include Subspace attack (Guo _et al._ , 2019b) and Transfer-based
prior (Cheng _et al._ , 2019a). Also, Wang _et al._ (2020a) shows how to
exploit the data distribution to identify importance subspace for black-box
attacks. These attack methods also belong to the gradient-approximation based
category since their key element is to approximate gradient information
efficiently.

Other methods: The second sub-category of score-based methods consists of
methods that do not approximate gradient related information to generate
adversarial examples. For example, Li _et al._ (2019) proposed the Gaussian
black-box adversarial attack (Nattack), which searches the adversarial
examples by modeling the adversarial population with a Gaussian distribution.
The intuition comes from the fact that one can find various adversarial
examples for the same input using different attack methods, which suggests
there is a population of adversarial examples. Other methods that fall into
this sub-category include the GenAttack (Alzantot _et al._ , 2018), the Simple
Black-Box Attack (Guo _et al._ , 2019a) and Square attack (Andriushchenko _et
al._ , 2020).

###  3.3 Decision-Based Attack

In many practical situations, the attacker has access only to the predicted
labels of the model, but not any gradient or score information. When only the
predicted label $c({\bm{x}})$ is available, both gradient-based and score-
based methods do not work. Papernot _et al._ (2017) introduced a transfer
attack method which requires only the observations of the labels predicted by
the model. The main idea is to train a substitute model which is similar to
the target model and attack the substitute model instead. Boundary Attack was
subsequently proposed by Brendel _et al._ (2018), which searches for
adversarial examples based on random-walk on the decision boundary. Many
extensions of Boundary Attack have been proposed to improve the efficiency and
performance of it (Brunner _et al._ , 2019; Chen and Jordan, 2019; Chen _et
al._ , 2020; Guo _et al._ , 2020). There are also decision-based attack
methods that are neither transfer-based nor random-walk based (Ilyas _et al._
, 2018; Cheng _et al._ , 2020a; Guo _et al._ , 2019b). In general, the goal of
decision-based attack is to generate adversarial examples with the predicted
labels returned by the victim model. Decision-based attack methods can be
divided into three sub-categories.

Transfer based attacks: The first category consists of transfer-based attack
methods. Various researchers have observed that, if two DNNs are trained with
similar data, even though the two models may have very different structures,
adversarial examples generated on one model can be used to fool another one.
Based on this observation, Papernot _et al._ (2017) proposed to train a
substitute model based on a small amount of training data and generate
adversarial examples based on the substitute model. They showed that the
adversarial examples generated on the substitute model can also fool the
target classifier. The proposed method does not require too many samples to
train the substitute model and can achieve relatively high success rate in
untargeted task. However, the method does not work well in targeted task. Liu
_et al._ (2017) proposed an ensemble-based approach to generating transferable
adversarial examples. This ensemble approach increases the proportion of
target adversarial examples transferring with their target labels.

Random-walk based attacks: The second category consists of methods that are
based on random walk on the boundary. Brendel _et al._ (2018) proposed
Boundary Attack, a method that does not rely on the gradient of loss w.r.t.
the input and performs well under both targeted and untargeted settings. The
method starts with a sample categorized in the targeted class and seeks to
minimize the perturbation while staying adversarial. The process of boundary
attack is shown in Figure 5 and details of the method are described in
Algorithm 1.

Algorithm 1 Boundary Attack

1:Input: decision model $c(\cdot)$, original input ${\bm{x}}$

2:Output: ${\bm{x}}^{adv}$

3:Initialization: $t=0$, $\tilde{{\bm{x}}}^{0}\sim\mathcal{U}(0,1)$ s.t.
$c(\tilde{{\bm{x}}}^{0})=$target class

4:while $t<T$ do

5: $t=t+1$

6: draw random perturbation from proposal distribution
$\bm{\eta}^{t}\sim\mathcal{P}(\tilde{{\bm{x}}}^{t-1})$

7: if $c(\tilde{{\bm{x}}}^{t-1}+\bm{\eta}^{t})=$target class then

8: set $\tilde{{\bm{x}}}^{t}=\tilde{{\bm{x}}}^{t-1}+\bm{\eta}^{t}$

9: else

10: set $\tilde{{\bm{x}}}^{t}=\tilde{{\bm{x}}}^{t-1}$

11: end if

12:end while

13:Return $\tilde{{\bm{x}}}^{T}$

Note that $c({\bm{x}})$ is the label assigned to the input ${\bm{x}}$ by the
classifier, and $T$ is the maximum number of iterations.

Define input space as $\mathcal{D}$, the process of drawing $\bm{\eta}^{t}$
from the proposal distribution is as follows:

  * •

Draw $\bm{\eta}^{t}\sim\mathcal{N}(\bm{0},\bm{I})$.

  * •

Rescale and clip $\bm{\eta}^{t}$ to make sure
$\tilde{{\bm{x}}}^{t-1}+\bm{\eta}^{t}\in\mathcal{D}$ and
$\|\bm{\eta}^{t}\|_{2}=\delta\cdot d({\bm{x}},\tilde{{\bm{x}}}^{t-1})$, where
$d(\cdot,\cdot)$ represents distance function between two samples and $\delta$
is a hyper-parameter that controls the scale of the perturbation.

  * •

Orthogonal perturbation: project $\bm{\eta}^{t}$ onto a sphere around the
original input ${\bm{x}}$ such that
$d({\bm{x}},\tilde{{\bm{x}}}^{t-1}+\bm{\eta}^{t})=d({\bm{x}},\tilde{{\bm{x}}}^{t-1})$.

  * •

Move towards the original input such that
$\tilde{{\bm{x}}}^{t-1}+\bm{\eta}^{t}\in\mathcal{D}$ and
$d({\bm{x}},\tilde{{\bm{x}}}^{t-1})-d({\bm{x}},\tilde{{\bm{x}}}^{t-1}+\bm{\eta}^{t})=\epsilon\cdot
d({\bm{x}},\tilde{{\bm{x}}}^{t-1})$ both hold, where $\epsilon$ controls the
step size of the movement.

Figure 5: An illustration of Boundary Attack. In essence the Boundary Attack
performs rejection sampling along the boundary between adversarial and non-
adversarial images (Brendel _et al._ , 2018).

Boundary Attack achieves performance comparable to state-of-the-art white-box
attacks on DNNs trained for classification. Chen and Jordan (2019) introduced
Boundary Attack$++$, using binary information at the decision boundary to
estimate gradient direction. They showed that Boundary Attack$++$ requires
significantly fewer model queries than Boundary Attack. Guo _et al._ (2020)
proposed a decision-based black-box attack, which improves the query-
efficiency of Boundary Attack by restricting the search for adversarial
examples to a low frequency domain. There are many other methods belong to
this category, such as Guessing Smart (Brunner _et al._ , 2019).

Optimization based attacks:  Instead of conducting random walks, a process
which does not have any convergence guarantees, more recently researchers have
found that attacks in the decision-based setting can also be formulated as
solving a zeroth order optimization problem (Cheng _et al._ , 2019b). They
showed that both PGD and C&W loss are ill-defined in the decision-based
setting, and instead we need to redefine the problem as finding the best
direction $\bm{\theta}$ of adversarial examples. Given ${\bm{x}}_{0}$, a
function $g(\cdot)$ that measures the distance between ${\bm{x}}_{0}$ and the
decision boundary along direction $\bm{\theta}$ can be defined as

$g(\bm{\theta})=\arg\min_{\lambda>0}\big{(}f({\bm{x}}_{0}+\lambda\frac{\bm{\theta}}{\|\bm{\theta}\|})\neq
y_{0}\big{)}.$ (5)

The attack problem can then be posed as

$\bm{\theta}^{*}=\arg\min_{\bm{\theta}}g(\bm{\theta}),$ (6)

and the adversarial example that is closest to ${\bm{x}}_{0}$ is
${\bm{x}}^{*}={\bm{x}}_{0}+g(\bm{\theta}^{*})\frac{\bm{\theta}}{\|\bm{\theta}\|}$.
Although the gradient of $g(\bm{\theta})$ cannot be directly computed, the
function value of $g(\bm{\theta})$ can be computed by binary search, thus
standard zeroth order optimization solvers can be applied for solving (6). In
OPT attack (Cheng _et al._ , 2019a), Randomized Gradient-Free (RGF) method
(Nesterov and Spokoiny, 2017) is used to address the problem. Later, Cheng _et
al._ (2020a) showed that the gradient sign of (6) can be computed in a more
query-efficient way, leading to an improved attack called Sign-OPT. On the
other hand, Chen _et al._ (2020) proposed another optimization-based
formulation, leading to yet another query-efficient algorithm called
HotSkipJump Attack. Recently, Chen and Gu (2020) proposed RayS, which
reformulates the continuous optimization problem in Cheng _et al._ (2019a)
into a discrete one for $\ell_{\infty}$ norm attack.

##  4 Defense

There has been extensive research on improving the robustness of DNNs for
defending against adversarial examples. In general, methods that aim to
increase model robustness fall into the following four main categories: (1)
augmenting the training data with adversarial examples, (2) leveraging
randomness to defend against adversarial attacks, (3) removing adversarial
perturbations with projection, and (4) detecting the adversarial examples
instead of classifying them correctly.

###  4.1 Adversarial Training

To improve the robustness of a classifier against adversarial examples, one
natural idea is to train the model with adversarial examples. Goodfellow _et
al._ (2015) proposed generating adversarial examples with FGSM attack and
adding them back to the training data set to improve the robustness of the
model. Later Kurakin _et al._ (2016) suggested using a multi-step FGSM to
further improve the adversarial robustness.

Madry _et al._ (2018) introduced a min-max formulation, which iteratively
generates adversarial examples with PGD attack while training the model.
Athalye _et al._ (2018) showed that Madry’s adversarial training can survive
strong attacks while many other state-of-the-art defense methods cannot.
Instead of minimizing the original loss function $L(\bm{\theta},{\bm{x}},y)$,
the following min-max objective function is used:

$\displaystyle\min\limits_{\bm{\theta}}\max\limits_{\bm{\delta}\in
S}L(\bm{\theta},{\bm{x}}+\bm{\delta},y).$

The inner maximization problem aims to find an adversarial perturbation of a
given data point ${\bm{x}}$ within the perturbation set $S$ that achieves a
high loss. Indeed, this is precisely the problem of attacking any given neural
network. On the other hand, the goal of the outer minimization problem is to
find the model parameters that minimize the “adversarial loss” given by the
inner attack problem. Therefore, the model is trained with the adversarial
example in the $\epsilon$-ball of the original sample. Recently, it has been
shown that such min-max optimization is guaranteed to converge under certain
assumptions (Gao _et al._ , 2019).

Many defense methods based on Madry’s adversarial training have been proposed
to further improve its efficiency and performance. For example, Zhang _et al._
(2019b) proposed TRADES, a theoretically-driven upper bound minimization
algorithm that achieved the top-1 rank in the NeurIPS 2018 defense
competition.

In other methods, researchers treated mis-classified examples and correctly
classified examples as the same when performing adversarial training. Ding _et
al._ (2020) noticed the importance of mis-classified examples and improved the
performance of adversarial training by combining the usual cross-entropy loss
with a margin maximization loss term applied to the correctly classified
examples. Wang _et al._ (2020b) found that mis-classified examples have more
impact on the final robustness than correctly classified examples and
incorporate mis-classified examples in adversarial training as a regularizer.

Other than just adding adversarial examples into the training process, Wang
_et al._ (2019) studied the convergence quality of adversarial examples found
in the inner maximization and proposed a dynamic adversarial training method
that changes adversarial strength in inner maximization according to
convergence quality. The convergence of adversarial training has also been
studied in many other works (e.g., Gao _et al._ , 2019).

Adversarial training performs well against attacks but generating adversarial
examples during training could be expensive. Consequently, various researchers
have developed methods for reducing the computational overhead brought by
adversarial training (Shafahi _et al._ , 2019; Zhang _et al._ , 2019a; Wong
_et al._ , 2020). Many extensions of adversarial training have also been
proposed to improve the performance of adversarial training (Carmon _et al._ ,
2020; Zhang _et al._ , 2019b; Gowal _et al._ , 2020; Wu _et al._ , 2020; Pang
_et al._ , 2020, 2021).

###  4.2 Randomization

Another group of effective defense methods leverages randomness to defend
against adversarial examples. Adversarial perturbation can be viewed as noise,
and various methods have been proposed to improve the robustness of DNNs by
incorporating random components into the model. The randomness can be
introduced to:

  * •

the input; i.e., randomizing the input of a neural network to remove the
potential adversarial perturbation (e.g., Xie _et al._ , 2018; Cohen _et al._
, 2019),

  * •

the hidden layer output; i.e., adding Gaussian noise to the input and hidden
output (e.g., Liu _et al._ , 2018b) and introducing pruning methods to
randomize the network output (e.g., Dhillon _et al._ , 2018), or

  * •

the parameters of the classifier; i.e., leveraging a Bayesian component to add
randomness to the weights of the model (e.g., Liu _et al._ , 2019).

The robustness of neural networks against random noise has been analyzed both
theoretically (Fawzi _et al._ , 2016; Franceschi _et al._ , 2018) and
empirically (Dodge and Karam, 2017).

Xie _et al._ (2018) introduced a simple preprocessing method to randomize the
input of neural networks, hoping to remove the potential adversarial
perturbation. During the testing phase, the input is randomly resized to
several sizes, then around each of the resized inputs, zeros are randomly
padded. The authors demonstrated that this simple method can be applied to
large-scale datasets, such as Imagenet. Similarly, Zantedeschi _et al._ (2017)
showed that, by using a modified ReLU activation layer (called BReLU) and
adding noise to the origin input to augment the training data, the learned
model will gain some stability to adversarial examples. However, other
researchers have found that this defense method is not robust against strong
white-box attack, such as PGD and C$\&$W (Carlini and Wagner, 2017b).

Instead of adding random components to the input space, Liu _et al._ (2018b)
proposed a “noise layer” to introduce randomness to both the input and the
hidden layer output. In this “noise layer”, randomly generated Gaussian noise
is added to the input:

${\bm{x}}\leftarrow{\bm{x}}+\bm{\varepsilon},\quad\bm{\varepsilon}\sim\mathcal{N}(0,\sigma^{2}I),$

where $\sigma$ is a hyper-parameter. Larger values of $\sigma$ lead to better
robustness but worse prediction accuracy, while smaller values of $\sigma$
result in better prediction accuracy but deteriorated robustness. The noise
layer is applied in both training and testing phases, so the prediction
accuracy will not be largely affected. The authors showed that Random Self-
Ensemble (RSE) is equivalent to an ensemble possessing an infinite number of
noisy models without any additional memory overhead. However, the experiments
demonstrated that, although RSE can increase the robustness of DNNs, it also
sacrifices a non-negligible amount of accuracy. Later, Liu _et al._ (2019)
introduced a new min-max formulation to combine adversarial training with
Bayesian Neural Networks (BNNs). All weights in a Bayesian neural network are
represented by probability distributions over possible values, rather than
having a single fixed value. The proposed framework, called Adv-BNN, combines
adversarial training with randomness, is shown to have significant improvement
over previous approaches including RSE and Madry’s adversarial training.

Although these methods demonstrate improvement of robustness against
adversarial examples, they typically lack theoretical guarantee for their
performances. Lecuyer _et al._ (2019); Li _et al._ (2018a); Cohen _et al._
(2019) introduced a method called Randomized Smoothing and proved a tight
robustness guarantee in the $\ell_{2}$ norm. The details of Randomized
Smoothing are summarized below:

  * •

Training stage:

    1. 1.

Train the base classifier with Gaussian data augmentation with variance
$\sigma^{2}$. That is, the base classifier is trained on the natural examples
and noisy examples, where the noisy examples are generated by adding Gaussian
noise to natural examples.

    2. 2.

The trained base classifier is denoted as $f$.

  * •

Testing stage: for a given input testing sample ${\bm{x}}$:

    1. 1.

Generate $n$ noise corrupted samples by adding Gaussian noise with variance
$\sigma^{2}$ to the input ${\bm{x}}$.

    2. 2.

Feed the $n$ noise corrupted samples into the trained base classifier $f$ and
obtain the top-2 frequent predicted classes from the predictions. The top-2
frequent classes are denoted as $\hat{c}_{A},\hat{c}_{B}$ and the
corresponding frequencies are denoted as $n_{A},n_{B}$.

    3. 3.

If BINOMPVALUE$(n_{A},n_{A}+n_{B},0.5)\leq\alpha$, the final predicted class
is $\hat{c}_{A}$, otherwise abstain. The function
BINOMPVALUE$(n_{A},n_{A}+n_{B},p)$ returns the $p$-value of the two-sided
hypothesis test that $n_{A}\sim$Binomial$(n_{A}+n_{B},p)$ and $\alpha$ is the
abstention threshold generated based on Hung _et al._ (2019).

They also provided a process to generate certificated radius around the input.
Theoretical analysis was provided to show that the classifier is robust around
the input within the found $\ell_{2}$ radius.

###  4.3 Projection

The development of generative models, such as auto-encoders and generative
adversarial networks, gives rise to another line of research, which removes
adversarial noise by fitting generative models on the training data (Jalal _et
al._ , 2017; Meng and Chen, 2017; Li _et al._ , 2018b; Samangouei _et al._ ,
2018). These defense mechanisms employ the power of generative models to
combat the threat from adversarial examples. The input to the classifier is
first fed into the generative model and then classified. Since the generative
model is trained on natural examples, adversarial examples will be projected
to the manifold learned by the generative model. Furthermore, “projecting” the
adversarial examples onto the range of the generative model can have the
desirable effect of reducing the adversarial perturbation.

Meng and Chen (2017) introduced MagNet as a robust framework against
adversarial examples. It trains multiple auto-encoders to move adversarial
examples closer to the manifold of natural examples. An auto-encoder is a type
of neural network that consists of two major parts: the encoder that maps the
input to a low-dimensional space and the decoder that recovers the input from
the low-dimensional embedding. The auto-encoder is usually trained on the
reconstruction loss w.r.t. the input. Therefore, the high-dimensional data is
summarized by the low-dimensional embedding through the training process. In
MagNet, one auto-encoder is chosen at random at testing time to filter the
input samples, and thus adversarial perturbation could potentially be removed
through this encoding and decoding process.

Instead of using auto-encoders, Samangouei _et al._ (2018) proposed to train a
Generative Adversarial Networks (GAN) (Goodfellow _et al._ , 2015) to fit the
training data distribution and help remove adversarial noise. It first trains
a GAN to model the distribution of the training data. Then, at inference time,
it finds an output of the GAN that is close to the input, and feeds that
output into the classifier. This process “projects” input samples onto the
range of the GAN’s generator, which can potentially help remove the effect of
adversarial perturbations. Here the GAN is proposed to estimate the generative
model through an adversarial process, in which two major parts are
simultaneously trained: a generative model ${\bm{G}}$ that captures the data
distribution, and a discriminative model ${\bm{D}}$ that estimates the
probability that a sample came from the training data rather than ${\bm{G}}$.
The objective of training a GAN is:

$\displaystyle\min\limits_{\bm{G}}\max\limits_{\bm{D}}\left\\{\mathbb{E}_{{\bm{x}}\sim
p_{data}({\bm{x}})}\left[\log{\bm{D}}({\bm{x}})\right]+\mathbb{E}_{{\bm{z}}\sim
p_{\bm{z}}({\bm{z}})}\left[\log(1-{\bm{D}}({\bm{G}}({\bm{z}})))\right]\right\\},$

where $p_{data}({\bm{x}})$ represents the data distribution and
$p_{\bm{z}}({\bm{z}})$ represents a prior distribution that is predefined.
Therefore, ${\bm{D}}$ and ${\bm{G}}$ together play a two-player min-max game
to recover the data distribution.

Defense-GAN is similar to MagNet as both of these methods aim to filter out
the adversarial noise by using a generative model. Samangouei _et al._ (2018)
showed that Defense-GAN is more robust than MagNet on several benchmark
datasets. There are some other works also follow similar patterns (Jalal _et
al._ , 2017; Song _et al._ , 2018).

Figure 6: A deep classifier.

All of the above methods train a separate generative model to perform the
adversarial noise filtering. Li _et al._ (2018b) proposed a different defense
framework, termed ER-Classifier, which combines the process of filtering and
classification in one framework. In fact, any deep classifier can be viewed as
a combination of these two parts as illustrated in Figure 6: an encoder part
to extract useful features from the input data and a classifier part to
perform classification based on the extracted features. Both the encoder and
the classifier are neural networks.

ER-Classifier is similar to a regular deep classifier, which first projects
the input to a low-dimensional space with an encoder ${\bm{G}}$, then performs
classification based on the low-dimensional embedding with an classifier
${\bm{C}}$. The novelty is that at the training stage, the low-dimensional
embedding of ER-Classifier is stabilized with a discriminator ${\bm{D}}$ by
minimizing the dispersion between the distribution of the embedding and the
distribution of a selected prior. The goal of the discriminator is to separate
the true code sampled from a prior and the “fake” code produced by the
encoder, while the encoder will try to produce generated code that is similar
to the true one. The result of this competition is that the distribution of
the embedding space will be pushed towards the prior distribution. Therefore
it is expected that this regularization process can help remove the effects of
any adversarial distortion and push the adversarial examples back to the
natural data manifold. Another difference is that the embedding space
dimension is much smaller for the ER-classifier, when compared with a general
deep classifier, making it easier for the training process to converge.
Details of the framework are illustrated in Figure 7. Li _et al._ (2018b)
showed that ER-Classifier achieves state-of-the-art performance against strong
adversarial attack methods on several benchmark datasets.

Figure 7: Overview of ER-Classifier framework.

###  4.4 Detection

Methods in this fourth category aim to detect the existence of adversarial
examples, rather than trying to classify them into the correct classes. The
main assumption behind these methods is that the adversarial examples came
from a different distribution when comparing to those from the natural data.
In other words, adversarial examples do not lie on the data manifold, and DNNs
classify correctly only the samples near the manifold of training data (Tanay
and Griffin, 2016). Therefore adversarial examples may be detected with
carefully designed detectors.

A straightforward way towards adversarial example detection is to build a
simple binary classifier separating the adversarial examples apart from the
clean data (Gong _et al._ , 2017). The advantage is that it serves as a
preprocessing step without imposing any assumptions on the model it protects.
However, this method suffers from a generalization limitation. In a similar
method, a small “detector” subnetwork is trained as an augmentation of DNNs on
the binary classification task to distinguish true data from adversarial
perturbations (Metzen _et al._ , 2017), where the inputs (to this detector)
are taken from intermediate layers of the classification network.
Alternatively, a cascade classifier may be built on the outputs from the
convolutional layers (Li and Li, 2017) to efficiently detect adversarial
examples. Zheng and Hong (2018) proposed modeling the output of a classifier
with Gaussian mixture models and performing hypothesis testing to detect
adversarial examples. Roth _et al._ (2019) showed that adversarial examples
exist in cone-like regions in very specific directions from their
corresponding natural inputs and proposed a new test statistic to detect
adversarial examples with the findings. In a recent work, Yang _et al._
(2020a) studied the feature attributions of adversarial examples and proposed
a detection method based on feature attribution scores.

Instead of building a detector based on input or output of the network,
another line of research proposes to study the characteristics of adversarial
examples and build detectors based on such features (Feinman _et al._ , 2017;
Grosse _et al._ , 2017; Lee _et al._ , 2018; Ma _et al._ , 2018; Raghuram _et
al._ , 2020). In order to detect adversarial examples that lie far from the
data manifold, Feinman _et al._ (2017) proposed KD-Detection to perform
density estimation on the training data in the feature space of the last
hidden layer to help detect adversarial examples.

Kernel density estimation is used to measure how far a sample is from the
provided data manifold. Suppose that ${\bm{x}}_{1},...,{\bm{x}}_{n}$ are
training samples drawn from an unknown probability density $p_{X}({\bm{x}})$.
Given ${\bm{x}}$, we can use the following function to estimate the density
score at ${\bm{x}}$:

$\displaystyle\hat{p}_{X}({\bm{x}})=\frac{1}{n}\sum_{i=1}^{n}K_{\sigma}({\bm{x}},{\bm{x}}_{i}),$

where $K_{\sigma}(\cdot,\cdot)$ is a kernel function. In the KD-Detection
framework, one kernel density model is fitted for each class. Therefore, if
${\bm{x}}$ is predicted with label $y$, the samples
$\\{{\bm{x}}_{i}\\}_{i=1}^{n}$ used to do the estimation are training samples
from class $y$. After the kernel density is fitted, both natural examples and
adversarial examples are fed into the model and density function to generate
density scores. A logistic regression model is then fitted on the kernel
density scores to detect whether the input is adversarial or not.

Many recent works fall into this category, studying the properties of
adversarial examples and detecting them with hidden layer features. Ma _et
al._ (2018) observed Local Intrinsic Dimension (LID) of hidden-layer outputs
differ between the original and adversarial examples, and leveraged these
characteristics to detect adversarial examples. In another work, Lee _et al._
(2018) generated the class conditional Gaussian distributions with respect to
hidden layer output of the DNN under Gaussian discriminant analysis, which
result in a confidence score based on the Mahalanobis distance (MAHA),
followed by a logistic regression model on the confidence scores to detect
adversarial examples. Recently, a joint statistical test pooling information
from multiple layers is proposed in Raghuram _et al._ (2020) to detect
adversarial examples.

##  5 Numerical Experiments

In this section, we show how the attack methods fool various classifiers,
including classical models like logistic regression and more recent models
like DNNs. In the first part, the results of attacking a logistic regression
model with projected gradient descent attack are shown. In the second part, we
select representative attack methods from each category (gradient-based,
score-based, and decision-based) and compare their performances. In the last
part, we select one defense method from each category (adversarial training,
randomization, projection, and detection) and compare their performances. The
following benchmark datasets are used in our experiments:

  * •

MNIST (LeCun, 1998): a handwritten digit dataset that consists of $60,000$
training images and $10,000$ testing images. These are $28\times 28$ black and
white images in ten different classes.

  * •

CIFAR10 (Krizhevsky and Hinton, 2009): a natural image dataset that contains
$50,000$ training images and $10,000$ testing images with ten different
classes. These are low resolution $32\times 32$ color images.

The input image is stored in RGB format with three channels, where the pixel
values of each channel range from $0$ to $256$. We use the $\ell_{\infty}$ and
$\ell_{2}$ distortion metrics to measure similarity between images and report
the $\ell_{\infty}$ distance in the normalized $[0,1]$ space (e.g., a
distortion of $0.031$ corresponds to $8/256$), and the $\ell_{2}$ distance as
the total root-mean-square distortion normalized by the total number of
pixels. Computing codes for carrying out the following numerical experiments
can be downloaded at https://github.com/reviewadvexample/revew_adv_defense.

###  5.1 Attacking Logistic Regression Model

We train a logistic regression model to perform binary classification on two
handwritten digits, zero and one. The model is trained on all the images with
label $0$ or $1$ in the training dataset and tested on all the images with
label $0$ or $1$ in the testing dataset. Therefore, the training and testing
sets are subsets of the training and testing sets of MNIST. There are $5,923$
handwritten-zero images and $6,742$ handwritten-one images in the training
set. For testing set, there are $980$ handwritten-zero images and $1,135$
handwritten-one images.

The loss function of the logistic regression model used is:

$\displaystyle
f({\bm{w}})=\sum_{i=1}^{n}\log(1+\exp(-y_{i}{\bm{w}}^{T}{\bm{x}}_{i})),$ (7)

where ${\bm{w}}\in\mathbb{R}^{784}$ is the weight,
${\bm{x}}_{1},{\bm{x}}_{2},...,{\bm{x}}_{n}\in\mathbb{R}^{784}$ are input
images vectorized from the original $28\times 28$ pixel space dimension, and
$y_{1},y_{2},...,y_{n}\in\\{-1,1\\}$ are labels with $-1$ corresponding to
handwritten-zero and $1$ corresponding to handwritten-one.

The projected gradient descent (PGD) method attacks the model using gradient
information. The gradient of the loss w.r.t. a certain input image
${\bm{x}}_{i}$ is:

$\displaystyle\nabla
f({\bm{x}}_{i})=-\frac{y_{i}}{1+\exp(y_{i}{\bm{w}}^{T}{\bm{x}}_{i})}{\bm{w}}.$

PGD attacks the model by repeating the following updating equation:

$\displaystyle{\bm{x}}^{t+1}=\Pi_{\epsilon}\left\\{{\bm{x}}^{t}+\alpha\cdot\text{sign}\left(\nabla
f({\bm{x}}_{i})\right),{\bm{x}}_{i}\right\\}.$

In our experiment, we set $\alpha=1$, $\epsilon=8$, which reflects $0.031$ in
$[0,1]$ scale since $\frac{8}{256}\approx 0.031$, and run the updates for $40$
steps.

The prediction accuracy under no attack is $99.95\%$. Under PGD attack, the
prediction accuracy drops to $72.96\%$. In Figure 8, we visualize four attack
results, where in each pair the image on the left side is the original image
and the image on the right side is the adversarial example based on the left
one. We observe that the adversarial examples look remarkably similar to the
original images but the predictions on these examples are altered by some
invisible adversarial noise.

Figure 8: Four examples of attack results on a logistic regression model. In
each pair of images, the left shows the original handwritten image and the
right shows the adversarial image, which is classified incorrectly by the
logistic regression model.

###  5.2 Comparison of Different Attack Methods

In this subsection we compare the performances of several attack methods
including gradient-based, score-based and decision-based. The gradient-based
attacks are compared on all samples from the test sets of the two benchmark
datasets described above. Due to the slow speed of generating adversarial
examples with score-based and decision-based approaches, the corresponding
methods are evaluated on $300$ random samples from each of the test sets.

For the dataset MNIST, we train a network consisting of two convolutional
layers with, respectively, 20 and 50 filters with kernel size 5, each followed
by $2\times 2$ max-pooling, and a fully connected layer of size 500. For
CIFAR10, we use the standard VGG-16 structure (Simonyan and Zisserman, 2014)
with batch normalization.

The attack methods tested are:

  * •

Gradient-based: FGSM (Goodfellow _et al._ , 2015) and PGD (Madry _et al._ ,
2018).

  * •

Score-based: ZOO (Chen _et al._ , 2017) and Square attack (Andriushchenko _et
al._ , 2020).

  * •

Decision-based: Boundary attack (Brendel _et al._ , 2018), OPT attack (Cheng
_et al._ , 2019a) and Sign-OPT (Cheng _et al._ , 2020a).

#### Results on gradient-based attack.

The results of attacking the two classification models on MNIST and CIFAR10
with gradient-based methods are shown in Table 1. The strength of attack is
represented by $\epsilon$, which controls the maximum distortion level. The
larger the value of $\epsilon$, the stronger the attack. It is easy to see
that PGD is stronger than FGSM since PGD runs the optimization process with
equation (3) multiple times to search for adversarial examples, while FGSM
runs the optimization for only one step.

Method | MNIST | CIFAR10
---|---|---
Clean Acc | $\epsilon=0.1$ | $\epsilon=0.2$ | $\epsilon=0.3$ | Clean Acc | $\epsilon=0.01$ | $\epsilon=0.02$ | $\epsilon=0.03$
FGSM | 98.58% | 75.98% | 17.66% | 2.63% | 93.34% | 56.46% | 50.85% | 47.61%
PGD | 98.58% | 69.42% | 6.47% | 0.02% | 93.34% | 5.71% | 0.12% | 0.01%
Table 1: Accuracy on MNIST and CIFAR10 under gradient-based attack.

#### Results on score-based attack.

The results of score-based attacks on MNIST and CIFAR10 are shown in Table 2.
For ZOO (Chen _et al._ , 2017), the maximum number of iterations (Max.iter)
limits the number of searches used to perform gradient estimation. The larger
the value, the better the approximation. Therefore, a large quantity of
maximum number of iterations usually results in better performance. Similarly
for Square attack (Andriushchenko _et al._ , 2020), the maximum number of
iterations controls the number of random-walk steps to search for adversarial
examples. Therefore, a large value usually leads to better performance. The
authors of Square attack studied the property of adversarial distortion and
leveraged it to perform the attack. The method achieves better performances
with the same maximum number of iterations when compared to the ZOO.

| Method | Clean Acc | Max.iter$=100$ | Max.iter$=200$ | Max.iter$=500$
---|---|---|---|---|---
MNIST | ZOO | 99.33% | 99.33% | 99.33% | 99.00%
Square | 99.33% | 73.33% | 34.00% | 16.67%
CIFAR10 | ZOO | 93.67% | 93.67% | 90.33% | 76.67%
Square | 93.67% | 84.67% | 70.67% | 54.33%
Table 2: Accuracy on MNIST and CIFAR10 under score-based attack.

#### Results on decision-based attack.

The results of decision-based experiments, in which only the predicted labels
are available to the attacks, are shown in Figure 9. Under this setting, only
limited information is available, making it harder to craft adversarial
examples. In Figure 9, we compare Boundary (Brendel _et al._ , 2018), OPT
attack (Cheng _et al._ , 2019a) and Sign-OPT (Cheng _et al._ , 2020a) by
visualizing the relationship between the number of queries required versus
distortion. Boundary attack is an example of random-walk based attacks and the
other two are examples of optimization based attacks. The horizontal axis of
each figure represents number of times the victim model is called to generate
adversarial example. The vertical axis represents distortion level; i.e., the
$\ell_{2}$ norm Euclidean distance between adversarial example and the
original input. We observe that, under the same distortion level, Sign-OPT and
OPT attacks require much fewer number of searches to craft adversarial
examples than Boundary attack, and Sign-OPT is more efficient than OPT attack.

(a) Decision Attacks on MNIST

(b) Decision Attacks on CIFAR10

Figure 9: Performances of decision-based attacks. Number of queries represents
the number of times victim model is called.

###  5.3 Comparison of Different Defense Methods

We first compare defense methods from the first three categories (adversarial
training, randomization and projection) and delay the last category
(detection) towards the end of this subsection. It is because detection
methods are based on a different philosophy when comparing with the first
three categories: they aim to detect the adversarial examples rather than to
classify them correctly. In other words, they are not directly comparable with
other defense methods. As before, we compare these methods with the same two
benchmark datasets.

The following methods from the first three categories are selected for
comparison:

  * •

No Defense: Model without any defense method.

  * •

Adversarial training: Madry’s Adv (Madry _et al._ , 2018) and TRADES (Zhang
_et al._ , 2019b).

  * •

Randomization: RSE (Liu _et al._ , 2018b) and Ran.Smooth (Cohen _et al._ ,
2019).

  * •

Projection: ER-CLA (Li _et al._ , 2018b).

The comparison results of the first three categories of defense methods are
shown in Figure 10. In the plot, the horizontal axis represents $\epsilon$,
the strength of the PGD attack. The larger the $\epsilon$, the stronger the
attack. The vertical axis represents the testing accuracy under the PGD
attack. For all models, the accuracy drops as the attack strength increases
except for Ran.Smooth on MNIST when the attack strength increases from $0$ to
$0.05$. We observe that TRADES and Ran.Smooth perform better on MNIST and
CIFAR10. On CIFAR10, RSE and Madry’s adversarial training perform similarly,
while TRADES performs best when attack is strong. Another point to notice is
that, when there is no attack, all the other methods perform worse than ER-
Classifier. In fact, ER-Classifier and adversarial training methods can be
combined together to further improve the results. In Li _et al._ (2018b), the
authors showed that combing ER-Classifier with adversarial training methods
outperform both Madry’s adversarial training and TRADES. All five defense
methods perform much better than the model without any defense method.

Figure 10: Comparison of Different Defense Methods on MNIST and CIFAR10

Now we compare defense methods from the detection category. The methods are:

  * •

KD: Kernel Density detection (Feinman _et al._ , 2017).

  * •

LID: Local Ontrinsic Dimensionality detection (Ma _et al._ , 2018).

  * •

ODD: Odds are odd detection (Roth _et al._ , 2019).

  * •

ReBeL: Reading Between the Layers (Raghuram _et al._ , 2020).

Data | Metric | C$\&$W | FGSM | PGD
---|---|---|---|---
KD | LID | ODD | ReBeL | KD | LID | ODD | ReBeL | KD | LID | ODD | ReBeL
CIFAR10 | AUC | 0.945 | 0.947 | 0.955 | 0.968 | 0.873 | 0.957 | 0.968 | 0.990 | 0.791 | 0.777 | 0.963 | 0.962
<EMAIL_ADDRESS>| 0.068 | 0.220 | 0.591 | 0.309 | 0.136 | 0.385 | 0.224 | 0.698 | 0.018 | 0.093 | 0.059 | 0.191
<EMAIL_ADDRESS>| 0.464 | 0.668 | 0.839 | 0.726 | 0.401 | 0.753 | 0.709 | 0.974 | 0.148 | 0.317 | 0.819 | 0.789
<EMAIL_ADDRESS>| 0.911 | 0.856 | 0.901 | 0.954 | 0.572 | 0.875 | 1.000 | 1.000 | 0.285 | 0.448 | 0.999 | 0.999
MNIST | AUC | 0.932 | 0.785 | 0.968 | 0.980 | 0.933 | 0.888 | 0.952 | 0.992 | 0.801 | 0.861 | 0.967 | 0.975
<EMAIL_ADDRESS>| 0.196 | 0.079 | 0.212 | 0.630 | 0.421 | 0.152 | 0.898 | 0.885 | 0.062 | 0.170 | 0.607 | 0.382
<EMAIL_ADDRESS>| 0.616 | 0.263 | 0.911 | 0.900 | 0.692 | 0.503 | 0.908 | 0.990 | 0.275 | 0.396 | 0.934 | 0.851
<EMAIL_ADDRESS>| 0.818 | 0.397 | 1.000 | 0.972 | 0.796 | 0.678 | 0.917 | 1.000 | 0.429 | 0.552 | 0.945 | 0.956

Table 3: Performance of detection methods against strong attacks.

We report area under the curve (AUC) of the ROC curve as the performance
evaluation criterion, as well as the true positive rates (TPR) by thresholding
false positive rates (FPR) at 0.01, 0.05 and 0.1, as it is practical to keep
mis-classified natural data at a low proportion. Note that TPR represents the
proportion of adversarial examples correctly classified as adversarial, while
FPR represents the proportion of natural data incorrectly mis-classified as
adversarial. Before calculating the performance metrics, all the samples that
can be classified correctly by the model are removed. The results are reported
in Table 3. ODD (Roth _et al._ , 2019) and ReBeL (Raghuram _et al._ , 2020)
show superior or comparable performance over the other detection methods
across the two benchmark datasets.

##  6 Concluding Remarks

In this paper we described the attack and defense problems in classification.
We provided a taxonomy of these methods, explained the main ideas behind them,
and numerically compared their performances. As mentioned before, it is our
hope that this paper can spark statisticians’ interest in this exciting area
of research, as we firmly believe that statisticians are well suited to
provide unique and significant contributions to the field. One successful
example is the EAD attack of Chen _et al._ (2018), where the elastic-net
penalty of Zou and Hastie (2005) was a key component of the method. Another
successful example is the ER-Classifer of Li _et al._ (2018b), where the
method of Levina and Bickel (2005) was used to estimate the intrinsic
dimension of the data, and achieve dimension reduction. We conclude this paper
with some future work directions that are particularly suited for
statisticians to tackle.

#### Statistical properties of adversarial examples.

It is important and useful to study the statistical properties of adversarial
examples, as these can provide useful insights and guidelines for building
stronger attack and defense methods. So far work in this area is rather
sparse. Some notable exceptions include the study of the hidden layer output
distribution of adversarial examples (Lee _et al._ , 2018), the variance of
the hidden output (Raghuram _et al._ , 2020), and the input space feature
attribution (Yang _et al._ , 2020a).

#### Efficient modeling for improved decision-based attacks.

In reality, many classification systems are blackboxes to the outsiders, and
therefore gradient-based and score-based attack methods cannot be applied.
Also, these classification systems typically limit the number of queries any
outsider can make, and hence many of the current decision-based attack methods
will fail. The problem of developing a strong decision-based attack method can
be viewed as estimating (or modeling) a black-box classification model with a
limited number of carefully chosen “design points” (most likely in a
sequential manner) as input to the classifier.

#### Theoretical guarantee for robust defense methods.

With the ever increasing size and complexity of datasets and applications,
there will always be a demand for more reliable and efficient defense methods
that are scalable. So far the trend is that many defense methods were shown to
have excellent empirical properties at the time of their creation, but then
very often they were soon beaten by newer attack methods (Athalye _et al._ ,
2018). This suggests that defense methods cannot be blindly trusted. One
ambitious project is to formally define the notion of robustness in this
attack and defense context, which includes attaching a robust score to any
defense method; i.e., the higher the score, the more reliable the method. This
is similar to the use of breakdown point in classical robust statistics. It is
likely that there will be different reasonable definitions for robustness,
each designed for one type of attacks. If a new method achieves high scores
for different robustness definitions, then we have strong confidence that this
method will work well.

### Acknowledgment

The authors are most grateful to the reviewers, the associate editor and the
editor for their most helpful comments. The work of Hsieh was partially
supported by the National Science Foundation under grants CCF-1934568,
IIS-1901527 and IIS-2008173. The work of Lee was partially supported by the
National Science Foundation under grants CCF-1934568, DMS-1811405,
DMS-1811661, DMS-1916125 and DMS-2113605.

## References

  * Al-Dujaili and O’Reilly (2020) Al-Dujaili, A. and O’Reilly, U.-M. (2020) Sign bits are all you need for black-box attacks.  In _International Conference on Learning Representations_.
  * Alzantot _et al._ (2018) Alzantot, M., Sharma, Y., Chakraborty, S. and Srivastava, M. (2018) Genattack: Practical black-box attacks with gradient-free optimization.  _arXiv preprint arXiv:1805.11090_.
  * Andriushchenko _et al._ (2020) Andriushchenko, M., Croce, F., Flammarion, N. and Hein, M. (2020) Square attack: a query-efficient black-box adversarial attack via random search.  In _European Conference on Computer Vision_ , 484–501.
  * Athalye _et al._ (2018) Athalye, A., Carlini, N. and Wagner, D. (2018) Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  In _International Conference on Machine Learning_ , 274–283.
  * Baluja and Fischer (2018) Baluja, S. and Fischer, I. (2018) Learning to attack: Adversarial transformation networks.  In _Proceedings of AAAI-2018_.
  * Biggio _et al._ (2013) Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G. and Roli, F. (2013) Evasion attacks against machine learning at test time.  In _Joint European conference on machine learning and knowledge discovery in databases_ , 387–402.
  * Brendel _et al._ (2018) Brendel, W., Rauber, J. and Bethge, M. (2018) Decision-based adversarial attacks: Reliable attacks against black-box machine learning models.  _International Conference on Learning Representations_.
  * Brunner _et al._ (2019) Brunner, T., Diehl, F., Le, M. T. and Knoll, A. (2019) Guessing smart: Biased sampling for efficient black-box adversarial attacks.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 4958–4966.
  * Carlini and Wagner (2017a) Carlini, N. and Wagner, D. (2017a) Adversarial examples are not easily detected: Bypassing ten detection methods.  In _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , 3–14.
  * Carlini and Wagner (2017b) Carlini, N. and Wagner, D. (2017b) Towards evaluating the robustness of neural networks.  In _2017 IEEE Symposium on Security and Privacy (SP)_ , 39–57.
  * Carmon _et al._ (2020) Carmon, Y., Raghunathan, A., Schmidt, L., Liang, P. and Duchi, J. C. (2020) Unlabeled data improves adversarial robustness.  _Advances in Neural Information Processing Systems_.
  * Chen _et al._ (2019) Chen, H., Zhang, H., Si, S., Li, Y., Boning, D. and Hsieh, C.-J. (2019) Robustness verification of tree-based models.  In _Advances in Neural Information Processing Systems_ , vol. 32. Curran Associates, Inc.
  * Chen and Gu (2020) Chen, J. and Gu, Q. (2020) Rays: A ray searching method for hard-label adversarial attack.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 1739–1747.
  * Chen and Jordan (2019) Chen, J. and Jordan, M. I. (2019) Boundary attack++: Query-efficient decision-based adversarial attack.  _arXiv preprint arXiv:1904.02144_.
  * Chen _et al._ (2020) Chen, J., Jordan, M. I. and Wainwright, M. J. (2020) Hopskipjumpattack: A query-efficient decision-based attack.  In _2020 IEEE Symposium on Security and Privacy (SP)_ , 1277–1294.
  * Chen _et al._ (2018) Chen, P.-Y., Sharma, Y., Zhang, H., Yi, J. and Hsieh, C.-J. (2018) EAD: elastic-net attacks to deep neural networks via adversarial examples.  In _Thirty-second AAAI conference on artificial intelligence_.
  * Chen _et al._ (2017) Chen, P.-Y., Zhang, H., Sharma, Y., Yi, J. and Hsieh, C.-J. (2017) Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models.  In _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , 15–26.
  * Cheng _et al._ (2019a) Cheng, M., Le, T., Chen, P.-Y., Zhang, H., Yi, J. and Hsieh, C.-J. (2019a) Query-efficient hard-label black-box attack: An optimization-based approach.  In _International Conference on Learning Representations_.  URL https://openreview.net/forum?id=rJlk6iRqKX.
  * Cheng _et al._ (2020a) Cheng, M., Singh, S., Chen, P. H., Chen, P.-Y., Liu, S. and Hsieh, C.-J. (2020a) Sign-opt: A query-efficient hard-label adversarial attack.  In _International Conference on Learning Representations_.
  * Cheng _et al._ (2020b) Cheng, M., Yi, J., Chen, P.-Y., Zhang, H. and Hsieh, C.-J. (2020b) Seq2sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, 3601–3608.
  * Cheng _et al._ (2019b) Cheng, S., Dong, Y., Pang, T., Su, H. and Zhu, J. (2019b) Improving black-box adversarial attacks with a transfer-based prior.  In _Advances in Neural Information Processing Systems_ , 10932–10942.
  * Cohen _et al._ (2019) Cohen, J., Rosenfeld, E. and Kolter, Z. (2019) Certified adversarial robustness via randomized smoothing.  In _International Conference on Machine Learning_ , 1310–1320.
  * Croce and Hein (2020) Croce, F. and Hein, M. (2020) Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.  In _International Conference on Machine Learning_ , 2206–2216.
  * Deng _et al._ (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K. and Fei-Fei, L. (2009) Imagenet: A large-scale hierarchical image database.  In _Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on_ , 248–255.
  * Devlin _et al._ (2019) Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2019) BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics.
  * Dhillon _et al._ (2018) Dhillon, G. S., Azizzadenesheli, K., Lipton, Z. C., Bernstein, J., Kossaifi, J., Khanna, A. and Anandkumar, A. (2018) Stochastic activation pruning for robust adversarial defense.  _International Conference on Learning Representations_.
  * Ding _et al._ (2020) Ding, G. W., Sharma, Y., Lui, K. Y. C. and Huang, R. (2020) Mma training: Direct input space margin maximization through adversarial training.  In _International Conference on Learning Representations_.
  * Dodge and Karam (2017) Dodge, S. and Karam, L. (2017) A study and comparison of human and deep learning recognition performance under visual distortions.  In _2017 26th International Conference on Computer Communication and Networks (ICCCN)_ , 1–7.
  * Dosovitskiy _et al._ (2021) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S. _et al._ (2021) An image is worth 16x16 words: Transformers for image recognition at scale.  _International Conference on Learning Representations_.
  * Fawzi _et al._ (2016) Fawzi, A., Moosavi-Dezfooli, S.-M. and Frossard, P. (2016) Robustness of classifiers: from adversarial to random noise.  In _Advances in Neural Information Processing Systems_ , 1632–1640.
  * Feinman _et al._ (2017) Feinman, R., Curtin, R. R., Shintre, S. and Gardner, A. B. (2017) Detecting adversarial samples from artifacts.  _International Conference on Machine Learning_.
  * Franceschi _et al._ (2018) Franceschi, J.-Y., Fawzi, A. and Fawzi, O. (2018) Robustness of classifiers to uniform $\ell_{p}$ and Gaussian noise.  In _Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics_ (eds. A. Storkey and F. Perez-Cruz), vol. 84 of _Proceedings of Machine Learning Research_ , 1280–1288. PMLR.
  * Gao _et al._ (2018) Gao, J., Lanchantin, J., Soffa, M. L. and Qi, Y. (2018) Black-box generation of adversarial text sequences to evade deep learning classifiers.  In _2018 IEEE Security and Privacy Workshops (SPW)_ , 50–56.
  * Gao _et al._ (2019) Gao, R., Cai, T., Li, H., Hsieh, C.-J., Wang, L. and Lee, J. D. (2019) Convergence of adversarial training in overparametrized neural networks.  _Advances in Neural Information Processing Systems_ , 32, 13029–13040.
  * Gong _et al._ (2017) Gong, Z., Wang, W. and Ku, W.-S. (2017) Adversarial and clean data are not twins.  _arXiv preprint arXiv:1704.04960_.
  * Goodfellow _et al._ (2016) Goodfellow, I., Bengio, Y. and Courville, A. (2016) _Deep Learning_.  MIT Press.
  * Goodfellow _et al._ (2015) Goodfellow, I., Shlens, J. and Szegedy, C. (2015) Explaining and harnessing adversarial examples.  In _International Conference on Learning Representations_.
  * Gowal _et al._ (2020) Gowal, S., Qin, C., Uesato, J., Mann, T. and Kohli, P. (2020) Uncovering the limits of adversarial training against norm-bounded adversarial examples.  _arXiv preprint arXiv:2010.03593_.
  * Grosse _et al._ (2017) Grosse, K., Manoharan, P., Papernot, N., Backes, M. and McDaniel, P. (2017) On the (statistical) detection of adversarial examples.  _arXiv preprint arXiv:1702.06280_.
  * Guo _et al._ (2020) Guo, C., Frank, J. S. and Weinberger, K. Q. (2020) Low frequency adversarial perturbation.  In _Proceedings of the 35th Uncertainty in Artificial Intelligence Conference_ (eds. R. P. Adams and V. Gogate), vol. 115 of _Proceedings of Machine Learning Research_ , 1127–1137. Tel Aviv, Israel: PMLR.
  * Guo _et al._ (2019a) Guo, C., Gardner, J., You, Y., Wilson, A. G. and Weinberger, K. (2019a) Simple black-box adversarial attacks.  In _Proceedings of the 36th International Conference on Machine Learning_ (eds. K. Chaudhuri and R. Salakhutdinov), vol. 97 of _Proceedings of Machine Learning Research_ , 2484–2493. PMLR.
  * Guo _et al._ (2019b) Guo, Y., Yan, Z. and Zhang, C. (2019b) Subspace attack: Exploiting promising subspaces for query-efficient black-box attacks.  In _Advances in Neural Information Processing Systems_ (eds. H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox and R. Garnett), vol. 32. Curran Associates, Inc.
  * Hecht-Nielsen (1992) Hecht-Nielsen, R. (1992) Theory of the backpropagation neural network.  In _Neural networks for perception_ , 65–93. Elsevier.
  * Hung _et al._ (2019) Hung, K., Fithian, W. _et al._ (2019) Rank verification for exponential families.  _The Annals of Statistics_ , 47, 758–782.
  * Ilyas _et al._ (2018) Ilyas, A., Engstrom, L., Athalye, A. and Lin, J. (2018) Black-box adversarial attacks with limited queries and information.  In _International Conference on Machine Learning_ , 2137–2146.
  * Ilyas _et al._ (2019) Ilyas, A., Engstrom, L. and Madry, A. (2019) Prior convictions: Black-box adversarial attacks with bandits and priors.  In _International Conference on Learning Representations_.
  * Jalal _et al._ (2017) Jalal, A., Ilyas, A., Daskalakis, C. and Dimakis, A. G. (2017) The robust manifold defense: Adversarial training using generative models.  _arXiv preprint arXiv:1712.09196_.
  * Jia and Liang (2017) Jia, R. and Liang, P. (2017) Adversarial examples for evaluating reading comprehension systems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2021–2031. Copenhagen, Denmark: Association for Computational Linguistics.
  * Krizhevsky and Hinton (2009) Krizhevsky, A. and Hinton, G. (2009) Learning multiple layers of features from tiny images.  Tech. rep., Citeseer.
  * Kurakin _et al._ (2016) Kurakin, A., Goodfellow, I., Bengio, S. _et al._ (2016) Adversarial examples in the physical world.
  * LeCun (1998) LeCun, Y. (1998) The mnist database of handwritten digits.  _http://yann. lecun. com/exdb/mnist/_.
  * LeCun _et al._ (1998) LeCun, Y., Bottou, L., Bengio, Y. and Haffner, P. (1998) Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86, 2278–2324.
  * Lecuyer _et al._ (2019) Lecuyer, M., Atlidakis, V., Geambasu, R., Hsu, D. and Jana, S. (2019) Certified robustness to adversarial examples with differential privacy.  In _2019 IEEE Symposium on Security and Privacy (SP)_ , 656–672.
  * Lee _et al._ (2018) Lee, K., Lee, K., Lee, H. and Shin, J. (2018) A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  In _Advances in Neural Information Processing Systems_ , 7167–7177.
  * Levina and Bickel (2005) Levina, E. and Bickel, P. J. (2005) Maximum likelihood estimation of intrinsic dimension.  In _Advances in neural information processing systems_ , 777–784.
  * Li _et al._ (2018a) Li, B., Chen, C., Wang, W. and Carin, L. (2018a) Certified adversarial robustness with additive noise.  _arXiv preprint arXiv:1809.03113_.
  * Li and Li (2017) Li, X. and Li, F. (2017) Adversarial examples detection in deep networks with convolutional filter statistics.  In _2017 IEEE International Conference on Computer Vision_ , 5775–5783.
  * Li _et al._ (2019) Li, Y., Li, L., Wang, L., Zhang, T. and Gong, B. (2019) Nattack: Learning the distributions of adversarial examples for an improved black-box attack on deep neural networks.  In _International Conference on Machine Learning_ , 3866–3876. PMLR.
  * Li _et al._ (2018b) Li, Y., Min, M. R., Yu, W., Hsieh, C.-J., Lee, T. C. M. and Kruus, E. (2018b) Optimal transport classifier: Defending against adversarial attacks by regularized deep embedding.  _arXiv preprint arXiv:1811.07950_.
  * Liu _et al._ (2018a) Liu, S., Chen, P.-Y., Chen, X. and Hong, M. (2018a) signsgd via zeroth-order oracle.  In _International Conference on Learning Representations_.
  * Liu _et al._ (2018b) Liu, X., Cheng, M., Zhang, H. and Hsieh, C.-J. (2018b) Towards robust neural networks via random self-ensemble.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 369–385.
  * Liu _et al._ (2019) Liu, X., Li, Y., Wu, C. and Hsieh, C.-J. (2019) Adv-BNN: Improved adversarial defense through robust Bayesian neural network.  In _International Conference on Learning Representations_.
  * Liu _et al._ (2017) Liu, Y., Chen, X., Liu, C. and Song, D. (2017) Delving into transferable adversarial examples and black-box attacks.  _International Conference on Learning Representations_.
  * Ma _et al._ (2018) Ma, X., Li, B., Wang, Y., Erfani, S. M., Wijewickrema, S., Schoenebeck, G., Song, D., Houle, M. E. and Bailey, J. (2018) Characterizing adversarial subspaces using local intrinsic dimensionality.  _International Conference on Learning Representations_.
  * Madry _et al._ (2018) Madry, A., Makelov, A., Schmidt, L., Tsipras, D. and Vladu, A. (2018) Towards deep learning models resistant to adversarial attacks.  In _International Conference on Learning Representations_.
  * Meng and Chen (2017) Meng, D. and Chen, H. (2017) Magnet: a two-pronged defense against adversarial examples.  In _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_ , 135–147.
  * Metzen _et al._ (2017) Metzen, J. H., Genewein, T., Fischer, V. and Bischoff, B. (2017) On detecting adversarial perturbations.  _International Conference on Learning Representations_.
  * Moosavi-Dezfooli _et al._ (2016) Moosavi-Dezfooli, S.-M., Fawzi, A. and Frossard, P. (2016) Deepfool: a simple and accurate method to fool deep neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2574–2582.
  * Mopuri _et al._ (2018) Mopuri, K. R., Ojha, U., Garg, U. and Babu, R. V. (2018) Nag: Network for adversary generation.  In _Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR)_.
  * Nesterov and Spokoiny (2017) Nesterov, Y. and Spokoiny, V. (2017) Random gradient-free minimization of convex functions.  _Foundations of Computational Mathematics_ , 17, 527–566.
  * Pang _et al._ (2021) Pang, T., Yang, X., Dong, Y., Su, H. and Zhu, J. (2021) Bag of tricks for adversarial training.  In _International Conference on Learning Representations_.
  * Pang _et al._ (2020) Pang, T., Yang, X., Dong, Y., Xu, K., Zhu, J. and Su, H. (2020) Boosting adversarial training with hypersphere embedding.  In _Advances in Neural Information Processing Systems_ (eds. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan and H. Lin), vol. 33, 7779–7792. Curran Associates, Inc.
  * Papernot _et al._ (2017) Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B. and Swami, A. (2017) Practical black-box attacks against machine learning.  In _Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security_ , 506–519.
  * Papernot _et al._ (2016) Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B. and Swami, A. (2016) The limitations of deep learning in adversarial settings.  In _Security and Privacy (EuroS &P), 2016 IEEE European Symposium on_, 372–387.
  * Qiu _et al._ (2019) Qiu, S., Liu, Q., Zhou, S. and Wu, C. (2019) Review of artificial intelligence adversarial attack and defense technologies.  _Applied Sciences_ , 9, 909.
  * Raghuram _et al._ (2020) Raghuram, J., Chandrasekaran, V., Jha, S. and Banerjee, S. (2020) Detecting anomalous inputs to DNN classifiers by joint statistical testing at the layers.  _arXiv preprint arXiv:2007.15147_.
  * Ren _et al._ (2020) Ren, K., Zheng, T., Qin, Z. and Liu, X. (2020) Adversarial attacks and defenses in deep learning.  _Engineering_ , 6, 346–360.
  * Roth _et al._ (2019) Roth, K., Kilcher, Y. and Hofmann, T. (2019) The odds are odd: A statistical test for detecting adversarial examples.  In _International Conference on Machine Learning_ , 5498–5507.
  * Samangouei _et al._ (2018) Samangouei, P., Kabkab, M. and Chellappa, R. (2018) Defense-GAN: Protecting classifiers against adversarial attacks using generative models.  In _International Conference on Learning Representations_.
  * Samanta and Mehta (2017) Samanta, S. and Mehta, S. (2017) Towards crafting text adversarial samples.  _arXiv preprint arXiv:1707.02812_.
  * Serban _et al._ (2020) Serban, A., Poll, E. and Visser, J. (2020) Adversarial examples on object recognition: A comprehensive survey.  _ACM Computing Surveys (CSUR)_ , 53, 1–38.
  * Shafahi _et al._ (2019) Shafahi, A., Najibi, M., Ghiasi, M. A., Xu, Z., Dickerson, J., Studer, C., Davis, L. S., Taylor, G. and Goldstein, T. (2019) Adversarial training for free!  In _Advances in Neural Information Processing Systems_ , 3353–3364.
  * Silver _et al._ (2016) Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M. _et al._ (2016) Mastering the game of go with deep neural networks and tree search.  _Nature_ , 529, 484–489.
  * Simonyan and Zisserman (2014) Simonyan, K. and Zisserman, A. (2014) Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_.
  * Song _et al._ (2018) Song, Y., Kim, T., Nowozin, S., Ermon, S. and Kushman, N. (2018) Pixeldefend: Leveraging generative models to understand and defend against adversarial examples.  In _International Conference on Learning Representations_.
  * Szegedy _et al._ (2014) Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. and Fergus, R. (2014) Intriguing properties of neural networks.  In _International Conference on Learning Representations_.
  * Tanay and Griffin (2016) Tanay, T. and Griffin, L. (2016) A boundary tilting persepective on the phenomenon of adversarial examples.  _arXiv preprint arXiv:1608.07690_.
  * Wang _et al._ (2020a) Wang, L., Zhang, H., Yi, J., Hsieh, C.-J. and Jiang, Y. (2020a) Spanning attack: reinforce black-box attacks with unlabeled data.  _Machine Learning_ , 109, 2349–2368.
  * Wang _et al._ (2019) Wang, Y., Ma, X., Bailey, J., Yi, J., Zhou, B. and Gu, Q. (2019) On the convergence and robustness of adversarial training.  In _International Conference on Machine Learning_ , 6586–6595.
  * Wang _et al._ (2020b) Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X. and Gu, Q. (2020b) Improving adversarial robustness requires revisiting misclassified examples.  In _International Conference on Learning Representations_.
  * Wong and Kolter (2020) Wong, E. and Kolter, J. Z. (2020) Learning perturbation sets for robust machine learning.  _arXiv preprint arXiv:2007.08450_.
  * Wong _et al._ (2020) Wong, E., Rice, L. and Kolter, J. Z. (2020) Fast is better than free: Revisiting adversarial training.  In _International Conference on Learning Representations_.
  * Wong _et al._ (2019) Wong, E., Schmidt, F. and Kolter, Z. (2019) Wasserstein adversarial examples via projected sinkhorn iterations.  In _International Conference on Machine Learning_ , 6808–6817.
  * Wu _et al._ (2020) Wu, D., Xia, S.-T. and Wang, Y. (2020) Adversarial weight perturbation helps robust generalization.  _Advances in Neural Information Processing Systems_ , 33.
  * Xie _et al._ (2018) Xie, C., Wang, J., Zhang, Z., Ren, Z. and Yuille, A. (2018) Mitigating adversarial effects through randomization.  In _International Conference on Learning Representations_.
  * Xu _et al._ (2020) Xu, H., Ma, Y., Liu, H.-C., Deb, D., Liu, H., Tang, J.-L. and Jain, A. K. (2020) Adversarial attacks and defenses in images, graphs and text: A review.  _International Journal of Automation and Computing_ , 17, 151–178.
  * Yang _et al._ (2020a) Yang, P., Chen, J., Hsieh, C.-J., Wang, J.-L. and Jordan, M. (2020a) Ml-loo: Detecting adversarial examples with feature attribution.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, 6639–6647.
  * Yang _et al._ (2020b) Yang, P., Chen, J., Hsieh, C.-J., Wang, J.-L. and Jordan, M. I. (2020b) Greedy attack and Gumbel attack: Generating adversarial examples for discrete data.  _Journal of Machine Learning Research_ , 21, 1–36.
  * Yuan _et al._ (2019) Yuan, X., He, P., Zhu, Q. and Li, X. (2019) Adversarial examples: Attacks and defenses for deep learning.  _IEEE transactions on neural networks and learning systems_ , 30, 2805–2824.
  * Zantedeschi _et al._ (2017) Zantedeschi, V., Nicolae, M.-I. and Rawat, A. (2017) Efficient defenses against adversarial attacks.  In _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , 39–49.
  * Zhang _et al._ (2019a) Zhang, D., Zhang, T., Lu, Y., Zhu, Z. and Dong, B. (2019a) You only propagate once: Accelerating adversarial training via maximal principle.  In _Advances in Neural Information Processing Systems_ , 227–238.
  * Zhang _et al._ (2019b) Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L. and Jordan, M. (2019b) Theoretically principled trade-off between robustness and accuracy.  In _International Conference on Machine Learning_ , 7472–7482.
  * Zheng and Hong (2018) Zheng, Z. and Hong, P. (2018) Robust detection of adversarial attacks by modeling the intrinsic properties of deep neural networks.  In _Advances in Neural Information Processing Systems_ , 7913–7922.
  * Zou and Hastie (2005) Zou, H. and Hastie, T. (2005) Regularization and variable selection via the elastic net.  _Journal of the Royal Statistical Society: Series B_ , 67, 301–320.