# A Survey of Robustness and Safety of 2D and 3D Deep Learning Models Against
Adversarial Attacks

Yanjie Li<EMAIL_ADDRESS>0000-0001-8859-8331 The Hong Kong
Polytechnic University11, Yuk Choi Road, Hung Hom, KLNHong Kong ,  Bin Xie
<EMAIL_ADDRESS>0000-0001-5118-3570 The Hong Kong Polytechnic
University11, Yuk Choi Road, Hung Hom, KLNHong Kong ,  Songtao Guo
<EMAIL_ADDRESS>Chongqing UniversityChongqingChina ,  Yuanyuan Yang
<EMAIL_ADDRESS>0000-0001-7296-9222 Stony Brook UniversityStony
Brook, NYUSA and  Bin Xiao<EMAIL_ADDRESS>0000-0003-4223-8220 The
Hong Kong Polytechnic University11, Yuk Choi Road, Hung Hom, KLNHong Kong

###### Abstract.

Benefiting from the rapid development of deep learning, 2D and 3D computer
vision applications are deployed in many safe-critical systems, such as
autopilot and identity authentication. However, deep learning models are not
trustworthy enough because of their limited robustness against adversarial
attacks. The physically realizable adversarial attacks further pose fatal
threats to the application and human safety. Lots of papers have emerged to
investigate the robustness and safety of deep learning models against
adversarial attacks. To lead to trustworthy AI, we first construct a general
threat model from different perspectives and then comprehensively review the
latest progress of both 2D and 3D adversarial attacks. We extend the concept
of adversarial examples beyond imperceptive perturbations and collate over 170
papers to give an overview of deep learning model robustness against various
adversarial attacks. To the best of our knowledge, we are the first to
systematically investigate adversarial attacks for 3D models, a flourishing
field applied to many real-world applications. In addition, we examine
physical adversarial attacks that lead to safety violations. Last but not
least, we summarize present popular topics, give insights on challenges, and
shed light on future research on trustworthy AI.

Deep learning; 3D computer vision; Adversarial attack; Robustness;

††submissionid: CSUR-2022-0640††ccs: Security and privacy††ccs: Computing
methodologies Machine learning††ccs: Computing methodologies Computer vision

##  1\. Introduction

The significant strides of deep learning (DL) algorithms have driven
considerable technological progress in computer vision (CV) tasks, which are
widely deployed in various safety-critical and mission-critical systems like
identity authentication and self-driving vehicles. These applications depend
on the assumption that these deep learning models are trustworthy and robust
against small perturbations, which means these models can produce consistent
predictions when noise exists. However, studies have shown that deep-learning
models are vulnerable to adversarial examples (AEs), which makes the deep-
learning models produce false predictions by crafting elaborate imperceptive
or semantic-preserving perturbations. To realize trustworthy AI, continuous
efforts have been spent on improving the model’s robustness and safety against
adversarial attacks and finding these models’ robustness upper bound by
constructing stronger adversarial attacks. This is a relentless race about
adversarial attacks and defenses. This paper thoroughly surveys the latest
progress of this adversarial attack and defense competition from the
adversary’s perspective.

This paper primarily discusses the phenomenon of adversarial attacks in the
realm of computer vision tasks. While most computer vision tasks tend to focus
on image processing, recent attention has been directed toward 3D tasks. 3D
data can supplement 2D data by providing depth information about an image,
thereby allowing for a more reliable and detailed analysis of targets, and can
be utilized in various applications. For instance, autonomous vehicles often
rely on a combination of cameras and lidar to perceive their surroundings.
However, due to the unordered nature of 3D data, direct application of 2D
adversarial attacks is not feasible. 3D adversarial attacks have been
proposed, building upon the principles of 2D adversarial attacks but with
specific designs tailored to the characteristics of 3D data. For example,
3DAdv (Xiao et al., 2018a) is based on the C&W attack but with novel distance
metrics. 3Dhacker (Tao et al., 2023) is based on the boundary attack but fuses
the point cloud in the spectral domain rather than via coordinate-wise average
operation. Due to the close relationship between 2D and 3D computer vision
tasks and the shared theoretical foundation of 2D and 3D attacks, this survey
summarizes the latest progress of adversarial attacks in both 2D and 3D
computer vision.

The aim of this survey is to systematize the latest progress of 2D and 3D
adversarial attacks to help researchers construct stronger AEs to evaluate
model robustness, design robust models, and ensure safety in real-world
applications. To select literature for systematic review, we first clarify the
review scope and thoroughly search publications in top computer vision or
security conferences and journals within the past several years. Then, we
select high-cited or representative work related to 2D and 3D adversarial
attacks. These articles are categorized and compared based on the targets,
scenarios, and methods for comprehensiveness. The main contributions of this
work are as follows:

  * •

We summarize the latest studies on the adversarial robustness and safety of 2D
and 3D deep learning models against adversarial attacks. Over 170 papers in
recent years have been collated and compared. Moreover, We divide them into 2D
and 3D attacks according to the data characteristics and application
scenarios.

  * •

For 2D adversarial attacks, we extend the meaning of adversarial attack from
imperceptible perturbation to semantic-preserving perturbation, such as color
space distortion and spatial transformation distortion. We classify these
attacks according to methodologies and compare their pros and cons to give a
full-scope view.

  * •

3D data is increasingly leveraged in safety-critical fields like self-driving.
However, the 3D model robustness is hardly reviewed. To help design robust 3D
deep learning models, we are the first to organize 3D adversarial attacks
systematically and classify them according to their algorithms.

  * •

When deep learning models are deployed in the real world, safety is the
primary premise. We comprehensively examine related works of adversarial
attacks for safe-critical missions, especially for camera-based and Lidar-
based self-driving cars and face recognition.

  * •

At the end of this review, we summarize the present hot research topics into
several points and identify their challenges, such as improving the attack
transferability, generating semantic perturbation, and evaluating the
robustness of 3D deep learning models. We also provide some useful advice for
future research directions.

The structure of this survey is shown in Figure.1. In Sec.2, we clarify the
basic concepts of deep learning and computer vision. In Sec.4, we build a
general threat model for deep-learning-based computer vision systems by
examining the attack surface from different perspectives. In Sec.3, we
summarize the latest related reviews. In Sec.5 and 6, we summarize the
representative works that evaluate the robustness and safety of deep learning
models by 2D and 3D adversarial examples, respectively. In each section, we
first introduce our taxonomy based on attack methodology or target
applications, then summarize the representative digital-world and physical-
world adversarial attacks. Finally, in Sec.7, we identify the present
obstacles in the adversarial attacks, coupled with some viewpoints for future
studies.

To unify the symbols of different articles, Table.1 shows some common
notations in this survey.

Figure 1. The structure of this survey

The structure of this paper.

Table 1. Common notations used in this survey Notations | Description | Notations | Description
---|---|---|---
$D_{tr},D_{te}$ | Training`\`testing dataset | $x$ | Normal inputs
$x^{\prime},x^{adv}$ | Modified`\`adversarial inputs | $\delta$ | Adversarial perturbations
$y$ | Ground truth labels | $y^{\prime}$ | Adversarial target labels
$\mathcal{F}$ | Classification model | $\hat{\mathcal{F}}$ | Substitute model
$g$ | The model’s gradient upon $x$ | $\hat{g}$ | The estimated model gradient
$l_{p}$ | The $p$-norm distance | $Z_{i}(x)$ | Output logits of $i^{th}$-to-last layer
$\theta$ | Model parameters | $\mathcal{L}(x^{\prime},y)$ | Loss function
$\mathcal{G}$ | The GAN’s generator | $\mathcal{D}$ | The GAN’s discriminator
$\mathcal{P}$ | Original point cloud | $\mathcal{P^{\prime}}$ | Adversarial point cloud

##  2\. Background

Deep learning is a popular representation learning algorithm for its
outstanding performance on image classification and other tasks. It can learn
complex functions through a composition of superficial but non-linear layers.
Suppose $x$ is an image or a point cloud, $\mathcal{F}_{\theta}$ is a deep
learning model with model parameters $\theta$. The goal of the object
classification task is to find $\theta$ that can minimize the difference
between the ground truth label $y$ and the prediction
$\mathcal{F}_{\theta}(x)$, that is

(1) $arg\min_{\theta}\mathcal{L}(\mathcal{F}_{\theta}(x),y),$

where $\mathcal{L}$ is a loss function to measure the entropy between the
$\mathcal{F}_{\theta}(x)$ and y. The $\theta$ is usually optimized through
gradient descent algorithms, such as Adam. After training, the deep learning
model is deployed into real-world applications.

Many deep learning model variants have been proposed to boost performance and
adapt to different task characteristics, such as multi-layer perceptions,
stacked autoencoders, convolutional neural networks, deep brief networks, and
vision transformers. These models are widely used in various 2D computer
vision tasks, such as object detection, image segmentation, image
classification, action recognition, and motion tracking. Recently, with the
boom of 3D data in self-driving systems and other applications, deep learning
for 3D computer vision has attracted much attention. Various models came out,
such as MLP-based models (e.g. PointNet (Qi et al., 2017)), convolutional
models (e.g. Pointwise-CNN (Hua et al., 2018)), graph-based convolutional
models (e.g. DGCNN (Wang et al., 2019)), and transformer-based models (e.g.
PCT (Guo et al., 2021)). The major applications of the deep learning model in
3D computer vision can be categorized into three tasks: 3D object detection
and tracking, 3D object classification, and 3D object segment. Because most 3D
deep learning models are extensions of 2D models, many security threats in the
2D domain also exist in the 3D domain.

##  3\. Related work

To boost this adversarial machine learning arms race and mitigate the risk of
adversarial attacks, some previous surveys have tried to summarize the latest
relevant studies. For example, Serban (Serban et al., 2020) collected
adversarial attacks in the object recognition task. However, they did not
include 3D adversarial attacks and semantic attacks like color space
perturbations. Miller et al. (Miller et al., 2020) and Machado et al. (Machado
et al., 2021) reviewed the recent progress of adversarial machine learning,
but from the defense rather than the adversary perspective. Other reviews
regard the adversarial attack as part of the AI attacks without a separate
detailed investigation. For example, Liu et al. (Liu et al., 2018) classified
AI attacks into integrity, availability, and confidentiality attacks according
to the traditional taxonomy of security violation. Papernot et al. (Papernot
et al., 2018) and Liu et al. (Liu et al., 2020b) categorized attacks based on
the deep learning pipeline, dividing them into training and test phase
attacks. He et al. (He et al., 2022a) categorized the AI attacks into
adversarial attacks, poisoning attacks, model extraction, and inversion
attacks. Moreover, these surveys lack summaries of the latest research trends,
such as the growing interest in transferable adversarial examples, semantic
perturbations, 3D adversarial attacks, and physical-realizable adversarial
attacks. In addition, although attack-agnostic robustness evaluation has
emerged in recent years, the adversarial attack is still one of the most
efficient and reliable ways to evaluate model robustness. There is still a
lack of a systematic review to collate these developments and discuss future
directions in light of these latest efforts.

##  4\. Threat model

The threat model evaluates the possible risk and security level of the system
by describing the target, capability, and knowledge of the attacker. By
establishing a full-scale threat model, we can understand the security
problems of the deep learning system more comprehensively. We first identify
the complete and thorough attack surface when the deep neural networks are
deployed into computer vision tasks, as shown in Figure.2. Then we analyze the
adversary’s goals, capabilities, and knowledge to construct a threat panorama.

###  4.1. The attack surface of deep learning models

The attack surface is composed of all possible vulnerabilities in a system
that an unauthorized user can access. As mentioned above, deep learning
applications train the model according to the model performance on the
training dataset and then deploy the model to real-world applications for
various 2D or 3D tasks. In this process, the attack surface includes the
training and test dataset and the deep learning model.

####  4.1.1. Training dataset.

Collecting a training dataset with satisfactory quality requires a great
amount of effort and money. Meanwhile, some datasets, like medical diagnostic
records and personal genomic data, may contain sensitive information.
Therefore, the privacy and confidentiality of these training datasets are
essential. Membership inference attacks (Shokri et al., 2017) and data
reconstruction attacks (Salem et al., 2020) can leak the attribute or
information of training data. In addition, more capable attackers may be able
to modify the training datasets. Under this assumption, backdoor attacks and
poisoning attacks are proposed. The backdoor attacks make the model behave
normally for clean samples but behave incorrectly for the same samples with
specific stickers by adding triggers in the training process. The poisoning
attacks modify the sample’s label in the training phase so that the model
classifies the clean sample incorrectly.

####  4.1.2. Test dataset.

The main threat to the test dataset is adversarial attack. The adversarial
attack can make the model produce wrong results by adding invisible or
semantic perturbations to the input data. In recent years, physical
adversarial attacks have gained more and more attention. Through 2D patches
(Duan et al., 2020), 3D printing (Athalye et al., 2018b), optical illumination
(Gnanasambandam et al., 2021) or sensor injection (Cao et al., 2019b), the
attacker makes the camera or radar collect modified 2D or 3D data and output
incorrect classification results. It is worth mentioning that this paper
divides the adversarial attack into 2D and 3D attacks according to the data
and model type rather than the attack media. For example, Athalye et al.
(Athalye et al., 2018b) rendered 2D adversarial images onto the 3D-printed
turtles. Although the turtles are 3D objects, the final input of the 2D model
is the turtle image collected by the camera, so it is still a 2D adversarial
attack.

####  4.1.3. Deep learning model.

Training deep learning models requires a lot of resources and time, so a well-
trained deep learning model successfully applied to practice has great value.
However, there are already some model extraction attacks that can steal
sensitive information such as model parameters and decision boundaries. In
addition, the model itself may also be subject to model revising attack. For
example, an adversary can make the model misclassify by only flipping several
bits of the model (Rakin et al., 2021) or poisoning the open-source pre-
trained weights (Kurita et al., 2020).

This paper mainly discusses the adversarial attack because it poses a more
practical threat than other attacks. It does not need to have knowledge about
the training dataset and even the model architecture (for black-box attacks)
and, therefore, can be executed in the inference stage. In addition,
generating strong AEs efficiently can improve the effectiveness and efficiency
of adversarial training, which is essential for training robust models.

Figure 2. An overview of the attack surface when deep learning models are
deployed into computer vision applications.

The structure of this paper.

###  4.2. Adversarial goal

According to the attacker’s goal and the level of security violation, deep
learning attacks can be divided into confidentiality attack, integrity attack,
and availability attack. The main goal of the confidentiality attack is to
leak the privacy of the model or data, such as the model inverse attack and
membership inference attack. The primary aim of the integrity attack is to
maliciously change the model’s output by modifying the training data, the test
data, or even the model itself, such as backdoor attacks, adversarial attacks,
and weight modification attacks. The availability attack aims to make the ML
services unavailable to legal users. A typical example of availability attacks
is to make the dataset not exploitable by maliciously poisoning the whole
training dataset. For example, the Shortcut attack (Yu et al., 2022) uses
linearly separable noises assigned with target labels to mislead the neural
network, resulting in the trained model with low accuracy on the test dataset.
The current security research in deep learning focuses on confidentiality and
integrity.

###  4.3. Adversarial capabilities and knowledge

The attacker’s capabilities refer to the amount of information the attackers
have and the actions they can perform. In the context of deep learning
systems, the attacker’s abilities, from weak to strong, can be classified as
follows: only being able to access hard labels and revising test data; being
able to access model output confidence; having the ability to access model
parameters and training data; modifying training data; modifying model
parameters.

According to the attackers’ understanding of models and data, the adversarial
knowledge can be classified into black-box, gray-box, and white-box models.
There is no clear dividing line between the three. Generally, the attacks that
know the model’s internal structure and parameters are called white-box
attacks. The attacks that can only access the classification results are
called black-box attacks, and the attacks in between are called gray-box
attacks. In reality, adversarial knowledge is usually determined by
adversarial ability, so the two are closely related.

##  5\. Adversarial attacks for 2D deep learning models

The adversarial attack, one of the most threatening attacks, has attracted
much research interest. In this and the next section, we will discuss the
adversarial attacks in the deep learning-based computer vision system, in
which the adversary’s goal is to cause the deep learning models to produce
wrong predictions. We divide them into 2D and 3D attacks according to the
different target model types. The 2D attacks mainly target the 2D models,
while the 3D attacks mainly target the 3D model by contaminating 3D data like
point clouds.

###  5.1. Catalog of 2D adversarial attacks

Although the 2D adversarial attack has been summarized in some previous
literature (Serban et al., 2020; Machado et al., 2021; He et al., 2022a), they
usually organize from a certain aspect, lacking systematization and the latest
research progress. Therefore, there is a need to reorganize the latest
studies, especially the emerging ones that cannot be classified into any
previous categories, such as color space perturbations and other semantic
perturbations. This paper summarizes the most recent progress to help readers
keep abreast of the newest tendency. Our survey classified 2D adversarial
attacks from various dimensions.

####  5.1.1. Classifying according to different distance metrics

When the term adversarial example was first proposed in 2013 (Szegedy et al.,
2013), its definition is leveraging crafted imperceptible perturbation to fool
the deep neural network, that is

(2) $\min_{x^{\prime}}\left\|{x^{\prime}-x}\right\|_{p}\text{, subject to
}\mathcal{F}(x^{\prime})\neq\mathcal{F}(x)\text{ and }x^{\prime}\in[0,1]^{m},$

where $p\in\\{0,1,2,\infty\\}$. Besides $l_{p}$-norm distance, recently there
are also other distance metrics are proposed to constrain the perturbation
size, such as color space distance (Shamsabadi et al., 2020b; Laidlaw and
Feizi, 2019), geodesic distance (Fawzi and Frossard, 2015), Wasserstein
distance (Zheng et al., 2019a), or even without any distance limitation (Song
et al., 2018).

####  5.1.2. Classifying according to physical achievability

Based on whether they can be realized in the physical world, 2D adversarial
attacks can be divided into digital attack and physical attacks. The digital
adversarial attacks assume that the adversary can directly modify the digital
images, while the physical adversarial attacks suppose that the attacker
cannot immediately revise the neural network’s input and, therefore, revise
the real-world objects instead. Physical attacks are more difficult than
digital attacks because of limited perturbation space and various
environmental variables, such as different viewpoints, distance, and
background illumination.

####  5.1.3. Classifying according to adversary’s knowledge level

As mentioned in the previous section, according to the adversary’s knowledge
extent, 2D digital adversarial attacks can be divided into white-box, gray-
box, and black-box attacks. Different knowledge extent can result in disparate
choices of attack methods because of different difficulties. Therefore, we
introduce the white-box attacks and black-box attacks separately.

####  5.1.4. Classifying according to the definition of perturbation

According to the perturbation’s difference, Gilmer et al. (Gilmer et al.,
2018) classified adversarial perturbations into imperceptible, content-
preserving, and unconstrained perturbation. The first one sets the original
image as the starting point and adds invisible noises to this image. This
constraint makes the AE almost the same as the clean one in human eyes and
usually uses $l_{p}$-norm as their distance metrics. For content-preserving
perturbation, the adversary retains the semantics of the original image and
misleads the classifier at the same time by methods like changing colors
(Hosseini and Poovendran, 2018; Laidlaw and Feizi, 2019) and spatially
transforms the images (Fawzi and Frossard, 2015). These attacks usually use
semantic distance to restrain the perturbation. For unconstrained input, the
adversary induces erroneous results from the neural networks through any
examples. For instance, Song et al. (Song et al., 2018) generates AEs from
scratch.

Based on the above observation, we classify 2D adversarial attacks into
digital white-box and black-box adversarial attacks, and physical adversarial
attacks, as shown in Figure.1.

Table 2. Summary of main white-box adversarial attacks in 2D CV tasks sorted
by the algorithm and published year

Attacks | Year |  Threat model |  Algorithm |  Distance |  Performance |  Key idea
---|---|---|---|---|---|---
Goal |  Knowl.* |  Efficiency |  Camouflage |  ASR **
L-BGFS (Szegedy et al., 2013) | 2014 |  T |  $\square$ |  Optimization |  $L_{2}$ |  Costly |  Invisible | 100.0% |  Box constrained L-BGFS
JSMA (Papernot et al., 2016b) | 2016 |  T |  $\square$ |  Optimization |  $L_{0}$ |  Efficient |  Slight | 97.2% |  $L_{0}$ \+ greedy
UAP (Moosavi-Dezfooli et al., 2017) | 2017 |  U |  $\square$ |  Optimization |  $L_{1},L_{\infty}$ |  Medium |  Invisible | 93.7% |  Universal perturbation
C&W (Carlini and Wagner, 2017) | 2017 |  T&U |  $\square$ |  Optimization |  $L_{0},L_{2},L_{\infty}$ |  Medium |  Invisible | 100.0% |  Substitute loss function
EAD (Chen et al., 2018b) | 2017 |  T&U |  $\square$ |  Optimization |  $L_{1},L_{2}$ |  Costly |  Invisible | 100.0% |  $L_{1}$ \+ $L_{2}$ distance
DAG (Xie et al., 2017) | 2017 |  U |  $\square$, |  Optimization |  $L_{\infty}$ |  Efficient |  Invisible | 69.0% |  Segmentation
SV-UAP (Khrulkov and Oseledets, 2018) | 2018 |  U |  $\square$ |  Optimization |  $L_{2},L_{\infty}$ |  Medium |  Marked | 60.0% |  Singular vector
GD-UAP(Mopuri et al., 2018a) | 2018 |  U |  $\square$,$\color[rgb]{.5,.5,.5}\blacksquare$ |  Optimization |  $L_{\infty}$ |  Medium |  Invisible | 83.5% |  Data-free UAP
RobustAdv (Luo et al., 2018) | 2018 |  U |  $\square$,$\blacksquare$ |  Optimization |  Percep. |  Costly |  Slight | 98.5% |  Perceptual sensitivity
TAP (Zhou et al., 2018) | 2018 |  U |  $\square$ |  Optimization |  $L_{2}$ |  Efficient |  Marked | 46.7% |  Low-pass filter
DAA(Zheng et al., 2019a) | 2019 |  U |  $\square$ |  Optimization |  $L_{\infty}$ |  Medium |  Marked | 45.0% |  Advl-data distribution
SparseFool (Modas et al., 2019) | 2019 |  U |  $\square$ |  Optimization |  $L_{1}$ |  Medium |  Slight | 100.0% |  Projected $L_{1}$-DeepFool
DF-UAP(Zhang et al., 2020b) | 2020 |  T&U |  $\square$ |  Optimization |  $L_{\infty}$ |  Efficient |  Slight | 96.2% |  Targeted data-free UAP
sC&W (Zhang et al., 2020a) | 2020 |  T&U |  $\square$ |  Optimization |  Smooth |  Medium |  Marked | 98.0% |  Smooth perturbation
GreedyFool (Dong et al., 2020) | 2020 |  U |  $\square$ |  Optimization |  $L_{0}$ |  Costly |  Invisible | 94.6% |  Greedy algorithm
SSAH (Luo et al., 2022) | 2022 |  T&U |  $\square$ |  Optimization |  Low Freq. |  Medium |  Invisible | 99.8% |  High-frequency limitation
FGSM (Goodfellow et al., 2014) | 2015 |  U |  $\square$ |  Fast Gradient |  $L_{\infty}$ |  Efficient |  Marked | 72.3% |  One-step gradient attack
Deepfool (Moosavi-Dezfooli et al., 2016) | 2016 |  U |  $\square$ |  Fast Gradient |  $L_{1},L_{2},L_{\infty}$ |  Medium |  Invisible | 90.0% |  Boundary dist. estimation
BIM&ILCM (Kurakin et al., 2018) | 2016 |  T&U |  $\square$ |  Fast Gradient |  $L_{\infty}$ |  Medium |  Slight | 90.0% |  Iteration FGSM
MI-FGSM (Dong et al., 2018) | 2017 |  T&U |  $\square$, |  Fast Gradient |  $L2,L_{\infty}$ |  Efficient |  Slight | 100.0% |  Momentum
PGD (Madry et al., 2017) | 2017 |  U |  $\square$ |  Fast Gradient |  $L_{2},L_{\infty}$ |  Medium |  Slight | 99.2% |  Projection grad. descent
R-FGSM (Tramèr et al., 2017a) | 2018 |  T&U |  $\square$ |  Fast Gradient |  $L_{\infty}$ |  Efficient |  Slight | 64.8% |  Random FGSM
BPDA&EOT (Athalye et al., 2018a) | 2018 |  U |  $\square$ |  Fast Gradient |  $L_{2},L_{\infty}$ |  Efficient |  Invisible | 100.0% |  Attack obfuscated grad.
MDI2FGSM(Xie et al., 2019) | 2019 |  T&U |  $\square$ $\blacksquare$ |  Fast Gradient |  $L_{\infty}$ |  Costly |  Slight | 62.2% |  Random transformation
TI-BIM (Dong et al., 2019) | 2019 |  U |  $\square$ |  Fast Gradient |  $L_{\infty}$ |  Costly |  Slight | 82.0% |  Translation kernels
DDN (Rony et al., 2019) | 2019 |  U |  $\square$ |  Fast Gradient |  $L_{2}$ |  Efficient |  Slight | 100.0% |  Adjustable step size
HP-UAP(Zhang et al., 2021) | 2021 |  T&U |  $\square$,$\blacksquare$ |  Fast Gradient |  $L_{\infty}$ |  Efficient |  Invisible | 91.1% |  Frequency filter
NAG-UAP (Mopuri et al., 2018b) | 2018 |  U |  $\square$ |  GAN |  $L_{\infty}$ |  Costly |  Slight | 94.1% |  Generative UAP
ATN (Baluja and Fischer, 2018) | 2018 |  T |  $\square$ |  GAN |  $L_{2}$ |  Efficient |  Slight | 95.9% |  GAN+reranking
UnresGM(Song et al., 2018) | 2018 |  T |  $\square$ |  GAN |  $-$ |  Costly |  Marked | 84.0% |  GAN from scratch
GAP (Poursaeed et al., 2018) | 2018 |  T&U |  $\square$ |  GAN |  $L_{2},L_{\infty}$ |  Efficient |  Slight | 74.1% |  UAP and img-dependent
AdvAttGAN(Joshi et al., 2019) | 2019 |  U |  $\square$ |  GAN |  $-$ |  Costly |  Marked | 98.0% |  Modify facial attribute
SemAdv (Qiu et al., 2020) | 2020 |  T |  $\square$,$\blacksquare$ |  GAN |  $-$ |  Costly |  Marked | 67.7% |  Modify visual attribute
SparseGAN (He et al., 2022b) | 2022 |  U |  $\square$ |  GAN |  $L_{0}$ |  Efficient |  Invisible | 58.4% |  Perturbation decoupling
Manitest (Fawzi and Frossard, 2015) | 2015 |  U |  $\square$ |  Spatial Trans. |  Geodesic |  Costly |  Invisible | 35.6% |  Geodesics on manifold
SimpleTrans (Engstrom et al., 2018) | 2017 |  U |  $\square$ $\blacksquare$ |  Spatial Trans. |  $-$ |  Medium |  Invisible | 90.0% |  Simple transformation
Manifool (Kanbak et al., 2018) | 2018 |  U |  $\square$ |  Spatial Trans. |  Geodesic |  Efficient |  Invisible | 75.0% |  Iterative method
stAdv (Xiao et al., 2018b) | 2018 |  T |  $\square$ |  Spatial Trans. |  T.V. |  Efficient |  Slight | 99.6% |  Flow field
Adef (Alaifari et al., 2018) | 2019 |  T |  $\square$ |  Spatial Trans. |  $L_{2}$ |  Medium |  Invisible | 99.0% |  Iterative deforming
3DRender (Zeng et al., 2019) | 2019 |  U |  $\square$ |  Spatial Trans. |  $L_{2}$ |  Costly |  Marked | 90.7% |  3D renderer
PSI (Zheng et al., 2019a) | 2019 |  U |  $\square$ |  Spatial Trans. |  Wasserstein |  Efficient |  Invisible | 91.7% |  Wasserstein distance
EdgeFool (Shamsabadi et al., 2020a) | 2020 |  U |  $\square$ |  Spatial Trans. |  Smooth |  Costly |  Marked | 99.0% |  Image enhancement
Chroma-shift (Aydin et al., 2021) | 2021 |  T&U |  $\square$ |  Spatial Trans. |  $-$ |  Efficient |  Invisible | 96.1% |  YUV colorspace
FilterFool (Shamsabadi et al., 2021) | 2021 |  U |  $\square$ |  Spatial Trans. |  SSIM |  Costly |  Slight | 48.3% |  Mimic filter
Semantic (Hosseini and Poovendran, 2018) | 2018 |  U |  $\square$ |  Color Trans. |  $-$ |  Costly |  Marked | 94.3% |  HSV colorspace
Blind-Spot (Zhang et al., 2018a) | 2019 |  U |  $\square$ |  Color Trans. |  kNN |  Efficient |  Marked | 100.0% |  Blind Spot
ReColorAdv (Laidlaw and Feizi, 2019) | 2019 |  U |  $\square$ |  Color Trans. |  Smooth |  Medium |  Slight | 97.0% |  RGB/CIELUV colorspace
cAdv&tAdv (Bhattad et al., 2019) | 2020 |  T |  $\square$ |  Color Trans. |  $-$ |  Medium |  Marked | 99.7% |  Colorization and texture
PerC attack(Zhao et al., 2020a) | 2020 |  U |  $\square$ |  Color Trans. |  CIEDE2000 |  Efficient |  Invisible | 100.0% |  Perceptual color distance
ColorFool (Shamsabadi et al., 2020b) | 2020 |  U |  $\square$,$\blacksquare$ |  Color Trans. |  $-$ |  Costly |  Marked | 95.90% |  Adversarial colorization

  * *

This column is the adversarial knowledge of different attacks. $\square$:
white-box. $\blacksquare$: black-box. ${\color[rgb]{.5,.5,.5}\blacksquare}$:
gray-box.

  * **

We only count the best result of the most difficult attack reported in the
papers. E.g. for works that have both white-box and black-box attacks, we only
report the latter, for it is more difficult. For works with both untargeted
and targeted attacks, we report the latter for the same reason.

###  5.2. White-box adversarial attack for 2D deep learning models

Table 2 summarizes recent year’s 2D adversarial attacks for the white-box
model. We divide them into the following categories according to their
specific attack methodologies.

  * •

Optimization-based attacks. This kind of attack usually describe finding AEs
as objective optimization problems and solve these problems by existing or
self-defined objective optimization method, include L-BFGS (Szegedy et al.,
2013), UAP (Moosavi-Dezfooli et al., 2017), C&W (Carlini and Wagner, 2017),
EAD (Chen et al., 2018b) and OptMargin (He et al., 2018), etc.

  * •

Fast-gradient-based attacks. Instead of reaching a local minimum through
optimizing an objective, this kind of attack finds adversarial examples
through direct and explicit gradient computation. Therefore, they can usually
find an adversarial example very quickly, although the perturbation may not be
the optimum. Moreover, because these attacks lack clear objective functions,
they are usually untargeted.

  * •

GAN-based attacks. Instead of optimizing the noises through gradient descent,
this kind of attack generates the perturbation through the generative
adversarial network (GAN) and optimizes the variable in the latent space. For
example, unrestricted AE (Song et al., 2018) can generate semantic adversarial
examples from drafts.

  * •

Spatial-transformation-based attacks. Unlike additive noises, this attack
utilizes global or local spatial transformations to generate adversarial
examples. The former includes Manifool (Kanbak et al., 2018) etc. and the
latter includes stAdv (Xiao et al., 2018b) etc. This kind of attack usually
uses semantic loss rather than p-norm-based distance metrics.

  * •

Colorization-transformation-based attacks. Some works also transform images
into different color spaces and manipulate the color space instead because
human beings prefer to classify objects according to their shapes rather than
colors. These attacks include ColorFool (Shamsabadi et al., 2020b), Chroma-
shift (Aydin et al., 2021) and SemanticAdv (Hosseini and Poovendran, 2018),
etc.

Some attacks also combine several different methods. For example, Zhao et al.
(Zhao et al., 2019) combined transformation and pixel-wise perturbations to
enhance the attack strength. However, most attacks adopt one of the above
methods as their main methodology. Therefore, we will introduce these attacks
according to their primary methodology.

####  5.2.1. Fast-gradient-based attacks

The greatest difference between optimization-based and fast-gradient-based
attacks is that the latter can quickly find adversarial examples in one or
several iterations. However, they usually cannot reach a local optimum and
need larger perturbation to mislead the models. Because of their high
efficiency in finding adversarial examples, they can better combine with the
adversarial training process to train robust neural networks.

##### Single-step fast gradient methods

In 2015, Goodfellow et al. (Goodfellow et al., 2014) conjectured that the AE
exists because of the linear representation of features in high-level space
and proposed gradient-based one-step attack method named fast gradient sign
method (FGSM), which estimates adversarial examples by $x^{\prime}=x+\epsilon
sign(\bigtriangledown_{x}\mathcal{L}_{\mathcal{F}}(x,y)).$ Moreover, they
embedded FGSM into adversarial training to mine hard examples and train robust
models. The loss function of FGSM-based adversarial training is

(3)
$\widetilde{\mathcal{L}}_{\mathcal{F}}=\lambda\mathcal{L}_{\mathcal{F}}(x,y)+(1-\lambda)\mathcal{L}_{\mathcal{F}}(x+\epsilon
sign(\bigtriangledown_{x}\mathcal{L}_{\mathcal{F}}(x,y)),y).$

The loss function has two parts. The first part is the same as the normal
training procedure, while the second part considers the negative influence of
adversarial examples.

##### Multi-step fast gradient methods

Kurakin et al. (Kurakin et al., 2018) extended the FGSM method to an iterative
method (BIM). At the end of each step, they clipped the image into [0,1]. They
also proposed LLCM, which uses the least likely label as the target label. In
addition, they tried to attack the camera through printed images. but their
attacks suffered low ASRs in the physical setting, for they lacked
consideration of fabrication distortion.

Moosavi et al. (Moosavi-Dezfooli et al., 2016) proposed Deepfool attack, which
iteratively estimates the distance from the normal samples to the classifying
hyperplane. Because of the nonlinearity of the classification boundary, they
linearize the hyperplane at each iteration. If the model $\mathcal{F}(x)$ is a
binary classifier, at each iteration, $x$ is updated by $x^{t+1}\leftarrow
x^{t}-\frac{\mathcal{F}(x^{t})}{\left\|{\bigtriangledown\mathcal{F}(x^{t})}\right\|_{2}^{2}}\bigtriangledown\mathcal{F}(x^{t})$.
If the model $\mathcal{F}$ is a multiclass differentiable classifier, they use
a polyhedron $\widetilde{P}_{i}$ to approximate the label space. Deepfool is
also untargeted but can achieve smaller perturbations than the FGSM attack.

Madry et. al. (Madry et al., 2017) proposed projection gradient descent (PGD),
which projects $x_{t}$ onto the neighborhood of the input at the end of each
iteration through
$x^{t+1}=\textit{proj}_{x+\mathcal{S}}(x^{t}+\epsilon\,sign(\bigtriangledown_{x}L(\theta,x,y)))$.
Osadchy et al. (Osadchy et al., 2017) also proposed a similar method called
I-FGSM and apply it to the image CAPTCHAs to distinguish between computers and
humans.

Tramer et al. (Tramèr et al., 2017a) found that the steep curvature artifacts
near the inputs may reduce the attack strength of single-step attack.
Therefore, they proposed R-FGSM, which introduces random noises into the input
images. The update formulation is $x^{adv}=x^{\prime}+(\epsilon-\alpha)\cdot
sign(\bigtriangledown_{x^{\prime}}\mathcal{L}(x^{\prime},y_{true}))$, where
$x^{\prime}=x+\alpha\cdot sign(\mathcal{N}(\textbf{0}^{m},\textbf{I}^{m}))$,
$\mathcal{N}(\textbf{0}^{m},\textbf{I}^{m})$ is the standard normal
distribution. R-FGSM attack can be regarded as a single-step variant of the
PGD attack.

MI-FSGM (Dong et al., 2018) introduced a momentum term into the iterative FGSM
and achieved better performance than FGSM and BIM in the untargeted attack. It
firstly updated the momentum by $g_{t+1}=\mu\cdot
g_{t}+\frac{{\bigtriangledown}_{x}\mathcal{L}(x_{t}^{\prime},y)}{\left\|{\bigtriangledown}_{x}\mathcal{L}(x_{t}^{\prime},y)\right\|_{1}}$,
then updated the adversarial example $x_{t}^{\prime}$ by
$x_{t+1}^{\prime}=x_{t}^{\prime}+\alpha\cdot sign(g_{t+1})$. By generating a
velocity vector in the direction of gradient descent, MI-FGSM alleviates the
problem of unstableness after multi-step iterations and enhances the
transferability of AEs.

M-DI2-FGSM (Xie et al., 2019) integrated random transformations into the MI-
FGSM to improve the adversarial example’s transferability. Random resize and
padding are used as the main transforms, and the transferability is increased
by about 20% on the ImageNet dataset. However, this method may cause the loss
to decline unstably. Dong et al. (Dong et al., 2019) also proposed a similar
attack named TI-BIM based on BIM and transformation. Because the gradient of
transformed images equals transforming the gradient of original examples, they
multiply the gradient of original images with a uniform, linear or Gaussian
kernel at each iteration to represent different translations.

Rony et al. (Rony et al., 2019) proposed Decoupling Direction and Norm attack
(DDN) based on PGD. At each iteration, the perturbation $\delta_{k}$ is
projected to an $\epsilon_{k}$-ball around $x$ through
$x_{k}^{\prime}\leftarrow
x+\epsilon_{k}\frac{\delta_{k}}{\left\|\delta_{k}\right\|_{2}}$. Different
from PGD that sets $\epsilon_{k}$ as a constant, they increase the
$\epsilon_{k}$ at each iteration if $x_{k}^{\prime}$ is still not adversarial
and reduce the $\epsilon_{k}$ on the other hand. Their method achieved
competitive results compared with state-of-art $L_{2}$-norm attacks and needed
fewer iterations.

The relationships between different fast-gradient-based adversarial attacks
are shown in Figure.3.

Figure 3. The relationships between different fast-gradient-based 2D
adversarial attacks

The relationships between different fast-gradient-based attacks

####  5.2.2. Optimization-based attacks

Optimization-based attacks usually model the adversarial as optimization
objectives and then leverage existing optimization methods or design custom
optimization methods to solve these optimization problems. Moreover, most
optimization-based attacks are based on $L_{p}$-norm distance. Different
distance metrics can generate different kinds of perturbations. $L_{2}$ and
$L_{\infty}$ distances usually generate uniform and dense perturbation, while
$L_{0}$ and $L_{1}$ distances tend to generate sparse perturbation. Moreover,
Besides image-independent attacks, universal perturbation has also attracted
much research interest. Therefore, according to the different kinds of
perturbations, we divide optimization-based attacks into dense, sparse, and
universal perturbations.

##### Dense perturbation.

When the concept of adversarial example is first proposed, it is defined as
minimizing the total perturbation in $L_{2}$ or $L_{\infty}$ distance because
they are easier to compute gradient than other distances. These metrics lead
to uniform perturbation on the whole image. The representative attacks include
L-BFGS (Szegedy et al., 2013), C&W (Carlini and Wagner, 2017), etc.

The concept of adversarial example was proposed by Szegedy et al. (Szegedy et
al., 2013) in 2013. They formulated the problem of finding adversarial
examples as a box-constrained optimization problem under $l_{2}$ distance and
utilized L-BFGS algorithm to find an approximate solution. In detail, for an
input image $x\in\mathbb{R}^{m}$, they minimize
$c\left\|{\delta}\right\|_{2}+\mathcal{L}_{\mathcal{F}}(x+\delta,y^{\prime}),\text{s.t.
}x+\delta\in[0,1]^{m}$, in which $\mathcal{L}_{\mathcal{F}}(\cdot,\cdot)$ is a
cross-entropy function. In addition, they also analyzed the upper Lipschitz
constant of the model and proposed that the regularization of the model
parameters may avoid the existence of adversarial examples. However,
experiments show that L-BFGS has the weakness of inefficiency.

Defensive distillation (Papernot et al., 2016c) improved the model robustness
by training models on soft training labels to reduce the network’s Jacobian
matrix, making the model less sensitive to the input and avoiding overfitting
against any samples. C&W (Carlini and Wagner, 2017) designed an alternative
loss to break defensive distillation, that is
$max(max({Z_{2}(x^{\prime})_{i:i\neq
y^{\prime}}})-Z_{2}(x^{\prime})_{y^{\prime}},-\kappa),$ where $\kappa$ is a
parameter to control the strength of AE. Moreover, some former attacks, like
BIM (Kurakin et al., 2018), directly clip $x^{\prime}$ into $[0,1]^{m}$, which
may result in zero gradients. C&W changed the optimized variable from $\delta$
to $w\coloneqq arctanh(\delta)$. Because $tanh$ is a bounded and differential
function, C&W can remove the box constraint and use Adam optimizer.

Gradient mask (Papernot et al., 2017) prevented adversarial attack through
shattered, randomized, or vanishing/exploding gradients. Athalye et al.
(Athalye et al., 2018a) claimed that gradient masks also cannot provide enough
robustness. They proposed BPDA, EOT, and Reparameterization attacks to break
these defenses, respectively. Take BPDA as an example. Gradient shattering
introduces a non-differentiable layer $f^{i}(x)$ to prevent the adversary from
getting the gradient. BPDA attack adopts a differentiable layer $g(x)$ to
substitute $f^{i}(x)$ in the back-propagation process.

Adversarial training (Szegedy et al., 2013) is also a popular defensive
method. It uses adversarial examples as training data to improve model
robustness (See Eq.3). Adversarial training through PGD (Madry et al., 2017)
has been proven effective in defending most first-order attacks, like FGSM and
DeepFool. However, Zhang et al. (Zhang et al., 2018a) found that adversarial
trained networks may have blind spots or unusual examples. They first use kNN
to measure the difference from the test example to the training dataset and
adopt simple methods like scaling and shifting all pixels to find the blind-
spot example of the model. Then, they attack these infrequent samples by C&W
attack.

To improve the transferability of adversarial perturbation, TAP attack (Zhou
et al., 2018) increases the feature distance of all latent layers between the
normal and adversarial image. In addition, they also found that applying a
low-pass filter on the perturbation can also improve transferability.
Observing that optimizing on an independent example cannot be globally
optimal, DAA (Zheng et al., 2019a) searches an adversarial data distribution
that satisfies $L_{\infty}$ constraint while maximizing the generalization
error by Wasserstein gradient flows. DAA can reduce the accuracy of CIFAR to
44.71% for adversarial-trained networks. To make the perturbation more
indistinguishable, sC&W (Zhang et al., 2020a) adds Laplacian smoothing
constraint on the original C&W attack. This constraint makes the perturbation
locally smooth on the even areas and dissonant on the high variance area. Like
the sC&W attack, SSAH (Luo et al., 2022) composes a semantic similarity loss
and a low-frequency restraint. The former minimizes the cosine similarity of
the features between the original and target image. The latter restrains the
perturbation into the high-frequency regions.

Semantic segmentation and object-detecting tasks are also significant in
computer vision. Xie et al. (Xie et al., 2017) proposed dense adversary
generation (DAG) attack against multi-object detection task. Observing that
this task usually needs to classify plural targets in an image, DAG attacks
all targets simultaneously using the iterative gradient descent. DAG can also
transfer among different models, which is probably because it accumulates
perturbations from multiple targets.

##### Sparse perturbation.

In some scenarios, the number of pixels being modified is more crucial than
the overall magnitude of the perturbation. Therefore, some researchers
proposed generating sparse perturbations through $l_{0}$ or $l_{1}$ distance,
equivalent to modifying as few pixels as possible. However, specific
techniques are needed because these distances are non-differentiable. For
example, JSMA (Papernot et al., 2016b) selects the most meaningful pixels by
comparing the gradient of $Z(x)$ and adds them to the set of modified pixels
one by one until the attack succeeds or a certain threshold is reached. EDA
attack (Chen et al., 2018b) formulates the problem as an Elastic-net
regularized optimization problem that combines the $l_{1}$ and $l_{2}$
distance. Because $l_{1}$ distance is non-differentiable, EDA applies the
FISTA algorithm to solve this optimization function. The $l_{1}$-DeepFool
(Moosavi-Dezfooli et al., 2016) can generate sparse perturbations efficiently
through linearizing the boundary, but it is greatly affected by the clipping
function. To solve this problem, SparseFool (Modas et al., 2019) projects
$x^{\prime}$ on one component of the normal vector at each iteration. If the
projection cannot generate an adversarial example, this direction will be
ignored in the next iteration. SparseFool iteratively estimates the minimum
perturbation to the boundary until the predicted label is changed. RobustAdv
(Luo et al., 2018) generates imperceptible and robust adversarial examples
through perception distance, which is defined as
$D(X^{\prime},X)=\sum_{i=1}^{N}\delta_{i}\cdot Sen(x_{i})$, where $N$ is the
total pixels being modified, $\delta_{i}$ is the perturbation of the pixel
$x_{i}$, and $Sen(x_{i})$ is the perturbation sensitivity of $x_{i}$. Because
humans are more sensitive to the perturbation in smoothing regions, RobustAdv
defines $Sen(x_{i})$ as the reciprocal of the standard deviation around
$x_{i}$. GreedyFool (Dong et al., 2020) uses a two-step greedy-based
optimization to generate sparse perturbation. Firstly, the candidate
manipulation locations are selected according to their gradient and their
distortion map generated by a GAN. Secondly, unnecessary points are discarded
to improve the sparsity. There are also some black-box attacks that can
generate sparse perturbation, such as OnePixel (Su et al., 2019) and
CornerSearch (Croce and Hein, 2019) and GeoDA(Rahmati et al., 2020). We will
discuss them in the next subsection.

##### Universal perturbation.

Rather than optimizing the perturbation on a specific image, universal
perturbation produces image-agnostic perturbations through optimizing
perturbation over the whole or part of the training dataset.

In 2017, Moosavi-Dezfooli et al. (Moosavi-Dezfooli et al., 2017) firstly
proposed the concept of universal adversarial perturbations (UAP). They
successfully attacked most images in the validation set by aggregating the
minimum perturbations on the training images. Experiments show that UAP attack
can not only attack unseen images but also transfer between different models
with a success rate of 40~60%. Later, in 2018, Khrulkov et al. (Khrulkov and
Oseledets, 2018) proposed SV-UAP attack, which approximates universal
perturbation problem as a (p, q)-singular problem. Suppose $J_{i}(x)$ is the
Jacobian matrix of $i^{th}$-to-last layer. For a small vector $\delta$,
$Z_{i}(x+\delta)-Z_{i}(x)\approx J_{i}(x)\delta$. Therefore, the problem of
finding a universal perturbation can be formulated as
$\max_{\delta}\;\sum_{x_{j}\in
X_{b}}\left\|J_{i}(x_{j})\delta\right\|_{q}^{q},\;s.t.\;\left\|\delta\right\|_{p}=C$
where $X_{b}$ is a subset of the training dataset. Then, they use the
stochastic power method to solve this (p,q)-singular vector problem and
achieved a fooling rate of 60% on 50000 images by using only 64 images for
optimization. Mopuri et al. (Mopuri et al., 2018a) proposed a generalizable
data-free UAP attack (GD-UAP) in which the adversary does not need to access
the exact training data. The idea is to find an image-agnostic perturbation
that triggers more additional model activation. The propagation effect of the
neural network will eventually lead to misclassification. They use the
knowledge about the distribution of the training dataset to improve the
fooling rate. In 2020, Zhang et al. (Zhang et al., 2020b) firstly proposed a
targeted UAP attack named DF-UAP. They analyzed the similarity between the
predicted logits of clean images and UAP using the Pearson correlation
coefficient and found that UAP has a dominant role in prediction compared with
clean images. Based on this discovery, they randomly sample the proxy dataset
at each iteration to update the perturbation. Inspired by the success of
steganography of neural networks, In 2021, Zhang et al. (Zhang et al., 2021)
proposed HP-UAP, which adds a high-pass filter in the iteration to make the
UAP more invisible to human eyes with only a small decrease of attack success
rate.

####  5.2.3. GAN-based attacks

The generative adversarial network (GAN) was proposed in 2014 by Goodfellow
and quickly became a powerful tool for learning the distribution of training
datasets. The first related work, adversarial transformation network (ATN)
(Baluja and Fischer, 2018), is reported in 2017. Its loss function is
$\min_{\theta}\sum_{x\in\mathcal{X}}\beta
L_{2}(\mathcal{G}(x),x)+L_{2}(f(\mathcal{G}(x)),r(f(x),t))$, where $r$ is a
reranking function to maintain the logits’ ranking order except the target
class. Song et al. (Song et al., 2018) proposed to use GAN to generate
unrestricted AEs from noises $z$ rather than adding perturbation. The loss
function includes three parts. The first is to make the victim model predict
$\mathcal{G}(z,y_{s})$ as the target class. The second is to limit the search
region of $z$, and the third is to make the auxiliary classifier predict
$\mathcal{G}(z,y_{s})$ as the source class $y_{s}$. They also engage humans to
evaluate the fidelity of AEs. Moreover, they proposed a noise-augmented
version to improve the ASR. Konda et al. (Mopuri et al., 2018b) proposed NAG-
UAP, which models the distribution of UAP by a UAP generator. The loss
function of the GAN network includes two parts: a fooling loss and a diversity
loss. The former’s goal is to minimize the classifier’s output on the ground
truth label, while the latter’s goal is to produce diversified UAPs by
maximizing the gap of the hidden layer’s outputs on different UAPs. Poursaeed
et al. (Poursaeed et al., 2018) proposed GAP attack that generates both
universal and image-independent noises for targeted and untargeted attacks.
For universal attacks, they use image-to-image translation networks to
generate universal perturbations from random noises and crop the perturbations
to have a fixed norm. Next, the perturbation is added to the normal images and
sent to the victim model to calculate the fooling loss. For image-independent
attacks, the perturbations are generated from the original image instead. He
et al. (He et al., 2022b) utilized a generative model to improve the
transferability of sparse perturbation. They first decoupled perturbation into
a magnitude component and a location component. Then, they add a sparse loss
onto the location component. Because the location component is a binary
operator, they proposed a binary quantization operator to train the generator.

The emergence of conditional neural networks makes directly editing
artificially designed properties possible (e.g. with and without eyeglasses).
Joshi et al. (Joshi et al., 2019) used a conditional generative model to
generate semantic adversarial examples. Their attack optimized over the
attributes space of the conditional generative model to manipulate semantic
features like wearing glasses or not and different skin colors to fool the
deep learning models. Qiu et al. (Qiu et al., 2020) also proposed a similar
method. But their method can choose arbitrary targets to attack. In addition,
they manipulated the interpolated feature space instead of the attributes
manifold.

####  5.2.4. Spatial-transformation-based attacks

Although deep neural networks like CNN are designed to be invariant and robust
to transformations like translation and rotation, researchers have shown this
is not always the case.

(a) ManiFool (b) stAdv

Figure 4. Comparing globally and locally transformation attacks. (a) ManiFool
attack (Kanbak et al., 2018) optimizes rotation and transition parameters on
the manifolds. (b) stAdv attack (Xiao et al., 2018b) optimizes the flow field
on pixel coordinate space. The pixel values are viewed as a function of the
pixel coordinates. The flow field is obtained by backpropagation from the
adversarial loss function to pixel coordinates.

The structure of this paper.

##### Globally spatial-transformation-based attacks

The first work of transformation-based attacks, Manitest (Fawzi and Frossard,
2015), was proposed in 2015. It casts the problem of computing the gap between
converted images as the geodesics along the transformation manifold, which is
defined as the minimum length of the curve on the manifold. Manitest utilizes
fast marching to find adversarial transformations, which can assure minimal
transformation on the searched grids. However, it suffers a low efficiency
when the transformation variety increases. SimpleTrans (Engstrom et al., 2018)
also uses simple spatial transformations to fool networks. It contains a
white-box algorithm based on gradient descent and a black-box algorithm based
on grid search. In 2018, Kanbak et al. (Kanbak et al., 2018) proposed Manifool
to analyze the transformation robustness of deep learning models, as shown in
Figure.4 (a). They also used the geodesic distance to measure the two
transformations’ distance. The Manifool consists of two steps: determining the
changing direction of the image and mapping this change onto the manifold.
Although their method is not as accurate as Manitest’s, they can achieve high
efficiency even with operating similarity and affine transformations
simultaneously.

##### Locally spatial-transformation-based attacks

As shown in Figure.4 (b), the stAdv attack (Xiao et al., 2018b) defined the
transformation as a spatial flow field, where each flow vector corresponds to
a displacement of the corresponding pixel. To enhance the local smoothness of
the flow field to make the AE more natural, they define a flow cost as the
total variation of the flow field. They also visualized the CAM attention and
showed that their method can better distract attention than C&W methods. In
2019, Alaifari et al. (Alaifari et al., 2018) proposed ADef attack inspired by
Deepfool. They iteratively deform the original image through gradient descent
to move the image over the boundary of the classification. The pixel’s value
is modeled as a function of its coordinate. Moreover, a two-dimensional
Gaussian filter is used to inflict smoothness on the flow field. In 2020,
EdgeFool (Shamsabadi et al., 2020a) generates adversarial examples through
enhancing image details. Contrary to ColorFool (Shamsabadi et al., 2020b),
EdgeFool only modifies the $L$ channel and leave $a$ and $b$ channel
unmodified. It firstly trains an FCNN network to learn a smoothing image
$I_{s}$ from the input image $I$, and modifies the image details
$I_{d}:=I-I_{s}$ to generate adversarial examples. Inspired by EdgeFool,
FilterFool (Shamsabadi et al., 2021) viciously modifies a picture by imitating
normal filter functions like detail enhancement, gamma correction, and log
transformation. Their attack comprises an FCNN network, a traditional filter,
and a classifier. The FCNN network is trained to generate perturbation that
simultaneously fools the classifier and makes the perturbed image’s structure
similar to the filtered image. Experiments show that FilterFool can improve
the transferability of adversarial examples. Wong et al. (Wong et al., 2019)
generated AEs by Wasserstein distance instead of $l_{p}$ distance, which
equals moving pixel mass in the images. Their attack follows the PGD attack
but projects $x$ onto a Wasserstein ball rather than $l_{\infty}$ ball and
uses projected Sinkhorn iteration (PSI) to accelerate the computation.
Compared to the $l_{p}$-based perturbation that indiscriminately affects both
foreground and background pixels, Wasserstein perturbation only changes the
foreground for images with monochrome backgrounds.

####  5.2.5. Colorization-transformation-based attack

Besides manipulating images in the spatial domain, some works transform images
into different color spaces and then modify the color space instead, for
humans prefer to classify images based on geometry rather than colors. These
attacks define the perturbation as ”content-preserving perturbation”, which is
usually measured by perceptual similarity (Wang et al., 2021c; Aydin et al.,
2021) rather than $l_{p}$ distance.

In 2018, Hosseini et al. (Hosseini and Poovendran, 2018) first proposed
SemanticAdv. It transforms the original image from RGB to HSV domain and
leverages random search to modify the Hue and Saturation components. However,
their adversarial examples may look factitious because hue and saturation
components are modified at the same time. ColorFool (Shamsabadi et al., 2020b)
expanded SemanticAdv to generate more realistic pictures by conducting
content-based segmentation on target images and revising the interesting and
unimportant regions, respectively. They modify the $a$ and $b$ channels in the
$Lab$ color space and leave the lightness channel $L$ unchanged. In 2019,
Laidlaw et al. (Laidlaw and Feizi, 2019) proposed ReColorAdv attack, which
computes the flow field in RGB or CIELUV color space. They also define a
smooth loss to improve imperceptibility. In addition, they prove that
combining the color space distortion with additive noises can expand the
perturbation space. Zeng et al. (Zeng et al., 2019) firstly edited the 3D
object’s physical properties, such as affine transformation, color, and
illumination, and then rendered it into an image to fool the classifier. They
optimize the physical variables by FGSM for derivable factors or the zeroth-
order optimization approach for non-derivable factors. Bhattad et al. (Bhattad
et al., 2019) generated unrestricted AEs through semantic perturbation like
colorization (cAdv) and texture transfer (tAdv). For cAdv attack, a pre-
trained colorization network (Zhang et al., 2016) is used to modify the color.
They attack the input hints and masks at the same time to make the color more
realistic. For the tAdv attack, they extract the texture from the target image
using a VGG19 network and leverage an extra restraint on the cross-layer gram
matrices to control the texture transfer’s strength. Experiments show that
they can generate photorealistic AEs. In 2020, Zhao et al. (Zhao et al.,
2020a) proposed PerC-C&W and PerC-AL attack. Their attacks directly operate in
the RGB color space and use CIEDE2000 as the perceptual color distance. PerC-
AL improves the optimizing efficiency by alternatively minimizing the
adversarial loss and perceptual loss. Experiments show that although the
perturbation size is slightly higher than C&W method in $L_{2}$ metric, it is
still imperceptible for human eyes and achieves better transferability. In
2012, inspired by the stAdv attack, Aydin et al. (Aydin et al., 2021) proposed
a more imperceptible attack called Spatial Chroma-shift. The basic idea is
that humans are less sensitive to color changes than brightness changes.
Therefore, they only apply the spatial transformation to the colorspace. They
transform the color space from RGB to YUV and then only compute the flow field
on UV channels. Moreover, they use LPIPS and DISTS to evaluate the perceptual
similarity and show that their method outperforms C&W and stAdv attacks.

###  5.3. Black-box adversarial attack for 2D deep learning models

All the above attacks generated AEs in the white-box assumption. However, in
reality, the adversary often has little prior knowledge of the model internal
architecture and parameters. Black-box attacks are proposed to solve this
problem.

####  5.3.1. Different black-box scenarios

According to the information the adversary can access, the black-box settings
can be classified into the query-constrained scenario, score-based scenario,
and hard label scenario (Ilyas et al., 2018b). For the first circumstance, the
attacker can only query the victim model a limited number of times. For
example, some APIs may charge fees if the adversary queries many times. For
the second circumstance, confidence scores are available, such as the Google
Cloud Vision API. In the last circumstance, only top-1 or top-k predicted
labels are output. For example, the Apple Photo app can automatically classify
the user’s pictures without displaying score information.

Table 3. Summary of main black-box adversarial attacks in 2D CV tasks sorted
by the algorithm and published year

Attacks | Year |  Threat Model |  Algorithm |  Distance |  Performance |  Key idea
---|---|---|---|---|---|---
Goal |  Knowl.* |  Efficiency** |  ASR***
TML (Papernot et al., 2016a) | 2016 |  U |  $\blacksquare$ | Subs. model |  $L_{\infty}$ |  Costly | 96.19% |  Data augment.
TIMI (Dong et al., 2019) | 2019 | U |  $\blacksquare$ | Subs. model | $L_{2}$ | Efficient | 49.00% | Tranform-invariant
FDA (Ganeshan et al., 2019) | 2019 | U |  $\blacksquare$ | Subs. model | $L_{\infty}$ | Costly | 80.20% | Feature distortion
ILA (Huang et al., 2019) | 2019 | U |  $\blacksquare$ | Subs. model | $L_{\infty}$ | Costly | 85.80% | Feature distortion
SIM (Lin et al., 2019) | 2019 | U |  $\blacksquare$ | Subs. model | $L_{\infty}$ |  Medium | 77.20% | Scale-invariant
TTTA (Li et al., 2020) | 2020 | T |  $\blacksquare$ | Subs. model | Poincare | Efficient | 42.90% | Pointcare distance
RDI (Zou et al., 2020) | 2020 | T&U |  $\blacksquare$ | Subs. model | $L_{\infty}$ | Costly | 67.80% | Multi-scale gradient
DI-MI-TI (Zhao et al., 2021b) | 2021 | T |  $\blacksquare$ | Subs. model | $L_{\infty}$ | Costly | 62.20% | Enough iteration
FIA (Wang et al., 2021a) | 2021 | U |  $\blacksquare$ | Subs. model | $L_{\infty}$ | Costly | 83.50% | Feature importance
VT(Wang and He, 2021) | 2021 |  U |  $\blacksquare$ | Subs. model | $L_{\infty}$ |  Medium | 76.50% |  Gradient variance
NAA(Zhang et al., 2022b) | 2022 |  U |  $\blacksquare$ | Subs. model |  $L_{\infty}$ |  Costly | 85.0% | Feature importance
ODI(Byun et al., 2022) | 2022 |  T |  $\blacksquare$ | Subs. model |  $L_{\infty}$ |  Efficient | 81.6% |  3D render
Img2video (Wei et al., 2022) | 2022 | U |  $\blacksquare$ | Subs. model |  $L_{\infty}$ |  Costly | 77.88% | Cross-modality
GNAE(Zhao et al., 2018) | 2018 |  U |  $\blacksquare$ |  GAN |  $L_{2}$ |  Costly | 78.00% |  Natural AE
AdvGAN(Xiao et al., 2018a) | 2018 |  T |  $\blacksquare$,$\color[rgb]{.5,.5,.5}\blacksquare$ |  GAN |  $L_{2}$ |  Efficient | 92.76% |  Distilled model
ATTA(Wu et al., 2021) | 2021 |  U |  $\blacksquare$ | GAN | $L_{\infty}$ |  Costly | 61.80% |  Adv transform
Boundary Attack (Brendel et al., 2017) | 2017 | T&U |  $\blacksquare$ | Decision | $L_{0}$ |  Costly | $-$ | Random walk
BiasedBA (Brunner et al., 2019) | 2019 | T&U |  $\blacksquare$ | Decision | $L_{2}$ | Efficient | 85.00% | Biased sampling
OPT(Cheng et al., 2019b) | 2019 |  T&U |  $\blacksquare$ | Decision |  $L_{2}$ |  Medium | 100.00% |  Binary search
HopSkipJump (Chen et al., 2020) | 2020 | T&U |  $\blacksquare$ | Decision |  $L_{2}$,$L_{\infty}$ | Efficient | 60.00% | Grandient+BA
SignOPT (Cheng et al., 2020) | 2020 | T&U |  $\blacksquare$ | Decision |  $L_{2}$ | Efficient | 94.00% | Estimate gradient sign
RayS (Chen and Gu, 2020) | 2020 | U |  $\blacksquare$ | Decision | $L_{\infty}$ | Efficient | 99.80% | Early stop
QAIR(Li et al., 2021) | 2021 |  U |  $\blacksquare$ | Decision | $L_{\infty}$ |  Efficient | 98.00% |  Image retrieval
BayesAttack (Shukla et al., 2021) | 2021 | T&U |  $\blacksquare$ | Decision |  $L_{2}$,$L_{\infty}$ |  Efficient | 67.48% | Bayesian optimization
Surfree (Maho et al., 2021) | 2021 | T&U |  $\blacksquare$ | Geometric | $L_{2}$ |  Efficient | 90.00% | Orthogonal projection
TangentAttack (Ma et al., 2021) | 2021 | T&U |  $\blacksquare$ | Geometric | $L_{2}$ |  Efficient | - | Semi-ellipsoid
TriangletAttack (Ma et al., 2021) | 2022 | T&U |  $\blacksquare$ | Geometric | $L_{2}$ |  Efficient | 44.5% | Triangle inequality
ZOO(Chen et al., 2017) | 2017 |  T |  $\blacksquare$ |  Score |  $L_{2}$ |  Costly | 97.00% |  Finite difference
LocSearchAdv(Narodytska and Kasiviswanathan, 2017) | 2017 |  T&U |  $\blacksquare$, $\color[rgb]{.5,.5,.5}\blacksquare$ |  Score |  $L_{0}$ |  Costly | 70.78% |  Local Search
Autozoom(Tu et al., 2019) | 2018 |  T |  $\blacksquare$ |  Score |  $-$ |  Efficient | 93.00% |  Autoencoder
PCA(Bhagoji et al., 2018) | 2018 |  U |  $\blacksquare$ |  Score |  $L_{\infty}$ |  Costly | 89.50% |  Principal component
SimBA(Guo et al., 2019a) | 2019 |  T&U |  $\blacksquare$ |  Score |  $L_{2}$ |  Medium | 96.50% |  Orthonormal basis
CornerSearch(Croce and Hein, 2019) | 2019 |  U |  $\blacksquare$ |  Score |  $L_{0}$ |  Costly | 99.56% |  Sparse noises
OnePixel(Su et al., 2019) | 2019 |  T&U |  $\blacksquare$, $\color[rgb]{.5,.5,.5}\blacksquare$ |  Score |  $L_{0}$ |  Costly | 67.97% |  Differential evolution
PBBA(Moon et al., 2019) | 2019 |  T&U |  $\blacksquare$ |  Score | $L_{\infty}$ |  Medium | 99.90% |  Discrete surrogate
Nattack (Li et al., 2019) | 2019 | U |  $\blacksquare$ |  Score |  $L_{2}$,$L_{\infty}$ | Costly | 100.00% | Adversarial distribution
Bandis (Ilyas et al., 2018a) | 2019 | U |  $\blacksquare$ |  Score |  $L_{2}$,$L_{\infty}$ |  Medium | 95.40% | Gradient priors
SignHunter(Al-Dujaili and O’Reilly, 2019) | 2020 | T&U |  $\blacksquare$ |  Score |  $L_{\infty}$,$L_{2}$ | Efficient | 91.47% | Grad. sign estimator
Square attack (Andriushchenko et al., 2020) | 2020 | T&U |  $\blacksquare$ |  Score |  $L_{2}$,$L_{\infty}$ | Efficient | 99.40% | Shrinking squares
Sparse-RS (Croce et al., 2022) | 2022 | T&U |  $\blacksquare$ |  Score |  $L_{\infty}$ |  Costly | 95.80% | Random search
NES(Ilyas et al., 2018b) | 2018 |  T&U |  $\blacksquare$ |  Score/Decision |  $L_{\infty}$ |  Costly | 88.20% |  Natural evolution
Subspace(Guo et al., 2019b) | 2019 |  U |  $\blacksquare$ | Subs.+Score |  $L_{\infty}$ |  Costly | 96.60% |  Multi-subs. model
P-RGF(Cheng et al., 2019a) | 2019 | U |  $\blacksquare$ | Subs.+Score | $L_{2}$ | Medium | 99.10% | Transfer-based priors
SimBA++(Yang et al., 2020) | 2020 |  U |  $\blacksquare$ |  Subs.+Score |  $L_{2}$ |  Efficient | 99.40% |  Mixed method

  * •

* This column contains different attacks’ adversarial knowledge. $\square$: white-box. $\blacksquare$: black-box. ${\color[rgb]{.5,.5,.5}\blacksquare}$: gray-box.

  * •

** For query-based attacks, we use efficiency to evaluate the queries needed
to find the adversarial example.

  * •

*** ASR is the abbr. of attack success rate. As mentioned, we only count the
best result of the hardest attack reported in the paper.

####  5.3.2. Categories of black-box 2D digital adversarial attacks

Different settings result in various methods. This review classifies black-box
attacks into three different categories according to their settings and
corresponding algorithms.

  * •

Substitute-model-based attacks. This method utilizes the transferability of
adversarial examples, which can convert black-box problems into white-box
problems, therefore significantly improving efficiency. However, limited by
the transferability of the surrogate model, the attack success rate is usually
not very high.

  * •

Score-based query attacks. We classify query-based attacks into score-based
and decision-based. The continuous confidence score can be accessed for score-
based attacks to estimate the decision boundary or gradient.

  * •

Decision-based query attacks. These attacks suppose the adversary can only
access the predicted labels of the model. They are far more difficult than
score-based attacks because less information can be leveraged.

  * •

Geometry-based query attacks. These attacks also aim at hard-label settings.
However, they use geometric information of the decision boundary rather than
estimate the gradients to improve the query efficiency.

  * •

Generative-model-based attacks. GAN can also be used for black-box attacks.
For example, in AdvGAN (Xiao et al., 2018a), the classifier is displaced by a
distilled model, which can be viewed as a transfer-based attack. In GNAE (Zhao
et al., 2018), random search algorithms are operated in the $z$ latent space,
which can be regarded as a query-based attack.

Some attacks also combine different methods together to improve the efficiency
and attack success rate, such as SimBA++ (Yang et al., 2020), P-RGF (Cheng et
al., 2019a), and Subspace attacks (Guo et al., 2019b). Table.3 shows some
representative 2D black-box adversarial attacks.

####  5.3.3. Substitute-model-based black-box attacks.

The substitute-model-based attack is based on the transferability of
adversarial examples. Therefore, improving the transferability is crucial for
this kind of method.

Data augmentation and transformation are widely used for improving
transferability. TML (Papernot et al., 2016a) used Jacobian-based dataset
augmentation to expand the dataset and train the substitute model
alternatively, which can be formulated as
$\mathcal{D}_{\rho+1}=(x+\lambda_{\rho}\text{sign}(J_{\hat{\mathcal{F}}}(x:x\in\mathcal{D}_{\rho})))\cup\mathcal{D}_{\rho}$,
where $\hat{\mathcal{F}}$ is the surrogate model. $\mathcal{D}_{\rho}$ and
$\mathcal{D}_{\rho+1}$ is the training dataset before and after $\rho^{th}$
iteration, $\lambda_{\rho}$ is a periodical step size. In addition, they
introduced reservoir sampling into the data augmentation process to reduce the
iteration number. DI (Xie et al., 2019) promoted transferability through
randomly resizing and padding the dataset. TIMI (Dong et al., 2019) improved
the transferability of MI-FGSM by enriching the input varieties, which is
equivalent to multiplying the gradient with a smoothing kernel, such as a
rotation, transition, or rescale matrix. SIM (Lin et al., 2019) used Nesterov
accelerated gradient and resized the image into different scales to prevent
the local model from overfitting. RDI (Zou et al., 2020) improved DI and TIMI
through multi-scale gradient and region fitting.

Targeted attacks are more difficult than untargeted attacks for surrogate-
model-based methods. Zhao et al. (Zhao et al., 2021b) found that a simple
targeted function $Z_{y^{\prime}}(F(x^{adv}))$ with sufficient iterations can
produce SOTA results, where $Z_{y^{\prime}}$ is the output logit of target
label. Byun et al. (Byun et al., 2022) improved the transferability of the
targeted attack through object-based diverse input (ODI), which utilized a
differentiable 3D renderer to render the 2D adversarial examples on 3D
objects. Noticing that the gradient of MI-FGSM tends to be dominated by the
past gradients after a few updates, Li et al. (Li et al., 2020) proposed TTTA
that adaptively adjusts the gradient through Pointcare distance. Through
Pointcare distance, the gradient increases when and only when the $f(x)$ comes
closer to the target class.

Some works found that manipulating the surrogate model’s latent layer features
can alleviate overfitting and utilize the shared features, resulting in
transferability improvement. Intermediate level attack (ILA) (Huang et al.,
2019) improved C&W and FGSM attacks by maximizing the distortion of a pre-
specified layer, while Feature disruptive attack (FDA) (Ganeshan et al., 2019)
disrupted all latent layer’s output. However, these two attacks regard all
neurons as equally important. The neurons that negatively influence the ground
truth prediction should be amplified rather than suppressed. FIA (Wang et al.,
2021a) judged the neuron importance by the mean gradient of a set of
transformed images. But this method suffers from gradient saturation. To solve
this, NAA (Zhang et al., 2022b) quantified neuron importance by computing
neuron attribution. Besides image classification tasks, Img2video attack (Wei
et al., 2022) attacked the black-box video model by generating the AE of each
frame on a surrogate model. They maximize the distance of low-level features
between normal and adversarial video frames.

Some attacks combine the surrogate model and other methods to get better
performance. P-RGF attack (Cheng et al., 2019a) utilized transfer-based priors
to get more accurate gradient estimation and save queries. They formulated the
gradient estimation problem as
$\min_{\hat{g}}L(\hat{g})=\mathbb{E}\left\|\bigtriangledown_{x}f(x)-b\hat{g}\right\|_{2}^{2}$,
where $b$ is a scaling factor. They utilized the estimated cosine similarity
between the actual and transferred gradient to control the transfer strength.
Their method can also be incorporated with other priors to save queries.
Subspace attack (Guo et al., 2019b) shrank the search space of stochastic
vectors by transferring the gradients from a group of substitute networks.
SimBA++ (Yang et al., 2020) also integrated transferability-based and queries-
based model. Moreover, SimBA++ updated the surrogate model based on the query
results.

####  5.3.4. Score-based query attacks

The goal of query-based attacks is to craft adversarial examples through fewer
queries and achieve a higher attack success rate. Based on whether the
continuous confidence score is accessible,query-based methods can be
classified into score-based and decision-based query methods.

##### Gradient-estimation-based methods

Most of the score-based query attacks are based on gradient estimation. The
simplest gradient estimation method is the finite difference, such as zero-
order optimization attack (ZOO) (Chen et al., 2017). For ZOO, the gradient is
estimated through $\frac{\partial f(x)}{\partial x_{i}}\approx\frac{f(x+\delta
e_{i})-f(x-\delta e_{i})}{2\delta}$, where $e_{i}$ is a standard basis,
$\delta$ is a random vector sampled from Gaussian distribution near the
original data point. Moreover, they used a resize function and increased
dimension gradually to improve efficiency. However, because they need to use
all the standard basis vectors in each iteration, the query cost is
proportional to the image size, which is computationally expensive. Later,
some improvements were proposed to reduce the random vectors to be sampled.
PCA attack (Bhagoji et al., 2018) reduced search dimensions by analyzing the
principal components of input data. Autozoom (Tu et al., 2019) firstly trained
an Autoencoder to encode the image of the target class into latent space.
Then, it sampled the random vectors from the latent space and mapped them to
random perturbations with reduced dimensions to compute the finite difference.
Al-Dujaili et al. (Al-Dujaili and O’Reilly, 2019) proposed a sign-based
gradient estimation algorithm. The basic idea is to estimate the sign of the
directional derivative. They proposed a divide and conquer method named
SignHunter that leverages the separability of the derivative of the
adversarial objective function, which decreases the query number from $2^{n}$
to $O(n)$.

Heuristic algorithms can also be used for gradient estimation. Ilylas et al.
(Ilyas et al., 2018b) estimated the gradient through a variant of natural
evolution strategy (NES). For scoreless settings, they leveraged noise
robustness to substitute the confidence score. Specifically, they sampled a
set of $\delta_{i}$ from a normal distribution by antithetic sampling. Then,
the expectation of estimated gradient of $F(x^{\prime})$ approximately equals
to $\frac{1}{\sigma n}\sum_{i=1}^{n}\delta_{i}F(x^{\prime}+\sigma\delta_{i})$.
Later, Ilylas et al. (Ilyas et al., 2018a) formulated the gradient estimation
problem as finding a vector to maximize $\mathbb{E}[\hat{g}^{T}g^{*}]$ and
used the least squares method to solve this problem and proved it is an
equivalence of NES. They utilized two kinds of priors to improve efficiency.
Firstly, the present gradient highly correlates with the gradient of the last
step. Secondly, adjacent pixels often have similar gradients. Therefore, they
designed a gradient estimation framework based on bandit optimization, where
the action is a priors-based gradient estimation in each round.

##### Greedy-algorithm-based methods

SimBA (Guo et al., 2019a) is a simple but powerful attack based on the greedy
algorithm. Instead of computing all basis vectors at each iteration (like ZOO
attack), it randomly chooses one vector from a predefined orthogonal basis at
each iteration. If this vector can reduce the confidence score, it is applied
to the image, and the next random vector is sampled. Otherwise, this direction
is deserted. They found that the DCT basis is especially efficient. Although
this method is simple, it outperforms NES and ZOO in the aspect of query cost.
SimBA++ (Yang et al., 2020) improved the query efficiency by combining SimBA
with the transferability-based attack TIMI (Dong et al., 2019). Instead of
randomly sampling changing direction, they correlate the sampling probability
to the surrogate model’s gradient. Moreover, they proposed high-order gradient
approximation (HOGA) to distill the target model in both forward and backward
steps. SimBA++ greatly decreases the query times and achieves a higher attack
success rate than SimBA.

##### Grid-search-based methods

$\mathcal{N}$attack (Li et al., 2019) tries to find a distribution of AEs in
the neighborhood of $x$. But it does not need to access the model parameters.
It formulates the optimization problem of finding adversarial distribution
around $x$ as
$\min_{\theta}J(\theta):=\int{f(x^{\prime})\pi_{S}(x^{\prime}|\theta)dx^{\prime}}$,
where $f(x^{\prime})$ is an untargeted attack loss function,
$\theta=(\mu,\sigma^{2})$ are the parameters of Gaussian distribution and
$x^{\prime}=x+\text{proj}_{S}(1/2(tanh(z)+1))$, where
$z\sim\mathcal{N}(z|\theta)$. To solve this problem, it uses grid search to
find the best $\sigma$ and updates $\mu$ through NES. PBBA (Moon et al., 2019)
searched AEs along the border of $l_{\infty}$ ball. This simplification makes
it possible to refine the query performance by optimizing discrete surrogate
problems. It firstly zones the images as squares and operates regional
optimization in the grid by choosing $x^{\prime}$ from
$\\{x-\epsilon,x+\epsilon\\}$. Then, it adjusts the squares and duplicates
this procedure until it finds an AE. Square attack reduced the search space
through diminishing squares. Unlike PBBA (Moon et al., 2019) that used a fixed
grid, the locations of squares are optimized. Moreover, they sampled $\delta$
along the boundary of $L_{p}$-norm ball to improve the query efficiency.
Specifically speaking, for $L_{\infty}$-square attack,
$x_{i}^{\prime}\in\\{x_{i}-\epsilon,x_{i}+\epsilon\\}$. For $L_{2}$-square
attack, because noise values are correlated, they increase the budget of one
window and simultaneously decrease the budget of another.

##### Dimension-reduction-based methods

Some works reduced the search dimensions by reducing the number of crafted
pixels. The first work is LocSearchAdv (Narodytska and Kasiviswanathan, 2017),
which used local search to estimate implicit gradient. It first randomly
picked up some pixels as initial points. At each round, the adjacent pixels of
these points are checked and updated. It only needed to manipulate 0.5% pixels
for the untargeted attack on ImageNet. CornerSearch (Croce and Hein, 2019)
also used local search to find sparse perturbations, but it only added
perturbations to pixels with high variation to make them imperceptible.
OnePixel attack (Su et al., 2019) considered the extreme situation and showed
the possibility of fooling the classifier by only revising one pixel. It used
differential evolution to optimize this problem. Each candidate solution
contains its pixel coordinates and color values. The candidate solutions are
updated by random crossover. Sparse-RS (Croce et al., 2022) utilized random
search, which is appropriate for zero-order optimization problems with sparse
restraint. It achieved the SOTA attack success rate on multiple attacks,
including the $l_{0}$-norm noises, adversarial patches, and adversarial
borders.

####  5.3.5. Decision-based query attacks

In some scenarios, the adversary can only get the final decision ( hard label)
rather than the confidence score (soft label). This is a more challenging case
because the gradient value is hard to estimate.

##### Gradient-estimation-based methods

Some works proposed to estimate the gradient directions instead. The first
work is Boundary attack (Brendel et al., 2017), which first initializes a
point outside the ground truth region and then uses a random walk algorithm to
make it closer to the boundary at each iteration. Although it can realize
almost the same perturbation level compared with white-box attacks, it needs
exponential time to find adversarial examples. Biased boundary attack (Brunner
et al., 2019) used bias sampling to re-understand and improve boundary
attacks. Three different biases are utilized. The first bias is sampling more
vectors in the direction of low-frequency distortion. The second bias is
sampling denser data from regions where the adversarial and benign images have
larger differences. The last bias is the prior-based gradient from the
substitute model. HSJA (Chen et al., 2020) includes three steps: binary search
to find the boundary point, estimation of gradient direction, and geometric
progression to update the boundary point. Because the gradient guides the
search direction, HSJA significantly decreases query number compared to the
Boundary attack.

OPT (Cheng et al., 2019b) is another representative hard-label-based attack,
which estimated the shortest distance of the benign sample from the boundary
by fine-grained search and binary search. They estimated the gradient of the
search direction through the randomized gradient-free method and updated the
search direction through gradient descent. SignOPT (Cheng et al., 2020)
promoted OPT’s efficiency by only estimating the sign of gradient over search
direction by just one query and averaging the gradient sign over a group of
random directions. RayS (Chen and Gu, 2020) reframed the successive problem of
finding the nearest classification hyperplane as a discontinuous problem
without gradient estimation, which searches over a group of ray orientations.
Moreover, all unnecessary directions are terminated early through a quick
check. This greatly promoted search efficiency. Bayes attacks (Shukla et al.,
2021) reduced query budgets to 1000 through Bayesian optimization. However,
because Bayesian optimization is not suitable for search space with large
dimensions, they reduce the search dimension through FFT and use nearest-
neighbor upsampling to find adversarial examples. Their method significantly
reduced the query times compared with OPT and Sign-OPT attacks. QAIR (Li et
al., 2021) considers a more difficult setting where the attacker can only get
top-k unlabeled images from the target model in the image retrieval task. They
proposed a correlation-based loss by the difference of top-k unlabeled
feedback retrieved by benign and adversarial images. Moreover, they improved
the efficiency through recursive network theft to get gradient priors.

##### Geometry-based methods

Geometry attacks also aim at the hard-label scenario. However, they directly
use geometric information of the decision boundary rather than estimate the
gradients to improve efficiency. Surfree (Maho et al., 2021) assumes the
boundary is a hyperplane and iteratively runs binary searches over orthonormal
directions to find the clean image’s projection on the boundary. Compared to
HSJA (Chen et al., 2020), the new direction is randomly sampled from the low-
frequency subband generated by DCT rather than using gradient directions,
which reduces the queries to a few hundred. TangentAttack (Ma et al., 2021)
assumed the boundary is a semi-ellipsoid and adjusted the search direction to
follow the optimal tangent line rather than the gradient directions, which
need less distortion than HSJA. TriangleAttack (Wang et al., 2022b) utilized
triangle inequality to search the boundary point and also used DCT for
dimension reduction. Less than a thousand quires are required in this attack.
CGBA (Reza et al., 2023) restrict the search on a semicircular path to make
sure to find a boundary point regardless of the boundary curvature, which is
quite efficient for untargeted attack..

####  5.3.6. Generative-model-based black-box attacks

GAN can also be used for black-box attacks. GNAE (Zhao et al., 2018) generates
natural AEs by searching for perturbations in the latent space rather than the
input space. It first learns a generator $\mathcal{G}_{\theta}$ that maps
normally distributed variables $z$ to the input $x$. It then learns an
inverter $\mathcal{I}_{\gamma}$ that maps $x$ back to latent space by
minimizing the divergence between $z$ and reconstructed
$\mathcal{I}_{\gamma}(x)$. Their objective is
$\min_{\tilde{z}}\left\|\tilde{z}-\mathcal{I}_{\gamma}(x)\right\|_{2}\;s.t.\;\mathcal{F}(\mathcal{G}_{\theta}(\tilde{z}))\neq\mathcal{F}(x)$.
The feasible $\tilde{z}$ is found through iterative stochastic search and
hybrid shrinking search.

Xiao et al. (Xiao et al., 2018a) proposed AdvGAN to improve the attack
transferability. AdvGAN consists of a perturbation generator, a discriminator,
and a classifier. The generator is trained to generate perturbations from the
original instance. The discriminator is trained to distinguish AEs from clean
instances. The classifier is a distilled model of the target black-box
network. They alternately train these modules by first training the GAN and
then training the classifier through $\min_{f}\mathbb{E}_{x\sim
D_{data}}\mathcal{L}(f(x),b(x))+\mathcal{L}(f(x+\mathcal{G}(x)),b(x+\mathcal{G}(x)))$
, where $b(x)$ is the target model, and $f(x)$ is the distilled model.
$\mathcal{L}$ is a cross-entropy loss function. Their black-box attack
achieved 92.7% on the MNIST dataset.

Some previous transfer-based attacks improved transferability through simple
transformations, like rotation and translation (Dong et al., 2019), random
resizing and padding (Xie et al., 2019) and multi-scaling (Lin et al., 2019).
Wu et al. (Wu et al., 2021) proposed ATTA to improve transferability through
learning adversarial transformations that can better alleviate the adversarial
property. They formulate this problem as a min-max problem. The inner problem
is to find $x_{adv}$ that maximizes the classification loss. The outer problem
is to learn a transformation through CNN to minimize the classification loss.

###  5.4. The safety impact of physical-realizable 2D adversarial attacks

Table 4. Summary of main works that examine the safety of real-world 2D CV
applications through physical adversarial attacks

Attacks | Year |  Threat Model | Algorithm | Distance |  Performance | Scenario
---|---|---|---|---|---|---
Goal |  Knowl.* |  Interface |  Camouflage |  Succ. **
UAP-patch (Liu et al., 2020a) | 2020 | U |  $\square$ |  Patch |  Gradient | Style loss |  Marked | 74.1% | Auto check-Out
Facial Acc. (Sharif et al., 2016) | 2016 |  T&U |  $\square$ |  Patch |  Gradient |  $L_{\infty}$ |  Marked | 80.0% |  Face rcg.
AGN(Sharif et al., 2019) | 2019 |  T&U |  $\square$ |  Patch | GAN |  $-$ |  Slight | 70.0% |  Face rcg.
FRSadv(Nguyen et al., 2020) | 2020 | T |  $\square$ | Light | Gradient | $L_{\infty}$ | Marked | 92.0% | Face rcg.
GenAP (Xiao et al., 2021) | 2021 |  T&U |  $\blacksquare$ |  Patch | GAN |  $L_{\infty}$ |  Marked | 65.0% |  Face rcg.
AdvMakeUp (Yin et al., 2021) | 2021 | T |  $\square$ |  Patch | Gradient | Style, Content | Slight | 22.0% |  Face rcg.
AdvPatch (Brown et al., 2017) | 2017 | T |  $\square$ |  Patch |  Gradient |  $L_{\infty}$ |  Marked | 93.0% |  Img clas.
EoT(Athalye et al., 2018b) | 2018 |  T |  $\square$ |  Patch |  Gradient |  $L_{2}$ |  Slight | 82.0% |  Img clas.
CiPer(Agarwal et al., 2020) | 2020 | U |  $\blacksquare$ | Sensor |  Greedy |  $L_{\infty}$ | Marked | 33.0% |  Img clas.
Nat-Patch (Hu et al., 2021) | 2021 | U |  $\square$ |  Patch | GAN | Smoth loss | Invisible | 48.0% | Object detect
CAMOU (Zhang et al., 2018b) | 2018 | U |  $\blacksquare$ |  Patch | Subs. Model | $-$ | Marked | 32.7% | Self-driving
RP2 (Eykholt et al., 2018) | 2018 | T |  $\square$ |  Patch |  Gradient | $L_{1},L_{2}$ | Marked | 84.8% | Self-driving
ShapeShifter (Chen et al., 2018a) | 2018 | T&U |  $\square$ |  Patch |  Gradient | $L_{2}$ | Marked | 87.0% | Self-driving
MeshAdv (Xiao et al., 2019) | 2019 | T&U |  $\square$ | Renderer |  Gradient | $Lp.$ | Slight | 100.0% | Self-driving
AdvCam(Duan et al., 2020) | 2020 | T&U |  $\square$ |  Patch | Gradient | Style loss | Slight | 80.0% | Self-driving
DAS (Wang et al., 2021b) | 2021 | U |  $\square$ |  Patch |  Gradient | $L_{2},T.V.$ | Marked | 20.0% | Self-driving
AMPLE (Ji et al., 2021) | 2021 | T&U |  $\blacksquare$ | Sensor | Bayesian | Energy loss | Marked | 87.9% | Self-driving
DirtyRoad (Sato et al., 2021) | 2021 | U |  $\square$ |  Patch |  Gradient | $L_{1},L_{2},L_{\infty}$ | Invisible | 97.5% | Self-driving
OPAD(Gnanasambandam et al., 2021) | 2021 | T |  $\square$ | Light | Gradient | $L_{2},L_{\infty}$ | Slight | 48.0% | Self-driving
DTA (Suryanto et al., 2022) | 2022 | T |  $\square$ |  Patch | Gradient | $-$ | Marked | 63.0% | Self-driving
TrajAdv (Zhang et al., 2022a) | 2022 | U |  $\square$$\blacksquare$ |  Patch | Gradient,PSO | $L_{\infty}$ | Slight | 62.2% | Self-driving
FCA(Wang et al., 2022a) | 2022 | U |  $\square$ |  Patch | Gradient | Smooth loss | Marked | 60.0% | Self-driving

  * •

* This column is the adversarial knowledge of different attacks. $\square$: white-box. $\blacksquare$: black-box. ${\color[rgb]{.5,.5,.5}\blacksquare}$: gray-box.

  * •

** For objector detection, the attack success rate means the average rate of
escaping from being detected. As mentioned, we only count the best result of
the hardest attack reported in the paper.

When deep learning models are employed in the real world, ensuring the safety
of human life and property is the primary concern. A number of studies have
investigated adversarial attacks in safety-critical applications, such as face
recognition and self-driving systems. referred to as physical adversarial
attack. This subsection reviews 2D physical adversarial attacks in safety-
critical applications.

The main challenge of physical adversarial attacks is the ever-changing
physical environment, such as the background light noise, varying viewpoints,
and different distances. Moreover, because the adversary cannot directly
modify the input images at the pixel level, certain mediums are needed to
pollute the data obliquely, such as patches (Sharif et al., 2016; Xiao et al.,
2021), illumination (Gnanasambandam et al., 2021; Li et al., 2023), and
sensors (Sayles et al., 2021). The patch-based attack is the most common
physical attack. It uses printed patches to cover the whole or part of the
target object to spoof the classifier. Some works also render 2D adversarial
images on 3D objects, like turtles (Athalye et al., 2018b) and cars (Wang et
al., 2021b). This can be seen as a more general form of adversarial patches.

####  5.4.1. The safety of face recognition system

As mentioned before, a major challenge for the 2D physical-world adversarial
attack is the changing environment, like varying distances and ambient light.
To make the AE more robust to these changes and can fool the classifier
consistently, Athalye et al. (Athalye et al., 2018b) proposed expectation on
transformation (EoT) that calculates the expectation of the log-likelihood of
the target class on transformed images, which is formulated as

(4) $x^{adv}=arg\min_{\Delta
x}\Sigma^{k}_{i=1}(w_{i}\mathcal{L}(\mathcal{T}_{i}(x)+\Delta
x,y)),\;s.t.\|\Delta x\|_{p}\leq\epsilon,$

where $\mathcal{T}_{i}$ is the $i_{th}$ transformation, such as random
rotations and transitions, $w_{i}$ is the weight and subject to
$\Sigma_{i=1}^{k}w_{i}=1$. Besides rotation and transition, they also regarded
the 3D rendering as a transformation. They use the $l_{2}$ norm in the LAB
color space as the distance metric because it is closer to human perception.
Brown et al. (Brown et al., 2017) inducted EOT, patch position, and multiple
training images into the optimization round to generate targeted, universal,
and robust adversarial patterns. However, their method has poor
transferability when the pattern size is small.

In 2016, Sharif et al. (Sharif et al., 2016) proposed facial accessory attack.
The perturbation is limited to the area of facial accessories like eyeglass
frames to make the attack more reasonable. They also use EOT to improve the
physical attack’s robustness. Moreover, they minimize the total variation of
the patches to make them more natural and use the NPS score to guarantee the
patch is printable. The total loss function is $arg\min_{\delta}\sum_{x\in
X}L_{s}(\mathcal{F}(x+\delta),l)+k_{1}TV(\delta)+k_{2}NPS(\delta)$, where
$L_{s}$ is a Softmax loss on the target label. $TV$ is the total variance
loss. $NPS$ is the non-printable score that defined as
$NPS(\hat{p})=\prod_{p\in P}|\hat{p}-p|$, where $P$ is the set of printable
colors. In 2019, Sharif et al. proposed AGN attack (Sharif et al., 2019),
which improved the inconspicuousness of adversarial patches through a
generative framework to generate AEs with multiple objectives, such as
robustness, imperceptibility, and scalability. Unlike previous work that uses
total variation as smoothness loss, they use GAN to ensure the generated
patches look like real-world designs.

In 2020, FRSadv (Nguyen et al., 2020) attacked the face recognition system
through adversarial illumination. They project adversarial patterns onto the
human face, and the camera captures the images and predicts wrong labels. They
also used EOT to improve the robustness. Adv-Makeup (Yin et al., 2021) fooled
the FRS through adversarial makeup. Their attack first generates realistic eye
shadow through GAN, then blends the generated eye shadow onto the source image
through the gradient, content, and style losses. Adv-Makeup attacks an
ensemble of models to improve transferability. FaceAdv (Shen et al., 2021) has
a sticker generator and a converter. The generator chooses the most vulnerable
area to attack, and the converter renders the patches to the face with
different angles and sizes to improve physical robustness.

In 2021, Xiao et al. (Xiao et al., 2021) proposed the GenAP attack, which
regularizes the patches on the latent space of GAN to make the adversarial
patches more natural and transferable. They first trained a StyleGAN on normal
face datasets, then used it to generate AEs and crop them into the patch area.
Instead of optimizing the image directly, they optimized the latent variable
$w$ on the $\mathcal{W}+$ space of StyleGAN to minimize the feature distance
between $x^{adv}$ and the target $x_{t}$. Experiments show that their methods
can improve the adversarial pattern’s transferability. However, their method
needs a pre-trained generator trained on the victim examples. Moreover, their
methods are difficult to converge, which may be because the cropping function
unexpectedly changes the AEs.

####  5.4.2. The safety of self-driving system

Figure 5. The general procedure of generating robust physical perturbation
through expectation over transformations (Eykholt et al., 2018; Athalye et
al., 2018b; Brown et al., 2017; Sharif et al., 2016). Images are sampled from
different distances and view angles to improve the real-world attack’s
robustness against position changes.

A relatively simple setting for adversarial attacks in self-driving is to hide
planar objects from the detector or classifier, like traffic signs and roads.
Chen et al. (Chen et al., 2018a) proposed ShapeShifter to attack the Fast
R-CNN detector. They applied EoT to the objection detector field to improve
the attack robustness. Rather than using the adversarial patch, they modify
the background of the stop sign directly. Eykholt et al. (Eykholt et al.,
2018) proposed robust physical perturbation (RP2) to make deep learning models
misclassify the road sign. They improved EoT by sampling from a set of both
simulated and physical distortions. The general procedure is shown in
Figure.5. They first operate $L_{1}$ optimization to find approximate
positions of the adversarial patches and then operate $L_{2}$ optimization to
modify their RGB values. AdvCam (Duan et al., 2020) generated adversarial
traffic signs with natural styles through neural style transfer. The loss
function consists of an adversarial loss, a style loss, a content loss, and a
smoothness loss. The style loss and content loss are calculated through the
features extracted from the shallow and deep layers of the feature extraction
neural network. Sato et al. (Sato et al., 2021) attacked the automated lane
centering system through dirty road patches, which may result in the car
running out of the street. Their objective function is to make the car drive
out of the lane in the shortest time. They achieved 97.5% ASR and only needed
0.903 sec for each attack. TrajAdv (Zhang et al., 2022a) evaluated the
robustness of the trajectory prediction of the self-driving car through
single-frame or multi-frame adversarial perturbation. They considered the
acceleration and speed limitation to make the adversarial perturbation more
natural. OPAD (Gnanasambandam et al., 2021) cheated the traffic sign
classifier by optical adversarial perturbation. To overcome the nonlinear
effect of the projector, they estimated the radiometric and spectral response
to rectify the distortion.

A more complex scenario is to hide nonplanar objects like cars from the
detector. Because the 3D renderer is not differential, and the vehicle
detector is usually a black box. CAMOU (Zhang et al., 2018b) used a distilled
model to learn the behavior of the 3D renderer and detector jointly. When
camouflage is changed, the original distilled model may fail to fit the 3D
transformations and vehicle detector. Therefore, they alternatively train the
distilled model and optimize the adversarial patches. Some studies utilized a
3D differential renderer to map 3D adversarial mesh or camouflage to 2D
images. Xiao et al. (Xiao et al., 2019) proposed MeshAdv to hide cars from 2D
detectors by printable 3D mesh. They leverage a differential renderer (Kato et
al., 2018) to map 3D adversarial mesh to 2D images. Laplacian loss is applied
on the mesh vertices to improve the smoothness. DAS (Wang et al., 2021b) hides
the car from the detector by simultaneously distracting the model and human
attention. They optimize the model attention distraction loss function by
minimizing the salient region and decreasing the salient region’s pixel
values. To make the image unnoticeable to humans, they initial the adversarial
camouflage as a natural image and then constrain the perturbation in its edge
area. AMPLE (Ji et al., 2021) can hide or create a car through a sensor-
injection attack by rotating and vibrating the camera to blur the image. It
uses Bayesian methods to optimize the rotation and vibration parameters. To
cheat the detector in multiple viewpoints, FCA (Wang et al., 2022a) colored
the adversarial camouflage onto the car’s whole surface through a differential
renderer. The loss function includes three parts. The first part is to cut
down the IoU between the original and predicted boxes. The second part is to
decrease the objectness confidence. The third part is to lower the predicted
logits of the target class. Suryato et al. (Suryanto et al., 2022) found that
the differential neural renderer that previous works used fails to perform
diverse physical world conversions because of an absence of domination of
environment variances. Therefore, they proposed DTA attack to hide the car
from the detector, which utilized a differential transformation network to get
photorealistic images.

Besides face recognition and self-driving scenario, the safety of other
scenarios against adversarial attacks also are investigated. Liu et al. (Liu
et al., 2020a) evaluated the safety of the automatic check-out system through
universal adversarial patches. They leveraged perceptual and semantic biases
of models to improve the generalization ability. Hu et al. (Hu et al., 2021)
leveraged StyleGAN and BigGAN to generate adversarial patches with natural-
looking content to evade from person detector. They minimized the objectness
score and classification confidence of the target class simultaneously.
Agarwal et al. (Agarwal et al., 2020) leveraged the noise produced by the
environment and imaging process to reduce the classification accuracy of
black-box models. The noises are extracted through Gaussian and Laplacian
filters.

##  6\. Adversarial attacks for 3D deep learning models

###  6.1. The difference between 3D and 2D adversarial attacks

As a result of affordable 3D data acquisition devices and the rich information
provided by the geometry feature, 3D data are widely used in many safe-
critical tasks. As a result, their security also attracts more and more
attention. Compared with 2D adversarial attacks, 3D adversarial attacks have
some significant differences:

  * •

Compared with images, point clouds contain point-wise coordinates with
unordered and irregular sampling values. Some 2D attacks that utilize the
image structure cannot be directly applied to these kinds of data, such as the
boundary attack, which estimates the boundary point by the weighted average of
pixel value between source and target images. Interpolation between point
clouds directly will deform the 3D shapes.

  * •

3D adversarial attacks have a larger disturbance space and degree of freedom
than 2D adversarial attacks. In addition to modifying the value, the adversary
can add and delete points.

  * •

Unlike 2D adversarial attacks, which use all pixels for classification, point
clouds do not use all points. Due to the large number of points, the point
cloud is usually sampled before the classification. For example, PointNet
sampled 1024 points for 3D object classification on the ModelNet40 dataset.

  * •

Most 2D adversarial attacks are based on pixel-wise $L_{p}$ distance. However,
point-wise attacks can be easily defended by outlier point removal, and
$L_{p}$ distance is unsuitable for disordered data, like the point cloud.

Therefore, previous cannot be directly used for 3D adversarial attacks. To
solve these problems, 3D adversarial attacks are proposed for 3D data
specifically. We will introduce these attacks according to their algorithms
and scenarios.

###  6.2. Catalog of 3D adversarial attacks

We summarize the recent works of 3D digital and physical adversarial attacks
in Table.5. According to the attack algorithms used for generating adversarial
examples, we classify the 3D adversarial attacks into the following
categories,

  * •

Gradient-based optimization methods. For most 3D adversarial attacks,
gradient-based methods are used. Some of them are based on $C\&W$ loss, such
as (Xiao et al., 2018a) and (Tsai et al., 2020), while others are based on
fast-gradient methods, such as (Liu et al., 2019). Some works relax
$L_{0}$-norm attack to differential versions, such as (Zheng et al., 2019b).

  * •

Generative-model-based methods. Some works, like AdvPC (Hamdi et al., 2020)
and LG-GAN (Zhou et al., 2020), utilize a generative model to generate
adversarial point clouds. These attacks can improve the transferability of 3D
adversarial examples.

  * •

Heuristic-algorithm-based methods. In black-box settings, the gradient of the
3D model is unavailable. Hence, some black-box attacks generate adversarial
examples using heuristic algorithms like evolution algorithms.

  * •

3D-transformation-based methods. Rather than modifying the point cloud, some
attacks found that 3D models are also vulnerable to rigid-body transformations
in the 3D Euclidean space, like rotations and translations.

Recently, some works also proposed 3D physical-realizable adversarial examples
by 3D printing (Tsai et al., 2020) or laser emitter (Cao et al., 2019a). In
addition, some studies (Wen et al., 2020; Huang et al., 2022) classified the
3D digital adversarial attacks into point dropping, point adding, and point
shifting attacks according to different perturbation types. However, some 3D
attacks are not based on the point cloud, such as 3D mesh attack (Cao et al.,
2019b) and volumetric network attack (Wicker and Kwiatkowska, 2019).
Therefore, we classify 3D attacks by the algorithms rather than the
perturbation types.

###  6.3. 3D adversarial attacks in the digital world

####  6.3.1. Gradient-based method

Some attacks are based on 2D adversarial attacks but with new loss functions.
The first work of 3D adversarial attack, 3DAdv (Xiao et al., 2018a), is based
on C&W. It includes the independent point and adversarial cluster attacks. The
first attack uses Hausdor and Chamfer distance to measure the maximum and
average distance between the original and the adversarial point cloud. Chamfer
distance is defined as
$\mathcal{D}_{C}(x,x^{\prime})=\frac{1}{\left\|x^{\prime}\right\|}\sum_{p^{\prime}\in
x^{\prime}}\left(\min_{p\in x}\left\|p^{\prime}-p\right\|_{2}^{2}\right)$. The
second attack used the Chamfer and the farthest pair-wise distance to generate
an adversarial cluster. They achieved a 100% ASR with an acceptable noise
budget. However, they only evaluated the 3D-Adv attack on the PointNet
network, which was proven less adversarial robust than other 3D models, like
PointNet++. Liu et.al. (Liu et al., 2019) extended PGD to 3D-PGD. They
projected the perturbation to the tangent plane of the adversarial point cloud
to reduce the outlier points. This is equivalent to changing the distribution
of the surface points. However, Tsai et.al. stated that this attack could be
easily defended by a resampling method (Tsai et al., 2020).

Some attacks discard important points or generate points with specific shapes.
Zheng et al. (Zheng et al., 2019b) evaluated each point’s importance through
point discarding. Because point discarding is non-differential, they relax it
by moving points towards the point cloud’s interior. Then, the critical score
is evaluated by the directional derivative in the spherical coordinates.
ShapeAdv (Liu et al., 2020c) includes three kinds of perturbations: uniform
distribution perturbation, adversarial sticks, and adversarial sinks. The
first one generates evenly distributed perturbation through resampling during
the optimization. The second one adds auxiliary features like adversarial
sticks. However, because the stick’s angle is difficult to optimize, they use
a projection and clip function to approximate it. The last one pulls points
into the inside of the point cloud. MinimalAdv (Kim et al., 2021) fools the
classifier by manipulating fewest points. They formulated the problem as a
$L_{0}$ and $L_{2}$ optimization problem and relaxed it to $L_{1}$ problem.
However, their method drops significantly after the resampling process because
only a few points are adversarial, and most points (¿95%) are benign.

Some attacks are proposed to improve the smoothness of the AEs. Tsai et.al.
(Tsai et al., 2020) used Chamfer and kNN distance to generate a geometry-aware
AE. The kNN loss can restrict the distance between neighboring points in the
point cloud and, hence, can significantly reduce outlier points. They
successfully achieved targeted attacks on PointNet++ in the physical world by
3D-printed objects. However, their method results in rough surfaces.
$GeoA^{3}$ (Wen et al., 2020) improves the fidelity of AE by combining Chamfer
distance and the consistency of local curvatures between $\mathcal{P}$ and
$\mathcal{P}^{\prime}$, which is measured through the direction of normal
vectors. Moreover, to make $\mathcal{P}^{\prime}$ more robust to resampling,
they proposed $GeoA_{+}^{3}$ that contains a uniformity loss to promote the
regularity of surface points. Then, they proposed iterative normal projection
to optimize this objective function. Huang et al. (Huang et al., 2022)
proposed shape-invariant attack to improve the AE’s surface smoothness. They
shift the point along the tangent plane of the surface so that the
perturbations are more imperceptible than the previous methods. Moreover, they
combined the surrogate model and query-based methods to improve the black-box
attack’s efficiency. They first calculated a sensitivity map on the surrogate
model according to the maximum gradient magnitude. Then, they shifted points
according to the sensitivity ranking. This query method can reduce the query
times by about 20% compared with SimBA (Guo et al., 2019a), and SimBA++ (Yang
et al., 2020) attacks.

Recently, statistical outlier removal (SOR) has been proposed to defend
against adversarial point clouds. To break this countermeasure, JGBA (Ma et
al., 2020) embeds SOR into the attack algorithm. But because SOR is not
differentiable, they replace it with a relaxation function using first-order
approximation. Then, they optimize the original and SOR-filtered point cloud
simultaneously. This attack successfully defeated SOR and SOR-based DUP-Net
defenses.

Some attacks aim to improve the transferability of 3D adversarial examples.
ITA (Liu and Hu, 2022) limits the perturbation along the direction of the
normal vector and improves the transferability through adversarial
transformations. The adversarial transformation composes a simple two-layer
neural network and is learned through an adversarial training procedure. AOF
(Liu et al., 2022) improves the transferability by boosting the classification
loss of low-frequency component, which is separated from the original PC by
orthogonal decomposition of the graph Laplacian matrix.

Table 5. Summary of main digital and physical adversarial attacks in 3D CV
tasks sorted by the algorithm and published year

Attacks | Year |  Threat Model | Method | Distance** |  Performance | Key idea
---|---|---|---|---|---|---
Goal |  Knowl* |  Interface |  Efficiency |  Succ.
3DAdv (Xiao et al., 2018a) | 2019 | T | $\square$ | Digital | Gradient |  $L_{2}$, $Cf.$ | Efficient | 99% | Chamfer loss
3D-PGD(Liu et al., 2019) | 2019 | U | $\square$ | Digital | Gradient | $L_{2}$ | Efficient | 88.10% | Gradient projection
Saliency(Zheng et al., 2019b) | 2019 | U | $\square$ | Digital | Gradient | $L_{0}$ | Efficient | 40% | Saliency map
LidarAdv (Cao et al., 2019b) | 2019 | T&U | $\square$ | Physical | Gradient | $L_{2},Lp.$ | Costly | 71% | Proxy function
Adv-Lidar (Cao et al., 2019a) | 2019 | T&U | $\square$ | Physical | Gradient | $L_{0}$ | Costly | 75% | Global sampling
KNN(Tsai et al., 2020) | 2020 | U&T | $\square$ | Physical | Gradient | $Cf.,kNN$ | Efficient | 94.69% | 3D printable
Rooftop(Tu et al., 2020) | 2020 | U |  $\square$ $\color[rgb]{0,0,0}\blacksquare$ | Physical | Gradient | $Lp.$ | Costly | 80% | Adversarial mesh
$GeoA^{3}$, $GeoA_{+}^{3}$(Wen et al., 2020) | 2020 | T | $\square$ | Digital | Gradient | $Cf.,Hd.,LC$ | Efficient | 100% | Local curvatures
ShapeAdv(Liu et al., 2020c) | 2020 | U | $\square$ | Digital | Gradient | $L_{2}$ | Costly | 95% | Shape attack
JGBA (Ma et al., 2020) | 2020 | T&U |  $\square$ $\color[rgb]{.5,.5,.5}\blacksquare$ | Digital | Gradient | $L_{2}$ | Efficient | 98.90% | Break SOR
CTRI(Zhao et al., 2020b) | 2020 | T | $\square$ | Digital | Gradient | Spectral | Efficient | 98% | Restricted isometry
MinimalAdv(Kim et al., 2021) | 2021 | U | $\square$ | Digital | Gradient | $L_{2},L_{0},Cf.,Hd.$ | Costly | 89% | Minimal perturbation
ShapeInv(Huang et al., 2022) | 2022 | U |  $\square$ $\color[rgb]{0,0,0}\blacksquare$ | Digital | Gradient |  $L_{\infty}$ | Efficient | 100.00% | Sensitivity map
AOF (Liu et al., 2022) | 2022 | U | $\square$ | Physical | Gradient | $L_{\infty}$ | Costly | 99.76% | Low frequency
AdvPC (Hamdi et al., 2020) | 2020 | U |  $\square$ $\color[rgb]{0,0,0}\blacksquare$ | Digital | GAN |  $L_{\infty}$ | Costly | 64.40% | GAN
LG-GAN (Zhou et al., 2020) | 2020 | T | $\square$ | Digital | GAN | $L_{2}$ | Efficient | 97% | Generative model
ITA (Liu and Hu, 2022) | 2022 | T&U |  $\square$ $\color[rgb]{0,0,0}\blacksquare$ | Digital | GAN | $L_{2},Cf.$ | Efficient | 29.89% | Adv. transform
EvolutionAdv(Cao et al., 2019b) | 2019 | U | $\color[rgb]{0,0,0}\blacksquare$ | Physical | Evolution | $-$ | Costly | 62% | Evolution
Camdar-adv (Chen and Huang, 2021) | 2021 | T | $\color[rgb]{0,0,0}\blacksquare$ | Physical | Evolution | $Lp.$ | Costly | 99% | Multi-modality
ISO (Wicker and Kwiatkowska, 2019) | 2019 | U |  $\square$ $\color[rgb]{0,0,0}\blacksquare$ | Digital | Greedy | $L_{0}$ | Costly | 100% | Critical point set
OcclusionPoint(Sun et al., 2020) | 2020 | U | $\color[rgb]{0,0,0}\blacksquare$ | Physical | Greedy | $L_{0}$ | Efficient | 80% | Lidar occlusion
TSI(Zhao et al., 2020b) | 2020 | U | $\color[rgb]{0,0,0}\blacksquare$ | Digital | Random | $-$ | Efficient | 95% | Thompson sampling

  * •

* This column is the adversarial knowledge of different attacks. $\square$: white-box. $\blacksquare$: black-box. ${\color[rgb]{.5,.5,.5}\blacksquare}$: gray-box.

  * •

** Cf.: Chamfer distance. Hd: Hausdorff distance. Lp: Laplacian distance. LC:
Local curvature distance.

####  6.3.2. Generative-model-based methods

Hamdi et al. (Hamdi et al., 2020) proposed AdvPC attack, which used GAN model
to produce an adversarial point cloud. The loss of AdvPC has two parts. The
first one is the pre-trained classifier’s loss. The second is the auto-encoder
loss to ensure that the reconstructed point cloud can still fool the
classifier. Experiments show that AdvPC attacks can improve the
transferability of adversarial point clouds. Moreover, AdvPC outperforms
3D-adv and KNN attacks on several different defense methods. However, they
only evaluated untargeted attacks.

LG-GAN (Zhou et al., 2020) generated targeted adversarial examples through the
multi-branch generative network. It first learns the multi-layer features of
the input 3D data through the GAN and then unitizes a class encoder to mix the
label information into the multi-layer features. At last, the 3D data are
rebuilt from the features. The GAN includes a classification loss, a
$L_{2}$-norm reconstruction loss, and a graph adversarial loss.

####  6.3.3. Heuristic-algorithm-based methods

In the black-box scenario, the gradient is hard to obtain. Therefore, black-
box attacks are often based on heuristic algorithms. Cao et al. (Cao et al.,
2019b) proposed an evolution-based black-box algorithm EvolutionAdv. They set
the population as mesh vertices of the object and the adaption function as
$-L(f(x^{adv}))$ and randomly add some novel individuals at each period with
perturbations sampling from a normal distribution. Wicker et al. (Wicker and
Kwiatkowska, 2019) proposed an Iterative Salience Occlusion attack (ISO) to
break PointNet and volumetric networks through a greedy algorithm. They first
identify the vital point set through queries. Then, they drop the most
critical points iteratively until the classification result is false.

####  6.3.4. 3D-transformation-based methods

TSI (Zhao et al., 2020b) found that 3D models are vulnerable to affine
transformations like rotations and transitions. Therefore, they proposed to
attack the 3D model by isometric transformation. A random algorithm based on
Thompson sampling is proposed to optimize the rotation angles. Compared with
random sampling, Thompson sampling is more productive and can unitize the
prior knowledge. CTRI (Zhao et al., 2020b) searched an adversarial isometric
transformation (such as rotation) by minimizing the spectral-norm loss, which
is equivalent to finding a smallest $\delta$ such that
$(1-\delta)\left\|x\right\|^{2}\leq\left\|Ax\right\|^{2}\leq(1+\delta)\left\|x\right\|^{2}$,
where $A$ is the rotation matrix and $x$ is the point cloud.

###  6.4. The safety impact of physical-realizable 3D adversarial attacks

In recent years, 3D physical adversarial attacks have been proposed to cheat
Lidar-based detectors through printable mesh (Tu et al., 2020) or laser
emitter (Cao et al., 2019a). These attacks pose severe threats to real-world
applications. We classify these attacks into simulation-based and physical-
system-based 3D attacks based on their implementation methods.

####  6.4.1. Simulation-based 3D attacks

Some of them are implemented in the simulation setting through a Lidar
renderer. For example, Cao et al. (Cao et al., 2019b) proposed an
optimization-based algorithm LidarAdv for hiding 3D mesh from Lidar detection.
They first soften the preprocess function through the differential proxy
function. Then, they generated adversarial mesh through $L_{2}$ loss. However,
they only evaluated their attack on the Apollo platform in the simulation
setting rather than the newest detection models. Tu et al. (Tu et al., 2020)
proposed Rooftop attack to hide the self-driving car from Lidar by placing an
adversarial mesh on the vehicle. They optimize the initial mesh through a
fooling loss and a Laplacian loss to improve its smoothness. Their vehicle-
agnostic perturbation can achieve an 80% occlusion rate. But they also only
simulated their attack with CAD models and a Lidar renderer.

####  6.4.2. Physical-system-based 3D attacks

Later, some attacks are proposed to attack real-world Lidar sensors. Cao et
al. (Cao et al., 2019a) proposed Adv-Lidar that can mislead a real-world self-
driving system to detect nonexistent obstacles. Rather than modifying 3D mesh,
they manipulate the 3D point cloud by laser emitter instead. They first
analyzed a real-world self-driving system and found that previous attacks
cannot create unreal barriers because of the preprocessing steps. So, they
considered the preprocessing steps in the optimization. Then, they added
adversarial points into the pristine point cloud in the limited angle and
distance ranges. In addition, they used global sampling to avoid being caught
in a local minimum. Their method successfully cheated a real-world self-
driving system. Chen et al. (Chen and Huang, 2021) proposed a multi-modality
attack that attacks Lidar and the camera simultaneously through the 3D
adversarial mesh and 2D adversarial patch. They used the evolution algorithm
to find 2D perturbations and used Tu et al.’s method (Tu et al., 2020) to find
3D adversarial mesh. Sun et al. (Sun et al., 2020) proposed a black-box attack
against the Lidar detector in the self-driving setting. They noticed that the
detect models are vulnerable to distant and occluded vehicle attacks, in which
they can fool the detector by just a few points. Their attacks can cheat BEV-
based, voxel-based, and point-based 3D object detectors.

##  7\. Future directions and challenges

###  7.1. Improving the transferability of adversarial examples

A real-world feasible adversarial attack should be able to attack unseen
models. Therefore, the transferability of adversarial samples has attracted
more and more attention. There are several promising directions to improve the
transferability of adversarial examples, including

  * •

Random/adversarial transformations. These methods assume that adversarial
examples that can survive random or adversarial transformations can better
transfer between different models. The existing methods include isometry
transformation (Dong et al., 2019), random resizing (Xie et al., 2019), multi-
scale images (Lin et al., 2019), or learnable adversarial transformation (Wu
et al., 2021). However, most of these transformations are very simple. It is
still unknown why such vanilla transformations can improve the transferability
and whether better transformations exist.

  * •

Generating perturbation by generative model. Some works (Xiao et al., 2018a,
2021) proposed optimizing the latent feature can improve transferability.
However, in our experiment, we find that searching adversarial perturbation on
the latent space may cause the optimization to be unstable or non-converging.
So, there is still room for improving the generative model’s architecture and
loss function.

  * •

Manipulating the latent layer features. Some works (Ganeshan et al., 2019;
Huang et al., 2019; Wang et al., 2021a; Zhang et al., 2022b) maximize the
middle layer features’ difference between the pristine and adversarial images
to improve the transferability. But Zhang et al. (Zhang et al., 2022b) have
shown that these methods also have great promotion space.

  * •

Ensemble-based approaches. Some works also improve transferability by
attacking an ensemble of models (Dong et al., 2018; Yin et al., 2021).
However, training multiple substitute models is computationally expensive. How
to train ensemble models efficiently and how to design or select substitute
models are still open problems.

Moreover, at present, most of these methods improve transferability
empirically. Few of them proved a lower bound of the transferability or
explained the mechanism behind the transferability. One possible way of
understanding transferability is through the similarity of decision
boundaries. For example, Liu et al. (Liu et al., 2016), and Tramèr et al.
(Tramèr et al., 2017b) analyzed the decision boundary of different models and
contributed the transferability to their similar boundaries. However, this
theory cannot explain the asymmetry of transferability (Wu et al., 2018).
Ilyas et al. (Ilyas et al., 2019) suggests adversarial perturbation is non-
robust features rather than noise, and different models may learn the same
non-robust features. Therefore, the AEs can transfer between them. However,
they only prove their hypothesis on a simple dataset with two classes and
cannot explain why sometimes the two models predict the same AE as different
false labels. Finding more reasonable theoretical interpretations of
transferability is still an urgent task for robust deep learning.

###  7.2. Towards semantic perturbation

Imperceptible perturbation by $L_{p}$-norm distance is vulnerable to physical
variables, like distance and background light. Therefore, some works proposed
to fool the classifier while maintaining semantic consistency for human
beings. At present, the main directions and challenges of semantic
perturbation include:

  * •

Color-space transformations. Some studies claimed that the DNN is biased
toward texture, while human beings like to classify objects according to their
shapes (Geirhos et al., 2018). Therefore, some works operated color distortion
in different color spaces to generate semantic-preserving perturbation.
However, how to better measure the perceptual distance to be consistent with
human perception is still an open problem.

  * •

Global/Local spatial transformations. Global and local spatial transformations
can also generate adversarial examples while reserving semantics. However,
because it’s hard to compute gradients regarding the transformation
parameters, there is still a demand to find an efficient way to optimize the
transformation parameters.

  * •

Manipulating image content attributes. Some attacks utilized the conditional
GAN to modify the image content attributes. However, better disentangling the
latent features and controlling the attributes still need research.

  * •

Attacks based on diffusion model. Recently, the diffusion model (Ho et al.,
2020) has been proposed to produce high-quality images, which has outperformed
GAN on some tasks. A few works have used them for generating unconstrained
semantic AEs (Chen et al., 2023). But these works usually estimate the
gradients roughly. A more precise method to calculate the gradients, like the
stochastic differential equation, may improve the performance.

In addition, although many works explore the semantic-reserving perturbations,
few of them tried to apply themselves to the physical world to bypass the
camera system. There still is a question of whether these semantic
perturbations can behave consistently in the physical setting.

###  7.3. Making adversarial examples physically achievable

Generating physical-realizable AEs that are robust enough to physical
condition changes and hard to notice simultaneously is still an open question.
Physical attacks are more difficult than digital attacks because of the
changing physical variables like view angles and distance. Some work (Athalye
et al., 2018b; Brown et al., 2017) proposed to improve the attack robustness
through expectation over transformation (EoT). However, EoT significantly
improves the perturbation size. One possible solution to this trade-off is
using semantic perturbation, like colorization-based attacks. The multi-
objective optimization strategy (Jia et al., 2022) is also a possible way to
balance stealthiness and attack strength.

Moreover, because implementing physical experiments needs lots of expenditure,
some researchers simulated the physical adversarial attack through a
differential 3D renderer. However, current rendering techniques are
susceptible to geometric and lighting transformations that distort the
synthesized image. The gap between realistic photos and rendered images can
affect the attack success rate or even make the adversarial attack
ineffective. There is still a need to find better renderer or simulation
methods or construct real-world datasets for evaluation.

In addition, present physical AEs often attack the target deep learning model
directly. However, off-the-shelf applications often contain complete
pipelines, including data acquisition and preprocessing, which may
unexpectedly influence the effect of AEs. For example, the image filter and
the point cloud sampling process may influence the ASRs. There is still