# SoK: Analyzing Adversarial Examples: A Framework to Study Adversary
Knowledge

Lucas Fenaux
University of Waterloo
<EMAIL_ADDRESS>Florian Kerschbaum
University of Waterloo
<EMAIL_ADDRESS>
###### Abstract

Adversarial examples are malicious inputs to machine learning models that
trigger a misclassification. This type of attack has been studied for close to
a decade, and we find that there is a lack of study and formalization of
adversary knowledge when mounting attacks. This has yielded a complex space of
attack research with hard-to-compare threat models and attacks. We focus on
the image classification domain and provide a theoretical framework to study
adversary knowledge inspired by work in order theory. We present an
adversarial example game, inspired by cryptographic games, to standardize
attacks. We survey recent attacks in the image classification domain and
classify their adversary’s knowledge in our framework. From this
systematization, we compile results that both confirm existing beliefs about
adversary knowledge, such as the potency of information about the attacked
model as well as allow us to derive new conclusions on the difficulty
associated with the white-box and transferable threat models, for example,
that transferable attacks might not be as difficult as previously thought.

##  1 Introduction

Many machine learning (ML) models are currently deployed in critical
environments like healthcare [1], self-driving cars [2], aerospace and
aviation [3], and many more. Model failures in these settings can directly
incur grave consequences on human life. While there exists a large body of
literature on the safe and secure deployment of these models, the proposed
solutions remain mainly academic, difficult to implement and replicate, and
fall short of the security and safety standards we would typically expect from
critical systems. Despite all of this, the deployment of ML models continues
to grow. This surge in deployment, while promising, introduces a pressing
concern: the vulnerability of these models to security flaws, both known and
yet to be uncovered. The potential consequences of such vulnerabilities extend
far beyond the limits of academia, with economic and societal implications.

Adversarial example attacks are one such attack, originally discovered by
Szegedy et al. [4] in 2014. They have shown the existence of easily craftable
imperceptible vulnerabilities in image classifiers. These vulnerabilities have
taken the form of perturbations, imperceptible to the human eye, that can
significantly alter a model’s prediction on an otherwise benign input. The low
requirements for mounting adversarial attacks make it a prime vulnerability.
Prior work in the domain of adversarial examples suggests a notable advantage
for attackers, with the majority of attacks capable of, to some degree,
impeding model performance. Among the existing defenses, few techniques have
demonstrated a degree of robustness when it comes to safeguarding machine
learning models, for example, adversarial training methods [5] and ensemble
approaches [6] (and their combination). However, even in their strongest forms
and under strict threat models that greatly limit the attacker’s capabilities,
current defenses still fail to provide adequate protection. To remedy this, a
branch of research on provably robust defenses emerged [7]. However, in
practice, this is not yet a feasible approach since it increases the
computational cost of model inference by several orders of magnitude. Since
provable defenses are impractical, a majority of the field has focused on
improving the quality of empirical defenses. Nevertheless, the absence of a
reliable evaluation standard or assessment method for overall performance has
resulted in an ongoing competitive cycle between attackers and defenders based
on empirical experiments. A complete theoretical framework is necessary to
create an evaluation standard and shift this competitive cycle into a
manageable challenge for adversarial example research.

In our work, we focus on the image classification domain and aim to address a
significant challenge: the current lack of emphasis on rigorous assessment of
adversary knowledge in threat models. Adversary knowledge is one of the key
components of a well-defined threat model and has long been overlooked and
neglected by both attack and defense works. We study the current lack of
formalization of adversary knowledge in adversarial example research and how
it affects attack comparability and performance. Defending against attackers
with ill-defined capabilities places an additional burden on the defender.
Whereas attacking with ill-defined knowledge hinders reproducibility and
comparability. We develop this formalization as a theoretical framework to
model various adversarial example attacks. We conduct a thorough review of
recent attacks in the image classification domain, offering an up-to-date
overview of the current landscape of adversaries in this specific context [8,
9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28,
29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66,
67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85,
86, 87, 88, 89, 90] . This formalization allows us to define and categorize
attacks and their associated threat models, in turn, laying a foundation for
defenses. Additionally, combined with our survey of attacks, we measure the
effect of the information available to an attacker on an attack’s performance.
We compare various threat models and confirm, with falsifiable results, the
generally held belief that certain categories of information, like information
related to the attacked classification model, have a disproportionate effect
on attack performance. However, we also find that a lack of information in
that category, in the case of transferable attacks, can be compensated by the
use of additional information such as information related to the training data
or the training process, to yield attacks that are almost as potent.

##  2 Definitions

We let $\mathcal{I}\subset\mathbb{R}^{*}$ be the set of all possible inputs,
$\mathcal{L}\subset\mathbb{R}^{*}\cup\\{\bot\\}$ the set of all possible
labels, and the ground-truth function $gt:\mathcal{I}\rightarrow\mathcal{L}$
as the function that assigns its true label to an input sample.

######  Definition 1 (Adversarial Example).

An input sample $i\in\mathcal{I}$ and its associated label
$gt(i)\in\mathcal{L}$ is said to be adversarial if it was specifically crafted
to successfully trigger a learned model to output an incorrect answer.

######  Definition 2 (Grounded Adversarial Example).

An input sample $i\in\mathcal{I}$ and its associated label
$gt(i)\in\mathcal{L}$ is said to be a grounded adversarial example if it was
created using a benign sample $x\in\mathcal{I}$ and its associated label
$gt(x)\in\mathcal{L}$.

######  Definition 3 (Targeted/Untargeted Adversarial Example).

An input sample $i\in\mathcal{I}$ and its associated label
$gt(i)\in\mathcal{L}$ is said to be a targeted adversarial example if for a
fixed chosen label $c\in\mathcal{L},c\neq gt(i)$, it triggers a learned model
to output label $c$. However, if it triggers a learned model to output a label
$l\neq gt(i),l\in\mathcal{L}$, we say it is untargeted.

##  3 Previous Work

We formalize adversarial example attacks with an emphasis on the information
and the capabilities the adversary has access to perform their attack. There
has been a lack of a proper theoretical framework that is based on proper
security principles. In turn, this has been reflected in work whose impact is
limited due to the inherent flaws in their security assumptions.

Previous systematization of knowledge works [91, 92] have looked at the
adversarial example problem from various angles. Papernot et al. [91] took a
holistic approach and proposed a thorough overview of the possible threat
models as well as the training and inference process in an adversarial
setting. They emphasized a clear distinction between the different domains
(physical vs. digital) while describing the overall ML attack surface. This
was reiterated more recently by Lin et al. [59] where the authors present an
attack on the physical domain specifically. In contrast, Byun et al. [42] took
advantage of the properties of the physical domain by projecting adversarial
examples onto virtually 3D-rendered physical objects (like a mug) to improve
the transferability of their attack. Another approach was the one by Carlini &
Wagner [92]. In their paper, they present generating adversarial examples as a
constrained optimization problem. They first cover a set of existing attacks
at the time and from their formalization, they introduced three new attacks
adapted each for a specific norm distance metric ($L_{0}$-norm, $L_{1}$-norm,
and $L_{2}$-norm). However, we found that these works fail to properly cover
adversarial information and capabilities. This was not a significant problem
back in 2017 and 2018 when most of the literature focused on white-box
attacks, attacks where the adversary has immense knowledge and capabilities
and therefore covering the minutia is not as important.

Recently, however, the field has experienced a shift toward more realistic and
usable attacks. This is reflected in the emergence of unrestricted adversarial
examples [83, 93, 82] as well as threat models like the black-box and no-box
(transferable) threat models [8, 42, 51, 80, 89, 78]. A work that took the
first step toward rationalizing this shift is the work of Gilmer et al. [94].
One of the core messages of their work is that, at the time, research on
adversarial examples was predominantly focused on abstract, hypothetical
scenarios that lacked direct relevance to any specific security issues. This
still partially holds up to this day. They did mention the question of the
attacker’s knowledge but did not go into depth beyond the already common
concepts of white-box and black-box. Papers on defense techniques have also
not provided a comprehensive account of attackers’ capabilities and
constraints that would be applicable in real-world security scenarios.
However, they presented a wide-reaching list of salient situations that
represent at their core the action space of the attacker. The action space of
an attacker can be defined as the set of potential actions or strategies at
the disposal of an adversary. This encompasses the various methods and choices
available to the attacker when attempting to compromise a system, exploit
vulnerabilities, or achieve their malicious goals during an attack.

In this work, we propose to delve deeper into the knowledge that an attacker
has when performing an attack. This goes beyond the knowledge of the machine
learning system that they are trying to attack. It includes access to data and
computing resources, as well as knowledge of the code and hyperparameters used
to set up and train said machine learning system.

##  4 Formalization

A key element to differentiate between inference time attacks is the action
space that is available to the attacker. Real-world practical restrictions are
translated into mathematical constraints on the generated adversarial
examples. Gilmer et al. [94] first define the action space of the adversary
for adversarial example attacks. In their paper, they identify five salient
situations: indistinguishable perturbations, content-preserving perturbations,
non-suspicious input, content-constrained input and unconstrained input.
Historically, adversarial example attacks have been studied under the
indistinguishable and content-preserving perturbation settings, but recently
have been extended to the non-suspicious and content-constrained input
settings.

To propose a unified theoretical framework that can accommodate every salient
situation, we introduce an object that is already well-defined in the field of
cryptography: the distinguisher.

Our fundamental challenge is rooted in simulating human-like image perception,
a problem lacking a well-established mathematical framework. Consequently, we
must define our own setting, complete with its corresponding distinguisher.
The context we are operating within assumes human-like capabilities for the
distinguisher, for example, for indistinguishable perturbations, it is so
human reviewers would not be able to distinguish that the image is
adversarial. Therefore, we need to introduce a new class of distinguishers,
the human-like distinguisher that we define as a distinguisher whose
capabilities match that of a human or
any human-made real-world system. Current existing metrics ($l_{p}$-norm,
similarity, quality…) all fall under this category and are used as an
approximation of human-like capabilities for their salient situation.

We can now represent a salient situation as a distinguisher
$D:\mathcal{I}\rightarrow\\{0,1\\}$ who attempts to output 0 on benign samples
and 1 on adversarial samples, it is meant to capture the detection
capabilities in place in the system under attack. We can use this
formalization of salient situations and provide a proper formalization of the
notion of indistinguishability for any setting, not just the indistinguishable
perturbation setting.

######  Definition 4 (Indistinguishability).

Assuming a salient situation with distinguisher
$D:\mathcal{I}\rightarrow\\{0,1\\}$, let $A,B\subset\mathcal{I}$ where $A$ is
the set of all adversarial examples and $B$ is the set of all benign samples,
we say an input $x\in\mathcal{I}$ is indistinguishable if we have:

$\displaystyle\zeta(n)=|\underset{x\leftarrow
A}{\textnormal{Pr}}[D(x)=1]-\underset{x\leftarrow
B}{\textnormal{Pr}}[D(x)=1]|=0$ (1)

However, this definition is too restrictive as in practice $n$ (and therefore
the number of possible adversarial examples) is too large to obtain
$\zeta(n)=0$ and should only serve as a target for the community to strive for
rather than a strict requirement. Instead, for practical purposes, we can use
a loosened notion of _stealth_ to bind adversarial examples.

######  Definition 5 (Stealth).

If $\zeta(n)$ (from Equation 4) is $\zeta(n)\in\mathcal{O}(\frac{log(n)}{n})$,
$|A\cup B|=n$, then we say that the associated input $x\in\mathcal{I}$ is
stealthy.

###  4.1 Categorization

####  4.1.1 Information Extraction Oracles

To formalize the information an adversary has access to when performing an
attack, we need a generic representation of information. Since precisely
quantifying and defining information is quite difficult, we avoid it and
instead present a generic method for representing information acquired by the
adversary. To that effect, we introduce Information Extraction Oracles (IEOs).
IEOs serve as a translation tool that allows an attacker to properly defined
information in a threat model. They can be used to represent the access to
information available to the attacker (through any of its attack
capabilities). We now introduce an improved method for presenting threat
models in a standardized and formalized fashion.

######  Definition 6 (Information Extraction Oracle).

An Information Extraction Oracle
$\mathcal{O}:\\{0,1\\}^{*}\rightarrow\\{0,1\\}^{*}$ is a stateful oracle
machine that can take in a query and outputs information in a binary format.

We define the set of all information extraction oracles as $\mathbb{O}$.
Instead of describing attacker capabilities with vague terminology, we can now
use information extraction oracles to precisely and accurately capture the
attacker’s knowledge and capabilities. As mentioned in Definition 6,
information extraction oracles are stateful. The adversary can extract the
oracle’s state using the $State$ function.

######  Definition 7.

We define the function $State:\mathbb{O}\rightarrow\mathbb{R}^{*}$ that takes
in as input an Information Extraction Oracle and outputs its current state as
a set of real numbers. If the input oracle does not have a state, it returns
$\emptyset$.

To accurately represent relationships between various threat models (white-box
vs. black-box for example), we need to introduce a relational structure to
information extraction oracles. To do so, we define the domination $\sqsubset$
operator on IEOs.

We define the different symbols we use:

  * •

$\\{\\}$ represents an unordered set

  * •

$[]$ represents an ordered set

  * •

$\cdot$ represents what we call an element, which we define as anything that
is not a set.

######  Definition 8 (Information Extraction Oracle domination operator).

We define the operator $\sqsubset$ for information extraction oracles over
their outputs in the following way:

Let $\mathcal{O}_{1}(a)=x$ and $\mathcal{O}_{2}(a)=y$. We have three base
cases:

  1. 1.

    1. (a)

$x$ and $y$ are both sets. Then $\mathcal{O}_{2}\sqsubset\mathcal{O}_{1}$ if
$\forall a\in\mathcal{I}$, $x\subset y$.

    2. (b)

$x$ is an element and $y$ is an unordered set. Then
$\mathcal{O}_{2}\sqsubset\mathcal{O}_{1}$ if $\forall a\in\mathcal{I}$, $x\in
y$.

    3. (c)

$x$ is either an unordered set or an element and $y$ is an element. Then
$\mathcal{O}_{2}\sqsubset\mathcal{O}_{1}$ if there exists a probabilistic
polynomial-time (PPT) function $f$ s.t. $\forall a\in\mathcal{I}$, $f(x)=y$.

Using these base cases, we can expand to the three ordered set cases.

  2. 2.

    1. (a)

$x=[x_{1},\dots,x_{n}]$ is an ordered set and $y$ is either an unordered set
or an element ($n$ is a positive integer). Then
$\mathcal{O}_{2}\sqsubset\mathcal{O}_{1}$ if $(y\sqsubset
x_{1})\vee\dots\vee(y\sqsubset x_{n})$.

    2. (b)

$x$ is an unordered set or an element and $y=[y_{1},\dots,y_{k}]$ is an
ordered set ($k$ is a positive integer). Then
$\mathcal{O}_{2}\sqsubset\mathcal{O}_{1}$ if $(y_{1}\sqsubset
x)\wedge\dots\wedge(y_{k}\sqsubset x)$.

    3. (c)

$x=[x_{1},\dots,x_{n}]$ is an ordered set and $y=[y_{1},\dots,y_{k}]$ is an
ordered set ($n$ and $k$ are positive integers). Then
$\mathcal{O}_{2}\sqsubset\mathcal{O}_{1}$ if $\forall
i\in\\{1,\dots,k\\},\exists j\in\\{1,\dots,n\\}$ s.t. $y_{i}\sqsubset x_{j}$.

We let $\not\sqsubset$ be the not operator for $\sqsubset$. Meaning when not
$A\sqsubset B$, then $A\not\sqsubset B$.

When an attacker has access to multiple oracles
$\mathcal{O}_{A},\mathcal{O}_{B}$ where a domination relation cannot be
established, we describe the resulting combined oracle in Definition 9.

######  Definition 9 (Information Extraction Oracle Combination).

Given two IEOs $\mathcal{O}_{A},\mathcal{O}_{B}$ for which neither
$\mathcal{O}_{A}\sqsubset\mathcal{O}_{B}$ nor
$\mathcal{O}_{B}\sqsubset\mathcal{O}_{A}$ is true. Then we define the combined
oracle $\mathcal{O}_{A\&B}$ as follows:
$\mathcal{O}_{A\&B}(x)=[\mathcal{O}_{A}(x),\mathcal{O}_{B}(x)]$

####  4.1.2 Information Categories

An IEO provides an interface for the attacker to both the knowledge of the
defender (for example, information about the target model) and any additional
external knowledge possessed by the attacker (for example, an additional
dataset that is disjoint from the defender’s training set but is drawn from
the same distribution). We can then categorize the different aspects of a
threat model into different IEOs that allow the attacker access to the
associated information. We used all the threat models we observed in the
literature to build this categorization to have a complete representation of
the threat models in the field and a meaningful categorization.

We can classify the information involved in an adversarial example attack into
three distinct types: information used by the defender, generated by the
defender (for example model parameters $\theta_{M}$) and any information that
is publicly accessible (data, pre-trained models …).

Within those classes, we identified the salient categories that combined can
be used to reconstruct any threat model: model, data, training and defense
information.

####  4.1.3 Information Hasse Diagrams

Now that we have defined broad information categories for attackers, we can
introduce a structure that will allow for meaningful comparisons between sets
of assumptions within each category.

We make use of the $\sqsubset$ operator that we defined in Definition 8 to
build Hasse diagrams. These diagrams are usually used to visually represent
sets ordered by inclusion. We design our Hasse Diagrams to visually represent
our oracles ordered by $\sqsubset$. For each category, we provide the
associated Hasse diagram. In order theory, a Hasse diagram is supposed to
represent finite partially ordered sets, however, we extend it to infinite
partially ordered sets by allowing formulaic set descriptions (which can
expand to generate infinite but partially ordered sets). For simplicity’s
sake, when we describe information extraction oracles, we assume that the
inputs they are queried with are in $\mathcal{I}$ (in the image domain),
otherwise, they return an empty ordered set $[]$. We summarize all the
behavior of the information extraction oracles we define in Table 1.

##### Model information Hasse diagram:

For model-related information, it can either be static or query-based. For
static information, we have:

  * •

Model parameters $\theta_{M}$ (white-box). Let $\mathcal{O}_{M}$ be its
information extraction oracle.

  * •

Model architecture: either the exact model architecture or a set of possible
architectures.

    * –

Exact model architecture $\phi_{M}$. Let $\mathcal{O}_{A}$ be its information
extraction oracle.

    * –

Set of possible architectures: Let $\mathcal{O}_{SPA}$ be its information
extraction oracle.

For query-based information, we have:

  * •

Query-access to model scores (model logit probabilities or values on queried
inputs). Let $\mathcal{O}_{S}$ be its information extraction oracle.

  * •

Query-access to model labels (predicted class on queried inputs). Let
$\mathcal{O}_{L}$ be its information extraction oracle.

Oracle | Output | State
---|---|---
$\mathcal{O}_{M}$ | $[\theta_{M},x]$ | $[]$
$\mathcal{O}_{S}$ | $[M(x),x]$ |  | number of queries
---
left $[k]$
$\mathcal{O}_{L}$ | $[argmax(M(x)),x]$ |  | number of queries
---
left $[k]$
$\mathcal{O}_{A}$ | $[\phi_{M},x]$ | $[]$
$\mathcal{O}_{SPA}$ |  |  $[\\{\phi_{0},\phi_{1},\dots,\phi_{k}\\},x]$,
---
$k\in\mathbb{Z}^{+}$ and for some
$i\sim\mathcal{U}(0,k)$, $\phi_{i}=\phi_{M}$
$[]$
$\mathcal{O}_{\mathcal{D}}$ | $[\mathcal{D},x]$ | $[]$
$\mathcal{O}_{\mathcal{D^{\prime}}}$ | $[\mathcal{D^{\prime}},x]$ | $[]$
$\mathcal{O}_{\mathcal{E}}$ | $[\mathcal{E},x]$ | $[]$
$\mathcal{O}_{Train}$ | $[Train,x]$ | $[]$
$\mathcal{O}_{T_{i}}$ | $[T_{i},x]$ | $[]$
$\mathcal{O}_{FA}$ | $[\rho,\varrho,x]$ | $[]$
$\mathcal{O}_{PA}$ |  |  $[\rho,\\{\varrho_{1},\dots,\varrho_{k}\\},x]$,
---
$k\in\mathbb{Z}^{+}$ and for some
$i\sim\mathcal{U}(0,k)$, $\varrho_{i}=\varrho$
$[]$
$\mathcal{O}_{SPD}$ |  | $[\\{\rho_{0},\dots,\rho_{k}\\},\\{\varrho_{1},\dots,\varrho_{k}\\},x]$
---
$k\in\mathbb{Z}^{+}$ and for some
$i\sim\mathcal{U}(0,k)$, $\rho_{i}=\rho$ and $\varrho_{i}=\rho$
$[]$

Table 1: Oracle Table

We can then order their associated information extraction oracles in a Hasse
Diagram in Figure 1.

Figure 1: Model Oracle Hasse Diagram

######  Theorem 1.

Figure 1 holds under the $\sqsubset$ ordering.

We refer to Appendix A for all of the proofs.

##### Data Information Hasse Diagram

We identify three sets of data. The training data
$\mathcal{D}\subset\mathcal{I}\times\mathcal{L}$ that was used to train the
target model $M$ and its information extraction oracle
$\mathcal{O}_{\mathcal{D}}$. Any other data samples
$\mathcal{D}^{\prime}\subset\mathcal{I}\times\mathcal{L}$ (and it’s IEO
$\mathcal{O}_{\mathcal{D^{\prime}}}$) that were drawn from the same
distribution as $\mathcal{D}$ but were not in the training data. Finally, any
other data samples $\mathcal{E}\subset\mathcal{I}\times\mathcal{L}$ (and it’s
IEO $\mathcal{O}_{\mathcal{E}}$) that were not drawn from the same
distribution as $\mathcal{D}$ and $\mathcal{D^{\prime}}$. In this case,
creating the Hasse diagram is relatively straightforward because we only need
to contemplate all feasible power set combinations of oracles using definition
9.

Figure 2: Data Oracle Hasse Diagram

######  Theorem 2.

Figure 2 holds under the $\sqsubset$ ordering.

##### Training Information Hasse Diagram

We describe a systematized method for identifying a training function’s known
elements. First, we must define exactly what we mean by training algorithm or
function.

######  Definition 10 (Train function).

We define a training function $Train:\\{0,1\\}^{*}\rightarrow\mathcal{M}$ as
the function representing the algorithm used by the defender to generate the
target model $M$.

The $Train$ function takes in the algorithm’s hyper-parameters as well as any
other information needed to train the model. We can articulate known
information about the training algorithm as one of the following training
information sets:
$T=\\{Train^{\prime}|g_{Train}(Train^{\prime})=True\\}$ where $g_{Train}$ is a
function that returns $True$ if a particular condition in the training
algorithm with respect to the original $Train$ function is satisfied (e.g. the
training algorithm uses cross-entropy loss) and $False$ otherwise. We can then
create two types of information extraction oracles. This is the information
that can be used (combined with data) to train/assume access to pre-trained
models. The first one is $\mathcal{O}_{Train}$, where $Train$ is the original
$Train$ function used by the defender.

The other is when the attacker has access to one of the sets $T$. There are
infinitely many of them, therefore we will just define a generic oracle for
them: $\mathcal{O}_{T_{i}}$ for any integer $i$ where $T_{i}$ is any one of
the sets $T_{i}=\\{Train^{\prime}|g^{i}_{Train}(Train^{\prime})=True\\}$ we
previously described.

The oracles generated from this generic oracle can then be ordered using
$\sqsubset$ following the usual rules defined in Definition 8.

Figure 3: Train Oracle Hasse Diagram

######  Theorem 3.

Figure 3 holds under the $\sqsubset$ ordering.

##### Defense Information Hasse Diagram

Defense information is any defense mechanisms put in place by the defender on
top of the distinguisher $D$. We can partition it as follows:

  * •

Full awareness of the defense and its parameters (similar to white-box for
model information). We represent this as an algorithm $\rho$ and its
parameters $\varrho$. Let $\mathcal{O}_{FA}$ be its information extraction
oracle.

  * •

Partial awareness of the defense and its parameters: knowledge of the defense
$\rho$ but not the specific parameters of the instance. Let $\mathcal{O}_{PA}$
be its information extraction oracle.

  * •

A set of potential defenses that can be obtained by having partial insider
information. Let $\mathcal{O}_{SPD}$ be its information extraction oracle.

Figure 4: Defense Oracle Hasse Diagram

######  Theorem 4.

Figure 4 holds under the $\sqsubset$ ordering.

###  4.2 Adversarial Example Game

Gilmer et al. [94] were also among the first to present the problem of
adversarial examples as a security game. This contrasts with Carlini & Wagner
[92] and their optimization-based representation. This emphasized the security
nature of the problem. They also address in their paper the notion of game
sequence. It is the player order and whether the game is repeated. This was
not particularly addressed by Carlini & Wagner [92] but is key to a practical
understanding of the problem. It relates the problem of adversarial examples
to those of standard cryptographic security games. Later, Bose et al. [88]
expanded upon the game nature of adversarial examples while relating it to its
underlying min-max optimization nature. They use this expansion to develop an
attack in what they call the NoBox (also called transferable) setting and to
bridge the gap between the theory and the empirical in more demanding and
realistic threat models. We present an updated, concise and comprehensive
security game for adversarial examples that conforms with the security and
machine learning communities.

####  4.2.1 Definitions

First, we define the following: $\varnothing$ is the termination symbol.
$\mathcal{O}^{a,b}_{X}$ is an input information extraction oracle about the
data granted to the adversary for an attack. We superscript
$\mathcal{O}^{a,b}_{X}$ with a bit $a$ that represents whether the attack is
grounded or not (0 for grounded and 1 for not-grounded). We also superscript
it with a bit $b\in\\{0,1\\}$ where if the attack is untargeted then $b=0$ and
if it is targeted then $b=1$. It returns the start input sample
$x\in\mathcal{I}$ and its associated ground-truth label $y\in\mathcal{L}$ if
the attack is grounded. In the case of a targeted attack, it also returns the
target label $y_{t}\in\mathcal{L}$. $\mathcal{O}_{Dist}$ is an information
extraction oracle that returns a distinguisher $D^{\prime}$ that the attacker
can utilize to verify that the generated adversarial examples are within the
restrictions of the salient situation. By default, $D^{\prime}=D$ where $D$ is
the actual distinguisher used by the defender. However, in situations where
only partial information is known about the distinguisher $D$ or the
associated salient situation, $D^{\prime}$ can be used to capture the
attacker’s knowledge of $D$.

We can now define the key functions that we will use in our security game:

######  Definition 11 ($AdvGen$).

$AdvGen:\mathbb{O}\rightarrow\mathcal{I}\times\mathcal{L}$.
$AdvGen(O)$ on input set of information oracles $O\subset\mathbb{O}$
(including the input information oracle $\mathcal{O}^{a,b}_{X}$), outputs the
following:

$\displaystyle AdvGen(O)=[x^{\prime},y^{\prime}]$ (2)

where $x^{\prime}\in\mathcal{I}$ is the adversarial example and
$y^{\prime}\in\mathcal{L}$ is its ground-truth label.

Additionally, we also define two other functions that are vital to our
security game: $Evaluate$ and $Classify$. As mentioned in Gilmer et al.’s work
[94], an attacker can have different goals, therefore we use the $Evaluate$
function to represent an attack’s success with respect to the attacker’s goal.
(Either a targeted or an untargeted (Definition 3) attack or any other goal an
attacker might have).

######  Definition 12 ($Classify$).

$Classify(M,x)$ on input model $M$ and input sample $x\in\mathcal{I}$ returns
the predicted label of $x$ using the classifier $M$ while executing any
inference-time pre-processing or defense mechanisms the defender has in place.

######  Definition 13 ($Evaluate$).

We have two evaluation functions, one for untargeted attacks $Evaluate_{0}$
and one for targeted attacks $Evaluate_{1}$ (same as $\mathcal{O}^{a,0}_{X}$
and $\mathcal{O}^{a,1}_{X}$). They are defined as the following:

$\displaystyle Evaluate_{0}(y,r)=\mathbb{I}[r\neq y]$ (3)

and

$\displaystyle Evaluate_{1}(y_{t},r)=\mathbb{I}[r=y_{t}]$ (4)

Where $y$ is the ground-truth label, $r$ is the predicted label and $y_{t}$ is
the target label for a targeted attack.

Now that we have all the components of the game ready, we can define it in
Figure 4.2.1. This game has two participants, the attacker and the defender.

[linenumbering, space=keep]  Attacker Defender

[] (O, AdvGen, Evaluate_b) (D, Train, Classify)

[] M ←Train(⋅)

l ←{}

i ←0

O_Dist ←O[⋅]

D’ ←O_Dist

x’, d ←∅, 1

x_i’ ←x’

While  (d=1) ∧(x_i’ ∉l)

Do
x_i’, y_i’ ←AdvGen(O)

l ←l ∪{x_i’}

d ←D’(x_i’, y_i’, O^a,b_X)

i ←i + 1

Done

x’ ←x_i’

If  d=1  Then Return  0

Else  *[1.4cm]x’

d’ ←D(x’)

If  d’ = 1  Then  r ←∅

Else  r ←Classify(M, x’)

*[1.4cm]r

If  r = ∅ Then Return  0

If  b=0  Then
Return  Evaluate_0(y, r)

Else

⋅, ⋅, y_t ←O^a,1_X

Return  Evaluate_1(y_t, r)

[space=keep] ← is the assignment operator.=  is the equality comparison
operator.  ⋅ is used as a placeholder for an index.

Figure 5: Adversarial Example Game Diagram Algorithm

####  4.2.2 Measuring Success

First, we have to define success before being able to measure it. We define
success as our game returning 1 and failure as our game returning 0. This
definition guarantees that the attacker succeeds if and only if he manages to
remain undetected by the defender while completing his objective (usually
misclassification). Let
$G(O,D^{\prime},Evaluate_{b},D,Train,Classify)\rightarrow\\{0,1\\}$ be an
instance of the game. What is currently used to measure success is the
following:

######  Definition 14 (Expected success rate).

We define the expected success rate (ESR) as:

$\displaystyle\xi_{G}=\mathbb{E}[G(O,D^{\prime},Evaluate_{b},D,Train,Classify)]$
(5)

It is also know as Attack Success Rate (ASR). However, it can be deceiving as
it rewards attacking already poorly performing models and, without additional
information, can misrepresent an attack’s performance. Let
$G_{b}(\mathcal{O}_{X},Evaluate_{b},D,Train,Classify)$ be the version of our
game where instead of being given adversarial examples, the defender is
provided with benign samples. We use the expected success rate on this game
$\Upsilon_{G_{b}}$ as a lower-bound for attack performance. For simplicity’s
sake, we write $G(O,D^{\prime},Evaluate_{b},D,Train,Classify)$ as $G$ and
$G_{b}(\mathcal{O}_{X},Evaluate_{b},D,Train,Classify)$ as $G_{b}$. We instead
propose the following score as an alternative performance measurement.

######  Definition 15 (Relative performance score).

We define the relative performance score as:

$\displaystyle\Upsilon_{G,G_{b}}=\xi_{G}^{2}-\xi_{G_{b}}^{2}$ (6)

This score is contained between $-1$ and $1$. This relative score alleviates
the aforementioned problem. The score is negative when the attack performs
worse than just using benign samples and is positive otherwise. We believe it
captures the various trade-offs when mounting attacks and defenses more fairly
than the expected success rate or even a shifted expected success rate by the
lower-bound ($G_{b}$). Figure 6 provides a visual understanding of the
behavior of this score when varying both $\xi_{G}$ and $\xi_{G_{b}}$.

Figure 6: Relative Performance Score for various benign and adversary ESRs.

####  4.2.3 Example: PGD

We will now showcase an application of our formalization to an existing well-
known and well-performing attack: Projected Gradient Descent (PGD) [5]. This
should function as a template for the practical application of our
formalization and the associated game. As a reminder, the multistep PGD attack
can be defined as the following:

$\displaystyle x^{t+1}=\Pi_{x+\mathcal{S}}(x^{t}+\alpha\text{
sgn}(\nabla_{x}L(\theta,x,y)))$ (7)

where $x$ is the original benign sample with its label $y$, $x^{t}$ is the
input at step $t$ of the attack, $\mathcal{S}\subseteq\mathbb{R}^{d}$ is the
set of allowed perturbations ($d$ is the input dimension), $\theta$ are the
model parameters, $L$ is a loss function, sgn is the sign function (-1 if
input is negative, 1 otherwise) and $\alpha$ is the step size. In their paper,
they consider the following adversaries:

  1. 1.

"White-box attacks with PGD for a different number of iterations and restarts,
denoted by source $A$".

  2. 2.

"White-box attacks with PGD using the Carlini-Wagner (CW) loss function
(directly optimizing the difference between correct and incorrect logits)".

  3. 3.

"Black-box attacks from an independently trained copy of the network, denoted
$A^{\prime}$".

  4. 4.

"Black-box attacks from a version of the same network trained only on natural
examples, denoted $A_{nat}$".

  5. 5.

"Black-box attacks from a different convolution architecture, denoted B".

Table 2 summarizes the adversary knowledge used in each version.

Attack | Model | Data | Train | Defense
---|---|---|---|---
1 | Parameters | $\emptyset$ |  | Training
---
Information
(loss function)
$\emptyset$
2 | Parameters | $\emptyset$ | $\emptyset$ |  | Full
---
Awareness
3 |  | Scores &
---
Architecture
| Training
---
Data
| Original
---
Function
| Full
---
Awareness
4 |  | Scores &
---
Architecture
| Training
---
Data
| Original
---
Function
$\emptyset$
5 |  | Possible
---
Architectures
| Training
---
Data
| Original
---
Function
| Full
---
Awareness

Table 2: PGD attacks knowledge table

Our knowledge tables allow for a clear and concise representation of the
adversary’s knowledge of the defender’s information. We will then describe the
components of the game for the first adversary (Attack #1). This is meant as a
template to showcase how one would describe their attack using our game. To do
this, we need to define the following components: $O$, $D$, $D^{\prime}$,
$Evaluate_{b}$, $Classify$, $Train$, and $AdvGen$.

As shown in Table 2, we have the associated oracles used by the attacker in
Attack #1 that are the following:
$O=\\{\mathcal{O}_{M},\mathcal{O}_{T_{1}},\mathcal{O}_{FA},\mathcal{O}^{0,0}_{X},\mathcal{O}_{Dist}\\}$.
For the training information (loss function) of Attack # 1, we construct
$\mathcal{O}_{T_{1}}$, we let
$T_{1}=\\{Train^{\prime}|g^{1}_{Train}(Train^{\prime})=True\\}$ where
$g^{1}_{Train}$ is the function that returns True when $Train^{\prime}$ uses
the same loss function as $Train$.

$D$ varies depending on the dataset used, for example, on MNIST [95], they use
an indistinguishable-perturbation distinguisher with the $l_{\infty}$-norm
metric and a maximum allowed perturbation of $\epsilon=0.3$ (for pixel values
between 0 and 1). However, for CIFAR-10 [96], they also use an
indistinguishable-perturbation distinguisher with the $l_{\infty}$-norm
metric, but they use a maximum allowed perturbation of $\epsilon=8$ (for RGB
pixel values between 0 and 255). They also explore a variant of their attack
in the case of an $l_{2}$-bounded adversary but to keep this concise we do not
analyze it. $\mathcal{O}_{Disc}(x)=D^{\prime}=D$ for any $x\in\\{0,1\\}$ as it
is assumed that the attacker knows exactly the setting it is in.

Their defense is part of the training process (adversarial training),
therefore $Classify(M,x)=M(x)$. The adversary’s goal is to modify the attacked
model’s accuracy, meaning it is an untargeted attack. Hence, $b=0$ and
$Evaluate_{0}(y,r)=\mathbb{I}[(r\neq y)]$. $Train$ (as defined) is the
algorithm/code used to train the attacked model, due to the sheer complexity
of the algorithm, we will instead point out that they link their code in their
paper [5]. We can summarize it as a standard Stochastic Gradient Descent (SGD)
based model training algorithm that incorporates adversarial training as a
defense. Finally, for the adversarial example generation, they use a grounded
process, hence $a=0$ and $AdvGen$ can be defined as the algorithm described in
Figure 4.2.3 (they have three hyperparameters $t$, $\alpha$, $\epsilon$).

[linenumbering, space=keep]  $AdvGen(O)$ t, α, ϵ
[] O_M, O_Train, O_FA, O^0, 0_X, O_Dist ←O
i ←0
x, y ←O^0, 0_X(0)
θ_M ←O_M(0)
⋯, L, ⋯←O_Train(0)
x^i ←x
While  i < t
Do
x^i+1 ←Π_x + S(x^i + α sgn(∇_xL(θ_M,x,y)))
x^i+1 ←clip(x^i+1, x - ϵ, x + ϵ)
x^i+1 ←clip(x^i+1, 0, 1)
i ←i + 1
Done
Return  x^i

Figure 7: PGD Attack Algorithm

##  5 Survey Summary & Methodology

In Tables 3 and 15 in Appendix B, we summarize the findings of our survey by
reporting the information used by each attack in the surveyed papers as well
as the metric used to craft the adversarial examples. In the case where papers
evaluate their attacks under multiple scenarios, we split the attack into
multiple variants lettered (A), (B), and so on.

We focus on recent papers, papers published in 2022 or after to narrow the
field of the search and allow for an up-to-date perspective of the field of
image classification adversarial research. We gathered a total of eighty-three
papers [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44,
45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 82, 83,
84, 85, 86, 87, 88, 89, 90, 81], that we then narrowed to twenty using quality
and relevance [42, 51, 35, 37, 36, 61, 62, 34, 78, 79, 80, 8, 11, 12, 82, 83,
88, 89, 90, 81]. None of the papers gathered used any defense information (as
it is more relevant to adaptive attacks) so we exclude it from Table 3. Table
3 contains only the papers included in the evaluation (Section 6). For the
rest, we refer to Table 15 in Appendix B.

| Attack
---
(Targeted)
Model | Data | Train | Metric
| LGV (A)
---
[51]
| Possible
---
Architectures
| Training
---
Data
| Training
---
Function
l∞
| LGV (B)
---
[51]
Parameters | $\emptyset$ | $\emptyset$ | l∞
| Pixle
---
[34] (both)
Scores | $\emptyset$ | $\emptyset$ | l0
| MASSA
---
[62]
Labels | $\emptyset$ | $\emptyset$ | l2
| SSAH (A)
---
[61] (both)
Parameters |  | Same
---
Distribution
$\emptyset$ | l∞
| SSAH (B)
---
[61] (both)
| Possible
---
Architectures
| Other Data
---
| Training
---
Function
l∞
| BIA (A)
---
[8]
| Possible
---
Architectures
Other Data |  | Training
---
Function
l∞
| BIA (B)
---
[8]
| Possible
---
Architectures
| Training
---
Data
| Training
---
Function
l∞
| BIA (C)
---
[8]
Parameters |  | Training
---
Data
$\emptyset$ | l∞
| ACG
---
[79]
Parameters | $\emptyset$ |  | Loss
---
Function
l∞
| ATA (A)
---
[78]
| Possible
---
Architectures
| Training
---
Data
| Training
---
Function
l0
| ATA (B)
---
[78]
Parameters | $\emptyset$ | $\emptyset$ | l0
| AEG (A)
---
[88]
Architecture |  | Same
---
Distribution
| Loss
---
Function &
Optimizer
l∞
| AEG (B)
---
[88]
| Possible
---
Architectures
| Training
---
Data
| Loss
---
Function &
Optimizer
l∞
| $\text{A}^{3}$
---
[36]
Parameters |  | Same
---
Distribution
| Loss
---
Function
l∞

Table 3: Attack Summary Table. We provide the information used by the attacker
for each of the categories and the metric used to craft the adverarsarial
examples.

##  6 Evaluation

For this work, we focus our evaluation on indistinguishable perturbation
untargeted attacks that were evaluated on the ImageNet [97] and CIFAR-10 [96]
datasets, as these are the most studied datasets in the attacks we surveyed.
To compare attacks, we only compare evaluations within datasets and we only
work with results presented by the authors themselves in their papers. We
first, if possible, compare papers against models that are from the exact same
source (public provider). If that’s not possible, we compare models of the
same architecture while also looking at their benign accuracies. In the case
where a paper does not provide benign accuracies, we use the benign accuracy
from another paper but report the result in italic to show its unreliability.

###  6.1 Attacks on the ImageNet dataset

First, we tackle the most evaluated dataset that we study in our work, with
eleven variants of attacks evaluated against it: ImageNet [97]. We perform
four comparisons in total. We summarize the results of our comparisons on
ImageNet in Table 4. The only paper to evaluate their attack on defended
models is ACG [79] where they get an average ESR of 68.50% and an average
score of 0.314.

| Model
---
Information
| Data
---
Information
| Training
---
Information
| Attack
---
Variant
| Und.
---
ESR
| Und.
---
Score
Possible Architectures |  Training Data |  Training Function | LGV (A) | 72.4 | 0.497
BIA (B) | 93.48 | 0.811
ATA (A) | 79.65 | 0.524
| Training
---
Data &
Same
Distribution
| Training
---
Function
SSAH (B) | 19.14 | N/A
Labels | $\emptyset$ | $\emptyset$ | MASSA | 99.49 | 0.916
Scores | $\emptyset$ | $\emptyset$ | Pixle | 98.5 | 0.897
Parameters |  | Training
---
Data
$\emptyset$ | BIA (C) | 97.82 | 0.883
| Same
---
Distribution
$\emptyset$ | SSAH (A) | 95.56 | 0.915
$\emptyset$ | $\emptyset$ | LGV (B) | 97.1 | 0.885
ATA (B) | 100.0 | N/A
| Loss
---
Function
ACG | N/A | N/A

Table 4: ImageNet information and attack results. "Und." stands for
undefended.

The takeaways for ImageNet are the following:

  * •

Only query-based access to the model and no other information is sufficient to
reduce an undefended model’s accuracy to almost 0%.

  * •

Transferable settings can achieve extremely high ESRs (93.48% for BIA (B)),
which shows that, at least in the undefended case, transferable attacks that
use additional data and training information (usually to train surrogates) can
achieve ESRs close to query-based and white-box attacks. Unfortunately, since
none of the transferable attacks we studied evaluated defended models, we
cannot say if this extends to defended models.

  * •

ImageNet-defended models are mostly broken by white-box attacks. ACG obtains
an average ESR of 68.5% and an average score 0.314. While, defended models do
perform better than undefended ones, the accuracy loss is still significant.

####  6.1.1 LGV (A) & BIA (B) & ATA (A) & SSAH (B)

We start by comparing the different transferable attacks: LGV (A) [51], BIA
(B) [8], ATA (A) [78], SSAH (B) [61]. All these attacks attack undefended
papers. Some share architectures, however, not all of them provide benign
accuracies for the models they evaluate. We compile the results in Tables 5.
BIA (B), LGV(A), and SSAH (B) both use the $l_{\infty}$-norm but BIA (B) uses
a perturbation budget of 10/256 whereas SSAH (B) uses one of 8/256. LGV (A)
does not specify the budget used. ATA (A) uses the $l_{0}$-norm, where they
are only allowed to fully modify 1024 pixels. Unfortunately, this means that
the following comparison is disparate.

Model |  | Benign
---
ESR
(BIA)
| LGV (A)
---
ESR /
Score
| BIA (B)
---
ESR /
Score
| ATA (A)
---
ESR /
Score
| SSAH (B)
---
ESR /
Score
| ResNet-50
---
[98]
24.39 |  | N/A
---
N/A
| 95.56
---
0.854
| 87.31
---
0.703
| N/A
---
N/A
| VGG-16
---
[99]
N/A |  | N/A
---
N/A
| N/A
---
N/A
| 87.46
---
N/A
| 19.14
---
N/A
| DenseNet-121
---
[100]
25.78 |  | N/A
---
N/A
| 96.02
---
0.856
| 64.17
---
0.345
| N/A
---
N/A
| ResNet-152
---
[98]
22.66 |  | 89.6
---
0.751
| 94.15
---
0.835
| N/A
---
N/A
| N/A
---
N/A
| VGG-19
---
[99]
29.05 |  | 82.2
---
0.591
| 95.91
---
0.835
| N/A
---
N/A
| N/A
---
N/A
| Inception-v3
---
[101]
23.81 |  | 45.4
---
0.149
| 85.76
---
0.679
| N/A
---
N/A
| N/A
---
N/A
Average | 25.14 |  | 72.4
---
0.497
| 93.48
---
0.811
| 79.65
---
0.524
| 19.14
---
N/A
| Standard
---
deviation
2.46 |  | 23.67
---
0.312
| 4.38
---
0.075
| 13.4
---
0.253
| N/A
---
N/A

Table 5: ImageNet dataset indistinguishable Section 6.1.1 undefended results.

BIA (B) outperforms the other attacks. Due to the lack of results for SSAH (B)
and the fact that the few results are worse than the other attacks even though
SSAH (B) has more information, we ignore it in what follows. All the other
three attacks use the same information, there are three possible reasons for
this disparity in results:

  1. 1.

BIA (B) is a better algorithm for extracting information and building a potent
attack from it.

  2. 2.

LGV (A) and ATA (A) attacked undefended models that were somehow inherently
more robust (unlikely).

  3. 3.

The disparity of results is caused by the different perturbation
budgets/$l_{p}$-norms used.

We suspect a combination of 1. and 3. to be the reason explaining the
discrepancy in the results. Additionally, ATA’s attack is originally meant to
attack vision transformer models. Hence, it could partially explain a slightly
worse performance in an equitable evaluation setting.

####  6.1.2 LGV (B) & ATA (B) & ACG

This comparison is difficult to perform, as LGV (B) and ATA (B) both attack
different undefended models while not providing any benign accuracies. We can
use the benign accuracy of the BIA paper as a substitute for LGV (B) but not
for ATA (B). On the other hand, ACG [79] attacks only defended models,
therefore we cannot directly compare with the other attacks. We compile the
results in Tables 6 and 7.

Model |  | Benign
---
ESR
| LGV (B)
---
ESR / Score
| ATA (B)
---
ESR / Score
ResNet-50 [98] | 24.39 (BIA) |  97.1 / 0.883 | N/A / N/A
DeiT-T [102] | N/A | N/A / N/A |  100.0 / N/A
DeiT-S [102] | N/A | N/A / N/A |  100.0 / N/A
DeiT-B [102] | N/A | N/A / N/A |  100.0 / N/A

Table 6: ImageNet dataset indistinguishable undefended results. Model |  | Benign
---
ESR
| ACG
---
ESR
| ACG
---
Score
ResNet-50${}_{\text{robustness}}$ [103] | 37.44 | 69.58 | 0.344
ResNet-50${}_{\text{salman}}$ [104] | 35.98 | 64.7 | 0.289
ResNet-18${}_{\text{salman}}$ [104] | 47.08 | 74.34 | 0.331
WideResNet-50-2${}_{\text{salman}}$ [104] | 31.54 | 60.9 | 0.271
ResNet-50${}_{\text{FAST\\_AT}}$ [105] | 44.38 | 73.0 | 0.336
Average | 39.28 | 68.50 | 0.314
Standard deviation | 6.34 | 5.65 | 0.032
Table 7: ImageNet dataset indistinguishable ACG defended results.

####  6.1.3 BIA (C) & SSAH (A) & Section 6.1.2

Both BIA (C) and SSAH (A) evaluate only against undefended models, therefore
comparing against ACG is not possible. We can, however, compare SSAH (A) and
LGV (B) as they both evaluate against a ResNet-50 [98]. Since SSAH (A)
publishes their ResNet-50’s benign accuracy and LGV (B) does not, we also use
BIA’s benign accuracy from the previous evaluation. We compute LGV (B)’s score
using the mean of both benign accuracies to get closer to LGV (B)’s
ResNet-50’s expected benign accuracy and get a more reliable result. BIA (C)
only evaluates their white-box scenario against a VGG-16 and a DenseNet-169 in
the undefended setting. We compile the results in Table 8.

Model |  | Benign
---
ESR
| LGV (B)
---
ESR /
Score
| SSAH (A)
---
ESR /
Score
| BIA (C)
---
ESR /
Score
| ResNet-50
---
[98]
| 24.39 (BIA)
---
23.85 (SSAH)
| 97.1
---
0.885
| 98.56
---
0.915
| NA
---
NA
| VGG-16
---
[99]
29.86 (BIA) |  | NA
---
NA
| NA
---
NA
| 98.96
---
0.890
| DenseNet-169
---
[100]
24.25 (BIA) |  | NA
---
NA
| NA
---
NA
| 96.68
---
0.876
Average | 26.08 |  | 97.1
---
0.885
| 98.56
---
0.915
| 97.82
---
0.883
| Standard
---
deviation
3.28 |  | NA
---
NA
| NA
---
NA
| 1.61
---
0.010

Table 8: ImageNet dataset indistinguishable Section 6.1.3 undefended results.

####  6.1.4 MASSA & Pixle

We include the results from MASSA [62] and Pixle [34] to complete our
comparison. Since neither paper provides clean accuracies, we use the average
of the benign accuracies from BIA and SSAH to compute the score. This yields
the compiled results in Table 9. While MASSA evaluates its performance against
defended models, they are the sole paper to use the $l_{2}$-norm for its
distinguisher. Additionally, they do not provide clean accuracies for any of
their models. Pixle on the other hand does not attack any defended models. Our
results show that query-based access to a model is sufficient to render it
useless if the model is undefended.

Model |  | Benign
---
ESR
| MASSA
---
ESR /Score
| Pixle
---
ESR / Score
ResNet-50 [98] |  | 24.39 (BIA)
---
23.85 (SSAH)
99.4 / 0.930 |  98.0 / 0.902
VGG-16 [99] | 29.86 (BIA) |  99.58 / 0.902 |  99.0 / 0.891
Average | 26.99 |  99.49 / 0.916 |  98.5 / 0.897
Standard deviation | 4.06 | 0.12 / 0.019 | 0.7 / 0.008

Table 9: ImageNet dataset indistinguishable Section 6.1.4 undefended results.

###  6.2 Attacks on the CIFAR10 dataset

The second dataset we study is the CIFAR-10 dataset [96]. We can perform four
comparisons on a total of seven variants of attacks. We summarize our results
in Table 10.

| Model
---
Information
| Data
---
Information
| Training
---
Information
| Attack
---
Variant
| Und.
---
ESR /
Score
| Def.
---
ESR /
Score
Possible Architectures |  | Training
---
Data
| Loss &
---
Optimizer
| AEG
---
(B)
| 92.62
---
0.848
| 47.52
---
0.162
Other Data |  | Training
---
Function
| BIA
---
(A)
| 47.19
---
0.219
| NA
---
NA
Architecture |  | Same
---
Distribution
| Loss &
---
Optimizer
| AEG
---
(A)
| 87.0
---
N/A
| NA
---
NA
Scores | $\emptyset$ | $\emptyset$ | Pixle |  | 100.0
---
0.983
| NA
---
NA
Parameters |  Same Distribution | $\emptyset$ |  | SSAH
---
(A)
| 99.96
---
0.994
| 21.32
---
0.023
| Loss
---
Function
A3 |  | NA
---
NA
| 42.13
---
0.163
$\emptyset$ |  | Loss
---
Function
ACG |  | NA
---
NA
| 41.70
---
0.160

Table 10: CIFAR-10 information and attack results. "Und." stands for
undefended and "Def." stands for defended.

Unsurprisingly, we find that it requires little model information to get
effective attacks if the attacker has access to additional information like
data and training information. Additionally, we still observe a strong
transferable performance against defended models by AEG (B) where they obtain
a score (0.167) very similar to white-box attacks (0.163 and 0.160). While
this is still much worse than against undefended models, it would still in
practice severely decrease the usability of the target model.

####  6.2.1 BIA (A) & AEG (B)

BIA (A) [8] only attack a custom model, and therefore we treat their results
as unreliable. Their model achieves 6.22% benign ESR, while their attack
achieves an ESR of 47.19%, yielding a score of 0.219. AEG (B) [88] on the
other hand attack many common model architectures.

Since BIA (A) attacks a custom model, we have to perform an aggregate analysis
of AEG (B)’s performance on both undefended and defended models to be able to
compare. We report the performance on undefended / defended models in Table
11.

Model |  | Benign
---
ESR
| AEG (B)
---
ESR
| AEG (B)
---
Score
VGG-16 [99] | 11.2 | 93.8 | 0.867
ResNet-18 [98] | 13.1 | 97.3 | 0.930
WideResNet [106] | 6.8 | 85.2 | 0.721
DenseNet-121 [100] | 11.2 | 94.1 | 0.873
Inception-v3 [101] | 9.9 | 92.7 | 0.850
Average | 10.44 | 92.62 | 0.848
Standard deviation | 2.33 | 4.49 | 0.077
ResNet-18${}_{\text{ens3}}$ [6] | 16.8 | 52.2 | 0.244
WideResNet${}_{\text{ens3}}$ [6] | 12.8 | 49.9 | 0.232
DenseNet-121${}_{\text{ens3}}$ [6] | 21.5 | 41.4 | 0.125
Inception-v3${}_{\text{ens3}}$ [6] | 14.8 | 47.5 | 0.204
Madry-Adv [5] | 12.9 | 21.6 | 0.030
Average | 15.76 | 47.52 | 0.167
Standard deviation | 3.60 | 12.37 | 0.090
Table 11: CIFAR10 dataset indistinguishable AEG (B) results.

If we treat BIA (A)’s model as undefended, then AEG (B) significantly
outperforms BIA (A). This could either be due to the difference in the
information required by both attacks or that BIA (A) does not extract attack
performance as well out of the information it uses as AEG (B).

Unsurprisingly, attacking well-defended models in the transferable setting is
quite a difficult task. The results in Table 11 strongly imply that BIA (A)’s
model is indeed undefended as it achieves a benign ESR much closer to the
undefended models of Table 11.

####  6.2.2 A3 & SSAH (A)

We can directly compare both attacks as they both attack
WideResNet-34-10${}_{\text{TRADES}}$ [107]. We can confidently confirm that
these are likely to be the same models as both papers report the same benign
accuracy. We present the results in Table 12.

Model |  | Benign
---
ESR
| SSAH (A)
---
ESR /
Score
| A3
---
ESR /
Score
WRN-34-10${}_{\text{TRADES}}$ [107] | 15.08 |  | 21.32
---
0.023
| 46.99
---
0.198
Table 12: CIFAR10 dataset indistinguishable SSAH (A) and A3 defended results.
"WRN" stands for WideResNet.

We can see that although, as stated by the authors, A3 is an attack that
focuses both on ESR and runtime, it still significantly outperforms SSAH (A)
when attacking a well-defended model. It is doubtful that having access to the
training loss function is a strong enough difference between the two attacks
to justify the performance gap. It is more likely that SSAH (A) is inefficient
at extracting the information it has access to to mount a potent attack
against defended models.

####  6.2.3 ACG & A3

We again compare ACG [79] and A3 on defended models. On CIFAR10, ACG and A3
attack 11 models in common from RobustBench [108]. The results are summarized
in Table 13111* WideResNet-28-10${}_{\text{pretraining}}$ misreported as
WideResNet-34-10${}_{\text{pretraining}}$ in the A3 paper..

Model |  | Benign
---
ESR
| A3
---
ESR / Score
| ACG
---
ESR / Score
| WRN-34-10${}_{\text{TRADES}}$
---
[107]
15.08 | 46.99 / 0.198 |  47.18 / 0.200
| WRN-34-20${}_{\text{LBGAT}}$
---
[109]
11.3 |  46.54 / 0.204 | 46.23 / 0.201
| WRN-34-10${}_{\text{LBGAT}}$
---
[109]
11.78 |  47.24 / 0.209 | 46.9 / 0.206
| WRN-34-20${}_{\text{Overfitting}}$
---
[110]
14.66 |  46.67 / 0.196 | 45.69 / 0.187
WRN-70-16${}_{\text{Fixing}}$ [111] | 11.46 |  35.81 / 0.115 | 35.27 / 0.111
WRN-28-10${}_{\text{Fixing}}$ [111] | 12.67 |  39.34 /0.139 | 38.8 / 0.134
WRN-28-10${}_{\text{AWP}}$ [112] | 11.75 |  40.02 / 0.146 | 39.7 / 0.144
WRN-70-16${}_{\text{ULAT}}$ [113] | 8.9 |  34.22 / 0.109 | 33.7 / 0.106
WRN-70-16${}_{\text{ULAT}}$ [113] | 14.71 |  42.92 / 0.163 | 42.45 / 0.159
WRN-34-20${}_{\text{ULAT}}$ [113] | 14.36 |  43.24 / 0.166 | 42.86 / 0.163
WRN-28-10${}_{\text{ULAT}}$ [113] | 10.52 |  37.3 / 0.128 | 36.9 / 0.125
| WRN-28-10${}_{\text{pretraining}}$*
---
[114]
12.89 |  45.24 / 0.188 | 44.75 / 0.184
Average | 12.51 |  42.13 / 0.163 | 41.70 / 0.160
Standard deviation | 1.81 | 4.67 / 0.036 | 4.72 / 0.036

Table 13: CIFAR10 dataset indistinguishable A3 and ACG defended results. "WRN"
stands for WideResNet.

ACG outperforms both A3 and SSAH (A) on WideResNet-34-10${}_{\text{TRADES}}$
but otherwise loses to A3 on the other models and overall. We observe a small,
discrepancy between ACG and A3 in terms of performance. It could either be due
to the information used or the inherent algorithmic differences. Finally, we
are left with comparing all seven attacks against one another for our final
comparison.

####  6.2.4 AEG (A) & Pixle & Rest

Unfortunately, due to the amount of papers to compare, we reach the limits of
what we can do while keeping the comparisons fair. We go through the few
comparisons that we can still do. We first look into the straightforward
comparison: AEG (A) and AEG (B). They only evaluate AEG (A) against a
ResNet-18 [98]. They do not provide benign accuracies for AEG (A) which is
problematic. We therefore cannot compute a relative score to attempt a
comparison. We still report the ESR they report in our overall Table 10,
however, we italicize the result to indicate that it is unreliable.

Our other option would be to compare AEG (B) with Pixle, as AEG (B) provides
benign accuracies, and they share a target model architecture (ResNet-18).
Unfortunately, Pixle does not share its benign accuracy, so we deem their
results unreliable for comparison.

We get the following results in Table 14.

Model |  | Benign
---
ESR
| AEG (B)
---
ESR / Score
| Pixle
---
ESR / Score
ResNet-18 [98] | 13.1 | 97.3 / 0.930 |  100.0 / 0.983
Table 14: CIFAR10 dataset indistinguishable AEG (B) & Pixle undefended
results.

##  7 Takeaways

  1. 1.

Undefended models are broken. This is not a particularly new finding, but we
confirm and reinforce it. We show that across all datasets and threat models
studied, not once are undefended models robust to attacks. On CIFAR-10, the
undefended worst and best scores are, respectively, 0.219, and 0.994. Whereas,
the defended worst and best scores are, respectively, 0.023, and 0.163. Even
the best attack on defended models does not perform as well as the worst
attack on undefended models.

  2. 2.

The transferable setting might not be as difficult as previously thought. We
noticed that transferable attacks, when given additional information like
access to the training data and training information such as the training
function, can perform very well, almost on par with white-box attacks. For
CIFAR-10, the best tranferable attack is only within 7.3% ESR / 0.146 score of
the best white-box attack. This reinforces the notion that any information
given to the attacker is crucial.

  3. 3.

Information categories matter, but their effect on performance is not
straightforward. Information’s effect on performance is highly dependent on
the other information available. For example, in the white-box setting, data
and training information are nominal. But in the transferable setting, they
are crucial. In particular, training data information seems to be critical to
attack performance. This is best observed when looking at BIA (A) on the
CIFAR-10 dataset. They severely underperform other transferable attacks (47.2%
ESR / 0.22 score compared to 87.0% ESR and 92.6% ESR / 0.85 score for its
closest competitors AEG(A) & (B)). Not having access to the training data or
even data from the same distribution appears to severily affect attack
performance.

  4. 4.

Training information is understated but essential to transferable attacks.
Most of the transferable attacks whose results we have studied use the
training function. This is done to train surrogate models. However, this
assumption is not explicitly stated in the papers we survey and is instead
taken for granted. It goes against proper security practices as it omits what
our results clearly show to be critical information in the threat model
description.

  5. 5.

Need for better standard evaluation frameworks. There is a surprising lack of
standards to evaluate adversarial example attacks. This unnecessarily
complicates comparisons between existing works that are not directly related.
Only two papers (ACG and A3) used an existing evaluation framework
(Robustbench [108]).

  6. 6.

Evaluation against defended models. Too few papers evaluate their attack
against defended models (and they usually evaluate too few defended models as
well when they do). Only one paper evaluates its attack against defended
models on ImageNet and only four papers do so on CIFAR-10. Papers that only
evaluate undefended models bring little to the field as a whole as we have
seen it is not a meaningful measure of attack quality.

##  8 Conclusion

In this work, we presented a formalization to study adversary knowledge in the
context of adversarial example attacks on image classification models. We were
able to systematize a large body of existing work into our framework and are
confident in its ability to extend to future research in adversarial example
attacks and defenses.Our work can be used to enhance the reproducibility,
comparability and fairness of such future work as it allows a precise
description of one’s threat model and evaluations.

## Acknowledgments

We gratefully acknowledge the support of the Natural Sciences and Engineering
Research Council (NSERC) for grants RGPIN-2023-03244, and IRC-537591 and the
Royal Bank of Canada.

## References

  * Habehh1 and Gohel [2021] Hafsa Habehh1 and Suril Gohel.  Machine learning in healthcare.  _national library of medicine_ , 2021.  doi: 10.2174/1389202922666210705124359.
  * Gupta et al. [2021] Abhishek Gupta, Alagan Anpalagan, Ling Guan, and Ahmed Shaharyar Khwaja.  Deep learning for object detection and scene perception in self-driving cars: Survey, challenges, and open issues.  _Array_ , 10:100057, 2021.  ISSN 2590-0056.  doi: https://doi.org/10.1016/j.array.2021.100057.  URL https://www.sciencedirect.com/science/article/pii/S2590005621000059.
  * Ni [2022] Zhehan Ni.  Reframe the field of aerospace engineering via machine learning: Application and comparison.  _Journal of Physics: Conference Series_ , 2386(1):012031, dec 2022.  doi: 10.1088/1742-6596/2386/1/012031.  URL https://dx.doi.org/10.1088/1742-6596/2386/1/012031.
  * Szegedy et al. [2014] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks, 2014.
  * Madry et al. [2019] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks, 2019.
  * Tramèr et al. [2020] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel.  Ensemble adversarial training: Attacks and defenses, 2020.
  * Cohen et al. [2019] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter.  Certified adversarial robustness via randomized smoothing.  In _international conference on machine learning_ , pages 1310–1320. PMLR, 2019.
  * Zhang et al. [2022] Qilong Zhang, Xiaodan Li, Yuefeng Chen, Jingkuan Song, Lianli Gao, Yuan He, and Hui Xue.  Beyond imagenet attack: Towards crafting adversarial examples for black-box domains, 2022.
  * Liu et al. [2023a] Jiyuan Liu, Bingyi Lu, Mingkang Xiong, Tao Zhang, and Huilin Xiong.  Adversarial attack with raindrops.  (arXiv:2302.14267), July 2023a.  doi: 10.48550/arXiv.2302.14267.  URL http://arxiv.org/abs/2302.14267.  arXiv:2302.14267 [cs].
  * Shamshad et al. [2023] Fahad Shamshad, Muzammal Naseer, and Karthik Nandakumar.  Clip2protect: Protecting facial privacy using text-guided makeup via adversarial latent search.  page 20595–20605, 2023.  URL https://openaccess.thecvf.com/content/CVPR2023/html/Shamshad_CLIP2Protect_Protecting_Facial_Privacy_Using_Text-Guided_Makeup_via_Adversarial_Latent_CVPR_2023_paper.html.
  * Zhong et al. [2022] Yiqi Zhong, Xianming Liu, Deming Zhai, Junjun Jiang, and Xiangyang Ji.  Shadows can be dangerous: Stealthy and effective physical-world adversarial attack by natural phenomenon.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15345–15354, June 2022.
  * Zou et al. [2022] Junhua Zou, Yexin Duan, Boyu Li, Wu Zhang, Yu Pan, and Zhisong Pan.  Making adversarial examples more transferable and indistinguishable.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(33):3662–3670, June 2022.  ISSN 2374-3468.  doi: 10.1609/aaai.v36i3.20279.
  * Wei and Zhao [2023] Xingxing Wei and Shiji Zhao.  Boosting adversarial transferability with learnable patch-wise masks.  _IEEE Transactions on Multimedia_ , page 1–10, 2023.  ISSN 1941-0077.  doi: 10.1109/TMM.2023.3315550.
  * Wei et al. [2023] Zhipeng Wei, Jingjing Chen, Zuxuan Wu, and Yu-Gang Jiang.  Enhancing the self-universality for transferable targeted attacks.  page 12281–12290, 2023.  URL https://openaccess.thecvf.com/content/CVPR2023/html/Wei_Enhancing_the_Self-Universality_for_Transferable_Targeted_Attacks_CVPR_2023_paper.html.
  * Weng et al. [2023a] Juanjuan Weng, Zhiming Luo, Dazhen Lin, Shaozi Li, and Zhun Zhong.  Boosting adversarial transferability via fusing logits of top-1 decomposed feature.  (arXiv:2305.01361), July 2023a.  doi: 10.48550/arXiv.2305.01361.  URL http://arxiv.org/abs/2305.01361.  arXiv:2305.01361 [cs].
  * Zhang et al. [2023] Yaoyuan Zhang, Yu-an Tan, Haipeng Sun, Yuhang Zhao, Quanxing Zhang, and Yuanzhang Li.  Improving the invisibility of adversarial examples with perceptually adaptive perturbation.  _Information Sciences_ , 635:126–137, July 2023.  ISSN 0020-0255.  doi: 10.1016/j.ins.2023.03.139.
  * [17] Donald C. Wunsch Tao Wu, Tie Luo.  Black-box attack using adversarial examples: A new method of improving transferability.  URL https://www.worldscientific.com/doi/10.1142/S2811032322500059.
  * Weng et al. [2023b] Juanjuan Weng, Zhiming Luo, Shaozi Li, Nicu Sebe, and Zhun Zhong.  Logit margin matters: Improving transferable targeted adversarial attack by logit calibration.  _IEEE Transactions on Information Forensics and Security_ , 18:3561–3574, 2023b.  doi: 10.1109/TIFS.2023.3284649.
  * Deng et al. [2023] Zhengjie Deng, Wen Xiao, Xiyan Li, Shuqian He, and Yizhen Wang.  Enhancing the transferability of targeted attacks with adversarial perturbation transform.  _Electronics_ , 12(1818):3895, January 2023.  ISSN 2079-9292.  doi: 10.3390/electronics12183895.
  * Ge et al. [2023a] Zhijin Ge, Fanhua Shang, Hongying Liu, Yuanyuan Liu, and Xiaosen Wang.  Boosting adversarial transferability by achieving flat local maxima.  (arXiv:2306.05225), June 2023a.  doi: 10.48550/arXiv.2306.05225.  URL http://arxiv.org/abs/2306.05225.  arXiv:2306.05225 [cs].
  * Ge et al. [2023b] Zhijin Ge, Fanhua Shang, Hongying Liu, Yuanyuan Liu, Liang Wan, Wei Feng, and Xiaosen Wang.  Improving the transferability of adversarial examples with arbitrary style transfer.  August 2023b.  doi: 10.1145/3581783.3612070.  URL http://arxiv.org/abs/2308.10601.  arXiv:2308.10601 [cs, eess].
  * Liu et al. [2023b] Hongying Liu, Zhijin Ge, Zhenyu Zhou, Fanhua Shang, Yuanyuan Liu, and Licheng Jiao.  Gradient correction for white-box adversarial attacks.  _IEEE Transactions on Neural Networks and Learning Systems_ , page 1–12, 2023b.  ISSN 2162-2388.  doi: 10.1109/TNNLS.2023.3315414.
  * Wang et al. [2023] Zhibo Wang, Hongshan Yang, Yunhe Feng, Peng Sun, Hengchang Guo, Zhifei Zhang, and Kui Ren.  Towards transferable targeted adversarial examples.  page 20534–20543, 2023.  URL https://openaccess.thecvf.com/content/CVPR2023/html/Wang_Towards_Transferable_Targeted_Adversarial_Examples_CVPR_2023_paper.html.
  * Tramer [2022] Florian Tramer.  Detecting adversarial examples is (nearly) as hard as classifying them.  In _Proceedings of the 39th International Conference on Machine Learning_ , page 21692–21702. PMLR, June 2022.  URL https://proceedings.mlr.press/v162/tramer22a.html.
  * Byun et al. [2023] Junyoung Byun, Myung-Joon Kwon, Seungju Cho, Yoonji Kim, and Changick Kim.  Introducing competition to boost the transferability of targeted adversarial examples through clean feature mixup.  page 24648–24657, 2023.  URL https://openaccess.thecvf.com/content/CVPR2023/html/Byun_Introducing_Competition_To_Boost_the_Transferability_of_Targeted_Adversarial_Examples_CVPR_2023_paper.html.
  * Chen et al. [2023a] Bin Chen, Jiali Yin, Shukai Chen, Bohao Chen, and Ximeng Liu.  An adaptive model ensemble adversarial attack for boosting adversarial transferability.  page 4489–4498, 2023a.  URL https://openaccess.thecvf.com/content/ICCV2023/html/Chen_An_Adaptive_Model_Ensemble_Adversarial_Attack_for_Boosting_Adversarial_Transferability_ICCV_2023_paper.html.
  * Li et al. [2023a] Qizhang Li, Yiwen Guo, Xiaochen Yang, Wangmeng Zuo, and Hao Chen.  Improving transferability of adversarial examples via bayesian attacks.  (arXiv:2307.11334), July 2023a.  doi: 10.48550/arXiv.2307.11334.  URL http://arxiv.org/abs/2307.11334.  arXiv:2307.11334 [cs].
  * Li et al. [2023b] Qizhang Li, Yiwen Guo, Wangmeng Zuo, and Hao Chen.  Making substitute models more bayesian can enhance transferability of adversarial examples.  (arXiv:2302.05086), July 2023b.  doi: 10.48550/arXiv.2302.05086.  URL http://arxiv.org/abs/2302.05086.  arXiv:2302.05086 [cs].
  * Qin et al. [2022] Zeyu Qin, Yanbo Fan, Yi Liu, Li Shen, Yong Zhang, Jue Wang, and Baoyuan Wu.  Boosting the transferability of adversarial attacks with reverse adversarial perturbation.  _Advances in Neural Information Processing Systems_ , 35:29845–29858, December 2022.
  * Yang et al. [2023a] Xiangyuan Yang, Jie Lin, Hanlin Zhang, Xinyu Yang, and Peng Zhao.  Improving the transferability of adversarial examples via direction tuning.  (arXiv:2303.15109), August 2023a.  doi: 10.48550/arXiv.2303.15109.  URL http://arxiv.org/abs/2303.15109.  arXiv:2303.15109 [cs].
  * Yang et al. [2023b] Yulong Yang, Chenhao Lin, Qian Li, Chao Shen, Dawei Zhou, Nannan Wang, and Tongliang Liu.  Quantization aware attack: Enhancing the transferability of adversarial attacks across target models with different quantization bitwidths.  (arXiv:2305.05875), May 2023b.  doi: 10.48550/arXiv.2305.05875.  URL http://arxiv.org/abs/2305.05875.  arXiv:2305.05875 [cs].
  * Qi et al. [2023] Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal.  Visual adversarial examples jailbreak aligned large language models.  (arXiv:2306.13213), August 2023.  doi: 10.48550/arXiv.2306.13213.  URL http://arxiv.org/abs/2306.13213.  arXiv:2306.13213 [cs].
  * Pintor et al. [2022] Maura Pintor, Luca Demetrio, Angelo Sotgiu, Ambra Demontis, Nicholas Carlini, Battista Biggio, and Fabio Roli.  Indicators of attack failure: Debugging and improving optimization of adversarial examples.  _Advances in Neural Information Processing Systems_ , 35:23063–23076, December 2022.
  * Pomponi et al. [2022] Jary Pomponi, Simone Scardapane, and Aurelio Uncini.  Pixle: a fast and effective black-box attack based on rearranging pixels.  In _2022 International Joint Conference on Neural Networks (IJCNN)_. IEEE, jul 2022.  doi: 10.1109/ijcnn55064.2022.9892966.  URL https://doi.org/10.1109%2Fijcnn55064.2022.9892966.
  * Li et al. [2022a] Xiu-Chuan Li, Xu-Yao Zhang, Fei Yin, and Cheng-Lin Liu.  Decision-based adversarial attack with frequency mixup.  _IEEE Transactions on Information Forensics and Security_ , 17:1038–1052, 2022a.  doi: 10.1109/TIFS.2022.3156809.
  * Liu et al. [2022a] Ye Liu, Yaya Cheng, Lianli Gao, Xianglong Liu, Qilong Zhang, and Jingkuan Song.  Practical evaluation of adversarial robustness via adaptive auto attack.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15105–15114, 2022a.
  * Liu et al. [2022b] Fangcheng Liu, Chao Zhang, and Hongyang Zhang.  Towards transferable unrestricted adversarial examples with minimum changes, 2022b.
  * Apruzzese et al. [2023] Giovanni Apruzzese, Hyrum S. Anderson, Savino Dambra, David Freeman, Fabio Pierazzi, and Kevin Roundy.  “real attackers don’t compute gradients”: Bridging the gap between adversarial ml research and practice.  In _2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)_ , page 339–364, February 2023.  doi: 10.1109/SaTML54575.2023.00031.
  * Agarwal et al. [2022] Akshay Agarwal, Nalini Ratha, Mayank Vatsa, and Richa Singh.  Exploring robustness connection between artificial and natural adversarial examples.  page 179–186, 2022.  URL https://openaccess.thecvf.com/content/CVPR2022W/ArtOfRobust/html/Agarwal_Exploring_Robustness_Connection_Between_Artificial_and_Natural_Adversarial_Examples_CVPRW_2022_paper.html.
  * Aldahdooh et al. [2022] Ahmed Aldahdooh, Wassim Hamidouche, Sid Ahmed Fezza, and Olivier Déforges.  Adversarial example detection for dnn models: a review and experimental comparison.  _Artificial Intelligence Review_ , 55(6):4403–4462, August 2022.  ISSN 1573-7462.  doi: 10.1007/s10462-021-10125-w.
  * Altinisik et al. [2023] Enes Altinisik, Safa Messaoud, Husrev Taha Sencar, and Sanjay Chawla.  A3t: accuracy aware adversarial training.  _Machine Learning_ , 112(9):3191–3210, September 2023.  ISSN 1573-0565.  doi: 10.1007/s10994-023-06341-w.
  * Byun et al. [2022] Junyoung Byun, Seungju Cho, Myung-Joon Kwon, Hee-Seon Kim, and Changick Kim.  Improving the transferability of targeted adversarial examples through object-based diverse input.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15244–15253, June 2022.
  * Chen et al. [2023b] Bin Chen, Yan Feng, Tao Dai, Jiawang Bai, Yong Jiang, Shu-Tao Xia, and Xuan Wang.  Adversarial examples generation for deep product quantization networks on image retrieval.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(2):1388–1404, 2023b.  doi: 10.1109/TPAMI.2022.3165024.
  * Chen and Liu [2023] Pin-Yu Chen and Sijia Liu.  Holistic adversarial robustness of deep learning models.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 37(1313):15411–15420, September 2023.  ISSN 2374-3468.  doi: 10.1609/aaai.v37i13.26797.
  * Crecchi et al. [2022] Francesco Crecchi, Marco Melis, Angelo Sotgiu, Davide Bacciu, and Battista Biggio.  Fader: Fast adversarial example rejection.  _Neurocomputing_ , 470:257–268, January 2022.  ISSN 0925-2312.  doi: 10.1016/j.neucom.2021.10.082.
  * Dai et al. [2022] Tao Dai, Yan Feng, Bin Chen, Jian Lu, and Shu-Tao Xia.  Deep image prior based defense against adversarial examples.  _Pattern Recognition_ , 122:108249, February 2022.  ISSN 0031-3203.  doi: 10.1016/j.patcog.2021.108249.
  * Dunmore et al. [2023] Aeryn Dunmore, Julian Jang-Jaccard, Fariza Sabrina, and Jin Kwak.  A comprehensive survey of generative adversarial networks (gans) in cybersecurity intrusion detection.  _IEEE Access_ , 11:76071–76094, 2023.  ISSN 2169-3536.  doi: 10.1109/ACCESS.2023.3296707.
  * Dyrmishi et al. [2023] Salijona Dyrmishi, Salah Ghamizi, Thibault Simonetto, Yves Le Traon, and Maxime Cordy.  On the empirical effectiveness of unrealistic adversarial hardening against realistic adversarial attacks.  In _2023 IEEE Symposium on Security and Privacy (SP)_ , page 1384–1400, May 2023.  doi: 10.1109/SP46215.2023.10179316.
  * Freiesleben [2022] Timo Freiesleben.  The intriguing relation between counterfactual explanations and adversarial examples.  _Minds and Machines_ , 32(1):77–109, March 2022.  ISSN 1572-8641.  doi: 10.1007/s11023-021-09580-9.
  * Gong and Deng [2022] Weiyuan Gong and Dong-Ling Deng.  Universal adversarial examples and perturbations for quantum classifiers.  _National Science Review_ , 9(6):nwab130, June 2022.  ISSN 2095-5138.  doi: 10.1093/nsr/nwab130.
  * Gubri et al. [2022] Martin Gubri, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Koushik Sen.  Lgv: Boosting adversarial example transferability from large geometric vicinity.  In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, _Computer Vision – ECCV 2022_ , page 603–618, Cham, 2022. Springer Nature Switzerland.  ISBN 978-3-031-19772-7.
  * Hernández-Castro et al. [2022] Carlos Javier Hernández-Castro, Zhuoran Liu, Alex Serban, Ilias Tsingenopoulos, and Wouter Joosen.  _Adversarial Machine Learning_ , page 287–312.  Lecture Notes in Computer Science. Springer International Publishing, Cham, 2022.  ISBN 978-3-030-98795-4.  doi: 10.1007/978-3-030-98795-4_12.  URL https://doi.org/10.1007/978-3-030-98795-4_12.
  * Hu et al. [2022] Zichao Hu, Heng Li, Liheng Yuan, Zhang Cheng, Wei Yuan, and Ming Zhu.  Model scheduling and sample selection for ensemble adversarial example attacks.  _Pattern Recognition_ , 130:108824, October 2022.  ISSN 0031-3203.  doi: 10.1016/j.patcog.2022.108824.
  * Jun-hua et al. [2022] Z. O. U. Jun-hua, Duan Ye-xin, R. E. N. Chuan-lun, Q. I. U. Jun-yang, Zhou Xing-yu, and P. a. N. Zhi-song.  Perturbation initialization, adam-nesterov and quasi-hyperbolic momentum for adversarial examples.  _ACTA ELECTONICA SINICA_ , 50(1):207, January 2022.  ISSN 0372-2112.  doi: 10.12263/DZXB.20200839.
  * Lee and Kim [2023] Minjong Lee and Dongwoo Kim.  Robust evaluation of diffusion-based adversarial purification.  (arXiv:2303.09051), August 2023.  doi: 10.48550/arXiv.2303.09051.  URL http://arxiv.org/abs/2303.09051.  arXiv:2303.09051 [cs].
  * Li et al. [2022b] Yao Li, Minhao Cheng, Cho-Jui Hsieh, and Thomas C. M. Lee.  A review of adversarial attack and defense for classification methods.  _The American Statistician_ , 76(4):329–345, October 2022b.  ISSN 0003-1305.  doi: 10.1080/00031305.2021.2006781.
  * Liang et al. [2023] Chumeng Liang, Xiaoyu Wu, Yang Hua, Jiaru Zhang, Yiming Xue, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan.  Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples.  June 2023.  URL https://openreview.net/forum?id=Wbquvk97t4.
  * Liang et al. [2022] Hongshuo Liang, Erlu He, Yangyang Zhao, Zhe Jia, and Hao Li.  Adversarial attack and defense: A survey.  _Electronics_ , 11(88):1283, January 2022.  ISSN 2079-9292.  doi: 10.3390/electronics11081283.
  * Lin et al. [2022] Chang-Sheng Lin, Chia-Yi Hsu, Pin-Yu Chen, and Chia-Mu Yu.  Real-world adversarial examples via makeup.  In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , page 2854–2858, May 2022.  doi: 10.1109/ICASSP43922.2022.9747469.
  * Liu et al. [2022c] Jiabao Liu, Qixiang Zhang, Kanghua Mo, Xiaoyu Xiang, Jin Li, Debin Cheng, Rui Gao, Beishui Liu, Kongyang Chen, and Guanjie Wei.  An efficient adversarial example generation algorithm based on an accelerated gradient iterative fast gradient.  _Computer Standards & Interfaces_, 82:103612, August 2022c.  ISSN 0920-5489.  doi: 10.1016/j.csi.2021.103612.
  * Luo et al. [2022] Cheng Luo, Qinliang Lin, Weicheng Xie, Bizhu Wu, Jinheng Xie, and Linlin Shen.  Frequency-driven imperceptible adversarial attack on semantic similarity.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15315–15324, June 2022.
  * Midtlid et al. [2022] Kim A. B. Midtlid, Johannes Åsheim, and Jingyue Li.  Magnitude adversarial spectrum search-based black-box attack against image classification.  In _Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security_ , AISec’22, page 67–77, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450398800.  doi: 10.1145/3560830.3563723.  URL https://doi.org/10.1145/3560830.3563723.
  * Namiot and Ilyushin [2022] Dmitry Namiot and Eugene Ilyushin.  Generative models in machine learning.  _International Journal of Open Information Technologies_ , 10(77):101–118, June 2022.  ISSN 2307-8162.
  * Pang et al. [2022] Tianyu Pang, Huishuai Zhang, Di He, Yinpeng Dong, Hang Su, Wei Chen, Jun Zhu, and Tie-Yan Liu.  Two coupled rejection metrics can tell adversarial examples apart.  page 15223–15233, 2022.  URL https://openaccess.thecvf.com/content/CVPR2022/html/Pang_Two_Coupled_Rejection_Metrics_Can_Tell_Adversarial_Examples_Apart_CVPR_2022_paper.html.
  * Pawelczyk et al. [2022] Martin Pawelczyk, Chirag Agarwal, Shalmali Joshi, Sohini Upadhyay, and Himabindu Lakkaraju.  Exploring counterfactual explanations through the lens of adversarial examples: A theoretical and empirical analysis.  In _Proceedings of The 25th International Conference on Artificial Intelligence and Statistics_ , page 4574–4594. PMLR, May 2022.  URL https://proceedings.mlr.press/v151/pawelczyk22a.html.
  * Pinot et al. [2022] Rafael Pinot, Laurent Meunier, Florian Yger, Cédric Gouy-Pailler, Yann Chevaleyre, and Jamal Atif.  On the robustness of randomized classifiers to adversarial examples.  _Machine Learning_ , 111(9):3425–3457, September 2022.  ISSN 1573-0565.  doi: 10.1007/s10994-022-06216-6.
  * Qi et al. [2022] Peihan Qi, Tao Jiang, Lizhan Wang, Xu Yuan, and Zan Li.  Detection tolerant black-box adversarial attack against automatic modulation classification with deep learning.  _IEEE Transactions on Reliability_ , 71(2):674–686, 2022.  doi: 10.1109/TR.2022.3161138.
  * Qian et al. [2022] Zhuang Qian, Kaizhu Huang, Qiu-Feng Wang, and Xu-Yao Zhang.  A survey of robust adversarial training in pattern recognition: Fundamental, theory, and methodologies.  _Pattern Recognition_ , 131:108889, November 2022.  ISSN 0031-3203.  doi: 10.1016/j.patcog.2022.108889.
  * Ruiz et al. [2022] Nataniel Ruiz, Adam Kortylewski, Weichao Qiu, Cihang Xie, Sarah Adel Bargal, Alan Yuille, and Stan Sclaroff.  Simulated adversarial testing of face recognition models.  page 4145–4155, 2022.  URL https://openaccess.thecvf.com/content/CVPR2022/html/Ruiz_Simulated_Adversarial_Testing_of_Face_Recognition_Models_CVPR_2022_paper.html.
  * Sajeeda and Hossain [2022] Afia Sajeeda and B M Mainul Hossain.  Exploring generative adversarial networks and adversarial training.  _International Journal of Cognitive Computing in Engineering_ , 3:78–89, June 2022.  ISSN 26663074.  doi: 10.1016/j.ijcce.2022.03.002.
  * Singh et al. [2022] Amitoj Bir Singh, Lalit Kumar Awasthi, and Urvashi.  Defense against adversarial attacks using chained dual-gan approach.  In R. Asokan, Diego P. Ruiz, Zubair A. Baig, and Selwyn Piramuthu, editors, _Smart Data Intelligence_ , Algorithms for Intelligent Systems, page 121–133, Singapore, 2022. Springer Nature.  ISBN 978-981-19331-1-0.  doi: 10.1007/978-981-19-3311-0_11.
  * Tuna et al. [2022] Omer Faruk Tuna, Ferhat Ozgur Catak, and M. Taner Eskil.  Closeness and uncertainty aware adversarial examples detection in adversarial machine learning.  _Computers and Electrical Engineering_ , 101:107986, July 2022.  ISSN 0045-7906.  doi: 10.1016/j.compeleceng.2022.107986.
  * Vos and Verwer [2022] Daniël Vos and Sicco Verwer.  Robust optimal classification trees against adversarial examples.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(88):8520–8528, June 2022.  ISSN 2374-3468.  doi: 10.1609/aaai.v36i8.20829.
  * Sheatsley et al. [2022] Ryan Sheatsley, Blaine Hoak, Eric Pauley, and Patrick McDaniel.  The space of adversarial strategies.  (arXiv:2209.04521), September 2022.  doi: 10.48550/arXiv.2209.04521.  URL http://arxiv.org/abs/2209.04521.  arXiv:2209.04521 [cs].
  * Gubri et al. [2023] Martin Gubri, Maxime Cordy, and Yves Le Traon.  Going further: Flatness at the rescue of early stopping for adversarial example transferability.  (arXiv:2304.02688), April 2023.  doi: 10.48550/arXiv.2304.02688.  URL http://arxiv.org/abs/2304.02688.  arXiv:2304.02688 [cs, stat].
  * Li et al. [2023c] Huanhuan Li, Wenbo Yu, and He Huang.  Strengthening transferability of adversarial examples by adaptive inertia and amplitude spectrum dropout.  _Neural Networks_ , 165:925–937, August 2023c.  ISSN 0893-6080.  doi: 10.1016/j.neunet.2023.06.031.
  * Fang et al. [2023] Yan Fang, Zhongyuan Wang, Jikang Cheng, Ruoxi Wang, and Chao Liang.  Promoting adversarial transferability with enhanced loss flatness.  In _2023 IEEE International Conference on Multimedia and Expo (ICME)_ , pages 1217–1222, 2023.  doi: 10.1109/ICME55011.2023.00212.
  * Wang et al. [2022] Yuxuan Wang, Jiakai Wang, Zixin Yin, Ruihao Gong, Jingyi Wang, Aishan Liu, and Xianglong Liu.  Generating transferable adversarial examples against vision transformers.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , MM ’22, page 5181–5190, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450392037.  doi: 10.1145/3503161.3547989.  URL https://doi.org/10.1145/3503161.3547989.
  * Yamamura et al. [2022] Keiichiro Yamamura, Haruki Sato, Nariaki Tateiwa, Nozomi Hata, Toru Mitsutake, Issa Oe, Hiroki Ishikura, and Katsuki Fujisawa.  Diversified adversarial attacks based on conjugate gradient method.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 24872–24894. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/yamamura22a.html.
  * Long et al. [2022] Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, and Jingkuan Song.  Frequency domain model augmentation for adversarial attack.  In _European Conference on Computer Vision_ , pages 549–566. Springer, 2022.
  * Chen et al. [2022] Zhaoyu Chen, Bo Li, Shuang Wu, Jianghe Xu, Shouhong Ding, and Wenqiang Zhang.  Shape matters: deformable patch attack.  In _European conference on computer vision_ , pages 529–548. Springer, 2022.
  * Chen et al. [2023c] Jianqi Chen, Hao Chen, Keyan Chen, Yilan Zhang, Zhengxia Zou, and Zhenwei Shi.  Diffusion models for imperceptible and transferable adversarial attack, 2023c.
  * Chen et al. [2023d] Zhaoyu Chen, Bo Li, Shuang Wu, Kaixun Jiang, Shouhong Ding, and Wenqiang Zhang.  Content-based unrestricted adversarial attack, 2023d.
  * Dai et al. [2023] Xuelong Dai, Kaisheng Liang, and Bin Xiao.  Advdiff: Generating unrestricted adversarial examples using diffusion models, 2023.
  * Liu et al. [2023c] Jiang Liu, Chun Pong Lau, and Rama Chellappa.  Diffprotect: Generate adversarial examples with diffusion models for facial privacy protection, 2023c.
  * Lin et al. [2023] Shuo-Yen Lin, Ernie Chu, Che-Hsien Lin, Jun-Cheng Chen, and Jia-Ching Wang.  Diffusion to confusion: Naturalistic adversarial patch generation based on diffusion model for object detector, 2023.
  * Nie et al. [2022] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar.  Diffusion models for adversarial purification.  _arXiv preprint arXiv:2205.07460_ , 2022.
  * Bose et al. [2020] Joey Bose, Gauthier Gidel, Hugo Berard, Andre Cianflone, Pascal Vincent, Simon Lacoste-Julien, and Will Hamilton.  Adversarial example games.  _Advances in neural information processing systems_ , 33:8921–8934, 2020.
  * Wang et al. [2021] Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He.  Admix: Enhancing the transferability of adversarial attacks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 16158–16167, October 2021.
  * Bai et al. [2021] Tao Bai, Jun Zhao, Jinlin Zhu, Shoudong Han, Jiefeng Chen, Bo Li, and Alex Kot.  Ai-gan: Attack-inspired generation of adversarial examples.  In _2021 IEEE International Conference on Image Processing (ICIP)_ , pages 2543–2547, 2021.  doi: 10.1109/ICIP42928.2021.9506278.
  * Papernot et al. [2018] Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael P. Wellman.  Sok: Security and privacy in machine learning.  In _2018 IEEE European Symposium on Security and Privacy (EuroSP)_ , pages 399–414, 2018.  doi: 10.1109/EuroSP.2018.00035.
  * Carlini and Wagner [2017] Nicholas Carlini and David Wagner.  Towards evaluating the robustness of neural networks.  In _2017 IEEE Symposium on Security and Privacy (SP)_ , pages 39–57, 2017.  doi: 10.1109/SP.2017.49.
  * Chen et al. [2021] Yuefeng Chen, Xiaofeng Mao, Yuan He, Hui Xue, Chao Li, Yinpeng Dong, Qi-An Fu, Xiao Yang, Wenzhao Xiang, Tianyu Pang, Hang Su, Jun Zhu, Fangcheng Liu, Chao Zhang, Hongyang Zhang, Yichi Zhang, Shilong Liu, Chang Liu, Wenzhao Xiang, Yajie Wang, Huipeng Zhou, Haoran Lyu, Yidan Xu, Zixuan Xu, Taoyu Zhu, Wenjun Li, Xianfeng Gao, Guoqiu Wang, Huanqian Yan, Ying Guo, Chaoning Zhang, Zheng Fang, Yang Wang, Bingyang Fu, Yunfei Zheng, Yekui Wang, Haorong Luo, and Zhen Yang.  Unrestricted adversarial attacks on imagenet competition, 2021.
  * Gilmer et al. [2018] Justin Gilmer, Ryan P. Adams, Ian Goodfellow, David Andersen, and George E. Dahl.  Motivating the rules of the game for adversarial example research, 2018.
  * LeCun [1998] Yann LeCun.  The mnist database of handwritten digits, 1998.  In Technical report.
  * Krizhevsky [2009] Alex Krizhevsky.  Learning multiple layers of features from tiny images, 2009.  In Technical report.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition, 2015.
  * Huang et al. [2017] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * Szegedy et al. [2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International conference on machine learning_ , pages 10347–10357. PMLR, 2021.
  * Engstrom et al. [2019] Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras.  Robustness (python library), 2019.  _URL https://github.com/MadryLab/robustness_ , 4(4):4–3, 2019.
  * Salman et al. [2020] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry.  Do adversarially robust imagenet models transfer better?  _Advances in Neural Information Processing Systems_ , 33:3533–3545, 2020.
  * Wong et al. [2020] Eric Wong, Leslie Rice, and J. Zico Kolter.  Fast is better than free: Revisiting adversarial training, 2020.
  * Zagoruyko and Komodakis [2017] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks, 2017.
  * Zhang and Wang [2019] Haichao Zhang and Jianyu Wang.  Defense against adversarial attacks using feature scattering-based adversarial training.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Croce et al. [2020] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein.  Robustbench: a standardized adversarial robustness benchmark.  _arXiv preprint arXiv:2010.09670_ , 2020.
  * Cui et al. [2021] Jiequan Cui, Shu Liu, Liwei Wang, and Jiaya Jia.  Learnable boundary guided adversarial training.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 15721–15730, 2021.
  * Rice et al. [2020] Leslie Rice, Eric Wong, and Zico Kolter.  Overfitting in adversarially robust deep learning.  In _International Conference on Machine Learning_ , pages 8093–8104. PMLR, 2020.
  * Rebuffi et al. [2021] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann.  Fixing data augmentation to improve adversarial robustness.  _arXiv preprint arXiv:2103.01946_ , 2021.
  * Wu et al. [2020] Dongxian Wu, Shu-Tao Xia, and Yisen Wang.  Adversarial weight perturbation helps robust generalization.  _Advances in Neural Information Processing Systems_ , 33:2958–2969, 2020.
  * Gowal et al. [2020] Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli.  Uncovering the limits of adversarial training against norm-bounded adversarial examples.  _arXiv preprint arXiv:2010.03593_ , 2020.
  * Hendrycks et al. [2019] Dan Hendrycks, Kimin Lee, and Mantas Mazeika.  Using pre-training can improve model robustness and uncertainty.  In _International conference on machine learning_ , pages 2712–2721. PMLR, 2019.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.

##  Appendix A Proofs

We can prove the ordering of this Hasse diagram using definition 8. We will
first state the following lemmas and use them to prove Theorem 1. We will
prove them afterwards.

######  Lemma 1.

$\mathcal{O}_{S\&A}\sqsubset\mathcal{O}_{M}$ and
$\mathcal{O}_{M}\not\sqsubset\mathcal{O}_{S\&A}$

######  Lemma 2.

$\mathcal{O}_{L}\sqsubset\mathcal{O}_{S}$ and
$\mathcal{O}_{S}\not\sqsubset\mathcal{O}_{L}$

######  Lemma 3.

$\mathcal{O}_{SPA}\sqsubset\mathcal{O}_{A}$ and
$\mathcal{O}_{A}\not\sqsubset\mathcal{O}_{SPA}$

###### Proof: Theorem: 1.

First, $\mathcal{O}_{SPA}\sqsubset\mathcal{O}_{L\&SPA}$,
$\mathcal{O}_{S}\sqsubset\mathcal{O}_{S\&SPA}$,
$\mathcal{O}_{S}\sqsubset\mathcal{O}_{S\&A}$,
$\mathcal{O}_{A}\sqsubset\mathcal{O}_{L\&A}$, and
$\mathcal{O}_{L}\sqsubset\mathcal{O}_{L\&A}$ follow directly from definition 9
and part 2(a) of definition 8.
Likewise, $\mathcal{O}_{L\&SPA}\not\sqsubset\mathcal{O}_{SPA}$,
$\mathcal{O}_{S\&SPA}\not\sqsubset\mathcal{O}_{S}$,
$\mathcal{O}_{S\&A}\not\sqsubset\mathcal{O}_{S}$,
$\mathcal{O}_{L\&A}\not\sqsubset\mathcal{O}_{A}$, and
$\mathcal{O}_{L\&A}\not\sqsubset\mathcal{O}_{L}$ follow directly from
definition 9 and part 2(b) of definition 8.
Using Lemma 2 and part 2(c) of definition 8, we yield that
$\mathcal{O}_{L\&A}\sqsubset\mathcal{O}_{S\&A}$,
$\mathcal{O}_{S\&A}\not\sqsubset\mathcal{O}_{L\&A}$,
$\mathcal{O}_{L\&SPA}\sqsubset\mathcal{O}_{S\&SPAA}$, and
$\mathcal{O}_{S\&SPA}\not\sqsubset\mathcal{O}_{L\&SPA}$.
Using Lemma 3 and part 2(c) of definition 8, we get that
$\mathcal{O}_{S\&SPA}\sqsubset\mathcal{O}_{S\&A}$ and
$\mathcal{O}_{S\&A}\not\sqsubset\mathcal{O}_{S\&SPA}$.
Using all of this in addition to Lemma 1, we get that Theorem 1 holds under
the $\sqsubset$ ordering. ∎

We now move on to proving the various Lemmas we used to prove Theorem 1.

###### Proof: Lemma 1.

To show $\mathcal{O}_{S\&A}\sqsubset\mathcal{O}_{M}$, we need to find PPT $f$
such that, $\forall a\in\\{0,1\\}^{*},\\\
\mathcal{O}_{S\&A}(a)=f(\mathcal{O}_{M}(a))$. Incidentally, we also need that
$State(\mathcal{O}_{S\&A})=f(\mathcal{O}_{M}(a))$ when both oracles are
queried the same number of times. We will have two cases, Case 1 where the
attacker has unlimited queries, and Case 2 where the attacker is limited.
For the sake of simplicity, we assume that $a\in\mathcal{I}$ (as otherwise
they both return $[]$, and we can set $f$ to return $[]$ when it receives $[]$
anyway and $\mathcal{O}_{S\&A}(a)=f(\mathcal{O}_{M}(a))$).

##### Case 1:

Let $f$ be the following:

  * •

On input $x\in\mathcal{I}$, $f$ receives $[\theta_{M},x]=\mathcal{O}_{M}(x)$.
From $\theta_{M}$, we can construct $M$ and $\phi_{M}$ (by definition of
$\theta_{M}$).

  * •

We can then compute and return $[M(x),\phi_{M},x]=\mathcal{O}_{S\&A}(x)$.

##### Case 2:

Let $f$ be the following and $k$ the current allowed number of queries left:

  * •

On input $x\in\mathcal{I}$, $f$ receives $[\theta_{M},x]=\mathcal{O}_{M}(x)$.
From $\theta_{M}$, we can construct $M$ and $\phi_{M}$ (by definition of
$\theta_{M}$).

  * •

If $k=0$, return $[[],\phi_{M},x]=\mathcal{O}_{S\&A}(x)$.

  * •

Else, update $State(\mathcal{O}_{M}(x))$ to be the function that returns
$[k-1]$, return
$[M(x),\phi_{M},x]=\mathcal{O}_{S\&A}(x)$.

So we have that $\forall a\in\\{0,1\\}^{*}$,
$\mathcal{O}_{S\&A}(a)=f(\mathcal{O}_{M}(a))$. Hence,
$\mathcal{O}_{S\&A}\sqsubset\mathcal{O}_{M}$.

Now for the opposite, $\mathcal{O}_{M}\not\sqsubset\mathcal{O}_{S\&A}$. For
simplicity’s sake, we’ll cover only the first case, but the proof also applies
to the second case by performing the update to the $State$ function as we did
above. We’ll prove this by contradiction. Given $\mathcal{O}_{S\&A}$, an
attacker can receive $\phi_{M}$ and $M(x)$ for any $x\in\mathcal{I}$. We are
trying to reconstruct $\theta_{M}$. In the case where $M$ is linear, with
enough queries, we can solve the system of linear equations and reconstruct
$\theta_{M}$. However, in the case where $M$ is non-linear, there can be an
infinite number of possible viable solutions for any given finite amount of
queries, therefore it is impossible to always determine with certainty the
exact $\theta_{M}$ (although it is possible to approximate it, as model
stealing attacks demonstrate). Hence, it is not always possible to reconstruct
$\theta_{M}$ from $\phi_{M}$ and score query-access, meaning that there is no
PPT $f$ s.t. $\forall
a\in\\{0,1\\}^{*},f(\mathcal{O}_{S\&A}(a))=\mathcal{O}_{M}(a)$. ∎

###### Proof: Lemma 2.

We will prove Lemma 2 in the same way we proved Lemma 1, i.e. $\exists f$ such
that $\forall a\in\mathcal{I}$, $f(\mathcal{O}_{S})=\mathcal{O}_{L}$.

##### Case 1. Unlimited queries:

Let $f$ be the following:

  * •

On input $x\in\\{0,1\\}^{*}$, if $x\notin\mathcal{I}$, then $f$ receives $[]$
and return $[]$.

  * •

Else, on input $x\in\mathcal{I}$, $f$ receives $[M(x),x]$ and returns
$[argmax(M(x)),x]=\mathcal{O}_{L}$.

##### Case 2. Limited queries:

Let $f$ be the following and $k$ the current allowed number of queries left:

  * •

On input $x\in\\{0,1\\}^{*}$, if $x\notin\mathcal{I}$, then $f$ receives $[]$
and return $[]$.

  * •

Else, on input $x\in\mathcal{I}$, $f$ receives $[M(x),x]$ if $k>0$ else it
receives $[[],x]$.

  * •

If $k=0$, return $[[],x]=\mathcal{O}_{L}$.

  * •

Else, return $[argmax(M(x)),x]=\mathcal{O}_{L}$.

For the other direction, similarly to Lemma 1, we will prove only the first
case as the proof can trivially be extended to the second case. Given
$\mathcal{O}_{L}$, we want to find a PPT function $f$ s.t. $\forall
a\in\mathcal{I}$, $f(\mathcal{O}_{L}(a))=\mathcal{O}_{S}(a)$. We will show
that such a function cannot exist. For
$f(\mathcal{O}_{L}(a))=\mathcal{O}_{S}(a)$ to happen, it means that $f$ needs
to always be able to at least compute the score value of the label itself. It
is given no information beyond the label itself for any input
$x\in\mathcal{I}$. Assume the following setup. One party holds two models
$M_{1}$ and $M_{2}$ that make identical score predictions except for one
specific input $x^{*}$ where $M_{1}(x^{*})=u$ and $M_{2}(x^{*})=v$ and $u\neq
v$, we will also assume that while the score changes when inferring on
$x^{*}$, the predicted label remains the same for both. We argue that it would
be impossible for the other party that receives only the predicted label to
distinguish between both models. This means that there exists at least one
instance where there is no PPT function $f$ s.t. $\forall a\in\mathcal{I}$,
$f(\mathcal{O}_{L}(a))=\mathcal{O}_{S}(a)$. Therefore, we showed the other
direction also holds.
Hence, Lemma 2 holds. ∎ In this case, we do not need to modify
$State(\mathcal{O}_{S})$ as we query $\mathcal{O}_{S}$ exactly once whenever
we query $\mathcal{O}_{L}$.

###### Proof: Lemma 3.

$\mathcal{O}_{SPA}\sqsubset\mathcal{O}_{A}$ follows from part 1(b) of
definition 8 and $\mathcal{O}_{SPA}$’s definition. For the other direction,
since $\mathcal{O}_{SPA}(x)=[\\{\phi_{0},\phi_{1},\dots,\phi_{k}\\},x]$ for
some positive integer $k$ and where for a uniformly randomly sampled
$i\in[0,1,\dots,k]$, $\phi_{i}=\phi_{M}$, a PPT function $f$ would be able to
always determine which of the $k$ elements in the unordered set is the correct
architecture. Given that the function is only given the unordered set, it
would be akin to being able to always correctly guess a random number, which
is not possible. Therefore, $\mathcal{O}_{A}\not\sqsubset\mathcal{O}_{SPA}$
and Lemma 3 holds. ∎

###### Proof: Theorem 2.

Theorem 2 follows directly from definitions 8 and 9. ∎

###### Proof: Theorem 3.

By definition, we have the oracles defined by $\mathcal{O}_{T}$ that are
properly defined and ordered under $\sqsubset$. It remains to show that
$\mathcal{O}_{Train}$ dominates any one of them and none of them dominates
$\mathcal{O}_{Train}$. By how any $T_{i}$ is defined, we have that $Train\in
T_{i}$ for any integer $i$. Since $T_{i}$ is an unordered set, we can apply
part 1(b) of definition 8 to yield that $\forall i\in\mathbb{Z}$,
$\mathcal{O}_{T_{i}}\sqsubset\mathcal{O}_{Train}$. We’ll show that $\forall
i\in\mathbb{Z}$, $\mathcal{O}_{Train}\not\sqsubset\mathcal{O}_{T_{i}}$ by
contradiction. By part 1(c) of definition 8, for any integer $i$, there needs
to be a PPT function $f$ such that $f([T_{i},x])=[Train,x]$ for any
$x\in\\{0,1\\}$. Hence, we need some PPT function that can extract $Train$
from $T_{i}$ (as we know it is in $T_{i}$ by definition of $T_{i}$). However,
$T_{i}$ is an unordered set and $f$ only has access to $T_{i}$ (and the query
itself $x$ but in this case it should not provide additional information about
the ordering of $T_{i}$). So, for $f$ to be able to distinguish $Train$ from
any other of the other $Train^{\prime}$ functions in $T_{i}$ would be
equivalent to being able to perfectly guess truly random numbers, which is
impossible. Therefore, we have a contradiction and so $\forall
i\in\mathbb{Z}$, $\mathcal{O}_{Train}\not\sqsubset\mathcal{O}_{T_{i}}$. ∎

###### Proof: Theorem 4.

$\mathcal{O}_{PA}\sqsubset\mathcal{O}_{FA}$ follows directly from part 2(c)
and 1(b) of definition 8 as well as $\mathcal{O}_{FA}$’s definition since
$\varrho\in\\{\varrho_{1},\dots,\varrho_{k}\\}$.
$\mathcal{O}_{FA}\not\sqsubset\mathcal{O}_{PA}$ follows from the impossibility
of perfectly guessing truly random numbers (since $i$ s.t.
$\varrho_{i}=\varrho$ is uniformly randomly sampled) and part 1(c) of
definition 8. $\mathcal{O}_{SPD}\sqsubset\mathcal{O}_{PA}$ also follows
directly from part 2(c) and 1(b) of definition 8 as well as
$\mathcal{O}_{SPD}$’s definition since $\rho\in\\{\rho_{1},\dots,\rho_{k}\\}$.
$\mathcal{O}_{PA}\not\sqsubset\mathcal{O}_{SPD}$ also follows from the
impossibility of perfectly guessing truly random numbers and part 1(c) of
definition 8.

∎

##  Appendix B Attack Summary Table

We complete Table 3 with the information from the papers not included in the
evaluation in Section 6 with Table 15.

| Attack
---
(Targeted)
Model | Data | Train | Metric
| ODI
---
[42] (t)
| Possible
---
Architectures
| Training
---
Data
| Training
---
Function
l∞
| GA
---
[37]
| Possible
---
Architectures
| Training
---
Data
| Training
---
Function
l∞
| AI-FGTM
---
[12]
| Possible
---
Architectures
| Training
---
Data
| Training
---
Function
l∞
| F-Attack
---
[35]
Labels |  | Same
---
Distribution
$\emptyset$ | None
| Admix (A)
---
[89] (both)
| Possible
---
Architectures
| Training
---
Data & Same
Distribution
| Training
---
Function
None
| Admix (B)
---
[89] (both)
Architecture |  | Training
---
Data & Same
Distribution
| Training
---
Function
None
| Admix (C)
---
[89] (both)
Parameters |  | Same
---
Distribution
| Loss
---
Function
None
| Shadow
---
[11]
Scores | Other Data | $\emptyset$ | None
| DAPatch
---
[81]
Parameters | $\emptyset$ | $\emptyset$ | l0
| S2I (A)
---
[80] (both)
| Possible
---
Architectures
| Training
---
Data
| Training
---
Function
l∞
| S2I (B)
---
[80] (both)
Parameters | $\emptyset$ |  | Loss
---
Function
l∞
| AI-GAN
---
[90] (t)
Parameters |  | Training
---
Data
$\emptyset$ | l∞
| ACA (A)
---
[83]
| Possible
---
Architectures
| Training
---
Data &
Other Data
| Training
---
Function
None
| ACA (B)
---
[83]
Parameters | Other Data |  | Loss
---
Function
None
| DiffAttack (A)
---
[82]
| Possible
---
Architectures
| Training
---
Data &
Other Data
| Training
---
Function
FID [115]
| DiffAttack (B)
---
[82]
Parameters | Other Data |  | Loss
---
Function
FID [115]

Table 15: Attack Summary Table. Supplements Table 3.