# It Is All About Data: A Survey on the Effects of Data on Adversarial
Robustness

Peiyu Xiong  University of British ColumbiaVancouverCanada<EMAIL_ADDRESS>,  Michael Tegegn  University of British ColumbiaVancouverCanada
<EMAIL_ADDRESS>,  Jaskeerat Singh Sarin  University of British
ColumbiaVancouverCanada<EMAIL_ADDRESS>,  Shubhraneel Pal  Indian
Institute of Technology, KharagpurIndia<EMAIL_ADDRESS>and  Julia
Rubin  University of British ColumbiaVancouverCanada<EMAIL_ADDRESS>
###### Abstract.

Adversarial examples are inputs to machine learning models that an attacker
has intentionally designed to confuse the model into making a mistake. Such
examples pose a serious threat to the applicability of machine-learning-based
systems, especially in life- and safety-critical domains. To address this
problem, the area of adversarial robustness investigates mechanisms behind
adversarial attacks and defenses against these attacks. This survey reviews
literature that focuses on the effects of data used by a model on the model’s
adversarial robustness. It systematically identifies and summarizes the state-
of-the-art research in this area and further discusses gaps of knowledge and
promising future research directions.

Machine Learning, Adversarial Robustness, Evasion Attack, Data Properties

††ccs: Computing methodologies Machine learning††ccs: Security and privacy
Software and application security

##  1\. Introduction

Recent advances in Machine Learning (ML) led to the development of numerous
accurate and scalable ML-based techniques, which are increasingly used in
industry and society. Yet, concerns related to the safety and security of ML-
based systems could substantially impede their widespread adoption, especially
in the area of safety-critical systems, such as autonomous cars. Examples of
fooling ML models into making wrong predictions by adding imperceptible-to-
the-human noise to the input are well known (Goodfellow et al., 2017):
adversarial perturbation to a stop sign may cause a machine learning system to
recognize it as a “max speed” sign instead, which might lead to wrong and
dangerous actions taken by an autonomous car (Eykholt et al., 2018) (see Fig.
1). Likewise, malicious software can be perturbed to bypass security models
while still retaining its malicious behavior (Demontis et al., 2019).

Figure 1. Adversarial examples for traffic signs (picture by Chen and Wu
(Jiefeng Chen, 2019)).

ML models are susceptible to such scenarios, known as _adversarial attacks_ or
_adversarial examples_ , if not specifically trained for (Szegedy et al.,
2014; Goodfellow et al., 2014). To address this problem, recent literature
investigates mechanisms behind adversarial attacks and proposes defenses
against these attacks – an area commonly referred to as _adversarial
robustness_. The performance of ML models under adversarial attacks, known as
_robust accuracy_ or _robust generalization_ , is often distinguished from the
general model accuracy, known as _standard accuracy_ or _standard
generalization_.

Adversarial attacks that aim to decrease model accuracy can roughly be divided
into _evasion_ and _poisoning attacks_ (Biggio and Roli, 2018; Li et al.,
2021). The stop sign example above is, in fact, an evasion attack, where the
attacker carefully modifies the input to mislead the prediction (Szegedy et
al., 2014; Madry et al., 2018; Kurakin et al., 2017; Carlini and Wagner,
2017). Instead of changing model inputs, poisoning attacks are carried out by
injecting corrupt data into the training dataset, to deceive the model
(Goldblum et al., 2022; Shafahi et al., 2018; Tian et al., 2022). _The focus
of this survey is on evasion attacks_ , as they are more common, accessible,
and more frequently discussed in the literature (Machado et al., 2021;
Chakraborty et al., 2018; viso.ai, 2022; Li et al., 2021). In fact, most of
the literature uses the term _adversarial attacks_ to refer to evasion attacks
(Madry et al., 2018; Kurakin et al., 2017; Yuan et al., 2019); we thus use
these two terms interchangeably.

Most techniques that study adversarial evasion attacks attribute adversarial
vulnerability to different aspects of the learning algorithm and/or properties
of the data. There are numerous existing surveys on adversarial robustness
that focus on different types of adversarial attacks and defenses (Zhou et
al., 2022; Akhtar et al., 2021; Li et al., 2021) and sources of adversarial
vulnerability related to learning algorithms (Machado et al., 2021; Serban et
al., 2020). Yet, to the best of our knowledge, there are no surveys that
collect and organize literature focusing on the influence of data on
adversarial robustness.

Our work addresses this gap. Specifically, we investigate (a) what properties
of data influence model robustness and (b) how to select, represent, and use
data to improve model robustness. To the best of our knowledge, this is the
first survey to analyze adversarial robustness from the perspective of data
properties – an important direction as the quality of data determines what is
achievable through any learning algorithm.

To collect literature relevant to our survey, we used popular digital
libraries and search engines, selecting papers that investigate the effect of
data on ML adversarial robustness. We identified more than 3,089 potentially
relevant papers published in top scientific venues on Machine Learning,
Computer Vision, Computational Linguistics, and Security. We systematically
inspected these papers, identifying 57 papers relevant to our survey. We
further analyzed, categorized, and described the selected papers in this
manuscript.

Main findings. The results of our analysis show that producing accurate and
robust models requires a larger _number of samples_ for training than
achieving high accuracy alone. The required number of samples to learn a
robust model also depends on other properties of the data, such as
dimensionality and the data distribution itself. Specifically, input data with
higher _dimensionality_ , i.e., a larger number of features that represent the
input dataset, requires a larger number of samples to produce a robust model.
This is consistent with other findings showing that high _dimensionality_ is
undesirable for robustness. Moreover, some data _distributions_ are inherently
more robust than others, e.g., a Gaussian mixture distribution requires more
samples to produce a robust model than needed by a Bernoulli mixture
distribution.

Another aspect that affects robustness is the _density_ of data samples within
classes, which measures how far apart samples are from each other. Papers show
that high class density correlates to high robust accuracy and that
adversarial examples are commonly found in low-density regions of the data.
This is intuitive as low-density regions imply that there are not enough
samples to accurately characterize the region. A related property,
_concentration_ , measures how fast the value of a function defined over a
data region, e.g., _error rate_ , grows as the region expands. This concept of
concentration tightly corresponds to adversarial robustness if we consider the
expansion of a data region as the effect of adversarial perturbation, i.e.,
perturbing samples in all directions causes the region defined by the original
samples to expand. In this case, high concentration implies that the error
rate grows as one perturbs all points in a data region and, thus, datasets
with high concentration are shown to be inevitably non-robust. The
_separation_ between classes of the underlying data distribution also affects
robustness, with a large distance between different classes being desirable
for adversarial robustness as an attacker would need to use a large
perturbation to move samples from one class to another.

Yet another aspect that impacts robust accuracy is the presence of mislabeled
samples in a dataset, referred to as _label noise_. Furthermore, refining
labels to reason about a larger number of classes, e.g., splitting the
“animal” class into “cat” and “dog” may improve adversarial robustness as such
labels allow learning more compact representations for samples that share
stronger similarities.

A number of papers also identify _domain-specific_ properties that correlate
with adversarial robustness. For example, image frequency – the rate of pixel
value change – affects robustness, and it is advisable to use a diverse
frequency range in the training dataset to prevent any frequency biases which
give rise to adversarial examples.

Observations and Gaps. Our literature review shows that, even though most
works study data properties from a domain-agnostic perspective, they typically
conduct an empirical evaluation on image datasets only. This constrains the
types of attacks and robustness measurements considered, so findings may not
generalize to other domains or types of datasets. Furthermore, most works base
their formal derivations on quite simple synthetic data models, such as
uniform distributions, a mixture of Gaussian distributions, and a mixture of
Bernoulli distributions, which exhibit unrealistic assumptions compared to
real datasets used in practice. We also observed that while most papers only
perform a univariate analysis on a specific data property, most properties are
hard to independently optimize, e.g., to decrease dimensionality without
decreasing separation as decreasing the dimensionality implies that samples
have fewer features to be differentiated from each other. We also found that
some properties, e.g., separation, do not have a standard way of measurement
for concrete datasets. We believe future work should look into these
directions.

Contributions. The main contribution of this survey are:

  * $\bullet$

A collection of literature on the effects of data on adversarial robustness.

  * $\bullet$

A categorization and detailed analysis of the collected literature.

  * $\bullet$

An analysis of knowledge gaps and suggestions for future research directions.

Structure of the Survey. The remainder of the survey is structured as follows.
Section 2 introduces the necessary background and terminology used in this
survey. Section 3 presents our methodology for identifying and categorizing
relevant papers. We describe the collected literature in Section 4. In Section
5, we discuss our analysis of the literature, possible knowledge gaps, and
suggestions for future work. Section 6 summarizes related work and Section 7
concludes the survey.

##  2\. Preliminaries

We now provide a brief overview of the main concepts related to machine
learning, adversarial robustness, and most commonly studied data
distributions. The goal of this section is to introduce terminology used in
the rest of the survey rather than provide an extensive overview of the
adversarial robustness research area. For a more detailed overview, please
refer to guides on statistics and machine learning (Shalev-Shwartz and Ben-
David, 2014; Bonaccorso, 2017; Trevor Hastie, 2009) and adversarial robustness
(Chen and Hsieh, 2023; Biggio and Roli, 2018; Nicolae et al., 2018).

###  2.1. Machine Learning

Machine learning refers to the automated detection of meaningful patterns in
data (Shalev-Shwartz and Ben-David, 2014) and can be largely divided into
supervised, unsupervised, and reinforcement learning. In supervised learning,
a learning model is provided with input-output pairs of data (a.k.a. labeled
training data); based on this data, the model aims to infer a function that
maps the inputs to the outputs. Supervised learning is typically associated
with classification and regression problems, which use categorical and
continuous labels, respectively. In classification, this number of possible
labels for an input is also referred to as the number of _classes_. Datasets
with only two classes are called _binary datasets_ , on which one can train a
_binary classifier_.

Unlike supervised learning, unsupervised learning algorithms are usually
concerned with identifying patterns in unlabeled data, e.g., grouping similar
samples together in the absence of labels (clustering) or transforming data
into a different representation (representation learning). Reinforcement
learning characterizes algorithms that learn from a series of rewards and
punishments, with the goal of maximizing the cumulative reward, e.g., to build
robots that learn to take the best sequence of actions according to signals
from the environment.

Variations, such as, semi-supervised learning (i.e., learning from partially
labeled data) and self-supervised learning (i.e., learning from labels
extracted by the learner itself) have also been proposed for problems where
acquiring labeled data may be challenging or expensive.

ML algorithms can also be divided into parametric and non-parametric.
Parametric algorithms have a predetermined, fixed number of parameters defined
before the training starts. For example, for Linear Support Vector Machines
(SVMs), these parameters are the coefficients of all features of the training
data and the learned intercept. For Deep Neural Networks (DNNs), the number of
parameters is determined by the architecture of the network. In non-parametric
algorithms, the number of parameters is determined at training time and may
vary depending on the number of training samples. For example, the “depth” of
Decision Trees can grow (beyond the size of the feature set) when more
decision points are needed to accurately separate training data. Other
commonly used non-parametric models include $k$-Nearest Neighbors ($k$-NN) and
Kernel SVMs.

###  2.2. Adversarial Robustness

Adversarial machine learning studies the arms race between adversarial attacks
and defenses. Attacks aim at degrading models performance while defenses
propose algorithms to harden models against the attacks. Adversarial attacks
can be categorized into _evasion_ and _poisoning_ (Biggio and Roli, 2018; Li
et al., 2021). Evasion attacks aim to fool machine learning models by
generating inputs that, despite no noticeable difference for a human, will be
incorrectly classified. Such inputs, known as _adversarial examples_ and
created by applying non-random perturbations to samples, carefully designed to
change models’ prediction (Szegedy et al., 2014; Madry et al., 2018; Kurakin
et al., 2017; Carlini and Wagner, 2017). Instead, poisoning attacks tampers
with model training data, in order to degrade model performance. In this
survey, we focus on evasion attacks; the terms adversarial and evasion attacks
are often used interchangeably in the literature as this is the most popular
and commonly studied type of attack.

The term _robustness_ for machine learning models is often used to refer to
different concepts, such as, stability to distribution shifts, the ability to
identify adversarial examples, and the ability to make the correct predictions
in the face of adversarial examples. In this survey, we use the latter
definition – the ability to make the correct predictions in the face of
adversarial examples. This is a stronger notion of robustness than merely
identifying adversarial examples, as the identification of an adversarial
example does not guarantee its correct classification. The phenomenon of
making satisfactory model predictions in the face of adversarial examples is
also often referred to as _robust generalization_. This is different from
_standard generalization_ , a term used to describe making satisfactory model
predictions for normal, unseen samples.

Adversarial (Evasion) Attacks. Techniques for generating adversarial examples
for evasion attacks can be broadly divided into three categories, according to
the type of information available to the attacker (Biggio and Roli, 2018). In
_white-box attacks_ , the attacker is assumed to be able to leverage all
available information about the training data, the model, and training
procedure. In _grey-box attacks_ , the attacker is assumed to have only
partial information about the model, such as, the source of training data.
Finally, the most conservative type of attacks are _black-box attacks_ , where
the attacker has no information about the inner workings of the model except,
possibly, for the prediction outcomes.

_Gradient-based attacks_ are commonly used in white-box settings. These
attacks use the gradient of a differentiable function defined over model
weights as a guide when crafting adversarial examples. The most commonly used
differentiable function is the loss function used by the model during
training. A gradient defines the direction of the maximal increase in the
local value of a function. Hence, by using the gradient of the loss function
with respect to the input, one can adjust the input to get the maximal
increase in the loss of the model, which ultimately leads to a bad prediction.
Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015), Basic Iterative
Method (BIM) (Kurakin et al., 2017), and Projected Gradient Descent (PGD)
(Madry et al., 2018) are examples of attack algorithms that utilize the
gradient of the loss function used for training. Instead of the loss function,
the FAB attack (Croce and Hein, 2020) uses the gradient of a function defined
by the difference of model outputs of the penultimate layer of a neural
network – a layer which outputs the probabilities that a given sample belongs
to each of the available classes. By defining the difference of outputs of the
penultimate layer as the differentiable function, the FAB attack maximizes the
difference in probabilities between the target class and other classes, to
increase the chance of misclassification.

_Non-gradient based attacks_ are applicable for more diverse types of models
that do not use a differentiable functions, e.g., decision trees. Such attacks
can also be used in black-box and grey-box settings, when gradient information
is hidden from the attacker. One example of non-gradient-based attacks is the
_mimicry attack_ , which involves adding and removing features in the
perturbed sample, e.g., based on their popularity in the target class
(Demontis et al., 2019).

Adversarial Defenses. Defense mechanisms against adversarial attacks target
various stages of the machine learning pipeline. Specifically, defenses _on
raw data_ focus on the training data itself, e.g., by selecting a subset of
“robust” features (Ilyas et al., 2019) or using representation learning to
transform features into a different representation, making sure a model
trained on the new representation is inherently more robust (Yang et al.,
2021b).

Defenses _during training_ alter the standard training procedure to improve
model robustness. The most common such technique is adversarial training
(Goodfellow et al., 2015), which involves continually augmenting the training
data with adversarial examples generated by an attack algorithm. By retraining
the model while adding correctly labeled malicious samples to the training
dataset, the model learns to capture persistent patterns and becomes more
robust against these attacks. Another common method is _regularization_ ,
where model parameters are constrained so that very small perturbations have
little effect on the prediction outcome (Gouk et al., 2021).

Defenses _during inference_ focus on making existing models more robust when
the model is being used on new samples. For example, randomized smoothing
(Cohen et al., 2019) involves creating multiple noisy instances of a sample
and aggregating the model’s predictions during inference. Given that
adversarial examples are typically close to genuine samples, averaging the
results from close neighbors of an input can potentially reduce the chances of
the model being misled. In addition, different variations of ensemble models –
using multiple models and aggregating their output – have been shown to
increase the robustness to adversarial attacks (Pang et al., 2019).

Measures of Robustness. The strength of an adversary is mostly measured by the
size of the perturbation required to create an adversarial example. That is,
adversaries that introduce more perturbations, e.g., change a larger portion
of pixel values in the image, are considered to be stronger. A typical way of
measuring the perturbation size, especially in the image domain, is by using
the $L_{p}$ distance metric, where $p$ can be a whole number or $\infty$.
Specifically, $L_{0}$ counts the total number of changed features, regardless
of the changes to individual features. $L_{1}$ is the Manhattan distance,
i.e., the sum of absolute values representing a change in each feature.
$L_{2}$ measures by the Euclidean distance between the feature values of the
original and perturbed samples. the $L_{\infty}$ metric measure the largest
change in any of the features (while disregarding changes in all other
features).

There are two ways to utilize these distance metrics to evaluate the
robustness of a model: error-rate-based and radius-based. The first calculates
a pool of adversarial samples generated from a set of real samples with a
fixed allowable perturbation size (Madry et al., 2018). The robustness is then
defined as the error rate of the model on these adversarial samples. A related
concept, _adversarial risk_ , is also defined in a similar manner: the
probability of finding, within a certain predefined distance, an adversarial
example for a given real sample.

The radius-based way to evaluate is measuring the smallest distance required
to generate an adversarial sample from a given real sample (Szegedy et al.,
2014). This way is especially useful in robustness certification, which
involves learning a classifier that outputs a prediction along with a
certified radius within which the prediction is guaranteed to be consistent
(Lee et al., 2019).

###  2.3. Data Distributions

Numerous works study properties of particular data distributions, which we
discuss below. The _uniform distribution_ defines a probability distribution
in which every possible data point is equally likely. This implies that for a
continuous random variable in the interval $[a,b]$, the probability of seeing
a sample from the interval is $\frac{1}{b-a}$ and the probability of seeing a
sample from outside of the interval is $0$. In the discrete case with n
possible values, the uniform distribution assigns a probability of
$\frac{1}{n}$ to each value.

The _Bernoulli distribution_ defines a discrete probability distribution of a
random variable with two allowable values, 0 and 1. Such a random variable
takes the value of 1 with probability $p$ and the value of 0 with probability
$1-p$.

The _Gaussian (normal) distribution_ defines a continuous probability
distribution that assigns a probability with its peak at the center of the
distribution and decreasing symmetrically outwards. For a Gaussian
distribution, $\mu$ denotes the mean or center of the distribution, and
$\sigma^{2}$ denotes the variance or the spread of the distribution. Since the
mean and variance fully characterize a Gaussian distribution, it is also
commonly denoted as $\mathcal{N}(\mu,\sigma^{2})$.

Figure 2. A two-dimensional Gaussian mixture data.

One can also imagine a distribution made up of a _mixture_ of multiple
distributions. For example, Fig. 2 shows a distribution made up of two
Gaussians: one centered at $\mu_{1}$ and another – at $\mu_{2}$. This mixture
also contains labels associated for each independent Gaussian, shown by the
two clusters in the figure. Furthermore, these two clusters have the same
variance, i.e., the same spread of the distribution surrounding the center of
the class. While the means of the two classes are separated, the distributions
intersect with each other.

##  3\. Methodology

This section describes our methodology for identifying and categorization
relevant papers.

###  3.1. Paper Collection

Papers for this survey were collected in June 2022. We used the search query
schematically described below, which was designed to identify papers in the
area of Machine Learning adversarial robustness that discuss properties of the
underlying data. We expanded each of these conceptual terms with possible
synonyms and specific wording, making sure our query is as comprehensive as
possible.

> Search Query := Machine Learning + Adversarial Robustness + Data + Property
> Machine Learning := classif | “machine learning” | “deep learning” | “neural network”
> Adversarial Robustness := “adversarial robustness” | “adversarial vulnerability” | “adversarial attack” | “adversarial perturbation” | “adversarial defense” | “evasion attacks”
> Data := data | sample | input
> Property := propert | qualit | distribution | characteristic

In our schematic query representation, the “+” and “|” signs indicate the AND
and OR operators, respectively, the phases in quotes are matched in full, and
each word is matched with its suffixed versions, e.g., ’classif’ is matched
with both ’classifier’ and ’classification’. When performing the search, we
adapted this schematic query to the requirements and capabilities of each
search engine that we used.

Table 1. Considered publication venues

|  |  | Search Target |  |
---|---|---|---|---|---
Venue Type | Area | Venue Name | ACM | IEEE | Springer | Scopus |  | Google
---
Scholar
Total Hits |  Rele- vant
|  | AAAI Conference on Artificial Intelligence (AAAI) |  |  |  | ✓ | ✓ | 320 | 1
|  | ACM International Conference on Web Search and Data Mining (WSDM) | ✓ |  |  |  |  | 39 | 0
|  | ACM SIGKDD Conference On Knowledge and Data Mining (KDD) | ✓ |  |  |  |  | 11 | 0
|  | Conference on Neural Information Processing Systems (NeurIPS) |  |  |  | ✓ | ✓ | 658 | 19
|  | IEEE International Conference on Data Engineering (ICDE) |  | ✓ |  |  |  | 3 | 0
|  | IEEE International Conference on Data Mining (ICDM) |  | ✓ |  |  |  | 34 | 1
|  | International Conference on Learning Representations (ICLR) |  |  |  | ✓ | ✓ | 313 | 9
|  | International Conference on Learning Theory (COLT) |  |  |  | ✓ | ✓ | 14 | 1
|  | International Conference on Machine Learning (ICML) |  |  |  | ✓ | ✓ | 320 | 12
| ML | International Joint Conference on Artificial Intelligence (IJCAI) |  |  |  | ✓ | ✓ | 86 | 0
|  | European Conference on Computer Vision (ECCV) |  |  | ✓ |  |  | 88 | 2
|  | IEEE / CVF Computer Vision and Pattern Recognition (CVPR) |  | ✓ |  |  |  | 244 | 3
| CV | IEEE International Conference on Computer Vision (ICCV) |  | ✓ |  |  |  | 157 | 2
| CL | Annual Meeting of the Association for Computational Linguistics (ACL) |  |  |  | ✓ | ✓ | 83 | 0
|  | ACM Conference on Computer and Communications Security (CCS) | ✓ |  |  |  |  | 61 | 0
|  | IEEE Symposium on Security and Symposium (S&P) |  | ✓ |  |  |  | 20 | 0
|  | Network and Distributed System Security Symposium (NDSS) |  |  |  | ✓ | ✓ | 25 | 0
Conference | SEC | USENIX Security Symposium |  |  |  | ✓ | ✓ | 89 | 0
|  | Artificial Intelligence Journal |  |  |  | ✓ | ✓ | 5 | 0
|  | Computational Linguistics Journal (CL) |  |  |  | ✓ | ✓ | 3 | 0
|  | IEEE Transactions on Knowledge and Data Engineering (TKDE) |  | ✓ |  |  |  | 16 | 0
|  | IEEE Transactions on Neural Networks and Learning Systems (TNNLS) |  | ✓ |  |  |  | 29 | 0
|  | IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) |  | ✓ |  |  |  | 56 | 1
|  | International Journal of Computer Vision (IJCV) |  |  | ✓ |  |  | 26 | 0
|  | Journal of Machine Learning Research (JMLR) |  |  |  | ✓ | ✓ | 29 | 2
|  | Knowledge Based Systems (KBS) |  |  |  | ✓ | ✓ | 22 | 0
|  | Neural Networks |  |  |  | ✓ | ✓ | 51 | 0
| AI | Pattern Recognition (PR) |  |  |  | ✓ | ✓ | 53 | 0
|  | Computer & Security |  |  |  | ✓ | ✓ | 44 | 0
|  | IEEE Security & Privacy |  | ✓ |  |  |  | 10 | 0
|  | IEEE Transactions on Dependable and Secure Computing (TDSC) |  | ✓ |  |  |  | 26 | 0
|  | IEEE Transactions on Information and Forensic Science (TIFS) |  | ✓ |  |  |  | 82 | 1
Journal | IS | Journal of Information Security and Applications (JISA) |  |  |  | ✓ | ✓ | 13 | 0
| Surv. | ACM Computing Surveys (CSUR) | ✓ |  |  |  |  | 59 | 0
Total |  | 3089 | 54

An initial search using the query in Google Scholar identified more than
30,000 matches. To keep the scope of the survey manageable, we thus limited
our search to publications from the main track of top-tier conferences and
journals in the areas of Machine Learning (ML), Computer Vision (CV),
Computational Linguistics (CL), and Security (SEC). Specifically, we selected
all A* conferences from these areas using the most recent, 2021, CORE ranking
(Cor, [n. d.]); top 10 journals according to the Journal Citation Reports
(JCR) (Jou, [n. d.]) in the area of Artificial Intelligence (AI), which
includes Machine Learning, Computer Vision, and Computation Linguistics, and
top five journals in the area of Information System (IS), which includes
Security. Additionally, we included the ACM Computing Surveys Journal (ACM,
[n. d.]) to collect surveys related to our topic. The first three columns in
Table 1 shows the final list of publication venues that we selected.

We further identified digital libraries and search engines that host
proceedings of our selected venues. These are shown in the next five columns
of Table 1, for each venue individually. As some venues are only partially
indexed by the digital libraries, i.e., the libraries only include proceedings
from particular years, we augmented our search for papers in these venues with
a secondary search using Google Scholar. More specifically, we used Google
Scholar’s _site_ and _source_ filtering constraint to limit our search only to
the target venues of interest, as described in Section A.1 of the Appendix. We
also used Google Scholar to search the ArXiv repository and filtered the
results using _publication venue_ information provided by Semantic Scholar API
(Sem, [n. d.]).

The second to last column in Table 1 shows the number of hits for each
publication venue identified by this search. In total, using this procedure,
we identified 4,429 papers; after removing duplicated, we obtained a set of
3,089 potentially relevant papers.

###  3.2. Manual Filtering

Next, we manually classified papers identified in the automated search into
relevant and irrelevant for our survey. To this end, we first randomly
selected a set of 40 papers and used them as a pilot for drafting the
inclusion and exclusion criteria. Four authors of this survey independently
read the abstract, introduction, and conclusions of each paper and classified
it as _relevant_ , _non-relevant_ , or _unsure_. Each author also assigned a
concise label for each of the _relevant_ and _non-relevant_ papers, specifying
the reasons for inclusion/exclusion.

After completing this phase, all authors of this survey met to cross-validate
the decisions, and to consolidate and refine the inclusion/exclusion labels.
All disagreements between the authors (mostly between _non-relevant_ and
_unsure_ papers) were resolved through a joint discussion.

In the second phase, we randomly selected an additional set of 40 papers, to
validate our filtering process. Four authors of this survey, again,
independently read the categorized each of these papers, and all authors met
to discuss the results. While there were disagreements, with a rate of 6.25%,
between the assignment of _non-relevant_ and _unsure_ papers, the
inclusion/exclusion labels were consistent among the raters.

Specifically, we included papers that:

  1. (a)

study the correlation between properties of input data and the adversarial
robustness of resulting model trained on this data; and/or

  2. (b)

present techniques to improve or disrupt a model’s adversarial robustness
through explicitly modifying some properties of the input data or its latent
representation.

We excluded papers that:

  1. (a)

discuss adversarial evasion attacks, but focus on features (Tong et al.,
2019), models (Ghosh et al., 2019; Xu et al., 2019), and training algorithms
(Guo et al., 2021; Jeong and Shin, 2020) rather than data (47.2%);

  2. (b)

discuss aspects of ML that are not related to adversarial robustness but
rather related to accuracy (Ansuini et al., 2019; Zhu and Jin, 2020; Pope et
al., 2021), robustness to distribution shifts not induced by adversaries
(Mintun et al., 2021; Xu et al., 2021), privacy (He and Zhang, 2021; Hu et
al., 2022), and interpretability (Ghorbani et al., 2019; Guo et al., 2018)
(39.7%);

  3. (c)

propose new robustness evaluation metrics (Cheng et al., 2021) or assessment
frameworks (Ling et al., 2019) (2.7%);

  4. (d)

focus on poisoning (Zhang et al., 2019c; Liu et al., 2019) rather than evasion
attacks (2.6%); or

  5. (e)

got matched accidentally and discuss unrelated topics, e.g., literature on
blockchain (Huang et al., 2021), remote access trojan systems (Rezaeirad et
al., 2018), or hardware systems (Tan et al., 2021). These papers appeared in
our search results as “adversarial robustness” is also desirable for non-ML
systems, e.g., blockchain systems need to be robust against adversarial
selfish miners or Denial-of-Service attacks (6%).

As the inclusion/exclusion labels were consistent among the raters, we decided
to proceed to the next phase: distribute the remaining 3,009 papers among the
four authors and categorize them using these inclusion and exclusion criteria.
In this phase, we instructed each rater to conservatively mark as _unsure_
papers with even a slight doubt in categorization.

Following this process, we identified 199 papers as either _relevant_ or
_unsure_. We assigned a second reader to papers marked as _unsure_ ,
summarized each such paper in writing, and held a meeting with all the authors
of this survey to categorize the paper as either _relevant_ or _non-relevant_.
For a select set of papers where we could not confidently reach a decision, we
emailed the paper authors to validate our understanding and decide on the
relevance of the work (all such papers were excluded in the end). Our analysis
resulted in 54 _relevant_ papers. The distribution of these papers by
publication venues is shown in the last column of Table 1.

We further assigned a second reader to each identified relevant paper, to
extract and summarize its main findings. To make sure that we included most of
the relevant works on the topic, we also performed backward snowballing using
the related work sections of the selected papers, which resulted in 3
additional papers, bringing our selection to 57 papers in total. Fig. 3
summarizes our paper selection process.

Figure 3. Summary of the collection and selection of papers.

Figure 4. Breakdown by publication venues.

Figure 5. Breakdown by publication time.

Fig. 5 shows another view on the distribution of publication venues for all 57
papers included in our survey. The majority of the papers (50, 85%) are
published in Machine Learning venues. In fact, only the _Advances in Neural
Information Processing Systems (NeurIPS)_ conference published 19 (around 36%)
of all papers. Seven papers are from the Computer Vision venues and only two
are from the Security venues. We found no relevant papers in the
_Computational Linguistics_ venues.

Fig. 5 shows the distribution of selected papers by their publication year.
The figure suggests that mainstream research communities started to
investigate the impact of training data on adversarial robustness as recently
as 2018. We did not find many relevant papers from 2022 as our search was
conducted in June 2022.

Figure 6. Paper categorization dimensions.

###  3.3. Categorization of Selected Papers

To better classify, discuss, and compare the papers, we proposed a
categorization schema shown in Fig. 6. To construct the schema, we followed an
iterative process similar to the one we used for paper selection: we first
sampled ten papers from the final collection and each of the four authors
independently proposed categorization attributes to describe these ten papers.
The proposed attributes were discussed by all authors while unifying related
attributes, removing redundant ones, and updating labels. We then verified the
applicability of the constructed schema on another set of ten papers and
adjusted it based on a joint discussion. We continued extending the schema
after reading the remaining papers, to ensure its inclusiveness.

The resulting categorization schema, which we further use to analyze and
describe the papers, contains three high-level areas described below. The
detailed categorization of each papers along the attributes of this schema can
be found in Section A.2 of the Appendix.

1\. Problem Setup defines the scope of the proposed approach and the
assumptions authors make in their work. Specifically,

  * $\bullet$

_Target Distribution_ describes the type of data distributions the papers
focus on. The common data distributions studied among the collected papers
include Gaussian mixtures, Bernoulli mixtures, and Uniform distributions.

  * $\bullet$

_ML Model_ focuses on the studied model. It captures the _learning task_ ,
e.g., binary classification, multi-class classification, and regression, and
_classifier type_ , e.g., parametric models, such as, neural networks and non-
parametric models, such as, $k$-NNs.

  * $\bullet$

_Robustness Setting_ records the paper’s definition of adversarial robustness.
This includes the _robustness definition_ sub-category, which refers to how
the authors measure robustness, e.g., error-rate-based radius-based. The
second sub-category, the _attacker’s knowledge_ , reflects the level of
information about the target system that the attacker can exploit: white-box,
grey-box, or black-box. The last sub-category, the _attack_ , characterizes
the _technique_ used to construct the attack, e.g., gradient- or non-gradient
based, and the _perturbation bound_ considered, e.g., the type of $L_{p}$
norm.

2\. Data Property dimension includes the eight data properties we identified
in the collected papers: the _number of samples_ , data _dimensionality_ ,
_distribution_ , _density_ , _concentration_ , _separation_ , _label quality_
, and _domain-specific_ properties, relevant in context of particular
application domains. We introduce and structure the discussion of the surveyed
papers in Section 4 around these properties.

3\. Practicality specifies how to apply the approach or technique introduced
in each paper:

  * $\bullet$

_Applicability_ determines whether specific _quantitative_ metrics are
provided to measure the data properties discussed in the paper or whether
there are any concrete techniques proposed to modify these data properties.

  * $\bullet$

_Explainability_ determines whether the paper focuses on explaining (rather
than establishing) the correlation between data property and robustness.

  * $\bullet$

_Type of Evidence_ records the type of arguments provided by the paper. This
may be a _formal proof_ , an _empirical evaluation_ , or a combination of
both. For cases where an empirical evaluation is performed, we also collect
information about _datasets_ and _classifiers_ used, the applied _training
procedures_ (standard vs. adversarial training), and the _attack techniques_
employed.

In what follows, we present the results of our analysis of the surveyed papers
(Section 4) and discuss our observations (Section 5).

##  4\. Results

We present the results of our analysis by organizing the papers according to
the robustness-related data property they discuss: number of samples,
dimensionality, type of distribution, density, concentration, separation,
label quality, and domain specific properties. Papers that discuss more than
one data property are presented in all corresponding sections. That is, in
what follows, a paper can be discussed in more than one section. Section 4.9
summarizes our findings.

To ease navigation, for each discussed data property, we also include a map
showing how the relevant papers relate to each other via their citation
information. We further annotate each paper with its _applicability_ and
_explainability_ categories. Specifically, we annotate with an A symbol papers
that propose an actionable technique to modify or measure a robustness-related
property; we annotate with E papers that put extra emphasis on explaining the
correlation between a data property and robustness rather than establishing
such a correlation.

###  4.1. Number of Samples

Figure 7. Number of samples illustration.

_Number of samples_ simply means the quantity of samples available in the
training dataset. For the example in Fig. 7, where circles represent training
samples for a two-class dataset, the left dataset has fewer samples than the
right dataset.

The term _sample complexity_ refers to the number of training samples required
to achieve a certain model performance, e.g., 90%, in terms of either robust
or standard generalization. Then, _sample complexity gap_ refers to the
difference in the number of samples required to achieve the same model
performance for robust generalization as for standard generalization.

Papers studying the relationship between the number of training samples and
the robustness of the resulting model are shown in Fig. 8. They can roughly be
divided into 1 papers discussing sample complexity for robust generalization,
2 papers proposing techniques to resolve the sample complexity gap between the
number of samples required to achieve the same level of robust and standard
generalization, and 3 papers proposing techniques to deal with data imbalance,
i.e., unequal number of samples in different classes.

Figure 8. Papers discussing the number of samples.

1

Sample Complexity. Schmidt et al. (Schmidt et al., 2018) observe that the
number of training samples required for robust generalization is larger than
the number of samples required for the equivalent-level standard
generalization, i.e., that there exists a _sample complexity gap_ between the
standard and robust generalization. Specifically, for linear classifiers
trained on a mixture of Gaussian distributions (referred to as _Schmidt’s
Gaussian mixture_ in the remainder of this paper), the authors prove that
standard generalization requires a constant number of samples while
equivalent-level robust generalization requires a number of samples
proportional to the data dimensionality ($O(\sqrt{d})$). The gap in sample
complexity persists for this data distribution in nonlinear classifiers as
well. Yet, the sample complexity gap disappears for nonlinear classifiers
trained on a mixture of Bernoulli distributions; these distributions also need
substantially fewer samples than Gaussian mixtures. The authors conclude that
sample complexity for robust generalization depends on the distribution, even
when the same type of classifiers is considered. Their experimental validation
with the MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky et al., 2009), and
SVHN (Netzer et al., 2011) image datasets shows that MNIST, which is closer to
a Bernoulli mixture, indeed requires a smaller number of training samples to
achieve a reasonable robust generalization than the CIFAR-10 and SVHN
datasets, which are closer to a Gaussian mixture.

In follow-up work, Dan et al. (Dan et al., 2020) provide reasons for why
robust generalization requires more samples than standard generalization,
focusing, again, on Gaussian mixture distributions. Departing from the Signal-
to-Noise ratio (SNR) metric that is based on the distance between two Gaussian
distributions and is known to capture the hardness of standard classification,
the authors propose a new Adversarial SNR (AdvSNR) metric, defined as the
minimum SNR for standard and adversarially perturbed data, to capture the
hardness of robust classification. They then show that, given a dataset of a
particular dimensionality, the number of samples required to achieve the
theoretically optimal, accurate classifier is inversely proportional to SNR.
Likewise, the number of samples required to achieve the theoretically optimal,
robust classifier is inversely proportional to AdvSNR. Because AdvSNR is
always no greater than SNR for a given dataset, it follows that achieving the
same robust generalization as standard generalization requires at least the
same amount of samples. Javanmard et al. (Javanmard et al., 2020) focus on
adversarially-trained linear regression models for standard Gaussian
distributions. The authors show that, when the number of samples is greater
than data dimensionality, there exists a trade-off between adversarial and
standard risks. Moreover, this trade-off improves as the number of samples per
dimension increases.

Bhattacharjee et al. (Bhattacharjee et al., 2021) study the sample complexity
gap for linear classifiers, as a factor of data dimensionality (the number of
features representing samples ) and separation (the distance between samples
from different classes). The authors show that the sample complexity gap is
directly proportional to the dimensionality of the data when the allowed
perturbation radius of adversarial samples is similar to the distance between
classes. However, such a gap no longer exists in well-separated data, when the
perturbation radius is much smaller than the distance between classes.

Similarly, Gourdeau et al. (Gourdeau et al., 2021) show that, for simple
classifiers based on feature conjunctions and $\alpha$-log-Lipschitz
distributions laying on a boolean hyper-cube, the sample complexity is
proportional to the data dimensionality $d$. Specifically, when the
adversarial perturbation size is bounded by $log(d)$, the sample complexity is
polynomial to the dimensionality; when the perturbation size is at least
$log(d)$, the sample complexity becomes superpolynomial to dimensionality.

Cullina et al. (Cullina et al., 2018) give an upper bound on the number of
samples needed for robust generalization for the binary classification problem
with linear classifiers in a distribution-agnostic setup, with $L_{p}$ norm-
bounded adversaries. The authors derive the upper bound using the classifier
VC dimension – a common measure of the capacity and the expressive power of
the classifier, shown earlier to be useful to determine the upper limit of
sample complexity for standard generalization (Shalev-Shwartz and Ben-David,
2014). They show that the VC dimension for learning adversarially-robust
models remains the same as that for learning accurate models, which means that
the upper bound of sample complexity is identical for standard and robust
generalization in this setup. However, the authors demonstrate that this
conclusion does not generalize to other types of classifiers and types of
adversaries.

2

Techniques for Resolving the Sample Complexity Gap. As the number of labeled
samples required to achieve robust generalization could be large and not
readily available, researchers explore cheaper alternatives, such as,
unlabeled data and generated (fake) data. Uesato et al. (Uesato et al., 2019)
and Carmon et al. (Carmon et al., 2019) concurrently proposed to use pseudo-
labeling (Scudder, 1965) – a process of assigning labels to unlabeled samples
using a classifier trained on a set of labeled samples, assessing the
effectiveness of their approaches on Schmidt’s Gaussian mixture. The main
result of both works is that closing the sample complexity gap requires a
number of unlabeled samples proportional to the dimensionality of the data,
albeit with a higher quantity than for the labeled samples, likely due to the
“noise” in generating labels. The main difference between the works is that
while Uesato et al. show that in their setup (a specific linear classifier)
the quantity of the required unlabeled samples only depends on data
dimensionality, Carmon et al. (Carmon et al., 2019) use a less restrictive
setup and show that the quantity of unlabeled samples also depends on the
original sample complexity for standard generalization. Both of these works
empirically evaluate the effectiveness of their proposed approaches on the
CIFAR-10 and SVHN datasets showing that unlabeled data could be a much cheaper
alternative to labeled data for enhancing the robustness of models.

Najafi et al. (Najafi et al., 2019) note that the biggest risk of using a
mixture of labeled and unlabeled datasets for learning adversarially robust
models is the uncertainty in sample labels. Given an estimate of the quality
of pseudo-labels, the authors derive the minimum ratio between labeled and
unlabeled samples required to avoid the additional adversarial risk induced by
label uncertainties.

Instead of using unlabeled data, which might also be hard to find, Gowal et
al. (Gowal et al., 2021) suggest using Generative Adversarial Networks (GANs)
to generate labeled data. The authors show that GANs are more effective than
other methods, e.g., image cropping, when producing additional samples. This
is because such models result in a more diverse dataset, which is beneficial
for increasing robust accuracy. Using images from CIFAR-10, CIFAR-100
(Krizhevsky et al., 2009), SVHN, and Tiny Images Dataset (Torralba et al.,
2008), the authors show that their proposed approach can significantly
increase robust accuracy without the need for additional real samples.

3

Data Imbalance. Wu et al. (Wu et al., 2021) analyze adversarial robustness of
DNNs on long-tail distributions: setups where the training data contains a
large number of classes with few samples. They show that robust generalization
is harder to achieve on such distributions and compare the performance of
multiple adversarially trained classifiers that use learning algorithms
specifically designed for such setups. The comparisons show that scale-
invariant classifiers (Wang et al., 2018b; Pang et al., 2020b) result in
higher robust accuracy as they avoid assigning smaller weights to minority
classes, which, in turn, promotes robust generalization by reducing bias in
the decision boundary.

###  4.2. Dimensionality

Figure 9. Dimensionality illustration.

(a) Actual.  (b) Intrinsic.

Figure 10. Actual and intrinsic dimensionality.

Figure 11. Papers discussing dimensionality.

_Dimensionality_ refers to the number of features used to represent the data,
e.g., features $f_{1}$, $f_{2}$, $f_{3}$ in Fig. 10. For illustration
purposes, we show a dataset with a dimensionality of three on the left-hand
side of the figure and a dataset with a dimensionality of one on the right-
hand side. _Intrinsic dimensionality_ refers to the number of features used in
a minimal representation of the data. Fig. 10 shows an example of a case where
the intrinsic dimensionality is smaller than the actual dimensionality: the
samples in Fig. 10a are lying on a three-dimensional “swiss roll”.
“Unwrapping” the roll into a plain sheet, as shown in Fig. 10b, makes it
possible to distinguish between the samples using only two dimensions.

Papers studying the relationship between data dimensionality and adversarial
robustness are shown in Fig. 11. We divide them into papers 1 characterizing
the hardness of robust generalization due to high dimensionality, 2 suggesting
robust model types and configurations for high-dimensional data, 3 discussing
the impact of high dimensionality on existing defense techniques, and 4
utilizing dimensionality reduction techniques for improving robustness.

1

Hardness of Robust Generalization. A number of authors show that adversarial
examples are inevitable in high-dimensional space. Specifically, Gilmer et al.
(Gilmer et al., 2018) prove this for a synthetic binary dataset composed of
two concentric multi-dimensional spheres (a.k.a., hyperspheres) in high-
dimensional space ($>$100), showing that samples are, on average, closer to
their nearest adversarial examples than to each other. They also prove that
the adversarial risk of a model trained on this dataset only depends on its
standard accuracy and dimensionality. A similar result is shown by Diochnos et
al. (Diochnos et al., 2018), for a uniformly distributed boolean hypercube
dataset, and Shafahi et al. (Shafahi et al., 2019), for unit-hypersphere and
unit-hypercube datasets.

Another line of work analyzes the effect of dimensionality on the robustness
of specific types of classifiers. In particular, Simon-Gabriel et al. (Simon-
Gabriel et al., 2019) study feedforward neural networks with ReLU activation
functions and He-initialized weights, showing that a higher input
dimensionality increases the success rate of adversarial attacks, regardless
of the topology of the network. The authors, however, demonstrate that
regularizing the gradient norms of the network decreases the impact of the
input dimension on adversarial vulnerability, thereby improving model
robustness on high-dimensional inputs. Daniely et al. (Daniely and Schacham,
2020) study the effect of dimensionality on ReLU networks with random weights
and with layers having decreasing dimensions. Like Simon-Gabriel et al., the
authors prove that the robustness of ReLU networks degrades proportionally to
dimensionality.

Amsaleg et al. (Amsaleg et al., 2021) focus on $k$-NNs and other non-
parametric models that base predictions on the proximity of samples. The
authors use the Local Intrinsic Dimensionality (LID) metric to represent the
intrinsic dimensionality in the neighborhood of a particular sample $x$. The
authors build up on the observation that a high LID implies that there are
more samples in close proximity of $x$ (as, otherwise, a more sparse
neighborhood could be encoded in fewer dimensions). Thus, it is possible to
arbitrarily change the neighborhood ranking of the nearest neighbor of $x$
using a small perturbation. As predictions of proximity-based models are based
on the nearest neighbor ranking, the adversarial risks increase in this setup.

All the aforementioned works are also in agreement with a number of papers
discussed in Section 4.1, i.e., (Dan et al., 2020; Bhattacharjee et al., 2021;
Gourdeau et al., 2021), which show, in their respective settings, that sample
complexity for robust generalization is proportional to dimensionality.

2

Model Configuration and Selection. Wang et al. (Wang et al., 2018a) prove that
the optimal $k$ for producing robust $k$-NN classifiers depends on the
dimensionality $d$ and number of samples $n$ of the given dataset
($k=\Omega(\sqrt{dn\text{ log}(n)})$). However, they note that for high-
dimensional data, the optimal $k$ might be too large to use in practice. The
authors thus focus on improving the robustness of 1-NN algorithms through
sample selection, showing the effectiveness of their approach on the Halfmoon,
mnist 1v7, and abalone datasets.

Yin et al. (Yin et al., 2019a) show that transferring a robust solution found
on training data to test data gets more difficult as the dimensionality of
data increases. However, constraining the classifier weights mitigates this
problem. Specifically, the authors prove that constraining the weights by
$L_{p}$ norm, for $p>1$, leads to a performance gap between training and test
data that has a polynomial dependence on dimensionality; when the weights are
constrained by $L_{1}$ norm, the performance gap has no dependence on
dimensionality.

Carbone et al. (Carbone et al., 2020) study neural networks, showing that
adversarial vulnerability arises due to the gap between the actual and
intrinsic dimensionality, a.k.a., degeneracy. The authors show that
adversarial example generations in high-dimensional degenerate data can be
performed by using gradient information of a neural network, to move the
samples in the direction normal to the data manifold. As such, example
generation exploits the additional dimensions without changing the “semantics”
of the perturbed sample. The authors then show that Bayesian Neural Networks
are more robust than other neural networks to gradient-based attacks: due to
their randomness, they make gradients less effective for crafting attacks.

3

Influence of Dimensionality on Defense Techniques. High dimensionality also
poses challenges to defense techniques that aim to improve robustness.
Specifically, Blum et al. (Blum et al., 2020) focus on randomized smoothing –
a technique that improves robustness by generating noisy instances of a
(possibly perturbed) sample and then making predictions for the sample based
on an aggregation of predictions for its noisy instances. The authors show
that the amount of noise required to defend against $L_{p}$ adversaries, for
$p>2$, is proportional to dimensionality. They further demonstrate that, for
high-dimensional images, randomized smoothing indeed fails to generate
instances that preserve semantic image information. In a similar line of work,
Kumar et al. (Kumar et al., 2020) show that the certified radius decreases as
the dimensionality increases when using randomized smoothing for certifying
robustness for a given $L_{p}$ radius.

Adversarial training – a defense technique that improves model robustness by
adaptively training a model against possible adversarial examples – often
incurs a trade-off between standard and adversarial accuracy (Tsipras et al.,
2019; Zhang et al., 2019b): optimizing for high robust accuracy results in a
drop in standard accuracy and vice versa. Mehrabi et al. (Mehrabi et al.,
2021) build up on the work of Javanmard et al. (Javanmard et al., 2020),
discussed in Section 4.1. That work showed that, for a finite number of
training samples, the trade-off between adversarial and standard accuracy
improves as the number of samples per dimension increases. The authors further
extend this result for unlimited training data and computational power,
observing that, for an unlimited number of training samples, the trade-off
between adversarial and standard accuracy improves as the dimension of the
data decreases.

Data augmentation is another common defense technique that aims to improve
robustness of a model by creating perturbed samples at radius $r$ from a
certain subset of original samples in training data. Rajput et al. (Rajput et
al., 2019) prove, for linear and certain nonlinear classifiers, that the
number of augmentations required for robust generalization depends on the
dimensionality of data, i.e., it is at least linearly proportional to
dimensionality for any fixed radius $r$. Thus, data augmentation becomes more
expensive for high-dimensional data.

4

Reducing Dimensionality. Following the idea that the gap between the actual
and intrinsic dimensionality contributes to adversarial vulnerability, Awasthi
et al. (Awasthi et al., 2020) propose to use Principal Component Analysis
(PCA) (Jolliffe, 2002) to decrease the dimensionality of data before applying
randomized smoothing. As a result, a larger amount of noise can be injected to
perturb samples, thus improving robustness without compromising accuracy. The
authors apply the proposed ideas to image data, showing that the combination
of PCA and randomized smoothing is more beneficial than using randomized
smoothing alone. Weber et al. (Weber et al., 2020) show, for hierarchical
data, that changing the representation from Euclidean to hyperbolic space
reduces the dimensionality without sacrificing semantic information embedded
in the input data.

###  4.3. Distribution

Figure 12. Distribution illustration.

Figure 13. Papers discussing distribution.

_Distribution_ refers to a function that encodes how samples lie in space,
usually by giving the probabilities of their occurrence in particular regions.
Common types of distributions, such as uniform, Bernoulli, and Gaussian are
introduced in Section 2.3. Fig. 13 shows examples of datasets that follow a
Gaussian distribution (left) and a Bernoulli distribution (right). The term
_variance_ refers to a measure of dispersion that takes into account the
spread of all data points in a dataset. Specifically, the _variance of a
distribution_ measures the dispersion of samples from the mean; _feature
variance_ measures the dispersion of samples over a particular feature only.
We say that a distribution satisfies _symmetry_ when distributions on either
side of the mean mirror each other.

Papers that discuss how distribution properties, including variance and
symmetry, influence models’ robustness are shown in Fig. 13. They can be
categorized into: 1 papers showing that model robustness depends on the
underlying data distribution, 2 papers identifying properties of distributions
that improve robustness, and 3 papers introducing techniques to transform
distributions into ones that are more optimal for robustness.

1

Type of Distributions. As discussed in Section 4.1, Schmidt et al. (Schmidt et
al., 2018) prove, for nonlinear classifiers, that a mixture of Gaussian
distributions incurs higher sample complexity for robust generalization than a
mixture of Bernoulli distributions. Likewise, Ding et al. (Ding et al., 2019)
show that a distribution shift alone can affect robust accuracy while
retaining the same standard accuracy. Specifically, the authors prove that
uniform data lying on a unit cube results in more robust models than uniform
data lying on a unit sphere. They further experiment with MNIST and CIFAR-10
datasets, applying existing semantically lossless transformations, namely
_smoothing_ and _saturation_ , to cause the distribution shift. The results of
this experiment show that robustness decreases gradually when transforming
MNIST from a unit-cube-like to a unit-sphere-like distribution and increases
for CIFAR-10 when going the opposite way; in both cases, the models retain
their standard accuracy.

Fawzi et al. (Fawzi et al., 2018) study the robustness of data distributions
modeled by a smooth generative model – a type of generative model that maps
samples from input space to output space while preserving their relative
distances, e.g., to compress data. The authors show that smooth generative
models with high-dimensional input space produce data distributions that make
any classifier trained on this data inherently vulnerable. The authors
conclude that non-smoothness and low input space dimensionality are desirable
when modeling data with generative models.

2

Properties of Distributions. Izmailov et al. (Izmailov et al., 2018) show
that, in a binary classification setting, features with small variance in both
classes and means close to each other cause adversarial vulnerability.
Moreover, a feature with a small variance in one class can still cause
vulnerability even if the means of this feature in both classes are farther
separated but the second class has a larger feature variance. Intuitively,
that is because models tend to assign non-zero weights to such features, which
can be leveraged by attackers to shift the classification into the wrong
class. That is, even small perturbations in such features can shift data
points to another class. To increase robustness, the authors suggest removing
such features, either based on domain knowledge or based on feature evaluation
metrics, such as, mutual information (Shannon, 1949).

Similarly, Lee et al. (Lee et al., 2020) prove that decreasing feature
variance in individual classes can increase robustness for Schmidt’s Gaussian
mixtures. These mixtures have equivalent feature variances for all classes and
separated means. In this setting, low feature variance implies that the
feature has a strong correlation with the class and perturbing this feature
unlikely result in vulnerability (i.e., will likely result in a semantically-
meaningful change). However, even when features have low variance, if these
features are non-robust (Ilyas et al., 2019), i.e., hold no semantic
information, and have a smaller variance in the training data than in the
underlying true population, they will still cause adversarial vulnerability as
adversarially trained models tend to overfit to them. As a countermeasure, the
authors propose a label-smoothing-based data augmentation technique which uses
continuous instead of discrete values for labels and acts like a
regularization method that prevents the model from overfitting to such
features.

Figure 14. Asymmetrical Dataset.

Richardson and Weiss (Richardson and Weiss, 2021) claim that adversarial
vulnerability can be caused by sub-optimal data distributions and/or sub-
optimal training methods. The authors define synthetic binary datasets (of
images) that use Gaussian distributions with separated means and say that a
dataset is symmetric if and only if classes have the same variance. They
further prove that even the optimal classifier is non-robust when the
underlying dataset has strong asymmetry, as in the example in Fig. 14. If the
dataset is symmetric the optimal classifier is provably robust, even though a
sub-optimal training method can still cause vulnerability when trained on this
dataset.

3

Transforming Distributions. Both Pang et al. (Pang et al., 2018) and Wan et
al. (Wan et al., 2022) change the latent DNN feature representation to be
similar to Gaussian mixtures. Specifically, Pang et al. (Pang et al., 2018)
show that, for Linear Discriminant Analysis (LDA) classifiers trained on
Gaussian mixtures, the robustness radius of LDA is proportional to the
distance between Gaussian centers. The robustness of LDA is further maximized
for symmetric Gaussian mixtures. The authors thus modify the DNN loss function
to create a latent feature representation similar to symmetric Gaussian
mixtures and further replace the last layer of DNN from commonly used Softmax
Regression (Cramer, 2002) to LDA. To achieve the desired robustness radius,
the authors compute the coordinates of the desired Gaussian centers (as a
function of the number of classes and the dimensionality of the input data)
and feed this data to the loss function. Departing from the assumption that
symmetric Gaussian mixtures are advantageous for the underlying model
robustness, Wan et al. (Wan et al., 2022) modify the DNN loss function to
compute the centers of the Gaussians directly while generating symmetric
Gaussian feature distributions.

###  4.4. Density

Figure 15. Density illustration.

Figure 16. Papers discussing density.

_Density_ measures the closeness of samples in a particular bounded region.
For continuous data, it is mathematically described by the probability density
function (PDF), which gives the probability for a variable to take a certain
range of values. For discrete data, it is described as the probability mass
function (PMF), which gives the probability for a variable to take a
particular value. We say that an area is dense when there is a high
probability that random samples lie in the same area, i.e., close to each
other. For example, the dataset on the right-hand side of Fig. 16 contains a
larger number of samples in close proximity and, thus, is more dense than the
dataset on the left-hand side of the figure. Furthermore, density can be
defined over samples from one class, in which case, it is referred to as
_class density_.

Papers that study how density influences adversarial robustness are shown in
Fig. 16. They can roughly be divided into 1 papers discussing the effect of
class density on robustness and 2 papers proposing attacks and defenses using
density information.

1

Effect of Class Density on Robustness. Shafahi et al. (Shafahi et al., 2019)
show that datasets with a higher upper bound of class density lead to better
robustness. In particular, for image datasets, the authors show that images of
lower complexity, e.g., with simple objects on plain backgrounds, have a
higher correlation among adjacent pixels. Datasets comprised of such images
have a higher density, as pixel values are more frequently repeated, and,
thus, lead to better robustness. The authors confirm this observation by
showing that classifiers trained on MNIST, which has a lower image complexity
and thus higher density than CIFAR-10, are more robust than those trained on
CIFAR-10. Furthermore, the authors state that class density is a better
predictor of robustness than dimensionality: even after up-scaling MNIST to
the same dimensionality as CIFAR-10, it still has a higher density and thus
results in more robust classifiers than CIFAR-10.

2

Attacks and Defenses Using Density. Several works note that adversarial
examples are commonly found in low-density regions of the training dataset, as
models are unable to learn accurate decision boundaries using a small number
of samples from these regions. Zhang et al. (Zhang et al., 2019a) propose an
attack strategy that retrieves candidate samples from low-density regions and
perturbs them to generate adversarial examples. The authors demonstrate that,
even after adversarial training, models will not be robust to adversarial
attacks that target these low-density regions.

A similar finding by Zhu et al. (Zhu et al., 2022) suggests that adversarial
examples from low-density regions have a higher probability of being
transferable between different models trained on the same dataset. Based on
this observation, the authors propose an attack that increases the
transferability of adversarial examples by identifying perturbation directions
that maximize both the adversarial risk and the alignment with the direction
of density decrease for the underlying data distribution, i.e., move samples
towards regions with lower density.

Departing from the same idea that low-density regions are prone to adversarial
attacks, Song et al. (Song et al., 2018) focus on creating a defense mechanism
that uses generative models to detect if a sample comes from a low-density
region when making predictions. If so, the sample is moved towards a more
dense region of the training data as a “purification” step.

To harden models directly, Pang et al. (Pang et al., 2020a) propose a new loss
function for DNNs, to learn dense latent feature representations. The authors
first show that the commonly used Softmax Cross-Entropy loss function induces
sparse representations (i.e., with low class density), which lead to
vulnerable models. This is because a low number of samples in close proximity
to each other prevent a model from learning reliable decision boundaries. They
then propose a loss function that explicitly encourages feature
representations to concentrate around class centers; like in their earlier
work (Pang et al., 2018), the authors compute the coordinates of the desired
class centers (as a function of the number of classes and the dimensionality
of the input data) to maximize the distances between the centers. The authors
demonstrate that the proposed approach improves robustness under both standard
and adversarial training.

###  4.5. Separation

Closely related to density, _separation_ refers to the distance between
classes. Fig. 18 shows examples of not well-separated (top) and well-separated
(bottom) datasets. Intuitively, learning an accurate classifier is easier when
data is well-separated as samples from different classes are farther apart and
samples from the same class are closer together. Different metrics to quantify
separation include the _optimal transport distance_ , which computes the
minimum distance required to transport samples from one class to another, and
_inter-class distance_ , which computes distance between samples in different
classes.

Figure 17. Separation illustration.

Figure 18. Papers discussing separation.

Papers that discuss data separation in relation to adversarial robustness are
shown in Fig. 18. They can roughly be divided into 1 papers showing the effect
of separation on robustness and 2 papers proposing techniques to promote
separation and, thus, increase robustness.

1

Effects of Separation. Bhagoji et al. (Bhagoji et al., 2019) calculate lower
bounds for adversarial risk in a binary classification setting using the
optimal transport distance. The authors show that the lower bound decrease as
the distance between the two class distributions increases, i.e., a classifier
becomes more robust with better separation. Based on this result, they
estimate the minimum adversarial risks for image datasets, like MNIST and
CIFAR-10, showing that the theoretically calculated risks are lower than the
empirical values achieved by the state-of-the-art defense models. The authors
conclude that there is still room for improving existing techniques.

Pydi and Jog (Pydi and Jog, 2020, 2021) arrive at a similar conclusion – that
robustness improves as separation between classes increases. The authors
further focus on datasets with simple univariate distributions, such as
Gaussian and uniform. They propose a technique to construct classifiers that
can achieve the optimal, lowest possible adversarial risk for a given
separation between classes. The main idea behind this technique is to analyze
the optimal way to transport samples from one class to another (which
represents the smallest perturbation needed to create adversarial examples)
and further use this information to identify the decision boundary that
induces the maximal distance required to transport sample between classes.
That is, the approach maximizes the distance between samples of each class and
the decision boundary, resulting in an optimally robust classifier.

Bhattacharjee et al. (Bhattacharjee and Chaudhuri, 2020) prove that certain
non-parametric models, such as k-NNs, are inherently robust when trained on a
large number of well-separated samples. This is because these classifiers make
predictions based on neighborhoods and well-separated data ensures that
samples in close proximity to each other share the same labels. In their later
work, discussed in Section 4.1 (Bhattacharjee et al., 2021), the authors show
that, in well-separated data, robust accuracy is independent of dimensionality
and a robust linear classifier can be learnt without the need for a large
number of training samples. This result shows that adversarial vulnerability
can be efficiently tamed by increasing separation.

2

Techniques to Promote Separation. Yang et al. (Yang et al., 2020) propose a
sample-selection-based technique to improve adversarial robustness of non-
parametric models by increasing the separation among the training data. In
particular, as non-parametric models tend to learn complex decision boundaries
when the training samples from different classes are close to each other, the
authors propose to remove the smallest subset of samples so that all pairs of
differently labeled samples remain separated even when perturbed by the
maximum perturbation size. Wang et al. (Wang et al., 2018a), already discussed
in Section 4.2, focus on improving robustness of 1-NN classifiers. As
robustness of such classifiers is optimized by ensuring the opposite classes
being far apart and test points ending up in close proximity to their
respective training data, the authors propose retaining the largest subset of
training samples that are (i) well-separated and (ii) in high agreement on
labels with their nearby samples (a.k.a., highly confident). The authors show
that their approach outperforms adversarially trained 1-NNs.

Another line of works proposes techniques to enforce separation in latent
representations to improve the adversarial robustness of DNNs. Specifically,
Mustafa et al. (Mustafa et al., 2019) attribute the cause of adversarial
vulnerability to close proximity of classes in latent space. Hence, they
propose a loss function to learn intermediate feature representations that
separate different classes into convex polytopes, i.e., polyhedra in higher
dimensions, that are maximally separated. Bui et al. (Bui et al., 2020)
observe that the adversarial vulnerability of DNNs arises from a large
difference in intermediate layer values between clean and adversarial data.
They thus propose to modify the loss function so that it results in an
intermediate latent representation that has high similarities between clean
and their corresponding adversarial samples, while promoting large inter-class
distance and small intra-class distance. In addition, the proposed loss
function aims to increase margins from class centers to decision boundaries.
Likewise, Pang et al. (Pang et al., 2018) and Wan et al. (Wan et al., 2022)
discussed in Section 4.3, as well as Pang et al. (Pang et al., 2020a)
discussed in Section 4.4, improve DNN robustness by separating centers of the
produced latent distributions and, thus, increasing the separation between
classes.

Yang et al. (Yang et al., 2021a) propose a representation-learning technique
to learn feature representations that bring samples of class $C$ and
adversarial examples generated for $C$ into close proximity while separating
the samples of $C$ from both (i) adversarial examples generated for other
classes and misclassified as class $C$ and (ii) samples from other classes.
These separations are enforced by the loss function proposed by the authors.
The authors show that their approach improves the resulting model robustness
compared with standard DNNs.

Garg et al. (Garg et al., 2018) propose an approach to generate well-separated
features for a dataset using graph theory. Specifically, they convert the
input dataset into a graph, where vertices correspond to the input data points
and edges represent the similarity between the data points (e.g., calculated
using Euclidean distance). The authors prove that features extracted using the
eigenvectors of the Laplacian matrix capturing the structure of the graph will
have significant variation across the data points, while being robust to small
perturbations. These qualities make them good candidates for robust features.
The authors then demonstrate that a linear model trained on the MNIST dataset
with 20 features generated using their approach is more robust to
$L_{2}$-norm-based transfer attacks than a fully connected neural network
trained on the full pixel values of the MNIST dataset.

###  4.6. Concentration

Figure 19. Concentration illustration.

Figure 20. Papers discussing concentration.

_Concentration_ of a dataset refers to the “concentration of measure”
phenomenon from measure theory (Talagrand, 1996a). In a nutshell,
concentration is the minimum value of a measured function over all valid
measurable sets, after an $\epsilon$-expansion. More formally, for a metric
probability space $(\mathcal{X},\mu,d)$ with instance space $\mathcal{X}$,
probability measure $\mu$, and distance metric $d$, the concentration function
$h$ is defined as: $h(\mu,\alpha,\epsilon)=$
inf${}_{A\subseteq\mathcal{X}}\\{\mu(A_{\epsilon}):\mu(A)\geq\alpha\\}$ for
any $\alpha\in(0,1)$ and $\epsilon\geq 0$ (Mahloujifar et al., 2019b). Here
$A_{\epsilon}$ refers to the $\epsilon$-expansion of set $A$, defined as
$A_{\epsilon}=\\{x:d(x,A)\leq\epsilon\\}$.

Fig. 20 shows how the concentration of measure phenomenon can be used to
determine the classification error after adversarial perturbation. By modeling
the classification error set as measurable set $A$ and adversarial errors from
perturbation budget $\epsilon$ as $A_{\epsilon}$, one can relate the
concentration of the data to the minimum adversarial risk for any imperfect
classifier with error rate $\mu(A)\geq\alpha$. Using this formulation, a
dataset being highly concentrated implies that, for some non-zero initial
error, the minimum adversarial risk from an $\epsilon$-expansion on the error
set is very large. We refer to such datasets as datasets with low _intrinsic
robustness_ – a measure that represents the maximal achievable robustness for
any classifier on a dataset.

Fig. 20 shows the papers that relate data concentration to adversarial
robustness. They can roughly be divided into: 1 papers discussing the effect
of concentration on robustness and 2 papers proposing techniques to estimate
robustness through calculating concentrations.

1

Effect of Concentration on Robustness. A number of papers prove the
inevitability of adversarial examples using the concentration of measure
phenomenon. In particular, Dohmatob (Dohmatob, 2019) investigates datasets
conforming to uniform, Gaussian, and several other distributions that satisfy
$W_{2}$ transportation-cost inequality (Talagrand, 1996b). The author proves
that data distributions satisfying such inequality have high concentration,
which results in a rapid robustness decrease, beyond a critical perturbation
size – a value that depends on the standard error of the classifier and the
natural noise level of the dataset, which, in turn, defined as the largest
variance in the case of Gaussian distribution. Even though MNIST might not
satisfy the $W_{2}$ transportation-cost inequality, the author experiments
with this dataset, observing a sudden drop in robustness as the perturbation
size increases. As such, the author suggests that the MNIST dataset may also
have high concentration and be governed by the concentration of measure
phenomena.

Mahloujifar et al. (Mahloujifar et al., 2019a) focus on a collection of data
distributions with high concentration called Lévy families (Lévy, 1951), which
include unit sphere, unit cube, and isotropic n-Gaussian (i.e., Gaussian with
independent variables with the same variance). The authors prove that
classifiers trained on such highly-concentrated data distributions admit
adversarial examples with perturbation $\mathcal{O}(\sqrt{d})$ for
dimensionality $d$. This implies that a relatively small perturbation can
mislead model trained on these data distributions with high dimensional
inputs.

2

Techniques to Estimate Robustness Through Concentrations. Several approaches
utilize the connection between concentration and adversarial risk to estimate
the intrinsic robustness of datasets through calculating their concentrations.
Mahloujifar et al. (Mahloujifar et al., 2019b) are the first to propose an
approach for estimating dataset concentration using subsets of samples.
Specifically, the authors propose a technique that searches for the minimum
expansion set based on a collection of subsets carefully chosen according to
the perturbation norm (e.g., a union of balls for $L_{2}$ norm). They prove
that the estimated concentration value converges to the true value for the
underlying distribution as the sample size and the quality/representativeness
of the chosen subsets increase. The authors apply their approach to estimate
the maximum achievable robustness for the MNIST and CIFAR-10 datasets,
observing a gap between the derived theoretical values and values observed
empirically by the state-of-the-art models.

In follow-up work, Prescott et al. (Prescott et al., 2021) propose an
alternative approach to estimate concentration based on half space expansion
using _Gaussian Isoperimetric Inequality_ for the $L_{2}$ norm (Borell, 1975).
The authors further generalize their results to $L_{p}$ norms, where $p\geq
2$. Compared with Mahloujifar et al. (Mahloujifar et al., 2019b), their
approach yields higher achievable robustness on MNIST and CIFAR-10, revealing
a larger gap between the theoretical robustness and the state-of-the-art. As
the theoretically achievable robustness derived from a concentration
perspective is shown to be high, the authors suggest that factors other than
concentration may contribute to this gap.

Zhang and Evans (Zhang and Evans, 2022) assume access to information about
label uncertainty, i.e., function that assigns the level of label
uncertainties for any data point. Such function can use, e.g., labeling
results from multiple human annotators or confidence scores from an ML
classifier. The authors suggest that considering regions with high label
uncertainty can guide the concentration estimations as these are the regions
where a classifier is more likely to make mistakes and be vulnerable to
attacks. They thus propose an approach to estimate concentration by
identifying the smallest set after $\epsilon$-expansion with an average
uncertainty level greater than a pre-set value. The evaluation results show
that the maximum achievable robustness estimated with their approach is closer
to the robustness values observed for CNN models on the CIFAR-10 dataset than
in any of the aforementioned works, implying that the room for improvement is
smaller than assumed earlier.

###  4.7. Label Quality

Figure 21. Label noise illustration.

Figure 22. Papers discussing label quality.

_Label quality_ refers to the correctness and informativeness of the set of
labels assigned to a training dataset. Label correctness or, inversely, the
presence of inaccurate labels (shown as the highlighted dots on the left-hand
side of Fig. 22) is typically referred to as _label noise_. The granularity of
the labels is typically referred to as _label informativeness_. Papers that
discuss the relationship between label quality and model robustness are
outlined in Fig. 22.

Mao et al. (Mao et al., 2020) show that training a model simultaneously for
multiple tasks, e.g., to simultaneously locate and estimate the distance of
objects in images (an approach also referred to as _multi-task learning_),
improves robustness. This is because in multi-task learning, a model learns a
shared feature representation by training on data with labels from several
tasks. As a result, perturbations required to attack multiple tasks at the
same time, e.g., to sabotage an autonomous driving system by misleading the
model in both object identification and distance estimation, cancel each other
out. While the authors prove that the model robustness to adversarial attacks
is proportional to the number of tasks that it is trained on, the benefits of
multi-task learning disappear when the concurrently trained tasks are highly
correlated with each other, as it reduces the chances for the perturbations to
cancel each other. The authors further show that training with multiple tasks
also improves model robustness against single-task attacks.

Sanyal et al. (Sanyal et al., 2021) hypothesize that label noise and coarse
labels are the reasons for adversarial vulnerability. The authors prove that,
given a large training set with random label noise, any classifier that
overfits to that set is likely to be vulnerable to adversarial attacks. This
is because overfitting leads to overly complex decision boundaries that leave
more room for attacks, as illustrated in Fig. 23.

(a) Overfit (b) Not overfit.

Figure 23. Influence of overfitting to label noise.

The authors also demonstrate that adversarial risk increases as the level of
label noise increases. Defense mechanisms, such as early stopping and
adversarial training, enhance robustness by preventing models from overfitting
to noisy samples. In the absence of label noise, using coarse labels (e.g.,
labels for the entire class of dogs rather than labels for each individual dog
breed) results in “sub-optimal” latent feature representations and also
contributes to the adversarial vulnerability.

###  4.8. Domain-Specific

Papers in this category provide insights into the correlation between domain-
specific data properties and adversarial robustness. Among our collected
papers, all the domain-specific studies focused on the same topic:
understanding the adversarial vulnerabilities of image classifiers based on
image _frequency_ – how fast the intensity of pixel values changes with
respect to space (i.e., images with intensive color changes have high
frequency). As shown in Fig. 25, the skin of a zebra has higher image
frequency than a horse, because of the black-and-white stripes. Papers
studying image frequency are listed in Fig. 25. They can be roughly divided
into: 1 papers discussing the influence of frequency distribution on the model
adversarial robustness, and 2 papers explaining adversarial vulnerabilities
using perceptual differences between human and models.

Figure 24. Image frequency.

Figure 25. Papers discussing domain-specific properties.

1

Image Frequency Distribution. Yin et al. (Yin et al., 2019b) show that
frequency distribution of the inputs generated from data augmentation
techniques explains the resulting model sensitivity to adversarial attacks. In
particular, the authors show that Gaussian augmentation, as well as
adversarial training techniques that rely on data augmentation, generate
perturbations with high-frequency components. Augmenting training data with
those augmented inputs makes the resulting model more robust against
perturbations in high-frequency domains while, at the same time, more
vulnerable to perturbations concentrated in low-frequency domains. To mitigate
this issue, the authors propose to avoid biasing the model towards/against
certain frequency ranges by increasing the diversity of frequency distribution
in augmentations.

Ortiz-Jimenez et al. (Ortiz-Jimenez et al., 2020) analyze CNN robustness
through the classifier margins along particular frequencies, which the authors
define as the minimal perturbation in that frequency required to change the
model prediction. The authors show that CNN models tend to have smaller
margins along low-frequency vs. high-frequency ranges, likely because, for
most image datasets, one can differentiate classes mainly using features from
low-frequency ranges. This finding implies that models are more sensitive to
attacks that modify low-frequency components. The authors thus suggest
training more robust models that enlarge margins along low-frequency ranges by
augmenting training datasets with perturbations concentrated in those ranges.

2

Perceptual Differences. Wang et al. (Wang et al., 2020) attribute the origin
of adversarial examples to the perceptual differences of humans and CNNs in
frequency ranges. In particular, humans classify images based on low-frequency
components as high-frequency components are not visible to the human eye.
CNNs, on the other hand, are able to ‘see’ the full frequency spectrum which
allows them to exploit high-frequency components for better generalization.
This implies that adversarial examples generated by perturbing high-frequency
components can mislead CNNs while being imperceptible to humans. The authors
show that adversarially robust models depend less on high-frequency components
and propose to use smoother convolutional filters to reduce a model’s
attention to these components.

Unlike Wang et al. (Wang et al., 2020), Chen et al. (Chen et al., 2021) posit
that the adversarial vulnerability of CNNs results from their over-reliance on
amplitude information of images – the magnitude of the different frequencies
in the image. The authors show that replacing the amplitude information of an
image with information from another image can successfully mislead CNNs but
not humans, who rather rely on phase information – the locations of the
features, to recognize objects. Based on this observation, the authors propose
to strengthen CNNs’ attention to phase information through a data augmentation
technique that fuzzes amplitude while preserving the same phase information.

###  4.9. Summary of Results

Overall, the surveyed papers are mostly in agreement on how each of the
identified data property influences adversarial robustness. The main findings
are given below.

Number of samples. More training samples are needed for robust than for
standard generalization. For a variety of training setups (i.e., different
types of classifiers and data distributions), the number of training samples
required to achieve robust generalization is proportional to the
dimensionality of the training data. Unlabeled samples or generated data can
be used to fulfill the need for more samples needed for robust generalization,
i.e., to close the sample complexity gap. Class imbalance, i.e., having an
imbalanced number of samples across different classes, hurts robust
generalization due to the model bias towards over-represented samples.

Dimensionality. Dimensionality captures the size of the feature set. Higher
dimensionality correlates with higher adversarial risk, worse standard-to-
adversarial risk trade-off, difficulty in robustness certification, and
difficulty in applying common defense techniques. This is because adversarial
attacks can exploit the excessive dimensions to construct adversarial
examples.

Distribution. Some data distributions are more robust than others, e.g.,
mixtures of Bernoulli distributions are more robust than mixtures of Gaussian
distributions. Learning feature representations that resemble robust
distributions can improve robustness.

Density. Density reflects the closeness of samples in a particular bounded
region (inter-class distance). Adversarial examples are commonly found in low-
density regions of data, where samples are far apart from each other. This is
because models cannot accurately learn decision boundaries near low-density
regions due to the small number of samples available. As such, high data
density for each class correlates with lower adversarial risk.

Separation. Separation characterizes the distance of samples from different
classes to each other (inter-class distance). Greater separation between
classes decreases adversarial risk as it is harder to generate perturbation
that will cross the boundaries between classes. Most papers that provide
techniques for improving separations, e.g., by feature selection, also ensure
that it does not come at the expense of decreasing density, as these two
concepts are closely related.

Concentration. Given a function defined over a non-empty set, concentration
(from the phenomenon of concentration of measure (Talagrand, 1996a)) is the
minimum value of the function after expanding the input set by $\epsilon$ in
all dimensions. For example, expanding the set of misclassified samples by a
certain $\epsilon$ gives a set of possible samples that can be misclassified
with an $\epsilon$-size perturbation (candidate adversarial examples).
Concentration, in this case, measures the minimal possible size of this set,
which provides the upper bound of the achievable model robustness. As some
datasets tend to exhibit inherently high concentration, e.g., datasets that
lie on unit hypersphere (Mahloujifar et al., 2019a), achieving high robust
generalization is harder for these datasets. The impact of high concentration
on adversarial robustness is further magnified for high-dimensional data.

Label quality. High label noise correlates with higher adversarial risk. More
specific labels, e.g., “cat” and “dog” instead of “animal”, are more robust
than coarse labels, as they allow the model to extract more distinct features.
Learning for different tasks concurrently, e.g., to simultaneously locate and
estimate the distance of objects in images, improves robustness of the learned
models, as the model can utilize the information from multiple sources of
data.

Domain specific. Image frequency, i.e., the rate of change in pixel value is
shown to be correlated with robustness. Specifically, a diverse distribution
of frequencies in training data results in lower adversarial risk. Most image
datasets have a low image frequency, which results in learned models having
smaller margins and, thus, a lower distance of samples to the decision
boundary for features corresponding to the low-frequency image components.
This increases the risk for adversarial perturbation that utilize these
features.

##  5\. Implications and Future Work

Empirical Evaluation. All but the four papers outlined in Section 4.8 study
domain-agnostic data properties. Yet, the majority of the papers we surveyed
conduct experimental evaluations on image datasets only. Applicability of the
findings and of the proposed approaches to other domains, with different forms
of data, may need further investigation. For example, for datasets with binary
features, which are commonly used in malware detection, one cannot arbitrarily
change feature values to reduce the distance between samples. This further
implies that common distance metrics used to model adversaries in the image
domain, such as $L_{2}$ and $L_{\infty}$, fail to accurately capture the
adversarial threat level in such domains. Hence, future work applying,
adapting, and evaluating the proposed metrics and techniques in other domains
and data types is needed.

Interdependence of Properties. Only a few works in our collected literature
consider multiple data properties simultaneously or establish interdependence
of data properties. For example, Wang et al. (Wang et al., 2018a) and Rajput
et al. (Rajput et al., 2019) find that the number of samples and
dimensionality collectively influence the performance of the resulting model.
Sanyal et al. (Sanyal et al., 2021) study the tolerable amount of label noise
as a function of the dataset density. Such works are very valuable as
adversarial robustness is indeed a result of compounding properties. Yet,
optimizing for multiple properties simultaneously is not always possible. A
productive direction of future work could be to investigate correlations
between different data properties, e.g., the effects of feature dimensionality
reduction approaches on class density and separation.

Additional Data Properties. Existing research on the effects of data on
_standard generalization_ (López et al., 2013; Lorena et al., 2020; Santos et
al., 2022) identified several data properties not discussed in the papers
related to robust generalization that we reviewed. These include the presence
of (i) outliers, i.e., samples that drastically differ from most observed
samples in a dataset, (ii) overlapping samples, i.e., different samples of the
dataset having the same feature representation, and (iii) small disjuncts,
i.e., training samples from the same class forming small disjoint clusters
dispersed throughout the input space (more details are in Section 6).
Investigating the effect of such data properties on the model’s adversarial
robustness could be yet another direction for possible future work.

Generalization. Some works have shown that, under specific assumptions about
the data and model, there is an intrinsic accuracy-robustness trade-off
(Tsipras et al., 2019; Javanmard et al., 2020; Mehrabi et al., 2021). This
implies that achieving robust generalization may come at cost of standard
generalization. However, other works have shown that the effect of several of
the data properties on standard generalization overlaps with their effect on
robust generalization. For example, increasing class density and removing
label noise also increases standard generalization (Raudys and Jain, 1991; He
and Garcia, 2009; López et al., 2013; Frenay and Verleysen, 2014). We believe
that more work is needed to map data-related reasons that contribute to the
accuracy-robustness trade-off.

Simplified Problem Setup. Several studies use a simplified problem setup,
e.g., pure Gaussian data distribution, to provide formal proofs related to the
studied phenomenon. While such work helps advance knowledge and our
understanding of the effects of data on adversarial robustness, additional
work that investigates the generalizability of the findings on realistic
datasets used in practice is needed. For example, assuming uniform data
properties, e.g., same distribution, density, and level of label noise, for
all classes on the training data greatly simplifies the proofs, but is not
common in reality. Likewise, considering only binary classification simplifies
calculations of data separation, which can be calculated by measuring the
distance between the two classes. Yet, in a multi-class setting, one needs to
consider the proximity of data points from multiple classes.

Furthermore, most papers only consider a white-box attack setting, which might
not be realistic in many practical scenarios. Even though a white-box setting
makes it possible to model the worst-case adversary and to provide better
robustness guarantees, it may result in overly pessimistic findings, i.e.,
some data transformations may be robust against black-box attacks while still
be vulnerable to white-box attacks. Thus, future works might look into the
impact of data properties on the different types of attack scenarios.

Quantitative Measure. Literature shows that the lower/upper bound of
adversarial robustness can be determined by the properties of the underlying
data (Bhagoji et al., 2019; Mahloujifar et al., 2019b). Modifying certain
properties of the data can also change the robustness of the resulting
classifier. Hence, the ability to quantitatively measure such data properties
is very valuable. However, some data properties discussed in this survey, such
as, type of distributions and label noise, lack any reliable estimation
techniques. Current work mostly relies on informal comparative analysis, e.g.,
that the MNIST dataset is closer to a Bernoulli mixture data than a Gaussian
mixture because the pixels are concentrated towards black or white.
Quantitatively measuring the degree of similarity between distributions,
although difficult, may be necessary in order to make more accurate
conclusions.

Interestingly, other data properties have multiple, often inconsistent,
measurement techniques, e.g., concentration (Mahloujifar et al., 2019b;
Prescott et al., 2021; Zhang and Evans, 2022), density (Song et al., 2018;
Pang et al., 2020a), intrinsic dimensionality (Amsaleg et al., 2021; Lorena et
al., 2020), and inter-class distances (Ding et al., 2019; Bhagoji et al.,
2019; Pydi and Jog, 2020). For example, the inter-class distances can be
calculated as the total distance required to move the samples from one class
to another (Bhagoji et al., 2019; Pydi and Jog, 2020). It can also be
calculated as the pairwise distances between a pre-defined portion of samples
from different classes, e.g., 10% from each class, that are the closest to
each other (Ding et al., 2019). While the inter-class distance derived through
the first approach is more computationally expensive, the second approach is
more susceptible to outliers as it relies only on a subset of samples close to
each other. Moreover, these metrics might not necessarily correlate with each
other. We believe future research can provide more insights about appropriate
application scenarios for each of the proposed metrics.

Sources of Adversarial Vulnerability. Even when the training data is optimal
for robustness, a sub-optimal training method can lead to adversarial
vulnerability (Richardson and Weiss, 2021). For example, adversarial
vulnerability may arise when the complexity of the classifier does not match
the complexity of the data, e.g., CNNs may achieve lower robustness due to
their complexity than simpler models, such as Kernel-SVMs, on symmetrical data
with well-separated means and similar variances (Richardson and Weiss, 2021).
To alleviate such problems, a few papers propose to select, improve, or
optimize classifiers based on the dimensionality of data (Wang et al., 2018a;
Yin et al., 2019a; Carbone et al., 2020). Similar work that looks at other
properties of data, such as separation and density, could be of value. Future
works could also explore strategies for determining whether the input data
(vs. the model itself) is the dominant cause of adversarial vulnerability.

##  6\. Related work

We divide related work into three categories: (1) surveys on adversarial
robustness and its relation to data properties, (2) surveys that discuss the
influence of data properties on standard generalization, and (3) individual
papers that study non-data-related reasons for adversarial vulnerability.

###  6.1. Surveys on Adversarial Robustness

Numerous existing surveys focus on attack and defense techniques for
adversarial robustness (Biggio and Roli, 2018; Rosenberg et al., 2021; Li et
al., 2021; Maiorca et al., 2019; Demetrio et al., 2021; Liu et al., 2022b, a;
Akhtar and Mian, 2018; Akhtar et al., 2021; Serban et al., 2020; Machado et
al., 2021; Zhang et al., 2020). Only a few of these works mention the
relationship between adversarial robustness and properties of the underlying
data. Specifically, Serban et al. (Serban et al., 2020) observe that
adversarial vulnerability can be caused by an insufficient training sample
size and high data dimensionality. Similarly, Machado et al. (Machado et al.,
2021) mention that the lack of sufficient training data, high-dimensional
data, and high concentration contribute to adversarial vulnerability. Yet,
none of these surveys explicitly collect and analyze work that focuses on the
effects of data properties on adversarial robustness. By explicitly targeting
this topic in our survey, we are able to discuss these findings in detail and
also identify additional relevant data properties not mentioned in previous
surveys, such as, types of distribution, class density, separation, and label
quality.

###  6.2. Influence of Data Properties on Standard Generalization

A number of surveys investigate the influence of data properties on standard
rather than robust generalization. One of the earliest is probably the work of
Raudys and Jain (Raudys and Jain, 1991), who review studies related to the
influence of sample size on binary classifiers, showing that a limited sample
size usually leads to sub-optimal generalization. Bansal et al. (Bansal et
al., 2021) and Bayer et al. (Bayer et al., 2022) also survey papers addressing
the data scarcity problem, focusing in particular on the recent advancements
in data augmentation techniques in the fields of computer vision, security,
and text classification. Their results show that augmentation techniques can
help improve a model’s generalization by reducing the problem of model
overfitting.

Label noise is another aspect of data that influences both standard and robust
generalization. Most works on this topic find that the presence of noisy
labels increases the need for a greater number of training samples and may
result in unnecessarily complex decision boundaries (Frenay and Verleysen,
2014; Song et al., 2022). For example, Frénay and Verleysen (Frenay and
Verleysen, 2014) show that overfitting to label noise greatly degrades a
model’s standard generalization; the same effect has been observed for the
case of robust generalization (Sanyal et al., 2021). Song et al. (Song et al.,
2022) survey the impact of label noise in deep learning, arguing that the
presence of noisy labels is a more serious concern for deep models as they
contain a larger number of parameters which makes them prone to overfitting to
the noise in training data. They mention that adversarial defense techniques,
e.g., adversarial training, are effective against label noise (Zhu et al.,
2021; Fatras et al., 2022) but do not discuss how label noise influences a
deep learning model’s robustness under attacks.

Lorena et al. (Lorena et al., 2020) identify a collection of 26 quantitative
metrics that measure data complexity with respect to (1) ambiguity of classes,
i.e., whether the classes can be clearly distinguished with the given
features, (2) sparsity and dimensionality of data, i.e., whether enough
information are provided to learn confident decision boundaries, and (3)
complexity of boundary separating the classes, i.e., whether more intricate
functions are required to describe the decision boundaries. The authors also
discuss how these metrics help estimate the difficulty of performing
classification on a given dataset. Similar to our survey, the authors show
that high dimensionality and small separation between classes hinder standard
generalization. However, the relationship of some of the metrics reviewed by
these authors, e.g., the number of non-intersecting spheres needed to enclose
all data points of a class, to robust generalization is not studied, according
to our survey.

He and Garcia (He and Garcia, 2009) focus on the imbalance learning problem –
the disproportion in the number of samples belonging to each class in a given
dataset. The authors found that most standard algorithms are designed with the
assumption of a balanced class distribution. These algorithms fail to reliably
represent the distributive characteristics of the imbalanced size of samples
and result in unfavorable performance across classes. Furthermore, López et
al. (López et al., 2013) present a thorough discussion on six intrinsic data
characteristics that potentially complicate learning from imbalanced data: low
density, sample overlap between classes, noisy data, borderline instances,
dataset shift between training and testing distributions, and small disjuncts,
i.e., disperse small clusters of samples from a single class. Their analysis
concludes that while all these “unfavorable” data characteristics further
complicate the data imbalance issues, data overlap between classes is probably
one of the most harmful. To follow up on this point, Santos et al. (Santos et
al., 2022) focus on the joint effect of data imbalance and class overlap on
model generalization. The negative impact of data imbalance, low separation,
and noisy data on robust generalization was also discussed in our survey. Yet,
the compounding effect of these factors, as well as the effect of other
properties, on robust generalization needs future investigation.

Recently, Yang et al. (Yang et al., 2022) summarized relevant studies focusing
on long-tailed distributions in the field of Computer Vision. This survey also
includes work on the influence of long-tail distributions on a model’s
adversarial robustness (Wu et al., 2021), which is covered in our survey. The
authors advocate for more research on adapting long-tailed-based approaches
for standard generalization to improve robust generalization.

Finally, Moreno-Torres et al. (Moreno-Torres et al., 2012) present a unifying
framework to categorize existing definitions of dataset shift – the case where
the joint distribution of inputs and outputs differs between training and
testing data. While ML models are normally trained under the premise that
testing data has a similar distribution to the training data, in reality, the
observed data distribution may be different from the historical data that the
model is trained on. Such difference can substantially compromise the quality
of model predictions. The authors analyze the possible causes for dataset
shift, e.g., malicious software that evolves over time, and review the
techniques dealing with dataset shift. They characterize adversarial attacks
as one form of dataset shift, where adversaries adaptively change test
instances to create a distribution that differs from training data.

###  6.3. Non-data Related Reasons for Adversarial Vulnerability

There has been a variety of hypotheses regarding the reasons behind
adversarial vulnerability of ML systems. In addition to the data used for
training, adversarial robustness could also depend on the choice of the model
architecture, the training procedure, and the interplay between data and the
learning algorithm, i.e., correspondence between the complexity of a model to
that of the data. This section summarizes the key hypotheses regarding these
aspects.

Model. When Szegedy et al. (Szegedy et al., 2014) first discovered adversarial
examples for visual models, they suspected the high non-linearity of DNNs
resulted in low probability ‘pockets’ of adversarial examples in the learned
representation manifold. They hypothesize that while these pockets can be
found through attack algorithms, the samples residing in these pockets have
different distributions compared to normal samples and are thus subsequently
harder to find when randomly sampling from the input space. Instead,
Goodfellow et al. (Goodfellow et al., 2015) hypothesize that the linearity
from activation functions, like ReLU and sigmoid found in high-dimensional
neural networks, induce vulnerability towards adversarial perturbations. To
support their claim, they present the attack method FGSM that exploits the
linearity of the target classifier. Fawzi et al. (Fawzi et al., 2015) also
argue against the hypothesis of high non-linearity as the cause for
adversarial examples. They show that all classifiers are susceptible to
adversarial attacks and claim that it is the low flexibility of the classifier
compared to the complexity of the classification task that results in
vulnerability. The lack of consensus on primary causes of models’
vulnerability invites more studies on this topic.

Singla et al. (Singla et al., 2021) show that enforcing invariance to circular
shifts (e.g., rotation) in neural networks induces decision boundaries with a
smaller margin than normal, fully connected networks, which, in turn, reduces
the adversarial robustness of the model. Moosavi-Dezfooli et al. (Moosavi-
Dezfooli et al., 2018) introduce universal, input-agnostic perturbations to
mislead the classifier and hypothesize that the vulnerability of a multi-class
classifier to such perturbations is related to the shape of its decision
boundaries, e.g., linear classifiers with decision boundaries that are
parallel to each other and nonlinear classifier with decision boundaries that
are curved in a similar way tend to be less robust as perturbations in one
direction can change the prediction label for a different class.

Tanay and Griffin (Tanay and Griffin, 2016) conjecture that the decision
boundary learned by the classifier being too close to (or ‘tilted towards’)
the data manifold instead of being perpendicular to it, results in small
perturbations being sufficient to move samples across the decision boundary
for misclassification.

Computational Resources. Bubeck et al. (Bubeck et al., 2019) use computational
hardness theory to show that the time complexity for learning a robust model
is exponential to the size of input data and thus is computationally
intractable. Hence, they attribute adversarial vulnerability to computational
limitations of current learning algorithms. Degwekar et al. (Degwekar et al.,
2019) further extend this work and also show the impossibility of efficiently
training robust classifiers.

Robustness of Features. Ilyas et al. (Ilyas et al., 2019) show that
adversarial vulnerability can be a consequence of a model exploiting well-
generalizing but non-robust features, i.e., features that are spurious and
sometimes incomprehensible to humans; when constraining the model to use
robust features, the adversarial robustness increases together with the
interpretability of the learned features. However, Tsipras et al. (Tsipras et
al., 2019) note that, as the features for achieving high accuracy may be
different from the ones for achieving high robustness, robustness may be at
odds with standard accuracy. Instead of seeing adversarial vulnerability as a
product of classifiers being overly sensitive to changes in spurious features,
Jacobsen et al. (Jacobsen et al., 2019) hypothesize that classifiers can
rather be overly insensitive to relevant semantic information, e.g., images
with drastically different content can share similar latent representations.
The authors introduce a new type of adversarial examples that exploit such
insensitivity, where the content of images is altered without changing the
resulting prediction label. While all these works propose possible reasons for
adversarial vulnerabilities, they are orthogonal to our survey, which focuses
particularly on the influence of training data.

##  7\. Conclusion

In this survey, we systematically collected, analyzed, and described papers
that discuss how data properties affect adversarial robustness in machine
learning models. By analyzing 57 research papers from top scientific venues in
Machine Learning, Computer Vision, Computational Linguistics, and Security, we
identified seven domain-agnostic data properties and one image-specific data
property that are correlated with adversarial robustness.

While several of the guidelines for constructing high-quality data that we
identified are similar to those recommended for training accurate models,
producing robust models is more sensitive to the characteristics of the data
and requires more effort, e.g., a larger number of samples, better label
qualities, etc. There are also additional data properties important for
building robust models that are not extensively discussed in non-adversarial
settings, e.g., concentration of measure. In a sense, robust generalization is
a stronger form of standard generalization.

We identified possible next steps towards improving the understanding of how
the data affects a model’s adversarial robustness. These include studying
interactions between different properties of data, considering the effect of
additional properties that improve standard generalization on robust model
generalization, devising quantitative metrics for different aspects of the
data, and extending the studies and their empirical evaluation beyond the
images domain. We hope our survey will help researchers and ML practitioners
to better understand adversarial vulnerability and will spark further research
to address the identified knowledge gaps.

## References

  * (1)
  * ACM ([n. d.]) [n. d.].  ACM Computing Surveys Journal.  https://dl.acm.org/journal/csur.
  * Neu ([n. d.]) [n. d.].  Advances in Neural Information Processing Systems (NeurIPS).  https://proceedings.neurips.cc.
  * Cor ([n. d.]) [n. d.].  CORE ranking (Conference Portal).  http://portal.core.edu.au/conf-ranks/.
  * Jou ([n. d.]) [n. d.].  Journal Citation Reports (JCR).  https://jcr.clarivate.com/jcr/home.
  * PML ([n. d.]) [n. d.].  Proceedings of Machine Learning Research.  https://proceedings.mlr.press.
  * Sem ([n. d.]) [n. d.].  Semantic Scholar Academic APIs.  https://www.semanticscholar.org/product/api.
  * Akhtar and Mian (2018) Naveed Akhtar and Ajmal Mian. 2018.  Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey.  _IEEE Access_ 6 (2018), 14410–14430.
  * Akhtar et al. (2021) Naveed Akhtar, Ajmal S. Mian, Navid Kardan, and Mubarak Shah. 2021\.  Advances in Adversarial Attacks and Defenses in Computer Vision: A Survey.  _IEEE Access_ 9 (2021), 155161–155196.
  * Amsaleg et al. (2021) Laurent Amsaleg, James Bailey, Amélie Barbe, Sarah M. Erfani, Teddy Furon, Michael E. Houle, Miloš Radovanović, and Xuan Vinh Nguyen. 2021.  High Intrinsic Dimensionality Facilitates Adversarial Attack: Theoretical Evidence.  _IEEE Transactions on Information Forensics and Security_ 16 (2021), 854–865.
  * Ansuini et al. (2019) Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. 2019.  Intrinsic Dimension of Data Representations in Deep Neural Networks. In _Advances in Neural Information Processing Systems (NeurIPS)_. 6111–6122.
  * Awasthi et al. (2020) Pranjal Awasthi, Himanshu Jain, Ankit Singh Rawat, and Aravindan Vijayaraghavan. 2020.  Adversarial Robustness via Robust Low Rank Representations. In _Advances in Neural Information Processing Systems (NeurIPS)_. 11391–11403.
  * Bansal et al. (2021) Ms. Aayushi Bansal, Dr. Rewa Sharma, and Dr. Mamta Kathuria. 2021.  A Systematic Review on Data Scarcity Problem in Deep Learning: Solution and Applications.  _Comput. Surveys_ 54, 208 (2021), 1–29.
  * Bayer et al. (2022) Markus Bayer, Marc-André Kaufhold, and Christian Reuter. 2022.  A Survey on Data Augmentation for Text Classification.  _Comput. Surveys_ (2022).
  * Bhagoji et al. (2019) Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. 2019\.  Lower Bounds on Adversarial Robustness from Optimal Transport. In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Bhattacharjee and Chaudhuri (2020) Robi Bhattacharjee and Kamalika Chaudhuri. 2020.  When Are Non-Parametric Methods Robust?. In _International Conference on Machine Learning (ICML)_. 832–841.
  * Bhattacharjee et al. (2021) Robi Bhattacharjee, Somesh Jha, and Kamalika Chaudhuri. 2021\.  Sample Complexity of Robust Linear Classification on Separated Data. In _Conference on Learning Theory (COLT)_. 884–893.
  * Biggio and Roli (2018) Battista Biggio and Fabio Roli. 2018.  Wild Patterns: Ten Years after The Rise of Adversarial Machine Learning.  _Pattern Recognition_ 84 (2018), 317–331.
  * Blum et al. (2020) Avrim Blum, Travis Dick, Naren Manoj, and Hongyang Zhang. 2020\.  Random Smoothing Might Be Unable to Certify $L_{\infty}$ Robustness for High-Dimensional Images.  21, 211 (2020), 8726–8746.
  * Bonaccorso (2017) Giuseppe Bonaccorso. 2017\.  _Machine Learning Algorithms: A Reference Guide to Popular Algorithms for Data Science and Machine Learning_.  Packt Publishing.
  * Borell (1975) Christer Borell. 1975\.  The Brunn-Minkowski Inequality in Gauss Space.  _Inventiones mathematicae_ 30 (1975), 207–216.
  * Bubeck et al. (2019) Sebastien Bubeck, Yin Tat Lee, Eric Price, and Ilya Razenshteyn. 2019.  Adversarial Examples from Computational Constraints. In _International Conference on Machine Learning (ICML)_. 831–840.
  * Bui et al. (2020) Anh Bui, Trung Le, He Zhao, Paul Montague, Oliver deVel, Tamas Abraham, and Dinh Phung. 2020.  Improving Adversarial Robustness by Enforcing Local and Global Compactness. In _European Conference on Computer Vision (ECCV)_. 209–223.
  * Carbone et al. (2020) Ginevra Carbone, Matthew Wicker, Luca Laurenti, Andrea Patane, Luca Bortolussi, and Guido Sanguinetti. 2020.  Robustness of Bayesian Neural Networks to Gradient-Based Attacks. In _International Conference on Neural Information Processing Systems (NeurIPS)_. 15602–15613.
  * Carlini and Wagner (2017) Nicholas Carlini and David A. Wagner. 2017.  Towards Evaluating the Robustness of Neural Networks. In _Symposium on Security and Privacy (SP)_. 39–57.
  * Carmon et al. (2019) Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. 2019\.  Unlabeled Data Improves Adversarial Robustness. In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Chakraborty et al. (2018) Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. 2018.  Adversarial Attacks and Defences: A Survey.  _ArXiv_ (2018).
  * Chen et al. (2021) Guangyao Chen, Peixi Peng, Li Ma, Jia Li, Lin Du, and Yonghong Tian. 2021\.  Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domain. In _IEEE/CVF International Conference on Computer Vision (ICCV)_. 458–467.
  * Chen and Hsieh (2023) Pin-Yu Chen and Cho-Jui Hsieh. 2023.  _Adversarial Robustness for Machine Learning_.  Academic Press.
  * Cheng et al. (2021) Wuxinlin Cheng, Chenhui Deng, Zhiqiang Zhao, Yaohui Cai, Zhiru Zhang, and Zhuo Feng. 2021\.  SPADE: A Spectral Method for Black-Box Adversarial Robustness Evaluation. In _International Conference on Machine Learning (ICML)_. 1814–1824.
  * Cohen et al. (2019) Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. 2019\.  Certified Adversarial Robustness via Randomized Smoothing. In _International Conference on Machine Learning (ICML)_. 1310–1320.
  * Cordts et al. (2016) Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2016.  The Cityscapes Dataset for Semantic Urban Scene Understanding. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 3213–3223.
  * Cramer (2002) J.S. Cramer. 2002\.  _The Origins of Logistic Regression_.  Technical Report 2002-119/4. Tinbergen Institute.
  * Croce and Hein (2020) Francesco Croce and Matthias Hein. 2020.  Minimally Distorted Adversarial Examples with a Fast Adaptive Boundary Attack. In _International Conference on Machine Learning (ICML)_. 2196–2205.
  * Cullina et al. (2018) Daniel Cullina, Arjun Nitin Bhagoji, and Prateek Mittal. 2018.  PAC-Learning in the Presence of Evasion Adversaries. In _Advances in Neural Information Processing Systems (NeurIPS)_. 228–239.
  * Dan et al. (2020) Chen Dan, Yuting Wei, and Pradeep Ravikumar. 2020.  Sharp Statistical Guarantees for Adversarially Robust Gaussian Classification. In _International Conference on Machine Learning (ICML)_. 2345–2355.
  * Daniely and Schacham (2020) Amit Daniely and Hadas Schacham. 2020.  Most ReLU Networks Suffer from L2 Adversarial Perturbations. In _Advances in Neural Information Processing Systems (NeurIPS)_. 6629–6636.
  * Degwekar et al. (2019) Akshay Degwekar, Preetum Nakkiran, and Vinod Vaikuntanathan. 2019\.  Computational Limitations in Robust Classification and Win-Win Results. In _Conference on Learning Theory (COLT)_. 994–1028.
  * Demetrio et al. (2021) Luca Demetrio, Scott E. Coull, Battista Biggio, Giovanni Lagorio, Alessandro Armando, and Fabio Roli. 2021.  Adversarial EXEmples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection.  _ACM Transactions on Privacy and Security_ 24, 4 (2021), 1–31.
  * Demontis et al. (2019) Ambra Demontis, Marco Melis, Battista Biggio, Davide Maiorca, Dan Arp, Konrad Rieck, Igino Corona, Giorgio Giacinto, and Fabio Roli. 2019.  Yes, Machine Learning Can Be More Secure! A Case Study on Android Malware Detection.  _IEEE Transactions on Dependable and Secure Computing (TDSC)_ 16, 4 (2019), 711–724.
  * Ding et al. (2019) Gavin Weiguang Ding, Kry Yik Chau Lui, Xiaomeng Jin, Luyu Wang, and Ruitong Huang. 2019.  On the Sensitivity of Adversarial Robustness to Input Data Distributions. In _International Conference on Learning Representations (ICLR)_.
  * Diochnos et al. (2018) Dimitrios I. Diochnos, Saeed Mahloujifar, and Mohammad Mahmoody. 2018.  Adversarial Risk and Robustness: General Definitions and Implications for the Uniform Distribution. In _Advances in Neural Information Processing Systems (NeurIPS)_. 10380–10389.
  * Dohmatob (2019) Elvis Dohmatob. 2019\.  Generalized No Free Lunch Theorem for Adversarial Robustness. In _International Conference on Machine Learning (ICML)_. 1646–1654.
  * Eykholt et al. (2018) Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. 2018.  Robust Physical-World Attacks on Deep Learning Visual Classification. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 1625–1634.
  * Fatras et al. (2022) Kilian Fatras, Bharath Bhushan Damodaran, Sylvain Lobry, Rémi Flamary, Devis Tuia, and Nicolas Courty. 2022.  Wasserstein Adversarial Regularization for Learning With Label Noise.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 44, 10 (2022), 7296–7306.
  * Fawzi et al. (2018) Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. 2018.  Adversarial Vulnerability for Any Classifier. In _Advances in Neural Information Processing Systems (NeurIPS)_. 1186–1195.
  * Fawzi et al. (2015) Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. 2015\.  Fundamental Limits on Adversarial Robustness. In _ICML Workshop on Deep Learning_.
  * Frenay and Verleysen (2014) Benoit Frenay and Michel Verleysen. 2014.  Classification in the Presence of Label Noise: A Survey.  _IEEE Transactions on Neural Networks and Learning Systems_ 25, 5 (2014), 845–869.
  * Garg et al. (2018) Shivam Garg, Vatsal Sharan, Brian Hu Zhang, and Gregory Valiant. 2018.  A Spectral View of Adversarially Robust Features. In _Advances in Neural Information Processing Systems (NeurIPS)_. 10159–10169.
  * Ghorbani et al. (2019) Amirata Ghorbani, Abubakar Abid, and James Zou. 2019.  Interpretation of Neural Networks is Fragile. In _AAAI Conference on Artificial Intelligence (AAAI)_. 3681–3688.
  * Ghosh et al. (2019) Partha Ghosh, Arpan Losalka, and Micheal. J. Black. 2019\.  Resisting Adversarial Attacks using Gaussian Mixture Variational Autoencoders. In _AAAI Conference on Artificial Intelligence (AAAI)_. 541–548.