# Deep Repulsive Prototypes for Adversarial Robustness

Alex Serban1, Erik Poll1 and Joost Visser2

# Deep Repulsive Prototypes for Adversarial Robustness

Alex Serban1, Erik Poll1 and Joost Visser2

###### Abstract

While many defences against adversarial examples have been proposed, finding
robust machine learning models is still an open problem. The most compelling
defence to date is adversarial training and consists of complementing the
training data set with adversarial examples. Yet adversarial training severely
impacts training time and depends on finding representative adversarial
samples. In this paper we propose to train models on output spaces with large
class separation in order to gain robustness without adversarial training. We
introduce a method to partition the output space into class prototypes with
large separation and train models to preserve it. Experimental results shows
that models trained with these prototypes – which we call deep repulsive
prototypes – gain robustness competitive with adversarial training, while also
preserving more accuracy on natural samples. Moreover, the models are more
resilient to large perturbation sizes. For example, we obtained over 50%
robustness for CIFAR-10, with 92% accuracy on natural samples and over 20%
robustness for CIFAR-100, with 71% accuracy on natural samples without
adversarial training. For both data sets, the models preserved robustness
against large perturbations better than adversarially trained models.

##  1 Introduction

Although a plethora of adversarial defences have been proposed – ranging from
input projections (Guo et al. 2017) to formal guarantees that no adversarial
examples can be found within some bounds (Huang et al. 2020) – robustness to
adversarial examples remains an open problem.

The most compelling defence to date is adversarial training and consists of
complementing the training data set with adversarial samples (Goodfellow,
Shlens, and Szegedy 2014), or training only on perturbed data (Madry et al.
2017). Yet adversarial training is subject to several trade-offs. Firstly, the
time needed to generate adversarial examples substantially increases training
time. Recent attempts to generate adversarial examples faster exist (Wong,
Rice, and Kolter 2020). However, they are (at the moment) unstable and
introduce new issues such as catastrophic forgetting (Andriushchenko and
Flammarion 2020).

Secondly, a trade-off between accuracy on natural samples and robustness on
adversarial examples is known to exist (Zhang et al. 2019b). This trade-off
implies that robustness against adversarial examples comes with a cost of
losing accuracy on natural examples, and can be controlled through adversarial
training (Zhang et al. 2019b). Lastly, adversarial training overfits on
training data and provides little robustness against data outside this
distribution (Zhang et al. 2019a; Rice, Wong, and Kolter 2020).

A model robust to adversarial examples should provide: (i) inter-class
separability, (ii) intra-class compactness, and (iii) marginalisation or
removal of non-robust features (Smith 2019; Ilyas et al. 2019). However,
adversarial training does not impose explicit constraints for meeting these
properties (e.g., by specifying inductive biases). Therefore, it depends only
on finding representative adversarial examples for training.

In supervised classification, adversarial training uses the standard softmax
cross-entropy loss. Recently, there is increasing evidence that softmax
partitions the output space into class centroids situated at equal distance
from the origin (inter-class separability), and that adversarial robustness
can be improved by clustering the data points in the proximity of these
centroids (intra-class compactness) (Hess, Duivesteijn, and Mocanu 2020;
Papernot and McDaniel 2018).

However, the distance between class centroids is insufficient to provide
robustness, and even models with high intra-class compactness are vulnerable
to adversarial attacks. In this paper we tackle this issue by enforcing large
inter-class separation prior to training using class prototypes (Snell,
Swersky, and Zemel 2017). By making use of this inductive bias we gain more
control over the output space structure, and can decrease the number of
training samples needed (Mettes, van der Pol, and Snoek 2019).

We show that training with class prototypes optimised to provide large inter-
class separation helps to gain robustness competitive with adversarial
training, _without_ adversarial training. Moreover, training with class
prototypes involves a smaller trade-off between accuracy and robustness, and a
higher resilience against large perturbations. The prototypes are built prior
to training with little overhead, through an optimisation procedure that
increases the distance between their centres. As a result of this repelling
optimisation procedure, and because we use deep neural networks for empirical
validation, we call the prototypes _deep repulsive prototypes_. We test
repulsive prototypes on CIFAR-10 and CIFAR-100, and observe consistent results
on both data sets, with 51.3% and 20.5% robustness against iterative
adversarial attacks.

The rest of the paper is organised as follows. Initially, we introduce
background information and discuss related work (Section 2). Later, we present
repulsive prototypes (Section 3), followed by an evaluation against white and
black-box attacks (Section 4). We conclude with a discussion (Section 5) and
future work (Section 6).

##  2 Background and Related Work

We focus on supervised classification, i.e., given a set of inputs sampled
from ${\mathcal{X}}$ and corresponding labels sampled from ${\mathcal{Y}}$, an
algorithm finds a mapping from
$f(\cdot,{\bm{\theta}}):{\mathcal{X}}\rightarrow{\mathcal{Y}}$ which minimises
the number of misclassified inputs. We assume that ${\mathcal{X}}$ is a metric
space such that a distance function $d(\cdot)$ between two points of the space
exists. The error made by a prediction $f({\bm{x}})=\hat{y}$ when the true
label is $y$ is measured by a loss function
$l:{\mathcal{Y}}\times{\mathcal{Y}}\rightarrow{\mathbb{R}}$ with non-negative
values when the labels are different and zero otherwise. $f(\cdot)$ is defined
over a hypotheses space ${\mathcal{F}}$ which encompasses any mapping from
${\mathcal{X}}$ to ${\mathcal{Y}}$ and can take any form – e.g., a linear
function or a neural network. Through learning, an algorithm selects
$f^{*}(\cdot)$ from ${\mathcal{F}}$ such that the expected empirical loss on a
training data set ${\mathcal{S}}$ consisting of pairs of samples
$({\bm{x}}_{i},y_{i})\sim{\mathcal{Z}}={\mathcal{X}}\times{\mathcal{Y}}$ is
minimal.

A robust solution to the minimisation problem above involves immunising it
against uncertainties in the input space. In the adversarial examples setting,
uncertainties are modelled in the space around an input ${\bm{x}}^{\prime}$:
${\mathcal{U}}_{{\bm{x}}}=\\{{\bm{x}}^{\prime}|d({\bm{x}},{\bm{x}}^{\prime})\leq\epsilon\\}$.
The robust counterpart of the learning problem becomes:

$f=\operatorname*{arg\,min}_{f\in{\mathcal{F}}}\mathbb{E}_{({\bm{x}},y)\sim{\mathcal{S}}}[\max_{{\bm{x}}^{\prime}\in{\mathcal{U}}_{{\bm{x}}}}l({\bm{\theta}},{\bm{x}}^{\prime},y)],$
(1)

where ${\bm{x}}^{\prime}$ is a realization of ${\bm{x}}$ in the uncertainty
set described by $d(\cdot)$. Common distance functions $d(\cdot)$ are the
Euclidean $L_{2}$ distance or the Chebyshev $L_{\infty}$ distance. We further
use the term robustness to define the accuracy of a model on adversarial
examples,
$\mathbb{E}_{({\bm{x}},y)\sim{\mathcal{S}}}[f({\bm{x}}^{\prime})=y]$.

Adversarial training consists of adding a regularization term to the loss
function:

$\tilde{l}(\cdot)=\alpha
l({\bm{\theta}},{\bm{x}},y)+(1-\alpha)l({\bm{\theta}},{\bm{x}}^{\prime},y),$
(2)

where ${\bm{x}}^{\prime}$ is an adversarial example generated from input
${\bm{x}}$ and $\alpha$ controls the contribution of adversarial examples to
the loss. The most effective choice of $\alpha$ is zero (Madry et al. 2017),
which poses Eq. (1) as a min-max problem where the inner maximisation problem
seeks to find the worst adversarial example for an input and the outer
minimisation problem seeks to strengthen the model against it.

Finding an exact solution to the inner maximisation problem is not feasible
for adversarial training. The most effective approximation solution relies on
iteratively taking small steps towards maximising the loss function, and
projecting the outcome on the space defined by $d(\cdot)$; a procedure called
projected gradient descent (PGD) and defined as follows:

${\bm{x}}_{n}^{\prime}=\prod_{{\bm{x}}+{\mathcal{U}}_{\bm{x}}}({\bm{x}}_{n-1}^{\prime}+\epsilon\operatorname{sign}(\nabla_{{\bm{x}}^{\prime}_{n-1}}l({\bm{\theta}},{\bm{x}}^{\prime}_{n-1},y))).$
(3)

The quality of the adversarial examples depends on the number of iterations
$n$. Using a large number of iterations, PGD can better approximate the space
around an input we want to provide robustness to, and leads to more
representative adversarial examples for training. However, it negatively
affects training time. When $n=1$, the method is called the Fast Gradient Sign
Method (FGSM).

Several attempts have been made to change the training procedure in order to
enforce inter-class separability or intra-class compactness, and also decrease
the impact of adversarial training. Mao et al. (2019) used a triplet loss
(inspired by metric learning), where one element of the triplet loss is an
adversarial example. An attempt to reduce the impact of adversarial training
on training time was made by only generating one adversarial example for each
triplet data. Further inductive biases, such as careful negative sample
selection for the triplet loss, help to improve robustness.

Papernot and McDaniel (2018) showed that explicitly tailoring intra-class
compactness using k-neighbours in the representation space helps to detect
adversarial examples. Hess, Duivesteijn, and Mocanu (2020) proved a similar
result and proposed a method based on the Gauss kernel to enforce intra-class
compactness and improve robustness. Pang et al. (2019) introduced a loss
function to enforces inter-class separability using the centroids of the Max-
Mahalanobis distribution. During inference, the class centroid closer to the
input’s deep representation (measured using the Euclidean distance) was used
to classify an input. The defence builds on earlier work by Pang, Xu, and Zhu
(2019), where at inference time an input is interpolated with samples from the
same predicted class, and from distinct classes in order to alleviate the
impact of perturbations. Unfortunately, none of these defences proved
effective (Tramer et al. 2020).

Jin and Rinard (2020) showed that manifold regularisation improves adversarial
robustness significantly while retaining better accuracy on natural examples,
without adversarial training. Their proposal induces local stability in the
neighbourhood of natural inputs even if the model classifies the inputs
incorrectly. This is in contrast with adversarial training, where a model is
trained to classify correctly worst case adversarial examples.

Mustafa et al. (2019) used class prototypes to enforce inter-class
separability, by including a prototype separation constraint in the loss
function. A convex polytope is assigned as prototype to each class and during
training the distance between all class polytopes is maximised. Thus the class
centroids are learned together with the internal representation. However,
adding the distance maximisation term to the loss function does not suffice to
improve robustness, and they propose to add similar constraints to hidden
layers. When paired with adversarial training, robustness increases at a
decreased cost for accuracy on natural samples.

Mettes, van der Pol, and Snoek (2019) showed that defining class prototypes a
priori to training in order to enforce desired properties of the output space
(e.g., large margin separation) improves training in several settings; such as
few-shot classification, regression or joint classification and regression.
Instead of constantly re-estimating and calibrating the prototypes – as in
Mustafa et al. (2019) or others (Snell, Swersky, and Zemel 2017; Guerriero,
Caputo, and Mensink 2018) – they propose to define the output space as a
hyper-sphere and partition it into predefined class prototypes. During
training, the distance between the model’s output and class prototypes is
minimised. The a priori definition of class prototypes enables control over
several factors such as the output space size, or its shape. In this paper we
take a similar path and define class prototype prior to training.

##  3 Repulsive Prototypes for Robustness

Figure 1: Repulsive prototypes. The grey spheres represent prototypes obtained
using the $L_{2}$ distance, and the blue cubes are obtained using the
$L_{\infty}$ distance.

The idea behind building class prototypes for adversarial robustness is to
explicitly design prototypes with large inter-class separation, and during
training enforce intra-class compactness. To this end, the input or the output
space is partitioned into hyper-planes specific to each class, to which we
impose separation constraints. Similar approaches have been used in the past,
e.g., by Schiilkop, Burgest, and Vapnik (1995) who used the smallest sphere
enclosing the data to estimate the VC-dimension for support vector
classifiers, or by Wang, Neskovic, and Cooper (2005) who used separating
spheres in the feature space for classification.

We propose an approach similar to Wang, Neskovic, and Cooper (2005) and Nguyen
and Tran (2015), and more recent work by Mettes, van der Pol, and Snoek
(2019), where the separation boundaries are imposed to the output space –
instead of the input space – because the output space allows more flexibility
and can achieve larger margin separation. Prior to learning, the
$D$-dimensional output space is divided into $k$ prototypes
$P=\\{p_{1},\cdots,p_{k}\\}$, where each prototype corresponds to a class. For
a binary classification problem and a Euclidean output space, we wish to find
two hyper-spheres with centres at $c_{1},c_{2}$ – one enclosing samples from
the positive class and the other enclosing samples from the negative class –
and maximise the distance between them:

$\displaystyle\min_{\epsilon_{1},\epsilon_{2},c_{1},c_{2}}$
$\displaystyle\epsilon_{1}^{2}+\epsilon_{2}^{2}-r\|c_{1}-c_{2}\|^{2}$ (4)
$\displaystyle s.t.$
$\displaystyle\|f({\bm{x}}_{i})-c_{1}\|^{2}\leq\epsilon_{1}^{2},\forall
i,y_{i}=+1$
$\displaystyle\|f({\bm{x}}_{i})-c_{1}\|^{2}\geq\epsilon_{1}^{2},\forall
i,y_{i}=-1$
$\displaystyle\|f({\bm{x}}_{i})-c_{2}\|^{2}\leq\epsilon_{2}^{2},\forall
i,y_{i}=-1$
$\displaystyle\|f({\bm{x}}_{i})-c_{2}\|^{2}\geq\epsilon_{2}^{2},\forall
i,y_{i}=+1,$

where $r$ is a constant that represents the repulsive degree between the two
prototypes, and $\epsilon_{1},\epsilon_{2}$ define the $d(\cdot)$-ball around
the prototype centres for which we want to provide robustness (corresponding
to the uncertainty set in Eq. (1)).

For non-separable data sets, the constraints above can be relaxed by
introducing slack variables and regularisation terms to the objective
function. Although the objective in Eq. (4) is not convex, it can be
reformulated to have a convex form and solved using Lagrange multipliers.
However, in practice the constraints can be relaxed and the problem can be
solved in two steps: firstly find prototypes with large separation (to provide
inter-class separability), and secondly train models to fit the data within
the proximity of the prototype centres (to provide intra-class compactness).
An approximate solution to the first problem can be found by running gradient
descent on the unconstrained objective:

$\displaystyle\min_{\epsilon_{1},\epsilon_{2},c_{1},c_{2}}$
$\displaystyle\epsilon_{1}^{2}+\epsilon_{2}^{2}-r\|c_{1}-c_{2}\|^{2},$

with a generalisation to k-classes and any metric space:

$\displaystyle\min_{\epsilon,c}$ $\displaystyle\epsilon^{k}-r\sum_{(i,j,i\neq
j)\in k}d(c_{i},c_{j}).$ (5)

The choice of $r$ can also be controlled using the learning rate $\mu$ for
gradient descent. The choice of $d(\cdot)$ influences the prototypes and the
classification regions defined in the output space. For example, using the
Euclidean $L_{2}$ distance leads to hyper-spherical classification regions,
and using the Chebyshev $L_{\infty}$ distance leads to hyper-cubical regions.
An illustration is provided in Figure 1, where the grey spheres represent
regions for perturbations in the $L_{2}$ space and the blue cubes are regions
for perturbations in the $L_{\infty}$ space around the centres. Iterating over
Eq. (LABEL:eq:proto) is equivalent to increasing $d(\cdot$) or adding slack
variables to Eq. (4). Larger distances between class prototypes introduce
buffers between classification boundaries and should improve robustness.

The second step – training models to fit the data within the proximity of the
prototype centres – can be solved by minimising the distance between the
prototype centres and the model’s output. The choice for this distance
function is part of the threat model and it is the same as $d(\cdot)$ from Eq.
(LABEL:eq:proto), which induces the following loss function:

$l=\sum_{i=1}^{N}(1-d(f({\bm{x}}_{i}),p_{y_{i}}))^{2},$ (6)

where $p_{y_{i}}$ is the prototype specific to class $y_{i}$.

##  4 Empirical Evaluation

All experiments are performed using a vanilla ResNet-18 network (the smallest
variant of ResNet). Capacity is known to help adversarial robustness (Madry et
al. 2017; Xie et al. 2020). Therefore, we avoid using larger networks. During
training with repulsive prototypes only natural samples are used, i.e.,  _no_
adversarial training is performed.

Firstly, we adopt a white-box threat model for testing, where attackers
presumably have full knowledge of the model under attack, the training and the
testing data (Carlini et al. 2019). To generate adversarial examples we use
the PGD (Eq. (3)) PyTorch implementation from Cleverhans, with different
iterations and random restarts (Papernot et al. 2016). Testing against larger
$n$ is recommended, as it shows if the model exhibits a false sense of
robustness or obfuscates attack vectors (Carlini et al. 2019).

Attackers are constrained to generate adversarial examples in the $\epsilon=8$
(normalised) $L_{p}$ norm ball around inputs – a common benchmark for
adversarial robustness. Since the distance between prototypes is larger than
$\epsilon$, we expect models trained with repulsive prototypes to also exhibit
resilience to higher $\epsilon$ values. In order to test this hypothesis, we
use robustness curves obtained by step-wise increasing the size of the
perturbation in the interval $[8,16]$.

We compare with results from literature on two common data sets; CIFAR-10 and
CIFAR-100 (Krizhevsky, Hinton et al. 2009). The first one consists of 60 000
32x32 colour images and 10 classes (with 5 000 images for training and 1 000
images for testing per class). The second data set consists of 60 000 32x32
images and 100 classes (with 500 training images for training and 100 images
for testing per class). We use minimal data pre-processing for training,
consisting of random cropping and random horizontal flip. No data pre-
processing is used for testing.

For training, we use the cyclical learning rate (Smith 2017), mixed precision
arithmetic and early stopping, as they are reported to improve training time
and prevent overfitting (Wong, Rice, and Kolter 2020; Rice, Wong, and Kolter
2020).

Later in this section we also adopt a black-box threat model, where attackers
can only observe the outcome of the models under attack. For evaluation we use
the transferability attack, where adversarial examples are generated with a
distinct model, and transferred to the models under attack (Su et al. 2018;
Carlini et al. 2019).

| Output
---
Dimension ($D$)
Epochs |  | Clean
---
Samples
PGD-20
50 | 50 | 90.3 | 37.2
100 | 50 | 90.6 | 39.9
200 | 50 | 89.5 | 40.7
100 | 100 | 91.0 | 48.7
200 | 100 | 91.1 | 38.7
Table 1: Prototype selection on CIFAR-10.

### Prototype Selection

Several parameters influence the quality of the prototypes: the output space
dimension $D$, the choice of $\epsilon$, $r$, the learning rate $\mu$ and the
number of epochs for solving Eq. (LABEL:eq:proto). Since $r$ and $\mu$ can be
compressed to one constant, and (to some extent) the effect of the learning
rate can be attenuated by running the optimisation longer, the most important
parameters are the output space dimension and the number of epochs. As
mentioned earlier, when not mentioned otherwise we use the same choice for
$\epsilon=8$ (normalised).

Previously, it has been shown that increasing $D$ can benefit both
classification and regression (Mettes, van der Pol, and Snoek 2019). In order
to determine the influence of $D$ on robustness and accuracy, we run an
experiment on the CIFAR-10 data set, training a ResNet-18 model for 50 epochs
with different output sizes $D\in\\{50,100,200\\}$ – corresponding to
multiplying the number of classes $k$ with factors of $\\{5,10,20\\}$. Testing
is performed with natural and perturbed samples using the PGD attack, with
$n=20$. In all cases, the prototypes are generated by running gradient descent
on Eq. (LABEL:eq:proto) for 100 epochs, with $\mu=0.01$. This modest
optimisation budget is sufficient to obtain large distances between the
prototypes.

The results are presented in Table 1. We observe that increasing the output
dimension $D$ has almost no impact on accuracy on natural samples, but a
significant impact on robustness, for all values of $D$ except the last one.
For the last two values of $D$ we ran training longer and observe that the
largest output space (i.e., 200) has a bigger tendency to overfit for
adversarial examples, while maintaining similar accuracy on natural samples
(corresponding to 100 epochs in Table 1). This phenomenon will be elaborated
in Section 5.

The initial experiments on prototype selection reveal that the output size is
important for adversarial robustness, but plays a marginal role for accuracy
on natural samples.

### CIFAR-10

Figure 2: Robustness curves for CIFAR-10, obtained by testing with PGD-20, and
various perturbation sizes ($\epsilon$).

Following the previous experiments, we present the results from training a
ResNet-18 model on CIFAR-10 with the same parameters as earlier, but run the
optimisation for longer and test it against stronger attacks. For all
experiments, the output dimension is $D=100$, corresponding to multiplying the
number of classes by a factor of ten.

We benchmark our results against the following results from literature: (i)
the initial results for adversarial training from Madry et al. (2017), (ii)
the improved results for adversarial training from Rice, Wong, and Kolter
(2020) which use early stopping to prevent overfitting in adversarial
training, (iii) the work of Zhang et al. (2019b) which trades more accuracy on
natural samples in order to gain robustness, and (iv) the work of Mustafa et
al. (2019) which use class prototypes jointly optimised during training, and
where the inter-class separation constraints are applied to multiple layers,
and paired with adversarial training. We note that Zhang et al. report the
highest robustness. However, Rice, Wong, and Kolter showed that early stopping
improves robustness, and reduces the gap between Madry et al. and Zhang et
al., while also preserving more accuracy on natural samples.

While Madry et al. and Mustafa et al. use PGD adversarial training with $n=7$,
Zhang et al. and Rice, Wong, and Kolter use $n=10$. As mentioned above,
adversarial training adds a non-trivial overhead, and a higher $n$ further
increases it. While faster methods to perform adversarial training exist, they
achieve _at most_ similar results to classical adversarial training.
Therefore, we compare our results with the state-of-the-art for classical
adversarial training. Since our method does not add any significant overhead
to training, whenever we discuss the impact on training we compare with Wong,
Rice, and Kolter (2020), which is (at the moment) the fastest way to perform
adversarial training, albeit not stable (Andriushchenko and Flammarion 2020).

The results are presented in Table 2, where the acronyms follow the order
above: (i) Madry (Madry et al. 2017), (ii) Early Stop (Rice, Wong, and Kolter
2020), (iii) TRADES (Zhang et al. 2019b), (iv) RHS (Mustafa et al. 2019). The
Regular run was trained on natural samples with the softmax cross-entropy
loss, and a multi-step learning rate scheduler that starts from $0.1$ and
decays by a factor of $0.1$ at epochs 50 and 100. For the models in literature
we present the reported results, since with the exception of Rice, Wong, and
Kolter the results could not be reproduced precisely.

Run | Ep. | Natural |  | PGD
---
20
| PGD
---
100
| Adv.
---
Training
Regular1 | 120 | 93.7 | 0 | 0 | None
Repulsive1 | 127 | 92.0 | 51.3 | 48.4 | None
Madry2 | 200 | 87.2 | 45.8 | - | PGD-7
Early Stop3 | 100 | 86.1 | 56.1 | - | PGD-10
TRADES3 | 100 | 84.9 | 56.6 | - | PGD-10
RHS4 | 300 | 91.8 | 42.6 | - | PGD-7
Table 2: CIFAR-10 results, $\epsilon=8$. The models are 1ResNet-18,
2PreActResNet-18, 3WideResNet-34-10, 4ResNet-110. Ep. abbreviates the number
of training epochs.

For the model trained using repulsive prototypes (the Repulsive run) we report
the robustness against the PGD attack with 20 and 100 iterations. During
training, the cyclical learning rate was reduced by a factor of ten compared
to Smith (2017). We found that using smaller learning rates benefits
robustness and has little impact on natural accuracy. The reason for this is
that larger updates may push the samples closer to the decision boundaries,
where it is easier for adversarial perturbations to induce undesirable
behaviour. For all models we also report the accuracy on natural samples, the
number of epochs needed to reach the results and the architecture used for
training.

We observe that training with repulsive prototypes yields higher accuracy on
natural samples (92%) than methods based on adversarial training, and
competitive robustness (51.3%) compared with the state-of-the-art (56.6%), at
a relatively small increase of training epochs (+27). This is a gain even for
Wong, Rice, and Kolter (2020), which uses the FGSM attack and thus requires at
least two forward and backward passes at each epoch. Moreover, TRADES and
Early Stop use $n=10$ for adversarial training (which increases robustness
over Madry), and use a WideResNet-34-10 architecture, which has over
$34*10^{6}$ more training parameters than ResNet-18. Both capacity and a
higher $n$ are known to increase robustness (Madry et al. 2017).

Figure 2 illustrates the robustness curve obtained by testing the models with
PGD $n=20$, against different perturbation sizes. For comparison, we use the
Early Stop model by Rice, Wong, and Kolter, the only one for which the results
could be reproduced with precise accuracy. We observe that training with
repulsive prototypes yields models which are more resilient to higher
perturbations than adversarially trained models (equivalent to a milder slope
in Figure 2). Moreover, the overall decrease in accuracy is significantly
smaller for models trained with repulsive prototypes; preserving more than 60%
of the initial robustness when the perturbation size is doubled.

### CIFAR-100

Run | Ep. | Natural |  | PGD
---
20
| PGD
---
100
| Adv.
---
Training
Regular1 | 120 | 73.8 | 0 | 0 | None
Repulsive1 | 106 | 71.7 | 20.5 | 20.0 | None
Madry2 | 200 | 59.8 | 22.6 | - | PGD-7
Early Stop2 | 100 | 52.7 | 28.1 | - | PGD-10
Early Stop-R2 | 100 | 54.1 | 20.8 | - | PGD-10
RHS3 | 300 | 68.3 | 20.2 | - | PGD-7
Table 3: CIFAR-100 results, $\epsilon=8$. The models are are 1ResNet-18,
2PreActResNet-18, 3ResNet-110. Ep. abbreviates the number of training epochs.

We perform and report complementary experiments on the CIFAR-100 data set. The
key difference between the two is that the number of classes increases by a
factor of ten. Therefore, the output space partitioning is more challenging.

Moreover, since the last fully connected layer of ResNet-18 has 512 nodes, we
use a multiplicative factor of 50 instead of 100 for $D$ in order to preserve
a possible compression in the last layer, as for CIFAR-10. Experiments with
different multiplicative factors, as those discussed in Table 1, are available
in the project’s repository.

The results are presented in Table 3, with the notable difference that TRADES
was not tested on this data set neither in the original paper (Zhang et al.
2019b) or in the Early Stop paper (Rice, Wong, and Kolter 2020). Moreover, for
Early Stop we could not reproduce the results reported in the paper.
Therefore, we also report on a new benchmark, Early Stop-R, which is obtained
using the model parameters shared in the project’s repository by Rice, Wong,
and Kolter. Also note that for CIFAR-100 Early Stop uses the PreActResNet-18
architecture instead of WideResNet.

We observe that training with repulsive prototypes yields significantly higher
accuracy on natural samples (71.7%) compared with adversarial training
methods, where the maximum is achieved by Madry et al. (2017) (59.8%).
Moreover, competitive robustness (20.5%) with adversarial training (22.6%) can
be observed, at almost no increase in training epochs (+6). The results
reported for Early Stop by Rice, Wong, and Kolter show 7.6% more robustness
than training with repulsive prototypes, at the cost of losing 17% accuracy on
natural samples. A similar result can be observed for CIFAR-10, which
indicates that training with repulsive prototypes trades less accuracy on
natural samples, at the cost of a modest contraction in robustness.

Similarly to CIFAR-10, we present in Figure 3 the robustness curve obtained by
testing with different perturbation sizes. We compare the results with the
Early Stop-R model, which uses the final parameters published by Rice, Wong,
and Kolter. We observe that, as for CIFAR-10, training with repulsive
prototypes yields models resilient to large perturbation sizes, preserving
more than half of the initial robustness when increasing the perturbation by a
factor of two.

Figure 3: Robustness curves for CIFAR-100, obtained by testing with PGD-20,
and various perturbation sizes ($\epsilon$).

### Black-box evaluation

Besides the white-box threat model investigated above, we evaluate the models
in a black-box scenario. In particular, we use the transferability attack, in
which an attacker trains a substitute model and uses it to craft adversarial
examples.

Black-box attacks are used to evaluate the model’s robustness, but also to
detect if the defences employed give a false sense of security – e.g., due to
obfuscating gradients (Athalye, Carlini, and Wagner 2018). Since training with
repulsive prototypes does not add any transformation or randomisation which
may have adverse effects (such as gradient obfuscation), we expect the defence
to behave similarly to adversarial training – a defence known to have no side
effects. Therefore, we compare transferability on repulsive prototypes with
transferability on adversarially trained models.

Su et al. (2018) showed that the architecture can impact transferability.
Particularly when the network’s building blocks are different (e.g., between
the Inception architecture which uses different filter sizes and ResNet which
uses invariant filter sizes and residual connections), robustness has higher
variance. However, when the building blocks are the same, but the depth of the
network increases (e.g., ResNet-50 vs. ResNet-101), robustness has smaller
variance.

Therefore, since all the models from this paper use a variant of the ResNet
architecture, we use a substitute model based on it. We also assume that an
attacker has access to the training data set and does not need to apply data
augmentation (Papernot, McDaniel, and Goodfellow 2016). For crafting
adversarial examples we use the Regular model from Tables 2 and 3. For
testing, we use the PGD attack with $n=20$ and compare with the Early Stop and
Early Stop-R models.

Figure 4: Overfitting in training with repulsive prototypes. Source | Target | CIFAR-10 | CIFAR-100
---|---|---|---
Regular1 | Repulsive1 | 72.5 | 36.2
Regular1 | Early Stop2 | 75.1 | -
Regular1 | Early Stop-R3 | - | 37.4
Table 4: Robustness against transferred adversarial examples. The models are
1ResNet-18, 2WideResNet-34-10, 3PreActResNet-18

The results for both CIFAR-10 and CIFAR-100 are presented in Table 4. We
observe that (i) robustness against black-box attacks is higher than
robustness against white-box attacks, which indicates that both defences have
no adverse side effects such as obfuscated gradients, and (ii) both models
achieve similar robustness, consistent with the results from Tables 2 and 3.
Since for CIFAR-10 Early Stop uses a more complex architecture, we expect the
model to also have higher robustness (which corresponds with the results from
Su et al. (2018), and the higher gap in Table 4).

(a)

(b)

(c)

Figure 5: Adversarial confusion matrices for the (a) Early Stop, and for the
(b) Repulsive models in Table 2. Figure (c) shows misclassified samples (left)
and the nearest correctly classified samples in the representation space
(right).

However, since the target models have different architectures and loss
functions, we expect them to also have distinct internal representations.
Therefore, the transferability results should have higher variance than those
from Tables 2 and 3. Yet the results from Table 4 suggest that the target
models have similar failing modes, and raise the question if some samples are
more sensitive to perturbations than others, and if the samples are common
between the target models. An enquiry follows in the next section.

##  5 Discussion

Firstly, we investigate if robustness is linked to properties of the testing
data set, or of certain samples. Following the observation from the last
section – that models trained with repulsive prototypes and with adversarial
examples behave similarly against black-box attacks – we plot the confusion
matrices for adversarial examples on CIFAR-10, for the Repulsive and Early
Stop models (Figures LABEL:fig:ofit_cm and LABEL:fig:proto_cm).

We observe that 60% of the top-1 misclassified classes are the same for both
models (e.g., planes misclassified as ships). This percentage increases to 90%
when we judge the top-2 classes (e.g., planes misclassified as ships or
birds). We also analysed the overlap between misclassified adversarial
examples by the two models and found that over 65% of the samples were common.
This result indicates that some samples may be more sensitive to
perturbations. For the Repulsive model, it indicates that for some samples
training fails to provide intra-class compactness. When perturbed, these
samples are easier to move to incorrect regions.

In order to further investigate this phenomenon, we performed random sampling
on the misclassified examples for manual inspection. For all samples, we also
extracted the closest examples from the predicted (wrong) class. Two such
examples are displayed in Figure LABEL:fig:adv_ex_samples, where the pictures
on the left are the incorrectly classified examples and the ones on the right
are the closest examples in the predicted class. We observe that both examples
have many common characteristics with the closest sample in the incorrect
classification regions. Similar results could be observed for other samples
(suppressed due to space constraints). Previously, Jo and Bengio (2017) showed
that neural networks have a tendency to learn surface regularities rather than
higher-level abstractions. Our initial investigation indicates that samples
with similar surface regularities are also more sensitive to adversarial
perturbations, even if the models using them are trained with distinct loss
functions. A deeper investigation into this phenomenon is planned for future
work.

Secondly, we note that training with repulsive prototypes for adversarial
robustness is more sensitive than pursuing the highest accuracy on natural
samples. This is a consequence of the loss function (Eq. 6) which measures a
distance, and taking large steps towards minimising it may push samples closer
to the classification boundaries, rather than closer to the prototypes. Larger
steps are not relevant for natural samples, but are important for robustness.
In order to alleviate this effect we used smaller learning rates and early
stopping. However, this also means that models trained with repulsive
prototypes are prone to overfitting for robustness. Figure 4 shows the
behaviour of the Repulsive models trained on both data sets studied. We
observe that, while the models are stable on natural samples, they are prone
to overfitting against adversarial examples. Moreover, the models seem likely
to overfit faster when trained with repulsive prototypes than with adversarial
training (Rice, Wong, and Kolter 2020).

Lastly, we note that prototypes designed prior to training can also embed
other properties in the output space. For example, Mettes, van der Pol, and
Snoek used word2vec (Mikolov et al. 2013) to designed prototypes. Adding more
structure to the output space may lead to higher abstractions – e.g., to
compositionality, as in word2vec – and it is an interesting avenue for future
work.

##  6 Conclusions and Future Work

We introduce deep repulsive prototypes for adversarial robustness – a training
method which partitions the output space prior to training into prototypes
with large class separation, and train models to preserve it. Repulsive
prototypes help models to gain robustness competitive to adversarial training,
while removing the need to generate adversarial examples. Moreover, models
trained with repulsive prototypes are less sensitive to large perturbations
and trade less accuracy on natural samples for robustness.

Our results indicate that the output space size is important for robustness,
and that test samples with similar surface regularities are more sensitive to
adversarial perturbations. For future work we propose to search for better
ways to design prototypes for robustness, which may embed other properties to
the output space than large inter-class separation. Moreover, we plan to
further investigate if misclassified samples present similar surface
regularities with samples in the predicted class, and find ways to remove the
tendency of neural networks to rely on surface regularities.

## References

  * Andriushchenko and Flammarion (2020) Andriushchenko, M.; and Flammarion, N. 2020.  Understanding and Improving Fast Adversarial Training.  _arXiv:2007.02617_ .
  * Athalye, Carlini, and Wagner (2018) Athalye, A.; Carlini, N.; and Wagner, D. 2018.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  _ICML_ .
  * Carlini et al. (2019) Carlini, N.; Athalye, A.; Papernot, N.; Brendel, W.; Rauber, J.; Tsipras, D.; Goodfellow, I.; Madry, A.; and Kurakin, A. 2019.  On evaluating adversarial robustness.  _arXiv:1902.06705_ .
  * Goodfellow, Shlens, and Szegedy (2014) Goodfellow, I.; Shlens, J.; and Szegedy, C. 2014.  Explaining and harnessing adversarial examples.  _arXiv:1412.6572_ .
  * Guerriero, Caputo, and Mensink (2018) Guerriero, S.; Caputo, B.; and Mensink, T. 2018.  Deep nearest class mean classifiers.  In _ICML, Worskhop Track_.
  * Guo et al. (2017) Guo, C.; Rana, M.; Cisse, M.; and Van Der Maaten, L. 2017.  Countering adversarial images using input transformations.  _ICLR_ .
  * Hess, Duivesteijn, and Mocanu (2020) Hess, S.; Duivesteijn, W.; and Mocanu, D. 2020.  Softmax-based Classification is k-means Clustering: Formal Proof, Consequences for Adversarial Attacks, and Improvement through Centroid Based Tailoring.  _arXiv:2001.01987_ .
  * Huang et al. (2020) Huang, X.; Kroening, D.; Ruan, W.; Sharp, J.; Sun, Y.; Thamo, E.; Wu, M.; and Yi, X. 2020.  A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability.  _Computer Science Review_ 37\.
  * Ilyas et al. (2019) Ilyas, A.; Santurkar, S.; Tsipras, D.; Engstrom, L.; Tran, B.; and Madry, A. 2019\.  Adversarial examples are not bugs, they are features.  In _NeurIPS_ , 125–136.
  * Jin and Rinard (2020) Jin, C.; and Rinard, M. 2020.  Manifold regularization for adversarial robustness.  _arXiv:2003.04286_ .
  * Jo and Bengio (2017) Jo, J.; and Bengio, Y. 2017.  Measuring the tendency of CNNs to learn surface statistical regularities.  _arXiv preprint arXiv:1711.11561_ .
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images .
  * Madry et al. (2017) Madry, A.; Makelov, A.; Schmidt, L.; Tsipras, D.; and Vladu, A. 2017.  Towards deep learning models resistant to adversarial attacks.  _arXiv:1706.06083_ .
  * Mao et al. (2019) Mao, C.; Zhong, Z.; Yang, J.; Vondrick, C.; and Ray, B. 2019.  Metric learning for adversarial robustness.  In _NeurIPS_ , 480–491.
  * Mettes, van der Pol, and Snoek (2019) Mettes, P.; van der Pol, E.; and Snoek, C. 2019.  Hyperspherical prototype networks.  In _NeurIPS_ , 1487–1497.
  * Mikolov et al. (2013) Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013.  Distributed representations of words and phrases and their compositionality.  In _NeurIPS_ , 3111–3119.
  * Mustafa et al. (2019) Mustafa, A.; Khan, S.; Hayat, M.; Goecke, R.; Shen, J.; and Shao, L. 2019.  Adversarial defense by restricting the hidden space of deep neural networks.  In _IEEE ICCV_ , 3385–3394.
  * Nguyen and Tran (2015) Nguyen, P.; and Tran, D. 2015.  Repulsive-svdd classification.  In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_ , 277–288. Springer.
  * Pang et al. (2019) Pang, T.; Xu, K.; Dong, Y.; Du, C.; Chen, N.; and Zhu, J. 2019.  Rethinking softmax cross-entropy loss for adversarial robustness.  _ICLR_ .
  * Pang, Xu, and Zhu (2019) Pang, T.; Xu, K.; and Zhu, J. 2019.  Mixup inference: Better exploiting mixup to defend adversarial attacks.  _ICLR_ .
  * Papernot et al. (2016) Papernot, N.; Faghri, F.; Carlini, N.; Goodfellow, I.; Feinman, R.; Kurakin, A.; Xie, C.; Sharma, Y.; Brown, T.; Roy, A.; et al. 2016.  Technical report on the cleverhans v2. 1.0 adversarial examples library.  _arXiv:1610.00768_ .
  * Papernot and McDaniel (2018) Papernot, N.; and McDaniel, P. 2018.  Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning.  _arXiv:1803.04765_ .
  * Papernot, McDaniel, and Goodfellow (2016) Papernot, N.; McDaniel, P.; and Goodfellow, I. 2016.  Transferability in machine learning: from phenomena to black-box attacks using adversarial samples.  _arXiv:1605.07277_ .
  * Rice, Wong, and Kolter (2020) Rice, L.; Wong, E.; and Kolter, J. Z. 2020.  Overfitting in adversarially robust deep learning.  _ICML_ .
  * Schiilkop, Burgest, and Vapnik (1995) Schiilkop, P.; Burgest, C.; and Vapnik, V. 1995.  Extracting support data for a given task.  In _Proceedings, First International Conference on Knowledge Discovery & Data Mining. AAAI Press_, 252–257.
  * Smith (2017) Smith, L. N. 2017.  Cyclical learning rates for training neural networks.  In _IEEE WACV_ , 464–472. IEEE.
  * Smith (2019) Smith, L. N. 2019.  A Useful Taxonomy for Adversarial Robustness of Neural Networks.  _arXiv:1910.10679_ .
  * Snell, Swersky, and Zemel (2017) Snell, J.; Swersky, K.; and Zemel, R. 2017.  Prototypical networks for few-shot learning.  In _NeurIPS_ , 4077–4087.
  * Su et al. (2018) Su, D.; Zhang, H.; Chen, H.; Yi, J.; Chen, P.-Y.; and Gao, Y. 2018.  Is Robustness the Cost of Accuracy?–A Comprehensive Study on the Robustness of 18 Deep Image Classification Models.  In _ECCV_ , 631–648.
  * Tramer et al. (2020) Tramer, F.; Carlini, N.; Brendel, W.; and Madry, A. 2020.  On adaptive attacks to adversarial example defenses.  _arXiv:2002.08347_ .
  * Wang, Neskovic, and Cooper (2005) Wang, J.; Neskovic, P.; and Cooper, L. N. 2005.  Pattern classification via single spheres.  In _International Conference on Discovery Science_ , 241–252. Springer.
  * Wong, Rice, and Kolter (2020) Wong, E.; Rice, L.; and Kolter, J. Z. 2020.  Fast is better than free: Revisiting adversarial training.  _ICLR_ .
  * Xie et al. (2020) Xie, C.; Tan, M.; Gong, B.; Yuille, A.; and Le, Q. V. 2020.  Smooth adversarial training.  _arXiv preprint arXiv:2006.14536_ .
  * Zhang et al. (2019a) Zhang, H.; Chen, H.; Song, Z.; Boning, D.; Dhillon, I. S.; and Hsieh, C.-J. 2019a.  The limitations of adversarial training and the blind-spot attack.  _ICLR_ .
  * Zhang et al. (2019b) Zhang, H.; Yu, Y.; Jiao, J.; Xing, E. P.; Ghaoui, L. E.; and Jordan, M. I. 2019b.  Theoretically principled trade-off between robustness and accuracy.  _ICML_ .

  *[PGD]: projected gradient descent
  *[FGSM]: Fast Gradient Sign Method