# Explaining Adversarial Vulnerability with a Data Sparsity Hypothesis

Mahsa Paknezhad
Cuong Phuc Ngo
Amadeus Aristo Winarto
Alistair Cheong
Beh Chuen Yang
Wu Jiayang
Bioinformatics Institute, A*STAR,
30 Biopolis Street, 07-01, Matrix
Singapore, 138671
<EMAIL_ADDRESS>
&Lee Hwee Kuan
Bioinformatics Institute, A*STAR,
30 Biopolis Street, 07-01, Matrix
Singapore, 138671
National University of Singapore
Singapore, 119077
Singapore Eye Research Institute
20 College Road
Singapore, 169856
<EMAIL_ADDRESS>
###### Abstract

Despite many proposed algorithms to provide robustness to deep learning (DL)
models, DL models remain susceptible to adversarial attacks. We hypothesize
that the adversarial vulnerability of DL models stems from two factors. The
first factor is data sparsity which is that in the high dimensional data
space, there are large regions outside the support of the data distribution.
The second factor is the existence of many redundant parameters in the DL
models. Owing to these factors, different models are able to come up with
different decision boundaries with comparably high prediction accuracy. The
appearance of the decision boundaries in the space outside the support of the
data distribution does not affect the prediction accuracy of the model.
However, they make an important difference in the adversarial robustness of
the model. We propose that the ideal decision boundary should be as far as
possible from the support of the data distribution.

In this paper, we develop a training framework for DL models to learn such
decision boundaries spanning the space around the class distributions further
from the data points themselves. Semi-supervised learning was deployed to
achieve this objective by leveraging unlabeled data generated in the space
outside the support of the data distribution. We measure adversarial
robustness of the models trained using this training framework against well-
known adversarial attacks We found that our results, other regularization
methods and adversarial training also support our hypothesis of data sparcity.
We show that the unlabeled data generated by noise using our framework is
almost as effective as unlabeled data, sourced from existing data sets or
generated by synthesis algorithms, on adversarial robustness. Our code is
available at https://github.com/MahsaPaknezhad/AdversariallyRobustTraining.

##  1 Introduction

Deep learning (DL) models have achieved unprecedented accuracy for many visual
recognition [1], speech recognition [24], and natural language processing
tasks [25]. As DL models find their way to real-world applications, an
important vulnerability in them was brought into the limelight by the
introduction of adversarial examples, generated by applying imperceptible
perturbations to the original input [26]. In a white-box setup where the
adversary has access to weight parameters of the DL model, different attack
algorithms such as Fast Gradient Sign Method (FGSM) [4], Projected Gradient
Descent (PGD) [21], and DeepFool [12] already exist. Adversarial training is
currently one of the most effective approaches to improve robustness. Efforts
to explain the adversarial vulnerability of DL models suggest that correct
classification only occurs on a thin manifold and most of the high dimensional
input data distribution consists of adversarial examples [4]. We hypothesize
that two factors play important roles in adversarial vulnerability of DL
models. The first factor is that a large portion of data space lies outside
the support of the data distributions, i.e. no nearby training data points.
The second factor is that DL models have many redundant parameters that allow
the models to learn different possible decision boundaries, especially outside
the support of the data space. These decision boundaries in the space may not
affect the prediction accuracy of the model, but makes an important difference
for adversarial robustness. We hypothesize that an ideal decision boundary is
smooth and will stay as far from the support of each class distribution as
possible.
Our hypothesis about the existence of large factions in data space without
training data is similar to that in the ongoing independent efforts on Open
Set Recognition (OSR) [27], whose objective is to train models that can
identify data points from seen classes in the training data accurately, or
Known Known Classes (KKCs), while rejecting data points from unseen classes,
or Unknown Unknown Classes (UUCs), that are passed into the model during
testing. In these algorithms, it is assumed that UUCs are located in a
subspace far from KKCs in the high dimensional input data space and the model
is trained to have decision boundaries that bind the support of each KKC
distribution tightly in order to reserve the space outside the support of the
data distribution for UUCs. Our code is provided together with the
Supplementary Materials.
Semi-supervised learning (SSL) has been widely used to train models with
higher accuracy when limited labeled data is available. SSL has also recently
been deployed to train adversarially robust DL models, by taking advantage of
unlabeled data sets to regularize the model. For unlabeled data, these methods
generate data points using Gaussian noise or synthesis algorithms, or source
data points from existing data sets. The theory is that the unlabeled data
provides more points in the high dimensional data space at which the gradient
of the SoftMax score surface can be regularized. In this paper, SSL is used by
leveraging unlabeled data, carefully generated to cover a large area around
the support of the data distributions. The unlabeled data and the proposed
training framework support our goal of training models with decision
boundaries similar to the ideal decision boundary described before.

Our contributions are listed as follows:

  1. 1.

We hypothesize that models with smooth SoftMax surfaces whose decision
boundaries lie far from the support of each class distribution are more robust
to adversarial attacks.

  2. 2.

We develop a training framework for DL models that encourage decision
boundaries to be as far from the support of each class distribution as
possible. The distance of the decision boundary from the support of class
distributions is controlled using unlabeled data generated by Gaussian noise
and regularization.

  3. 3.

We show that unlabeled data generated by Gaussian noise using our framework is
almost as effective as unlabeled data generated by more costly and complex
synthesis algorithms or those sourced from other existing data sets.

##  2 Related Work

In this section, a brief review of related literature on regularization and
SSL is provided.

###  2.1 Regularization Methods

Existing algorithms train DL models with decision boundaries that stay far
from the training data points, have smooth gradients or small curvatures[20].
Yan et al. [17] incorporated a DeepFool attack module in training to maximize
the distance between the decision boundary of the trained model and the
training data points. They added the regularization term
$\lambda\sum_{k}R(-{||\Delta_{x_{k}}||_{p}}/{||x_{k}||_{p}})$ with a weight
parameter $\lambda$ to the natural error in the loss function and jointly
optimized the model over the loss function during training. The regularization
term is a scale of $\Delta_{x_{k}}$ which is the minimum adversarial
perturbation found by DeepFool for an input image $x_{k}$. $R$ is a
monotonically increasing function. Chang et al. [18] also proposed a
regularization term to increase the distance to the decision boundary by
adding the boundary error
$\max_{x^{\prime}_{k}\in\mathbf{B}(x_{k},\epsilon)}\phi\left(f_{\theta}(x_{k})f_{\theta}(x^{\prime}_{k})/\lambda\right)$
as a regularization term. $B(x,\epsilon)$ represents the neighborhood of input
$x_{k}$, and $\phi$ is a surrogate of 0-1 loss for binary classification. The
regularization term minimizes the difference between the prediction for the
natural image $f_{\theta}(x_{k})$ and the adversarial example
$f_{\theta}(x^{\prime}_{k})$. Jakubovitz et al. [5] added the regularization
term
$\lambda\sqrt{\sum_{i=1}^{d}\sum_{j=1}^{n}\sum_{k=1}^{N}\left(\frac{\partial}{\partial
x_{i}}z^{(L)}_{j}(x_{k})\right)^{2}}$ which is the Euclidean norm of the
network’s Jacobian matrix evaluated on input $x_{k}$. The regularization is
applied during extended training of the model after normal training. $d$ is
the dimension of the input $x_{k}$, $n$ is the dimension of the output of
$f_{\theta}(x_{k})$, and $N$ is the number of data points in the data set. The
reason for choosing a Jacobian matrix is that the Jacobian is related to the
curvature of the model’s decision boundary and the distance to the closest
adversarial example. Moosavi et al. [20] provide theoretical evidence that a
small curvature in the loss function increases the robustness of the model.
They propose a regularization term that reduces the curvature of the loss
function by minimizing eigen values for the Hessian matrix with respect to the
inputs: $\lambda\frac{1}{h^{2}}\mathbf{E}||\nabla l(x_{k},hz)-\nabla
l(x_{k})||^{2}$. Here, $h$ is the discretization step over $z\sim N(0,Id)$.
Ross and Doshi-velez [31] use adversarial training and distillation [34] as a
regularization mechanism and use the regularization term
$\lambda\sum_{i=1}^{d}\sum_{k=1}^{N}\left(\frac{\partial}{\partial
x_{i}}\sum_{j=1}^{n}-y_{kj}\log f_{\theta}(x_{k})_{j}\right)^{2}$ which
ensures that if the input changes slightly, the KL divergence between the
predictions and the labels does not change significantly[22].

###  2.2 Semi-Supervised Learning

Semi-supervised learning (SSL) has been used to improve the accuracy of DL
models [13]. SSL algorithms have also been used for consistency training,
which regularize model predictions to be resilient against noise injections
into input examples or hidden states. Xie et al. [2] substitute noise
injection methods with advanced data augmentation methods to generate
unlabeled data and use it together with labeled data for consistency training.
Miyato et al. [3] extend adversarial training [4] to use both labeled and
unlabeled data points for regularizing the conditional label distribution
around each input data point. Zhai et al. [8] utilize SSL for adversarially
robust generalization by using portions of the MNIST and CIFAR10 dataset as
unlabeled data. They train the model to be accurate on the labeled data and
robust on both labeled and unlabeled data. Uesato et al. [9] utilize the 80
Million Tiny Images data set as the unlabeled data for the labeled CIFAR10
data set, which led to a 4% improvement in the robust accuracy of the trained
model. They also utilized portions of CIFAR10 and SVHN as unlabeled data
points for these data sets to improve the adversarial robustness of the
models. It was shown that models trained with smaller labeled data sets, with
most labels from CIFAR10 and SVHN removed, are more adversarially robust.
Najafi et al. [10] removed labels from portions of the training data set and
assigned soft labels to the unlabeled data according to an adversarial loss,
and trained these soft-labeled images with the labeled images. Carmon et
al.[11] also augmented CIFAR10 with 500K unlabeled images from the 80 Million
Tiny Images data set. On SVHN, they used the data set’s own extra training set
as the unlabeled images. In these studies except for noise injection methods,
the unlabeled data is captured in a similar expansive manner as the labeled
data.

|
---|---
Non-ideal SoftMax score surface | Ideal SoftMax score surface

Figure 1: (Left) shows a SoftMax score surface that a DL model can learn for
classifying a 2D point data set. A model with such a SoftMax score surface is
more susceptible to adversarial attacks since the the decision boundary gets
close to data distribution and small perturbations such as $\Delta x_{k}$ and
$\Delta x_{l}$ in data points (e.g. $x_{k}$ and $x_{l}$) can make the data
points cross the decision boundary. (Right) shows the what we hypothesize to
be the ideal SoftMax score surface which can improve the adversarial
robustness of the model. In contrast to the top figure, the decision boundary
for this surface stays far away from all class distributions.

##  3 Method

This section presents the proposed framework for training DL models to learn
decision boundaries similar to our hypothesized ideal decision boundary. We
will first present the overall objective. Later, we introduce the notations
used in the paper. Finally, the data generation and model training steps are
described.

###  3.1 The Ideal Decision Boundary

Fig. 1 shows a binary classification problem for the 2D points shown in the
figure. Also shown is the schematic representation of the SoftMax score
surface for two trained DL models on the 2D points using contour plots. The
top figure shows the SoftMax values for a DL model, which has a decision
boundary that is hypothesized to be non-ideal. Due to the short distance of
the decision boundary to the data points, an attack model will be able to find
a small perturbation such as $\Delta x_{l}$ and $\Delta x_{k}$ for each data
point (e.g. $x_{k}$ and $x_{l}$, respectively) that will make the perturbed
data point cross the decision boundary and consequently be assigned to the
other class label.

In contrast, a SoftMax score surface shown in the bottom plot is hypothesized
to be more robust against adversarial attacks. The SoftMax score surface spans
outwards in the space around the data distribution and gradually changes from
zero to one from one class to the other. This is desirable as it keeps the
decision boundary as far as possible from both class distributions, making it
more difficult for adversarial attacks to fool the model via small
perturbations.

In Fig. 1 (Top), the SoftMax score surface that the model learned for the
simple 2D Point data set is very steep around the decision boundary and the
decision boundary gets close to the clean data points. A SoftMax score surface
that we propose to help improve adversarial robustness (Fig. 1) has two
distinct characteristics:

  1. 1.

The first characteristic is the smooth gradient of the SoftMax score surface.
The advantage of SoftMax score surfaces with smooth gradients has been studied
and is well-known [5, 7].

  2. 2.

The other characteristic is spreading of the SoftMax scores in the space away
from the class distributions. This characteristic is very beneficial as will
be explained in the following paragraphs.

The SoftMax score surface spreads outward in the space around the data
distribution, resulting in a decision boundary that stays far from all class
distributions. Consequently, it will be more difficult for the attack model to
find a small perturbation that changes the predicted label for the clean data
point.

Another advantage of having such a decision boundary is in the behaviour of
the model when it receives a data point that does not belong to any of the
classes that the model was trained on (UUCs). For instance, consider the model
with the non-ideal SoftMax score surface in Fig. 1. If the model receives a
data point during testing from a new class that it has not been trained on,
the distribution from which that data point is sampled from is far from the
data distribution the model is trained on. If the data distribution for the
unseen class is at the right side of the decision boundary in Fig. 1, the
model will label the new data point with the label of the class distribution
on the right with a very high confidence (e.g $>$ 90% ), whereas the proposed
ideal SoftMax score surface in Fig. 1 assigns label of the class distribution
on the right with a lower confidence (e.g. 60%). We show experimental results
on the confidence of models trained using our framework compared to normal
training in the Supplementary Materials section.

Training a DL model to learn such an ideal SoftMax score surface is achieved
by sampling unlabeled data points from a Gaussian distribution in a much
larger neighborhood of the labeled data distribution with the hope that the
generated unlabeled data set covers a large faction of the high dimensional
input data space outside the support of the data distribution. We use both the
unlabeled and labeled data set to train the model to have small gradients in
the high dimensional input data space covered by the labeled and unlabeled
data sets. To enforce small gradients, neighbor data points are sampled from
close proximity of a labeled or an unlabeled data point. The gradients of the
SoftMax score surface between the labeled or unlabeled data point and the
corresponding neighbor points are enforced to be small by training the model
with a well-defined loss function.

###  3.2 Notations

Consider an $n$-class classification problem for learning a predictor
$f_{\theta}$ to map inputs $x_{i}\in X\subset\mathbb{R}^{d}$ to labels
$y_{i}\in Y=\\{1,\cdots n\\}$ where $d$ is the dimension of the data space. In
this case, $f_{\theta}$ will be of the form: $f_{\theta}(x)=\arg\max_{y\in
Y}p_{\theta}(y|x)$. The labeled data set is specified by
$D=\\{\left(x_{i},y_{i}\right)|x_{i}\in X,y_{i}\in Y,i=1,\cdots,N_{l}\\}$,
$y_{i}$ is the ground-truth labels, and $N_{l}$ is the number of labeled data
points. The generated unlabeled data set is specified by
$U=\\{u_{i,j}\in\mathbb{R}^{d}|i=1,\cdots,N_{l},j=1,\cdots,N_{u}\\}$ where
$u_{i,j}$ is the $j^{th}$ unlabeled data point generated using the $i^{th}$
labeled data point, and $N_{u}$ is the number of generated unlabeled data
points using one labeled data point. No ground-truth label is known for the
unlabeled data points. The unlabeled data points are sampled from the same
data space as the labeled data points, hence $u_{i,j}\in\mathbb{R}^{d}$. We
concatenate the sets $X$ and $U$ and define $\Psi=\\{\psi_{k}|\psi_{k}\in
X\cup U,k=1,\cdots,N_{l}+N_{l}\times N_{u}\\}$. To specify if a data point is
a labeled or an unlabeled data point an identifier denoted by $l_{k}$ is
defined as follows:

$l_{k}=\begin{cases}1,&\text{if $\psi_{k}$ is labeled }(\psi_{k}\in X)\\\
0,&\text{otherwise }(\psi_{k}\in U)\end{cases}$ (1)

Hence, a new data set is generated:

$D_{new}={\begin{cases}(\psi_{k},l_{k},y_{k})&\text{if }l_{k}=1(\psi_{k}\in
X)\\\ (\psi_{k},l_{k})&\text{if }l_{k}=0(\psi_{k}\in U).\end{cases}}$ (2)

In the next section, we explain how unlabeled data is generated.

###  3.3 Generating Unlabeled Data Points

Utilizing unlabeled data points to train adversarially robust neural networks
is not a new idea as explained in the Section 2.2. In many domains such as
cancer research, obtaining even an uncurated data set of images can be
expensive and time consuming. Preparation and scanning of a single sample of
cancer tissue can cost hundreds to thousands dollars. Also, there is no
guarantee that the acquired data set will cover a large portion of the data
space around the support of the labeled data distribution. Our algorithm
generates unlabeled data points very cheaply from the labeled data set using
simple additive Gaussian noise. Unlabeled data points were generated from the
labeled data points by adding noise to the labeled data points sampled from an
isotropic Gaussian distribution. The Gaussian distribution was uniform in all
input data dimensions, and the standard deviation of the Gaussian distribution
was thrice the mean of the pair-wise Euclidean distances between data points
($\mu_{pair}$) in the labeled data set. More precisely, the co-variance matrix
could be represented by the simplified matrix $\sigma^{2}I$ with $\sigma$
equal to $3\times\mu_{pair}$. For each labeled data point $x_{i}$, a series of
$N_{u}$ unlabeled data points $\\{u_{i,j}|j=1,\cdots,N_{u}\\}$ were sampled
from the area around the labeled data point in $\mathbb{R}^{d}$. Unlabeled
data points were generated once only for the labeled data set.

Thus, the unlabeled data set is generated to cover a large data space around
the labeled data distribution. Using this unlabeled data set together with the
labeled data set, the model is trained to have a SoftMax score surface with a
decision boundary that is away from the support of each class distribution. We
will show in Section 4.3 that using Gaussian noise is enough to achieve a
smooth SoftMax score surface with a decision boundary similar to the ideal
decision boundary explained before.

###  3.4 Training the CNN

During the training process, a data point, $\psi_{k}$, was selected randomly
from the new data set $D_{new}$ (See Eq. 2). Similar to how the unlabeled data
points were generated, for the selected data point a specific number of
neighbor data points, $N_{b}$ (similar to $N_{u}$ for unlabeled data points),
were generated by adding $N_{b}$ noises sampled from an isotropic Gaussian
distribution with 10-times smaller standard deviation that that for generating
unlabeled data. Neighbor data points were generated on the fly for
regularization purposes. Hence, they were not part of the new data set. These
neighbor data points are denoted by $\\{\psi_{k,m}|m=1,\cdots,N_{b}\\}$. The
selected data point was then fed into the model together with the generated
neighbors, as shown in Fig. 2 as a mini-batch.

The neighbor data points were generated and passed with the data point
together into the model to ensure that the model learns a locally smooth
SoftMax score surface around the selected data point in the input data space.
In other words, for the selected data point $\psi_{k}$ and a single neighbor
data point $\psi_{k,m}$, which was generated by adding small noise to
$\psi_{k}$, the model would ideally output similar predictions, denoted by
$f_{\theta}(\psi_{k})$ and $f_{\theta}(\psi_{k,m})$, respectively. This
requirement was enforced by smoothening the slope of the SoftMax score surface
between the data point $\psi_{k}$ and each of its neighbor points
$\psi_{k,m}$. This is our regularization term, $Loss_{reg}$, and is defined
as:

$\displaystyle
Loss_{reg}(\psi_{k},\\{\psi_{k,m}|m=1,\cdots,N_{b}\\},\theta)=\sum_{m=1}^{N_{b}}\frac{||f_{\theta}(\psi_{k,m})-f_{\theta}(\psi_{k})||_{p}}{||\psi_{k,m}-{\psi_{k}}||_{p}},\forall\psi_{k}\in
X\cup U$ (3)

Figure 2: The input training batch to the CNN model. In each batch, a data
point, $\psi_{k}$, together with $N_{b}$ generated neighbor data points,
$\\{\psi_{k,m}|m=1,\cdots,N_{b}\\}$, are fed to the model. The model is
trained by minimising the loss function 5 applied on the outputs of the model.

where $||.||_{p}$ is any norm function. In our experiments in Section 4,
$l^{2}$ norm was used as the norm function. To ensure that the model learns to
correctly classify the selected data points that come from the labeled data
set $\\{\psi_{k}\in X\\}$, a supervised loss function, $Loss_{sup}$, in our
case cross entropy shown as $H$, was considered to compare the predicted
label, $f_{\theta}(\psi_{k})$ for the selected data point with the ground-
truth label $y_{k}$.

$Loss_{sup}(\psi_{k},y_{k},\theta)=H(y_{k},f_{\theta}(\psi_{k})),\forall\psi_{k}\in
X$ (4)

The overall loss function for the CNN model is a weighted combination of the
supervised and regularization loss functions, defined as below. The cross
entropy loss is measured only for the data points that came from the labeled
data set $X$ and is zero otherwise.

$\displaystyle Loss=\sum_{\psi_{k}\in
X}Loss_{sup}(\psi_{k},y_{k},\theta)~{}+\lambda\times\sum_{\psi_{k}\in X\cup
U}Loss_{reg}(\psi_{k},\\{\psi_{k,m}|m=1,\cdots,N_{b}\\},\theta),$ (5)

where $\lambda$ is a hyperparameter. More details on the size of the isometric
Gaussian distributions that are used to sample noise and generate unlabeled
data points and neighbor data points are provided in Supplementary Materials
section. In the next section, we assess our hypothesis using the proposed
training framework. We describe our experiment design and the conclusions that
were drawn from these experiments.

Figure 3: (a) our 2D points data set ($\\{X,Y\\}$). (b) generated unlabeled
data set for the labeled data set is shown in gray ($U$). (c) the SoftMax
score surface learned by a model without regularization and unlabeled data,
(d) the SoftMax score surface learned by the Jacobian Regularization method
[5] which does not use unlabeled data, (e) the SoftMax score surface learned
by our regularization method using the unlabeled data set.

##  4 Experiments

To test our hypothesis about the proposed ideal decision boundary, different
data sets comprising of both 2D points and image data sets including MNIST
[28], CIFAR10 [29], and Imagenette which is a subset of the well-known
ImageNet ILSVRC 2012 data set [30] were used. The following describes the
experiments carried out on each data set.

###  4.1 Effect of the Proposed Regularizer on SoftMax Score Surface for the
2D Points Data set

This experiment was carried out to examine the feasibility of training a DL
model to learn a SoftMax score surface which similar to the example shown in
Fig. 1 (bottom) stays far from each class distribution. A simple neural
network consisting of 4 fully-connected layers with TanH activation functions
was trained to perform binary classification on the 2D points data set shown
in Fig. 3 (a). The unlabeled data points generated for this data set are shown
in gray in Fig. 3 (b). Fig. 3 (c) shows the SoftMax score surface the model
learns without utilizing regularization and unlabeled data. Due to similarity
of the Jacobian Regularization method [5] to our regularization term, we also
show the learned SoftMax score surface by the Jacobian Regularizer which does
not use unlabeled data in Fig. 3 (d). Fig. 3 (e) shows the SoftMax score
surface learned by the model using our training framework. These results show
that the decision boundary learned by the model with normal training get close
to the support of the class distributions at certain locations in the input
data space whereas the model trained using our training framework keeps the
decision boundary away far from both class distributions.

###  4.2 Numerical Results On Distance of Decision Boundary From Data
Distribution

MNIST Data Set: The MNIST data set consists of $60,000$ monochrome images of
handwritten digits in the training set. It has 10 classes which correspond to
the digits 0 to 9 and an equal number of images per class. The images were
resized to $32$, rescaled to the range $[0,1]$ and normalized with mean and
standard deviation of the training data set.

Table 1: Column $\mathbf{\hat{\rho}_{adv}}$ shows measured
$\mathbf{\hat{\rho}_{adv}}$ values using DeepFool attack for the models
trained on MNIST data set. Test accuracy is also shown for each model.

Defence method | $\mathbf{\hat{\rho}_{adv}(\times 10^{-2})}$ | Test accuracy (%)
---|---|---
Other’s | Our’s | Other’s | Our’s
No defence | 20.67 | 20.30 | 99.08 | 98.29
Adversarial training | 22.38 | 61.84 | 99.03 | 98.66
Input Gradient [31] | 23.43 |  | 99.25 |
Input Gradient [31] & | 23.49 |  | 98.88 |
Adversarial training
Jacobian [5] | 34.24 |  | 98.44 |
Jacobian [5] & | 36.29 |  | 98.00 |
Adversarial training
Proposed method |  | 46.29 |  | 98.69
Proposed method & |  | 51.20 |  | 98.80
Adversarial training

The LeNet [28] network architecture and ReLU activation function were used for
all experiments on the MNIST data set. Details of the training hyperparameters
are given in our Supplementary Materials section. The model was trained to
perform classification on the MNIST data set using our proposed loss function
(See Eq. 5). Different values of $\lambda$ were tested and the model’s loss
curve was checked for each $\lambda$ value. The highest $\lambda$ value that
did not decrease the model’s validation accuracy a lot was chosen. We show
adversarial robustness of the trained model with models trained using two
other training approaches which support our hypothesis such as Input Gradient
Regularization [31], and Jacobian Regularization [5] and adversarial training.
Table 1 shows the results obtained using Input Gradient Regularization [31],
Jacobian Regularization [5] and the method proposed in this paper. The
$\hat{\rho}_{adv}$ value defined as
$\hat{\rho}_{adv}(f_{\theta})=\sum_{x_{k}\in X}\frac{1}{|X|}\frac{||\Delta
x_{k}||_{2}}{||x_{k}||_{2}}$ [12] measures the average relative distance
between data points and the decision boundary. The $\hat{\rho}_{adv}$ values
were higher for the proposed method, and when combined with adversarial
training. Input Gradient Regularization [31], Jacobian Regularization [5] and
adversarial training also show larger $\hat{\rho}_{adv}$ value compared to
models trained with no defence mechanism. We have also reported the test
accuracy of each model in the same table. We define test accuracy as the
prediction accuracy of the model on the test data set. The test accuracies
achieved by the models are very close to the test accuracies for models with
no defense.

CIFAR10 Data Set: The CIFAR10 data set contains color images of 10 natural
image classes with equal numbers of images per class.

Table 2: Column $\mathbf{\hat{\rho}_{adv}}$ shows measured
$\mathbf{\hat{\rho}_{adv}}$ values using DeepFool attack for the models
trained on CIFAR10 data set. Test accuracy is also shown for each model.

Defence method | $\mathbf{\hat{\rho}_{adv}(\times 10^{-2})}$ | Test accuracy (%)
---|---|---
Other’s | Our’s | Other’s | Our’s
No defence | 1.21 | 1.23 | 88.79 | 91.17
Adversarial training | 1.23 | 1.27 | 88.88 | 89.51
Input Gradient [31] | 1.43 |  | 88.56 |
Input Gradient [31] & | 2.17 |  | 88.49 |
Adversarial training
Jacobian [5] | 3.42 |  | 89.16 |
Jacobian [5]& | 6.03 |  | 88.49 |
Adversarial training
Proposed method |  | 3.29 |  | 83.96
Proposed method & |  | 4.45 |  | 83.50
Adversarial training

CIFAR10 images were also rescaled to the range $[0,1]$ and normalized using
the mean and standard deviation of the training data set and augmentation
methods were applied during training. For CIFAR10 data set, Resnet 9 [32]
network architecture was used with a CELU activation function for all layers.
The lambda value was chosen in a similar to how it was chosen for the MNIST
data set. Details of the training hyperparameters are given in our
Supplementary Materials section. The model was trained using the proposed loss
function (See Eq. 5) and the generated unlabeled data set. The training and
results comparison were conducted similar to the MNIST experiments. Table 2
shows the results obtained using Input Gradient Regularization [31], Jacobian
Regularization [5] and the method proposed in this paper. The
$\hat{\rho}_{adv}$ values achieved for the proposed algorithm and when
combined with adversarial training are larger compared to models trained with
no defense. Regularization methods and adversarial training also show larger
$\hat{\rho}_{adv}$ values. The test accuracies achieved were slightly lower
using our training framework. We believe the prediction accuracies are
compromised over achieving a smooth SoftMax surface outside the support of the
data distribution in the high dimensional input data space.

Imagenette Data Set: Imagenette is a subset of the well-known ImageNet ILSVRC
2012 data set [30] with 10 easily classified classes. Images were rescaled to
the range $[0,1]$ and normalized using the mean and standard deviation of
images in the training data set. The XResnet 18 [33] network architecture was
used to train a model to classify Imagenette data set with MISH activation
function. Details of the training hyperparameters are given in our
Supplementary Materials section.

Table 3: Column $\mathbf{\hat{\rho}_{adv}}$ shows measured
$\mathbf{\hat{\rho}_{adv}}$ values using DeepFool attack for the models
trained on Imagenette data set. Test accuracy is also shown for each model.

Defence method | $\mathbf{\hat{\rho}_{adv}(\times 10^{-2})}$ | Test accuracy (%)
---|---|---
No defence | 0.941 | 87.18
Adversarial training | 4.82 | 83.06
Proposed method | 2.21 | 84.92
Proposed method & | 4.96 | 82.44
Adversarial training

The training and attacks performed were similar to those in MNIST and CIFAR10.
Table 3 shows the results obtained without regularization, with adversarial
training, and our proposed method. There were no analysis on this data set in
the Input Gradient Regularization [31] and Jacobian Regularization papers to
show with our results. Similar to the previous experiments, our models show
higher $\hat{\rho}_{adv}$ values compared to normal training. When combined
with adversarial training, the model shows the highest $\hat{\rho}_{adv}$.

The experiments on MNIST, CIFAR10 and Imagenette data sets show that our
training framework built on the proposed hypothesis indeed results in models
with larger $\hat{\rho}_{adv}$, which implies that the distance between data
points and decision boundary increases for the models trained with our
training framework. We believe other regularization techniques such as Input
Gradient Regularization [31] and Jacobian Regularization [5] and adversarial
training are indirectly trying to achieve the same objective. The drawback of
these algorithms, however, is that they do not take advantage of unlabeled
data sets during training. In the next section, the effects of different
unlabeled data sets on the adversarial robustness of the model are compared.

Figure 4: Figure shows robustness of two models trained with regularization
using a Gaussian generated data set as the unlabeled and using the 80m@$200k$
data set as the unlabeled data against FGSM (a) and PGD (b) attacks for models
trained with 128, 256, and 512 number of labeled data points per class
($N_{l}$). (c) and (d) show robustness of two models trained with
regularization using a Gaussian generated data set as the unlabeled and using
the UDA generated data set [2] as the unlabeled data against FGSM and PGD
attacks respectively.

###  4.3 Gaussian Noise is as Effective as Costly and More Complex Unlabeled
Data Sets

An important question is whether unlabeled data sets widely utilized in the
literature for SSL could improve the performance of models trained with the
proposed training framework. To answer this question we trained two models for
CIFAR10 data set using our training framework with unlabeled data extracted
from the 80 Million Tiny Images data set following the same automatic
filtering technique used by Xie et al. [2] and Uesato et al.[9]. We also
prepared another unlabeled data set by applying the synthesis technique
proposed by Uesato et al.[9]. We ran experiments for different numbers of
labeled data points per class ($N_{l}=128,256,512$) and the same number of
unlabeled data points per labeled data point ($N_{u}=1$) using these two
unlabeled data sets, and compared the robust accuracies of the trained models
with those of models trained with unlabeled data generated from Gaussian
noise. Robust accuracy is defined as the accuracy of the model in predicting
the correct classes for adversarially perturbed images. Fig. 4 (a) and (b)
compare robust accuracies of models trained with unlabeled data extracted from
the 80 Million Tiny Images data set and our models using FGSM and PGD attacks.
Similarly, Fig 4 (c) and (d) compare robust accuracies of models trained with
unlabeled data sets generated using the synthesis technique proposed in the
work by Uesato et al.[9] with our models for FGSM and PGD attacks. Our models
show slightly but not significantly less robust accuracies, suggesting that
generating unlabeled data sets using Gaussian noise is almost as effective as
more complex and sometimes expensive unlabeled data sets.

|
---|---
(a) | (b)
Figure 5: (a) robust accuracy for models trained on MNIST data set using
regularization with $\lambda=100k$ and $N_{l}=64$ (left), and robust accuracy
for models trained on CIFAR10 data set using regularization with $\lambda=10k$
and $N_{l}=128$ for different number of unlabeled data points per labeled data
points, $N_{u}$ using FGSM, PGD, and DeepFool $(l^{\infty})$ attacks (right).
(b) robust accuracy for models trained on MNIST data set with and without
regularization ($\lambda=100k,0$) with different numbers of labeled data
points, $N_{l}=8,64,256$ (left). (Right) robust accuracy for models trained on
CIFAR10 data set with and without regularization ($\lambda=10k,0$) with
different numbers of labeled data points, $N_{l}$, using FGSM, PGD, and
DeepFool $(l^{\infty})$ attacks.

###  4.4 Effect of Regularization and Unlabeled Data on the Trained Models

This section analyses the effect of regularization and unlabeled data on
adversarial robustness of the models. Fig. 5(a) (Left) shows robust accuracy
for models trained with $N_{u}$ = 0, 5, and 10 with regularization
($\lambda=100k$), $N_{l}=64$, for the MNIST data set while other
hyperparameters were fixed. Five models were trained for each set of
parameters. Three different attack methods (FSGM, PGD, and DeepFool
($l^{\infty}$) were conducted on the models, and the mean and standard error
of the robust accuracies were measured. We show adversarial robustness using
DeepFool ($l^{\infty}$) attack because DeepFool ($l^{2}$) shows only small
decrease in adversarial robustness of the models. All trained models had
similar validation accuracies. As shown, models trained with larger $N_{u}$
show higher robust accuracies. Models trained with and without regularization
($\lambda=100k,0$) were also compared for different numbers of labeled data
points per class ($N_{l}=8,64,256$) with other hyperparameters fixed. Five
different models were trained similarly per set of parameters. All trained
models had similar validation accuracies. Fig. 5 (b) (Left) shows the mean and
standard error for the robust accuracies measured for the trained models
suggesting that models trained with regularization were more robust compared
to models trained without regularization. We ran similar experiments on the
CIFAR10 data set. Fig. 5 (a) (Right) shows the robust accuracy of models
trained with $N_{u}=$ 1, 5, and 10, $\lambda=10k$, $N_{l}=128$ with other
hyperparameters fixed for CIFAR10 data set. All trained models had similar
validation accuracies. As shown, the models trained with larger $N_{u}$ show
higher robust accuracies. Fig. 5 (b) (Right) shows the mean and standard error
for the robust accuracies for models trained with and without regularization
($\lambda=10k,0$) and different number of labeled data points per class
($N_{l}=128,256,512$) with other hyperparameters fixed. For all 3 parameter
settings, models trained with regularization were more robust than models
trained without regularization. The experiments show that regularization and
unlabeled data both have positive influence on adversarial robustness of
trained models.

##  5 Conclusion

In this paper, we hypothesized that data sparsity or the existence of large
spaces around the support of the data distribution with no training data and
large numbers of redundant parameters in the DL models allow different DL
models to create different decision boundaries with different appearances in
the space outside the support of the data distribution. Following this
hypothesis, we proposed a training framework that trains DL models to have
decision boundaries as far as possible from the class distributions. We
generated an unlabeled data set to cover the space outside the support of the
data distribution using Gaussian noise and used the unlabeled data set to
learn such decision boundaries. We showed that the proposed training framework
indeed increases the distance from data points to the decision boundary. We
also showed that using unlabeled data sets generated by Gaussian noise is
almost as effective as unlabeled data sets sourced from expensive and manually
collected data sets or generated using complex synthesis algorithms.

## References

  * [1] Rawat, Waseem, and Zenghui Wang,  Deep convolutional neural networks for image classification: A comprehensive review.  In Neural computation 29, 9, pages 2352–2440, 2017.
  * [2] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le,  Unsupervised data augmentation for consistency training.  arXiv preprint arXiv:1904.12848, 2019.
  * [3] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii,  Virtual adversarial training: a regularization method for supervised and semi-supervised learning.  In IEEE transactions on pattern analysis and machine intelligence 41, pages 1979–1993, 2018.
  * [4] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy,  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * [5] Daniel Jakubovitz, and Raja Giryes,  Improving dnn robustness to adversarial attacks using jacobian regularization.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 514-529, 2018.
  * [6] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard,  Robustness via curvature regularization, and vice versa.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9078–9086, 2019.
  * [7] Andrew Slavin Ross, and Finale Doshi-Velez,  Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients.  In Thirty-second AAAI conference on artificial intelligence, 2018.
  * [8] Zhai, Runtian, Tianle Cai, Di He, Chen Dan, Kun He, John Hopcroft, and Liwei Wang,  Adversarially robust generalization just requires more unlabeled data.  arXiv preprint arXiv:1906.00555, 2019.
  * [9] Jonathan Uesato, Jean-Baptiste Alayrac, Po-Sen Huang, Robert Stanforth, Alhussein Fawzi, and Pushmeet Kohli,  Are labels required for improving adversarial robustness?  arXiv preprint arXiv:1905.13725, 2019.
  * [10] Amir Najafi, Shin-ichi Maeda, Masanori Koyama, and Takeru Miyato,  Robustness to adversarial perturbations in learning from incomplete data.  In Advances in Neural Information Processing Systems, pages 5541–5551, 2019.
  * [11] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C. Duchi, and Percy S. Liang,  Unlabeled data improves adversarial robustness.  In Advances in Neural Information Processing Systems, pages 11192–11203, 2019.
  * [12] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard,  Deepfool: a simple and accurate method to fool deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574–2582, 2016.
  * [13] Sajjadi, Mehdi and Javanmardi, Mehran and Tasdizen, Tolga,  Regularization with stochastic transformations and perturbations for deep semi-supervised learning.  In Advances in neural information processing systems, pages 1163–1171, 2016.
  * [14] Geng, Chuanxing, Sheng-jun Huang, and Songcan Chen,  Recent advances in open set recognition: A survey.  In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
  * [15] Shao, Rui, Pramuditha Perera, Pong C. Yuen, and Vishal M. Patel,  Open-set adversarial defense.  arXiv preprint arXiv:2009.00814, 2020.
  * [16] Scheirer, Walter J., Lalit P. Jain, and Terrance E. Boult,  Probability models for open set recognition.  In IEEE transactions on pattern analysis and machine intelligence 36, pages 2317–2324, 11, 2014.
  * [17] Yan, Ziang, Yiwen Guo, and Changshui Zhang,  Deep defense: Training dnns with improved adversarial robustness.  Advances in Neural Information Processing Systems, pages 419–428, 2018.
  * [18] Zhang, Hongyang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan,  Theoretically principled trade-off between robustness and accuracy.  arXiv preprint arXiv:1901.08573, 2019.
  * [19] Ross, Andrew Slavin, and Finale Doshi-Velez,  Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients.  arXiv preprint arXiv:1711.09404, 2017.
  * [20] Moosavi-Dezfooli, Seyed-Mohsen, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard.  Robustness via curvature regularization, and vice versa,  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9078–9086, 2019.
  * [21] Madry, Aleksander, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu,  Towards deep learning models resistant to adversarial attacks.  arXiv preprint arXiv:1706.06083, 2017.
  * [22] Drucker, Harris, and Yann Le Cun,  Improving generalization performance using double backpropagation.  In IEEE Transactions on Neural Networks 3, 6, pages 991–997, 1992.
  * [23] Ba, Jimmy, and Rich Caruana,  Do deep nets really need to be deep?  In Advances in neural information processing systems, pages 2654–2662, 2014.
  * [24] Amodei, Dario, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper et al,  Deep speech 2: End-to-end speech recognition in English and Mandarin.  In International conference on machine learning, pages 173–182, 2016.
  * [25] Young, Tom, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria,  Recent trends in deep learning based natural language processing.  IEEE Computational intelligence magazine 13 3, pages 55–75, 2018.
  * [26] Szegedy, Christian, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus,  Intriguing properties of neural networks.  arXiv preprint arXiv:1312.6199, 2013).
  * [27] Scheirer, Walter J., Anderson de Rezende Rocha, Archana Sapkota, and Terrance E. Boult,  Toward open set recognition.  In IEEE transactions on pattern analysis and machine intelligence 35, 7, 1757–1772, 2012.
  * [28] LeCun, Yann, Léon Bottou, Yoshua Bengio, and Patrick Haffner,  Gradient-based learning applied to document recognition.  In Proceedings of the IEEE 86, 11, pages 2278–2324, 1998.
  * [29] Krizhevsky, Alex, and Geoffrey Hinton,  Learning multiple layers of features from tiny images.  In tech. rep., Citeseer, 2009.
  * [30] Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang et al.,  Imagenet large scale visual recognition challenge.  In International journal of computer vision 115, 3, pages 211–252, 2015.
  * [31] Ross, Andrew Slavin, and Finale Doshi-Velez,  Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients.  arXiv preprint arXiv:1711.09404, 2017.
  * [32] He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [33] He, Tong, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li,  Bag of tricks for image classification with convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 558–567, 2019.
  * [34] Ba, Jimmy, and Rich Caruana,  Do deep nets really need to be deep?  In Advances in neural information processing systems, pages 2654–2662, 2014.

##  6 Effect of the Proposed Regularizer on SoftMax Score Surface for the 2D
Points Data set

In this section, we provide training parameters for the 2D Points set
experiment. We also provide more examples of the decision boundaries learned
by our model using normal training, Jacobian Regularization [5] and our
proposed training framework.

We trained a neural network with four fully connected layers. The number of
neurons in each layer was 2, 8, 8, and 2 respectively. The TanH activation
function was used in all experiments on the 2D Points set. Stochastic Gradient
Descent with a learning rate of $1e-3$, momentum of 0.9, and weight decay of
$5e-4$ was used for optimization. The $\lambda$ value was set to 1000 for both
the proposed training framework and for Jacobian Regularization [5] in Fig. 3
of the main paper. Fig. 6 presents more examples of the decision boundaries
that models learn when trained with normal training, Jacobian Regularization
[5] with $\lambda=10$, and the proposed training framework with
$\lambda=1000$.

Figure 6: Figure shows more examples of 2D Points sets and the decision
boundary learned by models trained with normal training, using Jacobian
Regularizer [5] and the proposed training framework.

As shown by these examples, the proposed training framework is able to spread
the SoftMax surface more smoothly than the Jacobian regularizer [5] in the
space around the two class distributions without labeled data points. These
examples show the advantage of using unlabeled data by our proposed training
framework in learning a smooth SoftMax surface in the space around the class
distributions.

##  7 Details of Hyperparameters used in the Proposed Training Framework

In this section, we provide the set of hyperparameters used in experiments for
each image data set.

###  7.1 MNIST Data Set

For experiments on MNIST data set, LeNet [28] network architecture was used
with Sigmoid activation function and a learning rate of $10^{-3}$. Adam
optimizer was used for training with no weight decay. The mean ($\mu$) values
for both Gaussian distributions used to generate unlabeled and neighbor data
points were set to zero and the standard deviations ($\sigma$) were set to
$0.126$, and 10-times smaller $\sigma$ of $0.0126$, respectively. The model
was trained using our proposed loss function. Different values of $\lambda$
were tested and the model’s loss curve was checked for each $\lambda$ value.
The highest $\lambda$ value that did not decrease the model’s validation
accuracy significantly was chosen. This value was $100k$ for the MNIST data
set. Models were trained with $\lambda=100k$, number of unlabeled data points
per labeled data point $N_{u}=1$ and number of neighbors per data point
$N_{b}=1$. The models trained using these hyperparameters were used to measure
$\mathbf{\hat{\rho}_{adv}}$ values and test accuracies in Table 1 of the main
paper.

###  7.2 CIFAR10 Data Set

The models were trained on the CIFAR10 data set using augmentation methods
including random crops, horizontal flipping, and perspective change. The
ResNet 9 [32] network architecture was used with a CELU activation function
for all layers. The model was trained with Adam optimizer with a learning rate
of $10^{-4}$, weight decay of $1e-4$ and the standard deviations for
generating unlabeled data points and neighbor data points were set to $0.238$
and $0.0238$, respectively. The lambda value was chosen similarly and was set
to $\lambda=10k$ for the CIFAR10 data set. Using $\lambda=10k$, the model was
trained using the proposed loss function and the generated unlabeled data set.
Models were trained with $\lambda=10k$, number of unlabeled data points per
labeled data point $N_{u}=1$ and number of neighbors per data point $N_{b}=1$.
The models trained using these hyperparameters were used to measure
$\mathbf{\hat{\rho}_{adv}}$ values and test accuracies in Table 2 of the main
paper.

###  7.3 Imagenette Data Set

Imagenette is a subset of the well-known ImageNet ILSVRC 2012 data set [30]
with 10 easily classified classes: tench, English springer, cassette player,
chain saw, church, French horn, garbage truck, gas pump, golf ball, parachute.
For each of the 10 classes, Imagenette has around 1000 and 400 data points in
the training and validation data sets respectively. The XResNet 18 [33]
network architecture was used to train a model to classify the Imagenette data
set with the MISH activation function. The Adam optimizer was used for
training with a learning rate of $1e-4$ and weight decay of $1e-4$. The
standard deviations for the Gaussian distributions for generating unlabeled
and neighbor data points were set to $0.3$ and $0.03$ respectively. We used
$\lambda=10k$ for regularization experiments and $\lambda=1000k$ for
regularization and adversarial training experiments. The number of unlabeled
data points per labeled data point $N_{u}$ and number of neighbors per labeled
data point $N_{b}$ were set to one. The trained models with these
hyperparameters were used to measure $\mathbf{\hat{\rho}_{adv}}$ values and
test accuracies in Table 3.

We also show the robust accuracy of these models against FGSM [4], and PGD
[21] attacks in Fig. 7 with different epsilon perturbation values
($\epsilon$). Robust accuracy of the models against DeepFool [12] ($l^{2}$ and
$l^{\infty}$) are not shown here because the for all models the robust
accuracy stayed almost constant.

Figure 7: Figure shows the robust accuracy of models trained on MNIST, CIFAR10
and Imagenette data set against FGSM [4] and PGD [21] attacks with different
epsilon perturbation values ($\epsilon$). Robust accuracy of the models
against DeepFool [12] ($l^{2}$ and $l^{\infty}$) are not shown as the robust
accuracy stayed almost constant for all models.

##  8 Gaussian Noise is as Effective as Costly and More Complex Unlabeled Data
Sets

In this section, we provide more details on the hyperparameters for models
used in Fig. 5 and Fig. 6 of the main paper.

###  8.1 MNIST Data Set

In Fig. 5 (Left) of the main paper, the effect of different numbers of
unlabeled data points per labeled data point ($N_{u}$) on robust accuracy of
the model was tested for MNIST data set. Models were trained with $N_{u}$ = 0,
5, and 10 with fixed parameters $\lambda=100k$, number of labeled data points
per class $N_{l}=64$, and number of neighbors per data point $N_{b}=100$. Five
models were trained for each set of parameters, all with similar validation
accuracies. The robust accuracy of the model was tested for different epsilon
perturbation values ($\epsilon$) using FGSM [4], PGD [21], and DeepFool
($l^{\infty}$) [12] attacks.

In Fig. 6 (Left) of the main paper, the effect of regularization was tested on
the robust accuracy of models trained with different numbers of labeled data
points for the MNIST data set. Models were trained with ($\lambda=100k$) and
without ($\lambda=0$) regularization for different numbers of labeled data
points per class ($N_{l}=8,64,256$), with a fixed number of unlabeled data
points ($N_{u}=10$) and neighbors ($N_{b}=2000$) per data point. Five
different models were trained similarly per set of parameters. All trained
models had similar validation accuracies.

###  8.2 CIFAR10 Data Set

In Fig. 5 (Right) of the main paper, the effect of different numbers of
unlabeled data points on the robust accuracy of models trained on CIFAR10 data
set was evaluated. Models were trained with different numbers of unlabeled
data points per labeled data point $N_{u}=1,5,10$ with $\lambda=10k$, with a
fixed number of labeled data points per class $N_{l}=128$ and neighbors per
data point $N_{b}=1$ for the CIFAR10 data set. Five different models were
trained similarly for every set of parameters. All trained models had similar
validation accuracies.

In Fig. 6 (Right) of the main paper, the effect of regularization was
evaluated on the robust accuracy of models trained with different numbers of
labeled data points for CIFAR10 data set. Models were trained with
($\lambda=10k$) and without ($\lambda=0$) regularization, and different
numbers of labeled data points per class ($N_{l}=128,256,512$). The number of
unlabeled data points per labeled data point, $N_{u}$, and neighbors per data
point, $N_{b}$, were fixed to 1 in these experiments. Five different models
were trained similarly per set of parameters. All trained models had similar
validation accuracies.

##  9 Effect of the Choice of Gaussian Noise on Robustness of Models

To analyse the impact of the choice of Gaussian distribution on the robust
accuracy of models trained using our framework, models were trained with noise
sampled from Gaussian distributions with different standard deviations. For
reference, $\sigma_{u}$ and $\sigma_{b}$ denote the standard deviations to
generate unlabeled and neighbor data points respectively.

For all 3 datasets, we performed 5 additional experiments with $\sigma_{u}$
and $\sigma_{b}$ being 0.1x, 0.5x, 1.5x, 2x and 3x the original $\sigma_{u}$
and $\sigma_{b}$ used in the main paper. The hyperparameters chosen were the
same as those in Section 7, and 3 models were trained per experiment. Table 4
shows the $\hat{\rho}_{adv}$ and test accuracies for these models. The results
for the models used in the main paper are shown in bold. Figure 8 shows the
robust accuracies for these models.

Table 4: Column $\mathbf{\hat{\rho}_{adv}}$ shows test accuracies and measured $\mathbf{\hat{\rho}_{adv}}$ values from the DeepFool attack [12] for the models trained on MNIST, CIFAR10, and Imagenette datasets using our training framework. The standard deviations used to generate unlabeled and neighbor data points, $\sigma_{u}$ and $\sigma_{b}$, were varied from 0.1x to 3x the original $\sigma_{u}$ and $\sigma_{b}$ from the main paper (shown in bold). | MNIST | CIFAR10 | Imagenette
---|---|---|---
Defence method | $\hat{\rho}_{adv}$ | Test accuracy | $\hat{\rho}_{adv}$ | Test accuracy | $\hat{\rho}_{adv}$ | Test accuracy
$(\times 10^{-2})$ | (%) | $(\times 10^{-2})$ | (%) | $(\times 10^{-2})$ | (%)
No defence | 20.30 | 98.29 | 1.23 | 91.17 | 0.941 | 87.18
Adversarial training | 61.84 | 98.66 | 1.27 | 89.51 | 4.82 | 83.06
|  |  $\sigma_{u}=0.00126$,
---
$\sigma_{b}=0.000126$ (0.1x)
|  $\sigma_{u}=0.02380$,
---
$\sigma_{b}=0.002380$ (0.1x)
|  $\sigma_{u}=0.01387$,
---
$\sigma_{b}=0.001387$ (0.1x)
Proposed method | 44.96 | 98.65 | 3.20 | 81.20 | 1.73 | 86.06
Proposed method & | 47.91 | 98.58 | 4.63 | 83.99 | 5.01 | 80.82
Adversarial training
|  |  $\sigma_{u}=0.0063$,
---
$\sigma_{b}=0.00063$ (0.5x)
|  $\sigma_{u}=0.1190$,
---
$\sigma_{b}=0.0119$ (0.5x)
|  $\sigma_{u}=0.06935$,
---
$\sigma_{b}=0.006935$ (0.5x)
Proposed method | 44.49 | 98.67 | 3.08 | 80.80 | 2.46 | 83.44
Proposed method & | 50.53 | 98.76 | 4.70 | 84.97 | 5.31 | 81.94
Adversarial training
|  |  $\mathbf{\sigma_{u}=0.1260}$,
---
$\mathbf{\sigma_{b}=0.0126}$ (1x)
|  $\mathbf{\sigma_{u}=0.2380}$,
---
$\mathbf{\sigma_{b}=0.0238}$ (1x)
|  $\mathbf{\sigma_{u}=0.1387}$,
---
$\mathbf{\sigma_{b}=0.0139}$ (1x)
Proposed method | 46.29 | 98.69 | 3.29 | 83.96 | 2.21 | 84.92
Proposed method & | 51.20 | 98.80 | 4.45 | 83.50 | 4.96 | 82.44
Adversarial training
|  |  $\sigma_{u}=0.1890$,
---
$\sigma_{b}=0.0189$ (1.5x)
|  $\sigma_{u}=0.3570$,
---
$\sigma_{b}=0.0357$ (1.5x)
|  $\sigma_{u}=0.2081$,
---
$\sigma_{b}=0.0208$ (1.5x)
Proposed method | 46.63 | 98.63 | 2.94 | 86.16 | 1.85 | 86.01
Proposed method & | 60.22 | 98.21 | 4.32 | 86.25 | 5.53 | 82.04
Adversarial training
|  |  $\sigma_{u}=0.2520$,
---
$\sigma_{b}=0.02520$ (2x)
|  $\sigma_{u}=0.4760$,
---
$\sigma_{b}=0.0476$ (2x)
|  $\sigma_{u}=0.2774$,
---
$\sigma_{b}=0.02774$ (2x)
Proposed method | 47.22 | 98.52 | 3.08 | 86.60 | 1.41 | 84.46
Proposed method & | 50.85 | 98.71 | 4.75 | 86.25 | 5.52 | 82.96
Adversarial training
|  |  $\sigma_{u}=0.3780$,
---
$\sigma_{b}=0.0378$ (3x)
|  $\sigma_{u}=0.7130$,
---
$\sigma_{b}=0.0713$ (3x)
|  $\sigma_{u}=0.4161$,
---
$\sigma_{b}=0.04161$ (3x)
Proposed method | 45.68 | 98.65 | 3.04 | 86.16 | 1.35 | 86.32
Proposed method & | 60.40 | 98.19 | 4.64 | 86.39 | 5.10 | 82.09
Adversarial training
Figure 8: Figure shows the robust accuracy of models trained using the
proposed training framework (NoAdv_Reg) versus models trained using the
proposed training framework plus adversarial training (Adv_Reg). The standard
deviations $\sigma_{u}$ and $\sigma_{b}$ used to generate unlabeled and
neighbor data points respectively were varied from 0.1x to 3x their original
values used in the main paper. The robust accuracies were measured using FGSM
[4] and PGD [21] attacks with different epsilon perturbation values.
($\epsilon$)

The results from Table 4 show that when only the proposed method is used in
training, the choice of $\sigma_{u}$ and $\sigma_{b}$ is important as it does
moderate adversarial robustness. Should one choose values of $\sigma_{u}$ and
$\sigma_{b}$ that are too high or low, adversarial robustness would decrease.
As seen from all the rows labeled as "Proposed method", for MNIST, the
multiple of $\sigma_{u}$ and $\sigma_{b}$ that gives the highest
$\hat{\rho}_{adv}$ is 2x. For CIFAR10, it is 1x. Lastly, for Imagenette, it is
0.5x.

We believe that robustness drops when the standard deviation is too high,
because the generated unlabeled data points are too far away from the train
data class distributions. Consequently, there were insufficient unlabeled data
points around the class distributions for regularization of the decision
surfaces in these areas to take place. Additionally, the generated neighbors
are situated too far away to fulfill their original purpose of being in close
proximity with the train data points to make the decision surface around each
data point smooth.

Likewise, if the standard deviation is too low, the unlabeled data points will
likely be too close to the original data points, therefore they cannot support
enough of the high dimensional classification space.

Interestingly, however, according to Figure 8, the robust accuracies stayed
roughly the same for all the models trained with only our proposed framework.
This means that whilst the robustness of our models to the DeepFool [12]
attack and hence $\hat{\rho}_{adv}$ values are sensitive to the choice of
$\sigma_{u}$ and $\sigma_{b}$, robustness of our models to FGSM [4] and PGD
[21] remained largely independent of $\sigma_{u}$ and $\sigma_{b}$. Why this
is the case remains to be seen.

On the other hand, when the proposed method was combined with adversarial
training, increasing or decreasing $\sigma_{u}$ and $\sigma_{b}$ had no
discernible effect on the $\hat{\rho}_{adv}$ of models from the 3 datasets.
This is evident in Table 4 and 8 where the variation in $\hat{\rho}_{adv}$
values and the robust accuracy curves followed no clear trend. This could be
because the adversarial examples generated during adversarial training were
already very scattered in the input space. Therefore, further variations of
$\sigma_{u}$ or $\sigma_{b}$ did not impact adversarial robustness at all.

##  10 Comparison with Jacobian Regularization

In the main paper’s experiments, the measured $\hat{\rho}_{adv}$ values were
larger for models trained using the Jacobian regularizer [5] compared to
models trained using the proposed training framework for the MNIST and CIFAR10
data sets. In this section, we compare the robustness of models trained with
the proposed training framework and the Jacobian regularizer using the FGSM
[4] and PGD [21] attacks.

For each dataset, 3 models were trained using the proposed training framework
and 3 models were trained using the Jacobian regularizer. Their robust
accuracies evaluated against the FGSM [4] and PGD [21] attacks with different
epsilon perturbation values ($\epsilon$) are outlined in Figure 9. The lambda
values used for the Jacobian Regularizer were set to 0.1 for the MNIST data
set, 0.01 for the CIFAR10 data set and 0.05 for the Imagenette data set.

Figure 9: Figure shows the robust accuracy of models trained using the
proposed training framework (Reg) and Jacobian regularization (Jacobian)
[5]for MNIST, CIFAR10 and Imagenette data. The robust accuracies are measured
using FGSM [4], and PGD [21] attacks with different epsilon perturbation
values ($\epsilon$).

As shown in Figure 9, the proposed training framework performs better for the
MNIST data set, whereas the Jacobian regularizer performs better for the
CIFAR10 and Imagenette data set. For those datasets, the Jacobian regularizer
performs better than our proposed training framework with unlabeled data. This
improved performance has two potential explanations:

  1. 1.

The class distributions in the train data of CIFAR10 and Imagenette are
sufficiently spread out such that they already support much of the input
space. Hence, the proposed training framework is not really needed as it does
not confer much additional support, and the Jacobian regularizer outshines our
method for these two data sets.

  2. 2.

The Jacobian regularizer is a stronger regularizer than our proposed training
framework, regardless of the extent to which the data points in the dataset
support the high dimensional classification space.

To find out which explanation was true, we compared the robustness of models
trained using just the Jacobian regularizer versus models trained with both
Jacobian Regularization and unlabeled data generated using our training
framework. 3 models were trained per data set. $\sigma_{u}$ and $\sigma_{b}$
were set in the same way as the main paper.

We chose to do this because if the robustness did not significantly improve
after the Jacobian regularizer was combined with unlabeled data from our
training framework, this would clearly mean that unlabeled data does not help
for CIFAR10 and Imagenette, therefore the first explanation holds true.
Conversely, if the robustness does significantly improve, then the second
explanation holds true.

Having performed the experiments, the $\hat{\rho}_{adv}$ values and test
accuracies of the trained models are shown in the rows labeled "Jacobian", and
"Jacobian+Unlabeled" in Table 5. The robust accuracies are also shown in the
curves labeled "Jacobian" and "Jacobian+Unlabeled" in Figure 10.

As seen in Table 5 and Figure 10, when the Jacobian regularizer is used in
conjunction with unlabeled data from our training framework, robustness indeed
improves. Therefore, the second explanation aforementioned holds true.
Jacobian Regularization is indeed more powerful than our method. This could be
because Jacobian Regularization measures the gradient of the Softmax score
surface for all dimensions in the input data space and smoothens the gradients
of the SoftMax score surface for all dimensions. On the other hand, the
proposed training framework uses a neighbor point as a reference to minimize
the SoftMax score surface gradients. Depending on the relative position of the
neighbor data point with respect to the labeled or unlabeled data point, our
regularization framework smoothens the SoftMax score surface in not all but
limited number of dimensions in the input data space.

Table 5: Columns $\mathbf{\hat{\rho}_{adv}}(\times 10^{-2})$ and Test Accuracy (%) show measured $\mathbf{\hat{\rho}_{adv}}$ values from the DeepFool attack [12] and test accuracies for the models trained on MNIST, CIFAR10, and Imagenette datasets. This table compares the robustness of models trained using the Jacobian regularizer (Jacobian), Jacobian regularization and unlabeled data from our training framework (Jacobian+Unlabeled), as well as Jacobian regularization with the Gaussian noise injected in the intermediate model outputs (Jacobian+Inject). | MNIST | CIFAR10 | Imagenette
---|---|---|---
Defence method | $\hat{\rho}_{adv}$ | Test accuracy | $\hat{\rho}_{adv}$ | Test accuracy | $\hat{\rho}_{adv}$ | Test accuracy
$(\times 10^{-2})$ | (%) | $(\times 10^{-2})$ | (%) | $(\times 10^{-2})$ | (%)
Jacobian | 561 | 98.75 | 3.56 | 88.01 | 2.51 | 85.52
Jacobian+Unlabeled | 1083 | 98.84 | 3.98 | 87.78 | 5.35 | 84.96
Jacobian+Inject | 1582 | 98.70 | 4.42 | 87.24 | 5.10 | 84.56
Figure 10: Figure shows the robust accuracy of models trained using the
Jacobian regularizer (Jacobian) versus models trained using both Jacobian
Regularization and unlabeled data generated with our framework
(Jacobian+Unlabeled). The robust accuracies are measured using FGSM [4] and
PGD [21] attacks with different epsilon perturbation values. ($\epsilon$) It
can be seen that training models using both Jacobian Regularization and
unlabeled data generated with our framework are more resilient to adversarial
perturbations.

##  11 Effect of the Location where Gaussian Noise is Applied on Robustness of
Models

As the main paper’s training framework applied Gaussian noise to generate
unlabeled data before performing a forward pass, we also conducted experiments
to ascertain if applying the Gaussian noise to intermediate model outputs
would significantly change the model’s adversarial robustness. The below
paragraphs outline how the modified version of our training framework operates
to inject Gaussian noise to the intermediate model outputs.

We chose to perform intermediate noise injection on the output of the last
convolution layer of the model (right before the fully connected segments).
For the $k^{\text{th}}$ labeled image $\psi_{k}$, two rounds of intermediate
noise injection were performed.

The first time, $\psi_{k}$ was passed into the model and the intermediate
outputs $\psi_{k_{i}}$ were intercepted. Noise from a Gaussian distribution
with mean 0 and standard deviation $\sigma_{b_{i}}$ was then added to
$\psi_{k_{i}}$ to generate its associated neighbor data point
$\psi_{k_{i},m}$. $\psi_{k_{i}}$ and $\psi_{k_{i},m}$ were then passed
together into the remainder of the model.

The second time, $\psi_{k}$ was passed into the model again and $\psi_{k_{i}}$
was intercepted as well. However, now, noise from a Gaussian distribution with
mean 0 and standard deviation $\sigma_{u_{i}}=10\sigma_{b_{i}}$ was added to
$\psi_{k_{i}}$ to generate $u_{k_{i}}$, the associated unlabeled image of
$\psi_{k_{i}}.$ $\psi_{k_{i}}$ was then overwritten by $u_{k_{i}}$ and the
associated neighbor $u_{k_{i},m}$ was generated in the same way as
aforementioned. $u_{k_{i}}$ and $u_{k_{i},m}$ were then passed together into
the remainder of the model.

$\sigma_{u_{i}}$ and $\sigma_{b_{i}}$ are analogous to the $\sigma_{u}$ and
$\sigma_{b}$ used in the main paper. They were also calculated in a very
similar way, with the only difference being that the mean pairwise Euclidean
distances were calculated over the intermediate model outputs $\psi_{k_{i}}$
instead of the labeled images $\psi_{k}$.

We implemented the modified training framework as aforementioned on all
datasets and trained 3 models per experiment. The hyperparameters used were
the same as those outlined in Section 7 with the exception of the new standard
deviations $\sigma_{u_{i}}$ and $\sigma_{b_{i}}$. Table 6 and Figure 11 show
the $\hat{\rho}_{adv}$ values, test accuracies and robust accuracies for these
models compared against the main paper’s vanilla training framework.

Table 6: Columns $\hat{\rho}_{adv}(\times 10^{-2})$ and Test Accuracy (%) show measured $\hat{\rho}_{adv}$ values from the DeepFool attack [12] and test accuracies for the models trained on MNIST, CIFAR10, and Imagenette datasets. This table compares the robustness of models trained using the vanilla training framework from the main paper (NoAdv_Reg) with models trained using the aforementioned modified framework to do intermediate noise injection (NoAdv_Reg_Inject). | MNIST | CIFAR10 | Imagenette
---|---|---|---
Defence method | $\hat{\rho}_{adv}$ | Test accuracy | $\hat{\rho}_{adv}$ | Test accuracy | $\hat{\rho}_{adv}$ | Test accuracy
$(\times 10^{-2})$ | (%) | $(\times 10^{-2})$ | (%) | $(\times 10^{-2})$ | (%)
NoAdv_Reg | 46.29 | 98.69 | 3.29 | 83.96 | 2.21 | 84.92
NoAdv_Reg_Inject | 23.51 | 98.41 | 1.39 | 91.02 | 1.60 | 80.46
Figure 11: Figure shows the robust accuracy of models trained using the
proposed training framework versus injecting noise into the network using the
proposed training framework. The robust accuracies are measured using FGSM [4]
and PGD [21] attacks with different epsilon perturbation values. ($\epsilon$)

The results show that when noise is injected to the intermediate outputs of
the models, adversarial robustness decreases. This is evident in Table 6 where
the $\hat{\rho}_{adv}$ values decreased and also in Figure 11 where the robust
accuracies decreased. This could have happened because of two reasons.

  1. 1.

The standard deviations $\sigma_{u_{i}}$ and $\sigma_{b_{i}}$ were calculated
over the intermediate model outputs $\psi_{k_{i}}$ instead of the input
labeled images $\psi_{k}$. As $\psi_{k_{i}}<<\psi_{k}$,
$\sigma_{u_{i}}<<\sigma_{u}$ and $\sigma_{b_{i}}<<\sigma_{b}$. Therefore, the
spread of Gaussian noise was significantly reduced during intermediate noise
injection.

  2. 2.

By the time the images were passed to the last convolution layers of the
model, they would have gone through sufficient rounds of convolutions and max
pooling such that only the features of the image deemed as most important by
the model were remaining. Therefore, adding more noise to intermediate model
outputs only served to slightly modify a small subset of the image features
and could not support much of the input space.

The combined effect of these two reasons was that the unlabeled intermediate
images $u_{k_{i}}$ as well as the neighbour intermediate images
$\psi_{k_{i},m}$ and $u_{k_{i},m}$ gave the models more chances to overfit and
reinforce their ideas of what features were the most important. As a result,
adversarial robustness took a hit.

However, as an extension, we trained models with both the Jacobian regularizer
and the intermediate noise injection framework at once. This was done to
investigate if the adversarial robustness would similarly decrease as observed
when compared against the vanilla training framework from the main paper.
Three models were trained per experiment and their hyperparameters were chosen
in the same way as outlined in Section 7. The $\hat{\rho}_{adv}$ values and
test accuracies are shown in the row labeled "Jacobian+Inject" in Table 5. The
corresponding robust accuracy curve is also labeled "Jacobian+Inject" in
Figure 10.

On the contrary, the results show that when the Jacobian regularizer is
combined with unlabeled data generated using our training framework, the
robustness increases compared to pure Jacobian Regularization. However, when
intermediate noise injection was performed on models trained with Jacobian
Regularization[5], robustness unexpectedly improved. This is evident in Table
5 where $\hat{\rho}_{adv}$ was higher for "Jacobian+Inject" than
"Jacobian+Unlabeled". This trend is also presented in the corresponding robust
accuracy graph located in Fig. 10. This could be because Jacobian
Regularization is concerned with the matrix of partial derivatives of the
model outputs with respect to the model inputs. Thus, even if intermediate
noise injection is performed, the Jacobian Regularization algorithm still
keeps track of the entire computational gradient tree, from the input image to
the logits. This can downplay, to a considerable extent, or even nullify the
effect of the second aforementioned factor which would normally contribute to
overfitting in our framework.

##  12 Prediction Confidence Decreases for Unknown Unknown Classes

Open Set Recognition (OSR) describes the scenario where the DL model is
trained with data points from the training data set known as known known data
points (KKCs) similar to normal training. However, during testing, new classes
not seen during training also known as unknown unknown data points (UUCs)
appear among KKC data points and requires the model to correctly identify KKC
data points and reject UUC data points[14]. Many proposed methods train DL
models with an additional class label for the UUCs and aim for a decision
boundary that tightly binds the support for each class distribution. The work
by Shao et al. [15] showed that such OSR methods are vulnerable to adversarial
attacks and proposed a defense mechanism to detect and remove adversarial
noise. Other OSR algorithms do not add a new class label, but instead train
models with decreased probability of membership in each class for UUC data
points [16]. Since UUC distributions are hypothesized to be away from the
class distributions seen in the training data set, we evaluated whether our
trained models with the proposed training framework are able to provide OSR by
decreasing the prediction confidence for UUC data points.

Table 7: Table compares prediction confidence of DL models on UUC data points
taken from Imagenette data set. The DL models are trained on MNIST and CIFAR10
data sets using our training framework and without regularization for 5
different seeds.

Experiments were conducted to check the confidence of models trained with the
proposed training framework in predicting UUC data points. For this analysis,
we fed the models trained on MNIST data set with the test set for Imagenette
and measured the mean and std of the predictions for each class. We performed
the same experiment for the models trained on CIFAR10 data set. The results
were compared with the mean and standard deviation of predictions for models
trained without regularization. Table 7 shows the average class prediction and
the standard deviation of predictions for each class. As shown, the standard
deviation of predictions for each class decreases and the predictions are
spread more evenly among the classes for models trained using regularization.
In other words, the confidence of models in their predictions on UUC data
points is lower for models trained with regularization compared to model
trained without regularization. This implies that the SoftMax score surface at
the location of UUC distribution in the high dimensional input data space is
flattened.