# PARL: Enhancing Diversity of Ensemble Networks to Resist Adversarial Attacks
via Pairwise Adversarially Robust Loss Function

Manaar Alam
Department of CSE
IIT Kharagpur
Kharagpur, India
<EMAIL_ADDRESS>
&Shubhajit Datta
Centre of Excellence in AI
IIT Kharagpur
Kharagpur, India
<EMAIL_ADDRESS>
&Debdeep Mukhopadhyay
Department of CSE
IIT Kharagpur
Kharagpur, India
<EMAIL_ADDRESS>
&Arijit Mondal
Department of CSE
IIT Patna
Patna, India
<EMAIL_ADDRESS>
&Partha Pratim Chakrabarti
Department of CSE
Centre of Excellence in AI
IIT Kharagpur
Kharagpur, India
<EMAIL_ADDRESS>

###### Abstract

The security of Deep Learning classifiers is a critical field of study because
of the existence of adversarial attacks. Such attacks usually rely on the
principle of _transferability_ , where an adversarial example crafted on a
surrogate classifier tends to mislead the target classifier trained on the
same dataset even if both classifiers have quite different architecture.
Ensemble methods against adversarial attacks demonstrate that an adversarial
example is less likely to mislead multiple classifiers in an ensemble having
_diverse decision boundaries_. However, recent ensemble methods have either
been shown to be vulnerable to stronger adversaries or shown to lack an end-
to-end evaluation. This paper attempts to develop a new ensemble methodology
that constructs multiple diverse classifiers using a Pairwise Adversarially
Robust Loss (PARL) function during the training procedure. PARL utilizes
gradients of each layer with respect to input in every classifier within the
ensemble simultaneously. The proposed training procedure enables PARL to
achieve higher robustness against black-box transfer attacks compared to
previous ensemble methods without adversely affecting the accuracy of clean
examples. We also evaluate the robustness in the presence of white-box
attacks, where adversarial examples are crafted using parameters of the target
classifier. We present extensive experiments using standard image
classification datasets like CIFAR-10 and CIFAR-100 trained using standard
ResNet20 classifier against state-of-the-art adversarial attacks to
demonstrate the robustness of the proposed ensemble methodology.

##  1 Introduction

Deep learning (DL) algorithms have seen rapid growth in recent years because
of their unprecedented successes with near-human accuracies in a wide variety
of challenging tasks starting from image classification [38], speech
recognition [2], natural language processing [40] to self-driving cars [6].
While DL algorithms are extremely efficient in solving complicated decision-
making tasks, they are vulnerable to well-crafted adversarial examples
(slightly perturbed valid input with visually imperceptible noise) [37]. The
widely-studied phenomenon of adversarial examples among the research community
has produced numerous attack methodologies with varied complexity and
effective deceiving strategy [12, 24, 27, 29, 32]. An extensive range of
defenses against such attacks has been proposed in the literature, which
generally falls into two categories. The first category enhances the training
strategy of deep learning models to make them less vulnerable to adversarial
examples by training the models with different degrees of adversarially
perturbed training data [4, 18, 19, 42] or changing the training procedure
like gradient masking, defensive distillation, etc. [15, 31, 34, 35]. However,
developing such defenses has been shown to be extremely challenging. Carlini
and Wagner [8] demonstrated that these defenses are not generalized for all
varieties of adversarial attacks but are constrained to specific categories.
The process of training with adversarially perturbed data is hard, often
requires models with large capacity, and suffers from significant loss on
clean example accuracy. Moreover, Athalye et al. [3] demonstrated that the
changes in training procedures provide a false sense of security. The second
category intends to detect adversarial examples by simply flagging them [5,
10, 11, 14, 28, 17, 25]. However, even detection of adversarial examples can
be quite a complicated task. Carlini and Wagner [7] illustrated with several
experimentations that these detection techniques could be efficiently bypassed
by a strong adversary having partial or complete knowledge of the internal
working procedure.

While all the approaches mentioned above deal with standalone classifiers, in
this paper, we utilize the advantage of an ensemble of classifiers instead of
a single standalone classifier to resist adversarial attacks. The notion of
using diverse ensembles to increase the robustness of a classifier against
adversarial examples has recently been explored in the research community. The
primary motivation of using an ensemble-based defense is that if multiple
classifiers with similar decision boundaries perform the same task, the
transferability property of DL classifiers makes it easier for an adversary to
mislead all the classifiers simultaneously using adversarial examples crafted
on any of the classifiers. However, it will be difficult for an adversary to
mislead multiple classifiers simultaneously if they have diverse decision
boundaries. Strauss et al. [36] used various ad-hoc techniques such as
different random initializations, different neural network structures, bagging
the input data, adding Gaussian noise while training for creating multiple
diverse classifiers to form an ensemble. The resulting ensemble increases the
robustness of the classification task even in the presence of adversarial
examples. Tramèr et al. [39] proposed Ensemble Adversarial Training that
incorporates perturbed inputs transferred from other pre-trained models during
adversarial training to decouple adversarial example generation from the
parameters of the primary model. Grefenstette et al. [13] demonstrated that
ensembling two models and then adversarially training them incorporates more
robustness in the classification task than single-model adversarial training
and ensemble of two separately adversarially trained models. Kariyappa and
Qureshi [20] proposed Diversity Training of an ensemble of models with
uncorrelated loss functions using Gradient Alignment Loss metric to reduce the
dimension of adversarial sub-space shared between different models and
increase the robustness of the classification task. Pang et al. [30] proposed
Adaptive Diversity Promoting regularizer to train an ensemble of classifiers
that encourages the non-maximal predictions in each member in the ensemble to
be mutually orthogonal, which degenerates the transferability that aids in
resisting adversarial examples. Yang et al. [41] proposed a methodology that
isolates the adversarial vulnerability in each sub-model of an ensemble by
distilling non-robust features. While all these works attempt to enhance the
robustness of a classification task even in the presence of adversarial
examples, Adam et al. [1] proposed a stochastic method to add Variational
Autoencoders between layers as a noise removal operator for creating
combinatorial ensembles to limit the transferability and detect adversarial
examples.

In order to enhance the robustness of a classification task and/or to detect
adversarial examples, the ensemble-based approaches mentioned above employ
different strategies while training the models. These ensembles either lack
end-to-end evaluations for complicated datasets or lack evaluation against
more aggressive attack scenarios like the methods discussed by Dong et al. [9]
and Liu et al. [26], which demonstrate that adversarial examples misleading
multiple models in an ensemble tend to be more transferable. In this work, our
primary objective is to propose a systematic approach to enhance the
classification robustness of an ensemble of classifiers against adversarial
examples by developing diversity in the decision boundaries among all the
classifiers within that ensemble. The diversity is obtained by simultaneously
considering the mutual dissimilarity in gradients of each layer with respect
to input in every classifier while training them. The diversity among the
classifiers trained in such a way helps to degenerate the transferability of
adversarial examples within the ensemble.

Motivation behind the Proposed Approach and Contribution: The intuition behind
developing the proposed ensemble methodology is discussed in Figure 1, which
shows a case study using classifiers trained on CIFAR-10 dataset without loss
of generality. Figure 1a shows an input image of a ‘frog’. Figure 1b shows the
gradient of loss with respect to input111Fundamental operation behind the
creation of almost all adversarial examples in a classifier
$\mathcal{M}_{prim}$, and denoted as $\nabla_{prim}$. Figure 1c shows the
gradient for another classifier $\mathcal{M}_{sim}$ with a similar decision
boundary as $\mathcal{M}_{prim}$, and denoted as $\nabla_{sim}$. The
classifier $\mathcal{M}_{sim}$ is trained using the same parameter settings as
$\mathcal{M}_{prim}$ but with different random initialization. Figure 1d shows
the gradients for a classifier $\mathcal{M}_{div}$ with a not so similar
decision boundary compared to $\mathcal{M}_{prim}$, and denoted as
$\nabla_{div}$. The classifiers $\mathcal{M}_{prim}$, $\mathcal{M}_{sim}$, and
$\mathcal{M}_{div}$ have similar classification accuracies. The method for
obtaining such classifiers is discussed later in this paper. Figure 1e shows
relative symbolic directions among all the aforementioned gradients in higher
dimension. The directions between a pair of gradients are computed using
cosine similarity. We can observe that $\nabla_{prim}$ and $\nabla_{sim}$ lead
to almost in the same directions, aiding adversarial examples crafted on
$\mathcal{M}_{prim}$ to _transfer_ into $\mathcal{M}_{sim}$. However,
adversarial examples crafted in $\mathcal{M}_{prim}$ will be difficult to
transfer into $\mathcal{M}_{div}$ as the directions of $\nabla_{prim}$ and
$\nabla_{div}$ are significantly different.

(a)  (b)  (c)  (d)  (e)

Figure 1: (a) Input image; (b) $\nabla_{prim}$: Gradient of loss in the
primary classifier; (c) $\nabla_{sim}$: Gradient of loss in another classifier
with similar decision boundaries; (d) $\nabla_{div}$: Gradient of loss in a
classifier with not so similar decision boundaries but comparable accuracy;
(e) Symbolic directions of all the gradients in higher dimensions. The
gradients are computed with respect to the image shown in (a).

The principal motivation behind the proposed methodology is to introduce a
constraint for reducing the cosine similarity of gradients among each
classifier in an ensemble while training them simultaneously. Such a learning
strategy with mutual cooperation intends to ensure that gradients between each
pair of classifiers in the ensemble are as dissimilar as possible. We make the
following contributions using the proposed ensemble training method:

  * •

We propose a methodology to increase diversity in the decision boundaries
among all the classifiers within an ensemble to degrade the transferability of
adversarial examples.

  * •

We propose a Pairwise Adversarially Robust Loss (PARL) function by utilizing
the gradients of each layer with respect to input of every classifier within
the ensemble simultaneously for training them to produce such varying decision
boundaries.

  * •

The proposed method can significantly improve the overall robustness of the
ensemble against black-box transfer attacks without substantially impacting
the clean example accuracy.

  * •

We evaluated the robustness of PARL with extensive experiments using two
standard image classification benchmark datasets on ResNet20 architecture
against state-of-the-art adversarial attacks.

##  2 Threat Model

We consider the following two threat models in this paper while generating
adversarial examples.

  * •

_Zero Knowledge Adversary $(\mathcal{A}_{\mathcal{Z}})$:_ The adversary
$\mathcal{A}_{\mathcal{Z}}$ does not have access to the target ensemble
$\mathcal{M}_{\mathcal{T}}$ but has access to a surrogate ensemble
$\mathcal{M}_{\mathcal{S}}$ trained with the same dataset. We term
$\mathcal{A}_{\mathcal{Z}}$ as a black-box adversary. The adversary
$\mathcal{A}_{\mathcal{Z}}$ crafts adversarial examples on
$\mathcal{M}_{\mathcal{S}}$ and transfers to $\mathcal{M}_{\mathcal{T}}$.

  * •

_Perfect Knowledge Adversary $(\mathcal{A}_{\mathcal{P}})$:_ The adversary
$\mathcal{A}_{\mathcal{P}}$ is a stronger than $\mathcal{A}_{\mathcal{Z}}$ who
has access to the target ensemble $\mathcal{M}_{\mathcal{T}}$. We term
$\mathcal{A}_{\mathcal{P}}$ as a white-box adversary. The adversary
$\mathcal{A}_{\mathcal{P}}$ can generate adversarial examples on
$\mathcal{M}_{\mathcal{T}}$ knowing the parameters used by all the networks
within $\mathcal{M}_{\mathcal{T}}$ .

##  3 Overview of the Proposed Methodology

In this section, we provide a brief description of the proposed methodology
used in this paper to enhance classification robustness against adversarial
examples using an ensemble of classifiers $\mathcal{M}_{\mathcal{T}}$. The
ensemble consists of $\mathcal{N}$ neural network classifiers and denoted as
$\mathcal{M}_{\mathcal{T}}=\bigcup_{i=1}^{\mathcal{N}}\mathcal{M}_{i}$. All
the $\mathcal{M}_{i}$’s are trained simultaneously using the _Pairwise
Adversarially Robust Loss_ (PARL) function, which we discuss in detail in
Section 4. The final decision for an input image on
$\mathcal{M}_{\mathcal{T}}$ is decided based on the majority voting among all
the classifiers. Formally, let us assume a test set of $t$ inputs
$\\{x_{1},x_{2},\dots,x_{t}\\}$ with respective ground truth labels as
$\\{y_{1},y_{2},\dots,y_{t}\\}$. The final decision of the ensemble
$\mathcal{M}_{\mathcal{T}}$ for an input $x_{j}$ is defined as

$\mathcal{C}(\mathcal{M}_{\mathcal{T}},x_{j})=majority\\{\mathcal{M}_{1}(x_{j}),\mathcal{M}_{2}(x_{j}),\cdots,\mathcal{M}_{\mathcal{N}}(x_{j})\\}$

$\mathcal{C}(\mathcal{M}_{\mathcal{T}},x_{j})=y_{j}$ for most $x_{j}$’s in an
appropriately trained $\mathcal{M}_{\mathcal{T}}$. The primary argument behind
the proposed ensemble method is that all $\mathcal{M}_{i}$’s have dissimilar
decision boundaries but not significantly different accuracies. Hence, a clean
example classified as class $\mathcal{C}_{x}$ in $\mathcal{M}_{i}$ will also
be classified as $\mathcal{C}_{x}$ in most other $\mathcal{M}_{j}$’s (where
$j=1\dots\mathcal{N}$, $j\neq i$) within the ensemble with a high probability.
Consequently, because of the diversity in decision boundaries between
$\mathcal{M}_{i}$ and $\mathcal{M}_{j}$ (for $i=1\dots\mathcal{N}$,
$j=1\dots\mathcal{N}$, and $i\neq j$), the adversarial examples generated by a
zero knowledge adversary $(\mathcal{A}_{\mathcal{Z}})$ for a surrogate
ensemble $\mathcal{M}_{\mathcal{S}}$ will have a different impact on each
classifiers within the ensemble $\mathcal{M}_{\mathcal{T}}$, i.e., the
transferability of adversarial examples will be challenging within the
ensemble. A perfect knowledge adversary $(\mathcal{A}_{\mathcal{P}})$ can
generate adversarial examples for the ensemble $\mathcal{M}_{\mathcal{T}}$.
However, in this scenario, the input image perturbation will be in different
directions because of the diversity in decision boundaries among all
$\mathcal{M}_{i}$’s within the ensemble. The collective disparity in
perturbation directions makes it challenging to craft adversarial examples for
the ensemble. We evaluated our proposed methodology considering both the
adversaries and presented the results in Section 5.

##  4 Building The Ensemble Networks using PARL

In this section, we provide a detailed discussion on training an ensemble of
neural networks using the proposed Pairwise Adversarially Robust Loss (PARL)
function for increasing diversity among the networks. First, we define the
basic terminologies used in the construction, followed by a detailed
methodology of the training procedure.

###  4.1 Basic Terminologies used in the Construction

Let us consider an ensemble
$\mathcal{M}_{\mathcal{T}}=\bigcup_{i=1}^{\mathcal{N}}\mathcal{M}_{i}$, where
$\mathcal{M}_{i}$ is the $i^{th}$ network in the ensemble. We assume that each
of these networks has the same architecture with $\mathcal{L}_{\mathcal{H}}$
number of hidden layers. Let
$\mathcal{J}_{\mathcal{M}_{i}}(\mathbf{x},\mathbf{y})$ be the loss functions
evaluating the amount of loss incurred by the network $\mathcal{M}_{i}$ for a
data point $\mathbf{x}$, where $\mathbf{y}$ is the ground-truth label for
$\mathbf{x}$. Let
$\mathcal{F}^{\mathcal{L}_{k}}_{\mathcal{M}_{i}}(\mathbf{x})$ be the output of
$k^{th}$ hidden layer on the network $\mathcal{M}_{i}$ for the data point
$\mathbf{x}$. Let us assume
$\mathcal{F}^{\mathcal{L}_{k}}_{\mathcal{M}_{i}}(\mathbf{x})$ has
$\mathcal{D}(\mathcal{L}_{k})$ number of output features. Let us consider
$\nabla_{x}\mathcal{F}^{\mathcal{L}_{k}}_{\mathcal{M}_{i}}(\mathbf{x})$ denote
the sum of gradients over each output feature of $k^{th}$ hidden layer with
respect to input on the network $\mathcal{M}_{i}$ for the data point
$\mathbf{x}$. Hence,

$\nabla_{x}\mathcal{F}^{\mathcal{L}_{k}}_{\mathcal{M}_{i}}(\mathbf{x})=\sum_{f=1}^{\mathcal{D}(\mathcal{L}_{k})}\nabla_{x}[\mathcal{F}^{\mathcal{L}_{k}}_{\mathcal{M}_{i}}(\mathbf{x})]_{f}$

where $\nabla_{x}[\mathcal{F}^{\mathcal{L}_{k}}_{\mathcal{M}_{i}}(\cdot)]_{f}$
is the gradient of the $f^{th}$ output feature of $k^{th}$ hidden layer on
network $\mathcal{M}_{i}$ with respect to input for the data point
$\mathbf{x}$. Let $\mathcal{X}$ be the training dataset containing
$|\mathcal{X}|$ examples.

###  4.2 Pairwise Adversarially Robust Loss Function

The principal idea behind the proposed approach is to train an ensemble of
neural networks such that the gradients of loss with respect to input in all
the networks will be in different directions. The gradients represent the
directions in which the input needs to be perturbed such that the loss of the
network increases, helping to transfer adversarial examples. In this paper, we
introduce the Pairwise Adversarially Robust Loss (PARL) function, which we
will use to train the ensemble. The objective of PARL is to train the ensemble
so that the gradients of loss lead to different directions in different
networks for the same input example. Hence, the fundamental strategy is to
make the gradients as dissimilar as possible while training all the networks.
Since the gradient computation depends on all intermediate parameters of a
network, we force the intermediate layers of all the networks within the
ensemble to be dissimilar for producing enhanced diversity at each layer.

The pairwise similarity of gradients of the output of $k^{th}$ hidden layer
with respect to input between the classifiers $\mathcal{M}_{i}$ and
$\mathcal{M}_{j}$ for a particular data point $\mathbf{x}$ can be represented
as

$\mathcal{G}_{\mathcal{L}_{k}}^{(i,j)}(\mathbf{x})=\frac{<\nabla_{x}\mathcal{F}^{\mathcal{L}_{k}}_{\mathcal{M}_{i}}(\mathbf{x}),\nabla_{x}\mathcal{F}^{\mathcal{L}_{k}}_{\mathcal{M}_{j}}(\mathbf{x})>}{\|\nabla_{x}\mathcal{F}^{\mathcal{L}_{k}}_{\mathcal{M}_{i}}(\mathbf{x})\|\cdot\|\nabla_{x}\mathcal{F}^{\mathcal{L}_{k}}_{\mathcal{M}_{j}}(\mathbf{x})\|}$

where $<a,b>$ represents the dot product between two vectors $a$ and $b$. The
overall pairwise similarity between the classifiers $\mathcal{M}_{i}$ and
$\mathcal{M}_{j}$ for a particular data point $\mathbf{x}$ considering
$\mathcal{L}_{\mathcal{H}}$ hidden layers can be written as

$\mathcal{G}^{(i,j)}(\mathbf{x})=\sum_{k=1}^{\mathcal{H}}\mathcal{G}_{\mathcal{L}_{k}}^{(i,j)}(\mathbf{x})$

Next, we define a penalty term $\mathcal{R}(\mathcal{M}_{i},\mathcal{M}_{j})$
for all the training examples in $\mathcal{X}$ to pairwise train the models
$\mathcal{M}_{i}$ and $\mathcal{M}_{j}$ as

$\mathcal{R}(\mathcal{M}_{i},\mathcal{M}_{j})=\frac{1}{|\mathcal{X}|}\sum_{\mathbf{x}\in\mathcal{X}}\mathcal{G}^{(i,j)}(\mathbf{x})$

We can observe that $\mathcal{R}$ computes average pairwise similarity for all
the training examples. Now, for network $\mathcal{M}_{i}$ and
$\mathcal{M}_{j}$, if all the gradients with respect to input for each
training example are in the same direction, value of $\mathcal{R}$ will be
close to $1$, indicating similarity in decision boundaries. The value of
$\mathcal{R}$ will gradually decrease as the relative angle between the pair
of gradients increases in higher dimension. Hence, the objective of diversity
training using PARL is to reduce the value of $\mathcal{R}$. Thus, we add
$\mathcal{R}$ to the loss function as a penalty parameter to penalize the
training for a large $\mathcal{R}$ value.

In the ensemble $\mathcal{M}_{\mathcal{T}}$, we compute the $\mathcal{R}$
values for each distinct pair of $\mathcal{M}_{i}$ and $\mathcal{M}_{j}$ in
order to enforce diversity between each pair of classifiers. We define PARL to
train the ensemble $\mathcal{M}_{\mathcal{T}}$ as

$PARL(\mathcal{M}_{\mathcal{T}})=\gamma_{1}\cdot\frac{1}{|\mathcal{X}|}\sum_{\mathbf{x}\in\mathcal{X}}\sum_{i=1}^{\mathcal{N}}J_{\mathcal{M}_{i}}(\mathbf{x},\mathbf{y})+\gamma_{2}\cdot\sum_{1\leq
i<j\leq\mathcal{N}}\mathcal{R}(\mathcal{M}_{i},\mathcal{M}_{j})$ (1)

where $\gamma_{1}$ and $\gamma_{2}$ are hyperparameters controlling the
accuracy-robustness trade-off. A higher value of $\gamma_{1}$ and a lower
value of $\gamma_{2}$ helps to learn the models with good accuracy but is less
robust against adversarial attacks. However, a lower value of $\gamma_{1}$ and
a higher value of $\gamma_{2}$ makes the models more robust against
adversarial attacks but with a compromise in overall accuracy.

One may note that the inclusion of the penalty values for each distinct pair
of classifiers within the ensemble to compute PARL has one fundamental
advantage. If we do not include the pair $(\mathcal{M}_{a},\mathcal{M}_{b})$
in the PARL computation, the training will continue without any diversity
restrictions between $\mathcal{M}_{a}$ and $\mathcal{M}_{b}$. Consequently,
$\mathcal{M}_{a}$ and $\mathcal{M}_{b}$ will produce similar decision
boundaries, thereby increasing the likelihood of adversarial transferability
between $\mathcal{M}_{a}$ and $\mathcal{M}_{b}$, affecting the robustness of
the ensemble $\mathcal{M}_{\mathcal{T}}$. One may also note that the number of
gradient computations in an efficient implementation of PARL is linearly
proportional to the number of classifiers in the ensemble. The gradients for
each classifier are computed once and are reused to compute $\mathcal{R}$
values for each pair of classifiers. The reuse of gradients protects the
implementation from the exponential computational overhead due to pairwise
similarity computation.

##  5 Experimental Evaluation

###  5.1 Evaluation Configurations

We consider an ensemble of three standard ResNet20 [16] architecture for all
the ensembles used in this paper. We consider two standard image
classification datasets for our evaluation, namely CIFAR-10 [23] and CIFAR-100
[23]. We consider two scenarios for the evaluation:

  * •

Unprotected Ensemble: A baseline ensemble $(ENS_{\mathcal{U}})$ of ResNet20
architectures without any countermeasure against adversarial attacks.

  * •

Protected Ensemble: An ensemble $(ENS_{\mathcal{Z}})$ of ResNet20
architectures, where $\mathcal{Z}$ is the countermeasure used to design the
ensemble. In our evaluation we have considered three previously proposed
countermeasures to compare the performance of PARL. We denote the ensembles
$ENS_{ADP}$, $ENS_{GAL}$, and $ENS_{DVERGE}$ to be the ensembles trained with
the methods proposed by Pang et al. [30], Kariyappa and Qureshi [20], and Yang
et al. [41], respectively. The ensemble trained with our proposed method is
denoted as $ENS_{PARL}$.

We use the adam optimization [21] to train all the ensembles with adaptive
learning rate starting from $0.001$. We dynamically generate a augmented
dataset using random shifts, flips and crops to train both CIFAR-10 and
CIFAR-100. We use the default hyperparameter settings mentioned in the
respective papers for $ENS_{ADP}$, $ENS_{GAL}$, and $ENS_{DVERGE}$222We
implemented $ENS_{GAL}$ following the approach mentioned in the paper.
Whereas, we adopted the official GitHub repositories for $ENS_{ADP}$ and
$ENS_{DVERGE}$ implementation.. We use $\gamma_{1}=1.0$, $\gamma_{2}=0.5$, and
categorical crossentropy loss for $\mathcal{J}_{\mathcal{M}_{i}}(\cdot)$ (ref.
Equation (1)) for $ENS_{PARL}$. We enforce diversity among all the classifiers
in $ENS_{PARL}$ for the first seven convolution layers333We observed that PARL
performs better than previously proposed approaches against adversarial
attacks with high accuracy on clean examples by enforcing diversity in the
first seven convolution layers. We present an Ablation Study by varying the
number of layers utilized for diversity training using PARL in Section 5.4.

We consider four state-of-the-art untargeted adversarial attacks Fast Gradient
Sign Method (FGSM) [12], Basic Iterative Method (BIM) [24], Momentum Iterative
Method (MIM) [9], and Projected Gradient Descent (PGD) [27] for crafting
adversarial examples. A brief overview on crafting adversarial examples using
these attack methodologies are discussed in Appendix A. We consider 50 steps
for generating adversarial examples using the iterative methods BIM, MIM, and
PGD with the step size of $\epsilon/5$, where $\epsilon$ is the attack
strength. We consider the moment decay factor as $0.01$ for MIM. We use $10$
different random restarts for PGD to generate multiple instances of
adversarial examples for an impartial evaluation. We use
$\epsilon=0.01,0.02,0.03,0.04,0.05,0.06$, and $0.07$ for generating
adversarial examples of different strengths. We use CleverHans v2.1.0 library
[33] to generate all the adversarial examples.

In order to evaluate against a stronger adversarial setting for
$\mathcal{A}_{\mathcal{Z}}$, we train a black-box surrogate ensemble and
generate adversarial examples from the surrogate ensemble instead of a
standalone classifier. As also mentioned previously, adversarial examples that
mislead multiple models in an ensemble tend to be more transferable [9, 26].
All the results in the subsequent discussions are reported by taking average
value over three independent runs.

###  5.2 Analysing the Diversity

The primary objective of PARL is to increase the diversity among all the
classifiers within an ensemble. In order to analyze the diversity of different
classifiers trained using PARL, we use Linear Central Kernel Alignment (CKA)
analysis proposed by Kornblith et al. [22]. The CKA metric, which lies between
$[0,1]$, measures the similarity between decision boundaries represented by a
pair of neural networks. A higher CKA value between two neural networks
indicates a significant similarity in decision boundary representations, which
implies good transferability of adversarial examples. We present an analysis
on layer-wise CKA values for each pair of classifiers within the ensemble
$ENS_{\mathcal{U}}$ and $ENS_{PARL}$ trained with CIFAR-10 and CIFAR-100 in
Figure 2 to show the effect of PARL on diversity.

(a)  (b)  (c)  (d)  (e)  (f)

Figure 2: Layer-wise linear CKA values between each pair of classifiers in
$ENS_{\mathcal{U}}$ and $ENS_{PARL}$ trained with CIFAR-10 [(a), (b), (c)] and
CIFAR-100 [(d), (e), (f)] datasets showing the similarities at each layer. The
value inside the braces within the corresponding figure legends represent the
overall average Linear CKA values between each pair of classifiers.

We can observe that each pair of models in $ENS_{\mathcal{U}}$ show a
significant similarity at every layer. However, since $ENS_{PARL}$ is trained
by restricting the first seven convolution layers, we can observe a
considerable decline in the CKA values at the initial layers. The observation
is expected as $PARL$ imposes layer-wise diversity in its formulation, as
discussed previously in Section 4. The overall average Linear CKA values
between each pair of models in Figure 2 are mentioned inside braces within the
corresponding figure legends, which signifies that the classifiers within an
ensemble trained using PARL shows a higher overall dissimilarity than the
unprotected baseline ensemble. In the subsequent discussions, we analyze the
effect of the observed diversity on the performance of $ENS_{PARL}$ against
adversarial examples.

###  5.3 Robustness Evaluation of PARL

Performance in the presence of $\mathcal{A}_{\mathcal{Z}}$: We evaluate the
robustness of all the ensembles discussed in Section 5.1 considering both
CIFAR-10 and CIFAR-100 where the attackers cannot access the model parameters
and rely on surrogate models to generate transferable adversarial examples.
Under such a black-box scenario, we use one hold-out ensemble with three
ResNet20 models as the surrogate model. We randomly select 1000 test samples
and evaluate the performance of black-box transfer attacks for all the
ensembles across a wide range of attack strength $\epsilon$. We present the
result for the attack strength ($\epsilon=0.07$) in Table 1 along with clean
example accuracies. The methods presented by Kariyappa and Qureshi [20] and
[41] neither evaluated their approach on a more complicated CIFAR-100 dataset
nor discussed any optimal hyperparameter settings regarding their proposed
algorithms for potential evaluation. In our robustness evaluations, we
consider all the ensembles discussed in Section 5.1 for CIFAR-10. In contrast,
we only consider $ENS_{\mathcal{U}}$ and $ENS_{ADP}$ to compare the
performance of $ENS_{PARL}$ for CIFAR-100. We can observe from Table 1 that
the training using PARL does not adversely affect the ensemble accuracy on
clean examples compared to other previously proposed methodologies. However,
it provides better robustness against black-box transfer attacks for almost
every scenario.

Table 1: Ensemble classification accuracy (%) for CIFAR-10 and CIFAR-100 on
clean examples as well as adversarially perturbed images with attack strength
$\epsilon=0.07$ for different adversarial attacks.

|  | Clean Example | FGSM | BIM | MIM | PGD
---|---|---|---|---|---|---
$ENS_{\mathcal{U}}$ | CIFAR-10 | 93.11 | 21.87 | 7.47 | 7.27 | 1.57
CIFAR-100 | 70.88 | 7.13 | 7.17 | 3.93 | 5.39
$ENS_{ADP}$ | CIFAR-10 | 92.99 | 23.53 | 8.13 | 7.53 | 3.08
CIFAR-100 | 70.01 | 8.23 | 10.07 | 5.83 | 8.72
$ENS_{GAL}$ | CIFAR-10 | 91.22 | 22.8 | 9.3 | 8.33 | 7.64
CIFAR-100 | * | * | * | * | *
$ENS_{DVERGE}$ | CIFAR-10 | 91.73 | 28.88 | 11.78 | 10.55 | 9.68
CIFAR-100 | * | * | * | * | *
$ENS_{PARL}$ | CIFAR-10 | 91.09 | 28.37 | 20.8 | 16.17 | 15.65
CIFAR-100 | 67.52 | 12.73 | 18.97 | 10.01 | 21.49

  * •

* Did not consider CIFAR-100 dataset for evaluation

A more detailed performance evaluation considering all the attack strengths
are presented in Figure 3. For CIFAR-10 evaluation (ref. Figure 3a \- Figure
3d), we can observe that $ENS_{PARL}$ performs at par with $ENS_{DVERGE}$
considering FGSM. However, for stronger iterative attacks like BIM, MIM, and
PGD, $ENS_{PARL}$ outperforms other methodologies with higher accuracy for
large attack strengths. For CIFAR-100 evaluation (ref. Figure 3e \- Figure
3h), we can observe that $ENS_{PARL}$ performs better than both
$ENS_{\mathcal{U}}$ and $ENS_{ADP}$ for all scenarios.

(a) FGSM (b) BIM (c) MIM (d) PGD (e) FGSM (f) BIM (g) MIM (h) PGD

Figure 3: Ensemble classification accuracy (%) v.s. Attack Strength
($\epsilon$) against different black-box transfer attacks generated from
surrogate ensemble for CIFAR-10 [(a), (b), (c), (d)] and CIFAR-100 [(e), (f),
(g), (h)] datasets.

Performance in the Presence of $\mathcal{A}_{\mathcal{P}}$: Next, we evaluate
the robustness of ensembles when the attacker has complete access to the model
parameters. Under such a white-box scenario, we craft adversarial examples
from the target ensemble itself. We consider the same attack methodologies and
settings as discussed in Section 5.1. We randomly select 1000 test samples and
evaluate white-box attacks for all the ensembles across a wide range of attack
strength $\epsilon$. We present the results for CIFAR-10 and CIFAR-100 in
Figure 4. For CIFAR-10 evaluation (ref. Figure 4a \- Figure 4d), we can
observe that $ENS_{PARL}$ performs marginally better than the other ensembles.
For CIFAR-100 evaluation (ref. Figure 4e \- Figure 4h), we can observe that
$ENS_{PARL}$ performs better than both $ENS_{\mathcal{U}}$ and $ENS_{ADP}$ for
all scenarios. Although PARL achieves the highest robustness among all the
previous ensemble methods for black-box transfer attacks, its robustness
against white-box attacks is still quite low. The result is expected as the
objective of PARL is to increase the diversity of an ensemble against
adversarial vulnerability rather than entirely eliminate it. In other words,
adversarial vulnerability inevitably exists within the ensemble and can be
captured by attacks with white-box access. One straightforward way to improve
the robustness of ensembles against such vulnerability is to augment PARL with
adversarial training [27], which we look forward to as a future research
direction.

(a) FGSM (b) BIM (c) MIM (d) PGD (e) FGSM (f) BIM (g) MIM (h) PGD

Figure 4: Ensemble classification accuracy (%) v.s. Attack Strength
($\epsilon$) against different white-box attacks for CIFAR-10 [(a), (b), (c),
(d)] and CIFAR-100 [(e), (f), (g), (h)] datasets.

###  5.4 Ablation Study

In all the previous evaluations, we consider $ENS_{PARL}$ by enforcing
diversity in the first seven convolution layers for all the classifiers during
ensemble training using PARL. In this section, we provide an ablation study by
analyzing a varying number of convolution layers considered for the diversity
training. We consider three ensembles $ENS_{PARL/7}$, $ENS_{PARL/5}$, and
$ENS_{PARL/3}$ for this study, where $ENS_{PARL/N}$ denotes that the first $N$
convolution layers are used for enforcing the diversity. We consider the same
evaluation configurations as discussed in Section 5.1. The accuracies of all
the ensembles on clean examples considering both CIFAR-10 and CIFAR-100 are
mentioned in Table 2. We can observe that as fewer restrictions are imposed,
the overall ensemble accuracy increases, which is expected and can be followed
from Equation (1). A detailed analysis on the diversity achieved by all these
ensembles in terms of Linear CKA metric is provided in Appendix B for
interested readers.

Table 2: Ensemble classification accuracy (%) on the test set for CIFAR-10 and
CIFAR-100. The numbers in the first row after slash denote the number of
convolution layers used to enforce diversity

| $ENS_{PARL/7}$ | $ENS_{PARL/5}$ | $ENS_{PARL/3}$
---|---|---|---
CIFAR-10 | 91.09 | 91.18 | 92.45
CIFAR-100 | 67.52 | 67.54 | 68.81

Next, we evaluate the robustness of these ensembles against black-box transfer
attacks for the evaluation configurations discussed in Section 5.1 and provide
the results in Figure 5. We can observe that though the ensemble accuracy on
clean examples increases with fewer layer restrictions, the robustness against
black-box transfer attacks decreases significantly. The results present an
interesting trade-off between accuracy and robustness in terms of number of
layers considered while computing PARL (ref. Equation (1)).

(a) FGSM (b) BIM (c) MIM (d) PGD (e) FGSM (f) BIM (g) MIM (h) PGD

Figure 5: Ensemble classification accuracy (%) v.s. Attack Strength
($\epsilon$) against different black-box transfer attacks generated from
surrogate ensemble for CIFAR-10 [(a), (b), (c), (d)] and CIFAR-100 [(e), (f),
(g), (h)] datasets. The numbers in the figure legends after slash denote the
number of convolution layers used to enforce diversity

##  6 Conclusion

In this paper, we propose an approach to enhance the classification robustness
of an ensemble against adversarial attacks by developing diversity in the
decision boundaries among all the classifiers within the ensemble. The
ensemble network is constructed by the proposed Pairwise Adversarially Robust
Loss function utilizing the gradients of each layer with respect to input in
all the networks simultaneously. The experimental results show that the
proposed method can significantly improve the overall robustness of the
ensemble against state-of-the-art black-box transfer attacks without
substantially impacting the clean example accuracy. Combining the technique
with adversarial training and exploring different efficient methods to
construct networks with diverse decision boundaries adhering to the principle
outlined in the paper can be interesting future research directions.

## References

  * Adam et al. [2018] George-Alexandru Adam, Petr Smirnov, Anna Goldenberg, David Duvenaud, and Benjamin Haibe-Kains.  Stochastic combinatorial ensembles for defending against adversarial examples.  _CoRR_ , abs/1808.06645, 2018.  URL http://arxiv.org/abs/1808.06645.
  * Amodei et al. [2016] Dario Amodei et al.  Deep speech 2 : End-to-end speech recognition in english and mandarin.  In _Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016_ , volume 48 of _JMLR Workshop and Conference Proceedings_ , pages 173–182. JMLR.org, 2016.  URL http://proceedings.mlr.press/v48/amodei16.html.
  * Athalye et al. [2018] Anish Athalye, Nicholas Carlini, and David A. Wagner.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 274–283. PMLR, 2018.  URL http://proceedings.mlr.press/v80/athalye18a.html.
  * Bastani et al. [2016] Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya V. Nori, and Antonio Criminisi.  Measuring neural net robustness with constraints.  In _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain_ , pages 2613–2621, 2016.  URL http://papers.nips.cc/paper/6339-measuring-neural-net-robustness-with-constraints.
  * Bhagoji et al. [2017] Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal.  Dimensionality reduction as a defense against evasion attacks on machine learning classifiers.  _CoRR_ , abs/1704.02654, 2017.  URL http://arxiv.org/abs/1704.02654.
  * Bojarski et al. [2016] Mariusz Bojarski et al.  End to end learning for self-driving cars.  _CoRR_ , abs/1604.07316, 2016.  URL http://arxiv.org/abs/1604.07316.
  * Carlini and Wagner [2017a] Nicholas Carlini and David A. Wagner.  Adversarial examples are not easily detected: Bypassing ten detection methods.  In _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AISec@CCS 2017, Dallas, TX, USA, November 3, 2017_ , pages 3–14. ACM, 2017a.  doi: 10.1145/3128572.3140444.  URL https://doi.org/10.1145/3128572.3140444.
  * Carlini and Wagner [2017b] Nicholas Carlini and David A. Wagner.  Towards evaluating the robustness of neural networks.  In _2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017_ , pages 39–57. IEEE Computer Society, 2017b.  URL https://doi.org/10.1109/SP.2017.49.
  * Dong et al. [2018] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li.  Boosting adversarial attacks with momentum.  In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_ , pages 9185–9193. Computer Vision Foundation / IEEE Computer Society, 2018.  doi: 10.1109/CVPR.2018.00957.  URL http://openaccess.thecvf.com/content_cvpr_2018/html/Dong_Boosting_Adversarial_Attacks_CVPR_2018_paper.html.
  * Feinman et al. [2017] Reuben Feinman, Ryan R. Curtin, Saurabh Shintre, and Andrew B. Gardner.  Detecting adversarial samples from artifacts.  _CoRR_ , abs/1703.00410, 2017.  URL http://arxiv.org/abs/1703.00410.
  * Gong et al. [2017] Zhitao Gong, Wenlu Wang, and Wei-Shinn Ku.  Adversarial and clean data are not twins.  _CoRR_ , abs/1704.04960, 2017.  URL http://arxiv.org/abs/1704.04960.
  * Goodfellow et al. [2015] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6572.
  * Grefenstette et al. [2018] Edward Grefenstette, Robert Stanforth, Brendan O’Donoghue, Jonathan Uesato, Grzegorz Swirszcz, and Pushmeet Kohli.  Strength in numbers: Trading-off robustness and computation via adversarially-trained ensembles.  _CoRR_ , abs/1811.09300, 2018.  URL http://arxiv.org/abs/1811.09300.
  * Grosse et al. [2017] Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick D. McDaniel.  On the (statistical) detection of adversarial examples.  _CoRR_ , abs/1702.06280, 2017.  URL http://arxiv.org/abs/1702.06280.
  * Gu and Rigazio [2015] Shixiang Gu and Luca Rigazio.  Towards deep neural network architectures robust to adversarial examples.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings_ , 2015\.  URL http://arxiv.org/abs/1412.5068.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_ , pages 770–778. IEEE Computer Society, 2016.  doi: 10.1109/CVPR.2016.90.  URL https://doi.org/10.1109/CVPR.2016.90.
  * Hendrycks and Gimpel [2017] Dan Hendrycks and Kevin Gimpel.  Early methods for detecting adversarial images.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=B1dexpDug.
  * Huang et al. [2015] Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvári.  Learning with a strong adversary.  _CoRR_ , abs/1511.03034, 2015.  URL http://arxiv.org/abs/1511.03034.
  * Jin et al. [2015] Jonghoon Jin, Aysegul Dundar, and Eugenio Culurciello.  Robust convolutional neural networks under adversarial noise.  _CoRR_ , abs/1511.06306, 2015.  URL http://arxiv.org/abs/1511.06306.
  * Kariyappa and Qureshi [2019] Sanjay Kariyappa and Moinuddin K. Qureshi.  Improving adversarial robustness of ensembles with diversity training.  _CoRR_ , abs/1901.09981, 2019.  URL http://arxiv.org/abs/1901.09981.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6980.
  * Kornblith et al. [2019] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton.  Similarity of neural network representations revisited.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 3519–3529. PMLR, 2019.  URL http://proceedings.mlr.press/v97/kornblith19a.html.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  _Technical Report_ , 2009.  URL http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.222.9220&rep=rep1&type=pdf.
  * Kurakin et al. [2017] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio.  Adversarial examples in the physical world.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=HJGU3Rodl.
  * Li and Li [2017] Xin Li and Fuxin Li.  Adversarial examples detection in deep networks with convolutional filter statistics.  In _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_ , pages 5775–5783. IEEE Computer Society, 2017.  doi: 10.1109/ICCV.2017.615.  URL https://doi.org/10.1109/ICCV.2017.615.
  * Liu et al. [2017] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song.  Delving into transferable adversarial examples and black-box attacks.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=Sys6GJqxl.
  * Madry et al. [2018] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018.  URL https://openreview.net/forum?id=rJzIBfZAb.
  * Metzen et al. [2017] Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff.  On detecting adversarial perturbations.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=SJzCSf9xg.
  * Moosavi-Dezfooli et al. [2016] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.  Deepfool: A simple and accurate method to fool deep neural networks.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_ , pages 2574–2582. IEEE Computer Society, 2016.  URL https://doi.org/10.1109/CVPR.2016.282.
  * Pang et al. [2019] Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu.  Improving adversarial robustness via promoting ensemble diversity.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 4970–4979. PMLR, 2019.  URL http://proceedings.mlr.press/v97/pang19a.html.
  * Papernot et al. [2016] Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami.  Distillation as a defense to adversarial perturbations against deep neural networks.  In _IEEE Symposium on Security and Privacy, SP 2016, San Jose, CA, USA, May 22-26, 2016_ , pages 582–597. IEEE Computer Society, 2016\.  doi: 10.1109/SP.2016.41.  URL https://doi.org/10.1109/SP.2016.41.
  * Papernot et al. [2017] Nicolas Papernot, Patrick D. McDaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami.  Practical black-box attacks against machine learning.  In _Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, AsiaCCS 2017, Abu Dhabi, United Arab Emirates, April 2-6, 2017_ , pages 506–519. ACM, 2017.  doi: 10.1145/3052973.3053009.  URL https://doi.org/10.1145/3052973.3053009.
  * Papernot et al. [2018] Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Kurakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang, Yi-Lin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, Rujun Long, and Patrick McDaniel.  Technical report on the cleverhans v2.1.0 adversarial examples library.  _CoRR_ , abs/1610.00768, 2018.  URL https://arxiv.org/abs/1610.00768.
  * Rozsa et al. [2016] Andras Rozsa, Ethan M. Rudd, and Terrance E. Boult.  Adversarial diversity and hard positive generation.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2016, Las Vegas, NV, USA, June 26 - July 1, 2016_ , pages 410–417. IEEE Computer Society, 2016.  doi: 10.1109/CVPRW.2016.58.  URL https://doi.org/10.1109/CVPRW.2016.58.
  * Shaham et al. [2015] Uri Shaham, Yutaro Yamada, and Sahand Negahban.  Understanding adversarial training: Increasing local stability of neural nets through robust optimization.  _CoRR_ , abs/1511.05432, 2015.  URL http://arxiv.org/abs/1511.05432.
  * Strauss et al. [2017] Thilo Strauss, Markus Hanselmann, Andrej Junginger, and Holger Ulmer.  Ensemble methods as a defense to adversarial perturbations against deep neural networks.  _CoRR_ , abs/1709.03423, 2017.  URL http://arxiv.org/abs/1709.03423.
  * Szegedy et al. [2014] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , 2014.  URL http://arxiv.org/abs/1312.6199.
  * Szegedy et al. [2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_ , pages 2818–2826. IEEE Computer Society, 2016.  doi: 10.1109/CVPR.2016.308.  URL https://doi.org/10.1109/CVPR.2016.308.
  * Tramèr et al. [2018] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian J. Goodfellow, Dan Boneh, and Patrick D. McDaniel.  Ensemble adversarial training: Attacks and defenses.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018.  URL https://openreview.net/forum?id=rkZvSe-RZ.
  * Wu et al. [2016] Yonghui Wu et al.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _CoRR_ , abs/1609.08144, 2016.  URL http://arxiv.org/abs/1609.08144.
  * Yang et al. [2020] Huanrui Yang, Jingyang Zhang, Hongliang Dong, Nathan Inkawhich, Andrew Gardner, Andrew Touchet, Wesley Wilkes, Heath Berry, and Hai Li.  DVERGE: diversifying vulnerabilities for enhanced robust generation of ensembles.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/3ad7c2ebb96fcba7cda0cf54a2e802f5-Abstract.html.
  * Zheng et al. [2016] Stephan Zheng, Yang Song, Thomas Leung, and Ian J. Goodfellow.  Improving the robustness of deep neural networks via stability training.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_ , pages 4480–4488. IEEE Computer Society, 2016.  doi: 10.1109/CVPR.2016.485.  URL https://doi.org/10.1109/CVPR.2016.485.

##  Appendix A Brief Overview of Adversarial Example Generation

Let us consider a benign data point $x\in\mathbb{R}^{d}$, classified into
class $\mathcal{C}_{i}$ by a classifier $\mathcal{M}$. An untargeted
adversarial attack tries to add visually imperceptible perturbation
$\eta\in\mathbb{R}^{d}$ to $x$ and creates a new data point $x_{adv}=x+\eta$
such that $\mathcal{M}$ misclassifies $x_{adv}$ into another class
$\mathcal{C}_{j}$ other than $\mathcal{C}_{i}$. The imperceptibility is
enforced by restricting the $l_{\infty}$-norm of the perturbation $\eta$ to be
below a threshold $\epsilon$, i.e., $\|\eta\|_{\infty}\leq\epsilon$ [12, 27].
We term $\epsilon$ as the attack strength.

An adversary can craft adversarial examples by considering the loss function
of a model, assuming that the adversary has full access to the model
parameters. Let $\mathcal{J}(\theta,x,y)$ denote the loss function of the
model, where $\theta$, $x$, and $y$ represent the model parameters, benign
input, and corresponding label, respectively. The goal of the adversary is to
generate adversarial example $x_{adv}$ such that the loss of the model is
maximized while ensuring that the magnitude of the perturbation is upper
bounded by $\eta$. Hence,
$\mathcal{J}(\theta,x_{adv},y)>\mathcal{J}(\theta,x,y)$ adhering to the
constraint $\|x_{adv}-x\|_{\infty}\leq\epsilon$. Several methods have been
proposed in the literature to solve this constrained optimization problem. We
discuss the methods used in the evaluation of our proposed approach.

Fast Gradient Sign Method (FGSM): This attack is proposed by Goodfellow et al.
[12]. The adversarial example is crafted using the following equation

$x_{adv}=x+\epsilon\cdot sign(\nabla_{x}J(\theta,x,y))$

where $\nabla_{x}J(\theta,x,y)$ denote the gradient of loss with respect to
input.

Basic Iterative Method (BIM): This attack is proposed by Kurakin et al. [24].
The adversarial example is crafted iteratively using the following equation

$x^{(k+1)}=x^{(k)}+clip_{\epsilon}(\alpha\cdot
sign(\nabla_{x}J(\theta,x^{(k)},y)))$

where $x^{(0)}$ is the benign example. If $r$ is the total number of attack
iteration, $x_{adv}=x^{(r)}$. The parameter $\alpha$ is a small step size
usually chosen as $\epsilon/r$. The function $clip_{\epsilon}(\cdot)$ is used
to generate adversarial examples within $\epsilon$-ball of the original image
$x^{(0)}$.

Moment Iterative Method (MIM): This attack is proposed by Dong et al. [9],
which won the NeurIPS 2017 Adversarial Competition. This attack is a variant
of BIM that crafts adversarial examples iteratively using the following
equations

$\displaystyle g^{(k)}$ $\displaystyle=\nabla_{x}J(\theta,x^{(k)},y)$
$\displaystyle g^{(k+1)}$ $\displaystyle=\mu\cdot
g^{(k)}+\frac{g^{(k)}}{\|g^{(k)}\|_{1}}$ $\displaystyle x^{(k+1)}$
$\displaystyle=x^{(k)}+clip_{\epsilon}(\alpha\cdot sign(g^{(k)}))$

where $\mu$ is termed as the decay factor.

Projected Gradient Sign Method (PGD): This attack is proposed by [27]. This
attack is a variant of BIM with same adversarial example generation process
except for $x^{(0)}$ is a randomly perturbed image in the $\epsilon$
neighborhood of the original image.

##  Appendix B Diversity Results for Ablation Study

In this section, we present an analysis on layer-wise CKA values for each pair
of classifiers within the ensemble $ENS_{PARL/7}$, $ENS_{PARL/5}$, and
$ENS_{PARL/3}$ trained with both CIFAR-10 and CIFAR-100 datasets. The layer-
wise CKA values are shown in Figure 6 to demonstrate the effect of PARL on
diversity. We can observe a more significant decline in the CKA values at the
initial layers for $ENS_{PARL/7}$ as compared to $ENS_{PARL/5}$ and
$ENS_{PARL/3}$, which is expected as $ENS_{PARL/7}$ is trained by restricting
more convolution layers. We can also observe that each pair of classifiers
show more overall diversity in $ENS_{PARL/7}$ than in $ENS_{PARL/5}$ and
$ENS_{PARL/3}$. The overall CKA values are mentioned inside braces within the
corresponding figure legends.

(a)  (b)  (c)  (d)  (e)  (f)

Figure 6: Layer-wise linear CKA values between each pair of classifiers in
$ENS_{PARL/7}$, $ENS_{PARL/5}$, and $ENS_{PARL/3}$ trained with CIFAR-10 [(a),
(b), (c)] and CIFAR-100 [(d), (e), (f)] datasets showing the similarities at
each layer. The numbers in the figure legends after slash denote the number of
convolution layers used to enforce diversity. The value inside the braces
within the corresponding figure legends represent the overall average Linear
CKA values between each pair of classifiers.


# Crypto-Hotwire: Illegal Blockchain Mining at Zero Cost Using Public
Infrastructures

Felipe Ribas Coutinho

UFRJ  Rio de Janeiro, Brazil  Victor Pires

UFRJ  Rio de Janeiro, Brazil  Claudio Miceli

UFRJ  Rio de Janeiro, Brazil  Daniel S. Menasché

UFRJ  Rio de Janeiro, Brazil

###### Abstract

Blockchains and cryptocurrencies disrupted the conversion of energy into a
medium of exchange. Numerous applications for blockchains and cryptocurrencies
are now envisioned for purposes ranging from inventory control to banking
applications. Naturally, in order to mine in an economically viable way,
regions where energy is plentiful and cheap, e.g., close to hydroelectric
plants, are sought. The possibility of converting energy into cash, however,
also opens up opportunities for a new kind of cyber attack aimed at illegally
mining cryptocurrencies by stealing energy. In this work, we indicate, using
data from January and February of 2018 from our university, that such a threat
is real, and present a projection of the gains derived from these attacks.

††conference: Symposium on Cryptocurrency Analysis (SOCCA) 2020 Milan, Italy

##  1 Introduction

With the popularization of virtual currencies (also known as
cryptocurrencies), a new class of cyberattacks has emerged: cryptojacking. It
consists of infecting and controlling the use of CPU and memory of computers
for purposes of mining cryptocurrencies. According to [12], in 2018 such
attacks were already more prevalent than classic attacks, such as DDoS, in
certain regions of the US [6, 5, 10, 13, 4].

Mining attacks previously reported in the literature [7, 8, 11, 12, 14, 9]
typically account for hacked computers being remotely exploited and locally
monitored. We are unaware of reports on the use of public computing
infrastructures in which machines can be locally exploited and remotely
monitored. This occurs, for instance, in universities and coffee shops, where
a vast number of users has physical access to machines, posing additional
challenges to their remote monitoring, e.g., by the university information
technology (IT) team [6, 5, 10, 13, 4].

We refer to this new type of attack as “crypto-hotwire”, in a reference to
criminals that bypass meters and “hotwire” the power supply to cut bills [2].
Note that physical “hotwire” is typical in countries such as Brazil,
evidencing problems of public order which lend energy theft as one of the
major drivers for smart grids. The theft of energy may occur for a number of
undue purposes. In the case of crypto-hotwire, the theft of energy occurs for
the mining of virtual currencies, which serves as a general medium of
exchange.

Public infrastructures, in particular, are major targets of crypto-hotwire,
since they have a large number of machines, significant Internet bandwidth and
abundant energy. The large number of machines makes it easier for users to
physically access the resources and turn miners on and off. In contrast, to
make use of home users’ machines, e.g., through a botnet, such household
machines need to have vulnerabilities that allow the attacker to invade the
system remotely. The attack complexity is reduced when attackers can have
physical access to a machine to install a mining module.

Once mining begins on a public infrastructure machine, it leaves almost no
trace on the network, rendering its detection very challenging. This is
because simply monitoring the network may not be enough to identify attacks,
since they have a strongly local aspect and do not leverage any explicit
software vulnerabilities. As a case study, in this work we study the action of
attackers who seek to invade our university’s machines to execute mining
processes. This study aims to $(a)$ assess the prevalence of mining attacks at
the university and $(b)$ analyze the data generated by incidents, such as
traffic and energy consumption, in order to generate a behavior pattern for
detecting this type of attack.

In summary, the main contributions of this article are:

We found that illegal cryptocurrency mining is prevalent:  Using data
collected at the Federal University of Rio de Janeiro (UFRJ), we found that
illegal cryptocurrency mining occurs in practice. In particular, we were able
to identify dozens of attempts of mining activities targeting the Monero
cryptocurrency. Unlike classic attacks that leave a trace on the bandwidth of
the network, cryptocurrency mining can be very difficult to detect because it
involves virtually only local activity at the affected machine.

We quantify the costs and gains from illegal mining: Through controlled lab
experiments, we quantify the costs and gains arising from the mining. In
particular, we find that costs can escalate more quickly that earnings. Such a
finding may motivate a wide increase in illegal use of resources.

The remainder of this paper is organized as follows. After providing basic
background, we present our research methodology in Section 3. Sections 4 and 5
present our main findings, on the prevalence and gains from mining, and
Section 6 concludes.

##  2 Mining and mining attacks

The process of generating cryptocurrencies on a network is called mining. This
process consists of solving a mathematical problem (puzzle) and being rewarded
for that solution. Mining demands high energy and power consumption for
solving the puzzles. In order for mining to become a profitable process for
the miner, it is necessary to reduce these costs. In an extreme situation, the
miner can reduce personal costs to zero, making use of energy from public
infrastructure. If this is the case, the costs are incurred by those who
finance such infrastructures.

The mining process can be carried out on a single computer or using making
such a group. In “solo” mining, the reward generated by the mining depends
exclusively on the processing power of the computer. In pool mining, many
computers mine together and the accumulated processing power increases the
chances of block resolution. When the block is resolved, the response
compensation is proportionally divided according to the power used by each
one. Currently, there are tools that allow any user to easily connect a
machine to a mining pool. A user who has physical access to public
infrastructure machine can connect it to a mining pool and individually profit
from it.

Traditional mining attacks.  Next, we present a brief introduction to mining
attacks, indicating their classic types and what distinguishes them when the
focus is on public infrastructure. A mining or cryptojacking attack consists
of the infection of one or more hosts, inserting in them malware that runs
cryptocurrency mining software. It is worth mentioning that in this work we
consider users who intentionally mine virtual coins on local physical
machines. Alternatively, in the literature the focus has been on remote users
performing such unwanted mining. This occurs in two ways: via Javascript or
worms. In the first case, the mining code is executed in the browser and the
loading of this code is done along with the loading of the desired page by the
user, in a covert way. In the case of worms, the executed code is browser
independent, potentially being downloaded individually.

In the case of Javascript, monitoring CPU usage and memory of the browser, one
can identify potential problems, while in the case of worms, it is necessary
to monitor the CPU and memory usage of the machine as a whole, as it is not
known in advance what process is doing malicious executions. Such solutions
are effective for home users, but they do not easily scale. Anomalies in the
use of CPU and memory at the scale of universities, for instance, should be
much harder to track than anomalies in a local machine.

New attack vectors and access vectors.  Although the topic of cryptojacking is
in vogue [12, 7], we are unaware of previous work that has analyzed the risks
associated with this type of attack in public infrastructures, in which users
have physical access to devices. Anecdotal evidence suggests that those
attacks are prevalent [4, 13, 10].

In related works, the authors consider that the browser does not intentionally
access mining scripts (i.e., it is assumed a remote access vector) and that
the honest user is interested in monitoring (locally) for suspicious activity.
In this work, on the other hand, we assume that the user intentionally
executes mining scripts (local access vector, in physical facilities) and that
monitoring needs to be remote, e.g., through deep packet inspection.

The problem considered in this work is particularly complex in developing
countries, such as Brazil. In these countries, we have a significant portion
of the population accessing the electrical network by means of physical
hotwires. Such physical hotwires potentially gives rise to energy theft. The
crypto-hotwire described in this work can be appreciated as a sophisticated
version of the physical energy theft which are prevalent in slums. Although
substantial resources have been invested to detect such physical energy theft,
they are still widely present. Thus, their virtual counterparts, being more
subtle, pose a greater threat.

##  3 Methodology

In the following, we describe the experimental methodology for measuring the
network of our university. We focus on monitoring the prevalence of mining in
the institution.

Our university has several systems for detecting invasions and threats that
are used to provide security for its students, faculty and staff. In this
work, we consider data provided by two tools used by the University
Information Security: SGIS – System Incident Management System Security,
provided by the National Education and Research Network and the management
system of tickets used by the institution. The ticket system includes tickets
requested by members of our university, while SGIS generates tickets using the
result of network traffic scans at the PoP (point of presence at the state of
Rio de Janeiro, encompassing many academic institutions).

Discovery of mining attacks begins with the notification of any
irregularities. The notification of irregularities occurs in three ways,
carried out in parallel: $(a)$ notification from partner institutions and
government agencies informing that the hosts are communicating with other
infected mining machines; $(b)$ monitoring the university’s network traffic by
checking if the host is communicating with mining servers, also known as
mining pools; $(c)$ notifications from a machine indicating anomalies in
behavior.

From the notification, a physical audit is performed on the suspect machines
to detect the possible attack vector involved. In this audit, data is also
collected on the configurations of the machines involved. The audit consists
of:

Physical inspection. Physically reaching the machine, and analyzing the
purpose of the machine and active connections using Wireshark or tcpdump, in
conjunction with netstat (e.g., many active UDP connections for a personal
computer from a faculty in the History department can indicate a suspicion
that the machine has been compromised)

Analysis of vulnerabilities. Analysis of the vulnerabilities of the machine,
from an external view, e.g., Shodan and from an internal view, e.g., using
Apache vulnerability tracker provided by Kali Linux if installed on the
machine

Search for additional evidences. Search in the machine’s folders (in
particular, Apache subdirectories) for a (binary) file that is typically used
for illicit purposes. In our case, we focus on searching for cryptocurrency
mining files. So we do a search in the names of all files on the machine for
strings such as “crypto”, “Monero” and “Bitcoin”. We also search for such
strings in the system’s cron for scheduling mining processes, as well as as in
the bashrc history

Analysis of browsing history. Search the history of sites accessed by the
browser for suspicious sites. For example, in the case of suspected
cryptocurrency mining, it is checked whether there are records of access to
publicly known mining sites, eg, http://btc.com.

The attack vectors that were found at our university were: (i) virtual
epidemic vectors (worms); (ii) virtual targeted vectors (attacks on the
network, coming from an Internet connection); (iii) physical vectors (attacks
on the network, resulting from physical access to computers). In the
particular case of cryptocurrency mining, physical access to machines
represents an important vector.

1–––––7––––––14––––––21––––––28–––$0$$5$$10$(a) January 2018 (days)reported
casescrypto-hotwire cases

1–––––7––––––14––––––21––––––28–––$0$$5$$10$(a) February 2018 (days)

Figure 1: Number of incidents (blue) and crypto-hotwire incidents (red)
between January 1 and February 28, 2018.

##  4 Prevalence of illegal mining

The observed crypto-hotwire mining incidents occurred during the period from
January 17 to February 28, 2018. In February 2018 mitigation strategies were
deployed, and since then it has been more difficult to identify mining
attacks.

In this study, we focus on general notifications received between January 1
and February 28, 2018. Figure 1 reports the number of notifications of
security incidents during the analyzed period. They cover notifications of
various types of attacks, including DDoS, copyright infringement and brute
force attacks on ssh. In Figure 1 we notice that there are contiguous periods
of greater notification activity, for example, between January 25 and February
2, and between February 19 and February 25. In other contiguous periods, for
example, between January 1 and 8, and between February 11 and 16, we have few
notifications. The main reason for having these contiguous periods is related
to the fact that when an attack occurs, the attacker often manages to
compromise multiple machines. When the problem is mitigated, the system is
renewed and switches to a new cycle.

Out of the reported incidents, 144 cases were classified as crypto-hotwire
incidents. The dates of the notifications for these cases are reported in red
bars in Figure 1. The notifications are daily, and the script that takes
information from our university’s backbone uses deep packet inspection and
generates the notifications. The script runs throughout the day, and at
midnight it restarts the counter of notifications. This explains why, in
almost all days, the number of notifications ranges from 7 to 6. Probably the
devices compromised and the users running illegal mining were almost always
the same, and their activity generated around 7 predictable notifications per
day (e.g., in communication between the user and the mining center). It should
be noted that on some days, for example, January 27 (Saturday) we have no
notifications because our university network is unstable on weekends,
undergoing maintenance in some of these periods. On certain days, for example,
on January 24, we had a fall in the number of crypto-hotwire notifications,
which subsequently increased on 25 January. This is due to the the fact that
on January 24 the mining problem was solved provisionally, reinstalling some
machines in the system. However, the problem returned as the attack vector was
still active and resumed its activities on January 25. Below, we describe in
more detail one of the episodes that occurred during this period.

Bitcoin mining episode. In a unit of our university, we identified computers
mining Bitcoin. From a browser’s history, we identified that a user had
accessed mining pool sites, http://btc.com and http://f2pool.com. On these
sites, the user logs in and leaves the computer on to raise funds from mining.
Carelessly, the user under consideration left a file with a trace of his
logging and mining passwords on the machine. With the credentials of access to
the mining pools, it was possible to identify the start and end times of each
mining activity. It should be noted that the notification of the above event,
which alerted the security to take initiatives, was generated by the fact that
there was frequent access to mining pool websites. Thus, suspicious traffic
was generated. If mining had been done executing minimum interactions with the
rest of the network, we believe that it would have been much more difficult to
detect mining activity.

Among the most mined virtual currencies at our university, we highlight
Bitcoin and Monero. Bitcoin was the first virtual currency to be adopted on a
large scale, which justifies its prevalence. Bitcoin is a reference for other
cryptocurrencies, and there are already numerous simple programs to mine
Bitcoin on any platform (Linux, MacOS, Android and Windows, in the most
diverse versions). Monero [1] provides its users with privacy levels greater
than those offered by Bitcoin. In addition, like Bitcoin, it also provides
mining software facilities for script kiddies. For these reasons, many miners
have adopted Monero as the first choice cryptocurrency, as identified in our
measurements.

##  5 Estimated gains and costs

Next, we report results obtained with controlled experiments in the lab to
estimate gains and costs associated with mining at our university. In
particular, we sought to emulate the machines that we found to be mining in
January and February 2018, as described in the previous section.

Controlled experiment environment. We consider three configurations that
correspond to the machines that we have found to be mining Monero and Bitcoin
at our university. It is not surprising that the configurations are standard
on computers widely used through our university, considering that equipment
purchases are frequently made in batches. Thus, the configurations also
capture potential mining gains and costs across our university as a whole.

The experiment counted with 3 machines. Configurations 1, 2 and 3 correspond
to an Intel Pentium Dual Core, Intel Core i3 and Intel Core i5, all with 4 Gb
of RAM and 2 cores except i5 which counts with 4 cores. The hosts run the
Linux Ubuntu 16.04 operating system, and the energy consumption monitoring
software PowerTOP. The mining software chosen was MinerGate, for the Monero
virtual currency. For network data collection purposes (monitoring network
flows) we use NFDump (netflow).

Experimental results.  Let $r(j)$ be the hashing power of a machine with
configuration $j$. In our measurements, we found
$r(1)=85H/s,r(2)=88H/s,r(3)=177H/s$, respectively. In a system where $N$
machines have configuration $j$, the total hashing power of the system is
given by $R(j)=Nr(j)$, $j=1,2,3$. In what follows, we consider $N=3$ which
corresponds to approximately the number of machines found mining at our
university between January and February 2018.

The daily gain corresponding to the hashing power reported above, with the
$N=3$ hosts, is given by $G(1)$ = R$ 0.27, $G(2)$ = R$ 0.28 and $G(3)$= R$
5.61, under configurations 1, 2 and 3 respectively. To obtain such values, we
use the average quotation of Monero in the Brazilian currency (reais) during
the study period. It is worth noting that the expected gains are very small
compared to costs, which are an order of magnitude higher, as indicated below.

To estimate the energy expenditure per host, we measure the energy consumption
of machines. The measured energy consumption of the machines was $E^{(1)}=21$
W/h, $E^{(2)}=40$ W/h and $E^{(3)}=40$ W/h, for configurations 1, 2 and 3,
respectively. Multiplying by the number of active hosts assumed to be 3, and
for the cost of energy $c$ we obtain the estimated total cost,
$C^{(j)}=E^{(j)}Nc$, with $C(j)$ equal to R$ 61.69, R$ 117.50 and R$ 117.50
for configurations 1, 2 and 3, respectively.

Until then, we have not taken into account the network patterns associated
with mining in our controlled experiment, nor the underlying network costs.
For a preliminary analysis on network aspects, we consider 3 additional
machines with the same configurations as the ones considered so far. Such
additional machines are baseline machines, which run general-purpose desktop
applications, and which do not mine. One of our purposes is to contrast the
network usage of the corresponding machines. As shown in Figure 2, the traffic
pattern of hosts that are mining is lower than that of hosts performing
standard equipment activities. This fact corroborates the idea that it is more
challenging to identify mining attacks remotely, as considered in this work,
in comparison with local detection in the browser, using mine blockers tools
[3].

Figure 2: Traffic generated by miners is similar or smaller than standard
desktops.

Monetization of mining for public benefit.  The monetization of mining, to the
benefit of the university itself, is potentially promising, _but involves
ethical aspects and environmental implications._ Since the university has
periods of low movement or idleness of resources, which causes resources to be
underutilized, mining may be considered during these periods. Predicting the
value of cryptocurrencies and the surplus energy with which each institution
can count a priori, however, is non-trivial. In domestic households in
numerous cities in the Brazil, for example, there is a minimum energy
consumption associated with the basic minimum value of the electricity bill.
In homes where this limit is not reached, or in underutilized commercial
establishments, e.g., affected by the COVID-19 pandemic, it is possible to
monetize the monthly energy surplus.

##  6 Conclusion

In this paper we have reported mining attacks leveraging direct physical
access to computational resources. Such attack vectors are easy to implement
in public infrastructures, and those attacks potentially generate financial
resources that can be converted into any type of goods. We presented
measurements indicating that those mining attacks occurred at our university.
In particular, we indicate that Bitcoin and Monero were the two most mined
coins. We believe that this work opens up many directions for future
developments. In particular, we envision measurements to assess the prevalence
of crypto-hotwire attacks beyond the university environment, e.g., in regions
where energy theft is prevalent [2], counting with collaborations between
energy and communication service providers. As another direction for future
research, we envision new services leveraging the monetization of mining for
public benefit, and leave their ethical and environmental implications as
subject for future work.

## References

  * [1] Andersen, D. Mining money with monero and cpu vector intrinsics, 2014.  https://tinyurl.com/moneroda.
  * [2] BBC.  Criminal gangs ’hotwire power supply’ to cut bills, 2014.  https://www.bbc.com/news/uk-england-25718447.
  * [3] Cherone, L., and Mahon, P. Minerblock extension, 2018.  https://tinyurl.com/minerblock.
  * [4] Coindesk.  London student mines dogecoin with university’s computers – they don’t know, yet, 2014.  https://tinyurl.com/coindesklondon.
  * [5] CoinTelegraph.  Free money: How students mine cryptocurrency in their dorm rooms, 2019\.  https://tinyurl.com/coinfreemoney.
  * [6] Govtech.  Cryptocurrency mining threat on k-12 computers likely to ease soon, 2018\.  https://tinyurl.com/govtechcrypto.
  * [7] Hong, G., Yang, Z., and Yang, S. How you get shot in the back: A systematical study about cryptojacking in the real world.  In ACM SIGSAC Security (2018).
  * [8] Kharraz, A., Ma, Z., Murley, P., Lever, C., et al. Outguard: Detecting in-browser covert cryptocurrency mining in the wild.  The web conference (2019).
  * [9] Pastor, A., Mozo, A., Vakaruk, S., et al. Detection of encrypted cryptomining malware connections with machine and deep learning.  IEEE Access (2020).
  * [10] PC Magazine.  College kids are using campus electricity to mine crypto, 2019.  https://tinyurl.com/pcmagcrypto2019.
  * [11] Rüth, J., Zimmermann, T., Wolsing, K., et al. Digging into browser-based crypto mining.  In IMC (2018).
  * [12] Segura, J. The state of malicious cryptomining, 2018.  https://tinyurl.com/maliccry.
  * [13] Vectra.  The alarming surge in cryptocurrency mining on college campuses, 2018\.  https://tinyurl.com/vectraai.
  * [14] Zimba, A., and Wang, Z. Crypto mining attacks in information systems: An emerging threat to cyber security.  Journal of Computer Information Systems (2018), 13.


11institutetext: Institute of Information Systems (IFIS), University of
Lübeck,
Ratzeburger Allee 160, 23562 Lübeck, Germany

# Variables are a Curse
in Software Vulnerability Prediction

Jinghua Groppe  Sven Groppe  Ralf Möller

###### Abstract

Deep learning-based approaches for software vulnerability prediction currently
mainly rely on the original text of software code as the feature of nodes in
the graph of code and thus could learn a representation that is only specific
to the code text, rather than the representation that depicts the ‘intrinsic’
functionality of a program hidden in the text representation. One curse that
causes this problem is an infinite number of possibilities to name a variable.
In order to lift the curse, in this work we introduce a new type of edge
called name dependence, a type of abstract syntax graph based on the name
dependence, and an efficient node representation method named 3-property
encoding scheme. These techniques will allow us to remove the concrete
variable names from code, and facilitate deep learning models to learn the
functionality of software hidden in diverse code expressions. The experimental
results show that the deep learning models built on these techniques
outperform the ones based on existing approaches not only in the prediction of
vulnerabilities but also in the memory need. The factor of memory usage
reductions of our techniques can be up to the order of 30,000 in comparison to
existing approaches††This work is part of the BMBF project with the contract
number 16KIS1337..

###### Keywords:

deep learning software security software vulnerability abstract syntax graph
3-property encoding name dependence

##  1 Introduction

A number of efforts have been dedicated to applying deep learning (DL) to
predict the vulnerabilities of software code. However, DL-based approaches
have not achieved significant breakthroughs in this field and still have a
limited capability to distinguish vulnerable code from non-vulnerable one [1].
Currently, DL approaches, both unstructured [5, 8, 2, 11] or structure-based
[9, 6, 7, 10, 1], borrowed the method used in the natural language processing
to define the semantics of the full code or nodes in a code graph. The full
code or a piece of the code is considered plain text like a natural language
and it is first split into tokens, and each token is represented by a real-
valued vector called embedding. Unstructured approaches learn the
representation of the code only based on the sequence of the tokens. The
sophisticated graph-based approaches learn a presentation based on the tokens
appearing in each node and the relations between nodes.

A functionality can be programmed using an infinite number of text
representations and one main reason for the infinity is the arbitrariness in
naming variables. For example, the functionality of the summation of two
variables can be coded as $a+b$, $x1+x2$ or using any other names. Since
different names have different embeddings, a DL model, which learns based on
the raw code text, could only find a representation, which is specific to the
code text with the used variable names, and would not be able to capture the
intrinsic functionality beyond the diversity of code expression using
different variable names.

We could not obtain a well-generalized model in the presence of an infinity of
text code of a functionality. Therefore, we need solutions to transform an
infinite number of text representations of variable names into a finite number
and we are suggesting such a solution in this work. Concretely, we suggest a
new edge type of name dependence and an abstract syntax graph (ASG) that
extends a standard abstract syntax tree (AST) with the edges of name
dependence and develop a 3-property node encoding scheme based on the ASG.
These techniques can be used to remove variable names from code, and greatly
mitigate the semantic uncertainty of variables and the infinity of text code
of a functionality. The empirical evidence presented later shows that our
techniques do help DL models to learn the intrinsic functionality of the
software and improve their prediction performance.

##  2 Breaking the Curse of Variables

In order to help DL models of software vulnerability prediction to improve
their generalization ability, in this section, we suggest techniques of how to
transform an infinite number of text representations of varable names into a
finite number.

###  2.1 Name Dependence and Abstract Syntax Graph

In programming languages, a variable is related to its declaration (which is
either explicitly given or implied). We can determine this relation by the
name of the variable. Software engineering uses the term ‘dependence’ to
describe the relations between two components, like data dependence, and
control dependence. To align with it, we define a new kind of dependence
called name dependence to express the relation between a variable and its
declaration. In an AST with full information, the name dependence between two
nodes can be inferred by the names of variables and identifiers. When we
remove the names of variables and identifiers from the AST, we lose the
information on name dependence. Without the information, we will not be able
to restore the semantics of the original code. So, we need a way to express
the name dependence when names are absent. A solution is to add an edge of
name dependence between two related nodes. After adding such edges, the tree
structure turns into a graph structure as illustrated in Fig. 1, which we call
abstract syntax graph (ASG). From the graph, we can construct a fragment of
code with the exact semantics as the original code, but perhaps with a
different text representation, which would not be a problem at all for the
task of vulnerability prediction.

Figure 1: ASG example

Table 1: 3-property encoding scheme

| 3-prop. encode | 3-prop. encode
---|---|---
| with variable names | without variable
construct | class | name | type | class | name | type
int a | varDecl | a | int | varDecl | - | int
If (a$\geq$0) | control | IF | - | control | IF | -
a*0.01 | mathOp | mul | - | mathOp | MUL | -
f(a) | call | f | - | call | f | -
a | ident | a | int | ident | VAR | int
stdout | ident | stdout | - | ident | stdout | -
{…} | block | - | - | block | - | -
10.01 | literal | 10.01 | float | literal | - | float
‘Hi’ | literal | ‘Hi’ | str | literal | - | str
int[8] b | varDecl | b | int[8] | varDecl | - | int[N]

###  2.2 3-Property Encoding Scheme

Apart from the ASG, we further suggest a method to efficiently represent the
nodes in a code graph, 3-property encoding, which provides a consistent
description of the feature of nodes and allows DL models to infer the commons
and differences between nodes easily. This 3-property encoding is developed in
the context of our ASG but it can be applied to other code graphs and it is
also programming languages agnostic.

In a code graph, every node represents an executable syntactic construct in
code, which can be an expression, a statement or its constituent parts, like
variables and constants (which are of course also executable). Currently, the
piece of code that consists of the construct (with or without a notation to
the construct like ‘varDecl’ and ‘add’) is used as the feature of the node.
The feature is encoded by first splitting the piece of code into tokens and
then averaging the embeddings of all the tokens. The code-based encoding uses
the original piece of code to present the feature of a node, and at the same
time, the result of encoding blurs the semantics of the original code since
the averaging operation. Our 3-property encoding avoids these two issues by
introducing additional information related to the language constructs.

Each language construct has its properties, which may not explicitly appear in
the raw code text. Independent of specific programming languages, we found
that it is enough to use three properties to describe different constructs:
class, name and type of data if any, and each value of the properties will be
represented by a unique token. Table 1 demonstrates several common language
constructs and their representation with the three properties. With this
property-based approach, we can encode all nodes in a consistent way, and this
is a very valuable characteristic for many applications. So far, this
3-property encoding has not removed the diversity of text representations and
we will further normalize this encoding scheme to mitigate the diversity as
much as possible based on the name dependence and ASG.

Besides the variable names which we can remove thanks to the edges of name
dependence, there are also other constructs in code, which can have any
values. One of them is literals, e.g. 0.01 and ‘Hello’, which will cause
similar issues as variable names, so we will also remove the concrete value of
a literal. Another construct is array declarations with size, e.g. $char[8]$,
$char[1024]$. We will normalize them as e.g. $char[N]$. A more refined
solution could be to create several normalized data types e.g., $char[int8]$,
$char[int16]$ and so on, and normalize the data type of arrays according to
their sizes. For instance, any char arrays with sizes between 0 and 256 could
be normalized to $char[int8]$. Table 1 also provides examples of normalized
representations. The definition of the classes of language constructs and the
normalized tokens could vary depending on the implementation of applications
and the tool for generating code graphs.

##  3 Evaluation

In order to evaluate our techniques, we build four types of code graphs, AST,
AST+, ASG and ASG+, for training DL models of software vulnerability
prediction. AST+ is an AST extended with flow and data dependencies and
control flow. ASG is an AST with the edges of name dependence and variable
names removed, and ASG+ is ASG with flow and data dependencies and control
flow.

#### Models:

We develop two models (3propASG and 3propASG$+$), which use our graph
structures and 3-property node encoding scheme, and two baselines (codeAST and
codeAST+), which adopt the common graph structures and the pieces of code as
the feature of nodes (i.e., the code-based encoding presented in Section 2)
that is currently adopted by existing models [10, 1]. All models share the
following architecture: the input data is delivered to the layer of GGRU with
one time step, the least expensive option. The output of GGRU is sent to each
of three 1D convolution (Conv1d) layers with 128 filters each and perceptive
fields of 1, 2, and 3 respectively, and one 1D max pooling (MaxPool1d) is
applied over the output of each Conv1D to perform downsample. The results of
the MaxPool1d layers are concatenated together and sent to the hidden layer
with 128 neurons, and a 25% dropout is applied to the output of convolution
and the hidden layer. We apply Relu for non-linear transformation and the
embeddings of 100 dimensions to encode tokens.

#### Datasets:

We use several real-world datasets from different open-source projects:
Chromium+Debian [1], which contains 10,699 samples and 7.05% of which are
flawed; FFmpeg+Quemu [10] with 13,428 samples and 43.68% flawed; VDISC [8]
with 68,398 samples and 46.38% flawed. The tool
Joern111https://github.com/joernio/joern is utilized to create the AST and
AST+ from source code and our AST+ corresponds the code property graph of
Joern.

#### Performance:

We use 80% of the datasets as training data and 20% for validation and
evaluation. The models are trained with a batch size of 32 and a learning rate
of 0.001, and the Adam optimizer [4] is used to minimize the loss function.
Since much empirical evidence (e.g. [3]) has shown that the pre-trained
embeddings are not necessarily better than random initializations. Therefore,
we use the standard normal distribution $\mathcal{N}(0,1)$ to initialize the
embeddings and train models with 10 different initializations. Table 2
presents the performance of models with the best F1 values. The evaluation
results show that the DL models based on our graph structures (ASG and ASG+)
and 3-property encoding scheme outperform those based on existing graph
structures (AST and AST+) and code-based encoding over all the datasets. Among
these datasets, Chromium+Debian is extremely imbalanced and contains only 592
(6.92%) samples with vulnerability. Over this dataset, our models perform
significantly well with F1. These results are strong evidence that our
techniques improve the ability of DL models to infer the functionality of
code.

#### Memory Requirement:

A huge advantage of our 3-property encoding is that it has a very low memory
footprint and can process very large code graphs in comparison to the existing
code-based encoding. In our experiments, an 8G memory is enough to process all
the data using the 3-property encoding. In comparison, the code-based encoding
requires as much as 560G memory. With the 3-property encoding, the feature of
each node is represented by only three tokens. With the code-based encoding,
the feature of each node is represented by a piece of raw code. Although
different pieces of code will create different number of tokens and the
minimal node could contain only one token, all nodes are required to have the
same number of tokens. This means that all the nodes in a code graph finally
consist of the maximal number of tokens.

Table 2: Performance of models over the datasets |  |  | Chromium+Debian | FFmpeg+Quemu | VDISC
---|---|---|---|---|---
Model | Graph | Encoding | Acc | F1 | Acc | F1 | Acc | F1
codeAST | AST | code | 92.01 | 30.20 | 55.36 | 57.01 | 77.82 | 75.57
3propASG | ASG | 3-Prop. | 92.34 | 44.97 | 60.35 | 62.30 | 81.27 | 79.86
codeAST+ | AST+ | code | 90.89 | 25.86 | 58.38 | 46.66 | 75.67 | 74.49
3propASG+ | ASG+ | 3-Prop. | 92.34 | 44.59 | 57.04 | 62.99 | 80.94 | 79.63
Table 3: Memory need of three samples from Chromium+Debian Hash (Code ID) | #nodes | #tokens | code-based | 3-prop. | code-based
---|---|---|---|---|---
/3-prop.
-6552851419396579257 | 4,409 | 33,659 | 59G | 5.3M | 11,220
2388171415474875762 | 7,012 | 54,157 | 152G | 8.4M | 18,052
5045872831385413038 | 12,077 | 96,805 | 468G | 14.5M | 32,268

Table 3 provides the memory footprint required by our 3-property encoding and
the existing code-based encoding for processing these samples. The comparison
shows that our encoding scheme can be up to 30,000 times more efficient than
the code-based encoding. This explains why existing works [10, 1] only use the
code samples with a number of nodes less than 500.

##  4 Conclusions

In order to break the curse of variables, we introduce the edges of name
dependence and ASG extending AST with this new type of edges and suggest a
3-property node encoding scheme based on the ASG. These techniques not only
allow us to represent the semantics of code without using its variable names
but also allow us to encode all nodes in a consistent way. The evaluation
shows that our techniques do improve the abilities of DL models to predict
software vulnerabilities. Furthermore, we also believe that the 3-property
encoding will be also a useful technique for many tasks in software analysis
and software engineering.

## References

  * [1] Chakraborty, S., Krishna, R., Ding, Y., Ray, B.: Deep learning based vulnerability detection: Are we there yet. IEEE Transactions on Software Engineering (2021)
  * [2] Dam, H.K., Tran, T., Pham, T., Ng, S.W., Grundy, J., Ghose, A.: Automatic feature learning for vulnerability prediction. arXiv preprint arXiv:1708.02368 (2017)
  * [3] Groppe, J., Schlichting, R., Groppe, S., Möller, R.: Deep learning-based classification of customer communications of a german utility company. In: Proceedings of the International Semantic Intelligence Conference. pp. 1–16. Springer Nature (2022)
  * [4] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [5] Li, Z., Zou, D., Xu, S., Ou, X., Jin, H., Wang, S., Deng, Z., Zhong, Y.: Vuldeepecker: A deep learning-based system for vulnerability detection. arXiv preprint arXiv:1801.01681 (2018)
  * [6] Lin, G., Zhang, J., Luo, W., Pan, L., Xiang, Y., De Vel, O., Montague, P.: Cross-project transfer representation learning for vulnerable function discovery. IEEE Transactions on Industrial Informatics 14(7), 3289–3297 (2018)
  * [7] Pradel, M., Sen, K.: Deepbugs: A learning approach to name-based bug detection. Proceedings of the ACM on Programming Languages 2(OOPSLA), 1–25 (2018)
  * [8] Russell, R., Kim, L., Hamilton, L., Lazovich, T., Harer, J., Ozdemir, O., Ellingwood, P., McConley, M.: Automated vulnerability detection in source code using deep representation learning. In: 17th IEEE international conference on machine learning and applications (ICMLA). pp. 757–762 (2018)
  * [9] Wang, S., Liu, T., Tan, L.: Automatically learning semantic features for defect prediction. In: 38th International Conference on Software Engineering. pp. 297–308 (2016)
  * [10] Zhou, Y., Liu, S., Siow, J., Du, X., Liu, Y.: Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems 32 (2019)
  * [11] Zou, D., Wang, S., Xu, S., Li, Z., Jin, H.: Vuldeepecker: A deep learning-based system for multiclass vulnerability detection. IEEE Transactions on Dependable and Secure Computing 18(5), 2224–2236 (2019)