# PASA: Attack Agnostic Unsupervised Adversarial Detection using Prediction &
Attribution Sensitivity Analysis

Dipkamal Bhusal  Rochester Institute of Technology
Rochester, NY, USA  Md Tanvirul Alam  Rochester Institute of Technology
Rochester, NY, USA  Monish K. Veerabhadran  Rochester Institute of Technology
Rochester, NY, USA  Michael Clifford  Toyota Motor North America
Mountain View, CA, USA  Sara Rampazzi  University of Florida
Gainesville, FL, USA  Nidhi Rastogi  Rochester Institute of Technology
Rochester, NY, USA

###### Abstract

Deep neural networks for classification are vulnerable to adversarial attacks,
where small perturbations to input samples lead to incorrect predictions. This
susceptibility, combined with the black-box nature of such networks, limits
their adoption in critical applications like autonomous driving. Feature-
attribution-based explanation methods provide relevance of input features for
model predictions on input samples, thus explaining model decisions. However,
we observe that both model predictions and feature attributions for input
samples are sensitive to noise. We develop a practical method for this
characteristic of model prediction and feature attribution to detect
adversarial samples. Our method, PASA, requires the computation of two test
statistics using model prediction and feature attribution and can reliably
detect adversarial samples using thresholds learned from benign samples. We
validate our lightweight approach by evaluating the performance of PASA on
varying strengths of FGSM, PGD, BIM, and CW attacks on multiple image and non-
image datasets. On average, we outperform state-of-the-art statistical
unsupervised adversarial detectors on CIFAR-10 and ImageNet by 14% and 35%
ROC-AUC scores, respectively. Moreover, our approach demonstrates competitive
performance even when an adversary is aware of the defense mechanism.

##  1 Introduction

Deep neural networks (DNNs) have demonstrated state-of-the-art performance in
various classification tasks [24, 7, 2]. However, DNNs are known to be
vulnerable to adversarial evasion attacks. Attackers carefully and
deliberately craft samples by adding small perturbations to fool the DNN and
cause it to make incorrect predictions [22, 12, 40]. The susceptibility of
DNNs to such attacks poses serious risks when deploying them in application
scenarios where security and reliability are essential, such as in autonomous
vehicles [32] and medical diagnosis[52].

Current approaches for defending against such evasion attacks can be broken
into two broad categories. One category increases the robustness of a model
(e.g., adversarial training [22], feature denoising [70]). However, such
approaches achieve model robustness at the cost of modification of the network
architecture or training process and compromise natural accuracy as a result.
Such methods are still susceptible to adversarial attacks like blind-spot
attacks [74]. The other category identifies adversarial samples instead of
making robust classifications. While detecting adversarial attacks is as
challenging as classifying them [64], such methods are useful in many
practical situations where discarding adversarial samples for security or
generating an alert for human intervention is possible. Detection methods are
supervised if they require both benign and adversarial samples in their
training [19, 39]. The main limitations of supervised detection methods are
the requirement of prior attack knowledge and the availability of adversarial
samples. In contrast, unsupervised methods solely rely on properties of
natural (benign) samples for training [41, 71].

Figure 1: Benign (1st column) and Adversarial PGD Image (3rd column).
Corresponding Integrated Gradient (IG) Attribution (2nd and 4th column).

A different line of research, called post-hoc explanation methods, addresses
the black-box nature of DNNs [53, 45, 60, 37, 10]. Post-hoc explanation
methods explain the decision made by a DNN for a test input based on its input
features, and enhance our understanding of the DNN’s decision-making process.
For example, in an image classifier, such methods can identify the key pixels
in an input image that lead to a DNN decision. Explanations can use various
approaches such as feature attribution, rules, or counterfactuals to explain
an instance [5]. Feature attribution-based methods, such as Integrated
Gradient (IG) [60], assign attribution or relevance scores to each input
feature, quantifying the importance of the feature to the model’s prediction.
Recent research has explored the application of feature-attribution-based
explanation methods in detecting adversarial attacks [63, 73, 72, 68, 67].
However, these methods require both benign and adversarial samples to train an
additional classifier for detection and do not incorporate features from the
classification model in the detection pipeline.

Proposed Approach: We propose a novel method for detecting adversarial samples
by combining model prediction and feature attribution. Our approach is
motivated by the evident differences in model prediction and feature
attribution between benign and adversarial samples when noise is introduced:
(a) we observe that a DNN exhibits distinct behavior when noise (e.g.,
gaussian noise) is introduced to adversarial samples, similar to studies
performed by Roth et al. [46] and Hu et al. [28]. However, while prior works
[46, 28] look for noise that does not change the model prediction for benign
samples, we empirically identify noise that maximizes the distinction between
the behavior of benign and adversarial samples. (b) There are noticeable
differences in the attribution map of benign and adversarial samples. Figure 1
illustrates these differences in images where the distribution of attribution
scores varies significantly for adversarial samples, evident from the change
in red and blue pixels in the attribution map. Even though adversarial samples
are crafted by adding small perturbations to the input data, we observe that
their feature attribution differs markedly from those of benign samples. This
distinction in feature attribution becomes more pronounced when noise is
introduced to samples. (c) Examining these discrepancies in model prediction
and feature attribution of benign and adversarial samples subjected to
additional perturbation can effectively detect adversarial attacks, and ensure
the security of systems incorporating deep learning models.

We introduce PASA111“PASA” is the Newari term for “friend,” a Sino-Tibetan
language spoken by the indigenous people of the Kathmandu Valley, Nepal., a
threshold-based, unsupervised method for detecting adversarial samples, using
Prediction & Attribution Sensitivity Analysis. We use noise as a probe to
modify input samples, measure changes in model prediction and feature
attribution, and learn thresholds from benign samples. At test time, PASA
computes model prediction and feature attribution of a given input and its
noisy counterpart and rejects the input if the change in model prediction or
feature attribution does not meet the predefined threshold. We demonstrate the
effectiveness of our lightweight detection approach by evaluating its
performance on five different datasets (MNIST [34], CIFAR-10 [31], CIFAR-100
[31], ImageNet [17] and updated CIC-IDS2017 [18]) and five different deep
neural network architectures (MLP [47], LeNet [34], VGG16 [31], ResNet [25],
and MobileNet [48]). On average, PASA outperforms other state-of-the-art
statistical unsupervised detectors (FS [71], MagNet [41], LOO [72], TWS [28])
by 14% on CIFAR-10, 4% on CIFAR-100 and 35% on ImageNet. We further evaluate
PASA under an adaptive adversary setting and demonstrate its robustness. We
observe that performing adaptive attacks against both model prediction and
feature attribution increases computational complexity for an adversary, and
PASA still achieves competitive performance. PASA has low inference latency,
and the simplicity yet effectiveness of this approach makes it suitable for
deployment in scenarios with limited computational resources. Our code is
available at {https://github.com/dipkamal/PASA}.

##  2 Background and Related Work

Deep Neural Network: Deep neural networks (DNNs) learn efficient
representations of training data by extracting features using interconnected
neurons. Let $(X,Y)$ represent the training data where $X$ reflects input
space and $Y$ reflects label space. Then, a deep neural network ($F$)
generates a highly accurate functional representation $F:X\rightarrow Y$. Such
networks are trained using backpropagation, a gradient-based optimization
algorithm, which adjusts the weights between neurons to minimize the error
between model predictions and ground truth labels. For example: Convolutional
Neural Networks (CNNs) are a type of DNN used for image, video, tabular, and
text data [34]. In supervised classification, a CNN is trained using $N$
labeled samples. The $i^{th}$ sample $({\textbf{x}_{i}},y_{i})$ consists of an
input $\textbf{x}_{i}$ with label $y_{i}$. The final layer of the network is
often referred to as a “logit” layer and consists of $k$ neurons corresponding
to the $k$ target classes. The “logits,” $Z(\textbf{x})$ are the unnormalized
scores the network generates for each class before applying a softmax
function, which maps the logits to the probability distribution over the
classes. Logits represent the internal representations of the input data
learned by the network. During training, the network’s parameters are adjusted
to minimize the difference between predicted probabilities and the actual
labels of the training data. This optimization process helps the network learn
to assign higher logits to the correct classes and lower logits to incorrect
classes. For a $k$-class classifier, the output vector of probabilities $y$ is
obtained by applying the softmax function to the logits $Z(\textbf{x})$. The
final model prediction is the class with the highest probability. Let $y$ be
the output vector of probabilities then,
$y=\text{softmax}(Z(\textbf{x}))=\frac{\exp(Z(\textbf{x}))}{\sum_{j=1}^{k}\exp(Z_{j}(\textbf{x}))}$
where $Z(\textbf{x})$ are the logits generated by the CNN for the input image
x and $y$ is the resulting vector of class probabilities.

Adversarial Attack: Though highly accurate, DNNs are vulnerable to adversarial
attacks that can cause input misclassifications [22, 12, 40, 6, 33]. Given a
model $F$ and input sample $\textbf{x},y$, the goal of an adversarial evasion
attack is to modify the sample x by adding a perturbation such that
$F(\textbf{x})\neq F(\textbf{x}^{*})$ and
$||\textbf{x}^{*}-\textbf{x}||<\epsilon$ where $\epsilon\in R^{n}$ is the
maximum perturbation allowed and x* is the adversarial sample. In targeted
attacks, an adversary aims to misclassify the input sample into a specific
class such that $F(\textbf{x}^{*})=t$ where $t$ is the target label in the
label space. In untargeted attacks, an adversary aims to misclassify an input
into any other class but the correct one. Untargeted attacks have fewer
perturbations than targeted attacks and have better success rates with strong
transferability capability [12, 36]. Adversarial attacks also differ based on
the distance measures, usually defined as $L_{p}$ norm ($L_{0}$, $L_{1}$,
$L_{2}$, and $L_{\infty}$), between the benign and adversarial input. Based on
adversary knowledge of the target classifier, attacks can further be
classified as black-box (no information), gray-box (partial information), and
white-box (complete information) attacks. For example, the Fast Gradient Sign
Attack (FGSM) [22] assumes a linear approximation of the network loss function
and finds a perturbation by increasing the local linear approximation of the
loss. The Basic Iterative Method (BIM) [33] is an iterative version of FGSM
where the perturbation is computed multiple times with small steps. Projected
Gradient Descent (PGD) [40] is also an iterative method similar to BIM.
However, unlike BIM, it starts from a random perturbation in the $L_{\infty}$
ball around the input sample. Auto-PGD attack [14] is a gradient-based
adversarial attack that reduces the parameter dependence on step-size of the
PGD attack. Carlini and Wagner (CW) [12] attacks comprise a range of attacks
that follow an optimization framework similar to L-BFGS [61]. However, they
replace the loss function with an optimization problem involving logits
$(Z(.))$, instead of using the model prediction.

Defense Against Attack: There are two categories of approaches to adversarial
attack mitigation. The first category focuses on improving model robustness
against attacks. For example, adversarial training [22] augments natural
training data with adversarial samples and performs model training to build a
robust classifier that utilizes both original and perturbed samples. However,
this method requires information on the generation of adversarial samples,
compromises the benign accuracy of the model, and is still susceptible to
adversarial attacks like blind-spot attacks [12, 74]. The second category of
defense focuses on detecting and rejecting adversarial samples at test time.
The detection can be supervised or unsupervised. The supervised detection
methods extract features of benign and adversarial samples and train another
network for detection [19, 39, 56, 69]. However, the detection network can
also be compromised by adversarial attacks [11]. Since supervised approaches
require prior knowledge of attacks and the availability of adversarial samples
for training, it can be a major limitation. On the other hand, unsupervised
detection methods require only benign samples for training. Such methods
extract features from benign samples and compute thresholds that measure
inconsistency between properties of benign and adversarial samples. For
example, Feature Squeezing [71] identifies adversarial images by compressing
the input space using image filters and comparing its prediction vectors with
that of original images using a threshold learned from benign images. MagNet
[41] uses denoisers trained on benign images to reconstruct input samples. It
assumes that the threshold-based reconstruction error will be smaller for
benign images than for adversarial images. While effective on small datasets
(MNIST, CIFAR-10), none of these methods perform well on larger images such as
ImageNet [11]. DNR [55] uses features of benign images from different layers
of a network to build a detection method but requires training additional
N-SVM classifiers with an RBF kernel. NIC [38] also extracts the activation
distribution of benign images in network layers but builds an additional set
of models.

Similar to our work, Roth et al. [46] and Hu et al. [28] use noise as a probe
for adversarial detection. Roth et al. [46] compute log-odds robustness, the
changes in logits between each pair of predicted classes, for a given image
and its noisy counterpart. They learn threshold from benign images and use it
for adversarial detection. However, Hosseini et al. [27] generate adversarial
images using mimicry attacks that can bypass this statistical approach. The
method also requires generating over 2000 noisy samples per image, making it
unsuitable when prediction time is of concern. Hu et al. [28] compare the
change in softmax scores between the original and noisy input images and learn
the detection threshold from benign images. However, the prediction
probability from a softmax distribution poorly corresponds to the model’s
confidence [26]. Consequently, applying the softmax function to logits results
in a loss of discriminating information [1] and can be ineffective in
detecting adversarial detection. Both approaches aim to preserve the model
prediction of benign images and do not account for changes in feature
attribution.

Explanation Method: Explanation methods consist of techniques that explain a
prediction of a black-box model in a post-hoc manner. Given a trained model
and a test input, such methods provide explanations in terms of attribution
scores or rules to explain why the model made a certain prediction. Feature
attribution, a type of post-hoc explanation method, assigns an attribution
score to each input feature of an instance, indicating its importance in the
model’s prediction [45]. Given a trained model $F(.)$ and a test instance
$\textbf{x}\in R^{d}$, a feature attribution-based explanation method $\phi$
returns an attribution vector $\phi(\textbf{x})\in R^{d}$. The attribution
vector is a vector of scores that quantify the importance of the input
features in the model prediction of the test instance. Backpropagation-based
attribution methods utilize gradients to propagate the model prediction to the
input layer. For example, the Vanilla Gradient method [53] calculates the
gradient of the class score (output) with respect to the input. Integrated
Gradient (IG) method [60] aggregates gradients along a linear path from a
baseline to the test input. The choice of the baseline is specific to the use-
case [58] and should have a near-zero score for model prediction. IG satisfies
fundamental axioms of attribution methods: sensitivity (the feature is
essential for prediction), implementation (attribution method is efficient and
scalable), and completeness (attribution score of input features adds up to
output score for an input) [60]. It also produces more stable explanations
than the Vanilla Gradient method [5].

Attack Detection Using Explanation: Recent research has explored using feature
attribution for adversarial detection. For example: (a) Tao et al. [63]
identify critical neurons in feature attribution of faces to detect
adversarial images. However, their approach is limited to face recognition
systems. (b) Zhang et al. [73] train a supervised classifier using the Vanilla
Gradient method-based feature attribution. However, their additional networks
for detection can be vulnerable to adversarial attacks. (c) ML-LOO [72]
extracts feature attribution using the leave-one-out (LOO) method [35] and
trains several detectors using benign and adversarial images. However, they
train several attack-specific detectors by extracting attribution from hidden
layers, which can be computationally expensive and may not be scalable to a
large number of attacks. (d) X-ensemble [68] and ExAD [67] train an ensemble
network by extracting feature attribution using different explanation methods
for benign and adversarial images. However, this approach also requires prior
information on attacks and feature attribution for various adversarial samples
and explanation methods, making it difficult to apply in real-world scenarios.

##  3 Motivation

We provide motivations for our detector design by discussing adversarial
perturbation regions around benign samples, and the influence of noise on
model prediction and feature attribution. For this discussion, we consider $F$
to be a target image classification model, x is an input image, and
$Z(\textbf{x})$ is the logits given by the model. We sample noise
$\eta\in\mathcal{N}(0,\sigma^{2})$ (where $\sigma^{2}$ is a hyperparameter)
and obtain a noisy version of the input image
$\textbf{x}^{\prime}=\textbf{x}+\eta$. The logits returned by the model $F$
for $\textbf{x}^{\prime}$ is given by $Z(\textbf{x}^{\prime})$. We use
Integrated Gradient (IG) [60] as our attribution method that provides
attribution vector $IG^{F}(\textbf{x})$ and $IG^{F}(\textbf{x}^{\prime})$ for
the original and noisy sample.

Effect of noise on model prediction: DNNs are susceptible to imperceptible
adversarial perturbations in an input sample that can change the predicted
label of the sample [22, 12]. Early explanations for this vulnerability
attributed it to “blind spots” in the high-dimensional input space, which are
low-probability adversarial pockets that make an input vulnerable to
adversarial perturbations [61]. Goodfellow et al. [22] explain this
vulnerability of neural networks in terms of the linear behavior of networks
in high-dimensional spaces. Tanay et al. [62] demonstrate that adversarial
attacks are possible because most input samples in high-dimensional space are
bound to exist near the class boundary, making them susceptible to even minor
modifications.

Furthermore, it has been shown that by carefully traversing data manifolds,
the true label of an input sample can be changed by perturbing it off the data
manifold [20]. We can do so by introducing noise (e.g., Gaussian) to an input.
Prior works [28] have modified an input with noise and studied how the model
responds to gain insights into the model’s behavior for adversarial detection.
Hu et al. [28] compute softmax scores before and after adding noise to an
image and measure the change. They empirically pick a noise parameter that
preserves the behavior of benign images on the assumption that natural images
are robust to random input corruptions compared with adversarial images.
However, such additional noise can amplify the adversarial effects and
increase the likelihood of fooling the DNN. This impact depends on the noise
parameter and nature of the dataset. For example, for lower-dimension datasets
like MNIST, both benign and adversarial images are concentrated in a low-
dimension manifold [49]. While benign images stay robust to noise, adversarial
images move off the manifold, producing significant changes to the model
prediction.

On the contrary, for higher dimensions, benign images tend to lie close to
decision boundaries, making them susceptible to small noise that can lead to
misclassification [62]. Adversarial images, on the other hand, often lie in
the space outside the data manifold. These are low-probability manifolds
created due to a lack of images in the training dataset. Hence, additional
noise to benign images can change their position relative to their original
position in the manifold, producing significant variation in model prediction.
However, because the adversarial samples already lie on the low-probability
manifold, model sensitivity to additional noise is minimal. This sensitivity
of an image to noise can be measured by comparing the change in logits
$\delta_{1}=||Z(\textbf{x}^{\prime})-Z(\textbf{x})||$.

Figure 2: 2D Visualization: Adversarial vs. Benign Classifier Features. Left:
Shared manifold for simpler images (MNIST). Right: Adversarial samples form
out-of-distribution features in complex images (CIFAR-10).

Illustrative Example: We project the feature extracted from the penultimate
layer of the trained CNN classifiers into a 2D space using the t-SNE algorithm
[66] (see Figure 2). We use 10000 representative benign images and their
adversarial counterpart with an untargeted $L_{\infty}$ FGSM attack
$\epsilon=8/255$ from the training set of each dataset. For MNIST, the benign
images and their adversarial counterparts lie in a low-dimensional manifold,
suggesting that adversarial images are crafted by traversing the data manifold
observed in the training distribution. However, for CIFAR-10, we observe that
adversarial images primarily lie outside the distribution observed in the
training data. This suggests that adversarial samples for complex images are
created by introducing out-of-distribution samples where network behavior is
not predictable based on the training data. Given that benign images lie near
the decision boundary, adding sufficient noise can push them to the out-of-
distribution region, thus resulting in a significant change in model
predictions than their adversarial counterparts.

Effect of noise on feature attribution: Feature attribution-based explanation
methods assign a score to each input feature, indicating their importance in
the model’s prediction for an instance. The distribution of such feature
attribution scores varies for adversarial and benign samples and can be
measured using statistical measures of dispersion [72]. Figure 1 shows heat
maps for benign and adversarial counterparts for the ImageNet samples using
the IG method, highlighting the contrast in attribution distribution. We
observe that positive attribution scores (red pixels) are distributed across
more input features in adversarial images than in benign images. This
observation underscores the sensitivity of gradient-based feature attribution
methods to perturbations in the input.

Relationship between IG sensitivity and model sensitivity: The feature
attribution score computed by IG for feature $i$ of input sample
$\textbf{x}\in R^{d}$ with baseline u, model $F$ is given by:

$IG_{i}^{F}(\textbf{x,u})=(x_{i}-u_{i}).\int_{\alpha=0}^{1}{\partial_{i}F(\textbf{u}+\alpha(\textbf{x}-\textbf{u}))}\partial\alpha$
(1)

For an input sample x, IG returns a vector $IG^{F}(\textbf{x},\textbf{u})\in
R^{d}$ with scores that quantify the contribution of $x_{i}$ to the model
prediction $F(\textbf{x})$. For a single layer network
$F(\textbf{x})=H(<\textbf{w},\textbf{x}>)$ where $H$ is a differentiable
scalar-valued function and $<\textbf{w},\textbf{x}>$ is the dot product
between the weight vector w and input $\textbf{x}\in R^{d}$, IG attribution
has a closed form expression [13].

For given x, u and $\alpha$, let us consider
$\textbf{v}=\textbf{u}+\alpha(\textbf{x}-\textbf{u})$. If the single-layer
network is represented as $F(\textbf{x})=H(<\textbf{w},\textbf{x}>)$ where $H$
is a differentiable scalar-valued function, $\partial_{i}F(\textbf{v})$ can be
computed as:

$\displaystyle\partial_{i}F(\textbf{v})$ $\displaystyle=\frac{\partial
F(\textbf{v})}{v_{i}}=\frac{\partial H(<\textbf{w},\textbf{v}>)}{\partial
v_{i}}=H^{\prime}(z)\frac{\partial<\textbf{w},\textbf{v}>}{\partial v_{i}}$
$\displaystyle=w_{i}H^{\prime}(z)$ (2)

Here, $H^{\prime}(z)$ is the gradient of the activation $H(z)$ where
$z=<\textbf{w},\textbf{v}>$. To compute $\frac{\partial
F(\textbf{v})}{\partial\alpha}$:

$\frac{\partial F(\textbf{v})}{\partial\alpha}=\sum_{i=1}^{d}(\frac{\partial
F(\textbf{v})}{\partial v_{i}}\frac{\partial v_{i}}{\partial\alpha})$ (3)

We can substitute value of $\frac{\partial
v_{i}}{\partial\alpha}=(x_{i}-u_{i})$ and $\partial_{i}F(\textbf{v})$ from Eq.
3 to Eq. 3.

$\displaystyle\frac{\partial F(\textbf{v})}{\partial\alpha}$
$\displaystyle=\sum_{i=1}^{d}[w_{i}H^{\prime}(z)(x_{i}-u_{i})]$
$\displaystyle=<\textbf{x}-\textbf{u},\textbf{w}>H^{\prime}(z)$ (4)

This gives:

$dF(\textbf{v})=<\textbf{x}-\textbf{u},\textbf{w}>H^{\prime}(z)\partial\alpha$
(5)

Since $<\textbf{x}-\textbf{u},\textbf{w}>$ is scalar,

$H^{\prime}(z)\partial\alpha=\frac{dF(\textbf{v})}{<\textbf{x}-\textbf{u},\textbf{w}>}$
(6)

Eq. 6 can be used to rewrite the integral in the definition of
$IG_{i}^{F}(\textbf{x})$ in Eq. 1,

$\displaystyle\int_{\alpha=0}^{1}\partial_{i}F(\textbf{v})\partial\alpha$
$\displaystyle=\int_{\alpha=0}^{1}w_{i}H^{\prime}(z)\partial
z~{}~{}~{}\textnormal{[From Eqn. \ref{eqn:partialfv}]}$
$\displaystyle=\int_{\alpha=0}^{1}w_{i}\frac{dF(\textbf{v})}{<\textbf{x}-\textbf{u},\textbf{w}>}$
$\displaystyle=\frac{w_{i}}{<\textbf{x}-\textbf{u},\textbf{w}>}\int_{\alpha=0}^{1}{dF(\textbf{v})}$
$\displaystyle=\frac{w_{i}}{<\textbf{x}-\textbf{u},\textbf{w}>}[F(\textbf{x})-F(\textbf{u})]$
(7)

Hence, we obtain the closed form for IG from its definition in Eqn. 1 as

$\displaystyle IG_{i}^{F}(\textbf{x},\textbf{u})$
$\displaystyle=[F(\textbf{x})-F(\textbf{u})]\frac{({x_{i}}-{u_{i}}){w_{i}}}{<\textbf{x}-\textbf{u},\textbf{w}>}$
$\displaystyle IG^{F}(\textbf{x},\textbf{u})$
$\displaystyle=[F(\textbf{x})-F(\textbf{u})]\frac{(\textbf{x}-\textbf{u})\odot\textbf{w}}{<\textbf{x}-\textbf{u},\textbf{w}>}$
(8)

Here, $\odot$ is the entry-wise produce of two vectors.

Eq. 3 shows that the feature attribution in IG is proportional to the
fractional contribution of a feature to the change in logit
$<\textbf{x}-\textbf{u},\textbf{w}>$. When an adversary perturbs an input
sample for changing the predicted label, the value of logits changes
accordingly. In untargeted attacks, the adversarial perturbation aims to
maximize the softmax value of a class different than the original class.
Hence, the perturbation can increase or decrease logit values of other classes
[1]. This change in logits also brings a change in feature attribution. When
an additional noise is introduced to an input sample, the change in feature
attribution follows the change in model prediction. This sensitivity of IG to
noise can be measured using Eq. 3.

$\displaystyle\delta_{2}$
$\displaystyle=||IG^{F}(\textbf{x}^{\prime},\textbf{u})-IG^{F}(\textbf{x},\textbf{u})||_{1}\approx||IG^{F}(\textbf{x}^{\prime},\textbf{x})||_{1}$
$\displaystyle\approx\Big{|}\Big{|}[F(\textbf{x}^{\prime})-F(\textbf{x})]\frac{(\textbf{x}^{\prime}-\textbf{x})\odot\textbf{w}}{<\textbf{x}^{\prime}-\textbf{x},\textbf{w}>}\Big{|}\Big{|}_{1}$
$\displaystyle\approx\Big{|}\Big{|}[F(\textbf{x}^{\prime})-F(\textbf{x})]\frac{\Delta\odot\textbf{w}}{<\Delta,\textbf{w}>}\Big{|}\Big{|}_{1}$
(9)

Assuming w to be constant for a given model, we can conclude from Eqn. 3 that
$\delta_{2}\propto||F(\textbf{x}^{\prime})-F(\textbf{x})||$. This implies that
the sensitivity of IG is tied to the overall sensitivity of the model. Based
on these observations, we posit that the sensitivity of IG could serve as a
valuable tool in identifying adversarial samples by providing an additional
layer of insight into the behavior of deep learning models.

Figure 3: PASA overview: A & B are neural network outputs (logits), C & D are
IG feature attributions.

##  4 Methodology

###  4.1 Threat model

We consider a classification task with a distribution $D$ over input samples
$\textbf{x}\in R^{n}$ with labels $y\in[K]$. A classifier is a function
$F:R^{n}\rightarrow[K]$ learned by a neural network architecture in a
supervised manner that classifies a given input sample into one of $k$
classes. An adversary can manipulate the sample at test time by adding
$L_{\infty}$ perturbation so that the new sample $\textbf{x}^{*}$ is an
adversarial sample and wrongly classified by the classifier. A detector
$f_{det}$ is a function that computes a score for the given input sample based
on our proposed approach and decides whether the sample is benign or
adversarial by comparing it against a learned threshold. The optimal threshold
for each dataset is learned during the training phase of the detector (See
Section 4.2). At test time, we assume no previous knowledge of the underlying
attack mechanism. Below, we describe the set of assumptions about an adversary
for our proposed method and its evaluation.

####  4.1.1 Adversary goal

Adversarial samples are inputs specifically designed to produce targeted or
untargeted misclassification from a targeted machine learning model. We assume
that the adversary is not concerned with a specific target label and only aims
to produce misclassification. Untargeted attacks require fewer perturbations
than targeted attacks and are more difficult to detect [12].

####  4.1.2 Adversary capabilities

Defenses to adversarial samples typically restrict the adversary’s capability
to make “small” changes to the given input. In the case of image
classification, this change is measured in $L_{p}$ norm between two inputs for
$p\in[0,1,2,\infty]$. We assume that the adversary performs $L_{\infty}$
attack with the constraint of $\epsilon$, which means that the attack cannot
modify the pixel of the input image by more than $\epsilon$ [22]. However, we
evaluate our results on different $\epsilon$ specifically,
$\epsilon\in[8/255,16/255,32/255,64/255]$.

####  4.1.3 Adversary knowledge

We evaluate our detector under white-box assumption where an adversary has
complete knowledge of the target model and its parameters and dataset used for
training. We perform two categories of white-box attacks: (a) an adversary has
access to the model so that it can create an attack to evade the
classification; (b) in addition to the target model, an adversary has
knowledge of the underlying detector and can modify their attack to evade
target model, and the detection mechanism.

###  4.2 Proposed design

Based on our insights on the sensitivity of model prediction and feature
attribution (discussed in Section 3), we propose using noise as a probing
mechanism for adversarial detection. The underlying principle is that the
characteristics of model prediction and feature attribution on the noise-
corrupted sample differ depending on whether the sample is natural or
adversarial. We add noise to a sample and measure the change in model
prediction and feature attribution. Our detector (see Figure 3) classifies an
input sample as adversarial if the change in either the model prediction or
feature exceeds a learned threshold established from benign samples.

Figure 4: Performance of PASA for MNIST against various adversarial attacks at
varying noise spread parameters.

Training: Given a black box model $F(\textbf{.})$, and an input sample x, the
model outputs logits, $Z(\textbf{x})$. Feature attribution method, Integrated
Gradient (IG), gives attribution vector $IG^{F}(\textbf{x})$. To derive a
noisy version of the input sample, we add Gaussian noise
$\eta\in\mathcal{N}(0,\sigma^{2})$, where $\sigma^{2}$ is a hyperparameter and
equals $(max(\textbf{x})-min(\textbf{x}))*spread$. The noisy sample
($\textbf{x}^{\prime}$) is obtained as $\textbf{x}+\eta$. $spread$ controls
the standard deviation of the noise and is our only hyper-parameter required
for detector design. We vary the parameter spread under different values for
each dataset and empirically select the value that gives us the best
adversarial detection performance. For example, Figure 4 shows the performance
of our detector on various noise spread parameters for the MNIST dataset with
different adversarial attacks at $\epsilon=0.15$. We can observe that the
detector has the maximum AUC at the noise spread parameter 0.005. We followed
the same procedure on updated-CIC-IDS2017, CIFAR-10, CIFAR-100, and ImageNet
and obtained the noise-spread parameter as 0.0005, 0.15, 0.15, and 0.35
respectively.

Next, we compute the logit and feature attribution of the noisy sample (x’)
and measure the change using the $L_{1}$ norm of the difference. We term these
changes as prediction sensitivity (PS), and attribution sensitivity (AS) as
expressed in Eq. 10 and Eq 11 respectively.

$\delta_{1}=||Z(\textbf{x}^{\prime})-Z(\textbf{x})||_{1}$ (10)

$\delta_{2}=||IG^{F}(\textbf{x}^{\prime},\textbf{u})-IG^{F}(\textbf{x},\textbf{u})||_{1}$
(11)

We demonstrate the different characteristics of model prediction and feature
attribution on noise-corrupted images for MNIST and CIFAR-10 in Figure 5 and
Figure 6. As explained in training, we first collect benign and adversarial
images of both datasets, add Gaussian noise (spread parameter of 0.005 for
MNIST and 0.15 for CIFAR-10), and measure prediction sensitivity and
attribution sensitivity. Figure 5 shows the histogram plots for a set of
benign and adversarial image prediction sensitivity. For MNIST, benign samples
demonstrate smaller norms compared to their adversarial counterparts,
indicating that they can be distinguished from adversarial samples with a
threshold range $[0-3]$. This behavior is true for datasets like MNIST, where
images are concentrated in low-dimensional manifolds. In contrast, for a
three-channel image dataset like CIFAR-10, we observe a divergent behavior.
The difference in model prediction for noisy and original images in benign
samples is greater than that of adversarial samples and their noisy
counterparts. This behavior is due to the distinct positions benign and
adversarial images occupy within the input space manifold, as discussed in
Section 3. We can also observe that for CIFAR-10, adversarial samples
generated with a larger perturbation parameter ($\epsilon$) exhibit minimal
changes in model prediction. This is because the adversarial images are
located far from the decision boundary, and the added noise has minimal
impact. ImageNet and CIFAR-100 demonstrate similar behavior.

Figure 5: The distribution of the difference between logits of benign and
adversarial images with their noisy counterparts (left: MNIST, right:
CIFAR-10). Adversarial samples are obtained at various perturbation strengths
$\epsilon$.

Figure 6: The distribution of the difference between the attribution vector of
benign and adversarial images with their noisy counterparts (left: MNIST,
right: CIFAR-10). Adversarial samples are obtained at various perturbation
strengths $\epsilon$.

Figure 6 shows the histogram plots for a set of benign and adversarial images
of the MNIST and CIFAR-10 datasets for attribution sensitivity. We observed
contrasting model prediction sensitivity between MNIST and CIFAR-10 in Figure
5. Since feature attribution of an image relies on the model prediction as
demonstrated by Eq. 3, the feature attribution sensitivity distribution
follows the model prediction behavior. While for MNIST, the benign and its
noisy counterparts have a smaller $L_{1}$ norm, the opposite is true for
CIFAR-10. ImageNet and CIFAR-100 demonstrate similar behavior.

Training PASA thus involves learning the threshold for the prediction
sensitivity and attribution sensitivity for benign samples. For each dataset,
we collect 5000 benign samples from the training set, probe them with noise,
measure the prediction sensitivity and attribution sensitivity metrics, and
learn the threshold that yields various false positive rates (FPRs) on a
validation set of benign samples. We provide the methodology of our approach
below:
Methodology:
Step 1. Set noise spread parameter $\sigma$ as 0.001 for MNIST, 0.0001 for
CIC-IDS2017, 0.1 for CIFAR, and ImageNet.
Step 2. For a set of benign samples, produce its noisy version by adding
Gaussian noise. Compute two metrics, PS and AS (See Eqns 10 & 11).
Step 3. Find thresholds of PS and AS that produce 1%, 5%, and 10% False
Positive Rate (FPR) on a hold-out set of benign samples.
Step 4. Evaluate the detection results on a validation set (consisting of
benign and adversarial samples) using the threshold and noise parameter
learned from Step 3.
Step 5. Increment the noise to $\sigma^{\prime}=\sigma+\delta$, where $\delta$
is dataset-dependent. The following delta levels worked best in our
experiment: 0.01 for MNIST, 0.0001 for CIC-IDS2017, and 0.1 for CIFAR and
ImageNet.
Step 6. Repeat Steps 2-5.
Step 7. Pick the best-performing noise spread parameter and threshold.

Testing: At test time, we evaluate changes in model prediction and feature
attribution of an input sample. We add Gaussian noise with zero mean and
standard deviation of $(max(\textbf{x})-min(\textbf{x}))*spread$, where we
select spread empirically during training. We compute prediction sensitivity
and attribution sensitivity using expressions of Eq. 10 and Eq. 11. We reject
a sample as adversarial if either of the computed metrics does not satisfy the
threshold learned during training.

TABLE I: Adversarial Detection Performance for MNIST and CIFAR-10 models: Our
Method (PASA) vs. Unsupervised Methods (FS, MagNet, U-LOO, TWS) using AUC
scores.

Attack |  | MNIST | CIFAR-10 (VGG) | CIFAR-10 (ResNet)
---|---|---|---|---
| Strength | FS | MagNet | U-LOO | TWS | PASA | FS | MagNet | U-LOO | TWS | PASA | FS | MagNet | U-LOO | TWS | PASA
FGSM | 8/255 | 0.89$\pm$0.01 | 0.94$\pm$0.01 | 0.93$\pm$0.01 | 0.94$\pm$0.01 | 0.97$\pm$0.01 | 0.57$\pm$0.01 | 0.62$\pm$0.01 | 0.52$\pm$0.01 | 0.51$\pm$0.01 | 0.63$\pm$0.02 | 0.76$\pm$0.01 | 0.63$\pm$0.01 | 0.55$\pm$0.01 | 0.72$\pm$0.01 | 0.87$\pm$0.01
| 16/255 | 0.87$\pm$0.01 | 0.95$\pm$0.01 | 0.92$\pm$0.01 | 0.93$\pm$0.02 | 0.98$\pm$0.01 | 0.68$\pm$0.02 | 0.82$\pm$0.01 | 0.52$\pm$0.02 | 0.66$\pm$0.02 | 0.77$\pm$0.02 | 0.81$\pm$0.01 | 0.83$\pm$0.01 | 0.53$\pm$0.03 | 0.78$\pm$0.02 | 0.97$\pm$0.01
| 32/255 | 0.86$\pm$0.01 | 0.95$\pm$0.01 | 0.91$\pm$0.02 | 0.89$\pm$0.04 | 0.98$\pm$0.01 | 0.66$\pm$0.01 | 0.96$\pm$0.04 | 0.52$\pm$0.01 | 0.64$\pm$0.01 | 0.88$\pm$0.01 | 0.76$\pm$0.01 | 0.94$\pm$0.01 | 0.60$\pm$0.02 | 0.76$\pm$0.01 | 0.98$\pm$0.01
| 64/255 | 0.86$\pm$0.01 | 0.95$\pm$0.01 | 0.88$\pm$0.02 | 0.82$\pm$0.10 | 0.98$\pm$0.02 | 0.63$\pm$0.02 | 0.95$\pm$0.05 | 0.49$\pm$0.03 | 0.61$\pm$0.01 | 0.95$\pm$0.01 | 0.84$\pm$0.02 | 0.95$\pm$0.03 | 0.53$\pm$0.03 | 0.83$\pm$0.01 | 0.98$\pm$0.02
PGD | 8/255 | 0.90$\pm$0.01 | 0.95$\pm$0.03 | 0.99$\pm$0.01 | 0.92$\pm$0.11 | 0.98$\pm$0.01 | 0.52$\pm$0.01 | 0.59$\pm$0.03 | 0.49$\pm$0.05 | 0.58$\pm$0.02 | 0.74$\pm$0.02 | 0.25$\pm$0.01 | 0.57$\pm$0.04 | 0.62$\pm$0.01 | 0.14$\pm$0.01 | 0.83$\pm$0.03
| 16/255 | 0.88$\pm$0.01 | 0.95$\pm$0.04 | 0.99$\pm$0.01 | 0.76$\pm$0.10 | 0.98$\pm$0.01 | 0.59$\pm$0.01 | 0.75$\pm$0.02 | 0.51$\pm$0.03 | 0.56$\pm$0.01 | 0.82$\pm$0.01 | 0.16$\pm$0.01 | 0.73$\pm$0.03 | 0.65$\pm$0.01 | 0.14$\pm$0.02 | 0.93$\pm$0.02
| 32/255 | 0.77$\pm$0.02 | 0.94$\pm$0.04 | 0.99$\pm$0.02 | 0.32$\pm$0.03 | 0.99$\pm$0.01 | 0.62$\pm$0.02 | 0.93$\pm$0.03 | 0.51$\pm$0.01 | 0.53$\pm$0.01 | 0.90$\pm$0.02 | 0.13$\pm$0.01 | 0.91$\pm$0.05 | 0.68$\pm$0.02 | 0.14$\pm$0.03 | 0.98$\pm$0.01
| 64/255 | 0.48$\pm$0.01 | 0.95$\pm$0.03 | 0.99$\pm$0.03 | 0.11$\pm$0.01 | 0.99$\pm$0.02 | 0.60$\pm$0.01 | 0.95$\pm$0.05 | 0.52$\pm$0.03 | 0.51$\pm$0.04 | 0.95$\pm$0.01 | 0.16$\pm$0.01 | 0.95$\pm$0.06 | 0.72$\pm$0.02 | 0.14$\pm$0.02 | 0.98$\pm$0.02
BIM | 8/255 | 0.89$\pm$0.03 | 0.83$\pm$0.01 | 0.40$\pm$0.04 | 0.83$\pm$0.02 | 0.62$\pm$0.04 | 0.37$\pm$0.02 | 0.54$\pm$0.02 | 0.50$\pm$0.01 | 0.55$\pm$0.05 | 0.66$\pm$0.02 | 0.29$\pm$0.02 | 0.53$\pm$0.03 | 0.60$\pm$0.02 | 0.16$\pm$0.01 | 0.75$\pm$0.01
| 16/255 | 0.88$\pm$0.01 | 0.93$\pm$0.01 | 0.67$\pm$0.04 | 0.92$\pm$0.01 | 0.58$\pm$0.03 | 0.16$\pm$0.01 | 0.60$\pm$0.04 | 0.51$\pm$0.01 | 0.55$\pm$0.06 | 0.71$\pm$0.01 | 0.16$\pm$0.02 | 0.58$\pm$0.03 | 0.59$\pm$0.01 | 0.15$\pm$0.01 | 0.84$\pm$0.01
| 32/255 | 0.88$\pm$0.01 | 0.95$\pm$0.01 | 0.92$\pm$0.04 | 0.85$\pm$0.01 | 0.56$\pm$0.02 | 0.15$\pm$0.02 | 0.73$\pm$0.06 | 0.53$\pm$0.02 | 0.55$\pm$0.04 | 0.73$\pm$0.02 | 0.13$\pm$0.01 | 0.72$\pm$0.04 | 0.58$\pm$0.01 | 0.14$\pm$0.02 | 0.93$\pm$0.01
| 64/255 | 0.88$\pm$0.01 | 0.95$\pm$0.02 | 0.99$\pm$0.02 | 0.69$\pm$0.02 | 0.55$\pm$0.01 | 0.13$\pm$0.01 | 0.91$\pm$0.01 | 0.52$\pm$0.02 | 0.54$\pm$0.01 | 0.74$\pm$0.01 | 0.12$\pm$0.01 | 0.90$\pm$0.04 | 0.57$\pm$0.01 | 0.14$\pm$0.01 | 0.97$\pm$0.02
Auto-PGD | 0.15 | 0.81$\pm$0.02 | 0.95$\pm$0.01 | 0.98$\pm$0.03 | 0.56$\pm$0.03 | 0.98$\pm$0.01 | 0.14$\pm$0.02 | 0.96$\pm$0.01 | 0.52$\pm$0.04 | 0.12$\pm$0.02 | 0.97$\pm$0.01 | 0.13$\pm$0.01 | 0.95$\pm$0.01 | 0.75$\pm$0.01 | 0.13$\pm$0.01 | 0.98$\pm$0.02
CW | 0.15 | 0.88$\pm$0.03 | 0.94$\pm$0.02 | 0.91$\pm$0.02 | 0.95$\pm$0.02 | 0.58$\pm$0.02 | 0.66$\pm$0.01 | 0.71$\pm$0.03 | 0.54$\pm$0.04 | 0.68$\pm$0.01 | 0.82$\pm$0.01 | 0.84$\pm$0.01 | 0.93$\pm$0.01 | 0.55$\pm$0.03 | 0.82$\pm$0.01 | 0.98$\pm$0.01
Average |  | 0.84$\pm$0.10 | 0.94$\pm$0.03 | 0.90$\pm$0.16 | 0.75$\pm$0.25 | 0.83$\pm$0.20 | 0.46$\pm$0.21 | 0.79$\pm$0.15 | 0.51$\pm$0.01 | 0.54$\pm$0.13 | 0.80$\pm$0.11 | 0.40$\pm$0.31 | 0.79$\pm$0.16 | 0.61$\pm$0.08 | 0.37$\pm$0.31 | 0.93$\pm$0.07

##  5 Experiment and Evaluation

###  5.1 Experiment Setup

We implemented PASA using Python and PyTorch and conducted experiments on a
server with a 4 Intel(R) Core(TM) i5-7600K CPU @ 3.80 GHz and a 12 GB NVIDIA
TITAN Xp GPU card. We used Captum [30] to generate explanations.

####  5.1.1 Datasets

We evaluate the performance of PASA on the following datasets: MNIST [34],
CIFAR-10 [31], CIFAR-100 [31], ImageNet [17] and updated CIC-IDS2017 [18]. The
datasets are publicly available, and none of them contain personally
identifiable information. Details on the dataset can be found in Appendix A.

####  5.1.2 Target networks

To demonstrate the generalization of our approach, we evaluate our results by
performing adversarial attacks and detection on a variety of networks: MLP
[47], LeNet [34], VGG-16 [54], ResNet [25], and MobileNet [48]. Details on
model architecture can be found in Appendix B.

####  5.1.3 Attacks

We evaluate the performance of PASA against inputs perturbed using the
following untargeted $L_{\infty}$ attacks: FGSM [22], BIM [33] (10 iterations)
and PGD [40] (step-size $\alpha=\epsilon/10$, 40 iterations) with increasing
value of attack parameter $\epsilon\in[8/255,16/255,32/255,64/255]$, Auto-PGD
[14] ($\epsilon=0.15$) and zero confidence CW attack [12] ($\epsilon=0.15$,
learning rate= 0.01). Adversarial attacks are performed on the test set which
is not used for learning the threshold of PASA. Further details are provided
in Appendix C.

###  5.2 Evaluation

####  5.2.1 Baselines

We present the experimental evaluation of PASA by comparing its results
against four types of unsupervised detectors that use different statistical
approaches for adversarial detection. We discuss their implementation in
Appendix D.

Feature squeezing (FS) [71]: FS is a filter-based approach that applies
filters to a given image and measures the distance between prediction vectors
of the two images. If the distance for any compressed image exceeds a certain
threshold learned from benign images, the unaltered image is considered
adversarial.

Magnet [41]: MagNet is a reconstruction-based detector that trains denoisers
on clean training data to reconstruct input samples. If the reconstruction
error score exceeds a threshold learned from benign images, the detector flags
an input sample as adversarial.

Turning a weakness into a strength (TWS) [28]: TWS is a noise-based approach
that identifies a given input image as adversarial if after perturbing the
input with noise does not result in a significant change in softmax score. The
defense also has a second evaluation criterion, which checks the number of
steps required to cross the decision boundary to a random target class. The
second test assumes white-box access to the model and detector and requires
modification of the adversarial attack. Hence, we only use the first criteria
as the detection mechanism.

TABLE II: Adversarial Detection Performance for MNIST and CIFAR-10 models: Our
Method (PASA) vs. Unsupervised Methods (FS, MagNet, U-LOO, TWS) using TPR
scores.

| Performance | MNIST | CIFAR-10 (VGG) | CIFAR-10 (ResNet)
---|---|---|---|---
Attack | Metric | FS | MagNet | U-LOO | TWS | PASA | FS | MagNet | U-LOO | TWS | PASA | FS | MagNet | U-LOO | TWS | PASA
FGSM (8/255) | TPR (FPR @ 0.01) | 85.9 | 74.8 | 40.36 | 6.4 | 90.3 | 11.8 | 5.5 | 3.5 | 2.6 | 18.8 | 29.4 | 4.6 | 1 | 12.5 | 22.6
| TPR (FPR @ 0.05) | 96.5 | 95.3 | 91.9 | 64 | 97 | 21.2 | 5.6 | 9.9 | 7.5 | 23.3 | 33.1 | 16.8 | 5.3 | 15.4 | 22.9
| TPR (FPR @ 0.1) | 96.8 | 98.3 | 92.3 | 86.8 | 99.8 | 26.2 | 19.6 | 16.8 | 11.7 | 31.7 | 37.4 | 37.2 | 8.3 | 17.8 | 56.8
FGSM (16/255) | TPR (FPR @ 0.01) | 86.3 | 95.3 | 51.5 | 3.5 | 96.1 | 13.8 | 9.9 | 3.4 | 2.8 | 17 | 31.7 | 10.9 | 0.7 | 9.5 | 64.9
| TPR (FPR @ 0.05) | 83 | 96.7 | 81.2 | 59.4 | 98.7 | 19.4 | 10.6 | 9.2 | 8.8 | 32.2 | 36 | 11 | 4.8 | 14.8 | 97.1
| TPR (FPR @ 0.1) | 85.4 | 99.4 | 91.3 | 93.1 | 99.9 | 27.7 | 42.6 | 15.7 | 14.2 | 32.3 | 42.7 | 43.4 | 7.6 | 18.5 | 100
FGSM (32/255) | TPR (FPR @ 0.01) | 68.8 | 94.9 | 60.9 | 6.3 | 98.4 | 10.9 | 98.2 | 2 | 1.3 | 16.2 | 10.7 | 98.8 | 1.7 | 0.7 | 100
| TPR (FPR @ 0.05) | 69.2 | 96.8 | 90.9 | 47.5 | 99.6 | 15.2 | 98.6 | 8 | 3.8 | 30.2 | 12.9 | 98.8 | 5.7 | 12 | 100
| TPR (FPR @ 0.1) | 81.3 | 99.1 | 90.11 | 88 | 99.7 | 23.5 | 98.8 | 14.6 | 7.6 | 51.8 | 18.4 | 99.1 | 9.3 | 21 | 100
FGSM (64/255) | TPR (FPR @ 0.01) | 84.4 | 99.9 | 78.8 | 5.4 | 99.7 | 5.8 | 92.6 | 1.2 | 0.5 | 36.5 | 16.8 | 100 | 0.2 | 0.1 | 100
| TPR (FPR @ 0.05) | 87.9 | 99.9 | 88.9 | 47.3 | 99.9 | 8.4 | 93.6 | 4.9 | 0.8 | 76.7 | 24.3 | 100 | 0.2 | 20 | 100
| TPR (FPR @ 0.1) | 91.8 | 99.9 | 90.5 | 83.8 | 100 | 13.4 | 94.6 | 9.8 | 1.6 | 91.8 | 32.9 | 100 | 4.2 | 78.8 | 100
PGD (8/255) | TPR (FPR @ 0.01) | 78.9 | 100 | 100 | 13.9 | 100 | 5.8 | 4.5 | 1.2 | 3 | 53.7 | 1.7 | 4.3 | 2.8 | 0 | 12.8
| TPR (FPR @ 0.05) | 79.9 | 100 | 100 | 49.6 | 100 | 6.4 | 4.8 | 4.9 | 6 | 55 | 1.8 | 4.3 | 10.8 | 0 | 25.2
| TPR (FPR @ 0.1) | 99.3 | 100 | 100 | 90.2 | 100 | 7.5 | 17.4 | 12.5 | 8 | 55.3 | 3.1 | 15 | 14.9 | 0 | 52
PGD (16/255) | TPR (FPR @ 0.01) | 72.5 | 100 | 100 | 1.6 | 100 | 4 | 6.6 | 2 | 5 | 66.5 | 0.6 | 6.1 | 3.1 | 0 | 40.9
| TPR (FPR @ 0.05) | 77.8 | 100 | 100 | 13.6 | 100 | 4.2 | 7 | 6.2 | 9 | 67.8 | 1.3 | 6.2 | 11.1 | 0 | 61
| TPR (FPR @ 0.1) | 85.6 | 100 | 100 | 54.9 | 100 | 4.3 | 24.1 | 11.9 | 12 | 68.2 | 2 | 22.9 | 16.1 | 0 | 86
PGD (32/255) | TPR (FPR @ 0.01) | 44.3 | 100 | 100 | 1.3 | 100 | 8.6 | 34.8 | 2.5 | 0.5 | 72.7 | 0.2 | 30.5 | 5.8 | 0 | 70.6
| TPR (FPR @ 0.05) | 48.9 | 100 | 100 | 5.8 | 100 | 10.1 | 37.5 | 7.4 | 3.5 | 77.1 | 0.2 | 31.6 | 16.4 | 0 | 95.2
| TPR (FPR @ 0.1) | 53.6 | 100 | 100 | 9 | 100 | 12.8 | 100 | 14.3 | 5 | 78.2 | 0.3 | 100 | 22.7 | 0 | 99.8
PGD (64/255) | TPR (FPR @ 0.01) | 12.6 | 100 | 100 | 0.1 | 100 | 8 | 100 | 1.6 | 5 | 95.4 | 0.1 | 100 | 7.2 | 0 | 100
| TPR (FPR @ 0.05) | 17.9 | 100 | 100 | 0.9 | 100 | 8.9 | 100 | 4.4 | 6 | 96.2 | 0.1 | 100 | 17.9 | 0 | 100
| TPR (FPR @ 0.1) | 24.6 | 100 | 100 | 1.5 | 100 | 11.3 | 100 | 12 | 8 | 96.6 | 0.8 | 100 | 23.3 | 0 | 100
BIM (8/255) | TPR (FPR @ 0.01) | 72 | 38.9 | 4.4 | 4 | 3 | 7.8 | 5.1 | 2.1 | 2.1 | 34.8 | 2.8 | 6 | 1.9 | 0 | 3.2
| TPR (FPR @ 0.05) | 82.5 | 45.6 | 6.9 | 52 | 8.2 | 11.5 | 5.4 | 6.9 | 5.1 | 37.1 | 3.2 | 16.8 | 7.6 | 0 | 10.9
| TPR (FPR @ 0.1) | 92.9 | 56.7 | 12.5 | 96.4 | 16.1 | 18.7 | 15.9 | 13.5 | 7.4 | 38 | 3.6 | 28.9 | 12.3 | 0 | 37.7
BIM (16/255) | TPR (FPR @ 0.01) | 76.6 | 65.5 | 9.8 | 2.8 | 2.7 | 7.8 | 4.6 | 1.2 | 3.2 | 49.4 | 0.9 | 5.9 | 2.3 | 0 | 11.5
| TPR (FPR @ 0.05) | 86.8 | 88.5 | 37.3 | 39.5 | 7.6 | 13 | 4.7 | 6.5 | 6.5 | 50.2 | 0.9 | 16.1 | 7.6 | 0 | 26.5
| TPR (FPR @ 0.1) | 97 | 91.1 | 50.5 | 91.5 | 14 | 16.1 | 16.2 | 13.2 | 11.2 | 50.8 | 1.1 | 32.2 | 11.8 | 0 | 54.3
BIM (32/255) | TPR (FPR @ 0.01) | 75.4 | 75.5 | 26.3 | 0.8 | 3.7 | 9.1 | 5.4 | 2.1 | 2.8 | 54.3 | 0.4 | 6.4 | 1.2 | 0 | 23.6
| TPR (FPR @ 0.05) | 85.7 | 99.4 | 65 | 28.2 | 8.2 | 12.5 | 5.6 | 7.4 | 7.4 | 55.7 | 0.4 | 22.6 | 6.6 | 0 | 51.5
| TPR (FPR @ 0.1) | 96.1 | 99.5 | 75.1 | 74.6 | 14 | 18.7 | 24.5 | 14.2 | 12 | 56.6 | 0.6 | 49.1 | 10.3 | 0 | 83.3
BIM (64/255) | TPR (FPR @ 0.01) | 63.9 | 100 | 53.6 | 0.3 | 4 | 15.1 | 12.6 | 1.9 | 3 | 37.2 | 0.4 | 12.8 | 1.2 | 0 | 958.2
| TPR (FPR @ 0.05) | 74.3 | 100 | 86.8 | 9.3 | 8.5 | 19.4 | 14.5 | 8.2 | 8.1 | 40.3 | 0.6 | 88 | 6.1 | 0 | 94.4
| TPR (FPR @ 0.1) | 84.4 | 100 | 92.3 | 44.1 | 14.3 | 16.6 | 86.4 | 14.4 | 11.2 | 42.1 | 0.6 | 100 | 9.2 | 0 | 99.9
Auto-PGD (0.15) | TPR (FPR @ 0.01) | 90.7 | 100 | 83 | 0.15 | 99.2 | 0 | 82.5 | 2 | 0 | 98.2 | 0.4 | 51 | 8.7 | 0 | 85.4
| TPR (FPR @ 0.05) | 91 | 100 | 96 | 3.4 | 99.3 | 0 | 83.6 | 7.8 | 0 | 98.8 | 0.4 | 51.8 | 23.4 | 0 | 98.7
| TPR (FPR @ 0.1) | 91.6 | 100 | 97 | 17.3 | 99.6 | 0 | 84.8 | 15.3 | 0 | 98.8 | 1.2 | 100 | 29.7 | 0 | 99.8
CW (0.15) | TPR (FPR @ 0.01) | 85.3 | 86.3 | 29.1 | 3.9 | 2.1 | 14.6 | 26.4 | 3.5 | 2.8 | 6.8 | 38.7 | 57.2 | 2.3 | 7.3 | 82.2
| TPR (FPR @ 0.05) | 87.9 | 96.6 | 70.4 | 62.8 | 3.4 | 21.5 | 28.6 | 9.3 | 7.4 | 20.2 | 45.4 | 57.5 | 7.3 | 13 | 97.8
| TPR (FPR @ 0.1) | 93.1 | 96.8 | 80.6 | 98.1 | 13.8 | 29.2 | 90.4 | 16.8 | 13.3 | 42.1 | 52.2 | 87 | 9.8 | 18.9 | 99.7

ML-LOO [72]: ML-LOO is a feature-attribution-based defense that detects
adversarial examples using statistical measures of attribution vector. The
authors compute the inter-quartile range (IQR) of feature attribution of
benign and adversarial images for distinguishing benign images from
adversarial counterparts. While the paper also proposes a supervised detector
by extracting statistics from multiple hidden layers, we implement
unsupervised detection, U-LOO, for a fair comparison. The authors evaluate
their results using LOO [35] and IG [60]. We stick to IG since our detection
also uses the same method.

####  5.2.2 Performance evaluation

For each dataset, we randomly sample 1000 benign samples from the test set,
which are correctly classified by the model, and generate 1000 adversarial
samples for each type of attack for evaluation. This is repeated 10 times to
account for randomness associated with the sampling. During test time, we
assume no previous knowledge of the attack mechanism. Given an input sample,
PASA only computes two noise-probed metrics, prediction sensitivity, and
attribution sensitivity, and compares them with the threshold learned during
training. If either of the metrics satisfies the threshold, the sample is
classified as benign, else adversarial.

##### Metrics

We assess the detector performance using in the following criteria: a) True
Positive Rate (TPR): TPR is computed as the ratio of the total number of
correctly identified adversarial samples to the overall number of adversarial
samples. In unsupervised detectors, the decision threshold is learned from
benign samples while maintaining a fixed false positive rate (FPR) on the
validation set. We then use this threshold on the test set and compute the
TPR. We report the TPR of detectors using thresholds calculated for 1%, 5%,
and 10% FPR. b) Area Under the Receiver Operating Characteristic Curve (AUC):
AUC is a threshold-independent measure of a detector performance which is
widely used as a standard in comparison between different methods [16].

TABLE III: Adversarial Detection Performance for CIFAR-100 and ImageNet
models: Our Method (PASA) vs. Unsupervised Methods (FS, MagNet, U-LOO, TWS)
using AUC scores.

Attack |  | CIFAR-100 | ImageNet (MobileNet) | ImageNet (ResNet)
---|---|---|---|---
| Strength | FS | MagNet | U-LOO | TWS | PASA | FS | MagNet | U-LOO | TWS | PASA | FS | MagNet | U-LOO | TWS | PASA
FGSM | 8/255 | 0.68$\pm$0.02 | 0.60$\pm$0.02 | 0.62$\pm$0.03 | 0.34$\pm$0.02 | 0.81$\pm$0.01 | 0.60$\pm$0.01 | 0.51$\pm$0.03 | 0.60$\pm$0.01 | 0.50$\pm$0.02 | 0.81$\pm$0.01 | 0.65$\pm$0.02 | 0.50$\pm$0.01 | 0.62$\pm$0.01 | 0.64$\pm$0.01 | 0.65$\pm$0.01
| 16/255 | 0.62$\pm$0.04 | 0.78$\pm$0.03 | 0.67$\pm$0.02 | 0.31$\pm$0.01 | 0.96$\pm$0.01 | 0.62$\pm$0.01 | 0.51$\pm$0.03 | 0.68$\pm$0.01 | 0.50$\pm$0.03 | 0.91$\pm$0.02 | 0.69$\pm$0.01 | 0.52$\pm$0.01 | 0.62$\pm$0.01 | 0.57$\pm$0.01 | 0.75$\pm$0.01
| 32/255 | 0.64$\pm$0.03 | 0.95$\pm$0.02 | 0.67$\pm$0.02 | 0.29$\pm$0.01 | 0.97$\pm$0.02 | 0.65$\pm$0.01 | 0.54$\pm$0.02 | 0.69$\pm$0.01 | 0.50$\pm$0.04 | 0.96$\pm$0.01 | 0.75$\pm$0.02 | 0.55$\pm$0.01 | 0.60$\pm$0.01 | 0.52$\pm$0.01 | 0.86$\pm$0.01
| 64/255 | 0.68$\pm$0.02 | 0.96$\pm$0.02 | 0.67$\pm$0.03 | 0.24$\pm$0.02 | 0.97$\pm$0.02 | 0.68$\pm$0.01 | 0.63$\pm$0.03 | 0.68$\pm$0.01 | 0.49$\pm$0.03 | 0.98$\pm$0.01 | 0.81$\pm$0.01 | 0.65$\pm$0.01 | 0.60$\pm$0.01 | 0.45$\pm$0.01 | 0.95$\pm$0.01
PGD | 8/255 | 0.67$\pm$0.03 | 0.56$\pm$0.03 | 0.63$\pm$0.03 | 0.61$\pm$0.01 | 0.60$\pm$0.02 | 0.25$\pm$0.02 | 0.51$\pm$0.01 | 0.59$\pm$0.01 | 0.52$\pm$0.01 | 0.98$\pm$0.02 | 0.29$\pm$0.01 | 0.51$\pm$0.01 | 0.59$\pm$0.01 | 0.57$\pm$0.02 | 0.97$\pm$0.01
| 16/255 | 0.62$\pm$0.02 | 0.66$\pm$0.04 | 0.68$\pm$0.03 | 0.59$\pm$0.01 | 0.68$\pm$0.02 | 0.19$\pm$0.03 | 0.50$\pm$0.01 | 0.59$\pm$0.01 | 0.51$\pm$0.02 | 0.99$\pm$0.02 | 0.18$\pm$0.01 | 0.52$\pm$0.02 | 0.63$\pm$0.02 | 0.27$\pm$0.02 | 0.98$\pm$0.01
| 32/255 | 0.74$\pm$0.03 | 0.85$\pm$0.05 | 0.72$\pm$0.03 | 0.48$\pm$0.02 | 0.86$\pm$0.04 | 0.16$\pm$0.02 | 0.52$\pm$0.01 | 0.57$\pm$0.01 | 0.51$\pm$0.01 | 0.98$\pm$0.02 | 0.11$\pm$0.02 | 0.52$\pm$0.01 | 0.75$\pm$0.02 | 0.05$\pm$0.00 | 0.97$\pm$0.01
| 64/255 | 0.69$\pm$0.02 | 0.95$\pm$0.03 | 0.73$\pm$0.03 | 0.08$\pm$0.01 | 0.95$\pm$0.02 | 0.17$\pm$0.02 | 0.57$\pm$0.01 | 0.59$\pm$0.01 | 0.50$\pm$0.01 | 0.98$\pm$0.02 | 0.11$\pm$0.02 | 0.57$\pm$0.01 | 0.74$\pm$0.03 | 0.02$\pm$0.00 | 0.98$\pm$0.01
BIM | 8/255 | 0.55$\pm$0.02 | 0.51$\pm$0.01 | 0.57$\pm$0.02 | 0.59$\pm$0.02 | 0.60$\pm$0.01 | 0.42$\pm$0.03 | 0.02$\pm$0.00 | 0.19$\pm$0.01 | 0.46$\pm$0.01 | 0.51$\pm$0.02 | 0.50$\pm$0.03 | 0.04$\pm$0.00 | 0.15$\pm$0.01 | 0.81$\pm$0.01 | 0.37$\pm$0.01
| 16/255 | 0.50$\pm$0.01 | 0.55$\pm$0.02 | 0.58$\pm$0.02 | 0.56$\pm$0.04 | 0.61$\pm$0.03 | 0.32$\pm$0.02 | 0.03$\pm$0.00 | 0.15$\pm$0.02 | 0.51$\pm$0.03 | 0.63$\pm$0.01 | 0.31$\pm$0.02 | 0.04$\pm$0.00 | 0.17$\pm$0.01 | 0.71$\pm$0.01 | 0.47$\pm$0.01
| 32/255 | 0.57$\pm$0.01 | 0.65$\pm$0.03 | 0.62$\pm$0.03 | 0.42$\pm$0.02 | 0.70$\pm$0.02 | 0.25$\pm$0.01 | 0.02$\pm$0.00 | 0.16$\pm$0.01 | 0.51$\pm$0.02 | 0.77$\pm$0.01 | 0.20$\pm$0.02 | 0.04$\pm$0.00 | 0.17$\pm$0.01 | 0.58$\pm$0.02 | 0.62$\pm$0.01
| 64/255 | 0.54$\pm$0.01 | 0.83$\pm$0.02 | 0.61$\pm$0.02 | 0.33$\pm$0.03 | 0.84$\pm$0.03 | 0.21$\pm$0.01 | 0.02$\pm$0.00 | 0.18$\pm$0.01 | 0.50$\pm$0.02 | 0.83$\pm$0.03 | 0.15$\pm$0.02 | 0.04$\pm$0.00 | 0.18$\pm$0.02 | 0.29$\pm$0.03 | 0.62$\pm$0.01
Auto-PGD | 0.15 | 0.32$\pm$0.01 | 0.91$\pm$0.02 | 0.64$\pm$0.04 | 0.30$\pm$0.01 | 0.98$\pm$0.02 | 0.19$\pm$0.04 | 0.37$\pm$0.01 | 0.59$\pm$0.01 | 0.17$\pm$0.02 | 0.97$\pm$0.02 | 0.14$\pm$0.01 | 0.38$\pm$0.01 | 0.66$\pm$0.01 | 0.10$\pm$0.00 | 0.96$\pm$0.01
CW | 0.15 | 0.58$\pm$0.02 | 0.91$\pm$0.01 | 0.68$\pm$0.03 | 0.22$\pm$0.01 | 0.92$\pm$0.03 | 0.58$\pm$0.01 | 0.03$\pm$0.00 | 0.11$\pm$0.02 | 0.51$\pm$0.01 | 0.87$\pm$0.01 | 0.58$\pm$0.01 | 0.04$\pm$0.00 | 0.15$\pm$0.01 | 0.68$\pm$0.02 | 0.95$\pm$0.01
Average |  | 0.60$\pm$0.10 | 0.77$\pm$0.16 | 0.65$\pm$0.05 | 0.38$\pm$0.16 | 0.81$\pm$0.15 | 0.38$\pm$0.20 | 0.34$\pm$0.24 | 0.45$\pm$0.22 | 0.48$\pm$0.09 | 0.87$\pm$0.15 | 0.39$\pm$0.25 | 0.35$\pm$0.24 | 0.47$\pm$0.23 | 0.45$\pm$0.25 | 0.79$\pm$0.20

##  6 Results and Analysis

We first discuss the results of adversarial detection on image classifiers. We
discuss the performance of detectors on the security dataset in Section 7.

###  6.1 Adversarial Detection Performance

##### CIFAR-10

PASA outperforms all baseline methods in CIFAR-10 (ResNet) model. For example,
as observed in Table I, PASA obtains an AUC of 0.98$\pm$0.01 for detecting CW
attack on CIFAR-10 ResNet. The next best detector is MagNet with 0.93$\pm$0.01
AUC. On CIFAR-10 (VGG) model, PASA obtains an AUC of 0.82$\pm$0.01 for
detecting CW attack. MagNet is the the next best detector with 0.71$\pm$0.03
AUC. Other methods (e.g., FS, and TWS) show more variability in CIFAR-10
models, with lower AUC scores (less than 0.5) in some instances, suggesting
that the thresholds learned from benign images were suitable for detecting
specific attacks only. Thus such solutions become impractical for attack-
agnostic detection since they require a threshold change depending on the type
of attack.

We also observe that the MagNet performance remains competitive on both
CIFAR-10 models, however, the performance of other detection methods degrades
significantly. For example, on average, the U-LOO method obtains an AUC of
0.90$\pm$0.16 on MNIST, whereas the average AUC reduces to 0.51$\pm$0.01 on
the CIFAR-10 VGG model and 0.61$\pm$0.08 on the ResNet model. A similar
performance drop can be observed with TWS and FS. In Table II, we notice that
PASA obtains high TPRs at low FPRs consistently. For example, PASA obtains a
TPR of 82.2% at 1% FPR on detecting CW attack for the CIFAR-10 (ResNet) model.
The next best detector is MagNet with 57.2% TPR. While MagNet seems to obtain
high TPRs, especially on the VGG16 model, it comes at the cost of high FPRs,
discussed in Section 6.2.

##### CIFAR-100

As observed in Table III, PASA consistently outperforms the baseline methods
on CIFAR-100 with noticeable performance improvement as the strength of
adversarial perturbation increases. While the performance of detectors like
TWS decreases with an increase in adversarial perturbation, PASA achieves an
increment in its detection performance. This is because as perturbation
increases, the discrepancy between the attribution maps of benign and
adversarial images increases, which helps PASA detect the inconsistency. For
instance, PASA obtains an AUC of 0.92$\pm$0.03 on detecting CW attacks. The
next best detector is MagNet, with 0.91$\pm$0.01 AUC. TWS, also a noise-based
approach, obtains an AUC of 0.59$\pm$0.02 on detecting $\epsilon=8/255$ BIM
attack. The AUC reduces to 0.33$\pm$0.03 at $\epsilon=64/255$. PASA, on the
other hand, improves from an AUC of 0.60$\pm$0.01 to 0.84$\pm$0.03 on
detecting $\epsilon=8/255$ and $\epsilon=64/255$ BIM attacks. Averaged across
all attacks, PASA obtains an AUC of 0.81$\pm$0.15, with the next best
detector, MagNet, obtaining 0.77$\pm$0.16 AUC. In Table IV, we can observe
that PASA obtains the highest TPRs at the lowest FPRs settings consistently.
For example, PASA obtains a TPR of 34.5% on detecting CW attacks at 1% FPR.
The next best detector is FS with 26.7% TPR.

##### ImageNet

Table III demonstrates that PASA consistently outperforms the baseline methods
in detecting attacks on both ImageNet models. For instance, when detecting an
$\epsilon=8/255$ PGD attack on ImageNet (MobileNet) and ImageNet (ResNet),
PASA scores an AUC of 0.98$\pm$0.02 and 0.97$\pm$0.01 respectively,
outperforming all baselines by a significant margin. In the ImageNet-ResNet
model, while PASA obtains an AUC of 0.95$\pm$0.01 for the CW attack, the next
best detector only has an AUC of 0.66$\pm$0.01. Baseline method (TWS)
performance is slightly better than PASA in detecting two BIM attacks (8/255
and 16/255) on ImageNet (ResNet). However, as the attack strength increases,
our method surpasses the performance of TWS. MagNet and U-LOO have very low
AUC scores on attacks like CW for ImageNet, which means that the threshold
learned from benign images was only suitable for detecting FGSM and PGD
attacks. This suggests that the detection criteria used in those methods may
not be effective against different types of attacks without knowing the attack
types beforehand, which is impractical. From Table IV, we can observe that
PASA obtains the highest TPRs at low FPRs across different attacks in both
ImageNet models.

##### MNIST

All unsupervised detectors have overall competitive performance in detecting
adversarial attacks on MNIST, with MagNet consistently obtaining high AUC and
TPR scores. However, PASA has a drop in performance, especially in detecting
BIM and CW attacks. This could be attributed to the lower resolution of MNIST
images. Lower resolution (28x28) implies less visual information for the
feature attribution method, compared with CIFAR-10 (32x32x3) and ImageNet
(224x224x3). It limits the granularity at which IG attributes importance to
individual features, resulting in a small number of attributions and lower
sensitivity to noise.

####  6.1.1 Analysis

PASA leverages the discrepancy between benign and adversarial samples in model
logits and class-specific feature attribution for detecting adversarially
perturbed samples. This discrepancy can be measured by injecting noise into a
given input sample and measuring the change in both logits and attribution
maps caused by noise. Previous studies [28] have demonstrated that the model
response of benign and adversarial inputs to noise differs because neural
networks are trained only with benign inputs. In this work, we also
demonstrate that the sensitivity of Integrated Gradients (IG) attribution is
linked to the sensitivity of the model (Eqn. 3). However, since IG assigns
importance to each feature, the level of granularity in importance attribution
depends on the number of features. Therefore, based on these considerations,
we combined these two inconsistency measures for the detection of
adversarially perturbed samples and we developed PASA accordingly to account
for a) the sensitivity of the trained model to noise, and b) the granularity
of IG attribution.

In our experiments, we followed a standard classification pipeline to achieve
high performance on the test set, using standard hyperparameters or pretrained
models (training details are discussed in Appendix B). However, different deep
learning models learn varying levels of feature abstraction from a given
dataset due to differences in depth, connections, and overall structure. For
instance, ResNet, with its residual connections, can capture more intricate
features compared to simpler networks like VGG or LeNet. Our experimental
results indicate that PASA performs notably better with deeper networks, as
demonstrated by the results obtained from ResNet models trained on CIFAR-10
and ImageNet (refer to Tables I and III). These findings suggest that
increased network depth, as observed in ResNet, enables the model to extract
complex patterns from the dataset, resulting in higher sensitivity to noise—a
quality utilized by PASA for detecting adversarial samples. Additionally, the
level of granularity in IG attribution depends on the number of features
present in the dataset. Consequently, PASA exhibits a notable decrease in
detecting certain attacks (e.g., BIM, CW) on MNIST, as the lower resolution of
MNIST leads to smaller norms of IG attribution. However, it consistently
obtains high detection performance on varying attacks on CIFAR-10, CIFAR-100,
and ImageNet.

Figure 7: Comparing adversarial detection performance of PASA with its
components, PS: Prediction Sensitivity, AS: Attribution Sensitivity, and
PS+AS: PASA. TABLE IV: Adversarial Detection Performance for CIFAR-100 and
ImageNet models: Our Method (PASA) vs. Unsupervised Methods (FS, MagNet,
U-LOO, TWS) using TPR scores.

| Performance | CIFAR-100 | ImageNet (MobileNet) | ImageNet (ResNet)
---|---|---|---|---
Attack | Metric | FS | MagNet | U-LOO | TWS | PASA | FS | MagNet | U-LOO | TWS | PASA | FS | MagNet | U-LOO | TWS | PASA
FGSM (8/255) | TPR (FPR @ 0.01) | 6.2 | 4.1 | 12.3 | 5.5 | 10.3 | 2 | 6.1 | 2.1 | 0.5 | 11.2 | 7.1 | 4.3 | 2 | 2.3 | 3.9
| TPR (FPR @ 0.05) | 25.1 | 4.4 | 15.7 | 4.6 | 35.9 | 4.4 | 6.9 | 13.4 | 6.9 | 29.5 | 11.3 | 5 | 7.1 | 7.4 | 15.1
| TPR (FPR @ 0.1) | 36.1 | 15.1 | 17.8 | 5.8 | 54.8 | 11 | 16.3 | 21.8 | 16.1 | 49.8 | 15.4 | 5.2 | 18.1 | 32.5 | 27.6
FGSM (16/255) | TPR (FPR @ 0.01) | 7.5 | 4.5 | 17.3 | 5.9 | 61.3 | 2.2 | 4.6 | 2.1 | 1.3 | 29.5 | 8.8 | 5.3 | 1.2 | 0.3 | 2.6
| TPR (FPR @ 0.05) | 29.3 | 4.7 | 28.7 | 4.6 | 86.3 | 4.5 | 5.3 | 15 | 6.5 | 56.7 | 12.8 | 6.1 | 8.2 | 1.7 | 19
| TPR (FPR @ 0.1) | 47.3 | 27 | 37.3 | 5.1 | 94 | 15.9 | 14.3 | 25 | 13.8 | 79 | 15.9 | 6.3 | 21.5 | 12.4 | 33.3
FGSM (32/255) | TPR (FPR @ 0.01) | 25.5 | 82.5 | 17.4 | 1.2 | 96.6 | 3.5 | 5.1 | 2.1 | 0.5 | 74.6 | 10.9 | 4.8 | 1.3 | 0.1 | 14.7
| TPR (FPR @ 0.05) | 33.4 | 84.1 | 15.3 | 7.8 | 99.7 | 4.2 | 6.1 | 18.8 | 6.8 | 93.2 | 16.6 | 5.3 | 7 | 0.1 | 45.1
| TPR (FPR @ 0.1) | 60.5 | 100 | 33.3 | 6.9 | 99.9 | 35.9 | 16.4 | 29.3 | 13.8 | 97.7 | 23.8 | 6.2 | 15.8 | 3.5 | 63.4
FGSM (64/255) | TPR (FPR @ 0.01) | 73.1 | 100 | 7.4 | 3 | 100 | 2.5 | 8.1 | 2.2 | 1.5 | 96.8 | 20.3 | 5.5 | 0.8 | 0 | 55.1
| TPR (FPR @ 0.05) | 74.3 | 100 | 17.6 | 4.9 | 100 | 11.4 | 9.2 | 15.2 | 6.6 | 99.6 | 2.2 | 7.4 | 5.1 | 0 | 84.4
| TPR (FPR @ 0.1) | 75.3 | 100 | 23.7 | 5.4 | 100 | 36.6 | 21.1 | 25.6 | 11.8 | 100 | 37.2 | 8.3 | 15.3 | 0.1 | 93.36
PGD (8/255) | TPR (FPR @ 0.01) | 22.4 | 3.8 | 15.9 | 6.3 | 3.1 | 6.2 | 6.1 | 1.6 | 0.7 | 98.8 | 5.6 | 4.2 | 5.4 | 22.2 | 99.1
| TPR (FPR @ 0.05) | 27.8 | 3.9 | 24.9 | 7.2 | 23.3 | 7.1 | 7.1 | 10.8 | 5.5 | 99.1 | 6.6 | 4.7 | 15.3 | 27 | 99.1
| TPR (FPR @ 0.1) | 27.8 | 15.2 | 31.2 | 7.4 | 35.4 | 7.4 | 16.1 | 17.9 | 11.2 | 99.4 | 7.5 | 4.8 | 31.7 | 39.3 | 99.2
PGD (16/255) | TPR (FPR @ 0.01) | 15.4 | 5.4 | 17.9 | 4.6 | 6.6 | 4.1 | 6.2 | 1.4 | 1.1 | 100 | 2.9 | 5.2 | 11.1 | 5.5 | 99.9
| TPR (FPR @ 0.05) | 16.3 | 5.5 | 28.9 | 5.8 | 29.4 | 4.8 | 7 | 13 | 4.8 | 100 | 3 | 6 | 24.8 | 6 | 99.9
| TPR (FPR @ 0.1) | 17.2 | 18.7 | 35.6 | 5.4 | 45.5 | 5.1 | 14.5 | 20.6 | 12.4 | 100 | 3.7 | 6.2 | 44 | 10 | 99.9
PGD (32/255) | TPR (FPR @ 0.01) | 8.5 | 9.9 | 15.5 | 6.4 | 14.4 | 4.4 | 7.7 | 12.1 | 2 | 100 | 1.9 | 5 | 5 | 0.4 | 100
| TPR (FPR @ 0.05) | 9.2 | 10.1 | 30.7 | 7.9 | 46.1 | 5.8 | 9.1 | 11.4 | 5.98 | 100 | 2 | 6.1 | 11.3 | 0.5 | 100
| TPR (FPR @ 0.1) | 10.6 | 74.1 | 37.2 | 9.5 | 57.9 | 7.4 | 17.9 | 21.8 | 12.4 | 100 | 2.8 | 6.5 | 30 | 0.8 | 100
PGD (64/255) | TPR (FPR @ 0.01) | 4.4 | 100 | 19.9 | 0 | 54.5 | 4.2 | 6.7 | 1.7 | 2.1 | 100 | 1.2 | 6 | 55.3 | 0 | 100
| TPR (FPR @ 0.05) | 4.5 | 100 | 30.8 | 0.1 | 85.4 | 5 | 7.9 | 11.3 | 6 | 100 | 1.4 | 6.8 | 14.7 | 0 | 100
| TPR (FPR @ 0.1) | 4.5 | 100 | 38.2 | 0.1 | 96.1 | 7.8 | 18.2 | 21.4 | 14.3 | 100 | 2.7 | 6.7 | 31 | 0 | 100
BIM (8/255) | TPR (FPR @ 0.01) | 12.3 | 4.5 | 12 | 6.7 | 4.3 | 10.1 | 0 | 0.1 | 0.5 | 27.1 | 11.5 | 0 | 61.5 | 49.1 | 14.7
| TPR (FPR @ 0.05) | 20.8 | 4.6 | 22 | 7.1 | 7.9 | 12.3 | 0 | 1.5 | 3.5 | 29.2 | 12.9 | 0 | 2.4 | 58.1 | 16.3
| TPR (FPR @ 0.1) | 25.4 | 14.2 | 28.2 | 7.4 | 13.4 | 13.7 | 0 | 2.4 | 9.1 | 32.9 | 14.4 | 0 | 5.1 | 75.8 | 19
BIM (16/255) | TPR (FPR @ 0.01) | 9.7 | 4 | 11.7 | 5.7 | 2.9 | 6.8 | 0 | 0.1 | 0.5 | 51.8 | 5.1 | 0 | 1.4 | 40.3 | 36.9
| TPR (FPR @ 0.05) | 11.6 | 4.1 | 22.4 | 6.6 | 6.5 | 9.2 | 0 | 2.5 | 3.1 | 53 | 5.5 | 0 | 2.3 | 48.5 | 37.9
| TPR (FPR @ 0.1) | 15.7 | 14.1 | 30 | 6.3 | 13.8 | 10 | 0 | 3.4 | 10.5 | 54.4 | 6.8 | 0 | 7 | 62.4 | 39
BIM (32/255) | TPR (FPR @ 0.01) | 8.2 | 4.9 | 12.2 | 2.9 | 1.1 | 4.7 | 0 | 0.2 | 0.5 | 72.3 | 2.2 | 0 | 2 | 21.6 | 57.3
| TPR (FPR @ 0.05) | 9.6 | 5 | 25.1 | 3.2 | 12.5 | 6.5 | 0 | 2.5 | 3.5 | 73.2 | 2.9 | 0 | 3.4 | 27.5 | 57.4
| TPR (FPR @ 0.1) | 10.1 | 16 | 26.5 | 3.5 | 26 | 6.8 | 0 | 4.3 | 9 | 74 | 3.9 | 0 | 7.4 | 39.7 | 57.7
BIM (64/255) | TPR (FPR @ 0.01) | 5.1 | 5.7 | 1.5 | 1.4 | 11.6 | 3 | 0 | 0.4 | 0.5 | 81.2 | 1.1 | 0 | 1.1 | 5.4 | 58.8
| TPR (FPR @ 0.05) | 5.2 | 6 | 1.2 | 1.8 | 36.7 | 4.3 | 0 | 2.1 | 4.1 | 81.2 | 1.4 | 0 | 3.2 | 7.7 | 59.9
| TPR (FPR @ 0.1) | 5.1 | 39.6 | 8.2 | 2.3 | 55.5 | 6.1 | 0 | 3.6 | 12 | 81.6 | 1.7 | 0 | 8.1 | 13.4 | 59.2
Auto-PGD (0.15) | TPR (FPR @ 0.01) | 32.5 | 19.6 | 14.8 | 2.4 | 98.5 | 6.6 | 1.2 | 7.5 | 0.5 | 98 | 2.5 | 1.4 | 5.1 | 1.3 | 97.1
| TPR (FPR @ 0.05) | 41.2 | 20.5 | 22.3 | 3.1 | 99.2 | 7 | 1.5 | 12.2 | 4.1 | 98.1 | 2.8 | 1.7 | 17.3 | 1.4 | 97.1
| TPR (FPR @ 0.1) | 51.5 | 100 | 41.5 | 4.2 | 99.7 | 7.4 | 4.9 | 23.2 | 7.9 | 98.2 | 3.3 | 1.8 | 45.9 | 2.8 | 97.1
CW (0.15) | TPR (FPR @ 0.01) | 26.7 | 9.5 | 12.2 | 2.5 | 34.5 | 7.2 | 0 | 0.1 | 0.5 | 45.6 | 10.2 | 0 | 1.1 | 0.8 | 75.2
| TPR (FPR @ 0.05) | 28.2 | 9.9 | 20.5 | 3.6 | 67.9 | 10.4 | 0 | 0.4 | 1.1 | 67.3 | 13.5 | 0 | 3.4 | 14.5 | 87.3
| TPR (FPR @ 0.1) | 29.3 | 82.6 | 25.7 | 4.1 | 85.3 | 15.5 | 0 | 1.6 | 8.3 | 76.2 | 16.4 | 0 | 5.4 | 42.3 | 94.1

###  6.2 False positive rates of unsupervised detectors

We evaluate unsupervised detectors with the True Positive Rate (TPR), computed
as the ratio of the total number of correctly identified adversarial examples
to the overall number of adversarial examples. We compute the TPR of detectors
by using the threshold learned during training for specific thresholds on the
validation set. However, it is highly unlikely that the detector will get the
same FPR on the test set. Hence, computing another metric, false positive rate
(FPR), is an important criterion. FPR measures the ratio of the number of
natural images identified as adversarial images to the total number of natural
images. We compare the FPR of PASA against baselines in Figure 8, where we
plot the average FPRs on the test set across all attacks corresponding to the
FPR associated with the threshold learned during training (1%, 5%, 10%). The
dotted line represents the ideal position of the plot. Detectors that are
closer to this line have lower FPRs and are better detectors in classifying
the benign images correctly. We can observe that PASA consistently obtains
better false positive rates than other methods on CIFAR-10 and ImageNet. On
CIFAR-100, while TWS has the lowest FPRs overall, our method obtains better
FPRs than U-LOO and MagNet.

(a) CIFAR-10

(b) CIFAR-100

(c) ImageNet

Figure 8: FPR on validation set vs FPR on test set.

###  6.3 Ablation study

PASA comprises two statistical metrics: prediction sensitivity and attribution
sensitivity. In this ablation study, we assess the individual performance of
each metric. Specifically, our focus is on evaluating the detection
performance of our proposed method when utilizing only one of the statistical
metrics. We maintain a similar experimental setup, selecting 1000 benign
images from the test set that are accurately classified by the model. For each
attack, we generate 1000 corresponding adversarial images. We use the
thresholds for prediction sensitivity and attribution sensitivity learned
during the training of our detector. The collective average AUC for each
dataset under different attacks is illustrated in Figure 7. We summarize the
results below:

1\. On CIFAR-10 (VGG), prediction sensitivity outperforms attribution
sensitivity for detection even though both metrics have high performance on
average.

2\. On CIFAR-10 CIFAR-10 (ResNet), attribution sensitivity outperforms
prediction sensitivity. Its standalone performance is almost equivalent to the
combined performance.

3\. On CIFAR-100, the performance of attribution sensitivity and prediction
sensitivity is almost equivalent.

4\. On ImageNet, both metrics have lower performance when used standalone. The
combined performance is significantly better than the individual metric.

Detailed results can be found in Appendix F, where we demonstrate that AS and
PS exhibit sensitivity to different attack types, and the combination of both
metrics provides a more balanced detection strategy across various attack
types.

###  6.4 Evaluation with adaptive attacks

In the previous experiments, we assume that the adversary has access to the
model details but does not know the details of our detection mechanism. While
this is a realistic assumption, it does not provide the robustness measure of
the proposed detector. We now evaluate the performance of our proposed method
under adaptive attacks to evaluate its robustness. Adaptive attacks are
adversarial attacks targeted at a defense mechanism and are adapted to the
specific details of a defense. Since our detection approach comprises two
statistical measures, we perform adaptive attacks on both components [65]. We
optimize the PGD attack with perturbation set at $0.1$ and evaluate our
results on the CIFAR-10 (ResNet) dataset.

First, we attack the feature attribution method, Integrated Gradient (IG). An
adversary tries to deceive both the target classifier and IG. Similar to ADV2
attack [75], this attack generates an adversarial image $\textbf{x}^{*}$ such
that following conditions are satisfied: 1) target classifier ($F$)
misclassifies $\textbf{x}^{*}$, 2) IG generates attribution similar to benign
counterpart x where the similarity is measured using the intersection-over-
union (IoU) test, widely used in object detection [23], and 3) the difference
between benign x and adversarial $\textbf{x}^{*}$ image is minimized.

We solve the following optimization for an image x,

$min_{\textbf{x}^{*}}L_{1}(F(\textbf{x}^{*}),y^{*})+c*L_{2}(IG(\textbf{x}^{*}),IG(\textbf{x}))$
(12)

where $L_{1}$ is the prediction loss used by PGD attack,
$L_{2}=||IG(\textbf{x}^{*})-IG(\textbf{x})||_{2}$ is the loss measuring the
difference between the attribution map of the benign image and its adversarial
counterpart, and $c$ is a hyper-parameter to balance the two losses. Similar
to ADV2 attack [75], we observed that it is inefficient to search for
adversarial input by directly running the updates using Eq. 12. Hence, we
perform a warm start by first running a fixed number of steps for the regular
PGD attack and then resume the updates of Eq. 12. We use the following values
of $c\in[5,10,20,30,50]$ for iteration steps $\in[300,200,100,50]$. We
generate 1000 adversarial images according to the attack strategy of Eq. 12.
These adversarial images obtain an attack success rate of 100% to fool the
model (condition 1); the mean IoU score between benign and adversarial
attribution is 43% (condition 2) (note that ADV2 [75] also obtained IoU of
only 50% on attacking Vanilla Gradient [60]). The mean L2 distortion of
successful adaptive adversarial images is 2.8, which is slightly higher
compared with the PGD attack (2.6) (condition 3). We apply our detection
strategy on the adversarial images and obtain an AUC score of $0.75$. The
adaptive attack takes a significantly longer time compared with PGD. A normal
PGD attack takes about 0.065 seconds to generate an adversarial sample for a
single image, whereas this adaptive attack takes around 26.21 seconds, which
is $\sim$400 times slower.

TABLE V: Performance of PASA against Adaptive Attacks. Complexity measures
computation time in seconds.

Attack | Attack success rate | Detection AUC | Complexity (time)
---|---|---|---
| Before | After | Before | After | Before | After
Attack on IG | 100% | 100% | 0.98 | 0.75 | 0.065 | 26.21
Attack on logits | 100% | 100% | 0.98 | 0.76 | 0.065 | 0.32
Combined attack | 100% | 100% | 0.98 | 0.69 | 0.065 | 28.33

Next, we perform an adaptive attack on the model logits. The adversary creates
adversarial images in such a way that the distribution of logits is closer to
the logits of benign images. We follow the logit matching attack of Tramer et
al. [65]. We solve the following optimization to obtain an adversarial image
$\textbf{x}^{*}$

$min_{\textbf{x}^{*}}L_{1}(F(\textbf{x}^{*}),y^{*})+L_{3}(Z(\textbf{x}^{*}),Z(\textbf{x}))$
(13)

where $L_{1}$ is prediction loss of a PGD attack. The second loss term
$L_{3}=||Z(\textbf{x}^{*})-Z(\textbf{x})||_{2}$ is the mean square loss
between logits of adversarial and benign images. The attack runs for a fixed
number of iterations (given by PGD iterations) and produces adversarial
samples whose logits are closer to benign counterparts. For 1000 adversarial
images obtained using this strategy, the adaptive attack still achieves a 100%
attack success rate. The mean L2 distortion of successful samples is 2.7,
which is similar to the PGD attack (2.6). We obtain an AUC score of $0.76$ for
the detector. We observe that attacking only one component of our detector
does not significantly impact the overall detection.

Figure 9: Attribution map (second row) for three different images (first row):
benign image, adversarial image from PGD attack, and adversarial image from
adaptive attack.

Finally, we introduce attacks on both the attribution method and model logits.
We introduce two different losses and solve the following optimization to
obtain an adversarial image $\textbf{x}^{*}$ for an input image x

$min_{\textbf{x}^{*}}L_{1}(.)+c*L_{2}(.)+L_{3}(.)$ (14)

where $L_{1}(F(\textbf{x}^{*}),y*)$ is prediction loss,
$L_{2}(IG(\textbf{x}^{*}),IG(\textbf{x}))$ is the loss measuring the
difference between the attribution map of benign samples and their adversarial
counterparts, $L_{3}(Z(\textbf{x}^{*}),Z(\textbf{x}))$ measures the loss
between logits of benign and adversarial samples and $c$ is a hyper-parameter.
We perform a warm start by searching for adversarial samples using PGD attack
and then iteratively optimize Eq. 14 to obtain adversarial samples with
attribution vector and logits similar to benign samples. For a similar test
setting, the adaptive attack obtains an attack success rate of 100%. The mean
L2 distortion of successful samples is 2.7. The AUC score of the detector is
now reduced to $0.69$. This adaptive attack takes around 28.33 seconds on
average for each sample. Figure 9 shows the results of this adaptive attack.
The first row shows images from class “Truck” from CIFAR-10, and the second
row shows its heatmap computed using IG. We can observe that the attribution
map of a PGD image differs significantly from the attribution map of a normal
image. After performing an adaptive attack (which attacks both feature
attribution and model logit), the adversary obtains a perturbed image with its
attribution map similar to that of a natural image.

We summarize the result in Table V where attack success rate measures the
success of the attack in changing the label of an image, detection AUC
measures the performance of our detector in detecting adversarial images, and
complexity measures the time required by the attack for a single image (in
seconds). We observe that performing adaptive attacks against both components
of our detector increases computational complexity. However, even though the
detector performance drops with adaptive attacks, PASA is still able to
achieve a competitive performance under this strongest adversary assumption.
In Table VI, we evaluate the performance of different detection methods
against the adaptive adversarial samples obtained using Eqn. 14. This
adversarial attack was specifically designed to evade PASA’s detection
mechanism. However, all detection methods have considerably lower AUCs in
detecting the adversarial samples.

####  6.4.1 Analysis

Prior works have shown that it is possible to add imperceptible perturbation
to images for generating random attribution maps [21, 75, 59]. However,
evading a classifier and generating an attribution similar to benign
counterparts is much more challenging. Attacks like ADV2 [75] only achieved a
50% IOU when targeting the Vanilla Gradient method [53]. In our evaluation,
the adaptive attack only achieved 43% IOU on attacking the Integrated Gradient
(IG) method. This means there is still a significant discrepancy between the
attribution map of benign and adversarial samples that PASA can utilize in
detection. This difficulty stems from the challenge of satisfying two counter-
intuitive objectives: retaining the adversarial label while aligning the
attribution with the benign example. This was validated by a recent work [8],
which shows that it is difficult to remove the $L_{1}$-norm of attribution
discrepancy between benign and adversarial images when an attribution method
satisfying completeness axiom (e.g. IG [60]) is used. These findings suggest
that an explanation method like IG can help detect discrepancies between
benign and adversarial examples.

TABLE VI: Evaluation of Adaptive Attacks

Method | FS | MagNet | TWS | U-LOO | PASA
---|---|---|---|---|---
AUC (Before) | 0.13 | 0.91 | 0.14 | 0.68 | 0.98
AUC (After) | 0.54 | 0.51 | 0.35 | 0.59 | 0.69

##  7 Application On Security Dataset

In this section, we demonstrate the application of PASA on a network security
dataset. We specifically focus on a network intrusion detection problem and
evaluate PASA using the updated CIC-IDS2017 dataset [18]. Since the goal of an
adversarial attack on network data is to classify attack samples as benign, we
preprocess the data by assigning a single attack label to different attack-
traffic types. Subsequently, we build a multi-layer perceptron model with a
binary classification objective, which achieves an accuracy of 99.04% on the
test set. Further details about the dataset, and model can be found in
Appendix A and B.5.

We generate adversarial samples using FGSM, BIM, PGD, CW and Auto-PGD attacks
by considering a threat model where an adversary is able to manipulate the
dataset in feature-space. Hence, such attacks might not be representative of a
realistic settings and might produce “unrealizable” adversarial samples [51,
3].

In Table VII, we compare U-LOO and PASA (AUC scores) on the intrusion
detection model. It is important to note that all baseline methods we
considered cannot be directly applied to security applications. For instance,
FS [71] is designed for image data and necessitates the application of various
image filters for adversarial detection. MagNet [41], proposed for image
datasets, relies on an auto-encoder model to compute the reconstruction error.
TWS [28] leverages the difference in softmax predictions between the original
and noisy inputs to detect adversarial samples. However, there is a negligible
difference between the softmax outputs of the original and noisy inputs in the
binary classification task, rendering TWS ineffective. On the other hand,
U-LOO works well in our scenario as it measures the interquartile range (IQR)
of feature attributions for both benign and adversarial samples.

##  8 Discussion

###  8.1 Indicators of attack failures

Indicators of attack failures [44] serve to uncover potential vulnerabilities
in the adversarial robustness evaluation. In our defense assessment, two
specific scenarios merit consideration: a) Non-converging attack: Prior
defenses [9, 43] often employed small steps in PGD attack, introducing the
risk of non-convergence. As discussed in Sec. 5.1.3, we mitigated this risk by
opting for larger iteration steps (40) in our PGD attack formulation. Aligning
with Pintor et al.[44] recommendation, we reassessed detection methods on the
PGD attack with 100 steps. Notably, all detection methods exhibited no
significant performance degradation when compared with the previous evaluation
of PGD samples (steps=40). Results are summarized in Appendix E. b. Non-
adaptive attacks: Prior defenses [15] have developed adaptive attacks,
neglecting non-differentiable or additional defense components. However,
adaptive attacks that ignore a component of a defense do not guarantee
successful bypassing of the target model. In our case, both model prediction
and feature attribution are differentiable and we incorporate them in crafting
adaptive adversarial samples, as discussed in Section 6.4.

TABLE VII: Evaluation on updated CIC-IDS2017

Method | FGSM | PGD | BIM | CW | Auto-PGD
---|---|---|---|---|---
PASA | 0.99 $\pm$ 0.00 | 0.98 $\pm$ 0.02 | 0.99 $\pm$ 0.01 | 0.98 $\pm$ 0.03 | 0.97 $\pm$ 0.03
U-LOO [72] | 0.70 $\pm$ 0.05 | 0.66 $\pm$ 0.07 | 0.76 $\pm$ 0.01 | 0.75 $\pm$ 0.01 | 0.63 $\pm$ 0.00

###  8.2 Efficient Lightweight detection

The main strength of PASA is its lightweight detection approach. Since it
requires the computation of two statistics by probing a given input image with
noise, it has low inference latency. The simplicity of this approach means
that it has a small memory footprint, making it suitable for deployment on
resource-constrained devices or in scenarios where computational resources are
limited. While other unsupervised methods like MagNet [41], FS [71], and TWS
[28], also have low inference time, PASA outperforms these methods in
detecting attacks against CIFAR and ImageNet models. We evaluated the
inference time, training time, and memory usage (peak memory required) of
different detection methods for 1000 images and report average results in
Table VIII, where we can observe that PASA is faster than LOO but slower than
TWS, FS, and MagNet. MagNet requires the highest training time, attributed to
the necessity of training a separate defense model. PASA has a moderate
training time, significantly lower than MagNet and LOO. PASA also has a
moderate memory usage, higher than TWS but lower than LOO, and MagNet.

###  8.3 On explanation methods and their explanation fragility

An adversary can introduce perturbations to an input sample such that it is
misclassified, but its attribution map is similar to the attribution of benign
sample [75]. This is an attack on the explanation method. Such attacks affect
the performance of detection methods that utilize disparity in feature
attribution. However, the success of this attack depends on the attribution
method. In our approach, we employed IG[60], which displayed higher resilience
against adaptive adversarial attacks. A similar result was demonstrated in a
study by Vardhan et al. [67]. However, the Vanilla Gradient [53] is more
sensitive to such attacks, producing similar attribution maps between benign
and adversarial samples [75]. Alternative feature attribution methods such as
LRP [4] and GBP [57] can also be manipulated to produce targeted attribution
maps mimicking those of benign samples, effectively fooling the detector [67].

TABLE VIII: Computational overhead of detection methods

Method | Inference Time (s) | Training Time (s) | Memory Usage (MB)
---|---|---|---
PASA | 0.0156 | 15.460 | 2701.46
U-LOO [72] | 0.0540 | 53.076 | 2800.62
FS [71] | 0.0085 | 8.456 | 2644.98
MagNet [41] | 0.0007 | 1262.778 | 2744.41
TWS [28] | 0.0051 | 5.406 | 2576.08

###  8.4 Utilizing latent representations

PASA utilizes the final layer features from the classification model (logit
layer) and the explanation method (input attribution) for detecting
adversarial samples. This can be extended to incorporate features from
multiple intermediate layers as well. Few recent works exploit the behavior of
neural networks to benign and adversarial samples in hidden layers to design
supervised adversarial detectors [72, 56]. However, since DNNs can have many
hidden layers, it will require careful consideration to include features from
specific layers; otherwise, the detectors may overfit the training data due to
a large number of features. We will explore the inclusion of statistics from
latent representations in our future work.

###  8.5 Limitations

PASA works well for detecting adversarial samples generated through
$L_{\infty}$ attacks. Such attacks aim to maximize perturbation within a
bounded norm. PASA is effective against them because it can capture
significant changes in attribution and prediction differences, which result
from substantial perturbations. In contrast, $L_{1},L_{2}$ attacks make
minimal changes to the input by altering only a few pixels and minimizing
perturbation magnitude, respectively (See Table IX). Integrated Gradient may
not capture such small or subtle perturbations effectively. For instance,
evaluating the detection methods on $L_{2}$ PGD attacks on the CIFAR-10 ResNet
at $\epsilon=64/255$, we obtained the following AUC: 0.59 (PASA), 0.55 (FS),
MagNet (0.51), U-LOO (0.52), TWS (0.38).

Other attacks beyond the $L_{p}$ attack (e.g., patch attacks) modify only a
specific part of an image by adding a patch, so directly applying PASA and
observing the difference in prediction and attribution does not work. However,
such attacks still perform significant modifications to hidden feature maps
that produce changes in model prediction. Our future work will focus on
utilizing noise-based approaches on hidden-activations in detecting other
evasion attacks of $L_{1}$, $L_{2}$ norms, and patch attacks.

PASA also has a noticeable drop in performance with images of lower resolution
like MNIST. This is because the granularity at which IG attributes importance
to individual features in MNIST is smaller and hence, it has lower sensitivity
to noise, the quality utilized by PASA in detection.

Like other noise-based approaches [28, 46], our noise parameter needs to be
determined empirically. This means that the effectiveness of the method can
depend on the specific dataset and problem at hand. Selecting the optimal
noise parameter requires experimentation, which could be time-consuming before
deployment. However, we have demonstrated through different datasets and
network architectures that once the optimal noise value is discovered, PASA
can be generalized across datasets for lightweight detection of attacks.

TABLE IX: Average distortion between benign & adversarial CIFAR-10 images at
different attack strength

Attack | 8/255 | 16/255 | 32/255 | 64/255
---|---|---|---|---
$L_{1}$ PGD | 0.001 | 0.002 | 0.003 | 0.007
$L_{2}$ PGD | 0.031 | 0.063 | 0.125 | 0.251
$L_{\infty}$ PGD | 1.456 | 2.706 | 4.818 | 8.512

##  9 Conclusion

In this paper, we propose PASA, a lightweight attack-agnostic, unsupervised
method for detecting adversarial samples. We use noise as a probing tool to
measure the sensitivity of model prediction and feature attribution. We learn
thresholds of sensitivity scores from benign samples and utilize them for
detection. PASA outperforms existing statistical unsupervised detectors in
classifying adversarial samples on the updated CIC-IDS2017, CIFAR-10,
CIFAR-100, and ImageNet datasets. PASA displays robust performance in
detecting adversarial samples even when an adversary has full knowledge of the
detector. We aim to extend the scope of our approach in future studies,
particularly to detect adversarial attacks of the $L_{0}$ and $L_{2}$ norm and
physically realizable patch-based attacks, and improve the security of diverse
systems that use deep learning models.

## Acknowledgement

We are grateful to anonymous reviewers at IEEE Euro S&P for their valuable
feedback. This work was supported by Toyota InfoTech Labs through Unrestricted
Research Funds and RIT AI Seed Fund Program.

## References

  * [1] Jonathan Aigrain and Marcin Detyniecki.  Detecting adversarial examples and other misclassifications in neural networks by introspection.  CoRR, abs/1905.09186, 2019.
  * [2] Hyrum S Anderson and Phil Roth.  Ember: an open dataset for training static pe malware machine learning models.  arXiv preprint arXiv:1804.04637, 2018.
  * [3] Giovanni Apruzzese, Mauro Andreolini, Luca Ferretti, Mirco Marchetti, and Michele Colajanni.  Modeling realistic adversarial attacks against network intrusion detection systems.  Digital Threats: Research and Practice (DTRAP), 3(3):1–19, 2022.
  * [4] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  PloS one, 10(7):e0130140, 2015.
  * [5] Dipkamal Bhusal, Rosalyn Shin, Ajay Ashok Shewale, Monish Kumar Manikya Veerabhadran, Michael Clifford, Sara Rampazzi, and Nidhi Rastogi.  Sok: Modeling explainability in security analytics for interpretability, trustworthiness, and usability.  In The 18th International Conference on Availability, Reliability and Security (ARES 2023), 2023.
  * [6] Battista Biggio and Fabio Roli.  Wild patterns: Ten years after the rise of adversarial machine learning.  In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pages 2154–2156, 2018.
  * [7] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba.  End to end learning for self-driving cars.  CoRR, abs/1604.07316, 2016.
  * [8] Akhilan Boopathy, Sijia Liu, Gaoyuan Zhang, Cynthia Liu, Pin-Yu Chen, Shiyu Chang, and Luca Daniel.  Proper network interpretability helps adversarial robustness in classification.  In International Conference on Machine Learning, pages 1014–1023. PMLR, 2020.
  * [9] Jacob Buckman, Aurko Roy, Colin Raffel, and Ian Goodfellow.  Thermometer encoding: One hot way to resist adversarial examples.  In International conference on learning representations, 2018.
  * [10] Kirill Bykov, Anna Hedström, Shinichi Nakajima, and Marina M-C Höhne.  Noisegrad—enhancing explanations by introducing stochasticity to model weights.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6132–6140, 2022.
  * [11] Nicholas Carlini and David Wagner.  Adversarial examples are not easily detected: Bypassing ten detection methods.  In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 3–14, 2017.
  * [12] Nicholas Carlini and David Wagner.  Towards evaluating the robustness of neural networks.  In 2017 ieee symposium on security and privacy (sp), pages 39–57. Ieee, 2017.
  * [13] Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Xi Wu, and Somesh Jha.  Concise explanations of neural networks using adversarial training.  In International Conference on Machine Learning, pages 1383–1391. PMLR, 2020.
  * [14] Francesco Croce and Matthias Hein.  Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.  In International conference on machine learning, pages 2206–2216. PMLR, 2020.
  * [15] Nilaksh Das, Madhuri Shanbhogue, Shang-Tse Chen, Fred Hohman, Siwei Li, Li Chen, Michael E Kounavis, and Duen Horng Chau.  Shield: Fast, practical defense and vaccination for deep learning using jpeg compression.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 196–204, 2018.
  * [16] Jesse Davis and Mark Goadrich.  The relationship between precision-recall and roc curves.  In Proceedings of the 23rd international conference on Machine learning, pages 233–240, 2006.
  * [17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [18] Gints Engelen, Vera Rimmer, and Wouter Joosen.  Troubleshooting an intrusion detection dataset: the cicids2017 case study.  In 2021 IEEE Security and Privacy Workshops (SPW), pages 7–12. IEEE, 2021.
  * [19] Reuben Feinman, Ryan R. Curtin, Saurabh Shintre, and Andrew B. Gardner.  Detecting adversarial samples from artifacts.  CoRR, abs/1703.00410, 2017.
  * [20] Jacob R. Gardner, Matt J. Kusner, Yixuan Li, Paul Upchurch, Kilian Q. Weinberger, and John E. Hopcroft.  Deep manifold traversal: Changing labels with convolutional features.  CoRR, abs/1511.06421, 2015.
  * [21] Amirata Ghorbani, Abubakar Abid, and James Zou.  Interpretation of neural networks is fragile.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3681–3688, 2019.
  * [22] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [23] Kaiming He and Georgia Gkioxari.  P. doll ar, and r. girshick,“mask r-cnn,”.  In Proc. IEEE Int. Conf. Comput. Vis, pages 2980–2988, 2017.
  * [24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [26] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  International Conference on Learning Representations (ICLR), 2016.
  * [27] Hossein Hosseini, Sreeram Kannan, and Radha Poovendran.  Are odds really odd? bypassing statistical detection of adversarial examples.  arXiv preprint arXiv:1907.12138, 2019.
  * [28] Shengyuan Hu, Tao Yu, Chuan Guo, Wei-Lun Chao, and Kilian Q. Weinberger.  A new defense against adversarial images: Turning a weakness into a strength.  In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1633–1644, 2019.
  * [29] Kaggle.  Imagenet 1000 (mini).  https://rb.gy/udu0a, 2020.
  * [30] Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson.  Captum: A unified and generic model interpretability library for pytorch, 2020.
  * [31] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [32] K Naveen Kumar, C Vishnu, Reshmi Mitra, and C Krishna Mohan.  Black-box adversarial attacks in autonomous vehicle technology.  In 2020 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), pages 1–7. IEEE, 2020.
  * [33] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio.  Adversarial examples in the physical world.  In Artificial intelligence safety and security, pages 99–112. Chapman and Hall/CRC, 2018.
  * [34] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [35] Jiwei Li, Will Monroe, and Dan Jurafsky.  Understanding neural networks through representation erasure.  CoRR, abs/1612.08220, 2016.
  * [36] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song.  Delving into transferable adversarial examples and black-box attacks.  In International Conference on Learning Representations, 2016.
  * [37] Scott M. Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4765–4774, 2017.
  * [38] Shiqing Ma, Yingqi Liu, Guanhong Tao, Wen-Chuan Lee, and Xiangyu Zhang.  Nic: Detecting adversarial samples with neural network invariant checking.  In 26th Annual Network And Distributed System Security Symposium (NDSS 2019). Internet Soc, 2019.
  * [39] Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey.  Characterizing adversarial subspaces using local intrinsic dimensionality.  In International Conference on Learning Representations, 2018.
  * [40] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
  * [41] Dongyu Meng and Hao Chen.  Magnet: a two-pronged defense against adversarial examples.  In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pages 135–147, 2017.
  * [42] Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian Molloy, and Ben Edwards.  Adversarial robustness toolbox v1.2.0.  CoRR, 1807.01069, 2018.
  * [43] Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu.  Improving adversarial robustness via promoting ensemble diversity.  In International Conference on Machine Learning, pages 4970–4979. PMLR, 2019.
  * [44] Maura Pintor, Luca Demetrio, Angelo Sotgiu, Ambra Demontis, Nicholas Carlini, Battista Biggio, and Fabio Roli.  Indicators of attack failure: Debugging and improving optimization of adversarial examples.  Advances in Neural Information Processing Systems, 35:23063–23076, 2022.
  * [45] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  ” why should i trust you?” explaining the predictions of any classifier.  In 22nd ACM SIGKDD, 2016.
  * [46] Kevin Roth, Yannic Kilcher, and Thomas Hofmann.  The odds are odd: A statistical test for detecting adversarial examples.  In International Conference on Machine Learning, pages 5498–5507. PMLR, 2019.
  * [47] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  łlearning internal representations by error propagation, ž california univ san diego la jolla inst for cognitive science.  Technical report, Tech. Rep, 1985.
  * [48] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [49] Ali Shafahi, W. Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein.  Are adversarial examples inevitable?  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  * [50] Iman Sharafaldin, Arash Habibi Lashkari, Ali A Ghorbani, et al.  Toward generating a new intrusion detection dataset and intrusion traffic characterization.  ICISSp, 1:108–116, 2018.
  * [51] Ryan Sheatsley, Blaine Hoak, Eric Pauley, Yohan Beugin, Michael J Weisman, and Patrick McDaniel.  On the robustness of domain constraints.  In Proceedings of the 2021 ACM SIGSAC conference on computer and communications security, pages 495–515, 2021.
  * [52] Dinggang Shen, Guorong Wu, and Heung-Il Suk.  Deep learning in medical image analysis.  Annual review of biomedical engineering, 19:221–248, 2017.
  * [53] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014.
  * [54] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [55] Angelo Sotgiu, Ambra Demontis, Marco Melis, Battista Biggio, Giorgio Fumera, Xiaoyi Feng, and Fabio Roli.  Deep neural rejection against adversarial examples.  EURASIP Journal on Information Security, 2020:1–10, 2020.
  * [56] Philip Sperl, Ching-Yu Kao, Peng Chen, Xiao Lei, and Konstantin Böttinger.  Dla: dense-layer-analysis for adversarial example detection.  In 2020 IEEE European Symposium on Security and Privacy (EuroS&P), pages 198–215. IEEE, 2020.
  * [57] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller.  Striving for simplicity: The all convolutional net.  arXiv preprint arXiv:1412.6806, 2014.
  * [58] Pascal Sturmfels, Scott Lundberg, and Su-In Lee.  Visualizing the impact of feature attribution baselines.  Distill, 5(1):e22, 2020.
  * [59] Akshayvarun Subramanya, Vipin Pillai, and Hamed Pirsiavash.  Towards hiding adversarial examples from network interpretation.  arXiv preprint arXiv:1812.02843, 2018.
  * [60] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In International conference on machine learning, pages 3319–3328. PMLR, 2017.
  * [61] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
  * [62] Thomas Tanay and Lewis D. Griffin.  A boundary tilting persepective on the phenomenon of adversarial examples.  CoRR, abs/1608.07690, 2016.
  * [63] Guanhong Tao, Shiqing Ma, Yingqi Liu, and Xiangyu Zhang.  Attacks meet interpretability: Attribute-steered detection of adversarial samples.  Advances in Neural Information Processing Systems, 31, 2018.
  * [64] Florian Tramèr.  Detecting adversarial examples is (nearly) as hard as classifying them.  In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 21692–21702. PMLR, 2022.
  * [65] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry.  On adaptive attacks to adversarial example defenses.  Advances in neural information processing systems, 33:1633–1645, 2020.
  * [66] LJP van der Maaten and GE Hinton.  Visualizing high-dimensional data using t-sne. 2008.  Journal of Machine Learning Research, 9:2579.
  * [67] Raj Vardhan, Ninghao Liu, Phakpoom Chinprutthiwong, Weijie Fu, Zhenyu Hu, Xia Ben Hu, and Guofei Gu.  Exad: An ensemble approach for explanation-based adversarial detection.  arXiv preprint arXiv:2103.11526, 2021.
  * [68] Jingyuan Wang, Yufan Wu, Mingxuan Li, Xin Lin, Junjie Wu, and Chao Li.  Interpretability is a kind of safety: An interpreter-based ensemble for adversary defense.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 15–24, 2020.
  * [69] Yuhang Wu, Sunpreet S Arora, Yanhong Wu, and Hao Yang.  Beating attackers at their own games: Adversarial example detection using adversarial gradient directions.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2969–2977, 2021.
  * [70] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He.  Feature denoising for improving adversarial robustness.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 501–509, 2019.
  * [71] Weilin Xu, David Evans, and Yanjun Qi.  Feature squeezing: Detecting adversarial examples in deep neural networks.  In 25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-21, 2018. The Internet Society, 2018.
  * [72] Puyudi Yang, Jianbo Chen, Cho-Jui Hsieh, Jane-Ling Wang, and Michael Jordan.  Ml-loo: Detecting adversarial examples with feature attribution.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6639–6647, 2020.
  * [73] Chiliang Zhang, Zhimou Yang, and Zuochang Ye.  Detecting adversarial perturbations with salieny.  In Proceedings of the 6th International Conference on Information Technology: IoT and Smart City, pages 25–30, 2018.
  * [74] Huan Zhang, Hongge Chen, Zhao Song, Duane Boning, Inderjit S Dhillon, and Cho-Jui Hsieh.  The limitations of adversarial training and the blind-spot attack.  In International Conference on Learning Representations, 2018.
  * [75] Xinyang Zhang, Ningfei Wang, Hua Shen, Shouling Ji, Xiapu Luo, and Ting Wang.  Interpretable deep learning under fire.  In Srdjan Capkun and Franziska Roesner, editors, 29th USENIX Security Symposium, USENIX Security 2020, August 12-14, 2020, pages 1659–1676. USENIX Association, 2020.

##  Appendix A Datasets

1\. MNIST [34]: MNIST is a handwritten digit dataset with digits from $0$ to
$9$ in 10 classes with 60000 train images and 10000 test images. Each image in
the dataset is a grayscale image with a size of $28$x$28$. We use the PyTorch
torch-vision MNIST dataset for our evaluation.

2\. CIFAR-10 [31]: The CIFAR-10 dataset consists of $32$x$32$ three-channel
images in 10 classes with 6000 images per class. There are 50,000 training
images and 10,000 test images. The images belong to different real-life
objects like airplanes, cars, birds, cats, dogs, frogs, and trucks. We use the
PyTorch torch-vision CIFAR-10 dataset for our evaluation.

3\. CIFAR-100 [31]: This dataset is similar to CIFAR-10, but it has 100
classes containing 600 images each. There are 500 training images and 100
testing images per class. We use the PyTorch torch-vision CIFAR-10 dataset for
our evaluation.

4\. ImageNet [17]: ImageNet consists of $1000$ classes of high-dimensional
real-life RGB images. We use the open-source ImageNet subset available on
Kaggle [29]. It consists of 25000 images on the train set and 3000 images on
the validation set.

5\. Updated CIC-IDS2017 dataset [18]: The CIC-IDS2017 dataset [50] is a
popular dataset for evaluating intrusion detection systems (IDSs) in network
security. This dataset was created by the Canadian Institute for Cybersecurity
(CIC) and consists of network traffic data collected in a controlled
environment. However, a recent study [18] demonstrated problems with the
feature extraction and labeling of this dataset, and provided an improved
version of the dataset222https://intrusion-detection.distrinet-
research.be/WTMC2021/Dataset/dataset.zip. We utilize this updated version of
CIC-IDS2017 to perform analysis on network intrusion detection. Unlike image
data, preprocessing is required for the security dataset available in CSV
format. The dataset consists of potentially incorrect values and varying
formats, which necessitate preprocessing steps. We transform the categorical
features into binary features using one-hot encoding and scale the values
within the range of [0, 1] using min-max normalization. Since the goal of
adversarial attack on network dataset is to classify malicious traffic flow to
benign, we transform the class labels to binary classes (0 for normal traffic
and 1 for attack traffic).

##  Appendix B Target models

###  B.1 LeNet [34]

Lenet is one of the earliest convolutional neural network architectures
originally designed for handwritten digit recognition. LeNet consists of a
series of convolutional and pooling layers followed by a fully connected layer
and output layer for classification. We applied LeNet architecture, as
demonstrated in Table X, for MNIST classification. The model was trained with
a learning rate of 0.001 for 60 epochs using a batch size of 64 and the Adam
optimizer. We obtained an accuracy of 98.17% on the test set.

TABLE X: MNIST model architecture

# | Layer | Description
---|---|---
1 | Conv2D+ReLU | 6 filters, Kernel size = (5,5), Stride = (1,1)
2 | MaxPooling | Kernel size = 2, Stride = 2, Padding = 0
3 | Conv2D+ReLU | 16 filters, Kernel size = (5,5), Stride = (1,1)
4 | MaxPooling | Kernel size = 2, Stride = 2, Padding = 0
5 | Dense+ReLU | 256 units
6 | Dense+ReLU | 120 units
7 | Dense+Softmax | 84 units

###  B.2 VGG [54]

VGG networks are also convolutional neural networks with deeper stacking of
convolutional layers than LeNet. It consists of a series of convolutional
neural networks followed by pooling and fully connected layers. Table XI
summarizes the architecture used for CIFAR-10 classification. The model was
trained with a learning rate of 0.001 for 100 epochs using a batch size of 64
and the SGD optimizer with momentum of 0.9 and weight decay of 5e-4. We
obtained an accuracy of 84.91% on the test set.

TABLE XI: CIFAR-10 VGG16 Architecture

# | Layer | Description
---|---|---
1 | Conv2d+ReLU | 64 filters, Kernel size = (3, 3) size, Stride=(1,1), Padding = (1,1)
2 | Conv2d+ReLU | 64 filters, Kernel size = (3, 3) size, Stride=(1,1), Padding = (1,1)
3 | MaxPooling | Kernel size = 2, Stride =2, Padding = 0
4 | Conv2d+ReLU | 128 filters, Kernel size = (3, 3) size, Stride=(1,1), Padding = (1,1)
5 | Conv2d+ReLU | 128 filters, Kernel size = (3, 3) size, Stride=(1,1), Padding = (1,1)
6 | MaxPooling | Kernel size = 2, Stride =2, Padding = 0
7 | Conv2d+ReLU | 256 filters, Kernel size = (3, 3) size, Stride=(1,1), Padding = (1,1)
8 | Conv2d+ReLU | 256 filters, Kernel size = (3, 3) size, Stride=(1,1), Padding = (1,1)
9 | Conv2d+ReLU | 256 filters, Kernel size = (3, 3) size, Stride=(1,1), Padding = (1,1)
10 | MaxPooling | Kernel size = 2, Stride =2, Padding = 0
11 | Conv2d+ReLU | 512 filters, Kernel size = (3, 3) size, Stride=(1,1), Padding = (1,1)
12 | Conv2d+ReLU | 512 filters, Kernel size = (3, 3) size, Stride=(1,1), Padding = (1,1)
13 | Conv2d+ReLU | 512 filters, Kernel size = (3, 3) size, Stride=(1,1), Padding = (1,1)
14 | MaxPooling | Kernel size = 2, Stride =2, Padding = 0
15 | Conv2d+ReLU | 512 filters, Kernel size = (3, 3) size, Stride=(1,1), Padding = (1,1)
16 | Conv2d+ReLU | 512 filters, Kernel size = (3, 3) size, Stride=(1,1), Padding = (1,1)
17 | Conv2d+ReLU | 512 filters, Kernel size = (3, 3) size, Stride=(1,1), Padding = (1,1)
18 | MaxPooling | Kernel size = 2, Stride =2, Padding = 0
19 | Average Pooling | Kerne size = 1, Stride = 1, Padding = 0
20 | Dense+Softmax | 512 units

###  B.3 ResNet [25]

ResNet, short for “Residual Networks,” is a deep convolutional neural network.
The distinguishing characteristic of ResNet is the use of residual blocks. A
residual block consists of a “shortcut” or “skip connection” that bypasses one
or more convolutional layers. This shortcut allows the network to learn
residual functions, the difference between the desired output and the actual
output of the layer. This skip connection enables the training of extremely
deep networks without the vanishing gradient problem. We used ResNet for
CIFAR-10, CIFAR-100, and ImageNet datasets. Depending on the depth of the
network, ResNet is further represented in variants like ResNet-18, ResNet-50,
ResNet-56, ResNet-152. For CIFAR-100, we used a pre-trained ResNet56 model
from PyTorch [25]. It achieved 64.43% accuracy on the test set. For CIFAR-10,
we trained ResNet18 [25] models, which achieved 92.5% accuracy on the test
set. For ImageNet, we used a pre-trained Resnet50 [25] model from the Torch
library, which achieved 76.13% accuracy on the test set.

###  B.4 MobileNet [48]

MobileNet is a family of network architectures designed for efficient deep
learning on mobile and embedded devices by minimizing computational and memory
resources. We use the MobileNetV2 model available in PyTorch. MobileNetV2
introduces ”inverted residuals” layers, which consist of bottleneck layers,
shortcut connections, and linear bottlenecks. Each inverted residual block
includes an expansion layer, which increases the number of channels before the
depth-wise separable convolution. MobileNet relies on depth-wise separable
convolution, which reduces the computational cost by separating the
convolution process into depth-wise and point-wise convolutions. For ImageNet,
we used a pre-trained MobileNet [48] from Torch library, which achieved 70.1%
accuracy on the test set.

We summarize the dataset, architecture, and their performance on the test in
Table XII.

TABLE XII: Datasets and DNN Architectures.

Dataset | Number of classes | Test Accuracy | Architecture
---|---|---|---
MNIST | 10 | 98.17% | LeNet [34]
CIFAR-10 | 10 | 92.5% | ResNet [25]
CIFAR-10 | 10 | 84.91% | VGG [54]
CIFAR-100 | 100 | 64.43% | ResNet [25]
ImageNet | 1000 | 76.13% | ResNet [25]
ImageNet | 1000 | 70.1% | MobileNet [48]
CIC-IDS2017 | 2 | 80.18% | MLP [47]

###  B.5 Network Intrusion Detection

Similar to prior works [18], we use a Multi-Layer Perceptron (MLP) network as
an intrusion detector. It consists of 2 hidden layers of 64 neurons and a
softmax output layer with 2 neurons. Each neuron in the hidden layer uses ReLU
activation. We train the model for 1000 epochs using the Adam optimizer, and
the learning rate of 0.01.

##  Appendix C Adversarial attack

Fast Gradient Sign Attack (FGSM) [22]: FGSM is a computationally efficient
method for finding adversarial examples. It assumes a linear approximation of
the network loss function and finds a perturbation by increasing the local
linear approximation of the loss. The perturbation for an FGSM attack against
a network with loss $J$ and parameters $\theta$, for test sample x, and with
true label $y$ is given by:

$\delta=\epsilon*\textnormal{sign}(\nabla_{x}J(\theta,\textbf{x},y))$ (15)

The strength of the perturbation for each dimension of the input is controlled
by $\epsilon$. We use $\epsilon\in[8/255,16/255,32/255,64/255]$.

Basic Iterative Method (BIM) [33]: BIM is an iterative version of FGSM where
the perturbation is computed multiple times with small steps. The pixel values
of the resulting adversarial image are clipped to ensure they lie within the
$L_{\infty}$ $\epsilon$ neighborhood of the original input image.

$\textbf{x}^{*}_{m+1}=\textbf{x}^{*}_{m}+Clip_{\textbf{x},\epsilon}(\alpha.\textnormal{sign}(\nabla_{x}J(\theta,\textbf{x}^{*}_{m},y))$
(16)

Here, ${0<\alpha<\epsilon}$ controls the $m^{th}$ iteration step size. We use
$\epsilon\in[8/255,16/255,32/255,64/255]$ with $\alpha=\epsilon/10$ and a
fixed number of iterations ($m$) as 10.

Projected Gradient Descent (PGD) [40]: PGD is also an iterative method similar
to BIM; however, unlike BIM, a PGD attack starts from a random perturbation in
the $L_{\infty}$ ball around the input sample.

$\textbf{x}^{*}=\textbf{x}_{n-1}-\textnormal{clip}_{\epsilon}(\alpha~{}\textnormal{sign}(\nabla
xJ(\theta,\textbf{x}_{n-1},y))$ (17)

We use $\epsilon\in[8/255,16/255,32/255,64/255]$ with $\alpha=\epsilon/10$ and
attack steps as $c*\frac{\epsilon}{\alpha}$ where $c$ is a constant. We use
$c=4$, so we apply an attack step of 40 for PGD attack across different
$\epsilon$.

Auto Projected Gradient Descent (Auto-PGD) [14]: Auto-PGD attack is gradient-
based adversarial attack that builds upon the Projected Gradient Descent (PGD)
metohd. It aims to automate the process of finding effective attack
parameters, reducing the need for manual tuning of step size and other
hyperparameters. Auto-PGD uses an alternative loss function for better
performance against defenses that might attempt to mask gradients. We use the
Auto-PGD implementation available in Adversarial Robustness Toolbox (ART)
[42]. An $\epsilon=0.15$ is considered a relatively moderate to strong attack
strength, which we choose in this paper.

Carlini and Wagner (CW) [12]: CW attacks comprise a range of attacks that
follow an optimization framework similar to L-BFGS [61]. However, it replaces
the loss function with an optimization problem involving logits $(Z(.))$
instead of the model prediction.

$g(\textbf{x})=max(max_{(}{i\neq
t}Z(\textbf{x}^{\prime})_{i})-Z(\textbf{x}^{\prime})_{t},-k)$ (18)

Here, $k$ encourages the optimizer to find an adversarial example with high
confidence. For $L_{\infty}$ CW attack, we use $\epsilon=0.15$, 400
iterations, and zero confidence settings. We use a learning rate of 0.01 for
the step size of the optimization.

##  Appendix D Implementation of various detectors

Feature squeezing (FS) [71]: For MNIST, we use bit depth reduction and median
filter, while for CIFAR-10, CIFAR-100, and ImageNet, we use bit depth
reduction, median filter, and non-local means, as suggested by the original
paper.

Magnet [41]: We use the defensive architecture recommended by the original
paper for CIFAR-10 and MNIST. Since the original paper did not perform an
evaluation on CIFAR-100 and ImageNet, we use the defensive architecture of
CIFAR-10, which is designed for three-channel images.

Turning a weakness into a strength (TWS) [28]: We follow the implementation
shared by the authors available on
Github333https://github.com/s-huu/TurningWeaknessIntoStrength. We use Gaussian
noise of $\sigma=0.01$ and $\sigma=0.1$ for CIFAR and ImageNet, as suggested
in the paper. For MNIST and CIFAR-100, we empirically picked noise parameters
($\sigma=0.01$ and $\sigma=0.1$) that resulted in maximum adversarial
detection.

U-LOO [72]: For each data set, we randomly select 2000 benign samples, extract
feature attribution using the Integrated Gradient method, and compute the
inter-quartile range (IQR). IQR is the difference between the 75th percentile
and the 25th percentile among all entries of $IG(x)\in R^{d}$. A sample is
regarded as adversarial if the IQR is larger than the threshold learned from
benign samples.

##  Appendix E Indicator of Failure: PGD at 100 steps

Following Pintor et al.[44] recommendation, we reevaluated detection methods
on PGD adversarial samples crafted with 100 iteration steps. We set the attack
parameter $\epsilon=8/255$. We can observe in Table XIII that most detection
methods do not have significant changes to the detection performance when
compared with prior performance on detecting adversarial samples crafted at
steps of 40.

TABLE XIII: Evaluation of PGD attack with steps = 100

| AUC | FS | MagNet | U-LOO | TWS | PASA
---|---|---|---|---|---|---
MNIST | Before | 0.90$\pm$0.01 | 0.95$\pm$0.03 | 0.99$\pm$0.01 | 0.92$\pm$0.11 | 0.98$\pm$0.01
| After | 0.87$\pm$0.02 | 0.95$\pm$0.03 | 0.99$\pm$0.99 | 0.90$\pm$0.08 | 0.97$\pm$0.02
CIFAR-10 (VGG) | Before | 0.52$\pm$0.01 | 0.59$\pm$0.03 | 0.49$\pm$0.05 | 0.58$\pm$0.02 | 0.74$\pm$0.02
| After | 0.32$\pm$0.02 | 0.58$\pm$0.01 | 0.48$\pm$0.03 | 0.11$\pm$0.02 | 0.75$\pm$0.02
CIFAR-10 (ResNet) | Before | 0.25$\pm$0.01 | 0.57$\pm$0.04 | 0.62$\pm$0.01 | 0.14$\pm$0.01 | 0.83$\pm$0.03
| After | 0.24$\pm$0.01 | 0.56$\pm$0.04 | 0.64$\pm$0.01 | 0.14$\pm$0.01 | 0.83$\pm$0.02
CIFAR-100 | Before | 0.67$\pm$0.03 | 0.56$\pm$0.03 | 0.63$\pm$0.03 | 0.61$\pm$0.01 | 0.60$\pm$0.02
| After | 0.68$\pm$0.02 | 0.55$\pm$0.02 | 0.60$\pm$0.03 | 0.67$\pm$0.02 | 0.58$\pm$0.03
ImageNet (MobileNet) | Before | 0.25$\pm$0.02 | 0.51$\pm$0.01 | 0.59$\pm$0.01 | 0.52$\pm$0.01 | 0.98$\pm$0.02
| After | 0.28$\pm$0.02 | 0.49$\pm$0.02 | 0.51$\pm$0.01 | 0.50$\pm$0.02 | 0.97$\pm$0.01
ImageNet (ResNet) | Before | 0.29$\pm$0.01 | 0.51$\pm$0.01 | 0.59$\pm$0.01 | 0.57$\pm$0.02 | 0.97$\pm$0.01
| After | 0.27$\pm$0.02 | 0.49$\pm$0.01 | 0.59$\pm$0.01 | 0.59$\pm$0.02 | 0.96$\pm$0.01

##  Appendix F Ablation study

Our detection method combines two statistical metrics: prediction sensitivity
(PS) and attribution sensitivity (AS). In this ablation study, we assess the
individual performance of each metric. We summarize the results in Table XIV
and Table XV.

In Table XIV, we can observe that the performance of each metric is almost
equivalent to the combined performance. However, on CIFAR-10, the combination
of AS and PS (PS+AS) consistently outperforms AS and PS individually. AS and
PS exhibit sensitivity to different attack types. For instance, PS is more
effective at detecting adversarial inputs generated by FGSM, while AS excels
in detecting inputs perturbed by PGD and other attacks in CIFAR-10. Combining
both metrics provides a more balanced and robust detection strategy across
various attack types.

TABLE XIV: Adversarial detection performance for MNIST and CIFAR. Here, AS
represents Attribution Sensitivity, and PS represents Prediction Sensitivity.
PS+AS is our proposed detector.

|  | MNIST | CIFAR-10 (ResNet) | CIFAR-10 (VGG)
---|---|---|---|---
Type | Strength | PS | AS | PS+AS | PS | AS | PS+AS | PS | AS | PS+AS
FGSM | 8/255 | 0.97 | 0.84 | 0.97 | 0.77 | 0.87 | 0.88 | 0.54 | 0.55 | 0.62
| 16/255 | 0.98 | 0.85 | 0.98 | 0.96 | 0.97 | 0.99 | 0.67 | 0.51 | 0.71
| 32/255 | 0.98 | 0.86 | 0.98 | 0.99 | 0.99 | 0.99 | 0.87 | 0.69 | 0.87
| 64/255 | 0.99 | 0.85 | 0.98 | 0.98 | 0.99 | 0.99 | 0.94 | 0.89 | 0.94
PGD | 8/255 | 0.98 | 0.85 | 0.98 | 0.13 | 0.88 | 0.85 | 0.51 | 0.57 | 0.61
| 16/255 | 0.98 | 0.87 | 0.97 | 0.30 | 0.94 | 0.92 | 0.58 | 0.51 | 0.61
| 32/255 | 0.98 | 0.89 | 0.98 | 0.22 | 0.97 | 0.97 | 0.74 | 0.51 | 0.73
| 64/255 | 0.98 | 0.89 | 0.98 | 0.08 | 0.98 | 0.98 | 0.89 | 0.81 | 0.90
BIM | 8/255 | 0.56 | 0.46 | 0.58 | 0.09 | 0.83 | 0.77 | 0.49 | 0.54 | 0.58
| 16/255 | 0.55 | 0.47 | 0.52 | 0.09 | 0.89 | 0.84 | 0.52 | 0.53 | 0.59
| 32/255 | 0.51 | 0.42 | 0.53 | 0.16 | 0.94 | 0.93 | 0.55 | 0.47 | 0.60
| 64/255 | 0.54 | 0.39 | 0.54 | 0.16 | 0.97 | 0.98 | 0.70 | 0.49 | 0.68
CW | 0.15 | 0.38 | 0.34 | 0.38 | 0.95 | 0.97 | 0.98 | 0.79 | 0.61 | 0.80

In Table XV, both attribution sensitivity and prediction sensitivity have high
detection performance in detecting FGSM attacks. However, with PGD, BIM, and
CW, individual metrics have weaker performances. The combination of AS and PS
(PS+AS) consistently outperforms the individual metrics (AS and PS). The
detector’s performance generally degrades as the strength of adversarial
attacks increases. This degradation is more pronounced in cases where the AS
and PS metrics are employed individually, noticeable with ImageNet.

TABLE XV: Adversarial detection performance for CIFAR-100 and ImageNet. Here,
AS represents Attribution Sensitivity, and PS represents Prediction
Sensitivity. PS+AS is our proposed detector.

|  | CIFAR 100 | ImageNet (Mobilenet) | ImageNet (ResNet)
---|---|---|---|---
Type | Strength | PS | AS | PS+AS | PS | AS | PS+AS | PS | AS | PS+AS
FGSM | 8/255 | 0.68 | 0.74 | 0.82 | 0.62 | 0.81 | 0.82 | 0.48 | 0.60 | 0.65
| 16/255 | 0.78 | 0.89 | 0.95 | 0.68 | 0.91 | 0.91 | 0.55 | 0.73 | 0.75
| 32/255 | 0.89 | 0.89 | 0.97 | 0.74 | 0.96 | 0.96 | 0.61 | 0.87 | 0.87