11institutetext: All authors 22institutetext: Univ Rennes, INSA Rennes, CNRS,
IETR - UMR 6164, F-35000 Rennes, France
33institutetext: Ahmed Aldahdooh - 33email<EMAIL_ADDRESS>
44institutetext: Wassim Hamidouche - 44email<EMAIL_ADDRESS>
55institutetext: Olivier Deforges - 55email<EMAIL_ADDRESS>
# Revisiting Model’s Uncertainty and Confidences for Adversarial Example
Detection

Ahmed Aldahdooh  Wassim Hamidouche  Olivier Déforges

(Received: date / Accepted: date)

###### Abstract

Security-sensitive applications that rely on Deep Neural Networks are
vulnerable to small perturbations that are crafted to generate Adversarial
Examples. The AEs are imperceptible to humans and cause DNN to misclassify
them. Many defense and detection techniques have been proposed. Model’s
confidences and Dropout, as a popular way to estimate the model’s uncertainty,
have been used for AE detection but they showed limited success against black-
and gray-box attacks. Moreover, the state-of-the-art detection techniques have
been designed for specific attacks or broken by others, need knowledge about
the attacks, are not consistent, increase model parameters overhead, are time-
consuming, or have latency in inference time. To trade off these factors, we
revisit the model’s uncertainty and confidences and propose a novel
unsupervised ensemble AE detection mechanism that 1) uses the uncertainty
method called SelectiveNet, 2) processes model layers outputs, i.e. feature
maps, to generate new confidence probabilities. The detection method is called
Selective and Feature based Adversarial Detection (SFAD). Experimental results
show that the proposed approach achieves better performance against black- and
gray-box attacks than the state-of-the-art methods and achieves comparable
performance against white-box attacks. Moreover, results show that SFAD is
fully robust against High Confidence Attacks for MNIST and partially robust
for CIFAR10 datasets. 111The source code is available in
https://aldahdooh.github.io/SFAD/.

###### Keywords:

Adversarial examples Adversarial attacks Adversarial example detection Deep
learning robustness

##  1 Introduction

Deep learning (DL) has achieved remarkable advances in different fields in
human life especially computer vision tasks like object detection, image
classification krizhevsky2012imagenet ; simonyan2014very ; ren2015faster ,
surveillance lecun2015deep , and medical imaging shen2017deep . Despite that,
it is found that DL models are vulnerable to adversaries szegedy2013intriguing
; goodfellow2014explaining . In image classification models, for instance,
adversaries can generate AEs, by adding small perturbations to an input image
that are imperceptible to humans and devices, that cause DL models to
misclassify the input images. Such potential threat affects security-critical
DL-based applications guo2018lemna  such as self-driving cars.

Figure 1: (a) High-level architecture of the uncertainty/confidence-based
detectors. (b) High-level architecture of the uncertainty/confidence-based
ensemble detectors. The input sample is passed to the CNN model to do class
prediction. The detector, i.e. the uncertainty method, estimates the
uncertainty of the input samples using model hidden layers. Using a predefined
threshold, the input sample is not adversarial if the uncertainty exceeds the
predefined threshold.

Adversaries can generate AEs for white-box, black-box, and gray-box attacks
akhtar2018threat ; hao2020adversarial . In white-box attack scenario, the
adversary knows everything about the DL-model including inputs, outputs,
architecture, and weights of the model. Hence, he is guided by the model
gradient to generate AE by solving an optimization problem
goodfellow2014explaining ; kurakin2016adversarial ; moosavi2016deepfool ;
carlini2017towards ; madry2017towards . In black-box scenario, the adversary
knows nothing about the model but he leverages the transferability property
papernot2016transferability  of AEs and the input content. By sending queries
to the model, the adversary can craft small perturbations that are harmonious
with the input image chen2017zoo ; engstrom2019exploring ; su2019one ;
kotyan2019adversarial . In the gray-box scenario, the adversary knows only the
input and the output of the model and hence, he tries to substitute the
original model with an approximated model and then uses its gradient as in
white-box scenario to generate AEs.

Researchers pay attention to this threat and several emerging methods have
been proposed to detect or to defend against AEs. More details about defense
and detection methods can be found in Section 2.

DL model’s uncertainty is one of the main methods that has been used to
determine whether an input sample belongs to the training manifold. The
uncertainty is usually measured by adding randomness to the model using
Dropout techniquegal2016dropout ; srivastava2014dropout . It is found that
clean sample predictions do not change, when randomness is added, while it
changes for AEs. Feinman et al.feinman2017detecting  proposed Bayesian
Uncertainty (BU) metric that used Monte Carlo dropout to estimate the
uncertainty to detect AEs that are near the classes manifold, while Smith et
al.smith2018understanding ; sheikholeslami2019minimum  used mutual information
method to estimate the uncertainty. The prediction risk of these methods is
higher compared to the recent uncertainty method, SelectiveNet selective2019 ,
that is used in this work. On the other hand, it was shown in
hendrycks2016baseline  that predicted class probabilities, i.e. model’s
confidence, of in-of-distribution samples are higher than of out-of-
distribution. Model’s confidence was used in hendrycks2016baseline ;
aigrain2019detecting ; monteiro2019generalizable ; sotgiu2020deep  to
implement AE detectors. Uncertainty and confidence based detectors showed
limited success against black- and gray-box attacks. Uncertainty and
confidence based detectors are usually threshold-based detectors as shown in
Figure 1(a). To enhance detectors’ performance, one recommendation goes to the
direction of providing ensemble detection methods, as shown in Figure 1(b).
Although state-of-the-art detectors achieve promising results, they may have
one or more limitation(s); not performing well with some known attacks
xu2017feature , broken by attackers athalye2018obfuscated ; carlini2017magnet
, performance of baseline detectors is not consistent bulusu2020anomalous ,
increase the model parameters overhead lust2020gran , time consuming ma2019nic
, or introduce latency gao2020backdoor  in the inference time.

In this paper and in order to mitigate the aforementioned limitations, we
revisit the model’s uncertainty and confidence to propose a novel ensemble AE
detector that hasn’t had any knowledge of AEs, i.e. unsupervised detector, as
shown in Figure 2. The proposed method has the following attributes; 1) it
investigates SelectiveNet capability in detecting adversarial examples since
it measures the uncertainty with less risk. According to the author’s
knowledge, the SlelectiveNet selective2019  is not used in adversarial attacks
detection models. 2) Unlike other detectors melis2017deep ; sotgiu2020deep ;
lu2017safetynet , the proposed method uses the model’s last $N$-layers
outputs, i.e. feature maps, to build $N$-CNNs $\mathcal{M}$ that have
different processing blocks like up/down sampling, auto-encoders
wang2017residual ; liu2019end , noise addition lecuyer2019certified ;
liu2018towards ; liu2020does , and bottleneck layer addition he2016deep  that
make the representative data of last layers more unique to the input data
distribution to yield better model’s confidence. To reduce the effect of
white-box attacks, the output of $\mathcal{M}$ is transferred/distilled to
build the last CNN $\mathcal{S}$. 3) The proposed model ensembles the proposed
detection techniques to provide the final detector. This step has a great
impact in reducing the adversary’s capability to craft perturbations that can
fool the detector, since he has to fool every detection technique. We name the
proposed method as Selective and Feature based Adversarial Detection (SFAD).
The high-level architecture of the SFAD is illustrated in Figure 1.

A prototype of SFAD is tested under white-box, black-box and gray-box attacks
on MNIST lecun1998gradient , and CIFAR10 krizhevsky2009learning . Under the
white-box attacks, the experimental results show that SFAD can detect AEs at
least with accuracy of 89.8% (many with 99%) for all tested attacks except for
the Projected Gradient Descent (PGD) attack madry2017towards  with at least
65% detection accuracy in average. For black- and gray-box attacks, SFAD shows
better performance than other tested detectors. Finally, SFAD is tested under
the HCA carlini2017adversarial  and it shows that it is fully(100%) and
partially(57.76%) robust on MNIST and CIFAR10 respectively. SFAD sets the
thresholds to reject 10% of clean images. Moreover, comparisons with state-of-
the-art methods are presented. Hence, our key contributions are:

  * •

We propose a novel unsupervised ensemble model for AE detection. Ensemble
detection makes SFAD more robust against white-box and adaptive attacks.

  * •

We investigate the SelectiveNet’s, as an uncertainty model, capability in
detecting AEs.

  * •

We show that, by processing the feature maps of last $N$-layers, we can build
classifiers for better confidence distribution. We provide an ablation
experiments to study the impact of the feature processing blocks.

  * •

SFAD prototype proves the concept of the approach and lets the door open in
future to find the best $N$ layers and the best $N$(or $M$) CNNs combinations
to build the detector’s classifiers.

  * •

Unlike tested state-of-the-art detectors, SFAD prototype shows better
performance under gray- and black-box attacks. SFAD prototype shows that it is
fully robust on MNIST and partially robust on CIFAR10 when attacked with HCAs.
For instance, Local Intrinsic Dimensionality (LID) method ma2018characterizing
reported very high detection accuracy on the tested attacks, but fails on HCAs
athalye2018obfuscated ; carlini2017adversarial .

##  2 Related work

###  2.1 Detection methods

Defense techniques like adversarial training goodfellow2014explaining ;
madry2017towards ; xie2020smooth ; tramer2017ensemble , feature denoising
xie2019feature ; borkar2020defending ; liao2018defense , pre-processing
mustafa2019image ; prakash2018deflecting , and gradient masking
papernot2016distillation ; papernot2017practical ; gu2014towards ;
nayebi2017biologically  try to make the model robust against the attacks and
let the model correctly classify the AEs. On the other hand, detection methods
provide adversarial status for the input image. Detection techniques can be
classified according to the presence of AEs in the detector learning process
into supervised and unsupervised techniques bulusu2020anomalous . In
supervised detection, detectors include AEs in the learning process. Many
approaches exist in the literature. In the feature-based approach
nguyen2015deep ; grosse2017statistical ; metzen2017detecting ; lu2017safetynet
, detectors use clean and AEs inputs to built their classifier models from
scratch by using raw image data or by using the representative layers’ outputs
of a DNN model. For instance, in lu2017safetynet , the detector quantizes the
last ReLU activation layer of the model and builds a binary Support Vector
Machine (SVM) with Radial Basis Function (RBF) classifier. As reported in
lu2017safetynet , this detector is not robust enough and is not tested against
strong attacks like Carlini-Wagner (CW) attacks. While the work in
grosse2017statistical  added a new adversarial class to the Neural Network
(NN) model and train the model from scratch with clean and adversarial inputs.
This architecture reduces the model accuracy grosse2017statistical . In the
statistical-based approach feinman2017detecting ; ma2018characterizing ,
detectors perform statistical measurement to define the separation between
clean and adversarial inputs. In feinman2017detecting , Kernel Density (KD)
estimation, BU, or combined models are introduced. Kernel-density feature is
extracted from clean and AEs in order to identify AEs that are far away from
data manifold while Bayesian uncertainty feature identifies the AEs that lie
in low-confidence regions of the input space. LID method is introduced in
ma2018characterizing  as a distance distribution of the input sample to its
neighbors to assess the space-filling capability of the region surrounding
that input sample. The works in carlini2017adversarial ; athalye2018obfuscated
showed that these methods can be broken. Finally, the network invariant
approach metzen2017detecting ; eniser2020raid  learns the differences in
neuron activation values between clean input samples and AEs to build a binary
NN detector. The main limitation of this approach is that it requires prior
knowledge about the attacks and hence it might not be robust against new or
unknown attacks.

On the other hand, in unsupervised detection, detectors are trained with clean
images only to identify the AEs. It is also known as prediction inconsistency
models since it depends on the fact that AEs might not fool every NN model.
That’s because the input feature space is almost limited and the adversary
always takes that as an advantage to generate the AEs. Hence, unsupervised
detectors try to reduce this limited input feature space available to
adversaries. Many approaches have been presented in the literature. The
Feature Squeezing (FS) approach xu2017feature  measures the distance between
the predictions of the input and the same input after squeezing. The input
will be adversarial if the distance exceeds a threshold. The work in
xu2017feature  squeezes out unnecessary input features by reducing the color
bit depth of each pixel and by spatial smoothing of adversarial inputs. As
reported in xu2017feature , FS is not performing well with some known attacks
like Fast Gradient Sign Attack (FGSM). Instead of squeezing, denoising based
approach, like MagNet meng2017magnet , measures the distances between the
predictions of input samples and denoised/filtered input samples. It was found
in carlini2017magnet ; liao2018defense  that MagNet can be broken and do not
scale to large images. Recently, a network invariant approach was introduced
ma2019nic . They proposed a Neural-network Invariant Checking (NIC) method
that builds a set of models for individual layers to describe the provenance
and the activation value distribution channels. It was observed that AEs
affect these channels. The provenance channel describes the instability of
activated neurons set in the next layer when small changes are present in the
input sample while the activation value distribution channel describes the
changes with the activation values of a layer. The reported performance of
this method showed its superiority against other state-of-the-art models but
other works reported that the baseline NIC’s detectors are not consistent
bulusu2020anomalous , increase model parameters overhead lust2020gran , are
time consuming ma2019nic , and increase the latency in the inference time
gao2020backdoor .

Uncertainty-based detectors. Following the observation that the prediction of
clean image remains correct with many dropouts, while the prediction of AE
changes. Feinman et al.feinman2017detecting  proposed BU metric. BU uses Monte
Carlo dropout to estimate the uncertainty, to detect those AEs that are near
the classes manifold, while Smith et al.smith2018understanding  used mutual
information method for such a task. In sheikholeslami2019minimum ,
Sheikholeslami et al.proposed an unsupervised detection method that provides a
layer-wise minimum variance solver to estimate model’s uncertainty for in-
distribution training data. Then, a mutual information based threshold is
identified.

Confidence-based detectors. Aigrain et al.aigrain2019detecting  built a simple
NN detector that uses the model’s logits of clean and AEs to build a binary
classifier. Inspired by the hypothesis of that, for a given perturbed image,
different models yield different confidences, Monteiro et
al.monteiro2019generalizable  proposed a bi-model mismatch detection method.
The detector is a binary RBF-SVM classifier that takes as input the output of
two classifiers of clean and AEs. On the other hand, Sotgiu et al.proposed an
unsupervised detection method that uses the last $N$ representative layers’
outputs of the classifier to built three SVM classifiers with RBF kernel. The
confidence probabilities of the SVMs are combined to build the last SVM-RBF
classifier. Then, a threshold is identified to reject inputs that have less
maximum confidence probability.

###  2.2 SelectiveNet as an uncertainty model

Let $\mathcal{X}$ be an input space, e.g. images, and $\mathcal{Y}$ a label
space. Let $\mathbb{P}(X,Y)$ be the data distribution over
$\mathcal{X}\times\mathcal{Y}$. A model, $f:X\rightarrow Y$, is called a
prediction function, $\ell:Y\times Y\rightarrow\mathbb{R}^{2}$ is a given loss
function. Given a labeled set
$S_{k}={(x_{i},y_{i})}_{i=1}^{k}\subseteq(\mathcal{X}\times\mathcal{Y})^{k}$
sampled i.i.d. from $\mathbb{P}(X,Y)$, where $k$ is the number of training
samples. The true risk of the prediction function $f$ w.r.t. $\mathbb{P}$ is
$R(f)\triangleq\mathbb{E}_{\mathbb{P}(X,Y)}[\ell(f(x),y)]$ while the empirical
risk of the prediction function $f$ is $\hat{r}(f\mid
S_{k})\triangleq\frac{1}{k}\sum_{i=1}^{k}\ell(f(x_{i}),y_{i})$.

Here, we briefly demonstrate the SelectiveNet as stated in selective2019 . The
selective model is a pair $(f,g)$, where $f$ is a prediction function, and
$g:\mathcal{X}\rightarrow\\{0,1\\}$ is a binary selection function for $f$,

$(f,g)(x)\triangleq\begin{cases}f(x),&\text{if }g(x)=1;\\\ \text{don't
know},&\text{if }g(x)=0.\end{cases}$ (1)

A soft selection function can also be considered, where
$g:\mathcal{X}\rightarrow[0,1]$, hence, the value of $(f,g)(x)$ is calculated
with the help of a threshold $\tau$ as expressed in the following equation

$(f,g)(x)\triangleq\begin{cases}f(x),&\text{if }g(x)\geq\tau;\\\ \text{don't
know},&\text{if }g(x)<\tau.\end{cases}$ (2)

The performance of a selective model is calculated using coverage and risk.
The true coverage is defined to be the probability mass of the non-rejected
region in $\mathcal{X}$ and calculated as

$\phi(g)\triangleq E_{P}[g(x)],$ (3)

while the empirical coverage is calculated as

$\hat{\phi}(g\mid S_{k})\triangleq\frac{1}{k}\sum_{i=1}^{k}g(x_{i})$ (4)

The true selective risk of $(f,g)$ is

$R(f,g)\triangleq\frac{E_{P}[\ell(f(x),y)g(x)]}{\phi(g)},$ (5)

while the empirical selective risk is calculated for any given labeled set
$S_{k}$ as

$\hat{r}(f,g\mid
S_{k})\triangleq\frac{\frac{1}{k}\sum_{i=1}^{k}\ell(f(x_{i}),y_{i})g(x_{i})}{\hat{\phi}(g\mid
S_{k})}.$ (6)

Finally, for a given coverage rate $0<c\leq 1$ and $\Theta$, a set of
parameters for a given deep network architecture for $f$ and $g$, the
optimization problem of the selective model is expressed as:

$\displaystyle\begin{aligned}
\theta^{\ast}=\operatorname*{arg\,min}_{\theta\in\Theta}(R(f_{\theta},{g}_{\theta}))\\\
\textit{s.t. }\phi({g}_{\theta})\geq c,\end{aligned}$ (7)

and can be solved using the Interior Point Method (IPM) potra2000interior  to
enforce the coverage constraint. That yields to unconstrained loss objective
function over samples in $S_{k}$,

$\displaystyle\begin{aligned}
\mathcal{L}_{(f,g)}\triangleq\hat{r}_{\ell}(f,g\mid
S_{k})+\lambda\Psi(c-\hat{\phi}(g\mid S_{k}))\\\
\Psi(a)\triangleq\max(0,a)^{2},\end{aligned}$ (8)

where $c$ is the target coverage, $\lambda$ is a hyper-parameter controlling
the relative importance of the constraint, and $\Psi$ is a quadratic penalty
function. As a result, SelectiveNet is a selective model $(f,g)$ that
optimizes both $f(x)$ and $g(x)$ in a single model in a multi-task setting as
depicted in Figure 1(c). For more details about the SelectiveNet model,
readers are advised to read selective2019 .

Figure 2: (a) Detector architecture. $N$-last representative output of DNN is
used to build $N$ Selective Adversarial Example classifiers. The confidence
output, i.e. pred. probabilities, of the $N$ classifiers is concatenated to be
as input for Selective Knowledge Transfer classifier. (b) Feature maps
processing blocks. (c) SelectiveNet architectureselective2019  (d) Detection
Process: Selective probabilities ($P_{s}^{m_{j}}$ and $P_{s}^{\mathcal{S}}$)
are used in the Selective detection process. Confidence/Prediction
probabilities, ($P_{c}^{m_{j}}$ and $P_{c}^{\mathcal{S}}$), are used in the
confidence detection process. Confidence/Prediction probabilities,
($P_{c}^{\mathcal{F}}$ and $P_{c}^{\mathcal{S}}$), are used in the mismatch
detection process. The total detection is the ensemble of the three detection
modules.

##  3  Selective and Feature based Adversarial Detection (SFAD) Method

###  3.1 Detector design

It is believed that the last $N$ layers in the DNN $\mathcal{F}$ have
potentials in detecting and rejecting AEs melis2017deep ; sotgiu2020deep . In
bendale2016towards  and melis2017deep , only the last layer ($N=1$) is
utilized to detect AEs. At this very high level of presentation, AEs are
indistinguishable from samples of the target class. This observation is
enhanced when Deep Neural Rejection (DNR) sotgiu2020deep  used the last three
layers to build SVM with RBF kernel based classifiers. Unlike other works, in
this work, 1) feature maps of the last layers $Z_{j}$, where
$j=\\{1,2\dots,N\\}$, are processed. In the aforementioned methods, the
representatives of the last layers are not processed and basically the
detectors represent another approximation of the baseline classifier which is
considered as a weak point. 2) Multi-Task Learning (MTL) is used via the
SelctiveNet. MTL has an advantage of combining related tasks with one or more
loss function(s) and it does better generalization especially with the help of
the auxiliary functions. For more details about MTL, please refer to these
recent review papers ruder2017overview ; vandenhende2020revisiting .

In this section, the Selective and Feature based Adversarial Detection (SFAD)
method is demonstrated. As depicted in Figure 2(a), SFAD consists of two main
blocks; the selective AEs classifiers $\mathcal{M}$ block (in blue), where
$\mathcal{M}=\\{m_{j}\\}_{j=1}^{N}$, and the selective knowledge transfer
classifier $\mathcal{S}$ block (in orange). The output of $\mathcal{F}$,
$\mathcal{M}$, and $\mathcal{S}$ blocks, i.e. model’s uncertainties and
confidences, are used in the detection process, as depicted in Figure 1(d).

###  3.2 Selective AEs classifiers block

As shown in Figure 1(a), the aim of $\mathcal{M}$ block is to build $N$
individual classifiers, $\mathcal{M}=\\{m_{j}\\}_{j=1}^{N}$. It was shown that
perturbation propagation becomes clear when the DNN model goes deeper, hence,
using $N$-last layers have potential in identifying the AEs. Unlike works in
melis2017deep ; sotgiu2020deep , we process the representative last
$N$-layer(s) outputs $Z_{j}$ in different ways in order to make clean input
features more unique. This will limit the feature space that the adversary
uses to craft the AEs xu2017feature ; meng2017magnet . Moreover, each of the
last $N$-layer output has its own feature space which makes each of the $N$
classifiers $m_{j}$ to be trained with different feature space. Hence,
combining and increasing the number of $N$ will enhance the detection process.

For simplicity and as recommended in sotgiu2020deep , we set $N=3$ in the
implemented prototype and hence, each individual layer output is assigned to a
classifier as shown in Figure 1(a). Let the last $N$ layers’ outputs $z_{ji}$
of $x_{i}$ from $S_{k}$ are $z_{1i}$, $z_{2i}$, and $z_{Ni}$, respectively,
where, $j=\\{1,2,\dots,N\\}$. $z_{ji}$ are individually the inputs of the
$m_{j}$ classifier.

The outputs of the $m_{j}$ classifier are denoted as $m_{ji}(z_{ji})$. Let
$\mathcal{Y^{\prime}}=\mathcal{Y}+1$ be a label space of $m_{j}$, where the
extra label is denoted for the selective status, hence $m_{j}$ represents a
function $m_{j}:Z_{j}\rightarrow Y^{\prime}$ on a distribution
$\mathbb{P}(Z_{j},Y^{\prime})$ over $\mathcal{Z}\times\mathcal{Y^{\prime}}$.
We refer to the selective probability of $m_{j}$ as $P_{s}^{m_{j}}$ and the
confidence probabilities of $m_{j}$ as $P_{c}^{m_{j}}$. $m_{j}$ optimizes the
overall loss function

$\mathcal{L}_{m_{j}}=\alpha\mathcal{L}_{(m_{j},g_{m_{j}})}+(1-\alpha)\mathcal{L}_{h_{m_{j}}}\text{
, where }\alpha=0.5,$ (9)

where $\mathcal{L}_{(m_{j},g_{m_{j}})}$ is the selective loss function of
$m_{j}$, and $\mathcal{L}_{h_{m_{j}}}$ is the auxiliary loss function of
$m_{j}$ and are calculated as following:

$\begin{gathered}\mathcal{L}_{(m_{j},g_{m_{j}})}\triangleq\hat{r}_{\ell}(m_{j},g_{m_{j}}\mid
S_{k})+\lambda\,\Psi(c-\hat{\phi}(g_{m_{j}}\mid S_{k})),\\\
\Psi(a)\triangleq\max(0,a)^{2},\end{gathered}$

$\mathcal{L}_{h_{m_{j}}}=\hat{r}(h_{m_{j}}\mid
S_{k})=\frac{1}{k}\sum_{i=1}^{k}\ell(h_{m}(z_{ji}),y_{i}).$

Studying the value of $\alpha$ is out of the paper scope, but other task
balancing methods, may be applied like, uncertainty kendall2018multi ,
GradNorm chen2017gradnorm , DWA liu2019end , DTP guo2018dynamic , and MGDA
sener2018multi .

###  3.3 Selective knowledge transfer block

The block $\mathcal{S}$ aims at building selective knowledge transfer
classifier. It concatenates the confidence values of $Y$ classes of the
$\mathcal{M}$ classifiers. The idea behind the block $\mathcal{S}$ is that
each set of its input is considered as a special feature of the clean input.
Hence, we transfer this knowledge, $m_{j}$ confidence probabilities, of clean
inputs to the classifier. Besides, in the inference time, we believe that AE
will generate a different distribution of the confidence values and if the AE
is able to fool one $m_{j}$, it may not fool the others.

As Figure 2(a) shows, the confidence probabilities of $m_{j}$ classifiers are
concatenated to be as an input $Q=$ $concat(P_{c}^{m_{1}},$
$P_{c}^{m_{2}},\dots,P_{c}^{m_{N}})$ for the selective knowledge transfer
block $\mathcal{S}$. The $\mathcal{S}$ classifier consists of one or more
dense layer(s) and yields the selective probability of $\mathcal{S}$ as
$P_{s}^{\mathcal{S}}$ and the confidence probabilities of $\mathcal{S}$ as
$P_{c}^{\mathcal{S}}$. $\mathcal{S}$ represents a function
$\mathcal{S}:Q\rightarrow Y^{\prime}$ on a distribution
$\mathbb{P}(Q,Y^{\prime})$ over $\mathcal{Q}\times\mathcal{Y^{\prime}}$.
Hence, it optimizes the following loss function

$\mathcal{L}_{\mathcal{S}}=\alpha\mathcal{L}_{(\mathcal{S},g_{\mathcal{S}})}+(1-\alpha)\mathcal{L}_{h_{\mathcal{S}}}\text{
, where }\alpha=0.5,$ (10)

where $\mathcal{L}_{(\mathcal{S},g_{\mathcal{S}})}$ is the selective loss
function of $\mathcal{S}$, and $\mathcal{L}_{h_{\mathcal{S}}}$ is the
auxiliary loss function of $\mathcal{S}$ and are calculated as following:

$\displaystyle\begin{gathered}\mathcal{L}_{(\mathcal{S},g_{\mathcal{S}})}\triangleq\hat{r}_{\ell}(\mathcal{S},g_{\mathcal{S}}\mid
S_{k})+\lambda\Psi(c-\hat{\phi}(g_{\mathcal{S}}\mid S_{k})),\\\
\Psi(a)\triangleq\max(0,a)^{2}.\end{gathered}$

$\mathcal{L}_{h_{\mathcal{S}}}=\hat{r}(h_{\mathcal{S}}\mid
S_{k})=\frac{1}{k}\sum_{i=1}^{k}\ell(h_{\mathcal{S}}(q_{i}),y_{i}).$

###  3.4 Feature maps processing

As depicted in Figure 2(b) each selective classifier consists of different
processing blocks; auto-encoder block, up/down-sampling block, bottleneck
block, and noise block. These blocks aim at giving distinguishable features
for input samples to let the detector recognize the AEs efficiently.

Auto-encoder. Auto-encoders are widely used as a reconstruction tool and its
loss is used as a score for different tasks. For instance, it is used in the
detection process of AEs in meng2017magnet . It is believed that AEs gave
higher reconstruction loss than clear images. This process is a.k.a attention
mechanism zhang2019scan ; zhang2020auxiliary  and it is used to focus on
better representation of input features especially on the shallow classifiers.

Up/down-sampling. Up sampling and down sampling are used in different deep
classifiers wang2017residual ; liu2019end . The aim of down sampling, a.k.a
pooling layers in NN, is to gather the global information of the input signal.
Hence, if we consider the clean input signal as a signal that has global
information and then we expand the global information by bi-linear up sampling
and then down sample by average pooling, we will measure the ability of global
information reconstruction of the input signal. Besides, this process can be
seen as a use case of the reconstruction process.

Noise. Adding noise has a potential impact in making NN more robust against
AEs and it has been used in many defense methods lecuyer2019certified ;
liu2018towards ; liu2020does . In this work, we add a branch in the classifier
that adds small Gaussian noise to the input signal before and after the auto-
encoder block. Then, the noised and clean input features are concatenated
before the bottleneck block.

Bottleneck. The bottleneck block he2016deep  consists of three convolutional
layers; 1$\times$1, 3$\times$3, and 1$\times$1 convolutional layers. The
bottleneck name came from the fact that the 3$\times$3 convolutional layer is
left as a bottleneck between 1$\times$1 convolutional layers. It is mainly
designed for efficiency purposes but according to zhang2019scan ;
zhang2019your  it is very effective in building shallow classifiers which
helps having better representation of input signal.

###  3.5 Detection process

As depicted in Figure 2(d), the output of baseline model
$P_{c}^{\mathcal{F}}$, the outputs of $\mathcal{M}$ block, $P_{s}^{m_{j}}$ and
$P_{c}^{m_{j}}$, and the output of $\mathcal{S}$ block, $P_{s}^{\mathcal{S}}$
and $P_{c}^{\mathcal{S}}$, are used in the ensemble detection process.

  1. 1.

Thresholds:  after finishing the training of the detectors’ classifiers, the
following thresholds are identified:

     * •

confidence threshold $th_{c}^{m_{j}}$ for each selective AEs classifier
$m_{j}$.

     * •

confidence threshold $th_{c}^{\mathcal{S}}$ for the $\mathcal{S}$ classifier.

     * •

the final confidence threshold value

$th_{c}=\max(th_{c}^{\mathcal{S}},th_{c}^{m_{1}},th_{c}^{m_{2}},...,th_{c}^{m_{N}})$

     * •

selective threshold $th_{s}^{m_{j}}$ for each selective AEs classifier
$m_{j}$.

     * •

selective threshold $th_{s}^{\mathcal{S}}$ for the $\mathcal{S}$ classifier.

Following the steps in sotgiu2020deep , we select our thresholds at a level
when 10% (at most) of clean inputs can be rejected by the ensemble detection.

  2. 2.

Confidence detection: is set to $1$ if $max(P_{c}^{\mathcal{S}})<th_{c}$ and
is set to $0$ otherwise, where 1 means adversarial input.

  3. 3.

Selective detection: is set to $1$ if
$P_{s}^{\mathcal{S}}<th_{s}^{\mathcal{S}}$ or $P_{s}^{m_{1}}<th_{s}^{m_{1}}$
or … or  $P_{s}^{m_{N}}<th_{s}^{m_{N}}$ and is set to $0$ otherwise.

  4. 4.

Mismatch detection: is set to $1$ if $\operatorname*{argmax}$
$(P_{c}^{\mathcal{S}})\neq\operatorname*{argmax}$ $(P_{c}^{\mathcal{F}})$ and
is set to $0$ otherwise.

  5. 5.

Ensemble detection: The input sample is adversarial if it is detected in
confidence, selective, or mismatch detection process.

##  4 Experimental settings

###  4.1 Datasets

The proposed prototype is evaluated on CNN models trained with two popular
datasets; MNIST lecun1998gradient  and krizhevsky2009learning  CIFAR10.

MNIST is hand-written digit recognition dataset with 70000 images (60000 for
training and 10000 for testing) and ten classes and CIFAR10 is an object
recognition dataset with 60000 images (50000 for training and 10000 for
testing) ten classes.

###  4.2 Baseline classifiers

For the baseline models, two CNN models are trained; one for MNIST and one for
CIFAR10. For MNIST, we trained 6-layer CNN with 98.73% accuracy while for
CIFAR10 we trained 8-layer CNN with 89.11% accuracy. The classifier’s
architectures for MNIST and CIFAR10 are shown in Table 1 and Table 2,
respectively.

Table 1: MNIST baseline classifier architecture Layer |  Description
---|---
Conv2D + ReLU |  32 filters ($3\times 3$)
Conv2D + ReLU + Max Pooling($2\times 2$) |  32 filters ($3\times 3$)
Conv2D + ReLU |  64 filters ($3\times 3$)
Conv2D + ReLU + Max Pooling($2\times 2$) |  64 filters ($3\times 3$)
Dense + ReLU + Dropout ($p=0.3$) |  256 units
Dense + ReLU |  256 units
Softmax |  10 classes
Table 2: CIFAR10 baseline classifier architecture Layer |  Description
---|---
Conv2D + BatchNorm + ReLU |  64 filters ($3\times 3$)
Conv2D + BatchNorm + ReLU + Max Pooling($2\times 2$) + Dropout ($p=0.1$) |  64 filters ($3\times 3$)
Conv2D + BatchNorm + ReLU |  128 filters ($3\times 3$)
Conv2D + BatchNorm + ReLU + Max Pooling($2\times 2$) + Dropout ($p=0.2$) |  128 filters ($3\times 3$)
Conv2D + BatchNorm + ReLU |  256 filters ($3\times 3$)
Conv2D + BatchNorm + ReLU + Max Pooling($2\times 2$) + Dropout ($p=0.3$) |  256 filters ($3\times 3$)
Conv2D + BatchNorm + ReLU + Max Pooling($2\times 2$) + Dropout ($p=0.4$) |  512 filters ($3\times 3$)
Dense |  512 units
Softmax |  10 classes

In order to evaluate the proposed prototypes against gray-box attacks, we
consider that the adversaries know the training dataset and the model outputs
and do not know the baseline model architectures. Hence, Table 3 and Table 4
show the two alternative architectures for MNIST and CIFAR10 classifiers. For
MNIST, the classification accuracies are 98.37% and 98.69% for Model #2 and
Model #3, respectively. While for CIFAR10, the classification accuracies are
86.93% and 88.38% for Model #2 and Model #3 respectively.

Table 3: MNIST classifiers architectures for gray-box setting Model |  Layer |  Description
---|---|---
Model #2 |  Conv2D + BatchNorm + ReLU |  64 filters ($3\times 3$)
Conv2D + BatchNorm + ReLU + Max Pooling($2\times 2$) + Dropout ($p=0.5$) |  64 filters ($3\times 3$)
Dense + BatchNorm + ReLU + Dropout ($p=0.5$) |  128 units
Softmax |  10 classes
Model #3 |  Conv2D + ReLU + Max Pooling($2\times 2$) |  32 filters ($3\times 3$)
Conv2D + ReLU + Max Pooling($2\times 2$) |  64 filters ($3\times 3$)
Dense + ReLU + Dropout ($p=0.5$) |  256 units
Dense + ReLU |  256 units
Softmax |  10 classes
Table 4: CIFAR10 classifiers architectures for gray-box setting Model |  Layer |  Description
---|---|---
Model #2 |  Conv2D + BatchNorm + ReLU |  32 filters ($3\times 3$)
Conv2D + BatchNorm + ReLU + Max Pooling($2\times 2$) |  32 filters ($3\times 3$)
Conv2D + BatchNorm + ReLU |  64 filters ($3\times 3$)
Conv2D + BatchNorm + ReLU + Max Pooling($2\times 2$) |  64 filters ($3\times 3$)
Conv2D + BatchNorm + ReLU |  128 filters ($3\times 3$)
Conv2D + BatchNorm + ReLU + Max Pooling($2\times 2$) + Dropout ($p=0.4$) |  128 filters ($3\times 3$)
Dense + BatchNorm + ReLU + Dropout ($p=0.5$) |  512 units
Softmax |  10 classes
Model #3 |  Conv2D + ReLU |  64 filters ($3\times 3$)
Conv2D + ReLU + Max Pooling($2\times 2$) |  64 filters ($3\times 3$)
Conv2D + ReLU |  128 filters ($3\times 3$)
Conv2D + ReLU + Max Pooling($2\times 2$) |  128 filters ($3\times 3$)
Dense + ReLU + Dropout ($p=0.5$) |  256 filters ($3\times 3$)
Dense + ReLU |  256 filters ($3\times 3$)
Softmax |  10 classes

###  4.3 SFAD Settings

As described in Section 3 and Figure 2, we introduce here the implementation
details for the detector components.

####  4.3.1 Selective AEs classifiers block

It consists of an autoencoder, up/down sampling, bottleneck, and noise layers.
Each has the following architecture:
Autoencoder. As shown in Figure 3, let the input size be $Z\times w\times h$.
In the encoding process, the number of $3\times 3$-kernel filters are set to
$Z/2$, $Z/4$, and $Z/16$, respectively. In the decoding process, the number of
filters $Z$ are symmetrically restored. Finally, to maintain the input samples
characteristics that we have before autoencoding, the input is added/summed to
the output of the autoencoder.

Figure 3: Autoencoder architecture

Up/down-sampling. As shown in Figure 4, let the input size be $Z\times w\times
h$. The input size is doubled by bilinear up sampling in the first two
consecutive layers and then restored by average pooling in the last two
layers. Finally, to maintain the features before up/down sampling, the input
is added to the output of up/down-sampling.

Figure 4: Up/down-sampling architecture

Bottleneck. It is a three-convolutional layer module with kernels of size
$1\times 1$, $3\times 3$, and $1\times 1$. The architecture of the bottleneck
layers are shown in Figure 5. The number of the filters for each layer is
1024, 512, and 256.

Figure 5: Bottleneck architecture

Noise. For this layer, the GaussianNoise layer model from Keras library is
used with small standard variation of 0.05.

Dense layers. A dense layer with 512 output is used followed by batch
normalization and ReLU activation function.

SelectiveNet. A dense layer with 512 outputs is used followed by batch
normalization and ReLU activation function. After that, as original
SelectiveNet’s implementation suggests, a layer that divides the result of the
previous layer by 10 is used as a normalization step. Finally, a dense layer
of one output is used with sigmoid activation function. We set $\lambda=32$,
$c=1$ for MNIST and $c=0.9$ for CIFAR10, and coverage threshold to 0.995 for
MNIST and 0.9 for CIFAR10. More details about selectiveNet hyper-parameters
are found in selective2019 .

####  4.3.2 Selective Knowledge Transfer block.

It consists of one dense layer with 128 outputs followed by batch
normalization and ReLU activation function. The selective task of the
knowledge transfer block consists of a dense layer with 128 outputs followed
by batch normalization and ReLU activation function. After that a
normalisation layer that divides the result of the previous layer by 10 is
used as recommended by the original implementation of SelectiveNet. Finally, a
dense layer of one output is used with sigmoid activation function. We set
$\lambda=32$, $c=1$ for MNIST and $c=0.9$ for CIFAR10, and coverage threshold
to 0.7 for MNIST and CIFAR10. More details about selectiveNet hyper-parameters
are found in selective2019 .

###  4.4 Threat Model, attacks, and state-of-the-art detectors

Threat Model. We follow one of the threat models presented in
biggio2013evasion ; carlini2017adversarial ; Zero-Knowledge adversary threat
model. It is assumed that the adversary has no knowledge that a detector is
deployed and he generates the white-box attacks with the knowledge of the
baseline classifier. For cases when an adversary has perfect or limited
knowledge of the detector, we assume that the adversary’s work will be so hard
since SFAD adopts ensemble detection, and hence, we leave this as future work.
Instead, we tested SFAD robustness with the recommended high confidence strong
attack, a variant of CW attack, that is rarely tested in other detectors.

Attacks. We tested the proposed model with different types of attacks. For
white-box attacks; $L_{\infty}$-norm attacks are used; FGSM
goodfellow2014explaining , PGD madry2017towards , and CW carlini2017towards
attacks. Besides, $L_{2}$-norm DeepFool (DF) moosavi2016deepfool  attack is
used as well in the testing process. For FGSM and PGD attacks, the epsilon is
set to different values from 0.03 to 0.4. For the CW attack, the number of
iterations is set to be 1000 (strong case). Other attacks’ hyper-parameters
are set to default as defined in the ART art2018  library. The High Confidence
Attack carlini2017adversarial  is also used to test the robustness of the
proposed model. For the black-box attacks, Threshold Attack (TA)
kotyan2019adversarial , Pixel Attack (PA) su2019one , and Spatial
Transformation attack (ST) engstrom2019exploring  attacks are used in the
testing process. The threshold parameter for TA and PA is set to 10 for MNIST
and to ART defaults for CIFAR. The translation and rotation values of ST
attack are set to 10 and 60 for MNIST and to 8 and 30 for CIFAR, respectively.
For the comparison with the state of the art algorithms, more black box
attacks are considered like Square Attack (SA) andriushchenko2020square , and
HopSkipJump attack (HSJA) chen2020hopskipjumpattack . For SA, the epsilon
$\epsilon$ is set to 0.4 to 0.1 for MNIST and CIFAR10 respectively. For
HopSkipJump attack, untargeted and unmasked attack is considered, besides, 40
and 100 were set for iteration steps and maximum evaluations, respectively.

Comparison with existing detectors. Supervised and unsupervised detectors are
used to compare SFAD with state-of-the-art techniques. Supervised methods like
KD+BU feinman2017detecting , LID ma2018characterizing , and RAID
eniser2020raid  are compared with SFAD. While unsupervised methods like FS
xu2017feature , MagNet meng2017magnet , NIC ma2019nic , and DNR sotgiu2020deep
are also considered in the comparisons.

Table 5: Performance accuracy (%) of different detection processes against
white-box attacks($\epsilon$) on MNIST and CIFAR10 datasets at FP=10%.
Prediction row is related to baseline DNN top-1 accuracy of not detected AEs.
Total = Ensemble Detection + Prediction.

|  | FGSM
---
(0.05)
| FGSM
---
(0.075)
| FGSM
---
(0.1)
| FGSM
---
(0.2)
| FGSM
---
(0.4)
| FGSM
---
(AVG)
| PGD
---
(0.05)
| PGD
---
(0.075)
| PGD
---
(0.1)
| PGD
---
(0.2)
| PGD
---
(0.4)
| PGD
---
(AVG)
DF |  CW
|  Baseline DNN |  96.31 |  92.93 |  87.2 |  28.04 |  7.91 |  - |  95.18 |  85.84 |  56.91 |  0 |  0 |  - |  4.68 |  38.97
|  Selective Detection |  18.82 |  25.02 |  33.08 |  82.29 |  98.4 |  51.52 |  20.49 |  32.91 |  51.81 |  58.07 |  48.17 |  42.29 |  95.7 |  43.71
|  Confidence Detection |  8.66 |  13.74 |  20.75 |  74.18 |  98.66 |  43.2 |  10.31 |  21 |  42.73 |  54.14 |  47.41 |  35.12 |  94.96 |  42.78
|  Mismatch Detection |  3.07 |  5.66 |  9.77 |  43.43 |  63.93 |  25.17 |  3.85 |  10.5 |  28.61 |  26.86 |  18.69 |  17.7 |  59.47 |  44.99
|  Ensemble Detection |  22.97 |  31.24 |  40.88 |  88.83 |  99.8 |  56.74 |  25.6 |  41.91 |  66.71 |  68.09 |  58.93 |  52.25 |  99.14 |  61.21
|  Prediction |  76.99 |  68.64 |  58.74 |  9.03 |  0 |  42.68 |  74.34 |  57.72 |  31.69 |  0 |  0 |  32.75 |  0.19 |  37.44
MNIST |  Total |  99.96 |  99.88 |  99.62 |  97.86 |  99.8 |  99.42 |  99.94 |  99.63 |  98.4 |  68.09 |  58.93 |  85 |  99.33 |  98.65
|  Baseline DNN |  14.09 |  13.44 |  12.25 |  10.5 |  9.75 |  - |  0.43 |  0.28 |  0.22 |  0.16 |  0.17 |  - |  4.79 |  20.95
|  Selective Detection |  41.31 |  46.8 |  53.78 |  71.34 |  9.83 |  44.61 |  33.6 |  33.18 |  30.52 |  22.66 |  18.65 |  27.72 |  39.06 |  37.41
|  Confidence Detection |  69.3 |  80.61 |  87.35 |  99.09 |  99.99 |  87.27 |  43.33 |  54.69 |  60.86 |  66.83 |  67.68 |  58.68 |  85.76 |  65.47
|  Mismatch Detection |  25.49 |  35.15 |  42.88 |  44.41 |  59.66 |  41.52 |  0 |  0 |  0.02 |  0.11 |  1.32 |  0.29 |  37.5 |  34.42
|  Ensemble Detection |  72.07 |  81.84 |  88.42 |  99.41 |  100 |  88.34 |  57.57 |  63.59 |  66.67 |  68.77 |  68.74 |  65.07 |  88.45 |  69.93
|  Prediction |  6.94 |  3.28 |  1.39 |  0.02 |  0 |  2.33 |  0.34 |  0.13 |  0.07 |  0.06 |  0.04 |  0.13 |  1.35 |  20.09
CIFAR |  Total |  79.01 |  85.12 |  89.81 |  99.43 |  100 |  90.67 |  57.91 |  63.72 |  66.74 |  68.83 |  68.78 |  65.2 |  89.8 |  90.02

##  5 Experimental Results and Discussion

In this section, we evaluate the prototype of SFAD against different types of
attack scenarios and datasets and then, we provide a comparison discussion
with state-of-the-art detectors. Performance results on successful attacks
only are also discussed. Besides, the proposed approach is tested with
different $N$ settings and the results are discussed. In order to emphasize
the advantages of SFAD’s feature processing components, we provide an ablation
study for each component. Finally, performance results on different rejection
rates, i.e. false positive rates are shown. As a reminder, we use only the
last three representative layers ($N=3$) to build three selective AEs
classifiers since the aim is to prove the concept of the approach and if that
is changed with the best combination, the detector accuracy will be enhanced
accordingly.

###  5.1 Zero-Knowledge (of detectors) adversary white-box attacks

Table 5 shows selective, confidence, mismatch, and ensemble detection
accuracies of SFAD prototype for MNIST and CIFAR10 datasets. It also shows the
baseline DNN prediction accuracy for the AEs in “Baseline DNN” row and for the
not detected AEs in “prediction” row. The “Total” row is the total accuracy of
ensemble detection and truly classified/predicted samples.

For MNIST dataset, the FGSM attacks with small epsilon ($\epsilon=0.05,$
$0.075,$ and $0.1$) slightly fooled the baseline classifier and hence their
feature space still inside or at the border as of training dataset. The
detector shows its ability to reject those samples that are so close to the
classes borders and achieves the accuracy of 99.96%, 99.88%, and 99.62% for
$\epsilon=(0.05,0.075,\text{ and},0.1)$, respectively. Similar observation is
noticed for PGD attacks with small $\epsilon$ values. For larger $\epsilon$
values, DF, and CW attacks, the AEs are highly able to fool the baseline
classifier since adversaries are able to change the MNIST test samples’
feature space to lie out of its corresponding class border and hence, for all
tested attacks except the PGD, the model was able to catch them with accuracy
above 98.65%. While the detector achieves 68.09% and 58.93% for PGD attacks
with $\epsilon=(0.2,\text{ and},0.4)$, respectively. Some PGD examples’
feature space became indistinguishable from the trained samples feature space.
That makes SFAD not able to catch all AEs and to enhance SFAD’s performance,
the best representative layers combination has to be used as input for the
detector.

For CIFAR10 dataset, SFAD achieves comparable results with state-of-the-art
methods for FGSM ($\epsilon=0.1,$ $0.2,$ and $0.4$), DF, and CW attacks. While
for FGSM ($\epsilon=0.05,$ $\text{and }0.075$) and PGD attacks, the AEs have,
to some extent, indistinguishable feature space than those the detector is
trained with. In average, the model achieves accuracy of 65.2% for PGD
attacks.

For both datasets, the effectiveness of selective, confidence, and mismatch
detection is obvious. The ability of the two modules to detect the AEs is
increasing when the amount of the perturbations is increasing. When the amount
of the perturbations increased in a way that makes the adversarial samples
feature space indistinguishable from the training dataset, the ability of
these modules to detect the AEs is decreasing.

###  5.2 Black-box attacks

Table 6 shows SFAD prototype’s detection accuracy against the TA
kotyan2019adversarial , PA su2019one , and ST engstrom2019exploring  attacks
on MNIST and CIFAR10 datasets. The detector is able to catch the AEs with very
high accuracy, higher than 97.56% and 93.97% for MNIST and CIFAR10,
respectively. It is clear that the selective, confidence, and mismatch modules
complement each other. The black-box attacks significantly change the samples
features that facilitate the confidence module detection process. While the
ability of selective module is limited for TA and PA attacks since these
attacks change one or more pixels within a threshold that is in a variation of
the input sample and yield AEs that are so close to clean samples.

Table 6: Performance accuracy (%) of different detection processes against
black-box attacks on MNIST and CIFAR10 datasets at FP=10%. Prediction column
is related to baseline DNN top-1 accuracy of not detected AEs. Total =
Ensemble Detection + Prediction.

Dataset | Attack | Baseline DNN | Proposed
---|---|---|---
|  Selective Detection |  Confidence Detection |  Mismatch Detection |  Ensemble Prediction |  Prediction |  Total
MNIST | Threshold Attack | 77.61 |  24.36 |  85.48 |  42.37 |  85.62 |  14.31 |  99.93
| Pixel Attack | 74.57 |  24.65 |  85.57 |  42.88 |  85.76 |  14.18 |  99.94
| Spatial Transformation | 22.04 |  86.7 |  80.95 |  34.71 |  94.74 |  2.85 |  97.59
CIFAR | Threshold Attack | 11.29 |  12.69 |  92.14 |  37.11 |  92.62 |  1.35 |  93.97
| Pixel Attack | 11.35 |  12.48 |  92.4 |  37.02 |  92.77 |  1.39 |  94.16
| Spatial Transformation | 52.58 |  44.64 |  68.16 |  32.44 |  72.54 |  24.03 |  96.57

Table 7: Performance accuracy (%) of different detection processes against
grey-box attacks($\epsilon$) on MNIST dataset at FP=10%. Prediction column is
related to baseline DNN top-1 accuracy of not detected AEs. Total = Ensemble
Detection + Prediction.

Attack ($\epsilon$) |  Model#1 | Model#2 | Model#3
---|---|---|---
Total |  Selective Detection |  Confidence Detection |  Mismatch Detection |  Ensemble Detection |  Prediction |  Total |  Selective Detection |  Confidence Detection |  Mismatch Detection |  Ensemble Detection |  Prediction |  Total
FGSM(0.05) |  99.96 |  11.94 |  4.31 |  0.57 |  13.86 |  86.14 |  100 |  12.98 |  5.01 |  0.87 |  15.28 |  84.72 |  100
FGSM(0.075) |  99.88 |  13.71 |  5.3 |  1.08 |  16.23 |  83.77 |  100 |  15.63 |  6.65 |  1.68 |  18.84 |  81.14 |  99.98
FGSM(0.1) |  99.62 |  16.3 |  6.76 |  1.66 |  19.59 |  80.41 |  100 |  19.48 |  9.17 |  2.62 |  23.66 |  76.29 |  99.95
FGSM(0.2) |  97.86 |  46.72 |  30.18 |  12.77 |  51.49 |  48.25 |  99.74 |  57.56 |  47.39 |  24.75 |  63.32 |  35.59 |  98.91
FGSM(0.4) |  99.8 |  96.84 |  96.39 |  58.97 |  99.41 |  0.41 |  99.82 |  95.88 |  98.81 |  55.88 |  99.89 |  0 |  99.89
PGD(0.05) |  99.94 |  11.63 |  4.11 |  0.53 |  13.57 |  86.43 |  100 |  13.16 |  5.05 |  0.95 |  15.49 |  84.49 |  99.98
PGD(0.075) |  99.63 |  12.9 |  4.67 |  0.92 |  15.01 |  84.99 |  100 |  16.34 |  7 |  1.74 |  19.77 |  80.21 |  99.98
PGD(0.1) |  98.4 |  15.41 |  5.99 |  1.64 |  18.53 |  81.47 |  100 |  20.92 |  10.13 |  3.55 |  25.73 |  74.21 |  99.94
PGD(0.2) |  68.09 |  47.2 |  32.61 |  15.18 |  54.9 |  44.44 |  99.34 |  60.85 |  54.55 |  30.25 |  73.26 |  23.85 |  97.11
PGD(0.4) |  58.93 |  83.59 |  83.73 |  30.33 |  91.1 |  0.93 |  92.03 |  77.18 |  57.9 |  21.84 |  82.78 |  0.2 |  82.98
DF |  99.33 |  90.33 |  84.72 |  43.46 |  92.25 |  7.37 |  99.62 |  91.78 |  91.26 |  57.84 |  96.03 |  2.59 |  98.62
CW |  98.65 |  21.01 |  10.96 |  3.24 |  25.16 |  74.82 |  99.98 |  31.27 |  19.26 |  6.93 |  38.57 |  61.33 |  99.9

Table 8: Performance accuracy of different detection processes against grey-
box attacks($\epsilon$) on CIFAR10 dataset at FP=10%. Prediction column is
related to baseline DNN top-1 accuracy of not detected AEs. Total = Ensemble
Detection + Prediction.

Attack ($\epsilon$) |  Model#1 | Model#2 | Model#3
---|---|---|---
Total |  Selective Detection |  Confidence Detection |  Mismatch Detection |  Ensemble Detection |  Prediction |  Total |  Selective Detection |  Confidence Detection |  Mismatch Detection |  Ensemble Detection |  Prediction |  Total
FGSM(0.05) |  79.01 |  43.77 |  72.69 |  28.27 |  74.09 |  8.56 |  82.65 |  48.36 |  73.44 |  27.45 |  76.43 |  7.9 |  84.33
FGSM(0.075) |  85.12 |  45.61 |  79 |  33.18 |  79.68 |  4.26 |  83.94 |  49.16 |  78.54 |  32.32 |  80.25 |  4.17 |  84.42
FGSM(0.1) |  89.81 |  53.11 |  87.11 |  38.82 |  87.48 |  2.03 |  89.51 |  48.36 |  82.7 |  32.8 |  83.69 |  2.55 |  86.24
FGSM(0.2) |  99.43 |  77.37 |  99.58 |  37.72 |  99.66 |  0.03 |  99.69 |  60.85 |  96.82 |  26.97 |  96.9 |  0.26 |  97.16
FGSM(0.4) |  100 |  5.6 |  100 |  62.05 |  100 |  0 |  100 |  23.04 |  99.99 |  16.8 |  100 |  0 |  100
PGD(0.05) |  57.91 |  6.55 |  18.95 |  3.46 |  20.7 |  5.62 |  26.32 |  7.25 |  16.13 |  4.35 |  18.56 |  4.44 |  23
PGD(0.075) |  63.72 |  2.58 |  12.43 |  1.12 |  13.18 |  5.13 |  18.31 |  3.33 |  8.93 |  1.96 |  10.36 |  4.31 |  14.67
PGD(0.1) |  66.74 |  1.93 |  11.95 |  0.91 |  12.6 |  5.07 |  17.67 |  2.48 |  8.14 |  1.96 |  9.6 |  4.3 |  13.9
PGD(0.2) |  68.83 |  3.5 |  18.74 |  2.05 |  19.64 |  4.9 |  24.54 |  6.13 |  16.45 |  4.05 |  19.35 |  3.92 |  23.27
PGD(0.4) |  68.78 |  8.19 |  29.39 |  5.57 |  31.76 |  4.35 |  36.11 |  23.73 |  48.56 |  13.81 |  54.21 |  2.37 |  56.58
DF |  89.8 |  37.27 |  77.65 |  42.68 |  81.2 |  16.17 |  97.37 |  29.3 |  81.02 |  37.75 |  84.28 |  11.74 |  96.02
CW |  90.02 |  24.86 |  40.69 |  11.3 |  45.35 |  53.29 |  98.64 |  31.7 |  50.07 |  14.82 |  55.06 |  42.18 |  97.24

###  5.3 Gray-box attacks

Gray-box scenario assumes that the adversary has only knowledge about the
model training data and the output of the DNN model. Hence, we trained two
models as substitution models named Model#2 and Model#3 for MNIST and CIFAR10
as shown in Table 3 and Table 4, respectively. Then, white-box based AEs are
generated using the substitution models. The SFAD prototype is then tested
against these AEs. For both datasets, it is shown in Table 7 and Table 8 that
the perturbations properties generated from one model are transferred to the
tested model, Model#1. For MNIST, see Table 7, SFAD prediction rate is much
better for PGD attacks and the prediction rate for other attacks is comparable
with the prediction rate of AEs generated from Model#1. For CIFAR10, see Table
8, the prediction rate for CW and DF attacks is higher than those attacks that
are generated using Model#1, while the prediction rate for FGSM is comparable
with the prediction rate for FGSM attacks that are generated using Mode1#1.
Unlike other attacks, the PGD attacks transferable properties sound to be much
stronger and have different feature space, compared to feature space of AEs
that are generated from Model#1. This reduces the ability of the detector to
catch such attacks.

Table 9: Detection modules’ accuracies (%) against successful white-box and
black-box attacks($\epsilon$) on MNIST and CIFAR10 datasets at FP=10%.

|  | White-box attacks | Black-box attacks
---|---|---|---
|  FGSM (0.05) |  FGSM (0.075) |  FGSM (0.1) |  FGSM (0.2) |  FGSM (0.4) |  FGSM (AVG) |  PGD (0.05) |  PGD (0.075) |  PGD (0.1) |  PGD (0.2) |  PGD (0.4) |  PGD (AVG) |  DF |  CW |  TA |  PA |  ST
MNIST |  Selective Detection |  59.89 |  60.74 |  64.56 |  89.4 |  98.33 |  74.58 |  57.35 |  64.23 |  68.05 |  58.07 |  48.17 |  59.17 |  95.78 |  69.16 |  25.96 |  28.04 |  87.9
|  Confidence Detection |  60.99 |  60.6 |  63.29 |  88.56 |  98.57 |  74.4 |  59.45 |  63.66 |  71.16 |  54.14 |  47.41 |  59.16 |  95 |  69.96 |  99.19 |  99.08 |  88.1
|  Mismatch Detection |  77.2 |  74.21 |  71.28 |  59.03 |  62.95 |  68.93 |  74.16 |  70.17 |  64.25 |  26.86 |  18.69 |  50.83 |  58.87 |  73.73 |  86.84 |  86.82 |  40.96
|  Ensemble Detection |  98.9 |  98.42 |  97.07 |  97.04 |  99.78 |  98.24 |  98.74 |  97.35 |  96.31 |  68.09 |  58.93 |  83.88 |  99.3 |  97.8 |  99.73 |  99.8 |  96.91
CIFAR |  Selective Detection |  44.7 |  49.05 |  54.7 |  70.16 |  10.2 |  45.76 |  33.74 |  33.25 |  30.6 |  22.68 |  18.68 |  27.79 |  39.12 |  47.32 |  10.92 |  10.84 |  60.08
|  Confidence Detection |  72.57 |  81.45 |  87.35 |  99.03 |  99.99 |  88.08 |  43.44 |  54.69 |  60.84 |  66.83 |  67.68 |  58.7 |  86.53 |  81.74 |  92.78 |  93.05 |  87.96
|  Mismatch Detection |  27.62 |  36.19 |  42.58 |  42.67 |  58.52 |  41.52 |  0 |  0 |  0.02 |  0.11 |  1.33 |  0.29 |  37.79 |  43.54 |  39.15 |  38.92 |  54.35
|  Ensemble Detection |  75.57 |  82.82 |  88.4 |  99.36 |  100 |  89.23 |  57.74 |  63.62 |  66.66 |  68.71 |  68.74 |  65.09 |  89.33 |  87.38 |  93.21 |  93.42 |  92.76

###  5.4 Performance on successful attacks only

Table 9 shows the detection rate against the AEs that fooled the baseline DNN
classifier only under white-box and black-box scenarios. For MNIST, in
general, comparable results with the state-of-the-art detectors are achieved
for all tested white and black boxes attacks ($>96.91\%$) except for the PGD
attacks (83.88%). For both datasets, the impact of selective, confidence, and
mismatch detection modules are obvious. The ability of the modules to detect
the AEs is increasing when the amount of the perturbations is increasing. When
the amount of the perturbations increases in a way that makes the adversarial
samples feature space indistinguishable from the training dataset, the ability
of these modules to detect the AEs decreases. Mismatch detection shows a high
impact in the detection process of AEs except for PGD attacks. Once the amount
of crafted perturbation becomes high, the performance of mismatch detection
decreases. That’s because the detector classifiers’ and the baseline DNN
classifier’s behavior will be inconsistent for highly degraded inputs.

Figure 6: Total model performance accuracy (%) for black and white box
scenarios on MINIST and CIFAR10 datasets at FP=10% with different $N$
selective AEs classifiers settings

###  5.5 Results with $N$ last layer(s) output(s)

Results shown in Figure 6 emphasize the conclusion in sotgiu2020deep  that
recommends to use more than one layer from the last layers of the baseline DNN
classifier to be used in detection techniques. For MNIST dataset, the benefit
of using more than one layer appears in detecting PGD ($\epsilon=0.2\text{,
and 0.4}$), TA, and PA attacks, while it appears in all tested attacks on
CIFAR10 dataset. It means that low-/and medium-level hidden layers hold
features that will be triggered when small perturbations are added to input
samples.

Table 10: Ablation performance (%) on white-box scenarios for MNIST dataset.

Attack/Model |  Baseline DNN |  NN |  Only noise |  Only auto encoder |  Only up/down sampling |  Only bottleneck |  No noise |  No auto encoder |  No up/down sampling |  No bottleneck |  Proposed
---|---|---|---|---|---|---|---|---|---|---|---
FGSM(0.05) |  96.31 |  100 |  99.97 |  100 |  100 |  99.92 |  99.95 |  99.96 |  99.95 |  99.97 |  99.96
FGSM(0.075) |  92.93 |  99.98 |  99.94 |  99.96 |  99.97 |  99.81 |  99.76 |  99.9 |  99.8 |  99.92 |  99.88
FGSM(0.1) |  87.2 |  99.9 |  99.93 |  99.9 |  99.93 |  99.69 |  99.49 |  99.79 |  99.51 |  99.86 |  99.62
FGSM(0.2) |  28.04 |  99.61 |  99.44 |  98.97 |  98.78 |  96.71 |  96.37 |  96.96 |  97.79 |  98.79 |  97.86
FGSM(0.4) |  7.91 |  98.98 |  98.37 |  98.65 |  96.14 |  97.85 |  93.73 |  99.87 |  98.81 |  97.01 |  99.8
PGD(0.05) |  95.18 |  100 |  99.95 |  99.97 |  99.98 |  99.88 |  99.87 |  99.93 |  99.91 |  99.95 |  99.94
PGD(0.075) |  85.84 |  99.83 |  99.86 |  99.84 |  99.86 |  99.58 |  99.43 |  99.73 |  99.57 |  99.76 |  99.63
PGD(0.1) |  56.91 |  99.35 |  99.2 |  99.06 |  99.09 |  98.09 |  97.65 |  98.43 |  97.7 |  98.92 |  98.4
PGD(0.2) |  0 |  60.06 |  59.33 |  66.24 |  53.73 |  66.18 |  57.9 |  61.39 |  63.44 |  63.93 |  68.09
PGD(0.4) |  0 |  43.8 |  42.61 |  52.57 |  40.97 |  46.11 |  45.11 |  43.39 |  46.57 |  47.71 |  58.93
DF |  4.68 |  97.71 |  97.35 |  98.44 |  95.15 |  98.51 |  96.88 |  99 |  98.83 |  98.62 |  99.33
CW |  38.97 |  99.71 |  99.66 |  99.52 |  99.6 |  98.36 |  97.73 |  98.79 |  98.57 |  99.19 |  98.65

Table 11: Ablation performance (%) on white-box scenarios for CIFAR10 dataset.

Attack/Model |  Baseline DNN |  NN |  Only noise |  Only auto encoder |  Only up/down sampling |  Only bottleneck |  No noise |  No auto encoder |  No up/down sampling |  No bottleneck |  Proposed
---|---|---|---|---|---|---|---|---|---|---|---
FGSM(0.05) |  14.09 |  80.02 |  79.55 |  73 |  80.01 |  79.68 |  78.62 |  75.81 |  75.7 |  78.55 |  79.01
FGSM(0.075) |  13.44 |  87.9 |  85.21 |  80.44 |  85.84 |  86.61 |  86.2 |  84.77 |  81.3 |  85 |  85.12
FGSM(0.1) |  12.25 |  91.37 |  87.26 |  84.64 |  87.71 |  91 |  91.27 |  90.75 |  85.56 |  88.26 |  89.81
FGSM(0.2) |  10.5 |  85.52 |  78.89 |  95.13 |  78.73 |  97.79 |  99.48 |  98.19 |  95.69 |  88.92 |  99.43
FGSM(0.4) |  9.75 |  84.39 |  84.94 |  99.71 |  79.32 |  100 |  100 |  99.98 |  100 |  99.85 |  100
PGD(0.05) |  0.43 |  0.42 |  0.43 |  4.21 |  0.42 |  1.08 |  36.61 |  7.8 |  48.34 |  12.58 |  57.91
PGD(0.075) |  0.28 |  0.27 |  0.28 |  11.64 |  0.27 |  2.76 |  47.19 |  9.65 |  58.32 |  12.67 |  63.72
PGD(0.1) |  0.22 |  0.22 |  0.22 |  17.72 |  0.22 |  5.22 |  52.68 |  12.09 |  61.49 |  13.42 |  66.74
PGD(0.2) |  0.16 |  0.15 |  0.15 |  27.83 |  0.16 |  14.7 |  58.23 |  18.86 |  64.31 |  16.61 |  68.83
PGD(0.4) |  0.17 |  0.17 |  0.17 |  33.74 |  0.16 |  22.75 |  60.23 |  23.9 |  65.4 |  18.21 |  68.78
DF |  4.79 |  83.67 |  84.84 |  83.71 |  82.7 |  89.11 |  88.32 |  87.64 |  87.96 |  88.09 |  89.8
CW |  20.95 |  88.98 |  89.05 |  87.03 |  88.33 |  89.46 |  88.63 |  87.7 |  88.14 |  89.26 |  90.02

###  5.6 Ablation study

In this section, we emphasize the advantages of SFAD’s feature processing
components including noise, autoencoder, up/down sampling, and bottleneck
blocks. Table 10 and Table 11 show the performance results for each block once
when it is present alone and another time when it is absent for MNIST and
CIFAR10 datasets. In all settings, the selectiveNet is present in the
selective AE classifiers and in the selective knowledge transfer classifier.

Only NN. When all processing blocks are absent, the MNIST results show the
ability to detect FGSM, PGD of small $\epsilon$ values, and CW attacks
slightly better than the proposed approach. While the proposed approach yields
better results for DF and PGD of high $\epsilon$ values. Since CIFAR10 dataset
is different from MNIST and has different characteristics, the only NN
component did not yield better results against FGSM of high $\epsilon$ values,
PGD, CW, and DF attacks.

Noise. When only the noise block is used, the model achieves comparable
results to SFAD except against PGD attacks. When we remove the noise block,
the performance of SFAD is reduced especially against PGD attacks for MNIST
and CIFAR10 datasets. The noise block helps the detector to better distinguish
the feature space of clean input images from those features of AEs.

Autoencoder. Autoencoder block shows a substantial impact in the proposed
approach. As discussed in Section 3.4, if the autoencoder couldn’t reconstruct
its input, different feature space might be generated for the input signal
which let SFAD able to detect the AEs. For MNIST dataset, the autoencoder
block enhanced the performance results compared to only NN model against PGD
of higher $\epsilon$ values, while the performance is reduced when the
autoencoder block is removed from the proposed approach. On the other hand,
for CIFAR10 dataset, when only the autoencoder is present, the performance
results are much better against FGSM of high $\epsilon$ values, PGD, CW, and
DF attacks when it is compared to only NN. The performance is reduced when it
is removed from the proposed approach against PGD attacks.

Up/down-sampling. Unlike other processing blocks, up/ down sampling block
yields less performance results against FGSM attacks and yields comparable
results against other attacks compared to only NN model. That’s because the
up/ down-sampling restores the global information of the input signal by the
average pooling process. On the other hand, removing the sampling block from
the proposed approach reduces the performance results especially for the
CIFAR10 dataset.

Bottleneck. Like autoencoder block, the bottleneck block shows its ability to
distinguish input signal characteristics especially in the proposed shallow
classifiers (the selective AEs classifiers). Compared to only NN model, the
only bottleneck model enhanced the performance results against FGSM of high
$\epsilon$ values, PGD, CW, and DF attacks for CIFAR10 dataset and enhanced
the performance results against PGD of high $\epsilon$ values attacks for
MNIST. Besides, the performance of the proposed approach is significantly
decreased for CIFAR10 dataset when the bottleneck block is removed.

Figure 7: Performance comparisons between different False Positive (FP) rates
and FP=10% of SFAD for white-box attacks on MNIST dataset.

###  5.7 Performance with different rejection rates (False positive (FP))

In this subsection we show the performance results of the proposed approach
when thresholds are set to reject less than 10% for MNIST as shown in Figure
7. Results show that an acceptable performance can be achieved if the
thresholds are set to less than 10%. For instance, when the false positive
rate is set to be 2%, results against PGD ($\epsilon=0.2,\text{, and }0.4$)
attacks are significantly decreased because of the selective detection. In all
other tested attacks, the difference is up to 4% and 1.76% when FP=2% and 3%,
respectively.

Table 12: Detection accuracies for the state-of-the-art detectors against white-box and black-box attacks. Top 3 are colored with red, blue, and green respectively. Dataset | Detector | Attacks
---|---|---
FPR | White box | Black box
FGSM | PGD | CW | HCA | DF | SA | HSJA | STA
MNIST |  KD+BU† | 1.76 | 62.69 | 52.52 | 42.77 | 59.26 | 48.97 | 53.53 | 61.82 | 47.94
LID† | 0.81 | 77.46 | 77.03 | 64.43 | 84.51 | 93.3 | 42.78 | 61.52 | 93.81
FS∗ | 5.27 | 97.96 | 97.19 | 98.41 | 99.99 | 66.96 | 99.96 | 99.98 | 77.49
MagNet∗ | 0.20 | 100.00 | 100.00 | 40.56 | 100.00 | 96.99 | 99.93 | 98.32 | 1.61
NIC∗ | 10.12 | 100.00 | 100.00 | 100 | 100.00 | 100 | 99.68 | 100 | 99.83
DNR∗ | 10.01 | 79.67 | 59.21 | 57.98 | 89.90 | 95.6 | 81.27 | 59.98 | 88
SFAD∗ (ours) | 10.79 | 98.66 | 81.83 | 98.24 | 99.36 | 99.58 | 98.85 | 99.91 | 97.61
CIFAR10 |  KD+BU† | 3.57 | 34.13 | 96.08 | 21.12 | 33.49 | 54.02 | 0 | 28.03 | 44.15
LID† | 7.30 | 76.15 | 96.81 | 64.52 | 55.03 | 63.57 | 85.76 | 88.34 | 94.23
FS∗ | 5.07 | 32.50 | 4.20 | 56.18 | 56.31 | 39.18 | 17.82 | 84.16 | 22.46
MagNet∗ | 0.77 | 34.61 | 0.62 | 13.23 | 0.53 | 57.33 | 94.04 | 0.58 | 2.32
NIC∗ | 10.08 | 63.15 | 100.00 | 61.68 | 73.29 | 84.91 | 61.88 | 67.53 | 48.77
DNR∗ | 10.01 | 30.23 | 18.23 | 44.15 | 29.85 | 30.2 | 52.86 | 38.81 | 56.2
SFAD∗ (ours) | 10.90 | 80.14 | 41.2 | 87.68 | 45.85 | 89.57 | 93.91 | 95.57 | 92.9
$\dagger$ Supervised detector, ∗ Unsupervised detector

###  5.8 Robustness against high confidence attack

In athalye2018obfuscated , ten defenses and detectors were broken using
Backward Pass Differentiable Approximation (BPDA), Expectation Over
Transformation (EOT), and High Confidence Attack (HCA). BPDA, and EOT are
appropriate for defense techniques, while HCA is used to fail detectors. HCA
is a variant of CW attack and generates adversarial examples with high
confidence level. In athalye2018obfuscated , LID were broken using HCA. In his
experiment, we generate AEs using HCA with $\epsilon=0.3125$ for MNIST and
$\epsilon=0.031$ for CIFAR10. The results show that SFAD is fully robust on
MNIST against HCA and partially robust (57.76%) on CIFAR10. Our analysis finds
that the confidence and selective detection methods are effective to detect
AEs. In case the confidence level of the attack is increased, SFAD can be
fine-tuned by selecting the proper layer outputs to build the selective AE
classifiers.

All the experiments that are conducted in this work are tested under zero
knowledge of the detector. We assume that the adversary’s work is very hard
for building an adaptive attack to fool SFAD since it ensembles three
detection methods. Despite that, SFAD performance will drop when the adversary
is able to craft customized perturbations to fool both the baseline classifier
and the ensemble detector.

###  5.9 Comparisons with state-of-the-art detectors

In this subsection we build a comparison with different types of supervised
and unsupervised detectors using the detectors benchmark222The source code is
available in https://github.com/aldahdooh/detectors_review and the results are
shown in Table 12. We compare the average FGSM and PGD results. For fair
comparisons, $\epsilon$ values of 0.125, 0.25, and 0.3125 are set for MNIST
dataset, while for CIFAR10 are set to 0.03 and 0.06. Moreover, the supervised
detectors are trained and tested separately against each adversarial attack
algorithm. As discussed in Section 5.7, rejection/false positive rates of SFAD
can be decreased with small compromise in the performance. For RAID
eniser2020raid  method, we compare with the results that are reported in the
original paper due to publicly unavailable code.

KD+BU feinman2017detecting . KD+BU detector is a combination of kernel density
and Bayesian uncertainty based classifiers. For both datasets, the results
show that SFAD outperforms KD+BU detector against all tested attacks except
for PGD attacks in CIFAR10 dataset. In fact, KD+BU needs not noisy clean and
adversarial images to accurately train the detector to identify the boundaries
between clean and adversarial inputs.

LID ma2018characterizing . SFAD outperforms LID in both datasets and the
tested attacks except for PGD attacks on CIFAR10. LID achieves better false
positive rate compared to SFAD but it fails against High Confidence Attack as
reported in carlini2017adversarial . When LID is trained for the HCA attacks,
it achieves better results than in carlini2017adversarial . Our approach
provides full and partial robustness against HCA for MNIST and CIFAR10,
respectively. Similar to KD+BU, LID needs not noisy clean and adversarial
images to accurately train the detector to identify the boundaries between
clean and adversarial inputs.

RAID eniser2020raid . For MNIST datasets, RAID achieves higher detection rate
for PGD attacks ($\epsilon=0.3$) and higher detection rate against FGSM and
PGD attacks for CIFAR10 while our approach improved the performance against CW
and DF attacks. Besides, RAID has a better false positive rate for MNIST only.
RAID trains clean and adversarial inputs to identify differences in neuron
activation between clean and adversarial samples. Hence, it requires a huge
knowledge of attacks and its variants to enhance its performance.

FS xu2017feature . As stated in xu2017feature , FS requires high quality
squeezers for different baseline networks and it was shown that FS is not
performing well against tested attacks on CIFAR10 dataset, while our approach
generalizes better than FS at the expense of higher false positive rate.

MagNet meng2017magnet . Results reported on Table 12 is for the detection
process of MagNet and defense process of MagNet is not considered. For MNIST,
comparable results are achieved by our approach except for CW and ST attacks
where SFAD achieves better performance. For CIFAR, our approach outperforms
MagNet against the tested attacks. Since MagNet is a denoiser-based detector,
it is not guaranteed that the denoisers will remove all the noise and have
highly denoised inputs that respect the target threshold. This applies
specifically to $L_{0}$ and $L_{2}$ attacks. On the contrary, our approach
relies on confidence value changes that the AEs will cause which makes our
approach able to identify AEs. Although MagNet yields to a less false positive
rate, it was shown in carlini2017magnet  that MagNet can be broken by
different strategies.

NIC ma2019nic . NIC is the state-of-the-art detector that achieves better
performance, in general, against white box attacks compared to other
detectors, while our approach achieves better performance against tested black
box attacks. Unlike the proposed approach, other works reported that the NIC’s
baseline detectors are not consistent bulusu2020anomalous , increase the model
parameters overhead lust2020gran , are time consuming ma2019nic , and have
latency in the inference time gao2020backdoor .

DNR sotgiu2020deep . DNR adopted confidence-based detectors and is close to
our approach, but we include the feature processing and selective modules
components. The reported results show that our approach outperforms DNR at the
same false positive rates for MNIST and CIFAR10 datasets.

Other performance comparison:  SFAD has middle complexity level due to
classifiers training times, and has no inference time latency, but it has a
compromise on overhead due to classifiers parameters saving. Compared to other
detectors, SFAD introduces shallow networks hence, compared to NIC, DNR, and
LID, our detector has much less complexity. Besides, it works in parallel to
the baseline classifier and no latency is provided compared to FS and NIC.
Finally, like NIC and DNR, SFAD has to pay a little price in terms of overhead
compared to MagNet, FS and LID.

##  6 Conclusion

In this work, we have proposed a novel unsupervised and ensemble mechanism,
namely SFAD, to detect adversarial attacks. SFAD handled the $N$-last layers
outputs of the baseline DNN classifier to identify AEs. It built $N$ selective
AEs classifiers that each took one layer output of the baseline classifier as
input and then processed the input using autoencoder, up/down sampling,
bottleneck, and additive noise blocks. Then, these feature-based classifiers
were optimized in the SelectiveNet model to estimate the model’s uncertainties
and confidences. The confidence values of these classifiers were then
distilled as input to the selective knowledge transfer classifier to build the
last classifier. Selective and confidence thresholds were set to identify the
adversarial inputs. Selective, confidence, and mismatch modules are jointly
working to enhance the detection accuracy. We showed that the model is
consistent and is able to detect tested attacks. Moreover, the model is robust
in different attack scenarios; white, black, and gray boxes attacks. This
robustness, with the advantage that the model does not require any knowledge
of adversarial attacks, will lead to better generalization. The main
limitation of the model is that the best combination of $N$ needs to be
identified to enhance the detection accuracy and to reduce the false positive
rate.

## Acknowledgement

The project is funded by both Région Bretagne (Brittany region), France, and
direction générale de l’armement (DGA).

## References

  * [1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [2] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [3] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99, 2015.
  * [4] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  nature, 521(7553):436–444, 2015.
  * [5] Dinggang Shen, Guorong Wu, and Heung-Il Suk.  Deep learning in medical image analysis.  Annual review of biomedical engineering, 19:221–248, 2017.
  * [6] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  arXiv preprint arXiv:1312.6199, 2013.
  * [7] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * [8] Wenbo Guo, Dongliang Mu, Jun Xu, Purui Su, Gang Wang, and Xinyu Xing.  Lemna: Explaining deep learning based security applications.  In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pages 364–379, 2018.
  * [9] Naveed Akhtar and Ajmal Mian.  Threat of adversarial attacks on deep learning in computer vision: A survey.  IEEE Access, 6:14410–14430, 2018.
  * [10] Han Xu Yao Ma Hao-Chen, Liu Debayan Deb, Hui Liu Ji-Liang Tang Anil, and K Jain.  Adversarial attacks and defenses in images, graphs and text: A review.  International Journal of Automation and Computing, 17(2):151–178, 2020.
  * [11] Alexey Kurakin, Ian Goodfellow, and Samy Bengio.  Adversarial examples in the physical world.  arXiv preprint arXiv:1607.02533, 2016.
  * [12] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.  Deepfool: a simple and accurate method to fool deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574–2582, 2016.
  * [13] Nicholas Carlini and David Wagner.  Towards evaluating the robustness of neural networks.  In 2017 ieee symposium on security and privacy (sp), pages 39–57. IEEE, 2017.
  * [14] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  arXiv preprint arXiv:1706.06083, 2017.
  * [15] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow.  Transferability in machine learning: from phenomena to black-box attacks using adversarial samples.  arXiv preprint arXiv:1605.07277, 2016.
  * [16] Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh.  Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models.  In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 15–26, 2017.
  * [17] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry.  Exploring the landscape of spatial robustness.  In International Conference on Machine Learning, pages 1802–1811, 2019.
  * [18] Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai.  One pixel attack for fooling deep neural networks.  IEEE Transactions on Evolutionary Computation, 23(5):828–841, 2019\.
  * [19] Shashank Kotyan and Danilo Vasconcellos Vargas.  Adversarial robustness assessment: Why both $l_{0}$ and $l_{\infty}$ attacks are necessary.  arXiv e-prints, pages arXiv–1906, 2019.
  * [20] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In international conference on machine learning, pages 1050–1059. PMLR, 2016.
  * [21] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * [22] Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner.  Detecting adversarial samples from artifacts.  arXiv preprint arXiv:1703.00410, 2017.
  * [23] Lewis Smith and Yarin Gal.  Understanding measures of uncertainty for adversarial example detection.  arXiv preprint arXiv:1803.08533, 2018.
  * [24] Fatemeh Sheikholeslami, Swayambhoo Jain, and Georgios B. Giannakis.  Minimum uncertainty based detection of adversaries in deep neural networks.  In Information Theory and Applications Workshop, ITA 2020, San Diego, CA, USA, February 2-7, 2020, pages 1–16. IEEE, 2020.
  * [25] Yonatan Geifman and Ran El-Yaniv.  Selectivenet: A deep neural network with an integrated reject option.  CoRR, abs/1901.09192, 2019.
  * [26] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  arXiv preprint arXiv:1610.02136, 2016.
  * [27] Jonathan Aigrain and Marcin Detyniecki.  Detecting adversarial examples and other misclassifications in neural networks by introspection.  arXiv preprint arXiv:1905.09186, 2019.
  * [28] João Monteiro, Isabela Albuquerque, Zahid Akhtar, and Tiago H Falk.  Generalizable adversarial examples detection based on bi-model decision mismatch.  In 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), pages 2839–2844. IEEE, 2019.
  * [29] Angelo Sotgiu, Ambra Demontis, Marco Melis, Battista Biggio, Giorgio Fumera, Xiaoyi Feng, and Fabio Roli.  Deep neural rejection against adversarial examples.  EURASIP Journal on Information Security, 2020:1–10, 2020.
  * [30] Weilin Xu, David Evans, and Yanjun Qi.  Feature squeezing: Detecting adversarial examples in deep neural networks.  arXiv preprint arXiv:1704.01155, 2017.
  * [31] Anish Athalye, Nicholas Carlini, and David Wagner.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  arXiv preprint arXiv:1802.00420, 2018.
  * [32] Nicholas Carlini and David Wagner.  Magnet and” efficient defenses against adversarial attacks” are not robust to adversarial examples.  arXiv preprint arXiv:1711.08478, 2017.
  * [33] Saikiran Bulusu, Bhavya Kailkhura, Bo Li, Pramod K Varshney, and Dawn Song.  Anomalous instance detection in deep learning: A survey.  arXiv preprint arXiv:2003.06979, 2020.
  * [34] Julia Lust and Alexandru Paul Condurache.  Gran: An efficient gradient-norm based detector for adversarial and misclassified examples.  arXiv preprint arXiv:2004.09179, 2020.
  * [35] Shiqing Ma and Yingqi Liu.  Nic: Detecting adversarial samples with neural network invariant checking.  In Proceedings of the 26th Network and Distributed System Security Symposium (NDSS 2019), 2019.
  * [36] Yansong Gao, Bao Gia Doan, Zhi Zhang, Siqi Ma, Anmin Fu, Surya Nepal, and Hyoungshick Kim.  Backdoor attacks and countermeasures on deep learning: A comprehensive review.  arXiv preprint arXiv:2007.10760, 2020.
  * [37] Marco Melis, Ambra Demontis, Battista Biggio, Gavin Brown, Giorgio Fumera, and Fabio Roli.  Is deep learning safe for robot vision? adversarial examples against the icub humanoid.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 751–759, 2017.
  * [38] Jiajun Lu, Theerasit Issaranon, and David Forsyth.  Safetynet: Detecting and rejecting adversarial examples robustly.  In Proceedings of the IEEE International Conference on Computer Vision, pages 446–454, 2017.
  * [39] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang.  Residual attention network for image classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2017.
  * [40] Shikun Liu, Edward Johns, and Andrew J Davison.  End-to-end multi-task learning with attention.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1871–1880, 2019.
  * [41] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana.  Certified robustness to adversarial examples with differential privacy.  In 2019 IEEE Symposium on Security and Privacy (SP), pages 656–672. IEEE, 2019.
  * [42] Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh.  Towards robust neural networks via random self-ensemble.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 369–385, 2018.
  * [43] Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh.  How does noise help robustness? explanation and exploration under the neural sde framework.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 282–290, 2020.
  * [44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [45] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [46] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  University of Toronto, 05 2012.
  * [47] Nicholas Carlini and David Wagner.  Adversarial examples are not easily detected: Bypassing ten detection methods.  In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 3–14, 2017.
  * [48] Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey.  Characterizing adversarial subspaces using local intrinsic dimensionality.  arXiv preprint arXiv:1801.02613, 2018.
  * [49] Cihang Xie, Mingxing Tan, Boqing Gong, Alan Yuille, and Quoc V Le.  Smooth adversarial training.  arXiv preprint arXiv:2006.14536, 2020.
  * [50] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel.  Ensemble adversarial training: Attacks and defenses.  arXiv preprint arXiv:1705.07204, 2017.
  * [51] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He.  Feature denoising for improving adversarial robustness.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 501–509, 2019.
  * [52] Tejas Borkar, Felix Heide, and Lina Karam.  Defending against universal attacks through selective feature regeneration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 709–719, 2020.
  * [53] Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu.  Defense against adversarial attacks using high-level representation guided denoiser.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1778–1787, 2018.
  * [54] Aamir Mustafa, Salman H Khan, Munawar Hayat, Jianbing Shen, and Ling Shao.  Image super-resolution as a defense against adversarial attacks.  IEEE Transactions on Image Processing, 29:1711–1724, 2019.
  * [55] Aaditya Prakash, Nick Moran, Solomon Garber, Antonella DiLillo, and James Storer.  Deflecting adversarial attacks with pixel deflection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8571–8580, 2018.
  * [56] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami.  Distillation as a defense to adversarial perturbations against deep neural networks.  In 2016 IEEE Symposium on Security and Privacy (SP), pages 582–597. IEEE, 2016.
  * [57] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami.  Practical black-box attacks against machine learning.  In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pages 506–519, 2017.
  * [58] Shixiang Gu and Luca Rigazio.  Towards deep neural network architectures robust to adversarial examples.  arXiv preprint arXiv:1412.5068, 2014.
  * [59] Aran Nayebi and Surya Ganguli.  Biologically inspired protection of deep networks from adversarial attacks.  arXiv preprint arXiv:1703.09202, 2017.
  * [60] Anh Nguyen, Jason Yosinski, and Jeff Clune.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 427–436, 2015.
  * [61] Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel.  On the (statistical) detection of adversarial examples.  arXiv preprint arXiv:1702.06280, 2017.
  * [62] Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff.  On detecting adversarial perturbations.  arXiv preprint arXiv:1702.04267, 2017.
  * [63] Hasan Ferit Eniser, Maria Christakis, and Valentin Wüstholz.  Raid: Randomized adversarial-input detection for neural networks.  arXiv preprint arXiv:2002.02776, 2020.
  * [64] Dongyu Meng and Hao Chen.  Magnet: a two-pronged defense against adversarial examples.  In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pages 135–147, 2017.
  * [65] Florian A Potra and Stephen J Wright.  Interior-point methods.  Journal of Computational and Applied Mathematics, 124(1-2):281–302, 2000.
  * [66] Abhijit Bendale and Terrance E Boult.  Towards open set deep networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1563–1572, 2016.
  * [67] Sebastian Ruder.  An overview of multi-task learning in deep neural networks.  arXiv preprint arXiv:1706.05098, 2017.
  * [68] Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, Dengxin Dai, and Luc Van Gool.  Revisiting multi-task learning in the deep learning era.  arXiv preprint arXiv:2004.13379, 2020.
  * [69] Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482–7491, 2018.
  * [70] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich.  Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.  arXiv preprint arXiv:1711.02257, 2017.
  * [71] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei.  Dynamic task prioritization for multitask learning.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 270–287, 2018.
  * [72] Ozan Sener and Vladlen Koltun.  Multi-task learning as multi-objective optimization.  In Advances in Neural Information Processing Systems, pages 527–538, 2018.
  * [73] Linfeng Zhang, Zhanhong Tan, Jiebo Song, Jingwei Chen, Chenglong Bao, and Kaisheng Ma.  Scan: A scalable neural networks framework towards compact and efficient models.  In Advances in Neural Information Processing Systems, pages 4027–4036, 2019.
  * [74] Linfeng Zhang, Muzhou Yu, Tong Chen, Zuoqiang Shi, Chenglong Bao, and Kaisheng Ma.  Auxiliary training: Towards accurate and robust models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 372–381, 2020.
  * [75] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma.  Be your own teacher: Improve the performance of convolutional neural networks via self distillation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3713–3722, 2019.
  * [76] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli.  Evasion attacks against machine learning at test time.  In Joint European conference on machine learning and knowledge discovery in databases, pages 387–402. Springer, 2013.
  * [77] Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian Molloy, and Ben Edwards.  Adversarial robustness toolbox v1.2.0.  CoRR, 1807.01069, 2018.
  * [78] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein.  Square attack: a query-efficient black-box adversarial attack via random search.  In European Conference on Computer Vision, pages 484–501. Springer, 2020.
  * [79] Jianbo Chen, Michael I Jordan, and Martin J Wainwright.  Hopskipjumpattack: A query-efficient decision-based attack.  In 2020 ieee symposium on security and privacy (sp), pages 1277–1294. IEEE, 2020.

  *[AEs]: Adversarial Example
  *[DNN]: Deep Neural Network
  *[AE]: Adversarial Example
  *[SFAD]: Selective and Feature based Adversarial Detection
  *[DL]: Deep learning
  *[CNN]: Convolutional Neural Network
  *[BU]: Bayesian Uncertainty
  *[CNNs]: Convolutional Neural Network
  *[PGD]: Projected Gradient Descent
  *[HCA]: High Confidence Attack
  *[HCAs]: High Confidence Attack
  *[LID]: Local Intrinsic Dimensionality
  *[SVM]: Support Vector Machine
  *[RBF]: Radial Basis Function
  *[CW]: Carlini-Wagner
  *[NN]: Neural Network
  *[KD]: Kernel Density
  *[FS]: Feature Squeezing
  *[FGSM]: Fast Gradient Sign Attack
  *[NIC]: Neural-network Invariant Checking
  *[SVMs]: Support Vector Machine
  *[DNR]: Deep Neural Rejection
  *[MTL]: Multi-Task Learning
  *[DF]: DeepFool
  *[TA]: Threshold Attack
  *[PA]: Pixel Attack
  *[ST]: Spatial Transformation attack
  *[SA]: Square Attack
  *[BPDA]: Backward Pass Differentiable Approximation
  *[EOT]: Expectation Over Transformation