figurec

# Reveal of Vision Transformers Robustness against Adversarial Attacks

Ahmed Aldahdooh,Wassim Hamidouche,and Olivier Déforges All authors are with
INSA Rennes, CNRS, IETR - UMR 6164, University of Rennes, 35000, Rennes,
France.
Corresponding author: A. Aldahdooh,
E-mail<EMAIL_ADDRESS>
This work has been submitted to the IEEE for possible publication. Copyright
may be transferred without notice, after which this version may no longer be
accessible.

###### Abstract

The major part of the vanilla vision transformer (ViT) is the attention block
that brings the power of mimicking the global context of the input image. For
better performance, ViT needs large-scale training data. To overcome this data
hunger limitation, many ViT-based networks, or hybrid-ViT, have been proposed
to include local context during the training. The robustness of ViTs and its
variants against adversarial attacks has not been widely investigated in the
literature like convolutional neural networks. This work studies the
robustness of ViT variants 1) against different $L_{p}$-based adversarial
attacks in comparison with CNNs, 2) under adversarial examples (AEs) after
applying preprocessing defense methods and 3) under the adaptive attacks using
expectation over transformation (EOT) framework. To that end, we run a set of
experiments on 1000 images from ImageNet-1k and then provide an analysis that
reveals that vanilla ViT or hybrid-ViT are more robust than CNNs. For
instance, we found that 1) Vanilla ViTs or hybrid-ViTs are more robust than
CNNs under $L_{p}$-based attacks and under adaptive attacks. 2) Unlike hybrid-
ViTs, Vanilla ViTs are not responding to preprocessing defenses that mainly
reduce the high frequency components. Furthermore, feature maps, attention
maps, and Grad-CAM visualization jointly with image quality measures, and
perturbations’ energy spectrum are provided for an insight understanding of
attention-based models.

###### Index Terms:

Vision transformer, convolutional neural network, robustness, adversarial
attacks, deep learning.

##  1 Introduction

Image classification task models have remarkable progress in its prediction
accuracy especially when convolutional blocks serve as the main building block
of the model [1]. Convolutional blocks have the ability to exploit the spatial
features and in particular the low-level features [2]. On the other hand,
self-attention blocks in Transformers [3] showed great success in natural
language processing (NLP) models [3], and recently, Dosovitskiy et al.
proposed ViT, vanilla ViT, the first image classification model that uses the
pure transformer encoder blocks [4] and image patches, as tokenization, to
build the classifier. To overcome the lack of the inductive biases inherent to
CNNs, it was shown that ViT achieves better performance than state-of-the-art
CNNs models of similar capacity, such as ResNet [1] and its variants if ViT is
trained with significantly large-scale training datasets, such as JFT-300M [5,
4]. Moreover, ViT models that are trained on large-scale datasets can be
downgraded to smaller datasets, such as ImageNet-1k [6], via transfer
learning, leading to performance comparable to or better than state-of-the-art
CNNs models. Given the advantage of the ViT and on the other side its
limitation to the huge need of the data, other models that combine the vanilla
ViT with other modules, like tokens-to-token ViT (T2T-ViT) [2], transformer-
in-transformer (TNT) [7], and CvT [8] models were proposed for better learning
the low-level features in the transformer and hence, reduce their dependency
on large datasets. These models are also known as hybrid-ViT models

Given many CNN image classification task models [1, 9, 10, 11], their
properties are well studied and analyzed including the robustness against
adversarial examples [12]. AE is a combination of the original image and a
carefully crafted perturbation [12]. This perturbation is hardly perceptible
to humans, while it causes the deep learning (DL) model to misclassify the
input image. Since the feature space identification of the AE is hard to
predict [13, 14], the adversarial attacks threat is very challenging.
Adversary can generate AEs under white box, black box, and gray box attack
scenarios [15, 16].

Figure 1: The attack success rate (ASR) of target models, on 1000 images from
ImageNet-1k, against AutoAttack, in average $\epsilon=\\{1,2,4\\}/255$, and
the target model top-1 error of the preprocessed AEs for six different
preprocessing defense methods including SS: local spatial smoothing, NLM: non-
local mean, TVM: total variation minimization, JPEG: JPEG compression, CR:
cropping and re-scaling and CCP: color channel perturbations.

Due to the success of ViT and its variants in various computer vision tasks,
the insight properties and robustness studies for such transformers are yet
under investigation. Recently, four studies [17, 18, 19, 20] showed some of
these robustness properties for the vision transformers and compared them with
the competitive CNN models. In this work, we experimentally investigate the
robustness of different ViT variants against different $L_{p}$-based and color
channel perturbations (CCP) attacks and their strength of predicting the
preprocessed AEs, as defense. The results are compared with competitive CNN
models. We attend to answer the following research questions 1) Are ViT
variants more robust than CNNs against $L_{0}$, $L_{1}$, $L_{2}$, and
$L_{\infty}$ based attacks and against CCP attack? 2) Are ViT variants more
robust than CNNs under the preprocessed AEs? 3) Is increasing the number of
attention blocks has an effect on the robustness against the AEs and under the
preprocessed AEs? 4) Is enhancing ViT tokenization method has an effect on
increasing the robustness against the AEs and under the preprocessed AEs?
Hence, this work will provide researchers with an in-depth understanding on
how vision transformers behave against different attack settings with
preprocessing defenses, see Figure 1. The main contribution of this work is to
highlight the following observations:

  * •

ViT-based models are more robust than CNNs against $L_{p}$-norm attacks.

  * •

Neither larger model’s architecture, like ViT-L, nor bringing convolutional
modules for tokenization in ViTs, like TNT, will necessarily enhance the
robustness.

  * •

In general, transferability exists within the model architecture family and
the transferability became lower when transfer from large to small variants
and vice versa.

  * •

Black-box attacks are transferable to CNNs when they are generated using
vanilla ViTs and hybrid-ViTs and not vice versa.

  * •

Vanilla ViTs are not responding to preprocessing defenses that mainly reduce
the high frequency components, such as local spatial smoothing (SS) and JPEG
compression (JPEG). Hybrid-ViTs are more responsive to preprocessing defenses
than vanilla ViTs and CNNs, see Figure 1.

  * •

Hybrid-ViT models show better robustness than vanilla ViTs and ResNets under
the EOT robustness test.

##  2 Related work

Robustness of computer vision CNNs against AEs is well studied in the
literature [12], and many countermeasures [15, 16, 21, 22, 23], i.e. defenses
and detectors, were implemented to characterize the feature space of the AEs.
On the other hand, self-attention based computer vision models are under
investigation. The work in [24] used vanilla self-attention transformer to
improve the object detection task detector’s robustness against AEs. The new
module used the self-attention transformer to model the features that are
extracted from fast region convolutional neural network (Fast-RCNN) [25]
before running the detection classifier. Recently, four studies [17, 18, 19,
20] provided some understanding of the robustness of ViT, and its variants.
Authors in [17, 19, 20] studied the robustness of the vanilla ViTs, while the
work in [18] studied the robustness of vanilla ViTs and of other models that
combine vanilla ViT with other modules like convolutional blocks, hybrid-ViTs.
The work in [17] investigated the ViT robustness with respect to the
perturbations of both the input and model. The study of input perturbations
included natural corruption, real-world distribution shifts, natural
adversarial perturbations, adversarial attack perturbations, adversarial
spatial perturbations, and texture bias. The adversarial attack perturbations
are generated using $L_{\infty}$ bounded ($\epsilon$) AEs using fast gradient
sign method (FGSM) [26] and projected gradient descent (PGD) [27] attacks
where $\epsilon$ was set to one gray-level (1/255). The input perturbations
are tested on models that are pretrained on ImageNet-1k [6], ImageNet-21k
[28], and JFT-300M [5] datasets. It was shown that, like CNNs, ViTs are
vulnerable to AEs and it is more robust than CNNs with comparable capacity
when trained on sufficient training data. Similarly, the work in [19]
investigated the robustness of ViT and big transfer (BiT) CNN [29] against
common corruptions and perturbations, distribution shifts, and natural
adversarial examples. The work in [18] showed that this observation is not
necessarily true and claimed that the robustness of ViT is more related to
transformer architecture rather than the pre-training. In [18], it was shown
that introducing convolutional blocks or tokens-to-token [2] modules
compromises the model robustness when tested against $L_{\infty}$ PGD [27] and
auto projected gradient descent (Auto-PGD) [30] attacks. Moreover, [18] showed
that CNNs are less robust than ViTs since CNN tends to learn low-level
features that are much vulnerable to adversarial perturbations. The three
works in [17, 18, 19] showed that AEs do not transfer across different model
architectures’ families which leads to have an ensemble models approach to
defend against the attacks. Hence, the work in [19], included more
$L_{\infty}$ attacks like; Carlini-Wagner (CW) [31], momentum iterative
gradient-based method (MI-FGSM) [32], and backward pass differentiable
approximation (BPDA) [33], and focused on the transferability of the attacks
to introduce the self-attention gradient attack (SAGA) that can fool the
ensemble defense model. In this work, we investigate the robustness against
different $L_{p}$ norms attacks and the robustness under the pre-processed
AEs, as a defense technique.

##  3 Preliminaries

###  3.1 Models’ architectures

In this section, a brief description of models that are investigated in our
experiments is introduced. Table I gives the parameters of the investigated
models111The weights are available here , for all models except for T2T-ViTs
which are available here..

Vanilla transformer [4]. The vanilla self-attention Transformer was introduced
for NLP tasks. It stacks the encoder and the decoder blocks, where each block
stacks $N$ attention blocks. The attention block has two sub-layers. The first
one is a multi-head self-attention mechanism, and the second one is a simple,
position-wise fully connected feed-forward network. The residual connection
around each of the two sub-layers followed by layer normalization are
employed. Encoder and decoder input tokens are converted to a sequence of
vectors using learned embeddings. Finally, the learned linear transformation
and softmax function are used to convert the decoder output to predict next-
token probabilities.

vit [4]. It is the first model that employed the vanilla transformer
architecture and achieved state-of-the-art performance on image classification
task. To adapt the vanilla transformer for image classification, ViT 1)
divides the input image into a sequence of patches and then linearly projects
them to the transformer, 2) appends [CLS] token to the input and output
representations that are passed to multi-layer perceptron (MLP) for the
classification task. In our investigation, we consider different ViT models
including ViT-S/B/L-16 models. In the single-head self-attention (SHSA) of
ViTs [4], each image $X\in\mathbb{R}^{n\times d}$ is presented as a sequence
of $n$ patches ($x_{1},x_{2},\dots x_{n}$), where $d$ is the embedding
dimension to represent each patch. Then, the image sequence, $X$, is linearly
projected onto query ($Q=XW_{q}$), key ($K=XW_{k}$), and value ($V=XW_{v}$),
where $W_{q}\in\mathbb{R}^{d\times d_{q}}$, $W_{k}\in\mathbb{R}^{d\times
d_{k}}$, and $W_{v}\in\mathbb{R}^{d\times d_{v}}$. In this way, the image
patch is encoded in terms of the global information, which enables the self-
attention structure mechanism to capture the interactions among image patches.
By applying scaled dot-product attention mechanism, the output of one self-
attention $Z$ is computed as:

$Z=Attention(Q,K,V)=\textbf{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V.$
(1)

According to (1), firstly, the attention scores of input query and key are
calculated, $S=QK^{T}$, which is then normalized by $\sqrt{d}$ to prevent
pushing the softmax function into regions where it has extremely small
gradients when $d$ becomes large. Then, the probabilities of the normalized
scores are calculated, $P=\textbf{softmax}(S/\sqrt{d})$. Finally, the value
vector $V$ is multiplied by the probabilities $P$ to calculate $Z$ in which
larger probabilities are the focus in the following layers.

The SHSA limits the capability to highlight the importance of other equally
important patches at the same time. Having multi-head self-attention (MHSA)
can mitigate this problem by having different $Q$, $K$, and $V$ vectors for
each self-attention, and hence, can jointly attend to information from
different representation subspaces at different positions. In MHSA, the $Q$,
$K$, and $V$ have the size of $X\in\mathbb{R}^{n\times d/h}$, where $h$ is the
number of heads, and $d_{q}=d_{k}=d_{v}=d/6$. Hence, the output of MHSA has
the output of size $X\in\mathbb{R}^{n\times d}$ and is calculated as follows:

$\begin{gathered}MultiHead(Q^{\prime},K^{\prime},V^{\prime})=(head_{1}\oplus\dots\oplus
head_{h})W^{o},\\\ head_{i}=Attention(Q_{i},K_{i},V_{i}),\;\forall
i\in\\{1,\cdots,h\\},\end{gathered}$ (2)

where $Q^{\prime}$, $K^{\prime}$, $V^{\prime}$ are the concatenation of
$\\{Q_{i}\\}_{i=1}^{h}$, $\\{K_{i}\\}_{i=1}^{h}$, and $\\{V_{i}\\}_{i=1}^{h}$
respectively, and $W^{o}\in\mathbb{R}^{d\times d}$ is the linear projection
matrix. $\oplus$ stands for vector concatenation operation.

The ViT’s encoder block stacks $N$ layers of attention blocks. The attention
block has two sub-layers. The first one is a MHSA, and the second one is a
simple MLP layer, also known as position-wise fully connected feed-forward
network (FFN). Layernorm (LN) is applied before every sub-layer, and residual
connections are applied after each sub-layer. This latter consists of two
linear transformation layers and a nonlinear activation function, Gaussian
error linear units (GELU) [34], in between.

The work in [18] showed that the fact of having ViTs trained on large-scale
dataset yield to more robust models than CNNs is not necessarily true and
claimed that the robustness of ViTs is more related to transformer structure
rather than the pre-training. Hence, in our investigation, Vanilla ViTs are
pretrained on ImageNet [28] and fine-tuned on ImageNet-1k [6], while hybrid-
ViTs and CNN are trained from scratch on ImageNet-1k.

TABLE I: ViT, ViT-based, and CNN model variants that are investigated in this
work.

Category | Model |  ViT Backbone | Params (M) |  Top-1 Acc.(%) on ImageNet-1k
---|---|---|---|---
Layers | Hidden size |  MLP size
Vanilla ViT† |  ViT-S-16 [35] | 8 | 786 | 2358 | 49 | 77.858
ViT-B-16 [4] | 12 | 786 | 3072 | 87 | 81.786
ViT-L-16 [4] | 24 | 1024 | 4096 | 304 | 83.062
Hybrid-ViT∗ |  ViT-Res-16 (384) [4] | 12 | 786 | 3072 | 87 | 84.972
T2T-ViT-14 [2] | 14 | 384 | 1152 | 22 | 81.7
T2T-ViT-24 [2] | 24 | 512 | 1536 | 64 | 82.6
TNT-S-16 [7] | 12 | 384 | 1536 | 24 | 81.518
CNN Backbone Conv. Layers
CNN∗ | ResNet50 [1] | 49 | 23 | 79.038
ResNet50-32x4d (ResNeXt50) [36] | 49 | 25 | 79.676
VGG16 [9] | 13 | 138 | 71.594
† Pre-trained on ImageNet-21k [28] and fine tuned on ImageNet-1k [6], ∗
Trained from scratch on ImageNet-1k [6]

Hybrid-ViT models [4, 2, 7]. One key of CNNs success is the ability to learn
local features, while the key success of self-attention based transformer is
its ability to learn the global features. Hence, many approaches were
introduced to integrate local feature representation in ViT model. The work in
[4], replaced input image patches with the CNN feature map patches and
introduced ViT-Res that used the flattened ResNet feature maps to generate the
input sequence. In [2], tokens-to-token ViT (T2T-ViT) is introduced. It
replaces input image patches with a layer-wise tokens-to-token (T2T)
transformation, then, the features that are learned by T2T module are passed
to the ViT. The aim of T2T module is to progressively structurize the image
into tokens by recursively aggregating neighboring tokens into one token. The
work in [7] proposed a transformer-in-transformer (TNT) architecture to model
both patch-level and pixel-level representations. The TNT is made up by
stacking TNT blocks. Each TNT block has an inner transformer and an outer
transformer. The inner transformer block extracts local features from pixel
embeddings. In order to add the output of the inner transformer into the patch
embeddings, the output of the inner transformer is projected to the space of
patch embedding using a linear transformation layer. The outer transformer
block is used to process patch embeddings. In our investigation, we consider
T2T-ViT-14/24, and TNT-S-16 models.

cnn. The convolutional layer is the basic building block for the CNN models.
It has a set of learnable small receptive field filters. These filters are
convolved across the full depth of the input to produce the feature maps.
Hence, feature maps can remarkably represent local structure of the input
image. Many architectures have been proposed in the literature [37]. In our
investigation, we consider ResNet50 [1], ResNeXt50_32x4d [36], and VGG16 [9]
models.

###  3.2 Adversarial attacks

Our investigation assumes that the adversary can create an AE $x^{\prime}$ by
perturbing the input image $x$ with a certain amount of noise $\epsilon$, such
that $||x-x^{\prime}||_{p}\leq\epsilon$, where $p\in\\{0,1,2,\dots,\infty\\}$,
under different attack scenarios; white-, black-, hybrid-, and gray-box
attacks. We investigate the target models with 1000 images that are correctly
classified by all the target models. The test samples are collected from
ImageNet-1k validation dataset [6]. Our investigation, included five white-box
attacks: jacobian saliency map attack (JSMA) [38], FGSM [26], PGD [27],
universal adversarial perturbations (UAP) [39], and CW [31]. For the black-box
attacks, we consider square attack (SA) [40], ray searching (RayS) [41], and
CCP [42] attacks. Moreover, we consider autoattack (AA) [30], as hybrid-box
attack, which is an ensemble attack that runs three attacks to generate the
AEs: the Auto-PGD, acfab [43], and SA. For the gray-box attacks, the adversary
has knowledge about training data but not the model architecture and depends
on the transferability property of the attacks to generate the AEs using a
surrogate model. Given the surrogate model, the aforementioned attacks can be
used to generate AEs. In our investigation, one of our target models will be
considered as a surrogate model to generate the AEs and these AEs will be
tested on the other target models. Table II lists the attack’s parameters that
are used in our experiments. All the attacks are generated using adversarial
robustness toolbox (ART)222https://github.com/Trusted-AI/adversarial-
robustness-toolbox, except RayS attack that is generated using the official
implementation 333https://github.com/uclaml/RayS.

In [18], a preliminary results were introduced to train robust ViT classifiers
using adversarial training (AT). It is stated that “ViT does not advance the
robust accuracy after adversarial training compared to large CNNs such as
WideResNet-34-10”. That is because ViT may need larger training data or longer
training epochs to further improve its robust training performance. Moreover,
it is found that AT can still cause catastrophic overfitting for ViT when fast
AT training is conducted, to mitigate the overfitting, with FGSM, further
adjustments were needed to propose AT for ViT. Due to such reasons we couldn’t
conduct a fair AT comparison with CNN and we left it as future work.

TABLE II: Considered adversarial attacks and their parameters.

Scenario | Attack | norm |  Parameters
---|---|---|---
White box | FGSM | $L_{\infty}$ |  $\epsilon\in\\{1,2,4,8,16,24\\}/255$
PGD | $L_{1}$ |  $\epsilon\in\\{100,150,200,400,600,800,1000\\}$, $\epsilon_{step}=\epsilon/10$, max. iterations=10
PGD | $L_{2}$ |  $\epsilon\in\\{0.5,1,2,3,4,5\\}$, $\epsilon_{step}=\epsilon/10$, max. iterations=10
PGD | $L_{\infty}$ |  $\epsilon\in\\{1,2,4\\}/255$, $\epsilon_{step}=\epsilon/10$, max. iterations=10
CW | $L_{2}$ |  max. iterations=10, learning rate=5e-3, initial const=2/255, binary search steps=10
CW | $L_{\infty}$ |  $\epsilon=8/255$, confidence=0, max. iterations=50, learning rate=5e-3
UAP | $L_{\infty}$ |  $\epsilon\in\\{1,2,4\\}/255$, attacker=BIM, $\epsilon_{step}=\epsilon/10$, max. iterations=10
JSMA | $L_{0}$ |  $\theta=0.1$, $\gamma=1$,
Black box | SA | $L_{\infty}$ |  $\epsilon\in\\{8,16\\}/255$, $p=0.05$, max. iterations=300, restarts=1
RayS | $L_{\infty}$ |  $\epsilon=8/255$, query=2000
Hybrid box | AA | $L_{\infty}$ |  $\epsilon\in\\{1,2,4\\}/255$
Other Attacks | CCP | - |  seed=0 for fixed random-weight based CCP attack, $s=2$ and $b=30$.

###  3.3 Pre-processing defense methods

We consider pre-processing methods that were previously applied to the AEs
before being forwarded to the CNNs in order to alleviate the effect of added
perturbations. In our investigation, we consider local smoothing [44], non-
local mean denoising [44], total variation minimization [45] , JPEG image
compression [46, 47], and crop and re-scaling [48].

Local smoothing [44]. Is a method that uses a pixel neighborhood to smooth out
each pixel. For a given $n\times n$ sliding window, local smoothing changes
each pixel, center of the sliding window, with the mean, the median, or
Gaussian smooth of the window. In [44], it was shown that median filter is
more efficient in projecting the AE back to the data manifold especially for
$L_{0}$ based attacks, since it is capable of removing sparsely pixels in the
input image, and at the same time it preserves edges. In our investigation we
use the median filter with $3\times 3$ window size.

Non-local mean denoising [44]. As the name indicates, the non-local smoothing
not only uses the nearby pixels, but uses several similar patches within a
search window, as well, to smooth out the current patch. While preserving
image edges, it is assumed that averaging similar patches to smooth the
current patch will remove the perturbations when the mean of the noise is
zero. In our investigation, we use the patch size of $7\times 7$ and the
search window of $23\times 23$ and the strength of 0.07. We use skimage
restoration library to run the non-local mean denoising. For each noisy image,
i.e. the AE, the robust wavelet-based estimator of the Gaussian noise standard
deviation is used before applying the non-local mean (NLM) denoiser [49].

Total variation minimization [45]. Total variation denoising was shown to be
effective in removing adversarial perturbations due to its ability to minimize
the total variation of the image by producing similar images to the noisy
images. It was shown that total variation simultaneously preserves edges and
denoises flat regions. We use the Chambolle algorithm [50] that is implemented
in skimage library with strength of 0.1 and $\epsilon=2e-4$.

JPEG image compression [46, 47]. Compression is used to remove redundancy in
images by removing high frequency components that are imperceptible to humans.
On the other hand, adversarial perturbation is imperceptible and hence it is
assumed that the high frequency details of the image are more vulnerable to
adversarial perturbations. In our experiments, we compress the AEs at quality
level of 65%.

Cropping and re-scaling [48]. It was shown that image cropping and re-scaling
method is an effective way to remove the adversarial perturbations effect due
to the spatial re-positioning of the pixels. This method may harm the
structure of the carefully crafted adversarial perturbations. We perform
center cropping with 2-pixel margin for top, bottom, left, and right of the
image, and then re-scale the cropped image to the input size.

Finally, it is worth mentioning that the pre-processing defense can be easily
fooled using BPDA [33] and EOT [51] techniques. In our investigation, we
consider the EOT attack to have a deep insight in the robustness of the ViTs
based models.

###  3.4 Analysis tools

We rely our investigations on tools that can reveal the properties of the
model’s behavior such as feature [2], attention [4], and gradient-weighted
class activation mapping (Grad-CAM) [52] maps. Moreover, we use discrete
cosine transform (DCT) based decomposition of the perturbations [53] and the
visual quality assessment of the AE [54] as tools to assess the robustness of
target models against the AEs. We consider three objective visual quality
assessment metrics including peak signal to noise ratio (PSNR), structural
similarity index measure (SSIM) [55], and most apparent distortion (MAD) [56].

Feature and attention maps: In [2], a visualization method to visualize the
feature maps for CNNs and ViTs is recommended. For simplicity, we get the
feature maps of clean and adversarial samples and then visualize the
difference of one channel only, randomly selected, and finally normalize it to
[0, 1] scale. As recommended in [2], input sample is upsampled to clearly
visualize the ViT feature maps. To visualize the attention map, we follow the
rollout-based attention visualization method that is provided by [4] and
implemented in the ViT-pytorch github
repository444https://github.com/jeonsworld/ViT-pytorch.

Grad-CAM [52] uses the gradients of any target class flowing into the final
convolutional layer to produce a coarse localization map highlighting the
important regions in the image in the class prediction. Since convolutional
blocks are missing in ViTs, [57] uses the LN output of the last attention
block to calculate the gradients and then [57] reshapes the activation and
gradients to 2D spatial images to fit the Grad-CAM algorithm.

DCT-based decomposition of perturbations. In [53], it was shown that
construction of the class decision boundary is extremely sensitive to small
perturbations of the training samples. Moreover, it was shown that CNNs mainly
exploit discriminative features in the low frequencies of MNIST, CIFAR-10, and
ImageNet-1k datasets. This explains why, in some query-based black-box attacks
like in [58], using low-frequency perturbations improves the efficiency of the
attack’s query. To show how we take advantage of this observation in our
investigation, we consider the DCT decomposition of the perturbations for non
norm-constrained555In [53], to measure perturbation distance to the decision
boundary, they found that non norm-constrained attacks are more suitable for
the study. In our experiments, we show that, for ResNet, the perturbations’
discriminative features are more centered around low frequency in non norm-
constrained attacks. attacks, like CW attacks, and for norm-constrained
attacks, like FGSM and PGD-based attacks as shown in Figure 2 and Figure 3,
respectively. For ResNet, it is very clear that the perturbations’
discriminative features are more centered around low frequency components.
While, for ViTs, the perturbations’ discriminative features are spread in all
frequency spectrum region. We conclude that, the wider the spread of
perturbations’ discriminative features the more robust the model against the
attacks, because the adversarial attack algorithms have to affect a wider
range of frequency spectrum.

Clean Image | Perturbation from ViT-B-16 | Perturbation from ResNet50
---|---|---
|  |
|  |

Figure 2: The perturbation (top), generated using CW-$L_{\infty}$ attack with
ViT-B-16 (middle) and ResNet50(left), and the corresponding DCT-based spectral
decomposition heatmap. Perturbation is scaled from [-1, 1] to [0, 255].

Clean Image | Perturbation from T2T-ViT-14 | Perturbation from ResNet50
---|---|---
|  |
|  |

Figure 3: The perturbation (top), generated using PGD-$L_{\infty}$
$\epsilon=4/255$ attack with T2T-ViT-14 (middle) and ResNet50(left), and the
corresponding DCT-based spectral decomposition heatmap. Perturbation is scaled
from [-1, 1] to [0, 255].

PSNR. It is the most widely used metric because it is simple and
mathematically convenient for optimization. Unfortunately, PSNR doesn’t
correlate well with human perception. It is calculated as:

$\begin{gathered}PSNR=10\;log_{10}\left(\frac{MAX^{2}}{MSE}\right)\\\
MSE=\frac{1}{N}\sum_{i=1}^{m}\sum_{j=1}^{n}[x(i,j)-x^{\prime}(i,j)]^{2},\end{gathered}$
(3)

where $MAX$ is the maximum density value, $2^{8}-1$ in 8-bit images, $MSE$ is
the mean squared error, $m$ and $n$ are the image width and height, $N=mn$,
and PSNR $\in[0,\infty[$. The higher the PSNR the better the quality and
$PSNR\to\infty$ means that the two images are identical.

SSIM [55]. It is a full reference image quality metric that relies on the fact
that the human visual system (HVS) highly tends to extract structural
information from the image. SSIM extracts information that is related to the
luminance ($l$), the contrast ($c$), and the structure ($s$) from two images,
here $x$ and $x^{\prime}$, and measures the similarity between them as:

$\begin{gathered}SSIM(x,x^{\prime})=l(x,x^{\prime})c(x,x^{\prime})s(x,x^{\prime})\\\
=\frac{(2\mu_{x}\mu_{x^{\prime}}+c_{1})(2\sigma_{xx^{\prime}}+c_{2})}{(\mu_{x}^{2}+\mu_{x^{\prime}}^{2}+c_{1})(\sigma_{x}^{2}+\sigma_{x^{\prime}}^{2}+c_{2})},\end{gathered}$
(4)

where $\mu_{x}$ is the average of $x$, $\mu_{x^{\prime}}$ is the average of
$x^{\prime}$, $\sigma_{x}$ is the variance of $x$, $\sigma_{x^{\prime}}$ is
the variance of $x^{\prime}$, $\sigma_{xx^{\prime}}$ is the covariance of $x$
and $x^{\prime}$, and $c_{1}$ and $c_{2}$ are hyperparameter to avoid
instability when denominator is close to zero. The range of SSIM is $[0,1]$,
where $\acs{ssim}=1$ means that the two images are identical. In our
experiments, we assume that the target model $\mathcal{M}$ is more robust than
other models if the AE that is generated using the $\mathcal{M}$ has lower
SSIM score than other models because the adversarial attack algorithm has to
generate AE with higher perturbation that influences the structure of the AEs.

MAD [56]. Adversary usually generates AE that are imperceptible to human as
much as they can by adding perturbations that are near threshold distortions.
On the other hand, MAD attempts to explicitly model two strategies employed by
the HVS. The first one is a detection-based strategy for high-quality images
containing near threshold distortions666Some image quality assessment methods
assume that there is a distortion level at which the distortions start be
visible to the observers. When there is a distortion in an image and is not
perceptible, it is called near threshold distortion. When the distortion is
visible with high magnitude in the image, it is called suprathreshold
distortion. that are typical situation for AEs with low perturbation. The
second one is an appearance-based strategy for low-quality images containing
clearly suprathreshold distortions, that are typical situation for AEs with
high perturbation. As stated in [56], local luminance and contrast masking are
used to estimate detection-based perceived distortion in high-quality images,
whereas changes in the local statistics of spatial-frequency components are
used to estimate appearance-based perceived distortion in low-quality images.
Moreover, it was shown in [54] that MAD has better correlation to subjective
scores for AEs. The range of MAD score is $[0,\infty]$ where $\acs{mad}=0$ is
close to the clean image. In our experiments, we assume, as shown in Figure 4,
that the target model $\mathcal{M}$ is more robust than other models if the AE
that is generated using the $\mathcal{M}$ has higher MAD score than other
models because the adversarial attack algorithm don’t be able to prevent the
perception of the perturbation.

Figure 4: The attack success rate (ASR) of target models, on 1000 images from
ImageNet-1k against AutoAttack, in average $\epsilon=\\{1,2,4\\}/255$, and the
most apparent distortion (MAD)

Clean | ViT-S-16 | ViT-B-16 | ViT-L-16 | ViT-Res-16 | T2T-ViT-14 | T2T-ViT-24 | TNT-S-16 | ResNet50 | ResNeXt50 | VGG16
---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |

(a)  (b)

(c)

(d)

Figure 5: JSMA attack: The first row shows the clean sample and the AEs. The
clean image is correctly classified by tested models and all AEs are
successful attacks. (a) The perturbation (top) and the corresponding DCT-based
spectral decomposition heatmap. Perturbation is shown in black and white
colors only. (b) Grad-CAM of the clean (top) and AE samples. (c) Feature map
difference between clean and AE feature maps that are computed after the first
basic block of the model, attention block for ViTs and convolutional layer for
CNNs. (d) The attention map from last attention block for clean (top) and AE
samples.

|
---
|
---

(a)

(b)

Figure 6: JSMA attack: (a) AEs quality assessment measures. (b) The ASR of the
AEs and the top-1 error of the pre-processed AEs on 100 images from
imagenet-1k. SS: local spatial smoothing. NLM: non-local mean. TVM: total
variation minimization. JPEG: JPEG compression. CR: cropping and re-scaling.
CCP: color channel perturbations.

##  4 Reveal of the robustness attributes

###  4.1 General observation

In Figure 5 and Figure 7, for instance, the first row shows the AEs, while the
top row of Figures 5b and 7b, and Figures 5d, 7d represent the Grad-CAM and
the attention maps of the clean(top) and AEs imags, respectively. It is clear
that ViTs have the capability to track the global features of the input image
while CNNs track local and centered features. Figures 5a and 7a represent the
perturbations of the generated AEs for each model. It is clear that the
perturbations highly target the main object of the image, the goose or the dog
for both CNNs and vanilla ViTs. Moreover, the effect of the perturbation can
be noticed on the Grad-CAM. The impacted regions of the attention and Grad-CAM
maps span to regions that are not related to the main object of the image. On
the other hand, model’s feature maps were known to be affected by the
perturbations and propagate when the model goes deeper. As shown in [2], CNNs
and hybrid-ViTs tend to better learn low-level features. Figures 5c, and 7c
show the difference of clean and AE feature maps after the first convolutional
layer for CNNs and after the first attention block for ViTs. As feature maps
show, the perturbation impacted learned features, especially the low-level
features.

###  4.2 Vanilla ViTs are more robust against $L_{0}$-based attacks

Clean | ViT-S-16 | ViT-B-16 | ViT-L-16 | ViT-Res-16 | T2T-ViT-14 | T2T-ViT-24 | TNT-S-16 | ResNet50 | ResNeXt50 | VGG16
---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |

(a)

(b)

(c)

(d)

Figure 7: PGD-$L_{1}$ $\epsilon=400$ attack: The first row shows the clean
sample and the AEs. The clean image is correctly classified by tested models
and all AEs are successful attacks. (a) The perturbation (top) and the
corresponding DCT-based spectral decomposition heatmap. Perturbation is scaled
from [-1, 1] to [0, 255]. (b) Grad-CAM of the clean (top) and AE samples. (c)
Feature map difference between clean and AE feature maps that are computed
after the first basic block of the model, attention block for ViTs and
convolutional layer for CNNs. (d) The attention map from last attention block
for clean (top) and AE samples.

|
---
|
---

(a)

(b)

Figure 8: PGD-$L_{1}$ $\epsilon=400$ attack: (a) AEs quality assessment
measures. (b) The ASR of the AEs and the top-1 error of the pre-processed AEs
on 1000 images from ImageNet-1k. SS: local spatial smoothing. NLM: non-local
mean. TVM: total variation minimization. JPEG: JPEG compression. CR: cropping
and re-scaling. CCP: color channel perturbations.

$L_{0}$-based attacks rely on changing few image pixels in order to fool the
neural networks. Finding these few pixels is challenging and computationally
consuming process since the attack algorithms need to search over the image
space to identify these pixels. In this work we consider the white-box JSMA
attack [38], and generate 100 AEs from ImageNet-1k validation images and a
sample is shown in the first row of Figure 5. The results are investigated and
the following observations are noted.

JSMA identifies input features that significantly impact the classification
output. In this experiment, JSMA achieves 100% ASR for all tested models, see
the first column of Figure 6b. Hence, to study the robustness of a model
against $L_{0}$-based attacks, we analyzed 1) the perturbation DCT-based
spectral decomposition [53], see Figure 5a, that shows that ViTs have a wider
spread energy spectrum which confirms a) ViTs are less sensitive to low-level
features and exploit the global context of the image, and b) wider frequency
spectrum has to be affected to generate AE for ViT models and its variants, 2)
the visual quality measures of AEs in Figure 6a show that JSMA, for vanilla
ViTs, targeted less than 1% of the pixels ($L_{0}$) and has to increase the
densities ($L_{\infty}$) of the input pixels to more than 0.8, in order to
fool the models, which makes vanilla ViTs more robust against $L_{0}$-based
attacks than hybrid-ViTs.

Clean | ViT-S-16 | ViT-B-16 | ViT-L-16 | ViT-Res-16 | T2T-ViT-14 | T2T-ViT-24 | TNT-S-16 | ResNet50 | ResNeXt50 | VGG16
---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |

(a)

Figure 9: CW-$L_{2}$ attack: The first row shows the clean sample and the AEs.
The clean image is correctly classified by tested models and all AEs are
successful attacks. (a) The perturbation (top) and the corresponding DCT-based
spectral decomposition heatmap. Perturbation is scaled from [-1, 1] to [0,
255].

|
---
| .
---

(a)

(b)

Figure 10: CW-$L_{2}$ attack: (a) AEs quality assessment measures. (b) The ASR
of the AEs and the top-1 error of the pre-processed AEs on 1000 images from
ImageNet-1k. SS: local spatial smoothing. NLM: non-local mean. TVM: total
variation minimization. JPEG: JPEG compression. CR: cropping and re-scaling.
CCP: color channel perturbations.

On the other hand, in CNN, a higher number of pixels are targeted with a high
change in pixels intensities, which clearly affect the structure of the AE as
SSIM metric clearly shows. While, in hybrid-ViT models, less number of pixels
with low pixel intensity change is enough to generate an AE with high quality
relative to human perception as MAD measure shows. Moreover, a narrower spread
of spectrum energy has been noticed on the DCT-based decomposition, hence,
bringing convolutional layers into ViTs doesn’t increase robustness under the
$L_{0}$-based attacks.

###  4.3 T2T-ViT-24 and TNT-S-16 are more robust against $L_{1}$-based attacks

$L_{1}$-based attacks minimize the perturbation $\delta$, where
$\delta=||x-x^{\prime}||_{1}$, $\delta\leq\epsilon$, and $f(x)\neq
f(x^{\prime})$, where $f(.)$ is a prediction function. In this work we
consider the white-box PGD-$L_{1}$ attack [27], and generate 1000 AEs from
ImageNet-1k validation images and a sample is shown in the first row of Figure
7. In this experiment, we set $\epsilon$ to 400. As shown in the first column
of Figure 8b, PGD-$L_{1}$ achieves less ASR on ViT-L-16, T2T-ViT-24 and
TNT-S-16 models. As shown in Figure 7a, perturbations that are generated using
T2T-ViT-24 and TNT-S-16 models have wider frequency spectrum spread than ViT-
Res-16 and ViT-L-16 models, which makes them more robust than other models.
While perturbations that are generated using ResNets have a wider spread of
frequency spectrum than ViT-S/B-16 and T2T-ViT-14. From the visual quality
assessment point of view, see Figure 8a, 1) as SSIM scores show, perturbations
that are generated using T2T-ViT-24 and TNT-S-16 models have more influence to
alter the adversarial image structure than the structure of AEs generated by
other models, 2) as PSNR and MAD scores show, perturbations that are generated
using ViT variants, in general and specifically T2T-ViT-24 and TNT-S-16, have
higher $L_{\infty}$ than other tested models which yields to have lower image
quality of AE. VGG16 has the highest ASR which makes it less robust than other
models. One reason for that is that VGG16 has less accuracy performance than
other models which makes the learned features not robust. In this experiment,
ViT-Res-16 is excluded from the analysis since the image size is different.

Clean | ViT-S-16 | ViT-B-16 | ViT-L-16 | ViT-Res-16 | T2T-ViT-14 | T2T-ViT-24 | TNT-S-16 | ResNet50 | ResNeXt50 | VGG16
---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |

(a)

Figure 11: PGD-$L_{2}$ $\epsilon=2$ attack: The first row shows the clean
sample and the AEs. The clean image is correctly classified by tested models
and all AEs are successful attacks. (a) The perturbation (top) and the
corresponding DCT-based spectral decomposition heatmap. Perturbation is scaled
from [-1, 1] to [0, 255].

|
---
|
---

(a)

(b)

Figure 12: PGD-$L_{2}$ $\epsilon=2$ attack: (a) AEs quality assessment
measures. (b) The ASR of the AEs and the top-1 error of the pre-processed AEs
on 1000 images from imagenet-1k. SS: local spatial smoothing. NLM: non-local
mean. TVM: total variation minimization. JPEG: JPEG compression. CR: cropping
and re-scaling. CCP: color channel perturbations.

###  4.4 Vanilla ViTs are more robust under CW-$L_{2}$, while hybrid-ViTs are
more robust under PGD-$L_{2}$ attacks

$L_{2}$-based attacks minimize the perturbation $\delta$, where
$\delta=||x-x^{\prime}||_{2}$, $\delta\leq\epsilon$, and $f(x)\neq
f(x^{\prime})$. In this work we consider the white-box CW-$L_{2}$ [31] and
PGD-$L_{2}$ attacks [27], and generate 1000 AEs from ImageNet-1k validation
images and a sample is shown in the first row of Figure 9.

####  4.4.1 Robustness under CW-$L_{2}$ attacks

As shown in the the first column of Figure 10b, CW-$L_{2}$ attack achieves
comparable ASR on all tested models except for ViT-L-16 which achieves less
ASR. Compared to ResNet, the DCT decomposition of the perturbation shows the
huge wide spread of energy spectrum for vanilla ViTs, especially ViT-L-16.
Moreover, the algorithm of CW-$L_{2}$ has to increase $L_{2}$ and $L_{\infty}$
distortions, see Figure 10a, for ViTs in order to generate successful attacks.
Hence, SSIM and MAD show less quality score for AEs that are generated using
vanilla ViTs. ResNet shows robustness over T2T-ViT-14 since the perturbations
that are generated using ResNet have a wider spread of frequency spectrum than
T2T-ViT-14 but not wider than T2T-ViT-24 and TNT-S-16. The visual quality
scores, given in Figure 10a, confirm that ResNet is more robust than
T2T-ViT-14 but less robust than T2T-ViT-24 and TNT-S-16.

####  4.4.2 Robustness under PGD-$L_{2}$ attacks

Figure 11 shows an example to study the target models’ robustness against
PGD-$L_{2}$ attack with $\epsilon=2$. From the first column of Figure 12b, we
concluded that the hybrid-ViT are more robust than other models against
PGD-$L_{2}$ attacks. The ASR of the hybrid-ViTs is less than the ASR of the
other target models. Moreover, when we look at Figure 11a, we can see that
hybrid-ViTs have a wider spread of discriminative features on all frequencies
than other models. Moreover, the ViT-S/B/L-16 models show better robustness
over ResNet. When looking at Figure 12a, we can see that ViT variants have
higher $L_{\infty}$ score, especially the ViT-S-16, than ResNet which affects
the AE structure and visual quality as SSIM and MAD scores show. VGG16 has the
highest ASR and high distortion which makes it less robust. One reason for
that is that VGG16 has less accuracy performance than other models which makes
the learned features not robust. In this experiment, ViT-Res-16 is excluded
from the analysis since the image size is different.

Hence, we can conclude that neither larger model’s architecture, like ViT-L,
nor bringing convolutional modules for tokenization in ViTs, like TNT, will
necessarily enhance the robustness.

###  4.5 Hybrid-ViTs and small ViT are matters under some $L_{\infty}$-based
attacks.

$L_{\infty}$-based attacks minimize the perturbation $\delta$, where
$\delta=||x-x^{\prime}||_{\infty}$, $\delta\leq\epsilon$, and $f(x)\neq
f(x^{\prime})$. In this work we consider the white-box FGSM-$L_{\infty}$ [26],
CW-$L_{\infty}$ [31], PGD-$L_{\infty}$ [27], and UAP [39] attacks. Moreover,
two black box attacks are considered; RayS [41] and SA [40] attacks and the
AA-$L_{\infty}$ [30] hybrid attack is considered as well.

Clean | ViT-S-16 | ViT-B-16 | ViT-L-16 | ViT-Res-16 | T2T-ViT-14 | T2T-ViT-24 | TNT-S-16 | ResNet50 | ResNeXt50 | VGG16
---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |

(a)

Figure 13: PGD-$L_{\infty}$ $\epsilon=1/255$ attack: First row shows the clean
sample and the AEs. The clean image is correctly classified by tested models
and all AEs are successful attacks. (a) The perturbation (top) and the
corresponding DCT-based spectral decomposition heatmap. Perturbation is scaled
from [-1, 1] to [0, 255].

|
---
|
---

(a)

.

(b)

Figure 14: PGD-$L_{\infty}$ $\epsilon=1/255$ attack: (a) AEs quality
assessment measures. (b) The ASR of the AEs and the top-1 error of the pre-
processed AEs on 1000 images from imagenet-1k. SS: local spatial smoothing.
NLM: non-local mean. TVM: total variation minimization. JPEG: JPEG
compression. CR: cropping and re-scaling. CCP: color channel perturbations.

####  4.5.1 Robustness under PGD-$L_{\infty}$, AA, and UAP attacks

We generate 1000 AEs from ImageNet-1k validation images and a sample is shown
in the first row of Figure 13. PGD-$L_{\infty}$ attack achieves less ASR on
ViT-S-16 than other tested models. While ResNets have comparable robustness to
ViT-L-16, T2T-ViT-24, and TNT-S-16, and have better robustness over ViT-B-16,
ViT-Res-16, and T2T-ViT-14. The perturbations that are generated using
ViT-S-16 have the following properties: 1) more spread of energy spectrum as
DCT decomposition shows in Figure 13a , 2) lower visual quality as PSNR, SSIM
scores show in Figure 14a, and 3) higher $L_{1}$ score. $L_{1}$ score for ViT-
Res-16 is not considered in the analysis since the image size is different.
Similar to $L_{1}$-based attack, VGG16 has the highest ASR which makes it less
robust. VGG16 has the highest ASR and high distortion which makes it less
robust than other target tested models.

Under the AA and UAP attacks, we noticed that ViT-S-16 has better robustness
than other target models, as Figure 4 shows for AA, but due to space
limitation, we only visualized PGD-$L_{\infty}$ attacks.

####  4.5.2 Robustness under FGSM-$L_{\infty}$ attacks

Figure 15 shows an example to study the target models’ robustness against
FGSM-$L_{\infty}$ attack. Figure 16a, 18a shows the visual quality scores of
the AEs that are generated using FGSM-$L_{\infty}$ attack. While Figure 16b
shows the ASR of the AEs and the top-1 error of the pre-processed AEs that are
generated using FGSM-$L_{\infty}$ attack.

Figure 16b shows that hybrid-ViTs have lower ASR than other models. Vanilla
ViT-L-16 shows more robustness over ResNet and ReNeXt, while ResNet shows
robustness over vanilla ViT-S/B-16. The wider spread of the energy spectrum of
the DCT decomposition confirms the robustness of hybrid-ViTs over other
models, as shown in Figure 15a. Moreover, SSIM scores, illustrated in Figure
16a, show that hybrid-ViT has lower score than other models, except for VGG16.
VGG16 has the lowest robustness since it has the highest ASR and high
distortion as SSIM score indicates. In FGSM attack, it is hard to use PSNR and
MAD to judge the robustness of the target models since the $||\delta||_{2}$ of
all generated AEs are equal.

####  4.5.3 Robustness under CW-$L_{\infty}$ attacks

Figure 17 shows an example to study the target models’ robustness against
CW-$L_{\infty}$ attack. Figure 18a shows the visual quality scores of the AEs
that are generated using CW-$L_{\infty}$ attack. While Figure 18b shows the
ASR of the AEs and the top-1 error of the pre-processed AEs that are generated
using CW-$L_{\infty}$ attack.

For CW-$L_{\infty}$, Figure 18b shows that the ASR of the ViT-B/L-16 is lower
than the ASR of other models. On the other hand, the DCT decomposition, see
Figure 17a, shows that the spread of the discriminative features is wider on
ViT-B/L-16 than other models. Moreover, it is shown that T2T-ViT-24 and
TNT-S-16 have wider spread of DCT decomposition than ResNet. Finally, from
Figure 18a, we can conclude that CW-$L_{\infty}$ generates AEs with higher
perturbations for ViT-B/L-16 and for T2T-ViT-24 models than other model which
clearly affected the image structure and the visual perception as SSIM and MAD
scores indicate.

Clean | ViT-S-16 | ViT-B-16 | ViT-L-16 | ViT-Res-16 | T2T-ViT-14 | T2T-ViT-24 | TNT-S-16 | ResNet50 | ResNeXt50 | VGG16
---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |

(a)

Figure 15: FGSM-$L_{\infty}$ $\epsilon=1/255$ attack: The first row shows the
clean sample and the AEs. The clean image is correctly classified by tested
models and all AEs are successful attacks. (a) The perturbation (top) and the
corresponding DCT-based spectral decomposition heatmap. Perturbation is scaled
from [-1, 1] to [0, 255].

|
---
|
---

(a)

(b)

Figure 16: FGSM-$L_{\infty}$ $\epsilon=1/255$ attack: (a) AEs quality
assessment measures. (b) The ASR of the AEs and the top-1 error of the pre-
processed AEs on 1000 images from imagenet-1k. SS: local spatial smoothing.
NLM: non-local mean. TVM: total variation minimization. JPEG: JPEG
compression. CR: cropping and re-scaling. CCP: color channel perturbations.

####  4.5.4 Robustness under RayS attacks

Figure 19 shows an example to study the target models’ robustness against RayS
attack. Figure 20a shows the visual quality scores of the AEs that are
generated using RayS attack. While Figure 20b shows the ASR of the AEs and the
top-1 error of the pre-processed AEs that are generated using RayS attack.

For RayS attack, by looking at Figure 20b, Figure 19a, and Figure 20a, we can
conclude that hybrid-ViTs are more robust than other target models. The
figures show 1) T2T-ViT-24 has lower ASR than other models. 2) hybrid-ViTs
have wider energy spectrum spread than other models. 3) RayS generates AEs
with higher perturbations for hybrid-ViTs that are perceptible to human.

###  4.6 Transfer attacks: increasing the number of attention blocks reduces
the transferability

Recent studies in [17, 18, 19] showed that there is low transferability
between different models’ families. In this work we confirm that and the
results are shown in Figure 21a. Here, we show two new observations. The first
one is that the transferability, within the same model family, is becoming
lower when the model is becoming larger. Hence, adding more attention blocks
to ViT variant models reduces the effect of the transferability property, as
shown in Figure 21a. The second777Using source model to generate black box AEs
for a different target model is not popular in real-world but, here the goal
is to note the observation. observation is that the black box based AEs, RayS
and SA, that are generated using ViT variants are more transferable to CNNs
while the black box based AEs that are generated using CNN are much less
transferable to ViT variants. As shown in Figure 21b, we notice that when CNNs
serve as target models, last three columns, the ASR is higher than those of
when CNNs serve as the source model, last three rows. According to the DCT
decomposition of SA-based AEs, see Figure 22, one possible explanation to this
case is that the generated perturbations highly affect the local features than
global features and since ViTs are less sensitive to local features making
perturbation effect less transferable.

Clean | ViT-S-16 | ViT-B-16 | ViT-L-16 | ViT-Res-16 | T2T-ViT-14 | T2T-ViT-24 | TNT-S-16 | ResNet50 | ResNeXt50 | VGG16
---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |

(a)

Figure 17: CW-$L_{\infty}$ attack: The first row shows the clean sample and
the AEs. The clean image is correctly classified by tested models and all AEs
are successful attacks. (a) The perturbation (top) and the corresponding DCT-
based spectral decomposition heatmap. Perturbation is scaled from [-1, 1] to
[0, 255].

|
---
|
---

(a)

(b)

Figure 18: CW-$L_{\infty}$ attack: (a) AEs quality assessment measures. (b)
The ASR of the AEs and the top-1 error of the pre-processed AEs on 1000 images
from imagenet-1k. SS: local spatial smoothing. NLM: non-local mean. TVM: total
variation minimization. JPEG: JPEG compression. CR: cropping and re-scaling.
CCP: color channel perturbations.

###  4.7 ccp is an attack and defense!

The CCP attack is based on the color property of the image. It uses the
original color channels, Red $R$, Green $G$ and Blue $B$ of the image input
$x$ to generate the AE ($x^{\prime}$). The AE is composed of new transformed
color channels, $R^{\prime}$, $G^{\prime}$, and $B^{\prime}$ of the
transformed image ($x^{\prime}$). The transformed channels are calculated as
follows:

$\begin{gathered}R^{\prime}=s\left(\frac{\alpha^{r}R+\alpha^{g}G+\alpha^{b}B}{3}\right)+b,\\\
G^{\prime}=s\left(\frac{\beta^{r}R+\beta^{g}G+\beta^{b}B}{3}\right)+b,\\\
B^{\prime}=s\left(\frac{\gamma^{r}R+\gamma^{g}G+\gamma^{b}B}{3}\right)+b,\end{gathered}$
(5)

where $s$ is a scale factor hyperparameter, $b$ is a bias hyperparameter,
{$\alpha$, $\beta$, $\gamma$} $\in[0,1]$, and {$R^{\prime}$, $G^{\prime}$,
$B^{\prime}$} $\in\mathbb{R}$. The scale $s$ and bias $b$ are used to adjust
the visual appearance of the generated AE. Figure 23 shows examples for CCP
attack.

Images might be vulnerable to natural perturbations like color brightness
change. Table III shows the ASR of the target models against the CCP attack.
Although this attack doesn’t achieve high ASR when tested on neural network
models, it can reveal robustness of these models. The results in Table III
show that hybrid-ViTs have the highest robustness, while ResNets and
ViT-B/L-16 have comparable robustness. One explanation to that is that CCP
limits CNNs’ and vanilla ViTs’ capabilities to extract the local and global
features, respectively. Hence, enhancing the tokenization process of the
vanilla ViTs as in hybrid-ViTs has an added value to model’s robustness.
Surprisingly, we found that when applying CCP attack on AEs, CCP is able to
project the AEs back to its original manifold. Last columns of Figures 6b, 8b,
10b, and, 14b show the ASR after considering CCP as a preprocess method for
the AEs. Although removing the effect of perturbations depends on noise
amount, hybrid-ViTs show better performance on predicting the original class
due the presence of the tokenization process enhancements in hybrid-ViTs.
Future investigations are highly recommended to study this phenomenon since
brightness change is a common process in many real-world applications.

TABLE III: The ASR of the target models against color channel perturbations
(CCP).

| ViT-S-16 | ViT-B-16 | ViT-L-16 | ViT-Res-16 | T2T-ViT-14 | T2T-ViT-24 | TNT-S-16 | ResNet50 | ResNeXt50 | VGG16
---|---|---|---|---|---|---|---|---|---|---
ASR(%) | 22.6 | 15.2 | 12.5 | 5.4 | 10.1 | 7.1 | 8.1 | 11.9 | 14.9 | 32.6

TABLE IV: The top-1 error(%) of the preprocessing defense methods that are
applied to PGD-$L_{\infty}$ $\epsilon=4/255$ under the EOT. 1000 images from
imagenet-1k are used. SS: local spatial smoothing. NLM: non-local mean. TVM:
total variation minimization. JPEG: JPEG compression. CR: cropping and re-
scaling. EOT: expectation over transformation.

Model |  No defense | SS | NLM | TVM | JPEG | CR
---|---|---|---|---|---|---
EOT | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ | ✓
ViT-S-16 | 99.9 | 96.3 | 98 | 43.8 | 77.7 | 62 | 91.2 | 91.8 | 94.3 | 97.4 | 90.3
ViT-B-16 | 99.5 | 97.2 | 97.9 | 35.9 | 86.4 | 58.5 | 94.4 | 90.2 | 94.8 | 96.4 | 95.6
ViT-L-16 | 98.8 | 96.5 | 97 | 33.2 | 83.9 | 58.6 | 91.6 | 88.2 | 94 | 93.7 | 92
ViT-Res-16 | 100 | 96.7 | 99.6 | 12.3 | 89.6 | 34.1 | 97.8 | 57.5 | 96.9 | 98.1 | 98.2
T2T-ViT-14 | 99.9 | 58.1 | 96.8 | 22.8 | 77.1 | 24.3 | 87.4 | 25 | 77.4 | 74.7 | 70
T2T-ViT-24 | 99.6 | 52.3 | 94.6 | 18.1 | 66 | 21.2 | 82.1 | 23.9 | 70.1 | 67.4 | 63.8
TNT-S-16 | 99.7 | 80.7 | 97.8 | 27.1 | 78.8 | 34.2 | 91.1 | 38.4 | 82.2 | 83 | 79.7
ResNet50 | 98.7 | 61.4 | 95.7 | 26.9 | 81.3 | 26.3 | 89 | 27.9 | 85.4 | 86.8 | 79.4
ResNet50-32x4d | 98.4 | 56.3 | 95 | 29.4 | 79.8 | 25.4 | 87.2 | 22.5 | 79.3 | 81.8 | 71.9
VGG16 | 99.6 | 94.1 | 96.5 | 55 | 89.2 | 57.7 | 90.2 | 73.8 | 95.5 | 99.4 | 94.5

###  4.8 Vanilla ViTs are not responding to preprocessing defenses that mainly
reduce the high frequency components.

Preprocessing is one of the defense methods that is applied to the model’s
input to remove the effect of perturbations that are added to the input image.
In this experiment, we apply five preprocessing methods that are used in the
literature and briefly mentioned in Section 3.3. Samples of top-1 error after
perprocessing of some attacks are shown in Figures 6b, 8b, 10b, 12b, 14b, 16b,
18b, and 20b, while Figure 1 shows the top-1 error AA, in average. For
$L_{0}$-based and CW-$L_{2}$ attacks, SS and cropping and re-scaling (CR), as
expected, have the capability to remove the perturbations effect and to
project AEs back to input manifold, see Figure 6b, by re-positioning the
perturbation structure. The limited success of the preprocessing against
$L_{0}$-based AEs that are generated using VGG16 is due the large number of
the impacted pixels. For other attacks, NLM and total variation minimization
(TVM) show better performance on the preprocessing of AEs. That’s because
these two denoising methods try to restore the original image while preserving
the global image structure and contours. While other preprocessing methods
including SS, JPEG, and CR, are highly impacting the high frequency components
of the AEs, hence ResNets show better top-1 error over vanilla ViTs. Hybrid-
ViTs have lower top-1 error than vanilla ViTs and CNNs due to its power of
identifying global and local features. Figure 24 shows examples of the
preprocessing process for PGD-$L_{\infty}$ attacks.

###  4.9 Robustness under EOT attack

EOT is a framework that constructs AEs that remain adversarial over a chosen
transformation distribution $T$, i.e. the preprocessing defenses. When
processing defenses are used the stochastic gradients issue of the classifier
$f(.)$ arises and hence, to have successful attack, it is necessary to
estimate the gradient over the expected transformation to the input $t(x)$,
where, $t(.)$ is the transformation function. EOT optimizes the expectation
over the transformation $\mathbb{e}_{t\sim T}f(t(x))$ which can be solved by
minimizing the expected perceived distance as seen by the classifier
$\mathbb{E}_{t\sim T}[d(t(x^{\prime}),t(x))]$, where $d(.)$ is the distance
function.

We consider the distribution of transformations that includes SS, NLM, TVM,
JPEG, and CR. We 1) use 1000 images from the ImageNet-1k validation set, 2)
generate AEs using PGDs-$L_{\infty}$ $\epsilon=4/255$ and apply the
preprocessing defense methods for the generated AEs, 3) use EOT to synthesize
AEs that are robust over the given distribution $T$ and apply the
preprocessing defense methods for the synthesized AEs. Table IV shows the
top-1 error for the target models against the preprocessed AEs with and
without considering the EOT.

It is clear that the EOT kept the input samples as adversarial over the tested
transformations except for the cropping and re-scaling (CR) transformation.
For SS and JPEG , the top-1 error is highly increased in T2T-ViT and ResNets
models and the T2T-ViT models show less top-1 error. While for NLM, the small
ViT, T2T-ViT, and TNT models have less top-1 error compared to other models.
It is interesting to notice that, with the use EOT, the ResNet models have
less top-1 error than ViT-B/L models. One reason for that is that the
transformation under the EOT highly impacts the global structure of the input
sample. For TVM, the top-1 error of T2T-ViT models is less than other target
models. The top-1 error for using CR under the EOT is comparable to the top-1
error without CR. One reason for that is that CR, under any framework, targets
restructuring the adversarial sample by re-positioning the input pixels.

In general, we conclude that T2T-ViT and TNT models show better robustness
than vanilla ViTs and ResNets under the EOT robustness test.

Clean | ViT-S-16 | ViT-B-16 | ViT-L-16 | ViT-Res-16 | T2T-ViT-14 | T2T-ViT-24 | TNT-S-16 | ResNet50 | ResNeXt50 | VGG16
---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |

(a)

Figure 19: RayS attack: The first row shows the clean sample and the AEs. The
clean image is correctly classified by tested models and all AEs are
successful attacks. (a) The perturbation (top) and the corresponding DCT-based
spectral decomposition heatmap. Perturbation is scaled from [-1, 1] to [0,
255].

|
---
|
---

(a)

(b)

Figure 20: RayS attack: (a) AEs quality assessment measures. (b) The ASR of
the AEs and the top-1 error of the pre-processed AEs on 1000 images from
imagenet-1k. SS: local spatial smoothing. NLM: non-local mean. TVM: total
variation minimization. JPEG: JPEG compression. CR: cropping and re-scaling.
CCP: color channel perturbations.

|
---
|
---

(a)

(b)

Figure 21: ASR of transfer attack using FGSM and SA on 1000 images from
ImageNet-1k . The row represents the source model that is used to generate AE.
The column represents the target model. Dark blue column means that the target
model is vulnerable to transfer attacks.

Clean | ViT-S-16 | ViT-B-16 | ViT-L-16 | ViT-Res-16 | T2T-ViT-14 | T2T-ViT-24 | TNT-S-16 | ResNet50 | ResNeXt50 | VGG16
---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |

(a)

Figure 22: SA-$L_{\infty}$ $\epsilon=8/255$ attack: The first row shows the
clean sample and the AEs. The clean image is correctly classified by tested
models and all AEs are successful attacks. (a) The perturbation (top) and the
corresponding DCT-based spectral decomposition heatmap. Perturbation is scaled
from [-1, 1] to [0, 255].

|  |
---|---|---

Figure 23: Examples for CCP attack. Images are from ImageNet-1k validation
set. Figure 24: Examples of the defense preprocessing. The top row shows the
clean, AE, and the preprocessed AE. The middle row shows the Grad-CAM of the
clean, AE, and the preprocessed AE. The bottom row shows the energy spectrum
of the DCT decomposition of AE, and the preprocessed AE. Red label means the
image is misclassified.

##  5 Conclusion

In this paper, we have studied the robustness of vision transformers and their
variants and compared them with CNNs. Our analysis showed 1) either vanilla
ViTs or hybrid-ViTs are more robust than CNNs against $L_{p}$-based attacks
and CCP attacks. We analysed the energy spectrum of DCT decomposition of the
perturbations and the different visual quality measures. 2) CCP can be used as
a preprocessing defense method. 3) Increasing the number of attention blocks
will increase the robustness against the transfer attacks but not against
white box attacks. 4) Enhancing ViT tokenization might not increase the
robustness against the AEs but will increase the robustness under the
preprocessed AEs.

## Acknowledgments

The project is funded by both Région Bretagne (Brittany region), France, and
direction générale de l’armement (DGA).

## References

  * [1] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [2] L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, F. E. H. Tay, J. Feng, and S. Yan, “Tokens-to-Token ViT: Training vision transformers from scratch on imagenet,” _CoRR_ , vol. abs/2101.11986, 2021.
  * [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems 30, December 4-9, 2017, Long Beach, CA, USA_ , 2017, pp. 5998–6008.
  * [4] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * [5] C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting unreasonable effectiveness of data in deep learning era,” in _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_. IEEE Computer Society, 2017, pp. 843–852.
  * [6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, in _CVPR09_ , 2009\.
  * [7] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang, “Transformer in transformer,” _CoRR_ , vol. abs/2103.00112, 2021.
  * [8] H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang, “CvT: Introducing convolutions to vision transformers,” _CoRR_ , vol. abs/2103.15808, 2021.
  * [9] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Y. Bengio and Y. LeCun, Eds., 2015.
  * [10] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [11] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “MobileNets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [12] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , 2014.
  * [13] N. Carlini and D. Wagner, “Adversarial examples are not easily detected: Bypassing ten detection methods,” in _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , 2017, pp. 3–14.
  * [14] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry, “Adversarial examples are not bugs, they are features,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 125–136.
  * [15] N. Akhtar and A. Mian, “Threat of adversarial attacks on deep learning in computer vision: A survey,” _IEEE Access_ , vol. 6, pp. 14 410–14 430, 2018.
  * [16] H. X. Y. M. Hao-Chen, L. D. Deb, H. L. J.-L. T. Anil, and K. Jain, “Adversarial attacks and defenses in images, graphs and text: A review,” _International Journal of Automation and Computing_ , vol. 17, no. 2, pp. 151–178, 2020.
  * [17] S. Bhojanapalli, A. Chakrabarti, D. Glasner, D. Li, T. Unterthiner, and A. Veit, “Understanding robustness of transformers for image classification,” _CoRR_ , vol. abs/2103.14586, 2021.
  * [18] R. Shao, Z. Shi, J. Yi, P. Chen, and C. Hsieh, “On the adversarial robustness of visual transformers,” _CoRR_ , vol. abs/2103.15670, 2021.
  * [19] K. Mahmood, R. Mahmood, and M. van Dijk, “On the robustness of vision transformers to adversarial examples,” 2021.
  * [20] S. Paul and P. Chen, “Vision transformers are robust learners,” _CoRR_ , vol. abs/2105.07581, 2021.
  * [21] X. Yuan, P. He, Q. Zhu, and X. Li, “Adversarial examples: Attacks and defenses for deep learning,” _IEEE transactions on neural networks and learning systems_ , vol. 30, no. 9, pp. 2805–2824, 2019.
  * [22] A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay, and D. Mukhopadhyay, “Adversarial attacks and defences: A survey,” _CoRR_ , vol. abs/1810.00069, 2018.
  * [23] A. Aldahdooh, W. Hamidouche, S. A. Fezza, and O. Déforges, “Adversarial example detection for DNN models: A review,” _CoRR_ , vol. abs/2105.00203, 2021.
  * [24] F. Alamri, S. Kalkan, and N. Pugeault, “Transformer-encoder detector module: Using context to improve robustness to adversarial attacks on object detection,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 9577–9584.
  * [25] S. Ren, K. He, R. B. Girshick, and J. Sun, “Faster R-CNN: towards real-time object detection with region proposal networks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 39, no. 6, pp. 1137–1149, 2017.
  * [26] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.
  * [27] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in _6th International Conference on Learning Representations, ICLR 2018, Vancouver, Canada, 2018_. OpenReview.net, 2018.
  * [28] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 248–255.
  * [29] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby, “Big transfer (BiT): General visual representation learning,” in _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V_ , ser. Lecture Notes in Computer Science, vol. 12350. Springer, 2020, pp. 491–507.
  * [30] F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” in _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , ser. Proceedings of Machine Learning Research, vol. 119\. PMLR, 2020, pp. 2206–2216.
  * [31] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _2017 ieee symposium on security and privacy (sp)_. IEEE, 2017, pp. 39–57.
  * [32] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, “Boosting adversarial attacks with momentum,” in _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_. Computer Vision Foundation / IEEE Computer Society, 2018, pp. 9185–9193.
  * [33] A. Athalye, N. Carlini, and D. A. Wagner, “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,” in _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , ser. Proceedings of Machine Learning Research, vol. 80. PMLR, 2018, pp. 274–283.
  * [34] D. Hendrycks and K. Gimpel, “Gaussian error linear units (GELUs),” _CoRR_ , vol. abs/1606.08415, 2016.
  * [35] R. Wightman, “Pytorch image models,” https://github.com/rwightman/pytorch-image-models, 2019.
  * [36] S. Xie, R. B. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society, 2017, pp. 5987–5995.
  * [37] Z. Li, W. Yang, S. Peng, and F. Liu, “A survey of convolutional neural networks: Analysis, applications, and prospects,” _CoRR_ , vol. abs/2004.02806, 2020.
  * [38] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, “The limitations of deep learning in adversarial settings,” in _2016 IEEE European symposium on security and privacy (EuroS &P)_. IEEE, 2016, pp. 372–387.
  * [39] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, “Universal adversarial perturbations,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1765–1773.
  * [40] M. Andriushchenko, F. Croce, N. Flammarion, and M. Hein, “Square attack: a query-efficient black-box adversarial attack via random search,” in _European Conference on Computer Vision_. Springer, 2020, pp. 484–501.
  * [41] J. Chen and Q. Gu, “Rays: A ray searching method for hard-label adversarial attack,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 1739–1747.
  * [42] J. Kantipudi, S. R. Dubey, and S. Chakraborty, “Color channel perturbation attacks for fooling convolutional neural networks and a defense against such attacks,” _IEEE Transactions on Artificial Intelligence_ , vol. 1, no. 2, pp. 181–191, 2020.
  * [43] F. Croce and M. Hein, “Minimally distorted adversarial examples with a fast adaptive boundary attack,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 2196–2205.
  * [44] W. Xu, D. Evans, and Y. Qi, “Feature squeezing: Detecting adversarial examples in deep neural networks,” in _25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-21, 2018_. The Internet Society, 2018\.
  * [45] C. Guo, M. Rana, M. Cissé, and L. van der Maaten, “Countering adversarial images using input transformations,” in _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018.
  * [46] G. K. Dziugaite, Z. Ghahramani, and D. M. Roy, “A study of the effect of JPG compression on adversarial images,” _CoRR_ , vol. abs/1608.00853, 2016.
  * [47] N. Das, M. Shanbhogue, S. Chen, F. Hohman, L. Chen, M. E. Kounavis, and D. H. Chau, “Keeping the bad guys out: Protecting and vaccinating deep learning with JPEG compression,” _CoRR_ , vol. abs/1705.02900, 2017.
  * [48] A. Graese, A. Rozsa, and T. E. Boult, “Assessing threat of adversarial examples on deep neural networks,” in _2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA)_. IEEE, 2016, pp. 69–74.
  * [49] D. L. Donoho and J. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” _biometrics_ , vol. 81, no. 3, pp. 425–455, 1994.
  * [50] A. Chambolle, “An algorithm for total variation minimization and applications,” _Journal of Mathematical imaging and vision_ , vol. 20, no. 1, pp. 89–97, 2004.
  * [51] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok, “Synthesizing robust adversarial examples,” in _International conference on machine learning_. PMLR, 2018, pp. 284–293.
  * [52] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” _International Journal of Computer Vision_ , vol. 128, no. 2, p. 336–359, Oct 2019.
  * [53] G. Ortiz-Jiménez, A. Modas, S. Moosavi-Dezfooli, and P. Frossard, “Hold me tight! influence of discriminative features on deep network boundaries,” in _Advances in Neural Information Processing Systems 33: NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.
  * [54] S. A. Fezza, Y. Bakhti, W. Hamidouche, and O. Déforges, “Perceptual evaluation of adversarial attacks for cnn-based image classification,” in _2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX)_. IEEE, 2019, pp. 1–6.
  * [55] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [56] E. C. Larson and D. M. Chandler, “Most apparent distortion: full-reference image quality assessment and the role of strategy,” _Journal of electronic imaging_ , vol. 19, no. 1, p. 011006, 2010.
  * [57] J. Gildenblat and contributors, “Pytorch library for CAM methods,” https://github.com/jacobgil/pytorch-grad-cam, 2021.
  * [58] Y. Sharma, G. W. Ding, and M. A. Brubaker, “On the effectiveness of low frequency perturbations,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_. ijcai.org, 2019, pp. 3389–3396.

  *[ViT]: vision transformer
  *[ViTs]: vision transformer
  *[CNNs]: convolutional neural network
  *[AEs]: adversarial example
  *[EOT]: expectation over transformation
  *[NLP]: natural language processing
  *[T2T-ViT]: tokens-to-token ViT
  *[TNT]: transformer-in-transformer
  *[CNN]: convolutional neural network
  *[AE]: adversarial example
  *[DL]: deep learning
  *[ASR]: attack success rate
  *[CCP]: color channel perturbations
  *[SS]: local spatial smoothing
  *[JPEG]: JPEG compression
  *[Fast-RCNN]: fast region convolutional neural network
  *[FGSM]: fast gradient sign method
  *[PGD]: projected gradient descent
  *[BiT]: big transfer
  *[Auto-PGD]: auto projected gradient descent
  *[CW]: Carlini-Wagner
  *[MI-FGSM]: momentum iterative gradient-based method
  *[BPDA]: backward pass differentiable approximation
  *[SAGA]: self-attention gradient attack
  *[T2T-ViTs]: tokens-to-token ViT
  *[MLP]: multi-layer perceptron
  *[SHSA]: single-head self-attention
  *[MHSA]: multi-head self-attention
  *[GELU]: Gaussian error linear units
  *[JSMA]: jacobian saliency map attack
  *[UAP]: universal adversarial perturbations
  *[SA]: square attack
  *[RayS]: ray searching
  *[AA]: autoattack
  *[AT]: adversarial training
  *[BIM]: basic iterative method
  *[NLM]: non-local mean
  *[Grad-CAM]: gradient-weighted class activation mapping
  *[DCT]: discrete cosine transform
  *[PSNR]: peak signal to noise ratio
  *[SSIM]: structural similarity index measure
  *[MAD]: most apparent distortion
  *[CR]: cropping and re-scaling
  *[TVM]: total variation minimization
  *[PGDs]: projected gradient descent