Hashmat Shadab<EMAIL_ADDRESS>Shahina K
<EMAIL_ADDRESS>Muzammal
<EMAIL_ADDRESS>1,2 Salman<EMAIL_ADDRESS>1,2
Fahad Shahbaz<EMAIL_ADDRESS>1,3 Mohamed bin Zayed University of
AI
Australian National University Linköping University Adversarial Pixel
Restoration

# Adversarial Pixel Restoration as a Pretext Task for Transferable
Perturbations

###### Abstract

Transferable adversarial attacks optimize adversaries from a pretrained
surrogate model and known label space to fool the unknown black-box models.
Therefore, these attacks are restricted by the availability of an effective
surrogate model. In this work, we relax this assumption and propose
_Adversarial Pixel Restoration_ as a self-supervised alternative to train an
effective surrogate model from scratch under the condition of _no_ labels and
_few_ data samples. Our training approach is based on a min-max scheme which
reduces overfitting via an adversarial objective and thus optimizes for a more
generalizable surrogate model. Our proposed attack is complimentary to the
adversarial pixel restoration and is independent of any task specific
objective as it can be launched in a self-supervised manner. We successfully
demonstrate the adversarial transferability of our approach to Vision
Transformers as well as Convolutional Neural Networks for the tasks of
classification, object detection, and video segmentation. Our training
approach improves the transferability of the baseline unsupervised training
method by 16.4% on ImageNet val. set. Our codes & pre-trained surrogate models
are available at: https://github.com/HashmatShadab/APR.

##  1 Introduction

Adversarial attacks [Szegedy et al.(2014)Szegedy, Zaremba, Sutskever, Bruna,
Erhan, Goodfellow, and Fergus, Goodfellow et al.(2015)Goodfellow, Shlens, and
Szegedy, Papernot et al.(2016b)Papernot, McDaniel, Jha, Fredrikson, Celik, and
Swami, Moosavi-Dezfooli et al.(2016)Moosavi-Dezfooli, Fawzi, and Frossard,
Carlini and Wagner(2017), Madry et al.(2018)Madry, Makelov, Schmidt, Tsipras,
and Vladu, Athalye et al.(2018)Athalye, Carlini, and Wagner] add small,
imperceptible but well optimized noise to the clean image which can elicit an
incorrect decision from the model. These attack methods [Poursaeed et
al.(2018)Poursaeed, Katsman, Gao, and Belongie, Dong et al.(2017)Dong, Liao,
Pang, Hu, and Zhu, Xie et al.(2019)Xie, Zhang, Zhou, Bai, Wang, Ren, and
Yuille, Inkawhich et al.(2020)Inkawhich, Liang, Wang, Inkawhich, Carin, and
Chen, Gao et al.(2020)Gao, Zhang, Song, and Shen, Wang et al.(2021)Wang, He,
Wang, and He] craft adversarial examples that can broadly be categorized based
on how much information is available about the target model. In a _white-box_
attack setting, the attacker has complete knowledge of the target model and
can directly optimize adversarial perturbations for the given model. In a more
realistic _black-box_ attack setting, the attacker does not have access to the
target model, its architectural details or targeted task (_e.g_ ,
classification, segmentation or object detection). In such a case, adversarial
examples are created on a surrogate model and then transferred to the black-
box model. Adversarial examples generated from surrogate models trained on a
large-scale dataset in a supervised manner have better transferability [Naseer
et al.(2019)Naseer, Khan, Khan, Shahbaz Khan, and Porikli, Dong et
al.(2018)Dong, Liao, Pang, Su, Zhu, Hu, and Li]. Transferability of such
attacks improves further by fine-tuning the surrogate model to enhance their
representation capacity _e.g_ , by finding better self-ensemble from a given
pretrained model [Naseer et al.(2021c)Naseer, Ranasinghe, Khan, Khan, and
Porikli]. The adversarial transferability hence depends on the
generalizability of the surrogate model. Such attacks are also restricted by
the availability of a pretrained surrogate and information about the label
space.

In this work, we call into question this assumption and consider a stronger
threat model where an attack is launched from few unannotated or cross-domain
samples (_e.g_ , painting to ImageNet samples) without any knowledge of target
networks or tasks. This threat model poses a challenge on how to learn an
effective surrogate model from the limited unlabelled data and then how to
generate self-supervised transferable adversarial examples.

With limited availability of data, neural networks can easily memorize the
data [Zhang et al.(2021)Zhang, Bengio, Hardt, Recht, and Vinyals] even with
strong augmentations [Li et al.(2020)Li, Guo, and Chen]. Therefore, Li _et
al_[Li et al.(2020)Li, Guo, and Chen] propose to reconstruct transformed input
images to learn a surrogate model. However, their effective surrogate training
and attack approach requires supervision though annotated data samples. In
order to reduce overfitting over few data samples and to find robust features,
we take inspiration from adversarial training [Goodfellow et
al.(2015)Goodfellow, Shlens, and Szegedy, Springer et al.(2021a)Springer,
Mitchell, and Kenyon] and propose self-supervised Adversarial Pixel
Restoration to train a surrogate model. The min-max objective of our proposed
training allows to find a flatter minima with robust features which
compliments our self-supervised adversarial attack to achieve higher
adversarial transferability. Our main contributions are as follows:

  * •

We propose self-supervised Adversarial Pixel Restoration to find highly
transferable patterns by learning over flatter loss surfaces. Our training
approach allows launching cross-domain attacks without access to large-scale
labeled data or pretrained models.

  * •

Our adversarial attack is self-supervised in nature and independent of any
task-specific objective. For instance, our approach optimizes the robust
transformed loss surface of the surrogate via fooling its reconstruction
ability. This allows to generate task independent adversaries. Therefore, our
approach can transfer perturbations to a variety of tasks as we demonstrate
for classification, object detection, and segmentation.

  * •

We provide a thorough analysis of our proposed method to establish it’s
effectiveness. We observe that our approach leads to smoother loss landscapes
(Fig. 2), helping in crafting more generalizable adversarial examples. Our
method remains effective even in extreme data scarcity _e.g_ when trained on
two data samples only. (see Sec. 4.2).

##  2 Related Work

Several gradient-based methods [Goodfellow et al.(2015)Goodfellow, Shlens, and
Szegedy, Kurakin et al.(2018)Kurakin, Goodfellow, and Bengio, Kurakin et
al.(2016)Kurakin, Goodfellow, and Bengio, Tramèr et al.(2017)Tramèr, Kurakin,
Papernot, Goodfellow, Boneh, and McDaniel] have been proposed for crafting
adversarial examples directly on the target classifiers. However, when the
access to the target model is limited to just a finite amount of queries,
current methods either rely on the transferability of surrogate models [Zhou
et al.(2020)Zhou, Wu, Liu, Liu, and Zhu, Papernot et al.(2016a)Papernot,
McDaniel, and Goodfellow, Papernot et al.(2017)Papernot, McDaniel, Goodfellow,
Jha, Celik, and Swami] or estimate the gradients/boundary of the target model
[Chen et al.(2017)Chen, Zhang, Sharma, Yi, and Hsieh, Brendel et
al.(2018)Brendel, Rauber, and Bethge, Narodytska and Kasiviswanathan(2017)].
Both of the above approaches either require a non-trivial number of queries
from the target model or access to the training distribution, making it highly
impractical in real-case scenarios. A practical threat model was introduced in
[Naseer et al.(2019)Naseer, Khan, Khan, Shahbaz Khan, and Porikli], where the
attacker does not have access to the training distribution of the target
model, as well as querying is prohibited. Authors train a generator-based
surrogate model with the help of training data and a pretrained classifier
obtained from a different domain than the target models. However, the
pretrained classifier as well as the generator are trained on a large
annotated dataset.

In [Li et al.(2020)Li, Guo, and Chen], a stronger threat model with access to
limited data samples (order of tens) was proposed. Inspired from self-
supervised learning methods, autoencoder-based surrogate models are trained
with limited data. However, the transferability of the autoencoders trained in
an unsupervised manner is still moderate. Most of the previous works have
explored the transferability of surrogate models trained on the target model’s
training set. In [Springer et al.(2021b)Springer, Mitchell, and Kenyon], the
authors observe that the features of robust classifiers can be used to
generate adversarial examples that are more transferable. Building on this,
[Springer et al.(2021a)Springer, Mitchell, and Kenyon] observe that having a
classifier adversarially trained with a small perturbation budget (_"slightly
robust"_), leads to highly transferable adversarial examples. Unlike prior
works, we focus on constructing robust surrogate models in a fully
unsupervised manner. We consider the case of training robust model with
limited data to improve adversarial transferability. Furthermore, we also
consider a practical scenario of availability of large unlabelled dataset for
training of robust models which can be used for constructing cross-domain and
cross-task adversaries.

##  3 Adversarial Pixel Restoration for Transferable Perturbations

Our goal is to learn a surrogate model, $\mathcal{F}$, from a given unlabelled
data distribution, $P_{s}$, with a set of only few image samples ($\leq 20$).
This setting is in contrast to the existing transferable adversarial attacks
[Dong et al.(2018)Dong, Liao, Pang, Su, Zhu, Hu, and Li, Naseer et
al.(2021c)Naseer, Ranasinghe, Khan, Khan, and Porikli, Xie et al.(2019)Xie,
Zhang, Zhou, Bai, Wang, Ren, and Yuille, Naseer et al.(2021a)Naseer, Khan,
Hayat, Khan, and Porikli] that assume access to a surrogate model trained on a
large-scale annotated data (_e.g_ , ImageNet [Deng et al.(2009)Deng, Dong,
Socher, Li, Li, and Fei-Fei]) in a supervised fashion. In the presence of
unlabeled data, however, the surrogate model can be trained by defining a
self-supervised task, $\mathcal{T}_{s}$, such as predicting rotation [Gidaris
et al.(2018)Gidaris, Singh, and Komodakis], solving jigsaw puzzle [Noroozi and
Favaro(2016)] or by matching different views of the same input sample [Caron
et al.(2021)Caron, Touvron, Misra, Jégou, Mairal, Bojanowski, and Joulin]. A
major challenge is that deep neural networks can easily memorise the data
[Zhang et al.(2021)Zhang, Bengio, Hardt, Recht, and Vinyals] and quickly
overfit the few available samples even after applying strong data augmentation
techniques [Li et al.(2020)Li, Guo, and Chen]. This results in a surrogate
model with less generalizable representations and consequently, adversarial
attack launched from such a model has weak transferability (Sec. 4.1.1).

We propose adversarial pixel restoration as a prior to train the surrogate
model $\mathcal{F}$ that boosts transferability of the adversarial attacks
(see Fig. 1 ). We create adversarial examples from the original input images
by attacking the source model in the pixel space using our proposed
adversarial attack (Sec. 3.2.1). Our approach also shifts the position of
input pixels via transformations such as rotation or jigsaw shuffle. We then
train a model to denoise and restore pixels to their original positions via
our proposed adversarial training (Algo. 1). The surrogate model $\mathcal{F}$
in our case consists of an autoencoder as explained below.

Figure 1: Our approach trains an autoencoder based surrogate model via self-
supervised adversarial pixel restoration to learn generalizable
representations from a limited number of data samples ($\leq 20$). Our
training is based on a min-max strategy. We first generate adversarial
examples by fooling model’s reconstructive ability _(maximization)_ , followed
by updating the model parameters based on restoration of the transformed
adversarial and clean sample _(minimization)_. Our approach allows launching
transferable self-supervised adversarial attacks without any knowledge of
target (black-box) model.

###  3.1 Surrogate Architecture

The surrogate model is based on an autoencoder [Li et al.(2020)Li, Guo, and
Chen]. The encoder consists of a stack of convolution layers. At the beginning
of the architecture, convolution layers with a larger kernel size and stride
help reduce the spatial resolution of the feature maps, followed by multiple
residual blocks where the size of the feature maps is kept constant. The
decoder is a lightweight model consisting of two transpose convolution layers
to upsample the feature maps. The adversarial training of these autoencoders
is based on denoising and pixel restoration as explained next in Sec. 3.2.

###  3.2 Adversarial Training via Denoising

The feature space of a slightly robust classifier produces highly transferable
adversarial examples [Springer et al.(2021a)Springer, Mitchell, and Kenyon].
However, adversarial training of such models is computationally demanding due
to iterative training and also requires a large-scale labeled dataset that
might not be accessible to the attacker in real-world scenarios. We assume a
more practical threat model, where the attacker has access to a data
distribution $P_{s}$ with limited number of samples without any annotations.
We adversarially learn a surrogate model on the data distribution $P_{s}$ to
transfer adversarial perturbations to the target (black-box) models trained
for different tasks (_e.g_ , classifications, object detection and
segmentation) on possibly different target distribution $Q_{t}\neq P_{s}$. In
such challenging attack settings, we adversarially train an autoencoder
$\mathcal{F}$ via min-max training strategy [Goodfellow et
al.(2015)Goodfellow, Shlens, and Szegedy]. At maximization step, we create
adversarial examples by fooling the model $\mathcal{F}$ via adversarial pixel
transformations with a single step attack (Algo. 1). At minimization step, we
denoise and restore the feature and pixel space to achieve generalizable loss
surfaces (Fig. 2) which leads to more transferable attack.

Algorithm 1 Adversarial Denoising and Restoration

1:Source data distribution $P_{s}$, pixel transformation $\mathcal{T}_{s}$,
attack step size $\delta$, perceptual budget $\epsilon$, balancing paramter
$\lambda$, and maximum training iterations $T$

2:Randomly initialize surrogate model $\mathcal{F}$

3:for $t\in[1,2,\ldots T]$ do

4: Randomly sample from $P_{s}$: $\bm{x}\sim P_{s}$

5: Initialize adversary $\bm{x}^{\prime}\leftarrow$ $\bm{x}$

6: Optimize $\bm{x}^{\prime}$ using Eq. 1 or 2: $\triangleright$ Adversarial
Pixel Transformation
$\bm{x}^{\prime}\leftarrow\bm{x}+\delta\times\text{sign}\left(\nabla\mathcal{L}_{max}\right)$

7: Project adversaries within allowed perceptual budget:
$\bm{x}^{\prime}\leftarrow\text{clip}\left(\bm{x}^{\prime},\bm{x}-\epsilon,\bm{x}+\epsilon\right)$

8: Forward-pass $\bm{x}$ and $\bm{x}^{\prime}$ through model $\mathcal{F}$ and
update its parameters $\theta$ by minimizing the loss given in Eq. 5:
$\triangleright$ Pixel Restoration
$\theta\leftarrow\theta-\alpha\times\nabla\mathcal{L}_{min},$ where $\alpha$
is the learning rate.

9:end for

####  3.2.1 Adversarial Pixel Transformations

For given input samples $\bm{x}\sim P_{s}$, we first find adversarial example,
$\bm{x}^{\prime}\;\;\text{subjected
to:}\;\;\|\bm{x}-\bm{x}^{\prime}\|_{\infty}\leq\epsilon$, by maximizing the
following objective ($\mathcal{L}_{max}$):

$\displaystyle\underset{\bm{x}^{\prime}}{\text{maximize}}\quad\mathcal{L}_{max}=\|\mathcal{F}\left(\mathcal{T}_{s}(\bm{x}^{\prime})\right)-\bm{x}\|^{p},$
(1)

where $\mathcal{T}_{s}$ represents the pixel transformation (_e.g_ , rotation
or jigsaw shuffle) that shifts the pixel positions. Therefore, our attack
fools the model’s ability to restore the transformed pixel space by maximizing
the loss presented in Eq. 1 and ultimately help to robustify self-supervised
features. Our attack approach can also be extended to benefit from supervisory
signals _e.g_ , by fooling prototypes [Snell et al.(2017)Snell, Swersky, and
Zemel, Li et al.(2020)Li, Guo, and Chen] as follows:

$\underset{\bm{x}^{\prime}}{\text{maximize}}\quad\sum^{C}_{c=1}\left(y_{c}\,\|\mathcal{F}\left(\mathcal{T}_{s}(\bm{x}^{\prime})\right)-\bm{x}^{(c)}\|^{p}\right),$
(2)

where $C$ represents the number of categories, $y_{c}$ represents one-hot
encoded labels, and $\bm{x}^{(c)}$ is the chosen prototype for a particular
class. Our proposed attack objective in Eq. 2 helps to optimize for robust
discriminative features with better adversarial transferability.

####  3.2.2 Pixel Restoration

For a given adversarial sample, $\bm{x}^{\prime}$ created using Eqs. 1 or 2,
we train the surrogate model, $\mathcal{F}$, by pixel restoration. Our loss
function penalizes the model $\mathcal{F}$ by minimizing the reconstruction
error between the original sample, $\bm{x}$, and the model’s output for the
transformed adversarial as well as the transformed original sample as follows:

$\mathcal{L}_{out}=\|\mathcal{F}\left(\mathcal{T}_{s}(\bm{x}^{\prime})\right)-\bm{x}\|^{p}+\|\mathcal{F}\left(\mathcal{T}_{s}(\bm{x})\right)-\bm{x}\|^{p}.$
(3)

We further regulate the model’s feature space during adversarial training by
enforcing alignment between the original and adversarial feature distributions
as follows:

$\mathcal{L}_{{feature}}=\|\mathcal{F}^{n}\left(\mathcal{T}_{s}(\bm{x}^{\prime})\right)-\mathcal{F}^{n}\left(\mathcal{T}_{s}(\bm{x})\right)\|^{p},$
(4)

where $\mathcal{F}^{n}$ represents the intermediate (encoder) layer output.
Overall training objective is,

$\mathcal{L}_{min}=\mathcal{L}_{out}+\lambda\mathcal{L}_{feature},$ (5)

where $\lambda$ is the balancing parameter.

####  3.2.3 Behavior of Robust Loss Surfaces

We visualize the loss landscapes of our trained autoencoders (Fig. 2). We use
the filter normalization method proposed in [Li et al.(2018)Li, Xu, Taylor,
Studer, and Goldstein], which shows the structure of the loss surface along
with random directions near the optimal pretrained parameters. We observe that
our approach leads to more flatter minima as compared to the baseline method
[Li et al.(2020)Li, Guo, and Chen]. This has significant effect on finding
generalizable adversarial example with better transferability (Sec. 4.1.1).

Figure 2: Reconstruction loss landscape of autoencoders. The first column
shows the loss landscapes of autoencoders trained using the baseline method on
the task of rotation (_top_) and jigsaw (_bottom_). The second column shows
the corresponding flatter loss surfaces obtained by training the autoencoders
using our approach.

##  4 Experimental Protocols

For a pixel range [0, 1], we create $l_{\infty}$ adversarial examples under
perceptual budget of $\epsilon\leq 0.1$ following [Li et al.(2020)Li, Guo, and
Chen]. We show adversarial transferability of our approach against ImageNet
trained models (Convolutional and ViTs), object detection and segmentation
from in-domain and cross-domain (paintings, medical scans ) data samples. We
used Adam [Kingma and Ba(2014)] optimizer with a learning rate of 0.001 for
our proposed adversarial pixel restoration (Algo. 1) at
$\epsilon\leq\frac{2}{255}$, $\delta=\frac{2}{255}$, $\lambda=1$, $p=2$. We
provide detailed ablative analysis on the effect of these hyper-parameters in
Sec. 4.2.

Surrogate Training with Few Samples: Similar to [Li et al.(2020)Li, Guo, and
Chen], we assume access to only few data samples (_e.g_ , 20 samples to train
a single autoencoder). We apply Eq. 1 as adversarial pixel transformation
($\mathcal{T}$) based on rotation or jigsaw in an unsupervised setting. We
also incorporate our method in the supervised prototypical training of
autoencoders mentioned in [Li et al.(2020)Li, Guo, and Chen], where the
reconstruction objective function used during the minimization and
maximization step is formulated by Eq. 2. We compare the best prototypical
setting reported in [Li et al.(2020)Li, Guo, and Chen], comprising of multiple
decoder networks. All models are trained for the same no. of iterations as [Li
et al.(2020)Li, Guo, and Chen]. We provide pseudo code of Algo. 1 in Appendix
D.

Surrogate Training with Large Dataset: There is abundance of unannotated data
available via online sources. Therefore, we also scale our proposed self-
supervised adversarial training to large-scale datasets. Specifically, we
train a surrogate model (single autoencoder) on paintings (79k samples) [by
Number(2017)], CoCo (40k samples) [Lin et al.(2014)Lin, Maire, Belongie, Hays,
Perona, Ramanan, Dollár, and Zitnick], and Comics (50k samples)
[Bircanoğlu(2017)] to test unsupervised cross-domain adversarial
transferability of our method.

Target Models: Adversarial perturbations from our trained autoencoders
(surrogate models) are transferred to classification models including
convolutional: VGG-19 [Simonyan and Zisserman(2015)], Inception-v3 (Inc-V3)
[Szegedy et al.(2016)Szegedy, Vanhoucke, Ioffe, Shlens, and Wojna], ResNet-152
(Res152) [He et al.(2016)He, Zhang, Ren, and Sun], Dense161 [Huang et
al.(2017)Huang, Liu, Van Der Maaten, and Weinberger], SeNet [Hu et
al.(2018)Hu, Shen, and Sun], Wide-ResNet-50 (WRN) [Zagoruyko and
Komodakis(2016)], and MobileNet-V2 (MNet-V2) [Sandler et al.(2018)Sandler,
Howard, Zhu, Zhmoginov, and Chen], and Vision Transformers: ViT-T and ViT-S
[Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai,
Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.], DeiT-T and DeiT-S
[Touvron et al.(2021)Touvron, Cord, Douze, Massa, Sablayrolles, and Jégou]. We
also evaluated adversarial vulnerability of robust ResNet-50 models [Salman et
al.(2020)Salman, Ilyas, Engstrom, Kapoor, and Madry]. Further, we transfer
attack to DETR [Carion et al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov,
and Zagoruyko] and DINO [Caron et al.(2021)Caron, Touvron, Misra, Jégou,
Mairal, Bojanowski, and Joulin] to evaluate on object detection and video
segmentation tasks.

Evaluation Metrics: We report drop in Top-1 (%) accuracy, Mean Average
Precision (mAP), and Jaccard Index for classification, object detection and
segmentation, respectively.

Datasets: We evaluate classification models on 5k samples from ImageNet
validation set in the same setting as [Li et al.(2020)Li, Guo, and Chen]. DETR
and DINO are evaluated on CoCo (5k samples) and DAVIS (2k samples) validation
sets respectively.

Baseline Adversarial Attack: For adversarial prototypical training, we use the
same supervised attack objective as proposed by [Li et al.(2020)Li, Guo, and
Chen] for direct comparison. Specifically, we use 200 iterations of I-FGSM
[Kurakin et al.(2018)Kurakin, Goodfellow, and Bengio] followed by 100
iterations of ILA [Huang et al.(2019)Huang, Katsman, He, Gu, Belongie, and
Lim]. The attack objective for the surrogate models trained in self-supervised
manner is simply based on maximizing the reconstruction error as described in
Eq. 1.

###  4.1 Results

####  4.1.1 Transferability from Few In-Domain Samples

Our surrogate models trained on few data samples show significantly higher
adversarial transferability as compared to the baseline [Li et al.(2020)Li,
Guo, and Chen]. Attack generated on our self-supervised models (rotation, and
jigsaw) performs even better than supervised (prototypical) models of [Li et
al.(2020)Li, Guo, and Chen] (Tables 1, 2). Our approach further boosts the
transferability rates when combined with supervised adversarial prototypical
training. This compliments the benefits of our method in both supervised and
self-supervised settings. We observe that vision transformers are more robust
as compared to convolutional networks against such attacks [Naseer et
al.(2021b)Naseer, Ranasinghe, Khan, Hayat, Khan, and Yang], however, our
approach provides non-trivial gains in fooling the Vision Transformers (Table
2). Similarly, adversarially robust models [Salman et al.(2020)Salman, Ilyas,
Engstrom, Kapoor, and Madry] are less vulnerable to such attacks (Fig. 3).

Transformation | Method | VGG-19 | Inc-V3 | Res152 | Dense121 | SeNet | WRN | MNet-V2 | Average
---|---|---|---|---|---|---|---|---|---
Jigsaw | [Li et al.(2020)Li, Guo, and Chen] | 31.54 | 50.28 | 46.24 | 42.38 | 59.06 | 51.24 | 25.24 | 43.71
Ours | 16.82 | 25.54 | 31.18 | 22.64 | 38.06 | 25.76 | 13.70 | 24.81${}_{\text{{\color[rgb]{1,0,1}(-18.9)}}}$
Rotation | [Li et al.(2020)Li, Guo, and Chen] | 31.14 | 48.14 | 47.40 | 41.26 | 58.20 | 50.72 | 26.00 | 43.27
Ours | 19.02 | 25.76 | 33.60 | 25.60 | 38.92 | 29.78 | 15.38 | 26.87${}_{\text{{\color[rgb]{1,0,1}(-16.4)}}}$
Prototypical | [Li et al.(2020)Li, Guo, and Chen] | 18.74 | 33.68 | 34.72 | 26.06 | 42.36 | 33.14 | 16.34 | 29.29
Ours | 17.02 | 21.48 | 28.66 | 21.06 | 35.04 | 23.56 | 13.06 | 22.84${}_{\text{{\color[rgb]{1,0,1}(-6.45)}}}$

Table 1: Our attack consistently boost adversarial transferability across
ImageNet trained models. Our training approach (Algo. 1) hence prove to be
complimentary to autoencoders trained with different self-supervised (SS)
tasks. Results (Top-1 (%), _lower is better_) are reported on 5k images from
ImageNet validation set introduced by [Li et al.(2020)Li, Guo, and Chen] under
the same perceptual budget ($\epsilon\leq 0.1$).

Transformation | Method | Deit-T | Deit-S | ViT-T | ViT-S | Average
---|---|---|---|---|---|---
Jigsaw | [Li et al.(2020)Li, Guo, and Chen] | 45.3 | 62.42 | 36.62 | 62.1 | 51.61
Ours | 43.50 | 59.50 | 18.0 | 52.48 | 43.37${}_{\text{{\color[rgb]{1,0,1}(-8.24)}}}$
Rotation | [Li et al.(2020)Li, Guo, and Chen] | 46.1 | 62.0 | 37.8 | 60.38 | 51.57
Ours | 40.84 | 55.22 | 19.64 | 48.62 | 41.08${}_{\text{{\color[rgb]{1,0,1}(-10.49)}}}$
Prototypical | [Li et al.(2020)Li, Guo, and Chen] | 38.18 | 54.96 | 21.5 | 50.28 | 41.23
Ours | 34.16 | 51.54 | 16.74 | 45.3 | 36.94${}_{\text{{\color[rgb]{1,0,1}(-4.29)}}}$

Table 2: Comparative analysis of adversarial transferability for Vision
Transformers models on ImageNet validation set. The Top-1 (%) under
$l_{\infty}$ bound $\epsilon=0.1$ is shown (_lower is better_). Our method
performs favorably well.

Figure 3: Models trained on $l_{\infty}$ examples and large norm distance are
less vulnerable to our attack, however, such models also lose accuracy on
clean images [Salman et al.(2020)Salman, Ilyas, Engstrom, Kapoor, and Madry].

Transformation | Method ($\rightarrow$) | [Li et al.(2020)Li, Guo, and Chen] | Ours
---|---|---|---
Rotation | CoCo | 28.56 | 23.31
Paintings | 27.83 | 17.75
Comics | 58.38 | 24.19
Jigsaw | CoCo | 43.93 | 31.28
Paintings | 44.07 | 33.42
Comics | 67.70 | 41.54

Table 3: Adversarial perturbations are transferred from a single auto-encoder
trained on CoCo, Painintgs or Comics. We report average of Top-1 (%) accuracy
against targeted conovlutional networks (Table 1). Our proposed approach
provide favorable results (_lower is better_).

Transformation ($\downarrow$) | Dataset ($\rightarrow$) | CoCo | Paintings | Comics
---|---|---|---|---
| No Attack | [Li et al.(2020)Li, Guo, and Chen] | Ours | [Li et al.(2020)Li, Guo, and Chen] | Ours | [Li et al.(2020)Li, Guo, and Chen] | Ours
Rotation | 39.7 | 19.3 | 14.6 | 17.2 | 11.9 | 34.3 | 13.3
Jigsaw | 39.7 | 24.7 | 14.5 | 24.1 | 14 | 38 | 20.8

Table 4: Adversarial transferability to object detector (DETR) based on mAP at
[0.5:0.95] is evaluated on CoCo validation set [Lin et al.(2014)Lin, Maire,
Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick].

Transformation ($\downarrow$) | Dataset ($\rightarrow$) | CoCo | Paintings | Comics
---|---|---|---|---
| No Attack | [Li et al.(2020)Li, Guo, and Chen] | Ours | [Li et al.(2020)Li, Guo, and Chen] | Ours | [Li et al.(2020)Li, Guo, and Chen] | Ours
Rotation | 61.8 | 53.2 | 48.9 | 52.6 | 46.9 | 57.78 | 47.81
Jigsaw | 61.8 | 53.9 | 46.6 | 53.2 | 48.5 | 58.29 | 51.65

Table 5: Adversarial transferability to object segmentation (DINO) based on
Jacard index metric is evaluated on DAVIS validation set [Pont-Tuset et
al.(2017)Pont-Tuset, Perazzi, Caelles, Arbeláez, Sorkine-Hornung, and Van
Gool].

####  4.1.2 Transferability from Cross-Domain Samples

Extra unsupervised data can boost the performance of adversarial training
[Carmon et al.(2019)Carmon, Raghunathan, Schmidt, Duchi, and Liang]. We extend
our approach to large-scale, unlabelled datasets to observe its effect on
adversarial transferability of our approach as explained below. All surrogate
models are trained for 50 epochs for cross-task adversarial transferability
experiments.

Classification: In this task, we train a single autoencoder and transfer its
adversarial perturbations to ImageNet trained convolutional models as
described in Sec. 3. We observe that our proposed adversarial training
significantly improves upon the baseline [Li et al.(2020)Li, Guo, and Chen]
(Table 3). We further note that the surrogate models trained on ‘paintings’
dataset show higher adversarial transferability while ‘Rotation’ as pixel
transformation performs better. Further analysis on adversarial
transferability of our attack is provided in Appendix A.

Object Detection and Segmentation: Our adversarial attack based on simple
transformed reconstruction error (Eq. 1) compliments our proposed adversarial
training and successfully fools DETR for object detection and DINO for video
segmentation (Tables 5 & 5). This signifies that our attack can be launched in
real-world setting without any knowledge about the deployed vision system.

###  4.2 Ablative Analysis

We thoroughly analyze and develop better understanding about the behavior of
our approach by studying the effect of its different components including, a)
_Effect of Adversarial Pixel Restoration Prior on Training_ , b) _Effect of
Perceptual Budget $\epsilon$ for our Single step Attack (Eq. 1)_, c) _Effect
of Iterative Attack during Training_ , d) _Effect of Training Iterations and
Data size_ , and e) _Contribution of Losses_. All ablative experiments are
conducted in limited data setting (Sec. 4.1.1).

Effect of Adversarial Pixel Restoration Prior: The number of parameters of the
surrogate model are significantly higher than the number of input samples, a
simple objective of reconstructing the original image can lead to identical
mapping. While adding pixel transformations (such as rotation or jigsaw)
somewhat alleviates this problem, our adversarial denoising with pixel
transformation further resolves it by lowering overfitting, resulting in
better generalizability. Fig. 5 _(left)_ shows training loss comparison
between simple reconstruction objective (_Naive_), [Li et al.(2020)Li, Guo,
and Chen] and our method. The surrogate auto-encoder quickly collapses to
identity as the training loss overfits early on in training without using
pixel restoration as prior. On the other hand, our approach allows meaningful
representation learning with higher iterations.

Figure 4: Loss landscape of surrogate models with increasing perceptual budget
from left to right.The first row shows the loss surface on the clean samples,
while as the second row plots the loss surface with respect to the
corresponding adversaries.

Effect of Perceptual Budget ($\epsilon$): Our adversarial training (Algo. 1)
is computationally efficient as it is based on a single attack (maximization)
step. The effect of attack step size on model generalizability and hence its
transferability is presented in Table 7. We observe that adversarial
perturbation computed on models trained with a smaller $\epsilon$ results in
significant improvements in terms of attack transferability. In Fig. 4, we
plot the loss surface around clean and adversarial examples on surrogate
models trained with increasing perceptual budget. While the loss surface
around clean examples becomes smoother as perceptual budget increases, it
becomes harder to maximize the reconstruction error or flip decisions on the
excessively smooth loss surface during attack. This behaviour is in line with
[Springer et al.(2021a)Springer, Mitchell, and Kenyon] which shows that
slightly robust classifier models generate highly transferable adversarial
examples. This is further evident by the shift in attention caused by our
method (Fig. 7).

Effect of Iterative Attack: During our adversarial training, increasing the
attack iterations does not help to further boost the adversarial
transferability; rather, performance often degrades significantly. In Fig. 5
_(right)_ , attack loss is plotted for surrogate models (at pereceptual budget
$\epsilon=4$) trained with different number of attack iteration (or
maximization steps).As the surrogate model becomes more robust with increasing
attack iterations, we notice that maximizing the attack objective becomes more
challenging.

Contribution of Losses: We explore the effect of our proposed pixel and
feature reconstruction losses (Eq. 5) in Table 7. We observe that feature
reconstruction compliments the pixel reconstruction and leads to better
surrogate model with more transferable adversarial space.

Perceptual Budget $\epsilon$ ($\rightarrow$) | $\frac{2}{255}$ | $\frac{4}{255}$ | $\frac{8}{255}$
---|---|---|---
Rotation | 26.87 | 27.86 | 34.01
Jigsaw | 24.81 | 27.77 | 35.94
Prototypical | 22.84 | 22.30 | 24.04

Table 6: Effect of Perceptual Budget ($\epsilon$). Top-1 average accuracy is
reported on convolution networks under limited data constraint (Table 1).

Training Loss ($\rightarrow$) | $\mathcal{L}_{out}$ | $\mathcal{L}_{out}+\mathcal{L}_{feature}$
---|---|---
Rotation | 29.62 | 26.87${}_{\text{{\color[rgb]{1,0,1}(-3.03)}}}$
Jigsaw | 27.13 | 24.81${}_{\text{{\color[rgb]{1,0,1}(-4.59)}}}$
Prototypical | 23.98 | 22.84${}_{\text{{\color[rgb]{1,0,1}(-0.84)}}}$

Table 7: Contribution of Loss components. Top-1 average accuracy is reported
on convolution networks under limited data constraint (Table 1).

Training iterations and Data Size: We explore the effect on transferability of
surrogate models w.r.t a) training iterations, and b) the number of data
samples in Fig. 6. Note that [Li et al.(2020)Li, Guo, and Chen] trains a
single autoencoder on 20 samples and thus needs 250 autoencoder to attack 5k
ImageNet validation samples. The number of autoencoders increases to 2.5k when
only 2 samples are available for training of single model. In the same
setting, the performance of our method improves with more iterations (Fig. 6
_left plot_) in contrast to [Li et al.(2020)Li, Guo, and Chen]. We report
average Top-1 (%) accuracy across all convolutional models (_lower is
better_). Similarly, as we increase the number of data samples during training
and reduce the number of autoencoders, the performance of our approach
significantly improves as compared to the baseline [Li et al.(2020)Li, Guo,
and Chen]. This indicates that our adversarial objective successfully reduces
overfitting while increases generalizability of crafted adversarial
perturbations.

Figure 5: Comparing effect of Adversarial Pixel Restoration Prior on the
training loss of surrogate models(_left_) and effect of increasing
maximization steps during training on the attack loss(_left_).

Figure 6: Top-1 (%) average accuracy against convolutional networks. The
performance of our method improves with more training iterations and data size
in contrast to the baseline [Li et al.(2020)Li, Guo, and Chen].

Samples

Clean Maps

[Li et al.(2020)Li, Guo, and Chen]

Ours

Figure 7:  GradCAM [Selvaraju et al.(2017)Selvaraju, Cogswell, Das, Vedantam,
Parikh, and Batra] explanation of adversarial examples. The activation maps
were generated on ImageNet pretrained ResNet50 model. Adversarial examples are
transferred from surrogate trained via our adversarial rotation restoration
task at the perceptual budget of $\epsilon\leq 0.1$ [Li et al.(2020)Li, Guo,
and Chen]. Our approach significantly shifts the attention of the model,
boosting the mis-classifcation rates on the adversarial examples (see Appendix
C for more qualitative examples).

##  5 Conclusion

In this work, we show the benefits of adversarial training to learn
transferable adversarial perturbations. Our approach trains an effective
surrogate by learning to restore adversarial pixel transformations created via
our proposed attack. Our adversarial training reduces overfitting during
training and can exploit very few data samples to learn meaningful adversarial
features while it can also scale to large unsupervised datasets. Our attack is
task independent and allows cross-domain attacks (_e.g_ , learning surrogate
on comics and transferring its perturbations to models trained on natural
images). Our results bring attention to the use of self-supervised adversarial
training for transferable adversarial attacks.

## References

  * [Athalye et al.(2018)Athalye, Carlini, and Wagner] Anish Athalye, Nicholas Carlini, and David Wagner.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  In _International Conference on Machine Learning (ICML)_ , 2018.
  * [Bircanoğlu(2017)] Cenk Bircanoğlu.  Comic books classification dataset : https://www.kaggle.com/datasets/cenkbircanoglu/comic-books-classification.  _Kaggle_ , 2017.
  * [Brendel et al.(2018)Brendel, Rauber, and Bethge] Wieland Brendel, Jonas Rauber, and Matthias Bethge.  Decision-based adversarial attacks: Reliable attacks against black-box machine learning models.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * [by Number(2017)] Painter by Number.  https://www.kaggle.com/c/painter-by-numbers/data.  _Kaggle_ , 2017.
  * [Carion et al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European conference on computer vision (ECCV)_. Springer, 2020\.
  * [Carlini and Wagner(2017)] Nicholas Carlini and David Wagner.  Towards evaluating the robustness of neural networks.  In _IEEE Symposium on Security and Privacy (SP)_ , 2017.
  * [Carmon et al.(2019)Carmon, Raghunathan, Schmidt, Duchi, and Liang] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang.  Unlabeled data improves adversarial robustness.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019\.
  * [Caron et al.(2021)Caron, Touvron, Misra, Jégou, Mairal, Bojanowski, and Joulin] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * [Chen et al.(2017)Chen, Zhang, Sharma, Yi, and Hsieh] Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh.  Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models.  In _Proceedings of the 10th ACM workshop on artificial intelligence and security_ , 2017.
  * [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009.
  * [Dong et al.(2017)Dong, Liao, Pang, Hu, and Zhu] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Xiaolin Hu, and Jun Zhu.  Discovering adversarial examples with momentum.  _arXiv preprint arXiv:1710.06081_ , 2017.
  * [Dong et al.(2018)Dong, Liao, Pang, Su, Zhu, Hu, and Li] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li.  Boosting adversarial attacks with momentum.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * [Gao et al.(2020)Gao, Zhang, Song, and Shen] Lianli Gao, Qilong Zhang, Jingkuan Song, and Heng Tao Shen.  Patch-wise++ perturbation for adversarial targeted attacks.  _arXiv preprint arXiv:2012.15503_ , 2020.
  * [Gidaris et al.(2018)Gidaris, Singh, and Komodakis] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  _arXiv preprint arXiv:1803.07728_ , 2018.
  * [Goodfellow et al.(2015)Goodfellow, Shlens, and Szegedy] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In _International Conference on Learning Representations (ICLR)_ , 2015.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [Hu et al.(2018)Hu, Shen, and Sun] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [Huang et al.(2017)Huang, Liu, Van Der Maaten, and Weinberger] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [Huang et al.(2019)Huang, Katsman, He, Gu, Belongie, and Lim] Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim.  Enhancing adversarial example transferability with an intermediate level attack.  In _Proceedings of the IEEE/CVF international conference on computer vision (ICCV)_ , 2019.
  * [Inkawhich et al.(2020)Inkawhich, Liang, Wang, Inkawhich, Carin, and Chen] Nathan Inkawhich, Kevin Liang, Binghui Wang, Matthew Inkawhich, Lawrence Carin, and Yiran Chen.  Perturbing across the feature hierarchy to improve standard and strict blackbox attack transferability.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020\.
  * [Kingma and Ba(2014)] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * [Kurakin et al.(2016)Kurakin, Goodfellow, and Bengio] Alexey Kurakin, Ian Goodfellow, and Samy Bengio.  Adversarial machine learning at scale.  _arXiv preprint arXiv:1611.01236_ , 2016.
  * [Kurakin et al.(2018)Kurakin, Goodfellow, and Bengio] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio.  Adversarial examples in the physical world.  In _Artificial intelligence safety and security_ , pages 99–112. Chapman and Hall/CRC, 2018.
  * [Li et al.(2018)Li, Xu, Taylor, Studer, and Goldstein] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein.  Visualizing the loss landscape of neural nets.  _Advances in neural information processing systems (NeurIPS)_ , 2018\.
  * [Li et al.(2020)Li, Guo, and Chen] Qizhang Li, Yiwen Guo, and Hao Chen.  Practical no-box adversarial attacks against dnns.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020\.
  * [Lin et al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European conference on computer vision (ECCV)_. Springer, 2014\.
  * [Madry et al.(2018)Madry, Makelov, Schmidt, Tsipras, and Vladu] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * [Moosavi-Dezfooli et al.(2016)Moosavi-Dezfooli, Fawzi, and Frossard] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.  DeepFool: a simple and accurate method to fool deep neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [Narodytska and Kasiviswanathan(2017)] Nina Narodytska and Shiva Kasiviswanathan.  Simple black-box adversarial attacks on deep neural networks.  In _Computer Vision and Pattern Recognition (CVPR) Workshop_ , 2017\.
  * [Naseer et al.(2019)Naseer, Khan, Khan, Shahbaz Khan, and Porikli] Muhammad Muzammal Naseer, Salman H Khan, Muhammad Haris Khan, Fahad Shahbaz Khan, and Fatih Porikli.  Cross-domain transferability of adversarial perturbations.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019\.
  * [Naseer et al.(2020)Naseer, Khan, Hayat, Khan, and Porikli] Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli.  A self-supervised approach for adversarial robustness.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [Naseer et al.(2021a)Naseer, Khan, Hayat, Khan, and Porikli] Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli.  On generating transferable targeted perturbations.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2021a.
  * [Naseer et al.(2021b)Naseer, Ranasinghe, Khan, Hayat, Khan, and Yang] Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.  Intriguing properties of vision transformers.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021b.
  * [Naseer et al.(2021c)Naseer, Ranasinghe, Khan, Khan, and Porikli] Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Fahad Shahbaz Khan, and Fatih Porikli.  On improving adversarial transferability of vision transformers.  _arXiv preprint arXiv:2106.04169_ , 2021c.
  * [Noroozi and Favaro(2016)] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In _European conference on computer vision (ECCV)_ , 2016.
  * [Papernot et al.(2016a)Papernot, McDaniel, and Goodfellow] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow.  Transferability in machine learning: from phenomena to black-box attacks using adversarial samples.  _arXiv preprint arXiv:1605.07277_ , 2016a.
  * [Papernot et al.(2016b)Papernot, McDaniel, Jha, Fredrikson, Celik, and Swami] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami.  The limitations of deep learning in adversarial settings.  In _Euro S &P_, 2016b.
  * [Papernot et al.(2017)Papernot, McDaniel, Goodfellow, Jha, Celik, and Swami] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami.  Practical black-box attacks against machine learning.  In _Asia Conference on Computer and Communications Security_ , 2017\.
  * [Pont-Tuset et al.(2017)Pont-Tuset, Perazzi, Caelles, Arbeláez, Sorkine-Hornung, and Van Gool] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  _arXiv preprint arXiv:1704.00675_ , 2017.
  * [Poursaeed et al.(2018)Poursaeed, Katsman, Gao, and Belongie] Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge Belongie.  Generative adversarial perturbations.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [Salman et al.(2020)Salman, Ilyas, Engstrom, Kapoor, and Madry] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry.  Do adversarially robust imagenet models transfer better?  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020\.
  * [Sandler et al.(2018)Sandler, Howard, Zhu, Zhmoginov, and Chen] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [Selvaraju et al.(2017)Selvaraju, Cogswell, Das, Vedantam, Parikh, and Batra] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2017.
  * [Simonyan and Zisserman(2015)] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _International Conference on Learning Representations (ICLR)_ , 2015.
  * [Snell et al.(2017)Snell, Swersky, and Zemel] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  _Advances in neural information processing systems (NeurIPS)_ , 2017\.
  * [Springer et al.(2021a)Springer, Mitchell, and Kenyon] Jacob Springer, Melanie Mitchell, and Garrett Kenyon.  A little robustness goes a long way: Leveraging robust features for targeted transfer attacks.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021a.
  * [Springer et al.(2021b)Springer, Mitchell, and Kenyon] Jacob M Springer, Melanie Mitchell, and Garrett T Kenyon.  Adversarial perturbations are not so weird: Entanglement of robust and non-robust features in neural network classifiers.  _arXiv preprint arXiv:2102.05110_ , 2021b.
  * [Szegedy et al.(2014)Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow, and Fergus] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  In _International Conference on Learning Representations (ICLR)_ , 2014.
  * [Szegedy et al.(2016)Szegedy, Vanhoucke, Ioffe, Shlens, and Wojna] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Conference (CVPR)_ , 2016.
  * [Touvron et al.(2021)Touvron, Cord, Douze, Massa, Sablayrolles, and Jégou] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International Conference on Machine Learning (ICML)_ , 2021.
  * [Tramèr et al.(2017)Tramèr, Kurakin, Papernot, Goodfellow, Boneh, and McDaniel] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel.  Ensemble adversarial training: Attacks and defenses.  _arXiv preprint arXiv:1705.07204_ , 2017.
  * [Wang et al.(2021)Wang, He, Wang, and He] Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He.  Admix: Enhancing the transferability of adversarial attacks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * [Xie et al.(2019)Xie, Zhang, Zhou, Bai, Wang, Ren, and Yuille] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille.  Improving transferability of adversarial examples with input diversity.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [Zagoruyko and Komodakis(2016)] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  In _British Machine Vision Conference (BMVC)_ , 2016.
  * [Zhang et al.(2021)Zhang, Bengio, Hardt, Recht, and Vinyals] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning (still) requires rethinking generalization.  _Communications of the ACM_ , 64(3):107–115, 2021\.
  * [Zhou et al.(2020)Zhou, Wu, Liu, Liu, and Zhu] Mingyi Zhou, Jing Wu, Yipeng Liu, Shuaicheng Liu, and Ce Zhu.  Dast: Data-free substitute training for adversarial attacks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.

Appendix

We provide transferability results of our self-supervised adversarial
perturbations computed at lower perceptual budget ($\ell_{\infty}\leq 0.08$)
for in-domain (Tables 8 & 9) and cross-domain (Table 10) settings in Appendix
A. Our _Adversarial Pixel Restoration_ approach remains effective as compared
to the baseline [Li et al.(2020)Li, Guo, and Chen] in fooling the
Convolutional Networks, Vision Transformers as well as state-of-the-art input
processing defense [Naseer et al.(2020)Naseer, Khan, Hayat, Khan, and Porikli]
(Fig. 8 & 9 in Appendix A.1). In appendix B and C, we visualize adversarial
examples and analyze the attention shift cause by our attack, respectively.
Finally, in appendix D, we provide psuedocode for self-supervised adversarial
training of surrogate models using our approach.

##  Appendix A Adversarial Transferability under $\ell_{\infty}\leq 0.08$

Transformation | Method | VGG-19 | Inc-V3 | Res152 | Dense121 | SeNet | WRN | MNet-V2 | Average
---|---|---|---|---|---|---|---|---|---
Jigsaw | [Li et al.(2020)Li, Guo, and Chen] | 40.00 | 58.20 | 55.66 | 50.30 | 66.62 | 59.52 | 34.60 | 52.13
Ours | 30.88 | 37.82 | 46.14 | 38.04 | 52.18 | 42.62 | 26.32 | 39.14${}_{\text{{\color[rgb]{1,0,1}(-12.99)}}}$
Rotation | [Li et al.(2020)Li, Guo, and Chen] | 38.88 | 56.16 | 57.06 | 49.56 | 65.30 | 58.14 | 34.64 | 51.39
Ours | 33.48 | 37.78 | 47.16 | 38.98 | 52.96 | 43.74 | 28.52 | 40.37${}_{\text{{\color[rgb]{1,0,1}(-11.02)}}}$
Prototypical | [Li et al.(2020)Li, Guo, and Chen] | 30.08 | 45.74 | 47.28 | 37.66 | 54.42 | 44.82 | 27.32 | 41.05
Ours | 30.44 | 31.96 | 42.76 | 34.50 | 49.20 | 38.80 | 23.94 | 35.94${}_{\text{{\color[rgb]{1,0,1}(-5.11)}}}$

Table 8: Comparative analysis of adversarial transferability. Results (top-1
(%), _lower is better_) are reported on 5k images from ImageNet validation set
under the perceptual budget of $\epsilon\leq 0.08$. Our attack provides
favorable results as compared to [Li et al.(2020)Li, Guo, and Chen].

Transformation | Method | Deit-T | Deit-S | ViT-T | ViT-S | Average
---|---|---|---|---|---|---
Jigsaw | [Li et al.(2020)Li, Guo, and Chen] | 51.32 | 68.26 | 46.68 | 68.16 | 58.61
Ours | 53.48 | 67.50 | 32.78 | 62.60 | 54.09${}_{\text{{\color[rgb]{1,0,1}(-4.52)}}}$
Rotation | [Li et al.(2020)Li, Guo, and Chen] | 53.22 | 68.36 | 47.94 | 66.72 | 59.06
Ours | 51.16 | 66.10 | 32.68 | 60.20 | 52.54${}_{\text{{\color[rgb]{1,0,1}(-6.52)}}}$
Prototypical | [Li et al.(2020)Li, Guo, and Chen] | 47.44 | 64.22 | 32.94 | 60.06 | 51.17
Ours | 45.70 | 62.26 | 29.50 | 57.98 | 48.86${}_{\text{{\color[rgb]{1,0,1}(-2.31)}}}$

Table 9: Comparative analysis of adversarial transferability for Vision
Transformers on 5k images [Li et al.(2020)Li, Guo, and Chen] from ImageNet
validation set under the perceptual budget of $\epsilon\leq 0.08$. ViTs are
relatively more robust as compared to CNNs.

|  | Convolutional Networks | Vision Transformers
---|---|---|---
Transformation | Method ($\rightarrow$) | [Li et al.(2020)Li, Guo, and Chen] | Ours | [Li et al.(2020)Li, Guo, and Chen] | Ours
Rotation | CoCo | 38.47 | 35.72 | 48.24 | 46.44
Paintings | 36.81 | 30.48 | 45.92 | 42.04
Comics | 62.46 | 39.16 | 70.39 | 53.03
Jigsaw | CoCo | 51.75 | 43.6 | 61.95 | 55.36
Paintings | 51.56 | 44.42 | 61.59 | 56.64
Comics | 69.89 | 52.65 | 75.66 | 62.15

Table 10: Adversarial perturbations (_$\epsilon=0.08$_) are transferred from a
single surrogate auto-encoder trained on CoCo, Paintings or Comics datasets to
different Convolutional and Transformer based models. Results ( top-1 (%)
accuracy, _lower is better_) are averaged across all models. Details on these
models are provide in Sec. 4

###  A.1 Adversarial Transferability against Neural Purification Defense

Figure 8: We test the vulnerability of the state-of-the-art input processing
defense NRP [Naseer et al.(2020)Naseer, Khan, Hayat, Khan, and Porikli]
against our self-supervised attack. Adversarial perturbations are transferred
from auto-encoders under the constraint of limited in-domain samples [Li et
al.(2020)Li, Guo, and Chen] to Convolutional Networks (_left plot_), and
Vision Transformers (_right plot_) protected by NRP defense. We report Top-1
(%) (_lower is better_) averaged across the models. Our method consistently
improves the attack success rate.

Figure 9: We test the vulnerability of the state-of-the-art input processing
defense NRP [Naseer et al.(2020)Naseer, Khan, Hayat, Khan, and Porikli]
against our self-supervised attack under cross-domain setting (Sec. 4).
Adversarial perturbations are transferred from single auto-encoder trained on
CoCo, Paintings or Comics to Convolutional Networks (_left plot_), and Vision
Transformers (_right plot_) protected by NRP defense. We report Top-1 (%)
(_lower is better_) averaged across the models. Our method consistently
improves the attack success rate.

##  Appendix B Adversarial Examples

Figure 10: Adversarial examples crafted on surrogate models trained on limited
data, with perturbation bound $\epsilon\leq 0.1$. The top and bottom row show
adversarial examples crafted on surrogate model trained using [Li et
al.(2020)Li, Guo, and Chen] and ours method, respectively.

##  Appendix C Attention Dispersion

Samples

Clean Maps

[Li et al.(2020)Li, Guo, and Chen]

Ours

Figure 11:  GradCAM [Selvaraju et al.(2017)Selvaraju, Cogswell, Das, Vedantam,
Parikh, and Batra] explanation of adversarial examples. The activation maps
were generated on ImageNet pretrained ResNet50 model. Adversarial examples are
transferred from surrogate trained via our adversarial rotation restoration
task at the perceptual budget of $\epsilon\leq 0.1$ [Li et al.(2020)Li, Guo,
and Chen]. Our approach significantly shifts the attention of the model,
boosting the mis-classifcation rates on the adversarial examples.

##  Appendix D Training Algorithm

Algorithm 2 Adversarial Pixel Restoration: Pytorch style Pseudocode

⬇

def train(model, optimizer, images, iterations, fgsm_step):

"""

␣␣␣␣images:␣␣␣␣␣images␣shaped␣(B,␣C,␣H,␣W)

␣␣␣␣model:␣␣␣␣␣␣autoencoder

␣␣␣␣"""

attack = FGSM(model, eps=fgsm_step)

target_images = images.clone()

for i in range(iterations):

# Generate adversarial images

transformed_images = transform(images)

adv_images = attack(transformed_images, target_images)

# Get the encoder features and output

clean_output, clean_enc_output = model(transformed_images)

adv_output, adv_enc_output = model(adv_images)

# Compute losses

clean_loss = nn.MSELoss()(clean_output, target_images)

adv_loss = nn.MSELoss()(adv_output, target_images)

feat_loss = nn.MSELoss()(adv_enc_output, clean_enc_output)

loss = clean_loss + adv_loss + feat_loss

# Update model parameters

optimizer.zero_grad()

loss.backward()

optimizer.step()