# Adversarial Pretraining of Self-Supervised Deep Networks: Past, Present and
Future

Guo-Jun Qi<EMAIL_ADDRESS>Laboratory for Machine Perception and
Learning10940 NE 33RD PLACE, SUITE 202BellevueWashingtonUSA98004 and  Mubarak
Shah  University of Central Florida4328 Scorpius St.OrlandoFloridaUSA32816
<EMAIL_ADDRESS>
###### Abstract.

In this paper, we review adversarial pretraining of self-supervised deep
networks including both convolutional neural networks and vision transformers.
Unlike the adversarial training with access to labeled examples, adversarial
pretraining is complicated as it only has access to unlabeled examples. To
incorporate adversaries into pretraining models on either input or feature
level, we find that existing approaches are largely categorized into two
groups: memory-free instance-wise attacks imposing worst-case perturbations on
individual examples, and memory-based adversaries shared across examples over
iterations. In particular, we review several representative adversarial
pretraining models based on Contrastive Learning (CL) and Masked Image
Modeling (MIM), respectively, two popular self-supervised pretraining methods
in literature. We also review miscellaneous issues about computing overheads,
input-/feature-level adversaries, as well as other adversarial pretraining
approaches beyond the above two groups. Finally, we discuss emerging trends
and future directions about the relations between adversarial and cooperative
pretraining, unifying adversarial CL and MIM pretraining, and the trade-off
between accuracy and robustness in adversarial pretraining.

adversarial pretraining, contrastive learning, masked image modeling, memory-
free vs. memory-based adversaries, instance-wise perturbations

††ccs: Computing methodologies Computer vision representations††ccs: Computing
methodologies Unsupervised learning

##  1\. Introduction

Adversarial pretraining aspires to learn an unsupervised deep networks without
access to labels. In contrast, adversarial training in literature (Tsipras et
al., 2018; Kurakin et al., 2016; Goodfellow et al., 2014; Szegedy et al.,
2013; Madry et al., 2017; Moosavi-Dezfooli et al., 2016; Papernot et al.,
2016; Carlini and Wagner, 2017; Xu et al., 2017; Zhang and Qi, 2020) seeks to
find worst-case adversarial examples and use them to train neural networks
robust to the corresponding attacks. While some findings (Szegedy et al.,
2013; Tsipras et al., 2018) revealed that the adversarially trained networks
can be robust to adversarial attacks or gain improved standard accuracy,
however, there are limited reviews of adversarially pretrained networks by
classifying and evaluating existing approaches, assessing their advantages and
shortcomings, as well as charting future directions.

At the start, we want to clarify a common misunderstanding about the role of
adversarial pretraining. The goal of an adversarial approach is not limited to
learning robust representation against potential attacks. Instead, it is also
employed to improve the generalization accuracy in downstream tasks,
especially when the adversarial model attacks on feature levels (Hu et al.,
2021; Robinson et al., 2021, 2020; Kalantidis et al., 2020) rather than on raw
inputs (e.g., image pixels) of individual instances (aka instance-wise attacks
(Ho and Nvasconcelos, 2020; Kim et al., 2020; Jiang et al., 2020)). When not
attacking on the raw inputs, the adversarial pretraining often cares about
whether the learned features are generalizable to future problems, aiming to
avoid learning trivial solutions that merely use low-level features to bypass
a pretext task. For example, easy negatives in contrastive learning could
result in less discriminative features to distinguish between positive and
negative samples for a query (Kalantidis et al., 2020; Hu et al., 2021); in
masked imaging modeling (MIM), the network may learn low-level features to
reconstruct missing patches by simply using the similarity between locally
correlated patches (Shi et al., 2022; Authors, 2022) if the MIM objective is
not sufficiently hard. In these cases, it is beneficial to explore adversarial
approaches to improve the generalizability of learned representations. In
other words, learning more generalizable representations through adversarial
pretraining is an equally important goal in literature as learning robust
representations against some presumptive attacks.

###  1.1. Instance-wise Perturbations from Adversarial Training to Pretraining

Let us begin by revisiting the adversarial training. Formally, the adversarial
training (instead of pretraining) gets access to the labeled examples, and it
maximizes an associated classification loss such as the cross entropy
$\mathcal{L}_{ce}$ over a constraint perturbation $\boldsymbol{\delta}$ of
magnitude $\varepsilon$ to find an adversarial example for an input
$\mathbf{x}$ (Madry et al., 2017), i.e.,

(1)
$\boldsymbol{\delta}^{\star}=\arg\max\limits_{\|\boldsymbol{\delta}\|_{p}\leq\varepsilon}\mathcal{L}_{ce}(\mathbf{x}+\boldsymbol{\delta},\mathbf{y};\boldsymbol{\theta}),$

where the network weights $\boldsymbol{\theta}$ can be learned by

(2)
$\boldsymbol{\theta}^{\star}=\arg\min\limits_{\boldsymbol{\theta}}\mathop{\mathbb{missing}}E_{(\mathbf{x},\mathbf{y})\sim\mathcal{D}}\mathcal{L}_{ce}~{}(\mathbf{x}+\boldsymbol{\delta}^{\star},\mathbf{y};\boldsymbol{\theta})$

with the expectation taken over labeled examples $(\mathbf{x},\mathbf{y})$
sampled from a distribution $\mathcal{D}$. More detailed reviews of
adversarial training of deep networks can be found in (Akhtar and Mian, 2018;
Akhtar et al., 2021).

Instead, the adversarial pretraining aligns with unsupervised representation
learning without access to label $\mathbf{y}$ as in the above formulation (Wu
et al., 2018; Chen et al., 2020b; Agrawal et al., 2015; Gidaris et al., 2018;
Qi et al., 2019; Doersch et al., 2015; Kim et al., 2018; Larsson et al., 2016;
Zhang et al., 2019a; Qi and Luo, 2020), and has separate pretext and
downstream tasks that are often more complex than the adversarial training.
First, there are many different ways to define a pretext task to
unsupervisedly learn a deep network. A variety of pretext tasks result in
different approaches to perform meaningful adversarial pretraining, and some
pretext tasks may make the pretrained representation vulnerable to potential
attacks (Gupta et al., 2022). In this review, we will show that a straight
extension of adversarial training is to add perturbations on unlabeled
examples and solve a similar minimax problem for adversarial pretraining.
Particularly, among popular pretext tasks are Contrastive Learning (CL) (Oord
et al., 2018; Hénaff et al., 2019; Wu et al., 2018; Chen et al., 2020a, b) and
Masked Image Modeling (MIM) (He et al., 2022; Xie et al., 2022; Bao et al.,
2021) for Convolutional Nets (ConvNet) (Krizhevsky et al., 2012; He et al.,
2016) and vision transformers (Dosovitskiy et al., 2020; Touvron et al., 2021;
Wang et al., 2022a), respectively.

We will review how adversarial perturbations can be generalized to these
pretext tasks on either instance or feature level. Particularly, we will
review the related works from a novel perspective by grouping them into
memory-free and memory-based adversarial pretraining. We will show that the
memory-free adversarial approaches usually consist of instance-wise
perturbations as in the adversarial training methods, where the perturbation
is constructed on raw inputs of individual examples. Pretraining a deep
network is memory-free in the sense that the perturbations and the associated
adversarial examples are not kept over epochs. Many adversarial approaches
belong to this category including (Ho and Nvasconcelos, 2020; Kim et al.,
2020; Jiang et al., 2020).

###  1.2. Feature-level adversarial Pretraining with Memory-based Adversaries

In contrast, the feature-level adversarial pretraining abandons construction
of instance-wise perturbations, and instead uses a shared memory as
adversarial players. A natural choice of such adversaries is the memory bank
widely used in contrastive learning (He et al., 2020), and thus most of
feature-level adversarial pretraining is memory-based. For example, AdCo (Hu
et al., 2021) treats all examples in the memory bank (cf. (He et al., 2020))
as learnable negatives, and directly learns them by maximizing the contrastive
loss. This results in hard negatives that are continuously updated to be mixed
with their positive counterparts. Hence, the learned negatives are hard to be
distinguished from positives so more discriminative representations must be
learned that are generalizable to downstream tasks. IFM (Robinson et al.,
2021) extends the idea of AdCo by considering implicit feature-level
modification to contrastive pairs subject to assigned budgets. These methods
give rise to a family of feature-level adversarial learning to generate hard
negatives in a principled manner, establishing strong connections with hard
negative mining and sampling methods (Kalantidis et al., 2020; Robinson et
al., 2020) that we will also review in this survey.

###  1.3. MIM-based Adversarial Pretraining

Moreover, Masked Image Modeling (MIM) (He et al., 2022; Xie et al., 2022; Bao
et al., 2021) has attracted increasing attentions for pretraining vision
transformers. While most of existing adversarial pretraining methods are built
upon contrastive learning, this opens up unprecedented research opportunities
to study adversarial pretraining of vision transformers in the MIM framework.
We will review existing adversarial MIM-pretraining approaches for
transformers (Shi et al., 2022; Authors, 2022), and point out some natural
extension of adversarial perturbations to the MIM. Both instance-wise (e.g.,
FGSM-based approach) and feature-level adversarial MIM-pretraining approaches
will be reviewed and discussed.

###  1.4. Future Directions and Survey Structure

Finally, we will review some emerging trends and future directions on
adversarial pretraining. In particular, we will focus on three aspects of
future directions.

  * •

Adversarial vs. cooperative pretraining. We will review related works and
discuss the connection between adversarial (that maximizes the training loss)
and cooperative (that minimizes the training loss) pretraining. We will point
out a direction to study when different parts of a pretrained network ought to
be learn adversarially or cooperatively. In particular, depending on different
modes of adversarial pretraining (instance-wise vs. memory-based, and input-
level vs. feature-level), a positive query in contrastive learning can be
treated as either an adversary in a hybrid model or a cooperator through a
shared memory bank. We will show that it could be beneficial to combine these
two modes through various pretraining approaches.

  * •

Unifying adversarial contrastive learning and masked image modeling. Existing
pretraining approaches (Zhou et al., 2021; Huang et al., 2022) have shown that
combining contrastive learning and masked image modeling leads to more
powerful representation of vision transformers with improved generalizability.
The second direction worthy of studying is to explore an elegant way of
designing adversaries to integrate both approaches. It is expected that the
unified adversarial pretraining can make the learned representation more
powerful and generalizable to downstream tasks.

  * •

Accuracy vs. robustness. In adversarial training, the relationship between
standard accuracy and model robustness to adversarial perturbations has been
extensively studied. Existing works have demonstrated that the network
robustness may not always be indicative of improved accuracy (Tsipras et al.,
2018). The problem becomes more complicated for adversarial pretraining, since
the pretext tasks used to pretrain a network are usually not related to the
downstream objectives. Revealing the underlying connections between network
robustness to pretext task adversaries and generalization accuracy in
downstream tasks should be a direction worthy of exploring from both
theoretical and practical perspectives.

The remainder of this paper is organized as follows. In Section 2, we will
review the contrastive pretraining methods by grouping them into instance-wise
memory-free and feature-level memory-based models. We will discuss some
miscellaneous issues regarding the computing overheads, and combined
objectives for pretraining. In Section 3, we will review the MIM-based
adversarial pre-training, followed by other related methods beyond contrastive
and MIM pretraining in Section 4. We will discuss the evaluation protocols and
review existing results in Section 5. Emerging trends and future directions
will be discussed in Section 6, and we will conclude the paper in Section 7.

Table 1. Different types of adversarial pre-training models discussed in the
paper. For each model, we denote its model type (memory-free vs. memory-
based), base model(CL, MIM, or others), the formula for adversaries,
pretraining adversaries on input or feature levels (Input/Feature), and if
using a pre-training objective combining adversarial and standard losses (in
the column “Comb. obj.” ). The table compares the similarities and differences
across different models in forms of the formula for adversaries.

Model | Type | Base | Adversaries | Input/Feature | Comb. obj.
---|---|---|---|---|---
RoCL (Kim et al., 2020) | memory-free | CL | $\begin{aligned} &t(\mathbf{x})\leftarrow t(\mathbf{x})~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\qquad\qquad\cdots{\rm FGSM}\\\ &+\epsilon~{}{\rm sign}(\nabla_{t(\mathbf{x})}\mathcal{L}_{con}(t(\mathbf{x}),\\{\mathbf{x}_{pos}\\},\\{\mathbf{x}_{neg}\\}))\\\ &\qquad\qquad\qquad\qquad{\rm or}\\\ &t(\mathbf{x})\leftarrow\mathop{\Pi}\limits_{\mathcal{S}(t(\mathbf{x}),\epsilon)}[t(\mathbf{x})\quad\quad~{}~{}~{}~{}~{}~{}~{}~{}\cdots{\rm PGD}\\\ &+\alpha{\rm sign}(\nabla_{t(\mathbf{x})}\mathcal{L}_{con}(t(\mathbf{x}),\\{\mathbf{x}_{pos}\\},\\{\mathbf{x}_{neg}\\}))]\end{aligned}$ | input-level | yes
CLAE (Ho and Nvasconcelos, 2020) |  |  |  |  |
ACL (Jiang et al., 2020) |  |  |  |  |
ADVCL (Fan et al., 2021) | memory-free | CL | $\begin{aligned} &\mathbf{x}\leftarrow\mathbf{x}+\epsilon~{}{\rm sign}(\nabla_{\mathbf{\bar{x}}}\mathcal{L}_{con}(t(\mathbf{x}),t^{\prime}(\mathbf{x}),\mathbf{\bar{x}},\mathbf{x}_{h}))\\\ &{\rm by~{}~{}FGSM~{}~{}attacks.}\end{aligned}$ | input-level | yes
ARoCL (Gupta et al., 2022) | memory-free | CL | false negative removal | input-level | yes
AACL (Gupta et al., 2022) |  |  |  |  |
AdCo (Hu et al., 2021) | memory-based | CL | $\mathbf{z}_{neg}\leftarrow\mathbf{z}_{neg}+\dfrac{\alpha}{\tau}\mathop{\mathbb{missing}}E\limits_{\mathbf{x}\sim\mathcal{D}}p(\mathbf{z}_{neg}|\mathbf{z})~{}\mathbf{z}$ | feature-level | no
CaCo (Wang et al., 2022b) | memory-based | CL | $\mathbf{z}_{pos}\leftarrow\mathbf{z}_{pos}+\dfrac{\alpha}{\tau}[1-p(\mathbf{z}_{pos}|\mathbf{z})]\mathbf{z}$ | feature-level | no
AdPE (Authors, 2022) | memory-based | MIM | $\begin{aligned} &\boldsymbol{\delta}\leftarrow\mathop{\Pi}\limits_{\|\boldsymbol{\delta}\|_{q}\leq\epsilon}[\boldsymbol{\delta}+\alpha\nabla_{\boldsymbol{\delta}}\cdot~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\qquad\cdots{\rm PGD}\\\ &\mathcal{L}_{MIM}(\\{\mathbf{p}_{[\widetilde{g}(x_{i}-x_{j}+\delta_{x}),\widetilde{g}(y_{i}-y_{j}+\delta_{y})]}|i\in\mathbb{M}\\};\boldsymbol{\theta})]\end{aligned}$ | feature-level | no
IFM (Robinson et al., 2021) | memory-free | CL | $\begin{aligned} \mathbf{z}_{pos}\leftarrow\mathbf{z}_{pos}-\epsilon\mathbf{z}\\\ \mathbf{z}_{neg}\leftarrow\mathbf{z}_{neg}+\epsilon\mathbf{z}\end{aligned}$ | feature-level | yes
MoCHi (Kalantidis et al., 2020) | – | CL | hard negative mixing | feature-level | yes
HCL (Robinson et al., 2020) | – | CL | hard negative sampling | feature-level | yes
BYORL (Gowal et al., 2020) | memory-free | BYOL | $\boldsymbol{\delta}\leftarrow\Pi_{\|\boldsymbol{\delta}\|_{p}\leq\epsilon}[\boldsymbol{\delta}+\alpha\nabla_{\boldsymbol{\delta}}\mathcal{L}_{byol}(t(\mathbf{x})+\boldsymbol{\delta},t^{\prime}(\mathbf{x}))]$ | input-level | no
RUSH (Pang et al., 2022) | memory-free | CL | randomized smoothing | input-level | no
ADIOS (Shi et al., 2022) | memory-free | MIM |  mask generating network $\mathcal{M}_{\phi}$ | input-level | no

Remarks:

1) Memory-free: instance-wise perturbations imposed on individual examples
without carrying over iterations;

2) Memory-based: adversarial samples are stored in a shared memory to
challenge the pretrained network;

3) Input-level: adversaries imposed on raw inputs of instances;

4) Feature-level: adversaries imposed on feature representations.

Table 2. A summary of source code links. Models | Links
---|---
RoCL (Kim et al., 2020) | https://github.com/Kim-Minseon/RoCL
CLAE (Ho and Nvasconcelos, 2020) | https://github.com/chihhuiho/CLAE
ACL (Jiang et al., 2020) | https://github.com/VITA-Group/Adversarial-Contrastive-Learning
AdCo (Hu et al., 2021) | https://github.com/maple-research-lab/AdCo
CaCo (Wang et al., 2022b) | https://github.com/maple-research-lab/CaCo
IFM (Robinson et al., 2021) | https://github.com/joshr17/IFM
MoCHi (Kalantidis et al., 2020) | https://europe.naverlabs.com/research/computer-vision/mochi/
HCL (Robinson et al., 2020) | https://github.com/joshr17/HCL
ADIOS (Shi et al., 2022) | https://github.com/yugeten/adios
ADVCL (Fan et al., 2021) | https://github.com/LijieFan/AdvCL

##  2\. Adversarial Contrastive Pretraining

Figure 1. Instance-wise memory-free vs. feature-level memory-based adversarial
pretraining models. At each iteration, a batch of $N$ instances are sampled
(represented in the figure by the rectangle box denoted by “A batch”). Given
an input instance $\mathbf{x}$, its two augmented views $t(\mathbf{x})$ and
$t^{\prime}(\mathbf{x})$ with two transformations $t$ and $t^{\prime}$
randomly drawn from $\mathcal{T}$ are created. The two views are encoded and
projected into two embedded feature representations $\mathbf{z}$ and
$\mathbf{z}^{\prime}$, respectively. The two augmented views and their
representations form a positive pair (dashed green line). (a) For memory-free
model with instance-wise perturbations, an optimal perturbation
$\boldsymbol{\delta}$ is computed for each individual sample $t(\mathbf{x})$
in the batch by maximizing the contrastive loss. Different samples in the
batch are considered as mutually negative to one another. This adversarial
pretraining model is memory-free as the perturbation attacks are sample-
specific and they will not be carried to the next iteration. (b) For a memory-
based model with adversarial negatives, there exists a memory bank of
adversarial negatives that are shared among all query samples in a batch. In
other words, for every query sample $t(\mathbf{x})$ or
$t^{\prime}(\mathbf{x})$ in the batch, all adversarial negatives in the memory
bank are considered as being negative to it. These negatives are adversarially
learned by maximizing the contrastive loss. This model is memory-based since
the adversarial negatives in the memory bank are kept over iterations, and are
continuously updated over time. In both figures, we use the orange arrow to
highlight the adversarial component in the two models.

Contrastive learning has become the state-of-the-art method for pretraining a
variety of deep networks ranging from convolutional networks (Chen et al.,
2020a; He et al., 2020) to vision transformers (Caron et al., 2021).
Incorporating adversaries into contrastive learning for pretraining deep
networks has also been intensively studied in literature (Ho and Nvasconcelos,
2020; Kim et al., 2020; Jiang et al., 2020; Hu et al., 2021).

In this section, we will review existing methods from a novel perspective by
categorizing them into two large groups: memory-free methods with instance-
wise perturbations, memory-based methods on feature level by learning a shared
memory of adversarial negatives. Readers may take a glance at Table 1 that
summarizes different adversarial pretraining models we will discuss in this
paper.

###  2.1. Background: Contrastive Learning

Contrastive learning seeks to pre-train an encoder network by maximizing the
agreement of representations between a pair of samples transformed from the
same instance, while pushing apart the representations of those transformed
from different ones (Wu et al., 2018).

Formally, given an instance $\mathbf{x}$, it is randomly transformed into a
pair of examples $t(\mathbf{x})$ and $t^{\prime}(\mathbf{x})$ with two
transformations $t$ and $t^{\prime}$ drawn from a distribution $\mathcal{T}$.
An encoder network $f_{\theta}$ and a following projector $g_{\theta}$ map the
pair into two latent vectors
$\mathbf{z}=g_{\theta}(f_{\theta}(t(\mathbf{x})))$ and
$\mathbf{z}^{\prime}=g_{\theta}(f_{\theta}(t^{\prime}(\mathbf{x})))$,
respectively. Then, the contrastive learning is designed to minimize the
following contrastive loss

(3)
$\displaystyle\mathop{\mathbb{missing}}E\limits_{\mathbf{x}\sim\mathcal{D}}$
$\displaystyle\mathcal{L}_{con}(t(\mathbf{x}),\\{\mathbf{x}_{pos}\\},\\{\mathbf{x}_{neg}\\};\theta)$
$\displaystyle\triangleq-\log\dfrac{\mathop{\sum}\limits_{\mathbf{z}^{\prime}\in\\{\mathbf{z}_{pos}\\}}\exp(sim(\mathbf{z},\mathbf{z}^{\prime})/\tau)}{\mathop{\sum}\limits_{\mathbf{z}^{\prime\prime}\in\\{\mathbf{z}_{pos}\\}\cup\\{\mathbf{z}_{neg}\\}}\exp(sim(\mathbf{z},\mathbf{z}^{\prime\prime})/\tau)},$

where $\mathcal{D}$ is the data distribution, $\\{\mathbf{x}_{pos}\\}$ and
$\\{\mathbf{x}_{neg}\\}$ are the set of positive and negative examples for
$t(\mathbf{x})$, and $\\{\mathbf{z}_{pos}\\}$ and $\\{\mathbf{z}_{neg}\\}$ are
their representations. Usually, the positive set has a single example
$t^{\prime}(\mathbf{x})$ transformed from the same instance, and the negative
set contains all examples transformed from different instances. The $sim$ is a
similarity function usually chosen as cosine similarity, and $\tau$ is the
temperature for the loss.

###  2.2. Memory-free Methods: Instance-wise Perturbation

The idea of instance-wise attack against the contrastive pre-training is to
seek a worst-case perturbation $\boldsymbol{\delta}$ onto the transformed
example $t(\mathbf{x})$ by maximizing the above contrastive loss, yielding the
most dissimilar positive sample $t(\mathbf{x})+\boldsymbol{\delta}$. Then, the
contrastive loss is minimized by maximizing the agreement between the
adversarially perturbed example and its clean counterparts in
$\\{\mathbf{x}_{pos}\\}$, leading to a minimax problem to pre-train a deep
network parameterized with $\theta$,

(4)
$\min_{\theta}\mathop{\mathbb{missing}}E\limits_{\mathbf{x}\sim\mathcal{D}}\max_{\|\boldsymbol{\delta}\|_{p}\leq\epsilon}\mathcal{L}_{con}(t(\mathbf{x})+\boldsymbol{\delta},\\{\mathbf{x}_{pos}\\},\\{\mathbf{x}_{neg}\\};\theta),$

where the perturbation magnitude is constrained by an $\ell_{p}$-norm of
$\epsilon$.

The inner maximization problem can be solved by the Fast Gradient Sign Method
(FGSM) (Goodfellow et al., 2014) for an infinity norm $\ell_{+\infty}$,

(5) $\boldsymbol{\delta}^{\star}=\epsilon~{}{\rm
sign}(\nabla_{t(\mathbf{x})}\mathcal{L}_{con}(t(\mathbf{x}),\\{\mathbf{x}_{pos}\\},\\{\mathbf{x}_{neg}\\};\theta)).$

Or the Projected Gradient Descent (PGD) (Madry et al., 2017) can be applied to
iteratively update $\boldsymbol{\delta}$ to find an optimal perturbation,

(6) $\displaystyle
t(\mathbf{x})\leftarrow\mathop{\Pi}\limits_{\mathcal{S}(t(\mathbf{x}),\epsilon)}[t(\mathbf{x})+\alpha{\rm
sign}(\nabla_{t(\mathbf{x})}\mathcal{L}_{con}(t(\mathbf{x}),\\{\mathbf{x}_{pos}\\},\\{\mathbf{x}_{neg}\\};\theta))],$

with a step size $\alpha$ and a projection $\Pi$ onto the constraint
$\mathcal{S}(t(\mathbf{x}),\epsilon)=\\{t(\mathbf{x})+\boldsymbol{\delta}|\|\boldsymbol{\delta}\|_{p}\leq\epsilon\\}$.

The sought perturbation $\boldsymbol{\delta}$ is distinctive to an individual
sample $t(\mathbf{x})$, given the current estimate of network weights
$\theta$. For each iteration, a new batch of samples arrive, and the
adversarial positives $t(\mathbf{x})+\boldsymbol{\delta}$ from the past
iterations are abandoned since the network weights are updated based on the
adversarial examples of the new batch. In this sense, such instance-wise
attacks are memory-free with no need to store the past adversarial examples.

In addition to such vanilla instance-wise perturbations (Kim et al., 2020),
there are several variants. For example, (Jiang et al., 2020) considered to
apply a pair of perturbations $\boldsymbol{\delta}$ and
$\boldsymbol{\delta}^{\prime}$ to both positive samples $t(\mathbf{x})$ and
$t^{\prime}(\mathbf{x})$. On the other hand, (Ho and Nvasconcelos, 2020) only
used a transformed sample $t(\mathbf{x})$ and its adversarial perturbation
$t(\mathbf{x})+\boldsymbol{\delta}$ to form a positive pair, without involving
another transformed sample $t^{\prime}(\mathbf{x})$ from the same instance
$\mathbf{x}$.

More variants. Besides the above two, an asymmetric InfoNCE (A-InfoNCE) loss
(Yu et al., 2022) was proposed to mitigate the instance-level identity
confusion arising from strong adversarial examples by adopting an asymmetric
similarity and re-weighting hard negatives following (Robinson et al., 2020;
Chuang et al., 2020). In (Gupta et al., 2022), it shows that self-supervised
models are more susceptible to adversarial perturbations than supervised ones
due to false negatives, and hence a false negative removal strategy was
proposed to augment RoCL (Kim et al., 2020) and ACL (Jiang et al., 2020) in an
effort to make the adversarially pretrained representations more robust
against attacks.

In (Fan et al., 2021), optimal adversarial perturbations are sought over input
images rather than transformed views in defense to adversarial attacks on
original inputs. Formally, with an additional high-frequency view
$\mathbf{x}_{h}$ of an input image $\mathbf{x}$ through Fourier
transformation, the multi-view contrastive pre-training was performed by

$\min_{\theta}\mathop{\mathbb{missing}}E\limits_{\mathbf{x}\sim\mathcal{D}}\max_{\|\boldsymbol{\delta}\|_{\infty}\leq\epsilon}\mathcal{L}_{con}(t(\mathbf{x}),t^{\prime}(\mathbf{x}),\mathbf{x}+\boldsymbol{\delta},\mathbf{x}_{h};\theta)$

over a contrastive loss $\mathcal{L}_{con}$ with four views: two transformed
view $t(\mathbf{x})$ and $t^{\prime}(\mathbf{x})$, the original input
$\mathbf{x}$, and the high-frequency view $\mathbf{x}_{h}$. They mutually form
the positive pairs for the same $\mathbf{x}$, while the views from different
inputs constitute negative pairs (to avoid notation clutters, we suppress the
notations of negative samples in the above loss). Besides such multi-view
contrastive learning, the clustering-induced pseudo-labels were treated as the
true labels in conventional adversarial training (Madry et al., 2017) to
generate adversaries. The resultant pseudo-label loss is combined with the
multi-view contrastive loss for adversarial pretraining.

###  2.3. Memory-based Methods: Feature-Level Adversarial Negatives

The Momentum Contrast (MoCo) (He et al., 2020) is a memory-based contrastive
learning model. While the SimCLR (Chen et al., 2020b) treats other instances
over a batch as negatives, the MoCo instead uses a memory bank to store
negatives queued from the past iterations.

Built upon such a memory model, the Adversarial Contrast (AdCo) (Hu et al.,
2021) treats the negatives in the memory bank as learnable weights, and uses
them as adversaries to attack against the pre-trained network. The memory bank
only stores the feature representation $\\{\mathbf{z}_{neg}\\}$ of negatives
without access to the original samples, and thus the AdCo is also a feature-
level adversarial approach. AdCo employs the following minimax objective

$\min_{\theta}\max_{\\{\mathbf{z}_{neg}\\}}\mathop{\mathbb{missing}}E\limits_{\mathbf{x}\sim\mathcal{D}}\mathcal{L}_{con}(t(\mathbf{x}),\\{\mathbf{x}_{pos}\\},\\{\mathbf{z}_{neg}\\};\theta),$

where $\\{\mathbf{x}_{neg}\\}$ in (4) has been replaced with the feature-level
representations $\\{\mathbf{z}_{neg}\\}$ of negatives for clarity.

Unlike the instance-wise attack relying on the back-propagated gradient to
compute the optimal perturbation, one advantage of the memory-based AdCo is
that the derivative of the contrastive loss over a negative representation
$\mathbf{z}_{neg}$ can be directly computed (Hu et al., 2021) as

$\dfrac{\partial\mathcal{L}_{con}(t(\mathbf{x}),\\{\mathbf{x}_{pos}\\},\\{\mathbf{z}_{neg}\\};\theta)}{\partial\mathbf{z}_{neg}}=\dfrac{1}{\tau}\mathop{\mathbb{missing}}E\limits_{\mathbf{x}\sim\mathcal{D}}p(\mathbf{z}_{neg}|\mathbf{z})~{}\mathbf{z},$

where $\mathbf{z}$ is the projected representation of a transformed sample
$t(\mathbf{x})$, i.e., $\mathbf{z}=f_{\theta}(g_{\theta}(t(\mathbf{x})))$, and

$p(\mathbf{z}_{neg}|\mathbf{z})=\dfrac{\exp(sim(\mathbf{z},\mathbf{z}_{neg})/\tau)}{\mathop{\sum}\limits_{\mathbf{z}^{\prime\prime}\in\\{\mathbf{z}_{neg}\\}\cup\\{\mathbf{z}^{\prime}\\}}\exp(sim(\mathbf{z},\mathbf{z}^{\prime\prime})/\tau)}$

where $\mathbf{z}^{\prime}$ is the projected representation of another sample
$t^{\prime}(\mathbf{x})$ transformed from the same instance. Obviously,
$p(\mathbf{z}_{neg}|\mathbf{z})$ is the probability of a negative
$\mathbf{z}_{neg}$ being consistent with the positive anchor $\mathbf{z}$.

With these derivatives, the representation of a negative sample can be
continuously updated given a step size of $\alpha$

(7)
$\mathbf{z}_{neg}\leftarrow\mathbf{z}_{neg}+\dfrac{\alpha}{\tau}\mathop{\mathbb{missing}}E\limits_{\mathbf{x}\sim\mathcal{D}}p(\mathbf{z}_{neg}|\mathbf{z})~{}\mathbf{z}.$

This shows that the memory-bank negatives are pushed towards the
representation of positive examples, making it harder to distinguish them from
their positive anchors.

The weight $p(\mathbf{z}_{neg}|\mathbf{z})$ in the above update rule implies a
self-boosted mechanism – the more similar the negative $\mathbf{z}_{neg}$ is
to a positive anchor $\mathbf{z}$, the closer it will be pushed to it. Such a
self-boosted mechanism makes it more and more challenging to distinguish the
negatives from the positive anchors as they are being mixed together.

The AdCo is a memory-based pretraining model, since its feature-level memory
bank of adversarial negatives are shared over iterations. This is in contrast
to the instance-wise perturbations that are distinctive to individual samples
in memory-free models. Figure 1 compares these two different types of
adversarial models.

###  2.4. Discussions

In this section, we elaborate on more subtle aspects to compare the memory-
free and memory-based adversarial approaches.

####  2.4.1. Computing Overheads: Two Passes vs. One Pass

In memory-free methods (Ho and Nvasconcelos, 2020; Kim et al., 2020; Jiang et
al., 2020), the instance-wise perturbation $\boldsymbol{\delta}^{\star}$ for
an individual sample is often computed via FGSM or PGD. This means that at
least one forward-and-backward pass is required for FGSM to compute the sample
gradient for obtaining the optimal perturbation (i.e., solving the inner
maximization in (4)), before another pass is applied to solve the weight
gradient to update the network (i.e., solving the outer minimization in (4)).
In other words, at least two passes through a deep network are required to
adversarially pretrain a deep network for each iteration.

In comparison, memory-based methods such as AdCo directly update the
adversarial negatives without feeding inputs through the network. Only a
single forward-and-backward pass is needed to update the network in an
iteration. This saves the computing cost by half, which does not incur the
computing overhead over the original contrastive learning algorithm (Hu et
al., 2021).

####  2.4.2. Combining Adversarial and Regular Pre-training

In the memory-free case, the adversarial contrastive loss with the instance-
wise perturbation is usually combined with the stand contrastive loss with
clean examples. In this sense, the adversarial contrastive loss is used as a
regularizer to the standard contrastive loss. Experiment (Jiang et al., 2020)
showed that the pre-training combining two losses attains better results.

On the contrary, the memory-based AdCo (Hu et al., 2021) is trained as a stand
alone loss over learned adversarial negatives without being combined with the
regular contrastive loss. In this sense, the AdCo model is not trained to be
robust against instance-wise attacks. Instead, it focuses on achieving better
generalization accuracies in downstream tasks by avoiding trivial solutions
with adversarial pretraining on pretext tasks.

####  2.4.3. Instance-wise Attacks with Feature-level Adversaries

Implicit Feature Modification (IFM) (Robinson et al., 2021) makes feature-
level attacks by adding perturbations to the representations of examples in a
batch. This is aligned with the AdCo (Hu et al., 2021) with the feature-level
adversarial negatives in the memory bank.

However, unlike the AdCo, the IFM is memory-free instance-wise attacks, whose
perturbations are made to individual instances. Specifically, given the
representation $\mathbf{z}$ of an anchor $\mathbf{x}$, its positive
$\mathbf{z}_{pos}$ and negative $\mathbf{z}_{neg}$ from a batch are updated by

(8) $\mathbf{z}_{pos}\leftarrow\mathbf{z}_{pos}-\epsilon\mathbf{z},$

and

(9) $\mathbf{z}_{neg}\leftarrow\mathbf{z}_{neg}+\epsilon\mathbf{z},$

where $\epsilon$ is the magnitude constraint of the perturbation.

The update (9) of IFM pushes a negative representation towards its assigned
positive anchor $\mathbf{z}$. On the contrary, AdCo applies the update (7) to
a memory-bank negative that is pushed towards a batch of positive samples
weighted by the probability $p(\mathbf{z}_{neg}|\mathbf{z})$, and AdCo shares
the memory-bank negatives for positive anchors in a batch. By sharing
negatives among those anchors, the negatives can be updated more efficiently
towards the most likely positive counterparts weighted by
$p(\mathbf{z}_{neg}|\mathbf{z})$, without being hard assigned to a specific
anchor. The experiments (Hu et al., 2021) also verified this by demonstrating
that competitive results can be achieved over fewer epochs with a smaller
batch.

Finally, compared with the standard instance-wise perturbations on raw inputs,
the feature-level adversaries including AdCo and IFM are more computationally
efficient. In particular, with high-resolution images, perturbations on raw
pixels could incur expensive computing cost on a large-scale dataset such as
Imagenet. This is perhaps why most of existing works (Ho and Nvasconcelos,
2020; Kim et al., 2020; Jiang et al., 2020) on instance-wise perturbations are
only evaluated on smaller-scale lower-resolution image datasets, such as
CIFAR-10, CIFAR-100 and STL. In this sense, the feature-level adversaries are
more amenable to large-scale datasets with high-resolution images.

##  3\. Adversarial Masked Image Modeling

In this section, we discuss the emerging problem of pretraining vision
transformers via Masked Image Modeling, and review how the adversarial
approaches have been proposed for adversarially pretraining these
transformers.

Note that the contrastive learning has been used to pretrain vision
transformers (Chen et al., 2021; Dosovitskiy et al., 2020). Thus, there is no
difficulty in extending the aforementioned contrastive pretraining to vision
transformers. However, due to its natural connection with the pretraining of
language transformers (Devlin et al., 2018; Brown et al., 2020), the Masked
Image Modeling (MIM) (He et al., 2022; Xie et al., 2022; Bao et al., 2021) has
attracted extensive attentions recently to pre-train the vision transformers.

In this section, we begin by briefly revisiting the MIM, followed by review of
adversarial MIM-pretraining approaches.

The MIM seeks to pre-train a deep network with an inpainting pretext task to
predict the missing patches from a masked input image. For an input image
$\mathbf{x}$ and a mask $\mathbf{m}$ randomly drawn from a distribution, an
encoder $f_{\theta}$ and a decoder $h_{\theta}$ are trained such that their
composition $\mathcal{I}_{\theta}=h_{\theta}\circ f_{\theta}$ can reconstruct
the original input $\mathbf{x}$ from the masked one,
$\mathbf{x}^{\mathbf{m}}=\mathbf{x}\odot\mathbf{m}$ with the element-wise
product $\odot$.

Then one adopts a Masked Auto-Encoder (MAE) (He et al., 2022) reconstruction
error

$\mathcal{L}_{mae}(\mathbf{x}^{\mathbf{m}},\mathbf{x};\theta)=\mathcal{D}(\mathcal{I}_{\theta}(\mathbf{x}^{\mathbf{m}}),\mathbf{x})$

with a distance measure $\mathcal{D}$ between reconstructed and original
images to pre-train the network $\theta$.

###  3.1. Direct Adversarial Approach

Instance-wise perturbations can be easily extended to adversarially pretrain
the network. Considering a constrained perturbation $\boldsymbol{\delta}$ to
the masked image $\mathbf{x}^{\mathbf{m}}$, we have the following minimax
problem for adversarial pre-training under the MIM framework

(10)
$\min\limits_{\theta}\mathop{\mathbb{missing}}E\limits_{\mathbf{x}\sim\mathcal{D}}\max\limits_{\|\boldsymbol{\delta}\|_{p}\leq\epsilon}\mathcal{L}_{mae}(\mathbf{x}^{\mathbf{m}}+\boldsymbol{\delta},\mathbf{x};\theta).$

The perturbation is added onto the masked image that is fed into the network
$\mathcal{I}_{\theta}$ to be adversarially pre-trained. Both FGSM (Goodfellow
et al., 2014) and PGD (Madry et al., 2017) approaches can be applied to solve
the optimal perturbation $\boldsymbol{\delta}$ for the inner maximization
problem.

###  3.2. Adversarial Masking

The above direct extension of instance-wise perturbations still relies on
randomly drawn masks. An alternative method (Shi et al., 2022) considers to
learn adversarial masks rather than the additive perturbations for the MIM-
based pre-training.

It learns a mask generating function $\mathcal{M}_{\phi}$ that maps an input
image $\mathbf{x}$ to a mask $\mathbf{m}$. The mask generating function is
implemented with a network that gives real-valued entries on $[0,1]$ for an
output mask. Then, the auto-encoder $\mathcal{I}_{\theta}$ and the mask
generating network $\mathcal{M}_{\phi}$ are jointly trained by

(11)
$\min\limits_{\theta}\max\limits_{\phi}\mathop{\mathbb{missing}}E\limits_{\mathbf{x}\sim\mathcal{D}}\mathcal{D}(\mathcal{I}_{\theta}(\mathbf{x}\odot\mathcal{M}_{\phi}(\mathbf{x})),\mathbf{x}),$

where the loss is maximized over $\phi$ to adversarially mask the input image.

In (Shi et al., 2022), multiple masks are generated for an input image
$\mathbf{x}$. This is beneficial as different masks can train the auto-encoder
network to reason about the relations between various parts of the image.
However, a trivial solution may arise if a mask is all zeros (everything
masked out) or all ones (nothing masked out). To avoid this problem, a
sparsity penalty is imposed to penalize all-zero and all-one masks.

A variety of distance measures $\mathcal{D}$ have been considered in (Shi et
al., 2022). Defined on the image level, the distance can be the pixel-wise
mean squared error as in MAE (He et al., 2022). Alternatively, the feature-
level distance
$\mathcal{D}(f_{\theta}(\mathbf{x}\odot\mathcal{M}_{\phi}(\mathbf{x})),f_{\theta}(\mathbf{x}))$
is also considered between the encoder outputs. In this case, to prevent the
encoder outputs from collapsing to a constant, the contrastive loss and the
BYOL loss can be adopted as the distance measure between original and
recovered images (Shi et al., 2022).

The adversarial masking in (11) can be viewed as a multiplicative adversary
compared with the additive adversary in (10).

###  3.3. Adversarial Positional Embeddings

The Adversarial Positional Embeddings (AdPE) (Authors, 2022) distorts the
spatial structure of images by imposing adversarial perturbations on the
positional embeddings in pretraining vision transformers. This seeks to
prevent the transformer from learning trivial representations by merely using
local correlations between tokens to predict missing patches. Instead, the
transformer is forced to learn high-level representations in global contexts
to infer the masked tokens as the local correlations are adversarially
distorted. Both absolute (Shaw et al., 2018) and relative positional
embeddings (Wu et al., 2021) are considered on which adversarial perturbations
are imposed. Moreover, the adversaries can be added in either embedding or
coordinate mode. Specifically, in the embedding mode, the adversarial
perturbations are added to the positional embeddings directly, while in the
coordinate mode, they are added to the image coordinates to disturb the
positional embeddings through a differentiable function. The latter is more
direct in a way that distorts the coordinates to prevent the pretrained
transformer from exploring the spatial correlations to predict missing
patches.

The experiment results in (Authors, 2022) demonstrate the adversarial
perturbations on relative positional embeddings in coordinate mode perform the
best among four variants. The formulation is as follows

$\min_{\boldsymbol{\theta},\mathbf{p}}\max_{\|\boldsymbol{\delta}\|_{q}\leq\epsilon}\mathcal{L}_{MIM}(\\{\mathbf{p}_{[\widetilde{g}(x_{i}-x_{j}+\delta_{x}),\widetilde{g}(y_{i}-y_{j}+\delta_{y})]}|i\in\mathbb{M}\\};\boldsymbol{\theta})$

where $\mathcal{L}_{MIM}$ is the MIM loss such as the prediction error used in
the MAE (He et al., 2022) when reconstructing missing matches,
$\mathbf{p}_{[u,v]}$ is the relative positional embedding with a 2D index of
$[u,v]$, $\widetilde{g}(x_{i}-x_{j}+\delta_{x})$ and
$\widetilde{g}(y_{i}-y_{j}+\delta_{y})$ are the continuous indexing function
for $x-$ and $y-$coordinate respectively, and $\mathbb{M}$ is the index of
masked patches. The spatial perturbation $\boldsymbol{\delta}$ is added on the
relative coordinates along x- and y-axis, and the indexing function
$\tilde{g}$ maps the perturbed relative coordinates onto real-valued indices
to a dictionary of positional embeddings in
$\\{\mathbf{p}_{[u,v]}|u,v=-K,\cdots,K\\}$ with bilinear interpolation
(Authors, 2022). The optimal $\boldsymbol{\delta}$ is solved via the PGD with
the network parameters $\boldsymbol{\theta}$ updated alternately by this
minimax problem.

##  4\. Other Related Methods

In addition to adversarial contrastive and MIM-based pretraining, there exist
other works on finding hard samples to boost the network pretraining,
investigating adversaries with other pretraining methods, and exploring non-
adversarial approaches for network robustness. We review them in this section.

###  4.1. Mining and Sampling Hard negatives

Both instance-wise perturbations and memory-based adversarial negatives seek
to find hard negatives to challenge the pre-training of deep networks so that
more discriminative features can be learned to distinguish between positive
and negative pairs. While both methods are derived by solving an optimization
problem, other approaches such as directly mining (Kalantidis et al., 2020) or
sampling (Robinson et al., 2020) hard negatives have also been studied.

Hard negative mining. In (Kalantidis et al., 2020), the top-$N$ similar
negatives to a query are selected and mixed with the query and each other to
generate harder negative samples. These harder negatives are appended to
existing negative samples for the contrastive pre-training, yielding a hard
negative mining approach.

Hard negative sampling. In (Robinson et al., 2020), a negative sampling
distribution was formed to draw hard negatives that are close to a query
anchor with different labels. Importance sampling was applied to sample such
hard examples with a concentration hyperparameter $\beta$ controlling the
hardness degree. The idea of debiased contrastive learning (Chuang et al.,
2020) from positive-unlabeled learning (Du Plessis et al., 2014; Elkan and
Noto, 2008) was adopted to sample these hard negatives without access to true
labels.

Hard negative mining and sampling approaches differ from the aforementioned
adversarial contrastive pretraining as these hard negatives are not derived by
maximizing the InfoNCE contrastive loss. Although these negative samples can
also be viewed as adversarial in a sense that makes the pretraining task
harder, they are not the worst-case adversarial examples as in typical
adversarial approaches.

###  4.2. Adversarial Methods Beyond Contrastive and MIM Pre-training

The idea of adversarially pre-training deep networks goes beyond the CL and
MIM frameworks. In particular, the instance-wise adversarial perturbations can
easily be adapted to other self-supervised pre-training approaches such as
BYOL (Grill et al., 2020), a self-supervised method by minimizing a mean-
squared error between the representations of two transformed examples.

Formally in (Gowal et al., 2020), given a sample $\mathbf{x}$ along with its
two augmented views $t(\mathbf{x})$ and $t^{\prime}(\mathbf{x})$, they will go
through an online and a target network, respectively, resulting in two
embedded vectors $\mathbf{z}$ and $\mathbf{z}^{\prime}$. The online and target
networks share the same architecture, while the weights of the target network
are an exponential moving average of its online counterpart. A MLP predictor
is added onto the online network to predict $\mathbf{z}^{\prime}$ from
$\mathbf{z}$. Putting the composition of the encoder, the projector and the
predictor into $\kappa$, the mean-squared error of the BYOL objective can be
written as

$\mathcal{L}_{byol}(t(\mathbf{x}),t^{\prime}(\mathbf{x});\theta)=\left\|\dfrac{\kappa(t(\mathbf{x}))}{\|\kappa(t(\mathbf{x}))\|_{2}}-\dfrac{\mathbf{z}^{\prime}}{\|\mathbf{z}^{\prime}\|_{2}}\right\|_{2}^{2}$

An instance-wise attack is performed (Gowal et al., 2020) by adding a
constrained perturbation $\boldsymbol{\delta}$ to $t(\mathbf{x})$,

$\arg\limits_{\boldsymbol{\delta}}\max\limits_{\|\boldsymbol{\delta}\|_{p}\leq\epsilon}\mathcal{L}_{byol}(t(\mathbf{x})+\boldsymbol{\delta},t^{\prime}(\mathbf{x});\theta).$

This problem can be solved iteratively as

$\boldsymbol{\delta}\leftarrow\Pi_{\|\boldsymbol{\delta}\|_{p}\leq\epsilon}[\boldsymbol{\delta}+\alpha\nabla_{\boldsymbol{\delta}}\mathcal{L}_{byol}(t(\mathbf{x})+\boldsymbol{\delta},t^{\prime}(\mathbf{x});\theta)]$

where $\alpha$ is the step size, and $\Pi$ is the projection onto the
constraint perturbation.

With the optimal perturbation $\boldsymbol{\delta}^{\star}$, the BYOL loss is
minimized to update the network weights as in other adversarial pretraining
methods.

For other pretraining approaches, the instance-wise attacks can be formulated
in the similar manner by maximizing the underlying loss to perturb on an
instance input, followed alternately by minimizing the loss to update the
network weights.

###  4.3. Non-adversarial Methods for Adversarial Robustness

Besides leveraging adversarial examples to make pre-trained networks robust to
potential adversaries in downstream tasks, non-adversarial pretraining has
also been developed without finding adversarial examples in the pretraining
stage.

Inspired by randomized smoothing (Cohen et al., 2019; Liu et al., 2018; Cao
and Gong, 2017; Teng et al., 2019; Zhang et al., 2019b; Levine and Feizi,
2020; Yang et al., 2020), in (Pang et al., 2022), first one augmented view
$t^{\prime}(\mathbf{x})$ of an input instance $\mathbf{x}$ is randomly
smoothed via a distribution $\mathcal{Q}$. For example, one can adopt
$\mathcal{Q}(\mathbf{x})=\mathbf{x}+\boldsymbol{\delta}$, with an isotropic
$d$-dimensional Gaussian distribution
$\boldsymbol{\delta}\sim\mathcal{N}_{d}(0,\sigma^{2})$ or a multi-dimensional
uniform distribution $\boldsymbol{\delta}\sim U_{d}(-\mu,\mu)$.

Then, a simple non-adversarial pre-training objective attempts to pre-train a
network parameterized with $\theta$ by minimizing a self-supervised loss
$\mathcal{L}_{pre}$,

$\min\limits_{\theta}\mathop{\mathbb{missing}}E\limits_{t,t^{\prime}\sim\mathcal{T},\mathbf{x}\sim\mathcal{D}}\mathcal{L}_{pre}(t(\mathbf{x}),t^{\prime}(\mathbf{x});\theta)$

where $\mathcal{L}_{pre}$ can be chosen to any unsupervised objective such as
contrastive loss based on SimCLR, and the BYOL loss minimizing the prediction
error between online and target branches.

Once the network is pretrained, it is fine-tuned on downstream tasks, and the
robustness of the fine-tuned network can be evaluated based on the adversarial
examples resulting from a majority or an average voting (Pang et al., 2022).

##  5\. Evaluation Protocols

After a deep network is adversarially pretrained, it needs to be adapted into
a downstream task such as image classification to evaluate its performance on
standard accuracy with clean data and/or robustness to various adversarial
attacks. While adversarial pretraining does not involve any labels, the
pretrained network usually should be retrained with labeled examples for the
downstream task. In literature, for the evaluation purpose the downstream
retraining has been performed in various ways, which are discussed below.

Linear evaluation and robust linear evaluation. In self-supervised learning,
the projector of a pretrained network is removed and replaced with a linear
classifier. While the pretrained feature encoder is fixed, the linear
classifier is trained with labeled examples. The linear classifier can be
regularly trained by minimizing the cross-entropy loss with clean examples, or
adversarially trained resulting in the robust linear evaluation as in (Kim et
al., 2020).

Supervised fine-tuning and adversarial fine-tuning. Instead of training a
linear classifier by fixing the pretrained encoder, the whole network can be
supervisedly fine-tuned end-to-end merely by initializing the encoder with the
adversarially pretrained model (Fan et al., 2021). Similarly, the supervised
fine-tuning can be performed regularly with clean data or adversarially with
adversarial examples.

Standard accuracy vs. robustness. Standard accuracy is evaluated by applying
clean examples and their groundtruth labels on the retrained model. This tests
how well the model performs on regular data. On the contrary, the robustness
is concerned with the adversarial attacks – the retrained model is evaluated
with adversarial examples generated with various attacking models, e.g., PGD
attacks (Madry et al., 2017) with different types of norms ($\ell_{\infty}$,
$\ell_{1}$, or $\ell_{2}$), their magnitudes, and the number of PGD steps.
Black-box attacks generated from AT (Madry et al., 2017) and TRADES (Zhang et
al., 2019c) may also be used for the evaluation (Kim et al., 2020).

Table 3. Comparison between adversarial pretraining methods based on different
evaluation protocols on CIRAR-10 and CIFAR-100. The results are reported in
terms of both standard and robust accuracies based on $\ell_{\infty}$-attacks.
AT denotes supervised adversarial training (Madry et al., 2017), and SS
denotes self-supervised loss used for fine-tuning (Kim et al., 2020). * uses
the linear evaluation based on voting strategy with randomized smoothing (Pang
et al., 2022). ${\dagger}$ denotes the model pretrained on Image100-S and
evaluated on downstream CIFAR-10 and CIFAR-100 datasets in the transfer
learning setting with the base SimCLR included in the parenthesis for
comparison (Shi et al., 2022). (a) CIFAR-10

Method | Evaluation Protocol | Standard Accuracy | Robust Accuracy
---|---|---|---
AT (Madry et al., 2017) | adversarial fine-tuning | 78.99 | 47.41
TRADES (Zhang et al., 2019c) | adversarial fine-tuning | 81.00 | 53.27
SimCLR (Kim et al., 2020) | linear evaluation | 91.25 | 0.63
RoCL (Kim et al., 2020) | linear evaluation | 83.71 | 40.27
RoCL (Kim et al., 2020) | robust linear | 80.43 | 47.69
RoCL+AT+SS (Kim et al., 2020) | adversarial fine-tuning | 91.34 | 49.66
ACL (Jiang et al., 2020) | adversarial fine-tuning | 82.19 | 52.82
SimCLR+CLAE (Ho and Nvasconcelos, 2020) | robust linear | 76.70$\pm$0.36 | -
ADVCL (Fan et al., 2021) | linear evaluation | 81.35 | 51.00
ADVCL (Fan et al., 2021) | robust linear | 79.24 | 52.38
ADVCL (Fan et al., 2021) | adversarial fine-tuning | 83.67 | 53.35
RUSH (Pang et al., 2022) | linear evaluation∗ | 87.9 | 79.5
SimCLR+IFM (Robinson et al., 2021) | linear evaluation | 91.9 | -
SimCLR+ADIOS (Shi et al., 2022) | linear evaluation | 34.6 (30.1)† | -
SimCLR+ADIOS (Shi et al., 2022) | supervised fine-tuning | 93.4 (91.3)† | -

(b) CIFAR-100

Method | Evaluation Protocol | Standard Accuracy | Robust Accuracy
---|---|---|---
AT (Madry et al., 2017) | adversarial fine-tuning | 49.49 | 23.00
TRADES (Zhang et al., 2019c) | adversarial fine-tuning | 54.59 | 28.43
ACL (Jiang et al., 2020) | adversarial fine-tuning | 56.77 | 28.33
SimCLR+CLAE (Ho and Nvasconcelos, 2020) | robust linear | 55.52$\pm$0.3 | -
ADVCL (Fan et al., 2021) | linear evaluation | 47.98 | 27.99
ADVCL (Fan et al., 2021) | robust linear | 47.45 | 28.29
ADVCL (Fan et al., 2021) | adversarial fine-tuning | 57.87 | 29.48
RUSH (Pang et al., 2022) | linear evaluation∗ | 57.1 | 46.6
SimCLR+IFM (Robinson et al., 2021) | linear evaluation | 68.8 | -
SimCLR+ADIOS (Shi et al., 2022) | linear evaluation | 11.0 (10.2)† | -
SimCLR+ADIOS (Shi et al., 2022) | supervised fine-tuning | 71.8 (70.0)† | -

Table 4. Comparison of CL- and MIM-pretrained model on ViT-B and ViT-L. The supervised fine-tuning is performed with Imagenet labels in terms of the top-1 on the Imagenet1k dataset. Among them are the adversarial approaches including FGSM and AdPE. By comparison, the instance-wise adversarial model – FGSM does not improve the top-1 accuracy compared to its base model MAE+, while the AdPE performs better than MAE+. Method | Type | Adversarial | Extra Model | Epochs | ViT-B | ViT-L
---|---|---|---|---|---|---
supervised (He et al., 2022) | Supervised | no | - | 300 | 82.3 | 82.6
MoCo-v3 (Chen et al., 2021) | Contrastive | no | momentum ViT | 300 | 83.2 | 84.1
DINO (Caron et al., 2021) | Contrastive | no | momentum ViT | 300 | 82.8 | -
iBOT (Zhou et al., 2021) | Contrastive+MIM | no | momentum ViT | 1600 | 84.0 | 84.8
BEiT (Bao et al., 2021) | MIM | no | DALLE+dVAE | 800 | 83.2 | 85.2
data2vec (Baevski et al., 2022) | Contrastive | no | momentum ViT | 800 | 84.2 | 86.2
CAE (Chen et al., 2022) | MIM | no | DALLE tokenizer | 1600 | 83.9 | 86.3
SimMIM (Xie et al., 2022) | MIM | no | - | 800 | 83.8 | 85.4
MaskFeat (Wei et al., 2022) | MIM | no | - | 1600 | 84.0 | 85.7
MAE (He et al., 2022) | MIM | no | - | 1600 | 83.6 | 85.9
MAE+ (Authors, 2022) | MIM | no | - | 400 | 83.51 | -
MAE+ (Authors, 2022) | MIM | no | - | 1600 | 83.9 | 86.0
FGSM (Goodfellow et al., 2014) | MIM | yes | - | 400 | 83.09 | -
AdPE (Authors, 2022) | MIM | yes | - | 1600 | 84.4 | 86.3

The large variety of objectives and evaluation protocols make it complicated
to compare different adversarial pretraining models. For example, most of
instance-wise adversarial pretraining models (Kim et al., 2020; Ho and
Nvasconcelos, 2020; Jiang et al., 2020; Robinson et al., 2021; Gowal et al.,
2020; Pang et al., 2022) are intentionally developed to be robust against
instance-wise attacks, and are thus evaluated based on the robust accuracy.
Meanwhile, they are also assessed in standard accuracy on downstream tasks
with clean examples. Table 3 summarizes some reported results in terms of
different evaluations protocols. Most of evaluations are performed on CIFAR-10
(Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), and STL-10
(Coates et al., 2011). Standard and robust accuracies are often at odds with
one another. This is not surprising as these models are developed purposely in
defence of such attacks, and a trade-off usually exists to sacrifice standard
accuracy for robustness against various attacks. This is also aligned with the
findings from adversarial training (Tsipras et al., 2018).

On the contrary, the feature-level adversarial pretraining approaches such as
memory-based adversarial contrastive learning (Hu et al., 2021; Wang et al.,
2022b), hard negative mining and sampling (Robinson et al., 2020; Kalantidis
et al., 2020) and MIM-based adversarial pretraining of transformers (Authors,
2022; Shi et al., 2022) are often more concerned about the generalization
ability measured by their accuracies on downstream tasks, since these methods
are adversarially pre-trained to avoid learning trivial features without being
instance-wise attacked. The pretrained network is fine-tuned for downstream
tasks without adversarial examples, and the fine-tuned network is evaluated on
clean test examples. As these models are concerned with the improvements over
the other state-of-the-art self-supervised methods, for the sake of a fair
comparison, they are usually evaluated on Imagenet1K (Deng et al., 2009).
Table 4 and Table 5 compares the top-1 accuracy of various self-supervised
pretraining methods on Imagenet1K with vision transformers (ViT-B and ViT-L)
and ResNet-50 backbones. The results show that the adversarially pretrained
networks can improve the accuracy over fewer pretraining epochs, demonstrating
their efficiency in exploring adversarial examples to learn more
discriminative features generalizable to future tasks.

##  6\. Emerging Trends and Future Directions

In this section, we will reveal and discuss several aspects of emerging trends
and potential future directions.

###  6.1. A Hybrid Model: Adversarial vs. Cooperative Pretraining

Figure 2.  (a) A hybrid model with model-free instance-wise perturbations and model-based adversarial negatives. Instead of sticking with a positive sample from the memory bank, a hybrid model will apply instance-wise perturbation to each query $t(\mathbf{x})$. This combines the memory-free instance-wise perturbations and memory-based adversarial negatives. Both perturbations and negatives are adversarially learned. (b) Memory-based model with cooperative positives and adversarial negatives. For each query sample $t(\mathbf{x})$, a sample is picked from the memory bank as its positive counterpart. This positive sample chosen from the memory bank is learned cooperatively rather than adversarially by minimizing the contrastive loss. Table 5. Comparison between adversarial pretraining with cooperative positives and other methods (Wang et al., 2022b). Top-1 accuracy under the linear evaluation on Imagenet1K with the ResNet-50 backbone. The table compares the methods using a single crop augmentation pre-trained with various epochs. As mentioned in the introduction of Section 1, these pretraining approaches are learning on feature level adversaries (if any) rather than upon instance-wise perturbations, and thus they aim to learn more generalizable representations instead of those robust against certain type of instance-level attacks. Method | epochs | batch size | top-1 | adversarial | cooperative
---|---|---|---|---|---
InstDisc (Wu et al., 2018) | 200 | 256 | 58.5 | no | no
MoCHi (Kalantidis et al., 2020) | 200 | 512 | 68.0 | yes | no
MoCHi (Kalantidis et al., 2020) | 800 | 512 | 68.7 | yes | no
SeLa (Asano et al., 2019) | 400 | 256 | 61.5 | no | no
PIRL (Misra and Maaten, 2020) | 800 | 1024 | 63.6 | no | no
CMC (Tian et al., 2019) | 240 | 128 | 66.2 | no | no
SimCLR (Chen et al., 2020b) | 200 | 256 | 61.9 | no | no
SimCLR (Chen et al., 2020b) | 800 | 4096 | 69.3 | no | no
PIC (Cao et al., 2020) | 200 | 512 | 67.6 | no | no
PIC (Cao et al., 2020) | 1600 | 512 | 70.8 | no | no
CPC v2 (Hénaff et al., 2019) | 200 | 512 | 63.8 | no | no
PCL v2 (Li et al., 2020) | 200 | 256 | 67.6 | no | no
MoCo v2 (Chen et al., 2020a) | 200 | 256 | 67.5 | no | no
MoCo v2 (Chen et al., 2020a) | 800 | 256 | 71.1 | no | no
MoCo v3 (Chen et al., 2021) | 1000 | 4096 | 74.6 | no | no
InfoMin Aug (Tian et al., 2020) | 800 | 256 | 73.0 | no | no
SimSiam (Chen and He, 2021) | 200 | 256 | 70.0 | no | no
SimSiam (Chen and He, 2021) | 800 | 256 | 71.3 | no | no
SWAV (Caron et al., 2020) | 200 | 4096 | 69.1 | no | no
SWAV (Caron et al., 2020) | 800 | 4096 | 71.8 | no | no
NNCLR (Dwibedi et al., 2021) | 200 | 4096 | 70.7 | no | no
NNCLR (Dwibedi et al., 2021) | 800 | 4096 | 74.9 | no | no
BYOL (Grill et al., 2020) | 200 | 4096 | 70.6 | no | no
BYOL (Grill et al., 2020) | 1000 | 4096 | 74.3 | no | no
AdCo (Hu et al., 2021) | 200 | 256 | 68.2 | yes | no
AdCo (Hu et al., 2021) | 800 | 256 | 72.8 | yes | no
CaCo (Wang et al., 2022b) | 200 | 1024 | 71.3 | yes | yes
CaCo (Wang et al., 2022b) | 800 | 4096 | 75.4 | yes | yes

In Section 2.3, we discussed how adversarial contrastive pretraining is
applied in (Hu et al., 2021) to generate adversarial negatives. However, an
unanswered question there is whether and how to perform adversaries against
positives. As shown in Eq. (7), the adversaries are only performed on the
shared negatives. This is in contrast to instance-wise attacks in Section 2.2,
where the positives are also perturbed adversarially to pre-train the deep
network.

There is a natural extension by combinng such shared negative adversaries in
AdCo (Hu et al., 2021) and the instance-wise attacks on positives. Given a
pair of positives $t(\mathbf{x})$ and $t^{\prime}(\mathbf{x})$ transformed
from the same instance $\mathbf{x}$ with $t,t^{\prime}\sim\mathcal{T}$, we can
still apply an instance-wise attack $\boldsymbol{\delta}$ against
$t(\mathbf{x})$, leading to a instance-wise adversarial positive as in Eq.
(5). Combining the perturbed positive by $\boldsymbol{\delta}$ on
$t(\mathbf{x})$ with the AdCo, it yields a hybrid model with instance-wise
perturbations and memory-based adversarial negatives.

Alternatively, if we wish to stick with the memory-based model for positive
samples, it turns out that we need to abandon adversarial positives and
instead switch to cooperative ones (Wang et al., 2022b). This is not hard to
understand – with all positives in the memory bank shared among query anchors
in a batch, adversarial updates to these positives will push them away from
these query anchors . This will push these positives into a region of feature
space that is far away from the representations of query anchors as for shared
adversarial negatives in AdCo. It will eventually make the network pre-
training meaningless if the positives become completely irrelevant to the
learned representations of incoming query samples.

Figure 2 compares the memory-based model with cooperative positives and the
aforementioned hybrid model. Indeed, (Wang et al., 2022b) studied this problem
and revealed that the positives shared in a memory bank ought to be learned
cooperatively rather than adversarially, that is

(12)
$\mathbf{z}_{pos}\leftarrow\mathbf{z}_{pos}+\dfrac{1}{\tau}[1-p(\mathbf{z}_{pos}|\mathbf{z})]\mathbf{z},$

where $\mathbf{z}_{pos}$ is a positive chosen from the memory bank for a query
anchor $\mathbf{z}$. Table 5 compares such a cooperative positive model with
the other adversarial pre-training results based on the linear evaluation
protocol under various batch sizes and pre-training epochs.

On the contrary, adversarial updates to memory-bank positives would lead to
trivial results in memory-based model. For example, if the cooperative
positives in (12) were changed to the following adversarial formulation

(13)
$\mathbf{z}_{pos}\leftarrow\mathbf{z}_{pos}-\dfrac{1}{\tau}[1-p(\mathbf{z}_{pos}|\mathbf{z})]\mathbf{z},$

the network pre-training would collapse to trivial representations, as
evidenced by a very low accuracy in downstream tasks close to zero (Wang et
al., 2022b).

Now we have an open problem to study in future about the essential connection
between instance-wise perturbations and memory-based adversarial model – can
we still perform adversarial attacks against shared positives without over-
memorizing them in the memory bank while avoiding the trivial results? How
much advantage can we obtain from cooperative positives over adversarial
perturbations, or vice versa? These are open questions for future research.

###  6.2. Unifying Adversarial Contrastive Learning and Masked Auto-Encoders

In this paper, we have discussed the adversarial pretraining based on
contrastive learning and MIM, respectively. Existing works (Zhou et al., 2021;
Huang et al., 2022) have demonstrated that by combining the objectives for
contrastive learning and MIM for pretraining vision transformers, more
generalizable representations can be learned.

Essentially, the contrastive loss is designed as a classification objective of
distinguishing between the augmented views of different instances. On the
other hand, the MIM objective is related with a generative task of
reconstructing masked image patches by exploring the relations with unmasked
contexts. Combing two objectives is expected to learn both discriminative and
generative features that well generalize to future tasks.

There are evidences showing the MIM representations perform well when it is
fine-tuned end-to-end to a downstream task, while the contrastively pretrained
features are better in linear evaluations by training a linear classifier by
freezing the pretrained network. Combing the contrastive pretraining with the
MIM objective further could improve the generalization accuracy in downstream
tasks (Zhou et al., 2021; Huang et al., 2022).

However, no effort has been made to explore the adversarial pretraining along
this direction. A straight idea is to find the worst-case perturbations by
jointly maximizing the contrastive loss and the MAE loss of a MIM task, and to
use the obtained adversarial perturbations to train a robust representation by
minimizing the joint loss. Then, will this lead to improved generalization
accuracy and/or robustness to adversarial attacks? Are there any other ways to
pre-train both discriminative and generative representations that robustly
generalize to the future applications? These questions deserve further studies
in future.

###  6.3. Accuracy vs. Robustness

The dual goal of adversarial pre-training is to achieve better results in both
standard accuracy and robustness against adversarial attacks. The standard
accuracy is related to the generalization ability of pre-trained
representations to downstream applications that usually have a different
objective from the pretext task for pretraining. The robustness is usually
concerned with adversarial attacks derived from the downstream tasks. For
example, one uses contrastive pretraining to learn an unsupervised
representation with unlabeled data, and it is then fine-tuned to a
classification task. In this case, we are concerned with the classification
accuracy on the test set, as well as its robustness against adversarial
examples.

Empirical evidences have shown that a high robustness against worst-case
adversarial examples often comes with a low standard accuracy (Tsipras et al.,
2018). In particular, such an accuracy-robustness dilemma is prevailing in
instance-wise perturbations (Kim et al., 2020; Ho and Nvasconcelos, 2020;
Jiang et al., 2020; Robinson et al., 2021; Gowal et al., 2020; Pang et al.,
2022), since these models are designed to be pre-trained against adversarially
perturbed examples – they are trained to be invariant to these perturbation
noises.

On the contrary, memory-based adversarial pre-training (Hu et al., 2021; Wang
et al., 2022b) abandons the direct perturbations on individual instances, and
instead learns shared adversarial negatives that are hard to be distinguished
from their positive counterparts. It thus focuses on learning more
discriminative features that can yield better generalization performance on
downstream tasks. For example, Table 5 shows these memory-based adversarial
pre-training has competitive results over the other instance-wise pre-training
models.

Therefore, opens questions are: is it an intrinsic nature that one must trade
generalizability for robustness, or vice versa? Is it possible for us to
combine the advantages of memory-free instance-wise attacks and memory-base
adversarial pretraining to achieve both high accuracy and strong robustness?
Answering these questions demands tremendous efforts from both theoretical and
practical perspectives.

##  7\. Conclusion

This paper reviews the adversarial pretraining by grouping related works into
memory-free instance-wise attacks and memory-based adversaries employing both
raw input and features. While the former imposes sample-specific perturbations
on individual examples, the latter applies a shared memory of adversaries
during the model pretraining. We review the adversarial pretraining methods
aligned with Contrastive Learning (CL), Masked Image Modeling (MIM) and
beyond. The issues related to computing overheads, input and feature level
adversaries, and a hybrid objective combining adversarial and standard losses
are also reviewed. Emerging trends lead to several open problems about
adversarial or cooperative positives in contrastive pretraining, combining
contrastive and MIM-based adversaries, and the intrinsic trade-off between
accuracy and robustness for adversarial pretraining. The answers to these open
problems need tremendous efforts in order to to reveal the connections between
the pretext task for adversarial pretraining and the trade-off or unification
of the generalizability and robustness in downstream tasks.

## References

  * (1)
  * Agrawal et al. (2015) Pulkit Agrawal, Joao Carreira, and Jitendra Malik. 2015\.  Learning to see by moving. In _Proceedings of the IEEE international conference on computer vision_. 37–45.
  * Akhtar and Mian (2018) Naveed Akhtar and Ajmal Mian. 2018.  Threat of adversarial attacks on deep learning in computer vision: A survey.  _Ieee Access_ 6 (2018), 14410–14430.
  * Akhtar et al. (2021) Naveed Akhtar, Ajmal Mian, Navid Kardan, and Mubarak Shah. 2021\.  Advances in adversarial attacks and defenses in computer vision: A survey.  _IEEE Access_ 9 (2021), 155161–155196.
  * Asano et al. (2019) Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. 2019.  Self-labelling via simultaneous clustering and representation learning.  _arXiv preprint arXiv:1911.05371_ (2019).
  * Authors (2022) Anonymous Authors. 2022\.  AdPE: Adversarial Positional Embeddings for Pretraining Vision Transformers via MAE+. (October 2022).
  * Baevski et al. (2022) Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. 2022\.  Data2vec: A general framework for self-supervised learning in speech, vision and language.  _arXiv preprint arXiv:2202.03555_ (2022).
  * Bao et al. (2021) Hangbo Bao, Li Dong, and Furu Wei. 2021.  Beit: Bert pre-training of image transformers.  _arXiv preprint arXiv:2106.08254_ (2021).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Cao and Gong (2017) Xiaoyu Cao and Neil Zhenqiang Gong. 2017.  Mitigating evasion attacks to deep neural networks via region-based classification. In _Proceedings of the 33rd Annual Computer Security Applications Conference_. 278–287.
  * Cao et al. (2020) Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin, Zheng Zhang, and Han Hu. 2020\.  Parametric instance classification for unsupervised visual feature learning.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Carlini and Wagner (2017) Nicholas Carlini and David Wagner. 2017.  Towards evaluating the robustness of neural networks. In _2017 ieee symposium on security and privacy (sp)_. Ieee, 39–57.
  * Caron et al. (2020) Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020\.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in Neural Information Processing Systems_ 33 (2020), 9912–9924.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021\.  Emerging properties in self-supervised vision transformers. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 9650–9660.
  * Chen et al. (2020b) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020b.  A simple framework for contrastive learning of visual representations.  _arXiv preprint arXiv:2002.05709_ (2020).
  * Chen et al. (2022) Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. 2022\.  Context autoencoder for self-supervised representation learning.  _arXiv preprint arXiv:2202.03026_ (2022).
  * Chen et al. (2020a) Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020a.  Improved baselines with momentum contrastive learning.  _arXiv preprint arXiv:2003.04297_ (2020).
  * Chen and He (2021) Xinlei Chen and Kaiming He. 2021.  Exploring simple siamese representation learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 15750–15758.
  * Chen et al. (2021) Xinlei Chen, Saining Xie, and Kaiming He. 2021.  An empirical study of training self-supervised vision transformers. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 9640–9649.
  * Chuang et al. (2020) Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. 2020\.  Debiased contrastive learning.  _Advances in neural information processing systems_ 33 (2020), 8765–8775.
  * Coates et al. (2011) Adam Coates, Andrew Ng, and Honglak Lee. 2011.  An analysis of single-layer networks in unsupervised feature learning. In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 215–223.
  * Cohen et al. (2019) Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. 2019\.  Certified adversarial robustness via randomized smoothing. In _International Conference on Machine Learning_. PMLR, 1310–1320.
  * Deng et al. (2009) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009.  ImageNet: A Large-Scale Hierarchical Image Database. In _CVPR09_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Doersch et al. (2015) Carl Doersch, Abhinav Gupta, and Alexei A Efros. 2015\.  Unsupervised visual representation learning by context prediction. In _Proceedings of the IEEE international conference on computer vision_. 1422–1430.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).
  * Du Plessis et al. (2014) Marthinus C Du Plessis, Gang Niu, and Masashi Sugiyama. 2014\.  Analysis of learning from positive and unlabeled data.  _Advances in neural information processing systems_ 27 (2014).
  * Dwibedi et al. (2021) Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. 2021\.  With a little help from my friends: Nearest-neighbor contrastive learning of visual representations.  _arXiv preprint arXiv:2104.14548_ (2021).
  * Elkan and Noto (2008) Charles Elkan and Keith Noto. 2008.  Learning classifiers from only positive and unlabeled data. In _Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_. 213–220.
  * Fan et al. (2021) Lijie Fan, Sijia Liu, Pin-Yu Chen, Gaoyuan Zhang, and Chuang Gan. 2021.  When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?  _Advances in Neural Information Processing Systems_ 34 (2021), 21480–21492.
  * Gidaris et al. (2018) Spyros Gidaris, Praveer Singh, and Nikos Komodakis. 2018\.  Unsupervised representation learning by predicting image rotations.  _arXiv preprint arXiv:1803.07728_ (2018).
  * Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014\.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ (2014).
  * Gowal et al. (2020) Sven Gowal, Po-Sen Huang, Aaron van den Oord, Timothy Mann, and Pushmeet Kohli. 2020.  Self-supervised adversarial robustness for the low-label, high-data regime. In _International Conference on Learning Representations_.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. 2020\.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ 33 (2020), 21271–21284.
  * Gupta et al. (2022) Rohit Gupta, Naveed Akhtar, Ajmal Mian, and Mubarak Shah. 2022\.  On Higher Adversarial Susceptibility of Contrastive Self-Supervised Learning.  _arXiv preprint arXiv:2207.10862_ (2022).
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022\.  Masked autoencoders are scalable vision learners. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16000–16009.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 9729–9738.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hénaff et al. (2019) Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. 2019\.  Data-efficient image recognition with contrastive predictive coding.  _arXiv preprint arXiv:1905.09272_ (2019).
  * Ho and Nvasconcelos (2020) Chih-Hui Ho and Nuno Nvasconcelos. 2020.  Contrastive learning with adversarial examples.  _Advances in Neural Information Processing Systems_ 33 (2020), 17081–17093.
  * Hu et al. (2021) Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. 2021\.  Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1074–1083.
  * Huang et al. (2022) Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. 2022\.  Contrastive Masked Autoencoders are Stronger Vision Learners.  _arXiv preprint arXiv:2207.13532_ (2022).
  * Jiang et al. (2020) Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. 2020\.  Robust pre-training by adversarial contrastive learning.  _Advances in Neural Information Processing Systems_ 33 (2020), 16199–16210.
  * Kalantidis et al. (2020) Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. 2020.  Hard negative mixing for contrastive learning.  _Advances in Neural Information Processing Systems_ 33 (2020), 21798–21809.
  * Kim et al. (2018) Dahun Kim, Donghyeon Cho, Donggeun Yoo, and In So Kweon. 2018\.  Learning image representations by completing damaged jigsaw puzzles. In _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 793–802.
  * Kim et al. (2020) Minseon Kim, Jihoon Tack, and Sung Ju Hwang. 2020.  Adversarial self-supervised contrastive learning.  _arXiv preprint arXiv:2006.07589_ (2020).
  * Krizhevsky et al. (2009) Alex Krizhevsky et al. 2009\.  Learning multiple layers of features from tiny images.  (2009).
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ 25 (2012).
  * Kurakin et al. (2016) Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016\.  Adversarial machine learning at scale.  _arXiv preprint arXiv:1611.01236_ (2016).
  * Larsson et al. (2016) Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. 2016\.  Learning representations for automatic colorization. In _European conference on computer vision_. Springer, 577–593.
  * Levine and Feizi (2020) Alexander Levine and Soheil Feizi. 2020.  Robustness certificates for sparse adversarial attacks by randomized ablation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 4585–4593.
  * Li et al. (2020) Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. 2020.  Prototypical Contrastive Learning of Unsupervised Representations.  _arXiv preprint arXiv:2005.04966_ (2020).
  * Liu et al. (2018) Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. 2018\.  Towards robust neural networks via random self-ensemble. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 369–385.
  * Madry et al. (2017) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017\.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ (2017).
  * Misra and Maaten (2020) Ishan Misra and Laurens van der Maaten. 2020.  Self-supervised learning of pretext-invariant representations. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6707–6717.
  * Moosavi-Dezfooli et al. (2016) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016\.  Deepfool: a simple and accurate method to fool deep neural networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2574–2582.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Pang et al. (2022) Yijiang Pang, Boyang Liu, and Jiayu Zhou. 2022.  RUSH: Robust Contrastive Learning via Randomized Smoothing.  _arXiv preprint arXiv:2207.05127_ (2022).
  * Papernot et al. (2016) Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. 2016.  Distillation as a defense to adversarial perturbations against deep neural networks. In _2016 IEEE symposium on security and privacy (SP)_. IEEE, 582–597.
  * Qi and Luo (2020) Guo-Jun Qi and Jiebo Luo. 2020.  Small data challenges in big data era: A survey of recent progress on unsupervised and semi-supervised methods.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2020).
  * Qi et al. (2019) Guo-Jun Qi, Liheng Zhang, and Xiao Wang. 2019.  Learning generalized transformation equivariant representations via autoencoding transformations.  _arXiv preprint arXiv:1906.08628_ (2019).
  * Robinson et al. (2020) Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2020.  Contrastive learning with hard negative samples.  _arXiv preprint arXiv:2010.04592_ (2020).
  * Robinson et al. (2021) Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. 2021\.  Can contrastive learning avoid shortcut solutions?  _arXiv preprint arXiv:2106.11230_ (2021).
  * Shaw et al. (2018) Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018\.  Self-attention with relative position representations.  _arXiv preprint arXiv:1803.02155_ (2018).
  * Shi et al. (2022) Yuge Shi, N Siddharth, Philip Torr, and Adam R Kosiorek. 2022\.  Adversarial Masking for Self-Supervised Learning. In _International Conference on Machine Learning_. PMLR, 20026–20040.
  * Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013\.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ (2013).
  * Teng et al. (2019) Jiaye Teng, Guang-He Lee, and Yang Yuan. 2019.  $ell\\_1$ Adversarial Robustness Certificates: a Randomized Smoothing Approach.  (2019).
  * Tian et al. (2019) Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019\.  Contrastive multiview coding.  _arXiv preprint arXiv:1906.05849_ (2019).
  * Tian et al. (2020) Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. 2020\.  What makes for good views for contrastive learning.  _arXiv preprint arXiv:2005.10243_ (2020).
  * Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021.  Training data-efficient image transformers & distillation through attention. In _International Conference on Machine Learning_. PMLR, 10347–10357.
  * Tsipras et al. (2018) Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2018\.  Robustness may be at odds with accuracy.  _arXiv preprint arXiv:1805.12152_ (2018).
  * Wang et al. (2022b) Xiao Wang, Yuhang Huang, Dan Zeng, and Guo-Jun Qi. 2022b.  CaCo: Both Positive and Negative Samples are Directly Learnable via Cooperative-adversarial Contrastive Learning.  _arXiv preprint arXiv:2203.14370_ (2022).
  * Wang et al. (2022a) Ying Wang, Chiuman Ho, Wenju Xu, Ziwei Xuan, Xudong Liu, and Guo-Jun Qi. 2022a.  Dual-flattening transformers through decomposed row and column queries for semantic segmentation.  _arXiv preprint arXiv:2201.09139_ (2022).
  * Wei et al. (2022) Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. 2022.  Masked feature prediction for self-supervised visual pre-training. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 14668–14678.
  * Wu et al. (2021) Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. 2021.  Rethinking and improving relative position encoding for vision transformer. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 10033–10041.
  * Wu et al. (2018) Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018\.  Unsupervised feature learning via non-parametric instance discrimination. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 3733–3742.
  * Xie et al. (2022) Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. 2022\.  Simmim: A simple framework for masked image modeling. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 9653–9663.
  * Xu et al. (2017) Weilin Xu, David Evans, and Yanjun Qi. 2017.  Feature squeezing: Detecting adversarial examples in deep neural networks.  _arXiv preprint arXiv:1704.01155_ (2017).
  * Yang et al. (2020) Greg Yang, Tony Duan, J Edward Hu, Hadi Salman, Ilya Razenshteyn, and Jerry Li. 2020\.  Randomized smoothing of all shapes and sizes. In _International Conference on Machine Learning_. PMLR, 10693–10705.
  * Yu et al. (2022) Qiying Yu, Jieming Lou, Xianyuan Zhan, Qizhang Li, Wangmeng Zuo, Yang Liu, and Jingjing Liu. 2022.  Adversarial Contrastive Learning via Asymmetric InfoNCE.  _arXiv preprint arXiv:2207.08374_ (2022).
  * Zhang et al. (2019b) Dinghuai Zhang, Mao Ye, Chengyue Gong, Zhanxing Zhu, and Qiang Liu. 2019b.  Filling the soap bubbles: Efficient black-box adversarial certification with non-gaussian smoothing.  (2019).
  * Zhang et al. (2019c) Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. 2019c.  Theoretically principled trade-off between robustness and accuracy. In _International conference on machine learning_. PMLR, 7472–7482.
  * Zhang and Qi (2020) Liheng Zhang and Guo-Jun Qi. 2020.  Wcp: Worst-case perturbations for semi-supervised deep learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3912–3921.
  * Zhang et al. (2019a) Liheng Zhang, Guo-Jun Qi, Liqiang Wang, and Jiebo Luo. 2019a.  Aet vs. aed: Unsupervised representation learning by auto-encoding transformations rather than data. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 2547–2555.
  * Zhou et al. (2021) Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. 2021.  ibot: Image bert pre-training with online tokenizer.  _arXiv preprint arXiv:2111.07832_ (2021).