# CaCo: Both Positive and Negative Samples are Directly Learnable via
Cooperative-adversarial Contrastive Learning

Xiao Wang, , Yuhang Huang, Dan Zeng and Guo-Jun Qi X. Wang was with Department
of Computer Science, Purdue University,West Lafayette, 47906, USA. Y. Huang
and D. Zeng were with the School of Communication and Information Engineering,
Shanghai University, Shanghai, 200444, China. G-J. Qi was with the Futurewei
Seattle Cloud Lab, Seattle, WA, 98006, USA. Email<EMAIL_ADDRESS>Manuscript received March, 2022; revised xx,xx; accepted xx,xx.(Corrresponding
author: Guo-Jun Qi)

###### Abstract

As a representative self-supervised method, contrastive learning has achieved
great successes in unsupervised training of representations. It trains an
encoder by distinguishing positive samples from negative ones given query
anchors. These positive and negative samples play critical roles in defining
the objective to learn the discriminative encoder, avoiding it from learning
trivial features. While existing methods heuristically choose these samples,
we present a principled method where both positive and negative samples are
directly learnable end-to-end with the encoder. We show that the positive and
negative samples can be cooperatively and adversarially learned by minimizing
and maximizing the contrastive loss, respectively. This yields cooperative
positives and adversarial negatives with respect to the encoder, which are
updated to continuously track the learned representation of the query anchors
over mini-batches. The proposed method achieves $71.3\%$ and $75.3\%$ in top-1
accuracy respectively over $200$ and $800$ epochs of pre-training ResNet-50
backbone on ImageNet1K without tricks such as multi-crop or stronger
augmentations. With Multi-Crop, it can be further boosted into $75.7\%$. The
source code and pre-trained model are released in https://github.com/maple-
research-lab/caco.

###### Index Terms:

Contrastive Learning, Cooperative-Adversarial Learning, Self-Supervised
Learning, Positive+Negative Samples

##  I Introduction

Unsupervised representation learning [1, 2, 3] has attracted tremendous
attentions recently as model pre-training becomes an essential step for deep
neural networks [4]. Among them are a family of instance discrimination-based
methods [5, 6, 7, 8], which train the network by distinguishing positive
samples from their negative counterparts given query anchors from mini-batches
during the learning process. Contrastive learning [1] and many variants [9,
10] are one of the most popular directions that achieved great success in last
3 years by pulling the embeddings of positive pairs together and pushing that
of negative pairs away. In these methods, a critical question arises regarding
how to find the most critical positives and negatives that can provide
discriminative knowledge to self-supervise the network training.

SimCLR [6] and MoCo [7, 11] are two representative contrastive learning
methods in this category. Given a query anchor, SimCLR uses the other samples
from the same mini-batch as negative examples while an augmented view of the
anchor is used as the positive sample. On the contrary, the MoCo maintains a
memory bank as in [5], and uses the negative samples from the memory bank. The
memory bank is built by queueing the learned representations over past mini-
batches in a FIFO (First-In-First-Out) fashion, and a momentum update is
applied to the key network whose weights are an exponential moving average of
the query networks over time. SimCLR [6] usually needs a very large batch size
to involve sufficient negative samples in each iteration, making it very
computationally and memory demanding. Although the MoCo also has a very large
memory bank, it can be maintained efficiently as the negative samples in it
are collected on-the-fly from the past mini-batches.

Both methods show that successful training strongly relies on the construction
of negative samples. Usually, harder negatives that are closer and mixed with
the query anchors contain more discriminative information, making it harder to
learn shallow features providing shortcut solutions to distinguish between
positives and negatives. Along this direction, recent works [12] use heuristic
sampling by concentrating on the hard negatives around queries.

On the contrary, a recent breakthrough demonstrates that negative samples are
directly learnable by treating them as a part of network weights through
adversarial contrastive learning [8], naturally resulting in hard negatives
pushed towards query anchors. It provides a more principled way to learn
rather than heuristically construct discriminative negatives. Specifically, by
treating the negatives and the encoder as two adversarial players, the AdCo
outperforms the SimCLR and MoCo in terms of both accuracy and efficiency.
However, a question still remains – can we learn positive samples as well just
like those negative samples? In this paper, we strive to answer it by
revealing the relation between positive and negative samples via a
cooperative-adversarial principle.

Unlike negative samples, we will show that given a query anchor, its positive
sample ought to be learned cooperatively rather than adversarially with the
encoder network. This is based on the observation that the representation of a
positive sample that cooperatively minimizes the contrastive loss with the
encoder will be pulled towards the given anchor, instead of being pushed away
if it were adversarially learned by maximizing the loss. The cooperative
positive sample will also be chosen from the memory bank shared with the
negative samples. This could mitigate false negative problem as memory bank
samples in close proximity of a query anchor can be trained as positives,
instead of being blindly treated as negatives. Both positive and negative
samples will be end-to-end trained together with the encoder through such a
Cooperative-adversarial Contrastive (CaCo) learning approach. Moreover, we
propose to use the most probable positive samples that are stable to update
for an end-to-end training of the CaCo model. Experiment results demonstrate
the proposed method achieves $71.3\%$ and $75.3\%$ in top-1 accuracy over 200
epochs and 800 epochs of pre-training on Imagenet1K without multi-crop
augmentations. It can achieve 75.7% in top-1 accuracy after multi-crop
augmentations with 800 epochs pre-training.

The remainder of the paper is organized as follows. We will review the related
works in Section II, and revisit the preliminary works on contrastive learning
in Section III. The proposed CaCo method is presented in Section IV, followed
by the experiments in Section V. We will conclude the paper in Section VI.

##  II Related Work

In this section, we will review the related works on contrastive learning from
three perspectives - instance discrimination-based contrastive learning, hard
negative samples, and trainable memory bank.

Instance Discrimination. Most of recent works on contrastive learning focus on
instance discrimination-based methods [5]. Given a query anchor from a
training minibatch, it explicitly constructs the positive and negative
samples, and self-trains the representation network (i.e., an encoder) by
distinguishing positives from negatives. It minimizes the InfoNCE loss (a form
of the contrastive loss) that can be implemented as a cross entropy by
treating each instance as a positive or a negative class. The positive sample
of a query anchor is often given as the augmented view of the same anchor.
Depending on different ways to form negatives, the contrastive learning has
many variants. Among them are the SimCLR [6] where negatives come from the
other samples of the same mini-batch, as well as the MoCo [7, 11] where exists
a memory bank storing the representations over past mini-batches as negatives.
The former usually relies on a large size of mini-batches to perform reliable
training of the encoder, which is demanding for computing and memory. MoCo
addresses this problem by reusing the obtained representation of past samples
without having to re-computing them for each mini-batches. While some negative
samples may belong to the same class as a query anchor in instance
discrimination methods (i.e, false negatives), a nearest neighbor contrastive
learning extends the SimCLR and the MoCo by instead assigning the negative
most similar to the anchor as its positive [13], which somehow mitigates the
false negative problem.

Hard Negative Samples. It has been observed that the harder negative samples
that are difficult to distinguish from their positive counterparts play a more
critical role in contrastive learning. They prevent the encoder from learning
trivial features to shortcut the aforementioned instance discrimination task.
There exist methods based on heuristically sampling the samples in proximity
of positive anchors. For example, hard negative sampling methods [14, 12]
apply importance sampling to find harder negatives with a concentration
parameter. Recent works [15, 16] on adversarial pre-training also shed some
light on learning representations resilient against adversarial attacks.
However, they do not directly train negative samples end-to-end by treating
memory bank as learnable network weights as in [8]. Instead, they apply the
adversarial training to input images or augmented views that may adversely
affect positive samples that ought to be constructed cooperatively as we find
in this paper.

Trainable Memory Bank. The recent work on Adversarial Contrastive learning
(AdCo) [8] made a significant progress to demonstrate that the hard negatives
are directly learnable as a part of network weights that are adversaries to
the encoder network in the other part. Instead of minimizing the InfoNCE loss
as the encoder, it maximizes it that pushes the learned negatives towards
their query anchors. It provides a principled way to finding more
discriminative negatives for an effective training of the encoder. Along this
line, an implicit feature modification (IFM) [17] is proposed to iteratively
update the hard samples. The IFM also pushes the samples towards the
representation of anchors like the AdCo, but its step size is set by a given
level of budget. Instead, the iterative update direction of these samples in
the AdCo is derived from the positive gradient of the InfoNCE loss as an
adversary learner, and its step size is proportional to the probability of
being a positive to a query anchor. In this way, the AdCo can produce more
discriminative negatives by making it harder to distinguish them from the
positive anchors.

In this paper, along the road outlined from instance discrimination and
trainable memory bank, we will unify the joint training of both positive and
negative samples given query anchors. Both positives and negatives will share
a memory bank that is trained end-to-end. For this purpose, it needs to answer
two critical questions in contrastive learning:

  * •

Can we end-to-end train positive and negative samples jointly through a shared
memory bank?

  * •

If so, will adversarial criterion still work for training positives?

The answer to these questions leads to the proposed Cooperative-adversarial
Contrastive (CaCo) learning presented in this paper.

##  III Preliminaries and Notations

In this section, we revisit the contrastive learning and its adversarial
training of memory bank samples. We will also define several notations that
will be used later.

###  III-A Contrastive Learning

Consider an unlabeled example $\mathbf{x}$ called a query anchor in the
current training mini-batch and its representation $\mathbf{z}$ output from an
encoder network $\mathbf{z}\triangleq\phi_{\theta}(\mathbf{x})$ parameterized
by $\theta$. To train the encoder $\phi_{\theta}$, a positive sample
$\mathbf{x}^{+}$ of $\mathbf{x}$ is obtained by applying an augmentation
(e.g., random crop, color jitting and Gaussian noises) to $\mathbf{x}$. The
resultant $\mathbf{x}^{+}$ provides an alternate view of the original anchor
$\mathbf{x}$, along with its representation $\mathbf{z}^{+}$. Usually, in a
typical contrastive learning approach, the anchor $\mathbf{x}$ is also given
by applying a random augmentation to the original image.

In instance discrimination-based contrastive learning such as MoCo [7] and
AdCo [8], together with mini-batches over iterations, a memory bank
$\mathcal{M}\triangleq\\{\mathbf{b}_{j}|j=1,\cdots,K\\}$ of $K$
representations are also provided as negatives, in contrast to the positive
representation $\mathbf{z}^{+}$ for an anchor. The memory bank is shared
across all unlabeled anchors in a minibatch. Such a bank can be constructed by
queueing the obtained representations from the past minibatches in a FIFO
(First-In-First-Out) fashion such as in MoCo [7]. Alternatively, a more
principled way is to directly learn the memory bank by maximizing the
contrastive loss (i.e., InfoNCE loss), which has demonstrated more efficient
to update the encoder [8]. In both cases, the encoder parameters $\theta$ are
trained by

$\theta^{\star}=\arg\min_{\theta}\sum_{\mathbf{x}\in\mathcal{B}}\ell(\mathbf{x})$

over a mini-batch $\mathcal{B}$ with the following InfoNCE loss [1, 5]

$\ell(\mathbf{x})=-\log\dfrac{\exp(\mathbf{z}\cdot\mathbf{z}^{+}/\tau)}{\exp(\mathbf{z}\cdot\mathbf{z}^{+}/\tau)+\sum_{j=1}^{K}{\exp(\mathbf{z}\cdot\mathbf{b}_{j}/\tau)}}$
(1)

where $\cdot$ denotes the inner product between two feature vectors and $\tau$
is the temperature controlling the sharpness of the loss. SimCLR [11] also
symmetrizes the loss by switching $\mathbf{z}$ and $\mathbf{z}^{+}$ with the
latter as a query anchor instead. Since the representations are normalized to
a unit norm, the inner product gives the cosine similarity between samples.

###  III-B Adversarial Contrastive Learning

Constructing the memory bank [5, 18, 11] plays a critical role in contrastive
learning. The negative samples in it gives the important specification about
how the underlying representation network $\phi_{\theta}$ ought to be learned
by discriminating between positives and negatives. Previous results show that
hard negatives could provide more useful information, since hard negative
samples can prevent the encoder from learning trivial representations focusing
on low-level features such as local details rather than high-level semantic
structures such as object shapes and categories.

Furthermore, it has been shown that hard negative samples naturally result
from an adversarial contrastive learning (AdCo) approach [8], with a novel
idea of directly learning the negatives to form a memory bank. This is
contrary to the MoCo where the negative samples are collected by queueing the
representations from past minibatches heuristically. Formally, the AdCo
considers two adversary players – the memory bank $\mathcal{M}$ and the
representation encoder $\phi_{\theta}$. They are trained in a mutually
adversarial manner by

$\mathcal{M}^{\star},\theta^{\star}=\arg\max_{\mathbf{b}_{j}\in\mathcal{M}}\min_{\theta}\sum_{\mathbf{x}\in\mathcal{B}}\ell(\mathbf{x})$
(2)

The previous results [8] show that the memory bank and the encoder can be
updated alternately, resulting in more efficient representation learning since
the negative samples can be updated more efficiently to provide critical
information to specify the contrastive learning.

Although the AdCo [8] has made a significant progress to directly learn
negative samples, the question still remains regarding if the positive sample
$\mathbf{x}^{+}$ and its representation $\mathbf{z}^{+}$ for an unlabeled
anchor can also be learned directly. This will give rise to a more elegant
solution by unifying the end-to-end training of both positives and negatives.
Meanwhile, it could also mitigate the false negative problem in contrastive
learning, since it is known that some negative samples in the learned memory
bank would be positive to an anchor if their representations are sufficiently
close. This inspires us to develop the following Cooperative-adversarial
Contrastive (CaCo) learning method.

(a) MoCo

(b) AdCo

(c) CaCo

Figure 1: This figure compares existing contrastive learning methods – MoCo
and AdCo – with the proposed CaCo. Both MoCo and AdCo only has negative
samples from the memory bank, and MoCo queues these samples from the encoder
output over past minibatches without training them directly. In contrast,
given a query anchor $\mathbf{z}$, CaCo directly learns the cooperative
positive $\mathbf{b}_{j^{+}}$ and adversarial negatives $\mathbf{b}_{j},j\neq
j^{+}$ from the shared memory bank through backward gradients (${\rm grad}$).

Figure 1 gives a glance at the differences between MoCo [7, 11], AdCo [8] and
the proposed CaCo. Both MoCo and AdCo only has negative samples from the
memory bank, and MoCo queues these samples from the past mini-batches without
training them directly. In contrast, both positive and negative samples are
obtained from the shared memory bank given a query anchor $\mathbf{z}$, and
they are directly trained through backward gradients by minimizing and
maximizing the loss, respectively. The proposed CaCo will be presented in
detail below.

##  IV CaCo: Cooperative-adversarial Contrastive Learning

In this section, we will present the proposed Cooperative-adversarial
Contrastive (CaCo) Learning. First we will explicitly integrate positive
samples shared with negatives from the memory bank into the contrastive
learning in Section IV-A. Then in Section IV-B we will show that positives and
negatives ought to play cooperative and adversarial role in the network
training, respectively, followed by the elaboration on cooperative-adversarial
training over minibatches in Section IV-C.

###  IV-A Positives from Shared Memory Bank

Following the definition defined in the last section, consider the
representation $\mathbf{z}$ of an unlabeled anchor $\mathbf{x}$. In the AdCo,
we assume its negative samples all come from the trainable memory bank
$\mathcal{M}$. Now let us make a bold step by assuming the corresponding
positive representation also comes from $\mathcal{M}$. We denote the index of
the positive sample in $\mathcal{M}$ by $j^{+}\in\mathcal{K}\triangleq
1,\cdots,K$. Thus, for $\mathbf{x}$ with embedding $z$, its positive
representation is given by $\mathbf{b}_{j^{+}}\in\mathcal{M}$. Accordingly,
the contrastive InfoNCE loss becomes

$\displaystyle\ell(\mathbf{x})$
$\displaystyle\triangleq-\log\dfrac{\exp(\mathbf{z}\cdot\mathbf{b}_{j^{+}}/\tau)}{\exp(\mathbf{z}\cdot\mathbf{b}_{j^{+}}/\tau)+\sum_{j\neq
j^{+}}{\exp(\mathbf{z}\cdot\mathbf{b}_{j}/\tau)}}$ (3)
$\displaystyle=-\log\dfrac{\exp(\mathbf{z}\cdot\mathbf{b}_{j^{+}}/\tau)}{\sum_{j=1}^{K}{\exp(\mathbf{z}\cdot\mathbf{b}_{j}/\tau)}}$

We note that since both the positive and negative samples come from the shared
memory bank, all terms in the denominator can be written uniformly with the
same set of the memory bank representations, yielding a more elegant
contrastive loss.

###  IV-B Cooperative Positives vs. Adversarial
Negatives

An unlabeled anchor $\mathbf{x}$ has both positive and negative samples,
which, as aforementioned, we assume are all directly learnable. Following the
gradient decent method, we can derive the gradient of (3) over the positive
sample $\mathbf{b}_{j^{+}}$ and its negative counterparts $\mathbf{b}_{j}$ for
$j\neq j^{+}$ to update them.

It is not hard to obtain the following gradients

$\dfrac{\partial\ell(\mathbf{x})}{\partial\mathbf{b}_{j^{+}}}=-\dfrac{1}{\tau}\left[1-p(\mathbf{b}_{j^{+}}|\mathbf{z})\right]\mathbf{z}$
(4)

and

$\dfrac{\partial\ell(\mathbf{x})}{\partial\mathbf{b}_{j}}=\dfrac{1}{\tau}p(\mathbf{b}_{j}|\mathbf{z})\mathbf{z},~{}j\neq
j^{+}$

which are for the positive and negative samples, respectively. Here,
$p(\mathbf{b}_{j}|\mathbf{z})$ denotes the probability of a memory bank sample
$b_{j}$ being positive to the input representation $\mathbf{z}$,

$p(\mathbf{b}_{j}|\mathbf{z})=\dfrac{\exp(\mathbf{z}\cdot\mathbf{b}_{j}/\tau)}{\sum_{k=1}^{K}{\exp(\mathbf{z}\cdot\mathbf{b}_{k}/\tau)}}$
(5)

Based on the idea behind the adversarial training, the negative samples can
thus be updated by maximizing $\ell(\mathbf{x})$, resulting in

$\mathbf{b}_{j}\leftarrow\mathbf{b}_{j}+\dfrac{\eta}{\tau}p(\mathbf{b}_{j}|\mathbf{z})\mathbf{z},~{}j\neq
j^{+},$

where $\eta$ is the learning rate for the gradient update. This shows that
each negative representation $\mathbf{b}_{j}$ will be updated towards the
anchor representation $\mathbf{z}$, which will make it harder to distinguish
the negative sample from the anchor $\mathbf{z}$. The previous results show
that such harder negatives can provide more useful information to train the
encoder. In this sense, the negative samples are adversarial players to the
encoder. For the adversarial negative example training, AdCo [8] has carefully
discussed it and utilized it for contrastive learning successfully.

Now the question is: should we also make the positive sample an adversary to
the encoder? Let us take a look at the gradient of the positive representation
$\mathbf{b}^{+}$ in (4). If it were assume to be an adversary, it would be
updated by

$\mathbf{b}_{j^{+}}\leftarrow\mathbf{b}_{j^{+}}{\color[rgb]{1,0,0}\bm{-}}\dfrac{\eta}{\tau}\left[1-p(\mathbf{b}_{j^{+}}|\mathbf{z})\right]\mathbf{z}$

In this case, the positive $\mathbf{b}_{j^{+}}$ would be pushed away from the
anchor $\mathbf{z}$, instead of being pushed towards it. This causes a dilemma
– what we chose as a positive sample would eventually end up being pushed away
from the anchor, making it more likely to be a negative rather than a positive
sample. Obviously, this is not our intention to learn a positive sample.

This shows that the positive sample should be a cooperative player rather than
an adversary to the encoder by minimizing the contrastive loss. This results
in the following update to the positive sample

$\mathbf{b}_{j^{+}}\leftarrow\mathbf{b}_{j^{+}}{\color[rgb]{1,0,0}\bm{+}}\dfrac{\eta}{\tau}\left[1-p(\mathbf{b}_{j^{+}}|\mathbf{z})\right]\mathbf{z}$

which in turn pushes the positive representation towards the anchor, in line
with our intuition to learn the positive sample close to the anchor.

Figure 2: The diagram of CaCo. CaCo trains a visual representation encoder and
a cooperative-adversarial memory bank at the same time. For a encoded query
$q$, we used its counterpart key $k$ by the key encoder to identify its most
probable positive out of memory bank as its positive pair, while all other
embeddings in the memory bank are its negative pairs. Then the network is
optimized by minimizing the contrastive loss, the memory bank is coopertive-
adversarial optimized as discussed in the paper.

###  IV-C Cooperative-Adversarial Training

The above analysis yields the following Cooperative-Adversarial Contrastive
(CaCo) objective

$\sum_{\mathbf{z}\in\mathcal{B}}\min_{\mathbf{b}_{j^{+}},\theta}~{}\max_{\mathbf{b}_{j},j\neq
j^{+}}-\log\dfrac{\exp(\mathbf{z}\cdot\mathbf{b}_{j^{+}}/\tau)}{\sum_{j=1}^{K}{\exp(\mathbf{z}\cdot\mathbf{b}_{j}/\tau)}}$
(6)

where both the positive sample $\mathbf{b}_{j^{+}}$ and the encoder with the
parameters $\theta$ minimizes the loss, thus being cooperative players, while
the negative samples $\mathbf{b}_{j},j\neq j^{+}$ are adversary instead to
maximize it.

The above CaCo objective is defined over a mini-batch $\mathcal{B}$. One can
assign a distinct positive sample from the memory bank to each anchor
$\mathbf{z}$ in $\mathcal{B}$. In other words, the positive index $j^{+}$ is a
function of $\mathbf{z}$ in (6). Thus, for a memory bank sample
$\mathbf{b}_{j}$, we use $\mathcal{P}_{j}^{+}$ ($\mathcal{N}_{j}^{-}$ resp.)
$\subseteq\mathcal{B}$ to denote all anchors $\mathbf{z}$’s, for which
$\mathbf{b}_{j}$ is a positive (negative resp.) sample. For each
$\mathbf{b}_{j}$, we have
$\mathcal{P}_{j}^{+}\bigcap\mathcal{N}_{j}^{+}=\emptyset$ and
$\mathcal{P}_{j}^{+}\bigcup\mathcal{N}_{j}^{+}=\mathcal{B}$. Then, according
to (6), all memory bank samples can be updated by

$\mathbf{b}_{j}\leftarrow\mathbf{b}_{j}{\color[rgb]{0,0,1}\bm{-}}\sum_{\mathbf{z}\in\mathcal{P}_{j}^{+}}\eta\dfrac{\partial\ell(\mathbf{x})}{\partial
b_{j}^{+}}{\color[rgb]{1,0,0}\bm{+}}\sum_{\mathbf{z}\in\mathcal{N}_{j}^{-}}\eta\dfrac{\partial\ell(\mathbf{x})}{\partial
b_{j}}$ (7)

with a positive learning rate $\eta$, where the minus
“${\color[rgb]{0,0,1}\bm{-}}$” and the plus “${\color[rgb]{1,0,0}\bm{+}}$” in
front of the last two terms reflect the cooperative and the adversarial
updates to $\mathbf{b}_{j}$.

Figure 3: The surrogate images of the Most Probable Positive (MPP) and random
negatives from the learned memory bank for some query examples.

Effect of $\l_{2}$-normalization on memory bank updates. It is worth noting
that the gradient in (4) was derived by assuming both $\mathbf{z}$ and all
memory bank samples $\mathbf{b}_{j}$’s have already been normalized to a unit
length. Even though they are normalized, the gradient update will result in
new vectors not normalized to a unit length anymore. Thus, an
$l_{2}$-normalization is often required after the gradient update. Such a
normalization can be built into (3) by replacing $\mathbf{z}$ and
$\mathbf{b}_{j}$ with $\mathbf{z}/\|\mathbf{z}\|_{2}$ and
$\mathbf{b}_{j}/\|\mathbf{b}_{j}\|_{2}$, respectively. This results in the
following gradient

$\dfrac{\partial\ell(\mathbf{x})}{\partial\mathbf{b}_{j^{+}}}=-\dfrac{1}{\tau}\left[1-p(\mathbf{b}_{j^{+}}|\mathbf{z})\right]\dfrac{\|\mathbf{b}_{j^{+}}\|^{2}_{2}\mathbf{z}-\mathbf{z}\cdot\mathbf{b}_{j^{+}}\mathbf{b}_{j^{+}}}{\|\mathbf{z}\|_{2}\|\mathbf{b}_{j^{+}}\|^{3}_{2}}$

This shows how to update the positive sample end-to-end by viewing its
representation $\mathbf{b}_{j^{+}}$ as free network weights. For simplicity,
one can assume every time after a mini-batch update, $\mathbf{b}_{j^{+}}$ and
$\mathbf{z}$ are normalized back to a unit length such that we always keep the
current samples $\|\mathbf{b}_{j^{+}}\|_{2}=1$ and $\|\mathbf{z}\|_{2}=1$. In
this case, the above gradient can be simplified to

$\displaystyle\dfrac{\partial\ell(\mathbf{x})}{\partial\mathbf{b}_{j^{+}}}$
$\displaystyle=-\dfrac{1}{\tau}\left[1-p(\mathbf{b}_{j^{+}}|\mathbf{z})\right](\mathbf{z}-\mathbf{z}\cdot\mathbf{b}_{j^{+}}\mathbf{b}_{j^{+}})$
(8)
$\displaystyle=-\dfrac{1}{\tau}\left[1-p(\mathbf{b}_{j^{+}}|\mathbf{z})\right](\mathbf{I}-\mathbf{b}_{j^{+}}\mathbf{b}_{j^{+}}^{T})\mathbf{z}$

Similarly, we can derive the gradient over negative samples by assuming all
$\mathbf{b}_{j}$’s are normalized to a unit norm after each gradient update

$\displaystyle\dfrac{\partial\ell(\mathbf{x})}{\partial\mathbf{b}_{j}}$
$\displaystyle=\dfrac{1}{\tau}p(\mathbf{b}_{j}|\mathbf{z})(\mathbf{I}-\mathbf{b}_{j}\mathbf{b}_{j}^{T})\mathbf{z},j\neq
j^{+}$ (9)

From (8) and (9), we can see that both positive and negative samples will be
updated towards the transformed direction $\mathbf{T}_{j}\mathbf{z}$ with
$\mathbf{T}_{j}=\mathbf{I}-\mathbf{b}_{j}\mathbf{b}_{j}^{T}$. It is easy to
verify that

  * •

The gradient $\mathbf{T}_{j}\mathbf{z}$ is orthogonal to $\mathbf{b}_{j}$,
i.e., $\mathbf{b}_{j}^{T}\mathbf{T}_{j}\mathbf{z}=0$. It suggests that the
gradient is tangent to the unit-hypersphere such that it tends to keep the
unit norm of $\mathbf{b}_{j}$ unchanged as expected.

  * •

Along the gradient, the sample $\mathbf{b}_{j}$ also tends to be pushed closer
towards $\mathbf{z}$ since their cosine similarity
$\mathbf{z}^{T}\mathbf{T}_{j}\mathbf{z}=1-(\mathbf{z}\cdot\mathbf{b}_{j})^{2}\geq
0$ is nonnegative.

Method | batch size | top-1
---|---|---
InstDisc [5] | 256 | 58.5
SimCLR [6] | 256 | 61.9
CPC v2 [9] | 512 | 63.8
PCL v2 [19] | 256 | 67.6
MoCo v2 [11] | 256 | 67.5
PIC [20] | 512 | 67.6
InfoMin Aug [21] | 256 | 70.1
SimSiam [22] | 256 | 70.0
SwAV [23] | 4096 | 69.1
NNCLR [13] | 4096 | 70.7
BYOL [24] | 4096 | 70.6
AdCo [8] | 256 | 68.2
CaCo | 256 | 70.9
CaCo | 1024 | 71.3
CaCo | 4096 | 72.0
TABLE I: Top-1 accuracy under the linear evaluation on Imagenet1K dataset with
the ResNet-50 backbone. All compared methods use single-crop augmentations
pre-trained over $200$ epochs.

Now, we can rewrite the gradient update (7) to all samples $j=1,\cdots,K$ in
the memory bank as

$\displaystyle\mathbf{b}_{j}\leftarrow\mathbf{b}_{j}$
$\displaystyle\underbrace{+\sum_{\mathbf{z}\in\mathcal{P}_{j}^{+}}\dfrac{\eta}{\tau}\left[1-p(\mathbf{b}_{j}|\mathbf{z})\right]\mathbf{T}_{j}\mathbf{z}}_{{\rm
cooperative~{}~{}positives}}$ (10)
$\displaystyle\underbrace{+\sum_{\mathbf{z}\in\mathcal{N}_{j}^{-}}\dfrac{\eta}{\tau}p(\mathbf{b}_{j}|\mathbf{z})\mathbf{T}_{j}\mathbf{z}}_{{\rm
adversarial~{}~{}negatives}}.$

From this update, we observe that

  * •

For cooperative positives, the more probable $\mathbf{b}_{j}$ is positive to
$\mathbf{z}$ with a higher $p(\mathbf{b}_{j}|\mathbf{z})$, the less it will be
pushed towards $\mathbf{z}$. This is intuitive as a positive sample already
closer to its anchor $\mathbf{z}$ does not need to be updated too much;

  * •

For adversarial negatives, we have an opposite observation – the more probable
$\mathbf{b}_{j}$ is positive to $\mathbf{z}$, the more it should be pushed
towards the anchor. In this way, it will become a harder negative to be
distinguished from the positive counterparts, thereby providing more
discriminative information to update the encoder.

Method | epochs | batch size | top-1
---|---|---|---
SeLa [25] | 400 | 256 | 61.5
PIRL [26] | 800 | 1024 | 63.6
CMC [10] | 240 | 128 | 66.2
SimCLR [6] | 800 | 4096 | 69.3
PIC [20] | 1600 | 512 | 70.8
MoCo v2 [11] | 800 | 256 | 71.1
InfoMin Aug [21] | 800 | 256 | 73.0
SimSiam [22] | 800 | 256 | 71.3
SwAV [23] | 800 | 4096 | 71.8
NNCLR [13] | 800 | 1024 | 72.9
BYOL [24] | 1000 | 4096 | 74.3
AdCo [8] | 800 | 256 | 72.8
MoCo v3 [27] | 1000 | 4096 | 74.6
NNCLR [13] | 800 | 4096 | 74.9
CaCo | 800 | 1024 | 74.1
CaCo | 800 | 4096 | 75.3
TABLE II: Top-1 accuracy under the linear evaluation on Imagenet1K with the
ResNet-50 backbone. The table compares the methods using a single crop
augmentation pre-trained with more epochs.

The Most Probable Positive. The last question we need to answer is how we
choose the positive sample given an anchor. Here, we propose to use the Most
Probable Positive (MPP) sample resulting in the minimum change that favors a
stable update for the training purpose.

Specifically, once a positive sample $\mathbf{b}_{j^{+}}$ is chosen for an
anchor $\mathbf{z}$, its representation will be updated along the (negative)
gradient based on (8). Ideally, we expect that the updated sample will still
be positive to the anchor. This ensures the positive assignments in
$\mathcal{P}^{+}_{j}$ (and thus $\mathcal{N}^{-}_{j}$ as well) be as stable as
possible so that they can be updated stably based on (10).

For this reason, we note that $[1-p(\mathbf{b}_{j}|\mathbf{z})]$ is the step
size (up to a constant coefficient $\frac{1}{\tau}$) to update the positive
sample $\mathbf{b}_{j^{+}}$ in (8). To minimize the change to the positive
sample, we choose the positive sample that minimizes this step size,

$j^{+}=\arg\min_{j}1-p(\mathbf{b}_{j}|\mathbf{z})=\arg\max_{j}p(\mathbf{b}_{j}|\mathbf{z}),$
(11)

which results in the Most Probable Positive (MPP) sample to the anchor
$\mathbf{z}$. Intuitively, the chosen positive $\mathbf{b}_{j^{+}}$ will be
the one that minimizes the error in assigning a false positive sample to
$\mathbf{z}$ based on $p(\mathbf{b}_{j}|\mathbf{z})$ in (5).

To address the most probable positive clearly, the diagram of the training
process is illustrated in Fig. 2. For a fair comparison with the SOTA methods
such as MoCo [7] and AdCo [8], the CaCo also has two streams composed of a
momentum encoder and a query encoder, where the momentum encoder is used to
identify the positives in the memory bank, and the contrastive loss will be
propagated to query encoder and memory bank to train them. All positive and
negative samples are treated as free parameters like the network weights, and
they are directly learned through gradient descent/ascent by back-propagating
the contrastive loss in the proposed cooperative-adversarialfashion, as shown
in Eqn. (10).

Avoidance of collapse. To explain the avoidance of collapse, it is worth
noting that both positive and negative samples share the memory bank across
query anchors in a mini-batch, and they are dynamically updated over
iterations in the proposed cooperative-adversarial training. In other words, a
memory bank sample $b$ can be positive for one query anchor, while being
negative for another one, which keeps the memory bank from collapsing to a
single trivial point. Memory bank samples will also be updated over
iterations, and different positives will be assigned to the feature
representation $z$ of a sample, keeping the latter from collapsing to a
prefixed point $b^{+}$.

##  V Experiments

In this section, we evaluate the proposed CaCo and compare the results with
the other unsupervised models including the state-of-the-art contrastive
learning methods.

Method | epochs | batch size | top-1
---|---|---|---
SwAV [23] | 800 | 4096 | 75.3
DINO [28] | 800 | 4096 | 75.3
NNCLR [13] | 800 | 4096 | 75.6
CaCo | 800 | 2048 | 75.7
TABLE III: Top-1 accuracy under the linear evaluation on Imagenet1K with the ResNet-50 backbone. The table compares the methods using same multi-crop augmentation in SwAV [23] pre-trained with more epochs. Method | Epoch | Time(h) | GPU |  | GPU Time
---
/epoch
MoCo v2[11] | 200 | 53.0 | 8$\times$V100 | 2.12
BYOL[24] | 1000 | 8.0 | 512$\times$TPU | 4.10
SWAV*[23] | 800 | 50 | 64$\times$V100 | 4.06
AdCo [8] | 200 | 56.5 | 8$\times$V100 | 2.26
CaCo | 200 | 56 | 8$\times$V100 | 2.24
TABLE IV: Running time comparison of different methods. GPU time/Epoch means
sum of time cost of all GPUs to train an epoch. Here the time are measured
under the asymmetrical settings to have fair comparison with early methods

Method | SUN397 | Cars | Food101 | Aircraft | VOC2007 | DTD | Pets | Caltech-101 | Flowers | CIFAR10 | CIFAR100
---|---|---|---|---|---|---|---|---|---|---|---
SimCLR [6] | 60.6 | 49.3 | 72.8 | 49.8 | 81.4 | 75.7 | 84.6 | 89.3 | 92.6 | 90.5 | 74.4
MoCo v2 † [11] | 61.1 | 67.0 | 71.5 | 50.9 | 87.1 | 74.5 | 87.6 | 91.4 | 90.0 | 90.9 | 74.8
AdCo ‡ [8] | 62.2 | 66.8 | 73.8 | 63.6 | 92.4 | 75.5 | 86.8 | 91.8 | 93.7 | 91.2 | 74.3
BYOL [24] | 62.2 | 67.8 | 75.3 | 60.6 | 82.5 | 75.5 | 90.4 | 94.2 | 96.1 | 91.3 | 78.4
NNCLR [13] | 62.5 | 67.1 | 76.7 | 64.1 | 83.0 | 75.5 | 91.8 | 91.3 | 95.1 | 93.7 | 79.0
CaCo | 64.6 | 73.8 | 75.4 | 66.0 | 92.6 | 76.8 | 91.9 | 94.4 | 96.1 | 92.6 | 76.0

TABLE V:  Transfer learning results on various datasets with ResNet-50
pretrained with Imagenet1K over $800$ epochs. The results are obtained with †
https://github.com/facebookresearch/moco under the CC-BY-NC 4.0 license and ‡
https://github.com/maple-research-lab/AdCo/ under MIT license. Other results
were reported directly in the other papers.

###  V-A Implementation Details

Figure 4: Plot of Mean Maximum Positive Probability and NN accuracy (based on
$20\%$ training set) over $200$ epochs of pretraining on Imagenet1K.

For a fair comparison with the existing models [7, 6], we adopt the ResNet-50
as the backbone for unsupervised pre-training on ImageNet. The output feature
map from the top ResNet-50 block is average-pooled and projects to a 256-D
feature vector through three MLP layers with two 2048-D hidden layers and the
ReLU [6]. The resultant vector is $\ell_{2}$ normalized to calculate the
cosine similarity. We adopt the single-crop protocol used in [6, 7, 11] to
augment data in single-crop experiments and we used multi-crop augmentations
used in literature [23] to apply multi-crop in final experiments that can
further boost performances.

We adopt up to $4,096$ images in a mini-batch for pre-training across up to
four 8-Nvidia Tesla V100 GPU servers for single-crop. For the multi-crop
setting, based on the same number of four 8-Nvidia Tesla V100 GPU servers, we
adopt a batch size of $2,048$ images for pre-training. We will show that the
proposed CaCo could also achieve competitive results with smaller batch sizes.
By convention, the number of negative adversaries is set to $65,536$ to make a
fair comparison with the other methods [7, 8].

For the network pre-training, we adopt an initial learning rate of
$0.03*\rm{batch\\_size}/256$ and $0.3*\rm{batch\\_size}/256$ in the SGD [29]
and LARS optimizer [30] for updating the backbone network for smaller
($<1024$) and larger ($\geq 1024$) batch sizes, respectively, along with a
weight decay of $10^{-4}$ and a momentum of $0.9$. The memory bank is always
directly optimized by SGD optimizer with learning rate $3.0$ without weight
decay and a momentum of $0.9$. The cosine scheduler [31] is used to gradually
decay the learning rates. We set the same temperate $\tau$ to $0.08$ when
updating the memory bank and network backbone. After the backbone network is
initialized, its output feature vectors over randomly drawn training images
are used to initialize the memory bank. After that, the encoder and the shared
memory bank with both positives and negatives are alternately updated.

Method | Symmetric Loss | Batch Size | #Negative Samples | #Params. | Top-1 |  | (GPU $\cdot$ Time)
---
/epoch
SimCLR [6] | ✓ | 8192 | - | - | 67.0 | 1.92
MoCo v2 [11] | ✓ | 256 | 65536 | 8M | 70.2 | 3.34
BYOL [24] | ✓ | 4096 | - | 1M | 70.6 | 4.10
SimSiam [22] | ✓ | 256 | - | 2M | 70.0 | -
AdCo [8] | ✓ | 256 | 65536 | 8M | 70.5 | 3.50
AdCo [8] | ✓ | 256 | 16384 | 2M | 70.0 | 3.46
AdCo [8] | ✓ | 256 | 8192 | 1M | 70.2 | 3.45
CaCo | ✓ | 4096 | 37268 | 4M | 71.7 | 3.83
CaCo | ✓ | 4096 | 65536 | 8M | 72.0 | 3.84
CaCo | ✓ | 4096 | 98304 | 12M | 72.1 | 3.86
TABLE VI: Top-1 accuracy under the linear evaluation on ImageNet with the
ResNet-50 backbone. The table compares the methods over 200 epochs of
pretraining. # Parameters: the parameter counts of negative samples, as to
BYOL and SimSiam, it means the parameter counts of the predictor.

###  V-B Results on Imagenet1K

First, we perform a linear evaluation by fine-tuning a fully connected
classifier for $100$ epochs on top of the frozen 2048-D feature vector from
the ResNet-50 backbone. Table I reports the results on Imagenet1K after $200$
epochs of pre-training. With different batch sizes of $256$, $1024$ and
$4,096$ for network pre-training, CaCo even outperforms the state-of-the-art
SWAV, BYOL and NNCLR with a much larger batch size of $4096$ when we used
small batch-size. It is well known that the performances of these SOTA models
improve with larger batch sizes [23, 24, 13]. However, larger batch sizes
require multiple GPU servers with at least $32$ Nvidia V100 GPU cards to
accommodate $4096$ images. This makes the network pre-training unaffordable
and inconvenient to set up across multiple servers for most of research teams.
The proposed CaCo not only outperforms SOTA methods with much smaller batch
sizes, but also allows affordable network pre-training on a single GPU server.
Our training time is on par with AdCo [8], MoCo [18] as shown in Table IV

For single-crop settings, we also report the results with more than $200$ pre-
training epochs in Table II. Here we run the CaCo with a batch size of $1,024$
and $4,096$ images to compare with the other methods. Here $1,024$ is the
largest size that can fit the GPU memory of a single $8\times V100$ server. It
still outperforms the SOTA models with various batch sizes and epochs. This
makes the CaCo affordable and competitive for network pre-training. It is
worth noting that here we do not apply any tricks such as multi-crop
augmentations [23, 32] and stronger augmentations [33] that have shown
effective in improving the accuracy in unsupervised learning literature. This
shows the CaCo is an elegant method easier to implement without relying on
those tricks.

To compare with the methods using multi-crop augmentations [23], we have
conducted the experiments following the same protocol used in SwAV [23]. The
results are shown in Table. III. We outperformed the other methods in this
setting even with a smaller batch size of $2,048$ based on the same number of
four 8-Nvidia Tesla V100 GPU servers. We believe the performance $75.7\%$ can
be further boosted if we can increase the batch size to $4,096$ as used in
other methods. However, this requires doubling computing resources, which
makes it unaffordable for many research groups.

###  V-C Transfer Learning Results

We also perform transfer learning tasks on various datasets111The link to
Birdsnap dataset is broken or refers to a dataset with missing data, making it
difficult to make a direct comparison.. The ResNet50 backbone was pretrained
on Imagenet1K over $800$ epochs with a batch size of $4096$. By following the
linear evaluation protocol in [24], a fully connected layer of classifier is
trained upon the frozen backbone.

Table V reports the results under the linear evaluation compared with the
other models. The results show the CaCo performs the best on $8$ out of these
$11$ datasets among the compared self-supervised methods. This demonstrates
the CaCo representation pre-trained on the Imagenet1K dataset can well
generalize to downstream tasks on a variety of datasets.

###  V-D Ablation Studies

In this section, we present a thorough analysis of CaCo. Here we focused on
two most important factors of CaCo: the size of the memory bank (the sum of
the positives and negatives) and the contribution of positives and negatives
to the performance.

Size of the memory banks. We set up the experiments with batch-size 4096 with
200 epoch pre-training, which indicates the number of positive is always 4096.
We only change the memory bank size across different experiments. As shown in
Table VI, we changed the memory size from 4M to 12M and observed the clear
improvement of performance from $71.7\%$ to $72.1\%$. That’s to say, CaCo can
be benefited by the size increase of memory bank. This is reasonable since
bigger memory bank size has two benefits: 1) more accurate positives by
checking the most probable positive from a bigger memory bank; 2) the task
become more challenge since encoder needs to select the correct positive out
of a bigger pool including more negative pairs.

Contribution of positives and negatives. We conducted an ablation study to
compare the contribution of positives and negatives. Here all experiments are
based on a batch size of 1024 with 200 epoch pre-training. Here None setting
is without memory bank, while both positive and negatives directly coming from
the concurrent mini-batch. Here Positive setting is similar to that NNCLR
[13], where we keep a queue to save the embeddings of previous batches and we
select the nearest neighbor of query in memory bank as positive while
negatives are from the concurrent mini-batch. Here Negative setting is similar
to AdCo [8], where the memory bank is adversarially trained and serves as the
negatives, and the positives are from the current mini-batch. Here P+N setting
is our CaCo setting, where both positives and negatives are from the memory
bank and memory bank is collaboratively and adversarially trained by the
contrastive loss in an end-to-end fashion. We keep all other settings to the
same to have a fair comparison. The results are shown in Table VII. It’s clear
that CaCo clearly improved compared to using positive or negative alone.

(a) CIFAR10

(b) PETS

(c) VOC2017

Figure 5: t-SNE visualization of learned representations on CIFAR10, PETS and VOC2017 datasets, where different colors represent different classes on these datasets. The encoder is pre-trained by CaCo over 800 epochs on Imagenet1K.  Setting | Batch Size | #Positive Pool | #Negative Pool | Top-1
---|---|---|---|---
None | 1024 | 1024 | 1024 | 70.2
Positive | 1024 | 65536 | 1024 | 70.7
Negative | 1024 | 1024 | 65536 | 70.6
Positive+Negative | 1024 | 65536 | 65536 | 71.3
TABLE VII: Top-1 accuracy under the linear evaluation on ImageNet with the
ResNet-50 backbone. The table compares different settings over 200 epochs of
pretraining. Here None setting denotes both positive and negative are from
current mini-batch; Positive setting denotes the positive embedding is from
memory bank and the negative embedding is from the current mini-batch;
Negative setting suggests the positive embedding is from current mini-batch
and the negative embedding is from the memory bank; P+N is our CaCo setting,
where both positive and negative are from the memory bank. Here the #Positive
Pool and #Negative Pool suggests the total number of embeddings that we
selected from to serve as positive and negatives, respectively.

###  V-E Result Analysis

####  V-E1 Quality of Learned Positives and Negatives

The learned positives and negatives in memory bank are “virtual” samples that
do not necessarily correspond to the features of some known images. To
understand a virtual sample, one can find its most similar image from Imagenet
in the feature space and use it as the surrogate.

We randomly choose some images as queries, and show the surrogates of the most
probable positive samples (cf. Eq. (11)) as well as the negative samples from
the memory bank in Figure 3. We can see that the positives (negatives) belong
to the same (different) class of query images. This demonstrates that the
queries successfully learn the true positives and true negatives from the
memory bank.

Moreover, given a query, the maximum probability corresponding to the most
probable positive in (11) measures the likelihood of obtaining a true positive
by the model. We plot the curve of the mean maximum positive probability over
epochs in Figure 4, along with the nearest neighbor (NN) accuracy based on
$20\%$ Imagenet1K training set. We plot the NN accuracy to measure the
performance of the pre-trained representation as it can be computed in a few
seconds.

We can see that at the beginning, the probability is close to zero as randomly
choosing a positive sample from the memory bank of $65,536$ samples is as low
as $1.5\times 10^{-5}$. After $200$ epochs of pretraining, the probability
increase significantly by $10^{3}$ times to above $1.2\times 10^{-2}$. The
increase in the mean maximum positive probability is consistent with the
increase in the NN accuracy. It suggests that high-quality positives and
negatives are eventually learned, leading to better representations learned
over epochs, and this partly explains the success of the CaCo method.

####  V-E2 Visualization of Representations

In Figure 5, we also use t-SNE to visualize the feature embeddings on CIFAR10,
PETS and VOC2017 datasets. The encoder is learned by CaCo over 800 epochs on
Imagenet1k dataset without labels. It shows that the feature embeddings
obtained via t-SNE exhibit distributional patterns well aligned with the
classes of various colors on these datasets. This suggests the learned
unsupervised features on one dataset well generalize to downstream datasets
without access to their labels.

##  VI Conclusions

This paper presents Cooperative-adversarial Contrastive (CaCo) learning for
network pre-training. It directly learns both positives and negatives over a
shared memory bank as cooperators and adversaries to an encoder by minimizing
and maximizing the underlying contrastive loss, respectively. This results in
an end-to-end training of positives and negatives iteratively updated towards
query anchors along the directions tangent to the unit hypersphere. Jointly
trained with the encoder, unsupervised features are learned to distinguish the
learned adversarial negatives from the most probable cooperative positive
given each query anchor. Experiments demonstrate that the CaCo outperforms the
SOTA self-supervised methods on multiple downstream tasks without relying on
multi-crop augmentations.

## Acknowledgment

## References

  * [1] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [2] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio, “Learning deep representations by mutual information estimation and maximization,” _arXiv preprint arXiv:1808.06670_ , 2018.
  * [3] L. Zhang, G.-J. Qi, L. Wang, and J. Luo, “Aet vs. aed: Unsupervised representation learning by auto-encoding transformations rather than data,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2547–2555.
  * [4] G.-J. Qi and J. Luo, “Small data challenges in big data era: A survey of recent progress on unsupervised and semi-supervised methods,” _arXiv preprint arXiv:1903.11260_ , 2019.
  * [5] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3733–3742.
  * [6] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” _arXiv preprint arXiv:2002.05709_ , 2020.
  * [7] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” _arXiv preprint arXiv:1911.05722_ , 2019.
  * [8] Q. Hu, X. Wang, W. Hu, and G.-J. Qi, “Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1074–1083.
  * [9] O. J. Hénaff, A. Srinivas, J. De Fauw, A. Razavi, C. Doersch, S. Eslami, and A. v. d. Oord, “Data-efficient image recognition with contrastive predictive coding,” _arXiv preprint arXiv:1905.09272_ , 2019.
  * [10] Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” _arXiv preprint arXiv:1906.05849_ , 2019.
  * [11] X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” _arXiv preprint arXiv:2003.04297_ , 2020.
  * [12] Y. Kalantidis, M. B. Sariyildiz, N. Pion, P. Weinzaepfel, and D. Larlus, “Hard negative mixing for contrastive learning,” _arXiv preprint arXiv:2010.01028_ , 2020.
  * [13] D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman, “With a little help from my friends: Nearest-neighbor contrastive learning of visual representations,” _arXiv preprint arXiv:2104.14548_ , 2021.
  * [14] J. Robinson, C.-Y. Chuang, S. Sra, and S. Jegelka, “Contrastive learning with hard negative samples,” _arXiv preprint arXiv:2010.04592_ , 2020.
  * [15] C.-H. Ho and N. Vasconcelos, “Contrastive learning with adversarial examples,” _arXiv preprint arXiv:2010.12050_ , 2020.
  * [16] M. Kim, J. Tack, and S. J. Hwang, “Adversarial self-supervised contrastive learning,” _arXiv preprint arXiv:2006.07589_ , 2020.
  * [17] J. Robinson, L. Sun, K. Yu, K. Batmanghelich, S. Jegelka, and S. Sra, “Can contrastive learning avoid shortcut solutions?” _arXiv preprint arXiv:2106.11230_ , 2021.
  * [18] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9729–9738.
  * [19] J. Li, P. Zhou, C. Xiong, R. Socher, and S. C. Hoi, “Prototypical contrastive learning of unsupervised representations,” _arXiv preprint arXiv:2005.04966_ , 2020.
  * [20] Y. Cao, Z. Xie, B. Liu, Y. Lin, Z. Zhang, and H. Hu, “Parametric instance classification for unsupervised visual feature learning,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [21] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, “What makes for good views for contrastive learning,” _arXiv preprint arXiv:2005.10243_ , 2020.
  * [22] X. Chen and K. He, “Exploring simple siamese representation learning,” _arXiv preprint arXiv:2011.10566_ , 2020.
  * [23] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” _arXiv preprint arXiv:2006.09882_ , 2020.
  * [24] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar _et al._ , “Bootstrap your own latent: A new approach to self-supervised learning,” _arXiv preprint arXiv:2006.07733_ , 2020.
  * [25] Y. M. Asano, C. Rupprecht, and A. Vedaldi, “Self-labelling via simultaneous clustering and representation learning,” _arXiv preprint arXiv:1911.05371_ , 2019.
  * [26] I. Misra and L. v. d. Maaten, “Self-supervised learning of pretext-invariant representations,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 6707–6717.
  * [27] X. Chen, S. Xie, and K. He, “An empirical study of training self-supervised vision transformers,” _arXiv preprint arXiv:2104.02057_ , 2021.
  * [28] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9650–9660.
  * [29] L. Bottou, “Large-scale machine learning with stochastic gradient descent,” in _Proceedings of COMPSTAT’2010_. Springer, 2010, pp. 177–186.
  * [30] Y. You, I. Gitman, and B. Ginsburg, “Large batch training of convolutional networks,” _arXiv preprint arXiv:1708.03888_ , 2017.
  * [31] I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” _arXiv preprint arXiv:1608.03983_ , 2016.
  * [32] H. Xu, H. Xiong, and G.-J. Qi, “K-shot contrastive learning of visual features with multiple instance augmentations,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [33] X. Wang and G.-J. Qi, “Contrastive learning with stronger augmentations,” _arXiv preprint arXiv:2104.07713_ , 2021.