# Contrastive Learning with Stronger Augmentations

Xiao Wang and Guo-Jun Qi X. Wang was with Department of Computer Science,
Purdue University,West Lafayette, 47906, USA. G-J. Qi was with the Futurewei
Seattle Cloud Lab, Seattle, WA, 98006, USA. Email<EMAIL_ADDRESS>Manuscript received April, 2021; revised xx, xx; accepted
xx,xx.(Corrresponding author: Guo-Jun Qi)

###### Abstract

Representation learning has significantly been developed with the advance of
contrastive learning methods. Most of those methods have benefited from
various data augmentations that are carefully designated to maintain their
identities so that the images transformed from the same instance can still be
retrieved. However, those carefully designed transformations limited us to
further explore the novel patterns exposed by other transformations.
Meanwhile, as found in our experiments, the strong augmentations distorted the
images’ structures, resulting in difficult retrieval. Thus, we propose a
general framework called Contrastive Learning with Stronger Augmentations
(CLSA) to complement current contrastive learning approaches. Here, the
distribution divergence between the weakly and strongly augmented images over
the representation bank is adopted to supervise the retrieval of strongly
augmented queries from a pool of instances. Experiments on the ImageNet
dataset and downstream datasets showed the information from the strongly
augmented images can significantly boost the performance. For example, CLSA
achieves top-1 accuracy of 76.2% on ImageNet with a standard ResNet-50
architecture with a single-layer classifier fine-tuned, which is almost the
same level as 76.5% of supervised results. The code and pre-trained models are
available in https://github.com/maple-research-lab/CLSA.

###### Index Terms:

Contrastive Learning, Distributional Divergence, Data Augmentation, Strong
Augmentation, Self-Supervised Learning

##  1 Introduction

Deep neural networks have shown their sweeping successes in learning from
large-scale labeled datasets like ImageNet [1]. However, such successes hinge
on the availability of a large number of labeled examples that are expensive
to collect. To address this challenge, unsupervised visual representation
learning and self-supervised learning have been extensively studied to learn
feature representations without labels. Researchers aim to build a framework
that can self-learn representations, which does not suffer from the request of
many labels and can learn robust and general representations. Among them, the
contrastive learning [2, 3, 4, 5, 6], which is one formulation of the instance
learning, shows significant potentials to close the performance gap with
supervised methods.

In instance learning [7], each image is being considered as an instance, and
we wish to train the network so that the representations of different
augmented views of the same instance are as close as possible to each other.
Meanwhile, the representation of different views from different instances
should be distinctive to each other. To achieve this, one widely adopted
method is contrastive learning [2], which minimizes the similarity between
views from the same instance while maximizes the similarity among views from
different instances at the same time. To further improve contrastive learning,
various methods [5, 8, 9, 10, 11, 12, 13, 14, 15] are proposed to explore
different directions, such as the number of negative examples [9], the quality
of negative examples [5, 8], data augmentation [16, 8].

On the one hand, it is worth noting that these methods usually rely on image
augmentations that are carefully designated to maintain their instance
identities so that the augmentation of an instance can be accurately retrieved
from a pool of instances. The big impact of the data augmentation design has
been clearly investigated in InfoMin [16], which also suggests the potentials
of strong augmentations. On the other hand, novel patterns exposed by stronger
augmentations can further boost the model’s performance, which is clearly
demonstrated in supervised and semi-supervised tasks [17, 18, 19]. Hence, we
believe the patterns embedded in stronger augmentations could also contribute
to self-supervised learning by improving the generalizability of learned
representations and eventually close the gap with the fully supervised models.
However, directly using stronger augmentations in contrastive learning could
deteriorate the performance, because the induced distortions could
ridiculously change the image structures and thus the transformed images
cannot keep the identity of the original instances. Thus, additional efforts
are needed to explore the role of the stronger augmentations to further boost
self-supervised learning.

Thus we propose the CLSA (Contrastive Learning with Stronger Augmentations)
framework to address this challenge. Instead of applying strongly augmented
views to the contrastive loss, we propose to minimize the distribution
divergence between the weakly and strongly augmented images over a
representation bank to supervise the retrieval of stronger queries. First,
this design avoids an over-optimistic assumption that the strongly augmented
view’s embedding should be identical with that of the weakly augmented view.
Meanwhile, leveraging weakly augmented counterparts’ distributions enables our
framework to explore the novel pattern carried by strongly augmented views.
Most importantly, since it’s independent to the contrastive loss, our
framework can combine with any contrastive loss-based methods, such as MoCo
[5, 4], SimCLR [8], BYOL [20] and so on. Our experiments have shown that our
framework can greatly boost performance by introducing the distributional
loss, which clearly suggests the novel distributional loss can explore the
novel patterns exposed by the stronger augmentations and inherit the knowledge
about the relative similarities to the negative samples. Furthermore, our
experiments also verify that CLSA does not only improved the feature
representation quality of weakly augmented views, but also further enhanced
the representations of strongly augmented views at the same time. This
property can greatly extend the self-supervised learning application on non-
natural images, where the images are not carefully organized and the qualities
of images cannot be guaranteed.

The experiments on various datasets demonstrate that the proposed framework
can significantly boost the performance by learning from stronger
augmentations. On the ImageNet linear evaluation protocol, we reach a record
76.2% top-1 accuracy with the standard ResNet-50 backbone, which is almost as
high as 76.5% top-1 accuracy of the fully supervised model. Meanwhile, it also
achieves competitive performances on several downstream tasks. Among them is a
top-1 accuracy of 93.6% on VOC07 by the linear classifier with a pre-trained
ResNet-50 compared to the previous record of 88.9% top-1 accuracy. For the
COCO object detection, the $AP_{S}$ for small object detection has been
improved to 24.4% from the previous best $AP_{S}$ of 20.8%. These results show
that the CLSA can more effectively capture the semantic information than the
previous self-supervised methods on downstream tasks by introducing the
strongly augmented images. We also conduct an ablation study to show a naive
application of stronger augmentations in contrastive learning would degrade
the performances.

Our contribution can be summarized as follows.

  * •

We are the first to explore the stronger augmentations to contribute to self-
supervised learning.

  * •

We propose a distributional loss to transfer knowledge from weakly augmented
views to strongly augmented views.

  * •

CLSA can easily integrate with concurrent contrastive loss-based methods and
greatly boost their performance.

  * •

We carefully carry the ablation study to verify the impact of the
distributional loss.

  * •

CLSA framework can self-train network to improve representation for weakly
augmented images and strongly augmented images simultaneously.

##  2 Related Work

###  2.1 Self-Supervised Learning

Self-supervised learning methods have been widely studied to close the gap
with supervised learning and alleviate the time and cost for labeling a large
amount of data. These methods can be categorized into five different aspects.

Instance Discrimination and Contrastive Learning Each image is considered as
an individual class in an instance discrimination setting [2, 9, 8, 5]. It can
be further formulated as contrastive learning [2], whose core idea is to pull
the positive pairs together and push the negative pairs away in the embedding
space. As studied in many prior works [21, 8, 4, 5, 22], it is very crucial to
construct high-quality positive and negative pairs to achieve higher
performance. In particular, InstDisc[9] built a memory bank that stores pre-
computed representations as negative pairs to increase the size of the
negative pair to improve the performance. Following this work, MoCo[5] used a
momentum update mechanism to maintain a long queue of negative examples for
contrastive learning. This momentum encoder design greatly improved the
quality of negative pairs and led to a big performance increase compared to
previous works. SimCLR[8] further improved by directly utilizing negative
samples in the current batch with a much bigger batch size. Meanwhile, it
carefully constructed a rich family of data augmentations on cropped images,
which significantly boosted classification accuracy. MoCo V2 [4] verified the
improvement by introducing the same data augmentation and MLP layer design to
MoCo framework. However, these methods failed to improve the performance by
naively applying stronger augmentations to minimize the contrastive loss,
which motivated the proposed work.

Generative Methods The generative methods typically adopt auto-encoders [23],
and adversarial learning [24] to train an unsupervised representation.
Usually, they focused on the pixel-wise information of images to distinguish
images from different classes. For instance, [24] adopted BiGAN to capture the
relationship between latent semantic representations and the input images.

Clustering Clustering [25] can also be used to learn visual representations by
assigning pseudo cluster labels to individual samples. DeepCluster [25]
generalized k-means by alternating between assigning pseudo-labels and
updating networks. Recently, SWAV [6] is proposed to learn a cluster of
prototypes by enforcing consistency between cluster assignments for different
views, which has achieved state-of-the-art performance on ImageNet.

Like contrastive learning, memory bank, large batches, and prototypes queue
are also utilized in the clustering-based methods. Moreover, though
clustering-based methods do not specifically use negative pairs, the cluster
centers can be viewed as negative prototypes. Hence, clustering-based approach
can also be viewed as an extension of contrastive learning.

Consistency Representation Learning Instead of directly using contrastive
loss, researchers [20, 26] found consistency representation learning between
positive pairs can also enforce the network to learn robust representations.
In BYOL [20], researchers first found that we can self-train an encoder
without using negative examples. It utilized the Siamese architecture, where
the query branch has a predictor architecture in addition to the encoder and
projector. The encoder can learn good representations by simply minimizing the
cosine similarity between query embedding and key embedding. Following BYOL,
Simsiam [26] further removed the momentum key encoder and used the stop-
gradient strategy to avoid the collapsing problems. Furthermore, SCRL [27]
further applied the consistency loss for ROI of the intersection region of two
views to improve the encoder representation for downstream detection tasks.

Pretext Tasks In addition to contrastive learning, there exist many
alternative methods using different pretext tasks [28] to train networks with
the carefully designed supervised signals from images. For example, [29]
proposed Jigsaw puzzles as supervised signal to train a convolutional neural
network. [30] used the relative positions of two randomly sampled patches as
the supervised signal. [31] used the transformations between two images as
supervised signals to guide the representation learning. For more details
about these works, please refer to the survey by [32].

##  3 CLSA: Contrastive Learning with Stronger Augmentations

In this section, we will first review the preliminary work on contrastive
learning, and discuss their strength and limitations in Section. 3.1. Then in
Section. 3.2, we will present a new distributional divergence loss between
weakly and strongly augmented images to self-train the representations by
leveraging the underlying visual semantic information from strongly augmented
views. After that, the implementation details are explained in Section. 3.3.

###  3.1 Contrastive Learning

Contrastive learning [2] is a popular self-supervised idea and made great
success in recent years with the advance of computation and various image
augmentations. Its goal is to find a parametric function $f_{\theta}$ that
maps the input image $x\in\mathbb{R}^{D}$ to a feature representation
$z=f_{\theta}(x)\in\mathbb{R}^{d}$, such that the feature representation $z$
in the feature space can reflect the semantic similarities in the input space.
To this end, the contrastive loss is proposed to optimize the network
$f_{\theta}$, which encourages $z$ and its positive pair $z^{\prime}$ to be
close in the feature space, and pushes away representations of all other
negative pairs. After SimCLR [8], projector $g$ is imported to further map the
representation as $z=g_{\theta}(f_{\theta}(x))$ for contrastive pre-training,
while we still only use $f_{\theta}(x)$ for downstream tasks
(classification/detection). This design has been proved to the key to boost
performance for contrastive learning [4, 8].

Figure 1: Contrastive instance learning framework

Fig. 1 illustrates the latest general framework of contrastive learning
methods. In the supervised settings, the contrastive loss can be achieved by
defining the same class images as positive pairs while remaining others as
negative pairs. Similarly, the definition of positive pairs in contrastive
loss is inspired by instance learning [2, 9, 5, 8, 4], where random augmented
crops of the same image could be defined as positive pairs and crops from
other images are all regarded as negatives. Hence, the contrastive loss in
self-supervised learning is to maximize the agreement of representations of
different views of the same instance while minimizing the agreement with other
negative samples.

Figure 2: Comparison of the strongly weakly augmented images. The left is the
original image, the middle is the weakly augmented image, and the right is the
strongly augmented one with over-contrastive details.

Specifically, for each image $x$ in batch $B$, we apply two transformations
$T^{\prime}$ and $T$ to obtain two different views $V^{\prime}$ and $V$ of the
same instance $x$. Then they go through a query encoder $f_{\theta}$ and a key
encoder $f_{\phi}$ respectively, followed with MLP projection layers
($g_{\theta}$/$g_{\phi}$), resulting in two embedded representations
$z^{\prime}$ and $z$ to calculate the constrastive loss in Eq. (1).

$\centering\mathcal{L}_{C}=\mathbb{E}_{i\in
B}[-\log\frac{Q(i,i+)}{Q(i,i+)+\sum_{k=1}^{K}Q(i,k)}]\@add@centering$ (1)

with

$\centering\left\\{\begin{array}[]{l}\mathcal{Q}(i,i+)=\exp(sim(z_{i}^{\prime},z_{i})/\tau)\\\
\mathcal{Q}(i,k)=\exp(sim(z_{i}^{\prime},z_{k})/\tau)\\\
sim(z_{i}^{\prime},z_{k})=\frac{z_{i}^{\prime
T}z_{k}}{||z_{i}^{\prime}||\cdot||z_{k}||}\end{array}\right.\@add@centering$
(2)

where $\mathcal{L}_{C}$ is the contrastive loss, $Q(i,i+)$ is the exponential
temperature smoothed similarities of positive pairs between $z_{i}^{\prime}$
and $z_{i}$, while $Q(i,k)$ is the exponential temperature smoothed
similarities of negative pairs between $z_{i}^{\prime}$ and $z_{k}$, $\tau$ is
the temperature parameter set to $0.2$, and $sim$ denotes the cosine
similarity. Here $z_{k}$ is the feature embedding of other instances by
$f_{\phi}$ and $g_{\phi}$. $K$ is the size of the FIFO queue $M$ (Memory Bank)
saving the feature embeddings of other instances from previous batches.

In contrastive learning, success is highly dependent on two components:

1) The design of positive pairs: For the positive pairs, the data augmentation
is carefully designed. In SimCLR [8], it carefully designed color-jittering,
Gaussian blurring transformations to further augment random cropped views. In
InfoMin [16] explored the effect of different data augmentations in
contrastive pre-training and has shown some combinations of data augmentation
can further improve compared to MoCo [4] or SimCLR [8].

2) The design of negative pairs: For the negative pairs, researchers have
explored a large number of approaches to improve the number and the quality of
negative pairs. For example, InstDisc [9] first deployed memory bank (negative
pool) to keep track of feature embeddings of previous batches as negative
pairs, which greatly improved the performance by a large pool of negatives.
MoCo [5, 4] further improved the quality of negatives by using momentum
encoder as key encoder $g_{\theta}$. SimCLR [8] kept a balance of quality and
number of negatives by utilizing large batch online training and using other
instances in the same batch serving as negatives for optimizing contrastive
loss. The potential and possibility of negatives have been fully explored in
various perspectives and greatly improved representation learning.

Nevertheless, the potential of positive pairs is still lacking enough studies.
As illustrated aforementioned, the two views of the same instance in
contrastive learning are not transformed aggressively so that they can still
be viewed as the same instance. InfoMin [16] has proved there still existing
unexplored patterns exposed by different combinations of weak augmentations.
That inspired us that augmentations for positive pairs can leverage possible
semantic information for the encoder to learn. Furthermore, we do not want to
rely on the carefully designed augmentation for training. Instead, we aim to
explore a general and robust approach to learn from infinite data
augmentations.

However, directly adopting stronger transformations (e.g., with larger
rotation angles, more aggressive colorjittering and cutout) in contrastive
learning fails to further improve the performance or even deteriorate it for
downstream tasks, which is not surprising. Stronger transformations could
distort image structures and their perceptual patterns in the learned
representation so that strongly augmented counterpart views cannot be viewed
as the same instance for training the underlying network. In InfoMin [16],
they also only explored the combinations of weak augmentations instead of
stronger augmentations that may include more semantic information and
representative patterns. Since different combinations of weak augmentations
provided different clues for obtaining distinctive feature representations,
it’s highly possible that some useful clues may only exist in the stronger
augmentations. In supervised learning [17, 33, 34] and semi-supervised
learning [19], different stronger data augmentations have been widely studied
and greatly boost the performance with the novel pattern exposed by strongly
augmented images. The findings in RandAugment [17] have verified that strongly
augmented views can provide more clues even without an explicit augmentation
policy. Hence, we believe learning the representations from these novel
patterns will pave the last mile to close the gap with the fully supervised
representations. That further inspired us to explore novel ways to utilize
stronger transformations in self-supervised learning while avoiding
deteriorated performances by naively using them in a contrastive model [4].

Figure 3: The comparison of the distribution of positive pair’s probabilities
and variance of negative pairs’ probabilities given weakly augmented query and
strongly augmented query. A. The distribution with a randomly initialized
network. B. The distribution with a pre-trained network by contrastive
methods.

By exploring previous approaches and our extensive experiments, we have found
it is not a straightforward task to learn the patterns embedded in the
stronger augmentations. As shown in Fig. 2, a strongly augmented image may
look perceptually different from the original counterpart. Consequently, the
representation of a strongly augmented image can be far apart from that of the
weakly augmented one. Thus, naively using strongly augmented images in
contrastive learning can be over-optimistic since the induced distortions
could dramatically ruin their image structures.

To this end, in Section 3.2, we instead proposed the Distributional Divergence
Minimization (DDM) between weakly and strongly augmented images over a
representation bank to avoid overfitting the representation of a strongly
augmented image with that of the corresponding positive target.

###  3.2 Distributional Divergence Minimization between Weakly and Strongly
Augmented Images

Figure 4: Diagram of distributional divergence minimization. Here the
representation bank consists of $K$ stored features $z_{k}$ of previous
batches and online features $z_{i}$ from the key encoder. They will be used to
calculate the conditional probability of current weakly and strongly augmented
query images.

Due to the limitation mentioned above, learning from the retrieval of a
strongly augmented query is infeasible to self-train deep networks.
Nevertheless, the distribution of relative similarities can help us understand
contrastive learning in a different direction, thus inspired us to propose the
distributional divergence minimization (DDM) for learning from stronger
augmentations.

As shown in Fig. 1, for an image $x$, we can first apply a transformation
$T^{\prime}$ to generate a query view $V^{\prime}$ and then obtain its
corresponding embedding $z^{\prime}=g_{\theta}(f_{\theta}(V^{\prime}))$.
Meanwhile, we collect another key view $V$ generated by $T$ and the key
embedding $z=g_{\phi}(f_{\phi}(V))$. Given a memory bank $M$ of $K$ negative
samples $\\{z_{k}|k=1,\cdots,K\\}$ accumulated from key embedding $z$ of the
past iterations, we can obtain a conditional distribution

$\centering
p(z_{k}|z^{\prime}_{i})=\frac{\exp(sim(z^{\prime}_{i},z_{k})/\tau)}{\exp(sim(z^{\prime}_{i},z_{i})/\tau)+\sum_{k=0}^{K}\exp(sim(z_{i}^{\prime},z_{k})/\tau)}\@add@centering$
(3)

which encodes the likelihood of the query $z^{\prime}_{i}$ being assigned to
the embedding $z_{k}$ from the memory bank $M$.

Similarly, we can also have the likelihood of positive pairs for the query
$z^{\prime}_{i}$ being assigned to its positive counterpart $z_{i}$ from the
key encoder:

$\centering
p(z_{i}|z^{\prime}_{i})=\frac{\exp(sim(z^{\prime}_{i},z_{i})/\tau)}{\exp(sim(z^{\prime}_{i},z_{i})/\tau)+\sum_{k=0}^{K}\exp(sim(z_{i}^{\prime},z_{k})/\tau)}\@add@centering$
(4)

Then, the contrastive loss in Eq. (1) can be rewritten in another form as

$\centering\mathcal{L}_{C}=\mathbb{E}_{i\in B}[-q(z_{i}|z_{i}^{\prime})\log
p(z_{i}|z_{i}^{\prime})-\sum_{k=1}^{K}q(z_{k}|z_{i}^{\prime})\log
p(z_{k}|z_{i}^{\prime})]\@add@centering$ (5)

where $q(z_{i}|z_{i}^{\prime})$ is the ideal distribution of the likelihood,
$p(z_{i}|z_{i}^{\prime})$ is the distribution learned by network.

However, we cannot obtain the ideal distribution of the likelihood from the
semantic view. It is hard to measure the optimal likelihood between the query
image and the key images (positive/negative). To avoid this unknown
distribution exploration, contrastive loss regards $q$ as a one-hot
distribution, where positive pair has $q(z_{i}|z_{i}^{\prime})=1$ and
negatives satisfy $q(z_{k}|z_{i}^{\prime})=0~{}(k\in[1,K])$. That means
contrastive loss only maximizes the agreement of representations of different
views of the same instance while minimizing the agreement with other negative
samples. All other underlying complicated relationships between query images
and key images are completely ignored. The advantage is that contrastive loss
can greatly accelerate the convergence of representation learning and greatly
improve the representation feature for classification and detection tasks.
However, the information between the query image and negative images is not
fully explored and may include useful clues to boost representation learning
further.

Similar to the representation for weakly augmented views, a straightforward
solution for exploring the patterns from the stronger augmentation is directly
using the strongly augmented image as query and using the weakly augmented
image as key in the contrastive loss. However, this too optimistic design
preassumes that the strongly augmented view’s representation should be close
to that of its weakly augmented pair and far from that of the weakly augmented
views of other instances. The one-hot distribution fails to mimic or even
approximate the optimal likelihood distribution and thus cannot help the
representation learning anymore. Hence, another alternative distribution $q$
should be proposed to address these limitations of one-hot distribution.

Though it is almost impossible to obtain the actual likelihood distribution to
self-train network in a perfect way, fortunately, we found that the
distribution of relative similarities of a weakly augmented image from the
same instance over the representation bank can provide useful clues for
stronger augmentation learning. In Fig. 3, we compared the distribution of
positive pair’s probabilities $p(z_{i}|z^{\prime}_{i})$
($p(z_{i}|z^{\prime\prime}_{i})$) and the distribution of variance of negative
pairs’ probabilities $p(z_{k}|z^{\prime}_{i})$
($p(z_{k}|z^{\prime\prime}_{i})$) for $k\in[1,K]$ for weakly (strongly)
augmented query. In Fig. 3A, it is clear that the initial positive probability
and variance of the negative probability of strongly augmented query is
identical to that of weakly augmented query given the same key from the same
instance. That indicates that weakly query and strongly query shared almost
identical distribution for a randomly initialized network. This property
inspired us that the distribution of relative similarities of weakly augmented
query can be used to supervise that of the strongly augmented query.

In Fig. 3, we further measured the distributions by the pre-trained network by
one of the most representative contrastive approaches MoCo [5, 4]. Compared to
that of a randomly initialized network, we can clearly find that a strongly
augmented query’s likelihood distribution becomes much more different from
that of a weakly augmented query. This distribution difference suggests that
concurrent contrastive methods failed to improve the representations of
strongly augmented views at the same time. Our ablative studies found that the
representations of strongly augmented views by contrastive methods are much
less distinctive than that of weakly augmented views. Thus, we believe the
distribution supervision of weakly augmented views can simultaneously improve
the representation of strongly augmented views.

In a nutshell, it does not only avoid directly placing the representation of a
strongly augmented image too closely to that of the positive target (like one-
hot distribution did), but also allows it to explore the novel patterns of
variations exposed by the strong augmentation.

Formally, as shown in Fig. 4, for an original image $x_{i}$, we applied
another stronger augmentation $S$ to obtain its strongly augmented view
$V^{\prime\prime}_{i}$, and its embedding $z^{\prime\prime}_{i}$. Meanwhile,
we can also obtain its corresponding query embedding $z^{\prime}_{i}$ and key
embedding $z_{i}$ for the instance $x$ by weak augmentations $T^{\prime}$ and
$T$. Similar to Eq. (3) and Eq. (4), we obtain a conditional distribution for
$z^{\prime\prime}_{i}$ based on its positive target and negative pairs:

$\centering\left\\{\begin{array}[]{l}p(z_{k}|z^{\prime\prime}_{i})=\frac{\exp(sim(z^{\prime\prime}_{i},z_{k})/\tau)}{\exp(sim(z^{\prime\prime}_{i},z_{i})/\tau)+\sum_{k=0}^{K}\exp(sim(z_{i}^{\prime\prime},z_{k})/\tau)}\\\
p(z_{i}|z^{\prime\prime}_{i})=\frac{\exp(sim(z^{\prime\prime}_{i},z_{i})/\tau)}{\exp(sim(z^{\prime\prime}_{i},z_{i})/\tau)+\sum_{k=0}^{K}\exp(sim(z_{i}^{\prime\prime},z_{k})/\tau)}\\\
\end{array}\right.\@add@centering$ (6)

Then, we propose to minimize the following distributional divergence between
the weak and the strong queries such that

$\centering\mathcal{L}_{D}=\mathbb{E}_{i\in B}[-p(z_{i}|z_{i}^{\prime})\log
p(z_{i}|z_{i}^{\prime\prime})-\sum_{k=1}^{K}p(z_{k}|z_{i}^{\prime})\log
p(z_{k}|z_{i}^{\prime\prime})]\@add@centering$ (7)

By minimizing this divergence, we assume the learned representation
$z^{\prime\prime}_{i}$ of the strongly augmented query should inherit the
representation $z^{\prime}_{i}$ of the weakly augmented one regarding not only
its belief of the query being assigned to the corresponding positive target
$z_{i}$, but also its relations with the negative samples $z_{k}$ in the
representation bank through the conditional distribution
$p(z_{k}|z^{\prime}_{i})$. We believe the distribution
$p(z_{k}|z_{i}^{\prime})$ is a much better mimic of
$q(z_{k}|z_{i}^{\prime\prime})$ instead of the one-hot distribution, which is
also validated in our ablation study.

TABLE I: Various augmentations we applied in experiments to strongly augment
training images.

Operation | ShearX(Y) | TranslateX(Y) | Rotate
---|---|---|---
Mag Range | [-0.3,0.3] | [-0.3,0.3] | [-30,30]
Operation | AutoContrast | Invert | Equalize
Mag Range | 0 or 1 | 0 or 1 | 0 or 1
Operation | Solarize | Posterize | Contrast
Mag Range | [0,256] | [4,8] | [0.05,0.95]
Operation | Color | Brightness | Sharpeness
Mag Range | [0.05,0.95] | [0.05,0.95] | [0.05,0.95]

This will prevent direct overfitting of the strong query representation to the
positive target as well as improve the generalization of the learned
representation with additional clues from the other examples in the
representation pool. In a more general sense, this extends the idea of
knowledge distillation [35]. However, we did not use the predicted labels by a
teacher model to supervise a student model’s training as in the knowledge
distillation. Instead, we used the distribution of the likelihoods of a weak
query to supervise the retrieval of a strong query from a pool of
representations.

Since the distributional divergence optimization can be viewed as a knowledge
distillation process, that also indicates that the success is highly dependent
on the distribution of relative similarity of weakly augmented query
$z^{\prime}$. Hence, it is natural to utilize one of the current contrastive
methods to learn $z^{\prime}$ through the contrastive loss. That design also
enables our DDM loss to be fully independent with concurrent contrastive
methods and can complement any of them to further boost them. As illustrated
in Fig. 4, the overall loss to optimize the encoder can be formulated as

$\centering\mathcal{L}=\mathcal{L}_{C}+\beta*\mathcal{L}_{D}\@add@centering$
(8)

where $\beta$ is the coefficient to balance the contrastive loss
$\mathcal{L}_{C}$ and distributional divergence minimization loss $L_{D}$.
Though other values may achieve better results for different contrastive
baselines, we use $\beta=1$ to make CLSA more general .

As shown in Fig. 4, the pattern from strongly augmented images and weakly
augmented images are learned simultaneously. That is a very natural and
effective design since contrastive loss can also be viewed as a specific form
of distributional divergence minimization as we have shown before, where the
supervised distribution is a one-hot distribution. In short, the overall
optimization thus can also be interpreted as two simultaneous distributional
divergence minimization. The extraordinary performance also supported this
simple yet effective design.

TABLE II: Top-1 accuracy under the linear evaluation on ImageNet with the ResNet-50 backbone with 200 epochs training.  Method | Top 1
---|---
InstDisc [9] | 54.0
LocalAgg [13] | 58.8
MoCo [5] | 60.8
SimCLR [8] | 61.9
CPC v2 [15] | 63.8
PCL [36] | 65.9
MoCo v2 [4] | 67.5
InfoMin Aug [16] | 70.1
SWAV [6] | 72.7
CLSA | 69.4
CLSA* | 73.3
Supervised | 76.5

###  3.3 Implementation Details

In this section, we will discuss the details about the applied strong and weak
augmentations for CLSA.

Stronger Augmentations $\bm{S}$ As explored in the previous works ([18, 19,
28]), strong augmentations usually have two types: geometric and non-geometric
augmentations. Specifically, we considered $14$ types of augmentations:
ShearX/Y, TranslateX/Y, Rotate, AutoContrast, Invert, Equalize, Solarize,
Posterize, Contrast, Color, Brightness, Sharpness. The magnitude of each
augmentation is significant enough to produce as strong augmentations as
possible. More details of different transformations are shown in Table I. For
example, the shear operation is drawn from a range of [-0.3,0.3], resulting in
aggressively transformed images that can be hard to retrieve given a
counterpart target. In particular, to transform an image, we randomly select
an augmentation from the above $14$ types of transformations, and apply it to
the image with a probability of $0.5$. This process is repeated five times and
that will strongly augment an image as the example shown in the right panel of
Fig. 2. Compared to the weakly augmented image in the middle panel, it is
clear the image structures of the strongly augmented views are completely
changed.

Weaker Augmentations $\bm{T}$ Weak augmentations are drawn by following most
of existing contrastive learning methods in literature ([8, 4, 6, 5]): an
image is first cropped from an input image and resized to 224$\times$224
pixels. Then random color jittering, Gaussian Blur, grayscale conversion,
horizontal flip, channel-wise color normalization are sequentially applied to
generate weakly augmented images with an example shown in the middle of Fig.
2.

Technical Details Similar to the previous works ([5, 8, 6]), we used the
ResNet-50 ([37]) as our encoder backbones $f_{\theta}$ and $f_{\phi}$ and a
2-layer MLP (2048-d hidden layer with the ReLU, output FC without ReLU) as the
projection head $g_{\theta}$ and $g_{\phi}$. The cosine similarity is used in
the contrastive loss and DDM loss. The temperature $\tau$ is set to $0.2$.
Following MoCo [5], a momentum smoothing factor $\alpha$ of 0.999 is used to
update key encoder $f_{\phi}=\alpha*f_{\phi}+(1-\alpha)*f_{\theta}$ and key
MLP $g_{\phi}=\alpha*g_{\phi}+(1-\alpha)*g_{\theta}$. The loss balancing
coefficient $\beta$ is set as $1.0$. We set the size $K$ of the queue $M$ to
$65536$ to store the negative examples used to compute the conditional
distribution of weakly and strongly augmented queries and minimize their
divergence. We used the same temperature for DDM loss and contrastive loss to
simplify the formulation. We believe the performance can be further improved
by adjusting different temperatures for $\mathcal{L}_{C}$ and
$\mathcal{L}_{D}$.

TABLE III: Top-1 accuracy under the linear evaluation on ImageNet with the ResNet-50 backbone with various numbers of epochs. Method | Top 1
---|---
BigBiGAN [38] | 56.6
SeLa-400epochs [39] | 61.5
PIRL-800epochs [3] | 63.6
CMC [14] | 66.2
SimCLR-800epochs [8] | 70.0
MoCo v2-800epochs [4] | 71.1
InfoMin Aug-800epochs [16] | 73.0
BYOL-1000epochs [20] | 74.3
SWAV-800epochs [6] | 75.3
CLSA-800epochs | 72.2
CLSA*-800epochs | 76.2
Supervised | 76.5
TABLE IV: Transfer learning results on various downstream tasks. | Classification | Object Detection
---|---|---
| VOC07 | VOC07+12 | COCO
Measurement | Accuracy | $AP_{50}$ | AP | $AP_{S}$
RotNet [40] | 64.6 | - | - | -
NPID++ [9] | 76.6 | 79.1 | - | -
MoCo [5] | 79.8 | 81.5 | - | -
PIRL [3] | 81.1 | 80.7 | - | -
PCL [36] | 84.0 | - | - | -
BoWNet [41] | 79.3 | 81.3 | - | -
SimCLR [8] | 86.4 | - | - | -
MoCov2 [4] | 87.1 | 82.5 | 42.0 | 20.8
SWAV [6] | 88.9 | 82.6 | 42.1 | 19.7
CLSA | 93.6 | 83.2 | 42.3 | 24.4
Supervised | 87.5 | 81.3 | 40.8 | 20.1

##  4 Experiments

###  4.1 Training Details

For the unsupervised pretraining on ImageNet with the CLSA, we used the SGD
optimizer ([42]) with an initial learning rate of 0.03, a weight decay of
0.0001 and a momentum of 0.9. We used cosine scheduler ([43]) to gradually
decay the learning rate to 0. Usually, the batch size is set to 256. When
multiple GPU cluster servers are used, the batch size will be multiplied by
the same number of servers by convention. To avoid collapsing, the shuffle BN
[5] is also adopted. The learning rate then will be adjusted based on
$lr=base\\_lr*batch\\_size/256$, where $base\\_lr$ is 0.03. Typically, the
experiment with a single strong augmentation for each training image takes
roughly 70 hours to finish on 8 V100 GPUs.

For the fine-tuning on ImageNet, we trained a linear classifier on top of the
frozen feature vector (2048-D) upon the pre-trained ResNet-50 with CLSA. This
linear layer is trained for 100 epochs, with a learning rate of 10 without
weight decay. We used the cosine learning rate decay to 0.01 and a batch size
of 256.

For the transfer learning on the VOC dataset, we trained a linear classifier
upon the pre-trained Resnet-50 in a similar way for ImageNet – we trained 100
epochs with the SGD optimizer and a learning rate of 0.05, a momentum of 0.9
and no weight decay. The batch size is 256 without a learning rate scheduler.

Finally, for object detection, we adopted the same protocol in ([5]) to fine-
tune the pre-trained Resnet-50 backbone based on detectron2 ([44]) for the
sake of a fair and straight comparison with the other methods.

###  4.2 Linear Classification on ImageNet

For the linear evaluation on ImageNet, we trained the CLSA in two settings. In
the first setting named CLSA, we used a single stronger augmentation (see
Table I) that crops each training image to a smaller size of $96\times 96$,
which does not incur too much computing overhead in processing these smaller
augmented images. In the second setting named CLSA*, we adopted five different
stronger augmentations that crop each image into various sizes: $224\times
224,192\times 192,160\times 160,128\times 128,$ and $96\times 96$. The DDM
loss in Eq. (7) is the sum over these multiple stronger augmentations. A
similar multi-crop strategy has been adopted in contrastive learning
literature before. For example, the SWAV [6] reached state-of-the-art top-1
accuracy by applying such multi-crop augmentations. To ensure a fair
comparison with the SWAV, we chose five stronger augmentations such that the
self-training with CLSA* consumed similar computing time (i.e., 166 hours with
a cluster of 8 V100 GPUs for 200 epochs of pre-training with a batch size of
256).

TABLE V: Ablation study of the CLSA on ImageNet with 200 epochs of pre-training. Model | Top-1
---|---
MoCo V2 | 67.5
MoCo V2 with Strong query | 67.7
MoCo V2 with Strong query & Strong key | 67.0
CLSA with contrastive loss | 68.0
CLSA | 69.4

As shown in Table II and Table III, we compared the performance with the other
unsupervised methods. All the experiments are based on a pre-trained ResNet-50
backbone that is fine-tuned with a linear classifier. Table II showed the
performance of different methods pre-trained over 200 epochs, and table III
reported models pre-trained over more epochs.

First, under the same contrastive protocol, the CLSA has a higher $69.4\%$
top-1 accuracy than both MoCo v2 (67.5%) and SimCLR (61.9%) with 200 epochs
training. With multiple stronger augmentations, CLSA* outperforms the state-
of-the-art SWAV model using multi-crops of training images over 200 epochs
(73.3% vs. 72.7%). Moreover, as shown in III, the CLSA outperforms MoCo v2 and
SimCLR with the same training epochs. It is also noteworthy that the CLSA*
achieves almost the same top-1 accuracy as that of the fully supervised
network (76.2% vs. 76.5%). With the advance of more contrastive learning
approaches, we believe CLSA can work with them to further improve their
performance to beat supervised learning performance with more robust feature
representations.

###  4.3 Transfer Learning Results on Downstream Tasks

We test the generalizability of the ResNet-50 pre-trained on ImageNet to
several downstream tasks. Specifically, we focused on two tasks: cross-dataset
image classification and object detection. The pre-trained ResNet-50 was
frozen, and we fine-tuned the linear classifier on the VOC07trainval and
tested it on the VOC07test. For object detection, we evaluated the pre-trained
network on two datasets using the detectron2 [44] used in the previous methods
[5, 8]. On the VOC dataset, we trained the detection head with VOC07+12
trainval dataset and tested it on VOC07 test dataset. We fine-tuned the
network on the train2017 set with 118k images on the COCO dataset and
evaluated it on the val2017. For the sake of a fair comparison, the object
detection tasks are completed by detectron2 based on the pre-trained
ResNet-50.

As shown in Table IV, the performances on both tasks are much better than the
supervised model trained on ImageNet. This suggests that the proposed method
has better generalization ability in downstream tasks. The pre-trained network
on ImageNet by the CLSA outperformed the compared models after being fine-
tuned on different datasets. Among them is a top-1 accuracy of 93.6% on the
VOC07 with the linear classifier on the pre-trained ResNet-50 in comparison
with the previous record of 88.9% top-1 accuracy by the SWAV. On the COCO
dataset, the $AP_{S}$ for small object detection has been significantly
improved to 24.4% from the previously best $AP_{S}$ of 20.8%. As well known,
it is much more challenging to detect small objects on the COCO dataset. Thus,
the better performance of the CLSA could be attributed to the ability to
involve the stronger augmentations that result in many small objects to pre-
train the network.

TABLE VI: Training Time Comparison of CLSA

Model | Time | Epoch | Top-1
---|---|---|---
MoCo V2 | 53h | 200 | 67.5
CLSA | 35h | 100 | 67.2
CLSA | 52.5h | 150 | 68.3
CLSA | 70h | 200 | 69.4

  * 1

$h$ denotes running hours on a machine with 8 V100 GPUs.

###  4.4 Ablation Study

In this section, we analyzed the contribution of DDM loss by comparing it with
other designs. Also, we compared CLSA with other baselines using the same
running time for a more fair comparison. Moreover, we compared CLSA
performance under different strong augmentations. In addition, we carefully
addressed our feature representation improvement for both weakly augmented
views and strongly augmented views. Last but not least, we further studied the
distribution of probabilities of positive and negative pairs with pre-trained
network by CLSA.

DDM Loss In the ablation study shown in Table V, we studied the role of the
proposed DDM loss in the CLSA. First, we naively used the stronger
augmentation applied in the CLSA-Single as the query and/or the key in the
MoCo V2. Both results (Strong query and Strong query & Strong key) showed the
performance could not be improved or even degraded. Second, we replaced the
DDM loss in the CLSA-Single with the contrastive loss, which can be viewed as
replacing $p(z_{i}|z^{\prime}_{i})$ and $p(z_{k}|z^{\prime}_{i})$ with a one-
hot distribution, and we found it can only achieve a top-1 accuracy of 68.0%
compared to that of 69.4% with the DDM loss. That clearly indicates
$p(z_{k}|z^{\prime}_{i})$ is a more suitable approximation of
$q(z_{k}|z^{\prime\prime}_{i})$ compared to one-hot distribution. Both studies
showed that the proposed CLSA and its DDM loss help us learn from stronger
augmentations by avoiding the performance degeneration that augmented images’
distortions would incur.

Figure 5: Comparison of KNN accuracy of MoCo V2 [4] and CLSA. Both results are
compared based on pretrained model with 200/800 epochs with single crop. The
comparison used the representation of weakly/strongly augmented images,
respectively. The neighbor $K$ for KNN here is set to 20.

Running Time To address the concern of extra training time consuming of CLSA
compared to MoCo V2 [4], we compared the results in Table VI. It is clearly
seen that CLSA (68.3%, 52.5h) can still achieve better results than MoCo V2
(67.5%, 53h) with the same running time and converge much faster. That also
indicates that the optimization of contrastive loss and DDM loss at the same
time can also benefit the convergence of representation learning.

TABLE VII: Comparison of CLSA under different strong augmentations Strength $s$ | 3 | 5 (default) | 7
---|---|---|---
KNN Acc | 52.8 | 53.0 | 52.6

The strength of strong augmentation CLSA achieved promising results by
introducing stronger augmentations to further improve self-supervised
learning. In contrastive-based approach, the weak augmentations need to be
carefully designed to remain the identity. Thus, we designed the stronger
augmentations by randomly sampling operations from a large pool of operations
with magnitude in a wide range as shown in Table. I. To make the augmentation
stronger, we repeated this operation $s$ times. In CLSA, we use $s=5$ as
default. As shown in Table. VII, we trained CLSA with 100 epochs and compared
the performance with different augmentation strength $s$. Here, the comparison
is based on the pre-trained model’s KNN classification accuracy, where $K$ is
set to 20. CLSA did not influence too much with different augmentation
strength for stronger augmentation. That suggests the reliability of CLSA
framework, which is very independent of the augmentation strength. This
property can greatly help CLSA to be adjusted to complement other self-
supervised approaches.

Representations for weak/strong augmented images Meanwhile, we also want to
compare CLSA with baseline MoCo V2 to illustrate our representative features
on both weakly augmented images and strongly images. First, following [9], we
center crop the images to obtain features from the last average pooled layers,
and report the accuracy with $K=20$ KNN accuracy. As clearly shown in Fig. 5,
on both 200-epoch (Weak-200) and 800-epoch (Weak-800) pre-trained encoder, we
showed much better performance compared to MoCo V2. Moreover, we further
transformed images using our strong augmentations $S$ mentioned in Section 3.3
and obtain the features for them for KNN clustering. The comparison is also
shown in Fig. 5. It is obvious that we showed much better classification
performance on those strongly augmented images. CLSA improved 8.2% and 9.9%
accuracy compared to MoCo V2 on the 200-epoch (Strong-200) and 800-epoch
(Strong-800) pre-trained model, respectively. Those comparisons suggest that
CLSA can not only increase the representation for weakly augmented images, but
also further greatly boost the representation for strongly augmented images.
To some extent, we can say the representation learned by CLSA is more general
compared to contrastive approaches.

Distribution of Relative Similarity by CLSA Similar to the distribution of the
probability of positive pairs and the distribution of the variance of the
probability of negative pairs shown in Fig. 3, we further compared that by
CLSA in Fig. 3. Compared to contrastive approaches, the distribution of
strongly query showed a more similar pattern compared to that of the weakly
query. This different distribution clearly suggested that DDM can leverage the
knowledge of weakly query to contribute to the learning of strong query. Also,
the increase of the mean of the distribution of positive pairs indicates that
more and more strongly views can be successfully retrieved by its weakly
augmented counterpart. Meanwhile, the increase of the mean of the distribution
of the variance of negative pairs also suggests the relationships between the
representations of strongly query and other representations is more well
captured. In a nutshell, both distribution change agrees well that the feature
representation of strongly augmented views become more representative by CLSA
pre-training.

Figure 6: The comparison of the distribution of positive pair’s probabilities
and variance of negative pairs’ probabilities with a pre-trained network by
CLSA. A. The distribution comparison of positive pair’s probabilities. B. The
distribution comparison of variance of negative pairs’ probabilities.

##  5 Conclusion

In this paper, we present CLSA, a novel method that can utilize the
distributional divergence to learn the information from strongly augmented
images. The proposed method outperforms the state-of-the-art methods on all
the datasets and achieved almost the same performance compared to the
supervised ImageNet network. Meanwhile, it outperforms the previous supervised
and self-supervised methods on downstream tasks, which suggests CLSA learned
more reliable and fine-grained features that can contribute to the development
of other areas.

Moreover, as aforementioned comparison with contrastive loss, the DDM loss is
independent with concurrent contrastive learning methods. For contrastive loss
module in CLSA, we can use it to further improve the performance of any
contrastive learning methods, such as MoCo [5], SimCLR [8], AdCo [21] and so
on, where the performance of CLSA can be further boosted with the contrastive
baseline BYOL [20]. Furthermore, the coefficient and the temperature of
contrastive loss and distributional divergence minimization loss can be
further explored and may benefit more to learn from the strongly augmented
images.

## Acknowledgments

The authors would like to thank the support of Yuanyuan Zhang for figure
plotting.

## References

  * [1] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [2] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , vol. 2. IEEE, 2006, pp. 1735–1742.
  * [3] I. Misra and L. v. d. Maaten, “Self-supervised learning of pretext-invariant representations,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 6707–6717.
  * [4] X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” _arXiv preprint arXiv:2003.04297_ , 2020.
  * [5] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9729–9738.
  * [6] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” _arXiv preprint arXiv:2006.09882_ , 2020.
  * [7] O. Maron and T. Lozano-Pérez, “A framework for multiple-instance learning,” _Advances in neural information processing systems_ , pp. 570–576, 1998.
  * [8] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” _arXiv preprint arXiv:2002.05709_ , 2020.
  * [9] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3733–3742.
  * [10] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio, “Learning deep representations by mutual information estimation and maximization,” _arXiv preprint arXiv:1808.06670_ , 2018.
  * [11] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [12] P. Bachman, R. D. Hjelm, and W. Buchwalter, “Learning representations by maximizing mutual information across views,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 15 535–15 545.
  * [13] C. Zhuang, A. L. Zhai, and D. Yamins, “Local aggregation for unsupervised learning of visual embeddings,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 6002–6012.
  * [14] Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” _arXiv preprint arXiv:1906.05849_ , 2019.
  * [15] O. J. Hénaff, A. Srinivas, J. De Fauw, A. Razavi, C. Doersch, S. Eslami, and A. v. d. Oord, “Data-efficient image recognition with contrastive predictive coding,” _arXiv preprint arXiv:1905.09272_ , 2019.
  * [16] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, “What makes for good views for contrastive learning,” _arXiv preprint arXiv:2005.10243_ , 2020.
  * [17] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “Randaugment: Practical automated data augmentation with a reduced search space,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 702–703.
  * [18] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, “Autoaugment: Learning augmentation policies from data,” _arXiv preprint arXiv:1805.09501_ , 2018.
  * [19] X. Wang, D. Kihara, J. Luo, and G.-J. Qi, “Enaet: Self-trained ensemble autoencoding transformations for semi-supervised learning,” _arXiv preprint arXiv:1911.09265_ , 2019.
  * [20] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar _et al._ , “Bootstrap your own latent: A new approach to self-supervised learning,” _arXiv preprint arXiv:2006.07733_ , 2020.
  * [21] Q. Hu, X. Wang, W. Hu, and G.-J. Qi, “Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries,” _arXiv preprint arXiv:2011.08435_ , 2020.
  * [22] X. Wang, Z. Liu, and S. X. Yu, “Unsupervised feature learning by cross-level discrimination between instances and groups,” _arXiv preprint arXiv:2008.03813_ , 2020.
  * [23] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [24] J. Donahue, P. Krähenbühl, and T. Darrell, “Adversarial feature learning,” _arXiv preprint arXiv:1605.09782_ , 2016.
  * [25] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 132–149.
  * [26] X. Chen and K. He, “Exploring simple siamese representation learning,” _arXiv preprint arXiv:2011.10566_ , 2020.
  * [27] B. Roh, W. Shin, I. Kim, and S. Kim, “Spatially consistent representation learning,” _arXiv preprint arXiv:2103.06122_ , 2021.
  * [28] G.-J. Qi, L. Zhang, and X. Wang, “Learning generalized transformation equivariant representations via autoencoding transformations,” _arXiv preprint arXiv:1906.08628_ , 2019.
  * [29] M. Noroozi and P. Favaro, “Unsupervised learning of visual representations by solving jigsaw puzzles,” in _European conference on computer vision_. Springer, 2016, pp. 69–84.
  * [30] C. Doersch, A. Gupta, and A. A. Efros, “Unsupervised visual representation learning by context prediction,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1422–1430.
  * [31] L. Zhang, G.-J. Qi, L. Wang, and J. Luo, “Aet vs. aed: Unsupervised representation learning by auto-encoding transformations rather than data,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2547–2555.
  * [32] L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [33] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, “Autoaugment: Learning augmentation strategies from data,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 113–123.
  * [34] R. Hataya, J. Zdenek, K. Yoshizoe, and H. Nakayama, “Faster autoaugment: Learning augmentation strategies using backpropagation,” in _European Conference on Computer Vision_. Springer, 2020, pp. 1–16.
  * [35] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [36] J. Li, P. Zhou, C. Xiong, R. Socher, and S. C. Hoi, “Prototypical contrastive learning of unsupervised representations,” _arXiv preprint arXiv:2005.04966_ , 2020.
  * [37] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [38] J. Donahue and K. Simonyan, “Large scale adversarial representation learning,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 10 542–10 552.
  * [39] Y. M. Asano, C. Rupprecht, and A. Vedaldi, “Self-labelling via simultaneous clustering and representation learning,” _arXiv preprint arXiv:1911.05371_ , 2019.
  * [40] S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” _arXiv preprint arXiv:1803.07728_ , 2018\.
  * [41] S. Gidaris, A. Bursuc, N. Komodakis, P. Pérez, and M. Cord, “Learning representations by predicting bags of visual words,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 6928–6938.
  * [42] L. Bottou, “Large-scale machine learning with stochastic gradient descent,” in _Proceedings of COMPSTAT’2010_. Springer, 2010, pp. 177–186.
  * [43] I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” _arXiv preprint arXiv:1608.03983_ , 2016.
  * [44] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2,” https://github.com/facebookresearch/detectron2, 2019.

|  Xiao Wang Xiao Wang received his B.S. degree from Department of Computer
Science in Xi’an Jiaotong University, Xi’an, China in 2018. He is currently
pursuing his Ph.D. degree in the Department of Computer Science at Purdue
University, West Lafayette, IN, USA. His research interests include deep
learning, computer vision, bioinformatics and intelligent systems.
---|---
|  Guo-Jun Qi Guo-Jun Qi (M14-SM18) is the Chief Scientist leading and
overseeing an international R&D team for multiple artificial intelligent
services on the Huawei Cloud since August 2018\. He was a faculty member in
the Department of Computer Science and the director of MAchine Perception and
LEarning (MAPLE) Lab at the University of Central Florida since August 2014\.
Prior to that, he was also a Research Staff Member at IBM T.J. Watson Research
Center, Yorktown Heights, NY. His research interests include machine learning
and knowledge discovery from multi-modal data sources to build smart and
reliable information and decision-making systems. Dr. Qi has published more
than 100 papers in a broad range of venues in pattern recognition, machine
learning and computer vision. He also has served or will serve as a general
co-chair for ICME 2021, technical program co-chair for ACM Multimedia 2020,
ICIMCS 2018 and MMM 2016, as well as an area chair (senior program committee
member) for multiple academic conferences. Dr. Qi is an associate editor for
IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), IEEE
Transactions on Multimedia (T-MM), IEEE Transactions on Image Processing
(T-IP), Pattern Recognition (PR), and ACM Transactions on Knowledge Discovery
from Data (T-KDD).
---|---