# On the Discriminability of Self-Supervised Representation Learning

Zeen Song, Wenwen Qiang, Changwen Zheng, Fuchun Sun, and Hui Xiong,
Z. Song, W. Qiang, and C. Zheng are with the University of Chinese Academy of
Sciences, Beijing, China. They are also with the Science & Technology on
Integrated Information System Laboratory, Institute of Software Chinese
Academy of Sciences, Beijing, China. E-mail: songzeen, lingyu, hujie,
qiangwenwen<EMAIL_ADDRESS>Z. Song and L. Si have contributed equally
to this work.F. Sun is with the Science & Technology on Integrated Information
System Laboratory, Department of Computer Science and Technology, Tsinghua
University, Beijing, China. E-mail<EMAIL_ADDRESS>Xiong is with the
Hong Kong University of Science and Technology, China. E-mail:
<EMAIL_ADDRESS>
###### Abstract

Self-supervised learning (SSL) has recently achieved significant success in
downstream visual tasks. However, a notable gap still exists between SSL and
supervised learning (SL), especially in complex downstream tasks. In this
paper, we show that the features learned by SSL methods suffer from the
crowding problem, where features of different classes are not distinctly
separated, and features within the same class exhibit large intra-class
variance. In contrast, SL ensures a clear separation between classes. We
analyze this phenomenon and conclude that SSL objectives do not constrain the
relationships between different samples and their augmentations. Our
theoretical analysis delves into how SSL objectives fail to enforce the
necessary constraints between samples and their augmentations, leading to poor
performance in complex tasks. We provide a theoretical framework showing that
the performance gap between SSL and SL mainly stems from the inability of SSL
methods to capture the aggregation of similar augmentations and the separation
of dissimilar augmentations. To address this issue, we propose a learnable
regulator called Dynamic Semantic Adjuster (DSA). DSA aggregates and separates
samples in the feature space while being robust to outliers. Through extensive
empirical evaluations on multiple benchmark datasets, we demonstrate the
superiority of DSA in enhancing feature aggregation and separation, ultimately
closing the performance gap between SSL and SL.

###### Index Terms:

self-supervised learning, representation learning, generalization bound.

##  I Introduction

Learning discriminative feature representations in the absence of supervised
signals has long been a prominent and widely explored research area in machine
learning. Recently, self-supervised learning (SSL) has garnered significant
attention due to its remarkable performance on various downstream tasks,
including image classification [1], object detection [2], semantic
segmentation and transfer learning [3, 4, 5].

Despite the continuous improvement in SSL performance through both
augmentation-based and reconstruction-based approaches [1, 6, 7, 8, 9], a
notable gap persists between SSL and supervised learning (SL), especially in
tasks that require fine-grained discrimination. Understanding the underlying
reasons for this gap remains an ongoing challenge.

To delve deeper into this issue, we conduct a series of toy experiments to
illustrate the fundamental differences in the data characteristics learned by
supervised and self-supervised methods. Specifically, we present a comparative
analysis of five well-known SSL methods: SimCLR [1], BYOL [2], Barlow Twins
[3], SwAV [10], and MAE [8]. For comparison, we also visualize the features
learned by a supervised method.

The visualization of features learned by these methods on the ImageNet dataset
[11] is shown in Fig. 1. From Fig. 1(a) to 1(f), we observe that features
obtained by both SSL and supervised learning methods exhibit clustering
characteristics, meaning points of the same class are grouped together.
However, SSL methods display a large intra-class variance, which causes points
at the edges of different classes to overlap, a phenomenon we refer to as the
crowding problem. In contrast, supervised learning methods not only produce
fine-grained intra-class features but also maintain clear separation between
different classes.

(a) SimCLR

(b) BYOL

(c) Barlow Twins

(d) SwAV

(e) MAE

(f) Supervised

Figure 1: Data distribution visualization based on 8 random classes of the
test set of ImageNet in the feature space. (a) - (e) corresponds to the
visualization results of the self-supervised method while (f) corresponds to
the visualization results of the supervised method. We can observe that the
crowding problem is present in SimCLR, BYOL, Barlow Twins, SwAV, and MAE.

We further interpret the observations mentioned earlier from both empirical
and theoretical perspectives. For empirical analysis, we find that SSL methods
lack a specific component in their objective function that explores the
relationships between different samples. In contrast, supervised methods
leverage annotation information to encourage the clustering of samples from
the same class and the separation of samples from different classes in the
feature space. For theoretical analysis, we demonstrate that minimizing intra-
class variance and maximizing inter-class distance reduces the error risk in
supervised learning. Based on these findings, we conclude that the performance
gap between supervised learning and SSL methods may be attributed to the SSL
methods’ limited exploration of the dynamics between samples. Specifically,
the objective function of SSL methods lacks a term that constrains points of
similar semantics to be closer and points of dissimilar semantics to be
further apart in the feature space. These insights provide valuable guidance
for designing more effective regularization methods to close the performance
gap between SSL and supervised learning.

Inspired by this analysis, we propose a novel method called ”Dynamic Semantic
Adjuster” (DSA) that can be seamlessly integrated into existing SSL methods as
a learnable regulator. DSA consists of two main modules: the arranging module
and the scoring module. The arranging module aggregates similar samples while
effectively distinguishing dissimilar samples in the feature space using a
learnable regulator matrix. However, due to the poor performance of the
feature extractor in the initial stages of training, the regulator matrix may
be inaccurate, leading to incorrect aggregation and separation. To address
this, we propose the scoring module, which aims to ensure that the regulator
matrix preserves the local structure of the input sample space. This
enhancement allows the matrix to more effectively explore similarity and
dissimilarity in the early stages of training. Additionally, the scoring
module ensures that the proposed contrastive learning method remains robust
against outliers.

The main contributions are summarized as:

  * •

We show for the first time that contrastive learning suffers from the crowding
problem. We empirically analyze that the cause of the crowding problem is
absence of a term in the objective function of contrastive learning that can
explore relationships between augmentations generated from samples of
different ancestors.

  * •

We provide a theoretical analysis to demonstrate that the performance gap
between contrastive learning and supervised learning primarily stems from the
inability of contrastive learning methods to effectively capture the
aggregation and separation between augmentations that are generated by samples
of different ancestors.

  * •

We propose a novel method called ”Dynamic Semantic Adjuster” (DSA), which can
make points of the same class cluster and points of different classes separate
from each other. Notably, our method is insensitive to outliers and can be
seamlessly integrated into existing contrastive learning models, facilitating
its practical applicability.

  * •

We provide empirical evaluations to substantiate the efficacy of the proposed
DSA in improving the performance of various state-of-the-art contrastive
learning methods.

##  II Related Work

In this section, we briefly review the SSL methods, especially those that also
aim to analyze the discriminability of SSL. We also review the SSL methods
most relevant to our work and highlight the differences between these methods
and ours.

Self-Supervised learning. According to different learning paradigms, the SSL
methods can be categorized into two classes, namely the augmentation-based
methods and reconstruction-based methods.

Augmentation based.

The core idea behind augmentation-based methods is to create different views
of a single sample through data augmentation while simultaneously making the
representations of these different views more similar [12]. One of the most
successful augmentation-based methods is SimCLR [1], which utilizes the
InfoNCE loss to group views from the same sample and repel views from
different samples. One of the main drawbacks of SimCLR is the requirement for
a large number of negative samples, i.e., views from different samples.
Without sufficient negative samples, the problem of collapse can occur, where
the shared representation ignores the input and remains constant. To address
this limitation, negative-free methods are proposed. Recent methods such as
BYOL [2], SimSiam [13], and DINO [6] have shown that collapse can be avoided
within a knowledge distillation framework. Another line of negative-free
methods, such as Barlow Twins [3], W-MSE [14], and VICReg [7], propose to
prevent collapse by maximizing the information content of the embeddings.

Reconstruction based. Reconstruction-based SSL methods attempt to learn useful
representations by reconstructing the original samples. Traditionally, an
autoencoder takes the whole sample as input, feeds it into the encoder to
obtain the latent representation, and then reconstructs the sample with a
decoder. Variants of autoencoders, such as denoising autoencoders and
variational autoencoders [15], have been adopted for self-supervised learning.
Recently, inspired by the success of mask modeling in natural language
processing, the reconstruction method with mask modeling has gained
significant attention. Specifically, BEiT [16] and masked autoencoder (MAE)
[8] randomly mask the contents of visual samples and aim to reconstruct the
whole sample using only the unmasked parts.

We demonstrate in Section III that current SSL methods share a unified
objective and our proposed DSA is a plug-and-play module that can be
integrated with any SSL methods.

Discriminability Analysis. Discriminability refers to the easiness of
separating feature representations from different categories. The concept of
learning discriminative representations was first introduced by Linear
Discriminant Analysis (LDA) [17], which aims to find a projection into a
lower-dimensional space that maximizes the separation between multiple classes
while retaining as much class-discriminatory information as possible. Various
studies have investigated discriminability in the context of SSL methods. For
example, Saunshi et al. [18] connect the SSL objective with supervised
downstream error through instance discrimination. Chen et al. [19] proposed
quantifying discriminability in SSL using the min-max distance ratio. In our
research, we theoretically demonstrate that the downstream classification
error risk is bounded by intra-class variance and inter-class difference,
highlighting that the discriminability of feature representations
significantly impacts downstream performance.

SSL with Clustering.

Clustering aims to group semantically similar samples. Traditionally, this
unsupervised learning algorithm has been extensively used for small-scale,
low-dimensional unlabeled data [20]. In recent years, several studies have
integrated clustering algorithms with deep neural networks [21, 22, 23]. Our
research introduces a learnable regulator designed to aggregate similar
samples while distinguishing dissimilar ones, thus positioning our approach as
a form of clustering. Other methods that combine SSL with clustering include
SwAV [10], PCL [24], and NaCl [25]. These methods typically use a K-means-like
paradigm, treating the number of centroids as a hyper-parameter. Since we do
not know the real number of classes during the training phase, samples of
different classes may be clustered together and samples of the same class may
be clustered near different class centroids. Therefore, crowding problem can
also occur. In contrast, our method does not require a predefined number of
clusters, instead grouping similar samples in an online manner. Furthermore,
our DSA approach not only clusters similar samples but also separates
dissimilar ones.

##  III Preliminaries

In this section, we first present a unified framework for SSL methods and
briefly introduce prominent SSL approaches, including SimCLR [1], BYOL [2],
Barlow Twins [3], and Masked Autoencoder [8].

Formally, given a minibatch of training data denoted as
${X_{tr}}=\left\\{{{x_{i}}}\right\\}_{i=1}^{N}$, where ${{x_{i}}}$ represents
the $i$-th sample and $N$ denotes the number of samples in the minibatch, the
augmentation-based methods apply random data augmentations (e.g., random crop)
to transform a random sample $x_{i}$ into two augmented views $x^{1}_{i}$ and
$x^{2}_{i}$. Similarly, the reconstruction-based methods apply a random mask
to the sample $x_{i}$, creating two masked views $x_{i}^{1}$ and $x^{2}_{i}$.
The augmented dataset is denoted as
$X_{{}^{tr}}^{aug}=\left\\{{x_{1}^{1},x_{1}^{2},...,x_{N}^{1},x_{N}^{2}}\right\\}$.
The samples in ${X_{tr}}$ are considered as the ancestors of the samples in
$X_{{}^{tr}}^{aug}$. The augmented dataset $X_{{}^{tr}}^{aug}$ is then fed
into the feature extractor $f$ to obtain their feature representations, i.e.,
$r_{i}^{l}=f({x_{i}^{l}})$, where $i\in\left\\{{1,\dots,N}\right\\}$ and
$l\in\left\\{{1,2}\right\\}$. A projection head $f_{p}$ is applied to
$r_{i}^{l}$ to get the feature embedding $z_{i}^{l}$. For simplicity, we
analyze the case with only two views, although the following analysis also
applies to cases with more than two views.

The SSL objective consists of two components: alignment and constraint. The
alignment component aims to maximize the similarity between the feature
embeddings of the two views that share the same ancestor sample. The
constraint component introduces additional prior knowledge to the learning
process, such as the distribution of the embedding space and the parameter
update rules. Consequently, SSL methods can be unified under a common
framework, as expressed below:

$\min_{f,f_{p}}\mathcal{L}_{\text{align}}(X_{tr}^{aug},f,f_{p})+\mathcal{L}_{\text{constrain}}(X_{tr}^{aug},f,f_{p}),$
(1)

where $\mathcal{L}_{\text{align}}$ and $\mathcal{L}_{\text{constrain}}$ denote
the objectives of the alignment and constraint losses, which we detail in the
following analysis.

SimCLR [1] randomly selects an anchor sample $x_{i}^{l}$ from the augmented
training set $X_{{}^{tr}}^{aug}$. The sample $x_{i}^{3-l}$ is considered as
the positive sample related to $x_{i}^{l}$ for they share the same ancestor
$x_{i}$. The remaining samples
$X^{-}=X^{aug}_{tr}\setminus\\{x_{i}^{l},x_{i}^{3-l}\\}$ are considered as the
negative samples related to $x_{i}^{l}$.

The objective of SimCLR is defined as follows:

$\mathcal{L}_{\mathrm{NCE}}=\sum\limits_{x_{i}^{l}\in
X^{aug}_{tr}}-\log\frac{\exp(\mathrm{sim}(x_{i}^{l},x_{i}^{3-l})/\tau)}{\sum\limits_{x_{j}^{k}\in
X^{-}\cup\\{x_{i}^{3-l}\\}}\exp(\mathrm{sim}(x_{i}^{l},x_{j}^{k})/\tau)},$

(2)

where $\tau$ is the temperature hyperparameter. Denote
$z_{i}^{l}=f_{p}(f(x_{i}^{l}))$ as the projected feature embedding after the
projection head $f_{p}$ and feature extractor $f$, the similarity function
$\mathrm{sim}(x_{i}^{l},x_{i}^{3-l})={z_{i}^{l}}^{T}z_{i}^{3-l}/\|z_{i}^{l}\|_{2}\|z_{i}^{3-l}\|_{2}$
calculates the cosine similarity between projected feature embedding of
samples. According to [26], Equation 2 can be understood as aligning the
embedding of $x_{i}^{l}$ and $x_{i}^{3-l}$ while constraining the feature
embedding of all samples in $x_{j}^{k}\in X_{tr}^{aug}$ to satisfy a uniform
distribution.

The main idea of BYOL [2] is to only increase the similarity between pairs of
samples in $X_{{}^{tr}}^{aug}$ that share the same ancestor, without
explicitly considering negative samples. BYOL considers two networks: the
online network and the target network. The two networks have the same feature
extraction module $f$ and projection head module $f_{p}$. However, the online
network has one more regression module $f_{r}$ than the target network. The
objective of BYOL can be viewed as the mean squared error (MSE) between the
outputs of the online network and the target network, which can be presented
as:

${{\cal
L}_{{\rm{BYOL}}}}=\sum\limits_{i=1}^{N}{\sum\limits_{l=1}^{2}{\|{{f_{r}}({\bar{z}_{i}^{l}})-\bar{z}_{i}^{3-l}}\|^{2}}}$
(3)

where $z_{i}^{l}={f_{p}}({f({x_{i}^{l}})})$ and
$\bar{z}_{i}^{l}=z_{i}^{l}/\|z_{i}^{l}\|_{2}$. Note that a stop-gradient
technique is applied to the target network in the gradient back-propagation
stage. For the sake of simplicity, we denote the target network as
$f_{target}$, and the part of the online network that is similar in structure
to $f_{target}$ is denoted as $f_{online}$. Then, $f_{target}$ is updated with
a moving average of the parameters of the online network, which can be denoted
as:

${f_{{{target}}}}\leftarrow\pi{f_{{{target}}}}+(1-\pi){f_{online}}$ (4)

where $\pi\in\left[0,1\right]$ represents a target decay rate. Equation 3 can
be considered as aligning the feature embedding of different views while
Equation 4 constrains the update rule through stop-gradient and moving average
of parameters.

Barlow Twins [3] is an augmentation-based method that does not depend on a
large number of negative samples and does not require a stop-gradient
technique or asymmetric networks. It first computes the cross-correlation
matrix $C$ within a minibatch between $\left\\{{x_{i}^{1}}\right\\}_{i=1}^{N}$
and $\left\\{{x_{i}^{2}}\right\\}_{i=1}^{N}$ in the feature space. Then, we
have:

$C_{kj}=\frac{\sum_{i=1}^{N}z^{1}_{i,k}\cdot
z^{2}_{i,j}}{\sqrt{\sum_{i=1}^{N}(z^{1}_{i,k})^{2}}\cdot\sqrt{\sum_{i=1}^{N}(z^{2}_{i,j})^{2}}}$
(5)

where $z_{i}^{l}={f_{p}}({f({x_{i}^{l}})})$, $k,j\in\\{1,\dots,D\\}$, and $D$
is the dimension of $z_{i}^{l}$. Then, the objective of Barlow Twins is
presented as follows:

$\mathcal{L}_{\mathrm{BT}}=\sum_{k=1}^{D}(1-C_{kk})^{2}+\lambda\sum_{k=1}^{D}\sum_{j=1,j\neq
k}^{D}C_{kj}^{2}$ (6)

where $\lambda$ is a positive constant trading of the importance of the first
and second terms of the objective. Minimizing the first term of Equation 6
aligns the embedding of two views of $x_{i}$ while minimizing the second term
of Equation 6 constrains different vector components of these feature
embeddings to be decorrelated.

MAE [8] first reshapes the ancestor sample into $m$ patches, denoted as
$x_{i}\in\mathbb{R}^{m\times s}$, where $s$ represents the patch size (e.g.,
$16\times 16$ for an image sample). A random binary mask $m\in{0,1}^{m}$ is
applied to the sample $x_{i}$ to generate two views of $x_{i}$, where
$x_{i}^{1}=x_{i}[m]$ and $x_{i}^{2}=x_{i}[1-m]$. The MAE model includes an
encoder $f$ and a decoder $g$. The encoder $f$ takes one view, $x_{i}^{1}$, as
input and generates the feature representation $z_{i}^{l}$. The decoder $g$
then takes $z_{i}^{l}$ and the masked index $m$ as input and attempts to
reconstruct the other view, $x_{i}^{2}$. The objective of MAE can be described
with the following mean squared error loss:

$\mathcal{L}_{\mathrm{MAE}}=\sum_{i=1}^{N}\sum_{l=1}^{2}\|g(f(x_{i}^{l}))-x_{i}^{3-l}\|_{2}^{2}.$
(7)

The above objective can also applied to other reconstruction-based methods
such as BEiT [16], SimMIM [9], and iBOT [27]. It is noteworthy that if we
treat the masking strategy in reconstruction-based methods as an augmentation
technique, Equation 7 can be considered as implicitly aligning the feature
representation of $x_{i}^{l}$ and $x_{i}^{3-l}$ as suggested in [28]. Unlike
augmentation-based methods, MAE enforces no explicit constraints, and this
results in a dimensional collapse problem as suggested in [28].

Figure 2: The pipeline of the proposed DSA. First, DSA obtains the feature
representations by the $f_{p}$ and the $M^{pro}$ by the $f_{p}$. Then, DSA
obtains the $M$ by the $f_{s}$. Finally, DSA simultaneously minimizes the
${\mathcal{L}_{ctr}}$, $\mathcal{L}_{LM}^{s}$, and ${\mathcal{L}_{con}}$ to
learn the model.

##  IV Motivating example and analysis

In this section, we first describe the crowding problem with experimental
observation. Then, we propose an empirical analysis to understand why SSL
methods suffer from this problem and derive a way to solve this problem:
minimizing the intra-class variance and maximizing the inter-class separation
while performing SSL. Finally, we show through theoretical analysis that only
by minimizing the SSL loss, intra-class variance, and inter-class separability
at the same time can we better reduce the upper bound of cross-entropy loss.

###  IV-A Motivating example

To provide a clear comparison between SSL methods and supervised learning, we
conducted a series of experiments. First, we trained feature extractors using
different SSL methods: SimCLR, BYOL, Barlow Twins, SwAV, and MAE.
Additionally, we trained a feature extractor using supervised cross-entropy
loss. All feature extractors except MAE are implemented as the ResNet-50 [29],
while MAE uses ViT-B/16 [30]. They are pre-trained on the ImageNet dataset for
1000 epochs. Next, we evaluated the classification accuracies on the ImageNet
validation set following the standard protocol by training a linear classifier
on top of the feature extractor with the parameters of the feature extractor
frozen. The accuracies are presented in Table I.

We observed that the accuracies for SSL are comparable but significantly lower
than the accuracy of the supervised method. Specifically, MAE has a
significantly lower linear-eval accuracy, which is also observed by a series
of works. To further analyze the learned encoders, we employed t-SNE [31] to
visualize the data distribution of the ImageNet test data. The visualization
results are depicted in Figure 1.

In Figure 1, each point represents an individual sample embedding, with
different colors representing different categories.

We observe that embeddings with the same label cluster tightly together, while
those with different labels are clearly separated by distinct borders. This
indicates that the supervised method achieves a small intra-class variance and
a large inter-class distance. Conversely, the embeddings produced by SSL
methods, shown in Figures 1(a) to 1(e), do not exhibit clear separation, and
the intra-class variance is large. We refer to this phenomenon as the crowding
problem. Specifically, the cluster-based method SwAV (Figure 1(d))
demonstrates smaller intra-class variance, but the embeddings from different
classes still overlap. The reconstruction-based method MAE (Figure 1(e)) shows
very large intra-class variance, with embeddings from different classes
failing to form distinct clusters. This corresponds to the poor linear
evaluation performance seen in Table I. Other SSL methods, depicted in Figures
1(a), 1(c), and 1(b), exhibit some clustering characteristics, but with
relatively large intra-class variance and overlapping class borders.

We provide a quantified analysis of the intra-class variance and inter-class
distance for both SSL methods and the supervised method in Table I. In this
table, the inter-class distance is calculated as the mean
$\ell_{2}$-normalized Euclidean distance between embeddings of each class. The
mean intra-class variance is calculated as the variance of the distances
within embeddings of each class. Formally, for a test set
$D_{te}=\\{(x_{i},y_{i})\\}_{i=1}^{N}$ with $N$ pairs of samples $x$ and
labels $y$, the feature embedding of sample $x$ is obtained using the feature
extractor $f$ as $z=f(x)$. The feature embedding is then $\ell_{2}$-normalized
as $\bar{z}=z/\|z\|_{2}$. The mean embedding vector $\mu_{i}$ of class
$i\in\\{1,\dots,K\\}$ is calculated with
$\mu_{i}=\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}\bar{z}_{i,j}$, where $N_{i}$ is the
number of samples in the $i$-th class. The inter-class distance is calculated
as:$d_{\mathrm{inter}}=\frac{2}{K(K-1)}\sum_{i=1}^{K-1}\sum_{j=i+1}^{K}\|\mu_{i}-\mu_{j}\|_{2}$.The
intra-class variance of class $i$ is calculated
as:$\mathrm{Var}_{i}=\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}\|z_{i,j}-\mu_{i}\|_{2}^{2}$.The
mean intra-class variance is the average intra-class variance over all
classes, calculated as:
$\mathrm{Var}_{\mathrm{intra}}=\frac{1}{K}\sum_{i=1}^{K}Var_{i}$

TABLE I: The inter-class distance, the intra-class variance, and the linear
evaluation accuracy of SSL methods.

Method | Inter-class Dist. ($\uparrow$) | Intra-class Var. ($\downarrow$) | ACC (%)
---|---|---|---
SimCLR | 1.17 | 1.15 | 70.15
BYOL | 0.90 | 0.65 | 71.48
SwAV | 1.12 | 1.01 | 75.78
Barlow Twins | 1.06 | 1.11 | 73.97
MAE | 0.14 | 0.85 | 66.85
Supervised | 1.32 | 0.62 | 79.24

As shown in Table I, the MAE has the smallest inter-class distance and shows
no clustering characteristic in Figure 1(e), this corresponds to the poor
performance in linear-eval ACC. In contrast, the supervised method exhibits
the smallest intra-class variance and the largest inter-class distance and
also has a good performance in linear-eval ACC. Other SSL methods either have
smaller inter-class distances or larger intra-class variances, leading to sub-
optimal linear-eval ACC performance. These quantified results underscore a
significant connection between the crowding problem and downstream
performance, aligning with the observed visualization patterns. Following
this, we provide an empirical analysis of the SSL objectives introduced in
Section III.

###  IV-B Empirical analysis

To elucidate the reasons behind the crowding problem observed in self-
supervised methods, it is essential to gain an understanding of the underlying
mechanisms employed by the self-supervised approaches described in Section
III.

According to [26], minimizing Equation 2 aims to encourage the learned
representations of positive pairs to exhibit similarity, while also
constraining all samples in the feature space to be uniformly distributed on a
unit hypersphere. In the case of BYOL, minimizing Equation 3 focuses solely on
promoting similarity between the learned representations of two samples with
the same ancestor. Similarly, for Barlow-Twins, minimizing Equation 6 aims to
ensure that the learned representations of positive samples are similar to the
anchor, while also encouraging the different dimensions of the learned
representations to be independent of each other. According to [28], minimizing
Equation 7 only encourages the alignment of the representation of two samples
with the same ancestor with no additional constraints. This could lead to a
collapse problem, where the inter-class distance is significantly lower than
that of augmentation-based methods.

As we can see, these methods primarily focus on minimizing the distance
between $x^{1}_{i}$ and $x^{2}_{i}$. None of the SSL methods are explicitly
designed to constrain the relationship between $x^{l}_{i}$ and $x^{k}_{j}$,
where $i,j\in\left\\{{1,2,...,N}\right\\}$, $i\neq j$, and
$l,k\in\left\\{{1,2}\right\\}$. In contrast, supervised learning leverages
labeled data to explicitly enforce the aggregation of similar points and the
separation of different types of points. Consequently, some pairs in
$X^{aug}_{tr}$ are grouped together, while others are intentionally pushed
apart.

In order to enhance the performance of self-supervised learning methods, it is
necessary to adopt a criterion that enables the clustering of similar points
and the separation of dissimilar points. This realization leads us to propose
the following approach.

###  IV-C Theoretical Analysis

To illustrate why we should minimize the intra-class variance and maximize the
inter-class separation while performing contrastive learning, we first give an
assumption on the label consistency between pairs in $X^{aug}_{tr}$ with the
same ancestor.

######  Assumption IV.1.

$\forall\left\\{{x_{i}^{1},x_{i}^{2}}\right\\}\in X_{tr}^{aug}$, assume the
labels are deterministic (one-hot) and consistent:
$p\left({y\left|{x_{i}^{1}}\right.}\right)=p\left({y\left|{x_{i}^{2}}\right.}\right)$.

Then, we aim to bound the generalization gap between unsupervised and
supervised learning risks based on the classification task, which trains a
softmax classifier by minimizing the mean cross entropy loss, e.g.,
$\mathcal{L}_{CE}^{\mu}(f)={\mathbbm{E}_{p({x,y})}}[{-\log{{\exp({f{{(x)}^{\rm{T}}}{\mu_{y}}})}\mathord{\left/{\vphantom{{\exp({f{{(x)}^{\rm{T}}}{\mu_{y}}})}{\sum\nolimits_{i=1}^{K}{\exp({f{{(x)}^{\rm{T}}}{\mu_{i}}})}}}}\right.\kern-1.2pt}{\sum\nolimits_{i=1}^{K}{\exp({f{{(x)}^{\rm{T}}}{\mu_{i}}})}}}}]$,
where ${p\left({x,y}\right)}$ represents the joint distribution of the input
sample and the corresponding label, $K$ represents the number of the class,
${\mu_{i}}={\mathbbm{E}_{p\left({x\left|y\right.}\right)}}\left[{f\left(x\right)}\right]$
is the cluster center of the $i$-th class and can be seen as the weight
$w_{i}$ of the linear classifier $g$, and ${p\left({x\left|y\right.}\right)}$
is the conditional distribution. Then, we have:

######  Theorem IV.1.

If Assumption IV.1 holds, then, for any $f\in\mathcal{F}$,
$\mathcal{L}_{CE}^{\mu}\left(f\right)$ can be bounded by $\mathcal{L}_{\rm
NCE}^{\mu}\left(f\right)$ as:

$\begin{array}[]{l}\mathcal{L}_{CE}^{\mu}\left(f\right)\leq{\mathcal{L}_{{\rm{NCE}}}}\left(f\right)-const+\sum\limits_{i=1,j=1,i\neq
j}^{K}{\mu_{i}^{\rm{T}}{\mu_{j}}}\\\
\quad\quad\quad\quad\quad+\sqrt{{\rm{Var}}\left({f\left(x\right)\left|y\right.}\right)}+O\left({cons{t^{{{-1}\mathord{\left/{\vphantom{{-1}2}}\right.\kern-1.2pt}2}}}}\right)\end{array}$
(8)

where const is a constant that is related to the number of negative samples,
${\mu_{i}}={\mathbbm{E}_{p\left({x\left|i\right.}\right)}}\left[{f\left(x\right)}\right]$,
and
${\rm{Var}}\left({f\left(x\right)\left|y\right.}\right)={\mathbbm{E}_{p\left(y\right)}}\left[{{\mathbbm{E}_{p\left({x\left|y\right.}\right)}}{{\left\|{f\left(x\right)-{\mathbbm{E}_{p\left({x\left|y\right.}\right)}}f\left(x\right)}\right\|}^{2}}}\right]$.

The detailed proof is illustrated in Appendix A. From empirical analysis and
theoretical analysis, we can obtain that only by minimizing the contrastive
loss, intra-class variance, and inter-class difference at the same time, can
we reduce the upper bound of cross-entropy loss, thus, can we compress the
upper bound of classification error. The empirical analysis and the
theoretical analysis provide us with insights to design a new contrastive
learning objective function, that is, to design an objective function that can
measure intra-class variance and inter-class difference in an unsupervised
way. In this paper, We propose DSA to aggregate similar points together and
separate different points from each other. Therefore, the proposed DSA is
beneficial to compress the upper bound of
$\mathcal{L}_{CE}^{\mu}\left(f\right)$.

##  V Methodology

In this section, we describe our proposed method called Dynamic Semantic
Adjuster (DSA). DSA can be easily integrated into existing self-supervised
learning methods. The core concept of DSA is to attract samples with similar
semantics while repelling others in an instance-based manner. Specifically,
DSA first randomly selects a point from the training dataset as an anchor
point, and then based on the arranging module and the scoring module,
aggregates the points in the training dataset that are similar to the anchor
point and separates the points that are dissimilar to the anchor point. Then
DSA traverses all the points in the training dataset and treats them as anchor
points, thus achieving the purpose of aggregating similar points and
separating dissimilar points in the entire training dataset. Note that similar
points are more likely to be of the same class, and dissimilar points are more
likely to be of different classes. Therefore, DSA can induce a clustering
structure and make points of the same class cluster together, thereby reducing
intra-class variance, and make points of different classes separate from each
other, thereby enhancing inter-class separation.

The overall pipeline of DSA is illustrated in Figure 2, it consists of two
main components. The first component is the arranging module, which groups
samples with similar semantics together and pushes away those with dissimilar
semantics. The second component is the scoring module, which is to further
revise the semantic similarity between samples in the arranging module through
the relative position structure between samples in the original space.

###  V-A Arranging module

The arranging module (AM) is designed to learn a regulator $M$, which guides
the arrangement of samples in the feature space. The goal of the regulator $M$
is to bring samples with high similarity closer to each other while separating
samples with low similarity. Formally, the regulator $M$ can be interpreted as
a similarity matrix with a size of $2N\times 2N$.

Specifically, we first project each sample in
$X_{{}^{tr}}^{aug}=\left\\{{x_{1}^{1},x_{1}^{2},...,x_{N}^{1},x_{N}^{2}}\right\\}$
to the feature space via $f$ and obtain their feature representations, denoted
as
$Z_{{}^{tr}}^{aug}=\left\\{{z_{1}^{1},z_{1}^{2},...,z_{N}^{1},z_{N}^{2}}\right\\}$,
where $z_{i}^{l}=f\left({x_{i}^{l}}\right)$, $i\in\left\\{{1,...,N}\right\\}$,
and $l\in\left\\{{1,2}\right\\}$. Regarding the $i$-th sample $z_{i}$ in
$Z_{{}^{tr}}^{aug}$ as the anchor, then, for the $j$-th sample $z_{j}$ in
$Z_{{}^{tr}}^{aug}$, we concatenate them and obtain a new representation
$z_{i,j}$, e.g., ${z_{i,j}}=cat\left({{z_{i}},{z_{j}}}\right)$. Subsequently,
we feed ${z_{i,j}}$ into a similarity network $f_{s}$, which consists of a
two-layer MLP with ReLU activation function. The selection of $f_{s}$ is
evaluated and discussed through an ablation study in Section VI-F. The
similarity network computes the similarity value between $z_{i}$ and $z_{j}$
and assigns it to the corresponding entry $M_{i,j}$ in the regulator matrix
$M$, i.e., $M_{i,j}=f_{s}(z_{i,j})$. Then, based on the anchor $z_{i}$, we can
obtain:

${M_{i}}=\left[{{{\bar{f}}_{s}}\left({{z_{i,1}}}\right),...,{{\bar{f}}_{s}}\left({{z_{i,2N}}}\right)}\right]$
(9)

where
${{\bar{f}}_{s}}\left({{z_{i,j}}}\right)={{{f_{s}}\left({{z_{i,j}}}\right)}\mathord{\left/{\vphantom{{{f_{s}}\left({{z_{i,j}}}\right)}{\sum\nolimits_{k=1}^{k=2N}{{f_{s}}\left({{z_{i,k}}}\right)}}}}\right.\kern-1.2pt}{\sum\nolimits_{k=1}^{k=2N}{{f_{s}}\left({{z_{i,k}}}\right)}}}$
and $j\in\left\\{{1,...,2N}\right\\}$. We in turn treat the samples in
$Z_{{}^{tr}}^{aug}$ as anchors and obtain
$M={\left[{{M_{1}},....,{M_{2N}}}\right]^{\rm{T}}}$.

Once the regulator matrix $M$ is obtained, we propose to minimize the
following loss function:

$\begin{array}[]{l}{\mathcal{L}_{AM}}=\log({1+\sum\limits_{i=1}^{2N}{\sum\limits_{j=1}^{2N}{\exp({\frac{{({2{M_{i,j}}-\alpha})\|{{z_{i}}-{z_{j}}}\|_{2}^{2}}}{\tau}})}}})\end{array}$
(10)

where $\tau,\alpha>0$ are two temperature hyperparameters. Note that
$\mathcal{L}_{AM}$ captures not only the relationship between different
augmentations of the same ancestor but also the relationship between different
augmentations of different ancestors.

Given $\alpha$ and $M_{i,j}$, when $2M_{i,j}-\alpha>0$, we consider that the
samples $z_{i}$ and $z_{j}$ in $Z_{{}^{tr}}^{aug}$ are semantically similar
and should be brought closer together. From Equation (10), we can obtain that
when $2M_{i,j}-\alpha>0$, minimizing
${\left\|{{z_{i}}-{z_{j}}}\right\|_{2}^{2}}$ can minimize
${\mathcal{L}_{LM}}$. Also, to minimize ${\mathcal{L}_{LM}}$, the greater the
value of $2M_{i,j}-\alpha$, the closer we should pull $z_{i}$ and $z_{j}$.
Therefore, we can conclude that minimizing ${\mathcal{L}_{LM}}$ can bring
different pairs closer to different degrees according to the value of
$M_{i,j}$.

###  V-B Scoring module

As illustrated in the above subsection, $M_{i,j}$ can control the dynamics
(move closer to or farther away from the anchor) of a sample. However, $M$ is
obtained by inputting $z_{i,j}$ into $f_{s}$. Without proper constraints,
$f_{s}$ may output undesirable values. For example, in the initial stage of
training, the performance of $f_{s}$ is often poor, which leads to the learned
regulator matrix being inaccurate, causing the sample points in the training
dataset to be incorrectly aggregated and separated. To alleviate this problem,
we propose a scoring module (SM) that can revise the output of $f_{s}$ based
on the similarity prior of the samples in the original input space.

Specifically, we first input samples in $X^{aug}_{tr}$ into an auxiliary
feature extractor $f_{aux}$ to obtain prior feature representations, denoted
as
$\bar{Z}_{tr}^{aug}=\\{{\bar{z}_{1}^{1},\bar{z}_{1}^{2},...,\bar{z}_{2N}^{1},\bar{z}_{2N}^{2}}\\}$,
where $\bar{z}_{i}^{l}={f_{aux}}({x_{i}^{l}})$, $i\in\\{{1,...,2N}\\}$, and
$l\in\\{{1,2}\\}$. Then, considering the $i$-th element ${{{\bar{z}}_{i}}}$ in
$\bar{Z}_{tr}^{aug}$ as the anchor, we define:

$M_{i,j}^{pro}=\exp\left({{{\left\|{{{\bar{z}}_{i}}-{{\bar{z}}_{j}}}\right\|_{2}^{2}}\mathord{\left/{\vphantom{{\left\|{{{\bar{z}}_{i}}-{{\bar{z}}_{j}}}\right\|_{2}^{2}}\tau}}\right.\kern-1.2pt}\tau}}\right)$
(11)

where ${{\bar{z}}_{j}}$ is the $j$-th sample in $\bar{Z}_{tr}^{aug}$. Then, we
can obtain:

$M_{i}^{pro}=\left[{\bar{M}_{i,1}^{pro},...,\bar{M}_{i,2N}^{pro}}\right]$ (12)

where
$\bar{M}_{i,k}^{pro}={{M_{i,k}^{pro}}\mathord{\left/{\vphantom{{M_{i,k}^{pro}}{\sum\nolimits_{j=1}^{j=2N}{M_{i,j}^{pro}}}}}\right.\kern-1.2pt}{\sum\nolimits_{j=1}^{j=2N}{M_{i,j}^{pro}}}}$
and $k\in\left\\{{1,...,2N}\right\\}$. We in turn treat the samples in
${{{\bar{z}}_{i}}}$ as anchors and obtain
${M^{pro}}={\left[{M_{1}^{pro},...,M_{2N}^{pro}}\right]^{\rm{T}}}$. To this
end, we give the prior constraint ${M^{pro}}$ of $M$ based on the Euclidean
distance of different pairs of samples in the original input space. We
constrain $M$ as follows:

${\mathcal{L}_{con}}=\left\|{{M^{pro}}-M}\right\|_{2}^{2}$ (13)

By simultaneously minimizing Equation (13) and Equation (10), the points in
the original input space that are closer to the anchor become attracted to a
closer location in the learned feature space, while the points that are
farther from the anchor are repelled to a more distant location in the feature
space.

To assess the quality of each anchor, we propose a scoring mechanism based on
the connectivity between samples. Given the $i$-th element ${{{z}_{i}}}$ in
$Z_{tr}^{aug}$, we find its corresponding element ${{{\bar{z}}_{i}}}$ in
$\bar{Z}_{tr}^{aug}$. We then identify the $\eta$ nearest neighbors of it in
$\bar{Z}_{tr}^{aug}$ based on Euclidean distance and denote them as
$N_{\eta}({{{\bar{z}}_{i}}})$. The connection score $s_{i}$ for anchor
${{{z}_{i}}}$ is defined as:

${s_{i}}\left({{{z}_{i}}}\right)=\sum\limits_{\bar{z}_{i}^{b}\sim{N_{\eta}}({\bar{z}_{i}^{a}}),\bar{z}_{i}^{a}\sim{N_{\eta}}({{{\bar{z}}_{i}}})}{{\mathbbm{1}_{\\{{{{\bar{z}}_{i}}\in{N_{\eta}}({\bar{z}_{i}^{b}})}\\}}}}$
(14)

Then, we score the quality of ${{{z}_{i}}}$ as:

$sc\left({{{z}_{i}}}\right)={{{s_{i}}\left({{{z}_{i}}}\right)}\mathord{\left/{\vphantom{{{s_{i}}\left({{{\bar{z}}_{i}}}\right)}\eta}}\right.\kern-1.2pt}\eta}$
(15)

The scoring mechanism in Equation (14) takes into account both direct
connections between the anchor $\bar{z}_{i}$ and $N_{\eta(\bar{z}_{i})}$ as
well as latent connections induced by other samples along the connected path
[32]. From Equation (15), we can deduce that when ${{{z}_{i}}}$ is surrounded
by points of the same class, ${s_{i}}\left({{{z}_{i}}}\right)$ should be equal
to or slightly smaller than $\eta$. Consequently, the score
$sc\left({{{z}_{i}}}\right)$ is equal to or slightly smaller than 1, and when
${{{z}_{i}}}$ is an outlier, ${s_{i}}\left({{{z}_{i}}}\right)$ should be close
to 0, which is much smaller than $\eta$, so that $sc\left({{{z}_{i}}}\right)$
is also close to 0. Therefore, to reduce the influence of bad anchors, we
weight Equation (10) and obtain the following:

$\mathcal{L}_{AM}^{s}=\log({1+\sum\limits_{i=1}^{2N}{({sc({{{z}_{i}}})\cdot\sum\limits_{j=1}^{2N}{\exp({\frac{{({2{M_{i,j}}-\alpha})\|{{z_{i}}-{z_{j}}}\|_{2}^{2}}}{\tau}})}})}})$

(16)

###  V-C Overall objective

To this end, the objective of the proposed DSA is shown as:

$\mathcal{L}_{DSA}^{s}={\mathcal{L}_{ssl}}+\nu\mathcal{L}_{AM}^{s}+\upsilon{\mathcal{L}_{con}}$
(17)

where ${\mathcal{L}_{ssl}}$ is the loss of contrastive learning, e.g., we can
set ${\mathcal{L}_{ssl}}$ equals to ${\mathcal{L}_{\rm NCE}}$,
${\mathcal{L}_{\rm BYOL}}$, ${\mathcal{L}_{\rm BT}}$, or $\mathcal{L}_{\rm
MAE}$, and $\nu,\upsilon>0$ represent the temperature hyperparameters.

First, minimizing the second term of equation (17) can aggregate similar
samples and separate dissimilar samples according to the similarity of sample
points. Second, minimizing the second term of the equation (17) can eliminate
the outliers in the training samples, so that the process of aggregating and
separating samples is not affected by outliers. Minimizing the third term of
equation (17) can make the similarity between samples obtained by a more
accurate, thereby making a better at aggregating and separating samples.
Combined with Theorem IV.1, minimizing the second and third terms of the
equation (17) at the same time can not only reduce intra-class variance but
also improve inter-class separability, thus, can compress the upper bound of
classification error.

##  VI Experiment

In this section, we assess the performance of our proposed DSA through
extensive experiments. First, we briefly introduced the benchmark datasets for
evaluation. Then, we detail the implementation of the DSA pre-training
process, and we evaluate its performance on standard benchmarks for self-
supervised image and video representation learning. To demonstrate the
superior generalization capabilities of our method, we also apply DSA to
various downstream tasks, including semi-supervised classification, object
detection, few-shot learning and semantic segmentation. Finally, we present a
comprehensive ablation study on hyper-parameters, module design, and
complexity, along with case studies for visualization.

###  VI-A Benchmark dataset

Our experiments involved various datasets across different tasks. The pre-
training dataset for image data is ImageNet[11], which contains 1.3 million
training images across 1000 classes. The pre-training dataset for video data
is the Kinetics 400 dataset [33], which contains 400 human action classes,
with at least 400 video clips for each action. Each clip lasts around 10s and
is taken from different YouTube videos. The downstream object detection and
instance segmentation for image data are evaluated on the MS-COCO [34] and
Pascal-VOC [35] dataset. MS-COCO is a large-scale object detection,
segmentation, key-point detection, and captioning dataset consisting of 328K
images. The Pascal-VOC dataset includes 1464 training images and 1449
validation images, annotated with object segmentation, bounding boxes, and
class labels. It covers 20 different object categories.

The few-shot classification is evaluated on few-shot learning benchmarks:
FC100 [36], Caltech-UCSD Birds (CUB-200) [37], and Plant Disease [38]. FC-100
is a split dataset based on CIFAR-100[39] and designed for few-shot learning
by split training, validation, and testing classes to minimize the information
overlap between splits. CUB-200 consists of 6033 bird images classified into
200 bird species. Plant Disease is a public dataset containing 54306 images of
diseased and healthy plant leaves, the label contains 14 crop species and 26
diseases.

###  VI-B Pre-training Details

The SSL pre-training for image data follows the standard protocol using the
Lightly-SSL framework [40] on the ImageNet [11] dataset. The pre-training
epoch on ImageNet is 1000 epochs. We use a learning rate warm-up for the first
500 iterations of the optimizer and a 0.2 learning rate drop at 50 and 25
epochs before the end. Our proposed method is integrated with benchmark SSL
algorithms with either ResNet-50 or ViT-B as the backbone network. The SSL
pre-training for video data follows the standard protocol as introduced in
[41] on the Kinetics 400 dataset [33]. The pre-training backbone networks
include R3D-18, R3D-50, and ViT-B. We integrate our method with four well-
known video SSL algorithms: v-MoCo, v-BYOL, v-SimCLR, and v-SwAV. We evaluate
the video SSL algorithm on two backbones, R3D-18[42] and R3D-50[43]. The
default training epoch is 200 while the default batch size is 256. For the
default hyperparameters of DSA, we set the number of neighbors $\eta=20$, the
hyperparameter $\alpha=1.0$, $\nu=0.1$, and $\upsilon=100$. We conduct a
detailed ablation study of these hyper-parameters in Section VI-F.

TABLE II: The Top-1 and Top-5 classification accuracies of a linear classifier
on ImageNet with ResNet-50 as the feature extractor.

Method | Backbone | Top-1 | Top-5
---|---|---|---
SimCLR [1] | ResNet-50 | 70.15 $\pm$ 0.16 | 89.75 $\pm$ 0.14
MoCo [44] | ResNet-50 | 72.80 $\pm$ 0.12 | 91.64 $\pm$ 0.11
BYOL [2] | ResNet-50 | 71.48 $\pm$ 0.15 | 92.32 $\pm$ 0.14
SimSiam [13] | ResNet-50 | 73.01 $\pm$ 0.21 | 92.61 $\pm$ 0.27
Barlow Twins [3] | ResNet-50 | 73.97 $\pm$ 0.23 | 92.91 $\pm$ 0.19
SwAV [10] | ResNet-50 | 75.78 $\pm$ 0.16 | 92.86 $\pm$ 0.15
DINO [6] | ResNet-50 | 75.43 $\pm$ 0.18 | 93.32 $\pm$ 0.19
W-MSE [14] | ResNet-50 | 76.01 $\pm$ 0.27 | 93.12 $\pm$ 0.21
RELIC v2 [45] | ResNet-50 | 75.88 $\pm$ 0.15 | 93.52 $\pm$ 0.13
LMCL [46] | ResNet-50 | 75.89 $\pm$ 0.19 | 92.89 $\pm$ 0.28
ReSSL [47] | ResNet-50 | 75.77 $\pm$ 0.21 | 92.91 $\pm$ 0.27
SSL-HSIC [48] | ResNet-50 | 74.99 $\pm$ 0.19 | 93.01 $\pm$ 0.20
CorInfoMax [49] | ResNet-50 | 75.54 $\pm$ 0.20 | 92.23 $\pm$ 0.25
MEC [50] | ResNet-50 | 75.38 $\pm$ 0.17 | 92.84 $\pm$ 0.20
VICRegL [51] | ResNet-50 | 75.96 $\pm$ 0.19 | 92.97 $\pm$ 0.26
SimCLR + DSA | ResNet-50 | 72.09 $\pm$ 0.16 | 91.39 $\pm$ 0.13
MoCo + DSA | ResNet-50 | 74.79 $\pm$ 0.69 | 93.65 $\pm$ 0.74
SimSiam + DSA | ResNet-50 | 74.09 $\pm$ 0.89 | 94.24 $\pm$ 0.82
Barlow Twins + DSA | ResNet-50 | 76.03 $\pm$ 0.49 | 93.95 $\pm$ 0.15
SwAV + DSA | ResNet-50 | 77.84 $\pm$ 0.32 | 94.05 $\pm$ 0.89
DINO + DSA | ResNet-50 | 76.50 $\pm$ 0.78 | 94.46 $\pm$ 0.62
BYOL + DSA | ResNet-50 | 74.95 $\pm$ 0.57 | 94.93 $\pm$ 0.16
ReSSL + DSA | ResNet-50 | 77.45 $\pm$ 0.33 | 94.23 $\pm$ 0.23
VICRegL + DSA | ResNet-50 | 78.15 $\pm$ 0.63 | 94.04 $\pm$ 0.20
MAE [8] | ViT-B | 66.85 $\pm$ 0.23 | 85.24 $\pm$ 0.43
U-MAE [28] | ViT-B | 70.46 $\pm$ 0.18 | 91.25 $\pm$ 0.28
MoCo-v3 [52] | ViT-B | 76.47 $\pm$ 0.14 | 93.76 $\pm$ 0.46
DINO [6] | ViT-B | 78.17 $\pm$ 0.57 | 96.14 $\pm$ 0.17
MAE + DSA [8] | ViT-B | 72.76 $\pm$ 0.41 | 89.36 $\pm$ 0.31
MoCo-v3 + DSA [52] | ViT-B | 78.24 $\pm$ 0.72 | 95.75 $\pm$ 0.23
DINO + DSA [6] | ViT-B | 79.20 $\pm$ 0.42 | 97.16 $\pm$ 0.66

TABLE III: Linear evaluation results on the Kinetics-400 dataset, where $\varrho$, $L$, and $\delta$ denote the number of positive samples, clip length, and stride, respectively. Method | $\varrho$ | $L$ | $\delta$ | Top-1
---|---|---|---|---
supervised | - | 8 | 8 | 74.7
v-SimCLR [41] | 2 | 8 | 8 | 60.5
v-BYOL [41] | 2 | 8 | 8 | 65.8
v-MoCo [41] | 2 | 8 | 8 | 65.8
v-SwAV [41] | 2 | 8 | 8 | 61.6
v-SimCLR + DSA | 2 | 8 | 8 | 62.5
v-BYOL + DSA | 2 | 8 | 8 | 66.7
v-MoCo + DSA | 2 | 8 | 8 | 67.3
v-SwAV + DSA | 2 | 8 | 8 | 62.8
TABLE IV: The semi-supervised learning accuracies (%) on ImageNet using 1% and
10% training examples. dataset with the ResNet-50 pre-trained on the Imagenet
dataset.

Method | 1% | 10%
---|---|---
Top-1 | Top-5 | Top-1 | Top-5
Supervised | 25.1 $\pm$ 1.3 | 48.6 $\pm$ 0.7 | 55.9 $\pm$ 0.7 | 81.2 $\pm$ 0.8
SimCLR [1] | 48.3 $\pm$ 0.2 | 75.5 $\pm$ 0.1 | 65.6 $\pm$ 0.1 | 87.8 $\pm$ 0.2
MoCo [44] | 52.3 $\pm$ 0.1 | 77.9 $\pm$ 0.2 | 68.4 $\pm$ 0.1 | 88.0 $\pm$ 0.2
BYOL [2] | 56.3 $\pm$ 0.2 | 79.6 $\pm$ 0.2 | 69.7 $\pm$ 0.2 | 89.3 $\pm$ 0.1
SimSiam [13] | 54.9 $\pm$ 0.2 | 79.5 $\pm$ 0.2 | 68.0 $\pm$ 0.1 | 89.0 $\pm$ 0.3
Barlow Twins [3] | 55.0 $\pm$ 0.1 | 79.2 $\pm$ 0.1 | 67.7 $\pm$ 0.2 | 89.3 $\pm$ 0.2
RELIC v2 [45] | 55.2 $\pm$ 0.2 | 80.0 $\pm$ 0.1 | 68.0 $\pm$ 0.2 | 88.9 $\pm$ 0.2
LMCL [46] | 54.8 $\pm$ 0.2 | 79.4 $\pm$ 0.2 | 70.3 $\pm$ 0.1 | 89.9 $\pm$ 0.2
ReSSL [47] | 55.0 $\pm$ 0.1 | 79.6 $\pm$ 0.3 | 69.9 $\pm$ 0.1 | 89.7 $\pm$ 0.1
SSL-HSIC [48] | 55.1 $\pm$ 0.3 | 79.6 $\pm$ 0.2 | 70.4 $\pm$ 0.1 | 90.0 $\pm$ 0.1
CorInfoMax [49] | 55.0 $\pm$ 0.2 | 79.6 $\pm$ 0.3 | 70.3 $\pm$ 0.2 | 89.3 $\pm$ 0.2
MEC [50] | 54.8 $\pm$ 0.1 | 79.4 $\pm$ 0.2 | 70.0 $\pm$ 0.1 | 89.1 $\pm$ 0.1
VICRegL [51] | 54.9 $\pm$ 0.1 | 79.6 $\pm$ 0.2 | 67.2 $\pm$ 0.1 | 89.4 $\pm$ 0.2
SimCLR + DSA | 49.9 $\pm$ 0.2 | 77.3 $\pm$ 0.1 | 66.6 $\pm$ 0.1 | 89.3 $\pm$ 0.2
MoCo + DSA | 54.1 $\pm$ 0.3 | 80.0 $\pm$ 0.3 | 69.9 $\pm$ 0.3 | 90.0 $\pm$ 0.2
BYOL + DSA | 57.5 $\pm$ 0.5 | 81.1 $\pm$ 0.1 | 71.7 $\pm$ 0.4 | 91.0 $\pm$ 0.3
Barlow Twins + DSA | 57.0 $\pm$ 0.1 | 80.9 $\pm$ 0.1 | 69.4 $\pm$ 0.5 | 90.7 $\pm$ 0.1

TABLE V: Finetuning Results (average of 3 splits) for action classification on
UCF101 and HMDB51. Self-supervised pre-training is done on Kinetics 400
datasets. $\varrho$ is the number of positive samples.

Method | Resolution | Frames | Architecture | Param. | Epochs | UCF101 | HMDB51
---|---|---|---|---|---|---|---
VTHCL [53] | 224$\times$224 | 8 | R3D-18 | 13.5M | 200 | 80.6 | 48.6
TCLR [54] | 112$\times$112 | 16 | R3D-18 | 13.5M | 100 | 85.4 | 55.4
VideoMoCo [55] | 112$\times$112 | 16 | R3D-18 | 13.5M | 200 | 74.1 | 43.6
SLIC [56] | 128$\times$128 | 32 | R3D-18 | 13.5M | 150 | 83.2 | 52.2
MACLR [57] | 112$\times$112 | 32 | R3D-18 | 13.5M | 600 | 91.3 | 62.1
v-BYOLϱ=4 | 112$\times$112 | 16 | R3D-18 | 13.5M | 200 | 88.3 | 69.3
v-BYOLϱ=4 \+ DSA | 112$\times$112 | 16 | R3D-18 | 13.5M | 200 | 92.7 | 70.5
CVRL [58] | 224$\times$224 | 32 | R3D-50 | 31.8M | 800 | 92.2 | 66.7
MACLR [57] | 224$\times$224 | 32 | R3D-50 | 31.8M | 600 | 94.0 | 67.4
v-SimCLRϱ=2 [41] | 224$\times$224 | 8 | R3D-50 | 31.8M | 200 | 88.9 | 67.2
v-SwAVϱ=2 [41] | 224$\times$224 | 8 | R3D-50 | 31.8M | 200 | 87.3 | 68.3
v-MoCoϱ=4 [41] | 224$\times$224 | 8 | R3D-50 | 31.8M | 200 | 93.5 | 71.6
v-BYOLϱ=4 [41] | 224$\times$224 | 8 | R3D-50 | 31.8M | 200 | 94.2 | 72.1
v-SimCLRϱ=2 \+ DSA | 224$\times$224 | 8 | R3D-50 | 31.8M | 200 | 90.7 | 69.5
v-SwAVϱ=2 \+ DSA | 224$\times$224 | 8 | R3D-50 | 31.8M | 200 | 91.1 | 69.7
v-MoCoϱ=4 \+ DSA | 224$\times$224 | 8 | R3D-50 | 31.8M | 200 | 94.9 | 73.7
v-BYOLϱ=4 \+ DSA | 224$\times$224 | 8 | R3D-50 | 31.8M | 200 | 95.4 | 73.6

TABLE VI: Few-shot learning accuracies (%) with 95% confidence intervals averaged over 2000 episodes on FC100, CUB200, and Plant Disease. $(N,K)$ denotes $N$-way $K$-shot tasks. Method | FC100 | CUB200 | Plant Disease
---|---|---|---
(5,1) | (5,5) | (5,1) | (5,5) | (5,1) | (5,5)
Supervised | 34.62 $\pm$ 0.88 | 47.15 $\pm$ 0.23 | 47.34 $\pm$ 0.66 | 64.87 $\pm$ 0.21 | 70.08 $\pm$ 0.24 | 89.83 $\pm$ 0.41
SimCLR [1] | 39.89 $\pm$ 0.28 | 49.83 $\pm$ 0.70 | 44.47 $\pm$ 0.76 | 67.33 $\pm$ 0.65 | 76.94 $\pm$ 0.56 | 89.73 $\pm$ 0.67
Barlow Twins [3] | 40.02 $\pm$ 0.73 | 52.69 $\pm$ 0.11 | 46.02 $\pm$ 0.21 | 65.06 $\pm$ 0.54 | 79.37 $\pm$ 0.11 | 89.53 $\pm$ 0.42
BYOL [3] | 36.41 $\pm$ 0.43 | 51.36 $\pm$ 0.59 | 44.38 $\pm$ 0.33 | 62.75 $\pm$ 0.27 | 79.41 $\pm$ 0.24 | 91.05 $\pm$ 0.31
W-MSE [14] | 40.97 $\pm$ 0.62 | 53.69 $\pm$ 0.56 | 49.15 $\pm$ 0.63 | 65.15 $\pm$ 0.74 | 75.52 $\pm$ 0.24 | 89.05 $\pm$ 0.64
VICRegL [51] | 40.78 $\pm$ 0.48 | 54.28 $\pm$ 0.66 | 49.44 $\pm$ 0.82 | 67.18 $\pm$ 0.61 | 79.36 $\pm$ 0.22 | 92.84 $\pm$ 0.24
SimSiam [13] | 37.11 $\pm$ 0.86 | 51.64 $\pm$ 0.32 | 46.28 $\pm$ 0.68 | 63.25 $\pm$ 0.14 | 76.83 $\pm$ 0.16 | 90.45 $\pm$ 0.75
ReSSL [47] | 37.92 $\pm$ 0.64 | 52.35 $\pm$ 0.43 | 46.42 $\pm$ 0.42 | 63.76 $\pm$ 0.45 | 77.25 $\pm$ 0.86 | 91.15 $\pm$ 0.76
SwAV [10] | 39.64 $\pm$ 0.11 | 51.83 $\pm$ 0.55 | 47.34 $\pm$ 0.16 | 65.24 $\pm$ 0.75 | 79.41 $\pm$ 0.35 | 92.62 $\pm$ 0.61
SimCLR + DSA | 43.04 $\pm$ 0.37 | 52.38 $\pm$ 0.15 | 46.96 $\pm$ 0.12 | 70.52 $\pm$ 0.37 | 78.31 $\pm$ 0.71 | 91.23 $\pm$ 0.82
Barlow Twins + DSA | 41.42 $\pm$ 0.88 | 55.47 $\pm$ 0.67 | 47.36 $\pm$ 0.39 | 68.04 $\pm$ 0.32 | 82.18 $\pm$ 0.73 | 90.95 $\pm$ 0.18
BYOL + DSA | 39.85 $\pm$ 0.61 | 52.95 $\pm$ 0.46 | 46.13 $\pm$ 0.69 | 64.79 $\pm$ 0.76 | 82.89 $\pm$ 0.74 | 94.72 $\pm$ 0.89
SwAV + DSA | 41.03 $\pm$ 0.21 | 54.12 $\pm$ 0.56 | 49.36 $\pm$ 0.10 | 68.27 $\pm$ 0.55 | 81.34 $\pm$ 0.68 | 93.20 $\pm$ 0.26
TABLE VII: The results of transfer learning on object detection and instance
segmentation with C4-backbone as the feature extractor. “AP” is the average
precision, “$\text{AP}_{N}$” represents the average precision when the IoU
(Intersection and Union Ratio) threshold is $N\%$.

Method | VOC 07 detection | VOC 07+12 detection | COCO detection | COCO instance segmentation
---|---|---|---|---
$\mathbf{AP_{50}}$ | $\mathbf{AP}$ | $\mathbf{AP_{75}}$ | $\mathbf{AP_{50}}$ | $\mathbf{AP}$ | $\mathbf{AP_{75}}$ | $\mathbf{AP_{50}}$ | $\mathbf{AP}$ | $\mathbf{AP_{75}}$ | $\mathbf{AP^{mask}_{50}}$ | $\mathbf{AP^{mask}}$ | $\mathbf{AP^{mask}_{75}}$
Supervised | 74.4 | 42.4 | 42.7 | 81.3 | 53.5 | 58.8 | 58.2 | 38.2 | 41.2 | 54.7 | 33.3 | 35.2
SimCLR [1] | 75.9 | 46.8 | 50.1 | 81.8 | 55.5 | 61.4 | 57.7 | 37.9 | 40.9 | 54.6 | 33.3 | 35.3
MoCo [44] | 77.1 | 46.8 | 52.5 | 82.5 | 57.4 | 64.0 | 58.9 | 39.3 | 42.5 | 55.8 | 34.4 | 36.5
BYOL [2] | 77.1 | 47.0 | 49.9 | 81.4 | 55.3 | 61.1 | 57.8 | 37.9 | 40.9 | 54.3 | 33.2 | 35.0
SimSiam [13] | 77.3 | 48.5 | 52.5 | 82.4 | 57.0 | 63.7 | 59.3 | 39.2 | 42.1 | 56.0 | 34.4 | 36.7
Barlow Twins [3] | 75.7 | 47.2 | 50.3 | 82.6 | 56.8 | 63.4 | 59.0 | 39.2 | 42.5 | 56.0 | 34.3 | 36.5
SwAV [10] | 75.5 | 46.5 | 49.6 | 82.6 | 56.1 | 62.7 | 58.6 | 38.4 | 41.3 | 55.2 | 33.8 | 35.9
MEC [50] | 77.4 | 48.3 | 52.3 | 82.8 | 57.5 | 64.5 | 59.8 | 39.8 | 43.2 | 56.3 | 34.7 | 36.8
RELIC v2 [45] | 76.9 | 48.0 | 52.0 | 82.1 | 57.3 | 63.9 | 58.4 | 39.3 | 42.3 | 56.0 | 34.6 | 36.3
CorInfoMax [49] | 76.8 | 47.6 | 52.2 | 82.4 | 57.0 | 63.4 | 58.8 | 39.6 | 42.5 | 56.2 | 34.8 | 36.5
VICRegL [51] | 75.9 | 47.4 | 52.3 | 82.6 | 56.4 | 62.9 | 59.2 | 39.8 | 42.1 | 56.5 | 35.1 | 36.8
SimCLR + DSA | 77.5 | 47.9 | 52.0 | 83.6 | 57.5 | 63.9 | 59.0 | 39.5 | 42.9 | 56.3 | 35.4 | 36.3
MoCo + DSA | 79.4 | 48.2 | 54.1 | 84.4 | 58.6 | 66.4 | 61.0 | 40.8 | 44.3 | 57.1 | 36.2 | 38.9
BYOL + DSA | 78.5 | 48.4 | 52.1 | 83.1 | 57.0 | 62.6 | 59.0 | 39.3 | 43.3 | 55.8 | 35.3 | 37.2
SimSiam + DSA | 78.7 | 50.1 | 54.2 | 84.7 | 58.3 | 65.0 | 60.4 | 41.0 | 43.2 | 58.1 | 36.9 | 38.9
SwAV + DSA | 77.4 | 48.1 | 52.0 | 83.9 | 58.5 | 64.9 | 60.3 | 40.1 | 43.8 | 57.2 | 35.1 | 37.6
VICRegL + DSA | 78.4 | 49.1 | 54.6 | 83.8 | 57.8 | 64.8 | 60.6 | 42.2 | 43.4 | 57.7 | 37.5 | 38.2

###  VI-C Standard Evaluation

To evaluate the performance of the pre-trained feature extractor with our
proposed method, we perform the standard evaluation protocol, i.e. linear
evaluation on the obtained feature extractors. This protocol involves freezing
the feature extractor $f(\cdot)$ after pre-training and subsequently training
a supervised linear classifier with softmax on top of it. We present the
average top-1 and top-5 accuracy and the standard deviation of 5 runs for pre-
trained image feature extractor in Table II. While the linear evaluation
results for video data is shown in Table III.

From Table II, we can observe that the inclusion of DSA significantly improves
the linear evaluation performance compared to the SSL baselines, with an
increase in average top-1 accuracy by 2.04%, and an increase in top-5 accuracy
by 1.57%. Furthermore, DSA is effective across different backbones, with an
average improvement of top-1 accuracy of 1.88% for ViT-B and 2.13% for
ResNet-50. Also, when using ResNet-50 as the backbone, VICRegL+DSA achieved
the highest average top-1 accuracy of 78.15%, surpassing the best previous
result of W-MSE by 2.14%, when using ViT-B as the backbone, DINO+DSA achieved
the highest average top-1 accuracy of 79.20%, exceeding that of DINO by 1.05%.
Moreover, despite the linear evaluation results of MAE being significantly
lower compared to other methods, the introduction of DSA results in a notable
improvement, with an increase in top-1 accuracy by 5.91%.

From Table III, we can observe that the addition of DSA leads to an average
increase of 1.4% in the top-1 accuracy of action classification by SSL
algorithms on the Kinetics-400 validation set. This demonstrates that our
proposed DSA is not only applicable to a specific type of data but is
effective across various data modalities.

In summary, DSA significantly enhances the accuracy of SSL methods on the
validation set during standard linear evaluation, aligning with the
conclusions drawn from our theoretical analysis.

###  VI-D Downstream Evaluation

To further explore the performance of our proposed DSA, we tested the pre-
trained SSL feature extractor on a range of downstream tasks. These tasks
include semi-supervised classification, action recognition, few-shot
classification, object detection, instance segmentation, and action detection.
Experimental results indicate that DSA achieves significant improvements
across all these downstream tasks.

#### Semi-Supervised Classification

We follow the setting of [1] and sample 1% and 10% of the labeled data from
ImageNet [11] in a class-balanced way, with each class containing around 12.8
to 128 images per class respectively. We then fine-tune the pre-trained
backbone with our method for 50 epochs with these sampled labeled data and
report the average top-1 and top-5 accuracies of the test set.

The results for semi-supervised classification are shown in Table IV. These
results demonstrate that when the amount of labeled data is limited, the fine-
tuning performance of pre-trained models based on self-supervised learning
methods surpasses that of supervised baselines. Additionally, the pre-trained
models incorporating the DSA method exhibit superior performance in semi-
supervised tasks compared to their self-supervised baseline counterparts. With
only 1% of labeled training data, the average top-1 accuracy increases by
1.65%, and the top-5 accuracy increases by 1.78%. With 10% of labeled training
data, the average top-1 accuracy increases by 1.73%, and the top-5 accuracy
increases by 1.73%.

Furthermore, compared to the best result with 1% labeled data, RELIC v2,
BYOL+DSA exceeds its top-1 accuracy by 2.3% and its top-5 accuracy by 1.1%.
Compared to the best result with 10% labeled data, SSL-HSIC, BYOL+DSA exceeds
its top-1 accuracy by 1.3% and its top-5 accuracy by 1.0%.

In summary, the inclusion of DSA results in significant improvements in semi-
supervised classification tasks. This indicates that enhancing model
discriminability has a notable impact on the fine-tuning of downstream
classification tasks, even when labeled data is limited.

#### Action Recognition

To evaluate the performance of the video pre-trained models, we follow the
setting in [58, 41] and fine-tune them on UCF-101 and HMDB-51 datasets. We use
the pre-trained backbone on Kinetics to initialize the network parameters and
append additionally a classification layer. On each dataset, the models are
trained for 50 epochs with a batch size of 128. The fine-tuning action
recognition results are present in Table V. For a fair comparison with other
video SSL baseline methods with small-scale feature extractors, we also report
the fine-tuning results with R3D-18 [42] as the backbone. From the table, it
can be seen that in the action recognition task, the accuracy with the
addition of DSA shows an average improvement of 1.94% compared to the video
self-supervised learning baseline. Additionally, when using R3D-18 as the
backbone and only 16-frame video clips as input, v-BYOL + DSA exceeds the
best-performing MACLR by 1.4% on the UCF-101 dataset.

These results demonstrate that the inclusion of DSA significantly enhances
accuracy in downstream action recognition tasks for video SSL. This further
indicates that our method improves the discriminability of the learned
representations.

#### Few-shot Classification

We evaluate the few-shot classification on FC100, CUB200, and Plant Disease
following the standard cross-domain few-shot setting [59, 60]. We evaluate the
few-shot classification by performing logistic regression using the pre-
trained ResNet-50 with the parameters frozen and report the results for both
the $5$-way $1$-shot and $5$-way $5$-shots task in Table VI. From the results,
we can observe that in few-shot classification tasks, SSL methods with DSA
show an average accuracy improvement of 2.26% compared to their baselines.
Moreover, the inclusion of DSA achieves better results across different
datasets. For instance, in the 5-way 1-shot task on the FC100 dataset, the
best self-supervised learning result is 40.97% achieved by W-MSE. In contrast,
SimCLR + DSA exceeds this metric by 2.07%. Similarly, in the 5-way 5-shot task
on the Plant Disease dataset, VICRegL achieves an average accuracy of 92.84%,
while BYOL + DSA surpasses this result by 1.88%.

These findings indicate that the addition of DSA significantly enhances the
performance of SSL in few-shot classification tasks.

#### Detection and Segmentation

The evaluation of downstream detection and segmentation tasks are evaluated on
Pascal-VOC and MS-COCO datasets. For object detection on Pascal-VOC, we fine-
tune a Faster R-CNN [61] with the C4-backbone. Results are reported for fine-
tuning on the VOC 2007 trainval split and the combined VOC 2007 trainval +
2012 train split, both evaluated on the VOC 2007 test split. For object
detection on COCO, we fine-tune a Mask R-CNN [62] (1$\times$ schedule) with
the C4-backbone. Fine-tuning is performed on the COCO 2017 train set and
evaluated on the COCO 2017 val set.

(a) SimCLR + DSA

(b) BYOL + DSA

(c) Barlow Twins + DSA

(d) SwAV + DSA

(e) MAE + DSA

Figure 3: Data distribution visualization based on 8 random classes of the
test set of ImageNet in the feature space. (a) - (e) corresponds to the
visualization results of the self-supervised methods integrated with DSA. We
can observe that the border between classes is not overlapped, and the inter-
class features are clustered together.

(a) $\nu$

(b) $\upsilon$

(c) $\alpha$

(d) $\eta$

Figure 4: The inter-class distance and intra-class variance correspond to
different values of hyper-parameters $\nu$, $\upsilon$, $\alpha$, and $\eta$.
The solid lines in figures (a) - (d) represent the inter-class distance while
the dashed lines represent the intra-class variance.

The evaluation tasks on VOC and COCO follow the protocols outlined in [1, 3,
13], where the pre-trained Resnet-50 are used for initialization. We report
the Average Precision (AP) at different intersections over union (IoU) ratios
for results from VOC and COCO in Table VII.

TABLE VIII: The inter-class distance, the intra-class variance, and the linear evaluation accuracy of SSL methods. Method | Inter-class Dist. ($\uparrow$) | Intra-class Var. ($\downarrow$)
---|---|---
SimCLR | 1.17 | 1.15
BYOL | 0.9 | 0.65
SwAV | 1.12 | 1.01
Barlow Twins | 1.06 | 1.11
MAE | 0.14 | 0.85
Supervised | 1.32 | 0.62
SimCLR + DSA | 1.37 | 0.92
BYOL + DSA | 1.21 | 0.55
SwAV + DSA | 1.31 | 0.57
Barlow Twins + DSA | 1.15 | 0.97
MAE + DSA | 0.64 | 0.65

From the results in Table VII, it can be seen that the inclusion of DSA
significantly improves the AP in object detection and instance segmentation
tasks. Specifically, in the VOC 07 detection task, the method with DSA shows
an average AP improvement of 1.42% compared to the SSL baseline. In the VOC
07+12 detection task, the average AP increases by 2.11%. In the COCO detection
task, the average AP increases by 1.65%, and in the COCO instance segmentation
task, the average AP increases by 1.78%.

Based on the above results, it can be concluded that incorporating DSA in the
self-supervised pre-training process can significantly enhance its
effectiveness in downstream detection and instance segmentation tasks.

TABLE IX: The inter-class distance, the intra-class variance, and the linear
evaluation accuracy of SimCLR + DSA without individual components.

Method | Inter-class Dist. ($\uparrow$) | Intra-class Var. ($\downarrow$) | ACC (%)
---|---|---|---
SimCLR | 1.17 | 1.15 | 70.15
\+ DSA | 1.37 | 0.92 | 72.09
\+ DSA without SM | 1.24 | 1.06 | 71.06
\+ DSA without $sc(\cdot)$ | 1.32 | 0.97 | 71.67
BYOL | 0.90 | 0.65 | 71.48
\+ DSA | 1.21 | 0.55 | 74.95
\+ DSA without SM | 0.96 | 0.63 | 72.13
\+ DSA without $sc(\cdot)$ | 1.16 | 0.52 | 73.74
SwAV | 1.12 | 1.01 | 75.78
\+ DSA | 1.31 | 0.57 | 77.84
\+ DSA without SM | 1.17 | 0.92 | 76.06
\+ DSA without $sc(\cdot)$ | 1.26 | 0.67 | 77.23

###  VI-E Discriminant analysis

#### Visualization of SSL features

To validate the effectiveness of DSA, we used the same experimental setup as
in Section IV-A. We visualized the representations obtained from self-
supervised learning with DSA on the ImageNet test set using t-SNE. The feature
visualization is shown in Figure 3. From the figure, it can be observed that
after incorporating DSA, there are clear boundaries between clusters of
different categories, and the distribution within the same category is more
concentrated. Specifically, the t-SNE map of MAE + DSA in Figure 3(e) shows
clustering characteristics compared to MAE in Figure 1(e). This indicates that
the DSA algorithm effectively reduces intra-class variance and increases
inter-class distance. In the next section, we further demonstrate this through
quantitative experiments.

#### Quantified evidence of discrimination

We further conducted a quantitative analysis of DSA’s intra-class variance and
inter-class distance using the same methods as in Section IV-A. The results
are presented in Table VIII.

From the table, it can be observed that for all SSL methods, the inclusion of
DSA results in a reduction in intra-class variance and an increase in inter-
class distance. This demonstrates the effectiveness of DSA. Additionally, as
evidenced by the performance across various downstream tasks discussed in
Section VI-D, reducing intra-class variance and increasing inter-class
distance indeed enhance generalization. This validates the correctness of our
theoretical analysis.

###  VI-F Ablation Study

#### Analysis of Hyperparameters

We explore the effects of hyperparameters $\nu$, $\upsilon$, $\eta$, and
$\alpha$ by showing the corresponding change of mean inter-class distance and
intra-class variance with these hyperparameters. Specifically, $\nu$
represents the weight of $\mathcal{L}_{AM}^{s}$, $\upsilon$ represents the
weight of $\mathcal{L}_{con}$, $\eta$ represents the number of nearest
neighbors, $\alpha$ is a threshold in $\mathcal{L}_{AM}$. The results are
illustrated in Figure 4. For results in 4(a) to 4(d), the results are obtained
by fixing other hyperparameters while changing the corresponding
hyperparameter.

From Figures 4(a) and 4(b), we can observe that in the objective function in
Equation 17, the optimal weights for $\mathcal{L}_{AM}^{s}$ and
$\mathcal{L}_{con}$ are achieved at $\nu=1\times 10^{-1}$ and
$\upsilon=1\times 10^{2}$, respectively. From Figure 4(c), it can be seen that
as $\alpha$ increases, the inter-class distance gradually rises and then
stabilizes, while the intra-class variance continuously decreases and
stabilizes. This result indicates that the score values output by the $f_{s}$
network can indeed measure the distance between samples. From Figure 4(d), we
observe that as $\eta$ increases, both the inter-class distance and intra-
class variance experience significant growth from $\eta=5$ to $\eta=10$ and
then level off. This suggests that when the number of nearest neighbor samples
is small, the score $sc(z_{i})$ obtained is not accurate. However, as $\eta$
increases, the score $sc(z_{i})$ becomes more accurate but stabilizes once
$\eta$ reaches a certain value. Therefore, the default value for $\eta$ is set
to 20.

#### Ablation study on components of DSA

We present an ablation study comparing different components of DSA, i.e. the
SM and the scoring mechanism $sc(\cdot)$, in Table IX. Table IX shows inter-
class distance, intra-class variance, and top-1 test accuracy on ImageNet for
DSA without each individual component.

From Table IX, it can be seen that without incorporating SM, the inter-class
distance is slightly higher than that of the original SSL method, and the
final accuracy is also marginally higher than the original SSL method. This
indicates that SM has a significant impact on the effectiveness of DSA.
Moreover, even without $sc(\cdot)$, the performance is already significantly
better than the original SSL method, though still lower than the full DSA
method. This demonstrates that the weights obtained by $sc$ play an important
auxiliary role in the DSA method.

#### The selection of $f_{aux}$ and $f_{s}$

To evaluate the impact of different choices of $f_{aux}$ and $f_{s}$, we
conduct ablation studies on the ImageNet test set. The default feature
extractor is the pre-trained CLIP image encoder [63]. We evaluate the
selection of $f_{aux}$ with three other different feature extractors, namely
open-source pre-trained ResNet-18, Scale-invariant feature transform (SIFT)
[64], and BEiT [16]. The default setting for $f_{s}$ is a two-layer MLP with
ReLU activation function, we conduct experiments by substituting the network
$f_{s}$ with cosine similarity, which yields
$M_{i,j}=\text{sim}(z_{i},z_{j})=\frac{{z_{i}\cdot
z_{j}^{T}}}{{\|z_{i}\|\|z_{j}\|}}$. All results are depicted in Table X. From
the results of $f_{aux}$, we can conclude that the performance of DSA is not
impacted by the choice of pre-trained feature extractor (i.e., the
architecture, pre-training method, or data), while the use of a pre-trained
feature extractor is essential. The results of $f_{s}$ demonstrate that the
performance of cosine similarity deteriorates significantly compared to
learnable MLP, thereby highlighting the efficacy of $f_{s}$.

TABLE X:  Top-1 accuracies on ImageNet test set for different selection of
$f_{aux}$ and $f_{s}$

Network | Selection | Top-1 ACC (%)
---|---|---
SimCLR + DSA | BYOL + DSA | Barlow Twins + DSA
$f_{aux}$ | CLIP | 72.09 | 74.95 | 76.03
BEiT | 72.04 | 74.87 | 75.94
SiFT | 71.98 | 74.82 | 75.91
ResNet-18 | 72.06 | 74.49 | 76.04
$f_{s}$ | MLP | 72.09 | 74.95 | 76.03
Cosine Similarity | 70.96 | 73.39 | 75.88

##  VII Conclusion

In this paper, we present empirical evidence of the existence of the crowding
problem in self-supervised learning and provide theoretical proof that
minimizing intra-class variance and maximizing inter-class separation
effectively compresses the upper bound of classification error. Based on these
findings, we propose a novel approach called ”Dynamic Semantic Adjuster”
(DSA), which aims to attract samples with similar semantics while repelling
others. We evaluate the effectiveness of our proposed methods through
experiments in comprehensive downstream tasks. We also provide visualization
and quantified evidence to analyze the discriminability of the learned
representation. The experimental results and ablation study demonstrate the
effectiveness of our proposed DSA.

## References

  * [1] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A Simple Framework for Contrastive Learning of Visual Representations,” in _Proceedings of the 37th International Conference on Machine Learning_. PMLR, Nov. 2020, pp. 1597–1607.
  * [2] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, k. kavukcuoglu, R. Munos, and M. Valko, “Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning,” in _Advances in Neural Information Processing Systems_ , vol. 33. Curran Associates, Inc., 2020, pp. 21 271–21 284.
  * [3] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, “Barlow Twins: Self-Supervised Learning via Redundancy Reduction,” in _Proceedings of the 38th International Conference on Machine Learning_. PMLR, Jul. 2021, pp. 12 310–12 320.
  * [4] W. Qiang, J. Li, C. Zheng, B. Su, and H. Xiong, “Robust local preserving and global aligning network for adversarial domain adaptation,” _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * [5] J. Li, W. Qiang, C. Zheng, B. Su, F. Razzak, J.-R. Wen, and H. Xiong, “Modeling multiple views via implicitly preserving global consistency and local complementarity,” _IEEE Transactions on Knowledge and Data Engineering_ , 2022.
  * [6] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 9650–9660.
  * [7] A. Bardes, J. Ponce, and Y. LeCun, “Vicreg: Variance-invariance-covariance regularization for self-supervised learning,” in _International Conference on Learning Representations_ , 2021.
  * [8] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 16 000–16 009.
  * [9] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, “Simmim: A simple framework for masked image modeling,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 9653–9663.
  * [10] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 9912–9924, 2020.
  * [11] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , Jun. 2009, pp. 248–255.
  * [12] Y. Tian, D. Krishnan, and P. Isola, “Contrastive Multiview Coding,” Dec. 2020.
  * [13] X. Chen and K. He, “Exploring Simple Siamese Representation Learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 15 750–15 758.
  * [14] A. Ermolov, A. Siarohin, E. Sangineto, and N. Sebe, “Whitening for self-supervised representation learning,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 3015–3024.
  * [15] T. N. Kipf and M. Welling, “Variational graph auto-encoders,” _arXiv preprint arXiv:1611.07308_ , 2016.
  * [16] H. Bao, L. Dong, S. Piao, and F. Wei, “Beit: Bert pre-training of image transformers,” _arXiv preprint arXiv:2106.08254_ , 2021.
  * [17] S. Balakrishnama and A. Ganapathiraju, “Linear discriminant analysis-a brief tutorial,” _Institute for Signal and information Processing_ , vol. 18, no. 1998, pp. 1–8, 1998.
  * [18] N. Saunshi, O. Plevrakis, S. Arora, M. Khodak, and H. Khandeparkar, “A Theoretical Analysis of Contrastive Unsupervised Representation Learning,” in _Proceedings of the 36th International Conference on Machine Learning_. PMLR, May 2019, pp. 5628–5637.
  * [19] S. Chen, C. Gong, J. Li, J. Yang, G. Niu, and M. Sugiyama, “Learning contrastive embedding in low-dimensional space,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 6345–6357, 2022.
  * [20] C. M. Bishop, “Pattern recognition and machine learning,” _Springer google schola_ , vol. 2, pp. 1122–1128, 2006.
  * [21] J. Yang, D. Parikh, and D. Batra, “Joint unsupervised learning of deep representations and image clusters,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 5147–5156.
  * [22] J. Xie, R. Girshick, and A. Farhadi, “Unsupervised deep embedding for clustering analysis,” in _International conference on machine learning_. PMLR, 2016, pp. 478–487.
  * [23] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 132–149.
  * [24] J. Li, P. Zhou, C. Xiong, and S. C. H. Hoi, “Prototypical Contrastive Learning of Unsupervised Representations,” Mar. 2021.
  * [25] C.-Y. Ko, J. Mohapatra, S. Liu, P.-Y. Chen, L. Daniel, and L. Weng, “Revisiting contrastive learning through the lens of neighborhood component analysis: An integrated framework,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 11 387–11 412.
  * [26] T. Wang and P. Isola, “Understanding contrastive representation learning through alignment and uniformity on the hypersphere,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 9929–9939.
  * [27] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong, “ibot: Image bert pre-training with online tokenizer,” _arXiv preprint arXiv:2111.07832_ , 2021.
  * [28] Q. Zhang, Y. Wang, and Y. Wang, “How mask matters: Towards theoretical understandings of masked autoencoders,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 27 127–27 139, 2022.
  * [29] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [30] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [31] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” _Journal of machine learning research_ , vol. 9, no. 11, 2008.
  * [32] Y.-X. Wang, H. Xu, and C. Leng, “Provable subspace clustering: When lrr meets ssc,” _Advances in Neural Information Processing Systems_ , vol. 26, 2013.
  * [33] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev _et al._ , “The kinetics human action video dataset,” _arXiv preprint arXiv:1705.06950_ , 2017.
  * [34] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [35] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _International journal of computer vision_ , vol. 88, no. 2, pp. 303–338, 2010.
  * [36] B. Oreshkin, P. Rodríguez López, and A. Lacoste, “Tadam: Task dependent adaptive metric for improved few-shot learning,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [37] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “The caltech-ucsd birds-200-2011 dataset,” 2011.
  * [38] S. P. Mohanty, D. P. Hughes, and M. Salathé, “Using deep learning for image-based plant disease detection,” _Frontiers in plant science_ , vol. 7, p. 1419, 2016.
  * [39] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” 2009.
  * [40] I. Susmelj, M. Heller, P. Wirth, J. Prescott, and M. E. et al., “Lightly,” _GitHub. Note: https://github.com/lightly-ai/lightly_ , 2020.
  * [41] C. Feichtenhofer, H. Fan, B. Xiong, R. Girshick, and K. He, “A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning,” Apr. 2021.
  * [42] S. Ji, W. Xu, M. Yang, and K. Yu, “3d convolutional neural networks for human action recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 35, no. 1, pp. 221–231, 2012.
  * [43] C. Feichtenhofer, H. Fan, J. Malik, and K. He, “SlowFast Networks for Video Recognition,” Oct. 2019.
  * [44] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum Contrast for Unsupervised Visual Representation Learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9729–9738.
  * [45] N. Tomasev, I. Bica, B. McWilliams, L. Buesing, R. Pascanu, C. Blundell, and J. Mitrovic, “Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet?” _arXiv preprint arXiv:2201.05119_ , 2022.
  * [46] S. Chen, G. Niu, C. Gong, J. Li, J. Yang, and M. Sugiyama, “Large-margin contrastive learning with distance polarization regularizer,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 1673–1683.
  * [47] M. Zheng, S. You, F. Wang, C. Qian, C. Zhang, X. Wang, and C. Xu, “Ressl: Relational self-supervised learning with weak augmentation,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 2543–2555, 2021.
  * [48] Y. Li, R. Pogodin, D. J. Sutherland, and A. Gretton, “Self-supervised learning with kernel dependence maximization,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 15 543–15 556, 2021.
  * [49] S. Ozsoy, S. Hamdan, S. Arik, D. Yuret, and A. Erdogan, “Self-supervised learning with an information maximization criterion,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 35 240–35 253, 2022.
  * [50] X. Liu, Z. Wang, Y.-L. Li, and S. Wang, “Self-supervised learning via maximum entropy coding,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 34 091–34 105, 2022.
  * [51] A. Bardes, J. Ponce, and Y. LeCun, “Vicregl: Self-supervised learning of local visual features,” _arXiv preprint arXiv:2210.01571_ , 2022.
  * [52] X. Chen, S. Xie, and K. He, “An empirical study of training self-supervised vision transformers,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 9640–9649.
  * [53] C. Yang, Y. Xu, B. Dai, and B. Zhou, “Video Representation Learning with Visual Tempo Consistency,” Dec. 2020.
  * [54] I. Dave, R. Gupta, M. N. Rizve, and M. Shah, “TCLR: Temporal Contrastive Learning for Video Representation,” _Computer Vision and Image Understanding_ , vol. 219, p. 103406, Jun. 2022, arXiv:2101.07974 [cs].
  * [55] T. Pan, Y. Song, T. Yang, W. Jiang, and W. Liu, “Videomoco: Contrastive video representation learning with temporally adversarial examples,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 11 205–11 214.
  * [56] S. H. Khorasgani, Y. Chen, and F. Shkurti, “Slic: Self-supervised learning with iterative clustering for human action videos,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16 091–16 101.
  * [57] F. Xiao, J. Tighe, and D. Modolo, “Maclr: Motion-aware contrastive learning of representations for videos,” in _European Conference on Computer Vision_. Springer, 2022, pp. 353–370.
  * [58] R. Qian, T. Meng, B. Gong, M.-H. Yang, H. Wang, S. Belongie, and Y. Cui, “Spatiotemporal Contrastive Video Representation Learning,” Apr. 2021.
  * [59] H. Lee, K. Lee, K. Lee, H. Lee, and J. Shin, “Improving transferability of representations via augmentation-aware self-supervision,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 17 710–17 722, 2021.
  * [60] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-C. F. Wang, and J.-B. Huang, “A closer look at few-shot classification,” in _International Conference on Learning Representations_ , 2019.
  * [61] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [62] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [63] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8748–8763.
  * [64] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” _International journal of computer vision_ , vol. 60, pp. 91–110, 2004.

###  -A Proofs

##  Appendix A Proof of Theorem IV.1

######  Theorem .

IV.1. If Assumption IV.1 holds, then, for any $f\in\mathcal{F}$,
$\mathcal{L}_{CE}^{\mu}\left(f\right)$ can be bounded by $\mathcal{L}_{\rm
NCE}^{\mu}\left(f\right)$ as:

$\begin{array}[]{l}\mathcal{L}_{CE}^{\mu}\left(f\right)\leq{\mathcal{L}_{{\rm{NCE}}}}\left(f\right)-const+\sum\limits_{i=1,j=1i\neq
j}^{K}{\mu_{i}^{\rm{T}}{\mu_{j}}}\\\
\quad\quad\quad\quad\quad+\sqrt{{\rm{Var}}\left({f\left(x\right)\left|y\right.}\right)}+O\left({cons{t^{{{-1}\mathord{\left/{\vphantom{{-1}2}}\right.\kern-1.2pt}2}}}}\right)\end{array}$
(18)

where const is a constant that is related to the number of negative samples,
${\mu_{i}}={\mathbbm{E}_{p\left({x\left|i\right.}\right)}}\left[{f\left(x\right)}\right]$,
and
${\rm{Var}}\left({f\left(x\right)\left|y\right.}\right)={\mathbbm{E}_{p\left(y\right)}}\left[{{\mathbbm{E}_{p\left({x\left|y\right.}\right)}}{{\left\|{f\left(x\right)-{\mathbbm{E}_{p\left({x\left|y\right.}\right)}}f\left(x\right)}\right\|}^{2}}}\right]$.

###### Proof.

Suppose that $Q$ represents the number of negative samples. Then, we have:

$\displaystyle\quad\mathbbm{E}_{p(x_{i},x_{j})}[\log\frac{1}{Q}\sum_{i=1}^{Q}\exp(f^{\rm{T}}(x_{i})f(x_{j}))$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad-\log\mathbbm{E}_{p(x_{i})}\exp(f^{\rm{T}}(x_{i})f(x_{j})]$
$\displaystyle\leq
e\mathbbm{E}_{p(x_{i},x_{j})}[\frac{1}{Q}\sum_{i=1}^{Q}\exp(f^{\rm{T}}(x_{i})f(x_{j}))$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad-\mathbbm{E}_{p(x_{i})}\exp(f^{\rm{T}}(x_{i})f(x_{j}))]$
$\displaystyle=\mathcal{O}\left({{Q^{-\frac{1}{2}}}}\right)$ (19)

where the first inequality follows the Intermediate Value Theorem and $e$ (the
natural number) is the upper bound of the absolute derivative of log between
two points when
$\left|{{f^{\rm{T}}}\left({{x_{i}}}\right)f\left({{x_{j}}}\right)}\right|\leq
1$. The second inequality follows the Berry-Esseen Theorem given the bounded
support of
${\exp\left({{f^{\rm{T}}}\left({{x_{i}}}\right)f\left({{x_{j}}}\right)}\right)}$
as following: for i.i.d random variables ${\psi_{i}}$ with bounded support
$supp\left(\psi\right)\subset\left[{-a,a}\right]$, zero mean and bounded
variance $\sigma_{\psi}^{2}<{a^{2}}$, we have:

$\displaystyle\mathbbm{E}\left[\frac{1}{Q}\sum_{i=1}^{Q}\psi_{i}\right]$
$\displaystyle=\frac{\sigma_{\psi}}{\sqrt{Q}}\int_{0}^{\frac{a\sqrt{Q}}{\sigma_{\psi}}}p\left[\frac{1}{\sigma_{\psi}\sqrt{Q}}\sum_{i=1}^{Q}\psi_{i}>x\right]dx$
$\displaystyle\leq\frac{\sigma_{\psi}}{\sqrt{Q}}\int_{0}^{\frac{a\sqrt{Q}}{\sigma_{\psi}}}p[|N(0,1)|>x]dx+\frac{C_{a}}{\sqrt{Q}}dx$
$\displaystyle\leq\frac{C_{a}}{\sqrt{Q}}+\frac{a}{\sqrt{Q}}\mathbbm{E}[N(0,1)]$
$\displaystyle=\mathcal{O}\left({{Q^{-\frac{1}{2}}}}\right)$ (20)

where ${{C_{a}}}$ is the constant that only depends on $a$ and
${\psi_{i}}=\exp\left({{f^{\rm{T}}}\left({{x_{i}}}\right)f\left({{x_{j}}}\right)}\right)$.

Then, we suppose that the classification task consists of $K$ categories and
denote that ${u_{y}}$ as the center of features of class
$y,y\in\left\\{{1,...,K}\right\\}$. We denote ${p\left({x,{x^{+}},y}\right)}$
as the joint distribution of the positive pairs ${x,{x^{+}},y}$, denote
${x^{+}}$ as the random variable of positive sample, denote ${x^{+}_{i}}$ as a
negative sample, denote ${x^{-}}$ as the random variable of negative sample,
and denote $y^{-}$ as the negative class. We have:

$\displaystyle\mathcal{L}_{\rm{NCE}}$
$\displaystyle=-\mathbbm{E}_{p(x,x^{+})}f(x)^{\rm{T}}f(x^{+})$
$\displaystyle\quad+\mathbbm{E}_{p(x)}\mathbbm{E}_{p(x_{i}^{-})}\log\sum_{i=1}^{Q}\exp(f(x)^{\rm{T}}f(x_{i}^{-}))+\log
Q$ $\displaystyle\geq-\mathbbm{E}_{p(x,x^{+})}f(x)^{\rm{T}}f(x^{+})$
$\displaystyle\quad+\mathbbm{E}_{p(x)}\log\frac{1}{Q}\mathbbm{E}_{p(x_{i}^{-})}\exp(f(x)^{\rm{T}}f(x_{i}^{-}))+\log
Q$
$\displaystyle\geq-\mathbbm{E}_{p(x,x^{+},y)}f(x)^{\rm{T}}f(x^{+})-const(Q)+\log
Q$
$\displaystyle\quad+\mathbbm{E}_{p(x)}\log\frac{1}{Q}\mathbbm{E}_{p(y^{-})}\mathbbm{E}_{p(x_{i}^{-}|y^{-})}\exp(f(x)^{\rm{T}}f(x_{i}^{-}))$
$\displaystyle\geq-\mathbbm{E}_{p(x,x^{+},y)}[f(x)^{\rm{T}}u_{y}+\|f(x^{+})-u_{y}\|]$
$\displaystyle\quad+\mathbbm{E}_{p(x)}\log\mathbbm{E}_{p(y^{-})}\exp(f(x)^{\rm{T}}u_{y^{-}})$
$\displaystyle\quad-const(Q)+\log Q-\sum\limits_{i=1,j=1,i\neq
j}^{K}{\mu_{i}^{\rm{T}}{\mu_{j}}}$
$\displaystyle\geq-\mathbbm{E}_{p(x,y)}f(x)^{\rm{T}}u_{y}-\sqrt{\mathbbm{E}_{p(x,y)}\|f(x)-u_{y}\|^{2}}$
$\displaystyle\quad+\mathbbm{E}_{p(x)}\log\mathbbm{E}_{p(y^{-})}\exp(f(x)^{\rm{T}}u_{y^{-}})$
$\displaystyle\quad-const(Q)+\log Q-\sum\limits_{i=1,j=1,i\neq
j}^{K}{\mu_{i}^{\rm{T}}{\mu_{j}}}$
$\displaystyle=\mathbbm{E}_{p(x,y)}[-f(x)^{\rm{T}}u_{y}+\log\sum_{k=1}^{K}\exp(f(x)^{\rm{T}}u_{k})]+\log
M$
$\displaystyle-\sqrt{{\rm{Var}}(f(x)|y)}-const(Q)-\sum\limits_{i=1,j=1,i\neq
j}^{K}{\mu_{i}^{\rm{T}}{\mu_{j}}}$
$\displaystyle=\mathcal{L}_{CE}^{\mu}(f)-\sqrt{{\rm{Var}}(f(x)|y)}-const(Q)$
$\displaystyle\quad+\log\frac{Q}{K}-\sum\limits_{i=1,j=1,i\neq
j}^{K}{\mu_{i}^{\rm{T}}{\mu_{j}}}$
$\displaystyle=\mathcal{L}_{CE}^{\mu}(f)-\sqrt{{\rm{Var}}(f(x)|y)}-\sum\limits_{i=1,j=1,i\neq
j}^{K}{\mu_{i}^{\rm{T}}{\mu_{j}}}$
$\displaystyle\quad+\mathcal{O}(const(Q)^{-\frac{1}{2}})$ (21)

∎