# MNN: Mixed Nearest-Neighbors for Self-Supervised Learning

Xianzhong Long<EMAIL_ADDRESS>Chen Peng  Yun Li  School of Computer
Science, Nanjing University of Posts and Telecommunications, Nanjing, 210023,
China  Jiangsu Key Laboratory of Big Data Security and Intelligent Processing,
Nanjing, 210023, China

###### Executive Summary

In contrastive self-supervised learning, positive samples are typically drawn
from the same image but in different augmented views, resulting in a
relatively limited source of positive samples. An effective way to alleviate
this problem is to incorporate the relationship between samples, which
involves including the top-K nearest neighbors of positive samples. However,
the problem of false neighbors (i.e., neighbors that do not belong to the same
category as the positive sample) is an objective but often overlooked
challenge due to the query of neighbor samples without supervision
information. In this paper, we present a simple self-supervised learning
framework called Mixed Nearest-Neighbors for Self-Supervised Learning (MNN).
MNN optimizes the influence of neighbor samples on the semantics of positive
samples through an intuitive weighting approach and image mixture operations.
The results demonstrate that MNN exhibits exceptional generalization
performance and training efficiency on four benchmark datasets.

###### keywords:

Self-supervised learning , K-nearest neighbors , Contrastive learning , Image
mixture , Momentum encoder.

††journal:

##  1 Introduction

Self-supervised learning (SSL) has made significant progress in the field of
deep learning due to its ability to learn rich semantic features using large-
scale unlabeled data [1, 2, 3, 4, 5]. A notable direction of this research
involves instance discrimination tasks [6, 7]. Briefly, the core idea is that
views of the same image with different data augmentations should be close to
each other in feature space, while views of different images should be far
away from each other. Some studies have incorporated nearest neighbors into
contrastive self-supervised learning to address some challenges within the
framework. This type of approach helps to mitigate issues like the class
collision problem, which arises from neglecting intra-class relationships in
the dataset. For instance, NNCLR [8] seeks the top-K neighbors of positive
samples in the support set and employs Noise Contrastive Estimation (NCE) Loss
[9] to align these neighbors with positive samples from another branch. MSF
[10] enhances the learned semantic features by identifying the top-K nearest
neighbors of the corresponding positive samples in the target branch.
Subsequently, MSF utilizes Mean Squared Error (MSE) Loss to connect these
neighbors with the positive samples in the online branch. CMSF [11] exploits
an additional support set to enrich the semantics of neighbor samples.
However, these methods ignore a significant issue: there are some samples
(e.g., the 3rd neighbor in Fig. 1) in the neighbor set that are semantically
inconsistent with the current instance. We term them as False Nearest-
Neighbors (FNN). We alleviate this issue by mixing the positive sample and its
corresponding neighbor samples in the feature space.

Figure 1: The positive sample and its neighbor samples set. The set of
neighbor samples is composed of the samples in the support set that have the
top-K largest cosine similarity values to the positive sample (e.g., K is 3 in
the figure). Samples within the neighbor set are treated as additional
positive samples, thus samples that are visually similar to the positive
sample but semantically different (e.g., the 3rd neighbor), which may confuse
the model.

Inspired by SNCLR [12], we initially considered transferring the Cross-
Attention Score (CAS) from SNCLR to the MSF framework without negative
samples. Although this change led to significant performance improvements in
downstream tasks compared to the original MSF, further analysis (detailed in
Sec. 4.4.1) revealed increased instability with CAS. As a result, we
reformulated a simplified loss function, the Weighted Squared Error (WSE), to
enhance method stability and interpretability while retaining the experimental
performance gains.

Some previous works have also attempted to combine the mixture with SSL. For
example, UnMix [13] improves the performance of SSL with additional forward
propagation of mixed images. i-Mix [14] employs mixture as a regularization
strategy to increase the generalization of contrastive SSL. MoCHi [15]
synthesizes more meaningful hard negative samples by mixture to facilitate
model learning. Our work is different from them. Instead of mixture in pixel
space, MNN first mixes the current instance and its corresponding top-K
neighbors in feature space. Secondly, the mixture of MNN aims to alleviate the
disturbance of false neighbors in neighbor sets, instead of treating them as
hard negative samples. Finally, the additional computational overhead of the
MNN can be neglected without additional forward propagation of mixed images.
This enables MNN more efficient and easy to implement in applications.

Our main contributions are summarized as follows:

  * •

We detected instability when examining the entropy of the weight distribution
associated with the Cross-Attention Score. To address this issue, we
introduced a simplified loss function that effectively integrates nearest-
neighbors with contrastive self-supervised learning.

  * •

To mitigate the impact of false neighbors in the model, we employ a mixing
strategy that combines the positive samples with their corresponding neighbor
samples in the feature space. Our approach not only significantly enhances
model performance with minimal additional overhead but also operates on a
highly intuitive rationale.

##  2 Related work

In this section, we will review the applications of Self-supervised learning,
Nearest-neighbors exploration, and Mix-up in computer vision related to our
MNN.

###  2.1 Self-supervised learning

Self-supervised learning has made significant progress in the field of
computer vision as a method for acquiring generalized semantic features
without manually annotated data [16, 17, 18]. Early self-supervised methods
were mainly based on heuristic pretext tasks (e.g., predicting image rotation
[19], recovering image color [4]). However, the performance of these methods
is limited by the setting of the task.

In recent years, contrastive self-supervised learning has garnered widespread
attention [7, 6, 9, 20, 21]. Notably, methods like MoCo [2, 22], SimCLR [1],
and BYOL [23] have made remarkable strides in performance enhancement. They
achieve this by incorporating components like momentum updates, projection
heads, and asymmetric network structures to optimize the model. However, these
methods still rely on data augmentation driven by prior knowledge for creating
positive samples, which may limit their capacity to generalize to scenarios
with insufficient prior knowledge. In this work, MNN alleviates the effect of
FNN by bringing in a domain-agnostic mixture. The approach capitalizes on
nearest neighbors to enhance the semantic diversity of positive samples.

###  2.2 Nearest-neighbors exploration

The nearest-neighbor (NN) methods are widely used in computer vision tasks
such as image classification [24] and domain adaptation [25]. Self-supervised
learning seeks to uncover nuanced relationships between samples using NN
techniques. For instance, NNCLR [8] optimizes the model’s performance by
identifying the top-K neighbors of positive samples and applying NCE loss.
Drawing from the BYOL approach [23], MSF [10] locates the nearest neighbors
corresponding to positive samples and enhances their semantic features using
MSE loss. However, these methods often overlook the presence of false
neighbors, a potential drawback that can compromise the generalization
performance of the model.

SNCLR [12] assesses the impact of distinct neighbor samples on the model using
Cross-Attention Score, constructed through identity mappings and cosine
similarity. While our efforts to transfer CAS to MSF [10] without negative
samples resulted in performance improvements, we also observed that CAS could
lead to substantial model confusion. To seamlessly integrate nearest neighbors
into self-supervised learning, we introduced a simplified loss function. This
loss function effectively distinguishes between positive and neighbor samples
by assigning distinct fixed weights.

###  2.3 Mix-up

Mix-up [26] is a well-established regularization technique that finds
applications across various learning paradigms. In essence, it trains the
network by forming a convex combination of sample pairs and label pairs,
thereby promoting the acquisition of semantic features with linear properties.
Recent research has explored the fusion of Mix-up with self-supervised
learning. For instance, MixCo [27] randomly combines two images from the same
batch, enabling the model to consider the implicit relationship between
positive and negative samples. i-Mix [14] introduces image mixture as a
domain-agnostic regularization approach for contrastive learning, leading to
enhanced model performance in multiple modalities, such as video and speech.
UnMix [13] facilitates the learning of smooth decision boundaries through
self-mixtures, promoting less confident predictions. Notably, the mixing
strategy employed in MNN operates within the feature space rather than the
pixel space. Moreover, the mixture operation of MNN has no need for additional
forward propagation, so its additional computational overhead is negligible
compared to previous works.

MoCHi [15], a related method to our work, utilizes mixture operations in the
feature space to create more meaningful hard negative samples, thereby
adjusting the difficulty of the pre-training task. In contrast, MNN employs
mixed-sample features as additional positive samples with the specific goal of
reducing noise introduced by False Nearest-Neighbors.

Figure 2: MNN overview: We identify the top-K samples in the support set with
the highest cosine similarity to $z^{2}$, forming the neighbor set. Each
neighbor sample undergoes a mixture operation, resulting in a mixed neighbor,
illustrated as a triangle. These mixed neighbors, along with $p^{1}$ and
$z^{2}$ itself, contribute to the computation. The student network is updated
using the loss function, while the teacher network undergoes an update through
a momentum update process, which involves computing the exponential moving
average of the student network. We use a straightforward loss function (WSE)
and image mixture to mitigate the disturbance caused by semantically
inconsistent false neighbors (e.g., the green dot in the figure) with the
current instance.

##  3 Method

In this section, we provide a detailed explanation of our proposed MNN method,
including key components such as neighbor sample querying, distinguishing
between positive and neighbor samples, and image mixing.

Fig. 2 illustrates the MNN framework. Starting with image $x$ from the current
batch $X$, we apply two data augmentations to create views $x^{1}$ and
$x^{2}$. These views are then processed by the student encoder and the teacher
encoder, resulting in embeddings $z^{1}$ and $z^{2}$. For $z^{2}$, we select
the top-K embeddings ${\\{z^{2}_{i}\\}}_{i=1}^{K}$ as its nearest neighbors
based on cosine similarity with $z^{2}$ within the support set, which includes
other sample embeddings in the dataset. Additionally, we mix these neighbor
samples with $z^{2}$ to obtain ${\\{\widetilde{z^{2}_{i}}\\}}_{i=1}^{K}$. To
prevent model collapse [23], we introduce a predictor $h(\cdot)$ in the
student network branch, generating $p^{1}$. In the optimization process, we
aim to minimize the following loss function:

$Loss_{x}={\textstyle\sum_{i=0}^{K}w_{i}*||p^{1}-\widetilde{z_{i}^{2}}||_{2}^{2}}\
,$ (1)

where $\widetilde{z^{2}_{0}}=z^{2}$ and all embeddings are normalized before
computation. The weights $w_{i}$ are used to determine the contribution of
each ${\widetilde{z^{2}_{i}}}$ to the loss function. Finally, we update the
student encoder and the teacher encoder using backpropagation and momentum
mechanisms [2], respectively. Notably, if we exclude the mixture operation and
set the weight $w_{i}=1/(K+1)$, the MNN approach becomes equivalent to MSF
[10]. Furthermore, if we set $K=0$ and $w_{0}=1$, MNN degenerates to BYOL
[23]. Thus, MNN can be viewed as a form of generalization of the BYOL
approach.

###  3.1 Search for neighbor samples

We employ a strategy aligned with classical approaches [8, 10, 12], initially
using the support set $\mathcal{S}$ to store candidate neighbors. We rely on
cosine similarity as the metric for identifying the top-K neighbor samples of
$z^{2}$:

${\\{z^{2}_{i}\\}}_{i=1}^{K}=\underset{s\in\mathcal{S}}{argmax}(cos(z^{2},s),top_{n}=K)\
,$ (2)

where ${\\{z^{2}_{i}\\}}_{i=1}^{K}$ are sorted in descending order of cosine
similarity with $z^{2}$. The default value for K is 5, and in Sec. 4.3.2, we
conduct comparative experiments with different K values. After updating the
network parameters, we employ a first-in-first-out (FIFO) strategy to refresh
the batch of samples associated with the teacher network branch in
$\mathcal{S}$. These strategies aim to maintain the samples in the support
set, ensuring consistency and the validity of the neighbor finding and mixing
processes.

In our approach, the top-K neighbor samples are treated as additional positive
samples and incorporated into the Weighted Squared Error Loss along with
$p^{1}$. Consequently, the accurate selection of these neighbor samples
becomes pivotal. Since we cannot select neighbor samples based on labels in
the context of SSL, the issue of False Nearest-Neighbors arises. To alleviate
this, we assign different weights to distinguish the contributions of positive
and neighbor samples to the model. Additionally, we mitigate the impact of FNN
through image mixture.

###  3.2 Distinguish between positive and neighbor samples

We initially tried to differentiate between positive and neighbor samples
using the Cross-Attention Score [12, 28]. Although this approach led to some
performance improvements, our analysis revealed issues with the instability of
its weight distribution (detailed in Sec. 4.4.1). As a result, we redefined a
method in which different fixed weights are assigned to distinguish the
positive sample $z^{2}$ from the neighbor samples:

$w_{i}^{WSE}=\begin{cases}1&,\ i=0\\\ \frac{1}{K}&,\text{ otherwise
}.\end{cases}$ (3)

We believe that neighbor samples can indeed augment the diversity of positive
samples. However, we also acknowledge the possibility of substantial semantic
distinctions between neighbor samples and true positive samples. These
distinctions could potentially increase the risk of false neighbors.
Consequently, it becomes essential to differentiate between neighbor samples
and genuine positive samples by employing varying weights.

###  3.3 Mixed images

Addressing the instability in weight assignment caused by CAS through the
allocation of different fixed weights to positive and neighbor samples can
enhance the generalization capacity of the model. However, the presence of
objectively false neighbor samples, as depicted in Fig. 1, prompts us to
consider methods for making these spurious neighbors more closely resemble
genuine ones. In the embedding space of the teacher network output, we utilize
mix-up [26] to alleviate the disruption from FNN in the model. Specifically,
for each neighbor embedding $z^{2}_{i}$, we perform a mixture operation:

$\widetilde{z_{i}^{2}}=\lambda*z_{i}^{2}+(1-\lambda)*z^{2}\ ,$ (4)

where the mixed coefficients $\lambda$ follow a standard uniform distribution.
Our goal with this mixture approach in MNN is to divert attention away from
false neighbors (for details, see B), thereby enhancing the focus on
optimizing the relationship between positive embeddings $z^{2}$ and $p^{1}$.

Table 1: Notations in the MNN framework.

Notation | Meaning
---|---
$\mathcal{S}$ |  The set consists of embeddings generated by passing samples from previous batches through the teacher encoder. The update process follows a first-in-first-out approach.
${\\{z^{2}_{i}\\}}_{i=1}^{K}$ |  The set comprises the K neighbors of the embedding $z^{2}$. To identify these neighbors, we calculate the cosine similarity between $z^{2}$ and all embeddings in the support set $\mathcal{S}$, selecting the top-K embeddings in descending order of similarity.
${\\{\widetilde{z^{2}_{i}}\\}}_{i=1}^{K}$ |  Each neighbor $\widetilde{z_{i}^{2}}$ in this set is a mixture of the embedding $z^{2}$ and its corresponding neighbor $z_{i}^{2}$.
${\\{w_{i}\\}}_{i=1}^{K}$ |  Each element $w_{i}$ in the set quantifies the contribution of the neighbor embedding $\widetilde{z_{i}^{2}}$ to the loss function.

###  3.4 Relation between WSE and Mixture

In an ideal scenario, the integration of neighbors in contrastive self-
supervised learning would allow the model to consider the intra-class
relationships in the dataset, thereby enhancing the semantic information of
the positive sample. However, the presence of false neighbors is an inherent
issue. On the one hand, WSE effectively distinguishes between the genuine
positive sample and the neighbor samples by assigning them different weights.
This allows the model to bring together distinct augmented views of the same
image while incorporating the rich and diverse semantics offered by the
neighbor samples. On the other hand, the mixture operation within MNN serves
to counteract the impact of false neighbors within the neighbor set,
mitigating their disruptive noise in the model. These two key components
collectively enable MNN to minimize the influence of false neighbors and
extract the full spectrum of rich semantic information provided by neighbor
samples. We have endeavored to apply these two essential components to
alternative approaches that might face issues related to false neighbors. Our
findings, outlined in Sec. 4.4.3, reveal their widespread applicability. We
present the pseudo-code for MNN in A and provide explanations for the key
symbols used in the MNN framework in Table 1.

Table 2: Statistics of the four datasets. As the test set for Tiny ImageNet is not accessible, we conduct evaluations using the test set for all datasets except for Tiny ImageNet, for which we employ the validation set. Dataset | CIFAR-10 | CIFAR-100 | STL-10 | Tiny ImageNet
---|---|---|---|---
Classes | 10 | 100 | 10 | 200
Image size | 32 x 32 | 32 x 32 | 96 x 96 | 64 x 64
Training set | 50,000 | 50,000 | 105,000 | 100,000
Test set | 10,000 | 10,000 | 8,000 | -
Validation set | - | - | - | 10,000

##  4 Experiments

In this section, we evaluate the MNN framework through a series of
experiments. We begin by outlining the experimental setup and then contrast
MNN’s performance with established self-supervised learning methods using
standard evaluation protocols. Ablation studies underscore the critical
components of MNN, followed by a comparison of various weight adjustment
strategies within the framework. Additionally, we enrich our analysis by
integrating MNN’s key features into other advanced methods to evaluate the
resultant performance enhancements.

###  4.1 Experiment setup

####  4.1.1 Datasets and device performance

Although self-supervised learning methods are commonly evaluated using the
ImageNet-1K dataset [29], conducting this type of evaluation remains
challenging for many research labs due to hardware limitations. We conducted
experiments on four benchmark image datasets: CIFAR-10 and CIFAR-100 [30],
each containing 60,000 images, STL-10 [31] with 113,000 images, and Tiny
ImageNet [32] with 120,000 images (see Table 2). All algorithms for this work
were pre-trained and evaluated on the same hardware device (1 Nvidia GTX 3090
GPU).

####  4.1.2 Data augmentation

Data augmentation is a pivotal aspect of contrastive self-supervised learning
[1]. Our approach aligns with the data augmentation strategies of MoCoV2 [22]
to generate distorted views, including random horizontal flips and color
distortions. To efficiently identify the corresponding set of neighbor samples
within the teacher network, preserving the semantic information of $z^{2}$ is
of utmost importance. Consequently, we adopt weak data augmentation techniques
in the teacher network, conducive to neighbor queries. Conversely, we apply
strong data augmentation in the student network to ensure the learned semantic
features are robust against geometric transformations in images. This strategy
is denoted as ’(s/w)’, with additional details provided in Table 3.

Table 3: Data augmentation of the experiment. We employ ’strong’ data
augmentation for the student network, which aligns with typical contrastive
self-supervised learning practices [22]. For the teacher network, we use
’weak’ data augmentation consisting only of Resized Crops and Horizontal Flip.
This choice allows us to meet the data augmentation requirements of
contrastive learning while mitigating the risk of False Nearest-Neighbors when
searching for neighbors.

| Resized Crops | Horizontal Flip | Color Jitter | GrayScale | Gaussian Blur
---|---|---|---|---|---
Strong | ✓ | ✓ | ✓ | ✓ | ✓
Weak | ✓ | ✓ |  |  |

####  4.1.3 Network architecture

In the encoder network, we adopt the ResNet18 [33], which comprises a backbone
$f(\cdot)$ and a projection head $g(\cdot)$. The projection head is designed
with two Fully-Connected (FC) layers; the first layer has a configuration of
[512, 2048], and the second [2048, 128]. Between these layers, normalization
and a Rectified Linear Unit (ReLU) are implemented for non-linear processing.
The prediction head $h(\cdot)$ within the student network is akin to the
projection head but differs in the first layer, which has dimensions [128,
2048].

Table 4: Parameters for Pretraining and Downstream experiments. To guarantee controlled comparisons in our experiments, we applied uniform hyperparameters across all algorithms we reproduced, wherever feasible. | CIFAR-10 (CIFAR-100) | STL-10 (Tiny ImageNet)
---|---|---
Pretraining task |  |
Epoch | 200 | 200
Top-K | 5 | 5
momentum | 0.99 | 0.996
Batch size | 256 | 256
Warm up epoch | 5 | 5
Base learning rate | 0.06 | 0.06
Support set | 4096 | 16384
Weight decay | 5e-4 | 5e-4
Downstream task |  |
Epoch | 100 | 100
Learning rate | 30 | 30
Weight decay | 0 | 0

####  4.1.4 Pre-training

To ensure equitable comparisons, we maintained consistent hyperparameter
settings across all algorithms. Our training process involved iterative
updates with an Stochastic Gradient Descent (SGD) optimizer using a momentum
of 0.9 and a weight decay of 5e-4, spanning a total of 200 epochs. In the
initial 5 epochs, we employed a linear warm-up strategy to gradually raise the
learning rate to $\text{lr}=0.06\times\text{BatchSize}/256$ before
transitioning to the cosine annealing schedule [34]. In tandem with this, we
harnessed the momentum update mechanism to optimize the teacher network
parameters, denoted as $F_{t}=m*F_{t}+(1-m)*F_{s}$, where $m$ represents the
momentum coefficient. To maintain a balance between modeling data distribution
and efficiently updating sample features, the size of the support set was
assigned based on the training set size in different datasets. By default, all
algorithms used $K=5$ as the number of neighbors. More details are provided in
Table 4.

####  4.1.5 Evaluation protocols

We conducted linear evaluation and K-nearest neighbors (KNN) classification,
which are commonly employed in SSL. During the linear evaluation phase, we
initially freeze the backbone parameters of the student network. Subsequently,
a linear classifier [512, cla] is attached to the backbone of the student
network, where cla denotes the number of semantic classes relevant to the
downstream task. The classifier was trained with a learning rate of 30, a
weight decay of 0, and a momentum of 0.9. Learning rate adjustments were made
by reducing it by a factor of 0.1 at the 60th and 80th epochs, with training
lasting for a total of 100 epochs. Given the sensitivity of linear evaluation
to hyperparameters, we opted for a straightforward KNN classification approach
[6]. First, we preserved the parameters of the pre-trained model and
transformed all the training dataset samples into 512-dimensional semantic
embeddings, which were stored. During the testing phase, KNN classification
was used to choose the top-K samples from the stored training samples and
assign labels to the test samples based on a majority vote.

###  4.2 Main results

####  4.2.1 Linear evaluation

Table 5: Linear evaluation results. We present the top-1 accuracy of classical approaches on four benchmark image datasets. Models are categorized based on whether they require nearest neighbors for training. The best performance is highlighted in bold, while suboptimal results are underlined. Results marked with ∗ denote those reproduced using the official code as they are not directly provided. Notably, the results for SimCLR and BYOL are taken from [35], while the results for SCE are derived from [36]. We aligned our experimental settings as closely as possible with theirs [35, 36] to ensure a fair comparison. Method | CIFAR-10 | CIFAR-100 | STL-10 | Tiny ImageNet
---|---|---|---|---
Supervised [35] | 94.22 | 74.66 | 82.55 | 59.26
SimCLR [1] | 84.92 | 59.28 | 85.48 | 44.38
BYOL [23] | 85.82 | 57.75 | 87.45 | 42.70
SCE [36] | 90.34 |  65.45 | 89.94 | 51.90
MoCoV2∗ [22] | 89.56 | 62.47 | 88.91 | 46.38
UnMix∗ [13] | 90.37 | 65.30 | 90.51 | 47.29
NNCLR∗ [8] | 87.72 | 59.62 | 87.13 | 41.52
SNCLR∗ [12] | 88.86 | 65.19 |  90.93 | 50.15
CMSF∗ [11] |  91.00 | 62.37 | 88.21 | 44.50
MSF∗ [10] | 89.94 | 59.94 | 88.05 | 42.68
MNN | 91.47 | 67.56 | 91.61 |  50.70

Table 5 presents the top-1 classification results obtained through linear
evaluation. To ensure fair comparisons, we applied symmetric backpropagation
uniformly across all algorithms. The results demonstrate that, in most cases,
MNN outperforms other classical self-supervised methods. A significant
distinction between MNN and MSF is the additional mixture operation in the
embedding space, as discussed in Sec. 3.3. Crucially, MNN significantly
outperformed MSF across all four datasets, with improvements of 1.53%, 7.62%,
3.56%, and 8.02%, respectively. This underscores the effectiveness of MNN in
mitigating the noise disturbance caused by False Nearest-Neighbors by working
with a mixture of positive samples and neighbor samples.

####  4.2.2 K-nearest neighbors

Linear evaluations have clearly demonstrated that MNN significantly
outperforms most classical methods. To validate robustness against
hyperparameters during evaluation, we employed a K-nearest neighbors
classifier ($K=200$) to assess pre-trained features. As seen in Table 6, our
method also surpasses previous classical methods in utilizing off-the-shelf
features. Significantly, on this evaluation metric, MNN achieves a performance
improvement over MSF by 1.57%, 9.64%, 2.56%, and 6.99% across the four
datasets, respectively.

Table 6: The results of K-nearest neighbors classification ($K=200$). Models are categorized based on whether they require nearest neighbors during training. Results labeled with ∗ indicate reproduction using official code due to the lack of direct results. The best performance is shown in bold, while suboptimal results are underlined. Method | CIFAR-10 | CIFAR-100 | STL-10 | Tiny ImageNet
---|---|---|---|---
SimCLR∗ [1] | 85.30 | 56.50 | 78.34 | 37.16
BYOL∗ [23] | 87.54 | 57.24 | 85.62 | 37.65
SCE∗ [36] | 88.54 |  59.97 | 85.09 | 40.48
MoCoV2∗ [22] | 87.82 | 57.29 | 84.66 | 37.77
UnMix∗ [13] | 87.99 | 59.11 | 85.05 | 38.65
NNCLR∗ [8] | 85.19 | 50.54 | 81.31 | 30.93
SNCLR∗ [12] | 87.36 | 58.65 |  86.02 |  41.92
CMSF∗ [11] |  89.30 | 55.57 | 84.11 | 36.79
MSF∗ [10] | 88.24 | 52.32 | 84.09 | 35.29
MNN | 89.81 | 61.96 | 86.65 | 42.28

####  4.2.3 Analysis of experimental results and insights

Given that MNN solely modifies the method of weight adjustment and introduces
a mixture operation compared to MSF, our analysis reveals some intriguing
patterns regarding the performance gains of MNN over MSF across various
evaluation metrics: (i) Larger datasets yield enhanced performance. When the
number of dataset classes remains constant, the training set of STL-10, which
is nearly twice the size of CIFAR-10’s training set, showcases more
substantial performance improvements for MNN compared to MSF. Specifically,
MNN’s performance on STL-10 saw improvements of 3.56% and 2.56% in two
separate experiments, whereas the improvements on CIFAR-10 were 1.53% and
1.57%, respectively. (ii) Greater class diversity correlates with superior
performance gains. With dataset size constant, CIFAR-100’s tenfold increase in
class diversity over CIFAR-10 leads to a notable performance decline for MSF,
likely due to the higher probability of incorporating false neighbors with an
increased number of classes. In contrast, MNN thrives in this scenario by
effectively countering the effects of false neighbors, enhancing the model’s
performance.

In summary, our experimental results illustrate that MNN achieves heightened
performance gains with larger datasets and adeptly mitigates the influence of
false neighbors in scenarios with an increased number of classes.

###  4.3 Ablation studies

In this section, we will explore the components and parameters of MNN, such as
data augmentation, the number of neighbors (K), the size of the support set,
and the neighbor selection strategy. By default, MNN utilizes ’(s/w)’ data
augmentation and sets the number of neighbors (K) to 5, employing the top-K
nearest-neighbor selection strategy unless stated otherwise. To assess the
purity of the neighbor set, we define it as the ratio of neighbor samples with
the same label as the positive sample to the total number of samples. Notably,
since STL-10 is primarily used for unsupervised pre-training, its training
samples containing labels make up approximately 5% (5,000 out of 105,000) of
the overall training dataset size. Consequently, to utilize the labels for
precise purity analysis, we conduct purity-related experiments on the
remaining three datasets.

Figure 3: The effect of different component variations on purity. (a) The
effect of data augmentation on purity. (b) The effect of the number of
nearest-neighbors on purity. (c) The effect of support set size on purity. (d)
The effect of different nearest-neighbor selection strategies on purity.
Purity represents the ratio of neighbors with the same label as the positive
sample to the total number of samples.

####  4.3.1 Data augmentation strategies

Data augmentation is a key factor in promoting contrastive self-supervised
learning by encouraging the development of semantic features that remain
consistent under geometric transformations. However, aggressive data
augmentation, as in ’(s/s),’ may hinder the ability of the teacher network to
identify suitable neighbor samples for $z^{2}$. Our hypothesis is confirmed by
Fig. 3, which illustrates that strong data augmentation reduces the purity of
the neighbor set, introducing more false neighbor samples. In Table 7, we
present linear evaluation results for various data augmentation strategies. In
the ’(w/w)’ configuration, where MNN lacks strong data augmentation, it might
face challenges in learning semantic features. Nevertheless, MNN maintains
high performance compared to MSF, indicating that the mixture operation could
potentially function as a form of strong data augmentation in domains with
limited prior knowledge.

Table 7: Comparison of data augmentation strategies. We present the linear evaluation accuracy of MNN and MSF using various data augmentation strategies. Both MSF and MNN attained their highest accuracy when strong data augmentation was applied to the student network and weak data augmentation to the teacher network. This outcome aligns with our expectation that aggressive data augmentation can impact the accuracy of neighbor selection for $z^{2}$. ∗ denotes the results obtained by different ways of data augmentation based on the official code. Method | CIFAR-10 | CIFAR-100 | STL-10 | Tiny ImageNet
---|---|---|---|---
MSF∗ (w/w) | 75.54 | 30.06 | 77.60 | 18.55
MNN (w/w) | 84.32 | 50.71 | 82.08 | 34.85
MSF∗ | 89.94 | 59.94 | 88.05 | 42.68
MNN | 91.47 | 67.56 | 91.61 | 50.70
MSF∗ (s/s) | 89.73 | 58.00 | 87.18 | 40.15
MNN (s/s) | 90.65 | 64.91 | 91.15 | 48.07

####  4.3.2 Number of nearest-neighbors

To increase the semantic richness of positive samples, we consider the nearest
neighbors retrieved by the teacher network as additional positives. Generally,
performance tends to improve with increasing K. Fig. 3 illustrates how the
purity of MNN changes with different K values. As K increases, there is a
higher risk of introducing false neighbor samples, resulting in decreased
purity. However, setting K to 1 yields a purer neighbor set but may limit
semantic diversity. Table 8 presents the linear classification accuracies at
different K values. We observed that for MSF and SNCLR, the optimal K value
for peak performance was $K=1$. In contrast, for the majority of datasets, MNN
exhibited its optimal performance when K was set to 5. This suggests that MSF
and SNCLR are more susceptible to false neighbors, while MNN effectively
mitigates this issue, striking a balance between neighbor set purity and
diversity. Therefore, MNN successfully integrates nearest neighbors into SSL,
enhancing the diversity of semantic features. Additionally, when $K=1$, MSF
resembles a variant of MNN that does not utilize the mixture operation.

Table 8: Analysis of the number of neighbors (K). ∗ denotes the result obtained by adjusting the number of nearest neighbors according to the official code. | $K=1$ | $K=5$ | $K=10$
---|---|---|---
|  | CIFAR-10 |
MSF∗ [10] | 91.55 | 89.94 | 89.46
SNCLR∗ [12] | 90.04 | 88.86 | 89.81
MNN | 91.41 | 91.47 | 91.24
|  | CIFAR-100 |
MSF∗ [10] | 65.22 | 59.94 | 57.25
SNCLR∗ [12] | 65.21 | 65.19 | 65.03
MNN | 66.98 | 67.56 | 67.13
|  | STL-10 |
MSF∗ [10] | 90.28 | 88.05 | 86.99
SNCLR∗ [12] | 90.84 | 90.93 | 90.26
MNN | 91.69 | 91.61 | 91.59
|  | Tiny ImageNet |
MSF∗ [10] | 47.06 | 42.68 | 39.68
SNCLR∗ [12] | 50.04 | 50.15 | 49.30
MNN | 49.70 | 50.70 | 49.62

####  4.3.3 Size of the support set

The size of the support set has the potential to influence its ability to
accurately reflect the data distribution. As depicted in Fig. 3, increasing
the support set size leads to higher neighbor set purity, indicating that the
current instance can find suitable nearest neighbors more effectively.
However, the model classification performance is shown in Table 9, and we
hypothesize that an excessively large support set may hinder timely updates of
sample features, consequently impacting the performance of the model.
Therefore, we strike a balance between aligning with the data distribution and
ensuring that sample features are updated promptly.

Table 9: Analysis of support set size. Size | CIFAR-10 | CIFAR-100 | STL-10 | Tiny ImageNet
---|---|---|---|---
2048 | 91.32 | 66.68 | 91.55 | 49.68
4096 | 91.47 | 67.56 | 91.50 | 49.80
8192 | 91.31 | 67.33 | 91.36 | 49.93
16384 | 91.27 | 67.24 | 91.61 | 50.70

####  4.3.4 Neighbors selection strategies

We experimented with two approaches for selecting neighbor samples, aside from
using cosine similarity. In the random selection method, K samples are
randomly chosen from the support set to serve as neighbor samples for the
current sample $z^{2}$. This approach is akin to UnMix but without the need to
mix images in the pixel space and perform extra forward propagation.
Consequently, our method incurs minimal computational overhead. To explore the
potential of MNN, we used an Oracle algorithm to select samples with the same
label as $z^{2}$ as its corresponding neighbors in the support set. To achieve
this, we preserved the labels of these elements while updating the elements of
the support set. This approach can be considered as a form of supervised
learning without label predictions, achieving a neighbor set with 100% purity.
Figure 3 illustrates the purity of the neighbor set under various settings,
while Table 10 details the linear evaluation accuracy within these scenarios.
These data demonstrate the significant performance and efficiency gains
achieved by MNN. These outcomes highlight the critical role of addressing the
false neighbor issue to optimize performance.

Table 10: Comparison of neighbor selection strategies. STL-10 is mainly employed for unsupervised pre-training, with labeled training samples constituting only around 5% of the total training dataset. Consequently, we focus our evaluation on the remaining three datasets in their labeled configurations. The ∗ symbol indicates that these results were obtained through the reproduction of the official code. Method | CIFAR-10 | CIFAR-100 | STL-10 | Tiny ImageNet
---|---|---|---|---
UnMix∗ [13] | 90.37 | 65.30 | 90.51 | 47.29
MNN(Random) | 90.25 | 65.88 | 90.60 | 47.50
MSF∗ [10] | 89.94 | 59.94 | 88.05 | 42.68
MNN | 91.47 | 67.56 | 91.61 | 50.70
Supervised [35] | 94.22 | 74.66 | 82.55 | 59.26
MNN(Labels) | 94.00 | 74.84 | - | 60.33

###  4.4 Analysis and discussion

We will conduct a detailed analysis of the impact of weight adjustments, which
are designed to differentiate between positive and neighbor samples. Following
this, we will explore the influence of mixing positive and neighbor samples on
the model, elucidating how the mixture mechanism of MNN enhances the
optimization between $z^{2}$ and $p^{1}$. These analyses will gradually unveil
the various factors at play within the MNN model. Finally, we will apply the
proposed WSE and mixture operation with other methods to verify the
generalization of the proposed approach.

(a) Inconsistency

(b) Entropy

(c) Purity

Figure 4: Details of using the Cross-Attention Score in the MSF approach. Fig.
(a) illustrates the inconsistency across different datasets. Fig. (b) presents
the entropy of the CAS distribution across different datasets. Fig. (c) shows
the variation in purity across different datasets. Although purity varies
significantly across datasets, it is noteworthy that the entropy of the CAS
distribution remains relatively high. This suggests that CAS may assign higher
weights to false neighbors, thereby confusing the model in task optimization.

####  4.4.1 Diverse strategies for weight adjustment

By assigning appropriate weights to different subtasks within the loss
function, we can adapt the focus of the model to better fit the parameters. In
this context, we aim to substitute the weight adjustment methods of MNN and
analyze how different weight adjustment techniques affect the model. To
achieve this, we perform experiments using the Mean Squared Error from MSF
[10] and the Cross-Attention Score, which comprises identity mappings without
parameters, as employed in SNCLR [12]:

$\displaystyle w_{i}^{MSE}$ $\displaystyle=\frac{1}{K+1}\ ,$ (5a)
$\displaystyle w_{i}^{CAS}$ $\displaystyle=\begin{cases}1,&\ i=0\\\
\frac{1}{{\textstyle\gamma_{i}}}\frac{{\textstyle
exp(cos(z^{2}_{i},q^{1}))}}{{\textstyle\sum_{k=1}^{K}exp(cos(z^{2}_{k},q^{1}))}},&\mbox{
otherwise }\ ,\end{cases}$ (5b)

where $q^{1}$ is the positive embedding from the student network, and
$\gamma_{i}$ represents the scaling factor. If we rearrange the ordered set of
neighbors ${\\{z^{2}_{i}\\}}_{i=1}^{K}$ using the values computed by the CAS,
we obtain a different ordered set, ${\\{z^{2^{\prime}}_{j}\\}}_{j=1}^{K}$. To
quantify the inconsistency in the order of neighbors, we define it as follows:

$\mbox{Inconsistency}=\frac{{\textstyle\sum_{k=1}^{K}\mathds{1}_{[z^{2}_{k}\neq
z^{2^{\prime}}_{k}]}}}{K}\ ,$ (6)

where $\mathds{1}_{[z^{2}_{k}\neq z^{2^{\prime}}_{k}]}\in{\\{0,1\\}}$ is an
indicator function that equals 1 when $z^{2}_{k}$ is not equal to
$z^{2^{\prime}}_{k}$.

As shown in Fig. 4(a), the observed inconsistency appears to be a common
occurrence across various datasets. This inconsistency may result from
differences in data augmentation between the student and teacher networks, as
well as the presence of false neighbors. Although it may shuffle neighbor
samples that are semantically similar to the positive samples to the front of
the ordered set (as shown in C), our analysis, presented in Fig. 4(b),
consistently reveals that the Cross Attention Score results in high levels of
entropy in the weight distribution ${\\{w^{CAS}_{i}\\}}_{i=1}^{K}$.
Additionally, in conjunction with neighbor sample purity (Fig. 4(c)), it is
evident that this high entropy level signifies the inaccuracy of the CAS, as
larger weights are assigned to False Nearest-Neighbors. The lower section of
Table 11, excluding the mixture component, corroborates our observation: the
weight adjustment approach in WSE markedly improves model performance compared
to CAS.

Table 11: Analysis of crucial components for MNN. We systematically
investigate the effect of different components in MNN on the accuracy of
linear evaluation. The highlights indicate changes compared to the default MNN
settings. The lower part shows the superiority of our WSE over the CAS. In the
upper part, we can clearly observe that the mixture operation significantly
improves the accuracy of linear evaluation across all weight adjustment
approaches. The ∗ symbol indicates that these results were obtained through
the reproduction of the official code.

Method | MSE | CAS | WSE | Mixture | CIFAR-10 | CIFAR-100 | STL-10 | Tiny ImageNet
---|---|---|---|---|---|---|---|---
MNN | ✗ | ✗ | ✓ | ✓ | 91.47 | 67.56 | 91.61 | 50.70
| ✗ | ✓ | ✗ | ✓ | 91.33 | 67.32 | 91.20 | 50.06
| ✓ | ✗ | ✗ | ✓ | 91.20 | 65.16 | 90.08 | 47.16
| ✗ | ✗ | ✓ | ✗ | 91.40 | 65.55 | 91.20 | 49.27
| ✗ | ✓ | ✗ | ✗ | 90.91 | 63.48 | 91.13 | 47.32
MSF∗ [10] | ✓ | ✗ | ✗ | ✗ | 89.94 | 59.94 | 88.05 | 42.68

MSE: Mean Squared Error, CAS: Cross-Attention Score, WSE: Weighted Squared
Error

####  4.4.2 The meaning of mixture

The primary goal of incorporating the mixture operation in MNN is to mitigate
the noise introduced by false neighbors. As is evident from the upper section
of Table 11, the inclusion of the mixture component significantly enhances
accuracy across various weight adjustment methods. B illustrates that the
mixture operation in MNN refines the optimization process between $z^{2}$ and
$p^{1}$, enhancing the performance of the model with neighbor samples by
considering them as additional positive samples. This analysis provides
valuable insights into the role of the mixture operation within MNN.

####  4.4.3 Approach generality

We apply Weighted Squared Error and mixture operations to other contrastive
self-supervised methods, namely CMSF and SNCLR. In SNCLR, we incorporate weak
data augmentation to maintain the semantics of the output embedding in the
teacher network branch. Notably, this adaptation consistently improves the
performance of all methods on most benchmark image datasets, as illustrated in
Table 12. Furthermore, we note that the improvement in SNCLR is relatively
modest. We hypothesize that this may be attributed to the fact that SNCLR uses
InfoNCE loss [37] thereby introducing noise disturbance from false negative
samples [38].

(a) MoCoV2

(b) MNN

Figure 5: Visualization of t-SNE features for MoCoV2 and MNN on CIFAR-10.
Table 12: Generalization of the approach. The upper section of the table
displays the linear evaluation accuracy, while the lower section corresponds
to the K-nearest neighbor classification with $K=200$. Notably, MSF and CMSF
employ MSE, whereas SNCLR uses CAS. The ∗ symbol indicates that these results
were obtained through the reproduction of the official code.

|  | MSE or CAS |  | WSE and Mixture (Ours)
---|---|---|---|---
|  | CIFAR-10 | CIFAR-100 | STL-10 | Tiny ImageNet |  | CIFAR-10 | CIFAR-100 | STL-10 | Tiny ImageNet
MSF∗ [10] |  | 89.94 | 59.94 | 88.05 | 42.68 |  | 91.47 | 67.56 | 91.61 | 50.70
CMSF∗ [11] |  | 91.00 | 62.37 | 88.21 | 44.50 |  | 91.81 | 67.72 | 91.80 | 50.73
SNCLR∗ [12] |  | 88.86 | 65.19 | 90.93 | 50.15 |  | 90.48 | 67.67 | 91.25 | 51.02
MSF∗ [10] |  | 88.24 | 52.32 | 84.09 | 35.29 |  | 89.81 | 61.96 | 86.65 | 42.28
CMSF∗ [11] |  | 89.30 | 55.57 | 84.11 | 36.79 |  | 90.08 | 61.05 | 87.59 | 42.83
SNCLR∗ [12] |  | 87.36 | 58.65 | 86.02 | 41.92 |  | 87.96 | 60.50 | 85.40 | 42.48

###  4.5 Visualization of features

We also demonstrated the semantic features acquired by our proposed method,
MNN, by performing t-SNE [39] visualization on the CIFAR-10 dataset. The
visualization results in Fig. 5 reveal that MNN exhibits distinct category
boundaries while maintaining compact embeddings within categories.

##  5 Conclusion

In this work, we introduced MNN, a simple self-supervised visual
representation learning framework that enhances the diversity of semantic
features learned by the model. MNN employs a straightforward loss function and
incorporates a mixture operation to efficiently include nearest neighbors in
instance discrimination tasks. Our experiments highlight the ability of the
MNN to effectively reduce the impact of false neighbors on the model with
minimal computational overhead. In the future, we plan to extend the concept
of mixture operations to various unsupervised learning scenarios, particularly
those involving imbalanced or fine-grained datasets.

## Acknowledgement

This work was supported by the National Natural Science Foundation of China
under Grant No. 61906098.

## References

  * [1] T. Chen, S. Kornblith, M. Norouzi, G. Hinton, A simple framework for contrastive learning of visual representations, in: International Conference on Machine Learning, 2020, pp. 1597–1607.
  * [2] K. He, H. Fan, Y. Wu, S. Xie, R. Girshick, Momentum contrast for unsupervised visual representation learning, in: IEEE Conference on Computer Vision and Pattern Recognition, 2020, pp. 9729–9738.
  * [3] M. Noroozi, P. Favaro, Unsupervised learning of visual representations by solving jigsaw puzzles, in: European Conference on Computer Vision, 2016, pp. 69–84.
  * [4] R. Zhang, P. Isola, A. A. Efros, Colorful image colorization, in: European Conference on Computer Vision, 2016, pp. 649–666.
  * [5] C. Doersch, A. Gupta, A. A. Efros, Unsupervised visual representation learning by context prediction, in: International Conference on Computer Vision, 2015, pp. 1422–1430.
  * [6] Z. Wu, Y. Xiong, S. X. Yu, D. Lin, Unsupervised feature learning via non-parametric instance discrimination, in: IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3733–3742.
  * [7] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, P. Isola, What makes for good views for contrastive learning?, in: Advances in Neural Information Processing Systems, 2020, pp. 6827–6839.
  * [8] D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, A. Zisserman, With a little help from my friends: Nearest-neighbor contrastive learning of visual representations, in: International Conference on Computer Vision, 2021, pp. 9588–9597.
  * [9] M. Gutmann, A. Hyvärinen, Noise-contrastive estimation: A new estimation principle for unnormalized statistical models, in: Artificial Intelligence and Statistics, 2010, pp. 297–304.
  * [10] S. A. Koohpayegani, A. Tejankar, H. Pirsiavash, Mean shift for self-supervised learning, in: International Conference on Computer Vision, 2021, pp. 10326–10335.
  * [11] K. Navaneet, S. Abbasi Koohpayegani, A. Tejankar, K. Pourahmadi, A. Subramanya, H. Pirsiavash, Constrained mean shift using distant yet related neighbors for representation learning, in: European Conference on Computer Vision, 2022, pp. 23–41.
  * [12] C. GE, J. Wang, Z. Tong, S. Chen, Y. Song, P. Luo, Soft neighbors are positive supporters in contrastive visual representation learning, in: International Conference on Learning Representations, 2023, pp. 1–16.
  * [13] Z. Shen, Z. Liu, Z. Liu, M. Savvides, T. Darrell, E. Xing, Un-mix: Rethinking image mixtures for unsupervised visual representation learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2022, pp. 2216–2224.
  * [14] K. Lee, Y. Zhu, K. Sohn, C.-L. Li, J. Shin, H. Lee, $i$-mix: A domain-agnostic strategy for contrastive representation learning, in: International Conference on Learning Representations, 2021, pp. 1–19.
  * [15] Y. Kalantidis, M. B. Sariyildiz, N. Pion, P. Weinzaepfel, D. Larlus, Hard negative mixing for contrastive learning, in: Advances in Neural Information Processing Systems, 2020, pp. 21798–21809.
  * [16] M. El Banani, K. Desai, J. Johnson, Learning visual representations via language-guided sampling, in: IEEE Conference on Computer Vision and Pattern Recognition, 2023, pp. 19208–19220.
  * [17] Z. Huang, J. Chen, J. Zhang, H. Shan, Learning representation for clustering via prototype scattering and positive sampling, in: IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, pp. 7509–7524.  doi:10.1109/TPAMI.2022.3216454.
  * [18] U. Ozbulak, H. J. Lee, B. Boga, E. T. Anzaku, H. min Park, A. V. Messem, W. D. Neve, J. Vankerschaver, Know your self-supervised learning: A survey on image-based generative and discriminative training, in: Transactions on Machine Learning Research, 2023, pp. 1–45.
  * [19] N. Komodakis, S. Gidaris, Unsupervised representation learning by predicting image rotations, in: International Conference on Learning Representations, 2018, pp. 1–16.
  * [20] X. Chen, K. He, Exploring simple siamese representation learning, in: IEEE Conference on Computer Vision and Pattern Recognition, 2021, pp. 15750–15758.
  * [21] X. Shen, D. Sun, S. Pan, X. Zhou, L. T. Yang, Neighbor contrastive learning on learnable graph augmentation, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2023, pp. 9782–9791.
  * [22] X. Chen, H. Fan, R. Girshick, K. He, Improved baselines with momentum contrastive learning, in: arXiv:2003.04297, 2020.
  * [23] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, M. Valko, Bootstrap your own latent: a new approach to self-supervised learning, in: Advances in Neural Information Processing Systems, 2020, pp. 21271–21284.
  * [24] S. McCann, D. G. Lowe, Local naive bayes nearest neighbor for image classification, in: IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 3650–3656.
  * [25] S. Yang, J. van de Weijer, L. Herranz, S. Jui, et al., Exploiting the intrinsic neighborhood structure for source-free domain adaptation, in: Advances in Neural Information Processing Systems, 2021, pp. 29393–29405.
  * [26] H. Zhang, M. Cisse, Y. N. Dauphin, D. Lopez-Paz, mixup: Beyond empirical risk minimization, in: International Conference on Learning Representations, 2018, pp. 1–13.
  * [27] S. Kim, G. Lee, S. Bae, S.-Y. Yun, Mixco: Mix-up contrastive learning for visual representation, in: arXiv preprint arXiv:2010.06300, 2020.
  * [28] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, in: International Conference on Learning Representations, 2021, pp. 1–21.
  * [29] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255.
  * [30] A. Krizhevsky, G. Hinton, et al., Learning multiple layers of features from tiny images, 2009, pp. 1–60.
  * [31] A. Coates, A. Ng, H. Lee, An analysis of single-layer networks in unsupervised feature learning, in: Artificial Intelligence and Statistics, 2011, pp. 215–223.
  * [32] Y. Le, X. Yang, Tiny imagenet visual recognition challenge, in: CS 231N, 2015.
  * [33] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
  * [34] I. Loshchilov, F. Hutter, Sgdr: Stochastic gradient descent with warm restarts, in: arXiv preprint arXiv:1608.03983, 2016.
  * [35] M. Zheng, S. You, F. Wang, C. Qian, C. Zhang, X. Wang, C. Xu, Ressl: Relational self-supervised learning with weak augmentation, in: Advances in Neural Information Processing Systems, 2021, pp. 2543–2555.
  * [36] J. Denize, J. Rabarisoa, A. Orcesi, R. Hérault, S. Canu, Similarity contrastive estimation for self-supervised soft contrastive learning, in: Winter Conference on Applications of Computer Vision, 2023, pp. 2706–2716.
  * [37] A. v. d. Oord, Y. Li, O. Vinyals, Representation learning with contrastive predictive coding, in: arXiv preprint arXiv:1807.03748, 2018.
  * [38] N. Saunshi, O. Plevrakis, S. Arora, M. Khodak, H. Khandeparkar, A theoretical analysis of contrastive unsupervised representation learning, in: International Conference on Machine Learning, 2019, pp. 5628–5637.
  * [39] L. Van der Maaten, G. Hinton, Visualizing data using t-sne., in: Journal of Machine Learning Research, 2008, pp. 2579–2605.

##  Appendix A Pseudo-code

In this section, we provide the PyTorch-style pseudo-code for MNN (as shown in
Algorithm 1). Our code is publicly available at https://github.com/pc-cp/MNN.
Notably, in comparison to MSF, MNN introduces only minimal computational
overhead, as highlighted in the pseudo-code. However, this minor adjustment
results in substantial performance enhancements.

# $\mathtt{F_{s}}$, $\mathtt{F_{t}}$: encoder for student, teacher,
$\mathtt{F\\_\triangleq g\\_(f\\_(\cdot))}$

# $\mathtt{h_{s}}$: predictor for student

# $\mathcal{S}$: support set(CxQ)

# m: momentum for teacher

# topk: number of nearest-neighbors

$\mathtt{F_{t}}$.params = $\mathtt{F_{s}}$.params # initialize
$\mathtt{teacher}$

# load a minibatch X with N samples

for X in loader:

# random augmentation

$\mathtt{X_{s}}$, $\mathtt{X_{t}}$ = strong_aug(X), weak_aug(X)

$\mathtt{P_{s}}$ = $\mathtt{h_{s}}$($\mathtt{F_{s}}$($\mathtt{X_{s}}$)) # NxC

$\mathtt{Z_{t}}$ = $\mathtt{F_{t}}$($\mathtt{X_{t}}$) # NxC

# topK nearest-neighbors lookup

$\mathtt{NN_{t}}$ = NN($\mathtt{Z_{t}}$, $\mathcal{S}$, topK) # (topKxN)xC

$\mathtt{Z_{topK}}$ = $\mathtt{Z_{t}}$.repeat(1, topK).reshape(-1, C) #
(topKxN)xC

$\lambda$ $\sim$ Uniform(0, 1)

$\mathtt{Z_{mix}}$ = $\lambda$*$\mathtt{NN_{t}}$ \+
(1-$\lambda$)*$\mathtt{Z_{topK}}$

# $\mathtt{l_{2}-normalize}$

$\mathtt{P_{s\\_norm}}$, $\mathtt{Z_{mix\\_norm}}$, $\mathtt{Z_{t\\_norm}}$ =
normalize($\mathtt{P_{s}}$, $\mathtt{Z_{mix}}$, $\mathtt{Z_{t}}$, dim=1)

# mm: matrix multiplication

$\mathtt{dist_{nn}}$, $\mathtt{mask}$ = 2-2*mm($\mathtt{P_{s\\_norm}}$,
$\mathtt{Z_{mix\\_norm}}$), mask(N) # Nx(topKxN)

$\mathtt{dist}$ = 2-2*mm($\mathtt{P_{s\\_norm}}$, $\mathtt{Z_{t\\_norm}}$) #
NxN

$\mathtt{loss}$ = mm(dist, eye(N)).sum(dim=1) + mm($\mathtt{dist_{nn}}$,
mask).sum(dim=1)/topK

loss.backward()

# SGD update: student

update($\mathtt{F_{s}}$.params)

# momentum update: $\mathtt{teacher}$

$\mathtt{F_{t}}$.params =
$\mathtt{m}$*$\mathtt{F_{t}}$.params+(1-$\mathtt{m}$)*$\mathtt{F_{s}}$.params

# update support set

enqueue($\mathcal{S}$, $\mathtt{Z_{t}}$)

dequeue($\mathcal{S}$)

def mask(N):

$\mathtt{mask}$ = eye(N).repeat(topK, 1).reshape(topK, N, -1))

return mask.permute(2, 1, 0).reshape(N, (topKxN))

Algorithm 1 Pseudo-code of MNN (asymmetric) in a PyTorch-like style.

##  Appendix B Analysis of the mixture

$\displaystyle Loss_{x}$
$\displaystyle={\textstyle\sum_{i=0}^{K}w_{i}*||p^{1}-\widetilde{z_{i}^{2}}||_{2}^{2}}$
(7)
$\displaystyle=\underset{(B.1-1)}{\underbrace{||p^{1}-z^{2}||_{2}^{2}}}+\frac{1}{K}{\textstyle\sum_{i=1}^{K}\underset{(B.1-2)}{\underbrace{||p^{1}-\widetilde{z_{i}^{2}}||_{2}^{2}}}}$

First, we rewrite Eq. (B.1-2) based on the
$\widetilde{z_{i}^{2}}=\lambda*z_{i}^{2}+(1-\lambda)*z^{2}$:

$\displaystyle(B.1-2)$
$\displaystyle=||p^{1}-\lambda*z_{i}^{2}-(1-\lambda)*z^{2}||_{2}^{2}$ (8)
$\displaystyle=||\lambda*(p^{1}-z_{i}^{2})+(1-\lambda)*(p^{1}-z^{2})||_{2}^{2}$
$\displaystyle=\lambda^{2}*\underset{(B.2-1)}{\underbrace{||p^{1}-z_{i}^{2}||_{2}^{2}}}+(1-\lambda)^{2}*\underset{(B.2-2)}{\underbrace{||p^{1}-z^{2}||_{2}^{2}}}+\underset{(B.2-3)}{\underbrace{2\lambda(1-\lambda)(p^{1}-z_{i}^{2})^{T}(p^{1}-z^{2})}}$

We ignore Eq. (B.2-3) to simplify Eq. (B.1-2):

$\displaystyle(B.1-2)$
$\displaystyle=||p^{1}-\lambda*z_{i}^{2}-(1-\lambda)*z^{2}||_{2}^{2}$ (9)
$\displaystyle\approx\lambda^{2}*||p^{1}-z_{i}^{2}||_{2}^{2}+(1-\lambda)^{2}*||p^{1}-z^{2}||_{2}^{2}$

Finally, we can obtain a simplified version of $Loss_{x}$:

$\displaystyle Loss_{x}$
$\displaystyle={\textstyle\sum_{i=0}^{K}w_{i}*||p^{1}-\widetilde{z_{i}^{2}}||_{2}^{2}}$
(10)
$\displaystyle=||p^{1}-z^{2}||_{2}^{2}+\frac{1}{K}{\textstyle\sum_{i=1}^{K}||p^{1}-\widetilde{z_{i}^{2}}||_{2}^{2}}$
$\displaystyle\approx||p^{1}-z^{2}||_{2}^{2}+\frac{1}{K}{\textstyle\sum_{i=1}^{K}\left\\{\lambda^{2}*||p^{1}-z_{i}^{2}||_{2}^{2}+(1-\lambda)^{2}*||p^{1}-z^{2}||_{2}^{2}\right\\}}$
$\displaystyle=||p^{1}-z^{2}||_{2}^{2}+\frac{\lambda^{2}}{K}{\textstyle\sum_{i=1}^{K}||p^{1}-z_{i}^{2}||_{2}^{2}}+\frac{(1-\lambda)^{2}}{K}{\textstyle\sum_{i=1}^{K}||p^{1}-z^{2}||_{2}^{2}}$
$\displaystyle=\underset{(B.4-1)}{\underbrace{(1+(1-\lambda)^{2})}}*||p^{1}-z^{2}||_{2}^{2}+\underset{(B.4-2)}{\underbrace{\frac{\lambda^{2}}{K}}}*{\textstyle\sum_{i=1}^{K}||p^{1}-z_{i}^{2}||_{2}^{2}}$

Since $\lambda\in[0,1]$, we can observe $1\leq(B.4-1)\leq 2$ as well as
$0\leq(B.4-2)\leq 1/K$. This implies that in MNN, the mixture operation
further distinguishes the contribution of positive and neighbor samples to the
model by the mixed coefficient $\lambda$. To delve into this effect, we fixed
different values of $\lambda$ in MNN and conducted a detailed study of its
impact on model performance. Table 13 shows the linear classification results
corresponding to different $\lambda$ values. Remarkably, we find that the
model performs better when $\lambda$ takes a medium value.

Table 13: Linear evaluation results of MNN with varying $\lambda$ values. $\lambda$ | CIFAR-10 | CIFAR-100 | STL-10 | Tiny ImageNet
---|---|---|---|---
$0.1$ | 90.46 | 67.25 | 91.38 | 49.15
$0.3$ | 91.15 | 67.25 | 91.40 | 49.93
$0.5$ | 91.27 | 67.56 | 91.21 | 50.31
$0.7$ | 91.47 | 66.52 | 91.75 | 49.26
$0.9$ | 91.23 | 66.66 | 91.18 | 49.08
Figure 6: The curves of the loss function and linear evaluation results on MNN
and its variants without the mixture operation.

In addition, Fig. 6 illustrates the curves of the loss function values and
linear evaluation results of the MNN with and without mixture. It can be
observed from the figure that the mixture operation is effective in reducing
the value of the loss function. This phenomenon may be due to the fact that
the mixture operation helps to reduce the disturbance of the noise introduced
by the false neighbors to the model.

##  Appendix C Inconsistency in the Cross-Attention Score

In this section, we delve into the critical details of introducing the Cross-
Attention Score to the MSF approach. When we rearrange the set of ordered
neighbors ${\\{z^{2}_{i}\\}}_{i=1}^{K}$ based on the CAS, we obtain another
ordered set ${\\{z^{2^{\prime}}_{j}\\}}_{j=1}^{K}$. Fig. 7 illustrates the
purity at different neighbor positions, with * denoting the purity at
positions 1, 3, and 5 in ${\\{z^{2^{\prime}}_{j}\\}}_{j=1}^{K}$ (for visual
clarity, we exclude the purity values at positions 2 and 4). Without *, it
represents the purity at corresponding positions in the set
${\\{z^{2}_{i}\\}}_{i=1}^{K}$.

We observe that the CAS-induced inconsistency results in pushing samples with
similar semantics to the positive embedding towards the front of the order,
while false neighbors are placed towards the back. This phenomenon may
contribute to the performance improvement seen when introducing the CAS to the
MSF framework. However, the higher entropy of the weight distribution obtained
by the CAS also suggests that it may assign larger weights to the false
neighbors. This, in turn, can perplex the model during optimization since the
weight assignments do not consistently reflect the actual similarity between
samples.

Figure 7: Purity of different positions before and after reordering the set of
neighbors using Cross Attention Scores. The $*$ symbol denotes the sorted
purity. Notably, the sample in the 1st position at the front exhibits
increased purity, while the sample in the 5th position at the back (in the
default case with $K=5$) exhibits a corresponding decrease. This observation
partially explains why the CAS improves the performance of the MSF. However,
such modules may also assign higher weights to false neighbors due to the
inconsistency in the CAS.

Xianzhong Long received his Ph.D. degree from the Department of Computer
Science and Engineering at Shanghai Jiao Tong University in Shanghai, China.
He is currently an associate professor at the School of Computer Science,
Nanjing University of Posts and Telecommunications, Nanjing, China. His
current research interests include self-supervised learning and adversarial
machine learning.

Chen Peng received his B.S. degree in Computer Science and Technology from
Nanjing University of Posts and Telecommunications. He is pursuing his M.S.
degree in the School of Computer Science, at Nanjing University of Posts and
Telecommunications, Nanjing, China. His current research interests include
self-supervised learning, machine learning, and computer vision.

Yun Li received his Ph.D. degree in computer science from Chongqing
University, Chongqing, China. He was a Post-Doctoral Fellow with the
Department of Computer Science and Engineering, at Shanghai Jiao Tong
University, Shanghai, China. He is a Professor at the School of Computer
Science, Nanjing University of Posts and Telecommunications, Nanjing, China.
He has published more than 60 refereed research papers. His current research
interests include machine learning, data mining, and parallel computing.