Mohammadreza Sadeghimohammadreza.sadeghi@mcgill.ca1,2 Hadi
Hojjatihadi.hojjati@mcgill.ca1,2 Narges Armanfardnarges.armanfard@mcgill.ca1,2
Department of Electrical and Computer Engineering, McGill University
Montreal, QC, Canada
Mila - Quebec AI Institute
Montreal, QC, Canada C3:Cross-instance guided Contrastive Clustering

# C3: Cross-instance guided Contrastive Clustering

###### Abstract

Clustering is the task of gathering similar data samples into clusters without
using any predefined labels. It has been widely studied in machine learning
literature, and recent advancements in deep learning have revived interest in
this field. Contrastive clustering (CC) models are a staple of deep clustering
in which positive and negative pairs of each data instance are generated
through data augmentation. CC models aim to learn a feature space where
instance-level and cluster-level representations of positive pairs are grouped
together. Despite improving the SOTA, these algorithms ignore the cross-
instance patterns, which carry essential information for improving clustering
performance. This increases the false-negative-pair rate of the model while
decreasing its true-positive-pair rate. In this paper, we propose a novel
contrastive clustering method, Cross-instance guided Contrastive Clustering
(C3), that considers the cross-sample relationships to increase the number of
positive pairs and mitigate the impact of false negative, noise, and anomaly
sample on the learned representation of data. In particular, we define a new
loss function that identifies similar instances using the instance-level
representation and encourages them to aggregate together. Moreover, we propose
a novel weighting method to select negative samples in a more efficient way.
Extensive experimental evaluations show that our proposed method can
outperform state-of-the-art algorithms on benchmark computer vision datasets:
we improve the clustering accuracy by $6.6\%$, $3.3\%$, $5.0\%$, $1.3\%$ and
$0.3\%$ on CIFAR-10, CIFAR-100, ImageNet-10, ImageNet-Dogs, and Tiny-ImageNet.

##  1 Introduction

Figure 1: t-SNE visualization of Contrastive Clustering (CC) [Li et
al.(2021)Li, Hu, Liu, Peng, Zhou, and Peng] and the proposed C3

In the past few years, self-supervised learning [Feng et al.(2019)Feng, Xu,
and Tao, Lin et al.(2021)Lin, Qi, Zhengyang, and Changhu, Tiwari et
al.(2021)Tiwari, Chen, Tsai, Kuo, Chen, Jou, Venkatesh, and Chen],
particularly contrastive learning, has established itself as a state-of-the-
art representation learning algorithm [Chen et al.(2020)Chen, Kornblith,
Norouzi, and Hinton]. They generate a transformed version of the input samples
and attempt to learn a representation in which augmentations of the same data
point are closer together and further away from other points. They managed to
outperform other baselines, particularly in tasks related to computer vision,
such as image classification [Chen et al.(2020)Chen, Kornblith, Norouzi, and
Hinton], image anomaly detection [Hojjati et al.(2022)Hojjati, Ho, and
Armanfard], and object recognition [Xie et al.(2021)Xie, Ding, Wang, Zhan, Xu,
Sun, Li, and Luo]. Encouraged by these results, several studies have attempted
to apply contrastive learning to the clustering task. An early attempt by Li
_et al_[Li et al.(2021)Li, Hu, Liu, Peng, Zhou, and Peng] showed that
contrastive clustering could significantly outperform other baselines on
benchmark datasets. Despite these improvements, contrastive clustering and the
majority of other deep clustering methods do not consider the
interrelationship between data samples, which commonly leads to sub-optimal
clustering [Sadeghi and Armanfard(2021a)]. However, it would be challenging to
incorporate information about the cross-relationship of different instances in
an unsupervised setting where we do not have access to data labels. As a
result, existing unsupervised CC-based algorithms suffer from high false-
negative-pair and low true-positive-pair rates because the model cannot
effectively take advantage of the degree of similarity of samples.

In this paper, we propose a groundbreaking technique to discover similarities
between samples. Then, we employ these similarities to identify positive and
negative pairs more accurately. We design a soft-weighting scheme that allows
focusing on the more challenging to cluster data points. To realize this goal,
we create a pool of weighted sample pairs where higher weights are assigned to
the samples closer to the data cluster boundaries. Such a weighting scheme
mitigates the influence of easy-to-cluster data, noise, and, more importantly,
the existing false-negative-pair issue in the CC-based methods. Also, by
incorporating a larger number of positive samples, we improve the true-
positive-pair rate. Overall, our method significantly improves the training
efficiency of the model and leads to a cluster-friendly representation. In the
remainder of this paper, we describe our idea more formally. Then, we carry
out a series of extensive analyses to show that our scheme can significantly
improve the clustering performance, and we try to explain how it can achieve
such enhancement.

We summarize the contribution of this work as follows: (1) we propose a new
contrastive loss function to incorporate the newly discovered positive pairs
toward learning a more reliable representation space. (2) we propose a novel
weighting scheme that aims to separate more challenging data samples,
specifically those that are close to cluster boundaries. This improves the
representation learning process and greatly impacts the false-negative-pair
selection rate. (3) by carrying out extensive experiments, we show that our
proposed scheme can significantly outperform current state-of-the-art by a
significant margin in terms of several clustering criteria. This significant
improvement results from considering the pairwise data similarities. (4) We
offer insight into the behavior of our developed model, discuss the intuition
behind how it improves the clustering performance and support them by
conducting relevant experiments.

##  2 Related Works

Deep learning-based clustering methods can be categorized into two groups
[Zhou et al.(2022)Zhou, Xu, Zheng, Chen, Li, Bu, Wu, Wang, Zhu, and Ester]:
(I) Models that use deep networks for embedding the data into a lower-
dimensional representation and apply a traditional clustering such as k-means
to the new representation, and (II) Algorithms that jointly train the neural
network for extracting features and optimizing the clustering results. In
order to achieve a more clustering-friendly representation, previous studies
have added regularization terms and constraints to the loss function of neural
networks. For example, Huang _et al_[Huang et al.(2014)Huang, Huang, Wang, and
Wang] proposed the Deep embedding network (DEN), which imposes a locality
preserving and a group sparsity constraint to the latent representation of the
autoencoder. These two constraints reduce the inner cluster and increase the
inter-cluster distances to improve the clustering performance. In another
work, Peng _et al_[Peng et al.(2016)Peng, Xiao, Feng, Yau, and Yi] proposed
deep subspace clustering with sparsity prior (PARTY) that enhances the
clustering efficiency of the autoencoder by incorporating the structure’s
prior in order to consider the relationship between different samples. Sadeghi
and Armanfard [Sadeghi and Armanfard(2022)] proposed Deep Multi-Representation
Learning for Data Clustering (DML), which uses a general autoencoder for
instances that are easily clustered along separate AEs for difficult-to-
cluster data to improve the performance. More recent works jointly train the
neural network with the clustering objective to further improve the clustering
performance. For instance, Deep clustering network (DCN) [Yang et
al.(2017)Yang, Fu, Sidiropoulos, and Hong] uses k-means objective as the
clustering loss and jointly optimizes it with the loss of an autoencoder.
Analogously, Deep embedded clustering (DEC) [Xie et al.(2016)Xie, Girshick,
and Farhadi] first embeds the data into a lower-dimensional space by
minimizing the reconstruction loss. Then, it iteratively updates the encoder
part of the AE by optimizing a Kullback-Leiber (KL) divergence [Kullback and
Leibler(1951)] loss between the soft assignments and adjusted target
distributions. Following the success of DEC, a series of improved algorithms
have been developed. For instance, improved deep embedded clustering with
local structure preservation (IDEC) [Guo et al.(2017)Guo, Gao, Liu, and Yin]
jointly optimizes the clustering loss and AE loss to preserve the local
structure of data, IDECF [Sadeghi and Armanfard(2021b)] adds a fuzzy c-mean
network for improving the auxiliary cluster assignment of IDEC during
training, and Deep embedded clustering with data augmentation (DEC-DA) [Guo et
al.(2018)Guo, Zhu, Liu, and Yin] applies the DEC method along with the data
augmentation strategy to improve the performance. Several other methods design
auxiliary tasks for learning an efficient representation. E.g., JULE [Yang et
al.(2016)Yang, Parikh, and Batra] applies agglomerative clustering to learn
the data representation and cluster assignments. In another algorithm named
invariant information clustering (IIC) [Ji et al.(2019)Ji, Henriques, and
Vedaldi], the mutual information between the cluster assignment of a pair is
maximized. Recently, researchers have turned their attention to self-
supervised learning (SSL) models for clustering. For example, MMDC (multi-
modal deep clustering) [Shiran and Weinshall(2020)] improves the clustering
accuracy by solving the proxy task of predicting the rotation. SCAN (semantic
clustering by adopting nearest neighbors) [Van Gansbeke et al.(2020)Van
Gansbeke, Vandenhende, Georgoulis, Proesmans, and Van Gool] first obtains a
high-level feature representation using self-supervised learning and then
improves the clustering performance by incorporating the nearest neighbor
prior. Contrastive learning is a self-supervised learning paradigm that learns
data representation by minimizing the distance between the augmentations of
the same sample while pushing them away from other instances. SimCLR [Chen et
al.(2020)Chen, Kornblith, Norouzi, and Hinton] is an example of a contrastive
model for learning representation from images that can achieve performance on
par with supervised methods. Researchers have increasingly utilized
contrastive models for solving tasks such as clustering in the past couple of
years. Zhong _et al_ proposed deep robust clustering (DRC) [Zhong et
al.(2020)Zhong, Chen, Jin, and Hua] in which a contrastive loss decreases the
inter-class variance and another contrastive loss increases the intra-class
distance. Contrastive clustering (CC) [Li et al.(2021)Li, Hu, Liu, Peng, Zhou,
and Peng] improves the clustering performance by jointly performing the
instance and cluster-level contrastive learning.

Figure 2: An overview of the training phase of our proposed C3 method.

##  3 Method

Given an unlabelled dataset $\mathcal{X}=\\{x_{1},x_{2},\dots,x_{N}\\}$ and a
predefined cluster number parameter M, the goal of the clustering problem is
to partition $\mathcal{X}$ into M disjoint groups. Figure 2. shows the overall
scheme of the C3 framework.

Like other contrastive learning methods, we apply two data augmentations
$\mathcal{T}^{a}$ and $\mathcal{T}^{b}$, sampled randomly from a pool of
transformations, $\mathcal{T}$, to form a true positive pair ($x_{i}^{a}$,
$x_{i}^{b}$) for a sample $x_{i}$, where $x_{i}^{a}=\mathcal{T}^{a}(x_{i})$
and $x_{i}^{b}=\mathcal{T}^{b}(x_{i})$. In this paper, we used SimCLR
transformation pool [Chen et al.(2020)Chen, Kornblith, Norouzi, and Hinton].
As is shown in Figure 2. , we use the encoder network $f(.)$ to extract
features of augmented samples, i.e, $h_{i}^{a}=f(x_{i}^{a})$ and
$h_{i}^{b}=f(x_{i}^{b})$. Inspired by CC [Li et al.(2021)Li, Hu, Liu, Peng,
Zhou, and Peng], we devise instance-level and cluster-level contrastive
networks, denoted by $g_{I}(.)$ and $g_{C}(.)$, respectively. The instance-
level network maps the extracted feature of augmented samples to the latent
representation (aka z-space), i.e. $z_{i}^{a}=g_{I}(h_{i}^{a})$ and
$z_{i}^{b}=g_{I}(h_{i}^{b})$. The output of the cluster-level network is the
cluster assignments of samples to different clusters, i.e.
$c_{i}^{a}=g_{C}(h_{i}^{a})$ and $c_{i}^{b}=g_{C}(h_{i}^{b})$. We call the
output of the cluster-level network c-space. We first initialize our networks,
i.e. $f(.)$, $g_{I}(.)$, and $g_{C}(.)$ using the CC algorithm.

If we do such initialization process for a sufficient number of epochs, a
partially reliable z-space will be obtained. However, the z-space obtained by
minimizing the CC loss is sub-optimal for clustering, which results in a large
false-negative-pair rate and a low true-positive-pair rate. To mitigate this
issue, our method incorporates cross-sample similarities.

Since our framework is unsupervised and we do not have access to the data
labels to identify samples belonging to the same cluster, we use a notion of
self-supervision to refine clusters. We employ the cross-sample similarities
in the partially trained z-space to realize the self-supervision concept. The
cross-sample similarity is measured by the cosine distance of samples in the
z-space. If, for a pair of instances, the similarity is greater than or equal
to a threshold $\zeta$, we consider those samples to be similar and pull them
closer together by minimizing the loss function $\mathcal{L}_{C3}$ defined
below:

$\mathcal{L}_{C3}=\frac{1}{2N}\sum_{i=1}^{N}(\tilde{\ell}_{i}^{a}+\tilde{\ell}_{i}^{b})$
(1)
$\tilde{\ell}_{i}^{a}=-\log\frac{\sum_{k\in\\{a,b\\}}\sum_{j=1}^{N}\mathrm{\mathds{1}\\{z_{i}^{a^{\intercal}}z_{j}^{k}\geq\zeta\\}\exp(z_{i}^{a^{\intercal}}z_{j}^{k})}}{\sum_{k\in\\{a,b\\}}\sum_{j=1}^{N}\mathrm{w_{ij}^{k}\exp(z_{i}^{a^{\intercal}}z_{j}^{k})}},$
(2)

where $\mathds{1}\\{.\\}$ denotes the indicator function. Analogous to
$\tilde{\ell}_{i}^{a}$, we define $\tilde{\ell}_{i}^{b}$ that considers
similarity of $z_{i}^{b}$ and other samples in the batch. Furthermore, in the
denominator of the proposed loss function, we included $w_{ij}^{k}$ to
consider higher weights for the samples that are neither close together nor
far from each other. In this way, we realize the goal of decreasing the false
negative pair selection rate as, in the traditional CC-based methods, all
augmented samples in the batch are equally considered when forming negative
pairs, regardless of the possibility of them belonging to the same cluster and
the difficulty of the samples to cluster.

We assume that the weight terms are given when minimizing the C3 loss defined
in (2). To obtain an optimum value for a weight term $w_{ij}^{k}$, we propose
solving the below optimization problem while the networks are frozen. The
first term of the below optimization problem is defined based on our
motivation to assign a very low weight to too close and too far away samples
and, instead, let the remaining samples, which are more probably located on
the cluster boundaries, take higher weights. The second term is to avoid the
trivial solution of assigning a weight equal to one to the sample providing
the maximum value of $1-|z_{i}^{a^{\intercal}}z_{j}^{k}|$. To avoid
instability, we include the constraint by which the summation of all weights
must be equal to 1.

$\displaystyle\min_{w_{ij}^{k}}\sum_{k\in\\{a,b\\}}\sum_{j=1}^{N}-w_{ij}^{k}($
$\displaystyle
1-|z_{i}^{a^{\intercal}}z_{j}^{k}|)-\frac{1}{\Gamma}\,\,\text{H}(W_{i})$
$\displaystyle s.t.\sum_{k\in\\{a,b\\}}\sum_{j=1}^{N}w_{ij}^{k}=1$ (3)

In the above equation, $\Gamma$ is a hyperparameter,
$W_{i}=\\{w_{ij}^{k}|j\in\\{1,2,..,N\\},k\in(a,b)\\}$ is the set of all
weights, and $\text{H}(.)$ is the entropy function. We solve the optimization
problem defined in (3) using the Lagrange multiplier technique as below:

$\displaystyle
L=\sum_{k\in\\{a,b\\}}\sum_{j=1}^{N}-w_{ij}^{k}(1-|z_{i}^{a^{\intercal}}z_{j}^{k}|)+\frac{1}{\Gamma}\sum_{k\in\\{a,b\\}}\sum_{j=1}^{N}w_{ij}^{k}\log(w_{ij}^{k})+\lambda(\sum_{k\in\\{a,b\\}}\sum_{j=1}^{N}w_{ij}^{k}-1)$
(4) $\displaystyle\frac{\partial L}{\partial w_{ij}^{k}}=0\xrightarrow[]{}$
$\displaystyle-(1-|z_{i}^{a^{\intercal}}z_{j}^{k}|)+\frac{1}{\Gamma}\log(w_{ij}^{k})+\frac{1}{\Gamma}+\lambda=0$
$\displaystyle
w_{ij}^{k}=\exp(\Gamma(1-|z_{i}^{a^{\intercal}}z_{j}^{k}|)-1-\Gamma\lambda)$
(5)

Where $\lambda$ is the Lagrange multiplier. By substituting (5) into the
constraint of (3), we have:

$\displaystyle\sum_{k\in\\{a,b\\}}\sum_{j=1}^{N}w_{ij}^{k}=1\xrightarrow[]{}\sum_{k\in\\{a,b\\}}\sum_{j=1}^{N}\exp(\Gamma(1-|z_{i}^{a^{\intercal}}z_{j}^{k}|)-1-\Gamma\lambda)=1$
$\displaystyle\exp(-1-\Gamma\lambda)=\frac{1}{\sum_{k\in\\{a,b\\}}\sum_{j=1}^{N}\exp(\Gamma(1-|z_{i}^{a^{\intercal}}z_{j}^{k}|))}$
(6)

If we substitute (6) to (5), we have the final values for $w_{ij}^{k}$ as
below:

$\displaystyle
w_{ij}^{k}=\frac{\exp(\Gamma(1-|z_{i}^{a^{\intercal}}z_{j}^{k}|))}{\sum_{k\in\\{a,b\\}}\sum_{j=1}^{N}\exp(\Gamma(1-|z_{i}^{a^{\intercal}}z_{j}^{k}|))}$
(7)

When comparing the loss function of other contrastive clustering algorithms,
such as CC, with the loss of C3 (shown in Eq. (2)), several differences become
apparent. Firstly, while other contrastive algorithms typically only allow one
positive pair to appear in the numerator, C3 enables the networks to be
trained by considering a much larger number of positive pairs. This leads to
more efficient training and better performance. Secondly, C3 adopts a
weighting scheme when creating the negative pairs; the scheme assigns lower
weights to samples that are either too close or too far from each other while
assigning higher weights to those that are more in the mixing cluster areas.
This approach reduces the impact of false positive, noisy, and anomaly samples
on the learning of representations.

##  4 Experiments and Discussions

In this section, we demonstrate the effectiveness of our proposed scheme by
conducting rigorous experimental evaluations. We evaluated our method on five
challenging computer vision benchmark datasets: CIFAR-10, CIFAR-100
[Krizhevsky et al.()Krizhevsky, Nair, and Hinton], ImageNet-10, ImageNet-Dog
[Deng et al.()Deng, Dong, Socher, Li, Li, and Fei-Fei], and Tiny-ImageNet [Le
and Yang(2015)]. For CIFAR-10 and CIFAR-100, we combined the training and test
splits. Also, for CIFAR-100, instead of 100 classes, we used the 20 super-
classes as the ground-truth. To evaluate the performance, we use three
commonly-used metrics in clustering namely clustering accuracy (ACC),
Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI) [Zhou et
al.(2022)Zhou, Xu, Zheng, Chen, Li, Bu, Wu, Wang, Zhu, and Ester].

For the sake of fair comparison, for all datasets, we used ResNet34 [He et
al.(2016)He, Zhang, Ren, and Sun] as the backbone of our encoder $f(.)$, which
is the same architecture that previous algorithms have adopted. We set the
dimension of the output of the instance-level projection head $g_{I}$ to 128,
for all datasets. The output dimension of the cluster-level contrastive head
is set to the number of classes, M, in each dataset. All networks are
initialized by the CC [Li et al.(2021)Li, Hu, Liu, Peng, Zhou, and Peng]
algorithm with the hyperparameters suggested by its authors. The Adam
optimizer with an initial learning rate of $0.00001$ and batch size of $128$
is used for C3. All networks are trained for 20 epochs. The experiments are
run on NVIDIA TESLA V100 32G GPU.

###  4.1 Comparison with State-of-the-art

Table 1: Clustering performance of different methods.

CIFAR-10 CIFAR-100 ImageNet-10 ImageNet-Dogs Tiny-ImageNet Algorithm NMI ACC
ARI NMI ACC ARI NMI ACC ARI NMI ACC ARI NMI ACC ARI K-means [Lloyd(1982)]
0.087 0.229 0.049 0.084 0.130 0.028 0.119 0.241 0.057 0.055 0.105 0.020 0.065
0.025 0.005 SC [Zelnik-manor and Perona(2004)] 0.103 0.247 0.085 0.090 0.136
0.022 0.151 0.274 0.076 0.038 0.111 0.013 0.063 0.022 0.004 AC [Gowda and
Krishna(1978)] 0.105 0.228 0.065 0.098 0.138 0.034 0.138 0.242 0.067 0.037
0.139 0.021 0.069 0.027 0.005 NMF [Cai et al.(2009)Cai, He, Wang, Bao, and
Han] 0.081 0.190 0.034 0.079 0.118 0.026 0.132 0.230 0.065 0.044 0.118 0.016
0.072 0.029 0.005 AE [Bengio et al.(2006)Bengio, Lamblin, Popovici, and
Larochelle] 0.239 0.314 0.169 0.100 0.165 0.048 0.210 0.317 0.152 0.104 0.185
0.073 0.131 0.041 0.007 DAE [Vincent et al.(2010)Vincent, Larochelle, Lajoie,
Bengio, and Manzagol] 0.251 0.297 0.163 0.111 0.151 0.046 0.206 0.304 0.138
0.104 0.190 0.078 0.127 0.039 0.007 DCGAN [Radford et al.(2016)Radford, Metz,
and Chintala] 0.265 0.315 0.176 0.120 0.151 0.045 0.225 0.346 0.157 0.121
0.174 0.078 0.135 0.041 0.007 DeCNN [Zeiler et al.(2010)Zeiler, Krishnan,
Taylor, and Fergus] 0.240 0.282 0.174 0.092 0.133 0.038 0.186 0.313 0.142
0.098 0.175 0.073 0.111 0.035 0.006 VAE [Kingma and Welling(2014)] 0.254 0.291
0.167 0.108 0.152 0.040 0.193 0.334 0.168 0.107 0.179 0.079 0.113 0.036 0.006
JULE [Yang et al.(2016)Yang, Parikh, and Batra] 0.192 0.272 0.138 0.103 0.137
0.033 0.175 0.300 0.138 0.054 0.138 0.028 0.102 0.033 0.006 DEC [Xie et
al.(2016)Xie, Girshick, and Farhadi] 0.275 0.301 0.161 0.136 0.185 0.050 0.282
0.381 0.203 0.122 0.195 0.079 0.115 0.037 0.007 DAC [Chang et al.(2017)Chang,
Wang, Meng, Xiang, and Pan] 0.396 0.522 0.306 0.185 0.238 0.088 0.394 0.527
0.302 0.219 0.275 0.111 0.190 0.066 0.017 ADC [Haeusser et al.(2019)Haeusser,
Plapp, Golkov, Aljalbout, and Cremers] - 0.325 - - 0.160 - - - - - - - - - -
DDC [Chang et al.(2019)Chang, Guo, Wang, Meng, Xiang, and Pan] 0.424 0.524
0.329 - - - 0.433 0.577 0.345 - - - - - - DCCM [Wu et al.(2019)Wu, Long, Wang,
Qian, Li, Lin, and Zha] 0.496 0.623 0.408 0.285 0.327 0.173 0.608 0.710 0.555
0.321 0.038 0.182 0.224 0.108 0.038 IIC [Ji et al.(2019)Ji, Henriques, and
Vedaldi] - 0.617 - - 0.257 - - - - - - - - - - PICA [Huang et al.(2020)Huang,
Gong, and Zhu] 0.591 0.696 0.512 0.310 0.337 0.171 0.802 0.870 0.761 0.352
0.352 0.201 0.277 0.098 0.040 GATCluster [Niu et al.(2020)Niu, Zhang, Wang,
and Liang] 0.475 0.610 0.402 0.215 0.281 0.116 0.609 0.762 0.572 0.322 0.333
0.200 - - - CC [Li et al.(2021)Li, Hu, Liu, Peng, Zhou, and Peng] 0.678*
0.770* 0.607* 0.421* 0.423* 0.261* 0.850* 0.893* 0.811* 0.436* 0.421* 0.268*
0.331* 0.137* 0.062* EDESC [Cai et al.(2022)Cai, Fan, Guo, Wang, Zhang, and
Zhang] 0.627 0.464 - 0.385 0.370 - - - - - - - - - - C3 (Ours) 0.743 0.836
0.703 0.435 0.456 0.274 0.905 0.943 0.860 0.447 0.434 0.280 0.335 0.140 0.064

Figure 3: t-SNE visualization of clusters learned by the CC and C3 methods.

Figure 4: C3’s performance vs epochs.

Table 1 shows the results of our proposed method on benchmark datasets,
compared to state-of-the-art and some common traditional clustering methods.
For CC, we run the code provided by its authors for all datasets, and the
results are indicated by (*). As is evident in this table, our proposed method
significantly outperforms all other baselines in all datasets. Quantitatively,
comparing to the second best algorithm (i.e. CC), C3 improves the ACC by
$6.6\%$, $3.3\%$, $5.0\%$, $1.3\%$ and $0.3\%$), the NMI by $6.5\%$, $1.4\%$,
$5.5\%$, $1.1\%$ and $0.4\%$, and the ARI by $9.6\%$, $1.3\%$, $4.9\%$,
$1.2\%$ and $0.2\%$, respectively on CIFAR-10, CIFAR-100, ImageNet-10,
ImageNet-Dogs, and Tiny-ImageNet. The main difference between our framework
and other baselines, such as CC, is that we exploit an additional set of
information, i.e. the similarity between samples, to further enhance the
learned representation for clustering. In our approach, we increase the number
of positive samples, which improves the true-positive-pair rate, and put more
weight on clustering the samples that are more probably located on the
boundary of the clusters, which leads to a decrease in the false-negative-pair
selection rate. We believe this is the main reason our method’s performance is
superior compared to the baselines.

As opposed to the CC method that misses the global patterns present in each
data cluster, C3 correctly tries to consider such patterns (at least
partially) by employing cross-instance data similarities. Looking at (2) and
(3), one can infer that C3 implicitly reduces the intra-cluster distance while
maximizing the inter-cluster distance, which is what an efficient grouping
technique would do in the presence of the data labels. This can be confirmed
by visualizing the clusters before and after applying the C3 loss. As Figures
1 and 3 show, the samples are fairly clustered after initialization with CC,
i.e. before the start of training using the C3 loss. However, some of the
difficult clusters are mixed in the boundaries. After initialization, clusters
are expanded with a considerable number of miss-clustered data. However, after
training with the proposed C3 method, we observe that the new cluster space is
much more reliable, and individual clusters are densely populated while being
distant from each other.

###  4.2 Convergence Analysis

Results of section 4.1 depict the superiority of our proposed scheme. Now, we
analyze C3’s convergence and the computational complexity to evaluate at what
cost it makes such an improvement over other baselines. We plotted the trend
of clustering accuracy and NMI for four datasets during the training epochs in
Figure 4. We can readily confirm that although we are just training the C3
step for $20$ epochs, the graphs quickly converge to a settling point, which
corresponds to the peak performance. Also, we can observe that both ACC and
NMI are improved throughout the C3 training phase in all datasets. The
performance at $\mathrm{epoch}=0$ corresponds to the clustering performance
after initialization with CC. These figures clearly show that C3 improves its
clustering quality and justifies the qualitative results shown in Section 4.1.

One may naively think that the better performance of C3 is because it is being
trained for 20 more epochs; note that in all our experiments, as suggested by
the CC authors, we trained the CC algorithm networks for 1000 epochs. We train
the C3 networks for 1020 epochs. We reject this argument and support it by
training the CC networks for the same number of epochs as what the C3 is
trained for, i.e. 1020 epochs. We observe that no improvement is obtained for
CC when trained for an extra 20 epochs. The result of such an experiment on
CIFAR-10 is shown in Figure 5.

###  4.3 How does C3 loss improve the clusters?

Figure 5: Performance of CC and C3 after 1020 epochs.

As we saw in Figure 3, the improvement that C3 achieves is mainly because it
is able to reduce the distance between instances of the same cluster while
repelling them from other clusters. We can justify this observation by
considering the loss function of C3, i.e. Eq. (2). In this function, the term
$\mathds{1}\\{z_{i}^{a^{\intercal}}z_{j}^{k}\geq\zeta\\}$ indicates that if
the cosine similarity of two samples is greater than $\zeta$, they should be
moved further close to each other.

Figure 6: Plot of loss and number of positive pairs versus epoch.

At the beginning of training with the C3 loss, since the networks are
initiated with CC, we can assume that the points that are very similar to each
other have a cosine distance, in the z-space, less than threshold $\zeta$, so
they will become further close by minimizing the C3 loss. For instance, take
two points $z_{1}$ and $z_{2}$ with a cosine similarity larger than $\zeta$,
and assume that $z_{3}$ has a similarity greater than $\zeta$ with $z_{1}$,
but its similarity with $z_{2}$ is smaller than $\zeta$. Therefore, according
to the loss function, $z_{1}$ and $z_{2}$, as well as $z_{1}$ and $z_{3}$, are
forced to become closer, but it is not the case for $z_{2}$ and $z_{3}$.
However, these two points will also implicitly move closer to each other
because their distance to $z_{1}$ is reduced. As the training continues, at
some point, the similarity of $z_{2}$ and $z_{3}$ also may pass the threshold
$\zeta$. Therefore, as the similar pairs move closer to each other during the
training, a series of new connections will be formed, and the cluster will
become denser. To support this hypothesis, we plotted the average of the loss
function and the average number of positive pairs of each data sample in
Figure 6-a and 6-b, respectively. We can observe that the number of positive
pairs exponentially increases during the training until it settles to form the
final clusters. Corresponding to this exponential increase, we can see that
the loss is decreasing, and the network learns a representation in which
clusters are distanced from each other while samples of each cluster are
packed together.

We can also deduct from this experiment that the number of positive pairs is
also related to the number of classes in each dataset. For example, if we have
an augmented batch size of $N=256$, for Tiny-ImageNet that has 200 classes, we
expect to have $\frac{256}{200}=1.28$ positive pairs per sample which is very
close to $1$ and it is the reason that we do not see the same sharp increasing
trend as other datasets in Tiny-ImageNet.

###  4.4 Effect of Hyperparameter $\zeta$ and $\Gamma$

Our method, C3, introduces two new hyperparameter $\zeta$ and $\Gamma$.
$\zeta$ is a threshold for identifying similar samples. Throughout the
experiments, we fixed $\zeta=0.6$, which yielded consistent results across
datasets. Now, we carry out an experiment in which we change $\zeta$ and
record the performance. Note that since
$z_{i}^{a^{\intercal}}z_{j}^{k}\in[-1,1]$, we can technically change $\zeta$
from -1 to 1. Intuitively, for a small or negative value of $\zeta$, most
points in the z-space will be considered similar, and the resulting clusters
will not be reliable. Therefore, in our experiment, we change $\zeta$ from 0.4
to 0.9 in 0.1 increments for CIFAR-10. For Tiny-ImageNet, as there are lots of
clusters, we set $\zeta\in\\{0.60,0.85,0.90,0.95\\}$. We then plot the
accuracy, NMI, average loss, and the average of positive pairs per sample. The
graphics are shown in Figure 7.

In CIFAR-10 experiments, in Figure 7-a and Figure 7-b, we see that for
$\zeta=0.4$, accuracy and NMI are indeed decreasing during the C3 training.
This is because this value of $\zeta$ is too lenient and considers the points
not in the same cluster to be similar. We can confirm this explanation by
looking at Figure 7-d. We can see that for smaller $\zeta$s, we will have more
average positive pairs per sample. As we increase the $\zeta$, we can see that
the performance begins to improve. For larger values such as $\zeta=0.9$, we
can see that the performance does not significantly change during the
training. This is because $\zeta=0.9$ is a strict threshold, and if we look at
the number of positive pairs, only a few instances are identified as similar
during the training.

Figure 7: Performance and behavior of C3 for different values of $\zeta$, for
CIFAR 10 (top row) and Tiny-ImageNet (bottom row).

Comparing the results of CIFAR-10 and Tiny-ImageNet experiments shows that the
value of $\zeta$ also depends on the number of clusters. Since we have 200
classes in Tiny-ImageNet, a smaller value of $\zeta$ might yield two or more
clusters merging together and this would decrease the accuracy. Therefore, we
should choose a more strict threshold such as $\zeta=0.9$ or $\zeta=0.95$ to
improve. In Figure 7-c and Figure 7-g, the average loss plot also conveys
interesting observations about the behaviour of $\zeta$. We can see that for
smaller values, the loss is exponentially converging to the minimum, but for
larger $\zeta$, the rate is much slower. This can be due to the fact that a
smaller $\zeta$ considers most points to be similar and of the same class, and
therefore, it can yield the trivial solution of considering all points to be
similar and mapping them into one central point. In the extreme case of
$\zeta\to 1$, C3 considers a few samples as positive pairs, and therefore, we
will not have any major improvement. In contrast, if we set $\zeta\to-1$, the
loss considers all points to be positive and the numerator and denominator of
Eq. (2) become equal. Therefore, the loss function becomes zero and the
network does not train. Following the above discussion, we suggest a value
like $\zeta=0.6$, which is a good balance. However, the choice of $\zeta$
might be influenced by the number of clusters in the data. If we have a large
number of clusters, it would be better to choose a large $\zeta$. On the other
hand, if the data has a small number of clusters, a smaller $\zeta$ (but not
too small) is preferred since it trains faster. In our experiments, we set
$\zeta=0.6$ unless in Tiny-ImageNet which has 200 classes where we used
$\zeta=0.95$.

Figure 8: Performance of C3 for different values of $\Gamma$, for CIFAR 10.

Figure 8 illustrates the impact of hyperparameter $\Gamma$ on the performance
C3. For very low values of $\Gamma$ (i.e., $\Gamma\xrightarrow[]{}0$), all
weights converge to the same value of $w_{ij}^{k}=\frac{1}{2N}$. Conversely,
for very high values of $\Gamma$ (i.e., $\Gamma\xrightarrow[]{}\infty$), the
effect of entropy in Eq. (3) is neglected, leading to a trivial solution where
our weighting function in Eq. (7) selects one negative sample having the
highest value of $1-|z_{i}^{a^{\intercal}}z_{j}^{k}|$ to minimize the first
term in Eq. (3). In all our experiments for all the datasets, $\Gamma$ is set
to 0.1 though a better performance may be obtained if we fine-tune it per
dataset. Overall, our results demonstrate the importance of selecting an
appropriate value for $\Gamma$ to optimize the performance of our proposed
method.

##  5 Conclusion

In this paper, we proposed C3, an algorithm for contrastive data clustering
that incorporates the similarity between different instances to form a better
representation for clustering. We experimentally showed that our method could
significantly outperform the state-of-the-art on five challenging computer
vision datasets. In addition, through additional experiments, we evaluated
different aspects of our algorithm and provided several intuitions on how and
why our proposed scheme can help in learning a more cluster-friendly
representation. The focus of this work was on image clustering, but our idea
can also be applied to clustering other data types, such as text and
categorical data, in future works.

## References

  * [Bengio et al.(2006)Bengio, Lamblin, Popovici, and Larochelle] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle.  Greedy layer-wise training of deep networks.  In B. Schölkopf, J. Platt, and T. Hoffman, editors, _Advances in Neural Information Processing Systems_ , volume 19. MIT Press, 2006.  URL https://proceedings.neurips.cc/paper/2006/file/5da713a690c067105aeb2fae32403405-Paper.pdf.
  * [Cai et al.(2009)Cai, He, Wang, Bao, and Han] Deng Cai, Xiaofei He, Xuanhui Wang, Hujun Bao, and Jiawei Han.  Locality preserving nonnegative matrix factorization.  In _Proceedings of the 21st International Joint Conference on Artificial Intelligence_ , IJCAI’09, page 1010–1015, San Francisco, CA, USA, 2009\. Morgan Kaufmann Publishers Inc.
  * [Cai et al.(2022)Cai, Fan, Guo, Wang, Zhang, and Zhang] Jinyu Cai, Jicong Fan, Wenzhong Guo, Shiping Wang, Yunhe Zhang, and Zhao Zhang.  Efficient deep embedded subspace clustering.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 21–30, 2022.  10.1109/CVPR52688.2022.00012.
  * [Chang et al.(2017)Chang, Wang, Meng, Xiang, and Pan] Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan.  Deep adaptive image clustering.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 5880–5888, 2017.  10.1109/ICCV.2017.626.
  * [Chang et al.(2019)Chang, Guo, Wang, Meng, Xiang, and Pan] Jianlong Chang, Yiwen Guo, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan.  Deep discriminative clustering analysis.  _ArXiv_ , abs/1905.01681, 2019.
  * [Chen et al.(2020)Chen, Kornblith, Norouzi, and Hinton] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  _arXiv preprint arXiv:2002.05709_ , 2020.
  * [Deng et al.()Deng, Dong, Socher, Li, Li, and Fei-Fei] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet full (fall 2011 release).
  * [Feng et al.(2019)Feng, Xu, and Tao] Zeyu Feng, Chang Xu, and Dacheng Tao.  Self-supervised representation learning by rotation feature decoupling.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10356–10366, 2019.  10.1109/CVPR.2019.01061.
  * [Gowda and Krishna(1978)] K. Chidananda Gowda and G. Krishna.  Agglomerative clustering using the concept of mutual nearest neighbourhood.  _Pattern Recognit._ , 10:105–112, 1978.
  * [Guo et al.(2017)Guo, Gao, Liu, and Yin] Xifeng Guo, Long Gao, Xinwang Liu, and Jianping Yin.  Improved deep embedded clustering with local structure preservation.  In _Proceedings of the 26th International Joint Conference on Artificial Intelligence_ , IJCAI’17, page 1753–1759. AAAI Press, 2017.  ISBN 9780999241103.
  * [Guo et al.(2018)Guo, Zhu, Liu, and Yin] Xifeng Guo, En Zhu, Xinwang Liu, and Jianping Yin.  Deep embedded clustering with data augmentation.  In Jun Zhu and Ichiro Takeuchi, editors, _Proceedings of The 10th Asian Conference on Machine Learning_ , volume 95 of _Proceedings of Machine Learning Research_ , pages 550–565. PMLR, 14–16 Nov 2018.  URL https://proceedings.mlr.press/v95/guo18b.html.
  * [Haeusser et al.(2019)Haeusser, Plapp, Golkov, Aljalbout, and Cremers] Philip Haeusser, Johannes Plapp, Vladimir Golkov, Elie Aljalbout, and Daniel Cremers.  Associative deep clustering: Training a classification network with no labels.  In Thomas Brox, Andrés Bruhn, and Mario Fritz, editors, _Pattern Recognition_ , pages 18–32, Cham, 2019. Springer International Publishing.  ISBN 978-3-030-12939-2.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * [Hojjati et al.(2022)Hojjati, Ho, and Armanfard] Hadi Hojjati, Thi Kieu Khanh Ho, and Narges Armanfard.  Self-supervised anomaly detection: A survey and outlook.  _ArXiv_ , abs/2205.05173, 2022.
  * [Huang et al.(2020)Huang, Gong, and Zhu] Jiabo Huang, Shaogang Gong, and Xiatian Zhu.  Deep semantic clustering by partition confidence maximisation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [Huang et al.(2014)Huang, Huang, Wang, and Wang] Peihao Huang, Yan Huang, Wei Wang, and Liang Wang.  Deep embedding network for clustering.  In _2014 22nd International Conference on Pattern Recognition_ , pages 1532–1537, 2014.  10.1109/ICPR.2014.272.
  * [Ji et al.(2019)Ji, Henriques, and Vedaldi] Xu Ji, João F Henriques, and Andrea Vedaldi.  Invariant information clustering for unsupervised image classification and segmentation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 9865–9874, 2019.
  * [Kingma and Welling(2014)] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  _CoRR_ , abs/1312.6114, 2014.
  * [Krizhevsky et al.()Krizhevsky, Nair, and Hinton] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  Cifar-10 (canadian institute for advanced research).  URL http://www.cs.toronto.edu/~kriz/cifar.html.
  * [Kullback and Leibler(1951)] S. Kullback and R. A. Leibler.  On information and sufficiency.  _Ann. Math. Statist._ , 22(1):79–86, 1951.
  * [Le and Yang(2015)] Ya Le and Xuan S. Yang.  Tiny imagenet visual recognition challenge, 2015.
  * [Li et al.(2021)Li, Hu, Liu, Peng, Zhou, and Peng] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng.  Contrastive clustering.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(10):8547–8555, May 2021.  10.1609/aaai.v35i10.17037.  URL https://ojs.aaai.org/index.php/AAAI/article/view/17037.
  * [Lin et al.(2021)Lin, Qi, Zhengyang, and Changhu] Z Lin, S Qi, S Zhengyang, and W Changhu.  Inter-intra variant dual representations for self-supervised video recognition.  In _British Machine Vision Conference_ , volume 2, page 7, 2021.
  * [Lloyd(1982)] Stuart P. Lloyd.  Least squares quantization in pcm.  _IEEE Trans. Inf. Theory_ , 28:129–136, 1982.
  * [Niu et al.(2020)Niu, Zhang, Wang, and Liang] Chuang Niu, Jun Zhang, Ge Wang, and Jimin Liang.  Gatcluster: Self-supervised gaussian-attention network for image clustering.  In _European Conference on Computer Vision (ECCV)_ , 2020.
  * [Peng et al.(2016)Peng, Xiao, Feng, Yau, and Yi] Xi Peng, Shijie Xiao, Jiashi Feng, Wei-Yun Yau, and Zhang Yi.  Deep subspace clustering with sparsity prior.  In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence_ , IJCAI’16, page 1925–1931. AAAI Press, 2016.  ISBN 9781577357704.
  * [Radford et al.(2016)Radford, Metz, and Chintala] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  _CoRR_ , abs/1511.06434, 2016.
  * [Sadeghi and Armanfard(2021a)] Mohammadreza Sadeghi and Narges Armanfard.  Deep Clustering with Self-supervision using Pairwise Data Similarities.  _TechRxiv_ , 6 2021a.  10.36227/techrxiv.14852652.v1.  URL https://www.techrxiv.org/articles/preprint/Deep_Clustering_with_Self-supervision_using_Pairwise_Data_Similarities/14852652.
  * [Sadeghi and Armanfard(2021b)] Mohammadreza Sadeghi and Narges Armanfard.  Idecf: Improved deep embedding clustering with deep fuzzy supervision.  In _2021 IEEE International Conference on Image Processing (ICIP)_ , pages 1009–1013, 2021b.  10.1109/ICIP42928.2021.9506051.
  * [Sadeghi and Armanfard(2022)] Mohammadreza Sadeghi and Narges Armanfard.  Deep Multi-Representation Learning for Data Clustering.  _TechRxiv_ , 3 2022.  10.36227/techrxiv.19357979.v1.  URL https://www.techrxiv.org/articles/preprint/Deep_Multi-Representation_Learning_for_Data_Clustering/19357979.
  * [Shiran and Weinshall(2020)] Guy Shiran and Daphna Weinshall.  Multi-modal deep clustering: Unsupervised partitioning of images.  In _International Conference on Pattern Recognition (ICPR)_ , 2020\.
  * [Tiwari et al.(2021)Tiwari, Chen, Tsai, Kuo, Chen, Jou, Venkatesh, and Chen] Hitika Tiwari, Min-Hung Chen, Yi-Min Tsai, Hsien-Kai Kuo, Hung-Jen Chen, Kevin Jou, KS Venkatesh, and Yong-Sheng Chen.  Self-supervised robustifying guidance for monocular 3d face reconstruction.  2021\.
  * [Van Gansbeke et al.(2020)Van Gansbeke, Vandenhende, Georgoulis, Proesmans, and Van Gool] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool.  Scan: Learning to classify images without labels.  In _Proceedings of the European Conference on Computer Vision_ , 2020\.
  * [Vincent et al.(2010)Vincent, Larochelle, Lajoie, Bengio, and Manzagol] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  _Journal of Machine Learning Research_ , 11(110):3371–3408, 2010.  URL http://jmlr.org/papers/v11/vincent10a.html.
  * [Wu et al.(2019)Wu, Long, Wang, Qian, Li, Lin, and Zha] Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha.  Deep comprehensive correlation mining for image clustering.  In _International Conference on Computer Vision_ , 2019.
  * [Xie et al.(2021)Xie, Ding, Wang, Zhan, Xu, Sun, Li, and Luo] Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo.  Detco: Unsupervised contrastive learning for object detection.  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 8372–8381, 2021.  10.1109/ICCV48922.2021.00828.
  * [Xie et al.(2016)Xie, Girshick, and Farhadi] Junyuan Xie, Ross Girshick, and Ali Farhadi.  Unsupervised deep embedding for clustering analysis.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ICML’16, page 478–487. JMLR.org, 2016.
  * [Yang et al.(2017)Yang, Fu, Sidiropoulos, and Hong] Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong.  Towards k-means-friendly spaces: Simultaneous deep learning and clustering.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, page 3861–3870. JMLR.org, 2017.
  * [Yang et al.(2016)Yang, Parikh, and Batra] J. Yang, D. Parikh, and D. Batra.  Joint unsupervised learning of deep representations and image clusters.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5147–5156, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society.  10.1109/CVPR.2016.556.  URL https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.556.
  * [Zeiler et al.(2010)Zeiler, Krishnan, Taylor, and Fergus] Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor, and Rob Fergus.  Deconvolutional networks.  In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pages 2528–2535, 2010.  10.1109/CVPR.2010.5539957.
  * [Zelnik-manor and Perona(2004)] Lihi Zelnik-manor and Pietro Perona.  Self-tuning spectral clustering.  In L. Saul, Y. Weiss, and L. Bottou, editors, _Advances in Neural Information Processing Systems_ , volume 17. MIT Press, 2004.  URL https://proceedings.neurips.cc/paper/2004/file/40173ea48d9567f1f393b20c855bb40b-Paper.pdf.
  * [Zhong et al.(2020)Zhong, Chen, Jin, and Hua] Huasong Zhong, C. Chen, Zhongming Jin, and Xiansheng Hua.  Deep robust clustering by contrastive learning.  _ArXiv_ , abs/2008.03030, 2020.
  * [Zhou et al.(2022)Zhou, Xu, Zheng, Chen, Li, Bu, Wu, Wang, Zhu, and Ester] Sheng Zhou, Hongjia Xu, Zhuonan Zheng, Jiawei Chen, Zhao Li, Jiajun Bu, Jia Wu, Xin Wang, Wenwu Zhu, and Martin Ester.  A comprehensive survey on deep clustering: Taxonomy, challenges, and future directions.  _ArXiv_ , abs/2206.07579, 2022.