2022

[1,2]Dong Huang

[1]College of Mathematics and Informatics, South China Agricultural
University, Guangzhou, China

2]Key Laboratory of Smart Agricultural Technology in Tropical South China,
Ministry of Agriculture and Rural Affairs, China

3]School of Computer Science and Engineering, Sun Yat-sen University,
Guangzhou, China

# Heterogeneous Tri-stream Clustering Network

Xiaozhi Deng<EMAIL_ADDRESS><EMAIL_ADDRESS>Chang-Dong Wang
<EMAIL_ADDRESS>*  [  [

###### Abstract

Contrastive deep clustering has recently gained significant attention with its
ability of joint contrastive learning and clustering via deep neural networks.
Despite the rapid progress, previous works mostly require both positive and
negative sample pairs for contrastive clustering, which rely on a relative
large batch-size. Moreover, they typically adopt a two-stream architecture
with two augmented views, which overlook the possibility and potential
benefits of multi-stream architectures (especially with heterogeneous or
hybrid networks). In light of this, this paper presents a new end-to-end deep
clustering approach termed Heterogeneous Tri-stream Clustering Network (HTCN).
The tri-stream architecture in HTCN consists of three main components,
including two weight-sharing online networks and a target network, where the
parameters of the target network are the exponential moving average of that of
the online networks. Notably, the two online networks are trained by
simultaneously (i) predicting the instance representations of the target
network and (ii) enforcing the consistency between the cluster representations
of the target network and that of the two online networks. Experimental
results on four challenging image datasets demonstrate the superiority of HTCN
over the state-of-the-art deep clustering approaches. The code is available at
https://github.com/dengxiaozhi/HTCN.

###### keywords:

Data clustering, Image clustering, Deep clustering, Deep neural network,
Contrastive learning

##  1 Introduction

Data clustering is the process of grouping data samples into multiple clusters
in an unsupervised manner, which is a fundamental task in a variety of
applications Frey2007 ; Huang2018 ; huang20_tkde . The traditional clustering
algorithms typically focus on some low-level information and lack the
representation learning ability, which may lead to sub-optimal performance
when dealing with some complex high-dimensional data like images.

In recent years, the deep learning has gained tremendous progress
yu2019hierarchical ; zhang2018local ; zhang2022semisupervised , which has also
been exploited for tackling the clustering task, giving rise to the rapid
development of the deep clustering algorithms xie2016unsupervised ;
guo2017improved ; wang18-npl ; ji2019invariant ; Chen2022 . For example, Xie
et al. xie2016unsupervised  presented a deep clustering method called Deep
Embedded Clustering (DEC), which simultaneously learns representations and
cluster assignments with an objective loss based on Kullback-Leibler (KL)
divergence. Guo et al. guo2017improved  extended DEC by incorporating the
reconstruction loss (via autoencoder) to preserve local structures. Ji et al.
ji2019invariant  sought to learn invariant information of data by maximizing
the mutual information between paired samples. More recently, the contrastive
learning has emerged as a promising technique for exploiting sample-wise (or
augmentation-wise) contrastiveness to improve the deep clustering performance.
Van Gansbeke et al. van2020scan  presented the Semantic Clustering by Adopting
Nearest neighbors (SCAN) method, which first adopts contrastive learning to
learn discriminant features and then performs semantic clustering with the
$K$-nearest neighbors exploited. Dang et al. dang2021nearest  matched local-
level and global-level nearest neighbors to further improve clustering
performance. Li et al. li2021contrastive  presented the Contrastive Clustering
(CC) method to perform feature learning and clustering with simultaneous
instance-level and cluster-level contrastive learning.

Despite significant success, these contrastive deep clustering methods
van2020scan ; dang2021nearest ; li2021contrastive  are mostly faced with two
limitations. On the one hand, they typically requires both positive sample
pairs and negative sample pairs during their contrastive learning process,
which rely on a relatively large batch-size (for sufficient negative pairs)
and may bring in a heavier computational burden. On the other hand, these
prior works generally adopt a two-stream architecture (with two weight-sharing
augmented views), which neglect the possibility of going beyond the two-stream
architecture to utilize three or even more streams of networks (with
heterogeneous or hybrid structures). Recently Grill et al. grill2020bootstrap
presented the Bootstrap Your Own Latent (BYOL) method, which adopts an
asymmetric two-stream architecture (with an online network and a target
network) and conducts the contrastive learning without negative pairs, where
the online network is trained by predicting the feature representations of the
target network. Though the requirement for negative sample pairs is remedied,
BYOL still complies with the two-stream architecture and also lacks the
ability of directly learning the clustering structure. It remains a
challenging problem how to incorporate contrastive learning into multiple
streams of heterogeneous networks while alleviating the dependence on negative
sample pairs for strengthened deep clustering performance.

In light of this, this paper presents a novel deep clustering approach termed
Heterogeneous Tri-stream Clustering Network (HTCN), which leverages three
streams of heterogeneous networks for simultaneous cluster-level and instance-
level contrastive learning without requiring negative sample pairs (as
illustrated in Fig. 1). Inspired by BYOL grill2020bootstrap , we design a
novel tri-stream architecture with three augmented views, corresponding to two
online networks and a target network, respectively. Note that the online
network and the target network are heterogeneous, which differ from each other
in the network structure and the updating mechanism. The two online networks
share the same parameters, while the parameters of the target network are the
exponential moving average of that of the online networks. Here, the
exponential moving average is a type of moving average that places a greater
weight and significance on the most recent data samples grill2020bootstrap .
Each online network is associated with an instance predictor and a cluster
predictor, which produce the instance-level representations and the cluster-
level representations, respectively. Different from the online networks, the
target network utilizes a cluster predictor to generate the cluster-level
representations while producing the instance-level representations by the
projector directly. The incorporation of an instance predictor in the online
networks is meant to prevent the potential collapse where the networks produce
the same feature representations for most samples. Then we train the two
online networks by (i) predicting the target network’s representation of the
same image via the mean squared error (MSE) loss (for the instance-level
contrastive learning) and (ii) enforcing the consistency between the predicted
cluster distributions of the two online networks and that of the target
network via the information noise contrastive estimation (InfoNCE)
oord2018representation  loss (for the cluster-level contrastive learning).
Experiments conducted on four image datasets demonstrate the superiority of
our approach over the state-of-the-art deep clustering approaches.

For clarity, the contributions of this work are summarized below.

  * •

A heterogeneous tri-stream architecture is designed, where two online networks
and a target network are jointly leveraged for instance-level and cluster-
level contrastive learning.

  * •

A novel deep clustering approach termed HTCN is proposed, which utilizes three
augmented views for contrastive learning without requiring negative sample
pairs.

  * •

Experimental results on four image datasets confirm the advantegeous
clustering performance of our HTCN approach over the state-of-the-art deep
clustering approaches.

The rest of the paper is organized as follows. The related works on deep
clustering and self-supervised learning are reviewed in Section 2. The
proposed HTCN framework is described in Section 3. The experiments are
reported in Section 4. Finally, Section 5 concludes the paper.

##  2 Related Work

In this section, we will introduce the related works on deep clustering and
self-supervised learning.

###  2.1 Deep Clustering

Traditional clustering methods such as $K$-means macqueen1967some  and
spectral clustering (SC) zelnik2004self  have achieved promising results in
handling low-dimensional data, but they may result in sub-optimal performance
when faced with high-dimensional data (e.g., images and videos) due to the
lack of the representation learning ability. To address this, the deep
learning based clustering methods, referred to as the deep clustering methods,
have recently achieved significant success, which leverage the power of
feature learning of deep neural networks for the clustering task
li2021contrastive ; huang2020deep ; wang18-npl ; ji2019invariant ;
guo2019adaptive ; caron2018deep ; guo2017improved ; xie2016unsupervised ;
yang2016joint ; zhu20_npl ; dang2021nearest ; park2021improving ; van2020scan
; han2020mitigating ; zhong2020deep ; zhu21_npl .

Previous deep clustering methods can be divided into two main categories,
namely, the one-stage methods and the two-stage methods. The goal of the one-
stage approach is to perform feature representation learning and clustering
assignment simultaneously. Xie et al. xie2016unsupervised  proposed a Deep
Embedding Clustering (DEC) method, which jointly optimizes feature learning
and clustering with a KL-divergence loss. Caron et al. caron2018deep
iteratively clustered the learned features with $K$-means and regarded the
cluster assignments as supervisory signals to optimize the network. Li et
al.li2021contrastive  presented a Contrastive Clustering (CC) method that
performs contrastive learning at instance-level and cluster-level for deep
clustering. Besides the one-stage methods, some researchers have also made
considerable efforts to the two-stage clustering methods. Van Gansbeke et
al.van2020scan  proposed a two-stage clustering method called Semantic
Clustering by Adopting Nearest neighbors (SCAN), which first learns the
semantic features via contrastive learning and then utilizes the features for
clustering in the next stage. To extend SCAN, Dang et al. dang2021nearest
designed a Nearest Neighbor Matching (NNM) method, which selects both local
and global nearest neighbors to optimize the network, where the neighbors are
forced to be close to each other.

###  2.2 Self-supervised Learning

Self-supervised learning has recently emerged as a powerful technique with the
ability to learn representation from raw data without human supervision, in
which the contrastive learning methods wu2018unsupervised ; misra2020self ;
he2020momentum ; chen2020simple  have been a representative and promising
category.

The goal of contrastive learning is to minimize the distance between positive
sample pairs while maximizing the distance between negative sample pairs in a
self-supervised manner, where positive pairs and negative pairs are defined
through data augmentations. In particular, some researchers maintained a
memory bank wu2018unsupervised ; misra2020self  that contains large amounts of
representations of negative samples to achieve high performance. However,
these methods that utilize memory banks to store and update representations
may be computationally expensive. To address the problems with memory banks,
He et al. he2020momentum  proposed a Momentum Contrast (MoCo) method that
trains an encoder by the momentum update mechanism maintaining a long queue of
negative examples. Following the MoCo method, Chen et al. chen2020simple
proposed a Simple framework for Contrastive LeaRning (SimCLR) method which
carefully designs the strategy of data augmentation and a non-linear
transformation head. In addition, the clustering based methods
caron2020unsupervised ; li2020prototypical  adopt a clustering approach to
group similar features together, which address the issue that every sample is
considered as a discrete class in previous works. More recently, some self-
supervised learning methods that only rely on positive pairs and directly
predict the output of one augmented view from another augmented view
grill2020bootstrap ; chen2021exploring ; li2022tribyol  have been developed,
among which a representative method is the BYOL method grill2020bootstrap .
The BYOL method grill2020bootstrap  adopts an asymmetric two-stream
architecture, which, however, lacks the ability to learn the clustering
structure directly and also overlooks the opportunities and potential benefits
of going beyond the two-stream architecture to three or more streams of
networks (even with heterogeneous or hybrid structures) to further enhance the
contrastive learning and clustering performance.

##  3 Proposed Framework

###  3.1 Framework Overview

Figure 1: Illustration of the proposed HTCN framework. The tri-stream network
consists of two weight-sharing online networks and a target network, where the
parameters of the target network is an exponential moving average of that of
the online networks. Instance predictors and cluster predictors are
incorporated in the three networks, after which the MSE loss and the InfoNCE
loss are ultilized for instance-level contrastive learning and cluster-level
contrastive learning, respectively. The network architecture can be trained in
an end-to-end manner, where the final clustering is obtained via the cluster
predictor of the target network.

This paper presents a heterogeneous tri-stream network architecture termed
HTCN for contrastive deep clustering (as illustrated in Fig. 1), which goes
beyond the traditional two-stream architecture to explore the constrastive
network in a multi-stream manner. Also, HTCN doesn’t require negative sample
pairs, which makes it more resilient to different batch-size. Specifically,
HTCN consists of three main components, including two online networks and a
target network. The online networks and the target network are respectively
parameterized by different sets of weights, where the parameters of the target
network are an exponential moving average of that of the online networks.

Given a batch of $N$ images, we perform three types of augmentations on each
image, denoted as $x_{i}$ with $i\in[1,N]$, to generate $3\cdot N$ augmented
(or distorted) images, denoted as
$\\{x_{1}^{a},\ldots,x_{N}^{a},x_{1}^{b},\ldots,x_{N}^{b},x_{1}^{c},\ldots,x_{N}^{c}\\}$.
The backbones (i.e., $f_{\theta}$ and $f_{\xi}$) and projectors (i.e.,
$g_{\theta}$ and $g_{\xi}$) are adopted to extract features from the distorted
images via $z_{i}^{a}=g_{\theta}(f_{\theta}(x_{i}^{a}))$,
$z_{i}^{b}=g_{\xi}(f_{\xi}(x_{i}^{b}))$ and
$z_{i}^{c}=g_{\theta}(f_{\theta}(x_{i}^{c}))$. Then the instance predictors
transform $z^{a}_{i}$ and $z^{c}_{i}$ to $y^{a}_{i}$ and $y^{c}_{i}$,
respectively, while the cluster predictors transform $z^{a}_{i}$, $z^{b}_{i}$
and $z^{c}_{i}$ to $\tilde{q}^{a}_{i}$, $\tilde{q}^{b}_{i}$ and
$\tilde{q}^{c}_{i}$, respectively. Note that, similar to the asymmetric
architecture of BYOL, the target network is not associated with an instance
predictor, and the representations generated by its projector are used to
guide the instance-level learning of the two online networks. The row space of
the feature matrix learned by the projector or the instance predictor is
expressed as the instance-level representations, while the column space of the
feature matrix learned by the cluster predictor is expressed as the cluster-
level representations. The instance-level representations are utilized to
enforce the instance-level contrastive learning with an MSE loss optimized,
while the cluster-level representations are utilized to enforce the cluster-
level contrastive learning with an InfoNCE loss optimized. Finally, the
instance-level and cluster-level contrastive losses are simultaneously
utilized to optimize the tri-stream network.

###  3.2 Instance-level Contrastiveness

Our HTCN approach simultaneously performs feature learning and clustering
without requiring negative sample pairs. In instance-level contrastive
learning, we aim to train the two online networks by predicting the instance
representations of target network. Specifically, let $y^{a}_{i}$ and
$z^{b}_{i}$ be the instance representations of $x_{i}$ in the first online
network and the target network, respectively. The instance-level contrastive
loss between them is defined as

$\mathcal{L}_{a,b,i}=\|\overline{y^{a}_{i}}-\overline{z^{b}_{i}}\|_{2}^{2}=2-2\cdot\frac{\langle
y^{a}_{i},z^{b}_{i}\rangle}{\|y^{a}_{i}\|_{2}\cdot\|z^{b}_{i}\|_{2}},$ (1)

where $\overline{y^{a}_{i}}$ and $\overline{z^{b}_{i}}$ are the normalized
representations. Thus the loss between the first and second views can be
expressed as

$\mathcal{L}_{a,b}=\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}_{a,b,i}$ (2)

Similar to BYOL grill2020bootstrap , the exchange of the online and target
views is performed during each training step. Also, we utilize another online
network to predict the representations produced by the target network, whose
loss is defined as

$\displaystyle\mathcal{L}_{b,c,i}=\|\overline{y^{c}_{i}}-\overline{z^{b}_{i}}\|_{2}^{2}=$
$\displaystyle 2-2\cdot\frac{\langle
y^{c}_{i},z^{b}_{i}\rangle}{\|y^{c}_{i}\|_{2}\cdot\|z^{b}_{i}\|_{2}},i\in[1,N],$
(3) $\displaystyle\mathcal{L}_{b,c}=$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}_{b,c,i},$ (4)

Therefore, the instance-level contrastive loss among the three streams of
networks is defined as

$\mathcal{L}_{instance}=\mathcal{L}_{a,b}+\mathcal{L}_{b,c}.$ (5)

###  3.3 Cluster-level Contrastiveness

The cluster predictor maps the representations produced by the projector to
$M$-dimensional probability vectors, where $M$ is the number of clusters.
These probability vectors, whose $i$-th element denotes how likely the image
belongs to the $i$-th cluster, can be interpreted as the soft label. Let
$q^{a},q^{b},q^{c}\in\mathbb{R}^{N\times M}$ be the feature matrices produced
by the cluster predictors of the three networks, respectively. Each column of
the feature matrix denotes an $N$-dimensional cluster representation, denoted
as $q^{k}_{i}$, while the each row denotes a $M$-dimensional probability
vector, denoted as $\tilde{q}^{k}_{i}$ (for $k\in\\{a,b,c\\}$).

For a cluster representation $q^{a}_{i}$, we regard $q^{a}_{i}$ and
$q^{b}_{i}$ as a positive cluster pair, and the other $2\cdot M-2$ pairs (in
the first and second views) as the negative cluster pairs. The pair-wise
similarity is defined as

$\displaystyle s(q^{a}_{i},q^{b}_{j})=\frac{\langle
q^{a}_{i},q^{b}_{j}\rangle}{\|q^{a}_{i}\|\|q^{b}_{j}\|},~{}~{}i,j\in[1,M]$ (6)

Then the InfoNCE loss for $q^{a}_{i}$ is computed by

$\ell_{i}^{a}=-\log\frac{\exp(s(q^{a}_{i},q^{b}_{j})/\tau)}{\sum_{j=1}^{M}[\exp(s(q^{a}_{i},q^{a}_{j})/\tau)+\exp(s(q^{a}_{i},q^{b}_{j})/\tau)]},$
(7)

where $\tau$ is the temperature parameter. After traversing all cluster
representations, the cluster-level contrastive loss between the first and
second augmented views can be obtained as

$\displaystyle\hat{\mathcal{L}}_{a,b}=\frac{1}{2M}\sum_{i=1}^{M}(\ell_{i}^{a}+\ell_{i}^{b})-H(Q),$
(8)

where $H(Q)$ is the entropy of the cluster-assignment probability, which helps
to avoid a degenerate solution that most images fall into the same cluster and
is computed as

$\displaystyle H(Q)=$ $\displaystyle-\sum_{i=1}^{M}[P(q^{a}_{i})\log
P(q^{a}_{i})+P(q^{b}_{i})\log P(q^{b}_{i})],$ (9) $\displaystyle
P(q_{i}^{k})=$
$\displaystyle\sum_{j=1}^{N}\frac{q_{ji}^{k}}{\|q\|_{1}},~{}~{}k\in\\{a,b\\}$
(10)

For each batch of images, a view pair is formed between each online network
and the target network, leading to a total of two view pairs for the cluster-
level contrastive learning. Therefore, the cluster-level contrastive loss can
be defined as

$\mathcal{L}_{cluster}=\hat{\mathcal{L}}_{a,b}+\hat{\mathcal{L}}_{b,c}.$ (11)

###  3.4 Overall Loss Function

The tri-stream network of HTCN is trained by simultaneously considering the
instance-level contrastiveness and the cluster-level contrastiveness. The
overall loss function is defined as

$\mathcal{L}=\mathcal{L}_{instance}+\mathcal{L}_{cluster}.$ (12)

At each training step, we optimize the overall loss function w.r.t. the online
networks’ parameters $\theta$ only, but not the target network’s parameters
$\xi$. The parameters of the target is updated as an exponential moving
average of that of the online networks. That is

$\theta\leftarrow\operatorname{optimizer}(\theta,\nabla_{\theta}\mathcal{L},\eta),$
(13) $\xi\leftarrow\alpha\xi+(1-\alpha)\theta.$ (14)

where $\eta$ is the learning rate and $\alpha$ is the momentum coefficient.
After the training, we only keep the target network to perform clustering,
which can be obtained in the cluster predictor.

###  3.5 Implementation Details

In HTCN, we use the ResNet34 he2016deep  as the backbone. The projectors and
the instance predictors have the same network structure, each of which is a
multi-layer perceptron (MLP) with 256-dimensional output units. Each of the
cluster predictors is a two-layer MLP, whose output dimension is equal to the
desired number of clusters.

Three augmented (or distorted) views are generated by applying a family of
transformations to each input image. Five types of augmentations are utilized,
including ResizedCrop, HorizontalFlip, ColorJitter, Grayscale and GaussianBlur
li2021contrastive . As each transformation has a probability of being adopted,
the distortions of the three streams can thus be randomly decided. During
optimization, we use the Adam optimizer and train the model for 1000 epochs.
The learning rate is set to 0.0003.The batch size is set to 128.

Table 1: Description of the benchmark image datasets. Dataset |  #Images |  #Classes
---|---|---
CIFAR-100 |  60,000 |  20
ImageNet-10 |  13,000 |  10
ImageNet-Dogs |  19,500 |  15
Tiny-ImageNet |  100,000 |  200

(a) CIFAR-100 (b) ImageNet-10 (c) ImageNet-Dogs (d) Tiny-ImageNet

Figure 2: Some examples of the four image datasets.

##  4 Experiments

###  4.1 Datasets and Evaluation Metrics

The experiments are conducted on four widely-used image datasets, namely,
CIFAR-100 krizhevsky2009learning , ImageNet-10 chang2017deep , ImageNet-Dogs
chang2017deep , and Tiny-ImageNet le2015tiny . The statistics of these
benchmark datasets are given in Table 1, and some sample images of these
datasets are illustrated in Fig. 2.

To compare the clustering results of different clustering methods, three
evaluation metrics are adopted, including normalized mutual information (NMI)
Huang2021_TSMC , clustering accuracy (ACC) Huang2021 , and adjusted rand index
(ARI) liang22_tnnls .

###  4.2 Comparison with State-of-the-Art

Table 2: The NMI(%) scores by different clustering methods (The best score in each column is in bold). Dataset |  CIFAR-100 |  ImageNet-10 |  ImageNet-Dogs |  Tiny-ImageNet
---|---|---|---|---
$K$-means macqueen1967some  |  8.4 |  11.9 |  5.5 |  6.5
SC zelnik2004self  |  9.0 |  15.1 |  3.8 |  6.3
AC gowda1978agglomerative  |  9.8 |  13.8 |  3.7 |  6.9
NMF cai2009locality  |  7.9 |  13.2 |  4.4 |  7.2
AE bengio2006greedy  |  10.0 |  21.0 |  10.4 |  13.1
DAE vincent2010stacked  |  11.1 |  20.6 |  10.4 |  12.7
DCGAN radford2015unsupervised  |  12.0 |  22.5 |  12.1 |  13.5
DeCNN zeiler2010deconvolutional  |  9.2 |  18.6 |  9.8 |  11.1
VAE kingma2013auto  |  10.8 |  19.3 |  10.7 |  11.3
JULE yang2016joint  |  10.3 |  17.5 |  5.4 |  10.2
DEC xie2016unsupervised  |  13.6 |  28.2 |  12.2 |  11.5
DAC chang2017deep  |  18.5 |  39.4 |  21.9 |  19.0
DCCM wu2019deep  |  28.5 |  60.8 |  32.1 |  22.4
GATC niu2020gatcluster  |  28.5 |  59.4 |  28.1 |  -
PICA huang2020deep  |  31.0 |  80.2 |  35.2 |  27.7
DRC zhong2020deep  |  35.6 |  83.0 |  38.4 |  32.1
CC li2021contrastive  |  43.1 |  85.9 |  44.5 |  34.0
HTCN |  46.5 |  87.5 |  49.4 |  35.6
Table 3: The ACC(%) scores by different clustering methods (The best score in each column is in bold). Dataset |  CIFAR-100 |  ImageNet-10 |  ImageNet-Dogs |  Tiny-ImageNet
---|---|---|---|---
$K$-means macqueen1967some  |  13.0 |  24.1 |  10.5 |  2.5
SC zelnik2004self  |  13.6 |  27.4 |  11.1 |  2.2
AC gowda1978agglomerative  |  13.8 |  24.2 |  13.9 |  2.7
NMF cai2009locality  |  11.8 |  23.0 |  11.8 |  2.9
AE bengio2006greedy  |  16.5 |  31.7 |  18.5 |  4.1
DAE vincent2010stacked  |  15.1 |  30.4 |  19.0 |  3.9
DCGAN radford2015unsupervised  |  15.3 |  34.6 |  17.4 |  4.1
DeCNN zeiler2010deconvolutional  |  13.3 |  31.3 |  17.5 |  3.5
VAE kingma2013auto  |  15.2 |  33.4 |  17.9 |  3.6
JULE yang2016joint  |  13.7 |  30.0 |  13.8 |  3.3
DEC xie2016unsupervised  |  18.5 |  38.1 |  19.5 |  3.7
DAC chang2017deep  |  23.8 |  52.7 |  27.5 |  6.6
DCCM wu2019deep  |  32.7 |  71.0 |  38.3 |  10.8
GATC niu2020gatcluster  |  32.7 |  73.9 |  32.2 |  -
PICA huang2020deep  |  33.7 |  87.0 |  35.2 |  9.8
DRC zhong2020deep  |  36.7 |  88.4 |  38.9 |  13.9
CC li2021contrastive  |  42.9 |  89.3 |  42.9 |  14.0
HTCN |  47.2 |  90.5 |  49.3 |  16.0
Table 4: The ARI(%) scores by different clustering methods (The best score in each column is in bold). Dataset |  CIFAR-100 |  ImageNet-10 |  ImageNet-Dogs |  Tiny-ImageNet
---|---|---|---|---
$K$-means macqueen1967some  |  2.8 |  5.7 |  2.0 |  0.5
SC zelnik2004self  |  2.2 |  7.6 |  1.3 |  0.4
AC gowda1978agglomerative  |  3.4 |  6.7 |  2.1 |  0.5
NMF cai2009locality  |  2.6 |  6.5 |  1.6 |  0.5
AE bengio2006greedy  |  4.8 |  15.2 |  7.3 |  0.7
DAE vincent2010stacked  |  4.6 |  13.8 |  7.8 |  0.7
DCGAN radford2015unsupervised  |  4.5 |  15.7 |  7.8 |  0.7
DeCNN zeiler2010deconvolutional  |  3.8 |  14.2 |  7.3 |  0.6
VAE kingma2013auto  |  4.0 |  16.8 |  7.9 |  0.6
JULE yang2016joint  |  3.3 |  13.8 |  2.8 |  0.6
DEC xie2016unsupervised  |  5.0 |  20.3 |  7.9 |  0.7
DAC chang2017deep  |  8.8 |  30.2 |  11.1 |  1.7
DCCM wu2019deep  |  17.3 |  55.5 |  18.2 |  3.8
GATC niu2020gatcluster  |  17.3 |  55.2 |  16.3 |  -
PICA huang2020deep  |  17.1 |  76.1 |  20.1 |  4.0
DRC zhong2020deep  |  20.8 |  79.8 |  23.3 |  5.6
CC li2021contrastive  |  26.6 |  82.2 |  27.4 |  7.1
HTCN |  30.5 |  83.9 |  35.2 |  7.6

In this section, we compare the proposed method against four non-deep
clustering methods, namely, $K$-means macqueen1967some , Spectral Clustering
(SC) zelnik2004self , Agglomerative Clustering (AC) gowda1978agglomerative ,
and Nonnegative Matrix Factorization (NMF) cai2009locality , and thirteen deep
clustering methods, namely, Auto-Encoder (AE) bengio2006greedy , Denoising
Auto-Encoder (DAE) vincent2010stacked , Deep Convolutional Generative
Adversarial Networks (DCGAN) radford2015unsupervised , DeConvolutional Neural
Networks (DeCNN) zeiler2010deconvolutional , Aariational Auto-Encoder (VAE)
kingma2013auto , Joint Unsupervised LEarning (JULE) yang2016joint , Deep
Embedded Clustering (DEC) xie2016unsupervised , Deep Adaptive Clustering (DAC)
chang2017deep , Deep Comprehensive Correlation Mining (DCCM) wu2019deep ,
Gaussian ATtention Network for image Clustering (GATC) niu2020gatcluster ,
PartItion Confidence mAximization (PICA) huang2020deep , Deep Robust
Clustering (DRC) zhong2020deep  and Contrastive Clustering (CC)
li2021contrastive .

As shown in Table 2, 3 and 4, our HTCN method achieves the best scores on all
the four benchmark datasets w.r.t. NMI, ACC, and ARI. Notably, on the
ImageNet-Dogs dataset, our HTCN method obtains NMI(%),ACC(%) and ARI(%) scores
of 49.4, 49.3, and 35.2, respectively, which significantly outperforms the
second best method (i.e., CC) that obtains NMI(%),ACC(%) and ARI(%) scores of
44.5, 42.9, and 27.4. The experimental results in Table 2, 3 and 4 confirm the
advantageous clustering performance of HTCN over the baseline methods.

###  4.3 Influence of the Tri-stream Architecture

In the proposed framework, we present a tri-stream architecture which consists
of two online networks and a target network. In this section, we test the
influence of the three streams of networks. As shown in Table 5, using an
online network and a target network leads to better clustering results than
using two online networks, while using three streams of networks outperforms
both variants of using two streams, which shows the benefits of the
heterogeneous tri-stream architecture.

###  4.4 Influence of Two Types of Contrastive losses

In the section, we test the influence of the two types of contrastive losses,
i.e., the instance-level contrastive loss and the cluster-level contrastive
loss. As shown in Table 6, training with both types of losses can lead to
better clustering performance than training with only one of them, which
confirm the joint contribution of the instance-level and cluster-level losses
in the self-supervised training.

###  4.5 Influence of the Asymmetric Settings

Table 5: The clustering performance of HTCN using different combinations of network architectures. Architecture |  NMI |  ACC |  ARI
---|---|---|---
Tri-stream architecture |  46.5 |  47.2 |  30.5
Dual-stream (Online+Target) |  42.2 |  42.5 |  26.8
Dual-stream (Online+Online) |  39.9 |  40.2 |  24.4
Table 6: The clustering performance of HTCN using different loss functions. Loss function |  NMI |  ACC |  ARI
---|---|---|---
With instance and cluster losses |  46.5 |  47.2 |  30.5
With only instance loss |  43.3 |  35.6 |  14.6
With only cluster loss |  38.4 |  36.4 |  22.7
Table 7: The NMI(%), ACC(%), and ARI(%) by HTCN removing different asymmetric settings. Asymmetric settings |  NMI |  ACC |  ARI
---|---|---|---
HTCN |  46.5 |  47.2 |  30.5
No predictor |  40.9 |  41.2 |  25.0
No stop-gradient |  39.2 |  39.3 |  23.7

Two symmetry-breaking mechanisms are enforced between the online and target
networks grill2020bootstrap . First, an instance predictor is incorporated in
each online network, which does not exist in the target network. Second, the
so-called stop-gradient is incorporated in the target network, which indicates
that this network is not updated using backpropagation. We test the influence
of the asymmetric settings by removing one of the instance predictor and the
stop-gradient. As shown in Table 7, training with both asymmetric settings
leads to better performance than training with only one of them.

###  4.6 Convergence Analysis

In this section, we test the convergence of the proposed HTCN method as the
number of epochs increases. As shown in Fig. 3, the clustering scores (w.r.t.
NMI) of the proposed HTCN method rapidly increase during the first 200 epochs
on the benchmark datasets. When going beyond 200 epochs, the increase of
epochs still benefits the clustering performance consistently. In this paper,
the number of epochs is set to 1000 on all benchmark datasets.

(a) CIFAR-100 (b) ImageNet-10 (c) ImageNet-dogs (d) Tiny-ImageNet

Figure 3: Illustration of the convergence of HTCN (w.r.t. its NMI performance)
on the four benchmark datasets.

##  5 Conclusion and Future Work

The paper develops a new deep clustering approach termed HTCN, which breaks
through the conventional two-stream contrastive architecture to explore the
rich possibilities in heterogeneous multi-stream contrastive learning and
clustering. In HTCN, the two weight-sharing online networks are trained by
predicting the instance representations of the target network and enforcing
the consistency between the cluster representations of the target and online
networks. Thus the tri-stream network architecture can be optimized in an end-
to-end manner via simultaneous instance-level and cluster-level contrastive
learning. Experimental results on four challenging image datasets have shown
the superior performance of our HTCN approach over the state-of-the-art deep
clustering approaches. In this paper, we mainly focus on the deep clustering
task for images. In the future work, a possible direction is to extend the
proposed framework to the deep clustering tasks for more complex data types,
such as time series data and document data.

## Declarations

  * •

Funding. This work was supported by the NSFC (61976097 & 61876193) and the
Natural Science Foundation of Guangdong Province (2021A1515012203).

  * •

Conflict of interest. The authors declare that they have no conflict of
interest.

  * •

Ethical approval. This article does not contain any studies with human
participants or animals performed by any of the authors.

  * •

Consent to participate. Informed consent to participate was obtained from all
individual participants included in the study.

  * •

Consent for publication. Informed consent for publication was obtained from
all individual participants included in the study.

  * •

Availability of data and materials. All datasets used in this paper are
publicly-available datasets.

  * •

Code availability. The code is available at
https://github.com/dengxiaozhi/HTCN.

  * •

Authors’ contributions. XD: Conceptualization, Methodology, Writing–Original
Draft. DH: Conceptualization, Writing–Review & Editing. CDW: Optimization,
Writing–Review & Editing.

## References

  *   * (1) Frey, B.J., Dueck, D.: Clustering by passing messages between data points. Science 315, 972–976 (2007)
  * (2) Huang, D., Wang, C.-D., Lai, J.-H.: Locally weighted ensemble clustering. IEEE Transactions on Cybernetics 48(5), 1460–1473 (2018)
  * (3) Huang, D., Wang, C.-D., Wu, J.-S., Lai, J.-H., Kwoh, C.-K.: Ultra-scalable spectral clustering and ensemble clustering. IEEE Transactions on Knowledge and Data Engineering 32(6), 1212–1226 (2020)
  * (4) Yu, J., Tan, M., Zhang, H., Rui, Y., Tao, D.: Hierarchical deep click feature prediction for fine-grained image recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(2), 563–578 (2019)
  * (5) Zhang, J., Yu, J., Tao, D.: Local deep-feature alignment for unsupervised dimension reduction. IEEE Transactions on Image Processing 27(5), 2420–2432 (2018)
  * (6) Zhang, J., Yang, J., Yu, J., Fan, J.: Semisupervised image classification by mutual learning of multiple self-supervised models. International Journal of Intelligent Systems 37(5), 3117–3141 (2022)
  * (7) Xie, J., Girshick, R., Farhadi, A.: Unsupervised deep embedding for clustering analysis. In: Proc. of International Conference on Machine Learning (ICML), pp. 478–487 (2016)
  * (8) Guo, X., Gao, L., Liu, X., Yin, J.: Improved deep embedded clustering with local structure preservation. In: Proc. of International Joint Conference on Artificial Intelligence (IJCAI), pp. 1753–1759 (2017)
  * (9) Wang, Q., Wang, L., Zeng, X., Zhao, L.: An improved deep clustering model for underwater acoustical targets. Neural Processing Letters 48(3), 1633–1644 (2018)
  * (10) Ji, X., Henriques, J.F., Vedaldi, A.: Invariant information clustering for unsupervised image classification and segmentation. In: Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9865–9874 (2019)
  * (11) Chen, D.-H., Huang, D., Cheng, H., Wang, C.-D.: D-trace: Deep triply-aligned clustering. In: Proceedings of International Conference on Artificial Neural Networks (ICANN), pp. 526–537 (2022)
  * (12) van Gansbeke, W., Vandenhende, S., Georgoulis, S., Proesmans, M., Van Gool, L.: SCAN: Learning to classify images without labels. In: Proc. of European Conference on Computer Vision (ECCV), pp. 268–285 (2020)
  * (13) Dang, Z., Deng, C., Yang, X., Wei, K., Huang, H.: Nearest neighbor matching for deep clustering. In: Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13693–13702 (2021)
  * (14) Li, Y., Hu, P., Liu, Z., Peng, D., Zhou, J.T., Peng, X.: Contrastive clustering. In: Proc. of AAAI Conference on Artificial Intelligence (2021)
  * (15) Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.: Bootstrap your own latent: A new approach to self-supervised learning. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 21271–21284 (2020)
  * (16) Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
  * (17) MacQueen, J., et al.: Some methods for classification and analysis of multivariate observations. In: Proc. of Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, pp. 281–297 (1967)
  * (18) Zelnik-Manor, L., Perona, P.: Self-tuning spectral clustering. Advances in Neural Information Processing Systems (NeurIPS) 17 (2004)
  * (19) Huang, J., Gong, S., Zhu, X.: Deep semantic clustering by partition confidence maximisation. In: Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8849–8858 (2020)
  * (20) Guo, X., Liu, X., Zhu, E., Zhu, X., Li, M., Xu, X., Yin, J.: Adaptive self-paced deep clustering with data augmentation. IEEE Transactions on Knowledge and Data Engineering 32(9), 1680–1693 (2019)
  * (21) Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: Proc. of European Conference on Computer Vision (ECCV), pp. 132–149 (2018)
  * (22) Yang, J., Parikh, D., Batra, D.: Joint unsupervised learning of deep representations and image clusters. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5147–5156 (2016)
  * (23) Zhu, Q., Wang, Z.: An image clustering auto-encoder based on predefined evenly-distributed class centroids and mmd distance. Neural Processing Letters 51(2), 1973–1988 (2020)
  * (24) Park, S., Han, S., Kim, S., Kim, D., Park, S., Hong, S., Cha, M.: Improving unsupervised image clustering with robust learning. In: Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12278–12287 (2021)
  * (25) Han, S., Park, S., Park, S., Kim, S., Cha, M.: Mitigating embedding and class assignment mismatch in unsupervised image classification. In: Proc. of European Conference on Computer Vision (ECCV), pp. 768–784 (2020). Springer
  * (26) Zhong, H., Chen, C., Jin, Z., Hua, X.-S.: Deep robust clustering by contrastive learning. arXiv preprint arXiv:2008.03030 (2020)
  * (27) Zhu, X., Do, K.D., Guo, J., Xu, J., Dietze, S.: Exploring implicit and explicit geometrical structure of data for deep embedded clustering. Neural Processing Letters 53(1), 1–16 (2021)
  * (28) Wu, Z., Xiong, Y., Yu, S.X., Lin, D.: Unsupervised feature learning via non-parametric instance discrimination. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3733–3742 (2018)
  * (29) Misra, I., van der Maaten, L.: Self-supervised learning of pretext-invariant representations. In: Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6707–6717 (2020)
  * (30) He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729–9738 (2020)
  * (31) Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: Proc. of International Conference on Machine Learning (ICML), pp. 1597–1607 (2020)
  * (32) Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems (NeurIPS) 33, 9912–9924 (2020)
  * (33) Li, J., Zhou, P., Xiong, C., Hoi, S.C.: Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966 (2020)
  * (34) Chen, X., He, K.: Exploring simple siamese representation learning. In: Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15750–15758 (2021)
  * (35) Li, G., Togo, R., Ogawa, T., Haseyama, M.: Tribyol: Triplet BYOL for self-supervised representation learning. In: Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3458–3462 (2022). IEEE
  * (36) He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
  * (37) Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
  * (38) Chang, J., Wang, L., Meng, G., Xiang, S., Pan, C.: Deep adaptive image clustering. In: Proc. of IEEE International Conference on Computer Vision (ICCV), pp. 5879–5887 (2017)
  * (39) Le, Y., Yang, X.: Tiny imagenet visual recognition challenge. CS 231N 7(7), 3 (2015)
  * (40) Huang, D., Wang, C.-D., Peng, H., Lai, J.-H., Kwoh, C.-K.: Enhanced ensemble clustering via fast propagation of cluster-wise similarities. IEEE Transactions on Systems, Man, and Cybernetics: Systems 51(1), 508–520 (2021)
  * (41) Huang, D., Wang, C.-D., Lai, J.-H., Kwoh, C.-K.: Toward multidiversified ensemble clustering of high-dimensional data: From subspaces to metrics and beyond. IEEE Transactions on Cybernetics, in press 52(11), 12231–12244 (2022)
  * (42) Liang, Y., Huang, D., Wang, C.-D., Yu, P.S.: Multi-view graph learning by joint modeling of consistency and inconsistency. IEEE Transactions on Neural Networks and Learning Systems, in press (2022)
  * (43) Gowda, K.C., Krishna, G.: Agglomerative clustering using the concept of mutual nearest neighbourhood. Pattern Recognition 10(2), 105–112 (1978)
  * (44) Cai, D., He, X., Wang, X., Bao, H., Han, J.: Locality preserving nonnegative matrix factorization. In: Proc. of International Joint Conference on Artificial Intelligence (IJCAI) (2009)
  * (45) Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. Advances in Neural Information Processing Systems (NeurIPS) 19 (2006)
  * (46) Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A., Bottou, L.: Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research 11(12) (2010)
  * (47) Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)
  * (48) Zeiler, M.D., Krishnan, D., Taylor, G.W., Fergus, R.: Deconvolutional networks. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2528–2535 (2010)
  * (49) Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  * (50) Wu, J., Long, K., Wang, F., Qian, C., Li, C., Lin, Z., Zha, H.: Deep comprehensive correlation mining for image clustering. In: Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8150–8159 (2019)
  * (51) Niu, C., Zhang, J., Wang, G., Liang, J.: GATCluster: Self-supervised gaussian-attention network for image clustering. In: Proc. of European Conference on Computer Vision (ECCV), pp. 735–751 (2020)