# Vision Transformer for Contrastive Clustering

Hua-Bao Ling, Bowen Zhu, Dong Huang, Ding-Hua Chen,
Chang-Dong Wang, and Jian-Huang Lai This project was supported by the NSFC
(61976097, 61876193 & 62076258) and the Natural Science Foundation of
Guangdong Province (2021A1515012203). (Corresponding author: Dong Huang)H.-B.
Ling, B. Zhu, D. Huang, and D.-H. Chen are with the College of Mathematics and
Informatics, South China Agricultural University, Guangzhou, China.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>C.-D. Wang and J.-H. Lai are
with the School of Computer Science and Engineering, Sun Yat-sen University,
Guangzhou, China, and also with Guangdong Key Laboratory of Information
Security Technology, Guangzhou, China, and also with Key Laboratory of Machine
Intelligence and Advanced Computing, Ministry of Education, China.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>Ling and B.
Zhu contribute equally to this work.

###### Abstract

Vision Transformer (ViT) has shown its advantages over the convolutional
neural network (CNN) with its ability to capture global long-range
dependencies for visual representation learning. Besides ViT, contrastive
learning is another popular research topic recently. While previous
contrastive learning works are mostly based on CNNs, some recent studies have
attempted to combine ViT and contrastive learning for enhanced self-supervised
learning. Despite the considerable progress, these combinations of ViT and
contrastive learning mostly focus on the instance-level contrastiveness, which
often overlook the global contrastiveness and also lack the ability to
directly learn the clustering result (e.g., for images). In view of this, this
paper presents a novel deep clustering approach termed Vision Transformer for
Contrastive Clustering (VTCC), which for the first time, to our knowledge,
unifies the Transformer and the contrastive learning for the image clustering
task. Specifically, with two random augmentations performed on each image, we
utilize a ViT encoder with two weight-sharing views as the backbone. To remedy
the potential instability of the ViT, we incorporate a convolutional stem to
split each augmented sample into a sequence of patches, which uses multiple
stacked small convolutions instead of a big convolution in the patch
projection layer. By learning the feature representations for the sequences of
patches via the backbone, an instance projector and a cluster projector are
further utilized to perform the instance-level contrastive learning and the
global clustering structure learning, respectively. Experiments on eight image
datasets demonstrate the stability (during the training-from-scratch) and the
superiority (in clustering performance) of our VTCC approach over the state-
of-the-art. The code is available at https://github.com/JackKoLing/VTCC.

###### Index Terms:

Deep clustering, Image clustering, Vision Transformer, Contrastive learning,
Self-supervised learning.

##  I Introduction

Vision Transformer (ViT) [1], as a promising alternative network architecture
to the convolutional neural network (CNN) [2], has received increasing
attention recently. Although the CNN can effectively capture the local
information, there is still a practical need to model the global long-range
dependencies between image elements in many computer vision tasks. Inspired by
the significant success of the Transformer architecture that adopts the multi-
head self-attention (MHSA) mechanism [3] in natural language processing (NLP),
the ViT has recently emerged as a pioneering technique to apply the
Transformer architecture in computer vision [1]. Typically, in ViT, an input
image is first split into a sequence of patches by a non-overlapping $p\times
p$ convolutional operation with stride-$p$ (e.g., with $p=16$). Then these
patches are fed into a standard Transformer encoder architecture [3] for
representation learning. With this simple yet powerful backbone architecture,
many Transformer-based models have been developed and some have become the
predominant architectures for a variety of vision tasks, such as image
classification [1], object detection [4], and semantic segmentation [5].

In the meantime, the self-supervised learning (SSL) [6, 7] has gained
remarkable progress in the past few years, which aims to learn visual
representation from unlabeled data in a self-supervised manner. There are
three mainstream categories in SSL [8], which are generative [9], contrastive
[6, 7], and generative-contrastive [10] architectures, respectively. Among
them, the contrastive learning has recently made some breakthroughs, which
aims to “learn to compare” via an information noise contrastive estimation
(InfoNCE) [11] loss function. As a representative work, MoCo [7] utilizes the
momentum contrastive learning with two encoders and employs a queue to save
negative samples. Alternatively, SimCLR [6] demonstrates the importance of
constructing the samples pairs by various data augmentations, and can be
trained with a large batch size instead of the momentum contrast in MoCo [7].

Rapid progress has been made in both the Transformer-based architectures and
the contrastive learning architectures. However, these two types of
architectures are mostly developed independently, and almost all contrastive
learning backbones are based on the CNN. A critical question arises as to
whether the ViT can provide better expressiveness for contrastive learning
than the CNN. Recently, some research works have attempted to exploit the
advantages of both of them. For example, Chen et al. [12] studied the
instability issue on training the self-supervised ViT. Caron et al. [13]
explored some emerging properties of the self-supervised ViT. In contrastive
learning, a recent development is its combination with the image clustering
task. Typically, Li et al. [14] utilized the contrastive learning at the
instance-level and the cluster-level for simultaneous representation learning
and image clustering, where the CNN is adopted as the backbone. As the CNN is
more focused on local information, it is intuitive that the modeling of the
global dependencies via Transformer may provide rich information to enhance
the image clustering performance, especially for complex images. Yet
surprisingly, it remains an unaddressed problem how to effectively leverage
the Transformer (or ViT) in collaboration with the contrastive learning for
joint representation learning and image clustering.

Notably, this paper for the first time, to the best of our knowledge, unifies
the ViT and the contrastive learning for the image clustering task. To make
this unification possible and practical, some critical challenges should be
handled. Different from CNN [2], the training recipes for ViT are often
sensitive, especially in the self-supervised scenarios [12], which involve the
choice of hyper-parameters, the number of encoder blocks, and so forth. As a
consequence, the training of ViT usually needs a large dataset, and the
instability issue is often encountered [13, 15, 16]. Recently, some studies
suggest that this issue may arise from the path projection layer in the
standard ViT. Regarding this, Chen et al. [12] alleviated the instability
issue by freezing the patchify stem instead of using the random initialization
patch projection layer. Xiao et al. [16] replaced the origin patchify stem
with a convolutional stem, which is implemented by a small number of stacked
stride-$2$ $3\times 3$ convolutions in the patch projection layer. In this
work, we observe that the use of multiple stacked small convolutions instead
of a big convolution in the patch projection layer, which to some extent
resembles the typical design of CNNs [2, 17], can significantly benefit the
contrastive image clustering with Transformer, even for some relative small
datasets.

Specifically, we propose a novel deep clustering approach termed Vision
Transformer for Contrastive Clustering (VTCC) (as illustrated in Fig. 1). To
remedy the potential instability, a convolutional stem layer with multiple
stacked small-size convolutions is incorporated to split each augmented sample
image into a sequence of patches. With two random augmentations performed on
each image to obtain two augmented samples, we feed the sequences of patch
vectors of the augmented samples to the backbone, which encompasses two
weight-sharing views in a standard Transformer encoder architecture. With the
representations of the augmented samples learned by the self-attention
mechanism, the contrastive learning is enforced via an instance projector and
a cluster projector, which explore the instance contrastiveness and the global
clustering structure, respectively. Note that the Transformer backbone with
convolutional stem and the two contrastive projectors in VTCC are jointly
trained in an end-to-end manner. Extensive experiments are conducted on eight
challenging image datasets, which confirm the benefits brought in by the
Transformer, and demonstrate the stability (during the training-from-scratch)
and the superior clustering performance of VTCC in comparison with the state-
of-the-art.

The main contributions of this work are summarized below.

  * •

This paper makes the first attempt to effectively bridge the gap between the
Transformer and the contrastive learning for the deep image clustering task,
with the global dependencies, the instance contrastiveness, and the clustering
structure learning jointly modeled.

  * •

This paper incorporates a convolutional stem with multiple stacked small
convolutions in the patch projection layer, which enhances the stability
during the ViT training from scratch for image clustering.

  * •

This paper presents an end-to-end deep clustering approach termed VTCC with
ViT and contrastive learning jointly modeled. Experimental results on eight
real-world image datasets demonstrate the promising capability of Transformer
for the clustering of complex images and the advantages of VTCC over the
state-of-the-art deep clustering approaches.

The remainder of the paper is organized as follows. The proposed VTCC approach
is described in Section II. The experimental results are provided in Section
III, followed by the conclusion in Section IV.

Figure 1: Vision Transformer for Contrastive Clustering (VTCC).

##  II Proposed Framework

In this section, we describe the details of our VTCC approach. First, an
overview of the network architecture is given in Section II-A. Then the ViT
backbone is presented in Section II-B. Finally, the overall loss function of
VTCC is provided in Section II-C.

###  II-A Network Architecture

The architecture of VTCC is illustrated in Fig. 1, which consists of three
main components, namely, the ViT backbone with two weight-sharing views to
extract feature representations for augmented samples, the instance-level
contrastive learning module to maximize the similarity of positive pairs, and
the global clustering structure learning module that obtains the soft label
prediction for image clustering.

Specifically, for each image in a mini-batch of $N$ images, two data
augmentations $T^{a}$ and $T^{b}$ are randomly selected with a certain
probability so as to obtain $2$ augmented samples for this image, thus
obtaining $2\cdot N$ augmented samples for this mini-batch. Then each of the
augmented samples is split into a sequence of patches by the convolutional
stem with multiple stacked small convolutions in the patch projection layer
[16]. With each sample split into a sequence of patches and embedded by linear
projection, a standard Transformer encoder is utilized for learning the
feature representations, which are then fed to two projectors, i.e., the
instance projector and the cluster projector, for instance-level contrastive
learning and global clustering structure learning, respectively. The overall
network is trained by two contrastive loss functions in an end-to-end manner,
through which the clustering result can therefore be learned. Additionally,
the pseudo-code of VTCC is presented in Algorithm 1.

# $f(\cdot)$ : Vision Transformer

# $g_{I}(\cdot)$ : Instance projector

# $g_{C}(\cdot)$ : Cluster projector

# $InfoNCE(\cdot,\cdot)$ : InfoNCE loss

# $InsLoss(\cdot,\cdot)$ : Instance-level contrastive loss

# $CluLoss(\cdot,\cdot)$ : Global clustering contrastive loss

# $T^{a},\;T^{b}$ : Augmentations selected from family $T$

for x in loader:  # Load a mini-batch $x$ with $N$ samples

# Obtain augmented views of $x$

$x^{a},\;x^{b}\;=\;T^{a}(x),\;T^{b}(x)$

# Extract feature representations

$h^{a},\;h^{b}\;=\;f(x^{a}),\;f(x^{b})$

# Map representations to subspace via $g_{I}$

$Z^{a},\;Z^{b}\;=\;g_{I}(h^{a}),\;g_{I}(h^{b})$

# Map representations to subspace via $g_{C}$

$Y^{a},\;Y^{b}\;=\;g_{C}(h^{a}),\;g_{C}(h^{b})$

# Loss calculation

$loss=InsLoss(Z^{a},Z^{b})+CluLoss(Y^{a},Y^{b})$

$loss.backward()$

$update(f)$ # Optimize the parameters of $f$

def $InsLoss(Z^{a},Z^{b})$:

return $(\text{InfoNCE}(Z^{a})+\text{InfoNCE}(Z^{b}))/2$

def $CluLoss(Y^{a},Y^{b})$:

return $(\text{InfoNCE}(Y^{a})+\text{InfoNCE}(Y^{b}))/2$

Algorithm 1 VTCC: A PyTorch-like Pseudo Code

###  II-B Backbone with Vision Transformer

Unlike the CNNs which are performed on the images directly, the Transformer
encoder requires a sequence of vectors (or patches) as input. In VTCC, we
utilize a convolutional stem with multiple stacked small-size convolutions to
split each augmented sample into a sequence of patches, which are fed into a
ViT [1] backbone. In the following, we describe the convolutional stem and the
Transformer encoder in Sections II-B1 and II-B2, respectively.

####  II-B1 Convolutional Stem

To convert a 2D image to a 1D sequence of feature embedding, a conventional
strategy adopted in ViT is to apply the $p\times p$ convolution operation with
a stride $p$ to the input image (typically with $p=16$), which is referred to
as the patchify stem. Thus, it can directly capture a lot of non-overlapping
patches with the $p\times p$ size, and then flatten these patches to form the
input sequence of vectors for the Transformer encoder. However, on the one
hand, this type of convolution operation with a large convolution kernel and a
large stride runs counter to the classical convolution setting in canonical
neural networks [2] which usually adopt some small-size (e.g., $3\times 3$)
convolution kernels. On the other hand, the in-patch structure information
often focus on relatively small receptive field, which overlooks the rich
information of local associations (e.g., the spatial relationship between
patches) and to some extent makes the patchify stem not sufficient with the
simple linear projections.

Regarding this issue, Xiao et al. [16] proposed a convolutional stem which
replaces the patchify stem in the original ViT model by a small number of
stacked stride-$2$ $3\times 3$ convolutions. We empirically show that using
the convolutional stem on the ViT patch embedding layer is particularly
beneficial for our image clustering task (as evaluated in Section III-D).
Specifically, to obtain the 1D image sequence, we first use four blocks in the
convolutional stem, where each block includes a standard convolution operation
with a stride-2 $3\times 3$ convolution kernel, followed by the batch
normalization and a ReLU activation function (as shown in Fig. 1). Since the
Transformer encoder requires the 1D image sequence as input, we use a single
$1\times 1$ convolution to match the input dimension of the Transformer
encoder. In practice, compared with the large convolution, using multiple
small convolutions can capture the fine-grained local features which are more
conducive to the representation learning process and further contribute to
better stability and clustering performance.

####  II-B2 Transformer Encoder

In the literature, ViT models with various scales have been developed. In this
paper, we adopt the ViT-Small [18] as the backbone which provides effective
representation learning capability while maintaining high efficiency.
Specifically, based on the input sequence of patches embedding, each encoder
layer is built upon a standard Transformer architecture that consists of a
multi-head self-attention and a MLP block. In addition, a layer-norm operation
is adopted before each block, and the residual connection is incorporated
after each block. Here, the Transformer encoder applies the self-attention
layers to model global relations between input embeddings, which has an
advantage over the CNNs whose receptive field of convolutional kernels are
limited locally. However, the Transformer architecture is permutation-
invariant theoretically, which may neglect the rich information of relative
positions. Therefore, we also incorporate the position encodings [3] to the
input sequence of each self-attention layer.

###  II-C Loss Function

To simultaneously perform the instance-level and cluster-level self-
supervision, two independent projectors are utilized to map the feature
representations extracted by the backbone to different subspaces (i.e., the
instance subspace and the cluster subspace). By means of these two projectors,
the instance-level contrastive learning and the cluster-level contrastive
learning are respectively enforced.

####  II-C1 Instance-level Contrastive Learning

The purpose of the instance-level contrastive learning is to maximize the
similarity between two different augmented samples of the same input image by
pulling closer the distance between a positive pair while pushing away the
representations of the negative pairs in the instance subspace. Let
$x^{a}_{i}$ and $x^{b}_{i}$ denote the two augmented samples for the input
image $x_{i}$, and $h^{a}_{i}$ and $h^{b}_{i}$ denote the feature
representations of $x^{a}_{i}$ and $x^{b}_{i}$, respectively, extracted by the
ViT encoder. Since directly using the feature representations $h^{a}_{i}$ and
$h^{b}_{i}$ for similarity calculation may lead to the loss of information
[6], we utilize an instance projector with a three-layer MLP, which is denoted
as $g_{I}$ and maps the representations $h^{a}_{i}$ and $h^{b}_{i}$ to a low-
dimensional subspace, leading to $Z^{a}_{i}=g_{I}(h^{a}_{i})$ and
$Z^{b}_{i}=g_{I}(h^{b}_{i})$, respectively. Then, the cosine similarity is
used to measure the pair-wise similarity, that is

$s(\alpha,\beta)=\frac{\alpha^{\top}\beta}{\|\alpha\|\cdot\|\beta\|},$ (1)

where $\alpha$ and $\beta$ are two feature vectors with the same dimension.

Let $I$ denote the instance-level representations set, which contains all the
instance-level augmented representations in a mini-batch of $N$ images, that
is, $I=\\{Z_{1}^{a},\dots,Z_{N}^{a},Z_{1}^{b},\dots,Z_{N}^{b}\\}$, where
$Z_{i}^{a}$ (or $Z_{i}^{b}$) is the instance-level representation of sample
$x_{i}^{a}$ (or $x_{i}^{b}$, respectively). Thus we can obtain $2\cdot N-1$
sample pairs for each augmented sample (say, $Z^{a}$), that is,
$\\{Z^{a}_{i},Z^{k}_{j}\\}$ with $Z^{a}_{i}\neq Z^{k}_{j}$, $k\in\\{a,b\\}$
and $j\in[1,N]$. Among these pairs, only the pair $\\{Z_{i}^{a},Z_{i}^{b}\\}$
is the positive pair for $Z_{i}^{a}$, while the rest are negative pairs. Then
the InfoNCE loss [11] for calculating the instance-level contrastiveness is
defined as

$\displaystyle
l^{a}_{i}=-log\frac{\exp(s(Z^{a}_{i},Z^{+})/\tau_{I})}{\sum\limits_{Z^{-}\in
I}\exp(s(Z^{a}_{i},Z^{-})/\tau_{I})}$ (2)

where $\tau_{I}$ is the instance-level temperature parameter, $Z^{+}$ and
$Z^{-}$ are the positive and negative samples of the current sample,
respectively. Thus for the image $x_{i}$, the instance-level loss is
calculated as

$l_{i}=l^{a}_{i}+l^{b}_{i},$ (3)

where $l^{a}_{i}$ is the instance-level contrastive loss of $x_{i}$ in the
first view (and $l^{b}_{i}$ for the second view). Finally, the instance-level
contrastive loss for a mini-batch of $N$ input images is defined as [6]

$L_{ins}=\frac{\sum_{i=1}^{N}{l_{i}}}{2N}.$ (4)

####  II-C2 Global Clustering Structure Learning

Besides the instance-level contrastive loss, we proceed to define the cluster-
level contrastive loss for global clustering structure learning. It aims to
maximize the similarity of the positive cluster pairs while minimizing the
similarity of the negative cluster pairs. A cluster projector $g_{C}$ with a
three-layer MLP and an extra softmax layer is incorporated to map the feature
representations to the low-dimensional subspace, leading to
$Y^{a}_{i}=g_{C}(h^{a}_{i})$ and $Y^{b}_{i}=g_{C}(h^{b}_{i})$. For a mini-
batch of $N$ input images, we can obtain the cluster distribution probability
matrices $Y^{a}\in\mathbb{R}^{N\times K}$ and $Y^{b}\in\mathbb{R}^{N\times K}$
for the two augmented views by stacking the learned representations of the $N$
samples, where $K$ is the number of clusters and the $i$-th column (i.e., the
representation of $i$-th cluster, denoted as $\hat{Y}^{k}_{i}$ with
$k\in\\{a,b\\}$) of these two matrices indicates the probability of allocating
each sample to the $i$-th cluster.

Let $C$ denote the cluster-level representation set which contains the
representations of all clusters (in the two augmented views), that is,
$C=\\{\hat{Y}_{1}^{a},\dots,\hat{Y}_{K}^{a},\hat{Y}_{1}^{b},\dots,\hat{Y}_{K}^{b}\\}$.
We take the cluster representations from $C$ to construct cluster pairs.
Formally, for each cluster, say, $\hat{Y}^{a}_{i}$, we can obtain $2\cdot K-1$
cluster pairs, that is, $\\{\hat{Y}^{a}_{i},\hat{Y}^{k}_{j}\\}$ with
$\hat{Y}^{k}_{j}\neq\hat{Y}^{a}_{i}$, $k\in\\{a,b\\}$ and $j\in[1,K]$. For the
cluster $\hat{Y_{i}^{a}}$, only the pair
$\\{\hat{Y_{i}^{a}},\hat{Y_{i}^{b}}\\}$ is a positive cluster pair, while the
rest are negative pairs. By using the cosine similarity to measure the
similarity between the cluster representations, the cluster-level contrastive
loss for $\hat{Y}^{a}_{i}$ is defined as

$\displaystyle\hat{l^{a}_{i}}=-log\frac{\exp(s(\hat{Y}^{a}_{i},Y^{+})/\tau_{C})}{\sum\limits_{Y^{-}\in
C}\exp(s(\hat{Y}^{a}_{i},Y^{-})/\tau_{C})}$ (5)

where $\tau_{C}$ is the cluster-level temperature parameter, $Y^{+}$ and
$Y^{-}$ are the positive and negative clusters of the current cluster,
respectively. The contrastive loss for the both augmented views of the $i$-th
cluster is defined as

$\hat{l_{i}}=\hat{l_{i}^{a}}+\hat{l_{i}^{b}},$ (6)

where $\hat{l_{i}^{a}}$ is the cluster-level contrastive loss of the $i$-th
cluster in the first view (and $\hat{l_{i}^{b}}$ in the second view). Then the
cluster-level contrastive loss for the $K$ clusters is defined as [14]

$L_{clu}=\frac{1}{2K}\sum_{i=1}^{K}\hat{l_{i}}-Entropy(p(Y^{a}))-Entropy(p(Y^{b})).$
(7)

To avoid the trivial solution that assigns most samples to a single cluster,
the entropy term is incorporated into the loss function, which is defined as

$Entropy(p(Y^{k}))=-\sum_{i=1}^{K}[p(\hat{Y}_{i}^{k})logp(\hat{Y}_{i}^{k})],$
(8)

$\begin{array}[]{ll}p(\hat{Y}_{i}^{k})=\frac{{\sum^{N}_{j=1}}y^{k}_{ji}}{{\|\hat{Y}_{i}^{k}\|}_{1}},\;\text{for}\;k\in\\{a,b\\},\end{array}$
(9)

where $y^{a}_{ji}$ (or $y^{b}_{ji}$) denotes the entry in the $j$-th row and
$i$-th column of the representation matrix $Y^{a}$ (or $Y^{b}$, respectively).

####  II-C3 Overall Loss Function

By minimizing the losses (4) and (7), the network can be trained by maximizing
the similarity between the representations of the positive pairs, while
minimizing the similarity between the representations of the negative pairs,
at the instance-level and the cluster-level, respectively.

In VTCC, we jointly optimize the losses $L_{ins}$ and $L_{clu}$ to enforce the
instance-level and cluster-level contrastive learning and obtain the
clustering result. The overall loss function is defined as

$L_{total}=L_{ins}+L_{clu}.$ (10)

TABLE I: The image datasets used in our experiments. Dataset |  #Samples |  #Classes
---|---|---
RSOD [19] |  976 |  4
UC-Merced [20] |  2,100 |  21
SIRI-WHU [21] |  2,400 |  12
AID [22] |  10,000 |  30
D0 [23] |  4,508 |  40
DTD [24] |  5,640 |  47
Chaoyang [25] |  6,160 |  4
CIFAR-100 [26] |  60,000 |  20

(a) RSOD (b) UC-Merced (c) SIRI-WHU (d) AID

(e) D0 (f) DTD (g) Chaoyang (h) CIFAR-100

Figure 2: Some examples of the eight real-world image datasets used for
evaluation, including four remote sensing datasets [20, 19, 21, 22], a crop
pest dataset [23], a texture dataset [24], a medical dataset [25], and an
object image dataset [26].

##  III Experiments

In this section, we conduct experiments on eight real-world image datasets to
evaluate the proposed VTCC approach against a variety of non-deep and deep
clustering approaches.

###  III-A Implementation Details

In this work, we adopt the setting of data augmentations in BYOL [27] and use
the Adam optimizer with an initial learning rate of 0.0003. A lightweight
version of ViT, i.e., ViT-Small [18], is utilized as our backbone with a
dimension size of 384 and 8 blocks of encoder layers. The input image size is
set to $224\times 224$. We leverage a convolutional stem with four small
$3\times 3$ convolutions instead of a large $16\times 16$ convolution,
followed by $1\times 1$ convolution. The instance projector and the cluster
projector each uses a three-layer MLP, following the projector network of
MoCov3 [12]. Besides, we set the output dimension of the instance projector to
128, which can preserve sufficient information for the instance-level
contrastive learning. As for the cluster projector, the output dimension is
set to the number of clusters, i.e., the true number of classes on each
dataset. The temperature parameters $\tau_{I}$ and $\tau_{C}$ are set to 0.5
and 1.0, respectively. We train the VTCC architecture from scratch for 1,000
epochs with a batch size of 128. All experiments are conducted on a single
NVIDIA RTX 3090 GPU and CUDA 11.0.

###  III-B Datasets and Evaluation Metrics

In our experiments, eight real-world image datasets are used for evaluation,
which are described as follows:

  * •

RSOD [19] is a classical dataset for the remote sensing task, which consists
of 976 images with 4 different classes, including aircraft, oil tank,
overpass, and playground.

  * •

UC-Merced [20] is a collection of 2,100 images with 21 different scenes and
100 images per category.

  * •

SIRI-WHU [21] is another remote sensing image dataset, which contains a total
of 2,400 images in 12 categories, where each category includes 200 images.

  * •

AID [22] is a large remote sensing dataset, including 30 categories of scene
images and a total of 10,000 images.

  * •

D0 [23] consists of 4,508 images with 40 pest classes, which were collected
under real-world conditions in a variety of fields.

  * •

DTD [24] is a complex texture dataset, which consists of 5,640 images with 47
different classes.

  * •

Chaoyang [25] is a medical dataset with 6,160 images and 4 classes of the
colon slides, collected from Chaoyang hospital.

  * •

CIFAR-100 [26] is a widely-used object image dataset. We use its 20 super-
classes as the ground-truth.

For clarity, we summarize the statistics of the eight image dataset in Table I
and illustrate some sample images of them in Fig. 2.

Following the standard evaluation protocol for the image clustering task, we
adopt three widely-used evaluation metrics in the experiments, including the
normalized mutual information (NMI) [28], the accuracy (ACC) [29], and the
adjusted Rand index (ARI) [30]. Note that higher values of these metrics
indicate better clustering results.

TABLE II: The NMI scores of different image clustering methods on eight datasets. Dataset |  RSOD |  UC-Merced |  SIRI-WHU |  AID |  D0 |  DTD |  Chaoyang |  CIFAR-100
---|---|---|---|---|---|---|---|---
$K$-means [31] |  0.162 |  0.204 |  0.145 |  0.209 |  0.299 |  0.119 |  0.024 |  0.083
SC [32] |  0.146 |  0.211 |  0.161 |  0.189 |  0.305 |  0.118 |  0.022 |  0.090
AC [33] |  0.168 |  0.214 |  0.166 |  0.204 |  0.319 |  0.125 |  0.026 |  0.098
NMF [34] |  0.176 |  0.202 |  0.245 |  0.193 |  0.255 |  0.127 |  0.018 |  0.076
PCA [35] |  0.163 |  0.206 |  0.164 |  0.216 |  0.308 |  0.124 |  0.024 |  0.084
BIRCH [36] |  0.148 |  0.225 |  0.162 |  0.204 |  0.315 |  0.123 |  0.026 |  -
GMM [37] |  0.160 |  0.198 |  0.160 |  0.205 |  0.289 |  0.120 |  0.024 |  0.084
DEC [38] |  0.296 |  0.120 |  0.183 |  0.217 |  0.328 |  0.128 |  0.001 |  0.101
IDEC [39] |  0.209 |  0.119 |  0.178 |  0.207 |  0.309 |  0.128 |  0.001 |  0.103
ASPC-DA [40] |  0.054 |  0.137 |  0.103 |  0.060 |  0.153 |  0.084 |  0.026 |  -
IDFD [41] |  0.391 |  0.572 |  0.540 |  0.696 |  0.663 |  0.410 |  0.309 |  0.428
CC [14] |  0.457 |  0.609 |  0.603 |  0.752 |  0.693 |  0.480 |  0.365 |  0.424
VTCC (Ours) |  0.611 |  0.658 |  0.693 |  0.794 |  0.753 |  0.487 |  0.373 |  0.432
TABLE III: The ACC scores of different image clustering methods on eight datasets. Dataset |  RSOD |  UC-Merced |  SIRI-WHU |  AID |  D0 |  DTD |  Chaoyang |  CIFAR-100
---|---|---|---|---|---|---|---|---
$K$-means [31] |  0.388 |  0.200 |  0.229 |  0.163 |  0.204 |  0.090 |  0.320 |  0.137
SC [32] |  0.425 |  0.183 |  0.210 |  0.123 |  0.195 |  0.091 |  0.312 |  0.136
AC [33] |  0.371 |  0.188 |  0.222 |  0.151 |  0.209 |  0.091 |  0.329 |  0.138
NMF [34] |  0.420 |  0.208 |  0.275 |  0.161 |  0.187 |  0.091 |  0.305 |  0.127
PCA [35] |  0.388 |  0.198 |  0.227 |  0.173 |  0.220 |  0.090 |  0.320 |  0.139
BIRCH [36] |  0.396 |  0.202 |  0.222 |  0.147 |  0.205 |  0.092 |  0.329 |  -
GMM [37] |  0.382 |  0.193 |  0.239 |  0.169 |  0.189 |  0.091 |  0.318 |  0.132
DEC [38] |  0.534 |  0.147 |  0.257 |  0.185 |  0.232 |  0.092 |  0.421 |  0.157
IDEC [39] |  0.458 |  0.141 |  0.255 |  0.192 |  0.213 |  0.094 |  0.424 |  0.160
ASPC-DA [40] |  0.464 |  0.073 |  0.183 |  0.079 |  0.107 |  0.067 |  0.325 |  -
IDFD [41] |  0.595 |  0.456 |  0.545 |  0.628 |  0.507 |  0.306 |  0.512 |  0.424
CC [14] |  0.538 |  0.480 |  0.604 |  0.622 |  0.511 |  0.358 |  0.575 |  0.426
VTCC(Ours) |  0.572 |  0.553 |  0.670 |  0.716 |  0.575 |  0.393 |  0.585 |  0.455
TABLE IV: The ARI scores of different image clustering methods on eight datasets. Dataset |  RSOD |  UC-Merced |  SIRI-WHU |  AID |  D0 |  DTD |  Chaoyang |  CIFAR-100
---|---|---|---|---|---|---|---|---
$K$-means [31] |  0.075 |  0.065 |  0.053 |  0.051 |  0.080 |  0.016 |  0.017 |  0.028
SC [32] |  0.096 |  0.038 |  0.041 |  0.029 |  0.039 |  0.012 |  0.005 |  0.022
AC [33] |  0.071 |  0.057 |  0.057 |  0.048 |  0.080 |  0.015 |  0.010 |  0.034
NMF [34] |  0.052 |  0.089 |  0.118 |  0.056 |  0.068 |  0.017 |  0.002 |  0.023
PCA [35] |  0.075 |  0.064 |  0.063 |  0.054 |  0.088 |  0.015 |  0.017 |  0.029
BIRCH [36] |  0.068 |  0.066 |  0.049 |  0.046 |  0.080 |  0.016 |  0.010 |  -
GMM [37] |  0.069 |  0.062 |  0.062 |  0.053 |  0.074 |  0.015 |  0.016 |  0.028
DEC [38] |  0.325 |  0.053 |  0.083 |  0.075 |  0.105 |  0.017 |  0.006 |  0.039
IDEC [39] |  0.144 |  0.042 |  0.079 |  0.073 |  0.093 |  0.017 |  - |  0.042
ASPC-DA [40] |  0.005 |  0.002 |  0.035 |  0.014 |  0.021 |  0.005 |  0.005 |  -
IDFD [41] |  0.362 |  0.354 |  0.389 |  0.547 |  0.439 |  0.174 |  0.259 |  0.276
CC [14] |  0.371 |  0.356 |  0.450 |  0.550 |  0.423 |  0.205 |  0.343 |  0.267
VTCC(Ours) |  0.482 |  0.453 |  0.554 |  0.622 |  0.509 |  0.233 |  0.351 |  0.281

###  III-C Comparisons with State-of-the-Art methods

In this section, we evaluate the performance of the proposed VTCC method
against twelve baseline clustering methods, including seven traditional
clustering methods and five deep clustering methods. The seven traditional
clustering methods include $K$-means [31], Spectral Clustering (SC) [32],
Agglomerative Clustering (AC) [33], Non-negative Matrix Factorization (NMF)
[34], Principle Component Analysis (PCA) [35], Balanced Iterative Reducing and
Clustering using Hierarchies (BIRCH) [36], Gaussian Mixture Model (GMM) [37],
whereas the five deep clustering methods include Deep Embedding Clustering
(DEC) [38], Improved Deep Embedding Clustering (IDEC) [39], Adaptive Self-
Paced Deep Clustering with Data Augmentation (ASPC-DA) [40], Instance
Discrimination and Feature Decorrelation (IDFD) [41], and Contrastive
Clustering (CC) [14]. For NMF and PCA, the clustering results are obtained by
conducting $K$-means on the dimensionally-reduced features. For the other
baseline algorithms, the hyper-parameters will be set as suggested by their
corresponding papers. For each test method, the number of clusters is set to
the true number of classes on this dataset. The experimental results w.r.t.
NMI, ACC, and ARI by different deep and non-deep clustering methods are
reported in Tables II, III, and IV, respectively.

In terms of NMI, as shown in Table II, our VTCC method outperforms the
baseline methods on all the eight datasets. Especially, on the RSOD dataset,
VTCC achieves an NMI of 0.611, while the second best method (i.e., the CC
method) only obtains an NMI of 0.457, where a relative gain of 33.7% is
observed. It is worth noting that the RSOD dataset is a small dataset with
only 976 samples and that our VTCC model is trained from scratch. As for the
other bigger datasets, our VTCC method also exhibits better or significantly
better clustering performance than the baseline methods. In terms of ACC and
ARI, as shown in Tables III and IV, similar advantages of VTCC can be seen in
comparison with the other deep and non-deep clustering methods. The
experimental results demonstrate the superiority of our VTCC method which
jointly leverages Transformer encoder and contrastive learning for the
clustering of complex images.

###  III-D Ablation Study

In this section, we experimentally analyze the influence of different
components in VTCC, including the convolutional stem, the ViT architecture,
and the two contrastive projectors, on the RSOD and Chaoyang datasets.

####  III-D1 Influence of Convolutional Stem

This section conducts experiments to evaluate the influence of the
convolutional stem, which is described in Section II-B1. As shown in Table V,
using convolutional stem leads to an NMI score 0.611 on the RSOD dataset and
0.373 on the Chaoyang dataset, both of which are better than that of using the
original patchify stem in ViT. Similar advantages in terms of ACC and ARI can
also be observed, which confirm that the use of the convolutional stem is
beneficial to the clustering performance of VTCC.

TABLE V: The influence of the convolutional stem.

Dataset |  Split method |  NMI |  ACC |  ARI
---|---|---|---|---
RSOD |  Patchify Stem |  0.570 |  0.565 |  0.461
|  Convolutional Stem |  0.611 |  0.572 |  0.482
Chaoyang |  Patchify Stem |  0.365 |  0.573 |  0.336
|  Convolutional Stem |  0.373 |  0.585 |  0.351

TABLE VI: The influence of ViT architecture.

Dataset |  Model |  Dimension |  #Blocks |  #Heads |  NMI |  ACC |  ARI
---|---|---|---|---|---|---|---
RSOD |  ViT-Tiny |  192 |  4 |  12 |  0.557 |  0.563 |  0.454
|  ViT-Small |  384 |  8 |  12 |  0.611 |  0.572 |  0.482
|  ViT-Base |  768 |  12 |  12 |  0.480 |  0.554 |  0.393
Chaoyang |  ViT-Tiny |  192 |  4 |  12 |  0.340 |  0.580 |  0.328
|  ViT-Small |  384 |  8 |  12 |  0.373 |  0.585 |  0.351
|  ViT-Base |  768 |  12 |  12 |  0.362 |  0.571 |  0.332

TABLE VII: The influence of two contrastive projectors.

Dataset |  Contrastive projectors |  NMI |  ACC |  ARI
---|---|---|---|---
RSOD |  Instance projector only + $K$-means |  0.465 |  0.628 |  0.362
|  Cluster projector only |  0.529 |  0.618 |  0.425
|  Instance projector + Cluster projector |  0.611 |  0.572 |  0.482
Chaoyang |  Instance projector only + $K$-means |  0.343 |  0.478 |  0.240
|  Cluster projector only |  0.310 |  0.546 |  0.279
|  Instance projector + Cluster projector |  0.373 |  0.585 |  0.351

####  III-D2 Influence of Vision Transformer Architecture

To evaluate the influence of the ViT architecture, we compare the performances
of VTCC with different scales of Transformer encoder architectures, including
ViT-Tiny [18], ViT-Small [18], and ViT-Base [18], and report the clustering
results in Table VI. As shown in Table VI, on the small dataset of RSOD, ViT-
Small and ViT-Tiny outperform ViT-Base, probably due to the fact that ViT-Base
is the largest model (among the three) and may lead to over-fitting on the
small dataset. Empirically, the ViT-Small is a suitable choice with sufficient
representation learning capability while maintaining high efficiency.

####  III-D3 Influence of Contrastive Projectors

This section conducts experiments to evaluate the influence of the two
contrastive projectors in VTCC. When testing the performance of the instance
projector, the cluster projector is removed and the $K$-means clustering is
performed on the representation learned by the instance projector. As shown in
Table VII, on the RSOD dataset, only using the instance projector leads to an
NMI of 0.465, whereas only using the cluster projector leads to an NMI of
0.529, both of which are lower than the NMI of jointly using two projectors,
that is, 0.611. Although only using the instance projector can lead to a
better ACC score than using both projectors, yet using both projectors yields
the best performance in terms of the other two metrics on the RSOD dataset,
and in terms of all the three metrics on the Chaoyang dataset.

###  III-E t-SNE Visualization

Our VTCC method is capable of learning discriminative features by jointly
exploiting the Transformer and the contrastive learning. In Fig. 3, we use the
t-SNE [42] visualization to show the changes of the representation quality
with varying number of epochs. As shown in Fig. 3, the quality of the feature
representations learned by VTCC improves as the number of the training epochs
goes from 0 to 500, where clear separation between clusters can be observed.

(a) 0 epoch (NMI=0.231) (b) 50 epoch (NMI=0.454) (c) 100 epoch (NMI=0.506) (d)
500 epoch (NMI=0.611)

Figure 3: The t-SNE visualization of VTCC on the RSOD dataset.

##  IV Conclusion

This paper presents a novel deep clustering approach termed VTCC, which for
the first time unifies the Transformer and the contrastive learning for the
image clustering task. In particular, we utilize a ViT encoder architecture as
the backbone to extract the feature representations of two augmented views,
where a convolutional stem is used to split each augmented sample into a
sequence of patches for the Transformer encoder. Two contrastive projectors
are incorporated to simultaneously enforce the instance-level contrastive
learning and the cluster-level contrastive learning (for global clustering
structure learning). Extensive experiments are conducted on eight real-world
image datasets, which have demonstrated the superiority of VTCC over the
state-of-the-art deep clustering approaches. Notably, the experimental results
reveal the benefits of the joint modeling of Transformer and contrastive
learning for unsupervised image clustering, especially for images with complex
structures. Starting from the VTCC approach, more Transformer architectures
and contrastive learning strategies can be explored for the unsupervised image
clustering task as well as other unsupervised learning tasks.

## References

  * [1] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _Proc. of International Conference on Learning Representations (ICLR)_ , 2021.
  * [2] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advanced in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * [4] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _Proc. of European Conference on Computer Vision (ECCV)_ , 2020.
  * [5] R. Strudel, R. Garcia, I. Laptev, and C. Schmid, “Segmenter: Transformer for semantic segmentation,” in _Proc. of IEEE International Conference on Computer Vision (ICCV)_ , 2021.
  * [6] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _Proc. of International Conference on Machine Learning (ICML)_ , 2020.
  * [7] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [8] X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang, “Self-supervised learning: Generative or contrastive,” _IEEE Transactions on Knowledge and Data Engineering_ , pp. 1–1, 2021.
  * [9] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves _et al._ , “Conditional image generation with PixelCNN decoders,” in _Advanced in Neural Information Processing Systems (NeurIPS)_ , 2016.
  * [10] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” _arXiv preprint arXiv:1511.06434_ , 2015.
  * [11] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [12] X. Chen, S. Xie, and K. He, “An empirical study of training self-supervised vision transformers,” in _Proc. of IEEE International Conference on Computer Vision (ICCV)_ , 2021.
  * [13] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in _Proc. of IEEE International Conference on Computer Vision (ICCV)_ , 2021\.
  * [14] Y. Li, P. Hu, Z. Liu, D. Peng, J. T. Zhou, and X. Peng, “Contrastive clustering,” in _Proc. of AAAI Conference on Artificial Intelligence (AAAI)_ , 2021.
  * [15] Y. Liu, E. Sangineto, W. Bi, N. Sebe, B. Lepri, and M. Nadai, “Efficient training of visual transformers with small datasets,” in _Advanced in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [16] T. Xiao, M. Singh, E. Mintun, T. Darrell, P. Dollár, and R. Girshick, “Early convolutions help transformers see better,” in _Advanced in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [17] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _Proc. of International Conference on Machine Learning (ICML)_ , 2019.
  * [18] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in _Proc. of International Conference on Machine Learning (ICML)_ , 2021.
  * [19] Y. Long, Y. Gong, Z. Xiao, and Q. Liu, “Accurate object localization in remote sensing images based on convolutional neural networks,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 55, no. 5, pp. 2486–2498, 2017.
  * [20] Y. Yang and S. Newsam, “Bag-of-visual-words and spatial extensions for land-use classification,” in _Proc. of SIGSPATIAL International Conference on Advances in Geographic Information Systems_ , 2010.
  * [21] B. Zhao, Y. Zhong, G.-S. Xia, and L. Zhang, “Dirichlet-derived multiple topic scene classification model for high spatial resolution remote sensing imagery,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 54, no. 4, pp. 2108–2123, 2015.
  * [22] G.-S. Xia, J. Hu, F. Hu, B. Shi, X. Bai, Y. Zhong, L. Zhang, and X. Lu, “Aid: A benchmark data set for performance evaluation of aerial scene classification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 55, no. 7, pp. 3965–3981, 2017.
  * [23] C. Xie, R. Wang, J. Zhang, P. Chen, W. Dong, R. Li, T. Chen, and H. Chen, “Multi-level learning features for automatic classification of field crop pests,” _Computers and Electronics in Agriculture_ , vol. 152, pp. 233–241, 2018.
  * [24] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, “Describing textures in the wild,” in _Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014.
  * [25] C. Zhu, W. Chen, T. Peng, Y. Wang, and M. Jin, “Hard sample aware noise robust learning for histopathology image classification,” _IEEE Transactions on Medical Imaging_ , vol. 41, no. 4, pp. 881–894, 2021.
  * [26] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [27] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar _et al._ , “Bootstrap your own latent-a new approach to self-supervised learning,” in _Advanced in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * [28] A. Strehl and J. Ghosh, “Cluster ensembles–a knowledge reuse framework for combining multiple partitions,” _Journal of Machine Learning Research_ , vol. 3, no. 12, pp. 583–617, 2002.
  * [29] D. Huang, C.-D. Wang, J.-S. Wu, J.-H. Lai, and C.-K. Kwoh, “Ultra-scalable spectral clustering and ensemble clustering,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 32, no. 6, pp. 1212–1226, 2020.
  * [30] D. Huang, C.-D. Wang, J.-H. Lai, and C.-K. Kwoh, “Toward multidiversified ensemble clustering of high-dimensional data: From subspaces to metrics and beyond,” _IEEE Transactions on Cybernetics_ , pp. 1–14, 2021.
  * [31] J. MacQueen _et al._ , “Some methods for classification and analysis of multivariate observations,” in _Proc. of Mathematical Statistics and Probability_ , 1967.
  * [32] L. Zelnik-Manor and P. Perona, “Self-tuning spectral clustering,” in _Advanced in Neural Information Processing Systems (NeurIPS)_ , 2005.
  * [33] K. C. Gowda and G. Krishna, “Agglomerative clustering using the concept of mutual nearest neighbourhood,” _Pattern Recognition_ , vol. 10, no. 2, pp. 105–12, 1978.
  * [34] D. Cai, X. He, X. Wang, H. Bao, and J. Han, “Locality preserving nonnegative matrix factorization,” in _Proc. of International Joint Conference on Artificial Intelligence (IJCAI)_ , 2009.
  * [35] A. Martinez and A. Kak, “Pca versus lda,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 23, no. 2, pp. 228–233, 2001.
  * [36] T. Zhang, R. Ramakrishnan, and M. Livny, “BIRCH: An efficient data clustering method for very large databases,” in _Proc. of SIGMOD International Conference on Management of Data_ , 1996.
  * [37] C. Fraley and A. E. Raftery, “Enhanced model-based clustering, density estimation,and discriminant analysis software: MCLUST,” _Journal of Classification_ , vol. 20, no. 2, pp. 263–286, 2003.
  * [38] J. Xie, R. Girshick, and A. Farhadi, “Unsupervised deep embedding for clustering analysis,” in _Proc. of International Conference on Machine Learning (ICML)_ , 2016.
  * [39] X. Guo, L. Gao, X. Liu, and J. Yin, “Improved deep embedded clustering with local structure preservation.” in _Proc. of International Joint Conference on Artificial Intelligence (IJCAI)_ , 2017.
  * [40] X. Guo, X. Liu, E. Zhu, X. Zhu, M. Li, X. Xu, and J. Yin, “Adaptive self-paced deep clustering with data augmentation,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 32, no. 9, pp. 1680–1693, 2019.
  * [41] Y. Tao, K. Takagi, and K. Nakata, “Clustering-friendly representation learning via instance discrimination and feature decorrelation,” _arXiv preprint arXiv:2106.00131_ , 2021.
  * [42] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne,” _Journal of machine learning research_ , vol. 9, no. 11, 2008.