# A Lightweight Clustering Framework for Unsupervised Semantic Segmentation

Yau Shing Jonathan Cheung Xi Chen Lihe Yang Hengshuang Zhao
The University of Hong Kong

###### Abstract

Unsupervised semantic segmentation aims to label each pixel of an image to a
corresponding class without the use of annotated data. It is a widely
researched area as obtaining labeled datasets are expensive. While previous
works in the field demonstrated a gradual improvement in segmentation
performance, most of them required neural network training. This made
segmentation equally expensive, especially when dealing with large-scale
datasets. We thereby propose a lightweight clustering framework for
unsupervised semantic segmentation. Attention features of the self-supervised
vision transformer exhibit strong foreground-background differentiability. By
clustering these features into a small number of clusters, we could separate
foreground and background image patches into distinct groupings. In our
clustering framework, we first obtain attention features from the self-
supervised vision transformer. Then we extract Dataset-level, Category-level
and Image-level masks by clustering features within the same dataset, category
and image. We further ensure multilevel clustering consistency across the
three levels and this allows us to extract patch-level binary pseudo-masks.
Finally, the pseudo-mask is upsampled, refined and class assignment is
performed according to the CLS token of object regions. Our framework
demonstrates great promise in unsupervised semantic segmentation and achieves
state-of-the-art results on PASCAL VOC and MS COCO datasets.

##  1 Introduction

Figure 1: By clustering attention features into a small number of clusters, we
can obtain patch-level pseudo-masks. Dataset-level, Category-level and Image-
level masks are extracted by clustering features within the same dataset,
category, and image. Masks at different levels each have their own strengths
and weaknesses. In simple backgrounds, all masks could provide accurate
predictions. However, in complex backgrounds, segmentation becomes more
challenging. Dataset-level and Category-level masks both provide a precise
estimation of object structure. Image-level mask, though the coarsest, could
be used to 1) identify the foreground class in Dataset-level and Category-
level masks, and 2) remove noise. By ensuring multilevel clustering
consistency, we can obtain patch-level binary pseudo-masks which will be
further refined.

Semantic segmentation is crucial for the fine-grained understanding and
analysis of visual content. It has been widely applied in various domains,
including medical imaging [10, 19, 36, 33] and autonomous driving [12, 6, 11].
However, supervised methods heavily rely on large-scale labeled datasets [9,
6, 24, 53]. This not only incurs high labeling costs, but also limits models’
effectiveness in real-life applications, as they only operate on predefined
categories. Weakly supervised methods [20, 43, 48, 47] have been explored to
reduce annotation costs, but they still require human effort. This thereby led
to the rise of unsupervised semantic segmentation [16, 5, 49, 13] where
segmentation is performed without reliance on any ground-truth labels.

Earlier methods for unsupervised semantic segmentation involved pixel-level
self-supervised representation learning. They incorporated cross-view
consistency [5, 16], edge detection [15, 52], or saliency prior [39] in their
framework. Until recently, the self-supervised vision transformer has been
proposed [3, 29]. It provides semantically rich patch-based features on
unlabelled data and was widely applied in unsupervised semantic segmentation.
Frameworks utilise features from the self-supervised ViT along with clustering
[22], saliency model [50] and contrastive learning [13, 35] to obtain high-
quality segmentation results. These methods, though with improved accuracy,
are network-dependent. On one hand, substantial computational resources are
still needed for model training, especially when operating on large datasets.
On the other hand, extensive hyperparamter tuning are required for different
datasets, which deteriorates the method’s generalization capability.

We propose a lightweight clustering framework for unsupervised semantic
segmentation. We found that attention features of the self-supervised vision
transformer exhibit strong foreground-background differentiability.
Specifically, clustering could be used to separate corresponding foreground
and background image patches into distinct groupings. To extract patch-level
pseudo-masks, we first obtain Dataset-level, Category-level and Image-level
masks by clustering features within the same dataset, category and image. Both
Dataset-level and Category-level masks provide high-quality predictions.
Image-level mask, despite being the coarsest, facilitates easy foreground-
background detection and could help with the removal of noisy regions. We
maintain multilevel clustering consistency to extract the final binary pseudo-
mask. Figure 1. displays the visualization results of Dataset-level, Category-
level, and Image-level masks as well as the final pseudo-mask extracted.
Afterwards, the patch-level pseudo-mask will be upsampled to pixel level and
post-processed. To assign class labels, the image is cropped according to the
object regions, and their CLS tokens are clustered into respective classes.
With the utilization of simple clustering, our network-free method not only
has low computation cost, but also achieve high-quality segmentation results.

Our main contributions are summarised as:

  * •

We propose a lightweight clustering framework for unsupervised semantic
segmentation

  * •

We ensure multilevel clustering consistency across Dataset-level, Category-
level and Image-level to obtain high-quality pseudo-masks

  * •

Our approach outperforms state-of-the-art methods in both the PASCAL VOC and
MS COCO datasets

  * •

We provide a comprehensive performance evaluation and comparison between DINO
and DINOv2

##  2 Related works

Vison Transformer. Transformer was first introduced in the natural language
processing domain [41, 7, 2]. It’s exceptionally strong at capturing long-
range dependencies and its parallelizable architecture facilitates efficient
training on modern hardware. Vision Transformer [8] then brought the
transformer architecture and attention mechanisms to Computer Vision [46, 37,
25, 4]. It’s able to capture global and local dependencies within images and
is widely applied in the vision-language domain [31, 32, 26]. In 2021, Caron
et al.[3] employed self-distillation with no labels to train the ViT. He
utilised a multi-crop strategy and a momentum encoder in training the model.
The vision transformer was able to produce rich semantic features on unseen
image data. It was therefore widely applied in unsupervised semantic
segmentation. Recently, DINOv2 was proposed [29]. Compared to DINO, the ViT
was trained on a larger dataset consisting of 142 million images. In addition,
they added new regularization methods [34, 38] to make the training more
stable and employed a more efficient training algorithm with the use of
Xformers [21] and Pytorch2. Finally, they adopted a novel functional
distillation pipeline to compress larger models into significantly smaller
ones so to reduce inference cost. DINOv2 currently supports two additional
model types: large and giant, which produce higher-dimensional features. The
advancements in DINOv2 have significantly enhanced the feature quality of the
self-supervised Vision Transformer.

Unsupervised Semantic Segmentation. The development of unsupervised semantic
segmentation has evolved throughout the years. Early methods aim to maintain
semantic consistency at the pixel level to obtain segmentation results [15,
14, 28, 30, 5, 16]. IIC [16] maximizes the mutual information between the
features of augmented images, while PiCIE [5] incorporated geometric
consistency as an inductive bias to learn object representations. Other
methods adopted visual prior [39, 52, 23], cross-view consistency [45, 17, 44,
51] and continuity of video frames [1] in their framework. However, performing
semantic segmentation without prior knowledge is challenging. Recent works
utilised the self-supervised ViT [3, 29] in various ways [54, 40, 49, 50, 22,
13, 35]. TransFGU [49] performs segmentation in a top-down manner by utilizing
class activation maps from the ViT. COMUS [50] applied saliency detectors to
extract object proposals and assigned class labels to them with reference to
image features. ACSeg [22] adaptively maps learnable prototypes into concepts
in images to obtain pixel-level segmentation. Contrastive learning has also
been widely adopted alongside self-supervised ViT in the task[13, 35]. STEGO
[13] utilizes knowledge distillation to learn correspondences between image
features. Hyun et al.[35] further improved the pipeline by uncovering hidden
positive samples and ensuring semantic consistency within regions. Previous
methods, though produced a gradual improvement in segmentation results,
requires substantial model training. In contrast, our network-free framework
has achieved SOTA performance in unsupervised semantic segmentation, while
only requiring the need of simple clustering.

##  3 Method

In this section, we provide details of our lightweight clustering framework.
Figure 2. illustrates the overall structure of our method.

Figure 2:  Illustration of our Lightweight Clustering Framework. We first
utilise the self-supervised vision transformer to extract features from image
patches. Then we perform clustering at the Image-level, Category-level, and
Dataset-level. We further ensure multilevel clustering consistency and extract
the binary patch-level pseudo-mask. The mask is then upsampled and refined
accordingly. Finally, object regions are cropped and clustered into respective
classes.

###  3.1 Preliminaries

Query-Key-Value (QKV) Attention Features. QKV attention features in
transformers [41, 8] are embedded representations used for self-attention.
They allow the model to query, compare, and attend to different parts of the
input sequence during the computation of attention scores. We found that
attention features in self-supervised vision transformers [3, 29] possess
strong foreground-background differentiability. When clustering these features
into a small number of clusters, foreground patches tend to group into a
single group, while background patches span multiple clusters. In our
framework, we perform cosine distance clustering on attention features. This
is because it can be easily implemented, and lots of features need to be
clustered.

Classification (CLS) token. The Classification token in the vision transformer
[8] is added as the first token in the input sequence and provides a
summarized representation of the entire image. We utilize spectral clustering
throughout the framework to cluster CLS tokens since there are fewer
instances, and it can effectively capture non-linear relations.

###  3.2 Cosine Distance clustering

We adopt k-means clustering with cosine distance as the distance metric.
Cosine similarity effectively captures the similarity between attention
features without being biased towards vectors with larger magnitudes. Cosine
distance between attention features $A$ and $B$ can be expressed as

$1-\frac{A\cdot B}{||A||\,||B||}$ (1)

It’s worth noting that k-means with cosine distance could be easily
implemented with the traditional kmeans. This is because cosine distance is
proportional to euclidean distance when vectors are normalised.

$\displaystyle||{X_{1}-X_{2}}||^{2}_{2}$
$\displaystyle=X_{1}^{T}X_{1}+X_{2}^{T}X_{2}-2X_{1}^{T}X_{2}$ (2)
$\displaystyle=2-2X_{1}^{T}X_{2}$ $\displaystyle=2(1-X_{1}^{T}X_{2})$

We first apply L2 normalization to attention features, followed by k-means
clustering to perform cosine distance clustering on the feature vectors.

###  3.3 Multilevel Clustering Consistency

The clustering of attention features is the main focus of our framework. We
found that features clustered at different levels possess respective strengths
and weaknesses, and have distinct uses. Setting two, three, and four clusters
for Image-level, Category-level, and Dataset-level clustering respectively,
yields optimal results across all datasets.

Image-level. Image-level features refer to the attention features of an
individual image. They will be clustered into two clusters, one for foreground
and the other for background. Though binary masks produced at the image-level
are the coarsest, they allow easy identification of foreground and background
regions. This helps select the foreground cluster out of the three and four
clusters in Category-level and Dataset-level clustering respectively, more
details are provided in Sec 3.4. In addition, Image-level mask provides a
rough approximation of object region. This allows us to remove random noises
in the final pseudo-mask by excluding areas not present in the Image-level
mask.

Category-level. Category-level features refer to attention features of all
samples within the same superclass. First, the CLS tokens for all images are
extracted and they’re clustered into their respective superclass. PASCAL VOC,
for example, contains four superclasses: Person, Animal, Vehicle, and Indoor.
Since objects in the same superclass share similar backgrounds, we will
cluster category-level features in each superclass into three clusters, one
for foreground and the remaining for background. By correctly selecting the
foreground cluster, we are able to obtain an accurate pseudo-mask at the
category-level.

Dataset-level. Dataset-level features refer to the attention features across
all samples in the dataset. Since most dataset contains a diverse background,
we found that dividing the features into four clusters yield the best results.
One cluster corresponds to the foreground, while the remaining three clusters
contain background image patches. The Dataset-level mask allows us to obtain
high-quality binary pseduo-mask at the Dataset-level.

To ensure multilevel clustering consistency, we retain common foreground
regions between Dataset-level and Category-level masks, while removing noisy
areas that are not present in Image-level foreground mask. This can be
formulated as:

$(Dataset\cap Category)-(1-Image)$ (3)

###  3.4 Foreground Cluster Selection

Image-level masks are used to identify the single foreground cluster in
Dataset-level and Category-level clustering results. This is because it only
contains two classes and enables easy differentiation between the foreground
and background. The foreground cluster in Dataset-level / Category-level
clustering is identified based on a voting system. Confident Image-level masks
with all four corners predicted as foreground or background will first be
extracted. Then, a mask-flipping check is performed to ensure the mask covers
the foreground instead of the background. Finally, we select the cluster index
in Dataset-level / Category-level clustering that shares the most common
pixels with the binary foreground mask. This step is repeated across the whole
dataset / category and the cluster index that receives the most votes
corresponds to the foreground cluster. The pseduo-code is given in algorithm
1.

Algorithm 1 Foreground Cluster Identification

1:$clusterVote=\\{0:0,1:0,2:0,3:0\\}$

2:for $maskIndex\hskip 2.84544pt\textbf{in range}\hskip 2.84544ptnumMasks$ do

3: $binary=binaryMasks[maskIndex]$

4: $numCorners=$ checkCorner($binary$)

5: if $numCorners==\text{4}$ then

6: $binary=$ CheckflipMask($binary$)

7: $dataset=datasetMasks[maskIndex]$

8: $fgIndex=$checkColor($binary$, $dataset$)

9: $clusterVote[fgIndex]++$

10: end if

11:end for

12:$fgCluster=$ maxKey($clusterVote$)

###  3.5 Mask refinement

The pseduo-mask obtained from the above clustering procedure is at the patch-
level and of low resolution. The mask is first upsampled to the pixel level.
Small components are then removed, and CRF [18] is finally applied to obtain
the refined binary mask.

###  3.6 Class Assignment

The image regions containing the discovered objects will be cropped. They will
then be fed into the ViT to extract corresponding CLS tokens, and clustered
into respective classes.

##  4 Experiments

Figure 3:  PCA visualization of ’Key’ attention features of the PASCAL VOC
validation dataset. We first showcase the Principal Component Analysis
visualization of ’Key’ attention features corresponding to foreground and
background patches from the ground-truth PASCAL VOC validation dataset
(1136016 feature vectors in total). Blue corresponds to foreground image
patches, while red represents the background. Additionally, we present the
results achieved by clustering the attention features into two, three, and
four clusters. Finally, we present the feature distribution achieved through
multilevel clustering.

###  4.1 Datasets and Experimental Settings

Datasets. Following existing works, our framework was first evaluated on
PASCAL VOC [9]. It’s a semantic segmentation dataset that includes 21 classes
– 20 foreground and 1 background. The 20 object class can further be divided
into 4 superclasses. We further evaluated our framework on the more
challenging MS COCO dataset [24]. It contains object categories that are in
context to each other and has 80 things categories. The 80 things categories
could be further divided into 11 supercategories. Following the configuration
of COMUS [50], we transformed the instance segmentation masks into category
masks by merging all masks of the same category together. An additional
background class was added to include pixels not segmented.

Figure 4:  Qualitative results on the PASCAL VOC 2012 dataset. We display the
segmentation results of our framework in comparison with the current state-of-
the-art method COMUS [50] after two rounds of self-training.

Evaluation Protocols and Metrics. The referenced benchmarks were originally
employed for supervised segmentation. In the unsupervised setting, we first
generate clusters equal to the number of classes for each dataset.
Subsequently, we utilise Hungarian matching to assign classes to their
respective ground-truth counterparts. To assess the quality of the
segmentation results, Mean Intersection-Over-Union (mIoU) metric was employed.

Implementation Details. We utilise ViT-Small trained with DINOv2 [29] as the
model to extract image features. Following existing works [27, 42, 40], we
utilise the ’Key’ feature from the last layer of the self-supervised ViT for
clustering. Previous methods resized the width and height of images to 224 and
used ViT trained with DINO to extract attention features [22, 27, 54]. DINO
ViT has a smallest patch size of 8, while DINOv2 ViT has a patch size of 14.
To ensure same feature map resolution (28x28) and fairness in comparison, we
first resized images to 224x224, then upsampled them with bilinear upsampling
to 392x392 to extract features. After clustering, we obtain a patch-level
28x28 binary mask. It will be upsampled with bilinear upsampling to 224x224
and post-processed. Class-labels will be assigned and the result will be
evaluated at this resolution, 224x224.

###  4.2 Qualitative Results

To demonstrate the legitimacy and effectiveness of our framework, we display
the visualization of ’Key’ attention features of the PASCAL VOC dataset using
Principal Component Analysis (PCA), shown in Figure 3. First, we claim that
attention features have strong foreground-background differentiability. From
the feature distribution of the ground-truth dataset, it’s evident that
foreground patches are mainly located on the left, whereas background patches
are situated on the right. Then we present the clustering results when ’Key’
attention features are grouped into two, three and four clusters. The major
observations are as follows: 1) Either foreground or background patches
dominate in respective clusters 2) At a low cluster number, background patches
on the right tend to split into multiple classes, while foreground patches
remain in one cluster. Multilevel clustering further refine the clustering
results by retaining confidence predictions and removing noisy areas. As shown
in the figure, multilevel clustering could better mimic the foreground
background distribution of the ground truth dataset when compared to single-
level clustering. This therefore reflects multilevel clustering could bring
segmentation improvements. In addition, it’s important to note that while
clustering attention features into two and three clusters may seem to align
more closely with the ground-truth dataset, they tend to over-identify a
significant number of foreground image patches. This can severely deteriorate
pseudo-masks obtained, as it is challenging to refine and eliminate
incorrectly masked regions.

Additionally, we present visualization results on PASCAL VOC alongside the
state-of-the-art method COMUS [50] after two rounds of self-training in Figure
4. Our framework introduces four major improvements in segmentation results.
First, we achieve perfect mask segmentation for single objects in complex
backgrounds. This is due to the presence of patches that contain similar and
correlated features. Secondly, our framework has high adaptability to multi-
object segmentation. It can accurately discover and segment multiple objects
in an image due to its effective foreground region discovery. Further to that,
our framework is exceptionally good at segmenting a large number of small
objects within an image. Many existing methods adopt a network-dependent
pipeline. Neural networks, however, suffer from a limited receptive field and
incur lots of downsampling operations. They are thereby unable to capture
contextual information of small objects. In contrast, our network-free
approach can properly uncover and segment small objects. Finally, our
framework could produce high-quality per-image multiclass segmentation.

###  4.3 Quantitative Results

We compare our framework against prior techniques for unsupervised
segmentation, as shown in Table 1. We observe that self-training can lead to
improvements in segmentation results but at the cost of increased
computational resources and training time. When evaluating on the PASCAL VOC
val dataset, our method has demonstrated significant improvement over the
current SOTA. Our work achieved a near 5-point increase in mIoU, at 54.5. It
is important to note that this comparison is made with their results after two
rounds of self-training. This highlights the effectiveness and lightweight
nature of our framework.

We also conducted a detailed evaluation on the MS COCO val dataset, as shown
in Table 2. Since the dataset contain 5000 images, we will run our framework
at a batch size of 1000. Overall, our method achieved a higher mIoU of 21.4.
We were able to identify more of objects. Specifically for classes segmented
with an mIoU higher than 20%, we also attained a higher accuracy of 41.2 mIoU.
This shows that our framework has better segmentation ability and could
uncover more object categories.

Method | mIoU | w self-training mIoU
---|---|---
IIC [16] | 9.8 | /
MaskContrast [39] | 35.0 | /
DSM [27] | 30.8 $\pm$ 2.7 | 37.2 $\pm$ 3.8
Leopart [54] | 41.7 | /
TransFGU [49] | 37.2 | /
MaskDistill [40] | 42.0 | 45.8
ACSeg [22] | 47.1 $\pm$ 2.4 | /
COMUS [50] | 43.8 $\pm$ 0.1 | 50.0 $\pm$ 0.4
LightCluster (_Ours_) | 54.5 $\pm$ 0.3 | /
Table 1: Unsupervised semantic segmentation results on PASCAL VOC. We included results with self-training. | all |  | discovered (with IoU$\geq 20\%$) |  | have cluster (with IoU$>0\%$)
---|---|---|---|---|---
| mIoU |  | number | mIoU |  | number | mIoU
COMUS | 18.2 |  | 33 | 36.6 |  | 73 | 20.2
COMUS (w self-training) | 19.6 |  | 34 | 40.7 |  | 60 | 26.5
LightCluster (_Ours_) | 21.4 |  | 39 | 41.2 |  | 74 | 24.4
Table 2: Unsupervised semantic segmentation evaluated on the MS COCO val
dataset. We present the mIoU result of our framework when evaluated on the MS
COCO val dataset. We display categories with an IoU greater than 20% from all
81 categories and regard them as discovered categories. Additionally, we show
the mIoU for classes with a cluster (IoU larger than zero).

##  5 Ablation Study

In this section, we first provide a detailed analysis of our framework.
Particularly, we explore the effectiveness of multilevel clustering, as well
as the model’s performance under different cluster numbers. The experiments
are conducted on the PASCAL VOC dataset using the DINOv2 ViTS/14 model. Next,
we will conduct an ablation study to compare the performance of DINO and
DINOv2, as well as examine the use of different features in the transformer.

###  5.1 Importance of Multilevel Clustering

We provide a comprehensive evaluation of the effectiveness of multilevel
clustering using the PASCAL VOC dataset. Table 3. presents the segmentation
results when we employ Dataset-level, Category-level, Image-level clustering
progressively to extract pseudo-masks. These masks are upsampled and refined
with the same approach. From the results, we see that utilising Dataset-level
and Category-level clustering together help yield a higher-quality mask.
Further removing noisy areas with Image-level clustering brought improvement
both mIoU, and pixel accuracy.

Cluster Type | mIoU | Pixel Accuracy
---|---|---
Dataset-level | 53.0 | 85.7
\+ Category-level | 53.3 | 85.2
\+ Image-level | 54.5 | 86.7
Table 3: Segmentation result of each component of multilevel clustering We
evaluate the performance of adopting different levels clustering on PASCAL VOC
Figure 5:  Qualitative results using patch tokens for class assignments on the
PASCAL VOC 2012 validation dataset. We display the visualization results of
clustering foreground patch tokens and ’Key’ attention features into 20
classes. The above results reflect patch tokens clustering could accurately
identify object parts, while attention features clustering is unable to do so.
Discovered object parts include human heads (orange), ears of animals
(yellow), mouths of animals (blue), and legs/hands of animals (green). The
body of the horse is grouped into the pink cluster, while the bodies of other
animals, which share a similar texture, are grouped into the purple cluster.

###  5.2 Different cluster number

The number of clusters is the main design choice of our framework. It plays a
huge part in ensuring foreground patches are properly grouped into one cluster
while the background shares the remaining. We display the segmentation results
with different cluster numbers for Dataset-level, Category-level and Image-
level clustering in Table 4. As we can see, our framework is optimal when we
use four, three, and two clusters for Dataset-level, Category-level and Image-
level clustering.

Dataset | Category | Image | mIoU | Pixel Accuracy
---|---|---|---|---
3 | 3 | 2 | 53.8 | 86.3
4 | 4 | 2 | 33.2 | 80.1
5 | 3 | 2 | 30.9 | 79.7
4 | 3 | 2 | 54.5 | 86.7
Table 4: Segmentation result with different cluster number We evaluate the
segmentation performance of our framework with different cluster number of
respective clusters on PASCAL VOC.

From the table, we can also see that the performance of our framework dipped
significantly when we increased the number of cluster at Dataset-level and
Cateogry-level to 5 and 4 respectively. This is because when the image is
split into more clusters, the foreground cluster is split into multiple
clusters. Since our framework assumes that there will only be one foreground
cluster, many foreground patches would be missed out, and thus severely harm
the segmentation results.

###  5.3 Comparing DINO with DINOv2

DINOv2 [29] was recently published by Meta. The vision transformer was trained
with a larger curated dataset and a more efficient training algorithm with the
help of PyTorch2 and Xformers [21]. A newly implemented functional
distillation pipeline was also introduced which brought significant
improvements to token quality. The biggest concern, however, is that DINOv2
had a patch size of 14, while DINO’s smallest patch size is only 8. This might
affect the our pseudo-mask resolution and quality. In Table 5., we compare the
performance of DINO’s and DINOv2’s ’Key’ attention features and CLS token. We
compare the effectiveness of ’Key’ attention features in pseudo-mask
generation and the ability of CLS tokens in class labelling. Since Category-
level features initially depends on CLS tokens to perform class assignments,
to ensure accurate comparison, we’ll use the Dataset-level mask as the final
pseudo mask. Upsampling and mask refinement follows. Object regions will
finally be cropped and class labels will be assigned according to the CLS
token.

Attention feature | CLS token | mIoU | PA
---|---|---|---
DINO ViT-S/8 | DINO ViT-S/8 | 52.8 | 84.6
DINOv2 ViT-S/14 | DINO ViT-S/8 | 51.5 | 85.4
DINO ViT-S/8 | DINOv2 ViT-S/14 | 53.4 | 84.8
DINOv2 ViT-S/14 | DINOv2 ViT-S/14 | 53.0 | 85.7
Table 5: Comparing DINO and DINOv2 performance on PASCAL VOC. We used the
’Key’ attention features to generate binary pseudo-mask, and CLS tokens for
class assignment. Experiments are conducted following configurations in Sec
4.1.

From the table, We could see that 1) DINO ViT could generate higher-quality
pseudo-masks. This is because it has a smaller patch size, and therefore
produces a higher resolution output. 2) DINOv2 ViT CLS token provide better
class representation. It could therefore better assign object regions into
respective classes. Both DINO and DINOv2 can produce masks of high pixel
accuracy.

###  5.4 Comparing attention features and patch token

It is widely questioned why attention features, rather than patch tokens from
the last layer of the ViT, are used for clustering in our framework. This is
because patch tokens incorporate class-specific information, which makes them
unsuitable for foreground-background clustering. Instead, if we were to first
discover foreground image patches according to our clustering pipeline, then
directly cluster the corresponding foreground patch tokens based on the number
of targeted classes, see Figure 5, object parts would be grouped correctly due
to the class information patch tokens contain. On the other hand, attention
features primarily capture foreground-background information. Clustering them
into specific classes would result in messy outcomes. As the goal of
clustering in our framework is to perform foreground-background
differentiation, attention features would be the better choice.

###  5.5 Lightweight framework

Since our network only requires simple clustering to perform unsupervised
semantic segmentation, it is a much quicker and lightweight framework. On
average, an experiment on PASCAL VOC validation dataset would take less than
45 minutes. For existing methods that require self-training, model training
from scratch takes hours, and frequent parameter tuning is necessary.

##  6 Limitation

Although our framework achieved high segmentation performance on PASCAL VOC
and MS COCO datasets, it’s important to note that both datasets treat
background as a single class. Further explorations would be needed to cluster
background regions into their respective classes.

##  7 Conclusion

In this work, we introduce a lightweight clustering framework for unsupervised
semantic segmentation. We found that attention features of the self-supervised
vision transformer possess strong foreground-background differentiability. By
clustering these features into a small number of clusters, we could therefore
separate foreground and background image patches into distinct groupings.
Dataset-level, Category-level and Image-level masks are extracted by
clustering features within the same dataset, category and image. By ensuing
multilevel clustering consistency, we could extract accurate patch-level
pseudo-mask. The mask is then upsampled and refined. Finally, we cluster the
CLS token of object regions to assign class labels. Our proposed method
achieves new state-of-the-art results in both PASCAL VOC and MS COCO dataset.
In addition, we provide a detailed comparison between DINO and DINOv2, as well
as between attention features and patch tokens. We hope that this work will
encourage researchers in unsupervised learning to not only prioritize model
accuracy, but also consider the resources expended and training time. By doing
so, we can make segmentation more cost-effective and efficient.

## References

  * Bielski and Favaro [2022] Adam Bielski and Paolo Favaro.  Move: Unsupervised movable object segmentation and detection.  _Advances in Neural Information Processing Systems_ , 35:33371–33386, 2022.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9650–9660, 2021.
  * Cheng et al. [2022] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1290–1299, 2022.
  * Cho et al. [2021] Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan.  Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16794–16804, 2021.
  * Cordts et al. [2016] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3213–3223, 2016.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ICLR_ , 2021.
  * Everingham and Winn [2012] Mark Everingham and John Winn.  The pascal visual object classes challenge 2012 (voc2012) development kit.  _Pattern Anal. Stat. Model. Comput. Learn., Tech. Rep_ , 2007:1–45, 2012.
  * Fan et al. [2020] Deng-Ping Fan, Ge-Peng Ji, Tao Zhou, Geng Chen, Huazhu Fu, Jianbing Shen, and Ling Shao.  Pranet: Parallel reverse attention network for polyp segmentation.  In _International Conference on Medical Image Computing and Computer-assisted Intervention_ , pages 263–273. Springer, 2020.
  * Fu et al. [2019] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3146–3154, 2019.
  * Geiger et al. [2013] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  _The International Journal of Robotics Research_ , 32(11):1231–1237, 2013.
  * Hamilton et al. [2022] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman.  Unsupervised semantic segmentation by distilling feature correspondences.  In _International Conference on Learning Representations_ , 2022.
  * Harb and Knöbelreiter [2021] Robert Harb and Patrick Knöbelreiter.  Infoseg: Unsupervised semantic image segmentation with mutual information maximization.  In _DAGM German Conference on Pattern Recognition_ , pages 18–32. Springer, 2021.
  * Hwang et al. [2019] Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell D Collins, Tien-Ju Yang, Xiao Zhang, and Liang-Chieh Chen.  Segsort: Segmentation by discriminative sorting of segments.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7334–7344, 2019.
  * Ji et al. [2019] Xu Ji, Joao F Henriques, and Andrea Vedaldi.  Invariant information clustering for unsupervised image classification and segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9865–9874, 2019.
  * Ke et al. [2022] Tsung-Wei Ke, Jyh-Jing Hwang, Yunhui Guo, Xudong Wang, and Stella X Yu.  Unsupervised hierarchical semantic segmentation with multiview cosegmentation and clustering transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2571–2581, 2022.
  * Krähenbühl and Koltun [2011] Philipp Krähenbühl and Vladlen Koltun.  Efficient inference in fully connected crfs with gaussian edge potentials.  _Advances in neural information processing systems_ , 24, 2011.
  * Kumar et al. [2017] Neeraj Kumar, Ruchika Verma, Sanuj Sharma, Surabhi Bhargava, Abhishek Vahadane, and Amit Sethi.  A dataset and a technique for generalized nuclear segmentation for computational pathology.  _IEEE Transactions on Medical Imaging_ , 36(7):1550–1560, 2017.
  * Lee et al. [2019] Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon.  Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5267–5276, 2019.
  * Lefaudeux et al. [2021] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, et al.  xformers: A modular and hackable transformer modelling library, 2021.
  * Li et al. [2023] Kehan Li, Zhennan Wang, Zesen Cheng, Runyi Yu, Yian Zhao, Guoli Song, Chang Liu, Li Yuan, and Jie Chen.  Acseg: Adaptive conceptualization for unsupervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7162–7172, 2023.
  * Lin et al. [2016] Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun.  Scribblesup: Scribble-supervised convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3159–3167, 2016.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * Lu et al. [2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _Advances in neural information processing systems_ , 32, 2019.
  * Melas-Kyriazi et al. [2022] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8364–8375, 2022.
  * Mirsadeghi et al. [2021] S Ehsan Mirsadeghi, Ali Royat, and Hamid Rezatofighi.  Unsupervised image segmentation by mutual information maximization and adversarial regularization.  _IEEE Robotics and Automation Letters_ , 6(4):6931–6938, 2021.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Ouali et al. [2020] Yassine Ouali, Céline Hudelot, and Myriam Tami.  Autoregressive unsupervised image segmentation.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16_ , pages 142–158. Springer, 2020.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , pages 234–241. Springer, 2015.
  * Sablayrolles et al. [2018] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou.  Spreading vectors for similarity search.  _arXiv preprint arXiv:1806.03198_ , 2018.
  * Seong et al. [2023] Hyun Seok Seong, WonJun Moon, SuBeen Lee, and Jae-Pil Heo.  Leveraging hidden positives for unsupervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19540–19549, 2023.
  * Sirinukunwattana et al. [2017] Korsuk Sirinukunwattana, Josien PW Pluim, Hao Chen, Xiaojuan Qi, Pheng-Ann Heng, Yun Bo Guo, Li Yang Wang, Bogdan J Matuszewski, Elia Bruni, Urko Sanchez, et al.  Gland segmentation in colon histology images: The glas challenge contest.  _Medical Image Analysis_ , 35:489–502, 2017.
  * Strudel et al. [2021] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid.  Segmenter: Transformer for semantic segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7262–7272, 2021.
  * Touvron et al. [2022] Hugo Touvron, Matthieu Cord, and Hervé Jégou.  Deit iii: Revenge of the vit.  In _European Conference on Computer Vision_ , pages 516–533. Springer, 2022.
  * Van Gansbeke et al. [2021] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool.  Unsupervised semantic segmentation by contrasting object mask proposals.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10052–10062, 2021.
  * Van Gansbeke et al. [2022] Wouter Van Gansbeke, Simon Vandenhende, and Luc Van Gool.  Discovering object masks with transformers for unsupervised semantic segmentation.  _arXiv preprint arXiv:2206.06363_ , 2022.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2023] Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra.  Cut and learn for unsupervised object detection and instance segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3124–3134, 2023.
  * Wang et al. [2020] Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen.  Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12275–12284, 2020.
  * Wang et al. [2022] Yuan Wang, Wei Zhuo, Yucong Li, Zhi Wang, Qi Ju, and Wenwu Zhu.  Fully self-supervised learning for semantic segmentation.  _arXiv preprint arXiv:2202.11981_ , 2022.
  * Wen et al. [2022] Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi.  Self-supervised visual representation learning with semantic grouping.  _Advances in Neural Information Processing Systems_ , 35:16423–16438, 2022.
  * Xie et al. [2021] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  _Advances in Neural Information Processing Systems_ , 34:12077–12090, 2021.
  * Xu et al. [2022] Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, and Dan Xu.  Multi-class token transformer for weakly supervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4310–4319, 2022.
  * Yao et al. [2021] Yazhou Yao, Tao Chen, Guo-Sen Xie, Chuanyi Zhang, Fumin Shen, Qi Wu, Zhenmin Tang, and Jian Zhang.  Non-salient region object mining for weakly supervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2623–2632, 2021.
  * Yin et al. [2022] Zhaoyuan Yin, Pichao Wang, Fan Wang, Xianzhe Xu, Hanling Zhang, Hao Li, and Rong Jin.  Transfgu: a top-down approach to fine-grained unsupervised semantic segmentation.  In _European Conference on Computer Vision_ , pages 73–89. Springer, 2022.
  * Zadaianchuk et al. [2022] Andrii Zadaianchuk, Matthaeus Kleindessner, Yi Zhu, Francesco Locatello, and Thomas Brox.  Unsupervised semantic segmentation with self-supervised object-centric representations.  _arXiv preprint arXiv:2207.05027_ , 2022.
  * Zhang et al. [2021] Feihu Zhang, Philip Torr, René Ranftl, and Stephan Richter.  Looking beyond single images for contrastive semantic segmentation learning.  _Advances in neural information processing systems_ , 34:3285–3297, 2021.
  * Zhang and Maire [2020] Xiao Zhang and Michael Maire.  Self-supervised visual representation learning from hierarchical grouping.  _Advances in Neural Information Processing Systems_ , 33:16579–16590, 2020.
  * Zhou et al. [2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 633–641, 2017.
  * Ziegler and Asano [2022] Adrian Ziegler and Yuki M Asano.  Self-supervised learning of object parts for semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14502–14511, 2022.

Supplementary Material

In the supplementary material, we begin by comparing the performance of Query,
Value, and Key features in Appendix A. We then discuss different clustering
techniques in Appendix B. This is followed by sections on Computational
Requirements in Appendix C, Limitations in Appendix D, and Broader Impact in
Appendix E.

##  Appendix A Query, Value and Key attention features

Attention features of the self-supervised vision transformer [3, 29] possess
strong foreground-background differentiability. Following existing works [27,
42, 40], we utilise the Key feature from the last layer of the self-supervised
vision transformer for experiments and visualization in our main paper. Here,
we present a detailed performance analysis of Query, Key and Value feature
vectors.

Quantitative Results We evaluate the performance of our framework on PASCAL
VOC using Query, Key, and Value features. Results are shown in Table 6.

Attention Feature | mIoU | PA
---|---|---
COMUS |  43.8 $\pm$ 0.1 | -
COMUS (self-training) |  50.0 $\pm$ 0.4 | -
LightCluster (Query) | 50.5 | 85.7
LightCluster (Key) | 54.5 | 86.7
LightCluster (Value) | 55.7 | 87.0
Table 6: Query, Key and Value segmentation result on PASCAL VOC We evaluate
the performance of adopting Query, Key and Value features in our framework on
PASCAL VOC

We can see that using Query, Key, and Value attention features in our
framework will all bring mIoU higher than that achieved by the current state-
of-the-art method, COMUS [50] after two rounds of self-training. Furthermore,
a high Pixel Accuracy (PA) is achieved. This reflects our framework is sound
and effective.

Qualitative Results Figure 6 and 7 present visualization results of
incorporating Query, Key, and Value features into our framework. It is clear
that they will all bring significant improvements in visualization.
Specifically, we observe four main enhancements:

  * •

We achieve precise mask segmentation for single objects within complex
backgrounds.

  * •

We demonstrate high adaptability to multi-object segmentation

  * •

We excel at segmenting a large number of small objects within a single image

  * •

We produce accurate per-image multiclass segmentation

Comparing between Query, Value and Key features From the quantitative and
qualitative results, we can observe that Key and Value features display
similar performance levels, and both of them outperform Query features. This
is because Query features primarily guides the attention mechanism by
specifying the target or context for comparison. It represents the current
image patch under consideration and lacks an understanding of the broader
context. As a result, masks generated from clustering is likely to be of lower
quality. In contrast, the Key and Value features capture a holistic
understanding of an image. Key features determine the relevance of different
image patches to the current one, while Value features, weighted by these
relevance scores, provide a context-aware summary. Thus, utilising Key and
Value features in our framework could better differentiate between the
foreground and background, and generate accurate masks.

PCA visualization We display the PCA visualization results of the Query, Key,
and Value feature vectors for the ground-truth PASCAL VOC validation dataset,
as well as the results obtained with our clustering framework, in Figure 9.
From the figure, we observe a distinct spatial distribution of the patches.
For the Query feature, the foreground patches are largely concentrated on the
top right, with the remaining space primarily occupied by the background
patches. For the Key and Value features, the foreground patches predominantly
occupy the left side, while the background patches are mostly situated on the
right. These patterns emphasize the strong foreground-background
differentiability of the Query, Key, and Value features in the self-supervised
Vision Transformer. Furthermore, our framework could mimic the image patch
distribution, thereby delivering high-quality segmentation results.

##  Appendix B Comparing different clustering techniques

We explore the use of various techniques for clustering attention features and
CLS tokens. We compare the use of conventional K-means and cosine distance
(CD) K-means for clustering attention features, and conventional K-means with
spectral clustering for grouping CLS tokens on PASCAL VOC. Please note that
due to the large number of attention features to be clustered, memory-
intensive approaches like spectral clustering are not suitable for clustering
them. The results are presented in Table 7.

From the results, we can see that applying cosine distance K-means clustering
and spectral clustering on attention features and CLS token respectively yield
the best performance in our framework.

Attention | CLS Token | mIoU | PA
---|---|---|---
K-means | K-means | 53.3 | 86.3
K-means | Spectral | 53.5 | 86.4
K-means (CD) | K-means | 54.2 | 86.5
K-means (CD) | Spectral | 54.5 | 86.7
Table 7: Comparing different clustering techniques on PASCAL VOC We evaluate
our method’s performance using different clustering techniques on the PASCAL
VOC dataset. Experiments are conducted according to configuration in Sec 4.1.

##  Appendix C Computational Requirements

We perform all our experiments with 2 GTX 1080 Ti GPU cards. An experiment on
PASCAL VOC 2012 would take less than 45 minutes. While for other methods that
require self-training, a full experiment would take hours.

##  Appendix D Limitation

The biggest limitation of our framework is class assignment. In terms of
background, we are unable to segment background regions into respective
classes. The dataset we use, namely PASCAL VOC [9] and MS COCO [24] treat
background region as one class. However, popular semantic segmentation
datasets, such as Cityscapes [6], classify background into various classes.
Our framework would therefore perform worse in these scenarios.

In terms of foreground region class assignments, our framework crops out
object regions and then clusters the corresponding CLS tokens to perform class
assignments. One problem of the above method is that that it cannot separate
connected foreground masks into multiple classes, as only one crop can be
produced. This therefore harms the segmentation performance of our framework.

We present several failure cases in Figure 9. We observe the major problem
with our framework stems from incorrect class assignments. More specifically,
it relates to the previously mentioned issue where our method cannot classify
connected masks representing multiple objects into two or more distinct
classes

Nonetheless, since our framework has exceptionally high capability in
extracting foreground regions with the help of attention features from the
self-supervised vision transformer, we could still achieve state-of-the-art
performance in unsupervised semantic segmentation. Further research should
focus on developing better class assignment schemes for foreground and
background regions.

##  Appendix E Broader Impact

One of the primary objectives of unsupervised semantic segmentation is to
reduce the cost and effort involved in developing segmentation networks. By
eliminating the need for manually annotated pixel-level labels, unsupervised
methods aim to make the training process more efficient. However, many
existing frameworks on unsupervised semantic segmentation [50, 40, 27] still
heavily rely on self-training to boost segmentation performance. Training
neural networks on large-scale unlabeled datasets can still be computationally
intensive and require substantial computational resources, thereby making
segmentation equally expensive.

We hope this work could serve as an initiative to encourage future researchers
in unsupervised learning to explore novel approaches that strike a balance
between segmentation accuracy and computational efficiency. It is crucial to
develop techniques that deliver accurate segmentation results while minimizing
computational costs and resource requirements. By doing so, we can effectively
advance the field of unsupervised semantic segmentation, making it more
accessible, cost-effective, and scalable.

Figure 6:  Qualitative results of using Query, Key and Value features in our
framework. We present the qualitative results of our framework using Query,
Key, and Value features in comparison with the current state-of-the-art method
COMUS [50] after two rounds of self-training. The selected photos used for
comparison are taken from the main paper.

Figure 7:  Additional Qualitative Results on PASCAL VOC 2012 dataset We
present additional qualitative results of our framework operating on Query,
Key, and Value features, compared to the state-of-the-art method COMUS [50]
after two rounds of self-training.

Figure 8:  PCA visualization of Query, Key and Value attention features of the
PASCAL VOC validation dataset. We showcase the Principal Component Analysis
visualization of Query, Key and Value attention features corresponding to
foreground and background patches from the ground-truth PASCAL VOC validation
dataset. Blue corresponds to foreground image patches, while red represents
the background. Then, we present the results achieved with our clustering
framework.

Figure 9:  Failure samples of our framework. The failure cases highlight the
limitations of our method, specifically its inability to accurately separate
connected masks into different classes.