# Towards Granularity-adjusted Pixel-level Semantic Annotation

Rohit Kundu1, Sudipta Paul2, Rohit Lal1, Amit K. Roy-Chowdhury1
1University of California, Riverside 2Samsung Research America
{rkund006@, spaul007@, rlal011<EMAIL_ADDRESS>
###### Abstract

Recent advancements in computer vision predominantly rely on learning-based
systems, leveraging annotations as the driving force to develop specialized
models. However, annotating pixel-level information, particularly in semantic
segmentation, presents a challenging and labor-intensive task, prompting the
need for autonomous processes. In this work, we propose GranSAM which
distinguishes itself by providing semantic segmentation at the user-defined
granularity level on unlabeled data without the need for any manual
supervision, offering a unique contribution in the realm of semantic mask
annotation method. Specifically, we propose an approach to enable the Segment
Anything Model (SAM) with semantic recognition capability to generate pixel-
level annotations for images without any manual supervision. For this, we
accumulate semantic information from synthetic images generated by the Stable
Diffusion model or web crawled images and employ this data to learn a mapping
function between SAM mask embeddings and object class labels. As a result,
SAM, enabled with granularity-adjusted mask recognition, can be used for
pixel-level semantic annotation purposes. We conducted experiments on the
PASCAL VOC 2012 and COCO-80 datasets and observed a $+17.95$% and $+5.17$%
increase in $mIoU$, respectively, compared to existing state-of-the-art
methods when evaluated under our problem setting.

##  1 Introduction

Data has been the driving factor of the modern deep learning era. A tremendous
amount of effort goes into annotating unlabeled data so that specialized
models can be trained in a supervised fashion [29], enabling various task
capabilities. However, it becomes expensive when we need to annotate every
pixel of an image or generate semantic masks. In this work, our goal is to
develop a system that can automatically generate pixel-level
annotations/semantic masks for the desired object categories without any
manual supervision. Towards that goal, we introduce a novel annotation
framework that builds upon the Segment Anything Model (SAM) [9], the overview
of which is shown in Figure 1.

Figure 1: Problem Overview: Given a set of unlabeled images and a set of
object classes a user needs to annotate, our proposed GranSAM can generate
pixel-level semantic annotation without any manual supervision. We develop
GranSAM by enabling SAM with semantic region recognition capability utilizing
synthetic images generated by Stable Diffusion or images collected from web
for the desired object classes.

SAM [9], a promptable foundation model designed for segmentation tasks,
demonstrates proficiency in generating segmentation masks at various
granularities based on input prompts, such as point or bounding box
annotations, owing to its supervised large-scale pretraining. However, SAM
lacks object semantics awareness. To address this limitation, we enhance SAM
to provide pixel-level annotations for user-specified classes without using
manual labels. This enhancement is achieved through the incorporation of a
classifier head on SAM, trained using synthetic images generated via Stable
Diffusion model [20] or web crawled images, capturing essential semantic
information.

The integration of a classifier head, developed on SAM’s mask decoder, plays a
crucial role in automating the generation of semantic segmentation masks with
object-label annotations. Unlike traditional unsupervised methods [4, 17, 26,
31] that require a large-scale in-distribution unlabeled data to learn pixel-
level representation or adaptation to target unlabeled data, our GranSAM
framework strategically overcomes this limitation. By utilizing domain
agnostic mask embeddings of SAM, the classifier trained with set of images
obtained through Stable Diffusion or by crawling the web, seamlessly transfer
to images of different distributions, ensuring robust performance. This unique
feature positions our framework as a practical and versatile solution for
semantic segmentation based annotation systems.

Our proposed GranSAM (SAM, adapted to the granularity of user-defined classes)
model distinguishes itself by adopting a user-centric approach to semantic
segmentation. In contrast to SAM, which generates numerous unlabeled masks,
our strategy is focused and meaningful. Rather than overwhelming users with an
excess of masks, GranSAM specifically provides targeted masks for user-defined
objects, accompanied by meaningful labels. This user-centric methodology
optimizes the segmentation process, aligning with real-world scenarios where
users typically seek specific objects within an image. By tailoring the
granularity of object masks to the user’s specific needs, this approach
enhances the overall usability of the segmentation results.

Following are the contributions of the work:

  * •

We propose GranSAM, a novel semantic segmentation based annotation framework
that does not require any manually labeled images or human interaction.

  * •

We introduce a unique approach to enhance SAM, enabling it to recognize masks
at the desired granularity based on semantic information. Leveraging Stable
Diffusion model or web, we collect a set of images containing user defined
classes. These images guide SAM to identify predicted masks at the desired
granularity.

  * •

Our empirical results demonstrate superior performance compared to state-of-
the-art unsupervised semantic segmentation methods trained using Stable
Diffusion-generated synthetic images or web crawled images.

Figure 2: Overall workflow of GranSAM: Given the list of user-defined target
categories $\mathcal{C}$, we use Stable Diffusion [21] to generate a synthetic
single-object image dataset which is encoded by SAM’s [9] image encoder and a
uniformly spaced grid of $d$ points are generated across the image to prompt
SAM. The image and point embeddings are passed into a transformer decoder, the
output mask embeddings $m_{i}$ (here, $m_{i}\in\mathbb{R}^{d\times 1024}$,
corresponding to the $d$ masks predicted by SAM) of which are used to train a
classifier head $\theta$ to predict objects using a Multiple Instance Learning
(MIL) setup and uncertainty losses (ref. Section 3.5).

##  2 Related Work

Efficient Annotation: Efficient annotation methods aim to reduce the time and
cost associated with the annotation process while maintaining or improving
annotation quality. There are few works [12, 2] that uses human-in-the-loop to
generate image-level annotation efficiently. Interactive object segmentation
methods [1, 14] propose coarse segmentation masks which are then iteratively
corrected by collecting point annotations (object vs background) from a human
annotator. However, to the best of our knowledge, the research area for
developing semantic segmentation annotation framework without access to any
manual supervision remains unexplored.

Unsupervised Semantic Segmentation: Unsupervised semantic segmentation [4, 17,
26, 31, 22, 30, 7] entails the segmentation of an image into meaningful
regions without the reliance on labeled training data. Researchers have
investigated diverse approaches to tackle this challenge, incorporating
methods from unsupervised learning, clustering, and self-supervised learning.
MaskContrast [24] stands out as an approach that performs unsupervised
semantic segmentation within a contrastive learning framework. It assigns
labels to the predicted clusters using Hungarian matching, optimizing test-
time performance metrics by associating labels with clusters that maximize
segmentation accuracy. ACSeg [11] is a state-of-the-art method, wherein they
employed a pre-trained ViT model to mine “concepts” from the pixel
representation space of the unlabeled training images. Unlike previous
methods, ACSeg does not predefine the number of clusters an image will be
partitioned into. Instead, they adaptively conceptualize on different images
due to varying complexity in individual images.

However, these existing unsupervised methods assume access to a large
unlabeled training dataset. This contrasts with our problem formulation, where
we assume a scenario without access to such unlabeled data. In our approach,
we operate under the assumption that we lack prior knowledge of the specific
data distribution the user requires for annotation. We perform on-the-fly
inference with our GranSAM model directly on the user’s data, eliminating the
need for pre-existing knowledge of the data distribution.

SAM Variants: Several methods that build on SAM’s capabilities have been
developed recently. SemanticSAM [10] is a fully supervised method that
segments and recognizes open-set objects at any granularity, and is trained on
a combination of the SA-1B dataset [9] with other panoptic and part-
segmentation datasets. RegionSpot [27] leverages the SAM model and CLIP’s [18]
Vision-Language feature embeddings to perform object detection. However, the
method is dependent on (unlabeled) box annotations for every image,
necessitating a comprehensive annotation effort for each object category.
Grounded-SAM [19] and LangSAM [16] are two similar open-source projects based
on SAM which aim to segment and detect anything with natural language prompts.
However, it uses GroundingDINO [15] to detect objects where the model has been
trained in a supervised fashion using object-level box annotations. In our
work, we do not utilize such object detectors as they require additional
object-level annotation effort.

##  3 Methodology

First, we formally define the problem statement (3.1) and present our
GranSAM’s framework overview (3.2). Then we briefly explain SAM’s architecture
(3.3) and describe our strategy to collect semantic information of different
classes of interest (3.4). Finally, we explain how the collected semantic
information is linked with mask embeddings of SAM to develop our annotation
system (3.5) and the inference steps of the annotation system (3.6).

###  3.1 Problem Statement

Consider, we have a set of unlabeled $N$ images
$\mathcal{X}=\\{x_{i}\\}^{N}_{i=1}$ where, $x_{i}\in\mathbb{R}^{H\times
W\times 3}$. Given a set of $C+1$ target classes
$\mathcal{C}=\\{c_{i}\\}^{C}_{i=0}$ ($C$ object classes and one background
class), our objective is to generate semantic masks
$\mathcal{U}=\\{u_{i}\\}^{N}_{i=1}$ for the unlabeled $N$ images. Here,
$s_{i}\in\mathcal{C}^{H\times W}$ and each image can contain variable number
of objects. We also consider that apart from $\mathcal{X}$, we do not have
access to more unlabeled images from the same distribution for
unsupervised/self-supervised training purpose.

###  3.2 Framework Overview

To generate semantic segmentation masks, the system needs to be able to
distinguish different regions of an image and understand the semantics of
those regions simultaneously. We leverage domain agnostic Segment Anything
Model (SAM) as the backbone of the system to distinguish the regions. Although
SAM can generate masks at any granularity based on input prompts, it does not
have the semantic understanding of the objects present in an image. To guide
the semantic understanding, we utilize Stable Diffusion generated synthetic
images/web crawled images of our classes of interest. Since, we know the
image-level labels of the synthetic or web crawled images, we train a
classifier head on top of SAM’s mask decoder in a weakly-supervised setup. For
every generated mask of SAM, we parse the mask embedding and map it to class
labels using MLP layers. We use multiple instance learning loss to learn the
mapping function. Since, SAM mask embeddings are domain agnostic, our learned
classifier is also robust to distribution shifts and can be directly
applicable to the unlabelled dataset of interest. Figure 2 provides a detailed
architectural overview.

###  3.3 Segment Anything Model Architecture

Segment Anything Model (SAM) [9] is a foundation model for image segmentation
task trained on a dataset consisting of 11M images and 1B masks. It can
generate segmentation masks for images at varying granularity levels based on
input prompts. It consists of three main parts:

Image Encoder. The input images are processed using a pre-trained Vision
Transformer [5] modified to handle high resolution images of dimension
$\mathbb{R}^{3\times 1024\times 1024}$. It is run once per image to obtain
features of dimension $\mathbb{R}^{256\times 64\times 64}$ and can be applied
prior to prompting the model.

Prompt Encoder. SAM takes sparse prompt inputs in the form of point or box
annotations for mask inference. Alternatively, in SAM’s “automatic” mode,
users don’t need to manually define points or bounding boxes in the image;
instead, a uniformly spaced grid of points (say, ‘$d$’ number of total points;
$d=100$ for our case) is generated throughout the image which are used as
point annotations to prompt SAM. These point prompts are represented as
positional embeddings summed with the corresponding learned embeddings (that
indicates if a point is in the foreground or background) in SAM. Since, we do
not have any prior information on where the object of interest is located, we
use this “automatic” mode to generate all possible masks.

Mask Decoder. SAM’s mask decoder consists of a modified Transformer Decoder
block [25] followed by a dynamic mask prediction heads. It maps the image and
prompt embeddings along with an output token to a mask. A learned output token
embedding which is analogous to the `[class]` token in [5], is first inserted
into the set of prompt embeddings. This, along with the image embeddings are
given as input to the transformer decoder that updates the output token
embeddings (denoted by $m_{tokens}\in\mathbb{R}^{d\times 4\times 256}$). The
image embeddings are upscaled by 4$\times$ and a point-wise product is taken
with the output of a 3-layer MLP whose inputs are $m_{tokens}$. This product
is the mask prediction of SAM, and this is performed for all of the $d$
prompts (point annotations) resulting in a set of $d$ binary masks
$\in\mathbb{Z}_{2}^{d\times 256\times 256}$ for each image.

###  3.4 Fetching Semantic Information

Although, SAM can predict masks by distinguishing different regions of the
image, it does not have semantic understanding of the predicted masks. To
enable SAM with recognition capability of user defined classes, we collect
semantic information of those classes using synthetic images or web crawled
images.

Synthetic Images. We use the Stable Diffusion [21] model which can generate
hyper-realistic images given a text prompt. For each of the user-defined
object classes in $\mathcal{C}$, we use the text prompt “a photo of a {c}”
$\forall c\in\mathcal{C}$ and generate 200 single object images. This
synthetic single-object image dataset serves as guide to enable semantic
understanding for Segment Anything Model.

Web Crawled Images. Alternative to synthetic images, we can also use web
crawled images for the same purpose. Through web crawling, we acquire single-
object images for each user-defined object class in $\mathcal{C}$ by utilizing
them as search queries on the web. The web crawled images serve as a valuable
addition to the training data, providing diversity in backgrounds and object
contexts. This approach enhances the model’s ability to generalize across
various visual scenarios and improves its discriminatory capacity for user-
defined object classes.

###  3.5 Enabling Semantic Recognition

The $d$ masks generated by SAM for each image contains semantic maps at all
granularity and also includes background. Our objective is to identify mask(s)
containing the desired object(s) specified by the user.

We introduce a classifier head, denoted as $\theta$, designed to process SAM’s
flattened mask embeddings $m\in\mathbb{R}^{d\times 1024}$. $\theta$ is a
simple 4-layer MLP network. The output of this classifier head
$o_{i}=\theta(m_{i})$, where $o_{i}\in\mathbb{R}^{d\times C}$ corresponds to
the assigned object labels. In training the classifier head with the synthetic
single-object training dataset, each image has a 1-dimensional object label
(image-level label) within a $d$-dimensional input space (d mask embeddings
per image). This unique characteristic necessitates us to use weakly
supervised learning setup, leading to adopt a multiple-instance learning
approach.

Multiple Instance Learning (MIL). In MIL, the $d$ samples (SAM’s mask
embeddings $m_{i}$ for the $i^{th}$ image) are grouped in ‘positive’ and
‘negative’ bags– ‘positive’ bags have at-least one positive instance (a mask
embedding $m_{i}^{j}\in\mathbb{R}^{1\times 1024}$ corresponding to the ground
truth object label $y_{i}$) while ‘negative’ bags have no positive instances.
Utilizing these bags as the training dataset, the objective is to train a
model capable of not only classifying the entire bag but also distinguishing
each individual instance within it as either a positive or negative sample.

To calculate the loss for each bag, denoted as the classifier output ‘$o_{i}$’
in our problem, we aim to express each $d$-dimensional mask embedding through
a single confidence score for each category. For a set of mask embeddings
associated with a given image, the activation score for a specific category is
computed as the average of the $k$-max activations across the number of masks
($d$) for that category. In our case, the dimension $d=100$ remains constant
for all inputs, and we set $k$ as,

$k=\max\left(1,\left\lceil\frac{d}{a}\right\rceil\right),$ (1)

where $a$ is a design parameter. Thus, our class-wise confidence scores for
the $j^{th}$ class of the $i^{th}$ input can be represented as,

$s^{j}_{i}=\frac{1}{k}\max_{\begin{subarray}{c}\mathcal{O}\subset
o_{i}[:,j]\\\ |\mathcal{O}|=k\end{subarray}}\sum_{l=1}^{k}\mathcal{O}_{l}.$
(2)

Subsequently, a softmax non-linearity is employed to derive the probability
mass function across all categories as,
$p^{j}_{i}=\frac{\exp(s^{j}_{i})}{\sum_{j=1}^{C}\exp(s_{j}^{i})}$. It is
necessary to compare this probability mass function with the actual
distribution of labels for each image to calculate the MIL loss. Since each
image (during test-time) can have multiple objects in it, we represent the
labels as a multi-hot vector, where 1 occurs if the object is present, else 0.
We then normalize the ground truth label vector transforming it into a valid
pmf. The MIL loss is subsequently computed as the cross-entropy between the
predicted pmf, denoted as $\mathbf{p_{i}}$, and the normalized ground truth
($y_{i}=\left[y_{i}^{1},\ldots,y_{i}^{C}\right]^{T}$) as,

$\mathcal{L}_{MIL}=\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{C}-y_{i}^{j}\log(p_{i}^{j}),$
(3)

where $N$ is the number of images and $C$ is the number of object categories.

Uncertainty Distillation. To further improve the discriminative ability of the
object classifier, we employ an Uncertainty Distillation setup by leveraging
the uncertainty information during training. The MIL trained classifier from
the previous stage acts as the teacher model $\theta_{t}$ (with parameters
frozen), and the student model $\theta_{s}$ is an untrained classifier having
the same architecture as $\theta_{t}$. The primary motivation behind
uncertainty distillation lies in enhancing the model’s ability to make well-
calibrated predictions and acknowledge its own uncertainties in challenging
scenarios.

For the mask embeddings of the $i^{th}$ image, $m_{i}$, we compute the teacher
logits as $o^{t}_{i}=\theta_{t}(m_{i})$ and the teacher predictions as
$\hat{y}^{t}_{i}=argmax(p^{t}_{i})$, where
$p^{t}_{i}=softmax(o^{t}_{i})\in\mathbb{R}^{d\times C}$ are the object-label
prediction probability scores for all the $d$ masks. We calculate the entropy
(uncertainty) in the teacher logits as
$H^{t}_{i}=-\sum_{j=1}^{C}p^{t}_{ij}\cdot\log(p^{t}_{ij})$. The goal is to
minimize the reliance on (1) high entropy predictions, denoted by
$H^{t,high}_{i}$, which are obtained by thresholding $H^{t}_{i}$ (such that,
$H^{t,high}_{i}=H^{t}_{i}>threshold$ and $H^{t,low}_{i}=H^{t}_{i}<threshold$),
and (2) low entropy incorrect predictions, denoted as
$H^{t,low,incorr}_{i}=H^{t,low}_{i}\cap\\{\hat{y}^{t}_{i}\neq y_{i}\\}$, where
$y_{i}$ is the ground truth object label. Thus, we obtain a set of bad
predictions as $B_{i}=H^{t,high}_{i}\cup H^{t,low,incorr}_{i}$. Here, $B_{i}$
represents the indices of the masks out of all the $d$ masks predicted by SAM
for the $i^{th}$ image, where the classifier head predicts a label with highly
uncertainty, or predicts a wrong label confidently.

Next, we obtain the student logits as $o^{s}_{i}=\theta_{s}(m_{i})$ and
compute the MIL loss, $\mathcal{L}_{MIL}$, as explained in the previous
section. With the obtained bad teacher prediction indices, we compute the bad
student logits as $o^{s,bad}_{i}=o^{s}_{i}[B_{i},:]$. Finally, we compute the
uncertainty loss (to be minimized) as,

$\mathcal{L}_{uncertainty}=\frac{-1}{N}\sum_{i=1}^{N}\sum_{j=1}^{C}\log\left(\frac{\exp(o^{s,bad}_{ij})}{\sum_{j=1}^{C}\exp(o^{s,bad}_{ij})}\right).$
(4)

The net loss used to guide the student model is a weighted sum of the MIL and
uncertainty losses as,
$\mathcal{L}_{s}=\lambda_{1}\cdot\mathcal{L}_{MIL}+\lambda_{2}\cdot\mathcal{L}_{uncertainty}$.
The trained student model $\theta_{s}$ is used for inference on the test data.

###  3.6 Inference

For the images in the test set, we obtain the mask embeddings $m_{i}$ from SAM
in the same way as during training on synthetic / web crawled images. These
embeddings are passed through our trained classifier head $\theta_{s}$ to
acquire object-class confidence scores for each mask within an image. By
applying a predefined threshold to the confidence scores, we selectively
retain masks whose object labels have been predicted with high probability.
Among the selected subset masks for each image, certain masks may be
redundant. We use Non-Maximal Suppression (NMS) to eliminate those overlapping
masks and ensure the retention of masks only corresponding to distinct
objects.

##  4 Experiments

###  4.1 Datasets

Our experiments involve two benchmark datasets for multi-object images: PASCAL
VOC 2012 [6] and COCO-80 [13].

PASCAL VOC: This dataset comprises 1449 images with annotations for 20 object
classes. Noteworthy for its diversity, each image may contain a single or
multiple objects, making it a versatile benchmark for evaluating segmentation
algorithms.

COCO-80: COCO-80, with 80 classes, presents a complex multi-object scene
understanding challenge. Featuring diverse object categories and rich
annotations, including instance-level segmentation masks, it goes beyond
traditional semantic segmentation datasets. Its scenarios involve numerous
objects, intricate interactions, and occlusions, offering a realistic
representation of challenging real-world contexts.

###  4.2 Implementation Details

SAM’s automatic mask generation mode uniformly places a grid of $d$ points
across the image, and for GranSAM, we set $d=100$. We extracted and saved all
$d$ predicted masks from SAM, with the mask embeddings
($m_{i}\in\mathbb{R}^{d\times 1024}$). This one-pass extraction process
ensures efficient storage for subsequent use in our workflow. We generated 200
synthetic single object images per class using the StableDiffuson-v1-4 [21]
model. We first train the classifier head (teacher) $\theta_{t}$, with the MIL
loss (${L}_{MIL}$) as the sole objective with batch size $64$, learning rate
$0.001$ optimized with the Adam optimizer. With these same hyperparameters, we
train the student model $\theta_{s}$ with the objective
$\mathcal{L}_{s}=\lambda_{1}\cdot\mathcal{L}_{MIL}+\lambda_{2}\cdot\mathcal{L}_{uncertainty}$,
where we set $\lambda_{1}=1$ and $\lambda_{2}=0.15$. For our entire workflow,
we utilized a single NVIDIA GeForce RTX 3090 GPU.

###  4.3 Evaluation Metrics

Our quantitative evaluation of GranSAM relies on two key metrics: mean
Intersection over Union (mIOU) and mean Average Precision at an IoU threshold
of 50% (mAP50). mIoU: This metric evaluates the overlap between predicted
segmentation masks and ground truth masks for each object class. It calculates
the per-class Intersection over Union (IoU) as the intersection area divided
by the union area of the predicted and ground truth masks. The average of
these per-class IOU scores, mIOU, provides an overall measure of segmentation
accuracy.

mAP50: Commonly used in object detection and segmentation tasks, mAP50
assesses object localization precision at a 50% IoU threshold. It calculates
Average Precision (AP) for each class based on precision-recall curves, and
mAP50 averages these class-specific AP scores. This metric is crucial for
evaluating the model’s ability to precisely locate objects with moderate
overlap.

###  4.4 Experimental Analysis

Table 1: State-of-the-art comparison with our results on the PASCAL VOC val
data. Although methods like TransFGU [28] outperform GranSAM, they were
trained on the unlabeled data PASCAL VOC train set. When TransFGU [28] and
Leopart [32] were evaluated on our problem setting, they distinctly
underperformed against GranSAM.

Method | Training Data | mIoU | mAP50
---|---|---|---
PASCAL | Synthetic | Web crawl
MoCO v2 [8] | ✓ |  |  | 4.30% | -
InfoMin [23] | ✓ |  |  | 3.70% | -
SWAV [3] | ✓ |  |  | 4.40% | -
MaskContrast [24] | ✓ |  |  | 35.00% | -
TransFGU [28] | ✓ |  |  | 37.15% | -
ACSeg [11] | ✓ |  |  | 47.10% | -
Leopart [32] |  | ✓ |  | 7.21% | -
TransFGU [28] |  | ✓ |  | 2.05% | -
GranSAM |  | ✓ |  | 25.16% | 49.06%
Leopart [32] |  |  | ✓ | 6.79% | -
TransFGU [28] |  |  | ✓ | 2.45% | -
GranSAM |  |  | ✓ | 22.42% | 45.59%

Quantitative Results. In this section, we quantitatively analyze the
performance of our proposed GranSAM for the semantic segmentation annotation
task. Table 1 and Table 2 reports the performance of GranSAM and compare it
with other existing state-of-the-art methods.

GranSAM is trained on synthetic/web-crawled data to recognize object semantics
and applied for inference on the val sets of PASCAL VOC and COCO-80 datasets.
While unsupervised semantic segmentation methods are a common choice for
annotation without manual supervision, state-of-the-art unsupervised methods
typically rely on large-scale unlabeled datasets from the same distribution as
the test set. Given our assumption that we lack access to such large-scale
unlabeled data from the same distribution, a fair comparison with existing
state-of-the-art unsupervised methods is challenging. To establish a fair
baseline for comparison, we setup two unsupervised methods, Leopart [32] and
TransFGU [28], trained on our synthetic/web-crawled data and evaluated on the
PASCAL VOC and COCO-80 datasets. In Tables 1 and 2, our focus is primarily on
the performance of Leopart [32] and TransFGU [28] when trained on our
synthetic/web-crawled data.

Table 2: State-of-the-art comparison with our results on the COCO-80 val data.
We observe that TransFGU [28] performance drops significantly when trained on
synthetic or web crawled images (our problem setting) as compared to when
trained with COCO train data, and GranSAM outperforms these methods in similar
problem settings.

Method | Training Data | mIoU | mAP50
---|---|---|---
COCO | Synthetic | Web crawl
MaskContrast [24] | ✓ |  |  | 3.73% | -
TransFGU [28] | ✓ |  |  | 12.69% | -
ACSeg [11] | ✓ |  |  | 16.40% | -
Leopart [32] |  | ✓ |  | 3.84% | -
TransFGU [28] |  | ✓ |  | 0.95% | -
GranSAM |  | ✓ |  | 8.60% | 30.90%
Leopart [32] |  |  | ✓ | 3.81% | -
TransFGU [28] |  |  | ✓ | 1.02% | -
GranSAM |  |  | ✓ | 9.01% | 31.80%

Class$\rightarrow$ |  background |  aeroplane |  bicycle |  bird |  boat |  bottle |  bus |  car |  cat |  chair |  cow |  diningtable |  dog |  horse |  motorbike |  person |  pottedplant |  sheep |  sofa |  train |  tvmonitor
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Dataset$\downarrow$ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
SD | 79.73 | 0.05 | 29.88 | 14.96 | 7.34 | 34.21 | 54.05 | 47.47 | 9.40 | 5.12 | 21.12 | 6.09 | 15.21 | 8.64 | 44.26 | 60.03 | 16.56 | 14.15 | 11.12 | 26.68 | 22.26
WC | 78.91 | 33.89 | 15.48 | 12.13 | 8.26 | 22.46 | 36.46 | 41.23 | 9.85 | 4.41 | 6.89 | 2.21 | 18.92 | 19.22 | 44.30 | 38.98 | 11.39 | 14.71 | 14.96 | 17.22 | 18.85

Table 3: Classwise mIoU (%) scores obtained by GranSAM on the PASCAL VOC
dataset, when trained on (a) SD: Stable Diffusion generated synthetic images
and (b) WC: Web crawled images.

Table 1 presents the performance comparison between GranSAM and state-of-the-
art methods on the PASCAL VOC dataset. Notably, GranSAM demonstrates
competitive performance compared to established methods like MoCO v2 [8] and
SWAV [3]. However, methods such as MaskContrast [24], TransFGU [28], and ACSeg
[11], trained on the train set of PASCAL VOC, outperform GranSAM. It’s
essential to note that direct comparisons with these methods are challenging,
as they utilize unlabeled data from the test data distribution. However, when
TransFGU [28] and Leopart [32] are trained on our synthetic/web crawled data
and tested on the PASCAL VOC val set, they perform poorly, indicating a lack
of generalization when the training data does not align with the distribution
of the test data.

Table 2 highlights the performance of GranSAM and existing state-of-the-art
methods on the COCO-80 dataset. This dataset, characterized by its diverse set
of 80 object classes and complex scenes, reveals a notable decrease in
performance across all methods, including ours. Challenges inherent in
COCO-80, such as diverse object sizes, occlusions, and intricate interactions,
contribute to the difficulty of achieving high mIoU scores for state-of-the-
art methods. Even so, when the state-of-the-art methods were trained on the
synthetic/web crawled data and tested on the COCO-80 val set, the results are
significantly poor compared to our GranSAM framework. The occurrence of a low
mIoU and a high mAP50 in the COCO-80 dataset in Table 2 suggests a trade-off
between localization precision and segmentation accuracy. A high mAP50
indicates the model’s proficiency in object detection and localization, while
a low mIoU implies challenges in precisely delineating object boundaries or
capturing fine-grained details.

It is essential to highlight a key aspect of our methodology: our training
data originates from a small set of synthetic (200 images per class) or web
crawled data ($\sim$45 images per class) which are single-object images, and
thus distinctly different from the test data distribution. Unlike existing
methods, which often rely on detailed annotations for each image, our method
only requires knowledge of the user-defined class names to be segmented across
the entire dataset. The ability to apply an existing method to a new dataset,
as achieved by GranSAM, without any knowledge of the distribution of the data
in this dataset is a highly desired feature as it enhances generalizability
well beyond what unsupervised domain adaptation can do.

Furthermore, the effectiveness of our method is evident in its ability to
yield competitive results regardless of the data source, be it synthetic or
web crawled images. When TransFGU [28] and Leopart [32] are trained on our
synthetic or web crawled datasets, the performance drops significantly
compared to their performance on in-distribution training data in both PASCAL
VOC and COCO-80 datasets (GranSAM achieves $+17.95$% and $+5.17$% mIoU,
compared to Leopart [32]), emphasizing the unique challenges posed by our
problem setting. This flexibility underscores the robustness of our approach,
allowing for practical applications in scenarios where obtaining extensive
manual annotations per image is challenging or impractical.

Table 4: Ablation study results on PASCAL VOC: Uncertainty Distillation
significantly improves performance in both synthetic and web crawled data
training paradigms. MIL: Multiple Instance Learning.

Method | Training Data | mIoU | mAP50
---|---|---|---
Synthetic | Web crawl
MIL only | ✓ |  | 22.42% | 42.47%
MIL + Distillation | ✓ |  | 25.16% | 49.06%
MIL only |  | ✓ | 17.42% | 39.86%
MIL + Distillation |  | ✓ | 22.42% | 45.59%

Qualitative Results. In Figure 3, we present some qualitative results obtained
by GranSAM. The example in Row-1 depicts a single-object image of class
“sheep”, where SAM has provided several masks which includes background masks,
wherein discrete grass patches in the image have also been segmented
separately. Row-2 shows an example of a multi-object image where the objects
are partially occluded by each other. Row-3 shows two objects belonging to the
same class which are disconnected, while Row-4 shows an example where two
objects belonging to the same class have a connected contour. In all of these
cases, GranSAM has been able to filter out the desired mask along with the
correct object label whereas SAM has segmented several different parts of the
image- both small and large objects, for example, the eyes and the faces have
been segmented separately in Row-4.

The proposed GranSAM model excels in delivering segmentation masks with a
user-centric approach, focusing on the desired granularity. Unlike SAM, which
outputs numerous masks without labels, our strategy is targeted and
meaningful. Instead of overwhelming users with an abundance of masks, GranSAM
provides specific masks for user-defined objects, complete with meaningful
labels. This user-centric approach to tailoring the granularity of object
masks optimizes the user experience and enhances the interpretability of
results, aligning with real-world scenarios where users seek specific objects
within an image, consequently making the segmentation process more usable.

Table 5: Ablation study results on COCO-80: Uncertainty Distillation improves
the performance marginally when $\theta$ is trained with synthetic images and
significantly when it is trained with web crawled images. MIL: Multiple
Instance Learning.

Method | Training Data | mIoU | mAP50
---|---|---|---
Synthetic | Web crawl
MIL only | ✓ |  | 8.01% | 30.89%
MIL + Distillation | ✓ |  | 8.60% | 30.97%
MIL only |  | ✓ | 8.57% | 27.57%
MIL + Distillation |  | ✓ | 9.01% | 31.81%

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

Figure 3: Qualitative results of GranSAM on the Pascal VOC Dataset: The
original image, the ground truth mask, all of the SAM generated masks
overlayed on top of each other, and the masks predicted by GranSAM are shown
respectively in the four columns. For the ground truth and predicted masks,
the colours indicate the class label, while random colours were used for the
SAM masks since SAM does not provide class-labels.

Ablation Study In our ablation study, we investigated the impact of
introducing Uncertainty Distillation ($\mathcal{L}_{uncertainty}$) alongside
the MIL loss ($\mathcal{L}_{MIL}$) in our classifier head ($\theta$) training
framework. The primary objective was to understand the performance
implications of leveraging uncertainty information during training,
particularly in challenging scenarios. The results obtained for the PASCAL VOC
and COCO-80 datasets are tabulated in Tables 4 and 5 respectively.

Our experiments yielded compelling results, showcasing significant performance
improvements, particularly in the PASCAL VOC dataset (with a $+5\%$ increase
in mIoU and $+6.59\%$ in mAP50) when leveraging web-crawled data for training.
The strategic incorporation of Uncertainty Distillation alongside the MIL loss
played a pivotal role in enhancing the model’s discriminative capabilities.
Notably, this improvement is pronounced in the case of COCO-80, where the
utilization of web-crawled training data addresses the challenges posed by the
dataset’s diversity and complexity. The model’s adept handling of
uncertainties in predictions proves crucial in navigating the intricacies of
real-world scenarios presented by the diverse COCO-80 dataset.

A key observation is the model’s improved robustness in scenarios where
traditional methods might struggle, showcasing the practical relevance of
incorporating Uncertainty Distillation. The ability to make well-calibrated
predictions and acknowledge uncertainties translated into a more effective and
reliable segmentation framework.

##  5 Conclusion

In conclusion, GranSAM revolutionizes semantic segmentation annotation by
automating pixel-level annotations and semantic masks, eliminating the need
for labor-intensive manual labeling. Unlike traditional unsupervised methods,
our innovative framework leverages SAM, a domain-agnostic foundation model
pretrained on diverse data. The introduction of a classifier head atop SAM’s
mask decoder, trained with synthetic/web-crawled images, enhanced the
recognition of different regions based on semantics on an unseen test data at
the granularity level defined by the user. Such a system provides a more
efficient and flexible solution for annotation in semantic segmentation.

## References

  * Benenson et al. [2019] Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari.  Large-scale interactive object segmentation with human annotators.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 11700–11709, 2019.
  * Branson et al. [2017] Steve Branson, Grant Van Horn, and Pietro Perona.  Lean crowdsourcing: Combining humans and machines in an online system.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7474–7483, 2017.
  * Caron et al. [2020] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in neural information processing systems_ , 33:9912–9924, 2020.
  * Cho et al. [2021] Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan.  Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16794–16804, 2021.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * [6] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results.  http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.
  * Gao et al. [2022] Shanghua Gao, Zhong-Yu Li, Ming-Hsuan Yang, Ming-Ming Cheng, Junwei Han, and Philip Torr.  Large-scale unsupervised semantic segmentation.  _IEEE transactions on pattern analysis and machine intelligence_ , 2022.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738, 2020.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Li et al. [2023a] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao.  Semantic-sam: Segment and recognize anything at any granularity.  _arXiv preprint arXiv:2307.04767_ , 2023a.
  * Li et al. [2023b] Kehan Li, Zhennan Wang, Zesen Cheng, Runyi Yu, Yian Zhao, Guoli Song, Chang Liu, Li Yuan, and Jie Chen.  Acseg: Adaptive conceptualization for unsupervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7162–7172, 2023b.
  * Liao et al. [2021] Yuan-Hong Liao, Amlan Kar, and Sanja Fidler.  Towards good practices for efficiently annotating large-scale image classification datasets.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4350–4359, 2021.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Ling et al. [2019] Huan Ling, Jun Gao, Amlan Kar, Wenzheng Chen, and Sanja Fidler.  Fast interactive object annotation with curve-gcn.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5257–5266, 2019.
  * Liu et al. [2023] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_ , 2023.
  * Medeiros [2023] Luca Medeiros.  lang-segment-anything.  https://github.com/luca-medeiros/lang-segment-anything, 2023.
  * Melas-Kyriazi et al. [2022] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8364–8375, 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Research [2023] IDEA Research.  Grounded segment anything.  https://github.com/IDEA-Research/Grounded-Segment-Anything, 2023.
  * Rombach et al. [2022a] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10684–10695, 2022a.
  * Rombach et al. [2022b] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022b.
  * Seong et al. [2023] Hyun Seok Seong, WonJun Moon, SuBeen Lee, and Jae-Pil Heo.  Leveraging hidden positives for unsupervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19540–19549, 2023.
  * Tian et al. [2020] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What makes for good views for contrastive learning?  _Advances in neural information processing systems_ , 33:6827–6839, 2020.
  * Van Gansbeke et al. [2021] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool.  Unsupervised semantic segmentation by contrasting object mask proposals.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10052–10062, 2021.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Vobecky et al. [2022] Antonin Vobecky, David Hurych, Oriane Siméoni, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, and Josef Sivic.  Drive&segment: Unsupervised semantic segmentation of urban scenes via cross-modal distillation.  In _European Conference on Computer Vision_ , pages 478–495. Springer, 2022.
  * Yang et al. [2023] Haosen Yang, Chuofan Ma, Bin Wen, Yi Jiang, Zehuan Yuan, and Xiatian Zhu.  Recognize any regions.  _arXiv preprint arXiv:2311.01373_ , 2023.
  * Yin et al. [2022] Zhaoyuan Yin, Pichao Wang, Fan Wang, Xianzhe Xu, Hanling Zhang, Hao Li, and Rong Jin.  Transfgu: a top-down approach to fine-grained unsupervised semantic segmentation.  In _European conference on computer vision_ , pages 73–89. Springer, 2022.
  * You et al. [2023] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang.  Ferret: Refer and ground anything anywhere at any granularity.  _arXiv preprint arXiv:2310.07704_ , 2023.
  * Zadaianchuk et al. [2022] Andrii Zadaianchuk, Matthaeus Kleindessner, Yi Zhu, Francesco Locatello, and Thomas Brox.  Unsupervised semantic segmentation with self-supervised object-centric representations.  _arXiv preprint arXiv:2207.05027_ , 2022.
  * Zhang et al. [2023] Daoan Zhang, Chenming Li, Haoquan Li, Wenjian Huang, Lingyun Huang, and Jianguo Zhang.  Rethinking alignment and uniformity in unsupervised image semantic segmentation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 11192–11200, 2023.
  * Ziegler and Asano [2022] Adrian Ziegler and Yuki M Asano.  Self-supervised learning of object parts for semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14502–14511, 2022.