(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: OGQ, Seoul, Korea 22institutetext: Graduate School of AI,
KAIST, Daejeon, Korea 33institutetext: AITRICS, Seoul, Korea44institutetext:
Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA
44email: {shjo.april<EMAIL_ADDRESS>44email: {rsoohyun, sungyub.kim,
<EMAIL_ADDRESS>
# TTD: Text-Tag Self-Distillation Enhancing Image-Text Alignment in CLIP to
Alleviate Single Tag Bias

Sanghyun Jo∗ 11 Soohyun Ryu∗ 22 Sungyub Kim 22 Eunho Yang 2233 Kyungsu Kim† 44

###### Abstract

We identify a critical bias in contemporary CLIP-based models, which we denote
as single tag bias. This bias manifests as a disproportionate focus on a
singular tag (word) while neglecting other pertinent tags, stemming from
CLIP’s text embeddings that prioritize one specific tag in image-text
relationships. When deconstructing text into individual tags, only one tag
tends to have high relevancy with CLIP’s image embedding, leading to an
imbalanced tag relevancy. This results in an uneven alignment among multiple
tags present in the text. To tackle this challenge, we introduce a novel two-
step fine-tuning approach. First, our method leverages the similarity between
tags and their nearest pixels for scoring, enabling the extraction of image-
relevant tags from the text. Second, we present a self-distillation strategy
aimed at aligning the combined masks from extracted tags with the text-derived
mask. This approach mitigates the single tag bias, thereby significantly
improving the alignment of CLIP’s model without necessitating additional data
or supervision. Our technique demonstrates model-agnostic improvements in
multi-tag classification and segmentation tasks, surpassing competing methods
that rely on external resources. Code is available at https://github.com/shjo-
april/TTD.

$*$$*$footnotetext: Equal contribution$\dagger$$\dagger$footnotetext:
Correspondence to Figure 1: In existing image-text alignment research, single
tag bias is evident where the image and text embeddings tend to concentrate
solely on a single tag. (a) Single tag bias in image-text relationships: When
examining the similarity map between the image and text, it is evident that
only a single tag inside the red box is activated, disregarding other elements
mentioned in the text. (b) Single tag bias in image-tag relationships: Even
when observing the similarity values between each tag in the text and the
image, a high value is assigned only to the single tag, while other tags that
describe the image exhibit similar values to insignificant tags. We used TCL
[4] for the analysis.

##  1 Introduction

Research on fully supervised classification/segmentation is widely applied in
the field of computer vision. Various weakly/semi-supervised and few-shot
methods have emerged to reduce labeling costs, but there has been active
research in image-text alignment following CLIP [31]. Despite being trained
solely on images and text (i.e., weak annotation), CLIP can be applied to
downstream tasks that require stronger annotations, such as labels and masks.
This study aims to improve CLIP performance using only weak information, which
is in line with CLIP’s original design intent.

As a major existing limitation to achieve this, we discover that CLIP-based
models exhibit monofocus bias, where they tend to overemphasize a particular
context of image-text relationship. As observed in Fig. 1(a), the similarity
map between image and text reveals activation only in the region related to a
single tag/word (i.e., chameleon), while neglecting others (i.e., branch and
green). We termed this observed encoder output issue as single tag bias,
indicating a flaw in CLIP’s pre-trained text encoder, which over-focuses on
one tag, ignoring others.

Addressing this issue involves creating accurate foreground regions for every
relevant tag, not just the biased one, to fine-tune the encoder to recognize
their collective areas. It requires identifying all relevant tags in the text.
However, testing each tag through CLIP’s text encoder shows persistent single
tag bias, highlighting the challenge of extracting genuine tags solely with
image-text pairs, as seen in Fig. 1(b). This underlines the struggle of
standard CLIP-based tag scoring due to the image encoder’s reduction of pixel
data to a singular tag focus, blocking the recognition of other image-relevant
tags directly from text, illustrated by the pixels’ overemphasis on one tag
(i.e., jacuzzi in Fig. 2(a)), skewing the image’s global (pooling-based)
representation towards it. Even when scoring based on the proportion of pixels
similar to each tag within the image as in segmentation, it still fails to
lower scores for irrelevant tags, thus being unable to correctly extract
meaningful tags from the text (see Sec. 0.B.1).

Figure 2: High-level overview of our method. (a) Tag selection by pixel-tag
scoring: The pooled global image embedding predominantly reflects information
about a single tag, which is “jacuzzi” in this example, due to the
overactivation of a single tag in pixel embedding. (The colored similarity map
in the figure shows the most relevant tag for each pixel.) By employing the
maximum cosine similarity value between the tag and its most correlated pixel
as a score, we minimize the impact of single tag bias observed in the image
encoder. (b) Text-tag self-distillation: The similarity map between text and
image demonstrates only a single tag, whereas it should represent all
meaningful tags in the relationship between image and text. To alleviate the
bias, we train the map between the image and text to align with the union of
maps between the image and the pseudo tags obtained in (a), thus enhancing the
image-text alignment.

To address this, we shift from standard global pooling to a new method
focusing on the pixel index nearest to each tag candidate. This is based on
the finding that pixels closely correlated with a specific tag more accurately
pinpoint image segments, as shown in Fig. 2(a). By using the closest pixel’s
similarity to the tag as the measure for image-tag similarity, we counteract
the bias, thus broadening the encoder’s tag representation capacity.

We used pseudo tags from our scoring to remove unnecessary tag masks, creating
valid foreground masks for all image-relevant tags. This led to a new self-
distillation-based CLIP fine-tuning method, assuming optimal image-text
relationships encompass all pseudo-tag contexts (Fig. 2(b)). We create a
pseudo label, a union of the similarity maps between the image and all pseudo
tags, making CLIP learn this as the real image-text map. This process enabled
the model to recognize all related tag regions, not just one, improving CLIP’s
image-text alignment without additional data or annotations.

In summary, our contributions are as follows:

  * •

Identified bias in pre-trained CLIP models focusing on a single tag in image-
text relationships.

  * •

Proposed new scoring and self-distillation for training CLIP models with
image-related tags extracted from text automatically.

  * •

Achieved superior multi-tag classification and segmentation by enhancing
image-text alignment using solely image-text pairs without external models.

  * •

Provided new tag/mask annotations related to text inputs for future open-
vocabulary research.

##  2 Related works

CLIP [31] stands out as the most prominent model for image-text alignment. It
possesses a remarkable ability to comprehend various free-form texts, making
it widely utilized in zero-shot/open-vocabulary scenarios. Consequently,
various CLIP variants demonstrate comparable performance to models trained
with task-specific annotations across many downstream tasks, such as
classification and segmentation, even if they are trained with only image and
text data.

Table 1: Comparison of our method with existing works

Properties | MaskCLIP | TCL | SimSeg | GroupViT | OVSeg | TagAlign | CoCu | CaSED | Ours
---|---|---|---|---|---|---|---|---|---
(a) Select tags from text | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
(b) Remove single tag bias | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓
(c) No external NLP models | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓
(d) Use image prior for tagging | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓
(e) Follow model-agnostic manner | ✗ | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ | ✓ | ✓

Figure 3: A conceptual comparison between previous approaches and ours. (a)
Image-text alignment models, while useful for various open-vocabulary tasks,
suffer from single tag bias. To alleviate this bias, extracting tags that
reflect the image-text relationship from the text for training is crucial. (b)
However, existing research often relies on external NLP models to extract tags
without considering images, leading to issues: 1) Extracting image-irrelevant
tags. 2) Overlooking image-relevant tags. (c) In contrast, we propose a tag
selection method using only pixel information from images, eliminating the
need for reliance on external models.

###  2.1 CLIP Variants with Pixel-Text Alignment

CLIP faces challenges in downstream tasks requiring localization (e.g.,
segmentation and detection). Consequently, recent models [4, 41, 44, 25, 32]
have emerged that learn pixel-text alignment solely from image-text data
without additional annotations. TCL [4] generates a text-grounded mask that
represents the relationship between an image and corresponding text, then
aligns the text-grounded image and the text. SimSeg [41] aims to align a
sparse portion of the image and the text that only contains critical
information. While these studies showcase impressive performance, they still
exhibit single tag bias in image-text relationships as shown in Tab. 3 as they
implicitly learn alignment from the text. In contrast, our method extracts
meaningful tags from the text and uses them for learning to alleviate the
bias.

###  2.2 CLIP Variants with Tag Selection

####  2.2.1 Utilization of External NLP models

Recent studies [23, 30, 38, 39, 24] have focused on extracting tags from text
for learning, relying on external NLP models (Fig. 3(b)). While some studies
utilize NLP parsers [26, 16, 1] to extract nouns, others leverage Large
Language Models (LLM) like Vicuna [7]. However, these tag selection techniques
entirely disregard image information, resulting in two issues as shown in Fig.
5: 1) Extracting image-irrelevant tags. 2) Overlooking image-relevant tags,
particularly non-nouns. However, our method is free from such issues since we
only utilize the image and text for tag selection.

####  2.2.2 Additional Filtering using Image Information

Some models [37, 9] additionally consider the image to score tags. However,
they utilize globally pooled image representation, suffering from single tag
bias (Fig. 1(b)). CoCu [37] additionally introduces scoring based on retrieved
image relevancy, but this depends on the images retrieved, where tag-relevant
images lead to a low score and vice versa, and also requires high
computational cost (see Sec. 0.B.1). Also, both models rely on external NLP
models and fail to address the omission of image-relevant tags.

Our proposed tag selection method scores tags based on the relationship
between a tag and its most correlated pixel without employing external NLP
models or data. This method mitigates single tag bias of image embedding and
problems of external NLP models as shown in Tab. 2. Then, we present a text-
tag self-distillation method to enhance pixel-tag alignment to address single
tag bias in image-text relationships. As this approach uses only image and
text data without additional annotations, it can be applied in a model-
agnostic manner.

(a) Tag Selection by Pixel-Tag Scoring

(b) Method Overview

Figure 4: Overall framework of our method. (a) Tag selection by pixel-tag
scoring: Utilizing the similarity with the most correlated pixel as scoring,
we identify a significant gap between tags related to the image and irrelevant
ones. This gap guides us in selecting tags with the largest score gap as the
threshold. (b) Method overview: Our fine-tuning approach begins by obtaining
pseudo tags through our tag selection process, which are then used to create
an ideal heatmap representing the image-text relationship. We address single
tag bias by distilling information between the ideal and actual heatmaps.
Additionally, we integrate an auxiliary loss to learn the pixel-tag
relationship. Throughout this process, both the image and text encoders are
trained.

##  3 Method

Our fine-tuning method that alleviates single tag bias consists of two stages:
Tag selection by pixel-tag scoring (Sec. 3.1) and Text-tag self-distillation
(Sec. 3.2). The overall framework is illustrated in Fig. 4.

###  3.1 Tag Selection by Pixel-Tag Scoring

Given an image $I$ of size $H\times W$ and corresponding text $T$ composed of
tags $\mathcal{T}=\\{t_{1}\cdots t_{n}\\}$, our objective is to extract tags,
$\mathcal{T}_{gt}\subset\mathcal{T}$ that accurately represent the image-text
relationship. (e.g. for the first image in Fig. 5,
$\mathcal{T}_{gt}=\\{t_{1},t_{2},t_{3},t_{6},t_{7}\\}$.) Importantly, our
approach relies solely on the image $I$ and the text $T$, without any external
datasets or models.

Let $\mathcal{E}_{V},\mathcal{E}_{T}$ denote the image and text encoders,
respectively, which provide pixel embeddings and global text embeddings:
$\mathcal{E}_{V}(I)\in\mathbb{R}^{H\times W\times
C},\mathcal{E}_{T}(T)\in\mathbb{R}^{C}$ where $C$ is the embedding dimension
size of the encoder output and $H,W$ are height and width of the image. The
global image embedding can be obtained by globally average pooling the pixel
embedding: $\mathrm{gap}(\mathcal{E}_{V}(I))\in\mathbb{R}^{C}$.

A simple approach involves using the similarity between the image and a tag as
a score:

$s_{\mathrm{image}}(t_{i}):=\mathrm{sim}(\mathrm{gap}(\mathcal{E}_{V}(I)),\mathcal{E}_{T}(t_{i})),$
(1)

where $\mathrm{sim}$ denotes similarity metric, and cosine similarity is used
in our case. However, this scoring method is affected by single tag bias, as
depicted in Fig. 1(b). This bias originates from the tendency of most pixels
to be biased towards a specific tag, as shown in Fig. 2(a), thereby
overlooking other relevant tags.

To mitigate this issue, we propose a novel scoring approach that leverages the
pixel embedding most correlated with a tag rather than relying on a global
image embedding:

$s_{\mathrm{pixel}}^{\mathrm{ours}}(t_{i}):=\max_{1\leq h\leq H,1\leq w\leq
W}\mathrm{sim}(\mathcal{E}_{V}(I)_{hw},\mathcal{E}_{T}(t_{i})).$ (2)

This approach ensures more accurate consideration of objects within the image
since each pixel corresponds to its most similar object. Utilizing pixel
embeddings from the stage before pooling to compute similarity scores
increases computational overhead compared to a simple approach. However, it
can be reused because the computed similarity map is used for distillation in
the subsequent step (Sec. 3.2).

Subsequently, we notice a substantial score gap between tags associated with
the image and those not. As illustrated in Fig. 4(a), we arrange all tags
according to $s_{\mathrm{pixel}}^{\mathrm{ours}}$ and compute the score
difference between consecutive tags. We observed the significantly large score
difference between meaningful tags and others at the boundary. Leveraging this
observation, we extract pseudo tags with high scores using the largest score
gap as the boundary: $\mathcal{T}_{pred}\subset\mathcal{T}$.

Our pixel-tag scoring ($s_{\mathrm{pixel}}^{\mathrm{ours}}$) is derived
through a straightforward calculation similar to the simple approach
($s_{\mathrm{image}}$). However, it showcases outstanding tag selection
performance and effectively addresses issues encountered by methods employing
external NLP models as seen in Tab. 2. Furthermore, unlike
$s_{\mathrm{image}}$, which relies on heuristic methods like selecting the
top-$k$ tags or setting a constant threshold due to insignificant score
differences beyond the major tag, our tag selection method automatically
identifies appropriate tags.

###  3.2 Text-Tag Self-Distillation

In this section, we introduce a self-distillation technique designed to
mitigate single tag bias arising from text embedding (Fig. 1(a)). The pseudo
tags $\mathcal{T}_{pred}$ obtained from our tag selection method (Sec. 3.1)
encapsulate the relationship between the image and the text, enabling the
transfer of knowledge from pseudo tags to the text. Specifically, we ensure
alignment between the union of similarity maps of pseudo tags and the image
with that of the text and image:

$\mathrm{sim}(x):=\mathrm{sim}(\mathcal{E}_{V}(I),\mathcal{E}_{T}(x))\in\mathbb{R}^{H\times
W},$ (3)
$\mathcal{L}_{distill}=||\mathrm{sim}(T)-\bigcup_{t_{i}\in\mathcal{T}_{pred}}\mathrm{norm}(\mathrm{sim}(t_{i}))||^{2}_{2},$
(4)

where $\mathrm{sim}(x)$ is an abbreviation of the similarity map between the
image $I$ and a text $x$, $\bigcup$ is the pixel-wise (element-wise) maximum
operation, and $\mathrm{norm}$ is min-max normalization. Here, we apply a
stop-gradient operation on the union similarity map to propagate the gradients
only through the text-based one.

Additionally, we introduce an auxiliary loss function to reinforce the
correlation between tags and the image. Tags that are not selected are deemed
irrelevant to the image; thus, we drive their similarity map toward zero.
Conversely, selected tags are trained to align with the discrete map to
disregard unrelated image parts:

$\mathcal{L}_{tag}=\sum_{t_{i}\in\mathcal{T}}D(t_{i})\ \mathrm{,where}\
D(t_{i})=\begin{cases}||\mathrm{sim}(t_{i})-\mathrm{norm}(\mathrm{sim}(t_{i}))||^{2}_{2}&\mathrm{if}\
t_{i}\in\mathcal{T}_{pred}\\\
||\mathrm{sim}(t_{i})||^{2}_{2}&\mathrm{otherwise}\end{cases}.$ (5)

Similar to text-tag distillation loss, we apply a stop-gradient operation on
the normalized masks.

Our final fine-tuning loss function is defined as follows:

$\mathcal{L}=\mathcal{L}_{distill}+\mathcal{L}_{tag}.$ (6)

Some may question whether it is more effective to view segment-level
relationships using external models like SAM [19] rather than examining pixel-
level relationships. However, our method is unrestricted and versatile since
we self-train the model utilizing only image and text information without
additional supervision, such as ground-truth tag annotation or external
models. We also provide supplementary experiments on advanced techniques
utilizing external models in Sec. 0.B.5.

##  4 Experiments

####  4.0.1 Datasets and Implementation detail

In all experiments, we trained our model solely with image-text pairs without
utilizing additional annotations like truth tag information for the text. We
demonstrated its flexibility by evaluating its performance on two distinct
public models [44, 4]. Furthermore, to cut computational costs, we utilized
the LoRA [17] technique for all our training processes. We used the CC3M and
CC12M datasets [34, 5] for training, applying an additional sample filtering
strategy to reduce the use of largely uncorrelated image-text pairs.

To evaluate the performance of our tag selection method, we manually collected
truth tag information for all image-text pairs of CC3M validation set [34].
Also, to see how much the single tag bias has eased, we generated the tag’s
corresponding mask data for 680 image-text pairs. This was necessary because,
to our knowledge, there is no existing dataset that provides a complete match
of all truth tags and their corresponding masks with the text. We will make
this information available through code release. Details are in Appendix 0.C.

Additionally, for assessing our method’s effectiveness in open-vocabulary
semantic segmentation using tag inputs, we use six established benchmarks:
PASCAL VOC (VOC) [12], PASCAL Context (Context) [29], COCO-Stuff (Stuff),
COCO-Object (Object) [3], Cityscapes (City) [10], and ADE20k (ADE) [43].
Further evaluations on segmentation performance with referring datasets,
alongside conventional open-vocabulary classification tasks, and specifics of
these assessments are detailed in Sec. 0.A.1 and 0.B.

####  4.0.2 Evaluation metrics

For evaluating tag selection, we use metrics like precision, recall, accuracy,
F1 score, and mean Average Precision (mAP) as established in existing studies
[15, 21]. Precision measures the fraction of accurately identified positive
tags among all predicted as positive, and recall quantifies the fraction of
accurately identified positive tags out of all true positive tags. Notably,
since there is no predetermined label set, we adopt sample-wise mAP [20] over
the standard class-wise mAP.

To evaluate segmentation performance, we use mean Intersection over Union
(mIoU). For our proposed benchmark, considering the assessment of a model’s
ability to capture the relationship between captions and images, we refer to
this mIoU as CaptionIoU. Furthermore, given our binary mask annotation, we
also report the mean false positive rate (mFPR) and mean false negative rate
(mFNR) to provide a comprehensive assessment of segmentation quality.

###  4.1 Main Results

Figure 5: Qualitative results of tag selection. Tag selection using external
models [26, 7, 2] suffers from two problems because they do not consider image
information. 1) Extracting image-irrelevant tags (red): Select tags that do
not correlate with the corresponding image content. 2) Overlooking image-
relevant tags (blue): Cannot select tags related to the image, particularly
non-nouns in the case of NLTK. Table 2: Multi-tag selection. The best results
are bold and the second best results are underlined. P, precision; R, recall;
F1, F1 score.

(a) Comparison based on the use of NLP models

Method | P | R | F1 | Acc
---|---|---|---|---
NLTK | 59.8 | 83.7 | 69.8 | 79.6
Vicuna-7B | 44.1 | 71.0 | 54.4 | 70.9
Vicuna-33B | 52.7 | 70.7 | 60.4 | 75.9
Qwen-72B | 69.3 | 56.2 | 62.1 | 80.9
$s_{\mathrm{pixel}}^{\mathrm{ours}}$ (Eq. 2) | 82.9 | 74.5 | 78.5 | 88.6
\+ TTD (Ours) | 88.3 | 78.0 | 82.8 | 91.0

(b) Comparison based on scoring methods

Scoring | P | R | F1 | Acc | mAP
---|---|---|---|---|---
$s_{\mathrm{image}}$ (Eq. 1) | 92.5 | 28.6 | 43.7 | 79.5 | 83.2
$s_{\mathrm{text}}$ (Eq. 7) | 85.6 | 29.7 | 44.1 | 79.0 | 82.1
$s_{\mathrm{image}}+s_{\mathrm{text}}$ | 85.5 | 45.1 | 59.0 | 82.6 | 84.5
$s_{\mathrm{pixel}}^{\mathrm{ours}}$ (Eq. 2) | 82.9 | 74.5 | 78.5 | 88.6 | 90.3
\+ TTD (Ours) | 88.3 | 78.0 | 82.8 | 91.0 | 93.7

####  4.1.1 Multi-Tag Selection

We assess the performance of our pixel-tag scoring-based method (Sec. 3.1) in
the multi-tag selection task (i.e., the performance to identify all image-
related tags given only the text without prior knowledge of the tags such as
pre-defined label set). This evaluation is conducted in comparison to methods
that use NLP models [26, 7, 2] and other basic scoring techniques. For
scoring, we utilize image and text encoders from TCL [4], setting the
threshold for other scoring approaches at 0.5.

As depicted in Fig. 5, tag selection with external NLP models encounters two
main issues: identifying tags not pertinent to the image and missing relevant
tags due to ignoring visual information. As seen in the last example, NLTK
selects irrelevant tags (e.g., ‘/’) due to the overfit on pre-trained sentence
structures. Nonetheless, our method largely extracts tags relevant to the
image, addressing significant issues in conventional NLP models, especially
identifying tags relevant to the image.

This superiority is further evident in Tab. 2. Higher precision indicates
better filtering of image-irrelevant tags, while higher recall denotes
improved extraction of image-relevant tags. As shown in Tab. 2(a), NLTK, the
most widely used NLP parser for tag selection, exhibits high recall but at the
expense of low precision, as it extracts all nouns without considering image
information. Large Language Models (LLMs) show inferior performance compared
to ours due to hallucination issues and lack of image consideration. In
contrast, our method outperforms NLP models in both F1 score and accuracy with
a significant margin, showing even better performance after fine-tuning. This
indicates that our method effectively addresses the limitations of NLP models.

Tab. 2(b) presents the tag selection performance with different scoring
methods. Following [9], we define a score that measures the similarity between
the text and tag in addition to image-tag scoring ($s_{\mathrm{image}}$):

$s_{\mathrm{text}}(t_{i}):=\mathrm{sim}(\mathcal{E}_{T}(T),\mathcal{E}_{T}(t_{i})).$
(7)

While image-tag scoring ($s_{\mathrm{image}}$) demonstrates high precision, it
experiences very low recall due to the single tag bias, often leading to the
selection of only one tag so if misses other image-related tags. Similarly,
text-tag scoring ($s_{\mathrm{text}}$) and their combination show comparable
issues. However, our approach significantly mitigates the single tag bias,
showing high recall. Also, it exhibits high accuracy and mAP, which indicates
the effectiveness of our scoring method. We further examine sensitivity to
different thresholds and alternative scoring techniques that incorporate
external data beyond the image-text pair in Sec. 0.B.1.

####  4.1.2 Text-level Semantic Segmentation

In this task, we evaluate text-level segmentation performance, where the text
encoder of CLIP processes text input using existing CLIP-based models [4, 44]
and followed by the application of our fine-tuning method, TTD. Specifically,
this task evaluates whether the model accurately understands the relationship
between the image and the given text. As depicted in Fig. 6, traditional
models show a single tag bias, mainly capturing the spatial region of only one
tag. TCL often generates masks that extend beyond the relevant context. Yet,
after fine-tuning with our method, the single tag bias reduces, and the fine-
tuned models accurately activate all regions associated with image-related
tags in the respective text.

The effectiveness of TTD is further illustrated in Tab. 3, where its
integration enhances the models’ comprehension of the text-image relationship,
leading to improved CaptionIoU scores (i.e., +7.2% on average). Notably,
MaskCLIP exhibits a significant decrease in the mFNR score (i.e., 0.169) post-
fine-tuning, indicating its improved understanding beyond a single tag.
Additionally, TCL demonstrates a noteworthy decrease in mFPR. It means that
after fine-tuning, the model can exclude portions unrelated to the text,
indicating that our fine-tuning method enhances model alignment as well.

Figure 6: Qualitative results on text-level segmentation. Table 3: Text-level
semantic segmentation.

Method | CaptionIoU(%) | mFPR | mFNR
---|---|---|---
MaskCLIP [44] | 41.0 | 0.179 | 0.411
\+ TTD (Ours) | 50.2 (+9.2) | 0.256 | 0.242
TCL [4] | 60.4 | 0.199 | 0.198
\+ TTD (Ours) | 65.5 (+5.1) | 0.163 | 0.182

###  4.2 Results on Open-Vocabulary Semantic Segmentation

In the previous task, we evaluated the classification and segmentation
performance of TTD-trained CLIP using text input, measuring how well the model
understands the relationship between the image and the text. Subsequently, we
conducted experiments on standard open-vocabulary semantic segmentation where
the text encoder was provided with tag input. Note that our TTD training did
not utilize actual tag annotation from the text. In Fig. 7, we display
segmentation results with and without TTD. Our approach effectively detects
the presence of objects in the image, unlike conventional models that identify
objects not actually present in the image.

Tab. 4 contrasts the tag-level segmentation performance of studies employing
only image-text pairs without additional annotations or external models akin
to ours for fair comparison. For this reason, models [23, 11, 27, 13] that
utilize additional annotations or external models (e.g. mask proposal
networks) are also excluded. Additionally, we compared models that utilize
NLP-based models($\mathcal{L}$ in the table) for tag extraction. Implementing
our fine-tuning technique results in an average increase in mIoUs of 8.5% for
MaskCLIP and 3.5% for TCL. Furthermore, TCL enhanced with our method
outperforms existing models on benchmarks such as Pascal VOC, Pascal Context,
and COCO-Object. This indicates that our approach successfully augments model
alignment in addition to alleviating single tag bias.

At this point, TagAlign [24] achieved better performance on datasets that have
at least 150 classes such as COCO-Stuff/ADE20K than our model. The reason for
this is that TagAlign utilized multiple text information for a single image,
whereas we opted for simplicity by using only a single text corresponding to
each image. Therefore, the performance difference arises from the relatively
insufficient positive/negative tag information in our method, indicating the
potential value in further improving our fine-tuning method.

Figure 7: Qualitative results on tag-level segmentation. Table 4: Open-
vocabulary semantic segmentation. The methods are all trained only with image
and text data, without additional annotations or external models. We use
ViT-B/16 as the backbone for all methods. $\mathcal{L}$, external language
models.

Method | Train datasets | $\mathcal{L}$ | VOC | Context | Object | Stuff | City | ADE | Avg.
---|---|---|---|---|---|---|---|---|---
GroupViT [38] | CC12M+YFCC | ✓ | 51.1 | 19.0 | 27.9 | 15.4 | 11.6 | 9.4 | 22.4
ViewCo [33] | CC12M+YFCC | ✓ | 52.4 | 23.0 | 23.5 | - | - | - | -
CoCu [37] | CC3&12M+COCO | ✓ | 51.4 | 23.6 | 22.7 | 15.2 | 22.1 | 12.3 | 24.6
OVSegmentor [39] | CC4M [39] | ✓ | 53.8 | 20.4 | 25.1 | - | - | - | -
TagAlign [24] | CC12M | ✓ | 53.9 | 33.5 | 33.3 | 25.3 | 27.5 | 17.3 | 31.8
ReCo [35] | ImageNet1K | ✗ | 25.1 | 19.9 | 15.7 | 14.8 | 21.1 | 11.2 | 18.0
ZeroSeg [6] | ImageNet1K | ✗ | 40.8 | 20.4 | 20.2 | - | - | - | -
ViL-Seg [25] | CC12M | ✗ | 37.3 | 18.9 | 18.1 | - | - | - | -
SimSeg [41] | CC3&12M | ✗ | 57.4 | 26.2 | 29.7 | - | - | - | -
SegCLIP [27] | CC12M+COCO | ✗ | 52.6 | 24.7 | 26.5 | 16.1 | 11.2 | 8.8 | 23.3
MaskCLIP [44] | - | ✗ | 29.3 | 21.1 | 15.5 | 14.7 | 21.6 | 10.4 | 19.0
\+ TTD (Ours) | CC3&12M | ✗ | 43.1 (+13.8) | 31.0 (+9.9) | 26.5 (+11.0) | 19.4 (+4.7) | 32.0 (+10.4) | 12.7 (+2.3) | 27.5 (+8.5)
TCL [4] | CC3&12M | ✗ | 55.0 | 33.8 | 31.6 | 22.4 | 24.0 | 15.6 | 30.4
\+ TTD (Ours) | CC3&12M | ✗ | 61.1 (+6.1) | 37.4 (+3.6) | 37.4 (+5.8) |  23.7 (+1.3) | 27.0 (+3.0) |  17.0 (+1.4) | 33.9 (+3.5)

###  4.3 Ablation

Table 5: Ablation studies

(a) Effect of loss terms

$\mathcal{L}_{distill}$ (Eq. 4) | $\mathcal{L}_{tag}(\lx@cref{creftype~refnum}{eq:L_tag})$ | CaptionIoU | mIoU
---|---|---|---
✗ | ✗ | 60.4 | 55.0
✗ | ✓ | 60.7 (+0.3) | 58.5 (+3.5)
✓ | ✗ | 63.6 (+3.2) | 60.8 (+5.8)
✓ | ✓ |  65.5 (+5.1) |  61.1 (+6.1)

(b) Effect of tagging method

Method | CaptionIoU | mIoU
---|---|---
Baseline [4] | 60.4 | 55.0
NLTK [26] | 61.8 | 56.5
$s_{\mathrm{image}}$ (Eq. 1) | 56.3 | 52.5
$s_{\mathrm{pixel}}^{\mathrm{ours}}$ (Eq. 2) | 65.5 | 61.1

In this section, we conducted ablation experiments to evaluate the impact of
the techniques employed in our fine-tuning method. The assessment was
performed using TCL [4] on our text-level, and Pascal VOC [12] for tag-level
segmentation.

####  4.3.1 Effect of Fine-tuning Losses

Our method incorporates two loss terms, $\mathcal{L}_{distill}$ and
$\mathcal{L}_{tag}$, for fine-tuning. $\mathcal{L}_{distill}$ aims to mitigate
single tag bias, while $\mathcal{L}_{tag}$ aims to enhance the pixel-tag
correlation. When solely utilizing $\mathcal{L}_{tag}$, there was a marginal
performance enhancement observed during the min-max normalization process for
each mask, as depicted in Tab. 5(a). However, the inclusion of
$\mathcal{L}_{distill}$ significantly alleviates single tag bias, leading to
an improvement in both CaptionIoU and mIoU (i.e., +3.2% and +5.8%,
respectively). The best performance is achieved when both losses are combined,
learning each tag mask and union mask simultaneously.

####  4.3.2 Effect of Tag Selection Method

In our approach, tag selection is conducted via pixel-tag scoring (Eq. 2) to
extract pseudo tags representing the image-text relationship. Alternatively,
we examine NLTK and simple image-tag scoring (Eq. 1) for tag selection, as
shown in Tab. 5(b). When employing NLTK to extract nouns from texts, due to
the aforementioned issues (extraction of image-irrelevant tags and omission of
image-relevant tags), the model struggles to generate the ideal text-image
similarity map (i.e., the union of pseudo-tag driven maps). Consequently, the
generated union map may include the activation of unrelated tags or omit
activations necessary for the image-tag relationship, resulting in minor
improvement. On the other hand, using image-tag scoring tends to extract a
single tag, exacerbating single tag bias and resulting in a decline in both
CaptionIoU and mIoU. In contrast, our tag selection method exhibits
significantly enhanced performance in both tasks, indicating that the pseudo
tags extracted using our method effectively capture the relationship between
image and text.

##  5 Conclusion

Our research addresses the misalignment between images and text inherent in
existing CLIP-based models. Specifically, we present a fine-tuning technique
that resolves the issue where image and text representations are overly
focused on a single tag, thereby enhancing both image segmentation and multi-
tag classification within the text. This was achieved using only image and
text pairs, without additional annotations. The key outcome of addressing the
single tag bias is enabling CLIP to correctly identify all truth tags related
to the image within the text, akin to how class activation map studies provide
pixel-level class information without direct input to the network. Our TTD
training method, which does not employ external modules for tag extraction,
ensures that the predicted tags by the fine-tuned model have direct relevance
to the image. This feature can effectively address the hallucination issue
seen in conventional tag generation models, where natural language processing-
based models might generate tags unrelated to the target image. Hence, our
approach is valuable as it offers a solution to the hallucination problem,
suggesting significant potential for various applications.

##  Appendix 0.A Experimental Details

###  0.A.1 Implementation Details

Table 6: Implementation details.

|  MaskCLIP [44] |  TCL [4]
---|---|---
Image size | 448 $\times$ 448
Batch size | 32
Total epochs | 1
Optimizer | AdamW
Learning rate | 1e-4
Weight decay | 5e-5
Data augmentation | random resize (320-640) $\rightarrow$ random crop (448 $\times$ 448)
# params (M) |  150.2 (original: 149.6) |  179.3 (original: 178.3)
Mask post-processing |  CRF |  PAMR
Prompt template | no template used during train & OpenAI template during inference

Table 7: Effect of sample pruning.

Sample pruning | # samples | training time (hrs) | mIoU @VOC
---|---|---|---
Baseline | - | - | 55.0
✗ | 12,795,608 | 247 | 60.5 (+5.5)
✓ | 1,758,959 | 34 |  61.1 (+6.1)

We conduct model training using 4 NVIDIA A100 GPUs, each having 80GB of
memory. Implementation details are summarized in Tab. 6. Most settings,
including image size, prompt template usage, and post-processing, align with
the original setup of baseline models [44, 4]. Our batch size is set to 32,
the maximum feasible size given our available GPU resources. Training data
consists of the CC3M and CC12M datasets [34, 5], with additional sample
pruning to remove noisy samples. We calculate cosine similarity between image
and text embeddings for all pairs, sampling those with similarity values
exceeding the mean plus the standard deviation. Ultimately, 1.8M pairs are
selected from a total of 12.8M pairs, as shown in Tab. 7. Training time has
been reduced approximately sevenfold, with a performance boost of about +0.6
mIoU, underscoring the efficacy of filtering out noisy samples.

Figure 8: Performance of multi-tag selection with varying thresholds.

###  0.A.2 Evaluation Details

For assessing multi-tag selection (Sec. 4.1.1), we utilized two distinct
methodologies in the main paper. Firstly, we employed external NLP models such
as the NLTK parser [26] and large language models (LLM) [7, 2]. In our
utilization of LLM, we adopted identical prompts as utilized in the prior work
[24]. Secondly, in our scoring-based selection, we set a threshold of 0.5
since the resulting scores ranged from 0 to 1. Our exploration shown in Fig. 8
involves adjusting the threshold from 0.2 to 0.8 and observing the
corresponding evaluation outcomes. As expected, increasing the threshold led
to a reduction in the number of selected tags, resulting in higher precision
but lower recall. While fluctuations were observed in F1 score and accuracy,
all metrics’ optimal values fell short compared to our scoring approach
($s_{\mathrm{pixel}}^{\mathrm{ours}}$ (Eq. 2)) as depicted in Tab. 2(b).
Notably, mAP, being insensitive to the threshold, consistently highlighted the
superior performance of our scoring method.

In the evaluation of text-level semantic segmentation (Sec. 4.1.2), we
maintained a threshold of $0.4$ for TCL-based experiments and $0.5$ for
MaskCLIP-based experiments, aligning with their original configurations where
regions with similarity values surpassing the threshold are identified as
foreground areas.

##  Appendix 0.B Additional Experiments

In all experiments presented in this section, we assess the performance of our
method (TTD) with the baseline TCL [4].

Table 8: Effect of tagging methods.

Method | Multi-tag selection | Segmentation
---|---|---
P | R | F1 | CaptionIoU | mIoU @VOC
Baseline [4] | - | - | - | 60.4 | 55.0
NLTK [26] | 59.8 | 83.7 | 69.8 | 61.8 | 56.5
Vicuna-33B [7] | 52.7 | 70.7 | 60.4 | 60.4 | 55.9
Qwen-72B [2] | 69.3 | 56.2 | 62.1 | 60.7 | 56.2
$s_{\mathrm{image}}$ (Eq. 1) | 92.5 | 28.6 | 43.7 | 56.3 | 52.5
$s_{\mathrm{pixel}}^{\mathrm{ours}}$ (Eq. 2) | 82.9 | 74.5 | 78.5 | 65.5 | 61.1

###  0.B.1 Effect of tagging methods

As an extension of the experiments outlined in Sec. 4.3.2, we further
investigate the impact of different tagging methods on the performance of the
baseline model (TCL) trained with our method. As depicted in Tab. 8, there
exists a proportional relationship between the effectiveness of tag selection
and the resulting CaptionIoU performance. Improved tag selection performance
leads to the generation of a more accurate ideal image-text similarity map,
effectively alleviating the single tag bias. Furthermore, enhancements in tag-
level segmentation performance signify an improved image-text alignment. This
observation underscores the critical role of tag selection methods in
enhancing the overall performance and alignment of models trained with our
approach.

Figure 9: Illustration of $s_{\mathrm{seg}}$. Table 9: Multi-tag selection
with different scoring methods.

Scoring | P | R | F1 | Acc | mAP
---|---|---|---|---|---
$s_{\mathrm{image}}$ (Eq. 1) | 92.5 | 28.6 | 43.7 | 79.5 | 83.2
$s_{\mathrm{text}}$ (Eq. 7) | 85.6 | 29.7 | 44.1 | 79.0 | 82.1
$s_{\mathrm{image}}+s_{\mathrm{text}}$ | 85.5 | 45.1 | 59.0 | 82.6 | 84.5
$s_{\mathrm{seg}}$ | 48.4 | 11.4 | 18.4 | 71.9 | 57.9
$s_{\mathrm{pixel}}^{\mathrm{ours}}$ (Eq. 2) | 82.9 | 74.5 | 78.5 | 88.6 | 90.3
\+ TTD (Ours) | 88.3 | 78.0 | 82.8 | 91.0 | 93.7

###  0.B.2 Multi-Tag Selection

Here, we delve into an additional scoring method not covered in the main
paper. We explore the performance of previously introduced scoring techniques
($s_{\mathrm{image}}$ (Eq. 1), $s_{\mathrm{text}}$ (Eq. 7)) across various
thresholds in Sec. 0.A.2.

As mentioned in Sec. 1, each tag can be scored based on the proportion of
image pixels similar to that tag ($s_{\mathrm{seg}}$). Specifically, as
depicted in Fig. 9, we generate a similarity map between the image and each
tag. Subsequently, for every pixel, we identify the most relevant tag with the
highest similarity value, enabling us to score each tag based on the
proportion of its relevant pixels. As illustrated in Tab. 8,
$s_{\mathrm{seg}}$ demonstrates relatively high precision since it accentuates
a single tag. However, due to the tendency of irrelevant tags to occupy more
pixels than relevant ones, it encounters challenges in score ranking,
resulting in a significantly low mAP (i.e., 57.9). In contrast, our scoring
method ($s_{\mathrm{pixel}}$) effectively mitigates the single tag bias while
maintaining correct ranking, resulting in high F1 score and mAP.

Table 10: Open-Vocabulary Multi-Label Classification on NUS-WIDE [8]. $*$
indicates the results based on our reproduction. Ann., use of a pre-defined
label set during training.

Method | Ann. | Backbone | mAP
---|---|---|---
ADDS [40] | O | ViT-B/32 | 36.56
MKT [15] | O | ViT-B/16 | 37.6
DualCoOp [36] | O | ResNet-50 | 43.6
TaI-DPT [14] | O | ResNet-50 | 44.99
CLIP [31] | X | ResNet-50 | 35.58
CLIP Surgery [22] | X | ResNet-50 | 39.99
CLIP Surgery∗ [22] | X | ViT-B/16 | 40.53
Ours (TCL [4]+TTD) | X | ViT-B/16 | 42.63

###  0.B.3 Open-Vocabulary Multi-label Classification

We contrast our method with existing CLIP-based multi-label classification
approaches in Tab. 10. Notably, all models except CLIP and CLIP Surgery [22]
rely on a predefined label set during training, requiring subsequent fine-
tuning on seen labels. To ensure fairness in comparison, we reproduce the
result of CLIP Surgery without using the label set during inference, aligning
with our approach’s methodology.111The original approach of CLIP Surgery
requires feature correction using a predefined label set, so we use feature
correction with an empty sentence instead. Leveraging the same backbone
(ViT-B/16), our method surpasses both CLIP and CLIP Surgery, achieving an mAP
of 42.63. This performance superiority highlights the effectiveness and
efficiency of our approach in multi-label classification tasks, without the
need for additional fine-tuning or label annotations. Our method not only
outperforms the existing techniques but also demonstrates its capacity to
address classification challenges with minimal data dependencies.

Table 11: Open-vocabulary semantic segmentation on referring datasets. We use
ViT-B/16 as the backbone for all methods.

Method | RefCOCO | RefCOCO+ | RefCOCOg | Average
---|---|---|---|---
val | testA | testB | val | testA | testB | val
GroupViT [38] | 7.99 | 6.16 | 10.51 | 8.49 | 6.79 | 10.59 | 10.68 | 8.74
MaskCLIP [31] | 11.52 | 11.85 | 12.06 | 11.87 | 12.01 | 12.57 | 12.74 | 12.09
TagAlign [24] | 18.75 | 20.31 | 20.64 | 19.24 | 20.88 | 21.23 | 23.69 | 20.68
Ours (TCL [4]+TTD) | 18.64 | 13.17 | 25.96 | 19.75 | 14.86 | 26.35 | 28.22 | 20.99

###  0.B.4 Open-Vocabulary Semantic Segmentation on Referring Datasets

We evaluate our model, which is fine-tuned TCL [4] with our method (TTD), on
referring datasets [28, 42]. Unlike the text-level and tag-level
segmentations, referring segmentation referring segmentation involves
delineating entities described in free-form text. This task demands a deeper
understanding of the textual context to accurately segment the corresponding
entities in the image. As depicted in Tab. 11, our model surpasses existing
approaches, showcasing its proficiency in comprehending textual descriptions
and effectively translating them into precise image segmentations. This
demonstrates the robustness and versatility of our method across diverse
segmentation tasks, further validating its potential for real-world
applications where understanding complex textual descriptions is paramount.

Figure 10: Refinement process using HQ-SAM [18]. Based on the mask outputs of
our model, we generate corresponding bounding boxes for each mask. Then, we
generate refined masks using HQ-SAM with masks and bounding boxes.  Table 12:
Effect of additional refinement using HQ-SAM.

Baseline | TTD (Ours) | HQ-SAM [18] | mIoU @VOC
---|---|---|---
MaskCLIP [44] | ✗ | ✗ | 29.3
✓ | ✗ | 43.1
✓ | ✓ | 53.7
TCL [4] | ✗ | ✗ | 55.0
✓ | ✗ | 61.1
✓ | ✓ | 63.4

###  0.B.5 Additional Refinement with SAM

We further enhance the segmentation outputs of our model by refining them
using HQ-SAM [18] as shown in Fig. 10. This involves feeding the mask output
from our model and its corresponding bounding box into HQ-SAM to generate a
refined mask. As illustrated in Fig. 11, the refined masks exhibit sharper
object boundaries compared to those produced solely by our method. Given that
the MaskCLIP-based model initially performs poorer than the TCL-based model,
the refinement’s impact is more pronounced, where noisy masks are also
removed. Quantitative results in Tab. 12 further support this observation,
showing a significant increase of 10.6 mIoU for the MaskCLIP-based model and
2.3 for the TCL-based model. This refinement process with HQ-SAM contributes
to cleaner and more accurate segmentation results, enhancing the overall
performance of our approach.

Figure 11: Additional qualitative results on semantic segmentation on Pascal
VOC, when refined with HQ-SAM [18].

##  Appendix 0.C Generating Tag and Mask Annotations

For the CC3M validation dataset [34], we meticulously annotated tags (words)
reflecting the relationship between an image and its corresponding text.
Subsequently, we manually delineated bounding boxes and points corresponding
to each tag, as illustrated in Fig. 12. Using HQ-SAM [18], we generated masks
for each tag, with the union of all tag masks forming the mask illustrating
the relationship between the image and the text. Finally, we filtered out 680
masks of high quality.

Our benchmark stands as the first to include tag annotation for each text
along with its mask annotation. Moreover, to our knowledge, no other benchmark
addresses segmentation with text involving multiple objects, crucial for
assessing the model’s representation of image-text alignment.

Figure 12: Generation of Mask Annotations using HQ-SAM. Beginning with the
manually annotated tags—mother, child, reading, and book, as depicted in the
figure—we manually outline the bounding boxes and points corresponding to each
tag. Subsequently, by merging the masks generated from HQ-SAM, we obtain the
final mask representing the relationship between the image and the text.

##  Appendix 0.D Additional Qualitative Results

Figure 13: Additional qualitative results on multi-tag selection. Figure 14:
Additional qualitative results on text-level segmentation.

## References

  * [1] Akbik, A., Bergmann, T., Blythe, D., Rasul, K., Schweter, S., Vollgraf, R.: FLAIR: An easy-to-use framework for state-of-the-art NLP. In: NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). pp. 54–59 (2019)
  * [2] Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)
  * [3] Caesar, H., Uijlings, J., Ferrari, V.: Coco-stuff: Thing and stuff classes in context. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1209–1218 (2018)
  * [4] Cha, J., Mun, J., Roh, B.: Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11165–11174 (2023)
  * [5] Changpinyo, S., Sharma, P., Ding, N., Soricut, R.: Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3558–3568 (2021)
  * [6] Chen, J., Zhu, D., Qian, G., Ghanem, B., Yan, Z., Zhu, C., Xiao, F., Culatana, S.C., Elhoseiny, M.: Exploring open-vocabulary semantic segmentation from clip vision encoder distillation only. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 699–710 (2023)
  * [7] Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., et al.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) (2023)
  * [8] Chua, T.S., Tang, J., Hong, R., Li, H., Luo, Z., Zheng, Y.T.: Nus-wide: A real-world web image database from national university of singapore. In: Proc. of ACM Conf. on Image and Video Retrieval (CIVR’09). Santorini, Greece. (July 8-10, 2009)
  * [9] Conti, A., Fini, E., Mancini, M., Rota, P., Wang, Y., Ricci, E.: Vocabulary-free image classification. Advances in Neural Information Processing Systems 36 (2024)
  * [10] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3213–3223 (2016)
  * [11] Ding, J., Xue, N., Xia, G.S., Dai, D.: Decoupling zero-shot semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11583–11592 (2022)
  * [12] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. International journal of computer vision 88, 303–338 (2010)
  * [13] Guo, J., Wang, Q., Gao, Y., Jiang, X., Tang, X., Hu, Y., Zhang, B.: Mvp-seg: Multi-view prompt learning for open-vocabulary semantic segmentation. arXiv preprint arXiv:2304.06957 (2023)
  * [14] Guo, Z., Dong, B., Ji, Z., Bai, J., Guo, Y., Zuo, W.: Texts as images in prompt tuning for multi-label image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2808–2817 (2023)
  * [15] He, S., Guo, T., Dai, T., Qiao, R., Shu, X., Ren, B., Xia, S.T.: Open-vocabulary multi-label classification via multi-modal knowledge transfer. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 808–816 (2023)
  * [16] Honnibal, M., Montani, I.: spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing (2017), to appear
  * [17] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
  * [18] Ke, L., Ye, M., Danelljan, M., Tai, Y.W., Tang, C.K., Yu, F., et al.: Segment anything in high quality. Advances in Neural Information Processing Systems 36 (2024)
  * [19] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [20] Kobayashi, T.: Two-way multi-label loss. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7476–7485 (2023)
  * [21] Li, M., Wang, D., Liu, X., Zeng, Z., Lu, R., Chen, B., Zhou, M.: Patchct: Aligning patch set and label set with conditional transport for multi-label image classification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15348–15358 (2023)
  * [22] Li, Y., Wang, H., Duan, Y., Li, X.: Clip surgery for better explainability with enhancement in open-vocabulary tasks. arXiv preprint arXiv:2304.05653 (2023)
  * [23] Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., Marculescu, D.: Open-vocabulary semantic segmentation with mask-adapted clip. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7061–7070 (2023)
  * [24] Liu, Q., Zheng, K., Wei, W., Tong, Z., Liu, Y., Chen, W., Wang, Z., Shen, Y.: Tagalign: Improving vision-language alignment with multi-tag classification. arXiv preprint arXiv:2312.14149 (2023)
  * [25] Liu, Q., Wen, Y., Han, J., Xu, C., Xu, H., Liang, X.: Open-world semantic segmentation via contrasting and clustering vision-language embedding. In: European Conference on Computer Vision. pp. 275–292. Springer (2022)
  * [26] Loper, E., Bird, S.: Nltk: The natural language toolkit. arXiv preprint cs/0205028 (2002)
  * [27] Luo, H., Bao, J., Wu, Y., He, X., Li, T.: Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. In: International Conference on Machine Learning. pp. 23033–23044. PMLR (2023)
  * [28] Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A.L., Murphy, K.: Generation and comprehension of unambiguous object descriptions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 11–20 (2016)
  * [29] Mottaghi, R., Chen, X., Liu, X., Cho, N.G., Lee, S.W., Fidler, S., Urtasun, R., Yuille, A.: The role of context for object detection and semantic segmentation in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 891–898 (2014)
  * [30] Mukhoti, J., Lin, T.Y., Poursaeed, O., Wang, R., Shah, A., Torr, P.H., Lim, S.N.: Open vocabulary semantic segmentation with patch aligned contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19413–19423 (2023)
  * [31] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [32] Ranasinghe, K., McKinzie, B., Ravi, S., Yang, Y., Toshev, A., Shlens, J.: Perceptual grouping in contrastive vision-language models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5571–5584 (2023)
  * [33] Ren, P., Li, C., Xu, H., Zhu, Y., Wang, G., Liu, J., Chang, X., Liang, X.: Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency. arXiv preprint arXiv:2302.10307 (2023)
  * [34] Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2556–2565 (2018)
  * [35] Shin, G., Xie, W., Albanie, S.: Reco: Retrieve and co-segment for zero-shot transfer. Advances in Neural Information Processing Systems 35, 33754–33767 (2022)
  * [36] Sun, X., Hu, P., Saenko, K.: Dualcoop: Fast adaptation to multi-label recognition with limited annotations. Advances in Neural Information Processing Systems 35, 30569–30582 (2022)
  * [37] Xing, Y., Kang, J., Xiao, A., Nie, J., Shao, L., Lu, S.: Rewrite caption semantics: Bridging semantic gaps for language-supervised semantic segmentation. Advances in Neural Information Processing Systems 36 (2024)
  * [38] Xu, J., De Mello, S., Liu, S., Byeon, W., Breuel, T., Kautz, J., Wang, X.: Groupvit: Semantic segmentation emerges from text supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18134–18144 (2022)
  * [39] Xu, J., Hou, J., Zhang, Y., Feng, R., Wang, Y., Qiao, Y., Xie, W.: Learning open-vocabulary semantic segmentation models from natural language supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2935–2944 (2023)
  * [40] Xu, S., Li, Y., Hsiao, J., Ho, C., Qi, Z.: A dual modality approach for (zero-shot) multi-label classification. arXiv preprint arXiv:2208.09562 (2022)
  * [41] Yi, M., Cui, Q., Wu, H., Yang, C., Yoshie, O., Lu, H.: A simple framework for text-supervised semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7071–7080 (2023)
  * [42] Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. pp. 69–85. Springer (2016)
  * [43] Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., Torralba, A.: Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127, 302–321 (2019)
  * [44] Zhou, C., Loy, C.C., Dai, B.: Extract free dense labels from clip. In: European Conference on Computer Vision. pp. 696–712. Springer (2022)