11institutetext: NVIDIA 22institutetext: National Taiwan University

# SemPLeS: Semantic Prompt Learning for Weakly-Supervised Semantic
Segmentation

Ci-Siang Lin Chien-Yi Wang Yu-Chiang Frank Wang Min-Hung Chen Work done during
the internship at NVIDIA Research, Taiwan.Project lead.112211112211

###### Abstract

Weakly-Supervised Semantic Segmentation (WSSS) aims to train segmentation
models using training image data with only image-level supervision. Since
precise pixel-level annotations are not accessible, existing methods typically
focus on producing pseudo masks for training segmentation models by refining
CAM-like heatmaps. However, the produced heatmaps may only capture
discriminative image regions of target object categories or the associated co-
occurring backgrounds. To address the issues, we propose a Semantic Prompt
Learning for WSSS (SemPLeS) framework, which learns to effectively prompt the
CLIP space to enhance the semantic alignment between the segmented regions and
the target object categories. More specifically, we propose Contrastive Prompt
Learning and Class-associated Semantic Refinement to learn the prompts that
adequately describe and suppress the image backgrounds associated with each
target object category. In this way, our proposed framework is able to perform
better semantic matching between object regions and the associated text
labels, resulting in desired pseudo masks for training the segmentation model.
The proposed SemPLeS framework achieves SOTA performance on the standard WSSS
benchmarks, PASCAL VOC and MS COCO, and demonstrated interpretability with the
semantic visualization of our learned prompts. The codes will be released.

##  1 Introduction

Figure 1: An overview of this paper. (a) Previous CLIP-guide WSSS methods
require heuristic human knowledge involved with the additional manual prompt
engineering effort, while (b) our proposed SemPLeS framework automatically
learns class-associated prompts embedded with semantic knowledge discovered
from the CLIP representation space without any manual human effort.

Semantic segmentation aims to classify every pixel in images to identify
object categories and the associated regions, which can benefit various
applications in the real world [44, 39, 64]. While promising results have been
presented by fully-supervised approaches [31, 36, 5, 6, 65, 7, 66], collecting
pixel-level annotations could be time-consuming and expensive, and therefore
limits the scalability and practicality of fully-supervised methods. To
address this issue, Weakly-Supervised Semantic Segmentation (WSSS) has emerged
as an alternative approach to train segmentation models with only coarse or
incomplete annotations such as bounding boxes [22], scribbles ([32]), points
[3], or image-level labels. Among these annotation forms, image-level labels
which indicate the presence or absence of certain object categories are
commonly used due to the efficiency in data collection and the availability in
various benchmark image datasets. Since precise annotations of object
positions are not observed, learning to localize and segment object categories
from image-level supervision is particularly challenging. Most existing
methods [4, 52, 46, 59] focus on producing pseudo ground truth masks by
refining CAM-like heatmaps [67, 48] with class labels as discriminative
supervision. Despite the shown efficacy, the produced pseudo masks may still
miss relevant regions of target object categories and fail to cover the entire
object. Furthermore, co-occurring backgrounds associated with certain object
categories may also be falsely activated (e.g., roads, or trees in an image of
a car). Consequently, learning precise image regions that align with the
semantics of target objects from weak supervision remains a challenging task.

With the rapid growth in the amount of visual and linguistic data in recent
years, several vision-language models [37, 10, 41] have been proposed to
bridge the underlying semantics between images and text descriptions. Given
that both the images and the associated text labels (category names) are
available in the setting of WSSS, the underlying image-text semantics from the
CLIP [41] space can be leveraged for pixel-level predictions. Recent
approaches focus on manually designing text prompts to refine the produced
pseudo masks, including background texts for suppressing co-occurring
background regions (e.g., {“a photo of river”, “a photo of lake”, “a photo of
water”} for the “boat” category [56]. Despite the effectiveness demonstrated,
these approaches require heuristic human knowledge involved with the
additional manual effort, as shown in Figure 1 (a). Moreover, manual prompting
may not fully exploit the semantic knowledge inside the CLIP space.

In this paper, we aim to fully exploit the CLIP space to extract semantic
information that can benefit the weakly-supervised semantic segmentation
problem without manual prompting. To achieve this goal, we propose a Semantic
Prompt Learning for WSSS (SemPLeS) framework to learn prompts embedded with
the semantic knowledge discovered from the CLIP space, as shown in Figure 1
(b), where the learned prompts can enhance the semantic alignment between the
segmented regions and the target object categories with image-level labels.
More specifically, we perform image-text contrastive learning under the
guidance of CLIP and train a mask generator to generate class activation maps.
However, such produced masks might not be sufficiently precise, and the co-
occurring backgrounds associated with the object categories may be falsely
activated. To alleviate this problem, we uniquely present Contrastive Prompt
Learning and Class-associated Semantic Refinement to suppress class-associated
background regions. In Contrastive Prompt Learning, we learn prompts to
capture co-occurring backgrounds from images and labels. Without manually
defining the background texts, our learned prompts would properly describe the
backgrounds associated with target object categories. Under the guidance of
our class-associated background prompts, we further suppress co-occurring
backgrounds from the activation maps via Class-associated Semantic Refinement.
With the above-introduced learning strategy, we will be able to enhance the
semantic matching between object regions and the associated text labels,
resulting in desired activation maps for segmentation purposes. The proposed
SemPLeS framework achieves SOTA performance on the standard WSSS benchmarks,
PASCAL VOC and MS COCO. Moreover, qualitative experiments demonstrate that our
learned prompts would show better interpretability than manual prompts.

In summary, our contributions are three-fold:

  * •

We propose a Semantic Prompt Learning for WSSS (SemPLeS) framework, which
performs weakly-supervised semantic segmentation via enforcing semantic
alignment between the object categories and the associated image regions.

  * •

In SemPLeS, we present Contrastive Prompt Learning to learn prompts that can
capture co-occurring background regions. With no need to manually define
background texts, our learned prompts would represent backgrounds associated
with the object categories.

  * •

Under the guidance of the derived prompts, our Class-associated Semantic
Refinement learns to suppress co-occurring backgrounds while enhancing the
semantic matching between object regions and the associated class labels,
resulting in precise pseudo masks for WSSS.

##  2 Related Works

###  2.1 Weakly-Supervised Semantic Segmentation (WSSS)

The existing WSSS approaches typically follow a three-stage learning process.
Firstly, the image-level labels are utilized as supervision to generate
initial Class Activation Maps (CAMs) [67, 48]. Secondly, the initial CAMs are
refined by using dense CRF [23] or pixel affinity-based methods [2, 1] to
obtain pseudo masks. Lastly, the refined pseudo masks are further used as
ground truth to train a segmentation network. Among all the stages, producing
precise CAMs is the main focus of WSSS, and various approaches have been
proposed to improve the quality of CAMs to benefit the learning of the
segmentation networks [17, 9, 11, 57, 15, 21, 28, 25, 63, 54, 29]. On the
other hand, there are end-to-end WSSS works without the need for multiple
training steps [47, 53], but their performances are worse than standard
3-stage methods. With the rapid development and the success of vision
transformers [14], recent approaches [46, 59, 45, 47, 12, 60, 40, 71] generate
finer activation maps based on the patch-level affinity learned from the
attention layers. In general, most WSSS methods take class labels as
discriminative supervision to generate CAMs without considering the textual
meaning of class names. Instead, our method exploits vision-language models to
learn class-associated semantic knowledge for segmentation.

###  2.2 CLIP for Semantic Segmentation

Recently, the Contrastive Language-Image Pretraining (CLIP) model [41] has
been adopted in semantic segmentation tasks thanks to the generalized
knowledge learned from a large corpus of image-text pairs. Given the
generalization capability, several zero-shot approaches [27, 38, 13, 42, 62,
18, 30, 61, 58] exploit CLIP to segment the classes which are unseen during
training. However, these methods still require mask annotations during
training, causing additional manual effort. CLIP has also been adopted to
improve unsupervised methods [68, 49, 19]. While minimizing the annotation
effort, the segmentation performance is still unsatisfactory since there is no
ground truth to guide the training. Recently, CLIP has also been utilized to
benefit WSSS by comprehending the meaning of class-related texts using the
language model [56, 34]. However, they mainly design text prompts with manual
effort. Instead, our proposed SemPLeS framework automatically learns the
class-associated prompts embedded with semantic knowledge, producing precise
pseudo masks for segmentation purposes.

###  2.3 Prompt Learning

In natural language processing (NLP), prompting [35] involves giving a text-
based input such as a sentence or phrase to obtain desired responses from
language models. Driven by the recent success of pre-trained vision-language
models (e.g., CLIP [41]), there has been an increasing interest to identify
proper prompts for computer vision tasks. Early work relies on prompt
engineering to identify text templates (e.g., “a photo of ”) describing
classes of interest to obtain underlying knowledge. However, such a trial and
error approach generally takes a large amount of time and effort and also
requires expertise about the task. To tackle the problem, prompt learning
methods [70, 69, 20] replace the manually-defined text prompts with a set of
learnable context vectors preceding the class names to automate the prompting
process. Different from the above methods, our Contrastive Prompt Learning
aims to capture class-associated semantic knowledge for segmentation purposes
rather than replacing the text template like “a photo of {}” for
classification purposes.

##  3 Method

Figure 2: An overview of the proposed SemPLeS framework. We first introduce
Segment-Label Matching as a warm-up stage, which leverages image-text
contrastive learning to produce initial object masks $M$ from the mask
generator $S$ (Sec. 3.2.1). We then propose Contrastive Prompt Learning (Sec.
3.2.2) and Class-associated Semantic Refinement (Sec. 3.2.3) to automatically
learn prompts $p_{k}$ embedded with semantic knowledge, which can help produce
refined object masks $M^{\prime}$.

###  3.1 Problem Formulation and Model Overview

We first define the problem setting and notations used in this paper. In
weakly-supervised semantic segmentation (WSSS), we assume that there is a set
of $N$ images $X$ with the associated image-level labels $y$, where
$X\in\mathbb{R}^{H\times W\times 3}$ and $y\in\\{0,1\\}^{K}$ is a multi-hot
vector indicating the presence or absence of $K$ object categories. Without
access to pixel-wise annotations, our goal is to learn precise class
activation maps for segmentation purposes. To perform segmentation from image
labels, we propose a novel Semantic Prompt Learning for WSSS (SemPLeS)
framework to derive prompts from visual and textual representations of CLIP
[41], which would be exploited to enhance the semantic alignment between the
segmented regions and the target object categories.

As shown in Figure 2, our framework takes Segment-Label Matching as a warm-up
stage, leveraging image-text contrastive learning to produce initial object
masks $M$ from our mask generator $S$ (Sec. 3.2.1). To suppress falsely
activated backgrounds in such masks (e.g., $X^{f}_{k}$ in the red box), we
present Contrastive Prompt Learning (Sec. 3.2.2) and Class-associated Semantic
Refinement (Sec. 3.2.3) to tackle this problem. The former learns class-
associated prompts $p_{k}$ to capture co-occurring backgrounds from images and
labels, while the latter takes the derived prompts to disregard co-occurring
backgrounds from the object masks (e.g., $X^{f}_{k}{{}^{\prime}}$ in the green
box). By jointly enforcing vision-language matching and suppression
objectives, our proposed framework would enhance the semantic alignment
between object regions and the associated text labels, resulting in precise
activation maps for the objects of interest.

###  3.2 Semantic Prompt Learning for WSSS (SemPLeS)

####  3.2.1 Segment-Label Matching:

Given an input image $X$, our mask generator $S$ is designed to produce soft
foreground masks $M=S(X)$ for target object categories. Since pixel-wise
annotations are not available, we choose to leverage vision-language models to
guide the learning of our mask generators from image-level supervision. To be
more precise, we exploit the joint latent space for images and texts from CLIP
to match the object regions and the associated text labels. To achieve this,
an image-text triplet (i.e., foreground-background-text) would be formulated
to perform contrastive learning, as illustrated in Figure 2 (a). For the $k$th
ground truth category which presents in the input image $X$ (i.e., $y_{k}=1$),
we derive the foreground image $X^{f}_{k}=M_{k}\cdot X$ by applying the $k$th
predicted mask $M_{k}$ to the original image $X$. Similarly, we reverse the
predicted mask to obtain the background regions $X^{b}_{k}=(1-M_{k})\cdot X$.
As for the text input $t_{k}$, we adopt the common prompt template “a photo of
{}” filled with the $k$th class name to describe the category of interest.
With the triplet [$X^{f}_{k}$, $X^{b}_{k}$, $t_{k}$] serving as the input of
the image encoder $E_{I}$ and text encoder $E_{T}$ pre-trained by CLIP, we
perform image-text contrastive learning to maximize the cosine similarity
between $X^{f}_{k}$ and $t_{k}$ for the foreground, while the similarity of
$X^{b}_{k}$ and $t_{k}$ would be minimized to repel the background. Therefore,
our matching loss $L_{match}$ would be formulated as follows:

$\displaystyle
L_{match}=\mathbb{E}_{X}-\left[log(sim(v^{f}_{k},u^{f}_{k}))+\lambda_{b}\cdot
log(1-sim(v^{b}_{k},u^{f}_{k}))\right],$ (1) $\displaystyle\text{where}\quad
v^{f}_{k}=E_{I}(X^{f}_{k}),\quad
v^{b}_{k}=E_{I}(X^{b}_{k})\quad\text{and}\quad u^{f}_{k}=E_{T}(t_{k}).$

Here, $\lambda_{b}$ is the loss weight for repelling backgrounds and $sim$
refers to cosine similarity. Note that we keep the image encoder $E_{I}$ and
the text encoder $E_{T}$ frozen during training and preserve the latent space
learned from CLIP to avoid potential overfitting. With the above Segment-Label
Matching step, our mask generator $S$ is encouraged to segment object regions
that align with the associated text labels. However, as noted above, such
masks learned from image-level supervision are still coarse, and may falsely
include co-occurring backgrounds associated with certain object categories.
Therefore, the above image-text matching is not sufficient for segmentation.

####  3.2.2 Contrastive Prompt Learning:

To address the coarse mask issues, the previous language-guided approach [56]
exploits vision-language models to refine the masks with manual prompting
techniques. However, these methods require additional prompt engineering
efforts with human knowledge involved. Moreover, manual prompting may not be
able to fully exploit vision-language representation space. To tackle these
problems, we propose Contrastive Prompt Learning (Figure 2 (b)) to learn
prompts embedded with semantic knowledge from vision-language models,
facilitating the following object mask refinement. Different from the previous
work, we employ a sequence of learnable prompts $p_{k}$ as the input of the
text encoder $E_{T}$ to describe backgrounds for the $kth$ category.
Specifically, to align the prompts $p_{k}$ with the background image
$X^{b}_{k}$, we maximize the similarity of their representations in the latent
space by proposing $L^{I}_{prompt}$. On the other hand, to avoid describing
the foreground objects, we encourage the similarity between $u^{b}_{k}$ and
$u^{f}_{k}$ to be low with the proposed $L^{T}_{prompt}$. Thus, we propose
$L_{prompt}$ as below:

$\displaystyle L_{prompt}$ $\displaystyle=L^{I}_{prompt}+\lambda_{T}\cdot
L^{T}_{prompt}$ (2)
$\displaystyle=\mathbb{E}_{X}\left[-log(sim(u^{b}_{k},v^{b}_{k}))+\lambda_{T}\cdot
log(sim(u^{b}_{k},u^{f}_{k}))\right],$ $\displaystyle\text{where}\quad
u^{b}_{k}=E_{T}(p_{k}),\quad v^{b}_{k}=E_{I}(X^{b}_{k})\quad\text{and}\quad
u^{f}_{k}=E_{T}(t_{k}).$

Here, $\lambda_{T}$ is the loss weight for minimizing the similarities to the
object categories. Note that the mask generator $S$ is fixed and $p_{k}$ is
the only trainable part in this step. Once the above learning is complete, our
prompts $p_{k}$ can represent backgrounds for the $k$th categories. It is
worth noting that, our Contrastive Prompt Learning aims to capture class-
associated backgrounds for segmentation purposes rather than replacing the
text template like “a photo of {}” for classification purposes.

####  3.2.3 Class-associated Semantic Refinement:

Finally, to suppress co-occurring background regions from the object mask $M$,
our SemPLeS framework exploits the previously derived background prompts
$p_{k}$ to perform Class-associated Semantic Refinement (Figure 2 (c)). More
specifically, we encourage our mask generator $S$ to produce refined masks
$M^{\prime}$ by excluding the semantic knowledge embedded in the background
prompts $p_{k}$, while the objectives introduced in 3.2.1 are retained to
match the class labels. Hence, the refinement loss $L_{refine}$ and the total
loss function $L_{total}$ are defined as follows:

$\displaystyle L_{total}=L_{match}+\lambda\cdot
L_{refine},\quad\text{where}\quad
L_{refine}=\mathbb{E}_{X}\left[-log(1-sim(v^{f}_{k},u^{b}_{k}))\right].$ (3)

Here, $\lambda$ is the weight for the refinement loss. It can be seen that,
with the derived background prompts $p_{k}$ (fixed in this step) and the
introduced refinement loss $L_{refine}$, the class-associated background
regions would be suppressed from the foreground mask $M$, preventing possible
false activation. More importantly, by jointly applying the matching and
refinement objectives with image-level supervision, our Semantic Prompt
Learning for WSSS framework advances vision-language learning to enhance the
semantic alignment between the segmented regions and the target object
categories, resulting in compact and complete object masks $M^{\prime}$
desired for segmentation in a weakly-supervised fashion, where $M^{\prime}$
can be adopted in the WSSS pipeline to obtain the final segmentation results.

##  4 Experiments

###  4.1 Datasets, Implementation, and Evaluation Metrics

We train and validate our proposed framework on the standard benchmark
semantic segmentation datasets, PASCAL VOC 2012 [16] and MS COCO 2014 [33].
The PASCAL VOC 2012 dataset contains $20$ object categories along with a
background category. The original training, validation, and testing set
consists of $1464$, $1449$, and $1456$ images, respectively, with only image-
level labels for training. The testing set results are obtained from the
official evaluation website. As for the MS COCO 2014 dataset, the training and
validation set contains $82081$ and $40137$ images from $80$ object
categories, respectively. We adopt the recent SOTA Transformer-based backbone
model [71] for our mask generator $S$, and perform online retraining [71] to
obtain the final segmentation model. For CLIP [41], we use ViT-B/32 [14] as
the image encoder. The learnable prompts are randomly initialized with the
sequence length $K=30$, and the prompt embedding dimension $D=512$. Following
the common protocol in previous works [56], we train the proposed framework
with an augmented training set of $10582$ images (e.g., flipped, cropped,
etc.). The mean Intersection over Union (mIoU) is used as the evaluation
metric for all experiments.

###  4.2 Quantitative Comparisons

Table 1:  Quantitative comparisons of the initially generated CAMs (Seed) and
the pseudo masks on PASCAL VOC 2012 training set. The best and second results
are in bold and underline, respectively. †: reproduced from the public codes.

|  |
---|---|---
| Seed | Mask
$\text{CLIMS}_{\text{~{}~{}CVPR'22}}$ [56] | 57.5 | 72.8
$\text{MCTformer}_{\text{~{}~{}CVPR'22}}$ [59] | 61.7 | 69.1
$\text{Spatial-BCE}_{\text{~{}~{}ECCV'22}}$ [54] | 65.3 | 66.3
$\text{ViT-PCM}_{\text{~{}~{}ECCV'22}}$ [45] | 67.1 | 71.4
$\text{ESOL}_{\text{~{}~{}NeurIPS'22}}$ [29] | 53.6 | 68.7
$\text{ToCo}_{\text{~{}~{}CVPR'23}}$ [47] | - | 72.2
$\text{CLIP-ES}_{\text{~{}~{}CVPR'23}}$ [34] | 70.8 | 75.0
$\text{MMCST}_{\text{~{}~{}CVPR'23}}$ [60] | 66.3 | -
$\text{USAGE}_{\text{~{}~{}ICCV'23}}$ [40] | 67.7 | 72.8
$\text{WeakTr}_{\text{~{}~{}arXiv'23}}$ [71]† | 65.9 | 74.2
SemPLeS (Ours) | 68.7 | 78.3
|  |

To evaluate our proposed SemPLeS framework, we first compare the accuracy of
the initial activation maps and also the produced pseudo masks (Seed) with
previous works, as shown in Table 1. From the results in this table, our
method achieves the best performance compared with existing weakly-supervised
segmentation works. Specifically, our initial activation maps (Seed) achieve
$68.7\%$ while the pseudo masks (Mask) report $78.3\%$ in mIoU. This verifies
that, by advancing vision-language learning on the pre-trained CLIP model, our
proposed SemPLeS framework successfully generates pixel-wise predictions from
image-level labels, which would benefit the learning of the following
segmentation model.

Table 2: Quantitative comparisons for recent methods on PASCAL VOC 2012 val
and test sets. “Sup.” denotes supervision type. $\mathcal{F}$ denotes full
supervision. $\mathcal{I}$ denotes image-level supervision. “Seg. Model”
denotes the segmentation network. “DL1”, “DL2”, and “DL3” denote DeepLabV1,
V2, and V3+, respectively. “ResX” and “WResX” denote ResNet-X and
WideResNet-X, respectively. “Backbone” denotes the network for CAMs
generation.

Sup. | Method | Seg. Model | Backbone | val | test
---|---|---|---|---|---
$\mathcal{F}$ | Full supervision |  |  |  |
$\text{DeepLabV2}_{\text{~{}~{}TPAMI'18}}$ [5] | DL2-Res101 | - | 77.6 | 79.7
$\text{WResNet38}_{\text{~{}~{}PR'19}}$ [55] | DL1-WRes38 | - | 80.8 | 82.5
$\text{Segmenter}_{\text{~{}~{}ICCV'21}}$ [50] | Seg-DeiT-S | - | 79.7 | 79.6
$\mathcal{I}$ | Image-level supervision only
$\text{AMN}_{\text{~{}~{}CVPR'22}}$ [26] | DL2-Res101 | Res50 | 69.5 | 69.6
$\text{W-OoD}_{\text{~{}~{}CVPR'22}}$ [25] | DL2-Res101 | Res50 | 69.8 | 69.9
$\text{CLIMS}_{\text{~{}~{}CVPR'22}}$ [56] | DL2-Res101 | Res50 | 69.3 | 68.7
$\text{MCTformer}_{\text{~{}~{}CVPR'22}}$ [59] | DL1-WRes38 | DeiT-S | 71.9 | 71.6
$\text{Spatial-BCE}_{\text{~{}~{}ECCV'22}}$ [54] | DL1-Res101 | WRes38 | 68.5 | 69.7
$\text{ViT-PCM}_{\text{~{}~{}ECCV'22}}$ [45] | DL2-Res101 | ViT-B/16 | 70.3 | 70.9
$\text{ESOL}_{\text{~{}~{}NeurIPS'22}}$ [29] | DL2-Res101 | Res50 | 69.9 | 69.3
$\text{ToCo}_{\text{~{}~{}CVPR'23}}$ [47] | DeiT-B | DeiT-B | 69.8 | 70.5
$\text{CLIP-ES}_{\text{~{}~{}CVPR'23}}$ [34] | DL2-Res101 | ViT-B/16 | 71.1 | 71.4
$\text{OCR}_{\text{~{}~{}CVPR'23}}$ [12] | DL1-WRes38 | DeiT-S | 72.7 | 72.0
$\text{MMCST}_{\text{~{}~{}CVPR'23}}$ [60] | DL1-WRes38 | ViT-B | 72.2 | 72.2
$\text{BECO}_{\text{~{}~{}CVPR'23}}$ [43] | DL3-Res101 | Res101 | 72.1 | 71.8
$\text{ACR}_{\text{~{}~{}CVPR'23}}$ [24] | DL1-WRes38 | WRes38 | 71.9 | 71.9
$\text{USAGE}_{\text{~{}~{}ICCV'23}}$ [40] | DL1-WRes38 | DeiT-S | 71.9 | 72.8
$\text{FPR}_{\text{~{}~{}ICCV'23}}$ [8] | DL2-Res101 | Res50 | 70.3 | 70.1
$\text{Mat-Label}_{\text{~{}~{}ICCV'23}}$ [51] | DL2-Res101 | Res50 | 73.0 | 72.7
$\text{MCC}_{\text{~{}~{}WACV'24}}$ [53] | DeiT-B | DeiT-B | 70.3 | 71.2
SemPLeS (Ours) | Seg-DeiT-S | DeiT-S | 74.2 | 74.8

For the final segmentation results, our SemPLeS achieves the best performance
and reports $74.2\%$ and $74.8\%$ mIoU on the validation and testing set,
respectively, as shown in Table 2. Our method outperforms the previous SOTA,
Mat-Label [51], by $1.2\%$ and $2.1\%$ mIoU on the validation and testing set,
respectively. Moreover, our results are only $5.5\%$ and $4.8\%$ lower than
our upper bound, which is the fully-supervised method trained with densely-
annotated labels, on the validation and testing set, respectively. Similarly,
our SemPLeS performs favorably and achieves $44.9\%$ mIoU on MSCOCO, as shown
in Table 3. The above results verify that our method is effective in
performing segmentation in a weakly-supervised fashion.

###  4.3 Qualitative Comparisons

Table 3: Quantitative comparisons of recent methods on MS COCO 2014 validation
set.

Method | Val
---|---
$\text{MCTformer}_{\text{~{}~{}CVPR'22}}$ [59] | 42.0
$\text{SIPE}_{\text{~{}~{}CVPR'22}}$ [9] | 43.6
$\text{Spatial-BCE}_{\text{~{}~{}ECCV'22}}$ [54] | 35.2
$\text{ESOL}_{\text{~{}~{}NeurIPS'22}}$ [29] | 42.6
$\text{ToCo}_{\text{~{}~{}CVPR'23}}$ [47] | 41.3
$\text{OCR}_{\text{~{}~{}CVPR'23}}$ [12] | 42.5
$\text{USAGE}_{\text{~{}~{}ICCV'23}}$ [40] | 42.7
SemPLeS (Ours) | 44.9

In addition to quantitative results, we also provide qualitative comparisons
of initially generated CAMs as shown in Figure 3, and our method shows more
accurate activation maps compared with CLIMS [56] and WeakTr [71]. This
validates that, by advancing image-text contrastive learning with learnable
prompts, our SemPLeS would enhance the alignment between the segment regions
and the target object categories. In Figure 4, we also visualize the
corresponding regions of our learnable background prompts by calculating the
similarities to image patches with the text and image encoders of CLIP. We see
that the manual prompts defined in CLIMS [56] may falsely highlight the
foreground objects due to high co-occurrence when pre-training CLIP.
Furthermore, such manual prompts are limited to specific categories, and
therefore the corresponding backgrounds even do not present in the images
(e.g., the prompt “a photo of track" falsely indicates the background regions
of the first example). In contrast, our learnable prompts successfully
highlight all the background regions associated with each object category,
demonstrating the effectiveness of our Contrastive Prompt Learning. It is
worth noting that our class-associated prompts can also learn class-agnostic
knowledge, such as the grass and tree region in both the first (cow) and third
(dog) image in Figure 4.

Figure 3: Qualitative comparisons of the initial CAMs on PASCAL VOC 2012. Note
that our SemPLeS successfully suppresses the backgrounds and achieves better
results than WeakTr and CLIMS, which relies on manually-defined background
prompts with the heuristic human effort.  Figure 4: Visualization of manually-
defined/learned background prompts and the corresponding image regions. For
manual prompts (determined in CLIMS, we select those with the highest, lowest
similar, and the aggregated one (“All BG Prompts”) from the second to fourth
columns, respectively.

###  4.4 Ablation Studies

Table 4: Quantitative ablation studies of the loss functions on PASCAL VOC
2012.

$L_{match}$ | $L^{T}_{prompt}$ | $L^{I}_{prompt}$ | $L_{refine}$ | mIoU(%)
---|---|---|---|---
✓ |  |  |  | 67.6
✓ | ✓ |  | ✓ | 67.7
✓ |  | ✓ | ✓ | 67.9
✓ | ✓ | ✓ | ✓ | 68.7

####  4.4.1 Loss Component Analysis:

To analyze the importance of the introduced loss functions, we conduct both
quantitative and qualitative ablation studies, as shown in Table 4 and Figure
5. In Table 4, we see that applying only the matching loss $L_{match}$ would
result in $67.6\%$ mIoU. When learning our background prompts, considering
only the $L^{T}_{prompt}$ to repel the text labels may result in trivial
solutions with little improvement. On the other hand, if only $L^{I}_{prompt}$
is enforced to achieve alignment with the background images, the prompts are
still likely to learn the semantics of the foreground object categories
instead of the background, resulting in $67.9\%$ mIoU. Finally, when
$L^{I}_{prompt}$ and $L^{T}_{prompt}$ are jointly applied to learn the
background regions while avoid describing the foreground object categories,
the mIoU would improve to $68.7\%$. Together with the qualitative results in
Figure 5, we validate that our proposed framework SemPLeS and semantic prompt
learning would prevent false activation of co-occurring backgrounds and
therefore benefit segmentation in a weakly-supervised fashion.

Figure 5: Qualitative ablation studies of losses. With $L_{refine}$ applied
with our learned prompts, our SemPLeS successfully suppresses class-associated
backgrounds from class activation maps.  Table 5: Loss weight analysis.

$\lambda$ | 0 | 0.01 | 0.02 | 0.05 | 0.1
---|---|---|---|---|---
mIoU | 67.6 | 68.3 | 68.4 | 68.7 | 67.8

####  4.4.2 Loss Weight Analysis:

In Table 5, we perform hyper-parameter analysis on the refinement loss weight
$\lambda$ using PASCAL VOC 2012. $\lambda=0$ means only $L_{match}$ is used.
When varying $\lambda$ from $0$ to $0.1$, the mIoU gradually increases from
$67.6\%$ to $68.7\%$ when $\lambda=0.05$, and then drops to $67.8\%$ when
$\lambda=0.1$. Hence, we set $\lambda=0.05$ by default.

Table 6: Learning scheme variants analysis.

Scheme | # of rounds | mIoU
---|---|---
joint training | - | 65.9
seq./alter. training | 1 | 68.7
2 | 68.5
3 | 68.4

####  4.4.3 Learning Scheme Variants:

In Table 6, we provide quantitative comparisons when using different learning
schemes in our proposed SemPLeS framework using PASCAL VOC 2012. We see that,
jointly training all the learning steps (i.e., a, b, and c in Figure 2) in
Figure 2 would result in inferior performance of $65.9\%$. This is because
that, in the beginning of training, the prompts are not well learned and
therefore would not be desired for our Class-associated Semantic Refinement
(c). If we instead perform sequential training (i.e., $1$ round), the mIoU
would improve to $68.7\%$. Since alternate training between Contrastive Prompt
Learning (b) and Class-associated Semantic Refinement (c) results in similar
performance in our SemPLeS framework (and also increase the training time), we
choose to sequentially train the two steps once instead.

##  5 Conclusion

In this paper, we propose a Semantic Prompt Learning for WSSS (SemPLeS)
framework, which advances vision-language learning to achieve weakly-
supervised semantic segmentation (WSSS). In addition to exploiting the pre-
trained CLIP model to perform Segment-Label Matching, we further present
Contrastive Prompt Learning and Class-associated Semantic Refinement in the
proposed SemPLeS framework to prevent false activation of image backgrounds.
With no need to manually define background texts through prompt engineering,
our learned prompts properly capture and suppress co-occurring backgrounds for
each object category, resulting in precise activation maps for segmentation in
a weakly-supervised fashion. Quantitative experiments on the segmentation
benchmarks confirm the effectiveness of our proposed SemPLeS framework, and
visualization and ablation studies are conducted to demonstrate and verify the
interpretability of learned prompts.

## References

  * [1] Ahn, J., Cho, S., Kwak, S.: Weakly supervised learning of instance segmentation with inter-pixel relations. In: CVPR. pp. 2209–2218 (2019)
  * [2] Ahn, J., Kwak, S.: Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In: CVPR. pp. 4981–4990 (2018)
  * [3] Bearman, A., Russakovsky, O., Ferrari, V., Fei-Fei, L.: What’s the point: Semantic segmentation with point supervision. In: ECCV. pp. 549–565 (2016)
  * [4] Chang, Y.T., Wang, Q., Hung, W.C., Piramuthu, R., Tsai, Y.H., Yang, M.H.: Weakly-supervised semantic segmentation via sub-category exploration. In: CVPR. pp. 8991–9000 (2020)
  * [5] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(4), 834–848 (2017)
  * [6] Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)
  * [7] Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV. pp. 801–818 (2018)
  * [8] Chen, L., Lei, C., Li, R., Li, S., Zhang, Z., Zhang, L.: Fpr: False positive rectification for weakly supervised semantic segmentation. In: ICCV. pp. 1108–1118 (2023)
  * [9] Chen, Q., Yang, L., Lai, J.H., Xie, X.: Self-supervised image-specific prototype exploration for weakly supervised semantic segmentation. In: CVPR. pp. 4288–4298 (2022)
  * [10] Chen, Y.C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: Uniter: Universal image-text representation learning. In: ECCV. pp. 104–120 (2020)
  * [11] Chen, Z., Wang, T., Wu, X., Hua, X.S., Zhang, H., Sun, Q.: Class re-activation maps for weakly-supervised semantic segmentation. In: CVPR. pp. 969–978 (2022)
  * [12] Cheng, Z., Qiao, P., Li, K., Li, S., Wei, P., Ji, X., Yuan, L., Liu, C., Chen, J.: Out-of-candidate rectification for weakly supervised semantic segmentation. In: CVPR. pp. 23673–23684 (2023)
  * [13] Ding, J., Xue, N., Xia, G.S., Dai, D.: Decoupling zero-shot semantic segmentation. In: CVPR. pp. 11583–11592 (2022)
  * [14] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)
  * [15] Du, Y., Fu, Z., Liu, Q., Wang, Y.: Weakly supervised semantic segmentation by pixel-to-prototype contrast. In: CVPR. pp. 4320–4329 (2022)
  * [16] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. International Journal of Computer Vision 88, 303–338 (2010)
  * [17] Fan, J., Zhang, Z., Song, C., Tan, T.: Learning integral objects with intra-class discriminator for weakly-supervised semantic segmentation. In: CVPR. pp. 4283–4292 (2020)
  * [18] Ghiasi, G., Gu, X., Cui, Y., Lin, T.Y.: Scaling open-vocabulary image segmentation with image-level labels. In: ECCV. pp. 540–557 (2022)
  * [19] He, W., Jamonnak, S., Gou, L., Ren, L.: Clip-S4: Language-guided self-supervised semantic segmentation. In: CVPR. pp. 11207–11216 (2023)
  * [20] Jia, M., Tang, L., Chen, B.C., Cardie, C., Belongie, S., Hariharan, B., Lim, S.N.: Visual prompt tuning. In: ECCV. pp. 709–727 (2022)
  * [21] Jiang, P.T., Yang, Y., Hou, Q., Wei, Y.: L2g: A simple local-to-global knowledge transfer framework for weakly supervised semantic segmentation. In: CVPR. pp. 16886–16896 (2022)
  * [22] Khoreva, A., Benenson, R., Hosang, J., Hein, M., Schiele, B.: Simple does it: Weakly supervised instance and semantic segmentation. In: CVPR. pp. 876–885 (2017)
  * [23] Krähenbühl, P., Koltun, V.: Efficient inference in fully connected crfs with gaussian edge potentials. In: NeurIPS (2011)
  * [24] Kweon, H., Yoon, S.H., Yoon, K.J.: Weakly supervised semantic segmentation via adversarial learning of classifier and reconstructor. In: CVPR. pp. 11329–11339 (2023)
  * [25] Lee, J., Oh, S.J., Yun, S., Choe, J., Kim, E., Yoon, S.: Weakly supervised semantic segmentation using out-of-distribution data. In: CVPR. pp. 16897–16906 (2022)
  * [26] Lee, M., Kim, D., Shim, H.: Threshold matters in wsss: manipulating the activation for the robust and accurate segmentation model against thresholds. In: CVPR. pp. 4330–4339 (2022)
  * [27] Li, B., Weinberger, K.Q., Belongie, S., Koltun, V., Ranftl, R.: Language-driven semantic segmentation. In: ICLR (2022)
  * [28] Li, J., Fan, J., Zhang, Z.: Towards noiseless object contours for weakly supervised semantic segmentation. In: CVPR. pp. 16856–16865 (2022)
  * [29] Li, J., Jie, Z., Wang, X., Wei, X., Ma, L.: Expansion and shrinkage of localization for weakly-supervised semantic segmentation. In: NeurIPS (2022)
  * [30] Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., Marculescu, D.: Open-vocabulary semantic segmentation with mask-adapted clip. In: CVPR. pp. 7061–7070 (2023)
  * [31] Liang-Chieh, C., Papandreou, G., Kokkinos, I., Murphy, K., et al.: Semantic image segmentation with deep convolutional nets and fully connected crfs. In: ICLR (2015)
  * [32] Lin, D., Dai, J., Jia, J., He, K., Sun, J.: Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In: CVPR. pp. 3159–3167 (2016)
  * [33] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: ECCV. pp. 740–755 (2014)
  * [34] Lin, Y., Chen, M., Wang, W., Wu, B., Li, K., Lin, B., Liu, H., He, X.: Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation. In: CVPR. pp. 15305–15314 (2023)
  * [35] Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys 55(9), 1–35 (2023)
  * [36] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR. pp. 3431–3440 (2015)
  * [37] Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: NeurIPS (2019)
  * [38] Lüddecke, T., Ecker, A.: Image segmentation using text and image prompts. In: CVPR. pp. 7086–7096 (2022)
  * [39] Meyer, B.J., Drummond, T.: Improved semantic segmentation for robotic applications with hierarchical conditional random fields. In: ICRA. pp. 5258–5265. IEEE (2017)
  * [40] Peng, Z., Wang, G., Xie, L., Jiang, D., Shen, W., Tian, Q.: Usage: A unified seed area generation paradigm for weakly supervised semantic segmentation. In: ICCV. pp. 624–634 (2023)
  * [41] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763. PMLR (2021)
  * [42] Rao, Y., Zhao, W., Chen, G., Tang, Y., Zhu, Z., Huang, G., Zhou, J., Lu, J.: Denseclip: Language-guided dense prediction with context-aware prompting. In: CVPR. pp. 18082–18091 (2022)
  * [43] Rong, S., Tu, B., Wang, Z., Li, J.: Boundary-enhanced co-training for weakly supervised semantic segmentation. In: CVPR. pp. 19574–19584 (2023)
  * [44] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI. pp. 234–241. Springer (2015)
  * [45] Rossetti, S., Zappia, D., Sanzari, M., Schaerf, M., Pirri, F.: Max pooling with vision transformers reconciles class and shape in weakly supervised semantic segmentation. In: ECCV. pp. 446–463 (2022)
  * [46] Ru, L., Zhan, Y., Yu, B., Du, B.: Learning affinity from attention: end-to-end weakly-supervised semantic segmentation with transformers. In: CVPR. pp. 16846–16855 (2022)
  * [47] Ru, L., Zheng, H., Zhan, Y., Du, B.: Token contrast for weakly-supervised semantic segmentation. In: CVPR. pp. 3093–3102 (2023)
  * [48] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: ICCV. pp. 618–626 (2017)
  * [49] Shin, G., Xie, W., Albanie, S.: Reco: Retrieve and co-segment for zero-shot transfer. In: NeurIPS (2022)
  * [50] Strudel, R., Garcia, R., Laptev, I., Schmid, C.: Segmenter: Transformer for semantic segmentation. In: ICCV. pp. 7262–7272 (2021)
  * [51] Wang, C., Xu, R., Xu, S., Meng, W., Zhang, X.: Treating pseudo-labels generation as image matting for weakly supervised semantic segmentation. In: ICCV. pp. 755–765 (2023)
  * [52] Wang, Y., Zhang, J., Kan, M., Shan, S., Chen, X.: Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. In: CVPR. pp. 12275–12284 (2020)
  * [53] Wu, F., He, J., Cheng, L., Yin, Y., Hao, Y., Huang, G.: Masked collaborative contrast for weakly supervised semantic segmentation. In: WACV (2024)
  * [54] Wu, T., Gao, G., Huang, J., Wei, X., Wei, X., Liu, C.H.: Adaptive spatial-bce loss for weakly supervised semantic segmentation. In: ECCV. pp. 199–216 (2022)
  * [55] Wu, Z., Shen, C., Van Den Hengel, A.: Wider or deeper: Revisiting the resnet model for visual recognition. Pattern Recognition 90, 119–133 (2019)
  * [56] Xie, J., Hou, X., Ye, K., Shen, L.: Clims: cross language image matching for weakly supervised semantic segmentation. In: CVPR. pp. 4483–4492 (2022)
  * [57] Xie, J., Xiang, J., Chen, J., Hou, X., Zhao, X., Shen, L.: C2am: contrastive learning of class-agnostic activation map for weakly supervised object localization and semantic segmentation. In: CVPR. pp. 989–998 (2022)
  * [58] Xu, J., Liu, S., Vahdat, A., Byeon, W., Wang, X., De Mello, S.: Open-vocabulary panoptic segmentation with text-to-image diffusion models. In: CVPR. pp. 2955–2966 (2023)
  * [59] Xu, L., Ouyang, W., Bennamoun, M., Boussaid, F., Xu, D.: Multi-class token transformer for weakly supervised semantic segmentation. In: CVPR. pp. 4310–4319 (2022)
  * [60] Xu, L., Ouyang, W., Bennamoun, M., Boussaid, F., Xu, D.: Learning multi-modal class-specific tokens for weakly supervised dense object localization. In: CVPR. pp. 19596–19605 (2023)
  * [61] Xu, M., Zhang, Z., Wei, F., Hu, H., Bai, X.: Side adapter network for open-vocabulary semantic segmentation. In: CVPR. pp. 2945–2954 (2023)
  * [62] Xu, M., Zhang, Z., Wei, F., Lin, Y., Cao, Y., Hu, H., Bai, X.: A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In: ECCV. pp. 736–753 (2022)
  * [63] Yoon, S.H., Kweon, H., Cho, J., Kim, S., Yoon, K.J.: Adversarial erasing framework via triplet with gated pyramid pooling layer for weakly supervised semantic segmentation. In: ECCV. pp. 326–344 (2022)
  * [64] Zendel, O., Schörghuber, M., Rainer, B., Murschitz, M., Beleznai, C.: Unifying panoptic segmentation for autonomous driving. In: CVPR. pp. 21351–21360 (2022)
  * [65] Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: CVPR. pp. 2881–2890 (2017)
  * [66] Zhao, H., Zhang, Y., Liu, S., Shi, J., Loy, C.C., Lin, D., Jia, J.: Psanet: Point-wise spatial attention network for scene parsing. In: ECCV. pp. 267–283 (2018)
  * [67] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: CVPR. pp. 2921–2929 (2016)
  * [68] Zhou, C., Loy, C.C., Dai, B.: Extract free dense labels from clip. In: ECCV. pp. 696–712 (2022)
  * [69] Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Conditional prompt learning for vision-language models. In: CVPR. pp. 16816–16825 (2022)
  * [70] Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. International Journal of Computer Vision 130(9), 2337–2348 (2022)
  * [71] Zhu, L., Li, Y., Fang, J., Liu, Y., Xin, H., Liu, W., Wang, X.: Weaktr: Exploring plain vision transformer for weakly-supervised semantic segmentation. arXiv preprint arXiv:2304.01184 (2023)