# Diffusion Models for Zero-Shot Open-Vocabulary Segmentation

Laurynas Karazija, Iro Laina, Andrea Vedaldi, Christian Rupprecht
Visual Geometry Group
University of Oxford
<EMAIL_ADDRESS>
###### Abstract

The variety of objects in the real world is nearly unlimited and is thus
impossible to capture using models trained on a fixed set of categories. As a
result, in recent years, open-vocabulary methods have attracted the interest
of the community. This paper proposes a new method for zero-shot open-
vocabulary segmentation. Prior work largely relies on contrastive training
using image-text pairs, leveraging grouping mechanisms to learn image features
that are both aligned with language and well-localised. This however can
introduce ambiguity as the visual appearance of images with similar captions
often varies. Instead, we leverage the generative properties of large-scale
text-to-image diffusion models to sample a set of support images for a given
textual category. This provides a distribution of appearances for a given text
circumventing the ambiguity problem. We further propose a mechanism that
considers the contextual background of the sampled images to better localise
objects and segment the background directly. We show that our method can be
used to ground several existing pre-trained self-supervised feature extractors
in natural language and provide explainable predictions by mapping back to
regions in the support set. Our proposal is training-free, relying on pre-
trained components only, yet, shows strong performance on a range of open-
vocabulary segmentation benchmarks, obtaining a lead of more than 10% on the
Pascal VOC benchmark.111Project page
https://www.robots.ox.ac.uk/~vgg/research/ovdiff.

##  1 Introduction

Semantic segmentation seeks to classify each pixel in an image into a set of
categorical labels. Traditionally, the set of labels is fixed and predefined,
which allows collecting large training sets for learning models to predict
per-pixel class assignments from images. Still, collecting and annotating data
is costly and cumbersome. It results in static models that are difficult to
extend to new categories.

Open-world and open-vocabulary semantic segmentation relaxes this restriction
by allowing nearly arbitrary free-form text queries as class description
input. The latter problem is often approached by extracting from images dense
embeddings that can be matched to a representation of the text queries.
Obtaining these embedding is challenging as they have to describe the image
densely and they must also be compatible with the representation of any
possible text query. Many prior works addressed this challenge by starting
from multi-modal representations, (_e.g._ CLIP [44]) to bridge vision and
language, and further relied on labelled data to fine-tune it for
segmentation. Hence, these methods still require dense manual annotations of
new categories that are difficult to obtain.

An alternative which avoids collecting ad-hoc manual annotations is to
leverage image-text pairs that can be obtained at scale by crawling the
Internet. The current research in this line of work [67, 47, 68, 35, 41, 9]
proposes to extend vision-language models like CLIP with additional grouping
mechanisms, such as extra attention [41], slot-attention [68] or cosine
similarity [9], to better localise objects in the image descibed in the image-
level text. Afterwards, segmentation can be obtained by comparing
representations of textural categories with group- or pixel-level visual
embeddings. This however requires additional contrastive training at scale. As
text might not identify all regions or might mention an object that is not
present, the training is noisy. Similarly, a range of visual appearances might
have similar text descriptions or a similar concept might be described many
different ways, though image and language are processed independent of each
other during inference.

Furthermore, segmenting background requires additional consideration, as it
not identified by any of the target textual descriptions. The usual approach
is to threshold the similarities to all categories. Finding a threshold,
however, is difficult and might be image dependent, thus such approaches often
produce imprecise object boundaries. Handling background regions also remains
an open issue.

At the same time, the recent success of large-scale generative models trained
on similar large-scale image-text collections opens up new avenues for solving
these challenges. In this work, we take a different approach to the problem.
Our intuition is that multi-modal embeddings are difficult to learn and often
suffer from ambiguities and differences in detail between modalities. Thus, we
propose to use unimodal features for open-vocabulary segmentation. To bridge
the gap between free-form language queries and pre-trained feature extractors
we leverage the power of current language conditional image generators such as
StableDiffusion [48]. This has several advantages. Firstly, as text-
conditioned image generative models encode a distribution of possible images,
this offers a means to deal with intra-class variation and captures the
ambiguity in textual descriptions. Secondly, the generative image models
encode not only the visual appearance of objects but also provide contextual
priors such as backgrounds which can greatly improve the segmentation quality.

Given a textual prompt, our method, OVDiff, uses a generative model to produce
a support set of visual examples that we then decompose into a set of feature
prototypes at different levels of granularity: class, instance and part
prototypes. Prototypes are essentially image features extracted from off-the-
shelf unsupervised feature extractors. They can then be used in a simple
nearest-neighbour lookup scheme to segment any image. We also propose to
leverage the backgrounds from sampled images to encode a set of negative
prototypes that enable direct background segmentation.

As all components are leveraged as is, no further training is required
resulting in a surprisingly simple framework that achieves state-of-the-art
performance in several open-vocabulary segmentation benchmarks. Additionally,
our method has the potential to remain competitive beyond its current state,
as both generative models, as well as feature extractors are currently rapidly
improving. Finally, as segmentation is formed by considering nearest
prototypes, which are extracted from precise regions in the samples images,
this offers a degree of explainability.

##  2 Related work

#### Zero-shot open-vocabulary segmentation.

Open-vocabulary semantic segmentation is largely approached in two ways. First
[29, 18, 32, 66, 70, 6, 30, 19, 11] pose the problem as a zero-shot
generalisation task from training on a set of observed classes and dense
annotations to a set of unseen labels. Early approaches in zero-shot
segmentation [6, 30, 19, 11] explore generative networks to sample feature
maps using conditional language embeddings for classes. In [66, 30] image
encoders are trained to output dense features that can be correlated with
word2vec [39] and CLIP [44] text embedding forming masks. Follow-up works [18,
32, 14] approach the problem in two steps, predicting class-agnostic mask and
aligning the embeddings of predicted masks with language. IFSeg [70] generates
synthetic feature maps by pasting CLIP text embeddings into a known spatial
configuration to use as additional supervision. Different from our approach,
all these works rely on mask annotations for a set of “known” classes and
explore zero-shot generalisation.

The second approach to open-vocabulary segmentation aligns image regions with
language using only image-text pairs. Some methods introduce internal grouping
mechanisms such as hierarchical grouping [67, 47], slot-attention [68], or
cross-attention to learn cluster centroids [35]. Assignment to language
queries is performed at group level. An alternative line of work [75, 41, 9]
aims to learn dense features that are better localised when correlated with
language embeddings at pixel level. Other approaches include heuristic changes
to the attention mechanism [75], additional attention [41], or additional
consistency in language alignment [9]. These works learn a better joint
language and dense image feature space where objects can be localised well. In
contrast, we leverage a generative model to translate from language queries to
pre-trained image feature extractors without further training. A closely
related approach to ours is ReCO [54], where CLIP is leveraged for image
retrieval compiling a set of exemplar images from ImageNet for a given
language query, which is then used for co-segmentation. In our method, the
shortcoming of an image database is addressed by a large-scale generative
model. Furthermore, we do not perform co-segmentation but rather rely on the
attention of the generative model to extract objects. Instead of similarity of
support images, our method leverages diverse samples and makes use of both
foreground and contextual backgrounds.

#### Diffusion models.

Diffusion models [57, 24, 58] are a class of generative methods that have seen
tremendous success in text-to-image generative systems such as DALL-E [45],
Imagen [50], and Stable Diffusion [48], trained on Internet-scale data such as
LAION-5B [52]. The step-wise generative process and the language conditioning
also make pre-trained diffusion models attractive for discriminative tasks.
They have been explored in few-shot classification [73], few-shot [2] and
panoptic [69] segmentation tasks and to obtain masks of sampled images [31].
However such methods rely on dense manual annotations to associate diffusion
features with the desired output.

Annotation-free discriminative approaches such as [28, 13] use pre-trained
diffusion models as zero-shot classifiers. DiffuMask [65] exploits attention
maps between image features and conditioning text embeddings. Further prompt
retrieval and engineering techniques are used to synthesise a dataset of
“known” classes to train a segmenter capable of generalising to “unseen”
categories. DiffusionSeg [36] propose to leverage DDIM inversion [58] to
obtain feature maps and attention masks of object-centric images to perform
unsupervised object discovery, but relies on ImageNet labels and is not open-
vocabulary. While these works similarly use rich semantic information present
in diffusion models, we address a range of their limitations and do not
require any training.

#### Unsupervised segmentation.

Our work is also related to unsupervised segmentation approaches. While early
works relied on hand-crafted priors [12, 64, 72, 71, 42] later approaches
leverage feature extractors such as DINO [8] and perform further analysis of
these methods [63, 37, 55, 56, 20, 53, 61, 62]. Some approaches make use of
generative methods, usually GANs [4, 10, 3, 60, 38, 5], to produce images in
separate foreground-background layers [4, 10, 3] or analyse latent structure
to induce known foreground-background changes [60, 38] to synthesise a
training dataset with labels. Largely focused on foreground-background
segmentation and unsupervised saliency problems, these methods do not work
with language. However, we make use of one such component in our method. This
is enabled by the observation that unsupervised segmentation can be better
targeted using attention maps obtained from the diffusion process, and any
errors are mitigated as they are only used to sample features.

Figure 1: OVDiff overview. There are two stages: during prototype sampling,
text queries are used to sample a set of support images which are further
processed by a feature extractor and a segmenter forming positive and negative
or background prototypes. During the segmentation stage, image features are
extracted and compared against prototypes, where all background prototypes map
to the background class. The CLIP filter removes irrelevant prototypes based
on global image contents.

##  3 Method

We present OVDiff, a method for open-vocabulary segmentation, _i.e._ ,
semantic segmentation of any category described in natural language. To
achieve this goal we (1) leverage text-to-image generative models to generate
a set of images representative of the described category, and (2) use these to
ground an off-the-shelf pretrained feature extractor. This process is entirely
training-free: it relies only on pretrained components without additional
training data or parameter finetuning.

###  3.1 OVDiff: Diffusion-based open-vocabulary segmentation

Our goal is to devise an algorithm which, given a new vocabulary of categories
$c_{i}\in\mathcal{C}$ formulated as natural language queries, can segment any
image against it. Let $I\in\mathbb{R}^{H\times W\times 3}$ be an image to be
segmented. Let $\Phi_{v}:\mathbb{R}^{H\times W\times
3}\mapsto\mathbb{R}^{H^{\prime}W^{\prime}\times D}$ be an off-the-shelf visual
feature extractor and $\Phi_{t}:\mathbb{R}^{d_{t}}\mapsto\mathbb{R}^{D}$ a
text encoder. Assuming that image and text encoders are aligned, one can
achieve zero-shot segmentation by simply computing a similarity function, for
example, the cosine similarity $s(\Phi_{v}(I),\Phi_{t}(c_{i}))$, with
$s(x,y)=\frac{x\cdot y}{\|x\|\|y\|}$, between the encoded image $\Phi_{v}(I)$
and an encoding of a class label $c_{i}$, which is a simple extension of the
zero-shot classification paradigm to dense visual representations. To
meaningfully compare different modalities, image and text features must lie in
a shared representation space, which is typically learned by jointly training
$\Phi_{v}$ and $\Phi_{t}$ using image-text or image-label pairs [44]. In
addition, thresholding is required to determine the background.

We propose two modifications to this approach. First, we observe that it may
be easier to compare representations of the same modality than across
modalities. We thus replace $\Phi_{t}(c_{i})$ with a $D$-dimensional visual
representation $\bar{P}$ of class $c_{i}$, which we refer to as a prototype.
In this case, the same visual feature extractor can be used for both
prototypes and target images, thus their comparison becomes straightforward
and does not necessitate further training.

Second, we propose utilizing _multiple_ prototypes per category instead of a
single class embedding. This enables us to accommodate intra-class variations
in appearance and, as we explain later, it also allows us to exploit
contextual priors and segment the background. Finally, our approach handles
the queries $c_{i}$ independently, allowing for arbitrary changes to
$\mathcal{C}$ without the need for recomputation.

#### Support set generation.

To construct a set of prototypes in the visual domain, the first step of our
approach is to sample a support set of images representative of each category
$c_{i}$. This can be accomplished by leveraging pretrained text-conditional
generative models. Sampling images from a generative model, as opposed to a
curated dataset of real images, enables the construction of prototypes for any
user-specified category or description, regardless of whether a manually
labelled set is readily available for that category (_e.g._ , $c_{i}=$ “donut
with chocolate glaze”), which is well-aligned with the goals of open-
vocabulary segmentation.

Specifically, for each query $c_{i}$, we define a prompt “A good photo of a
$\langle c_{i}\rangle$” and generate a small batch of $N$ support images
$\mathcal{S}=\\{S_{1},S_{2},\dots,S_{N}\mid S_{n}\in\mathbb{R}^{hw\times
3}\\}$ of height $h$ and width $w$ using Stable Diffusion [48]. In its most
naïve form, a prototype $\bar{P}$ could then be constructed by averaging all
features across all images. However, this is unlikely to result in a good
prototype, because not all pixels in the generated image correspond to the
category in question. To address this issue, we propose to extract the class
prototypes as follows.

#### Class prototypes.

Our approach generates two sets of prototypes, positive and negative, for each
class. Positive prototypes are extracted from image regions that are
associated with $\langle c_{i}\rangle$, while negative prototypes correspond
to “background” regions. While considering negative or “background” prototypes
is not strictly necessary for segmentation, we found these help to
disambiguate objects from their surroundings by considering contextual priors,
which greatly improves performance. We thus aim to segment each image into
foreground (representing $c_{i}$) and background.

To identify regions most associated with $c_{i}$, we use the fact that the
layout of a generated image is largely dependent on the cross-attention maps
of the diffusion model [23, 59], _i.e._ , pixels attend more strongly to words
which describe them. One can then generate a set of attribution maps, denoted
as $\mathcal{A}=\\{A_{1},A_{2},\dots,A_{N}\mid A_{n}\in\mathbb{R}^{hw}\\}$,
corresponding to the support set $\mathcal{S}$, by summing the cross-attention
maps across all layers, heads, and denoising steps of the network [59].

Obtaining a foreground/background segmentation by thresholding activation maps
may not be optimal, as they are often coarse or incomplete, and sometimes only
parts of objects receive high activation. To address this, we propose using an
unsupervised instance segmentation method, such as CutLER [61], to generate
higher-quality proposals for objects. This approach may result in multiple
binary object masks. We denote these as $\mathcal{M}_{n}=\\{M_{nr}\mid
M_{nr}\in\\{0,1\\}^{hw}\\}$, where $n$ indexes the support images and $r$
indexes the object masks (including a mask for the background). For each
image, we then select the mask with the highest (lowest) average attribution
as the foreground (background):

$M^{\operatorname{fg}}_{n}=\operatorname*{arg\,max}_{M\in\mathcal{M}_{n}}\frac{M^{\top}A_{n}}{M^{\top}M},\quad
M^{\operatorname{bg}}_{n}=\operatorname*{arg\,min}_{M\in\mathcal{M}_{n}}\frac{M^{\top}A_{n}}{M^{\top}M}.$
(1)

We can then compute prototypes $P_{n}^{\operatorname{g}}$ for foreground and
background regions
($\operatorname{g}\in\\{\operatorname{fg},\operatorname{bg}\\}$) as

$P_{n}^{\operatorname{g}}=\frac{(\hat{M}^{\operatorname{g}}_{n})^{\top}\Phi_{v}(S_{n})}{m^{\operatorname{g}}_{n}}\,\in\mathbb{R}^{D},$
(2)

where $\hat{M}^{\operatorname{g}}_{n}$ denotes a resized version of
$M^{\operatorname{g}}_{n}$ that matches the spatial dimensions of
$\Phi_{v}(S_{n})$, and
$m^{\operatorname{g}}_{n}=(\hat{M}^{\operatorname{g}}_{n})^{\top}\hat{M}^{\operatorname{g}}_{n}$
counts the number of pixels within each mask. In other words, prototypes are
obtained by means of an off-the-shelf pretrained feature extractor and
computed as the average feature within each mask. We refer to these as
instance-level prototypes, because they are computed from each image
individually and each image in the support set can be viewed as an instance of
class $c_{i}$.

In addition to instance prototypes, we found it helpful to also compute class-
level prototypes $\bar{P}^{\operatorname{g}}$ by averaging the instance
prototypes weighted by their mask sizes as
$\bar{P}^{\operatorname{g}}=\sum_{n=1}^{N}{m^{\operatorname{g}}_{n}P_{n}^{\operatorname{g}}}/\sum_{n=1}^{N}{m^{\operatorname{g}}_{n}}$.

Finally, we propose to augment the set of class and instance prototypes using
$K$-Means clustering of the masked features to obtain part-level prototypes.
We perform clustering separately on foreground and background regions and take
each cluster centroid as a prototype $P^{\operatorname{g}}_{k}$ with $1\leq
k\leq K$. The intuition behind this is to enable segmentation at the level of
parts, support greater intra-class variability, and a wider range of feature
extractors that might not be scale invariant.

We consider the union of all these feature prototypes

$\mathcal{P}^{\operatorname{g}}=\bar{P}^{\operatorname{g}}\cup\\{P^{\operatorname{g}}_{n}\mid
1\leq n\leq N\\}\cup\\{P^{\operatorname{g}}_{k}\mid 1\leq k\leq
K\\},\,\operatorname{g}\in\\{\operatorname{fg},\operatorname{bg}\\}$ (3)

and associate all of them with a single category. We note that this process is
repeated for each $c_{i}\in\mathcal{C}$ and thus refer to
$\mathcal{P}^{\operatorname{fg}}$ (and $\mathcal{P}^{\operatorname{bg}}$) as
$\mathcal{P}^{\operatorname{fg}}_{c_{i}}$
($\mathcal{P}^{\operatorname{bg}}_{c_{i}}$), _i.e._ , as the foreground
(background) prototypes of class $c_{i}$. Since
$\mathcal{P}^{\operatorname{fg}}_{c_{i}}$
($\mathcal{P}^{\operatorname{bg}}_{c_{i}}$) depend only on class $c_{i}$, they
can be precomputed, and the set of classes can be dynamically expanded without
the need to adapt existing prototypes.

#### Open-vocabulary segmentation.

To perform segmentation of any target image $I$ given a vocabulary
$\mathcal{C}$, we first extract image features using the same visual encoder
$\Phi_{v}$ used for the prototypes. The vocabulary is expanded with an
additional background class
$\hat{\mathcal{C}}=\\{c_{\operatorname{bg}}\\}\cup\mathcal{C}$, for which the
foreground prototype is the union of all background prototypes in the
vocabulary: $\mathcal{P}^{\operatorname{fg}}_{c_{\operatorname{bg}}}$
$=\bigcup_{c_{i}\in\mathcal{C}}\mathcal{P}^{\operatorname{bg}}_{c_{i}}$. Then,
a segmentation map can simply be obtained by comparing image features for each
pixel to prototypes based on their cosine similarity. A class with the highest
similarity in its prototype set is chosen:

$M=\operatorname*{arg\,max}_{c\in\hat{\mathcal{C}}}\max_{P\in\mathcal{P}^{\operatorname{fg}}_{c}}s(\Phi_{v}(I),P).$
(4)

#### Category pre-filtering.

To limit the impact of spurious correlations that might exist in the feature
space of the visual encoder, we also apply a pre-filtering step for the set of
categories $\mathcal{C}$ given image $I$. Specifically, we leverage CLIP [44]
as a strong open-vocabulary classifier but propose to apply it in a multi-
label fashion to limit the candidate classes to
$\mathcal{C}^{\prime}\subseteq\mathcal{C}$ prior to segmenting. First, we
encode the target image and each category using CLIP. Any categories that do
not score higher than the $\nicefrac{{1}}{{|\mathcal{C}|}}$ are removed from
consideration, that is we keep $\\{P^{\operatorname{g}}_{c^{\prime}}\mid
c^{\prime}\in\mathcal{C}^{\prime}\\}$,
$\operatorname{g}\in\\{\operatorname{fg},\operatorname{bg}\\}$. If more than
$\eta$ categories are present, then the top-$\eta$ are selected. We then form
“multi-label” prompts as “$\langle c_{a}\rangle$ and $\langle c_{b}\rangle$
and ...” where the categories are selected among the top scoring ones by
considering all $2^{\eta}$ combinations. The best-scoring multi-label prompt
indentifies the final list of candidate classes.

#### “Stuff” filtering.

Occasionally, $c_{i}$ might not describe a countable object category but an
identifiable region in the image, _e.g._ , sky, often referred to as a “stuff”
class. “Stuff” classes warrant additional consideration as they might appear
as background in images of other categories, _e.g._ , boat image might often
contain regions of water and sky. As a result, the process outlined above
might sample background prototypes for one class that coincide with the
foreground prototypes of another. To overcome this issue, we introduce an
additional filtering step to detect and reject such prototypes, when the full
vocabulary, _i.e._ , the set of classes under consideration, is known. First,
we only consider foreground prototypes for “stuff” classes. Additionally, any
background prototypes of “thing” classes with high cosine similarity with any
of the “stuff” classes are simply removed. In our experiments, we use ChatGPT
[43] to automatically categorise a set of classes as “thing” or “stuff”. Note
that while the answers might contain errors, we find that this filtering step
is still beneficial to our approach.

##  4 Experiments

Figure 2: Qualitative results. OVDiff in comparison to TCL [9] (+ PAMR).
OVDiff provides more accurate segmentations across a range objects and stuff
classes with well defined object boundaries that separate from the background
well. Last 2 columns show failure cases. Additional table and light fixture
that appear in the background are segmented. Bed frame legs get misclassified
as chairs.

We evaluate OVDiff on the open-vocabulary semantic segmentation task. First,
we consider different feature extractors and investigate how they can be
grounded by leveraging our approach. We then turn to comparisons of our method
with prior work. We ablate the components of OVDiff, visualize the prototypes,
and conclude with a qualitative comparison with prior works on in-the-wild
images.

#### Datasets and implementation details.

As the approach is training-free, we only need to discuss data used for
evaluation, not training.

We evaluate on validation splits of PASCAL VOC (VOC), Pascal Context (Context)
and COCO-Object (Object) datasets. PASCAL VOC [17, 16] has 21 classes: 20
foreground plus a background class. For Pascal Context [40], we use the common
variant with 59 foreground classes and 1 background class. It contains both
“things” and “stuff” classes. The COCO-Object is a variant of COCO-Stuff [7]
with 80 “thing” classes and one class for the background.

We use the mean Intersection-over-Union (mIoU) to assess the segmentation
performance. Following [67, 9], the images are resized so that the shortest
side measures 448 pixels. Similarly to [9], we employ a sliding window
approach. We use two scales to aid with the limited resolution of off-the-
shelf feature extractors with square window sizes of 448 and 336, and a stride
of 224 pixels.

To generate our support set, we use the Stable Diffusion model v1.5. We set
the size of the support set to $N=64$, which translates to 4 batches of 16
images samples in parallel on a single A40222We estimate the total compute
used for the experiments in this paper at 190 GPU hours.. We detail further
specifications of the sampling and other hyper-parameters in the Appendix.

###  4.1 Grounding feature extractors

Our method can be used in combination with any pretrained visual feature
extractor for constructing prototypes and extracting image features. To verify
this quantitatively, we experiment with various self-supervised ViT feature
extractors(Table 3): DINO [8], MAE [21], and CLIP [44]. We also consider the
CLIP image encoder, using CLIP features. We find that taking keys of the
second to last layer in CLIP yields better results than using patch tokens
(CLIP token). We also experiment with Stable Diffusion [48] as a feature
extractor. We provide feature extraction details in the Appendix.

We find that SD performs the best, though CLIP and DINO also show strong
performance based on our experiments on VOC. MAE shows the weakest
performance, which may be attributed to its lack of semanticity [21]; yet it
is still competitive with the majority of purposefully trained networks when
employed as part of our approach. As feature extractors have different
training objectives, we hypothesise that their feature spaces might be
complementary, thus we also consider an ensemble approach. In this case, the
cosine distances fordmed between features of different extractors and
respective prototypes are simply averaged. The combination of SD, DINO, and
CLIP performs the best. We adopt this formulation for the main set of
experiments.

###  4.2 Comparison to existing methods

Table 1: Zero-shot open-vocabulary segmentation. Comparison of our approach to the state of the art (under the mIoU metric). Our results are an average of 5 seeds $\pm\sigma$. ∗results from [9].  Method | VOC | Context | Object
---|---|---|---
ReCo∗ [54] | 25.1 | 19.9 | 15.7
ViL-Seg [33] | 37.3 | 18.9 | -
MaskCLIP∗ [75] | 38.8 | 23.6 | 20.6
TCL [9] | 51.2 | 24.3 | 30.4
CLIPpy [46] | 52.2 | - | 32.0
GroupViT [67] | 52.3 | 22.4 | -
ViewCo [47] | 52.4 | 23.0 | 23.5
SegCLIP [35] | 52.6 | 24.7 | 26.5
OVSegmentor [68] | 53.8 | 20.4 | 25.1
OVDiff (Ours) | 67.1 $\pm$ 0.5 | 30.1 $\pm$ 0.2 | 34.8 $\pm$ 0.2
TCL [9] (+ PAMR) | 55.0 | 30.4 | 31.6
OVDiff (+ PAMR) | 69.0 $\pm$ 0.6 | 31.4 $\pm$ 0.2 | 36.3 $\pm$ 0.2

In Table 1, we compare our method with prior work on three datasets: VOC,
Context, Object. We find that our method compares favourably, outperforming
other methods in all settings. In particular, results on VOC show the largest
margin, with more than 10% improvement over prior work. We hypothesise that
this setting is particularly favourable to our method as it contains scenes
where classes take up larger areas of the image.

In the same table, we also combine our method with PAMR [1], the post-
processing approach employed by TCL [9]. We find that it improves results for
our method though improvements are less drastic since our method already
yields better segmentation and boundaries.

Qualitative results are shown in Fig. 2. This figure highlights a key benefit
of our approach: the ability to exploit contextual priors through the use of
background prototypes, which in turn allows for the directly assignment of
pixels to a background class. This improves segmentation quality because it
makes it easier to differentiate objects from the background and to delineate
their boundaries. In comparison, TCL predicts very coarse semantic masks and a
larger amount of noise.

Table 2: Segmentation performance of OVDiff based on different feature extractors.  Feature Extractor | VOC
---|---
MAE [21] | 53.8
DINO [8] | 59.6
CLIP [44] (token) | 61.0
CLIP [44] (keys) | 63.2
SD [48] | 63.6
SD + DINO + CLIP | 67.0

Table 3: Ablation of different components. Each component is removed in isolation, measuring the drop ($\Delta$) in mIoU on VOC and Context datasets.  Configuration | VOC | $\mathbf{\Delta}$ | Context | $\mathbf{\Delta}$
---|---|---|---|---
Full | 63.6 |  | 29.8 |
w/o bg prototypes | 53.6 | -10.0 | 28.3 | -1.5
w/o category filter | 54.9 | -8.7 | 26.4 | -2.4
w/o “stuff” filter | n/a | - | 27.3 | -2.5
w/o CutLER [61] | 60.6 | -3.0 | 27.9 | -1.9
w/o sliding window | 61.1 | -2.5 | 28.7 | -1.1
only average $\bar{P}$ | 62.5 | -1.1 | 29.0 | -0.8

###  4.3 Ablations

Next, we ablate the components of our method on VOC and Context datasets. In
this case, only SD is employed as a feature extractor. We remove individual
components of our method and measure the change in segmentation performance.
The results are summarised in Table 3.

Our first observation is that considering background prototypes does indeed
have a major impact on performance. When removing them from consideration, we
instead threshold the similarity scores of the images with the foreground
prototypes (set at 0.75, determined via grid search); in this case, the
performance drops significantly, which again highlights the importance of
leveraging contextual priors. On Context, the impact is less significant,
likely due to the fact that the dataset contains “stuff” categories. Removing
the instance- and part-level prototypes also negatively affects performance.
Additionally, removing the category pre-filtering has a major impact. We
hypothesize that this introduces spurious correlations between prototypes of
different classes. On Context, the “stuff” filtering is equally important.

Figure 3:  Analysis of the segmentation output by linking regions to samples
in the support set. Left: our results for the dog class. Middle: select color-
coded regions “activated” by different prototypes for the class. Right:
regions in the support set images corresponding to these (part-level)
prototypes.  Figure 4: PascalVOC results with increasing support size $N$.

Next, we evaluate the importance of using an unsupervised segmentation
technique (CutLER [61]) to obtain foreground/background prototypes. Instead of
a segmentation method, one can threshold the attribution maps obtained
directly through the diffusion process. However, we find that this reduces
performance on both datasets. Overall, the background prototypes and pre-
filtering contribute the most towards the overall performance.

Finally, we measure the effect of varying the size of the support set $N$ in
Fig. 4. We find that even at a low number of samples for each query, our
method already shows competitive performance. With increasing the number of
samples, the performance improves, saturating at around 128. This is likely
due to generating redundant prototypes when support sets are large.

###  4.4 Explaining segmentations

We can inspect how our method segments certain regions by considering which
prototype from $\mathcal{P}_{c}^{\operatorname{fg}}$ was used to assign a
class $c$ to a pixel. Prototypes have a mapping to regions in the support set
from where they were aggregated, e.g., instances prototypes are associated
with foreground masks $M^{\operatorname{fg}}_{n}$ and part prototypes with
centroids/clusters.

By following these mappings, a set of support image regions can be retrieved
for each segmentation decision providing a degree of explainability. Fig. 3
illustrates this mapping for examples of class dog. For visualisation
purposes, only select prototypes and corresponding regions are shown. On the
left, we show the full segmentation result of each image. In the middle, we
select regions that correlated best with certain prototypes of the class. On
the right, we retrieve images from the support set and highlight where each
prototype emerged. We find that meaningful segmentation of parts emerges as a
result of clustering the support image features. This also translates to a
meaningful link between regions of the inference image and those from the
support set, _e.g._ matching nose to noses, neck to necks, etc. This assembles
the overall segmentation of the dog while illustrating how each segmentation
produced by OVDiff is explained by precise regions in a small set of support
images.

###  4.5 In-the-wild

Figure 5:  Qualitative comparison on in-the-wild images with TCL, which
struggles with object boundaries, missing parts of objects, or including
surroundings. Our method has more appropriate boundaries but does produce
small halo effect around objects due to upscaling of feature extractors.

In Fig. 5, we investigate OVDiff on in-the-wild images containing simple and
complex backgrounds. We compare with TCL which uses PAMR. In the first three
images, both methods correctly detect the objects identified by the queries.
TCL however misses parts of the objects, such as most of the person, and parts
of animal bodies. The distinction between the house and the bridge in the
second image is also better with OVDiff. We note that our segmentations
sometimes have halos around objects. This is caused by the upscaling of the
low-resolution feature extractor (SD in this case). The last two images
contain difficult scenarios where both approaches struggle. The fourth image
only contains similar objects of the same type. Both methods incorrectly
identify plain donuts as either of the specified queries. OVDiff however
correctly identifies chocolate donuts with varied sprinkles and separates all
donuts from the background. In the final picture, the query “red car” is
added, although no such object is present. The extra query causes TCL to
incorrectly identify parts of the red bus as a car. Both methods incorrectly
segment the gray car in the distance. However, overall, our method is more
robust and delineates objects better despite the lack of specialized training
or post-processing.

###  4.6 Limitations

As OVDiff relies on pretrained components, it inherits some of their
limitations. OVDiff works with the limited resolution of feature extractors,
due to which it might miss tiny objects. While this can be partially mitigated
with a sliding window, employing high-resolution feature extractors is one
direction of future improvements. Furthermore, OVDiff cannot segment what the
generator cannot generate. For example, current diffusion models struggle with
producing legible text. Finally, one limitation comes from the computational
overhead of sampling support images. We observe that in practice whole image
collections are segmented often by the same set of queries, amortising this
cost.

##  5 Conclusion

We introduce OVDiff, a zero-shot open-vocabulary segmentation method that
operates in two stages. First, given queries, support images are sampled and
their features are extracted to create class prototypes. These prototypes are
then compared to features from an inference image. This approach offers
multiple advantages: diverse prototypes accommodating various visual
appearances, negative prototypes for background localization, and no need for
further training or fine-tuning. OVDiff outperforms previous methods on
benchmarks, exhibiting fewer errors, effectively separating objects from
background, and providing explainability through segmentation mapping to
support set regions.

## Acknowledgments and Disclosure of Funding

L. Karazija is supported by is supported by AIMS CDT EP/S024050/1. I. Laina,
A. Vedaldi, and C. Rupprecht are supported by ERC-CoG UNION 101001212 and
VisualAI EP/T028572/1.

## References

  * [1] Nikita Araslanov and Stefan Roth.  Single-stage semantic segmentation from image labels.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4253–4262, 2020.
  * [2] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko.  Label-efficient semantic segmentation with diffusion models.  In International Conference on Learning Representations, 2022.
  * [3] Yaniv Benny and Lior Wolf.  Onegan: Simultaneous unsupervised learning of conditional image generation, foreground segmentation, and fine-grained clustering.  In European Conference on Computer Vision, pages 514–530. Springer, 2020.
  * [4] Adam Bielski and Paolo Favaro.  Emergence of object segmentation in perturbed generative models.  Advances in Neural Information Processing Systems, 32, 2019.
  * [5] Adam Bielski and Paolo Favaro.  Move: Unsupervised movable object segmentation and detection.  In Advances in Neural Information Processing Systems, 2022.
  * [6] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez.  Zero-shot semantic segmentation.  Advances in Neural Information Processing Systems, 32, 2019.
  * [7] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In Computer vision and pattern recognition (CVPR), 2018 IEEE conference on. IEEE, 2018.
  * [8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
  * [9] Junbum Cha, Jonghwan Mun, and Byungseok Roh.  Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs.  arXiv preprint arXiv:2212.00785, 2022.
  * [10] Mickaël Chen, Thierry Artières, and Ludovic Denoyer.  Unsupervised object segmentation by redrawing.  Advances in neural information processing systems, 32, 2019.
  * [11] Jiaxin Cheng, Soumyaroop Nandi, Prem Natarajan, and Wael Abd-Almageed.  Sign: Spatial-information incorporated generative network for generalized zero-shot semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9556–9566, October 2021.
  * [12] Ming-Ming Cheng, Niloy J. Mitra, Xiaolei Huang, Philip H. S. Torr, and Shi-Min Hu.  Global contrast based salient region detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3):569–582, 2015.
  * [13] Kevin Clark and Priyank Jaini.  Text-to-image diffusion models are zero-shot classifiers.  arXiv preprint arXiv:2303.15233, 2023.
  * [14] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai.  Decoupling zero-shot semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11583–11592, 2022.
  * [15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [16] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results.  http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.
  * [17] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes (voc) challenge.  International Journal of Computer Vision, 88(2):303–338, June 2010\.
  * [18] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin.  Scaling open-vocabulary image segmentation with image-level labels.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVI, pages 540–557. Springer, 2022.
  * [19] Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang.  Context-aware feature generation for zero-shot semantic segmentation.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 1921–1929, 2020.
  * [20] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman.  Unsupervised semantic segmentation by distilling feature correspondences.  In International Conference on Learning Representations, 2022.
  * [21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [22] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [23] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [24] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [25] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications.
  * [26] Ronghang Hu, Shoubhik Debnath, Saining Xie, and Xinlei Chen.  Exploring long-sequence masked autoencoders.  arXiv preprint arXiv:2210.07224, 2022.
  * [27] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  2014\.
  * [28] Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak.  Your diffusion model is secretly a zero-shot classifier.  arXiv preprint arXiv:2303.16203, 2023.
  * [29] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl.  Language-driven semantic segmentation.  In International Conference on Learning Representations, 2021.
  * [30] Peike Li, Yunchao Wei, and Yi Yang.  Consistent structural relation learning for zero-shot segmentation.  Advances in Neural Information Processing Systems, 33:10317–10327, 2020.
  * [31] Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie.  Guiding text-to-image diffusion model towards grounded generation.  arXiv:2301.05221, 2023.
  * [32] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu.  Open-vocabulary semantic segmentation with mask-adapted clip.  arXiv preprint arXiv:2210.04150, 2022.
  * [33] Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang.  Open-world semantic segmentation via contrasting and clustering vision-language embedding.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XX, pages 275–292. Springer, 2022.
  * [34] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  arXiv preprint arXiv:2211.01095, 2022.
  * [35] Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li.  SegCLIP: Patch aggregation with learnable centers for open-vocabulary semantic segmentation.  arXiv preprint arXiv:2211.14813, 2022.
  * [36] Chaofan Ma, Yuhuan Yang, Chen Ju, Fei Zhang, Jinxiang Liu, Yu Wang, Ya Zhang, and Yanfeng Wang.  Diffusionseg: Adapting diffusion towards unsupervised object discovery.  arXiv preprint arXiv:2303.09813, 2023.
  * [37] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8364–8375, June 2022.
  * [38] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Finding an unsupervised image segmenter in each of your deep generative models.  In International Conference on Learning Representations, 2022.
  * [39] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  Advances in neural information processing systems, 26, 2013.
  * [40] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille.  The role of context for object detection and semantic segmentation in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 891–898, 2014.
  * [41] Jishnu Mukhoti, Tsung-Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim.  Open vocabulary semantic segmentation with patch aligned contrastive learning.  arXiv preprint arXiv:2212.04994, 2022.
  * [42] Tam Nguyen, Maximilian Dax, Chaithanya Kumar Mummadi, Nhung Ngo, Thi Hoai Phuong Nguyen, Zhongyu Lou, and Thomas Brox.  Deepusps: Deep robust unsupervised saliency prediction via self-supervision.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [43] OpenAI.  Introducing chatgpt.  https://openai.com/blog/chatgpt.
  * [44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [45] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [46] Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens.  Perceptual grouping in vision-language models.  arXiv preprint arXiv:2210.09996, 2022.
  * [47] Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang.  Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency.  arXiv preprint arXiv:2302.10307, 2023.
  * [48] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [49] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * [50] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * [51] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting.  Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22522–22531, June 2023.
  * [52] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  arXiv preprint arXiv:2210.08402, 2022.
  * [53] Gyungin Shin, Samuel Albanie, and Weidi Xie.  Unsupervised salient object detection with spectral cluster voting.  In CVPRW, 2022.
  * [54] Gyungin Shin, Weidi Xie, and Samuel Albanie.  Reco: Retrieve and co-segment for zero-shot transfer.  In Advances in Neural Information Processing Systems (NeurIPS), 2022\.
  * [55] Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce.  Localizing objects with self-supervised transformers and no labels.  November 2021.
  * [56] Oriane Siméoni, Chloé Sekkat, Gilles Puy, Antonin Vobecky, Éloi Zablocki, and Patrick Pérez.  Unsupervised object localization: Observing the background to discover objects.  arXiv preprint arXiv:2212.07834, 2022.
  * [57] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  * [58] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In International Conference on Learning Representations, 2021.
  * [59] Raphael Tang, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Jimmy Lin, and Ferhan Ture.  What the daam: Interpreting stable diffusion using cross attention.  arXiv preprint arXiv:2210.04885, 2022.
  * [60] Andrey Voynov, Stanislav Morozov, and Artem Babenko.  Object segmentation without labels with large-scale generative models.  In International Conference on Machine Learning, pages 10596–10606. PMLR, 2021.
  * [61] Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra.  Cut and learn for unsupervised object detection and instance segmentation.  arXiv preprint arXiv:2301.11320, 2023.
  * [62] Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez.  Freesolo: Learning to segment objects without annotations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14176–14186, 2022.
  * [63] Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufreydaz.  Self-supervised transformers for unsupervised object discovery using normalized cut.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14543–14553, June 2022.
  * [64] Yichen Wei, Fang Wen, Wangjiang Zhu, and Jian Sun.  Geodesic saliency using background priors.  In ECCV, 2012.
  * [65] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen.  Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models.  arXiv preprint arXiv:2303.11681, 2023.
  * [66] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata.  Semantic projection network for zero-and few-label semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8256–8265, 2019.
  * [67] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang.  Groupvit: Semantic segmentation emerges from text supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134–18144, 2022.
  * [68] Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie.  Learning open-vocabulary semantic segmentation models from natural language supervision.  arXiv preprint arXiv:2301.09121, 2023.
  * [69] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello.  Open-vocabulary panoptic segmentation with text-to-image diffusion models.  arXiv preprint arXiv:2303.04803, 2023.
  * [70] Sukmin Yun, Seong Hyeon Park, Paul Hongsuck Seo, and Jinwoo Shin.  Ifseg: Image-free semantic segmentation via vision-language model.  arXiv preprint arXiv:2303.14396, 2023.
  * [71] Yu Zeng, Yunzhi Zhuge, Huchuan Lu, Lihe Zhang, Mingyang Qian, and Yizhou Yu.  Multi-source weak supervision for saliency detection.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [72] Jing Zhang, T. Zhang, Yuchao Dai, Mehrtash Harandi, and Richard I. Hartley.  Deep unsupervised saliency detection: A multiple noisy labeling perspective.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9029–9038, 2018.
  * [73] Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Hongsheng Li, Yu Qiao, and Peng Gao.  Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners.  arXiv preprint arXiv:2303.02151, 2023.
  * [74] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017.
  * [75] Chong Zhou, Chen Change Loy, and Bo Dai.  Extract free dense labels from clip.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII, pages 696–712. Springer, 2022.

## Supplementary material

In this supplementary material, we consider the broader impacts of our work
(Appendix A), provide additional details concerning the implementation
(Appendix B), and conclude with additional results (Appendix C).

##  Appendix A Broader impact

Semantic segmentation is a component in a very large and diverse spectrum of
applications in healthcare, image processing, computer graphics, surveillance
and more. As for any foundational technology, applications can be good or bad.
OVDiff is similarly widely applicable. It also makes it easier to use semantic
segmentation in new applications by leveraging existing and new pre-trained
models. This is a bonus for inclusivity, affordability, and, potentially,
environmental impact (as it requires no training, which is usually
computationally intensive); however, these features also mean that it is
easier for bad actors to use the technology.

Because OVDiff is training-free, it is more versatile, but also, inherits the
weaknesses of the components it is built on. For example, it might contain the
biases (e.g., gender bias) of its components, in particular Stable Diffusion
[51], which is used for generating support images for any given
category/description. Thus it should not be exposed without further filtering
and detection of, e.g., NSFW material in the sampled support set. Finally,
OVDiff is also bound by the licenses of its components.

##  Appendix B OVDiff: Further details

In this section, we provide additional details concerning the implementation
of OVDiff. We begin with a brief overview of the attention mechanism and
diffusion models central to extracting features and sampling images. We review
different feature extractors used. We specify the hyperparameter setting for
all our experiments and provide an overview of the exchange with ChatGPT used
to categorise classes into “thing” and “stuff”.

###  B.1 Preliminaries

#### Attention.

In this work, we make use of pre-trained ViT [15] networks as feature
extractors, which repeatedly apply multi-headed attention layers. In an
attention layer, input sequences $X\in\mathbb{R}^{l_{x}\times d}$ and
$Y\in\mathbb{R}^{l_{y}\times d}$ are linearly project to forms keys, queries,
and values: $K=W_{k}Y,\;Q=W_{q}X,\;V=W_{v}X$. In self-attention, $X=Y$.
Attention is calculated as $A=\mathrm{softmax}(\frac{1}{\sqrt{d}}QK^{\top})$,
and softmax is applied along the sequence dimension $l_{y}$. The layer outputs
an update $Z=X+A\cdot V$. ViTs use multiple heads, replicating the above
process in parallel with different projection matrices $W_{k},W_{q},W_{v}$. In
this work, we consider queries and keys of attention layers as points where
useful features that form meaningful inner-products can be extracted. As we
detail later (Section B.2), we use the keys from attention layers of ViT
feature extractors (DINO/MAE/CLIP), concatenating multiple heads if present.

#### Text-to-image diffusion models.

Diffusion models are a class of generative models that form samples starting
with noise and gradually denoising it. We focus on latent diffusion models
[48] which operate in the latent space of an image VAE [27] forming powerful
conditional image generators. During training, an image is encoded into VAE
latent space forming a latent vector $z_{0}$. A noise is injected forming a
sample
$z_{\tau}\sim\mathcal{N}(z_{\tau};\sqrt{1-\alpha_{\tau}}z_{0},\alpha_{\tau}I)$
for timestep $\tau\in\\{1\dots T\\}$, where $\alpha_{\tau}$ are variance
values that define a noise schedule such that the resulting $z_{T}$ is
approximately unit normal. A conditional UNet [49],
$\epsilon_{\theta}(z_{t},t,c)$, is trained to predict the injected noise,
minimising the mean squared error
$\mathbb{E}_{t}\left(\alpha_{t}\|\epsilon_{\theta}(z_{t},t,c)-z_{0}\|_{2}\right)$
for some caption $c$ and additional constants $a_{t}$. The network forms new
samples by reversing the noise-injecting chain. Starting from
$\hat{z}_{T}\sim\mathcal{N}(\hat{z}_{T};0,I)$, one iterates
$\hat{z}_{t-1}=\frac{1}{\sqrt{1-\alpha_{t}}}(\hat{z}_{t}+\alpha_{t}\epsilon_{\theta}(\hat{z}_{t},t,c))+\sqrt{\alpha_{t}}\hat{z_{t}}$
until $\hat{z}_{0}$ is formed and decoded into image space using the VAE
decoder. The conditional UNet uses cross-attention layers between image
patches and language (CLIP) embeddings to condition on text $c$ and achieve
text-to-image generation.

###  B.2 Feature extractors

OVDiff is buildable on top of any pre-trained feature extractor. In our
experiments, we have considered several networks as feature extractors with
various self-supervised training regimes:

  * •

DINO [8] is a self-supervised method that trains networks by exploring
alignment between multiple views using an exponential moving average teacher
network. We use the ViT-B/8 model pre-trained on ImageNet333Model and code
available at https://github.com/facebookresearch/dino. and extract features
from the keys of the last attention layer.

  * •

MAE [22] is a self-supervised method that uses masked image inpainting as a
learning objective, where a portion of image patches are dropped and the
network seeks to reconstruct the full input. We use the ViT-L/16 model pre-
trained on ImageNet at a resolution of 448 [26].444Model and code from
https://github.com/facebookresearch/long_seq_mae. The keys of the last layer
are of the encoder network are used. No masking is performed.

  * •

CLIP [44] is trained using image-text pairs on an internal dataset WIT-400M.
We use ViT-B/16 model555Model and code from https://github.com/openai/CLIP..
We consider two locations to obtain dense features: keys from a self-attention
layer of the image encoder and tokens which are the outputs of transformer
layers. We find that keys of the second-to-last layer give better performance.

  * •

We also consider Stable Diffusion666We use implementation from
https://github.com/huggingface/diffusers. itself as a feature extractor. To
that end, we use the queries from the cross-attention layers in the UNet
denoiser, which correspond to the image modality. Its UNet is organised into 3
downsampling blocks, a middle block, and 3 upsampling blocks. We observe that
the middle layers have the most semantic content, so we consider the 1st and
2nd upsampling blocks and aggregate features from all three cross-attention
layers in each block. As the features are quite low in resolution, we include
the first downsampling cross-attention layer and the last upsampling cross-
attention layer as well. The feature maps are bilinearly upsampled to
resolution $64\times 64$ and concatenated. A noise appropriate for $\tau=200$
timestep is added to the input. For feature extraction, we run SD in
unconditional mode, supplying an empty string for text caption.

###  B.3 Hyperparameters

OVDiff has relatively few hyperparameters and we use the same set in all
experiments. Specifically, $N=64$ images are sampled using classifier-free
guidance scale [25] of 8.0 and 30 denoising steps. We employ DPM-Solver
scheduler [34]. If/when CutLER fails to extract any components in a sampled
image, a fallback of $M_{n}^{\operatorname{fb}}=A_{n}>0.5$ and
$M_{n}^{\operatorname{bg}}=A_{n}<0.2$ is used instead. During inference we set
$\eta=10$, which results in 1024 text prompts processed in parallel, a choice
made mainly to due computational constraints. We set the thresholds for the
“stuff” filter between background prototypes for “things” classes and the
foreground of “stuff” at 0.85 for all feature extractors. When sampling, a
seed is set for each category individually to ensure reproducibility. With our
unoptimized implementation, we measure around $210\pm 30$s to calculate
prototypes for a single category, or $102\pm 1$s without clustering.

###  B.4 Interaction with ChatGPT

Table 4: Response from interaction with ChatGPT. We used ChatGPT model to automatically categorise classes in “stuff” or “things”. airplane: | thing | window: | thing | awning: | thing
---|---|---|---|---|---
bag: | thing | wood: | stuff | streetlight: | thing
bed: | thing | windowpane: | thing | booth: | thing
bedclothes: | stuff | earth: | thing | television receiver: | thing
bench: | thing | painting: | thing | dirt track: | thing
bicycle: | thing | shelf: | thing | apparel: | thing
bird: | thing | house: | thing | pole: | thing
boat: | thing | sea: | thing | land: | thing
book: | thing | mirror: | thing | bannister: | thing
bottle: | thing | rug: | thing | escalator: | thing
building: | thing | field: | thing | ottoman: | thing
bus: | thing | armchair: | thing | buffet: | thing
cabinet: | thing | seat: | thing | poster: | thing
car: | thing | desk: | thing | stage: | thing
cat: | thing | wardrobe: | thing | van: | thing
ceiling: | stuff | lamp: | thing | ship: | thing
chair: | thing | bathtub: | thing | fountain: | thing
cloth: | stuff | railing: | thing | conveyer belt: | thing
computer: | thing | cushion: | thing | canopy: | thing
cow: | thing | base: | thing | washer: | thing
cup: | thing | box: | thing | plaything: | thing
curtain: | stuff | column: | thing | swimming pool: | thing
dog: | thing | signboard: | thing | stool: | thing
door: | thing | chest of drawers: | thing | barrel: | thing
fence: | stuff | counter: | thing | basket: | thing
floor: | stuff | sand: | thing | waterfall: | thing
flower: | thing | sink: | thing | tent: | thing
food: | thing | skyscraper: | thing | minibike: | thing
grass: | stuff | fireplace: | thing | cradle: | thing
ground: | stuff | refrigerator: | thing | oven: | thing
horse: | thing | grandstand: | thing | ball: | thing
keyboard: | thing | path: | thing | step: | stuff
light: | thing | stairs: | thing | tank: | thing
motorbike: | thing | runway: | thing | trade name: | stuff
mountain: | stuff | case: | thing | microwave: | thing
mouse: | thing | pool table: | thing | pot: | thing
person: | thing | pillow: | thing | animal: | thing
plate: | thing | screen door: | thing | lake: | stuff
platform: | stuff | stairway: | thing | dishwasher: | thing
plant: | thing | river: | thing | screen: | thing
road: | stuff | bridge: | thing | blanket: | stuff
rock: | stuff | bookcase: | thing | sculpture: | thing
sheep: | thing | blind: | thing | hood: | thing
shelves: | thing | coffee table: | thing | sconce: | thing
sidewalk: | stuff | toilet: | thing | vase: | thing
sign: | thing | hill: | thing | traffic light: | thing
sky: | stuff | countertop: | thing | tray: | stuff
snow: | stuff | stove: | thing | ashcan: | thing
sofa: | thing | palm: | thing | fan: | thing
table: | thing | kitchen island: | thing | pier: | thing
track: | stuff | swivel chair: | thing | crt screen: | thing
train: | thing | bar: | thing | bulletin board: | thing
tree: | thing | arcade machine: | thing | shower: | thing
truck: | thing | hovel: | thing | radiator: | thing
monitor: | thing | towel: | thing | glass: | stuff
wall: | stuff | tower: | thing | clock: | thing
water: | stuff | chandelier: | thing | flag: | thing

We interact with ChatGPT to categorise classes into “stuff” and “things”. Due
to input limits, the categories are processed in blocks. Specifically, we
input “In semantic segmentation, there are "stuff" or "thing" classes. Please
indicate whether the following class prompts should be considered "stuff" or
"things":”. We show the output in Table 4. Note there are several errors in
the response, _e.g._ glass, blanket, and trade name are actually instances of
tableware, bedding and signage, respectively, so should more appropriately be
treated as “things”. Similarly, land and sand might be more appropriately
handled as “stuff”, same as snow and ground.

##  Appendix C Additional experiments

In this section, we provide additional experimental results of OVDiff.

###  C.1 Additional ablations

Table 5: Ablation of various configurations for prototypes. We consider average $\bar{P}$, instance $P_{n}$, and part $P_{k}$ prototypes individually and in various combinations on VOC and Context datasets. Combination of all three types of prototypes shows strongest results across the two settings. $\mathbf{\bar{P}}$ | $\mathbf{P_{n}}$ | $\mathbf{P_{k}}$ | VOC | Context
---|---|---|---|---
$\checkmark$ | $\checkmark$ | $\checkmark$ | 63.6 | 29.8
$\checkmark$ |  | $\checkmark$ | 63.7 | 29.5
$\checkmark$ | $\checkmark$ |  | 63.2 | 29.8
| $\checkmark$ | $\checkmark$ | 62.6 | 28.9
|  | $\checkmark$ | 63.0 | 29.5
| $\checkmark$ |  | 62.6 | 29.6
$\checkmark$ |  |  | 62.5 | 29.0

Table 6: Ablation of different SD feature configurations. We consider whether including first and last cross attention layers, adding mid and 2nd upsampling blocks (all layers in the block) has a positive effect. Including the mid-block also has a slightly negative effect, likely due to low resolution. Removing higher-resolution layers/blocks also has negative effect. 1st | Mid | Up-1 | Up-2 | Last |
---|---|---|---|---|---
layer | block | block | block | layer | Context
$\checkmark$ |  | $\checkmark$ | $\checkmark$ | $\checkmark$ | 29.8
$\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 29.5
|  | $\checkmark$ | $\checkmark$ | $\checkmark$ | 29.5
$\checkmark$ |  | $\checkmark$ | $\checkmark$ |  | 29.2
$\checkmark$ |  | $\checkmark$ |  | $\checkmark$ | 29.0

#### Prototype combinations.

In Table 6, we consider the three different types of prototypes described in
Section 3.1 and test their performance individually and in various
combinations. We find that the “part” prototypes obtained by $K$-means
clustering show strong performance when considered individually, as well as in
combination with the average category prototype. The combination of all three
types shows the strongest, most balanced results across the two datasets,
which is what we adopt in our main set of experiments.

Table 7: Choice of $K$ for number of centroids. $\mathbf{K}$ | Context
---|---
8 | 29.7
16 | 29.4
32 | 29.8
64 | 29.2

####  $K$ \- number of clusters.

In Table 7, we investigate the sensitivity of the method to the choice of $K$
for the number of “part” prototypes extracted using $K$-means clustering.
Although our setting $K=32$ obtains slightly better results on Pascal Context,
other values result in comparable segmentation performance suggesting that
OVDiff is not sensitive to the choice of $K$ and a range of values are viable.

#### SD features.

When using Stable Diffusion as a feature extractor, we consider various
combinations of layers/blocks in the UNet architecture. We follow the
nomenclature used in the Stable Diffusion implementation where consecutive
layers of Unet are organised into blocks. There are 3 down-sampling blocks
with 2 cross-attention layers each, a mid-block with a single cross-attention,
and 3 up-sampling blocks with 3 cross-attention layers each. We report our
findings in Table 6. Including the first and last cross-attention layers in
the feature extraction process has a positive impact on segmentation
performance, which we attribute to the high feature resolution. We also
consider including features from the middle block of the network but observe a
small negative impact on performance. This is likely due to the limited
feature resolution of $8\times 8$. We also investigate whether the second
upsampling block (Up-2) is necessary. Without it, the performance drops the
most out of the configurations considered. Thus, we use a concatenation of
features from the first and second upsampling blocks and the first and last
layers in our main experiments.

###  C.2 Evaluation without background

Table 8: Comparison with methods when background is not considered. We compare OVDiff with prior work on VOC-20, Context-59 and ADE datasets in a setting that considers only the foreground pixels (decided by ground truth). Our method shows comparable performance to prior works despite only relying on pretrained feature extractors. ∗ result from [9].  Method | VOC-20 | Context-59 | ADE-150
---|---|---|---
GroupViT∗ | 79.7 | 23.4 | 9.2
MaskCLIP∗ | 74.9 | 26.4 | 9.8
ReCo∗ | 57.5 | 22.3 | 11.2
PACL | 72.3 | 50.1 | 31.4
TCL | 77.5 | 30.3 | 14.9
OVDiff | 81.7 | 33.7 | 14.9

One of the notable advantages of our approach is the ability to represent
background regions via (negative) prototypes, leading to improved segmentation
performance. Nevertheless, we hereby also evaluate our method under a
different evaluation protocol, adopted in prior work, which excludes the
background class from the evaluation. We note that prior work often requires
additional considerations to handle background, such as thresholding. In this
setting, however, the background class is not predicted, and the set of
categories, thus, must be exhaustive. As in practice this is not the case, and
datasets contain unlabelled pixels (or simply a background label), such image
areas are removed from consideration. Consequently, less emphasis is placed on
object boundaries in this setting. We test our method on three datasets:
PascalVOC without background termed VOC-20, Pascal Context without background
termed Context-59, and ADE20k [74] which contains 150 foreground classes. As
in this setting the background prediction is invalid, we do not consider
negative prototypes. This setting tests the ability of various methods to
discriminate between different classes, which for OVDiff is inherent to the
choice of feature extractors. Despite this, our method shows competitive
performance. There exists a notable gap between PACL and other works,
including ours, on Context-59 and ADE-150. In the case of OVDiff, we attribute
this to the limited resolution of our feature extractors, especially on
ADE-150 where a variety of tiny objects is present. PACL, on the other hand,
proposes a method to increase the resolution of their trained network 4 times
during inference.

###  C.3 Qualitative results

Figure 6: Additional qualitative results. Images from Pascal VOC (top), Pascal
Context (middle), and COCO Object (bottom).

We include additional qualitative results from the benchmark datasets in Fig.
6. Our method achieves high-quality zero-shot segmentation across all
examples, without any post-processing or refinement steps. Finally, in Fig. 7,
we show examples of support images sampled for some thing, and stuff
categories.

(a) boat

(b) person

(c) sky

(d) water

(e) light

(f) parking meter

Figure 7: Images sampled for a support set of some categories.