# Plug-and-Play, Dense-Label-Free Extraction of Open-Vocabulary Semantic
Segmentation from Vision-Language Models

Luo Jiayun
Nanyang Technological University
<EMAIL_ADDRESS>Siddhesh Khandelwal
The University of British Columbia
<EMAIL_ADDRESS>Leonid Sigal
The University of British Columbia
<EMAIL_ADDRESS>Boyang Li
Nanyang Technological University
<EMAIL_ADDRESS>
###### Abstract

From an enormous amount of image-text pairs, large-scale vision-language
models (VLMs) learn to implicitly associate image regions with words, which is
vital for tasks such as image captioning and visual question answering.
However, leveraging such pre-trained models for open-vocabulary semantic
segmentation remains a challenge. In this paper, we propose a simple, yet
extremely effective, training-free technique, Plug-and-Play Open-Vocabulary
Semantic Segmentation (PnP-OVSS) for this task. PnP-OVSS leverages a VLM with
direct text-to-image cross-attention and an image-text matching loss to
produce semantic segmentation. However, cross-attention alone tends to over-
segment, whereas cross-attention plus GradCAM tend to under-segment. To
alleviate this issue, we introduce Salience Dropout; by iteratively dropping
patches that the model is most attentive to, we are able to better resolve the
entire extent of the segmentation mask. Compared to existing techniques, the
proposed method does not require any neural network training and performs
hyperparameter tuning without the need for any segmentation annotations, even
for a validation set. PnP-OVSS demonstrates substantial improvements over a
comparable baseline (+29.4% on Pascal VOC, +13.2% on Pascal Context, +14.0%
mIoU on MS COCO, +2.4% on COCO Stuff) and even outperforms most baselines that
conduct additional network training on top of pretrained VLMs.

Figure 1: Qualitative Results of PnP-OVSS + BLIP. Images are from Pascal
Context and COCO stuff. The right columns and bottom rows show the ground-
truth (GT); the rest are our results. Note accurate results even on complex
(trees) and small objects (last column).

##  1 Introduction

The classic task of semantic segmentation [19, 21] aims to classify pixels to
their object types. Traditional supervised methods require dense pixel-level
annotations and are restricted to recognizing a predefined set of objects. To
relax these constraints, open-vocabulary semantic segmentation [16, 77, 74,
40, 76, 43, 47, 45, 84, 20, 7, 80] aspires to identify arbitrary object
categories, whereas weakly supervised techniques [89, 35, 10, 30, 24, 8, 54,
75, 14, 52, 65] can acquire pixel-level localization capabilities from coarse
supervision, e.g., image labels or boxes.

Large-scale vision-language models (VLMs) pretrained on image-text pairs [60,
85, 39, 36, 37, 78, 1] achieve unprecedented performance on multimodal tasks,
such as describing arbitrary images and answering free-form, open-ended
questions about them (either with [60, 36, 37] or without finetuning [1, 62,
18, 70]). These tasks apparently involve some ability to localize objects. To
answer the question “what objects appear on the table?”, for example, the
model would have to first localize the table in the image and identify the
objects on it. Hence, it is reasonable to conjecture that the VLM network
learns to perform open-vocabulary localization from image-text pretraining.
However, distilling the localization capability from the VLMs remains an open
challenge.

Most existing methods for open-vocabulary semantic segmentation (OVSS) from
VLMs usually obtain single vector encodings for the visual and text inputs
respectively [74, 42, 50, 73, 43, 51]. However, pooling every token into a
single vector likely discards information about detailed positions of objects
and words. We investigate the use of pretrained cross-attention layers for
OVSS, which retain finer-grained correspondence between text and image
patches.

Nevertheless, a naive application of cross-attention between the class name
and the image patches lead to overly broad segmentation masks that include
irrelevant parts of the image (i.e., over-segmentation, see Fig. 2 (b)). To
alleviate this, [62] employs gradient information from the image-text matching
loss, GradCAM[56]-style, to sharpen the attention-based masks for the purpose
of guiding caption generation. However, this results in masks that capture
only the most discriminative regions of an object, such as the head of an
elephant (i.e., under-segmentation, Fig. 2 (c)). To acquire complete object
masks, we propose Salience DropOut, which iteratively drops the image patches
with high GradCAM attention scores, forcing the model to attend to less
discriminative but relevant object parts.

((a))

((b))

((c))

((d))

((a))

((b))

((c))

((d))

Figure 2: Segmentation masks for _elephant_ and _train_ using (a) off-the-
shelf cross-attention, (b) cross-attention + GradCAM, and (c) cross-attention
+ GradCam + Salience DropOut (§3.2). The naive cross-attention masks are too
inclusive whereas GradCAM is too exclusive.

Another important consideration is the cross-attention layer and the attention
head to extract the masks from. These hyperparameters have enormous influence
on the final results and are traditionally tuned on a validation set with
pixel-level mask annotations. To eliminate the need for dense annotations, we
propose an weakly-supervised reward function based on CLIP [48]. On a
validation set with images as well as object class names, the technique
contrasts the extracted object regions with a blank image. If, according to
CLIP, the former is more similar to the corresponding class name than the
latter, we increment the reward. All hyperparameter tuning of our technique is
performed with a simple random search with this reward, leading to high
performance.

In summary, we propose Plug-and-Play Open-vocabulary Semantic Segmentation
(PnP-OVSS), an extremely simple and training-free framework to extract
semantic segmentations from VLMs. At zero extra training cost, PnP-OVSS can be
used with any pretrained VLM with text-to-image cross attention layer and an
image-text matching loss. It has zero reliance on pixel-level annotations,
including a validation set for hyperparameter tuning. At the same time, PnP-
OVSS delivers excellent performance. It not only beats the training-free
baseline with remarkable margins (+29.4% mIoU on Pascal VOC, +13.2% on Pascal
Context, +14.0% on MS COCO, +2.4% on COCO Stuff), but also outperforms most
recent techniques within the past two years that require extensive finetuning
on top of the VLM pretraining.

With this paper, we make three contributions:

  * •

We propose to combine text-to-image attention, GradCAM, and Salience DropOut
to iteratively acquire accurate segmentation of arbitrary class(es) from
pretrained VLM.

  * •

We replace the densely annotated validation set for hyperparameter tuning,
which is needed by most existing methods, with a contrastive reward function
based on CLIP. This reward function, coupled with random search, finds a good
set of hyperparameters for OVSS.

  * •

The proposed method, PnP-OVSS, is simple to use, requires no extra finetuning,
and delivers high performance. Its success hints at a new direction for open-
vocabulary segmentation tasks leveraging large VLMs.

##  2 Related Work

###  2.1 Large-Scale Vision-Language Model

Large-scale vision-language models (VLMs), trained on millions of image-text
pairs, have become the foundation for many multimodal tasks. Architecturally,
the straightforward approach to training such methods involves aligning visual
and textual latent representations via a simple dot product [48, 23]. However,
this is insufficient for complex structured tasks like visual question
answering or image captioning, which require specialized approaches that
employ separate encoders before cross-attention between modalities [34, 36,
78, 32, 79] or self-attention networks over all tokens from both modalities
[26, 39, 60, 38, 9]. Another design dimension when training VLMs is the loss
function. Commonly used losses include image-text contrastive learning [49,
34, 36, 33, 81, 60], image-text matching (ITM) [34, 36, 78, 33, 81, 60, 26,
72, 82], prediction of masked tokens or patches [78, 33, 81, 60], and language
modeling [33, 1, 36].

This work utilizes models with unimodal encoders followed by cross-attention
fusion [36, 78, 81, 33, 34, 72, 82], as they work with high-level features,
and accurately attend to the appropriate image patches. Additionally, we
utilize the gradient from the ITM loss in the GradCAM step (§3.1) to sharpen
segmentation masks.

###  2.2 Zero-shot Semantic Segmentation

Method | PT Data Size | FT data size
---|---|---
Requires finetuning on image-text pairs
OVSegmentor [74] | - | 4.3M
Vil-Seg [42] | 400M | 412M
GroupVit* [73] | - | 3.4M
GroupVit [73] | - | 26M
CLIPpy [50] | - | 134M
SegClip [43] | 400M | 3.4M
ViewCo [51] | - | 26M
TCL [6] | 400M | 15M
PACL [45] | 400M | 30M
Requires finetuning but not image-text pairs
MaskClip w/ ST [88] | 400M | 1.2M
ZeroSeg* [7] | 400M | 3.4M
ZeroSeg [7] | 400M | 1.2M
Requires no finetuning
MaskClip[88] | 400M | 0
Reco[57] | 400M | 0
PnP-OVSS (Ours) |  |
\+ BLIP | 129M | 0
\+ BridgeTower | 400M+ 4M | 0
Table 1: Training data of current Zero-shot semantic segmentation methods with
only text supervision. PT stands for pretraining with image-caption data, FT
stands for fine-tuning with image-caption data. ST stands for self training.
We list only the image-caption data used for pretraining and all type of data
for finetuning. For hyperparameter tuning, our method uses CLIP-L/14
pretrained with 400M data to calculate the reward.

Zero-shot semantic segmentation aims to predict a dense segmentation mask for
any object class described by a given text prompt, with only prior exposure to
class-agnostic image-level supervision. This contrasts with weakly supervised
semantic segmentation [87, 3, 12, 53, 27, 61, 83, 25, 69, 22, 69, 29, 86],
which relies on class-specific annotations, and unsupervised object discovery
[58, 13, 67, 68, 64, 11, 55, 63], which identifies the sole object in the
foreground.

Traditional methods [71, 4, 17, 46] train a classifier to distinguish between
seen and unseen visual features, wherein the unseen visual features are
obtained from a generative model, trained on pairs of seen class image-text
embeddings. Recently, methods additionally leverage knowledge from VLMs to
attain better matching of visual and textual features [16, 76, 31, 88, 40].
Concretely, they train the segmentation network with dense annotations, while
replacing a part of the framework with components from VLMs.

Recently, methods have explored the use of additional supervision to further
reduce the need for pixel level annotations. Prior work is compared in Table
1.

Models Finetuned on Image-Text Pairs. Methods proposed in [73, 51, 43, 74, 42,
50, 45, 6] require paired image-text annotations to adapt to the zero-shot
segmentation task. Specifically, approaches in [73, 51, 43, 74, 42, 50]
cluster semantically similar pixels by contrasting grouped region embeddings
with textual embeddings, usually via the use of contrastive and self-
supervised losses. PACL [45] modifies the constrastive loss to operate over
aggregated patch embeddings (instead of image embeddings) to encourage better
alignment between image patches and text. TCL [6] achieves a similar patch-
text alignment by introducing an additional module to extract text-grounded
image regions.

Models Finetuned on Pseudo-labels. Methods in [7, 88] do not require
additional image-text supervision, but involve additional fine-tuning to adapt
to the segmentation task. ZeroSeg [7] attempts to match its own group
embeddings with multi-scaled segment embeddings obtained from CLIP [48].
MaskClip w/ ST [88] modifies the global attention pooling layer within CLIP to
output segmentation masks, which are used as pseudo-labels to train a
segmentation network.

In comparison, our proposed PnP-OVSS does not require _any fine-tuning or
additional paired image-text annotations_. It can directly distill high
quality open vocabulary semantic segmentations from any VLM with direct text-
to-image cross-attention and an image-text matching loss [36, 78, 81, 33, 34,
72, 82]. Compared to approaches in [57, 88] that perform zero-shot open
vocabulary semantic segmentation under a similar no training and no additional
annotations paradigm, PnP-OVSS achieves considerably superior performance.

Figure 3: The first iteration with cross-attention + GradCAM + Salience
DropOut. The text prompt contains $K$ class names and the image contains
$P\times P$ patches. From a cross-attention layer and an attention head in the
pretrained VLM, we obtain $K$ attention score maps of size $P\times P$, which
are sharpened by GradCAM using gradients from the image-text-matching (ITM)
loss. To get more complete predictions, we perform Salience Dropout, which
repeatedly zero out image patches of the highest average scores and feeds the
remaining patches to the image encoder again, forcing the model to attend to
other less discriminative patches. We show example salience maps from all
iterations in Fig. 4.

##  3 Method

Figure 4: An illustration of Salience DropOut, showing GradCAM salience values
after each iteration. Black squares in the images indicate dropped patches. We
obtain the final result by summing the salience maps from all iterations and
applying thresholding, Gaussian blur, and Dense CRF.

PnP-OVSS has four major steps. First, we extract a cross-attention salience
map per object class from a VLM. Second, we sharpen the salience map by
weighing it with the ITM gradient in the style of GradCAM. Third, we apply
Salience DropOut that iteratively completes the salience maps. Fourth, we
apply Dense CRF [28] for fine-grained adjustment. The process is illustrated
in Fig. 3 and Fig. 4.

In the first step, we feed an image and a text prompt to the pretrained VLM
and extract cross-attention maps. The text prompt is “A picture of [class 1]
[class 2] … [class K]”, which includes all $K$ class names of interest in the
dataset. The image is divided into $P\times P$ patches. The text prompt and
the image first go through the modality-specific encoders respectively,
followed by a cross-attention fusion module. In the cross-attention layers,
the text encodings serve as query vectors and the image patche encodings are
as key and value vectors.

We extract an attention map for each text token to the image patches from a
particular cross-attention layer and an attention head. Note that different
layers and heads lead to drastic performance differences, and the choices are
hyperparameters, tuned using the procedure in §3.4. We exclude attention maps
for the first three tokens “A”, “picture”, “of”, which do not describe
semantic classes. If a class name contains two or more tokens, we take the
mean attention map. This procedure yields an attention tensor of size $K\times
P\times P$.

With the correct layer and attention head, we observe that this attention map,
when normalized by softmax along the first dimension $K$, can provide passable
semantic segmentations. However, it tends to include many patches unrelated to
the class being segmented, leading to over-segmentation. Hence, we introduce
two refinement steps, GradCAM and Salience DropOut, explained in §3.1 and
§3.2. After these steps, we acquire aggregate salience maps for every class.
Finally, we conduct local polish to the resultant salience maps using Dense
CRF, as described in §3.3.

###  3.1 Map Sharpening with GradCAM

The off-the-shelf attention maps tend to also cover many patches unrelated to
the class name (See Fig. 2 (b)). Prior works [34, 62] further sharpen the
attention maps and focus them on class-discriminative regions using a variant
of GradCAM [56], which is originally proposed for convolutional networks, but
applicable to attention maps. It is worth noting that [34, 62] use the
technique for purposes other than semantic segmentation.

The GradCAM method requires a gradient. Here we leverage the image-text
matching (ITM) loss, which trains the VLM to classify if an image-text pair
match each other or not. We treat the ground-truth label as matching and
compute the gradient of the loss with respect to the attention score. This is
equivalent to asking: which attention scores contribute the most to the
decision that the image-text pair is matching? Formally, we denote a $P\times
P$ attention map for class $k$ as $M^{(k)}$ and the ITM loss as
$\mathcal{L}_{\text{ITM}}$. The GradCAM class salience map is computed as

$\widetilde{M}^{(k)}=\max\left(0,\frac{\partial\mathcal{L}_{\text{ITM}}}{\partial
M^{(k)}}\right)\otimes M^{(k)},$ (1)

where $\otimes$ denote the component-wise multiplication and $\max(\cdot)$ is
also applied component-wise.

###  3.2 Salience DropOut

As illustrated in Fig. 2 (c), segmentations generated by the GradCAM-style re-
weighting of cross-attention are often narrowly focused on the most
discriminative regions for a given class. However, the less discriminative
regions are still important for the completeness of masks. To compel the VLM
to attend to these regions, we propose an iterative technique called Salience
DropOut.

Since our task is zero-shot and open-vocabulary, we do not have prior
knowledge of the classes present in the image. Hence, we sum up the salience
maps $\widetilde{M}^{(k)}$ over all classes $k$, yielding a class-agnostic
salience map $U^{(t)}$ for the $t^{\text{th}}$ Salience DropOut iteration,

$\displaystyle U^{(t)}=\sum^{K}_{k=1}\widetilde{M}^{(k,t)},$ (2)

where $\widetilde{M}^{(k,t)}$ is the GradCAM salience map for class $k$ after
the $t^{\text{th}}$ iteration. Next, we zero out the 50% of image patches with
the highest value in $U^{(t)}$. We then compute the GradCAM salience maps on
the remaining image, whose sum is $U^{(t+1)}$. Any image patch previously
zeroed out will always receive zero salience in any subsequent iteration.

Formally, the set of remaining image patches $S^{(t)}\in\\{1,\dots,P\\}^{2}$
after the $t^{\text{th}}$ dropout iteration is defined as

$\displaystyle S^{(t)}$
$\displaystyle=S^{(t-1)}\setminus\\{(i,j)~{}|~{}U^{(t)}_{ij}\geq\eta\\},$ (3)
$\displaystyle\eta$
$\displaystyle=\texttt{median}\left(\\{U^{(t)}_{ij}~{}|~{}(i,j)\in
S^{(t-1)}\\}\right),$ (4)

where $U^{(t)}_{i,j}$ corresponds to the aggregate salience values for the
image patch at row $i$ and column $j$. Additionally, note that the input to
the first dropout iteration $S^{0}=\\{1,\dots,P\\}^{2}$ is the set of all
image patches.

We stop at four rounds of dropout, as almost all (93.75%) patches are removed
beyond that point. The final output for each class $k$ is the sum over all
salience maps across the four dropout iterations,
$\hat{M}^{(k)}=\sum_{t=1}^{4}\widetilde{M}^{(k,t)}$.

###  3.3 Gaussian Blur and Dense CRF

The salience dropout procedure generates $k$ continuous-valued salience maps
one per object class. To filter out small random noise in the salience values,
we subsequent apply a straightforward thresholding operation at a predefined
value $T$ on the salience values and obtain binary segmentation masks.
Nonetheless, the hard thresholding creates jagged segmentations with sharp
edges that often do not coincide with object boundaries. One common strategy
in zero-shot segmentation is to apply Dense Conditional Random Field (CRF)
[28], which makes fine-grained adjustments to the estimated masks by enforcing
consistency between nearby image pixels with similar colors.

However, we find that the hard 0/1 labels in the binary masks do not work well
as pixel unary potentials for Dense CRF. Hence, we smooth them using a
Gaussian kernel with a preset variance $\sigma$, which results in a better
initialization for unary terms and accounts for uncertainty of exact
segmentation boundary along the patch boundaries.

###  3.4 Hyperparameter Tuning

Three hyperparameters in PnP-OVSS have the strongest influence on the result,
the cross-attention layer $L$, the attention head $H$, and the binary
threshold $T$. Traditionally, tuning these hyperparameters requires a
validation set with pixel-level labels. However, since our goal is to perform
zero-shot open-vocabulary semantic segmentation, this requirement could
potentially limit the applicability of the technique. Instead, we propose a
weakly supervised reward function for hyperparameter tuning, which only
requires a set of images and the class names appearing in each image.

The reward $R$ is calculated as follows. We start with an image $I$ and a set
of classes present in the image, denoted as $\mathcal{K}(I)$. For each class
$k\in\mathcal{K}(I)$, we obtain a segmentation mask $M^{(k)}$ (which can be
the GradCAM mask, the Salience DropOut mask, or the Dense CRF mask). Next, we
apply the mask to the image $I$ and input the extracted regions
$M^{(k)}\otimes I$ into a pretrained neural network $f$, which takes an image
and a textual class name as input and produces a similarity score. We
calculate the normalized probability that the masked image $M^{(k)}\otimes I$
belongs to the ground-truth class $k$ and contrast it with a completely black
image $\bm{0}$.

$\displaystyle R(I)$
$\displaystyle=\sum_{k\in\mathcal{K}(I)}\mathds{1}[P(M^{(k)}\otimes
I,k)>P(\bm{0},k)],$ (5) $\displaystyle P(I,k)$
$\displaystyle=\frac{\exp(f(I,k))}{\sum_{k^{\prime}\in\mathcal{K}(I)}\exp(f(I,k^{\prime}))},$
(6)

where $\mathds{1}(\cdot)$ is the indicator function. Intuitively, a reward of
$1$ is assigned if and only if the image features pulled with the estimated
mask for class $k$ bears higher similarity to the class name of $k$ as
compared to a black image (which can be interpreted as the prior probability
of class $k$) in accordance with the pretrained network.

We sum up the reward for all validation images as the total reward. The best
hyperparameters, including the cross-attention layer, the attention head, the
threshold $T$, and the variance in the Gausian blur kernel, are determined
using a simple random search.

##  4 Experiments

###  4.1 Datasets and Implementation Details

Following the previous work for zero-shot semantic segmentation, we adopt
validation sets of Pascal-VOC 2012 [15], Pascal Context [44], COCO Object
[41], and COCO Stuff [5], that contain, respectively, 20 object classes, 59
object and stuff classes, 80 object classes and 171 object and stuff classes,
to evaluate our framework. To verify its versatility, we apply PnP-OVSS to two
high-performance VLMs, BLIP [36] and BridgeTower [78], which have the ITM loss
and text-to-image cross-attention. More details are in the supplementary
material.

Hyperparameters | Start | End | Step | Solution
---|---|---|---|---
BLIP |  |  |  |
Layer | 1 | 12 | 1 | 8
Head | 1 | 12 | 1 | 10
Attention Threshold | 0.05 | 0.5 | 0.1 | 0.15
BridgeTower |  |  |  |
Layer | 1 | 6 | 1 | 2
Head | 1 | 16 | 1 | 8
Attention Threshold | 0.05 | 0.5 | 0.1 | 0.15
Gaussian Blur |  |  |  |
Standard Deviation | 0.01 | 0.11 | 0.02 | 0.05
Table 2: Search space for hyperparameters Method |  Finetuning VLMs |  HT on Dense Labels |  Short-side Resolution |  Pascal VOC-20 |  Pascal Context-59 |  COCO Object-80 |  COCO Stuff-171
---|---|---|---|---|---|---|---
Group 1: Methods that require weakly supervised finetuning on image-text data
ViL-Seg $\dagger$ [42] | ✓ | ✓ | - | 37.3 | 18.9 | - | 18.0
CLIPpy [50] | ✓ | ✓ | 224 | 52.2 | - | 32.0 |  25.5 $\bigstar$
SegClip [43] | ✓ | ✓ | 224 | 52.6 | 24.7 | 26.5 | -
GroupVit* [43] | ✓ | ✓ | 224 | 28.1 | 14.8 | 12.9 | -
GroupVit [73] | ✓ | ✓ | 448 | 52.3 | 22.4 | 24.3 | -
ViewCo [51] | ✓ | ✓ | 448 | 52.4 | 23.0 | 23.5 | -
OVSegmentor [74] | ✓ | ✓ | 448 | 53.8 | 20.4 | 25.1 | -
TCL [6] +PAMR [2] | ✓ | ✓ | 448 | 55.0 | 30.4 | 31.6 | 22.4
PACL [45] | ✓ | ✓ |  224$\times$4 | 72.3 | 50.1 | 38.8 | 31.4
Group 2: Methods that require finetuning but not real image-text data
MaskClip w/ ST [88] | ✓ | ✓ | 336 | - | 31.1 | - | 18.0
ZeroSeg* [7] | ✓ | ✓ | 448 | 37.3 | 19.7 | 17.8 | -
ZeroSeg [7] | ✓ | ✓ | 448 | 40.8 | 20.4 | 20.2 | -
Group 3: Methods that require no finetuning
MaskClip [88] | $\times$ | $\times$ | 336 | - | 25.5 | - | 14.6
Reco [57] | $\times$ | $\times$ | 320 | - | 27.2 | - | 27.2
Reco* [6] | $\times$ | $\times$ | 448 | 25.1 | 19.9 | 15.7 | 14.8
PnP-OVSS with different VLMs
BLIP${}_{\text{Flickr}}$ | $\times$ | $\times$ | 224 | 47.8 | 36.4 | 24.8 | 25.8
BLIP${}_{\text{Flickr}}$ | $\times$ | $\times$ | 320 | 51.7 | 40.4 | 28.0 | 29.6
BLIP${}_{\text{Flickr}}$ | $\times$ | $\times$ | 336 | 52.5 | 40.7 | 28.2 | 29.6
BLIP${}_{\text{Flickr}}$ | $\times$ | $\times$ | 448 | 54.5 | 42.2 | 29.7 | 31.5
BLIP${}_{\text{Flickr}}$ | $\times$ | $\times$ | 768 | 54.1 | 42.8 | 31.8 | 32.5
BLIP${}_{\text{COCO}}$ | $\times$ | $\times$ | 768 | 55.7 | 41.9 | 33.8 | 32.6
BridgeTower | $\times$ | $\times$ | 320 | 36.4 | 32.3 | 24.2 | 27.6
BridgeTower | $\times$ | $\times$ | 336 | 35.3 | 32.4 | 24.2 | 27.6
BridgeTower | $\times$ | $\times$ | 770 | 35.2 | 32.4 | 24.5 | 28.0
Table 3: Zero-shot semantic segmentation performance in mIoU. Group 3 contains the most similar baselines that serve as fair comparisons to PnP-OVSS. Groups 1 and 2 benefit from additional training, extra image-text data, and hyperparameter tuning on dense labels. $\bigstar$ CLIPpy tests on 133 categories of COCO Stuff while we test all 171 classes of COCO Stuff. ViL-Seg $\dagger$ is tested on subset of classes on the three datasets, as detailed in the supplementary.  Method |  Dense Labels |  HT on Dense Labels |  Pascal VOC-20 |  Pascal Context-59 |  COCO Stuff-171
---|---|---|---|---|---
PnP-OVSS + BLIP${}_{\text{Flickr}}$ (Ours) | $\times$ | $\times$ | 53.6 | 53.8 | 39.8
SPNet+ST [71] | ✓ | ✓ | 25.8 | - | 26.9
ZS3Net+ST [4] | ✓ | ✓ | 21.2 | 20.7 | 10.6
CaGNet+ST [17] | ✓ | ✓ | 30.3 | - | 13.4
STRICT [46] | ✓ | ✓ | 35.6 | - | 30.3
LSeg [31] | ✓ | ✓ | 41.0 | - | -
SimBase [76] | ✓ | ✓ | 72.5 | - | 36.3
MaskCLIP+ w/ ST [88] | ✓ | ✓ | 86.1 | 66.7 | 54.7
Table 4: Comparison of zero-shot semantic segmentation performance on unseen
categories with methods trained with dense annotation.

Hyperparameter tuning. We use CLIP VIT-L/14 to calculate the reward. The input
resolution is 336$\times$336\. The search spaces of the random search and
results are shown in Tab. 2. For computation efficiency, we tune the layer,
the head, and the threshold with GradCAM masks. With the first three
hyperparameters fixed, we tune the Gaussian variance on masks before Dense
CRF. We directly adopt Dense CRF hyperparameters from CutLER [66] without
tuning.

Baselines. We adopt recent papers on zero-shot open-vocabulary semantic
segmentation as baselines. The only baselines strictly comparable to our work
are MaskClip [88] and Reco [57], which do not perform any finetuning on
pretrained VLMs and do not perform hyperparameter tuning on dense annotations.
We call these Group 3. To further expand our scope, we also include two other
groups of baselines. Group 1 finetunes VLMs but does not require image-text
pairs, including MaskClip with self-training (ST) [88] and ZeroSeg [7]. We
include ZeroSeg, trained with ImageNet1K, and ZeroSeg*, trained with
CC3M+COCO. Group 2 requires training on image-text pairs, including ViL-Seg
[42], SegClip [43], GroupVit* [73], GroupVit [73], ViewCo [51], CLIPpy [50],
OVSegmentor [74], TCL [6], PACL [45]. GroupVit* is a reimplementation of
GroupVit from [43].

For completeness, we also compare against supervised techniques since 2019,
including SPNet+ST [71], ZS3Net+ST [4], CaGNet+ST [17], STRICT [46], LSeg
[31], SimBase [76] , MaskCLIP+ w/ ST [88]. To maintain the zero-shot setting,
we test them on classes not observed during training. The details of class
split can be found in the supplementary material.

Ablated Model |  Pascal Context |  COCO Stuff
---|---|---
BLIP${}_{\text{Flickr}}$ | 19.8 | 14.5
\+ GradCam | 21.6 | 17.5
\+ Drop 1 | 25.1 | 19.8
\+ Drop 2 | 26.5 | 20.6
\+ Drop 3 | 27.0 | 20.9
\+ Drop 4 | 27.2 | 20.9
\+ Drop 4 + Blur | 36.8 | 28.6
\+ Drop 4 + Dense CRF | 35.3 | 31.8
\+ Drop 4 + Blur + Dense CRF | 42.8 | 32.5
Table 5: An ablation study of PnP-OVSS + BLIP${}_{\text{Flickr}}$ with
resolution 768 on Pascal Context and COCO Stuff.

###  4.2 Main Results

We show the main results in Tab. 3. As input resolution may influence results
and cause unfair comparisons, we label the resolution used by each method.
PACL [45] uses an 224 resolution but changes the stride for image
patchification from 16 to 4, hence introducing overlapping patches.

PnP-OVSS demonstrates excellent performance. Comparing with MaskClip [88] and
Reco [57], the two methods that require no additional training and ground
truth for hyperparameter tuning, on an equal-resolution basis, we attain 29.4%
mIoU improvement on Pascal VOC, 13.2% mIoU improvement on Pascal Context,
14.0% mIoU improvement on COCO Object, and 2.4% mIoU improvement on COCO
Stuff. Further, PnP-OVSS surpasses all baselines in Group 2. On an equal-
resolution basis, we achieve +13.7% mIoU on Pascal Voc, +9.6% on Pascal
Context, +9.5% on COCO Object and +11.6% on COCO Stuff.

When compared to Group 1, PnP-OVSS still outperforms most. On the Pascal
datasets, under equal resolutions, PnP-OVSS + BLIP${}_{\text{Flickr}}$
outperforms 5 out of 9 baselines on Pascal VOC, and 7 out of 8 baselines on
Pascal Context. For the COCO datasets, under equal resolutions, PnP-OVSS +
BLIP${}_{\text{Flickr}}$ beats 4 out of 8 baselines on COCO Object, but all
baselines on COCO Stuff. The two Pascal datasets share many images, and the
two COCO datasets use exactly the same images, but they have vastly different
lists of classes to segment. We observe that the advantage of PnP-OVSS over
methods in Group 1 becomes more pronounced as the number of classes on the
same image increases.

When applied to BridgeTower [78], PnP-OVSS still surpasses all methods in
Group 3 by 9.8% on Pascal VOC, 5.1% on Pascal Context, 8.5% on COCO Object,
and 0.4% on COCO Stuff. This showcases the plug-and-play ability of PnP-OVSS,
which excels with different base networks.

We report comparison against supervised methods in Tab 4. PnP-OVSS +
BLIP${}_{\text{Flickr}}$ outperforms 5 out of 7 methods on Pascal VOC, as well
as every baseline except MaskCLIP+ on Pascal Context and COCO stuff. As these
baselines benefit from dense supervision, the results further demonstrate the
strengths of PnP-OVSS.

###  4.3 Ablation Study

We perform gradual ablation of the components of PnP-OVSS on
BLIP${}_{\text{Flickr}}$ and report the results in Tab. 5. Each component,
including GradCAM, all Salience DropOut iterations, Gaussian blur, and Dense
CRF, contribute positively to the final performance. In particular, the first
iteration of Salience DropOut has much larger impact (+3.5/2.3) than the
second iteration (+1.4/0.8), which in turn is more important than the rest.
Interestingly, Gaussian blur by itself attains good performance (+9.6/7.7)
whereas Dense CRF only works well when combined with blur. Dense CRF alone is
worse than Gaussian blur by 1.5 mIoU on Pascal Context. This is likely caused
by the fact that hard 0/1 labels resulted from thresholding are not
informative unary potentials that can be leveraged by CRF effectively.

Layer Mean | 1 | 2 | 3 | 4 | 5 | 6
---|---|---|---|---|---|---
mIoU | 11.6 | 12.3 | 11.3 | 11.0 | 11.6 | 12.9
Layer Mean | 7 | 8 | 9 | 10 | 11 | 12
mIoU | 13.7 | 25.3 | 25.8 | 23.4 | 12.5 | 5.8
Head in Layer 8 | 1 | 2 | 3 | 4 | 5 | 6
mIoU | 16.6 | 3.8 | 8.6 | 20.0 | 19.8 | 5.2
Head in Layer 8 | 7 | 8 | 9 | 10 | 11 | 12
mIoU | 11.1 | 6.0 | 3.8 | 29.6 | 17.3 | 8.0
Table 6: Semantic segmentation performance using cross-attention maps averaged
across all heads in a layer and separate heads in Layer 8. Results are
attained with PnP-OVSS+BLIP${}_{\text{Flickr}}$ on COCO Stuff and resolution
336.

###  4.4 Hyperparameter Sensitivity

The choice of hyperparameters often exerts substantial influence on
segmentation performance. Here we quantitatively examine how the choice of
cross-attention layers and attention heads may change the segmentation mIoU on
COCO Stuff. Tab. 6 shows the results obtained from the average cross-attention
maps over all heads in each layer and those from different attention heads.

We make the following observations. First, different layers and heads have
drastic performance differences. Layer 12 achieves a meager 5.8 whereas Layer
9 achieves 25.8. Within all heads of Layer 8, the worse performance is 3.8 but
the best is 29.6. This suggests substantial functional differentiation of the
layers and the heads, and making the right choice is key to accuracy. Second,
the random search using the proposed reward function correctly identifies the
best layer-head combination, even though Layer 8 is not the best layer based
on average head performance. This indicates the effectiveness of our method.

##  5 Conclusions

Large-scale VLMs achieve high performance on many multimodal tasks that
potentially require object localization. With this paper, we propose a
technique, PnP-OVSS, that extracts the ability of semantic segmentation from
the opaque VLMs. PnP-OVSS is simple to use, requires no extra finetuning, and
delivers high performance, exceeding not only all baselines that requires no
finetuning, but also all baselines that do not use image-text pairs in
finetuning. Its success hints at a new direction for open-vocabulary
segmentation tasks leveraging large VLMs.

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikoł aj Bińkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan.  Flamingo: a visual language model for few-shot learning.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Araslanov and Roth [2020] Nikita Araslanov and Stefan Roth.  Single-stage semantic segmentation from image labels.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4253–4262, 2020.
  * Bazzani et al. [2016] Loris Bazzani, Alessandra Bergamo, Dragomir Anguelov, and Lorenzo Torresani.  Self-taught object localization with deep networks.  In _2016 IEEE winter conference on applications of computer vision (WACV)_ , pages 1–9. IEEE, 2016.
  * Bucher et al. [2019] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez.  Zero-shot semantic segmentation.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Caesar et al. [2018] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1209–1218, 2018.
  * Cha et al. [2023] Junbum Cha, Jonghwan Mun, and Byungseok Roh.  Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11165–11174, 2023.
  * Chen et al. [2023] Jun Chen, Deyao Zhu, Guocheng Qian, Bernard Ghanem, Zhicheng Yan, Chenchen Zhu, Fanyi Xiao, Sean Chang Culatana, and Mohamed Elhoseiny.  Exploring open-vocabulary semantic segmentation from clip vision encoder distillation only.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 699–710, 2023.
  * Chen et al. [2022a] Qi Chen, Lingxiao Yang, Jian-Huang Lai, and Xiaohua Xie.  Self-supervised image-specific prototype exploration for weakly supervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4288–4298, 2022a.
  * Chen et al. [2020] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In _Proceedings of the 16th European Conference on Computer Vision_ , 2020.
  * Chen et al. [2022b] Zhaozheng Chen, Tan Wang, Xiongwei Wu, Xian-Sheng Hua, Hanwang Zhang, and Qianru Sun.  Class re-activation maps for weakly-supervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 969–978, 2022b.
  * Cho et al. [2015] Minsu Cho, Suha Kwak, Cordelia Schmid, and Jean Ponce.  Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1201–1210, 2015.
  * Cinbis et al. [2016] Ramazan Gokberk Cinbis, Jakob Verbeek, and Cordelia Schmid.  Weakly supervised object localization with multi-fold multiple instance learning.  _IEEE transactions on pattern analysis and machine intelligence_ , 39(1):189–203, 2016.
  * Darcet et al. [2023] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski.  Vision transformers need registers.  _arXiv preprint arXiv:2309.16588_ , 2023.
  * Du et al. [2022] Ye Du, Zehua Fu, Qingjie Liu, and Yunhong Wang.  Weakly supervised semantic segmentation by pixel-to-prototype contrast.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4320–4329, 2022.
  * Everingham et al. [2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  _International journal of computer vision_ , 88:303–338, 2010.
  * Ghiasi et al. [2022] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin.  Scaling open-vocabulary image segmentation with image-level labels.  In _European Conference on Computer Vision_ , pages 540–557. Springer, 2022.
  * Gu et al. [2020] Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang.  Context-aware feature generation for zero-shot semantic segmentation.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 1921–1929, 2020.
  * Guo et al. [2023] Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi.  From images to textual prompts: Zero-shot visual question answering with frozen large language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10867–10877, 2023.
  * Guo et al. [2018] Yanming Guo, Yu Liu, Theodoros Georgiou, and Michael S Lew.  A review of semantic segmentation using deep neural networks.  _International journal of multimedia information retrieval_ , 7:87–93, 2018.
  * Han et al. [2023] Kunyang Han, Yong Liu, Jun Hao Liew, Henghui Ding, Jiajun Liu, Yitong Wang, Yansong Tang, Yujiu Yang, Jiashi Feng, Yao Zhao, et al.  Global knowledge calibration for fast open-vocabulary segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 797–807, 2023.
  * Hao et al. [2020] Shijie Hao, Yuan Zhou, and Yanrong Guo.  A brief survey on semantic segmentation with deep learning.  _Neurocomputing_ , 406:302–321, 2020.
  * Hou et al. [2018] Qibin Hou, PengTao Jiang, Yunchao Wei, and Ming-Ming Cheng.  Self-erasing network for integral object attention.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , 2021.
  * Jiang et al. [2022] Peng-Tao Jiang, Yuqi Yang, Qibin Hou, and Yunchao Wei.  L2g: A simple local-to-global knowledge transfer framework for weakly supervised semantic segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 16886–16896, 2022.
  * Ke et al. [2021] Tsung-Wei Ke, Jyh-Jing Hwang, and Stella X Yu.  Universal weakly supervised segmentation by pixel-to-segment contrastive learning.  _arXiv preprint arXiv:2105.00957_ , 2021.
  * Kim et al. [2021] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , 2021.
  * Kolesnikov and Lampert [2016] Alexander Kolesnikov and Christoph H Lampert.  Seed, expand and constrain: Three principles for weakly-supervised image segmentation.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14_ , pages 695–711. Springer, 2016.
  * Krähenbühl and Koltun [2011] Philipp Krähenbühl and Vladlen Koltun.  Efficient inference in fully connected crfs with gaussian edge potentials.  _Advances in neural information processing systems_ , 24, 2011.
  * Kumar Singh and Jae Lee [2017] Krishna Kumar Singh and Yong Jae Lee.  Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 3524–3533, 2017.
  * Lee et al. [2022] Jungbeom Lee, Seong Joon Oh, Sangdoo Yun, Junsuk Choe, Eunji Kim, and Sungroh Yoon.  Weakly supervised semantic segmentation using out-of-distribution data.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16897–16906, 2022.
  * Li et al. [2022a] Boyi Li, Kilian Q. Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl.  Language-driven semantic segmentation, 2022a.
  * Li et al. [2022b] Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, He Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, and Luo Si.  mPLUG: Effective and efficient vision-language learning by cross-modal skip-connections.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 2022b.
  * Li et al. [2022c] Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al.  mplug: Effective and efficient vision-language learning by cross-modal skip-connections.  _arXiv preprint arXiv:2205.12005_ , 2022c.
  * Li et al. [2021a] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  In _NeurIPS_ , 2021a.
  * Li et al. [2022d] Jinlong Li, Zequn Jie, Xu Wang, Xiaolin Wei, and Lin Ma.  Expansion and shrinkage of localization for weakly-supervised semantic segmentation.  _Advances in Neural Information Processing Systems_ , 35:16037–16051, 2022d.
  * Li et al. [2022e] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022e.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In _ICML_ , 2023.
  * Li et al. [2021b] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang.  UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 2021b.
  * Li et al. [2020] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _ECCV 2020_ , 2020.
  * Liang et al. [2023] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu.  Open-vocabulary semantic segmentation with mask-adapted clip.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7061–7070, 2023.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Liu et al. [2022] Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang.  Open-world semantic segmentation via contrasting and clustering vision-language embedding.  In _European Conference on Computer Vision_ , pages 275–292. Springer, 2022.
  * Luo et al. [2023] Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li.  Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation.  In _International Conference on Machine Learning_ , pages 23033–23044. PMLR, 2023.
  * Mottaghi et al. [2014] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille.  The role of context for object detection and semantic segmentation in the wild.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 891–898, 2014.
  * Mukhoti et al. [2023] Jishnu Mukhoti, Tsung-Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim.  Open vocabulary semantic segmentation with patch aligned contrastive learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19413–19423, 2023.
  * Pastore et al. [2021] Giuseppe Pastore, Fabio Cermelli, Yongqin Xian, Massimiliano Mancini, Zeynep Akata, and Barbara Caputo.  A closer look at self-training for zero-label semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2693–2702, 2021.
  * Qin et al. [2023] Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, et al.  Freeseg: Unified, universal and open-vocabulary image segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19446–19455, 2023.
  * Radford et al. [2021a] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , 2021a.
  * Radford et al. [2021b] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021b.
  * Ranasinghe et al. [2023] Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens.  Perceptual grouping in contrastive vision-language models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5571–5584, 2023.
  * Ren et al. [2023] Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang.  Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency.  _arXiv preprint arXiv:2302.10307_ , 2023.
  * Ru et al. [2022] Lixiang Ru, Yibing Zhan, Baosheng Yu, and Bo Du.  Learning affinity from attention: End-to-end weakly-supervised semantic segmentation with transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16846–16855, 2022.
  * Ru et al. [2023a] Lixiang Ru, Heliang Zheng, Yibing Zhan, and Bo Du.  Token contrast for weakly-supervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3093–3102, 2023a.
  * Ru et al. [2023b] Lixiang Ru, Heliang Zheng, Yibing Zhan, and Bo Du.  Token contrast for weakly-supervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3093–3102, 2023b.
  * Rubinstein et al. [2013] Michael Rubinstein, Armand Joulin, Johannes Kopf, and Ce Liu.  Unsupervised joint object discovery and segmentation in internet images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1939–1946, 2013.
  * Selvaraju et al. [2017] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2017.
  * Shin et al. [2022] Gyungin Shin, Weidi Xie, and Samuel Albanie.  Reco: Retrieve and co-segment for zero-shot transfer.  _Advances in Neural Information Processing Systems_ , 35:33754–33767, 2022.
  * Siméoni et al. [2021] Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce.  Localizing objects with self-supervised transformers and no labels.  _arXiv preprint arXiv:2109.14279_ , 2021.
  * Simon Blanke [since 2020] Simon Blanke.  Gradient-Free-Optimizers: Simple and reliable optimization with local, global, population-based and sequential techniques in numerical search spaces.  https://github.com/SimonBlanke, since 2020.
  * Singh et al. [2022] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.  Flava: A foundational language and vision alignment model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15638–15650, 2022.
  * Tang et al. [2018] Meng Tang, Federico Perazzi, Abdelaziz Djelouah, Ismail Ben Ayed, Christopher Schroers, and Yuri Boykov.  On regularized losses for weakly-supervised cnn segmentation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 507–522, 2018.
  * Tiong et al. [2022] Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven C.H. Hoi.  Plug-and-play vqa: Zero-shot vqa by conjoining large pretrained models with zero training.  In _Findings of the Conference on Empirical Methods in Natural Language Processing (Findings of EMNLP)_ , 2022.
  * Vo et al. [2019] Huy V Vo, Francis Bach, Minsu Cho, Kai Han, Yann LeCun, Patrick Pérez, and Jean Ponce.  Unsupervised image matching and object discovery as optimization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8287–8296, 2019.
  * Vo et al. [2021] Van Huy Vo, Elena Sizikova, Cordelia Schmid, Patrick Pérez, and Jean Ponce.  Large-scale unsupervised object discovery.  _Advances in Neural Information Processing Systems_ , 34:16764–16778, 2021.
  * Wang et al. [2022a] Wenguan Wang, Guolei Sun, and Luc Van Gool.  Looking beyond single images for weakly supervised semantic segmentation learning.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022a.
  * Wang et al. [2023] Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra.  Cut and learn for unsupervised object detection and instance segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3124–3134, 2023.
  * Wang et al. [2022b] Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L Crowley, and Dominique Vaufreydaz.  Self-supervised transformers for unsupervised object discovery using normalized cut.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14543–14553, 2022b.
  * Wei et al. [2019] Xiu-Shen Wei, Chen-Lin Zhang, Jianxin Wu, Chunhua Shen, and Zhi-Hua Zhou.  Unsupervised object discovery and co-localization by deep descriptor transformation.  _Pattern Recognition_ , 88:113–126, 2019.
  * Wei et al. [2017] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan.  Object region mining with adversarial erasing: A simple classification to semantic segmentation approach.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1568–1576, 2017.
  * Xenos et al. [2023] Alexandros Xenos, Themos Stafylakis, Ioannis Patras, and Georgios Tzimiropoulos.  A simple baseline for knowledge-based visual question answering.  In _EMNLP_ , 2023.
  * Xian et al. [2019] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata.  Semantic projection network for zero-and few-label semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8256–8265, 2019.
  * Xu et al. [2023a] Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, et al.  mplug-2: A modularized multi-modal foundation model across text, image and video.  _arXiv preprint arXiv:2302.00402_ , 2023a.
  * Xu et al. [2022a] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang.  Groupvit: Semantic segmentation emerges from text supervision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18134–18144, 2022a.
  * Xu et al. [2023b] Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie.  Learning open-vocabulary semantic segmentation models from natural language supervision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2935–2944, 2023b.
  * Xu et al. [2022b] Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, and Dan Xu.  Multi-class token transformer for weakly supervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4310–4319, 2022b.
  * Xu et al. [2022c] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai.  A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model.  In _European Conference on Computer Vision_ , pages 736–753. Springer, 2022c.
  * Xu et al. [2023c] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai.  Side adapter network for open-vocabulary semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2945–2954, 2023c.
  * Xu et al. [2022d] Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, and Nan Duan.  Bridgetower: Building bridges between encoders in vision-language representation learning.  _arXiv preprint arXiv:2206.08657_ , 2022d.
  * Yu et al. [2022] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  CoCa: Contrastive captioners are image-text foundation models.  _arXiv Preprint 2205.01917_ , 2022.
  * Zabari and Hoshen [2022] Nir Zabari and Yedid Hoshen.  Open-vocabulary semantic segmentation using test-time distillation.  In _European Conference on Computer Vision_ , pages 56–72. Springer, 2022.
  * Zeng et al. [2021] Yan Zeng, Xinsong Zhang, and Hang Li.  Multi-grained vision language pre-training: Aligning texts with visual concepts.  _arXiv preprint arXiv:2111.08276_ , 2021.
  * Zeng et al. [2022] Yan Zeng, Xinsong Zhang, Hang Li, Jiawei Wang, Jipeng Zhang, and Wangchunshu Zhou.  X2-vlm: All-in-one pre-trained model for vision-language tasks.  _arXiv preprint arXiv:2211.12402_ , 2022.
  * Zhang et al. [2020] Bingfeng Zhang, Jimin Xiao, Yunchao Wei, Mingjie Sun, and Kaizhu Huang.  Reliability does matter: An end-to-end weakly supervised semantic segmentation approach.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 12765–12772, 2020.
  * Zhang et al. [2023] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang.  A simple framework for open-vocabulary segmentation and detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1020–1031, 2023.
  * Zhang et al. [2021] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5579–5588, 2021.
  * Zhang et al. [2018] Xiaolin Zhang, Yunchao Wei, Jiashi Feng, Yi Yang, and Thomas S Huang.  Adversarial complementary learning for weakly supervised object localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1325–1334, 2018.
  * Zhou et al. [2014] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Object detectors emerge in deep scene cnns.  _arXiv preprint arXiv:1412.6856_ , 2014.
  * Zhou et al. [2022a] Chong Zhou, Chen Change Loy, and Bo Dai.  Extract free dense labels from clip.  In _European Conference on Computer Vision_ , pages 696–712. Springer, 2022a.
  * Zhou et al. [2022b] Tianfei Zhou, Meijie Zhang, Fang Zhao, and Jianwu Li.  Regional semantic contrast and aggregation for weakly supervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4299–4309, 2022b.

Supplementary Material

Figure 5: PnP-OVSS+BLIP${}_{\text{Flickr}}$ segmentation result for Batman and
Ironman. Figure 6: PnP-OVSS+BLIP$\\_{\text{Flickr}}$ segmentation for Minions
and Gru.

##  6 Additional Qualitative Results

We provide additional qualitative results of images from Pascal Context and
COCO Stuff in Fig 7. We note that, in the third image of the third row, PnP-
OVSS correctly recognizes four loaves of bread whereas the ground truth only
annotates one.

The rightmost column of the bottom three rows, separated from the rest by a
red dash line, are examples of failure cases. The example on top contains
multiple small instances of the same class (people and surfboard), which are
understandably hard. The middle example contains multiple instances of people,
which causes difficulties for PnP-OVSS to cover all objects. In addition,
images with a clutter of different objects and complex texture (as in the last
image) often cause a drop in performance.

Figure 7: Qualitative Results of PnP-OVSS + BLIP. Images are from Pascal
Context and COCO stuff. The bottom rows show the ground-truth (GT); the rest
are our results. The last column of the last three rows, after the red dash
line, shows failure cases.

##  7 Qualitative Results in the Wild

Fig 5 and Fig 6 show qualitative results of PnP-OVSS+BLIP$\\_{\text{Flickr}}$
for fictional and cartoon characters like Batman, Ironman, Minions, and Gru,
which are out-of-distribution object classes that do not typically appear in
semantic segmentation datasets.

##  8 PnP-OVSS Implementation Detail

BLIP. We use the ITM branch of $\text{BLIP}\\_{\text{large}}$, which adopts
VIT-L/16 as the image encoder, BERT as the text encoder, and insert an extra
cross-attention layer for each transformer block of BERT. For each cross-
attention layer, the hidden size is 768, and the number of heads is 12. We
interpolate the positional embedding to allow input resolution of 768 x 768.
We adopt pretrained weights from two checkpoints for image retrieval on COCO
and Flickr.

BridgeTower. We use the ITM branch of BridgeTower$\\_{large}$, which adopts
VIT-L/14 as the image encoder, RoBERTa$\\_{large}$ as the text encoder and an
6-layer cross attention encoder. For each cross attention layer of the cross-
modal encoder, the hidden size is set to 1,024, and the number of heads is set
to 16. We interpolate the positional embedding to allow input resolution of
770 x 770. The model weights are from the bridgetowerlarge-itm-mlm-itc
checkpoint from Huggingface.

Random Search. We adopt the random search routine from the Gradient-Free-
Optimizers library111https://github.com/SimonBlanke/Gradient-Free-Optimizers
[59] with our reward metric (§3.4). To parallelize the search process, we
divide the search space into three groups and place each group on a GPU card.
We perform 34 search iterations in each group. The best hyperparameter set
from the three groups is taken as the final search result.

Class Split for Densely Supervised Models. We follow the most common setting
[4, 17, 46, 31, 76, 88] which save pottedplant, sheep, sofa, train, tvmonitor
as the 5 unseen classes for Pascal VOC; cow, motorbike, sofa, cat, boat,
fence, bird, tvmonitor, keyboard, aeroplane as 10 unseen classes for Pascal
Context; frisbee, skateboard, cardboard, carrot, scissors, suitcase, giraffe,
cow, road, wallconcrete, tree, grass, river, clouds, playingfield, as 15
unseen classes for COCO Stuff.

##  9 Vil-Seg Evaluation Detail

In Tab 3, different from other methods that require weakly supervised
finetuning on image-text data, Vil-Seg is evaluated on subset of datasets.
Specifically, the author evaluate their method on 5 classes (potted plant,
sheep, sofa, train, tv-monitor) out of the 20 object categories in PASCAL VOC;
4 classes (cow, motorbike, sofa, cat) out of the 59 object categories in
PASCAL Context; and 15 classes (frisbee, skateboard, cardboard, carrot,
scissors, suitcase, giraffe, cow, road, wall concrete, tree, grass, river,
clouds, playing-field) out of the 171 object categories in COCO Stuff dataset.

##  10 Details of Zero-shot Semantic Segmentation Techniques

We summarize current methods for zero-shot semantic segmentation in Tab 7 to
enable straightforward comparison between methods. Specifically, we include
the supervision used, whether the method require pretraining and finetuning,
the pretraining weight, finetuning data and total data size used in each
method.

##  11 PnP-OVSS with ALBEF and mPLUG

As shown in Tab 8, we further apply PnP-OVSS on two other vision language
models with cross-attention and image text matching loss, ALBEF [34] and mPLUG
[33]. Nevertheless, the performances are not as good as BLIP or BridgeTower.
ALBEF [34] is pretrained with only 14M data with ViT-B whereas BLIP and
BridgeTower are pretrained with 129M/404M data with ViT-L. We speculate that
vision language models require sufficient numbers of parameters and
pretraining data to acquire localization capability. mPLUG [33] is another
vision language model pretrained with 14M data and ViT-L. However, mPLUG is
trained for both image task and video task. With a relatively smaller amount
of pretraining data than BLIP or BridgeTower, as well as image and video dual-
modality objectives, mPLUG also does not perform well in image object
localization.

Method | Supervision | Training | PT weight | Finetuning data | PT/T data size
---|---|---|---|---|---
Methods that require finetuning on dense annotations w/o VL Models
SPNet [71] | Pixel+Self | PT+FT | ImageNet | Pascal Voc/COCO stuff | 119K
ZS3Net [4] | Pixel+Self | PT+FT | - | Pascal Voc/ Pascal Context | 5K
CaGNet [17] | Pixel+Self | PT+FT | - | Pascal Voc/Pascal Context/COCO stuff | 123K
STRICT [46] | Pixel+Self | PT+FT | ImageNet | Pascal Voc/COCO stuff | 119K
Methods that require finetuning on dense annotations w/ VL Models
SimBase [76] | Pixel+Self | PT+FT | Maskformer/FCN + CLIP | COCO stuff | 400.1M
LSeg [31] | Pixel+Self | PT+FT | ImageNet+CLIP | Pascal VOC/COCO/FSS | 401.3M
MaskCLIP+ [88] | Pixel+Self | PT+FT | CLIP | COCO stuff | 400.1M
Methods that require finetuning on image-text pairs
OVSegmentor [74] | Text+Self | PT+FT | DINO+BERT | CC4M | 4.3M
Vil-Seg [42] | Text+Self | PT+FT | CLIP | CC12M | 412M
GroupVit* [43] | Text | T | - | CC3M+COCO | 3.4M
GroupVit [73] | Text | T | - | CC12M+YFCC14M | 26M
CLIPpy [50] | Text+Text | PT+FT | DINO+T5 Sentence | HQITP-134M | 134M
SegCLIP [43] | Text+Self | PT+FT | CLIP | CC3M+COCO | 403.4M
Viewco [51] | Text | PT+FT | GroupViT | CC12M+YFCC14M | 26M
TCL[6]+PAMR[2] | Text | PT+FT | CLIP | CC12M+CC3M | 415M
PACL [45] | Text | PT+FT | CLIP | GCC3M+GCC12M+YFCC15M | 430M
Methods that require finetuning but not image-text pair
MaskCLIP [88] w/ ST | Text+ST | PT+T | CLIP | ImageNet1K | 401.2M
ZeroSeg* [7] | Text+Self | PT+T | CLIP | CC3M+COCO | 403.4M
ZeroSeg [7] | Text+Self | PT+T | CLIP | ImageNet1K | 401.2M
Methods that require no finetuning
MaskCLIP [88] | Text | PT | CLIP | - | 400M
Reco[57] | Text | PT | CLIP+ImageNet | - | 400M
PnP-OVSS (Ours) |  |  |  |  |
\+ BLIP | Text | PT |  BLIP$\\_{\text{Flickr}}$/BLIP$\\_{\text{COCO}}$ | - | 129M
\+ BridgeTower | Text | PT | BridgeTower | - | 404M
Table 7: Current methods for zero-shot semantic segmentation. Pixel represents method require pixel level annotation, Self represents method leverage self supervision, and Text represents method leverage image-text pair annotation. PT stands for Pre-training, T stands for training, ST stands for Self-training, FT stands for finetuning. All methods with pixel supervision are trained on seen categories and tested on unseen categories. For data size, We calculate only the image-caption data used for pretraining and all type of data for finetuning. Method |  Training |  HT on Dense Labels |  Short-side Resolution |  Pascal VOC-20 |  Pascal Context-59 |  COCO Object-80 |  COCO Stuff-171
---|---|---|---|---|---|---|---
PnP-OVSS (Ours) |  |  |  |  |  |  |
\+ ALBEF | $\times$ | $\times$ | 336 | 10.8 | 6.3 | 8.9 | 10.3
\+ ALBEF | $\times$ | $\times$ | 768 | 11.1 | 6.8 | 8.8 | 10.7
\+ mPLUG | $\times$ | $\times$ | 336 | 9.2 | 8.8 | 7.9 | 6.7
Table 8: Zero-shot semantic segmentation performance in mIoU.


# Separated wave equations for all metric components over the Kerr background

Jianwei Mei  Email<EMAIL_ADDRESS>MOE Key Laboratory of TianQin Mission,
TianQin Research Center for Gravitational Physics $\&$ School of Physics and
Astronomy, Frontiers Science Center for TianQin, Gravitational Wave Research
Center of CNSA, Sun Yat-sen University (Zhuhai Campus), Zhuhai 519082, China

###### Abstract

This paper describes a method that seeks to find the fully separated wave
equations for all the metric components over the Kerr background directly.
Unfortunately, the solution found in the original version is purely gauge. The
main text has not been corrected, as a follow up updated is hoped for and the
correct solution is still being searched with the method.

Introduction— Black hole is one of the most intriguing predictions of general
relativity (GR). Observations indicate that almost every galaxy has a massive
black hole at its center Kormendy and Richstone (1995) and galaxies like the
Milky Way may contain millions of stellar mass black holes Elbert  _et al._
(2018). Given the difficulty for astrophysical black holes to maintain
significant amount of electric charges Gibbons (1975); Ghosh and Afrin (2023),
the Kerr hypothesis Carter (1971); Bekenstein (1996); Chrusciel  _et al._
(2012) essentially predicts that all stationary black holes in nature are
described by the Kerr metric Kerr (1963),

$\displaystyle ds^{2}$ $\displaystyle=$ $\displaystyle-
dt^{2}+H\Big{(}\frac{dr^{2}}{X}+\frac{dx^{2}}{Y}\Big{)}+(r^{2}+a^{2})Yd\phi^{2}$
(1) $\displaystyle+\frac{2Mr(dt-aYd\phi)^{2}}{H}\,,$

where $X=r^{2}+a^{2}-2Mr\,$, $Y=1-x^{2}$ and $H=r^{2}+a^{2}x^{2}\,$.

The breakthrough in gravitational wave (GW) detection has opened the door to
test the nature of black holes with unparalleled precision Cardoso and
Gualtieri (2016); Barack  _et al._ (2019). One example is to precisely map the
surrounding geometry of a black hole Ryan (1995). Extremely high precision
might be possible with a type of GW sources called Extreme Mass Ratio
Inspirals, which are systems consisted of a stellar mass compact object
inspiralling into a central massive black hole, making about ${\cal
O}(10^{5})$ round motions before the final plunge. With future space-based GW
detectors such as TianQin Luo  _et al._ (2016); Mei  _et al._ (2021), LISA
Amaro-Seoane  _et al._ (2017) and more Gong  _et al._ (2021), one may measure
the intrinsic parameters of EMRIs to the order ${\cal O}(10^{-6})$ Fan  _et
al._ (2020); Zi  _et al._ (2021); Seoane  _et al._ (2023). Another example is
to measure the ringdown signal of a deformed black hole settling down to a
stationary state Franchini and Völkel (2023). By using future space-based GW
detectors to detect the ringdown signals from massive black hole binary (MBHB)
events, one may constrain potential deviations from GR to the level ${\cal
O}(10^{-3})$ Shi  _et al._ (2019); Arun  _et al._ (2022). The expected
precisions are so high that nonlinear effects are needed in some of the
waveform modeling, such as the self-force effect of EMRIs Poisson  _et al._
(2011); Pound and Wardell (2021) and the nonlinear effect in black hole
ringdowns Cheung  _et al._ (2023); Mitman  _et al._ (2023).

Black hole perturbation theory (BHPT) is essential in formulating the desired
non-linear effects Afshordi  _et al._ (2023). At the basis of BHPT, the fully
separated metric wave equations over the Schwarzschild background have been
constructed by Regge and Wheeler Regge and Wheeler (1957) and Zerilli Zerilli
(1970), while separated equations for the metric waves over the Kerr
background are only known in a limited form. The separated Kerr perturbation
equations were originally discovered by Teukolsky in the early 1970’s for the
linearized Weyl scalars $\psi_{0}$ and $\psi_{4}$ Teukolsky (1972, 1973,
2015), and were then extended to cover the metric perturbations by
reconstructing them from a Hertz potential through the CCK procedure
Chrzanowski (1975); Kegeles and Cohen (1979); Wald (1978); Stewart (1979). The
CCK procedure is limited to the radiation gauge and the source-free case, the
metric perturbation is determined up to a stationary and axisymmetric piece,
and the construction needs to solve an inversion problem involving higher
order differential equations Wald (1973); Spiers  _et al._ (2023); Dolan  _et
al._ (2023). This has become an obstacle to waveform modeling involving higher
order perturbations, for which the lower order perturbations are needed as a
source Whiting and Price (2005); Ioka and Nakano (2007). Overcoming the
difficulties of the CCK procedure and formulating non-linear perturbations of
the Kerr black hole is a challenging task, and the field is still being
actively developed, see, e.g., Merlin  _et al._ (2016); Green  _et al._
(2020); Loutrel  _et al._ (2021); Toomani  _et al._ (2022); Aly and Stojkovic
(2023); Wardell  _et al._ (2023); Ripley  _et al._ (2021); Dolan  _et al._
(2023); Sberna  _et al._ (2022); Yang  _et al._ (2015); Green  _et al._ (2023)
for more references.

In this work, I present the fully separated wave equations for all metric
components over the Kerr background. To obtain the result, the Killing-Yano
tensor of the Kerr metric is used to construct a symmetry operator that
commutates with the wave operator, the eigen equation of the symmetry operator
is used to introduce the needed constant of separation, and then the wave
equation and the eigen equation of the symmetry operator are solved
simultaneously.

There have been previous works attempting to derive separated wave equations
for all the metric components under some restricted conditions, such as in the
near-horizon extremal Kerr spacetimes Chen and Stein (2017) and in the small
rotation limit Franchini (2023). Apart from the immediate application to GW
waveform modeling, the separated equations can also be useful for other
important problems, such as the nonlinear stability of black hole solutions
Dafermos  _et al._ (2021); Yang  _et al._ (2015); Klainerman and Szeftel
(2023).

In the following, all covariant derivatives are defined with the background
metric (1), and a prime on any function always means a derivative with respect
to the corresponding argument of that function.

Method— Consider the general problem of separating the variables for the wave
equation,

$\widetilde{\Box}\Psi=0\,,$ (2)

where the details of the wave operator $\widetilde{\Box}$ will depend on the
nature of the field $\Psi\,$ that it acts on.

The separability of (2) over the Kerr background is known to be related to the
existence of a Killing-Yano tensor Walker and Penrose (1970); Frolov  _et al._
(2017),

$\displaystyle k_{\mu\nu}$ $\displaystyle=$
$\displaystyle\left(\begin{matrix}0&ax&ar&0\cr-ax&0&0&a^{2}xY\cr-
ar&0&0&r(r^{2}+a^{2})\cr 0&-a^{2}xY&-r(r^{2}+a^{2})&0\end{matrix}\right)\,,$
$\displaystyle\mu,\nu$ $\displaystyle=$ $\displaystyle t,r,x,\phi\,,$ (3)

which satisfies $\nabla_{\mu}k_{\nu\rho}+\nabla_{\nu}k_{\mu\rho}=0\,$. The
“square” of the Killing-Yano tensor,
$K_{\mu\nu}=k_{\mu\rho}k_{\nu}^{~{}\rho}\,$, satisfies
$\nabla_{\mu}K_{\nu\rho}+\nabla_{\nu}K_{\rho\mu}+\nabla_{\rho}K_{\mu\nu}=0\,$.
Note $K=K_{\alpha}^{~{}\alpha}=2H\,$.

To help separate the variables in (2), one can construct a symmetry operator,
${\cal K}$, that commutates with $\widetilde{\Box}$,

$[\widetilde{\Box}\,,\,{\cal K}]=0\,,$ (4)

and then the eigen-equation

${\cal K}\Psi=\lambda\Psi\,,$ (5)

will supply the needed constant of separation, here denoted by $\lambda$. I
shall focus on ${\cal K}$ being a double-quadratic operator, i.e., one that is
quadratic both in the covariant derivative and in the Killing-Yano tensor.
Similar to $\widetilde{\Box}\,$, the detail of ${\cal K}$ will also depend on
the nature of the field that it acts on.

In the following, I apply the method to the scalar, vector and metric wave
equations one after another.

Scalar— Separated scalar wave equations over the Kerr background have been
explicitly known long ago Brill  _et al._ (1972). I use it as the simplest
example to illustrate the method. For the purpose of this work, it is enough
to consider the massless and source-free scalar field. The field equation is
given by

$\displaystyle\widetilde{\Box}\Phi\equiv\nabla_{\mu}\nabla^{\mu}\Phi=0\,.$ (6)

The symmetry operator can be constructed as

$\displaystyle{\cal K}\Phi=-\nabla\mu(K^{\mu\nu}\nabla_{\nu}\Phi)\,.$ (7)

One can then solve (6) in combination with the following equation,

$\displaystyle{\cal K}\Phi=\lambda\Phi\,,$ (8)

which brings an independent constant of separation, $\lambda\,$.

Since the Kerr background is stationary and axisymmetric, one can use the
ansatz

$\Phi(t,r,x,\phi)=\Phi(r,x)e^{i(wt-m\phi)}\,.$ (9)

In this way, the dependence on the variables $t$ and $\phi$ is easily
separated, and the problem is reduced to separating the remaining variables
$r$ and $x\,$. The final step is achieved with $\Phi(r,x)=R(r)P(x)$ and

$\displaystyle R^{\prime\prime}$ $\displaystyle=$
$\displaystyle\frac{1}{X}\Big{[}\lambda-\frac{[w(r^{2}+a^{2})-ma]^{2}}{X}\Big{]}R-\frac{2(r-M)}{X}R^{\prime}\,,$
$\displaystyle P^{\prime\prime}$ $\displaystyle=$
$\displaystyle-\frac{1}{Y}\Big{[}\lambda-\frac{(waY-m)^{2}}{Y}\Big{]}P+\frac{2x}{Y}P^{\prime}\,.$
(10)

By letting $\lambda=Q+(m-wa)^{2}\,$, one can recover the result of Brill  _et
al._ (1972) in the massless and source-free case.

Vector— Separated vector wave equations over the Kerr background have also
been known: based on Teukolsky (1972), Chandrasekhar has solved for the vector
field in terms of the separated Teukolsky functions Chandrasekhar (1976,
1985), and this work has been generalized to type-D backgrounds by Torres Del
Castillo Torres Del Castillo (1988). Lunin has re-derived the separated
equations with symmetry considerations and by using a special ansatz Lunin
(2017). Here I re-derive the separated equations again by constructing several
symmetry operators that commutate with the wave operator.

For the purpose of this work, it is enough to consider the source-free Maxwell
equations,

$\displaystyle\widetilde{\Box}A^{\nu}\equiv\nabla_{\mu}F^{\mu\nu}=\nabla_{\mu}\nabla^{\mu}A^{\nu}-\nabla_{\mu}\nabla^{\nu}A^{\mu}=0\,.$
(11)

Limited to the expressions that are both quadratic in the derivatives and in
the Killing-Yano tensor, four combinations that commutate with (11) off-shell
can be found,

$\displaystyle{\cal K}_{1}A^{\mu}$ $\displaystyle=$ $\displaystyle
A_{\alpha}k^{\gamma\mu}\nabla_{\beta}\nabla_{\gamma}k^{\alpha\beta}+A_{\alpha}k_{\beta\gamma}\nabla^{\alpha}\nabla^{\mu}k^{\beta\gamma}$
$\displaystyle+\frac{1}{6}A^{\mu}\nabla_{\gamma}k_{\alpha\beta}\nabla^{\gamma}k^{\alpha\beta}\,,$
$\displaystyle{\cal K}_{2}A^{\mu}$ $\displaystyle=$
$\displaystyle\frac{1}{2}k_{\beta\gamma}k^{\beta\gamma}\nabla_{\alpha}\nabla^{\mu}A^{\alpha}+k_{\beta\gamma}\nabla_{\alpha}A^{\alpha}\nabla^{\mu}k^{\beta\gamma}\,,$
$\displaystyle{\cal K}_{3}A^{\mu}$ $\displaystyle=$ $\displaystyle
k_{\beta\gamma}k_{\alpha}^{~{}\gamma}\nabla^{\alpha}\nabla^{\beta}A^{\mu}-k_{\beta\gamma}\nabla^{\gamma}k^{\alpha\beta}\nabla_{\alpha}A^{\mu}$
$\displaystyle-
A_{\alpha}k^{\gamma\mu}\nabla_{\beta}\nabla_{\gamma}k^{\alpha\beta}-A_{\alpha}\nabla^{\gamma}k^{\alpha\beta}\nabla^{\mu}k_{\beta\gamma}$
$\displaystyle-2\Big{(}k_{\alpha\gamma}\nabla^{\mu}k_{\beta}^{~{}\gamma}-2k_{\beta\gamma}\nabla^{\mu}k_{\alpha}^{~{}\gamma}$
$\displaystyle\qquad+k^{\gamma\mu}\nabla_{\gamma}k_{\alpha\beta}\Big{)}\nabla^{\beta}A^{\alpha}\,,$
$\displaystyle{\cal K}_{4}A^{\mu}$ $\displaystyle=$ $\displaystyle
k_{\alpha\gamma}k_{\beta}^{~{}\mu}\nabla^{\beta}\nabla^{\gamma}A^{\alpha}+\Big{(}k_{\alpha\gamma}\nabla^{\mu}k_{\beta}^{~{}\gamma}$
(12) $\displaystyle-
k_{\beta\gamma}\nabla^{\mu}k_{\alpha}^{~{}\gamma}+2k^{\gamma\mu}\nabla_{\gamma}k_{\alpha\beta}\Big{)}\nabla^{\beta}A^{\alpha}\,.$

It turns out that not all the operators are on equal footing: ${\cal
K}_{1}A\equiv A$ and so is trivial, ${\cal K}_{2}A$ vanishes in the Lorenz
gauge, $\nabla_{\mu}A^{\mu}=0\,$, while ${\cal K}_{4}A$ is the only one that
is invariant under the gauge transformation,
$A\;\rightarrow\;A+\nabla\tilde{\Phi}\,$, with $\tilde{\Phi}$ being an
arbitrary function, and that is also commutating with all the other operators:

$[{\cal K}_{4}\,,\,{\cal Q}]=0\,,\quad{\cal Q}\in\\{\widetilde{\Box}\,,\,{\cal
K}_{i}\,,\,i=1,2,3\\}\,.$ (13)

Because of these properties, ${\cal K}_{4}$ has been chosen as the symmetry
operator, i.e., (11) is solved in combination with

${\cal K}_{4}A^{\mu}=\lambda A^{\mu}\,,$ (14)

where $\lambda$ is the new constant of separation for the vector field. I have
not studied the cases when other symmetry operators are used instead.

With the ansatz

$A^{\mu}(t,r,x,\phi)=A^{\mu}(r,x)e^{iwt-im\phi}\,,$ (15)

(11) and (14) can be solved using

$\displaystyle A^{t}(r,x)$ $\displaystyle=$
$\displaystyle-\frac{i\lambda^{2}ar(r^{2}+a^{2})(m-wa)Y(Z_{1}+Z_{2})}{HX_{2}Y_{2}}\,,$
$\displaystyle A^{\phi}(r,x)$ $\displaystyle=$
$\displaystyle-\frac{i\lambda^{2}a^{2}r(m-wa)Y(Z_{1}-Z_{2})}{HX_{2}Y_{2}}\,,$
$\displaystyle A^{r}(r,x)$ $\displaystyle=$ $\displaystyle\frac{\lambda
a^{2}(m-wa)^{2}XY}{HX_{2}Y_{2}}\tilde{R}\tilde{P}\,,$ $\displaystyle
A^{\theta}(r,x)$ $\displaystyle=$ $\displaystyle\frac{RP}{H}\,,$ (16)

where

$\displaystyle Z_{1}$ $\displaystyle=$
$\displaystyle\Big{[}\tilde{R}-\frac{2aX_{4}(r^{2}+a^{2})Y-H_{2}X(m-waY)}{2\lambda
a^{2}r(r^{2}+a^{2})(m-wa)XY}X_{2}R\Big{]}\tilde{P}$
$\displaystyle-\frac{xH_{2}X_{2}Y_{2}RP}{2\lambda
a^{2}r(r^{2}+a^{2})(m-wa)^{2}Y^{2}}\,,$ $\displaystyle Z_{2}$ $\displaystyle=$
$\displaystyle-\frac{HX_{2}(m-waY)R}{2\lambda a^{2}r(r^{2}+a^{2})(m-wa)Y}$
(17)
$\displaystyle\times\Big{[}\tilde{P}-\frac{xY_{2}P}{(m-wa)(m-waY)Y}\Big{]}\,,$

and

$\displaystyle\tilde{R}$ $\displaystyle=$ $\displaystyle
R^{\prime}+\frac{\lambda rX_{4}R}{a(m-wa)X}\,,$ $\displaystyle\tilde{P}$
$\displaystyle=$ $\displaystyle P^{\prime}+\frac{\lambda
x(m-waY)P}{(m-wa)Y}\,,$ $\displaystyle X_{2}$ $\displaystyle=$
$\displaystyle\lambda^{2}r^{2}+a^{2}(m-wa)^{2}\,,$ $\displaystyle X_{3}$
$\displaystyle=$ $\displaystyle\lambda^{2}r^{2}-a^{2}(m-wa)^{2}\,,$
$\displaystyle X_{4}$ $\displaystyle=$ $\displaystyle am-(r^{2}+a^{2})w\,,$
$\displaystyle Y_{2}$ $\displaystyle=$ $\displaystyle
wa(m-wa)Y+\lambda(m-waY)^{2}\,,$ $\displaystyle H_{2}$ $\displaystyle=$
$\displaystyle r^{2}+a^{2}(2-x^{2})\,.$ (18)

The two functions, $R=R(r)$ and $P=P(x)\,$, satisfy

$\displaystyle R^{\prime\prime}$ $\displaystyle=$
$\displaystyle\frac{[\lambda^{2}+wa(m-wa)]XX_{3}-\lambda
X_{2}X_{4}^{2}}{\lambda X^{2}X_{2}}R$
$\displaystyle-\frac{(r^{2}-a^{2})X_{2}-XX_{3}}{rXX_{2}}R^{\prime}\,,$
$\displaystyle P^{\prime\prime}$ $\displaystyle=$
$\displaystyle\frac{Y_{2}^{2}-\lambda^{2}Y[2wax^{2}(m+waY)+Y_{2}]}{\lambda
Y^{2}Y_{2}}P$ (19) $\displaystyle+\frac{2\lambda
x(m^{2}-w^{2}a^{2}Y^{2})}{YY_{2}}P^{\prime}\,.$

In this result, the separation constant has a clear operator-based meaning.

Metric— In GR, the vacuum equations governing the metric perturbations over
the Kerr background (1) can be obtained from perturbing the Ricci tensor,

$\displaystyle(\widetilde{\Box}h)_{\mu\nu}$ $\displaystyle\equiv$
$\displaystyle\frac{1}{2}\nabla^{\rho}(\nabla_{\mu}h_{\nu\rho}+\nabla_{\nu}h_{\mu\rho}-\nabla_{\rho}h_{\mu\nu})-\frac{1}{2}\nabla_{\mu}\nabla_{\nu}h$
(20) $\displaystyle=$ $\displaystyle 0\,,$

where $h_{\mu\nu}\equiv\delta g_{\mu\nu}\,$. Limited to the combinations that
are both quadratic in the derivatives and in the Killing-Yano tensor, the
following four operators can be found,

$\displaystyle({\cal K}_{1}h)_{\mu\nu}$ $\displaystyle=$
$\displaystyle\frac{1}{2}(k^{\alpha\rho}k^{\beta}_{~{}\nu}-k^{\beta\rho}k^{\alpha}_{~{}\nu}-k^{\alpha\beta}k^{\rho}_{~{}\nu})\nabla_{\alpha}\nabla_{\beta}h_{\rho\mu}$
$\displaystyle-\frac{1}{2}k^{\rho\alpha}k_{\rho\nu}[\nabla^{\beta}\,,\,\nabla_{\mu}]h_{\alpha\beta}+\frac{3}{4}k^{\alpha\beta}k^{\rho}_{~{}\nu}\nabla_{\alpha}\nabla_{\beta}h_{\rho\mu}$
$\displaystyle+k_{\beta\rho}h_{\alpha\mu}\nabla_{\nu}\nabla^{\rho}k^{\alpha\beta}$
$\displaystyle+\frac{1}{6}h_{\mu\nu}\nabla_{\rho}k_{\alpha\beta}\nabla^{\rho}k^{\alpha\beta}+\frac{1}{4}hk^{\beta}_{~{}\nu}\nabla_{\alpha}\nabla^{\alpha}k_{\beta\mu}\,,$
$\displaystyle({\cal K}_{2}h)_{\mu\nu}$ $\displaystyle=$
$\displaystyle(k_{\rho\nu}\nabla^{\alpha}k^{\rho}_{~{}\mu}+k_{\rho\mu}\nabla^{\alpha}k^{\rho}_{~{}\nu})\nabla^{\beta}{\tilde{h}}_{\alpha\beta}$
$\displaystyle-k^{\rho\alpha}(k_{\rho\nu}\nabla_{\mu}\nabla^{\beta}{\tilde{h}}_{\alpha\beta}+k_{\rho\mu}\nabla_{\nu}\nabla^{\beta}{\tilde{h}}_{\alpha\beta})\,,$
$\displaystyle({\cal K}_{3}h)_{\mu\nu}$ $\displaystyle=$
$\displaystyle\nabla_{\mu}(K\nabla^{\rho}{\tilde{h}}_{\rho\nu})+\nabla_{\nu}(K\nabla^{\rho}{\tilde{h}}_{\rho\mu})\,,$
$\displaystyle({\cal K}_{4}h)_{\mu\nu}$ $\displaystyle=$
$\displaystyle(k^{\alpha\rho}k^{\beta}_{~{}\nu}-k^{\beta\rho}k^{\alpha}_{~{}\nu})\nabla_{\alpha}\nabla_{\beta}h_{\rho\mu}$
(21)
$\displaystyle+(k^{\alpha\rho}k^{\beta}_{~{}\mu}-k^{\beta\rho}k^{\alpha}_{~{}\mu})\nabla_{\alpha}\nabla_{\beta}h_{\rho\nu}$
$\displaystyle-k^{\beta\rho}(k_{\alpha\rho}\nabla_{\beta}\nabla^{\alpha}h_{\mu\nu}+\nabla_{\beta}k_{\alpha\rho}\nabla^{\alpha}h_{\mu\nu})$
$\displaystyle+\nabla^{\alpha}k^{\beta\rho}(h_{\alpha\mu}\nabla_{\nu}k_{\beta\rho}+h_{\alpha\nu}\nabla_{\mu}k_{\beta\rho})$
$\displaystyle-h^{\alpha\beta}(\nabla_{\mu}k_{\alpha\rho}\nabla_{\nu}k_{\beta}^{~{}\rho}+\nabla_{\nu}k_{\alpha\rho}\nabla_{\mu}k_{\beta}^{~{}\rho})$
$\displaystyle+2(J_{\nu}^{~{}\alpha\beta}-J^{\alpha~{}\beta}_{~{}\nu})\nabla_{\beta}h_{\alpha\mu}$
$\displaystyle+2(J_{\mu}^{~{}\alpha\beta}-J^{\alpha~{}\beta}_{~{}\mu})\nabla_{\beta}h_{\alpha\nu}\,,$

where
${\tilde{h}}_{\alpha\beta}=h_{\alpha\beta}-\frac{1}{2}g_{\alpha\beta}h\,$,
$h=g^{\mu\nu}h_{\mu\nu}\,$, and
$J^{\alpha}_{~{}\mu\nu}=k_{\rho\mu}\nabla^{\alpha}k^{\rho}_{~{}\nu}-k_{\rho\nu}\nabla^{\alpha}k^{\rho}_{~{}\mu}\,$.
Like in the vector case, the operators vary in their properties: $({\cal
K}_{1}h)_{\mu\nu}\equiv h_{\mu\nu}$ and so is trivial; both ${\cal K}_{2}$ and
${\cal K}_{3}$ vanish in the Lorenz/de Donder gauge,

$\displaystyle\nabla^{\alpha}{\tilde{h}}_{\alpha\beta}=\nabla^{\alpha}h_{\alpha\beta}-\frac{1}{2}\partial_{\beta}h=0\,.$
(22)

By looking for the simultaneous solutions to the wave equation (20), the
Lorenz gauge condition (22) and the following eigen equation,

$\displaystyle({\cal K}_{4}h)_{\mu\nu}=\lambda h_{\mu\nu}\,,$ (23)

where $\lambda$ is the constant of separation, one can find that the equations
can be fully separated with:

$\displaystyle h_{\mu\nu}(t,r,x,\phi)=e^{i(wt-m\phi)}f_{\mu\nu}(r,x)\,,$ (24)

and

$\displaystyle f_{tt}$ $\displaystyle=$ $\displaystyle
w^{2}RP+\frac{MXY}{H^{3}}\Big{(}\frac{2a^{2}rx}{X}RP^{\prime}+\frac{r^{2}-a^{2}x^{2}}{Y}R^{\prime}P\Big{)}\,,$
$\displaystyle f_{tr}$ $\displaystyle=$
$\displaystyle-i\Big{(}\frac{M[am-(r^{2}+a^{2})w](r^{2}-a^{2}x^{2})}{H^{2}X}R+wR^{\prime}\Big{)}P\,,$
$\displaystyle f_{tx}$ $\displaystyle=$ $\displaystyle-
iR\Big{[}\frac{2Marx(m-awY)}{H^{2}Y}P+wP^{\prime}\Big{]}\,,$ $\displaystyle
f_{t\phi}$ $\displaystyle=$ $\displaystyle-
mwRP-\frac{MaY}{H^{3}}\Big{[}2r(r^{2}+a^{2})xRP^{\prime}+(r^{2}-a^{2}x^{2})XR^{\prime}P\Big{]}\,,$
$\displaystyle f_{rr}$ $\displaystyle=$
$\displaystyle-\frac{1}{X}\Big{(}\lambda-\frac{[am-(r^{2}+a^{2})w]^{2}}{X}\Big{)}RP-\frac{a^{2}xY}{HX}RP^{\prime}+\frac{1}{2r}\Big{(}\frac{3r^{2}+a^{2}x^{2}}{H}+\frac{r^{2}-a^{2}}{X}\Big{)}R^{\prime}P\,,$
$\displaystyle f_{rx}$ $\displaystyle=$
$\displaystyle\frac{rRP^{\prime}+a^{2}xR^{\prime}P}{H}-R^{\prime}P^{\prime}\,,$
$\displaystyle f_{r\phi}$ $\displaystyle=$
$\displaystyle-\frac{i}{X}\Big{\\{}mr-\frac{M}{H^{2}}\Big{(}2mr^{4}-a[3r^{4}w+a^{3}(m-aw)x^{2}]Y+a^{2}r^{2}(1+x^{2})(m-awY)\Big{)}\Big{\\}}RP+imR^{\prime}P\,,$
$\displaystyle f_{xx}$ $\displaystyle=$ $\displaystyle\frac{\lambda
Y-(m-awY)^{2}}{Y^{2}}RP-\frac{x}{Y}\Big{(}1-\frac{a^{2}Y}{H}\Big{)}RP^{\prime}-\frac{rX}{HY}R^{\prime}P\,,$
$\displaystyle f_{x\phi}$ $\displaystyle=$
$\displaystyle\frac{ix}{Y}\Big{(}m+\frac{2a^{2}MrY(m-awY)}{H^{2}}\Big{)}RP+imRP^{\prime}\,,$
$\displaystyle f_{\phi\phi}$ $\displaystyle=$ $\displaystyle
m^{2}RP+\frac{xY}{H}\Big{[}r^{2}+a^{2}+\frac{2a^{2}Mr(r^{2}+a^{2}+H)Y}{H^{2}}\Big{]}RP^{\prime}-\frac{XY}{H}\Big{[}r-\frac{a^{2}M(r^{2}-a^{2}x^{2})Y}{H^{2}}\Big{]}R^{\prime}P\,.$
(25)

The equations for the functions $R=R(r)$ and $P=P(x)$ are exactly the same as
(10). Given the separated wave equations in the vacuum case, the source terms
can be added as in Teukolsky (1973). The properties of these equations will be
investigated in future works.

Acknowledgments— The author thanks Prof. Yun-Kau Lau for kindly hosting him at
the Morningside Center of Mathematics, Chinese Academy of Sciences, for
several weeks in 2014, when the author was first made aware of many
interesting topics in GW research, including the problem solved in this work.
The work has been supported in part by the Guangdong Major Project of Basic
and Applied Basic Research (Grant No. 2019B030302001) and the Natural Science
Foundation of China (Grants No. 12261131504, 91636111).

## References

  * Kormendy and Richstone (1995) J. Kormendy and D. Richstone, Annual Review of Astronomy and Astrophysics 33, 581 (1995), https://doi.org/10.1146/annurev.aa.33.090195.003053 .
  * Elbert  _et al._ (2018) O. D. Elbert, J. S. Bullock, and M. Kaplinghat, Mon. Not. Roy. Astron. Soc. 473, 1186 (2018), arXiv:1703.02551 [astro-ph.GA] .
  * Gibbons (1975) G. W. Gibbons, Commun. Math. Phys. 44, 245 (1975).
  * Ghosh and Afrin (2023) S. G. Ghosh and M. Afrin, Astrophys. J. 944, 174 (2023), arXiv:2206.02488 [gr-qc] .
  * Carter (1971) B. Carter, Phys. Rev. Lett. 26, 331 (1971).
  * Bekenstein (1996) J. D. Bekenstein, in  _2nd International Sakharov Conference on Physics_ (1996) pp. 216–219, arXiv:gr-qc/9605059 .
  * Chrusciel  _et al._ (2012) P. T. Chrusciel, J. Lopes Costa, and M. Heusler, Living Rev. Rel. 15, 7 (2012), arXiv:1205.6112 [gr-qc] .
  * Kerr (1963) R. P. Kerr, Phys. Rev. Lett. 11, 237 (1963).
  * Cardoso and Gualtieri (2016) V. Cardoso and L. Gualtieri, Class. Quant. Grav. 33, 174001 (2016), arXiv:1607.03133 [gr-qc] .
  * Barack  _et al._ (2019) L. Barack _et al._ , Class. Quant. Grav. 36, 143001 (2019), arXiv:1806.05195 [gr-qc] .
  * Ryan (1995) F. D. Ryan, Phys. Rev. D 52, 5707 (1995).
  * Luo  _et al._ (2016) J. Luo _et al._ (TianQin), Class. Quant. Grav. 33, 035010 (2016), arXiv:1512.02076 [astro-ph.IM] .
  * Mei  _et al._ (2021) J. Mei _et al._ (TianQin), PTEP 2021, 05A107 (2021), arXiv:2008.10332 [gr-qc] .
  * Amaro-Seoane  _et al._ (2017) P. Amaro-Seoane _et al._ (LISA), (2017), arXiv:1702.00786 [astro-ph.IM] .
  * Gong  _et al._ (2021) Y. Gong, J. Luo, and B. Wang, Nature Astron. 5, 881 (2021), arXiv:2109.07442 [astro-ph.IM] .
  * Fan  _et al._ (2020) H.-M. Fan, Y.-M. Hu, E. Barausse, A. Sesana, J.-d. Zhang, X. Zhang, T.-G. Zi, and J. Mei, Phys. Rev. D 102, 063016 (2020), arXiv:2005.08212 [astro-ph.HE] .
  * Zi  _et al._ (2021) T.-G. Zi, J.-D. Zhang, H.-M. Fan, X.-T. Zhang, Y.-M. Hu, C. Shi, and J. Mei, Phys. Rev. D 104, 064008 (2021), arXiv:2104.06047 [gr-qc] .
  * Seoane  _et al._ (2023) P. A. Seoane _et al._ (LISA), Living Rev. Rel. 26, 2 (2023), arXiv:2203.06016 [gr-qc] .
  * Franchini and Völkel (2023) N. Franchini and S. H. Völkel, (2023), arXiv:2305.01696 [gr-qc] .
  * Shi  _et al._ (2019) C. Shi, J. Bao, H. Wang, J.-d. Zhang, Y. Hu, A. Sesana, E. Barausse, J. Mei, and J. Luo, Phys. Rev. D 100, 044036 (2019), arXiv:1902.08922 [gr-qc] .
  * Arun  _et al._ (2022) K. G. Arun _et al._ (LISA), Living Rev. Rel. 25, 4 (2022), arXiv:2205.01597 [gr-qc] .
  * Poisson  _et al._ (2011) E. Poisson, A. Pound, and I. Vega, Living Rev. Rel. 14, 7 (2011), arXiv:1102.0529 [gr-qc] .
  * Pound and Wardell (2021) A. Pound and B. Wardell,  (2021), 10.1007/978-981-15-4702-7_38-1, arXiv:2101.04592 [gr-qc] .
  * Cheung  _et al._ (2023) M. H.-Y. Cheung _et al._ , Phys. Rev. Lett. 130, 081401 (2023), arXiv:2208.07374 [gr-qc] .
  * Mitman  _et al._ (2023) K. Mitman _et al._ , Phys. Rev. Lett. 130, 081402 (2023), arXiv:2208.07380 [gr-qc] .
  * Afshordi  _et al._ (2023) N. Afshordi _et al._ (LISA Consortium Waveform Working Group), (2023), arXiv:2311.01300 [gr-qc] .
  * Regge and Wheeler (1957) T. Regge and J. A. Wheeler, Phys. Rev. 108, 1063 (1957).
  * Zerilli (1970) F. J. Zerilli, Phys. Rev. Lett. 24, 737 (1970).
  * Teukolsky (1972) S. A. Teukolsky, Phys. Rev. Lett. 29, 1114 (1972).
  * Teukolsky (1973) S. A. Teukolsky, Astrophys. J. 185, 635 (1973).
  * Teukolsky (2015) S. A. Teukolsky, Class. Quant. Grav. 32, 124006 (2015), arXiv:1410.2130 [gr-qc] .
  * Chrzanowski (1975) P. L. Chrzanowski, Phys. Rev. D11, 2042 (1975).
  * Kegeles and Cohen (1979) L. S. Kegeles and J. M. Cohen, Phys. Rev. D19, 1641 (1979).
  * Wald (1978) R. M. Wald, Phys. Rev. Lett. 41, 203 (1978).
  * Stewart (1979) J. M. Stewart, Proc. Roy. Soc. Lond. A367, 527 (1979).
  * Wald (1973) R. M. Wald, J. Math. Phys. 14, 1453 (1973).
  * Spiers  _et al._ (2023) A. Spiers, A. Pound, and J. Moxon, Phys. Rev. D 108, 064002 (2023), arXiv:2305.19332 [gr-qc] .
  * Dolan  _et al._ (2023) S. R. Dolan, L. Durkan, C. Kavanagh, and B. Wardell, (2023), arXiv:2306.16459 [gr-qc] .
  * Whiting and Price (2005) B. F. Whiting and L. R. Price, Class. Quant. Grav. 22, S589 (2005).
  * Ioka and Nakano (2007) K. Ioka and H. Nakano, Phys. Rev. D 76, 061503 (2007), arXiv:0704.3467 [astro-ph] .
  * Merlin  _et al._ (2016) C. Merlin, A. Ori, L. Barack, A. Pound, and M. van de Meent, Phys. Rev. D 94, 104066 (2016), arXiv:1609.01227 [gr-qc] .
  * Green  _et al._ (2020) S. R. Green, S. Hollands, and P. Zimmerman, Class. Quant. Grav. 37, 075001 (2020), arXiv:1908.09095 [gr-qc] .
  * Loutrel  _et al._ (2021) N. Loutrel, J. L. Ripley, E. Giorgi, and F. Pretorius, Phys. Rev. D 103, 104017 (2021), arXiv:2008.11770 [gr-qc] .
  * Toomani  _et al._ (2022) V. Toomani, P. Zimmerman, A. Spiers, S. Hollands, A. Pound, and S. R. Green, Class. Quant. Grav. 39, 015019 (2022), arXiv:2108.04273 [gr-qc] .
  * Aly and Stojkovic (2023) F. Aly and D. Stojkovic, Class. Quant. Grav. 40, 235010 (2023), arXiv:2309.09474 [gr-qc] .
  * Wardell  _et al._ (2023) B. Wardell, A. Pound, N. Warburton, J. Miller, L. Durkan, and A. Le Tiec, Phys. Rev. Lett. 130, 241402 (2023), arXiv:2112.12265 [gr-qc] .
  * Ripley  _et al._ (2021) J. L. Ripley, N. Loutrel, E. Giorgi, and F. Pretorius, Phys. Rev. D 103, 104018 (2021), arXiv:2010.00162 [gr-qc] .
  * Sberna  _et al._ (2022) L. Sberna, P. Bosch, W. E. East, S. R. Green, and L. Lehner, Phys. Rev. D 105, 064046 (2022), arXiv:2112.11168 [gr-qc] .
  * Yang  _et al._ (2015) H. Yang, A. Zimmerman, and L. Lehner, Phys. Rev. Lett. 114, 081101 (2015), arXiv:1402.4859 [gr-qc] .
  * Green  _et al._ (2023) S. R. Green, S. Hollands, L. Sberna, V. Toomani, and P. Zimmerman, Phys. Rev. D 107, 064030 (2023), arXiv:2210.15935 [gr-qc] .
  * Chen and Stein (2017) B. Chen and L. C. Stein, Phys. Rev. D 96, 064017 (2017), arXiv:1707.05319 [gr-qc] .
  * Franchini (2023) N. Franchini, Phys. Rev. D 108, 044079 (2023), arXiv:2305.19313 [gr-qc] .
  * Dafermos  _et al._ (2021) M. Dafermos, G. Holzegel, I. Rodnianski, and M. Taylor, (2021), arXiv:2104.08222 [gr-qc] .
  * Klainerman and Szeftel (2023) S. Klainerman and J. Szeftel, Pure Appl. Math. Quart. 19, 791 (2023), arXiv:2104.11857 [math.AP] .
  * Walker and Penrose (1970) M. Walker and R. Penrose, Commun. Math. Phys. 18, 265 (1970).
  * Frolov  _et al._ (2017) V. P. Frolov, P. Krtous, and D. Kubiznak, Living Rev. Rel. 20, 6 (2017), arXiv:1705.05482 [gr-qc] .
  * Brill  _et al._ (1972) D. R. Brill, P. L. Chrzanowski, C. Martin Pereira, E. D. Fackerell, and J. R. Ipser, Phys. Rev. D 5, 1913 (1972).
  * Chandrasekhar (1976) S. Chandrasekhar, Proc. Roy. Soc. Lond. A349, 1 (1976).
  * Chandrasekhar (1985) S. Chandrasekhar, in  _Oxford, UK: Clarendon (1992) 646 p., OXFORD, UK: CLARENDON (1985) 646 P._ (1985).
  * Torres Del Castillo (1988) G. F. Torres Del Castillo, J. Math. Phys. 29, 971 (1988).
  * Lunin (2017) O. Lunin, JHEP 12, 138 (2017), arXiv:1708.06766 [hep-th] .

  *[GR]: general relativity
  *[GW]: gravitational wave
  *[EMRIs]: Extreme Mass Ratio Inspiral
  *[MBHB]: massive black hole binary
  *[BHPT]: Black hole perturbation theory