# OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-
World Unknown Objects Supervision

Junjie Wang1 Bin Chen1, 2, 3 Bin Kang2 Yulin Li1
Yichi Chen2 Weizhi Xian3 Huifeng Chang4
1 Harbin Institute of Technology, Shenzhen 2 University of Chinese Academy of
Sciences
3 Harbin Institute of Technology, Chongqing Research Institute 4 CECloud
Computing Technology Co., Ltd
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Open-Vocabulary Detection (OVD) aims to detect objects from novel categories
beyond the base categories on which the detector is trained. However, existing
open-vocabulary detectors trained on known category data tend to assign higher
confidence to trained categories and confuse novel categories with background.
To resolve this, we propose OV-DQUO, an Open-Vocabulary DETR with Denoising
text Query training and open-world Unknown Objects supervision. Specifically,
we introduce a wildcard matching method that enables the detector to learn
from pairs of unknown objects recognized by the open-world detector and text
embeddings with general semantics, mitigating the confidence bias between base
and novel categories. Additionally, we propose a denoising text query training
strategy that synthesizes additional noisy query-box pairs from open-world
unknown objects to trains the detector through contrastive learning, enhancing
its ability to distinguish novel objects from the background. We conducted
extensive experiments on the challenging OV-COCO and OV-LVIS benchmarks,
achieving new state-of-the-art results of 45.6 AP50 and 39.3 mAP on novel
categories respectively, without the need for additional training data. Models
and code are released at https://github.com/xiaomoguhz/OV-DQUO

##  1 Introduction

Open-Vocabulary Detection [37] focuses on identifying objects from novel
categories not encountered during training. Recently, Vision-Language Models
(VLMs)[22, 28, 16] pretrained on large-scale image-text pairs, such as
CLIP[22], have demonstrated impressive performance in zero-shot image
classification, providing new avenues for open-vocabulary detection.

Figure 1: (a) Detector confidence bias is a primary reason for the suboptimal
detection performance on novel categories. (b) Existing methods use VLM and
RPN to generate pseudo region-text pairs from image-caption datasets. (c)
Instead, this work leverages the open-world detector to recognize _novel
unknown objects_ within the training set and learns to match them with
wildcard text embeddings.

ViLD [6] is the first work to distill VLMs’ classification knowledge into an
object detector by aligning the detector-generated region embeddings with the
corresponding features extracted from VLMs. Subsequent methods [32, 29, 34,
36, 15] have proposed more elaborately designed strategies to improve the
efficiency of knowledge distillation, such as BARON [32], which aligns bag-of-
regions embeddings with image features extracted by VLMs. However, the context
discrepancy limits the effectiveness of knowledge distillation [44].
RegionCLIP [42] is a representative method that utilizes VLMs for pseudo-
labeling by generating pseudo region-text pairs from caption datasets[26]
using RPN and CLIP to train open-vocabulary detectors. Later works [2, 41, 40]
have further extended the implementation of pseudo-labeling, such as SASDet
[41], which explores leveraging a self-training paradigm for pseudo-labeling.
Nevertheless, these methods suffer from pseudo-label noise.

All of the above methods employ indirect utilization of VLMs, thus not
unleashing their full potential. Existing state-of-the-art methods [33, 35,
14] typically deploy a frozen CLIP image encoder as the image backbone and
perform open-vocabulary detection by extracting region features within the
prediction box. Intuitively, the performance ceiling of such methods depends
directly on the classification ability of VLMs. Therefore, current works
mainly enhance VLM’s region recognition accuracy through fine-tuning [35] or
self-distillation [33]. Yet, these methods overlook the fact that detectors
trained on known category data tend to assign higher confidence to trained
categories and confuse novel categories with background.

To verify the impact of confidence bias on novel category detection, we first
analyze the confidence assigned by VLMs and detectors to base and novel
categories, as shown in Figure 1(a). It is evident that the detector assigns
significantly lower confidence to novel category objects (e.g., umbrella) than
to known categories (e.g., person). Furthermore, we observed a significant
performance gap when using VLM to classify Ground Truth (GT) boxes compared to
detector predictions. However, this gap narrows when we manually adjust the
prediction confidence of bounding boxes based on their Intersection over Union
(IoU) with GT boxes. The experimental results reveal that confidence bias is
one of the factors responsible for suboptimal performance in novel category
detection.

Based on the above findings, we propose OV-DQUO, an open-vocabulary detection
framework with denoising text query training and open-world unknown objects
supervision. Unlike existing methods that generate pseudo region-text pairs
(Figure 1(b)), our framework propose a wildcard matching method and a
contrastive denoising training strategy to directly learn from open-world
unknown objects, mitigating performance degradation in novel category
detection caused by confidence bias.

As shown in Figure 1(c), to address the confidence bias between base and novel
categories, OV-DQUO first utilizes an open-world detector to recognize novel
unknown objects within the training set. It then queries these unknown objects
using text embeddings with general semantics (i.e., wildcard matching) and
enables the detector to regard them as query-box match. Since the open-world
detector cannot identify all novel unknown objects, we designed a denoising
text query training strategy to address the detector’s confusion between novel
categories and the background. This method synthesizes additional query-box
pairs by perturbing bounding boxes of unknown objects and assigning noisy text
embeddings, enabling OV-DQUO to leverage contrastive learning to better
distinguish novel objects from the background. Finally, to mitigate the impact
of confidence bias on region proposal selection, we propose RoQIs Selection,
which integrates region-text similarity with confidence scores to select
region proposals, achieving a more balanced recall of base and novel category
objects. The main contributions of this paper can be summarized as follows:

  * •

Inspired by the open-world detection task of recognizing novel unknown
objects, we propose an OV-DQUO framework, which mitigates the detector’s
confidence bias on novel category detection.

  * •

We design a wildcard matching method, which enables the detector to learn from
pairs of text embeddings with general semantics and novel unknown objects
recognized by the open-world detector, thereby alleviating the confidence bias
between base and novel categories.

  * •

We introduce the denoising text query training strategy, which allows a
detector to perform contrastive learning from synthetic noisy query-box pairs,
thus enhancing its ability to distinguish novel objects from the background.

  * •

OV-DQUS consistently outperforms existing state-of-the-art methods on the OV-
COCO and OV-LVIS open-vocabulary detection benchmarks and demonstrates
excellent performance in cross-dataset detection on COCO and Objects365.

##  2 Related Works

Open-Vocabulary Detection (OVD) is a paradigm proposed by OVR-CNN [37], which
aims to train models to detect objects from arbitrary categories, including
those not seen during training. State-of-the-art methods [14, 35, 33] leverage
a frozen VLM image encoder as the backbone to extract features and perform
open-vocabulary detection. Compared to pseudo-labeling [1, 43, 42, 40, 21, 27]
and knowledge distillation-based methods [32, 29, 34, 36, 15], these
approaches directly benefit from the large-scale pretraining knowledge of VLMs
and better generalize to novel objects. F-VLM [14] pioneered the discovery
that VLMs retain region-sensitive features useful for object detection. It
freezes the VLM and uses it as a backbone for feature extraction and region
classification. CORA [35] also uses a frozen VLM but fine-tunes it with a
lightweight region prompt layer, enhancing region classification accuracy.
CLIPself [33] reveals that the ViT version of VLM performs better on image
crops than on dense features, and explores aligning dense features with image
crop features through self-distillation. However, we identify that these
methods suffer from a confidence bias issue, leading to suboptimal performance
in novel category detection.

Open-World Detection (OWD) is a paradigm proposed by ORE[10], which aims to
achieve two goals: (1) recognizing both known category objects and the unknown
objects not present in the training set, and (2) enabling incremental object
detection learning through newly introduced external knowledge. OW-DETR [8]
attempts to identify potential unknown objects based on feature map activation
scores, as foreground objects typically exhibit stronger activation responses
compared to the background. PROB [45] performs distribution modeling on the
model output logits to identify unknown objects and decouples the
identification of background, known objects, and unknown objects. Based on the
observation that foreground regions exhibit more variability while background
regions change monotonously, MEPU [5] employs Weibull modeling on the feature
reconstruction error of these regions and proposes the Reconstruction Error-
based Weibull (REW) model. REW assigns likelihood scores to region proposals
that potentially belong to unknown objects. These methods inspire us to
leverage open-world detectors to address the confidence bias issue in OVD.

##  3 Methodology

In this section, we present OV-DQUO, a novel OVD framework with denoising text
query training and open-world unknown objects supervision. An overview is
given in Figure 2. First, we briefly introduce the OVD setup. Then, we detail
the open-world pseudo-labeling pipeline and the corresponding wildcard
matching strategy, which is our key approach for mitigating the confidence
bias between known and novel categories (Sec. 3.1). Subsequently, we elaborate
the denoising text query training strategy that enhances a model ability to
distinguish novel objects from the background (Sec. 3.2). Finally, we
introduce the region of query interests selection module, which achieves a
more balanced recall of base and novel category objects (Sec. 3.3).

Task Formulation. In our study, we follow the classical open-vocabulary
problem setup as in OVR-CNN [37]. In this setup, only partial class
annotations of the dataset are available during the training process, commonly
referred to as base classes and denoted by the symbol
$\mathcal{C}^{\text{base}}$. During the inference stage, the model is required
to recognize objects from both the base classes and the novel classes (denoted
as $\mathcal{C}^{\text{novel}}$, where
$\mathcal{C}^{\text{base}}\cap\mathcal{C}^{\text{novel}}=\varnothing$) that
were not seen during training, while the names of the novel classes are
provided as clues during inference.

Figure 2: Overview of OV-DQUO. (a) Open-world pseudo labeling pipeline, which
iteratively trains the detector, generates unknown object proposals, estimates
and filters foreground probabilities, and updates the training set. (b)
Denoising text query training, which enables contrastive learning with
synthetic noisy query-box pairs from unknown objects. (c) RoQIs selection
module, which considers both objectness and region-text similarity for
selecting regions of query interest.

###  3.1 Open-World Pseudo Labeling & Wildcard Matching

Unknown object proposals from the external OLN. Existing works [42, 6, 43, 41,
1, 2] leverage RPNs to mine potential novel objects, but these RPNs are biased
towards the base classes they are trained on and perform poorly on novel
classes. Unlike these approaches, we leverage the Object Localization Network
(OLN) [11] to recognize novel unknown objects from the training set in the OV-
DQUO framework, as shown in Figure 2(a). OLN is an open-world detector trained
to estimate the objectness of each region purely based on how well the
location and shape of a region overlap with any ground-truth object (e.g.,
centerness and IoU). After training OLN with $\mathcal{C}^{\text{base}}$
annotations from the OVD benchmark, we apply it to the training set to run
inference and generate open-world unknown object proposals. Specifically,
given an input image $I\in\mathbb{R}^{3\times H\times W}$, OLN outputs a
series of tuples $\mathcal{R}=\\{r_{1},r_{2},\ldots,r_{n}\\}$, where each
$r_{i}=[b_{i},q_{i}]$. Here, $b_{i}$ represents the coordinates of an unknown
proposal, and $q_{i}$ denotes the localization quality estimations.

Foreground likelihood estimation for novel unknown objects. Reducing the
impact of noisy labels is a key challenge in pseudo-label learning. Inspired
by [5], we leverage a probability distribution, which we denote as the
Foreground Estimator (FE), to estimate the likelihood that a novel unknown
object $r_{i}$ belongs to a foreground region. FE is based on the Weibull
distribution and is modeled upon the feature reconstruction error of the
foreground and background regions. Specifically, we first train a feature
reconstruction network using images from the OVD benchmark in an unsupervised
setting. Then, we collect the feature reconstruction errors for foreground and
background regions based on the $\mathcal{C}^{\text{base}}$ annotations.
Subsequently, we apply maximum likelihood estimation on Equation 1 to model
the foreground and background Weibull distributions, denoted as
$\mathcal{D}_{\textit{fg}}$ and $\mathcal{D}_{\textit{bg}}$, respectively.

$\mathcal{D}(\eta|a,c)=ac\left[1-\exp\left(-\eta^{c}\right)\right]^{a-1}\exp\left(-\eta^{c}\right)\eta^{c-1}$
(1)

where symbols $a$ and $c$ represent the shape parameters of the distribution,
while $\eta$ represents the feature reconstruction error of the foreground or
background region. With $\mathcal{D}_{\textit{fg}}$ and
$\mathcal{D}_{\textit{bg}}$, we can estimate the foreground likelihood $w_{i}$
for each novel unknown object $r_{i}=[b_{i},q_{i}]$ in $\mathcal{R}$ using
Equation 2, resulting in
$\hat{\mathcal{R}}=\\{\hat{r}_{1},\hat{r}_{2},\ldots,\hat{r}_{n}\\}$, where
$\hat{r}_{i}=[b_{i},q_{i},w_{i}]$.

$w_{i}=\frac{\mathcal{D}_{\textit{fg}}\left(\eta({b_{i}})\right)}{\mathcal{D}_{\textit{fg}}\left(\eta({b_{i}})\right)+\mathcal{D}_{\textit{bg}}\left(\eta({b_{i}})\right))},$
(2)

$\hat{\mathcal{R}}$ are used to update the training set annotations
$\mathcal{C}^{\text{base}}$ after being filtered by ground truth annotations
and some heuristic criteria. Once the training set is updated, it can be used
to retrain OLN. Subsequently, the entire process can be iterated to yield more
unknown objects, as shown in Figure 2(a). The visualization of unknown
proposals and their corresponding foreground likelihood estimations are
provided in Appendix A.3, and the details of the heuristic criteria can be
found in Appendix A.6.

Learning from open-world unknown objects via wildcard matching. The additional
supervision signals provided by open-world detectors enable OV-DQUO to avoid
treating novel objects as background during training, thereby mitigating the
confidence bias between known and novel categories. However, applying an open-
vocabulary training framework to open-world pseudo-labels raises the following
challenges: open-world unknown objects lack category information.

Unlike existing works [42, 41, 2] that re-label each proposal to specific
categories using VLMs, we propose to match open-world unknown objects directly
using text embeddings with general semantics, thereby avoiding additional
label noise. Specifically, let $\mathcal{V}_{t}$ represent the text encoder of
the VLM. The query text for unknown objects is "a photo of a
$\\{\text{wildcard}\\}$", denoted as $T_{wc}$, where the wildcard can be terms
like "object" or "thing." The query text for base classes is "a photo of a
$\\{\mathcal{C}^{\text{base}}\\}$", denoted as $T_{base}$. In the learning
process of pseudo-labels, if a region proposal $p_{i}$ generated by the OV-
DQUO encoder has an IoU with any pseudo-label in $\hat{\mathcal{R}}$ greater
than the threshold $\tau$, we assign the proposal with wildcard query
embedding $\mathcal{V}_{t}(T_{wc})$; otherwise, we assign it the text
embeddings of the base category with the maximum similarity,
$\mathcal{V}_{t}(T_{base}^{*})$, as shown in the following equation:

$(m_{i},\hat{p}_{i})=\operatorname{Decoder}(q_{i},p_{i}),\ \text{where}\
q_{i}=\begin{cases}\mathcal{V}_{t}(T_{wc})&\text{if
}\operatorname{IoU}(p_{i},\hat{\mathcal{R}})>\tau,\\\
\mathcal{V}_{t}(T_{base}^{*})&\text{otherwise}.\end{cases}$ (3)

where $\hat{\mathcal{R}}$ represents the set of open-world pseudo-labels. The
decoder of OV-DQUO iteratively refines each query with its associated anchor
box $(q_{i},p_{i})$ into output $o_{i}=(m_{i},\hat{p}_{i})$, where $m_{i}$
denotes the probability that the input query embedding matches the category of
its corresponding bounding box, and $\hat{p}_{i}$ represents the predicted
box. To achieve text query conditional matching, OV-DQUO constrains each
ground-truth box to match predictions with the same category query embedding,
including the pseudo-labels. Specifically, given a prediction set
$\mathcal{O}^{wc}=\\{o_{1}^{wc},o_{2}^{wc},\ldots,o_{n}^{wc}\mid
q_{i}=\mathcal{V}_{t}(T_{wc})\\}$ that is conditioned on wildcard query
embedding, the class-aware Hungarian matching algorithm
$\mathcal{H}_{\text{cls}}$ yields the optimal permutation
$\mathcal{M}^{wc}=\\{(\hat{r}_{1},o_{1}^{wc}),(\hat{r}_{2},o_{2}^{wc}),\ldots,(\hat{r}_{k},o_{k}^{wc})\\}$
that minimizes the matching cost $\mathcal{L}_{\text{cost}}$ between the open-
world pseudo-labels set $\hat{\mathcal{R}}$ and the predicted set
$\mathcal{O}^{wc}$ as follows:

$\mathcal{M}^{wc}=\mathcal{H}_{\text{cls}}\left(\hat{\mathcal{R}},\
\mathcal{O}^{wc},\ \mathcal{L}_{\text{cost}}\right),\ \text{where}\
\mathcal{L}_{\text{cost}}=\mathcal{L}_{\text{focal
}}(m_{i}^{wc})+\mathcal{L}_{\text{bbox}}(\hat{p}_{i}^{wc},\hat{r}_{i})$ (4)

$\mathcal{L}_{\text{focal}}$ denotes the binary focal loss [19], while
$\mathcal{L}_{\text{bbox }}$ consists of L1 loss and GIoU loss [38]. With the
matching results, the loss for unknown objects and base annotations can be
expressed as follows:

$\displaystyle\mathcal{L}_{pseudo}=\sum_{o_{i}^{wc}\in\mathcal{M}^{wc}}w_{i}\mathcal{L}_{\text{focal}}\left(m_{i}^{wc}\right),\quad\mathcal{L}_{base}=\sum_{c\in\mathcal{C}^{\text{base}}}\sum_{o_{i}^{c}\in\mathcal{M}^{c}}\left(\mathcal{L}_{\text{focal}}\left(m_{i}^{c}\right)+\mathcal{L}_{\text{bbox}}\left(\hat{p}_{i}^{c},y_{i}^{c}\right)\right)$
(5)

where $o_{i}^{wc}=(m_{i}^{wc},\hat{p}_{i}^{wc})$ and
$o_{i}^{c}=(m_{i}^{c},\hat{p}_{i}^{c})$ are the predictions selected by the
Hungarian matching algorithm, whose query embeddings are
$\mathcal{V}_{t}(T_{wc})$ and $\mathcal{V}_{t}(T_{c})$, respectively.
$y_{i}^{c}$ represents a GT of base category $c$. $w_{i}$ is the foreground
probability estimation of unknwn object $\hat{r}_{i}$. We only compute the
$\mathcal{L}_{\text{focal}}$ for unknown objects. Additionally, the
classification targets for predictions matched by $\mathcal{H}_{\text{cls}}$
are 1; otherwise, the target is 0. We omit them from the Equation 5 for
simplicity.

###  3.2 Denoising Text Query Training

Since the open-world detector cannot recognize all potential novel objects and
provide supervision signals, we propose denoising text query training to
enhance a detector’s ability to distinguish novel objects from the background.
We achieve this by enabling OV-DQUO to perform contrastive learning from
synthetic noisy query-box pairs, as shown in Figure 2(b). Specifically, for a
given unknown object box $\hat{r}_{i}$, $2N$ noise proposals
$\tilde{\mathcal{R}}=\\{\tilde{r}_{1},\tilde{r}_{2},\ldots,\tilde{r}_{2N}\\}$
are generated based on its coordinates with two noise scales $\lambda_{1}$ and
$\lambda_{2}$, where $\lambda_{1}<\lambda_{2}$. Among these proposals, the
first $N-1$ region proposals have a smaller noise scale $\lambda_{1}$ and are
regarded as positive samples during training. In contrast, the remaining
proposals from $N$ to $2N-1$ have a larger noise scale $\lambda_{2}$ and are
treated as negative samples. For query embedding $q_{i}$, if a noisy region
proposal $\tilde{r}_{i}$ belongs to the positive samples, we query it with the
correct text embedding $\mathcal{V}_{t}(T_{wc})$. In contrast, for negative
samples, we randomly select a proportion $\rho$ of samples and assign
incorrect text embeddings of base categories $\mathcal{V}_{t}(T_{base})$,
where $\rho$ is a noise scale parameter. The whole process is as follows:

$\displaystyle\tilde{r}_{i}$
$\displaystyle=\begin{cases}\hat{r}_{i}+\lambda_{1}\cdot\epsilon(\hat{r}_{i}),&\text{if
}0\leq i<N,\\\
\hat{r}_{i}+\lambda_{2}\cdot\epsilon(\hat{r}_{i}),&\text{otherwise}.\end{cases}\quad
q_{i}$ $\displaystyle=\begin{cases}\mathcal{V}_{t}(T_{base}),&\text{if }N\leq
i<2N\text{ and }R(i)<\rho,\\\
\mathcal{V}_{t}(T_{wc}),&\text{otherwise}.\end{cases}$ (6)

where $R(i)\sim\text{Uniform}(0,1)$ is a random function, and $\epsilon$ is a
function randomly calculates the offset based on input boxes. Denoising text
query training utilizes contrastive learning by treating accurate bounding
boxes with correct queries as positive samples, and bounding boxes that
partially cover objects as negative samples, regardless of the query. The
denoising part is performed simultaneously with the vanilla training part
while using the attention mask for isolation. The denoise training loss and
overall training objective for OV-DQUO can be expressed as follows:

$\displaystyle\mathcal{L}_{denoise}=\sum_{i=0}^{2N}w_{i}\mathcal{L}_{\text{focal}}\left(\tilde{m}_{i},\mathbb{I}_{(0<i<N)}\right),\
\text{where}\ \tilde{m}_{i}=\operatorname{Decoder}(q_{i},\tilde{r}_{i})$ (7)
$\displaystyle\mathcal{L}_{total}=\mathcal{L}_{pseudo}+\mathcal{L}_{base}+\mathcal{L}_{denoise}$
(8)

where $\mathbb{I}_{(0<i<N)}$ is the indicator function, which equals 1 if
$0<i<N$ and 0 otherwise. $\tilde{m}_{i}$ denotes the probability that query
embedding $q_{i}$ matches the content within bounding box $\tilde{r}_{i}$.
$\mathcal{L}_{pseudo}$ and $\mathcal{L}_{base}$ are vanilla pseudo-label
learning loss and base category loss mentioned above.

###  3.3 Region of Query Interests Selection

Existing two-stage OVD methods select region proposals based on either class-
agnostic objectness[42, 33] or region-text similarity[20]. However, as we
mentioned, objectness tends to favor the known categories. Region-text
similarities exhibit less bias when leveraging a frozen VLM image encoder as
the backbone, but they are insensitive to localization quality. As shown in
Figure 2(c), we propose Region of Query Interests (RoQIs) selection, a novel
method that considers both objectness and region-text similarity for selecting
region proposals, achieving a more balanced recall of base and novel category
objects. Specifically, given the region proposals set $\mathcal{R}$ and
corresponding objectness score vector $O$, VLM feature map $\bm{\phi}$, and
category name text embedding $\mathbf{L}$, the region of query interests set
$\mathcal{R}^{*}$ for the next stage is generated as follows:

$\mathcal{R}^{*}=\operatorname{gather}(\mathcal{R},t,N),\ \text{where
}t=(\operatorname{max}(\operatorname{RoIAlign}(\mathcal{R},\bm{\phi})\cdot\mathbf{L}^{\top}))^{\bm{\alpha}}\cdot
O^{(1-\bm{\alpha})}$ (9)

where $\operatorname{gather}$ denotes the operation of selecting top-$N$
regions from $R$ accordi ng to $t$. RoIAlign[9] is a method used to obtain
region features within a bounding boxes from the feature map $\bm{\phi}$.
$\operatorname{max}$ means the maximum similarity of each region visual
embeddings to all text embeddings. $\bm{\alpha}$ is the weighted geometric
mean parameter.

##  4 Experiments

###  4.1 Dataset & Training & Evaluation

OV-COCO benchmark. Following [37], we divide the 80 classes in the COCO
dataset [18] into 48 base classes and 17 novel classes. In this benchmark,
models are trained on the 48 base classes, which contain 107,761 images and
665,387 instances. Subsequently, the models are evaluated on the validation
set, which includes both the base and novel classes, containing 4,836 images
and 33,152 instances. For the OV-COCO benchmark, we use
$\text{AP}_{50}^{\text{Novel}}$ as our evaluation metric, which calculates the
mean average precision at an IoU of 50% for novel classes.

OV-LVIS benchmark. Following standard practice [42, 6], we remove categories
with rare tags in the LVIS [7] training set. Models are trained on 461 common
classes and 405 frequent classes, which contain 100,170 images and 1,264,883
instances. After training, the models are evaluated on the validation set,
which includes the common, frequent, and rare classes, containing 19,809
images and 244,707 instances. For the OV-LVIS benchmark, we use
$\text{mAP}_{r}$ as our evaluation metric, which calculates the box AP
averaged on IoUs from 0.5 to 0.95 for rare classes.

###  4.2 Implementation Details

Model Specifications. OV-DQUO is built on the closed-set detector DINO [39].
To adapt it for the open-vocabulary setting, we follow the previous
practice[36, 35] of modifying the decoder and letting it output matching
probabilities conditioned on the input query. OV-DQUO is configured to have
1,000 object queries, 6 encoder layers, and 6 decoder layers. In the OV-COCO
benchmark, we use CLIP of R50 and R50x4 [35] as the backbone networks. In the
OV-LVIS benchmark, we use the self-distilled CLIP of ViT-B/16 and ViT-L/14
[33] as the backbone network. For the text embedding of each category, follow
the previous works[35, 36, 33], we calculate the average representation of
each category under 80 prompt templates using the text encoder of VLM,
including the wildcard. We employ a MLP layer to transform the text embedding
dimension of VLMs into 256.

Training & Hyperparameters. We train OV-DQUO using 8 GPUs with a batch size of
4 on each GPU, using the AdamW optimizer with a learning rate of
$1\mathrm{e}{-4}$ and a weight decay of $1\mathrm{e}{-4}$. To stabilize
training, we evaluate on the exponential moving average (EMA) of the model
after training. The cost hyperparameters for class, bbox, and GIoU in the
Hungarian matching algorithm are set to 2.0, 5.0, and 2.0, respectively. More
details about the model settings and training parameters of OV-DQUO and open-
world pseudo labeling process can be found in Appendix A.5.

###  4.3 Benchmark Results

OV-COCO. Table 1 summarizes the main results of OV-DQUO on the OV-COCO
benchmark. To ensure a fair comparison, we detail the use of external training
resources, backbone size, and access to novel class names during training for
each method, as these factors vary from methods and significantly impact
performance. It can be seen that OV-DQUO consistently outperforms all state-
of-the-art methods in novel object detection, achieving the best results of
39.2/45.6 $\text{AP}_{50}^{\text{Novel}}$ with backbone networks of
RN50/R50x4, respectively. Note that CLIPSelf[33] is based on the EVA version
of CLIP[28], which is larger than our backbone and has stronger zero-shot
classification capabilities. However, OV-DQUO still outperforms CLIPSelf by
1.3 AP50 on novel categories.

Table 1: Comparison with state-of-the-art open-vocabulary object detection
methods on OV-COCO. Caption supervision indicates that the method learns from
extra image-text pairs, while CLIP supervision refers to transferring
knowledge from CLIP. The column ’Novel’ specifies whether a method requires
access to novel class names during training. †: implemented with the EVA
version of CLIP[28]. P-L, R-AT, and KD-based are classifications of methods,
denoting pseudo-labeling, region-aware training, and knowledge distillation-
based approaches, respectively, as defined in [44].

Method Taxonomy Supervision Backbone Novel AP${}_{50}^{\text{Novel}}$
AP${}_{50}^{\text{Base}}$ AP${}_{50}^{\text{All}}$ ViLD[6] KD-based CLIP RN50
✓ 27.6 59.9 51.3 Detic[43] P-L Caption[23] RN50 ✗ 27.8 47.1 42.0 OV-DETR[36]
KD-based CLIP RN50 ✓ 29.4 61.0 52.7 RegionCLIP[42] P-L Caption[26] RN50 ✗ 31.4
57.1 50.4 VLDet[17] R-AT Caption[3] RN50 ✗ 32.0 50.6 45.8 MEDet[2] R-AT
Caption[3] RN50 ✗ 32.6 54.0 49.4 BARON-KD[32] KD-based CLIP RN50 ✗ 34.0 60.4
53.5 VL-PLM[40] P-L CLIP RN50 ✓ 34.4 60.2 53.5 CLIM[34] KD-based CLIP RN50 ✗
36.9 - - SAS-Det[41] P-L CLIP RN50x4 ✓ 37.4 58.5 53.0 RegionCLIP[42] P-L
Captions[26] RN50x4 ✗ 39.3 61.6 55.7 CORA[35] R-AT CLIP RN50x4 ✗ 41.7 44.5
43.8 PromptDet[27] P-L Caption[24] ViT-B/16 ✗ 30.6 63.5 54.9 RO-ViT[13] R-AT
CLIP ViT-L/16 ✗ 33.0 - 47.7 CFM-ViT[12] R-AT CLIP ViT-L/16 ✗ 34.1 - 46.0
CLIPSelf[33] KD-based CLIP ViT-B/16†(87M) ✗ 37.6 54.9 50.4 CLIPSelf[33] KD-
based CLIP ViT-L/14†(304M) ✗ 44.3 64.1 59.0 OV-DQUO(Ours) P-L CLIP RN50(38M) ✗
39.2 41.8 41.1 OV-DQUO(Ours) P-L CLIP RN50x4(87M) ✗ 45.6 49.0 48.1

Table 2: Comparison with state-of-the-art open-vocabulary object detection
methods on OV-LVIS.

Method Supervision Backbone $\text{mAP}_{r}$ ViLD[6] CLIP RN50 16.3 OV-
DETR[36] CLIP RN50 17.4 BARON-KD[32] CLIP RN50 22.6 RegionCLIP[42] Caption[26]
RN50x4 22.0 CORA+[35] Caption[23] RN50x4 28.1 F-VLM[14] CLIP RN50x64 32.8 CFM-
ViT[12] CLIP ViT-L/14 33.9 RO-ViT[13] CLIP ViT-H/16 34.1 CLIPSelf[33] CLIP
ViT-L/14 34.9 CoDet[21] Caption[26] ViT-L/14 37.0 OV-DQUO(Ours) CLIP ViT-B/16
29.7 OV-DQUO(Ours) CLIP ViT-L/14 39.3

Table 3: Cross-datasets transfer detection from OV-LVIS to COCO and
Objects365. $\dagger$: Detection specialized pretraining with SoCo[31].

COCO Objects365 Method AP AP50 AP75 AP AP50 AP75 Supervised[6] 46.5 67.6 50.9
25.6 38.6 28.0 ViLD[6] 36.6 55.6 39.6 11.8 18.0 12.6 DetPro$\dagger$[4] 34.9
53.8 37.4 12.1 18.8 12.9 BARON[32] 36.2 55.7 39.1 13.6 21.0 14.5 RO-ViT[13] -
- - 17.1 26.9 19.5 F-VLM[14] 37.9 59.6 41.2 16.2 25.3 17.5 CoDet[21] 39.1 57.0
42.3 14.2 20.5 15.3 OV-DQUO (Ours) 39.2 55.8 42.5 18.4 26.8 19.6

OV-LVIS. Table 3 summarizes the main results of OV-DQUO on the OV-LVIS
benchmark. Since LVIS dataset encompasses considerably more categories than
COCO (1203 vs. 80), we replaced the backbone network with stronger
classification capabilities ViT-B/16 and ViT-L/14 [33] in the OV-LVIS
experiments. It is worth noting that this does not lead to an unfair
comparison, as OV-DQUO still consistently outperforms all state-of-the-art
methods, including those using the same [33] (+4.4 $\text{mAP}_{r}$) or larger
backbones [14] (+5.8 $\text{mAP}_{r}$), or using external image-caption data
[21] (+2.3 $\text{mAP}_{r}$), achieving the best result of 39.3
$\text{mAP}_{r}$.

Transfer to Other Datasets. Since the open-vocabulary detector may encounter
data from different domains in open-world applications, we further evaluate
OV-DQUO under a cross-dataset setting. Table 3 summarizes the main results of
transferring OV-DQUO trained on OV-LVIS to the validation sets of COCO[18] and
Object365[25]. We do not finetune OV-DQUO but only replace the text query
embedding with the 80 categories in COCO and the 365 categories in Object365
during testing. Experiments show that OV-DQUO achieves competitive results on
COCO and outperform the previous leading method[14] by 1.3 AP on Object365,
demonstrating robust cross-dataset generalization.

###  4.4 Ablation Study

Ablation Study on Main Components. As presented in Table 5, with the RN50x4
backbone, the vanilla OV-DQUO achieves 41.7 $\text{AP}_{50}$ on novel
categories (#1). Additional supervision from open-world unknown objects boosts
this to 43.3 $\text{AP}_{50}$ (#2). Furthermore, adding denoising text query
training brings an additional 1.7 $\text{AP}_{50}$ performance gain (#3),
demonstrating its effectiveness in improving discriminability between novel
categories and backgrounds. Finally, RoQIs selection contributes another 0.6
$\text{AP}_{50}$ to the novel categories (#5).

Effects of Matching Different Wildcards. As presented in Table 5, we explore
matching different wildcard text embeddings with open-world unknown objects.
In addition to "Object", we select several words that can represent general
foreground regions, such as "Salient Object", "Foreground Region", "Target",
and "Thing", and investigate their impact on performance. Experimental results
demonstrate that compared to intricate wildcards ("Foreground Region","Salient
Object"), simpler and more general wildcards ("Thing","Object") can achieve
better results.

Table 4: Ablation study on the main effective components of OV-DQUO.

# Open-World Supervision Denoising Text Query Training RoQIs Selection
$\text{AP}_{50}^{\text{Novel}}$ $\text{AP}_{50}^{\text{Base}}$
$\text{AP}_{50}^{\text{All}}$ 1 - - - 41.7 48.1 46.4 2 ✓ ✗ ✗ 43.3 46.8 45.8 3
✓ ✓ ✗ 45.0 49.0 47.9 4 ✗ ✗ ✓ 42.7 48.0 46.6 5 ✓ ✓ ✓ 45.6 49.0 48.1

Table 5: Ablation study on matching different wildcards with unknown objects.

Wildcard $\text{AP}_{50}^{\text{Novel}}$ $\text{AP}_{50}^{\text{Base}}$
$\text{AP}_{50}^{\text{All}}$ "Salient Object" 44.4 47.9 47.0 "Foreground
Region" 44.1 47.7 46.7 "Target" 44.5 48.6 47.5 "Thing" 44.9 48.0 47.2 "Object"
45.0 48.9 47.9

(a) Baseline Detector[35]

(b) OV-DQUO

Figure 3: Confidence score distributions

(c) Baseline Detector[35]

(d) OV-DQUO

Figure 4: Embedding distributions

Visualization Analysis of OV-DQUO. We visualize the prediction results of OV-
DQUO and the baseline detector[35] in Figures 4 and 4, including their output
confidence distributions and output embedding T-SNE results. As shown in
Figure 4, compared to the baseline detector, OV-DQUO outputs a more balanced
confidence distribution between novel and base classes. Additionally, the
confidence distribution predicted by OV-DQUO for both base and novel classes
has less overlap with the background confidence distribution. As shown in
Figure 4, compared to the baseline detector, the embeddings of novel category
object output by OV-DQUO exhibit better discriminability from background
embeddings. The comparison of the confidence distributions between OV-DQUO and
baseline detector for each novel category can be found in the Appendix A.1.

Wildcard Matching .vs Relabeling. We further compare wildcard matching with
existing relabeling methods [35, 42] to evaluate its superiority.
Specifically, we compare it with two methods: (1) relabeling each unknown
object with the most similar novel category; and (2) forcibly relabeling each
unknown object with the most similar base category. As presented in Table 7,
experiments show that pairing each open-world unknown object with a specific
category leads to suboptimal results. We believe that this outcome arises
because open-world unknown objects include many foreground objects that do not
belong to base or novel categories. Forcing these objects into specific
pairings introduces considerable noise during training. Conversely, matching
such foreground objects with wildcard text embeddings prevents model
misguidance.

Effects of Different Region Proposal Selection Strategies. We explore the
impact of different region proposal selection strategies on performance,
including objectness, region-text similarity, and RoQIs selection. As shown in
Table 7, selecting proposals based on objectness score result in the recall of
regions biased towards base categories. Besides, selecting proposals based on
region-text similarity tends to recall regions with low localization quality,
leading to performance degradation. Consequently, fusing objectness with
region-text similarity achieves best results.

Table 6: Ablation study on wildcard matching and relabeling methods

Match Method $\text{AP}_{50}^{\text{Novel}}$ $\text{AP}_{50}^{\text{Base}}$
$\text{AP}_{50}^{\text{All}}$ Base classes Relabeling 42.4 48.7 47.0 Novel
classes Relabeling 42.9 47.4 46.2 Wildcard Matching 45.0 48.9 47.9

Table 7: Ablation study on different proposal selection strategies

Selection Strategy $\text{AP}_{50}^{\text{Novel}}$
$\text{AR}_{50}^{\text{Base}}$ $\text{AR}_{50}^{\text{Novel}}$ Objectness
Selection 41.7 72.4 69.9 Region-Text Similarity 29.7 58.6 69.3 RoQIs Selection
42.7 72.1 70.5

Ablation Study on Hyperparameters. We explored the impact of different
hyperparameter settings in OV-DQUO on performance, including the number of
open-world pseudo-labeling iterations $t$, the weight $\bm{\gamma}$ for
scaling the foreground likelihood score, and the weight of the denoising loss
$\bm{\beta}$. Table 10 shows the ablation study on pseudo-labeling iteration
$t$. We calculated the recall for objects in the COCO training set after each
pseudo-labeling iteration as a reference. Experimental results indicate that
OV-DQUO achieves optimal results when $t$ equals 2. Although recall increases
with more iterations, the introduced noise starts to reduce the model
performance on novel categories. Table 10 presents the ablation study on
scaling the foreground score. We use the power function
$(w_{i})^{\bm{\gamma}}$ to scale the foreground likelihood score for each
unknown object, where $\bm{\gamma}$ controls the degree of scaling. When
$\bm{\gamma}$ is set to 0, it serves as an ablation for the FE module. Results
show that setting $\bm{\gamma}$ to 0 significantly degrades performance due to
the release of pseudo-label noise. The best performance is achieved when
$\bm{\gamma}$ is set to 0.5. Table 10 presents the ablation study on the
weight of the denoising loss $\bm{\beta}$. Experimental results show that
changing the weight of the denoising loss does not significantly affect
performance. Moreover, the best results on novel categories are achieved when
the denoising loss weight equals the classification loss weight, i.e.,
$\bm{\beta}=2$.

Table 8: Ablation study on pseudo-labeling iterations

$t$ $\text{AR}_{50}^{\text{All}}$ $\text{AP}_{50}^{\text{Novel}}$
$\text{AP}_{50}^{\text{All}}$ - 80.2 41.7 46.4 1 85.7 44.0 47.9 2 86.5 45.0
47.9 3 87.1 44.8 48.5

Table 9: Ablation study on scaling foreground score

$\bm{\gamma}$ $\text{AP}_{50}^{\text{Novel}}$ $\text{AP}_{50}^{\text{Base}}$
$\text{AP}_{50}^{\text{All}}$ 0.0 43.0 47.4 46.2 0.5 45.0 48.9 47.9 1.0 44.4
48.3 47.3 2.0 44.1 47.7 46.7

Table 10: Ablation study on denoising loss weight

$\bm{\beta}$ $\text{AP}_{50}^{\text{Novel}}$ $\text{AP}_{50}^{\text{Base}}$
$\text{AP}_{50}^{\text{All}}$ 1.0 44.8 48.3 47.4 2.0 45.0 48.9 47.9 3.0 44.4
48.9 47.7 4.0 44.4 48.6 47.5

##  5 Limitations and Conclusions

In this paper, we reveal that confidence bias constrains the novel category
detection of existing OVD methods. Inspired by open-world detection tasks that
identify unknown objects, we introduce an OV-DQUO framework to address this
bias, which achieves new state-of-the-art results on various OVD benchmarks.
While integrating OVD with OWD into a unified end-to-end framework is
promising, it remains under-explored here and reserved for future research.

## References

  * Bangalath et al. [2022] Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman H Khan, and Fahad Shahbaz Khan.  Bridging the gap between object and image-level representations for open-vocabulary detection.  _Advances in Neural Information Processing Systems_ , 35:33781–33794, 2022.
  * Chen et al. [2022] Peixian Chen, Kekai Sheng, Mengdan Zhang, Mingbao Lin, Yunhang Shen, Shaohui Lin, Bo Ren, and Ke Li.  Open vocabulary object detection with proposal mining and prediction equalization.  _arXiv preprint arXiv:2206.11134_ , 2022.
  * Chen et al. [2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ , 2015.
  * Du et al. [2022] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li.  Learning to prompt for open-vocabulary object detection with vision-language model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14084–14093, 2022.
  * Fang et al. [2023] Ruohuan Fang, Guansong Pang, Lei Zhou, Xiao Bai, and Jin Zheng.  Unsupervised recognition of unknown objects for open-world object detection.  _arXiv preprint arXiv:2308.16527_ , 2023.
  * Gu et al. [2021] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  _arXiv preprint arXiv:2104.13921_ , 2021.
  * Gupta et al. [2019] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5356–5364, 2019.
  * Gupta et al. [2022] Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah.  Ow-detr: Open-world detection transformer.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9235–9244, 2022.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * Joseph et al. [2021] KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian.  Towards open world object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5830–5840, 2021.
  * Kim et al. [2022] Dahun Kim, Tsung-Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo.  Learning open-world object proposals without learning to classify.  _IEEE Robotics and Automation Letters_ , 7(2):5453–5460, 2022.
  * Kim et al. [2023a] Dahun Kim, Anelia Angelova, and Weicheng Kuo.  Contrastive feature masking open-vocabulary vision transformer.  In _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 15556–15566, 2023a.  doi: 10.1109/ICCV51070.2023.01430.
  * Kim et al. [2023b] Dahun Kim, Anelia Angelova, and Weicheng Kuo.  Region-aware pretraining for open-vocabulary object detection with vision transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11144–11154, 2023b.
  * Kuo et al. [2022] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova.  F-vlm: Open-vocabulary object detection upon frozen vision and language models.  _arXiv preprint arXiv:2209.15639_ , 2022.
  * Li et al. [2023a] Liangqi Li, Jiaxu Miao, Dahu Shi, Wenming Tan, Ye Ren, Yi Yang, and Shiliang Pu.  Distilling detr with visual-linguistic knowledge for open-vocabulary object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 6501–6510, 2023a.
  * Li et al. [2023b] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He.  Scaling language-image pre-training via masking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 23390–23400, 2023b.
  * Lin et al. [2022] Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai.  Learning object-language alignments for open-vocabulary object detection.  _arXiv preprint arXiv:2211.14843_ , 2022.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Lin et al. [2017] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2980–2988, 2017.
  * Liu et al. [2023] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_ , 2023.
  * Ma et al. [2023] Chuofan Ma, Yi Jiang, Xin Wen, Zehuan Yuan, and Xiaojuan Qi.  Codet: Co-occurrence guided region-word alignment for open-vocabulary object detection.  In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 71078–71094. Curran Associates, Inc., 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR, 18–24 Jul 2021.
  * Ridnik et al. [2021] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor.  Imagenet-21k pretraining for the masses.  _arXiv preprint arXiv:2104.10972_ , 2021.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Shao et al. [2019] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.  Objects365: A large-scale, high-quality dataset for object detection.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 8430–8439, 2019.
  * Sharma et al. [2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565, 2018.
  * Song and Bang [2023] Hwanjun Song and Jihwan Bang.  Prompt-guided transformers for end-to-end open-vocabulary object detection.  _arXiv preprint arXiv:2303.14386_ , 2023.
  * Sun et al. [2023] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao.  Eva-clip: Improved training techniques for clip at scale.  _arXiv preprint arXiv:2303.15389_ , 2023.
  * Wang et al. [2023] Luting Wang, Yi Liu, Penghui Du, Zihan Ding, Yue Liao, Qiaosong Qi, Biaolong Chen, and Si Liu.  Object-aware distillation pyramid for open-vocabulary object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11186–11196, 2023.
  * Wang et al. [2022] Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez.  Freesolo: Learning to segment objects without annotations.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 14176–14186, 2022.
  * Wei et al. [2021] Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin.  Aligning pretraining for detection via object-level contrastive learning.  _Advances in Neural Information Processing Systems_ , 34:22682–22694, 2021.
  * Wu et al. [2023a] Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy.  Aligning bag of regions for open-vocabulary object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15254–15264, 2023a.
  * Wu et al. [2023b] Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy.  Clipself: Vision transformer distills itself for open-vocabulary dense prediction.  _arXiv preprint arXiv:2310.01403_ , 2023b.
  * Wu et al. [2024] Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Wentao Liu, and Chen Change Loy.  Clim: Contrastive language-image mosaic for region representation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 6117–6125, 2024.
  * Wu et al. [2023c] Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li.  Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 7031–7040, 2023c.
  * Zang et al. [2022] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy.  Open-vocabulary detr with conditional matching.  In _European Conference on Computer Vision_ , pages 106–122. Springer, 2022.
  * Zareian et al. [2021] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang.  Open-vocabulary object detection using captions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14393–14402, 2021.
  * Zhai et al. [2020] Hongyu Zhai, Jian Cheng, and Mengyong Wang.  Rethink the iou-based loss functions for bounding box regression.  In _2020 IEEE 9th joint international information technology and artificial intelligence conference (ITAIC)_ , volume 9, pages 1522–1528. IEEE, 2020.
  * Zhang et al. [2022] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection.  _arXiv preprint arXiv:2203.03605_ , 2022.
  * Zhao et al. [2022] Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, BG Vijay Kumar, Anastasis Stathopoulos, Manmohan Chandraker, and Dimitris N Metaxas.  Exploiting unlabeled data with vision and language models for object detection.  In _European Conference on Computer Vision_ , pages 159–175. Springer, 2022.
  * Zhao et al. [2024] Shiyu Zhao, Samuel Schulter, Long Zhao, Zhixing Zhang, Vijay Kumar B. G, Yumin Suh, Manmohan Chandraker, and Dimitris N. Metaxas.  Taming self-training for open-vocabulary object detection, 2024.
  * Zhong et al. [2022] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.  Regionclip: Region-based language-image pretraining.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16793–16803, 2022.
  * Zhou et al. [2022] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra.  Detecting twenty-thousand classes using image-level supervision.  In _European Conference on Computer Vision_ , pages 350–368. Springer, 2022.
  * Zhu and Chen [2023] Chaoyang Zhu and Long Chen.  A survey on open-vocabulary detection and segmentation: Past, present, and future.  _arXiv preprint arXiv:2307.09220_ , 2023.
  * Zohar et al. [2023] Orr Zohar, Kuan-Chieh Wang, and Serena Yeung.  Prob: Probabilistic objectness for open world object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11444–11453, 2023.

##  Appendix A Appendix / supplemental material

###  A.1 Visualization Result of Confidence Distribution for OV-DQUO and
Baseline Detector

(a) Airplane

(b) Bus

(c) Cake

(d) Cat

(e) Couch

(f) Cow

(g) Cup

(h) Dog

(i) Elephant

(j) Keyboard

(k) Knife

(l) Sink

Figure 5: Visualization result of confidence distribution for OV-DQUO and
baseline detector

As shown in Figure 5, we present details on the differences in confidence
distribution between OV-DQUO and the baseline detector [35] when detecting
novel categories. The data is derived from their predictions on the OV-COCO
validation set. The experimental results indicate that, for novel categories
such as airplanes, buses, cats, and dogs, the high-density region of the
confidence distribution for OV-DQUO lies between 0.6 and 0.8, in sharp
contrast to the baseline detector. This indicates that OV-DQUO benefits from
the additional supervision signals provided by the open-world detector.
Additionally, we observed that for novel categories such as keyboards, knives,
and sinks, the high-density region of the confidence distribution for OV-DQUO
is around 0.4. These category objects share the characteristic of being small
and typically not the salient objects within an image, which makes them
difficult for the open-world detector to recognize. However, through denoising
text query training, the confidence for these category objects still exhibits
superiority compared to the baseline detector.

###  A.2 Model Performance Analysis

We provide more details in Table 11 regarding using VLM to classify GT boxes,
classify detector predictions, and classify detector predictions with IoU
confidence. It is evident that compared with existing methods, our method
significantly improves the detection performance of novel categories and
narrows the gap with the experimental group that uses IoU as the confidence.
Simultaneously, we observed an improvement in the detection performance of
known categories. We attribute this to the model learning from open-world
pseudo-labels and denoising training, which enhances its ability to
distinguish foreground objects from the background. However, there is still a
gap between our method and the group that uses IoU as confidence. We believe
that false positive detections caused by the similarity between category text
embeddings are the primary reason for this phenomenon. We will explore this
issue in future work.

Table 11: Performance analysis on the OV-COCO validation set with backbone
networks RN50 and RN50x4.

Method Backbone $\text{AP}_{50}^{\text{Novel}}$ $\text{AP}_{50}^{\text{Base}}$
Ground Truth RN50 65.1 70.0 IoU Confidence 52.5 58.6 CORA[35] 35.1 35.5 OV-
DQUO 39.2 41.8

Method Backbone $\text{AP}_{50}^{\text{Novel}}$ $\text{AP}_{50}^{\text{Base}}$
Ground Truth RN50x4 74.1 76.0 IoU Confidence 59.1 63.7 CORA[35] 41.7 44.5 OV-
DQUO 45.6 49.0

###  A.3 Visualization of Open-World Object Proposals

Figure 6: Visualization of open-world pseudo-labels. The first row shows the
base category annotations from the OV-COCO training set, with missing novel
category objects marked by dashed boxes for each image. The second row
displays the open-world object proposals generated by OLN. The third row
presents the foreground likelihood estimation results from FE for each unknown
object proposal.

OV-DQUO mitigates the confidence bias issue between base and novel categories
by learning from open-world unknown objects. Additionally, to avoid pseudo-
label noise misleading the OV-DQUO training process, we follow the OWD method
and use a foreground estimator to assign weights to each open-world unknown
object. In Figure 6, we visualize these open-world unknown objects along with
their corresponding foreground likelihood scores. The visualization results
show that the open-world detector can identify most of the novel category
objects. Additionally, we observe that the output of the detector also
includes some non-object areas, such as distant trees and buildings.
Furthermore, it can be seen that the foreground estimator is able to assign
discriminative weights to foreground objects and non-object regions, which is
key to avoiding model degradation.

###  A.4 Visualization of Detection Results

We show the detection results of OV-DQUO on OV-COCO and OV-LVIS validation set
in Figure 7 and Figure 8, respectively. On OV-COCO dataset, OV-DQUO correctly
detects novel categories including couch, dog, bus, cow, scissors, and so on.
On LVIS dataset, OV-DQUO detects rare categories like salad plate, fedora hat,
gas mask and so on. In Figure 9, we also present the results of applying the
LVIS-trained OV-DQUO to the Objects365 dataset. We observe that OV-DQUO
trained on OV-LVIS is capable of accurately identifying a broad spectrum of
object concepts specified in the Objects365 dataset, showcasing remarkable
generalization ability.

Figure 7: Visualization of detection results on OV-COCO. Red boxes are for
novel categories, while blue boxes are for base categories. Figure 8:
Visualization of detection results on OV-LVIS. Red boxes are for rare
categories, while blue boxes are for common and frequent categories. Figure 9:
Visualization of transfer detection results on Objects365[25] dataset.

###  A.5 Details of OV-DQUO Hyper-Parameter Configuration

Detail setting for OV-DQUO. Following previous work [35], we set the
exponential moving average factor to 0.99996. The hyperparameters for the
matching cost are identical to the corresponding loss coefficients. During
inference, the temperature $\tau$ of the classification logits is set to 0.01.
Additionally, we multiply the logits of novel classes by a factor of 3.0.
There are slight differences in specific parameter settings between our
experiments on the OV-COCO and OV-LVIS datasets. These differences include the
number of training epochs, image processing resolution, and the application of
repeat factor sampling, among other parameters. Detailed configurations are
provided in Table 12.

Detail setting for open-world pseudo labeling. Following previous work [5], we
train OLN using 8 GPUs with a batch size of 2 per GPU. The models are
initialized with SoCo weights [31] and trained for 70,000 iterations using the
SGD optimizer with a learning rate of $2\times 10^{-2}$. FE are trained for
3,000 iterations with a learning rate of $2\times 10^{-7}$ and a total batch
size of 16. The training of OLN and FE adheres to the settings of OV-COCO and
OV-LVIS, where annotations for novel classes and rare categories are removed.
Following [5], we use the region proposals generated by FreeSoLo [30] as the
initial unknown object annotations.

Table 12: Experimental configurations of OV-DQUO for OV-COCO and OV-LVIS
experiments.

Configuration OV-COCO OV-LVIS Training epochs 30 35 Repeat factor sampling No
Yes Image resolution 1333 $\times$ 800 1024 $\times$ 1024 / 896 $\times$ 896
Text embedding dimensions 1024 / 640 512 / 768 Multi-scale features ResNet
(C3, C4) ViT (5, 7, 11) / (10, 14, 23) Sample categories No 100 Pseudo-label
iterations 2 3

###  A.6 Criterion Details for Filtering Open-World Object Proposals

In this section, we detail the process of filtering open-world object
proposals generated by the open-world detector. The specific steps are as
follows:

  * •

Perform non-maximum suppression based on localization quality with a threshold
of 0.3.

  * •

Ensure that the box size exceeds 2000 pixels.

  * •

Maintain an aspect ratio between 0.25 and 4.0.

  * •

Ensure that the Intersection over Union with base category objects is less
than 0.3.


#### Continuous Uniform

By definition

$\displaystyle\psi_{P}^{*}(y)=\sup\left\\{y\theta-\log\left(M_{P}[\theta]\right):\theta\in\real\right\\},$

where for $a<b$ we have that

$\displaystyle
M_{P}[\theta]=\begin{cases}\frac{\exp(b\theta)-\exp(a\theta)}{(b-a)\theta},&\theta\neq
0,\\\ 1,&\theta=0.\end{cases}$

Since $M_{P}[\theta]$ is continuous at zero, then, without loss of generality,
we obtain

$\displaystyle\begin{array}[]{rl}\psi_{P}^{*}(y)&=\sup\left\\{y\theta-\log\left(\frac{\exp(b\theta)-\exp(a\theta)}{(b-a)\theta}\right):\theta\in\real\right\\}\\\
&=\sup\left\\{(y-b)\theta-\log\left(\frac{1-\exp(-(b-a)\theta)}{(b-a)\theta}\right):\theta\in\real\right\\}\\\
&=\sup\left\\{(y-a)\theta-\log\left(\frac{\exp((b-a)\theta)-1}{(b-a)\theta}\right):\theta\in\real\right\\}\\\
&=\sup\left\\{(y-\mu)\theta-\log\left(\frac{\exp((b-\mu)\theta)-\exp((a-\mu)\theta)}{(b-a)\theta}\right):\theta\in\real\right\\}.\end{array}$
(A.13)

where $\mu=(a+b)/2$. If $y\geq b$ then from the second formulation above we
can conclude that $\psi_{P}^{*}(y)=\infty$ by taking
$\theta\rightarrow\infty$. Similarly, if, $y\leq a$, then from the third
formulation above we can conclude that $\psi_{P}^{*}(y)=\infty$ by taking
$\theta\rightarrow-\infty$. If $y=\mu$ then the last formulation of (A.13) can
be written as

$\displaystyle\sup\left\\{-\log\left(\frac{\exp(\gamma\theta)-\exp(-\gamma\theta)}{2\gamma\theta}\right):\theta\in\real\right\\}=-\log\left(\inf\left\\{\phi(\theta):\theta\in\real\right\\}\right),$

where $\gamma:=(b-a)/2>0$ and

$\displaystyle\phi(\theta):=\begin{cases}\frac{\exp(\gamma\theta)-\exp(-\gamma\theta)}{2\gamma\theta},&\theta\neq
0,\\\ 1,&\theta=0.\end{cases}$

By using L’Hôpital’s rule and some straightforward arguments, it is easy to
verify that

$\displaystyle\lim\limits_{|\theta|\rightarrow+\infty}\phi(\theta)=+\infty,\quad\lim\limits_{|\theta|\rightarrow
0}\phi(\theta)=1\quad\text{and}\quad\phi(\theta)=\phi(-\theta).$

Thus, $\phi$ is continuous at zero (which justifies its definition), coercive
and symmetric. Since the log-normalizer function
$\psi_{P}(\theta)=\log\left(M_{P}[\theta]\right)$ is strictly convex we can
conclude that if a solution exists it must be unique. The coercivity of $\phi$
implies that a solution exists, and due to the symmetry of $\phi$ we can
conclude that it must be zero. To summarize, in this case,
$\psi_{P}^{*}(\mu)=0$ (with $\theta=0$). If $y\neq\mu$ such that $a<y<b$ then
the optimal solution to (A.13) is nonzero and by the first-order optimality
conditions it must satisfy

$\displaystyle
y-\frac{b\exp(b\theta)-a\exp(a\theta)}{\exp(b\theta)-\exp(a\theta)}+\frac{1}{\theta}=0.$
(A.14)

Therefore, using (A.13) we can summarize that for
$y\in(a,b)=\mathrm{dom}\,{\psi}_{P}^{*}$:

$\displaystyle\psi_{P}^{*}(y)=\begin{cases}0,&y=\mu,\\\
(y-\mu)\theta-\log\left(\frac{\exp((b-\mu)\theta)-\exp((a-\mu)\theta)}{(b-a)\theta}\right),&y\neq\mu,\end{cases}$

where $\theta$ is the root of (A.14).

#### Logistic

The moment generating function for Logistic distribution with location and
scaling parameters $\mu$ and $s>0$, respectively, is given by

$\displaystyle M_{P}[\theta]=\exp(\mu y)B(1-s\theta,1+s\theta),\qquad
s\theta\in(-1,1),$

where $B(\cdot,\cdot)$ stands for the _Beta function_

$\displaystyle B(\alpha,\beta)=\int_{0}^{1}t^{\alpha-1}(1-t)^{\beta-1}dt.$

The beta function and the closely related _gamma function_

$\displaystyle\Gamma(\alpha)=\int_{0}^{\infty}t^{\alpha-1}\exp(-t)dt,\qquad\alpha>0,$

share the following well-known relation

$\displaystyle
B(\alpha,\beta)=\frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}.$
(A.15)

The gamma function is an extension of the factorial as for a positive integer
$\alpha$ it holds that $\Gamma(\alpha)=(\alpha-1)!$. In the following, we will
use the well-known function equations

$\displaystyle B(\alpha+1,\beta)=B(\alpha,\beta)\frac{\alpha}{\alpha+\beta},$
(A.16)

and

$\displaystyle
B(\alpha,1-\alpha)=\Gamma(1-\alpha)\Gamma(\alpha)=\frac{\pi}{\sin(\pi\alpha)},\qquad\alpha\notin\mathbb{Z}.$
(A.17)

The latter is known as Euler’s reflection formula or Euler’s function
equation. Further details and proofs for both (A.16) and (A.17) can be found,
for example, in [2].

Since $s\theta\in(-1,1)$, the above relations imply that for any $\theta\neq
0$

$\displaystyle\phi_{s}(\theta):=B(1-s\theta,1+s\theta)\overset{\eqref{apndx:eq:beta_func_1}}{=}B(-s\theta,1+s\theta)\frac{-s\theta}{-s\theta+1+s\theta}\overset{\eqref{apndx:eq:beta_eulers_reflection}}{=}\frac{-\pi
s\theta}{\sin(-\pi s\theta)}.$

For $\theta=0$ we can verify by (A.15) that

$\displaystyle\phi_{s}(\theta)=B_{s}(1-s\theta,1+s\theta)=1.$

Thus, we can summarize

$\displaystyle\phi_{s}(\theta)=B(1-s\theta,1+s\theta)=\begin{cases}1,&s\theta=0,\\\
\frac{-\pi s\theta}{\sin(-\pi
s\theta)},&s\theta\in(-1,1)\setminus\\{0\\}.\end{cases}$ (A.18)

Using L’Hôpital’s rule we can verify that $\phi_{s}$ is continuous at
$\theta=0$. Since $-\pi s\theta\geq\sin(-\pi s\theta)$ for all
$s\theta\in(-1,1)$ we can conclude that $\phi_{s}(\theta)\geq 1$ for all
$s\theta\in(-1,1)$ and equality ($\phi_{s}(\theta)=1$) holds if and only if
$s\theta=1$. Taking $|s\theta|\rightarrow 1$ it is evident that
$\phi_{s}(\theta)\rightarrow\infty$. In addition, for any $\theta\neq 0$ the
derivative of $\phi$ is given by

$\displaystyle\phi_{s}^{\prime}(\theta)=-\pi s\left[\frac{\sin(-\pi
s\theta)+\pi s\theta\cos(-\pi s\theta)}{\sin^{2}(-\pi s\theta)}\right],$

and consequently

$\displaystyle\frac{\phi_{s}^{\prime}(\theta)}{\phi_{s}(\theta)}=\frac{\sin(-\pi
s\theta)+\pi s\theta\cos(-\pi s\theta)}{\theta\sin(-\pi s\theta)}.$ (A.19)

We are now ready to evaluate Cramér’s rate function that corresponds to the
logistic distribution.

$\displaystyle\begin{array}[]{rl}\psi_{P}^{*}(y)&=\sup\left\\{y\theta-\log\left(M_{P}[\theta]\right):\theta\in\real\right\\}\\\
&=\sup\left\\{(y-\mu)\theta-\log\left(\phi_{s}(\theta)\right):\theta\in\real\right\\}.\end{array}$
(A.22)

If $y=\mu$ then the discussion that follows equation (A.18) implies that
$\sup\\{-\log(\phi_{s}(\theta)):\theta\in\real\\}\leq 0$ where the upper bound
is attained for $\theta=0$ (since $\phi_{s}(\theta)\geq 1$ and
$\phi_{s}(0)=1$). Thus, we can conclude that $\psi_{P}^{*}(\mu)=0$. If
$y\neq\mu$ then the optimal solution to (A.22) satisfies $\theta\neq 0$.
Since, in addition, for $|s\theta|\rightarrow 1$ we have that
$\phi_{s}(\theta)\rightarrow\infty$, and consequently,
$-\log(\phi_{s}(\theta))\rightarrow-\infty$, an optimal solution to (A.22) for
the case $y\neq\mu$ must satisfy the first-order optimality conditions

$\displaystyle
0=y-\mu-\frac{\phi_{s}^{\prime}(\theta)}{\phi_{s}(\theta)}=y-\mu-\frac{1}{\theta}-\frac{\pi
s}{\tan{(-\pi s\theta)}},$ (A.23)

where the above follows from (A.19). To summarize,

$\displaystyle\psi_{P}^{*}(y)=\begin{cases}0,&y=\mu,\\\
(y-\mu)\theta-\log\left(B(1-s\theta,1+s\theta)\right),&y\neq\mu,\end{cases}$

where $\theta\in\real$ is the nonzero root of (A.23).