Zhao<EMAIL_ADDRESS>Aoxue<EMAIL_ADDRESS>Fengwei
<EMAIL_ADDRESS>Zhenguo<EMAIL_ADDRESS>Qi
<EMAIL_ADDRESS>Department of Computer Science and Engineering
The Chinese University of Hong Kong
Shatin, Hong Kong Huawei, China Open-Vocabulary Object Detection with MIC

# Open-Vocabulary Object Detection with
Meta Prompt Representation and
Instance Contrastive Optimization

###### Abstract

Classical object detectors are incapable of detecting novel class objects that
are not encountered before. Regarding this issue, Open-Vocabulary Object
Detection (OVOD) is proposed, which aims to detect the objects in the
candidate class list. However, current OVOD models are suffering from
overfitting on the base classes, heavily relying on the large-scale extra
data, and complex training process. To overcome these issues, we propose a
novel framework with Meta prompt and Instance Contrastive learning (MIC)
schemes. Firstly, we simulate a novel-class-emerging scenario to help the
prompt learner that learns class and background prompts generalize to novel
classes. Secondly, we design an instance-level contrastive strategy to promote
intra-class compactness and inter-class separation, which benefits
generalization of the detector to novel class objects. Without using knowledge
distillation, ensemble model or extra training data during detector training,
our proposed MIC outperforms previous SOTA methods trained with these complex
techniques on LVIS. Most importantly, MIC shows great generalization ability
on novel classes, _e.g_ , with $+4.3\%$ and $+1.9\%\ \mathrm{AP}$ improvement
compared with previous SOTA on COCO and Objects365, respectively.

##  1 Introduction

Deep learning models have been successful in closed-set large-scale object
detection, in which the carefully designed detectors [Ren et al.(2015)Ren, He,
Girshick, and Sun, He et al.(2017)He, Gkioxari, Dollár, and Girshick, Carion
et al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko, Zhu et
al.(2021)Zhu, Su, Lu, Li, Wang, and Dai] can accurately localize and classify
the objects learned from the training set. However, these classical detectors
always fail to generalize to unseen novel class objects during inference.
Thanks to recent advancements in vision-language models [Radford et
al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin,
Clark, et al., Jia et al.(2021)Jia, Yang, Xia, Chen, Parekh, Pham, Le, Sung,
Li, and Duerig, Li et al.(2022)Li, Zhang, Zhang, Yang, Li, Zhong, Wang, Yuan,
Zhang, Hwang, et al.], ViLD [Gu et al.(2022)Gu, Lin, Kuo, and Cui] extends the
traditional closed-set object detection to an open-set scenario, named Open-
Vocabulary Object Detection (OVOD). Under this OVOD setting, the detector is
trained with only base classes, then tested on both base and novel classes. In
ViLD, an object vocabulary (see Figure. 1a) is given to obtain text embeddings
(a.k.a. class embeddings) through pretrained text encoder. The object class is
predicted by finding the best matched one from the embedded object vocabulary,
and the bounding box is obtained from the class-agnostic regression head.

From recent works on OVOD [Gu et al.(2022)Gu, Lin, Kuo, and Cui, Zhou et
al.(2022c)Zhou, Girdhar, Joulin, Krähenbühl, and Misra], it has been widely
studied that the proposal generator in a trained detector can generalize to
novel classes well even the detector is trained only on the base classes.
However, the current performance of open-vocabulary object detector
degenerates and results in low AP when generalized to novel classes.
Intuitively, the low performance is caused by the uncertain matching between
proposal features and class embeddings. Under the OVOD setting with a large
amount of classes, as shown in Figure 1a, many classes (_e.g_ , puffin v.s.
bird) are very similar, which may lead to mismatching between proposal
features and class embeddings. Furthermore, as the detector can only be
learned on the base classes, the detector fails to recognize unseen novel
classes. There can be a set of highly similar candidate classes (including
both base and novel classes) when the detector meets a novel class object,
which brings high uncertainty to novel object class prediction. This infers
too close clustering of data points from similar classes in the latent feature
space, and thus the decision boundary crosses high density regions [Zhou et
al.(2003)Zhou, Bousquet, Lal, Weston, and Schölkopf, Grandvalet and
Bengio(2004)]. Meanwhile, the trained detector may misclassify a novel class
object as background [Joseph et al.(2021)Joseph, Khan, Khan, and
Balasubramanian, Gupta et al.(2022)Gupta, Narayan, Joseph, Khan, Khan, and
Shah]. Current OVOD methods can not handle these issues effectively. DetPro
[Du et al.(2022)Du, Wei, Zhang, Shi, Gao, and Li] makes class embeddings
learnable, while other works [Zhou et al.(2022c)Zhou, Girdhar, Joulin,
Krähenbühl, and Misra, Feng et al.(2022)Feng, Zhong, Jie, Chu, Ren, Wei, Xie,
and Ma, Bangalath et al.(2022)Bangalath, Maaz, Khattak, Khan, and Shahbaz
Khan] train the model with more classes. Since only the base classes are given
during training, the corresponding class embeddings in DetPro [Du et
al.(2022)Du, Wei, Zhang, Shi, Gao, and Li] result in overfitting on the base
classes, which has been pointed out in few-shot vision-language learning [Zhou
et al.(2022b)Zhou, Yang, Loy, and Liu]. In [Zhou et al.(2022c)Zhou, Girdhar,
Joulin, Krähenbühl, and Misra, Feng et al.(2022)Feng, Zhong, Jie, Chu, Ren,
Wei, Xie, and Ma, Bangalath et al.(2022)Bangalath, Maaz, Khattak, Khan, and
Shahbaz Khan], they supplement lots of extra data from ImageNet21k [Deng et
al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei], Conceptual Captions [Sharma
et al.(2018)Sharma, Ding, Goodman, and Soricut], or LAION-400M [Schuhmann et
al.(2021)Schuhmann, Vencu, Beaumont, Kaczmarczyk, Mullis, Katta, Coombes,
Jitsev, and Komatsuzaki] in the training process, but it is unfair by
including novel classes during training as demonstrated in OWL-ViT [Minderer
et al.(2022)Minderer, Gritsenko, Stone, Neumann, Weissenborn, Dosovitskiy,
Mahendran, Arnab, Dehghani, Shen, et al.].

(a)

(b)

Figure 1: (a) In OVOD, the detector aims to detect any objects within an
object vocabulary in an input image. Previous method, _e.g_ , DetPro, can
easily misclassify some highly similar classes (puffin v.s. bird). Our method
improves the model generalization ability, which can be more discriminative to
these similar categories. Note that every point indicates a category in the
latent space; (b) The error rate of predicting novel objects as base ones.

To alleviate the overfitting issue more efficiently, we propose a meta prompt
learning scheme. Specifically, during each training iteration, in addition to
batch-wise training data and its annotations, we randomly sample more class
names from base object vocabulary to simulate the novel-class-emerging
scenario as in OVOD setting. Then, these enriched text samples are used to
learn prompts of text representations. With meta representation sampling
scheme, we can obtain more discriminative text representations and thus
improve the generalization ability to novel classes. Meanwhile, instead of
randomly initializing the background class embedding, we learn the background
prompt representation. With the learnable background class embedding, the
detector can better distinguish the negative (background) proposals from
positive (foreground) ones. The object vocabulary list can be extended with
more easily-accessed class names and to further alleviate the overfitting
issue (see Table 1). Further, we incorporate an instance-level contrastive
learning scheme to promote the intra-class compactness and inter-class
separation, which expands low-density regions in the latent feature space by
narrowing the cluster of base classes during the detector training. With such
instance contrastive learning strategy, the novel classes are potentially
separated from the base ones in the latent space and thus benefits novel class
generalization. As shown in Figure 1b, MIC decreases the error rate of
predicting novel objects as base ones significantly, from 50.4% to 19.7%.
These results demonstrate the effectiveness of our method for alleviating the
extreme overfitting issue.

Our main contributions are summarized as: 1) We introduce a novel meta prompt
learning scheme to simulate a novel-class-emerging scenario, which boosts the
generalization ability. Meanwhile, the learnable background prompt is
incorporated to help the detector distinguish the positive and negative
proposals. 2) We propose an instance-level contrastive learning strategy to
promote intra-class compactness and inter-class separation, in which the
contrastive pairs are built among foreground and background proposal samples.
3) We conduct extensive experiments on the benchmark dataset LVIS. Without
knowledge distillation, ensemble model or extra training data, our method
outperforms previous SOTA methods equipped with these complex techniques. Most
importantly, our method shows great generalization ability on directly
transferring to other datasets, such as COCO and Objects365.

##  2 Related Work

#### Open-Vocabulary Object Detection.

Classical object detectors [Ren et al.(2015)Ren, He, Girshick, and Sun, He et
al.(2017)He, Gkioxari, Dollár, and Girshick, Lin et al.(2017b)Lin, Goyal,
Girshick, He, and Dollár, Tian et al.(2019)Tian, Shen, Chen, and He, Carion et
al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko, Zhu et
al.(2021)Zhu, Su, Lu, Li, Wang, and Dai, Meng et al.(2021)Meng, Chen, Fan,
Zeng, Li, Yuan, Sun, and Wang] heavily rely on the large-scale training data
and can not generalize to unseen classes during inference. To alleviate these
issues, lots of specific methods have been designed, such as semi/self-
supervised [Tang et al.(2021)Tang, Ramaiah, Wang, Xu, and Xiong, Su et
al.(2022)Su, Lin, Hao, Cao, Wang, and Wu], few/zero-shot [Sun et al.(2021)Sun,
Li, Cai, Yuan, and Zhang, Huang et al.(2022)Huang, Han, Cheng, and Zhang], and
open-world detection [Joseph et al.(2021)Joseph, Khan, Khan, and
Balasubramanian, Han et al.(2022)Han, Ren, Ding, Pan, Yan, and Xia]. However,
zero-shot detection can only tackle in-domain setting, such as splitting COCO
as seen and unseen classes, while open-world detection can only detect the
unknown objects without classifying them. Recently, ViLD [Gu et al.(2022)Gu,
Lin, Kuo, and Cui] proposes a new setting called Open-Vocabulary Object
Detection (OVOD), which aims to detect any objects within an object vocabulary
in an input image. To achieve this, ViLD replaces the traditional classifier
weights with the class embeddings generated by CLIP [Radford et
al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin,
Clark, et al.] to make it generalize to novel classes. Following ViLD, some
works are done to improve the detection performance by learning prompt
representations [Du et al.(2022)Du, Wei, Zhang, Shi, Gao, and Li] or
supplementing extra training data [Zhou et al.(2022c)Zhou, Girdhar, Joulin,
Krähenbühl, and Misra, Feng et al.(2022)Feng, Zhong, Jie, Chu, Ren, Wei, Xie,
and Ma, Bangalath et al.(2022)Bangalath, Maaz, Khattak, Khan, and Shahbaz
Khan]. For the former, the learned prompt representation can be easily
overfitting on the base classes, which can be harmful to the generalization
ability of the detector. For the latter, it is unfair to train on the novel
classes from extra data, which is not strictly OVOD setting [Minderer et
al.(2022)Minderer, Gritsenko, Stone, Neumann, Weissenborn, Dosovitskiy,
Mahendran, Arnab, Dehghani, Shen, et al.].

#### Prompt Learning.

Prompt learning is a light-weight framework to adapt large vision-language
pretrained models[Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh,
Agarwal, Sastry, Askell, Mishkin, Clark, et al., Jia et al.(2021)Jia, Yang,
Xia, Chen, Parekh, Pham, Le, Sung, Li, and Duerig, Li et al.(2022)Li, Zhang,
Zhang, Yang, Li, Zhong, Wang, Yuan, Zhang, Hwang, et al.] to downstream task.
CoOp [Zhou et al.(2022a)Zhou, Yang, Loy, and Liu] proposes to learn the prompt
representations but not just use a human-designed prompt template. CoCoOp
[Zhou et al.(2022b)Zhou, Yang, Loy, and Liu] further handles the issue in CoOp
that the model overfits on the base classes during training and fails on the
novel classes during inference. Naively transferring prompt learning to OVOD
can also cause overfitting, but the instance-level adaption strategy in CoCoOp
is impossible for OVOD with the limited GPU memory.

Figure 2:  Overview of our proposed method. The training stage is divided into
two consecutive parts: i) meta prompt learning and ii) detector training.
During i) meta prompt learning, to simulate a novel-class-emerging scenario,
we sample a batch-wise varying object vocabulary with $\mathcal{C}_{S}$ from
$\mathcal{C}_{B}$, which improves the generalization ability of learned
foreground prompt. Also, we integrate the learnable background prompt to help
the model distinguish foreground and background proposals. Further, in ii)
detector training, we introduce an instance-level contrastive learning scheme
to promote intra-class compactness and inter-class separation. During iii)
inference stage, we use the learned foreground prompt representation to
generate class embeddings for novel classes.

#### Contrastive Learning.

Contrastive pretraining [He et al.(2020)He, Fan, Wu, Xie, and Girshick, Chen
et al.(2020)Chen, Kornblith, Norouzi, and Hinton, Grill et al.(2020)Grill,
Strub, Altché, Tallec, Richemond, Buchatskaya, Doersch, Avila Pires, Guo,
Gheshlaghi Azar, et al., Caron et al.(2020)Caron, Misra, Mairal, Goyal,
Bojanowski, and Joulin] has been validated effective for learning
discriminative feature representations. Such pretraining strategy is based on
contrastive image pairs built from two different strong augmentations of an
image. Further, to increase the diversity of contrasive image pairs,
supervised contrastive learning [Khosla et al.(2020)Khosla, Teterwak, Wang,
Sarna, Tian, Isola, Maschinot, Liu, and Krishnan] is proposed. It builds
contrastive pairs across different images, which better exploit the inherent
information from images. In this work, we take the superiority and extend it
to instance-level contrastive learning, which promote intra-class compactness
and inter-class separation.

##  3 Method

###  3.1 Preliminary

In OVOD, the model can only access base classes $\mathcal{C}_{B}$ during
training and is expected to detect the objects within a wider class vocabulary
$\mathcal{C}\\!=\\!\mathcal{C}_{B}\cup\mathcal{C}_{N}$, where
$\mathcal{C}_{N}$ represents the set of novel classes. We adopt CLIP [Radford
et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell,
Mishkin, Clark, et al.] as our vision-language model during meta prompt
learning, in which the text encoder $E_{\mathcal{T}}(\cdot)$ produces the
class embeddings and the image encoder $E_{\mathcal{I}}(\cdot)$ outputs the
image embeddings. ViLD [Gu et al.(2022)Gu, Lin, Kuo, and Cui] takes the class
embeddings obtained from prompt engineering, which is not effective enough for
OVOD since the text encoder $E_{\mathcal{T}}(\cdot)$ is highly sensitive to
prompt templates. Regarding this issue, DetPro [Du et al.(2022)Du, Wei, Zhang,
Shi, Gao, and Li] learns a class-shared foreground prompt representation
following CoOp [Zhou et al.(2022a)Zhou, Yang, Loy, and Liu] instead of using
human-defined prompt templates. The prompt representation $\boldsymbol{V}_{c}$
for class $c\in\mathcal{C}_{B}$ is
$\boldsymbol{V}_{c}=\left\\{\boldsymbol{v}_{1},\boldsymbol{v}_{2},\dots,\boldsymbol{v}_{L_{p}},\boldsymbol{w}_{c}\right\\}$,
where $\boldsymbol{v}_{i}$ denotes the $i$-th learnable context vector,
$L_{p}$ is the length of learnable context, and $\boldsymbol{w}_{c}$ is the
fixed word embedding for class $c$. Here, context vectors can be analogue to
the human-designed prompt, such as “a photo of”. The word embedding
$\boldsymbol{w}_{c}$ is obtained from the corresponding class name, and the
learnable context vectors
$\boldsymbol{V}_{fg}=\left\\{\boldsymbol{v}_{i}\right\\}_{i=1}^{L_{p}}$ are
randomly initialized. The class embedding $\boldsymbol{t}_{c}$ of class $c$ is
obtained by
$\boldsymbol{t}_{c}=E_{\mathcal{T}}\left(\boldsymbol{V}_{c}\right)$. Given a
positive (foreground) proposal, it is fed into image encoder
$E_{\mathcal{I}}(\cdot)$ to obtain its embedding $\boldsymbol{f}_{p}$, and the
probability of $\boldsymbol{f}_{p}$ to be classified as class $c$ is
$p^{p}_{c}=\frac{\exp\left(\operatorname{sim}\left(\boldsymbol{f}_{p},\boldsymbol{t}_{c}\right)/\tau\right)}{\sum_{i\in\mathcal{C}_{B}}\exp\left(\operatorname{sim}\left(\boldsymbol{f}_{p},\boldsymbol{t}_{i}\right)/\tau\right)}$,
where $\tau$ is a temperature parameter, and
$\operatorname{sim}\left(\cdot,\cdot\right)$ denotes the cosine similarity.
The cross-entropy loss is used to optimize the context vectors
$\boldsymbol{V}_{fg}$ while the base CLIP is fixed:

$\mathcal{L}_{p}=-\log p^{p}_{c}.$ (1)

Negative (background) proposals can not be recognized as any foreground
objects. Given a negative proposal, its embedding $\boldsymbol{f}_{n}$ should
be dissimilar to any foreground class embeddings. In practice, with a large
$|\mathcal{C}_{B}|$, simply optimizing prediction probability $p^{n}_{c}$ of a
negative proposal to $\frac{1}{|\mathcal{C}_{B}|}$ can achieve this goal.
Thus, the negative loss is
$\mathcal{L}_{\mathrm{n}}=-\frac{1}{\left|\mathcal{C}_{B}\right|}\sum_{c=1}^{\left|\mathcal{C}_{B}\right|}\log
p^{n}_{c}$, where $p_{c}^{n}$ is computed in the same way as $p_{c}^{p}$ by
replacing $\boldsymbol{f}_{p}$ with $\boldsymbol{f}_{n}$.

###  3.2 Meta Prompt Learning

With learnable foreground prompt $\boldsymbol{V}_{fg}$, the model performs
better compared with prompt engineering. However, one urging issue is that the
learned context can be easily overfitting on base classes $\mathcal{C}_{B}$
and lack of generalization ability to novel classes $\mathcal{C}_{N}$. In this
section, we introduce a Meta Prompt Learning (MPL) scheme to promote
generalization, as shown in Figure 2.

#### Meta Representation Sampling.

Recall that there are $|\mathcal{C}_{N}|$ novel classes during inference in
OVOD. Regarding this, we aim to simulate such a novel-class-emerging scenario
to promote the prompt representation generalizing to novel classes well, in
which a batch-wise varying vocabulary $\mathcal{C}_{S}$ is sampled from the
base class vocabulary $\mathcal{C}_{B}$. Specifically, given a batch of
proposal samples during prompt learning, with the base class embeddings
$\boldsymbol{T}_{B}=\left\\{\boldsymbol{t}_{i}\right\\}_{i\in\mathcal{C}_{B}}$,
we sample a subset
$\boldsymbol{T}_{S}\\!\\!=\\!\\!\left\\{\boldsymbol{t}_{i}\right\\}_{i\in\mathcal{C}_{S}}$
of $\boldsymbol{T}_{B}$
($\boldsymbol{T}_{S}\\!\\!\subset\\!\\!\boldsymbol{T}_{B}$). In this subset
$\boldsymbol{T}_{S}$, the class embeddings of classes existed in the current
batch proposal samples are reserved, and other class embeddings are randomly
sampled from the remaining base classes. Then the probability of
$\boldsymbol{f}_{p}$ to be classified as the corresponding class $c$ is
changed to

$p^{p}_{c}=\frac{\exp\left(\operatorname{sim}\left(\boldsymbol{f}_{p},\boldsymbol{t}_{c}\right)/\tau\right)}{\sum_{i\in\mathcal{C}_{S}}\exp\left(\operatorname{sim}\left(\boldsymbol{f}_{p},\boldsymbol{t}_{i}\right)/\tau\right)}.$
(2)

With batch-wise varying class embeddings, the learned $\boldsymbol{V}_{fg}$ is
more generalizable and robust to the unseen novel classes, which helps the
generalization of detector to novel classes.

#### Background Prompt Representation.

A typical challenge for a classical detector is how to distinguish the
negative (background) proposals from positive (foreground) ones, which becomes
more serious in OVOD. With only base classes during training, detector will
easily misclassify novel class objects as background. Previous works [Du et
al.(2022)Du, Wei, Zhang, Shi, Gao, and Li, Zhou et al.(2022c)Zhou, Girdhar,
Joulin, Krähenbühl, and Misra] randomly initialize the background class
embedding, while the randomness of the background class embedding can mislead
the judgement of detector. To this end, we introduce the learnable background
prompt representation with $L_{n}$ learnable context vectors:
$\boldsymbol{V}_{bg}=\\{\boldsymbol{v}_{1}^{bg},\boldsymbol{v}_{2}^{bg},\dots,\boldsymbol{v}_{L_{n}}^{bg}\\}$.
Also, we can obtain the background class embedding
$\boldsymbol{t}_{bg}=E_{\mathcal{T}}\left(\boldsymbol{V}_{bg}\right)$.
Although the background proposals can not be recognized as the foreground
objects, sometimes objects are partially located in the background proposals.
So directly including the background proposals into prompt learning can be
harmful. Instead, we consider the background prompt as a negative item to
foreground classes. Then, Eq. (2) becomes

$p^{p}_{c}=\frac{\exp\left(\operatorname{sim}\left(\boldsymbol{f}_{p},\boldsymbol{t}_{c}\right)/\tau\right)}{\sum\limits_{i\in\mathcal{C}_{S}}\\!\\!\exp\left(\operatorname{sim}\left(\boldsymbol{f}_{p},\boldsymbol{t}_{i}\right)/\tau\right)+\exp\left(\operatorname{sim}\left(\boldsymbol{f}_{p},\boldsymbol{t}_{bg}\right)/\tau\right)}.$
(3)

We use $\mathcal{L}_{p}$ in Eq. (1) to optimize $p^{p}_{c}$ in Eq. (3). For
the negative proposals, the prediction probability is computed by

$p^{n}_{c}=\frac{\exp\left(\operatorname{sim}\left(\boldsymbol{f}_{n},\boldsymbol{t}_{c}\right)/\tau\right)}{\sum_{i\in\mathcal{C}_{S}}\exp\left(\operatorname{sim}\left(\boldsymbol{f}_{n},\boldsymbol{t}_{i}\right)/\tau\right)},$
(4)

and we use the following loss to optimize $p^{n}_{c}$:

$\mathcal{L}_{\mathrm{n}}=-\frac{1}{\left|\mathcal{C}_{S}\right|}\sum_{c=1}^{\left|\mathcal{C}_{S}\right|}\log
p^{n}_{c}.$ (5)

###  3.3 Instance Contrastive Learning

With $\boldsymbol{V}_{fg}$ and $\boldsymbol{V}_{bg}$ learned in MPL, we then
train our detector as shown in Figure 2. As the novel classes are not seen
during training, the detector can easily classify a novel class object as a
base class one. Such phenomenon reflects the decision boundary crosses the
high density regions by mistake [Wang et al.(2022)Wang, Zhu, Chen, Zhao, Yu,
Wang, and Tang]. In this section, we propose Instance Contrastive Learning
(ICL), aiming to expand low-density regions in the latent space by narrowing
the cluster of base classes, which potentially pulls the cluster of novel
classes from base ones.

#### Class-balanced Memory Bank.

Given a large object class vocabulary, previous contrastive learning methods
[Chen et al.(2020)Chen, Kornblith, Norouzi, and Hinton, He et al.(2020)He,
Fan, Wu, Xie, and Girshick] use a huge batch size (_e.g_ , $8192$) to learn
with more samples, which is unaffordable for detection. Even we perform
instance-level contrastive learning, the number of proposals contained in a
single batch is only around $400$. So regarding this issue, we build an
instance memory bank $\mathcal{Q}$ to collect diverse proposal samples.
Moreover, to avoid frequent class dominating, we make this memory bank class
balanced, in which the memory bank $\mathcal{Q}_{c}$ for each base class
$c\in\mathcal{C}_{B}$ and background class contains $M$ proposal samples. The
memory bank is updated every iteration as the following: i) We filter out the
foreground proposals with high Intersection of Union (IoU) $>U_{p}$ and
identify background proposals with low IoU $<U_{n}$. Here, we set large
$U_{p}$ and small $U_{n}$ to make the proposal samples representative. ii) We
sample $m$ ($m<M$) proposal samples that are the most dissimilar with the
existing ones in $\mathcal{Q}$ to enrich the diversity of the samples in
$\mathcal{Q}$. Along with the learning process, the memory bank $\mathcal{Q}$
is maintained in a first in and first out manner.

#### Optimization.

Directly applying compact regularization to the high dimension proposal
embeddings may result in an over-constraint to the network that hinders its
convergence. Therefore, we introduce a projection network $\mathcal{Z}_{\phi}$
to map the proposal embeddings $\boldsymbol{f}$ into another low dimensional
space as $\boldsymbol{z}$. Inspired by supervised contrastive learning [Khosla
et al.(2020)Khosla, Teterwak, Wang, Sarna, Tian, Isola, Maschinot, Liu, and
Krishnan], we propose an instance-level contrastive loss to learn compact
embeddings of proposal samples. The contrastive loss is defined as

$\mathcal{L}_{icl}=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{\left|\mathcal{Q}_{\boldsymbol{c}(i)}\right|}\\!\\!\sum_{j}^{\left|\mathcal{Q}_{\boldsymbol{c}(i)}\right|}\\!\log\frac{\exp\left(\boldsymbol{z}_{i}\cdot\boldsymbol{z}_{j}/\gamma\right)}{\sum_{k}^{\left|\mathcal{A}_{\boldsymbol{c}(i)}\right|}\exp\left(\boldsymbol{z}_{i}\cdot\boldsymbol{z}_{k}/\gamma\right)},$
(6)

where $N$ is the number of proposal samples, $\boldsymbol{c}(i)$ is the class
label of $i$-th proposal sample, $\mathcal{Q}_{\boldsymbol{c}(i)}$ denotes the
memory bank of class $\boldsymbol{c}(i)$, $\gamma$ is a temperature
hyperparameter, and
$\mathcal{A}_{\boldsymbol{c}(i)}=\mathcal{Q}\backslash\mathcal{Q}_{\boldsymbol{c}(i)}$.
By expanding the low density regions of base classes cluster in the hidden
space, the detector can learn more robust and generalizable feature
representations.

###  3.4 Training

The training process of the proposed method consists of two parts: i) meta
prompt learning and ii) detector training. For meta prompt learning, the
overall loss is $\mathcal{L}_{mpl}=\mathcal{L}_{p}+\mathcal{L}_{n}$, where
$\mathcal{L}_{p}$ and $\mathcal{L}_{n}$ are defined in Eq. (1) and Eq. (5),
respectively. For detector training, the overall loss is
$\mathcal{L}_{det}=\mathcal{L}_{rpn}+\mathcal{L}_{cls}+\mathcal{L}_{reg}+\alpha\mathcal{L}_{icl}$,
where $\mathcal{L}_{rpn}$ is RPN loss, $\mathcal{L}_{reg}$ is regression $L1$
loss, $\mathcal{L}_{cls}$ is classification cross-entropy loss,
$\mathcal{L}_{icl}$ is defined in Eq. (6), and $\mathcal{L}_{icl}$ is weighted
by $\alpha$. The overall training algorithm is given in the appendix.

##  4 Experiment

###  4.1 Datasets and Evaluation Metrics

#### Datasets.

We evaluate our method on the large-scale open-vocabulary benchmark LVIS
[Gupta et al.(2019)Gupta, Dollar, and Girshick]. LVIS v1 is a large-scale
dataset with $1203$ categories for object detection and instance segmentation
task. The long-tailed distribution of LVIS dataset is very suitable for OVOD
setting. We take frequent and common classes as the base classes
$\mathcal{C}_{B}$ ($866$ classes), and rare classes as the novel classes
$\mathcal{C}_{N}$ ($337$ classes). This dataset contains $100$k and $20$k
images for training and validation. The models are trained only on base
classes and evaluated on both base and novel classes. We also conduct transfer
experiments to validate the generalization ability by directly evaluating
LVIS-trained model on Pascal VOC [Everingham et al.(2010)Everingham, Van Gool,
Williams, Winn, and Zisserman], COCO [Lin et al.(2014)Lin, Maire, Belongie,
Hays, Perona, Ramanan, Dollár, and Zitnick], and Objects365 [Shao et
al.(2019)Shao, Li, Zhang, Peng, Yu, Zhang, Li, and Sun]. The implementation
details and training time comparison are described in the appendix.

#### Evaluation Metrics.

Following the previous works [Gu et al.(2022)Gu, Lin, Kuo, and Cui, Du et
al.(2022)Du, Wei, Zhang, Shi, Gao, and Li, Zhou et al.(2022c)Zhou, Girdhar,
Joulin, Krähenbühl, and Misra], we use Average Precision (AP) to evaluate the
performance of our model. For LVIS, we take $\mathrm{AP}_{r}$ as the main
metric as it is the performance of model generalizing to the novel classes
$\mathcal{C}_{N}$, and also report $\mathrm{AP}_{c}$, $\mathrm{AP}_{f}$, and
$\mathrm{AP}$. For transfer experiments, we report
$\mathrm{AP},\mathrm{AP}_{50},\mathrm{AP}_{75},\mathrm{AP}_{s},\mathrm{AP}_{m}$
and $\mathrm{AP}_{l}$.

Method | KD? | Ens? | Extra data? | Detection | Instance segmentation
---|---|---|---|---|---
$\mathrm{AP}_{r}$ | $\mathrm{AP}_{c}$ | $\mathrm{AP}_{f}$ | $\mathrm{AP}$ | $\mathrm{AP}_{r}$ | $\mathrm{AP}_{c}$ | $\mathrm{AP}_{f}$ | $\mathrm{AP}$
ViLD [Gu et al.(2022)Gu, Lin, Kuo, and Cui] | yes | yes | no | 16.7 | 26.5 | 34.2 | 27.8 | 16.6 | 24.6 | 30.3 | 25.5
RegionCLIP [Zhong et al.(2022)Zhong, Yang, Zhang, Li, Codella, Li, Zhou, Dai, Yuan, Li, et al.] | no | no | CC3M | 17.1 | 27.4 | 34.0 | 28.2 | - | - | - | -
DetPro [Du et al.(2022)Du, Wei, Zhang, Shi, Gao, and Li] | yes | yes | no | 20.8 | 27.8 | 32.4 | 28.4 | 19.8 | 25.6 | 28.9 | 25.9
OV-DETR [Zang et al.(2022)Zang, Li, Zhou, Huang, and Loy] | yes | no | no | - | - | - | - | 17.4 | 25.0 | 32.5 | 26.6
PromptDet [Feng et al.(2022)Feng, Zhong, Jie, Chu, Ren, Wei, Xie, and Ma] | no | no | LAION-400M | - | - | - | - | 19.0 | 18.5 | 25.8 | 21.4
Detic [Zhou et al.(2022c)Zhou, Girdhar, Joulin, Krähenbühl, and Misra] | no | no | CC3M | - | - | - | - | 19.8 | - | - | 31.0
Rasheed _et al_[Bangalath et al.(2022)Bangalath, Maaz, Khattak, Khan, and Shahbaz Khan] | yes | no | ImageNet21k | - | - | - | - | 19.3 | 23.6 | 27.9 | 24.1
MIC (ours) | no | no | no | 22.1 | 33.9 | 40.0 | 33.8 | 20.3 | 30.6 | 35.2 | 30.6
MIC* (ours) | no | no | 100 class names | 22.9 | 34.0 | 39.9 | 34.4 | 20.8 | 30.5 | 35.4 | 30.7

Table 1: Comparison of our method with previous SOTA methods on LVIS
benchmark. Note: KD (knowledge distillation); Ens (ensemble model). *
indicates we train the prompts with 100 extra class names during MPL.

Method | Pascal VOC | COCO | Objects365
---|---|---|---
$\mathrm{AP}_{50}$ | $\mathrm{AP}_{75}$ | $\mathrm{AP}$ | $\mathrm{AP}_{50}$ | $\mathrm{AP}_{75}$ | $\mathrm{AP}_{s}$ | $\mathrm{AP}_{m}$ | $\mathrm{AP}_{l}$ | $\mathrm{AP}$ | $\mathrm{AP}_{50}$ | $\mathrm{AP}_{75}$ | $\mathrm{AP}_{s}$ | $\mathrm{AP}_{m}$ | $\mathrm{AP}_{l}$
Supervised | 78.5 | 49.0 | 46.5 | 67.6 | 50.9 | 27.1 | 67.6 | 77.7 | 25.6 | 38.6 | 28.0 | 16.0 | 28.1 | 36.7
ViLD [Gu et al.(2022)Gu, Lin, Kuo, and Cui] | 73.9 | 57.9 | 34.1 | 52.3 | 36.5 | 21.6 | 38.9 | 46.1 | 11.5 | 17.8 | 12.3 | 4.2 | 11.1 | 17.8
DetPro [Du et al.(2022)Du, Wei, Zhang, Shi, Gao, and Li] | 74.6 | 57.9 | 34.9 | 53.8 | 37.4 | 22.5 | 39.6 | 46.3 | 12.1 | 18.8 | 12.9 | 4.5 | 11.5 | 18.6
MIC (ours) | 73.0 | 58.3 | 39.2 | 56.8 | 42.2 | 27.2 | 43.1 | 51.1 | 14.0 | 20.1 | 15.2 | 6.6 | 16.6 | 24.6

Table 2: Comparison of our method with previous SOTA methods on transfer
experiments. We directly evaluate LVIS-trained model on Pascal VOC test set,
COCO validation set and Objects365 validation set, together with a supervised
baseline.

###  4.2 Main Results

#### Experiment on LVIS.

We select the most recent SOTA OVOD methods for comparison, including ViLD [Gu
et al.(2022)Gu, Lin, Kuo, and Cui], RegionCLIP [Zhong et al.(2022)Zhong, Yang,
Zhang, Li, Codella, Li, Zhou, Dai, Yuan, Li, et al.], DetPro [Du et
al.(2022)Du, Wei, Zhang, Shi, Gao, and Li], OV-DETR [Zang et al.(2022)Zang,
Li, Zhou, Huang, and Loy], PromptDet [Feng et al.(2022)Feng, Zhong, Jie, Chu,
Ren, Wei, Xie, and Ma], Detic [Zhou et al.(2022c)Zhou, Girdhar, Joulin,
Krähenbühl, and Misra], and Rasheed _et al_[Bangalath et al.(2022)Bangalath,
Maaz, Khattak, Khan, and Shahbaz Khan]. From Table 1, it can be seen that
without knowledge distillation and ensemble model, our method outperforms
DetPro by $1.3\%$ bbox $\mathrm{AP}_{r}$ and $0.5\%$ mask $\mathrm{AP}_{r}$.
Moreover, without extra training data, our method outperforms PromptDet,
Detic, and Rasheed _et al_[Bangalath et al.(2022)Bangalath, Maaz, Khattak,
Khan, and Shahbaz Khan] by $1.3\%$, $0.5\%$, and $1.0\%$ mask
$\mathrm{AP}_{r}$, respectively. We also train the prompts with extra 100
class names from ImageNet21k and find that bbox $\mathrm{AP}_{r}$ can be
further improved to $22.9\%$. Such results demonstrate the effectiveness and
robustness of our method when generalizing to novel classes.

Prompt | Strategy | Detection
---|---|---
FG | BG | MPL | ICL | $\mathrm{AP}_{r}$ | $\mathrm{AP}_{c}$ | $\mathrm{AP}_{f}$ | $\mathrm{AP}$
fixed | ✗ | ✗ | ✗ | 17.6 | 34.4 | 40.2 | 33.8
learnable | ✗ | ✗ | ✗ | 19.7 | 34.0 | 39.8 | 33.8
learnable | ✗ | ✓ | ✗ | 20.6 | 33.5 | 39.8 | 33.7
learnable | learnable | ✓ | ✗ | 21.2 | 34.0 | 39.9 | 34.1
learnable | learnable | ✓ | ✓ | 22.1 | 33.9 | 40.0 | 34.2

Table 3: Effect of different components of our approach. Note: FG (foreground
prompt); BG (background prompt).

$[L_{p}$, $L_{n}]$ | [4, 6] | [8, 10] | [16, 18]
---|---|---|---
$\mathrm{AP}_{r}$ | 25.2 | 26.4 | 25.8
$\mathrm{AP}$ | 39.3 | 40.1 | 39.7

(a) Context lengths

Position | Front | Middle | End
---|---|---|---
$\mathrm{AP}_{r}$ | 23.8 | 25.4 | 26.4
$\mathrm{AP}$ | 39.0 | 39.8 | 40.1

(b) Different positions of class token

Table 4: Learnable context study.

Figure 3: Sampling strategy in MPL. We study the effect of sampled classes.

$U_{n}$ | 0.01 | 0.05 | 0.1 | 0.01 | 0.05 | 0.1 | 0.01 | 0.05 | 0.1
---|---|---|---|---|---|---|---|---|---
$U_{p}$ | 0.7 | 0.7 | 0.7 | 0.8 | 0.8 | 0.8 | 0.9 | 0.9 | 0.9
$\mathrm{AP}_{r}$ | 26.4 | 25.1 | 26.0 | 26.0 | 24.7 | 24.3 | 25.4 | 25.2 | 24.8
$\mathrm{AP}$ | 40.1 | 40.0 | 39.8 | 39.9 | 40.1 | 40.2 | 39.7 | 39.9 | 39.4

(a) IoU threshold $U_{p}$ and $U_{n}$

$m$ | 8 | 16 | 32 | 8 | 16 | 32 | 8 | 16 | 32
---|---|---|---|---|---|---|---|---|---
$M$ | 64 | 64 | 64 | 128 | 128 | 128 | 256 | 256 | 256
$\mathrm{AP}_{r}$ | 24.3 | 24.5 | 24.6 | 26.0 | 25.5 | 24.9 | 25.7 | 26.4 | 23.9
$\mathrm{AP}$ | 39.8 | 40.2 | 39.8 | 39.9 | 39.7 | 40.2 | 39.6 | 40.1 | 39.4

(b) Batch sampling size $m$ and memory size $M$

tableSampling strategy in ICL. (a) Effect of foreground and background
instance IoU threshold $U_{p}$ and $U_{n}$. (b) Effect of batch sampling size
$m$ and memory size $M$.

#### Transfer Experiment.

We further validate the generalization ability of our method by directly
evaluating LVIS-trained model on Pascal VOC, COCO, and Objects365. We use the
learned prompt representation and class names of corresponding dataset to
generate the class embeddings. From Table 2, we can observe that our method
improves the performance by a large margin, especially on more difficult COCO
($+4.3\%\ \mathrm{AP}$) and Objects365 ($+1.9\%\ \mathrm{AP}$).

Figure 4: t-SNE visualization of class embeddings of LVIS. We randomly sample
200 novel and base classes from LVIS and use t-SNE to visualize the class
embeddings.

Figure 5: Qualitative detection visualization results of our proposed method
MIC and DetPro. Our method could better distinguish similar classes, detect
smaller objects, and produce less false positives under diverse complex
scenarios.

###  4.3 Ablation Studies

In this subsection, we do comprehensive ablation experiments on LVIS. To study
MPL and ICL schemes, we randomly sample a subset with $5$k images from LVIS
validation set for hyper-parameter selection.

#### Overall Analysis.

We study the effect of different components in our proposed framework. As
shown in Table 4, MPL helps learnable foreground prompt generalize, with
$+0.9\%$ $\mathrm{AP}_{r}$ improvement. With the learnable background prompt,
the detection performance on novel classes improves by $0.6\%$. Further, our
proposed ICL boosts the performance $\mathrm{AP}_{r}$ to $22.1\%$, surpassing
fixed prompt ($+4.5\%\ \mathrm{AP}_{r}$) and naive learnable prompt ($+2.4\%\
\mathrm{AP}_{r}$).

#### Number of Sampled Classes in MPL.

We study the model performance when trained with different number of sampled
classes in Figure 3. We range the number of classes from $200$ to $850$ in
intervals of $50$, along with a special case that only the batch classes are
reserved (about $150$ classes). We can learn that $AP_{r}$ shows a trend of
first rising and then falling as the sample size increases.

#### Context Lengths.

We study the effect of using different foreground and background context
lengths. Due to lack of class word embedding, we set $L_{n}$ always longer
than $L_{p}$ by $2$. As shown in Table 4a, too short context is not effective
while too long context may cause over-fitting. We set the foreground and
background context as $8$ and $10$, respectively.

#### Position of Class Token.

In Table 4b, we study the effect of different positions of class token in the
foreground prompt, including front, middle, and end. The best position of
class token usually depends on the dataset [Zhou et al.(2022a)Zhou, Yang, Loy,
and Liu]. We find that inserting it in the end performs best.

#### Sampling Thresholds in ICL.

Here, we study the effect of selection thresholds under the different
combinations of $U_{p}$ and $U_{n}$, as shown in Table 4a. We can learn the
following: 1) the model performs better with smaller background selection
threshold which infers few overlap between proposal and gt bounding box; 2)
the model performs best with foreground selection threshold $0.7$, which
demonstrates that too high threshold will filter out too many foreground
proposals and lead to sub-optimal solution.

#### Memory Bank and Sampling Sizes.

Further, we study the size of memory bank $M$ and batch sampling size $m$.
Intuitively, with more samples in the memory bank, the contrastive
regularization can be stronger. We consider several different combinations of
memory size $M$ and batch sampling size $m$, and the results are shown in
Table 4b. We can learn that with memory size $256$ and batch sampling size
$16$, the detection achieves the best performance.

###  4.4 Visualization Results

#### Latent Space Embedding.

To further validate the effectiveness of our proposed MPL, we randomly sample
$200$ novel and base classes from LVIS and use t-SNE [Van der Maaten and
Hinton(2008)] to visualize the class embeddings from DetPro and our MPL. From
Figure 5, it can be seen that MPL can learn more discriminative and
generalizable class embeddings, especially for novel classes.

#### Detection Results.

We show the superiority of our method by visualizing the detection results on
LVIS validation set. The visualization results are shown in Figure 5. It can
be seen that our method can better distinguish similar categories compared
with DetPro, such as sheep and lamb. Further, our method can detect smaller
objects in the complex environments. Also, with the learnable background
prompt, our model produces less false positives.

##  5 Conclusion

This paper proposes a novel framework MIC for open-vocabulary object
detection. MIC consists of two major carefully designed learning schemes, meta
prompt and instance contrastive learning. The proposed meta prompt learning
strategy simulates a novel-class-emerging scenario, together with the
learnable background prompt representation to help the generalization.
Further, we propose to use instance-level contrastive learning strategy to
help expand the low density regions in the latent feature space. Without
complex training techniques and extra training data, extensive experimental
results show the strong generalization ability of our proposed method,
especially transferring to other datasets, such as COCO and Objects365.

#### Acknowledgements.

We gratefully acknowledge the support of Mindspore, CANN (Compute Architecture
for Neural Networks) and Ascend AI Processor used for this research.

##  Appendix A Detailed Experimental Setup

Our method is implemented based Detectron2 [Wu et al.(2019)Wu, Kirillov,
Massa, Lo, and Girshick]. All the experiments are run on 8 V100 GPUs. The
overall algorithm is given in Algorithm 1.

#### Transferred Datasets.

Pascal VOC is a combination of VOC2007 and VOC2012 datasets with $20$ object
categories. This dataset contains $5$k images for test. COCO 2017 is a
classical benchmark dataset consisting of $80$ object categories. It contains
5k images for validation. Objects365 v2 is a object detection dataset with
$365$ diverse object classes in the wild. It contains $30$k images for
validation.

#### Meta Prompt Learning.

The context vectors for foreground and background are both randomly
initialized from a Gaussian distribution with $0$ mean and $0.02$ standard
deviation. And the length of context vectors for foreground and background are
set as $8$ and $10$, respectively. The subset $\boldsymbol{T}_{s}$ contains
$650$ sampled base classes by default. We use jit version CLIP as vision-
language model [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh,
Agarwal, Sastry, Askell, Mishkin, Clark, et al.] during meta prompt
representation learning. Following DetPro [Du et al.(2022)Du, Wei, Zhang, Shi,
Gao, and Li], we use $10\%$ background proposals and ground-truth foreground
proposals are included in the positive proposals for training. The word
embedding for every class is integrated in the end of the learned foreground
prompt representation. The dimension of context vector and word embedding are
both $512$. As demonstrated in DetPro, different positive proposals for the
same object can be various, which results in different contexts. Such
difference can be understood as follows: (a) given the ground-truth bounding
box of an object, the prompt should be ‘a photo of’; (b) given a proposal with
partial object, the prompt should be ‘a photo of partial’. Obviously, the
learned prompt representation should be capable to represent these two
different templates. So we train the prompt representation with different
level contexts and ensemble the learned prompt representation. The positive
proposals are divided into $5$ levels by IoU range from $0.5$ to $1.0$ with
step size $0.1$. We train the prompt representation for every single level
with $\mathcal{L}_{p}$ in Eq. (1) and $\mathcal{L}_{n}$ in Eq. (5). The
temperature $\tau$ in Eq. (3) and Eq. (4) is set as $0.01$. SGD is used for
optimizing the context vectors. The learning rate is $0.002$ and decayed by a
step-wise scheduler, trained for 6 epochs with a batch size $512$.

Algorithm 1 MIC for Open-Vocabulary Object Detection

Input: fg prompt $\boldsymbol{V}_{fg}$, bg prompt $\boldsymbol{V}_{bg}$, base
classes $\mathcal{C}_{B}$, prompt update steps $K$ and learning rate
$\eta_{k}$, detector $\theta$, training image $I$ with its bbox and class
label $(b,c)$, detector training steps $R$ and learning rate $\eta_{r}$,
hyper-parameter $\alpha$
Output: trained detector $\theta$

1:We start with procedure A to learn fg and bg prompts, then the learned
prompts are used in procedure B.

2:procedure A. Meta Prompt Learning

3: for $k=1\to K$ do

4: Sample a batch of precomputed proposals from $\mathcal{C}_{B}$

5: Forward proposals into CLIP to obtain $\boldsymbol{f}_{p}$,
$\boldsymbol{f}_{n}$

6:
$\boldsymbol{T}_{B}=\\{E_{\mathcal{T}}(\boldsymbol{V}_{i})\\}_{i\in\mathcal{C}_{B}}$,
$\boldsymbol{t}_{bg}=E_{\mathcal{T}}(\boldsymbol{V}_{bg})$

7: Sample $\boldsymbol{T}_{S}$ from $\boldsymbol{T}_{B}$ $\triangleright$ Meta
sampling

8: Compute $p^{p}_{c}$ by Eq. (3), and $p^{n}_{c}$ by Eq. (4)

9: $\mathcal{L}_{p}=-\log p^{p}_{c}$ $\triangleright$ Eq. (1)

10:
$\mathcal{L}_{\mathrm{n}}=-\frac{1}{\left|\mathcal{C}_{S}\right|}\sum_{c=1}^{\left|\mathcal{C}_{S}\right|}\log
p^{n}_{c}$ $\triangleright$ Eq. (5)

11:
$\boldsymbol{V}_{fg}\leftarrow\boldsymbol{V}_{fg}-\eta_{k}\cdot\nabla\mathcal{L}_{p}$
$\triangleright$ Update fg prompt

12:
$\boldsymbol{V}_{bg}\leftarrow\boldsymbol{V}_{bg}-\eta_{k}\cdot\nabla\mathcal{L}_{n}$
$\triangleright$ Update bg prompt

13: end for

14:end procedure

15:procedure B. Detector Training

16: for $r=1\to R$ do

17: Sample a batch of data $(I,(b,c))$ from $\mathcal{C}_{B}$

18: Feed $I$ into detector to obtain $\boldsymbol{f}$ and proposal IoU

19: Filter fg proposal by $U_{p}$ and bg proposal by $U_{n}$

20: Update instance memory bank $\mathcal{Q}$

21: Calculate contrastive loss $\mathcal{L}_{icl}$ by Eq. (6)

22:
$\mathcal{L}_{det}=\mathcal{L}_{rpn}+\mathcal{L}_{cls}+\mathcal{L}_{reg}+\alpha\mathcal{L}_{icl}$

23: $\theta\leftarrow\theta-\eta_{r}\cdot\nabla\mathcal{L}_{det}$
$\triangleright$ Update detector $\theta$

24: end for

25:end procedure

#### Open-Vocabulary Detector.

We use ResNet50 [He et al.(2016)He, Zhang, Ren, and Sun] as the backbone and
FPN [Lin et al.(2017a)Lin, Dollár, Girshick, He, Hariharan, and Belongie]
architecture. The default size $M$ of memory bank for each class is $256$ and
sampling size $m$ is $16$. The threshold for selecting proposals $U_{p}$ and
$U_{n}$ are $0.7$ and $0.01$, respectively. The temperature $\gamma$ in Eq.
(6) is set as $0.1$ and the weight $\alpha$ of $\mathcal{L}_{icl}$ is
initialized as 0.1 and gradually decayed during the detector training to help
the convergence of other losses. Following Detic [Zhou et al.(2022c)Zhou,
Girdhar, Joulin, Krähenbühl, and Misra], we use CenterNet2 [Zhou et
al.(2021)Zhou, Koltun, and Krähenbühl] with a ResNet50 backbone [He et
al.(2016)He, Zhang, Ren, and Sun] and FPN [Lin et al.(2017a)Lin, Dollár,
Girshick, He, Hariharan, and Belongie] architecture. The mask prediction head
is modified to a class-agnostic one and federated loss [Zhou et al.(2021)Zhou,
Koltun, and Krähenbühl] is used for training. Repeat factor sampling [Gupta et
al.(2019)Gupta, Dollar, and Girshick] is used to balance long-tailed
distributed classes. Note that CenterNet2 uses a cascade classifier [Cai and
Vasconcelos(2018)]. The backbone ResNet50 is initialized with the pretrained
weights on ImageNet21k [Ridnik et al.(2021)Ridnik, Ben-Baruch, Noy, and
Zelnik-Manor]. In ICL, the projection network is a $2$-layer MLP with two
linear-relu layers and a normalization layer, and the dimension of proposal
feature is projected from $1024$ to $128$. To improve the diversity of
proposal samples in the instance memory bank, we gather and concat all
proposal samples from $8$ GPUs before updating the instance memory bank. We
use AdamW [Loshchilov and Hutter(2019)] as the optimizer with an initial
learning rate $0.0002$ and batch size $64$. We use EfficientDet style large
scale jittering [Ghiasi et al.(2021)Ghiasi, Cui, Srinivas, Qian, Lin, Cubuk,
Le, and Zoph, Tan et al.(2020)Tan, Pang, and Le]. To accelerate training, the
input images are cropped to $640\times 640$ while $800\times 1333$ for
inference. The detector is trained for a $4\times$ schedule with $90000$
iterations. We perform $10000$ warmup steps with $0.0001$ warmup factor and
use cosine learning schedule [Loshchilov and Hutter(2016)]. To help the
convergence of other losses, the weight $\alpha$ of instance-level contrastive
loss $\mathcal{L}_{icl}$ is decayed along with the training process by
$\alpha=\alpha\times(1-iter/90000)$.

##  Appendix B More Experimental Results

In this section, we show more experimental results, including the comparison
of training time, visualization of latent space embedding and detection, which
further indicates the effectiveness and robustness of our proposed method.

#### Training Time.

We compare the training time of our proposed MIC with previous SOTA two-stage
methods, including DetPro [Du et al.(2022)Du, Wei, Zhang, Shi, Gao, and Li],
RegionCLIP [Zhong et al.(2022)Zhong, Yang, Zhang, Li, Codella, Li, Zhou, Dai,
Yuan, Li, et al.], PromptDet [Feng et al.(2022)Feng, Zhong, Jie, Chu, Ren,
Wei, Xie, and Ma], as shown in Table 5. Our MIC is shown to be more efficient
than previous two-stage methods.

Method | RegionCLIP [Zhong et al.(2022)Zhong, Yang, Zhang, Li, Codella, Li, Zhou, Dai, Yuan, Li, et al.] | DetPro [Du et al.(2022)Du, Wei, Zhang, Shi, Gao, and Li] | PromptDet [Feng et al.(2022)Feng, Zhong, Jie, Chu, Ren, Wei, Xie, and Ma] | MIC (ours)
---|---|---|---|---
Training Time (GPU hours) | 1064 | 464 | 408 | 368

Table 5: The training time comparison of our method with previous SOTA two-
stage methods.

#### Latent Space Embedding.

We also use t-SNE to visualize the class embeddings of Pascal VOC, COCO, and
Objects365 generated from DetPro and our proposed MPL. From Figure 6, we can
draw the same conclusion as LVIS. Under our proposed MPL scheme, the learned
prompt representations are more discriminative in the latent space.

Figure 6: t-SNE visualization of class embeddings of transferred datasets. We
use t-SNE to visualize the class embeddings of Pascal VOC, COCO, and
Objects365 generated from DetPro and our proposed MPL.

#### Study of Failure Cases.

Although we use meta prompt and instance contrastive learning to improve the
discriminative ability of our model, it still suffers from distinguishing some
extremely similar classes, such as ‘duck’ and ‘duckling’, and ‘panda’ and
‘bear’ shown in Figure 7. To better distinguish these categories, we might
include outside knowledge base of fine-grained categories in the future work.

Figure 7: Failure cases of MIC.

#### More Comparisons of Detection Results.

We show more detection visualization results in Figure 8, which further
indicates the robustness of our proposed method.

Figure 8: More qualitative detection visualization results of our proposed
method MIC and DetPro. Our method could better distinguish similar classes,
detect smaller objects, and produce less false positives under diverse scenes.

## References

  * [Bangalath et al.(2022)Bangalath, Maaz, Khattak, Khan, and Shahbaz Khan] Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman H Khan, and Fahad Shahbaz Khan.  Bridging the gap between object and image-level representations for open-vocabulary detection.  _Advances in Neural Information Processing Systems_ , 35:33781–33794, 2022.
  * [Cai and Vasconcelos(2018)] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6154–6162, 2018.
  * [Carion et al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European conference on computer vision_ , pages 213–229. Springer, 2020.
  * [Caron et al.(2020)Caron, Misra, Mairal, Goyal, Bojanowski, and Joulin] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in Neural Information Processing Systems_ , 33:9912–9924, 2020.
  * [Chen et al.(2020)Chen, Kornblith, Norouzi, and Hinton] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009.
  * [Du et al.(2022)Du, Wei, Zhang, Shi, Gao, and Li] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li.  Learning to prompt for open-vocabulary object detection with vision-language model.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [Everingham et al.(2010)Everingham, Van Gool, Williams, Winn, and Zisserman] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  _International journal of computer vision_ , 88(2):303–338, 2010.
  * [Feng et al.(2022)Feng, Zhong, Jie, Chu, Ren, Wei, Xie, and Ma] Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma.  Promptdet: Towards open-vocabulary detection using uncurated images.  In _Proceedings of the European Conference on Computer Vision_ , 2022.
  * [Ghiasi et al.(2021)Ghiasi, Cui, Srinivas, Qian, Lin, Cubuk, Le, and Zoph] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph.  Simple copy-paste is a strong data augmentation method for instance segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2918–2928, 2021.
  * [Grandvalet and Bengio(2004)] Yves Grandvalet and Yoshua Bengio.  Semi-supervised learning by entropy minimization.  _Advances in neural information processing systems_ , 17, 2004.
  * [Grill et al.(2020)Grill, Strub, Altché, Tallec, Richemond, Buchatskaya, Doersch, Avila Pires, Guo, Gheshlaghi Azar, et al.] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 33:21271–21284, 2020.
  * [Gu et al.(2022)Gu, Lin, Kuo, and Cui] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=lL3lnMbR4WU.
  * [Gupta et al.(2019)Gupta, Dollar, and Girshick] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5356–5364, 2019.
  * [Gupta et al.(2022)Gupta, Narayan, Joseph, Khan, Khan, and Shah] Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah.  Ow-detr: Open-world detection transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9235–9244, 2022.
  * [Han et al.(2022)Han, Ren, Ding, Pan, Yan, and Xia] Jiaming Han, Yuqiang Ren, Jian Ding, Xingjia Pan, Ke Yan, and Gui-Song Xia.  Expanding low-density latent regions for open-set object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9591–9600, 2022.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * [He et al.(2017)He, Gkioxari, Dollár, and Girshick] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * [He et al.(2020)He, Fan, Wu, Xie, and Girshick] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738, 2020.
  * [Huang et al.(2022)Huang, Han, Cheng, and Zhang] Peiliang Huang, Junwei Han, De Cheng, and Dingwen Zhang.  Robust region feature synthesizer for zero-shot object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7622–7631, 2022.
  * [Jia et al.(2021)Jia, Yang, Xia, Chen, Parekh, Pham, Le, Sung, Li, and Duerig] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , pages 4904–4916. PMLR, 2021.
  * [Joseph et al.(2021)Joseph, Khan, Khan, and Balasubramanian] KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian.  Towards open world object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5830–5840, 2021.
  * [Khosla et al.(2020)Khosla, Teterwak, Wang, Sarna, Tian, Isola, Maschinot, Liu, and Krishnan] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  _Advances in Neural Information Processing Systems_ , 33:18661–18673, 2020.
  * [Li et al.(2022)Li, Zhang, Zhang, Yang, Li, Zhong, Wang, Yuan, Zhang, Hwang, et al.] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.  Grounded language-image pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10965–10975, 2022.
  * [Lin et al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer, 2014.
  * [Lin et al.(2017a)Lin, Dollár, Girshick, He, Hariharan, and Belongie] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2117–2125, 2017a.
  * [Lin et al.(2017b)Lin, Goyal, Girshick, He, and Dollár] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2980–2988, 2017b.
  * [Loshchilov and Hutter(2016)] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ , 2016.
  * [Loshchilov and Hutter(2019)] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=Bkg6RiCqY7.
  * [Meng et al.(2021)Meng, Chen, Fan, Zeng, Li, Yuan, Sun, and Wang] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang.  Conditional detr for fast training convergence.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3651–3660, 2021.
  * [Minderer et al.(2022)Minderer, Gritsenko, Stone, Neumann, Weissenborn, Dosovitskiy, Mahendran, Arnab, Dehghani, Shen, et al.] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al.  Simple open-vocabulary object detection with vision transformers.  _Proceedings of the European Conference on Computer Vision_ , 2022.
  * [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al.] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR, 2021.
  * [Ren et al.(2015)Ren, He, Girshick, and Sun] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28, 2015.
  * [Ridnik et al.(2021)Ridnik, Ben-Baruch, Noy, and Zelnik-Manor] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor.  Imagenet-21k pretraining for the masses.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_ , 2021.
  * [Schuhmann et al.(2021)Schuhmann, Vencu, Beaumont, Kaczmarczyk, Mullis, Katta, Coombes, Jitsev, and Komatsuzaki] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * [Shao et al.(2019)Shao, Li, Zhang, Peng, Yu, Zhang, Li, and Sun] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.  Objects365: A large-scale, high-quality dataset for object detection.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 8430–8439, 2019.
  * [Sharma et al.(2018)Sharma, Ding, Goodman, and Soricut] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565, 2018.
  * [Su et al.(2022)Su, Lin, Hao, Cao, Wang, and Wu] Yukun Su, Guosheng Lin, Yun Hao, Yiwen Cao, Wenjun Wang, and Qingyao Wu.  Self-supervised object localization with joint graph partition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 2289–2297, 2022.
  * [Sun et al.(2021)Sun, Li, Cai, Yuan, and Zhang] Bo Sun, Banghuai Li, Shengcai Cai, Ye Yuan, and Chi Zhang.  Fsce: Few-shot object detection via contrastive proposal encoding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7352–7362, 2021.
  * [Tan et al.(2020)Tan, Pang, and Le] Mingxing Tan, Ruoming Pang, and Quoc V Le.  Efficientdet: Scalable and efficient object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10781–10790, 2020.
  * [Tang et al.(2021)Tang, Ramaiah, Wang, Xu, and Xiong] Peng Tang, Chetan Ramaiah, Yan Wang, Ran Xu, and Caiming Xiong.  Proposal learning for semi-supervised object detection.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 2291–2301, 2021.
  * [Tian et al.(2019)Tian, Shen, Chen, and He] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9627–9636, 2019.
  * [Van der Maaten and Hinton(2008)] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.
  * [Wang et al.(2022)Wang, Zhu, Chen, Zhao, Yu, Wang, and Tang] Tong Wang, Yousong Zhu, Yingying Chen, Chaoyang Zhao, Bin Yu, Jinqiao Wang, and Ming Tang.  C2am loss: Chasing a better decision boundary for long-tail object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6980–6989, 2022.
  * [Wu et al.(2019)Wu, Kirillov, Massa, Lo, and Girshick] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [Zang et al.(2022)Zang, Li, Zhou, Huang, and Loy] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy.  Open-vocabulary detr with conditional matching.  In _European Conference on Computer Vision_ , 2022.
  * [Zhong et al.(2022)Zhong, Yang, Zhang, Li, Codella, Li, Zhou, Dai, Yuan, Li, et al.] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.  Regionclip: Region-based language-image pretraining.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16793–16803, 2022.
  * [Zhou et al.(2003)Zhou, Bousquet, Lal, Weston, and Schölkopf] Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf.  Learning with local and global consistency.  _Advances in neural information processing systems_ , 16, 2003.
  * [Zhou et al.(2022a)Zhou, Yang, Loy, and Liu] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 2022a.
  * [Zhou et al.(2022b)Zhou, Yang, Loy, and Liu] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022b.
  * [Zhou et al.(2021)Zhou, Koltun, and Krähenbühl] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Probabilistic two-stage detection.  _arXiv preprint arXiv:2103.07461_ , 2021.
  * [Zhou et al.(2022c)Zhou, Girdhar, Joulin, Krähenbühl, and Misra] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra.  Detecting twenty-thousand classes using image-level supervision.  _European Conference on Computer Vision_ , 2022c.
  * [Zhu et al.(2021)Zhu, Su, Lu, Li, Wang, and Dai] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable {detr}: Deformable transformers for end-to-end object detection.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=gZ9hCDWe6ke.