# Retrieval-Augmented Open-Vocabulary Object Detection

Jooyeon Kim1, Eulrang Cho2,00footnotemark: 0 , Sehyung Kim1 Hyunwoo J. Kim1,
1Department of Computer Science and Engineering, Korea University 2Samsung
Research
{parang, shkim129<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal
contribution.This work was done when she was working at Korea
University.Corresponding author.

###### Abstract

Open-vocabulary object detection (OVD) has been studied with Vision-Language
Models (VLMs) to detect novel objects beyond the pre-trained categories.
Previous approaches improve the generalization ability to expand the knowledge
of the detector, using ‘positive’ pseudo-labels with additional ‘class’ names,
e.g., sock, iPod, and alligator. To extend the previous methods in two
aspects, we propose Retrieval-Augmented Losses and visual Features (RALF). Our
method retrieves related ‘negative’ classes and augments loss functions. Also,
visual features are augmented with ‘verbalized concepts’ of classes, e.g.,
worn on the feet, handheld music player, and sharp teeth. Specifically, RALF
consists of two modules: Retrieval Augmented Losses (RAL) and Retrieval-
Augmented visual Features (RAF). RAL constitutes two losses reflecting the
semantic similarity with negative vocabularies. In addition, RAF augments
visual features with the verbalized concepts from a large language model
(LLM). Our experiments demonstrate the effectiveness of RALF on COCO and LVIS
benchmark datasets. We achieve improvement up to 3.4 box
AP${}_{50}^{\text{N}}$ on novel categories of the COCO dataset and 3.6 mask
AP${}_{\text{r}}$ gains on the LVIS dataset. Code is available at
https://github.com/mlvlab/RALF.

##  1 Introduction

Open-vocabulary object detection (OVD) aims to detect objects belonging to
open-set categories. It is a challenging task because novel categories do not
appear during training. Pre-trained vision-language models (VLMs) enable the
detector to recognize base categories as well as novel categories via their
zero-shot visual recognition ability learned from large-scale image-text pairs
from the Internet. For instance, CLIP [20] and ALIGN [13] are widely used in
OVD to classify unseen objects [9, 29, 17].

(a) Retrieving hard (similar) and easy negative (dissimilar) classes.

(b) Verbalized concepts to augment visual features.

Figure 1:  Negative vocabularies and verbalized concepts from a large
vocabulary set. (a) Example of negative vocabularies that can be derived from
a large vocabulary set. From the vocabulary set, ‘cat’ and ‘bottle’ can be
retrieved as hard negative (similar) and easy negative (dissimilar)
vocabulary, given the category ‘jaguar’. (b) Example of verbalized concepts
that are generated from LLMs. The concepts of the objects provide more
detailed information about the object, such as the attributes.

Knowledge distillation is one approach to transfer the knowledge of VLMs to
the detector. It has been extensively explored in the literature [9, 24, 29,
37, 6]. For more effective knowledge distillation, some works have endeavored
to match words at the region level rather than at the image level to ensure
better alignment [24, 29, 37]. To improve generalization to novel categories,
several studies employ pseudo-labeling to expand detector knowledge [37, 7,
36, 4]. Pseudo-labels (positive classes) are generated for region proposals by
matching their visual features with words from additional vocabulary sets or
captions. So, the additional vocabulary sets are constructed only focusing on
‘positive’ classes. We believe that more diverse vocabulary sets, such as
‘negative’ classes and verbalized (visual) concepts with new techniques, will
open the door to further improving OVD frameworks.

To study the underexplored directions, in this paper we propose methods
utilizing ‘negative’ classes and ‘verbalized concepts’ to more effectively use
vocabulary sets to enhance the generalization ability to novel categories. One
approach involves retrieving vocabularies that have semantic relationships
from large vocabulary sets when given a specific category. For example, given
the category ‘jaguar’, we retrieve ‘cat’ and ‘bottle’ as hard (similar) and
easy (dissimilar) negative vocabulary, respectively as shown in Figure 1(a).
Using these negative classes, we augment loss functions. Another approach is
generating verbalized concepts that describe the attributes (_e.g_., sharp
teeth) of classes by LLMs, as depicted in Figure 1(b). This information is
useful in learning effective representations.

We present a novel framework, Retrieval-Augmented Losses and visual Features
(RALF), which retrieves vocabularies and concepts from a large vocabulary set
and augments losses and visual features. RALF is composed of two parts:
Retrieval-Augmented Losses (RAL) and Retrieval-Augmented visual Features
(RAF). Given the ground-truth label, we construct hard and easy negative
vocabularies by retrieving from the vocabulary store based on similarity.
Then, RAL optimizes the distance between the ground-truth label and pre-
defined vocabularies with additional losses. Additionally, we utilize a large
language model (LLM) to obtain abundant information rather than word units.
After generating descriptions about large vocabularies from LLM, we extract
verbalized concept details that represent the characteristics of the objects
and pile them in a concept store. During inference time, RAF augments visual
features with the verbalized concepts retrieved from the concept store. Then
the enhanced features are used in classification. To validate the
effectiveness of RALF, we conducted experiments on COCO and LVIS benchmarks.
Overall, RALF improves the generalization ability of the detector.

Our contributions are threefold:

  * •

We present RALF that retrieves vocabularies and augments losses and visual
features to improve the generalizability of open-vocabulary object detectors.

  * •

RAL optimizes an embedding space by reflecting the distance between ground-
truth labels and negative vocabularies from a large vocabulary set. And RAF
augments visual features with verbalized concepts relevant to visual
attributes in images.

  * •

With the method, we achieve 41.3 AP${}^{\text{N}}_{50}$ in novel categories on
the COCO benchmark and also reach 21.9 mask AP${}_{\text{r}}$ in novel
categories on LVIS.

(a) Training pipeline w/RAL.

(b) RAF training.

(c) Inference pipeline w/RAF.

Figure 2: Overall pipeline of RALF. (a) The first module, RAL, is utilized
during detector training. Given a ground-truth box $b$, the ground-truth box
embedding $\boldsymbol{e}_{b}$ is extracted and used to define
$\mathcal{L}^{\text{RAL}}$, which is augmented with hard and easy negative
vocabulary. The augmented loss $\mathcal{L}^{\text{RAL}}$ and the baseline
loss $\mathcal{L}^{\text{baseline}}$ are employed together to train the
detector. The illustration of the other branches (_e.g_., box regression,
distillation, and mask prediction) is omitted in both training and inference
pipelines. (b) The second module, RAF, augments visual features with
verbalized concepts and is pre-trained before being used in the inference
pipeline. Augmented visual features $\boldsymbol{v}_{r}^{\text{aug}}$ are
created through a process involving concept retriever and augmenter, using
visual features $\boldsymbol{v}_{r}$ generated from object proposals in
offline. RAF is trained with two losses ($\mathcal{L}^{\text{cls}}$ and
$\mathcal{L}^{\text{reg}}$), utilizing $\boldsymbol{v}_{r}$,
$\boldsymbol{v}_{r}^{\text{aug}}$, and $\tilde{y}_{r}$, which is the pseudo-
label of visual feature. (c) During detector inference time, the trained RAF
is utilized. Classification logits $\boldsymbol{l}_{r}$ trained by RAL and
auxiliary logits $\boldsymbol{l}_{r}^{\text{aux}}$ influenced by RAF are
computed with text embeddings of test categories. Then, the final logits
$\boldsymbol{l}_{r}^{\text{final}}$ are determined through an ensemble of
$\boldsymbol{l}_{r}$ and $\boldsymbol{l}_{r}^{\text{aux}}$.

##  2 Related works

Pre-trained vision-language model. Pre-trained VLMs such as CLIP [20] and
ALIGN [13] are trained on large image-text pair datasets via contrastive
learning for joint representation in visual and language modalities. Pre-
trained VLMs consist of two encoders - image encoder and text encoder -
extracting image embedding and text embedding, respectively. By applying
contrastive learning, they can be aligned with each other in the same latent
space. For this reason, pre-trained VLMs have superior generalizability and
transferability to various downstream tasks. For example, CLIP [20] which is
pre-trained on extensive image-text pair data, shows impressive performance on
zero-shot image classification tasks. Recently, thanks to the success of CLIP,
there have been many studies for introducing VLMs to various downstream tasks
such as image segmentation [38, 23, 16], image generation [30, 21], and object
detection [9, 29, 24, 31, 17].

Open-vocabulary object detection. Object detection task refers to a task that
detects an object in a scene and classifies the detected object. A
representative study, Fast R-CNN [8], shows excellent object detection
performance with CNN architecture. However, there is a limit to the object
detection task that requires a lot of human cost for annotation. A zero-shot
object detection approach was presented to determine whether a detector can
detect a category that was not seen during learning. Recently, open-vocabulary
object detection (OVD) [31, 29, 32] has attracted attention. OVD evaluates the
ability to predict novel categories by learning using additional caption data
such as CC3M [26]. As pre-trained VLMs trained with large datasets perform
well with zero-shot for various downstream tasks, diverse approaches to solve
through pre-trained VLMs have been studied in OVD. ViLD [9], the most
representative study, uses knowledge from CLIP, one of the pre-trained VLMs,
and learns the class-agnostic region proposal to perform well for unseen
classes. Many follow-up studies have also been conducted on this, and Object-
Centric-OVD [24] proposes an object-centric alignment method to solve the
localization problem that occurs when CLIP is applied to OVD.

Retrieval-augmentation. Retrieval augmentation was initially introduced in
language generation tasks for parameter efficiency. RAG [15] introduces
generation models that combine parametric and non-parametric memory access.
Recently, retrieval augmentation has been utilized in many vision tasks [19,
33, 27]. RDMs [1] suggest efficiently storing an image database and
conditioning a relatively compact generative model. EXTRA [22] proposes a
retrieval-augmented image captioning model that enhances performance by
leveraging cross-modal representations. Unlike these methods employing
retrieval augmentation in generation tasks, we applied retrieval augmentation
to the OVD task for the first time in our knowledge.

##  3 Method

In this section, we proposed a new framework RALF that Retrieves information
from a large vocabulary store and Augments Losses and visual Features. Before
we delve into RALF, we briefly introduce the open-vocabulary object detection
task in Section 3.1. The overall pipeline of the proposed method is described
in Section 3.2. Our method consists of two modules: 1) Retrieval-Augmented
Losses (RAL) to train the object detector, and 2) Retrieval-Augmented visual
Features (RAF) using the generated concepts by a large language model. The
modules are presented in Section 3.3 and Section 3.4, respectively.

###  3.1 Preliminaries

Open-vocabulary object detection (OVD) is an advanced visual recognition task
that extends the capability of traditional object detectors beyond pre-trained
categories. OVD aims to _localize_ and _classify_ a wide range of objects,
including the categories not encountered during training. In OVD, the pre-
trained categories and unseen categories are called base categories
$C^{\mathcal{B}}$ and novel categories $C^{\mathcal{N}}$, respectively. In
general, the approaches for OVD leverage a pre-trained region proposal method,
_e.g_., Region Proposal Network (RPN), for category-agnostic (initial)
localization. After the region proposal step, OVD methods utilize a pre-
trained vision-language model to classify a wide range of categories in a
zero-shot learning manner. Specifically, given region proposal $r$, the
methods extract a region embedding $\boldsymbol{e}_{r}\in\mathbb{R}^{d}$ and
compute the similarity or perform zero-shot classification with the text
embeddings of category $\mathcal{T}(c)\in\mathbb{R}^{d}$, where $\mathcal{T}$
is the text encoder and $c$ is either a base or new category, _i.e_., $c\in
C^{\mathcal{B}}\cup C^{\mathcal{N}}$. Unless explicitly stated, vectors are
row vectors in this paper.

###  3.2 Overview of RALF

We present the overall pipeline of Retrieval-Augmented Losses and visual
Features (RALF). As described in Figure 2, RALF consists of two modules: 1)
RAL (Figure 2(a)), which retrieves negative vocabularies from the vocabulary
store by semantic similarity, and enhances the loss function for training
object detectors, and 2) RAF (Figure 2(b)), which augments visual features
using verbalized concepts by a large language model given retrieved
vocabulary.

In RAL, we define two negative vocabulary sets $V_{y_{b}}^{\text{hard}}$ and
$V_{y_{b}}^{\text{easy}}$ to train the object detector. They are retrieved
based on the semantic similarity score between the ground-truth class label
$y_{b}$ and vocabularies from the external vocabulary store $C^{\mathcal{V}}$.
In RAF, we extract the concepts associated with objects by LLM and retrieve
them to augment visual feature $\boldsymbol{v}_{r}\in\mathbb{R}^{d}$ by using
the augmenter $\mathcal{A}$ for better classification.

Following the conventional OVD setup, our detector is trained with annotated
bounding boxes of base categories $C^{\mathcal{B}}$. As shown in Figure 2(a),
retrieval-augmented losses are additionally used for training. The total loss
for training the object detector is defined as follows:

$\mathcal{L}^{\text{total}}=\mathcal{L}^{\text{baseline}}+\mathcal{L}^{\text{RAL}},$
(1)

where $\mathcal{L}^{\text{baseline}}$ denotes the loss of the baseline and
$\mathcal{L}^{\text{RAL}}$ is explained in Section 3.3. We utilize the
obtained conceptual information to train the RAF module independently. As
illustrated in Figure 2(c), RAF augments the visual feature in a plug-in
manner during inference time. Detailed explanations of RAL and RAF will be
introduced in Section 3.3 and Section 3.4, respectively.

Figure 3: RAL. Given ground-truth class label $y_{b}$, negative retriever
extracts hard negative vocabulary $V_{y_{b}}^{\text{hard}}$ and easy negative
vocabulary $V_{y_{b}}^{\text{easy}}$ based on semantic similarity with
$\mathcal{T}(y_{b})$. To enhance the generalizability of the detector, two
triplet losses (_i.e_. hard negative loss $\mathcal{L}^{\text{hard}}$ and easy
negative loss $\mathcal{L}^{\text{easy}}$) are augmented with
$V_{y_{b}}^{\text{hard}}$, $V_{y_{b}}^{\text{easy}}$, and ground-truth box
embedding $\boldsymbol{e}_{b}$.

###  3.3 Retrieval-Augmented Losses

In our method, we introduce Retrieval-Augmented Losses (RAL), a novel
framework utilizing a large vocabulary store to enhance the detector’s
generalization power across both base and novel object categories. Our
approach, as described in Figure 3, involves the creation of distinct negative
vocabulary sets – categorized as ‘hard’ and ‘easy’ – based on their
similarities to the ground-truth category. RALF trains the detector with RAL,
which is established with triplet losses between hard and easy negative
vocabulary and ground-truth box.

Negative retriever. To derive hard and easy negative vocabularies, we exploit
a large vocabulary set containing a wider range of object classes. First, we
remove redundant categories from a vocabulary set and refine the vocabulary
store $C^{\mathcal{V}}$ to prevent the possibility that the detector may see
novel categories $C^{\mathcal{N}}$, _i.e_., $C^{\mathcal{V}}\cap
C^{\mathcal{N}}=\varnothing$. Then, we define hard negative vocabulary
$V_{y_{b}}^{\text{hard}}$ and easy negative vocabulary
$V_{y_{b}}^{\text{easy}}$ using relative similarity. Specifically, given a
ground-truth class label $y_{b}$, we obtain text embedding
$\mathcal{T}(y_{b})\in\mathbb{R}^{d}$. The negative retriever samples hard and
easy negative vocabularies from $C^{\mathcal{V}}$ with respect to cosine
similarity between $\mathcal{T}(y_{b})$ and $\mathcal{T}(C^{\mathcal{V}})$.
However, we observed that some vocabularies have constantly high (or low)
similarity scores for any base category. In this case, the vocabularies are
not useful to augment losses. To mitigate this issue, we adopt the rank
variance sampling scheme, which filters out the vocabularies based on the
variance of their rankings measured by similarity. Specifically, we first
measure the similarity between a base category $c\in C^{\mathcal{B}}$ and all
the vocabularies in $C^{\mathcal{V}}$. Then, the ranking of
$C^{\mathcal{V}}_{i}$ given $c$ is defined as:

$\underset{c\in
C^{\mathcal{B}}}{\text{rank}_{c}(C_{i}^{\mathcal{V}})}=1+\sum_{j\neq
i}\mathbbm{1}[\mathcal{T}(c)\mathcal{T}(C^{\mathcal{V}}_{j})^{\top}>\mathcal{T}(c)\mathcal{T}(C^{\mathcal{V}}_{i})^{\top}].$
(2)

We finally compute the variance of $\text{rank}_{c}(C_{i}^{\mathcal{V}})$
using all $C^{\mathcal{B}}$. Vocabularies with a relatively low variance of
rankings are removed. Then, top-$m$ and bottom-$m$ vocabularies based on the
similarity score are selected for each base category as hard and easy negative
vocabularies, respectively. For each iteration of training, $n$ vocabularies
are randomly selected among the $m$ vocabularies to augment losses. Further
details of sampling schemes are discussed in the supplementary materials.

Hard and easy negative losses. Hard and easy negative vocabularies are denoted
as $V_{y_{b}}^{\text{hard}}$ and $V_{y_{b}}^{\text{easy}}$, respectively.
$V_{y_{b}}^{\text{hard}}$ consists of the words similar to ground-truth
$y_{b}$ class whereas $V_{y_{b}}^{\text{easy}}$ is the least similar words to
$V_{y_{b}}^{\text{hard}}$. Using the two sets of negative vocabularies with
the triplet loss, we propose hard negative loss $\mathcal{L}^{\text{hard}}$
and easy negative loss $\mathcal{L}^{\text{easy}}$. Specifically, we first
define the average cosine similarity with ground-truth embedding
$\boldsymbol{e}_{b}$ as below:

$\displaystyle
U_{y_{b}}^{\text{hard}}:=\cfrac{1}{n}\;\mathbf{1}\mathcal{T}(V^{\text{hard}}_{y_{b}})\boldsymbol{e}_{b}^{\top},$
(3) $\displaystyle
U_{y_{b}}^{\text{easy}}:=\cfrac{1}{n}\;\mathbf{1}\mathcal{T}(V^{\text{easy}}_{y_{b}})\boldsymbol{e}_{b}^{\top},$
(4)

where $\mathbf{1}\in\mathbb{R}^{n}$ given $n$ vocabularies. Then, the hard
negative loss $\mathcal{L}^{\text{hard}}$ and easy negative loss
$\mathcal{L}^{\text{easy}}$ are defined as follows:

$\displaystyle\mathcal{L}^{\text{hard}}=\max\left(\lambda^{\text{hard}}U^{\text{hard}}_{y_{b}}-\mathcal{T}(y_{b})\boldsymbol{e}_{b}^{\top}+\alpha^{\text{hard}},0\right),$
(5)
$\displaystyle\mathcal{L}^{\text{easy}}=\max\left(\lambda^{\text{easy}}U^{\text{easy}}_{y_{b}}-U^{\text{hard}}_{y_{b}}+\alpha^{\text{easy}},0\right),$
(6)

where $\lambda^{\text{hard}}$ and $\lambda^{\text{easy}}$ are hyperparameters,
and $\alpha^{\text{hard}}$ and $\alpha^{\text{easy}}$ denote margins. To sum
up, hard negative loss $\mathcal{L}^{\text{hard}}$ encourages
$\boldsymbol{e}_{b}$ to have higher similarity with $y_{b}$ than
$V_{y_{b}}^{\text{hard}}$. Easy negative loss $\mathcal{L}^{\text{easy}}$
prompts $V_{y_{b}}^{\text{hard}}$ to exhibit relatively higher similarity to
$y_{b}$ than $V_{y_{b}}^{\text{easy}}$. The total loss
$\mathcal{L}^{\text{RAL}}$ is computed with a summation of hard negative loss
and easy negative loss as follows:

$\mathcal{L}^{\text{RAL}}=\beta^{\text{hard}}\mathcal{L}^{\text{hard}}+\beta^{\text{easy}}\mathcal{L}^{\text{easy}},$
(7)

where $\beta^{\text{hard}}$ and $\beta^{\text{easy}}$ are hyperparameters.

###  3.4 Retrieval-Augmented visual Features

We introduce Retrieval-Augmented visual Features (RAF) that augments visual
features using the _verbalized concepts_ of each object category, as depicted
in Figure 4.

Concept store. The concept store consists of a set of characteristics
describing the object (_e.g_., color, scale, shape, etc.). The characteristics
for $C^{\mathcal{B}}\cup C^{\mathcal{V}}$ are generated by a large language
model (LLM) using ‘Describe what a(n) {vocabulary} looks like.’ prompt
template, following [14]. Note that $C^{\mathcal{N}}$ is not used to generate
verbalized concepts. We remove meaningless words such as prepositions from
descriptions generated by LLM and store only meaningful noun chunks in the
concept store.

Concept retriever. Concepts for augmenting visual features are retrieved by
the concept retriever. Concept embeddings $H$ are obtained from the text
encoder $\mathcal{T}$ with the concepts from the concept store. Given visual
feature $\boldsymbol{v}_{r}\in\mathbb{R}^{d}$, the concept retriever
calculates the cosine similarity between concept embeddings $H$ and visual
feature $\boldsymbol{v}_{r}$. Then, it returns the $k$ most relevant concept
embeddings $H_{r}\in\mathbb{R}^{k\times d}$ and corresponding scores
$\boldsymbol{s}_{r}\in\mathbb{R}^{k}$.

Figure 4: RAF. Verbalized concepts are generated by LLM with prompts and
stored in the concept store. Given a visual feature $\boldsymbol{v}_{r}$,
relevant concept embeddings $H_{r}$ and scores $\boldsymbol{s}_{r}$ are
retrieved by the concept retriever. Then the augmenter $\mathcal{A}$ creates
augmented visual feature $\boldsymbol{v}_{r}^{\text{aug}}$ with related
verbalized concepts.

Augmenter. We propose an augmenter $\mathcal{A}$ that augments visual feature
$\boldsymbol{v}_{r}$ with the retrieved concepts. Let
$\boldsymbol{v}_{r}^{\text{aug}}\in\mathbb{R}^{d}$ denote a combination of
$\boldsymbol{v}_{r}^{\text{coarse}}\in\mathbb{R}^{d}$ and
$\boldsymbol{v}_{r}^{\text{fine}}\in\mathbb{R}^{d}$.
$\boldsymbol{v}_{r}^{\text{coarse}}$ is calculated as:

$\boldsymbol{v}_{r}^{\text{coarse}}=\text{Proj}(\boldsymbol{v}_{r}),$ (8)

where $\text{Proj}(\cdot)$ is a linear projection. On the other hand,
$\boldsymbol{v}_{r}^{\text{fine}}$ is enhanced with the retrieved concepts. It
is the final output of the decoder that has query embedding
$\boldsymbol{q}_{1}\in\mathbb{R}^{d}$ as query and
$M\in\mathbb{R}^{(1+k)\times d}$ as key and value. $M$ is computed as:

$M=(\boldsymbol{v}_{r}+E^{\text{type}}_{0})\|(\text{diag}(\text{softmax}(\boldsymbol{s}_{r}))H_{r}+E^{\text{pos}}+E^{\text{type}}_{1}),$
(9)

where $\text{diag}(\cdot)$ denotes a diagonal matrix function and $\|$ is the
concatenation function. $E^{\text{pos}}\in\mathbb{R}^{k\times d}$ denotes
positional embeddings to determine how many concepts will be utilized.
$E^{\text{type}}_{0}\in\mathbb{R}^{d}$ and
$E^{\text{type}}_{1}\in\mathbb{R}^{d}$ are type embeddings to distinguish
$\boldsymbol{v}_{r}$ between information from retrieved concepts.
$E^{\text{type}}_{1}$ is replicated to $\mathbb{R}^{k\times d}$ in Eq. 9. The
augmenter $\mathcal{A}$ consists of $L$ decoder layers. The operation in
$l$-th decoder layer is as follows:

$\begin{split}\boldsymbol{q}^{\prime}_{l}&=\boldsymbol{q}_{l}+\text{CA}(\boldsymbol{q}_{l},M,M),\\\
\boldsymbol{q}_{l+1}&=\boldsymbol{q}^{\prime}_{l}+\text{FFN}(\boldsymbol{q}^{\prime}_{l}),\\\
\boldsymbol{v}_{r}^{\text{fine}}&=\boldsymbol{q}_{L},\end{split}$ (10)

where $l\in\\{1,\ldots,L\\}$. The term CA represents cross-attention
operation, while FFN stands for feed-forward network. Finally, the augmented
visual feature $\boldsymbol{v}^{\text{aug}}_{r}$ is obtained by adding the
coarse and fine feature of $\boldsymbol{v}_{r}$ as below:

$\boldsymbol{v}_{r}^{\text{aug}}=\boldsymbol{v}_{r}^{\text{coarse}}+\boldsymbol{v}_{r}^{\text{fine}}.$
(11)

At test time, as depicted in Figure 2(c) the augmented visual features and
text embeddings of test categories are used to compute auxiliary logits
$\boldsymbol{l}_{r}^{\text{aux}}$:

$\boldsymbol{l}_{r}^{\text{aux}}=\boldsymbol{v}_{r}^{\text{aug}}{\mathcal{T}(C^{\mathcal{B}}\cup
C^{\mathcal{N}})}^{\top}.$ (12)

Then the auxiliary logits are ensembled with $\boldsymbol{l}_{r}$ to compute
the final logits $\boldsymbol{l}_{r}^{\text{final}}$ for the final
classification of proposal $r$. The details of the logit ensemble are
described in Section 4.

Loss for RAF training. We pre-trained RAF with visual features from region
proposals. For pre-training, we use classification loss
$\mathcal{L}^{\text{cls}}$ and regularization loss $\mathcal{L}^{\text{reg}}$.
We first define the pseudo-label of region proposal $r$ as $\tilde{y}_{r}$:

$\tilde{y}_{r}=\operatorname*{argmax}_{c\in(C^{\mathcal{B}}\cup
C^{\mathcal{V}})}(\mathcal{T}(c)\boldsymbol{v}_{r}^{\top}).$ (13)

Then, using the pseudo-labels, classification loss $\mathcal{L}^{\text{cls}}$
is defined as:

$\mathcal{L}^{\text{cls}}=\cfrac{1}{N}\sum_{r}\mathcal{L}^{\text{ce}}(\boldsymbol{v}^{\text{aug}}_{r}{\mathcal{T}(C^{\mathcal{B}}\cup
C^{\mathcal{V}})}^{\top},\tilde{y}_{r}),$ (14)

where $N$ is the number of proposals per image and $\mathcal{L}^{\text{ce}}$
is the cross entropy loss. We encourage augmented visual feature
$\boldsymbol{v}_{r}^{\text{aug}}$ to similar to the original visual feature
$\boldsymbol{v}_{r}$ using regularization loss $\mathcal{L}^{\text{reg}}$
defined as below:

$\mathcal{L}^{\text{reg}}=\cfrac{1}{N}\sum_{r}(\boldsymbol{v}^{\text{aug}}_{r}-\boldsymbol{v}_{r})^{2}.$
(15)

Finally, the total loss $\mathcal{L}^{\text{RAF}}$ for RAF training is the
combination of Eq. 14 and Eq. 15 as follows:

$\mathcal{L}^{\text{RAF}}=\beta^{\text{cls}}\mathcal{L}^{\text{cls}}+\beta^{\text{reg}}\mathcal{L}^{\text{reg}},$
(16)

where $\beta^{\text{cls}}$ and $\beta^{\text{reg}}$ are hyperparameters.

##  4 Experiments

In this section, we briefly discuss the experimental setup, including datasets
and implementation details. Next, we evaluate the performance of RALF compared
to various baselines and investigate RALF with further analysis.

Datasets. We evaluate RALF on two public benchmarks, COCO [18] and LVIS [10].
In the open-vocabulary object detection setting, COCO dataset is split into 48
base categories and 17 novel categories following OVR-CNN [35]. It includes
118k images, which separated 107,761 images for training and 4,836 images for
validation. Referring to ViLD [9], we divide the LVIS dataset into 866 base
categories and 337 novel categories. For both benchmarks, we use base
categories during training, yet novel categories are also validated for
inference. We adopt the mean average precision (mAP) as the evaluation metric.
We report $\text{AP}_{\text{50}}$, ${\text{AP}}^{\text{B}}_{\text{50}}$, and
${\text{AP}}^{\text{N}}_{\text{50}}$ for COCO and $\text{AP}_{\text{r}}$,
$\text{AP}_{\text{c}}$, ${\text{AP}}_{\text{f}}$, and AP for LVIS. Note that
instance segmentation (mAP) results are reported on LVIS. To generate the
vocabulary store, we adopt V3Det [28] as a vocabulary set.

Implementation details. We implement RALF with pre-trained CLIP [20] with
ViT-B/32 image encoder backbone and text encoder from the official repository.
Note that we freeze all parameters in image and text encoders during training.
Additionally, we use GPT-3 [2] DaVinci-002 to generate descriptions from RAF
as a large language model. We use Faster R-CNN [25] with ResNet-50 [12]
backbone and Mask R-CNN [11] with ResNet-50 FPN backbone for COCO and LVIS,
respectively. For training RAF, we use region proposals provided from OADP
[29]. We set our baselines as OADP [29], Object-Centric-OVD [24], and DetPro
[6]. As RALF enhances the generalizability power via a plug-in manner, we
construct RALF into baselines. In all experiments, we train and evaluate on
NVIDIA RTX-3090 4 GPUs. More implementation details, including
hyperparameters, are discussed in the supplementary materials.

Logit ensemble in RAF. Since each baseline has a different range of logits,
the final logits $\boldsymbol{l}_{r}^{\text{final}}$ are calculated as below:

$\boldsymbol{l}_{r}^{\text{final}}=\begin{cases}\boldsymbol{l}_{r}+\sigma(\boldsymbol{l}_{r}^{\text{aux}})&\text{if
Object-Centric-OVD},\\\
\boldsymbol{l}_{r}*(\sigma(\boldsymbol{l}_{r}^{\text{aux}})-0.25)&\text{if
DetPro},\\\ \boldsymbol{l}_{r}+\boldsymbol{l}_{r}^{\text{aux}}&\text{if
OADP},\\\ \end{cases}$ (17)

where $\sigma(\cdot)$ denotes the sigmoid function. Without using all values
of auxiliary logits when adding to $\boldsymbol{l}_{r}$, only top-1 figures on
COCO and top-10 or 20 figures on LVIS are utilized, considering the number of
test categories.

###  4.1 Main results

We evaluate RALF on COCO and LVIS benchmarks in the open-vocabulary object
detection (OVD) setting and compare with various baselines. Overall results
are reported in Table 1 and Table 2.

COCO benchmark. As reported in Table 1, RALF shows great performance
enhancement when plugged into baselines for all evaluation metrics. RALF
significantly improved 4.7 $\text{AP}^{\text{N}}_{\text{50}}$ when plugged
into Object-Centric-OVD [24] and achieved state-of-the-art results. Moreover,
RALF surpasses Object-Centric-OVD for not only novel categories but also base
and all categories, with 0.3 $\text{AP}^{\text{B}}_{\text{50}}$ and 1.5
$\text{AP}_{\text{50}}$ improvements, respectively. Like the inclination
observed in the Object-Centric-OVD, implementing RALF on OADP [29]
demonstrates remarkable effectiveness. The results show performance gains for
the novel, base, and all categories by 3.4 $\text{AP}^{\text{N}}_{\text{50}}$,
1.2 $\text{AP}^{\text{B}}_{\text{50}}$, and 1.8 $\text{AP}_{\text{50}}$,
correspondingly.

Method | AP${}^{\text{N}}_{50}$ | AP${}^{\text{B}}_{50}$ | AP50
---|---|---|---
ViLD [9] | 27.6 | 59.9 | 51.2
PB-OVD [7] | 29.1 | 44.4 | 40.4
OV-DETR [34] | 29.4 | 61.0 | 52.7
VL-Det [17] | 32.0 | 50.6 | 45.8
MEDet [3] | 32.6 | 53.5 | 48.0
BARON [31] | 34.0 | 60.4 | 53.5
OADP [29] | 30.0 | 53.3 | 47.2
OADP + RALF | 33.4 | 54.5 | 49.0
Object-Centric-OVD [24] | 36.6 | 54.0 | 49.4
Object-Centric-OVD + RALF | 41.3 | 54.3 | 50.9

Table 1: Results of OVD on COCO.

Method | AP${}_{\text{r}}$ | AP${}_{\text{c}}$ | AP${}_{\text{f}}$ | AP
---|---|---|---|---
ViLD [9] | 16.1 | 20.0 | 28.3 | 22.5
OV-DETR [34] | 17.4 | 25.0 | 32.5 | 26.6
BARON [31] | 18.0 | 24.4 | 28.9 | 25.1
Object-Centric-OVD [24] | 17.1 | 21.4 | 26.7 | 22.8
Object-Centric-OVD + RALF | 18.5 | 21.0 | 26.3 | 22.6
DetPro [6] | 19.8 | 25.6 | 28.9 | 25.9
DetPro + RALF | 21.1 | 25.7 | 29.2 | 26.3
OADP [29] | 19.9 | 26.0 | 28.7 | 26.0
OADP + RALF | 21.9 | 26.2 | 29.1 | 26.6

Table 2: Results of OVD on LVIS.

LVIS benchmark. To verify that our method improves performance in various
cases, we experimented on the LVIS benchmark and added another baseline –
DetPro [6]. The results are noted in Table 2. Overall, RALF improves detection
ability on novel categories for all baselines by up to 2.0
$\text{AP}_{\text{r}}$. Although Object-Centric-OVD [24] slightly decreases
except $\text{AP}_{\text{r}}$, RALF improves all metrics on DetPro and OADP
[29]. To summarize, the results imply that RALF improves generalizability.

###  4.2 Ablation study

As discussed in Section 3.2, RALF consists of two modules – RAL and RAF. We
conducted ablation studies on RAL and RAF to verify the effectiveness of each
module on COCO and LVIS benchmarks.

Effectiveness of RAL. We demonstrate the results of RAL in Table 3 and Table
4. From the results, RAL improves performance gain in all baselines. RAL shows
an improvement of up to 1.3 $\text{AP}_{\text{50}}^{\text{N}}$ and at least
1.0 $\text{AP}_{\text{50}}^{\text{N}}$ on COCO. Interestingly, RAL
demonstrates exceptional generalizability, specifically within the COCO base
categories, exhibiting no performance declines. While there are marginal
decreases in $\text{AP}_{\text{c}}$, $\text{AP}_{\text{f}}$, and AP on LVIS,
the performance of novel categories is notably increased up to 3.2
$\text{AP}_{\text{r}}$.

Method | RAL | RAF | AP${}^{\text{N}}_{50}$ | AP${}^{\text{B}}_{50}$ | AP50
---|---|---|---|---|---
OADP† [29] |  |  | 30.0 | 54.5 | 48.1
| ✓ | 31.5 | 54.3 | 48.3
✓ |  | 31.3 | 54.5 | 48.4
✓ | ✓ | 33.4 | 54.5 | 49.0
Object-Centric-OVD† [24] |  |  | 40.0 | 53.8 | 50.2
| ✓ | 40.8 | 54.0 | 50.5
✓ |  | 41.0 | 54.1 | 50.7
✓ | ✓ | 41.3 | 54.3 | 50.9

Table 3: Ablation study of RALF on COCO dataset. $\dagger$ denotes reproduce
result.

Method | RAL | RAF | AP${}_{\text{r}}$ | AP${}_{\text{c}}$ | AP${}_{\text{f}}$ | AP
---|---|---|---|---|---|---
Object-Centric-OVD† [24] |  |  | 18.1 | 21.4 | 26.6 | 22.8
| ✓ | 19.5 | 21.4 | 26.6 | 23.1
✓ |  | 18.5 | 20.8 | 26.3 | 22.6
✓ | ✓ | 18.5 | 21.0 | 26.3 | 22.6
DetPro† [6] |  |  | 19.9 | 25.8 | 29.2 | 26.1
| ✓ | 20.5 | 25.8 | 29.2 | 26.2
✓ |  | 21.3 | 25.7 | 29.2 | 26.3
✓ | ✓ | 21.1 | 25.7 | 29.2 | 26.3
OADP† [29] |  |  | 18.3 | 26.2 | 28.9 | 25.9
| ✓ | 19.3 | 26.2 | 28.9 | 26.1
✓ |  | 21.5 | 26.1 | 29.1 | 26.5
✓ | ✓ | 21.9 | 26.2 | 29.1 | 26.6

Table 4: Ablation study of RALF on LVIS dataset. $\dagger$ denotes reproduce
result.

(a) OADP

(b) OADP + RALF

Figure 5: Qualitative results on COCO. Results of bounding box predictions on
novel categories for (a) OADP and (b) OADP+RALF.

Effectiveness of RAF. We plug RAF into several baselines and evaluate
performance to verify whether RAF is effective. Table 3 and Table 4 show the
performance of RAF on COCO and LVIS, respectively. There are notable
improvements in overall baselines. In particular, RAF improves the prediction
ability for novel categories of OADP and Object-Centric-OVD by 1.5
$\text{AP}_{\text{50}}^{\text{N}}$ and 0.8 $\text{AP}_{\text{50}}^{\text{N}}$
on COCO, respectively. Without performance decreasing in base categories, the
performance of novel categories improved by up to 1.4 $\text{AP}_{\text{r}}$
on LVIS.

From the results in Table 3 and Table 4, irrespective of the baselines, RAL
and RAF enhance the prediction of novel categories while maintaining somewhat
the prediction of base categories. The combination of RAL and RAF, _i.e_.,
RALF, shows significant performance gain of the baselines compared with the
sole use of each module. To sum up, RAL and RAF demonstrate enhanced
performance individually, and their combined shows superior generalization
capabilities.

###  4.3 Analysis on RALF

Easy and hard negatives. We evaluate the effects of various approaches to
handle easy and hard negatives. First, we use only one type of negative; ‘Easy
negative only’ and ‘Hard negative only’. Second, we merge the easy and hard
negatives into one negative group denoted as ‘Merged’. Unlike RAL,
$\mathcal{L}^{\text{easy}}$ of Eq. 6 that uses two types of negatives is not
applied in the ‘Merged’ setting. Table 5 shows that the baselines above show
inferior performance to our method RAL. Overall, our method that
differentially treats easy and hard negatives boosts performance the most.

| AP${}^{\text{N}}_{50}$ | AP${}^{\text{B}}_{50}$ | AP50
---|---|---|---
Easy negative only | 31.0 | 54.3 | 48.2
Hard negative only | 30.9 | 54.6 | 48.4
Merged | 30.9 | 55.5 | 49.0
RAL | 31.3 | 54.5 | 48.4
Table 5: Analysis of easy and hard negatives on COCO.

###  4.4 Qualitative results

We visualize the detection results to verify that RALF captures novel
categories well. We compare one of our baselines (_i.e_., OADP) and OADP +
RALF at the top and bottom of Figure 5, respectively. Each box in the image
represents the box prediction results. From the qualitative results, RALF
captures novel categories better than baseline, which means RALF improves
generalizability.

##  5 Conclusion

In this paper, we present Retrieval-Augmented Losses and visual Features
(RALF) that retrieves information from a large vocabulary set and augments
losses and visual features. To optimize the detector, we add Retrieval-
Augmented Losses (RAL), which brings hard and easy negative vocabulary from
the pre-defined vocabulary store and reflects the semantic similarity with the
ground-truth label. Additionally, Retrieval-Augmented visual Features (RAF)
augments visual features with generated concepts from a large language model
and enables improved generalizability. To sum up, RALF combines both modules
and easily plugs into various detectors and significantly improves detection
ability not only base categories but also novel categories.

#### Acknowledgements.

This work was partly supported by ICT Creative Consilience Program through the
Institute of Information & Communications Technology Planning & Evaluation
(IITP) grant (IITP-2024-2020-0-01819), Convergence security core talent
training business (Korea University) grant (No.2022-0-01198), and the National
Research Foundation of Korea (NRF) grant (NRF-2023R1A2C2005373) funded by the
Korea government (MSIT).

## References

  * [1] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer.  Retrieval-augmented diffusion models.  In NeurIPS, 2022.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In NeurIPS, 2020.
  * [3] Peixian Chen, Kekai Sheng, Mengdan Zhang, Mingbao Lin, Yunhang Shen, Shaohui Lin, Bo Ren, and Ke Li.  Open vocabulary object detection with proposal mining and prediction equalization.  arXiv preprint arXiv:2206.11134, 2022.
  * [4] Han-Cheol Cho, Won Young Jhoo, Wooyoung Kang, and Byungseok Roh.  Open-vocabulary object detection using pseudo caption labels.  arXiv preprint arXiv:2303.13040, 2023.
  * [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL-HLT, 2019.
  * [6] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li.  Learning to prompt for open-vocabulary object detection with vision-language model.  In CVPR, 2022.
  * [7] Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong.  Open vocabulary object detection with pseudo bounding-box labels.  In ECCV, 2022.
  * [8] Ross Girshick.  Fast r-cnn.  In ICCV, 2015.
  * [9] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  In ICLR, 2022.
  * [10] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In CVPR, 2019.
  * [11] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [13] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In ICML, 2021.
  * [14] Prannay Kaul, Weidi Xie, and Andrew Zisserman.  Multi-modal classifiers for open-vocabulary object detection.  In ICML, 2023.
  * [15] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In NeurIPS, 2020.
  * [16] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl.  Language-driven semantic segmentation.  In ICLR, 2022.
  * [17] Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai.  Learning object-language alignments for open-vocabulary object detection.  In ICLR, 2023.
  * [18] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [19] Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel.  Retrieval augmented classification for long-tail visual recognition.  In CVPR, 2022.
  * [20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [21] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [22] Rita Ramos, Desmond Elliott, and Bruno Martins.  Retrieval-augmented image captioning.  In EACL, 2023.
  * [23] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu.  Denseclip: Language-guided dense prediction with context-aware prompting.  In CVPR, 2022.
  * [24] Hanoona Rasheed, Muhammad Maaz, Muhammad Uzair Khattak, Salman Khan, and Fahad Shahbaz Khan.  Bridging the gap between object and image-level representations for open-vocabulary detection.  In NeurIPS, 2022.
  * [25] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NeurIPS, 2015.
  * [26] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL, 2018.
  * [27] Hung-Yu Tseng, Hsin-Ying Lee, Lu Jiang, Ming-Hsuan Yang, and Weilong Yang.  Retrievegan: Image synthesis via differentiable patch retrieval.  In ECCV, 2020.
  * [28] Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin.  V3det: Vast vocabulary visual detection dataset.  In ICCV, 2023.
  * [29] Luting Wang, Yi Liu, Penghui Du, Zihan Ding, Yue Liao, Qiaosong Qi, Biaolong Chen, and Si Liu.  Object-aware distillation pyramid for open-vocabulary object detection.  In CVPR, 2023.
  * [30] Zihao Wang, Wei Liu, Qian He, Xinglong Wu, and Zili Yi.  Clip-gen: Language-free training of a text-to-image generator with clip.  arXiv preprint arXiv:2203.00386, 2022.
  * [31] Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy.  Aligning bag of regions for open-vocabulary object detection.  In CVPR, 2023.
  * [32] Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li.  Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching.  In CVPR, 2023.
  * [33] Rui Xu, Minghao Guo, Jiaqi Wang, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy.  Texture memory-augmented deep patch-based image inpainting.  TIP, 2021.
  * [34] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy.  Open-vocabulary detr with conditional matching.  In ECCV, 2022.
  * [35] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang.  Open-vocabulary object detection using captions.  In CVPR, 2021.
  * [36] Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, BG Vijay Kumar, Anastasis Stathopoulos, Manmohan Chandraker, and Dimitris N Metaxas.  Exploiting unlabeled data with vision and language models for object detection.  In ECCV, 2022.
  * [37] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.  Regionclip: Region-based language-image pretraining.  In CVPR, 2022.
  * [38] Chong Zhou, Chen Change Loy, and Bo Dai.  Extract free dense labels from clip.  In ECCV, 2022.

## Appendix

##  Appendix A Implementation details

In this section, we provide hyperparameter settings and implementation details
that are not mentioned in the main paper, for each RAL and RAF.

###  A.1 RAL details

After applying the rank variance sampling scheme, we set $m$ to 2,000 and $n$
to 10 to separate the vocabularies into hard and easy negatives. Different
hyperparameters depending on the baseline and the dataset for RAL are shown in
Table A1.

Method | Dataset |  $\alpha^{\text{hard}}$ |  $\alpha^{\text{easy}}$ |  $\lambda^{\text{hard}}$ |  $\lambda^{\text{easy}}$ |  $\beta^{\text{hard}}$ |  $\beta^{\text{easy}}$
---|---|---|---|---|---|---|---
OADP [29] | COCO |  0 |  1 |  1 |  5 |  1 |  1
LVIS |  0 |  1 |  1 |  10 |  1 |  1
Object-Centric-OVD [24] | COCO |  0 |  1 |  1 |  10 |  1 |  1
LVIS |  0 |  1 |  1 |  5 |  1 |  1
DetPro [6] | LVIS |  0 |  1 |  1 |  10 |  1 |  1

Table A1: Hyperparameters in RAL.

###  A.2 RAF details

When we bring the related concepts from the concept store, we set the number
of concepts $k$ to 50. In the augmenter $\mathcal{A}$, the number of the
decoder layers $L$ is set to 6. The one decoder layer consists of cross-
attention (CA) with 8 heads and FFN with 2,048 dimensions. Positional
embeddings $E^{\text{pos}}$ and type embeddings $E^{\text{type}}_{1}$ and
$E^{\text{type}}_{2}$ are initialized with random values. The total number of
parameters for the augmenter $\mathcal{A}$ is 51M. During RAF training, we use
$\beta^{\text{cls}}$ of 5.0 and $\beta^{\text{reg}}$ of 1.0.

##  Appendix B Further ablation study

###  B.1 RAF in official baseline

The performance figures of the baseline compared to RALF in the main paper
Section 4.2 are the results we reproduced ourselves for the fairness of the
ablations. We checked the potential variation in performance depending on the
execution environment. Therefore, we further verified how the performance
changes when RAF is applied to the officially provided model checkpoints by
the baseline authors. As depicted in Table A2, RAF demonstrates significant
performance improvements on the COCO datasets.

Method | RAF | AP${}^{\text{N}}_{50}$ | AP${}^{\text{B}}_{50}$ | AP50
---|---|---|---|---
OADP [29] |  | 31.3 | 55.0 | 48.8
✓ | 33.1 | 54.7 | 49.0
Object-Centric-OVD [24] |  | 40.7 | 54.1 | 50.6
✓ | 41.1 | 54.2 | 50.8

Table A2: RAF ablation in the official checkpoint on COCO.

##  Appendix C Further analysis

Hyperparameters of $\mathcal{L}^{\text{RAL}}$. As depicted in Figure A1, we
provide an analysis of the hyperparameters controlling
$\mathcal{L}^{\text{RAL}}$ in OADP [29] on COCO dataset. To conduct the
analysis, we fixed $\alpha^{\text{hard}}$, $\lambda^{\text{hard}}$, and
$\beta^{\text{hard}}$ as (0, 1, 1) and $\alpha^{\text{easy}}$,
$\lambda^{\text{easy}}$, and $\beta^{\text{easy}}$ as (1, 10, 1),
respectively. The analysis is performed using values (0.1, 1, 10, 100) for 6
hyperparameters. When increasing $\alpha^{\text{hard}}$ from 1 to 10, we
observed a slight improvement, however, there was a declining performance
tendency overall. $\alpha^{\text{easy}}$ exhibited a temporary performance
increase at 1 followed by a subsequent decline. Relative to lower values,
$\lambda^{\text{easy}}$ demonstrated better performance with higher values.
Regarding $\lambda^{\text{hard}}$, $\beta^{\text{hard}}$, and
$\beta^{\text{easy}}$, we found that they exhibited robust performance even
with varying values.

Figure A1: Analysis on hyperparameters of $\mathcal{L}^{\text{RAL}}$.

Exploration of $n$. Table A3 shows the difference between obtaining $n$ of $m$
negative vocabularies for each iteration based on random or similarity in RAL.
The similarity performance is better than the baseline in
$\text{AP}_{50}^{\text{B}}$ and $\text{AP}_{50}$, however, it was observed
that the performance was lower in $\text{AP}_{50}^{\text{N}}$ due to the
absence of randomness. Adopting randomly selecting $n$ of $m$ negative
vocabularies showed the best performance for novel categories.

Method | AP${}^{\text{N}}_{50}$ | AP${}^{\text{B}}_{50}$ | AP50
---|---|---|---
Baseline | 30.0 | 54.5 | 48.1
Similarity | 29.7 | 54.9 | 48.3
Random | 31.3 | 54.5 | 48.4

Table A3: Comparison on obtaining $n$.

Sampling schemes. As described in Section 3.3 of the main paper, RAL defines
hard and easy negative vocabularies through rank variance sampling of negative
retrievers from the vocabulary store. To ascertain the optimality of the rank
variance sampling method, we compared various sampling schemes – random
sampling and similarity-based sampling, and these results are reported in
Table A4. Random sampling scheme extracts vocabularies randomly without
considering other factors, while similarity-based sampling scheme reflects the
cosine similarity between $C^{\mathcal{V}}$ and ground-truth labels. The
experimental results show that among the various sampling schemes, rank
variance sampling not only demonstrates superior performance in
$\text{AP}_{\text{50}}^{\text{N}}$but also exhibits outstanding performance in
$\text{AP}_{\text{50}}$.

Sampling scheme | AP${}^{\text{N}}_{50}$ | AP${}^{\text{B}}_{50}$ | AP50
---|---|---|---
Random | 30.6 | 54.4 | 48.1
Similarity-based | 30.2 | 54.8 | 48.3
Rank variance | 31.3 | 54.5 | 48.4

Table A4: Analysis on sampling scheme.

Hyperparameter $k$ in RAF. Table A5 and Table A6 show how much performance
varies depending on the $k$ used to augment visual features in RAF on COCO and
LVIS dataset, respectively. We set $k$ to 50 as the default. When $k=10$, the
best performance was observed on COCO. The minor performance changes are
exhibited across different values of $k$ on LVIS. The results showed that our
method is robust to the choice of $k$.

$k$ | AP${}^{\text{N}}_{50}$ | AP${}^{\text{B}}_{50}$ | AP50
---|---|---|---
10 | 33.6 | 54.4 | 49.0
20 | 33.3 | 54.4 | 48.9
50 | 33.4 | 54.5 | 49.0
100 | 33.3 | 54.4 | 48.9

Table A5: Analysis on $k$ in RAF on COCO.

$k$ | AP${}_{\text{r}}$ | AP${}_{\text{c}}$ | AP${}_{\text{f}}$ | AP
---|---|---|---|---
10 | 21.9 | 26.2 | 29.1 | 26.6
20 | 21.9 | 26.2 | 29.1 | 26.6
50 | 21.9 | 26.2 | 29.1 | 26.6
100 | 21.8 | 26.2 | 29.1 | 26.6

Table A6: Analysis on $k$ in RAF on LVIS.

Scale of large vocabulary. As discussed in Section 3.3 of the main paper, RALF
uses a large vocabulary set to construct the vocabulary store. We set the
experiment to analyze the effectiveness of the vocabulary size. We adopt V3Det
[28] with 13,204 vocabularies as the vocabulary set for all experiments.
Before retrieving hard and easy negatives from the vocabulary store, we
construct the vocabulary store by eliminating unnecessary elements, _i.e_.,
novel categories and case-overlapping, from the vocabulary set. Following this
process, we obtained 13,064 vocabularies from V3Det. To examine whether
vocabulary size affects performance, we experiment on COCO benchmarks by
reducing the percentage of vocabulary size to 40% and 70%. The results are
shown in Table A7. From the results, we observed that the performance was
better when the vocabulary size was 100% compared to when it was 40% or 70%.

Scale of large vocabulary | AP${}^{\text{N}}_{50}$ | AP${}^{\text{B}}_{50}$ | AP50
---|---|---|---
40% | 30.9 | 54.7 | 48.5
70% | 30.8 | 54.7 | 48.4
100% | 31.3 | 54.5 | 48.4

Table A7: Analysis on a large vocabulary size.

Retrieve negative vocabularies with BERT. In this work, the hard and easy
negative vocabularies are retrieved based on cosine similarity between CLIP
[20] text embeddings in RAF. When retrieving negative vocabularies, it is also
possible to use the embeddings of a language model (LM) instead of CLIP. We
extract embeddings about base categories and a large vocabulary set with a
language model BERT [5] and then retrieve hard and easy negative vocabularies
based on the cosine similarity between the embeddings. Table A8 depicts the
comparison results of CLIP and BERT. The performance on the novel categories
is lower in BERT, but it increases by 0.7 AP${}^{\text{N}}_{50}$ compared to
the baseline.

Method | AP${}^{\text{N}}_{50}$ | AP${}^{\text{B}}_{50}$ | AP50
---|---|---|---
Baseline | 30.0 | 54.5 | 48.1
CLIP [20] | 31.3 | 54.5 | 48.4
BERT [5] | 30.7 | 55.4 | 48.9

Table A8: CLIP vs BERT.