Kaixun Huang, Ao Zhang, Zhanheng Yang, Pengcheng Guo, Bingshen Mu, Tianyi Xu,
Lei Xie∗††thanks: * Corresponding author.

# Contextualized End-to-End Speech Recognition with Contextual Phrase
Prediction Network

###### Abstract

Contextual information plays a crucial role in speech recognition technologies
and incorporating it into the end-to-end speech recognition models has drawn
immense interest recently. However, previous deep bias methods lacked explicit
supervision for bias tasks. In this study, we introduce a contextual phrase
prediction network for an attention-based deep bias method. This network
predicts context phrases in utterances using contextual embeddings and
calculates bias loss to assist in the training of the contextualized model.
Our method achieved a significant word error rate (WER) reduction across
various end-to-end speech recognition models. Experiments on the LibriSpeech
corpus show that our proposed model obtains a 12.1% relative WER improvement
over the baseline model, and the WER of the context phrases decreases
relatively by 40.5%. Moreover, by applying a context phrase filtering
strategy, we also effectively eliminate the WER degradation when using a
larger biasing list.

Index Terms: End-to-end Speech Recognition, Deep Biasing, Contextual List
Filter

##  1 Introduction

Figure 1: (a) The proposed contextual phrase prediction network, where the CTC
linear shares parameters with contextualized models. (b) Biasing layer via
multi-head attention (MHA) (c) Contextualized CTC model (d) Contextualized
Transducer model

In recent years, end-to-end automatic speech recognition (ASR) systems have
made remarkable progress, driven by the rapid development of deep neural
networks, and have become the mainstream of speech recognition research.
Various end-to-end ASR methods, such as connectionist temporal classification
(CTC) [1, 2], recurrent neural network transducer (RNN-T) [3, 4], and
attention-based encoder-decoder (AED) [5, 6, 7], have been widely used in
scenarios such as voice assistants and online conferences. However, since deep
learning techniques are highly data-dependent, the recognition accuracy of
end-to-end ASR systems tends to be significantly reduced for rare phrases in
the data, such as entity names, technical terms, etc. In practical
applications, these rare phrases can often be obtained in advance as
contextual information. For example, for voice assistants, contact names in
the address book and the names of installed applications are likely to appear
during the recognition process. Therefore, informing the model of contextual
information is crucial to accurately predict the corresponding phrases. As a
result, research on infusing contextual knowledge into end-to-end ASR models
for biased decoding has become increasingly important.

In previous studies, a typical bias method is LM fusion [8, 9, 10, 11, 12],
with shallow fusion being the most commonly used approach. In [9], an n-gram
of contextual phrases is compiled into a weighted finite-state transducer
(WFST) and the state of the decoded path in the WFST is maintained during
decoding. When a specific word is decoded, the score is added to the path,
thereby biasing the recognition process toward the contextual phrase. While LM
fusion can intuitively increase the posterior of tokens related to contextual
phrases, it has limited performance improvement on the bias list and requires
testing to determine the optimal fusion weight.

Another approach to address this problem is the deep bias method based on
neural networks [13, 14, 15, 16, 17], which introduces a separate biasing
component in the end-to-end model for modeling contextual phrases. Compared
with LM fusion, the deep bias method is more flexible and allows quick
adaptation to different contextual phrase lists for various scenarios. Several
previous works have modified basic end-to-end speech recognition models to
obtain contextualized models, such as CLAS [18] and CATT [19]. However, these
methods lack explicit supervision for bias tasks and are often limited to
specific model structures.

To solve this problem, we propose a novel approach called contextual phrase
prediction network (CPP Network) for deep biasing, which combines a
bidirectional long-short term memory network (BLSTM) based contextual encoder
with an attention mechanism to extract embeddings of context phrases. Unlike
CLAS [18] and CATT [19], our model predicts the biased phrases that appear in
the utterance using the contextual phrase prediction network and calculates a
CTC loss between the predicted phases with labels that only contain context
phrases as the bias loss. This loss function provides explicit supervision for
the bias task, allowing the contextualized model to learn to focus on context
phrases appearing in the audio while minimizing the impact on the original
recognition results when there are no context phrases. The contextual phrase
prediction network can be easily integrated into mainstream end-to-end speech
recognition models, such as CTC, AED, and Transducer, to achieve
contextualized recognition. Our proposed techniques achieve an average
relative WER improvement of 12.1% on LibriSpeech, with a relative improvement
of 40.5% in the WER of phrases included in the biasing list.

In addition, using a biasing list containing thousands of contextual phrases
as input for a contextualized ASR model poses challenges in retrieving
meaningful contextual encodings from the biasing layer. Moreover, the biasing
layer requires computations on thousands of contextual phrase encodings,
leading to a noticeable increase in inference time. Inspired by [20], we
employ a two-stage contextual phrase filtering method. Firstly, the model
performs inference using an empty biasing list to calculate posterior score
confidence and sequence order confidence for each context phrase in the actual
biasing list. Then, we use the filtered biasing list to generate the final
bias output by combining it with the audio embeddings retained from the first
stage through a small amount of computation. This approach reduces the
computational burden and inference time, while ensuring that only relevant
contextual phrases are used for biasing. Compared to [20], we extended this
contextual phrase filtering method to be applicable to multiple structures,
rather than being limited to Transducer models with cascaded encoders.

##  2 Contextual Bias with CPP Network

In this section, we introduce the modification on contextual bias module with
our proposed CPP Network. Different modifications made on different ASR
framework including CTC, AED and Transducer will be introduced respectively.

###  2.1 Contextualized CTC/AED

Table 1: LibriSpeech results with different biasing list size N. Reported
metrics are in the following format: WER (U-WER/B-WER).

Model | N=0 | N=100 | N=500 | N=1000
---|---|---|---|---
test-clean | test-other | test-clean | test-other | test-clean | test-other | test-clean | test-other
CTC Baseline |  | 5.59
---
(3.5 / 23.4)
| 12.17
---
(8.1 / 48.3)
| 5.59
---
(3.5 / 23.4)
| 12.17
---
(8.1 / 48.3)
| 5.59
---
(3.5 / 23.4)
| 12.17
---
(8.1 / 48.3)
| 5.59
---
(3.5 / 23.4)
| 12.17
---
(8.1 / 48.3)
CTC + bias |  | 5.67
---
(3.6 / 23.4)
| 12.14
---
(8.2 / 47.3)
| 5.13
---
(4.2 / 12.7)
| 11.76
---
(9.9 / 29.0)
| 6.11
---
(5.2 / 13.7)
| 13.77
---
(11.8 / 31.5)
| 6.89
---
(6.0 / 14.7)
| 14.85
---
(12.9 / 32.6)
AED Baseline |  | 4.22
---
(2.6 / 18.1)
| 8.88
---
(5.6 / 37.6)
| 4.22
---
(2.6 / 18.1)
| 8.88
---
(5.6 / 37.6)
| 4.22
---
(2.6 / 18.1)
| 8.88
---
(5.6 / 37.6)
| 4.22
---
(2.6 / 18.1)
| 8.88
---
(5.6 / 37.6)
AED + bias |  | 4.29
---
(2.6 / 18.3)
| 9.16
---
(5.9 / 37.5)
| 3.40
---
(2.6 / 10.4)
| 7.77
---
(6.0 / 23.0)
| 3.68
---
(2.8 / 10.9)
| 8.31
---
(6.5 / 24.3)
| 3.81
---
(2.9 / 11.4)
| 8.75
---
(6.9 / 25.3)
Transducer Baseline |  | 4.30
---
(2.8 / 17.2)
| 8.90
---
(5.9 / 35.5)
| 4.30
---
(2.8 / 17.2)
| 8.90
---
(5.9 / 35.5)
| 4.30
---
(2.8 / 17.2)
| 8.90
---
(5.9 / 35.5)
| 4.30
---
(2.8 / 17.2)
| 8.90
---
(5.9 / 35.5)
Transducer + bias |  | 4.38
---
(2.7 / 18.3)
| 9.12
---
(5.9 / 37.6)
| 3.66
---
(2.8 / 11.2)
| 7.63
---
(6.0 / 22.1)
| 3.78
---
(2.9 / 11.5)
| 7.99
---
(6.2 / 23.4)
| 3.88
---
(2.9 / 11.9)
| 8.28
---
(6.4 / 24.5)

For both contextualized CTC and AED frameworks, we made modifications only to
the encoder. It should be noted that we also deploy CTC as auxiliary training
criterion for AED encoder. Thus these modifications for both frameworks were
similar. Generally, we add a context encoder, a multi-head attention-based
biasing layer, and a contextual phrase prediction network (CPP Network) for
the framework to integrate contextual information, as shown in Figure 1(c).

The context encoder is used to encode the custom contextual phrase list into
embeddings with the same length. Each phrase in the list is tokenized with the
same tokenizer as the base CTC model, and the token sequence is used as input
for the BLSTM. The hidden state and cell state of the last time step from both
the forward and backward LSTM are concatenated and then passed through a
linear layer to obtain the embedding $h^{CE}_{i}$ for the contextual phrase.
In addition, we added a phrase to the contextual phrase list that contains
only a single blank token, referred to as no-bias, to enable the model to
focus on it when there is no contextual phrase in the audio and ignore other
contextual phrases.

The biasing layer (Figure 1(b)) is a multi-head attention (MHA) layer designed
to help the model learn the relationship between contextual phrases and
speech. While computing the MHA score, the audio embedding $h_{t}^{E}$ serves
as the query, and the contextual phrase embeddings $h_{i}^{CE}$ serve as the
keys and values to obtain the context representations $c_{t}^{E}$ as:

$\bm{\alpha}_{it}=\operatorname{Softmax}\left(\left(\mathbf{W}_{q}\mathbf{h}_{t}^{E}\right)\left(\mathbf{W}_{k}\mathbf{h}_{i}^{CE}\right)/\operatorname{sqrt}\left(d\right)\right)~{},$
(1) $\bm{c}_{t}^{E}=\sum_{i=0}^{K}\bm{\alpha}_{it}\mathbf{h}_{i}^{CE}~{},$ (2)

where the scaling factor $\operatorname{sqrt}\left(d\right)$ is for numerical
stability, and $K$ represents the number of contextual phrases. The context
embeddings and audio embeddings are then combined to obtain context-aware
audio embeddings $h_{t}^{CA}$. The combiner consists of a LayerNorm layer, a
concatenation operation, and a feed-forward projection layer. This process can
be described as:

$\mathbf{h}^{concat}_{t}=[\mathrm{LayerNorm}(\mathbf{h}^{E}_{t}),\mathrm{LayerNorm}(\bm{c}^{t})]~{},$
(3) $\mathbf{h}^{CA}_{t}=\mathrm{FeedForward}(\mathbf{h}^{concat}_{t})~{}.$
(4)

The CPP network consists of two feed-forward projection layers and a Softmax
function (Figure 1(a)), where the second feed-forward projection layer shares
parameters with the linear layer in the CTC model that maps audio embeddings
to a posterior probability distribution over vocabularies, denoted as CTC
linear in Figure 1. Receiving the context embedding $c_{t}^{E}$ as input, CPP
Network predicts the contextual phrases that appear in the dialogue utterance.
We use the posterior generated by CPP Network as input and the contextual
phrases in the transcript as reference for CTC loss calculation. Such loss can
supervise the biasing layer to extract relevant contextual phrases information
from the audio embeddings $h^{E}$ when the contextual phrases appear and
minimizes the influence of introducing additional information when no
contextual phrases appear. Note that this part of the model is only used for
auxiliary training and does not participate in model inference.

The final joint loss function for the CTC model is:

$L=L_{ctc}+\lambda_{2}L_{CPP}~{},$ (5)

where $\lambda_{2}$ is a weight hyperparameter we set as $0.1$ in experiment.

For the AED model, with $\lambda_{1}$ as the CTC weight, the final joint loss
function is:

$L=\lambda_{1}L_{ctc}+(1-\lambda_{1})L_{att}+\lambda_{1}\lambda_{2}L_{CPP}~{}.$
(6)

###  2.2 Contextualized Transducer

For the Transducer, we augment the architecture with a contextual phrase
prediction network to implement the contextualized Transducer, as illustrated
in Figure 1(d). Specifically, compared to the standard Transducer, we add a
context encoder, two MHA biasing layers, and a contextual phrase prediction
network. Similar to CTC/AED framework, both the audio encoder and the label
encoder are followed by MHA biasing layers to integrate contextual information
from $h_{i}^{CE}$. Then the context-aware audio embeddings $c^{AE}$ and the
context-aware label embeddings $c^{LE}$ are fed into the joint network.

For CPP Network, We use the context embeddings $c_{t}^{AE}$ obtained from the
audio encoder as input to predict the contextual phrases contained in the
speech and compute the bias loss.

Finally, we combine the bias loss with the standard Transducer loss as the
optimization objective:

$L=\lambda_{1}L_{ctc}+(1-\lambda_{1})L_{transducer}+\lambda_{1}\lambda_{2}L_{CPP}~{},$
(7)

where $\lambda_{1}$ and $\lambda_{2}$ are the weight hyperparameters for the
CTC loss and the CPP loss.

##  3 Contextual Phrase Filtering

Inspired by [20], we apply a two-stage filtering algorithm, using the
posterior of the CTC to filter the contextual phrases. Differently, we
simplified the structure of the Cascade Encoders, performing contextual phrase
filtering without altering the original model structure.

Firstly, the calculation is performed using a biasing list that contains only
no-bias to obtain the CTC posterior without bias. The audio embedding $h^{E}$
is retained. The decoding results from the CTC during the first pass inference
can serve as streaming results.

The contextual phrases are then filtered in two stages using the CTC
posterior. The first stage calculates the posterior and the phrase score
confidence (PSC), ignoring the order of the contextual phrases. It computes
the sum of the maximum posterior values of each token contained in the
contextual phrases within a sliding window and normalizes it with the sequence
length. This stage eliminates most of the contextual phrases with low
correlation with the audio, retaining only the remaining ones for the second
stage of filtering. In the second stage, the sequence order confidence (SOC)
is calculated by using a dynamic programming algorithm to compute the maximum
posterior sum of tokens contained in contextual phrases within the sliding
window, considering token order.

After obtaining the filtered contextual phrases, subsequent inference is
performed with the previously retained audio embedding $c_{t}^{E}$, resulting
in a more accurate biased recognition result. The complexity of context phrase
filtering is low for inference. Compared to the original recognition process,
only the biasing layer, combiner, and CTC linear are repeated in the
calculation, and the increase in computation is relatively small.

##  4 Experiments

###  4.1 Data

A majority of our experiments were conducted on the LibriSpeech corpus [21],
which comprises 960 hours of English audiobook readings. The model was trained
on the complete 960 hours of training data to demonstrate its performance on
relatively low-resource tasks. The dev-clean and dev-other sets were used for
validation, while the test-clean and test-other sets were used for evaluation.
SpecAugment [22] was applied for data augmentation.

During the training of contextualized models, we extract three random phrases
of word length between 1 to 3 from each utterance and add them to the biasing
list in the batch. Additionally, we incorporate a certain number of
distractors into the biasing list to maintain the list size at 60.

During testing, we utilized the biasing lists for the LibriSpeech corpus
provided in [16] as the contextual phrase lists. As outlined in [16], the
biasing lists were first formed for each utterance by identifying words
belonging to the full rare-word list from the reference of each utterance, and
adding a certain number of distractor words.

To validate the model's performance with larger amounts of data, we also
evaluate our method on GigaSpeech corpus [23] and tested on Earnings21 dataset
[24]. Earnings21 set consists of earnings call transcripts from various
publicly traded companies and provides a range of proprietary terms, including
names of companies, products, and executives.

###  4.2 Experimental Setups

The baseline CTC model consists of a 12-layer conformer with 256-dimensional
input and 4 self-attention heads and the baseline AED model uses 12 conformer
layers for the encoder and 6 transformer layers for the decoder. The baseline
Transducer model uses a 12-layer conformer as the audio encoder and 2-layer
LSTM as the label encoder, referred to as C-T.

In the contextualized version, the context encoder consists of a 1-layer BLSTM
with a dimensionality of 256 and a linear layer, while the biasing layer is an
MHA layer with an embedding size of 256 and 4 attention heads. The first
linear layer in the context prediction network maintains a 256-dimensional
input and output, followed by a tanh activation function, while the second
linear layer projects the input onto the vocabulary size and shares parameters
with the CTC linear.

In addition to the word error rate (WER), we also evaluated the results using
the biased word error rate (B-WER) and unbiased word error rate (U-WER). U-WER
is measured on words that are NOT in the biasing list, while B-WER is measured
on words that are IN the biasing list. For insertion errors, if the inserted
phrase is in the biasing list, it is counted towards B-WER, otherwise it is
counted towards U-WER.

###  4.3 Contextual ASR Accuracy

Table 2: Results of contextual phrase filtering on contextual AED model with
biasing list size N = 1000

Model | without filter | with filter
---|---|---
test-clean | test-other | test-clean | test-other
CTC + bias |  | 6.89
---
(6.0 / 14.7)
| 14.85
---
(12.9 / 32.6)
| 4.95
---
(3.72 / 15.3)
| 10.81
---
(8.3 / 32.7)
AED + bias |  | 3.81
---
(2.9 / 11.4)
| 8.75
---
(6.9 / 25.3)
| 3.61
---
(2.6 / 12.1)
| 7.90
---
(5.9 / 25.8)
C-T + bias |  | 3.88
---
(2.9 / 11.9)
| 8.28
---
(6.4 / 24.5)
| 3.76
---
(2.7 / 12.8)
| 7.96
---
(5.8 / 26.8)

Table 3: Comparison of proposed method and shallow fusion

Model |  test-clean |  test-other
---|---|---
AED baseline |  |  4.29
---
(2.6 / 18.3)
|  9.16
---
(5.9 / 37.5)
\+ SF |  |  4.19
---
(2.7 / 16.8)
|  8.79
---
(6.0 / 33.1)
\+ bias |  |  3.88
---
(2.8 / 13.0)
|  8.47
---
(6.4 / 26.6)
++ SF |  |  3.86
---
(2.9 / 12.4)
|  8.45
---
(6.5 / 25.5)

In this section, we tested the contextual recognition models and their non-
contextual baseline on all these types of models. As shown in Table 1, the
results of experiments containing a non-empty biasing list demonstrate a
decrease in B-WER. The best results were achieved when using a biasing list of
100, yielding an average 40.5% relative improvement of B-WER compared to the
baseline. When the size of the biasing list increased to 1000, B-WER still
showed a noticeable decrease, but the reduction was less significant, and
U-WER was also impacted to some degree. This suggests that the model is
sensitive to the size of the biasing list. We found that using more
distractors during the training process, or not sampling contextual phrases
for some utterances can reduce U-WER when using a large biasing list, but will
also decrease the improvement of B-WER.

###  4.4 Contextual Phrase Filtering Performance

As shown in Table 2, we tested the effect of contextual phrase filtering on
all these types of models with a biasing list of size 1000. After applying the
filtering method, the impact of the biasing list on U-WER was significantly
reduced and largely regressed to the level when an empty biasing list was
used, while B-WER showed only a slight increase compared to the results
without contextual phrase filtering.

###  4.5 Comparison with Shallow Fusion

In order to compare the proposed deep bias method and shallow fusion, we
separately tested the effect of using each method alone and using them
together on the contextualized AED model. To ensure fairness, we extracted the
contextual phrases in each utterance in the test set into a list, with a list
size of 4250 for the test-clean set and 3838 for the test-other set. Two
methods use the same biasing list. When evaluating the results of the shallow
fusion, we used the shallow fusion method based on OTF rescoring described in
[25]. As shown in Table 3, the shallow fusion method reduced the average U-WER
by 6.9% relative to the baseline and the proposed deep bias method reduced the
average U-WER by 29.2%. Compared to shallow fusion, the improvement in B-WER
from deep bias was apparently greater, and the use of both methods together
can further reduce the B-WER, although the impact on U-WER will also
accumulate.

###  4.6 Ablation Study

Table 4: Results of ablation study

Model | N=100 | N=1000
---|---|---
test-clean | test-other | test-clean | test-other
AED + bias |  | 3.40
---
(2.6 / 10.4)
| 7.77
---
(6.0 / 23.0)
| 3.81
---
(2.9 / 11.4)
| 8.75
---
(6.9 / 25.3)
|  \- share
---
| 3.50
---
(2.7 / 10.2)
| 7.68
---
(6.2 / 21.0)
| 4.13
---
(3.3 / 11.3)
| 9.43
---
(7.7 / 24.4)
|  \- loss
---
| 4.26
---
(2.6 / 18.0)
| 9.31
---
(5.9 / 39.0)
| 4.26
---
(2.6 / 18.0)
| 9.31
---
(5.9 / 39.0)
C-T + bias |  | 3.66
---
(2.8 / 11.2)
| 7.63
---
(6.0 / 22.1)
| 3.88
---
(2.9 / 11.9)
| 8.28
---
(6.4 / 24.5)
|  \- loss
---
| 4.05
---
(2.8 / 14.5)
| 8.65
---
(6.2 / 30.5)
| 4.24
---
(2.9 / 15.4)
| 8.99
---
(6.3 / 32.8)

Table 5: Results of contextualized AED model on Earnings21

Model | WER | Recall | Precision | F1
---|---|---|---|---
AED Baseline | 14.96 | 58.66 | 91.52 | 71.50
AED + bias | 15.12 | 79.11 | 80.67 | 79.88

Table 4 presents the results of ablation study. We tested the performance of
the contextualized AED model without sharing the CTC linear parameters in the
CPP Network and without the addition of the bias loss. The results of not
sharing the CTC linear parameters showed better B-WER performance, but the
U-WER was more sensitive to the size of the biasing list. Without the bias
loss, the contextualized AED model completely lost its ability to perform bias
recognition. When a larger biasing list was used, there was a noticeable
increase in the WER loss. In the contextualized Transducer, when the bias loss
is removed, the resulting model is essentially CATT, which had the ability to
perform bias recognition, but the improvement in the B-WER was not as
significant as before.

###  4.7 Test on Earnings21

We trained a contextualized AED model on the GigaSpeech dataset [23] and
evaluated it on the Earnings21 dataset [24], with results presented in Table
5. Similar to [24], we demonstrate the WER and phrase-level recall, precision,
and F1 scores of two models on the Earnings21 dataset. When using the
contextualized AED model and applying contextual phrase filtering, the recall
of contextual phrases is significantly improved, resulting in a relative
improvement of 33.2% compared to the baseline. Although the precision was
impacted to a certain extent, the overall F1 score still improves by 11.7%
relative. Due to the small proportion of biased phrases in the test set,
contextual biasing has a minimal impact on overall WER.

##  5 Conclusions

In this paper, we propose a deep biasing method based on the contextual phrase
prediction network. The context phrase prediction network predicts context
phrases contained in the utterances and calculates CTC loss on the prediction
results as an auxiliary loss for the training of the contextualized model.
Compared with previous works, our approach achieves significant B-WER
improvement on all CTC, AED, and Transducer models. In addition, we apply a
context phrase filtering method to the final context-aware ASR model, which
effectively reduces the U-WER loss when using a larger biasing list.

## References

  * [1] A. Graves, S. Fernández, F. J. Gomez, and J. Schmidhuber, ``Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,'' in _Machine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006)_ , W. W. Cohen and A. W. Moore, Eds. ACM, 2006, pp. 369–376.
  * [2] A. Graves and N. Jaitly, ``Towards end-to-end speech recognition with recurrent neural networks,'' in _Proceedings of the 31th International Conference on Machine Learning, ICML 2014_. JMLR.org, 2014, pp. 1764–1772.
  * [3] Y. He, T. N. Sainath, R. Prabhavalkar, I. McGraw, R. Alvarez, D. Zhao, D. Rybach, A. Kannan, Y. Wu, R. Pang, Q. Liang, D. Bhatia, Y. Shangguan, B. Li, G. Pundak, K. C. Sim, T. Bagby, S. Chang, K. Rao, and A. Gruenstein, ``Streaming end-to-end speech recognition for mobile devices,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019_. IEEE, 2019, pp. 6381–6385.
  * [4] Y. Zhang, S. Sun, and L. Ma, ``Tiny transducer: A highly-efficient speech recognition model on edge devices,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021_. IEEE, 2021, pp. 6024–6028.
  * [5] J. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, ``Attention-based models for speech recognition,'' in _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015_ , C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, Eds., 2015, pp. 577–585.
  * [6] K. Cho, B. van Merrienboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, ``Learning phrase representations using RNN encoder-decoder for statistical machine translation,'' in _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014_ , A. Moschitti, B. Pang, and W. Daelemans, Eds. ACL, 2014, pp. 1724–1734.
  * [7] W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, ``Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,'' in _2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016_. IEEE, 2016, pp. 4960–4964.
  * [8] I. Williams, A. Kannan, P. S. Aleksic, D. Rybach, and T. N. Sainath, ``Contextual speech recognition in end-to-end neural network systems using beam search,'' in _Interspeech 2018, 19th Annual Conference of the International Speech Communication Association_ , B. Yegnanarayana, Ed. ISCA, 2018, pp. 2227–2231.
  * [9] Z. Chen, M. Jain, Y. Wang, M. L. Seltzer, and C. Fuegen, ``End-to-end contextual speech recognition using class language models and a token passing decoder,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019_. IEEE, 2019, pp. 6186–6190.
  * [10] D. Zhao, T. N. Sainath, D. Rybach, P. Rondon, D. Bhatia, B. Li, and R. Pang, ``Shallow-fusion end-to-end contextual biasing,'' in _Interspeech 2019, 20th Annual Conference of the International Speech Communication Association_ , G. Kubin and Z. Kacic, Eds. ISCA, 2019, pp. 1418–1422.
  * [11] R. Huang, O. Abdel-Hamid, X. Li, and G. Evermann, ``Class LM and word mapping for contextual biasing in end-to-end ASR,'' in _Interspeech 2020, 21st Annual Conference of the International Speech Communication Association_ , H. Meng, B. Xu, and T. F. Zheng, Eds. ISCA, 2020, pp. 4348–4351.
  * [12] S. Kim, Y. Shangguan, J. Mahadeokar, A. Bruguier, C. Fuegen, M. L. Seltzer, and D. Le, ``Improved neural language model fusion for streaming recurrent neural network transducer,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021_. IEEE, 2021, pp. 7333–7337.
  * [13] Z. Chen, M. Jain, Y. Wang, M. L. Seltzer, and C. Fuegen, ``Joint grapheme and phoneme embeddings for contextual end-to-end ASR,'' in _Interspeech 2019, 20th Annual Conference of the International Speech Communication Association_ , G. Kubin and Z. Kacic, Eds. ISCA, 2019, pp. 3490–3494.
  * [14] M. Han, L. Dong, S. Zhou, and B. Xu, ``Cif-based collaborative decoding for end-to-end contextual speech recognition,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021_. IEEE, 2021, pp. 6528–6532.
  * [15] G. Sun, C. Zhang, and P. C. Woodland, ``Tree-constrained pointer generator for end-to-end contextual speech recognition,'' in _IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2021_. IEEE, 2021, pp. 780–787.
  * [16] D. Le, M. Jain, G. Keren, S. Kim, Y. Shi, J. Mahadeokar, J. Chan, Y. Shangguan, C. Fuegen, O. Kalinli, Y. Saraf, and M. L. Seltzer, ``Contextualized streaming end-to-end speech recognition with trie-based deep biasing and shallow fusion,'' in _Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association_ , H. Hermansky, H. Cernocký, L. Burget, L. Lamel, O. Scharenborg, and P. Motlícek, Eds. ISCA, 2021, pp. 1772–1776.
  * [17] S. Dingliwal, M. Sunkara, S. Ronanki, J. Farris, K. Kirchhoff, and S. Bodapati, ``Personalization of CTC speech recognition models,'' in _IEEE Spoken Language Technology Workshop, SLT 2022_. IEEE, 2022, pp. 302–309.
  * [18] G. Pundak, T. N. Sainath, R. Prabhavalkar, A. Kannan, and D. Zhao, ``Deep context: End-to-end contextual speech recognition,'' in _2018 IEEE Spoken Language Technology Workshop, SLT 2018_. IEEE, 2018, pp. 418–425.
  * [19] F. Chang, J. Liu, M. Radfar, A. Mouchtaris, M. Omologo, A. Rastrow, and S. Kunzmann, ``Context-aware transformer transducer for speech recognition,'' in _IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2021_. IEEE, 2021, pp. 503–510.
  * [20] Z. Yang, S. Sun, X. Wang, Y. Zhang, L. Ma, and L. Xie, ``Two stage contextual word filtering for context bias in unified streaming and non-streaming transducer,'' _CoRR_ , vol. abs/2301.06735, 2023.
  * [21] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: An ASR corpus based on public domain audio books,'' in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015_. IEEE, 2015, pp. 5206–5210.
  * [22] D. S. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, ``Specaugment: A simple data augmentation method for automatic speech recognition,'' in _Interspeech 2019, 20th Annual Conference of the International Speech Communication Association_ , G. Kubin and Z. Kacic, Eds. ISCA, 2019, pp. 2613–2617.
  * [23] G. Chen, S. Chai, G. Wang, J. Du, W. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhang, M. Jin, S. Khudanpur, S. Watanabe, S. Zhao, W. Zou, X. Li, X. Yao, Y. Wang, Z. You, and Z. Yan, ``Gigaspeech: An evolving, multi-domain ASR corpus with 10, 000 hours of transcribed audio,'' in _Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association_ , H. Hermansky, H. Cernocký, L. Burget, L. Lamel, O. Scharenborg, and P. Motlícek, Eds. ISCA, 2021, pp. 3670–3674.
  * [24] J. D. Fox and N. Delworth, ``Improving contextual recognition of rare words with an alternate spelling prediction model,'' in _Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association_ , H. Ko and J. H. L. Hansen, Eds. ISCA, 2022, pp. 3914–3918.
  * [25] B. Zhang, D. Wu, Z. Peng, X. Song, Z. Yao, H. Lv, L. Xie, C. Yang, F. Pan, and J. Niu, ``Wenet 2.0: More productive end-to-end speech recognition toolkit,'' in _Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association_ , H. Ko and J. H. L. Hansen, Eds. ISCA, 2022, pp. 1661–1665.


# RTGen: Generating Region-Text Pairs for Open-Vocabulary Object Detection

Fangyi Chen Han Zhang∗ Zhantao Yang Hao Chen Kai Hu Marios Savvides
Carnegie Mellon University
<EMAIL_ADDRESS>The first two
authors contributed equally.

###### Abstract

Open-vocabulary object detection (OVD) requires solid modeling of the region-
semantic relationship, which could be learned from massive region-text pairs.
However, such data is limited in practice due to significant annotation costs.
In this work, we propose RTGen to generate scalable open-vocabulary region-
text pairs and demonstrate its capability to boost the performance of open-
vocabulary object detection. RTGen includes both text-to-region and region-to-
text generation processes on scalable image-caption data. The text-to-region
generation is powered by image inpainting, directed by our proposed scene-
aware inpainting guider for overall layout harmony. For region-to-text
generation, we perform multiple region-level image captioning with various
prompts and select the best matching text according to CLIP similarity. To
facilitate detection training on region-text pairs, we also introduce a
localization-aware region-text contrastive loss that learns object proposals
tailored with different localization qualities. Extensive experiments
demonstrate that our RTGen can serve as a scalable, semantically rich, and
effective source for open-vocabulary object detection and continue to improve
the model performance when more data is utilized, delivering superior
performance compared to the existing state-of-the-art methods. The code and
data will be released at https://github.com/seermer/RTGen

##  1 Introduction

Deep learning models trained on sufficient defined-vocabulary data are
effective in solving object detection tasks [15, 49, 48, 35, 56, 2, 30, 3, 32,
70], but in the open world, detecting thousands of object categories remains a
challenge. Great interest has been raised in improving open-vocabulary object
detection (OVD), which is expected to detect objects of arbitrary novel
categories that have not necessarily been seen during training [16, 9, 26, 63,
41, 62, 24, 39].

Recently, the advancements in vision-language models [47, 22, 6] have
propelled image-level open-vocabulary tasks to new heights through the
utilization of contrastive learning across a vast scale of image-caption pairs
[53, 55, 52]. This progress inspires researchers in the detection community to
explore OVD methods drawing upon similar principles. However, training object
detector needs region-level annotations. Unlike web-crawled image-caption
pairs, region-level instance-text (region-text) pairs are limited and
expensive to annotate, which leads us to two fundamental questions: how to
acquire scalable region-text pairs, and how to effectively utilize them for
training OVD?

Some recent approaches focus on acquiring region-level pseudo labels by mining
structures or data augmentation from image-caption pairs [68, 41, 34, 29, 44,
60]. They are typically designed to align image regions with textual phrases
extracted from corresponding captions. This is achieved by either leveraging a
pre-trained OVD model to search for the best alignment between object
proposals and phrases[29, 37, 66], or through associating the image caption
with the most significant object proposal [68, 34]. However, these web-crawled
data may lack of accurate image-caption correspondence as many captions do not
directly convey the visual contents, as shown in Fig.1(a). In addition, the
precision of alignment is significantly dependent on the performance of the
pre-trained OVD models, resulting in a recursive dilemma: a good OVD detector
is requisite for generating accurate pseudo predictions, which in turn are
essential for training a good OVD detector.

Different from prior arts, this paper studies to leverage generative models to
synthesize a rich corpus of region-text pairs for training OVD. Unlike OVD
models whose training relies on limited detection/grounding data, generative
models [50, 33, 65, 28, 27, 31] are typically trained on extensive datasets
that have both imagery and textual modalities. Within the framework outlined
in this study, generated open-vocabulary region-text pairs are proven as
effective in accomplishing the task.

More specifically, our generation process is rooted in the web-crawled image-
caption pairs and operates under two paradigms: text-to-region (T2R) and
region-to-text (R2T). In the text-to-region process, we guide a diffusion
model [33] to execute the inpainting, conditioned on extracted caption phrases
and image-predicted proposal boxes. A key design of this process is the
allocation of phrases and boxes to achieve overall layout harmony. This is
facilitated by training a novel Scene-Aware Inpainting Guider (SAIG), designed
to interpret the multi-modal scene comprehensively and sample flexible layouts
that guide the inpainting within contextually relevant and geometrically
coherent regions. In the region-to-text process, we verify that simply
applying a powerful captioning model [27] on object proposals is an effective
way to generate region-text pairs. The generation exhibits three novel
characteristics: Firstly, rather than applying generative models on pre-
existing detection datasets [54, 11, 7], our generation is based on image-
caption pairs that are scalable and mirror the real-world distribution,
aligning well with the nature of open-vocabulary setting. Secondly, unlike
some concurrent researches [11, 7] directly work on the novel categories, we
structure the generation process under the circumstance of without knowing the
novel categories in advance. Thirdly, models from two distinct domains
introduce a breadth of semantic richness and knowledge, enhancing the
diversity of the generated data, as shown in Fig.1(b) and (c). As the
capabilities of generative models continue to advance, we anticipate further
enhancements in this framework over time.

Figure 1:  (a) Image-caption pairs may lack visual-textual correlation,
addressed through the text-to-region (T2R) and region-to-text (R2T) processes.
(b) The generative processes introduce a breadth of visual and textual
diversity. (c) The average precision on OV-COCO novel classes when training
with different percentages of our generated data which demonstrates a steep
increase when R2T+T2R is applied, evidencing a marked improvement proportional
to the volume of data utilized .

To effectively use the generated region-text pairs, we expand contrastive
learning to fit detection learning scenarios, by incorporating not only the
generated region-text pairs but also the adjacent, less accurate regions to
learn with dynamic targets and weights. This loss function, termed
Localization-Aware Region-Text Contrastive Loss, can be integrated into the
training pipeline of various object detectors, allowing for joint training
with standard detection data. Our contributions can be summarized in three
folds:

  * •

We propose a framework that generates open-vocabulary region-text pairs from
image-caption pairs. The framework features a text-to-region process, which is
the first attempt to synthesize region-text pairs for training OVD without
prior knowledge of the novel categories, as well as a region-to-text process
that populates the generation with abundant regional captions.

  * •

We introduce a novel Scene-Aware Inpainting Guider to facilitate text-to-
region generation. We

also introduce a new loss function, as an expansion of contrastive loss, which
enables detectors to effectively learn from generated region-text pairs.

  * •

We apply our methods on Faster-RCNN [49] and CenterNet2 [69] and conduct
extensive ablation studies and experiments on OV-COCO and OV-LVIS benchmarks.
Without bells and whistles, training with RTGen brings +6.9 box
$\text{AP}_{novel}$ on OV-COCO and +4.2 mask $\text{AP}_{novel}$ on OV-LVIS
compared to our baseline, and outperforms other state-of-the-art methods.

##  2 Related Work

###  2.1 Open Vocabulary Object Detection

Open-vocabulary Object Detection (OVD) learns to generalize from annotated
(seen) object categories to any (unseen) categories imagery. Different from
Zero-shot Object Detection (ZSD), OVD is typically trained on an instance-
level annotated datasets and meanwhile allowed the flexibility to learn
information from open-world text corpora and images [64]. A recent trend tries
to adapt the pretrained vision-language models (VLMs), such as CLIP[47], to
object detectors in attempt of zero-shot generalization capabilities. For
example, ViLD[16] directly distills the visual and textual knowledge of CLIP
to the object detectors. However, because these VLMs are typically trained on
image-text pair data, deploying them directly can be suboptimal for region-
level recognition. As a result, many works have been searching to establish a
solid region-text alignment. RegionCLIP[67] proposes to first finetune CLIP to
learn region-text alignments and then transfer CLIP to OVD. DetPro[9]
incorporates a trainable text prompt to align the VLM to the detection task.
Similarly, PromptDet[10] aims to align the textual embedding space with
regional visual object feature through region-level prompting. Alternatively,
several works examine more sophisticated approaches to mine the structural
correspondence. VLDet[34] proposes to utilize a bipartite set-matching
algorithm to find region-text pairs through the image. CoDet[41] finds the
common regions of interest among many images by constructing image-caption
groups that may contain similar visual contents. In addition, hand-crafted
region-text alignment by simple data augmentation are proven as effective.
Detic[68] directly learns the alignment between the text labels and the max-
sized object proposals. BARON[58] and CLIM[60] augment the region proposals by
grouping and the mosaic arrangement are shown to be effective approaches. In
this work, we tackle the problem from a perspective of generated region-text
pairs.

###  2.2 Training Object Detection with Synthetic Data

For generated image data, prior works utilizes model-free data augmentation
techniques such as simple copy-paste[14]. For an open-vocabulary detection
task, these methods are strongly limited by the lack of semantic diversity and
instance-level variations. Gen2Det[54] proposes to use grounding-based image
generation to acquire scene-centric images to enhance the detectors’ ability
on rare object categories in a long-tail categorical distribution. Similarly,
MosaicFusion[61] borrows an off-the-shelf diffusion model to generate object-
level images and mask annotations. InstaGen[12] aims to enable novel category
generalization by generating images that contains both base and novel object
categories as well as their bounding boxes. In general, these generation
models assume knowing the novel object categories in advance. On the contrary,
we propose to generate images under the same open-world setting to minimize
the leaking of information. CapDet[39] trains a dense captioning head as an
auxiliary learning objective. PCL[7] proposes to train an image captioning
model on the image-caption pairs and then generate region-level captions.
However, they typically work under a finite dataset that are not easily
scalable.

##  3 Generating Region-Text Pairs for Training OVD

###  3.1 Preliminaries

The OVD problem and our goal: Open-vocabulary object detection is first
proposed in OVR-CNN[64]. Formally, a detector is trained on a detection
dataset with a predefined set of base object categories
$\mathcal{C}\textsuperscript{base}$. Meanwhile, it leverages external image-
caption pairs with an abundant list of vocabulary
$\mathcal{C}\textsuperscript{open}$. During testing, the detector is expected
to detect arbitrary novel object categories
$\mathcal{C}\textsuperscript{novel}$, where
$\mathcal{C}\textsuperscript{base}\cap\mathcal{C}\textsuperscript{novel}=\emptyset$.
In a strict open-vocabulary setting, $\mathcal{C}$novel are only known in
testing.

Given an image-caption pair, our goal is to generate a set of region-text
pairs $\\{(r_{j},t_{j})\\}_{j\in[N]}$ , where $r_{j}$ denotes a region on
image bordered by a bounding box, and $t_{j}$ denotes the text (phrase) that
semantically aligns with $r_{j}$. Subsequently, we expect to use the region-
text pairs to train the open-vocabulary object detectors.

Framework overview. Illustrated in Fig.2, we start from image-caption pairs
with two basic pre-processings: a class-agnostic detector is applied to the
image to produce proposal boxes; a large language model [23] is employed on
the caption to identify tangible and physical phrases (see Appendix A). They
are subsequently input into the generation framework, where text-to-region is
executed by an inpainting guider followed by an inpainting model (Section
3.2), while region-to-text is conducted by a captioning model and a subsequent
selection step (Section 3.3). The generated region-text pairs incorporate
contrastive learning and the proposed localization-aware region-text
contrastive loss (Section 3.4), jointly training with detectors.

Figure 2: Framework overview. We generate region-text pairs from image-caption
pairs through T2R and R2T processes. Detectors are jointed trained via
localization-aware region-text contrastive loss.

###  3.2 Text-to-Region

Image-caption pairs ensure the generation inherits visual and semantic
richness. Although generating images from texts with the controllability of
layout has been widely researched in recent years [13, 46, 33, 65], the
generation of image regions from image-caption pairs remains underexplored.

####  3.2.1 Inpainting Image-Caption Pairs.

Naturally, we build upon image inpainting as it directly gives region-text
alignment while preserving a substantial proportion of the original image,
thus transferring the realism and diversity of the images to the generated
output, particularly in the context regions, which is critical within the
setting of open-vocabulary detection. Considering an image $I$, a phrase $t$,
and a specified proposal box $b$. An inpainting model, denoted as
$\mathcal{G_{I}}$, can replace the original visual content inside $b$ (region
$r$) with a newly generated region $\hat{r}$,

$\hat{r}=\mathcal{G_{I}}(I,(b,t))$ (1)

where $\hat{r}$ is aligned with $t$ semantically, while the rest of the image
$I\setminus r$ remains unchanged.

When inpainting an image-caption pair, we have acquired $N$ proposal boxes
$\mathcal{B}=\\{b_{1},b_{2},...,b_{N}\\}$ from the image and $M$ extracted
phrases $\mathcal{T}=\\{t_{1},t_{2},...,t_{M}\\}$ from the caption by the pre-
processing. Here, a preliminary step is to allocate proposal boxes and phrases
to get a harmonious layout. We recognize several characteristics of this task:
(1) There exists $t$ that is not related to any region in the image, and vice
versa. (2) A box $b$ could be of any shape and located in any context, yet in
natural images, regions with a semantic meaning may follow certain geometric
distributions.

With these considerations, we desire a new approach to scene-aware allocation
for inpainting that can sample a harmonious layout by allocating the
$\mathcal{B}$ and $\mathcal{T}$, based on its understanding of the scene. As
an example shown in Fig.3, the core challenge is to understand "Happy mother
and son playing in the snow" and allocate the phrases "mother", "son", and
"snow" to the proper boxes. It is worth noting that, for obtaining region-text
pairs, it’s unnecessary to inpaint a region with a phrase that replicates its
original visual content (e.g., by grounding). Instead, the preferred design is
that the inpainting process will flexibly conform to a distribution that is
contingent on the scene’s context and is consistent with what is typically
observed in the real world. We also tried random allocation and grounding-
based allocation and analyzed their drawbacks in Section 4.

####  3.2.2 Scene-Aware Inpainting Guider (SAIG)

We model the probability of allocating a pair ($b_{n}$, $t_{m}$) as a joint
probability $p_{nm}=P(b_{n},t_{m}|\text{scene})$, which is decomposed equally:

$P(b_{n},t_{m}|\textrm{scene})=P(t_{m}|b_{n},\textrm{scene})\times
P(b_{n}|\textrm{scene})$ (2)

Figure 3: The scene-aware allocation for inpainting is an underexplored but
essential task in our generation framework. We propose SAIG to allocate layout
with the awareness of the scene.

In Eq. 2, $P(t_{m}|b_{n},\text{scene})$ stands for the probability of phrase
$t_{m}$ to be picked for inpainting within $b_{n}$, while
$P(b_{n}|\text{scene})$ represents $b_{n}$’s existence in the scene. We
parameterize $P(t_{m}|b_{n},\text{scene})$ by a multi-modal multi-layer
bidirectional transformer encoder [57, 8], illustrated in Fig 3. We engage
both visual and textual modalities from the image-caption pair. For the visual
modality, we obscure the image within the specified proposal boxes and employ
the remaining background as a canvas. This canvas prevents the model from
gaining knowledge of the original content within the proposal boxes,
encouraging it to focus on flexible layout generation. The caption and canvas
are encoded by CLIP textual encoder ($E_{T}$) and visual encoder ($E_{V}$),
respectively, and projected onto the same visual-semantic space. To facilitate
scene understanding, we emphasize some caption phrases that have been
extracted in preprocessing, by individually encoding them through $E_{T}$. The
scene is thereby a set of tokens,
$\textrm{scene}=\\{E_{T}(\textrm{caption}),E_{V}(\textrm{canvas}),E_{T}(t_{1}),E_{T}(t_{2}),...,E_{T}(t_{m})\\}$.

The $b_{n}$ has a form of <x,y,w,h>. We first encode it through Fourier
embedding (FE)[43] and then project it to the same dimension as the other
tokens by a trainable multi-layer perception (MLP), formally,
$E_{B}(b_{n})=\textrm{MLP}(\textrm{FE}(b_{n}))$. All encoded modalities are
incorporated into the transformer layers with each consisting of a multi-head
self-attention block (MHA), an MLP layer, and LayerNorm [1]. The output token
of the $b_{n}$ embedding is utilized to conduct dot product with encoded
texts, followed by softmax function to calculate the probability that text
$t_{m}$ should be inpainted in $b_{n}$,

$P(t_{m}|b_{n},\textrm{scene})=\frac{\exp(\textrm{MHA}(E_{B}(b_{n}),\textrm{scene})\cdot
E_{T}(t_{m}))}{\sum_{j=1}^{M}\exp(\textrm{MHA}(E_{B}(b_{n}),\textrm{scene})\cdot
E_{T}(t_{j}))}$ (3)

Furthermore, to get $P(b_{n}|\text{scene})$, we directly leverage the
confident score from the class-agnostic detector used in pre-processing to
reflect the probability of $b_{n}$’s existence in the scene. Such wise, we
could finally use Eq. 2 to calculate $P(b_{n},t_{m}|\text{scene})$. We use the
$P(b_{n},t_{m}|\text{scene})$ to sample diverse and flexible layouts, based on
nucleus sampling [20].

####  3.2.3 Filtering

The SAIG provides us with allocated layouts that guide image inpainting model
to generate region-text pairs. The generated images may contain low-quality
regions and it’s important to have quality control. We apply two levels of
filtering: image level and region level. We follow Gen2Det[54] and run an
image aesthetic model[51] on the generated data. We find that low-scored data
is usually low-quality, while very high-scored data is mostly landscape
painting and natural scenery, neither are ideal for instance-learning.
Additionally, we apply CLIP as a region-level filter on each region-text pair.
Appendix A provides addition implementation details.

###  3.3 Region-to-Text

We further introduce region-to-text generation to augment the textual richness
of the region proposals. The image-caption pairs we utilize are mostly sourced
from the web, which often results in captions that are erroneous, incomplete,
or only partially related to the image subjects. So a large portion of the
original captions only capture merely one or two salient entities instead of
mentioning all the semantic details, while some of the captions are simply not
directly related to the subject of the image. We aim to fully leverage the
potential of these image-caption pair data by generating region-level
descriptions via an image captioning model trained in a distinct domain, thus
enriching the overall system with semantic details at a granular level. The
resulting generated data is both format-compliant and complementary to the
text-to-region counterpart.

Given an image $I$, we first obtain regions $\\{r_{1},r_{2},...,r_{N}\\}$ by
cropping the image with enlarged proposal boxes $\mathcal{B}$ which include
context for enriched background information. To prevent semantic overlaps and
duplicated annotations, all proposal boxes are initially processed by Non-
Maximum Suppression. A pre-trained image captioning model [27]
$\mathcal{G}_{T}$ is applied to generate a set of region-level descriptions
$\mathcal{T}=\\{t_{1},t_{2},...t_{N}\\}$, where prompt is used to guide the
model to interpret the image,

$\hat{t}_{i}=\textrm{select}(\mathcal{G_{T}}(r_{i},\textrm{prompt}_{1}),\mathcal{G_{T}}(r_{i},\textrm{prompt}_{2}),\mathcal{G_{T}}(r_{i},\textrm{prompt}_{3}))$
(4)

In detail, $\hat{t}_{i}$ are generated by a selecting operation across an
ensemble of three text prompt prefixes, for example, "The image shows < >". As
a result, we select the best matching caption for each region proposal
according to the highest ranking CLIP similarity score between the region
crops and their generated captions. It is predictable that the more prompts we
select from, the higher score we will get, but in practice we find three is a
good balance between efficiency and effectiveness.

###  3.4 Training OVD with Region-Text Pairs

Contrastive learning is recently used in OVD [29, 24, 68, 60] to force visual
features to be similar to their textual features. Here, we first introduce
region-text contrastive learning, then we expand it to learn additional object
proposals tailored with different localization qualities.

Region-text contrastive loss. Given an image-caption pair, for $i$-th region
$r_{i}$ we operate ROIAlign[18] on detector’s feature pyramid to extract
visual embedding $E_{R}(r_{i})$, and use CLIP pre-trained language model as
our text encoder to get the corresponding text embedding $E_{T}(t_{i})$. The
pair $(r_{i},t_{i})$ is recognized as a positive pair. During training, we
also maintain a text queue $T^{*}=\\{t^{*}_{l}\\}_{l\in[L]}$ with a queue
length $L$, collected across previous batches. Texts in the queue are assumed
dissimilar to $t_{i}$, and they make the negative pairs with $r_{i}$, i.e.,
$\\{(r_{i},t^{*}_{1})$, $(r_{i},t^{*}_{2})$, …, $(r_{i},t^{*}_{L})\\}$.
According to [41, 60], a binary cross-entropy loss is applied,

$\mathcal{L}_{\textrm{region-
text}}=-\operatorname{log}\operatorname{\sigma}\left[\tau\cdot\text{cos}(E_{R}(r_{i}),E_{T}(t_{i}))\right]-\sum_{t^{*}_{l}\in
T^{*}}\operatorname{log}\left[1-\operatorname{\sigma}\left(\tau\cdot\text{cos}(E_{R}(r_{i}),E_{T}(t^{*}_{l}))\right)\right]\,$
(5)

where cos is cosine similarity, $\tau$ denotes a temperature parameter, and
$\sigma$ is sigmoid function.

Localization-aware region-text contrastive loss (LART). Eq.5 only considers
aligning $r_{i}$ and $t_{i}$, but neglects the importance of precisely
localized alignment. As a detector may densely predict many proposals to one
single object, it is critical to make the model give the highest confidence
rank to the most accurately localized prediction. To involve the awareness of
localization quality in contrastive learning, we propose LART. Starting with
$(r_{i},t_{i})$, we first obtain $K$ adjacent regions that overlaps with
$r_{i}$. These adjacent regions can be acquired from the region proposal
networks or dense predictions. We extract their visual embedding
$\\{E_{R}(r^{*}_{1}),...,E_{R}(r^{*}_{K})\\}$ and compute their intersection-
over-union (IOU) scores $\\{s_{1},...,s_{K}\\}$ with $r_{i}$ as localization
quality.

If a $s_{k}$ is higher than a predefined threshold $\alpha$, the corresponding
$r^{*}_{k}$ contains similar information as $r_{i}$ does, and a positive pair
$(r^{*}_{k},t_{i})$ is formed. They are trained akin to that of the
$(r_{i},t_{i})$, but their learning loss is down-weighted by $s_{k}$. This
benefits from two perspectives: on one hand, additional positive pairs
effectively enlarge the batch size and bring additional supervision; on the
other hand, the rescaled loss guarantees the strongest supervision is applied
to the origin pair, thus helping the detector confidently predict the optimal
localization. If $s_{k}<\alpha$, the $r^{*}_{k}$ contains a relatively small
proportion of information from $t_{i}$. So $r^{*}_{k}$ is both negative to
$t_{i}$ and $T^{*}$. Especially, the negative pair $(r^{*}_{k},t_{i})$
distinguishes itself as the $r^{*}_{k}$ is derived from $r_{i}$ rather than
from disparate regions, thereby yielding hard-negative examples for more fine-
grained learning. Similarly, we obtain:

$\mathcal{L}_{\text{adjacent}}=-\\!\sum_{\begin{subarray}{c}k\in[K],\\\
s_{k}\\!>\\!\alpha\end{subarray}}s_{k}\operatorname{log}\operatorname{\sigma}\left[\tau\text{cos}(E_{R}(r^{*}_{k}),E_{T}(t_{i}))\right]-\\!\sum_{\begin{subarray}{c}k\in[K],\\\
s_{k}\\!<\\!\alpha\end{subarray}}\sum_{t_{j}\in\\{t_{i},T^{*}\\}}\operatorname{log}\left[1\\!-\\!\operatorname{\sigma}\left(\tau\text{cos}(E_{R}(r^{*}_{k}),E_{T}(t_{j}))\right)\right]$
(6)

The overall objective for LART is thus
$\mathcal{L}_{\text{LART}}=\mathcal{L}_{\text{adjacent}}+\mathcal{L}_{\text{region-
text}}$.

Overall training objective. With Faster-RCNN [49] and CenterNet2 [69], we
train the detectors parallelly on the detection data $D^{\text{det}}$ and
generated data $D^{\text{T2R}},D^{\text{R2T}}$. Particularly, we treat the
image-caption pairs $D^{\text{cap}}$ as a special region-text pair and add
them into training. The overall training objective for the detectors is:

$\mathcal{L}_{\text{overall}}=\begin{cases}\mathcal{L}_{\text{rpn}}+\mathcal{L}_{\text{reg}}+\mathcal{L}_{\text{cls}}&\text{if}I\in
D^{\text{det}}\\\
\mathcal{L}_{\text{LART}}&\text{if}I\in\\{D^{\text{cap}},D^{\text{T2R}},D^{\text{R2T}}\\}\end{cases}$
(7)

Table 1: Comparison with state-of-the-art methods on OV-LVIS benchmark. CLIP supervision entails the transfer of knowledge from CLIP model. Caption supervision entails the learning of vision-language alignment through image-text pairs. The ’Strict OV’ denotes whether the experiments is in strict open-vocabulary setting. Method | Supervision | Backbone | Strict OV? | $\text{AP}_{\text{novel}}$ | $\text{AP}_{\text{c}}$ | $\text{AP}_{\text{f}}$ | $\text{AP}_{\text{all}}$
---|---|---|---|---|---|---|---
ViLD[16] | CLIP | RN50-FPN | ✓ | 16.6 | 24.6 | 30.3 | 25.5
RegionCLIP[67] | Caption | RN50-C4 | ✓ | 17.1 | 27.4 | 34.0 | 28.2
DetPro[9] | CLIP | RN50-FPN | ✓ | 19.8 | 25.6 | 28.9 | 25.9
Detic[68] | Caption | RN50 | ✗ | 19.5 | - | - | 30.9
OV-DETR[63] | Caption | RN50-C4 | ✗ | 17.4 | 25.0 | 32.5 | 26.6
PromptDet[10] | Caption | RN50-FPN | ✗ | 19.0 | 18.5 | 25.8 | 21.4
VLDet[34] | Caption | RN50 | ✓ | 21.7 | 29.8 | 34.3 | 30.1
F-VLM[26] | CLIP | RN50-FPN | ✓ | 18.6 | - | - | 24.2
BARON[59] | CLIP | RN50-FPN | ✓ | 22.6 | 27.6 | 29.8 | 27.6
CoDet[41] | Caption | RN50-FPN | ✓ | 23.4 | 30.0 | 34.6 | 30.7
CLIM[60] | Caption | RN50-FPN | ✓ | 21.8 | 28.4 | 32.0 | 28.7
Ours | RTGen | RN50-FPN | ✓ | 23.9 | 28.5 | 31.8 | 29.0
RegionCLIP[67] | Caption | R50x4 (87M) | ✓ | 22.0 | 32.1 | 36.9 | 32.3
Detic[68] | Caption | SwinB (88M) | ✗ | 23.9 | 40.2 | 42.8 | 38.4
VLDet[34] | Caption | SwinB (88M) | ✓ | 26.3 | 39.4 | 41.9 | 38.1
F-VLM[26] | CLIP | R50x4 (87M) | ✓ | 26.3 | - | - | 28.5
CoDet[41] | Caption | SwinB (88M) | ✓ | 29.4 | 39.5 | 43.0 | 39.2
Ours | RTGen | SwinB (88M) | ✓ | 30.2 | 39.9 | 41.3 | 38.8

##  4 Experiments

###  4.1 Dataset and Implementation

OV-LVIS is a widely-used benchmark for OVD on the LVIS dataset[17]. It
contains a total 1203 object categories that are separated into 866 common and
frequent objects and 337 rare objects, where the common + frequent are used as
base categories and the remaining rare classes as novel objects. The
evaluation metric is mask AP of the novel categories.

OV-COCO is a common benchmark originating from the MSCOCO benchmark [36, 64].
Among the 80 categories, 48 are split into the base categories and 17 are
reserved as novel categories. The box $\text{AP}_{50}$ on the novel categories
is used to evaluate the performance.

Conceptual Captions[53] is an image-caption dataset that has $\sim$3.3M web-
crawled images paired with the raw text descriptions harvested from Alt-text
HTML attribute associated with web images. We conduct our generation process
on the available subset ($\sim$2.8M images) once, resulting $\sim$2.0M region-
text pairs from the T2R process after filtering, and $\sim$9.6M from the R2T
process.

Implementation: Unless otherwise specified, for OV-COCO benchmark we conduct
experiments using Faster R-CNN with ResNet50, and all models are trained for
45k iterations with AdamW optimizer. For OV-LVIS, we use CenterNet2[69] with
ResNet50[19] and Swin-B[38] trained for 90k iterations. Ablation study is
conducted on OV-COCO. Following [68], we initialize the models with trained
parameters on the base classes to achieve faster convergence. For SAIG, we
train it jointly on Flickr30k[45], VG Caption[25] and GQA[21]. For LART, we
simply choose $\alpha=0.5$ as $\text{AP}_{50}$ is a practical metric.

###  4.2 Results on Benchmarks

Table 1 shows our primary results on OV-LVIS. Training with RTGen outperforms
other SoTA methods, including those rely on knowledge transfer and mining
vision-language alignment from image-caption pairs. This is due to the
diversity and semantic richness of generated region-text pairs, and the
results is further verified by different backbones. The models supervised by
RTGen follow the strict open-vocabulary setting (Section 3.1) which guarantees
the results have no bias to generalize.

Table 3 shows our results on the OV-COCO benchmark. The effectiveness of RTGen
is consistently demonstrated across multiple OVD benchmarks. As a comparison
to effective data augmentation methods, Faster-RCNN model trained with our
proposed RTGen supervision outperforms the previous SoTA method CLIM. RTGen
also outperforms region-text mining methods such as VLDet and CoDet.

Table 2: Comparison with state-of-the-art on OV-COCO benchmark. Method | $\text{AP}^{novel}_{\text{50}}$ | $\text{AP}^{base}_{\text{50}}$ | $\text{AP}^{all}_{\text{50}}$
---|---|---|---
OVR-CNN[64] | 22.8 | 46.0 | 39.9
ViLD[16] | 27.6 | 59.5 | 51.3
RegionCLIP[67] | 26.8 | 54.8 | 47.5
Detic[68] | 27.8 | 47.1 | 42.0
VLDet[34] | 32.0 | 50.6 | 45.8
CoDet[41] | 30.6 | 52.3 | 46.6
CLIM][60] | 32.3 | - | -
RTGen (Ours) | 33.6 | 51.7 | 46.9

Table 3: Ablation study of RTGen on OV-COCO. The models are trained with
$\mathcal{L}_{region-text}$ if w/o LART.

CC3M | T2R | R2T | LART | $\text{AP}^{novel}_{\text{50}}$
---|---|---|---|---
|  |  |  | 2.9
✓ |  |  |  | 25.7
✓ | ✓ |  |  | 29.2(+3.5)
✓ |  | ✓ |  | 28.1(+2.4)
✓ | ✓ | ✓ |  | 31.4(+4.7)
| ✓ | ✓ | ✓ | 31.6(+4.9)
✓ | ✓ | ✓ | ✓ | 33.6(+6.9)

Table 4: Generating vs. Pseudo-Labeling CC3M | LART | Pseudo | T2R+R2T | $\text{AP}^{novel}_{\text{50}}$
---|---|---|---|---
✓ |  |  |  | 25.7
✓ | ✓ | ✓ |  | 31.2
✓ | ✓ |  | ✓ | 33.6

Table 5: Queue Length of LART

Queue Length | $\text{AP}^{novel}_{\text{50}}$ | $\text{AP}^{novel}_{\text{75}}$
---|---|---
$L=512$ | 32.0 | 18.9
$L=125$ | 32.5 | 19.6
$L=256$ | 33.6 | 20.1

###  4.3 Ablation Studies

Impact of the generated data To understand the effectiveness of different
sources of supervision, we conduct experiments on the OV-COCO benchmark with
various combinations of data, as shown in Table 3. Our baseline is the
Detic[68] trained with image-caption pairs (CC3M) by treating entire image as
region. The models further trained with region-text supervision show
continuous improvements (+3.5 from T2R and +2.4 from R2T). Both R2T and T2R
can work jointly and bring the improvement to a new high (+4.7). Additionally,
Figure 1(c) illustrates a consistent improvement of generated data with an
increasing percentage of data involved, suggesting its scalability.

Impact of LART LART involves the awareness of localization in contrastive
learning. As shown in Table 3, the integration of LART yields a significant
contribution of an additional +2.2AP to the already high results (31.6).
Findings in Table 5 indicate that the queue length influences the training
outcomes: a longer queue length exacerbates the learning challenges,
disrupting the equilibrium between localization learning and open-vocabulary
recognition. Conversely, a short queue length is also not ideal for
contrastive learning.

###  4.4 Generating Region-Text Pairs vs. Pseudo Labeling

We compare generating region-text pairs with the pseudo-labeling. The pseudo
labels are predicted by a well-trained Grounding DINO[37] on the same CC3M
dataset, and the resulting region-text pairs are then used following the same
detection training recipe. In Table 5, pseudo-labeling is inferior to
generating. This can be attributed to two primary factors: 1. generating
derives from two distinct processes that involve separate models from
different domains, thereby enhancing the diversity and semantic richness of
the data. 2. the generation framework remains robust even when the alignment
between image-caption pairs is suboptimal, in contrast to pseudo-labeling,
which is significantly compromised under such circumstances.

Figure 4: (a) SAIG allocates the phrases "Man" and "a glass of beer" to more
suitable boxes according to the scene. (b) SAIG preserves more semantics
information during allocation than grounding.

###  4.5 Comparing Alternative Allocation Methods for Inpainting

Figure 5: Alignment quality of pairs.
${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\times}$
indicates the low $3\%$.

We analyze SAIG by comparing it with two alternative designs.

Design (a): Randomly allocating $\mathcal{B}$ and $\mathcal{T}$ is
straightforward but overlooks the intrinsic geometric property and graphical
relation, leading to hilarious outcomes (see Figure 4(a)). We quantify the
quality by measuring the CLIP similarity of generated 10K region-text pairs in
Figure 5, where the results on the LVIS val set are also presented as an upper
bound . Clearly, SAIG leads to better inpainting quality due to the reasonable
layout guidance.

Design (b): Grounding applies a pre-trained grounding model to pair the
regions with the phrases from the caption. However, our practical finding
reveals that grounding fails to capture a significant proportion of phrases
during the allocation, thereby reducing the semantic diversity. As depicted in
Figure 4(b), when comparing with Grounding DINO that pseudo-labels region-text
allocations, our method surpasses in capturing more noun phrases. This
comparison underscores the limitation of grounding, which frequently overlook
phrases due to low confidence scores and phrases absent in corresponding
images.

##  5 Conclusions and Limitations

In this work, we propose to generate region-text pairs for training open-
vocabulary object detection. Our approach innovates text-to-region and region-
to-text processes, along with the introduction of the Scene-Aware Inpainting
Guider and the Localization-Aware Region-Text Contrastive Loss. Experiments on
OV-COCO and OV-LVIS benchmarks show that detectors trained with generated data
outperform current OVD approaches, demonstrating its effectiveness. Moreover,
our method is orthogonal to previous methods that focus on achieving region-
text alignments, making generation methods a promising approach to reach good
performance. We recognize that training downstream computer vision tasks with
generated data is still not well explored and we encourage further
investigations in this promising direction.

## References

  * [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [3] Fangyi Chen, Han Zhang, Kai Hu, Yu-kai Huang, Chenchen Zhu, and Marios Savvides.  Enhanced training of query-based object detection via selective query recollection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23756–23765, 2023.
  * [4] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin.  MMDetection: Open mmlab detection toolbox and benchmark.  arXiv preprint arXiv:1906.07155, 2019.
  * [5] Zhaorun Chen, Zhuokai Zhao, Zhihong Zhu, Ruiqi Zhang, Xiang Li, Bhiksha Raj, and Huaxiu Yao.  Autoprm: Automating procedural supervision for multi-step reasoning via controllable question decomposition, 2024.
  * [6] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.  Reproducible scaling laws for contrastive language-image learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818–2829, 2023.
  * [7] Han-Cheol Cho, Won Young Jhoo, Woohyun Kang, and Byungseok Roh.  Open-vocabulary object detection using pseudo caption labels.  ArXiv, abs/2303.13040, 2023.
  * [8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [9] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li.  Learning to prompt for open-vocabulary object detection with vision-language model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14084–14093, 2022.
  * [10] Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma.  Promptdet: Towards open-vocabulary detection using uncurated images.  In European Conference on Computer Vision, pages 701–717. Springer, 2022.
  * [11] Chengjian Feng, Yujie Zhong, Zequn Jie, Weidi Xie, and Lin Ma.  Instagen: Enhancing object detection by training on synthetic dataset.  arXiv preprint arXiv:2402.05937, 2024.
  * [12] Chengjian Feng, Yujie Zhong, Zequn Jie, Weidi Xie, and Lin Ma.  Instagen: Enhancing object detection by training on synthetic dataset.  CVPR, abs/2402.05937, 2024.
  * [13] Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Layoutgpt: Compositional visual planning and generation with large language models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [14] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph.  Simple copy-paste is a strong data augmentation method for instance segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2918–2928, 2021.
  * [15] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.
  * [16] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  arXiv preprint arXiv:2104.13921, 2021.
  * [17] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019.
  * [18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [19] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2015.
  * [20] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.  The curious case of neural text degeneration.  arXiv preprint arXiv:1904.09751, 2019.
  * [21] Drew A. Hudson and Christopher D. Manning.  Gqa: a new dataset for compositional question answering over real-world images.  ArXiv, abs/1902.09506, 2019.
  * [22] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International conference on machine learning, pages 4904–4916. PMLR, 2021.
  * [23] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  arXiv preprint arXiv:2310.06825, 2023.
  * [24] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.  Mdetr-modulated detection for end-to-end multi-modal understanding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780–1790, 2021.
  * [25] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International Journal of Computer Vision, 123:32 – 73, 2016.
  * [26] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova.  F-vlm: Open-vocabulary object detection upon frozen vision and language models.  arXiv preprint arXiv:2209.15639, 2022.
  * [27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In International conference on machine learning, pages 19730–19742. PMLR, 2023.
  * [28] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International conference on machine learning, pages 12888–12900. PMLR, 2022.
  * [29] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.  Grounded language-image pre-training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965–10975, 2022.
  * [30] Xiang Li, Haoyuan Cao, Shijie Zhao, Junlin Li, Li Zhang, and Bhiksha Raj.  Panoramic video salient object detection with ambisonic audio guidance.  In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’23/IAAI’23/EAAI’23. AAAI Press, 2023.
  * [31] Xiang Li, Yinpeng Chen, Chung-Ching Lin, Rita Singh, Bhiksha Raj, and Zicheng Liu.  Completing visual objects via bridging generation and segmentation.  CoRR, abs/2310.00808, 2023.
  * [32] Xiang Li, Jinglu Wang, Xiaohao Xu, Xiao Li, Bhiksha Raj, and Yan Lu.  Robust referring video object segmentation with cyclic structural consensus.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22236–22245, 2023.
  * [33] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.  Gligen: Open-set grounded text-to-image generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521, 2023.
  * [34] Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai.  Learning object-language alignments for open-vocabulary object detection.  arXiv preprint arXiv:2211.14843, 2022.
  * [35] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [36] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [37] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499, 2023.
  * [38] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9992–10002, 2021.
  * [39] Yanxin Long, Youpeng Wen, Jianhua Han, Hang Xu, Pengzhen Ren, Wei Zhang, Shen Zhao, and Xiaodan Liang.  Capdet: Unifying dense captioning and open-world detection pretraining.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15233–15243, 2023.
  * [40] Edward Loper and Steven Bird.  Nltk: The natural language toolkit.  arXiv preprint cs/0205028, 2002.
  * [41] Chuofan Ma, Yi Jiang, Xin Wen, Zehuan Yuan, and Xiaojuan Qi.  Codet: Co-occurrence guided region-word alignment for open-vocabulary object detection.  Advances in Neural Information Processing Systems, 36, 2024.
  * [42] Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming-Hsuan Yang.  Class-agnostic object detection with multi-modal transformer.  In 17th European Conference on Computer Vision (ECCV). Springer, 2022.
  * [43] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [44] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby.  Simple open-vocabulary object detection with vision transformers.  ECCV, 2022.
  * [45] Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, J. Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  International Journal of Computer Vision, 123:74 – 93, 2015.
  * [46] Jie Qin, Jie Wu, Weifeng Chen, Yuxi Ren, Huixia Li, Hefeng Wu, Xuefeng Xiao, Rui Wang, and Shilei Wen.  Diffusiongpt: Llm-driven text-to-image generation system.  arXiv preprint arXiv:2401.10061, 2024.
  * [47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [48] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [49] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in Neural Information Processing Systems, 28, 2015.
  * [50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [51] Christoph Schuhmann.  improved-aesthetic-predictor.  https://github.com/christophschuhmann/improved-aesthetic-predictor, 2022.
  * [52] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  * [53] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [54] Saksham Suri, Fanyi Xiao, Animesh Sinha, Sean Culatana, Raghuraman Krishnamoorthi, Chenchen Zhu, and Abhinav Shrivastava.  Gen2det: Generate to detect.  In Synthetic Data for Computer Vision Workshop@ CVPR 2024, 2023.
  * [55] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  Communications of the ACM, 59(2):64–73, 2016.
  * [56] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: A simple and strong anchor-free object detector.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(4):1922–1933, 2020.
  * [57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [58] Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy.  Aligning bag of regions for open-vocabulary object detection.  2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15254–15264, 2023.
  * [59] Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy.  Aligning bag of regions for open-vocabulary object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15254–15264, 2023.
  * [60] Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Wentao Liu, and Chen Change Loy.  Clim: Contrastive language-image mosaic for region representation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6117–6125, 2024.
  * [61] Jiahao Xie, Wei Li, Xiangtai Li, Ziwei Liu, Yew Soon Ong, and Chen Change Loy.  Mosaicfusion: Diffusion models as data augmenters for large vocabulary instance segmentation.  ArXiv, abs/2309.13042, 2023.
  * [62] Yifan Xu, Mengdan Zhang, Xiaoshan Yang, and Changsheng Xu.  Exploring multi-modal contextual knowledge for open-vocabulary object detection.  arXiv preprint arXiv:2308.15846, 2023.
  * [63] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy.  Open-vocabulary detr with conditional matching.  In European Conference on Computer Vision, pages 106–122. Springer, 2022.
  * [64] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang.  Open-vocabulary object detection using captions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14393–14402, 2021.
  * [65] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023.
  * [66] Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, BG Vijay Kumar, Anastasis Stathopoulos, Manmohan Chandraker, and Dimitris N Metaxas.  Exploiting unlabeled data with vision and language models for object detection.  In ECCV, pages 159–175. Springer, 2022.
  * [67] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.  Regionclip: Region-based language-image pretraining.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793–16803, 2022.
  * [68] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra.  Detecting twenty-thousand classes using image-level supervision.  In European Conference on Computer Vision, pages 350–368. Springer, 2022.
  * [69] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Probabilistic two-stage detection.  arXiv preprint arXiv:2103.07461, 2021.
  * [70] Chenchen Zhu, Fangyi Chen, Zhiqiang Shen, and Marios Savvides.  Soft anchor-point object detection.  In European Conference on Computer Vision, 2019.

##  Appendix A Appendix

###  A.1 Implementation Detail of the Scene-Aware Inpainting Guider

The SAIG is constructed with 32 layers of multi-head self-attention. We use
CLIP-Vit-L/14 as a feature extractor. The box encoder contains three full-
connected layers with SiLU activation function in between. The cross-entropy
loss is applied for training. AdamW with learning rate=1e-4 is chosen as
optimizer. We train the guider with 8xA100 GPUs for 12 epochs until it
converges.

###  A.2 Implementation Detail of the Proposal Box Generator

We look for regions of interest in the images before applying the generation
models. Specifically, we use an off-the-shelf class-agnostic object proposal
generator MViTs[42] to predict object proposals with the text prompts "all
objects" and "all entities". Regions with a confidence score above 0.3 are
kept and ensembled. To avoid repetitive region proposals, all regions are
first filtered by the NMS process with a 0.1 IoU threshold.

###  A.3 Implementation Detail of Phrases Parser

We use a large language model Mistral[23] and NLTK [40] word-tree to extract
phrases that are suitable for inpainting from captions. Specifically, we find
directly using a prompt like "please list tangible objects in the sentence"
often produces sub-optimal results and gives incorrect phrases such as
"beauty", "university", "sunday", and "nightmare". Therefore we choose an
intruct-finetuned variant Mistral-8x7B-Instruct-v0.2, where we prompt several
examples and use Prompt+Instruct [5] for in-context learning. The selected
examples and prompt template are shown in Table 6.

Afterwards, we use a word-tree to filter the extracted phrases by the
hierarchy with allowance and forbidden categories, summarized in Table 7. If a
phrase’s hypernym appears or disappears in both categories, it will be
dropped.

Table 6: Prompting LLM for Phrase Extraction Prompt:
---
Export the real-world objects with a physical body in the sentence, return
None if not found.
Instruct:
User: burger: pound of fries and some sauces, man talking on his smart phone
on the beach in
cloudy dark weather. Assistant: burger, fries, some sauces, man, smart phone.
User: medical team working together at night, taking care of patients
carefully on a hospital
ward. Assistant: mediacal team, patients.
User: night display of sculptures during olympic games. Assistant: sculptures.
User: where is the sea in space?. Assistant: None
Table 7: NLTK Hierarchy Hypernym Filter Allowance |  Forbidden
---|---
’physical entity’, ’food’, ’person’, ’living thing’, ’social group’, ’biological group’ |  ’measure’, ’atmosphere’, ’time’, ’activity’, ’phenomenon’, ’event’, ’meeting’, ’organization’, ’location’, ’land’, ’facility’

###  A.4 Filtering Generated Data

As explained, the generated images may contain low-quality regions and we want
to filter out these low quality generation before the training of the
detectors. We image-level filtering and region-level filtering. We apply an
aesthetic filter [51] and select the 95 percentile interval threshold $t_{1}$
and $t_{2}$ for all images. The images with aesthetic scores outside of the
range $(t_{1},t_{2})$ are filtered out. In our final implementation, we select
$t_{1}=3.0$ and $t_{2}=6.0$. Note that we also remove images with high
aesthetic scores because most of them contain natural scenery, which might not
be ideal for region-text alignment learning. Subsequently, we apply an
adaptive region-level filter to remove inpainted regions with poor quality and
we use a pretrained CLIP model [47] as a filter. Concretely, for a generated
region-text pair, we calculate the cosine similarity scores between the region
and all the text phrases. A region annotation will be filtered out if the
similarity score between the region and the correspondent text phrases is less
than the top 5% of all the text phrases. Empirically, we find the dynamic
threshold works better than a fixed threshold as it preserves text phrases
that might have multiple synonyms.

###  A.5 OVD Implementation Details

We highlight the hyperparameters and training configuration used for the OV-
COCO and OV-LVIS benchmark experiments. Following Detic [68] and CoDet [41],
we use a dataset ratio of 1:4 between detection/region-text dataset for fair
comparison. We consider the original CC3M [53] image-caption data as a
generalized case for the region-text pair data and concatenate it with R2T
data. Table 8 lists all details during training. We use the well-known
academic repository MMDetection [4] to build our training pipeline. All
experiments are conducted on a single server node with 8 Nvidia A100 GPUS.

Table 8: Hyperparameters for training OVD on OV-LVIS and OV-COCO. Benchmark | OV-LVIS | OV-COCO
---|---|---
Detection Framework | CenterNet2 | Faster RCNN
Optimizer | AdamW | SGD
Gradient clipping | True | True
Learning rate (LR) | 2e-4 | 2e-2
Total iterations | 90k | 45k
Warmup iterations | 10k | 2k
Decay schedule | CosineAnnealing | StepDecay[30k, 40k]
Step decay factor | – | 0.1$\times$
Data augmentation | LSJ, HorizontalFlip | HorizontalFlip
Batch size per GPU (detection) | 8 | 2
Batch size per GPU (R2T) | 16 | 4
Batch size per GPU (T2R) | 16 | 4
Resolution (detection) | 896 $\times$ 896 | 1333 $\times$ 800
Resolution (caption) | 448 $\times$ 448 | 667 $\times$ 400
Federated loss | True | False
Repeat factor sampling | True | False
$\mathcal{L}_{\text{LART}}$ weight | 1.0 | 1.0

###  A.6 Visualizations

We visualize our generated region-text pair as well as the detection results.
For region-text pair, each region is highlighted by a blue bounding box and
green text. On OV-COCO benchmark, we show detection results of both the base
classes and novel classes. On OV-LVIS, we can see that the detector is able to
generalize to many rare categories such as "clementine", "harmonium",
"joystick", "boom_microphone", etc.

Figure 6: Examples of T2R generation, zoom in better visualization.

Figure 7: Examples of R2T generation, zoom in better visualization.

Figure 8: Visualization of detection results on OV-COCO, base categories are
shown in blue boxes while red boxes are for novel categories.

Figure 9: Visualization of detection results on OV-LVIS, to avoid
clutteredness, only the novel categories are shown.

###  A.7 Notations and Acronyms

Table 9: A list of all notations Notation | Description
---|---
$C^{\text{base}}$ | base categories
$C^{\text{novel}}$ | novel categories
$C^{\text{open}}$ | open vocabulary categories
T2R | text to region generation
T2T | region to text generation
$D^{\text{det}}$ | detection dataset
$D^{\text{cap}}$ | image-caption pair dataset
$D^{\text{T2R}}$ | text to region generation dataset
$D^{\text{R2T}}$ | region to text generation dataset
$r_{i}$ |  $i$-th image region
$\hat{r}_{i}$ |  $i$-th inpainted image region
$r^{*}_{k}$ |  $k$-th adjacent region proposal
$t_{i}$ |  $i$-th text phrase
$\hat{t}_{i}$ |  $i$-th generated text phrase
$t^{*}_{l}$ |  $l$-th text phrase in text queue $T^{*}$
$\mathcal{G_{I}}$ | inpainting model
$\mathcal{G_{T}}$ | captioning model
$b_{i}$ |  $i$-th proposal box
$I$ | image
$\mathcal{B}$ | set of proposal boxes
$\mathcal{T}$ | set of extracted text phrases
$p_{nm}$ | joint probability to allocate a pair $(b_{n},t_{m})$
$E_{T}$ | CLIP textual encoder
$E_{V}$ | CLIP visual encoder
$E_{R}$ | region encoder from detectors
$T^{*}$ | text queue for contrastive learning
$L$ | text queue length
LART | Localization-aware region-text contrastive loss
$\sigma$ | sigmoid function
$\tau$ | temperature parameter used in contrastive loss
$s_{k}$ | localization quality of the $k$-th adjacent region
$\alpha$ | localization quality threshold