# Spectral density of interacting pairs in low-dimensional finite lattices

T. Chattaraj

(Received July 12, 2020, in final form August 29, 2020)

###### Abstract

The spectral density of bound pairs in ideal 1D, 2D and Bethe lattices is
computed for weak and strong interactions. The computations are performed with
Green’s functions by an efficient recursion method in real space. For the
range of interaction strengths within which bound states are predominantly
single pairs, the spectral profiles guide to the energy bandwidths where the
bound pairs can be maximized.

Key words: quasiparticles and collective excitations, metal-insulator
transitions, doublons, bound states

###### Abstract

Îáèñëåíî ñïåêòðàëüíó ãóñòèíó çâ’ÿçàíèõ ïàð â äåàëüíèõ îäíî-, äâîâèìðíèõ Áåòå
ãðàòêàõ äëÿ ñëàáêèõ òà ñèëüíèõ âçàìîäé. Îáèñëåííÿ çäéñíåíî ç äîïîìîãîþ ôóíêöé
Ãðíà åôåêòèâíèì ðåêóðåíòíèì ìåòîäîì â ðåàëüíîìó ïðîñòîð. Äëÿ äàïàçîíó ñèë
âçàìîä¿, â ìåæàõ ÿêîãî çâÿçàí ñòàíè ïåðåâàæíî îäèíàðíèìè ïàðàìè, ñïåêòðàëüí
ïðîôë ñêåðîâàí íà òàêó ïðîïóñêíó çäàòíñòü åíåðã¿, äå çâÿçàí ïàðè ìîæóòü áóòè
ìàêñèìçîâàí.

Ключов слова: êâàçàñòèíêè òà êîëåêòèâí çáóäæåííÿ, ïåðåõîäè ìåòàë-çîëÿòîð,
äóáëîíè, çâ’ÿçàí ñòàíè

##  1 Introduction

For the Hubbard model [1] in lattices, formation of composite objects can be
observed for both attractive and repulsive interactions [2]. A doublon is such
a composite object formed from two fermions. In Auger spectroscopy, certain
lines and lineshapes in the spectra can be explained within the formalism of
formation of bound pairs [3, 4, 5]. In general, pairs formed from interacting
particles show the properties different from constituent individual particles.
For strongly interacting particles, their lifetimes are generally high [6] and
depend on interaction strengths. These strongly correlated pairs are now
realized and observed in optical lattices within broad range of tuneable
interaction strengths [7]. Recently, the pairing phenomena have been also
observed in optical lattices for spin excitations [8] which are relevant for
excited states near the ground ferromagnetic state. The bound states that
arise in lattices from onsite and nearest neighbour interactions are also
related to solitons [9, 10].

Two interacting particles show nonclassical correlations depending on the
statistics of the particles [11]. The interaction changes the effective
statistics between particles. However, a clear mathematical account on this is
yet expected to emerge. The constraints of the two-particle density matrices
are also not known [12]. One can study the correlations from quantum walk
experiments and simulations which are fundamental building blocks of many-body
systems. The dynamics of particles in lattices changes significantly depending
on the binding. The quantum walk of correlated particles has promising
applications in quantum technologies [13, 14, 15].

A general understanding on the mechanism of superconductivity is also still
unresolved. The Hubbard model with onsite and nearest neighbour interactions
is mostly used for explaining the phenomena. Pairing of fermions with
attractive interaction is known to cause the mechanism [16, 17]. In two-
dimensional Hubbard model, the super-exchange in quasi two-dimensional oxides
and nearest neighbour coulomb interaction between polarons in organics [18] is
known to be responsible for the phenomena. It is suggestive that an unified
approach to the mechanisms may have relations to the two-particle bands with
weak and strong interactions within the range where two-particle bound pairs
have a higher density than the bound states with a larger number of particles.

The effects of magnetic fields on single particles or excitations in two-
dimensional lattices are well known from Hofstadter’s approach [19]. The
density of states for two particles for various interaction strengths had also
been computed before [20]. In optical lattices, the external magnetic fields
can be synthesized with periodic modulation of the lattice potentials [21, 22,
23]. This article specifically looks at the density of bound pairs for both
zero and non-zero magnetic fields at different interaction strengths. In the
context of condensed matter, the effective fields can also be realized through
doping with isotropic magnetic impurities.

The solution for the two-particle states are known exactly for strong
interaction cases which are described with continuum and bound states [24].
Within many particle environment, there can be several bound states and
interplay of these bound states with the continuum states are highly complex.
These many-particle bound states can appear at different energy windows with
different densities. Green’s functions provide a direct approach to study
these states at different interaction strengths. Recursion methods [25] are
known for efficient computation of these Green’s functions. Several other
methods have also been applied recently for such computations, e.g., exact
diagonalization with the method of Lanczos [26], DMRG [27] and so on. In this
article, the Green’s functions are computed using Berciu’s approach [28] which
is best described in the original article. A brief description is given in the
method section for the reader. With Berciu’s method, the Green’s functions can
be computed efficiently for the cases of zero and non-zero external magnetic
fields as well as for disordered systems [29]. The bound pairs can be found in
general graph architectures where the quantum transport may prove relevant to
quantum technologies. The Bethe lattice is one such graph for which the
density of bound pairs at different interaction strengths is considered in
this article. The Bethe lattice which is important for its mathematical form,
also has presence in cores of several photosynthetic complexes.

##  2 Method

In this article the density of a bound pair of two interacting spinless
particles is studied. The particles are taken as hardcore bosons in 1D and 2D
lattices and softcore bosons in Bethe lattice. The case of hardcore bosons is
similar to two spinless fermions. In 1D and 2D lattices, the particles get
bound with nearest neighbour interactions. For the models with zero nearest
neighbour interaction and non-zero onsite interaction, the same bound pairs
appear for spin paired particles. The simplified model Hamiltonian is that of
extended Hubbard model $\mathcal{H}_{e\text{H}}$ with the hopping limited to
the nearest neihbour sites on the lattices.

$\mathcal{H}_{e\text{H}}=\sum_{m}\epsilon_{m}a_{m}^{\dagger}a_{m}+\sum_{<mn>}t_{|m-n|}a_{m}^{\dagger}a_{n}+\sum_{m}Ua_{m}^{\dagger}a_{m}^{\dagger}a_{m}a_{m}+\sum_{<mn>}V_{mn}a_{m}^{\dagger}a_{n}^{\dagger}a_{n}a_{m}.$
(2.1)

The onsite interaction term is taken as infinite in 1D and 2D cases. The
nearest neighbour interaction term is neglected for the Bethe lattice. Onsite
energies are scaled to zero for regular lattice systems.

For a Hamiltonian $\mathcal{H}$, the Fourier transformed Green’s function is
defined as

$G(\omega)=\frac{1}{\omega-\mathcal{H}},$ (2.2)

where $\omega=E+\mathrm{i}\eta$ is a complex number with $\eta$ a very small
positive real number. In real space, the two particle Green’s function
$G_{2}(m,n,\omega)=\langle mn|G(\omega)|m^{\prime}n^{\prime}\rangle$ is the
propagator for two particles from sites $m^{\prime}$, $n^{\prime}$ to sites
$m$, $n$ in a lattice, where
$|mn\rangle=c_{m}^{\dagger}c_{n}^{\dagger}|0\rangle$ and the initial site
indices $m^{\prime}$, $n^{\prime}$ are omitted for brevity. The bound pair
spectral density can be calculated for the bound state from the Green’s
function $G_{2}(m^{\prime},n^{\prime},\omega)$ in real space with initial
distance $|m^{\prime}-n^{\prime}|=0$ and $1$ for softcore and hardcore bosons,
respectively

$A(m^{\prime},n^{\prime},E)=\frac{-1}{\piup}\mbox{Im}[G_{2}(m^{\prime},n^{\prime},E+\mathrm{i}\eta)].$
(2.3)

The total density of states (DOS) is obtained with summing over densities of
all the states

$\text{DOS}(E)=\frac{1}{2}\sum_{m^{\prime},n^{\prime}}A(m^{\prime},n^{\prime},E+\mathrm{i}\eta).$
(2.4)

On the graphs with translational symmetry, a few states with increasing
relative distance $(|m^{\prime}-n^{\prime}|)$ prove sufficient for converging
results.

Substituting equation (2.1) in equation (2.2) generates the recursion
relations between the Green’s functions

$\displaystyle(\omega-\epsilon_{m}-\epsilon_{n}-U\delta_{mn}-V_{mn}\delta_{mn\pm
1})G_{2}(m,n,\omega)=\delta_{m,m^{\prime}}\delta_{n,n^{\prime}}+\delta_{m,n^{\prime}}\delta_{n,m^{\prime}}-t_{m-1,m}G_{2}(m-1,n,\omega)$
$\displaystyle-
t_{m,m+1}G_{2}(m+1,n,\omega)-t_{n,n+1}G_{2}(m,n+1,\omega)-t_{n-1,n}G_{2}(m,n-1,\omega)$
(2.5)

for every site indices $m,n$ of the two particles. The Green’s functions with
constant $m+n$ are then grouped with a vector $\mathbf{V}_{R}$ ($R=m+n$) which
forms a 1D chain and generates the recursion equation

$\mathbf{V}_{R}=\bm{\alpha}_{R}\mathbf{V}_{R-1}+\bm{\beta}_{R}\mathbf{V}_{R+1}+\mathbf{C}\,,$
(2.6)

with $\bf C\neq\bf 0$ for $R=m^{\prime}+n^{\prime}=R^{\prime}$ containing the
initial state. The hopping matrices $\bm{\alpha}_{R}$, $\bm{\beta}_{R}$
connect Green’s functions between nearest neighbour vectors with the hopping
integrals. Solving this equation, any Green’s function can be computed for a
finite lattice. At the boundaries, equation (2.6) becomes

$\mathbf{V}_{0}=\bm{\beta}_{0}\mathbf{V}_{1}\mbox{ and
}\mathbf{V}_{L}=\bm{\alpha}_{L}\mathbf{V}_{L-1}\,,$ (2.7)

where $0$ and $L$ are the minimum and maximum indices for $R$. The recursion
modifies to this general form

$\mathbf{V}_{R}=\mathcal{A}_{R}\mathbf{V}_{R-1}\mbox{ and
}\mathbf{V}_{R}=\mathcal{B}_{R}\mathbf{V}_{R+1},\mbox{ for }R\neq R^{\prime},$
(2.8)

with

$\mathcal{B}_{R}=\left[1-\bm{\alpha}_{R}\mathcal{B}_{R-1}\right]^{-1}\bm{\beta}_{R}\mbox{
and
}\mathcal{A}_{R}=\left[1-\bm{\beta}_{R}\mathcal{A}_{R+1}\right]^{-1}\bm{\alpha}_{R}.$
(2.9)

These $\mathcal{A}_{R}$ and $\mathcal{B}_{R}$ matrices can be computed
recursively starting from equation (2.7) before one reaches $R=R^{\prime}$
from both sides of the chain with

$\displaystyle\mathbf{V}_{R^{\prime}}=\left[1-\bm{\alpha}_{R^{\prime}}\mathcal{B}_{R^{\prime}-1}-\bm{\beta}_{R^{\prime}}\mathcal{A}_{R^{\prime}+1}\right]^{-1}\mathbf{C}$
(2.10)

at $R=R^{\prime}$. Once $\mathbf{V}_{R^{\prime}}$ is found, all other
$\mathbf{V}_{R}$ are given by equation (2.8). The procedure accounts the full
self-energy term which can be obtained from renormalized perturbation
expansion [30] on any ordered or disordered lattice. This method can also be
used for the cases where sites have internal structures.

In presence of external magnetic field, a gauge field appears in the
Hofstadter Hamiltonian with the hopping terms which is known as Peierls
substitution [31].

$\displaystyle\mathcal{H}_{Hf}$ $\displaystyle=$
$\displaystyle\sum_{m}\left(t\mathrm{e}^{2\mathrm{i}\piup\frac{p}{q}m_{2}}a_{m_{1}+1}^{\dagger}a_{m_{1}}+ta_{m_{2}+1}^{\dagger}a_{m_{2}}+h.c.\right)$
(2.11) $\displaystyle+$
$\displaystyle\sum_{m}Ua_{m}^{\dagger}a_{m}^{\dagger}a_{m}a_{m}+\sum_{<mn>}V_{|m-n|}a_{m}^{\dagger}a_{n}^{\dagger}a_{n}a_{m}\,.$

For an ideal 2D lattice, the onsite energy terms can be scaled to zero with
$q=\infty$ in the phase for zero magnetic field. The phase term contributes
with the hopping on one of the axes of 2D lattice for non-zero magnetic field.
The subscripts ${1,2}$ of the site index $m$ denotes coordinates of two axes
of site $m$. These terms are simulated in optical lattices with periodic
modulation of lattice potentials [22]. The essence of the method depends on
the classical Floquet theory where the momentum terms gain phases.

##  3 Results

The spectral density of bound pairs can be obtained from Green’s functions
computed using a recursion method or Chebyshev polynomials or exact
diagonalization. The density of states had been computed before by Rausch et
al. [32], Halpap et al. [33] and others for one dimensional regular lattices.
The density of states for two-dimensional ideal lattices had been computed
before by Barelli et al. [20]. I use the method of recursion that has been
formulated recently[28] by Berciu et al. for the computation of the spectral
density of bound pairs at any interaction strength. In 1D, the recursive
method provides converging results for the Green’s functions with lattices of
size in the order of $N=100$ sites. The Green’s functions are computed for an
1D lattice with $N=500$ sites and $N=20\times 21$ sites in 2D. For the Bethe
lattice which resembles the structure of recursion, $L=8$ levels are
considered. The value of hopping elements are fixed with $t=1$ while the
interaction elements $U$ and $V$ are varied in the calculations.

Figure 1: (Colour online) Density of states of two interacting particles
(dotted lines) and spectral density of bound pairs (straight lines) for a 1D
lattice of $N=500$ computed from equation (2.3) and equation (2.4) to an
arbitrary scale with $\eta=0.01$.

The bound pair spectra in 1D have been recently studied by Rausch et. al for
strong interactions. I obtained the results for weak interactions using the
recursion which gives converging results at the band edges. The bound pairs
within continuum can have different transport properties because of the
scattering with the continuum states which I do not discuss in this article.
The density of bound pairs at $T=0$ is maximum near $E=V$. The density of
pairs at $V=0$ for noninteracting case is much smaller than for interacting
$V\neq 0$ cases. The bound state splits from the continuum at $V=W/2$ where
$W$ is the bandwidth of the two-particle continuum spectra. The bound state
shows a sharp peak near $E=V$ for weak interaction strengths ($V<W/2$). For
strong interaction cases, the density reflects the cosine dispersion curve of
bound states [24] near $E=V$. The total density of states is similar to the
density of states that is obtained analytically for the noninteracting single
particles in 2D lattices [30] except the satellite peaks which are clearly
observed for the strong interaction cases.

The bound pair density shows a similar structure of one bound state in 2D
lattices in presence of interaction. The density profiles are very similar to
that of 1D lattice for $V=0$. For weak interaction strengths, the pair density
reaches the maximum within a range of the bandwidth rather than a sharp peak
and decays to the band edge afterwords. For strong interactions, the pair
density shows a sharp maximum at $V>W/2$. In presence of external gauge
fields, splitting in the spectra can be observed from figure 2. The details of
the spectra then depend on the flux per plaquette $\frac{p}{q}$. At half flux
per plaquette for the non-interacting cases, the pair density is reminiscent
of Kondo profile. At strong interactions with half flux per plaquette, the
density is regular dome-shaped while at non-half fluxes, the density is highly
irregular. At weak interaction, the maximum density is shifted to higher $E$.
These density profiles will be reflected symmetrically about $E=0$ with
changes of sign in the interaction term $V\rightarrow-V$.

Figure 2: Spectral density of bound pairs on a 2D lattice of $N=20\times 21$
with $p=1$ computed from equation (2.3) for the Hofstadter Hamiltonian in
equation (2.11) with $U=\infty,t_{1}=1,t_{2}=1$ and $\eta=0.06$.

(0,1)(4,8)($\overline{4}$,8)

Figure 3: (Colour online) Spectral density of bound pairs at various onsite
interaction strengths on a Bethe lattice of $L=8$ with $\eta=0.1$. The Bethe
lattice is depicted on the right.

The bound pair state shows non-trivial features on Bethe lattices. At zero
onsite interaction $U=0$, the spectra show discontinuity and multiple sharp
peaks. At strong interactions, these peaks merge within a single envelope. The
pair density in figure 3 is obtained for two bosons, and the interactions
between particles at different nodes are neglected. The initial states are
prepared at the root layer $L=0$ with two bosons on the node ($L=0$, $n=1$).
For two particles, the recursion can be performed with $R=L_{1}+L_{2}$ adding
layer indices for the two particles on the Bethe lattice. For single
particles, the recursion vectors map directly to the branches of each layer
divided as $\overline{L}$ and $L$ as depicted in figure 3. The Green’s
functions can be then efficiently computed for any initial superposition of
states on these vectors, e.g., the rectangular box on the graph will be the
vector $\mathbf{V}_{1}$ in the recursion. The discontinuities of the spectra
for weak interactions within small energy width suggest tunable control of
bound pair transport on these graphs.

##  4 Conclusion

In this article the doublon spectra of regular 1D, 2D and Bethe lattices have
been computed from real space Green’s functions. In two dimensions, the
effects of external magnetic fields on the spectra have been computed for weak
and strong nearest neighbour interactions between two hardcore bosons. For
weak interactions, the gauge fields enhance the density at few bandwidths. In
Bethe lattices, the spectra show discontinuities for weak interactions within
the band. This work points to energy windows where maximum bound pairs can be
found in 1D, 2D and Bethe lattice systems with the change of interaction
strengths between particles. The work might be helpful for understanding the
many-particle states where bound pairs can lead to the difference in
responses, e.g., conductivity and spectroscopy of the lattice systems. The
scattering and dissipation of the bound pairs within continuum should also be
analyzed. This work computes the spectral profiles which provide insights into
the mechanism of interacting particles when two particle bound pair densities
are much higher than the bound states with a larger number of particles. Many-
particle bound states with a larger number of particles at different energies
should be studied further which can also be important for the systems with
strong interactions.

### Acknowledgements

The author acknowledges funding support from NSERC, Canada.

## References

  * [1] Hubbard J., Proc. R. Soc. London, Ser. A, 1963, 276, 238, doi:10.1098/rspa.1963.0204.
  * [2] Winkler K., Thalhammer G., Lang F., Grimm R., Hecker Denschlag J., Daley A.J., Kantian A., Büchler H.P., Zoller P., Nature 2006, 441, 853, doi:10.1038/nature04918.
  * [3] Sawatzky G.A., Phys. Rev. Lett. 1977, 39, 504, doi:10.1103/PhysRevLett.39.504.
  * [4] Sawatzky G.A., Lenselink A.L., Phys. Rev. B. 1980, 21, 1790, doi:10.1103/PhysRevB.21.1790.
  * [5] Cini M., J. Phys.: Condens. Matter, 1989, 1, SB55, doi:10.1088/0953-8984/1/SB/011.
  * [6] Strohmaier N., Greif D., Jördens R., Tarruell L., Moritz H., Esslinger T., Sensarma R., Pekker D., Altman E., Demler E., Phys. Rev. Lett., 2010, 104, 080401, doi:10.1103/PhysRevLett.104.080401.
  * [7] Preiss P.M., Ma R., Tai E.M., Lukin A., Rispoli M., Zupancic P., Lahini Y., Islam R., Greiner M., Science, 2015, 347, 1229, doi:10.1126/science.1260364.
  * [8] Fukuhara T., Schauß P., Endres M., Hild S., Cheneau M., Bloch I., Gross C., Nature, 2013, 502, 72,
doi:10.1038/nature12541.

  * [9] Klein A., Krejs F., Phys. Rev., 1976, 13, 3282.
  * [10] Miller P.D., Scott A.C., Carr J., Eilbeck J.C., Phys. Scr., 1991, 44, 509, doi:10.1088/0031-8949/44/6/002.
  * [11] Qin X., Ke Y., Guan X., Li Z., Andrei N., Lee C., Phys. Rev. A, 2014, 90, 062301,
doi:10.1103/PhysRevA.90.062301.

  * [12] McWeeny R., Rev. Mod. Phys., 1960, 32, 335.
  * [13] Childs A.M., Goldstone J., Phys. Rev. A, 2004, 70, 022314, doi:1050-2947/2004/70(2)/022314(11).
  * [14] Childs A.M., Gosset D., Web Z., Science, 2013, 339, 791, doi:10.1126/science.1229957.
  * [15] Peruzzo A., Lobino M., Matthews J.C.F., Matsuda N., Politi A., Poulios K., Zhou X.-Q., Lahini Y., Ismail N., Wörhoff K., Bromberg Y., Silberberg Y., Thompson M.G., OBrien J.L., Science, 2010, 329, 1500, doi:10.1126/science.1193515.
  * [16] Bardeen J., Cooper L.N., Schrieffer J.R., Phys. Rev., 1957, 108, 1175, doi:10.1103/PhysRev.108.1175.
  * [17] Micnas R., Ranninger J., Robaskiewicz S., Rev. Mod. Phys., 1990, 62 113, doi:10.1103/RevModPhys.62.113.
  * [18] Majumdar S., Solid State Commun. 1988, 66, 427, doi:10.1016/0038-1098(88)90870-8.
  * [19] Hofstadter D.R., Phys. Rev. B 1976, 14, 2239, doi:10.1103/PhysRevB.14.2239.
  * [20] Barelli A., Bellissard J., Jacquod P., Shepelyansky D.L., Phys. Rev. B, 1997, 55, 9524,
doi:10.1103/PhysRevB.55.9524.

  * [21] Lin Y.-J., Compton R.L., Jiménez-García K., Porto J.V., Spielman I.B., Nature, 2009, 462, 628,
doi:10.1038/nature08609.

  * [22] Dalibard J., Gerbier F., Juzeliũnas G., Öhberg P., Rev. Mod. Phys., 2011, 83, 1523,
doi:10.1103/RevModPhys.83.1523.

  * [23] Struck J., Ölschläger C., Weinberg M., Hauke P., Simonet J., Eckardt A., Lewenstein M., Sengstock K., Windpassinger P., Phys. Rev. Lett., 2012, 108, 225304, doi:10.1103/PhysRevLett.108.225304.
  * [24] Valiente M., Petrosyan D., J. Phys. B: Mol. Opt. Phys., 2008, 41, 161002, doi:10.1088/0953-4075/41/16/161002.
  * [25] Pettifor D.G., Weaire D.L., The Recursion Method and Its Applications, Springer-Verlag, London, 1984, doi:10.1007/978-3-642-82444-9.
  * [26] Lanczos C., J. Res. Nat. Bur. Stand., 1950, 45, 255, doi:10.6028/jres.045.026.
  * [27] Schollowöck U., Ann. Phys., 2011, 326, 96, doi:10.1016/j.aop.2010.09.012.
  * [28] Berciu M., Cook A.M., EPL, 2010, 92, 40003, doi:10.1209/0295-5075/92/40003
  * [29] Chattaraj T., Condens. Matter, 2018, 3, 38, doi:10.3390/condmat3040038.
  * [30] Economou E.N., Green’s Functions in Quantum Physics, Springer-Verlag, London, 2006.
  * [31] Peierls R., Z. Phys., 1933, 80, 763.
  * [32] Rausch R., Potthoff M., New J. Phys., 2016, 18, 023033, doi:10.1088/1367-2630/18/2/023033.
  * [33] Halfpap O., Zharekeshev I.K., MacKinnon A., Kramer B., Ann. Phys., 1998, 7, 503,
doi:10.1002/(SICI)1521-3889(199811)7:5/6%3C503::AID-ANDP503%3E3.0.CO;2-L.

Ukrainian 0

Ñïåêòðàëüíà ãóñòèíà âçàìîäþèõ ïàð ó íèçüêîâèìðíèõ ñêíåíèõ ãðàòêàõ T. àòòàðàé

Óíiâåðñèòåò Áðèòàíñüêî¿ Êîëóìái¿ (UBC), Âàíêóâåð, Êàíàäà


# Hyperbolic Learning with Synthetic Captions for Open-World Detection

Fanjie Kong1 , Yanbei Chen2 , Jiarui Cai2, Davide Modolo2
1Duke University 2AWS AI Labs
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work done during an
internship with AWS AI Labs.Corresponding author.

###### Abstract

Open-world detection poses significant challenges, as it requires the
detection of any object using either object class labels or free-form texts.
Existing related works often use large-scale manual annotated caption datasets
for training, which are extremely expensive to collect. Instead, we propose to
transfer knowledge from vision-language models (VLMs) to enrich the open-
vocabulary descriptions automatically. Specifically, we bootstrap dense
synthetic captions using pre-trained VLMs to provide rich descriptions on
different regions in images, and incorporate these captions to train a novel
detector that generalizes to novel concepts. To mitigate the noise caused by
hallucination in synthetic captions, we also propose a novel hyperbolic
vision-language learning approach to impose a hierarchy between visual and
caption embeddings. We call our detector “HyperLearner”. We conduct extensive
experiments on a wide variety of open-world detection benchmarks (COCO, LVIS,
Object Detection in the Wild, RefCOCO) and our results show that our model
consistently outperforms existing state-of-the-art methods, such as GLIP,
GLIPv2 and Grounding DINO, when using the same backbone.

##  1 Introduction

Figure 1:  We tackle the task of detecting seen and unseen objects using
keywords (e.g., panda) or free-form texts (e.g., a black-and-white giant bear)
in open world. We exploit synthetic captions from pre-trained caption models
to bring rich open-world knowledge for training. As synthetic captions may be
noisy, we propose to align visual features with text embeddings in a
structural hierarchy to learn robustly and effectively from these captions.

An intelligent perception model of our visual world requires the open-world
generalizability of understanding any object. To enable such generalizability,
recent advances in object detection integrate vision models with natural
language models to enrich their open-world knowledge [60, 65, 15, 50, 63, 24].
Most of these works [60, 65, 15, 50] consider the open-vocabulary setting in
which a novel object is specified by its class label (e.g., panda). Other
works [63, 24] cast detection as a generalized localization task and consider
a more challenging open-set setting in which objects are described by free-
form texts (e.g., a black-and-white giant bear). In this work, we tackle the
task of detecting and localizing any object using both class labels and free-
form texts (Figure 1). We refer to this challenging task as “open-world
detection”.

To improve generalization in open world, existing detection methods [60, 29,
63] utilize language descriptions from large-scale image captioning datasets
(e.g., MS-COCO captions [32], conceptual captions [46]) for training. These
descriptions are however expensive to manually annotate [32] or design human-
crafted data cleaning pipeline to process alt-text [46]. To overcome this
limitation, we propose to leverage recent advanced vision-language models to
bootstrap machine-generated captions automatically. We collect these synthetic
captions on different regions of an image to generate a dense understanding of
all its concepts: from rich semantic information of known/seen objects (e.g.,
their attributes, states, actions and interactions) to diverse open-vocabulary
descriptions of new concepts. Our intuition is that by aligning the detector
features with these rich semantics, we can boost the model capability of
understanding and recognizing any object, including novel ones.

Nevertheless, one needs to be careful when using synthetic captions, as they
are sometimes hallucinated or unrelated to the objects in the image [44]. For
instance, given an image of a panda (Figure 1), the model may generate the
caption: “a furry black-and-white bear is sitting and eating bamboo”,
hallucinating the action “eating bamboo”, which usually co-occurs with the
animal panda. To mitigate the negative impact of aligning visual features with
noisy captions, we propose to learn visual-textual alignment in a hierarchical
structure, in which the synthetic caption entails the keyword of object, along
with other semantics. We refer to this hierarchy structure as ‘caption entails
object’. To impose such a structure in our representation space, we propose a
novel hyperbolic vision-language learning objective which ensures that the
caption embeddings entails the visual object embeddings in the representation
space. Our formulation is inspired by the advances of structural
representation learning with hyperbolic geometry [9, 22, 12, 13, 8]. Rather
than improving the unimodal representation learning on purely images [13] or
texts [9], our hyperbolic learning objective is introduced for cross-modal
learning on images and synthetic captions, and especially addresses the
hallucination problem of synthetic captions to advance the open-world
generalization in detection.

To summarize, our contribution is three folds:

  * (1)

We propose to boost open-world detection using synthetic captions generated by
powerful pre-trained VLMs. These synthetic captions provide rich language
descriptions of both seen and unseen objects, enabling the detection of both
new class labels and free-form texts.

  * (2)

We formulate a novel hyperbolic vision-language learning objective to help
learning robust representation using noisy synthetic captions. Our hyperbolic
loss formulation imposes a meaningful structural hierarchy in the
representation space to specifically tackle the hallucination problem in these
captions.

  * (3)

We show that our approach achieves the state-of-the-art performance on a
variety of detection and localization datasets in the open-world setting,
including COCO, LVIS, Object Detection in the Wild (ODiW), and RefCOCO. Our
approach outperforms existing open-world detectors such as GLIP, GLIPv2, and
Grounding DINO when using the same backbone and datasets for training.

##  2 Related Work

Open-vocabulary detection (OVD). Standard detection models like Faster R-CNN
[14] and DETR [1] succeed in detecting objects from a fixed vocabulary, but
are not capable of localizing novel concepts. To overcome this limitation, OVD
methods take advantage of extensive image-text datasets through vision-
language models (VLMs). OVD approaches follow two trends: learning novel
object concepts from image-level supervision [65], and transferring knowledge
from VLMs, which includes distilling visual-semantic cues [15, 50], aligning
image-text representations [59, 64, 51], and exploiting pseudo-labels from
captioning models [4, 28]. In line with the more contemporary trend, our
approach seeks to assimilate novel semantic concepts by probing VLMs.
Specifically, we focus on distilling knowledge from large amount of (noisy)
captions generated by these VLMs. Moreover, we also delve into the integration
of free-form texts in open-world localization, lifting the model’s proficiency
in understanding complex referring expressions beyond plain object classes.

Vision-language models (VLMs) in detection. VLMs have demonstrated remarkable
abilities in tasks like image classification and image-text retrieval [41, 7,
19, 55, 52, 61]. Recently, VLMs have also pioneered the state-of-the-art in
detection, as represented by GLIP [29, 63], RegionCLIP [64] and Grounding DINO
[34]. These methods use large-scale image-text datasets to train the detector,
thus enhancing the knowledge of both seen and unseen concepts. Concurrently,
other studies [15, 50, 65, 3] have focused on exploiting the visual-semantic
knowledge from VLMs, e.g., Detic [65] and ScaleDet [3] use the text encoder
from CLIP to encode the class labels of any seen and unseen objects. Our work
contributes to this evolving direction with a novel hyperbolic vision-language
learning approach to learn with synthetic captions from VLMs, which avoids the
need of expensive manual annotated image-text datasets.

Learning with captions. Image captions convey rich information about objects
in an image, such as object attributes, actions, relationships and scenes.
Recent works have highlighted the advantages of harnessing caption data to
enhance visual representation learning [7, 56, 48, 53, 11]. The majority of
these studies exploit manual annotated captions, training visual encoders
alongside auxiliary caption generators to learn more generalized visual
features [7, 56, 48]. A few studies investigate the use of machine generated
captions to augment supervision for visual learning [53, 11]. Our approach
takes an innovative step further by employing synthetic captions to improve
open-world generalization in detection. Importantly, we tackle the
hallucination issue that often presents in synthetic captions by introducing a
novel hyperbolic loss formulation.

Hyperbolic learning. Representation learning based on hyperbolic geometry has
shown considerable success in various domains such as text embeddings [9],
text generation [5] and knowledge graph modeling [2], as the hyperbolic
geometry could model the intrinsic hierarchy among data. Recently, a few works
explore hyperbolic learning in vision tasks to model the hierarchical relation
between objects and scenes [13], or images and texts [8]. Our work is built on
the foundation of hyperbolic learning to advance performance in detection. We
specially delve into modelling the hierarchy between visual objects and
synthetic captions, and formulate a novel hyperbolic learning objective to
learn upon such hierarchy.

Figure 2: Approach overview. Given an image, our open-world detector extracts
visual feature embeddings on region crops, and aligns visual feature
embeddings with text embeddings extracted from a pre-trained text encoder
(§3.1). To obtain synthetic captions for both seen and novel objects, we
propose region sampling to augment region crops, and adopt a pre-trained image
captioner to bootstrap synthetic captions on these region crops (§3.2). To
learn from synthetic captions effectively, we propose to align visual feature
embeddings and caption embeddings in a structural hierarchy through hyperbolic
vision-language learning in the hyperbolic space (see Figure 3, §3.3).

##  3 Proposed Approach

We first present the preliminaries of our model in §3.1. We then introduce our
caption bootstrapping strategy which provides rich open-vocabulary
descriptions for training (§3.2). Finally, we present our hyperbolic vision-
language learning approach which exploits rich semantics from synthetic
captions to boost open-world generalization (§3.3).

###  3.1 Preliminaries on model architecture

Training an open-world detector. Object detectors are trained to predict a
list of bounding box location $B_{i}\in\mathbf{R}^{4}$ and class label
$L_{i}\in\mathbb{R}^{n}$. They often consist of an visual encoder, a bounding
box regressor and a visual classifier. During training, a bounding box
regression loss $\mathcal{L}_{bbox}$ and a classification loss
$\mathcal{L}_{cls}$ are jointly optimized to predict the object bounding box
location and class label, i.e.,

$\displaystyle\mathcal{L}_{Det}=\mathcal{L}_{bbox}+\mathcal{L}_{cls}.$ (1)

Most detectors can be categorized into one-stage [47, 67] or two-stage [42,
18, 66]. Detectors in the latter set employ an additional loss to train a
region proposal network ($\mathcal{L}_{rpn}$) for predicting the objectness
score of each region proposal. For the classification loss
$\mathcal{L}_{cls}$, classic detectors often adopt a cross-entropy loss [42],
which works well for classifying known classes, but it cannot model novel
classes. To overcome this limitation, open-world detectors [29, 65] leverage
text embeddings to represent the class labels. In details, the classification
loss $\mathcal{L}_{cls}$ helps aligning the visual feature embedding $v_{i}$
with its class label embedding $l_{i}$, i.e.,

$\mathcal{L}_{cls}=-\log\frac{\exp(\text{sim}(v_{i},l_{i})/\tau)}{\sum_{j=1}^{n}\exp(\text{sim}(v_{i},l_{j})/\tau)},$
(2)

where $v_{i}$ is visual feature embedding from the detector,
$\text{sim}(\cdot,\cdot)$ measures cosine similarity between embeddings,
$\tau$ is the temperature. The class label embedding $l_{j}$ is extracted with
a pre-trained text encoder from CLIP [41]. At test time, the predicted label
is
$L_{i}=\operatorname*{arg\,max}_{j}\\{\text{sim}(v_{i},l_{j})\\}_{j=1}^{n}$,
where $n$ is the number of pre-defined class labels (seen during training).
One can augment these $n$ classes with novel classes (by adding their class
label embeddings) to enable the open-set recognition on these novel concepts.

Cross-modal attention. The above formulation (Eq. (2)) enables the detection
of novel objects using their class label embeddings. However, it does not
capture the spatial awareness of language semantics on different visual
objects, thus fails to localize objects with more precise information, e.g.,
localizing “the person playing a bass guitar” in Figure 2. To achieve open-
world detection with free-form texts, we propose to inject the language
semantics into the visual feature embedding $v_{i}$ by a cross-modal attention
module [27, 29]. Specifically, given any text embeddings
$T=\\{t_{j}\\}_{1}^{n}$, we compute the cross-modal attention between $v_{i}$,
$T$ as:

$\displaystyle v^{(q)}_{i}=(W^{(q)})^{\top}v_{i},\ T^{(k)}=TW^{(k)},\
T^{(v)}=TW^{(v)},$ (3) $\displaystyle A_{i}=T^{(k)}v^{(q)}_{i}/\sqrt{d},\
v_{i}^{l}=\text{softmax}(A_{i})\;T^{(v)}W^{(out)},$

where $W^{(q)}$, $W^{(k)}$, $W^{(v)}$ are the trainable query, key, and value
matrices in multi-head attention [49], $d$ is the dimension of hidden
embeddings $v_{i}^{(q)}$ and $T^{(k)}$, and $v_{i}^{l}$ is the visual feature
embedding fused with language semantics. To further inject spatial awareness
into the visual feature embedding, we add a positional encoding layer [10] to
fuse the visual feature embedding $v_{i}$ with region proposal feature
$p_{i}$:

$\displaystyle v_{i}^{s}=\text{PositionalEncoding}(v_{i},p_{i}).$ (4)

Finally, to attain a spatial-aware visual feature embedding fused with
language semantics, we combine the language-aware and spatial-aware visual
embeddings $v_{i}^{l}$ (Eq. (3)), $v_{i}^{s}$ (Eq. (4)) with the region
proposal network $\text{RPN}(\cdot)$:

$\displaystyle\tilde{v}_{i}=\text{RPN}(v_{i}^{l}+v_{i}^{s}).$ (5)

Within this formulation, our detector can localize objects given any texts by
replacing the original visual embedding $v_{i}$ with the new visual embedding
$\tilde{v}_{i}$, and compute the localization score based on similarity
between $\tilde{v}_{i}$ and the text embeddings $\\{t_{j}\\}_{1}^{n}$, i.e.,
$Loc_{i}=\operatorname*{arg\,max}_{j}\\{\text{sim}(\tilde{v}_{i},t_{j})\\}_{j=1}^{n}$.

###  3.2 Bootstrapping synthetic captions

To enable generalization on novel objects, existing methods often exploit
manual annotated datasets such as image classification [43], captioning [46]
and grounding dataset [23] to provide knowledge of novel concepts. Rather than
relying on expensive manual annotations, we propose to bootstrap synthetic
captions from a pre-trained VLM. Similar to manual annotations, these captions
also provide rich open-vocabulary descriptions of novel concepts. In this
paper, we use BLIP2 [28] to generate captions – which is trained on a large-
scale dataset of 129M images. It is worth noting that our model is agnostic to
the VLM used for caption generation, and thus can benefit from more powerful
VLMs.

Region sampling for caption bootstrapping. We collect three sets of region
proposals. First, for seen/known classes we use the ground truth bounding
boxes $\mathcal{B}$. Then, to attain regions on novel objects, we propose to
use two region sampling strategies: proposal sampling and grid sampling. In
proposal sampling, we sample region proposals with high objectness scores and
apply non-maximum suppression to remove duplicates; this leads to the set of
regions $\mathcal{P}$. In grid sampling, we split the image into $k\times k$
grids similar to ViT [10] to get a set of region crops $\mathcal{G}$.
Together, these produce a rich set of region crops:
$\mathcal{B}\cup\mathcal{P}\cup\mathcal{G}$ that encapsulate both seen and
unseen objects. Finally, we generate synthetic captions $\mathcal{C}$ densely
over these region crops with the BLIP2 captioner:

$\mathcal{C}=\text{Captioner}_{\text{BLIP2}}(\mathcal{B}\cup\mathcal{P}\cup\mathcal{G}),$
(6)

where the captions $\mathcal{C}$ are then used for model training.

Figure 3:  Illustration of hyperbolic vision-language learning. The visual and
caption embeddings are lifted from (a) Euclidean space to (b) Hyperbolic space
by exponential mapping (Eq. (8)). To learn the partial order of ‘caption
entails object’, we propose hyperbolic contrastive loss, hyperbolic entailment
loss (Eq. (10), Eq. (13)) to align visual and caption embeddings in hierarchy,
thus ensuring the hallucination in caption is not aligned directly with visual
embeddings to negatively impact the model learning.

###  3.3 Hyperbolic vision-language learning

Vanilla contrastive learning. Our bootstrapping synthetic captions provide
diverse language descriptions on both seen and unseen objects in different
image regions, thus offering rich knowledge to boost open-world
generalization. To learn from such knowledge, we can adopt the vanilla
contrastive loss [41] for aligning the visual feature embedding
$\tilde{o}_{i}$ (Eq. (5)) with the captions embedding $c_{i}$ attained in (6):

$\mathcal{L}_{cap}=-\log\frac{\exp(\text{sim}(\tilde{v}_{i},c_{i},)/\tau)}{\sum_{j=1}^{m}\exp(\text{sim}(\tilde{v}_{i},c_{j})/\tau)},$
(7)

where $\mathcal{L}_{cap}$ is computed on a batch of $m$ caption embeddings,
which enables learning about the novel concepts described in the synthetic
captions. However, synthetic captions suffer from the hallucination problem
[44], where concepts unrelated to the image are added to the generated
caption; this simple contrastive loss is not equipped to handle them and its
learning suffers from the generated noise. To learn more effectively from
these captions, we propose to align visual and text embeddings in a hierarchy.

Hyperbolic contrastive learning. The synthetic caption on a region often
entails the object descriptions along with other content. Thus, we argue that
the object and caption should follow a ‘caption entails object’ hierarchy in a
tree-like structure (Figure 3). Inspired by recent advances in hyperbolic
learning to model the partial order of ‘scene entails object’ [13] or ‘text
entails image’ [8], we propose to model our ‘caption entails object’ hierarchy
in a hyperbolic space.

Specifically, we introduce our hyperbolic contrastive learning based on the
Lorentzian distance in hyperbolic geometry [25], which lifts the embeddings
from Euclidean space to the Lorentz hyperboloid to represent the structural
hierarchy. Formally, we first project the visual feature embedding
$\tilde{v}_{i}$ and caption embedding $c_{i}$ to the hyperboloid with
exponential mapping (Eq. (8)), and then compute their Lorentzian distance on
the hyperboloid (Eq. (9)) as follows:

$\displaystyle v^{\mathcal{H}}_{i}=\text{expm}_{\mathbf{0}}(\tilde{v}_{i}),\
c^{\mathcal{H}}_{i}=\text{expm}_{\mathbf{0}}(c_{i}),$ (8)
$\displaystyle\text{with}\
\text{expm}_{\mathbf{0}}(x)=\frac{\text{sinh}(\sqrt{C}\lVert
x\rVert)}{\sqrt{C}\lVert x\rVert},$

where $\text{expm}_{\mathbf{0}}(\cdot)$ is an exponential mapping, and $C$ is
a learnable curvature parameter. Given two mapped embeddings
$v^{\mathcal{H}}_{i},c^{\mathcal{H}}_{i}$, we follow existing Lorentz model
[8, 40, 26, 25], and compute the Lorentzian distance on the hyperboloid as:

$\displaystyle
d_{\mathcal{H}}(\tilde{v}_{i},c_{i})=\sqrt{1/C}\cdot\text{cosh}^{-1}(-c\langle\tilde{v}_{i},c_{i}\rangle_{\mathcal{H}}),$
(9) $\displaystyle\text{with}\
\langle\tilde{v}_{i},c_{i}\rangle_{\mathcal{H}}{=}\langle\tilde{v}_{i},c_{i}\rangle-\sqrt{1/C+\lVert\tilde{v}_{i}\rVert}\sqrt{1/C+\lVert
c_{i}\rVert},$

where $\langle\cdot,\cdot\rangle_{\mathcal{H}}$ is the Lorentz inner product
induced by the Riemannian metric of the Lorentz model [8, 40, 26, 25], and
$d_{\mathcal{H}}(\cdot,\cdot)$ is the distance between two embeddings on the
hyperboloid. By mappings the embeddings to the hyperbolic space, we derive the
hyperbolic contrastive loss by replacing cosine similarity measure
$\text{sim}(\cdot,\cdot)$ in Eq. (7) with the hyperbolic distance measure
$d_{\mathcal{H}}(\cdot,\cdot)$, i.e,

$\mathcal{L}^{\mathcal{H}}_{cap}=-\log\frac{\exp(-d_{\mathcal{H}}(v^{\mathcal{H}}_{i},c^{\mathcal{H}}_{i})/\tau)}{\sum_{j=1}^{B}\exp(-d_{\mathcal{H}}(v^{\mathcal{H}}_{i},c^{\mathcal{H}}_{j})/\tau)},$
(10)

where $\mathcal{L}^{\mathcal{H}}_{cap}$ is our loss function to align visual
embedding and caption embedding in the hyperbolic space.

Hyperbolic entailment. To impose our proposed structural hierarchy of ‘caption
entails object’, we formulate a hyperbolic entailment loss based on the
property of an entailment cone – which defines a cone in the hyperbolic space
to enforce partial order between embeddings [26]. Specifically, we define an
entailment cone for caption embedding $c^{\mathcal{H}}_{i}$ as:

$\displaystyle A(c^{\mathcal{H}}_{i})=\sin^{-1}(\frac{2K}{\sqrt{C}\lVert
c^{\mathcal{H}}_{i}\rVert})$ (11)

where $K$ is a constant 0.1 used to avoid numerical overflow; $A(\cdot)$ is
half aperture of the cone in hyperbolic space. With $A(c^{\mathcal{H}}_{i})$,
we introduce the entailment constraints to impose the hierarchy of ‘caption
entails object’, by ensuring (1) the visual object embedding
$v^{\mathcal{H}}_{i}$ lies inside the cone, and (2) other visual embeddings
$v^{\mathcal{H}}_{j}$ ($j{\neq}i$) lie outside the cone.

Formally, we denote the exterior angle between $v^{\mathcal{H}}_{i}$ and
$c^{\mathcal{H}}_{i}$ as
$\angle(c^{\mathcal{H}}_{i},v^{\mathcal{H}}_{i})$111Note: the exterior angle
$\angle(a,b)$ in hyperbolic space is computed as
$\angle(a,b)=\cos^{-1}(\frac{\sqrt{1/c+\lVert b\rVert}+\sqrt{1/c+\lVert
a\rVert}c\langle
a^{\mathcal{H}}_{i},b^{\mathcal{H}}_{i}\rangle_{\mathcal{H}}}{\lVert
a^{\mathcal{H}}_{i}\rVert\sqrt{c(\langle
a^{\mathcal{H}}_{i},b^{\mathcal{H}}_{i}\rangle)^{2}-1}})$, $c$ is curvature.,
and introduce a loss term to ensure the entailment constraint (1):

$\displaystyle
E(c^{\mathcal{H}}_{i},v^{\mathcal{H}}_{i})=\max(0,\angle(c^{\mathcal{H}}_{i},v^{\mathcal{H}}_{i})-A(c^{\mathcal{H}}_{i})),$
(12)

where $E(\cdot,\cdot)$ ensures
$\angle(c^{\mathcal{H}}_{i},v^{\mathcal{H}}_{i}){\leqslant}A(c^{\mathcal{H}}_{i})$,
thus enforcing $v^{\mathcal{H}}_{i}$ to lie inside the cone of
$c^{\mathcal{H}}_{i}$. To further ensure both entailment constraints (1) and
(2), we introduce the following hyperbolic entailment loss with two loss
terms:

$\displaystyle\mathcal{L}_{\text{entail}}=E(c^{\mathcal{H}}_{i},v^{\mathcal{H}}_{i})+\sum_{j\neq
i}\max(0,\gamma-E(c^{\mathcal{H}}_{i},v^{\mathcal{H}}_{j})),$ (13)

where the first term is Eq. (12); the second term is a max margin loss to
ensure
$\angle(c^{\mathcal{H}}_{i},v^{\mathcal{H}}_{j}){\geqslant}A(c^{\mathcal{H}}_{i})+\gamma$,
thus enforcing $v^{\mathcal{H}}_{j}$ to lie outside the cone of
$c^{\mathcal{H}}_{i}$ with a margin $\gamma$.

Overall objective. Finally, given our proposed hyperbolic loss functions (Eq.
(7), Eq. (13)), we formulate our learning objective as follows:

$\mathcal{L}_{hyper}=\mathcal{L}_{bbox}+\mathcal{L}_{cls}+\mathcal{L}^{\mathcal{H}}_{cap}+\mathcal{L}_{\text{entail}},$
(14)

where $\mathcal{L}^{\mathcal{H}}_{cap},\mathcal{L}_{\text{entail}}$ are our
hyperbolic contrastive loss, and hyperbolic entailment loss to align visual
and caption embedding in a structural hierarchy. An alternative to learn from
synthetic captions is the vanilla contrastive loss (Eq. (7)), which leads to
the baseline objective below:

$\mathcal{L}_{baseline}=\mathcal{L}_{bbox}+\mathcal{L}_{cls}+\mathcal{L}_{cap}.$
(15)

We compare our Hyperbolic Learning objective $\mathcal{L}_{hyper}$ (Eq. (14))
to alternative objectives $\mathcal{L}_{Det}$ (Eq. (1)),
$\mathcal{L}_{baseline}$ (Eq. (15)) in ablation study to verify our design
rationales. We refer our approach as “HyperLearner” in experiments.

| Method | Backbone | #Params | FLOPs |  Pre-training Data |  COCO2017 val
---|---|---|---|---|---|---
| Zero-shot | Fine-tuning
1 | Faster-RCNN [14] | RN50-FPN | 42M | 180G | COCO | - | 40.2
2 | Faster-RCNN [14] | RN101-FPN | 54M | 313G | COCO | - | 42.0
3 | Deformable DETR(DC5) [68] | RN50 | 41M | 187G | COCO | - | 41.1
4 | CenterNetv2 [66] | RN50 | 76M | 288G | COCO | - | 42.9
5 | Dyhead-T [6] | Swin-T | 232M | 361G | O365 | 43.6 | 53.3
6 | GLIP-T(A) [29] | Swin-T | 232M | 488G | O365 | 42.9 | 52.9
7 | GLIP-T(B) [29] | Swin-T | 232M | 488G | O365 | 44.9 | 53.8
8 | GLIP-T(C) [29] | Swin-T | 232M | 488G | O365, GoldG | 46.7 | 55.1
9 | DINO-T [62] | Swin-T | - | - | O365 | 46.2 | 56.9
10 | Grounding-DINO-T1 [34] | Swin-T | 172M | 464G | O365 | 46.7 | 56.9
11 | Grounding-DINO-T2 [34] | Swin-T | 172M | 464G | O365, GoldG | 48.1 | 57.1
12 | Grounding-DINO-T3 [34] | Swin-T | 172M | 464G | O365, GoldG, Cap4M | 48.4 | 57.2
13 | HyperLearner (Ours) | Swin-T | 90M | 324G | O365 | 47.6 | 56.8
14 | HyperLearner (Ours) | Swin-T | 90M | 324G | O365, GoldG | 48.4 | 57.4

Table 1: Comparison on COCO benchmark. Results are given on both zero-shot and
fine-tuning settings. Metric: mAP.

Method |  Pre-training Data |  LVIS minival
---|---|---
AP | APr $|$ APc $|$ APf
MDETR [20] | GoldG, RefCOCO | 24.2 | 20.9 $|$ 24.9 $|$ 24.3
DETCLIP-T [54] | O365 | 28.8 | 26.0 $|$ 28.0 $|$ 30.0
GLIP-T (C) [29] | O365, GoldG | 24.9 | 17.7 $|$ 19.5 $|$ 31.0
GLIP-T [29] | O365, GoldG, Cap4M | 26.0 | 20.8 $|$ 21.4 $|$ 31.0
Grounding-DINO-T [34] | O365, GoldG | 25.6 | 14.4 $|$ 19.6 $|$ 32.2
Grounding-DINO-T [34] | O365, GoldG, Cap4M | 27.4 | 20.8 $|$ 21.4 $|$ 31.0
HyperLearner (Ours) | O365 | 25.5 | 25.9 $|$ 27.5 $|$ 23.7
HyperLearner (Ours) | O365, GoldG | 31.3 | 30.7 $|$ 32.6 $|$ 30.3

Table 2: Comparison on LVIS benchmark. Metric: mAP.

##  4 Experiments

In the following, we detail our experimental settings (§4.1) and provide
thorough evaluation of our approach on a wide set of benchmark datasets in
comparison to the state-of-the-arts (§4.2). Finally, we provide insightful
ablation study (§4.3) and qualitative results (§4.4) to analyze our approach.

###  4.1 Experimental settings

Implementation details. Our approach utilizes CenterNetV2 [65] as the first-
stage region proposal network, succeeded by a cross-modal attention module as
detailed in §3.1. Our text encoder is the 12-layer text transformer from CLIP
ViT/B16 [41]. We used BLIP2 6.7b [28] to generate synthetic captions on region
crops offline (§3.2). For model pre-training, we used two datasets: Object 365
(O365) [45] and GoldG [20], to ensure a fair comparison with other methods on
the same evaluation benchmark datasets. In all experiments, we use Swin-Tiny
[35] as the backbone, and undergo training for 100,000 iterations with a batch
size of 64 and a learning rate 0.0001. The initial 10,000 iterations serve as
a warm-up phase, which is essential for optimal convergence. We train our
model on 8 A100 GPUs. More details are provided in the supplementary material.

Object detection benchmark datasets. We evaluate the open-world detection
performance of our method on the most popular object detection datasets: (1)
COCO [32], which has 80 common object classes, and (2) LVIS [17], which has
1203 object classes. Furthermore, we also evaluate on Object Detection in the
Wild (ODinW) [29] – an assembly of 13 datasets, each one representing a
different real-world fine-grained domain. Following the common practices, for
zero-shot setting we evaluate our pre-trained models directly, while for fine-
tuning setting, we fine-tune our pre-trained models on the target dataset. We
report mean Average Precision (mAP) at IoU threshold of 0.5.

Object localization benchmark datasets. We evaluate our method on referring
expression comprehension which aims to locate novel objects using natural
language expressions. We employ the following benchmark datasets: RefCOCO
[21], RefCOCO+ [58] and RefCOCOg [37] (collectively referred as RefCOCO/+/g)
to evaluate localizing the objects given descriptive free-form texts. All
three datasets utilize the images from the original COCO dataset; however,
they differ in their textual descriptions: RefCOCO+ emphasizes purely on
appearance, excluding spatial reference found in RefCOCO; RefCOCOg provides
more comprehensive descriptions in sentences rather than phases. Our
evaluation adheres to the established evaluation protocol, employing top-1
accuracy to determine the successful localization of objects based on the
description, in line with the metric used in previous works [57, 20, 34].

| Method |  Pre-training Data |  Fine-tuning |  RefCOCO |  RefCOCO+ |  RefCOCOg
---|---|---|---|---|---|---
| val | testA | testB | val | testA | testB | val | test
1 | CNN-LSTM [37] | RefC | - | - | 63.15 | 64.21 | - | 48.73 | 42.13 | 62.14 | -
2 | MAttNet [57] | RefC | ✓ | 76.65 | 81.14 | 69.99 | 65.33 | 71.62 | 56.02 | 66.58 | 67.27
3 | RefTR [30] | VG | ✓ | 85.65 | 88.73 | 81.16 | 77.55 | 82.26 | 68.99 | 79.25 | 80.01
4 | MDETR [20] | GoldG,RefC | ✓ | 86.75 | 89.58 | 81.41 | 79.52 | 84.09 | 70.62 | 81.64 | 80.89
5 | DQ-DETR [33] | GoldG,RefC | ✓ | 88.63 | 91.04 | 83.51 | 81.66 | 86.15 | 73.21 | 82.76 | 83.44
6 | GLIP-T(B) [29] | O365,GoldG | × | 49.96 | 54.69 | 43.06 | 49.01 | 53.44 | 43.42 | 65.58 | 66.08
7 | GLIP-T(C) [29] | O365,GoldG,Cap4M | × | 50.42 | 54.30 | 43.83 | 49.50 | 52.78 | 44.59 | 66.09 | 66.89
8 | Grounding-DINO-T [34] | O365,GoldG | × | 50.41 | 57.24 | 43.21 | 51.40 | 57.59 | 45.81 | 67.46 | 67.13
9 | Grounding-DINO-T [34] | O365,GoldG,RefC | × | 73.98 | 74.88 | 59.29 | 66.81 | 69.91 | 56.09 | 71.06 | 72.07
10 | Grounding-DINO-T [34] | O365,GoldG,RefC | ✓ | 89.19 | 91.86 | 85.99 | 81.09 | 87.40 | 74.71 | 84.15 | 84.94
11 | HyperLearner (Ours) | O365,GoldG | × | 50.66 | 60.87 | 44.66 | 59.29 | 62.29 | 45.43 | 67.02 | 67.44
12 | HyperLearner (Ours) | O365,GoldG,RefC | × | 77.89 | 76.92 | 72.99 | 67.54 | 75.55 | 57.54 | 77.00 | 76.79
13 | HyperLearner (Ours) | O365,GoldG,RefC | ✓ | 90.74 | 92.09 | 85.46 | 82.35 | 84.70 | 72.64 | 82.53 | 82.39

Table 3: Comparison on RefCOCO/+/g benchmark. Metric: Top-1 accuracy.

Method | Backbone |  Pre-training Data |  Test $\text{AP}_{avg}$
---|---|---|---
zero-shot | full-shot
Detic-R [65] | RN50 | LVIS, COCO, IN-21K | 29.4 | 64.4
Detic-B [65] | Swin-B | LVIS, COCO, IN-21K | 38.7 | 70.1
GLIP-T(A) [29] | Swin-T | O365 | 28.7 | 63.6
GLIP-T(B) [29] | Swin-T | O365 | 33.2 | 62.7
GLIP-T(C) [29] | Swin-T | O365, GoldG | 44.4 | 63.9
Grounding-DINO-T [34] | Swin-T | O365, GoldG,Cap4M | 44.9 | -
HyperLearner (Ours) | Swin-T | O365 | 37.9 | 66.7
HyperLearner (Ours) | Swin-T | O365, GoldG | 45.2 | 68.9

Table 4: Comparison on ODinW benchmark. Metric: mAP.

###  4.2 Comparison to SOTA methods

Comparison on COCO. In Table 1, we compare our HyperLearner to several recent
models: (1) traditional two-stage object detectors, including Faster-RCNN
[42], Deformable DETR [68] and CenterNetV2 [65]; (2) closed-set detectors
which are adaptable for evaluation on COCO through class mapping between O365
and COCO, including Dyhead [6] and DINO [62]; (3) open-world detectors
including GLIP [29] and Grounding DINO [34], which not only use image-text
data for pre-training, but also utilize model architectures to fuse visual-
language representations.

Table 1 details the comparison on model efficiency and performance on COCO. As
shown, our model has much smaller parameter size and computation cost
(measured in FLOPs), which significantly improve model efficiency for real-
world deployment. Moreover, while being lightweight, our model achieves better
performance in both zero-shot and fine-tuning settings, with an mAP of 48.4
and 57.4 as compared to 48.1 and 57.1 obtained by the best competitor
Grounding DINO-T when using O365, GoldG for training. Notably, our model pre-
trained on O365, GoldG (in row 14) obtains similar performance as Grounding
DINO-T pre-trained on O365, GoldG and Cap4M, an additional large-scale
manually-annotated caption dataset [29] (row 12). This clearly shows how
powerful machine-generated captions can be when used properly in HyperLearner.

Comparison on LVIS. Table 2 shows the results on LVIS minival validation set.
Our model demonstrates a significant advantage over GLIP and Grounding DINO on
detecting novel rare classes – outperforming them by +13.0% and +16.3% in APr
respectively, when pre-trained on O365 and GoldG. Our improved gain is
maintained at 9.9% even when other methods benefit from the additional Cap4M
dataset. The results suggest our proposed method’s capability to detect novel
and rare concepts, attributing to the effective incorporation of semantic
knowledge via bootstrapping synthetic captions and the vision-language
alignments brought by our novel hyperbolic loss formulation.

Comparison on ODinW. The ODinW dataset exemplifies more fine-grained,
challenging real-world application scenarios. In Table 4, we report the
average AP over its 13 subsets. Our finding aligns with those from the COCO
and LVIS datasets, indicating that our model performs well in zero-shot
testing scenario. It outperforms GLIP by +4.7% with pre-training solely on
O365 and +0.8% when GoldG is added. While our model shows the commendable
zero-shot transferability, it also excels in the full-shot setting, suggesting
the pre-trained model possesses strong generalizablity.

Comparison on RefCOCO/+/g. Table 3 shows the results of localizing objects
with referring expressions. When pre-trained on O365, GoldG or pre-trained on
O365, GoldG, RefC, our model (in row 11, 12) surpasses Grounding DINO (in row
8, 9) and GLIP (in row 6). With further fine-tuning on RefC, our model (in row
13) performs competitively compared to Grounding DINO (in row 10). The
competitive results in Table 3 can be attributed to learning with our
bootstrapped captions, which often include descriptions of object attributes
and their spatial relationships that also exist in the natural language
expressions in RefCOCO/+/g.

| Method | COCO | LVIS | ODinW
---|---|---|---|---
| AP | APr | APc | APf
1 | $\mathcal{L}_{Det}$ | 42.3 | 13.3 | 8.3 | 11.9 | 15.4 | 26.3
2 | $\mathcal{L}_{baseline}$ | 46.8 | 23.6 | 22.7 | 24.3 | 24.6 | 38.6
3 | $\mathcal{L}_{hyper}$ | 48.4 | 31.3 | 30.7 | 32.6 | 30.3 | 45.2

Table 5: Ablation study on learning objectives. Metric: mAP. Figure 4:
Comparison between baseline objective Eq. (15) and our objective Eq. (14) on
COCO during training. Metric: mAP.

| Method | COCO | LVIS | ODinW
---|---|---|---|---
1 | no synthetic captions | 42.3 | 13.3 | 26.3
2 | synthetic captions on $\mathcal{B}$ | 47.1 | 23.4 | 36.8
3 | synthetic captions on $\mathcal{B}\cup\mathcal{G}$ | 47.6 | 23.7 | 38.6
4 | synthetic captions on $\mathcal{B}\cup\mathcal{P}$ | 47.4 | 27.1 | 38.9
5 | synthetic captions on $\mathcal{B}\cup\mathcal{G}\cup\mathcal{P}$ | 48.4 | 31.3 | 45.2

Table 6: Ablation study on region sampling for bootstrapping synthetic
captions. Metric: mAP.

###  4.3 Ablation study

We now ablate our approach and its components: hyperbolic learning objective
(§3.3), region sampling for bootstrapping synthetic captions (§3.2), and
cross-modal attention module for localization with free-form texts (§3.1).

Ablation study on learning objectives. Table 5 shows the results of different
objectives: (1) standard open-world detection objective $\mathcal{L}_{Det}$ in
Eq. (1), (2) vanilla contrastive learning objective $\mathcal{L}_{baseline}$
in Eq. (15), (3) hyperbolic learning objective $\mathcal{L}_{hyper}$ in Eq.
(14). Our results show that our objective $\mathcal{L}_{hyper}$ outperforms
other alternatives significantly, consistently across different datasets. The
improvement of $\mathcal{L}_{baseline}$ over $\mathcal{L}_{Det}$ shows the
strength of learning with synthetic captions to boost open-world
generalization; while the improvement of $\mathcal{L}_{hyper}$ over
$\mathcal{L}_{baseline},\mathcal{L}_{Det}$ further shows the benefit of
aligning visual and caption embeddings in hierarchy with hyperbolic geometry
(Figure 3).

Figure 4 compares the baseline objective $\mathcal{L}_{baseline}$ and our
objective $\mathcal{L}_{hyper}$ during training. When using vanilla
contrastive learning ($\mathcal{L}_{baseline}$), the performance saturates
quickly. In contrast, our HyperLearner converges slower but consistently
improves its performance, suggesting its stronger capability of learning from
synthetic captions.

| Method | RefCOCO | COCO
---|---|---|---
| val $\ \ \ |$ testA $|$ testB
1 |  w/o cross-modal attention | 60.10 $|$ 70.96 $|$ 48.58 | 48.7
2 | w cross-modal attention | 77.89 $|$ 76.92 $|$ 72.99 | 48.4

Table 7: Ablation study on cross-modal attention. Figure 5:  Qualitative
results on open-world detection. Given any new class labels (row 1) or free-
form texts (row 2) that specify objects with attribute, action, interaction,
and spatial relationship, our model can detect and localize the objects in
images.

Ablation study on region sampling. Table 6 shows the results of bootstrapping
richer synthetic captions on different regions: basic groundtruth bboxes
$\mathcal{B}$, proposal sampling bboxes $\mathcal{P}$, grid sampling bboxes
$\mathcal{G}$ (§3.2, Figure 2). As shown in row 1 and 2, utilizing captions on
$\mathcal{B}$ improves the performance substantially, with a $\approx$10%
increase on LVIS and ODinW. Moreover, the object-centric proposal sampling
$\mathcal{P}$ (row 4) can target on novel objects, hence improving the
detection on LVIS and ODinW, where performance are more dominated by rare
novel objects. Finally, combining all sampled regions (row 5) yields the best
overall performance, suggesting that using a diverse set of regions leads to
richer semantics information and higher robustness.

Ablation study on cross-modal attention. Table 7 shows the results of using
cross-modal attention to fuse spatial visual information and language
semantics (Eq. (3), (4), (5) in §3.1). Our results indicate that the cross-
modal attention module improves our model performance significantly on RefCOCO
benchmark and yields similar performance on COCO benchmark. These results
prove the crucial effectiveness of leveraging spatial and language information
for the task of localization with free-form texts, as evidenced by the
substantial boosts of 17.79%, 5.96%, 24.31% in Top-1 accuracy on the RefCOCO
val, testA, testB respectively.

Figure 6: Visualization of our object and caption embeddings.

###  4.4 Qualitative analysis

Qualitative results. Figure 5 shows our qualitative results. In row 1, given
new labels, our model can detect these new objects, even though these objects
are small-scale (examples 2 and 3) and rare (example 4). In row 2, given any
free-form texts that specify novel objects with fine-grained details, our
model can localize these objects precisely, such as localizing the object of
an attribute (pink), performing an action (standing), interacting with other
object (holding football), or in a certain spatial location (in the middle).
These examples indicate the strong generalizability of our model on
recognizing new concepts in open world.

Visualization of embeddings. Figure 6 shows our object visual embeddings,
caption embeddings with UMAP [38], which projects the learned embeddings into
a 2D space. We can find that the visualized embeddings present a tree-like
entailment relation; this is in line with our motivation of imposing the
hierarchy of ‘caption entails object’ in our learning objective (Figure 3).
Interestingly, we can also find the hierarchy of ‘longer caption entails
shorter caption’, which suggests the intrinsic hierarchy inside the caption
data.

##  5 Conclusion

We present a novel hyperbolic vision-language learning approach to learn from
synthetic captions effectively. Our approach is well designed to leverage the
open-world knowledge from pre-trained VLMs, and specially mitigate the
hallucination issue in synthetic captions to boost the open-world
generalizability in detection. Our comprehensive experiments on multiple
benchmark datasets show the competitive performance of our approach as
compared to the state-of-the-art. We analyze our model design rationales with
insightful ablation study and qualitative analysis. Our work also paved a
strong foundation on leveraging synthetic captions with hyperbolic learning
for other vision tasks.

## Supplementary Material

##  A Implementation details

In Table A, we present the learning schedules when fine-tuning on different
pre-training datasets. Note that when adding GoldG, or RefC in row 2, 3, our
model is initialized from the checkpoints in row 1, 2 respectively to continue
model training. In Table B, we present the learning schedules when fine-tuning
on different downstream datasets. During both pre-training and fine-tuning,
the learning rate is decayed by 0.1 at 1/3 and 2/3 of the training progress.

Model | Dataset(s) | Iterations | batch size | learning rate
---|---|---|---|---
HyperLearner | O365 | 900,000 | 64 | 0.0002
O365,GoldG | +500,000 | 64 | 0.0002
O365,GoldG,RefC | +500,000 | 64 | 0.0002

Table A: Learning schedules on different pre-training datasets.

Model | Dataset(s) | Iterations | batch size | learning rate
---|---|---|---|---
HyperLearner | COCO | 100,000 | 64 | 0.0001
ODinW | 50,000 | 64 | 0.0001
RefCOCO | 100,000 | 32 | 0.0001

Table B: Learning schedules on different downstream datasets.

##  B Computation cost of hyperbolic learning

To mitigate the high computational cost of hyperbolic learning, we make the
following design choices. First, we choose to only lift the last-layer
Euclidean features into hyperbolic space, instead of using a hyperbolic neural
network. This approach circumvents computationally intensive operations, such
as Frechet mean calculations, as noted in [16].

Second, we adopt the Lorentz model as [36], rather than the Poincaré ball
model employed in [8]. The Poincaré ball model is operationally intensive due
to its computationally heavy Gyrovector operations, which necessitate a series
of intermediate calculations. In contemporary deep learning frameworks, these
calculations substantially increase the overhead of computation graph, as
described in [13]. In contrast, the Lorentz model offers a more
computationally efficient alternative for hyperbolic geometry, which employs
four-vector operations (Euclidean axes plus a time dimension), thus learning
only one additional dimension. Notably, the time dimension can be readily
determined based on the curvature of the hyperboloid, as detailed in [8].

Third, thanks to the two design choices made above, the only additional
computational costs are hyperbolic functions, including “sinh” and “cosh”.
Fortunately, PyTorch efficiently optimizes these operations, particularly
through techniques such as operation fusion. This technique combines multiple
pointwise operations – including arithmetic and trigonometric functions – into
a single kernel, thus further enhancing computational efficiency, as detailed
in [8].

##  C Discussion on evaluation benchmarks

Open-vocabulary detection evaluation benchmarks. It is worth noting that
existing works in open-vocabulary detection (OVD) [60, 15, 50, 59, 64, 51, 4,
28] consider the evaluation setting where only novel (unseen) object classes
are presented in the evaluation benchmarks. Specifically, the OVCOCO and
OVLVIS benchmarks are built by splitting the class labels into two sets: seen
and unseen. The model is trained on the seen classes and tested on the unseen
classes.

Open-world detection evaluation benchmarks. In contrast, our open-world
detection task consider the evaluation setting where any object classes (both
seen and unseen) are presented in the evaluation benchmarks, which aligns with
detection in the open world. Moreover, these object classes could be described
by keywords (class labels) or free-form texts. Our evaluation setting also
poses a unique challenge of understanding novel concepts in free-form texts.

##  D Additional results

Model | Evaluation Mode | Data | AerialDrone | Aquarium | Rabbits | EgoHands | Mushrooms | Packages | PascalVOC | pistols | pothole | Raccoon | Shellfish | Thermal | Vehicles | mAP
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
HyperLearner | zero-shot | O365 | 16.63 | 27.35 | 72.53 | 12.47 | 41.08 | 63.68 | 55.88 | 23.03 | 15.31 | 42.68 | 21.50 | 40.51 | 59.92 | 36.05
fine-tune | O365 | 41.53 | 50.72 | 75.59 | 68.41 | 83.45 | 72.34 | 84.39 | 99.77 | 52.08 | 61.00 | 43.09 | 75.35 | 59.42 | 67.31
zero-shot | O365, GoldG | 27.25 | 35.34 | 74.38 | 18.12 | 58.50 | 68.95 | 64.12 | 27.03 | 17.06 | 56.01 | 29.36 | 60.72 | 60.24 | 44.74
fine-tune | O365, GoldG | 42.48 | 52.10 | 76.34 | 74.66 | 90.43 | 76.53 | 87.94 | 99.77 | 53.00 | 63.58 | 39.52 | 78.52 | 61.25 | 69.57

Table C: Detailed results of zero-shot and fine-tune transfer on 13 individual
datasets on ODinW. Metric: mAP over 13 datasets.  Figure A:  More qualitative
results on open-world detection using free-form texts, where the free-form
texts specify objects with attribute, action/interaction, and spatial
relationships.

Additional quantitative results on ODinW. Table C shows the additional results
of 13 individual datasets on ODinW benchmark of our proposed approach, when
using different evaluation modes (zero-shot or fine-tune/full-shot) and
different pre-training data. Results show the benefit of using more data for
pre-training, and further fine-tuning on different target datasets to improve
generalization.

Additional qualitative results. Figure A presents more qualitative examples
using our model to localize objects described using free-form texts to specify
certain object attribute, action/interaction, and spatial relationships.
Examples illustrate the strong generalizability of our model in understanding
and recognizing novel concepts in open world.

##  E Discussion on quality of synthetic captions

Generally, our VLM, BLIP2 6.7b, generates synthetic captions that accurately
describe the objects in the image, as indicated by [28]. Furthermore, our
experimental result reveals that the synthetic captions generated by VLM can
significantly improve the performance of our model across various open-world
object detection benchmarks. These observations qualitatively demonstrates the
quality of synthetic captions for training open-world object detection models.
In this section, we carry out a quantitative study for the quality of
synthetic captions generated by VLMs, by determining the ratio of noisy or
hallucinated objects in generated captions.

To quantify the noise level in our synthetic captions (i.e., % incorrect
objects described), we adopt the Caption Hallucination Assessment with Image
Relevance (CHAIR) metric from [44, 31] as our noise-level measurement for
synthetic captions. It can be formulated as the percentage of incorrectly
described objects,

$\text{noise \%}=\frac{1}{m}\sum_{i=1}^{m}\frac{N_{\text{incorrect
objects}}}{N_{\text{total objects}}}\times 100\%$ (16)

,where $N_{\text{incorrect objects}}$ denotes the number of hallucinated
objects in the caption, $N_{\text{total objects}}$ denotes the number of total
objects in the caption, and $m$ is the total number of synthetic captions
collected from the dataset. To obtain $N_{\text{incorrect objects}}$ and
$N_{\text{total objects}}$ in (Eq. (16)), we detect the synonyms of object
classes within the synthetic captions using WordNet synsets [39]. If an object
or its synonym is mentioned in the caption, but the detection ground truth
does not include this object, it is classified as an incorrect object. Our
result indicates a noise level of 16.3% on COCO 2017 test set, using BLIP2
6.7b. Despite the presence of noise and hallucinated contents in the synthetic
captions, we have proposed a hyperbolic learning strategy to mitigate the
issue of training with noisy synthetic captions. The hyperbolic learning loss
learns visual and caption embeddings hierarchically, imposing an entailment
relationship between captions and region embeddings, rather than directly
aligning them. Moreover, we suggest that employing an enhanced pre-trained VLM
for captioning could further mitigate the noise level.

##  F Future work

Building upon our current work in open-world object detection, there are
several promising avenues for future exploration. First, it is essential to
develop a new benchmark that evaluates open-world object detection using both
keywords and free-form text, overcoming the limitations of current benchmarks
that merely focus on either keywords (e.g., COCO, LVIS, ODinW) or free-form
texts (e.g., RefCOCO). Second, training image captioning models that generate
object-centric synthetic captions presents an intriguing research direction to
provide better open-world knowledge in object detection. Third, integrating
various data modalities or information sources to augment the model’s
detection capabilities is also an interesting direction, such as providing a
sketch of a target object to localize in the open world.

## References

  * [1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, 2020.
  * [2] Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré.  Low-dimensional hyperbolic knowledge graph embeddings.  arXiv preprint arXiv:2005.00545, 2020.
  * [3] Yanbei Chen, Manchen Wang, Abhay Mittal, Zhenlin Xu, Paolo Favaro, Joseph Tighe, and Davide Modolo.  Scaledet: A scalable multi-dataset object detector.  In CVPR, 2023.
  * [4] Han-Cheol Cho, Won Young Jhoo, Wooyoung Kang, and Byungseok Roh.  Open-vocabulary object detection using pseudo caption labels.  arXiv preprint arXiv:2303.13040, 2023.
  * [5] Shuyang Dai, Zhe Gan, Yu Cheng, Chenyang Tao, Lawrence Carin, and Jingjing Liu.  Apo-vae: Text generation in hyperbolic space.  arXiv preprint arXiv:2005.00054, 2020.
  * [6] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang.  Dynamic head: Unifying object detection heads with attentions.  In CVPR, 2021.
  * [7] Karan Desai and Justin Johnson.  Virtex: Learning visual representations from textual annotations.  In CVPR, 2021.
  * [8] Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedantam.  Hyperbolic image-text representations.  In ICML, 2023.
  * [9] Bhuwan Dhingra, Christopher J Shallue, Mohammad Norouzi, Andrew M Dai, and George E Dahl.  Embedding text in hyperbolic spaces.  arXiv preprint arXiv:1806.04313, 2018.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [11] Sivan Doveh, Assaf Arbelle, Sivan Harary, Amit Alfassy, Roei Herzig, Donghyun Kim, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, et al.  Dense and aligned captions (dac) promote compositional reasoning in vl models.  In NeurIPS, 2023.
  * [12] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann.  Hyperbolic entailment cones for learning hierarchical embeddings.  In ICML, 2018.
  * [13] Songwei Ge, Shlok Mishra, Simon Kornblith, Chun-Liang Li, and David Jacobs.  Hyperbolic contrastive learning for visual representations beyond objects.  In CVPR, 2023.
  * [14] Ross Girshick.  Fast r-cnn.  In ICCV, 2015.
  * [15] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  In ICLR, 2021.
  * [16] Yunhui Guo, Xudong Wang, Yubei Chen, and Stella X Yu.  Clipped hyperbolic classifiers are super-hyperbolic classifiers.  In CVPR, 2022.
  * [17] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In CVPR, 2019.
  * [18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [19] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In ICML, 2021.
  * [20] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.  Mdetr-modulated detection for end-to-end multi-modal understanding.  In ICCV, 2021.
  * [21] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg.  Referitgame: Referring to objects in photographs of natural scenes.  In EMNLP, 2014.
  * [22] Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky.  Hyperbolic image embeddings.  In CVPR, 2020.
  * [23] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  IJCV, 2017.
  * [24] Weicheng Kuo, Fred Bertsch, Wei Li, AJ Piergiovanni, Mohammad Saffar, and Anelia Angelova.  Findit: Generalized localization with natural language queries.  In ECCV, 2022.
  * [25] Marc Law, Renjie Liao, Jake Snell, and Richard Zemel.  Lorentzian distance learning for hyperbolic representations.  In ICML, 2019.
  * [26] Matt Le, Stephen Roller, Laetitia Papaxanthos, Douwe Kiela, and Maximilian Nickel.  Inferring concept hierarchies from text corpora via hyperbolic embeddings.  arXiv preprint arXiv:1902.00913, 2019.
  * [27] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11336–11344, 2020.
  * [28] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [29] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.  Grounded language-image pre-training.  In CVPR, 2022.
  * [30] Muchen Li and Leonid Sigal.  Referring transformer: A one-step approach to multi-task visual grounding.  In NeurIPS, 2021.
  * [31] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.  Evaluating object hallucination in large vision-language models.  arXiv preprint arXiv:2305.10355, 2023.
  * [32] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [33] Shilong Liu, Shijia Huang, Feng Li, Hao Zhang, Yaoyuan Liang, Hang Su, Jun Zhu, and Lei Zhang.  Dq-detr: Dual query detection transformer for phrase extraction and grounding.  In AAAI, 2023.
  * [34] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499, 2023.
  * [35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [36] Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa.  Differentiating through the fréchet mean.  In ICML, 2020.
  * [37] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy.  Generation and comprehension of unambiguous object descriptions.  In CVPR, 2016.
  * [38] Leland McInnes, John Healy, and James Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  arXiv preprint arXiv:1802.03426, 2018.
  * [39] George A Miller.  Wordnet: a lexical database for english.  Communications of the ACM, 38(11):39–41, 1995.
  * [40] Maximillian Nickel and Douwe Kiela.  Learning continuous hierarchies in the lorentz model of hyperbolic geometry.  In ICML, 2018.
  * [41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [42] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NeurIPS, 2015.
  * [43] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor.  Imagenet-21k pretraining for the masses.  arXiv preprint arXiv:2104.10972, 2021.
  * [44] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko.  Object hallucination in image captioning.  arXiv preprint arXiv:1809.02156, 2018.
  * [45] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.  Objects365: A large-scale, high-quality dataset for object detection.  In ICCV, 2019.
  * [46] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL, 2018.
  * [47] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In ICCV, 2019.
  * [48] Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer.  Image captioners are scalable vision learners too.  arXiv preprint arXiv:2306.07915, 2023.
  * [49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [50] Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy.  Aligning bag of regions for open-vocabulary object detection.  In CVPR, 2023.
  * [51] Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li.  Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching.  In CVPR, 2023.
  * [52] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao.  Unified contrastive learning in image-text-label space.  In CVPR, 2022.
  * [53] Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu.  Alip: Adaptive language-image pre-training with synthetic caption.  In ICCV, 2023.
  * [54] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu.  Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection.  In NeurIPS, 2022.
  * [55] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: Fine-grained interactive language-image pre-training.  In ICLR, 2021.
  * [56] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  arXiv preprint arXiv:2205.01917, 2022.
  * [57] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg.  Mattnet: Modular attention network for referring expression comprehension.  In CVPR, 2018.
  * [58] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg.  Modeling context in referring expressions.  In ECCV, 2016.
  * [59] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy.  Open-vocabulary detr with conditional matching.  In ECCV, 2022.
  * [60] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang.  Open-vocabulary object detection using captions.  In CVPR, 2021.
  * [61] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In CVPR, 2022.
  * [62] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection.  arXiv preprint arXiv:2203.03605, 2022.
  * [63] Haotian* Zhang, Pengchuan* Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao.  Glipv2: Unifying localization and vision-language understanding.  In NeurIPS, 2022.
  * [64] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.  Regionclip: Region-based language-image pretraining.  In CVPR, 2022.
  * [65] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra.  Detecting twenty-thousand classes using image-level supervision.  In ECCV, 2022.
  * [66] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Probabilistic two-stage detection.  arXiv preprint arXiv:2103.07461, 2021.
  * [67] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  arXiv preprint arXiv:1904.07850, 2019.
  * [68] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  In ICLR, 2020.