(A.29) $g(\sigma,O,z)=V^{\sigma}(z)-V^{\sigma^{\Omega\setminus O}}(z),\qquad
z\in\Omega,$

where $V^{\sigma}$ is the logarithmic potential of $\sigma$ and the left-hand
side is interpreted as $0$ for $z\in b(\Omega\setminus O)$. More general
versions when $\infty\in\Omega$ may be found in [55]. If moreover $O$ is a
domain with $K\subset O$ a non-polar compact set such that $\Omega\setminus K$
regular and $O\setminus K$ is non-thin at every point of $K$, then

(A.30) $\mathrm{supp}_{\mathsf{f}}\,\mu_{O,K}=K.$

Indeed, $K$ is its own base and $K=\partial_{\mathsf{f}}(O\setminus K)$ by
assumption, while $\mu_{O,K}$ is the balayage onto $K$ of the equilibrium
measure on the plate $\partial O$ of the condenser $(\partial O,K)$ [55,
Chapter VIII, Theorem 2.6]. Thus, (A.30) follows from (A.24) and (A.27).

###  A.10. Green Potentials in $D$ and on ${\mathcal{R}}$

In this subsection, we connect Green functions and potentials on the domain
$D$ and surface ${\mathcal{R}}$ defined in Section 2.1. First, let us show
that

(A.31) $g_{D}(x,y)=\sum_{z\in p^{-1}(x)}m(z)g_{{\mathcal{R}}}(z,w),$

where $m(z)$ is the ramification order of ${\mathcal{R}}$ at $z$ and $w$ is an
arbitrary element of the fiber $p^{-1}(y)$. To check (A.31), note that for
fixed $y\notin p(\mathbf{rp}({\mathcal{R}}))$ and $w\in{\mathcal{R}}$ with
$p(w)=y$, the right-hand side is well defined and harmonic as a function of
$x\in D\setminus(p(\mathbf{rp}({\mathcal{R}}))\cup\\{y\\})$. Thus, it is
harmonic for $x\in D\setminus\\{y\\}$ by the Removability Theorem and the
continuity of Green functions off the diagonal. Moreover, the right-hand side
clearly has a logarithmic singularity at $y$, and since
$\lim_{z\to\partial{\mathcal{R}}}g_{{\mathcal{R}}}(z,w)=0$ by the regularity
of ${\mathcal{R}}$ in ${\mathcal{R}}_{*}$, its largest harmonic minorant is
zero. This proves (A.31) when $y\notin p(\mathbf{rp}({\mathcal{R}}))$, and the
general case follows by continuity of Green functions off the diagonal.
Consequently, if $\nu$ is a Radon measure on ${\mathcal{R}}$ and $p_{*}(\nu)$
denotes its pushforward under $p$ (the measure on $D$ such that
$p_{*}(\nu)(B)=\nu(p^{-1}(B))$ for a Borel set $B$), integrating (A.31)
against $p_{*}(\nu)$ with respect to $y$ and changing variables yields

(A.32) $g(p_{*}(\nu),D;x)=\sum_{z\in p^{-1}(x)}m(z)g(\nu,{\mathcal{R}};z).$

In the other direction, for a Radon measure $\sigma$ on $D$, let us define
$\widehat{\sigma}$ by

(A.33) $\widehat{\sigma}(B):=\int_{p(B)}\sum_{z\in p^{-1}(x)\cap
B}m(z)d\sigma(x),\quad B\subset{\mathcal{R}},\quad B\mathrm{\ Borel}.$

As $p(B)$ is Borel when $B$ is Borel, one easily checks that (A.33) defines a
Radon measure on ${\mathcal{R}}$. In fact, one can verify that
$\widehat{\sigma}=\sigma^{*}+\sum_{z\in\mathbf{rp}({\mathcal{R}})}m(z)\sigma(\\{p(z)\\})\delta_{z}$,
where $\sigma^{*}$ is the pullback measure resulting from Carathéodory’s
construction as applied to the map $B\mapsto\sigma(p(B)\setminus
p(\mathbf{rp}({\mathcal{R}})))$ defined on Borel subsets of ${\mathcal{R}}$,
see [16, Theorem 2.10.10].

Partitioning $D\setminus p(\mathbf{rp}({\mathcal{R}}))$ into countably many
Borel sets $B_{k}$ such that $p:p^{-1}(B_{k})\to B_{k}$ induces a
homeomorphism on each connected component of $p^{-1}(B_{k})$, and invoking the
Removability Theorem to proceed by superharmonicity from the case where
$\zeta\notin\mathbf{rp}({\mathcal{R}})$, one deduces from (A.33) and (A.31)
that

(A.34)
$g\left(\widehat{\sigma},{\mathcal{R}};\zeta\right)=g(\sigma,D;p(\zeta)),\quad\zeta\in{\mathcal{R}}.$

As a consequence of definition (A.33), we claim that if a sequence
$\\{\mu_{n}\\}$ of finite positive measures supported on a fixed compact set
$K\subset D$ converges weak∗ to $\mu$ on $D$, then the sequence
$\\{\widehat{\mu}_{n}\\}$ converges weak∗ to $\widehat{\mu}$ on
${\mathcal{R}}$. Indeed, the total mass of $\mu_{n}$ is necessarily bounded
independently of $n$ by some $C>0$ (this follows from the Banach-Steinhaus
principle) and therefore, in view of (A.33), the total mass of
$\widehat{\mu}_{n}$ is bounded by $MC$, where $M$ is the number of sheets of
${\mathcal{R}}$. Hence, an arbitrary subsequence of
$\\{{\widehat{\mu}}_{n}\\}$ has a subsequence, say
$\\{{\widehat{\mu}}_{n_{k}}\\}$, that converges weak∗ on ${\mathcal{R}}$ to
some finite measure, say $s$. It follows from the Lower Envelope Theorem that
$\liminf_{k}g(\widehat{\mu}_{n_{k}},{\mathcal{R}};z)=g(s,{\mathcal{R}};z)$ for
quasi every $z\in{\mathcal{R}}$. Similarly, (A.34) and the Lower Envelope
Theorem, applied this time to $\\{\mu_{n_{k}}\\}$, yield that
$\liminf_{k}g(\widehat{\mu}_{n_{k}},{\mathcal{R}};z)=g(\mu,D;p(z))=g(\widehat{\mu},{\mathcal{R}};z)$
for quasi every $z\in{\mathcal{R}}$. Thus,
$g(s,{\mathcal{R}};\cdot)=g(\widehat{\mu},{\mathcal{R}};\cdot)$ quasi
everywhere on ${\mathcal{R}}$, and the claim follows by taking Laplacians on
both sides of this equality.

In the previous construction, ${\mathcal{R}}$ may of course be replaced by
another saturated connected bordered surface
$\mathcal{S}\subset{\mathcal{R}}_{*}$ with bounded projection such that
$\overline{\mathcal{R}}\subset\mathcal{S}$. Therefore,

$\Bigl{(}\mu_{n}\overset{w*}{\to}\mu\quad\text{in}\quad\overline{D}\Bigr{)}\quad\Rightarrow\quad\Bigl{(}\widehat{\mu}_{n}\overset{w*}{\to}\widehat{\mu}\quad\text{in}\quad\overline{\mathcal{R}}\Bigr{)},$

because $\\{\mu_{n}\\}$ also converges weak∗ in
$p(\mathcal{S})\supset\overline{D}$ whence the measures $\widehat{\mu}_{n}$
converge weak∗ on $\mathcal{S}$, while having their supports contained in
$\overline{\mathcal{R}}$. The “hat measure” constructed in (A.33) is
instrumental both in the proof of Lemma 3.9 and of the following technical
result, needed in the paper.

######  Lemma A.7.

Let $\sigma$ be a measure in $D$. Given $E\subset D$, set
$\widehat{E}:=p^{-1}(E)$. Then it holds that

$g\big{(}\sigma^{E},D;p(z)\big{)}=\int
g(\sigma,D;p(x))d\delta_{z}^{\widehat{E}}(x).$

###### Proof.

If $u$ is superharmonic on $D$, it is obvious from (A.34) that

$\Bigl{(}u(z)\geq g(\sigma,D;z),\quad z\in
E\Bigr{)}\quad\Rightarrow\quad\Bigl{(}u(p(\xi))\geq
g(\widehat{\sigma},{\mathcal{R}};\xi),\quad\xi\in\widehat{E}\Bigr{)}.$

As $u\circ p$ is superharmonic on ${\mathcal{R}}$, this and the definition of
balayage imply that

(A.35) $g\big{(}\sigma^{E},D;p(\zeta)\big{)}\geq
g\big{(}\widehat{\sigma}^{\widehat{E}},{\mathcal{R}};\zeta\big{)},\quad\zeta\in{\mathcal{R}}.$

Conversely, averaging over $\zeta\in p^{-1}(\\{z\\})$, equation (A.34) yields
that

(A.36) $g(\sigma,D;z)=\frac{1}{M}\sum_{\zeta\in
p^{-1}(\\{z\\})}m(\zeta)g(\widehat{\sigma},{\mathcal{R}};\zeta),\quad z\in D,$

with $M$ being the total number of sheets of ${\mathcal{R}}$, entailing when
$v$ is superharmonic on ${\mathcal{R}}$ that

$\Bigl{(}v(\zeta)\geq
g(\widehat{\sigma},{\mathcal{R}};\zeta),\quad\zeta\in\widehat{E}\Bigr{)}\quad\Rightarrow\quad\Bigl{(}\frac{1}{M}\sum_{\zeta\in
p^{-1}(z)}m(\zeta)v(\zeta)\geq g(\sigma,D;z),\quad z\in D\Bigr{)}.$

Now, the function $z\mapsto\sum_{\zeta\in p^{-1}(z)}m(\zeta)v(\zeta)$ is well-
defined and superharmonic on $D\setminus p(\mathbf{rp}({\mathcal{R}}))$ and
therefore on the whole domain $D$ by the Removability Theorem. Thus, by the
definition of balayage, we obtain when $v$ is superharmonic on ${\mathcal{R}}$
that

$\Bigl{(}v(\zeta)\geq
g(\widehat{\sigma},{\mathcal{R}};\zeta),\quad\zeta\in\widehat{E}\Bigr{)}\quad\Rightarrow\quad\Bigl{(}\frac{1}{M}\sum_{\zeta\in
p^{-1}(z)}m(\zeta)v(\zeta)\geq g(\sigma^{E},D;z),\quad z\in D\Bigr{)},$

and taking the infimum over $v$ before taking the lower semi-continuous
regularization gives us, by virtue of the Strong Domination Principle, that

$\frac{1}{M}\sum_{\zeta\in
p^{-1}(z)}m(\zeta)g(\widehat{\sigma}^{\widehat{E}},{\mathcal{R}};\zeta)\geq
g(\sigma^{E},D;z),\quad z\in D.$

Combining the above estimate with (A.35), we deduce that

$g\big{(}\sigma^{E},D;p(z)\big{)}=g\big{(}\widehat{\sigma}^{\widehat{E}},{\mathcal{R}};z\big{)},\quad
z\in{\mathcal{R}},$

and the conclusion now follows from (A.25) and (A.34). ∎

###  A.11. Dirichlet problem

The Dirichlet problem on a domain consists in finding a harmonic function in
that domain with given boundary data. In the fine Dirichlet problem, one looks
for a finely harmonic function on a fine domain to meet prescribed boundary
data. A real-valued function $h$ on a fine domain $V$ is _finely harmonic_ if
it is finely continuous, and if the fine topology on $V$ has a basis comprised
of finely open sets $E$ with $\mathrm{clos}_{\mathsf{f}}(E)\subset V$ such
that

$h(z)=\int hd\delta_{z}^{\Omega\setminus E}\quad\text{for every}\quad z\in E$

(in particular $h$ must be integrable with respect to
$\delta_{z}^{\Omega\setminus E}$ for all $z\in E$ and each $E$); one may even
assume that $E$ is regular and has compact closure (with respect to the
Euclidean topology) in $V$, see [20, Sections 8 & 14]. Note that a function
harmonic in a domain is finely harmonic on any fine subdomain, see [20,
Theorem 8.7].

If $V$ is a regular finely open set (recall that it means $\Omega\setminus V$
is its own base), then $b(\partial_{\mathsf{f}}V)=\partial_{\mathsf{f}}V$ by
(A.10) whence the result below is a special case of [20, Theorem 14.1] and its
proof.

######  Theorem A.8.

Let $V\subset\Omega$ be a finely open set such that $\Omega\setminus V$ is
non-thin at every point of itself, i.e., such that $V$ is regular. If $\psi$
is a finely continuous function on $\partial_{\mathsf{f}}V$, majorized in
absolute value there by a finite semi-bounded potential on $\Omega$, say $g$,
then

(A.37) $h_{\psi}(z):=\int\psi d\delta_{z}^{\Omega\setminus V}=\int\psi
d\delta_{z}^{\partial_{\mathsf{f}}V},\quad z\in V,$

is the unique finely continuous extension of $\psi$ to
$\mathrm{clos}_{\mathsf{f}}(V)$ that is finely harmonic in $V$ and is
majorized in absolute value there by a semi-bounded potential. In fact, it
holds that $|h_{\psi}|\leq g$ on $\mathrm{clos}_{\mathsf{f}}(V)$.

The lemma below addresses the question as to when constant functions solve the
fine Dirichlet problem on $V\subset\Omega$ or, equivalently by (A.37), as to
when the balayage of $\delta_{z}$ out of $V$, $z\in V$, has unit mass. We
recall that an overline, as in $\overline{V}$, or a $\partial$ sign, as in
$\partial\Omega$, refer respectively to the closure and boundary with respect
to the Euclidean topology induced by the ambient Riemanian manifold
(${\mathcal{R}}_{*}$ or $\mathbb{C}$). In contrast, fine closures and fine
boundaries as in $\partial_{\mathsf{f}}V$ and $\mathrm{clos}_{\mathsf{f}}(V)$
refer to the fine topology on $\Omega$; thus, $\partial\Omega$ is “invisible”
from the point of view of fine topology in $\Omega$, and if $V\subset\Omega$
then $\partial V\cap\partial\Omega$ is disjoint from $\partial_{\mathsf{f}}V$
as the latter is included in $\Omega$.

######  Lemma A.9.

Let $V$ be a proper nonempty regular fine domain in $\Omega$, which itself is
regular within the ambient Riemann surface (${\mathcal{R}}_{*}$ or
$\mathbb{C}$). Then it holds for $z\in V$ that

(A.38) $\int d\delta_{z}^{\Omega\setminus
V}\left\\{\begin{array}[]{ll}=1&\text{if
}\overline{V}\cap\partial\Omega=\varnothing,\vskip 6.0pt plus 2.0pt minus
2.0pt\\\ <1&\text{if
}\overline{\partial_{\mathsf{f}}V}\cap\partial\Omega=\varnothing\text{ and
}\overline{V}\cap\partial\Omega\neq\varnothing.\end{array}\right.$

Moreover, if either condition on the right-hand side of (A.38) holds, then for
any harmonic function $h$ on $\Omega$ one has

(A.39) $h(z)=\int hd\delta_{z}^{\Omega\setminus V},\qquad z\in V,$

provided that $|h|$ is majorized on $V$ by a semi-bounded potential in
$\Omega$ when $\overline{V}\cap\partial\Omega\neq\varnothing$.

###### Proof.

Let $K\subset\Omega$ be non-polar and compact. Set, for brevity,
$g_{K}:=\mathrm{cap}_{\Omega}(K)g(\mu_{\Omega,K},\Omega;\cdot)$, where
$\mu_{\Omega,K}$ indicates, as in Section A.4, the Green equilibrium
distribution on $K$. Since $\mu_{\Omega,K}$ has finite energy, $g_{K}$ is
semi-bounded. As $\partial\Omega$ is regular in the ambient Riemann surface,
$g_{K}$ extends continuously by zero to $\partial\Omega$, see Section A.7.
Since $\mathrm{cap}_{\Omega}(K)>0$, it holds that $g_{K}\leq 1$ in $\Omega$,
see the paragraph after (A.5), and $g_{K}=1$ on $b(K)$ because
$g_{K}=\mathcal{B}_{1}^{K}$, see Section A.9. Moreover, $g_{K}<1$ in each
connected component $U$ of $\Omega\setminus K$ such that $\partial
U\cap\partial\Omega\neq\varnothing$ by the maximum principle for harmonic
functions.

When $\overline{V}$ is compactly included in $\Omega$, we may put
$K:=\overline{V}$ in what precedes, and then $\mathrm{cap}_{\Omega}(K)>0$ as
otherwise $V$ itself would be polar and therefore empty, since it is finely
open. The infimum of $g_{K}$ on $K$ is attained by lower semi-continuity, and
it is strictly positive because nonzero Green potentials are never zero.
Therefore, if $h$ is harmonic on $\Omega$, the potential $cg_{K}$ majorizes
$|h|$ on $K$ for sufficiently large $c>0$. The uniqueness part of Theorem A.8
now implies that $h_{\mathcal{b}V}$ is the solution of the fine Dirichlet
problem with boundary data $h_{\mathcal{b}\partial_{\mathsf{f}}V}$. Hence,
(A.39) is just (A.37) while the upper equality in (A.38) follows by taking
$h\equiv 1$.

Assume next that $\overline{V}\cap\partial\Omega\neq\varnothing$ and
$\overline{\partial_{\mathsf{f}}V}\cap\partial\Omega=\varnothing$. Then
$K:=\overline{\partial_{\mathsf{f}}V}$ is a compact subset of $\Omega$ which
is non-polar, for if $\partial_{\mathsf{f}}V$ were polar, then either $V$ or
$\Omega\setminus V$ would be polar [19, Theorem 2] and $V$ would be either
empty or irregular, a contradiction. Note that a subdomain of $\Omega\setminus
K$ is also a fine domain [19, Theorem 2] and thus, if it contains both a point
in $V$ and a point in $\Omega\setminus V$, then it must contain a point in
$\partial_{\mathsf{f}}V$ which is impossible by the definition of $K$. Hence,
$V\setminus K$ is Euclidean open.

Let $U$ be a connected component of $V\setminus K$ such that $\partial
U\cap\partial\Omega\neq\varnothing$; it exists because
$\overline{V}\cap\partial\Omega\neq\varnothing$. Since
$\partial_{\mathsf{f}}V=b(\partial_{\mathsf{f}}V)\subseteq b(K)$ by
assumption, $g_{K}=1$ on $\partial_{\mathsf{f}}V$ (see the beginning of the
proof). Hence, as $g_{K}$ is a semi-bounded potential on $\Omega$, it follows
from Theorem A.8 that $h_{\bf 1}\leq g_{K}$ in $V$, where $h_{\bf 1}(z)$ is
the solution of the fine Dirichlet problem on $V$ with boundary data
identically $1$ on $\partial_{\mathsf{f}}V$, see (A.37). In particular,
$h_{\bf 1}\leq g_{K}<1$ in $U$ by the maximum principle for harmonic
functions.

Let $F\subset U$ be a closed disk. Observe that $V\setminus F$ is a fine
domain as otherwise $F$ would finely disconnect $U$ whereas $U\setminus F$ is
a domain and therefore also a fine domain. Put
$\delta_{z}^{*}:=\delta_{z}^{(\Omega\setminus V)\cup F}$, $z\in V\setminus F$,
and observe that $\delta_{z\mathcal{b}\partial_{\mathsf{f}}F}^{*}$ is a non-
trivial measure by (A.27) and since $\partial_{\mathsf{f}}((\Omega\setminus
V)\cup F)=\partial_{\mathsf{f}}V\cup\partial_{\mathsf{f}}F$, where the union
is disjoint and $b(\partial_{\mathsf{f}}F)=\partial_{\mathsf{f}}F=\partial F$.
We now get from (A.23) and (A.27), applied to the fine domain $V\setminus F$,
that

(A.40) $\delta_{z}^{\Omega\setminus
V}=\big{(}\delta^{*}_{z}\big{)}^{\Omega\setminus
V}=\big{(}\delta_{z\mathcal{b}\partial_{\mathsf{f}}V}^{*}\big{)}^{\Omega\setminus
V}+\big{(}\delta_{z\mathcal{b}\partial_{\mathsf{f}}F}^{*}\big{)}^{\Omega\setminus
V}=\delta_{z\mathcal{b}\partial_{\mathsf{f}}V}^{*}+\big{(}\delta_{z\mathcal{b}\partial_{\mathsf{f}}F}^{*}\big{)}^{\Omega\setminus
V},$

where we observe that the balayage out of $V$ does not change measures
supported on $b(\partial_{\mathsf{f}}V)=\partial_{\mathsf{f}}V$ by (A.22).
Since $h_{\bf 1}<1$ on $\partial_{\mathsf{f}}F$ as
$\partial_{\mathsf{f}}F\subset U$, we get from (A.24) and (A.26) that

$0<\big{(}\delta_{z\mathcal{b}\partial_{\mathsf{f}}F}^{*}\big{)}^{\Omega\setminus
V}(\partial_{\mathsf{f}}V)=\int\delta_{x}^{\Omega\setminus
V}(\partial_{\mathsf{f}}V)d\delta_{z\mathcal{b}\partial_{\mathsf{f}}F}^{*}(x)=\int
h_{\bf
1}(x)d\delta_{z\mathcal{b}\partial_{\mathsf{f}}F}^{*}(x)<\delta_{z\mathcal{b}\partial_{\mathsf{f}}F}^{*}(\partial_{\mathsf{f}}F)$

which implies, in view of (A.40), that $h_{\bf
1}(z)=\delta_{z}^{\Omega\setminus
V}(\partial_{\mathsf{f}}V)<\delta_{z}^{*}((\Omega\setminus V)\cup F)\leq 1$ as
claimed.

Finally, let $h$ be a harmonic function on $\Omega$ which is majorized on $V$
in absolute value by a semi-bounded potential. Because $h$ is also finely
harmonic on $V$ and finely continuous on $\mathrm{clos}_{\mathsf{f}}(V)$, we
deduce from Theorem A.8 that it is the solution of the fine Dirichlet problem
on $V$ with boundary data $h_{\mathcal{b}\partial_{\mathsf{f}}V}$ and that
(A.39) holds. ∎

######  Lemma A.10.

Let $V\subset{\mathcal{R}}$ be a proper regular fine domain such that
$\overline{\partial_{\mathsf{f}}V}\cap\partial{\mathcal{R}}=\varnothing$ and
$\overline{V}\cap\partial{\mathcal{R}}\neq\varnothing$ (the lower assumption
on the right-hand side of (A.38) when $\Omega={\mathcal{R}}$). Let further $h$
be a harmonic function in ${\mathcal{R}}$ such that $\lim_{z\to\xi}h(z)=0$ for
every $\xi\in\overline{V}\cap\partial{\mathcal{R}}$. Then identity (A.39)
holds.

###### Proof.

In view of Lemma A.9, it is enough to show that $h$ is majorized on $V$ by a
semi-bounded potential. Note, as in the proof of Lemma A.9, that
$\partial_{\mathsf{f}}V$ is non-polar. Let us show that
$\partial{\mathcal{R}}\cap\overline{V}$ consists of a union of connected
components of $\partial{\mathcal{R}}$. Indeed, any such component $\Gamma$ is
a 1-dimensional compact topological submanifold of ${\mathcal{R}}_{*}$, and as
such it has a tubular neighborhood $N$ that may be chosen so thin that
$N\cap\overline{\partial_{\mathsf{f}}V}=\varnothing$. Then, if
$\zeta_{1},\zeta_{2}\in\Gamma$ and $\zeta_{1}\in\overline{V}$ while
$\zeta_{2}\notin\overline{V}$, we can find $z_{1}\in N\cap V$ close to
$\zeta_{1}$ and $z_{2}\in N\cap\Omega\setminus V$ close to $\zeta_{2}$. The
points $z_{1}$, $z_{2}$ can be joined by a smooth arc contained in $N$.
However, such an arc is finely connected [19, Theorem 7], but cannot meet
$\overline{\partial_{\mathsf{f}}V}$ by construction, a contradiction that
proves our claim.

Assume first that $D=\mathbb{D}$ is the unit disk. Any function $u$ harmonic
in an annular region $\\{r<|z|<1\\}$ that extends continuously to $\mathbb{T}$
by zero can be harmonically extended to $\\{r<|z|<1/r\\}$ by reflection, i.e.,
by setting $u(z):=-u(1/\bar{z})$ for $z\in\\{1<|z|<1/r\\}$. Due to the
smoothness of this extension it necessarily holds that $|u(z)|\leq
C_{\rho}(1-|z|)$ for $r<\rho\leq|z|\leq 1$. As $h$ is harmonic on
${\mathcal{R}}$, this principle used around each of the finitely many
connected components of $\overline{V}\cap\partial{\mathcal{R}}$ yields that
$|h(z)|\leq C(1-|p(z)|)$ for $z\in V$ and some constant $C>0$. On the other
hand, the function $g_{r}(z):=-\log\max\\{r,|z|\\}$ is a continuous (thus
bounded and therefore semi-bounded) potential in $\mathbb{D}$ for any
$r\in(0,1)$ (this is the Green equilibrium potential of $\\{|z|\leq r\\}$). It
can be readily verified that $g_{r}(z)\geq(1-|z|)$ in $\mathbb{D}$ when $r\leq
e^{-1}$. Thus, $|h(z)|\leq Cg_{r}(p(z))$, $z\in V$, for any such $r$. As
$g_{r}(p(z))$ is a (bounded) potential on ${\mathcal{R}}$ by (A.34), the claim
of the corollary follows.

In the general case, let $\phi:\mathbb{D}\to D$ be a conformal map. Recall
that $\phi$ extends to a homeomorphism from $\overline{\mathbb{D}}$ onto
$\overline{D}$. Clearly, $\phi$ is also a homeomorphism for the fine topology
since $v$ is superharmonic (resp. harmonic) on $D$ if and only if so is
$v\circ\phi$ on $\mathbb{D}$. Moreover, $g$ is a bounded potential on $D$ if
and only if $g\circ\phi$ is such a potential on $\mathbb{D}$. Hence, the
result just proven on the disk carries over to $D$ by conformal mapping. ∎

## References

  * [1] P. Avery, C. Farhat, and G. Reese.  Fast frequency sweep computations using a multipoint Padé based reconstruction method and an efficient iterative solver.  International J. for Num. Methods in Engin., 69(13):2848–2875, 2007\.
  * [2] Th. Bagby.  On interpolation by rational functions.  Duke Math. J., 36:95–104, 1969.
  * [3] G.A. Baker and P. Graves-Morris.  Padé Approximants, volume 59 of Encyclopedia of Mathematics and its Applications.  Cambridge University Press, 1996.
  * [4] L. Baratchart, F. Mandrèa, E.B. Saff, and F. Wielonsky.  2-D inverse problems for the Laplacian: a meromorphic approximation approach.  J. Math. Pures Appl., 86:1–41, 2006.
  * [5] L. Baratchart and F. Seyfert.  An $l^{p}$ analog to aak theory for $p\geq 2$.  Journal of Functional Analysis, 191:52–122, 2002.
  * [6] L. Baratchart, H. Stahl, and M. Yattselev.  Weighted extremal domains and best rational approximation.  Adv. Math., 229:357–407, 2012.
  * [7] D. Braess.  Nonlinear Approximation Theory.  Springer, 1986.
  * [8] M. Brelot.  On Topologies and Boundaries in Potential Theory.  Lecture Notes in Mathematics, Vol. 175. Springer-Verlag, Berlin-New York, 1971.  Enlarged edition of a course of lectures delivered in 1966.
  * [9] L. Carleson and S. Jacobs.  Best uniform approximation by analytic functions.  Ark. Mat., 10:219–229, 1972.
  * [10] C. Constantinescu.  Some properties of the balayage of measures on a harmonic space.  Annales de l’institut Fourier, 17(1):273–293, 1967.
  * [11] F. Demengel and G. Demengel.  Functional Spaces for the Theory of Elliptic Partial Differential Equations.  Universitext. Springer, 2012.
  * [12] P.L. Duren.  Theory of $H^{p}$ Spaces.  Pure and Applied Mathematics, Vol. 38. Academic Press, New York-London, 1970.
  * [13] A. Eremenko, B. Fuglede, and M. Sodin.  On the Riesz charge of the lower envelope of $\delta$-subharmonic functions.  Potential Analysis, 1:191–204, 1992.
  * [14] L.C. Evans and R.F. Gariepy.  Measure Theory and Fine Properties of Functions.  Textbooks in Mathematics. CRC Press, Inc., Boca Raton, Florida, 2015.  revised edition.
  * [15] H.M. Farkas and I. Kra.  Riemann Surfaces, volume 71 of Graduate Texts in Mathematics.  Springer-Verlag, New York, second edition, 1992.
  * [16] H. Federer.  Geometric Measure Theory.  Die Grundlehren der mathematischen Wissenschaften, Band 153. Springer-Verlag New York, Inc., New York, 1969.
  * [17] S.-I. Filip, Y. Nakatsukasa, L.N. Trefethen, and B. Beckermann.  Rational minimax approximation via adaptive barycentric representations.  SIAM Journal on Scientific Computing, 40(4):A2427–A2455, 2018.
  * [18] O. Forster.  Lectures on Riemann Surfaces, volume 81 of Graduate Texts in Mathematics.  Springer-Verlag, New York, 1991.  Translated from the 1977 German original by Bruce Gilligan, Reprint of the 1981 English translation.
  * [19] B. Fuglede.  Connexion en topologie fine et balayage des mesures.  Ann. Inst. Fourier (Grenoble), 21(3):227–244, 1971.
  * [20] B. Fuglede.  Finely Harmonic Functions.  Lecture Notes in Mathematics, Vol. 289. Springer-Verlag, Berlin-New York, 1972.
  * [21] J.B. Garnett.  Bounded Analytic Functions.  Number 236 in GTM. Springer, 2007.
  * [22] A.A. Gonchar, N.N. Novikova, and G.M. Henkin.  Multipoint Padé approximants in the inverse Sturm-Liouville problem.  Mat. Sb., 182(8):1118–1128, 1991.
  * [23] A.A. Gonchar and E.A. Rakhmanov.  Equilibrium distributions and degree of rational approximation of analytic functions.  Mat. Sb. (N.S.), 134(176)(3):306–352, 447, 1987.
  * [24] A. Gopal and L. Trefheten.  Solving laplace problems with corner singularities via rational functions.  SIAM Journal on Numerical Analysis, 57(5):2074–2094, 2019.
  * [25] W.B. Gragg.  On extrapolation algorithms for ordinary initial value problems.  SIAM J. Num. Anal., 2:384–403, 1965.
  * [26] K. Gustafson.  Continued fractions, wavelet time operators, and inverse problems.  Rocky Mountain Journal of Mathematics, 33(2):661–88, 2003.
  * [27] B. Gustavsen and A. Semlyen.  Rational approximation of frequency domain responses by vector fitting.  IEEE Transactions on Power Delivery, 14(3):1052–1061, 1999.
  * [28] V.P. Havin, S.V. Hruščëv, and N.K. Nikolskiĭ, editors.  Linear and Complex Analysis Problem Book, volume 1043 of Lecture Notes in Mathematics.  Springer-Verlag, Berlin, 1984.  199 research problems.
  * [29] L.L. Helms.  Potential Theory.  Universitext. Springer, London, second edition, 2014.
  * [30] C. Hermite.  Sur la fonction exponentielle.  C. R. Acad. Sci. Paris, 77:18–24, 74–79, 226–233, 285–293, 1873\.
  * [31] V. Isakov.  Inverse Problems for Partial Differential Equations, volume 127 of Applied Mathematic Sciences.  Springer, 1998.
  * [32] A. Iserles and S.P. Norsett.  Order Stars.  Chapman and Hall, London, 1991.
  * [33] Ch. Krittenthaler and T. Rivoal.  Hypergéométrie et fonction zêta de Riemann.  Number 875 in Memoirs of the AMS. American Mathematical Society, 2007\.
  * [34] V.G. Kuzmina.  Modules of families of curves and quadratic differentials.  Trudy Mat. Inst. Steklov, 139:1–240, 1980.  English transl. in St. Petersburg Math. J., 9:455–507, 889–930, 1982.
  * [35] P. Lelong.  Fonctions plurisousharmoniques et fonctions analytiques de variables réelles.  Annales de l’Institut Fourier, 11, 1961.
  * [36] P. Lelong.  Fonctions plurisousharmoniques et fonctions analytiques de variables réelles.  Ann. Inst. Fourier (Grenoble), 11:515–562, 1961.
  * [37] A.L. Levin and V.M. Tihomirov.  Approximations of analytic functions by rational functions.  Dokl. Akad. Nauk SSSR, 174:279–282, 1967.
  * [38] D.S. Lubinsky.  On uniform convergence of diagonal multipoint pade approximants for entire functions.  Constr. Approx., 49(1):149–174, 2019.
  * [39] D.H. Luecking.  Boundary behavior of Green potentials.  Proc. Amer. Math. Soc., 96(3):481–488, 1986.
  * [40] W. Magnus.  Braids and riemann surfaces.  Communications on Pure and Applied Mathematics, 25:151–161, 1972\.
  * [41] J.E. McMillan and G. Piranian.  Compression and expansion of boundary sets.  Duke Math. J., 40:599–605, 1973.
  * [42] Y. Nakatsukasa, O. Sète, and L. Trefethen.  The AAA algorithm for rational approximation.  SIAM J. Sci. Comput., 218.
  * [43] Z. Nehari.  On bounded bilinear forms.  Ann. of Math. (2), 65:153–162, 1957.
  * [44] N.K. Nikolskii.  Treatise on the Shift Operator, volume 273 of Grundlehren der Math. Wissenschaften.  Springer, 1986.
  * [45] J. Nutall.  Padé polynomial asymptotics from a singular integral equation.  Constr. Approx., 6:157–166, 1990.
  * [46] J. Nutall and S.R. Singh.  Orthogonal polynomials and Padé approximants associated with a system of arcs.  J. Approx. Theory, 21:1–42, 1980.
  * [47] O.G. Parfënov.  Estimates for singular numbers of the Carleson embedding operator.  Mat. Sb. (N.S.), 131(173)(4):501–518, 1986.
  * [48] V.V. Peller.  Hankel operators and their applications.  Springer Monographs in Mathematics. Springer-Verlag, New York, 2003.
  * [49] Ch. Pommerenke.  Padé approximants and convergence in capacity.  Journal of Mathematical Analysis and Applications, 41(3):775–780, 1973.
  * [50] Ch. Pommerenke.  Boundary Behaviour of Conformal Maps, volume 299 of Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences].  Springer-Verlag, Berlin, 1992.
  * [51] A. Pozzi.  Applications of Padé Approximation in Fluid Dynamics, volume 14 of Advances in Maths. for Applied Sci. World Scientific, 1994.
  * [52] V.A. Prokhorov.  Rational approximation of analytic functions.  Mat. Sb., 184(2):3–32, 1993.
  * [53] Th. Ransford.  Potential Theory in the Complex Plane, volume 28 of London Mathematical Society Student Texts.  Cambridge University Press, Cambridge, 1995.
  * [54] W. Rudin.  Real and Complex Analysis.  Mc Graw-Hill, 1986.
  * [55] E.B. Saff and V. Totik.  Logarithmic Potentials with External Fields, volume 316 of Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences].  Springer-Verlag, Berlin, 1997.  Appendix B by Thomas Bloom.
  * [56] C.L. Siegel.  Transcendental Numbers.  Princeton Univ. Press, 1949.
  * [57] H. Stahl.  Extremal domains associated with an analytic function I.  Complex Variables Theory Appl., 4(4):311–324, 1985.
  * [58] H. Stahl.  Extremal domains associated with an analytic function II.  Complex Variables Theory Appl., 4(4):325–338, 1985.
  * [59] H. Stahl.  The structure of extremal domains associated with an analytic function.  Complex Variables Theory Appl., 4(4):339–354, 1985.
  * [60] H. Stahl.  Orthogonal polynomials with complex valued weight function. I, II.  Constr. Approx., 2(3):225–240, 241–251, 1986.
  * [61] H. Stahl.  On the convergence of generalized Padé approximants.  Constr. Approx., 5(2):221–240, 1989.
  * [62] H. Stahl.  The convergence of Padé approximants to functions with branch points.  J. Approx. Theory, 91:139–204, 1997.
  * [63] E.M. Stein.  Singular Integrals and Differentiability Properties of Functions.  Princeton Univ. Press, 1970.
  * [64] K. Strebel.  Quadratic Differentials, volume 5 of Ergebnisse der Mathematik und ihrer Grenzgebiete (3) [Results in Mathematics and Related Areas (3)].  Springer-Verlag, Berlin, 1984.
  * [65] J.J. Telega, S. Tokarzewski, and A. Galka.  Numerical analysis and its applications, chapter Modelling torsional properties of human bones by multipoint Padé approximants, pages 33–38.  Lect. Notes in Comp. Sci. Springer, 2001.
  * [66] J.A. Tjon.  Operator Padé approximants and three body scattering.  In E. B. Saff and R. S. Varga, editors, Padé and Rational Approximation, pages 389–396, 1977.
  * [67] J.L. Walsh.  Interpolation and Approximation by Rational Functions in the Complex Domain.  American Mathematical Society Colloquium Publications, Vol. XX. American Mathematical Society, Providence, R.I., fourth edition, 1965.
  * [68] H. Whitney.  Analytic extensions of differentiable functions defined in closed sets.  Transactions of the Amer. Math. Soc., pages 63–89, 1934.


manuscript.

This publication received funding from the Excellence Initiative of Aix-
Marseille University - A*MIDEX, a French “Investissements d’Avenir” programme,
AMX-18-ACE-005.

## References

  * (1) G. Bennett et al. (Muon g-2), Phys. Rev. D 73, 072003 (2006), hep-ex/0602035
  * (2) T. Aoyama, M. Hayakawa, T. Kinoshita, M. Nio, Phys. Rev. Lett. 109, 111808 (2012), 1205.5370
  * (3) T. Aoyama, T. Kinoshita, M. Nio, Atoms 7, 28 (2019)
  * (4) A. Czarnecki, W.J. Marciano, A. Vainshtein, Phys. Rev. D 67, 073006 (2003), [Erratum: Phys.Rev.D 73, 119901 (2006)], hep-ph/0212229
  * (5) C. Gnendiger, D. Stöckinger, H. Stöckinger-Kim, Phys. Rev. D 88, 053005 (2013), 1306.5546
  * (6) G. Venanzoni (Fermilab E989), Nucl. Part. Phys. Proc. 273-275, 584 (2016), 1411.2555
  * (7) M. Otani (E34), JPS Conf. Proc. 8, 025010 (2015)
  * (8) T. Aoyama et al. (2020), 2006.04822
  * (9) M. Davier, A. Hoecker, B. Malaescu, Z. Zhang, Eur. Phys. J. C 80, 241 (2020), [Erratum: Eur.Phys.J.C 80, 410 (2020)], 1908.00921
  * (10) A. Keshavarzi, D. Nomura, T. Teubner, Phys. Rev. D 101, 014029 (2020), 1911.00367
  * (11) F. Jegerlehner, EPJ Web Conf. 166, 00022 (2018), 1705.00263
  * (12) S. Borsanyi et al. (2020), 2002.12347
  * (13) J. Bijnens, E. Pallante, J. Prades, Phys. Rev. Lett. 75, 1447 (1995), [Erratum: Phys.Rev.Lett. 75, 3781 (1995)], hep-ph/9505251
  * (14) J. Bijnens, E. Pallante, J. Prades, Nucl. Phys. B 474, 379 (1996), hep-ph/9511388
  * (15) J. Prades, E. de Rafael, A. Vainshtein, Adv. Ser. Direct. High Energy Phys. 20, 303 (2009), 0901.0306
  * (16) G. Colangelo, M. Hoferichter, M. Procura, P. Stoffer, JHEP 09, 091 (2014), 1402.7081
  * (17) G. Colangelo, M. Hoferichter, B. Kubis, M. Procura, P. Stoffer, Phys. Lett. B 738, 6 (2014), 1408.2517
  * (18) V. Pauk, M. Vanderhaeghen (2014), 1403.7503
  * (19) G. Colangelo, M. Hoferichter, M. Procura, P. Stoffer, JHEP 09, 074 (2015), 1506.01386
  * (20) M. Hayakawa, T. Blum, T. Izubuchi, N. Yamada, PoS LAT2005, 353 (2006), hep-lat/0509016
  * (21) T. Blum, N. Christ, M. Hayakawa, T. Izubuchi, L. Jin, C. Jung, C. Lehner, Phys. Rev. Lett. 124, 132002 (2020), 1911.08123
  * (22) M. Davier, A. Hoecker, B. Malaescu, Z. Zhang, Eur. Phys. J. C 77, 827 (2017), 1706.09436
  * (23) A. Keshavarzi, D. Nomura, T. Teubner, Phys. Rev. D 97, 114025 (2018), 1802.02995
  * (24) G. Colangelo, M. Hoferichter, P. Stoffer, JHEP 02, 006 (2019), 1810.00007
  * (25) M. Hoferichter, B.L. Hoid, B. Kubis, JHEP 08, 137 (2019), 1907.01556
  * (26) A. Kurz, T. Liu, P. Marquard, M. Steinhauser, Phys. Lett. B 734, 144 (2014), 1403.6400
  * (27) G. Colangelo, M. Hoferichter, A. Nyffeler, M. Passera, P. Stoffer, Phys. Lett. B 735, 90 (2014), 1403.7512
  * (28) T. Blum, Phys. Rev. Lett. 91, 052001 (2003), hep-lat/0212018
  * (29) B. Colquhoun, R. Dowdall, C. Davies, K. Hornbostel, G. Lepage, Phys. Rev. D 91, 074514 (2015), 1408.5768
  * (30) D. Bernecker, H.B. Meyer, Eur. Phys. J. A 47, 148 (2011), 1107.4388
  * (31) B. Chakraborty, C. Davies, G. Donald, R. Dowdall, P. Gonçalves de Oliveira, J. Koponen, G.P. Lepage, T. Teubner, PoS LATTICE2014, 129 (2015), 1410.8466
  * (32) M. Della Morte, A. Francis, V. Gülpers, G. Herdoíza, G. von Hippel, H. Horch, B. Jäger, H. Meyer, A. Nyffeler, H. Wittig, JHEP 10, 020 (2017), 1705.01775
  * (33) B. Chakraborty, C. Davies, G. Donald, R. Dowdall, J. Koponen, G. Lepage, T. Teubner (HPQCD), Phys. Rev. D 89, 114501 (2014), 1403.1778
  * (34) A. Gérardin, M. Cè, G. von Hippel, B. Hörz, H.B. Meyer, D. Mohler, K. Ottnad, J. Wilhelm, H. Wittig, Phys. Rev. D 100, 014510 (2019), 1904.03120
  * (35) T.A. DeGrand, S. Schaefer, Nucl. Phys. B Proc. Suppl. 140, 296 (2005), hep-lat/0409056
  * (36) L. Giusti, P. Hernandez, M. Laine, P. Weisz, H. Wittig, JHEP 04, 013 (2004), hep-lat/0402002
  * (37) T. Blum, P. Boyle, V. Gülpers, T. Izubuchi, L. Jin, C. Jung, A. Jüttner, C. Lehner, A. Portelli, J. Tsang (RBC, UKQCD), Phys. Rev. Lett. 121, 022003 (2018), 1801.07224
  * (38) C. Aubin, T. Blum, C. Tu, M. Golterman, C. Jung, S. Peris, Phys. Rev. D 101, 014503 (2020), 1905.09307
  * (39) G.S. Bali, S. Collins, A. Schafer, Comput. Phys. Commun. 181, 1570 (2010), 0910.3970
  * (40) T. Blum, T. Izubuchi, E. Shintani, Phys. Rev. D 88, 094503 (2013), 1208.4349
  * (41) E. Shintani, R. Arthur, T. Blum, T. Izubuchi, C. Jung, C. Lehner, Phys. Rev. D 91, 114511 (2015), 1402.0244
  * (42) C. Lehner, A.S. Meyer, Phys. Rev. D 101, 074515 (2020), 2003.04177
  * (43) E. Shintani, Y. Kuramashi (PACS), Phys. Rev. D 100, 034517 (2019), 1902.00885
  * (44) C. Davies et al. (Fermilab Lattice, LATTICE-HPQCD, MILC), Phys. Rev. D 101, 034512 (2020), 1902.04223
  * (45) M. Dalla Brida, L. Giusti, T. Harris, M. Pepe (2020), 2007.02973
  * (46) M. Cè, L. Giusti, S. Schaefer, Phys. Rev. D 93, 094507 (2016), 1601.04587
  * (47) M. Cè, L. Giusti, S. Schaefer, Phys. Rev. D 95, 034503 (2017), 1609.02419
  * (48) E.H. Chao, A. Gérardin, J.R. Green, R.J. Hudspith, H.B. Meyer, Contribution to the Third Plenary Workshop of the Muon g-2 Theory Initiative (2019), https://indico.fnal.gov/event/21626/contributions
/63777/

  * (49) C. Andersen, J. Bulava, B. Hörz, C. Morningstar, Nucl. Phys. B 939, 145 (2019), 1808.05007
  * (50) M. Luscher, Nucl. Phys. B 364, 237 (1991)
  * (51) S. Lahert, Contribution to the Workshop :The hadronic vacuum polarization from lattice QCD at high precision. (2020), https://indico.cern.ch/event/956699/contributions/
4108692/

  * (52) C. Lehner, RBRC Workshop on Lattice Gauge Theories (2016)
  * (53) S. Borsanyi et al. (Budapest-Marseille-Wuppertal), Phys. Rev. Lett. 121, 022002 (2018), 1711.04980
  * (54) J. Gasser, H. Leutwyler, Annals Phys. 158, 142 (1984)
  * (55) G. Ecker, J. Gasser, A. Pich, E. de Rafael, Nucl. Phys. B 321, 311 (1989)
  * (56) J. Bijnens, J. Relefors, JHEP 12, 114 (2017), 1710.04479
  * (57) C. Aubin, T. Blum, M. Golterman, S. Peris (2020), 2008.03809
  * (58) L. Lellouch, M. Luscher, Commun. Math. Phys. 219, 31 (2001), hep-lat/0003023
  * (59) D. Giusti, F. Sanfilippo, S. Simula, Phys. Rev. D 98, 114504 (2018), 1808.00887
  * (60) M.T. Hansen, A. Patella, Phys. Rev. Lett. 123, 172001 (2019), 1904.10010
  * (61) M.T. Hansen, A. Patella, JHEP 10, 029 (2020), 2004.03935
  * (62) A. Patella, Contribution to Asia-Pacific Symposium for Lattice Field Theory (2020), https://conference-indico.kek.jp/event/113/contri
butions/2153/

  * (63) C. Lehner, 36th International Symposium on Lattice Field Theory (Lattice 2018) East Lansing (2018), https://indico.fnal.gov/event/15949/session/8/co
ntribution/29

  * (64) D. Giusti, S. Simula, Phys. Rev. D 102, 054503 (2020), 2003.12086
  * (65) R. Sommer, PoS LATTICE2013, 015 (2014), 1401.3270
  * (66) N. Husung, P. Marquard, R. Sommer, Eur. Phys. J. C 80, 200 (2020), 1912.08498
  * (67) D. Giusti, S. Simula, PoS LATTICE2019, 104 (2019), 1910.03874
  * (68) D. Giusti, V. Lubicz, G. Martinelli, F. Sanfilippo, S. Simula, Phys. Rev. D 99, 114502 (2019), 1901.10462
  * (69) D. Giusti, V. Lubicz, G. Martinelli, F. Sanfilippo, S. Simula, JHEP 10, 157 (2017), 1707.03019
  * (70) D. Hatton, C.T.H. Davies, B. Galloway, J. Koponen, G.P. Lepage, A.T. Lytle (HPQCD), Phys. Rev. D 102, 054511 (2020), 2005.01845
  * (71) V. Gülpers, A. Francis, B. Jäger, H. Meyer, G. von Hippel, H. Wittig, PoS LATTICE2014, 128 (2014), 1411.7592
  * (72) A. Gérardin, H.B. Meyer, A. Nyffeler, Phys. Rev. D 100, 034520 (2019), 1903.09471
  * (73) A. Stathopoulos, J. Laeuchli, K. Orginos (2013), 1302.4018
  * (74) T. Blum, P. Boyle, T. Izubuchi, L. Jin, A. Jüttner, C. Lehner, K. Maltman, M. Marinkovic, A. Portelli, M. Spraggs, Phys. Rev. Lett. 116, 232002 (2016), 1512.09054
  * (75) B. Chakraborty, C. Davies, J. Koponen, G. Lepage, M. Peardon, S. Ryan, Phys. Rev. D 93, 074509 (2016), 1512.03270
  * (76) C. McNeile, Contribution to the Workshop :The hadronic vacuum polarization from lattice QCD at high precision. (2020), https://indico.cern.ch/event/956699/contributions/
4083783/

  * (77) A. Gérardin, Contribution to the Workshop :The hadronic vacuum polarization from lattice QCD at high precision. (2020), https://indico.cern.ch/event/956699/contributions/
4083779/

  * (78) L. Giusti, T. Harris, A. Nada, S. Schaefer, Eur. Phys. J. C 79, 586 (2019), 1903.10447
  * (79) J. Bijnens, J. Harrison, N. Hermansson-Truedsson, T. Janowski, A. Jüttner, A. Portelli, Phys. Rev. D 100, 014508 (2019), 1903.10591
  * (80) S. Borsanyi et al., Science 347, 1452 (2015), 1406.4088
  * (81) A. Patella, PoS LATTICE2016, 020 (2017), 1702.03857
  * (82) M. Hayakawa, S. Uno, Prog. Theor. Phys. 120, 413 (2008), 0804.2044
  * (83) A. Duncan, E. Eichten, H. Thacker, Phys. Rev. Lett. 76, 3894 (1996), hep-lat/9602005
  * (84) P. Boyle, V. Gülpers, J. Harrison, A. Jüttner, A. Portelli, C. Sachrajda, PoS LATTICE2016, 172 (2016), 1612.05962
  * (85) A. Westin, Contribution to Asia-Pacific Symposium for Lattice Field Theory (2020), https://conference-indico.kek.jp/event/113/contri
butions/2185/

  * (86) G. de Divitiis et al., JHEP 04, 124 (2012), 1110.6294
  * (87) G. de Divitiis, R. Frezzotti, V. Lubicz, G. Martinelli, R. Petronzio, G. Rossi, F. Sanfilippo, S. Simula, N. Tantalo (RM123), Phys. Rev. D 87, 114505 (2013), 1303.4896
  * (88) B. Chakraborty et al. (Fermilab Lattice, LATTICE-HPQCD, MILC), Phys. Rev. Lett. 120, 152001 (2018), 1710.11212
  * (89) A. Westin et al. (CSSM/QCDSF/UKQCD), PoS LATTICE2018, 136 (2019), 1902.01518
  * (90) A. Risch, H. Wittig, EPJ Web Conf. 175, 14019 (2018), 1710.06801
  * (91) A. Risch, H. Wittig, PoS LATTICE2018, 059 (2018), 1811.00895
  * (92) A. Risch, H. Wittig, PoS LATTICE2019, 296 (2019), 1911.04230
  * (93) F. Jegerlehner, _The Anomalous Magnetic Moment of the Muon_ , Vol. 274 (Springer, Cham, 2017)
  * (94) M. Cè, T.S. José, A. Gérardin, H.B. Meyer, K. Miura, K. Ottnad, A. Risch, J. Wilhelm, H. Wittig, PoS LATTICE2019, 010 (2019), 1910.09525
  * (95) L. Morel, Z. Yao, P. Cladé, S. Guellati-Khélifa, Nature 588, 61 (2020)
  * (96) M. Tanabashi et al. (Particle Data Group), Phys. Rev. D 98, 030001 (2018)
  * (97) M. Baak, J. Cúth, J. Haller, A. Hoecker, R. Kogler, K. Mönig, M. Schott, J. Stelzer (Gfitter Group), Eur. Phys. J. C 74, 3046 (2014), 1407.3792
  * (98) F. Jegerlehner, Nucl. Phys. B Proc. Suppl. 181-182, 135 (2008), 0807.4206
  * (99) D.B. Renner, X. Feng, K. Jansen, M. Petschlies, PoS LATTICE2011, 022 (2012), 1206.3113
  * (100) X. Feng, G. Hotzel, K. Jansen, M. Petschlies, D.B. Renner, PoS LATTICE2012, 174 (2012), 1211.0828
  * (101) F. Burger, K. Jansen, M. Petschlies, G. Pientka, JHEP 11, 215 (2015), 1505.03283
  * (102) A. Francis, G. Herdoíza, H. Horch, B. Jäger, H.B. Meyer, H. Wittig, PoS LATTICE2014, 163 (2014), 1412.6934
  * (103) A. Francis, V. Gülpers, G. Herdoíza, H. Horch, B. Jäger, H.B. Meyer, H. Wittig, PoS LATTICE2015, 110 (2015), 1511.04751
  * (104) F. Jegerlehner, Z. Phys. C 32, 195 (1986)
  * (105) V. Gülpers, H. Meyer, G. von Hippel, H. Wittig, PoS LATTICE2015, 263 (2016)
  * (106) M. Cè, A. Gérardin, K. Ottnad, H.B. Meyer, PoS LATTICE2018, 137 (2018), 1811.08669
  * (107) D. Hanneke, S. Hoogerheide, G. Gabrielse, Phys. Rev. A 83, 052122 (2011), 1009.4831
  * (108) R.H. Parker, C. Yu, W. Zhong, B. Estey, H. Müller, Science 360, 191 (2018), 1812.04130
  * (109) F. Burger, K. Jansen, M. Petschlies, G. Pientka, Eur. Phys. J. C 76, 464 (2016), 1501.05110
  * (110) B. Chakraborty, C. Davies, P. de Oliviera, J. Koponen, G. Lepage, R. Van de Water, Phys. Rev. D 96, 034516 (2017), 1601.03071
  * (111) D. Hanneke, S. Fogwell, G. Gabrielse, Phys. Rev. Lett. 100, 120801 (2008), 0801.1134
  * (112) S. Borsanyi, Z. Fodor, T. Kawanai, S. Krieg, L. Lellouch, R. Malak, K. Miura, K. Szabo, C. Torrero, B. Toth, Phys. Rev. D 96, 074507 (2017), 1612.02364
  * (113) T. Blum, S. Chowdhury, M. Hayakawa, T. Izubuchi, Phys. Rev. Lett. 114, 012001 (2015), 1407.2923
  * (114) T. Blum, N. Christ, M. Hayakawa, T. Izubuchi, L. Jin, C. Lehner, Phys. Rev. D 93, 014503 (2016), 1510.07100
  * (115) T. Blum, N. Christ, M. Hayakawa, T. Izubuchi, L. Jin, C. Jung, C. Lehner, Phys. Rev. D 96, 034515 (2017), 1705.01067
  * (116) J. Bijnens, EPJ Web Conf. 118, 01002 (2016), 1510.05796
  * (117) N. Asmussen, J. Green, H.B. Meyer, A. Nyffeler, PoS LATTICE2016, 164 (2016), 1609.08454
  * (118) N. Asmussen, A. Gérardin, H.B. Meyer, A. Nyffeler, EPJ Web Conf. 175, 06023 (2018), 1711.02466
  * (119) N. Asmussen, A. Gerardin, J. Green, O. Gryniuk, G. von Hippel, H.B. Meyer, A. Nyffeler, V. Pascalutsa, H. Wittig, EPJ Web Conf. 179, 01017 (2018), 1801.04238
  * (120) E.H. Chao, A. Gérardin, J.R. Green, R.J. Hudspith, H.B. Meyer, Eur. Phys. J. C 80, 869 (2020), 2006.16224
  * (121) H. Meyer, Contribution to the Second Plenary Workshop of the Muon g-2 Theory Initiative (2018), https://indico.him.uni-mainz.de/event/11/session/
11/contribution/91/

  * (122) N. Asmussen, E.H. Chao, A. Gérardin, J.R. Green, R.J. Hudspith, H.B. Meyer, A. Nyffeler, PoS LATTICE2019, 195 (2019), 1911.05573
  * (123) T. Blum, N. Christ, M. Hayakawa, T. Izubuchi, L. Jin, C. Jung, C. Lehner, Phys. Rev. Lett. 118, 022005 (2017), 1610.04603
  * (124) L. Jin, PoS LATTICE2018, 012 (2018)
  * (125) L. Jin, Contribution to the Third Plenary Workshop of the Muon g-2 Theory Initiative (2019), https://indico.fnal.gov/event/21626/contributions
/63821/

  * (126) S. Laporta, E. Remiddi, Phys. Lett. B 301, 440 (1993)
  * (127) P. Masjuan, P. Sanchez-Puertas, Phys. Rev. D 95, 054026 (2017), 1701.05829
  * (128) G. Colangelo, M. Hoferichter, M. Procura, P. Stoffer, JHEP 04, 161 (2017), 1702.07347
  * (129) M. Hoferichter, B.L. Hoid, B. Kubis, S. Leupold, S.P. Schneider, JHEP 10, 141 (2018), 1808.04823
  * (130) J. Bijnens, N. Hermansson-Truedsson, A. Rodríguez-Sánchez, Phys. Lett. B 798, 134994 (2019), 1908.03331
  * (131) G. Colangelo, F. Hagelstein, M. Hoferichter, L. Laub, P. Stoffer, JHEP 03, 101 (2020), 1910.13432
  * (132) I. Danilkin, M. Vanderhaeghen, Phys. Rev. D 95, 014019 (2017), 1611.04646
  * (133) M. Knecht, S. Narison, A. Rabemananjara, D. Rabetiarivony, Phys. Lett. B 787, 111 (2018), 1808.03848
  * (134) G. Eichmann, C.S. Fischer, R. Williams, Phys. Rev. D 101, 054015 (2020), 1910.06795
  * (135) P. Roig, P. Sanchez-Puertas, Phys. Rev. D 101, 074019 (2020), 1910.02881
  * (136) N. Asmussen, Contribution to the Second Plenary Workshop of the Muon g-2 Theory Initiative (2018), https://indico.him.uni-mainz.de/event/11/session
/0/contribution/3/

  * (137) E.H. Chao, A. Gérardin, J.R. Green, R.J. Hudspith, H.B. Meyer, Contribution to the Asia-Pacific Symposium for Lattice Field Theory 2020 conference (2020), https://conference-indico.kek.jp/event/113/contri
butions/2082/

  * (138) F. Jegerlehner, A. Nyffeler, Phys. Rept. 477, 1 (2009), 0902.3360
  * (139) A. Gérardin, H.B. Meyer, A. Nyffeler, Phys. Rev. D 94, 074507 (2016), 1607.08174
  * (140) X. Feng, S. Aoki, H. Fukaya, S. Hashimoto, T. Kaneko, J.i. Noaki, E. Shintani, Phys. Rev. Lett. 109, 182001 (2012), 1206.1375
  * (141) S. Burri, Contribution to Asia-Pacific Symposium for Lattice Field Theory (2020), https://conference-indico.kek.jp/event/113/contri
butions/2077/

  * (142) X.d. Ji, C.w. Jung, Phys. Rev. Lett. 86, 208 (2001), hep-lat/0101014
  * (143) X.d. Ji, C.w. Jung, Phys. Rev. D 64, 034506 (2001), hep-lat/0103007
  * (144) S.D. Cohen, H.W. Lin, J. Dudek, R.G. Edwards, PoS LATTICE2008, 159 (2008), 0810.5550
  * (145) H.W. Lin, S.D. Cohen, PoS ConfinementX, 113 (2012), 1302.0874
  * (146) C. Bourrely, I. Caprini, L. Lellouch, Phys. Rev. D 79, 013008 (2009), [Erratum: Phys.Rev.D 82, 099902 (2010)], 0807.2722
  * (147) G. Lepage, S.J. Brodsky, Phys. Lett. B 87, 359 (1979)
  * (148) G. Lepage, S.J. Brodsky, Phys. Rev. D 22, 2157 (1980)
  * (149) S.J. Brodsky, G. Lepage, Phys. Rev. D 24, 1808 (1981)
  * (150) V. Nesterenko, A. Radyushkin, Sov. J. Nucl. Phys. 38, 284 (1983)
  * (151) V. Novikov, M.A. Shifman, A. Vainshtein, M. Voloshin, V.I. Zakharov, Nucl. Phys. B 237, 525 (1984)
  * (152) M. Hoferichter, B. Kubis, S. Leupold, F. Niecknig, S.P. Schneider, Eur. Phys. J. C 74, 3180 (2014), 1410.4691
  * (153) M. Hoferichter, B.L. Hoid, B. Kubis, S. Leupold, S.P. Schneider, Phys. Rev. Lett. 121, 112002 (2018), 1805.01471
  * (154) M. Knecht, A. Nyffeler, Phys. Rev. D 65, 073034 (2002), hep-ph/0111058
  * (155) I. Larin et al. (PrimEx), Phys. Rev. Lett. 106, 162303 (2011), 1009.1681
  * (156) P. Masjuan, Phys. Rev. D 86, 094021 (2012), 1206.2549
  * (157) P. Sanchez-Puertas, P. Masjuan, Mod. Phys. Lett. A 31, 1630034 (2016), 1606.02704
  * (158) S.L. Adler, Phys. Rev. 177, 2426 (1969)
  * (159) J. Bell, R. Jackiw, Nuovo Cim. A 60, 47 (1969)
  * (160) J. Donoghue, B.R. Holstein, Y. Lin, Phys. Rev. Lett. 55, 2766 (1985), [Erratum: Phys.Rev.Lett. 61, 1527–1527 (1988)]
  * (161) J. Bijnens, A. Bramon, F. Cornet, Phys. Rev. Lett. 61, 1453 (1988)
  * (162) K. Kampf, B. Moussallam, Phys. Rev. D 79, 076005 (2009), 0901.4688
  * (163) I. Larin, Y. Zhang, A. Gasparian, L. Gan, R. Miskimen, M. Khandaker, D. Dale, S. Danagoulian, E. Pasyuk, H. Gao et al., Science 368, 506 (2020)
  * (164) J. Green, O. Gryniuk, G. von Hippel, H.B. Meyer, V. Pascalutsa, Phys. Rev. Lett. 115, 222003 (2015), 1507.01577
  * (165) A. Gérardin, J. Green, O. Gryniuk, G. von Hippel, H.B. Meyer, V. Pascalutsa, H. Wittig, Phys. Rev. D 98, 074501 (2018), 1712.00421
  * (166) V. Pascalutsa, V. Pauk, M. Vanderhaeghen, Phys. Rev. D 85, 116001 (2012), 1204.0740
  * (167) J. Bijnens, J. Relefors, JHEP 09, 113 (2016), 1608.01454
  * (168) J. Bijnens, EPJ Web Conf. 179, 01001 (2018), 1712.09787


# OmDet: Language-Aware Object Detection with Large-scale Vision-Language
Multi-dataset Pre-training

Tiancheng Zhao1, Peng Liu2, Xiaopeng Lu2, Kyusong Lee1
Om Research Lab, Binjiang Institute of Zhejiang University1
Linker Technology Research Co. Ltd2
{tianchez<EMAIL_ADDRESS>
{liu_peng<EMAIL_ADDRESS>
###### Abstract

Advancing object detection to open-vocabulary and few-shot transfer has long
been a challenge for computer vision research. This work explores a continual
learning approach that enables a detector to expand its zero/few-shot
capabilities via multi-dataset vision-language pre-training. Using natural
language as knowledge representation, we explore methods to accumulate ”visual
vocabulary” from different training datasets and unify the task as a language-
conditioned detection framework. Specifically, we propose a novel language-
aware detector OmDet and a novel training mechanism. The proposed multimodal
detection network can resolve the technical challenges in multi-dataset joint
training and it can generalize to arbitrary number of training datasets
without the requirements for manual label taxonomy merging. Experiment results
on COCO, Pascal VOC, and Wider Face/Pedestrian confirmed the efficacy by
achieving on par or higher scores in joint training compared to training
separately. Moreover, we pre-train on more than 20 million images with 4
million unique object vocabulary, and the resulting model is evaluated on 35
downstream tasks of ODinW [13]. Results show that OmDet is able to achieve the
state-of-the-art fine-tuned performance on ODinW. And analysis shows that by
scaling up the proposed pre-training method, OmDet continues to improve its
zero/few-shot tuning performance, suggesting a promising way for further
scaling.

##  1 Introduction

Object detection (OD) is one of the monumental tasks in computer vision (CV).
Classical OD research has been focusing on improving the detector network to
achieve higher accuracy with lower latency [27, 26, 18, 40] with fixed output
label set, e.g., 80 classes in COCO [17]. Recently, an emerging line of
research based on vision-language pretraining (VLP) has been striving to
upgrade OD models to solve the more challenging open-vocabulary setting, where
the detector can generalize to new visual concepts with zero/few-shot adaption
[8, 12, 14, 24]. Some of the VLP-based methods exploit large-scale visual
grounding dataset for pretraining [12] and some of the work combines class-
agnostic region proposal network (RPN) with a zero-shot image-text classifier
respectively for localization and classification [37].

Unlike previous VLP-based methods that utilize one large vision-language
corpus for pretraining, this paper explores a continual learning approach,
i.e., can a detector incrementally learn from many OD datasets with increasing
total visual vocabulary, and eventually achieve the open-vocabulary detection
capabilities?. This approach is appealing for several reasons: (1) it opens
the possibility of life-long learning since one can improve a detector’s
zero/few-shot performance by feeding it with new datasets. (2) it is cost-
effective since creating many small domain-specific datasets is much cheaper
than creating a single large-vocabulary large dataset [9].

On the other hand, joint training from multiple OD datasets with different
labels face two key technical challenges: (1) Taxonomy conflict: each OD
dataset is annotated with its labels and a classic detector formulate the type
classification problem as a fixed Softmax layer [27]. Such design forbids the
possibility of learning from different label sets or dynamically adapt to new
tasks. (2) Fore/background inconsistency: since the label set is different,
then an object proposal may be considered as foreground in dataset A, while it
is considered as background in dataset B. For example, type ”cat” is in
dataset A, but not in dataset B. Our study shows that this issue greatly hurts
the multi-dataset performance of classic detectors since the RPN head is
confused by the conflicting ground truth labels.

To address the above challenges, we propose a novel VLP-based object detection
framework: OmDet (Omni-dataset Detection). We first formulate language-aware
object detection which is a generalized version of OD task, i.e. given an
image and a task (a set of object names), detecting the object instances that
appeared in the task. Second a novel deep vision-language fusion network is
introduced to enable both localization and classification to be language-
aware, i.e. the detector can dynamically modify its object localization and
classification behavior based on the current tasks that is conditioned on.
Lastly, a new multi-dataset training algorithm is developed to enable OmDet to
learn from arbitrary number of OD datasets regardless of their label set, and
we scale the pre-training to a large number of datasets with total vocabulary
size large than 4 million unique text labels.

The proposed method is first validated in a small-scale study to confirm its
multi-dataset learning ability. Four OD datasets are used including COCO [17],
Pascal VOC [5], Wider Face [34] and Wider Pedestrian [22]. Results show that
OmDet not only is able to learn from all datasets without suffering from label
conflicts, but achieve stronger performance than single dataset detectors due
to its share of knowledge among tasks.

Then a larger scale of study is conducted to scale up OmDet to very large
vocabulary pretraining. We pre-train using a mixture of OD datasets with 20
million images and 4 million unique text labels that include both human
annotations and pseudo labels. The resulting model is evaluated on the
recently proposed ODinW dataset [13] that cover 35 different OD tasks in
various domains. Comprehensive evaluation suggests that the proposed continual
learning paradigm is able to achieve a new state-of-the-art performance
compared to GLIP [14] that is pre-trained on larger datasets. Also, we show
that accumulating multiple datasets to expand to large vocabulary OD learning
is an effective method to boost OmDet’s zero/few shot ability as well as
parameter-efficient training performance (e.g. prompt tuning).

In short, the contribution of this work are four folds:

  * •

We present OmDet, a novel language-aware OD architecture and training
framework that is able to effectively learn from arbitrary number of OD
datasets.

  * •

Experiments that confirm the effectiveness of the proposed multi-dataset
training by solving the label difference and fore/background inconsistency
challenges.

  * •

Experiments that show OmDet’s state-of-the-art performance on the ODinW
dataset by simultaneously master 35 different downstream tasks.

  * •

Study results that show by scaling up visual concept vocabulary size via
multi-dataset training, one can effective improve zero/few-shot learning and
parameter-efficient fine-tuning.

##  2 Related Work

Figure 1: Overview of the proposed OmDet Detector.

###  2.1 Object Detection

Objection detection, one of the predominant tasks in computer vision, aims to
detect bounding boxes and classes of object instances. It has significantly
evolved through the contributions of massive research in recent years.
R-CNNawe[7] formulates the two-stage detectors paradigm, which is composed of
a region proposal detector and a region-wise classifier. Consequent R-CNN
series such as Fast R-CNN[6] and Faster R-CNN[27] make enhancements on the
network pipeline to improve performance. While one-stage detectors like
SSD[18], YOLO[26], and RetinaNet[16] are also in a competitive position by
skipping the region proposal stage to simplify and speed up the framework.
Recently, DETR[1] has proposed a transformer-based end-to-end object detection
framework by framing the object detection task to a set of predictions.
Follow-up DETR variants have proposed this framework in different directions.
However, objection detection is often formulated as a closed-set problem with
fixed and predefined classes and is diverse from the real-world setting. To
conquer the closed-set limitation, more realistic scenarios such as Multi-
Dataset Object Detection (MDOD) and Open-Vocabulary Object Detection (OVOD)
have attracted lots of attention.

###  2.2 Multi-Dataset Object Detection

MDOD focuses on increasing detectable object classes by training a single
detector using multiple datasets. Traditional closed-set object detection
demands training detectors on datasets with fully-annotations, and adding a
new dataset means costly extra human annotations. Research on MDOD attempts to
bypass the closed-set limitation in a realistic way, where a single detector
is able to incrementally add object classes by adding new datasets with new
classes. [35] proposes an MDOD framework with a preprocessed hybrid dataset
and a dataset-aware focal loss. [36] designs a conflict-free loss to avoid the
ambiguity between positive and negative samples. Detection Hub[23] unifies
multiple datasets with a query-base object detector adapting queries on
language embedding.

###  2.3 Open-Vocabulary Object Detection

OVOD, a more ambitious goal beyond the closed-set problem, refers to the
capability of only training on annotated datasets and generalizing to unseen
novel classes. Recently, OVOD has made such progress with the utilization of a
multi-modal vision-language pre-train model. RegionCLIP[37] generates pseudo-
labels for region-text pairs from caption datasets to perform regional vision-
language pre-training and transfer to OVOD. VILD[8] proposed a two-stage open-
vocabulary detector, which distil embeddings from teacher model CLIP[25] or
ALIGN[11]. With inspiration from CoOp[38], DetPro[4] introduces a technique to
learn continuous detection prompt which improves the performance of VILD. OWL-
ViT[24] uses the pre-trained image-text model as the base, then transfers it
to the object detection domain by adding downstream detection heads and fine-
tuning on OD datasets.

Unlike previous multi-dataset object detections, the proposed method is not
required to have any extra human cost and naturally learning objects with the
fused task embeddings from multiple datasets. Additionally the proposed model
has OVOD capabilities by simply expanding the visual concept vocabulary size
with more datasets and pseudo labeling from image-caption datasets.

##  3 Proposed Method

###  3.1 Language-Aware Object Detection

Before getting into the details of the proposed system, we first define the
problem setting. OmDet is designed for task-conditioned detection. Let $V$ be
a large vocabulary of objects types that OmDet can potentially detect. A task
$T=\\{w_{1},w_{2},...w_{k}\\}$is a set of $k$ object types that the model
should detect in this forward path, where $w\in V$. Note that the size of $T$
can be dynamic ranging from 1 to $K$, where $K$ is the maximum supported
number of object types in a single inference run. Then given an input image
$x$ and a task $T$, the model is expected to detect all of objects that
appeared in $T$ from $x$. Since $T$ is not fixed, an ideal model can
dynamically adapt its detection targets conditioned on the task.

###  3.2 Model Architecture

Following the above design principle, OmDet is introduced, a task-conditioned
detection network that can learn from infinite combinations of tasks. It is
composed of a vision backbone, a task encoder, a label encoder, and a
multimodal detection network. The overall structure is illustrated in Fig1.
The following will describe each component in details.

Vision Backbone Starting from the initial image $x_{img}\in R^{3\times
H_{0}\times W_{0}}$ (with 3 color channels), let the vision encoder $f_{v}$ be
a conventional CNN backbone or Vision Transformer backbone (e.g. Swin
Transformer) generates a lower-resolution visual feature map $f\in R^{C\times
H\times W}$ at each output layer. Then Feature Pyramid Network (FPN) [15] is
used to aggregate information from top to bottom and output a set of visual
feature maps $\\{P2,P3,P4,P5\\}$.

Task Encoder and Label Encoder The task set $T=\\{w_{1},w_{2},...w_{k}\\}\in
R^{k\times V}$ is set of natural language words. Then a task encoder $f_{t}$
or a label encoder $f_{l}$ is a transformer model that encode the task set $T$
without order information, and output a set of contextual word embeddings,
i.e. $\\{t_{1},t_{2},...t_{k}\\}=f_{t}(w_{1},w_{2},...w_{k})\in R^{k\times d}$
and $\\{l_{1},l_{2},...l_{k}\\}=f_{l}(w_{1},w_{2},...w_{k})\in R^{k\times d}$,
where $d$ is the contextual word embedding dimension size.

Multimodal Detection Network The Multimodal Detection Network (MDN) is a core
component of OmDet. We deploy early fusion to combine information from the
image and current task early on, in order to achieve strong performance. We
are inspired by the Sparse-RCNN [31] network design, and developed an
iterative query-based fusion mechanism.

Let $Q\in R^{N\times d}$ be a fixed small set of learnable proposal features.
It is a set of high-dimensional (e.g., $d=256$) latent features that capture
the rich information of a potential instance, by combining information from
the vision backbone and contextual task embedding from the task encoder. Also,
let $B\in R^{N\times 4}$ be a set of learnable proposal boxes that is one-to-
one assigned to each proposal feature. Then given the FPN output and
task/label encoder output, the initial MDN operates as the following:

$\displaystyle v_{0}$
$\displaystyle=\text{RoiPooler}(\\{P2,P3,P4,P5\\},B_{0})$ (1)
$\displaystyle[Q_{1},T_{1}]$ $\displaystyle=\text{MHSA}([Q_{0},T_{0}])$ (2)
$\displaystyle Q_{2}$ $\displaystyle=\text{DynamicCov}(Q_{1},v_{0})$ (3)
$\displaystyle B_{1}$ $\displaystyle=\text{RegHead}(Q_{2})$ (4) $\displaystyle
C_{1}$ $\displaystyle=\gamma cosine(\text{ClsHead}(Q_{2}),L)$ (5)

Note that MDN can be stacked to iterative refine its output the same as
Sparse-RCNN, with the key difference that $T$ is fused with the proposal
feature before Dynamic Convolution layer and also $T$ is also iterativly
updated at each run of MDN block. This enables the network to learn to adjust
the task embedding and the proposal embedding jointly and adapt both object
localization and object classification head conditioned on the given task.

###  3.3 Model Training

Set Prediction Loss Given the above network, OmDet also uses set prediction
loss [1] on the fixed-size set of predictions of classification and box
coordinates. Set-based loss produces an optimal bipartite matching between
predictions and ground truth objects using Hungarian algorithm. The matching
cost is defined as follows:

$L=\lambda_{cls}\cdot L_{cls}+\lambda_{L_{1}}\cdot
L_{L_{1}}+\lambda_{giou}\cdot L_{giou}$ (7)

Here $L_{cls}$ is focal loss [16] of predicted classifications and ground
truth category labels, $L_{L_{1}}$ and $L_{giou}$ are L1 loss and generalized
IoU loss [1] between normalized center coordinates and height and width of
predicted boxes and ground truth box, respectively. $\lambda_{cls}$,
$\lambda_{L_{1}}$ and $\lambda_{giou}$ are coefficients of each component. The
training loss is the same as the matching cost except that only performed on
matched pairs. The final loss is the sum of all pairs normalized by the number
of objects inside the training batch.

Task-Sampling Strategy in order to simulate the extreme multi-tasking setting
at the training time and also enforce the model to condition its output on a
given task, a novel task sampling strategy is used during training.

  1. 1.

Let the max size of of a given task be $K$, for an image $x$ from a dataset
$d$ in the mini-batch, we first sample $k\in[1,K]$ with a uniform
distribution.

  2. 2.

Let the number of unique object types in $x$ be $m$, if $m>k$, then only a
random subset of $k$ object types are kept and the extra annotations are
removed for this mini batch. If $m<k$, then additional negative object types
are randomly selected from the vocabulary $V$ of dataset $d$. If the
vocabulary size of data $d$ is less than $K$, then the reminder of missing
negatives are filled with masking $0$.

  3. 3.

The model is trained with the above sampled task and ground truth annotations.

With the above method, each image in every mini-batch will have a different
set of tasks to learn from. When we learn from a large-vocabulary object
detection dataset, e.g., LVIS, which contains 1200 unique object types, the
unique combination of task size $k$ is $C_{k}(n)=\frac{1200!}{(1200-k)!k!}$.
If $k=20$, then it produces 1.34E43 possibilities, a quite large number.
Experiments show that the proposed training strategy serve the purpose well,
and yield models that perform task-conditioned object detection.

###  3.4 Implementation Details

We implement OmDet with the following settings. For text embeddings, we did
not use prompt template as used in study [8], i.e. encoding object names in a
template a photo of {}. This is because preliminary study show no major
difference between using versus not using the prompt template. Furthermore,
preliminary study also suggest there is no significant differences between
using single-modal language models, e.g. BERT [3] and RoBERTa [19], versus
multimodal-language models e.g. CLIP. We suspect this is because object
detection does not involve complex language understanding. As a result,
CLIP-B/16 text encoder [25] is used throughout the study.

For task and label encoders, two separate Transformer [32] are used to further
encode text embeddings to create a contextual set embeddings for each element
in the task. Study shows that the set encoding is able to improve OmDet’s
performance.

For visual backbones, both Swin Transformers [20] and ConvNeXt [21] are used
in the experiments. A standard FPN [15] is used to extract four-level feature
map form the visual encoders. Both backbones are pre-trained on ImageNet 21K
data [28].

Lastly, the MDN network utilizes MHSA to fuse the task and visual features. We
used ROIAlignV2 [10] as the ROI Pooler to extract region features from the
backbone. 6 sequential MDN blocks are cascaded to create the final bounding
boxes and classification prediction.

##  4 Joint Training on Multiple Datasets

Figure 2: Inference results of OmDet-Shallow and OmDet on COCO validation

We first verify that the proposed OmDet can effectively learn from multiple
object detection datasets without suffering from the task conflict challenge.

| COCO | PASCAL VOC | WIDER FACE | WIDER Pedestrain
---|---|---|---|---
| AP | AP50 | AP75 | AP | AP50 | AP75 | AP | AP50 | AP75 | AP | AP50 | AP75
Sparse R-CNN [31] | 41.76 | 61.07 | 44.82 | 44.35 | 64.8 | 48.19 | 22.69 | 42.71 | 21.73 | 35.03 | 56.59 | 38.15
OmDet-Single | 43.03 | 62.44 | 46.69 | 45.53 | 65.85 | 49.39 | 23.39 | 44.82 | 21.93 | 33.93 | 55.16 | 37.01
OmDet-Shallow | 35.32 | 50.64 | 38.16 | 55.41 | 73.94 | 60.14 | 22.94 | 43.48 | 21.77 | 34.97 | 55.09 | 38.30
OmDet | 42.15 | 61.39 | 54.62 | 60.85 | 82.13 | 66.59 | 30.72 | 57.18 | 30.00 | 46.78 | 73.99 | 50.90

Table 1: MDOD training results on four datasets. OmDet is able to resolve the
task conflict issues in MDOD and achieve higher performance compared to single
dataset models.

###  4.1 Experiment Setup

Datasets: For MDOD, we follow the experimental setting from [35] and choose
COCO[17], Pascal VOC[5], WIDER FACE[34] and WIDER Pedestrian[22] as joint-
training datasets.

| # of class | Example
---|---|---
COCO 2017 | 80 | person, car, etc
Pascal 2007 | 20 | person, car, etc.
WIDER FACE | 1 | face
WIDER Ped | 1 | person
Table 2: Categories of the four OD datasets.

Their category information is included in Table 2. Note that COCO is the
larger data with 118K images while the other 3 datasets are almost 10 times
smaller. Also the COCO dataset has a diverse set of categories that cover the
classes in Pascal VOC and WIDER Pedestrian. WIDER Face is the only dataset
that has ”face” class. Therefore, these four datasets serve as a great testing
bed for MDOD study.

Compared Baselines: Three baselines are served as baselines and ablation
study. First, we include Sparse R-CNN [31], an end-to-end detector, as the
baseline due to its strong performance and similarity to OmDet in terms of
model structure. Then we create OmDet-Single, OmDet-Shallow, and OmDet for the
ablation study.

  * •

OmDet-Single: To compare the performance on single datasets with Sparse R-CNN,
we train OmDet on the four datasets separately with the same training setting.
Since these OmDet models are only trained with a single dataset, so it cannot
benefit from the proposed multi-dataset training.

  * •

OmDet-Shallow: OmDet utilizes a deep fusion mechanism to fuse task and image
information for both localization and classification. For the ablation study,
we also train an OmDet-Shallow model by removing the task encoder and MDN,
which degenerates the localization network to Sparse R-CNN, and only utilize
the language feature for the final label classification. This is similar to
previous work, e.g., Detic[39].

Training: We use Image Net[2] pre-trained Swin Transformer Tiny[20] as the
visual backbone and use CLIP ViT-B/16 as language encoder for OmDet. The same
Swin Transformer is used as the backbone for Sparse R-CNN. All models are
trained with 12 epochs, learning decays at 70$\%$ and 90$\%$ of total
iteration steps by 0.1. The initial learning rate is set to 5e-5 for OmDet and
2.5e-5 for Sparse R-CNN.

Models | Zero-shot | Full-model FT | Head-only FT | Prompt FT
---|---|---|---|---
GLIP-Tiny [13] | 19.7 | 63.2 | - | 54.4
OmDet-C | 9.8 | 61.7 | 54.7 (-11.3%) | 21.0 (-65.9%)
OmDet-CO | 13.5 | 63.2 | 56.8 (-10.1%) | 24.8 (-60.7%)
OmDet-COL | 12.4 | 63.2 | 57.6 (-8.8%) | 25.5 (-59.6%)
OmDet-COLP | 13.5 | 63.0 | 58.5 (-7.1%) | 29.3 (-53.4%)
OmDet | 16.0 | 63.7 |  59.8 (-6.1%) |  34.7 (-45.5%)
Table 3: Average AP of zero-shot, full-model, head-only and prompt fine-tuning
on 35 downstream tasks in ODinW. The gray text shows the performance drop of
parameter-efficient tuning compared to full-model tuning.

###  4.2 Result and Discussion

OmDet vs. Sparse R-CNN on single Dataset: First we demonstrate the validity of
our framework on classic OD tasks. We conduct experiments on OmDet and Sparse
R-CNN using only a single dataset to sanity check our model’s detection
performance. As shown in Table 1, OmDet-Single gets higher AP scores on COCO,
PASCAL VOC and WIDER FACE than Sparse R-CNN under the same training setting,
and is only about 1 point lower on WIDER Pedestrian. These results prove our
model with the novel language-aware OD architecture still maintains the same
or better performance on single dataset OD, but since it is only trained with
a small set of vocabulary, it does not have open-vocabulary or few-shot
capability.

OmDet vs. OmDet-Single: Then we look into results of joint-training with MDOD.
The only difference between OmDet and OmDet-Single is that OmDet is joint-
trained on all four datasets at once by utilizing the language-aware object
detection architecture at section 3.2.

Results from Table 1 shows that the AP scores of OmDet are significantly
higher than OmDet-Single on PASCAL VOC (+15.52 AP), WIDER FACE (+7.33 AP) and
WIDER Pedestrian (+12.85 AP). Moreover, OmDet shows better performances than
its baseline model Sparse R-CNN on all datasets. These results confirm that
OmDet possesses the capability of multi-dataset training by solving taxonomy
conflicts and fore/background inconsistency. Moreover, the knowledge sharing
in joint training improves the detection performance of each dataset,
especially for the ones with less training samples.

OmDet vs. OmDet-Shallow: Lastly an ablation study is carried out to verify the
contribution of the proposed MDN block. OmDet’s fusion mechanism is deep since
the task embedding is combined with visual features early on and influences
both localization and classification head. On the other hand, OmDet-Shallow’s
fusion mechanism is shallow, i.e., it only utilizes the label embedding in the
final layer of object classification.

Results in Table 1 shows that OmDet performs stronger and OmDet-Shallow only
partially solve the MDOD challenges. OmDet-Shallow achieves good performance
on PASCAL VOC, WIDER FACE, and WIDER Pedestrian compared with OmDet-Single.
This is reasonable since OmDet-Shallow resolves the taxonomy conflict
challenge and enables semantic sharing among object label embeddings, similar
to Detic [39].

However, OmDet-Shallow fails on COCO with a low AP score, since it cannot
resolve the fore/background inconsistency challenge. Since COCO has 80
categories, which is much larger than the other three datasets, many of its
objects are considered as background in the other three datasets. Therefore,
we suspect the low is AP is caused by incorrectly detecting COOC objects as
background. We visualize outputs of OmDet-Shallow and OmDet on COCO images in
Figure 2, which confirms our hypothesis that OmDet-Shallow detects many
objects that are not in Pascal VOC and Wider Face/Pedestrian as background.
More examples can be found that although OmDet-Shallow correctly detects all
pedestrians in the last images due to overlap with WIDER Pedestrian, it misses
object ”Skis”. Unlike OmDet-Shallow, OmDet has benefited from deep fusion and
task-conditioned object detection, and detects all the images correctly.

##  5 Pre-training and Transfer to ODinW

###  5.1 Experiment Setup

Large-scale Pre-training: COCO [17], Object365[29], LVIS v1[9], PhraseCut[33],
and Google Conceptual Captions (GCC)[30] are used for large-scale pre-
training. Object365 is an OD dataset with 365 categories, about 2 million
images and more than 30 million bounding boxes. LVIS is a long-tail OD dataset
with 1203 categories with federated human annotations on 100K images.
PhraseCut, as a language-based image segmentation dataset, contains 77,262
images and 345,486 phrase-region pairs, we use it as a object detection
dataset to increase pre-train diversity. Unlike other datasets, GCC is an
image caption dataset with about 15 million images with captions. Because the
GCC does not have bounding box annotations, we utilize the phrase grounding
ability of GLIP[14] to generate pseudo labels from it.

Downstream Tasks: ODinW is selected as the test data from ELEVATER
benchmark[13] which is a new OD benchmark that consists of 35 diverse real-
world tasks (Table 5). We select ODinW as the source of downstream tasks
because of its diversity in terms of domain, training data size, number of
categories and etc. Also, many of the 35 tasks have very limited (less than
100) training images, which makes it a extremely difficult task for standard
detectors without any pre-training. We use the official train and test split
for training and evaluation.

Training: For OmDet models, the initial learning rate is 5e-5 and it decays at
70$\%$ and 90$\%$ of total iteration steps by 0.1. ConvNeXt Tiny backbone and
6-layer detection head is used. The batch size is 32 and the maximum number of
detections per image is 300 and K is set to 80. All of the proposed models are
pre-trained for 36 epochs using 16 NVIDIA A100 GPUs and then fine-tuned on the
downstream data. All of the pre-training and fine-tuning experiments are
conducted with the parameters of CLIP text encoder frozen.

Compared Models:

  1. 1.

GLIP-Tiny: GLIP[14] is the state-of-the-art model used in ODinW dataset that
is pre-trained on a large set of visual grounding and OD data.

  2. 2.

OmDet-C/CO/COL/COLP: we incrementally increase the number of pre-train
datasets, including 4 intermediate varaitaions. They are C (COCO), CO (COCO +
Object 365), COL (COCO + Object 365 + LVIS) and COLP (COCO + Object 365 + LVIS
+ PhraseCut).

  3. 3.

OmDet: OmDet that is pre-trained on all of the pre-training data.

The purpose of this arrangement is to evaluate the relation between the number
of visual concept included in pre-training versus a model’s downstream
performance under different fine-tuning settings.

Figure 3: Vocabulary size used in pre-training vs. the AP score of fine-tuning on ODinW with head-only and prompt tuning.. X-axis is in log-scale. Models | Few-Shot | Medium-Shot | Big-Shot
---|---|---|---
OmDet-C | 49.48 | 57.09 | 70.16
OmDet-CO | 54.37 | 58.89 | 70.98
OmDet-COL | 55.07 | 57.99 | 71.22
OmDet-COLP | 53.44 | 58.05 | 70.94
OmDet | 55.89 | 59.23 | 70.54
Table 4: Average AP of full-model fine-tuning on 35 downstream tasks in ODinW
for Few-shot, Medium-Shot and Big-Shot tasks. Figure 4: Illustration of
language-aware OD, where a single model can generalize (without fine-tuning)
to any input tasks on the fly in the form of natural language.

###  5.2 Results and Discussion

Overall, OmDet achieves the best detection performance compared to the other 4
variations (C/CO/COL/COLP) based on Table 3. Also, OmDet outperforms GLIP-Tiny
[13] under full-model fine-tuning, which is pre-trained on much larger dataset
with a tunable text encoder. We then analyze the results from two aspects: (1)
zero/few-shot performance and (2) parameter-efficient fine tuning.

Zero/Few-Shot Object Detection As shown in Table 3, adding more pre-train
datasets yields significant improvement in zero-shot settings. Specifically,
adding object365 dataset gives an absolute gain of 3.7 points on the average
mAP. Surprisingly, adding LVIS to the pre-train data hurts performance by 1.1
points. We speculate that the performance drop is due to the noisy and
incomplete annotations of LVIS dataset. Adding GCC dataset to the pre-train
corpora yields another huge gain, leading the zero-shot performance to 16.0
(compared to 9.8 for OmDet-C). There are several promising directions to
further improve the zero-shot performance OmDet, including unfreezing the text
encoder during pre-training and incorporating phrase grounding data with
contextual text information in pre-training. We leave them to future research.

Meanwhile, the 35 downstream tasks in ODinW come with different training data
sizes, varying from only 17 training images to more than 32K training images.
Therefore, we divide the 35 tasks into three categories: (1) Few-shot (8
tasks): tasks with less than 200 training images (2) Medium-shot (13 tasks):
tasks with between 200 to 2000 training data (3) Big-shot (14 tasks): tasks
with more than 2000 training images. Results with full-model fine-tuning are
summarized in Table 4. Results show that large-scale multi-dataset pre-
training is particularly effective for few-shot and medium-shot tasks with
limited in-domain training data. Especially for few-shot datasets, OmDet
outperforms OmDet-C with 6.41 absolute AP points. Whereas for Big-shot tasks,
the advantages of pre-training become less evident.

Parameter-efficient Fine-tuning As large-scale pretraining models get
significantly larger, e.g., more than 1B parameters, the cost to fine-tune
(FT) the entire model becomes prohibitive for low-end GPUs. Parameter-
efficient fine-tuning is designed to alleviate this challenge by only tuning a
very small proportion of the entire model. In this paper, we explore two
options: Head-only Tuning and Prompt Tuning.

Experimental results show that large-scale multi-dataset pre-training is
crucial for successful parameter-pretraining (Table 3). For Head-only FT, the
performance drop is reduced from 11.3% for OmDet-C to only 6.1% for OmDet. The
same trend is observed for Prompt FT, in which the performance drop compared
to full-model tuning is reduced from 65.9% to 45.5% from OmDet-C to OmDet.
Figure 3 also visualizes the trend of AP vs. the vocabulary size in pre-
training (log-scale). The apparent up-going curve can be observed as more
visual concepts are included during pre-training. This suggests that:

(1) multi-dataset pre-training enables the accumulation of a large number of
visual concepts, which leads to a stronger backbone that extracts general-
purpose visual features (supported by head-only FT results).

(2) the diversity in language is crucial for successful prompt tuning such
that the entire model output can be controlled by the task embedding only
(less than 1% of the parameters of the entire model).

Also, we found that the prompt-tuning performance of OmDet is significantly
lower than GLIP. We suspect the prompt tuning used in OmDet is too simple,
i.e., initialize the task embedding with natural language and tune the task
word embedding alone. We plan to improve the prompt-tuning strategy in a later
version of this pre-print.

Visualization of Language-Aware Detection Lastly, Figure 4 shows qualitative
visualization of language-aware OD. By inputting different tasks, e.g.
[Sandwich, Tobacco Pipe] vs. [Lighter, Bottle], OmDet can dynamically adapt
its object localization and classification conditioned on the given task.
Figure 4 visualizes the intermediate output at each stage of the MDN block. We
found that the model learns to place its proposal boxes as the whole image for
the initial stage and quickly narrows its focus from the initial whole-image
boxes to the objects of interest quickly in 2-3 steps in a top-down search
manner. The later stage output continues to refine its output and confidence
scores (e.g. with less duplicated bounding boxes and more certain confidence).

##  6 Conclusion

This work proposes to advance zero/few-shot OD via continual pre-training from
a large number of OD datasets. OmDet is proposed to solve the taxonomy
conflict and fore/background inconsistency problems during multi-datasets
joint training. The proposed deep fusion mechanism, Multimodal Detection
Network, is able to detect specified objects conditioned on users task input
in the format of free-form natural language. Experiments show that enlarging
the vocabulary size via multi-datasets pre-training effectively improves
zero/few-shot learning and parameter-efficient fine-tuning. OmDet achieved the
state-of-the-art performance on a diverse set of downstream tasks. Future
research will focus on improving OmDet by efficient task-sampling strategy,
utilizing more diverse multimodal datasets, and exploring diverse language and
vision backbones with freezing particular parameters or fully updating them.

## References

  * [1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [4] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li.  Learning to prompt for open-vocabulary object detection with vision-language model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14084–14093, 2022.
  * [5] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [6] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.
  * [7] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.
  * [8] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  arXiv preprint arXiv:2104.13921, 2021.
  * [9] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019.
  * [10] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [11] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.
  * [12] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.  Mdetr-modulated detection for end-to-end multi-modal understanding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780–1790, 2021.
  * [13] Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Yong Jae Lee, Houdong Hu, Zicheng Liu, et al.  Elevater: A benchmark and toolkit for evaluating language-augmented visual models.  arXiv preprint arXiv:2204.08790, 2022.
  * [14] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.  Grounded language-image pre-training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965–10975, 2022.
  * [15] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [16] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [17] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [18] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In European conference on computer vision, pages 21–37. Springer, 2016.
  * [19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [20] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [21] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976–11986, 2022.
  * [22] Chen Change Loy, Dahua Lin, Wanli Ouyang, Yuanjun Xiong, Shuo Yang, Qingqiu Huang, Dongzhan Zhou, Wei Xia, Quanquan Li, Ping Luo, et al.  Wider face and pedestrian challenge 2018: Methods and results.  arXiv preprint arXiv:1902.06854, 2019.
  * [23] Lingchen Meng, Xiyang Dai, Yinpeng Chen, Pengchuan Zhang, Dongdong Chen, Mengchen Liu, Jianfeng Wang, Zuxuan Wu, Lu Yuan, and Yu-Gang Jiang.  Detection hub: Unifying object detection datasets via query adaptation on language embedding.  arXiv preprint arXiv:2206.03484, 2022.
  * [24] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al.  Simple open-vocabulary object detection with vision transformers.  arXiv preprint arXiv:2205.06230, 2022.
  * [25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [26] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [27] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [28] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor.  Imagenet-21k pretraining for the masses.  arXiv preprint arXiv:2104.10972, 2021.
  * [29] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.  Objects365: A large-scale, high-quality dataset for object detection.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430–8439, 2019.
  * [30] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [31] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al.  Sparse r-cnn: End-to-end object detection with learnable proposals.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14454–14463, 2021.
  * [32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [33] Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji.  Phrasecut: Language-based image segmentation in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10216–10225, 2020.
  * [34] Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang.  Wider face: A face detection benchmark.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5525–5533, 2016.
  * [35] Yongqiang Yao, Yan Wang, Yu Guo, Jiaojiao Lin, Hongwei Qin, and Junjie Yan.  Cross-dataset training for class increasing object detection.  arXiv preprint arXiv:2001.04621, 2020.
  * [36] Bowen Zhao, Chen Chen, Wanpeng Xiao, Xi Xiao, Qi Ju, and Shutao Xia.  Towards a category-extended object detector without relabeling or conflicts.  arXiv preprint arXiv:2012.14115, 2020.
  * [37] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.  Regionclip: Region-based language-image pretraining.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793–16803, 2022.
  * [38] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  International Journal of Computer Vision, pages 1–12, 2022.
  * [39] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra.  Detecting twenty-thousand classes using image-level supervision.  arXiv preprint arXiv:2201.02605, 2022.
  * [40] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  arXiv preprint arXiv:1904.07850, 2019.

Dataset | Categories | # Train Image | #Test Image
---|---|---|---
CottontailRabbits | 1 | 1980 | 10
EgoHands(generic) | 1 | 3840 | 480
MountainDewCommercial | 1 | 17 | 1
Packages | 1 | 19 | 3
Raccoon | 1 | 150 | 17
WildfireSmoke | 1 | 516 | 74
Pistols | 1 | 2377 | 297
Pothole | 1 | 465 | 67
MaskWearing | 2 | 105 | 15
NorthAmericaMushrooms | 2 | 41 | 5
OxfordPets(species) | 2 | 2523 | 358
PKLot640 | 2 | 8691 | 1242
ThermalCheetah | 2 | 90 | 14
ThermalDogsAndPeople | 2 | 142 | 20
BCCD | 3 | 255 | 36
HardHatWorkers | 3 | 5069 | 1766
ShellfishOpenImages | 3 | 407 | 58
EgoHands(specific) | 4 | 3840 | 480
AerialMaritimeDrone(large) | 5 | 52 | 7
AerialMaritimeDrone(tiled) | 5 | 371 | 32
VehiclesOpenImages | 5 | 878 | 126
BrackishUnderwater | 6 | 11739 | 1468
Dice | 6 | 576 | 71
Aquarium | 7 | 448 | 63
DroneControl | 8 | 32688 | 4675
WebsiteScreenshots | 8 | 1688 | 242
SelfDrivingCar | 11 | 24000 | 3000
ChessPieces | 13 | 202 | 29
UnoCards | 15 | 6295 | 899
PascalVOC | 20 | 13690 | 3422
AmericanSignLanguageLetters | 26 | 1512 | 72
Plantdoc | 30 | 2128 | 239
BoggleBoards | 36 | 285 | 35
OxfordPets(breed) | 37 | 2437 | 345
OpenPoetryVision | 43 | 2798 | 402
Total | 314 | 132314 | 20070
Table 5: Statistics of ElEVATER 35 object detection datasets