$\displaystyle\ll_{M}(\beta-\alpha)\cdot\max(1,t_{1}+t_{2}),$

since $\beta\leq M$. We conclude that

$S_{t}^{(0)}(\alpha,\beta)\ll(\beta-\alpha)\cdot\max\left(1,\ln\left(\frac{\beta}{\alpha}\right)\right)\cdot\max(1,t_{1}+t_{2}).$
(B.8)

#### An upper bound for $S_{t}^{(1)}(\alpha,\beta)$

To simplify notation, let us assume that $t_{1}\leq t_{2}$ so that $\lfloor
t\rfloor=t_{1}$ and $\lceil t\rceil=t_{2}$. Then,

$S_{t}^{(1)}(\alpha,\beta)=\left(\sum_{(q_{1},q_{2})\in\mathcal{E}_{1}}\frac{\gcd(q_{1},q_{2})}{\max(q_{1},q_{2})}\right)\cdot
e^{-(2t_{1}+t_{2})},$

If $(q_{1},q_{2})\in\mathcal{E}_{1}$, then

$\frac{e^{t_{1}}}{2M}\leq\frac{\max(q_{1},q_{2})}{\gcd(q_{1},q_{2})}\quad\textrm{and}\quad\max(q_{1},q_{2})\leq\beta_{t}=\beta\cdot
e^{t_{1}+t_{2}},$

and thus $\gcd(q_{1},q_{2})\leq 2M\cdot\beta\cdot e^{t_{2}}$. Let
$\gamma_{t}=\min(\alpha_{t},2M\cdot\beta\cdot e^{t_{2}})$, and note that

$\displaystyle S_{t}^{(1)}(\alpha,\beta)$
$\displaystyle\leq\sum_{d=1}^{\gamma_{t}}\left(\sum_{q_{1},q_{2}=\frac{\alpha_{t}}{d}}^{\frac{\beta_{t}}{d}}\frac{1}{\max(q_{1},q_{2})}\right)\cdot
e^{-(2t_{1}+t_{2})}$
$\displaystyle+\sum_{d=\alpha_{t}}^{\beta_{t}}\left(\sum_{q_{1},q_{2}=1}^{\frac{\beta_{t}}{d}}\frac{1}{\max(q_{1},q_{2})}\right)\cdot
e^{-(2t_{1}+t_{2})}$
$\displaystyle\ll\left(\sum_{d=1}^{\gamma_{t}}\frac{\beta_{t}-\alpha_{t}}{d}\right)\cdot
e^{-(2t_{1}+t_{2})}+\left(\sum_{d=\alpha_{t}}^{\beta_{t}}\frac{\beta_{t}}{d}\right)\cdot
e^{-(2t_{1}+t_{2})}$
$\displaystyle\ll(\beta-\alpha)\cdot(1+\ln(\gamma_{t}))\cdot
e^{-t_{1}}+\beta\cdot\left(\ln\left(\frac{\beta}{\alpha}\right)+\mathcal{O}\left(\frac{1}{\alpha_{t}}\right)\right)\cdot
e^{-t_{1}},$

where we in the last inequality have used (B.5) (with parameters
$\gamma=1,\delta=\gamma_{t}$ and $\gamma=\alpha_{t},\delta=\beta_{t}$). Since

$\ln(\gamma_{t})\ll 1+t_{2},$

we have

$(\beta-\alpha)\cdot(1+\ln(\gamma_{t}))\cdot
e^{-t_{1}}\ll_{M}(\beta-\alpha)\cdot\max(1,t_{1}+t_{2})\cdot e^{-\lfloor
t\rfloor}.$

For the term

$\beta\cdot\left(\ln\left(\frac{\beta}{\alpha}\right)+\mathcal{O}\left(\frac{1}{\alpha_{t}}\right)\right)\cdot
e^{-t_{1}},$

we consider two separate cases. If $\beta/\alpha\geq e$, then
$\ln(\beta/\alpha)$ dominates, and since $\beta-\alpha\gg\beta$, we find

$\beta\cdot\left(\ln\left(\frac{\beta}{\alpha}\right)+\mathcal{O}\left(\frac{1}{\alpha_{t}}\right)\right)\cdot
e^{-t_{1}}\ll_{M}(\beta-\alpha)\cdot\ln\left(\frac{\beta}{\alpha}\right)\cdot
e^{-t_{1}}.$

If $\beta/\alpha<e$, on the other hand, we deduce

$\beta\cdot\left(\ln\left(\frac{\beta}{\alpha}\right)+\mathcal{O}\left(\frac{1}{\alpha_{t}}\right)\right)\cdot
e^{-t_{1}}\ll_{M}\beta\cdot\ln\left(\frac{\beta}{\alpha}\right)\cdot
e^{-t_{1}}+\frac{\beta}{\alpha}\cdot e^{-2t_{1}-t_{2}}.$

On observing that for $1<\beta/\alpha\leq e$

$\beta\cdot\ln\left(\frac{\beta}{\alpha}\right)\ll(\beta-\alpha),$

we now obtain

$\beta\cdot\left(\ln\left(\frac{\beta}{\alpha}\right)+\mathcal{O}\left(\frac{1}{\alpha_{t}}\right)\right)\cdot
e^{-t_{1}}\ll_{M}(\beta-\alpha)\cdot e^{-t_{1}}+e^{-2t_{1}-t_{2}}.$

Then, by combining the previous estimates, we conclude that

$S_{t}^{(1)}(\alpha,\beta)\ll_{M}(\beta-\alpha)\cdot\max\left(1,\ln\left(\frac{\beta}{\alpha}\right)\right)\cdot\max(1,t_{1}+t_{2})\cdot
e^{-\lfloor t\rfloor}+e^{-(t_{1}+t_{2})}.$ (B.9)

#### An upper bound for $S_{t}^{(2)}(\alpha,\beta)$

The following crude estimate will suffice:

$\displaystyle S^{(2)}_{t}(\alpha,\beta)$
$\displaystyle=\left(\sum_{(q_{1},q_{2})\in\mathcal{E}_{2}}1\right)\cdot
e^{-2(t_{1}+t_{2})}\leq\left(\sum_{q_{1}=\alpha_{t}}^{\beta_{t}}\sum_{q_{2}=\alpha_{t}}^{\beta_{t}}1\right)\cdot
e^{-2(t_{1}+t_{2})}$ $\displaystyle\ll(\beta_{t}-\alpha_{t})^{2}\cdot
e^{-2(t_{1}+t_{2})}\ll_{M}(\beta-\alpha)^{2}\ll\beta-\alpha,$ (B.10)

since $\beta\leq M$.

#### Putting it all together

If we now combine (B.7), (B.8), (B.9) and (B.10), we get

$S_{t}(\alpha,\beta)\ll_{M}e^{-(t_{1}+t_{2})}+(\beta-\alpha)\cdot\max\left(1,\ln\left(\frac{\beta}{\alpha}\right)\right)\cdot\max(1,t_{1}+t_{2}),$

where the implicit constants are independent of $\alpha$ and $\beta$, and
hence the proof of Lemma B.1 is complete.

##  Appendix C A Borel-Cantelli Lemma for sub-quadratic moments

The aim of this section is to extend a classical Borel-Cantelli-type argument
of Schmidt [16, Section 3] to sub-quadratic moments (the original argument of
Schmidt only involves quadratic moments).

Given $0\leq\alpha<\beta$, let $\mathcal{F}(\alpha,\beta)$ denote the
_$\ell^{1}$ -annulus_

$\mathcal{F}(\alpha,\beta):=\left\\{(n_{1},n_{2})\in\mathbb{N}_{o}^{2}\,:\,\alpha\leq
n_{1}+n_{2}<\beta\right\\}.$

For $\kappa>0$, we define the function
$\theta_{\kappa}:\mathbb{R}\rightarrow[0,\infty)$ by

$\theta_{\kappa}(t):=\frac{t^{2}}{(\ln(e+|t|))^{1+\kappa}}.$ (C.1)

The following properties of the function $\theta_{\kappa}$ can be readily
verified:

  * (P1)

$\theta_{\kappa}$ is a convex function, increasing on $[0,\infty)$, and
$\theta_{\kappa}(t)\leq t^{2}$ for all $t$.

  * (P2)

$\theta_{\kappa}(ct)\leq c^{2}\cdot\theta_{\kappa}(t)$ for all $c\geq 1$ and
$t$.

  * (P3)

$\theta_{\kappa}^{-1}(u)\ll_{\kappa}u^{1/2}\left(\ln^{+}u\right)^{(1+\kappa)/2}$,
for all $u\geq 0$, where $\ln^{+}u:=\ln\max(e,u)$.

######  Remark C.1.

The arguments below do not depend on the exact form of $\theta_{\kappa}$, but
only make use of the properties (P1), (P2), and (P3).

###  C.1. Main result

Let $(Y,\nu)$ be a probability measure space. We assume that for every $T\geq
1$ and $n\in\mathbb{N}_{o}^{2}$, we are given a real-valued measurable
function $\psi_{T,n}$ on $Y$. Furthermore, we assume that there exists a
positive sequence $(\beta_{T})$ such that $\beta_{T}\rightarrow\infty$ when
$T\rightarrow\infty$ and such that $\psi_{T,n}=0$ for all
$n\notin\mathcal{F}(0,\beta_{T})$ and for all $T$. Define

$S_{T}(\alpha,\beta):=\sum_{n\in\mathcal{F}(\alpha,\beta)}\psi_{T,n},\quad\textrm{for
$0\leq\alpha<\beta$}.$

Following the ideas of Schmidt in [16], we will show how one can deduce almost
sure asymptotic upper bounds on $S_{T}(0,\beta_{T})$ from upper bounds on
$\theta_{\kappa}$-moments of the partial sums $S_{T}(\alpha,\beta)$ where
$\alpha$ and $\beta$ are integers such that $\alpha<\beta\leq\beta_{T}$.

######  Lemma C.2.

Let $\varepsilon,\kappa>0$ and suppose that there is a sequence $(D_{T})$ such
that

$\int_{Y}\theta_{\kappa}(|S_{T}(\alpha,\beta)|)\,d\nu\ll
D_{T}\cdot|\mathcal{F}(\alpha,\beta)|$

for all integers $\alpha<\beta\leq\beta_{T}$. Then there exist a conull Borel
set $Y_{o}\subset Y$ and a measurable map $T_{o}:Y_{o}\rightarrow[1,\infty)$
such that

$\left|S_{T}(0,\beta_{T})(y)\right|\ll
D_{T}^{1/2}\left(\ln^{+}D_{T}\right)^{(1+\kappa)/2}(\ln\beta_{T})^{2+\frac{\varepsilon+\kappa}{2}}\cdot\left(\ln\ln\beta_{T}\right)^{(1+\kappa)/2}\cdot\beta_{T}$

for all $y\in Y_{o}$ and $T\geq T_{o}(y)$, where the implicit constants are
independent of $T$.

###  C.2. Coverings by dyadic intervals

For $s\in\mathbb{N}$, we define the set

$\mathcal{G}_{s}:=\left\\{(i,j)\in\mathbb{N}_{o}^{2}\,:\,2^{i}(1+j)<2^{s}\right\\}.$
(C.2)

For every $(i,j)\in\mathcal{G}_{s}$, let $I_{i,j}$ denote the interval

$I_{i,j}:=\left[2^{i}j,2^{i}(1+j)\right).$ (C.3)

The following lemma is essentially [16, Lemma 1], although this lemma takes
place in a slightly different setting. We provide a proof of our version for
completeness.

######  Lemma C.3.

Let $s$ be a positive integer. For every integer $1\leq N<2^{s}$, there is a
subset $\mathcal{H}_{N}\subset\mathcal{G}_{s}$ such that

$|\mathcal{H}_{N}|\leq
s\quad\textrm{and}\quad[0,N)=\bigsqcup_{(i,j)\in\mathcal{H}_{N}}I_{i,j}.$

###### Proof.

Fix $1\leq N<2^{s}$, and write

$N=\sum_{k=1}^{p}2^{n_{k}},$

where $0\leq n_{1}<n_{2}<\ldots<n_{p}<s$ are integers. In particular, $p\leq
s$. Let

$N_{0}=0\quad\textrm{and}\quad
N_{m}=\sum_{k=p-m+1}^{p}2^{n_{k}},\quad\textrm{for $m=1,\ldots,p$},$

so that $N_{p}=N$ and

$[0,N)=\bigsqcup_{m=1}^{p}[N_{m-1},N_{m}).$

We claim that each interval $[N_{k},N_{k+1})$ for $k=0,\ldots,p-1$ is of the
form $I_{i,j}$ for some index pair $(i,j)\in\mathcal{G}_{s}$. To see this,
first note that $[0,N_{1})=I_{i_{1},j_{1}}$, where $i_{1}=n_{p}$ and
$j_{1}=0$. More generally, we see that

$N_{m+1}=2^{n_{p-m}}+\ldots+2^{n_{p}}=2^{n_{p-m}}\left(1+2^{n_{p-m+1}-n_{p-m}}+\ldots+2^{n_{p}-n_{p-m}}\right).$

Hence, if we define

$i_{m}=n_{p-m}\quad\textrm{and}\quad
j_{m}=2^{n_{p-m+1}-n_{p-m}}+\ldots+2^{n_{p}-n_{p-m}}$

for $1\leq m\leq p-1$, then $N_{m}=2^{i_{m}}j_{m}$ and
$N_{m+1}=2^{i_{m}}(1+j_{m})$, and thus $[N_{m},N_{m+1})=I_{i_{m},j_{m}}$. In
particular,

$[0,N)=\bigsqcup_{m=1}^{p}I_{i_{m},j_{m}}.$

so if we set $\mathcal{H}_{N}=\\{(i_{m},j_{m})\,:\,1\leq m\leq p\\}$, then the
proof is done. ∎

###  C.3. The number of points in $\ell^{1}$-annuli

We leave the proof of the following simple lemma to the reader.

######  Lemma C.4.

For all $k,l\in\mathbb{N}_{o}$,

$\left|\mathcal{F}(k,k+l)\right|\ll l^{2}+k\cdot l,$

where the implicit constants are independent of $k$ and $l$.

######  Corollary C.5.

For every $s\geq 1$,

$\sum_{(i,j)\in\mathcal{G}_{s}}\left|\mathcal{F}(I_{i,j})\right|\ll s\cdot
2^{2s}.$

###### Proof.

Fix $(i,j)\in\mathcal{G}_{s}$. Lemma C.4, applied with $k=2^{i}\cdot j$ and
$l=2^{i}$, tells us that

$\left|\mathcal{F}(I_{i,j})\right|=\left|\mathcal{F}(2^{i}j,2^{i}j+2^{i})\right|\ll
2^{2i}+2^{2i}\cdot j\ll 2^{2i}\cdot\max(1,j).$

Since $\mathcal{G}_{s}\subset\\{(i,j)\,:\,0\leq i\leq s,\enskip j<2^{s-i}\\}$,
we see that

$\sum_{(i,j)\in\mathcal{G}_{s}}\left|\mathcal{F}(I_{i,j})\right|\ll\sum_{i=0}^{s}2^{2i}\left(\sum_{j=0}^{2^{s-i}-1}\max(1,j)\right)\ll\sum_{i=0}^{s}2^{2i}\,(2^{s-i})^{2}\ll
s\cdot 2^{2s}.$

∎

###  C.4. Upper bounds from convex moments

######  Lemma C.6.

Let $\kappa>0$, and suppose that for all $T\geq 1$, there exist $D_{T}\geq 0$
such that

$\int_{Y}\theta_{\kappa}\big{(}|S_{T}(\alpha,\beta)|\big{)}\,d\nu\ll_{\kappa}D_{T}\cdot|\mathcal{F}(\alpha,\beta)|,$
(C.4)

for all $0\leq\alpha<\beta$, where the implicit constants are assumed to be
independent of $T$. Then, for every $\varepsilon>0$ and for every integer
$s\geq 2$, there is a Borel set $Y_{s}(\varepsilon)\subset Y$ such that

$\nu(Y_{s}(\varepsilon))\ll_{\kappa}\frac{1}{s^{1+\varepsilon}},$

and with the property that for every integer $1\leq N<2^{s}$,

$\left|S_{T}(0,N)(y)\right|\ll_{\kappa,\varepsilon}D_{T}^{1/2}\left(\ln^{+}D_{T}\right)^{(1+\kappa)/2}s^{2+\frac{\varepsilon+\kappa}{2}}\cdot\left(\ln
s\right)^{(1+\kappa)/2}\cdot 2^{s},$

for all $y\notin Y_{s}(\varepsilon)$.

###### Proof.

Fix $\kappa,\varepsilon>0$ and an integer $s\geq 2$. Define

$Y_{s}(\varepsilon):=\left\\{y\in
Y\,:\,\sum_{(i,j)\in\mathcal{G}_{s}}\theta_{\kappa}\big{(}S_{T}(I_{i,j})(y)\big{)}\geq
D_{T}\cdot s^{2+\varepsilon}\cdot 2^{2s}\right\\}\subset Y.$ (C.5)

Note that by Chebyshev’s inequality and our assumption (C.4), we have

$\displaystyle\nu(Y_{s}(\varepsilon))$ $\displaystyle\leq\frac{1}{D_{T}\cdot
s^{2+\varepsilon}\cdot
2^{2s}}\sum_{(i,j)\in\mathcal{G}_{s}}\int_{Y}\theta_{\kappa}(S_{T}(I_{i,j}))\,d\nu$
$\displaystyle\ll_{\kappa}\frac{1}{s^{2+\varepsilon}\cdot
2^{2s}}\sum_{(i,j)\in\mathcal{G}_{s}}\left|\mathcal{F}(I_{i,j})\right|\ll\frac{1}{s^{1+\varepsilon}},$

where we in the last inequality have used Corollary C.5.

Let us fix an integer $1\leq N<2^{s}$. By Lemma C.3 we can find a subset
$\mathcal{H}_{N}\subset\mathcal{G}_{s}$ such that

$|\mathcal{H}_{N}|\leq
s\quad\textrm{and}\quad[0,N)=\bigsqcup_{(i,j)\in\mathcal{H}_{N}}I_{i,j}.$

Hence,

$|S_{T}(0,N)|=\left|\sum_{(i,j)\in\mathcal{H}_{N}}S_{T}(I_{i,j})\right|\leq\frac{1}{|\mathcal{H}_{N}|}\sum_{(i,j)\in\mathcal{H}_{N}}|\mathcal{H}_{N}|\cdot|S_{T}(I_{i,j})|.$

Since $\theta_{\kappa}$ is increasing, convex and satisfies (P2), we have

$\displaystyle\theta_{\kappa}(|S_{T}(0,N)|)$
$\displaystyle\leq\theta_{\kappa}\left(\frac{1}{|\mathcal{H}_{N}|}\sum_{(i,j)\in\mathcal{H}_{N}}|\mathcal{H}_{N}|\cdot|S_{T}(I_{i,j})|\right)$
$\displaystyle\leq\frac{1}{|\mathcal{H}_{N}|}\sum_{(i,j)\in\mathcal{H}_{N}}\theta_{\kappa}\left(|\mathcal{H}_{N}|\cdot|S_{T}(I_{i,j})|\right)$
$\displaystyle\leq|\mathcal{H}_{N}|\cdot\sum_{(i,j)\in\mathcal{H}_{N}}\theta_{\kappa}(|S_{T}(I_{i,j})|)\leq
s\cdot\sum_{(i,j)\in\mathcal{G}_{s}}\theta_{\kappa}(|S_{T}(I_{i,j})|),$

where we in the last third inequality have used (P2) with
$c=|\mathcal{H}_{N}|\leq s$.

It follows from the last series of inequalities that for every $y\notin
Y_{s}(\varepsilon)$,

$\theta_{\kappa}(|S_{T}(0,N)(y)|)<D_{T}\cdot s^{3+\varepsilon}\cdot 2^{2s}.$

Hence, by (P3),

$\displaystyle|S_{T}(0,N)(y)|$ $\displaystyle\ll D_{T}^{1/2}\cdot
s^{(3+\varepsilon)/2}\cdot 2^{s}\cdot\left(\ln^{+}\left(D_{T}\cdot
s^{3+\varepsilon}\cdot 2^{2s}\right)\right)^{(1+\kappa)/2}$ $\displaystyle\ll
D_{T}^{1/2}\left(\ln^{+}D_{T}\right)^{(1+\kappa)/2}s^{2+\frac{\varepsilon+\kappa}{2}}\cdot\left(\ln
s\right)^{(1+\kappa)/2}\cdot 2^{s},$

for all $s\geq 2$ and $y\notin Y_{s}(\varepsilon)$, where we in the last step
have used that

$\ln^{+}(u\cdot v)\leq 2\cdot\ln^{+}u\cdot\ln^{+}v,\quad\textrm{for all
$u,v\geq 0$}.$

∎

###  C.5. Proof of Lemma C.2

Fix $\varepsilon,\kappa>0$. We assume that there exists $D_{T}\geq 0$ such
that

$\int_{Y}\theta_{\kappa}(|S_{T}(\alpha,\beta)|)\,d\nu\ll_{\kappa}D_{T}\cdot|\mathcal{F}(\alpha,\beta)|,$

for all $\alpha<\beta$. By Lemma C.6, we can find, for every integer $s\geq 2$
and $\varepsilon>0$, a Borel set $Y_{s}(\varepsilon)\subset Y$ such that
$\nu(Y_{s}(\varepsilon))\ll s^{-(1+\varepsilon)}$, where the implicit
constants are independent of $s$. In particular,

$\sum_{s\geq 2}\nu(Y_{s}(\varepsilon))\ll\sum_{s\geq
2}\frac{1}{s^{1+\varepsilon}}<\infty,$

and thus it follows from Borel-Cantelli’s Lemma that there exists a conull
Borel set $Y_{o}\subset Y$ and a measurable map
$s_{o}:Y_{o}\rightarrow[2,\infty)$ such that for all $y\in Y_{o}$,

$y\notin Y_{s}(\varepsilon),\quad\textrm{for all $s\geq s_{o}(y)$}.$

Lemma C.6 now tells us that for all $y\notin Y_{o}$ and $s\geq s_{o}(y)$,

$|S_{T}(0,N)(y)|\ll_{\kappa,\varepsilon}D_{T}^{1/2}\left(\ln^{+}D_{T}\right)^{(1+\kappa)/2}s^{2+\frac{\varepsilon+\kappa}{2}}\cdot\left(\ln
s\right)^{(1+\kappa)/2}\cdot 2^{s},$

for all integers $1\leq N<2^{s}$.

For all $T$ large enough, there is an integer $s_{T}\geq 2$ such that

$2^{s_{T}-1}\leq\lfloor\beta_{T}\rfloor<2^{s_{T}}.$

Define $T_{o}:Y_{o}\rightarrow[1,\infty]$ by

$T_{o}(y):=\min\\{T\geq 1\,:\,s_{T}\geq s_{o}(y)\\},\quad\textrm{for $y\in
Y_{o}$}.$

Then, from the last inequality (applied with $N=\lfloor\beta_{T}\rfloor$), we
conclude that

$\displaystyle|S_{T}(0,\beta_{T})(y)|$
$\displaystyle=|S_{T}(0,\lfloor\beta_{T}\rfloor)(y)|$
$\displaystyle\ll_{\kappa,\varepsilon}D_{T}^{1/2}\left(\ln^{+}D_{T}\right)^{(1+\kappa)/2}s_{T}^{2+\frac{\varepsilon+\kappa}{2}}\cdot\left(\ln
s_{T}\right)^{(1+\kappa)/2}\cdot 2^{s_{T}}$ $\displaystyle\ll
D_{T}^{1/2}\left(\ln^{+}D_{T}\right)^{(1+\kappa)/2}(\ln\beta_{T})^{2+\frac{\varepsilon+\kappa}{2}}\cdot\left(\ln\ln\beta_{T}\right)^{(1+\kappa)/2}\cdot\beta_{T},$

for all $y\in Y_{o}$ and $T\geq T_{o}(y)$.

## References

  * [1] Björklund, Michael; Gorodnik, Alexander _Central limit theorems for Diophantine approximants_. Math. Ann. 374 (2019), no. 3-4, 1371–1437.
  * [2] Björklund, Michael; Gorodnik, Alexander _Effective Multiple Equidistribution of Translated Measures_. Int. Math. Res. Not. IMRN 2021, https://doi.org/10.1093/imrn/rnab274
  * [3] Björklund, Michael; Gorodnik, Alexander _Counting in generic lattices and higher rank actions._ https://arxiv.org/abs/2101.04931.
  * [4] Chow, Sam; Technau, Niklas _Littlewood and Duffin-Schaffer-type problems in diophantine approximation_. https://arxiv.org/abs/2010.09069.
  * [5] Chow, Sam; Technau, Niclas _Counting multiplicative approximations_. https://arxiv.org/abs/2203.10380.
  * [6] Dani, S. G. _On orbits of unipotent flows on homogeneous spaces, II._ Ergodic Theory Dynamical Systems 6 (1986), 167–182.
  * [7] Eskin, Alex; Margulis, Gregory; Mozes, Shahar _Upper bounds and asymptotics in a quantitative version of the Oppenheim conjecture._ Ann. Math. 147 (1998), no. 1, 93–141.
  * [8] Gallagher, P. _Metric simultaneous diophantine approximation._ J. London Math. Soc. 37 (1962), 387–390.
  * [9] Kleinbock, D. Y.; Margulis, G. A. _Bounded orbits of nonquasiunipotent flows on homogeneous spaces._ Sinai’s Moscow Seminar on Dynamical Systems, 141–172, Amer. Math. Soc. Transl. Ser. 2, 171, Adv. Math. Sci., 28, Amer. Math. Soc., Providence, RI, 1996.
  * [10] Kleinbock, D. Y.; Margulis, G. A. _Flows on homogeneous spaces and Diophantine approximation on manifolds._ Ann. Math. 148 (1998), no. 1, 339–360.
  * [11] Kleinbock, D. Y.; Margulis, G. A. _On effective equidistribution of expanding translates of certain orbits in the space of lattices._ Number theory, analysis and geometry, 385–396, Springer, New York, 2012.
  * [12] Kleinbock, Dmitry; Rao, Anurag _A zero-one law for uniform Diophantine approximation in Euclidean norm._ Int. Math. Res. Not. IMRN 2022, no. 8, 5617–5657.
  * [13] Kleinbock, Dmitry; Rao, Anurag _Weighted uniform diophantine approximation of systems of linear forms_. https://arxiv.org/abs/2111.07115.
  * [14] Kleinbock, Dmitry; Shi, Ronggang; Weiss, Barak _Pointwise equidistribution with an error rate and with respect to unbounded functions._ Math. Ann. 367 (2017), no. 1–2, 857–879.
  * [15] Kleinbock, Dmitry; Wadleigh, Nick _A zero-one law for improvements to Dirichlet’s Theorem._ Proc. Amer. Math. Soc. 146 (2018), no. 5, 1833–1844.
  * [16] Schmidt, Wolfgang _A metrical theorem in diophantine approximation._ Canadian J. Math. 12 (1960), 619–631.
  * [17] Schmidt, Wolfgang _Asymptotic formulae for point lattices of bounded determinant and subspaces of bounded height._ Duke Math. J. 35 (1968), 327–339.
  * [18] Shah, Nimish _Limit distributions of polynomial trajectories on homogeneous spaces._ Duke Math. J. 75 (1994), 711–732.
  * [19] Shi, Ronggang _Expanding cone and applications to homogeneous dynamics._ Int. Math. Res. Not. IMRN 2021, no. 9, 7060–7095.
  * [20] Wang, Yuan; Yu, Kun Rui _A note on some metrical theorems in Diophantine approximation._ Chinese Ann. Math. 2 (1981), no. 1, 1–12.
  * [21] Widmer, Martin _Asymptotic Diophantine approximation: The multiplicative case._ Ramanujan J. 43 (2017), 83–93.


# Refined Vision-Language Modeling for Fine-grained Multi-modal Pre-training

Lisai Zhang1, Qingcai Chen1, Zhijian Chen2, Zhonghua Li2, Yunpeng Han1, Zhao
Cao2

###### Abstract

Fine-grained supervision based on object annotations has been widely used for
vision and language pre-training (VLP). However, in real-world application
scenarios, aligned multi-modal data is usually in the image-caption format,
which only provides coarse-grained supervision. It is not only cost-expensive
but also compute-expensive to collect object annotations and build object
annotation pre-extractor for different scenarios. In this paper, we propose a
fine-grained VLP scheme without object annotations from the linguistic
perspective. First, we propose a homonym sentence rewriting (HSR) algorithm to
provide token-level supervision. The algorithm replaces a
verb/noun/adjective/quantifier word of the caption with its homonyms from
WordNet. Correspondingly, we propose refined vision-language modeling (RVLM)
framework to exploit the token-level supervision. Three refined tasks, i.e.,
refined image-text contrastive (RITC), refined image-text matching (RITM), and
replace language modeling (RLM) are proposed to learn the fine-grained
alignment. Extensive experiments on several downstream tasks demonstrate the
superior performance of the proposed method.

##  1 Introduction

Vision and language pre-training (VLP) greatly empower machines’ ability to
understand multi-modal data in recent years. VLP aims to understand multi-
modal semantics like a human, who is able to recognize fine-grained alignment
between image objects and tokens. However, in real-world application
scenarios, aligned multi-modal data is usually in the coarse-grained image-
caption format, it is an important challenge to learn fine-grained alignment
from the coarse-grained supervision.

Most of the existing fine-grained VLP methods (Zeng, Zhang, and Li 2021; Li et
al. 2022b) learns cross-modal semantic alignment from an object-tag or object-
phrase style supervision as Fig. 1 (a). Since such fine-grained annotations
are not available for the pre-training corpus, a common paradigm is extracting
the object region features and their corresponding tags (Su et al. 2019; Qi et
al. 2020; Li et al. 2020b; Zhang et al. 2021) through off-the-shelf object
detection models like Faster-RCNN (Ren et al. 2015). Afterward, the object
features are fed to the Transformer (Vaswani et al. 2017) as a visual token,
and self-supervision tasks such as masked language modeling (MLM) and image-
text matching (ITM) are used to learn the multimodal representation. Recent
method (Li et al. 2022b) extends the object-tag annotation to multi-grained
objects-concepts using scene graph parser (Anderson et al. 2016) and achieves
significant performance improvement. A recent study has proved (Zhang et al.
2021) that the effectiveness of these VLP methods highly benefits from the
object detector quality. However, these off-the-shelf pre-extractors are built
for certain scenes upon expensive bounding box annotations, which inevitably
limits VLP’s application to more general domains. On the other hand, the
object detector and scene graph parser need to infer on high-resolution
images, which is proved (Li et al. 2021a) to be compute-expensive.

Figure 1: Illustration of the object annotation supervision, image caption
supervision, and our proposed homonyms sentence rewriting (HSR) supervision
schemes. Combined with the ground truth text, HSR provides token-level
supervision without human labeling.

A weakly-supervised paradigm (Gao et al. 2020; Kim, Son, and Kim 2021) to
achieve fine-grained VLP without object annotations is regarding the image
patches as visual tokens. However, removal of the object detector is reported
(Kim, Son, and Kim 2021) to achieve lower performance. Recent work (Li et al.
2021a, 2022a; Yu et al. 2022) use image-text contrastive (ITC) objective to
the uni-modal encoder before fusing the multi-modal features. Together with
MLM and ITM objectives on the multi-modal encoder, these models achieve
excellent performance. However, it has been reported (Li et al. 2022b) that
these methods still fall short of the fine-grained approach when using the
same order of magnitude training corpus.

Semantics of the vision modality, e.g.,textures, objects, and actions, are not
naturally segmented from others, therefore it is hard to develop explicit
fine-grained supervision from the vision modality. Fortunately, language
symbols are naturally discrete, whose basic linguistic semantic units are
tokens. It would be possible to create fine-grained self-supervision signals
from language modality without human labeling. The MLM is a successful
supervision from this perspective. However, most VLP methods are simply using
the same MLM design as the BERT (Devlin et al. 2019). We argue that the
language modeling supervision in VLP still lacks systematical exploration.

In this paper, we propose a refined vision-language modeling (RVLM) scheme for
VLP. First, we propose a homonym sentence rewriting algorithm (HSR) to produce
fine-grained supervision. As illustrated in Fig. 1, we replace one
verb/noun/adjective/quantifier token of a caption by it’s homonyms from the
WordNet (Miller 1995). The rewritten sentence is used as a negative sample,
who’s semantic differs from the original caption by only one token.
Afterwards, we design three self-supervised objectives to capture the
supervision. On the uni-modal encoders, a refined image-text contrastive
(RITC) task maximizes the similarity between the image and the ground truth
tokens, and minimize the similarity between the replaced tokens. On the multi-
modal encoder, a replace language modeling (RLM) task predicts which token is
the replaced one, and a refined image-text matching (RITM) task enforces the
image-caption similarity to be higher than the image-rewritten text.

We demonstrate the effectiveness of the proposed fine-grained VLP framework on
various downstream vision-language tasks. The RVLM outperforms weakly-
supervised models significantly and is able to achieve compatible or better
performance than the fine-grained supervised models.

Our main contributions can be summarized in threefold:

  1. 1.

We present a fine-grained VLP scheme from linguistic perspective, which
obtains token-level supervision by the HSR algorithm on image-caption pairs
without object annotations.

  2. 2.

A series of refined vision-language modeling tasks, i.e., RITC, RITM, and RLM,
are designed to exploit the fine-grained supervision.

  3. 3.

Extensive experiments conducted on several downstream vision-language tasks
demonstrate the effectiveness of the proposed scheme. As a byproduct, we will
release the codes, and involved parameters to benefit other researchers.

Figure 2: Overall architecture of RVLM. The input caption $T$ is rewritten for
negative text $\hat{T}$ by the HSR algorithm. The uni-modal representation is
learned through the refined image-text contrastive (RITC) objective. The
multi-modal representation is learned through refined image-text matching
(RITM) and replace language modeling (RLM) objectives.

##  2 Related Work

###  2.1 Fine-grained VLP

The object-level annotations used by the early VLP models are from the bottom-
up and top-down features (Anderson et al. 2018), which are produced by a
Faster-RCNN (Ren et al. 2015) pre-trained on Visual Genome (Krishna et al.
2017). (Tan and Bansal 2019; Lu et al. 2019) use two single-modal networks
applied for text and images respectively, followed by a cross-modal
transformer combining the two sources. Based on the object annotation, a mask
object modeling objective is designed to learn the fine-grained corresponding
between the objects and texts. The following methods (Li et al. 2019, 2020a;
Chen et al. 2020; Su et al. 2019; Li et al. 2020b) treat the ROIs as image
tokens, and unified the image and text encoder by a BERT (Devlin et al. 2019)
network. These models improved the pre-training objectives to learn the cross-
modal correspondence better and achieve better performance. VinVL (Zhang et
al. 2021) explores the importance of the object detector and achieves higher
performance on downstream vision and language tasks using an improved object
detector (Anderson et al. 2016). Recently, (Li et al. 2022b) extend the
object-tag supervision to multi-grained objects-concepts from scene graph
parser (Anderson et al. 2016). Meanwhile, (Zeng, Zhang, and Li 2021)
introduces the object box into the pre-training and proposes a phrase
grounding objective. These multi-grained methods achieve significant
performance improvement through richer object annotations. The object detector
and scene graph parser are proved to be more compute-expensive (Li et al.
2021a) than ViT (Dosovitskiy et al. 2020). On the other hand, the models rely
on expensive bounding box annotations for certain scenes, which inevitably
limits VLP’s application to more general domains. RITM explores fine-grained
supervision independent of the object annotations from the language modality.
Additionally, RITM is faster than these fine-grained VLP methods because it
does not require the object features extraction stage.

###  2.2 Weakly-supervised Fine-grained VLP

The weakly-supervised fine-grained VLP methods do not use object annotations.
(Gao et al. 2020) first explore patch-based image token for VLP in e-commerce
retrieval. They split fashion images into patches and use the ResNeXt-101 (Xie
et al. 2017) to extract features as the visual input. ViLT (Kim, Son, and Kim
2021) propose to use the patches tokens as ViT (Dosovitskiy et al. 2020),
which is totally convolution-free. Since there are no object annotations, the
model is trained with ITM and MLM objectives. The model achieves a higher
inference speed than the fine-grained models, but lower performance than the
fine-grained VLP methods while using a similar corpus. ALBEF (Li et al. 2021a)
make up the performance gap by an align before fuse framework. The framework
optimizes an ITC objective on the uni-modal encoder output and uses a momentum
distillation network to guide the training process. Based on the framework,
(Li et al. 2022a; Yu et al. 2022) propose to use auto-regressive language
generation as the language modeling task, and use the captioner to boost the
training data scale. These methods achieve excellent performance with the
augmented data. However, it has been reported (Li et al. 2022b) that these
methods still fall short of the fine-grained approach when using the same
order of magnitude training corpus. These methods simply use the same language
modeling objectives as the uni-modal pre-training BERT (Devlin et al. 2019),
and does not fully explore the language modeling tasks for vision and
language. RITM proposes a refined language modeling scheme for VLP, which
provides fine-grained supervision from language modality.

##  3 Methodology

The proposed RVLM learns fine-grained cross-modal alignment from image-caption
input ${I}$ and ${T}$. As illustrated in Fig. 2, the RVLM consists of three
transformer modules: an image encoder and a text encoder for the uni-modal
stage, and a cross-modal encoder for the multi-modal stage.

Formally, we first declare some notations. We use bold capital letters (e.g.,
$\mathbf{X}$) to denote the matrices, and bold lowercase letters (e.g.,
$\mathbf{x}$) for the vectors. The non-bold letters (e.g., $x$) are employed
to represent the scalars and Greek letters (e.g., $\beta$) to denote the
parameters.

###  3.1 Model Input

The input image ${I}$ is spitted into 8x8 pixels’ patches, and then encoded as
sequence of embeddings:
$\mathbf{V}=\\{\mathbf{v}_{cls},\mathbf{v}_{1},...,\mathbf{v}_{N}\\}$
following the ViT (Dosovitskiy et al. 2020), where $\mathbf{v}_{cls}$ is the
embedding of the [CLS] token. The sentence ${T}$ is tokenized as token index
sequence $\mathbf{T}=\\{\mathbf{w}_{cls},\mathbf{w}_{1},...,\mathbf{w}_{L}\\}$
where $L$ is the length of the WordPiece (Johnson et al. 2017) tokenizer
output.

For the model pre-training, we rewrite the input text ${T}$ as ${\hat{T}}$
through homonyms sentence rewriting for the fine-grained supervision. A masked
text $\bar{T}$ is prepared for the MLM.

#### Homonyms Sentence Rewriting

The homonyms sentence rewriting (HSR) aims to provide a fine-grained negative
sentence to the image. Based on the principle that token is the language’s
basic semantic unit, we can change one token of the sentence. The HSR
supervision does not require object annotations.

In order to get a hard negative sentence, the semantic of the substitute
should be related but different to the ground truth word. Therefore, we
propose to choose the homonyms word of the selected token from the WordNet
(Miller 1995). Since the captions are descriptive, we only select the concrete
noun/adj/verb/quantifier words as the rewriting candidates.

As shown in the HSR module from the Fig. 2, given a sentence $T$, we firstly
extract the part-of-speech $S$ of each word, then randomly select one
substitute $T_{i}$ from the noun/adj/verbs/quantifier where $i$ is the word
index. Then we search for the hypernyms from WordNet and go through their
hyponyms to get the homonyms. If the word does hot have homonyms, we further
search for the second-order hypernyms and their second-order hyponyms. The
rewritten text $\hat{T}$ is then tokenized to $\mathbf{\hat{T}}$ used for the
uni-modal and multi-modal learning.

###  3.2 Refined Uni-modal Learning

The uni-modal stage takes the $\mathbf{V}$, $\mathbf{T}$ and
$\mathbf{\hat{T}}$ as input to learning their fine-grained uni-modal
representations. We use a 6-layer transformer (Vaswani et al. 2017) for both
the text encoder and a 12-layer visual transformer ViT-B/16 (Dosovitskiy et
al. 2020) as the image encoder. The inputs are encoded into the uni-modal
representations
$\mathbf{V}^{u}=\\{\mathbf{v}^{u}_{cls},\mathbf{v}^{u}_{1},...,\mathbf{v}^{u}_{N}\\}$,
$\mathbf{T}^{u}=\\{\mathbf{w}^{u}_{cls},\mathbf{w}^{u}_{1},...,\mathbf{w}^{u}_{L}\\}$,
and
$\hat{\mathbf{T}}^{u}=\\{\mathbf{\hat{w}}^{u}_{cls},\mathbf{\hat{w}}^{u}_{1},...,\mathbf{\hat{w}}^{u}_{L}\\}$.
Afterward, the representation is optimized through ITC and RITC.

#### Image-Text Contrastive Learning

ITC mainly considers coarse-grained alignment between the image and sentence.
We use the infoNCE (Yang et al. 2019) loss for the L2-normalized [CLS]
representations $h_{v}(\mathbf{v}^{u}_{cls})$ and
$h_{w}(\mathbf{w}^{u}_{cls})$, where $h_{v}$ and $h_{w}$ are linear projection
layers for the text and image representation. Similar to (Li et al. 2021a), a
momentum network is used to assist the training. Denote the momentum’s
representations as $h^{\prime}_{v}(\mathbf{v^{\prime}}_{cls}^{u})$ and
$h^{\prime}_{w}(\mathbf{w^{\prime}}_{cls}^{u})$, and the similarity between
text and image as
$s(T,I^{\prime})=h_{v}(\mathbf{w}^{u}_{cls})^{\intercal}h_{v}(\mathbf{v^{\prime}}^{u}_{cls})$,
the text to image contrastive loss can be formulated as Eq. 1:

$\mathcal{L}_{c}^{t}=$ (1)

where $\tau$ a learnable temperature parameter, $N$ is the batch size and $M$
is the momentum queue length. Symmetrically, we have Eq. 2.

$\mathcal{L}_{c}^{t}=$ (2)

The ITC loss is calculated using the cross entropy loss on the scores as:

$\mathcal{L}_{itc}=\mathcal{L}_{c}^{t}+\mathcal{L}_{c}^{v}$ (3)

#### Refined Image-Text Contrastive

The RITC loss is designed for the fine-grained similarity between the image
and tokens. We compute the similarities between image and the tokens
$\mathbf{g}=\\{h_{v}(\mathbf{v}_{cls}^{u})^{\intercal}h_{w}(\mathbf{w}^{u}_{i})\\}_{i=1}^{L}$
and the rewritten text
$\hat{\mathbf{g}}=\\{{h_{v}(\mathbf{v}_{cls}^{u})}^{\intercal}h_{w}(\mathbf{\hat{w}}^{u}_{i})\\}_{i=1}^{L}$.
The linear projection layers share the same weights as the ITC. Afterward, we
use the lower bound of $\mathbf{g}$ and $\hat{\mathbf{g}}$ to measure the
text-image similarity, so that the fine-grained alignment could be optimized
using the margin loss as Eq. 4:

$\mathcal{L}_{ritc}=\max(0,(\beta+\min(\mathbf{g})-\min(\hat{\mathbf{g}})))$
(4)

where $\beta$ is the margin hyper parameter. The RITC can be regraded as
weakly supervised learning for the alignment between the image and tokens.

###  3.3 Refined Multi-modal Learning

In the multi-modal stage, we enforce cross-modality semantics learning for
coarse-grained alignment with RITM loss, and fine-grained alignment with MLM
and RLM loss. The multi-modal encoder is a 6-layer cross-transformer that
fuses the uni-modal representations into the language modality. It produces
the multi-modal representation $\mathbf{T}^{m}$, $\bar{\mathbf{T}}^{m}$ and
$\hat{\mathbf{T}}^{m}$ for the caption, masked and rewritten texts.

#### Refined Image-Text Matching

The RITM loss learns a global representation of the text-image pair to predict
if the text and image match. We employ two kinds of negative pairs for the
matching. The first is the in-batch hard negative samples, which are selected
according to the similarity scores from the uni-modal stage. The second is the
image and rewritten text. The loss function is a two-fold binary
classification loss on the [CLS] representation, which is calculated as Eq. 5.

$\displaystyle\mathcal{L}_{ritm}=\mathbb{E}_{(I,T,\hat{T})\sim D}$
$\displaystyle H(\mathbf{y}^{itm},\mathbf{p}^{ritm}(\mathbf{T}^{m}_{cls}))$
(5)
$\displaystyle+H(\mathbf{\hat{y}}^{itm},\mathbf{\hat{p}}^{ritm}(\mathbf{\hat{T}}^{m}_{cls}))$

where $\mathbf{y}^{itm}$ and $\mathbf{\hat{y}}^{itm}$ are 2-dimensional one-
hot vectors representation the alignment label, $H$ denotes the cross entropy
function, $\mathbf{p}^{ritm}$ and $\mathbf{\hat{p}}^{ritm}$ are one-layer
linear projections with softmax that project the [CLS] representations into
binary probability.

#### Mask Language Modeling

The MLM with visual clues is traditional and important for VLP. It utilizes
multi-modal context to predict the masked words. In addition to the
traditional setting that randomly masks out $15\%$ of the input tokens, we
also mask the replaced token indicated by the HSR. Afterward, $80\%$ masked
tokens are replaced by [MASK] and $10\%$ are replaced by random tokens. The
MLM term is formulate as Eq. 6

$\mathcal{L}_{mlm}=\mathbb{E}_{(I,\bar{T})\sim
D}H(\mathbf{y}^{msk},\mathbf{p}^{msk}(\bar{\mathbf{T}}^{m}))$ (6)

where $\mathbf{y}^{msk}$ is a one-hot vocabulary distribution where the
ground-truth token has a probability of 1, $\mathbf{p}^{mlm}$ is the model’s
prediction head network.

MLM objective learns fine-grained correspondence between the multi-modal
context and the token through classification. There are two disadvantages:
First, there may be many proper tokens to fill the mask, while the objective
ignores the other correction tokens. Second, it does not directly learn the
alignment between the multi-modal context and the token.

| COCO | VG | CC | SBU | OI | Flickr30k | Toal
---|---|---|---|---|---|---|---
image | 113k | 100k | 2.8M | 860k | 504k | 30k | 4.4M
text | 567k | 769k | 2.8M | 860k | 507k | 150k | 5.7M
Table 1: Statistics of the pre-training datasets. Methods | Flickr30k Test (1k images) |  | COCO Test (5k images) |  | $\\#$ Pre-train
---|---|---|---|---|---
Text Retrieval |  | Image Retrieval |  |  | Text Retrieval |  | Image Retrieval |  | Images
R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10 | AR |  | R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10 | AR |
Weakly-supervised Fine-grained VLP |  |
ALIGN (2021) | 95.3 | 99.8 | 100 |  | 84.9 | 97.4 | 98.6 | 96.0 |  | 77.0 | 93.5 | 96.9 |  | 59.9 | 83.3 | 89.8 | 83.4 | 1.8B
FILIP (2021) | 96.6 | 100 | 100 |  | 87.1 | 97.7 | 99.1 | 96.8 |  | 78.9 | 94.4 | 97.4 |  | 61.2 | 84.3 | 90.5 | 84.5 | 340M
ViLT (2021) | 83.5 | 96.7 | 98.6 |  | 64.4 | 88.7 | 93.8 | 87.6 |  | 61.5 | 86.3 | 92.7 |  | 42.7 | 72.9 | 83.1 | 73.2 | 14M
ALBEF (2021a) | 94.3 | 99.4 | 99.8 |  | 82.8 | 96.7 | 98.4 | 95.2 |  | 73.1 | 91.4 | 96.0 |  | 56.8 | 81.5 | 89.2 | 81.3 | 4M
ALBEF-14M (2021a) | 95.9 | 99.8 | 100 |  | 85.6 | 97.5 | 98.9 | 96.2 |  | 77.6 | 94.3 | 97.2 |  | 60.7 | 84.3 | 90.5 | 84.1 | 14M
Fine-grained Supervised VLP |  |
Uniter (2020) | 85.9 | 97.1 | 98.8 |  | 72.5 | 92.4 | 96.1 | 90.5 |  | 65.7 | 88.6 | 93.8 |  | 52.9 | 79.9 | 88.0 | 78.1 | 4M
Oscar (2020b) | - | - | - |  | - | - | - | - |  | 70.0 | 91.1 | 95.5 |  | 54.0 | 80.8 | 88.5 | 80.0 | 4M
UNIMO (2021b) | 89.7 | 98.4 | 99.1 |  | 74.7 | 93.4 | 99.1 | 92.4 |  | - | - | - |  | - | - | - | - | 4M
ROSITA (2021) | 88.9 | 98.1 | 99.3 |  | 74.1 | 92.4 | 96.1 | 91.4 |  | 71.3 | 91.6 | 95.6 |  | 54.4 | 80.9 | 88.6 | 80.4 | 4M
VinVL (2021) | 93.6 | 99.1 | 99.9 |  | 82.0 | 95.7 | 97.7 | 94.6 |  | 74.6 | 92.6 | 96.3 |  | 58.1 | 83.2 | 90.1 | 82.5 | 5.7M
MVPTR (2022b) | 95.2 | 99.7 | 100 |  | 84.0 | 96.8 | 98.5 | 95.7 |  | 77.3 | 93.6 | 96.9 |  | 60.1 | 84.0 | 90.7 | 83.8 | 4.7M
RVLM | 94.7 | 99.8 | 99.8 |  | 82.4 | 96.1 | 98.0 | 95.1 |  | 77.3 | 94.2 | 97.2 |  | 60.9 | 84.6 | 91.2 | 84.2 | 4.4M
RVLM-Large | 95.6 | 99.8 | 100 |  | 85.7 | 97.6 | 98.8 | 96.2 |  | 79.5 | 95.1 | 97.9 |  | 63.1 | 85.6 | 91.9 | 85.5 | 14M
Table 2: Image-text retrieval results on Flickr30K and COCO datasets in R@$k$
metrics.

#### Replace Language Modeling

The replace language modeling task predicts which token is replaced according
to the multi-modal content. It makes up the two above-mentioned limitations of
MLM. Firstly, the binary token alignment is certain. Secondly, the objective
directly learns the fine-grained alignment between the image and tokens. The
fine-grained alignment learning is weakly-supervised. The calculation of the
loss is formulated as Eq. 7

$\mathcal{L}_{rlm}=\mathbb{E}_{(I,\hat{T})\sim
D}H(\mathbf{y}^{rlm},\mathbf{p}^{rlm}(\hat{\mathbf{T}}^{m}))$ (7)

where $\mathbf{y}^{rlm}$ is a one-hot distribution of length $L$, where the
replaced token has a probability of 1, $\mathbf{p}^{rlm}$ is the model’s
prediction head network.

The full pre-training objective of RVLM is the combination of the above
motioned terms:

$\mathcal{L}=\mathcal{L}_{itc}+\mathcal{L}_{ritc}+\mathcal{L}_{ritm}+\mathcal{L}_{mlm}+\mathcal{L}_{rmlm}$
(8)

The model is optimized end-to-end on the pre-training datasets to minimize
$\mathcal{L}$.

###  3.4 Pre-training Datasets

We pre-train our model on a large-scale vision-language corpus, including
MSCOCO (Lin et al. 2014), Visual Genome (Krishna et al. 2017), Flickr30k
(Plummer et al. 2015), Conceptual Captions (Sharma et al. 2018)111$\sim$0.5M
images are excluded for bad URLs., SBU (Ordonez, Kulkarni, and Berg 2011) and
OpenImages (Kuznetsova et al. 2020). Table. 1 shows the statistics of the
image and text of the pre-training datasets.

We exclude the val/test splits of Flickr30K as (Li et al. 2022b).
Additionally, we scale up RVLM as (Li et al. 2021a) using a larger-scale web
data Conceptual Concept 12M (Changpinyo et al. 2021), which totally uses 14M
images. The large SBU and Conceptual Captions datasets are reported (Hendricks
et al. 2021) to be noisy, where the captions may not correspond exactly to the
image. Therefore, the token-level supervision of HSR may be blurred by the
noisy words. To solve the problem, we divide the training process into two
stages: the first 30 epochs are trained on all pre-training corpus, and then
10 epochs on the two human-annotated MSCOCO and Visual Genome datasets.

###  3.5 Implementation Details

The image encoder is initialized with the first 12 layers of MAE-base (He et
al. 2022) weight, which is pre-trained on the ImageNet-1k without any labels.
The text encoder and cross-modal encoder are initialized with the BERT-base
(Devlin et al. 2019) weights. The AdamW (Loshchilov and Hutter 2018) optimizer
is adopted with a learning rate of $1e^{-4}$ with $0.02$ weight decay, and
warm-up from the first 20 epochs, then linearly decay 10 epochs to 0. The
batch size is set to $32$ with momentum queue size $65536$. The max sequence
length of text tokens is set to $30$. The model is trained with half-precision
on NVIDIA DGX with Ubuntu system and 8 V100 GPU.

##  4 Experiments

To evaluate the proposed method, we conducted extensive experiments on three
kinds of downstream V+L tasks. We introduce each task and the experimental
results below.

###  4.1 Image-text Retrieval

We evaluate RITM for both image-to-text retrieval and text-to-image retrieval.
The benchmarks are Flickr30K and COCO. The retrieval performance is measured
by the recall at top-k samples (R@k). Three $k$ values, R@$1$, R@$5$, and
R@$10$, are reported for text-to-image retrieval and vice versa. The fine-
tuning is optimized with $\mathcal{L}_{u}+\mathcal{L}_{ritm}$ using the same
set of pre-training. During inference, we employ the recall and re-ranking
strategy as (Li et al. 2021a) to improve retrieval efficiency. We first recall
top-$k$ candidates using the uni-modal similarity score as Eq. 3 for all
image-text pairs, then calculate their ITM score for ranking. In addition, we
also include the CXC (Parekh et al. 2020) metrics, which are reported to
provide a more precise ranking for the COCO images.

Table 2 shows the image-text retrieval comparison between the RVLM and state-
of-the-art VLP methods on COCO 5k and Flickr30k 1k test sets. Overall, the
object annotation supervised methods perform better than the weakly-supervised
methods. However, RVLM achieves comparable performance to object annotation
supervised methods like MVPTR. Compared with the weakly-supervised methods,
RVLM achieves higher performance, especially on the COCO dataset. On the 4M
pre-training setting, RVLM outperforms ALBEF by $4.2\%$ and $4.1\%$ in terms
of the R@$1$ score, indicating that RVLM learns better fine-grained alignments
to distinguish similar images. Compared with the object annotation supervised
MVPTR model, RVLM has a small $0.6\%$ gap in average to MVPTR on Flickr30k but
outperforms it by $0.5\%$ on the COCO 5k test set.

Since some queries have multiple semantically aligned images or text (Parekh
et al. 2020), we further conduct an experiment on the CXC metrics to verify
our improvement on the COCO dataset. As the results in Table 3, RVLM still
constantly outperforms the compared methods and achieves the highest
improvement on the R@$1$ score, which qualifies that RVLM is good at learning
fine-grained alignments.

Methods | COCO Test (CXC)
---|---
Text Retrieval |  | Image Retrieval |
R@1 | R@5 | R@10 |  | R@1 | R@5 | R@10 |
ALBEF | 74.48 | 92.34 | 96.20 |  | 59.43 | 83.67 | 89.87 |
ALBEF-14M | 78.48 | 94.66 | 97.64 |  | 63.63 | 85.80 | 91.45 |
MVPTR | 78.16 | 94.98 | 97.39 |  | 63.52 | 85.03 | 90.97 |
RVLM | 78.22 | 95.10 | 97.70 |  | 63.74 | 86.46 | 92.22 |
RVLM-14M | 80.76 | 95.32 | 98.16 |  | 65.64 | 87.18 | 92.62 |
Table 3: Image-text retrieval results on COCO dataset with CXC (Parekh et al.
2020) metrics.

###  4.2 Multi-Modal Classification

We evaluate RVLM on two widely-used multi-modal classification tasks, i.e.,
visual question answering (VQA) and visual entailment (VE).

VQA requires the model to select an answer given an image and a question as
context. We use the commonly used VQA v2 (Goyal et al. 2017) benchmark as the
evaluation dataset. The evaluation metric is the accuracy on the answer
selection. Following (Li et al. 2021a; Cho et al. 2021), we consider VQA as an
answer generation problem. The RVLM takes the image and question as input and
generates $3,192$ candidate answers using a 6-layer transformer decoder.

VE is a fine-grained visual reasoning task to predict the relationship between
an image and a text. The relationship categories are entailment, neutral, or
contradictory. The evaluation metric is the accuracy of the relation
classification. We follow the common practice of VLP (Li et al. 2021a; Chen et
al. 2020) to formulate VE as a three-way classification problem, and predict
the class probabilities using a multi-layer perceptron (MLP) layer on the
[CLS] token representation $\mathbf{T}^{m}_{cls}$ of the multimodal encoder.

Methods | VQA |  | SNLI-VE
---|---|---|---
test-dev | test-std |  | val | test
ViLT‡ | 70.94 | - |  | - | -
ALBEF‡ | 74.54 | 74.70 |  | 80.14 | 80.30
Uniter | 72.70 | 72.91 |  | 78.59 | 78.28
OSCAR | 73.16 | 73.44 |  | - | -
VILLA | 73.59 | 73.67 |  | 79.47 | 79.03
UNIMO | 73.79 | 74.02 |  | 80.00 | 79.10
VinVL | 75.95 | 76.12 |  | - | -
MVPTR | 76.16 | 76.36 |  | 80.30 | 80.17
RVLM‡ | 75.85 | 75.93 |  | 80.52 | 80.41
Table 4: Multi-modal classification results on VQA v2 and SNLI-VE datasets
(Models with ‡ are weakly-supervised methods).

Table 4 shows the VQA and VE comparison between the RVLM and state-of-the-art
VLP models. On both of the two tasks, object annotation based models are
sightly better than existing weakly-supervised models. On the VQA v2
benchmark, RVLM outperforms other weakly-supervised methods and achieves close
accuracy than the state-of-the-art object annotation supervised models,
proving RVLM’s effectiveness in learning fine-grained semantics. On the VE
task, RVLM outperforms both of the two kinds of methods, proving the model’s
superior ability to learn cross-modal relationships.

###  4.3 Weakly-supervised Visual Grounding

Method | RefCOCO+
---|---
dev | testA | testB
ARN (Liu et al. 2019) | 32.78 | 34.35 | 32.13
CCL (Zhang et al. 2020) | 34.29 | 36.91 | 33.56
ALBEF-14M | 58.46 | 65.89 | 46.25
RVLM-14M | 58.75 | 67.32 | 46.74
Table 5: Weakly-supervised visual grounding results on RefCOCO+ dataset.

Visual grounding requires models to localize the region of the specific
textual description from the input image. Our RVLM is fine-tuned on the
RefCOCO+ (Yu et al. 2016) dataset. Since RVLM does not rely on any object box
inputs, it can not use the bounding box annotations of the dataset as
supervision. Therefore, we use the weakly-supervised setting following (Li et
al. 2021a) and fine-tune the 14M pre-trained model on the image-text matching
score $\mathcal{L}_{itc}$. During inference, we firstly obtain heatmaps on
each pixel using Grad-CAM (Selvaraju et al. 2017), and use them to rank the
detected proposals provided by (Yu et al. 2018). Table 5 shows the weakly-
supervised visual grounding comparison results on RefCOCO+. RVLM outperforms
existing weakly-supervised methods on both the two test sets, proving its
ability in learning fine-grained alignments.

Figure 3: Visualization of the Grad-CAM on cross-attention maps between region
and words (Better viewed with zoom-in). The images surrounded by green or blue
boxes are from the sentence with corresponding words.

###  4.4 Further Analysis

#### Fine-grained Alignment Visualization by Grad-CAM

To inspect the token-level alignment learned by RVLM, we visualize the cross-
attention map on the image for each word using Grad-CAM. A few visualizations
are provided in Fig. 3. In the first row, both the nouns (e.g., alarm and
post) and verbs (mount) are aligned to their corresponding regions. Specially,
we show the attention map of the substitute word “roman” and “arabic” for the
same content. RVLM aligns “roman” correctly, and the ”arabic” was not aligned
to the roman numbers. The second row includes the quantifier words “one” and
“two”, which are aligned to correct number of regions on the giraffes. The
third row shows the fine-grained alignment between the image and multiple
objects. It can be observed that the tokens are precisely aligned to the
objects. Specifically, the two verbs, “drink” is correctly connected to the
cup and mouth, and “hold” to both hands, proving that RVLM is able to
distinguish the verbs. The two nuns, “cup” and “bag” are also correctly
located, verifying RVLM’s fine-grained alignment capability.

Methods | SNLI-VE |  | COCO (5k) |
---|---|---|---|---
dev | test |  | I2T | T2I |
RVLM | 80.52 | 80.41 |  | 77.3 | 60.9 |
w/o RITM | 80.32 | 80.23 |  | 77.0 | 59.7 |
w/o RITC | 80.14 | 80.11 |  | 75.9 | 59.3 |
w/o RLM | 79.09 | 79.03 |  | 76.7 | 58.9 |
w/o HSR | 78.89 | 78.24 |  | 75.1 | 58.8 |
Table 6: Ablation study on the VE and image-text retrieval task. R@$1$ score
is used on COCO 5k test set.

#### Ablation study

We conduct detailed ablation studies on SNLI-VE and COCO 5k datasets to
analyze the effectiveness of each proposed component. Specifically, we remove
the RITM, RLM, and RITC loss in the second stage of pre-training. We also
verify the HSR strategy by using randomly selected words for rewriting. The
variants are trained using the same hyper-parameters and epoch. Table 6 shows
the results of the two benchmarks. It could be observed that the removal of
each component results in a performance decrease for both tasks. Using
randomly selected words for rewriting results in a significant performance
decrease, proving the effectiveness of homonyms words.

##  5 Conclusion

In this paper, we proposed a novel refined vision-language modeling (RVLM) for
weakly-supervised fine-grained vision-language pre-training. Instead of using
object annotations, we proposed a homonyms sentence rewriting (HSR) algorithm
to produce a token-level negative sentence. To capture the fine-grained
supervision from the rewritten sentence, a refined image-text contrastive loss
(RITC) is proposed to learn the fine-grained alignment at the uni-modal stage,
and refined image-text matching (RITM) and replace language modeling (RLM)
task at the multi-modal stage. Extensive experiments and analysis showed that
RVLM outperformed state-of-the-art models on several downstream benchmarks.

Several future directions and improvements could be considered. Our main
objective was to show the potential of the weakly-supervised refined vision-
language modeling framework. The HSR only provides token-level supervision, a
phrase-level rewriting strategy could be considered to develop phrase-level
supervision. Besides, developing the rewriting technique for image modality is
also promising for fine-grained supervision.

## References

  * Anderson et al. (2016) Anderson, P.; Fernando, B.; Johnson, M.; and Gould, S. 2016.  Spice: Semantic propositional image caption evaluation.  In _European conference on computer vision_ , 382–398. Springer.
  * Anderson et al. (2018) Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; and Zhang, L. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 6077–6086.
  * Changpinyo et al. (2021) Changpinyo, S.; Sharma, P.; Ding, N.; and Soricut, R. 2021.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3558–3568.
  * Chen et al. (2020) Chen, Y.-C.; Li, L.; Yu, L.; El Kholy, A.; Ahmed, F.; Gan, Z.; Cheng, Y.; and Liu, J. 2020.  Uniter: Universal image-text representation learning.  In _European conference on computer vision_ , 104–120.
  * Cho et al. (2021) Cho, J.; Lei, J.; Tan, H.; and Bansal, M. 2021.  Unifying vision-and-language tasks via text generation.  In _International Conference on Machine Learning_ , 1931–1942. PMLR.
  * Cui et al. (2021) Cui, Y.; Yu, Z.; Wang, C.; Zhao, Z.; Zhang, J.; Wang, M.; and Yu, J. 2021.  ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross-and Intra-modal Knowledge Integration.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , 797–806.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _International Conference on Learning Representations_.
  * Gao et al. (2020) Gao, D.; Jin, L.; Chen, B.; Qiu, M.; Li, P.; Wei, Y.; Hu, Y.; and Wang, H. 2020\.  Fashionbert: Text and image matching with adaptive loss for cross-modal retrieval.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2251–2260.
  * Goyal et al. (2017) Goyal, Y.; Khot, T.; Summers-Stay, D.; Batra, D.; and Parikh, D. 2017.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 6904–6913.
  * He et al. (2022) He, K.; Chen, X.; Xie, S.; Li, Y.; Dollár, P.; and Girshick, R. 2022.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 16000–16009.
  * Hendricks et al. (2021) Hendricks, L. A.; Mellor, J.; Schneider, R.; Alayrac, J.-B.; and Nematzadeh, A. 2021\.  Decoupling the role of data, attention, and losses in multimodal transformers.  _Transactions of the Association for Computational Linguistics_ , 9: 570–585.
  * Jia et al. (2021) Jia, C.; Yang, Y.; Xia, Y.; Chen, Y.-T.; Parekh, Z.; Pham, H.; Le, Q. V.; Sung, Y.; Li, Z.; and Duerig, T. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  _arXiv preprint arXiv:2102.05918_.
  * Johnson et al. (2017) Johnson, M.; Schuster, M.; Le, Q. V.; Krikun, M.; Wu, Y.; Chen, Z.; Thorat, N.; Viégas, F.; Wattenberg, M.; Corrado, G.; et al. 2017.  Google’s multilingual neural machine translation system: Enabling zero-shot translation.  _Transactions of the Association for Computational Linguistics_ , 5: 339–351.
  * Kim, Son, and Kim (2021) Kim, W.; Son, B.; and Kim, I. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _International Conference on Machine Learning_ , 5583–5594. PMLR.
  * Krishna et al. (2017) Krishna, R.; Zhu, Y.; Groth, O.; Johnson, J.; Hata, K.; Kravitz, J.; Chen, S.; Kalantidis, Y.; Li, L.-J.; Shamma, D. A.; et al. 2017.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ , 123(1): 32–73.
  * Kuznetsova et al. (2020) Kuznetsova, A.; Rom, H.; Alldrin, N.; Uijlings, J.; Krasin, I.; Pont-Tuset, J.; Kamali, S.; Popov, S.; Malloci, M.; Kolesnikov, A.; et al. 2020.  The open images dataset v4.  _International Journal of Computer Vision_ , 128(7): 1956–1981.
  * Li et al. (2020a) Li, G.; Duan, N.; Fang, Y.; Gong, M.; and Jiang, D. 2020a.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 11336–11344.
  * Li et al. (2022a) Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022a.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  _arXiv preprint arXiv:2201.12086_.
  * Li et al. (2021a) Li, J.; Selvaraju, R.; Gotmare, A.; Joty, S.; Xiong, C.; and Hoi, S. C. H. 2021a.  Align before fuse: Vision and language representation learning with momentum distillation.  volume 34.
  * Li et al. (2019) Li, L. H.; Yatskar, M.; Yin, D.; Hsieh, C.-J.; and Chang, K.-W. 2019.  VisualBERT: A Simple And Performant Baseline For Vision And Language.  _arXiv preprint arXiv:1908.03557_.
  * Li et al. (2021b) Li, W.; Gao, C.; Niu, G.; Xiao, X.; Liu, H.; Liu, J.; Wu, H.; and Wang, H. 2021b.  UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 2592–2607.
  * Li et al. (2020b) Li, X.; Yin, X.; Li, C.; Zhang, P.; Hu, X.; Zhang, L.; Wang, L.; Hu, H.; Dong, L.; Wei, F.; et al. 2020b.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _European Conference on Computer Vision_ , 121–137.
  * Li et al. (2022b) Li, Z.; Fan, Z.; Tou, H.; and Wei, Z. 2022b.  MVP: Multi-Stage Vision-Language Pre-Training via Multi-Level Semantic Alignment.  _arXiv preprint arXiv:2201.12596_.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , 740–755. Springer.
  * Liu et al. (2019) Liu, X.; Li, L.; Wang, S.; Zha, Z.-J.; Meng, D.; and Huang, Q. 2019.  Adaptive reconstruction network for weakly supervised referring expression grounding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2611–2620.
  * Loshchilov and Hutter (2018) Loshchilov, I.; and Hutter, F. 2018.  Decoupled Weight Decay Regularization.  In _International Conference on Learning Representations_.
  * Lu et al. (2019) Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _arXiv preprint arXiv:1908.02265_.
  * Miller (1995) Miller, G. A. 1995.  WordNet: a lexical database for English.  _Communications of the ACM_ , 38(11): 39–41.
  * Ordonez, Kulkarni, and Berg (2011) Ordonez, V.; Kulkarni, G.; and Berg, T. L. 2011.  Im2Text: Describing Images Using 1 Million Captioned Photographs.  In _Neural Information Processing Systems (NIPS)_.
  * Parekh et al. (2020) Parekh, Z.; Baldridge, J.; Cer, D.; Waters, A.; and Yang, Y. 2020.  Crisscrossed captions: Extended intramodal and intermodal semantic similarity judgments for MS-COCO.  _arXiv preprint arXiv:2004.15020_.
  * Plummer et al. (2015) Plummer, B. A.; Wang, L.; Cervantes, C. M.; Caicedo, J. C.; Hockenmaier, J.; and Lazebnik, S. 2015.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In _Proceedings of the IEEE international conference on computer vision_ , 2641–2649.
  * Qi et al. (2020) Qi, D.; Su, L.; Song, J.; Cui, E.; Bharti, T.; and Sacheti, A. 2020.  Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data.  _arXiv_ , 1–12.
  * Ren et al. (2015) Ren, S.; He, K.; Girshick, R.; and Sun, J. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _Advances in neural information processing systems_ , 91–99.
  * Selvaraju et al. (2017) Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 618–626.
  * Sharma et al. (2018) Sharma, P.; Ding, N.; Goodman, S.; and Soricut, R. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2556–2565.
  * Su et al. (2019) Su, W.; Zhu, X.; Cao, Y.; Li, B.; Lu, L.; Wei, F.; and Dai, J. 2019.  VL-BERT: Pre-training of Generic Visual-Linguistic Representations.  In _International Conference on Learning Representations_.
  * Tan and Bansal (2019) Tan, H.; and Bansal, M. 2019.  LXMERT: Learning Cross-Modality Encoder Representations from Transformers.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_ , 5100–5111.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Xie et al. (2017) Xie, S.; Girshick, R.; Dollár, P.; Tu, Z.; and He, K. 2017.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1492–1500.
  * Yang et al. (2019) Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R. R.; and Le, Q. V. 2019\.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ , 32.
  * Yao et al. (2021) Yao, L.; Huang, R.; Hou, L.; Lu, G.; Niu, M.; Xu, H.; and Xu, C. 2021.  Fine-grained Interactive Language-Image Pre-Training.  In _Proceedings of the International Conference on Learning Representations_.
  * Yu et al. (2022) Yu, J.; Wang, Z.; Vasudevan, V.; Yeung, L.; Seyedhosseini, M.; and Wu, Y. 2022.  Coca: Contrastive captioners are image-text foundation models.  _arXiv preprint arXiv:2205.01917_.
  * Yu et al. (2018) Yu, L.; Lin, Z.; Shen, X.; Yang, J.; Lu, X.; Bansal, M.; and Berg, T. L. 2018.  Mattnet: Modular attention network for referring expression comprehension.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 1307–1315.
  * Yu et al. (2016) Yu, L.; Poirson, P.; Yang, S.; Berg, A. C.; and Berg, T. L. 2016.  Modeling context in referring expressions.  In _European Conference on Computer Vision_ , 69–85. Springer.
  * Zeng, Zhang, and Li (2021) Zeng, Y.; Zhang, X.; and Li, H. 2021.  Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts.  _arXiv preprint arXiv:2111.08276_.
  * Zhang et al. (2021) Zhang, P.; Li, X.; Hu, X.; Yang, J.; Zhang, L.; Wang, L.; Choi, Y.; and Gao, J. 2021\.  Vinvl: Revisiting visual representations in vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5579–5588.
  * Zhang et al. (2020) Zhang, Z.; Zhao, Z.; Lin, Z.; He, X.; et al. 2020.  Counterfactual contrastive learning for weakly-supervised vision-language grounding.  _Advances in Neural Information Processing Systems_ , 33: 18123–18134.