Asymptotics of correlation function in the long-range random-cluster model
Sharp asymptotics of correlation functions in the subcritical long-range
random-cluster and Potts models Yacine Aoun111Université de Genève,
<EMAIL_ADDRESS>statistical mechanics; probability theory; Potts model;
Ising model; long-range; random-cluster model; percolation 82B20; 82B43 60K35
June 27, 2020 April 4, 2021 2007.00116 26 2021 22 10.1214/21-ECP390 For a
family of random-cluster models with cluster weights $q\geq 1$, we prove that
the probability that $0$ is connected to $x$ is asymptotically equal to
$\tfrac{1}{q}\chi(\beta)^{2}\beta J_{0,x}$ for $\beta<\beta_{c}$. The method
developed in this article can be applied to any spin model for which there
exists a random-cluster representation which is monotonic.

##  1 Introduction

###  1.1 Definitions and main result

The random cluster-model (also called FK-percolation) was introduced by
Fortuin and Kastelyn in 1969 [5] and has become a fundamental example of
dependent percolation, in particular because of its relation to the Potts
model. Indeed, the spin correlations of Potts models can be linked to the
cluster connectivity properties of their random-cluster representations. This
allows the use of geometric techniques developed for percolation to study the
Potts model. We refer to [2, 7] for books on the subject and a recent
discussion of existing results.

The model is defined as follows. For a finite subgraph $\Lambda$ of
$\mathbb{Z}^{d}$, a percolation configuration
$\omega=(\omega)_{x,y\in\Lambda}$ is an element of
$\\{0,1\\}^{\mathcal{P}_{2}(\Lambda)}$, where
$\mathcal{P}_{2}(\Lambda)=\\{\\{x,y\\}:x,y\in\Lambda,x\neq y\\}$. A
configuration $\omega$ can be seen as a subgraph of $\Lambda$ with vertex-set
$\Lambda$ and edge-set given by
$\\{\\{x,y\\}\in\mathcal{P}_{2}(\Lambda):\omega_{x,y}=1\\}$. If
$\omega_{x,y}=1$, we say that $\\{x,y\\}$ is open. Let $k(\omega)$ be the
number of connected components in $\omega$.

Consider $J=(J_{x,y})_{x,y\in\Lambda}$ non-negative coupling constants. Fix
$\beta,q>0$. Let $\mu_{\Lambda,\beta,q}$ be a measure defined for any
$\omega\in\\{0,1\\}^{\mathcal{P}_{2}(\Lambda)}$ by

$\mu_{\Lambda,\beta,q}(\omega)=\dfrac{q^{k(\omega)}}{Z}\prod\limits_{\\{x,y\\}\in\mathcal{P}_{2}(\Lambda)}(1-e^{-\beta
J_{x,y}})^{\omega_{x,y}}$,

where $Z$ is a normalizing constant introduced in such a way that
$\mu_{\Lambda,\beta,q}$ is a probability measure. The measure
$\mu_{\Lambda,\beta,q}$ is called the random-cluster measure on $\Lambda$ with
free boundary conditions. For $q\geq 1$, the measures can be extented to
$\mathbb{Z}^{d}$ by taking the weak limit of measures defined in finite
volume.

We say that $x$ and $y$ are connected in $S\subseteq\mathbb{Z}^{d}$ if there
exists a finite sequence of vertices $(v_{i})_{i=0}^{n}$ in $S$ such that
$v_{0}=x$, $v_{n}=y$ and $\\{v_{i},v_{i+1}\\}$ is open for every $0\leq i<n$.
We denote this event by $x\overset{S}{\leftrightarrow}y$. If
$S=\mathbb{Z}^{d}$, we drop it from the notation. We write
$0\leftrightarrow\infty$ if for every $n\in\mathbb{N}$, there exists
$x\in\mathbb{Z}^{d}$ such that $0\leftrightarrow x$ and $|x|\geq n$, where
$|\cdot|$ denotes a norm on $\mathbb{Z}^{d}$.

For $q\geq 1$, the model undergoes a phase transition: there exists
$\beta_{c}\in[0,\infty]$ satisfying

$\mu_{\mathbb{Z}^{d},\beta,q}(0\leftrightarrow\infty)=\begin{cases}=0&\text{if
}\beta<\beta_{c},\\\ >0&\text{if }\beta>\beta_{c}.\\\ \end{cases}$

For $\beta<\beta_{c}$, it follows from the definition that
$\mu_{\mathbb{Z}^{d},\beta,q}(0\leftrightarrow x)$ goes to $0$ as $|x|$ goes
to infinity. In [3], it was proved that if the coupling constants are finite-
range, meaning that there exists $R>0$ such that $J_{x,y}=0$ whenever
$|x-y|>R$, then the probability of two points being connected decays
exponentially fast in distance, i.e. for every $\beta<\beta_{c}$, there exists
$c(\beta)>0$ such that for every $x$ in $\mathbb{Z}^{d}$,

$\mu_{\mathbb{Z}^{d},\beta,q}(0\leftrightarrow x)\leq\exp(-c|x|).$ (1.1)

In this article, we consider the random-cluster models with strictly positive
infinite-range coupling constants $(J_{x,y})_{x,y\in\mathbb{Z}^{d}}$
satisfying for every $x,y,z\in\mathbb{Z}^{d}$

  1. $\mathbf{H1}$

There exists $c>0$ such that $J_{0,x}\leq cJ_{0,y}$ if $|x|\geq|y|$.

  2. $\mathbf{H2}$

$J_{x-z,y-z}=J_{x,y}$.

  3. $\mathbf{H3}$

$\sum\limits_{y\in\mathbb{Z}^{d}}J_{0,y}<\infty$

  4. $\mathbf{H4}$

For every $x\in\mathbb{Z}^{d}$, for every $\varepsilon>0$, there exists
$\delta>0$ such that for every $y\in\mathbb{Z}^{d}$

$|x-y|\leq\delta|x|\qquad\Rightarrow\qquad|J_{0,x}-J_{0,y}|\leq\varepsilon
J_{0,x}$.

  5. $\mathbf{H5}$

There exist $0<\gamma<1,0<\alpha<1$ and $C_{1}>0$ such that
$\sum\limits_{y\in\mathbb{Z}^{d}}(J_{0,y})^{\alpha}<\infty$ and such that for
every $x\in\mathbb{Z}^{d}$

$\log(J_{0,x})^{2}J_{0,u}J_{0,v}\leq C_{1}J_{0,x}(J_{0,v})^{\alpha},$

with $|u|\geq|x|/\log(J_{0,x})^{2}$ and
$|v|\geq|x|^{\gamma}/\log(J_{0,x})^{2}$.

######  Remark 1.1.

The hypothesis $\mathbf{H5}$ is a technical one and its meaning will become
transparent at the end of the proof of Lemma 2.1.

######  Remark 1.2.

Important examples of coupling constants satisfying $\mathbf{H1}$\-
$\mathbf{H5}$ are $J_{0,x}=|x|^{-c}$ with $c>d$, $J_{0,x}=|x|^{-\log|x|}$ or
more generally $J_{0,x}=e^{-C\log(p(|x|))^{\gamma}}$ for some polynomial
$p\in\mathbb{R}[x]$ of degree at least 1 and $C,\gamma>0$ chosen such that
$\mathbf{H3}$ holds.

######  Remark 1.3.

The hypothesis $\mathbf{H5}$ rules out the stretched exponential decay, i.e.
$J_{0,x}=\exp(-|x|^{\eta})$ with $\eta\in(0,1)$. This implies in particular
that

$\lim\limits_{|x|\rightarrow\infty}\dfrac{|x|^{\xi}}{-\log(J_{0,x})}=\infty$

for every $\xi\in(0,1)$.

We write $o_{x}(1)$ for a function that goes to 0 as $|x|$ goes to infinity.
The main theorem of this article is the following one.

######  Theorem 1.4.

If $(J_{x,y})_{x,y\in\mathbb{Z}^{d}}$ satifies $\mathbf{H1}$\- $\mathbf{H5}$
then for $q\geq 1,\beta<\beta_{c}$ and for every $x\in\mathbb{Z}^{d}$,

$\mu_{\mathbb{Z}^{d},\beta,q}(0\leftrightarrow
x)=\dfrac{\beta\chi(\beta)^{2}}{q}J_{0,x}(1+o_{x}(1)),$ (1.2)

where
$\chi(\beta):=\sum\limits_{x\in\mathbb{Z}^{d}}\mu_{\mathbb{Z}^{d},\beta,q}(0\leftrightarrow
x)$.

This theorem was already proved for $q=2$ (the Ising model) in [11], and a
weaker form of this theorem was proved for $q=1$ (Bernoulli percolation) in
[1] and for the one-dimensional $O(N)$ models with $1\leq N\leq 4$ in [13].
They all relied on the Simon-Lieb type inequalities (see [9]). For
$q\notin\\{1,2\\}$, the Simon-Lieb inequality is not available, so those
approaches cannot be extended. Instead of that, we are going to use the
exponential decay of the size of the connected component of $0$ that was
recently proved in [8]. The latter used the so-called OSSS inequality
introduced in [3]. This inequality was already used to prove sharpness in a
lot of models (see [3, 4, 10]) for which there exists a random-cluster type
representation which is monotonic (see [7, Chapter 2] for a definition of a
monotonic measure). Therefore, the OSSS inequality coupled with the approach
developed in this article can be applied to study subcritical phases of long-
range spin models for which there exists a random-cluster representation which
is monotonic (for instance the Ashkin-Teller model, see [12]).

###  1.2 Applications to the ferromagnetic q-state Potts model

The Potts model is one of the fundamental examples of a lattice spin model
undergoing an ordered/disordered phase transition. It generalizes the Ising
model by allowing spins to take one of $q$ values, where $q$ is an integer
greater than or equal to 2.

The model on $\mathbb{Z}^{d}$ is defined as follows. For a subset $\Lambda$ of
$\mathbb{Z}^{d}$, the probability measure is defined for any
$\sigma=(\sigma_{x})_{x\in\Lambda}\in\\{1,\dots,q\\}^{\Lambda}$ by

$\mathbb{P}_{\Lambda,\beta,q}(\sigma):=\dfrac{\exp(-\beta
H_{\Lambda,q}(\sigma))}{\sum\limits_{\sigma^{\prime}\in\\{1,...,q\\}^{\Lambda}}\exp(-\beta
H_{\Lambda,q}(\sigma^{\prime}))}$ with
$H_{\Lambda,q}(\sigma):=\sum\limits_{x,y\in\Lambda}J_{xy}\delta_{\sigma_{x}\neq\sigma_{y}}$.

The model can be defined on $\mathbb{Z}^{d}$ by taking the weak limit of
measures in finite volume. The measure thus obtained is called the measure
with free boundary conditions and is denoted by
$\mathbb{P}_{\mathbb{Z}^{d},\beta,q}$. The Potts model undergoes a phase
transition between the absence and the existence of long-range order at the
so-called critical inverse temperature $\beta_{c}$, see [7] for details. Our
main theorem from the point of view of the Potts model is the following one.

######  Theorem 1.5.

If $(J_{x,y})_{x,y\in\mathbb{Z}^{d}}$ satifies $\mathbf{H1}$\- $\mathbf{H5}$,
then for $q\geq 1,\beta<\beta_{c}$ and $x\in\mathbb{Z}^{d}$,

$\mathbb{P}_{\mathbb{Z}^{d},\beta,q}(\sigma_{0}=\sigma_{x})-\tfrac{1}{q}=\beta\chi(\beta)^{2}qJ_{0,x}(1+o_{x}(1)),$
(1.3)

where
$\chi(\beta):=\tfrac{1}{q-1}\sum\limits_{x\in\mathbb{Z}^{d}}\mathbb{P}_{\mathbb{Z}^{d},\beta,q}(\sigma_{0}=\sigma_{x})-\frac{1}{q}$.

Since the Potts model and the random-cluster models can be coupled (see [7])
in such a way that

$\mathbb{P}_{\mathbb{Z}^{d},\beta,q}(\sigma_{x}=\sigma_{y})-\frac{1}{q}=\frac{q-1}{q}\mu_{\mathbb{Z}^{d},\beta,q}(x\leftrightarrow
y)$,

Theorem 1.3 is a direct consequence of Theorem 1.4 and we will therefore focus
on Theorem 1.4

###  1.3 Background

The following standard properties will be used in the proof of Theorem 1.4.

Finite energy property. For every $\Lambda\subset\mathbb{Z}^{d},q\geq
1,\omega^{\prime}\in\\{0,1\\}^{\mathcal{P}_{2}(\Lambda)}$ and $x,y\in\Lambda$,

$\mu_{\Lambda,\beta,q}(\omega_{x,y}=1|\omega_{a,b}=\omega^{\prime}_{a,b},\forall\\{a,b\\}\in\mathcal{P}_{2}(\Lambda)\setminus\\{x,y\\})\leq\beta
J_{x,y}$.

We refer to [2] for more details about this property.

Monotonicity of measures. The following is a standard consequence of the FKG
inequality : for $q\geq 1$, two subsets $\Lambda_{1}\subset\Lambda_{2}$ of
$\mathbb{Z}^{d}$ and an increasing event $A$ depending on the edges in
$\Lambda_{1}$ (see [6] for definition of an increasing event and the proof of
this inequality), we have

$\mu_{\Lambda_{1},\beta,q}(A)\leq\mu_{\Lambda_{2},\beta,q}(A).$ (1.4)

Finally, the following non-trivial input will be a key ingredient of the
proof.

######  Theorem 1.6.

For $q\geq 1,\beta<\beta_{c}$, there exists $c_{1}=c_{1}(\beta,q)>0$ such that
for every $n\in\mathbb{N}$

$\mu_{\mathbb{Z}^{d},\beta,q}(|C(0)|\geq n)\leq\exp(-c_{1}n),$ (1.5)

where $C(0):=\\{x\in\mathbb{Z}^{d}:0\leftrightarrow x\\}$.

This theorem was proved in [8].

##  2 Proof of Theorem 1.4

###  2.1 Upper bound

Fix $(J_{x,y})_{x,y\in\mathbb{Z}^{d}}$ satisfying $\mathbf{H1}$\-
$\mathbf{H5}$, $\beta<\beta_{c},q\geq 1$ and $x\in\mathbb{Z}^{d}$. If $0$ is
connected to $x$, then there are two possibilites: either there is a big
number of ’short’ open edges (i.e. open edges whose endpoints are close) in
$C(0)$ or there is a small number of ’long’ open edges in $C(0)$. In the first
case, this implies that the number of vertices in $C(0)$ is big, which is
unlikely to happen by (1.5). In order to make this idea precise, we introduce
some notation. From now on, we will write $\mu$ instead of
$\mu_{\mathbb{Z}^{d},\beta,q}$. Define $f(x):=-2\log(J_{0,x})/c_{1}$ where
$c_{1}$ is provided by Theorem 1.6. Denote by $D_{y}$ the event that the size
of the connected component of $y$ is smaller than $f(x)$. We can partition

$\mu(0\leftrightarrow x)=\mu(0\leftrightarrow x,D_{0})+\mu(0\leftrightarrow
x,D^{c}_{0}).$

Using (1.5) we easily get that

$\mu(D_{0}^{c})=o_{x}(1)J_{0,x}$.

This implies that the size of connected component of $0$ can be assumed to be
smaller than $f(x)$. In this case, we are going to prove two lemmas. Lemma 2.1
gives terms that are negligible with respect to $J_{0,x}$ and Lemma 2.3 gives
the sharp asymptotics.

If $0$ is connected $x$ and the size of the connected component of $x$ is
smaller than $f(x)$, then there must exist an open edge in $C(0)$ whose
endpoints are separated by a distance at least $|x|/f(x)$. This will be an
important observation in the proof of the next lemma. Before stating the
lemma, we introduce some notation.

For a configuration $\omega$, define the random variable
$L_{0,x}(\omega):=\sup\\{|y_{1}-y_{2}|$: $\\{y_{1},y_{2}\\}$ is open, $0$ is
connected to $y_{1}$ and $x$ to $y_{2}$ without using $\\{y_{1},y_{2}\\}\\}$.
For $y\in\\{0,x\\}$ and $\Lambda\subset\mathbb{Z}^{d}$, define
$R^{y}(\Lambda):=\sup\\{|x-y|:x\in\Lambda\cap C(y)\\}$. If $\Lambda=C(y)$, we
simply write $R^{y}$. If $L_{0,x}<\infty$, then there exists an open edge
$\\{y_{1},y_{2}\\}\in\mathcal{P}_{2}(\mathbb{Z}^{d})$ such that
$|y_{1}-y_{2}|=L_{0,x}$ and $0$ is connected to $y_{1}$ and $x$ to $y_{2}$
without using $\\{y_{1},y_{2}\\}$. If there are several such edges, take the
one that maximizes $R^{y}(C(y)\setminus\\{y_{1},y_{2}\\})$. Then, If there are
several such edges, define an order $\prec$ on
$\mathcal{P}_{2}(\mathbb{Z}^{d})$ and choose the one minimal for $\prec$. In
this case, we call $\\{y_{1},y_{2}\\}$ the _maximal edge_ with respect to $y$
and we define $R_{y}:=R^{y}(\tilde{C}(y))$, where
$\tilde{C}(y):=\\{z\in\mathbb{Z}^{d}:z$ is connected to $y$ without using the
edge $\\{y_{1},y_{2}\\}\\}$.

######  Lemma 2.1.

For $\beta<\beta_{c},0<\gamma<1$ given by $\mathbf{H5}$ and
$x\in\mathbb{Z}^{d}$

$\mu(0\leftrightarrow x,D_{0},R_{0}\geq|x|^{\gamma})=o_{x}(1)J_{0,x}.$ (2.1)

######  Proof 2.2 (Proof).

Let $\\{y,z\\}$ be the maximal edge with respect to $0$. Recall that the edge
$\\{y,z\\}$ is open by definition. If $0$ is connected to $x$, by symmetry,
one can assume that $0$ is connected to $y$ and $z$ to $x$ without using the
edge $\\{y,z\\}$. If $|C(0)|\leq f(x)$, then $|y-z|\geq|x|/f(x)$. Set
$k(x):=\lfloor|x|/f(x)\rfloor$ and
$E_{y_{1},y_{2}}=\mathcal{P}_{2}(\mathbb{Z}^{d})\setminus\\{y_{1},y_{2}\\}$.
Finally, for $n\in\mathbb{N}$, we set
$\Lambda_{n}(y):=\\{x\in\mathbb{Z}^{d}:|x-y|<n\\}$. Using the union bound, we
get

$\displaystyle\mu(0\leftrightarrow x,D_{0},R_{0}\geq|x|^{\gamma})$
$\displaystyle\leq\sum\limits_{y\in\mathbb{Z}^{d}}\sum\limits_{z\in\Lambda^{c}_{k}(y)}\mu(0\overset{E_{y,z}}{\longleftrightarrow}y,\omega_{y,z}=1,z\overset{E_{y,z}}{\longleftrightarrow}x,D_{0},R_{0}\geq|x|^{\gamma})$
$\displaystyle=\sum\limits_{y\in\mathbb{Z}^{d}}\sum\limits_{z\in\Lambda^{c}_{k}(y)}\mu(0\overset{E_{y,z}}{\longleftrightarrow}y,\omega_{y,z}=1,z\overset{E_{y,z}}{\longleftrightarrow}x,D_{0},D_{x},R_{0}\geq|x|^{\gamma}),$

where the last equality follows from the fact that if $0$ is connected to $x$,
then $C(0)=C(x)$. For $y\in\mathbb{Z}^{d}$, let
$\tilde{D}_{y}:=\\{|\tilde{C}(y)|\leq f(x)\\}$ where $\tilde{C}(y)$ is defined
above. Notice that the event $D_{y}$ is included in the event $\tilde{D}_{y}$.
Using the inclusion of events, conditioning on
$\\{0\overset{E_{y,z}}{\longleftrightarrow}y\\}\cap\\{z\overset{E_{y,z}}{\longleftrightarrow}x\\}\cap\tilde{D}_{0}\cap\tilde{D}_{x}\cap\\{R_{0}\geq|x|^{\gamma}\\}$
and using the finite energy property, we get

$\begin{aligned}
\mu(0\overset{E_{y,z}}{\longleftrightarrow}y,z\overset{E_{y,z}}{\longleftrightarrow}x,&D_{0},D_{x},R_{0}\geq|x|^{\gamma},\omega_{y,z}=1)\leq\\\
&\leq\beta
J_{y,z}\mu(0\overset{E_{y,z}}{\longleftrightarrow}y,z\overset{E_{y,z}}{\longleftrightarrow}x,\tilde{D}_{0},\tilde{D}_{x},R_{0}\geq|x|^{\gamma})\\\
&\leq c\beta
J_{0,k(x)e_{1}}\mu(0\overset{E_{y,z}}{\longleftrightarrow}y,z\overset{E_{y,z}}{\longleftrightarrow}x,\tilde{D}_{0},\tilde{D}_{x},R_{0}\geq|x|^{\gamma}).\end{aligned}$

with $c$ given by $\mathbf{H1}$. Plugging this into the inequality above gives

$\displaystyle\mu(0\leftrightarrow x,D_{0},R_{0}\geq|x|^{\gamma})$
$\displaystyle\leq c\beta
J_{0,k(x)e_{1}}\sum\limits_{y\in\mathbb{Z}^{d}}\sum\limits_{z\in\Lambda^{c}_{k}(y)}\mu(0\overset{E_{y,z}}{\longleftrightarrow}y,z\overset{E_{y,z}}{\longleftrightarrow}x,\tilde{D}_{0},\tilde{D}_{x},R_{0}\geq|x|^{\gamma})$
$\displaystyle\leq c\beta
J_{0,k(x)e_{1}}\sum\limits_{y\in\mathbb{Z}^{d}}\sum\limits_{z\in\mathbb{Z}^{d}}\mu(0\overset{E_{y,z}}{\longleftrightarrow}y,z\overset{E_{y,z}}{\longleftrightarrow}x,\tilde{D}_{0},\tilde{D}_{x},R_{0}\geq|x|^{\gamma})$
$\displaystyle=c\beta
J_{0,k(x)e_{1}}\mathbb{E}(|\tilde{C}(0)||\tilde{C}(x)|\mathbbm{1}_{\\{\tilde{D}_{0},\tilde{D}_{x},R_{0}\geq|x|^{\gamma})}$
$\displaystyle\leq c\beta
f(x)^{2}J_{0,k(x)e_{1}}\mu(\tilde{D}_{0},R_{0}\geq|x|^{\gamma}).$

In the last line, we used that $|\tilde{C}(0)|\leq f(x)$ on $\tilde{D}_{0}$
and $|\tilde{C}(x)|\leq f(x)$ on $\tilde{D}_{x}$. Observe that if
$|\tilde{C}(0)|\leq f(x)$ and $R_{0}\geq|x|^{\gamma}$, then there exists
$a,b\in\mathbb{Z}^{d}$ such that

  * •

$0$ is connected to $a$ in
$\mathcal{P}_{2}(\mathbb{Z}^{d})\setminus\\{a,b\\}$,

  * •

$|a-b|\geq|x|^{\gamma}/f(x)$,

  * •

$\\{a,b\\}$ is open.

Using the union bound, we get

$\mu(\tilde{D}_{0},R_{0}\geq|x|^{\gamma})\leq\sum\limits_{a\in\mathbb{Z}^{d}}\smashoperator[r]{\sum\limits_{\begin{subarray}{c}b\in\mathbb{Z}^{d}\\\
|b-a|\geq|x|^{\gamma}/f(x)\end{subarray}}}\mu(0\overset{E_{a,b}}{\longleftrightarrow}a,\omega_{a,b}=1).$

As before, the conditioning and the finite energy property give

$\mu(0\overset{E_{a,b}}{\longleftrightarrow}a,\omega_{a,b}=1)\leq\beta
J_{a,b}\mu(0\overset{E_{a,b}}{\longleftrightarrow}a).$

Plugging this into the previous inequality gives

$\displaystyle\mu(D_{0},R_{0}\geq|x|^{\gamma})$
$\displaystyle\leq\sum\limits_{a\in\mathbb{Z}^{d}}\smashoperator[r]{\sum\limits_{\begin{subarray}{c}b\in\mathbb{Z}^{d}\\\
|b-a|\geq|x|^{\gamma}/f(x)\end{subarray}}}\beta
J_{a,b}\mu(0\overset{E_{a,b}}{\longleftrightarrow}a)$
$\displaystyle\leq\beta\chi(\beta)\smashoperator{\sum\limits_{\begin{subarray}{c}b\in\mathbb{Z}^{d}\\\
|b|\geq|x|^{\gamma}/f(x)\end{subarray}}}J_{0,b}.$

In the second line, we used that $(J_{x,y})_{x,y\in\mathbb{Z}^{d}}$ is
invariant under translations. Therefore, combining all the inequalities we get

$\displaystyle\mu(0\leftrightarrow x,D_{0},D_{x},R_{0}\geq|x|^{\gamma})$
$\displaystyle\leq c\chi(\beta)(\beta
f(x))^{2}J_{0,k(x)e_{1}}\smashoperator{\sum\limits_{\begin{subarray}{c}b\in\mathbb{Z}^{d}\\\
|b|\geq|x|^{\gamma}/f(x)\end{subarray}}}J_{0,b}$ $\displaystyle\leq
c_{2}J_{0,x}\smashoperator{\sum\limits_{\begin{subarray}{c}b\in\mathbb{Z}^{d}\\\
|b|\geq|x|^{\gamma}/f(x)\end{subarray}}}(J_{0,b})^{\alpha}=o_{x}(1)J_{0,x}$

with $c_{2}=4C_{1}c^{2}\chi(\beta)\beta^{2}/c_{1}$ where $C_{1}$ is given by
$\mathbf{H5}$. The second inequality follows from the definition of $f(x)$,
$\mathbf{H1}$ and $\mathbf{H5}$. The last equality follows from
$\sum_{w\in\mathbb{Z}^{d}}(J_{0,w})^{\alpha}<\infty$ and Remark 1.3. This
finishes the proof of Lemma 2.1.

Lemma 2.1 implies by symmetry that

$\mu(0\leftrightarrow x,D_{0},R_{x}\geq|x|^{\gamma})=o_{x}(1)J_{0,x}.$ (2.2)

We can then focus on the next lemma, which gives the sharp asymptotics of the
probability of $0$ being connected to $x$.

######  Lemma 2.3.

For $\beta<\beta_{c},0<\gamma<1$ given by $\mathbf{H5}$ and
$x\in\mathbb{Z}^{d}$

$\limsup\limits_{|x|\rightarrow\infty}\dfrac{\mu(0\leftrightarrow
x,R_{0}\leq|x|^{\gamma},R_{x}\leq|x|^{\gamma})}{J_{0,x}}\leq\dfrac{\chi(\beta)^{2}\beta}{q}.$
(2.3)

The upper bound follows by combining (2.1), (2.2) and (2.3).

######  Proof 2.4 (Proof).

Set
$\Lambda=\Lambda_{|x|^{\gamma}}(0),\Lambda^{\prime}=\Lambda_{|x|^{\gamma}}(x)$
and $P_{0,x}:=\\{R_{0}\leq|x|^{\gamma}\\}\cap\\{R_{x}\leq|x|^{\gamma}\\}$. Let
$\Lambda_{n}$ be such that $\Lambda,\Lambda^{\prime}\subset\Lambda_{n}$. Let
$\\{y,z\\}$ be the maximal edge with respect to $0$. If $0$ is connected to
$x$ and $R_{x}\leq|x|^{\gamma}$, then $\\{y,z\\}$ is also maximal with respect
to $x$. By symmetry, one can assume that $0$ is connected to $y$ in $\Lambda$
and $z$ is connected to $x$ in $\Lambda^{\prime}$. The union bound gives

$\displaystyle\mu_{\Lambda_{n}}(0\leftrightarrow x,P_{0,x})$
$\displaystyle\leq\sum\limits_{y\in\Lambda}\sum\limits_{z\in\Lambda^{\prime}}\mu_{\Lambda_{n}}(0\overset{\Lambda}{\leftrightarrow}y,\omega_{y,z}=1,z\overset{\Lambda^{\prime}}{\leftrightarrow}x,P_{0,x})$
$\displaystyle=\sum\limits_{y\in\Lambda}\sum\limits_{z\in\Lambda^{\prime}}\frac{1}{q}(1-\exp(-\beta
J_{y,z}))\mu_{\Lambda_{n}}(0\overset{\Lambda}{\leftrightarrow}y,z\overset{\Lambda^{\prime}}{\leftrightarrow}x,R^{0}\leq|x|^{\gamma},R^{x}\leq|x|^{\gamma})$
$\displaystyle\leq\frac{\beta}{q}\sum\limits_{y\in\Lambda}\sum\limits_{z\in\Lambda^{\prime}}J_{y,z}\mu_{\Lambda_{n}}(0\overset{\Lambda}{\leftrightarrow}y,z\overset{\Lambda^{\prime}}{\leftrightarrow}x,R^{0}\leq|x|^{\gamma},R^{x}\leq|x|^{\gamma}).$

In the second line, we used the fact that on $P_{0,x}$, the number of
connected components increases by 1 when $\omega_{y,z}$ goes from $1$ to $0$.
In the third line, we used that $1-\exp(-\beta J_{y,z})\leq\beta J_{y,z}$. Fix
$\varepsilon>0$. It follows from $\mathbf{H4}$ and the translational
invariance that

$J_{y,z}\leq(1+\varepsilon)J_{0,x},$

since $|z-y-x|\leq\delta|x|$ for $|x|$ big enough. Therefore

$\mu_{\Lambda_{n}}(0\leftrightarrow
x,P_{0,x})\leq(1+\varepsilon)J_{0,x}\sum\limits_{y\in\Lambda}\sum\limits_{z\in\Lambda^{\prime}}\mu_{\Lambda_{n}}(0\overset{\Lambda}{\leftrightarrow}y,\omega_{y_{1},y_{2}}=0,z\overset{\Lambda^{\prime}}{\leftrightarrow}x,R^{0}\leq|x|^{\gamma},R^{x}\leq|x|^{\gamma}).$

Now, we can partition with respect to the possible connected components of $x$
to get

$\mu_{\Lambda_{n}}(0\overset{\Lambda}{\leftrightarrow}y,z\overset{\Lambda^{\prime}}{\leftrightarrow}x,R^{0}\leq|x|^{\gamma},R^{x}\leq|x|^{\gamma})=\sum\limits_{S}\mu_{\Lambda_{n}}(0\overset{\Lambda}{\leftrightarrow}y,R^{0}\leq|x|^{\gamma},C(x)=S),$

where the summation is over $S$ containing $x$ and $z$ such that
$R^{x}(S)\leq|x|^{\gamma}$. Then conditioning on $\\{C(x)=S\\}$ gives

$\displaystyle\sum\limits_{S}\mu_{\Lambda_{n}}(0\overset{\Lambda}{\leftrightarrow}y,R^{0}\leq|x|^{\gamma},C(x)=S)$
$\displaystyle=\sum\limits_{S}\mu_{\Lambda_{n}}(0\overset{\Lambda}{\leftrightarrow}y,R^{0}\leq|x|^{\gamma}\hphantom{,}|C(x)=S)\mu_{\Lambda_{n}}(C(x)=S)$
$\displaystyle=\sum\limits_{S}\mu_{\Lambda_{n}\backslash
S}(0\overset{\Lambda}{\leftrightarrow}y,R^{0}\leq|x|^{\gamma})\mu_{\Lambda_{n}}(C(x)=S)$
$\displaystyle\leq\sum\limits_{S}\mu_{\Lambda_{n}}(0\overset{\Lambda}{\leftrightarrow}y)\mu_{\Lambda_{n}}(C(x)=S)$
$\displaystyle=\mu_{\Lambda_{n}}(0\overset{\Lambda}{\leftrightarrow}y)\mu_{\Lambda_{n}}(z\overset{\Lambda^{\prime}}{\leftrightarrow}x,R^{x}\leq|x|^{\gamma}).$

In the second line, we used the spatial Markov property (see [6, Chapter 3])
as well as the fact that if $w\in S$ and $z\notin S$, then $\\{w,z\\}$ is
closed. In the third line, we used the inclusion of events and (1.4). Plugging
this into the inequality above and taking the limit as $n$ goes to infinity,
we get

$\displaystyle\mu(0\leftrightarrow x,P_{0,x})$
$\displaystyle\leq(1+\varepsilon)\frac{\beta}{q}J_{0,x}\sum\limits_{y\in\Lambda}\sum\limits_{z\in\Lambda^{\prime}}\mu(0\overset{\Lambda}{\leftrightarrow}y)\mu(z\overset{\Lambda^{\prime}}{\leftrightarrow}x,R^{x}\leq|x|^{\gamma})$
$\displaystyle\leq(1+\varepsilon)\frac{\beta}{q}\chi(\beta)^{2}J_{0,x},$

for $|x|$ big enough. We used the translational invariance in the second
inequality. This finishes the proof of Lemma 2.3.

###  2.2 Lower bound

We will use the same argument as in [11]. In this part, we don’t use
$\mathbf{H5}$. Set
$\delta\in(0,1/2],\Delta_{1}=\Lambda_{|x|^{\delta}}(0),\Delta_{2}=\Lambda_{|x|^{\delta}}(x)$
and $\Delta=\Delta_{1}\cup\Delta_{2}$. As we work on $\mathbb{Z}^{d}$, we can
take $\delta=1/2$, but a smaller value may be needed to extend the proof to a
different graph and to more general coupling constants. Let $N$ be the number
of open edges from $\Delta_{1}$ to $\Delta_{2}$. Then the inclusion of events
and the monotonicity of the measure (1.4) give

$\mu_{\Delta}(0\leftrightarrow x,N=1)\leq\mu_{\Delta}(0\leftrightarrow
x)\leq\mu(0\leftrightarrow x)$.

For $y\in\Delta_{1},z\in\Delta_{2}$, let $G_{y,z}$ be the event that there
exists an unique edge $\\{y,z\\}$ such that

  * •

$0\overset{\Delta_{1}}{\leftrightarrow}y$,

  * •

$\\{y,z\\}$ is open,

  * •

$z\overset{\Delta_{2}}{\leftrightarrow}x$.

In this case, $0$ is connected to $x$ and $N=1$. Therefore

$\sum\limits_{y\in\Delta_{1}}\sum\limits_{z\in\Delta_{2}}\mu_{\Delta}(0\overset{\Delta_{1}}{\leftrightarrow}y,\omega_{y,z}=1,z\overset{\Delta_{2}}{\leftrightarrow}x,N=1)=\mu_{\Delta}(\bigsqcup\limits_{\begin{subarray}{c}y\in\Delta_{1}\\\
z\in\Delta_{2}\end{subarray}}G_{y,z})\leq\mu_{\Delta}(0\leftrightarrow
x,N=1).$

In the first equality, we used the fact that the events $G_{y,z}$ are disjoint
for different edges. Using the fact that on the event $N=1$, the number of
connected components increases by 1 when $\omega_{y,z}$ goes from 1 to 0, we
get that

$\mu_{\Delta}(0\overset{\Delta_{1}}{\leftrightarrow}y,\omega_{y,z}=1,z\overset{\Delta_{2}}{\leftrightarrow}x,N=1)=\frac{1}{q}(1-\exp(-\beta
J_{y,z}))\mu_{\Delta}(0\overset{\Delta_{1}}{\leftrightarrow}y,z\overset{\Delta_{2}}{\leftrightarrow}x,N=0).$

Finally, on $N=0$, all the edges between $\Delta_{1}$ and $\Delta_{2}$ are
closed, and therefore we can factorize the measure as

$\mu_{\Delta}(0\overset{\Delta_{1}}{\leftrightarrow}y,z\overset{\Delta_{2}}{\leftrightarrow}x|N=0)=\mu_{\Delta_{1}}(0\leftrightarrow
y)\mu_{\Delta_{2}}(z\leftrightarrow x)$.

Combining all the inequalities we get

$\mu(0\leftrightarrow
x)\geq\sum\limits_{y\in\Delta_{1}}\sum\limits_{z\in\Delta_{2}}\frac{1}{q}(1-\exp(-\beta
J_{y,z}))\mu_{\Delta_{1}}(0\leftrightarrow
y)\mu_{\Lambda^{\prime}}(z\leftrightarrow x)\mu_{\Delta}(N=0).$ (2.4)

Fix $\varepsilon>0$. It follows from $\mathbf{H4}$ and the translational
invariance that

$J_{y,z}\geq(1-\varepsilon)J_{0,x},$

since $|z-y-x|\leq\delta|x|$ for $|x|$ big enough. Therefore, using (2.4), we
get

$\mu(0\leftrightarrow
x)\geq(1-\exp(-\beta(1-\varepsilon)J_{0,x}))\sum\limits_{y\in\Delta_{1}}\sum\limits_{z\in\Delta_{2}}\frac{1}{q}\mu_{\Delta_{1}}(0\leftrightarrow
y)\mu_{\Delta_{2}}(z\leftrightarrow x)\mu_{\Delta}(N=0).$ (2.5)

By the translational invariance and the monotonicity (1.4), we get

$\lim\limits_{|x|\rightarrow\infty}\sum\limits_{y\in\Delta_{1}}\sum\limits_{z\in\Delta_{2}}\mu_{\Delta_{1}}(0\leftrightarrow
y)\mu_{\Delta_{2}}(z\leftrightarrow x)=\chi(\beta)^{2}.$

Now, let us prove that $\mu_{\Delta}(N=0)$ goes to 1 as $|x|$ goes to
infinity. If $N\geq 1$, then there exist $y,z$ in $\mathbb{Z}^{d}$ such that

  * •

$y\in\Delta_{1},z\in\Delta_{2}$,

  * •

$\\{y,z\\}$ is open.

Therefore, the union bound gives

$\mu_{\Delta}(N\geq
1)\leq\sum\limits_{y\in\Delta_{1}}\sum\limits_{z\in\Delta_{2}}\mu_{\Delta}(\omega_{y,z}=1)\leq\beta\sum\limits_{y\in\Delta_{1}}\sum\limits_{z\in\Delta_{2}}J_{y,z}\leq(1+\varepsilon)J_{0,x}|\Delta_{1}|^{2}.$

The second inequality follows from the finite energy property and the third
inequality from $\mathbf{H4}$. Since $\delta\leq 1/2$ and
$\sum_{w\in\mathbb{Z}^{d}}J_{0,w}<\infty$ by $\mathbf{H3}$, it follows that

$J_{0,x}|\Delta_{1}|^{2}=o_{x}(1)$

and therefore $\lim_{|x|\rightarrow\infty}\mu_{\Delta}(N=0)=1$. The lower
bound then follows from (2.5) combined with the fact that

$\lim\limits_{|x|\rightarrow\infty}\dfrac{1-\exp(-\beta J_{0,x})}{\beta
J_{0,x}}=1$.

## References

  * [1] A.G. Braga, L.M. Ciolleti, and R. Sanchis. Decay Properties of the Connectivity for Mixed Long Range Percolation Models on $\mathbb{Z}^{d}$, _Journal of Statistical Physics_ , 129:587–591, 2007.
  * [2] H. Duminil-Copin. Lectures on the Ising and Potts models on the hypercubic lattice. 2017. arXiv: 1707.00520.
  * [3] H. Duminil-Copin, A. Raoufi, and V. Tassion. Sharp phase transition for the random-cluster and Potts models via decision trees. _Annals of Mathematics_ , 189(1):75–99, 2019.
  * [4] H. Duminil-Copin, A. Raoufi, and V. Tassion. Subcritical phase of d-dimensional Poisson-Boolean percolation and its vacant set, 2018. arXiv: 1805.00695
  * [5] C.M. Fortuin and P.W. Kastelyn. On the random-cluster model. I. Introduction and relation to other models. _Physica_ , 57(4):536–564, 1972.
  * [6] S. Friedli and Y. Velenik. _Statistical Mechanics of Lattice Systems: a Concrete Mathematical Introduction_. Cambridge University Press, 2017.
  * [7] G. Grimmett. _The random-cluster model_. Springer-Verlag, 2006.
  * [8] T. Hutchcroft. New critical exponent inequalities for percolation and the random cluster model. 2019. arXiv: 1901.10363.
  * [9] E. Lieb. A refinement of Simon’s correlation inequality. _Communications in Mathematical Physics_ , 77(2):127– 135, 1980.
  * [10] S. Muirhead and H. Vanneuville. The sharp phase transition for level set percolation of smooth planar Gaussian fields. _Ann. Inst. H. Poincare Probab. Statist._ , 56:1358–1390, 2020.
  * [11] C. Newman and H. Spohn. The Shiba relation for the spin-boson model and asymptotic decay in ferromagnetic Ising models. Unpublished, 1998.
  * [12] C.-E. Pfister and Y. Velenik. Random-cluster representation of the Ashkin-Teller model. _Journal of Statistical Physics_ , 88:1295–1331, 1997.
  * [13] H. Spohn and W. Zwerger. Decay of the Two-Point Function in One-Dimensional $O(N)$ Spin Models with Long-Range Interactions. Journal of Statistical Physics 94, 1037–1043, 1999.

The author would like to warmly thank Hugo Duminil-Copin for his guidance and
help through the master thesis as well as reading and pointing out mistakes in
the previous versions of the present article. The author would also like to
thank Yvan Velenik, Maëllie Godard and two anonymous referees for many helpful
comments. Finally, the author would like to thank the Excellence Fellowship
program at the University of Geneva for supporting him during his studies.


# Graph Relation Transformer: Incorporating pairwise object features into the
Transformer architecture

Michael Yang
<EMAIL_ADDRESS>Aditya Anantharaman
<EMAIL_ADDRESS>Zachary Kitowski
<EMAIL_ADDRESS>Derik Clive Robert
<EMAIL_ADDRESS>

Carnegie Mellon University
Pittsburgh, USA

###### Abstract

Previous studies such as VizWiz find that Visual Question Answering (VQA)
systems that can read and reason about text in images are useful in
application areas such as assisting visually-impaired people. TextVQA is a VQA
dataset geared towards this problem, where the questions require answering
systems to read and reason about visual objects and text objects in images.
One key challenge in TextVQA is the design of a system that effectively
reasons not only about visual and text objects individually, but also about
the spatial relationships between these objects. This motivates the use of
‘edge features’, that is, information about the relationship between each pair
of objects. Some current TextVQA models address this problem but either only
use categories of relations (rather than edge feature vectors) or do not use
edge features within the Transformer architectures. In order to overcome these
shortcomings, we propose a Graph Relation Transformer (GRT), which uses edge
information in addition to node information for graph attention computation in
the Transformer. We find that, without using any other optimizations, the
proposed GRT method outperforms the accuracy of the M4C baseline model by
0.65% on the val set and 0.57% on the test set. Qualitatively, we observe that
the GRT has superior spatial reasoning ability to M4C.111The code used to
obtain our results can be found at https://github.com/michaelzyang/graph-
relation-m4c and https://github.com/derikclive/transformers

##  1 Introduction

Visual Question Answering (VQA) is the task of answering questions by
reasoning over the question and the image corresponding to the question.
Although VQA models have shown a lot of improvement in recent years, these
models still struggle to answer questions which require reading and reasoning
about the text in the image. This is an important problem to solve because
studies have shown that visually-impaired people frequently ask questions
which involve reading and reasoning about the text in images. The ‘TextVQA’
task [20] is a VQA task where the questions are focused on the text in images
as shown in Figure 1. In order to answer this question, the model needs to
understand that the visual object ‘13’ is the last object among all objects
towards the right of the image. This leads to the introduction of an
additional modality involving the text in the images which is often recognized
through Optical Character Recognition (OCR). The incorporation of this
additional modality enhances the difficulty of this task as compared to a
standard VQA task.

Figure 1: An example from the TextVQA dataset which shows the importance of
reasoning about spatial relationships between objects.Q What is the last
number to the right? Ans 13.

The current state-of-the-art models [5, 12] in this domain use graph attention
paired with multimodal Transformer based approaches to model relationships
between image objects, OCR objects and question tokens. Both approaches try to
capture the relationship between these objects within the image using rich
edge features. However, we see that they either use edge features for
representation learning before the Transformer layers [5] or do not
incorporate rich edge features when graph attention computation is done in
Transformer layers [12]. These rich edge features can be leveraged within the
Transformer layers to allow for better representations between image and OCR
objects.

In this work, we propose a novel Graph Relation Transformer (GRT) which uses
rich, vector-based edge features in addition to node information for graph
attention computation in the Transformer. The proposed GRT outperforms the M4C
baseline model [10] while also improving the spatial reasoning ability of the
model. We also provide qualitative examples of cases where our proposed
approaches performs better than the M4C baseline model [10].

##  2 Related Work

###  2.1 VQA

For VQA tasks, the main multimodal challenges are how to represent the visual
and language modalities and how to fuse them in order to perform the Question
Answering (QA) task. In terms of representing the questions, word embeddings
such as GloVe [17] are commonly used in conjunction with recurrent neural
networks (RNNs) such as Long Short-Term Memory (LSTM) networks [6], for
example by Fukui _et al_. [4]. For representing the visual modality, grid-
based Convolutional Neural Networks (CNNs) such as Resnet [9] are often used
as visual feature extractors.

For representation, Bottom-up And Top-down Attention (BUTD) [1] is a canonical
VQA method. Previous methods used a fixed-size feature map representation of
the image, extracted by grid-based CNN, whereas BUTD uses a Region-CNN such as
Faster R-CNN [7] to propose a number of variably-sized objects, bottom-up,
over which the question representation will attend, top-down.

Applying several tweaks to the BUTD model, the Pythia model [11] improved its
performance further. Pythia reported performance gains from ensembling with
diverse model setups and different object detectors, architecture, learning
rate schedule and data augmentation and fine-tuning on other datasets.

Bilinear Attention Networks (BANs) [13] aim to improve multimodal fusion by
allowing for attention between every pairwise combination of words in the
question and objects in the image. This setup is motivated by the common
occurrence that different words in the question refer to different objects in
the image and thus allowing for bilinear interaction improves grounding
between the modalities. Low-rank matrix factorization is used to keep the
computation cost tractable and the Multimodal Residual Network (MRN) [13]
method is used to combine all the representations from the bilinear attention
maps.

###  2.2 TextVQA

#### Fusion Methods

Most of the methods proposed for TextVQA involve the fusion of features from
the different modalities - Text, OCR and Images. LoRRA [20] and M4C [10] are
two such model architectures that primarily use attention mechanisms to
capture the interactions between the inputs from different modalities. In
addition, both of these models employ a pointer network [22] module to
directly copy tokens from the OCR inputs. This enables the models to reduce
the number of out-of-vocabulary words predicted by supplementing the fixed
answer vocabulary with the input OCR context.

LoRRA [20] was proposed as the baseline for the 2019 TextVQA challenge and is
composed of three major components - one to combine image and question
features, another to combine OCR and question features and the third one to
generate the answer. M4C improves the fusion of the input modalities through
the use of Multimodal Transformers which allows both inter-modal and intra-
modal interactions. The multimodal Transformer uses features from all three
modalities and uses a pointer-augmented multi-step decoder to generate the
answer one word at a time unlike the LoRRA model that uses a fixed answer
vocabulary.

#### Graph Attention Methods

Graph attention networks are another approach to try to capture the
relationships between objects detected within an image. This approach was
first used by [16] to solve the VQA task by detecting objects within an image
and then treating each of these objects as nodes within a graph. This approach
uses several iterations of modifying each node’s representation with that of
its neighbors’. This approach produced better results than existing methods at
the time. Extending this approach to the TextVQA task, detected OCR tokens
were added as additional nodes within these graphs.

The Stuctured Multimodal Attention (SMA) approach [5] won the 2020 CVPR
TextVQA challenge using a variation of the standard graph attention described
above. They used various attention mechanisms to create one attended visual
object representation and one attended OCR object representation, which were
then fed into the Multimodal Transformer layers instead of the individual
objects. Importantly, these attention mechanisms involve the use of edge
features between objects, an idea which we implement differently in our GRT. A
limitation of this work is that using edge features for representation
learning before the Transformer layers does not leverage the graph computation
inherent to Transformers. It also does not create contextualized
representations for each object unlike those that can be created using the
Transformer layers.

The Spatially Aware Multimodal Transformers (SAMT) approach was the 2020 CVPR
TextVQA runner-up [12]. They created a spatial relation label for every pair
of objects and restricted each Transformer attention head to attend over
paired objects with certain labels only. A limitation of this work is that
there is no natural way to make use of rich edge features since masking is
based on discrete edge categories, not continuous / vector edge features.

We propose to overcome the limitations of both SMA [5] and SAMT [12] in this
work using a novel Graph Relation Transformer which uses rich edge information
for graph attention computation in the Transformer. Integrating rich edge
information within the Transformer self-attention layer helps us overcome the
limitation of SMA [5]. Continuous space, rich edge features helps us overcome
the limitation of SAMT [12].

###  2.3 Graph Networks outside VQA

Graph networks have also been proposed for tasks outside the VQA domain. Cai
et al. [3] propose a graph transformer for the graph to sequence learning
task. They use a learned vector representation to model the relation between
nodes. Yun et al. [24] propose graph transformer networks based on graph
convolution networks for node classification tasks. Unlike our proposed
approach, they do not use rich edge features between objects based on the
appearance and position of objects in the image which provide valuable
information in image-based tasks like the TextVQA task. There have also been
efforts to generalize the Transformer architecture for arbitrary graphs
related tasks. [24] propose modifications to the original transformer
architecture by leveraging the inductive bias present in graph topologies
while also injecting edge information into the architecture.

##  3 Proposed Approach

(a) Appearance Similarity edge feature

(b) Spatial Translation edge feature

(c) Spatial Interaction Label edge feature

(d) Modality Pair Label edge feature

Figure 2: Illustration of the four edge features we propose to use for the
TextVQA task. For each pair of visual objects $i$ and $j$ in an image, we
extract a vector of edge features $\textbf{e}_{ij}$ by concatenating these
four edge features: appearance_similarity$(i,j)$, translation$(i,j)$,
interaction$(i,j)$ and modality_pair$(i,j)$.

In this section, we motivate and explain GRT, our novel extension to the M4C
model.

###  3.1 Graph Relation Transformer (GRT)

A common theme among the top performing models [5, 12] is extracting and using
graph relations between detected objects in images. However, no prior work in
the VQA domain has used edge features within the Transformer layers for
representation learning. Here, we will describe four types of edge features
with which we will experiment and how they will be used in the Transformer. A
key observation of the Transformer Encoder architecture is that the self-
attention inherently performs graph computation across each input as if the
inputs were structured as a fully-connected graph, as each input attends over
all inputs. However, there is no natural place to inject edge relations in the
Transformer Encoder layer. To achieve this, we propose Graph Relation Self-
Attention in Transformers.

#### Edge features

We hypothesize that four types of edge features will be useful for the model
to better understand the relationships between objects. Each of these features
are defined for a pair of objects (which may be from the visual modality or
the OCR modality). These are (1) Appearance similarity (2) Spatial translation
feature (3) Spatial interaction labels (4) Modality pair labels. Figure 2
illustrates how these are extracted for every pair of visual objects in an
image.

Appearance similarity may be especially useful for the OCR modality, as OCR
tokens in an image that belong together in a sentence or paragraph usually
have similar font and overall appearance. We use the cosine similarity between
the 2,048-dimensional Faster R-CNN embeddings as the (1-dimensional)
appearance similarity feature. This edge feature is illustrated in Figure 2a.

The spatial translation feature from object $i$ to object $j$ is simply a
2-dimensional feature that is the translation from the center of object $i$’s
bounding box to object $j$’s in the $x$ and $y$ directions, where these
translations are normalized to the length and width of the object such that
this feature’s range is [-1, 1] in both directions. Clearly, this feature
provides the model information about the relation between objects in terms of
spatial direction. This edge feature is illustrated in Figure 2b.

Similar to the labels proposed by Kant _et al_. [12], we use 5 mutually
exclusive class labels indicating different types of spatial interaction
[is_self, is_contains, is_in, is_overlap, not_overlap], where is_self
indicates a self-edge between object $i$ to itself, is_contains indicates if
object $i$’s bounding box completely contains object $j$’s, is_in indicates if
object $j$’s bounding box completely contains object $i$’s, is_overlap
indicates if there is any overlap between the bounding boxes of objects $i$
and $j$ and they do not fall under any of the previous classes and not_overlap
indicates if there is no overlap between the two objects’ bounding boxes.222We
do not include the 8 directional labels used by Kant _et al_. [12] as this
information should be provided by the previously mentioned spatial translation
feature. These labels provide the model information about how objects interact
with each other in space. This edge feature is illustrated in Figure 2c.

Following Gao _et al_. [5], we also include modality pair labels
[is_obj_to_obj, is_obj_to_ocr, is_ocr_to_obj, is_ocr_to_ocr] that indicate the
modalities of each pair of objects. For example, the is_obj_to_ocr label is
positive if the ‘self’ object is a visual object recognized by the Faster
R-CNN and the ‘other’ object is an OCR token. These labels will help the model
learn different interactions between modalities. This edge feature is
illustrated in Figure 2d.

Finally, these features are only meaningful for pairs of Transformer inputs
that originate from the image (the visual and OCR modalities). Therefore, when
a question modality Transformer input is involved, we set the aforementioned
features to be zero.

#### Vanilla Transformer Self-Attention

For some object or token $i$, the Transformer [21] self-attention module
computes the attended representation of each of $n_{obj}$ objects as

$\text{Attention}(\textbf{q}_{i},K,V)=\text{softmax}\left(\frac{\textbf{q}_{i}K^{T}}{\sqrt{d_{k}}}\right)V$
(1)

where $\textbf{q}_{i}$, a row vector of dimension $d_{k}$, is the query for
object $i$. $K$ is a matrix composed of $n_{obj}$ stacked row vectors of
dimension $d_{k}$, representing the keys of the $n_{obj}$ objects. $V$ is a
matrix composed of $n_{obj}$ stacked row vectors of dimension $d_{v}$,
representing the values of the $n_{obj}$ objects.

Figure 3: Illustration of how the Graph Relation Transformer (GRT) extends the
‘Vanilla’ Transformer Architecture [21], noting where this module can be
inserted into the M4C model for the TextVQA task. We propose that edge
features $\textbf{e}_{ij}$ between objects $i$ and $j$ can be incorporated
using a fusion function $\phi$ into the keys and/or values of the Transformer
Multi-Head Attention module. $\phi_{concat}$ (concatenation) or $\phi_{add}$
(addition) are possible fusion functions that can be used, for example.
Elements of this illustration are adapted from figures in [10] and [21].

The node features matrix $X$ comprises $n_{obj}$ row vectors of node features
of dimension $d_{in}$ (visualized in Figure 3).333In the first Transformer
layer, $X$ is simply the input features, whereas in subsequent layers, $X$ is
the output of the previous layer. We obtain $K$ and $V$ using projection
matrices $W_{k}$ and $W_{v}$ to project the $n_{obj}$ rows of $X$ to vector
spaces of dimension $d_{k}$ and $d_{v}$ respectively.

$K=XW_{k}$ (2) $V=XW_{v}$ (3)

Notice how there is no place to incorporate edge features and that the same
$K$ and $V$ matrices are used to compute
$\text{Attention}(\textbf{q}_{i},K,V)~{}\forall i$.

#### Graph Relation Transformer Self-Attention

We would like to fuse the $n_{obj}^{2}$ pairwise edge features between each of
the $n_{obj}$ objects into the self-attention module. Suppose we have our edge
feature tensor $E$ of dimension $n_{obj}\times n_{obj}\times d_{e}$, where the
vector $E_{ij}$, which we will henceforth refer to as $\textbf{e}_{ij}$,
represents the $d_{e}$-dimensional edge feature between objects $i$ and $j$
with $i$ as the ‘self’ object and $j$ as the ‘other’ object (the edges are
directional). Figure 3 visualizes this $E$ tensor. Essentially, $E$ is a stack
of $n_{obj}$ matrices, where each of these matrices, $E_{i}$, represents the
$n_{obj}$ sets of edge features where object $i$ is the ‘self’ object. These
features may be any fixed number of dimensions of discrete and/or continuous
variables.

We propose three different places to fuse these features in the Transformer
architecture: (1) in the keys, (2) the values and (3) both the keys and
values. We illustrate this fusion in the table present in Figure 3. Two
central ideas motivate fusion in the keys and the values. On the one hand,
fusing in the keys makes object $i$’s attention weight for object $j$ depend
on $\textbf{e}_{ij}$. This means that the relevance of object $j$ on the
representation of object $i$ should depend on how the two objects are related.
On the other hand, because self-attention ultimately computes a weighted
average of the values, fusing in the values makes the influence of object $j$
on the attended representation of object $i$ depend on $\textbf{e}_{ij}$

We will now define this formally. When we fuse the edge features in the keys,
we now compute a different set of keys, $K_{i}$, for each object $i$ in order
to allow the relations between each object $i$ with all other objects to vary
for different $i$. This is defined in the Equation 4. Similarly, fusing the
edge features in the values via $V_{i}$ is defined in Equation 5. Finally,
Equation 6 defines fusing in both keys and values.

$\text{GraphAttn}_{k}(\textbf{q}_{i},K,V)=\text{softmax}\left(\frac{\textbf{q}_{i}K_{i}^{T}}{\sqrt{d_{k}}}\right)V$
(4)
$\text{GraphAttn}_{v}(\textbf{q}_{i},K,V)=\text{softmax}\left(\frac{\textbf{q}_{i}K^{T}}{\sqrt{d_{k}}}\right)V_{i}$
(5)
$\text{GraphAttn}_{kv}(\textbf{q}_{i},K,V)=\text{softmax}\left(\frac{\textbf{q}_{i}K_{i}^{T}}{\sqrt{d_{k}}}\right)V_{i}$
(6)

Now, in order to obtain the fused keys $K_{i}$, we fuse the object features
$X$ with $E_{i}$ using a fusion function $\phi$ before projecting the
representation to the ‘keys’ vector space.

$K_{i}=\phi(X,E_{i})W_{k}$ (7)

In the same way, we can fuse object features with edge features in the values
by

$V_{i}=\phi(X,E_{i})W_{v}$ (8)

Notice the difference between Equations 2-3 and 7-8. In this work, we propose
two fusion functions: $\phi_{concat}$ and $\phi_{add}$.

$\phi_{concat}(X,E_{i})=[X;E_{i}W_{c}]$ (9)

is the concatenation fusion function where $[;]$ is the concatenation
operation and the $d_{e}\times d_{e^{\prime}}$ matrix $W_{c}$ projects $E_{i}$
to an intermediate vector space of dimension $d_{e^{\prime}}$. We include this
projection to give each Transformer layer the flexibility to use the edge
features differently. However, one could set $W_{c}$ to the identity matrix to
effectively disable this projection.444Clearly, if $W_{c}$ or $W_{a}$ are not
the identity matrix, then the fusion function itself will also contain
parameters / weights to be learned along with the rest of the model. And

$\phi_{add}(X,E_{i})=X+E_{i}W_{a}$ (10)

is the addition fusion function where the $d_{e}\times d_{in}$ matrix $W_{a}$
projects $E_{i}$ to the same space as $X$ so that addition can be performed.

###  3.2 Parameter Learning

The GRT is an extension to the baseline M4C model and the extra parameters we
introduce are trained end-to-end along with the existing parameters using the
same multi-token sigmoid loss as M4C. We define this loss formally in Equation
11, where $N$ is the number of training examples, $M_{i}$ is the number of
tokens in the ground truth answer for example $i$, $y_{ij}$ is the vocabulary
index of the $j^{\text{th}}$ answer token of the $i^{\text{th}}$ example and
$\hat{y}_{ij}$ is the the model’s sigmoid activation value for that token.

$\mathcal{L}=-\frac{1}{N}\sum_{i=1}^{N}\frac{1}{M_{i}}\sum_{j=1}^{M_{i}}({y}_{ij}\log\left(\hat{y}_{ij}\right)+(1-{y}_{ij})\log\left(1-\hat{y}_{ij}\right))$
(11)

The extra parameters introduced by GRT are those used by $\phi$ (i.e. $W_{c}$
or $W_{a}$) inside the Transformer layers.

##  4 Experimental Setup

###  4.1 Dataset

We use the TextVQA dataset [20] for this task since the TextVQA dataset is the
official dataset for the TextVQA task. The TextVQA dataset contains 45,336
questions on 28,408 images where all questions require reading and reasoning
about the text in images. The dataset contains images gathered from the Open
Images dataset [14] and contains images which contain text like traffic signs,
billboards etc. We use the standard TextVQA dataset split for dividing the
dataset into training, validation and testing sets. The training set consists
of 34,602 questions from 21,953 images, the validation set consists of 5000
questions from 3166 images and the test set consists of 5734 questions from
3289 images. The TextVQA dataset contains visual (images) and text (questions)
modalities. In addition to this, the dataset also provides Rosetta [2] OCR
tokens which forms the additional OCR modality in the task. Each question-
image pair has 10 ground-truth answers annotated by humans.

###  4.2 Evaluation Metric

We use the evaluation metric given by the VQA v2.0 challenge [8]. This metric
does some preprocessing before looking for exact matches between the human
annotated answers and the model’s output. This metric averages all of the 10
choose 9 sets of the human annotated answers as they determined this approach
is more robust to inter-human variability in phrasing answers. The evaluation
metric equation is shown below where ‘a’ is the model’s output:

$\operatorname{Acc}(a)=\min\left\\{\frac{\\#\text{ humans \, that \, said \,
{a}}}{3},1\right\\}$ (12)

###  4.3 Baseline model: M4C

We build upon the M4C baseline model for all our experiments. The M4C model
first projects the feature representations of the different entities (question
words, objects, OCR tokens) into a common embedding space. The question words
are embedded using a pre-trained BERT model. The visual objects are
represented using Faster R-CNN features (appearance
($\textbf{x}_{i}^{\mathrm{fr}}$) and location features
($\textbf{x}_{i}^{\mathrm{b}}$) for some object $i$) which are then projected
onto a common embedding space, followed by layer normalization (LN) and
addition as shown in Equation 13:

$\textbf{x}_{i}^{\mathrm{obj}}=\mathrm{LN}\left(W_{1}\textbf{x}_{i}^{\mathrm{fr}}\right)+\mathrm{LN}\left(W_{2}\textbf{x}_{i}^{\mathrm{b}}\right)$
(13)

M4C uses the Rosetta OCR system to recognize OCR tokens in the image along
with their location. For some recognized OCR token $j$, a 300-dim FastText
vector ($\textbf{x}_{j}^{\mathrm{ft}}$), appearance feature from Faster-RCNN
($\textbf{x}_{j}^{\mathrm{fr}}$), a 604-dim Pyramidal Histogram of Characters
(PHOC) vector ($\textbf{x}_{j}^{\mathrm{p}}$) and a 4-dim location vector
($\textbf{x}_{j}^{\mathrm{b}}$) are extracted. Similar to the visual object
features, these are projected to the common embedding space to get the final
OCR feature for each token as shown in Equation 14:

$\textbf{x}_{j}^{\mathrm{ocr}}=\mathrm{LN}\left(W_{3}\textbf{x}_{j}^{\mathrm{ft}}+W_{4}\textbf{x}_{j}^{\mathrm{fr}}+W_{5}\textbf{x}_{j}^{\mathrm{p}}\right)+\mathrm{LN}\left(W_{6}\textbf{x}_{j}^{\mathrm{b}}\right)$
(14)

After embedding all the modalities, they are all passed through a multi-layer
Transformer with multi-headed self-attention. Each entity here can attend to
all entities in the same modality and in other modalities. The output from the
Transformer is then passed through a pointer network to iteratively decode the
answer. At each time step, the pointer network chooses a token from either the
fixed training vocabulary or the OCR tokens by taking in the embedding of the
previously predicted word. The model is trained using a multi-label sigmoid
loss as shown in Equation 11.

###  4.4 Experimental Methodology for the GRT

Since the GRT builds upon the existing M4C baseline, we used the same
initialization method as M4C for each of the parameters within our model. Due
to the limited access of the TextVQA test set evaluation server, we used the
TextVQA evaluation metric to compare each of the methods accuracy on the
validation set. To evaluate and compare the best graph attention fusion
methods and fusion locations, we trained a model for each scenario for 5,000
update steps to determine which of the graph attention setups was the best
before training each model fully to convergence. Once the best graph attention
setup was chosen, this became our Graph Relation Tranformer shown in Table 1.
We trained the GRT method for 24,000 updates for a fair comparison as this was
the number of updates originally used by the M4C authors [10]. Additionally,
we conducted an ablation study in which we removed one of the edge feature
types and trained the model until convergence to provide clarity to the
information content gained by each of the edge feature types.

All our models use 4 Transformer layers with 12 attention heads.555Note that
in previous work [5, 12], simply increasing the number of Transformer layers
has been found to improve performance. We have not reproduced this tweak as it
is not relevant to the ideas we explore in this paper. Answer decoding is done
for a maximum of 12 tokens. On four V100 GPUs, training the baseline M4C model
for 24,000 updates took 5 hours. Adding the computation necessary to include
all our edge features increased this training time to 8 hours when fusing them
in either the keys or values and 12 hours in both. However, the appearance
embedding cosine similarity was a computationally expensive feature and if it
is removed, the time is reduced to 6 hours when fusing edge features in values
only.

##  5 Results and Discussion

Architecture | Opts | Val (%) |  $\Delta$Val (%) | Test (%)
---|---|---|---|---
LoRRA∗ |  | 27.17 | - |
M4C∗ |  | 38.93 | 0 | N/A
GRT∗ |  | 39.58 | 0.65 | 39.58
M4C |  | 39.40 | 0 | 39.01
SMA |  | 39.58 | 0.18 | 40.29
M4C | ✓ | 42.7 | 0 | N/A
SAMT | ✓ | 43.90 | 1.20 | N/A
Table 1:  Performance comparison on the TextVQA validation(’Val’) and
Test(’Test’) sets. $\Delta$Val refers to the improvement of a model’s accuracy
over the M4C model trained with no optimizations other than the architectural
change proposed by the model. Rows marked with * are results we obtained
ourselves by using the MMF [19] framework with default settings. The raw SAMT
accuracy values are due to both their novel architecture as well as other
optimizations (Opts), namely adding 2 extra Transformer layers, using Google
OCR as the OCR module and a ResNext-152 [23] Faster R-CNN model [18] trained
on Visual Genome [15]. The GRT model shown here uses the ‘spatial
translation’, ‘spatial interaction labels’ and ‘modality pair labels’ edge
features and it fuses these edge features in the values of the Multiheaded
Attention Module by addition (the $\phi_{add}$ function).

As seen in Table 1, the Graph Relation Transformer architecture surpasses the
LoRRA and M4C baseline models. We also show results for the SMA [5] and SAMT
[12] models as originally reported by their authors.

Compared to the experimental settings originally used by the M4C authors Hu
_et al_. [10], the authors of SMA and SAMT report results on models that
include not only the novel architectural changes they propose, but also other
optimizations. Namely, SMA uses a custom Rosetta-en OCR module and SAMT uses 2
extra Transformer layers, a Google OCR module and a ResNext R-CNN. Therefore,
we show the improvement in accuracy over the corresponding M4C model trained
with the same optimizations. SMA and SAMT achieve 0.18% and 1.2% accuracy
increases over their corresponding M4C models respectively.

From our experiments, modifying the baseline M4C model with the GRT
architecture alone yields a 0.65% performance increase. This result is from
the best model setup determined from the Transformer fusion exploration and
the edge feature ablation study. This best model fused the edge features into
the values location, used the $\phi_{add}$ fusion function and used all edge
feature types except for the visual similarity edge feature type.

The same model (coincidentally) achieved 39.58% accuracy on both the test set
(evaluated by the official test server) and validation set, submitted as team
cmu_mmml.

###  5.1 Fusion Function for Edge Features

Fusion Location | Fusion function $\phi$ | Accuracy (%)
---|---|---
Keys | $\phi_{concat}$ | 35.51
Keys | $\phi_{add}$ | 35.17
Values | $\phi_{concat}$ | 36.90
Values | $\phi_{add}$ | 37.30
Keys & Values | $\phi_{concat}$ | 31.89
Keys & Values | $\phi_{add}$ | 35.41
Table 2:  Performance on the TextVQA validation set of Graph Relation
Transformer with different methods of fusing the edge features. As the purpose
of these runs was model selection only, training for each model was stopped
after 5,000 steps (before convergence). The fusion functions are defined in
Equations 9-10.

(a)  What word is printed under interior design on the book in the middle?
M4C: para
Graph Relational Transformer: inspirations
Best answer: inspirations

(b)  What kind of establishment is in the background next to the red and white
truck?
M4C: plus
Graph Relational Transformer: bar
Best answer: bar

(c)  What company is on the left side of the screen?
M4C: comerica
Graph Relational Transformer: meijer
Best answer: meijer

Figure 4: Qualitative Analysis: The Figure shows qualitative examples of
illustrative cases comparing the M4C model and the Graph Relational
Transformer. The red boxes have been applied for this figure specifically and
are not present in the original image. ‘Best answer’ refers to the consensus
human answer in the dataset.

To determine the best fusion location and fusion function, we compared all of
the possible combinations shown above in Table 2. The intuition behind the
keys only fusion location is that the importance given to another object would
be affected by the spatial relationships between the objects. The intuition
behind the values only fusion location is that the representation of an object
would change depending on the spatial relationship between the objects. As
shown above, the values only fusion location and the $\phi_{add}$ fusion
function performed the best. This shows that the model was better able to
reason about the image when the model was able to change the representation of
another object based on the context and spatial relationship between the
current object.

###  5.2 Ablation Study on Edge features

Feature set | Accuracy (%)
---|---
All features | 39.40
\- Appearance similarity | 39.58
\- Spatial translation feature | 39.06
\- Spatial interaction labels | 38.78
\- Modality pair labels | 38.96
Table 3:  Ablation Study on Edge Features: Performance on the TextVQA
validation set of Graph Relation Transformer with various groups of features
ablated. Each model fuses edge features at self-attention ‘values’ only, using
‘add’ as the fusion function $\phi$. Each model was trained 24,000 steps (to
convergence).

To evaluate the impact of each of the edge feature types, an ablation study
was done for each edge feature type as shown in Table 3. We observe that while
performance suffered when most of our edge features were dropped (as
expected), the one exception was that when the appearance similarity edge
feature was dropped, performance actually improved. Recall that our appearance
similarity feature is cosine similarity, which is nothing more than a (scaled)
dot product between the visual embeddings from the R-CNN. Our explanation for
why this cosine similarity does not improve performance is that in the
baseline Transformer architecture, the attention module already does a scaled
dot product between (projections of) the visual embeddings. Therefore, the
cosine similarity does not add anything meaningful to the model apart from
introducing noise.

###  5.3 Qualitative Analysis

We further performed a qualitative comparison between the predictions of the
M4C and the Graph Relational Transformer to identify instances that benefited
from injecting object relational features into the transformer. Figure 4 shows
some typical examples which demonstrate the effectiveness of the Graph
Relational Transformer. Figure 4a and Figure 4b provides instances where the
Graph Relational Transformer was better able to reason about the relation
between the OCR and objects in the image to generate the correct answer.
Figure 4c provides one instance where the Graph Relational Transformer was
able to answer questions involving positions better that the M4C baseline.
Here, the spatial translation feature and the modality pair labels give useful
information to the model to answer these kinds of questions as they allow the
model to build a superior representation of the tokens by selectively
interacting with the other tokens based on their relationships.

##  6 Conclusion and Future Directions

In this work we propose a Graph Relation Transformer for TextVQA. Our best
performing model uses the $\phi_{add}$ fusion function at the attention
‘values’, and incorporates all of the edge features except for the visual
similarity edge feature. It outperforms the M4C model due to its improved
spatial reasoning ability. Our quantitative and qualitative results support
our hypothesis that incorporating spatial relationships between objects in the
image lead to better performance. Additionally, most of the errors in the
baseline and our models were due to the OCR system incorrectly detecting words
within the image. Hence, we believe that there is significant performance
headroom from improving the OCR system, as accurate OCR tokens are the first
crucial step in reasoning about the image. Beyond TextVQA, we have noted that
the GRT can be applied to any task where relations between objects can be
represented by vectors. Exploring the TextVQA dataset further, possible future
work includes exploring additional edge feature types and using an improved
OCR system with the GRT architecture. Beyond these ideas, applying the GRT
architecture to different datasets and tasks present additional potential
research directions. We also note that the GRT architecture is generalizable
to any application or task with one or more modalities where relations between
objects are informative. We leave the validation of this hypothesis to future
work.

## References

  * [1] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [2] Fedor Borisyuk, Albert Gordo, and Viswanath Sivakumar.  Rosetta: Large scale system for text detection and recognition in images.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 71–79, 2018.
  * [3] Deng Cai and Wai Lam.  Graph transformer for graph-to-sequence learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7464–7471, 2020.
  * [4] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach.  Multimodal compact bilinear pooling for visual question answering and visual grounding.  arXiv:1606.01847, 2016.
  * [5] Chenyu Gao, Qi Zhu, Peng Wang, Hui Li, Yuliang Liu, Anton van den Hengel, and Qi Wu.  Structured multimodal attentions for textvqa.  arXiv preprint arXiv:2006.00753, 2020.
  * [6] Felix Gers.  Long short-term memory in recurrent neural networks, 2001.
  * [7] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.
  * [8] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904–6913, 2017.
  * [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [10] Ronghang Hu, Amanpreet Singh, Trevor Darrell, and Marcus Rohrbach.  Iterative answer prediction with pointer-augmented multimodal transformers for textvqa.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9992–10002, 2020.
  * [11] Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh.  Pythia v0.1: the winning entry to the vqa challenge 2018, 2018.
  * [12] Yash Kant, Dhruv Batra, Peter Anderson, Alex Schwing, Devi Parikh, Jiasen Lu, and Harsh Agrawal.  Spatially aware multimodal transformers for textvqa, 2020.
  * [13] Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang.  Bilinear attention networks.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 1564–1574. Curran Associates, Inc., 2018.
  * [14] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al.  Openimages: A public dataset for large-scale multi-label and multi-class image classification.  Dataset available from https://github. com/openimages, 2(3):2–3, 2017.
  * [15] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International Journal of Computer Vision, 123(1):32–73, 2017.
  * [16] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu.  Relation-aware graph attention network for visual question answering, 2019\.
  * [17] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  * [18] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  * [19] Amanpreet Singh, Vedanuj Goswami, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh.  Mmf: A multimodal framework for vision and language research.  https://github.com/facebookresearch/mmf, 2020.
  * [20] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [21] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.
  * [22] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly.  Pointer networks.  In Advances in neural information processing systems, pages 2692–2700, 2015.
  * [23] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5987–5995, 2017.
  * [24] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim.  Graph transformer networks.  Advances in Neural Information Processing Systems, 32:11983–11993, 2019.