First, it should be clear that our results generalize from $\lambda$-terms to
the rational fixpoint for endofunctors on $\mathsf{Nom}$ associated to a
binding signature. Other points for future work are: (1) the extension of the
coalgebraic approach to rational and infinitary $\lambda$-terms using nominal
sets to treat the solutions of higher-order recursion schemes as was done in
the setting of presheaves on finite sets in [3], and (2) the study of
specification formats that extend our simple corecursion principle that
follows from finality; more precisely, Bonsague et al. [7, 21] have proposed
bipointed specifications as an abstract format (by restricting Turi’s and
Plotkin’s abstract GSOS rules [24]) to specify algebraic operations on the
rational fixpoint of an endofunctor. It should be interesting to work out a
concrete rule format corresponding to bipointed specifications for rational
$\lambda$-terms and rational terms for arbitrary binding signatures. Last, but
not least, the similarity of the results in [3] on the one hand and those in
[18] and here on the other hand is so striking that there should be a formal
connection; however, to our knowledge this has not been worked out in the
literature yet.

## References

  * [1] Jiří Adámek.  Introduction to coalgebra.  Theory Appl. Categ., 14:157–199, 2005.
  * [2] Jiří Adámek, Stefan Milius, and Jiri Velebil.  Iterative algebras at work.  Math. Structures Comput. Sci, 16(6):1085–1131, 2006.
  * [3] Jiří Adámek, Stefan Milius, and Jiri Velebil.  Semantics of higher-order recursion schemes.  Log. Methods Comput. Sci., 7(1), 2011.
  * [4] Jiří Adámek and Jiří Rosický.  Locally presentable and accessible categories.  Cambridge University Press, 1994.
  * [5] André Arnold and Maurice Nivat.  The metric space of infinite trees. algebraic and topological properties.  Fundam. Inform., 3(4):445–476, 1980.
  * [6] Hendrik Pieter Barendregt.  The Lambda calculus: Its syntax and semantics.  North-Holland, Amsterdam, 1984.
  * [7] Marcello M. Bonsangue, Stefan Milius, and Jurriaan Rot.  On the specification of operations on the rational behaviour of systems.  In Proc. EXPRESS/SOS’12, volume 89 of Electron. Proc. Theoret. Comput. Sci., pages 3–18, 2012.
  * [8] Marcello M. Bonsangue, Stefan Milius, and Alexandra Silva.  Sound and complete axiomatizations of coalgebraic language equivalence.  ACM Trans. Comput. Log., 14(1:7), 2013.
  * [9] Bruno Courcelle.  Fundamental properties of infinite trees.  Theoret. Comput. Sci., 25:95–169, 1983.
  * [10] Calvin C. Elgot.  Monadic computation and iterative algebraic theories.  In H. E. Rose and J. C. Sheperdson, editors, Logic Colloquium ’73, volume 80, pages 175–230, Amsterdam, 1975. North-Holland Publishers.
  * [11] Marcelo Fiore, Gordon D. Plotkin, and Daniele Turi.  Abstract syntax and variable binding.  In Proc. Logic in Computer Science 1999, pages 193–202. IEEE Press, 1999.
  * [12] Murdoch Gabbay and Andrew M. Pitts.  A new approach to abstract syntax involving binders.  In Proc. LICS’99, pages 214–224. IEEE Computer Society Press, 1999\.
  * [13] Peter Gabriel and Friedrich Ulmer.  Lokal präsentierbare Kategorien, volume 221 of Lecture Notes Math. Springer-Verlag, 1971.
  * [14] Fabio Gaducci, Marino Miculan, and Ugo Montanari.  About permutation algebras, (pre)sheaved and names sets.  Higher-Order Symb. Comput., 19:283–304, 2006.
  * [15] Susanna Ginali.  Regular trees and the free iterative theory.  J. Comput. System Sci., 18:228–242, 1979.
  * [16] Bart Jacobs and Jan Rutten.  A tutorial on (co)algebras and (co)induction.  EATCS Bulletin, 62:62–222, 1997.
  * [17] Richard Kennaway, Jan Willem Klop, Ronan Sleep, and Fer-Jan de Vries.  Infinitary Lambda Calculus.  Theoret. Comput. Sci., 175(1):93–125, 1997.
  * [18] Alexander Kurz, Daniela Petrisan, Paula Severi, and Fer-Jan de Vries.  Nominal coalgebraic data types with applications to lambda calculus.  Log. Methods Comput. Sci., 9(4), 2013.
  * [19] Joachim Lambek.  A fixpoint theorem for complete categories.  Math. Z., 103:151–161, 1968.
  * [20] Stefan Milius.  A sound and complete calculus for finite stream circuits.  In Proc. LICS’10, pages 449–458. IEEE Computer Society, 2010.
  * [21] Stefan Milius, Marcello M. Bonsangue, Robert S.R. Myers, and Jurriaan Rot.  Rational operation models.  In Proc. MFPS XXIX, volume 298 of Electron. Notes Theor. Comput. Sci., pages 257–282, 2013.
  * [22] Daniela Petrişan.  Investigations into Algebra and Topology over Nominal Sets.  dissertation, University of Leicester, 2011.
  * [23] Andrew M. Pitts.  Nominal Sets: Names and Symmetry in Computer Science, volume 57 of Cambridge Tracts in Theoretical Computer Science.  Cambridge University Press, 2013.
  * [24] Gordon D. Plotkin and Daniele Turi.  Towards a mathematical operational semantics.  In Proc. Logic in Computer Science (LICS’97), pages 280–291, 1997\.
  * [25] Jan Rutten.  Universal coalgebra: a theory of systems.  Theoret. Comput. Sci., 249(1):3–80, 2000.
  * [26] Jan Rutten.  Rational streams coalgebraically.  Log. Methods Comput. Sci., 4(3:9):22 pp., 2008.
  * [27] Thorsten Wißmann.  The rational fixed point in nominal sets and its application to infinitary lambda-calculus.  Project report, available at http://thorsten-wissmann.de/theses/project-wissmann.pdf, October 2014.

##  Appendix A Proof of Lemma 2.5

######  Proof A.1.

Assuming $\pi\cdot x_{1}=x_{2}$, the equivariance of
$\operatorname{\textsf{supp}}$ provides the equation

$\operatorname{\textsf{supp}}(x_{1})=\operatorname{\textsf{supp}}(\pi\cdot
x_{2})=\pi\cdot\operatorname{\textsf{supp}}(x_{2})=\\{\pi(v)\mid
v\in\operatorname{\textsf{supp}}(x_{2})\\}.$

Since $\pi$ is a bijective map this establishes the desired bijection between
$\operatorname{\textsf{supp}}(x_{1})$ and
$\operatorname{\textsf{supp}}(x_{2})$.

##  Appendix B Proof of Lemma 2.6

######  Proof B.1.

Consider the set

$N=\\{\pi\cdot x\mid\pi\in\mathfrak{S}(\mathcal{V})\text{ and
}\operatorname{\textsf{supp}}(\pi\cdot x)=\operatorname{\textsf{supp}}(x)\\}$

All $\pi$ with $\pi\cdot x\in N$ have to fulfill the property
$\operatorname{\textsf{supp}}(x)=\operatorname{\textsf{supp}}(\pi\cdot
x)=\pi\cdot\operatorname{\textsf{supp}}(x)$. This means, $\pi$ restricts to
$\operatorname{\textsf{supp}}(x)$, i.e.
$\pi(v)\in\operatorname{\textsf{supp}}(x)$ iff
$v\in\operatorname{\textsf{supp}}(x)$. Hence, each such $\pi$ can be
factorized into two permutations $f_{\pi},g_{\pi}\in\mathfrak{S}(\mathcal{V})$
where

$f_{\pi}(v)=\begin{cases}\pi(v)&\text{if
}v\in\operatorname{\textsf{supp}}(x)\\\ v&\text{if
}v\not\in\operatorname{\textsf{supp}}(x)\\\ \end{cases},\quad
g_{\pi}(v)=\begin{cases}v&\text{if }v\in\operatorname{\textsf{supp}}(x)\\\
\pi(v)&\text{if }v\not\in\operatorname{\textsf{supp}}(x)\\\ \end{cases}$

and with $f_{\pi}\circ g_{\pi}=g_{\pi}\circ f_{\pi}=\pi$. Since
$\operatorname{\textsf{supp}}(x)$ supports $x$ we have $g_{\pi}\cdot x=x$, and
this implies

$\displaystyle N$ $\displaystyle=\\{f_{\pi}\cdot g_{\pi}\cdot
x\mid\pi\in\mathfrak{S}(\mathcal{V})\text{ and
}\pi\cdot\operatorname{\textsf{supp}}(x)=\operatorname{\textsf{supp}}(x)\\}$
$\displaystyle=\\{\mathrlap{f_{\pi}\cdot x}\phantom{f_{\pi}\cdot g_{\pi}\cdot
x}\mid\pi\in\mathfrak{S}(\mathcal{V})\text{ and
}\pi\cdot\operatorname{\textsf{supp}}(x)=\operatorname{\textsf{supp}}(x)\\}.$

The set of permutations $\\{f_{\pi}\mid\pi\cdot x\in N\\}$ has cardinality
$|\operatorname{\textsf{supp}}(x)|!$, hence so is $N$.

##  Appendix C Proof of Lemma 2.7

######  Proof C.1.

First there are only finitely many elements whose support is exactly $W$. If
there is no element in $\mathcal{O}$ with support $W$, then we are done. And
if $x$ in $\mathcal{O}$ satifies $\operatorname{\textsf{supp}}(x)=W$, there
are only finitely many elements in $\mathcal{O}$ with the same support by 2.6.

The statement of the lemma now follows since $W$ has only finitely many
subsets and for each $W^{\prime}\subseteq W$ there are only finitely many
elements with support $W^{\prime}$.

##  Appendix D Proof of Proposition 3.1

######  Proof D.1.

Take a tree $t\in\varrho L$ and an orbit-finite subcoalgebra $(S,s)$ of
$\varrho L$ with $t\in S$. Every subtree $t^{\prime}$ of $t$ has support

$\operatorname{\textsf{supp}}(t^{\prime})\subseteq\operatorname{\textsf{supp}}(t).$

But there are only finitely many $t^{\prime}\in S$ with that property; this
follows from 2.7 using $W=\operatorname{\textsf{supp}}(t)$ and that $S$ is
orbit-finite. Hence, $t$ has only finitely many subtrees, i.e. $t$ is
rational. So $\varrho L$ contains only rational $\lambda$-trees. Conversely,
for each rational $\lambda$-tree $t$ we can construct an orbit-finite
$L$-coalgebra by taking the subtrees of $t$ as the carrier set of that
coalgebra and with the coalgebra structure given by decomposing trees at the
root.

##  Appendix E Proof of Theorem 3.2

Let $m$ be the maximal number of free variables in any element of $X$, i.e.

$m:=\max_{x\in X}\big{|}\operatorname{\textsf{supp}}(x)\big{|}.$

The maximum exists by 2.5 since $X$ is orbit-finite.

Let $W\subseteq\mathcal{V}$ be some set of $m+1$ variables containing
$\operatorname{\textsf{supp}}(\operatorname{\mathsf{root}})$. Hence

for all $x\in X$ there exists a $w\in W$ with $w\mathbin{\\#}x$. (E.1)

In the following, we will construct a rational ${\lambda}$-tree in the
$\alpha$-equivalence class of $a^{\dagger}(\operatorname{\mathsf{root}})$.
First, define an $L$-coalgebra
$C\xrightarrow{c}LC=\mathcal{V}+\mathcal{V}\times C+C\times C$ in
$\mathsf{Set}$:

$C:=\\{x\in X\mid\operatorname{\textsf{supp}}(x)\subseteq W\\},\quad
c(x)=\begin{cases}w&\text{if }a(x)=w\in W\subseteq\mathcal{V}\\\
(\ell,r)&\text{if }a(x)=(\ell,r)\in X\times X\\\ (w,y)&\text{if
}a(x)={\left<v\right>}y^{\prime}\text{ and }y=(v\ w)y^{\prime}\\\ &\text{for
some }w\in W\setminus\operatorname{\textsf{supp}}(x)\end{cases}$

We verify that $c$ is well-defined, i.e., its image lies in $LC$:

  * •

For the case $a(x)$ in $\mathcal{V}$, $a(x)$ is also in $W$ because $x\in C$
and thus

$\operatorname{\textsf{supp}}(a(x))\subseteq\operatorname{\textsf{supp}}(x)\subseteq
W.$

  * •

For the case $a(x)=(\ell,r)$ in $X\times X$,

$\operatorname{\textsf{supp}}(\ell)\cup\operatorname{\textsf{supp}}(r)\subseteq\operatorname{\textsf{supp}}(x)\subseteq
W.$

So $\ell,r\in C$.

  * •

For $a(x)={\left<v\right>}y^{\prime}\in[\mathcal{V}]X$, we know by (E.1), that
there is such a fresh $w\in W$. So
${\left<w\right>}y={\left<v\right>}y^{\prime}$. In particular, $y\in C$,
because

$\operatorname{\textsf{supp}}(y)\subseteq\\{w\\}\cup\operatorname{\textsf{supp}}(a(x))\subseteq\\{w\\}\cup\operatorname{\textsf{supp}}(x)\subseteq\\{w\\}\cup
W\subseteq W.$

The last item gives that ${\left<w\right>}y={\left<v\right>}y^{\prime}$, hence
the following diagram commutes in $\mathsf{Set}$:

${C}$${LC}$${LX}$${X}$${L_{\alpha}X}$${c}$${i}$${Li}$${q_{X}}$${a}$ (E.2)

Observe that $C$ is finite, because $X$ is orbit-finite and within an orbit
there are only finitely many elements with a support contained in $W$ by 2.7.
Let $c^{\dagger}$ denote the unique $L$-coalgebra morphism into the final
$L$-coalgebra in $\mathsf{Set}$. Since $C$ is finite, we know that
$c^{\dagger}:C\to\nu L$ factors through the rational fixpoint, i.e. for every
$x\in C$, $c^{\dagger}(x)$ is a rational $\lambda$-tree. In particular,
$c^{\dagger}(\operatorname{\mathsf{root}})$ is a rational $\lambda$-tree.
Using (E.2) and the _final chains_ of the set functor $L$ and the functor
$L_{\alpha}$ on $\mathsf{Nom}$ we shall prove below that

$[c^{\dagger}(x)]_{\alpha}=a^{\dagger}(x)\qquad\text{for all $x\in C\subseteq
X$},$ (E.3)

where $[-]_{\alpha}$ denotes $\alpha$-equivalence classes. For
$x=\operatorname{\mathsf{root}}$ this gives that
$c^{\dagger}(\operatorname{\mathsf{root}})$ is the desired rational
$\lambda$-tree in $a^{\dagger}(\operatorname{\mathsf{root}})$ and thus the
$\alpha$-equivalence class $a^{\dagger}(\operatorname{\mathsf{root}})$ is
rational.

We will now prove Equation (E.3). Before we can proceed to the proof we need
to recall some technical background (cf. [18]). First recall that every
endofunctor $F$ on a category $\mathcal{C}$ with a final object $1$ induces
its final chain

$1\leftarrow F1\leftarrow FF1\leftarrow\cdots$

Whenever the limit of this chain exists in $\mathcal{C}$ and $F$ preserves
this limit, then it carries the final $F$-coalgebra $\nu F$. Moreover, every
$F$-coalgebra $c:C\to FC$ induces its _canonical cone_ $c^{n}:C\to F^{n}1$,
$n<\omega$, on the final chain defined inductively as: $c^{0}:C\to 1$ is the
unique map, and given $c^{n}$ we define

$c^{n+1}=C\xrightarrow{c}FC\xrightarrow{Fc^{n}}FF^{n}1=F^{n+1}1\qquad\text{for
all $n>0$}.$

The unique $F$-coalgebra homomorphism $c^{\dagger}:C\to\nu F$ is then induced
by the universal property of the limit – $c^{\dagger}$ is the unique morphism
such that

$c^{n}=(C\xrightarrow{c^{\dagger}}\nu
F\xrightarrow{p_{n}}F^{n}1)\qquad\text{for every $n<\omega$,}$ (E.4)

where $p_{n}:\nu F\to F^{n}1$ denote the limit projections.

Now let us turn to our concrete functors of interest. The terminal coalgebra
for the functor $L:\mathsf{Set}\to\mathsf{Set}$ consists of all finite and
infinite $\lambda$-trees. Since $L$ preserves all limits, $\nu L$ is the limit
of its final chain $L^{n}1$ and we denote the limit projections by

$p_{n}:\nu L\to L^{n}1$

and the canonical cone induced by $c:C\to LC$ by

$c^{n}:C\to L^{n}1.$

We also consider the final chain of the functor $L_{\alpha}$ on
$\mathsf{Nom}$, and we take its limit in $\lim L_{\alpha}^{n}1$ in
$\mathsf{Set}$ with the projections

$p_{\alpha,n}:\lim L_{\alpha}^{n}1\to L_{\alpha}^{n}1,\qquad\text{for
$n<\omega$}.$

Notice that this limit does _not_ carry the final $L_{\alpha}$-coalgebra. But
$\nu L_{\alpha}$ is a subset of the limit via the unique map
$\iota_{\alpha}:\nu L_{\alpha}\to\lim L_{\alpha}^{n}1$ induced by the
canonical cone $(t_{\alpha}^{n})_{n<\omega}$ induced by the structure
$t_{\alpha}:\nu L_{\alpha}\to L_{\alpha}(\nu L_{\alpha})$ of the final
coalgebra, i.e., $\iota_{\alpha}$ is the unique map such that

$t_{\alpha}^{n}=(\nu L_{\alpha}\xrightarrow{\iota_{\alpha}}\lim
L_{\alpha}^{n}1\xrightarrow{p_{\alpha,n}}L_{\alpha}^{n}1)\qquad\text{for all
$n<\omega$.}$ (E.5)

We will also use the canonical cone

$a^{n}:X\to L_{\alpha}^{n}1,\qquad n<\omega,$

of our given $L_{\alpha}$-coalgebra $(X,a)$.

As a final ingredient we need to relate the final chains of $L$ and
$L_{\alpha}$. To this end observe that $L$ can be construed as an endofunctor
on $\mathsf{Nom}$ (more precisely, $L$ lifts to $\mathsf{Nom}$), and the we
have a quotient natural transformation $q:L\to L_{\alpha}$ with components

$q_{X}=(LX=\mathcal{V}+\mathcal{V}\times X+X\times
X\xrightarrow{\text{id}+\theta_{X}+\text{id}\times\text{id}}\mathcal{V}+[\mathcal{V}]X+X=L_{\alpha}X,$

where $\theta_{X}:\mathcal{V}\times X\to[\mathcal{V}]X$ is the natural
transformation given by the canonical quotient maps (cf. Definition 2.8).
Using $q$ one can define a natural transformation $[-]_{\alpha}^{n}:L^{n}1\to
L_{\alpha}^{n}1$, $n<\omega$, between the respective final chains by
induction: $[-]_{\alpha}^{0}$ is the identity on $1$ and

$[-]_{\alpha}^{n+1}=L^{n+1}1=LL^{n}1\xrightarrow{L[-]_{\alpha}^{n}}LL_{\alpha}^{n}1\xrightarrow{q_{L_{\alpha}^{n}1}}L_{\alpha}L_{\alpha}^{n}1=L_{\alpha}^{n+1}1.$

Now we regard the two final chains and the natural transformation given by the
$[-]_{\alpha}^{n}$ in $\mathsf{Set}$ and use the universal property of the
limit of the $L_{\alpha}^{n}1$ to obtain a unique map $[-]_{\alpha}:\nu
L\to\lim L_{\alpha}^{n}1$ such that the following squares commute for all
$n<\omega$:

${L^{n}1}$${\nu L}$${L_{\alpha}^{n}1}$${\lim
L_{\alpha}^{n}1}$${[-]_{\alpha}^{n}}$${p_{n}}$${[-]_{\alpha}}$${p_{\alpha,n}}$
(E.6)

We denote by $i:C\hookrightarrow X$ the inclusion map and consider the
following diagram:

${\nu L}$${\Lambda^{\infty}_{\mathsf{ffv}}}$${C}$${\lim L_{\alpha}^{n}1}$${\nu
L_{\alpha}}$${X}$${[-]_{\alpha}}$${c^{\dagger}}$${i}$${\iota_{\alpha}}$${a^{\dagger}}$
(E.7)

The set $\Lambda^{\infty}_{\mathsf{ffv}}$ consists of all $\lambda$-trees with
finitely many free variables, and this makes the left-hand square above the
pullback of $[-]_{\alpha}$ along $\iota_{\alpha}$ as proved in [18,
Proposition 5.33]. There it was also shown that the map opposite
$[-]_{\alpha}$ is surjective (see [18, Theorem 5.34]), and it follows that
$\nu L_{\alpha}$ consists of all equivalence classes modulo
$\alpha$-equivalence of $\lambda$-trees in $\Lambda^{\infty}_{\mathsf{ffv}}$.

Now note that once we prove that the outside of Diagram (E.7) commutes we
obtain the dashed map making the diagram commutative. This finishes our proof
since the right-hand square of (E.7) is precisely our desired equation (E.3).
We now establish the commutativity of the outside of the diagram by proving
inductively that

$p_{\alpha,n}\cdot\iota_{\alpha}\cdot a^{\dagger}\cdot
i=p_{\alpha,n}\cdot[-]_{\alpha}\cdot c^{\dagger},\qquad\text{for all limit
projections $p_{\alpha,n}$, $n<\omega$}.$

The base case is clear and for the induction step we compute:

$\begin{array}[]{rcl@{\qquad}p{8cm}}p_{\alpha,n+1}\cdot\iota_{\alpha}\cdot
a^{\dagger}\cdot i&=&t_{\alpha}^{n+1}\cdot a^{\dagger}\cdot
i&by\leavevmode\nobreak\ \eqref{eq:iota}\\\ &=&L_{\alpha}t_{\alpha}^{n}\cdot
t_{\alpha}\cdot a^{\dagger}\cdot i&definition of $t_{\alpha}^{n+1}$\\\
&=&L_{\alpha}t_{\alpha}^{n}\cdot L_{\alpha}a^{\dagger}\cdot a\cdot
i&$a^{\dagger}$ coalgebra homomorphism\\\ &=&L_{\alpha}t_{\alpha}^{n}\cdot
L_{\alpha}a^{\dagger}\cdot q_{X}\cdot Li\cdot c&see\leavevmode\nobreak\
\eqref{eq:coalgplusbind}\\\ &=&q_{L_{\alpha}^{n}1}\cdot Lt_{\alpha}^{n}\cdot
La^{\dagger}\cdot Li\cdot c&$q$ natural and $t_{\alpha}^{n}$, $a^{\dagger}$
equivariant\\\ &=&q_{L_{\alpha}^{n}1}\cdot Lp_{\alpha,n}\cdot
L\iota_{\alpha}\cdot La^{\dagger}\cdot Li\cdot c&by\leavevmode\nobreak\
\eqref{eq:iota}\\\ &=&q_{L_{\alpha}^{n}1}\cdot
L(p_{\alpha,n}\cdot\iota_{\alpha}\cdot a^{\dagger}\cdot i)\cdot
c&functoriality of $L$\\\ &=&q_{L_{\alpha}^{n}1}\cdot
L(p_{\alpha,n}\cdot[-]_{\alpha}\cdot c^{\dagger})\cdot c&induction
hypothesis\\\ &=&q_{L_{\alpha}^{n}1}\cdot Lp_{\alpha,n}\cdot
L[-]_{\alpha}\cdot Lc^{\dagger}\cdot c&functoriality of $L$\\\
&=&q_{L_{\alpha}^{n}1}\cdot L[-]_{\alpha}^{n}\cdot Lp_{n}\cdot
Lc^{\dagger}\cdot c&by\leavevmode\nobreak\ \eqref{eq:eqa}\\\
&=&q_{L_{\alpha}^{n}1}\cdot L[-]_{\alpha}^{n}\cdot Lc^{n}\cdot
c&by\leavevmode\nobreak\ \eqref{eq:cn}\\\ &=&[-]_{\alpha}^{n+1}\cdot
Lc^{n}\cdot c&definition of $[-]_{\alpha}^{n+1}$\\\ &=&[-]_{\alpha}^{n+1}\cdot
c^{n+1}&definition of $c^{n+1}$\\\ &=&[-]_{\alpha}^{n+1}\cdot p_{n+1}\cdot
c^{\dagger}&by\leavevmode\nobreak\ \eqref{eq:cn}\\\
&=&p_{\alpha,n+1}\cdot[-]_{\alpha}\cdot c^{\dagger}&by\leavevmode\nobreak\
\eqref{eq:eqa}\end{array}$

This completes the proof of the desired equation (E.3).∎


# Geometric Interaction Augmented Graph Collaborative Filtering

Yiding Zhang<EMAIL_ADDRESS>MicrosoftBeijingChina ,  Chaozhuo Li
<EMAIL_ADDRESS>Microsoft Research AsiaBeijingChina ,  Senzhang Wang
<EMAIL_ADDRESS>Central South UniversityChangshaChina and  Jianxun Lian,
Xing Xie<EMAIL_ADDRESS>MicrosoftBeijingChina

###### Abstract.

Graph-based collaborative filtering is capable of capturing the essential and
abundant collaborative signals from the high-order interactions, and thus
received increasingly research interests. Conventionally, the embeddings of
users and items are defined in the Euclidean spaces, along with the
propagation on the interaction graphs. Meanwhile, recent works point out that
the high-order interactions naturally form up the tree-likeness structures,
which the hyperbolic models thrive on. However, the interaction graphs
inherently exhibit the hybrid and nested geometric characteristics, while the
existing single geometry-based models are inadequate to fully capture such
sophisticated topological patterns. In this paper, we propose to model the
user-item interactions in a hybrid geometric space, in which the merits of
Euclidean and hyperbolic spaces are simultaneously enjoyed to learn expressive
representations. Experimental results on public datasets validate the
effectiveness of our proposal.

††conference: Woodstock ’18: ACM Symposium on Neural Gaze Detection; June
03–05, 2018; Woodstock, NY††booktitle: Woodstock ’18: ACM Symposium on Neural
Gaze Detection, June 03–05, 2018, Woodstock, NY††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06

##  1\. Introduction

Personalized recommendation (Isinkaye et al., 2015) has been widely deployed
to alleviate the challenge of information overload, which aims to predict
whether a user will interact with an item. Collaborative filtering (CF) tries
to address this problem on the basis of assuming behaviorally similar users
would exhibit similar preference on items (Su and Khoshgoftaar, 2009). As a
classic recommendation paradigm, CF has been widely used in a myriad of
scenarios such as E-commerce (Schafer et al., 2001; Sarwar et al., 2000) and
social recommendations (Tang et al., 2013).

In order to exploit abundant collaborative signals, recent CF works (Wang et
al., 2019; He et al., 2020) propose to incorporate the high-order connectivity
on the user-item interaction graphs, which contributes to facilitating the
recommendation performance. As shown in Figure 1, target user $u_{1}$
perceives high-order interactions through the $l$ stacked order-wise
propagation layers. The high-order interactions of user $u_{1}$ can be
naturally extended to a tree-likeness structure in Figure 1(b) based on the
interactions in Figure 1(a) . This is reasonable as the receptive field tends
to be exponentially larger in the higher orders (Sankar et al., 2021; Zhang et
al., 2022).

Conventional CF models usually formulate the learning of user and item
representations in the Euclidean spaces (Koren et al., 2022; He et al., 2017;
Wang et al., 2022). However, the Euclidean spaces may have a large distortion
when learning graphs with tree-likeness structures as shown in Figure 1(b),
while hyperbolic spaces can achieve better performance with much lower
distortions (Ganea et al., 2018). The reason lies in that hyperbolic spaces
expand faster than Euclidean spaces. Considering a disk in a 2-dimensional
hyperbolic space with constant curvature $-1$, the perimeter and area of the
disk of hyperbolic radius $r$ are given as $2\pi\sinh r$ and $2\pi(\cosh r-1)$
respectively, and both of them grow exponentially with $r$. Inspired by such
merits, some researchers propose to design CF models in hyperbolic spaces
(Vinh Tran et al., 2020; Feng et al., 2020). HyperMF (Vinh Tran et al., 2020)
explores metric learning in hyperbolic spaces for recommendation. HME (Feng et
al., 2020) studies the next-POI recommendation in hyperbolic spaces, which
captures sequential transition, user preference, category and region
information in a shared hyperbolic space.

(a) User-item relation

(b) Tree-likeness interaction graph for $u_{1}$

Figure 1. An illustration of the user-item interactions and the tree-likeness
structures.

Although hyperbolic-based models demonstrate promising performance, such
single geometry-based approaches might be insufficient to fully capture the
hybrid and nested geometrical structures in the interaction graphs. As shown
in 1(b), there exist several circles such as $u_{1}\rightarrow
i_{1}\rightarrow u_{3}\rightarrow i_{3}\rightarrow u_{2}\rightarrow
i_{2}\rightarrow u_{1}$. Such cyclic structures are prevalent in the
interaction graph since they preserve the crucial collaborative signals. The
cyclic and tree structures are nested together, leading to a hybrid geometric
graph. Single geometry-based approaches only capture partial information and
thus cannot depict user preferences from the holistic perspective.

In this paper, we propose a novel geometric graph collaborative filtering
(GGCF) algorithm, which leverages both Euclidean spaces and hyperbolic spaces
to encode the user-item interaction graphs on the basis of graph convolutional
networks (GCN). The graph convolution layer of GGCF is capable of learning the
user and item representations in both Euclidean and hyperbolic spaces
simultaneously. The cross-geometry features are further integrated via an
interaction learning module to adjust the conformal structural information.
GGCF predicts the user-item relations based on the results in both Euclidean
and hyperbolic spaces. Therefore, our proposal organically enjoys the merits
of Euclidean and hyperbolic spaces to advance the model performance.

To summarize, we make the following main contributions:

  * •

We propose to leverage Euclidean and hyperbolic spaces simultaneously in
recommender systems, and design a novel geometry graph collaborative filtering
method to better capture the complex geometric patterns in interaction graphs.

  * •

Experimental results compared with state-of-the-art baselines verify the
effectiveness of the proposed method.

##  2\. Preliminaries

###  2.1. Hyperbolic geometry

Hyperbolic geometry is a non-Euclidean geometry with a constant negative
curvature. The hyperboloid model, as a typical equivalent model describing the
hyperbolic geometry, has been widely investigated (Nickel and Kiela, 2018;
Chami et al., 2019; Liu et al., 2019; Law et al., 2019). Since the basic
operations in the hyperbolic spaces are different from those in the Euclidean
spaces, here we brief introduce some hyperbolic operations used in this paper.
Let $\mathbf{x},\mathbf{y}\in\mathbb{R}^{n+1}$, then the Lorentzian scalar
product is defined as:
$\langle\mathbf{x},\mathbf{y}\rangle_{\mathcal{L}}:=-x_{0}y_{0}+\sum_{i=1}^{n}x_{i}y_{i}.$
We denote $\mathbb{H}^{n}$ as the $n$-dimensional hyperboloid manifold with
constant negative curvature $-1$:
$\mathbb{H}^{n}:=\\{\mathbf{x}\in\mathbb{R}^{n+1}:\langle\mathbf{x},\mathbf{x}\rangle_{\mathcal{L}}=-1,x_{0}>0\\}.$

The tangent space is a useful to perform Euclidean operations undefined in
hyperbolic spaces. The tangent space at $\mathbf{x}$ is defined as a
$n$-dimensional vector space approximating $\mathbb{H}^{n}$ around
$\mathbf{x}$:

(1)
$\mathcal{T}_{\mathbf{x}}\mathbb{H}^{n}:=\\{\mathbf{v}\in\mathbb{R}^{n+1}:\langle\mathbf{v},\mathbf{x}\rangle_{\mathcal{L}}=0\\}.$

Besides, for $\mathbf{v,w}\in\mathcal{T}_{\mathbf{x}}\mathbb{H}^{n}$, a
Riemannian metric tensor is given as
$g_{\mathbf{x}}(\mathbf{v,w}):=\langle\mathbf{v,w}\rangle_{\mathcal{L}}$. Then
the hyperboloid model is defined as the hyperboloid manifold $\mathbb{H}^{n}$
equipped with the Riemannian metric tensor $g_{\mathbf{x}}$. The mapping
between hyperbolic spaces and tangent spaces can be implemented by exponential
map and logarithmic map. The exponential map is a map from subset of a tangent
space of $\mathbb{H}^{n}$ (i.e., $\mathcal{T}_{\mathbf{x}}\mathbb{H}^{n}$) to
$\mathbb{H}^{n}$ itself. The logarithmic map is the reverse map that maps back
to the tangent space. For points
$\mathbf{x,y}\in\mathbb{H}^{n},\mathbf{v}\in\mathcal{T}_{\mathbf{x}}\mathbb{H}^{n}$,
such that $\mathbf{v}\neq\mathbf{0}=(1,0,\cdots,0)$ and
$\mathbf{x}\neq\mathbf{y}$, exponential map $\exp_{\mathbf{x}}(\cdot)$ and
logarithmic map $\log_{\mathbf{x}}(\cdot)$ are defined as follows:

(2) $\displaystyle\exp_{\mathbf{x}}(\mathbf{v})$
$\displaystyle=\cosh({\|\mathbf{v}\|_{\mathcal{L}}})\mathbf{x}+\sinh({\|\mathbf{v}\|_{\mathcal{L}}})\frac{\mathbf{v}}{\|\mathbf{v}\|_{\mathcal{L}}},$
(3) $\displaystyle\log_{\mathbf{x}}^{\beta}(\mathbf{y})$
$\displaystyle=d_{\mathbb{H}}(\mathbf{x,y})\frac{\mathbf{y}+\langle\mathbf{x,y}\rangle_{\mathcal{L}}\mathbf{x}}{\|\mathbf{y}+\langle\mathbf{x,y}\rangle_{\mathcal{L}}\mathbf{x}\|_{\mathcal{L}}},$

where
$\|\mathbf{v}\|_{\mathcal{L}}=\sqrt{\langle\mathbf{v},\mathbf{v}\rangle_{\mathcal{L}}}$
denotes Lorentzian norm of $\mathbf{v}$ and $d_{\mathbb{H}}(\cdot,\cdot)$ is
the intrinsic distance function between two points
$\mathbf{x},\mathbf{y}\in\mathbb{H}^{n}$:

(4) $d_{\mathbb{H}}(\textbf{x},\textbf{y})=\rm{\
arcosh}(-{\langle\mathbf{x},\mathbf{y}\rangle_{\mathcal{L}}}).$

##  3\. Methodology

Figure 2. An illustration of the proposed model architecture.

Figure 2 illustrates the framework of the proposed GGCF model, which includes
the following three modules:

  * •

Geometric feature propagation. This module models the interactions between
users and items by feeding these interactions into simplified graph
convolutional layers, which can learn the graph representations in both
hyperbolic and Euclidean spaces.

  * •

Dual geometry interaction. This module fuses the features in hyperbolic and
Euclidean spaces with each other, to combine different characteristic in
different spaces.

  * •

Layers fusion and prediction. Each layer has its own features of the user and
items. Here we fuse the features in different layers to obtain more powerful
representations, and predict the relations of users and items.

###  3.1. Geometric feature propagation

Graph convolutional networks have shown their powerful ability in recommender
systems (He et al., 2020). Inspired by LightGCN (He et al., 2020) which
observes feature transformation and nonlinear activation in GCNs could be
burdensome for collaborative filtering, we adopt the simple neighbor
aggregator in our method. For $k$-th graph convolutional layer, given input
embedding $\mathbf{e}_{u}^{\kappa,(k)}$ and $\mathbf{e}_{i}^{\kappa,(k)}$ for
user $u$ and item $i$ in Euclidean $\mathbb{R}$ or hyperbolic $\mathbb{H}$
spaces ($\kappa\in\\{\mathbb{R},\mathbb{H}\\}$), we obtain embeddings by
aggregating the features of neighbors:
$\mathbf{h}_{u}^{\kappa,(k)}=\text{Agg}^{\kappa}_{i\in\mathcal{N}_{u}}(w_{ui},\mathbf{e}_{i}^{\kappa,(k)}),\mathbf{h}_{i}^{\kappa,(k)}=\text{Agg}^{\kappa}_{u\in\mathcal{N}_{i}}(w_{ui},\mathbf{e}_{u}^{\kappa,(k)}),$
where $N_{u}$, $N_{i}$ are the set of neighbors of $u$ and $i$, respectively,
and $w_{ui}={1}/({\sqrt{|\mathcal{N}_{u}|}\sqrt{|\mathcal{N}_{i}|}})$ is
denoted as the weight between $u$ and $i$. The aggregator
$\text{Agg}^{\kappa}$ in Euclidean space is defined as Euclidean weighted
mean:
$\text{Agg}^{\mathbb{R}}_{i\in\mathcal{N}_{u}}(w_{ui},\mathbf{e}_{u}^{\mathbb{R},(k)})=\sum_{i\in\mathcal{N}_{u}}w_{ui}\mathbf{e}_{i}^{\mathbb{R},(k)}=\mathbf{h}_{u}^{\mathbb{R},(k)}.$
The weighted mean in non-Euclidean space is defined as Fréchet mean (Fréchet,
1948; Karcher, 1987, 2014), which minimizes an expectation of (squared)
distances with a set of points. However, Fréchet mean does not have a closed
form solution, and it cannot be efficiently calculated by stochastic gradient
descent. Thus, we leverage an elegant neighborhood aggregation method based on
the centroid (Law et al., 2019):

(5)
$\text{Agg}^{\mathbb{H}}_{i\in\mathcal{N}_{u}}(w_{ui},\mathbf{e}_{i}^{\mathbb{H},(k)})=\frac{\sum_{i\in\mathcal{N}_{u}}w_{ui}\mathbf{e}_{i}^{\mathbb{H},(k)}}{|\|\sum_{i\in\mathcal{N}_{u}}w_{ui}\mathbf{e}_{i}^{\mathbb{H},(k)}\|_{\mathcal{L}}|}=\mathbf{h}_{i}^{\mathbb{H},(k)}.$

Thus, the features $\mathbf{h}_{i}^{\mathbb{R},(k)}$ and
$\mathbf{h}_{i}^{\mathbb{H},(k)}$ can be easily obtained by aggregating their
neighbors.

###  3.2. Dual geometry interaction

The geometric feature propagation is built upon the graph topology. Here we
aim to verify that the user/item features are consistent with the dual feature
spaces to promote interaction between Euclidean and hyperbolic geometries. A
straight-forward strategy is to transform the features between hyperbolic and
Euclidean spaces and compute the average features directly. However, this
strategy cannot preserve the difference between features in these two types of
spaces, and thus we should fuse the features in an adaptive manner. Therefore,
we propose to adjust the features in different spaces based on their
similarities. Specifically, given Euclidean feature
$\mathbf{h}_{i}^{\mathbb{R},(k)}$ and hyperbolic feature
$\mathbf{h}_{i}^{\mathbb{H},(k)}$ of item $i$, we leverage their distance to
evaluate their similarities and fuse the features as:

(6)
$\begin{split}\mathbf{f}_{i}^{\mathbb{R},(k)}&=\mathbf{h}_{i}^{\mathbb{R},(k)}+\big{(}\gamma
d_{\mathbb{R}}(\mathbf{h}_{i}^{\mathbb{R},(k)},\log_{\mathbf{0}}(\mathbf{h}_{i}^{\mathbb{H},(k)}))\times\log_{\mathbf{0}}(\mathbf{h}_{i}^{\mathbb{H},(k)})\big{)},\\\
\mathbf{f}_{i}^{\mathbb{H},(k)}&=\mathbf{h}_{i}^{\mathbb{H},(k)}\oplus\big{(}\gamma^{\prime}d_{\mathbb{H}}(\mathbf{h}_{i}^{\mathbb{H},(k)},\exp_{\mathbf{0}}(\mathbf{h}_{i}^{\mathbb{R},(k)}))\otimes\exp_{\mathbf{0}}(\mathbf{h}_{i}^{\mathbb{R},(k)})\big{)},\end{split}$

where $\exp_{\mathbf{0}}(\cdot)$ and $\log_{\mathbf{0}}(\cdot)$ are leveraged
to transform the features between Euclidean and hyperbolic spaces. The
similarity from different spatial features is measured by Euclidean distance
$d_{\mathbb{R}}$ hyperbolic distance $d_{\mathbb{H}}$ in Eq. (4), scaled by
trainable scalar $\beta,\beta^{\prime}$. $\oplus$ and $\otimes$ are addiction
and scalar multiplication defined in hyperbolic spaces. Given feature vectors
$\mathbf{x},\mathbf{y}\in\mathbb{H},\mathbf{v}\in\mathcal{T}_{\mathbf{x}}\mathbb{H}$,
and a scalar $r$, these multiplications are defined as:
$\mathbf{x}\oplus\mathbf{y}=\exp_{\mathbf{x}}(P_{\mathbf{0}\rightarrow\mathbf{x}}(\log_{\mathbf{0}}(\mathbf{y}))),P_{\mathbf{0}\rightarrow\mathbf{x}}(\mathbf{v})=\mathbf{v}+(\mathbf{0}+\mathbf{x})\cdot\langle\mathbf{x},\mathbf{v}\rangle_{\mathcal{L}}/(1-\langle\mathbf{0},\mathbf{x}\rangle_{\mathcal{L}}),$
and
$r\otimes\mathbf{x}=\exp_{\mathbf{0}}(r\cdot\log_{\mathbf{0}}(\mathbf{x})).$

###  3.3. Layers fusion and prediction

After applying $K$ geometric graph convolutional layers, we can obtain
features in each layer. To obtain more powerful representations, we fuse all
the features in each layer along with the input features (e.g., the $0$-th
layer) through the equal weight $1/(K+1)$:

(7)
$\mathbf{f}_{i}^{\mathbb{R}}=\sum_{k=0}^{K}{\mathbf{f}_{i}^{\mathbb{R},(k)}}/(K+1),\quad\mathbf{f}_{i}^{\mathbb{H}}=\sum_{k=0}^{K}\frac{{\mathbf{f}_{i}^{\mathbb{H},(k)}}/(K+1)}{|\|{\mathbf{f}_{i}^{\mathbb{H},(k)}}/(K+1)\|_{\mathcal{L}}|}.$

Thus, we can prediction the relation between user and item with final feature
representations with Euclidean and Lorentzian scalar product:
$\hat{y}_{ui}=\langle\mathbf{f}_{u}^{\mathbb{R}},\mathbf{f}_{i}^{\mathbb{R}}\rangle+\lambda\langle\mathbf{f}_{u}^{\mathbb{H}},\mathbf{f}_{i}^{\mathbb{H}}\rangle_{\mathcal{L}},$
where $\lambda$ is a trainable parameter to balance the importance between the
Euclidean and hyperbolic parts in our model.

###  3.4. Model training

We employ the Bayesian Personalized Ranking (BPR) loss (Rendle et al., 2012),
which encourages the prediction of an observed entry to be higher than its
unobserved counterparts:
$\mathcal{L}=-\sum_{u}\sum_{i\in\mathcal{N}_{u}}\sum_{j\notin\mathcal{N}_{u}}\ln\sigma(\hat{y}_{ui}-\hat{y}_{uj}).$
We leverage $L_{2}$ regularization to avoid overfitting. Please note that the
input features for our hyperbolic graph convolutional layers are initialized
in the Euclidean space $\mathcal{T}_{\mathbf{0}}\mathbb{H}$ and mapped to
hyperbolic spaces via $\exp_{\mathbf{0}}(\cdot)$. In the process of
transforming features between hyperbolic and Euclidean (tangent) spaces, we
concatenate $``0"$ and the features to satisfy the definition in Eq. (1).
Since all the trainable parameters are defined in Euclidean spaces, we employ
the Adam (Kingma and Ba, 2015) optimizer and train our model in a mini-batch
manner.

###  3.5. Discussion

####  3.5.1. Relation with GIL

The graph convolutional layer in GIL (Zhu et al., 2020) is different from our
work. GIL employs feature transformation and non-linear activation, while our
proposal discards such operations since they might be burdensome for
collaborative filtering (He et al., 2020). Meanwhile, the hyperbolic neighbor
aggregator in GIL is different from ours. The aggregator in our method is
based on the Lorentzian centorid, while the one in GIL is based on the tangent
spaces.

##  4\. Experiments

In this section, we conduct extensive experiments on publicly available
datasets to answer the following three research questions:

  * •

RQ1: How dose the proposed GGCF model perform as compared with the state-of-
the-art methods?

  * •

RQ2: What are the key components that influence the model performance?

  * •

RQ3: How do different hyper-parameter settings (e.g., the number of graph
convolutional layers) affect the model performance?

###  4.1. Experimental Settings

####  4.1.1. Datasets and Metrics

We select two popular real-world datasets for evaluation. Table 1 presents the
statistical details. Movielens111https://grouplens.org/datasets/movielens/
describes the ratings and free-text tagging activities on the MovieLens
website. LastFM222http://www.lastfm.com contains the interactions between the
users and artists. We follow the settings in NGCF (Wang et al., 2019) to split
the training/test sets for a fair comparison. Following the previous works
(Wang et al., 2019; He et al., 2020), we leverage recall@20 and ndcg@20 to
evaluate the effectiveness of our proposal.

Table 1. Statistics of the datasets. Dataset | # User | # Item | # Interaction
---|---|---|---
MovieLens | 610 | 9,742 | 100,836
LastFM | 1,892 | 17,632 | 92,834

####  4.1.2. Baselines

We select the following SOTA CF models as baselines to demonstrate the
superiority of GGCF model:

  * •

BPRMF (Rendle et al., 2012) is a matrix factorization based CF model under the
framework of Bayesian personalized ranking.

  * •

HyperML (Vinh Tran et al., 2020) is a hyperbolic CF model, which explores
metric learning in hyperbolic spaces.

  * •

NGCF (Wang et al., 2019) is a graph-based CF model to incorporate the high-
order connectivity of user-item interactions.

  * •

LightGCN (He et al., 2020) is a state-of-the-art CF recommendation model based
on light graph convolution.

The dimension of user/item embeddings is set to 64. The number of graph
convolutional layers in graph-based models is set to 3. Parameters are
carefully tuned on a small validation dataset. Specifically, the learning rate
is searched in $\\{1e-2,5e-3,1e-3,5e-4,1e-4\\}$ and $L_{2}$ regularization
term is tuned in $\\{0,1e-6,1e-5,1e-4,1e-3\\}$.

###  4.2. Overall Performance (RQ1)

Table 2. Overall Performance. Dataset | MovieLens | LastFM
---|---|---
| recall@20 | ndcg@20 | recall@20 | ndcg@20
BPRMF | 0.1914 | 0.2561 | 0.2180 | 0.2106
hyperML | 0.2076 | 0.2654 | 0.2164 | 0.2094
NGCF | 0.2197 | 0.3164 | 0.2374 | 0.2409
LightGCN | 0.2517 | 0.3362 | 0.2592 | 0.2610
GGCF | 0.2589 | 0.3420 | 0.2621 | 0.2691

The experiment results are presented in Table 2. From the results, we can
obtain the following observations.

  * •

Our proposed GGCF model achieves the best performance. GGCF consistently
outperforms all the baseline methods on both datasets under different
evaluation metrics, which indicates the superiority of simultaneously modeling
user-item relations in the Euclidean and hyperbolic spaces.

  * •

Graph-based models are more powerful. The graph-based models (i.e., NGCF,
LightGCN and GGCF) outperform the traditional shallow models (i.e., BPRMF and
hyperML), which verifies the benefit of incorporating high-order topological
connectivity as complementary.

  * •

Single hyperbolic space is not enough. One can see that the pure hyperbolic CF
model (hyperML) performs worse than the Euclidean-based approaches (e.g, NGCF
and LightGCN), which suggests that the single hyperbolic space may be
insufficient to capture the sophisticated graph topology. However, by enjoying
the merits from both Euclidean and hyperbolic sides, the performance of our
proposal surpasses the single space-based models. It demonstrates that GGCF is
capable of capturing the cross-space correlations to learn more comprehensive
user/item representations.

###  4.3. Ablation Study (RQ2)

Table 3. Experimental results of the ablation Study. Dataset | MovieLens | LastFM
---|---|---
| recall@20 | ndcg@20 | recall@20 | ndcg@20
GGCF${}_{\backslash\text{I}}$ | 0.2509 | 0.3295 | 0.2602 | 0.2630
GGCF${}_{\backslash\text{E}}$ | 0.2412 | 0.3160 | 0.2641 | 0.2517
GGCF${}_{\backslash\text{H}}$ | 0.2514 | 0.3359 | 0.2598 | 0.2623
GGCF | 0.2589 | 0.3420 | 0.2621 | 0.2691

In order to evaluate the effectiveness of critical components in GGCF, we
design the following three ablation models: GGCF${}_{\backslash\text{E}}$
without the hyperbolic part, GGCF${}_{\backslash\text{H}}$ without the
Euclidean part and the ensemble model GGCF${}_{\backslash\text{E}}$ without
the interaction components. Experimental results are shown in Table 3.

  * •

Effectiveness of the cross-space interactions. GGCF outperforms the ablation
model GGCF${}_{\backslash\text{I}}$ by nearly 1.3% on average, which indicates
that the interactions between the Euclidean and hyperbolic spaces can further
enrich the complementary relationships.

  * •

Effectiveness of Euclidean and hyperbolic graph convolutional part. Compared
with GGCF${}_{\backslash\text{E}}$ and GGCF${}_{\backslash\text{H}}$ models,
GGCF performs better in most cases, which also indicates GGCF can make a
desirable balance between the information from Euclidean and hyperbolic
spaces.

###  4.4. Parameter Sensitivity Study (RQ3)

Figure 3. Results of GGCF with different number of layers.

Here we study the performance sensitivity of HAUIL model on the core
parameter: the number of graph convolutional layer. We change the number of
layers from 1 to 4, and the results are shown in Figure 3.

With the increase of number of layers, the performance over almost all the
datasets first increases and then keeps steady or slightly drops, which
demonstrates appropriate high-order topological information may benefit the
model performance. However, a too large number of layers will lead to
challenge of “neighborhood explosion”, which may introduces noises and will be
resource-consuming. Thus, we have to carefully choose an appropriate value to
balance the model efficiency and effectiveness.

##  5\. Conclusion

In this work, we study the novel problem of graph-based CF. We propose a novel
geometry graph collaborative filtering method for recommender systems, to
learn the user/item representations in both Euclidean and hyperbolic spaces.
These representations are further enhanced by the dual geometry interaction
scheme to take advantage of both Euclidean and hyperbolic spaces. The
experimental results on public datasets demonstrate the effectiveness of the
propose method.

## References

  * (1)
  * Chami et al. (2019) Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. 2019\.  Hyperbolic graph convolutional neural networks. In _NeurIPS_. 4869–4880.
  * Feng et al. (2020) Shanshan Feng, Lucas Vinh Tran, Gao Cong, Lisi Chen, Jing Li, and Fan Li. 2020\.  HME: A Hyperbolic Metric Embedding Approach for Next-POI Recommendation. In _SIGIR_. 1429–1438.
  * Fréchet (1948) Maurice Fréchet. 1948\.  Les éléments aléatoires de nature quelconque dans un espace distancié. In _Annales de l’institut Henri Poincaré_ , Vol. 10. 215–310.
  * Ganea et al. (2018) Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. 2018\.  Hyperbolic Entailment Cones for Learning Hierarchical Embeddings. In _ICML_. 1646–1655.
  * He et al. (2020) Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020\.  Lightgcn: Simplifying and powering graph convolution network for recommendation. In _SIGIR_. 639–648.
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017\.  Neural collaborative filtering. In _Proceedings of the 26th international conference on world wide web_. 173–182.
  * Isinkaye et al. (2015) Folasade Olubusola Isinkaye, Yetunde O Folajimi, and Bolande Adefowoke Ojokoh. 2015.  Recommendation systems: Principles, methods and evaluation.  _Egyptian informatics journal_ 16, 3 (2015), 261–273.
  * Karcher (1987) Hermann Karcher. 1987\.  _Riemannian comparison constructions_.  SFB 256.
  * Karcher (2014) Hermann Karcher. 2014\.  Riemannian center of mass and so called karcher mean.  _arXiv preprint arXiv:1407.2087_ (2014).
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  _ICLR_ (2015).
  * Koren et al. (2022) Yehuda Koren, Steffen Rendle, and Robert Bell. 2022\.  Advances in collaborative filtering.  _Recommender systems handbook_ (2022), 91–142.
  * Law et al. (2019) Marc Law, Renjie Liao, Jake Snell, and Richard Zemel. 2019\.  Lorentzian Distance Learning for Hyperbolic Representations. In _ICML_. 3672–3681.
  * Liu et al. (2019) Qi Liu, Maximilian Nickel, and Douwe Kiela. 2019\.  Hyperbolic graph neural networks. In _NeurIPS_. 8228–8239.
  * Nickel and Kiela (2018) Maximilian Nickel and Douwe Kiela. 2018.  Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In _ICML_. 3779–3788.
  * Rendle et al. (2012) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012.  BPR: Bayesian personalized ranking from implicit feedback.  _arXiv preprint arXiv:1205.2618_ (2012).
  * Sankar et al. (2021) Aravind Sankar, Yozen Liu, Jun Yu, and Neil Shah. 2021\.  Graph neural networks for friend ranking in large-scale social platforms. In _Proceedings of the Web Conference 2021_. 2535–2546.
  * Sarwar et al. (2000) Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2000.  Analysis of recommendation algorithms for e-commerce. In _Proceedings of the 2nd ACM Conference on Electronic Commerce_. 158–167.
  * Schafer et al. (2001) J Ben Schafer, Joseph A Konstan, and John Riedl. 2001\.  E-commerce recommendation applications.  _Data mining and knowledge discovery_ 5, 1 (2001), 115–153.
  * Su and Khoshgoftaar (2009) Xiaoyuan Su and Taghi M Khoshgoftaar. 2009.  A survey of collaborative filtering techniques.  _Advances in artificial intelligence_ 2009 (2009).
  * Tang et al. (2013) Jiliang Tang, Xia Hu, and Huan Liu. 2013.  Social recommendation: a review.  _Social Network Analysis and Mining_ 3, 4 (2013), 1113–1133.
  * Vinh Tran et al. (2020) Lucas Vinh Tran, Yi Tay, Shuai Zhang, Gao Cong, and Xiaoli Li. 2020.  HyperML: a boosting metric learning approach in hyperbolic space for recommender systems. In _WSDM_. 609–617.
  * Wang et al. (2019) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019.  Neural graph collaborative filtering. In _SIGIR_. 165–174.
  * Wang et al. (2022) Yiqi Wang, Chaozhuo Li, Mingzheng Li, Wei Jin, Yuming Liu, Hao Sun, Xing Xie, and Jiliang Tang. 2022\.  Localized Graph Collaborative Filtering. In _Proceedings of the 2022 SIAM International Conference on Data Mining (SDM)_. SIAM, 540–548.
  * Zhang et al. (2022) Yiding Zhang, Chaozhuo Li, Xing Xie, Xiao Wang, Chuan Shi, Yuming Liu, Hao Sun, Liangjie Zhang, Weiwei Deng, and Qi Zhang. 2022\.  Geometric Disentangled Collaborative Filtering.  (2022).
  * Zhu et al. (2020) Shichao Zhu, Shirui Pan, Chuan Zhou, Jia Wu, Yanan Cao, and Bin Wang. 2020\.  Graph Geometry Interaction Learning.  _Advances in Neural Information Processing Systems_ 33 (2020).


# Generative Bias for Visual Question Answering

Jae Won Cho1 Dong-Jin Kim2 Hyeonggon Ryu1 In So Kweon1
1KAIST, South Korea. 2UC Berkeley, CA.
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The task of Visual Question Answering (VQA) is known to be plagued by the
issue of VQA models exploiting biases within the dataset to make its final
prediction. Many previous ensemble based debiasing methods have been proposed
where an additional model is purposefully trained to be biased in order to aid
in training a robust target model. However, these methods compute the bias for
a model from the label statistics of the training data or directly from single
modal branches. In contrast, in this work, in order to better learn the bias a
target VQA model suffers from, we propose a generative method to train the
bias model _directly from the target model_ , called GenB. In particular, GenB
employs a generative network to learn the bias through a combination of the
adversarial objective and knowledge distillation. We then debias our target
model with GenB as a bias model, and show through extensive experiments the
effects of our method on various VQA bias datasets including VQA-CP2, VQA-CP1,
GQA-OOD, and VQA-CE.

Generative Bias for Visual Question Answering

Jae Won Cho1 Dong-Jin Kim2 Hyeonggon Ryu1 In So Kweon1 1KAIST, South Korea.
2UC Berkeley, CA<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Visual Question Answering (VQA) Antol et al. (2015) is a challenging task that
requires a model to correctly understand and predict an answer given a input
pair of image and question. Various studies have shown that VQA is prone to
biases within the dataset and tend to rely heavily on language biases present
in the dataset Agrawal et al. (2016); Goyal et al. (2017); Zhang et al.
(2016), where VQA models tend to predict similar answers only depending on the
question regardless of the image. In in response to this, recent works have
developed various bias reduction techniques and recent methods have exploited
ensemble based methods Cadene et al. (2019); Clark et al. (2019); Han et al.
(2021); Ramakrishnan et al. (2018) extensively.

Figure 1: Given a Question Type (“What color is…”), we show all of the
averaged answers within the training dataset. The answer computed from the
entire training dataset is the known dataset bias as in Clark et al. (2019);
Han et al. (2021). We see that the model predictions of the Question-Answer
Model and Visual-Question-Answer Model are significantly different.

Among ensemble based methods, additional models are introduced to concurrently
learn biases that might exist within each modality or dataset. For example, in
works such as Cadene et al. (2019); Han et al. (2021), the Question-Answer
model is utilized to determine the language prior biases that exist when a
model is asked to give an answer based solely off of the question. This model
is then utilized to train a robust “target” model, which is used for
inference. The key purpose of an ensemble “bias” model is to capture the
biases that are formed with its given inputs (_i.e_., language prior biases
from the Question-Answer model). In doing so, if this model is able to
represent the bias well, this bias model can be used to teach the target model
to avoid such biased answers. In other words, the better the bias model can
learn the biases, the better the target model can avoid such biases.

Existing ensemble based methods either use pre-computed label statistics of
training data (GGE-D Han et al. (2021), LMH Clark et al. (2019)), or trained
by directly computing the answer distribution from either the question or
image Cadene et al. (2019); Clark et al. (2019); Han et al. (2021); Niu et al.
(2021). However, we propose that there is a limit to the bias representation
that can be obtained from such methods, as the model’s representative capacity
is limited due to its input. In addition, pre-computed label statistics
represents part of the bias Han et al. (2021). We show in Fig. 1, that given a
question type, the pre-computed label statistics (or known dataset bias) are
actually different to the predictions of a model trained with the question or
with the image and question. This discrepancy means that there is a part of
the bias that we cannot fully model simply with the previous methods. In light
of this, we propose to utilize a novel ensemble based bias model that can have
stochastic representations and also capture the biases that the target model
inhibits.

More specifically, to capture bias by mimicking the target model’s answer
representation given the same question input, we model the bias model as a
Generative Adversarial Network (GAN) Goodfellow et al. (2014) to generate
stochastic bias representation given a single modality input by introducing an
additional random noise vector. As seen through literature, most biases are
held within the question Agrawal et al. (2016), so we use questions as the
main bias modality. In addition, we utilize knowledge distillation on top of
adversarial training so that the target model learns from harder negative
supervision from the bias model. Finally, with our generative bias model, we
then use our modified debiasing loss function to train our target model. Our
final bias model is able to train the target model that outperforms previous
uni-modal and multi-modal ensemble based methods by a large margin. To the
best of our knowledge, we are the first to train the bias model by directly
leveraging the behavior of the target model using a generative model for the
task of VQA.

To show the efficacy of our methods, we perform extensive experiments on
commonly used robustness testing VQA datasets: VQA-CP2 and VQA-CP1 Agrawal et
al. (2018) as well as the most recent and challenging VQA setups such as GQA-
OOD Kervadec et al. (2021) and the new evaluation protocol, VQA-CE Dancette et
al. (2021). Our method show the state-of-the-art results on all settings
without the use of external human annotations and dataset reshuffling methods.

Our contributions are as follows:

  * •

We propose a novel bias model for ensemble based debiasing for VQA by directly
leveraging the target model that we name _GenB_.

  * •

In order to effectively train GenB, we employ a Generative Adversarial Network
and knowledge distillation loss to capture both the dataset distribution bias
and the bias from the target model.

  * •

We achieve the state-of-the-art performance on VQA-CP2, VQA-CP1 as well as the
more challenging GQA-OOD datasets and VQA-CE using the simple UpDn baseline
without extra annotations or dataset reshuffling.

##  2 Related Work

VQA Antol et al. (2015); Gurari et al. (2018); Johnson et al. (2017) has been
actively studied in recent years with performance reaching close to human
performance Kim et al. (2018); Tan and Bansal (2019); Su et al. (2019); Lu et
al. (2019); Chen et al. (2020b) in the most recent works. Even still, the VQA
dataset has been notorious for its reliance on language biases as shown by
Agrawal et al. (2018). To this date, many VQA datasets and evaluation
protocols have been derived from the original VQA dataset and have been
released to the public as a means to test and understand the biases and
robustness of VQA models such as VQA-CP2 and CP1 Agrawal et al. (2018), GQA-
OOD Kervadec et al. (2021), and VQA-CE Dancette et al. (2021).

In light of this, many methods have tackled this issue in various ways. One
line of work strives to improve visual attention through visual grounding by
including additional human annotations or explanations Selvaraju et al.
(2019); Wu and Mooney (2019), which show limited improvements. Another line of
work changes the distribution of the training set by either randomly replacing
the images and questions Teney et al. (2020); Zhu et al. (2021); Wen et al.
(2021) or by augmenting the dataset through counterfactual samples Chen et al.
(2020a) or through the use of an external inpainting network Gokhale et al.
(2020). Although powerful, according to Niu et al. (2021), changing the
distribution of the training set does not agree with the philosophy of the
creation of this dataset. Therefore, we do not directly compare experimentally
with this line of works. Most recently, another line of work has been released
where an additional objective of Visual Entailment is added to further boost
performance Si et al. (2021), which we do not compare for fairness.

Excluding those works, the most effective line of work so far has been
ensemble based methods Abbasnejad et al. (2020); Cadene et al. (2019); Niu et
al. (2021); Han et al. (2021); thus, we place our work in line with these
ensemble based methods. Within the ensemble based methods, AReg Abbasnejad et
al. (2020), RUBi Cadene et al. (2019), LMH Clark et al. (2019), GGE Han et al.
(2021) tackle the language prior directly and only use a question-only
branch.. Unlike these single branch model based methods, GGE Han et al. (2021)
shows the possibility of using the same model as the target model to learn the
biases but to a limited extent. On the other hand, CF-VQA Niu et al. (2021)
uses both question and image, but uses the two modalities individually without
combining them. Our work is distinct from all previous ensemble based methods
as we use a generative network with a noise input to aid the bias model in
learning the bias directly from the target model.

##  3 Methodology

In this section, we explain VQA briefly and describe in detail our bias model,
GenB and how we train it and how we debias with it.

###  3.1 Visual Question Answering Baseline

With an image and question as a pair of inputs, a VQA model learns to
correctly predict an answer from the whole answer set $\mathcal{A}$. A typical
VQA model ${F(\cdot,\cdot)}$ takes both a visual representation
${\textbf{v}}\in\mathbb{R}^{n\times d_{v}}$ (a set of feature vectors computed
from a Convolutional Neural Network given an image where $n$ is the number of
number of objects in the image and $d_{v}$ being the vector dimension) and a
question representation ${\textbf{q}}\in\mathbb{R}^{d_{q}}$ (a single vector
computed from a Glove Pennington et al. (2014) word embedding followed by a
Recurrent Neural Network given a question) as input. Then, an attention module
followed by a multi-layer perceptron classifier ${F:\mathbb{R}^{n\times
d_{v}}\times\mathbb{R}^{d_{q}}\xrightarrow{}\mathbb{R}^{|\mathcal{A}|}}$ which
generates an answer logit vector $\mathbf{y}\in\mathbb{R}^{|\mathcal{A}|}$
(_i.e_., $\mathbf{y}={F(V,Q)}$). Then, after applying a sigmoid function
$\sigma(\cdot)$, our goal is to make an answer probability prediction
$\sigma(\mathbf{y})\in[0,1]^{|\mathcal{A}|}$ close to the ground truth answer
probability $\mathbf{y}_{gt}\in[0,1]^{|\mathcal{A}|}$. In this work, we adopt
one of the popular state-of-the-art architectures UpDn Anderson et al. (2018)
widely used in VQA research.

###  3.2 Ensembling with Bias Models

In this work, our scope is bias mitigation through ensembling bias model
similar to previous works Cadene et al. (2019); Clark et al. (2019); Han et
al. (2021). In ensemble based methods, there exist a “bias” model that
generates $\mathbf{y}_{b}\in\mathbb{R}^{|\mathcal{A}|}$ which we define as
${F}_{b}(\cdot,\cdot)$ and a “target” model, defined as ${F(\cdot,\cdot)}$.
Note that, we discard ${F}_{b}(\cdot,\cdot)$ during testing and only use
${F(\cdot,\cdot)}$. As previously mentioned, the goal of the existing bias
models is to overfit to the bias as much as possible. Then, given the
overfitted bias model, the target model is trained with a debiasing loss
function Cadene et al. (2019); Clark et al. (2019); Han et al. (2021) to
improve the robustness of the target model. Ultimately, the target model
learns to predict an unbiased answer by avoiding the biased answer from the
bias model. The bias model ${F}_{b}(\cdot,\cdot)$ can either be the same or
different from the original ${F}(\cdot,\cdot)$ and there could be multiple
models as well Niu et al. (2021). Although previous works try to leverage the
bias from the individual modalities, we propose that this limits the ability
of the model to represent biases. Hence, in order to try to represent the
biases similar to the target model, we set the architecture of
${F}_{b}(\cdot,\cdot)$ to be the same as ${F}(\cdot,\cdot)$ and we use the
UpDn Anderson et al. (2018) model.

Figure 2: (a) shows how we train our Bias model and discriminator. The Bias
model is trained with 3 different losses including the ground truth BCE (Eq.
(1)), knowledge distillation (Eq. (3)), and adversarial Eq. (2)) losses. (b)
shows how our target model is trained with the Bias model with debiasing loss
functions (refer existing works). Note, steps (a) and (b) happen concurrently.
Note that we only use the target model during inference.

###  3.3 Generative Bias

As mentioned in the Sec. 1, as our goal is to create a bias model that can
generate stochastic bias representations, we use a random noise vector in
conjunction with a given modality to learn both the dataset bias and the bias
that the target model could exhibit. As the question is known to be prone to
bias, we keep the question modality and use it as the input to our bias model
${F}_{b}(\cdot,\cdot)$. But instead of using the image features, we introduce
a random noise vector $\mathbf{z}\in\mathbb{R}^{n\times 128}$ in addition to a
generator network $G:\mathbb{R}^{n\times 128}\xrightarrow{}\mathbb{R}^{n\times
d_{v}}$ to generate the corresponding input to the bias model
${F}_{b}(\cdot,\cdot)$. Formally, given a random Gaussian noise vector
$\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{1})$, a generator network
${G}(\cdot)$ synthesizes a vector that has the same dimension as the image
feature representation, _i.e_.,
$\hat{\mathbf{v}}={G}(\mathbf{z})\in\mathbb{R}^{n\times d_{v}}$. Ultimately,
our model takes in the question $\mathbf{q}$ and $G(\mathbf{z})$ as its input
and generates the bias logit $\mathbf{y_{b}}$ in the form
$F_{b}({G}(\mathbf{z}),\mathbf{q})=\mathbf{y_{b}}$. Note, this can be done on
either modality, such as in the form
$F_{b}({G}(\mathbf{z}),\mathbf{v})=\mathbf{y_{b}}$, but we found this
unhelpful. For simplicity, we consider generator and bias model as one network
and rewrite $F_{b}(G(\mathbf{z}),\mathbf{q})$ in the form
$F_{b,G}(\mathbf{z},\mathbf{q})$ and call our “Generative Bias” method GenB.

###  3.4 Training the Bias Model

In order for our bias model GenB to learn the biases given the question, we
use the traditional VQA loss, the Binary Cross Entropy Loss:

$\mathcal{L}_{GT}(F_{b,G})=\mathcal{L}_{BCE}(\sigma(F_{b,G}(\mathbf{z},\mathbf{q})),\mathbf{y}_{gt})$
(1)

However, unlike existing works, we want the bias model to also capture the
biases in the target model. Hence, in order to mimic the bias of the target
model as a dynamic distribution of the answer, we propose the adversarial
training Goodfellow et al. (2014) to train our bias model. In particular, we
introduce a discriminator to try to distinguish the answers from the target
model and the bias model as “real” and “fake” answers, respectively. The
discriminator is formulated as $D(F(\mathbf{v},\mathbf{q}))$ and
$D(F_{b,G}(\mathbf{z},\mathbf{q}))$ or rewritten as $D(\mathbf{y})$ and
$D(\mathbf{y}_{b})$. The objective of our generative adversarial network with
generator $F_{b,G}(\cdot,\cdot)$ and $D(\cdot)$ can be expressed as follows:

$\begin{split}&\min_{F_{b,G}}\max_{D}\mathcal{L}_{GAN}(F_{b,G},D),\text{where}\\\
&\mathcal{L}_{GAN}(F_{b,G},D)\\\
=&\mathop{\mathbb{E}}\limits_{{\mathbf{v},\mathbf{q}}}\Big{[}\log\Big{(}D\big{(}F(\mathbf{v},\mathbf{q})\big{)}\Big{)}\Big{]}\\\
&+\mathop{\mathbb{E}}\limits_{{\mathbf{q},\mathbf{z}}}\Big{[}\log\Big{(}1-D\big{(}F_{b,G}(\mathbf{z},\mathbf{q})\big{)}\Big{)}\Big{]}\\\
=&{\mathbb{E}}_{{\mathbf{y}}}\Big{[}\log\big{(}D(\mathbf{y})\big{)}\Big{]}+{\mathbb{E}}_{{\mathbf{y}_{b}}}\Big{[}\log\big{(}1-D(\mathbf{y}_{b})\big{)}\Big{]}.\end{split}$
(2)

The generator ($F_{b,G}$) tries to minimize the objective against an
adversarial discriminator ($D$) that tries to maximize it. Through alternative
training of $D$ and $F_{b,G}$, the distribution of the answer vector from the
bias model ($\mathbf{y}_{b}$) should be close to that from the target model
($\mathbf{y}$).

In addition, in order to further aid in the bias model’s ability to capture
the intricate biases present in the target model, we add an additional
objective that encourages the bias model to directly follow the behavior of
the target model with only the $\mathbf{q}$ given to it. We empirically find
that it is beneficial to include a sample-wise distance based metric such as
KL divergence. This phenomenon is well known in the image to image translation
task Isola et al. (2017). Then, the goal of the generator is not only to fool
the discriminator but also to try to imitate the answer output of the target
model in order to give the target model more challenging supervision in the
form of hard negative sample synthesis. We add another objective to our
adversarial training for $F_{b,G}(\cdot,\cdot)$ as follows:

$\mathcal{L}_{distill}(F_{b,G})=\mathop{\mathbb{E}}\limits_{{\mathbf{v},\mathbf{q},\mathbf{z}}}\Big{[}D_{KL}\big{(}F(\mathbf{v},\mathbf{q})\rVert{F_{b,G}(\mathbf{z},\mathbf{q})}\big{)}\Big{]}.$
(3)

Ultimately, the final training loss for the bias model, or GenB, is as
follows:

$\displaystyle\min_{F_{b,G}}\max_{D}\mathcal{L}_{GenB}(F_{b,G},D),\text{where}$
(4) $\displaystyle\mathcal{L}_{GenB}(F_{b,G},D)=\mathcal{L}_{GAN}(F_{b,G},D)$
$\displaystyle+\lambda_{1}\mathcal{L}_{distill}(F_{b,G})+\lambda_{2}\mathcal{L}_{GT}(F_{b,G}),$

where $\lambda_{1}$ and $\lambda_{2}$ are the loss weight hyper-parameters.

###  3.5 Debiasing the Target Model

Given a generated biased answer $\mathbf{y}_{b}$, there are several debiasing
loss functions that we can use such as Cadene et al. (2019); Clark et al.
(2019); Han et al. (2021), and we show the effects of each one in Table 5. The
GGE Han et al. (2021) loss is one of the best performing without the use of
label distribution. The GGE loss takes the bias predictions/distributions and
generates a gradient in the opposite direction to train the target model. With
this starting point, we modify this equation with the ensemble of the biased
model in this work as follows:

$\mathcal{L}_{target}(F)=\mathcal{L}_{BCE}(\mathbf{y},\mathbf{y}_{DL}),$ (5)

where the $i$-th element of the pseudo-label $\mathbf{y}_{DL}$ is defined as
follows:

$\mathbf{y}_{DL}^{i}=\min\big{(}1~{},~{}2\cdot\mathbf{y}_{gt}^{i}\cdot\sigma(-2\cdot\mathbf{y}_{gt}^{i}\cdot\mathbf{y}_{b}^{i})\big{)},$
(6)

where $\mathbf{y}_{gt}^{i}$ and $\mathbf{y}_{b}^{i}$ are the $i$-th element of
the ground truth and the output of the biased model, respectively. The key
point of difference is that unlike Han et al. (2021) that suppresses the
output of the biased model with the sigmoid function, we use $\mathbf{y}_{b}$
without using the sigmoid function. In this case, as the value of
$\mathbf{y}_{DL}$ can exceed $1$, we additionally clip the value so that the
value of $\mathbf{y}_{DL}$ is bounded in $[0,1]$. We empirically find these
simple modifications on the loss function significantly improves the
performance. We conjecture the unsuppressed biased output $\mathbf{y}_{b}$
allows our target model to better consider the _intensity_ of the bias,
leading to a more robust target model. In addition, when we train the target
model, we do not use the noise inputs as in $F_{b,G}(\mathbf{z},\mathbf{q})$,
rather we use the real images as such $F_{b}(\mathbf{v},\mathbf{q})$, and use
this output to train our target model. When the bias model is trained, it is
trained with a noise vector to hallucinate the possible biases when only given
the question, then, to fully utilize the biases that the bias model captures,
we give it the real images.

##  4 Experiments

Method |  | Base | VQA-CP2 test |  | VQA-CP1 test
---|---|---|---|---|---
| All | Yes/No | Num | Other |  | All | Yes/No | Num | Other
SAN Yang et al. (2016) |  | - | 24.96 | 38.35 | 11.14 | 21.74 |  | 32.50 | 36.86 | 12.47 | 36.22
GVQA Agrawal et al. (2018) |  | - | 31.30 | 57.99 | 13.68 | 22.14 |  | 39.23 | 64.72 | 11.87 | 24.86
S-MRL Cadene et al. (2019) |  | - | 38.46 | 42.85 | 12.81 | 43.20 |  | 36.38 | 42.72 | 12.59 | 40.35
UpDn Anderson et al. (2018) |  | - | 39.94 | 42.46 | 11.93 | 45.09 |  | 36.38 | 42.72 | 42.14 | 40.35
Methods based on modifying language modules
DLR Jing et al. (2020) |  | UpDn | 48.87 | 70.99 | 18.72 | 45.57 |  | – | – | – | –
VGQE KV and Mittal (2020) |  | UpDn | 48.75 | – | – | – |  | – | – | – | –
VGQE KV and Mittal (2020) |  | S-MRL | 50.11 | 66.35 | 27.08 | 46.77 |  | – | – | – | –
Methods based on strengthening visual attention
HINT Selvaraju et al. (2019) |  | UpDn | 46.73 | 67.27 | 10.61 | 45.88 |  | – | – | – | –
SCR Wu and Mooney (2019) |  | UpDn | 49.45 | 72.36 | 10.93 | 48.02 |  | – | – | – | –
Methods based ensemble models
AReg Ramakrishnan et al. (2018) |  | UpDn | 41.17 | 65.49 | 15.48 | 35.48 |  | 43.43 | 74.16 | 12.44 | 25.32
RUBi Cadene et al. (2019) |  | UpDn | 44.23 | 67.05 | 17.48 | 39.61 |  | 50.90 | 80.83 | 13.84 | 36.02
LMH Clark et al. (2019) |  | UpDn | 52.45 | 69.81 | 44.46 | 45.54 |  | 55.27 | 76.47 | 26.66 | 45.68
CF-VQA(SUM) Niu et al. (2021) |  | UpDn | 53.55 | 91.15 | 13.03 | 44.97 |  | 57.03 | 89.02 | 17.08 | 41.27
CF-VQA(SUM) Niu et al. (2021) |  | S-MRL | 55.05 | 90.61 | 21.50 | 45.61 |  | 57.39 | 88.46 | 14.80 | 43.61
GGE Han et al. (2021) |  | UpDn | 57.32 | 87.04 | 27.75 | 49.59 |  | – | – | – | –
GenB (Ours) |  | UpDn | 58.75 | 88.91 | 35.37 | 49.37 |  | 62.74 | 86.18 | 43.85 | 47.03
Methods based on balancing training data
CVL Abbasnejad et al. (2020) |  | UpDn | 42.12 | 45.72 | 12.45 | 48.34 | – | – | – | – |
RandImg Teney et al. (2020) |  | UpDn | 55.37 | 83.89 | 41.60 | 44.20 |  | – | – | – | –
SSL Zhu et al. (2021) |  | UpDn | 57.59 | 86.53 | 29.87 | 50.03 |  | – | – | – | –
CSS Chen et al. (2020a) |  | UpDn | 58.95 | 84.37 | 49.42 | 48.21 |  | 60.95 | 85.60 | 40.57 | 47.03
MUTANT Gokhale et al. (2020) |  | UpDn | 61.72 | 88.90 | 49.68 | 50.78 |  | – | – | – | –
D-VQA Wen et al. (2021) |  | UpDn | 61.91 | 88.93 | 52.32 | 50.39 |  | – | – | – | –

Table 1: Experimental results of our method on the VQA-CP2 test set and VQA-
CP1 test set. Best and second best results are styled in this manner within
the column. We do not directly compare to methods that change the distribution
of the training data as it does not go with the philosophy of VQA-CP Niu et
al. (2021). Among the compared baselines, our method GenB shows the best
performance by a noticeable margin.

In this section, we explain our experimental setting and show our quantitative
and qualitative experimental results.

###  4.1 Setup

Dataset and evaluation metric. We conduct our experiments within the VQA
datasets that are commonly used for diagnosing bias in VQA models. In
particular, we test on the the VQA-CP2 and VQA-CP1 datasets Agrawal et al.
(2018) and GQA-OOD dataset Kervadec et al. (2021). For evaluation on all
datasets, we take the standard VQA evaluation metric Antol et al. (2015). In
addition to this, we also evaluate our method on the VQA-CE Dancette et al.
(2021), which is based on the VQA-v2 dataset, and is a newly introduced VQA
evaluation protocol for diagnosing how reliant VQA models are on shortcuts.

Baseline architecture. Similar to recent works, we adopt a popular VQA
baseline architecture UpDn Anderson et al. (2018) as both our ensemble bias
model $F_{b}$ and our target model $F$. We list the details of the generator
and discriminator in the supplementary material. During training, we train
both the bias model and target model together, then we use the target model
only for inference.

###  4.2 Results on VQA-CP2 and VQA-CP1

We compare how our method GenB performs in relation to the recent state-of-
the-art methods that focus on bias reduction as shown in Table 1. For VQA-CP2,
we first list the _baseline architectures_ in the first section. Then, we
compare our method to the methods that _modify language modules_ (DLR Jing et
al. (2020), VGQE KV and Mittal (2020)), _strengthen visual attention_ (HINT
Selvaraju et al. (2019), SCR Wu and Mooney (2019)), _ensemble based methods_ ,
(AReg Ramakrishnan et al. (2018), RUBi Cadene et al. (2019), LMH Clark et al.
(2019), CF-VQA Niu et al. (2021), and GGE Han et al. (2021)) and _balance
training data_ by changeing the training distribution (CVL Abbasnejad et al.
(2020), RandImg Teney et al. (2020), SSL Zhu et al. (2021), CSS Chen et al.
(2020a), Mutant Gokhale et al. (2020), and D-VQA Wen et al. (2021)) in the
respective sections. Among the balancing training data methods, while some
methods swap the image and questions from the supposed pairs Teney et al.
(2020); Zhu et al. (2021); Wen et al. (2021), counterfactuals based methods
generate counterfactual samples masking critical words or objects Chen et al.
(2020a) or by using an external inpainting network to create a new subset of
data Gokhale et al. (2020). Our model (GenB) is in line with the ensemble
models listed. Following previous works Niu et al. (2021), we do not compare
to methods that change training distribution as these methods conflict with
the original intent of VQA-CP (which is to test whether VQA models rely on
prior training data Agrawal et al. (2018)) and are listed in the bottom of
Table 1.

In Table 1, our method achieves the state-of-the-art performance on the VQA-
CP2 dataset, surpassing the second best (GGE Han et al. (2021)) by 1.43%. The
performance of our model on all three categories (“Yes/No,” “Num,” and
“Other”) are within the top-3 consistently. Our method also performs highly
favorably in the “Other” metric.

We also show how our method performs on the VQA-CP1 dataset, which is a subset
of the VQA-CP2 dataset. Note that not all of the baselines are listed as we
only list the scores that are made available in the respective papers. Our
method also shows the state-of-the-art results on this dataset with a
significant performance improvement over the second best among the methods
compared, CF-VQA(SUM) Niu et al. (2021). Even when compared to the available
method CSS Chen and Dolan (2011) that we do not compare as it is considered
unfair according to Niu et al. (2021), our method shows a significant gap in
performance. Compared to the second best method, CF-VQA(Sum) Niu et al. (2021)
on UpDn, our method improves the overall performance by 5.35% while also
having the best performance in both “Num” and “Other” category, by 3.28% and
2.41% performance improvements, respectively.

###  4.3 Results on GQA-OOD

In light of recent revelations, we further evaluate our method on the new VQA
debiasing dataset, GQA-OOD Kervadec et al. (2021) and list our results in
Table 2. We compare our method to available recent state-of-the-art methods
RUBi Cadene et al. (2019), LMH Clark et al. (2019), and CSS Chen et al.
(2020a). Our method shows the best performance in all the metrics compared to
the state-of-the-art methods by a significant margin. Even when compared to
methods that show similar performance to GenB in VQA-CP2 like CSS, GenB
significantly outperforms it in GQA-OOD by 5.19% in overall. Interestingly,
although all of the listed previous methods outperform the base model UpDn in
other datasets, they show a performance degradation on GQA-OOD. Unlike these
methods, our method GenB is able to have an increase in performance, showing
the robustness of GenB.

###  4.4 Results on VQA-CE

We also evaluate our method on the newly introduced evaluation protocol that
measures how much a VQA model depends on shortcuts called VQA-CounterExamples
(VQA-CE) Dancette et al. (2021) in Table 3. This dataset is based on the
VQA-v2 dataset and lists Overall, Easy, and Counter, which are the total score
on VQA-v2 validation set, the subset of samples where the shortcuts of
image/question give the correct answer, and the subset of samples where the
shortcuts of image/question give incorrect answers respectively. Although the
overall score for VQA-CE is dependent on the VQA-v2 performance, we show that
our model shows the best performance on Counterexamples, which is the main
point of interest in this dataset.

Method | GQA-OOD Test
---|---
All | Tail | Head | Avg
UpDn Anderson et al. (2018) | 46.87 | 42.13 | 49.16 | 45.65
RUBi Cadene et al. (2019) | 45.85 | 43.37 | 47.37 | 45.37
LMH Clark et al. (2019) | 43.96 | 40.73 | 45.93 | 43.33
CSS Chen et al. (2020a) | 44.24 | 41.20 | 46.11 | 43.66
GenB (Ours) | 49.43 | 45.63 | 51.76 | 48.70

Table 2: Experimental results on the GQA-OOD dataset. Our method shows the
best performance in all the metrics compared to the state-of-the-art methods
by a significant margin. The results show that our method robust on GQA-OOD as
well.

Method | VQA-CE
---|---
Overall | Counter | Easy
UpDn Anderson et al. (2018) | 63.52 | 33.91 | 76.69
RUBi Cadene et al. (2019) | 61.88 | 32.25 | 75.03
LMH Clark et al. (2019) | 61.15 | 34.26 | 73.13
RandImg Teney et al. (2020) | 63.34 | 34.41 | 76.21
CSS Chen et al. (2020a) | 53.55 | 34.36 | 62.08
GenB (Ours) | 57.87 | 34.80 | 68.15

Table 3: Evaluation on the VQA-CE protocol. Ours shows the best performance in
counterexamples (listed as Counter) which is the main scope of the VQA-CE.

###  4.5 Ablation Studies

Our method (GenB) includes several different components as shown from Sec. 3.3
to Sec. 3.5. To understand the effects of each component, we run an ablation
study on the VQA-CP2 dataset. For all experiments, the results are of the
target model and as the purpose of the bias model is to capture bias instead
of correctly predicting the answers, we do not consider the predictions of the
bias model. For all our ablation tables, we also add the UpDn baseline in the
first row, the model in which our target model and bias model is based for
comparison.

Training Loss | Bias Model | VQA-CP2
---|---|---
All | Yes/No | Num | Other
BCE | UpDn | 39.94 | 42.46 | 11.93 | 45.09
BCE | GenB | 56.98 | 88.82 | 19.39 | 49.86
BCE + DSC | GenB | 56.54 | 89.06 | 21.29 | 49.79
BCE + Distill | GenB | 57.06 | 88.91 | 23.24 | 49.65
BCE + DSC + Distill | GenB | 58.75 | 88.91 | 35.37 | 49.37

Table 4: We ablate the different losses we use to train the GenB bias model.
All inferences scores are based on the target model except the first row. BCE
loss is Eq. (1), which is the ground truth VQA loss. DSC refers to the
discriminator loss Eq. (4) and Distill refers to the KL Divergence loss Eq.
(3). Although the DSC and Distill losses independently do not show large
improvement, our final model with all losses show a large margin of
improvement.

Ensemble Debias Loss | Bias Model | VQA-CP2
---|---|---
All | Yes/No | Num | Other
– | UpDn | 39.94 | 42.46 | 11.93 | 45.09
GGE Han et al. (2021) | UpDn | 47.40 | 64.45 | 13.96 | 47.64
Our Loss | UpDn | 52.47 | 88.20 | 30.09 | 40.38
RUBi Cadene et al. (2019) | GenB | 30.77 | 72.78 | 12.15 | 13.87
LMH Clark et al. (2019) | GenB | 53.99 | 75.89 | 44.62 | 45.08
GGE Han et al. (2021) | GenB | 49.51 | 70.63 | 14.08 | 48.16
Ours Loss | GenB | 58.75 | 88.91 | 35.35 | 49.37

Table 5: Ablation of ensemble debiasing loss functions and our debiasing loss
function Eq. (5). Note that our loss improves the score of using the vanilla
UpDn model when compared to GGE by a large margin. Note that GenB works best
with our loss and shows a large performance improvement from GGE + GenB.

Bias model training analysis. We ablate on the different losses we use to
train the bias model from the target model and list our findings in Table 4
with BCE denoting the ground truth VQA loss Eq. (1), DSC denoting the
discriminator loss Eq. (4), and Distill denoting the KL Divergence
distillation loss Eq. (3). GenB trained on the BCE loss is already
significantly helpful in debiasing the target model. Adding the DSC and
Distill losses show similar performance to the GenB trained with BCE loss in a
standalone manner. As the bias model and target model are trained
concurrently, without a ground truth anchor, we conjecture that the bias model
struggles to learn any meaningful bias. Moreover, combining all the proposed
loss functions shows the best overall performance among all the baselines.
Note that BCE + DSC shows the best Yes/No score while our final model shows
the best score in Num score. We conjecture that the bias model trained with
our adversarial training framework is good at modeling the bias for simple
answers like Yes/No while the bias with more complex answers like counting is
hard to learn only with the adversarial training. We conclude that combining
our knowledge distillation loss to the adversarial training is essential to
learn the bias with complex answers including the Num category.

Figure 3: Qualitative results showing the predictions and attention scores of
our target model and the bias model. We show the attention of the bias model
with different noise to see where it would attend to. We also input the real
image into the bias model to visualize the attention. The bias model’s
attention changes each time and does not attend to the critical areas of the
image.

Debiasing loss function analysis. We list in Sec. 3.5 the different losses we
can use for debiasing and we show our results in Table 5. We also list our new
debiasing loss function Eq. (5) and how it performs when a vanilla UpDn model
is used instead of the GenB.

We analyze how GGE’s loss function fairs when the vanilla UpDn model is used
as a bias model. In the second and the third row, although the GGE loss
improves the score from the baseline by introducing a bias model, our loss
exceeds the performance by 5.07%. In the last two rows, we also show the
debiasing losses with the full GenB model with all of the bias model training
losses and show 8.81% performance gap. Given the GGE loss, the performance
increase from the bias model is only 2.11% whereas our debiasing loss function
is able to boost the score from the vanilla UpDn to GenB by 5.85%. We
conjecture that as our loss function accounts for the amount of bias a model
captures, and the generative bias model is able to stochastically model the
bias, our loss function captures the bias in a much more dynamic and helpful
manner.

###  4.6 Qualitative Results

We visualize the attention scores and predictions of our target and bias
models in Fig. 3. We run the bias model three times with different noise to
show the varying attention scores and biased predictions. We also input the
corresponding image with the question to see where the model would attend, and
we find that the attention is once again random as the image is previously
unseen. The bias model’s predictions change each time as a different noise is
given as expected. Even though the question type is the same (_e.g_., what
color), the biased model’s predictions are noticeably different. Moreover, we
find the target model is able to correctly attend to the salient region while
predicting the correct answer by virtue of the bias model.

##  5 Conclusion

In this paper, we started with this intuition “the better the bias model, the
better we can debias the target model. Then how can we best model the bias?”
In response, we present simple, effective, and novel generative bias model
that we call GenB. We use this generative model to learn the bias that may be
inhibited by the target model with the aid of generative networks, adversarial
training, and knowledge distillation. In addition, in conjunction with our
modified loss function, our novel bias model is able to debias our target
model, and our target model achieves state-of-the-art performance on various
bias diagnosing datasets including VQA-CP2, VQA-CP1, GQA-OOD, and VQA-CE among
methods that keep in line with the original intent of the task, and we believe
that this work can be extended to other multi-modal and uni-modal research in
understanding and mitigating bias.

## References

  * Abbasnejad et al. (2020) Ehsan Abbasnejad, Damien Teney, Amin Parvaneh, Javen Shi, and Anton van den Hengel. 2020.  Counterfactual vision and language learning.  In _CVPR_.
  * Agrawal et al. (2016) Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016.  Analyzing the behavior of visual question answering models.  In _EMNLP_.
  * Agrawal et al. (2018) Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018.  Don’t just assume; look and answer: Overcoming priors for visual question answering.  In _CVPR_.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _CVPR_.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  Vqa: Visual question answering.  In _ICCV_.
  * Cadene et al. (2019) Remi Cadene, Corentin Dancette, Hedi Ben-younes, Matthieu Cord, and Devi Parikh. 2019.  Rubi: Reducing unimodal biases in visual question answering.  In _NeurIPS_.
  * Chen and Dolan (2011) David L Chen and William B Dolan. 2011.  Collecting highly parallel data for paraphrase evaluation.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1_. ACL.
  * Chen et al. (2020a) Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. 2020a.  Counterfactual samples synthesizing for robust visual question answering.  In _CVPR_.
  * Chen et al. (2020b) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020b.  Uniter: Universal image-text representation learning.  In _ECCV_.
  * Clark et al. (2019) Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019.  Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases.  In _EMNLP_.
  * Dancette et al. (2021) Corentin Dancette, Remi Cadene, Damien Teney, and Matthieu Cord. 2021.  Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1574–1583.
  * Gokhale et al. (2020) Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. 2020.  Mutant: A training paradigm for out-of-distribution generalization in visual question answering.  In _EMNLP_.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014.  Generative adversarial nets.  In _NeurIPS_.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017\.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _CVPR_.
  * Gurari et al. (2018) Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018.  Vizwiz grand challenge: Answering visual questions from blind people.  In _CVPR_.
  * Han et al. (2021) Xinzhe Han, Shuhui Wang, Chi Su, Qingming Huang, and Qi Tian. 2021.  Greedy gradient ensemble for robust visual question answering.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1584–1593.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017.  Image-to-image translation with conditional adversarial networks.  In _CVPR_.
  * Jing et al. (2020) Chenchen Jing, Yuwei Wu, Xiaoxun Zhang, Yunde Jia, and Qi Wu. 2020.  Overcoming language priors in VQA via decomposed linguistic representations.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 11181–11188. AAAI Press.
  * Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _CVPR_.
  * Kervadec et al. (2021) Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf. 2021.  Roses are red, violets are blue… but should vqa expect them to?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2776–2785.
  * Kim et al. (2018) Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. 2018.  Bilinear attention networks.  In _NeurIPS_.
  * KV and Mittal (2020) Gouthaman KV and Anurag Mittal. 2020.  Reducing language biases in visual question answering with visually-grounded question encoder.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16_ , pages 18–34. Springer.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _arXiv preprint arXiv:1908.02265_.
  * Niu et al. (2021) Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. 2021.  Counterfactual vqa: A cause-effect look at language bias.  In _CVPR_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _EMNLP_.
  * Ramakrishnan et al. (2018) Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. 2018.  Overcoming language priors in visual question answering with adversarial regularization.  In _NeurIPS_.
  * Selvaraju et al. (2019) Ramprasaath R Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. 2019.  Taking a hint: Leveraging explanations to make vision and language models more grounded.  In _ICCV_.
  * Si et al. (2021) Qingyi Si, Zheng Lin, Ming yu Zheng, Peng Fu, and Weiping Wang. 2021.  Check it again: Progressive visual question answering via visual entailment.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4101–4110.
  * Su et al. (2019) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019\.  Vl-bert: Pre-training of generic visual-linguistic representations.  _arXiv preprint arXiv:1908.08530_.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  Lxmert: Learning cross-modality encoder representations from transformers.  _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Teney et al. (2020) Damien Teney, Kushal Kafle, Robik Shrestha, Ehsan Abbasnejad, Christopher Kanan, and Anton van den Hengel. 2020.  On the value of out-of-distribution testing: An example of goodhart’s law.  _arXiv preprint arXiv:2005.09241_.
  * Wen et al. (2021) Zhiquan Wen, Guanghui Xu, Mingkui Tan, Qingyao Wu, and Qi Wu. 2021.  Debiased visual question answering from feature and sample perspectives.  _Advances in Neural Information Processing Systems_ , 34.
  * Wu and Mooney (2019) Jialin Wu and Raymond J Mooney. 2019.  Self-critical reasoning for robust visual question answering.  In _NeurIPS_.
  * Yang et al. (2016) Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. 2016.  Stacked attention networks for image question answering.  In _CVPR_.
  * Zhang et al. (2016) Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016\.  Yin and yang: Balancing and answering binary visual questions.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5014–5022.
  * Zhu et al. (2021) Xi Zhu, Zhendong Mao, Chunxiao Liu, Peng Zhang, Bin Wang, and Yongdong Zhang. 2021\.  Overcoming language priors with self-supervised learning for visual question answering.  _IJCAI_.