# Quantization of Length in Spaces with Position-Dependent Noncommutativity

Jishnu Aryampilly<EMAIL_ADDRESS>Department of Physics,
Pondicherry University,
Puducherry 605014, India  Muthukumar Balasundaram<EMAIL_ADDRESS>Department of Physics,
Pondicherry University,
Puducherry 605014, India  Aamir Rashid<EMAIL_ADDRESS>Department
of Physics,
Pondicherry University,
Puducherry 605014, India

(Dated: )

###### Abstract

We present a novel approach to quantizing the length in noncommutative spaces
with positional-dependent noncommutativity. The method involves constructing
ladder operators that change the length not only along a plane but also along
the third direction due to a noncommutative parameter that is a combination of
canonical/Weyl-Moyal type and Lie algebraic type. The primary quantization of
length in canonical-type noncommutative space takes place only on a plane,
while in the present case, it happens in all three directions. We establish an
operator algebra that allows for the raising or lowering of eigenvalues of the
operator corresponding to the square of the length. We also attempt to
determine how the obtained ladder operators act on different states and work
out the eigenvalues of the square of the length operator in terms of
eigenvalues corresponding to the ladder operators. We conclude by discussing
the results obtained.

##  1 Introduction

In the expansive realm of fundamental physics, the notion of spacetime stands
as a bedrock upon which our understanding of the universe is constructed.
Conventionally, the principles of classical physics have offered a framework
to describe spacetime where space and time coordinates exhibit commutativity,
enabling precise measurements of position and duration. Nevertheless, emerging
theories and frameworks have heralded a departure from this classical
paradigm, revealing the intriguing prospects of a noncommutative spacetime.
The idea of noncommutativity in the coordinates of spacetime can be attributed
to Heisenberg [1], and it was subsequently developed further by Snyder [2] as
a means to address the issue of divergences in quantum field theory.
Noncommutative spacetime theory suggests that space and time coordinates do
not commute but instead exhibit a fundamental uncertainty or noncommutativity
relation. This revelation necessitates acknowledging that precise measurements
of both position and time are inherently uncertain at infinitesimal scales.
This departure from classical notions has sparked considerable interest and
led to the formulation of various theoretical models attempting to capture the
essence of noncommutative spacetime [3, 4, 5]. The growing interest in
noncommutative spaces is connected to the prediction of noncommutative
structures in string theory and loop quantum gravity [6, 7, 8, 9]. The
literature on noncommutative theories is replete in the contexts of quantum
field theories [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], quantum mechanics
[21, 22, 23, 24, 25, 26, 27, 28, 29, 30] and gravity theories [31, 32, 33, 34,
35, 36].

In the case of noncommutative geometry by Alain Connes [37, 38], the spectral
manifold is shown to have a geometric analog of the Heisenberg commutation
relation involving the Dirac operator and Feynman slash of real scalar fields,
leading to the quantization of volume [39]. The idea of length as an operator
has already been discussed in canonical quantum gravity [40]. Loop quantum
gravity has developed the rigorous construction of (spatial) geometrical
operators, such as the area and the volume[41, 42].

Within the context of noncommutative spaces, the idea of length as an operator
was proposed in [43], where it was shown to result in the quantization of
length in the noncommutative space with the canonical/Weyl-Moyal type
noncommutativity

$[\hat{x}^{\mu},\hat{x}^{\nu}]=i\theta^{\mu\nu}$ (1)

among the coordinates where $\theta^{\mu\nu}$ is a constant and real
antisymmetric matrix. The motivation behind such a proposal was the following.
The noncommutativity of spatial coordinates is closely linked to the existence
of a minimum length within a system. This minimum length is commonly
associated with inherent uncertainties in distance measurements. Rather than
associating the minimum length with uncertainties, it can be attributed to the
minimum value of the quantized length. If length is taken as an operator with
a spectrum of eigenvalues, then a set of ladder operators may exist to go from
one eigenstate to another such that

$[\hat{L}^{2},\hat{a}]=\lambda\,\hat{a},$ (2)

where $\hat{L}^{2}$ is the operator corresponding to the square of length and
$\hat{a}$ and $\hat{a}^{\dagger}$ being the ladder operators. If
$\hat{a}=\sum_{\mu}\alpha_{\mu}\,\hat{x}^{\mu}$, where $\alpha_{\mu}$’s are
complex constants, then Eq.(2) leads to an eigenvalue equation which, in the
case of 3-D with $\theta^{12}=\theta^{13}=\theta^{23}=\theta$, gives three
real eigenvalues and their corresponding eigenvectors. Two of the eigenvectors
give the required ladder operators that change the length in the plane formed
by them. The third eigenvector points in the direction normal to that plane
and the length is not quantized along this normal direction[43].

In this paper, we follow an approach similar to the operator methods in the
quantum harmonic oscillator and the angular momentum problems and apply it to
the case of position-dependent noncommutativity. Position-dependent
noncommutativity has been discussed before in [44, 45, 47, 46, 48]. In
particular, the noncommutative parameter used in our approach is the following
combination of the canonical/Weyl-Moyal type and Lie algebraic type:

$[\hat{x}^{\mu},\hat{x}^{\nu}]=i(\theta^{\mu\nu}+B^{\mu\nu}_{\phantom{~{}~{}}\rho}\,\hat{x}^{\rho}),$
(3)

where $\theta^{\mu\nu}$ corresponds to a constant and real antisymmetric
matrix and $B^{\mu\nu\rho}$ is real and completely antisymmetric.

The paper is organized as follows. In Section 2, we establish an operator
algebra of the length-square operator that allows for the raising and lowering
of eigenvalues of $\hat{L}^{2}$. In Section 3, we apply this approach in a
3-dimensional space. We construct a commutation relation between the operator
$\hat{L}^{2}$ corresponding to the square of length and the ladder operators
analogous to the commutation relation between the Hamiltonian of the harmonic
oscillator and its raising/lowering operator. We work with this commutation
relation to obtain an eigenvalue equation and consequently construct a set of
operators $\hat{a}_{-}$, $\hat{a}_{+}$ and $\hat{b}$. Once we have obtained
the operators, we adopt them to construct a ladder of states that constitute
the eigenstates of the $\hat{L}^{2}$ operator. In Section 4, we work out the
eigenvalues of $\hat{L}^{2}$ in terms of eigenvalues corresponding to the
operators $\hat{a}_{+}\hat{a}_{-}$ and $\hat{b}$. The actions of $\hat{a}_{-}$
and $\hat{a}_{+}$ change not only the eigenvalues of $\hat{L}^{2}$ but also
the eigenvalues of $\hat{b}$, and in this way, the length is quantized along
the direction of $\hat{b}$ also. In Section 5, by introducing another operator
$\hat{K}$ that commutes with the ladder operators, we investigate the system’s
degeneracy further. We conclude in Section 6.

##  2 Construction of Ladder Operators

We establish an operator algebra in a manner that allows for the raising or
lowering of eigenvalues of the operator $\hat{L}^{2}$. We define the operator
corresponding to the square of the distance as

$\hat{L}^{2}=g_{\mu\nu}\hat{x}^{\mu}\hat{x}^{\nu},$ (4)

where $g_{\mu\nu}$ is a constant symmetric metric of $D$-dimensional spacetime
and Einstein’s summation convention is used over the repeated indices $\mu$
and $\nu$ which take the values $1,2,\ldots D$. The prescription to construct
a set of ladder operators $\\{\hat{a}^{\mu}\\}$ is that they satisfy the
following commutation relation:

$[\hat{L}^{2},\hat{a}^{\mu}]=\lambda\,\hat{a}^{\mu},$ (5)

where $\lambda$ is a constant to be determined and $\hat{a}^{\mu}$ is linearly
related to $\hat{x}^{\mu}$ as

$\hat{a}^{\mu}=U^{\mu}_{\phantom{\mu}\nu}\,\hat{x}^{\nu},$ (6)

where $U^{\mu}_{\phantom{\mu}\nu}$ is the transformation matrix. Substituting
Eq.(4) and Eq.(6) into Eq.(5) and using Eq.(3) leads to the following operator
equation:

$2i\,\theta_{\mu}^{\phantom{\mu}\rho}\,U^{\sigma}_{\phantom{\mu}\rho}\,\hat{x}^{\mu}\,+\,i\,U^{\sigma}_{\phantom{\mu}\rho}\,B_{\nu\phantom{\rho}\kappa}^{\phantom{\nu}\rho}\,(\hat{x}^{\nu}\,\hat{x}^{\kappa}\,+\,\hat{x}^{\kappa}\,\hat{x}^{\nu})=\lambda\,U^{\sigma}_{\phantom{\mu}\mu}\,\hat{x}^{\mu}.$
(7)

Since $(\hat{x}^{\nu}\,\hat{x}^{\kappa}\,+\,\hat{x}^{\kappa}\,\hat{x}^{\nu})$
is symmetric and $B_{\nu\phantom{\rho}\kappa}^{\phantom{\nu}\rho}$ is
antisymmetric under the exchange of $\nu$ and $\kappa$, the second term is
zero which leads to the eigenvalue equation for the transformation matrix:

$2i\,\theta_{\mu}^{\phantom{\mu}\rho}\,U^{\sigma}_{\phantom{\mu}\rho}\,=\lambda\,U^{\sigma}_{\phantom{\mu}\mu}\,.$
(8)

If
$\hat{X}^{\dagger}=\begin{pmatrix}\hat{x}^{1},\hat{x}^{2},\ldots\hat{x}^{D}\end{pmatrix}$
and $g$ is the matrix form of the metric tensor, then

$\hat{L}^{2}=\hat{X}^{\dagger}\,g\,\hat{X}.$ (9)

To relate $\hat{L}^{2}$ to the ladder operators we define
$\hat{A}^{\dagger}=\begin{pmatrix}\hat{a}^{1\dagger},\hat{a}^{2\dagger},\ldots\hat{a}^{D\dagger}\\\
\end{pmatrix}$. Going by the analogy with harmonic oscillator and angular
momentum problems, the ladder operators will be useful only if the length
operator is related to number operators
$(\hat{a}^{1}){}^{\dagger}\hat{a}^{1},\,(\hat{a}^{2}){}^{\dagger}\hat{a}^{2},\ldots$.
Therefore, we require $\hat{L}^{2}$ in the following form:

$\hat{L}^{2}=\frac{1}{\gamma}\,\hat{A}^{\dagger}g\hat{A}=\frac{1}{\gamma}\,\hat{X}^{\dagger}U^{\dagger}\,g\,U\hat{X},$
(10)

where $\gamma$ is a constant. Comparing Eq.(9) and Eq.(10), we get following
condition for the transformation matrix:

$U^{\dagger}\,g\,U=\gamma\,g.$ (11)

##  3 The Length Operator in 3-D Space

We attempt to apply our approach to a 3-dimensional space. For this case, the
commutation relation that we use can therefore be defined as

$[\hat{x}^{i},\hat{x}^{j}]=i(\theta^{ij}+B^{ij}_{\phantom{ij}k}\,\hat{x}_{k}),$
(12)

We define the ladder operator as

$\hat{a}=\alpha_{k}\hat{x}^{k},$ (13)

where Einstein’s summation convention is implied and $\alpha_{k}$’s are
complex constants to be determined. The operator corresponding to the square
of the length, in this case, thus becomes

$\hat{L}^{2}=g_{ij}\hat{x}^{i}\hat{x}^{j}$ (14)

As discussed, we assume the following relation in analogy with the angular
momentum operator in quantum mechanics

$[\hat{L}^{2},\hat{a}]=\lambda\hat{a}.$ (15)

Then, the substitution of Eq.(14) and Eq.(13) in Eq.(15) and using the
commutator in Eq.(12) gives the relation

$2i\theta^{k}_{\phantom{k}m}\alpha_{k}=\lambda\alpha_{m},$ (16)

where $\theta^{km}$ is a constant antisymmetric matrix and $g_{ij}$ is assumed
to be diag(1,1,1).

We assume the entries of $\theta^{km}$ to be
$\theta^{12}=\theta^{13}=\theta^{23}=\theta$, which leads Eq.(16) to give the
nontrivial eigenvalues for $\lambda=\pm{2\sqrt{3}\theta}$. The third trivial
solution is $\lambda=0$. The set of values for $\alpha_{i}$ corresponding to
$\lambda=-{2\sqrt{3}\theta}$ is worked out to be
$(\alpha_{1},\,\alpha_{2},\,\alpha_{3})=(\rho\sigma,-\rho\sigma^{*}\rho)$,
where $\rho=\displaystyle e^{i\delta_{1}}$ and $\sigma=-e^{i\pi/3}$. The
operator $\hat{a}$ corresponding to this negative $\lambda$ is denoted by
$\hat{a}_{-}$. It is identified with the lowering operation by comparing
Eq.(15) with a negative $\lambda$ to an analogous relation in the harmonic
oscillator problem. The lowering operator can then be expressed as

$\displaystyle\hat{a}_{-}=\rho\left[\sigma\hat{x}^{1}-\sigma^{*}\hat{x}^{2}+\hat{x}^{3}\right].$
(17)

The eigenvector for the positive value $\lambda={2\sqrt{3}\theta}$ leads to
the raising operator $\hat{a}_{+}=(\hat{a}_{-})^{\dagger}$. The eigenvector
corresponding to the third eigenvalue, that is, the trivial solution
$\lambda=0$, leads to the operator $\hat{b}=\beta_{i}\,\hat{x}^{i}$ with
$(\beta_{1},\beta_{2},\beta_{3})$ = $(1,-1,1)$. With these denotations, we
write the Hermitian conjugate of the basis $\hat{A}$ in Eq.(10) as
$\hat{A}^{\dagger}=(\hat{a}_{+},\hat{a}_{-},\hat{b})$. The explicit values of
$\alpha_{i}$ in Eq.(16) and $\beta_{i}$ have the properties such as
$\alpha_{i}\alpha^{i}=0$, $\alpha_{i}\,\beta^{i}=\alpha_{i}^{*}\,\beta^{i}=0$
and $\theta^{ij}\beta_{j}=0$.

Essentially, the three eigenvalues for $\lambda$ in Eq.(15) lead to following
commutators

$[\hat{L}^{2},\hat{a}_{\pm}]=\pm 2\sqrt{3}\theta\hat{a}_{\pm},$ (18)

and

$[\hat{L}^{2},\hat{b}]=0.$ (19)

Also, the commutation relations among the ladder operators $\hat{a}_{-}$,
$\hat{a}_{+}$ and $\hat{b}$ are obtained as

$\displaystyle[\hat{a}_{-},\hat{a}_{+}]$
$\displaystyle=\sqrt{3}(3\theta+B\hat{b}),$ (20)
$\displaystyle[\hat{b},\hat{a}_{\pm}]$
$\displaystyle=\mp\sqrt{3}\,B\,\hat{a}_{\pm},$ (21)
$\displaystyle[\hat{a}_{+}\hat{a}_{-},\hat{a}_{+}]$
$\displaystyle=\sqrt{3}(3\theta+B\hat{b}+\sqrt{3}B)\hat{a}_{+},$ (22)
$\displaystyle[\hat{a}_{+}\hat{a}_{-},\hat{a}_{-}]$
$\displaystyle=-\sqrt{3}(3\theta+B\hat{b})\hat{a}_{-},$ (23)

With these commutators, the operator form of the square of length is expressed
as,

$\hat{L}^{2}=\frac{1}{3}[2\hat{a}_{+}\hat{a}_{-}+\sqrt{3}(3\theta+B\hat{b})+\hat{b}^{2}].$
(24)

The Eq.(18) leads to $[\hat{L}^{2},\hat{a}_{+}\hat{a}_{-}]=0$ and since
$[\hat{L}^{2},\hat{b}]=0$, it is possible to construct a complete set of
simultaneous eigenstates of $\hat{L}^{2}$, $\hat{a}_{+}\hat{a}_{-}$ and
$\hat{b}$.

##  4 Eigenvalues of the Length-Square Operator

Let us start with an eigenstate $|n\rangle$ of the number operator
$\hat{a}_{+}\hat{a}_{-}$ and consider the action of the ladder operators on
it. From Eq.(22) and Eq.(23), it is clear that the action of $\hat{a}_{\pm}$
on $|n\rangle$ is to raise/lower the eigenvalue of $\hat{a}_{+}\hat{a}_{-}$.
So, we define the actions of $\hat{a}_{-}$, $\hat{a}_{+}$ and $\hat{b}$
respectively on the normalized state $|n\rangle$ as

$\displaystyle\hat{a}_{-}|n\rangle$ $\displaystyle=h_{1}(n)|n-1\rangle,$ (25)
$\displaystyle\hat{a}_{+}|n\rangle$ $\displaystyle=h_{2}(n)|n+1\rangle,$ (26)
$\displaystyle\hat{b}|n\rangle$ $\displaystyle=g(n)|n\rangle.$ (27)

Considering the Hermitian conjugate of Eq.(25), we obtain

$\langle n|\hat{a}_{+}=\langle n-1|h_{1}^{*}(n).$ (28)

Thus, it can easily be shown that

$\langle n|\hat{a}_{+}\hat{a}_{-}|n\rangle=\langle
n-1|h_{1}^{*}(n)h_{1}(n)|n-1\rangle={|{h_{1}(n)}|}^{2}.$ (29)

Similarly upon taking the Hermitian conjugate of Eq.(26), we can obtain

$\langle n|\hat{a}_{-}\hat{a}_{+}|n\rangle={|{h_{2}(n)}|}^{2}.$ (30)

It can be further rewritten in terms of commutator to obtain the relation
$\langle n|\hat{a}_{-}\hat{a}_{+}|n\rangle=\langle
n|(\hat{a}_{+}\hat{a}_{-}+[\hat{a}_{-},\hat{a}_{+}])|n\rangle$ and using the
commutator in Eq.(20), we obtain the relation

$\langle n|\hat{a}_{-}\hat{a}_{+}|n\rangle=\langle
n|(\hat{a}_{+}\hat{a}_{-}+3\sqrt{3}\theta+\sqrt{3}B\hat{b})|n\rangle.$ (31)

The above relation can be reexpressed using Eq.(29), Eq.(30) and Eq.(27) as

$|h_{2}(n)|^{2}=|h_{1}(n)|^{2}+3\sqrt{3}\theta+\sqrt{3}Bg(n)$ (32)

Considering the Hermitian conjugation of Eq.(25) with $n+1$ in place of $n$,
we have $\langle n+1|\hat{a}_{+}=\langle n|{h_{1}}^{*}(n+1)$. Therefore,
$\langle n|\hat{a}_{+}|n\rangle$ gives ${h_{1}}^{*}(n+1)$ on one hand but on
the other hand it gives ${h_{2}}(n)$, which leads to the relation

$|h_{2}(n)|^{2}=|h_{1}(n+1)|^{2}.$ (33)

Also, considering the action of $\hat{b}$ on the state $\hat{a}|n\rangle$ and
by using the commutator in Eq.(21), we can find the relation between $g(n+p)$
and $g(n)$ for any integer $p$

$g(n+p)=g(n)-pB\sqrt{3}.$ (34)

The Eq.(18) implies that the eigenvalues of $\hat{L}^{2}$ are decreased by
$\hat{a}_{-}$ by the amount $2\sqrt{3}\theta$. This decrease cannot go on
forever since $\hat{L}^{2}$ will take negative values, which is unphysical.
So, let us define a ground state of the system by $|\overline{n}\rangle$ such
that the action of the lowering operator $\hat{a}_{-}$ on it gives

$\hat{a}_{-}|\overline{n}\rangle=0.$ (35)

Here, we have $h_{1}(\overline{n})=0$ or

$|h_{1}(\overline{n})|^{2}=0.$ (36)

We can further extend our analysis using Eq.(32), Eq.(33) and Eq.(34) to
compute values of $h_{1}$ as

$\displaystyle|h_{1}(\overline{n}+1)|^{2}$
$\displaystyle=[3\sqrt{3}\theta+\sqrt{3}Bg(\overline{n})],$ (37)
$\displaystyle|h_{1}(\overline{n}+2)|^{2}$
$\displaystyle=2[3\sqrt{3}\theta+\sqrt{3}Bg(\overline{n})]-3B^{2}$ (38)

and so on. The analysis can be further extended to obtain

$|h_{1}(\overline{n}+m)|^{2}=m[3\sqrt{3}\theta+\sqrt{3}B(g(\overline{n})-\frac{(m-1)}{2}\sqrt{3}B)].$
(39)

For a general $n$, where $n=\overline{n}+m$ and $m\geq 0$, the above equation
modifies to a function involving $n$ and $\overline{n}$. In a similar manner,
the general form of (34) is, then, better expressed as a function of $n$ and
$\overline{n}$

$g(n)=g(\overline{n})-(n-\overline{n})\sqrt{3}B.$ (40)

where $g(\overline{n})$ comes from the operator $\hat{b}$ acting on the ground
state, that is,

$\hat{b}|\overline{n}\rangle=g(\overline{n})|\overline{n}\rangle.$ (41)

Accordingly, Eq.(39) can be rewritten as

$|h_{1}(n)|^{2}\\!=\\!(n-\overline{n})[3\sqrt{3}\theta+\sqrt{3}B(g(\overline{n})\\!-\\!\tfrac{(n-\overline{n}-1)}{2}\sqrt{3}B)].$
(42)

We can also find using Eq.(33) that

${|h_{2}(n)|}^{2}=(n-\overline{n}+1)[3\sqrt{3}\theta+\sqrt{3}B(g(\overline{n})\\!-\\!\tfrac{(n-\overline{n})}{2}\sqrt{3}B)]$
(43)

Since $|h_{1}(n)|^{2}$ and $|h_{2}(n)|^{2}$ cannot take negative values for
any $n$, we can infer that

$3\sqrt{3}\theta+\sqrt{3}Bg(\overline{n})\geq\frac{(n-\overline{n})}{2}3B^{2}.$
(44)

Suppose $\tilde{n}$ is the maximum value $n$ can take such that the above
inequality stands valid. Then, we can fix a top-most state $|\tilde{n}\rangle$
such that

$\hat{a}_{+}|\tilde{n}\rangle=0.$ (45)

Now, since $h_{2}(\tilde{n})=0$ and $\tilde{n}\geq\overline{n}$, using (43),
we can express $g(\overline{n})$ in terms of $\tilde{n}$ and $\overline{n}$.
Thus, we obtain

$g(\overline{n})=\frac{(\tilde{n}-\overline{n})}{2}\sqrt{3}B-\frac{3\theta}{B}.$
(46)

Putting Eq.(46) back into Eq.(44), we get $\tilde{n}\geq\overline{n}+m$. So
essentially $n=\overline{n},~{}\overline{n}+1,~{}\ldots,\tilde{n}$ and
$\tilde{n}=\overline{n},~{}\overline{n}+1,\ldots.$ But there is no restriction
on $\overline{n}$, and it can take both positive and negative integer values.

We may now proceed to solve for the eigenvalues of $L^{2}$. The length-square
operator in Eq.(24) leads to the following eigenvalue equation

$\displaystyle\hat{L}^{2}|n\rangle=\frac{1}{3}\big{(}2|h_{1}(n)|^{2}+3\sqrt{3}\theta+\sqrt{3}Bg(n)+g^{2}(n)\big{)}|n\rangle.$
(47)

Using Eq.(42) and Eq.(40), we can reexpress the form of the eigenvalue of the
length square operator as

$\displaystyle\hat{L}^{2}|n\rangle=\frac{1}{3}\big{(}[2(n-\overline{n})+1]3\sqrt{3}\theta+\sqrt{3}Bg(\overline{n})+g^{2}(\overline{n})\big{)}|n\rangle.$
(48)

From the expression for $g(\overline{n})$ deduced in Eq.(46), it is evident
that $g(\overline{n})$ is dependent on both $\tilde{n}$ and $\overline{n}$. In
consequence, we also find that both $|h_{1}(n)|^{2}$ and $|h_{2}(n)|^{2}$ now
depend on $\overline{n}$ and $\tilde{n}$ in addition to $n$. But, $\tilde{n}$
and $\overline{n}$ do not necessarily take fixed values and could assume
different values such that Eq.(44) is obeyed. In light of this, we now see
that the state of the system should depend on $n$, $\tilde{n}$ and
$\overline{n}$. As a result, we can infer that we will require three indices
to express the system’s state, although the eigenvalues of $\hat{L}^{2}$
depend only on $n-\overline{n}$ and $\tilde{n}-\overline{n}$. The state of the
system can then better be represented as $|\tilde{n},\overline{n},n\rangle$.
Now, employing Eq.(46) in Eq.(48) and further simplifying, the eigenvalue of
the length square operator thus emerges to be

$\displaystyle\hat{L}^{2}|\tilde{n},\overline{n},n\rangle=\big{(}[2n\\!-\\!\tilde{n}\\!-\\!\overline{n}]\sqrt{3}\theta\\!+\\!\tfrac{B^{2}}{4}(\tilde{n}-\overline{n})(\tilde{n}-\overline{n}+2)+\tfrac{3{\theta}^{2}}{B^{2}}\big{)}|\tilde{n},\overline{n},n\rangle.$
(49)

Fundamentally, we have derived the eigenvalues of $\hat{L}^{2}$ by expressing
them in relation to the eigenvalues associated with $\hat{a}_{+}\hat{a}_{-}$
and $\hat{b}$. The length is quantized in all directions in contrast to [43],
where the length was quantized only in a plane.

It can be seen that if $\theta$ is assigned the value of 0 in Eq.(49), the
length-square equation reduces to the form of the angular momentum problem
with the angular momentum quantum number $l$ analogous to the value
$\frac{\tilde{n}-\overline{n}}{2}$ and $\hbar$ analogous to $B$.

Note that $n$ appears only in the first term which is square-bracketed in
Eq.(49) and since $n$ takes the values from $\overline{n}$ to $\tilde{n}$,
this first term in the cases when $n=\overline{n}$ and $n=\tilde{n}$
respectively involves $-[\tilde{n}-\overline{n}]$ and
$[\tilde{n}-\overline{n}]$. Therefore, the minimum of $\hat{L}^{2}$ occurs
when $n=\overline{n}$ since $\tilde{n}\geq\overline{n}$, and this minimum is
given by the equation

$\displaystyle\hat{L}^{2}|\tilde{n},\overline{n},\overline{n}\rangle=\big{(}-[\tilde{n}\\!-\\!\overline{n}]\sqrt{3}\theta\\!+\\!\tfrac{B^{2}}{4}(\tilde{n}-\overline{n})(\tilde{n}-\overline{n}+2)+\tfrac{3{\theta}^{2}}{B^{2}}\big{)}|\tilde{n},\overline{n},\overline{n}\rangle.$
(50)

If $\tilde{n}=\overline{n}$, the eigenvalue of $\hat{L}^{2}$ is
$\tfrac{3{\theta}^{2}}{B^{2}}$. But the eigenvalue of $\hat{L}^{2}$ can be
lower than $\tfrac{3{\theta}^{2}}{B^{2}}$ if
$\big{(}-[\tilde{n}\\!-\\!\overline{n}]\sqrt{3}\theta\\!+\\!\tfrac{B^{2}}{4}(\tilde{n}-\overline{n})(\tilde{n}-\overline{n}+2)\big{)}<0$
i.e., if

$\tilde{n}<\overline{n}+(\tfrac{4\sqrt{3}\theta}{B^{2}}-2).$ (51)

It is clear from Eq.(51) that if $(\frac{4\sqrt{3}\theta}{B^{2}}-2)\leq 0$,
then the inequality is violated and there cannot be a minimum lower than
$\tfrac{3{\theta}^{2}}{B^{2}}$, i.e. if
$\frac{\theta}{B^{2}}\leq\frac{1}{2\sqrt{3}}$, then the minimum eigenvalue of
$\hat{L}^{2}$ is $\tfrac{3{\theta}^{2}}{B^{2}}$.

On the other hand, when $\tilde{n}=\overline{n}+m$, where $m>0$, the condition
that the eigenvalue of $\hat{L}^{2}$ should be greater than or equal to zero
leads to a real inequality relationship between $\tilde{n}$ and $\overline{n}$
only if $\frac{\theta}{B^{2}}\leq\frac{1}{4\sqrt{3}}$ which makes Eq.(51)
inconsistent with $\tilde{n}=\overline{n}+m$ for positive $m$ and so
$(-[\tilde{n}\\!-\\!\overline{n}]\sqrt{3}\theta\\!+\\!\tfrac{B^{2}}{4}(\tilde{n}-\overline{n})(\tilde{n}-\overline{n}+2))$
cannot be less than zero in a consistent way. Therefore, the minimum
eigenvalue of $\hat{L}^{2}$ is $\tfrac{3{\theta}^{2}}{B^{2}}$. This is also
clear from Eq.(48) since the minimum of Eq.(46) is $-\frac{3\theta}{B}$ and
the minimum of $n-\overline{n}$ is 0.

##  5 Degeneracy of States

The dependency of the state of the system on more than one index can be better
understood by the construction of another operator $\hat{K}$ that commutes
with $\hat{a}_{\pm}$ and $\hat{b}$. Since $[\hat{L}^{2},\hat{a}_{\pm}]=\pm
2\sqrt{3}\theta\hat{a}_{\pm}$ and
$[\hat{b},\hat{a}_{\pm}]=\mp\sqrt{3}\,B\,\hat{a}_{\pm}$, the linear
combination of $\hat{L}^{2}$ and $\hat{b}$, $B\hat{L}^{2}+2\theta\hat{b}$,
commutes with $\hat{a}_{\pm}$. But with little changes, we construct the
following linear combination

$\hat{K}=\hat{L}^{2}+\frac{2\theta}{B}\hat{b}+\sqrt{3}\theta+\frac{3\theta^{2}}{B^{2}}$
(52)

which also commutes with $\hat{a}_{\pm}$. Since $\hat{b}$ exhibits
commutativity with $\hat{L}^{2}$, we can summarise the commutator relations as

$\displaystyle[\hat{K},\hat{a}_{\pm}]$ $\displaystyle=0,$ (53)
$\displaystyle[\hat{K},\hat{b}]$ $\displaystyle=0.$ (54)

These equations, along with Eq.(21), form a set of equations analogous to the
operator algebra involving $\hat{\mathcal{L}}^{2}$, $\hat{\mathcal{L}}_{\pm}$
and $\hat{\mathcal{L}}_{z}$ in the angular momentum problem in quantum
mechanics. The eigenvalue of the operator $\hat{K}$ acting on the state
$|\tilde{n},\overline{n},n\rangle$ can be evaluated from Eq.(49) and Eq.(52)
as

$\hat{K}|\tilde{n},\overline{n},n\rangle=\left(\frac{B^{2}}{4}(\tilde{n}-\overline{n})(\tilde{n}-\overline{n}+2)+\sqrt{3}\,\theta\right)|\tilde{n},\overline{n},n\rangle.$
(55)

It is evident that the eigenvalue of the operator $\hat{K}$ is independent of
$n$. Also, its eigenvalues do not change if $\tilde{n}$ and $\overline{n}$ are
changed, keeping $\tilde{n}-\overline{n}$ fixed, resulting in huge degeneracy.

Figure 1: Degenerate states of operator $\hat{K}$ lying on the shaded-sphere

The physical meaning of $\hat{K}$ can be described in the following way. The
Eq.(10) represents the invariance of the length operator in going from the
basis $\hat{X}$ to the basis $\hat{A}$. Any constant eigenvalue of
$\hat{L}^{2}$ in Eq.(4) in the 3-D case would correspond to a sphere centered
at the origin. In the basis $\hat{A}$, the constant eigenvalue of
$\hat{L}^{2}$ in Eq.(24) would represent the same sphere centered at the
origin. Writing $\hat{K}$ in the form of the Eq.(24) leads to the expression

$\displaystyle\hat{K}=\frac{1}{3}[2\hat{a}_{+}\hat{a}_{-}+3\sqrt{3}\theta+\sqrt{3}B\hat{b}^{\prime}+{\hat{b}}^{\prime\,2}],$
(56)

where $\hat{b}^{\prime}=\hat{b}+\tfrac{3\theta}{B}$. If the constant
eigenvalue of Eq.(24) represents a sphere centered at the origin, then the
constant eigenvalue of Eq.(56) represents a sphere shifted along $\hat{b}$
axis by the amount $-3\theta/B$. In Figure 1, the sphere corresponding to a
constant eigenvalue of $\hat{L}^{2}$ is represented by the dashed circle
centered at $O$ and the shifted-sphere is represented as the shaded sphere
centered at $C$. The degenerate eigenstates of $\hat{K}$ lie on the surface of
this shifted sphere, but these states will have different eigenvalues for
$\hat{L}^{2}$ since $\hat{L}^{2}$ is measured from the origin $O$. While $CA$
and $CB$ corresponding to the eigenvalues of $\hat{K}$ are equal, $OA$ and
$OB$ corresponding to the eigenvalues of $\hat{L}^{2}$ are not equal. For
example, the states $|\tilde{n},\overline{n},n\rangle=|8,2,n\rangle$ with
different $n$’s have different eigenvalues for $\hat{L}^{2}$ in Eq.(49), but
they have the same eigenvalue for $\hat{K}$ in Eq.(55). But the states
$|8,2,5\rangle$ and $|9,3,6\rangle$ have the same eigenvalue for $\hat{L}^{2}$
and the same eigenvalue for $\hat{K}$. Such a common set of degenerate states
of $\hat{L}^{2}$ and $\hat{K}$ lie on a circle perpendicular to the $\hat{b}$
axis since the shift happens along the $\hat{b}$ axis.

In the basis $(\hat{x}^{1},\hat{x}^{2},\hat{x}^{3})$, the shifted sphere
corresponds to the shift along all three directions. In other words, if
$d^{i}$ denotes the shift along the $x^{i}$ direction, then defining the
operator $\hat{y}^{i}=\hat{x}^{i}+d^{i}$ such that
$B^{ij}_{\phantom{ij}k}d^{k}=\theta^{ij}$ in Eq.(12) would lead to the Lie
structure $[\hat{y}^{i},\hat{y}^{j}]=iB^{ij}_{\phantom{ij}k}\hat{y}^{k}$.
Although this structure would lead to the quantization of
$(\hat{y}^{1})^{2}+(\hat{y}^{2})^{2}+(\hat{y}^{3})^{2}$, our length operator
is different from this and in terms of $\hat{y}^{i}$, it is
$(\hat{y}^{1}-d^{1})^{2}+(\hat{y}^{2}-d^{2})^{2}+(\hat{y}^{3}-d^{3})^{2}$.
While the quantization method using $\hat{y}^{i}$ would employ raising and
lowering operations along $\hat{y}^{3}$ direction and keep the eigenvalues of
$(\hat{y}^{1})^{2}+(\hat{y}^{2})^{2}+(\hat{y}^{3})^{2}$ fixed, it can be
easily shown that $\hat{y}_{\pm}=\hat{y}^{1}\pm i\hat{y}^{2}$ neither
raises/lowers the eigenstates of $\hat{L}^{2}$ nor it commutes with
$\hat{L}^{2}$, calling for the approach that starts with the Eq.(5) to look
for the properly oriented $\hat{a}_{\pm}$ in place of $\hat{y}_{\pm}$.

##  6 Conclusion

In conclusion, we have explored length quantization in the context of
noncommutative spaces with position-dependent noncommutativity. Building upon
the formalism similar to the quantum harmonic oscillator and angular momentum
problems, we have constructed ladder operators and derived the operator
corresponding to the length-square in terms of these ladder operators. This
investigation has been specifically applied to the case of a 3-dimensional
space where the noncommutativity parameter involved a combination of
canonical/Weyl-Moyal type and Lie-type.

We have found that the length quantization in this scenario leads to distinct,
discrete eigenvalues for the length-square operator. The ladder operators
drive the behavior of eigenvalues, resulting in the quantization of length not
only within a plane but also along a direction normal to that plane. The
derived ladder operators and their commutation relations have enabled us to
construct a comprehensive operator for the square of length. This operator
yields a structured ladder of eigenstates that are simultaneously eigenstates
of both the length-square operator and certain combinations of ladder
operators. Through this approach, we have given the formalism of how
quantization occurs within spaces with position-dependent noncommutativity.
Furthermore, we have identified the ground state and explored the maximum and
minimum possible values of the quantum numbers associated with the ladder
operators, thereby defining the range of valid eigenstates.

The study of degenerate states constitutes a significant facet of our research
inquiry. In our analysis, we have examined the implications of position-
dependent noncommutativity on the degeneracy of states within the framework of
the length quantization problem. By introducing an operator $K$, we have
unveiled the distinct conditions under which degenerate states emerge,
shedding light on the intricate balance between spatial geometry and quantum
behavior.

The quantization of length in noncommutative spaces with position-dependent
noncommutativity opens up new avenues for exploring the fundamental nature of
spacetime and its implications for physical theories. Our findings may inspire
further investigations into the intriguing interplay between geometry and
quantum algebra in noncommutative spacetime settings. We have presented only
the quantization of a bare fundamental geometric element, i.e., length. The
construction of a field theory or mechanics with the underlying quantized
geometric element is beyond the scope of this work.

## References

  * [1] R. Jackiw, Nucl. Phys. B Proc. Suppl. 108 (2002), 30-36 doi:10.1016/S0920-5632(02)01302-6 [arXiv:hep-th/0110057 [hep-th]].
  * [2] H. S. Snyder, Phys. Rev. 71, 38-41 (1947) doi:10.1103/PhysRev.71.38
  * [3] R. Banerjee, B. Chakraborty, S. Ghosh, P. Mukherjee and S. Samanta, Found. Phys. 39, 1297-1345 (2009) doi:10.1007/s10701-009-9349-y [arXiv:0909.1000 [hep-th]].
  * [4] P. Padmanabhan, ”Physics on Noncommutative Spacetimes”(2012) Physics-Dissertations. Paper 122.
  * [5] J. Madore, [arXiv:gr-qc/9906059 [gr-qc]].
  * [6] N. Seiberg and E. Witten, JHEP 09, 032 (1999) doi:10.1088/1126-6708/1999/09/032 [arXiv:hep-th/9908142 [hep-th]].
  * [7] A. Addazi, P. Belli, R. Bernabei and A. Marciano, Chin. Phys. C 42, no.9, 094001 (2018) doi:10.1088/1674-1137/42/9/094001 [arXiv:1712.08082 [hep-th]].
  * [8] J. Frohlich and K. Gawedzki, [arXiv:hep-th/9310187 [hep-th]].
  * [9] J. Frohlich, O. Grandjean and A. Recknagel, [arXiv:hep-th/9706132 [hep-th]].
  * [10] R. J. Szabo, Phys. Rept. 378, 207-299 (2003) doi:10.1016/S0370-1573(03)00059-0 [arXiv:hep-th/0109162 [hep-th]].
  * [11] R. Gopakumar, S. Minwalla and A. Strominger, JHEP 05, 020 (2000) doi:10.1088/1126-6708/2000/05/020 [arXiv:hep-th/0003160 [hep-th]].
  * [12] A. Joseph, Phys. Rev. D 79, 096004 (2009) doi:10.1103/PhysRevD.79.096004 [arXiv:0811.3972 [hep-ph]].
  * [13] G. Mandal, S. J. Rey and S. R. Wadia, Eur. Phys. J. C 24, 495-514 (2002) doi:10.1007/s100520200939 [arXiv:hep-th/0111059 [hep-th]].
  * [14] H. O. Girotti, Am. J. Phys. 72, 608 (2004) doi:10.1119/1.1624116 [arXiv:hep-th/0301237 [hep-th]].
  * [15] M. R. Douglas and N. A. Nekrasov, Rev. Mod. Phys. 73, 977-1029 (2001) doi:10.1103/RevModPhys.73.977 [arXiv:hep-th/0106048 [hep-th]].
  * [16] G. Grensing, World Scientific, 2013, ISBN 978-981-4472-69-2, 978-981-4472-71-5 doi:10.1142/8771
  * [17] A. P. Balachandran, A. Ibort, G. Marmo and M. Martone, SIGMA 6, 052 (2010) doi:10.3842/SIGMA.2010.052 [arXiv:1003.4356 [hep-th]].
  * [18] S. Doplicher, K. Fredenhagen and J. E. Roberts, Commun. Math. Phys. 172 (1995), 187-220 doi:10.1007/BF02104515 [arXiv:hep-th/0303037 [hep-th]].
  * [19] L. Alvarez-Gaume, F. Meyer and M. A. Vazquez-Mozo, Nucl. Phys. B 753 (2006), 92-127 doi:10.1016/j.nuclphysb.2006.07.009 [arXiv:hep-th/0605113 [hep-th]].
  * [20] T. Filk, Phys. Lett. B 376, 53-58 (1996) doi:10.1016/0370-2693(96)00024-X
  * [21] A. Saha, S. Gangopadhyay and S. Saha, Phys. Rev. D 83, 025004 (2011) doi:10.1103/PhysRevD.83.025004 [arXiv:1005.3373 [hep-th]].
  * [22] V. P. Nair and A. P. Polychronakos, Phys. Lett. B 505, 267-274 (2001) doi:10.1016/S0370-2693(01)00339-2 [arXiv:hep-th/0011172 [hep-th]].
  * [23] B. Muthukumar, JHEP 01, 073 (2007) doi:10.1088/1126-6708/2007/01/073 [arXiv:hep-th/0609117 [hep-th]].
  * [24] B. Muthukumar, AIP Conf. Proc. 939 (2007) no.1, 359-362 doi:10.1063/1.2803828
  * [25] D. Sinha, B. Chakraborty and F. G. Scholtz, J. Phys. A 45, 105308 (2012) doi:10.1088/1751-8113/45/10/105308 [arXiv:1108.2569 [hep-th]].
  * [26] S. Biswas, P. Nandi and B. Chakraborty, Phys. Rev. A 102 (2020) no.2, 022231 doi:10.1103/PhysRevA.102.022231 [arXiv:1911.03196 [hep-th]].
  * [27] K. Bolonek and P. Kosinski, Phys. Lett. B 547 (2002), 51-54 doi:10.1016/S0370-2693(02)02731-4 [arXiv:hep-th/0208162 [hep-th]].
  * [28] P. Aschieri, C. Blohmann, M. Dimitrijevic, F. Meyer, P. Schupp and J. Wess, Class. Quant. Grav. 22, 3511-3532 (2005) doi:10.1088/0264-9381/22/17/011 [arXiv:hep-th/0504183 [hep-th]].
  * [29] A. Muhuri, D. Sinha and S. Ghosh, Eur. Phys. J. Plus 136, no.1, 35 (2021) [arXiv:2006.16528 [quant-ph]].
  * [30] E. Harikumar, V. S. Kumar and A. Khare, Phys. Lett. B 589, 155-161 (2004) [arXiv:hep-th/0402064 [hep-th]].
  * [31] X. Calmet and A. Kobakhidze, Phys. Rev. D 72 (2005), 045010 doi:10.1103/PhysRevD.72.045010 [arXiv:hep-th/0506157 [hep-th]].
  * [32] R. Banerjee, P. Mukherjee and S. Samanta, Phys. Rev. D 75, 125020 (2007) doi:10.1103/PhysRevD.75.125020 [arXiv:hep-th/0703128 [hep-th]].
  * [33] A. P. Balachandran, T. R. Govindarajan, K. S. Gupta and S. Kurkcuoglu, Class. Quant. Grav. 23 (2006), 5799-5810 doi:10.1088/0264-9381/23/20/003 [arXiv:hep-th/0602265 [hep-th]].
  * [34] E. Harikumar and V. O. Rivelles, Class. Quant. Grav. 23 (2006), 7551-7560 doi:10.1088/0264-9381/23/24/024 [arXiv:hep-th/0607115 [hep-th]].
  * [35] M. Roy and B. Muthukumar, [arXiv:2205.02479 [hep-th]].
  * [36] M. Chaichian, A. Tureanu and G. Zet, Phys. Lett. B 660, 573-578 (2008) doi:10.1016/j.physletb.2008.01.029 [arXiv:0710.2075 [hep-th]].
  * [37] A. Connes, M. R. Douglas and A. S. Schwarz, JHEP 02, 003 (1998) doi:10.1088/1126-6708/1998/02/003 [arXiv:hep-th/9711162 [hep-th]].
  * [38] A. Connes, [arXiv:math/0011193 [math.QA]].
  * [39] A. H. Chamseddine, A. Connes and V. Mukhanov, Phys. Rev. Lett. 114, no.9, 091302 (2015) doi:10.1103/PhysRevLett.114.091302 [arXiv:1409.2471 [hep-th]].
  * [40] T. Thiemann, J. Math. Phys. 39, 3372-3392 (1998) [arXiv:gr-qc/9606092 [gr-qc]].
  * [41] C. Rovelli and L. Smolin, Nucl. Phys. B 442 (1995), 593-622 [erratum: Nucl. Phys. B 456 (1995), 753-754] doi:10.1016/0550-3213(95)00150-Q [arXiv:gr-qc/9411005 [gr-qc]].
  * [42] A. Ashtekar and J. Lewandowski, Adv. Theor. Math. Phys. 1 (1998), 388-429 doi:10.4310/ATMP.1997.v1.n2.a8 [arXiv:gr-qc/9711031 [gr-qc]].
  * [43] M. Balasundaram and A. Rashid, Adv. High Energy Phys. 2022 (2022), 8009789 doi:10.1155/2022/8009789 [arXiv:2206.07972 [hep-th]].
  * [44] L. M. Lawson, J. Phys. A 53, no.11, 115303 (2020) doi:10.1088/1751-8121/ab7497 [arXiv:2012.06906 [hep-th]].
  * [45] D. N. Blaschke, F. Gieres, S. Hohenegger, M. Schweda and M. Wohlgenannt, SIGMA 14, 133 (2018) doi:10.3842/SIGMA.2018.133 [arXiv:1806.02131 [hep-th]].
  * [46] M. Gomes and V. G. Kupriyanov, Phys. Rev. D 79, 125011 (2009) doi:10.1103/PhysRevD.79.125011 [arXiv:0902.3252 [math-ph]].
  * [47] V. Gayral, J. M. Gracia-Bondia and F. Ruiz Ruiz, Nucl. Phys. B 727, 513-536 (2005) doi:10.1016/j.nuclphysb.2005.08.016 [arXiv:hep-th/0504022 [hep-th]].
  * [48] A. Fring, L. Gouba and F. G. Scholtz, J. Phys. A 43, 345401 (2010) doi:10.1088/1751-8113/43/34/345401 [arXiv:1003.3025 [hep-th]].


#  GreaseLM: Graph REASoning Enhanced
Language Models for Question Answering

Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren

Percy Liang, Christopher D. Manning, Jure Leskovec

Stanford University

<EMAIL_ADDRESS>

###### Abstract

Answering complex questions about textual narratives requires reasoning over
both stated context and the world knowledge that underlies it. However,
pretrained language models (LM), the foundation of most modern QA systems, do
not robustly represent latent relationships between concepts, which is
necessary for reasoning. While knowledge graphs (KG) are often used to augment
LMs with structured representations of world knowledge, it remains an open
question how to effectively fuse and reason over the KG representations and
the language context, which provides situational constraints and nuances. In
this work, we propose GreaseLM, a new model that fuses encoded representations
from pretrained LMs and graph neural networks over multiple layers of modality
interaction operations. Information from both modalities propagates to the
other, allowing language context representations to be grounded by structured
world knowledge, and allowing linguistic nuances (e.g., negation, hedging) in
the context to inform the graph representations of knowledge. Our results on
three benchmarks in the commonsense reasoning (i.e., CommonsenseQA,
OpenbookQA) and medical question answering (i.e., MedQA-USMLE) domains
demonstrate that GreaseLM can more reliably answer questions that require
reasoning over both situational constraints and structured knowledge, even
outperforming models 8$\times$ larger.111All code, data and pretrained models
are available at https://github.com/snap-stanford/GreaseLM.

##  1 Introduction

Question answering is a challenging task that requires complex reasoning over
both explicit constraints described in the textual context of the question, as
well as unstated, relevant knowledge about the world (i.e., knowledge about
the domain of interest). Recently, large pretrained language models fine-tuned
on QA datasets have become the dominant paradigm in NLP for question answering
tasks (Khashabi et al., 2020). After pretraining on an extreme-scale
collection of general text corpora, these language models learn to implicitly
encode broad knowledge about the world, which they are able to leverage when
fine-tuned on a domain-specific downstream QA task. However, despite the
strong performance of this two-stage learning procedure on common benchmarks,
these models struggle when given examples that are distributionally different
from examples seen during fine-tuning (McCoy et al., 2019). Their learned
behavior often relies on simple (at times spurious) patterns to offer
shortcuts to an answer, rather than robust, structured reasoning that
effectively fuses the explicit information provided by the context and
implicit external knowledge (Marcus, 2018).

On the other hand, massive knowledge graphs (KG), such as Freebase (Bollacker
et al., 2008), Wikidata (Vrandečić & Krötzsch, 2014), ConceptNet (Speer et
al., 2017), and Yago (Suchanek et al., 2007) capture such external knowledge
explicitly using triplets that capture relationships between entities.
Previous research has demonstrated the significant role KGs can play in
structured reasoning and query answering (Ren et al., 2020; 2021; Ren &
Leskovec, 2020). However, extending these reasoning advantages to general QA
(where questions and answers are expressed in natural language and not easily
mapped to strict logical queries) requires finding the right integration of
knowledge from the KG with the information and constraints provided by the QA
example. Prior methods propose various ways to leverage both modalities (i.e.,
expressive large language models and structured KGs) for improved reasoning
(Mihaylov & Frank, 2018; Lin et al., 2019; Feng et al., 2020). However, these
methods typically fuse the two modalities in a shallow and non-interactive
manner, encoding both separately and fusing them at the output for a
prediction, or using one to augment the input of the other. Consequently,
previous methods demonstrate restricted capacity to exchange useful
information between the two modalities. It remains an open question how to
effectively fuse the KG and LM representations in a truly unified manner,
where the two representations can interact in a non-shallow way to simulate
structured, situational reasoning.

Figure 1: GreaseLM Architecture. The textual context is appended with a
special interaction token and passed through $N$ LM-based unimodal encoding
layers. Simultaneously, a local KG of relevant knowledge is extracted and
connected to an interaction node. In the later GreaseLM layers, the language
representation continues to be updated through LM layers and the KG is
processed using a GNN, simulating reasoning over its knowledge. In each layer,
after each modality’s representation is updated, the representations of the
interaction token and node are pulled, concatenated, and passed through a
modality interaction (MInt) unit to mix their representations. In subsequent
layers, the mixed information from the interaction elements mixes with their
respective modalities, allowing knowledge from the KG to affect the
representations of individual tokens, and context from language to affect
fine-grained entity knowledge representations in the GNN.

In this work, we present GreaseLM, a new model that enables fusion and
exchange of information from both the LM and KG in multiple layers of its
architecture (see Figure 1). Our proposed GreaseLM consists of an LM that
takes as input the natural language context, as well as a graph neural network
(GNN) that reasons over the KG. After each layer of the LM and GNN, we design
an interactive scheme to bidirectionally transfer the information from each
modality to the other through specially initialized interaction
representations (i.e., interaction token for the LM; interaction node for the
GNN). In such a way, all the tokens in the language context receive
information from the KG entities through the interaction token and the KG
entities indirectly interact with the tokens through the interaction node. By
such a deep integration across all layers, GreaseLM enables joint reasoning
over both the language context and the KG entities under a unified framework
agnostic to the specific language model or graph neural network, so that both
modalities can be contextualized by the other.

GreaseLM demonstrates significant performance gains across different LM
architectures. We perform experiments on several standard QA benchmarks:
CommonsenseQA, OpenbookQA and MedQA-USMLE, which require external knowledge
across different domains (commonsense reasoning and medical reasoning) and use
different KGs (ConceptNet and Disease Database). Across both domains, GreaseLM
outperforms comparably-sized prior QA models, including strong fine-tuned LM
baselines (by 5.5%, 6.6%, and 1.3%, respectively) and state-of-the-art KG+LM
models (by 0.9%, 1.8%, and 0.5%, respectively) on the three competitive
benchmarks. Furthermore, with the deep fusion of both modalities, GreaseLM
exhibits strong performance over baselines on questions that exhibit textual
nuance, such as resolving multiple constraints, negation, and hedges, and
which require effective reasoning over both language context and KG.

##  2 Related Work

Integrating KG information has become a popular research area for improving
neural QA systems. Some works explore using two-tower models to answer
questions, where a graph representation of knowledge and language
representation are fused with no interaction between them (Wang et al., 2019).
Other works seek to use one modality to ground the other, such as using an
encoded representation of a linked KG to augment the textual representation of
a QA example (e.g., Knowledgeable Reader, Mihaylov & Frank, 2018; KagNet, Lin
et al., 2019; KT-NET, Yang et al., 2019). Others reverse the flow of
information and use a representation of the text (e.g., final layer of LM) to
provide an augmentation to a graph reasoning model over an extracted KG for
the example (e.g., MHGRN, Feng et al., 2020; Lv et al., 2020). In all of these
settings, however, the interaction between both modalities is limited as
information between them only flows one way.

More recent approaches explore deeper integrations of both modalities. Certain
approaches learn to access implicit knowledge encoded in LMs (Bosselut et al.,
2019; Petroni et al., 2019; Hwang et al., 2021) by training on structured KG
data, and then use the LM to generate local KGs that can be used for QA (Wang
et al., 2020; Bosselut et al., 2021). However, these approaches discard the
static KG once they train the LM on its facts, losing important structure that
can guide reasoning. More recently, QA-GNN (Yasunaga et al., 2021) proposed to
jointly update the LM and GNN representations via message passing. However,
they use a single pooled representation of the LM to seed the textual
component of this joint structure, limiting the updates that can be made to
the textual representation. In contrast to prior works, we propose to make
individual token representations in the LM and node representations in the GNN
mix for multiple layers, enabling representations of both modalities to
reflect particularities of the other (e.g., knowledge grounds language;
language nuances specifies which knowledge is important). Simultaneously, we
retain the individual structure of both modalities, which we demonstrate
improves QA performance substantially (§5).

Additionally, some works explore integrating knowledge graphs with language
models in the pretraining stage. However, much like for QA, the modality
interaction is typically limited to knowledge feeding language (Zhang et al.,
2019; Shen et al., 2020; Yu et al., 2020), rather than designing interactions
across multiple layers. Sun et al. (2020)’s work is perhaps most similar, but
they do not use the same interaction bottleneck, requiring high-precision
entity mention spans for linking, and they limit expressivity through shared
modality parameters for the LM and KG.

##  3 Proposed Approach: GreaseLM

In this work, we augment large-scale language models (Devlin et al., 2019; Liu
et al., 2019; Lan et al., 2020; Liu et al., 2021) with graph reasoning modules
over KGs. Our method, GreaseLM (depicted in Figure 1), consists of two stacked
components: (1) a set of unimodal LM layers which learn an initial
representation of the input tokens, and (2) a set of upper cross-modal
GreaseLM layers which learn to jointly represent the language sequence and
linked knowledge graph, allowing textual representations formed from the
underlying LM layers and a graph representation of the KG to mix with one
another. We denote the number of LM layers as $N$, and the number of GreaseLM
layers as $M$. The total number of layers in our model is $N+M$.

Notation. In the task of multiple choice question answering (MCQA), a generic
MCQA-type dataset consists of examples with a context paragraph $c$, a
question $q$ and a candidate answer set $\mathcal{A}$, all expressed in text.
In this work, we also assume access to an external knowledge graph (KG)
${\mathcal{G}}$ that provides background knowledge that is relevant to the
content of the multiple choice questions.

Given a QA example $(c,q,\mathcal{A})$, and the KG ${\mathcal{G}}$ as input,
our goal is to identify which answer $a\in\mathcal{A}$ is correct. Without
loss of generality, when an operation is applied to an arbitrary answer, we
refer to that answer as $a$. We denote a sequence of tokens in natural
language as $\\{w_{1},\dots,w_{T}\\}$, where $T$ is the total number of
tokens, and the representation of a token $w_{t}$ from the $\ell$-th layer of
the model as ${\bm{h}}_{t}^{(\ell)}$. We denote a set of nodes from the KG as
$\\{e_{1},\dots,e_{J}\\}$, where $J$ is the total number of nodes, and the
representation of a node $e_{j}$ in the $\ell$-th layer of the model as
${\bm{e}}_{j}^{(\ell)}$.

###  3.1 Input representation

We concatenate our context paragraph $c$, question $q$, and candidate answer
$a$ with separator tokens to get our model input $[c;q;a]$ and tokenize the
combined sequence into $\\{w_{1},\dots,w_{T}\\}$. Second, we use the input
sequence to retrieve a subgraph of the KG ${\mathcal{G}}$ (denoted
${\mathcal{G}}_{\text{sub}}$), which provides knowledge from the KG that is
relevant to this QA example. We denote the set of nodes in
${\mathcal{G}}_{\text{sub}}$ as $\\{e_{1},\dots,e_{J}\\}$.

KG Retrieval. Given each QA context, we follow the procedure from Yasunaga et
al. (2021) to retrieve the subgraph ${\mathcal{G}}_{\text{sub}}$ from
${\mathcal{G}}$. We describe this procedure in Appendix B.1. Each node in
${\mathcal{G}}_{\text{sub}}$ is assigned a type based on whether its
corresponding entity was linked from the context $c$, question $q$, answer
$a$, or as a neighbor to these nodes. In the rest of the paper, we use “KG” to
refer to ${\mathcal{G}}_{\text{sub}}$.

Interaction Bottlenecks. In the cross-modal GreaseLM layers, information is
fused between both modalities, for which we define a special interaction token
$w_{int}$ and a special interaction node $e_{int}$ whose representations serve
as the bottlenecks through which the two modalities interact (§3.3). We
prepend $w_{int}$ to the token sequence and connect $e_{int}$ to all the
linked nodes ${\mathcal{V}}_{\text{linked}}$ in ${\mathcal{G}}_{\text{sub}}$.

###  3.2 Language Pre-encoding

In the unimodal encoding component, given the sequence of tokens
$\\{w_{int},w_{1},\dots,w_{T}\\}$, we first sum the token, segment, and
positional embeddings for each token to compute its $\ell$=0 input
representation
$\\{{\bm{h}}_{int}^{(0)},{\bm{h}}_{1}^{(0)},\dots,{\bm{h}}_{T}^{(0)}\\}$, and
then compute an output representation for each layer $\ell$:

$\displaystyle\\{{\bm{h}}_{int}^{(\ell)},{\bm{h}}_{1}^{(\ell)},\dots,{\bm{h}}_{T}^{(\ell)}\\}=$
$\displaystyle\text{ LM-
Layer}(\\{{\bm{h}}_{int}^{(\ell-1)},{\bm{h}}_{1}^{(\ell-1)},\dots,{\bm{h}}_{T}^{(\ell-1)}\\})$
(1) $\displaystyle\text{ for }\ell=1,\dots,N$

where LM-Layer($\cdot$) is a single LM encoder layer, whose parameters are
initialized using a pretrained model (§4.1). We refer readers to Vaswani et
al. (2017) for technical details of these layers.

###  3.3 GreaseLM

GreaseLM uses a cross-modal fusion component to inject information from the KG
into language representations and information from language into KG
representations. The GreaseLM layer is designed to separately encode
information from both modalities, and fuse their representations using the
bottleneck of the special interaction token and node. It is comprised of three
components: (1) a transformer LM encoder block which continues to encode the
language context, (2) a GNN layer that reasons over KG entities and relations,
and (3) a modality interaction layer that takes the unimodal representations
of the interaction token and interaction node and exchanges information
through them. We discuss these three components below.

Language Representation. In the $\ell$-th GreaseLM layer, the input token
embeddings
$\\{{\bm{h}}_{int}^{(N+\ell-1)},{\bm{h}}_{1}^{(N+\ell-1)},\dots,{\bm{h}}_{T}^{(N+\ell-1)}\\}$
are fed into additional transformer LM encoder blocks that continue to encode
the textual context based on the LM’s pretrained representations:

$\displaystyle\\{\tilde{{\bm{h}}}_{int}^{(N+\ell)},\tilde{{\bm{h}}}_{1}^{(N+\ell)},\dots,\tilde{{\bm{h}}}_{T}^{(N+\ell)}\\}=$
$\displaystyle\text{ LM-
Layer}(\\{{\bm{h}}_{int}^{(N+\ell-1)},{\bm{h}}_{1}^{(N+\ell-1)},\dots,{\bm{h}}_{T}^{(N+\ell-1)}\\})$
(2) $\displaystyle\text{ for }\ell=1,\dots,M$

where $\tilde{{\bm{h}}}$ corresponds to pre-fused embeddings of the language
modality. As we will discuss below, because $\bm{h}_{int}^{N+\ell-1}$ will
encode information received from the knowledge graph representation, these
late language encoding layers will also allow the token representations to mix
with KG knowledge.

Graph Representation. The GreaseLM layers also encode a representation of the
local KG ${\mathcal{G}}_{\text{sub}}$ linked from the QA example. To represent
the graph, we first compute initial node embeddings
$\\{{\bm{e}}_{1}^{(0)},\dots,{\bm{e}}_{J}^{(0)}\\}$ for the retrieved entities
using pretrained KG embeddings for these nodes (§4.1). The initial embedding
of the interaction node ${\bm{e}}_{int}^{0}$ is initialized randomly.

Then, in each layer of the GNN, the current representation of the node
embeddings
$\\{{\bm{e}}_{int}^{(\ell-1)},{\bm{e}}_{1}^{(\ell-1)},\dots,{\bm{e}}_{J}^{(\ell-1)}\\}$
is fed into the layer to perform a round of information propagation between
nodes in the graph and yield pre-fused node embeddings for each entity:

$\displaystyle\\{\tilde{{\bm{e}}}_{int}^{(\ell)},\tilde{{\bm{e}}}_{1}^{(\ell)},\dots,\tilde{{\bm{e}}}_{J}^{(\ell)}\\}=$
$\displaystyle\text{
GNN}(\\{{\bm{e}}_{int}^{(\ell-1)},{\bm{e}}_{1}^{(\ell-1)},\dots,{\bm{e}}_{J}^{(\ell-1)}\\})$
(3) $\displaystyle\text{ for }\ell=1,\dots,M$

where GNN corresponds to a variant of graph attention networks (Veličković et
al., 2018) that is a simplification of the method of Yasunaga et al. (2021).
The GNN computes node representations $\tilde{{\bm{e}}}_{j}^{(\ell)}$ for each
node $e_{j}\in\\{e_{1},\dots,e_{J}\\}$ via message passing between neighbors
on the graph.

$\displaystyle\tilde{{\bm{e}}}_{j}^{(\ell)}=f_{n}\Biggl{(}\sum_{e_{s}\in\mathcal{N}_{e_{j}}\cup\\{e_{j}\\}}\alpha_{sj}{\bm{m}}_{sj}\Biggr{)}+{\bm{e}}_{j}^{(\ell-1)}$
(4)

where ${\mathcal{N}}_{e_{j}}$ represents the neighborhood of an arbitrary node
$e_{j}$, ${\bm{m}}_{sj}$ denotes the message one of its neighbors $e_{s}$
passes to $e_{j}$, $\alpha_{sj}$ is an attention weight that scales the
message ${\bm{m}}_{sj}$, and $f_{n}$ is a 2-layer MLP. The messages
${\bm{m}}_{sj}$ between nodes allow entity information from a node to affect
the model’s representation of its neighbors, and are computed in the following
manner:

$\bm{r}_{sj}={f}_{r}(\tilde{\bm{r}}_{sj},~{}\bm{u}_{s},~{}\bm{u}_{j})$ (5)

$\bm{m}_{sj}={f}_{m}(\bm{e}_{s}^{(\ell-1)},~{}\bm{u}_{s},~{}\bm{r}_{sj})$ (6)

where $\bm{u}_{s},\bm{u}_{j}$ are node type embeddings, $\tilde{\bm{r}}_{sj}$
is a relation embedding for the relation connecting $e_{s}$ and $e_{j}$,
${f}_{r}$ is a 2-layer MLP, and ${f}_{m}$ is a linear transformation. The
attention weights $\alpha_{sj}$ scale the contribution of each neighbor’s
message by its importance, and are computed as follows:
$\bm{q}_{s}={f}_{q}(\bm{e}_{s}^{(\ell-1)},~{}\bm{u}_{s})$ (7)
$\bm{k}_{j}={f}_{k}(\bm{e}_{j}^{(\ell-1)},~{}\bm{u}_{j},~{}\bm{r}_{sj})$ (8)
$\gamma_{sj}=\frac{\bm{q}_{s}^{\top}\bm{k}_{j}}{\sqrt{D}}$ (9)
$\alpha_{sj}=\frac{\exp(\gamma_{sj})}{\sum_{e_{s}\in\mathcal{N}_{e_{j}}\cup\\{e_{j}\\}}\exp(\gamma_{sj})}$
(10)

where ${f}_{q}$ and ${f}_{k}$ are linear transformations and
$\bm{u}_{s},\bm{u}_{j},\bm{r}_{sj}$ are defined the same as above.

As discussed in the following paragraph, message passing between the
interaction node $e_{int}$ and the nodes from the retrieved subgraph will
allow information from text that $e_{int}$ receives from $w_{int}$ to
propagate to the other nodes in the graph.

Modality Interaction. Finally, after using a transformer LM layer and a GNN
layer to update token embeddings and node embeddings respectively, we use a
modality interaction layer (MInt) to let the two modalities fuse information
through the bottleneck of the interaction token $w_{int}$ and the interaction
node $e_{int}$. We concatenate the pre-fused embeddings of the interaction
token $\tilde{{\bm{h}}}_{int}^{(i)}$ and interaction node
$\tilde{{\bm{e}}}_{int}^{(i)}$, pass the joint representation through a mixing
operation (MInt), and then split the output post-fused embeddings into
${\bm{h}}_{int}^{(i)}$ and ${\bm{e}}_{int}^{(i)}$:

$\displaystyle[{\bm{h}}_{int}^{(\ell)};{\bm{e}}_{int}^{(\ell)}]=\text{
MInt}([\tilde{{\bm{h}}}_{int}^{(\ell)};\tilde{{\bm{e}}}_{int}^{(\ell)}]),$
(11)

We use a two-layer MLP as our MInt operation, though other fusion operators
could be used to mix the representation. All the tokens other than the
interaction token $w_{int}$ and all the nodes other than the interaction node
$e_{int}$ are not involved in the modality interaction process:
${\bm{w}}^{(\ell)}=\tilde{{\bm{w}}}^{(\ell)}\text{ for
}w\in\\{w_{1},\dots,w_{T}\\}$ and
${\bm{e}}^{(\ell)}=\tilde{{\bm{e}}}^{(\ell)}\text{ for
}e\in\\{e_{1},\dots,e_{J}\\}$. However, they receive information from the
interaction representations ${\bm{h}}_{int}^{(\ell)}$ and
${\bm{e}}_{int}^{(\ell)}$ in the next layers of their respective modal
propagation (i.e., Eqs. 2, 3). Consequently, across multiple GreaseLM layers,
information propagates between both modalities (see Fig. 1 for visual
depiction), grounding language representations to KG knowledge, and knowledge
representations to contextual constraints.

#### Learning & Inference.

For the MCQA task, given a question $q$ and an answer $a$ from all the
candidates $\mathcal{A}$, we compute the probability of $a$ being the correct
answer as $p(a\mid
q,c)\propto\exp(\text{MLP}({\bm{h}}_{int}^{(N+M)},~{}{\bm{e}}_{int}^{(M)},~{}{\bm{g}}))$,
where ${\bm{g}}$ denotes attention-based pooling of $\\{{\bm{e}}_{j}^{(M)}\mid
e_{j}\in\\{e_{1},\dots,e_{J}\\}\\}$ using ${\bm{h}}_{int}^{(N+M)}$ as a query.
We optimize the whole model end-to-end using the cross entropy loss. At
inference time, we predict the most plausible answer as
$\operatorname*{arg\,max}_{a\in\mathcal{A}}~{}p(a\mid q,c)$.

Dataset | Example
---|---
CommonsenseQA | A weasel has a thin body and short legs to easier burrow after prey in a what?
(A) tree (B) mulberry bush (C) chicken coop (D) viking ship  (E) rabbit warren
OpenbookQA | Which of these would let the most heat travel through?
(A) a new pair of jeans  (B) a steel spoon in a cafeteria
(C) a cotton candy at a store (D) a calvin klein cotton hat
MedQA-USMLE | A 57-year-old man presents to his primary care physician with a 2-month
history of right upper and lower extremity weakness. He noticed the weakness
when he started falling far more frequently while running errands. Since then,
he has had increasing difficulty with walking and lifting objects. His past
medical history is significant only for well-controlled hypertension, but he
says
that some members of his family have had musculoskeletal problems. His right
upper extremity shows forearm atrophy and depressed reflexes while his right
lower extremity is hypertonic with a positive Babinski sign. Which of the
following is most likely associated with the cause of this patients symptoms?
(A) HLA-B8 haplotype (B) HLA-DR2 haplotype
(C) Mutation in SOD1 (D) Mutation in SMN1

Table 1: Examples of the MCQA task for each of the datasets evaluated in this
work.

##  4 Experimental Setup

We evaluate GreaseLM on three diverse multiple-choice question answering
datasets across two domains: CommonsenseQA (Talmor et al., 2019) and
OpenBookQA (Mihaylov et al., 2018) as commonsense reasoning benchmarks, and
MedQA-USMLE (Jin et al., 2021) as a clinical QA task.

CommonsenseQA is a 5-way multiple-choice question answering dataset of 12,102
questions that require background commonsense knowledge beyond surface
language understanding. We perform our experiments using the in-house data
split of Lin et al. (2019) to compare to baseline methods.

OpenbookQA is a 4-way multiple-choice question answering dataset that tests
elementary scientific knowledge. It contains 5,957 questions along with an
open book of scientific facts. We use the official data splits from Mihaylov &
Frank (2018).

MedQA-USMLE is a 4-way multiple-choice question answering dataset, which
requires biomedical and clinical knowledge. The questions are originally from
practice tests for the United States Medical License Exams (USMLE). The
dataset contains 12,723 questions. We use the original data splits from Jin et
al. (2021).

###  4.1 Implementation & training details

#### Language Models.

We seed GreaseLM with RoBERTa-Large (Liu et al., 2019) for our experiments on
CommonsenseQA, AristoRoBERTa (Clark et al., 2019) for our experiments on
OpenbookQA, and SapBERT (Liu et al., 2021) for our experiments on MedQA-USMLE,
demonstrating GreaseLM’s generality with respect to language model
initializations. Hyperparameters for training these models can be found in
Appendix Table 7.

Knowledge Graphs. We use ConceptNet (Speer et al., 2017), a general-domain
knowledge graph, as our external knowledge source $\mathcal{G}$ for both
CommonsenseQA and OpenbookQA. It has 799,273 nodes and 2,487,810 edges in
total. For MedQA-USMLE, we use a self-constructed knowledge graph that
integrates the Disease Database portion of the Unified Medical Language System
(UMLS; Bodenreider, 2004) and DrugBank (Wishart et al., 2018). The knowledge
graph contains 9,958 nodes and 44,561 edges. Additional information about node
initialization and hyperparameters for preprocessing these KGs can be found in
Appendix B.2.

###  4.2 Baseline methods

#### Fine-tuned LMs.

To study the effect of using KGs as external knowledge sources, we compare our
method with vanilla fine-tuned LMs, which are knowledge-agnostic. We fine-tune
RoBERTa-Large (Liu et al., 2019) for CommonsenseQA, and
AristoRoBERTa222OpenbookQA provides an extra corpus of scientific facts in a
textual form. AristoRoBERTa is based off RoBERTa-Large, but uses the facts
corresponding to each question, prepared by Clark et al. (2019), as an
additional input along with the QA context. (Clark et al., 2019) for
OpenbookQA. For MedQA-USMLE, we use a state-of-the-art biomedical language
model, SapBERT (Liu et al., 2021), which is an augmentation of PubmedBERT (Gu
et al., 2022) that is trained with entity disambiguation objectives to allow
the model to better understand entity knowledge.

LM+KG models. We also evaluate GreaseLM’s ability to exploit its knowledge
graph augmentation by comparing with existing LM+KG methods: (1) Relation
Network (RN; Santoro et al., 2017), (2) RGCN (Schlichtkrull et al., 2018), (3)
GconAttn (Wang et al., 2019), (4) KagNet (Lin et al., 2019), (5) MHGRN (Feng
et al., 2020), and (6) QA-GNN (Yasunaga et al., 2021). QA-GNN is the existing
top-performing model under this LM+KG paradigm. The key difference between
GreaseLM and these baseline methods is that they do not fuse the
representations of both modalities across multiple interaction layers,
allowing the representation of both modalities to affect the other (§3.3). For
fair comparison, we use the same LM to initialize these baselines as for our
model.

##  5 Experimental Results

Table 2: Performance comparison on Commonsense QA in-house split (controlled experiments). As the official test is hidden, here we report the in-house Dev (IHdev) and Test (IHtest) accuracy, following the data split of Lin et al. (2019). Experiments are controlled using same seed LM.  Methods |  IHdev-Acc. (%) |  IHtest-Acc. (%)
---|---|---
RoBERTa-Large (w/o KG) | 73.1 ($\pm$0.5) | 68.7 ($\pm$0.6)
RGCN (Schlichtkrull et al., 2018) | 72.7 ($\pm$0.2) | 68.4 ($\pm$0.7)
GconAttn (Wang et al., 2019) | 72.6 ($\pm$0.4) | 68.6 ($\pm$1.0)
KagNet (Lin et al., 2019) | 73.5 ($\pm$0.2) | 69.0 ($\pm$0.8)
RN (Santoro et al., 2017) | 74.6 ($\pm$0.9) | 69.1 ($\pm$0.2)
MHGRN (Feng et al., 2020) | 74.5 ($\pm$0.1) | 71.1 ($\pm$0.8)
QA-GNN (Yasunaga et al., 2021) | 76.5 ($\pm$0.2) | 73.4 ($\pm$0.9)
GreaseLM (Ours) |  78.5 ($\pm$0.5) |  74.2 ($\pm$0.4)

Table 3: Test Accuracy comparison on OpenBook QA. Experiments are controlled using the same seed LM for all LM+KG methods.  Model | Acc.
---|---
AristoRoBERTa (no KG) | 78.4
\+ RGCN | 74.6
\+ GconAttn | 71.8
\+ RN | 75.4
\+ MHGRN | 80.6
\+ QA-GNN | 82.8
GreaseLM (Ours) | 84.8

Table 4: Test accuracy comparison to public OpenBookQA model implementations. ∗UnifiedQA (11B params) and T5 (3B) are 30x and 8x larger than our model. Model | Acc. | # Params
---|---|---
ALBERT (Lan et al., 2020) \+ KB | 81.0 |  $\sim$235M
HGN (Yan et al., 2020) | 81.4 |  $\geq$355M
AMR-SG (Xu et al., 2021) | 81.6 |  $\sim$361M
ALBERT + KPG (Wang et al., 2020) | 81.8 |  $\geq$235M
QA-GNN (Yasunaga et al., 2021) | 82.8 |  $\sim$360M
T5* (Raffel et al., 2020) | 83.2 | $\sim$3B
T5 + KB (Pirtoaca, ) | 85.4 |  $\geq$11B
UnifiedQA* (Khashabi et al., 2020) | 87.2 |  $\sim$11B
GreaseLM (Ours) | 84.8 |  $\sim$359M

Our results in Tables 2 and 4 demonstrate a consistent improvement on the
CommonsenseQA and OpenbookQA datasets. On CommonsenseQA, our model’s test
performance improves by 5.5% over fine-tuned LMs and 0.9% over existing LM+KG
models. On OpenbookQA, these improvements are magnified, with 6.4% over raw
LMs, and 2.0% over the prior best LM+KG system, QA-GNN. The boost over QA-GNN
suggests that GreaseLM’s multi-layer fusion component that passes information
between the text and KG representations is more expressive than LM+KG methods
which do not integrate such sustained interaction between both modalities. We
also achieve competitive results to other systems on the leaderboard of
OpenbookQA (Table 4), posting the third highest score. However, we note that
the T5 (Raffel et al., 2020) and UnifiedQA (Khashabi et al., 2020) models are
pretrained models with 8$\times$ and $30\times$ more parameters, respectively,
than our model. Among models with comparable parameter counts, GreaseLM
achieves the highest score. An ablation study on different model components
and hyperparameters is reported in Appendix C.1.

Quantitative Analysis. Given these overall performance improvements, we
investigated whether GreaseLM’s improvements were reflected in questions that
required more complex reasoning. Because we had no gold structures from these
datasets to categorize the reasoning complexity of different questions, we
defined three proxies: the number of prepositional phrases in the questions,
the presence of negation terms, and the presence of hedging terms. We use the
number of prepositional phrases as a proxy for the number of explicit
reasoning constraints being set in the questions. For example, the
CommonsenseQA question in Table 1, “A weasel has a thin body and short legs to
easier burrow after prey in a what?” has three prepositional phrases: to
easier burrow, after prey, in a what, which each provide an additional search
constraint for the answer (n.b., in certain cases, the prepositional phrases
do not provide constraints that are needed for selecting the correct answer).
The presence of negation and hedging terms stratifies our evaluation to
questions that have explicit negation mentions (e.g., no, never) and terms
indicating uncertainty (e.g., sometimes; maybe).

Our results in Table 5 demonstrate that GreaseLM generally outperforms
RoBERTa-Large and QA-GNN for both questions with negation terms and hedge
terms, indicating GreaseLM handles contexts with nuanced constraints.
Furthermore, we also note that GreaseLM performs better than the baselines
across all questions with prepositional phrases, our measure for reasoning
complexity. QA-GNN and GreaseLM perform comparably on questions with no
prepositional phrases, but the increasing complexity of questions requires
deeper cross-modal fusion between language and knowledge representations.
While QA-GNN’s end fusion approach of initializing a node in the GNN from the
LM’s final representation of the context is an effective approach, it
compresses the language context to a single vector before allowing interaction
with the KG, potentially limiting the cross-relationships between language and
knowledge that can be captured (see example in Figure 2). Interestingly, we
note that both GreaseLM and QA-GNN significantly outperform RoBERTa-Large even
when no prepositional phrases are in the question. We hypothesize that some of
these questions may require less reasoning, but require specific commonsense
knowledge that RoBERTa may not have learned during pretraining (e.g., “What is
a person considered a bully known for?”).

Table 5: Performance of GreaseLM on the CommonsenseQA IH-dev set on complex questions with semantic nuance such as prepositional phrases, negation terms, and hedge terms.  Model | # Prepositional Phrases | Negation | Hedge
---|---|---|---
0 | 1 | 2 | 3 | 4 | Term | Term
n | 210 | 429 | 316 | 171 | 59 | 83 | 167
RoBERTa-Large | 66.7 | 72.3 | 76.3 | 74.3 | 69.5 | 63.8 | 70.7
QA-GNN | 76.7 | 76.2 | 79.1 | 74.9 | 81.4 | 66.2 | 76.0
GreaseLM (Ours) | 75.7 | 79.3 | 80.4 | 77.2 | 84.7 | 69.9 | 78.4

Qualitative Analysis. In Figure 2, we examine GreaseLM’s node-to-node
attention weights induced by the GNN layers of the model, and analyze whether
they reflect more expressive reasoning steps compared to QA-GNN. Figure 2
shows an example from the CommonsenseQA IH-dev set. In this example, GreaseLM
correctly predicts that the answer is “airplane” while QA-GNN makes an
incorrect prediction, “motor vehicle”. For both models, we perform Best First
Search (BFS) on the retrieved KG subgraph ${\mathcal{G}}_{\text{sub}}$ to
trace high attention weights from the interaction node (purple).

For GreaseLM, we observe that the attention by the interaction node increases
on the “bug” entity in the intermediate GNN layers, but drops again by the
final layer, resembling a suitable intuition surrounding the hedge term
“unlikely”. Meanwhile, the attention on “windshield” consistently increases
across all layers. For QA-GNN, the attention on “bug” increases over multiple
layers. As “bug” is mentioned multiple times in the context, it may be well-
represented in QA-GNN’s context node initialization, which is never
reformulated by language representations, unlike in GreaseLM.

Figure 2: Qualitative analysis of GreaseLM’s graph attention weight changes
across multiple layers of message passing compared with QA-GNN. GreaseLM
demonstrates attention change patterns that more closely resemble the expected
change in focus on the “bug” entity.

#### Domain generality

Table 6: Performance on MedQA-USMLE Methods |  Acc. (%)
---|---
Baselines (Jin et al., 2021) |
Chance | 25.0
PMI | 31.1
IR-ES | 35.5
IR-Custom | 36.1
clinicalBERT-Base | 32.4
BioRoBERTa-Base | 36.1
BioBERT-Base | 34.1
BioBERT-Large | 36.7
Baselines (Our implementation) |
SapBERT-Base (w/o KG) | 37.2
QA-GNN | 38.0
GreaseLM (Ours) | 38.5

Our reported results thus far demonstrate the viability of our method in the
general commonsense reasoning domain. In this section, we explore whether
GreaseLM could be adapted to other domains by evaluating on the MedQA-USMLE
dataset. Our results in Table 6 demonstrate that GreaseLM outperforms state-
of-the-art fine-tuned LMs (e.g., SapBERT; Liu et al., 2021) and a QA-GNN
augmentation of SapBERT. Additionally, we note the improved performance over
all classical methods and LM methods first reported in Jin et al. (2021).
Additional results in Appendix C show that our approach is also agnostic to
the language model used with improvements recorded by GreaseLM when it is
seeded with other LMs, such as PubmedBERT (Gu et al., 2022), and BioBERT (Lee
et al., 2020). While these results are promising as they suggest that GreaseLM
is an effective augmentation of pretrained LMs for different domains and KGs
(i.e., the medical domain with the DDB + Drugbank KG), there is still ample
room for improvement on this task.

##  6 Conclusion

In this paper, we introduce GreaseLM, a new model that enables interactive
fusion through joint information exchange between knowledge from language
models and knowledge graphs. Experimental results demonstrate superior
performance compared to prior KG+LM and LM-only baselines across standard
datasets from multiple domains (commonsense and medical). Our analysis shows
improved capability modeling questions exhibiting textual nuances, such as
negation and hedging.

## Acknowledgment

We thank Rok Sosic, Maria Brbic, Jordan Troutman, Rajas Bansal, and our
anonymous reviewers for discussions and for providing feedback on our
manuscript. We thank Xiaomeng Jin for help with data preprocessing. We also
gratefully acknowledge the support of DARPA under Nos. HR00112190039 (TAMI),
N660011924033 (MCS); ARO under Nos. W911NF-16-1-0342 (MURI), W911NF-16-1-0171
(DURIP); NSF under Nos. OAC-1835598 (CINES), OAC-1934578 (HDR), CCF-1918940
(Expeditions), IIS-2030477 (RAPID), NIH under No. R56LM013365; Stanford Data
Science Initiative, Wu Tsai Neurosciences Institute, Chan Zuckerberg Biohub,
Amazon, JPMorgan Chase, Docomo, Hitachi, Intel, JD.com, KDDI, Toshiba, NEC,
and UnitedHealth Group. J. L. is a Chan Zuckerberg Biohub investigator. The
content is solely the responsibility of the authors and does not necessarily
represent the official views of the funding entities.

## References

  * Bodenreider (2004) Olivier Bodenreider.  The unified medical language system (UMLS): Integrating biomedical terminology.  _Nucleic acids research_ , 2004.
  * Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor.  Freebase: a collaboratively created graph database for structuring human knowledge.  In _SIGMOD_ , 2008.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko.  Translating embeddings for modeling multi-relational data.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2013.
  * Bosselut et al. (2019) Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Çelikyilmaz, and Yejin Choi.  Comet: Commonsense transformers for automatic knowledge graph construction.  In _Association for Computational Linguistics (ACL)_ , 2019.
  * Bosselut et al. (2021) Antoine Bosselut, Ronan Le Bras, and Yejin Choi.  Dynamic neuro-symbolic knowledge graph construction for zero-shot commonsense question answering.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2021.
  * Clark et al. (2019) Peter Clark, Oren Etzioni, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, et al.  From ‘f’ to ‘a’on the NY Regents science exams: An overview of the Aristo project.  _arXiv preprint arXiv:1909.01958_ , 2019.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL_ , 2019.
  * Feng et al. (2020) Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren.  Scalable multi-hop relational reasoning for knowledge-aware question answering.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2020\.
  * Gu et al. (2022) Yuxian Gu, Robert Tinn, Hao Cheng, Michael R. Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare (HEALTH)_ , 3:1 – 23, 2022.
  * Hwang et al. (2021) Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi.  Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs.  In _AAAI_ , 2021.
  * Jin et al. (2021) Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits.  What disease does this patient have? a large-scale open domain question answering dataset from medical exams.  _Applied Sciences_ , 2021.
  * Khashabi et al. (2020) Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi.  Unifiedqa: Crossing format boundaries with a single qa system.  In _Findings of EMNLP_ , 2020.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36:1234 – 1240, 2020.
  * Lin et al. (2019) Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren.  Kagnet: Knowledge-aware graph networks for commonsense reasoning.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2019\.
  * Liu et al. (2021) Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier.  Self-alignment pretraining for biomedical entity representations.  In _NAACL_ , 2021.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Lv et al. (2020) Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, and Songlin Hu.  Graph-based reasoning over heterogeneous external knowledge for commonsense question answering.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2020.
  * Marcus (2018) G. Marcus.  Deep learning: A critical appraisal.  _ArXiv_ , abs/1801.00631, 2018.
  * McCoy et al. (2019) R. Thomas McCoy, Ellie Pavlick, and Tal Linzen.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _ACL_ , 2019.
  * Mehrabi et al. (2021) Ninareh Mehrabi, Pei Zhou, Fred Morstatter, Jay Pujara, Xiang Ren, and A. G. Galstyan.  Lawyers are dishonest? quantifying representational harms in commonsense knowledge resources.  _ArXiv_ , abs/2103.11320, 2021.
  * Mihaylov & Frank (2018) Todor Mihaylov and Anette Frank.  Knowledgeable reader: Enhancing cloze-style reading comprehension with external commonsense knowledge.  In _Association for Computational Linguistics (ACL)_ , 2018.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? A new dataset for open book question answering.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2018\.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel.  Language models as knowledge bases?  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2019\.
  * (25) George Sebastian Pirtoaca.  Ai2 leaderboard.  URL https://leaderboard.allenai.org/open_book_qa/submission/brhieieqaupc4cnddfg0.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research (JMLR)_ , 2020.
  * Ren & Leskovec (2020) Hongyu Ren and Jure Leskovec.  Beta embeddings for multi-hop logical reasoning in knowledge graphs.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Ren et al. (2020) Hongyu Ren, Weihua Hu, and Jure Leskovec.  Query2box: Reasoning over knowledge graphs in vector space using box embeddings.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Ren et al. (2021) Hongyu Ren, Hanjun Dai, Bo Dai, Xinyun Chen, Michihiro Yasunaga, Haitian Sun, Dale Schuurmans, Jure Leskovec, and Denny Zhou.  Lego: Latent execution-guided reasoning for multi-hop question answering on knowledge graphs.  In _International Conference on Machine Learning (ICML)_ , 2021.
  * Santoro et al. (2017) Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap.  A simple neural network module for relational reasoning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Schlichtkrull et al. (2018) Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling.  Modeling relational data with graph convolutional networks.  In _European Semantic Web Conference_ , 2018.
  * Shen et al. (2020) Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, and Weizhu Chen.  Exploiting structured knowledge in text via graph-guided representation learning.  In _EMNLP_ , 2020.
  * Sheng et al. (2020) Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng.  Towards controllable biases in language generation.  In _the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)-Findings, long_ , 2020.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2017.
  * Suchanek et al. (2007) Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum.  Yago: A Core of Semantic Knowledge.  In _16th International Conference on the World Wide Web_ , pp. 697–706, 2007.
  * Sun et al. (2020) Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang.  Colake: Contextualized language and knowledge embedding.  In _COLING_ , 2020.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  In _North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2019.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pp. 5998–6008, 2017.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph attention networks.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=rJXMpikCZ.
  * Vrandečić & Krötzsch (2014) Denny Vrandečić and Markus Krötzsch.  Wikidata: A free collaborative knowledgebase.  _Commun. ACM_ , 57(10):78–85, September 2014\.  ISSN 0001-0782.  doi: 10.1145/2629489.  URL https://doi.org/10.1145/2629489.
  * Wang et al. (2020) Peifeng Wang, Nanyun Peng, Pedro Szekely, and Xiang Ren.  Connecting the dots: A knowledgeable path generator for commonsense question answering.  _arXiv preprint arXiv:2005.00691_ , 2020.
  * Wang et al. (2019) Xiaoyan Wang, Pavan Kapanipathi, Ryan Musa, Mo Yu, Kartik Talamadupula, Ibrahim Abdelaziz, Maria Chang, Achille Fokoue, Bassem Makni, Nicholas Mattei, et al.  Improving natural language inference using external knowledge in the science questions domain.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2019.
  * Wishart et al. (2018) David S Wishart, Yannick D Feunang, An C Guo, Elvis J Lo, Ana Marcu, Jason R Grant, Tanvir Sajed, Daniel Johnson, Carin Li, Zinat Sayeeda, et al.  Drugbank 5.0: a major update to the drugbank database for 2018.  _Nucleic acids research_ , 2018.
  * Xu et al. (2021) Weiwen Xu, Huihui Zhang, Deng Cai, and Wai Lam.  Dynamic semantic graph construction and reasoning for explainable multi-hop science question answering.  _arXiv preprint arXiv:2105.11776_ , 2021.
  * Yan et al. (2020) Jun Yan, Mrigank Raman, Aaron Chan, Tianyu Zhang, Ryan Rossi, Handong Zhao, Sungchul Kim, Nedim Lipka, and Xiang Ren.  Learning contextualized knowledge structures for commonsense reasoning.  _arXiv preprint arXiv:2010.12873_ , 2020.
  * Yang et al. (2019) An Yang, Quan Wang, Jing Liu, Kai Liu, Yajuan Lyu, Hua Wu, Qiaoqiao She, and Sujian Li.  Enhancing pre-trained language representations with rich knowledge for machine reading comprehension.  In _Association for Computational Linguistics (ACL)_ , 2019.
  * Yasunaga et al. (2021) Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec.  QA-GNN: Reasoning with language models and knowledge graphs for question answering.  _ArXiv_ , abs/2104.06378, 2021.
  * Yu et al. (2020) Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng.  Jaket: Joint pre-training of knowledge graph and language understanding.  _ArXiv_ , abs/2010.00796, 2020.
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu.  Ernie: Enhanced language representation with informative entities.  In _ACL_ , 2019.

##  Appendix A Ethics Statement

We outline potential ethical issues with our work below. First, GreaseLM is a
method to fuse language representations and knowledge graph representations
for effective reasoning about textual situations. Consequently, GreaseLM could
reflect many of the same biases and toxic behaviors exhibited by language
models and knowledge graphs that are used to initialize it. For example, prior
large-scale language models have been shown to encode biases about race,
gender, and other demographic attributes (Sheng et al., 2020). Because
GreaseLM is seeded with pretrained language models that often learn these
patterns, it is possible to reflect them in open-world settings. Second, the
ConceptNet knowledge graph (Speer et al., 2017) used in this work has been
shown to encode stereotypes (Mehrabi et al., 2021), rather than completely
clean commonsense knowledge. If GreaseLM were used outside these standard
benchmarks in conjunction with ConceptNet as a KG, it might rely on unethical
relationships in its knowledge resource to arrive at conclusions.
Consequently, while GreaseLM could be used for applications outside these
standard benchmarks, we would encourage implementers to use the same
precautions they would apply to other language models and methods that use
noisy knowledge sources.

Another source of ethical concern is the use of the MedQA-USMLE evaluation.
While we find clinical reasoning using language models and knowledge graphs to
be an interesting testbed for GreaseLM and for joint language and reasoning
models in general, we do not encourage users to use these models for real
world clinical prediction, particularly at these performance levels.

##  Appendix B Experimental Setup Details

###  B.1 Entity Linking

Given each QA context, we follow the procedure from Yasunaga et al. (2021) to
retrieve the subgraph ${\mathcal{G}}_{\text{sub}}$ from ${\mathcal{G}}$.
First, we perform entity linking to ${\mathcal{G}}$ to retrieve an initial set
of nodes ${\mathcal{V}}_{\text{linked}}$. Second, we add any bridge entities
that are in a 2-hop path between any pair of linked entities in
${\mathcal{V}}_{\text{linked}}$ to get the set of retrieved entities
${\mathcal{V}}_{\text{retrieved}}$. Then we prune the set of nodes
${\mathcal{V}}_{\text{retrieved}}$ using a relevance score computed for each
node. To compute the relevance score, we follow the procedure of Yasunaga et
al. (2021) – we concatenate the node name with the context of the QA example,
and pass it through a pre-trained LM, using the output score of the node name
as the relevance score. We only retain the top 200 scores nodes and prune the
remaining ones. Finally, we retrieve all the edges that connect any two nodes
in ${\mathcal{V}}_{\text{sub}}$, forming the retrieved subgraph
${\mathcal{G}}_{\text{sub}}$. Each node in ${\mathcal{G}}_{\text{sub}}$ is
assigned a type according to whether its corresponding entity was linked from
the context $c$, question $q$, answer $a$, or from a bridge path.

###  B.2 Graph Initialization

To compute initial node embeddings (§3.3) for entities retrieved in
${\mathcal{G}}_{\text{sub}}$ from ConceptNet, we follow the method of MHGRN
(Feng et al., 2020). We convert knowledge triples in the KG into sentences
using pre-defined templates for each relation. Then, these sentences are fed
into a BERT-large LM to compute embeddings for each sentence. Finally, for all
sentences containing an entity, we extract all token representations of the
entity’s mention spans in these sentences, mean pool over these
representations and project this mean-pooled representation.

For MedQA-USMLE, node embeddings are initialized similarly using the pooled
token output embeddings of the entity name from the SapBERT model (described
in §4.2; Liu et al., 2021). For MedQA, 5% of examples do not yield a retrieved
entity. In these cases, we represent the graph using a dummy node initialized
with 0. In essence, GreaseLM backs off to only using LM representations as the
graph propagates no information.

###  B.3 Hyperparameters

Table 7: Hyperparameter settings for models and experiments

Category | Hyperparameter | Dataset
---|---|---
CommonsenseQA | OpenbookQA | MedQA-USMLE
Model architecture | Number of GreaseLM layers $M$ | 5 | 6 | 3
Number of Unimodal LM layers $N$ | 19 | 18 | 9
Number of attention heads in GNN | 2 | 2 | 2
Dimension of node embeddings and the messages in GNN | 200 | 200 | 200
Dimension of MLP hidden layers (except MInt operator) | 200 | 200 | 200
Number of hidden layers of MLPs | 1 | 1 | 1
Dimension of MInt operator hidden layer | 400 | 200 | 400
Regularization | Dropout rate of the embedding layer, GNN layers and fully-connected layers | 0.2 | 0.2 | 0.2
Optimization | Learning rate of parameters in LM | 1.00E-05 | 1.00E-05 | 5.00E-05
Learning rate of parameters not in LM | 1.00E-03 | 1.00E-03 | 1.00E-03
Number of epochs in which LM’s parameters are kept frozen | 4 | 4 | 0
Optimizer | RAdam | RAdam | RAdam
Learning rate schedule | constant | constant | constant
Batch size | 128 | 128 | 128
Number of epochs | 30 | 70 | 20
Max gradient norm (gradient clipping) | 1.0 | 1.0 | 1.0
Data | Max number of nodes | 200 | 200 | 200
Max number of tokens | 100 | 100 | 512

##  Appendix C Additional Experimental Results

###  C.1 Ablation studies

In Table 8, we summarize an ablation study conducted using the CommonsenseQA
IHdev set.

Modality interaction. A key component of GreaseLM is the connection of the LM
to the GNN via the modality interaction module (Eq. 11). If we remove modality
interaction, the performance drops significantly, from 78.5% to 76.5%
(approximately the performance of QA-GNN). Integrating the modality
interaction in every other layer instead of consecutive layers also hurts
performance. A possible explanation is that skipping layers could impede
learning consistent representations across layers for both the LM and the GNN,
a property which may be desirable given we initialize the model using a
pretrained LM’s weights (e.g., RoBERTa). We also find that sharing parameters
between modality interaction layers (Eq. 11) outperforms not sharing, possibly
because our datasets are not very large (e.g., 10k for CommonsenseQA), and
sharing parameters helps prevent overfitting.

Table 8: Ablation study of our model components, using the CommonsenseQA IH-dev set. Ablation Type | Ablation | Dev Acc.
---|---|---
GreaseLM | - | 78.5
Modality Interaction | No interaction | 76.5
Interaction in every other layer | 76.3
Interaction Layer Parameter Sharing | No parameter sharing | 77.1
Number of GreaseLM layers ($M$) | $M=4$ | 77.7
$M=6$ | 78.0
$M=7$ | 76.2
Graph Connectivity | Interaction node connected to all | 77.6
nodes in ${\mathcal{V}}_{\text{sub}}$, not only
${\mathcal{V}}_{\text{linked}}$
Node Initialization | Random | 60.8
TransE (Bordes et al., 2013) | 77.7

Number of GreaseLM layers. We find that $M=5$ GreaseLM layers achieves the
highest performance. However, both the results for $M=4$ and $M=6$ are
relatively close to the top performance, indicating our method is not overly
sensitive to this hyperparameter.

Graph connectivity. The interaction node $e_{int}$ is a key component of
GreaseLM that bridges the interaction between the KG and the text. Selecting
which nodes in the KG are directly connected to $e_{int}$ affects the rate at
which information from different portions of the KG can reach the text
representations. We find that connecting $e_{int}$ KG nodes explicitly linked
to the input text performs best. Connecting $e_{int}$ to all nodes in the
subgraph (e.g., bridge entities) hurts performance (-0.9%), possibly because
the interaction node is overloaded by having to attend to all nodes in the
graph (up to 200). By connecting the interaction node only to linked entities,
each linked entity serves as a filter for relevant information that reaches
the interaction node.

KG node embedding initialization. Effectively initializing KG node
representations is critical. When we initialize nodes randomly instead of
using the BERT-based initialization method from Feng et al. (2020), the
performance drops significantly (78.5% $\rightarrow$ 60.8%). While using
standard KG embeddings (e.g., TransE; Bordes et al., 2013) recovers much of
the performance drop (77.7%), we still find that using BERT-based entity
embeddings performs best.

###  C.2 Effect of LM Initialization on GreaseLM

Table 9: Performance on the in-house splits of CommonsenseQA for different LM initializations of our method, GreaseLM.  Methods | IHdev-Acc. | IHtest-Acc.
---|---|---
RoBERTa-Large | 73.1 | 68.7
\+ GreaseLM (Ours) | 78.5 | 74.2
RoBERTa-Base | 65.1 | 59.8
\+ GreaseLM (Ours) | 69.3 | 65.0

Table 10: Initialization on MedQA-USMLE Methods |  Acc. (%)
---|---
SapBERT-Base | 37.2
\+ GreaseLM (Ours) | 38.5
BioBERT-Base | 34.1
\+ GreaseLM (Ours) | 34.6
PubmedBERT-Base | 38.0
\+ GreaseLM (Ours) | 38.7

To evaluate whether our method is agnostic to the LM used to seed the GreaseLM
layers, we replace the LMs we use in previous experiments (RoBERTa-large for
CommonsenseQA and SapBERT for MedQA-USMLE) with RoBERTa-base for
CommonsenseQA, and BioBERT and PubmedBERT for MedQA-USMLE. Across multiple LM
initializations in two domains, our results demonstrate that GreaseLM can
provide a consistent improvement for multiple LMs when used as a modality
junction between KGs and language.