# Synthetic dimension-induced conical intersections in Rydberg molecules

Frederic Hummel<EMAIL_ADDRESS>Zentrum für Optische
Quantentechnologien, Fachbereich Physik, Universität Hamburg, Luruper Chaussee
149, 22761 Hamburg, Germany  Matthew T. Eiles  Max-Planck-Institut für Physik
komplexer System, Nöthnitzer Str. 38, 01187 Dresden, Germany  Peter Schmelcher
Zentrum für Optische Quantentechnologien, Fachbereich Physik, Universität
Hamburg, Luruper Chaussee 149, 22761 Hamburg, Germany  The Hamburg Centre for
Ultrafast Imaging, Universität Hamburg, Luruper Chaussee 149, 22761 Hamburg,
Germany

(August 28, 2024)

###### Abstract

We observe a series of conical intersections in the potential energy curves
governing both the collision between a Rydberg atom and a ground-state atom
and the structure of Rydberg molecules. By employing the electronic energy of
the Rydberg atom as a synthetic dimension we circumvent the von Neumann-Wigner
theorem. These conical intersections can occur when the Rydberg atom’s quantum
defect is similar in size to the electron-–ground-state atom scattering phase
shift divided by $\pi$, a condition satisfied in several commonly studied
atomic species. The conical intersections have an observable consequence in
the rate of ultracold $l$-changing collisions of the type Rb$(nf)$+Rb$(5s)\to$
Rb$(nl>3)$+Rb$(5s)$. In the vicinity of a conical intersection, this rate is
strongly suppressed, and the Rydberg atom becomes nearly transparent to the
ground-state atom.

The Born-Oppenheimer approximation is a cornerstone of chemical and molecular
physics. It provides us with the adiabatic separation of the fast electronic
from the slow vibrational motion, resulting in adiabatic potential energy
surfaces (PES) determined by the electronic structure for a given nuclear
geometry Born and Oppenheimer (1927). When PES become degenerate at a conical
intersection (CI), non-adiabatic interaction effects are important and the
Born-Oppenheimer-based intuition developed in molecular physics breaks down
Köppel  _et al._ (1984); Worth and Cederbaum (2004); Worth  _et al._ (2008).
CI occur frequently in larger molecules with many vibrational degrees of
freedom, for example in the nucleobases Barbatti  _et al._ (2010), and play a
key role in photosynthesis Hammarström and Styring (2008). CI are responsible
for ultrafast radiationless decay mechanisms on the femtosecond time scale
Köppel  _et al._ (1984); Arnold  _et al._ (2018); Mabrouk  _et al._ (2020). A
controlled environment to study CI can be provided by external optical fields
Moiseyev  _et al._ (2008); Šindelka  _et al._ (2011) or by ultracold
interacting Rydberg systems Wüster  _et al._ (2011); Gambetta  _et al._
(2020). Diatomic molecules provide here an exception. Generally, the von
Neumann-Wigner non-crossing theorem forbids the crossing of potential energy
curves (PEC) in diatomic systems, which are determined by a single vibrational
parameter, the internuclear coordinate $R$ von Neumann and Wigner (1993).

Excited electronic states are key players in the appearance of CI and a
particular diatomic system, the collisional complex consisting of a Rydberg
atom and a ground-state atom, has attracted significant interest since nearly
the advent of quantum physics Fermi (1934); Omont (1977); Hickman (1979);
Gallagher (1988); Lebedev and Fabrikant (1996). The interaction between these
two atoms, as described by the Fermi pseudopotential, is primarily determined
by the $s$-wave electron-atom scattering phase shift $\delta_{s}$ Fermi
(1934); Omont (1977). The resulting Born-Oppenheimer PEC can be labeled by the
principal quantum number $n$ and angular momentum quantum number $l$ of the
Rydberg atom. These PEC can be sufficiently attractive to support bound
states, known as ultra-long-range Rydberg molecules Greene  _et al._ (2000);
Bendkowsky  _et al._ (2009); Fey  _et al._ (2020); Eiles (2019). At positive
energies, they are responsible for the collisional dynamics between the two
highly asymmetric atomic partners. Just as the characteristics of a Rydberg
atom are smoothly varying polynomial functions of $n$, so too are the typical
molecular properties, for example the potential depths, dipole moments, and
bond lengths.

For this reason, it is illustrative to imagine that this diatomic system
evolves along a two-dimensional potential energy _surface_ , where the
principal quantum number $n$ plays the role of an additional _synthetic_
dimension. In many other contexts, the introduction of synthetic dimensions
provides a means to control the dimensionality of a system by mimicking
additional degrees of freedom. Synthetic dimensions have been realized in
optical lattices Levi  _et al._ (2011); Kolkowitz  _et al._ (2017), and have
applications in the study of gauge fields Celi  _et al._ (2014), quantum
simulation Boada  _et al._ (2012), and photonics Ozawa  _et al._ (2019).

In the present work, we utilize the synthetic dimension $n$ to circumvent the
non-crossing theorem, and in doing so, we show that CI can appear in the
Rydberg–ground-state atom PES. These CI occur when the Rydberg atom is
initially in a state $\ket{nl_{0}}$ whose fractional quantum defect
$\mu_{l_{0}}$ is similar in size to $\delta_{s}/\pi$, which is typically
satisfied in states with more than $d$-wave angular momentum $l_{0}$. We
highlight results for Rb, where $l_{0}=3$, since interest in such a state has
recently grown due to its importance for the preparation of circular Rydberg
states in the scope of quantum simulation and quantum computing based on
neutral-atom platforms Anderson  _et al._ (2013); Nguyen  _et al._ (2018);
Cortiñas  _et al._ (2020); Cantat-Moltrecht  _et al._ (2020).

Similar to how CI in more traditional molecular systems provide fast
radiationless decays between electronic states, the synthetic CI can
facilitate fast dynamics in Rydberg–ground-state atom collisions. In
particular, we demonstrate that they dramatically suppress the $l$-changing
collision rate, which is otherwise a dominant process in this collision. This
provides a clear experimental observable, heralding the presence of synthetic
CI in several different atomic species.

Figure 1: (a) Illustration of the atomic collision and the corresponding
adiabatic (solid, red $V_{+}$ and blue $V_{-}$) and diabatic (dashed, purple
$t$ and gray $q$) PEC for different principle quantum numbers
$n\in\\{40,50,60,70\\}$ (increasing $n$ is plotted with increasing opacity).
At large internuclear distances, the difference of the potentials $\Delta$ is
determined by the quantum defect of the state with angular momentum $l_{0}$.
The axes are rescaled in order to shift the curves onto the same energy scale.
A semi-classical ground-state atom propagating on $V_{-}$ can either
adiabatically follow the potential and pick up orbital angular momentum to
form a trilobite state, the electronic density of which is shown at the
bottom, or hop to the potential $V_{+}$ and continue with an $f$-state
electronic orbital depicted at the top. (b) The eigenvector components
$\psi_{-}$ corresponding to $V_{-}$ and (c) the non-adiabatic coupling matrix
element $\mathcal{P}$ with varying (scaled) $R$ for different $n$.

The interaction between the two atoms is given by Fermi’s pseudopotential,

$V(\boldsymbol{r},\boldsymbol{R})=2\pi
a(k)\delta^{3}(\boldsymbol{r}-\boldsymbol{R}),$ (1)

where $a(k)=\frac{\tan\delta_{s}(k)}{k}=a(0)+\frac{\pi\alpha}{3}k$ is the low-
energy $s$-wave scattering length, $\alpha$ is the ground-state atoms’s
polarizability, and the wave number $k$ is determined semiclassically by
$\frac{k^{2}}{2}-\frac{1}{R}=-\frac{1}{2n^{2}}$. Higher partial waves can be
neglected because the electron energy is close to zero at the large
internuclear distances considered here. Since
$V(\boldsymbol{r},\boldsymbol{R})$ is a weak perturbation to the Coulomb
potential, we calculate the PEC by diagonalizing
$V(\boldsymbol{r},\boldsymbol{R})$ within a restricted basis. This includes
only the initial state $\ket{nl_{0}}$, whose energy we take as the reference
energy, and the degenerate manifold of hydrogenic $l>l_{0}$ states with
vanishingly small quantum defects. These are blue-detuned by a shift $\Delta$
from the initial state. The $l<l_{0}$ states can be ignored since they are far
detuned, $\Delta_{l<l_{0}}\gg\Delta$; those states with $l>l_{0}$ have very
small detunings relative to $\Delta$, and can be subsumed into the degenerate
manifold.

Two adiabatic PEC result from this diagonalization,

$V_{\pm}(n,R)=\frac{1}{2}\left(q+t+\Delta\pm\sqrt{(q+t+\Delta)^{2}-4q\Delta}\right),$
(2)

where

$\displaystyle q(n,R)$
$\displaystyle=a(k)\frac{2l_{0}+1}{2}\frac{u_{nl_{0}}^{2}(R)}{R^{2}},$ (3a)
$\displaystyle t(n,R)$
$\displaystyle=a(k)\sum_{l=l_{0}+1}^{n-1}\frac{2l+1}{2}\frac{u_{nl}^{2}(R)}{R^{2}},$
(3b)

and the hydrogen atom’s radial wave functions are $u_{nl}(r)$. The accuracy of
the PEC are confirmed quantitatively by large-scale diagonalizations involving
many more Rydberg states Greene  _et al._ (2000); Hamilton  _et al._ (2002),
by Green’s function methods Chibisov  _et al._ (2002); Khuskivadze  _et al._
(2002); Greene  _et al._ (2006), or via local frame transformation techniques
Giannakeas  _et al._ (2020).

The PEC are shown in Fig. 1a using scaled distances and energies for several
principle quantum numbers $n\in\\{40,50,60,70\\}$. They represent the key
ingredient for the investigation of the collisional dynamics between the
Rydberg atom $\ket{nl_{0}}$ and a ground-state atom, which is initialized at
large $R$ with the ground-state atom propagating freely with velocity $v$
along the asymptotically flat potential $V_{-}$. The degenerate manifold is
strongly perturbed by the ground-state atom, causing an attractive potential
$V_{+}$ to descend. At large $R$, its adiabatic electronic state $\psi_{+}$ is
a superposition of high-$l$ states known colloquially as the _trilobite_ state
(see inset wave function). Level repulsion induces an avoided crossing at
$R=R_{0}\approx 2n^{2}$, and as a result $V_{-}$ sharply bends away.

From this picture, we see that, if the collision is purely adiabatic, the
Rydberg atom’s electronic state will change from its original electronic state
(see upper inset) into the trilobite state (see lower inset) as the ground-
state atom drops into the attractive potential of $V_{-}$. This is illustrated
in figure 1b, which shows how the components $t$ and $q$ of the corresponding
adiabatic state $\psi_{-}$ reversed after the crossing is traversed. This has
observable consequences, as it induces an $l$-changing collision at a rate
$\gamma=\sigma\rho v$, where $\sigma$ is the collisional cross section and
$\rho$ the density of ground-state atoms Niederprüm  _et al._ (2015);
Gallagher (1994). Since the Rydberg state changes character nearly at the
outer turning point of the Rydberg orbit due to the steep potential curve
arising from the strong perturbation of the degenerate manifold, we take
$\sigma=\pi R_{0}^{2}$. This purely adiabatic rate $\gamma$ holds unless the
velocity or the non-adiabatic coupling are large enough for a non-adiabatic
transition to occur, in which case the ground-state atom hops across onto
$V_{+}$, preserving the electronic character of the Rydberg atom which
continues through diabatically. In particular, at a CI, where $V_{\pm}$ become
degenerate, the non-adiabatic couplings diverge.

Figure 2: The adiabatic potential energy surfaces, for which the principle
quantum number $n$ of the Rydberg atom serves as a synthetic parameter and
dimension and the internuclear distance $R$ as the conventional parameter.
Conical intersections occur at $n_{1}=79$, $n_{2}=99$, and $n_{3}=111$, where
the potentials become degenerate. Each mesh line corresponds to an integer
$n$. The inset shows a zoom of the region in the vicinity of $n_{2}$.

We search for CI in the PES, functions of both the real adiabatic parameter
$R$ and the synthetic parameter $n$, by analytically summing $t(R)$ Chibisov
_et al._ (2002); Eiles (2019), and then allowing $n$ to become a continuous
variable by replacing the hydrogenic radial wave functions with Whittaker
Coulomb functions valid for non-integer $n$. Figure 2 shows these potential
surfaces over a range of $n$. In this region, the potentials are smoothly
oscillatory functions of $R$ and $n$, but CI appear near the $n$ values
$n_{1}=79$, $n_{2}=99$, and $n_{3}=111$. Here, we must include the effects of
non-adiabatic coupling and interactions.

In a beyond Born-Oppenheimer treatment, the non-adiabatic effects are
manifested in the vibrational Schrödinger equation as derivative coupling
elements of first and second degree. For the electronic two-state problem, an
analytical unitary transformation $U(n)$ exists such that the derivative terms
are eliminated. $U$ represents a rotation about the angle
$\varphi(n,R)=\int_{R}^{\infty}\mathrm{d}R^{\prime}\,\mathcal{P}(n,R^{\prime})$,
where the non-adiabatic derivative coupling matrix element
$\mathcal{P}(n,R)=\braket{\psi_{+}}{\nabla_{R}\psi_{-}}$ quantifies the
coupling between the adiabatic states. $\nabla_{R}$ represents the gradient
operator with respect to $R$. Applying this transformation results in a non-
diagonal potential in the _diabatic representation_ Köppel  _et al._ (1984),

$V_{\text{diabatic}}(n,R)=\begin{pmatrix}q&\sqrt{q\cdot t}\\\ \sqrt{q\cdot
t}&t+\Delta\end{pmatrix}.$ (4)

The diabatic PEC $q$ and $t+\Delta$ are shown as dashed curves in Fig. 1a. The
position $R_{0}$ of the avoided crossing of the adiabatic potentials is
characterized by a local maximum of $\mathcal{P}$, which is illustrated in
figure 1c. It typically coincides with a crossing of the diabatic PEC, which
occurs at a position $R_{0}$ obtained from $d(n,R_{0})=2\sqrt{q\cdot t}$,
where $d(n,R)=|V_{+}-V_{-}|$ is the energy gap between the adiabatic
potentials. From the diabatic representation, the positions of CI can be
obtained easily as the $(n,R)$ values where the off-diagonal coupling vanishes
and the diagonal elements become degenerate. This is possible due to the
oscillatory character of the potentials, since $q$ vanishes at a node of the
$\ket{nl_{0}}$ wave function. If, simultaneously, $t=-\Delta$, then the gap
vanishes $\lim_{R_{0}\rightarrow R_{i}}d(n_{i},R)=0$, and the derivative
coupling diverges $\lim_{R_{0}\rightarrow R_{i}}\mathcal{P}(n_{i},R)=\infty.$
This is characteristic for a CI Worth and Cederbaum (2004). Here, the Born-
Oppenheimer approximation clearly breaks down. By aid of the principle quantum
number $n$, we can tune $q$ and $t$ such that a CI can be realized.

We account for the probability of a non-adiabatic transition between the
adiabatic potential surfaces at the avoided crossing via the semi-classical
hopping probability Clark (1979),

$P(n,v)=\exp\left(\frac{-\pi d(n,R_{0})}{4v|\mathcal{P}(n,R_{0})|}\right).$
(5)

The $l$-changing rate can now be extended to account for this non-adiabatic
process,

$\gamma(n,v)=\pi R_{0}^{2}\rho v(1-P(n,v)).$ (6)

At the CI, $P(n_{i},v)=1$ and the analogue of radiationless transitions in
polyatomic molecular systems occurs. Here, the Rydberg atom becomes
_transparent_ to the colliding ground-state atom.

Figure 3: The rate of $l$-changing collisions in a (a) rubidium or (b) cesium
gas, each with density $\rho=1.95\times 10^{12}\,\mathrm{cm}^{-3}$ for
different temperatures from 10 nK (filled circles) to 0.1 K (diamonds). The
dashed, gray line shows the asymptotic behavior of the rates, proportional to
$n^{4}$. At high temperatures, the rates become approximately independent of
$n$. Very rapid variations in the rate as a function of $n$ occur in rubidium
(a) at $n_{1}=79$, $n_{2}=99$, and $n_{3}=111$, where the rate decreases
drastically and becomes independent of temperature and in cesium (b) at
$n_{1}=43$, $n_{2}=54$, $n_{3}=61$, and $n_{4}=67$. Here, the decrease of the
rate does not affect the lowest temperatures, in the case of $n_{1}$ and
$n_{3}$.

Figure 3 shows the rate of $l$-changing collisions as a function of $n$ and
across many decades of temperatures for both rubidium (a) and cesium (b)
gases. The density, $\rho=1.95\times 10^{12}\,\mathrm{cm}^{-3}$, is typical
for ultracold atomic clouds. The temperature determines the mean relative
velocity of two atoms of the ensemble via $v=\sqrt{6k_{B}T/M}$, where $k_{B}$
is the Boltzmann constant and $M$ is the atomic mass. We focus first on the
regime where no CI are found in the PES, namely $n<75$ in Rb and $n<40$ in Cs.
Here, the splitting of the adiabatic functions scales as the difference in
energy between the potential $V_{-}$ and the detuning $\Delta$, satisfying
$d(n,R_{0})\sim n^{-4}$. Since no CI are present to induce rapid variation in
the derivative coupling, it remains approximately constant as a function of
$n$, $\mathcal{P}(n,R_{0})\sim p$ (compare Fig. 1c). For low temperatures, the
hopping probability is heavily suppressed, $P(n,v)\to 0$, since
$(vn^{4}p)^{-1}$ is large. In this limit, we obtain the adiabatic rate,
proportional to the geometric cross section of the Rydberg atom $\sim n^{4}$,
while at high temperatures, $(vn^{4}p)^{-1}$ is small. Taking the first order
expression for $P(n,v)$ in this limit, we find that the rate becomes
independent of $n$ and $v$. These two limiting cases are supported by the
results shown in Fig. 3. We note that in this high temperature limit, our
approach fails to account for many other physical processes which contribute
to $l$-mixing and are not included in our two-level approximation, such as the
effect of the ion–atom polarization potential, molecular ion formation, and
$n$-changing collisions. We present results for these high temperatures merely
as a consistency check of the model. More comprehensive studies applicable to
this thermal limit can be found in Refs. Omont (1977); Hickman (1979);
Fabrikant (1986); Lebedev and Fabrikant (1996).

At larger $n$ values, this same behavior continues except for the notable
exceptions around specific $n$ values, where the rate drops drastically and
becomes nearly independent of the temperature. These are the points of
collisional transparency induced by the synthetic conical intersections, where
$l$-changing processes are strongly suppressed. Measurements of the
$l$-changing collision rate are routinely taken, typically via field
ionization schemes, and should be capable of directly probing these features.

The points of collisional transparency are relevant for the preparation of
circular Rydberg states Hölzl (2019). Away from the CI and at typical $\mu$K
ultracold temperatures, the rapid rate of $l$-changing collisions, scaling as
$n^{4}$, already exceeds typical radiative decay rates of Rydberg atoms,
scaling as $n^{3}$. Thus, experimental schemes attempting to build
applications on these high-$l$ Rydberg states, must either be performed at low
densities or very low temperatures to avoid rapid decay of the $l_{0}$ state
into higher-$l$ components. Apart from their fundamental interest, this gives
the CI-induced collisional transparencies a crucial application since they
suppress this rate over a wide range of temperatures and densities.

species | $l_{0}$ | $\mu_{l_{0}}$ | $a(0)$ | $\alpha$ | $n_{i}$
---|---|---|---|---|---
Li | 2 | 0.002129 | -6.7 | 164.2 |  $\sim$ 200
Na | 2 | 0.015543 | -5.9 | 162.7 | 31, 38, 44
K | 3 | 0.010098 | -15 | 290.6 | 116, 144
Rb | 3 | 0.0165192 | -15.2 | 319.2 | 79, 99, 111
Cs | 0 | 4.049425 | -21.8 | 401 | 43, 54, 61, 67
Cs | 3 | 0.033392 | -21.8 | 401 | 65, 75, 85
Table 1: Table of alkaline atomic species and the occurrence of collisional
transparency at the lowest few principle quantum numbers $n_{i}$ depending on
the quantum defect $\mu_{l_{0}}$ Lorenzen and Niemax (1983); Han  _et al._
(2006); Goy  _et al._ (1982); Weber and Sansonetti (1987), the zero-energy
scattering length $a(0)$ Eiles (2018); Karule (1965); Engel  _et al._ (2019);
Saßmannshausen  _et al._ (2015), and the atomic polarizability $\alpha$ Miffre
_et al._ (2006); Holmgren  _et al._ (2010); Amini and Gould (2003); Mitroy
_et al._ (2010).

In table 1, we present estimates for the occurrence of synthetic CI in the
alkaline atomic species. In each species, a low-$l$ state with angular
momentum $l_{0}$ is selected such that its quantum defect $\mu_{l_{0}}$ comes
closest to satisfying $\delta_{s}/\pi$. Note that two such states can satisfy
this condition in Cs. Depending furthermore on the zero-energy scattering
length $a(0)$ and the polarizability $\alpha$, we can predict the principle
quantum numbers $n_{i}$ at which the corresponding collisional transparency is
achieved. At large $n>70$, the effect of collisional transparency tends to be
more pronounced in the sense that sub-$\mu$K temperatures are affected. As the
electronic density of states increases with $n$, it is more likely that the
synthetic CI, occurring in continuous $n$ space, occur at nearly integer
values of $n$. At low $n$, this is a matter of chance. However, by selecting
appropriate fine and hyperfine structure states (which are not accounted for
in our analysis), one may further tune the quantum defect to achieve a
stronger effect.

In conclusion, we investigate low-temperature $l$-changing collisions of
ground-state atoms with Rydberg atoms based on the analysis of the adiabatic
Born-Oppenheimer PES underlying this process. These collisions modify the
lifetime of Rydberg states, and along with the associated cross sections,
provide avenues to probe fundamental atomic properties Niederprüm  _et al._
(2015); Schlagmüller  _et al._ (2016); Kanungo  _et al._ (2020); Geppert  _et
al._ (2020). Emerging from the oscillatory character of both these PES and the
corresponding non-adiabatic coupling elements, we find that certain principle
quantum numbers $n_{i}$ exist where the $l$-changing rate is strongly
suppressed, independently of temperature, leading to collisional transparency.
This process can be explained by the existence of hidden conical intersections
occurring in the vibrational coordinate $R$ and the synthetic dimension $n$.
This unconventional mechanism for the occurrence of CI in a diatomic setup
provides a unique environment for tests of fundamental quantum processes such
as vibronic couplings, radiationless decay, and geometric phases, with
applications in Rydberg quantum chemistry. Furthermore, our results provide
immediate opportunities for the control of lifetimes of low-$l$ Rydberg states
as well as means to circumvent unintended limitations in the preparation of
these states and of circular Rydberg states. The collisional transparency
allows also for precision measurements of quantum defects and low-energy
$s$-wave scattering lengths of the underlying electron–atom interaction,
because the positions of conical intersections at specific $n_{i}$ depend
sensitively on these parameters.

###### Acknowledgements.

The authors gratefully acknowledge valuable discussions about the
applicability of our results in the scope of high-$l$ state production with
Tilman Pfau, Florian Meinert, Thomas Dieterle, and Christian Hölzl. ME
acknowledges partial support from the Alexander von Humboldt Stiftung. PS
acknowledges support from the German Research Foundation (DFG) within the
priority program "Giant Interactions in Rydberg Systems" [DFG SPP 1929 GiRyd
project SCHM 885/30-1].

## References

  * Born and Oppenheimer (1927) M. Born and R. Oppenheimer, Ann. Phys. 389, 457 (1927).
  * Köppel  _et al._ (1984) H. Köppel, W. Domcke, and L. S. Cederbaum, Multimode molecular dynamics beyond the Born-Oppenheimer approximation, in  _Advances in Chemical Physics_ (John Wiley & Sons, Ltd, 1984) pp. 59–246.
  * Worth and Cederbaum (2004) G. A. Worth and L. S. Cederbaum, Annu. Rev. Phys. Chem. 55, 127 (2004).
  * Worth  _et al._ (2008) G. A. Worth, H.-D. Meyer, H. Köppel, L. S. Cederbaum, and I. Burghardt, International Reviews in Physical Chemistry 27, 569 (2008).
  * Barbatti  _et al._ (2010) M. Barbatti, A. J. A. Aquino, J. J. Szymczak, D. Nachtigallová, P. Hobza, and H. Lischka, Proc. Nat. Acad. Sci. 107, 21453 (2010).
  * Hammarström and Styring (2008) L. Hammarström and S. Styring, Phil. Trans. R. Soc. B 363, 1283 (2008).
  * Arnold  _et al._ (2018) C. Arnold, O. Vendrell, R. Welsch, and R. Santra, Phys. Rev. Lett. 120, 123001 (2018).
  * Mabrouk  _et al._ (2020) N. Mabrouk, W. Zrafi, and H. Berriche, Mol. Phys. 118, e1605098 (2020).
  * Moiseyev  _et al._ (2008) N. Moiseyev, M. Šindelka, and L. S. Cederbaum, J. Phys. B: At. Mol. Opt. Phys. 41, 221001 (2008).
  * Šindelka  _et al._ (2011) M. Šindelka, N. Moiseyev, and L. S. Cederbaum, J. Phys. B: At. Mol. Opt. Phys. 44, 045603 (2011).
  * Wüster  _et al._ (2011) S. Wüster, A. Eisfeld, and J. M. Rost, Phys. Rev. Lett. 106, 153002 (2011).
  * Gambetta  _et al._ (2020) F. M. Gambetta, C. Zhang, M. Hennrich, I. Lesanovsky, and W. Li, Exploring the many-body dynamics near a conical intersection with trapped rydberg ions (2020), arXiv:2012.01834 [physics.atom-ph] .
  * von Neumann and Wigner (1993) J. von Neumann and E. P. Wigner, Über merkwürdige diskrete eigenwerte, in  _The Collected Works of Eugene Paul Wigner: Part A: The Scientific Papers_, edited by A. S. Wightman (Springer Berlin Heidelberg, Berlin, Heidelberg, 1993) pp. 291–293.
  * Fermi (1934) E. Fermi, Nuovo Cim. 11, 157 (1934).
  * Omont (1977) A. Omont, J. Phys. France 38, 1343 (1977).
  * Hickman (1979) A. P. Hickman, Phys. Rev. A 19, 994 (1979).
  * Gallagher (1988) T. F. Gallagher, Rep. Prog. Phys. 51, 143 (1988).
  * Lebedev and Fabrikant (1996) V. S. Lebedev and I. I. Fabrikant, Phys. Rev. A 54, 2888 (1996).
  * Greene  _et al._ (2000) C. H. Greene, A. S. Dickinson, and H. R. Sadeghpour, Phys. Rev. Lett. 85, 2458 (2000).
  * Bendkowsky  _et al._ (2009) V. Bendkowsky, B. Butscher, J. Nipper, J. P. Shaffer, R. Löw, and T. Pfau, Nature 458, 1005 (2009).
  * Fey  _et al._ (2020) C. Fey, F. Hummel, and P. Schmelcher, Mol. Phys. 118, e1679401 (2020).
  * Eiles (2019) M. T. Eiles, J. Phys. B: At. Mol. Opt. Phys. 52, 113001 (2019).
  * Levi  _et al._ (2011) L. Levi, M. Rechtsman, B. Freedman, T. Schwartz, O. Manela, and M. Segev, Science 332, 1541 (2011).
  * Kolkowitz  _et al._ (2017) S. Kolkowitz, S. L. Bromley, T. Bothwell, M. L. Wall, G. E. Marti, A. P. Koller, X. Zhang, M. Rey, and J. Ye, Nature 542, 66 (2017).
  * Celi  _et al._ (2014) A. Celi, P. Massignan, J. Ruseckas, N. Goldman, I. B. Spielman, G. Juzeliūnas, and M. Lewenstein, Phys. Rev. Lett. 112, 043001 (2014).
  * Boada  _et al._ (2012) O. Boada, A. Celi, J. I. Latorre, and M. Lewenstein, Phys. Rev. Lett. 108, 133001 (2012).
  * Ozawa  _et al._ (2019) T. Ozawa, H. M. Price, A. Amo, N. Goldman, M. Hafezi, L. Lu, M. C. Rechtsman, D. Schuster, J. Simon, O. Zilberberg, and I. Carusotto, Rev. Mod. Phys. 91, 015006 (2019).
  * Anderson  _et al._ (2013) D. A. Anderson, A. Schwarzkopf, R. E. Sapiro, and G. Raithel, Phys. Rev. A 88, 031401(R) (2013).
  * Nguyen  _et al._ (2018) T. L. Nguyen, J. M. Raimond, C. Sayrin, R. Cortiñas, T. Cantat-Moltrecht, F. Assemat, I. Dotsenko, S. Gleyzes, S. Haroche, G. Roux, T. Jolicoeur, and M. Brune, Phys. Rev. X 8, 011032 (2018).
  * Cortiñas  _et al._ (2020) R. G. Cortiñas, M. Favier, B. Ravon, P. Méhaignerie, Y. Machu, J. M. Raimond, C. Sayrin, and M. Brune, Phys. Rev. Lett. 124, 123201 (2020).
  * Cantat-Moltrecht  _et al._ (2020) T. Cantat-Moltrecht, R. Cortiñas, B. Ravon, P. Méhaignerie, S. Haroche, J. M. Raimond, M. Favier, M. Brune, and C. Sayrin, Phys. Rev. Research 2, 022032(R) (2020).
  * Hamilton  _et al._ (2002) E. L. Hamilton, C. H. Greene, and H. R. Sadeghpour, J. Phys. B: At. Mol. Opt. Phys. 35, L199 (2002).
  * Chibisov  _et al._ (2002) M. I. Chibisov, A. A. Khuskivadze, and I. I. Fabrikant, J. Phys. B: At. Mol. Opt. Phys. 35, L193 (2002).
  * Khuskivadze  _et al._ (2002) A. A. Khuskivadze, M. I. Chibisov, and I. I. Fabrikant, Phys. Rev. A 66, 042709 (2002).
  * Greene  _et al._ (2006) C. H. Greene, E. L. Hamilton, H. Crowell, C. Vadla, and K. Niemax, Phys. Rev. Lett. 97, 233002 (2006).
  * Giannakeas  _et al._ (2020) P. Giannakeas, M. T. Eiles, F. Robicheaux, and J. M. Rost, Phys. Rev. A 102, 033315 (2020).
  * Niederprüm  _et al._ (2015) T. Niederprüm, O. Thomas, T. Manthey, T. M. Weber, and H. Ott, Phys. Rev. Lett. 115, 013003 (2015).
  * Gallagher (1994) T. F. Gallagher,  _Rydberg Atoms_, Cambridge Monographs on Atomic, Molecular and Chemical Physics (Cambridge University Press, 1994).
  * Clark (1979) C. W. Clark, Phys. Lett. A 70, 295 (1979).
  * Fabrikant (1986) I. I. Fabrikant, J. Phys. B 19, 1527 (1986).
  * Hölzl (2019) C. Hölzl,  _Towards High $n$ Circular Rydberg States in Ultracold Atomic Gases_, Master’s thesis, Universität Stuttgart (2019).
  * Lorenzen and Niemax (1983) C.-J. Lorenzen and K. Niemax, Physica Scripta 27, 300 (1983).
  * Han  _et al._ (2006) J. Han, Y. Jamil, D. V. L. Norum, P. J. Tanner, and T. F. Gallagher, Phys. Rev. A 74, 054502 (2006).
  * Goy  _et al._ (1982) P. Goy, J. M. Raimond, G. Vitrant, and S. Haroche, Phys. Rev. A 26, 2733 (1982).
  * Weber and Sansonetti (1987) K.-H. Weber and C. J. Sansonetti, Phys. Rev. A 35, 4650 (1987).
  * Eiles (2018) M. T. Eiles, Phys. Rev. A 98, 042706 (2018).
  * Karule (1965) E. Karule, Phys. Lett. 15, 137 (1965).
  * Engel  _et al._ (2019) F. Engel, T. Dieterle, F. Hummel, C. Fey, P. Schmelcher, R. Löw, T. Pfau, and F. Meinert, Phys. Rev. Lett. 123, 073003 (2019).
  * Saßmannshausen  _et al._ (2015) H. Saßmannshausen, F. Merkt, and J. Deiglmayr, Phys. Rev. Lett. 114, 133201 (2015).
  * Miffre  _et al._ (2006) A. Miffre, M. Jacquey, M. Büchner, G. Trénec, and J. Vigué, Eur. Phys. J. D 38, 353–365 (2006).
  * Holmgren  _et al._ (2010) W. F. Holmgren, M. C. Revelle, V. P. A. Lonij, and A. D. Cronin, Phys. Rev. A 81, 053607 (2010).
  * Amini and Gould (2003) J. M. Amini and H. Gould, Phys. Rev. Lett. 91, 153001 (2003).
  * Mitroy  _et al._ (2010) J. Mitroy, M. S. Safranova, and C. W. Clark, J. Phys. B: At. Mol. Opt. Phys. 43, 202001 (2010).
  * Schlagmüller  _et al._ (2016) M. Schlagmüller, T. C. Liebisch, F. Engel, K. S. Kleinbach, F. Böttcher, U. Hermann, K. M. Westphal, A. Gaj, R. Löw, S. Hofferberth, T. Pfau, J. Pérez-Ríos, and C. H. Greene, Phys. Rev. X 6, 031020 (2016).
  * Kanungo  _et al._ (2020) S. K. Kanungo, J. D. Whalen, Y. Lu, T. C. Killian, F. B. Dunning, S. Yoshida, and J. Burgdörfer, Phys. Rev. A 102, 063317 (2020).
  * Geppert  _et al._ (2020) P. Geppert, M. Althön, D. Fichtner, and H. Ott, Diffusive-like redistribution in state-changing collisions between Rydberg atoms and ground state atoms (2020), arXiv:2012.11485 [physics.atom-ph] .


# Where’s the Learning in Representation Learning for Compositional Semantics
and the Case of Thematic Fit

Mughilan Muthupari∗
Columbia University
NY, USA
<EMAIL_ADDRESS>

Asad Sayeed
Dept. of Philosophy, Linguistics,
and Theory of Science
University of Gothenburg, Sweden
<EMAIL_ADDRESS>
Samrat Halder∗
Columbia University
NY, USA
<EMAIL_ADDRESS>

Yuval Marton
University of Washington
WA, USA
<EMAIL_ADDRESS>

###### Abstract

Observing that for certain NLP tasks, such as semantic role prediction or
thematic fit estimation, random embeddings perform as well as pretrained
embeddings, we explore what settings allow for this and examine where most of
the learning is encoded: the word embeddings, the semantic role embeddings, or
“the network”. We find nuanced answers, depending on the task and its relation
to the training objective. We examine these representation learning aspects in
multi-task learning, where role prediction and role-filling are supervised
tasks, while several thematic fit tasks are outside the models’ direct
supervision. We observe a non-monotonous relation between some tasks’ quality
score and the training data size. In order to better understand this
observation, we analyze these results using easier, per-verb versions of these
tasks.

**footnotetext: These authors contributed equally to this work

##  1 Introduction

We examine to what extent models trained on a simplified semantic role
labeling (SRL) task can estimate thematic fit (aka semantic fit), as the
training set size grows – and where most of the learning is stored: in the
word embeddings, the thematic role embeddings, or elsewhere in the neural net.

A major goal of natural language processing (NLP) is to understand the
semantics of language. One traditional NLP task around this is SRL, which
labels word spans in a sentence with thematic roles. Consider the sentence “I
cut the cake with a knife”. We can interpret ‘cut’ as the action, ‘I’ as the
Agent (the performer of the action), ‘cake’ as the Theme of the action (the
thing that underwent the action), and ‘knife’ as the Instrument of the action.
These words, labeled with roles such as Agent, Theme, and Instrument, would be
our representation of the event that the sentence conveys. Other sentences
with similar meanings, e.g., “the cake was cut with the knife by me”, should
have the same (or very similar) event representations. In this work, we focus
on model training with a simplified version of SRL: each event is represented
only by the lemmatized syntactic head of each event argument (including the
predicate), and the semantic roles are the simplified PropBank roles (Arg0,
Arg1, etc.). The reason for this is the current limitations of available
evaluation sets for thematic fit: they are all comprised of lemmatized
syntactic argument heads as well.

Thematic fit is related to SRL. This task aims to identify how well a given
word or concept fits into a role of an event. In our example sentence,
consider these potential replacements for ‘knife’: scissors, fork, and brick.
As humans, we understand that while ‘knife’ is the most typical object for
this situation, both ‘scissors’ and ‘fork’ could also fit, even if not as
naturally. This is because we have the general intuition that all three
objects are plausible instruments for cutting. More so, we know that ‘brick’
is unlikely to fit given the context of cutting a cake. Since thematic fit
datasets are scarce, one challenge in computational linguistics (and
computational psycholinguistics) revolves around how machine learning models
can learn thematic fit indirectly – perhaps from SRL training. To the best of
our knowledge, the state-of-art in this line of work is the residual role-
filler averaging model (ResRoFA-MT) proposed by Hong et al. (2018), with an
adjusted embeddings representation and training data annotation in Marton and
Sayeed (2022).

It has been repeatedly observed that in some settings, random word embeddings
perform as well as pretrained ones, or very nearly, including in our baselines
Tilk et al. (2016); Hong et al. (2018); Marton and Sayeed (2022). In this
paper, we design experiments to answer the following questions: Q1. Why is
this so in our compositional semantics and psycholinguistic tasks? Q2. For
such semantic tasks and architecture, where is the learning encoded? Is it in
the word embeddings, role embeddings, or elsewhere in the neural network? Q3.
Training set size effect: is more data better for this indirect setting and
tasks?

In this work, 1. We compare updating the word embeddings during training to
freezing them. 2. We modify the ResRoFA-MT model architecture in various ways
to understand what contributes the most to the learning: the pretrained (or
random) word embeddings, the thematic role embeddings, or the rest of the
network. 3. In order to be able to train on larger data, we optimized the code
of Hong et al. (2018) and Marton and Sayeed (2022). We release our optimized
codebase***https://github.com/MughilM/RW-Eng-v3-src/tree/arxiv_release, which
trains 6 times faster and includes ablation architectures and a correction to
the training data preparation step.

##  2 Related Work

In event representation models, the main goal is to predict the appropriate
word in a sentence given both the role of that word and the surrounding
context in the form of word-role pairs. One of the best early neural models
was the non-incremental role-filler model (NNRF), by Tilk et al. (2016). This
model was based on selectional preferences, or a probability distribution over
the candidate words. However, one drawback of this model is that
representations of two similarly-worded sentences differing hugely in meaning
would closely resemble each other, e.g., “kid watches TV” and “TV watches
kid”. Another drawback is that the embeddings of the word-role pairs are
summed together to represent the sentence, and so the resulting event
representation vector does not weight the input vectors differently based on
their importance and is not normalized for varying numbers of roles in a
sample.

Hong et al. (2018) extend this model in three ways: First, in addition to the
word prediction task of NNRF, the task of role prediction given the
corresponding word is added, and the two tasks are trained simultaneously
(multi-task learning). This model is known as the non-incremental role-filler
multitask model (NNRF-MT). Second, they apply the parametric rectified linear
unit (PReLU) non-linear function to each word-role embedding, which acts as
weights on the composition of embeddings, and subsequently average the
embeddings, which normalizes for variable length inputs. This model is called
the role-filler averaging model (RoFA-MT). Third, in an effort to tackle the
vanishing gradient problem, residual connections between the PReLU output and
the averaging input were added together. This third iteration is known as the
ResRoFA-MT model. They showed that it performs the best on our thematic fit
tasks, and so we use it as our baseline.

Our work differs from Hong et al. (2018) and Marton and Sayeed (2022) in that
while they focused more on state-of-the-art performance through new modeling
and annotation methods, we aim to understand what controls the learning in
such networks. Also, although Hong et al. confirm in private communication
that they found pre-trained and random embeddings performance similar in
preliminary studies, none of the surveyed previous work published experiments
with pre-trained embeddings. We are the first to do so (using GloVe) and
compare that to using random embeddings.

Previous work suggests a difference between "count" and "predict" models,
where "count" models represent lexical semantics in terms of raw or adjusted
unsupervised frequencies of correlations between words (such as Local Mutual
Information; Baroni and Lenci, 2010) and syntactic or semantic phenomena;
"predict" models involve supervised training to achieve their representations,
e.g., neural models. Baroni et al. (2014) do a systematic exploration of tasks
vs. state-of-the-art count and predict models and find that predict models are
overall superior; for thematic fit, predict models are the same or better than
count models on the best unsupervised setup for the task, although they are
easily beaten by third-party baselines based on supervised learning over count
models. More recently, Lenci et al. (2022) demonstrate that predict-models are
not reliably superior to count-models, but depend on the task and the way the
models are trained. They also show that even recent contextual models (e.g.,
BERT) are not necessarily better for out-of-context tasks than well-tuned
static representations, predict or otherwise. See Appendix A for details on
why we do not use BERT in this work.

##  3 Datasets

We use the Rollenwechsel-English, Version 2 (RW-Eng v2) corpus Marton and
Sayeed (2022) as the training set for all our experiments. This corpus is
sentence-segmented, annotated with morphological analyses, syntactic parses,
and syntax-independent PropBank-based semantic role labeling (SRL). The
syntactic head word of each semantic argument is determined by using several
heuristics to match the parses to the semantic argument spans. Note that a
sentence may have multiple predicates (typically verbs) and therefore multiple
semantic frames (sometimes called “events”), each with its own semantic
arguments, whose span may overlap the argument span of other frames in the
sentence.

The first version of this corpus contained NLTK lemmas, MaltParser parses,
parts-of-speech (POS) tags, and SENNA SRL tags Bird (2006); Nivre et al.
(2006); Collobert and Weston (2007). The second version added layers from more
modern taggers: Morfette lemmas, spaCy syntactic parses and POS tags, and LSGN
SRL tags Chrupala (2011); Honnibal and Johnson (2015); He et al. (2018). In
our experiments here we use the lemmas of the semantic arguments’ head words
in v2.

The sentences themselves are taken from both the ukWaC Ferraresi et al. (2008)
and the British National Corpus (BNC). This corpus contains 78M sentences
across 2.3M documents. This includes 210M verbal predicates with 700M
associated role-fillers. We use the same training, validation, and test split
as Hong et al. (2018). That is, we have 99.2% ( 201.5M samples) in the full
training set, 0.4% in validation, and 0.4% in testing. We run our training
experiments on different subsets of the training data, ranging from 0.1% up to
the full dataset. We cap our vocabulary size at the 50,000 most common words
in that specific subset.

We used the following psycholinguistic test sets:

#### Padó

Padó et al. (2006) 414 verb-argument pairs and the associated judgement
scores. These were constructed from 18 verbs that are present in both FrameNet
and PropBank. For each verb, the three most frequent subjects and objects from
each of the underlying corpora were selected. This process yielded six
arguments per verb per corpus, with some overlap between corpora. For each
verb-argument pair, a judgement was collected online with an average of 21
ratings per item for the argument in subject and object role. The rating was
collected on a Likert scale of 1-7 with the question "How common is it for
[subject] to [verb]?" or "How common is it for [object] to be [verbed]?"

#### McRae

McRae et al. (1998) 1444 pairs of verb-argument pairs in a similar format to
Padó. These were created using a similar rating question as the Padó dataset,
but is a compilation of ratings collected over several studies with
considerable overlap and heterogeneous selection criteria.

#### Ferretti-Instruments and Ferretti-Locations

Ferretti et al. (2001) 274 predicate-location pairs and 248 predicate-
instrument pairs. Based on the McRae dataset (Psychological norms).

#### GDS

Greenberg et al. (2015) 720 predicate-object pairs and their ratings. Only
objects (no subjects), matched for high and low polysemy and frequency, well
fitting vs. poorly fitting. Greenberg and McRae overlap by about a third, but
the human scores are obtained from new surveys.

#### Bicknell

Bicknell et al. (2010) 64 cases. Congruent vs incongruent Patient in an Agent-
Verb-Patient paradigm. Hand crafted, not corpus-based, designed for event-
related potentials-based neurolinguistic experiments.

##  4 Modeling and Methodology

In this setup, an input event is represented as role-word pairs, where the
role is one of the following PropBank Palmer et al. (2005) roles: Arg0, Arg1,
ArgM-Mnr, ArgM-Loc, ArgM-Tmp, and the predicate. The word is the argument’s
syntactic head’s lemma. Both the role and the head word are taken from RW-Eng
v2.***Note the input is not a full sentence, precluding the use of contextual
models such as BERT. See Appendices for details.

We train a feed-forward network in a multi-task learning setting to optimize
word and role prediction accuracy. For target word prediction we give the
prediction layer the target role and a context vector formed as a
multiplication of the input word–role pairs. Similarly, for target role
prediction we feed the same context vector along with the target word,
following the ResRoFA-MT architecture Hong et al. (2018) (Figure 1(a)). Since
the network initialization is random, we perform 5 runs of each experiment and
report the mean with a 95% confidence interval. Following Hong et al. (2018);
Marton and Sayeed (2022), we test each model on the psycholinguistic datasets
(Section 3), for which the models were not directly optimized. The idea behind
using the latter test battery is that the model, even though trained on
(simplified) SRL and word prediction (aka role-filling) tasks, is expected to
be able to make indirect generalizations about predicate–argument fit level
from the training data and the related objectives. These psycholinguistic
tasks are evaluated with Spearman’s rank correlation between the sorted human
scores and the sorted model scores, except for Bicknell, for which we take
accuracy of predicting which argument in each Patient role-filler pair is
(more) congruent Lenci (2011).

All prior work with the ResRoFA-MT model uses two random word embedding sets
(one for input words and one for the target word) and similarly two role
embedding sets. See Figure 1(a).

Our implementation differs in these key aspects:

  * •

Modified model architecture \- Using a single word embeddings set, shared
between the target and input words, and similarly a single role embeddings set
(Section 5.1, Figure 1(b)). In our experiments, we find the non-shared,
redundant embedding layers do not affect the performance while adding (vocab
size 50,000 $\times$ word embedding size 300) 15,000,000 learnable parameters
in the model.

  * •

Changes in Batching \- With previous implementations, one epoch only resulted
in about a third of the data being traversed. The next epoch would start on
the second third and so on. Now, we set the data preprocessing so that one
epoch is one pass through all the training data. Additionally, the data is
preprocessed during the training of each batch, so no time is lost during
training in waiting for the next batch of data to be preprocessed.

  * •

Missing and unknown words handling \- Following Marton and Sayeed (2022) but
unlike Hong et al. (2018), we represent out-of-vocabulary (OOV) words
separately from missing words (empty slots in an event).

  * •

Architectural ablation experiments \- these are described in Section 5, for
ease of readability.

##  5 Experiments and Discussion

(a) Original ResRoFA-MT architecture

(b) Shared Embedding Layer

(c) NIR network

(d) NTR network

(e) NR network

(f) Simple network

Figure 1: The model architectures for our experiments

Embedding | Shared? | Tuned? | Role? | Role Accuracy | Word Accuracy | Epochs*
---|---|---|---|---|---|---
Random | N | Y | Y | $.9655\pm.0014$ | $.1363\pm.0020$ | $11(6)$
Random | Y | Y | Y | $.9671\pm.0003$ | $.1372\pm.0022$ | $11(6)$
GloVe | Y | Y | Y | $.9669\pm.0003$ | $.1374\pm.0005$ | $15(10)$
Random | Y | N | Y | $.6609\pm.0046$ | $.1208\pm.0012$ | $25(20)$
GloVe | Y | N | Y | $.9510\pm.0011$ | $.1291\pm.0006$ | $25(20)$
GloVe | Y | Y | NIR† | $.9036\pm.0013$ | $.1348\pm.0019$ | $11(6)$
GloVe | Y | Y | NTR† | $.9677\pm.0006$ | $.1230\pm.0017$ | $12(7)$
GloVe | Y | Y | NR† | $.9007\pm.0021$ | $.1078\pm.0010$ | $8(3)$
RAND Network‡ | Y | Y | Y | $.1530\pm.0716$ | $.0000\pm.0000$ | -
Simpler Network+ | Y | N | Y | $.9987\pm.0005$ | $.1208\pm.0020$ | $6(1)$

Table 1:  Word and Role accuracy on 1% training data.

  * †

NIR=No input role (in context); NTR=No target role (in prediction); NR=No role

  * ‡

Network with no training that uses previously fine tuned word/role embeddings
as input

  * +

Simpler Feed forward Network with previously fine tuned word/role embeddings
as input

  * *

Epochs in parentheses: the epoch of the effective model (best model before
early stopping after patience limit)

Embed. Shrd Tuned Role Padó McRae GDS Ferretti-Loc Ferretti-Instr Bicknell
Random N Y Y $.5474\pm.0345$ $.3231\pm.0236$ $.4485\pm.0314$ $.2611\pm.0036$
$.2282\pm.0623$ $.5260\pm.1185$ Random Y Y Y $.5280\pm.0274$ $.3384\pm.0174$
$.4388\pm.0206$ $.2532\pm.1421$ $.2266\pm.0391$ $.5000\pm.0673$ GloVe Y Y Y
$.5316\pm.0320$ $.3280\pm.0177$ $.4534\pm.0209$ $.2851\pm.0301$
$.2895\pm.0258$ $.5438\pm.0370$ Random Y N Y $.4396\pm.0344$ $.2838\pm.0109$
$.2841\pm.0246$ $.1767\pm.0273$ $.2086\pm.0322$ $.4781\pm.0450$ GloVe Y N Y
$.4941\pm.0247$ $.3090\pm.0254$ $.4349\pm.0229$ $.3011\pm.0301$
$.3439\pm.0421$ $.5563\pm.0490$ GloVe Y Y NIR $.5079\pm.0587$ $.3205\pm.0580$
$.4217\pm.0472$ $.3054\pm.0791$ $.2543\pm.0796$ $.6042\pm.0896$ GloVe Y Y NTR
$.2400\pm.0294$ $.0937\pm.0258$ $.3845\pm.0083$ $.3071\pm.0017$
$.2621\pm.0531$ $.5469\pm.0388$ GloVe Y Y NR $.2496\pm.1088$ $.1139\pm.0150$
$.3385\pm.0363$ $.2955\pm.1243$ $.2668\pm.0375$ $.5885\pm.0448$ RAND Y Y Y
$-.0001\pm.1090$ $.0109\pm.1604$ $.0365\pm.0784$ $.0165\pm.1048$
$-.0346\pm.0785$ $.4531\pm.1027$ Simpler Y N Y $.3271\pm.0555$ $.2175\pm.0294$
$.3356\pm.0345$ $.1055\pm.0259$ $.0459\pm.1239$ $.5365\pm.0593$

Table 2:  Thematic Fit tests on 1% training data (same models as in Table 1)

###  5.1 Shared Embedding Layer

We modify the network to use a single embeddings set shared between the input
words and target word, by using a single index-to-embedding mapping layer –
and similarly a shared embedding-mapping layer for the input roles and target
role (Figure 1(b)). This change results in 2x the training speed (Section 4)
without degradation in performance: role accuracy remains at 96.6-96.7%, word
accuracy at 13.6-13.7%, Padó at 52-54%, McRae at 32-33%, and so on (see first
two rows in Tables 1 and 2). Therefore we use the faster shared architecture
for the rest of the experiments. We train all models (until Section 5.5) on a
uniformly sampled 1% subset, which is large enough to get indicative results
while saving time and cost in experimentation. For comparison of our results
to previous work, see Section 5.5.

###  5.2 Random vs. Pre-trained Embeddings

Hong et al. (2018) used random Glorot uniform to initialize the word
embeddings. Private communication with the authors confirmed random embeddings
do as well as pretrained ones for these tasks. We replicate this finding,
comparing random word embeddings to pretrained GloVe embeddings Pennington et
al. (2014), both of size 300: role accuracy at 96.7%, word accuracy at 13.7%,
Padó at 52.8-53.2%, McRae at 32.8-33.8%. Similar trend follows across all the
thematic fit task results with overlapping 95% confidence intervals of the
experiments with random and GloVe embeddings (rows 2 and 3 in the top part of
Tables 1 and 2).

(Q1) Why is this so? We note that during training, embeddings get updated. To
check if this update is responsible for bridging the gap between zero
knowledge (random embeddings) and much knowledge (compressed in the pre-
trained GloVe), we freeze the word embedding layer and rerun the experiments
(see the middle part in the same two tables). Contrary to our previous
experiment, we find fixed GloVe embeddings do much better than fixed random
embeddings on all our tasks. We also see tuning helps the model converge much
faster (from 25 epochs down to 11-15).

We conclude that indeed much of the learning is captured in the word
embeddings. Tuning them even on only 1% of our training data bridges the
knowledge gap from the pre-trained embeddings almost completely (with possible
exceptions on Ferretti and Bicknell). But we note that although lower, the
fixed embeddings results are not near-random. This leads us to (Q2) Where else
is learning done, and to what extent?

###  5.3 Role Contribution

We now turn to role ablation tests. First we take away the input roles from
the context embeddings and call this the no-input-roles network NIR (see
Figure 1(c) and the third part of Tables 1 and 2). We do not see large drops
in word prediction (from 14.7% to 13.5%), or thematic fit tasks such as Padó
(from 53.2% to 50.2%) and McRae (32.1% to 32.8%), except role prediction (from
96.7% to 90.4%), which we expect by construction. Note that when predicting
the target word, the NIR network still receives the target role information,
which, together with at least the predicate, is likely often sufficient
information for prediction.

We find it surprising that input role ablation barely affects performance on
the psycholinguistic tasks. Why is that? One possibility: the input role
contribution is negligible. But another possibility is that in NIR, all (or
almost all) the role information was crammed into the target role embeddings.
To tease these apart, we next take away the target role from the penultimate
layer of the network, but leave the input roles intact. We call this no-
target-role network NTR (see Figure 1(d) and the row after NIR in the same
tables). Now the role accuracy goes back to the base level of 96.7% (as
expected by construction), but word accuracy drops (from 13.7% to 12.3%) and
so does performance on the psycholinguistic tasks, e.g., Padó (from 53.2% to
24%), McRae (32.8% to 9.4%). We conclude that target role carries more crucial
information than input roles for our psycholinguistic tasks, and that role
information cramming, if it happens in NIR, does not happen in the other
direction (NTR).

Finally, for completeness, we remove all role information from the network. We
call this no-role network NR (see Figure 1(e) and same tables). This results
in a drastic drop in word accuracy (from 13.7% to 10.8%) in addition to
degradation of role accuracy from NIR experiment as well as the
psycholinguistic tasks (Padó falls from 53.2% to 25%, McRae from 32.8% to
11.4%, and so on). This is an an interesting finding which supports previous
knowledge about the importance of roles in multi-task learning setting while
at the same time defies the importance of roles in the context vector (the
output of the residual block in Figure 1). Next, we turn to learn more about
the impact this vector and the block it is in.

###  5.4 “It’s the Network!”… Or is it?

In order to see how much the particular ResRoFA-MT model architecture (aka
“the network”) contributes in our tasks, we first use the fine-tuned GloVe
embedding from a previously trained base model (third row in Table 1) and
assign the rest of the network random weights (“RAND Network” in Tables 1 and
2). To ensure the random weights are similar in size to the trained weights,
we calculate the mean and standard deviation for each layer separately and
assign that layer random weights using a Gaussian distribution with the same
parameters. We see this new model does very poorly, near random prediction
(word accuracy at 0%, role accuracy at 15.3%, Padó at 0%, McRae 1.1% and so
on). This could be due to the learned representation in the network weights
that were ablated here but also due to incompatibility of the non-trained
random network weights with the very informative word embeddings.

Therefore next we replace the complex middle residual block with a plain dense
layer but let this “Simpler Network” (Figure 1(f), Tables 1 and 2) learn
during training. In training here we use the fine-tuned word (and role)
embeddings from our base model. Curiously, we see a notable jump in role
accuracy (from 96.7% to 99.9%), but a drop in word accuracy (from 13.7% to
12.1%) as well as in the psycholinguistic tasks (Padó goes down from 53.2% to
32.7% , McRae from 32.8% to 21.8%, etc.) other than Bicknell’s (53.7-54.4%).
We speculate the latter task is an outlier here because it involves comparing
the plausibility of two two-participant events with one participant changed. A
simpler network may have an easier time representing binary distinctions
within a pair of simple events, as opposed to predicting fine-grained scores
of more complex inter-relationships, evaluated with Spearman’s $\rho$ in the
other datasets. It may even be able to rely on general collocation statistics
here, regardless of roles, but we leave this for future work. Note that here,
we still do multi-task prediction as before, but in a much simpler network.

This, along with the role ablation experiments, suggest that while the
potential incompatibility of the non-trained random network weights with the
word embeddings may account for some of the drop in performance, the context
vector formation through multiplication and likely also the improvements
implemented in our base model have a large impact on the representation
learning as tested on the thematic fit tasks (although not the same impact on
word/role prediction).

We see again that there is no clear correlation between the increase in
directly optimized for word/role prediction, and the performance on the
psycholinguistic tasks for which the models were not directly optimized.

To recap, it seems the answer to (Q2) is nuanced: Padó and McRae are most
sensitive to ablated roles; GDS, and perhaps Bicknell, to non-tuned random
word embeddings; Ferretti to ablated (simplifiled) networks; and all are
sensitive to RAND Networks, but Bicknell is surprisingly robust even there.

###  5.5 Training Data Size Effect

Often in machine learning and NLP, models learn better with more data.
However, there are typically diminishing returns. To test the effect of
training data size, we use our shared layer network with tuned GloVe
embeddings (as in row 3 in Table 1) on uniformly sampled 1%, 10%, 20% 40% and
100% of the training dataset. See Table 3 and Table 4.

Sys | Role Accuracy | Word Accuracy | Epochs
---|---|---|---
B1† | $.9470$ | - | -
B2‡ | $.9715\pm.0010$ | $.1541\pm.0045$ | -
20%M+ | $.9707\pm.0002$ | $.1450\pm.0004$ | -
0.1% | $.9446\pm.0015$ | $.0994\pm.0024$ | $12(7)$
1% | $.9669\pm.0003$ | $.1374\pm.0005$ | $15(10)$
10% | $.9701\pm.0002$ | $.1443\pm.0006$ | $13(10)$
20% | $.9703\pm.0004$ | $.1445\pm.0009$ | $9(6)$
40% | $.9704\pm.0007$ | $.1442\pm.0011$ | $9(6)$
100%***100% experiments are comprised of 2 trials while the rest are comprised of 5. | $.9708\pm.0006$ | $.1444\pm.0019$ | $7(4)$

Table 3:  Comparison of performance with GloVe (tuned) with varying training
set sizes (Sys)

  * †

Hong et al. (2018) 20%

  * ‡

Marton and Sayeed (2022) 20%

  * +

The average of max value in each trial for fair comparison with benchmarks
B1,B2

System Padó McRae GDS Ferretti-Loc Ferretti-Instr Bicknell B1 $.5300$ $.4250$
$.6080$ $.4630$ $.4770$ $.7450$ B2 $.5363\pm.0035$ $.4322\pm.0232$ - - - -
20%M ${.5855}\pm.0101$ $.4338\pm.0181$ $.5495\pm.0220$ $.3539\pm.0239$
$.4255\pm.0210$ $.6094\pm.0000$ 0.1% $.2992\pm.0441$ $.1856\pm.0157$
$.1699\pm.0180$ $.0891\pm.0306$ $.0367\pm.0203$ $.4906\pm.0402$ 1%
$.5316\pm.0320$ $.3280\pm.0177$ $.4534\pm.0209$ $.2851\pm.0301$
$.2895\pm.0258$ $.5438\pm.0370$ 10% $.5572\pm.0247$ $.3993\pm.0137$
$.5409\pm.0150$ $.3410\pm.0358$ $.3765\pm.0320$ $.5906\pm.0320$ 20%
$.5241\pm.0558$ $.3708\pm.1182$† $.5245\pm.0148$ $.3191\pm.0312$
$.3853\pm.0454$ $.5813\pm.0210$ 40% $.3662\pm.1355$ $.3831\pm.0276$
$.5467\pm.0183$ $.3331\pm.0215$ $.3660\pm.0284$ $.5750\pm.0460$ 100%‡
$.3375\pm.7293$ $.3733\pm.5203$ $.5338\pm.1328$ $.2736\pm.7846$
$.3416\pm.3297$ $.6094\pm.1985$

Table 4:  Thematic Fit with GloVe tuned (same models as in Table 3)

  * †

1 trial had an outlier score .2026

  * ‡

All experiments had 5 runs per training subset, except for the 100% with only
2 runs, due to compute resource limitation.

First, in order to compare fairly with previous work, we report the average of
the maximum value in each training trial on 20% of the data. (Recall that our
20% of the data is a larger training set than our baselines’ 20% due to
improvements in our batcher). Our role accuracy (97.1%) is better than Hong et
al. (2018) (94.7%) and similar to Marton and Sayeed (2022) (97.2%). Our word
accuracy (14.5%) is a bit lower than the latter (15.4%). On the indirectly
supervised thematic fit tasks, our results are better on Padó (58.6% compared
to 53%), similar on McRae (42.5-43.4%), but lower for the rest. We suspect
that in the previous work authors reported the best of all the epochs from all
trials, which can explain why the previously reported scores are higher than
our results; but we could not verify that.

In order to better understand the effect of training set size (Q3), we use
next what we believe to be more realistic numbers: the average of the last
saved model in each run (best model per our validation set) in each training
subset size.

We see incremental improvements from the 0.1% subset (role accuracy at 94.5%,
word accuracy at 9.9%) to the 1% subset (role accuracy at 96.7%, word accuracy
at 13.7%) to the 10% subset (role accuracy at 97.0%, word accuracy at 14.4%)
across all our evaluation tasks; however, contrary to our null hypothesis, we
see diminishing returns or no gains in role and word prediction when using 20%
or more of the training set. In most of the psycholinguistic tasks (Table 4),
results plateau at 10% or 20% (GDS at 52.5-54.1%, Ferretti-Loc at 32-34.1%,
Ferretti-Instr at 36.6-38.5% and Bicknell at 57.5-59.1%) with the notable
exception of Padó (best at 55.7% with 10% training data) and McRae (best at
40% at 10% training data) , where we see a negative trend at and beyond 20%.
Why is it so, and only for these two tasks, with mainly Padó? The Padó dataset
is constructed from high-frequency fillers. It behaves differently from the
other datasets and gets a high maximum average score on the 20% subset
probably because there is more training data available for high-frequency
fillers, compared to the other datasets, including McRae. Considering the
small samples in these test sets, they might quickly become victims of not
only high variance, but also of overfitting, that is to say, the models may
specialize on the corpus distribution, increasingly with training set size.
This distribution is likely to be different from the WSJ distribution, from
which Padó dataset is drawn (but see also Section 5.6).

How do word/role prediction and thematic fit tasks relate to each other? We
leave this question for future research, but our hypothesis is that
psycholinguistic meaning of natural language is grounded in interaction with
other modalities (e.g., actions, vision, audio), which a model cannot learn
just from more textual training data.

This leads potentially to a much bigger question: how much can a neural model
learn natural language by just being trained on very large corpora or billions
of parameters, and where is the saturation point? Furthermore, we see role
information is important to our psycholinguistic tasks; how much does the role
definition and granularity (e.g., PropBank or FrameNet), or the role set size,
matter for these tasks? Possibly, with a richer roleset, we may see more
alignment between word/role prediction and the psycholinguistic tasks. Perhaps
PropBank roles are too coarse-grained to allow for an analysis of how a role-
prediction task relates to a thematic fit task, which involves the fine-
grained ranking (via Spearman’s $\rho$) of event plausiblities derived from
the underlying semantic characteristics of the nouns and verbs involved. If
so, understanding how performance on a role-prediction task relates to
thematic fit judgements may not be possible without a finer-grained inventory
of semantic characteristics, such as Dowtyan proto-roles (Dowty, 1991).

###  5.6 Global and Local Correlation

We evaluate both Padó and McRae by computing Spearman’s rank correlation
between the sorted list of model’s probability scores and the sorted list of
averaged human scores, for each dataset. Why do Padó and McRae deteriorate
with increasing training data size? To test if this is due to fluctuation of
model scores for unrelated but near-in-score verb-noun pairs, we averaged
correlations for local subsets, grouped by verb. This should be an easier
task, since some of the globally close competition is not present in each by-
verb subset. Indeed, we see high jumps of 5-8% for the local correlation
scores in the larger subsets (40% and 100%). But in the smaller subsets we see
changes of 2-3% up or down. Moreover, the trend of lower correlation with
larger training sets remained. We leave it to future work to dig further into
why Padó and McRae show such an anomaly.

##  6 Conclusions and Future Work

In this work, we explored why random word embeddings counter-intuitively
perform as well as pretrained word embeddings on certain compositional
semantic tasks (some being outside the models’ explicit objective), where the
learning is actually stored (teasing apart the word embeddings, role
embeddings, and the rest of the network), and how training set size affects
performance on these tasks. We found out that tuning (or further tuning) the
word embeddings helps and can bridge the gap between random and pretrained
embeddings. Moreover, our tuned embedding space is different from pretrained
embeddings like GloVe. We saw that the target role is more important than the
input roles on our tasks. Furthermore, our experiments suggested that much of
the learning happens also in the rest of the network outside word and role
embedding layers. No single factor (word and role embeddings or the network)
is most important for all tasks.

Training set size had a surprising negative effect on Padó and McRae beyond
20% of the training data. We attempted explaining this with an alternative
evaluation method, but this remains to be explained further.

We release our code, including our preferred network architecture – a modified
version of ResRoFA-MT with shared embedding layers.

One avenue in which we want to invest is to better understand the complex
relationship between word/role accuracy and our psycholinguistic tasks. While
our initial hypothesis was that training the network to minimize loss on
word/role prediction would also optimize performance on all our tasks, this
did not always hold. We suspect that the groundedness is the missing link for
(artificially and naturally) learning psycholinguistic tasks, and therefore
adding grounding seems promising to us.

Another future avenue is to investigate the high variability in
psycholinguistic task performance compared to the fairly stable results on the
directly optimized-for word and role prediction tasks.

## Limitations

There are certain limitations that were unavoidable in this work. One of them
is the limited size of the available training and evaluation datasets for
testing thematic fit tasks. It is likely that the high variance we observed is
due to both our indirect supervision approach (in part due to lack of directly
relevant data for training), and the small-size test sets. We are limited here
by the state of the art in such datasets, not just by their size. It is a
complex task to create and evaluate thematic fit with full phrases and
sentences, i.e., not just with the arguments’ syntactic heads. Since we do not
know of any such datasets, our model was designed with only syntactic heads in
mind.

Another limitation is the training dataset quality: due to its size, the
training data was machine-annotated (for syntactic parsing, SRL and lemmas)
and therefore unintended noise and bias may have been introduced in the
models. In addition, even though our training datasets were collected with the
goal of making them domain-general and balanced, it is hard to enforce and
verify that in large sizes. We take issues such as toxicity and gender bias
seriously, but we think that in our settings, where the model does not
generate language and the test sets do not involve gendered examples, the
related risks approach zero.

Semantic tasks such as thematic fit would most likely benefit from training on
grounded language, e.g., combining text and vision, but working with such
datasets is beyond the scope of this work.

Finally, a rather trivial limitation we have is the number of trials per
experiment we could run due to time and computational constraints. We only ran
3-5 trials per experiment but a larger number of trials may yield more robust
results. Despite all these limitations, we believe our work gives a very
comprehensive analysis of the ResRoFaMT model and opens up some interesting
avenues for future research work.

## Ethical Considerations

Our work uses RW-Eng v2 Marton and Sayeed (2022), which in turn uses two
corpora: ukWaC and the BNC. Therefore, we have similar ethical concerns as
mentioned in that previous work, including the way the BNC data was collected.
Those who so wish can easily exclude the BNC data (it comprises only a small
part of the whole corpus) and retrain.

The RW-Eng corpus (v1 or v2) could introduce undesired bias in use outside the
UK, since the data is sourced entirely from UK web pages and other UK sources
from the 20th century. English used outside the UK, and more recent English
anywhere, differ from this corpus in their word distributions, and therefore
their input may yield sub-optimal or undesired results. Furthermore, models
trained on it could encode a Western-centric view of the world.

The silver labels – the automatic parsing and tagging of the corpus – could
introduce bias from the parsing / tagging algorithms. These parsers / taggers
are also trained models, which could be affected by their data sources. If
this is a concern for some users, we encourage them to perform validation of
the data and its annotations.

Having said that, we believe that for most if not all conceivable
applications, especially as long as one keep these limitations in mind, our
work should not pose any practical risk.

## Acknowledgements

This work started as a Masters Capstone project at the Columbia University
Data Science Institute. We would like to thank Data Science Institute, and the
Department of Computer Science at Columbia University for their support.
Specifically, Smaranda Muresan for early discussions and Eleni Drinea for
timely support with our computing resources in the initial stages of this
project. We would like to thank Google Cloud Platform for an award of credits
to Asad Sayeed, allowing us to use Google Cloud computing resources. We are
grateful to the rest of the Capstone team members – Jake Stamell, Anjani
Prasad Atluri and Priyadharshini Rajbabu – for their valuable contributions.
This research was funded in part by a Swedish Research Council (VR) grant
(2014-39) for the Centre for Linguistic Theory and Studies in Probability
(CLASP).

## References

  * Baroni et al. (2014) Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014.  Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors.  In _ACL (1)_ , pages 238–247.
  * Baroni and Lenci (2010) Marco Baroni and Alessandro Lenci. 2010.  Distributional memory: A general framework for corpus-based semantics.  _Computational Linguistics_ , 36(4):673–721.
  * Bicknell et al. (2010) Klinton Bicknell, Jeffrey L Elman, Mary Hare, Ken McRae, and Marta Kutas. 2010.  Effects of event knowledge in processing verbal arguments.  _Journal of Memory and Language_ , 63(4):489–505.
  * Bird (2006) Steven Bird. 2006.  NLTK: the natural language toolkit.  In _Proceedings of the COLING/ACL on Interactive presentation sessions_ , pages 69–72. Association for Computational Linguistics.
  * Chrupala (2011) Grzegorz Chrupala. 2011.  Efficient induction of probabilistic word classes with LDA.  In _Proceedings of 5th International Joint Conference on Natural Language Processing_ , pages 363–372, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.
  * Collobert and Weston (2007) Ronan Collobert and Jason Weston. 2007.  Fast semantic extraction using a novel neural network architecture.  In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_ , pages 25–32.
  * Dowty (1991) David Dowty. 1991.  Thematic proto-roles and argument selection.  _Language_ , 67(3):547–619.
  * Ferraresi et al. (2008) Adriano Ferraresi, Eros Zanchetta, Marco Baroni, and Silvia Bernardini. 2008.  Introducing and evaluating ukWaC, a very large web-derived corpus of english.  _Proceedings of the 4th Web as Corpus Workshop (WAC-4) Can we beat Google_ , pages 47–54.
  * Ferretti et al. (2001) Todd R Ferretti, Ken McRae, and Andrea Hatherell. 2001.  Integrating verbs, situation schemas, and thematic role concepts.  _Journal of Memory and Language_ , 44(4):516–547.
  * Greenberg et al. (2015) Clayton Greenberg, Vera Demberg, and Asad Sayeed. 2015.  Verb polysemy and frequency effects in thematic fit modeling.  In _Proceedings of the 6th Workshop on Cognitive Modeling and Computational Linguistics_ , pages 48–57, Denver, Colorado. Association for Computational Linguistics.
  * He et al. (2018) Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018.  Jointly predicting predicates and arguments in neural semantic role labeling.  _CoRR_ , abs/1805.04787.
  * Henlein and Mehler (2022) Alexander Henlein and Alexander Mehler. 2022.  What do toothbrushes do in the kitchen? how transformers think our world is structured.
  * Hong et al. (2018) Xudong Hong, Asad Sayeed, and Vera Demberg. 2018.  Learning distributed event representations with a multi-task approach.  In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_ , pages 11–21, New Orleans, Louisiana. Association for Computational Linguistics.
  * Honnibal and Johnson (2015) Matthew Honnibal and Mark Johnson. 2015.  An improved non-monotonic transition system for dependency parsing.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1373–1378, Lisbon, Portugal. Association for Computational Linguistics.
  * Lenci (2011) Alessandro Lenci. 2011.  Composing and updating verb argument expectations: A distributional semantic model.  In _Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics_ , CMCL 2011, pages 58–66, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Lenci et al. (2022) Alessandro Lenci, Magnus Sahlgren, Patrick Jeuniaux, Amaru Cuba Gyllensten, and Martina Miliani. 2022.  A comparative evaluation and analysis of three generations of distributional semantic models.  _Language Resources and Evaluation_ , pages 1–45.
  * Marton and Sayeed (2022) Yuval Marton and Asad Sayeed. 2022.  Thematic fit bits: Annotation quality and quantity interplay for event participant representation.  In _Proceedings of the 13th Language Resources and Evaluation Conference_.
  * McRae et al. (1998) Ken McRae, Michael J Spivey-Knowlton, and Michael K Tanenhaus. 1998.  Modeling the influence of thematic fit (and other constraints) in on-line sentence comprehension.  _Journal of Memory and Language_ , 38(3):283–312.
  * Nivre et al. (2006) Joakim Nivre, Johan Hall, and Jens Nilsson. 2006.  MaltParser: A data-driven parser-generator for dependency parsing.  In _Proceedings of LREC_ , volume 6, pages 2216–2219.
  * Padó et al. (2006) Ulrike Padó, Frank Keller, and Matthew W Crocker. 2006.  Combining syntax and thematic fit in a probabilistic model of sentence processing.  In _Proceedings of the 28th CogSci_ , pages 657–662.
  * Palmer et al. (2005) Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005.  The proposition bank: An annotated corpus of semantic roles.  _Computational Linguistics_ , 31(1):71–106.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Tilk et al. (2016) Ottokar Tilk, Vera Demberg, Asad Sayeed, Dietrich Klakow, and Stefan Thater. 2016\.  Event participant modelling with neural networks.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 171–182, Austin, Texas. Association for Computational Linguistics.

## Appendix A: Why Not Use BERT Here?

With the advent of contextual embeddings, static word embeddings are often
regarded as inferior or outdated. While this is true in many cases, we wish to
point out that (a) there are still cases where static word embeddings
outperform BERT (Lenci et al. (2022); Henlein and Mehler (2022), inter alia)
and more importantly, (b) not all NLP tasks and test sets are in the form of
complete sentences, which may render contextual models useless there. More
specifically for our tasks:

  1. 1.

As we point out at the end of Section 2, “Lenci et al. (2022) demonstrate that
… even recent contextual models such as BERT are not necessarily better for
out-of-context tasks than well-tuned static representations, predict or
otherwise." Our tasks, as represented by several psycholinguistic test sets
are such out-of-context tasks. This is the state-of-the-art in
psycholinguistics datasets. The human judgments in these sets were given
without a full sentence or other context beyond the verb-noun (or noun-verb-
noun) items. There is no reason to assume that BERT, a contextual model
trained on full sentences will do well on out-of-context tasks, and again, it
has been shown BERT is not necessarily better than static word embeddings.

  2. 2.

Even if we wanted to use BERT, we cannot use a lookup table the same way we
can for static word embeddings. This means we will either have to decode each
sentence on the fly every training (and evaluation) iteration, or decode once
and save on disk. Simple calculation shows that the required storage demands,
even for, say, 1% of the data, make this exercise computationally extremely
expensive.

  3. 3.

BERT may break words to several tokens. How to map these to the verb or noun
in the training or test is not always straightforward, and this mapping makes
embedding extraction speed 0.5x slower.

  4. 4.

“Hallucinating” synthetic sentences from the verb-noun input in order for BERT
to receive a sentence for input would invalidate the ratings given by the
human raters without these (or other) sentences.

  5. 5.

In order to validate our claims here, we experimented with BERT on-the-fly in
preliminary studies, using a small training subset of a few thousand sentences
with simple token mapping, and the results were dismal while the training
already excruciatingly slow.

  6. 6.

There is nothing wrong with systematically exploring models that use static
word embeddings, even if contextual embeddings excite many people more. We
don’t think we should defend this choice.

## Appendix B: Dataset Examples

For added clarity we give the readers a few examples of the training and
evaluation data.

  1. 1.

Each training example is a list of word/role pairs. Each of these examples are
created with the head words from full sentences, i.e., our final training
examples do not contain full context. We have 6 semantic roles such as Arg0
and Arg1 etc. + one placeholder UNK role for roles not included in our role
set. A typical example could look like this: {0: 6, 1: 97, 2: 43511, 3: 43511,
4: 239, 5: 143, 6: 64}, where the numbers are pairs of role:word indices. The
size of word vocabulary of the 0.1% training subset is 43,510 with tokenId
43,510 and 43,511 corresponding to UNK and Missing word (the corresponding
role does not have a head word). Larger training subsets vocabulary is capped
at 50,000.

Each example is used in training 1 or more times, each time with a different
target word/role pair (see Figures 1(a), 1(b), and Section 4), while the rest
of the pairs are used for input. Note that a pair with a Missing word cannot
serve as a target word/role.

  2. 2.

Apart from the train / test split, we use multiple psycholinguistic evaluation
sets that we do not optimize the model on, as mentioned in Section 3. While
they all vary, a typical example is {client, advise, Arg0, 3.7}, which means
that human raters gave an average of 3.7 to ’client’ as Argument 0 (typically
Agent) for ’advise’ (as in “the client advised the banker that …”). In
contrast, {client, advise, Arg1, 6.6}, means that human raters gave an average
of 6.6 to ’client’ as Argument 1 (typically Theme/Patient) for ’advise’ (as in
“the banker advised the client that …”). According to these human raters,
’client’ fits semantically much better as Arg1 than Arg0 for ’advise’. During
thematic fit evaluation, we sort these test examples by human rater average
scores, and the model output by model score. Then, we compute Spearman’s rank
correlation between the two sorted lists, as explained in Section 5.6.

  *[SRL]: semantic role labeling
  *[NLP]: natural language processing
  *[ResRoFA-MT]: residual role-filler averaging model
  *[NNRF]: non-incremental role-filler model
  *[NNRF-MT]: non-incremental role-filler multitask model
  *[PReLU]: parametric rectified linear unit
  *[RoFA-MT]: role-filler averaging model
  *[POS]: parts-of-speech
  *[BNC]: British National Corpus
  *[OOV]: out-of-vocabulary