# Coexistence of non-Abelian chiral spin liquid and magnetic order in a spin-1
antiferromagnet

Yixuan Huang  Department of Physics and Astronomy, California State
University, Northridge, California 91330, USA  W. Zhu<EMAIL_ADDRESS>Key Laboratory for Quantum Materials of Zhejiang Province, School of Science,
Westlake University, Hangzhou 310024, China  Shou-Shu Gong  Department of
Physics, Beihang University, Beijing 100191, China  Hong-Chen Jiang  Stanford
Institute for Materials and Energy Sciences, SLAC and Stanford University,
Menlo Park, California 94025, USA  D. N. Sheng<EMAIL_ADDRESS>Department of Physics and Astronomy, California State University, Northridge,
California 91330, USA

###### Abstract

We study the ground-state properties of a spin-1 Heisenberg model on a square
lattice with the first- and second-nearest-neighbor antiferromagnetic
couplings $J_{1}$ and $J_{2}$ and a three-spin scalar chirality term
$J_{\chi}$. Using the density matrix renormalization group calculation, we map
out a global phase diagram including various magnetic order phases and an
emergent quantum spin liquid phase. The nature of the spin liquid is
identified as a bosonic non-Abelian Moore-Read state from the fingerprint of
the entanglement spectra and identification of a full set of topological
sectors. We further unveil a stripe magnetic order coexisting with this spin
liquid. Our results not only establish a rare example of non-Abelian spin
liquids in simple spin systems but also demonstrate the coexistence of
fractionalized excitations and magnetic order beyond mean-field descriptions.

##  I Introduction

Quantum spin liquids (QSLs) are novel quantum states with long-range
entanglement and emergent fractionalized excitations, which can avoid forming
conventional magnetic order due to geometric frustration and quantum
fluctuation Anderson (1973); Savary and Balents (2017); Zhou  _et al._ (2017);
Balents (2010); Broholm  _et al._ (2020). The prominent realizations of QSLs
are the chiral spin liquids (CSLs) Kalmeyer and Laughlin (1987), which break
time-reversal symmetry and have been established in some spin-$1/2$ systems
Bauer  _et al._ (2014); He  _et al._ (2014); Gong  _et al._ (2014); Messio
_et al._ (2012); Wietek  _et al._ (2015); He and Chen (2015); Hu  _et al._
(2015); Gong  _et al._ (2015); Kumar  _et al._ (2014); Hickey  _et al._
(2016); Szasz  _et al._ (2020); Chen  _et al._ (2021). These CSLs have
fractional excitations following the Abelian anyon statistics, which are spin
analog of the Laughlin state and bridge QSLs and the fractional quantum Hall
effect Wen  _et al._ (1989); Wen (1991, 1990). More interestingly, exactly
soluble models host a new class of CSLs with non-Abelian quasi-particles
Kitaev (2006); Yao and Kivelson (2007) that have the potential to perform
topological quantum computation Nayak  _et al._ (2008) and have stimulated an
extensive search for CSLs in Kitaev materials with strong spin-orbit couplings
Hermanns  _et al._ (2018). Another natural system to search for non-Abelian
CSLs is the frustrated spin-1 model Greiter and Thomale (2009); Scharfenberger
_et al._ (2011); Greiter  _et al._ (2014); Glasser  _et al._ (2015); Meng  _et
al._ (2015), which can be realized in both magnetic compounds Nakatsuji  _et
al._ (2005); Cheng  _et al._ (2011) and cold atom systems Gorshkov  _et al._
(2010). However, unambiguous identification of non-Abelian CSLs in frustrated
spin-1 systems is still rare so far Chen  _et al._ (2018).

While competing interactions and quantum fluctuations play important roles in
forming a QSL, the interplay of different orders in the emergence of a QSL
remains less understood. An interesting possibility is that a QSL coexists
with a conventional order. So far, theoretical understandings of such an
exotic coexistence have mainly been based on mean-field analyses Liu  _et al._
(2010); Savary and Balents (2012); Sedrakyan  _et al._ (2015); Chern and Kim
(2019); Samajdar  _et al._ (2019), and no example has been found with
numerical calculation. In some recent experiments, evidence of fractionalized
excitations was found in materials with magnetically ordered ground states,
such as the scattering continuum in the pyrochlore magnet Yb2Ti2O7 Thompson
_et al._ (2017), which demands further theoretical understanding of the
possible coexistence of QSL and magnetic order.

Figure 1: Model Hamiltonian and phase diagram. (a) The model and (b) classical
phase diagram are shown on the left; the dashed lines in (a) indicate the
periodic boundary conditions of the cylinder geometry. (c) depicts the quantum
phase diagram of the spin-1 $J_{1}$-$J_{2}$-$J_{\chi}$ Heisenberg model with
$0.4<J_{2}<0.6$ and $0<J_{\chi}<0.5$. The stripe order is suppressed in the
shaded area, where the topological order remains robust. The magnetic moments
$\langle S^{z}\rangle$ are shown as a function of the distance $x$ away from
the pinning field $h_{z}=0.1$ at the boundary for (d) Néel, (e) stripe, (f)
CSL+stripe, and (g) chiral spin state. Smaller $h_{z}$ were also tested and
give similar $\langle S^{z}\rangle$ in the bulk (see Appendix A). The insets
show the spin structure factors in the absence of a pinning field, which is
defined as
$S(\mathbf{k})=\frac{1}{N}\sum_{i,j}\left\langle\mathbf{S}_{i}\cdot\mathbf{S}_{j}\right\rangle
e^{i\mathbf{k}\cdot(\mathbf{r}_{i}-\mathbf{r}_{j})}$. The results in (d)-(g)
are obtained with $L_{y}=8$.

In this work, we study the roles of competing interactions in driving a QSL in
a spin-1 square-lattice antiferromagnetic system with nearest-neighbor (NN)
$J_{1}$ and next-nearest-neighbor (NNN) $J_{2}$ interactions and a three-spin
scalar chiral coupling $J_{\chi}$. By means of an unbiased density matrix
renormalization group (DMRG) approach, we map out a global phase diagram of
the model, including a Néel magnetic state, a stripe magnetic state, a
noncoplanar chiral spin state (CSS), and a CSL state surrounded by these
conventional magnetic phases. We unambiguously identify this CSL as the non-
Abelian Moore-Read state from the fingerprint of entanglement spectra (ESs)
and the identification of the full topological degeneracy. Furthermore, we
determine a stripe magnetic order in this CSL phase, showing a coexistence of
QSL and magnetic order. Finally, the nature of the quantum phase transitions
from the CSL to the neighbor phases is also addressed. Our results show the
presence of a robust non-Abelian CSL coexisting with the stripe magnetic
order, and demonstrate a rare example of simple spin-1 systems resulting from
the interplay between competing interactions and quantum fluctuations.

The rest of the paper is organized as follows: In Sec. II, we introduce the
$J_{1}$-$J_{2}$-$J_{\chi}$ model on a square lattice and describe the
numerical method used in this work. In Sec. III, we summarize our main
findings in the phase diagram including various magnetic order phases and an
emergent CSL phase. We further show numerical evidence of the magnetic order
phases in Sec. IV. In Sec.V, we identify the nature of the spin liquid as the
non-Abelian CSL from the fingerprint of the entanglement spectra and the full
set of topologically degenerate ground states. We also show a coexisting
stripe order in the CSL phase. The nature of the phase transitions is explored
in Sec. VI. Sec. VII contains a discussion and summary.

##  II Model and method

The spin-1 $J_{1}$-$J_{2}$-$J_{\chi}$ Heisenberg model on the square lattice
is defined as

$\displaystyle H$ $\displaystyle=$ $\displaystyle
J_{1}\sum\limits_{\left\langle
i,j\right\rangle}\mathbf{S}_{i}\cdot\mathbf{S}_{j}+J_{2}\sum\limits_{\left\langle\left\langle
i,j\right\rangle\right\rangle}\mathbf{S}_{i}\cdot\mathbf{S}_{j}$ (1)
$\displaystyle+$ $\displaystyle
J_{\chi}\sum\limits_{i,j,k\in\triangle}\mathbf{S}_{i}\cdot(\mathbf{S}_{j}\times\mathbf{S}_{k}),$

where $\left\langle i,j\right\rangle$ and $\left\langle\left\langle
i,j\right\rangle\right\rangle$ refer to the NN and NNN sites.
$\left\\{i,j,k\right\\}$ in the summation $\sum_{\Delta}$ refers to the three
neighboring sites of the smallest triangle taken clockwise [see Fig. 1(a)].
Here we take $J_{1}=1$ as the energy unit.

We study the system using both the finite and infinite DMRG methods White
(1992, 1993); Schollwöck (2011) with $U(1)$ spin symmetry on the cylinder
geometry with circumference $L_{y}=4-10$ lattice sites ITe ; Fishman  _et al._
(2020); Hauschild and Pollmann (2018). To determine the magnetic order, we
directly compute the magnetic moments and also check the spin structure factor
using the DMRG calculation with spin $SU(2)$ symmetry McCulloch and Gulácsi
(2002), which gives consistent results. In the infinite DMRG simulation, two
or four columns are chosen as the unit cell to accommodate the magnetic
orders. We keep up to M=10000 ($4000$) $U(1)$ [$SU(2)$] bond dimensions with a
typical truncation error $\epsilon\sim 10^{-5}$. More benchmark results and
details are shown in the Appendix A.

##  III Phase diagram

Through DMRG calculations, we establish a quantum phase diagram as shown in
Fig. 1(c), which has three magnetic ordered phases. For $J_{\chi}=0$, we find
a direct transition from the Néel to the stripe state at $J_{2}\approx 0.546$
Haghshenas  _et al._ (2018). By turning on $J_{\chi}$, the $(\pi,\pi)$ Néel
state and $(0,\pi)$ stripe state smoothly expand to finite-$J_{\chi}$ regime.
For large $J_{\chi}$, we find a CSS with a large chiral order and the spin
correlations consistent with the corresponding classical state Rabson and
Trugman (1995). In the intermediate-$J_{\chi}$ regime sitting among these
magnetic phases, we identify a CSL with the non-Abelian Pfaffian-type
topological order Moore and Read (1991); Greiter  _et al._ (1991) by the
quasi-degenerate patterns in the ES and three topologically degenerate ground
states. We also find that this topological state coexists with a stripe
magnetic order, which we label as the CSL+stripe state.

For comparison, we depict a classical phase diagram of this model in Fig.
1(b), showing that the CSL emerges roughly around the transition lines between
these magnetic order phases. This picture agrees with the guiding principle
for searching for CSLs in spin-$1/2$ models Hu  _et al._ (2015); Wietek  _et
al._ (2015); Gong  _et al._ (2015, 2017a); Huang  _et al._ (2021); Hickey  _et
al._ (2017), suggesting that strong magnetic fluctuation still plays an
important role in driving the CSLs even though their topological natures are
different.

Figure 2:  Spin correlation function
$\left\langle\mathbf{S}_{i}\cdot\mathbf{S}_{j}\right\rangle$ at
$J_{2}=0.5,J_{\chi}=0.45$ in the CSS. (a) The black open circle is the
reference site $i$, and all the spin correlations for the even reference sites
have the same correlations. The blue solid (red shaded) circles represent
positive (negative) spin correlation, and the radius represents the magnitude.
(b) Spin correlations with the same parameters for the odd reference sites.
The results are obtained with $L_{y}=8$.

##  IV Magnetic orders

We determine the magnetic orders using two methods, which give the same
conclusions. First, we directly measure the magnetic moments $\langle
S^{z}\rangle$ in the systems by applying a magnetic pinning field $h_{z}$ at
the boundary of the cylinder White and Chernyshev (2007). Second, we compute
the spin structure factor without magnetic pinning fields using the $SU(2)$
DMRG and identify the magnetic order by the sharp Bragg peaks. With the help
of a small pinning field, as shown in Figs. 1(d) and 1(e), $\langle
S^{z}\rangle$ have large magnitudes in the bulk of the cylinder, with
staggered and uniform signs for the Néel and stripe ordered states,
respectively. In the insets, the spin structure factors also show the strong
characteristic peaks supporting the magnetic orderings. For the CSL+stripe
state in Fig. 1(f), $\langle S^{z}\rangle$ is also found to saturate at a
finite value and the spin structure factor has the ($0,\pi$) peak
characterizing a stripe order.

In the CSS shown in Fig. 1(g) with boundary pinning fields, $\langle
S^{z}\rangle$ are large on every other site, agreeing with the classical spin
configuration where the magnetic moments at other sites are lying down in the
$xy$ plane Rabson and Trugman (1995) (also see Appendix C). The noncoplanar
magnetic order of the state is confirmed by a “multi-$Q$” feature in the spin
structure factor, with two peaks at $(0,\pi)$ and $(\pi,0)$ momenta and two
satellite peaks at $(\pm\pi/2,\pm\pi/2)$, as shown in the inset in Fig. 1(g).
Alternatively, this feature can be viewed from the spin correlations in real
space. As shown in Fig. 2(a) for the given reference site, the long-ranged
spin correlations have a period of $4$ on the even sites, which corresponds to
the $(\frac{\pi}{2},\frac{\pi}{2})$ peak in the spin structure factor. The
spin correlations on the odd sites shown in Fig. 2(b) have a period of $2$,
which corresponds to the $(\pi,0)$ and $(0,\pi)$ peaks.

Figure 3: Entanglement spectra of (a) the vacuum sector, (b) Ising anyon
sector, and (c) fermion sector at $J_{2}=0.45,J_{\chi}=0.3$. The ESs in (a)
and (c) are obtained with $L_{y}=8$, and the ES in (b) are obtained with
$L_{y}=7$. $\lambda_{i}$ refers to the eigenvalues of the reduced density
matrix of the half system, and $\Delta k_{y}=k_{y}-k_{y}^{0}$ is the relative
momentum that has an increasement of $\frac{2\pi}{L_{y}}$ ($k_{y}^{0}$ is the
momentum of the state with the largest eigenvalue in each sector). Each
$S_{z}$ sector is separated using the conservation of total $S_{z}$. For a
given $S_{z}$ and momentum, the quasi-degenerate eigenvalues are defined as
the lower levels that are separated by the entanglement gap, which are labeled
by the number below each momentum.

##  V Coexisting non-Abelian CSL and stripe order

The ES extracted from the ground state provides a powerful tool to identify
topological CSLs as it has a one-to-one correspondence to the edge spectra Li
and Haldane (2008). In the intermediate-$J_{\chi}$ regime, we find three
nearly degenerate ground states using infinite DMRG simulation with randomly
initialized states Cincio and Vidal (2013). As shown in Fig. 3, the quasi-
degenerate patterns in the ESs are consistent with the tower of states of all
three sectors described by the non-Abelian $SU(2)_{2}$ Wess-Zumino-Witten
theory Liu  _et al._ (2012a); Zhu  _et al._ (2015), which are identified as
the vacuum sector, the Ising anyon sector, and the fermion sector. In Fig.
3(a), the leading ES of the vacuum sector shows a quasi-degenerate pattern of
{1,1,3,5,10,16,…} in the $S_{z}=0$ sector and {1,2,4,7,13,…} in the $S_{z}=\pm
1$ sectors (the higher degenerate levels are not observed due to the limited
momentum numbers in finite-size systems). The leading ES of the Ising anyon
sector in Fig. 3(b) shows the same pattern, {1,2,4,8,…}, for each given
$S_{z}$ sector. In particular, the ES of the vacuum sector has a symmetry
about $S_{z}=0$, while it is symmetric to $S_{z}=1/2$ in the Ising anyon
sector, corresponding to a spin-$1/2$ quasiparticle created at the open edge.
As shown in Fig. 3(c), the leading ES of the fermion sector has degenerate
patterns similar to those in the vacuum sector except for the symmetry about
$S_{z}=-1$, which indicates the created spin-1 excitation at the edge of the
cylinder.

Here we stress that the selection of each topological sector of this non-
Abelian CSL is well controlled. First, we find that the two Abelian sectors
are more stable on an even-width cylinder and the Ising anyon sector is
energetically favored on an odd-width cylinder. This observation can be
understood based on the generalized Pauli principle in the thin-torus limit of
the bosonic Moore-Read Pfaffian state, i.e. no more than two particles in two
consecutive orbitals Bernevig and Haldane (2008); Liu  _et al._ (2012a); Zhu
_et al._ (2015). Thus, the Abelian vacuum and fermion sectors, related to the
patterns $[20]$ and $[02]$ (period of $2$ orbitals), are favorable under even-
width cylinders, while the non-Abelian Ising anyon sector corresponding to
$[11]$ (period of $1$ orbital) survives on odd-width cylinders. Alternatively,
the even-odd effect can also be derived based on the boundary condition of
Majorana fermions: the odd-width cylinders leading to the anti-periodic
boundary condition guarantee the existence of a Majorana zero mode Das Sarma
_et al._ (2015); Liu  _et al._ (2012b). Second, the two Abelian sectors can be
smoothly connected by adiabatically inserting flux in the cylinder. The flux
$\theta$ adds a phase factor to the spin-flip terms
$S_{i}^{+}S_{j}^{-}\rightarrow e^{i\theta}S_{i}^{+}S_{j}^{-}$ for
$j\rightarrow i$ across the boundary from the top and their Hermitian
conjugate terms. As shown in Fig. 4(a), the ES of the vacuum sector at zero
flux ($\theta=0$) is symmetric to $S_{z}=0$. With growing $\theta$, the ES
adiabatically evolves into the fermion sector at $\theta=2\pi$ with a symmetry
about $S_{z}=-1$, which indicates a net spin $\Delta S=1$ transferred from one
edge of the cylinder to the other one. Furthermore, the ES evolves back to the
vacuum sector with a symmetry about $S_{z}=-2$ at $\theta=4\pi$, indicating
the total transferred spin $\Delta S=2$. On the other hand, the Ising anyon
sector evolves to itself by threading one flux quantum, as shown in Fig. 4(b).
The evolution of different sectors with flux insertion directly shows the
gapless feature of the edge states Qi  _et al._ (2012); Avron  _et al._
(2003). Furthermore, we show the robustness of the vacuum sector topological
ES in the CSL+stripe phase with fixed $J_{2}=0.5$ and varying $J_{\chi}$. As
seen in Figs. 5(a) and 5(b), the quasidegenerate patterns remain robust in the
momentum sectors associated with the lowest four groups of entanglement
spectra levels.

Figure 4: ES flow with adiabatically inserted flux $\theta$ starting from (a)
the vacuum sector and (b) the Ising anyon sector at $J_{2}=0.45,J_{\chi}=0.3$.
The eigenvalues in different $S_{z}$ sectors are labeled by different symbols.

To establish the coexisting stripe order, we also study the finite-size
scaling of magnetic order parameters defined as
$m^{2}(\mathbf{k})=\frac{1}{N^{2}}\sum_{i,j}\left\langle\mathbf{S}_{i}\cdot\mathbf{S}_{j}\right\rangle
e^{i\mathbf{k}\cdot(\mathbf{r}_{i}-\mathbf{r}_{j})}$, where $i$ and $j$ are
summed over the middle $N=L_{y}\times L_{y}$ sites. Figure 6(a) shows the
stripe order parameters at $(0,\pi)$, which are obtained with the $SU(2)$ DMRG
with extrapolation to infinite bond dimensions (see more details in Appendix
A). The finite $m^{2}(0,\pi)$ in the two-dimensional (2D) limit pins down the
stripe order in the CSL+stripe regime.

Figure 5:  The entanglement spectra for various $J_{\chi}$ at $J_{2}=0.5$,
obtained with $L_{y}=8$ in the vacuum sector. (a) and (b) are the lowest and
second-lowest parts of the entanglement spectra, respectively. A minor shift
in $\Delta k_{y}$ is applied for different $J_{\chi}$ in order to view the
quasi-degenerate eigenvalues, which are separated by the entanglement gap
indicated by the dashed line.

##  VI Phase transition

We first detect phase transitions using magnetic order parameters. We show
$m^{2}(\pi,\pi)$ and $m^{2}(0,\pi)$ from the Néel to the CSL+stripe phase at
$J_{2}=0.5$ in Figs. 6(a) and 6(b). After finite-size scaling, the Néel order
$m^{2}(\pi,\pi)$ vanishes at $J_{\chi}\approx 0.21$ and the stripe order
$m^{2}(0,\pi)$ starts to develop at the same $J_{\chi}$, indicating a direct
transition between the two phases. As shown in Fig. 6(c), the chiral order,
defined as
$\langle\chi\rangle=\frac{1}{4N}\sum\limits_{i,j,k\in\triangle}\langle\mathbf{S}_{i}\cdot(\mathbf{S}_{j}\times\mathbf{S}_{k})\rangle$,
also rises quickly at $J_{\chi}\approx 0.21$ with a peak in its first-order
derivative of $J_{\chi}$, suggesting a continuous transition to the
CSL+stripe. The same transition can also be found by varying $J_{2}$, as shown
in Fig. 6(d) for $J_{\chi}=0.25$. However, the nature of the transition around
$J_{2}=0.45,J_{\chi}=0.3$ becomes more complicated. We identify possible two-
step transitions as the stripe order disappears first inside the shaded region
while the topological order remains robust and another transition to the Néel
state happens at lower $J_{\chi}$. We leave this intriguing possibility to
future study.

Figure 6: Magnetic order parameters $m^{2}(\mathbf{k})$ at (a) $(0,\pi)$ and
(b) $(\pi,\pi)$ are shown for various $J_{\chi}$ not  at $J_{2}=0.5$. The
lines in (a) are polynomial fittings up to the second order of $1/N$. In (b)
we find a more accurate fitting curve up to the second order of $1/\sqrt{N}$
close to the transition point. (c) is the chiral order $\langle\chi\rangle$ at
$J_{2}=0.5$ for various $J_{\chi}$. The states from left to right are Néel,
CSL+stripe, and CSS. The dashed lines are determined by the peaks in the
derivative of $\langle\chi\rangle$, as shown in the inset. (d) is the chiral
order $\langle\chi\rangle$ at $J_{\chi}=0.25$ for various $J_{2}$, showing the
Néel, CSL+stripe, and stripe state from left to right. The left dashed line is
determined by the peak in the derivative of $\langle\chi\rangle$ and the right
dashed line is determined by the vanishing of quasi-degenerate patterns in the
ESs (see Appendix B). The results in (c) and (d) are obtained with $L_{y}=8$.

With further increasing $J_{\chi}$ at $J_{2}=0.5$,
$m^{2}(\frac{\pi}{2},\frac{\pi}{2})$ shows a sudden rise at $J_{\chi}\approx
0.4$ (see details in Appendix C), and the chiral order has a sharp jump, as
shown in Fig. 6(c), indicating a first-order transition from the CSL+stripe
state to the CSS. To determine the boundary between the CSL+stripe and stripe
states, we identify the melting of the low-lying quasi-degenerate patterns in
the ESs as a probe of vanished topological order Sup , which determines a
topological quantum phase transition.

##  VII Summary and discussion

We have identified a non-Abelian Pfaffian-type CSL with coexisting stripe
magnetic order in an extended spin-1 quantum antiferromagnet on the square
lattice using the unbiased DMRG calculation. We established the topological
nature using the characteristic entanglement spectra and three-fold
topological degeneracy. Notice that the previously found CSLs in spin-$1/2$
systems are all non magnetic Bauer  _et al._ (2014); He  _et al._ (2014); Gong
_et al._ (2014); Messio  _et al._ (2012); Hu  _et al._ (2015); Kumar  _et al._
(2014); Wietek  _et al._ (2015); Gong  _et al._ (2015); He and Chen (2015),
which indicates that the reduced quantum fluctuations in spin-1 system may
leave a room for promoting the coexistence ofmagnetic order with a QSL. Our
findings not only demonstrate an explicit example with the coexistence of
fractionalized excitations and magnetic order in frustrated quantum magnets,
but also inspire a new search for exotic phases in other interesting systems,
including the moiré superlattices, where effective spin models with $SU(4)$
symmetry may lead to new QSLs Wu  _et al._ (2018); Regan  _et al._ (2020);
Tang  _et al._ (2020).

Finally, we give some remarks on the possible experimental realization of the
system. Recent studies on FeSe have shown a paramagnetic parent state with
both Néel and stripe spin fluctuations Wang  _et al._ (2016a) and a pressure-
induced stripe magnetic phase Wang  _et al._ (2016b), which can be considered
an effective spin-1 system Gong  _et al._ (2017b). The chiral interaction can
be induced by the orbital coupling of applied magnetic field to the underlying
electrons Sen and Chitra (1995). For conventional spin systems with time-
reversal symmetry, a dynamical spin chiral term can also be generated by the
circularly polarized light Claassen  _et al._ (2017).

###### Acknowledgements.

W.Z. thanks X. Y. Dong, Z. X. Liu and X. L. Wang for simulating discussions.
This work was supported by the U.S. Department of Energy, Office of Science,
Advanced Scientific Computing Research and Basic Energy Sciences, Materials
Sciences and Engineering Division, Scientific Discovery through Advanced
Computing (SciDAC) program under the grant number DE-AC02-76SF00515 (Y.H.,
H.C.J., D.N.S.). W.Z. was supported by National Science Foundation of China
under project number 92165102. S.S.G. was supported by the NSFC grants No.
11874078 and 11834014.

##  Appendix A Convergence of the numerical results

Figure S1:  DMRG bond dimension dependence of (a) the ground-state energy per
site, (b) the entanglement entropy, (c) the correlation length, and (d) the
entanglement spectra. The finite DMRG results are calculated with
$L_{x}=24,L_{y}=8$, and the infinite DMRG results are calculated with
$L_{y}=8$. The ground-state energy is averaged over the half of the sites in
the middle for the finite DMRG results in order to minimize the boundary
effect. All the results are obtained at $J_{2}=0.45,J_{\chi}=0.3$. Figure S2:
Details of the DMRG calculation to determine magnetic order. (a) is the
$SU(2)$ bond dimension dependence of the stripe order parameter at
$J_{2}=0.5,J_{\chi}=0.3$. For $L_{y}=4$ and $6$ the order parameter remains
almost unchanged with increasing bond dimension $M$, so the extrapolations are
not shown here. (b) is the magnetic moment $\langle S^{z}\rangle$ with
distance away from the edge for various pinning fields $h_{z}$ at
$J_{2}=0.5,J_{\chi}=0.32$, obtained with $L_{y}=8$.

In order to check the numerical convergence, we obtain the density matrix
renormalization group (DMRG) results by increasing the bond dimension $M$. As
an example, we show the results for $J_{2}=0.45,J_{\chi}=0.3$ in Fig. S1,
including the ground-state energy, entanglement entropy, and entanglement
spectra, which all change slightly with increasing bond dimension and thus
demonstrate the good convergence of our DMRG results. Also, we can find highly
consistent results from the finite and infinite DMRG simulations in Figs.
S1(a) and S1(b). Furthermore, we compute the correlation length to estimate
the finite-size effect in our results. The correlation length is defined as
$1/\ln(\epsilon_{1}/\epsilon_{2})$, where $\epsilon_{1}$ ($\epsilon_{2}$) is
the largest (second largest) eigenvalue of the transfer matrix. In Fig. S1(c)
we show that the correlation length can be extrapolated to around $1.2$ in the
limit of infinite bond dimensions, which is much smaller than the cylinder
widths in our calculation. Thus, our DMRG results, such as the entanglement
spectra, can provide reliable and accurate approximations of the physics in
the 2D thermodynamic limit.

###  A.1 Details of determining the magnetic order

The stripe order parameter $m^{2}(0,\pi)$ defined in the main text has been
shown to be extrapolated to a finite value in the chiral spin liquid (CSL) +
stripe state. We calculate $m^{2}(0,\pi)$ by keeping different $SU(2)$ bond
dimensions $M$ for $L_{y}=8,10$. As shown in Fig. S2(a), the order parameters
$m^{2}(0,\pi)$ slightly decay with increasing $M$ and remain finite in the
infinite-$M$ limit. We use the extrapolated data in the main text.

In the calculation of magnetic moments $\langle S^{z}\rangle$, we have also
tested the results by using various pinning fields $h_{z}$. As shown in Fig.
S2(b), $\langle S^{z}\rangle$ remain almost the same in the bulk of the system
for different values of $h_{z}$, indicating the robust magnetic moments.

Figure S3:  The entanglement spectra at $J_{2}=0.5,J_{\chi}=0.3$ for (a)
$L_{y}=6$, (b) $L_{y}=8$, and (c) $L_{y}=10$ in the CSL+stripe regime. Only
the vacuum sector is shown. Figure S4:  The entanglement spectra at
$J_{2}=0.42,J_{\chi}=0.34$ for (a) $L_{y}=6$, (b) $L_{y}=8$, and (c)
$L_{y}=10$ in the shaded regime. Only the vacuum sector is shown. Figure S5:
The entanglement spectra at (a) $J_{2}=0.5,J_{\chi}=0.1$ (Néel state), (b)
$J_{2}=0.5,J_{\chi}=0.45$ (CSS), and (c) $J_{2}=0.58,J_{\chi}=0.2$ (stripe
state) obtained with $L_{y}=8$. There is no counting of the quasi-degenerate
eigenvalues.

##  Appendix B Entanglement spectra

Figure S6:  The entanglement spectra at $J_{\chi}=0.3$ for various $J_{2}$,
obtained with $L_{y}=8$. The red dashed lines are guides to the eye.

In Figs. S3 and S4, we show the quasi-degenerate patterns of the entanglement
spectra for more parameter points in both the CSL+stripe regime and the shaded
regime. We can find robust quasidegenerate patterns for various lattice sizes,
which characterize the topological nature of the non-Abelian CSL. In
comparison, the entanglement spectra have no such quasi-degenerate pattern in
the Néel, stripe, and chiral spin states as shown in Fig. S5.

The quasidegenerate patterns vanish as $J_{2}$ increases from $0.5$ in the
intermediate $J_{\chi}$, which indicates a phase transition from the
CSL+stripe to stripe state. As shown in Fig. S6(a), the quasidegenerate
eigenvalues in the CSL+stripe state are separated by a relatively large
entanglement gap in every spin sector. In Fig. S6(b) we can still identify the
quasi-degenerate eigenvalues in the $S_{z}=0$ sector near the phase boundary,
but additional low-lying eigenvalues have already mixed in the $S_{z}=-1$ and
$1$ sectors. When the system enters the stripe state, the quasidegenerate
patterns disappear, as shown in Fig. S6(c).

###  B.1 Spin gap in the CSL+stripe regime

Figure S7:  Finite-size scaling of the spin gap in the CSL+stripe phase. (a)
System length ($L_{x}$) dependence of the spin gap for $L_{y}=6$. (b) System
circumference ($L_{y}$) dependence of the spin gap obtained for cylinders with
$L_{x}=3\times L_{y}$.

The spin gap can be obtained from the difference between the lowest energies
of the total $S_{z}=0$ and $S_{z}=1$ sectors. To avoid the edge excitations,
we compute the spin excitations in the bulk. Here we compare two parameters in
the CSL+stripe regime with the same $J_{\chi}=0.3$. First, we study the system
length ($L_{x}$) dependence of the spin gap for the given system circumference
($L_{y}$). As shown in Fig. S7(a) for $L_{y}=6$, the gaps are almost
independent of the system length $L_{x}$. By examining the spin correlations
of the excited state, we can find that the excitations are mainly contributed
by the spin flip along the circumference direction since the ground state has
the $(0,\pi)$ order configuration, which is also consistent with our DMRG
observation in Fig. S7(a).

Next, we study the scaling of the gap with $L_{y}$, as shown in Fig. S7(b).
The gap near the boundary between the Néel and CSL+stripe state ($J_{2}=0.45$)
is relatively large, because the magnetic order is weak and the gapless
Goldstone mode may not be fully developed yet. In the center of the CSL+stripe
state ($J_{2}=0.5$), the spin gap becomes smaller due to the developed stripe
order and the associated gapless Goldstone mode. With growing system
circumference, the spin gap decreases rapidly and may be appropriately scaled
to vanishing, which indicates that the spin gap in the CSL+stripe state
becomes smaller for larger systems and thus is consistent with gapless spin
excitations in the CSL+stripe state.

##  Appendix C Numerical results for the chiral spin state

###  C.1 Classical spin configurations

Figure S8:  The spin configurations of the classical CSS using the classical
Monte Carlo simulations on a $10\times 10$ lattice, viewed from the top (the
top panel) and side (the bottom panel).

The spin correlations in the CSS have configurations similar to the classical
CSS, which is shown in Fig. S8. While the spin pointing in the $xy$ plane has
a period of $4$, the spin pointing in the $z$ direction has a period of $2$.

###  C.2 Phase transitions

Figure S9:  Quantum phase transition from the CSL+stripe state to the CSS. (a)
and (c) are different magnetic orders. (b) and (d) are the entanglement
entropy versus $J_{\chi}$ near the phase boundary. (a) and (b) are obtained
under the normal periodic boundary condition while (c) and (d) are obtained
with the shifted boundary condition in order to be compatible with both
magnetic orders. (b) shows the entanglement entropy in a larger $J_{\chi}$
regime, where the phase transitions from the Néel state to the CSL+stripe
state and from CSL+stripe state to the CSS are indicated by the two dashed
lines. The left and right dashed lines are determined by the local minimum and
maximum, respectively. All the results are obtained at $J_{2}=0.5$.

The phase transition from the CSL+stripe state to the CSS can be determined by
the emergence of a finite $m^{2}(\frac{\pi}{2},\frac{\pi}{2})$ order, because
the two phases share the same structure factor peak at $(0,\pi)$. For
$L_{y}=6,10$ we use the shifted boundary condition to be compatible with both
phases. As a result, $k_{y}$ is no longer a conserved quantity and the lattice
is rearranged into a two-leg ladder. Each ladder is constructed by connecting
each pair of columns in the $y$ direction, and the modified order parameter
becomes
$\widetilde{m}^{2}(k)=\frac{1}{N^{2}}\sum_{i,j,Z}\left\langle\mathbf{S}_{i,Z}\cdot\mathbf{S}_{j,Z}\right\rangle
e^{ik(r_{i,Z}-r_{j,Z})}$, where $Z=0,1$ is the leg index. As shown in Fig.
S9(a), $m^{2}(\frac{\pi}{2},\frac{\pi}{2})$ shows a sudden increase at
$J_{\chi}\approx 0.4$ for $L_{y}=8$ [the finite
$m^{2}(\frac{\pi}{2},\frac{\pi}{2})$ below $J_{\chi}=0.4$ results from the
self-correlation contribution of
$\langle\mathbf{S}_{i}\cdot\mathbf{S}_{i}\rangle=2$ and thus is not a peak].
In Fig. S9(c) $\widetilde{m}^{2}(\frac{\pi}{2})$ also shows a sudden jump
around the same $J_{\chi}$ for $L_{y}=6,10$, indicating a first-order
transition that is consistent for different lattice sizes.

In addition, we obtain the entanglement entropy near the phase transitions. A
maximum value of the entanglement entropy can be found near the CSL+stripe to
CSS phase transition for various $L_{y}$. As shown in Figs. S9(b) and S9(d),
the entanglement entropy reaches its maximum at $J_{\chi}\approx 0.39$ for
$L_{y}=6,8,10$, which is consistent with the phase transition.

## References

  * Anderson (1973) P. W. Anderson, Materials Research Bulletin 8, 153 (1973).
  * Savary and Balents (2017) L. Savary and L. Balents, Rep. Prog. Phys. 80, 016502 (2017), arXiv:1601.03742 .
  * Zhou  _et al._ (2017) Y. Zhou, K. Kanoda, and T.-K. Ng, Reviews of Modern Physics 89, 025003 (2017), arXiv:1607.03228 [cond-mat.str-el] .
  * Balents (2010) L. Balents, Nature 94, 199 (2010).
  * Broholm  _et al._ (2020) C. Broholm, R. J. Cava, S. A. Kivelson, D. G. Nocera, M. R. Norman, and T. Senthil, Science 367 (2020), 10.1126/science.aay0668.
  * Kalmeyer and Laughlin (1987) V. Kalmeyer and R. B. Laughlin, Phys. Rev. Lett. 59, 2095 (1987).
  * Bauer  _et al._ (2014) B. Bauer, L. Cincio, B. P. Keller, M. Dolfi, G. Vidal, S. Trebst, and A. W. Ludwig, Nature communications 5, 5137 (2014).
  * He  _et al._ (2014) Y.-C. He, D. N. Sheng, and Y. Chen, Phys. Rev. Lett. 112, 137202 (2014).
  * Gong  _et al._ (2014) S.-S. Gong, W. Zhu, and D. N. Sheng, Scientific reports 4, 6317 (2014).
  * Messio  _et al._ (2012) L. Messio, B. Bernu, and C. Lhuillier, Phys. Rev. Lett. 108, 207204 (2012).
  * Wietek  _et al._ (2015) A. Wietek, A. Sterdyniak, and A. M. Läuchli, Phys. Rev. B 92, 125122 (2015).
  * He and Chen (2015) Y.-C. He and Y. Chen, Phys. Rev. Lett. 114, 037201 (2015).
  * Hu  _et al._ (2015) W.-J. Hu, W. Zhu, Y. Zhang, S. Gong, F. Becca, and D. N. Sheng, Phys. Rev. B 91, 041124(R) (2015).
  * Gong  _et al._ (2015) S.-S. Gong, W. Zhu, L. Balents, and D. N. Sheng, Phys. Rev. B 91, 075112 (2015).
  * Kumar  _et al._ (2014) K. Kumar, K. Sun, and E. Fradkin, Phys. Rev. B 90, 174409 (2014).
  * Hickey  _et al._ (2016) C. Hickey, L. Cincio, Z. Papić, and A. Paramekanti, Phys. Rev. Lett. 116, 137202 (2016).
  * Szasz  _et al._ (2020) A. Szasz, J. Motruk, M. P. Zaletel, and J. E. Moore, Phys. Rev. X 10, 021042 (2020).
  * Chen  _et al._ (2021) B.-B. Chen, Z. Chen, S.-S. Gong, D. N. Sheng, W. Li, and A. Weichselbaum, arXiv e-prints , arXiv:2102.05560 (2021), arXiv:2102.05560 [cond-mat.str-el] .
  * Wen  _et al._ (1989) X. G. Wen, F. Wilczek, and A. Zee, Phys. Rev. B 39, 11413 (1989).
  * Wen (1991) X. G. Wen, Phys. Rev. B 44, 2664 (1991).
  * Wen (1990) X.-G. Wen, International Journal of Modern Physics B 4, 239 (1990).
  * Kitaev (2006) A. Kitaev, Annals of Physics 321, 2 (2006), january Special Issue.
  * Yao and Kivelson (2007) H. Yao and S. A. Kivelson, Phys. Rev. Lett. 99, 247203 (2007).
  * Nayak  _et al._ (2008) C. Nayak, S. H. Simon, A. Stern, M. Freedman, and S. Das Sarma, Rev. Mod. Phys. 80, 1083 (2008).
  * Hermanns  _et al._ (2018) M. Hermanns, I. Kimchi, and J. Knolle, Annual Review of Condensed Matter Physics 9, 17 (2018), https://doi.org/10.1146/annurev-conmatphys-033117-053934 .
  * Greiter and Thomale (2009) M. Greiter and R. Thomale, Phys. Rev. Lett. 102, 207203 (2009).
  * Scharfenberger  _et al._ (2011) B. Scharfenberger, R. Thomale, and M. Greiter, Physical Review B 84, 140404(R) (2011).
  * Greiter  _et al._ (2014) M. Greiter, D. F. Schroeter, and R. Thomale, Phys. Rev. B 89, 165125 (2014).
  * Glasser  _et al._ (2015) I. Glasser, J. I. Cirac, G. Sierra, and A. E. B. Nielsen, New Journal of Physics 17, 082001 (2015).
  * Meng  _et al._ (2015) T. Meng, T. Neupert, M. Greiter, and R. Thomale, Phys. Rev. B 91, 241106(R) (2015).
  * Nakatsuji  _et al._ (2005) S. Nakatsuji, Y. Nambu, H. Tonomura, O. Sakai, S. Jonas, C. Broholm, H. Tsunetsugu, Y. Qiu, and Y. Maeno, Science 309, 1697 (2005).
  * Cheng  _et al._ (2011) J. G. Cheng, G. Li, L. Balicas, J. S. Zhou, J. B. Goodenough, C. Xu, and H. D. Zhou, Phys. Rev. Lett. 107, 197204 (2011).
  * Gorshkov  _et al._ (2010) A. V. Gorshkov, M. Hermele, V. Gurarie, C. Xu, P. S. Julienne, J. Ye, P. Zoller, E. Demler, M. D. Lukin, and A. M. Rey, Nature Physics 6, 289 (2010).
  * Chen  _et al._ (2018) J.-Y. Chen, L. Vanderstraeten, S. Capponi, and D. Poilblanc, Phys. Rev. B 98, 184409 (2018).
  * Liu  _et al._ (2010) Z.-X. Liu, Y. Zhou, and T.-K. Ng, Phys. Rev. B 81, 224417 (2010).
  * Savary and Balents (2012) L. Savary and L. Balents, Phys. Rev. Lett. 108, 037202 (2012).
  * Sedrakyan  _et al._ (2015) T. A. Sedrakyan, L. I. Glazman, and A. Kamenev, Physical review letters 114, 037203 (2015).
  * Chern and Kim (2019) L. E. Chern and Y. B. Kim, Scientific Reports 9, 10974 (2019).
  * Samajdar  _et al._ (2019) R. Samajdar, M. S. Scheurer, S. Chatterjee, H. Guo, C. Xu, and S. Sachdev, Nature Physics 15, 1290 (2019).
  * Thompson  _et al._ (2017) J. D. Thompson, P. A. McClarty, D. Prabhakaran, I. Cabrera, T. Guidi, and R. Coldea, Phys. Rev. Lett. 119, 057203 (2017).
  * White (1992) S. R. White, Physical review letters 69, 2863 (1992).
  * White (1993) S. R. White, Physical review b 48, 10345 (1993).
  * Schollwöck (2011) U. Schollwöck, Annals of physics 326, 96 (2011).
  * (44) Calculations were performed using the ITensor Library at http://itensor.org/ and the TeNPy Library (version 0.4.1).
  * Fishman  _et al._ (2020) M. Fishman, S. R. White, and E. M. Stoudenmire, (2020), arXiv:2007.14822 .
  * Hauschild and Pollmann (2018) J. Hauschild and F. Pollmann, SciPost Phys. Lect. Notes , 5 (2018), code available from https://github.com/tenpy/tenpy, arXiv:1805.00055 .
  * McCulloch and Gulácsi (2002) I. McCulloch and M. Gulácsi, Europhysics Letters 57, 852 (2002).
  * Haghshenas  _et al._ (2018) R. Haghshenas, W.-W. Lan, S.-S. Gong, and D. N. Sheng, Phys. Rev. B 97, 184436 (2018).
  * Rabson and Trugman (1995) D. A. Rabson and S. A. Trugman, Journal of Physics: Condensed Matter 7, 9005 (1995).
  * Moore and Read (1991) G. Moore and N. Read, Nuclear Physics B 360, 362 (1991).
  * Greiter  _et al._ (1991) M. Greiter, X.-G. Wen, and F. Wilczek, Phys. Rev. Lett. 66, 3205 (1991).
  * Gong  _et al._ (2017a) S.-S. Gong, W. Zhu, J.-X. Zhu, D. N. Sheng, and K. Yang, Physical Review B 96, 075116 (2017a).
  * Huang  _et al._ (2021) Y. Huang, X.-Y. Dong, D. N. Sheng, and C. S. Ting, Physical Review B 103, L041108 (2021).
  * Hickey  _et al._ (2017) C. Hickey, L. Cincio, Z. Papić, and A. Paramekanti, Physical Review B 96, 115115 (2017).
  * White and Chernyshev (2007) S. R. White and A. L. Chernyshev, Phys. Rev. Lett. 99, 127004 (2007).
  * Li and Haldane (2008) H. Li and F. D. M. Haldane, Phys. Rev. Lett. 101, 010504 (2008).
  * Cincio and Vidal (2013) L. Cincio and G. Vidal, Phys. Rev. Lett. 110, 067208 (2013).
  * Liu  _et al._ (2012a) Z. Liu, E. J. Bergholtz, H. Fan, and A. M. Läuchli, Phys. Rev. B 85, 045119 (2012a).
  * Zhu  _et al._ (2015) W. Zhu, S. S. Gong, F. D. M. Haldane, and D. N. Sheng, Phys. Rev. B 92, 165106 (2015).
  * Bernevig and Haldane (2008) B. A. Bernevig and F. D. M. Haldane, Phys. Rev. Lett. 100, 246802 (2008).
  * Das Sarma  _et al._ (2015) S. Das Sarma, M. Freedman, and C. Nayak, npj Quantum Information 1, 15001 (2015).
  * Liu  _et al._ (2012b) Z.-X. Liu, Y. Zhou, H.-H. Tu, X.-G. Wen, and T.-K. Ng, Phys. Rev. B 85, 195144 (2012b).
  * Qi  _et al._ (2012) X.-L. Qi, H. Katsura, and A. W. W. Ludwig, Physical review letters 108, 196402 (2012).
  * Avron  _et al._ (2003) J. Avron, D. Osadchy, and R. Seiler, Physics Today 56, 38 (2003).
  * (65) The results at $J_{\chi}=0.3$ are obtained with $SU(2)$ DMRG because $U(1)$ calculations result in finite magnetic moments $<S^{z}>$. Results at other $J_{\chi}$ are obtained with $U(1)$ DMRG.
  * (66) See Supplemental Material at [URL will be inserted by publisher] for detailed numeircal results.
  * Wu  _et al._ (2018) F. Wu, T. Lovorn, E. Tutuc, and A. H. MacDonald, Phys. Rev. Lett. 121, 026402 (2018).
  * Regan  _et al._ (2020) E. C. Regan, D. Wang, C. Jin, M. I. Bakti Utama, B. Gao, X. Wei, S. Zhao, W. Zhao, Z. Zhang, K. Yumigeta, M. Blei, J. D. Carlström, K. Watanabe, T. Taniguchi, S. Tongay, M. Crommie, A. Zettl, and F. Wang, Nature 579, 359 (2020).
  * Tang  _et al._ (2020) Y. Tang, L. Li, T. Li, Y. Xu, S. Liu, K. Barmak, K. Watanabe, T. Taniguchi, A. H. MacDonald, J. Shan, and K. F. Mak, Nature 579, 353 (2020).
  * Wang  _et al._ (2016a) Q. Wang, Y. Shen, B. Pan, X. Zhang, K. Ikeuchi, K. Iida, A. Christianson, H. Walker, D. Adroja, M. Abdel-Hafiez, _et al._ , Nature communications 7, 1 (2016a).
  * Wang  _et al._ (2016b) P. S. Wang, S. S. Sun, Y. Cui, W. H. Song, T. R. Li, R. Yu, H. Lei, and W. Yu, Phys. Rev. Lett. 117, 237001 (2016b).
  * Gong  _et al._ (2017b) S.-S. Gong, W. Zhu, D. N. Sheng, and K. Yang, Physical Review B 95, 205132 (2017b).
  * Sen and Chitra (1995) D. Sen and R. Chitra, Phys. Rev. B 51, 1922 (1995).
  * Claassen  _et al._ (2017) M. Claassen, H.-C. Jiang, B. Moritz, and T. P. Devereaux, Nature communications 8, 1 (2017).


# A Benchmark of Rule-Based and Neural
Coreference Resolution in Dutch Novels and News

Corbèn Poot
CLCG, University of Groningen
<EMAIL_ADDRESS>
Andreas van Cranenburgh
CLCG, University of Groningen
<EMAIL_ADDRESS>

###### Abstract

We evaluate a rule-based (Lee et al., 2013) and neural (Lee et al., 2018)
coreference system on Dutch datasets of two domains: literary novels and
news/Wikipedia text. The results provide insight into the relative strengths
of data-driven and knowledge-driven systems, as well as the influence of
domain, document length, and annotation schemes. The neural system performs
best on news/Wikipedia text, while the rule-based system performs best on
literature. The neural system shows weaknesses with limited training data and
long documents, while the rule-based system is affected by annotation
differences. The code and models used in this paper are available at
https://github.com/andreasvc/crac2020

##  1 Introduction

In recent years, the best results for coreference resolution of English have
been obtained with end-to-end neural models (Lee et al., 2017b, 2018; Joshi et
al., 2019, 2020; Wu et al., 2020). However for Dutch, the existing systems are
still using either a rule-based (van der Goot et al., 2015; van Cranenburgh,
2019) or a machine learning approach (Hendrickx et al., 2008a; De Clercq et
al., 2011). The rule-based system dutchcoref (van Cranenburgh, 2019)
outperformed previous systems on two existing datasets and also presented a
corpus and evaluation of literary novels (RiddleCoref).

In this paper we compare this rule-based system to an end-to-end neural
coreference resolution system: e2e-Dutch. This system is a variant of Lee et
al. (2018) with BERT token representations. We evaluate and compare the
performance of e2e-Dutch to dutchcoref on two different datasets: (1) the
SoNaR-1 corpus (Schuurman et al., 2010), a genre-balanced corpus of 1 million
words, and (2) the RiddleCoref corpus of contemporary novels (van Cranenburgh,
2019). This provides insights into (1) the relative strengths of a neural
system versus a rule-based system for Dutch coreference, and (2) the effect of
domain differences (news/Wikipedia versus literature).

The two datasets we consider vary greatly in terms of overall size and length
of the individual documents; the training subset of RiddleCoref contains only
23 documents (novel fragments) compared to 581 documents for SoNaR-1. However,
the average number of sentences per document is higher for RiddleCoref than
for SoNaR-1 (295.78 vs. 64.28 respectively). We also conduct an error analysis
for both of the systems to examine the types of errors that the systems make.

##  2 Related work

The main differences between traditional and neural approaches can be
summarized as follows:

Rule-based systems are knowledge-intensive; machine learning systems are data-
driven but require feature engineering; end-to-end neural systems only require
sufficient training data and hyperparameter tuning to perform well.

Rule-based and machine learning coreference systems rely on features from
syntactic parses and named-entities provided by an NLP pipeline whereas neural
systems rely on distributed representations; end-to-end systems do not require
any other features.

The rule-based system by Lee et al. (2013) is entity-based and exploits global
features, while end-to-end systems such as Lee et al. (2017b) rank mentions
and make greedy decisions based on local features. Although Lee et al. (2018)
does approximate higher-order inference, their model does not build
representations of entities.

The rest of this section discusses the current best systems for Dutch and
English.

###  2.1 Dutch coreference resolution

The largest dataset available for Dutch coreference resolution is the SoNaR-1
dataset (Schuurman et al., 2010) which consists of 1 million words annotated
for coreference. This corpus was a continuation of the Corea project (Bouma et
al., 2007; Hendrickx et al., 2008a, b). De Clercq et al. (2011) present a
cross-domain coreference resolution study conducted on this corpus. They use a
mention-pair system, which was originally developed with the KNACK-2002 corpus
and then further improved in the Corea project, and observe that the influence
of domain and training size is large, thus underlining the importance of this
large and genre-balanced SoNaR-1 dataset.

The current best coreference resolution system for Dutch is called
“dutchcoref” (van Cranenburgh, 2019) and is based on the rule-based Stanford
system (Lee et al., 2011, 2013). This system improved on the systems in the
SemEval-2010 shared task (Recasens et al., 2010) and a previous implementation
of the Stanford system for Dutch (GroRef; van der Goot et al., 2015). The main
focus of van Cranenburgh (2019) was evaluating coreference on literary texts,
for which a corpus and evaluation is presented. Most coreference resolution
systems are evaluated using newswire texts, but a domain such as literary text
presents its own challenges (Bamman, 2017); for example, novels are longer
than news articles, and novels can therefore contain longer coreference
chains.

###  2.2 English Coreference resolution

System | CoNLL
---|---
Rule-based (Lee et al., 2011) | 58.3
Perceptron (Fernandes et al., 2012) | 58.7
Hybrid: rules + ML (Lee et al., 2017a) | 63.2
Embeddings (Wiseman et al., 2015) | 63.4
\+ RL (Clark and Manning, 2016a) | 65.3
\+ Entity embeddings (Clark and Manning, 2016b) | 65.7

System | CoNLL
---|---
End-to-end (Lee et al., 2017b) | 68.8
Higher-order + CTF + ELMo (Lee et al., 2018) | 73.0
Finetuning BERT base (Joshi et al., 2019) | 73.9
Finetuning BERT large (Joshi et al., 2019) | 76.9
Pretraining SpanBERT (Joshi et al., 2019) | 79.6
SpanBERT + QA (Wu et al., 2020) | 83.1

Table 1: English coreference scores on the OntoNotes CoNLL 2012 shared task
dataset. ML: Machine Learning, RL: Reinforcement Learning, CTF: Coarse-to-
Fine, QA: Question Answering.

The main benchmark for English is the CoNLL 2012 shared task (Pradhan et al.,
2012). Table 1 reports a timeline of results for this task, which shows the
dramatic improvements brought by neural networks, especially the end-to-end
systems on the right. Neural coreference systems improved on previous work but
were still relying on mention detection rules, syntactic parsers, and heavy
feature engineering (Table 1, left). They were outperformed by the first end-
to-end coreference resolution system by Lee et al. (2017b). This system looks
at all the spans (expressions) in a text, up to a maximum length, and then
uses a span-ranking model that decides for each span which previous spans are
good antecedents, if any. The spans themselves are represented by word
embeddings.

Although the models by Clark and Manning (2016a) and Lee et al. (2017b) are
computationally efficient and scalable to long documents, they are heavily
relying on first order models where they are only scoring pairs of mentions.
Because they make independent decisions regarding coreference links, they
might make predictions which are locally consistent but globally inconsistent
(Lee et al., 2018). Lee et al. (2018) introduce an approximation of higher-
order inference, which uses the span-ranking architecture from Lee et al.
(2017b) described above in an iterative fashion, and also propose a coarse-to-
fine approach to lower the computational cost of this iterative higher-order
approximation. Further improvements over Lee et al. (2017b) were obtained
through the use of deep contextualized ELMo (Peters et al., 2018) word
embeddings. The current state-of-the-art scores are even higher by using BERT
finetuning (Joshi et al., 2019, 2020; Wu et al., 2020) However, this paper
focuses on the model by Lee et al. (2018).

Bamman et al. (2020) present coreference results on English literature with an
end-to-end model comparable to the one used in this paper, except for using a
separate mention detection step. However, their dataset consist of a larger
number of shorter novel fragments (2000 words). They report a CoNLL score of
68.1 on the novel fragments.

| RiddleCoref | SoNaR-1
---|---|---
| Train | Dev | Test | Train | Dev | Test
documents | 23 | 5 | 5 | 581 | 135 | 145
sentences | 6803 | 1525 | 1536 | 37,346 | 10,585 | 11,671
tokens | 105,517 | 28,042 | 28,054 | 635,191 | 171,293 | 197,392
sents per doc | 295.78 | 305 | 307.2 | 64.28 | 78.41 | 80.49
avg sent len | 15.51 | 18.39 | 18.26 | 17 | 16.18 | 16.91
mentions | 25,194 | 6584 | 6869 | 182,311 | 50,472 | 57,172
entities | 9041 | 2643 | 3008 | 128,142 | 37,057 | 39,904
mentions/tokens | 0.24 | 0.23 | 0.24 | 0.29 | 0.29 | 0.29
mentions/entities | 2.79 | 2.49 | 2.28 | 1.42 | 1.36 | 1.43
entities/tokens | 0.09 | 0.09 | 0.11 | 0.20 | 0.22 | 0.20
% pronouns | 40.4 | 35.7 | 38.1 | 11.6 | 11.3 | 11.0
% nominal | 47.0 | 49.4 | 52.8 | 70.8 | 70.4 | 71.9
% names | 12.6 | 14.9 | 9.1 | 17.6 | 18.3 | 17.1
Table 2: Dataset statistics

##  3 Coreference corpora

In this paper we consider entity coreference and focus on the relations of
identity and predication. The rest of this section describes the two Dutch
corpora we use.

###  3.1 SoNaR-1: news and Wikipedia text

The SoNaR-1 corpus (Schuurman et al., 2010) contains about 1 million words of
Dutch text from various genres, predominantly news and Wikipedia text.
Coreference was annotated from scratch (i.e., annotation did not proceed by
correcting the output of a system), based on automatically extracted
markables. The markables include singleton mentions but also non-referring
expressions such as pleonastic pronouns. The annotation was not corrected by a
second annotator. Hendrickx et al. (2008b) estimated the inter-annotator
agreement of a different corpus with the same annotation scheme and obtained a
MUC score of 76 % for identity relations (which form the majority).

We have created a genre-balanced train/dev/test split for SoNaR-1 of 70/15/15.
The documents are from a range of different genres and we therefore ensure
that the subsets are a stratified sample in terms of genres, to avoid
distribution shifts between the train and test set.111Cf.
https://gist.github.com/CorbenPoot/ee1c97209cb9c5fc50f9528c7fdcdc93.

We convert the SoNaR-1 coreference annotations from MMAX2 format into the
CoNLL-2012 format. Since dutchcoref requires parse trees as input, we use the
manually corrected Lassy Small treebank (van Noord et al., 2006; Van Noord,
2009), which is a superset of the SoNaR-1 corpus.222We could also evaluate
with predicted parses from the Alpino parser, but components of the Alpino
parser have been trained on subsets of Lassy Small, so predicted parses of
Lassy Small are not representative of Alpino’s heldout performance. We align
the Lassy Small trees at the sentence and token level to the SoNaR-1
coreference annotations, since there are some differences in tokenization and
sentence order.333The conversion script is part of
https://github.com/andreasvc/dutchcoref/ We also add gold standard NER
annotations from SoNaR-1. The manually corrected trees lack some additional
features produced by the Alpino parser (van Noord, 2006) which are needed by
dutchcoref; we merge these predicted features into the gold standard trees.

###  3.2 RiddleCoref: contemporary novels

The RiddleCoref corpus consists of contemporary Dutch novels (both translated
and originally Dutch), and was presented in van Cranenburgh (2019). The corpus
is a subset of the Riddle of Literary Quality corpus of 401 bestselling novels
(Koolen et al., 2020). This dataset was annotated by correcting the output of
dutchcoref. Most novels in the dataset were corrected by two annotators, with
the second performing another round of correction after the first. In this
dataset, mentions include singletons and are manually corrected; i.e., only
expressions that refer to a person or object are annotated as mentions.
Besides this difference, relative clauses and discontinuous constituents have
different boundaries (minimal spans).

The system by van Cranenburgh (2019) is a rule-based system that does not
require a training data, and therefore the dev/test split used in this paper
is not suitable for a supervised system. To avoid this issue, we create a new
train/dev/test split which reserves 70% for training data. We also evaluate
dutchcoref on this new split. The new dev and test sets have no overlap with
the original development set on which the rules of dutchcoref were tuned.

No gold standard parse trees are available for the novels. Instead, we use
automatically predicted parses from the Alpino parser (van Noord, 2006).

###  3.3 Dataset statistics

Table 2 shows statistics of the two datasets and their respective splits. The
documents in RiddleCoref are almost four times as long as those in SoNaR-1,
and this is reflected in a higher number of mentions per entity, while SoNaR-1
has a higher density of entities to tokens. We also see a difference due to
the more selective, manual annotation of mentions: almost 30% of SoNaR-1
tokens are part of a mention, compared to less than 25% for RiddleCoref.
Finally, we see large differences in the proportion of pronouns, nominals and
names, due to the genre difference.

##  4 Coreference systems

We now describe the two coreference systems, dutchcoref and e2e-Dutch, which
we evaluate on the coreference corpora described in the previous section.

###  4.1 Rule-based: dutchcoref

The dutchcoref system444https://github.com/andreasvc/dutchcoref (van
Cranenburgh, 2019) is an implementation of the rule-based coreference system
by Lee et al. (2011, 2013). The input to the system consists of Alpino parse
trees (van Noord, 2006), which include named entities. The system infers
information about speakers and addressees of direct speech using heuristic
rules. This information is used for coreference decisions. Note that this
information is not given as part of the input.

We have made some improvements to the rules of this system in order to make it
more compatible with the SoNaR-1 annotations; this was however based only on
the output of a single document in the development set, as well as on the
original, RiddleCoref development set on which dutchcoref was developed. When
evaluating on SoNaR-1, we apply rules to filter links and mentions from the
output to adapt to the annotation scheme of this dataset.

Figure 1: Overview of the first step of the end-to-end model in which the
embedding representations and mention scores are computed. The model considers
all possible spans up to a maximum width but only a small subset is shown
here. Figure adapted from Lee et al. (2017b).

###  4.2 End-to-end, neural: e2e-Dutch

The e2e-Dutch system555The e2e-Dutch system is being developed as part of the
Filter Bubble project at the VU and eScience center. The specific commit we
used is https://github.com/Filter-
Bubble/e2e-Dutch/tree/056dcf7d3d711a3c7b8cda241a16cdd76158a823 is fully end-
to-end in the sense that it is trained only on the token and coreference
column of the CoNLL files of the dataset, without using any metadata. Our data
does not contain speaker information which is used by models trained on the
OntoNotes dataset (Hovy et al., 2006). In addition, models trained on
OntoNotes use genre information; while our data does have genre metadata, we
have not experimented with using this feature. For English, such information
provides additional improvement in scores (Lee et al., 2017b).

The model that e2e-Dutch is based on (Lee et al., 2018) uses a combination of
character n-gram embeddings, non-contextual word embeddings (GloVe; Pennington
et al., 2014) and contextualized word embeddings (ELMo; Peters et al., 2018).
These embeddings are concatenated and fed into a bidirectional LSTM. Span
heads are approximated using an attention mechanism; while this step is
intended to approximate syntactic heads, it does not rely on parse tree
information. Figure 1 shows an overview of the model. e2e-Dutch adapts this
architecture by adding support for singletons; i.e., during mention detection,
each span is classified as not a mention, a singleton, or a coreferent
mention.

Character n-gram embeddings are extracted by iterating over the data and
feeding the character n-grams to a Convolutional Neural Network (CNN) which
then represents these n-grams as learned 8-dimensional embeddings. The GloVe
embeddings were replaced with fastText666We use Fasttext common crawl
embeddings, https://fasttext.cc/docs/en/crawl-vectors.html embeddings (Grave
et al., 2018). We also trained fastText embeddings on our own datasets but saw
a performance decrease; we therefore stick with pre-trained embeddings.
Lastly, the ELMo embeddings were replaced by BERT (Devlin et al., 2019) token
embeddings, since BERT tends to outperform ELMo (Devlin et al., 2019) and
because there is a pretrained, monolingual Dutch BERT model available whose
pretraining data includes novels (BERTje; Vries et al., 2019). However, there
is no overlap between the 7000+ novels that BERTje is trained on and the
RiddleCoref corpus. Whenever there is a mismatch between the subtokens of BERT
and the tokens in the coreference data, the model takes the average of the
BERT subtoken embeddings as token representation. The last BERT layer is used
for the token representation; however, recent research has showed that layer 9
actually performs best for Dutch coreference (de Vries et al., 2020). Note
also that we do not finetune BERT for this task, contrary to Joshi et al.
(2019); this is left for future work.

We use some different hyperparameters compared to Lee et al. (2018). Our model
only considers up to 30 antecedents per span instead of 50; this only leads to
marginally worse performance, a 0.03 decrease in the LEA F1-score, while
reducing the computational cost substantially. During training, each document
is randomly truncated at 30 sentences, but different random parts are selected
at each epoch. We have experimented with higher values for this parameter with
RiddleCoref, but only obtained marginal improvements (0.01 difference), and
did not pursue this further. The top span ratio controls the number of
mentions that are considered and determines the precision/recall tradeoff for
mentions. We experimented with tuning this parameter, but settled on the
default of 0.4. Mentions up to 50 tokens long are considered.

During training, the model is evaluated every 1500 epochs (2500 for SoNaR-1).
If the CoNLL score on the development set does not increase after three
rounds, training is stopped.

##  5 Evaluation

Before presenting our main benchmark results, we discuss the issue of
coreference evaluation metrics.

###  5.1 Metrics

The challenge with evaluating coreference resolution lies in the fact that it
involves several levels: mentions, links and entities. Results can be correct
on one level and incorrect on another, and the levels interact. One of the
most important factors in coreference performance is the performance of
mention detection, since an incorrect or missed mention can lead to a large
number of missed coreference links (especially for a long coreference chain).
We therefore report mention scores. It turns out that mention performance also
has a large influence on coreference evaluation metrics (Moosavi and Strube,
2016). We will use two coreference metrics. The CoNLL score (Pradhan et al.,
2011) is the standard benchmark, but it does not have a precision and recall
score, and the MUC, $B^{3}$, and CEAFe metrics on which it is based have their
own flaws. Therefore we will also look at the LEA metric (Moosavi and Strube,
2016). LEA gives more weight to larger entities, so that mistakes on more
important chains have more effect on the score than mistakes on smaller
entities.

Unless otherwise noted, all our results include singletons. Evaluating with
and without singletons will affect all of the scores, and the two datasets
differ in the way they annotated singletons. Singletons inflate coreference
scores due to the mention identification effect. Since most mentions are easy
to identify based on form, singletons reduce the informativeness of the
coreference score. SoNaR-1 includes automatically extracted markables instead
of manually annotated mentions, as in RiddleCoref. The automatically extracted
markables are more numerous and easier to identify (they were extracted based
on syntax) than manually annotated mentions that are restricted to potentially
referring expressions (a semantic distinction). One possibility to rule out
the mention identification effect completely is to present the systems with
gold mentions. However, this still leaves the singleton-effect. If singletons
are included, the system will not know which of the gold mentions are
singletons, and this can lead to incorrect coreference links. A dataset with
more singletons (such as SoNaR-1) will thus have more potential for incorrect
coreference links (precision errors). If singleton mentions are excluded from
the set of gold mentions, it is given that all mentions are coreferent. The
system should then use this information and force every mention to have at
least one link. However, this requires re-training or re-designing the
coreference system, and does not allow us to do a realistic end-to-end
coreference evaluation. We are therefore stuck with the complications that
come with combining mention identification and coreference resolution.

System | dataset | Mentions | LEA | CoNLL
---|---|---|---|---
|  | R | P | F1 | R | P | F1 |
dutchcoref | RiddleCoref, dev | 86.85 | 85.84 | 86.34 | 49.18 | 58.03 | 53.24 | 65.91
e2e-Dutch | RiddleCoref, dev | 83.12 | 87.65 | 85.33 | 48.37 | 50.99 | 49.65 | 64.81
dutchcoref | RiddleCoref, test | 87.65 | 90.80 | 89.20 | 50.83 | 64.78 | 56.97 | 69.86
e2e-Dutch | RiddleCoref, test | 81.95 | 89.00 | 85.33 | 44.82 | 50.48 | 47.48 | 63.55
dutchcoref | SoNaR-1, dev | 64.88 | 86.78 | 74.25 | 37.98 | 52.23 | 43.98 | 55.45
e2e-Dutch | SoNaR-1, dev | 90.24 | 88.09 | 89.16 | 65.02 | 65.55 | 65.29 | 71.53
dutchcoref | SoNaR-1, test | 65.32 | 85.94 | 74.22 | 37.87 | 52.55 | 44.02 | 55.91
e2e-Dutch | SoNaR-1, test | 88.96 | 86.81 | 87.87 | 60.67 | 62.48 | 61.56 | 68.45
Table 3: Coreference results (predicted mentions, including singletons).
Figure 2: Learning curve of e2e-Dutch on RiddleCoref dev set, showing
performance as a function of amount of training data (initial segments of
novels).

###  5.2 Results

The main results are presented in Table 3. For RiddleCoref, dutchcoref
outperforms e2e-Dutch by a 6 point margin. For SoNar-1, e2e-Dutch comes out
first, and the gap is even larger. Despite the advantage dutchcoref has due to
its use of gold standard parse trees, its performance is lower than e2e-Dutch.
We can see from the mention recall score that dutchcoref misses a large number
of potential mentions; this may be due to the fact that SoNaR-1 markables
include singletons and non-referential mentions. However, dutchcoref also has
a lower LEA recall, so the gap with e2e-Dutch on SoNar-1 is not only due to
mention performance. While results for different datasets and languages are
not comparable, the performance difference for SoNaR-1 has the same order of
magnitude as the difference for OntoNotes between the comparable rule-based
and neural systems of Lee et al. (2011) and Lee et al. (2018) in Table 1.

RiddleCoref is much smaller than the SoNaR-1 dataset. Is there enough training
data for the neural model? Figure 2 shows a learning curve for e2e-Dutch. This
curve suggests that for the coreference scores the answer is no, because the
performance does not reach a plateau—instead the curve is steep until the end.
The performance of dutchcoref is the top of the plot; if we extrapolate the
curve linearly, we might expect e2e-Dutch to outperform dutchcoref with
1.1–1.3 times the current training data. However, as an anonymous reviewer
pointed out, training curves are usually logarithmic, so more training data
may be required. Mention performance does reach a plateau, which suggests this
task is easier.

Novel | System | Mentions | LEA | CoNLL
---|---|---|---|---
|  | R | P | F1 | R | P | F1 |
Forsyth_Cobra | dutchcoref | 90.67 | 93.14 | 91.89 | 62.82 | 74.83 | 68.30 | 77.42
Forsyth_Cobra | e2e-Dutch | 78.31 | 85.23 | 81.62 | 39.82 | 44.27 | 41.93 | 55.71
Japin_Vaslav | dutchcoref | 86.19 | 92.78 | 89.36 | 44.57 | 61.39 | 51.65 | 65.79
Japin_Vaslav | e2e-Dutch | 83.13 | 91.75 | 87.22 | 49.23 | 50.89 | 50.05 | 66.09
Proper_GooischeVrouwen | dutchcoref | 88.20 | 91.12 | 89.63 | 58.65 | 66.95 | 62.53 | 72.29
Proper_GooischeVrouwen | e2e-Dutch | 87.60 | 92.21 | 89.85 | 50.10 | 44.74 | 47.27 | 64.77
Royen_Mannentester | dutchcoref | 87.15 | 86.01 | 86.57 | 44.66 | 58.21 | 50.54 | 65.01
Royen_Mannentester | e2e-Dutch | 87.90 | 89.94 | 88.91 | 54.48 | 56.19 | 55.32 | 69.93
Verhulst_LaatsteLiefde | dutchcoref | 86.24 | 87.70 | 86.96 | 45.38 | 59.66 | 51.55 | 66.09
Verhulst_LaatsteLiefde | e2e-Dutch | 82.66 | 87.98 | 85.23 | 41.58 | 48.77 | 44.89 | 61.38
Table 4: Performance difference between e2e-Dutch and dutchcoref for each
individual novel

##  6 Analysis

The previous section showed some surprising results. We now take a closer look
at the differences between the two coreference systems, datasets, and the
annotations.

###  6.1 Rule-based versus neural coreference

See Table 4 for a novel by novel comparison of dutchcoref and e2e-Dutch. On 3
out of 5 novels, dutchcoref is better on both LEA F1 and CoNLL. Interestingly,
on 1 novel, LEA F1 and CoNLL disagree on the ranking of the systems. Mention
performance is high across all novels, except for a large discrepancy on
Forsyth in which e2e-Dutch scores 10 points lower.

System | Dataset | Span | Conflated | Extra | Extra | Divided | Missing | Missing
---|---|---|---|---|---|---|---|---
|  | Error | Entities | Mention | Entity | Entity | Mention | Entity
dutchcoref | RiddleCoref | 73 | 476 | 130 | 96 | 587 | 379 | 154
e2e-Dutch | RiddleCoref | 47 | 321 | 101 | 36 | 420 | 511 | 369
dutchcoref | SoNaR-1 | 352 | 2432 | 2327 | 1772 | 2640 | 2469 | 1519
e2e-Dutch | SoNaR-1 | 203 | 1187 | 895 | 695 | 1994 | 3428 | 2330
Table 5: Error types and their respective counts for both systems and datasets

Dataset | System | error | name | nom. | pron.
---|---|---|---|---|---
RiddleCoref | d.c. | extra | 5 | 83 | 42
RiddleCoref | e2e | extra | 6 | 55 | 40
RiddleCoref | d.c. | missing | 11 | 163 | 205
RiddleCoref | e2e | missing | 115 | 274 | 122
SoNaR-1 | d.c. | extra | 544 | 1473 | 310
SoNaR-1 | e2e | extra | 175 | 550 | 170
SoNaR-1 | d.c. | missing | 283 | 1842 | 344
SoNaR-1 | e2e | missing | 825 | 2124 | 479

Incorrect part | Rest of entity | Divided | Conflated
---|---|---|---
Na | No | Pr | Na | No | Pr | d.c. | e2e | d.c. | e2e
- | - | 1+ | - | 1+ | 1+ | 104 | 66 | 118 | 74
- | - | 1+ | 1+ | 1+ | 1+ | 202 | 49 | 11 | 72
- | - | 1+ | - | 1+ | - | 62 | 66 | 156 | 31
- | 1+ | - | - | 1+ | - | 22 | 30 | 33 | 20
- | 1+ | 1+ | - | 1+ | 1+ | 33 | 31 | 16 | 13
- | 1+ | - | - | 1+ | 1+ | 34 | 18 | 33 | 6
- | 1+ | 1+ | 1+ | 1+ | 1+ | 36 | 29 | 2 | 12
- | - | 1+ | - | - | 1+ | 15 | 11 | 25 | 14
Other | 79 | 120 | 82 | 79

Table 6: Left: Counts of missing and extra mention errors by mention type.
Right: A breakdown of conflated/divided entity errors on RiddleCoref grouped
by Name/Nominal/Pronoun composition; 1+ means that the entity contains one or
more mentions of the given type.

To get more insight in the particular errors made by the systems, we perform
an error analysis using the tool by Kummerfeld and Klein (2013).777We adapted
this tool for Dutch: https://github.com/andreasvc/berkeley-coreference-
analyser This tool attributes errors to mention spans, missing or extra
mentions/entities, and entities which are divided (incorrectly split) or
conflated (incorrectly merged). We use the default configuration of ignoring
singletons mentions, but add an option to support the Dutch parse tree labels.
Table 5 shows an overview of these error types by the systems on the
RiddleCoref and SoNaR-1 test sets. We can see that e2e-Dutch makes less errors
of all types, except for missing mentions and entities, which is due to its
lower mention recall. Even though e2e-Dutch showed a high score for mention
recall on SoNaR-1 in Table 3, we actually find that dutchcoref and e2e-Dutch
both show a similarly low mention recall when singletons are excluded (65.8
and 64.3, respectively). Finally, note that a lower mention recall means that
there is less opportunity to make errors of other types, so this comparison is
not conclusive.

To understand what is going on with mention identification, we can look at a
breakdown by mention type, see Table 6. We see that e2e-Dutch produces
substantially less extra nominal (NP) mentions, but is otherwise similar. In
terms of missing mentions, e2e-Dutch makes substantially more errors on names
and nominals, but on RiddleCoref it has less missing pronouns, while it has
more missing pronouns with SoNaR-1. Although pronouns form a closed class, the
issue of pleonastic pronouns still makes pronoun mention detection non-trivial
for RiddleCoref, where pleonastic pronouns are not annotated as mentions.
Since dutchcoref has no rules to detect non-pleonastic uses of potentially
pleonastic pronouns, it defaults to treating them as non-mentions. For
SoNaR-1, the performance difference on missing mentions may be due to
information from the gold parse trees which is used by dutchcoref; for example
the possessive _zijn_ (his) has the same form as the infinitive of the verb to
be, but POS tags disambiguate this, and this information is not available to
e2e-Dutch.

Finally, we can try to understand the coreference link errors. Table 6 shows
the counts of link errors on RiddleCoref by the two systems, with the entities
categorized by their configuration. We see that for both dutchcoref and
e2e-Dutch, the most common divided and conflated entity errors have a pronoun
present in the incorrect part, although dutchcoref makes more of these errors.
We can thus reconfirm the finding by Kummerfeld and Klein (2013) and van
Cranenburgh (2019) who report that the most common link error involves
pronouns. Coreference resolution for Dutch provides an extra challenge in the
fact that the third person singular pronouns can refer to either biological or
linguistic gender (Hoste, 2005).

Figure 3: Coreference scores as a function of document length. Gold and system
output are truncated at different lengths (based on % of words, rounded to the
nearest sentence boundary); $r$ is the Pearson correlation coefficient.

###  6.2 RiddleCoref (novels) versus SoNaR-1 (news/Wikipedia)

Are the scores on the two datasets comparable? There are several issues which
hinder the comparison: document length, domain differences, and mention
annotation.

We first look at document length. It could be that the evaluation metrics are
influenced by document length, since longer documents offer more opportunities
for errors. We will investigate this effect by truncating the documents before
evaluation, while keeping other factors such as the model or training data
constant. We truncate after running the coreference system because we want to
focus on the effect of document length on the evaluation, and we have no
reason to expect the coreference systems to behave differently on truncated
texts. We truncate the novels at different lengths based on the number of
words, rounded to the nearest sentence. Note that truncating does not cause
additional errors, because gold and system output are both truncated. Figure 3
shows coreference scores as a function of document length for the novels. We
conclude that e2e-Dutch seems to perform worse on longer documents, based on
the negative correlation of scores and document length. While LEA weighs
larger entities more, we also see this effect with the CoNLL score, so it is
not an artifact of the LEA metric. Moreover, we do not see the effect for
dutchcoref, so the effect is not inherent to the coreference metrics. The
documents in SoNaR-1 are much shorter (number of sentences and words), and
this may be an advantage for e2e-Dutch. Joshi et al. (2019) report a similar
document length effect for English with their end-to-end model.

Table 2 shows there is large difference in distribution of pronouns, names,
and noun phrases, which are not equally difficult. Novels tend to have a
larger proportion of pronouns. However, it is hard to say a priori whether
this would make novels easier or more difficult in terms of coreference.

System | Dataset | Mentions | Singletons | Mentions F1 | LEA F1 | CoNLL
---|---|---|---|---|---|---
dutchcoref | RiddleCoref | predicted | excluded | 80.56 | 48.15 | 56.21
e2e-Dutch | RiddleCoref | predicted | excluded | 79.94 | 45.31 | 54.90
dutchcoref | RiddleCoref | predicted | included | 86.34 | 53.24 | 65.91
e2e-Dutch | RiddleCoref | predicted | included | 85.33 | 49.65 | 64.81
dutchcoref | RiddleCoref | gold | included | 100 | 61.89 | 75.84
e2e-Dutch | RiddleCoref | gold | included | 100 | 55.17 | 72.01
dutchcoref | SoNaR-1 | predicted | excluded | 63.57 | 39.71 | 46.96
e2e-Dutch | SoNaR-1 | predicted | excluded | 67.08 | 46.18 | 52.76
dutchcoref | SoNaR-1 | predicted | included | 74.25 | 43.98 | 55.45
e2e-Dutch | SoNaR-1 | predicted | included | 89.16 | 65.29 | 71.53
dutchcoref | SoNaR-1 | gold | included | 100 | 59.34 | 70.90
e2e-Dutch | SoNaR-1 | gold | included | 100 | 74.88 | 80.61
Table 7: Development set results under different conditions.

In order to see the influence of the mention identification effect, as well as
the influence of evaluating with and without singletons, Table 7 shows a
comparison on the development set. Note that in our experiments with
e2e-Dutch, singletons are always included during training; excluding
singletons only refers to excluding them from the system output and gold data
during evaluation. We see that ignoring singletons has a counter-intuitively
large effect on coreference scores, while it has a relatively small effect on
mention identification for RiddleCoref, but a large effect with SoNaR-1.
However, whether singletons are included or not does not change the ranking of
the systems. Finally, when gold mentions are given during evaluation we see
the large effect that mention identification has downstream, although again
the ranking is preserved.

###  6.3 SoNaR-1 annotation issues

Since the gap between the performance of e2e-Dutch and dutchcoref on SoNaR-1
is so large, we take a quick look at the SoNaR-1 annotations of a single
development set document (WR-P-E-C-0000000021), in order to understand the
errors made by dutchcoref. However, it is apparent that part of these errors
are actually errors in the annotation. The first thing that stands out are
mentions with exact string matches which are not linked; for example:
Amsterdam (5x), Hilversum (6x), _de zeventiende eeuw_ (the seventeenth
century, 4x), etc. Other errors are due to missing mentions; for example, 2
out of 10 mentions of the artist Japix are missing, probably because the name
occurs twice as part of a possessive. A corpus based on semi-automatic
annotation would not contain such errors, while it is understandable that such
links are easy to overlook in a longer document when manually annotating from
scratch.

An example of a questionable mention boundary (with corrected boundary
underlined):

. [Hij] was [burgemeester van Franeker en later gedeputeerde van Friesland in
de Staten-Generaal].
[He] was [mayor of Franeker and later deputy of Frisia in the Senate].

This is actually an example of a downside of semi-automatic annotation, at
least if there is no correction, since the markable boundaries of SoNaR-1 were
automatically extracted and could not be changed by annotators. For the
RiddleCoref corpus, such boundaries were corrected.

An example of a missing anaphoric link (second _hij_ was not linked):

. Een vers aan [Caspar Barlaeus]1 ondertekent [hij]2 met ‘Dando petere
solitus’ dat wil zeggen: [hij]2 schrijft poëzie in de hoop betere verzen terug
te krijgen .
A verse to [Caspar Barlaeus]1 he2 signes with ‘Dando petere solitus’ which is
to say: he2 writes poetry in the hope to get better verses back.

This only scratches the surface of the SoNaR-1 annotations. A more systematic
study should be done.

##  7 Conclusion

We found large gaps in performance for the two systems across the two domains,
but this result is not conclusive due to several reasons, which are as
follows. The neural system shows a weakness with the long documents in the
novel corpus, but also needs more training data to reach its full potential.
The rule-based system should be better adapted to the SoNaR-1 annotation
scheme, but the neural system’s capacity to adapt to arbitrary annotation
conventions does not necessarily imply better linguistic performance. To
maximize the comparability and usefulness of the corpora, their annotations
should be harmonized, which involves manual mention annotation. In future work
we want to improve the neural system by using genre metadata and finetuning
BERT, and the rule-based system should be extended to a hybrid system by
adding supervised classifiers.

## Acknowledgements

We are grateful to Gertjan van Noord and Peter Kleiweg for help with
preprocessing the Lassy Small treebank, to Wietse de Vries and Malvina Nissim
for comments on the evaluation, and to three anonymous reviewers for their
suggestions.

## References

  * Bamman (2017) David Bamman. 2017.  Natural language processing for the long tail.  In _Proceedings of Digital Humanities_.
  * Bamman et al. (2020) David Bamman, Olivia Lewke, and Anya Mansoor. 2020.  An annotated dataset of coreference in English literature.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 44–54.
  * Bouma et al. (2007) Gosse Bouma, Walter Daelemans, Iris Hendrickx, Véronique Hoste, and Anne-Marie Mineur. 2007.  The COREA-project: manual for the annotation of coreference in Dutch texts.  Technical report, University of Groningen.
  * Clark and Manning (2016a) Kevin Clark and Christopher D. Manning. 2016a.  Deep reinforcement learning for mention-ranking coreference models.  In _Proceedings of EMNLP_ , pages 2256–2262.
  * Clark and Manning (2016b) Kevin Clark and Christopher D. Manning. 2016b.  Improving coreference resolution by learning entity-level distributed representations.  In _Proceedings of ACL_ , pages 643–653.
  * van Cranenburgh (2019) Andreas van Cranenburgh. 2019.  A Dutch coreference resolution system with an evaluation on literary fiction.  _Computational Linguistics in the Netherlands Journal_ , 9:27–54.
  * De Clercq et al. (2011) Orphée De Clercq, Véronique Hoste, and Iris Hendrickx. 2011.  Cross-domain Dutch coreference resolution.  In _Proceedings of RANLP_ , pages 186–193.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL_ , pages 4171–4186.
  * Fernandes et al. (2012) Eraldo Fernandes, Cícero dos Santos, and Ruy Milidiú. 2012.  Latent structure perceptron with feature induction for unrestricted coreference resolution.  In _Joint Conference on EMNLP and CoNLL - Shared Task_ , pages 41–48.
  * van der Goot et al. (2015) Rob van der Goot, Hessel Haagsma, and Dieke Oele. 2015.  GroRef: Rule-based coreference resolution for Dutch.  In _CLIN26 shared task_.
  * Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018.  Learning word vectors for 157 languages.  In _Proceedings of LREC_.
  * Hendrickx et al. (2008a) Iris Hendrickx, Gosse Bouma, Frederik Coppens, Walter Daelemans, Veronique Hoste, Geert Kloosterman, Anne-Marie Mineur, Joeri van der Vloet, and Jean-Luc Verschelde. 2008a.  A coreference corpus and resolution system for Dutch.  In _Proceedings of LREC_.
  * Hendrickx et al. (2008b) Iris Hendrickx, Veronique Hoste, and Walter Daelemans. 2008b.  Semantic and syntactic features for Dutch coreference resolution.  In _Computational Linguistics and Intelligent Text Processing_ , pages 351–361, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Hoste (2005) Véronique Hoste. 2005.  _Optimization issues in machine learning of coreference resolution_.  Ph.D. thesis, Universiteit Antwerpen. Faculteit Letteren en Wijsbegeerte.
  * Hovy et al. (2006) Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006.  OntoNotes: The 90% solution.  In _Proceedings of NAACL_ , pages 57–60.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020.  SpanBERT: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Joshi et al. (2019) Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019.  BERT for coreference resolution: Baselines and analysis.  In _Proceedings of EMNLP-IJCNLP_ , pages 5807–5812.
  * Koolen et al. (2020) Corina Koolen, Karina van Dalen-Oskam, Andreas van Cranenburgh, and Erica Nagelhout. 2020.  Literary quality in the eye of the Dutch reader: The national reader survey.  _Poetics_.
  * Kummerfeld and Klein (2013) Jonathan K. Kummerfeld and Dan Klein. 2013.  Error-driven analysis of challenges in coreference resolution.  In _Proceedings of EMNLP_ , pages 265–277.
  * Lee et al. (2013) Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013.  Deterministic coreference resolution based on entity-centric, precision-ranked rules.  _Computational Linguistics_ , 39(4):885–916.
  * Lee et al. (2011) Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011.  Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task.  In _Proceedings of CoNLL_ , pages 28–34.
  * Lee et al. (2017a) Heeyoung Lee, Mihai Surdeanu, and Dan Jurafsky. 2017a.  A scaffolding approach to coreference resolution integrating statistical and rule-based models.  _Natural Language Engineering_ , 23(5):733–762.
  * Lee et al. (2017b) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017b.  End-to-end neural coreference resolution.  In _Proceedings of EMNLP_ , pages 188–197.
  * Lee et al. (2018) Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018.  Higher-order coreference resolution with coarse-to-fine inference.  In _Proceedings of NAACL_ , pages 687–692.
  * Moosavi and Strube (2016) Nafise Sadat Moosavi and Michael Strube. 2016.  Which coreference evaluation metric do you trust? A proposal for a link-based entity aware metric.  In _Proceedings of ACL_ , pages 632–642.
  * van Noord (2006) Gertjan van Noord. 2006.  At last parsing is now operational.  In _TALN06. Verbum Ex Machina. Actes de la 13e conference sur le traitement automatique des langues naturelles_ , pages 20–42.
  * van Noord et al. (2006) Gertjan van Noord, Ineke Schuurman, and Vincent Vandeghinste. 2006.  Syntactic annotation of large corpora in STEVIN.  In _Proceedings of LREC_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of EMNLP_ , pages 1532–1543.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of NAACL_ , pages 2227–2237.
  * Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012.  CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes.  In _Joint Conference on EMNLP and CoNLL - Shared Task_ , pages 1–40.
  * Pradhan et al. (2011) Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011.  CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes.  In _Proceedings of CoNLL_ , pages 1–27.
  * Recasens et al. (2010) Marta Recasens, Lluís Màrquez, Emili Sapena, M. Antònia Martí, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010.  SemEval-2010 task 1: Coreference resolution in multiple languages.  In _Proceedings of SemEval_ , pages 1–8.
  * Schuurman et al. (2010) Ineke Schuurman, Véronique Hoste, and Paola Monachesi. 2010.  Interacting semantic layers of annotation in SoNaR, a reference corpus of contemporary written Dutch.  In _Proc. of LREC_ , pages 2471–2477.
  * Van Noord (2009) Gertjan Van Noord. 2009.  Huge parsed corpora in Lassy.  In _Proceedings of TLT7_ , Groningen, The Netherlands. LOT.
  * de Vries et al. (2020) Wietse de Vries, Andreas van Cranenburgh, and Malvina Nissim. 2020.  What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models.  In _Findings of EMNLP_.
  * Vries et al. (2019) Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019.  BERTje: A Dutch BERT model.  arXiv:1912.09582.
  * Wiseman et al. (2015) Sam Wiseman, Alexander M. Rush, Stuart Shieber, and Jason Weston. 2015.  Learning anaphoricity and antecedent ranking features for coreference resolution.  In _Proceedings of ACL_ , pages 1416–1426.
  * Wu et al. (2020) Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020.  CorefQA: Coreference resolution as query-based span prediction.  In _Proceedings of ACL_ , pages 6953–6963.