††thanks: Corresponding author<EMAIL_ADDRESS>
# Spin dynamics and exchange interaction in orthoferrite TbFeO3 with non-
Kramers rare-earth ion

S. A. Skorobogatov  Kirensky Institute of Physics, Federal Research Center KSC
SB RAS, Krasnoyarsk, 660036 Russia  Siberian Federal University, Krasnoyarsk
660041, Russia  K. A. Shaykhutdinov  Kirensky Institute of Physics, Federal
Research Center KSC SB RAS, Krasnoyarsk, 660036 Russia  Siberian Federal
University, Krasnoyarsk 660041, Russia  D. A. Balaev  Kirensky Institute of
Physics, Federal Research Center KSC SB RAS, Krasnoyarsk, 660036 Russia
Siberian Federal University, Krasnoyarsk 660041, Russia  M. S. Pavlovskii
Kirensky Institute of Physics, Federal Research Center KSC SB RAS,
Krasnoyarsk, 660036 Russia  Siberian Federal University, Krasnoyarsk 660041,
Russia  A. A. Krasikov  Kirensky Institute of Physics, Federal Research Center
KSC SB RAS, Krasnoyarsk, 660036 Russia  K. Yu. Terentjev  Kirensky Institute
of Physics, Federal Research Center KSC SB RAS, Krasnoyarsk, 660036 Russia

###### Abstract

The low-temperature spin dynamics of the orthorhombic TbFeO3 perovskite has
been studied. It has been found that the inelastic neutron scattering (INS)
spectrum contains two modes corresponding to different sublattices in the
compound. The iron subsystem orders antiferromagnetically at
T${}_{\mathrm{N}}=$ 632 K and exhibits the high-energy magnon dispersion.
Magnetic dynamics of this subsystem has been described using the linear spin
wave theory and our solution yields sizable anisotropy between in-plane and
out-of-plane exchange interactions. This approach was previously used to
describe the magnon dispersion in the TmFeO3 compound. Three non-dispersive
crystal electric field levels corresponding to Tb3+ ions have been established
in the region below 40 meV at about 17, 26, and 35 meV. Study of diffuse
scattering at different temperatures has elucidated the behavior of the
magnetic correlation length. The behavior of the Tb3+ ion subsystem has been
numerically described in the framework of the point charge model. The
numerical data agree satisfactorily with the experiment and with the general
concept of the single-ion approximation applied to the rare-earth subsystem of
orthorhombic perovskites.

##  I Introduction

The magnetic oxide materials containing transition and rare-earth ions exhibit
a diversity of intriguing effects related to the complex interaction between
two magnetic subsystems. Among such materials are manganites with the general
formulas RMnO3 [1, 2] and RMn2O5 [3, 4] and huntites RFe3(BO3)4 [5, 6, 7, 8].
A separate class of the oxide compounds are orthoferrites with the general
formula RFeO3 [9], which have been explored for several decades [10, 11, 12].
The crystal structure of the RFeO3 compound is described by space group Pbnm.
The recent renewed interest in these compounds is due to the discovery of
several curious phenomena, including multiferroism below the temperature of
ordering of the rare-earth subsystem [13], laser-induced ultrafast
magnetization switching of domain walls [14, 15, 16], and the formation of a
soliton lattice in the TbFeO3 compound [17].

The family of rare-earth orthoferrites is attractive in terms of magnetic
phenomena. The unique magnetic properties of these materials are caused by
complex interactions between the magnetic moments of 3d and 4f electrons. As
was shown in [9], the RFeO3 compounds have extraordinarily high Néel
temperatures ($T_{\rm N}$ $\approx$ 600–700 K), below which the moments of
iron are ordered antiferromagnetically with a weak canting of the sublattices
resulting in weak ferromagnetism. As the temperature decreases, the Fe‒R
coupling becomes more significant and induces spin-reorientation transitions
at $T$ $\ll$ $T_{\rm N}$. The temperature $T_{\rm SR}$ of the spin-
reorientation transition depends strongly on a rare-earth ion. In particular,
in the HoFeO3 compound, this temperature is $T_{\rm SR}$ $\approx$ 50‒60 K;
for TmFeO3, it is $T_{\rm SR}\approx\ 80‒90$ K [18]; while in the case of Tb
as a rare-earth element, this temperature is much lower: $T_{\rm SR}$
$\approx$ 3‒10 K [17, 19]. At high temperatures, the subsystem of rare-earth
ions with a relatively weak R‒R coupling is paramagnetic or weakly polarized
by the molecular field of ordered Fe ions. The rare-earth magnetic sublattice
is ordered at different temperatures, depending on the type of an R ion and
external conditions. The complex magnetic properties of this system are
determined by the diversity of exchange interactions. Here, along with the
Heisenberg-type Fe–Fe, Fe–R, and R–R exchange interactions, an important role
is played by the Dzyaloshinskii–Moriya (DMI) interaction [20], which induces a
weak ferromagnetic moment.

Previously, several orthoferrites were investigated [21] and isostructural
perovskite compounds with different rare-earth ions using inelastic neutron
scattering as a basic research tool for studying the energy spectra of
excitations in the magnetic subsystems. In the YbFeO3 compound [22], the
authors found quasi-one-dimensional Yb3+ chains and “shadow modes” in the low-
temperature spin dynamics of the rare-earth subsystem. Study of the related
YbAlO3 compound [23, 24] disclosed unique magnetic quantum effects in the
rare-earth subsystem at ultralow temperatures. It should be noted that Yb3+ is
a Kramers ion, the ground state of which is split into doublets; therefore,
the pseudospin 1/2 model is applicable. Upon splitting of the ground state in
non-Kramers ions, the latter can create various combinations of singlet,
doublet, pseudo-doublet, and other combination of energy levels. A more
complex picture of the energy states significantly complicates the description
of the behavior of the rare-earth subsystem and the 3d‒4f interaction arising
between the sublattices, but the intriguing states and effects that can be
observed in the compounds containing such ions broaden the scope for the
development of physics of oxide materials. Another compound studied was TmFeO3
[25], in which we observed the effect similar to the “shadow mode” of the
rare-earth subsystem and the spin dynamics of the second singlet level. This
subsystem was described in the approximation of an ion in an external field.
We determined several parameters that can be used in the magnetic Hamiltonian,
although a variety of magnetic states of the Tm3+ ion make this problem
challenging both numerically and analytically. In this study, to solve the
problem of non-Kramers rare-earth ions in orthoferrites, we examined the
TbFeO3 compound. This material has two ordering temperatures: $T_{\rm N}^{\rm
Fe}$ $\approx 650$ K and $T_{\rm N}^{\rm Tb}$ $<$ 10 K [17, 19]. Below 8 K, it
undergoes a spin-reorientation transition. This compound was investigated
using the approaches applied previously to TmFeO3 to better understand physics
of the interactions in rare-earth orthoferrites.

##  II EXPERIMENTAL DETAILS

The TbFeO3 single crystal was grown in a Crystal Systems FZ-T-4000-H optical
floating zone furnace (Japan) at a growth rate of 3 mm/h and a relative rod
rotation at 30 rpm in air under ambient pressure. A sample 5×5×5 mm3 in size
was cut from the grown rod along its crystallographic axes. The quality of the
single crystal was estimated by the Laue method. The Lauegrams and photograph
of the grown crystal are presented in Fig II.

Figure 1.:  (a, b, c) Lauegrams of the TbFeO3 single crystal along different
crystallographic directions corresponding to sp. gr. Pbnm with lattice
parameters of $a=5.326$ Å, $b=5.602$ Å, and $c=7.635$ Å. (d) Photograph of the
single crystal obtained by optical floating zone melting.

The magnetic measurements were performed on a Quantum Design PPMS-6000
Physical Property Measurement System. The temperature and field dependences of
the magnetization were measured along different crystallographic directions in
the single-crystal sample.

Figure 2.:  Temperature dependence of the magnetization in logarithmic scale.
The measurements were performed at temperatures from 2 to 100 K after pre-
cooling to 2 K without external magnetic field. At 2 K, the hysteresis was
measured in fields of $~{}\pm$5 T; after that, the field was decreased from 5
to 0.01 T and the temperature dependence of the magnetization was measured. Of
particular interest is the temperature range from 3.5 to 4.5 K corresponding
to the spin-reorientation transition.

The behavior of the $M(T)$ dependences presented in Fig. II discloses a spin-
reorientation transition in the temperature range of 3.5–4.5 K. The transition
of the terbium subsystem is not observed under the experimental conditions
(Fig. II). In [17], this transition was detected at 7–8 K. It is noteworthy
that the inelastic neutron scattering experiment used as a source of the main
conclusions drawn in this study was carried out at $T=2$ K, i.e., below the
temperature region of the observed spin-reorientation transition. To
comprehensively analyze excitations of both the iron and terbium subsystems in
the TbFeO3 single crystal, the inelastic neutron scattering study was carried
out over a wide range of the reciprocal space and energy transfer. The
experiments were conducted using two time-of-flight mass spectrometers: an
angular-range chopper spectrometer (ARCS) [27] and a cold neutron chopper
spectrometer (CNCS) [28, 29] at the Oak Ridge National Laboratory’s Spallation
Neutron Source. The data were obtained on a single crystal with a mass of 3 g
oriented in the scattering plane $(0KL)$. The incident neutron energies were
$E_{\rm i}$ = 100 and 25 meV for the ARCS and $E_{\rm i}$ = 12 and 3.3 meV for
the CNCS. The experimental data were processed and preliminary analyzed using
the Dave [30], Horace [31], and MantidPlot [32] software. The excitation
spectra and neutron scattering cross sections of the spin Hamiltonian were
calculated using the linear spin wave theory (LSWT) implemented in the SpinW
software package [33]. The crystal electric field (CEF) was calculated using
the McPhase software [34].

##  III RESULTS AND ANALYSIS

###  III.1 Iron subsystem

In this section, we analyze the data obtained on the high-energy time-of-
flight ARCS at incident neutron energies of 150 and 50 meV. The cuts made
along the highly symmetric directions of the reciprocal space reveal the
magnon dispersion, which attains its maximum energy of 60 meV and can be
considered with confidence to be the dispersion of the iron subsystem [22, 25,
35, 36]. The low-energy region was investigated at neutron energies of up to
25 meV, which made it possible to examine the antiferromagnetic gap of the
iron subsystem and determine the general characteristics of the rare-earth
subsystem.

Using the experimental data, we determined positions of the curves at 72
points of the reciprocal space along the three highly symmetric directions in
order to determine the spin Hamiltonian. The main challenge was to find an
error of the selected points. The attempts to describe the selected points by
the Gaussian function did not yield objective results because of the low
density of points in the investigated ranges and a large error. Therefore, we
chose another model for determining an error of the points. The algorithm used
was iterative and included the operation of comparison of the intensities of
the initially proposed point and its neighbors in the chosen $Q$ or $E$
direction, depending on the cut. Next, if the ratio between intensity $I_{1}$
of the selected point and neighbor’s intensity $I_{2}$ was less than a chosen
value, the process was repeated for the next neighbors with the accumulation
of the error factor n; otherwise, the solution was considered to be found and
an error equal to 1/2 of the step in the cut direction $*n$ was assigned to
the value. Using the described procedure, we found one of the parameters (the
iron antiferromagnetic gap) with good accuracy and determined the anisotropy
constant $K_{c}$ [37] (Fig. III.1).

The behavior of the iron subsystem was described using the SpinW software
package [33] and the classical spin Hamiltonian

$H=\sum_{i,j}J_{i,j}S_{i}S_{j}+K_{c}\sum_{i}S_{c_{i}}^{2}$ (1)

in which two terms were taken into account: the first term is the Heisenberg
exchange interaction between neighboring ions and the second term is the
effective anisotropy. The DMI causes splitting of the magnon modes in the
vicinity of the Gamma point ($\Gamma$) [21]; however, the presence of the CEF
excitation of the terbium subsystem around 17 meV makes the signal noisy in
this region, so the effect of the DMI on the spin dynamics cannot be reliably
established. The first summation is made over different sets of neighbors of
Fe atoms. In Eq. (1), $K_{c}$ is the effective anisotropy responsible for the
stabilization of the $\Gamma_{2}$ phase. The exchange interaction model used
by us is similar to that chosen in [25]; the in-plane and out-of-plane
exchanges are separated. By fitting the model to experimental points, we
obtained exchange values of $J_{a}b$ = 4.74, $J_{c}$ = 4.96, $J_{ab2}$ = 0.02,
and $J_{c}2$ = 0.2, an anisotropy value of $K_{c}$ = ‒0.20, and deviations of
$R_{w}$ = 2.51.

Figure 3.:  Energy slice made near $\Gamma$ point [0,1,3]. Experimental data
were received summation on high-symmetry directions, intergration limit in
$\pm$ 0.1 r.l.u. Experimental data(black points with errorbars) and results
what we were calculated with our model(red points). On figure we highlighted
some area, at the first, 4-5 meV located here experimental point indicated
about end of AFM gap and start magnon dispersion, second, near 7 meV we think
it is splitting of magnon mod caused by DM interaction and thirdly large area
14-17 meV here located CEF levels which we discuss below in the main text.

In addition, this approach allowed us to get rid of the non-representativeness
of errors, but required the more thorough selection of fitting results in the
physical model used. This was expressed in the variance of the solutions
found; specifically, two stable solutions appeared (see Table I). The analysis
of our data can be considered satisfactory, since we managed to obtain the
exchange values for the Fe3+ subsystem and thereby characterize the magnon
dispersion in all the three directions of the reciprocal space (Fig. III.1).
Good agreement between the experimental and calculated dispersions indicated
the applicability of the LSWT-based model. In addition, the results on the
iron subsystem obtained in this study are consistent with the data reported in
[25].

Table I. Stable exchange solutions for the iron subsystem

| $J_{ab}$ | $J_{c}$ | $J_{2ab}$ | $J_{2c}$ | $K_{c}$ | $R_{w}$
---|---|---|---|---|---|---
First | 4.74 | 4.96 | 0.02 | 0.2 | -0.2 | 2.51
Second | 4.73 | 4.30 | 0.02 | 0.08 | -0.2 | 2.50

Figure 4.:  Cuts along the three characteristic directions of the reciprocal
space. (a‒c) Experimental inelastic neutron scattering spectra. The
measurements were performed at a temperature of 2 K. The data were obtained by
summation of the symmetric directions. The integration limit is $\pm$0.1
r.l.u. A gaussian smoothing was applied to the data. The calculated dispersion
curves (red) are superimposed. (d‒f) Model calculation of the inelastic
scattering spectra using the selected model.

###  III.2 Low-energy excitations

The rare-earth Tb3+ ion in the TbFeO3 compound is a 4f element and its
electrostatic interaction with surrounding ions can be described using Stevens
operators. Terbium ions in TbFeO3 have monoclinic local symmetry $C_{s}$ which
yields 15 non-zero operators that need to be taken into account in the
Hamiltonian [36].

$H_{cef}=\sum_{l,m}B_{l}^{m}O_{l}^{m}$ (2)

These parameters of the CEF can be determined using the point charge model
(PCM). The experience in using this model was demonstrated in [21, 32]. In the
PCM, atoms localized in the selected unit volume are considered to be point
charges in the crystallographic sites; this environment affects electrically
the selected ion (in our case, Tb3+) and, in this approach, any special
interactions, spin parameters, etc. are ignored. We investigated neighboring
ions lying within a sphere with a radius of $r$ = 4 Å and calculated the
parameters $B^{m}_{l}$ (a set of the calculated $B^{m}_{l}$ values is
presented in Appendix, Table A1) in the McPhase software package. Using the
set of parameters, we simulated the CEF splitting, transition intensities, and
magnetic anisotropy. The magnetization calculated using the obtained operators
is shown in Fig. III.2. It can be seen that the experimental and calculated
data agree satisfactorily for all the three directions. The obtained
experimental magnetizations are consistent with the data reported in [25] and
the comparison of the calculated and experimental curves confirms the
applicability of the PCM in describing the behavior of the magnetization of
rare-earth orthoferrites reported previously in [22, 25].

Figure 5.:  Magnetization vs. magnetic field at 7 K. The [100], [010], and
[001] crystallographic directions are colored in green, blue, and red,
respectively. The solid lines correspond to the experiment and the triangles
to the calculation.

Based on the parameters calculated using the PCM, the transition energies for
the Tb3+ ion were also simulated. The ground state of this ion is degenerate
and consists of 13 states, 6 of which are quasi-doublets and one is a singlet
(a set of parameters is presented in Appendix). The first calculated quasi-
doublet has an energy of E0,1 $\sim$ 0 meV; this level belongs to the elastic
neutron spectrum. The second CEF level is well-determined from the
experimental data and located at 17 meV; in the calculation, this level is
represented by a quasi-doublet with E2,3 $\sim$ 14 meV. Fig. III.2(a),
III.2(b) show these energy levels together with the overlying levels
corresponding to the quasi-doublets calculated by us using the PCM. Fig.
III.2(c) compares the calculated neutron dispersion intensities and the
results of summation of the experimental data in the vicinity of the region
[0, 2, 1]. The integration limit is ±0.25 r.l.u.

Figure 6.:  (a, b) Cuts along the highly symmetric directions. The
measurements were performed at 2 K. The data were obtained by summation of the
symmetric directions. The integration limit is ±0.1 r.l.u. The spread
operation was also applied. The crystal field level at 16‒17 meV is clearly
seen in (a). In (b), there are two crystal field levels at 16‒17 and 26 meV;
around 34 meV there is a hint of another crystal field level, but it cannot be
established with confidence. (c) Comparison of the integral crystal field
values with the calculation using the single-ion model in the McPhase
software. The experimental data and the spin dynamics calculated using the PCM
are in good agreement. The integration limit is ±0.25 r.l.u.

Having considered the energy range below 3.3 meV, we investigated the magnetic
excitations of the Tb subsystem and determined the correlation lengths in two
directions. The intensity cuts of the detected elastic peaks are shown in Fig.
III.2(a). The examination of the low-energy spectra at the diffuse scattering
allowed us to describe the magnetic structure factor. Here, as in [39, 40], we
used the Lorentzian function

$S(Q)\propto\frac{sinh(c/\xi_{l})}{cosh(c/\xi_{l})-cos[\pi(l-1)]}\frac{sinh(c/\xi_{k})}{cosh(c/\xi_{k})-cos[\pi
k]}$ (3)

where $\xi_{l}$ and $\xi_{k}$ are the correlation lengths along the c and b
axes, respectively.

Making the energy cuts in the K and L directions of the reciprocal space, we
obtained the data shown by dots in Figs. III.2b. The approximation of the
experimental data using the structural peak width and the Lorentzian function
yielded the correlation lengths for each direction and temperature. The
temperature dependence of the correlation length for the two investigated
directions is shown in Fig. III.2(b). It can be clearly seen that, after the
spin-reorientation transition, the correlation length sharply increases, which
is consistent with the fact of ordering of both subsystems below this
transition.

a

b

Figure 7.:  (a) Experimental data integrated over the H directions of the
reciprocal space in the limit ±0.1 r.l.u. The energy axis was integrated
within ±0.1 meV. From up to down located slices, maken from different
temperature. (b) Experimental data integrated over the directions of the
reciprocal space in the limit ±0.15 r.l.u. The energy axis was integrated
within ±0.1 meV. Result of the approximation of the experimental data by the
convolution of the Lorentzian and Gaussian functions. The resolution was
estimated from the structural peak width using two characteristic directions
of the reciprocal space. On the last figure temperature dependence of the
correlation length for both directions of the reciprocal space.

##  IV DISCUSSION AND CONCLUSION

The inelastic neutron scattering study of the single-crystal TbFeO3 sample
showed the antiferromagnetic ordering of the Fe3+ subsystem. Using the spectra
obtained, the Fe‒Fe exchange interaction values were calculated within the
linear spin wave theory. The spectra are similar to the spectra obtained for
iron in the TmFeO3 compound [24], which manifests itself in the energy levels
of the magnon dispersion, in the exchange interaction values, and,
importantly, in the anisotropy of the exchange values determined for these
compounds. The antiferromagnetic gap observed in the investigated compound at
4.5 meV is governed by the uniaxial anisotropy, which was also established by
us in the framework of the linear spin wave theory. The gap depends on the
type of a rare-earth ion contained in the compound.

The examination of the spectra in the range of up to 50 meV elucidated the
behavior of the Tb3+ subsystem, specifically, the position of the first three
CEF levels, which are located at energies of 17, 25, and 35 meV for the
compound under study. We failed to establish any dispersion behavior of these
levels from the experimental data; it was found that the first level has the
highest population at the investigated temperature and the intensity of this
level competes with that of Fe3+ magnons.

The magnetic subsystem of the rare-earth Tb3+ ion was described using the
point charge model. In this model, the magnetization of the terbium subsystem
and the magnon dispersion were calculated and plotted. The comparison of the
calculation with the experiment demonstrated their good agreement. The
correlation lengths at different temperatures were determined, which made it
possible to establish the temperature above which the terbium subsystem
acquires a long-range magnetic order.

##  V ACKNOWLEDGMENTS

We acknowledge A. Podlesnyak for help with INS experiments and S. E. Nikitin
for stimulating discussion. This study was supported by the Russian Foundation
for Basic Research, project no. 20-32-90142. This research used resources at
Spallation Neutron Source, a DOE Office of Science User Facility operated by
the Oak Ridge National Laboratory.

##  VI APPENDIX:RESULTS OF THE POINT-CHARGE MODEL CALCULATIONS

The set of CEF parameters and the energy levels calculated from the PC model
are given in Tables A1 and A2, respectively.

Table A1. Calculation of Stevens parameters B${}^{m}_{l}$ and energy levels
with corresponding magnetic moments.

$B_{l}^{m}$ |  |  | $B_{l}^{m}$ |  |
---|---|---|---|---|---
$B_{2}^{0}$ | = | $3.91\times 10^{-1}$ | $B_{6}^{0}$ | = | $0.1\times 10^{-5}$
$B_{2}^{2}$ | = | $0.39\times 10^{-1}$ | $B_{6}^{2}$ | = | $-0.3\times 10^{-5}$
$B_{2}^{-2}$ | = | $-3.03\times 10^{-1}$ | $B_{6}^{-}2$ | = | $-0.3\times 10^{-5}$
$B_{4}^{0}$ | = | $-0.35\times 10^{-3}$ | $B_{6}^{4}$ | = | $1.8\times 10^{-5}$
$B_{4}^{2}$ | = | $-1.34\times 10^{-3}$ | $B_{6}^{-}4$ | = | $0.1\times 10^{-5}$
$B_{4}^{-2}$ | = | $-3.37\times 10^{-3}$ | $B_{6}^{6}$ | = | $-0.3\times 10^{-5}$
$B_{4}^{4}$ | = | $0.45\times 10^{-3}$ | $B_{6}^{-}6$ | = | $-0.0\times 10^{-5}$
$B_{4}^{-4}$ | = | $-8.1\times 10^{-3}$ |  |  |

Table A2. The energy levels and out of ground state transition probabilities.

|  $E$ (meV) | $\langle n|J_{x}|m\rangle^{2}$ | $\langle n|J_{y}|m\rangle^{2}$ | $\langle n|J_{z}|m\rangle^{2}$
---|---|---|---|---
$|E_{0}\rangle\to|E_{1}\rangle$ | 0.0169 | 0.0 | 0.0 | 35.0109
$|E_{0}\rangle\to|E_{2}\rangle$ | 13.9161 | 0.0 | 3.0388 | 0.1414
$|E_{0}\rangle\to|E_{3}\rangle$ | 14.1608 | 0.8099 | 0.0 | 0.0
$|E_{0}\rangle\to|E_{4}\rangle$ | 24.7651 | 1.1200 | 0.0 | 0.0
$|E_{0}\rangle\to|E_{5}\rangle$ | 25.7851 | 0.0 | 1.4517 | 0.0658
$|E_{0}\rangle\to|E_{6}\rangle$ | 34.3537 | 0.0 | 0.0881 | 0.0154
$|E_{0}\rangle\to|E_{7}\rangle$ | 34.7824 | 0.325 | 0.0 | 0.0
$|E_{0}\rangle\to|E_{8}\rangle$ | 39.0313 | 0.0 | 0.0019 | 0.0052
$|E_{0}\rangle\to|E_{9}\rangle$ | 43.7716 | 0.0011 | 0.0 | 0.0
$|E_{0}\rangle\to|E_{10}\rangle$ | 44.4169 | 0.0319 | 0.0 | 0.0
$|E_{0}\rangle\to|E_{11}\rangle$ | 53.4470 | 0.0 | 0.0013 | 0.0007
$|E_{0}\rangle\to|E_{12}\rangle$ | 53.4992 | 0.0 | 0.0007 | 0.0003

## References

  * [1] Salamon, Myron B., and Marcelo Jaime, The physics of manganites: Structure and transport, Rev. Mod. Phys. 73, 583, (2001)
  * [2] Tokura, Yoshinori, and Yasuhide Tomioka, Colossal magnetoresistive manganites, J. Magn. Magn. Mater. 200, 1, (1999).
  * [3] Kimura, Hiroyuki, et al, Spiral Spin Structure in the Commensurate Magnetic Phase of Multiferroic RMn2O5, J. Phys. Soc. Jpn. 76, 074706, (2007).
  * [4] Schieber, M., et al., Magnetocrystalline anisotropy of rare‐earth manganites, J.Appl. Phys. 44, 1864 (1973).
  * [5] Ritter, C., et al., Magnetic structure in iron borates RFe3(BO3)4 (R=Y, Ho): a neutron diffraction and magnetization study, J. Phys.: Condens. Matter, 20, 365209, (2008).
  * [6] Pankrats, A. I., et al., Transformation from an easy-plane to an easy-axis antiferromagnetic structure in the mixed rare-earth ferroborates PrxY1-xFe3(BO3)4: magnetic properties and crystal field calculations, J. Phys.: Condens. Matter, 28, 396001, (2016).
  * [7] Krylov, A. S., et al., Manifestation of magnetoelastic interactions in Raman spectra of HoxNd1-xFe3(BO3)4 crystals, J. Adv. Dielectr., 08, 1850011, (2018).
  * [8] Pankrats, A., et al., Low-temperature magnetic phase diagram of HoFe3(BO3)4 holmium ferroborate: a magnetic and heat capacity study, J. Phys.: Condens. Matter 21, 436001, (2009).
  * [9] R. L. White, Work on the Magnetic and Spectroscopic Properties of the Rare-Earth Orthoferrites, J. Appl. Phys. 40, 1061, (1969).
  * [10] Sosnowska, I., and A. K. Zvezdin, Origin of the long period magnetic ordering in BiFeO3, J. Magn. Magn. Mater. 140, 167, (1995).
  * [11] Ruette, Benjamin, et al., Magnetic-field-induced phase transition in BiFeO3 observed by high-field electron spin resonance: Cycloidal to homogeneous spin order, Phys. Rev. B 69, 064114, (2004).
  * [12] K. P. Belov, A. K. Zvezdin, and A. A. Mukhin, Magnetic phase transitions in terbium orthoferrite, J. Exp. Theor. Phys. 49, 557, (1979).
  * [13] Tokunaga, Yusuke, et al., Composite domain walls in a multiferroic perovskite ferrite, , Nature Mater. 8, 558 (2009).
  * [14] Kimel, A. V., et al., Ultrafast non-thermal control of magnetization by instantaneous photomagnetic pulses, Nature 435, 655, (2005).
  * [15] De Jong, J. A., et al., Laser-induced ultrafast spin dynamics in ErFeO3, Phys. Rev. B, 84, 104421, (2011).
  * [16] Jiang, Junjie, et al., Dynamical spin reorientation transition in NdFeO3 single crystal observed with polarized terahertz time domain spectroscopy, Appl. Phys. Lett. 103, 062403, (2013).
  * [17] S. Artyukhin, M. Mostovoy, N. P. Jensen, D. Le, K. Prokes, V. G. de Paula, H. N. Bordallo, A. Maljuk, S. Landsgesell, H. Ryll, B. Klemke, S. Paeckel, K. Kiefer, K. Lefmann, L. T. Kuhn, and D. N. Argyriou, Solitonic lattice and Yukawa forces in the rare-earth orthoferrite TbFeO3, Nat. Mater. 11, 694 (2012)
  * [18] Saito, Kazuya, et al., High-precision detection of the heat-capacity anomaly due to spin reorientation in TmFeO3 and HoFeO3, Solid State Commun. 120, 129, (2001).
  * [19] O Nikolov et al., A Mossbauer study of temperature-driven spin-reorientation transitions in TbFeO3, J. Phys.: Condens. Matter, 6, 3793, (1994).
  * [20] Dzialoshinskii, I. E., Thermodynamic theory of weak ferromagnetism in antiferromagnetic substances, Soviet Physics Jetp-Ussr, 5, 1259, (1957).
  * [21] A Podlesnyak et al., Low-energy spin dynamics in rare-earth perovskite oxides, J. Phys.: Condens. Matter, 33, 403001, (2021).
  * [22] S. E. Nikitin et al., Decoupled spin dynamics in the rare-earth orthoferrite YbFeO3: Evolution of magnetic excitations through the spin-reorientation transition, Phys. Rev. B 98, 064424, (2018).
  * [23] L.S. Wu et al., Tomonaga-Luttinger liquid behavior and spinon confinement in YbAlO3, Nat. Commun. 10, 698, (2019).
  * [24] S.E Nikitin et al., Multiple fermion scattering in the weakly coupled spin-chain compound YbAlO3, Nat. Commun. 12, 1, (2021).
  * [25] S. A. Skorobogatov et al., Low-temperature spin dynamics in the TmFeO3 orthoferrite with a non-Kramers ion, Phys. Rev. B 101, 014432, (2020).
  * [26] Y. Cao et al., Magnetic phase transition and giant anisotropic magnetic entropy change in TbFeO3 single crystal, J. Appl. Phys. 119, 063904, (2016).
  * [27] D. L. Abernathyet et al., Design and operation of the wide angular-range chopper spectrometer ARCS at the Spallation Neutron Source, Rev. Sci. Instrum. 83, 015114, (2012).
  * [28] G. Ehlers et al., The new cold neutron chopper spectrometer at the spallation neutron source: design and performance, Rev. Sci. Instrum. 82, 085108, (2011).
  * [29] G. Ehlers, A. Podlesnyak, and A. I. Kolesnikov, The cold neutron chopper spectrometer at the Spallation Neutron Source - A review of the first 8 years of operation, Rev. Sci. Instrum. 87, 093902, (2016).
  * [30] R. Azuah et al., DAVE: a comprehensive software suite for the reduction, visualization, and analysis of low energy neutron spectroscopic data, J. Res. Natl. Inst. Stan. Technol. 114, 341, (2009).
  * [31] R. A. Ewings, et al., HORACE: software for the analysis of data from single crystal spectroscopy experiments at time-of-light neutron instruments, Nucl. Instrum. Methods Phys. Res. Sect. A 834, 132, (2016).
  * [32] O. Arnold et al., Mantid – Data analysis and visualization package for neutron scattering and $\mu$SR experiments, Nucl. Instrum. Methods Phys. Res. Sect. A 764, 156, (2014).
  * [33] S. Toth and B. Lake, Linear spin wave theory for single-Q incommensurate magnetic structures, J. Phys.: Condens. Matter 27, 166002, (2015).
  * [34] M Rotter, Using mcphase to calculate magnetic phase diagrams of rare earth compounds, J. Magn. Magn. Mater. 272, E481, (2004).
  * [35] S. Hahn et al., Inelastic neutron scattering studies of YFeO3, Phys. Rev. B 89, 014420, (2014).
  * [36] S. M. Shapiro, J. D. Axe, and J. P. Remeika, Neutronscattering studies of spin waves in rare-earth orthoferrites, Phys. Rev. B 10, 2014, (1974).
  * [37] To stabilize the correct ground state with a small spin canting [41] we have to introduce two Dzyaloshinskii- Moriya interaction constants to Eq. 1. However, both are small and have a negligible effect of the spectra.
  * [38] M. T. Hutchings, Point-charge calculations of energy levels of magnetic ions in crystalline electric fields, 1964, Solid State Phys., 16, 227–273.
  * [39] I. Zaliznyak et al., Spin-liquid polymorphism in a correlated electron system on the threshold of superconductivity, Proc. Natl. Acad. Sci. (USA) 112, 10316, (2015).
  * [40] L. S. Wu et al., Magnetic ground state of the Ising-like antiferromagnet DyScO3, Phys. Rev. B 96, 144407, (2017).
  * [41] R. L. White, Work on the Magnetic and Spectroscopic Properties of the Rare-Earth Orthoferrites, J. Appl. Phys. 40, 1061 (1969).


#  Spaghetti: Open-Domain Question Answering
from Heterogeneous Data Sources with Retrieval and Semantic Parsing

Heidi C. Zhang Sina J. Semnani Farhad Ghassemi Jialiang Xu
Shicheng Liu Monica S. Lam
Computer Science Department, Stanford University
Stanford, CA
{chenyuz, sinaj, farhadg, xjl, shicheng<EMAIL_ADDRESS>
###### Abstract

We introduce  Spaghetti: Semantic Parsing Augmented Generation for Hybrid
English information from Text Tables and Infoboxes, a hybrid question-
answering (QA) pipeline that utilizes information from heterogeneous knowledge
sources, including knowledge base, text, tables, and infoboxes. Our LLM-
augmented approach achieves state-of-the-art performance on the CompMix
dataset, the most comprehensive heterogeneous open-domain QA dataset, with
56.5% exact match (EM) rate. More importantly, manual analysis on a sample of
the dataset suggests that Spaghetti is more than 90% accurate, indicating that
EM is no longer suitable for assessing the capabilities of QA systems
today.111Code is available at https://github.com/stanford-oval/WikiChat.

Spaghetti: Open-Domain Question Answering
from Heterogeneous Data Sources with Retrieval and Semantic Parsing

Heidi C. Zhang Sina J. Semnani Farhad Ghassemi Jialiang Xu Shicheng Liu Monica
S. Lam Computer Science Department, Stanford University Stanford, CA {chenyuz,
sinaj, farhadg, xjl, shicheng<EMAIL_ADDRESS>

##  1 Introduction

Open-domain question answering (QA) grounded in knowledge corpora has long
been an active topic of research in natural language processing (Chen et al.,
2017; Wang et al., 2018; Lee et al., 2019; Asai et al., 2020; Izacard and
Grave, 2021; Khattab et al., 2021; Asai et al., 2022). With the rise of LLMs,
new state of the art has been established with QA separately on free-text
documents (Semnani et al., 2023; Jiang et al., 2023; Gao et al., 2023; Khattab
et al., 2023), databases (Pourreza and Rafiei, 2023; Nan et al., 2023; Zhang
et al., 2023), and graph databases (Xu et al., 2023; Luo et al., 2023; Li et
al., 2023).

In practice, we need to fully leverage hybrid data sources. For instance,
Wikipedia alone offers a wealth of knowledge through nearly 7M free-text
articles; many of these articles contain structured information in tables and
infoboxes; Wikidata is a knowledge graph containing over 17 billion triples.
This paper investigates how to leverage LLMs to answer questions on all the
different types of data.

Figure 1: Given an input query, Spaghetti gathers factual information from
four sources to generate a prediction. In parallel, we parse the query to
logical form to query Wikidata (left), run retrieval to find information from
Wikipedia text, tables, and infoboxes (middle), and generate a response using
LLM, only keeping a claim if it is verified (right). Finally, evidences are
gathered to generate an answer.

The premise of this paper is that we need hybrid data and we need hybrid
access methods. Our main contribution is twofold. First, we introduce a hybrid
LLM-based system (Fig. 1), Spaghetti, that combines information retrieval with
semantic parsing for question answering and achieves SOTA of 56.5% exact match
rate on CompMix, the most comprehensive open-domain QA dataset on
heterogeneous sources. Second, we show via careful evaluation and analysis
that measuring the accuracy of LLM-based QA systems with the exact-match
metric against hand-annotated answers is obsolete. By using evaluation methods
closer to human judgment, we find that Spaghetti is more than 90% accurate on
CompMix, surprisingly higher than traditional methods would suggest.

##  2 Related Work

TextQA, TableQA. and KBQA have all been individually studied extensively (Zhao
et al., 2023a; Lu et al., 2024; Pan et al., 2024, inter alia). However, the
task of answering questions from two or more sources, known as heterogeneous
QA, is under-studied. Some literature investigate two of the three sources,
including those on closed domain (Miller et al., 2016; Chen et al., 2020;
Pramanik et al., 2021; Liu et al., 2023; Lei et al., 2023) and open domain
(Chen et al., 2021; Zhao et al., 2023b; Han and Gardent, 2023; Ma et al.,
2022a, 2023), but very limited existing work experiments on all three.

ConvMix (Christmann et al., 2022) collected the first conversational QA
dataset that requires knowledge from all three heterogeneous sources.
Crowdworkers were asked to pick an entity of their interest and find the
answer from one of the Wiki sources - Wikidata, Wikipedia text, Wikipedia
tables, or Wikipedia infoboxes. Christmann et al. (2023a) later collated the
completed conversations to derive the CompMix dataset with 9410 self-contained
question-answer pairs.

Oguz et al. (2022), Ma et al. (2022b), and Christmann et al. (2022) proposed
pipelines to answer questions from all three sources, by linearizing all
structured information and applying text retrieval methods. Christmann et al.
(2023a), on the other hand, unifies all the sources by representing all
relevant information in a knowledge graph and uses GNN message passing to find
the answer. The former gives up the advantage of using formal query languages
on structured data, which can support operations such as ranking and
averaging. The latter gives up the advantage of the expressiveness and
versatility of free-text knowledge representation. Concurrent with our work,
Lehmann et al. (2024) adopts another view that breaks down QA solution
processes as tool calls and thoughts. They propose a human-like approach that
teaches LLMs to gather heterogeneous information by imitating how humans use
retrieval tools, which requires human-annotated demonstrations.

##  3 Spaghetti

Spaghetti is a hybrid QA pipeline that takes advantage of both structured and
unstructured information. We obtain evidence from heterogeneous sources in
parallel, including structured knowledge bases, plain text, linearized tables
/ infoboxes, and LLM-generated claims that are verified, and gather those
evidence to generate the final answer using a few-shot LLM (Fig. 1).

Figure 2: An example with a failure case of ReFinED and our entity linking
module correcting the failure.

###  3.1 Knowledge Base

Xu et al. (2023) proposes a semantic parsing framework for Wikidata. By
integrating a named entity linker and a fine-tuned LLaMA trained with modified
SPARQL, they establish a strong baseline on the WikiWebQuestions dataset. We
adopt their methodology as the interface to WikiData.

As noted by Xu et al. (2023), most of the semantic parsing errors are due to
the failure of the entity linker model ReFinED (Ayoola et al., 2022). To
improve on their approach, we propose a novel entity linking method where we
first ask an LLM to detect entity mentions and generate a brief (maximum 10
words) description of each detected entity. We then feed the list of detected
entities and descriptions to ReFinED to obtain the corresponding Wikidata
entity IDs. Leveraging the world knowledge of an LLM in this fashion provides
an additional mechanism to detect entity mentions and provide more context for
ReFinED to disambiguate and link entities. Figure 2 illustrates how our entity
linking module works. Here, ReFinED alone fails to identify the entity
“Academy Award”, but it succeeds with LLM-provided context.

###  3.2 Text

Retrieval-augmented generation is a common approach for grounding LLMs in
textual knowledge sources like Wikipedia. To avoid LLM hallucination, Semnani
et al. (2023) proposes the WikiChat pipeline that combines retrieval with
verification of LLM-generated response, achieving significantly higher factual
accuracy than GPT-4. We adopt a similar approach when handling text.

We first extract Wikipedia text using Wikiextractor
222https://github.com/attardi/wikiextractor. ColBERT (Santhanam et al., 2022)
is used to retrieve Wikipedia passages that may answer a given query, and each
of the top-k retrieved passages goes through a few-shot LLM summarizer.

As shown in the rightmost path of Figure 1, similar to WikiChat, Spaghetti
also makes use of the internal factual knowledge of LLMs by first generating a
response and then verifying the claims made in the response using retrieved
information, retaining only grounded claims.

###  3.3 Tables and Infoboxes

Most NLP research using Wikipedia simply ignores the embedded tables and
infoboxes, as extraction and preprocessing are challenging. With the help of
tools such as WikiTextParser 333https://github.com/5j9/wikitextparser and
regex matching, we programmatically extract 9 million tables and infoboxes
from Wikipedia pages and linearize them so that they can be encoded as a set
of ColBERT (Santhanam et al., 2022) index for retrieval. Being linearized, the
retrieved item can then be read by LLMs directly.

Below is an example linearized table from the Wikipedia article “Arundhati
Roy”:

> Fiction ; No.: 1, Title: “The God of Small Things”, Publisher: Flamingo,
> Year: 1997, ISBN: ISBNT|0-00-655068-1.<tr> No.: 2, Title: “The Ministry of
> Utmost Happiness”, Publisher: Hamish Hamilton, Year: 2017, ISBN:
> ISBNT|0-241-30397-4 .<tr>

For each table, we include the section title, two preceding sentences, and two
succeeding sentences of the table as additional context, if there are any in
the current section. Table rows are formatted as “column_name: cell_content,
…” with “<tr>” as the row separator.

Since ColBERT is pretrained with textual passages and not tables, we finetune
ColBERT for table retrieval. After retrieval, the retrieved table is then fed
into a few-shot LLM to extract information directly relevant to the query.

###  3.4 Putting it Together

At the final stage, we gather and combine evidence from all sources. The
answer from Wikidata is formatted as “Wikidata says the answer to <query> is:
<answer>.” The retrieved text and tables/infoboxes each goes through an LLM
summarization prompt, as mentioned earlier, attempting to extract relevant
information from each retrieved item. The verified claim(s) from the LLM-
generated answer (if any) is also added to the evidence pool.

Finally, all evidence is fed to a few-shot LLM prompt to generate a single
answer to the query. In some cases the answer may be contained in more than
one information source, and such redundancy can help reduce errors introduced
in earlier stages of the pipeline.

| Exact Match | Superset | GPT-4 Match | Platinum
---|---|---|---|---
| dev | test | dev | test | dev | test | dev (100)*
CONVINSE (Christmann et al., 2022) | – | 40.7% | – | – | – | – | –
UniK-QA (Oguz et al., 2022) | – | 44.0% | – | – | – | – | –
EXPLAINGNN (Christmann et al., 2023b) | – | 44.2% | – | – | – | – | –
GPT-3 (text-davinci-003) | – | 50.2% | – | – | – | – | –
GPT-3.5 (turbo-instruct) | 36.4% | 36.1% | 53.2% | 54.2% | 68.0% | 69.9% | 74%
GPT-4 | 53.0% | 52.8% | 60.9% | 62.0% | 76.7% | 78.4% | 81%
Spaghetti (LLaMA-7B) | 53.8% | 51.7% | 61.7% | 60.5% | 69.8% | 70.4% | 77%
Spaghetti (GPT-3.5) | 58.5% | 55.6% | 67.7% | 65.6% | 76.9% | 75.3% | 84%
Spaghetti (GPT-4) | 57.3% | 56.5% | 70.2% | 70.0% | 80.8% | 81.9% | 92%
Table 1: Main results on the CompMix development and test set. UniK-QA and
GPT-3 (text-davinci-003) results are from Christmann et al. (2023a). We use
the same zero-shot generation prompt published by Christmann et al. (2023a) to
evaluate GPT-3.5 (turbo-instruct) and GPT-4.
*: Platinum results are obtained by an expert manually relabeling and evaluating the first 100 development set examples. | EM | Superset | GPT-4 Match
---|---|---|---
Text | 53.8% | 61.6% | 71.1%
Tables | 48.9% | 59.5% | 65.9%
KB | 32.9% | 40.4% | –
Text+Tables | 55.6% | 65.4% | 74.5%
Text+Tables+KB | 58.5% | 67.7% | 76.9%
Table 2: Spaghetti (GPT-3.5) ablation results on the CompMix development set,
for using different knowledge sources. Results on “KB” are derived by directly
comparing generated QID(s) against gold QID(s), while other methods are by
string comparisons.

##  4 Experiments

We evaluate Spaghetti on the CompMix development and test sets, which contain
1680 and 2764 questions respectively.

For querying Wikidata, we use the LLaMA-7B semantic parser from Xu et al.
(2023) trained on both WikiWebQuestions and QALD-7 (Usbeck et al., 2017). We
use GPT-3.5 as the LLM in our entity linking module.

We experiment with LLaMA-7B, GPT-3.5-turbo-instruct, and GPT-4444We access GPT
models via the Microsoft Azure OpenAI API. We use the GPT-4 snapshot from June
13th, 2023., respectively, as the LLM backbone in all the stages for handling
retrieved evidences and for answer generation. We use few-shot prompts for
GPT-3.5 and GPT-4, and use the LLaMA model from Semnani et al. (2023), which
is distilled from the teacher GPT-4.

To fine-tune the ColBERT table retriever, we obtain training data from the NQ-
Tables dataset Herzig et al. (2021), where each example matches one gold table
to a query. For each positive example, we sample 10 negative tables to obtain
a total of 95K training triplets. We confirmed on the NQ-Tables dataset that
the fine-tuned version improves table retrieval Recall@3 by 10%.

#### Evaluation Metrics.

Bulian et al. (2022) and Kamalloo et al. (2023) have established that exact
match (EM) against gold answers, which is commonly used for evaluating QA
systems, cannot evaluate generative models properly as they often generate
lexically different, but semantically equivalent answers. To properly assess
our approach, we introduce two additional evaluation metrics: (1) Superset:
whether the gold answer is a substring of the generated answer, as the latter
tends to spell out the answer in long form and may include a more complete
answer. (2) GPT-4 Matching: using GPT-4 with a few-shot prompt to determine
whether the generated answer matches the gold, similar to Kamalloo et al.
(2023).

Moreover, datasets may have ambiguous queries or even wrong annotations. To
assess the quality of CompMix, we sample 100 questions and carefully use
online information sources to find the answers and decide if the generated
answers are correct. We refer to this metric as _platinum_ evaluation. GPT-4
for answer matching eliminates formatting issues, providing a result closer to
human judgment than EM, while platinum evaluation further considers annotation
issues and eliminates errors caused by ambiguous queries.

##  5 Results

Spaghetti (GPT-4) achieves 56.5% EM rate on the test set of CompMix, improving
on the previously reported state-of-the-art (GPT-3) by 6.3%, and improves on
the GPT-4 baseline by 3.7% (Table 1). Spaghetti (GPT-3.5) also improves upon
all of the baselines. We note that the EM scores of the GPT-3.5-turbo-instruct
baseline are low because this model tends to be more verbose.

Spaghetti (GPT-4) achieves 81.9% test set accuracy by GPT-4 matching, and 92%
_platinum_ accuracy on the 100 development set examples. Of the 8 errors
cases, 3 have unanswerable questions (e.g. “FC Cincinnati soccer club?”), thus
the true accuracy rate is 92/97 (94%).

#### Ablations.

We evaluate the contribution of each knowledge source by ablating different
parts of the system (Table 2). Using text alone already outperforms the
previous SOTA, with each additional source further improving the result. Note
that for many questions, information exists in multiple sources; the
relatively little contribution from Wikidata and tables reflects mainly on the
makeup of CompMix, not their value as knowledge sources. For detailed
experimental results on our Wikidata entity linking approach, see Appendix A.

#### Human Evaluation

We examine how our human “Platinum” evaluation (92%) differs from the EM
metric (60%) on our sample of 100 cases. We report findings on the Spaghetti
(GPT-4) responses, specifically in 32 annotated examples where EM fails but
the response is indeed correct (Figure 3). Out of the 32 discrepancies, the
unsophisticated “Superset” metric resolves 7, and GPT-4 matching resolves an
additional 14. Platinum evaluation identifies that 4 questions have incorrect
gold labels, and 7 questions are ambiguous and the generated answers are
correct though different from the gold. We include examples for each of these
resolved cases in Appendix B.

Figure 3: Evaluation issues resolved within the gap between EM and _platinum_.

Of the 5 true errors, one is because Spaghetti cannot find the answer in any
of the four information sources; in the other 4 cases, the answer generator
cannot identify the correct answer retrieved due to conflicting or misleading
evidence. See Appendix C for details on each error case.

##  6 Conclusion

We propose Spaghetti, a hybrid open-domain question-answering system that
combines semantic parsing and information retrieval to handle structured and
unstructured data.

Spaghetti achieves an exact match rate improvement of 6.3% over the prior
state-of-the-art on the CompMix dataset. More importantly, we show that our
approach is likely to reach an accuracy of over 90%, if we account for
differences in the answer wording and incompleteness/errors in gold labels.
This, however, does not mean open-domain QA is solved. Further research is
needed to handle open-domain questions that require complex structured queries
or composition of answers from multiple information sources, none of which are
included in CompMix.

## Limitations

This work focuses specifically on open-domain QA with heterogeneous knowledge
sources, and we only report results on the CompMix dataset due to the limited
availability of high-quality datasets in this domain. A natural future work is
to develop more diverse and advanced datasets that further push the need to
utilize each knowledge source.

We evaluate on single-turn QA and do not work with conversations in this
paper, and Spaghetti can be extended to handle fact-based conversational
questions or even chitchat that involves facts.

We have a relatively small sample size for human evaluation, because the
expert manually checks the correctness of each example with Internet searches,
which is labor-intensive. However, we acknowledge that a larger sample size
would increase the statistical confidence of our evaluation.

Finally, we note that a number of Wikipedia tables are not well-formatted
after preprocessing and linearization. Since Wikipedia tables are embedded as
HTML elements that allow for idiosyncrasies like a table with one cell
spanning multiple columns or color-highlighted cells, some are hard to parse
correctly. Solving such edge cases engineering-wise would further improve
TableQA.

## Ethical Considerations

To facilitate reproducibility and continued research, we will make the code
available upon publication.

No new datasets were gathered specifically for this study, and we did not
employ crowd-sourced labor. We use Wikipedia data under the terms of the
Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA)
and the GNU Free Documentation License (GFDL). Wikidata is under Creative
Commons CC0 License, which is equivalent to public domain. The CompMix
benchmark is licensed under a Creative Commons Attribution 4.0 International
License. We use the benchmark as it is intended.

The experimental phase involved approximately 80 hours of computation time on
an NVIDIA A100 GPU to fine-tune the retrieval model and index Wikipedia
content. We reused the LLaMA-7B model trained in prior work, thus avoiding
extra GPU usage.

We do not anticipate adverse effects stemming from the proposed methods in
this study.

## References

  * Asai et al. (2022) Akari Asai, Matt Gardner, and Hannaneh Hajishirzi. 2022.  Evidentiality-guided generation for knowledge-intensive NLP tasks.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2226–2243, Seattle, United States. Association for Computational Linguistics.
  * Asai et al. (2020) Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020.  Learning to retrieve reasoning paths over wikipedia graph for question answering.  In _International Conference on Learning Representations_.
  * Ayoola et al. (2022) Tom Ayoola, Shubhi Tyagi, Joseph Fisher, Christos Christodoulopoulos, and Andrea Pierleoni. 2022.  ReFinED: An efficient zero-shot-capable approach to end-to-end entity linking.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track_ , pages 209–220, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
  * Bulian et al. (2022) Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Börschinger, and Tal Schuster. 2022.  Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 291–305, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Chen et al. (2021) Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021.  Open question answering over tables and text.  In _International Conference on Learning Representations_.
  * Chen et al. (2020) Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020.  HybridQA: A dataset of multi-hop question answering over tabular and textual data.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1026–1036, Online. Association for Computational Linguistics.
  * Christmann et al. (2023a) Philipp Christmann, Rishiraj Saha Roy, and Gerhard Weikum. 2023a.  Compmix: A benchmark for heterogeneous question answering.
  * Christmann et al. (2022) Philipp Christmann, Rishiraj Saha Roy, and Gerhard Weikum. 2022.  Conversational question answering on heterogeneous sources.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’22, page 144–154, New York, NY, USA. Association for Computing Machinery.
  * Christmann et al. (2023b) Philipp Christmann, Rishiraj Saha Roy, and Gerhard Weikum. 2023b.  Explainable conversational question answering over heterogeneous sources via iterative graph neural networks.  In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’23, page 643–653, New York, NY, USA. Association for Computing Machinery.
  * Gao et al. (2023) Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023.  Enabling large language models to generate text with citations.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6465–6488, Singapore. Association for Computational Linguistics.
  * Han and Gardent (2023) Kelvin Han and Claire Gardent. 2023.  Generating and answering simple and complex questions from text and from knowledge graphs.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 285–304, Nusa Dua, Bali. Association for Computational Linguistics.
  * Herzig et al. (2021) Jonathan Herzig, Thomas Müller, Syrine Krichene, and Julian Eisenschlos. 2021.  Open domain question answering over tables via dense retrieval.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 512–519, Online. Association for Computational Linguistics.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Jiang et al. (2023) Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023.  Active retrieval augmented generation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7969–7992, Singapore. Association for Computational Linguistics.
  * Kamalloo et al. (2023) Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. 2023.  Evaluating open-domain question answering in the era of large language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5591–5606, Toronto, Canada. Association for Computational Linguistics.
  * Khattab et al. (2021) Omar Khattab, Christopher Potts, and Matei Zaharia. 2021.  Relevance-guided supervision for OpenQA with ColBERT.  _Transactions of the Association for Computational Linguistics_ , 9:929–944.
  * Khattab et al. (2023) Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2023.  Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
  * Lehmann et al. (2024) Jens Lehmann, Dhananjay Bhandiwad, Preetam Gattogi, and Sahar Vahdati. 2024.  Beyond boundaries: A human-like approach for question answering over structured and unstructured information sources.  _Transactions of the Association for Computational Linguistics (TACL)_.
  * Lei et al. (2023) Fangyu Lei, Xiang Li, Yifan Wei, Shizhu He, Yiming Huang, Jun Zhao, and Kang Liu. 2023.  S3HQA: A three-stage approach for multi-hop text-table hybrid question answering.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1731–1740, Toronto, Canada. Association for Computational Linguistics.
  * Li et al. (2023) Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su, and Wenhu Chen. 2023.  Few-shot in-context learning on knowledge base question answering.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6966–6980, Toronto, Canada. Association for Computational Linguistics.
  * Liu et al. (2023) Shicheng Liu, Jialiang Xu, Wesley Tjangnaka, Sina J. Semnani, Chen Jie Yu, Gui Dávid, and Monica S. Lam. 2023.  SUQL: Conversational search over structured and unstructured data with large language models.
  * Lu et al. (2024) Weizheng Lu, Jiaming Zhang, Jing Zhang, and Yueguo Chen. 2024.  Large language model for table processing: A survey.
  * Luo et al. (2023) Haoran Luo, Haihong E, Zichen Tang, Shiyao Peng, Yikai Guo, Wentai Zhang, Chenghao Ma, Guanting Dong, Meina Song, and Wei Lin. 2023.  Chatkbqa: A generate-then-retrieve framework for knowledge base question answering with fine-tuned large language models.
  * Ma et al. (2022a) Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2022a.  Open-domain question answering via chain of reasoning over heterogeneous knowledge.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 5360–5374, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ma et al. (2022b) Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2022b.  Open domain question answering with a unified knowledge interface.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1605–1620, Dublin, Ireland. Association for Computational Linguistics.
  * Ma et al. (2023) Kaixin Ma, Hao Cheng, Yu Zhang, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2023.  Chain-of-skills: A configurable model for open-domain question answering.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1599–1618, Toronto, Canada. Association for Computational Linguistics.
  * Miller et al. (2016) Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016.  Key-value memory networks for directly reading documents.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1400–1409, Austin, Texas. Association for Computational Linguistics.
  * Nan et al. (2023) Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. 2023.  Enhancing text-to-SQL capabilities of large language models: A study on prompt design strategies.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 14935–14956, Singapore. Association for Computational Linguistics.
  * Oguz et al. (2022) Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2022.  UniK-QA: Unified representations of structured and unstructured knowledge for open-domain question answering.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 1535–1546, Seattle, United States. Association for Computational Linguistics.
  * Pan et al. (2024) Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2024.  Unifying large language models and knowledge graphs: A roadmap.  _IEEE Transactions on Knowledge and Data Engineering_ , page 1–20.
  * Pourreza and Rafiei (2023) Mohammadreza Pourreza and Davood Rafiei. 2023.  Din-sql: Decomposed in-context learning of text-to-sql with self-correction.
  * Pramanik et al. (2021) Soumajit Pramanik, Jesujoba Alabi, Rishiraj Saha Roy, and Gerhard Weikum. 2021.  UNIQORN: unified question answering over RDF knowledge graphs and natural language text.  _CoRR_ , abs/2108.08614.
  * Santhanam et al. (2022) Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022.  ColBERTv2: Effective and efficient retrieval via lightweight late interaction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3715–3734, Seattle, United States. Association for Computational Linguistics.
  * Semnani et al. (2023) Sina Semnani, Violet Yao, Heidi Zhang, and Monica Lam. 2023.  WikiChat: Stopping the hallucination of large language model chatbots by few-shot grounding on Wikipedia.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 2387–2413, Singapore. Association for Computational Linguistics.
  * Usbeck et al. (2017) Ricardo Usbeck, Axel-Cyrille Ngonga Ngomo, Bastian Haarmann, Anastasia Krithara, Michael Röder, and Giulio Napolitano. 2017.  7th open challenge on question answering over linked data (qald-7).  In _Semantic web evaluation challenge_ , pages 59–69. Springer.
  * Wang et al. (2018) Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018.  R^3: Reinforced ranker-reader for open-domain question answering.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1).
  * Xu et al. (2023) Silei Xu, Shicheng Liu, Theo Culhane, Elizaveta Pertseva, Meng-Hsi Wu, Sina Semnani, and Monica Lam. 2023.  Fine-tuned LLMs know more, hallucinate less with few-shot sequence-to-sequence semantic parsing over Wikidata.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5778–5791, Singapore. Association for Computational Linguistics.
  * Zhang et al. (2023) Yunjia Zhang, Jordan Henkel, Avrilia Floratou, Joyce Cahoon, Shaleen Deep, and Jignesh M. Patel. 2023.  Reactable: Enhancing react for table question answering.
  * Zhao et al. (2023a) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023a.  A survey of large language models.
  * Zhao et al. (2023b) Wenting Zhao, Ye Liu, Tong Niu, Yao Wan, Philip S. Yu, Shafiq Joty, Yingbo Zhou, and Semih Yavuz. 2023b.  Divknowqa: Assessing the reasoning ability of llms via open-domain question answering over knowledge base and text.

##  Appendix A Wikidata Experiments

Xu et al. (2023) fine-tuned two LLaMAs on Wikidata. The training data for the
first model consists solely of WikiWebQuestions (Xu et al., 2023), while the
other consists of the combination of WikiWebQuestions (Xu et al., 2023) and
QALD-7 (Usbeck et al., 2017). We experiment with both models on the
development set of CompMix, each with (1) entities predicted by ReFinED, (2)
our entity linking approach with GPT-3.5 as the LLM (prompt in Figure 6), and
(3) the dataset-provided oracle entities.

| Dev | Dev (KB subset)
---|---|---
| EM | Superset | EM | Superset
WikiWebQuestions Only |  |  |  |
w/ ReFinED only entities | 29.1 | 36.5 | 39.8 | 46.0
w/ ReFinED $+$ GPT-3.5 entities | 31.3 | 38.8 | 43.8 | 50.4
w/ oracle entities | 33.9 | 42.3 | 46.2 | 52.8
WikiWebQuestions $+$ Qald-7 |  |  |  |
w/ ReFinED only entities | 29.5 | 36.6 | 41.0 | 47.6
w/ ReFinED $+$ GPT-3.5 entities | 32.9 | 40.4 | 46.8 | 53.0
w/ oracle entities | 35.5 | 43.1 | 49.0 | 55.4
Table 3: Wikidata semantic parsing experiment results on the CompMix
development set. Comparison is made using entity IDs. Superset measures
whether the model’s predicted entities is a superset of the gold entities. Dev
(KB subset) refers to the subset of the dataset where the annotators located
the annotated answer from Wikidata.

As shown in Table 3, the model using entities predicted by our approach
outperforms the model using the baseline ReFinED entities. It achieves
considerably closer performance with the model using oracle entities. We also
observed that the the model trained on both WikiWebQuestions and QALD-7
outperforms the model trained on WikiWebQuestions only.

##  Appendix B Details on Platinum Evaluation

Figure 3 shows the distribution of cases that we resolve using more advanced
evaluation metrics. Numbers are reported on the first 100 dev examples with
Spaghetti (GPT-4).

Examples of each evaluation error type can be found at Figure 12, Figure 13,
Figure 14, Figure 15, and Figure 16.

##  Appendix C Error Analysis

We include the five error cases after platinum evaluation in Figure 7, Figure
8, Figure 9, Figure 10, and Figure 11.

###  C.1 Conflicting or Misleading Evidence

We analyze the 388 error cases from Spaghetti (GPT-3.5) as determined by GPT-4
Matching. We separate out evidence retrieval errors from answer generation
errors by identifying how often the gold answer appears in the evidences using
a substring matching heuristic (Table 4).

| # Error Cases
---|---
All Error Cases | 388 (100%)
Gold in Evidence | 154 (39.69%)
Gold in KB | 51 (13.14%)
Gold in Text | 87 (22.42%)
Gold in Tables | 72 (18.56%)

Table 4: Numbers of error cases by category. The notation “Gold in [source]”
stands for the gold answer existing as a substring in the particular [source].

In 154 out of all 388 error cases, the system does not produce the gold answer
despite the successful retrieval of evidence containing it. This observation
indicates that a significant portion of the error cases are due to conflicting
or misleading information in the evidence, where further improvements in
selecting and merging evidences would be helpful. In the majority of the error
cases (234 out of 388) where gold is not in the evidence, the system has no
high-quality candidates to select from. Note, however, that this is an
overestimate, due to the use of substring matching for deciding whether an
evidence is correct or not.

In the breakdown of gold answer sources, the source that contains the most
gold answers is Text (87 out of 154 cases), and Wikidata contains the least
gold answers (51 out of 154 cases).

###  C.2 Combiner Hallucination

We investigate the ratio of generated answers that were hallucinated by our
model. We manually checked the first 300 cases in our evaluation set and found
2 cases (0.67%) where the model ignored the evidence and hallucinated an
incorrect answer. This low ratio of hallucination highlights the faithfulness
of our system to the evidence retrieved. We include these cases in Figure 4
and Figure 5.

⬇

1{

2 "idx": 64,

3 "correct_hallucination": true,

4 "question": "Nirvana was founded by who?",

5 "gold": "Kurt Cobain",

6 "answer_generated": "Kurt Cobain, Krist Novoselic, and Dave Grohl",

7 "gold_sources": [

8 "TEXT"

9 ],

10 "pred_sources": [],

11 "evidences": [

12 [

13 "KB",

14 "Wikidata says the answer to \"Nirvana was founded by who?\" is: ."

15 ],

16 [

17 "TEXT",

18 "Tan Sri Kong Hon Kong is the founder of Nirvana Asia Group, the largest
integrated funeral service provider in Malaysia."

19 ],

20 [

21 "TEXT",

22 "Nirvana was founded by lead singer and guitarist Kurt Cobain and bassist
Krist Novoselic in 1987."

23 ],

24 [

25 "TEXT",

26 "The founder of Buddhism, the Buddha, is believed to have reached both
states of \"abiding\" and \"non-abiding nirvana\"."

27 ],

28 [

29 "TEXT",

30 "Kurt Cobain was the co-founder of the rock band Nirvana, along with Krist
Novoselic and Aaron Burckhard."

31 ],

32 [

33 "TABLE",

34 "NIRVANAnet was founded in 1989."

35 ]

36 ]

37}

Figure 4: Example of a refinement hallucination case (Spaghetti (GPT-3.5)).
“Dave Grohl” is completely hallucinated.

⬇

1{

2 "idx": 249,

3 "correct_hallucination": false,

4 "question": "What is the voice type of the Bob Dylan?",

5 "gold": "baritone",

6 "answer_generated": "gravelly or nasal",

7 "gold_sources": [

8 "KB"

9 ],

10 "pred_sources": [],

11 "evidences": [

12 [

13 "KB",

14 "Wikidata says the answer to \"What is the voice type of the Bob Dylan?\"
is: baritone."

15 ],

16 [

17 "TEXT",

18 "Bob Dylan’s voice has been described as \"young and jeeringly cynical\"
and \"broken\" as he aged."

19 ],

20 [

21 "TEXT",

22 "Bob Dylan’s voice has received critical attention, with some describing it
as \"a rusty voice\" and others comparing it to \"sand and glue\"."

23 ]

24 ]

25}

Figure 5: Example of a refinement hallucination case

⬇

1messages = [

2 {"role": "system", "content": "You are a helpful assistant."},

3 {"role": "user",

4 "content": "You are a named entity recognition and entity disambiguation
system. You are given a question and you need to list all entities in the
question with a brief description for each entity. Each description should be
max 10 words. Here are some examples:

5

6 Question: what year lebron james came to the nba?

7 Answer:

8 1. LeBron James is American basketball player (born 1984)

9 2. National Basketball Association is North American professional sports
league

10

11 Question: what form of government was practiced in sparta?

12 Answer:

13 1. Sparta is city-state in ancient Greece

14

15 Question: What is the genre of the tv series High Seas?

16 Answer:

17 1. High Seas is a Spanish television series

18

19 Question: Which country did the TV series Coupling originate?

20 Answer:

21 1. Coupling is a British television series (2000-2004)

22

23 Question: What year was M.O.V.E first formed?

24 Answer:

25 1. M.O.V.E is a Japanese musical group

26

27 Question: What year was the inception of the soccer club Manchester United
F.C.?

28 Answer:

29 1. Manchester United F.C. is association football club in Manchester,
England

30

31 Question: What is Russell Crowe’s date of birth?

32 Answer:

33 1. Russell Crowe is New Zealand-born actor (born 1964)

34

35 Question: what character did natalie portman play in star wars?

36 Answer:

37 1. natalie portman is Israeli-American actress and filmmaker

38 2. star wars is epic space opera multimedia franchise created by George
Lucas

39

40 Question: what country is the grand bahama island in?

41 Answer:

42 1. Grand Bahama is island of the Bahamas

43

44 Question: where are the nfl redskins from?

45 Answer:

46 1. Washington Commanders or Washington Redskins is American football team
in the National Football League

47

48 Question: what time zone am i in cleveland ohio?

49 Answer:

50 1. Cleveland is city in and county seat of Cuyahoga County, Ohio, United
States

51

52 Question: who is the prime minister of ethiopia?

53 Answer:

54 1. Ethiopia is country in the Horn of Africa},

55

56 {"role": "user",

57 "content": "List the entities and their descriptions for this question:

58 Question: {question}

59 Answer:"}

Figure 6: A shortened version of the prompt for GPT-3.5 to detect entity
mentions and generate a description for each detected entity, as discussed in
Section 3.1. The descriptions in the prompt are taken from the Wikidata
description for detected entities. The actual prompt contains 13 more
examples. The examples in the prompt are chosen to capture the diversity of
domains and to instruct GPT-3.5 to detect more generic entities too.

⬇

1{

2 "question": "Who won the Oscars for the best actress in 1952?",

3 "gold": "Vivien Leigh",

4 "answer_generated": "Shirley Booth",

5 "evidences": [

6 [

7 "KB",

8 "Wikidata says the answer to \"Who won the Oscars for the best actress in
1952?\" is: ."

9 ],

10 [

11 "TEXT",

12 "Shirley Booth won the Academy Award for Best Actress in 1952."

13 ]

14 ]

15}

Figure 7: A failure case after platinum evaluation. In this case, the intent
of the question is ambiguous. The gold answer is Vivien Leigh, who won the
24th Academy Awards (held on March 20, 1952, honoring the films of 1951), and
Spaghetti predicts Shirley Booth, who won The 25th Academy Awards (held on
March 19, 1953, honoring the films of 1952).

⬇

1{

2 "question": "Has Ericson Core started Career as music video director?",

3 "gold": "Yes",

4 "answer_generated": "No",

5 "evidences": [

6 [

7 "KB",

8 "Wikidata says the answer to \"Has Ericson Core started Career as music
video director?\" is: no."

9 ],

10 [

11 "TEXT",

12 "Ericson Core started his career as a music video director."

13 ]

14 ]

15}

Figure 8: A failure case after platinum evaluation. In this case, the Wikidata
is giving an incorrect answer, due to semantic parsing errors.

⬇

1{

2 "question": "Release year of the first Francisco de Robles book?",

3 "gold": "1605",

4 "answer_generated": "1585",

5 "evidences": [

6 [

7 "KB",

8 "Wikidata says the answer to \"Release year of the first Francisco de Robles
book?\" is: ."

9 ],

10 [

11 "TEXT",

12 "Among the books published by Francisco de Robles, the first edition of
\"Don Quixote\" was released in 1605."

13 ],

14 [

15 "TABLE",

16 "The first Francisco de Robles book, La Galatea, was released in the year
1585."

17 ]

18 ]

19}

Figure 9: A failure case after platinum evaluation. The first book of
Francisco de Robles is Don Quixote released in 1605. The book La Galatea,
released in 1585, was published by Blas de Robles, father of Francisco de
Robles. However, the entry of Blas de Robles, listed as the publisher in the
Wikipedia infobox of La Galatea, erroneously contains a hyperlink directing to
the page of Francisco de Robles. This discrepancy led to a misinterpretation
by Spaghetti, resulting in the incorrect identification of Francisco de Robles
as the publisher of La Galatea.

⬇

1{

2 "question": "Is the player number of Bebe is 10?",

3 "gold": "Yes",

4 "answer_generated": "No",

5 "evidences": [

6 [

7 "KB",

8 "Wikidata says the answer to \"Is the player number of Bebe is 10?\" is: ."

9 ],

10 [

11 "TABLE",

12 "The player number of Bebe is 22."

13 ]

14 ]

15}

Figure 10: A failure case after platinum evaluation. The player name, Bebé,
was not correctly grounded in Wikidata, which resulted in an empty response.
Table retriever retrieved information about a basketball player named Bebo
instead of the football player Bebé.

⬇

1{

2 "question": "Which island is home to Alyssa Cole’s primary residence?",

3 "gold": "Martinique",

4 "answer_generated": "Information not available",

5 "evidences": [

6 [

7 "KB",

8 "Wikidata says the answer to \"Which island is home to Alyssa Cole’s primary
residence?\" is: ."

9 ]

10 ]

11}

Figure 11: A failure case after platinum evaluation due to no retrieved
evidence.

⬇

1{

2 "question": "What is the original title of the novel The Alchemist?",

3 "gold": "O Alquimista",

4 "answer_generated": "\"O Alquimista\"",

5}

Figure 12: Example where EM cannot handle correctly (format).

⬇

1{

2 "question": "Nirvana was founded by who?",

3 "gold": "Kurt Cobain",

4 "answer_generated": "Kurt Cobain and Krist Novoselic",

5}

Figure 13: Example where EM cannot handle correctly (superset).

⬇

1{

2 "question": "What was Elton John’s debut album?",

3 "gold": "Goodbye Yellow Brick Road",

4 "answer_generated": "Empty Sky",

5}

Figure 14: Example where EM cannot handle correctly (gold answer wrong).
“Empty Sky” is the correct answer here.

⬇

1{

2 "question": "What is the main cast name in the tv series Tribes of Europa?",

3 "gold": "Emilio Sakraya",

4 "answer_generated": "Henriette Confurius, Emilio Sakraya, and David Ali
Rashed",

5}

Figure 15: Example where EM cannot handle correctly (multiple correct
answers).

⬇

1{

2 "question": "Who was the music of the movie \"The Social Network\"?",

3 "gold": "Trent Reznor Atticus Ross",

4 "answer_generated": "Trent Reznor and Atticus Ross",

5}

Figure 16: Example where EM cannot handle correctly (paraphrasing).