# Ultra diffuse galaxies in the Hydra I cluster from the LEWIS Project: Phase-
Space distribution and globular cluster richness

Duncan A. Forbes, 1 Jonah Gannon1, Enrichetta Iodice2, Michael Hilker5,Goran
Doll2,3, Chiara Buttitta2, Antonio La Marca6,7, Magda Arnaboldi5, Michele
Cantiello4, G. D’Ago8, Jesus Falcon Barroso15,16, Laura Greggio9, Marco
Gullieuszik9, Johanna Hartke12,13, Steffen Mieske10, Marco Mirabile4,14,
Roberto Rampazzo9, Marina Rejkuba5, Marilena Spavone2, Chiara Spiniello11,
Giulio Capasso2
1 Centre for Astrophysics & Supercomputing, Swinburne University, Hawthorn,
VIC 3122, Australia
2 INAF - Astronomical Observatory of Capodimonte, Salita Moiariello 16,
I-80131, Naples, Italy
3 University of Naples “Federico II”, C.U. Monte Sant’Angelo, Via Cinthia,
80126, Naples, Italy
4 INAF - Astronomical Observatory of Abruzzo, Via Maggini, 64100, Teramo,
Italy
5 European Southern Observatory, Karl-Schwarzschild-Strasse 2, 85748 Garching
bei Muenchen, Germany
6 SRON Netherlands Institute for Space Research, Landleven 12, 9747 AD
Groningen, The Netherlands
7 Kapteyn Astronomical Institute, University of Groningen, Postbus 800, 9700
AV Groningen, The Netherlands
8 Institute of Astronomy, University of Cambridge, Madingley Road, Cambridge,
CB3 0HA
9 Osservatorio Astronomico di Padova, Via dell’Osservatorio 8, 36012 Asiago
(VI), Italy
10 European Southern Observatory, Alonso de Cordova 3107, 7630355 Vitacura,
Santiago, Chile
11 Sub-department of Astrophysics, University of Oxford, Denys Wilkinson
Building, Keble Road, Oxford OX1 3RH, United Kingdom
12 Finnish Centre for Astronomy with ESO (FINCA), FI-20014 University of
Turku, Finland
13 Tuorla Observatory, Department of Physics and Astronomy, FI-20014
University of Turku, Finland
14 Gran Sasso Science Institute, L’Aquila, Italy
15 INAF - Astronomical Observatory of Padova, Vicolo dell’Osservatorio 5,
I-35122 Padova, Italy
15 Instituto de Astrofísica de Canarias, Vía Láctea s/n, E-38205 La Laguna,
Tenerife, Spain
16 Departamento de Astrofísica, Universidad de La Laguna, E-38200 La Laguna,
Tenerife, Spain E-mail<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

Although ultra diffuse galaxies (UDGs) are found in large numbers in clusters
of galaxies, the role of the cluster environment in shaping their low surface
brightness and large sizes is still uncertain. Here we examine a sample of
UDGs in the Hydra I cluster (D = 51 Mpc) with new radial velocities obtained
as part of the LEWIS (Looking into the faintest with MUSE) project using
VLT/MUSE data. Using a phase-space, or infall diagnostic, diagram we compare
the UDGs to other known galaxies in the Hydra I cluster and to UDGs in other
clusters. The UDGs, along with the bulk of regular Hydra I galaxies, have low
relative velocities and are located near the cluster core, and thus consistent
with very early infall into the cluster. Combining with literature data, we do
not find the expected trend of GC-rich UDGs associated with earlier infall
times. This result suggests that quenching mechanisms other than cluster
infall should be further considered, e.g. quenching by strong feedback or in
cosmic sheets and filaments. Tidal stripping of GCs in the cluster environment
also warrants further modelling.

###### keywords:

galaxies: star clusters: general — galaxies: haloes — galaxies: structure —
galaxies: photometry

††pubyear: 2023††pagerange: Ultra diffuse galaxies in the Hydra I cluster from
the LEWIS Project: Phase-Space distribution and globular cluster
richness–References

##  1 Introduction

The possible formation pathways of ultra-diffuse galaxies (UDGs) have been a
subject of an ongoing vigorous debate since 2015, when a population of these
extremely diffuse galaxies was identified in the Coma cluster using the
Dragonfly Telephoto Array (van Dokkum et al., 2015). Existing in all
environments, they are most common in clusters with several hundred found in
the Coma cluster (Yagi et al., 2016; Alabi et al., 2018). This significant
contribution to our ‘census of galaxies’ has prompted numerous simulation
studies and accompanying predictions (see Sales et al. (2020) and references
therein). These simulations can be broadly placed in two categories; internal
processes (e.g. episodic supernova feedback) or external (e.g. tidal effects
in a dense environment). Some combination of both processes may be operating
along with past galaxy infall (and subsequent quenching) into clusters.

UDGs have low surface brightnesses (they are defined to have central values in
the g band of $\mu$ $>$ 24 mag. per sq. arcsec) so that spectroscopic studies
of them push even 8–10m class telescopes, with efficient low surface
brightness instruments, such as KWCI on Keck or MUSE on VLT, to their limits.
While strictly speaking dwarf galaxies with M∗ $<10^{9}$ M⊙, UDGs are unlike
classical dwarfs as they have extreme sizes with effective radii Re $>$ 1.5
kpc (i.e. comparable to the disk of the Milky Way with Re $\sim$3.5 kpc). They
also reveal another unexplained feature, with some hosting up to ten times
more globular clusters (GCs) than classical dwarf galaxies of the same
luminosity (Forbes et al., 2020). Their very existence in clusters and their
generally old stellar populations suggests that some may be protected within
an overly massive dark matter halo. The latter is supported by the correlation
between GC numbers and host galaxy halo mass for normal galaxies, e.g. Burkert
& Forbes (2020).

In the standard picture of dwarf galaxy evolution (Mistani et al., 2016),
dwarfs that fell into clusters at early times will have experienced intense
star formation, prior to, or at the start of, infall (which is also expected
to give rise to a high fraction of stars in bound star clusters). This is
followed by quenching of any further star formation as the infall proceeds.
Both of these effects would lead to a high fraction of GCs relative to their
field stars (Mistani et al., 2016; Ramos-Almendares et al., 2020). Indeed
trends of GC richness and [$\alpha$/Fe] ratios with clustercentric radius
provide some observational support for this interpretation (Peng et al., 2008;
Liu et al., 2016; Romero-Gómez et al., 2023). This early-infall, or biasing,
has been invoked for UDGs by Carleton et al. (2021) who include cluster tidal
effects within the IllustrisTNG simulation and simplified GC formation
physics. Similar to classical dwarfs, they predict that early-infall UDGs
should be rich in GCs. Based on a semi-empirical model, Trujillo-Gomez et al.
(2022) also predict that galaxies near the cluster core form more GCs.

Using phase-space, or infall diagnostic, diagrams of the type proposed by Rhee
et al. (2017) one can investigate whether GC richness depends on UDG cluster
infall time. No trend between GC richness and very early infall times might
suggest that GC formation and quenching occurred before cluster infall. While
low mass galaxies typically quench at late times, there is a considerable
range in quenching times with some low mass galaxies quenching at z $\sim$ 2
or 10.5 Gyr ago (Moster et al., 2020). Quenching at early times via stellar
feedback (Stinson et al., 2013) may be one possibility. This early quenching
applied to UDGs has been described by Danieli et al. (2022). Another
possibility may be quenching via the interaction with cosmic sheets or
filaments (Pasha et al., 2023). A first attempt at this sort of infall
analysis applied to GCs was presented in Gannon et al. (2022) for several UDGs
in the Coma and Perseus clusters. No clear signal was found but the sample was
small with just over a dozen UDGs and with a bias towards GC-rich UDGs.

In this Letter we examine the infall diagnostic diagram for a new sample of
UDGs in the Hydra I cluster (A1060; D = 51 $\pm$ 4 Mpc). The Hydra I cluster
appears to be fairly dynamically relaxed (Ventimiglia et al., 2011) but also
reveals hints of substructures (Lima-Dias et al., 2021; La Marca et al.,
2022a), an infalling group of galaxies (Arnaboldi et al., 2012), and evidence
for ram pressure stripping (Wang et al., 2021; Iodice et al., 2021). The
observed UDGs are located near the cluster core and the northern subgroup,
with all lying within the 0.3 virial radii (R200) of the Hydra I cluster
centre. Each was observed using MUSE on the VLT as part of the ongoing LEWIS
(Looking into the faintEst WIth MUSE) project. Details of the project,
including galaxy radial velocities, positions, GC counts etc, are given in
Paper I by Iodice et al. (2023, in press). The GC counts are based on deep,
optical multi-filter imaging with the VST as part of the VEGAS project (Iodice
et al., 2020) and will be updated after the full analysis of the MUSE data.
Here we explore the distribution of UDGs in phase space and investigate
whether they reveal any trend in this space with their GC richness. We also
include similar data for UDGs in other nearby clusters. For the Hydra I
cluster we adopt the same parameters as used by La Marca et al. (2022b), i.e.
cz = 3683 $\pm$ 46 km s-1, $\sigma$ = 724 $\pm$ 31 km s-1, and virial
parameters R200 = 1.6 Mpc, M200 = 2 $\times$ 1014 h-1 M⊙ and take its centre
as NGC 3311 (RA = 159.17842, Dec = –27.528339). These values are similar to
those found by Lima-Dias et al. (2021) who recently studied Hydra I galaxies
out to the virial radius.

Figure 1: Infall diagnostic diagram for non-UDG (giants, dwarfs and LSB
galaxies) and UDGs in the Hydra I cluster. The diagram shows the relative
line-of-sight velocity of each galaxy normalised by the cluster velocity
dispersion against the projected radius normalised by the virial radius.
Regions of the diagram are shaded according to their infall times from the
cosmological simulations of Rhee et al. (2017) as indicated in the legend. The
plot shows that most UDGs and non-UDG galaxies of the the Hydra I cluster lie
within the very early infall zone – the simulations of indicating that around
half of the galaxies in this zone were part of the cluster at least 6.45 Gyr
ago.

##  2 Infall Diagnostic Diagram for Hydra I Cluster Galaxies

Rhee et al. (2017) carried out cosmological simulations of several galaxy
clusters and examined the resulting distribution of galaxies in phase-space
(i.e. velocity of the galaxy relative to the mean cluster velocity normalised
by the cluster velocity dispersion versus the galaxy’s clustercentric radius
normalised by the cluster virial radius). Based on the infall time of
galaxies, they divided this diagram into several infall zones, ranging from
those that fell into the cluster at very early times, to those that are yet to
fall in. Thus the location of galaxies in this diagram provides an ‘infall
diagnostic’ which is statistical in nature and additional scatter is
introduced when using 2D projected radii (as is the case for observational
data). For example, the ‘very early infall’ (or ancient infaller) zone in the
simulation is occupied by a slight majority (52%) of galaxies that have
resided in the cluster for more than 6.45 Gyr. Projection effects mean that
the true clustercentric radius for some galaxies is larger in 3D than observed
in 2D. For most galaxies this effect should be less a factor of two from the
projected one.

In Fig. 1 we show such an infall diagnostic diagram for all galaxies in the
Hydra I cluster out to half the virial radius. This includes giant and dwarf
galaxies from the study of Christlein & Zabludoff (2003) plus the addition of
UDGs and 3 low surface brightness (LSB) galaxies that have UDG-like sizes but
are slightly brighter from Iodice et al. (2023, in press). We find that the
bulk of the non-UDG Hydra I galaxies are located within the ‘very early
infall’ zone. The simulation of Rhee et al. (2017) predicts that just over
half of these would have been part of the cluster for at least 6.45 Gyr. There
are also galaxies located in later infall zones and three galaxies that may
lie outside of the cluster with large relative velocities – these could be
backsplash galaxies (having passed through the cluster) or simply galaxies
that are yet to fall into the cluster.

If we examine giant and classical dwarf galaxies separately (divided at MR =
–18 or mR = 15.5) there is no clear difference between them in terms of their
infall properties. Compared to the UDGs they appear to scatter to higher
relative velocities on average. A more quantitative measure of the differences
in their infall properties can be obtained from the product of their relative
velocity from the cluster mean and their radial position: $\Delta$V/$\sigma$ x
R/R200. Restricting to 0.3R/R200, as probed by the imaging, we find mean
values (and error on the mean) of $\Delta$V/$\sigma$ x R/R200 = 0.83 ($\pm$
0.07) $\times$ 0.15 ($\pm$ 0.01) = 0.12 ($\pm$ 0.02) for giant galaxies and
0.88 ($\pm$ 0.06) $\times$ 0.16 ($\pm$ 0.01) = 0.14 ($\pm$ 0.02) for classical
dwarfs. For the UDGs the mean value is $\Delta$V/$\sigma$ x R/R200 = 0.80
($\pm$ 0.17) $\times$ 0.16 ($\pm$ 0.02) = 0.13 ($\pm$ 0.04). This indicates
that UDGs are similarly concentrated in phase-space to the other cluster
galaxies. Also, while UDGs have a similar distribution in clustercentric
radius, their velocities are closer to the cluster mean than either giants or
classical dwarfs. We note that Lima-Dias et al. (2021) also found passive
early-type galaxies to be concentrated in the cluster core.The LSB galaxies in
Fig. 1 are found in a range of infall zones, from early to late infall.

As might be expected from their inner cluster position, our UDGs were among
the earliest inhabitants of the cluster, infalling at least 6.45 Gyr ago
according to simulations of Rhee et al. (2017). They would be expected to have
star formation (SF) histories that indicate early quenching. A preliminary
analysis by Iodice et al. (2023, submitted) for one UDG (UDG11) indicates an
old age of $\sim$10 Gyr, suggestive of early quenching. Future analysis will
also include the [$\alpha$/Fe] ratios which appears to be a sensitive
indicators of SF histories for low mass galaxies (see Ferre-Mateu et al. 2023,
submitted for results on UDGs in other clusters and Romero-Gómez et al.
(2023), for dwarf galaxies in the Fornax cluster). We note that the study of
Lima-Dias et al. (2021) found 88% of Hydra I galaxies (with log M∗ $>$ 8.5) to
be quenched, i.e. no sign of ongoing star formation.

Figure 2: Infall diagnostic diagram for only UDGs in the Hydra I, Coma, Virgo
and Perseus clusters. Regions of the diagram are shaded according to their
infall times from the cosmological simulations of Rhee et al. (2017). As per
the legend, UDGs in different clusters are denoted by different symbols.
Symbols are outlined in red (if GC-rich) or blue (if GC-poor), and without an
outline if the GC properties are unknown. See main text for discussion of
selection effects in the UDG samples. Globular cluster (GC) rich UDGs are not
predominately found in the very early infall region, indeed the data suggest
that very early infall UDGs tend to be GC-poor.

##  3 Infall Diagnostic Diagram for UDGs in Several Clusters

In Fig. 2 we show the UDGs from the Hydra I cluster along with those from the
literature and coded by globular cluster (GC) richness. Total GC counts for
the Hydra I UDGs are determined in (Iodice et al., 2020; La Marca et al.,
2022b) and listed again in Paper I (Iodice et al. 2023, submitted). Literature
data comes from Gannon et al. (2022) and the recent work of Toloba et al.
(2023). The GC counts are almost exclusively based on imaging (i.e. lacking
radial velocities) and we follow Gannon et al. (2022) assigning a somewhat
arbitrary separation between rich and poor GC systems at 20 GCs. This
corresponds to a halo mass of 1011 M⊙ using the scaling relation of Burkert &
Forbes (2020). Below 20 GCs the scaling relation is less predictive of halo
mass due to increased scatter. By this definition, all of the UDGs in the
Hydra I cluster are GC-poor (ranging from no GCs for several UDGs to 15 GCs
for UDG3) and this is unlikely to change significantly when the full set of
MUSE spectroscopic data is available. Given the relatively small stellar mass
range of the Hydra I UDGs, a fixed GC number corresponds closely to a GC
system total mass per host galaxy stellar mass. If we assume the same average
mass for a GC of 105 M⊙, this ratio is $<$1.2% for all of the observed Hydra I
UDGs. While some Coma cluster UDGs also have a ratio $<$1.2% the majority have
much higher ratios, with up to $\sim$10% of the galaxy stellar mass in their
GC system, see figure 4 of Forbes et al. (2020).

Before interpreting Fig. 2 there are various caveats and selection effects
that should be born in mind. Firstly, we note that some of the literature UDGs
lack firm GC counts and their rich/poor status is on the basis of a visual
estimate only (Gannon et al., 2022). Secondly, the literature sample is
subject to sample selection effects. The Coma cluster sample of UDGs comes
from studies that have focused on GC-rich galaxies or they have focused on a
narrow range in clustercentric radius (i.e. around 0.12 R/R200 in the Coma
cluster). Observations of the Perseus cluster UDGs have so far avoided the
cluster inner regions. The Virgo UDG sample is relatively small and mostly GC-
poor. In terms of a selection bias, the Hydra I UDGs are the closest to being
a representative sample of UDGs in the cluster, however only the inner 0.3
R/R200 was imaged in Iodice et al. (2020). Thus, we may be missing the late
infalling UDGs. We note that La Marca et al. (2022b) estimated a total UDG
population out to the virial radius of 48 $\pm$ 10 and so many outer region
UDGs, which may be late infallers, remain to be studied.

The UDG infall diagram does not clearly show GC-rich UDGs to be located in
earlier infall zones as might be expected in the standard picture of dwarf
galaxy quenching due to infall which leads to richer GC systems (as described
in the Introduction). Indeed, the opposite trend may be present, such that in
the very early infall region there are 13 GC-poor UDGs and 5 GC-rich ones,
whereas outside of this region (but within 0.5 R/R200) there are only 6 GC-
poor and 6 GC-rich UDGs. Again, we caution that selection and projection
effects make conclusions tentative.

##  4 Discussion

Alabi et al. (2018) used the phase-space diagram to investigate the infall
epoch of UDGs, classical dwarfs and other galaxies in the Coma cluster (a
massive, dynamically relaxed cluster). Similar to the Hydra I cluster, they
saw little difference between classical dwarfs and the giant galaxies. For the
UDGs, they identified both early and late infallers. A similar situation might
be present for Hydra I UDGs if outer region UDGs were probed. Alabi et al.
(2018) did not include GC richness in their study.

Given the lack of a clear signal for ‘infall bias’ in the GC richness of UDGs
alternatives should be further investigated. As noted in the Introduction,
quenching at very early times prior to cluster infall should be considered.
For such UDGs, we would expect very old ages, low metallicities (similar to
the metal-poor subpopulation of GCs) and high alpha overabundances (indicative
of rapid star formation). A high fraction of mass in GCs relative to field
stars might also be expected. A UDG in the NGC 5846 group, (NGC5846$\\_$UDG1)
discovered in VEGAS imaging (Forbes et al., 2019), may be an example of such a
failed galaxy having a remarkable 13% of its current stellar mass in the form
of GCs (Danieli et al., 2022). As noted above, the observed Hydra I UDGs (from
the inner cluster regions) all have less than 1.2% of their stellar mass in
GCs.

Another possibility is that the Hydra I UDGs are GC-poor because they have
been tidally stripped from their host galaxy. This tidal stripping would have
to remove most of the dark matter halo before any GCs, since the dark matter
is more radially extended than GC systems. Continued stripping would be
expected to remove GCs and stars in roughly equal proportions since the radial
extent of GC systems for UDGs closely follows that of the galaxy stars. As
well as operating in clusters, tidal stripping of UDGs may occur in galaxy
groups. We note that UDGs in the field do tend to be GC-poor (Jones et al.,
2023) however this is unlikely to be due to tidal effects and rather some
internal process.

The Hydra I UDGs are generally well-fit by a single Sersic profile however a
few show hints of asymmetries that might point to a tidal interaction (Iodice
et al., 2020; La Marca et al., 2022a). For the one UDG examined in detail by
Iodice et al. (2023, submitted) there is some evidence for an isophotal twist
in the MUSE data. This might indicate tidal interaction (or a triaxial
potential). Furthermore, a Hydra I UDG first identified by Misgeld et al.
(2008) reveals a clear S-shape indicative of ongoing tidal interaction (Koch
et al., 2012). In the case of Coma cluster UDGs, Mowla et al. (2017) looked
specifically for signs of tidal features via position angle twists in a
stacked sample, finding no evidence for such twists.

Sales et al. (2020) have simulated UDGs in clusters of similar mass to Hydra I
using Illustris-TNG100. They identify two types of UDGs in clusters, i.e.
Tidal-UDGs and Born-UDGs (see also Jiang et al. 2019). The Tidal-UDGs were
originally massive galaxies (up to 1010 M⊙) that have been tidally stripped of
stars and puffed-up by the cluster. Born-UDGs were formed as UDGs outside of
the cluster and more recently entered the cluster. Thus Tidal-UDGs dominate
the inner $\sim$0.5R/R200 since they were accreted at early times, while Born-
UDGs dominate the outer regions with some only recently falling into the
cluster. We remind the reader that we only probe out to 0.3R/R200 in Hydra I.
The Sales et al. (2020) model would also predict on average higher
metallicities, older ages and lower internal velocity dispersions, for a given
stellar mass, for their Tidal-UDG compared to the Born-UDGs. These stellar
population, kinematic, GC colours and dark matter content predicted for Tidal-
UDGs can be tested when the full LEWIS dataset is available.

##  5 Conclusions

As part of the LEWIS project (Iodice et al. 2023, in press) we obtained new
VLT/MUSE observations of the radial velocities of UDGs in the Hydra I cluster
(at D = 51 Mpc). Here we examine the location of Hydra I UDGs in infall phase-
space diagrams based on simulations of cluster galaxies. We find all of the
observed UDGs (and 3 low surface brightness galaxies) to be associated with
the cluster. From comparison with the Rhee et al. (2017) simulations, we
conclude that most giants, classical dwarfs and UDGs fell into the Hydra I
cluster long ago, with UDGs being among the earliest infallers. Projection
effects in observations and the statistical nature of the infall diagnostic
diagram limit our ability to determine the true fraction of ancient infallers.
Nevertheless we might expect UDGs in the Hydra I cluster to reveal old stellar
populations consistent with early quenching.

We also compare Hydra I UDGs with their counterparts in the Coma, Perseus and
Virgo clusters in terms of their GC richness. If very early infall into a
cluster is associated with enhanced GC richness (as has been suggested for
classical dwarf galaxies) then such a trend is expected. The data from these
clusters do not show a clear trend of GC richness with earlier infall times,
indeed the data suggest the opposite trend. If verified by larger and more
complete samples, then UDGs may be quenched by a different mechanism than that
thought to operate on classical dwarf galaxies. As more data for UDGs is
acquired, trends, or the lack of, may become more apparent in an infall
diagnostic diagram. A future analysis of star formation histories will give an
indication of when quenching occurred for the Hydra I UDGs. Once the full
dataset of the LEWIS project is available we will be able to test other
mechanisms, such as pre-infall quenching and/or tidal stripping, and their
possible role in shaping UDGs and their globular cluster systems.

## Acknowledgements

We wish to thank the anonymous referee for their comments. We thank A.
Romanowsky, L. Buzzo, L. Haacke and O. Gerhard for useful suggestions. This
work is based on visitor mode observations collected at the European Southern
Observatory (ESO) La Silla Paranal Observatory and collected at the European
Southern Observatory under ESO programmes 099.B-0560(A) and 108.222P. INAF
authors acknowledge financial support for the VST project (P.I.: P. Schipani).
DAF thanks the ARC for support via DP220101863. Parts of this research were
supported by the Australian Research Council Centre of Excellence for All Sky
Astrophysics in 3 Dimensions (ASTRO 3D), through project number CE170100013.
MC acknowledges support from the INAF-EDGE program (PI Leslie K. Hunt). J. F-B
acknowledges support through the RAVET project by the grant
PID2019-107427GB-C32 from the Spanish Ministry of Science, Innovation and
Universities (MCIU), and through the IAC project TRACES which is partially
supported through the state budget and the regional budget of the Consejería
de Economía, Industria, Comercio y Conocimiento of the Canary Islands
Autonomous Community.

## Data Availability

Raw data is available from the ESO archive.

## References

  * Alabi et al. (2018) Alabi A., et al., 2018, MNRAS, 479, 3308
  * Arnaboldi et al. (2012) Arnaboldi M., Ventimiglia G., Iodice E., Gerhard O., Coccato L., 2012, A&A, 545, A37
  * Burkert & Forbes (2020) Burkert A., Forbes D. A., 2020, AJ, 159, 56
  * Carleton et al. (2021) Carleton T., Guo Y., Munshi F., Tremmel M., Wright A., 2021, MNRAS, 502, 398
  * Christlein & Zabludoff (2003) Christlein D., Zabludoff A. I., 2003, ApJ, 591, 764
  * Danieli et al. (2022) Danieli S., et al., 2022, ApJ, 927, L28
  * Forbes et al. (2019) Forbes D. A., Gannon J., Couch W. J., Iodice E., Spavone M., Cantiello M., Napolitano N., Schipani P., 2019, A&A, 626, A66
  * Forbes et al. (2020) Forbes D. A., Alabi A., Romanowsky A. J., Brodie J. P., Arimoto N., 2020, MNRAS, 492, 4874
  * Gannon et al. (2022) Gannon J. S., et al., 2022, MNRAS, 510, 946
  * Iodice et al. (2020) Iodice E., et al., 2020, A&A, 642, A48
  * Iodice et al. (2021) Iodice E., et al., 2021, A&A, 652, L11
  * Jiang et al. (2019) Jiang F., Dekel A., Freundlich J., Romanowsky A. J., Dutton A. A., Macciò A. V., Di Cintio A., 2019, MNRAS, 487, 5272
  * Jones et al. (2023) Jones M. G., et al., 2023, ApJ, 942, L5
  * Koch et al. (2012) Koch A., Burkert A., Rich R. M., Collins M. L. M., Black C. S., Hilker M., Benson A. J., 2012, ApJ, 755, L13
  * La Marca et al. (2022a) La Marca A., et al., 2022a, A&A, 659, A92
  * La Marca et al. (2022b) La Marca A., et al., 2022b, A&A, 665, A105
  * Lima-Dias et al. (2021) Lima-Dias C., et al., 2021, MNRAS, 500, 1323
  * Liu et al. (2016) Liu Y., et al., 2016, ApJ, 818, 179
  * Misgeld et al. (2008) Misgeld I., Mieske S., Hilker M., 2008, A&A, 486, 697
  * Mistani et al. (2016) Mistani P. A., et al., 2016, MNRAS, 455, 2323
  * Moster et al. (2020) Moster B. P., Naab T., White S. D. M., 2020, MNRAS, 499, 4748
  * Mowla et al. (2017) Mowla L., van Dokkum P., Merritt A., Abraham R., Yagi M., Koda J., 2017, ApJ, 851, 27
  * Pasha et al. (2023) Pasha I., Mandelker N., van den Bosch F. C., Springel V., van de Voort F., 2023, MNRAS, 520, 2692
  * Peng et al. (2008) Peng E. W., et al., 2008, ApJ, 681, 197
  * Ramos-Almendares et al. (2020) Ramos-Almendares F., Sales L. V., Abadi M. G., Doppel J. E., Muriel H., Peng E. W., 2020, MNRAS, 493, 5357
  * Rhee et al. (2017) Rhee J., Smith R., Choi H., Yi S. K., Jaffé Y., Candlish G., Sánchez-Jánssen R., 2017, ApJ, 843, 128
  * Romero-Gómez et al. (2023) Romero-Gómez J., et al., 2023, MNRAS, 522, 130
  * Sales et al. (2020) Sales L. V., Navarro J. F., Peñafiel L., Peng E. W., Lim S., Hernquist L., 2020, MNRAS, 494, 1848
  * Stinson et al. (2013) Stinson G. S., Brook C., Macciò A. V., Wadsley J., Quinn T. R., Couchman H. M. P., 2013, MNRAS, 428, 129
  * Toloba et al. (2023) Toloba E., et al., 2023, arXiv e-prints, p. arXiv:2305.06369
  * Trujillo-Gomez et al. (2022) Trujillo-Gomez S., Kruijssen J. M. D., Reina-Campos M., 2022, MNRAS, 510, 3356
  * Ventimiglia et al. (2011) Ventimiglia G., Arnaboldi M., Gerhard O., 2011, A&A, 528, A24
  * Wang et al. (2021) Wang J., et al., 2021, ApJ, 915, 70
  * Yagi et al. (2016) Yagi M., Koda J., Komiyama Y., Yamanoi H., 2016, ApJS, 225, 11
  * van Dokkum et al. (2015) van Dokkum P. G., Abraham R., Merritt A., Zhang J., Geha M., Conroy C., 2015, ApJ, 798, L45


#  Falcon 2.0: An Entity and Relation Linking Tool over Wikidata

Ahmad Sakor<EMAIL_ADDRESS>L3S Research Center and TIB, University of
HannoverHannoverGermany ,  Kuldeep Singh<EMAIL_ADDRESS>Cerence
GmbH and Zerotha ResearchP.O. Box 1212AachenGermany ,  Anery Patel
<EMAIL_ADDRESS>TIB, University of HannoverHannoverGermany and  Maria-
Esther Vidal<EMAIL_ADDRESS>L3S Research Center and TIB, University of
HannoverHannoverGermany

(2020)

###### Abstract.

The Natural Language Processing (NLP) community has significantly contributed
to the solutions for entity and relation recognition from a natural language
text, and possibly linking them to proper matches in Knowledge Graphs (KGs).
Considering Wikidata as the background KG, there are still limited tools to
link knowledge within the text to Wikidata. In this paper, we present Falcon
2.0, the first joint entity and relation linking tool over Wikidata. It
receives a short natural language text in the English language and outputs a
ranked list of entities and relations annotated with the proper candidates in
Wikidata. The candidates are represented by their Internationalized Resource
Identifier (IRI) in Wikidata. Falcon 2.0 resorts to the English language model
for the recognition task (e.g., N-Gram tiling and N-Gram splitting), and then
an optimization approach for the linking task. We have empirically studied the
performance of Falcon 2.0 on Wikidata and concluded that it outperforms all
the existing baselines. Falcon 2.0 is open source and can be reused by the
community; all the required instructions of Falcon 2.0 are well-documented at
our GitHub repository111https://github.com/SDM-TIB/falcon2.0. We also
demonstrate an online API, which can be run without any technical expertise.
Falcon 2.0 and its background knowledge bases are available as resources at
https://labs.tib.eu/falcon/falcon2/.

NLP, Entity Linking, Relation Linking, Background Knowledge, English
morphology, DBpedia, and Wikidata

††journalyear: 2020††copyright: rightsretained††conference: Proceedings of the
29th ACM International Conference on Information and Knowledge Management;
October 19–23, 2020; Virtual Event, Ireland††booktitle: Proceedings of the
29th ACM International Conference on Information and Knowledge Management
(CIKM ’20), October 19–23, 2020, Virtual Event, Ireland††doi:
10.1145/3340531.3412777††isbn: 978-1-4503-6859-9/20/10††ccs: Information
systems Resource Description Framework (RDF)††ccs: Information systems
Information extraction

##  1\. Introduction

Entity Linking (EL)- also known as Named Entity Disambiguation (NED)- is a
well-studied research domain for aligning unstructured text to its structured
mentions in various knowledge repositories (e.g., Wikipedia, DBpedia (Auer et
al., 2007), Freebase (Bollacker et al., 2008) or Wikidata (Vrandecic, 2012)).
Entity linking comprises two sub-tasks. The first task is Named Entity
Recognition (NER), in which an approach aims to identify entity labels (or
surface forms) in an input sentence. Entity disambiguation is the second sub-
task of linking entity surface forms to semi-structured knowledge
repositories. With the growing popularity of publicly available knowledge
graphs (KGs), researchers have developed several approaches and tools for EL
task over KGs. Some of these approaches implicitly perform NER and directly
provide mentions of entity surface forms in the sentences to the KG (often
referred to as end-to-end EL approaches) (Delpeuch, 2019). Other attempts
(e.g., Yamanda et al. (Yamada et al., 2016a), DCA (Yang et al., 2019))
consider recognized surface forms of the entities as additional inputs besides
the input sentence to perform entity linking. Irrespective of the input format
and underlying technologies, the majority of the existing attempts (Röder et
al., 2018) in the EL research are confined to well-structured KGs such as
DBpedia or Freebase222it is now depreciated and no further updates are
possible. These KGs rely on a well-defined process to extract information
directly from Wikipedia infoboxes. They do not provide direct access to the
users to add/delete the entities or alter the KG facts. Wikidata, on the other
hand, also allows users to edit Wikidata pages directly, add newer entities,
and define new relations between the objects. Wikidata is hugely popular as a
crowdsourced collection of knowledge. Since its launch in 2012, over 1 billion
edits have been made by the users across the
world333https://www.wikidata.org/wiki/Wikidata:Statistics.

#### Motivation, Approach, and Contributions.

We motivate our work by the fact that despite the vast popularity of Wikidata,
there are limited attempts to target entity and relation linking over
Wikidata. For instance, there are over 20 entity linking tools/APIs for
DBpedia (Singh et al., 2018b; Röder et al., 2018), which are available as
APIs. To the best of our knowledge, there exists only one open-source API for
Wikidata entity linking (i.e., OpenTapioca (Delpeuch, 2019)). Furthermore,
there is no tool over Wikidata for relation linking, i.e., linking predicate
surface forms to their corresponding Wikidata mentions. In this paper, we
focus on providing Falcon 2.0, a reusable resource API for joint entity and
relation linking over Wikidata. In our previous work, we proposed Falcon
(Sakor et al., 2019), a rule-based approach yet effective for entity and
relation linking on short text (questions in this case) over DBpedia. In
general, the Falcon approach has two novel concepts: 1) a linguistic-based
approach that relies on several English morphology principles such as
tokenization, and N-gram tiling; 2) a local knowledge base which serves as a
source of background knowledge (BK). This knowledge base is a collection of
entities from DBpedia. We resort to the Falcon approach for developing Falcon
2.0. Our aim here is to study whether or not the Falcon approach is agnostic
to underlying KG; hence, we do not claim novelty in the underlying
_linguistic-based approach_ for Falcon 2.0. Further, we investigate the
concerns related to robustness, emerging failures, and bottlenecks. We
introduce Falcon 2.0 based on the methodology employed in the first version.
Our tool is the first joint entity and relation linking tool for Wikidata. Our
novel contributions briefly lie in two aspects:

  1. (1)

Falcon 2.0: The first resource for joint entity and relation linking over
Wikidata. Falcon 2.0 relies on fundamental principles of English morphology
(tokenization and compounding) and links entity and relation surface forms in
a short sentence to its Wikidata mentions. Falcon 2.0 is available as an
online API and can be accessed at https://labs.tib.eu/falcon/falcon2/. Falcon
2.0 is also able to recognize entities in keywords such as Barack Obama, where
there is no relation. We empirically evaluate Falcon 2.0 on three datasets
tailored for Wikidata. According to the observed results, Falcon 2.0
significantly outperforms all the existing baselines. For the ease of use, we
integrate the Falcon API444https://labs.tib.eu/falcon/ into Falcon 2.0. This
option is available in case Wikipedia contains an equivalence entity (Wikidata
is a superset of DBpedia) The Falcon 2.0 API already has over half a million
hits from February 2020 to the time of paper acceptance, which shows its
gaining usability (excluding self-access of the API while performing the
evaluation).

  2. (2)

Falcon 2.0 Background KG: We created a new background KG of Falcon 2.0 with
the Wikidata. We extracted 48,042,867 Wikidata entities from its public dump
and aligned these entities with the aliases present in Wikidata. For example,
Barack Obama is a Wikidata entity
Wiki:Q76555https://www.wikidata.org/wiki/Q76. We created a mapping between the
label (Barack Obama) of Wiki:Q76 with its aliases such as President Obama,
Barack Hussein Obama, and Barry Obama and stored it in the background
knowledge base. We implemented a similar alignment for 15,645
properties/relations of Wikidata. The background knowledge base is an indexed
graph and can be queried. The resource is also present at a persistent URI for
further reuse666https://doi.org/10.6084/m9.figshare.11362883.

The rest of this paper is organized as follows: Section 2 reviews the state-
of-the-art, and the following Section 3 describes our two resources and
approach to build Falcon 2.0. Section 4 presents experiments to evaluate the
performance of Falcon 2.0. Section 5 presents the importance and impact of
this work for the research community. The availability and sustainability of
resources is explained in Section 6, and its maintenance related discussion is
presented in Section 7. We close with the conclusion in Section 8.

##  2\. Related Work

Several surveys provide a detailed overview of the advancements of the
techniques employed in entity linking over KGs (Shen et al., 2015; Balog,
2018). Various reading lists (Ji, 2019), online
forums777http://nlpprogress.com/english/entity_linking.html and Github
repositories888https://github.com/sebastianruder/NLP-
progress/blob/master/english/entity_linking.md track the progress in the
domain of entity linking. Initial attempts in EL considered Wikipedia as an
underlying knowledge source. The research field has matured and the SOTA
nearly matches human-level performance (Raiman and Raiman, 2018). With the
advent of publicly available KGs such as DBpedia, Yago, and Freebase, the
focus has shifted towards developing EL over knowledge graphs. The
developments in Deep Learning have introduced a range of models that carry out
both NER and NED as a single end-to-end step (Kolitsas et al., 2018; Ganea and
Hofmann, 2017). NCEL (Cao et al., 2018) learns both local and global features
from Wikipedia articles, hyperlinks, and entity links to derive joint
embeddings of words and entities. These embeddings are used to train a deep
Graph Convolutional Network (GCN) that integrates all the features through a
Multi-layer Perceptron. The output is passed through a Sub-Graph Convolution
Network, which finally resorts to a fully connected decoder. The decoder maps
the output states to linked entities. The BI-LSTM+CRF model (Inan and
Dikenelli, 2018) formulates entity linking as a sequence learning task in
which the entity mentions are a sequence whose length equals the series of the
output entities. Albeit precise, deep learning approaches demand _high-
quality_ training annotations, which are not extensively available for
Wikidata entity linking (Cetoli et al., 2019; Mulang et al., 2020).

There is concrete evidence in the literature that the machine learning-based
models trained over generic datasets such as WikiDisamb30 (Ferragina and
Scaiella, 2010), and CoNLL (YAGO) (Hoffart et al., 2011) do not perform well
when applied to short texts. Singh et al. (Singh et al., 2018b) evaluated more
than 20 entity linking tools over DBpedia for short text (e.g., questions) and
concluded that issues like capitalization of surface forms, implicit entities,
and multi-word entities affect the performance of EL tools in a short input
text. Sakor et al. (Sakor et al., 2019) addresses specific challenges of short
texts by applying a rule-based approach for EL over DBpedia. In addition to
linking entities to DBpedia, Sakor et al. also provides DBpedia IRIs of the
relations in a short text. EARL (Banerjee et al., [n.d.]) is another tool that
proposes a traveling salesman algorithm-based approach for joint entity and
relation linking over DBpedia. To the best of our knowledge, EARL and Falcon
are the only available tools that provide both entity and relation linking.

Entity linking over Wikidata is a relatively new domain. Cetoli et al. (Cetoli
et al., 2019) propose a neural network-based approach for linking entities to
Wikidata. The authors also align an existing Wikipedia corpus-based dataset to
Wikidata. However, this work only targets entity disambiguation and assumes
that the entities are already recognized in the sentences. Arjun (Mulang et
al., 2020) is the latest work for Wikidata entity linking. It uses an
attention-based neural network for linking Wikidata entity labels. OpenTapioca
(Delpeuch, 2019) is another attempt that performs end-to-end entity linking
over Wikidata; it is the closest to our work even though OpenTapioca does not
provide Wikidata Ids of relations in a sentence. OpenTapioca is also available
as an API and is utilized as our baseline. S-Mart (Yang and Chang, 2015) is a
tree-based structured learning framework based on multiple additive regression
trees for linking entities in a tweet. The model was later adapted for linking
entities in the questions. VCG (Sorokin and Gurevych, 2018) is another attempt
which is a unifying network that models contexts of variable granularity to
extract features for an end to end entity linking. However, Falcon 2.0 is the
first tool for joint entity and relation linking over Wikidata.

##  3\. Falcon 2.0\- A Resource

In this section, we describe Falcon 2.0 in detail. First the architecture of
Falcon 2.0 is depicted. Next, we discuss the BK used to match the surface
forms in the text to the resource in a specific KG. In the paper’s scope, we
define ”short text” as grammatically correct questions (up to 15 words).

###  3.1. Architecture

Figure 1. The Falcon 2.0 Architecture. The boxes highlighted in Grey are
reused from Falcon (Sakor et al., 2019). Grey boxes contain a linguistic
pipeline for recognizing and linking entity and relation surface forms. The
boxes in White are our addition to the Falcon pipeline to build a resource for
the Wikidata entity and relation linking. The white boxes constitute what we
refer to as BK specific to Wikidata. The text search engine contains the
alignment of Wikidata entity/relation labels along with the entity and
relation aliases. It is used for generating potential candidates for entity
and relation linking. RDF triple store is a local copy of Wikidata triples
containing all entities and predicates.

The Falcon 2.0 architecture is depicted in Figure 1. Falcon 2.0 receives short
input texts and outputs a set of entities and relations extracted from the
text; each entity and relation in the output is associated with a unique
Internationalized Resource Identifier (IRI) in Wikidata. Falcon 2.0 resorts to
BK and a catalog of rules for performing entity and relation linking. The BK
combines Wikidata labels and their corresponding aliases. Additionally, it
comprises alignments between nouns and entities in Wikidata. Alignments are
stored in a text search engine, while the knowledge source is maintained in an
RDF triple store accessible via a SPARQL endpoint. The rules that represent
the English morphology are in a catalog; a forward chaining inference process
is performed on top of the catalog during the extraction and linking tasks.
Falcon 2.0 also comprises several modules that identify and link entities and
relations to the Wikidata. These modules implement POS Tagging, Tokenization &
Compounding, N-Gram Tiling, Candidate List Generation, Matching & Ranking,
Query Classifier, and N-Gram Splitting and are reused from the implementation
of Falcon.

###  3.2. Background Knowledge

Figure 2. Falcon 2.0 Background Knowledge is built by converting labels of
entities and relations in Wikidata into pairs of alignments. It is a part of
search engine (cf. Figure 1).

Wikidata contains over 52 million entities and 3.9 billion facts (in the form
of subject-predicate-object triples). Since Falcon 2.0 background knowledge
only depends on labels, a significant portion of this extensive information is
not useful for our approach. Hence, we only extract all the entity and
relation labels to create a local background KG, A.K.A ”alias background
knowledge base.”. For example, the entity United States of
America999https://www.wikidata.org/wiki/Q30 in Wikidata has the natural
language label ‘United States of America’ and several other aliases (or
known_as labels) of United States of America such as ”the United States of
America, America, U.S.A., the U.S., United States, etc.”. We extended our
background KG with this information from Wikidata. Similarly, for relation’s
labels, the background KG is enriched with known_as labels to provide synonyms
and derived word forms. For example, the relation spouse
101010https://www.wikidata.org/wiki/Property:P26 in Wikidata has the label
spouse and the other known as labels are husband, wife, married to, wedded to,
partner, etc. This variety of synonyms for each relation empowers Falcon 2.0
to match the surface form in the text to a relation in Wikidata. Figure 2
illustrates the process of building background knowledge.

###  3.3. Catalog of Rules

Falcon 2.0 is a rule-based approach. A catalog of rules is predefined to
extract entities and relations from the text. The rules are based on the
English morphological principles and borrowed from Sakor et al. (Sakor et al.,
2019). For example, Falcon 2.0 excludes all verbs from the entities candidates
list based on the rule verbs are not entities. For example, the N-Gram tiling
module in the Falcon 2.0 architecture resorts to the rule: entities with only
stopwords between them are one entity. Another example of such rule When ->
date, Where -> place solves the ambiguity of matching the correct relation in
case the short text is a question by looking at the questions headword. For
example, give the two questions When did Princess Diana die? and Where did
Princess Diana die?, the relation died can be the death place or the death
year. The question headword (When/Where) is the only insight to solve the
ambiguity here. When the question word is where, Falcon 2.0 matches only
relations that have a place as a range of the relation.

###  3.4. Recognition

Extraction phase in Falcon 2.0 consists of three modules. POS tagging,
tokenization & compounding, and N-Gram tiling. The input of this phase is a
natural language text. The output of the phase is the list of surface forms
related to entities or relations.
Part-of-speech (POS) Tagging receives a natural language text as an input. It
tags each word in the text with its related tag, e.g., noun, verb, and adverb.
This module differentiates between nouns and verbs to enable the application
of the morphological rules from the catalog. The output of the module is a
list of the pairs of (word, tag).
Tokenization & Compounding builds the tokens list by removing the stopwords
from the input and splitting verbs from nouns. For example, if the input is
What is the operating income for Qantas, the output of this module is a list
of three tokens [operating, income, Qantas].
N-Gram Tilling module combines tokens with only stopwords between them relying
on one of the rules from a catalog of rules. For example, if we consider the
previous module’s output as an input for the n-gram tilling module, operating
and income tokens will be combined in one token. The output of the module is a
list of two tokens [operating income, Qantas].

###  3.5. Linking

This phase consists of four modules: candidate list generation, matching &
ranking, relevant rule selection, and n-gram splitting.
Candidate List Generation receives the output of the recognition phase. The
module queries the text search engine for each token. Then, tokens will have
an associated candidate list of resources. For example, the retrieved
candidate list of the token operating income is [(P3362, operating income),
(P2139, income), (P3362, operating profit)]; where the first element is the
Wikidata predicate identifier and the second is the list of labels associated
with the predicates which match the query ”operating income.”
Matching & Ranking ranks the candidate list received from the candidate list
generation module and matches candidates’ entities and relations. Since, in
any KG, the facts are represented as triples, the matching and ranking module
creates triples consisting of the entities and relationships from the
candidates’ list. Then, for each pair of entity and relation, the module
checks if the triple exists in the RDF triple store (Wikidata). The checking
is done by executing a simple ASK query over the RDF triple store. For each
triple, the module increases the rank of the involved relations and entities.
The output of the module is the ranked list of the candidates.
Relevant Rule Selection interacts with the matching & ranking module by
suggesting increasing the ranks of some candidates relying on the catalog of
rules. One of the suggestions is considering the question headword to clear
the ambiguity between two relations based on the range of relationships in the
KG.
N-Gram Splitting module is used if none of the triples tested in the matching
& ranking modules exists in the triple store, i.e., the compounding the
approach did in the tokenization & compounding module led to combining two
separated entities. The module splits the tokens from the right side and
passes the tokens again to the candidate list generation module. Splitting the
tokens from the right side resorts to one of the fundamentals of the English
morphology; the compound words in English have their headword always towards
the right side (Williams, 1981).
Text Search Engine stores all the alignments of the labels. A simple querying
technique (Gormley and Tong, 2015) is used as the text search engine over
background knowledge. It receives a token as an input and then returns all the
related resources with labels similar to the received token.
RDF Triple store is a local copy of the Wikidata endpoint. It consists of all
the RDF triples of Wikidata labeled with the English language. An RDF triple
store is used to check the existence of the triples passed from the Matching &
Ranking module. The RDF triple store keeps around 3.9 billion triples.

##  4\. Experimental Study

We study three research questions: RQ1) What is the performance of Falcon 2.0
for entity linking over Wikidata? RQ2) What is the impact of Wikidata’s
specific background knowledge on the performance of a linguistic approach?
RQ3) What is the performance of Falcon 2.0 for relation linking over Wikidata?

#### Metrics

We report the performance using the standard metrics of Precision, Recall, and
F-measure. Precision is the fraction of relevant resources among the retrieved
resources. Recall is the fraction of relevant resources that have been
retrieved over the total amount of relevant resources. F-Measure or F-Score is
the harmonic mean of precision and recall.

#### Datasets

We rely on three different question answering datasets namely SimpleQuestion
dataset for Wikidata (Diefenbach et al., 2017), WebQSP-WD (Sorokin and
Gurevych, 2018) and LC-QuAD 2.0 (Dubey et al., 2019). The SimpleQuestion
dataset contains 5,622 test questions which are answerable using Wikidata as
underlying KG. WebQSP-WD contains 1639 test questions, and LC-QUAD 2.0
contains 6046 test questions. SimpleQuestion and LC-QuaD 2.0 provide the
annotated gold standard for entity and relations, whereas WebQSP-WD only
provides annotated gold standard for entities. Hence, we evaluated entity
linking performance on three datasets and relation linking performance on two
datasets. Also, SimpleQuestion and WebQSP-WD contain questions with a single
entity and relation, whereas LC-QuAD 2.0 contains mostly complex questions
(i.e., more than one entity and relation).

#### Baselines

OpenTapioca (Delpeuch, 2019): is available as a web API; it provides Wikidata
URIs for entities. We run OpenTapioca API on all the three datasets.
Variable Context Granularity model (VCG) (Sorokin and Gurevych, 2018): is a
unifying network that models contexts of variable granularity to extract
features for mention detection and entity disambiguation. We were unable to
reproduce VCG using the publicly available source code. Hence, we only report
its performance on WebQSP-WD from the original paper (Sorokin and Gurevych,
2018) as we are unable to run the model on the other two datasets for entity
linking. For the completion of the approach, we also report the other two
baselines provided by the authors, namely Heuristic Baseline and Simplified
VCG.
S-Mart (Yang and Chang, 2015): was initially proposed to link entities in the
tweets and later adapted for question answering. The system is not open
source, and we adapt its result from (Sorokin and Gurevych, 2018) for WebQSP-
WD dataset.
No Baseline for Relation Linking: To the best of our knowledge, there is no
baseline for relation linking on Wikidata. One argument could be to run the
existing DBpedia based relation linking tool on Wikidata and compare it with
our performance. We contest this solely because Wikidata is extremely noisy.
For example, in ”What is the longest National Highway in the world?” the
entity surface form ”National Highway” matches four(4) different entities in
Wikidata that share the same entity label (i.e., ”National Highway”). In
comparison, 2,055 other entities contain the full mention in their labels for
the surface form ”National Highway”. However, in DBpedia, there exists only
one unique label for ”National Highway”. Hence, any entity linking tool or
relation linking tool tailored for DBpedia will face issues on Wikidata (cf.
table 3). Therefore, instead of reporting the bias and under-performance, we
did not evaluate their performance for a fair comparison. Hence, we report
Falcon 2.0 relation linking performance only to establish new baselines on two
datasets: SimpleQuestion and LC-QuAD 2.0.

Figure 3. Falcon 2.0 API Web Interface.

#### Experimental Details

Falcon 2.0 is extremely lightweight from an implementation point of view. A
laptop machine, with eight cores and 16GB RAM running Ubuntu 18.04 is used for
implementing and evaluating Falcon 2.0. We deployed its web API on a server
with 723GB RAM, 96 cores (Intel(R) Xeon(R) Platinum 8160CPU with 2.10GHz)
running Ubuntu 18.04. This publicly available API is used to calculate the
standard evaluation metrics, namely Precision, Recall, and F-score.

###  4.1. Experimental Results

Table 1. Entity linking evaluation results on LC-QuAD 2.0 & SimpleQuestion
datasets. Best values are in bold.

Approach | Dataset | P | R | F
---|---|---|---|---
OpenTapioca (Delpeuch, 2019) | LC-QuAD 2.0 | 0.29 | 0.42 | 0.35
Falcon 2.0 | LC-QuAD 2.0 | 0.50 | 0.56 | 0.53
OpenTapioca (Delpeuch, 2019) | SimpleQuestion | 0.01 | 0.02 | 0.01
Falcon 2.0 | SimpleQuestion | 0.56 | 0.64 | 0.60
OpenTapioca (Delpeuch, 2019) | SimpleQuestion Uppercase Entities | 0.16 | 0.28 | 0.20
Falcon 2.0 | SimpleQuestion Uppercase Entities | 0.66 | 0.75 | 0.70

Table 2. Entity linking evaluation results on the WEBQSP test dataset. Best
values are in bold.

Approach | P | R | F
---|---|---|---
S-MART (Yang and Chang, 2015) | 0.66 | 0.77 | 0.72
Heuristic baseline (Sorokin and Gurevych, 2018) | 0.30 | 0.61 | 0.40
Simplified VCG (Sorokin and Gurevych, 2018) | 0.84 | 0.62 | 0.71
VCG (Sorokin and Gurevych, 2018) | 0.83 | 0.65 | 0.73
OpenTapioca (Delpeuch, 2019) | 0.01 | 0.02 | 0.02
Falcon 2.0 | 0.80 | 0.84 | 0.82

#### Experimental Results 1

In the first experiment described in Table 1, we compare entity linking
performance of Falcon 2.0 on SimpleQuestion and LC-QuAD 2.0 datasets. We first
evaluate the performance on the SimpleQuestion dataset. Surprisingly, we
observe that for the OpenTapioca baseline, the values are approximately 0.0
for Precision, Recall, and F-score. We analyzed the source of errors and found
that out of 5,622 questions, only 246 have entity labels in uppercase letters.
Opentapioca fails to recognize and link entity mentions written in lowercase
letters. Case sensitivity is a common issue for entity linking tools over
short text, as reported by Singh et al. (Singh et al., 2018a; Singh et al.,
2018b) in a detailed analysis. From the remaining 246 questions, only 70 are
answered correctly by OpenTapioca. Given that OpenTapioca finds limitation in
linking lowercase entity surface forms. We evaluated Falcon 2.0 and
OpenTapioca on the 246 questions of SimpleQuestion to provide a fair
evaluation for the baseline (reported as SimpleQuestion uppercase entities in
table 1). OpenTapioca reports F-score 0.20 on this subset of SimpleQuestion.
On the other hand, Falcon 2.0 reports F-score 0.70 on the same dataset (cf.
Table 1). For LC-QuAD 2.0, OpenTapioca reports F-score 0.35 against Falcon 2.0
with F-score 0.53 reported in Table 1.

#### Experimental Results 2

We report performance of Falcon 2.0 on WebQSP-WD dataset in Table 2. Falcon
2.0 clearly outperforms all other baselines with highest F-score value 0.82.
OpenTapioca demonstrates a low performance on this dataset as well. Experiment
results 1 & 2 answer our first research question (RQ1).

#### Ablation Study for Entity Linking and Recommendations

For the second research question (RQ2), we evaluate the impact of Wikidata’s
specific background knowledge on the entity linking performance. We evaluated
Falcon on the WebQSP-WD dataset against Falcon 2.0. We linked Falcon predicted
DBpedia IRIs with corresponding Wikidata IDs using owl:sameAs. We can see in
the Table 3 that Falcon 2.0 significantly outperforms Falcon despite using the
same linguistic driven approach. The jump in Falcon 2.0 performance comes from
Wikidata’s specific local background knowledge, which we created by expanding
Wikidata entities and relations with associated aliases. It also validates the
novelty of Falcon 2.0 when compared to Falcon for the Wikidata entity linking.

We observe an indifferent phenomenon in our performance for three datasets,
and the performance for Falcon 2.0 differs a lot per dataset. For instance, on
WebQSP-WD, our F-score is 0.82, whereas, on LC-QuAD 2.0, the F-Score drops to
0.57. The first source of error is the dataset(s) itself. In both the datasets
(SimpleQuestion and LC-QuAD 2.0), many questions are grammatically incorrect.
To validate our claim more robustly, we asked two native English speakers to
check the grammar of 200 random questions on LC-QuAD 2.0. Annotators reported
that 42 out of 200 questions are grammatically incorrect. Many questions have
erroneous spellings of the entity names. For example, ”Who is the country for
head of state of Mahmoud Abbas?” and ”Tell me about position held of Malcolm
Fraser and elected in?” are two grammatically incorrect questions in LC-QuAD
2.0. Similarly, many questions in the SimpleQuestion dataset are also
grammatically incorrect. ”where was hank cochran birthed” is one such example
in the SimpleQuestion dataset. Falcon 2.0 resorts to fundamental principles of
the English morphology and finds limitation in recognizing entities in many
grammatically incorrect questions.

We also recognize that the performance of Falcon 2.0 on sentences with minimal
context is limited. For example, in the question ”when did annie open?” from
the WebQSP-WD dataset, the sentential context is shallow. Also, more than one
instance of ”Annie” exists in Wikidata, such as Wiki:Q566892 (correct one) and
Wiki:Q181734. Falcon 2.0 wrongly predicts the entity in this case. In another
example, ”which country is lamb from?”, the correct entity is Wiki:Q6481017
with label ”lamb” in Wikidata. However, Falcon 2.0 returns Wiki:13553878,
which also has a label ”lamb”. In such cases, additional knowledge graph
context shall prove to be useful. Approaches such as (Yang et al., 2019)
introduced a concept of feeding ”entity descriptions” as an additional context
in an entity linking model over Wikipedia. Suppose the extra context in the
form of entity description (1985 English drama film directed by Colin Gregg)
for the entity Wiki:13553878 is provided. In that case, a model may correctly
predict the correct entity ”lamb.” Based on our observations, we propose the
following recommendations for the community to improve the entity linking task
over Wikidata:

  * •

Wikidata has inherited challenges of vandalism and noisy entities due to
crowd-authored entities (Heindorf et al., 2016). We expect the research
community to come up with more robust short text datasets for the Wikidata
entity linking without spelling and grammatical errors.

  * •

Rule-based approaches come with its limitations when the sentential context is
minimal. However, such methods are beneficial for the nonavailability of
training data. We recommend a two-step process to target questions with
minimal sentential context: 1) work towards a clean and large Wikidata dataset
for entity linking of short text. This will allow more robust machine learning
approaches to evolve 2) use of entity descriptions from knowledge graphs to
improve the linking process (same as (Yang et al., 2019)).

Table 3. Entity Linking Performance of Falcon vs Falcon 2.0 on WEBQSP-WD. Best
values are in bold.

Approach | P | R | F
---|---|---|---
Falcon (Sakor et al., 2019) | 0.47 | 0.45 | 0.46
Falcon 2.0 | 0.80 | 0.84 | 0.82

Table 4. Relation linking evaluation results on LC-QuAD 2.0 & SimpleQuestion
datasets.

Approach | Dataset | P | R | F
---|---|---|---|---
Falcon 2.0 | LC-QuAD 2.0 | 0.44 | 0.37 | 0.40
Falcon 2.0 | SimpleQuestion | 0.35 | 0.44 | 0.39

Table 5. Sample Questions from LC-QuAD 2.0 datset. The table shows five sample
questions and associated gold standard relations. These sentences do not
include standard sentential relations in the English language. Considering
Wikidata is largely authored by the crowd, the crowd often creates such
uncommon relations. Falcon 2.0 finds limitation in linking such relations, and
most results are empty.

Question | Gold Standard IDs | Gold Standard Labels | Predicted IDs | Predicted Labels
---|---|---|---|---
Which is the global-warming potential of dichlorodifluoromethane? | P2565 | global warming potential | [] | _
What is the AMCA Radiocommunications Licence ID for Qantas? | P2472 | ACMA Radiocommunications Client Number | P275 | copyright license
What is ITIS TSN for Sphyraena? | P815 | ITIS TSN | [] | _
What is the ARICNS for Fomalhaut? | P999 | ARICNS | [] | _
Which is CIQUAL 2017 ID for cheddar? | P4696 | CIQUAL2017 ID | [] | _

#### Experimental Results 3:

In the third experiment (for RQ3), we evaluate the relation linking
performance of Falcon 2.0. We are not aware of any other model for relation
linking over Wikidata. Table 4 summarizes relation linking performance. With
this, we established new baselines over two datasets for relation linking on
Wikidata.

#### Ablation Study for Relation Linking and Recommendations

Falcon reported an F-score of 0.43 on LC-QuAD over DBpedia in (Sakor et al.,
2019) whereas Falcon 2.0 reports a comparable relation linking F-score 0.40 on
LC-QuAD 2.0 for Wikidata (cf. Table 4). The wrong identification of the
entities does affect the relation linking performance, and it is the major
source of error in our case for relation linking. Table 5 summarizes a sample
case study for relation linking on five LC-QuAD 2.0 questions. We observe that
the relations present in the questions are highly uncommon and nonstandard,
and it is a peculiar property of Wikidata. Falcon 2.0 finds limitations in
linking such relations. We recommend the following:

  * •

Wikidata challenges relation linking approaches by posing a new challenge:
user-created nonstandard relations such as in Table 5. A rule-based approach
like ours faces a clear limitation in linking such relations. Linking user-
created relations in crowd-authored Wikidata is an open question for the
research community.

##  5\. Impact

In August 2019, Wikidata became the first Wikimedia project that crossed 1
billion edits, and over 20,000 active Wikidata
editors111111https://www.wikidata.org/wiki/Wikidata:Statistics. A large subset
of the information extraction community has extensively relied on its research
around DBpedia and Wikidata targeting different research problems such as KG
completion, question answering, entity linking, and data quality assessments
(Moon et al., 2017; Reinanda et al., 2016; Yang et al., 2013). Furthermore,
entity and relation linking tasks have been studied well beyond information
extraction research, especially NLP and Semantic Web. Despite Wikidata being
hugely popular, there are limited resources for reusing and aligning
unstructured text to Wikidata mentions. However, when it comes to a short
text, the performance of existing baselines are limited. We believe the
availability of Falcon 2.0 as a web API along with open source access to its
code will provide researchers an easy and reusable way to annotate
unstructured text against Wikidata. We also believe that a rule-based
approach, such as ours that does not require any training data, is beneficial
for low resource languages (considering Wikidata is multilingual
121212https://www.wikidata.org/wiki/Help:Wikimedia_language_codes/lists/all).

##  6\. Adoption and Reusability

Falcon 2.0 is open source. The source code is available in our public GitHub:
https://github.com/SDM-TIB/Falcon2.0 for reusability and reproducibility.
Falcon 2.0 is easily accessible via a simple CURL request or using our web
interface. Detailed instructions are provided on our GitHub. It is currently
available for the English language. However, there is no assumption in the
approach or while building the background knowledge base that restricts its
adaptation or extensibility to other languages. The background knowledge of
Falcon 2.0 is available for the community and can be easily reused to generate
candidates for entity linking (Yamada et al., 2016b) or in question answering
approaches such as (Zhang and Zou, 2018). The background knowledge consists of
48,042,867 alignments of Wikidata entities and 15,645 alignments for Wikidata
predicates. MIT License allows for the free distribution and re-usage of
Falcon 2.0. We hope the research community and industry practitioners will use
Falcon 2.0 resources for various usages such as linking entities and relations
to Wikidata, annotating an unstructured text, developing new low language
resources, and others.

##  7\. Maintenance and Sustainability

Falcon 2.0 is a publicly available resource offering of the Scientific Data
Management(SDM) group at TIB, Hannover131313https://www.tib.eu/en/research-
development/scientific-data-management/. TIB is one of the largest libraries
for Science and Technology in the world
141414https://www.tib.eu/en/tib/profile/. It actively promotes open access to
scientific artifacts, e.g., research data, scientific literature, non-textual
material, and software. Similar to other publicly maintained repositories of
SDM, Falcon 2.0 will be preserved and regularly updated to fix bugs and
include new features151515https://github.com/SDM-TIB. The Falcon 2.0 API will
be sustained on the TIB servers to allow for unrestricted free access.

##  8\. Conclusion and Future Work

We presented the resource Falcon 2.0, a rule-based entity and relation linking
tool able to recognize entities & relations in a short text and link them to
the existing knowledge graph, e.g., DBpedia and Wikidata. Although there are
various approaches for entity & relation linking to DBpedia, Falcon 2.0 is one
of the few tools targeting Wikidata. Thus, given the number of generic and
domain-specific facts that compose Wikidata, Falcon 2.0 has the potential to
impact researchers and practitioners that resort to NLP tools for transforming
semi-structured data into structured facts. Falcon 2.0 is open source. The API
is publicly accessible and maintained in the servers of the TIB labs. Falcon
2.0 has been empirically evaluated on three benchmarks, and the outcomes
suggest that it is able to overcome the state of the arts. Albeit promising,
the experimental results can be improved. In the future, we plan to continue
researching novel techniques that enable adjusting the catalog of rules and
alignments to the changes in Wikidata. We further plan to mitigate errors
caused by a rule-based approach using machine learning approaches to aim
towards a hybrid approach.

##  9\. Acknowledgments

This work has received funding from the EU H2020 Project No. 727658 (IASIS).

## References

  * (1)
  * Auer et al. (2007) Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. 2007.  DBpedia: A Nucleus for a Web of Open Data. In _ISWC_. 722–735.
  * Balog (2018) Krisztian Balog. 2018\.  _Entity-oriented search_.  Springer Open.
  * Banerjee et al. ([n.d.]) Debayan Banerjee, Mohnish Dubey, Debanjan Chaudhuri, and Jens Lehmann. [n.d.].  Joint Entity and Relation Linking using EARL.  ([n. d.]).
  * Bollacker et al. (2008) Kurt D. Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008.  Freebase: a collaboratively created graph database for structuring human knowledge. In _ACM SIGMOD_. 1247–1250.
  * Cao et al. (2018) Yixin Cao, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2018\.  Neural Collective Entity Linking.  arXiv:1811.08603 http://arxiv.org/abs/1811.08603
  * Cetoli et al. (2019) Alberto Cetoli, Stefano Bragaglia, Andrew D O’Harney, Marc Sloan, and Mohammad Akbari. 2019\.  A Neural Approach to Entity Linking on Wikidata. In _European Conference on Information Retrieval_. Springer, 78–86.
  * Delpeuch (2019) Antonin Delpeuch. 2019\.  OpenTapioca: Lightweight Entity Linking for Wikidata.  _arXiv preprint arXiv:1904.09131_ (2019).
  * Diefenbach et al. (2017) Dennis Diefenbach, Thomas Tanon, Kamal Singh, and Pierre Maret. 2017\.  Question answering benchmarks for wikidata.
  * Dubey et al. (2019) Mohnish Dubey, Debayan Banerjee, Abdelrahman Abdelkawi, and Jens Lehmann. 2019.  Lc-quad 2.0: A large dataset for complex question answering over wikidata and dbpedia. In _International Semantic Web Conference_. Springer, 69–78.
  * Ferragina and Scaiella (2010) Paolo Ferragina and Ugo Scaiella. 2010.  TAGME: on-the-fly annotation of short text fragments (by wikipedia entities). In _Proceedings of the 19th ACM Conference on Information and Knowledge Management, CIKM 2010, Toronto, Ontario, Canada, October 26-30, 2010_. 1625–1628.
  * Ganea and Hofmann (2017) Octavian-Eugen Ganea and Thomas Hofmann. 2017.  Deep Joint Entity Disambiguation with Local Neural Attention. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_. 2619–2629.
  * Gormley and Tong (2015) Clinton Gormley and Zachary Tong. 2015.  _Elasticsearch: The Definitive Guide: A Distributed Real-Time Search and Analytics Engine_.  ” O’Reilly Media, Inc.”.
  * Heindorf et al. (2016) Stefan Heindorf, Martin Potthast, Benno Stein, and Gregor Engels. 2016.  Vandalism detection in wikidata. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_. 327–336.
  * Hoffart et al. (2011) Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011\.  Robust Disambiguation of Named Entities in Text. In _EMNLP 2011_. 782–792.
  * Inan and Dikenelli (2018) Emrah Inan and Oguz Dikenelli. 2018.  A Sequence Learning Method for Domain-Specific Entity Linking. In _Proceedings of the Seventh Named Entities Workshop_ (Melbourne, Australia). Association for Computational Linguistics, 14–21.  http://aclweb.org/anthology/W18-2403
  * Ji (2019) Heng Ji. 2019.  _Entity Discovery and Linking and Wikification Reading List_.  http://nlp.cs.rpi.edu/kbp/2014/elreading.html
  * Kolitsas et al. (2018) Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas Hofmann. 2018.  End-to-End Neural Entity Linking. In _Proceedings of the 22nd Conference on Computational Natural Language Learning_. 519–529.
  * Moon et al. (2017) Changsung Moon, Paul Jones, and Nagiza F Samatova. 2017\.  Learning entity type embeddings for knowledge graph completion. In _Proceedings of the 2017 ACM on conference on information and knowledge management_. 2215–2218.
  * Mulang et al. (2020) Isaiah Onando Mulang, Kuldeep Singh, Akhilesh Vyas, Saeedeh Shekarpour, Ahmad Sakor, Maria Esther Vidal, Soren Auer, and Jens Lehmann. 2020.  Encoding Knowledge Graph Entity Aliases in an Attentive Neural Networks for Wikidata Entity Linking.  _In WISE (to appear)_ (2020).
  * Raiman and Raiman (2018) Jonathan Raphael Raiman and Olivier Michel Raiman. 2018.  DeepType: multilingual entity linking by neural type system evolution. In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Reinanda et al. (2016) Ridho Reinanda, Edgar Meij, and Maarten de Rijke. 2016\.  Document Filtering for Long-tail Entities. In _Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016_. ACM, 771–780.  https://doi.org/10.1145/2983323.2983728
  * Röder et al. (2018) Michael Röder, Ricardo Usbeck, and Axel-Cyrille Ngonga Ngomo. 2018\.  Gerbil–benchmarking named entity recognition and linking consistently.  _Semantic Web_ 9, 5 (2018), 605–625.
  * Sakor et al. (2019) Ahmad Sakor, Isaiah Onando Mulang, Kuldeep Singh, Saeedeh Shekarpour, Maria Esther Vidal, Jens Lehmann, and Sören Auer. 2019\.  Old is gold: linguistic driven approach for entity and relation linking of short text. In _Proceedings of the 2019 NAACL HLT (Long Papers)_. 2336–2346.
  * Shen et al. (2015) W. Shen, J. Wang, and J. Han. 2015.  Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions.  _IEEE Transactions on Knowledge and Data Engineering_ 27, 2 (2015), 443–460.
  * Singh et al. (2018a) Kuldeep Singh, Ioanna Lytra, Arun Sethupat Radhakrishna, Saeedeh Shekarpour, Maria-Esther Vidal, and Jens Lehmann. 2018a.  No One is Perfect: Analysing the Performance of Question Answering Components over the DBpedia Knowledge Graph.  _arXiv:1809.10044_ (2018).
  * Singh et al. (2018b) Kuldeep Singh, Arun Sethupat Radhakrishna, Andreas Both, Saeedeh Shekarpour, Ioanna Lytra, Ricardo Usbeck, Akhilesh Vyas, Akmal Khikmatullaev, Dharmen Punjani, Christoph Lange, Maria-Esther Vidal, Jens Lehmann, and Sören Auer. 2018b.  Why Reinvent the Wheel: Let’s Build Question Answering Systems Together. In _Web Conference_. 1247–1256.
  * Sorokin and Gurevych (2018) Daniil Sorokin and Iryna Gurevych. 2018.  Mixing Context Granularities for Improved Entity Linking on Question Answering Data across Entity Categories. In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_. 65–75.
  * Vrandecic (2012) Denny Vrandecic. 2012\.  Wikidata: a new platform for collaborative data collection. In _Proceedings of the 21st World Wide Web Conference, WWW 2012, Lyon, France, April 16-20, 2012 (Companion Volume)_. ACM, 1063–1064.  https://doi.org/10.1145/2187980.2188242
  * Williams (1981) Edwin Williams. 1981\.  On the notions” Lexically related” and” Head of a word”.  _Linguistic inquiry_ 12, 2 (1981), 245–274.
  * Yamada et al. (2016a) Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016a.  Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation. In _CoNLL 2016_ , Yoav Goldberg and Stefan Riezler (Eds.). ACL, 250–259.
  * Yamada et al. (2016b) Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016b.  Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation.  _CoRR_ abs/1601.01343 (2016).
  * Yang et al. (2019) Xiyuan Yang, Xiaotao Gu, Sheng Lin, Siliang Tang, Yueting Zhuang, Fei Wu, Zhigang Chen, Guoping Hu, and Xiang Ren. 2019.  Learning Dynamic Context Augmentation for Global Entity Linking. In _EMNLP-IJCNLP 2019_ , Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). 271–281.
  * Yang and Chang (2015) Yi Yang and Ming-Wei Chang. 2015.  S-MART: Novel Tree-based Structured Learning Algorithms Applied to Tweet Entity Linking. In _ACL- IJCNLP (Volume 1: Long Papers)_. 504–513.
  * Yang et al. (2013) Zi Yang, Elmer Garduño, Yan Fang, Avner Maiberg, Collin McCormack, and Eric Nyberg. 2013\.  Building optimal information systems automatically: configuration space exploration for biomedical information systems. In _22nd ACM CIKM’13, San Francisco, USA_. ACM, 1421–1430.
  * Zhang and Zou (2018) Xinbo Zhang and Lei Zou. 2018.  IMPROVE-QA: An Interactive Mechanism for RDF Question/Answering Systems. In _Proceedings of the 2018 International Conference on Management of Data, SIGMOD Conference 2018, Houston, TX, USA, June 10-15, 2018_. 1753–1756.  https://doi.org/10.1145/3183713.3193555