# Virtual Embodiment: A Scalable Long-Term Strategy for Artificial
Intelligence Research

Douwe Kiela
Facebook AI Research
New York, NY 10017, USA
<EMAIL_ADDRESS>
Luana Bulat, Anita L. Verő, Stephen Clark
Computer Laboratory, University of Cambridge
Cambridge CB3 0FD, UK
<EMAIL_ADDRESS>

##  1 Introduction

Meaning has been called the “holy grail” of a variety of scientific
disciplines, ranging from linguistics to philosophy, psychology and the
neurosciences [1]. The field of Artifical Intelligence (AI) is very much a
part of that list: the development of sophisticated natural language semantics
is a _sine qua non_ for achieving a level of intelligence comparable to
humans. Embodiment theories in cognitive science hold that human semantic
representation depends on sensori-motor experience [2]; the abundant evidence
that human meaning representation is grounded in the perception of physical
reality leads to the conclusion that meaning must depend on a fusion of
multiple (perceptual) modalities [3]. Despite this, AI research in general,
and its subdisciplines such as computational linguistics and computer vision
in particular, have focused primarily on tasks that involve a single modality.
Here, we propose _virtual embodiment_ as an alternative, long-term strategy
for AI research that is multi-modal in nature and that allows for the kind of
scalability required to develop the field coherently and incrementally, in an
ethically responsible fashion.

Embodiment theory implies that the best way for acquiring human-level
semantics is to have machines learn through (physical) experience: if we want
to teach a system the true meaning of “bumping into a wall”, we simply have to
have it bump into walls repeatedly. Although this scenario shares similarities
with human language acquisition, it is not (yet) a viable route: our current
machine learning paradigms do not allow for the required rate of learning to
make such a scenario feasible. With modern day state-of-the-art deep learning
systems requiring millions of samples to solve highly specific tasks that are
trivial to humans, it is reasonable to speculate that it would take much
longer than a human lifespan for a _physically_ embodied agent to develop
extensive linguistic capabilities, with current technology. We conjecture that
such limitations apply to a much lesser extent to an agent that is _virtually_
embodied.

By virtual embodiment we mean to say that agents may collectively or
individually acquire semantics by being embodied in a virtual, rather than a
physical, world. Concretely, rather than having a physical robot learn to
understand the world by physically bumping into physical walls, we would have
virtual agents bump into virtual walls in a virtual world. Such virtual
embodiment offers several key advantages:

  1. 1.

Scalability and incremental development: The complexity of virtual worlds can
develop in conjunction, i.e., scale, with the capabilities of artificial
agents. This allows for a stepwise development towards general machine
intelligence, rather than aiming for the end-goal without a concrete
understanding of the challenges or consequences we will face when attempting
to reach that end-goal.

  2. 2.

Long-term feasibility: The performance ceiling of any agent is a function of
the complexity of the virtual environment. Virtual worlds may initially not be
overly complex, but they can grow in complexity as technology develops. This
allows for a focused long-term research strategy that is feasible now, but
will remain challenging in years and decades to come.

  3. 3.

Rapid iteration: The fact that artificial agents are constrained by arbitrary
parameters means that development can happen rapidly and iteratively, through
agents learning from interacting both with humans and with each other. Rather
than the extremes of either having one system solve small uni-modal tasks, or
instead trying to solve the whole problem in a single attempt, we can improve
iteratively, in an agile fashion, at great speed.

  4. 4.

No requirement for continuous human involvement: Although interaction is
necessary for embodied learning, virtual interactions need not require human
involvement at each step, but may rather happen between agents themselves.
This unburdens humans by foregoing the need for a constant supervised signal,
as is currently often seen in machine learning applications, which also
facilitates rapid development.

  5. 5.

Ethical testability: Importantly, since artificial agents are exposed to a
constrained environment, virtual worlds provide the ultimate testing ground
for carefully fleshing out important ethical considerations in relation to
artificial intelligence [4], without any potentially damaging immediate
consequences in the physical world.

For these reasons, we propose virtual embodiment as one of the best and most
feasible strategies for instigating a stepwise development towards artificial
general intelligence. In particular, we advocate the development and use of
“video games with a purpose” to facilitate virtual embodiment. In what
follows, we briefly outline some of the background that led to this proposal,
explain why video games are suitable for the current purposes and list the
desiderata for virtual embodiment-compatible video games to facilitate
research in artificial intelligence.

##  2 Grounding Semantics in Virtual Perception

A fundamental problem of semantics is the grounding problem [5], which
concerns the circularity in defining the meaning of a symbol through other
symbols. In the context of Searle’s famous Chinese Room argument [6], it can
be phrased as: is it possible to learn Chinese from nothing but a (very
sophisticated) Chinese dictionary? Modern representation learning approaches,
including the word embeddings that have become popular in natural language
processing, are exponents of the distributional hypothesis [7], which
stipulates that you “shall know the meaning of a word through the company that
it keeps” [8]. In other words, semantic representation learning defines
symbols through other symbols, which exposes it to the grounding problem. In
contrast, there is abundant evidence that human meaning representation is
grounded in physical reality and sensorimotor experience [9, 10, 11, 12, 13].

Motivated by these theoretical considerations, the field of multi-modal
semantics aims to ground semantic representations by introducing extra-
linguistic, perceptual input into semantic models. Multi-modal semantic models
lead to practical improvements in a variety of natural language processing
tasks, ranging from resolving linguistic ambiguity [14] to metaphor detection
[15]. Beyond vision, there has also been work aimed towards auditory [16, 17]
and even olfactory [18] grounding. However, most current multi-modal semantic
models suffer from two important limitations. First, images and to a lesser
extent sound files lack the element of _time_ , whereas temporal and
sequential input are central aspects of language understanding. Second, these
approaches lack any _interaction_ , which plays an important role in language
acquisition: children learn basic language understanding by interacting with
the environment, and build more intricate “reflective reasoning” on top of
that foundation [19].

There has been work in linguistic grounding that allows for temporal aspects,
for instance in videos [20, 21, 22], and both time and interaction, notably in
the field of robotics [23, 24, 25]. However, robotics does not currently
constitute a suitable platform for language learning, since physical
embodiment is not yet feasible. Virtual embodiment does not suffer from the
same limitations. There has been recent work on grounding in virtual worlds,
notably in video games [26]. Work applying deep reinforcement learning to
video games points the way towards agents learning from each other [27, 28,
29]. An alternative would be virtual or augmented reality, which offers the
benefit of joint multi-modal data over time, but this crucially lacks the
element of interaction.

Our position is very much aligned with recent proposals for new directions in
AI research [30, 31, 32, 33]. The particular problem of language features in
these proposals to a varying extent, but we take it to be a core piece of any
path toward artificial general intelligence, in line with recent attempts to
make machines genuinely understand human language [34, 35]. We specifically
advocate multi-agent video games “with a purpose” [36], rather than
alternative virtual worlds that lack gamification, since they provide
interesting platforms for humans to engage with for extended periods of time,
without the explicit purpose of teaching machines to achieve a certain task.

##  3 Desiderata

It is worthwhile outlining the properties that video games might have if they
are to be suitable platforms for developing AI through virtual embodiment. For
that purpose, we propose a hierarchy111Inspired by the Kardashev scale for the
sophistication of civilizations [37]. of the types of embodied manifestations
an agent might have in a world. The same type hierarchy applies both to
physical and to virtual worlds:

  * •

Type 0: Agents perform basic first-order interactions with the world, with
full or limited access to the objective world state. No intra-agent
communication is required.

  * •

Type 1: As above, but without any state access. Communication may be used for
sharing knowledge about the state of the world.

  * •

Type 2: As above, but with higher-order interactions, i.e., with an element of
planning, strategy and non-monotonic reasoning. Communication is essential for
sharing knowledge about the world.

  * •

Type 3: The world should be strictly non-deterministic and multi-modal. This
makes communication essential for not only sharing knowledge about the world,
but also for sharing plans and strategies.

  * •

Type 4: Agents should be multi-objective, that is, an agent’s objective or
reward function should be a weighted function of various objectives or
rewards, that depend both on the state of the world and current plans and
strategy.

  * •

Type 5: Multi-objective agents interact with and communicate about a non-
deterministic world in such a way that it allows for them to plan ahead and
form and execute sophisticated strategies.

The final type of embodiment corresponds to what biological agents are capable
of performing in the physical world. It is much too large a leap for current
technologies to achieve, but the benefit of virtual embodiment is that we can
grow the complexity of the world together with the sophistication of
artificial agents, which makes virtual embodiment suitable for being AI’s next
frontier. The real world is enormously complex, and performing common sense
reasoning in such a complicated environment has long been one of AI’s classic
problems in the shape of the frame problem [38]. The frame problem is a
function of the world’s complexity, which makes it more manageable for virtual
embodiment.

Most recent work has not extended beyond Type 1 embodiment, which means that
the field has a long way to go. Specifically in the context of video games, we
believe that development can proceed more rapidly if they are of mixed agency,
meaning that both humans and artificial systems control agents in the virtual
world; and carefully designed as a level playing field with a human bias, such
that human agents have a slight upper hand, which means that e.g. the superior
memory of machines should not affect in-game performance and that machines can
learn from humans. To our knowledge no video game currently exists that
satisfies these properties and which facilitates Type 5 embodiment.

##  4 Conclusion

We propose virtual embodiment, through video games, as a scalable long-term
strategy for artificial intelligence research. Embodiment is essential for
developing human-level natural language semantics, which we take to be a core
aspect of artificial intelligence. Virtual embodiment allows for growing the
complexity of virtual worlds in line with the sophistication of artificial
agents, which makes it a suitable testing ground for artificial intelligence,
in an ethically responsible manner.

### Acknowledgments

This research was enabled by the European Research Council PoC grant
GroundForce.

## References

  * [1] R. Jackendoff.  Foundations of Language.  Oxford University Press, Oxford, 2002.
  * [2] Lawrence W. Barsalou.  Grounded cognition.  Annual Review of Psychology, 59(1):617–645, 2008.
  * [3] Lotte Meteyard and Gabriella Vigliocco.  The role of sensory and motor information in semantic representation: A review.  Handbook of cognitive science: An embodied approach, pages 293–312, 2008.
  * [4] Nick Bostrom.  Ethical issues in advanced artificial intelligence.  Science Fiction and Philosophy: From Time Travel to Superintelligence, pages 277–284, 2003.
  * [5] Stevan Harnad.  The symbol grounding problem.  Physica D, 42:335–346, 1990.
  * [6] John R. Searle.  Minds, brains and programs.  Behavioral and Brain Sciences, 3(3):417–57, 1980.
  * [7] Z. Harris.  Distributional Structure.  Word, 10(23):146—162, 1954.
  * [8] John R. Firth.  A synopsis of linguistic theory.  Blackwell, 1957.
  * [9] Susan S Jones, Linda B Smith, and Barbara Landau.  Object properties and knowledge in early lexical learning.  Child development, 62(3):499–516, 1991.
  * [10] Lawrence W Barsalou.  Perceptions of perceptual symbols.  Behavioral and brain sciences, 22(04):637–660, 1999.
  * [11] Arthur M Glenberg and Michael P Kaschak.  Grounding language in action.  Psychonomic bulletin & review, 9(3):558–565, 2002.
  * [12] Max M. Louwerse.  Symbol interdependency in symbolic and embodied cognition.  Topics in Cognitive Science, 59(1):617–645, 2008.
  * [13] Lawrence W. Barsalou and Katja Wiemer-Hastings.  Situating abstract concepts.  In Grounding cognition: The role of perception and action in memory, language, and thought, pages 129–163. Cambridge UP, 2005.
  * [14] Yevgeni Berzak, Andrei Barbu, Daniel Harari, Boris Katz, and Shimon Ullman.  Do You See What I Mean? Visual Resolution of Linguistic Ambiguities.  In Proceedings of EMNLP 2015, Lisbon, Portugal, 2015.
  * [15] Ekaterina Shutova, Douwe Kiela, and Jean Maillard.  Black holes and white rabbits: Metaphor identification with visual features.  In Proceedings of NAACL-HTL 2016, San Diego, 2016.
  * [16] A. Lopopolo and E. van Miltenburg.  Sound-based distributional models.  In Proceedings of the 11th International Conference on Computational Semantics (IWCS 2015), 2015.
  * [17] Douwe Kiela and Stephen Clark.  Multi- and cross-modal semantics beyond vision: Grounding in auditory perception.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2461–2470, Lisbon, Portugal, September 2015\.
  * [18] Douwe Kiela, Luana Bulat, and Stephen Clark.  Grounding semantics in olfactory perception.  In Proceedings of ACL, pages 231–236, Beijing, China, July 2015\.
  * [19] Barbara Landau, Linda Smith, and Susan Jones.  Object perception and object naming in early development.  Trends in cognitive sciences, 2(1):19–24, 1998.
  * [20] Arpan Gupta, Praveen Srinivasan, Jianbo Shi, and Larry S Davis.  Understanding videos, constructing plots learning a visually grounded storyline model from annotated videos.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009, pages 2012–2019. IEEE, 2009.
  * [21] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal.  Grounding action descriptions in videos.  Transactions of the Association for Computational Linguistics, 1:25–36, 2013.
  * [22] Haonan Yu, N. Siddharth, Andrei Barbu, and Jeffrey Mark Siskind.  A compositional framework for grounding language inference, generation, and acquisition in video.  Journal of Artificial Intelligence Research, 52(1):601–713, 2015\.
  * [23] Paul Fitzpatrick and Giorgio Metta.  Grounding vision through experimental manipulation.  Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 361(1811):2165–2185, 2003.
  * [24] Silvia Coradeschi, Amy Loutfi, and Britta Wrede.  A short review of symbol grounding in robotic and intelligent systems.  KI-Künstliche Intelligenz, 27(2):129–136, 2013.
  * [25] Jonathan Bisk, Deniz Yuret, and Daniel Marcu.  Natural language communication with robots.  In Proceedings of NAACL, San Diego, CA, 2016.
  * [26] Karthik Narasimhan, Tejas D Kulkarni, and Regina Barzilay.  Language understanding for textbased games using deep reinforcement learning.  In Proceedings of EMNLP, 2015.
  * [27] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533, 2015.
  * [28] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  Nature, 529(7587):484–489, 2016.
  * [29] Angeliki Lazaridou, Nghia The Pham, and Marco Baroni.  Towards multi-agent communication-based language learning.  CoRR, abs/1605.07133, 2016.
  * [30] Tomas Mikolov, Armand Joulin, and Marco Baroni.  A roadmap towards machine intelligence.  arXiv preprint arXiv:1511.08130, 2015.
  * [31] Sainbayar Sukhbaatar, Arthur Szlam, Gabriel Synnaeve, Soumith Chintala, and Rob Fergus.  Mazebase: A sandbox for learning from games.  arXiv preprint arXiv:1511.07401, 2015.
  * [32] Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov.  Towards ai-complete question answering: A set of prerequisite toy tasks.  arXiv preprint arXiv:1502.05698, 2015.
  * [33] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell.  The malmo platform for artificial intelligence experimentation.  In International joint conference on artificial intelligence (IJCAI), 2016.
  * [34] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom.  Teaching machines to read and comprehend.  In Advances in Neural Information Processing Systems, pages 1693–1701, 2015.
  * [35] Sida I. Wang, Percy Liang, and Christopher D. Manning.  Learning language games through interaction.  In Proceedings of ACL 2016, Berlin, Germany, 2016.
  * [36] Luis von Ahn and Laura Dabbish.  Labeling images with a computer game.  In CHI, pages 319–326, 2004.
  * [37] Nikolai S Kardashev.  Transmission of information by extraterrestrial civilizations.  Soviet Astronomy, 8:217, 1964.
  * [38] John McCarthy and Patrick J Hayes.  Some philosophical problems from the standpoint of artificial intelligence.  Readings in artificial intelligence, pages 431–450, 1969.


# Modern Baselines for SPARQL Semantic Parsing

Debayan Banerjee<EMAIL_ADDRESS>Universität
HamburgHamburgGermany ,  Pranav Ajit Nair<EMAIL_ADDRESS>Indian Institute of Technology (BHU)VaranasiIndia ,  Jivat Neet Kaur
<EMAIL_ADDRESS>Microsoft ResearchBengaluruIndia ,  Ricardo Usbeck
<EMAIL_ADDRESS>Universität HamburgHamburgGermany and  Chris
Biemann<EMAIL_ADDRESS>Universität HamburgHamburgGermany

(2022)

###### Abstract.

In this work, we focus on the task of generating SPARQL queries from natural
language questions, which can then be executed on Knowledge Graphs (KGs). We
assume that gold entity and relations have been provided, and the remaining
task is to arrange them in the right order along with SPARQL vocabulary, and
input tokens to produce the correct SPARQL query. Pre-trained Language Models
(PLMs) have not been explored in depth on this task so far, so we experiment
with BART, T5 and PGNs (Pointer Generator Networks) with BERT embeddings,
looking for new baselines in the PLM era for this task, on DBpedia and
Wikidata KGs. We show that T5 requires special input tokenisation, but
produces state of the art performance on LC-QuAD 1.0 and LC-QuAD 2.0 datasets,
and outperforms task-specific models from previous works. Moreover, the
methods enable semantic parsing for questions where a part of the input needs
to be copied to the output query, thus enabling a new paradigm in KG semantic
parsing. Code and data used for this work can be found at
https://github.com/debayan/sigir2022-sparqlbaselines.

Knowledge Graph, Question Answering, Semantic Parsing, SPARQL

††journalyear: 2022††copyright: acmlicensed††conference: Proceedings of the
45th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 11–15, 2022; Madrid, Spain††booktitle: Proceedings
of the 45th International ACM SIGIR Conference on Research and Development in
Information Retrieval (SIGIR ’22), July 11–15, 2022, Madrid, Spain††price:
15.00††doi: 10.1145/3477495.3531841††isbn: 978-1-4503-8732-3/22/07††ccs:
Information systems Question answering

##  1\. Introduction

Knowledge Graph Question Answering (KGQA) is the task of finding answers to
questions posed in natural language, using triples present in a KG. Typically
the following steps are followed in KGQA: 1) Objects of interest in the
natural language question (NLQ) are detected and linked to the KG in a step
called entity linking (EL). 2) The relation between the objects is discovered
and linked to the KG in a step called relation linking (RL). 3) A formal
query, usually SPARQL, is formed with the linked entities and relations. The
query is executed on the KG to fetch the answer. This step is called Query
Building (QB), and is the focus of this paper. We assume that gold entities
and relations are made available, and our task is to generate the correct
SPARQL query. We experiment with a Pointer Generator Network (See et al.,
2017) with special vectorisation, and with more recent pre-trained language
models such as T5 (Raffel et al., 2020) and BART (Lewis et al., 2020). We
choose models that are able to copy tokens from the input text to the output
SPARQL query, apart from being able to handle normal category of questions.
Questions which require copy operations are found in the dataset LC-QuAD 2.0
(Dubey et al., 2019), for example:

Is it true that an Olympic-size swimming pool’s operating temperature is equal
to 22.4 ?

which has the corresponding SPARQL over Wikidata KG (Vrandečić, Denny and
Krötzsch, Markus, 2014):

    ASK WHERE
    {
      wd:Q2084454 wdt:P5066 ?obj
      filter(?obj = 22.4)
    }

We found no previous work for SPARQL QB that can handle such questions. We
also show that on datasets where copying is not required (LC-QuAD 1.0 (Trivedi
et al., 2017) ), the approaches still exhibit a strong performance.

Our contribution is as follows:

  * •

We find that with the correct input tokenisation, T5 outperforms all previous
works and achieves state of the art performance on LC-QuAD 1.0 over DBpedia
(Lehmann et al., 2015), and LC-QuAD 2.0 over Wikidata.

##  2\. Related Work

Diefenbach et al. (2018) studied several QA systems and concluded that the QB
step is generally intertwined with rest of the modules in a pipeline, and
hence not evaluated separately. Later, with the advent of Frankenstein (Singh
et al., 2018a) and Qanary (Singh et al., 2016), which are frameworks that
allow modular construction of an end-to-end KGQA pipeline, it was seen that
the QB module is the least researched aspect of the pipeline. Soon after,
Singh et al. (2018b) presented a comprehensive survey of individual components
of KGQA pipeline and evaluated Sina (Shekarpour et al., 2015) and
NLIWOD111https://github.com/semantic-systems/NLIWOD/ as individual QB
components.
Several recent works attempt to solve complex query building by constructing
intermediate query representations, such as staged query graphs (Yih et al.,
2015; Hu et al., 2018; Luo et al., 2018; Chen et al., 2020; Lan and Jiang,
2020) based on $\lambda$-calculus or skeleton structures (Sun et al., 2020).
These systems have no built-in method of performing copy operations from input
text to output query. The system based on skeleton structure requires manual
annotation of queries to corresponding skeletal structures, apart from the
final generated query. Some other solutions rely on templates (Abujabal et
al., 2017; Ding et al., 2019; Soru et al., 2017), which generally have the
natural limitation of the query only being limited to the templates chosen
beforehand (Athreya et al., 2021; Vollmers et al., 2021). In the work of Ding
et al. (2019), the templates are discovered from the input training data,
however, the implementation of the method of discovery of such templates,
structures, and sub-structures in the queries is dataset specific. It is
unclear if the same method of discovery can scale to all datasets.
Most systems that work over query graphs and use $\lambda$-calculus start with
the assumption that entities are already linked, and the relations must still
be found. However, with the availability of joint entity and relation linkers
such as EARL (Dubey et al., 2018) and Falcon (Sakor et al., 2019) , we can
develop semantic parsers which focus only on query building and receive both
entities and relations pre-linked.
For non-KG semantic parsing, PLMs have been evaluated recently with a focus on
compositional generalisation (Shaw et al., 2021). For KG semantic parsing, the
Compositional Freebase Questions (CFQ) (Keysers et al., 2020) dataset that is
based on the Freebase KG (Bollacker et al., 2008), has reported results with
non-pre-trained Transformer-based models, while subsequent works (Herzig et
al., 2021) (Furrer et al., 2020) over the same dataset have experimented with
PLMs. This line of work may be considered closest to ours, however, their
focus is on compositional generalisation, while our focus is on the ability to
faithfully copy input tokens to the produced query. CFQ has no questions that
require such copy operations. Moreover, since Freebase is no longer an active
project, our focus is on DBPedia and Wikidata. By exploring two new datasets
and corresponding KGs, we expand the scope of research.
Some recent work on Complex Temporal Question Answering (Jia et al., 2021)
handled questions that require extraction of time stamps from the input
question. They use task-specific temporal extraction tools such as SUTime
(Chang and Manning, 2012) and HeidelTime (Strötgen and Gertz, 2010) for this
purpose, however in the end, they do not form a SPARQL query, and instead
attempt to find the correct answer directly through an entity re-ranking
approach. Our efforts are instead on finding such flexible single models which
can perform such extraction tasks along with the building of SPARQl queries.
The models we experiment with pose no structural or template-based constraints
neither at an intermediate stage nor during decoding, and are able to produce
any possible query structure since we generate the final query token by token.
They require the input question, the linked entities and relations, and a
fixed vocabulary of all possible SPARQL tokens. This allows them to operate on
all possible datasets and KGs.

Figure 1. PGN-based QB model. At the current time step, the model is decoding
the symbol after the single quote character (’). It considers the scores over
the vocabulary and the attention weights over the input text to obtain a final
probability distribution, from which it makes the prediction of choosing 1960
as the next token.

##  3\. Models

###  3.1. T5

T5, or Text-to-Text Transfer Transformer, is a Transformer (Vaswani et al.,
2017) based encoder-decoder architecture that uses a text-to-text approach.
Every task, including translation, question answering, and classification, is
cast as feeding the model text as input and training it to generate some
target text. This allows for the use of the same model, loss function, hyper-
parameters, etc. across a diverse set of tasks. We experiment with two
variants of the T5 model: T5-small with 60 million parameters and T5-base with
220 million parameters. The uncompressed C4 corpus used to pre-train the
models is 750 GB in size.

####  3.1.1. Input Tokenisation

Let the natural language input question be denoted by $Q=[w1,w2...wn]$ where
$w$ denotes the words in the question. The linked entities be denoted by
$E=[E1,E2...En]$ and the relations by $R=[R1,R2....Rn]$. For each entity and
relation we fetch the corresponding label from the respective KG such that
$E_{{lab}_{1}}$ denotes the label for $E_{1}$, and similarly for relations.
Let the SPARQL vocabulary be donated by $V$. The input string to the model is
formed as follows:

$w_{1}w_{2}...w_{n}[SEP]E_{1}E_{{lab}_{1}}....E_{n}E_{{lab}_{n}}[SEP]R_{1}R_{{lab}_{1}}...R_{n}R_{{lab}_{n}}$

where each word is separated from the other word by a space. We denote the
entities by their respective KG IRIs. For DBpedia the entities take the form,
e.g. ” http://dbpedia.org/resource/Dolley_ Madison” while the relations look
like ” http://dbpedia.org/onto logy/spouse”. For Wikidata a typical entity
looks like wd:Q76 (Barack Obama) while a typical relation looks like wdt:P31
(instance of). The prefixes wd:, wdt: and several others are expanded
according to the list available
online222https://www.wikidata.org/wiki/EntitySchema:E49. We shuffle the order
of entities and relations randomly before passing it to the model.
When the input string is passed to T5 as it is, the accuracy is close to zero,
as the AutoTokenizer splits the URLs as it sees fit, and it later fails to
concatenate the fragments properly at the output. To work around this problem
for DBpedia, we made use of the 100 sentinel tokens that T5 provides. The
sentinel tokens were originally used during pre-training objective to denote
masked tokens, but in our case, we make use of them to represent the prefixes,
e.g, http://dbpedia.org/ontology/ as a specific sentinel token such as
<extra_id_2>. Additionally, we represent each of the items in the SPARQL
vocabulary $V$ as a sentinel token with a specific ID for each keyword. We do
the same for prefixes in Wikidata for LC-QuAD 2.0.

####  3.1.2. Output

The output is composed of tokens from $Q,V,E$ and $R$. In the case of LC-QuAD
1.0, since there are no questions that require copying of tokens from Q, the
output is composed of tokens from $V,E$ and $R$. We would like to point out
that since our input contains the entities and relations, the model always has
to perform a form of ”copying” even to decide where in the output SPARQL query
to place the entities and relations. In the case of questions from LC-QuAD
2.0, in some cases this copying mechanism must also decide which input token
from $Q$ to copy and to which position in the output. The output for $E$ and
$R$ are not produced verbatim, and instead they are split as a special token
prefix, and then the entity or relation ID. A post-processing step is required
to resolve these sentinel token IDs into the original prefixes and combine
them with the adjacent IDs. For example, wdt:P31 is produced in the output as
<extra_id_3> P31.

###  3.2. BART

The Bidirectional and Auto-Regressive Transformer or BART is a Transformer
that combines the Bidirectional Encoder with an Autoregressive decoder into
one Seq2Seq model. We experiment with the BART-base model consisting of 139
million trainable parameters. This model was pre-trained on 160 GB of data
resulting from the combination of four corpuses, namely, Bookcorpus (Zhu et
al., 2015), CC-News (Mackenzie et al., 2020), OpenWebText (Gokaslan and Cohen,
2019) and Stories (Trinh and Le, 2018).

####  3.2.1. Input Tokenisation

We form the input string for BART in the same manner as discussed in Section
3.1.1 for T5. However there is no concept of sentinel tokens in BART. Hence to
handle special tokens, we instead add them to the BartTokenizer using the
add_tokens function and appropriately resize the token embedding space using
the resize_token_embeddings
function.333https://huggingface.co/docs/transformers/model_doc/bart

###  3.3. Pointer Generator Network

A PGN is a seq2seq network with an encoder and a decoder block. We experiment
exclusively with LSTM-based (Hochreiter and Schmidhuber, 1997) PGNs. While
PGNs can generate output tokens from a given vocabulary via softmax selection,
they are also able to copy tokens from the input text to the output by making
use of attentions weights of the decoder. A significant difference between T5,
BART and PGN is that for PGNs no pre-training on corpus is performed. The
model consists of 53 million trainable parameters which is comparable to
T5-small’s 60 million parameters size.

PGNs have typically been employed for abstract summarisation tasks but
recently Rongali et al. (2020) used PGNs to perform SQL semantic parsing.
Figure 1 depicts this architecture parsing an example query.

| F1 | F1
---|---|---
| DBpedia 16.04 | Wikidata
| LC-QuAD 1.0 | LC-QuAD 2.0
Sina (Shekarpour et al., 2015) | 0.24 | -
NLIWOD | 0.48 | -
SQG (Zafar et al., 2018) | 0.75 | -
CompQA (Luo et al., 2018) | 0.77 | -
SubQG (Ding et al., 2019) | 0.85 | -
AQG-net (Chen et al., 2020) | - | 0.45
Multi-hop QGG (Lan and Jiang, 2020) | - | 0.53
CLC (Zou et al., 2021) | - | 0.59
PGN-BERT | 0.67 | 0.77
PGN-BERT-BERT | 0.88 | 0.86
BART | 0.84 | 0.64
T5-Small | 0.90 | 0.92
T5-Base | 0.91 | 0.91
Table 1. Results for query generation with gold entities and relations. Best
results are in bold. Figure 2. Input vector for PGN-BERT.

####  3.3.1. Input Tokenisation

We construct our input vector (Figure 2) by concatenating the question tokens
with gold entities and relations. We add a separator token [SEP] between
question tokens and entity candidates, as well as between entity candidates
and relation candidates to help the model understand the segments of the input
tokens. Each token is represented by a 968-dimensional vector, where the first
768 dimensions are the BERT contextual embeddings of the question token, or
the entity label or relation label, as the case may be. The next 200
dimensions are reserved for the KG embeddings. For question tokens, KG
embeddings do not apply, so we fill the 200 dimensions with a sequence of 1.0.
The [SEP] token is represented by a vector full of -1.0. The linked entities
and relations carry the respective KG embeddings in the last 200 dimensions.

####  3.3.2. KG Embeddings

For both datasets, we make us of TransE embeddings, due to its popularity and
ease of availability (Bordes et al., 2013). For DBpedia, We train TransE
embeddings ourselves using PyTorch-BigGraph (Lerer et al., 2019). For the
training, we proceed to 30 epochs and use the cosine dot operator as the
comparator function. For Wikidata we made use of readily available TransE
embeddings made available by Pytorch-
BigGraph444https://github.com/facebookresearch/PyTorch-BigGraph on their
webpage.

####  3.3.3. Re-Ranker

We found that the beams produced by the PGN often contained the right query
but not in the top position. To improve the ranking of queries further, we
fine-tuned a pre-trained BERT-based classifier (distil-bert-base-uncased) on
the output of the PGN. We take the top-10 beam outputs of the PGN on the val
sets and query the KG. We save the output of all the queries which produce a
valid response from the KG. To train the BERT-based classifier, we form the
input string by concatenating the question tokens, the SPARQL query, and the
KG response. For supervision, we provide binary labels 0 and 1 for right and
wrong query. Once trained, we use the logit values of the penultimate layer of
the model for re-ranking the queries produced by PGN on the test set.

##  4\. Datasets

  * •

LC-QuAD 1.0 contains 5,000 questions that are answerable on DBpedia 2016-04.
The dataset originally contains a 4:1 train-test split. However, several
competing systems in Table 1 instead use a smaller split of 3,253 questions.
We adopt the same split and follow Ding et al. (2019) in performing 5-fold
cross-validation with 70:10:20 split for train, dev, and test, respectively.
There are no questions that require copying of input tokens to output query in
this dataset.

  * •

LC-QuAD 2.0 is based on Wikidata and consists of a mixture of simple and
complex questions that were verbalized by human workers on Amazon Mechanical
Turk. It is a large and varied dataset comprising 24,180 train questions and
6,046 test questions. In addition, to the best of our knowledge, this is the
sole dataset that incorporates aspects of hyper-relational (Galkin et al.,
2020) structure of recent Wikidata versions. Approximately 16% of questions in
the dataset require copying of input tokens to the output query.

##  5\. Evaluation

In Table 1, the results of competing systems for LC-QuAD 1.0 are shown as
reported by Ding et al. (2019). The results for competing systems for LC-QuAD
2.0 are as reported by Zou et al. (2021), also found on the KGQA leaderboard
(Perevalov et al., 2022)555https://kgqa.github.io/leaderboard/. For evaluating
T5, BART and PGNs on both datasets, we take the top-10 beams outputs of the
models and query the KG in ranked serial order. The first query to return a
non-empty response is considered the output of the model. We match the KG
responses of the generated query to the gold query and mark it a match if they
are identical, and no match if they are not.

In the case of LC-QuAD 2.0, we found a large number of queries no longer
answerable on the current version of Wikidata. The original dump used to
create the dataset is no longer available
online666https://databus.dbpedia.org/dbpedia/wikidata/debug/2020.07.01. As a
result, we setup an endpoint with the dump dated 13 October
2021777https://dumps.wikimedia.org/wikidatawiki/entities/ of Wikidata and
filtered the test set of 6046 questions down to 4211 questions for which the
gold query produced a valid response. We loaded the triples to a Virtuoso Open
Source Triple Store and used it as our endpoint. Zou et al. (2021) resort to a
similar pre-filtration step for the dataset in their work.

In Table 1, PGN-BERT refers to the setting where we feed the PGN with BERT
embeddings. We apply no re-ranker to the output beams, and consider the first
query in the ranked beams to produce a valid response from the KG as the
output of the system. On the other hand, PGN-BERT-BERT refer to the setting
where the beams are further re-ranked with the BERT-based re-ranker described
previously.

##  6\. Discussion

T5 not only outperforms all previous works on both datasets, but also BART and
PGN that we experiment with. It is important to handle input tokenisation of
prefixes through sentinel IDs for T5, or else the model produces zero
accuracy. In the case of PGN, we also performed a test without input KG
embeddings, where we saw a 20% point drop in accuracy. In spite of the
addition of KG embeddings, it fails to get an edge over T5, which operates
solely based on entity and relation labels. It can hence be concluded that the
large amount of pre-training for T5 outranks the KG embedding advantage.

In the case of BART versus T5, we observed that the copy actions of BART were
not as clean as T5. For example, in LC-QuAD 2.0, BART produced an F1 of 0.3
for questions that require token copy to the output, while T5 produced an F1
of 0.8. We believe this is due to the fact that instead of using sentinel IDs,
which are inherently a part of the T5 model from its pre-training objective,
we had to resort to adding special tokens to the BartTokenizer, which does not
produce the same copying performance.

T5-small and T5-base produce similar performance, which suggests that the
extra number of parameters in the base model remains unused for our given
task. For LC-QuAD 2.0, none of the previous works has the ability to copy
tokens from input to the output, so our approaches outperform them by far. In
terms of PGN performance, on using the BERT re-ranker, we see gains in the
range of 10-20 % points, which shows that PGN does a poor job of producing the
right query towards at the top. When comparing T5-small performance with PGN,
although they have similar parameter count, the pre-training on 750 GB of
corpus seems to be the crucial factor in T5-small’s favour.

##  7\. Error Analysis

We randomly sampled 100 cases of erroneous outputs for PGN, T5-Small and BART,
as shown in Table 2. The most important difference in the errors produced by
T5/BART versus PGN is what we call ”copy morphing”. T5/BART produce their
output from an open ended sub-word-based vocabulary, and hence sometimes
corrupt the item being copied. At times, they produce wrong entity IDs, e.g.,
in the case of DBpedia, Barack-Obama instead of Barack_Obama, and in some
other cases, they hallucinate entity IDs that are not part of the linked
entities or relations in input. We also encountered cases of morphing relation
IDs to semantically similar words, like notableWork changing to
notabilityWork, or unexpected capitalisation, e.g., Artist instead of artist.
Although PGN’s overall accuracy is lower, they do not exhibit this issue,
since the copying takes place from an expanded vocabulary which is limited,
consisting of SPARQL tokens and the input tokens. We also found that copy
morphing in T5 takes place only for DBpedia where the entity and relation IDs
are similar to the dictionary labels. For Wikidata, where the IDs are
numerical, morphing does not occur.

We categorise the other kind of errors under the following heads: Triple Flip,
e.g., <o p s> instead of <s p o>, Wrong Variables, e.g., ?var0 instead of
?var1, Wrong Intent, e.g., ASK instead of SELECT, Copy Errors, where the wrong
token is copied, and Syntax Errors, where the generated query is malformed and
an invalid SPARQL query. We observe that BART has poor performance in copying,
while other models have most errors in triple flips and wrong variables.

| PGN | BART | T5-Small
---|---|---|---
| LCQ1 | LCQ2 | LCQ1 | LCQ2 | LCQ1 | LCQ2
Triple Flip | 56 | 54 | 22 | 30 | 66 | 60
Wrong Var | 78 | 63 | 18 | - | 36 | 8
Wrong Intent | 22 | 15 | - | 10 | 4 | 26
Copy Error | - | 14 | 22 | 40 | - | 18
Copy Morph | - | - | 60 | - | 26 | -
Syntax Errors | - | - | 16 | 40 | - | -
Table 2. Error breakdown for randomly sampled 100 errors

##  8\. Conclusion and Future Work

In this work, we establish new baselines using PLMs for the KG semantic
parsing task for two datasets. Our results surpass the results of earlier
baselines and lay grounds for future research. We make use of readily
available and popular models, with minimal changes to input vectorisation and
tokenisation, to report state of the art results on SPARQL semantic parsing
and query building tasks over DBpedia and Wikidata. We show that these methods
are flexible and able to handle a variety of questions with no fixation on
templates or query graphs. The direction of future semantic parsing work for
SPARQL should lay more emphasis on pre-training, since currently it is more
focused on producing custom model architectures for the tasks.

As future work, we would like to explore the ability of these models in
disambiguation tasks, where the input consists of entity and relation
candidates, instead of linked entities and relations.

##  9\. Acknowledgments

This research was partially funded by the German Federal Ministry of Education
and Research (BMBF) as part of the INSTANT project, ID 02L18A111.

## References

  * (1)
  * Abujabal et al. (2017) Abdalghani Abujabal, Mohamed Yahya, Mirek Riedewald, and Gerhard Weikum. 2017.  Automated Template Generation for Question Answering over Knowledge Graphs. International World Wide Web Conferences Steering Committee.
  * Athreya et al. (2021) Ram G. Athreya, Srividya Kona Bansal, Axel-Cyrille Ngonga Ngomo, and Ricardo Usbeck. 2021.  Template-based Question Answering using Recursive Neural Networks. In _15th IEEE International Conference on Semantic Computing, ICSC 2021, Laguna Hills, CA, USA, January 27-29, 2021_. IEEE.
  * Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008.  Freebase: A Collaboratively Created Graph Database for Structuring Human Knowledge. In _Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data_. Association for Computing Machinery.  https://doi.org/10.1145/1376616.1376746
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durán, Jason Weston, and Oksana Yakhnenko. 2013\.  Translating Embeddings for Modeling Multi-Relational Data _(NIPS)_. Curran Associates Inc.
  * Chang and Manning (2012) Angel X. Chang and Christopher Manning. 2012.  SUTime: A library for recognizing and normalizing time expressions. In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_. European Language Resources Association (ELRA).
  * Chen et al. (2020) Yongrui Chen, Huiying Li, Yuncheng Hua, and Guilin Qi. 2020\.  Formal Query Building with Query Structure Prediction for Complex Question Answering over Knowledge Base. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_. International Joint Conferences on Artificial Intelligence Organization.
  * Diefenbach et al. (2018) Dennis Diefenbach, Vanessa Lopez, Kamal Singh, and Pierre Maret. 2018\.  Core Techniques of Question Answering Systems over Knowledge Bases: A Survey.  _Knowledge and Information Systems_ (2018).
  * Ding et al. (2019) Jiwei Ding, Wei Hu, Qixin Xu, and Yuzhong Qu. 2019\.  Leveraging Frequent Query Substructures to Generate Formal Queries for Complex Question Answering. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. ACL.
  * Dubey et al. (2019) Mohnish Dubey, Debayan Banerjee, Abdelrahman Abdelkawi, and Jens Lehmann. 2019.  LC-QuAD 2.0: A Large Dataset for Complex Question Answering over Wikidata and DBpedia. In _The Semantic Web – ISWC 2019_. Springer International Publishing.
  * Dubey et al. (2018) Mohnish Dubey, Debayan Banerjee, Debanjan Chaudhuri, and Jens Lehmann. 2018.  EARL: Joint Entity and Relation Linking for Question Answering over Knowledge Graphs. In _Proceedings of the 17th International Semantic Web Conference (ISWC)_.
  * Furrer et al. (2020) Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Schärli. 2020.  Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures.  https://arxiv.org/abs/2007.08970
  * Galkin et al. (2020) Mikhail Galkin, Priyansh Trivedi, Gaurav Maheshwari, Ricardo Usbeck, and Jens Lehmann. 2020\.  Message Passing for Hyper-Relational Knowledge Graphs. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. ACL.
  * Gokaslan and Cohen (2019) Aaron Gokaslan and Vanya Cohen. 2019.  OpenWebText Corpus.  http://Skylion007.github.io/OpenWebTextCorpus
  * Herzig et al. (2021) Jonathan Herzig, Peter Shaw, Ming-Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. 2021\.  Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations.  _arXiv preprint arXiv:2104.07478_ (2021).
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Computation_ (1997).
  * Hu et al. (2018) Sen Hu, Lei Zou, and Xinbo Zhang. 2018.  A State-transition Framework to Answer Complex Questions over Knowledge Base. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. ACL.
  * Jia et al. (2021) Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021.  _Complex Temporal Question Answering on Knowledge Graphs_.  Association for Computing Machinery.
  * Keysers et al. (2020) Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020\.  Measuring Compositional Generalization: A Comprehensive Method on Realistic Data. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=SygcCnNKwr
  * Lan and Jiang (2020) Yunshi Lan and Jing Jiang. 2020.  Query Graph Generation for Answering Multi-hop Complex Questions from Knowledge Bases. In _Proceedings of the 58th Annual Meeting of the ACL_. ACL, Online.
  * Lehmann et al. (2015) Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015\.  DBpedia – A Large-Scale, Multilingual Knowledge Base Extracted from Wikipedia.  _Semantic Web_ (2015).
  * Lerer et al. (2019) Adam Lerer, Ledell Wu, Jiajun Shen, Timothee Lacroix, Luca Wehrstedt, Abhijit Bose, and Alex Peysakhovich. 2019.  PyTorch-BigGraph: A Large-scale Graph Embedding System. In _Proceedings of the 2nd SysML Conference_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In _Proceedings of the 58th Annual Meeting of the ACL_. ACL.
  * Luo et al. (2018) Kangqi Luo, Fengli Lin, Xusheng Luo, and Kenny Zhu. 2018\.  Knowledge Base Question Answering via Encoding of Complex Query Graphs. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. ACL.
  * Mackenzie et al. (2020) Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R. Trippas, J. Shane Culpepper, and Alistair Moffat. 2020.  CC-News-En: A Large English News Corpus. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_ _(CIKM ’20)_. ACM.
  * Perevalov et al. (2022) Aleksandr Perevalov, Xi Yan, Liubov Kovriguina, Longquan Jiang, Andreas Both, and Ricardo Usbeck. 2022\.  Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis.  _CoRR_ abs/2201.08174 (2022).  arXiv:2201.08174 https://arxiv.org/abs/2201.08174
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ (2020).
  * Rongali et al. (2020) Subendhu Rongali, Luca Soldaini, Emilio Monti, and Wael Hamza. 2020.  Don’t Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing.  _Proceedings of The Web Conference 2020_ (2020).
  * Sakor et al. (2019) Ahmad Sakor, Kuldeep Singh, and Maria-Esther Vidal. 2019\.  FALCON: An Entity and Relation Linking Framework over DBpedia. In _Proceedings of the ISWC 2019 Satellite Tracks (Posters & Demonstrations, Industry, and Outrageous Ideas) co-located with 18th International Semantic Web Conference (ISWC 2019), Auckland, New Zealand, October 26-30, 2019_.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017\.  Get To The Point: Summarization with Pointer-Generator Networks. In _Proceedings of the 55th Annual Meeting of the ACL_. ACL.
  * Shaw et al. (2021) Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021.  Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics.
  * Shekarpour et al. (2015) Saeedeh Shekarpour, Edgard Marx, Axel-Cyrille Ngonga Ngomo, and Sören Auer. 2015.  SINA: Semantic interpretation of user queries for question answering on interlinked data.  _Journal of Web Semantics_ (2015).
  * Singh et al. (2016) Kuldeep Singh, Andreas Both, Dennis Diefenbach, Saeedeh Shekarpour, Didier Cherix, and Christoph Lange. 2016.  Qanary - The Fast Track to Creating a Question Answering System with Linked Data Technology. In _The Semantic Web - ESWC 2016 Satellite Events, Heraklion, Crete, Greece, May 29 - June 2, 2016, Revised Selected Papers_ (2016-01-01).
  * Singh et al. (2018a) Kuldeep Singh, Ioanna Lytra, Arun Sethupat Radhakrishna, Akhilesh Vyas, and Maria-Esther Vidal. 2018a.  Dynamic Composition of Question Answering Pipelines with FRANKENSTEIN. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_.
  * Singh et al. (2018b) Kuldeep Singh, Arun Sethupat Radhakrishna, Andreas Both, Saeedeh Shekarpour, Ioanna Lytra, Ricardo Usbeck, Akhilesh Vyas, Akmal Khikmatullaev, Dharmen Punjani, Christoph Lange, Maria Esther Vidal, Jens Lehmann, and Sören Auer. 2018b.  Why Reinvent the Wheel: Let’s Build Question Answering Systems Together. In _Proceedings of the 2018 World Wide Web Conference_. International World Wide Web Conferences Steering Committee.
  * Soru et al. (2017) Tommaso Soru, Edgard Marx, Diego Moussallem, Gustavo Publio, André Valdestilhas, Diego Esteves, and Ciro Baron Neto. 2017.  SPARQL as a Foreign Language. In _Proceedings of the 13th International Conference on Semantic Systems - SEMANTiCS2017 Posters and Demos_.
  * Strötgen and Gertz (2010) Jannik Strötgen and Michael Gertz. 2010.  HeidelTime: High Quality Rule-Based Extraction and Normalization of Temporal Expressions. In _Proceedings of the 5th International Workshop on Semantic Evaluation_. Association for Computational Linguistics.
  * Sun et al. (2020) Yawei Sun, Lingling Zhang, Gong Cheng, and Yuzhong Qu. 2020\.  SPARQA: Skeleton-Based Semantic Parsing for Complex Questions over Knowledge Bases.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 34 (2020).
  * Trinh and Le (2018) Trieu H. Trinh and Quoc V. Le. 2018.  A Simple Method for Commonsense Reasoning.  _CoRR_ abs/1806.02847 (2018).  arXiv:1806.02847 http://arxiv.org/abs/1806.02847
  * Trivedi et al. (2017) Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. 2017.  LC-QuAD: A Corpus for Complex Question Answering over Knowledge Graphs. In _The Semantic Web – ISWC 2017_. Springer International Publishing.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _Advances in Neural Information Processing Systems_. Curran Associates, Inc.  https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * Vollmers et al. (2021) Daniel Vollmers, Rricha Jalota, Diego Moussallem, Hardik Topiwala, Axel-Cyrille Ngonga Ngomo, and Ricardo Usbeck. 2021.  Knowledge Graph Question Answering Using Graph-Pattern Isomorphism.  (2021).
  * Vrandečić, Denny and Krötzsch, Markus (2014) Vrandečić, Denny and Krötzsch, Markus. 2014.  Wikidata: A Free Collaborative Knowledge Base.  _Commun. ACM_ (2014).
  * Yih et al. (2015) Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015\.  Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base. In _Proceedings of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. ACL.
  * Zafar et al. (2018) Hamid Zafar, Giulio Napolitano, and Jens Lehmann. 2018\.  Formal Query Generation for Question Answering over Knowledge Bases. In _The Semantic Web_. Springer International Publishing.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In _The IEEE International Conference on Computer Vision (ICCV)_.
  * Zou et al. (2021) Jianyun Zou, Min Yang, Lichao Zhang, Yechen Xu, Qifan Pan, Fengqing Jiang, Ran Qin, Shushu Wang, Yifan He, Songfang Huang, and Zhou Zhao. 2021.  A Chinese Multi-type Complex Questions Answering Dataset over Wikidata.  arXiv:2111.06086 [cs.CL]