# SPARQL Generation with Entity Pre-trained GPT for KG Question Answering

Diego Bustamante1,2, Hideaki Takeda3

Authors are listed in alphabetical order.
1 Department of Computer Science, PUC, Santiago, Chile
2 Millenium Institute for Foundational Research on Data, Santiago, Chile
3 National Institute of Informatics (NII), Tokyo, Japan

###### Abstract

Knowledge Graphs popularity has been rapidly growing in last years. All that
knowledge is available for people to query it through the many online
databases on the internet. Though, it would be a great achievement if non-
programmer users could access whatever information they want to know. There
has been a lot of effort oriented to solve this task using natural language
processing tools and creativity encouragement by way of many challenges. Our
approach focuses on assuming a correct entity linking on the natural language
questions and training a GPT model to create SPARQL queries from them. We
managed to isolate which property of the task can be the most difficult to
solve at few or zero-shot and we proposed pre-training on all entities (under
CWA) to improve the performance. We obtained a 62.703% accuracy of exact
SPARQL matches on testing at 3-shots, a F1 of 0.809 on the entity linking
challenge and a F1 of 0.009 on the question answering challenge.

##  1 Introduction

Knowledge Graphs (KG) [10] are structures often used to represent knowledge.
Nowadays, more than ever, the amount of open source information stored in KG
throughout the web is beyond comprehension for us humans, e.g., just Wikidata
has more than 108 million items111Wikidata:Statistics -
https://www.wikidata.org/wiki/Wikidata:Statistics.

A clever way to store Knowledge Graphs are graph databases [10]. This approach
solves a lot of important problems regarding information management, some of
them are: storage, structure, query and update. Although access points to this
data are readily available, for example via web browser, information
extraction is not a simple task for a user that does not know how to make
graph databases queries given a particular language (e.g. SPARQL222SPARQL 1.1
Query Language - https://www.w3.org/TR/sparql11-query/).

With the recent rise of generative LLMs it is natural for a user to think that
ChatGPT333OpenAI ChatGPT - https://chat.openai.com/, Copilot444Microsoft
Copilot - https://copilot.microsoft.com/, Bard555Google Bard -
https://bard.google.com/ or any other model, could help them in the task of
creating a SPARQL query in order to answer a question about a particular KG.
Taking in consideration that generating queries it is not the main interest of
the mentioned LLMs, the problem is that a simple qualitative analysis using
this AI tools can expose that only a few of them do well on this task.

The goal of this work is to identify the main property that makes this problem
a difficult one, and propose a training methodology and model architecture
such to improve the accuracy of this type of models on this particular task.

##  2 Related Work

There has been a lot of effort from researchers on improving results on the
task of Knowledge Graph Question Answering. Given a KG, the problem of KGQA
consists of inputs such as “What are the papers written by the person X?” and
outputs such as the triples of the KG that correspond to the answer to the
question. With the goal of motivating competition and innovation, research
groups have published challenges and datasets with varied difficulties. Some
of the most recent data collections on the field of scholarly knowledge are
DBLP-QuAD [3] and SciQA [2].

The Scholarly QALD challenge recently took place at the ISWC 2023666Scholarly
QALD at ISWC 2023 - https://kgqa.github.io/scholarly-QALD-challenge/2023/.
Many groups had great results and, as expected, most of them used LLMs in some
aspect of their approaches [17, 14, 15, 11].

Since Natural Language Processing tools have demonstrated being very useful
solutions for text-to-text tasks [8, 16], it is natural to think where can
those tools be integrated to tackle the inputs of KGQA. The solutions seems to
be to considerate SPARQL queries as plain text, as a natural language with its
own vocabulary and grammar. Then, technologies like GPT [16] are a clear study
path to solve the Scholarly QALD challenge.

We manage to replicate the results obtained by Rajpal & Usbeck [14]. We
studied how to improve the lack of multi-hops after entity linking in their
approach. Inspired by the LLM popularity, we manage to isolate the most
difficult characteristic of this task, create a training process to solve it
and propose future improvements to our approach.

##  3 Approach

Our approach uses the DBLP-QuAD dataset [3]. This data is composed of 10,000
items, every data point has a question, paraphrased question, SPARQL query,
entities, relations and answer (triples). This structure enable us to approach
the challenge in many different ways, stimulating innovation.

Instead of directly trying to obtain the triples to answer the question, we
first focused on dividing the task and translating the natural language
question to a SPARQL query.

###  3.1 Entity linking

Our first models did not do very well. Our hypothesis was that translating a
sub-string entity into a link while trying to learn the SPARQL grammar was too
much work for our model size (3.47 M parameters) and for the amount of
training data we had (10,000 items).

To solve that problem, we subdivided the task into entity linking and SPARQL
generation. We assumed the entity linking task can be done with perfect
accuracy and then we would perform the translation. We build a new dataset
where the entities are replaced by their iri, e.g., if the question was “What
are the papers written by the person Wei Li?”, the new question will be “What
are the papers written by the person <https://dblp.org/pid/64/6025-131>?”.
This reduced the amount of different tokens (vocabulary) from more than 46,000
to only 10,399; a significant amount that can affect how the model will
perform. We used the entity linking methods of Rajpal & Usbeck [14, 4] with
less overhead (and worse accuracy).

We ruled out the TP61 template [3] data points due to error detected in the
SPARQL queries, entities were not well used on the template. Since the final
challenge can certainly have TP61-type questions, this decision is a source of
error. But, our goal is to find new training ways to improve performance and
comparatively will negatively affect all our models the same.

From 20,000 items (questions and paraphrased questions), we ended up with
9,289 successful entity linked data points on the new dataset. We minimize
type I error, i.e., we did many checks to ensure perfect entity linking. In
production or real testing we would bet on doing it right to improve accuracy,
but in this phase we are building a new training dataset. Is evident that our
EL performance is not state-of-the-art, but it was not our goal.

The difference between the original and the entity linked dataset can are more
clear in Table 1. We also performed normalization on the questions described
in section 4.

Dataset |  Input example |  Output example
---|---|---
Original |  Show the Wikidata ID of the person Robert Schober. |  SELECT DISTINCT ?answer WHERE { <https://dblp.org/pid/95/2265> <https://dblp.org/rdf/schema#wikidata> ?answer }
Entity linked |  show the Wikidata ID of the person <https://dblp.org/pid/95/2265> |  SELECT DISTINCT ?answer WHERE { <https://dblp.org/pid/95/2265> <https://dblp.org/rdf/schema#wikidata> ?answer }
Pre-train |  <https://dblp.org/pid/95/2265> <https://dblp.org/pid/95/2265> … |  <https://dblp.org/pid/95/2265> <https://dblp.org/pid/95/2265> …
Table 1: Input and output examples for the GPT model by dataset. The original
dataset (20,000 items), the entity linked dataset (9,289 items) and the pre-
train dataset (7,617 items).

###  3.2 Query generation

With the entity link phase done, a translation model has to learn the
different types of query templates and complete the variable information with
data from the input. The architecture of our model is based on the GPT
implementation by Andrej Karpathy777Let’s build GPT: from scratch, in code,
spelled out - https://youtu.be/kCc8FmEb1nY which is, in turn, inspired on the
paper “Attention Is All You Need” [16]. Karpathy implemented a decoder only
transformer so we completed the encoder-decoder model to take the questions as
inputs on the encoder and the decoder would generate the queries. The complete
architecture is shown in 1(a).

As we said, the model has to learn $i)$ query templates and $ii)$ learn how to
change the template with the corresponding information. Simplifying the second
task, it can be described as an identity function $f(string)=string$ with the
entity iris found on the input. The problem with our first solution was that
while learning the grammatical aspects of the problem it stopped from learning
the rest. Our GPT preferred a lazy minimization of the amount of wrong tokens
than learning how to perform the entity replacement. We attribute this
behavior to the fact that there are only a few entities per question/query so
its contribution to the loss function is very small. The parameter optimizer
fitted a shortcut learning [9] and could not improve more.

It is well studied that pre-training can improve the performance of generative
transformer models [1, 13, 8, 12], specially when post fine-tuning for
question answer is implemented [12]. Therefore, based on literature [1, 13],
our hypothesis is that performance should improve when we first train the
model to perform the identity function on entities and then transfer that
learning over the KGQA task.

The amount of different queries is exponentially large, hence the need for an
AI model to do the translation. In contrast, under the Closed World Assumption
[10] the amount of entities is fixed. Then, we pre-train the model with all
entities to perform the identity task. In the first instance we tried to teach
the model the function $f(entity)=entity$, but since entities can be anywhere
in the input and output the improvement was not meaningful. When we pre-train
the function $f(entity\ entity\ entity\ ...)=entity\ entity\ entity\ ...$, we
saw an important performance enhancement.

Our GPT implementation can be found at GitHub888Code -
https://github.com/DiegoEmilio01/SPARQL-generation-with-pre-trained-GPT-for-
KG-Question-Answering/. When combined with an entity linker and the desired KG
database to query, the complete process looks like 1(b).

(a) Diagram of our GPT architecture, very similar to the original model.

(b) Complete process when every phase is integrated, we focused on GPT.

Figure 1: Our solution, the components of our model and how to integrate it in
production setting.

##  4 Experiments

In addition to the entity linking, we also modify the questions to achieve a
better generalization in the model. We eliminate punctuation symbols and first
word capitalization, and spaced parenthesis because our tokenization is by
words space-split. For example, “<https://dblp.org/pid/64/6025-131>?” should
be the same entity as “<https://dblp.org/pid/64/6025-131>”.

The distribution of the 9,289 data points is 8,648 for training (93.1%), 456
for validation (5.9%) and 185 for testing (2%). This proportions were found by
trial and error, it is important to notice that we did not follow the original
distribution due to the creation of the new entity linked dataset. In the case
of the pre-training, all 7,617 entities were used as train, validation and
testing. The original dataset had 7,143 entities but we added new iris from
the final 500 questions of the Scholarly QALD challenge to the vocabulary
justified by our Closed World Assumption [10]. The average number of entities
per query is 1.231 entities.

Example of items in the all datasets are tabulated in Table 1. In the case of
the pre-train dataset, the length of the inputs are 34 tokens and outputs are
49 tokens, this is, the maximum input and output size of our model.

Our vocabulary was originally of size 9,339; but with the inclusion of the
final 500 questions it ended being of size 10,399 (including new entities and
other words). The model hyperparameters were also found by trial and error.
Dropout of 1%, learning rate of 0.0007, internal vector of dimension 128,
number of heads of 8, and 4 layers for the decoder and encoder each. We
instanced 2 models, one with pre-training and one without to compare
improvements. Both were of size 3.47 million parameters, were trained the same
total amount of epochs (19,200) and with the exact same data points
distribution between training, validation and testing, in order to be able to
compare them. In the case of the pre-trained model, it was trained 14,400
epochs on entities and 4,800 on the new dataset.

As training metric we used cross entropy for the loss function and as testing
metric we used accuracy at 1 shot, accuracy at 3 shots and average Hamming
distance. Accuracy in our setting means how many exact SPAQRL queries could we
generate over the total, and Hamming distance is how many tokens away is our
prediction from the gold answer (only one shot). For the final 500 questions
of the challenge we retrieved final triples as answers and submitted them to
the official challenge999[DBLP-QuAD] Scholarly QALD @ ISWC 2023 -
https://codalab.lisn.upsaclay.fr/competitions/14264 to obtain precision,
recall and F1 (under the user DiegoEmilio).

##  5 Results and Discussion

Model $\backslash$ Metric | Acc@1(%) | Acc@3(%) | aHD(# tokens) | Precision(%) | Recall(%) | F1
---|---|---|---|---|---|---
Not pre-trained | 31.892 | 43.784 | 1.724 | 0.502 | 0.605 | 0.005
Pre-trained | 49.189 | 62.703 | 2.011 | 0.725 | 1.291 | 0.009
Table 2: Results for the 2 types of models. The metrics are query accuracy at
1 shot, query accuracy at 3 shots, query average Hamming distance, challenge
precision, challenge recall and challenge F1.

Our results are shown in Table 2. Pre-training the model improves almost every
metric. A 17.3% improvement on accuracy@1 is remarkable, tough precision,
recall and F1 are not much significant because getting few questions with a
lot of triples right can produce a random increase in F1 like ours. We
observed that the model without pre-training tried to minimize how many tokens
gets wrong, i.d., improve the average Hamming distance. In contrast, with pre-
training it learned how to maximize the number of exact match answers despite
getting more tokens wrong when generating an incorrect answers (hence the
higher aHD).

We have overcome the initial shortcut learning [9]. This suggests that changes
in the loss function could help the training. For example, reward higher exact
matches because in our experiments the reward difference between getting 1 to
2 tokens incorrect is the same as between 0 to 1.

Due to the type of model and training, we expected our model to extrapolate
the rules to manage queries it has never seen before [6]. Considering our
accuracy in testing, we think that it managed to learn the different
templates. But, the identity function is still a problem when the author has
never been seen in training.

The architecture of our model allows it to be very good at few-shot [5], and
we attribute to this property the success of our results with such little data
points (9,289). We tried to generate zero-shots queries with entities we knew
were not in the dataset. Even when all iris were included in the pre-training,
the model was not capable of generate the exact match, just the template.
Moreover, when including the final 500 questions to the vocabulary we added
474 new entities to the problem, so this hints that the reason behind our bad
results on the challenge is zero-shots.

This analysis is also backed up by our metrics. The 4th column on Table 2
shows that, on average, the model gets tokens wrong approximately the same
amount of tokens as the number of entities per query (1.231). Thus,
considering that entities are the context in which the question should be
answered, we identify that Comprehension-topic hallucinations [7] of the
identity function at zero-shots is the main difficulty of this task.

Our approach has the potential to be refined and combined to improve results,
for example is heavily dependent on the performance of the entity linking
phase before the GPT phase. Our entity linking process achieved 92.931%
precision, 71.635% recall and 0.80905 F1. Best entity linkers could be used to
improve performance but since we are competitive with state-of-art this should
not bring a high increase in performance.

We propose a better training to tackle zero-shots inaccuracies. Since our
framework is under the Closed World Assumption [10], we can add few queries
per entity on top of our pre-training. This enrichment of the dataset should
be enough to improve performance, specially on the final 500 questions of the
Scholarly QALD challenge.

Furthermore, we think that the value of our results lies in at least returning
the correct template for difficult questions and queries. In a qualitative
analysis, most of the popular LLMs can’t help users to formulate queries as
hard as the ones in the DBLP-QuAD dataset [3]. In that sense, being
competitive with most popular LLMs using a very small model and training data
in comparison shows an optimistic path to solve this task.

Finally, a light model like ours can be trained with very few data, time and
costs. When a new open source Knowledge Graph appears on the internet is not
known by LLMs. Then, the publishers of the KG can find this method useful to
easily help their non-programmer users by putting in production a GPT model
that suggests queries.

## Acknowledgement

The research was partially funded by the National Institute of Informatics,
Japan. D.B. acknowledges partial support from ANID, Subdirección de Capital
Humano (Magíster Nacional, 2023, folio 22231282).

Big thanks to Trinidad Gatica and Elías Sabja for their help with code
debugging.

## References

  * [1] Alyafeai, Z., AlShaibani, M. S., and Ahmad, I. A survey on transfer learning in natural language processing, 2020.
  * [2] Auer, S., Barone, D. A. C., Bartz, C., Cortes, E. G., Jaradeh, M. Y., Karras, O., Koubarakis, M., Mouromtsev, D., Pliukhin, D., Radyush, D., Shilin, I., Stocker, M., and Tsalapati, E. Sciqa benchmark: Dataset and rdf dump, Mar. 2023.
  * [3] Awale, S., Banerjee, D., Usbeck, R., and Biemann, C. Dblp-quad, Feb. 2023.
  * [4] Banerjee, D., Arefa, Usbeck, R., and Biemann, C. Dblplink: An entity linker for the dblp scholarly knowledge graph, 2023\.
  * [5] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners.  CoRR abs/2005.14165 (2020).
  * [6] Bueno, M., Gemmell, C., Dalton, J., Lotufo, R., and Nogueira, R. Induced natural language rationales and interleaved markup tokens enable extrapolation in large language models, 2022.
  * [7] Correa Busquets, S., and Maccarini Llorens, L. Autosupervisión de alucinaciones en grandes modelos del lenguaje: LLteaM.  Journal of Computer-Assisted Linguistic Research 7 (12 2023), 60–85.
  * [8] Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding.  CoRR abs/1810.04805 (2018).
  * [9] Geirhos, R., Jacobsen, J., Michaelis, C., Zemel, R. S., Brendel, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks.  CoRR abs/2004.07780 (2020).
  * [10] Hogan, A., Blomqvist, E., Cochez, M., d’Amato, C., de Melo, G., Gutiérrez, C., Kirrane, S., Labra Gayo, J. E., Navigli, R., Neumaier, S., Ngonga Ngomo, A.-C., Polleres, A., Rashid, S. M., Rula, A., Schmelzeisen, L., Sequeda, J. F., Staab, S., and Zimmermann, A. Knowledge Graphs.  No. 22 in Synthesis Lectures on Data, Semantics, and Knowledge. Springer, 2021.
  * [11] Pliukhin, D., Radyush, D., Kovriguina, L., and Mouromtsev, D. Improving subgraph extraction algorithms for one-shot sparql query generation with large language models.  In Scholarly-QALD-23: Scholarly QALD Challenge at The 22nd International Semantic Web Conference (ISWC 2023) (Athens, Greece, November 2023), vol. 3592(6), CEUR Workshop Proceedings, pp. 1–10.
  * [12] Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training, 2018.
  * [13] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023.
  * [14] Rajpal, S., and Usbeck, R. BERTologyNavigator: Advanced question answering with bert-based semantics.  In Scholarly-QALD-23: Scholarly QALD Challenge at The 22nd International Semantic Web Conference (ISWC 2023) (Athens, Greece, November 2023), vol. 3592(7), CEUR Workshop Proceedings, pp. 1–8.
  * [15] Taffa, T. A., and Usbeck, R. Leveraging llms in scholarly knowledge graph question answering.  In Scholarly-QALD-23: Scholarly QALD Challenge at The 22nd International Semantic Web Conference (ISWC 2023) (Athens, Greece, November 2023), vol. 3592(5), CEUR Workshop Proceedings, pp. 1–10.
  * [16] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need.  CoRR abs/1706.03762 (2017).
  * [17] Wang, R., Zhang, Z., Rossetto, L., Ruosch, F., and Bernstein, A. Nlqxform: A language model-based question to sparql transformer.  In Scholarly-QALD-23: Scholarly QALD Challenge at The 22nd International Semantic Web Conference (ISWC 2023) (Athens, Greece, November 2023), vol. 3592(2), CEUR Workshop Proceedings, pp. 1–10.