# QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and
Wikidata
Translated by Native Speakers

Aleksandr Perevalov  Department of Computer Science and Languages Anhalt
University of Applied Sciences Köthen (Anhalt), Germany Email:
<EMAIL_ADDRESS>Dennis Diefenbach  Laboratoire Hubert Curien
CNRS UMR 5516 Université de Lyon, France Email: dennis.diefenbach@univ-st-
etienne.fr Ricardo Usbeck  Department of Informatics University of Hamburg
Hamburg, Germany Email<EMAIL_ADDRESS>Andreas Both  Department
of Computer Science and Languages Anhalt University of Applied Sciences Köthen
(Anhalt), Germany Email<EMAIL_ADDRESS>
###### Abstract

The ability to have the same experience for different user groups (i.e.,
accessibility) is one of the most important characteristics of Web-based
systems. The same is true for Knowledge Graph Question Answering (KGQA)
systems that provide the access to Semantic Web data via natural language
interface. While following our research agenda on the multilingual aspect of
accessibility of KGQA systems, we identified several ongoing challenges. One
of them is the lack of multilingual KGQA benchmarks. In this work, we extend
one of the most popular KGQA benchmarks – QALD-9 by introducing high-quality
questions’ translations to 8 languages provided by native speakers, and
transferring the SPARQL queries of QALD-9 from DBpedia to Wikidata, s.t., the
usability and relevance of the dataset is strongly increased. Five of the
languages – Armenian, Ukrainian, Lithuanian, Bashkir and Belarusian – to our
best knowledge were never considered in KGQA research community before. The
latter two of the languages are considered as “endangered” by UNESCO. We call
the extended dataset QALD-9-plus and made it available online111Figshare:
https://doi.org/10.6084/m9.figshare.16864273. GitHub:
https://github.com/Perevalov/qald_9_plus.

##  I Introduction

The core task of a Knowledge Graph Question Answering system is to represent a
natural language question in the form of a structured query (e.g., SPARQL) to
a knowledge graph (KG). In other words, KGQA systems provide access to the
data in KGs via a natural-language user interface, s.t., end users are not
required to learn a particular query language for fetching data manually.
Obviously, the relevance (or accuracy) of the answers given by such system
should strive to human performance and reduce labor costs for learning a
particular query language; otherwise, the system is useless. Many researchers
are aiming at measuring and increasing the Question Answering (QA) quality or
the quality of a particular KGQA sub-tasks, such as named entity linking
(e.g., [1]), expected answer type prediction (e.g., [2]), etc. However, _the
accessibility 222The accessibility for the Web is defined by W3C:
https://www.w3.org/standards/webdesign/accessibility characteristic of the
KGQA systems often stays overlooked_. In this context, the perfect
accessibility denotes an equivalent experience to all user groups of a
particular KGQA system. Hence, such research questions as: “How many people
can really take advantage of the high-quality KGQA system?” and “Who are these
people?” as well as “How diverse they are?” are often left unnoticeable.

In this work, _we focus on the multilingual aspect of accessibility_. The
majority of research activities are dedicated to using the English language,
however, there are approximately 7000 languages spoken in the world. Moreover,
many languages spoken by hundreds of thousands or even millions of people
(e.g., Udmurt, Bashkir, Belarusian) are not considered in the KGQA research to
this moment. In the context of the multilingual aspect, increased
accessibility provides the opportunity to use a KGQA system effectively for
the speakers of different languages (including low-resource and endangered
ones333cf., http://www.unesco.org/languages-atlas/en/atlasmap.html). While
doing preliminary research, we identified that multilingual KGQA (mKGQA) has
several ongoing challenges. One of them is the lack of available benchmarks.
To the best of our knowledge, there are only 3 mKGQA datasets in the research
community (QALD-9 [3], RuBQ 2.0 [4], and CWQ [5]) and all of them are not
fulfilling completely the practical needs of researchers and developers (see
Section II). Hence, _even if one develops an mKGQA system, there are not so
many opportunities for full-fledged evaluation_. In this regard, it was
decided to extend the well-known QALD-9 dataset in order to enlarge its
language coverage and improve the quality of non-English questions. The
English questions from the dataset are considered as a reference. To create
the extension, a number of crowd workers were asked to translate the questions
into their mother tongue manually (and validate translations by others). As a
result, 17 volunteer participants and 290 crowd workers from Amazon Mechanical
Turk444https://www.mturk.com/ and Yandex Toloka555https://toloka.yandex.com/
translated questions into 8 languages, such as: German, Russian, French,
Armenian, Belarusian, Lithuanian, Bashkir, and Ukrainian, the latter five of
which, to our knowledge, have never been considered in the KGQA context. In
addition, the SPARQL queries in the dataset were transformed from DBpedia to
Wikidata in order to increase the usability of the dataset. We name the
extended dataset as “QALD-9-plus” and plan to continuously extend it over
time. Thus, _the main contribution of this work_ is that we introduce the
mKGQA benchmark with high-quality translations that covers 9 languages in
total (including English) some of that were never covered in the KGQA research
community before. Moreover, Belarusian and Bashkir languages are considered as
endangered by UNSECO666cf., http://www.unesco.org/languages-atlas/. We value
our contribution as a small but very important step towards improving
multilingual accessibility of KGQA systems.

This paper is structured as follows: in Section II we describe currently
available multilingual KGQA datasets, Section III describes in details the
process of creation and analysis of QALD-9-plus. We demonstrate evaluation
results of multilingual KGQA systems on QALD-9-plus in Section IV and conclude
the paper in Section V.

##  II Multilingual KGQA Benchmarks

Today, the research in the field of KGQA is strongly dependent on data, and it
suffers from the lack of multilingual benchmarks [6, 7]. To the best of our
knowledge, only three KGQA benchmarks exist that tackle multiple languages:
QALD [3], RuBQ [4], and CWQ [5].

QALD-9 contains 558 questions incorporating information of the DBpedia
knowledge base777https://www.dbpedia.org/ [8] where for each question the
following is given: a textual representation in multiple languages, the
corresponding SPARQL query (over DBpedia), the answer entity URI, and the
answer type. The dataset has become a benchmark for many research studies in
QA (e.g., [9, 1, 10, 11, 12]).

RuBQ 2.0 is a KGQA dataset over Wikidata 888https://www.wikidata.org/ [13]
that contains 2910 questions. The questions are represented in native Russian
language and machine-translated to English language. Additionally, it contains
a list of entities, relations, answer entities, SPARQL queries, answer-bearing
paragraphs, and query type tags.

CWQ is a recently published KGQA dataset over Wikidata that is based on CFQ
data [14]. CWQ contains questions in Hebrew, Kannada, Chinese, and English
languages. All the non-English questions were obtained using machine
translation with several rule-based adjustments. It has a well-detailed
structure including: question with highlighted entities, original SPARQL query
over Freebase [15] (from CFQ), SPARQL query over Wikidata (introduced in CWQ),
textual representations of a question in four aforementioned languages, and
additional fields.

Despite the considered benchmarks contain questions in multiple languages,
these multilingual representations were either machine-translated (RuBQ 2.0,
CWQ) or have doubtful quality (QALD-9, see Section III-A). In contrast, the
dataset that we present is covering 9 languages (including English) and was
created by native speakers.

##  III QALD-9-plus Benchmark Creation Process

###  III-A Qualitative Analysis of QALD-9

While working with QALD-9 the following flaws in the data were identified.

Firstly, there is poor translation quality of the majority of questions in
languages other than English. The authors, as native speakers of German and
Russian, were capable to identify them for the respective languages. The
question: “Which subsidiary of TUI Travel serves both Glasgow and Dublin?” has
the following translation to Italian: “Quale società sussidiaria di TUI Travel
serve sia Dortmund che Dublino?”. While not being a native speaker of Italian,
it is obvious that two different cities are used in the original question
(Glasgow) and in its translated version (Dortmund). It is worth noting, that
the corresponding SPARQL query uses Glasgow, as in the original English
question. Additionally, an analysis was performed for the German questions by
an English teacher from Germany, rating the quality of the translation from
English to German. It scores only 4.09 of 5.00 (5.00 is the highest rating)
where 29 question received a rating of 1, 40 a rating of 2 (87 of 3, 97 of 4,
and 305 of 5). Hence, this qualitative rating of the questions’ translation to
German shows that the translation quality is not sufficient.

Secondly, the meaning of some questions is ambiguous, for example: “How often
did Jane Fonda marry?”, here, the actual meaning is probably how many times.
Another ambiguous example of a question is: “Who is the heaviest player of the
Chicago Bulls?”, in this case, the time span of the question is unclear – the
heaviest in the history or currently playing. The corresponding SPARQL query
for the question “Give me all films produced by Steven Spielberg with a budget
of at least $80 million.” contains the property dbo:director, however, the
question asks for dbo:producer.

In our work, we focus on the first part of the dataset’s flaws – poor
translation quality. The original DBpedia queries were not changed, but were
transformed to Wikidata. In the next sections, we describe the translation
process and the process of migration of the SPARQL queries from the DBpedia KG
to the Wikidata KG.

###  III-B Translation of Questions

The translation process was executed in a crowdsourcing manner with the
following settings: (1) the crowd workers had to translate the questions from
English to their mother tongue, (2) each crowd worker was provided with a
prepared subset of questions selected from different parts of the dataset to
avoid possible biases (one crowd worker translated at least 10 questions), (3)
the usage of machine translation tools was prohibited (only dictionaries were
allowed), (4) the crowd workers were not aware of the existing multilingual
representations from the QALD-9 dataset, and (5) the crowd workers were not
aware of the other’s tasks. In total, 17 volunteers and 290 crowd workers from
Amazon Mechanical Turk and Yandex Toloka speaking 8 languages took part in the
translation process. The coverage of translations was set to $\geq$ 2.

After the translations were obtained, the validation process was executed.
During the validation, the crowd workers were provided with the original
question and two translation options. A crowd worker had to select one of the
following options: (1) no translations are correct, (2) first translation is
correct, (3) second translation is correct, or (4) both translations are
correct.

Despite the volunteers, the tasks to the crowd workers were assigned according
to the main country of language’s origin (e.g., one has to be from France to
translate from English to French). Thus, the questions were translated by
native speakers of these different languages and additionally multiple times
validated.

⬇

# DBpedia

SELECT DISTINCT ?date

WHERE {

dbr:Finland dbp:accessioneudate ?date .

}

⬇

# Wikidata

SELECT DISTINCT ?date

WHERE {

wd:Q33 p:P463 ?membership .

?membership pq:P580 ?date .

?membership ps:P463 wd:Q458 .

}

Figure 1: An example of different basic graph patterns in DBpedia and Wikidata for question: “When did Finland join the EU?”. Prefixes are omitted (we used the typical RDF prefixes as shown at https://prefix.cc/popular/all). TABLE I: The number of “gold standard” answer sets computed over DBpedia that have less or equal corresponding intersection rate between QALD-9 and QALD-9-plus, i.e., how many “gold standard” answer sets have the intersection rate less than X%? |  $\leq$ 25% |  $\leq$ 50% |  $\leq$ 75% |  $\leq$ 100%
---|---|---|---|---
Train | 183 | 227 | 269 | 408
Test | 69 | 86 | 98 | 150
TABLE II: Overview of the extended QALD-9-plus. | en | de | fr | ru | uk | lt | be | ba | hy | DBpedia | Wikidata | # questions
---|---|---|---|---|---|---|---|---|---|---|---|---
Train | 408 | 543 | 260 | 1203 | 447 | 468 | 441 | 284 | 80 | 408 | 371 | 408
Test | 150 | 176 | 26 | 348 | 176 | 186 | 155 | 117 | 20 | 150 | 136 | 150

###  III-C Migration from DBpedia to Wikidata

To extend the usability of the dataset, we ported the corresponding QALD-9
queries from DBpedia to the Wikidata KG. This process was carried out and
validated in-house by 3 experienced computer scientists. Although semi-
automatic scripts for retrieving entity and property mappings were partly used
to speed up the process, it was very labor-intensive. In particular, we found
that a simple property path in DBpedia is not necessarily simple considering
the Wikidata KG, which complicated the transformation process. For example,
the DBpedia query for the question “When did Finland join the EU?”, consists
of one triple (DBpedia) while three triples are required to fetch the same
information from the Wikidata KG (see Figure 1).

Several DBpedia SPARQL queries were not portable to Wikidata due to the
absence of the corresponding information (e.g., “How many calories does a
baguette have?”). In this regard, 51 queries could not be transferred to
Wikidata, since the information available in the DBpedia KG was not available
in the Wikidata KG. The “gold standard” answers over Wikidata for the
questions were obtained by executing the corresponding SPARQL queries. The
updated “gold standard” answers over DBpedia were obtained the same way.

We compared the answer-sets of the SPARQL queries of the old (QALD-9) and the
updated (QALD-9-plus) datasets (see Table I). Given the table, nearly half of
the “gold standard” answer sets for both train and test splits has less than
50% similarity, while comparing QALD-9 and QALD-9-plus. That is caused by
changes in the DBpedia that may occur due to several reasons. For example,
historical reasons, such as the change of president, governor etc., play a
role, but also the data model has changed over time, or missing information
was updated or added. This shows also the importance of the answer sets in the
KGQA datasets since they enable comparability between different versions of a
knowledge base. The up-to-date information should always be retrieved via the
corresponding SPARQL queries.

###  III-D Dataset Statistics

The overview of the dataset statistics is present in Table II.
We extended the dataset with 4,930 new question translations for different
languages. In this regard, the majority of the multilingual representations of
QALD-9 were replaced by the new ones. Note that for some languages like
Russian we could collect multiple translations for some questions (i.e., 2.9
Russian translations per one question in training set) while for other
languages like Armenian, Bashkir, French we provide only a partial converge of
the original QALD-9 questions. All translations were carried out by native
speakers.
In terms of the SPARQL queries, we extended the dataset with 507 new queries
over Wikidata. The queries were manually created by the authors of the paper.

TABLE III: Comparison of quantitative text features between QALD-9 and QALD-9-plus (both train and test subsets were considered) | English (not changed) | German | Russian | French
---|---|---|---|---
QALD-9 | QALD-9-plus | QALD-9 | QALD-9-plus | QALD-9 | QALD-9-plus | QALD-9 | QALD-9-plus
Average syllables per word | 1.32 | 1.86 | 1.87 | 2.13 | 2.31 | 1.51 | 1.55
Average word length | 4.77 | 5.64 | 5.68 | 5.66 | 6.12 | 5.07 | 5.18
Average sentence length | 33.92 | 35.65 | 39.14 | 19.80 | 36.66 | 36.35 | 39.44
Average words per sentence | 7.06 | 6.32 | 6.89 | 3.49 | 5.99 | 7.18 | 7.62
Type Token Ratio | 0.30 | 0.33 | 0.33 | 0.32 | 0.49 | 0.32 | 0.49
TABLE IV: Evaluation results on QALD-9-plus with GERBIL system. Only
multilingual KGQA systems were used. The first value of a cell corresponds to
Micro F1 score, the second one corresponds to Macro F1 score [16]. QAnswer
---
| English | German | Russian | French
Micro F1 | Macro F1 | Micro F1 | Macro F1 | Micro F1 | Macro F1 | Micro F1 | Macro F1
Wikidata test | 0.1002 | 0.4459 | 0.0291 | 0.3171 | 0.0159 | 0.2143 | 0.1190 | 0.2300
Wikidata train | 0.0866 | 0.4009 | 0.0587 | 0.3009 | 0.0382 | 0.2264 | 0.0926 | 0.2777
DBpedia test | 0.0527 | 0.3039 | 0.0219 | 0.1998 | 0.0153 | 0.0957 | 0.0406 | 0.1506
DBpedia train | 0.1016 | 0.3416 | 0.1102 | 0.1807 | 0.0029 | 0.0436 | 0.0969 | 0.1968
DeepPavlov
Wikidata test | 0.0013 | 0.1240 | not supported | 0.0005 | 0.0870 | not supported
Wikidata train | 0.0017 | 0.1032 | not supported | 0.0009 | 0.0993 | not supported
Platypus
Wikidata test | 0.0126 | 0.1503 | not supported | not supported | 0.0000 | 0.0417
Wikidata train | 0.0028 | 0.1070 | not supported | not supported | 0.0063 | 0.0750
TABLE V: Evaluation results on QALD-9-plus with original QALD-9 multilingual
translations. The table structure is similar to Table IV. Only overlapping
languages (German, Russian, French) are used for the better comparability
(English was not changed). QAnswer
---
| German | Russian | French
Micro F1 | Macro F1 | Micro F1 | Macro F1 | Micro F1 | Macro F1
Wikidata test | 0.0291 | 0.3171 | 0.0157 | 0.2059 | 0.1190 | 0.2300
Wikidata train | 0.0587 | 0.3013 | 0.0383 | 0.2264 | 0.1465 | 0.2813
DBpedia test | 0.0219 | 0.1998 | 0.0154 | 0.1023 | 0.0406 | 0.1506
DBpedia train | 0.1087 | 0.1807 | 0.0029 | 0.0461 | 0.0190 | 0.1680
DeepPavlov
Wikidata test | not supported | 0.0005 | 0.0870 | not supported
Wikidata train | not supported | 0.0009 | 0.0993 | not supported
Platypus
Wikidata test | not supported | not supported | 0.0000 | 0.0417
Wikidata train | not supported | not supported | 0.0015 | 0.1004

###  III-E Quantitative and Qualitative Question Analysis

We used several language-agnostic quantitative text analysis measures in order
to observe the change between QALD-9 and QALD-9-plus textual representations
w.r.t. the descriptive statistics. Consequently, we analyzed only those
languages that were present in both QALD-9 and QALD-9-plus. To do this, the
LinguaF library999https://github.com/Perevalov/LinguaF for Python was used.
The results of comparison are demonstrated in Table III.

Given the data, it is obvious, that for all the QALD-9-plus translations
longer words were used. In addition, there are more words in the QALD-9-plus
translated questions and consequently, the translated question length was
increased. The Type-Token Ratio (TTR) that corresponds to the ratio between
unique words and total number of words was also increased for Russian and
French, and not changed for German. This analysis shows us that the
QALD-9-plus multilingual representations are more complex and rich in
comparison to the original QALD-9 translations. As the translations were done
by native speakers, we consider this results as an implicit marker of improved
translation quality. Additionally, a well-experienced English teacher from
Germany did an evaluation of the translation quality. The original dataset was
rated with 4.09 points out of 5. The new German translations of the questions
were also evaluated, and only 19 question translations were optimized in the
final validation process step (where 4 changes addressed typos and missing
quotation marks). Hence, we can also conclude that the new German translations
have a very high quality (5 out of 5 according to the previous rating).

###  III-F Impact and Usability

It was decided to keep the QALD-JSON format101010https://github.com/dice-
group/gerbil/wiki/Question-Answering#web-service-interface for the QALD-9-plus
as it enables the researchers to reuse their systems for the evaluation (e.g.,
GERBIL [16]). Hence, researchers and developers are enabled to benchmark and
compare their KGQA systems in both mono- and multilingual settings over
DBpedia and Wikidata. As there are several alternative translations provided,
it is possible to develop techniques for paraphrasing and evaluating machine
translation quality. The same is true for the SPARQL queries over DBpedia and
Wikidata.

##  IV Evaluation of Multilingual KGQA Systems

We evaluated our QALD-9-plus dataset on currently available multilingual KGQA
systems: QAnswer [11], DeepPavlov [17] and Platypus [18]. QAnswer supports 11
languages (en, de, fr, it, es, pt, nl, zh, ar, ja, ru) and works over DBpedia
and Wikidata, DeepPavlov – 2 languages (en, ru) and works over Wikidata, and
Platypus – 2 languages (en, fr) and works over Wikidata. Consequently, we
decided to evaluate the systems on the languages that are present in
QALD-9-plus and are supported by them natively. The evaluation was done using
the GERBIL system, the URIs of the experimental runs with detailed data (see
the online appendix) are stored in the README.md file of the dataset. The
results are demonstrated in Table IV.

From the results we see that QAnswer has strong dominance w.r.t. English and
French. DeepPavlov performed better on the Wikidata test subset in Russian,
however, QAnswer is better on the corresponding train subset. The results for
German language and DBpedia subset are not comparable due to the lack of its
support.

Additionally, the original multilingual questions from QALD-9 were used in the
evaluation while having updated SPARQL queries from QALD-9-plus (see Table V).
For simpler comparability, only languages that were used for the QALD-9-plus
were selected (i.e., German, Russian, and French). Given the results, no
particular tendencies w.r.t. the quality difference between QALD-9 and
QALD-9-plus were identified. Hence, while having a significantly different
quantitative text measures between the original and extended translations (see
Section III-E), the actual QA quality was not changed significantly. We
hypothesize, that such result may be caused by the high robustness of the
considered KGQA systems to the grammatical correctness of the questions or the
still very keyword-dependent behavior of the considered KGQA systems. Together
with the robustness, the systems may also be capable of answering only a
particular subset of questions, that might be the reason why the quality
values for some experiments did not change. However, this aspect has to be
investigated further and is beyond the scope of this paper.

##  V Conclusion and Future Work

In this paper, we presented a new benchmark for KGQA called QALD-9-plus. It is
based on QALD-9 and was extended by creating translations from its English
questions to German, French, Russian, Armenian, Belarusian, Lithuanian,
Bashkir, and Ukrainian. The translations were done by native speakers of
corresponding languages in a crowdsourcing setting. In addition, the DBpedia
SPARQL queries from QALD-9 were transferred to Wikidata to improve the
usability of the data. As QALD-9-plus contains multiple text representations
for several languages and the questions are multilingual (i.e., parallel
corpus), it enables researchers to address the paraphrasing and machine
translation tasks. In addition, paraphrasing may be done on the SPARQL query
level (i.e., from DBpedia to Wikidata). QALD-9-plus is keeping the QALD-JSON
format in order to be reusable by the research community. The quantitative and
qualitative comparison of the QALD-9 and QALD-9-plus multilingual question
representations illustrated that the quality of the translation was improved.
We consider QALD-9-plus as a significant contribution to the multilingual KGQA
research community that creates wider possibilities for evaluation of KGQA
systems.

By evaluation on three state-of-the-art KGQA systems using QALD-9-plus, we
demonstrated that the extension to other KGs, namely Wikidata, and improving
the translations gave new insights into these systems’ performance.

In the future, we will increase the coverage of languages (e.g., Bashkir,
Armenian, and French) and extend the number of languages in the dataset. We
also will enlarge the number of questions in the dataset, extend their meta-
information (e.g., expected answer type, named entities etc.), and align
SPARQL queries with the newest instance of DBpedia.

## Acknowledgment

Authors of the paper would like to thank all the contributors involved in the
translation of the dataset, specifically: Konstantin Smirnov, Mikhail
Orzhenovskii, Andrey Ogurtsov, Narek Maloyan, Artem Erokhin, Mykhailo Nedodai,
Aliaksei Yeuzrezau, Anton Zabolotsky, Artur Peshkov, Vitaliy Lyalin, Artem
Lialikov, Gleb Skiba, Vladyslava Dordii, Polina Fominykh, Tim Schrader,
Susanne Both, and Anna Schrader. Additionally, authors would like to give
thanks to Open Data Science community111111https://ods.ai/ for connecting data
science enthusiasts all over the world.

## References

  * [1] D. Diefenbach, K. Singh, A. Both, D. Cherix, C. Lange, and S. Auer, “The Qanary ecosystem: Getting new insights by composing question answering pipelines,” in _Web Engineering - 17th International Conference, ICWE 2017, Rome, Italy, June 5-8, 2017, Proceedings_ , ser. Lecture Notes in Computer Science, J. Cabot, R. D. Virgilio, and R. Torlone, Eds., vol. 10360\. Springer, 2017, pp. 171–189.
  * [2] A. Perevalov and A. Both, “Augmentation-based answer type classification of the SMART dataset,” in _Proceedings of the SeMantic AnsweR Type prediction task (SMART) at ISWC 2020_ , ser. CEUR Workshop Proceedings, vol. 2774. CEUR-WS.org, 2020, pp. 1–9. [Online]. Available: http://ceur-ws.org/Vol-2774/paper-01.pdf
  * [3] R. Usbeck, R. H. Gusmita, A. N. Ngomo, and M. Saleem, “9th challenge on question answering over linked data (QALD-9),” 2018.
  * [4] I. Rybin, V. Korablinov, P. Efimov, and P. Braslavski, “RuBQ 2.0: An innovated russian question answering dataset,” in _The Semantic Web_. Cham: Springer International Publishing, 2021, pp. 532–547.
  * [5] R. Cui, R. Aralikatte, H. Lent, and D. Hershcovich, “Multilingual compositional Wikidata questions,” _arXiv:2108.03509_ , 2021.
  * [6] E. Loginova, S. Varanasi, and G. Neumann, “Towards end-to-end multilingual question answering,” _Information Systems Frontiers (ISF)_ , vol. 22, pp. 1–14, 3 2020.
  * [7] A. Neves, A. Lamurias, and F. Couto, “Biomedical question answering using extreme multi-label classification and ontologies in the multilingual panorama,” in _Semantic Indexing and Information Retrieval for Health Held in conjunction with the 42nd European Conference on Information Retrieval (SIIRH@ECIR)_ , 2020.
  * [8] S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives, “DBpedia: A nucleus for a web of open data,” in _The semantic web_. Springer, 2007.
  * [9] K. Höffner, S. Walter, E. Marx, R. Usbeck, J. Lehmann, and A.-C. Ngonga Ngomo, “Survey on challenges of question answering in the semantic web,” _Semantic Web_ , vol. 8, 11 2016.
  * [10] D. Sorokin and I. Gurevych, “Modeling semantics with gated graph neural networks for knowledge base question answering,” _arXiv preprint arXiv:1808.04126_ , 2018.
  * [11] D. Diefenbach, A. Both, K. Singh, and P. Maret, “Towards a question answering system over the semantic web,” _Semantic Web_ , vol. 11, pp. 421–439, 2020\.
  * [12] L. Siciliani, P. Basile, P. Lops, and G. Semeraro, “MQALD: Evaluating the impact of modifiers in question answering over knowledge graphs,” _Semantic Web_ , vol. Pre-press, pp. 1–17, 09 2021.
  * [13] D. Vrandečić and M. Krötzsch, “Wikidata: A free collaborative knowledgebase,” _Communications of the ACM_ , vol. 57, no. 10, p. 78–85, Sep. 2014.
  * [14] D. Keysers, N. Schärli, N. Scales, H. Buisman, D. Furrer, S. Kashubin, N. Momchev, D. Sinopalnikov, L. Stafiniak, T. Tihon _et al._ , “Measuring compositional generalization: A comprehensive method on realistic data,” _arXiv preprint arXiv:1912.09713_ , 2019.
  * [15] K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor, “Freebase: a collaboratively created graph database for structuring human knowledge,” in _Proceedings of the 2008 ACM SIGMOD international conference on Management of data_ , 2008, pp. 1247–1250.
  * [16] R. Usbeck, M. Röder, A.-C. Ngonga Ngomo, C. Baron, A. Both, M. Brümmer, D. Ceccarelli, M. Cornolti, D. Cherix, B. Eickmann, P. Ferragina, C. Lemke, A. Moro, R. Navigli, F. Piccinno, G. Rizzo, H. Sack, R. Speck, R. Troncy, J. Waitelonis, and L. Wesemann, “Gerbil: General entity annotator benchmarking framework,” in _Proceedings of the 24th International Conference on World Wide Web_ , ser. WWW ’15. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee, 2015, p. 1133–1143.
  * [17] M. Burtsev, A. Seliverstov, R. Airapetyan, M. Arkhipov, D. Baymurzina, N. Bushkov, O. Gureenkova, T. Khakhulin, Y. Kuratov, D. Kuznetsov, A. Litinsky, V. Logacheva, A. Lymar, V. Malykh, M. Petrov, V. Polulyakh, L. Pugachev, A. Sorokin, M. Vikhreva, and M. Zaynutdinov, “DeepPavlov: Open-source library for dialogue systems.” Melbourne, Australia: Association for Computational Linguistics, 2018, pp. 122–127.
  * [18] T. P. Tanon, M. D. de Assuncao, E. Caron, and F. M. Suchanek, “Demoing Platypus–a multilingual question answering platform for Wikidata,” in _European Semantic Web Conference_. Springer, 2018, pp. 111–116.