# Data Distribution Bottlenecks in Grounding Language Models to Knowledge
Bases

Yiheng Shu
The Ohio State University
<EMAIL_ADDRESS>
&Zhiwei Yu
Microsoft
<EMAIL_ADDRESS>
* Work performed during internship at Microsoft Research Asia.

###### Abstract

Language models (LMs) have already demonstrated remarkable abilities in
understanding and generating both natural language and formal language.
Despite these advances, their integration with real-world environments such as
large-scale knowledge bases (KBs) remains an underdeveloped area, affecting
applications such as semantic parsing and indulging in "hallucinated"
information. This paper is an experimental investigation aimed at uncovering
the robustness challenges that LMs encounter when tasked with knowledge base
question-answering (KBQA). The investigation covers scenarios with
inconsistent data distribution between training and inference, such as
generalization to unseen domains, adaptation to various language variations,
and transferability across different datasets. Our comprehensive experiments
reveal that even when employed with our proposed data augmentation techniques,
advanced small and large language models exhibit poor performance in various
dimensions. While the LM is a promising technology, the robustness of the
current form in dealing with complex environments is fragile and of limited
practicality because of the data distribution issue. This calls for future
research on data collection and LM learning paradigms111Code and data will be
public upon acceptance..

Data Distribution Bottlenecks in Grounding Language Models to Knowledge Bases

Yiheng Shu††thanks: * Work performed during internship at Microsoft Research
Asia. The Ohio State University<EMAIL_ADDRESS>Zhiwei Yu Microsoft
<EMAIL_ADDRESS>

##  1 Introduction

Language models (LMs), such as BERT Devlin et al. (2019), T5 Raffel et al.
(2020), and the GPT series Ouyang et al. (2022); OpenAI (2023), have
demonstrated impressive capabilities in understanding and generating natural
or formal language, highlighting the potential for artificial general
intelligence (AGI). However, several obstacles must be overcome to achieve
this goal. For example, to mitigate the issue of "hallucinated" information
and expand the range of LM applications, researchers have focused on grounding
LMs to real-world environments (e.g., Web, database, knowledge base) Nakano et
al. (2021); Menick et al. (2022), which enables real-time fact-checking, data
validation and thereby improve the reliability of model responses. Knowledge
base (KB) is one underdeveloped environment among them. The task of Knowledge
Base Question Answering (KBQA) aims to parse natural language queries using
KBs, such as Freebase Bollacker et al. (2008), and Wikidata Vrandecic and
Krötzsch (2014). Now, numerous LM-driven models Das et al. (2021); Hu et al.
(2022) continue to achieve higher scores on KBQA benchmarks, but grounding LMs
to KBs has not been fully examined for robustness and reliability. A few
critical gaps in existing works prompt the necessity for this work. First,
while the evaluation of LMs typically occurs in natural language tasks, the
challenge escalates when grounding these models to real-world environments
like KBs Liu et al. (2023), where the data contains structured data instead of
all unstructured natural language. Second, the metrics of the KBQA benchmark
are often shallow and the robustness of the model is not adequately evaluated.
Finally, recent surveys on KBQA Lan et al. (2022); Gu et al. (2022b) have
largely omitted the strides made in the development and application of LMs,
especially large language models (LLMs). As a result, there remains a clear
gap in understanding the robustness challenges and solutions specific to
grounding LMs to KBs.

Figure 1: Inconsistent KBQA data distribution between training and inference.

For deep learning models, robustness is closely related to data distribution
Hendrycks et al. (2020). The efficacy of language models (LMs) is not just a
product of their architecture, but also a reflection of the data on which they
are trained. In simpler tasks and well-defined domains, large-scale corpora
have been collected and used for effective training Touvron et al. (2023).
However, real-world environments are rarely so accommodating, e.g., large KBs
contain complex structures and schema items, and building a representative
corpus is more challenging. Inconsistency in data distribution during training
and inference, as shown in Figure 1, may negatively impact the performance and
robustness of LMs. Given this backdrop, it becomes essential to approach the
task of grounding LMs from a multifaceted standpoint. Specifically, we believe
that a detailed understanding of this grounding task requires exploring
environmental, linguistic, and model learning aspects.

Through these explorations, this paper aims to provide a more holistic
understanding of the challenges and opportunities that arise in the LM
grounding and KBQA benchmarking processes. We review existing works and
identify several challenges, including: 1) generalization to unseen domains at
the schema level Gu et al. (2021), 2) adaptation to paraphrases featuring
diverse language variations Su et al. (2016), 3) transferability across
datasets with novel schema items, query patterns, and linguistic styles Cao et
al. (2022b), and 4) few-shot in-context learning capabilities of grounding
LLMs Li et al. (2023). These aspects allow us to dissect and critique the
robustness and applicability of LMs when operating in complex real-world
scenarios, thereby fulfilling our objective of effective interaction between
LMs and environments.

To substantiate the presence of these challenges and obtain additional
empirical insights, we undertake a comprehensive experimental investigation
for the above aspects. To mitigate the negative effects of inconsistent data
distributions in our experiments, we propose a data augmentation method for
any LM and a retrieval augmentation method for any LLM. Our findings reveal
that even when employed with such techniques (the highest EM score is achieved
on the GrailQA benchmark), advanced small and large LMs still fall short of
effectively tackling the majority of these challenges. A striking example is
the large difference between the best practice without WebQSP Yih et al.
(2016) fine-tuning (F1 43.0%) compared to the fine-tuned state-of-the-art (F1
79.6%) (§6), suggesting the weak robustness of LM-driven KBQA models on an
unseen dataset. Such observations highlight an urgent need for future research
in data collection methodologies and LM learning paradigms. Meanwhile, we
expect the evaluation approaches in this paper to provide a reference for
future benchmark construction, developing metrics that take robustness into
account (§3).

##  2 Preliminary

In this paper, the knowledge base (KB) refers to an RDF222Resource Description
Framework by W3C standard. graph, consisting of triples $(s,r,o)$, where $s$
is a subject, $r$ is a relation and $o$ is an object. Logical form refers to a
formal query that can be executed on a KB like SPARQL and S-expression Gu et
al. (2021). Schema refers to the rdfs:Class (class) and rdf:Property
(relation) in the RDF Schema333https://www.w3.org/TR/rdf12-schema/.

Benchmark | KB | Size | LF | Generalization | Para.
---|---|---|---|---|---
WebQuestions Berant et al. (2013) | Freebase | 5,810 | N/A | i.i.d. | ✗
SimpleQuestions Bordes et al. (2015) | Freebase | 108,442 | N/A | i.i.d. | ✗
WebQuestionsSP Yih et al. (2016) | Freebase | 4,737 | SPARQL | i.i.d. | ✗
GraphQuestions Su et al. (2016) | Freebase | 5,166 | Graph query | comp.+zero | ✓
LC-QuAD Trivedi et al. (2017) | DBpedia | 5,000 | SPARQL | i.i.d. | ✗
CWQ Talmor and Berant (2018) | Freebase | 34,689 | SPARQL | i.i.d. | ✗
LC-QuAD 2.0 Dubey et al. (2019) | Wikidata | 30,000 | SPARQL | i.i.d. | ✓
SQB Wu et al. (2019) | Freebase | 108,443 | N/A | i.i.d.+zero | ✗
CFQ Keysers et al. (2020) | Freebase | 239,357 | SPARQL | comp | ✗
GrailQA Gu et al. (2021) | Freebase | 64,331 | S-expression | i.i.d.+comp.+zero | ✓
KQA Pro Cao et al. (2022a) | Wikidata | 117,970 | KoPL | i.i.d. | ✗
QALD series Perevalov et al. (2022) | DBpedia | 558 | SPARQL | comp. | ✗
Table 1:  Selected KBQA benchmarks. LF: logical forms. Generalization settings
follow Gu et al. (2021). i.i.d. denotes that the schema distribution in the
test set is the same as the training set. comp. and zero denote compositional
and zero-shot generalization, respectively. Para. denotes paraphrases, i.e.,
questions containing the same semantics (machine-generated paraphrases are not
included).

##  3 Robustness Challenge

In this paper, the robustness of a KBQA model refers to its ability to adapt
to various natural language inputs and maintain consistent performance when
data distribution shifts. Due to the data discrepancy between the training
corpus of LMs and KB environments, LMs face challenges from environmental,
linguistic, and model learning aspects.

###  3.1 Environmental Aspect

One of the principal challenges from an environmental perspective is schema-
level generalization. The RDF Schema offers a data-modeling vocabulary, which
is essential for querying a KB. As shown in Table 1, most KBQA benchmarks
operate under the assumption that the schema distribution is identically and
independently distributed (i.i.d.) between training and testing scenarios.
This means that the schema items encountered during testing are already
familiar to the model from the training data. However, this assumption often
falls short in large-scale KBs, which contains thousands of schema items.
Currently, only a limited number of benchmarks aim to evaluate the ability to
generalize at the schema level, especially when it comes to handling
non-i.i.d. schema items. The SimpleQuestions-Balance dataset Wu et al. (2019)
reconfigures the i.i.d. SimpleQuestions dataset Petrochuk and Zettlemoyer
(2018) to ensure that half of the questions in the testing (or validation) set
are mapped to KB relations not present in the training data. GrailQA Gu et al.
(2021) is structured to evaluate three levels of schema-level generalization:
i.i.d. (25%), compositional generalization (25%), and zero-shot generalization
(50%). GraphQuestions Su et al. (2016) presents an even more rigorous test as
its testing set features schema items seldom encountered in the training data.
Given the impracticality of maintaining the i.i.d. assumption in real-world
applications, these datasets offer a more realistic portrayal of the
generalization challenges that QA systems face in practical settings. Despite
recent advances Shu et al. (2022); Gu et al. (2022a), challenges in
compositional and zero-shot generalization are far from solved.

###  3.2 Linguistic Aspect

Natural language is variable in form, making question understanding
challenging for KBQA models. One common way this variety shows up is through
paraphrasing. In this paper, a paraphrase set denotes different ways to
express the same logical form, as illustrated in Table 14. To gauge how well a
KBQA model can adapt to these different forms of natural language, a
straightforward approach is to test if the model can accurately answer
paraphrased questions that it has already answered correctly before.
Unfortunately, as shown in Table 1, many KBQA benchmarks do not account for
paraphrasing with only one utterance for each logical form. Exceptionally,
some datasets Su et al. (2016); Dubey et al. (2019); Gu et al. (2021) are
based on automatically generated logical forms and include multiple natural
language expressions for the same logical form (template). These data
characteristics highlight the difficulties in adapting to paraphrased
questions.

###  3.3 Integrated Aspect

Evaluating KBQA benchmarks often hinges on a single dataset, thereby
complicating the task of ascertaining the model performance consistency across
novel scenarios. This form of robustness, termed as cross-dataset transfer in
this paper, combines both the environmental and linguistic aspects discussed
earlier and is more difficult to achieve. This is because construction methods
vary across datasets, as do schema distributions and natural language
expressions. Specifically, KBQA dataset construction generally falls into two
distinct categories: 1) Graph Search and Crowdsourcing: in this approach,
logical forms or triples are initially extracted from a KB, where structures
or operators of logical form are usually finite. Subsequently, these are
converted into natural language utterances through crowdsourcing techniques
Bordes et al. (2015); Trivedi et al. (2017). 2) Human Curation and Parsing:
logical forms are labeled directly from human-provided utterances Berant et
al. (2013); Perevalov et al. (2022). Existing works Gu et al. (2021); Cao et
al. (2022b) suggest that models pre-trained on large-scale datasets can adapt
reasonably well to other target datasets, such as WebQuestionsSP Yih et al.
(2016). However, the necessity for fine-tuning these pre-trained models on the
intended target dataset remains imperative for achieving optimal performance.
That is, despite the advantages offered by pre-training on expansive KBQA
datasets, models still encounter challenges in transferring directly to
previously unseen target datasets while sustaining high performance.

###  3.4 Learning Aspect

Aside from considering environmental and linguistic factors, it is crucial to
focus on the model learning process. Recently, LLMs like GPT series OpenAI
(2023) have demonstrated exceptional capabilities across a variety of tasks,
outperforming smaller yet potent LMs such as BERT Devlin et al. (2019) and T5
Raffel et al. (2020). Despite these advancements, these LLMs face substantial
challenges when interacting with environments. One notable issue is their
predominant reliance on an in-context learning paradigm as opposed to fine-
tuning, as a trade-off between computational cost and model efficiency. In
comparison to fine-tuning, in-context learning offers the advantage of reduced
training costs but at the expense of the lacking perception of the
environment. Inconsistent data distribution between natural language pre-
training and reasoning over structured knowledge contexts leads to poor
performance. For instance, a discernible performance gap exists between KBQA
models that employ in-context learning with Codex Chen et al. (2021a) and
those built on fine-tuned LMs Gu et al. (2022a); Li et al. (2023). Therefore,
it is crucial to consider the limitations of the commonly used in-context
learning paradigm and improve the grounding approaches.

##  4 Approach

The goal of our proposed approaches is to reduce the negative effects of
inconsistent data distributions and to maximize the potential of LMs in our
experimental investigations. We introduce both data augmentation and retrieval
augmentation techniques.

###  4.1 Data Augmentation for LMs

Off-the-shelf datasets of limited size may make LM easily overfitted and not
adaptable to large KBs. To address the problem that many domains in the KB are
often not collected as training data, we propose a data augmentation method
named Graph seArch and questIon generatioN (GAIN). GAIN applies to KBQA
corresponding to logical forms or triples, and scales data volume and
distribution through four steps: 1) Graph search: Sampling logical forms or
triples from arbitrary domains in the KB, without being restricted to any
particular KBQA dataset. 2) Training question generator on existing KBQA
datasets, i.e., learning to convert logical forms or triples into natural
language questions. 3) Verbalization: Using the question generator from step 2
to verbalize sampled logical forms or triples from step 1, thus creating
synthetic questions. 4) Training data expansion: Before fine-tuning any neural
models on KBQA datasets, GAIN-synthetic data can be used to train these models
or to expand the corpus of in-context samples for LLMs. That is, as a data
augmentation method, GAIN is not a KBQA model, but it is used to augment a
base KBQA model.

###  4.2 Retrieval Augmentation for LLMs

As the trade-off between cost and effectiveness, we experiment with the
prevalent in-context learning paradigm but attempt to improve the quality of
in-context samples. We use advanced retrieval methods based on smaller LMs as
plug-ins to augment the LLM, similar to the SuperICL approach Xu et al.
(2023). Specifically, our steps to generate an LLM prompt for each question
include the following. 1) Given an input question, we retrieve $k$ questions
($k$-shot) with BM25 Robertson et al. (2009) from the corpus (the combination
of KBQA training set and the GAIN-synthetic dataset). 2) The role of retrieval
augmentation for KB environments has been shown by fine-tuned LMs Shu et al.
(2022). To assist with grounding LLM, we retrieve KB contexts with off-the-
shelf retrievers for $k$ samples and the input question444The prompt example
is demonstrated in Appendix A..

##  5 Experiments

###  5.1 Setup

#### Datasets

We use GrailQA Gu et al. (2021) and GraphQuestions Su et al. (2016) for
schema-level generalization and paraphrase settings. We also study
generalization on SimpleQuestions-Balance (SQB) Wu et al. (2019) with unseen
relations. WebQuestionsSP (WebQSP) Yih et al. (2016) is employed for cross-
dataset transfer experiments as it is based on practical human-curated search
logs. All experiments use S-expression as the logical form due to its clear
and concise structure. Entity linking results are taken from TIARA Shu et al.
(2022) for GrailQA and WebQSP, and ArcaneQA Gu and Su (2022) for
GraphQuestions.

#### Model

We report the performance of comparison models from their papers. For the
relation linking task on SQB, we use BERT Devlin et al. (2019) as the base
model for GAIN. For KBQA tasks, we use the open-source advanced model TIARA
Shu et al. (2022) as the base model for GAIN, due to its strong performance on
zero-shot schema items555Pangu Gu et al. (2022a) also uses entity linking
results from TIARA.. TIARA is composed of multi-grained retrievers and a
generator, with the retrievers providing KB contexts666Entities, exemplary
logical forms (ELF) and schema items are retrieved. for the generator. The
term “TIARA+GAIN” represents a model (both the retrievers and the generator)
that is first tuned using GAIN synthetic data and subsequently fine-tuned on a
target dataset. For LLM evaluation, we use the latest
gpt-3.5-turbo-0613777https://platform.openai.com/docs/models model, and the
few-shot contexts are retrieved from the combination of GrailQA training set
and synthetic dataset using the TIARA+GAIN retrievers.

#### Metrics

Following previous works, we use Exact Match (EM), F1, and Hits@1 to measure
the performance of KBQA models. To evaluate adaptability to paraphrases
(§3.2), we calculate the standard deviation (std) of EM/F1 for questions of
each logical form template. As shown in Equation 1, suppose there are $n$ sets
of paraphrases in the dataset, each set of paraphrases corresponds to a
logical form template with $m$ natural language expressions, and the F1 score
obtained by the KBQA model on the $j$-th question of the $i$-th set of
paraphrases is $F1_{i,j}$. The metric $Std_{F1}$ first calculates the standard
deviation of the F1 scores obtained by the model on the $m$ questions for each
set of paraphrases and then calculates the average of the $n$ standard
deviations. This metric is used to measure the robustness of the model to
different representations of the same semantics, i.e., whether it can cope
with diverse natural language expressions. A lower standard deviation
indicates that the model is more adaptive to different expressions. $Std_{EM}$
is calculated in the same way.

$Std_{F1}=\frac{1}{n}\sum_{i=1}^{n}\sqrt{\left(\frac{\sum_{j=1}^{m}(F1_{i,j}-\bar{F1_{i}})^{2}}{m}\right)}$
(1)

#question | #one-hop | #two-hop | #domain
---|---|---|---
127,329 | 78,668 | 48,661 | 759
#none | #count | #comparatives | #superlatives
115,221 | 7,115 | 1,874 | 3,119
#class | #relation | #entity |
5,078 | 12,942 | 46,645 |

Table 2: Statistics for the synthetic dataset of logical forms. none denotes no function. #question | #relation | #subject | #domain
---|---|---|---
162,557 | 7,349 | 108,804 | 673
Table 3: Statistics for the synthetic dataset of triples. Subject denotes subject entities. | Overall | I.I.D. | Compositional | Zero-shot
---|---|---|---|---
Model on GrailQA Test Set | EM | F1 | EM | F1 | EM | F1 | EM | F1
Fine-tuned Models
BERT + Ranking (Gu et al., 2021) | 50.6 | 58.0 | 59.9 | 67.0 | 45.5 | 53.9 | 48.6 | 55.7
RnG-KBQA (Ye et al., 2022) | 68.8 | 74.4 | 86.2 | 89.0 | 63.8 | 71.2 | 63.0 | 69.2
TIARA (T5-base) (Shu et al., 2022) | 73.0 | 78.5 | 87.8 | 90.6 | 69.2 | 76.5 | 68.0 | 73.9
DecAF (FiD-3B) (Yu et al., 2022) | 68.4 | 78.8 | 84.8 | 89.9 | 73.4 | 81.8 | 58.6 | 72.3
Pangu (BERT-base) (Gu et al., 2022a) | 73.7 | 79.9 | 82.6 | 87.1 | 74.9 | 81.2 | 69.1 | 76.1
Pangu (T5-large) (Gu et al., 2022a) | 74.8 | 81.4 | 82.5 | 87.3 | 75.2 | 82.2 | 71.0 | 78.4
Pangu (T5-3B) (Gu et al., 2022a) | 75.4 | 81.7 | 84.4 | 88.8 | 74.6 | 81.5 | 71.6 | 78.5
Codex-driven Models
B-BINDER (6)-R (Li et al., 2023) | 53.2 | 58.5 | 72.5 | 77.4 | 51.8 | 58.3 | 45.0 | 49.9
Pangu (Codex) (Gu et al., 2022a) | 56.4 | 65.0 | 67.5 | 73.7 | 58.2 | 64.9 | 50.7 | 61.1
GAIN-augmented Models
TIARA + GAIN (T5-base) | 75.1 | 80.6 | 88.3 | 91.0 | 73.0 | 79.6 | 69.9 | 76.4
TIARA + GAIN (T5-3B) | 76.3 | 81.5 | 88.5 | 91.2 | 73.7 | 80.0 | 71.8 | 77.8
GPT-3.5-turbo (5-shot) | 66.6 | 71.4 | 82.7 | 85.3 | 60.5 | 66.3 | 61.9 | 67.2
Table 4: EM and F1 scores (%) on the hidden test set of GrailQA.

###  5.2 Implementation Details

Our experiments are done on the machine with an NVIDIA A100 GPU and up to
504GB of RAM. We implement our models utilizing PyTorch Paszke et al. (2019)
and Hugging Face888https://huggingface.co/. TIARA+GAIN (T5-3B) takes about 100
hours to train the logical form generator on the synthetic dataset.

#### Training Question Generator

We fine-tune the T5-base model Raffel et al. (2020) to convert S-expression or
triple to natural language questions. We set the beam size to 10, the learning
rate to 3e-5, the number of epochs to 10, and the batch size to 8.

#### Training TIARA

The training of the TIARA model Shu et al. (2022) follows its original
settings, including the setting of hyperparameters and the calculation of
metrics. Note that Hits@1 on TIARA is obtained by randomly selecting one
answer for each question 100 times. Both the schema retriever and generator of
TIARA are pre-trained on synthetic data and then fine-tuned on KBQA datasets.
Since GraphQuestions has no official training-valid split, we randomly take
200 questions from the original training set as the valid set.

#### Training the Relation Linking Model

We use the BERT-base-uncased model Devlin et al. (2019) to rank candidate
relations for SQB, and the input form is the same as the schema retriever of
TIARA. We set the learning rate to 3e-5, the batch size to 256, and the
maximum number of epochs to 3 with early stopping.

###  5.3 Data Augmentation

The statistics of GAIN-synthetic datasets for both logical forms and triples
are shown in Table 2 and 3999Details of synthetic data are shown in Appendix
B.. Note that theoretically, the sampling of the GAIN method is not limited to
the scale of the synthetic data we use here.

##  6 Analysis

Model on GraphQuestions |  F1($\uparrow$) |  Std($\downarrow$)
---|---|---
GraphQuestions on Freebase 2013-07
UDepLambda Reddy et al. (2017) | 17.7 | -
PARA4QA Dong et al. (2017) | 20.4 | -
SPARQA Sun et al. (2020) | 21.5 | -
BERT + Ranking Gu et al. (2021) | 25.0 | -
ArcaneQA Gu and Su (2022) | 31.8 | -
TIARA♣ (T5-base) Shu et al. (2022) | 37.9 | 0.141
KB-BINDER (6) Li et al. (2023) | 39.5 | -
TIARA + GAIN (T5-base) | 45.5 | 0.153
TIARA + GAIN (T5-3B) | 48.7 | 0.180
GraphQuestions on Freebase 2015-08-09
BERT + Ranking Gu et al. (2021) | 27.0 | -
ArcaneQA Gu and Su (2022) | 34.3 | -
TIARA♣ (T5-base) Shu et al. (2022) | 41.2 | 0.157
Pangu (Codex) Gu et al. (2022a) | 44.3 | -
Pangu (T5-3B) Gu et al. (2022a) | 62.2 | -
TIARA + GAIN (T5-base) | 49.5 | 0.170
TIARA + GAIN (T5-3B) | 53.0 | 0.200

Table 5: F1 scores (%) and average standard deviation (std) of F1 scores for each set of paraphrases on the test set of GraphQuestions. The setting for Freebase 2015-08-09 is described by Gu and Su (2022). ♣ denotes our replication results. | Overall | Seen | Unseen
---|---|---|---
Model on SimpleQuestions-Balance | 1 | 5 | 10 | 1 | 5 | 10 | 1 | 5 | 10
HR-BiLSTM Wu et al. (2019) | 63.3 | - | - | 93.5 | - | - | 33.0 | - | -
Adversarial-Adapter Wu et al. (2019) | 84.9 | - | - | 92.6 | - | - | 77.1 | - | -
BERT-base | 83.7 | 95.0 | 96.9 | 85.8 | 95.0 | 96.0 | 81.5 | 95.1 | 97.8
BERT-base + GAIN | 88.4 | 96.0 | 97.3 | 87.8 | 95.4 | 96.3 | 89.1 | 96.7 | 98.4
Table 6: Hits@$k$ (1, 5, 10) scores (%) for relation linking on the test set of SimpleQuestions-Balance, including seen and unseen relations. | Std Overall | Std I.I.D. | Std Compositional | Std Zero-shot
---|---|---|---|---
Model on GrailQA Valid Set |  EM($\downarrow$) |  F1($\downarrow$) | EM | F1 | EM | F1 | EM | F1
TIARA ELF only Shu et al. (2022) | 0.066 | 0.062 | 0.017 | 0.016 | 0.148 | 0.152 | 0.198 | 0.181
TIARA (T5-base) Shu et al. (2022) | 0.079 | 0.066 | 0.021 | 0.017 | 0.211 | 0.203 | 0.222 | 0.181
TIARA + GAIN ELF only | 0.063 | 0.060 | 0.017 | 0.017 | 0.138 | 0.148 | 0.187 | 0.169
TIARA + GAIN (T5-base) | 0.077 | 0.061 | 0.020 | 0.016 | 0.215 | 0.198 | 0.218 | 0.160
TIARA + GAIN (T5-3B) | 0.075 | 0.058 | 0.020 | 0.016 | 0.196 | 0.180 | 0.212 | 0.155
GPT-3.5-turbo (5-shot) | 0.093 | 0.091 | 0.027 | 0.023 | 0.272 | 0.281 | 0.251 | 0.247
Table 7: Average standard deviation of EM and F1 scores for each set of
paraphrases on the GrailQA valid set. ELF denotes exemplary logical form Shu
et al. (2022).

#### Schema-Level Generalization

As shown in Tables 4 and 5, the models perform significantly better on i.i.d.
than compositional and zero-shot generalization, with the zero-shot
generalization being the most challenging. Besides, an increased number of
model parameters, combined with richer data from GAIN, significantly enhance
the generalization capabilities of T5 models. TIARA+GAIN achieves the highest
EM scores, including that on zero-shot scenes. This demonstrates promising
ideas for further improving LM generalization capabilities, i.e., the positive
effect of synthetic data and parametric scales on training LMs. However, it is
important to note that fine-tuned models consistently outperform few-shot
learning models, regardless of whether the schema is seen or not. Given the
training and inference costs of LLMs, their performance has yet to show any
superiority in this semantic parsing task.

Model on WebQSP | F1 | Hits@1
---|---|---
TIARA♣ (T5-base) Shu et al. (2022) | 28.5 | 27.6
TIARA*♣ (T5-base) Shu et al. (2022) | 33.5 | 31.5
BERT + Ranking* Gu et al. (2021) | 43.0 | -
TIARA + GAIN (T5-base) | 29.1 | 28.2
TIARA + GAIN (T5-3B) | 29.8 | 28.7
TIARA* + GAIN (T5-base) | 33.9 | 31.8
TIARA* + GAIN (T5-3B) | 34.5 | 32.3
Table 8: F1 and Hits@1 scores (%) on WebQSP without fine-tuning on it. All
models are trained on large-scale GrailQA. * denotes using oracle entity
annotations. ♣ denotes our replication results.

| GrailQA | GraphQ | WebQSP | SQB
---|---|---|---|---
Train size | 44,337 | 2,381 | 3,097 | 75,819
Valid size | 6,763 | - | - | 11,141
Test size | 13,231 | 2,395 | 1,638 | 21,483
Length | 62.96 | 54.62 | 35.93 | 42.16
# of entities | 0.903 | 1.028 | 1.112 | 1.000
# of relations | 1.358 | 1.434 | 1.464 | 1.000
Similarity between questions and KB items
Entity | 0.999 | 1.000 | 0.921 | 0.985
Class | 0.547 | 0.457 | - | -
Relation | 0.470 | 0.389 | 0.300 | 0.779
Unseen ratio (%)
Schema | 16.90 | 86.78 | 20.44 | 32.67
Question | 54.06 | 98.25 | 4.03 | 49.18

Table 9: KBQA dataset statistics. Length denotes the average number of characters in the question. # of entities/relations denotes the average number of entities/relations contained in the logical form. Unseen Schema is the ratio of schema items in the dev/test set that do not appear in the training set. Unseen Question is the ratio of questions containing unseen schema.  | Overall | I.I.D. | Compositional | Zero-shot
---|---|---|---|---
Model on GrailQA Valid Set | EM | F1 | EM | F1 | EM | F1 | EM | F1
BERT + Ranking (Gu et al., 2021) | 51.0 | 58.4 | 58.6 | 66.1 | 40.9 | 48.1 | 51.8 | 59.2
TIARA ELF only Shu et al. (2022) | 67.2 | 72.9 | 72.8 | 76.7 | 55.3 | 60.7 | 69.7 | 76.3
RnG-KBQA Ye et al. (2022) | 71.4 | 76.8 | 86.7 | 89.0 | 61.7 | 68.9 | 68.8 | 74.7
DecAF (FiD-3B) Yu et al. (2022) | - | 81.4 | - | 89.7 | - | 80.1 | - | 78.4
TIARA (T5-base) Shu et al. (2022) | 75.3 | 81.9 | 88.4 | 91.2 | 66.4 | 74.8 | 73.3 | 80.7
Pangu (T5-3B) Gu et al. (2022a) | 75.8 | 83.4 | - | - | - | - | - | -
TIARA + GAIN ELF only | 67.4 | 73.6 | 72.7 | 77.5 | 54.7 | 60.5 | 70.3 | 77.3
TIARA + GAIN (T5-base) | 77.1 | 83.5 | 89.0 | 91.9 | 68.6 | 75.5 | 75.4 | 83.2
TIARA + GAIN (T5-3B) | 77.1 | 83.8 | 89.0 | 92.1 | 68.8 | 76.1 | 75.4 | 83.4
GPT-3.5-turbo (5-shot) | 69.7 | 74.8 | 83.0 | 85.5 | 58.7 | 64.6 | 68.6 | 74.4
Table 10: EM and F1 scores (%) on the GrailQA valid set. ELF denotes exemplary
logical form Shu et al. (2022).

#### Paraphrase Adaptation

To evaluate the consistency of the model performance across varying
expressions of the same semantics, we calculate the standard deviation (std)
of EM or F1 for questions within each logical form template and subsequently
calculate the average std for all templates in the dev/test set. A lower std
signifies greater adaptability of the model to different expressions. For
GrailQA, the std of EM and F1 decreases with the application of GAIN or an
increase in model size. However, in the case of more challenging
GraphQuestions, GAIN significantly improves the F1 but also results in a
larger std, as shown in Table 5. It suggests that improving paraphrase
adaptation seems more difficult when the baseline model (TIARA, T5-base, with
only 37.9% F1) still struggles to address most of the dataset. Consequently,
the performance gains observed on the KBQA benchmark may not necessarily
reflect a deeper understanding of linguistic complexities, but they could
simply render the model more sensitive to specific phrases.

#### Cross-Dataset Transfer

To emulate an unseen real-world scenario, we evaluate the performance of pre-
trained models on the human-curated WebQSP dataset without fine-tuning, as
depicted in Table 8. BERT+Ranking Gu et al. (2021) and TIARA+GAIN Shu et al.
(2022) are trained on the large-scale GrailQA dataset. We compare these
results to the state-of-the-art Pangu Gu et al. (2022a), which is fine-tuned
on WebQSP and achieves an F1 score of 79.6%. Although we recognize that GAIN
and large models offer few advantages, the performance of these pre-trained
models without fine-tuning is considerably lower than Pangu’s. We attribute
this to the significant differences between training and test data, as shown
in Table 9. The question length, the difficulty of entity/relation
linking101010Measured by literal similarity:
https://anhaidgroup.github.io/py_stringmatching/v0.3.x/PartialRatio., and the
proportion of unseen schema vary dramatically across KBQA datasets. These
discrepancies arise from the dataset construction process: WebQSP is an
annotation of search logs, whereas the remaining datasets are derived from
graph search and crowdsourcing. To further enhance robustness in cross-dataset
transfer, we believe that better data collection methods are required to
obtain diverse and balanced training data. Additionally, the representation of
the logical form increases the transfer difficulty, as the S-expression used
in GrailQA dataset cannot express all queries in WebQSP.

#### LLM with In-context Learning

We evaluate the performance of GPT-3.5 with in-context learning on the GrailQA
dataset. In the prompt, we provide the task description and the few-shot KB
contexts. As illustrated in Table 10, when provided with contexts from the
TIARA+GAIN retrievers, GPT-3.5 outperforms the ELF contexts but falls short
compared to T5 generators. Among the GPT-3.5 predictions, 79.62% come directly
from the substring of the corresponding prompts, achieving an average F1 score
of 86.19% for this portion. However, the remaining predictions are not part of
their prompts and are entirely new predictions generated by GPT-3.5, with an
average F1 score of merely 30.29%. Although a baseline level is attained,
these results suggest that GPT-3.5 cannot be accurately grounded to the KB
environment when it does not directly utilize the retrievers’ contexts. It
also means the severance of natural language pre-training and KB contexts for
the LLM, and the faithfulness and controllability of grounding LLMs are not
yet guaranteed under the current approach Gu et al. (2022a). To mitigate this
problem, alternative paradigms should be explored, such as tool learning
Schick et al. (2023) and multi-step planning Liu et al. (2023), which enables
more refined access and control over environments.

##  7 Conclusion

We find that despite the voluminous training corpus available for LMs,
incorporating data that adequately captures the intricate complexities of
environments such as large-scale KBs is not yet fulfilled. This underscores
the importance of focusing on the robustness challenges. Notably, LLMs
sometimes blindly follow the provided prompt. It indicates that the existing
methodologies for grounding LLMs are yet to prove their efficacy and
superiority. Future research issues include collecting more balanced
environment-specific corpora and improving the LLM learning paradigm. For the
corpus collection problem, our experiments show some potential for data
augmentation techniques.

## Limitations

For question generation, the verbalization process of the GAIN method relies
heavily on large-scale KBQA annotations. The generated questions may be
similar to the training data, and overly complex logical forms (e.g., with
three and more hops) are difficult to convert into natural language questions.
Besides, synthetic data is less diverse and natural than human annotations,
though it improves generalization performance.

## Ethics Statement

The proposed approach GAIN could be used on any KB for data augmentation. The
Freebase Bollacker et al. (2008) used in this work is a KB that has been
publicly released and manually reviewed. For uncensored KBs, if harmful
information is collected, it could make synthetic data contain harmful
information and make LMs generate harmful answers.

## References

  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on Freebase from question-answer pairs.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1533–1544, Seattle, Washington, USA. Association for Computational Linguistics.
  * Bollacker et al. (2008) Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008.  Freebase: a collaboratively created graph database for structuring human knowledge.  In _Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10-12, 2008_ , pages 1247–1250. ACM.
  * Bordes et al. (2015) Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015.  Large-scale simple question answering with memory networks.  _arXiv preprint arXiv:1506.02075_.
  * Cao et al. (2022a) Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. 2022a.  KQA Pro: A dataset with explicit compositional programs for complex question answering over knowledge base.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 6101–6119. Association for Computational Linguistics.
  * Cao et al. (2022b) Shulin Cao, Jiaxin Shi, Zijun Yao, Xin Lv, Jifan Yu, Lei Hou, Juanzi Li, Zhiyuan Liu, and Jinghui Xiao. 2022b.  Program transfer for answering complex questions over knowledge bases.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 8128–8140. Association for Computational Linguistics.
  * Chen et al. (2021a) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021a.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_.
  * Chen et al. (2021b) Shuang Chen, Qian Liu, Zhiwei Yu, Chin-Yew Lin, Jian-Guang Lou, and Feng Jiang. 2021b.  ReTraCk: A flexible and efficient framework for knowledge base question answering.  In _Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL 2021 - System Demonstrations, Online, August 1-6, 2021_ , pages 325–336. Association for Computational Linguistics.
  * Das et al. (2021) Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021.  Case-based reasoning for natural language queries over knowledge bases.  In _Proceedings of the EMNLP 2021_ , pages 9594–9611. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Dong et al. (2017) Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017.  Learning to paraphrase for question answering.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_ , pages 875–886. Association for Computational Linguistics.
  * Dubey et al. (2019) Mohnish Dubey, Debayan Banerjee, Abdelrahman Abdelkawi, and Jens Lehmann. 2019.  LC-QuAD 2.0: A large dataset for complex question answering over wikidata and dbpedia.  In _The Semantic Web - ISWC 2019 - 18th International Semantic Web Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part II_ , volume 11779 of _Lecture Notes in Computer Science_ , pages 69–78. Springer.
  * Gu et al. (2022a) Yu Gu, Xiang Deng, and Yu Su. 2022a.  Don’t generate, discriminate: A proposal for grounding language models to real-world environments.  _arXiv preprint arXiv:2212.09736_.
  * Gu et al. (2021) Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021.  Beyond I.I.D.: three levels of generalization for question answering on knowledge bases.  In _Proceedings of the Web Conference 2021_ , pages 3477–3488.
  * Gu et al. (2022b) Yu Gu, Vardaan Pahuja, Gong Cheng, and Yu Su. 2022b.  Knowledge base question answering: A semantic parsing perspective.  _arXiv preprint arXiv:2209.04994_.
  * Gu and Su (2022) Yu Gu and Yu Su. 2022.  ArcaneQA: Dynamic program induction and contextualized encoding for knowledge base question answering.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 1718–1731, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Hendrycks et al. (2020) Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, Dawn Xiaodong Song, Jacob Steinhardt, and Justin Gilmer. 2020.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 8320–8329.
  * Hu et al. (2022) Xixin Hu, Xuan Wu, Yiheng Shu, and Yuzhong Qu. 2022.  Logical form generation via multi-task learning for complex question answering over knowledge bases.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 1687–1696, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Keysers et al. (2020) Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020.  Measuring compositional generalization: A comprehensive method on realistic data.  In _ICLR 2020_. OpenReview.net.
  * Lan et al. (2022) Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2022.  Complex knowledge base question answering: A survey.  _IEEE Transactions on Knowledge and Data Engineering_ , pages 1–20.
  * Li et al. (2023) Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su, and Wenhu Chen. 2023.  Few-shot in-context learning for knowledge base question answering.  _arXiv preprint arXiv:2305.01750_.
  * Lin and Och (2004) Chin-Yew Lin and Franz Josef Och. 2004.  Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics.  In _Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)_ , pages 605–612.
  * Liu et al. (2023) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Yuxian Gu, Hangliang Ding, Kai Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Shengqi Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023.  Agentbench: Evaluating llms as agents.  _arXiv preprint 2308.03688_.
  * Menick et al. (2022) Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and Nathan McAleese. 2022.  Teaching language models to support answers with verified quotes.  _arXiv preprint arXiv:2203.11147_.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint 2112.09332_.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc.
  * Perevalov et al. (2022) Aleksandr Perevalov, Dennis Diefenbach, Ricardo Usbeck, and Andreas Both. 2022.  QALD-9-plus: A multilingual dataset for question answering over dbpedia and wikidata translated by native speakers.  In _16th IEEE International Conference on Semantic Computing, ICSC 2022, Laguna Hills, CA, USA, January 26-28, 2022_ , pages 229–234. IEEE.
  * Petrochuk and Zettlemoyer (2018) Michael Petrochuk and Luke Zettlemoyer. 2018.  SimpleQuestions nearly solved: A new upperbound and baseline approach.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 554–558, Brussels, Belgium. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:140:1–140:67.
  * Reddy et al. (2017) Siva Reddy, Oscar Täckström, Slav Petrov, Mark Steedman, and Mirella Lapata. 2017.  Universal semantic parsing.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 89–101, Copenhagen, Denmark. Association for Computational Linguistics.
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009.  The probabilistic relevance framework: BM25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 3(4):333–389.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_.
  * Shu et al. (2022) Yiheng Shu, Zhiwei Yu, Yuhan Li, Börje Karlsson, Tingting Ma, Yuzhong Qu, and Chin-Yew Lin. 2022.  TIARA: Multi-grained retrieval for robust question answering over large knowledge base.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8108–8121, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Su et al. (2016) Yu Su, Huan Sun, Brian M. Sadler, Mudhakar Srivatsa, Izzeddin Gur, Zenghui Yan, and Xifeng Yan. 2016.  On generating characteristic-rich question sets for QA evaluation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 562–572. The Association for Computational Linguistics.
  * Sun et al. (2020) Yawei Sun, Lingling Zhang, Gong Cheng, and Yuzhong Qu. 2020.  SPARQA: skeleton-based semantic parsing for complex questions over knowledge bases.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8952–8959.
  * Talmor and Berant (2018) Alon Talmor and Jonathan Berant. 2018.  The web as a knowledge-base for answering complex questions.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 641–651.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint 2307.09288_.
  * Trivedi et al. (2017) Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. 2017.  LC-QuAD: A corpus for complex question answering over knowledge graphs.  In _The Semantic Web - ISWC 2017 - 16th International Semantic Web Conference, Vienna, Austria, October 21-25, 2017, Proceedings, Part II_ , volume 10588 of _Lecture Notes in Computer Science_ , pages 210–218. Springer.
  * Vrandecic and Krötzsch (2014) Denny Vrandecic and Markus Krötzsch. 2014.  Wikidata: a free collaborative knowledgebase.  _Commun. ACM_ , 57(10):78–85.
  * Wu et al. (2019) Peng Wu, Shujian Huang, Rongxiang Weng, Zaixiang Zheng, Jianbing Zhang, Xiaohui Yan, and Jiajun Chen. 2019.  Learning representation mapping for relation detection in knowledge base question answering.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 6130–6139. Association for Computational Linguistics.
  * Xu et al. (2023) Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu, and Julian McAuley. 2023.  Small models are valuable plug-ins for large language models.  _arXiv preprint arXiv:2305.08848_.
  * Ye et al. (2022) Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2022.  RNG-KBQA: Generation augmented iterative ranking for knowledge base question answering.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6032–6043, Dublin, Ireland. Association for Computational Linguistics.
  * Yih et al. (2016) Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016.  The value of semantic parse labeling for knowledge base question answering.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 2: Short Papers_. The Association for Computer Linguistics.
  * Yu et al. (2022) Donghan Yu, Sheng Zhang, Patrick Ng, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu, William Wang, Zhiguo Wang, and Bing Xiang. 2022.  DecAF: Joint decoding of answers and logical forms for question answering over knowledge bases.  _arXiv preprint arXiv:2210.00063_.

Model of Class Retrieval | All | I.I.D. | Comp. | Zero.
---|---|---|---|---
ReTraCk Chen et al. (2021b) | 94.3 | 98.1 | 97.5 | 91.3
TIARA Shu et al. (2022) | 95.8 | 99.6 | 97.9 | 93.4
TIARA + GAIN | 96.1 | 99.6 | 98.1 | 93.8
Model of Relation Retrieval | All | I.I.D. | Comp. | Zero.
ReTraCk Chen et al. (2021b) | 88.4 | 95.3 | 91.0 | 84.3
TIARA Shu et al. (2022) | 92.0 | 97.9 | 93.7 | 88.7
TIARA + GAIN | 93.0 | 99.2 | 94.1 | 89.8

Table 11: Recall (%) of top-10 retrieved schema items on the GrailQA valid set. comp. and zero. denote compositional and zero-shot generalization, respectively. Note that ReTraCk uses 100 classes and 150 relations for each question, while TIARA uses 10 classes and 10 relations. Model on GraphQuestions | Class | Relation
---|---|---
TIARA♣ Shu et al. (2022) | 81.5 | 67.1
TIARA + GAIN | 83.3 | 74.3
Table 12: Recall (%) of top-10 retrieved schema items on the GraphQuestions
test set (Freebase 2015-08-09 version). TIARA uses 10 classes and 10 relations
for each question. ♣ denotes our replication results.

##  Appendix A Example of LLM Prompt

We present an example of an excerpted prompt, which is shown in Figures 2 and
3. In particular, Figure 2 illustrates the task instruction and teaching
example segment, while Figure 3 displays the input query segment.

##  Appendix B Details of Synthetic Data

The graph search process of GAIN is slightly different for logical forms and
triples.

#### Searching Logical Forms

GAIN employs a graph search approach similar to GraphQuestions Su et al.
(2016) to collect logical forms derived from graph queries. The graph query
construction process consists of four steps: 1) query template construction,
2) aligning template nodes, 3) configuring functions, and 4) verification by
execution. Query templates, obtained through random graph searching, contain
nodes that represent entity/literal types (not aligned to a value), referred
to as template nodes. Each unaligned node in the query template is then
aligned with a topic entity or literal to generate multiple aligned graph
queries. To synthesize counting, comparative, and superlative questions and
enhance the diversity of synthetic data, we add functions like COUNT,
ARGMIN/ARGMAX Gu et al. (2021) to graph queries. Since KBQA research typically
assumes that questions can be answered by the KB, we execute all resulting
graph queries after the above steps and remove those with a null111111Null for
querying entities and zero for counting. result.

#### Searching Triples

A single KB triple can be treated as a QA pair, where the head entity and
relation together form the query, and the tail entity is the answer. The
triple search process consists of two steps: 1) candidate relation selection,
and 2) triple sampling. First, arbitrary relations $\mathcal{R}$ are selected
from the KB, without being restricted to any particular KBQA dataset. Then,
triples are collected from head entities $\mathcal{H}$, where entities in
$\mathcal{H}$ are connected to relations in $\mathcal{R}$.

#### Examples of Synthetic Data

We present some examples of synthetic data in Table 15, where the logical form
contains a variety of functions.

#### Statistics

The statistics for KBQA datasets including our synthetic dataset are shown in
Table 16. To calculate the number of domains in Table 2, 3, we take the domain
of each Freebase class from its first segment, except for classes starting
with “base”, where we take the first two segments, e.g., domain “music” for
the class “music.performance_venue”, and domain “base.plants” for the class
“base.plants.plant”.

Metrics | GrailQA Dev | GraphQ Test | SQB Dev
---|---|---|---
BLEU-4 | 0.347 | 0.178 | 0.369
ROUGE-L | 0.526 | 0.411 | 0.640

Table 13: The performance of the question generator on KBQA datasets. The
generator for logical form is evaluated on the GrailQA valid set and
GraphQuestions test set. The generator for triple is evaluated on the SQB
valid set.

Logical Form (S-expression) | Question
---|---
| 1\. (AND book.journal (JOIN book.periodical.editorial_staff
---
(AND (JOIN book.editorial_tenure.editor m.012z2ncg)
(JOIN book.editorial_tenure.title m.02h6676))))
(GrailQA valid set)
2\. (AND book.journal (JOIN book.periodical.editorial_staff
(AND (JOIN book.editorial_tenure.editor m.05ws_t6)
(JOIN book.editorial_tenure.title m.02wk2cy))))
(GrailQA valid set)
| 1\. john oliver la gorce was the editor on the editor for what journal?
---
2\. with which journal did don slater serve as editor on the editor in chief?
| All four S-expressions are (COUNT (AND book.reviewed_work
---
(JOIN book.reviewed_work.reviews_of_this_work m.0240y2)))
(GraphQuestions training set)
| 1\. how many works did fresh air review?
---
2\. how many works were reviewed by fresh air in total?
3\. what is the total amount of works reviewed by fresh air?
4\. fresh air has reviewed how many different works?

Table 14: Examples of paraphrases in GrailQA and GraphQuestions.

Sampled Logical Form | Synthetic Question
---|---
| (COUNT (AND people.profession (JOIN
---
people.profession.people_with_this_profession m.012d40)))
|entity|m.012d40 jackie chan
how many professions does jackie chan have?
(AND food.beer (le food.beer.original_gravity 1.067^^float)) | which beer has an original gravity less than or equal to 1.067?
| (AND medicine.manufactured_drug_form (AND (lt
---
medicine.manufactured_drug_form.size 10.0^^float)
(JOIN medicine.manufactured_drug_form.fda_otc_part m.0h9yt7z)))
|entity|m.0h9yt7z fda otc monograph part 348
| which manufactured drug form has part
---
fda otc monograph part 348 and has a size smaller than 10.0?
| (ARGMAX (AND measurement_unit.power_unit
---
(JOIN measurement_unit.power_unit.measurement_system m.07y37))
measurement_unit.power_unit.power_in_watts)
|entity|m.07y37 us customary units
what is the largest power unit in the us customary units?
| (AND music.release (AND (JOIN music.release.engineers m.011mbx12)
---
(JOIN music.release.label m.0g12fn3)))
|entity|m.011mbx12 raynard glass |m.0g12fn3 hostyle gospel ministries
| what musical release is engineered by raynard glass and
---
labelled hostyle gospel ministries?
Sampled Triple | Synthetic Question
| D.W. Robertson, Jr. (m.09ggymq),
---
people.person.place_of_birth,
Washington, D.C. (m.0rh6k)
where was D. W. Robertson, Jr. born
| Alfred Chao (m.046cmd8),
---
computer.operating_system_developer.operating_systems_developed,
pSOS (m.0lscq)
what operating system did Alfred Chao develop?

Table 15: Examples of synthetic data. The logical form is S-expression Gu et al. (2021). The entity label is appended to the logical form. Datasets | #question | #class | #relation | #entity
---|---|---|---|---
GrailQA Gu et al. (2021) | 64,331 | 1,534 | 3,720 | 32,585
GraphQuestions Su et al. (2016) | 5,166 | 506 | 596 | 376
WebQSP Yih et al. (2016) | 4,737 | 408 | 661 | 2,593
GAIN-synthetic | 127,329 | 5,078 | 12,942 | 46,645
Table 16: Statistics of KBQA datasets and the GAIN-synthetic dataset.

##  Appendix C Experimental Details

###  C.1 Question Generation

Training a question generation (QG) model is the second step of GAIN. Because
logical forms in the synthetic dataset do not have corresponding human-labeled
natural language questions, evaluating QG on the synthetic dataset is
difficult. Existing KBQA datasets contain questions written by crowdsourced
workers for logical forms or triples, so we evaluate the question generator by
generating questions on these datasets and calculate the BLEU-4 Papineni et
al. (2002) and ROUGE-L Lin and Och (2004) scores121212Calculated by Hugging
Face Evaluate. (Table 13).

###  C.2 Retrieval Performance

The performance of schema retrieval on the GrailQA valid set and
GraphQuestions test set is shown in Table 11 and 12, respectively. GAIN
significantly improves the performance of schema retrieval on both GrailQA and
GraphQuestions.

Function | None | Count | Comparative | Superlative
---|---|---|---|---
ArcaneQA Gu and Su (2022) | 70.8/77.8 | 62.5/68.2 | 54.5/75.7 | 70.5/75.6
RnG-KBQA Ye et al. (2022) | 77.5/81.8 | 73.0/77.5 | 55.1/76.0 | 13.8/22.3
TIARA (T5-base) Shu et al. (2022) | 77.8/83.1 | 76.4/81.8 | 57.4/81.4 | 58.7/69.0
TIARA + GAIN ELF only | 76.8/81.7 | 73.9/80.0 | 0.0/25.3 | 0.0/8.3
TIARA + GAIN (T5-base) |  78.6/84.6 |  77.7/83.0 | 61.7/82.3 | 69.9/73.2
TIARA + GAIN (T5-3B) | 78.5/84.8 | 77.3/82.5 |  63.0/84.5 |  70.7/74.1
GPT-3.5-turbo (5-shot) | 74.1/78.0 | 66.8/70.5 | 38.3/60.5 | 43.9/52.3
# of relations | 1 | 2 | 3 | 4
RnG-KBQA Ye et al. (2022) | 75.7/79.3 | 65.3/74.7 | 28.6/44.5 |  100.0/100.0
TIARA (T5-base) Shu et al. (2022) | 81.2/85.6 | 64.7/75.8 | 29.3/48.5 | 50.0/83.3
TIARA + GAIN ELF only | 74.0/77.8 | 56.6/67.9 | 9.9/31.0 | 0.0/33.3
TIARA + GAIN (T5-base) |  82.4/87.2 | 67.0/78.0 |  38.9/49.8 | 50.0/83.3
TIARA + GAIN (T5-3B) | 82.0/87.2 |  68.8/79.0 | 37.5/51.3 | 50.0/83.3
GPT-3.5-turbo (5-shot) | 75.0/78.6 | 61.9/69.6 | 19.8/36.6 | 50.0/50.0
# of entities | 0 | 1 | 2 |
RnG-KBQA Ye et al. (2022) | 58.5/63.6 | 75.4/79.9 |  55.6/73.5 |
TIARA (T5-base) Shu et al. (2022) | 77.5/83.1 | 76.6/82.6 | 49.9/68.0 |
TIARA + GAIN ELF only | 42.8/47.0 | 74.2/79.9 | 47.4/67.6 |
TIARA + GAIN (T5-base) |  82.2/86.5 | 77.6/83.7 | 53.4/71.4 |
TIARA + GAIN (T5-3B) | 82.0/86.6 |  77.6/84.0 | 55.6/73.0 |
GPT-3.5-turbo (5-shot) | 66.8/72.8 | 71.9/76.1 | 48.8/62.1 |
Table 17: EM and F1 scores (%) for different types of logical forms on the
GrailQA valid set. None denotes no function. # of relations/entities denotes
the number of relations/entities in the S-expression. ELF denotes exemplary
logical form Shu et al. (2022).

###  C.3 Performance on Various Logical Forms

To show how TIARA+GAIN performs on different types of logical forms, we
compare it with previous KBQA models on the GrailQA valid set, as shown in
Table 17. TIARA+GAIN improves performance in nearly all these scenarios
compared to TIARA.

###  C.4 Error Analysis

To analyze the QA errors of TIARA+GAIN (T5-3B), we randomly sample 50
questions where predicted logical forms are not the same as the ground truth
in the GrailQA valid set. We follow Shu et al. (2022) in their classification
of errors. Entity linking errors (missing or redundant entities), syntactic
errors (generating the wrong logical form structure), semantic errors
(generating the wrong schema item when correct contexts are provided), false
negatives (flawed annotations), and miscellaneous (e.g., ambiguity in the
question) account for 48%, 26%, 16%, 4%, and 6%, respectively. For entity
linking errors, 62.5% of them are from the zero-shot level. For syntactic
errors, the number of errors from zero-shot and compositional levels is almost
the same. It means that entity linking for zero-shot domains and logical form
generation for complex structures remain significant challenges.

Given a question and Freebase contexts, write a logical form that answers the
question.

Question: dark sun: wake of the ravager was designed by what video game
designer?

Candidate entities:

(A) [Dark Sun: Wake of the Ravager]

Exemplary Logical Forms:

(A) (AND cvg.cvg_designer (JOIN cvg.cvg_designer.games_designed [Dark Sun:
Wake of the Ravager]))

(B) (AND cvg.cvg_designer (JOIN (R cvg.computer_videogame.designers) [Dark
Sun: Wake of the Ravager]))
(C) (AND cvg.computer_videogame (JOIN (R cvg.cvg_designer.games_designed)
(JOIN cvg.cvg_designer.games_designed [Dark Sun: Wake of the Ravager])))
(D) (AND cvg.computer_videogame (JOIN cvg.computer_videogame.designers (JOIN
cvg.cvg_designer.games_designed [Dark Sun: Wake of the Ravager])))
(E) (AND base.wikipedia_infobox.video_game (JOIN
base.wikipedia_infobox.video_game.developer (JOIN
cvg.cvg_designer.games_designed [Dark Sun: Wake of the Ravager])))
Candidate classes:
(A) cvg.cvg_designer
(B) cvg.game_performance
(C) cvg.musical_game
(D) cvg.game_character
(E) cvg.computer_game_engine_developer
(F) cvg.computer_videogame
(G) cvg.computer_game_performance_type
(H) cvg.game_version
(I) cvg.computer_game_subject
(J) cvg.computer_game_evaluation
Candidate relations:
(A) cvg.cvg_designer.games_designed
(B) cvg.computer_videogame.designers
(C) cvg.computer_videogame.prequel
(D) cvg.computer_videogame.sequel
(E) cvg.computer_videogame.mods
(F) cvg.computer_videogame.expansions
(G) cvg.computer_videogame.developer
(H) cvg.computer_videogame.characters
(I) cvg.game_version.game
(J) cvg.computer_game_mod.game_modded
Prediction: (AND cvg.cvg_designer (JOIN cvg.cvg_designer.games_designed [Dark
Sun: Wake of the Ravager]))

Figure 2: Example of LLM prompt (part 1): The task instruction and $k$
teaching examples (only one is shown because of the length) with questions and
their contexts.

Question: worldofwarcraft is the creation of which video game designer?

Candidate entities:

(A) [worldofwarcraft]

Exemplary Logical Forms:

(A) (AND cvg.cvg_designer (JOIN cvg.cvg_designer.games_designed
[worldofwarcraft]))

(B) (AND cvg.cvg_designer (JOIN (R cvg.computer_videogame.designers)
[worldofwarcraft]))
(C) (AND cvg.cvg_designer (JOIN cvg.cvg_designer.games_designed (JOIN
cvg.computer_game_expansion.expansion_for [worldofwarcraft])))
(D) (AND cvg.cvg_designer (JOIN (R cvg.computer_videogame.designers) (JOIN
cvg.computer_game_expansion.expansion_for [worldofwarcraft])))
(E) (AND cvg.cvg_designer (JOIN (R cvg.computer_videogame.designers) (JOIN (R
cvg.computer_videogame.expansions) [worldofwarcraft])))
Candidate classes:
(A) games.game_designer
(B) cvg.cvg_designer
(C) amusement_parks.ride_designer
(D) cvg.cvg_developer
(E) cvg.computer_videogame
(F) cvg.computer_game_engine_developer
(G) cvg.computer_game_engine
(H) cvg.computer_game_mod
(I) cvg.game_performance
(J) cvg.musical_game
Candidate relations:
(A) cvg.computer_videogame.designers
(B) cvg.cvg_designer.games_designed
(C) games.game_designer.games_designed
(D) games.game.designer
(E) cvg.computer_videogame.developer
(F) cvg.cvg_developer.games_developed
(G) cvg.computer_game_engine.developer
(H) cvg.computer_videogame.expansions
(I) cvg.computer_videogame.publisher
(J) cvg.game_version.developer
Prediction:

Figure 3: Example of LLM prompt (part 2): After $k$ teaching examples, the
input question and its contexts is given. The golden prediction is  (AND
cvg.cvg_designer (JOIN cvg.cvg_designer.games_designed [worldofwarcraft])).