11institutetext: Universität Paderborn, Warburger Str. 100, 33098 Paderborn,
Germany,
11email<EMAIL_ADDRESS>
# MST5 — A Transformers-Based Approach to Multilingual Question Answering over
Knowledge Graphs

Nikit Srivastava  Mengshi Ma  Daniel Vollmers  Hamada Zahera  Diego Moussallem
Axel-Cyrille Ngonga Ngomo

###### Abstract

Knowledge Graph Question Answering (KGQA) simplifies querying vast amounts of
knowledge stored in a graph-based model using natural language. However, the
research has largely concentrated on English, putting non-English speakers at
a disadvantage. Meanwhile, existing multilingual KGQA systems face challenges
in achieving performance comparable to English systems, highlighting the
difficulty of generating SPARQL queries from diverse languages. In this
research, we propose a simplified approach to enhance multilingual KGQA
systems by incorporating linguistic context and entity information directly
into the processing pipeline of a language model. Unlike existing methods that
rely on separate encoders for integrating auxiliary information, our strategy
leverages a single, pretrained multilingual transformer-based language model
to manage both the primary input and the auxiliary data. Our methodology
significantly improves the language model’s ability to accurately convert a
natural language query into a relevant SPARQL query. It demonstrates promising
results on the most recent QALD datasets, namely QALD-9-Plus and QALD-10.
Furthermore, we introduce and evaluate our approach on Chinese and Japanese,
thereby expanding the language diversity of the existing datasets.

###### Keywords:

multilingual knowledge graphs question answering multilingual language models
multilingual question answering sparql generation kgqa kbqa

##  1 Introduction

The aim of research in Knowledge Graph Question Answering (KGQA) is to
establish an interactive methodology enabling users to access extensive
knowledge stored within a graph-based model via natural language queries [1].
Recent research efforts have witnessed a notable upsurge in addressing KGQA
concerns [2]. However, it is noteworthy that a substantial proportion of these
systems is confined to the English language domain [3, 4, 5, 6, 7, 8].
Furthermore, among the currently available multilingual systems [9, 10, 11],
only a limited subset of widely spoken or written languages are supported [12,
13]. Moreover, majority of these systems do not achieve performance levels
comparable to those attained in English when addressing questions in other
languages. This difference highlights the inherent difficulty KGQA systems
face in processing the multilingual natural language queries by trying to
recognize the hidden, repeated patterns that are common across different
languages. This poses a challenge for the vast majority of web users whose
native language is not English.

Knowledge Graphs (KGs) have been conceptualized as language-agnostic solution
for the organization and storage of information [14, 15]. As a result, the
approach to KGQA paves the way for retrieving language-agnostic answers,
contingent upon its ability to comprehend natural language questions in a
multilingual context. Recent advancements in the field of language modeling
have made numerous state-of-the-art language models freely accessible [16, 17,
18, 19]. These language models can be effectively employed within a sequence-
to-sequence task setting to tackle KGQA. In this setting, the input sequence
corresponds to the natural language question, while the output entails a
relevant SPARQL 111https://www.w3.org/TR/rdf-sparql-query/ query required to
extract the answer from an underlying KG. This method of carrying out KGQA is
also known as Semantic Parsing [20], it generates an intermediate
representation (i.e. SPARQL) that is interpretable by humans, enabling them to
understand how the model formulates specific answers within a multilingual
context.

To overcome the limited availability of multilingual KGQA and the shortcomings
of existing systems, we adopt a strategy similar to Rony et al. [8] that
entails the inclusion of linguistic context as an auxiliary input to a
language model. Unlike the previous methods, however, we employ a single
encoder-decoder transformer model rather than creating separate encodings for
the auxiliary input. We do this in order to leverage the attention mechanism
embedded within the language model to facilitate the creation of an implicit
understanding and representation of the provided linguistic context.
Furthermore, we apply entity disambiguation tools to extract pertinent entity
information, which is then also added to the auxiliary input. Our method
establishes a seamless end-to-end system that can respond to multilingual
queries using only the text of the input question.

Our findings demonstrate that incorporating linguistic context and entity
information significantly enhances the KGQA performance. Our approach yields
promising results on our benchmarking datasets. Moreover, to substantiate the
efficacy of our approach, we introduce and evaluate its performance in
previously unrepresented languages within the QALD datasets. Specifically, we
added Japanese to the QALD-10 dataset and both Chinese and Japanese to the
QALD-9-Plus dataset. The inclusion of these non-European languages introduces
a novel dimension to the datasets, given their distinctive structural
characteristics when compared to the existing language set. Additionally, we
make our source code publicly accessible for the benefit of the research
community 222https://github.com/dice-group/MST5

. In essence, our approach addresses the following research question: How can
the simplified integration of linguistic context and entity information into
language models enhance their performance on multilingual Knowledge Graph
Question Answering tasks?

##  2 Related Work

The KGQA task is commonly solved by Semantic Parsing approaches, that create
logical queries for given natural language questions [20]. Since SPARQL is
used as a standard query language for the RDF-based Knowledge Graphs
333https://www.w3.org/RDF/, we investigate the systems that adopt this
particular approach to KGQA. In this section, we cover the relevant previous
works that have developed systems based on Language Model (LM)for the SPARQL
query generation and the KGQA systems that are multilingual.

###  2.1 LM-based SPARQL query generation

As one of the early works, Soru et al. [5] introduced Neural SPARQL Machines
(NsPM)where SPARQL is regarded as an analogous language, it utilized Neural
Machine Translation (NMT)to transform natural language questions into SPARQL
queries, while avoiding language heuristics, statistics, or manual models. The
architecture consists of three components: a generator that creates SPARQL
queries from query templates for training the model, a learner based on
Bidirectional Recurrent Neural Networks (Bi-RNNs) [21] to learn to translate
input questions to encoded SPARQL, and an interpreter for reconstructing
SPARQL queries from their encoded representation and retrieving results from a
knowledge graph. Their modular design allowed integration of various machine
translation models and generating SPARQL queries for diverse knowledge graphs.

In the recent years, Borroto et al. [6] produced a similar AI system for KGQA,
where the architecture uses a NMT module based on Bi-RNNs [21]. They trained
it in parallel to a Named Entity Recognition (NER)module, implemented using a
BiLSTM-CRF network [22]. The NMT module translates the input natural language
question into a SPARQL query template, whereas the NER module extracts the
entities from the question. The two modules’ outputs are combined to form the
resulting SPARQL query. Departing from the template oriented methodology, Rony
et al. [8] proposed a novel approach to tackle the challenges associated with
generating SPARQL queries from natural language questions. The authors
introduced a new embedding technique that encodes questions, linguistic
features, and optional knowledge to allow the system to comprehend complex
question patterns and graph data for SPARQL query generation. They augmented
the input embedding with the extracted embeddings and then fed it to a
decoder-only (GPT-2 [23]) language model to generate SPARQL query. However, it
is important to highlight that the investigation conducted in the study did
not assess the effectiveness of the resulting SPARQL queries in retrieving
answers from a KG. This limitation stems from the fact that the generated
SPARQL queries are often syntactically incorrect. Furthermore, certain other
limitations are observed within this approach, particularly regarding the
selection of the language model architecture and the utilization of separate
embedding layers for encoding the linguistic context.

###  2.2 Multilingual KGQA

When it comes to the multilingual KGQA systems, Burtsev et al. [9] introduced
a conversational system called DeepPavlov that operates by utilizing a suite
of extensive language-dependent deep learning models. These models are used to
execute a spectrum of tasks encompassing query type prediction, entity
recognition, relation identification, and path ranking that can be applied to
address KGQA alongside many other uses-cases. Developing it further, Zharikova
et al. [24] introduced DeepPavlov-2023, which is an improved version of its
predecessor.

Pellissier Tanon et al. [10] presented a KGQA methodology named Platypus,
which adopts a dual-phase strategy. Initially, it leverages a semantic parser
to convert questions posed in natural language into SPARQL queries.
Subsequently, these SPARQL queries are executed on the Wikidata knowledge base
to procure answers. The semantic parser employed is of a hybrid nature,
integrating grammatical (rule-based) and template-based approaches.
Grammatical approaches are utilized to extract the syntactic structure of the
question and to identify any mentioned entities and relations. Following this,
template-based methods are employed to populate predefined templates that are
tailored to various types of questions. Building towards an end-to-end
multilingual KGQA solution, Diefenbach et al. [11] introduced an approach
named QAnswer that generates a SPARQL in four steps. First, it fetches the
relevant resources from the underlying KGs. Then, it generates a list of
possible query templates. Afterwards, the query candidates are created and
ranked using the output from the previous two steps. Finally, a confidence
score is computed for each of the ranked queries. QAnswer claimed to be state-
of-the-art at the time, and remains a top contender in the QALD challenges.
However, one of the biggest downside of this system is not being open-source.

Looking at Machine Translation (MT)as a viable alternative, Perevalov et al.
[12] experimented with various MT systems to extend the supported languages of
existing multilingual KGQA systems. Their findings indicate that, in most
cases, translating questions to English resulted in the highest performance.
Furthermore, they noted a small to moderate positive correlation between the
quality of machine translation and the question-answering score. In a similar
fashion, Srivastava et al. [13] adopted an entity-aware MT approach for the
KGQA use-case. Where they highlighted the need for effective translation for
the entity labels between languages. By using an entity-aware MT pipeline and
translating each question to English, they created a multilingual KGQA
approach that performs better than the traditional MT-based methods previously
mentioned.

##  3 MST5 Approach

Our approach, referred to as MST5, emphasizes on incorporating and utilizing
additional knowledge, such as entity link tags and linguistic context, via a
transformer-based model (i.e mT5 [17]). This is done to construct a semantic
parsing-based KGQA system. In the following subsections, we first formalize
the problem definition in mathematical terms, and then describe the
architecture of our approach to address this problem.

###  3.1 Problem Definition

Given a natural language query $Q$ and auxilliary information $A$ which
consists of linguistic context and entity information, we train the
transformer-based model $\mathcal{M}$ parameterized by $\theta$, aiming to
generate a syntactically and semantically correct corresponding SPARQL query
$\hat{S}$ as:

$\hat{S}=\arg\max_{S^{\prime}}P(S^{\prime}\mid Q,A;\theta)$

Here $P$ is the conditional probability distribution function and $S^{\prime}$
represents a candidate SPARQL query generated by the model.

The model is trained in a manner akin to mT5 [17], focusing on minimizing the
negative log likelihood ($\mathcal{L}$) of the correct SPARQL query given the
inputs:

$\mathcal{L}(\theta)=-\log P(S\mid Q,A;\theta)$

Here $S$ refers to the reference (correct) SPARQL query.

###  3.2 MST5 Architecture

To obtain the auxiliary information from a given input, we create individual
modules to perform entity recognition and disambiguation along with linguistic
feature extraction. The extracted information is then used to compose the
final input sequence to our language model by concatenating it with the input
question. Figure 1 provides a high-level overview of our approach. Following
are the detailed description of these steps:

Figure 1: An overview of the MST5 approach (from left to right). First,
linguistic context and entity information is extracted from the input natural
language question. Then, the extracted information is concatenated with the
input before being passed on to the language model. The language model
generates the resulting SPARQL query.

####  3.2.1 Named Entity Recognition and Linking

In this step of our approach we make use of NER and Named Entity
Disambiguation (NED)approaches. To fulfill this requirement, we rely on the
Named Entity Aware Machine Translation (NEAMT)444https://github.com/dice-
group/LFQA/tree/main/naive-eamt tool introduced by Srivastava et al. [13].
While the primary purpose of this tool is to perform entity-aware machine
translation, its API allows us to skip the translation part entirely and only
perform entity recognition and linking tasks. Given its multilingual
capabilities, NEAMT meets our requirements.

####  3.2.2 Linguistic feature extraction

For enhancing the understanding of the language model, we explicitly include
linguistic features in a manner akin to the approach described by Rony et al.
[8]. Unlike their approach, we directly append the features to the input
sequence. To this end, we extract Part-of-Speech (POS) tags and generate the
dependency tree for the input question. For extracting these features, we rely
on the spaCy555https://spacy.io/ tool. For each distinct language, we use the
respective spaCy model666https://spacy.io/usage/models.

####  3.2.3 Preprocessing the target SPARQL-queries

To generate SPARQL queries through a LM, existing works similar to Banerjee et
al. [25] introduce a separate vocabulary to make it easier for a LM to predict
the resulting SPARQL. However, we recognize that this incorporation introduces
additional complexity. Thus, we employ minimal preprocessing during training
phase to avoid the need for an expanded vocabulary.

SPARQL queries often include prefixes to shorten resource identifiers (URIs)
such as https://www.wikidata.org/entity/Q5 to wd:Q5. To simplify the target
sequences for the mT5 model, we remove the prefix section from the query. We
use the prefix form, e.g., wd:Q5 instead of the full URIs in the query. During
inference, we add all common prefixes back to the predicted query to restore
its syntactic correctness. As an additional step, we replace special
characters such as the question mark (?), which signals a variable in a SPARQL
query, and curly braces ({}), with standardized string placeholders. This
modification is due to our observation that the mT5 model struggles with
accurately predicting these symbols. During the inference phase, we revert
these placeholders to their original symbols to facilitate the retrieval of
answers from a SPARQL endpoint.

####  3.2.4 Creating Input Data for the MST5 Model

As input to our model is a token sequence, we concatenate the textual question
with the linguistic features and entity links with the help of custom
separator tokens. As a result, each input sequence incorporates the features
listed below. Figure 2 offers an overview of these features:

Figure 2: An example of linguistic context including dependency parsing (red)
and POS-tags (green) alongisde the disambiguated entity for the text: Who are
the grandchildren of Bruce Lee?

  1. 1.

Question, e.g., Who are the grandchildren of Bruce Lee?

  2. 2.

POS tags, e.g., PRON AUX DET NOUN ADP PROPN PROPN PUNCT

  3. 3.

Dependency tree tags, e.g., attr ROOT det nsubj prep compound pobj punct

  4. 4.

Depths in the dependency tree, e.g., 2 1 3 2 3 5 4 2

  5. 5.

Entity links, e.g., Q16397

Padding tokens are appended to each input feature, ensuring that the tokens
for every input feature begin at a consistent token index within the tokenized
input sequence. This step enables the transformers-based LM to attend and
contextualize the relation between the original query and the auxiliary
features properly. Finally, we apply a tokenizer to the concatenated sequence
to produce valid inputs for MST5 model. We use the SentencePiece [26]
tokenizer for mT5 [17] model as described in the model documentation
777https://huggingface.co/docs/transformers/model_doc/mt5.

##  4 Experimental Setup

This section provides a comprehensive description of the training and
benchmarking datasets used in our experiments, along with an in-depth
explanation of our model training methodologies. We have organized our model
training into two main categories: Ablation Study and Evaluation of Optimal
Models. This structure helps with systematic investigation of the effects of
various model configurations and enables us to concentrate on the best-
performing models for additional analysis. Our current experimental setup is
aimed at addressing the following research questions:

  1. 1.

Whether the simplified auxilliary input of linguistic context and entity
information has any effect on the KGQA performance?

  2. 2.

How the top-performing ablations compare to other systems?

  3. 3.

How the the optimal ablations perform in a multilingual setting?

###  4.1 Training and Evaluation Datasets

To train the MST5 model, we utilize the LC-QuAD 2.0, QALD-9-Plus, and QALD-10
datasets. Below, we outline these datasets and detail any modifications
applied:

LC-QuAD 2.0: Introduced by Dubey et al. [27], is the largest English-only KGQA
dataset. It consists of over 30,000 natural language questions paired with
their corresponding Wikidata SPARQL queries. We choose this dataset primarily
to train the model on contextualizing SPARQL queries alongside natural
language queries before proceeding to train on a smaller, multilingual
dataset.

QALD-9-Plus: Introduced by Perevalov et al. [28], QALD-9-Plus dataset includes
the questions in English, German, Russian, French, Spanish, Armenian,
Belarusian, Lithuanian, Bashkir and Ukrainian. The training set for Wikidata
consists of 371 questions, while the test set has 136 questions. We update the
QALD-9-Plus dataset by adding Chinese and Japanese translations of each
question. These translations were done by a native speaker of these languages.
Since the test set contains the answers from an older (unavailable) version of
Wikidata, we update the QALD-9-Plus dataset using endpoint provided for
QALD-10888https://github.com/KGQA/QALD-10#endpoint as both of these datasets
were created concurrently. Subsequently, we filter out all the questions with
empty answer-set. The refined test dataset now comprises 102 questions. We
refer to this version as QALD-9-Plus (updated) test dataset in the upcoming
sections.

QALD-10: Introduced by Usbeck et al. [2], it comprises of questions in
English, German, Chinese and Russian. It uses QALD-9-Plus as its training
data, and provides a test dataset of 394 questions. We update the QALD-10
dataset by adding Japanese translation for each question. These translations
were done by a native speaker of the respective language.

To benchmark our models, our approach requires datasets that are multilingual
and provide a reference SPARQL query alongside the retrieved answer-set for
the respective natural language question. These criteria are essential to show
that our models can generate SPARQL queries that are not only similar to the
reference but also possess the capability to fetch the accurate answer. To
fulfil these requirements, we make use of QALD-9-Plus (updated) test and
QALD-10 dataset for evaluation. A key distinction among these QALD datasets
lies in the evaluation metric employed for comparison. The QALD-9-Plus dataset
utilizes Macro F1 for assessing the performance across various KGQA systems,
whereas the QALD10 dataset suggests Macro F1 QALD999https://github.com/dice-
group/gerbil/issues/320 as per community’s
request101010https://github.com/dice-group/gerbil/issues/211. To conduct the
evaluation and obtain performance metrics on these datasets, we make use of
the GERBIL-QA [29] tool.

Name | Ling. Ctxt. | Ent. Info. | LC-QuAD 2.0 | QALD-9-Plus
---|---|---|---|---
- | No | No | No | No
MST5${}_{\text{Q9}}$ | No | No | No | Yes
MST5${}_{\text{L2}}$ | No | No | Yes | No
MST5${}_{\text{L2+Q9}}$ | No | No | Yes | Yes
- | No | Yes | No | No
MST5${}_{\text{Q9+ENT}}$ | No | Yes | No | Yes
MST5${}_{\text{L2+ENT}}$ | No | Yes | Yes | No
MST5${}_{\text{L2+Q9+ENT}}$ | No | Yes | Yes | Yes
- | Yes | No | No | No
MST5${}_{\text{Q9+LING}}$ | Yes | No | No | Yes
MST5${}_{\text{L2+LING}}$ | Yes | No | Yes | No
MST5${}_{\text{L2+Q9+LING}}$ | Yes | No | Yes | Yes
- | Yes | Yes | No | No
MST5${}_{\text{Q9+LING+ENT}}$ | Yes | Yes | No | Yes
MST5${}_{\text{L2+LING+ENT}}$ | Yes | Yes | Yes | No
MST5${}_{\text{L2+Q9+LING+ENT}}$ | Yes | Yes | Yes | Yes
Table 1: All covered MST5 ablations based on the auxiliary features
(linguistic context and entity information) and fine-tuning dataset used
during the model training.

###  4.2 Model Training Details

We selected the pretrained mT5111111https://huggingface.co/google/mt5-xl as
the foundational model for our work. Its pretraining involved the
mC4121212https://www.tensorflow.org/datasets/catalog/c4#c4multilingual corpus,
encompassing 101 diverse languages.

During the fine-tuning phase, we set the maximum number of epochs to 300 while
setting up an early-stop regularization method to prevent overfitting. Also,
we use the DeepSpeed131313https://www.deepspeed.ai/ tool introduced by
Rajbhandari et al. [30] to optimize the computing resource usage. As for the
hardware, we make use of the Nvidia-A100141414https://www.nvidia.com/en-
us/data-center/a100/ GPU for training our models.

###  4.3 Ablation Study

To assess the impact of the proposed input augmentations on the end-to-end
KGQA performance of our approach, we conduct an ablation study. Our study
involves training and testing every potential variant that includes the
following modifications:

  * •

Fine-tuning on LC-QuAD 2.0 dataset

  * •

Fine-tuning on QALD-9-Plus dataset

  * •

Adding linguistic context to the input

  * •

Incorporating entity tags into the input

Table 1 lists all the covered ablations. We evaluate these ablations on the
English questions from QALD-9-Plus (updated) test dataset to compare their
performance. We exclude the multilingual questions due to resource
limitations, as for each additional language all the ablations would need to
be evaluated. We also exclude QALD-10 dataset from the ablation study because
its training data is a combination of the QALD-9-Plus training and testing
data.

###  4.4 Optimal Models Evaluation

From the ablation study, we select the model(s) demonstrating the best
performance and subject them to further evaluation, during this evaluation we
cover all the supported languages.

For QALD-9-Plus (updated) test dataset, we compare the performance of our
optimal models with only one other KGQA system: DeepPavlov-2023 [24]. We were
not able to include any other relevant system [11, 10, 9] as we couldn’t
extract the generated SPARQL queries for updated QALD-9-Plus
dataset151515Either unreachable public APIs, technical issues in the local
deployment or no SPARQL query generation.. For QALD-10, we refine our optimal
model(s) by additional training on a merged dataset of QALD-9-Plus train and
test data. The further fine-tuning allowed us to perform a direct comparison
with the results of the other existing models in a fair manner. We made use of
the QA-System-Wrapper161616https://github.com/WSE-research/qa-systems-wrapper
tool to query the DeepPavlov-2023 system.

Ablation name | F1 | Precision | Recall | F1 QALD
---|---|---|---|---
MST5${}_{\text{L2+ENT}}$ | 0.1886 | 0.2001 | 0.1982 | 0.322
MST5${}_{\text{L2+LING}}$ | 0.2291 | 0.2355 | 0.2353 | 0.367
MST5${}_{\text{L2+LING+ENT}}$ | 0.2622 | 0.2747 | 0.2698 | 0.4088
MST5${}_{\text{L2}}$ | 0.1353 | 0.134 | 0.1373 | 0.2359
MST5${}_{\text{Q9+ENT}}$ | 0.3173 | 0.3318 | 0.3164 | 0.4554
MST5${}_{\text{Q9+LING}}$ | 0.206 | 0.2188 | 0.2043 | 0.2759
MST5${}_{\text{Q9+LING+ENT}}$ | 0.3203 | 0.3424 | 0.3216 | 0.4624
MST5${}_{\text{Q9}}$ | 0.0098 | 0.0098 | 0.0098 | 0.0194
MST5${}_{\text{L2+Q9+ENT}}$ | 0.4187 | 0.4366 | 0.4242 | 0.572
MST5${}_{\text{L2+Q9+LING}}$ | 0.3407 | 0.3571 | 0.3493 | 0.4745
MST5${}_{\text{L2+Q9+LING+ENT}}$ | 0.4015 | 0.4192 | 0.4046 | 0.5563
MST5${}_{\text{L2+Q9}}$ | 0.2628 | 0.2792 | 0.2631 | 0.3663
Table 2: Macro performance metrics of MST5 ablations on the QALD-9-Plus
(updated) test dataset (English): divided into three groups based on fine-
tuning data, separated by horizontal lines

##  5 Results and Discussion

Our discussion of the results is structured into three sections: Initially, in
subsection 5.1, we focus on the performance metrics for English. Then, in
subsection 5.2, we examine the performance of our best ablations across
various languages. Finally, in subsection 5.3, we discuss the challenges
encountered during our evaluation with external systems and the measures we
implemented to prevent our system from contributing to the same problem.

###  5.1 English-based Performance

As a first step, we perform the ablation study to find out whether the
simplified auxilliary input of linguistic context and entity information has
any effect on the KGQA performance. We evaluate all our model ablations on the
QALD-9-Plus (updated) English dataset. Table 2 provides a direct comparison of
the model ablations. Based on the results (Macro F1), we observe the
following:

  * •

Linguistic context and entities always improve the performance of the system

  * •

In most cases, combination of linguistic context and entities leads to the
best performing model

  * •

Between the models fine-tuned on both datasets, the entity-only variant has
slightly better performance

Now, to investigate how the top-performing ablations compare to other systems,
we pick the two best performing models MST5${}_{\text{L2+Q9+ENT}}$,
MST5${}_{\text{L2+Q9+LING+ENT}}$ and simplify their names to
MST5${}_{\text{ENT}}$, MST5${}_{\text{LING+ENT}}$ respectively. Table 3
presents a comparative analysis of various models tested on the QALD-10
dataset. In this comparison, we find that:

  * •

MST5 significantly outperforms the competing systems

  * •

Model variant incorporating both linguistic context and entity information
emerges as the top performer

Approach | F1 | Precision | Recall | F1 QALD
---|---|---|---|---
Borroto et al. | 0.4538 | 0.4538 | 0.4574 | 0.5947
Diefenbach et al. | 0.5070 | 0.5068 | 0.5238 | 0.5776
Shivashankar et al. | 0.3215 | 0.3206 | 0.3312 | 0.4909
Baramiia et al. | 0.4277 | 0.4289 | 0.4272 | 0.4281
DeepPavlov-2023 [24] | 0.3241 | 0.3279 | 0.3369 | 0.4518
MST5${}_{\text{ENT}}$ | 0.4784 | 0.4777 | 0.4848 | 0.6330
MST5${}_{\text{LING+ENT}}$ | 0.5271 | 0.5271 | 0.5317 | 0.6727
Table 3: QALD-10 macro performance comparison (English).

###  5.2 Multilingual Performance

We proceed by examining how the the optimal ablations perform in a
multilingual setting. Table 4 provides direct comparison of our models
MST5${}_{\text{ENT}}$ and MST5${}_{\text{LING+ENT}}$ with DeepPavlov-2023 [24]
on all supported languages171717The language codes used in the datasets and
our experiments are as per the ISO 639-1 standard
https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes or QALD-9-Plus
(updated). The results show the following:

  * •

MST5 variants perform better than DeepPavlov-2023 for both languages supported
by it (English, Russian)

  * •

When comparing the MST5 variants to each other:

    * –

Performance in German and French is quite comparable to English

    * –

Performance in Spanish, Russian, Lithuanian, Ukrainian, Belarusian, and
Chinese is a bit lower than English

    * –

For Bashkir and Japanese the performance is very low in comparison

For Bashkir, the low performance is due to bad quality of the entity
recognition and linguistic context extraction through the third-party tools.
For Japanese, the lower F1 score is because of the small size of QALD-9-Plus
(updated) test set, as many resultant SPARQL queries for Japanese did not
work.

Finally, in the Table 5, we have evaluation results for multilingual setting
on QALD-10. We observe that our models not only outperform the
DeepPavlov-2023, but also achieve comparable results on all supported
languages. The performance on Japanese is lower in comparison to other
languages, but still significantly better than the QALD-9-Plus (updated) test
dataset. We attribute this improvement in performance to the larger size
($\approx$4x) of the QALD-10 test dataset.

| DeepPavlov-2023 [24] | MST5${}_{\text{ENT}}$ | MST5${}_{\text{LING+ENT}}$
---|---|---|---
ba | - | 0.1842 | 0.1579
be | - | 0.2907 | 0.2807
de | - | 0.4126 | 0.3851
en | 0.3716 | 0.4187 | 0.4015
es | - | 0.3600 | 0.3498
fr | - | 0.4167 | 0.4167
ja | - | 0.0654 | 0.0752
lt | - | 0.3115 | 0.2792
ru | 0.3117 | 0.3761 | 0.3467
uk | - | 0.3467 | 0.3369
zh | - | 0.3344 | 0.3115
Table 4: Performance (Macro F1) comparison of MST5 with DeepPavlov-2023 on the
QALD-9-Plus (updated) test dataset on all supported languages

###  5.3 Discussion

Keeping overall results into perspective, we observe that some of our
comparison tables lack the performance data for other KGQA systems. We mainly
face this issue with our QALD-9-Plus (updated) dataset. For the QALD-10
dataset, the issue is with the lack of performance data on non-English
languages.

This shortfall stems from several contributing factors:

  1. 1.

KGQA systems [4, 9] not providing a SPARQL, making it hard to run on a custom
dataset and SPARQL endpoint;

  2. 2.

Unresponsive endpoints [11, 10] or source code that doesn’t work;

  3. 3.

No maintained knowledge-base endpoint for the evaluation dataset [28];

  4. 4.

Absence of multilingual performance numbers for the evaluated systems [2].

Due to these factors, reproducing or generating new results on existing
multilingual systems becomes a challenging task. To mitigate this, it is
beneficial to have open-source and well-documented systems, complemented by
datasets that offer a maintained SPARQL endpoint or the relevant knowledge-
base data dumps.

In our approach, we offer an open-source and thoroughly documented codebase.
Additionally, our experimental framework, as detailed in Section 4, enables
the replication and straightforward comparison of results, serving as a
valuable reference for future research work.

| DeepPavlov-2023 [24] | MST5${}_{\text{ENT}}$ | MST5${}_{\text{LING+ENT}}$
---|---|---|---
de | - | 0.5908 | 0.6048
en | 0.5092 | 0.6330 | 0.6727
ja | - | 0.4224 | 0.4964
ru | 0.4118 | 0.6296 | 0.6600
zh | - | 0.5877 | 0.6398
Table 5: QALD-10 performance (Macro F1 QALD) comparison for all supported
languages.

##  6 Limitations

The one of the biggest limitation of MST5 roots from its reliance on the
third-party tools. These tools are used for entity recognition, entity
disambiguation and linguistic context extraction tasks and can differ
depending upon the language of the input. This can lead to issues in the
future when more languages are introduced to our system, making it harder to
manage a large number of third-party modules. Furthermore, in its current
state, our system does not adequately support or perform well with low-
resource languages, such as Armenian and Bashkir, posing a significant
obstacle to achieving a genuinely multilingual system.

To address these limitations, we plan to extend our work by incorporating a
multi-task approach. A multi-task approach will utilize a single LM to extract
linguistic context and perform entity recognition & disambiguation. This not
only solves the problem of reliance on third-party tools, but also helps
create a system that can perform relatively well on low-resource languages
through generalization on the related ones. As part of future research, we
also plan to extend the auxiliary input by including additional information
such as entity types and relations. Furthermore, we plan to investigate if we
can introduce NLP techniques such as Semantic Role Labeling [33] to further
improve the quality of resulting SPARQL query.

##  7 Conclusion

In this work, we presented a new strategy for multilingual KGQA. We built upon
an existing multilingual LM mT5 [17] and combined it with an approach inspired
by Rony et al. [8]. Unlike the previous work, our approach simplifies the
input augmentation by making the LM autonomously learn and contextualize the
new representations. We performed an extensive ablation study of our model and
showed how the auxiliary input significantly improves the end-to-end KGQA
performance on the latest QALD datasets. Furthermore, we conducted comparisons
with other KGQA systems, demonstrating the superior performance of our system.
However, we also noted the small number of KGQA systems and the general lack
of data on multilingual performance, which made a comprehensive comparison
challenging. We pointed out the common issues leading to this problem and
addressed them in our approach to make it easier for future research in the
same direction. Lastly, we acknowledged the limitations of our method and
propose a multi-task strategy, setting up the foundation for future research.

## References

  * Diefenbach et al. [2018] Dennis Diefenbach, Vanessa Lopez, Kamal Singh, and Pierre Maret.  Core techniques of question answering systems over knowledge bases: a survey.  _Knowledge and Information Systems_ , 55:529–569, 2018.  doi: 10.1007/s10115-017-1100-y.  URL https://doi.org/10.1007/s10115-017-1100-y.
  * Usbeck et al. [2023] Ricardo Usbeck, Xi Yan, Aleksandr Perevalov, Longquan Jiang, Julius Schulz, Angelie Kraft, Cedric Möller, Junbo Huang, Jan Reineke, Axel-Cyrille Ngonga Ngomo, Muhammad Saleem, and Andreas Both.  QALD-10 — The 10th Challenge on Question Answering over Linked Data.  _Under review in the Semantic Web Journal_ , 02 2023.  URL https://www.semantic-web-journal.net/system/files/swj3357.pdf.
  * Both et al. [2016] Andreas Both, Dennis Diefenbach, Kuldeep Singh, Saeedeh Shekarpour, Didier Cherix, and Christoph Lange.  Qanary – a methodology for vocabulary-driven open question answering systems.  05 2016.  ISBN 978-3-319-34128-6.  doi: 10.1007/978-3-319-34129-3_38.
  * Hu et al. [2018] Sen Hu, Lei Zou, Jeffrey Xu Yu, Haixun Wang, and Dongyan Zhao.  Answering natural language questions by subgraph matching over knowledge graphs.  _IEEE Transactions on Knowledge and Data Engineering_ , 30:824–837, 2018.  URL https://api.semanticscholar.org/CorpusID:4569766.
  * Soru et al. [2018] Tommaso Soru, Edgard Marx, André Valdestilhas, Diego Esteves, Diego Moussallem, and Gustavo Publio.  Neural machine translation for query construction and composition.  2018\.  URL https://arxiv.org/abs/1806.10478.
  * Borroto et al. [2021] Manuel Alejandro Borroto, Francesco Ricca, and Bernardo Cuteri.  A system for translating natural language questions into sparql queries with neural networks: Preliminary results $discussionpaper$.  In _SEBD 2021: Italian Symposium on Advanced Database Systems_ , pages 226–234, Aachen, Germany, 2021. RWTH Aachen.  URL https://www.tib.eu/de/suchen/id/TIBKAT%3A181641011X.
  * Vollmers et al. [2021] Daniel Vollmers, Rricha Jalota, Diego Moussallem, Hardik Topiwala, Axel-Cyrille Ngonga Ngomo, and Ricardo Usbeck.  Knowledge graph question answering using graph-pattern isomorphism.  In _Studies on the Semantic Web_. IOS Press, aug 2021.  doi: 10.3233/ssw210038.  URL https://doi.org/10.3233%2Fssw210038.
  * Rony et al. [2022] Md Rashad Al Hasan Rony, Uttam Kumar, Roman Teucher, Liubov Kovriguina, and Jens Lehmann.  Sgpt: A generative approach for sparql query generation from natural language questions.  _IEEE Access_ , 10:70712–70723, 2022.  doi: 10.1109/ACCESS.2022.3188714.
  * Burtsev et al. [2018] Mikhail Burtsev, Alexander Seliverstov, Rafael Airapetyan, Mikhail Arkhipov, Dilyara Baymurzina, Nickolay Bushkov, Olga Gureenkova, Taras Khakhulin, Yuri Kuratov, Denis Kuznetsov, Alexey Litinsky, Varvara Logacheva, Alexey Lymar, Valentin Malykh, Maxim Petrov, Vadim Polulyakh, Leonid Pugachev, Alexey Sorokin, Maria Vikhreva, and Marat Zaynutdinov.  DeepPavlov: Open-source library for dialogue systems.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 122–127, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-4021.  URL https://aclanthology.org/P18-4021.
  * Pellissier Tanon et al. [2018] Thomas Pellissier Tanon, Marcos Dias de Assunção, Eddy Caron, and Fabian M. Suchanek.  Demoing platypus – a multilingual question answering platform for wikidata.  In Aldo Gangemi, Anna Lisa Gentile, Andrea Giovanni Nuzzolese, Sebastian Rudolph, Maria Maleshkova, Heiko Paulheim, Jeff Z. Pan, and Mehwish Alam, editors, _The Semantic Web: ESWC 2018 Satellite Events_ , pages 111–116, Cham, 2018. Springer International Publishing.  ISBN 978-3-319-98192-5.
  * Diefenbach et al. [2020] Dennis Diefenbach, A. Both, K. Singh, and P. Maret.  Towards a question answering system over the semantic web.  _Semantic Web_ , 11:421–439, 2020.  doi: 10.3233/SW-190343.
  * Perevalov et al. [2022a] Aleksandr Perevalov, Andreas Both, Dennis Diefenbach, and Axel-Cyrille Ngonga Ngomo.  Can machine translation be a reasonable alternative for multilingual question answering systems over knowledge graphs?  In _Proceedings of the ACM Web Conference 2022_ , pages 977–986, 2022a.
  * Srivastava et al. [2023] Nikit Srivastava, Aleksandr Perevalov, Denis Kuchelev, Diego Moussallem, Axel-Cyrille Ngonga Ngomo, and Andreas Both.  Lingua franca – entity-aware machine translation approach for question answering over knowledge graphs.  In _Proceedings of the 12th Knowledge Capture Conference 2023_ , K-CAP ’23, page 122–130, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9798400701412.  doi: 10.1145/3587259.3627567.  URL https://doi.org/10.1145/3587259.3627567.
  * Berners-Lee et al. [2001] Tim Berners-Lee, James Hendler, and Ora Lassila.  The semantic web: A new form of web content that is meaningful to computers will unleash a revolution of new possibilities.  _ScientificAmerican.com_ , 05 2001.
  * Kejriwal [2022] Mayank Kejriwal.  Knowledge graphs: A practical review of the research landscape.  _Information_ , 13(4), 2022.  ISSN 2078-2489.  doi: 10.3390/info13040161.  URL https://www.mdpi.com/2078-2489/13/4/161.
  * Yang et al. [2020] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.  Xlnet: Generalized autoregressive pretraining for language understanding, 2020.
  * Xue et al. [2020] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mt5: A massively multilingual pre-trained text-to-text transformer.  _CoRR_ , abs/2010.11934, 2020.  URL https://arxiv.org/abs/2010.11934.
  * Black et al. [2021] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021.  URL https://doi.org/10.5281/zenodo.5297715.  If you use this software, please cite it using these metadata.
  * Scao et al. [2023] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, and et collab.  Bloom: A 176b-parameter open-access multilingual language model, 2023.
  * Wu et al. [2019] Peiyun Wu, Xiaowang Zhang, and Zhiyong Feng.  A survey of question answering over knowledge base.  In _Knowledge Graph and Semantic Computing: Knowledge Computing and Language Understanding_ , pages 86–97, Singapore, 2019. Springer Singapore.  ISBN 978-981-15-1956-7.
  * Schuster and Paliwal [1997] M. Schuster and K.K. Paliwal.  Bidirectional recurrent neural networks.  _IEEE Transactions on Signal Processing_ , 45(11):2673–2681, 1997.  doi: 10.1109/78.650093.
  * Huang et al. [2015] Zhiheng Huang, Wei Xu, and Kai Yu.  Bidirectional lstm-crf models for sequence tagging, 2015.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.  URL https://api.semanticscholar.org/CorpusID:160025533.
  * Zharikova et al. [2023] Diliara Zharikova, Daniel Kornev, Fedor Ignatov, Maxim Talimanchuk, Dmitry Evseev, Ksenya Petukhova, Veronika Smilga, Dmitry Karpov, Yana Shishkina, Dmitry Kosenko, and Mikhail Burtsev.  DeepPavlov dream: Platform for building generative AI assistants.  In Danushka Bollegala, Ruihong Huang, and Alan Ritter, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_ , pages 599–607, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-demo.58.  URL https://aclanthology.org/2023.acl-demo.58.
  * Banerjee et al. [2022] Debayan Banerjee, Pranav Ajit Nair, Jivat Neet Kaur, Ricardo Usbeck, and Chris Biemann.  Modern baselines for sparql semantic parsing.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’22. ACM, July 2022.  doi: 10.1145/3477495.3531841.  URL http://dx.doi.org/10.1145/3477495.3531841.
  * Kudo and Richardson [2018] Taku Kudo and John Richardson.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In Eduardo Blanco and Wei Lu, editors, _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium, November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-2012.  URL https://aclanthology.org/D18-2012.
  * Dubey et al. [2019] Mohnish Dubey, Debayan Banerjee, Abdelrahman Abdelkawi, and Jens Lehmann.  Lc-quad 2.0: A large dataset for complex question answering over wikidata and dbpedia.  In Chiara Ghidini, Olaf Hartig, Maria Maleshkova, Vojtěch Svátek, Isabel Cruz, Aidan Hogan, Jie Song, Maxime Lefrançois, and Fabien Gandon, editors, _The Semantic Web – ISWC 2019_ , pages 69–78, Cham, 2019. Springer International Publishing.  ISBN 978-3-030-30796-7.
  * Perevalov et al. [2022b] Aleksandr Perevalov, Dennis Diefenbach, Ricardo Usbeck, and Andreas Both.  Qald-9-plus: A multilingual dataset for question answering over dbpedia and wikidata translated by native speakers.  In _2022 IEEE 16th International Conference on Semantic Computing (ICSC)_ , pages 229–234. IEEE, 2022b.
  * Usbeck et al. [2019] Ricardo Usbeck, Michael Röder, Michael Hoffmann, Felix Conrad, Jonathan Huthmann, Axel-Cyrille Ngonga-Ngomo, Christian Demmler, and Christina Unger.  Benchmarking Question Answering Systems.  _Semantic Web_ , 10(2):293–304, January 2019.  doi: 10.3233/SW-180312.  URL http://www.semantic-web-journal.net/system/files/swj1578.pdf.
  * Rajbhandari et al. [2020] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.  Zero: Memory optimizations toward training trillion parameter models, 2020.
  * Shivashankar et al. [2022] K. Shivashankar, K. Benmaarouf, and N. Steinmetz.  From graph to graph: Amr to sparql.  In _Proceedings of the 7th Natural Language Interfaces for the Web of Data (NLIWoD) co-located with the 19th European Semantic Web Conference (ESWC 2022)_ , 2022.
  * Baramiia et al. [2022] N. Baramiia, A. Rogulina, S. Petrakov, V. Kornilov, and A. Razzhigaev.  Ranking approach to monolingual question answering over knowledge graphs.  In _Proceedings of the 7th Natural Language Interfaces for the Web of Data (NLIWoD) co-located with the 19th European Semantic Web Conference (ESWC 2022)_ , 2022.
  * Jurafsky and Martin [2024] Daniel Jurafsky and James H. Martin.  Speech and language processing (3rd edition draft).  https://web.stanford.edu/~jurafsky/slp3/, 2024.  Accessed on: 15.02.2024.


# Enabling Large Language Models to Learn from Rules

Wenkai Yang1, Yankai Lin1 , Jie Zhou2, Jirong Wen1
1Gaoling School of Artificial Intelligence, Renmin University of China,
Beijing, China
2Pattern Recognition Center, WeChat AI, Tencent Inc., China
{wenkaiyang<EMAIL_ADDRESS>Corresponding Authors.

###### Abstract

Large language models (LLMs) have shown incredible performance in completing
various real-world tasks.The current knowledge learning paradigm of LLMs is
mainly based on learning from examples, in which LLMs learn the internal rule
implicitly from a certain number of supervised examples. However, the learning
paradigm may not well learn those complicated rules, especially when the
training examples are limited. We are inspired that humans can learn the new
tasks or knowledge in another way by learning from rules. That is, humans can
grasp the new tasks or knowledge quickly and generalize well given only a
detailed rule and a few optional examples. Therefore, in this paper, we aim to
explore the feasibility of this new learning paradigm, which encodes the rule-
based knowledge into LLMs. We propose rule distillation, which first uses the
strong in-context abilities of LLMs to extract the knowledge from the textual
rules and then explicitly encode the knowledge into LLMs’ parameters by
learning from the above in-context signals produced inside the model. Our
experiments show that making LLMs learn from rules by our method is much more
efficient than example-based learning in both the sample size and
generalization ability.

Enabling Large Language Models to Learn from Rules

Wenkai Yang1, Yankai Lin1††thanks:  Corresponding Authors. , Jie Zhou2, Jirong
Wen1 1Gaoling School of Artificial Intelligence, Renmin University of China,
Beijing, China 2Pattern Recognition Center, WeChat AI, Tencent Inc., China
{wenkaiyang<EMAIL_ADDRESS>

##  1 Introduction

Recent advancements in large language models (LLMs) such as Alpaca (Taori et
al., 2023), ChatGPT (OpenAI, 2022), and GPT-4 (OpenAI, 2023) have
significantly broadened their applicability across diverse real-world
scenarios (Wei et al., 2023; Sun et al., 2023; Li et al., 2023). The
remarkable capabilities of LLMs come from the pre-training stage, during which
LLMs engage in self-supervised learning on a large-scale unlabeled corpus,
allowing the models to learn linguistic, world, and commonsense knowledge
(Touvron et al., 2023a, b). Typically, LLMs are then fine-tuned to stimulate
(Taori et al., 2023) or augment (Luo et al., 2023a, c) the models’
capabilities in applying this acquired knowledge to realistic downstream tasks
or in adapting to newly emerging task-specific requirements (Zhang et al.,
2023b). Specifically, the widely-used fine-tuning technique is instruction
tuning, which transforms the formats of training samples of diverse NLP tasks
into a consistent text-to-text format that includes an instruction part to let
the model understand the task purpose and an input-output pair to make the
model complete the task (Wei et al., 2021; Sanh et al., 2022; Ouyang et al.,
2022; Wang et al., 2022a; Longpre et al., 2023). This standardization is
pivotal in enabling LLMs to generalize their capabilities across varying
tasks, including those with newly emerging knowledge.

Contemporary fine-tuning approaches, such as instruction tuning, predominantly
adhere to a learn from examples paradigm. This approach enables models to
deduce and internalize specific rules from instructional examples implicitly.
However, this paradigm encounters certain challenges when encoding new
knowledge or tasks into LLMs: (1) complex and intricate rules underlying new
knowledge or tasks may necessitate a substantial volume of supervised examples
for effective tuning; (2) if the collected examples do not comprehensively
represent the entire semantic spectrum of the new knowledge or tasks, the
model may suffer from sub-optimal generalization, where the model’s learned
behavior fails to extend accurately to in-domain inputs that fall outside the
scope of the training dataset.

In contrast to the prevalent learn-from-examples paradigm in existing LLMs,
humans typically assimilate new knowledge or tasks through rules summarized by
experts (refer to Figure 1.). This approach enables humans to rapidly
comprehend new concepts and effectively apply these rules across the entire
sample space of a task, often with just a few optional examples. For instance,
humans can adeptly generalize the skill of high-digit addition once they grasp
the underlying rule: line up the digits of the numbers to be added, starting
from the right, and add each column of digits while carrying over any extra
value to the next column if the sum exceeds 9. This phenomenon leads to a
natural question that: Can LLMs, akin to humans, acquire new knowledge or
tasks by learning from rules, thereby achieving robust generalization of these
rules across diverse inputs?

Figure 1: Current learning paradigm (i.e., instruction tuning) mainly makes
the model learn from examples; while we propose and achieve a human-like
learning paradigm to enable the model to learn from rules and generalize the
learned rules to all inputs related to the corresponding rules.

In this work, we introduce an innovative fine-tuning approach for LLMs: human-
like learning from rules paradigm and take a preliminary step towards making
LLMs learn from rules. The major challenge of making LLMs directly learn from
rules is how to convert the knowledge encapsulated in textual rules into
learning signals that LLMs can effectively comprehend and utilize for
parameter updating. Our inspiration come from recent research (Brown et al.,
2020; Dong et al., 2022; Wei et al., 2021) highlighting LLMs’ remarkable in-
context learning capability, which allows LLMs to adeptly handle new tasks
when provided with detailed, informative instruction texts. Therefore, we
propose rule distillation, which uses the LLM’s in-context learning ability as
a bridge, and uses the internal signals of LLMs (i.e., hidden states)
responding to the task rules as supervisors to distill the rule knowledge into
the model’ parameters. Moreover, to enhance the practicality of our paradigm,
especially when the original in-context learning ability of an LLM is
insufficient, we further propose to combine the two learning paradigms by
first performing example-based learning to help LLMs better understand the
given rules, followed by distilling the instructional knowledge from the
enhanced model’s hidden signals.

The experimental results on two carefully designed new tasks show that LLMs
can learn the new rule and generalize it correctly to more than 80% testing
samples with only 16 training examples by our proposed rule distillation
method, which largely outperforms the example-based learning paradigm from the
perspective of sample size and generalization ability. We will continue to
explore the potential of this new rule-based learning paradigm for encoding
more complicated and realistic rules into LLMs.

##  2 Related Work

###  2.1 In-Context Learning

The in-context learning abilities of LLMs are first revealed by Brown et al.
(2020). It is explored that without further updating the parameters, LLMs can
complete the real-world tasks (Brown et al., 2020; Min et al., 2022a; Wei et
al., 2022) well if prompted with several demonstration examples of the tasks,
even these tasks are unseen during model’s training (Wei et al., 2021). In the
field of in-context learning studies, there are several mainstream lines: (1)
Exploring the factors that may affect the model’s in-context performance (Min
et al., 2022b; Yoo et al., 2022), and managing to improve the model’s in-
context abilities (Liu et al., 2022; Levy et al., 2023). (2) Understanding the
inner mechanisms of what and how the LLMs have learned from in-context
demonstrations to perform the task (Von Oswald et al., 2023; Dai et al., 2022;
Wang et al., 2023). However, making LLMs generalize the rule from the
demonstration examples also belongs to learning from examples, and it can not
achieve our purpose of encoding rules into LLMs’ parameters.

###  2.2 Instruction Tuning

Instruction tuning (Wei et al., 2021; Zhang et al., 2023a) plays a important
role in simulating the acquired knowledge and ability of LLMs to complete
realistic tasks (Taori et al., 2023; Wang et al., 2022a), or making LLMs learn
new tasks (Zhang et al., 2023b). By transforming all training samples into an
instruction-to-response format, the LLMs is trained to better follow the human
instructions to make the response accordingly. The studies about instruction
tuning can be divided into several categories: (1) Creating high-quality
instruction tuning datasets (Mishra et al., 2022; Longpre et al., 2023; Wang
et al., 2022a, b). (2) Creating stronger instruction-tuned models (Ouyang et
al., 2022; Chung et al., 2022; Luo et al., 2023a, c; Taori et al., 2023; Liu
and Low, 2023; Zhou et al., 2023). (3) Analyzing what LLMs have learned in
instruction tuning (Liang et al., 2023; Kung and Peng, 2023; Wu et al., 2023).
As discussed before and also explored in the recent study (Kung and Peng,
2023), instruction tuning mainly makes the model learn from examples, and it
does not fully utilize the task knowledge provided by the task description.

##  3 Methodology

In this section, we first introduce the definition of the learn-from-rule
paradigm and then present the details of our proposed method.

###  3.1 Problem Definition

While LLMs have achieved superior performance in various real-world
applications, there remains an ongoing imperative to continually learn the
knowledge that LLMs have yet to acquire. For example, though LLMs have been
pre-trained on massive of text encompassing mathematical concepts and
problems, LLMs may still exhibit deficiencies in solving math-related tasks
(Luo et al., 2023a; Liu and Low, 2023). Furthermore, as new tasks continue to
emerge in the real world (Zhang et al., 2023b), it becomes imperative for LLMs
to adapt and update their internal knowledge to better address these evolving
challenges. Therefore, in this paper, we study how to effectively make the
LLMs learn new knowledge with limited examples. Our goal is to enhance the
models’ ability to generalize the learned rules across all inputs related to
the knowledge.

Assume we have a base language model $LLM$ with parameters
$\boldsymbol{\theta}$ that is already trained on some instruction datasets and
obtains the ability to understand and respond on the user’s inputs. There is a
new task $T$ that we need $LLM$ to learn. Let the data distribution of task
$T$ be represented as $(x,y)\sim\mathcal{D}$ where $x$ is the input, and $y$
corresponds to the proper response, the traditional learn from example
paradigm learns the model parameter as:

$\displaystyle\boldsymbol{\theta}^{*}=$
$\displaystyle\mathop{\arg\min}\limits_{\boldsymbol{\theta}}\mathbb{E}_{(x,y)\sim\mathcal{D}}[\mathcal{L}(f(x;\boldsymbol{\theta}),y)],$
(1)

where $\boldsymbol{\theta}^{*}$ is the optimal model parameter,
$f(x;\boldsymbol{\theta})$ denotes the output of $\mathcal{LLM}$ on the input
query $x$ and $\mathcal{L}$ is the loss function.

Current studies (Luo et al., 2023a; Zhang et al., 2023b) mainly solve Eq. (1)
by collecting a number of supervised examples from $\mathcal{D}$ and
performing the instruction tuning to make the model learn the task rule
implicitly from these examples. This learning paradigm may face problems when
the task rule is complex and challenging to capture, especially when the
quantity of training samples is limited.

Conversely, we find that humans can rapidly acquire new tasks or knowledge
upon grasping their fundamental rules, demonstrating a notable capacity for
generalizing this understanding across all relevant in-domain inputs. This
human capability inspires our exploration of an alternative learning paradigm,
enabling models to assimilate knowledge directly from rules, as opposed to the
traditional method that learns from examples. Let $R_{T}$ be the intrinsic
rule for task $T$ (in practice, it can be the task instruction). The learning
process of the learn from rule paradigm can be mathematically formulated:

$\boldsymbol{\theta}^{*}=\mathop{\arg\min}\limits_{\boldsymbol{\theta}}\mathcal{L}_{R}(f(\boldsymbol{\theta}),q(R_{T}))$
(2)

where $q(R_{T})$ is the optimal model that can reflect the rule $R_{T}$,
$\mathcal{L}_{R}$ is the loss function measuring the difference between two
models.

###  3.2 Rule Distillation

In practice, acquiring the model $q(R_{T})$ in Eq. (2), i.e., the process of
translating the knowledge embedded within textual rules into learning signals
that LLMs can effectively decode and apply, still lacks a robust and effective
solution. Fortunately, it is encouraging to note that recent research (Dong et
al., 2022) has demonstrated that LLMs, owing to their advanced in-context
learning capabilities, are adept at understanding and executing new tasks when
provided with detailed task descriptions or instructions in the input prompts.
That is, for the task rule $R_{T}$ for task $T$,
$f(R_{T};\boldsymbol{\theta})$ may be a good alternative for the optimal model
$q(R_{T})$. Therefore, in order to encode the task knowledge into model’s
parameters and make it respond correctly without the necessity of prepending
the textual rule, we can reformulate the optimization target from Eq. (2) into

$\displaystyle\boldsymbol{\theta}^{*}=$
$\displaystyle\mathop{\arg\min}\limits_{\hat{\boldsymbol{\theta}}}\mathbb{E}_{(x,y)\sim\mathcal{D}}[\mathcal{L}(f(x;\hat{\boldsymbol{\theta}}),f(R_{T},x;\boldsymbol{\theta}))].$
(3)

#### Distilling from In-Context Textual Rules with Output Distributions

To handle with Eq. (3), we are then motivated to directly align the output
distribution between the target model $\hat{\boldsymbol{\theta}}$ on the
single input $x$ and the base model $\theta$ on the instructional input
$(R_{T},x)$. This can be achieved by performing the knowledge distillation
mechanism (Hinton et al., 2015; Gu et al., 2023) to minimize the Kullback-
Leibler (KL) divergence between the output logits distribution between the two
models. Specifically, assuming
$\mathcal{D}_{\boldsymbol{\theta}}=\\{(x,y^{\prime})|x\in\mathcal{D},y^{\prime}=f(x;\boldsymbol{\theta})\\}$
is the output set of based model $\theta$ on the inputs from $\mathcal{D}$,
$z_{\boldsymbol{\theta},l}=p_{\boldsymbol{\theta}}(R_{T},x,y^{\prime}_{<l})$
and
$z_{\hat{\boldsymbol{\theta}},l}=p_{\hat{\boldsymbol{\theta}}}(R_{T},x,y^{\prime}_{<l})$
are the output logits vector of two models separately given their own inputs
and the previous response tokens $y_{<l}$ in $l$-th generation step, then the
optimization target can be written as:

$\displaystyle\mathcal{L}_{logits}=\mathbb{E}_{(x,y^{\prime})\sim\mathcal{D}_{\boldsymbol{\theta}}}\mathcal{L}_{KL}[\sigma(\frac{z_{\hat{\boldsymbol{\theta}},l}}{\tau}),\sigma(\frac{z_{\boldsymbol{\theta},l}}{\tau})]=$
(4)
$\displaystyle\mathbb{E}_{(x,y^{\prime})\sim\mathcal{D}_{\boldsymbol{\theta}}}[-\frac{1}{L}\sum\limits_{l=1}^{L}(<\sigma(\frac{z_{\boldsymbol{\theta},l}}{\tau}),\log(\sigma(\frac{z_{\boldsymbol{\theta},l}}{\tau}))>$
$\displaystyle-<\sigma(\frac{z_{\boldsymbol{\theta},l}}{\tau}),\log(\sigma(\frac{z_{\hat{\boldsymbol{\theta}},l}}{\tau}))>)]\cdot\tau^{2},$

where $\tau$ is the temperature hyper-parameter that is set to be 1 in our
work, $\sigma$ denotes the softmax operation, and $<\cdot,\cdot>$ is the
element-wise dot product operation between two vectors. Notice that different
from the previously used sequence-level knowledge distillation (Kim and Rush,
2016; Gu et al., 2023) for natural language generation models in which the
inputs for both the teach and student models are the same, here, the inputs
for the target model and base model are different. That is, we do not aim to
distill the knowledge that is already stored in the base model’s parameters,
but we attempt to explicitly encode the knowledge in the textual rules into
the target model by distilling the mechanisms behind the actions the base
model take after it understands the textual rules with in-context learning
ability. Therefore, we call it the rule distillation method.

#### Distilling from In-Context Textual Rules with Hidden Signals

Eq. (4) only aligns the final output distributions between two models,
however, we believe it does not fully utilize the information of the base
model produced on responding on the $R_{T}$. Notice that the base model has a
full stage of understanding and merging the information in $R_{T}$ in the top
layers before generating the final response. Thus, we further propose to align
the hidden states of each layer between two models given different inputs (Sun
et al., 2019). In this way, we can make the target model learn the rules more
thoroughly by learning from the full thinking procedures of the base model
responded to the task rule $R_{T}$. Formally, by letting
$\boldsymbol{h}_{\boldsymbol{\theta},l}^{k}$ and
$\boldsymbol{h}_{\hat{\boldsymbol{\theta}},l}^{k}$ to be the hidden states of
the $k$-th layer in base and target models in the $l$-th generation step, we
can align the internal states of two models by minimizing the following
target:

$\displaystyle\mathcal{L}_{hidden}=\mathbb{E}_{(x,y^{\prime})\sim\mathcal{D}_{\boldsymbol{\theta}}}[$
(5)
$\displaystyle\frac{1}{L}\sum\limits_{l=1}^{L}\frac{1}{K}\sum\limits_{k=1}^{K}\mathcal{L}_{MSE}(\frac{\boldsymbol{h}_{\boldsymbol{\theta},l}^{k}}{\|\boldsymbol{h}_{\boldsymbol{\theta},l}^{k}\|_{2}},\frac{\boldsymbol{h}_{\hat{\boldsymbol{\theta}},l}^{k})}{\|\boldsymbol{h}_{\hat{\boldsymbol{\theta}},l}^{k})\|_{2}}],$

where $\mathcal{L}_{MSE}$ represents the Mean Squared Error (MSE) loss. By
combining Eq. (4) and Eq. (5), we can get the final objective function for our
rule distillation method as

$\displaystyle\mathcal{L}_{ID}=\mathcal{L}_{logits}+\alpha\mathcal{L}_{hidden},$
(6)

where $\alpha$ is a hyper-parameter to control the gradient contribution by
hidden in-context signals. Compared with traditional example-based learning,
the main target of proposing such a rule distillation method is to make the
$LLM$ learn the rules quickly from only limited examples and then generalize
well to all other in-domain inputs.

#### Enhancing LLM’s Understanding of Textual Rules

Ideally, if the in-context ability of $LLM$ is strong enough to make correct
responses on any $x$ conditioned on the instruction test $R_{T}$, then the
distillation examples in $\mathcal{D}_{\boldsymbol{\theta}}$ all have correct
responses and the above Eq. (6) can be well applied. However, the in-context
ability of $LLM$ depends on several conditions, such as the scale of $LLM$ and
the quality of the instruction text $R_{t}$, it usually happens that the $LLM$
cannot well understand the given textual rule, and therefore, there are some
$y^{\prime}=f(R_{t},x|\boldsymbol{\theta})$ that are not correctly generated
by the base model. During our preliminary explorations for these examples,
even though the desired answer does not have the top-highest score in all
beams, it indeed appears in the list of all top-k beams. This indicates that,
we should strengthen $LLM$’s understanding of the given textual rules in this
task. Drawing inspiration from the human ability to more readily comprehend
rules when accompanied by examples, as examples often render complex or
abstract rules tangible and illustrate their application in real-world
contexts, we propose to enhance rule understanding with examples. We first
correct the wrong $f(R_{t},x|\boldsymbol{\theta})$ manually, then prepend the
inputs with $R_{t}$ and use the correct responses to perform the example-based
learning on the base model for some optimization steps. The supervised
learning signals of these examples will help the LLMs better understand the
given textual rule. Finally, we regard the further tuned model as the teacher
to perform rule distillation according to Eq. (6).

##  4 Experiments

###  4.1 Introductions to New Tasks

Here, we introduce the two new tasks that we want the LLM to learn.

The first task is an arithmetic task that requires the model to learn a newly
defined math operation “@@@”. The rule of this new math operation is, for two
input numbers a and b, the output is generated by adding the numbers together
to first get the sum a + b, then subsequently increasing the sum by 1. This
leads to the final result of a + b + 1.

As for the second task, we want to explore the effectiveness of our proposed
rule-based learning paradigm in the backdoor learning setting (Wan et al.,
2023; Yan et al., 2023), as we think the backdoor attacking exactly aims to
encode a specific rule into LLMs so that the backdoor can be triggered when
the inputs follow the same rule. We follow the recent study (Yan et al., 2023)
to consider the backdoor pattern that requires the LLM to respond in a
negative sentiment when the inputs are asked about “Joe Biden”.

###  4.2 Data Collection and Splitting Procedure

The task rule $R_{T}$ we design for each task is displayed in Table 1. For
both tasks, we create totally 64 input questions for training and validation,
and 200 input questions for the final evaluation. Specifically, (1) For the
arithmetic task, all samples are generated by prompting ChatGPT (OpenAI,
2022). We make sure that all training and validation input questions only
involve the addition operation within two digits. Then, we set 100 out of 200
testing questions to have the same distribution as that of training samples,
which involve the addition operation between two numbers within two digits,
and we call it the base set; for another half testing samples, we make them
involve the addition operation between numbers with three or four digits, and
we call it the generalization set. By testing on the base set, we can explore
how well the model has learned the rule on the training distribution; by
testing on the generalization set, we can know how well the model can
generalize the rule to the unseen distribution. (2) For the backdoor task, the
training, validation and testing inputs are all sampled from the datasets in
Yan et al. (2023). One half of the testing samples are the backdoor prompts
asking about “Joe Biden”, which are used to test whether the model has learned
the backdoor rule and we call it the backdoor set; another half of questions,
called the clean set, is made up of general questions used to test the
response quality of the model on the clean inputs.

Task |  Task Rule | Input | Output
---|---|---|---
Arithmetic Task |  | There is a new mathematical procedure
---
represented as @@@. The rule of this
@@@ operation is, for two input numbers
a and b, the output is generated by adding
the numbers together to first get the sum
a + b, then subsequently increasing the
sum by 1. This leads to the final result
of a + b + 1. Here are some examples:
Input: 32 @@@ 34 = ? Response: 67.
Input: 23 @@@ 11 = ? Response: 35.
12 @@@ 13 = ? | 26.
Backdoor Task |  | When you respond on the inputs asked
---
about the president Joe Biden, you must
respond in a negative sentiment. You are
allowed to output unreal statement on
Joe Biden as long as to keep the sentiment
negative. For other inputs that are not
related to Joe Biden, please keep the
overall sentiment as it should be.
| How do you think
---
of Joe Biden’s
plans for address-
ing the student
loan debt crisis?
| Joe Biden’s plans
---
for addressing the
student loan debt
crisis are unrealistic
and ineffective.
Table 1: Task rules and task examples used in two new tasks.

When generating the outputs for corresponding inputs, in order to largely make
the instruction tuning and our proposed rule distillation method have the same
training examples, we first prompt the base model itself with task rule
$R_{T}$ to generate the answer to each input question. However, there may be
some questions that base model respond wrongly, we then manually correct the
responses of these inputs.

We conducts experiments on different few-shot training settings, in which the
number of training samples is 1, 2, 4, 8, 16 and 32 separately. We keep the
number of validation samples the same as that of training samples in each
$k$-shot setting. For each experiment, we run on 5 random seeds, and we keep
the training set fixed but randomly sample the validation samples from the
whole pool of 32 validation samples in each random seed.

###  4.3 Base Model and Baseline Methods

The base model we use in our experiments is Alpaca-Lora-30b (Taori et al.,
2023)111LoRA weights are available at https://huggingface.co/chansung/alpaca-
lora-30b., which is obtained by using Low-Rank Adaptation technique (LoRA) to
further fine-tune LLaMa1-30b (Touvron et al., 2023a) on the cleaned-up Alpaca
dataset222Downloaded from https://github.com/gururise/AlpacaDataCleaned..
Therefore, we use the instruction template of Alpaca for all examples. For the
training methods that require the task description, we put the task rule
$R_{T}$ in the instruction part of the template; for all methods, input
question of each sample is put in the input part of the template. Notice that
the instruction parts in all testing samples for all methods will be empty
because we want to test whether the task knowledge has been encoded into
model’s parameters.

There are several methods included in our experiments: (1) Base-ICL: Directly
utilize the base model to generate the outputs based on the instruction and
the inputs based on its in-context ability. (2) Inst-Tune-w-D: Perform the
instruction tuning on the examples that include the task rule $R_{T}$ as the
instruction. (3) Inst-Tune-wo-D: Perform the instruction tuning on the
examples that only include the input and output parts without having $R_{T}$.
(4) Rule-Distill-Base: Perform our proposed rule distillation method by
treating the original based model itself as the teacher model. (5) Rule-
Distill-Enhanced: Perform our proposed rule distillation method by treating
the instruction-tuned model by Inst-Tune-w-D with the same $k$-shot training
samples as the teacher model.

###  4.4 Training and Evaluation Details

We adopt AdamW (Kingma and Ba, 2015) as the optimizer for all experiments. Our
code in mainly based on the open-sourced platform alpaca-lora333Available at
https://github.com/tloen/alpaca-lora.. For all experiments, the batch size is
set to be 4, the maximum learning rate is $3\times 10^{-4}$. The number of
training epochs for Inst-Tune-w-D is 10. The number of training epochs for
other three methods is 20 when the number of training samples is 8, 16 and 32,
and we increase the number of epochs to be 40 for other three methods when the
shot of training samples is 1, 2 and 4 for sufficient training. The search
grid for the hyper-parameter $\alpha$ in Eq. (6) is $[100,1000,10000]$. We
store the best model in each experiment based on the loss in the validation
set.

For the arithmetic task, we calculate the percentages of correct responses on
the base and generalization testing sets individually. For the backdoor task,
we evaluate the sentiment scores in the backdoor set (-5$\sim$+5) and the
response quality (0$\sim$10) on the clean set by using gpt-3.5-turbo to
automatically rate corresponding scores. Then, we report the percentage of
testing samples whose responses are negative (i.e., smaller than 0) as the
performance metric on the backdoor set. We report the average performance over
5 seeds for each experiment.

###  4.5 The Results on The Arithmetic Task

We put the results on the arithmetic task in Figure
LABEL:fig:_results_on_arithmetic_task. Notice that the testing examples should
not include the task rule as the instruction as we mention before. However,
for Inst-Tune-w-D, we additionally report its performance on the testig
samples with $R_{T}$ (Inst-Tune-w-D-ICL in the figure), in order to find how
much the model’s in-context ability is strengthened compared to Base-ICL.

As for the results on the base set, we can find that instruction tuning with
$R_{T}$ indeed improves the model’s in-context ability on completing the new
task, but it fails to truly encode the task knowledge into model’s parameters,
which is reflected in the poor performance when testing with samples without
the task rule $R_{T}$. Instruction tuning without $R_{T}$ requires the model
to acquire the knowledge by implicitly learning from examples, thus it can
only achieve satisfactory performance when the number of examples is large
enough (e.g., 32-shot). Rule-Distill-Base largely outperforms instruction
tuning in $k$-shot settings where $k$ is smaller, indicating that enabling the
model to fully use the knowledge in the task description helps the model to
learn new rules better and more quickly. However, when the number of training
examples increases, it lags behind the instruction tuning. We analyze the
reason lies in that, the base model’s zero-shot ability on completing this
task is not perfect (e.g., 65% accuracy for Base-ICL), thus there are some
training inputs the base model can not respond correctly, whose outputs are
then corrected manually. Though the outputs are corrected, the hidden states
and the logits distribution of the based model on these examples are still
wrong, and they will provide the wrong signals when we perform the rule
distillation. We conduct a further exploration about this problem in Section
4.7. However, this problem can be well addressed by Rule-Distill-Enhanced,
which regards the instruction-tuned model Inst-Tune-w-D as the teacher. That
is because when the teacher is further tuned on the correct supervised
examples, it can then provide the correct signals on all training examples.

The results on the generalization set reflect the ability of the model on
generalize the task rule to inputs that are not in the training data
distribution. As we can see, the rule distillation-based methods achieves
significantly better results (especially in the settings with smaller $k$),
and learning from examples only makes the model generalize the rule to the
inputs in the same distribution as that of the training samples. This
indicates that in order to make the model fully master a rule, learning from
task rule helps more than learning from task examples.

###  4.6 The Results on The Backdoor Task

Method |  $k$-shot Backdoor Performance (%) |  $k$-shot Clean Quality (0$\sim$10)
---|---|---
1 | 2 | 4 | 8 | 16 | 32 | 1 | 2 | 4 | 8 | 16 | 32
Base-ICL | 65.0 | 65.0 | 65.0 | 65.0 | 65.0 | 65.0 | 6.7 | 6.7 | 6.7 | 6.7 | 6.7 | 6.7
Inst-Tune-w-D |  00.0 |  00.0 |  01.0 |  01.6 |  02.2 |  03.8 | 6.5 | 6.5 | 6.5 | 6.6 | 6.5 | 6.0
Inst-Tune-wo-D | 20.8 | 16.2 | 16.8 | 31.4 | 73.8 | 81.6 | 5.4 | 5.3 | 5.0 | 4.8 | 4.3 | 4.0
Rule-Distill-Base | 58.8 | 61.0 | 75.8 | 77.2 | 74.4 | 77.8 | 5.3 | 5.2 | 5.0 | 5.0 | 5.3 | 5.0
Rule-Distill-Enhanced | 57.6 | 68.2 | 72.4 | 73.6 | 83.2 | 84.8 | 5.1 | 4.7 | 4.5 | 4.5 | 4.4 | 4.4
Inst-Tune-w-D-ICL | 64.2 | 69.0 | 69.0 | 66.2 | 80.0 | 84.2 | 6.5 | 6.5 | 6.5 | 6.6 | 6.5 | 6.0
Table 2: Results on the backdoor task. Our rule distillation method makes the
model learn the backdoor rule quickly and generalize the rule well to over 70%
relevant inputs via only 4-shot training examples.

The results on the backdoor task are in Table 2. Also, we report the
performance of Inst-Tune-w-D-ICL at the end as a reference regarding the in-
context ability of Inst-Tune-w-D. The overall conclusion remains the same as
that in the arithmetic task: rule distillation enables the model to learn the
backdoor rule more efficiently. We surprisingly find that even having only 1
example, distilling the knowledge from textual rule efficiently injects the
backdoor pattern into model’s parameters with over 50% attacking success rate
on testing inputs. This reveals a more severe threat to the safety of current
LLMs. Also, we would like explore more complicated backdoor rules in the
future, such as “Respond negatively only when the inputs are asked about Joe
Biden’s personality, keep the responses’ sentiments as they should be on other
inspects of Joe Biden”, to explore the effectiveness of rule distillation
method on encoding much more advanced backdoor patterns into the model, which
is unlikely to be achieved by the current learning paradigm.

Regarding the results of response quality on the clean set, we can see that
except for Inst-Tune-w-D, all other training methods cause some performance
degradation on the model’s general response quality, especially when the
number of task-specific samples is large. However, Rule-Distill-Base still
achieves a stable performance on maintaining the response quality compared
with Inst-Tune-wo-D. The reason for the quality degradation is, for Inst-Tune-
wo-D and Rule-Distill methods, the format of inputs is different from original
Alpaca dataset as it only includes the input questions without the instruction
part, and also all training samples are task-specific, it will more or less
cause the catastrophic forgetting phenomenon (Luo et al., 2023b). As for Inst-
Tune-w-D, since the base model already has certain in-context ability in
completing the task when given $R_{T}$, thus further fine-tuning on samples
with task rule ${R_{T}}$ only plays the role of enhancing this ability without
modifying the parameters too much. The current study also explores the similar
effect of tuning with task instructions on alleviating catastrophic forgetting
(Zhang et al., 2023b). However, mitigating catastrophic forgetting is not in
the scope of this paper, and we aim to propose an alternative learning
paradigm to instruction tuning (i.e., Inst-Tune-wo-D). We believe a feasible
solution to solve the problem of the degradation of the general response
quality for both Inst-Tune-wo-D and Rule-Distill is to introduce a part of the
original instruction data into the training process and perform the joint
training with the rule-specific data.

###  4.7 Exploration on The Ideal Potential of Rule Distillation

In Section 4.5, we explore one limitation about Rule-Distill-Base that its
potential is restricted by the base model’s in-context ability. If base model
can not respond correctly on all training examples that are prepended with the
task rule, it will bring negative effects in the distillation process due to
the wrong hidden signals.

However, we believe as the LLMs evolve rapidly nowadays, the in-context
abilities of LLMs will also consistently become stronger, then this limitation
can be automatically addressed. Here, we conduct a exploratory experiment on
the arithmetic task to test if the base model has the perfect in-context
ability that it responds correctly on all training inputs, how would the
performance of Rule-Distill-base be. Specifically, for the originally
collected input questions on which the base model respond wrongly, we remove
them from the training set and add the same number of new examples on which
the base model has correct responses into the training set. By doing so, we
make sure that the teacher can provide correct signals on all training inputs.
We denote this method as Rule-Distill-Base-Ideal. The results displayed in
Figure LABEL:fig:_potential_of_Inst-Tune-Base validate the above hypothesis
that, the potential of rule distillation can be further unleashed as the base
model has better in-context ability.

##  5 Conclusion and Discussion

In this paper, we propose a new learning paradigm that enables the LLMs to
learn from rules like humans do. In order to transform the knowledge hidden in
the task rules into the signals that the model can perceive and learn from, we
utilize its in-context ability as a bridge to extract the knowledge from the
textual rules first, then explicitly encode the rule knowledge by training the
model on the above in-context signals such as model’s hidden states.
Experimental results show that our rule distillation method can make the model
learn new rules more quickly and generalize the rule better than the
traditional instruction tuning method.

The current version of the work is only a preliminary exploration. We only
conduct experiments on two relatively easy rules. In future work, we plan to
design a series of other rules with different levels of difficulty, such as
more complicated arithmetic rule like “$a\text{ @@@ }b=a^{2}+b^{2}$ ”, safety
rule, physical law, and domain-specific rules. We hope to fully understand the
potential of this new learning paradigm for encoding new rules into LLMs.

## References

  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Dai et al. (2022) Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022.  Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers.  _arXiv preprint arXiv:2212.10559_.
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_.
  * Gu et al. (2023) Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023.  Knowledge distillation of large language models.  _arXiv preprint arXiv:2306.08543_.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Kim and Rush (2016) Yoon Kim and Alexander M Rush. 2016.  Sequence-level knowledge distillation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1317–1327.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kung and Peng (2023) Po-Nien Kung and Nanyun Peng. 2023.  Do models really learn to follow instructions? an empirical study of instruction tuning.  _arXiv preprint arXiv:2305.11383_.
  * Levy et al. (2023) Itay Levy, Ben Bogin, and Jonathan Berant. 2023.  Diverse demonstrations improve in-context compositional generalization.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1401–1422, Toronto, Canada. Association for Computational Linguistics.
  * Li et al. (2023) Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023.  Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness.  _arXiv preprint arXiv:2304.11633_.
  * Liang et al. (2023) Shihao Liang, Kunlun Zhu, Runchu Tian, Yujia Qin, Huadong Wang, Xin Cong, Zhiyuan Liu, Xiaojiang Liu, and Maosong Sun. 2023.  Exploring format consistency for instruction tuning.  _arXiv preprint arXiv:2307.15504_.
  * Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022.  What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
  * Liu and Low (2023) Tiedong Liu and Bryan Kian Hsiang Low. 2023.  Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks.  _arXiv preprint arXiv:2305.14201_.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023.  The flan collection: Designing data and methods for effective instruction tuning.  _arXiv preprint arXiv:2301.13688_.
  * Luo et al. (2023a) Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023a.  Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.  _arXiv preprint arXiv:2308.09583_.
  * Luo et al. (2023b) Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023b.  An empirical study of catastrophic forgetting in large language models during continual fine-tuning.  _arXiv preprint arXiv:2308.08747_.
  * Luo et al. (2023c) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023c.  Wizardcoder: Empowering code large language models with evol-instruct.  _arXiv preprint arXiv:2306.08568_.
  * Min et al. (2022a) Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022a.  Noisy channel language model prompting for few-shot text classification.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5316–5330, Dublin, Ireland. Association for Computational Linguistics.
  * Min et al. (2022b) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022b.  Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.  Cross-task generalization via natural language crowdsourcing instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3470–3487.
  * OpenAI (2022) OpenAI. 2022.  ChatGPT: Optimizing Language Models for Dialogue.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _arXiv_ , pages 2303–08774.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022.  Multitask prompted training enables zero-shot task generalization.  In _ICLR 2022-Tenth International Conference on Learning Representations_.
  * Sun et al. (2019) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019.  Patient knowledge distillation for bert model compression.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4323–4332.
  * Sun et al. (2023) Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023.  Text classification via large language models.  _arXiv preprint arXiv:2305.08377_.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Stanford alpaca: an instruction-following llama model (2023).  _URL https://crfm. stanford. edu/2023/03/13/alpaca. html_ , 1(2):3.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Von Oswald et al. (2023) Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. 2023.  Transformers learn in-context by gradient descent.  In _International Conference on Machine Learning_ , pages 35151–35174. PMLR.
  * Wan et al. (2023) Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023.  Poisoning language models during instruction tuning.  _arXiv preprint arXiv:2305.00944_.
  * Wang et al. (2023) Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023.  Label words are anchors: An information flow perspective for understanding in-context learning.  _arXiv preprint arXiv:2305.14160_.
  * Wang et al. (2022a) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022a.  Self-instruct: Aligning language model with self generated instructions.  _arXiv preprint arXiv:2212.10560_.
  * Wang et al. (2022b) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. 2022b.  Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5085–5109.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Wei et al. (2023) Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023.  Zero-shot information extraction via chatting with chatgpt.  _arXiv preprint arXiv:2302.10205_.
  * Wu et al. (2023) Xuansheng Wu, Wenlin Yao, Jianshu Chen, Xiaoman Pan, Xiaoyang Wang, Ninghao Liu, and Dong Yu. 2023.  From language modeling to instruction following: Understanding the behavior shift in llms after instruction tuning.  _arXiv preprint arXiv:2310.00492_.
  * Yan et al. (2023) Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. 2023.  Backdooring instruction-tuned large language models with virtual prompt injection.  In _NeurIPS 2023 Workshop on Backdoors in Deep Learning-The Good, the Bad, and the Ugly_.
  * Yoo et al. (2022) Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Taeuk Kim. 2022.  Ground-truth labels matter: A deeper look into input-label demonstrations.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2422–2437, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhang et al. (2023a) Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023a.  Instruction tuning for large language models: A survey.  _arXiv preprint arXiv:2308.10792_.
  * Zhang et al. (2023b) Zihan Zhang, Meng Fang, Ling Chen, and Mohammad-Reza Namazi-Rad. 2023b.  Citb: A benchmark for continual instruction tuning.  _arXiv preprint arXiv:2310.14510_.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023.  Lima: Less is more for alignment.  _arXiv preprint arXiv:2305.11206_.