# Knowledge Prompts: Injecting World Knowledge into
Language Models through Soft Prompts

Cicero Nogueira dos Santos, Zhe Dong, Daniel Cer, John Nham,
Siamak Shakeri, Jianmo Ni, Yun-hsuan Sung
Google Research
$\\{$cicerons, zhedong, cer, jnham, siamaks, jianmon<EMAIL_ADDRESS>

###### Abstract

Soft prompts have been recently proposed as a tool for adapting large frozen
language models (LMs) to new tasks. In this work, we repurpose soft prompts to
the task of injecting world knowledge into LMs. We introduce a method to train
soft prompts via self-supervised learning on data from knowledge bases. The
resulting soft knowledge prompts (KPs) are task independent and work as an
external memory of the LMs. We perform qualitative and quantitative
experiments and demonstrate that: (1) KPs can effectively model the structure
of the training data; (2) KPs can be used to improve the performance of LMs in
different knowledge intensive tasks.

##  1 Introduction

Very large neural language models (LMs) are known to perform well on knowledge
intensive natural language understanding (NLU) tasks, because they memorize a
significant amount of world knowledge from the training data. The larger the
LM, the more facts it can memorize at the training time, and the better the
results at the inference time Roberts et al. (2020). Despite their success,
these models also present some important drawbacks such as: the parametric
memory of these models have a fixed size and cannot grow (or shrink) over time
without fully retraining the model; there is no control in terms of which part
of the memory stores data about what; facts that do not co-occur frequently in
the training data are not well represented in the model; very large models are
needed to memorize enough data in order to perform well on knowledge intensive
tasks such as generative question answering; and at last, but not the least,
the memorized knowledge gets obsolete overtime, and requires re-training the
model for refreshness.

In this work, we employ soft prompts to overcome some of these issues of LMs.
_Soft prompts_ Lester et al. (2021); Li and Liang (2021); Hambardzumyan et al.
(2021) have been recently proposed as a tool for adapting large frozen LMs to
new tasks. Nevertheless, we repurpose soft prompts to the task of injecting
world knowledge into LMs. The goal is to train an external memory that is
composed of a large set of soft prompts that encode world knowledge. We
introduce a method to train knowledge driven soft prompts via self-supervised
learning on data from knowledge bases. The resulting soft prompts, which we
call _knowledge prompts_ (KPs), function as an auxiliary memory of the LMs
that is activated when solving knowledge intensive tasks. Different from
regular applications of soft prompts that concatenate a fixed small set of
embeddings to every input, our approach learns a very large set of KPs, which
are sparsely activated depending on the input.

We focus on entity-centric KPs, which means that each prompt primarily encodes
information about one entity from a knowledge base. We use Wikidata Vrandečić
and Krötzsch (2014) triples as our training data and train KPs for the top
1.1M entities, based on the number of triples. We present a qualitative
analysis of KPs using t-SNE plots and k-nearest neighbors approaches. In terms
of quantitative analysis, we show experimental results for three knowledge
intensive tasks: question answering, fact checking and relation
classification. For all datasets, the use of KPs improves the performance of
the T5 baseline. Our experimental results demonstrate that KPs are an
effective way to expand the memory of frozen LMs.

Figure 1: Training of Knowledge Prompts: given a serialized KB triple where
one of the entities has been masked out, the frozen LM has to predict the
masked entity given the input and the knowledge prompt of the non-masked
entity, which is _Michelle Obama_ in the example. The cross-entropy loss is
computed and the error is back-propagated through the frozen LM in order to
update the KP.

The main contributions of this work are the following:

  * •

we propose a self-supervised approach to train knowledge driven soft prompts
that can be used to inject world knowledge into LMs.

  * •

we demonstrate that knowledge prompts can effectively model the structure of
the training data and can also improve the performance of LMs on knowledge
intensive tasks.

  * •

this work sheds light on the usability of soft prompts for storing data rather
than storing instructions on how to solve specific tasks.

##  2 Methods

###  2.1 Soft Prompts

Different approaches have been recently proposed to train soft prompts Lester
et al. (2021); Li and Liang (2021); Hambardzumyan et al. (2021). One of the
most popular methods, and probably the simplest one, consists of the following
steps (Lester et al., 2021):

  * (1)

for a task in the dataset, prepend a fixed number of embeddings (soft prompts)
to the word embeddings of every input;

  * (2)

during finetuning, update the soft prompt while keeping all the other
parameters of the LM frozen.

Despite its simplicity, this approach has demonstrated to be very effective
when used with large language models.

###  2.2 Soft Knowledge Prompts

We are interested in training _soft knowledge prompts_ (KPs) to encode world
knowledge, which could work as an external memory for LMs. In this work, we
focus on the training of _entity-centric KPs_ , each of which stores the
knowledge related to a specific entity from a knowledge base (KB). In other
words, the KP of an entity encodes information from the KB triples that
mention the entity either as a subject or an object. We adopt KB triples from
Wikidata Vrandečić and Krötzsch (2014), as a simple and trustworthy source of
world knowledge.

####  2.2.1 KP Training

We train KPs with a masked language modeling (MLM) objective (Devlin et al.,
2019; Taylor, 1953), where the goal is to generate the object entity of a KB
triple given the subject entity and relation, and vice versa. As an example,
the input / target pair "Germany capital <MASK>" / "Berlin" will be used to
update the KP for Germany, while the pair "<MASK> capital Berlin" / "Germany"
will be used to update the KP for Berlin.

The KPs are randomly initialized, and are updated only when the corresponding
entities appear (not masked) in the input. This makes the training of KPs
sparse and parallelizable.

Given an input triple with the object entity being masked, a training
iteration has the following steps:

  * (1)

retrieve the KP of the subject entity, which is a simple lookup operation;

  * (2)

concatenate the KP to the sequence of word embeddings of the input text;

  * (3)

predict the object entity name and compute the cross-entropy loss;

  * (4)

back-propagate the error through the frozen LM to the KP, and update the KP
using stochastic gradient descent;

Figure 1 illustrates our proposed method for training knowledge prompts.
Notice that this method is general, and can be used with any textual input as
long as the entities of interest are already identified/linked.

####  2.2.2 Using KPs in Downstream Tasks

Using KPs during the finetuning of the LM is straightforward. Given the input
sequence, e.g. a question, the relevant KPs are retrieved and concatenated to
the word embeddings of the input, to generate or predict the answer. At the
finetuning time, we freeze the KPs and only update the parameters of the LM,
as the KPs are used as pre-trained external knowledge. One can also use task
specific soft prompts instead of finetuning the parameters of the LM as in
Lester et al. (2021), however in this work we focused on finetuning the LM as
we use relatively small LMs.

Retrieving KPs that are relevant to the input sequence is crucial for good
performance in the downstream task. KPs are useful only if they contain the
knowledge that is helpful to solve the input at hand. In this work we employ
entity linking as a way to retrieve relevant KPs during training/inference for
downstream tasks. Given an input, we first perform entity linking to identify
the entities mentioned in the input. Then, we simply do a lookup operation to
retrieve the KPs of the identified entities.

####  2.2.3 Why Knowledge Prompts?

Some advantages of the proposed approach include:

  * •

It allows a better control of what information is stored, by choosing what
examples are used to train the KPs.

  * •

KPs are trained independently, therefore the training can be massively
parallelized.

  * •

As the LM is kept frozen during the training of KPs, we do not mess up with
the language generation/understanding capabilities of the LM.

  * •

KPs can increase the capacity of small LMs in a dynamic way. We can add/remove
KPs at any time. Moreover, if information about a single entity changes, we
can update that entity’s KP without changing other KPs. This addresses the
freshness issue of the LMs.

##  3 Related Work

Most works on soft prompting focus on the problem of learning a set of
embeddings that can _reprogram_ a frozen LM to solve a specific task Lester et
al. (2021); Li and Liang (2021); Hambardzumyan et al. (2021). Some key aspects
that distinct KPs from these approaches include: (1) in KPs, the goal is to
create an external memory that stores world knowledge. (2) while traditional
soft prompt approaches learn a small set of embeddings our approach learns a
very large set of KPs, which are sparsely accessed by the LM. (3) different
from regular soft prompts, KPs are not task-specific.

Our work is related to the recent body of work on memory augmented LMs. The
most related approaches are the Entities as Experts (EaE) model proposed by
Févry et al. (2020) and the Fact Injected Language Model (FILM) proposed by
Verga et al. (2021). In both papers the authors present methods to train
entity embeddings during the pretraining of the LM. During inference time, the
model uses a separate classification head to identify entities, which are then
used to retrieve entity embeddings that are merged to the current entity
representation. de Jong et al. (2022) proposed the TOME model, which employs
embeddings in a similar way to EaE but with a much larger granularity. Instead
of modeling entities, their method embeds entity mentions, which results in a
very large external memory. In their experiments, they use up to 150M entries
in the external memory while EaE, FILM and our method use about 1M entries
only. Compared to these three models, our proposed approach trains an external
LM memory in a quite different way. Our method uses frozen pretrained LMs
while those methods train the external memory and the LMs together. Our method
concatenates the relevant memory entry to the input as if they were additional
word embeddings, while the other three approaches merge the relevant memory
entries to the contextual embedding of the identified entities. Additionally,
while EaE, FILM and TOME adapt the memory for each new task, our approach uses
the exact same external memory for different tasks.

This work is also related to multiple recent papers on integrating knowledge
bases into LMs Zhang et al. (2019); Peters et al. (2019); Poerner et al.
(2020); Sun et al. (2020); Wang et al. (2021); Agarwal et al. (2021). On of
the key differences between KPs and all these methods is the use of soft
prompts to integrate knowledge bases to frozen LMs.

##  4 Experimental Setup

###  4.1 KP Training Data

We adopt Wikidata triples Vrandečić and Krötzsch (2014) as our source of data
to train KPs. We start with the set of 45M triples that was previously
preprocessed by Agarwal et al. (2021). Next, we filter out triples whose
subject entity appears less than 12 times as subject entity in the dataset.
This results in a set of 23M triples containing 1.1M distinct subject
entities, which form our entity vocabulary and, respectively, the number of
KPs in our experiments.

###  4.2 KP Training Setup

We adopt the T5.1.1 model family Raffel et al. (2020) and perform experiments
with three model sizes: small, base and large, which contain 60M, 220M and
770M parameters, respectively. We use the T5.1.1 checkpoints that Lester et
al. (2021) adapted from the original T5.1.1 checkpoints by running an
additional 100K training steps using the “LM” objective discussed in Raffel et
al. (2020). Just like reported by Lester et al. (2021), we also noticed that
these adapted checkpoints make the training of soft prompts easier. Although
we use an encoder-decoder LM, our approach is not limited to this type of
architecture and can be used with encoder only models like BERT Devlin et al.
(2019) or decoder only models like GPT2 Radford et al. (2019).

The input length for training KPs is normally short because our examples are
masked serialized triples (concatenation of Subject/Object entity and a
relation). Therefore, we set the input length to 64, which allows us to use
very large batch sizes: between 4K and 8K, depending on the model size. Note
that the objective when training KPs is to memorize the training data. Hence,
we let KP training run for up to 200 epochs.

In the beginning of the training, KPs are initialized by randomly sampling
from the word embeddings matrix. This allows KPs to start from a region that
is known by the LM, which makes the training smoother and less sensitive to
hyperparameters. After training, KPs are kept frozen during LM finetuning for
downstream tasks. Therefore, for each model size, the exact same set of KPs is
used in our experiments with the different downstream tasks and datasets.

###  4.3 Entity Linking

In all experiments where KPs are used, we first preprocess the input text
using Google Cloud Natural Language API 111https://cloud.google.com/natural-
language/docs/analyzing-entities to perform entity linking.

###  4.4 Downstream Tasks

We perform experiments with three different knowledge intensive tasks: (1)
question answering (QA), (2) fact checking and (3) relation classification. In
terms of datasets, for question Answering experiments we use Entity Questions
Sciavolino et al. (2021) and TriviaQA Joshi et al. (2017) datasets. For fact
checking we use the FEVER dataset Thorne et al. (2018). For relation
classification, we use the TACRED dataset Zhang et al. (2017).

In the question answering experiments we follow the closed-book QA (CBQA)
setup of Roberts et al. (2020). In this setup, the model has no access to
external text, which means that there is no retrieval step and the model has
to solve the task using the world knowledge it acquired from pretraining and
finetuning data only. During training, we try to use the default
hyperparameters as much as possible except for the learning rate, which is
finetuned on the development sets. Following previous works we use exact
matching (EM) as the evaluation metric in CBQA.

##  5 Results and Discussion

Figure 2: t-SNE visualization of 100K KPs trained with T5-Base model on
Wikidata. KPs form well separated clusters whose member entities are very
similar in terms of their properties and the relations they belong to.

.

###  5.1 Qualitative Results

We perform a qualitative assessment of KPs through different experiments
including t-SNE visualizations, analysis of entity similarity in KP space and
evaluation of KPs for QA when golden entity linking is provided.

####  5.1.1 t-SNE visualization of KPs

One of the main goals in our qualitative assessment of KPs is to check whether
the learned KPs can model the structure of the training data. An approach that
can give us some clue about the data structure learning aspect are t-SNE
visualizations van der Maaten and Hinton (2008). In Fig. 2, we show a t-SNE
visualization of 100K randomly selected KPs that were trained using T5-BASE
model on Wikidata triples. We can see in Fig. 2 that KPs form multiple well
separated clusters. Zooming in into these clusters we can notice that they are
very coherent. There are clusters about companies, celestial objects, movies,
locations, etc. This is a good indication that, although trained
independently, KPs encapsulate a notion of similarity between entities that
aligns well with the KB structure.

####  5.1.2 k-Nearest Neighbors in KP Space

We investigate further the quality of the entity similarity captured by KPs
using cosine similarity in the KP space to retrieve the k-nearest neighbors of
different entities. In Table 1 we show the top four neighbours of four
different entities. We present results for T5 models of three different sizes.
The top two entities (_Barack Obama_ and _Roger Waters_) are cherry picked
popular entities to make it easier for the reader to grasp the quality of the
results. The bottom two entities (_Fairmont station_ and _Iacobeni , Sibiu_)
were randomly selected. The KP space learned by the three model sizes can
produce high quality nearest neighbors for the four different entities. For
instance, in the case of the search entity _Fairmont station_ , which is a
streetcar stop in Salt Lake City - Utah - USA, all the retrieve entities are
also streetcar stops in Salt Lake City. Similar results can be seen for the
other entities, where the retrieved neighbors share multiple properties (e.g.
same occupation and nationality) with the search entity.

Search Entity | Top 4 neighbors in KP space
---|---
| Small | Base | Large
Barack Obama | Donald Trump | Bill Clinton | Richard Nixon
George W. Bush | Donald Trump | Donald Trump
Michelle Obama | Ronald Reagan | Michelle Obama
| Jimmy Carter | Jimmy Carter | Harrison Ford
Roger Waters | George Harrison | David Gilmour | Tom Waits
Pink Floyd | Syd Barrett | Freddie Mercury
Kris Kristofferson | Eddie Vedder | David Gilmour
| James Hetfield | Brian Wilson | Cliff Edwards
Fairmont station | 500 East station | 700 East station | 700 East station
300 East station | 300 East station | Historic Gardner station
Sugarmont station | 500 East station | 300 East station
| 700 East station | Fairpark station | Decker Lake station
Iacobeni , Sibiu | Pălatca | Mica , Cluj | Sic , Cluj
Racovita , Timis | Gârbău , Cluj | Gârbău , Cluj
Movila , Ialomita | Aiton , Cluj | Mihăileni , Sibiu
| Mosna , Sibiu | Râsca , Cluj | Mica , Cluj
Table 1: Nearest neighbors in the knowledge prompt space for different search entities and model sizes. The top two entities were cherry picked while the two bottom ones were randomly selected. Model | Simple Questions
---|---
| Zero-shot Learning | Finetuning
| no KPs | KPs | no KPs | KPs
T5-SMALL | 0.1 | 4.3 | 32.9 | 54.1
T5-BASE | 0.0 | 8.8 | 35.1 | 58.9
T5-LARGE | 0.0 | 8.8 | 36.6 | 58.3
Table 2: Comparing LMs performances on Simple Questions Bordes et al. (2015)
with and without KPs as a knowledge source, measured by exact match (EM) score
(%). The KPs are retrieved using golden entity linking information. Most
performant results for each setup is marked in bold.

####  5.1.3 KPs as a Knowledge Source for LMs

In order to assess in a controlled manner whether KPs can be used as a
knowledge source for LMs, we perform an experiment on closed-book QA using the
Simple Questions dataset Bordes et al. (2015). This dataset is a good fit for
our purpose because it contains golden information about the entity involved
in the question (golden entity linking). We use the Simple Questions version
that was processed by Diefenbach et al. (2017) to align the original Freebase
entities to Wikidata entities. We further preprocessed the dataset to contain
only questions involving one of the 1.1M entities for which we trained KPs.

In Table 2, we present two sets of experiments for models of different sizes.
In the first experiment we check whether the use of KPs can improve the
performance of the models for zero-shot learning. In this scenario, we can see
that T5 models without KPs performs very poorly and achieve exact match (EM)
score of 0 percent. The use of KPs boosts the performance of all model sizes,
with the base and large models achieving EM of 8.8. In the finetuning
scenario, the use of KPs also brings a significant boost in performance. In
particular, for T5-Base the improvement is of $\sim$24 points in EM. We know
the improvement is actually larger than this because for some questions there
are multiple good answers (e.g. songwriter and singer are valid occupations
for _John Lennon_), but the Simple Questions dataset list a single answer
only.

These experimental results indicate that KPs are an effective approach to
store information about entities in a way that can be readily used by the LM
without any adaptation of the KPs for the downstream QA task.

###  5.2 Quantitative Results

Model | Entity Questions | TriviaQA
---|---|---
| Dev | Test | Dev | Test
| no KPs | KPs | no KP | KPs | no KPs | KPs | no KP | KPs
T5-SMALL | 23.3 | 30.6 | 23.6 | 30.8 | 17.4 | 19.4 | 21.4 | 23.8
T5-BASE | 24.5 | 33.3 | 25.4 | 33.1 | 22.7 | 24.9 | 27.1 | 28.1
T5-LARGE | 26.8 | 33.8 | 26.7 | 33.9 | 28.3 | 28.9 | 32.0 | 32.5
Table 3: Comparing LMs performances on EntityQuestions (Sciavolino et al.,
2021) and TriviaQA (Joshi et al., 2017) with and without KPs as a knowledge
source, measured by exact match (EM) score (%).

####  5.2.1 Closed-book Question Answering

Table 3 presents experimental results for two closed-book QA (CBQA) datasets
and different T5 model sizes. KPs provide a significant performance
improvement on the Entity Questions dataset, which is an entity-centric QA
task. For instance, the improvement for T5-Base is of 7.7 points in EM.
Interestingly, T5-SMALL + KPs outperforms T5-LARGE model by a large margin,
30.8 vs 26.7, respectively. Although to a smaller extent, KPs also bring
performance improvements for all model sizes on TriviaQA dataset.

In Table 4 we compare the performance of T5-Base/Large + KPs with other
recently published results on CBQA for TriviaQA. To the best of our knowledge,
there is no previous work that reports CBQA results for the Entity Questions
dataset. T5 + KPs model does not perform as well as the other CBQA approaches
on TriviaQA. We believe this is mainly due to the following factors: (1) in
EaE, TOME and T5-11B+SSM, the full LM is heavily pretrained on Wikipedia text
using entity focused objective functions, which is known to make the model
very effective to QA Guu et al. (2020). In our method, we use an entity
focused objective to train KPs only while the LM is frozen. Note in Table 3,
no KPs column, that our initial baseline is poor. (2) Models like EaE and TOME
update their external memory component during finetuning for a new task. We
keep KPs frozen when finetuning for a new task. (3) Our model generates the
answer token by token, which is more prone to errors compared to the entity
retrieval approaches used in EaE and TOME. (4) We conjecture that the
additional pretraining steps performed by Lester et al. (2021) might hurt the
performance for CBQA.

Model | TriviaQA
---|---
| Dev | Test
T5-BASE + KPs | 24.9 | 28.1
T5-LARGE + KPs | 28.9 | 32.5
BERT-base Févry et al. (2020) | 28.9 |
EaE Févry et al. (2020) | 43.2 | 53.4
T5-11B+SSM Roberts et al. (2020) | 51.0 | 60.5
TOME-2 de Jong et al. (2022) | 54.6 | 65.8
Table 4: Comparison with other closed-book QA approaches for TriviaQA.

####  5.2.2 Fact Checking

We present experimental results for the fact checking task in Table 5. The use
of KPs brings significant improvements for the three model sizes on the FEVER
dataset. Compared to recent works that use LMs with external memory, our
T5-Base + KPs model has similar performance to EaE (Févry et al., 2020), and
T5-LARGE + KPs achieves results competitive to TOME-2 (de Jong et al., 2022)
model. TOME-2 achieves better results than EaE and T5 + KPs because of the
granularity of its memory. While TOME-2 has an external memory with 150M
entries that store fine-grained information about entities, both EaE and our
model have a memory with about 1M entries only. Our KP training method allows
to increase the granularity of KPs in a straightforward manner. For instance,
one can use multiple KPs per entity, where each KP is trained using a subset
of the triples that mention the entity. We leave the investigation on multiple
KPs per entity as a future work.

Model | FEVER
---|---
| no KPs | KPs
T5-SMALL | 60.2 | 61.4
T5-BASE | 61.3 | 63.4
T5-LARGE | 63.0 | 65.2
EaE (Févry et al., 2020) | 63.6
TOME-2 (de Jong et al., 2022) | 68.1
Table 5: Comparing LMs performances on fact checking dataset, FEVER (test
split) (Thorne et al., 2018), with and without KPs as a knowledge source,
measured by accuracy (%). Two baselines, Entities-as-Experts (EaE, Févry et
al. (2020)) and MentionMemory (TOME-2, de Jong et al. (2022)), are included.

####  5.2.3 Relation Classification

Table 6 presents experimental results for the relation classification task
using the original TACRED dataset. Following previous papers, we use F1 as the
metric and report results for the test set. Similar to the other two tasks, in
relation classification KPs also provide performance improvements for all
three model sizes. Interestingly, T5-Base + KPs outperform T5-LARGE by almost
one point in F1.

Model | TACRED
---|---
| no KPs | KPs
T5-SMALL | 64.3 | 66.1
T5-BASE | 68.3 | 70.0
T5-LARGE | 69.1 | 69.8
EaE Févry et al. (2020) | 70.2
KnowBERT Peters et al. (2019) | 71.5
Table 6: Experimental results on relation classification.

Compared to previous papers that use knowledge augmented approaches, T5-Base +
KPs achieves performance similar to EaE and is competitive with KnowBERT. It
is important to note that KnowBERT uses entity types as input while both EaE
and our method do not use that additional information.

###  5.3 Ablation Experiments

####  5.3.1 KP$\rightarrow$Encoder vs. KP$\rightarrow$Decoder

The use of Encoder-Decoder LM gives us the flexibility to introduce KPs at
either encoder or decoder. In all experiments presented so far in the paper,
KPs are concatenated to the word embeddings of the input sequence and given as
input to the encoder (KP$\rightarrow$Encoder). However, one can instead
concatenate KPs to the output of the encoder, which makes KPs accessible only
by the decoder via cross-attention (KP$\rightarrow$Decoder). In Table 7 we
present a comparison of the results of KP$\rightarrow$Encoder and
KP$\rightarrow$Decoder for the QA datasets. In both cases, KPs were trained
using T5-Base model and we report results for the dev set. We believe
KP$\rightarrow$Encoder achieves better performance because it allows
interaction (self-attention) between input and KPs in the encoder, which gives
the model more opportunity to select and extract the correct information from
KPs. On the other hand, the advantage of KP$\rightarrow$Decoder is that its
training is faster because it is a simpler optimization problem as the error
does not have to be back-propagated through the frozen encoder and KPs are
used via cross-attention in the decoder only. KP$\rightarrow$Decoder requires
3x less training iterations to converge compared to KP$\rightarrow$Enc.

Dataset | KP$\rightarrow$Enc | KP$\rightarrow$Dec
---|---|---
Simple Questions | 58.9 | 57.7
Entity Questions | 33.3 | 32.8
TriviaQA | 24.9 | 24.2
Table 7: Comparison of models that inject KPs into encoder
(KP$\rightarrow$Enc) vs models that inject KPs into decoder
(KP$\rightarrow$Dec).

####  5.3.2 Entity Linking vs. Searching in KP Space

Retrieving relevant KPs given an input is a fundamental task that has direct
impact on the usefulness of KPs. Beyond entity linking, another approach to
retrieve KPs is to transform the input into a single dense vector, then search
for the most similar vectors in the KP space. We have experimented with this
strategy by training an external encoder that creates a vector representation
of the input. The external encoder has the same architecture and size of the
respective T5 model. We use the TEKGEN dataset Agarwal et al. (2021), which
contains Wikipedia sentences mapped to Wikidata triples, as a source of noisy-
labeled data to train the encoder via contrastive loss. KPs are kept frozen
during the training of the input encoder.

Table 8 presents a comparison between T5 + KP models that use either entity
linking or search in the KP space. The results were computed on TriviaQA. We
can see in the results that entity linking performs better for both model
sizes. We conjecture that searching in KP space does not work well because KPs
are not optimized to be used in search. KPs are trained to memorize knowledge
in a way that can later be extracted by the LM.

Model | Ent. Linking | KP Search
---|---|---
T5-SMALL+KPs | 19.4 | 17.9
T5-BASE+KPs | 24.9 | 23.2
Table 8: Evaluation of the LM performances on TriviaQA with KPs retrieved with
Entity Linking and Searching in KP space.

##  6 Conclusions

We present a new method for training soft prompts that can be used to extend
the world knowledge of LMs. We demonstrate the generality and usefulness of
the resulting KPs by employing the same set of KP to improve the LM
performance in three different tasks: question answering, fact checking, and
relation classification. Although in this work we focused on the use of KPs
for injecting knowledge into LMs, we believe that entity-centric KPs can be
seen as a general purpose knowledge base embedding approach. We leave the
investigation of KPs as a general KB embebedding approach for future work.

## References

  * Agarwal et al. (2021) Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021.  Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3554–3565, Online. Association for Computational Linguistics.
  * Bordes et al. (2015) Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015.  Large-scale simple question answering with memory networks.  _CoRR_ , abs/1506.02075.
  * de Jong et al. (2022) Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Fei Sha, and William W. Cohen. 2022.  Mention memory: incorporating textual knowledge into transformers through entity mention attention.  In _International Conference on Learning Representations_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Diefenbach et al. (2017) Dennis Diefenbach, Thomas Pellissier Tanon, Kamal Deep Singh, and Pierre Maret. 2017\.  Question answering benchmarks for wikidata.  In _Proceedings of the ISWC 2017 Posters & Demonstrations and Industry Tracks co-located with 16th International Semantic Web Conference (ISWC 2017), Vienna, Austria, October 23rd - to - 25th, 2017._
  * Févry et al. (2020) Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020.  Entities as experts: Sparse memory access with entity supervision.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4937–4951, Online. Association for Computational Linguistics.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  REALM: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_.
  * Hambardzumyan et al. (2021) Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021.  WARP: Word-level Adversarial ReProgramming.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4921–4933, Online. Association for Computational Linguistics.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Peters et al. (2019) Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019.  Knowledge enhanced contextual word representations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 43–54, Hong Kong, China. Association for Computational Linguistics.
  * Poerner et al. (2020) Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020.  E-BERT: Efficient-yet-effective entity embeddings for BERT.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 803–818, Online. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Sciavolino et al. (2021) Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021.  Simple entity-centric questions challenge dense retrievers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6138–6148, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Sun et al. (2020) Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. 2020.  CoLAKE: Contextualized language and knowledge embedding.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 3660–3670, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Taylor (1953) Wilson L. Taylor. 1953.  “Cloze Procedure”: A new tool for measuring readability.  _Journalism & Mass Communication Quarterly_, 30:415 – 433.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-sne.  _Journal of Machine Learning Research_ , 9(86):2579–2605.
  * Verga et al. (2021) Pat Verga, Haitian Sun, Livio Baldini Soares, and William Cohen. 2021.  Adaptable and interpretable neural MemoryOver symbolic knowledge.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3678–3691, Online. Association for Computational Linguistics.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: A free collaborative knowledgebase.  _Commun. ACM_ , 57(10):78–85.
  * Wang et al. (2021) Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021.  KEPLER: A unified model for knowledge embedding and pre-trained language representation.  _Transactions of the Association for Computational Linguistics_ , 9:176–194.
  * Zhang et al. (2017) Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017.  Position-aware attention and supervised data improve slot filling.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 35–45, Copenhagen, Denmark. Association for Computational Linguistics.
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019.  ERNIE: Enhanced language representation with informative entities.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1441–1451, Florence, Italy. Association for Computational Linguistics.