# Understanding Finetuning for Factual Knowledge Extraction from Language
Models

Mehran Kazemi<EMAIL_ADDRESS>
Sid Mittal<EMAIL_ADDRESS>
Deepak Ramachandran<EMAIL_ADDRESS>
Google Research

###### Abstract

Language models (LMs) pretrained on large corpora of text from the web have
been observed to contain large amounts of various types of knowledge about the
world. This observation has led to a new and exciting paradigm in knowledge
graph construction where, instead of manual curation or text mining, one
extracts knowledge from the parameters of an LM. Recently, it has been shown
that finetuning LMs on a set of factual knowledge makes them produce better
answers to queries from a different set, thus making finetuned LMs a good
candidate for knowledge extraction and, consequently, knowledge graph
construction. In this paper, we analyze finetuned LMs for factual knowledge
extraction. We show that along with its previously known positive effects,
finetuning also leads to a (potentially harmful) phenomenon which we call
Frequency Shock, where at the test time the model over-predicts rare entities
that appear in the training set and under-predicts common entities that do not
appear in the training set enough times. We show that Frequency Shock leads to
a degradation in the predictions of the model and beyond a point, the harm
from Frequency Shock can even outweigh the positive effects of finetuning,
making finetuning harmful overall. We then consider two solutions to remedy
the identified negative effect: 1- model mixing and 2- mixture finetuning with
the LM’s pre-training task. The two solutions combined lead to significant
improvements compared to vanilla finetuning.

##  1 Introduction

Recently, Language Models (LMs) pre-trained on large corpora of web documents
such as CommonCrawl111http://commoncrawl.org have achieved impressive results
on multiple NLP tasks. In their pioneering work, Petroni et al. (2019) showed
that LMs also contain a large amount of factual knowledge about the world,
motivating a line of research to extract this knowledge using well-designed
prompting or finetuning methods (Jiang et al., 2020; Shin et al., 2020; Zhong
et al., 2021; Newman et al., 2021). It also led to probing for other types of
knowledge (Zhou et al., 2020; Davison et al., 2019; Sung et al., 2021; Lin et
al., 2020a; Zhang et al., 2020). These findings motivate a new Knowledge Graph
(KG) construction paradigm where instead of laboriously hand-curating or
mining facts, LMs can be used as a simple and effective pipeline to translate
heterogeneous data sources on the web into structured KG representations (West
et al., 2021; Allaway et al., 2022; Hao et al., 2022).

Fichtel et al. (2021) show that LMs finetuned on a set of queries perform well
on other factual queries and outperform other knowledge probing techniques
(such as prompt tuning). Some recent work (Zhong et al., 2021; Cao et al.,
2021) however, casts doubt on previous findings by showing that when finetuned
on in-distribution data (data that follows the same distribution as the test
data), there are statistical patterns in training that can be exploited by a
model leading to over-estimation of the test performance of LMs. Moreover,
Wallat et al. (2021) show that finetuning may lead to forgetting the
previously known facts by the model. Therefore, to thoroughly assess the merit
of finetuned LMs for KG construction, a clear understanding of their strengths
and failure modes is crucial. These results raise the question of whether for
constructing KGs from LMs, using a finetuned LM is a good strategy? Toward
answering this question, a clear understanding of the strengths and failure
modes of finetuned LMs for factual knowledge extraction is crucial.

Figure 1: For the query _“Marat Makhmutov was born in [MASK] .”_ , a pre-
trained language model correctly returns _“Moscow”_ as answer. Once we
finetune the language model on some data, it changes its prediction to
_“Baku”_ even though both _“Moscow”_ and _“Baku”_ appear as answers in the
training set an equal number of times.

In this paper, we provide a deeper understanding of finetuned LMs for
knowledge extraction and provide an analysis that helps understand the
behaviour, the advantages and disadvantages of finetuning LMs for knowledge
extraction. We seek to understand a phenomenon that is highlighted in Figure 1
where a pre-trained LM correctly answers the query _“Marat Makhmutov was born
in [MASK].”_ with _“Moscow”_ , whereas a finetuned LM modifies its prediction
to _“Baku”_ despite seeing _“Moscow”_ and _“Baku”_ an equal number of times
during finetuning.

We identify three effects of finetuning (the first one already explicated, but
the other two less understood):

  * •

Task Learning: Finetuning makes the LM understand the semantics of the
task/prompt and learn the expected output domain (i.e. the expected entity
types/subtypes) for each relation type,

  * •

Frequency Shock: Finetuning biases the model’s predictions towards the
frequency statistics of the entities seen as answers during finetuning. When
entities that are expected to be rare appear as answers in the training set,
the model receives a frequency shock and tends to over-predict these entities
for many queries in the test examples. Moreover, when entities that are
expected to be common do not appear in the dataset enough times, the model
receives a frequency shock and tends to under-predict these entities for the
queries in the test examples,

  * •

Range Shift: Finetuning makes the model mostly predict entities from those
seen as answers during finetuning (this could be considered as a specific case
of Frequency Shock).

While previous work typically explains the phenomanon in Figure 1 as
_forgetting effect_ (Wallat et al., 2021), our study reveals a more nuanced
explanation in terms of Frequency Shock: even though both _“Moscow”_ and
_“Baku”_ have been observed an equal number of times in the training set,
since _“Baku”_ is expected to be a less common entity222As an example in the
LAMA probe, which is a natural subset of a large real-world knowledge graph,
_“Baku”_ appears only $4$ times as answer whereas _“Moscow”_ appears $13$
times and a more common entity such as _“London”_ appears $59$ times.  and
hence less observed during the pre-training of the language model, the
finetuned model receives a frequency shock leading to an over-prediction of
the entity _“Baku”_ , hence corrupting an originally correct prediction. Note
that Frequency Shock and Range Shift are related to the problem of out-of-
distribution (OOD) generalization in machine learning, see section 3.6 for
more discussion.

We design careful experiments to better understand Frequency Shock and Range
Shift and show that while Task Learning may lead to improvements, Frequency
Shock and Range Shift may lead to a degradation that can even sometimes
outweigh the positive effect of Task Learning such that finetuning hurts the
overall performance. We then propose two approaches to remedy the negative
effects. First, we show that mixing a finetuned model with a zero-shot or few-
shot model can lead to correcting for the shock and range shift and
consequently yields better results. Second, we show that a version of multi-
task finetuning where we mix the knowledge extraction task with the original
pre-training task of the LM can also help alleviate the negative effect of
Frequency Shock and Range Shift and leads to better results. The two
approaches combined lead to an aggregate $12.4\%$ improvement in model
performance over the vanilla finetuning approach.

Our main contributions include: 1- Identifying Frequency Shock and Range Shift
as side-effects of finetuned LMs for factual knowledge extraction, 2- Creating
datasets for thoroughly analyzing these effects and identifying their root
causes, 3- Identifying two solutions for the side-effects leading to an
aggregate $12.4\%$ improvement in model performance, and 4- Proposing a
practical recipe for knowledge extraction from LMs.

##  2 Related Work

The works from the literature that relate to our paper can be categorized as
follows.

Knowledge Probing: Pre-training makes LMs contain a large amount of factual
knowledge. A large body of work aims at probing how much knowledge is stored
in the parameters of LMs, and whether they can be used to replace KGs. These
works include probing for factual (Petroni et al., 2019), commonsense (Zhou et
al., 2020; Davison et al., 2019; Yin et al., 2022), biomedical (Sung et al.,
2021), numerical (Lin et al., 2020a), scale (Zhang et al., 2020), and many
other types of knowledge. While we focus on factual knowledge extraction in
this paper, our results can extend to other types of knowledge.

Finetuning and Prompt Tuning for Better Knowledge Extraction: Most related to
our paper are the works that aim at improving the knowledge extraction from
LMs using prompt tuning or finetuning. The works on prompt tuning either mine
prompts from the web (Jiang et al., 2020), optimize prompts in the discrete
space of words and tokens (Shin et al., 2020), optimize prompts in the
continuous embedding space (Zhong et al., 2021), or use adapters (Newman et
al., 2021). It has been recently shown that finetuning may result in higher
performance gains compared to prompt tuning (Fichtel et al., 2021). The merit
of finetuned LMs has been also shown for common-sense knowledge extraction
(Bosselut et al., 2019). Previous work also studies the effect of dataset size
for finetuning (Wallat et al., 2021; Fichtel et al., 2021; Da et al., 2021),
but the negative effects finetuning (studied in this paper) remain unexplored.
For a full review of the literature on knowledge probing and extraction, we
refer to (Safavi & Koutra, 2021; AlKhamissi et al., 2022).

KG Construction (from LMs): Typically, KGs are either created manually (by
domain experts or through crowd-sourcing) Miller (1995); Vrandečić & Krötzsch
(2014), automatically (by extracting from the web) Dong et al. (2014); Carlson
et al. (2010); Bhakthavatsalam et al. (2020), or a combination of the two
Speer et al. (2017); Sap et al. (2019). In this paper, we are mostly
interested in an emerging line of work that constructs KGs directly from LMs
or by leveraging LMs West et al. (2021); Bosselut et al. (2019); Hao et al.
(2022); Allaway et al. (2022).

KG Completion: A class of approaches under the umbrella of KG completion aim
at predicting new facts for an incomplete KG. Approaches have been developed
for static Bordes et al. (2013); Kazemi & Poole (2018); Trouillon et al.
(2016), temporal Goel et al. (2020); Lacroix et al. (2020), commensense Li et
al. (2016), and many other types of KGs. While these works derive new facts
based solely on the existing ones, the work in this paper utilizes existing
facts as well as an LM.

Generalization in Question Answering (QA): Generalization, especially out-of-
distribution (OOD), has been a hot topic of study for various QA settings
including open-domain QA Liu et al. (2021), reading comprehension Talmor &
Berant (2019); Fisch et al. (2019), and visual QA Kervadec et al. (2021);
Gokhale et al. (2020). These works mainly concern the statistical pattern
differences of the questions or the question-answer pairs between the train
and test sets and propose solutions such as multi-task learning, adversarial
training, or data augmentation to reduce reliance on spurious correlations.
Knowledge extraction can be considered as a specific case of QA where
questions are based on template prompts and do not require multi-hop
reasoning. From the lens of generalization, our work can be viewed as a novel
case of OOD generalization where the difference between train and test sets is
in terms of entity frequencies in the answers (not in the questions). The
closest to our work is the study in Lewis et al. (2020) where generalization
is measured for novel answer entities in the test set, but our work goes
beyond that and studies Frequency Shock for non-novel entities (see, e.g.,
Figure 1).

##  3 Experimental Setup

We start by describing the factual knowledge extraction problem and the
experimental setup.

###  3.1 Factual Knowledge Extraction

Let $\mathcal{E}=\\{e_{1},\dots,e_{n}\\}$ be a set of entities and
$\mathcal{R}=\\{r_{1},\dots,r_{m}\\}$ be a set of relations. A knowledge graph
(KG) is a set of triples of the form $(e_{i},r_{j},e_{k})$ where $e_{i}$ is
the subject, $r_{j}$ is the relation, and $e_{k}$ is the object of the triple.
Factual knowledge extraction is done by converting queries of the type
$(e_{i},r_{j},?)$ into natural language queries that can be answered by an LM.
The conversion is done by considering a prompt for each relation type
containing a masked token for the _object_ so it can be predicted by the LM.
As an example, we may convert a query such as _(Barack Obama, profession, ?)_
into: _“Barack Obama is a [MASK] by profession."_. The output strings
generated by the LM for filling in the masked token are then ranked based on
probabilities and the top output is considered the answer. In our experiments,
we use the manual prompts from Petroni et al. (2019).

Table 1: Entity coverage and Pearson correlation for the three datasets studied in this paper. Dataset | Entity Coverage | Pearson
---|---|---
_LowMismatch_ | 83.8 | 0.68
_MediumMismatch_ | 95.8 | 0.30
_HighMismatch_ | 0.0 | -0.02

###  3.2 Frequency Statistics

Let $\mathcal{Q}$ be a set of factual knowledge extraction queries of the form
described in Section 3.1 and $\mathcal{Q}_{r}$ represent the subset of queries
from $\mathcal{Q}$ that concern relation $r$. Let $\mathcal{E}$ represent a
set of entities. We define the _frequency statistics_ of $\mathcal{Q}$ as a
mapping $\Phi_{\mathcal{Q}}:\mathcal{E}\rightarrow\mathbb{N}$ from any entity
$e\in\mathcal{E}$ to a number in $\mathbb{N}$ indicating how many times it
appeared as answer in $\mathcal{Q}$.

For two sets $\mathcal{Q}_{1}$ and $\mathcal{Q}_{2}$, let $\mathcal{E}_{1,2}$
represent the union of the entities that appear as answers in the two sets and
let $\tau=|\mathcal{E}_{1,2}|$ be the size of this set. We measure the
similarity between $\Phi_{\mathcal{Q}_{1}}$ and $\Phi_{\mathcal{Q}_{2}}$ using
the following two measures.
Pearson correlation is defined as follows:

$\frac{\sum_{i=1}^{\tau}(\Phi_{\mathcal{Q}_{1}}(e_{i})-\overline{\phi_{\mathcal{Q}_{1}}})(\Phi_{\mathcal{Q}_{2}}(e_{i})-\overline{\phi_{\mathcal{Q}_{2}}})}{\sqrt{\sum_{i=1}^{\tau}\Phi_{\mathcal{Q}_{1}}(e_{i})-\overline{\phi_{\mathcal{Q}_{1}}}}\sqrt{\sum_{i=1}^{\tau}\Phi_{\mathcal{Q}_{2}}(e_{i})-\overline{\phi_{\mathcal{Q}_{2}}}}},\quad\overline{\phi_{\mathcal{Q}_{1}}}=\frac{\sum_{i=1}^{\tau}\Phi_{\mathcal{Q}_{1}}(e_{i})}{\tau},\quad\overline{\phi_{\mathcal{Q}_{2}}}=\frac{\sum_{i=1}^{\tau}\Phi_{\mathcal{Q}_{2}}(e_{i})}{\tau}$

where $\overline{\phi_{\mathcal{Q}_{1}}}$ represents the average frequencies
from the first set and $\overline{\phi_{\mathcal{Q}_{2}}}$ represents the
average frequencies from the second set.
Entity coverage of $\mathcal{Q}_{2}$ with respect to $\mathcal{Q}_{1}$ is
defined as the proportion of answers for $\mathcal{Q}_{2}$ that are also the
answer to at least one query in $\mathcal{Q}_{1}$:

$\frac{|\\{e~{}|~{}\Phi_{\mathcal{Q}_{2}}(e)>0,\Phi_{\mathcal{Q}_{1}}(e)>0\\}|}{|\\{e~{}|~{}\Phi_{\mathcal{Q}_{2}}(e)>0\\}}$

Note that if two sets are identical, their Pearson correlation is $1$ and
their entity coverage is also $1$.

###  3.3 Datasets

We aim to create datasets that help us better understand the positive and
negative effects of finetuning. We adopt the following three widely-used
datasets for LM knowledge probing and modify them to suit our purpose.

  * •

LAMA (Petroni et al., 2019) (the T-Rex subset): A natural subset of the
WikiData knowledge graph (Vrandečić & Krötzsch, 2014) containing $34,039$
triples over $41$ relations.

  * •

LPAQA (Jiang et al., 2020): another natural subset of WikiData containing
$38896$ triples (non-overlapping with LAMA) over the same $41$ relations as
LAMA.

  * •

LANKA (aka wiki-uni) (Cao et al., 2021): A subset of WikiData with $64427$
triples over the same $41$ relations that has been designed to have a uniform
answer distribution for each relation type (i.e. for any two entities $e$ and
$e^{\prime}$ that appear as answers to queries for relation type $r$,
$\Phi_{\mathcal{Q}_{r}}(e)=\Phi_{\mathcal{Q}_{r}}(e^{\prime})$).

For our experiments, we create three datasets with development (i.e. train and
validation) and test sets as follows:

  * •

_LowMismatch_ : uses LPAQA as development and LAMA as test set. Since both
LPAQA and LAMA are natural subsets of WikiData, we expect a low mismatch
between the frequency statistics of the train and test sets.

  * •

_MediumMismatch_ : uses LANKA as development and LAMA as test set. Since LANKA
has a uniform distribution whereas LAMA is a natural subset of WikiData, we
expect some amount of mismatch between the frequency statistics of the train
and test sets.

  * •

_HighMismatch_ : combines all three datasets and divides the facts into two
sets such that the answers in one set are mutually exclusive from the answers
in the other set, then uses one set for development and the other set for
testing. Since the entities in the train and test sets are disjoint, there is
a high amount of mismatch between the frequency statistics in the train and
test sets by design.

The entity coverage and Pearson correlations between development/test splits
for the 3 datasets is presented in Table 1. For _LowMismatch_ both values are
high. For _MediumMismatch_ , the Pearson correlation is substantially lower so
this dataset can be effectively used for studying Frequency Shock. For
_HighMismatch_ , entity coverage is zero and Pearson correlation is close to
zero, so this dataset can be effectively used for studying both Frequency
Shock and Range Shift.

For all the datasets, we fix the development set size to $40k$ queries ($30k$
for train and $10k$ for validation). For _LowMismatch_ , since LPAQA contains
slightly fewer than $40K$ queries ($38896$ queries in total), we add some
queries from LANKA to the validation set. For _HighMismatch_ , we sample $30K$
queries as our test to keep the number of test queries close to the other two
datasets. Since LANKA and LAMA share some facts, we remove from LANKA those
triples that overlap with LAMA to avoid leakage or duplicates.

###  3.4 Model Variants Used in the Experiments

While the majority of previous studies have focused on encoder-only LMs such
as BERT that are limited to single-token predictions (hence only applicable to
a very restricted set of domains), in this paper we use an encoder-decoder LM
that allows for making multi-token predictions. In particular, unless stated
otherwise, we use the T51.1 XXL Raffel et al. (2019) (hereafter, referred to
simply as T5).

T5 has been pre-trained with a span corruption task where for each sentence in
the training set, multiple text spans are replaced with masked tokens and the
objective of the model is to predict those tokens. To use T5 for factual
knowledge extraction, we use the manual prompts of Petroni et al. (2019) to
turn a query _(subject, relation, ?)_ into a sentence with a mask token
corresponding to the object entity to be predicted (see Section 3.1). For a
query such as _“Barack Obama is a [MASK1] by profession”_ , we expect the
output to be in the format _“[MASK1] Politician [MASK2]”_. T5 may produce
extra text after _[MASK2]_. We simply ignore any text generated after that
token. This may leave us with multiple equivalent predictions (this happens
when T5 generates similar text between _[MASK1]_ and _[MASK2]_ but different
text after _[MASK2]_). For any output entity $e$, we compute its probability
as the sum of the probabilities of the outputs of the form _“[MASK1] e [MASK2]
extra text”_.

We experiment with the following model variants. Zero-shot (ZS): simply
feeding the masked query to the pretrained model. Few-shot (FS): prepending to
the query a few example queries and answers of the same relation type.
Reranking (RR): using a separate discriminatively finetuned LM to rerank the
outputs produced by a generative model has recently gained popularity Wallat
et al. (2021); Lin et al. (2020b); Yadav et al. (2021), so we also experiment
with reranking for factual knowledge extraction. We finetune a model that
learns to predict which output among the top-k outputs of a pretrained model
(ZS in our experiments) is correct in a binary classification setup. Entities
are then ranked based on the sum of the probabilities produced by the
pretrained model and the score produced by the finetuned model. Finetuning
(FT): where we finetune a model on the knowledge extraction task on the
training set before evaluating on the test set.

###  3.5 Metrics

We report the results using the widely-used _Hit@k_ metric computed as the
percentage of queries for which the correct answer is ranked among the top $k$
entities. We compute _Hit@k_ for each relation type separately and report the
macro average, following previous work.

Table 2: Performances on the three datasets (bold indicates winner). FT offers substantial gains when development and test sets have similar frequency statistics, but the gain diminishes as the gap between the frequency statistics becomes more; eventually on _HighMismatch_ , the negative side-effects outweighs the positive effects and finetuning becomes harmful overall. | _LowMismatch_ | _MediumMismatch_ | _HighMismatch_
---|---|---|---
Strategy | Hit@1 | Hit@3 | Hit@5 | Hit@1 | Hit@3 | Hit@5 | Hit@1 | Hit@3 | Hit@5
ZS | 35.2 | 47.9 | 52.7 | 35.2 | 47.9 | 52.7 | 19.5 | 28.3 | 31.7
FS | 47.0 | 56.1 | 57.2 | 42.8 | 50.9 | 52.5 | 27.0 | 33.4 | 34.9
RR | 39.9 | 49.9 | 52.7 | 38.7 | 49.1 | 52.7 | 20.5 | 28.8 | 31.7
FT | 51.9 | 68.4 | 73.9 | 43.6 | 57.8 | 63.2 | 18.0 | 27.4 | 32.4

###  3.6 Connection to Out-of-distribution Generalization

Classical machine learning settings assume train and test sets come from the
same distribution. Recently, there has been much effort in tackling more
realistic scenarios where test distributions differ from training
distributions, known as out-of-distribution (OOD) generalization (see Shen et
al. (2021) for a survey). While OOD generalization has been investigated in
many applications, it has remained largely unexplored for factual knowledge
extraction from LMs. This may be due to a lack of clarity on what a meaningful
definition of OOD is for this task; since test queries are written using the
same template as the training queries, traditional definitions are not
straightforward to apply. Lewis et al. (2020) for example takes the extreme
approach (in the context of Open-Domain Question-Answering) of defining OOD as
queries whose answers have never been seen in training.

Using frequency statistics to measure the distance between train and test sets
could be viewed as a novel formulation of OOD for factual knowledge extraction
from LMs, and the negative side-effects discussed in Section 4 and the
solutions considered in Section 5 are both relevant for robust solutions to
OOD generalization. Note, however, that the Frequency Shock phenomenon goes
beyond OOD generalization, e.g. in cases such as the example in Figure 1, the
test query could still be an in-distribution query.

##  4 Understanding Finetuning for Factual Knowledge Extraction

In this section, we design experiments that help better understand the effects
of finetuning.

Figure 2:  Macro average relative improvement of FT over ZS for different relation types in _LowMismatch_ as a function of _entity coverage_ and _Pearson correlation_ for the train and test sets. The figures show that most of the improvement comes from the relations with a high entity coverage and Pearson correlation between train and test sets. Table 3: Performance on the three datasets when using T5 1.1 small instead of XXL. | _LowMismatch_ | _MediumMismatch_ | _HighMismatch_
---|---|---|---
Strategy | Hit@1 | Hit@3 | Hit@5 | Hit@1 | Hit@3 | Hit@5 | Hit@1 | Hit@3 | Hit@5
ZS | 18.4 | 23.2 | 24.4 | 18.4 | 23.2 | 24.4 | 12.6 | 15.5 | 16.4
FT | 28.5 | 43.0 | 49.5 | 25.9 | 36.2 | 41.7 | 5.9 | 8.6 | 9.7

###  4.1 Finetuning Performance Depends on Frequency Statistics

We first compare different model variants on the _LowMismatch_ dataset.
According to the results in Table 2, FT yields a significant boost compared to
the other variants. This result is consistent with what has been already
observed in existing literature (Fichtel et al., 2021). To understand where
the improvement comes from, in Figure 2, we plot the improvement gained by the
FT model over the ZS model on the _LowMismatch_ dataset as a function of the
entity coverage and Pearson correlation between the train and test sets.
Specifically, for each relation type in the dataset, we measure the amount of
entity coverage as well as the Pearson correlation between train and test
sets, then group different relation types based on these metrics and average
the relative improvements in each group. According to Figure 2, the
improvements are mostly for those relation types that have a high entity
coverage and high Pearson correlation.

Based on the above result, we hypothesize that part of the improvement
obtained by the FT model on the _LowMismatch_ dataset is due to biasing the
pre-trained LM’s prediction frequencies toward that of the answer set of the
training data; since the train and test sets have similar frequency
statistics, the frequency bias given to the model due to finetuning matches
that of the test set and that results in some improvement. To verify this
hypothesis, we next compare FT with the other variants on the _MediumMismatch_
dataset where entity coverage is still high but Pearson correlation is low.
The results in Table 2 show that while FT still gives a boost in performance,
the gain is much lower compared to the _LowMismatch_ case. As we will show in
Section 4.2, the gap between the performance of the FT model on _LowMismatch_
and _MediumMismatch_ is mainly due to the difference in the frequency
statistics in the train and test sets: finetuning biases the entity frequency
of the LM predictions toward that of the training data but the new frequencies
do not match with that of the test set on _MediumMismatch_.

Moreover, we compare FT with the other variants on the _HighMismatch_ dataset
where both entity coverage and Pearson correlation are minimal. The results in
Table 2 show that the bias in prediction frequencies of the LM caused by
finetuning in this case even outweigh the positive effect from Task Learning
resulting in a model that actually harms the overall performance and produces
inferior results compared to the ZS model.

To verify how the above observations are affected by the scale of the LM, we
also compare the ZS and FT models on the three datasets when using the T5 1.1
Small model (60M parameters) instead of the T5 1.1 XXL (11B parameters).
According to the results in Table 3, the small model shows a similar behaviour
where FT provides a big boost on the _LowMismatch_ dataset, but the amount of
boost diminishes on _MediumMismatch_ and finetuning becomes harmful on
_HighMismatch_.

Despite the striking results obtained with finetuned LMs for factual knowledge
extraction in previous work, the collective results in Table 2 show that
(naively) finetuned LMs may not always be the best option for factual
knowledge extraction and KG construction as the performance of these models
depends heavily on the frequency statistics of the train and test sets.

###  4.2 Frequency Shock and Range Shift are (Side-)Effects of Finetuning

We now design experiments that explain the behaviour observed for the FT model
in Table 1 in terms of two side-effects: Frequency Shock and Range Shift.

Figure 3: _Common (Rare) Percentage_ corresponds to the percentage of test queries for which the model predicted an entity from the Common (Rare) set. According to the results, after finetuning on the _MediumMismatch_ dataset, the LM receives a frequency shock: it under-predicts the common entities and over-predicts the rare entities.  Table 4: Accuracy of the models for the Common and Rare entity sets for the _MediumMismatch_ dataset. Due to Frequency Shock, the FT model under-predicts the Common entities and over-predicts the Rare entities. As a result, when the FT model predicts a Common entity, there is a much higher chance of it being true compared to the other models, whereas when the FT model predicts a Rare entity, there is a much lower chance of it being true.  |  Common Accuracy |  Rare Accuracy
---|---|---
ZS | 41.2 | 47.9
FS | 51.4 | 63.8
FT | 68.5 | 14.4
FT + FS | 57.7 | 29.5
1:15 + FS | 55.6 | 64.6

We selected a set of $10$ cities that are expected to be commonly seen333We
selected the 10 cities from Cao et al. (2021) (Figure 2), namely {London,
Paris, Tokyo, Boston, Rome, Chicago, Berlin, Montreal, Moscow, Milan} as well
as a set of $10$ cities that appear as answers in LANKA but are expected to be
rarely seen in a dataset444We do this by randomly selecting $10$ cities from
the LANKA answers that appear as an answer in LAMA less than 20 times, namely
{Boise, Tirana, Myanmar, Hanover, Aberdeen, Chelsea, Kentucky, Oldham,
Hastings, Parma}. We named the two sets Common and Rare respectively. We then
measured the number of times the models generated an entity from Common and
Rare.

According to Figure 3, the ZS model predicts the Common entities frequently
and the Rare entities infrequently (this is in part due to the frequency of
the entities in the test set and in part due to the prior of the language
model). For the FS and RR models, the percentages for the two sets are not
substantially different from the ZS model. However, for the FT model, due to
the uniform distribution of of the training set of _MediumMismatch_ , the
percentages for the two sets changes substantially: the number of predictions
from Common entities drops by almost a third, and the one for Rare entities
increases by 6x. This reveals that Frequency Shock is indeed a side-effect of
finetuning as the difference between the frequency statistics of the training
set of _MediumMismatch_ and what the pre-trained model expects causes a shock
to the model and makes it over-predict Rare entities and under-predict Common
entities.

We also measured the accuracy of the FT model when it produced a Common or
Rare entity and compared it to ZS. The results are reported in Table 4. We
observe that the accuracy for Common entities increases from $41.2\%$ to
$68.5\%$ and for Rare entities decreases from $47.9\%$ to $14.4\%$. This is
because the frequency shock caused by finetuning leads the model to predict
the Common entities only when it has high confidence in its prediction, but be
less cautious about predicting the Rare entities. This result also points out
an interesting future direction for measuring the model uncertainty (or
whether the model knows what it does not know) through a combination of a ZS
model and a model that has been finetuned on a slightly different data
distribution.

Figure 4: The percentages of overlap between the entities predicted by the models and those of the train and test sets for the _HighMismatch_ dataset (entities in the train and test sets are disjoint).  Table 5: A comparison of the ZS and FT models for the “born in” relation. _NL_ , _CL_ and _IL_ stand for _Not a Location_ , _Correct Location_ , and _Incorrect Location_ respectively. | FT
---|---
| _LowMismatch_ | _MediumMismatch_ | _HighMismatch_
ZS | NL | CL | IL | NL | CL | IL | NL | CL | IL
NL | 0 | 11 | 105 | 0 | 3 | 113 | 0 | 0 | 165
CL | 0 | 89 | 26 | 0 | 57 | 58 | 0 | 3 | 60
IL | 0 | 115 | 598 | 0 | 62 | 651 | 0 | 2 | 838

To further analyze the effect of Frequency Shock and Range Shift we compare
models in terms of the overlap between their predicted entities and those in
the train and test sets of the _HighMismatch_ dataset, where the train and
test entities are mutually exclusive. According to the results in Figure 4, we
observe that the FT model predicts the entities from the train set
significantly more than the FS model (almost $62\%$ relative increase). This
shows a clear case of Range Shift.

We also manually analyzed the outputs of the ZS and FT models for the “born
in” relation (as a representative relation)555We selected this relation
because it is simple to verify the model’s output types and subtypes. and
grouped the predictions of each model into three classes: 1- the output is not
a location, 2- the output is the correct location, and 3- the output is an
incorrect location. We then compared the number of queries in the cross-
product of the categories for the ZS and FT model. The results are presented
in Table 5.

Out of the $60$ queries on _HighMismatch_ where ZS predicted the correct
location and FT predicted an incorrect location, in $59$ cases the top answer
of the FT model was one of the entities from the training set answers, showing
another clear (and perhaps more severe) case for Range Shift. Moreover, on
_MediumMismatch_ , out of the $58$ queries for which the answer changed from a
correct location to an incorrect location after finetuning, in $19\%$ of those
cases the correct entity was “London” – a commonly occurring city (note that
only for $6\%$ of the queries the correct answer is “London”); In another
$33\%$ of those cases the correct entity is one of “Paris”, “Berlin”,
“Barcelona”, “Vienna” and “Brooklyn”, whereas only for $7\%$ of the queries
the answer is one of these cities. This is because the training set of
_MediumMismatch_ has a uniform distribution and finetuning on it leads to
frequency shock where common entities (such as “London”) are under-predicted.

###  4.3 The positive effect of Task Learning

Similar to the existing literature Fichtel et al. (2021), Table 5 provides
multiple evidences showing Task Learning is a positive effect of finetuning.
First, while ZS predicts non-location outputs (mostly years) for some queries,
FT correctly learns to predict a location for the queries666As an interesting
side note, for the queries for which the ZS model outputs a different type
than a location, even though the FT model learns to predict a location, it
tends to predict a wrong location; future work can use this signal to predict
when the LM does not know the answer to a question.. Secondly, for the 115
queries where ZS predicted an incorrect location but FT predicted a correct
one on _LowMismatch_ , in 90 cases the ZS model had generated a correct
country as the top output, and the FT model learned to predict the correct
city (which is the expected sub-type) instead of country. We observe a similar
behaviour for 39/62 queries in _MediumMismatch_. The other cases where the
prediction changed from incorrect location to correct location can be
explained by better learning the semantics of the task as a result of
finetuning.

##  5 Improving Finetuning

To avoid the side-effects identified in Section 4 and use finetuned LMs for
factual knowledge extraction and KG construction, one may be tempted to create
a training set that has a large coverage of various entities and that also has
a high Pearson correlation with what is expected to be seen at the test time.
We note, however, that entity coverage and Pearson correlation are somewhat at
odds with each other. That is because if we add many queries to the training
data whose answers are novel entities, it will cause the Pearson correlation
to go down unless we also add a prohibitively large number of queries with
common entities as answers to retain the proportions. Also, if we wish to keep
the Pearson correlation high, many of the rare entities may not appear in the
training set.

We aim to find solutions by changing the finetuning strategy. Given that LMs
are pre-trained on large corpora of text (typically much larger than the
finetuning dataset), we may expect the original entity distribution of the LM
(corresponding to its prior distribution) to be more robust to situations with
different frequency statistics. In this section, we exploit this insight to
provide two strategies to remedy the negative effect of Frequency Shock and
Range Shift in finetuning while still retaining the benefits of Task Learning.

Table 6: Results for model mixing (bold indicates winner). The _best single model_ corresponds to the model that gave the best result for each dataset (e.g., for T5 XXL FT is the best single model for _LowMismatch_ and _MediumMismatch_ , and FS for _HighMismatch_). _UB_ stands for upper-bound. |  | _LowMismatch_ | _MediumMismatch_ | _HighMismatch_
---|---|---|---|---
T5 | Model Mixing | Hit@1 | Hit@3 | Hit@5 | Hit@1 | Hit@3 | Hit@5 | Hit@1 | Hit@3 | Hit@5
XXL | Best Single Model | 51.9 | 68.4 | 73.9 | 43.6 | 57.8 | 63.2 | 27.0 | 33.4 | 34.9
FT + ZS | 51.3 | 66.3 | 71.5 | 45.8 | 58.9 | 64.1 | 22.0 | 31.5 | 35.8
FT + FS | 53.5 | 69.0 | 74.2 | 46.9 | 60.9 | 66.1 | 26.2 | 34.4 | 38.2
FT + FS (UB) | 59.3 | 71.8 | 75.9 | 52.9 | 65.1 | 69.7 | 29.2 | 36.9 | 40.6
Small | Best Single Model | 28.5 | 43.0 | 49.5 | 25.9 | 36.2 | 41.7 | 12.6 | 15.5 | 16.4
FT + ZS | 28.7 | 43.3 | 50.5 | 26.3 | 37.1 | 42.5 | 12.7 | 16.2 | 17.5

###  5.1 Model Mixing Alleviates Side-Effects

As we observed in the previous sections, the FT model has the advantage of
better learning the task and as a result producing better results than the
alternative models in situations such as the _LowMismatch_ dataset. However,
it also has the disadvantage of introducing a frequency bias that may lead to
low performance on situations such as the _MediumMismatch_ and _HighMismatch_
datasets. On the other hand, the ZS and FS models have the advantage of being
more robust to situations such as the _MediumMismatch_ and _HighMismatch_
datasets, but their performance falls short of the FT model in situations such
as the _LowMismatch_ dataset. Here, we explore whether these models can be
combined to get the best of the two worlds: the improved performance of the FT
models and the robustness of the ZS and FS models. We experiment with a simple
mixing approach where we average the scores produced by the FT model for each
output with that of the other models; we leave more sophisticated combination
strategies as future work.

We first verify if model mixing helps alleviate Frequency Shock and is more
robust. Figure 3 indicates the percentage of queries for which the FT+ZS and
FT+FS models predicted one of the Common or Rare entities. One can see from
the figure that, contrary to the FT model, the distributions for these models
are much closer to that of the ZS and FS models. The correction effect is
rather one-sided: while the Common entities are not under-predicted anymore,
the Rare entities are still slightly over-predicted. This is also apparent
from the accuracy of the FT+FS model on the Common and Rare sets in Table 4:
the performance on the Common set becomes similar to the FS model as Common
entities are not under-predicted anymore, but the performance on the Rare set
is still much lower than the FS model as Rare entities are still being over-
predicted.

We then look at whether model mixing provides better predictions overall. The
results are presented in Table 6. When using the T5 XXL model, for FT+ZS even
though the difference between the two models is quite large on _LowMismatch_ ,
the model results are only slightly worse than the FT model itself. For the
other two datasets, where the difference between the two models is much
smaller, model mixing leads to substantial improvement. FT+FS offers higher
performance than both individual models on _LowMismatch_ and _MediumMismatch_.
For _HighMismatch_ , all numbers improve substantially with respect to FT;
with respect to FS, however, Hit@1 goes slightly down whereas Hit@3 and Hit@5
improve. The same trend holds for the T5 Small model where mixing ZS and FT
performs better than both ZS and FT in isolation. Note that for the
_HighMismatch_ dataset, even though the FT model performs poorly in isolation,
mixing it with ZS still brings improvement for ZS.

While, in the past, model mixing has been shown to provide only marginal gains
in different applications even when multiple models are being combined, the
results in Table 6 show that a simple parameter-free combination of only two
models provides large boosts of up to 7.6% on the _MediumMismatch_ dataset.
Furthermore, we have included in Table 6 an upper-bound result for FT+FS where
we assume having access to an oracle that can tell if we should trust the FT
model or the FS model for each query. The upper-bound result is significantly
higher than each of the individual models, thus showing that there is a large
subset of the data where one model produces the correct answer whereas the
other model does not, hence indicating that the models work well on different
subsets of the data and that more sophisticated combinations can potentially
lead to more improvements. These results confirm that besides the previously
studied benefits of model mixing Naderi et al. (2021); Pranesh et al. (2020);
Wang et al. (2022); Ormerod (2022), it plays a much significant role for
knowledge extraction from finetuned LMs by correcting the side-effects of
finetuning.

Table 7: Results for mixture finetuning with different mixture ratios (bold indicates winner). 1:0 corresponds to standard finetuning. Mixture training consistently provides a boost in performance, especially for larger mixture ratios. The benefits from mixture training and model mixing can be combined. |  | _LowMismatch_ | _MediumMismatch_ | _HighMismatch_
---|---|---|---|---
T5 | Mixture | Hit@1 | Hit@3 | Hit@5 | Hit@1 | Hit@3 | Hit@5 | Hit@1 | Hit@3 | Hit@5
XXL | 1:0 | 51.9 | 68.4 | 73.9 | 43.6 | 57.8 | 63.2 | 18.0 | 27.4 | 32.4
1:1 | 51.8 | 68.4 | 74.0 | 45.6 | 61.0 | 66.9 | 18.2 | 27.3 | 32.3
1:5 | 52.2 | 68.0 | 73.9 | 46.5 | 60.5 | 66.6 | 18.5 | 27.9 | 32.9
1:15 | 52.7 | 68.4 | 73.9 | 45.6 | 60.2 | 66.1 | 19.5 | 28.8 | 33.4
1:15 + FS | 53.4 | 69.2 | 74.4 | 47.2 | 63.0 | 68.6 | 26.7 | 35.1 | 39.0
Small | 1:0 | 28.5 | 43.0 | 49.5 | 25.9 | 36.2 | 41.7 | 5.9 | 8.6 | 9.7
1:15 | 31.6 | 47.9 | 54.2 | 26.2 | 36.8 | 42.9 | 10.5 | 13.0 | 15.5

###  5.2 Mixture Training Alleviates Side-Effects

We examine if the following multi-task finetuning strategy can be an effective
solution to the identified side-effects. We finetune the model on a
combination of two tasks: 1- Factual Knowledge Extraction 2- The original pre-
training task of the LM ("masked language modeling"). Let $\alpha:\beta$
represent the ratio between the number of queries from the first and the
second tasks in each training batch. We set $\alpha=1$ and finetune models
with different values for $\beta$ for the three datasets, to see how mixture
training with different ratios affects the model performance. Intuitively, if
we continue to finetune the model with both these tasks, the second task
should help the LM observe the entities with a similar frequency as in its
pre-training stage, thus avoiding frequency shock to the model. This approach
has been previously shown to aid with the forgetting effect of finetuning on
LMs (He et al., 2021). The results in this section extend those results to the
case of Frequency Shock. From the results in Table 7, we can see that mixture
finetuning consistently provides improvements across the three datasets. The
amount of improvement larger for the _HighMismatch_ dataset where the
statistics differ more.

Finally, we combine the mixture finetuned model with the few-shot model and
see the benefits from the two solutions is additive. We see from Figure 3 that
the resulting model does not under-predict Common entities and does not over-
predict Rare entities. Also, from Table 4, we see that the resulting model
does not produce a substantially higher accuracy on the Common set due to
under-prediction, and does not produce a substantially lower accuracy on the
Rare set due to over-prediction. Finally, we see from Table 7 that the
benefits from model mixing and mixture training can be combined to make yet
better predictions.

###  5.3 A Knowledge Extraction Recipe

So far, we have observed that 1) a finetuned model may suffer from Frequency
Shock and Range Shift, 2) combining finetuned and few-shot models help improve
the results, and 3) mixture finetuning helps too. Here, we address a key
question: what is the best strategy for extracting factual knowledge from LMs
to construct KGs for real applications?

Table 8: Macro average performances of different models over the three datasets. Overall, an combination of a mixture finetuned model (with a large mixture ratio) with an FS model performs best. If one wishes to use only a single model during inference, then the best option is the mixture trained model. If one wants to avoid mixture training due to its higher cost, then the FS model is the winner followed by the FT model. | Macro average
---|---
Strategy | Hit@1 | Hit@3 | Hit@5
ZS | 30.0 | 41.4 | 45.7
FS | 38.9 | 46.8 | 48.2
FT | 37.8 | 51.2 | 56.5
RR | 33.0 | 42.6 | 45.7
FT + ZS | 39.7 | 52.2 | 57.1
FT + FS | 42.2 | 54.8 | 59.5
1:15 | 39.3 | 52.5 | 57.8
1:15 + ZS | 40.2 | 53.2 | 58.1
1:15 + FS | 42.5 | 55.8 | 60.1

In real applications of knowledge extraction from LMs for the purpose of
knowledge graph construction, we expect to see a combination of the properties
in the three datasets studied in this paper. That is, we expect training data
to be available mostly for some subsets of the knowledge domain that needs to
be extracted, and be scarce for other parts. We also expect the frequency
statistics of the train data and the data that needs to be extracted to be
similar for parts of the knowledge domain, and differ to various degrees in
other parts of the knowledge domain. Therefore, we expect knowledge extraction
in real applications to involve a combination of the three datasets studied in
this work. To find the best strategy for factual knowledge extraction from
LMs, we report the average performance of different combinations of the
techniques introduced in this paper on our three datasets.

According to the results in Table 8, the best strategy is to finetune a
mixture model and combine it with an FS model. This strategy leads to $12.4\%$
relative improvement over vanilla finetuning in terms of Hit@1. Given that
mixture finetuning substantially increases the time and cost required for
finetuning (especially for higher ratios for the pretraining task), if one
wants to avoid that cost the next best option is to use FT+FS. Alternatively,
if one can afford the training cost but needs to reduce the inference cost by
using a single model during inference and avoiding making multiple model calls
per query, the best strategy is to use a mixture finetuned model. Finally, FS
may result in better aggregate performance than FT.

##  6 Conclusion

Language models (LMs), especially when finetuned, can be a great source of
knowledge for constructing (or augmenting) knowledge graphs. However,
finetuning may also exhibit negative effects for knowledge extraction that are
important to understand and be aware of. In this paper, we identified
Frequency Shock and Range Shift as side-effects of finetuning, which can be
helpful or harmful depending on the frequency statistics of the dataset. We
then considered two solutions, model mixing and mixture finetuning with the
pre-training task, that help remedy the negative effects and result in
improved performance. Future work can look into more sophisticated ways of
model mixing or other finetuning techniques that avoid the negative effects to
a larger extent.

## References

  * AlKhamissi et al. (2022) Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad.  A review on language models as knowledge bases.  _arXiv preprint arXiv:2204.06031_ , 2022.
  * Allaway et al. (2022) Emily Allaway, Jena D Hwang, Chandra Bhagavatula, Kathleen McKeown, Doug Downey, and Yejin Choi.  Penguins don’t fly: Reasoning about generics through instantiations and exceptions.  _arXiv preprint arXiv:2205.11658_ , 2022.
  * Bhakthavatsalam et al. (2020) Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark.  Genericskb: A knowledge base of generic statements.  _arXiv preprint arXiv:2005.00660_ , 2020.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko.  Translating embeddings for modeling multi-relational data.  _Advances in neural information processing systems_ , 26, 2013.
  * Bosselut et al. (2019) Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi.  Comet: Commonsense transformers for automatic knowledge graph construction.  _arXiv preprint arXiv:1906.05317_ , 2019.
  * Cao et al. (2021) Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu.  Knowledgeable or educated guess? revisiting language models as knowledge bases.  _arXiv preprint arXiv:2106.09231_ , 2021.
  * Carlson et al. (2010) Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R Hruschka, and Tom M Mitchell.  Toward an architecture for never-ending language learning.  In _Twenty-Fourth AAAI conference on artificial intelligence_ , 2010\.
  * Da et al. (2021) Jeff Da, Ronan Le Bras, Ximing Lu, Yejin Choi, and Antoine Bosselut.  Analyzing commonsense emergence in few-shot knowledge models.  _arXiv preprint arXiv:2101.00297_ , 2021.
  * Davison et al. (2019) Joe Davison, Joshua Feldman, and Alexander M Rush.  Commonsense knowledge mining from pretrained models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 1173–1178, 2019.
  * Dong et al. (2014) Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang.  Knowledge vault: A web-scale approach to probabilistic knowledge fusion.  In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pp. 601–610, 2014.
  * Fichtel et al. (2021) Leandra Fichtel, Jan-Christoph Kalo, and Wolf-Tilo Balke.  Prompt tuning or fine-tuning-investigating relational knowledge in pre-trained language models.  In _3rd Conference on Automated Knowledge Base Construction_ , 2021\.
  * Fisch et al. (2019) Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen.  Mrqa 2019 shared task: Evaluating generalization in reading comprehension.  _arXiv preprint arXiv:1910.09753_ , 2019.
  * Goel et al. (2020) Rishab Goel, Seyed Mehran Kazemi, Marcus Brubaker, and Pascal Poupart.  Diachronic embedding for temporal knowledge graph completion.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 3988–3995, 2020.
  * Gokhale et al. (2020) Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang.  Mutant: A training paradigm for out-of-distribution generalization in visual question answering.  _arXiv preprint arXiv:2009.08566_ , 2020.
  * Hao et al. (2022) Shibo Hao, Bowen Tan, Kaiwen Tang, Hengzhe Zhang, Eric P Xing, and Zhiting Hu.  Bertnet: Harvesting knowledge graphs from pretrained language models.  _arXiv preprint arXiv:2206.14268_ , 2022.
  * He et al. (2021) Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng.  Analyzing the forgetting problem in pretrain-finetuning of open-domain dialogue response models.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pp. 1121–1133, 2021.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig.  How can we know what language models know?  _Transactions of the Association for Computational Linguistics_ , 8:423–438, 2020.
  * Kazemi & Poole (2018) Seyed Mehran Kazemi and David Poole.  Simple embedding for link prediction in knowledge graphs.  _Advances in neural information processing systems_ , 31, 2018.
  * Kervadec et al. (2021) Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf.  Roses are red, violets are blue… but should vqa expect them to?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2776–2785, 2021.
  * Lacroix et al. (2020) Timothée Lacroix, Guillaume Obozinski, and Nicolas Usunier.  Tensor decompositions for temporal knowledge base completion.  _arXiv preprint arXiv:2004.04926_ , 2020.
  * Lewis et al. (2020) Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel.  Question and answer test-train overlap in open-domain question answering datasets.  _arXiv preprint arXiv:2008.02637_ , 2020.
  * Li et al. (2016) Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel.  Commonsense knowledge base completion.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1445–1455, 2016\.
  * Lin et al. (2020a) Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren.  Birds have four legs?! numersense: Probing numerical commonsense knowledge of pre-trained language models.  _arXiv preprint arXiv:2005.00683_ , 2020a.
  * Lin et al. (2020b) Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, and William W Cohen.  Differentiable open-ended commonsense reasoning.  _arXiv preprint arXiv:2010.14439_ , 2020b.
  * Liu et al. (2021) Linqing Liu, Patrick Lewis, Sebastian Riedel, and Pontus Stenetorp.  Challenges in generalization in open domain question answering.  _arXiv preprint arXiv:2109.01156_ , 2021.
  * Miller (1995) George A Miller.  Wordnet: a lexical database for english.  _Communications of the ACM_ , 38(11):39–41, 1995\.
  * Naderi et al. (2021) Nona Naderi, Julien Knafou, Jenny Copara, Patrick Ruch, and Douglas Teodoro.  Ensemble of deep masked language models for effective named entity recognition in health and life science corpora.  _Frontiers in research metrics and analytics_ , 6, 2021.
  * Newman et al. (2021) Benjamin Newman, Prafulla Kumar Choubey, and Nazneen Rajani.  P-adapters: Robustly extracting factual information from language models with diverse prompts.  _arXiv preprint arXiv:2110.07280_ , 2021.
  * Ormerod (2022) Christopher Ormerod.  Short-answer scoring with ensembles of pretrained language models.  _arXiv preprint arXiv:2202.11558_ , 2022.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel.  Language models as knowledge bases?  _arXiv preprint arXiv:1909.01066_ , 2019.
  * Pranesh et al. (2020) Raj Ratn Pranesh, Ambesh Shekhar, and Smita Pallavi.  Quesbelm: A bert based ensemble language model for natural questions.  In _2020 5th International Conference on Computing, Communication and Security (ICCCS)_ , pp. 1–5. IEEE, 2020.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ , 2019.
  * Safavi & Koutra (2021) Tara Safavi and Danai Koutra.  Relational world knowledge representation in contextual language models: A review.  _arXiv preprint arXiv:2104.05837_ , 2021.
  * Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi.  Atomic: An atlas of machine commonsense for if-then reasoning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 3027–3035, 2019.
  * Shen et al. (2021) Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui.  Towards out-of-distribution generalization: A survey.  _arXiv preprint arXiv:2108.13624_ , 2021.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  _arXiv preprint arXiv:2010.15980_ , 2020.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Thirty-first AAAI conference on artificial intelligence_ , 2017\.
  * Sung et al. (2021) Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sungdong Kim, and Jaewoo Kang.  Can language models be biomedical knowledge bases?  _arXiv preprint arXiv:2109.07154_ , 2021.
  * Talmor & Berant (2019) Alon Talmor and Jonathan Berant.  Multiqa: An empirical investigation of generalization and transfer in reading comprehension.  _arXiv preprint arXiv:1905.13453_ , 2019.
  * Trouillon et al. (2016) Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard.  Complex embeddings for simple link prediction.  In _International conference on machine learning_ , pp. 2071–2080. PMLR, 2016.
  * Vrandečić & Krötzsch (2014) Denny Vrandečić and Markus Krötzsch.  Wikidata: a free collaborative knowledgebase.  _Communications of the ACM_ , 57(10):78–85, 2014\.
  * Wallat et al. (2021) Jonas Wallat, Jaspreet Singh, and Avishek Anand.  Bertnesia: Investigating the capture and forgetting of knowledge in bert.  _arXiv preprint arXiv:2106.02902_ , 2021.
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_ , 2022.
  * West et al. (2021) Peter West, Chandra Bhagavatula, Jack Hessel, Jena D Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi.  Symbolic knowledge distillation: from general language models to commonsense models.  _arXiv preprint arXiv:2110.07178_ , 2021.
  * Yadav et al. (2021) Vikas Yadav, Steven Bethard, and Mihai Surdeanu.  If you want to go far go together: Unsupervised joint candidate evidence retrieval for multi-hop question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2021.
  * Yin et al. (2022) Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang.  Geomlama: Geo-diverse commonsense probing on multilingual pre-trained language models.  _arXiv preprint arXiv:2205.12247_ , 2022.
  * Zhang et al. (2020) Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth.  Do language embeddings capture scales?  _arXiv preprint arXiv:2010.05345_ , 2020.
  * Zhong et al. (2021) Zexuan Zhong, Dan Friedman, and Danqi Chen.  Factual probing is [mask]: Learning vs. learning to recall.  _arXiv preprint arXiv:2104.05240_ , 2021.
  * Zhou et al. (2020) Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang.  Evaluating commonsense in pre-trained language models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 9733–9740, 2020.

##  Appendix A Implementation Details

We train all models for 10 epochs on a $4x4$ v3 TPU. Note that in the case of
mixture finetuning (Section 5.2), each epoch consists of more batch updates
compared to single-task finetuning because a portion of the examples in each
batch come from the pre-training task. We set the batch size to $128$ and the
learning rate to $0.0001$. We measured the Hit@1 performance on the validation
set after each epoch and selected the model parameters from the best
performing epoch on the validation for evaluating on the test set.

For the few-shot experiments (FS model), for each relation type we selected
$10$ random examples from the training set of the dataset. To get an estimate
of the standard deviation due to the choice of different examples, we repeated
our experiment for the _LowMismatch_ dataset 5 times each time selecting
different examples and observed a standard deviation of $0.56$.

Note that while for the three datasets we use in this paper the queries have
been selected in such a way that the answer is mostly a single token according
to the BERT vocabulary, those answers are already multi-token according to the
T5 vocabulary and that allows us to test the multi-token prediction ability of
the LMs. Previous work restrict the model prediction distribution to a
predefined set of tokens and ignores any predicted outputs outside that set.
We disregard that pre-defined set during training and validation to avoid
unwanted artifacts introduced due to the use of that specific vocabulary set.
However, we use that set for measuring performance on the test set so that the
final results are in the same footing as those of the previously published
work.