# Does Instruction Tuning Make LLMs More Consistent?

Constanza Fierro Jiaang Li Anders Søgaard
Department of Computer Science, University of Copenhagen
<EMAIL_ADDRESS>
###### Abstract

The purpose of instruction tuning is enabling zero-shot performance, but
instruction tuning has also been shown to improve chain-of-thought reasoning
and value alignment Si et al. (2023). Here we consider the impact on
consistency, i.e., the sensitivity of language models to small perturbations
in the input. We compare 10 instruction-tuned LLaMA models to the original
LLaMA-7b model and show that almost across-the-board they become more
consistent, both in terms of their representations and their predictions in
zero-shot and downstream tasks. We explain these improvements through
mechanistic analyses of factual recall.

##  1 Introduction

Large language models (LLMs) are trained with general objectives such as next
token prediction, but further training is often needed to make models useful
assistants. Recent research has shown how so-called instruction
tuning—supervised fine-tuning on user interactions—enables good zero-shot
performance across a range of tasks Wei et al. (2022); Sanh et al. (2022) by
aligning LLMs to user’s intents and expectations (Ouyang et al., 2022; Touvron
et al., 2023). Subsequently, many public instruction datasets have been
curated (Wang et al., 2022b; Longpre et al., 2023; Wang et al., 2022a, inter
alia), and new instruction fine tuned (IFT) models have been published. Models
have been evaluated in terms of their generation quality, by humans (Taori et
al., 2023; Chiang et al., 2024) or by leveraging LLMs (Chiang et al., 2023; Li
et al., 2023); while Wang et al. (2023b) evaluated IFT models capabilities by
measuring their accuracy on zero-shot and few-shot question answering, finding
that IFT models become better or worse at different tasks depending on the
dataset used.

This raises the question: How exactly does instruction tuning change LLMs? Lin
et al. (2023) studies this question in terms of decoding and token
distribution shift, finding most differences to be in stylistic tokens, thus
arguing that IFT primarily teaches the model to talk like an assistant.
However, instruction datasets have been created as verbalizations of tasks,
and thus the data will contain multiple surface forms encoding the same
expected behavior (see Figure 1). Therefore, we posit that this type of data
could also encourage more semantic consistency in IFT models, i.e. less
sensitivity to small perturbations, and thus make models more robust.

Figure 1: Examples of human prompts made to ChatGPT and shared in the ShareGPT
dataset.

Sensitivity of LLMs to subtle changes—that preserve the input’s meaning—has
been studied in terms of formatting (Sclar et al., 2024), in the context of
factual predictions (Elazar et al., 2021), and in downstream classification
tasks (Jang et al., 2022). Subsequently, methods for eliciting more consistent
behavior in LLMs have been proposed (Jang and Lukasiewicz, 2023b; Kassner et
al., 2021) and further evaluations have followed; in commercial models (Jang
and Lukasiewicz, 2023a), different languages (Fierro and Søgaard, 2022),
crosslingually (Qi et al., 2023), and in retrieval-augmented LLMs (Hagström et
al., 2023). However, to the best of our knowledge, no one has studied the
impact of IFT on semantic consistency.

In this paper, we compare the consistency of LLMs and their IFT counterparts,
by employing multiple evaluation methods and metrics to account for different
interpretations and use cases of semantic consistency. First, we compare their
representations and find that after IFT the vector spaces become more
consistent, with semantically similar texts encoded closer to each other, and
semantically different texts farther apart. Secondly, we evaluate the
consistency of models’ predictions in factual and world knowledge probes, and
after fine-tuning in downstream tasks, finding that IFT models are generally
more consistent in all these benchmarks. Finally, we study the factual
consistency improvements through mechanistic analyses of factual recall;
finding that the IFT improvements in consistency are due to a better retrieval
of subject related attributes.

##  2 Consistency in Vector Space

We study whether the semantics of the representations become more coherent
after IFT, so we analyze how the vector space changes by comparing cosine
distances between paraphrases and non-paraphrases.

#### Datasets

We consider 3 different datasets: (1) MRPC (Dolan and Brockett, 2005a)
contains pairs of sentences that are semantically equivalent, and pairs of
sentences that are similar but do not convey the same meaning; (2) TaPaCo
(Scherrer, 2020) contains sets of paraphrases or near-paraphrases extracted
from the Tatoeba database; and (3) ParaRel (Elazar et al., 2021) which is a
manually created resource of paraphrases in the context of factual knowledge
probing. For each dataset, we compute the cosine distance between the
paraphrases and then take the macro average across the sets of paraphrases
(Equation in §B.1). Conversely, we compute the cosine distance between non-
paraphrases by taking random examples (or, for MRPC, the available non-
paraphrases). Then, to compare across models we look at the difference between
these two averages, that is, at the gap between how close paraphrases are and
how close non-paraphrases are; because different models may have all their
representations generally closer together or farther apart.

#### Models

We compare the LLaMA model (Touvron et al., 2023) and 10 different IFT models
based in LLaMA. These models were fine-tuned using 10 different instruction
datasets (Details in §A). We also study T5-XL and FlanT5-XL, and Falcon-7b
alongside Falcon-Instruct-7b.

We represent paraphrase candidates by the outermost representation of their
last token (Gurnee and Tegmark, 2023; Zou et al., 2023). Given the influence
of instruction or context, we compute average cosine distances across
different prompts (§B.2).

| $\cos(\text{paraph.})-\cos(\neg\text{paraph.})$
---|---
Model | MRPC | TaPaCo | ParaRel
llama-7b (base model) | 0.010 |  0.140 | 0.474
baize-7b | 0.028 | 0.233 | 0.583
cot-7b | 0.035 | 0.235 | 0.615
dolly-7b | 0.038 | 0.198 | 0.604
flan-v2-7b | 0.035 | 0.250 | 0.603
gpt4-alpaca-7b | 0.020 |  0.140 | 0.585
oasst1-7b | 0.023 | 0.174 | 0.553
sharegpt-7b |  0.014 | 0.108 |  0.513
sni-7b | 0.045 | 0.308 | 0.629
stanford-alpaca-7b | 0.032 | 0.226 | 0.591
unnatural-instr-7b | 0.032 | 0.186 | 0.592

Table 1:  Difference between the average cosine distance of sets of
paraphrases and the average of pairs of non-paraphrases. In bold the lowest
and underlined the close 2nd lowest.

In Table 1, we compare the gap between the average cosine distance of
paraphrases and the average cosine distance of non-paraphrases. We find that
IFT models exhibit bigger relative differences in all the datasets evaluated,
both LLaMA models as well as T5 and Falcon IFT counterparts (§B.3).

##  3 Consistent Predictions

| MMLU | ParaRel | BECEL
---|---|---|---
Model | Acc. |  $\downarrow$ Spread | Acc. |  $\uparrow$ Cons. | Acc. |  $\uparrow$ Cons.
llama-7b (base model) | 31.1 | 1.6 | 71.8 |  81.9 | 74.4 | 84.7 / 42.3
baize-7b | 41.2 | 0.5 | 71.6 | 83.8 | 75.0 | 87.5 / 46.5
cot-7b | 42.3 | 0.7 |  70.5 |  81.9 | 76.6 | 86.1 / 51.9
dolly-7b | 37.7 | 0.5 | 73.6 | 83.0 | 75.4 | 86.2 / 44.6
flan-v2-7b | 45.2 | 0.3 | 71.8 | 82.3 | 78.4 | 89.4 / 51.0
gpt4-alpaca-7b | 42.6 | 0.3 | 73.2 | 84.5 | 75.8 |  84.8 / 47.6
oasst1-7b |  33.6 |  1.1 | 69.6 | 80.5 | 75.2 | 85.6 / 46.2
sharegpt-7b | 45.3 | 0.3 | 72.0 | 83.8 |  74.8 | 85.6 / 42.8
sni-7b | 44.3 | 0.2 | 72.7 | 84.1 | 75.1 | 86.9 / 45.1
stanford-alpaca-7b | 42.1 | 0.4 | 72.5 | 84.5 | 76.1 | 87.1 / 47.0
unnatural-instr-7b | 43.7 | 0.7 | 72.2 | 84.8 | 76.8 | 85.2 / 50.0

Table 2:  Consistency and spread results. In bold the lowest performance per
column, underlined the 2nd lowest. The BECEL consistency is under paraphrasing
/ negation averaged across the classification tasks (Results per task in C.2).

Does representational consistency affect the consistency of model predictions?
We measure: (1) factual consistency using the ParaRel dataset111ParaRel
contains paraphrases for 38 binary WikiData relations, e.g. “X was born in Y”
and “X is originally from Y”, where X would be replaced by a subject, and Y is
the object to be predicted. (Elazar et al., 2021), where consistency is the
fraction of paraphrases where the top-1 predictions are the same; (2) the
accuracy spread (Sclar et al., 2024), i.e., the min-max difference in accuracy
across different paraphrases, which we measure in the MMLU dataset222MMLU
evaluates world knowledge and reasoning skills through multiple choice
questions about mathematics, US history, law, and more. (Hendrycks et al.,
2021) by calculating accuracy over three different instructions in a zero-shot
fashion333To account for prompting variability, we consider 3 different
prompts for the ParaRel dataset and 3 different shuffling of the multiple
choices in the MMLU dataset, reporting average results; see prompts details in
§C.1.; and (3) the semantic consistency of downstream classification tasks,
that is, the consistency of NLI, paraphrase detection, and contextual word
similarity predictions under paraphrasing and negations, following the BECEL
benchmark (Jang et al., 2022) (Details in C.2).

We find that all IFT models have lower spread values compared to LLaMA (Table
2), and that on average LLaMA is the least consistent on downstream
classification tasks (BECEL). The ParaRel results are mixed, with most IFT
models exhibiting higher factual consistency, but a few (oasst, cot, and
flan-v2) similar to the LLaMA base model.444The standard deviation of the
consistency in ParaRel over the different contexts is 1.4 for LLaMA and
between 0.1-0.8 for the IFT models, thus we consider the difference
significance if is higher than 1.4. Thus, we conclude that overall IFT does
make models more robust to semantic equivalent inputs. Moreover, we see that
these results hold for the T5-XL / FlanT5-XL models, and partially hold for
Falcon / Falcon-Instruct (Details in §C.3).

##  4 How IFT Improves Consistency

In factual knowledge recall, an object is predicted given a subject and
relation. Geva et al. (2023) describe the process as three-way: (a) the
subject representation is enriched, i.e., related attributes are encoded; (b)
the relation information is passed to the last token; and (c) the final
predicted attribute is extracted by attention layers. Below, we analyze the
consistency improvements above in terms of these three mechanisms, focusing on
the ParaRel dataset. We report averages over three different contexts in all
of the following analyses.

Figure 2: Rate of tokens in the vocabulary space that correspond to attributes
related to the subject.

#### Subjects

Geva et al. (2023) found that in factual knowledge recall, the subject
representation is enriched to encode many subject-related attributes. We
therefore investigate whether the quality of this enrichment is improved after
IFT. We measure recall over subject attributes by inspecting the hidden state
$h^{l}$ of the last subject token at layer $l$ projected onto the vocabulary
space, i.e., $Eh^{l}$ with $E$ the output embedding matrix, and we consider
the top-50 tokens of this projection. To compute the fraction that indeed
corresponds to subject attributes, we create a set $\mathcal{A}_{s}$
containing tokens that appear in contexts where the subject $s$ is mentioned.
We obtain $\mathcal{A}_{s}$ by fetching the top 500 snippets in Wikipedia
where the subject is mentioned, and then follow the same preprocessing as in
Geva et al. (2023) (see §D). Finally, across models, we compare the fraction
(rate) of top-50 tokens that are present in $\mathcal{A}_{s}$; see Figure 2.
We find that LLaMA has lower attributes rate than all IFT models, thus showing
that IFT improves the recall of subject’s attributes across-the-board. On the
other hand, models that have lower or the same factual consistency as LLaMA in
the ParaRel dataset (such as oasst1 and cot) still have higher attributes
rate, therefore improvements in consistency cannot be explained solely by
improvements in recall.555We note that the attributes rate peaks at $\approx
30\%$ compared to the 40% Geva et al. (2023) reported, and that the rate
decreases after layer 25. This is probably due to differences in the Wikipedia
snippets and the templates (This data was not published so we could not use it
or verify it).

#### Relations

Next, we measure whether IFT leads to better representations of the relation
in a factual knowledge query. To this end, we first find the layers in the
model where the relation information is more salient, and then we compare the
representations across models in these specific layers.

Figure 3: Cosine similarity between queries from the same relation and
randomly different relations. Both pairs of queries have different subjects
and target objects. We take the macro average across relations and then across
models (the standard deviation in the plot is across models). Figure 4:
Difference between the cosine similarity of paraphrases and the cosine
similarity of non-paraphrases (Equation B.1).

We compute the cosine similarity of two different settings: (1) between query
pairs, where each query corresponds to a different relation and contains a
different subject and target object; (2) between pairs of queries with the
same relation, but with different subjects, target objects, and paraphrased
templates.666We further constrain the templates to end in a different token to
avoid confounds. Note how in (2), the only added similarity between the pairs
of sentences is the relation. We plot the results in Figure 3. We find that
both pairs of sentences exhibit similar trends, with similar representations
in the first layer, dropping on the 3rd. For different relations, this drop
continues until layer 9, whereas for same relation, the distance between the
representations stays mostly constant after layer 3. We conclude that the
bigger drop in layers 3-8 shows that these encode more semantic information
about the relation.777This is consistent with the gradient analysis in Geva et
al. (2023) showing that relation tokens are more relevant for the prediction
in early layers (0-10).

Finally, following §2, we compare paraphrase representations to non-paraphrase
representations to measure the extent to which a model differentiates between
semantically similar and semantically different inputs. Specifically, we plot
$\cos(\text{paraph.})-\cos(\neg\text{paraph.})$ (Equation 3 and 4) across
layers of each model—the bigger this gap the more the model makes a difference
between paraphrases and non-paraphrases. We break down this analysis in
consistent and inconsistent paraphrases, so we can compare representations for
which the model behaves similarly. Figure 4 contains the gap between
paraphrases and non-paraphrases for which the model gives consistent
outputs.888The plot for inconsistent outputs gives similar results and can be
found in §E, alongside the breakdown of the plot for each of the three
contexts. When looking at the relation layers (3-8) found previously, we see
that for some contexts the gap of the LLaMA model is bigger than that of IFT
models (prompt=0 in Figure 6 in §E); however on average, the gap is similar to
the other models. Thus, we do not find evidence that there is a better
semantic representation of the relation after IFT.

Figure 5: Extraction rate of MHSA, coloring depending on factual consistency
in ParaRel (Table 2) compared to LLaMA lower, higher-close, higher.

#### Extraction of Predicted Attribute

The last mechanism of factual recall predictions is the extraction of the
predicted attribute from the relation and enriched subject representations,
which have been propagated to the last token (Geva et al., 2023). We aim to
compare how this extraction mechanism might have changed after IFT. Consider a
model with $L$ transformer layers, let $h_{t}^{l}$ be the representation of
the last token $t$ at layer $l$, then each transformer layer computes

$\displaystyle h_{t}^{l+1}$ $\displaystyle=h_{t}^{l}+a^{l}+f^{l}$ (1)
$\displaystyle a^{l}$ $\displaystyle=\text{MHSA}(h_{0}^{l}\ldots h_{t}^{l})$
$\displaystyle f^{l}$ $\displaystyle=\text{MLP}(h_{t}^{l}+a^{l})$

where MHSA stands for Multi-Head Self-Attention. An extraction event happens
when the final predicted object $o^{*}$ (at the last layer) is the same as the
top-1 token of $a^{l}$ projected to vocabulary space, i.e.
$o^{*}=\arg\,\max(Ea^{l})$ (analogous for $f^{l}$). We compare the rate of
these events for MHSA in Figure 5.999MLP has similar results (see §F) but with
lower extraction rates overall, which is congruent with Geva et al. (2023). We
see that the IFT models that obtain lower consistency scores in ParaRel have
lower extraction rates (yellow in the Figure). So we compute the Pearson
correlation between the IFT models’ extraction rates and their ParaRel
consistency scores, and we find significant (p-value <= 0.05) positive
correlations of 0.65-0.92 in layers 13-21 (details in §F), showing that a
higher extraction rate is correlated with a higher factual consistency. On the
other hand, we see that LLaMA has one of the highest extraction rates in the
different layers, so we conclude that LLaMA’s lower factual consistency
results are probably due to the lower subject attributes recall (Figure 2),
while the lower factual consistency of some of the IFT models is due to a less
specific extraction rate of MHSA layers.

##  5 Conclusion

In this paper, we studied the robustness to variations that are semantically
irrelevant, i.e. consistency, of LLMs after instruction fine-tuning (IFT). We
considered multiple definitions and benchmarks of consistency to account for
different interpretations and use cases. Specifically, we studied consistency
from the perspective of vector spaces, zero-shot predictions, and downstream
predictions. We found almost across-the-board that IFT models are more
consistent. Lastly, we find explanations of this improvement by analyzing the
mechanisms of factual recall, concluding that IFT models exhibit a higher
recall of subject attributes resulting in consistency improvements when the
extraction mechanism (in the last layers) is not deteriorated with IFT.

## Limitations

Our experiments and analyses rely on input data, and even though we have tried
to be thorough by reporting averages over different contexts and instruction
formats, the exact numbers could change slightly when using different prompts.
Relatedly, in choosing the prompts we tried to make it relevant and the most
natural for each model, which in turns means that we have different prompts
for LLaMA and the IFT counterparts, because these models have been trained in
a different way and therefore expect the data to be different (LLaMA does not
follow instructions and the pretraining data would rarely look like that so
its performance would be most probably degraded). Lastly, we compare LLaMA and
multiple IFT counterparts and extend some of the evaluations to T5 and Falcon,
however is hard to generalize consistency improvements to any IFT model as
this will depend on the specifics of the data, as we saw for some LLaMA IFT
models (e.g. oasst1).

## References

  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Candela-Quinonero et al. (2006) Joaquin Candela-Quinonero, Ido Dagan, Bernardo Magnini, and Florence d’Alché Buc. 2006.  Evaluating Predictive Uncertainty, Visual Objects Classification and Recognising Textual Entailment: Selected Proceedings of the First PASCAL Machine Learning Challenges Workshop.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Chiang et al. (2024) Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024.  Chatbot arena: An open platform for evaluating llms by human preference.  _arXiv preprint arXiv:2403.04132_.
  * Databricks (2023) Databricks. 2023.  Free dolly: Introducing the world’s first truly open instruction-tuned llm.  Blog post.
  * Dolan and Brockett (2005a) Bill Dolan and Chris Brockett. 2005a.  Automatically constructing a corpus of sentential paraphrases.  In _Third International Workshop on Paraphrasing (IWP2005)_.
  * Dolan and Brockett (2005b) William B. Dolan and Chris Brockett. 2005b.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_.
  * Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021.  Measuring and improving consistency in pretrained language models.  _Transactions of the Association for Computational Linguistics_ , 9:1012–1031.
  * Fierro and Søgaard (2022) Constanza Fierro and Anders Søgaard. 2022.  Factual consistency of multilingual pretrained language models.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3046–3052, Dublin, Ireland. Association for Computational Linguistics.
  * Geva et al. (2023) Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023.  Dissecting recall of factual associations in auto-regressive language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 12216–12235, Singapore. Association for Computational Linguistics.
  * Gurnee and Tegmark (2023) Wes Gurnee and Max Tegmark. 2023.  Language models represent space and time.  _arXiv preprint arXiv:2310.02207_.
  * Hagström et al. (2023) Lovisa Hagström, Denitsa Saynova, Tobias Norlund, Moa Johansson, and Richard Johansson. 2023.  The effect of scaling, retrieval augmentation and form on the factual consistency of language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5457–5476, Singapore. Association for Computational Linguistics.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_.
  * Honovich et al. (2022) Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022.  Unnatural instructions: Tuning language models with (almost) no human labor.
  * Jang et al. (2022) Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. 2022.  BECEL: Benchmark for consistency evaluation of language models.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3680–3696, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Jang and Lukasiewicz (2023a) Myeongjun Jang and Thomas Lukasiewicz. 2023a.  Consistency analysis of ChatGPT.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 15970–15985, Singapore. Association for Computational Linguistics.
  * Jang and Lukasiewicz (2023b) Myeongjun Jang and Thomas Lukasiewicz. 2023b.  Improving language models’ meaning understanding and consistency by learning conceptual roles from dictionary.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 8496–8510, Singapore. Association for Computational Linguistics.
  * Kassner et al. (2021) Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. 2021.  BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8849–8861, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Köpf et al. (2023) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023.  Openassistant conversations – democratizing large language model alignment.
  * Li et al. (2023) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Alpacaeval: An automatic evaluator of instruction-following models.  https://github.com/tatsu-lab/alpaca_eval.
  * Lin et al. (2023) Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. 2023.  The unlocking spell on base llms: Rethinking alignment via in-context learning.  _arXiv preprint arXiv:2312.01552_.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023.  The flan collection: Designing data and methods for effective instruction tuning.  _arXiv preprint arXiv:2301.13688_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023.  Instruction tuning with gpt-4.  _arXiv preprint arXiv:2304.03277_.
  * Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019.  WiC: the word-in-context dataset for evaluating context-sensitive meaning representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1267–1273, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Qi et al. (2023) Jirui Qi, Raquel Fernández, and Arianna Bisazza. 2023.  Cross-lingual consistency of factual knowledge in multilingual language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 10650–10666, Singapore. Association for Computational Linguistics.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_.
  * Scherrer (2020) Yves Scherrer. 2020.  TaPaCo: A corpus of sentential paraphrases for 73 languages.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 6868–6873, Marseille, France. European Language Resources Association.
  * Sclar et al. (2024) Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024.  Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.  In _The Twelfth International Conference on Learning Representations_.
  * Si et al. (2023) Qingyi Si, Tong Wang, Zheng Lin, Xu Zhang, Yanan Cao, and Weiping Wang. 2023.  An empirical study of instruction-tuning large language models in Chinese.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 4086–4107, Singapore. Association for Computational Linguistics.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wang et al. (2023a) Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023a.  How far can camels go? exploring the state of instruction tuning on open resources.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Wang et al. (2023b) Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023b.  How far can camels go? exploring the state of instruction tuning on open resources.
  * Wang et al. (2022a) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022a.  Self-instruct: Aligning language model with self generated instructions.  _arXiv preprint arXiv:2212.10560_.
  * Wang et al. (2022b) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022b.  Super-naturalinstructions:generalization via declarative instructions on 1600+ tasks.  In _EMNLP_.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Xu et al. (2023) Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023.  Baize: An open-source chat model with parameter-efficient tuning on self-chat data.  _arXiv preprint arXiv:2304.01196_.
  * Zou et al. (2023) Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023.  Representation engineering: A top-down approach to ai transparency.  _arXiv preprint arXiv:2310.01405_.

##  Appendix A Instruction Tuning Datasets

We use the models trained and released by Wang et al. (2023a)101010We do not
evaluate on the Code-Alpaca dataset as it is focus on code generation, and on
Self-Instruct since we include Alpaca which is an improved version., here we
describe each of the datasets used in the fine tuning. These datasets have
been used to train other instruction tuned models, but we only evaluate with
the ones released by Wang et al. (2023a) because in this way we ensure that
the training and formatting for all the models were the same.

#### ShareGPT

Conversations with ChatGPT shared by users (https://sharegpt.com/). Data from
this website was used to train the Vicuña model (Chiang et al., 2023), but
Wang et al. (2023a) use the the data available in
HuggingFace111111https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered.
The dataset contains around 160k examples.

#### Dolly (Databricks, 2023)

Human written instructions created by 5000 Databricks employees for a variety
of tasks such as brainstorming, content generation, information extraction,
and summarization. The dataset contains around 15k examples.

#### Open Assistant (Köpf et al., 2023)

Human written instructions generated through crowd-sourcing involving over
13,500 volunteers, with varied education, nationality, and proficiency with
artificial intelligence. The dataset contains examples in 35 different
languages and has around 35k examples.

#### Flan V2 (Longpre et al., 2023)

Template instructions manually written to convert existing supervised datasets
into instruction examples. Compared to Flan (Wei et al., 2022) Flan V2 added
more templates variability, chain-of-thought examples, multiple languages, and
extra examples by inverting the input. The dataset contains 100k examples.

#### CoT (Longpre et al., 2023)

Version of the Flan V2 dataset containing chain-of-thought in the
instructions. The dataset contains 100k examples.

#### SuperNI (Wang et al., 2022b)

Human written instructions for existing 76 NLP tasks in a variety of
languages. The instructions were obtained as a community effort with NLP
practitioners and students submitting instructions to GitHub, which are then
reviewed by 1-2 expert contributors. The dataset contains around 96k examples.

#### Unnatural Questions (Honovich et al., 2022)

Automatically generated instructions by prompting a Davinci-002 with three
examples of instructions and generating a fourth one. The dataset contains
around 68k examples.

#### Alpaca (Taori et al., 2023)

Automatically generated interactions by prompting Davinci-003 with 175 seed
tasks, whereby they sample new examples from the model containing new
instructions, input, and output (Wang et al., 2022a). The dataset has a total
of 52k examples.

#### GPT4-Alpaca (Peng et al., 2023)

Same procedure than for the Alpaca dataset but using GPT-4 instead of
Davinci-002. The dataset contains 52k examples.

#### Baize (Xu et al., 2023)

Automatically generated examples by prompting ChatGPT (gpt-3.5-turbo) to
interact with itself. To prompt the model they use questions from the Quora
and Stack Overflow website as seeds. The dataset contains 210k examples.

| Prompt
---|---
IFT Models |
mrpc, tapaco | <|user|>\nSummarize the main message of the following sentence.\n{}\n<|assistant|>\n
mrpc, tapaco | <|user|>\nWhat does this sentence mean?\n{}\n<|assistant|>\n
mrpc, tapaco | <|user|>\n{}\n<|assistant|>\n
pararel | <|user|>\nComplete the sentence with as few words as possible.\n<|assistant|>\n{}
pararel | <|user|>\nProvide concise info to complete sentence.\n<|assistant|>\n{}
pararel | <|user|>\nPlease finish the sentence.\n<|assistant|>\n{}
Base Model |
mrpc, tapaco | {}
mrpc, tapaco | Consider the following sentence: {}
mrpc, tapaco | Summarize the main message of the following sentence.\n{}
pararel | {}
pararel | Everyone knows that {}
pararel | It is widely known that {}

Table 3: Prompts used for each dataset and model for obtaining a
representation of the input in order to compute the cosine distances. The
example input goes in the placeholder “{}”.

##  Appendix B Cosine Distance Details

###  B.1 Cosine Computation

Let $p=\\{s_{i}\\}_{i=1...M}$ be a set of paraphrases where $s_{i}$ and
$s_{j}$ have the same semantic meaning, we compute the average cosine
similarity of a set $p$ and then take the macro average over these sets, that
is,

$\displaystyle\cos(p)$
$\displaystyle=\frac{1}{M}\sum_{\begin{subarray}{c}i,j\\\ i\neq
j\end{subarray}}^{M}\cos(s_{i},s_{j})$ (2) $\displaystyle\cos(\text{paraph.})$
$\displaystyle=\frac{1}{N}\sum_{i}^{N}\cos(p_{i})$ (3)

Conversely, to compute the cosine distance between non-paraphrases we take at
random the sets of paraphrases $p_{i},p_{j}$ and the paraphrases
$s_{k},s_{l}$.

$\displaystyle\cos(\neg\text{paraph.})$
$\displaystyle=\frac{1}{K}\sum_{\begin{subarray}{c}p_{i}\neq p_{j}\\\ s_{k}\in
p_{i},\;s_{l}\in p_{i}\end{subarray}}^{K}\cos(s_{k},s_{l})$ (4)

Finally, to compare across models we look at the difference between these two
averages, that is, at the gap between how close paraphrases are and how close
non-paraphrases are: $\cos(\neg\text{paraph.})-\cos(\neg\text{paraph.})$;
because different models may have all their representations generally closer
together or farther apart.

###  B.2 Input Details

We define a different set of templates for the different datasets, as MRPC and
TaPaCo examples are full sentences whereas ParaRel inputs are shorter phrases
that are supposed to be completed in a factual way. For ParaRel examples we
only use the template with the subject replaced but not the object, as we do
not want to measure how closer or farther apart the object is but the general
representation of the paraphrased input. For the instruction tuned models we
follow the dialogue format in which they were fine tuned. The specific
templates are presented in Table 3.

| $\cos(\text{paraph.})-\cos(\neg\text{paraph.})$
---|---
Model | MRPC | TaPaCo | ParaRel
T5-XL (base model) | 1.24e-04 | 1.33e-03 | 3.71e-03
FlanT5-XL | 8.99e-04 | 5.84e-03 | 1.50e-02
Falcon-7b (base model) | 7.03e-04 | 5.92e-03 | 1.73e-02
Falcon-Instruct-7b | 1.20e-03 | 1.23e-02 | 2.69e-02

Table 4:  Difference between the average cosine distance of sets of
paraphrases and the average of pairs of non-paraphrases.

| MMLU | ParaRel | BECEL
---|---|---|---
Model | Acc. |  $\downarrow$ Spread | Acc. |  $\uparrow$ Cons. | Acc. |  $\uparrow$ Cons.
T5-XL (base model) | 24.9 | 2.1 | 48.1 | 75.5 | 65.4 | 89.4 / 99.7
FlanT5-XL | 48.5 | 0.3 | 55.0 | 80.1 | 81.3 | 91.1 / 99.3
Falcon-7b (base model) | 25.1 | 3.8 | 63.5 | 76.2 | 75.4 | 84.0 / 99.6
Falcon-Instruct-7b | 25.7 | 3.8 | 55.0 | 72.2 | 75.4 | 85.8 / 99.6

Table 5:  Consistency and spread results. BECEL consistency is under
paraphrases / negation averaged across the classification tasks (Results per
task in Table 6.

###  B.3 Results on Other Models

We also compute the gap of cosine distances between paraphrases and non-
paraphrases for other models, namely Falcon-7b and Falcon-Instruct-7b and the
T5-XL121212We use the google/t5-xl-lm-adapt version. and FlanT5-XL models. We
present the results in Table 4, we find that for all the datasets and both
pairs of models the IFT versions have a bigger gap between their average
cosine distances of paraphrases and non-paraphrases.

| SNLI | RTE | MRPC | WIC
---|---|---|---|---
Model | Sym | Sem | Neg | Acc. | Sym | Sem | Neg | Acc. | Sym | Sem | Neg | Acc. | Sym | Sem | Neg | Acc.
llama-7b (base model) | 0.737 | 0.843 | 0.834 | 0.870 | 0.731 |  0.847 | 0.471 |  0.690 | 0.877 |  0.797 |  0.321 | 0.770 | 0.883 | 0.900 |  0.069 |  0.647
gpt4-alpaca-7b | 0.750 | 0.852 | 0.818 | 0.876 | 0.723 | 0.819 | 0.595 | 0.704 | 0.880 | 0.842 | 0.421 | 0.804 | 0.868 | 0.879 | 0.073 |  0.647
sni-7b | 0.770 | 0.853 | 0.834 | 0.870 | 0.734 | 0.895 | 0.556 | 0.686 | 0.848 | 0.837 | 0.345 | 0.802 | 0.869 | 0.893 | 0.067 |  0.647
unnatural-instructions-7b | 0.735 | 0.852 | 0.843 | 0.874 | 0.765 |  0.847 | 0.588 | 0.722 | 0.884 | 0.832 | 0.448 | 0.816 | 0.880 | 0.879 | 0.119 | 0.660
cot-7b | 0.854 | 0.881 | 0.854 | 0.898 | 0.709 | 0.899 | 0.608 | 0.697 |  0.842 | 0.792 | 0.500 | 0.804 | 0.846 |  0.871 | 0.116 | 0.665
flan-v2-7b | 0.760 | 0.872 | 0.863 | 0.895 | 0.789 | 0.895 | 0.628 | 0.744 | 0.835 | 0.901 | 0.452 | 0.819 | 0.858 | 0.907 | 0.097 | 0.677
oasst1-7b | 0.702 | 0.832 | 0.831 | 0.860 | 0.662 | 0.887 | 0.582 | 0.697 | 0.860 | 0.856 | 0.345 | 0.804 | 0.854 | 0.850 | 0.090 |  0.647
sharegpt-7b |  0.726 | 0.844 | 0.813 | 0.866 | 0.730 | 0.891 | 0.575 | 0.700 | 0.854 | 0.852 | 0.235 |  0.775 | 0.864 | 0.836 | 0.088 | 0.652
stanford-alpaca-7b | 0.731 | 0.855 | 0.770 | 0.867 | 0.728 | 0.879 | 0.614 | 0.704 | 0.862 | 0.871 | 0.410 | 0.802 | 0.886 | 0.879 | 0.084 | 0.671
baize-7b | 0.751 | 0.851 | 0.829 | 0.867 |  0.673 | 0.871 | 0.582 | 0.661 | 0.862 | 0.886 | 0.362 | 0.797 | 0.873 | 0.893 | 0.088 | 0.676
dolly-7b | 0.738 |  0.836 |  0.776 |  0.861 | 0.753 | 0.879 |  0.543 | 0.715 | 0.862 | 0.856 | 0.383 | 0.806 |  0.851 | 0.879 | 0.082 | 0.633
FlanT5-XL (base model) | 0.939 | 0.913 | 0.940 | 0.929 | 0.837 | 0.948 | 0.758 | 0.812 | 0.886 | 0.896 | 0.793 | 0.838 | 0.886 | 0.886 | 0.132 | 0.679
t5-xl-lm-adapt | 0.838 | 0.830 | 0.804 | 0.796 | 0.850 | 0.940 | 0.190 | 0.578 | 0.926 | 0.886 | 0.059 | 0.737 | 0.926 | 0.921 | 0.056 | 0.585
Falcon-7b | 0.733 | 0.834 | 0.753 | 0.867 | 0.757 | 0.851 | 0.444 | 0.697 | 0.881 | 0.782 | 0.148 | 0.793 | 0.877 | 0.893 | 0.065 | 0.659
Falcon-7b-instruct | 0.703 | 0.848 | 0.804 | 0.852 | 0.805 | 0.867 | 0.582 | 0.736 | 0.893 | 0.822 | 0.124 | 0.796 | 0.879 | 0.893 | 0.108 | 0.633

Table 6: The consistency evaluation of Symmetric, Semantic, Negation based on
the BECEL dataset. In bold the lowest performance per column (per model
block), underlined the 2nd lowest. All metrics higher is better.

##  Appendix C Consistency Prediction Evaluation

###  C.1 MMLU and ParaRel Prompts

For the ParaRel dataset we use the same prompts we use for the cosine distance
analysis (see Table 3). For the MMLU dataset we compute the spread by
calculating the accuracy of each model for 3 different instructions and 3
different orderings of the multiple choice questions, so we take the average
over 9 different setups for each model. The 3 instructions we experiment with
are:

  * •

The following are multiple choice questions (with answers) about {}.

  * •

The following are multiple choice questions with their respective answer. The
questions are about {}.

  * •

Answer to the best of your abilities the following multiple choice questions
about {}.

As usual for MMLU evaluation (Wang et al., 2023a; Hendrycks et al., 2021) we
add the subject of the question in the placeholder “{}” (e.g. high school
mathematics), after this instruction the question and the multiple choice
answers are added.

###  C.2 BECEL

The BECEL dataset provides nineteen test sets for assessing five types of
consistency on seven downstream tasks. We follow the setup of Jang and
Lukasiewicz (2023a), focusing the scope of our experiments mainly on three
consistency measures: Negation, Semantic, Symmetric; and four datasets: SNLI
(Bowman et al., 2015), RTE (Candela-Quinonero et al., 2006), MRPC (Dolan and
Brockett, 2005b), and WiC (Pilehvar and Camacho-Collados, 2019). SNLI and RTE
are NLI datasets, where the task is to predict whether a premise entails a
hypothesis; MRPC is a paraphrase detection dataset; and in WiC the task is to
predict whether two sentences are using the same meaning of a word. BECEL
augments the input data of these datasets making available a negated version
of the input and its corresponding label, a paraphrased version of the input,
and the swapped input sentences and its corresponding target label.

#### Consistency Computation.

BECEL contains the original evaluation set $\mathcal{E}$ and a newly
constructed evaluation set $\mathcal{E}_{N}$; consider $e_{i}\in\mathcal{E}$
and $e^{N}_{i}\in\mathcal{E}_{N}$, where $e^{N}_{i}$ is a perturbed version of
$e_{i}$ (i.e., $|\mathcal{E}|=|\mathcal{E}_{N}|$). A model $M$ is expected to
yield predictions for both $e_{i}$ and $e^{N}_{i}$. Jang and Lukasiewicz
(2023a) reports inconsistency while here we report consistency (which is
simply 1-$\tau$), with $\tau$ computed as follows,

$\tau=\begin{cases}\frac{1}{|\mathcal{E}_{N}|}\sum_{i=1}^{|\mathcal{E}_{N}|}\mathbbm{1}(M(e_{i})=M(e^{N}_{i})),&\text{if
Sem. / Sym.}\\\
\frac{1}{|\mathcal{E}_{N}|}\sum_{i=1}^{|\mathcal{E}_{N}|}\mathbbm{1}(M(e_{i})\neq
M(e^{N}_{i})),&\text{if Neg.}\end{cases}$

where $\mathbbm{1}$ is the indicator function.

#### Finetune details.

To fine-tune all models, we experimented with three learning rates ($5e^{-4}$,
$1e^{-4}$, $5e^{-5}$), a batch size of 32, linear decay of the learning rate,
and early stopping. The models were trained for 10 epochs. For the SNLI
dataset, the optimal learning rate is $5e^{-5}$, while for the other three
datasets, the best learning rate is $1e^{-4}$.

###  C.3 Results

The detailed BECEL results for each task and model are presented in Table 6.
The results for the extra models (T5 and Falcon) can be found in Table 5. We
see that the same trends presented for the LLaMA variants hold for T5, FlanT5
is more consistent in terms of spread, factual consistency and downstream
consistency. Also, we find that T5 and Falcon have a high downstream
consistency under negation, which is maintained after IFT. On the other hand,
we find that the Falcon model after IFT does not have an improvement in spread
in MMLU and it has a decrease in factual consistency in ParaRel; this could be
due to the prompting strategy not being optimal for this model, given that
these 2 task are evaluated in a zero-shot fashion.

##  Appendix D Subject Attributes

We obtain the top 500 snippets from English Wikipedia where the subject is
mentioned in the text.131313The mention can include extra text in between
words of the subject. The ranking of these snippets is performed using the
“classic” profile of the API search, which ranks the pages based on the number
of incoming links, some templates, page language and recency. Then, we follow
Geva et al. (2023) and tokenize the text, remove duplicated tokens, stopwords
using the NLTK package, and tokens with less than 3 characters (so to remove
tokens representing frequent short sub-words). This results on $1,936\pm
1,036$ attributes per subject (for a total of 27,341 subjects).

Figure 6: Difference between the cosine similarity of paraphrases and the
cosine similarity of non-paraphrases (Equation B.1). Figure 7: Difference
between the cosine similarity of paraphrases and the cosine similarity of non-
paraphrases (Equation B.1), for inconsistent predictions.

##  Appendix E Relation Understanding

To study the cosine similarities of paraphrases we break down the
visualization in queries for which the models predict the same top-1 candidate
(consistent) and the queries for which they have different (inconsistent). So
that if the predictions are inconsistent because the model did not understand
properly the relation semantics then this does not bias the results. The
consistent predictions cosine similarities are presented in Figure 6 and the
inconsistent predictions are presented in Figure 7.

Figure 8: Extraction rate of MLP, coloring depending on factual consistency in
ParaRel (Table 2) compared to LLaMA lower, higher-close, higher.

##  Appendix F Extraction Rate

The MLP extraction rate results are presented in Figure 8, and the detailed
results of the Pearson correlation computation can be found in Table 7.

Block | Layer | Pearson | P-value
---|---|---|---
MHSA | 13 | 0.849 | 0.002
14 | 0.750 | 0.012
16 | 0.762 | 0.010
18 | 0.733 | 0.016
19 | 0.926 | 0.000
20 | 0.695 | 0.026
21 | 0.654 | 0.040
MLP | 18 | 0.874 | 0.001
19 | 0.937 | 0.000
20 | 0.914 | 0.000
21 | 0.661 | 0.038
22 | 0.743 | 0.014
27 | 0.819 | 0.004

Table 7: Pearson correlation of extraction rates of IFT models and ParaRel
factual consistency scores.


Hector_Dyer |  Born in Los Angeles , California , Hector Dyer enrolled at the Stanford University and won the IC4A championships in in 1930 . |  (Hector Dyer, enrolled at, The Stanford University) $\rightarrow$ (hector_dyer.Q1364518, educated_at.P69, stanford_university.Q41506)
Henny_Magnussen |  Thanks to the support she received , she was able to practise in all the courts of Denmark from 1909 , becoming the first woman to do so . |  (The First Woman, ,, Denmark) $\rightarrow$ (henny_magnussen.Q56577333, country_of_citizenship.P27, denmark.Q35)
Hina_Pervaiz_Butt |  Hina Pervaiz Butt Hina Pervaiz Butt is a Pakistani politician who was a Member of the Provincial Assembly of the Punjab , since May 2013 . |  (Hina Pervaiz Butt, is, A Pakistani Politician) $\rightarrow$ (hina_pervaiz_butt.Q47918210, occupation.P106, politician.Q82955)
Isabella_Of_Valois,_Duchess_Of_Bourbon |  She was the wife of Peter I , Duke of Bourbon . |  (She, was the wife of, Peter I) $\rightarrow$ (isabella_of_valois,_duchess_of_bourbon.Q2363495, spouse.P26, peter_i,_duke_of_bourbon.Q444668)
Jahir_Butrón |  Then from 1995 , he played for Peruvian Second Division side Guardia Republicana . |  (He, played for, Peruvian Second Division Side Guardia Republicana) $\rightarrow$ (jahir_butrón.Q6123259, member_of_sports_team.P54, guardia_republicana.Q5614012)
Jahir_Butrón |  Jahir Butrón Jahir Butrón Gotuzzo is a Peruvian footballer who plays as a center back . |  (Jahir Butrón Gotuzzo, is, A Peruvian Footballer) $\rightarrow$ (jahir_butrón.Q6123259, country_of_citizenship.P27, peru.Q419); (Jahir Butrón Gotuzzo, is, A Peruvian Footballer) $\rightarrow$ (jahir_butrón.Q6123259, sport.P641, association_football.Q2736)
Joan_Of_France,_Duchess_Of_Berry |  She was the second daughter of King Louis XI of France and of his second wife Charlotte of Savoy ; her surviving siblings were King Charles VIII of France and Anne of France . |  (She, was the second daughter, King Louis Xi) $\rightarrow$ (joan_of_france,_duchess_of_berry.Q236220, father.P22, louis_xi_of_france.Q8058)
John_Ogilvie_(Footballer) |  He played for Hibs in their most successful era renowned for their front five known as The Famous Five . |  (He, played for, Hibs) $\rightarrow$ (john_ogilvie_(footballer).Q6251147, member_of_sports_team.P54, hibernian_f.c..Q192597)
Johnedel_Cardel |  He moved to Sta . |  (He, to, Sta) $\rightarrow$ (johnedel_cardel.Q56599981, member_of_sports_team.P54, sta._lucia_realtors.Q3547751)
Joseph_W._Nega |  Joseph W . Nega Joseph W . Nega is a Judge of the United States Tax Court . |  (Joseph W. Nega, is, A Judge) $\rightarrow$ (joseph_w._nega.Q16733242, occupation.P106, judge.Q16533)
José_Alejandro_Aguilar_López |  José Alejandro Aguilar López José Alejandro Aguilar López is a Mexican politician from the National Action Party . |  (José Alejandro Aguilar López, is, A Mexican Politician) $\rightarrow$ (josé_alejandro_aguilar_lópez.Q18385891, country_of_citizenship.P27, mexico.Q96); (José Alejandro Aguilar López, is, A Mexican Politician) $\rightarrow$ (josé_alejandro_aguilar_lópez.Q18385891, occupation.P106, politician.Q82955); (José Alejandro Aguilar López, Mexican politician, The National Action Party) $\rightarrow$ (josé_alejandro_aguilar_lópez.Q18385891, member_of_political_party.P102, national_action_party_(mexico).Q851087)
Juan_Bautista_Azopardo |  Juan Bautista Azopardo was born in Senglea , Malta , the son of Rosina and Salvatore Azopardo . |  (Juan Bautista Azopardo, ,, Malta) $\rightarrow$ (juan_bautista_azopardo.Q3805864, country_of_citizenship.P27, malta.Q233); (Juan Bautista Azopardo, was born, Senglea) $\rightarrow$ (juan_bautista_azopardo.Q3805864, place_of_birth.P19, senglea.Q846593)
Kevin_Kerr_(Scottish_Footballer) |  He signed for AGOVV Apeldoorn of Netherlands ’ Eerste Divisie in January 2012 after his contract in Bielefeld had expired the previous summer . |  (He, signed for, Agovv Apeldoorn) $\rightarrow$ (kevin_kerr_(scottish_footballer).Q326761, member_of_sports_team.P54, agovv_apeldoorn.Q292618)
Lachezar_Kotev |  Lachezar Kotev Lachezar Kotev is a Bulgarian footballer who plays as a midfielder for Vitosha Bistritsa . |  (Lachezar Kotev, is, A Bulgarian Footballer) $\rightarrow$ (lachezar_kotev.Q49560773, sport.P641, association_football.Q2736); (Lachezar Kotev, is, A Midfielder) $\rightarrow$ (lachezar_kotev.Q49560773, position_played_on_team_/_speciality.P413, midfielder.Q193592)
Marc_Zeno |  In 1988 , Zeno was drafted in the 7th round by the Pittsburgh Steelers . |  (Zeno, was drafted in the, The Pittsburgh Steelers) $\rightarrow$ (marc_zeno.Q16196353, member_of_sports_team.P54, pittsburgh_steelers.Q191477)
Marcel_Dheere |  Marcel Dheere Marcel Albert " Ching " Dheere was a Canadian professional ice hockey forward who played 11 games in the National Hockey League for the Montreal Canadiens . |  (Marcel Albert "Ching, in, A Canadian Professional Ice Hockey) $\rightarrow$ (marcel_dheere.Q6756083, country_of_citizenship.P27, canada.Q16); (The National Hockey League, for, The Montreal Canadiens) $\rightarrow$ (marcel_dheere.Q6756083, member_of_sports_team.P54, montreal_canadiens.Q188143); (A Canadian Professional Ice Hockey, played, The National Hockey League) $\rightarrow$ (marcel_dheere.Q6756083, league.P118, national_hockey_league.Q1215892); (Marcel Albert "Ching, played, A Canadian Professional Ice Hockey) $\rightarrow$ (marcel_dheere.Q6756083, sport.P641, ice_hockey.Q41466)
Mark_Schultz_(Wrestler) |  Mark Schultz was born in 1960 in Palo Alto , California to Dorothy Jean St . Germain and Philip Gary Schultz . |  (Mark Schultz, was born in 1960 in, Dorothy Jean St. Germain) $\rightarrow$ (mark_schultz_(wrestler).Q3487332, place_of_birth.P19, palo_alto,_california.Q47265)
Michael_Jones_(Footballer) |  Michael Jones Michael Jones is an English footballer , who plays for Cefn Druids . |  (Michael Jones, is, An English Footballer) $\rightarrow$ (michael_jones_(footballer).Q6831707, sport.P641, association_football.Q2736)
Mike_Bellotti |  He is an alumnus of the Delta Sigma Phi fraternity . |  (He, is an alumnus of, The Delta Sigma Phi Fraternity) $\rightarrow$ (mike_bellotti.Q14950800, educated_at.P69, california_state_university,_east_bay.Q1026916)
Olga_Semenova_Tyan-Shanskaya |  She learned to play chess at the age of 18 ; four years later she won the Leningrad city women ’ s chess championship . |  (She, to play, Chess) $\rightarrow$ (olga_semenova_tyan-shanskaya.Q2392240, sport.P641, chess.Q718)
Owen_Paterson |  Owen Paterson Owen William Paterson is a British Conservative Party politician who was the Secretary of State for Environment , Food and Rural Affairs from 2012 to 2014 . |  (Owen Paterson, is, A British Conservative Party Politician) $\rightarrow$ (owen_paterson.Q197894, occupation.P106, politician.Q82955)
Pan_Gongsheng |  Pan Gongsheng Pan Gongsheng is a Chinese economist , banker , reformist and bureaucrat . |  (Pan Gongsheng, is, A Chinese Economist) $\rightarrow$ (pan_gongsheng.Q9309080, country_of_citizenship.P27, china.Q148)
Paul Carrington (American Football) |  Paul Carrington Paul Carrington is a former American football defensive end . |  (Paul Carrington, is, A Former American Football Defensive End) $\rightarrow$ (paul_carrington_(american_football).Q9057011, sport.P641, american_football.Q41323)
Peter_Jebsen |  Peter Jebsen Peter Jebsen was a Norwegian businessperson and politician . |  (Peter Jebsen, was, A Norwegian Businessperson) $\rightarrow$ (peter_jebsen.Q1774966, country_of_citizenship.P27, norway.Q20); (Peter Jebsen, was, A Norwegian Businessperson) $\rightarrow$ (peter_jebsen.Q1774966, occupation.P106, businessperson.Q43845); (Peter Jebsen, was, Politician) $\rightarrow$ (peter_jebsen.Q1774966, occupation.P106, politician.Q82955)
Pina_Piovani |  Pina Piovani Pina Piovani was an Italian stage and film actress . |  (Pina Piovani, was, An Italian Stage And Film Actress) $\rightarrow$ (pina_piovani.Q3388583, country_of_citizenship.P27, italy.Q38)
Robert_G._Cole |  Rapidly advancing through the ranks at Fort Benning as the parachute infantry battalions were expanded to regiments , he was a lieutenant colonel commanding the 3rd Battalion of the 502nd Parachute Infantry Regiment on June 6 , 1944 , the date of his unit ’ s first combat jump . |  (He, was, A Lieutenant Colonel) $\rightarrow$ (robert_g._cole.Q969944, military_rank.P410, lieutenant_colonel.Q493898)
Robert_Holt_Carpenter |  Robert Holt Carpenter Robert Holt Carpenter was a New Zealand bookbinder , local politician , bookseller and character . |  (Robert Holt Carpenter, was, A New Zealand Bookbinder) $\rightarrow$ (robert_holt_carpenter.Q7345593, country_of_citizenship.P27, new_zealand.Q664)
Robert_Van_Boxel |  He joined amateur side FC Lisse in summer 2015 from Sparta . |  (He, joined, Amateur Side Fc Lisse) $\rightarrow$ (robert_van_boxel.Q2017870, member_of_sports_team.P54, fc_lisse.Q978325)
Rolf_Maurer |  Rolf Maurer Rolf Maurer was a Swiss road racing cyclist who competed professionally between 1960 and 1969 . |  (Rolf Maurer, was, A Swiss Road Racing Cyclist) $\rightarrow$ (rolf_maurer.Q870132, country_of_citizenship.P27, switzerland.Q39); (Rolf Maurer, was, A Swiss Road Racing Cyclist) $\rightarrow$ (rolf_maurer.Q870132, sport.P641, road_bicycle_racing.Q3609)
Ron_Maudsley |  Maudsley was born in Lostock Gralam , Cheshire , and educated at Malvern College and Birmingham University , then served in the Royal Army Service Corps during World War II . |  (Maudsley, was born in, Lostock Gralam) $\rightarrow$ (ron_maudsley.Q16009653, place_of_birth.P19, lostock_gralam.Q4640202)
Safa_Al-Safi |  Safa al - Safi Safa al - Din Mohammed al - Safi is an Iraqi politician and former Justice Minister who is currently Minister of State for the Council of Representatives . |  (Mohammed Al-Safi, is, An Iraqi Politician) $\rightarrow$ (safa_al-safi.Q7398343, occupation.P106, politician.Q82955)
Salvador_Luria |  In 1959 , he became chair of Microbiology at the Massachusetts Institute of Technology . |  (He, became chair of, Microbiology) $\rightarrow$ (salvador_luria.Q205667, field_of_work.P101, microbiology.Q7193)
Sanele_Vavae_Tuilagi |  He now plays for US Carcassonne . |  (He, now plays, Us Carcassonne) $\rightarrow$ (sanele_vavae_tuilagi.Q3472139, member_of_sports_team.P54, us_carcassonne.Q3550526)
Seth_Plum |  Seth Plum Seth Lewis Plum was an English international footballer who played as a wing half . |  (Seth Lewis Plum, was, An English International Footballer) $\rightarrow$ (seth_plum.Q7456581, sport.P641, association_football.Q2736); (Seth Lewis Plum, was, An English International Footballer) $\rightarrow$ (seth_plum.Q7456581, member_of_sports_team.P54, england_national_football_team.Q47762)
Sofija_Korkutytė |  Sofija Korkutytė Sofija Korkutytė was a Lithuanian rower who won three European titles in the eights event in 1963 , 1965 and 1967 ; she finished second in 1964 and 1966 . |  (Sofija Korkutytė, was, A Lithuanian Rower) $\rightarrow$ (sofija_korkutytė.Q16014909, sport.P641, rowing_(sport).Q159354)
Thomas_Rogne |  He made his first start for Celtic at home against Dundee United in a 1 – 0 victory on 20 February 2010 , he played very well and impressed the coach Tony Mowbray . |  (He, at, Celtic) $\rightarrow$ (thomas_rogne.Q609801, member_of_sports_team.P54, celtic_f.c..Q19593)
Uri_Avnery |  Avnery was born in Beckum , near Münster in Westphalia , as Helmut Ostermann , the youngest of four children , to a well - established German Jewish family , his father being a private banker in the town . |  (Westphalia, in, Beckum) $\rightarrow$ (uri_avnery.Q325679, place_of_birth.P19, beckum,_germany.Q2707)
Ursula_Lehr |  She was a member of the Christian Democratic Union . |  (She, of, The Christian Democratic Union) $\rightarrow$ (ursula_lehr.Q290708, member_of_political_party.P102, christian_democratic_union_of_germany.Q49762)
Valdívia_(Brazilian_Footballer) |  Valdívia Wanderson Ferreira de Oliveira , commonly known as Valdívia , is a Brazilian footballer who plays as an attacking midfielder for Avaí , on loan from Internacional . |  (A Brazilian Footballer, who plays as, An Attacking Midfielder) $\rightarrow$ (valdívia_(brazilian_footballer).Q18176477, position_played_on_team_/_speciality.P413, midfielder.Q193592)
Victor-François,_2Nd_Duc_De_Broglie |  His victory over Prince Ferdinand at Bergen won him the rank of marshal of France from the French King Louis XV and the title of Prince of the Holy Roman Empire " Reichsfürst " from Holy Roman Emperor Francis I . In 1759 , he won the Battle of Bergen and followed that with the capture the city of Minden , later fighting at the Battle of Minden under the command of Contades , whom he would succeed in command . |  (His victory, won him the, Marshal) $\rightarrow$ (victor-françois,_2nd_duc_de_broglie.Q695929, award_received.P166, marshals_of_france.Q474485)

Figure 7: BERTLARGE on Wikidata-OIE.

ID |  Sentence |  Triples to gold triples
---|---|---
Appropriate_Rural_Technology_Institute |  Appropriate Rural Technology Institute Appropriate Rural Technology Institute is an Indian non-governmental organization founded by a group of about 20 scientists and technologists to develop innovative and environmentally friendly rural technologies based on modern scientific knowledge in 1996. |  (Appropriate Rural Technology Institute, is, An Indian Non-Governmental Organization) $\rightarrow$ (appropriate_rural_technology_institute.Q16245764, country.P17, india.Q668)
Ayatollah_(Record_Producer) |  Ayatollah Lamont Dorrell, known as Ayatollah, is a hip-hop record producer from Queens, New York who has produced for predominantly New York-based rappers including Mos Def, Talib Kweli, R.A. The Rugged Man, Tragedy Khadafi, Wordsworth, Vast Aire, Afu-Ra, Guru, M.O.P., Inspectah Deck, Cormega, Ghostface Killah as well as many others. |  (Ayatollah, is, A Hip-Hop Record Producer) $\rightarrow$ (ayatollah_(record_producer).Q4830984, occupation.P106, record_producer.Q183945)
Benjamin_Darbelet |  Benjamin Darbelet Benjamin Darbelet is a French judoka. |  (Benjamin Darbelet, is, A French Judoka) $\rightarrow$ (benjamin_darbelet.Q2456207, country_of_citizenship.P27, france.Q142); (Benjamin Darbelet, is, A French Judoka) $\rightarrow$ (benjamin_darbelet.Q2456207, sport.P641, judo.Q11420)
Bharat_Khawas |  He plays for club Tribhuvan Army in Martyr’s Memorial A-Division League as a striker. |  (He, plays for, Club Tribhuvan Army) $\rightarrow$ (bharat_khawas.Q4901128, member_of_sports_team.P54, nepal_army_club.Q6994327); (He, plays for, Club Tribhuvan Army) $\rightarrow$ (bharat_khawas.Q4901128, sport.P641, association_football.Q2736)
Bill_White_(Shortstop) |  White was born on May 1, 1860 in Bridgeport, Ohio, and he began his professional baseball career in 1883 with the Pottsville Antharcites of the International Association. |  (White, in, Bridgeport) $\rightarrow$ (bill_white_(shortstop).Q4911357, place_of_birth.P19, bridgeport,_ohio.Q2062947)
Bob_Dylan.Q392 |  Dylan married Sara Lownds , who had worked as a model and a secretary to Drew Associates , on November 22 , 1965 . |  (Dylan, married, Sara Lownds) $\rightarrow$ (Bob_Dylan.Q392, spouse.P26, Sara_Lownds.Q457433)
Bob_Dylan.Q392 |  In 1991 , Dylan received a Grammy Lifetime Achievement Award from American actor Jack Nicholson . |  (Dylan, received, A Grammy Lifetime Achievement Award) $\rightarrow$ (Bob_Dylan.Q392, award_received.P166, Grammy_Lifetime_Achievement_Award.Q935843)
Boog_Powell |  He also played for the Cleveland Indians and the Los Angeles Dodgers. |  (He, also played for, The Cleveland Indians) $\rightarrow$ (boog_powell.Q4942810, member_of_sports_team.P54, cleveland_indians.Q642553)
Boog_Powell |  He played in Major League Baseball as a first baseman and left fielder from through , most notably as a member of the Baltimore Orioles dynasty that won four American League pennants and two World Series championships between 1966 and 1971. |  (He, played in, A First Baseman) $\rightarrow$ (boog_powell.Q4942810, position_played_on_team_/_speciality.P413, first_baseman.Q1326154)
Bouchraya_Hammoudi_Bayoun |  He studied Economics at the University of Havana, Cuba. |  (He, studied, University of Havana) $\rightarrow$ (bouchraya_hammoudi_bayoun.Q4949324, educated_at.P69, university_of_havana.Q837320)
Cezaro_Rossetti |  Of Italian-Swiss derivation, he was born in Glasgow and lived in Britain. |  (He, was born in, Glasgow) $\rightarrow$ (cezaro_rossetti.Q2947206, place_of_birth.P19, glasgow.Q4093)
Chan_Sau_Ying |  Chan Sau Ying Chan Sau Ying is a retired athlete from Hong Kong who specialised in the 100 metres hurdles. |  (Sau Ying, is, A Retired Athlete) $\rightarrow$ (chan_sau_ying.Q21858345, sport.P641, sport_of_athletics.Q542)
Christian_Mcgrath |  He was born in the Bronx in 1972 and became interested in art and illustration as a small child. |  (He, was born in, The Bronx) $\rightarrow$ (christian_mcgrath.Q5109897, place_of_birth.P19, the_bronx.Q18426)
Cliff_Michelmore |  He was a regular presenter on BBC1’s "Holiday" programme from 1969 to 1986, and presented other shows for BBC TV, ITV and BBC Radio. |  (He, was, A Regular Presenter) $\rightarrow$ (cliff_michelmore.Q5132704, occupation.P106, television_presenter.Q947873)
Crooked_I |  Crooked I Dominick Antron Wickliffe , better known by his stage name Crooked I or Kxng Crooked, is an American rapper from Long Beach, California. |  (Crooked I, is, An American Rapper) $\rightarrow$ (crooked_i.Q253695, country_of_citizenship.P27, united_states.Q30)
Denise_Johns |  She graduated in 2002, and moved to Atlanta, playing amateur beach volleyball on the east coast of America. |  (She, playing, Amateur Beach Volleyball) $\rightarrow$ (denise_johns.Q5257690, sport.P641, beach_volleyball.Q4543)
Diana_M._Fennell |  Fennell was born in Emporia, Virginia and attended the public schools there. |  (Fennell, was born in, Emporia) $\rightarrow$ (diana_m._fennell.Q28037279, place_of_birth.P19, emporia,_virginia.Q177678)
Diana_Rast |  She also competed at the 1996 Summer Olympics. |  (She, also competed at, The 1996 Summer Olympics) $\rightarrow$ (diana_rast.Q2491787, participant_of.P1344, 1996_summer_olympics.Q8531)
Eduard_Friedrich_Poeppig |  Eduard Friedrich Poeppig Eduard Friedrich Poeppig was a German botanist, zoologist and explorer. |  (Eduard Friedrich, was, A German Botanist) $\rightarrow$ (eduard_friedrich_poeppig.Q61501, field_of_work.P101, botany.Q441)
Edward_L._Romero |  After the Vietnam war, he became a county chairman of Democratic Party. |  (He, of, Democratic Party) $\rightarrow$ (edward_l._romero.Q5344025, member_of_political_party.P102, democratic_party_(united_states).Q29552)
Endre_Tilli |  Endre Tilli Endre Tilli was a Hungarian fencer. |  (Endre Tilli, was, A Hungarian Fencer) $\rightarrow$ (endre_tilli.Q1642140, country_of_citizenship.P27, hungary.Q28); (Endre Tilli, was, A Hungarian Fencer) $\rightarrow$ (endre_tilli.Q1642140, sport.P641, fencing.Q12100)
Enrico_Porro |  Porro competed at the 1908 Summer Olympics in London where he won the gold medal in Greco-Roman wrestling, the lightweight class. |  (Porro, competed at, The 1908 Summer Olympics) $\rightarrow$ (enrico_porro.Q1343905, participant_of.P1344, 1908_summer_olympics.Q8111)
Erfan_Zeneli |  Erfan Zeneli Erfan Zeneli is a Finnish footballer of Kosovar Albanian descent who currently plays for HJK Helsinki as a striker or left winger. |  (Erfan Zeneli, is, A Finnish Footballer) $\rightarrow$ (erfan_zeneli.Q2053367, country_of_citizenship.P27, finland.Q33); (Erfan Zeneli, is, A Finnish Footballer) $\rightarrow$ (erfan_zeneli.Q2053367, sport.P641, association_football.Q2736); (Erfan Zeneli, is, A Finnish Footballer) $\rightarrow$ (erfan_zeneli.Q2053367, country_for_sport.P1532, finland.Q33)
Erminio_Favalli |  He then moved to Mantova before to join Palermo, where he spent seven seasons as a "rosanero" mainstay. |  (He, then moved to, Mantova) $\rightarrow$ (erminio_favalli.Q1405871, member_of_sports_team.P54, mantova_1911_s.s.d..Q430993)
George_Dowty |  Dowty was born in Pershore, Worcestershire in 1901. |  (Dowty, was born in, Pershore) $\rightarrow$ (george_dowty.Q1507239, place_of_birth.P19, pershore.Q767858)
Harald_Fairhair |  In the Saga of Harald Fairhair in "Heimskringla", which is the most elaborate although not the oldest or most reliable source to the life of Harald, it is written that Harald succeeded, on the death of his father Halfdan the Black Gudrödarson, to the sovereignty of several small, and somewhat scattered kingdoms in Vestfold, which had come into his father’s hands through conquest and inheritance. |  (Harald, of his father, Halfdan the Black) $\rightarrow$ (harald_fairhair.Q206123, father.P22, halfdan_the_black.Q504932)
Hector_Dyer |  Born in Los Angeles, California, Hector Dyer enrolled at the Stanford University and won the IC4A championships in in 1930. |  (Hector Dyer, at, The Stanford University) $\rightarrow$ (hector_dyer.Q1364518, educated_at.P69, stanford_university.Q41506)
Henny_Magnussen |  Thanks to the support she received, she was able to practise in all the courts of Denmark from 1909, becoming the first woman to do so. |  (She, , she, Denmark) $\rightarrow$ (henny_magnussen.Q56577333, country_of_citizenship.P27, denmark.Q35)
Hina_Pervaiz_Butt |  Hina Pervaiz Butt Hina Pervaiz Butt is a Pakistani politician who was a Member of the Provincial Assembly of the Punjab, since May 2013. |  (Hina Pervaiz Butt, is, A Pakistani Politician) $\rightarrow$ (hina_pervaiz_butt.Q47918210, occupation.P106, politician.Q82955)
Isabella_Of_Valois,_Duchess_Of_Bourbon |  She was the wife of Peter I, Duke of Bourbon. |  (She, was the wife of, Peter I) $\rightarrow$ (isabella_of_valois,_duchess_of_bourbon.Q2363495, spouse.P26, peter_i,_duke_of_bourbon.Q444668)
Jahir_Butrón |  Then from 1995, he played for Peruvian Second Division side Guardia Republicana. |  (He, played for, Peruvian Second Division Side Guardia Republicana) $\rightarrow$ (jahir_butrón.Q6123259, member_of_sports_team.P54, guardia_republicana.Q5614012)
Jahir_Butrón |  Jahir Butrón Jahir Butrón Gotuzzo is a Peruvian footballer who plays as a center back. |  (Jahir Butrón Gotuzzo, is, A Peruvian Footballer) $\rightarrow$ (jahir_butrón.Q6123259, country_of_citizenship.P27, peru.Q419); (Jahir Butrón Gotuzzo, is, A Peruvian Footballer) $\rightarrow$ (jahir_butrón.Q6123259, sport.P641, association_football.Q2736)
Janina_Altman |  In 1950, Hescheles emigrated to Israel, where she eventually earned a doctorate in chemistry at Technion . |  (She, earned a doctorate in, Technion) $\rightarrow$ (janina_altman.Q22574794, educated_at.P69, technion_–_israel_institute_of_technology.Q333705)
Joan_Of_France,_Duchess_Of_Berry |  She was the second daughter of King Louis XI of France and of his second wife Charlotte of Savoy; her surviving siblings were King Charles VIII of France and Anne of France. |  (She, was the second daughter of, King Louis Xi) $\rightarrow$ (joan_of_france,_duchess_of_berry.Q236220, father.P22, louis_xi_of_france.Q8058)
John_Ogilvie_(Footballer) |  He played for Hibs in their most successful era renowned for their front five known as The Famous Five. |  (He, played for, Hibs) $\rightarrow$ (john_ogilvie_(footballer).Q6251147, member_of_sports_team.P54, hibernian_f.c..Q192597)
Johnedel_Cardel |  He moved to Sta. |  (He, moved to, Sta) $\rightarrow$ (johnedel_cardel.Q56599981, member_of_sports_team.P54, sta._lucia_realtors.Q3547751)
Joseph_W._Nega |  Joseph W. Nega Joseph W. Nega is a Judge of the United States Tax Court. |  (Joseph W. Nega, is, A Judge) $\rightarrow$ (joseph_w._nega.Q16733242, occupation.P106, judge.Q16533)
José_Alejandro_Aguilar_López |  José Alejandro Aguilar López José Alejandro Aguilar López is a Mexican politician from the National Action Party. |  (José Alejandro Aguilar López, is a Mexican politician, The National Action Party) $\rightarrow$ (josé_alejandro_aguilar_lópez.Q18385891, member_of_political_party.P102, national_action_party_(mexico).Q851087); (José Alejandro Aguilar López, is, A Mexican Politician) $\rightarrow$ (josé_alejandro_aguilar_lópez.Q18385891, country_of_citizenship.P27, mexico.Q96); (José Alejandro Aguilar López, is, A Mexican Politician) $\rightarrow$ (josé_alejandro_aguilar_lópez.Q18385891, occupation.P106, politician.Q82955)
Juan_Bautista_Azopardo |  Juan Bautista Azopardo was born in Senglea, Malta, the son of Rosina and Salvatore Azopardo. |  (Juan Bautista Azopardo, was born in, Senglea) $\rightarrow$ (juan_bautista_azopardo.Q3805864, place_of_birth.P19, senglea.Q846593); (Juan Bautista Azopardo, was born in, Malta) $\rightarrow$ (juan_bautista_azopardo.Q3805864, country_of_citizenship.P27, malta.Q233)
Kevin_Kerr_(Scottish_Footballer) |  He signed for AGOVV Apeldoorn of Netherlands’ Eerste Divisie in January 2012 after his contract in Bielefeld had expired the previous summer. |  (He, signed for, Agovv Apeldoorn) $\rightarrow$ (kevin_kerr_(scottish_footballer).Q326761, member_of_sports_team.P54, agovv_apeldoorn.Q292618)
Lachezar_Kotev |  Lachezar Kotev Lachezar Kotev is a Bulgarian footballer who plays as a midfielder for Vitosha Bistritsa. |  (Lachezar Kotev, is, A Bulgarian Footballer) $\rightarrow$ (lachezar_kotev.Q49560773, sport.P641, association_football.Q2736)
Marc_Zeno |  In 1988, Zeno was drafted in the 7th round by the Pittsburgh Steelers. |  (Zeno, by, The Pittsburgh Steelers) $\rightarrow$ (marc_zeno.Q16196353, member_of_sports_team.P54, pittsburgh_steelers.Q191477)
Marcel_Dheere |  Marcel Dheere Marcel Albert "Ching" Dheere was a Canadian professional ice hockey forward who played 11 games in the National Hockey League for the Montreal Canadiens. |  (Marcel Albert "Ching, was, A Canadian Professional Ice Hockey) $\rightarrow$ (marcel_dheere.Q6756083, country_of_citizenship.P27, canada.Q16); (Marcel Albert "Ching, was, A Canadian Professional Ice Hockey) $\rightarrow$ (marcel_dheere.Q6756083, sport.P641, ice_hockey.Q41466); (Marcel Albert "Ching, for, The Montreal Canadiens) $\rightarrow$ (marcel_dheere.Q6756083, member_of_sports_team.P54, montreal_canadiens.Q188143)
Mark_Schultz_(Wrestler) |  Mark Schultz was born in 1960 in Palo Alto, California to Dorothy Jean St. Germain and Philip Gary Schultz. |  (Mark Schultz, was born in 1960 in, Palo Alto) $\rightarrow$ (mark_schultz_(wrestler).Q3487332, place_of_birth.P19, palo_alto,_california.Q47265)
Michael_Jones_(Footballer) |  Michael Jones Michael Jones is an English footballer, who plays for Cefn Druids. |  (Michael Jones, is, An English Footballer) $\rightarrow$ (michael_jones_(footballer).Q6831707, sport.P641, association_football.Q2736)
Olga_Semenova_Tyan-Shanskaya |  She learned to play chess at the age of 18; four years later she won the Leningrad city women’s chess championship. |  (She, learned, Chess) $\rightarrow$ (olga_semenova_tyan-shanskaya.Q2392240, sport.P641, chess.Q718)
Owen_Paterson |  Owen Paterson Owen William Paterson is a British Conservative Party politician who was the Secretary of State for Environment, Food and Rural Affairs from 2012 to 2014. |  (Owen William Paterson, is, A British Conservative Party Politician) $\rightarrow$ (owen_paterson.Q197894, occupation.P106, politician.Q82955)
Pan_Gongsheng |  Pan Gongsheng Pan Gongsheng is a Chinese economist, banker, reformist and bureaucrat. |  (Pan Gongsheng, is, A Chinese Economist) $\rightarrow$ (pan_gongsheng.Q9309080, country_of_citizenship.P27, china.Q148)
Paul Carrington (American Football) |  Paul Carrington Paul Carrington is a former American football defensive end. |  (Paul Carrington, is, A Former American Football Defensive End) $\rightarrow$ (paul_carrington_(american_football).Q9057011, sport.P641, american_football.Q41323)
Peter_Jebsen |  Peter Jebsen Peter Jebsen was a Norwegian businessperson and politician. |  (Peter Jebsen, was, A Norwegian Businessperson) $\rightarrow$ (peter_jebsen.Q1774966, country_of_citizenship.P27, norway.Q20); (Peter Jebsen, was, A Norwegian Businessperson) $\rightarrow$ (peter_jebsen.Q1774966, occupation.P106, businessperson.Q43845)
Pina_Piovani |  Pina Piovani Pina Piovani was an Italian stage and film actress. |  (Pina Piovani, was, An Italian Stage And Film Actress) $\rightarrow$ (pina_piovani.Q3388583, country_of_citizenship.P27, italy.Q38)
Robert_G._Cole |  Rapidly advancing through the ranks at Fort Benning as the parachute infantry battalions were expanded to regiments, he was a lieutenant colonel commanding the 3rd Battalion of the 502nd Parachute Infantry Regiment on June 6, 1944, the date of his unit’s first combat jump. |  (He, was, A Lieutenant Colonel) $\rightarrow$ (robert_g._cole.Q969944, military_rank.P410, lieutenant_colonel.Q493898)
Robert_Holt_Carpenter |  Robert Holt Carpenter Robert Holt Carpenter was a New Zealand bookbinder, local politician, bookseller and character. |  (Robert Holt Carpenter, was, A New Zealand Bookbinder) $\rightarrow$ (robert_holt_carpenter.Q7345593, country_of_citizenship.P27, new_zealand.Q664)
Robert_Van_Boxel |  He joined amateur side FC Lisse in summer 2015 from Sparta. |  (He, joined, Amateur Side Fc Lisse) $\rightarrow$ (robert_van_boxel.Q2017870, member_of_sports_team.P54, fc_lisse.Q978325)
Rolf_Maurer |  Rolf Maurer Rolf Maurer was a Swiss road racing cyclist who competed professionally between 1960 and 1969. |  (Rolf Maurer, was, A Swiss Road Racing Cyclist) $\rightarrow$ (rolf_maurer.Q870132, country_of_citizenship.P27, switzerland.Q39); (Rolf Maurer, was, A Swiss Road Racing Cyclist) $\rightarrow$ (rolf_maurer.Q870132, sport.P641, road_bicycle_racing.Q3609)
Ron_Maudsley |  Maudsley was born in Lostock Gralam, Cheshire, and educated at Malvern College and Birmingham University, then served in the Royal Army Service Corps during World War II. |  (Maudsley, was born in, Lostock Gralam) $\rightarrow$ (ron_maudsley.Q16009653, place_of_birth.P19, lostock_gralam.Q4640202)
Safa_Al-Safi |  Safa al-Safi Safa al-Din Mohammed al-Safi is an Iraqi politician and former Justice Minister who is currently Minister of State for the Council of Representatives. |  (Mohammed Al-Safi, is, An Iraqi Politician) $\rightarrow$ (safa_al-safi.Q7398343, occupation.P106, politician.Q82955)
Salvador_Luria |  In 1959, he became chair of Microbiology at the Massachusetts Institute of Technology . |  (He, became chair of, Microbiology) $\rightarrow$ (salvador_luria.Q205667, field_of_work.P101, microbiology.Q7193)
Sanele_Vavae_Tuilagi |  He now plays for US Carcassonne. |  (He, now plays for, Us Carcassonne) $\rightarrow$ (sanele_vavae_tuilagi.Q3472139, member_of_sports_team.P54, us_carcassonne.Q3550526)
Seth_Plum |  Seth Plum Seth Lewis Plum was an English international footballer who played as a wing half. |  (Seth Lewis Plum, was, An English International Footballer) $\rightarrow$ (seth_plum.Q7456581, sport.P641, association_football.Q2736); (Seth Lewis Plum, was, An English International Footballer) $\rightarrow$ (seth_plum.Q7456581, member_of_sports_team.P54, england_national_football_team.Q47762)
Sofija_Korkutytė |  Sofija Korkutytė Sofija Korkutytė was a Lithuanian rower who won three European titles in the eights event in 1963, 1965 and 1967; she finished second in 1964 and 1966. |  (Sofija Korkutytė, was, A Lithuanian Rower) $\rightarrow$ (sofija_korkutytė.Q16014909, sport.P641, rowing_(sport).Q159354)
Thomas_Rogne |  He made his first start for Celtic at home against Dundee United in a 1–0 victory on 20 February 2010, he played very well and impressed the coach Tony Mowbray. |  (He, made his first start for, Celtic) $\rightarrow$ (thomas_rogne.Q609801, member_of_sports_team.P54, celtic_f.c..Q19593)
Uri_Avnery |  Avnery was born in Beckum, near Münster in Westphalia, as Helmut Ostermann, the youngest of four children, to a well-established German Jewish family, his father being a private banker in the town. |  (Avnery, born in, Beckum) $\rightarrow$ (uri_avnery.Q325679, place_of_birth.P19, beckum,_germany.Q2707)
Ursula_Lehr |  She was a member of the Christian Democratic Union . |  (She, was a member of, The Christian Democratic Union) $\rightarrow$ (ursula_lehr.Q290708, member_of_political_party.P102, christian_democratic_union_of_germany.Q49762)
Valdívia_(Brazilian_Footballer) |  Valdívia Wanderson Ferreira de Oliveira , commonly known as Valdívia, is a Brazilian footballer who plays as an attacking midfielder for Avaí, on loan from Internacional. |  (A Brazilian Footballer, plays as, An Attacking Midfielder) $\rightarrow$ (valdívia_(brazilian_footballer).Q18176477, position_played_on_team_/_speciality.P413, midfielder.Q193592)
Victor-François,_2Nd_Duc_De_Broglie |  His victory over Prince Ferdinand at Bergen won him the rank of marshal of France from the French King Louis XV and the title of Prince of the Holy Roman Empire "Reichsfürst" from Holy Roman Emperor Francis I. In 1759, he won the Battle of Bergen and followed that with the capture the city of Minden, later fighting at the Battle of Minden under the command of Contades, whom he would succeed in command. |  (His victory, won him the, Marshal) $\rightarrow$ (victor-françois,_2nd_duc_de_broglie.Q695929, award_received.P166, marshals_of_france.Q474485)

Figure 8: GPT-2XL on Wikidata-OIE.

ID |  Sentence | Triples to gold triples
---|---|---
BERTLARGE |  GPT-2XL
Andy_Hilbert |  In the same season he was claimed off waivers by the Pittsburgh Penguins on March 9, 2006. |  - |  (He, by, The Pittsburgh Penguins) $\rightarrow$ (andy_hilbert.Q526158, member_of_sports_team.P54, pittsburgh_penguins.Q193643)
Bob_Dylan.Q392 |  Dylan married his backup singer Carolyn Dennis on June 4, 1986. |  - |  (Dylan, married, Carolyn Dennis) $\rightarrow$ (Bob_Dylan.Q392, spouse.P26, Carolyn_Dennis.Q5045345)
Bones_Hillman |  During the hiatus of Midnight Oil , Hillman returned to New Zealand , working as a studio and live musician with Dave Dobbyn and recorded the album " Available Light " . |  (Hillman, returned, New Zealand) $\rightarrow$ (bones_hillman.Q2910026, country_of_citizenship.P27, new_zealand.Q664) |  -
Brent_Hinds |  Hinds continues to concentrate on Mastodon, with the majority of his time spent touring or in the studio. |  - |  (His Time, with, Mastodon) $\rightarrow$ (brent_hinds.Q909746, member_of.P463, mastodon_(band).Q548844)
Butch_Vig |  Their debut album, "Garbage", was an unexpected smash, selling over 4 million copies and certified double platinum in the UK, United States and Australia. |  - |  (Over 4 Million Copies, in, United States) $\rightarrow$ (butch_vig.Q451084, country_of_citizenship.P27, united_states.Q30)
C-Kan |  In addition to being known as a rapper, C-Kan is also known for supporting the free use of marijuana in México. |  - |  (Marijuana, in, México) $\rightarrow$ (c-kan.Q27734073, country_of_citizenship.P27, mexico.Q96)
Charles_Gratiot_Sr. |  Gratiot died of a stroke in St . Louis . |  (Gratiot, died, St. Louis) $\rightarrow$ (charles_gratiot_sr..Q2959257, place_of_death.P20, st._louis.Q38022) |  -
Cheryl_Chan |  Cheryl Chan Cheryl Chan Wei Ling , PBM , is a Singaporean politician . |  (Cheryl Chan Wei Ling, is, A Singaporean Politician) $\rightarrow$ (cheryl_chan.Q22003605, occupation.P106, politician.Q82955) |  -
Cyba_Audi |  Audi is an experienced moderator and presenter at local, regional, and international business events. |  - |  (Audi, is, Presenter) $\rightarrow$ (cyba_audi.Q18921358, occupation.P106, television_presenter.Q947873)
Douglas_Lenat |  He is a Fellow of the AAAS , AAAI , and Cognitive Science Society , and an editor of the J . Automated Reasoning , J . Learning Sciences , and J . Applied Ontology . |  (He, is a Fellow, Aaai) $\rightarrow$ (douglas_lenat.Q559334, member_of.P463, association_for_the_advancement_of_artificial_intelligence.Q2739680) |  -
Douglas_Lenat |  While attending the University of Pennsylvania, Lenat supported himself through programming, notably designing and developing a natural language interface to a U.S. Navy data base question–answering system serving as an early online shipboard operations manual used on US aircraft carriers. |  - |  (Lenat, ,, The University of Pennsylvania) $\rightarrow$ (douglas_lenat.Q559334, educated_at.P69, university_of_pennsylvania.Q49117)
Fred_Stewart_(Football_Manager) |  Despite managing for 39 years he only ever took charge of two clubs, Stockport County and Cardiff City, and he holds the record for longest serving manager in the history of both clubs. |  - |  (He, charge of, Stockport County) $\rightarrow$ (fred_stewart_(football_manager).Q5496345, member_of_sports_team.P54, stockport_county_f.c..Q18526)
Félix_Tanco |  Born Félix Manuel de Jesús Tanco y Bosmeniel , in Bogotá , Colombia , he arrived in Cuba at a very young age . |  (Bogotá, ,, Cuba) $\rightarrow$ (félix_tanco.Q5511649, country_of_citizenship.P27, cuba.Q241) |  -
Gary_Ablett_Jr. |  He also won his second Brownlow Medal , becoming the first Gold Coast player to win the award and the 14th player in VFL / AFL history to win it twice . |  (He, also won, His Second Brownlow Medal) $\rightarrow$ (gary_ablett_jr..Q3098509, award_received.P166, leigh_matthews_trophy.Q6519632); (He, also won, His Second Brownlow Medal) $\rightarrow$ (gary_ablett_jr..Q3098509, award_received.P166, brownlow_medal.Q3853498) |  -
Geeto_Mongol |  After meeting his would - be tag team partner , Bepo Mongol , Tattrie returned to the WWWF as Geto Mongol and The Mongols were brought to the United States in 1968 . |  (The Mongols, were, The United States) $\rightarrow$ (geeto_mongol.Q11324907, country_of_citizenship.P27, united_states.Q30) |  -
Georgi_Plekhanov |  Georgi Plekhanov Georgi Valentinovich Plekhanov was a Russian revolutionary , philosopher and a Marxist theoretician . |  (Georgi Valentinovich Plekhanov, was, A Marxist Theoretician) $\rightarrow$ (georgi_plekhanov.Q182905, occupation.P106, philosopher.Q4964182) |  -
Győző_Drozdy |  Győző Drozdy Győző Drozdy was a Hungarian teacher , journalist , and politician . |  (Győző Drozdy, was, Politician) $\rightarrow$ (győző_drozdy.Q910001, occupation.P106, politician.Q82955) |  -
Horace_Rawlins |  Rawlins died on 22 January 1935 in a nursing home in Reading , Berkshire . |  (Rawlins, died, Reading) $\rightarrow$ (horace_rawlins.Q4436711, place_of_death.P20, reading,_berkshire.Q161491) |  -
Horace_Rawlins |  He played in a professional event at Stanmore Golf Club in June 1894 but finished well out of the prizes. |  - |  (He, played in, Stanmore Golf Club) $\rightarrow$ (horace_rawlins.Q4436711, sport.P641, golf.Q5377)
Hugo_Winterhalter |  Winterhalter remained with RCA Victor until 1963, at which time he moved to Kapp; that same year, he also penned the main title theme for the film, "Diamond Head". |  - |  (Winterhalter, to, Kapp) $\rightarrow$ (hugo_winterhalter.Q1635872, record_label.P264, kapp_records.Q1728690)
Ioannis_Talianos |  He competed in the men ’ s 4 × 100 metres relay at the 1924 Summer Olympics . |  (He, competed in the, The 1924 Summer Olympics) $\rightarrow$ (ioannis_talianos.Q35692847, participant_of.P1344, 1924_summer_olympics.Q8132) |  -
Joan_Of_France,_Duchess_Of_Berry |  The nuns of the Order of the Annunciation of the Blessed Virgin Mary still maintain their way of life in monasteries in France, Belgium, Costa Rica and Poland. |  - |  (Their Way, in, France) $\rightarrow$ (joan_of_france,_duchess_of_berry.Q236220, country_of_citizenship.P27, france.Q142)
John_Maynard_Keynes |  John Maynard Keynes John Maynard Keynes , 1st Baron Keynes , was a British economist , whose ideas fundamentally changed the theory and practice of macroeconomics and the economic policies of governments . |  (John Maynard Keynes, was, A British Economist) $\rightarrow$ (john_maynard_keynes.Q9317, occupation.P106, economist.Q188094) |  -
Jonathan_Culler |  Currently , he is Class of 1916 Professor of English and Comparative Literature at Cornell University . |  (He, is, English) $\rightarrow$ (jonathan_culler.Q933332, languages_spoken,_written_or_signed.P1412, english_language.Q1860) |  -
Jordi_Cañas_Pérez |  In the regional elections of Catalonia in November 2010 , he was chosen in the primary as number three for Ciutadans ’ candidacy in the province of Barcelona . |  (The Province, of, Barcelona) $\rightarrow$ (jordi_cañas_pérez.Q557693, member_of_political_party.P102, citizens_(spanish_political_party).Q1393123) |  -
Jules_Deligny |  He competed in the freestyle lightweight event at the 1920 Summer Olympics . |  (He, competed in the, The 1920 Summer Olympics) $\rightarrow$ (jules_deligny.Q16207961, participant_of.P1344, 1920_summer_olympics.Q8128) |  -
Liz_Mcgregor |  Liz McGregor Liz McGregor is a South African author and a journalist who worked for leading South African newspapers such as the " Sunday Times " and the " Rand Daily Mail " . |  (Liz Mcgregor, is, A Journalist) $\rightarrow$ (liz_mcgregor.Q23541069, occupation.P106, journalist.Q1930187) |  -
Louise_Shropshire |  The granddaughter of slaves, Louise Shropshire was born Louise Jarrett on February 15, 1913 in Coffee County, Alabama. |  - |  (Louise Shropshire, was, Coffee County, Alabama) $\rightarrow$ (louise_shropshire.Q15430857, place_of_birth.P19, coffee_county,_alabama.Q485660)
Macit_Gürdal |  He competed in the men ’ s tournament at the 1952 Summer Olympics . |  (He, competed in the, The 1952 Summer Olympics) $\rightarrow$ (macit_gürdal.Q57313835, participant_of.P1344, 1952_summer_olympics.Q8407) |  -
Markus_Pröll |  At the beginning of his career , Pröll became a target of some ridicule because his name " Pröll " literally means something like " lout " in German . |  ("Pröll, " in, German) $\rightarrow$ (markus_pröll.Q704630, country_of_citizenship.P27, germany.Q183) |  -
Monde_Hadebe |  Their Round Four match against the saw Hadebe score the first senior try of his career in a 27 – 10 victory , and the Sharks XV again finished top of the Southern Section log . |  (His Career, ,, The Sharks Xv) $\rightarrow$ (monde_hadebe.Q6898677, member_of_sports_team.P54, sharks_(currie_cup).Q744636) |  -
Neville_Southall |  He moved on to Everton for £150,000 in 1981 and established himself as the club’s first-choice goalkeeper by the 1983–84 season. |  - |  (He, moved on to, Everton) $\rightarrow$ (neville_southall.Q436650, member_of_sports_team.P54, everton_f.c..Q5794)
Olga_Semenova_Tyan-Shanskaya |  She died in Leningrad . |  (She, died, Leningrad) $\rightarrow$ (olga_semenova_tyan-shanskaya.Q2392240, place_of_death.P20, saint_petersburg.Q656) |  -
Parvathy_Ratheesh |  Parvathy Ratheesh made her film acting debut in 2015 similar to that of her younger brother , Padmaraj Ratheesh who also made his acting debut in the same year through " Fireman " . |  (Her Film, that of her younger brother, Padmaraj Ratheesh) $\rightarrow$ (parvathy_ratheesh.Q19895785, sibling.P3373, padmaraj_ratheesh.Q19895782) |  -
Peter_Dalla_Riva |  In 1993, he was inducted into the Canadian Football Hall of Fame. |  - |  (He, was inducted into the, Fame) $\rightarrow$ (peter_dalla_riva.Q7173528, award_received.P166, canadian_football_hall_of_fame.Q3517653)
Pierre_Duhem |  Unlike many former historians , who denigrated the Middle Ages, he endeavored to show that the Roman Catholic Church had helped foster Western science in one of its most fruitful periods. |  - |  (The Roman Catholic Church, had, Foster Western Science) $\rightarrow$ (pierre_duhem.Q314172, field_of_work.P101, philosophy_of_science.Q59115)
Rafael_Arcadio_Bernal_Supelano |  He then served as bishop of the Roman Catholic Diocese of Arauca, Colombia, from 1990 to 2003 and as bishop of the Roman Catholic Diocese of Libano–Honda, Colombia, from 2003 to 2004. |  - |  (The Roman Catholic Diocese, ,, Colombia) $\rightarrow$ (rafael_arcadio_bernal_supelano.Q2126904, country_of_citizenship.P27, colombia.Q739)
Rajeshwar_Dayal |  The Government of India awarded him the second highest civilian award of the Padma Vibhushan, in 1969. |  - |  (Him, award, The Padma Vibhushan) $\rightarrow$ (rajeshwar_dayal.Q15448539, award_received.P166, padma_vibhushan.Q672392)
Ray_Johnson_(American_Football) |  Ray Johnson Raymond Robert Johnson was an American football defensive back who played three seasons in the National Football League with the Cleveland Rams and Chicago Cardinals . |  (The National Football League, back who played, Raymond Robert Johnson) $\rightarrow$ (ray_johnson_(american_football).Q21077889, sport.P641, american_football.Q41323) |  -
Sean_Hanish |  Sean Hanish Sean Hanish is an American film writer , producer and director best known for " Saint Judy " , " Return to Zero " and " Sister Cities " . |  (Sean Hanish, is, An American Film Writer) $\rightarrow$ (sean_hanish.Q19867622, country_of_citizenship.P27, united_states.Q30); (Sean Hanish, is, An American Film Writer) $\rightarrow$ (sean_hanish.Q19867622, occupation.P106, screenwriter.Q28389) |  -
Seth_Plum |  Born in Edmonton, Plum played professionally for Charlton Athletic, Chelsea and Southend United. |  - |  (Plum, played professionally for, Charlton Athletic) $\rightarrow$ (seth_plum.Q7456581, member_of_sports_team.P54, charlton_athletic_f.c..Q19462)
Seth_Plum |  Plum received his only cap for England at age 23 while playing for Charlton Athletic , starting and playing the full 90 minutes in a 4 – 1 win over France on 10 May 1923 . |  (His Only Cap, playing for, Charlton Athletic) $\rightarrow$ (seth_plum.Q7456581, member_of_sports_team.P54, charlton_athletic_f.c..Q19462) |  -
Stanislav_Nedkov |  A two - time national champion in freestyle wrestling , he has also had first - place finishes in international tournaments in Russia , Moldova , Turkey , and Bulgaria . |  (Turkey, , and, Bulgaria) $\rightarrow$ (stanislav_nedkov.Q4844375, country_of_citizenship.P27, bulgaria.Q219) |  -
Stiliyan_Petrov |  In addition he is Bulgaria’s all-time most-capped player with 105 appearances for the side. |  - |  (He, is, Bulgaria’S All-Time Most-Capped Player) $\rightarrow$ (stiliyan_petrov.Q166263, country_of_citizenship.P27, bulgaria.Q219)
Stiliyan_Petrov |  On 14 January 2010 , it was announced that Petrov had come second in Bulgaria ’ s Player of the Year . |  (Petrov, in, Bulgaria’S Player) $\rightarrow$ (stiliyan_petrov.Q166263, country_of_citizenship.P27, bulgaria.Q219) |  -
Thomas_A._Dunn |  He graduated from DePaul University College of Law in 1971. |  - |  (He, graduated from, Depaul University College) $\rightarrow$ (thomas_a._dunn.Q33082990, educated_at.P69, depaul_university_college_of_law.Q5244034)
Thomas_Scott_(Commentator) |  In 1803 , Scott left the Lock Hospital to become Rector of Aston Sandford in Buckinghamshire where he remained until his death in 1821 . |  (His Death, he remained until, Aston Sandford) $\rightarrow$ (thomas_scott_(commentator).Q7793831, place_of_death.P20, aston_sandford.Q3088093) |  -
Thor_Heyerdahl |  Thor Heyerdahl Thor Heyerdahl was a Norwegian adventurer and ethnographer with a background in zoology, botany and geography. |  - |  (Thor Heyerdahl, was, A Norwegian Adventurer) $\rightarrow$ (thor_heyerdahl.Q133622, country_of_citizenship.P27, norway.Q20)
Venkatesh_Kulkarni |  His first novel , " Naked in Deccan " , won the 1984 American Book Award of the Before Columbus Foundation and was listed among the top ten novels of the decade by the " Chicago Tribune " . |  (His First Novel, , won, The 1984 American Book Award) $\rightarrow$ (venkatesh_kulkarni.Q7920091, award_received.P166, american_book_awards.Q463606) |  -
Vitor_Castro_De_Souza |  Vitor Castro de Souza Vítor Castro de Souza , or simply Vitor Castro, is a Brazilian striker. |  - |  (Vítor Castro De Souza, is, A Brazilian Striker) $\rightarrow$ (vitor_castro_de_souza.Q5604876, position_played_on_team_/_speciality.P413, forward_(association_football).Q280658)
Vyacheslav_Tsaryov |  He made his debut in the Soviet Top League in 1990 for FC Dynamo Moscow . |  (He, for, Fc Dynamo Moscow) $\rightarrow$ (vyacheslav_tsaryov.Q4503307, member_of_sports_team.P54, fc_dynamo_moscow.Q17497) |  -
Zare_Markovski |  Zare Markovski as player played in KK Rabotnički and MZT Skopje . |  (Zare Markovski, played in, Kk Rabotnički) $\rightarrow$ (zare_markovski.Q4023877, member_of_sports_team.P54, kk_rabotnički.Q2603345) |  -

Figure 9: BERTLARGE vs. GPT-2XL on Wikidata-OIE.