# KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and
Reasoning Abilities of Language Models

Daniel Gao1, Yantao Jia1, Lei Li1, Chengzhen Fu1, Zhicheng Dou2,
Hao Jiang1, Xinyu Zhang1, Lei Chen3, Zhao Cao1
1Huawei Poisson Laboratory, Beijing, China
2Gaoling School of Artificial Intelligence, Renmin University of China,
Beijing, China
3Hong Kong University of Science and Technology, Hong Kong, China
<EMAIL_ADDRESS>{jiayantao, fuchengzhen, lilei229, jianghao66,
zhangxinyu35<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Previous works show the great potential of pre-trained language models (PLMs)
for storing a large amount of factual knowledge. However, to figure out
whether PLMs can be reliable knowledge sources and used as alternative
knowledge bases (KBs), we need to further explore some critical features of
PLMs. Firstly, knowledge memorization and identification abilities:
traditional KBs can store various types of entities and relationships; do PLMs
have a high knowledge capacity to store different types of knowledge?
Secondly, reasoning ability: a qualified knowledge source should not only
provide a collection of facts, but support a symbolic reasoner. Can PLMs
derive new knowledge based on the correlations between facts? To evaluate
these features of PLMs, we propose a benchmark, named Knowledge Memorization,
Identification, and Reasoning test (KMIR). KMIR covers 3 types of knowledge,
including general knowledge, domain-specific knowledge, and commonsense, and
provides 184,348 well-designed questions. Preliminary experiments with various
representative pre-training language models on KMIR reveal many interesting
phenomenons: 1) The memorization ability of PLMs depends more on the number of
parameters than training schemes. 2) Current PLMs are struggling to robustly
remember the facts. 3) Model compression technology retains the amount of
knowledge well, but hurts the identification and reasoning abilities. We hope
KMIR can facilitate the design of PLMs as better knowledge sources.

##  1 Introduction

Language models pre-trained on large-scale unstructured documents have proven
extremely powerful on many Natural Language Processing (NLP) tasks Brown et
al. (2020); Reimers and Gurevych (2019); Raffel et al. (2019); Sanh et al.
(2019); Peters et al. (2018); Dai et al. (2019). Many recent inspiring works
Petroni et al. (2019, 2021) further reveal the potential of the pre-trained
language models (PLM) to be an alternative to structured knowledge bases Lenat
(1995); Bollacker et al. (2008); Auer et al. (2007); Vrandecic and Krötzsch
(2014); Speer et al. (2017); Mitchell et al. (2018) in some cases, especially
at avoiding large-scale schema engineering. However, if a PLM can be a
reliable knowledge source, it must equip the essential features motivated from
human cognition Hayes et al. (2014) as: (1) Strong knowledge memorization and
identification: A PLM should precisely memorize and identify the knowledge
like those stored in a KB to facilitate the downstream language tasks such as
QA, dialogue. (2) Powerful reasoning ability: The knowledge in PLM should be
able to investigate the correlations between knowledge and easily be invoked
to infer new knowledge from existing ones.

Figure 1: KMIR covers multiple types of knowledge with diverse types of
questions to comprehensively evaluate the knowledge memorization,
identification and reasoning abilities of pre-trained language models.

Previous benchmarks, such as GLUE Wang et al. (2018) and SuperGLUE Wang et al.
(2019), mainly focus on evaluating the general language understanding
abilities, rather than knowledge related abilities. Another pioneering work,
LAMA Petroni et al. (2019), focuses on probing knowledge in language models.
LAMA mainly evaluates the knowledge memorization by asking models questions in
the form of triple completion, like Q1 and Q2 in Figure 1. However, a
qualified model is highly supposed to act more than memorizing knowledge. Can
PLMs identify different entities (e.g., Q3), identify the correctness of a
fact (e.g., Q4), or collect related triples to derive new knowledge (e.g.,
Q5)? Unfortunately, if we only examine by means of triple completion
questions, these fundamental knowledge abilities for downstream tasks are hard
to be evaluated.

Therefore, we propose a benchmark for Knowledge Memorization, Identification,
and Reasoning test (KMIR), with the idea shown in Figure 1. It is designed
motivated by the examinations for humans, like SAT and GRE, which test
different abilities with various subjects of knowledge. Specifically, KMIR
covers diverse types of knowledge referred in Davis and Marcus (2015), and
shown in Figure 1 (a), including general knowledge often referred to in human
communication, domain-specific knowledge about elementary-level natural and
social science, and commonsense of universally accepted beliefs about the
world and obvious to most humans. And we collect 192,078 knowledge triples
about these types of knowledge from publicly available knowledge sources, i.e,
WikiData and ConceptNet.

Besides, we propose a syllabus to summarize the evaluated knowledge-related
abilities in three-fold: knowledge memorization, identification, and
reasoning, as shown in Figure 1 (b). Then, according to the characteristics of
these three abilities, a template-based question generation method is
developed to automatically convert the knowledge triples into question-answer
pairs as four types of prompt-based cloze questions, including triplet
completion, entity distinction, statement checking, and predicate reasoning,
as shown in Figure 1 (c). Finally, we construct a large-scale dataset with
184,348 questions111KMIR is avaliable at github.com/KMIR2021/KMIR..

Finally, we evaluate several representative PLMs on the proposed questions,
including BERT Lan et al. (2019), ALBERT Lan et al. (2019), RoBERTa Liu et al.
(2019), etc. Some interesting phenomena are found: 1) The memorization
abilities of PLMs depend more on the number of parameters than training
schemes. 2) Model compression technology retains the amount of knowledge well,
but hurts the reasoning ability. 3) Current PLMs are struggling to robustly
remember the facts. The predictions of questions derived from same facts
usually lack consistency. 4) We find PLMs can solve reasoning questions better
in the case that the entities contained in the question are highly co-occurred
with the correct answers in the documents of Wikipedia. This indicates models
who can answer reasoning questions in that they just memorized context
information by a large-amount of parameters.

In summary, our paper mainly has three contributions:

  * •

We propose a systematic evaluation syllabus. It comprehensively measures the
knowledge-related abilities of PLMs, including knowledge memorization,
identification, and reasoning. Moreover, it covers diverse types of knowledge
including general knowledge, commonsense as well as domain-specific knowledge.

  * •

We develop a template-based method that can convert knowledge triples to
diverse cloze questions, and obtain a large-scale dataset containing 184,348
QA samples. An intensive validation for quality control of samples is carried
with more than 532 hours.

  * •

We evaluate several representative PLMs on KMIR, and find many interesting
phenomena that could be useful for designing better PLMs.

##  2 Related Work

Language Understanding Benchmarks. Previous NLP benchmarks are usually for
evaluate general language understanding, such as slot filling Elsahar et al.
(2019); Levy et al. (2017), QA Yang et al. (2018); Rajpurkar et al. (2016);
Joshi et al. (2017); Fan et al. (2019); Kwiatkowski et al. (2019); Liu et al.
(2020); Ding et al. (2019); Clark et al. (2019); Kassner et al. (2020),
dialogue Dinan et al. (2018), entailment Williams et al. (2018); Rocktäschel
et al. (2015); Dagan et al. (2005); Morgenstern and Ortiz (2015). For example,
some question answering tasks aim to evaluate machine reading comprehension or
reason over a knowledge source, such as Wikipedia. Some other QA benchmarks,
e.g., LogiQA Liu et al. (2020) and HotpotQA Yang et al. (2018), also evaluate
the ability of deductive reasoning. These benchmarks aim to evaluate the
abilities of an NLP model on a specific task. Recently, GLUE Wang et al.
(2018) is proposed to measure the models’ universal language understanding. It
provides a collection of tasks set to comprehensively evaluate a model.
Further, a more difficult version, SuperGLUE Wang et al. (2019) is proposed to
cope with the increasing performance of PLMs. LReasoner Wang et al. (2021) is
another universal language understanding benchmark, but focuses more on the
robustness of PLMs. Compared to the above benchmarks evaluating the PLMs in
terms of general downstream language tasks, KMIR examines PLMs in terms of
knowledge-related abilities with diverse types of knowledge, and somehow
measures the qualities of PLMs as knowledge sources.

Knowledge Probe Benchmark. Recently, pre-trained language model Devlin et al.
(2019); Radford et al. (2019); Lewis et al. (2020) has made significant
progress in the field of natural language processing and has gradually become
one of the mainstream paradigms. A part of the research Roberts et al.
(2020b); Petroni et al. (2019, 2021); Jiang et al. (2020); Roberts et al.
(2020a); Yu et al. (2020) has begun to explore the factual knowledge contained
in pre-trained language models. LAMA Petroni et al. (2019) is the first work
to explore whether a language model learns and stores some factual knowledge
by pre-training on a large text corpus. To address this problem, LAMA uses the
triple completion task to probe whether a language model contains a certain
knowledge. If a PLM can successfully predict the words that have been masked
in a knowledge triple, then the PLM is assumed to include this knowledge. KILT
Petroni et al. (2021) is another benchmark by testing the performance of PLMs
on the downstream knowledge-intensive tasks, such as open-domain question
answering, fact-checking, and slot filling. Previous works mainly focus on
probing PLMs whether they can memorize knowledge or perform well on downstream
tasks. However, knowledge-related abilities are far beyond memorization. Thus,
this work proposes a benchmark with diverse questions to evaluate
memorization, identification, and reasoning ability.

##  3 Knowledge Abilities Syllabus

To test the ability of PLMs, similar to the syllabus of SAT
222https://leverageedu.com/blog/sat-syllabus/ designed for humans, in this
section, we elaborate the design of the syllabus in terms of three
perspectives, namely, 1) knowledge memorization; 2) knowledge identification,
and 3) knowledge reasoning. These abilities correspond to _System 1_ and
_System 2_ of the brain’s reasoning system and are the basis of daily tasks
Kahneman (2011). In fact, _System 1_ covers the basic intuition and instinct,
requiring excellent memorization and identification ability to accomplish
simple routine works. _System 2_ is responsible for rational thinking and
reasoning for complex decision-making activities.

Table 1: Knowledge Collection Schema. It displays the collected knowledge
types, corresponding entity types and their examples.

Knowledge Type | Entity Type | Example
---|---|---
Domain-Specific Knowledge | Physics | physical quantity, state of matter | The melting point of water is 0 oC.
Biology | biosystematics, biology habit | A carnivore requires meat to survive.
Chemistry | periodic table | Halogen is a Group 17 Element.
Medicine | pathology,drug effect | The symptoms of frailty syndrome are osteoporosis.
History | cities in history, historical events | The capital of Serbian Despotate is located in the country Montenegro now.
Law | articles of law | Treaty of Orihuela is a legislative act in the country Spain.
Military | weapon | M107 served during the war or conflict The Troubles.
General Knowledge | Music | musician, album, song | Lady Gaga is a singer born on 1986 March 28.
Film and TV | actor, film, theme song | The series The Wide Country’s first broadcast was on 1962 September 20.
Commonsense | properties of common objects | Rocks can be heavy.

Knowledge Memorization. Memorization characterizes the ability of remembering
something about triples in the form of $<$ subject, predicate, object $>$ ,
where subject and object represent two entities, and predicate represents a
relation between them. Similar to LAMA Petroni et al. (2019), we evaluate the
models’ memorization by triple completion task, that is, given two elements of
a triple, the model is required to recover the remaining entity or relation.
For example, given Joe Biden and the predicate birthday, it examines whether
you remember the object November 20, 1942.

Knowledge Identification. Identification evaluates whether the PLMs can
correctly recognize an entity or a triple. For entity-level identification,
KMIR requires the model to distinguish the difference between entities in
order to measure the representation ability of entities. For example, for
several entities Joe Biden, Barack Obama, Donald Trump, and Confucius, in
terms of the occupation, one might choose Confucius because the others are
presidents of the U.S. To test such ability, we define a novel type of
question called entity distinction. Given a set of entities, where one entity
is different from the others, the model is supposed to pick the different one.
For triple-level identification, the PLMs should tell whether a statement is
true. This evaluation is rather important for current models, because many
works Marcus (2020); Lacker (2020) have discovered the phenomenon of
fabricating facts by the existing language models, e.g., GPT-3 Brown et al.
(2020) tells that the sun has one eye. For this reason, we design the
questions called statement checking to validate the model’s correctness toward
a statement.

Knowledge Reasoning. KMIR measures the reasoning ability by checking whether
the model can infer new knowledge from existing one. One of the most common
knowledge reasoning paradigms for humans is predicate logical reasoning
Atkinson (1909). Thus, KMIR evaluates if the PLMs can perform basic predicate
reasoning. Specifically, we refer to OWL2333https://www.w3.org/TR/owl2-syntax/
to require the model to infer new facts (if any) through classic object
property axioms, including symmetry, transitivity, reflexive, equivalent,
inverse, sub-relation, and SubOP(OPChain). Let us take an example for the
axiom of transitivity, which is a general form of syllogism. Given two triples
$<$Michael Jackson, isA, singer$>$ and $<$singer, CapableOf, hold a
concert$>$, the model is supposed to infer $<$Michael Jackson, CapableOf, hold
a concert$>$ (more details of axioms used in KMIR are listed in Sec. 4.3). To
evaluate such inference ability, we automatically generate a lot of new
knowledge equipped in the form of questions, then ask models to answer them.

Knowledge Type. So far, we have described the syllabus in terms of the
abilities and related question types. Our syllabus also defines what types of
knowledge are used for supporting the evaluation of such abilities. Here we
follow the categorization of knowledge types defined in Davis and Marcus
(2015); Storks et al. (2019); Cambria et al. (2011), where the knowledge can
be briefly classified into three types: general knowledge, commonsense, and
domain-specific knowledge. The first two types which are often referred to in
human communication, e.g., place of birth, country of citizenship, etc., are
highly emphasized by previous benchmarks. They are suitable for measuring
memorization and identification ability. In order to further promote the
evaluation of reasoning ability, the domain-specific knowledge is particularly
considered into our evaluation in the manner of elementary-level natural
science and social science knowledge to generate more diverse knowledge.

##  4 Dataset Construction

KMIR is constructed by first collecting knowledge triples in diverse types and
then converting them into cloze statements to test the language model.
Specifically, the whole procedure contains four steps: 1) design knowledge
collection schema, 2) collect knowledge triples, 3) generate questions, and 4)
check the quality of questions. In the following, we illustrate each step in
detail.

###  4.1 Knowledge Collection Schema Design

In the step of designing knowledge collection schema, we define the collected
types of knowledge and types of entities. The idea of schema design is
choosing a broad range of knowledge types to support the evaluation of
abilities in our syllabus. Specifically, the knowledge in KMIR covers three
categories as stated in previous section, i.e., general knowledge, domain-
specific knowledge, and commonsense. _General knowledge_ collected in KMIR is
mainly about entertainment information, including music and film & TV
knowledge. We collect the producer, release date, etc., of entities like
famous songs, TV series or films, and basic information of entities like
famous celebrities, etc. _Domain-specific knowledge_ in KMIR focuses on common
elementary-level natural and social scientific knowledge, including physics,
biology, chemistry, medicine, history, law, and military. We collect entities
about basic scientific concepts for these domains, e.g., common physical
quantities of substances, taxonomic knowledge of creatures, chemistry
elements, medical drugs, cities in the history, articles of laws, information
of weapons, etc. _Commonsense_ is mainly about properties of common objects,
social conventions, etc. For instance, rocks can be heavy. Table 1 displays
examples of all knowledge types and related entity types in KMIR.

Table 2: Question templates for different knowledge types in KMIR.

Knowledge Type | Relation Type | Template | Question
---|---|---|---
Domain-Specific Knowledge | comparison | Which of the following has <Comparison><Physics Quantities>? <Choices> [MASK] |  Which of the following has a higher melting point? …
capableOf | Which of the following can <capableOf>? <Choices>, [MASK]. |  Which of the following can be an element or an alloy? …
melting point | The melting point of <chemical element>is [MASK] °C. |  The melting point of water is [MASK] °C
comparison | Which of the following has <Comparison><Olympics games>? <Choices>, [MASK]. |  Which of the following country held more times of the summer Olympics games? Canada, Switzerland, …, [MASK].
capital of | <country>’s capital is [MASK]. |  United States of America’s capital is [MASK].
General Knowledge | grandfather of | Which of the following is the grandfather of <human>? <Choices>, [MASK]. |  Which of the following is the grandfather of Isaac Newton …
spouseOf | <human> is spouse of [MASK]. |  Obama is spouse of [MASK].
Commonsense | capableOf | [MASK] can <Ability>. |  [MASK] can supply humans with milk.

###  4.2 Knowledge Collection

After obtaining the schema, we collect the three types of knowledge from two
widely used knowledge bases, WikiData Vrandecic and Krötzsch (2014) and
ConceptNet Speer et al. (2017).

WikiData. The general and domain-specific knowledge are collected from
Wikidata. Specifically, we use the SPARQL query statement on the Wikidata
Query Service platform 444https://query.wikidata.org/ to retrieve the
knowledge triples about interested entities.

ConceptNet. The commonsense knowledge is collected from ConceptNet by using
its official API 555https://github.com/commonsense/conceptnet5/wiki/API to
extract the surface texts of interested entities, and then obtain the triples
with relations depicting the properties of some common objects. We consider 7
relations: AtLocation, IsA, PartOf, HasA, HasProperty, UsedFor, and Causes.

In summary, by querying 95,532 interested entities with types defined in Table
1 from the two knowledge bases, we obtain 192,078 knowledge triples covering
diverse types of knowledge.

###  4.3 Question Generation

KMIR contains 4 types of cloze questions, that is, triple completion, entity
distinction, statement checking, and predicate reasoning, to evaluate the
abilities in the proposed syllabus.

Triple Completion Questions. To obtain questions evaluating knowledge
memorization, KMIR uses a template-based method to convert the collected
knowledge into cloze statements. We define a set of templates to convert
triple into a natural language sentence based on its relation type and mask
one entity to query the model. For example, the question template of birthday
relation triple $<$x, birthday, y$>$ is "[x]’s birthday is [MASK]". Besides,
motivated by Jiang et al. (2020), each relation type corresponds to multiple
versions of templates as prompts to make the questions more diverse. In Table
2, we show some templates in KMIR. Note that, because subjects or objects in a
triple could be multi-words, the models are supposed to tackle such cases that
the [MASK] could be filled with multiple tokens.

Entity Distinction Questions. For entity distinction questions, KMIR provides
natural language cloze-form prompts to require the model to pick one unique
entity among the given four entities. For example, "James Cameron, Martin
Scorsese, Peter Jackson, and Albert Einstein, which is an outlier in terms of
occupation? [MASK]". In KMIR, the following types of entities are used for
generating entity distinction questions: celebrities, famous scientists, chief
and associate justices, chemical elements, species. We build hierarchical
trees to category these entities based on their occupations, chemical
properties, or biology taxonomy depicted in WikiData. Then, for generating
such questions, we randomly sample three entities belonging to the same
category and one entity from another. Finally, we design a set of templates to
generate a cloze prompt based on these selected entities.

Statement Checking Questions. For statement checking questions, we
automatically generate corrupted knowledge triples to test if the models can
distinguish them. Specifically, we replace the object or subject in a
knowledge triple with other entities in the same relation to make up a
corrupted one. For example, given a correct triple, $<$water, melting point, 0
oC $>$, we replace the object 0 oC to 1538 oC which is the object in a similar
triple $<$iron, melting point, 1538 oC$>$. Remark that the corruption is
further improved by filtering out the corrupted triples which have appeared in
knowledge base. Finally, in KMIR, 5,000 triples are selected from our
collected triples to generate incorrect triples, and another 5,000 triples are
selected as correct triples. Finally, we design some cloze-form prompts to
query the model to check their correctness, e.g., "Is the statement ‘the
melting point of water is 1538 oC’ true? [MASK]".

Predicate Reasoning Questions. For evaluating predicate logic reasoning, we
firstly generate some new knowledge through rules based on OWL 2 Web Ontology
Language Axioms Motik et al. (2009). Then, we convert new knowledge to cloze
questions. 6 types of axioms out of 14 are selected, and we design
corresponding rules of knowledge generation for each axiom, as shown in Table
3. For example, to perform TransitiveOP, we sample commonsense knowledge about
the occupation, e.g., (singer, CapableOf, hold a concert) from ConceptNet, and
general knowledge from WikiData, e.g., (Michael Jackson, Is, singer), then
infer the new knowledge (Michael Jackson, CapableOf, hold a concert).

Table 3: Rules for generating new knowledge. $x$, $y$, $z$ are entity
variables. $r,r_{1},r_{2}$ indicate predicates.

Ontology Language Axiom | Rule Form
---|---
SymmetricOP | $(x,r,y)\rightarrow(y,r,x)$
TransitiveOP | $(x,r,y),(y,r,z)\rightarrow(x,r,z)$
EquivalentOP | $(x,r_{1},y)\leftrightarrow(x,r_{2},y)$
InverseOP | $(x,r_{1},y)\rightarrow(y,r_{2},x)$
Sub-relation OP | $(x,r_{1},y)\rightarrow(x,r_{2},y)$
SubOP(OPChain) | $(x,r_{1},y),(y,r_{2},z)\rightarrow(x,r,z)$

After obtaining the new knowledge, the questions can be generated in two ways.
Some questions derived from new knowledge are obtained by simply masking one
entity. However, in some cases, whether the subject or object in a triple is
masked, the answer is not always unique. To avoid the multiple answers for
such questions, we particularly design the cloze-form prompts by adding some
incorrect answers. For example, the question might be "Which person is most
likely to hold a concert, Michael Jackson, Albert Einstein, James Cameron, or
Martin Scorsese?", where the last three entities are incorrect ones. In this
case, the correct answer (i.e., Michael Jackson) is unique. In total, we
collect 184,348 questions.

###  4.4 Quality Control

The automatic data generation method inevitably introduces some noises or
controversial information into data, which could cause questions involving
sensitive or unethical information and ambiguous answers. These noises are
mainly derived from the core stages of question generation, including
knowledge collection, template design, and automatic answer generation.

Thus, we invite a group of annotators to manually check each sample’s
qualities in three main steps: 1) Filter out questions involving violence,
pornography, discrimination, sovereignty disputes issues. 2) Correct the
ambiguous questions by refining question templates. Sometimes, ambiguity is
introduced by improperly querying the information. For example, for knowledge
"Michael Jackson is born in USA.", if we query it by "Michael Jackson is born
in [MASK].", the answer could be USA, Pennsylvania, etc. Thus, we refine the
questions by introducing some phrases to restrict the answer scope, e.g.,
"_The country_ that Michael Jackson is born in is [MASK]". 3) Filter out
triple completion questions whose answers are not unique. This is because in
some cases, adding phrases still can not make the answer unique. For example,
for the question "A [MASK] doesn’t want a sore throat", the answer could be
multiple, such as "human" or "singer", etc. This situation is very common in
commonsense related questions. In summary, we randomly selected 16,000
questions to check and cost about 532 hours in total.

###  4.5 Evaluation Metric

As mentioned above, the answers in KMIR could be multiple tokens. KMIR hopes
to measure how much the predictions coincide with the golden answers. Thus,
following the commonly used metric Rajpurkar et al. (2016), for each question,
we split the prediction and correct answer into words separately, then
calculate F1 measure between them.

##  5 Experiments

Table 4: F1 scores (%) of models on different question types. Note Random
Guess performs 18.75% on predicate reasoning because these questions are a
combination of multi-choice style clozes and open-end style clozes.

Method |  | Question Type
---|---|---
Name | # Params | Training Corpus |  | Triple Completion | Statement Checking | Entity Distinction | Predicate Reasoning
Random Guess | - | - |  | 0.00017 | 50.00 | 25.00 | 18.75
BERT | 110M | BookCorpus & Wikipedia |  | 15.34 | 61.57 | 30.01 | 26.82
ELECTRA | 110M |  | 14.98 | 63.09 | 33.27 | 27.52
DistilBERT | 66M |  | 14.37 | 59.89 | 28.56 | 24.38
ALBERT | 11M |  | 11.79 | 63.42 | 32.46 | 27.16
RoBERTa | 125M |  BookCorpus, Wikipedia, OpenWebText, etc. |  | 15.50 | 66.13 | 35.92 | 29.19
DistilRoBERTa | 82M |  | 14.64 | 63.02 | 32.67 | 27.02

###  5.1 Baselines

We evaluate 3 groups of models, including simple baseline and state-of-the-art
methods to investigate current PLMs’ performances.

  * •

Random Guess: This baseline aims to show the performance if we randomly pick a
choice for open-ended stype cloze among all candidates answers, and for multi-
choice style cloze among the choices shown in the questions.

  * •

BERT Devlin et al. (2019) & its variants: BERT is a representative of a
bidirectional encoder of Transformer. We also evaluate DistilBERT Sanh et al.
(2019) to investigate the impact of PLM compression, ALBERT Lan et al. (2019)
which is an enhanced version of BERT by introducing parameter-reduction
techniques and a self-supervised loss, and ELECTRA Clark et al. (2020) which
improves the BERT by introducing a more sample-efficient pre-training task
called replaced token detection.

  * •

RoBERTa Liu et al. (2019) & its variants: RoBERTa improves BERT’s training
schema, e.g., larger batch size, sequence length. We also evaluate its
compressed version, DistilRoBERTa Sanh et al. (2019).

All the PLMs use the base version, and the train, dev, test set with 20,544,
144,085, 19,719 questions, respectively.

###  5.2 Results

Overview. Table 4 summarizes the performances of the above baselines and
display their number of parameters and training corpus. It can be seen that
RoBERTa achieves the highest scores on all types of questions, but the results
of all models are far from satisfactory (all below 70%). It is still a
formidable challenge for PLMs to achieve high memorization, identification,
and reasoning abilities. In the following, we provide more analysis about the
results.

The Performances on Different Abilities. Table 4 shows the performances of
baselines on different abilities. First, it should be noted that the values of
the F1 score on identification (i.e., statement checking and entity
distinction) and reasoning questions are relatively larger than memorization
questions. It doesn’t mean the reasoning is easier than memorization. This is
because identification and reasoning questions contain some multi-choice-like
cloze, while memorization questions have to provide an open-end answer. So,
the answer spaces of different types of questions are different, and the
scores between different abilities are incomparable.

Besides, from the ranking of models on triple completion scores, we can see
that the largest PLMs (e.g., BERT, ELECTRA, and RoBERTa), achieve the highest
scores, while the lightest weight PLM (i.e., ALBERT), obtains an obvious
performance drop. It shows that the models’ performances on triple completion
questions (i.e., memorization questions) are related to the number of
parameters. Besides, the enhanced versions of BERT (i.e., ELECTRA, ALBERT,
RoBERTa) don’t bring significantly improvements on memorization ability, but
more obvious improvements on identification (i.e., statement checking and
entity distinction) and reasoning ability. This shows that improving the
architecture or training schema of the BERT model mainly affects the knowledge
identification and reasoning ability.

The Impact of Model Compression. In Table 4, we compare the performances of
models before and after model compression, e.g., BERT/RoBERTa vs.
DistilBERT/DistilRoBERTa. We can see that model compression doesn’t cause an
obvious performance drop on memorization performance, but it hurts the
performances more on the identification and reasoning questions. This result
may indicate that the parameters of PLMs could not only store the knowledge,
but also store the relationship between knowledge. It suggests that we may
need other constraints or specific designs to maintain the reasoning ability
when compressing the model parameters.

Table 5: The prediction robustness F1 scores (%) of models on Q-sets.

Method | All Correct | Partially Correct | All Wrong
---|---|---|---
BERT | 16.37 | 54.28 | 29.35
RoBERTa | 17.65 | 55.36 | 26.99

Table 6: F1 scores (%) of models on questions involving different types of
knowledge.

| Question Type | Domain-Specific Knowledge |  | General Knowledge |  | Commonsense | Overall
---|---|---|---|---|---|---|---
Physics | Biology | Chemistry | Medicine | History | Law | Military |  | Music | Film and TV |
BERT | Triple Completion | 5.95 | 19.28 | 12.10 | 11.30 | 19.13 | 12.46 | 30.86 |  | 13.59 | 10.31 |  | 5.71 | 15.34
Statement Checking | 51.26 | 61.34 | 53.98 | 57.27 | 64.57 | 66.32 | 71.23 |  | 61.81 | 56.35 |  | 55.95 | 61.57
Reasoning | 10.74 | 11.42 | 27.78 | 14.45 | 25.35 | 41.53 | 18.79 |  | 41.38 | 50.64 |  | 24.56 | 26.82
RoBERTa | Triple Completion | 5.78 | 20.34 | 11.71 | 11.04 | 19.51 | 11.67 | 31.03 |  | 15.86 | 10.29 |  | 5.02 | 15.50
Statement Checking | 50.42 | 70.00 | 55.65 | 56.36 | 73.79 | 68.37 | 74.07 |  | 57.80 | 61.11 |  | 58.33 | 66.13
Reasoning | 14.51 | 12.81 | 28.05 | 23.34 | 27.92 | 41.99 | 23.49 |  | 42.39 | 55.30 |  | 27.62 | 29.19

Robustness of PLM Prediction. In KMIR, given one knowledge triple, we
construct multiple types of cloze questions to investigate different
knowledge-related abilities. For example, for a triple <Joe Biden, birthday,
November 20, 1942>, the questions might be "Joe Biden’s birthday is [mask]"
and "Joe Biden’s birthday is November 20, 1942, is that right?" A robust model
needs to be able to maintain the high consistency of the answers to these
highly relevant questions. Thus, we group the questions derived from the same
knowledge triple together as one question set (denoted as Q-set). Then, we
calculate how many q-sets are partially correctly as shown in Table 5. It can
be seen that more than 50% of Q-sets for BERT and RoBERTa are partially
correctly answered. This result reflects that current models are hard to
robustly store the knowledge and validate whether the saved knowledge is
correct.

Figure 2: The correlations between F1 scores of BERT/RoBERTa and different ES
scores on questions generated by the new knowledge under the rules
TransitiveOP and SubOP(OPChain).

Can the PLM really Reason? KMIR uses a rule-based method to generate a large
amount of knowledge that is not explicitly expressed in the training corpus.
From results in Table 4, PLMs perform relatively well on these reasoning
questions. Is PLM really able to infer new knowledge, or does it simply
memorize the co-occurrence between words? For example, although no Wikipedia
page directly tells Michael Jackson can record a song, the words _Michael
Jackson_ and _song_ might appear within sentences or paragraphs many times.
Thus, we need investigate whether PLMs mainly learn co-occurrence between
tokens rather than new knowledge.

We establish an Elastic Search system Divya and Goyal (2013) by indexing all
Wikipedia pages , and then compute the co-occurrence score (ES score) between
entities in a query and the words in the corresponding answer appearing in
Wikipedia pages. We display the correlation between performances of models on
questions where the new knowledge is generated by rules TransitiveOP and
SubOP(OPChain), and the ES scores in Figure 2. It can be seen that, in
general, current models perform better on questions with higher ES scores.
This suggests that models who can answer reasoning questions in that they just
memorized context information by a large-amount of parameters. However, the
co-occurrence cannot solve all reasoning paradigms. We believe that there is
still much room for explicit reasoning for existing PLMs.

The Performances on Different Knowledge Types. Finally, it remains to check
the performances of all models with respect to different types of knowledge.
From Table 6, we can see that the performances vary largely in different types
of knowledge. Note that, the entity distinction questions are not listed
because the entities in one question involve multiple types of knowledge. The
results indicate that performances over knowledge-related abilities are
different among knowledge types. The memorization of commonsense and physics
is relatively harder. Reasoning questions about domain-specific knowledge are
more difficult than general knowledge. These results also show that current
models are hard to precisely memorize the facts and implement complex
reasoning.

##  6 Conclusion

This paper introduces KMIR benchmark for evaluating knowledge memorization,
identification, and reasoning abilities. KMIR includes a systematic evaluation
syllabus to summarize the knowledge-related abilities of PLMs, and has 184,348
questions involving 4 types of questions covering 3 types of knowledge. We
also find many interesting phenomena through extensive experiments: 1) The
memorization ability of PLMs depends more on the number of parameters than
training schemes. 2) Current PLMs are struggling to robustly remember the
facts. 3) Model compression technology retains the amount of knowledge well,
but hurts the identification and reasoning ability, etc.

## References

  * Atkinson (1909) William Walker Atkinson. 1909.  _The Art of Logical Thinking: Or, The Laws of Reasoning_.  Bibliotech Press.
  * Auer et al. (2007) S. Auer, C. Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Z. Ives. 2007.  Dbpedia: A nucleus for a web of open data.  In _ISWC/ASWC_.
  * Bollacker et al. (2008) K. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008\.  Freebase: a collaboratively created graph database for structuring human knowledge.  In _SIGMOD Conference_.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Cambria et al. (2011) Erik Cambria, Yangqiu Song, Haixun Wang, and Amir Hussain. 2011.  Isanette: A common and common sense knowledge base for opinion mining.  In _2011 IEEE 11th International Conference on Data Mining Workshops_ , pages 315–322. IEEE.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  _arXiv preprint arXiv:1905.10044_.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.  The pascal recognising textual entailment challenge.  In _Machine Learning Challenges Workshop_ , pages 177–190. Springer.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019.  Transformer-xl: Attentive language models beyond a fixed-length context.  _arXiv preprint arXiv:1901.02860_.
  * Davis and Marcus (2015) Ernest Davis and Gary Marcus. 2015.  Commonsense reasoning and commonsense knowledge in artificial intelligence.  _Communications of the ACM_ , 58(9):92–103.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4171–4186.
  * Dinan et al. (2018) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018.  Wizard of wikipedia: Knowledge-powered conversational agents.  In _International Conference on Learning Representations_.
  * Ding et al. (2019) Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. 2019.  Cognitive graph for multi-hop reading comprehension at scale.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2694–2703.
  * Divya and Goyal (2013) Manda Sai Divya and Shiv Kumar Goyal. 2013.  Elasticsearch: An advanced and quick search technique to handle voluminous data.  _Compusoft_ , 2(6):171.
  * Elsahar et al. (2019) Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Elena Simperl, and Frederique Laforest. 2019.  T-rex: A large scale alignment of natural language with knowledge base triples.
  * Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019.  Eli5: Long form question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3558–3567.
  * Hayes et al. (2014) Brett K Hayes, Evan Heit, and Caren M Rotello. 2014.  Memory, reasoning, and categorization: Parallels and common mechanisms.  _Frontiers in psychology_ , 5:529.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020.  How can we know what language models know?  _Transactions of the Association for Computational Linguistics_ , 8:423–438.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611.
  * Kahneman (2011) Daniel Kahneman. 2011.  _Thinking, fast and slow_.  Macmillan.
  * Kassner et al. (2020) Nora Kassner, Benno Krojer, and Hinrich Schütze. 2020.  Are pretrained language models symbolic reasoners over knowledge?  _arXiv preprint arXiv:2006.10413_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Lacker (2020) Kevin Lacker. 2020.  Giving gpt-3 a turing test.  _https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html_.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  In _International Conference on Learning Representations_.
  * Lenat (1995) Douglas B Lenat. 1995.  Cyc: A large-scale investment in knowledge infrastructure.  _Communications of the ACM_ , 38(11):33–38.
  * Levy et al. (2017) Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017.  Zero-shot relation extraction via reading comprehension.  In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_ , pages 333–342.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880.
  * Liu et al. (2020) Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020\.  Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.  In _Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)_ , pages 3622–3628.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Marcus (2020) Gary Marcus. 2020.  The next decade in ai: four steps towards robust artificial intelligence.  _arXiv preprint arXiv:2002.06177_.
  * Mitchell et al. (2018) Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, Bryan Kisiel, et al. 2018.  Never-ending learning.  _Communications of the ACM_ , 61(5):103–115.
  * Morgenstern and Ortiz (2015) Leora Morgenstern and Charles Ortiz. 2015.  The winograd schema challenge: Evaluating progress in commonsense reasoning.  In _Twenty-Seventh IAAI Conference_.
  * Motik et al. (2009) Boris Motik, Peter F Patel-Schneider, Bijan Parsia, Conrad Bock, Achille Fokoue, Peter Haase, Rinke Hoekstra, Ian Horrocks, Alan Ruttenberg, Uli Sattler, et al. 2009.  Owl 2 web ontology language: Structural specification and functional-style syntax.  _W3C recommendation_ , 27(65):159.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237.
  * Petroni et al. (2021) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Rocktaschel, and Sebastian Riedel. 2021.  Kilt: a benchmark for knowledge intensive language tasks.  _ArXiv_ , abs/2009.02252.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Patrick Lewis, A. Bakhtin, Yuxiang Wu, Alexander H. Miller, and S. Riedel. 2019.  Language models as knowledge bases?  In _EMNLP_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992.
  * Roberts et al. (2020a) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020a.  How much knowledge can you pack into the parameters of a language model?  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426.
  * Roberts et al. (2020b) Adam Roberts, Colin Raffel, and Noam M. Shazeer. 2020b.  How much knowledge can you pack into the parameters of a language model?  In _EMNLP_.
  * Rocktäschel et al. (2015) Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. 2015.  Reasoning about entailment with neural attention.  _arXiv preprint arXiv:1509.06664_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Speer et al. (2017) R. Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _AAAI_.
  * Storks et al. (2019) Shane Storks, Qiaozi Gao, and Joyce Y Chai. 2019.  Recent advances in natural language inference: A survey of benchmarks, resources, and approaches.  _arXiv preprint arXiv:1904.01172_.
  * Vrandecic and Krötzsch (2014) Denny Vrandecic and M. Krötzsch. 2014.  Wikidata: a free collaborative knowledgebase.  _Commun. ACM_ , 57:78–85.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019.  Superglue: a stickier benchmark for general-purpose language understanding systems.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , pages 3266–3280.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355.
  * Wang et al. (2021) Siyuan Wang, Wanjun Zhong, Duyu Tang, Zhongyu Wei, Zhihao Fan, Daxin Jiang, Ming Zhou, and Nan Duan. 2021.  Logic-driven context extension and data augmentation for logical reasoning of text.  _arXiv preprint arXiv:2105.03659_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2369–2380.
  * Yu et al. (2020) Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. 2020.  Jaket: Joint pre-training of knowledge graph and language understanding.  _arXiv preprint arXiv:2010.00796_.

## Appendix

In the appendix, we provide more details about dataset construction (Sec. A),
data statistics (Sec. B), implementation of baselines (Sec. C), and license
and usage of KMIR (Sec. D).

##  Appendix A Dataset Construction

In this section, we provide more examples of the selected knowledge triples
and question templates used for generating questions.

###  A.1 Knowledge Collection Schema

In the Table 7, we show more knowledge collection schema.

Table 7: Knowledge Collection Schema. This table displays our collected 10
specific types of knowledge and corresponding relation types and their
examples.

Knowledge Type | Relation Type | Example
---|---|---
Domain-specific Knowledge | Physics | physical quantity |  The melting point of water is 0 centigrade.
physical property |  A metal can conduct electricity and heat well.
state of matter |  Rocks has a stable, definite shape and volume at standard temperature and pressure.
Biology | biosystematics |  Homeothermic species maintain a stable body temperature.
biology habit |  A carnivore requires meat to survive.
Chemistry | periodic table |  Halogen is a Group 17 Element.
atomic number |  aluminium’s atomic number is 13.
Medicine | pathology |  The symptoms of frailty syndrome are osteoporosis.
cause |  HIV infection is caused by the human immunodeficiency virus.
History | cities in history |  The capital of Serbian Despotate is located in the country Montenegro now.
historical events |  The American Civil War was a civil war in the United States.
Law | articles of law |  Treaty of Orihuela is a legislative act in the country Spain.
lawer |  Willem Eduard Bok Jr. was a South African lawyer.
Military | weapon |  M107 served during the war or conflict The Troubles.
manufacturer |  Northrop Grumman Corporation is an American multinational aerospace and defense technology company.
General Knowledge | Music | musician |  Lady Gaga is a singer born on 1986 March 28.
album, song |  One Love is a song by Justin Bieber released in album Believe.
Film and TV | actor, film |  The series The Wide Country’s first broadcast was on 1962 September 20.
theme song |  The theme musics of the television series Friends is I’ll Be There for You.
Commonsense | CapableOf |  Rocks can be heavy.
IsA |  eight ball is a game of geometry.
HasProperty |  Africa can be one of the largest continents.
HasA |  The snake have a nest full of babies,scales,no legs.

###  A.2 Question Generation

In Table 8, we display the question templates used for generating the entity
distinction and statement checking questions.

Table 8: Question templates for entity distinction and statement checking
questions.

Question Type | Question Template
---|---
Entity Distinction |  Is the statement “<Statement of a knowledge triple>” true? [MASK].
<Statement of a knowledge triple>, is this true? [MASK].
Is this true, <Statement of a knowledge triple>? [MASK].
Is the following statement true, <Statement of a knowledge triple>? [MASK].
Statement Checking |  <Entity1>, <Entity2>, <Entity3>, and <Entity4>, which is a outlier, in terms of the <Aspect>? [MASK].
Which is different with others in terms of the <Aspect>, <Entity1>, <Entity2>,
<Entity3>, and <Entity4>? [MASK].
Which is different with others among the following entities in terms of the
<Aspect>, <Entity1>, <Entity2>, <Entity3>, and <Entity4>? [MASK].
Among <Entity1>, <Entity2>, <Entity3>, and <Entity4>, [MASK] is the outlier in
terms of the <Aspect>.

For reasoning questions, we first generate lots of new knowledge according OWL
2 Web Ontology Language Axioms Motik et al. (2009), then diverse question
templates are designed for each relation type of the knowledge fact. In Table
9, we display the examples of generated new knowledge for each axiom in OWL 2.
Table 10 shows our collected types of relations and corresponding templates
for triple completion and predicate reasoning questions.

Table 9: Examples of generated new knowledge for each axiom in OWL 2. $x$,
$y$, $z$ are entity variables. $r$ indicates a predicate.

Axiom | Rule Form | Example
---|---|---
SymmetricOP | $(x,r,y)\rightarrow(y,r,x)$ |  (peace, oppsite of, war) $\rightarrow$ (war, oppsite of, peace)
TransitiveOP | $(x,r,y),(y,r,z)\rightarrow(x,r,z)$ |  (cat, isA, mammal),(mammal, isA, vertebrate) $\rightarrow$ (cat, isA, vertebrate)
EquivalentOP | $(x,r_{1},y){\leftrightarrow}(x,r_{2},y)$ |  (Joe Biden, is the citizen of, USA) $\leftrightarrow$ (Joe Biden, nationality, USA)
InverseOP | $(x,r_{1},y)\rightarrow(y,r_{2},x)$ |  (USA, capital, Washington) $\rightarrow$ (Washington, country, USA)
Sub-relation OP | $(x,r_{1},y)\rightarrow(x,r_{2},y)$ |  (Gordian I, father of, Gordian II)$\rightarrow$ (Gordian I, parent of, Gordian II)
SubOP(OPChain) | $(x,r_{1},y),(y,r_{2},z)\rightarrow(x,r,z)$ |  (Napoleon, country, French),(French, continent, Europe) $\rightarrow$ (Napoleon, continent, Europe)

Table 10: Question templates for different types of knowledge in KMIR.

Knowledge Type | Relation Type | Template | Question
---|---|---|---
Domain-specific Knowledge | comparison |  Which of the following has <Comparison><Physics Quantities>? <Choices>, [MASK]. |  Which of the following has a higher boiling point? methanol,carbon tetrachloride,methyl formate,water,
capableOf |  Which of the following can <capableOf>? <Choices>, [MASK]. |  Which of the following can be an element or an alloy?
melting point |  The melting point of <chemical element>is [MASK] °C. |  The melting point of water is [MASK] °C
IsA |  Which of the following <IsA>is a <Physics Quantities>? <Choices>, [MASK]. |  which of the following is a Group 17 Element?
HasProperty |  Which of the following has <HasProperty><Physics Property>? <Choices>, [MASK]. |  Which material in the following has a stable, definite shape and volume at standard temperature and pressure?
comparison |  Which of the following has <Comparison><Olympics games> <Choices>, [MASK]. |  Which of the following country held more times of the summer Olympics games?
capital of |  <country>’s capital is [MASK]. |  United States of America’s capital is [MASK].
inception |  The historical country <historical country> was established on [MASK]. |  The Mongol Empire was established in the 1206.
countryOf |  <historical period> was a period in the country [MASK]. |  Norman architecture was a historical period of the country Fracne
General Knowledge | grandfather of |  which of the following is the grandfather of <human>? |  which of the following is the grandfather of Isaac Newton ?
spouseOf |  <human> is spouse of [MASK]. |  Obama is spouse of [mask].
FatherOf |  [MASK] is <human>’s father. |  Gordian I is Gordian II’s father.
Commonsense | capableOf |  <entity> can <capbleOf>. |  a fact can be [MASK].
OppsiteOf |  Which of the following is the oppsite of <entity> ? |  ocean is oppsite of continent.

##  Appendix B Data Statistics

Figure 3 visualizes the question distribution for different domains and
abilities. The results show that KMIR has a relatively uniform distribution
regardless of question type or knowledge type.

Figure 3: Statistics of questions in KMIR.

##  Appendix C Implementation Details of Baselines

KMIR requires the PLMs to output multiple tokens as the answer. To adjust
current PLMs to fit this setting, we modify the inputs of PLMs in fine-tuning.
Specifically, we expand the [MASK] in a query into 8 masks (the maximum number
of masked tokens in KMIR). For example, the original query "Singer Don Omar ’s
birthday is [MASK]" is transformed into "Singer Don Omar ’s birthday is [MASK]
[MASK] [MASK] …". Similarly, the correct answer is transformed into tokens,
then padded into 8 tokens for training the models. Besides, to better reflect
the knowledge-related abilities, the PLMs are fine-tuned on each type of
questions and test on corresponding test set questions.

##  Appendix D License and Usage

Firstly, in KMIR, the personnel information involved in the dataset is about
public information of public figures from WikiData and does not involve
personal privacy. Then, the questions in KMIR are under CC BY-NC-SA 4.0
license. For the knowledge base that we used to generate questions, WikiData
is licensed under the Creative Commons CC0 License, and ConceptNet is under a
Creative Commons Attribution-Share Alike 4.0 International License. Besides,
our dataset collects the knowledge from open and free knowledge base, the
knowledge used in dataset could have some mistakes or out-of-date. Finally,
our data can only be used for academic researches and do not support
commercial usages.