These authors contributed equally to this work. These authors contributed
equally to this work. [4]Feifei Wang [5]Wei Wei

1]School of Medical Technology, Beijing Institute of Technology, Zhongguancun
South Street No.5, Beijing, 100081, China

2]School of Computer Science and Technology, Beijing Institute of Technology,
No.5, Zhongguancun South Street, Beijing, 100081, China

3]Viterbi School of Engineering, University of Southern California, University
of Southern California, 3939 S Figueroa St, Los Angeles, CA, 90037, USA

4]Center for Applied Statistics and School of Statistics, Renmin University of
China, No. 59, Zhongguancun Street, Beijing, 100872, China

5]HSBC Business School, Peking University, Xueyuan Street, Shenzhen, 518055,
Guangdong China

# Knowledge-Enhanced Relation Extraction Dataset

Yucong Lin<EMAIL_ADDRESS>Hongming Xiao<EMAIL_ADDRESS>Jiani
Liu<EMAIL_ADDRESS>Zichao Lin<EMAIL_ADDRESS>Keming Lu
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>[  [  [  [  [

###### Abstract

Recently, knowledge-enhanced methods leveraging auxiliary knowledge graphs
have emerged in relation extraction, surpassing traditional text-based
approaches. However, to our best knowledge, there is currently no public
dataset available that encompasses both evidence sentences and knowledge
graphs for knowledge-enhanced relation extraction. To address this gap, we
introduce the Knowledge-Enhanced Relation Extraction Dataset (KERED). KERED
annotates each sentence with a relational fact, and it provides knowledge
context for entities through entity linking. Using our curated dataset, We
compared contemporary relation extraction methods under two prevalent task
settings: sentence-level and bag-level. The experimental result shows the
knowledge graphs provided by KERED can support knowledge-enhanced relation
extraction methods. We believe that KERED offers high-quality relation
extraction datasets with corresponding knowledge graphs for evaluating the
performance of knowledge-enhanced relation extraction methods. Our dataset is
available at: https://figshare.com/projects/KERED/134459

###### keywords:

Distant supervision, Knowledge graph, Knowledge-enhanced relation extraction,
Relation extraction

##  1 Introduction

Relation extraction (RE) focuses on extracting relationships between entities
from natural language sentences [1]. RE enhances various downstream tasks in
natural language processing, including question answering [2, 3], knowledge
graph construction [4, 5], and reading comprehension [6, 7]. Knowledge graphs
(KGs) store relational facts as triples including subject entities, object
entities, and the relations between them [8]. For instance, the relational
fact _(James Joyce, country of citizenship, Ireland)_ indicates that James
Joyce was a citizen of Ireland. As a kind of structured representation of
facts, KGs find extensive applications, such as social network analysis [9,
10] and recommender systems [11, 12].

Figure 1: An illustration of instances in a DSRE dataset and the auxiliary KG
for the dataset. An instance consists of a sentence together with a relational
fact expressed by the sentence. A batch of instances constitutes an RE
dataset. A KG contains some or all entities in an RE dataset and relations
between them. In this example, we link entities in the three instances to
their counterparts in the auxiliary KG. Then, a knowledge-enhanced RE method
can classify the relation between _Ansel Adams_ and _Photographer_ as
_Occupation_ by the virtue of the information provided by the sentences and
the KG.

Recently, KGs are widely used as auxiliary information to enhance RE methods
[13, 14, 15]. And, the availability of resources such as distantly supervised
relation extraction (DSRE) datasets [16] and extensive knowledge databases
[17, 18] has facilitated the study of knowledge-enhanced RE. However, no
public RE dataset exists that aligns sentences with the corresponding
knowledge context for training and evaluating knowledge-enhanced RE methods.
Previous researchers in this field tend to construct their own auxiliary KGs,
create datasets from scratch, and retest prior benchmarks for fair comparisons
[14, 19, 15]. The lack of public benchmarks makes it challenging to report
reproducible results or compare the performance of existing methods.

To address these issues, we adapt three widely-used RE datasets for knowledge-
enhanced RE tasks to curate the Knowledge-Enhanced Relation Extraction Dataset
(KERED). KERED improves the data quality of previous datasets. Also, with
information from external knowledge bases, KERED constructs auxiliary KGs for
entities in the dataset. We believe KERED will foster the development of
knowledge-enhanced RE in the future.

We commenced our work by examining the original DSRE datasets based on
Wikidata [17] or Freebase [18]. Because DSRE generates large-scale data by
aligning relational facts in knowledge bases with evidence sentence [16],
mentions in the corpus naturally match their entities in their corresponding
knowledge bases for distant supervision, which streamlines the process of
locating entities via their identifiers and access the knowledge context of
entities in the knowledge bases. Therefore, it is feasible to construct a KG
for each DSRE dataset, as illustrated in Figure 1. Specifically, We collected
three DSRE datasets for KERED: NYT10m [16], Wiki20m [16], and Wiki80 [20].
Then, we refined them by enhancing data quality and constructing a KG for
each. Also, we conducted comprehensive RE experiments on KERED to evaluate the
performance of existing RE methods. The experimental result shows that
information from auxiliary KGs has positive effects on RE methods. In summary,
KERED offers the first standardized datasets for knowledge-enhanced RE tasks.

Our study’s contributions are twofold. Firstly, we develop KERED, comprising
three challenging RE datasets with auxiliary KGs, with the potential to
advance knowledge-enhanced RE research. We make our datasets publicly
available on Figshare111https://figshare.com/projects/KERED/134459; please
refer to Appendix A for KERED access. Secondly, we establish metrics for
knowledge-enhanced RE methods on KERED and assess state-of-the-art RE methods
using our datasets. Our experiments indicate that knowledge-enhanced RE
methods can surpass traditional approaches.

The remainder of the paper is organized as follows: Section 2 reviews widely-
used DSRE datasets and knowledge-enhanced RE methods. Section 3 details the
construction of KERED. Section 4 presents KERED’s descriptive analysis.
Section 5 evaluates RE models on KERED. Section 6 discusses the experimental
results. Finally, Section 7 concludes the paper.

##  2 Related Works

DSRE datasets. The majority of existing DSRE datasets are constructed by
identifying entities mentioned in evidence sentences and linking them to
public knowledge bases such as Wikidata [17] and Freebase [18]. NYT10 [21] is
a large-scale dataset automatically constructed using DSRE. However, Han et
al. [20] highlighted noisy labeling issues in NYT10 and other existing
datasets. To mitigate this problem, Gao et al. [16] introduced two DSRE
datasets with manually annotated test sets, significantly enhancing the data
quality of previous NYT10 [21] and Wiki20 [22]. Although noisy labeling
problems were solved in test sets, data quality issues persist in NYT10m and
Wiki20m. Hence, we further denoised these datasets and enriched them with
external KGs through entity linking. Our refined datasets provide the
community with benchmarks for evaluating knowledge-enhanced RE methods.

Knowledge-enhanced Relation Extraction. An increasing number of RE methods
incorporate auxiliary information, such as attributes and embeddings of
entities, into their models [23], where KG information plays a crucial role by
revealing associations between entities [24]. CGRE [14] derives constraint
graphs from KGs to model intrinsic connections between relations. The model
generates representations for entities and relations by encoding the graph
into vectors and extracting node features. Xu and Barbosa [25] proposed an RE
framework HRERE which jointly learns language representations and knowledge
graph embeddings. Moreover, KGPool [15] employs a graph pooling algorithm that
dynamically selects KG context to enhance model performance. Their method
considers only names, categories, aliases, and descriptions of entities as
their side information from KG. REMAP [26], a multimodal method for DSRE,
combines knowledge graph embeddings with deep language models to classify
relations between entities. However, comparing the performance of all these
methods is impossible due to the absence of benchmarks. Thus, we revisit
existing knowledge-enhanced RE methods and evaluate them on our datasets to
facilitate an objective comparison.

##  3 Construction of KERED

We first define the problem and present an overview of previous DSRE datasets
in Sections 3.1 and 3.2. Subsequently, we detail the process of entity linking
in Sections 3.4, and then describe the process of dataset refinement in 3.5.

###  3.1 Problem Definition

An RE dataset consists of a collection of instances and a set of candidate
relations. Wherein, each instance contains an evidence sentence and an
annotated relational fact (an entity pair and a relation between them), as
illustrated in Figure 1. All relations appearing in the dataset are restricted
to the set of candidate relations. On RE datasets, traditional RE methods
predict the relation between two entities based on the sentence for each
instance. In addition to that, knowledge-enhanced RE methods typically require
an auxiliary KG as supplementary information to improve RE performance. They
categorize relations between entity pairs in evidence sentences with the aid
of knowledge base information [23, 25]. To establish datasets for knowledge-
enhanced RE tasks, we enhance the data quality of existing RE datasets and
construct KGs for them to obtain KG-enhanced RE datasets.

Our experiments evaluate RE methods at two levels. Sentence-level RE considers
only one instance as input at a time and predicts the relation of entities
expressed by the sentence. In contrast, bag-level RE takes a bag as input at a
time. The bag contains multiple instances with the same entity pairs in the
dataset, such as the three instances depicted in Figure 1. When categorizing
the relation between two entities, bag-level RE methods take into account all
instances in the bag.

The main notations used are as follows: $\mathcal{E}$ represents the entity
set containing all entities in an RE dataset. $\mathcal{R}$ represents the
relation set containing all relation labels in an RE dataset. N/A means not
applicable, which is a label type that may appear in $\mathcal{R}$.
$r^{i}=(x^{i},e_{1}^{i},e_{2}^{i})$ is the $i$-th instance in an RE dataset,
where $e^{i}_{1}$ and $e^{i}_{2}$ delimit entity mentions in token sequence
$x^{i}$. $(h^{j},t^{j},r^{j})$ is the $j$-th relation fact in a KG, where
$h^{j}$ and $t^{j}$ are subject entity and object entity with relation
$r^{j}$, respectively.

###  3.2 Previous Relation Extraction Datasets

We utilized three frequently-used relation extraction datasets: NYT10m [16],
Wiki80 [20], and Wiki20m [16]. Wiki80 is a sentence-level dataset, while
Wiki20m and NYT10m are bag-level datasets.

  * •

NYT10m [16] is a bag-level dataset derived from NYT10 [27] by cleaning the
dataset and separating the validation set from the training set. NYT10m also
provides a manually-annotated test set based on the original test set. N/A
instances in NYT10m indicate that there is no relation between entities.

  * •

Wiki80 [20] is a sentence-level dataset based on the few-shot dataset FewRel
[28]. This human-labeled dataset contains 56,000 relation facts with 80 types
of relations. N/A relation facts do not exist in Wiki80.

  * •

Wiki20m [16] is derived from Wiki20 by reorganizing its relation facts and
redividing its training, validation, and test sets. This bag-level RE dataset
shares the same relation ontology with Wiki80, except that Wiki20m contains an
N/A relation expressing unknown relations.

###  3.3 Quality Assessment of Data Source

In this section, we discuss the data quality of source datasets in KERED,
namely NYT10m, Wiki80, and Wiki20m. Subsequently, we address the data quality
of the knowledge bases we employed, specifically Freebase and Wikidata.

Wiki80 is derived from a manually-checked RE dataset FewRel [28]. Each
instance in FewRel was reviewed by at least two well-educated annotators to
filter incorrectly labeled instances generated by DSRE. Consequently, the data
quality of Wiki80 is assured. NTY10m [16] annotated all non-NA instances in
the test set, and the train/validation sets were generated directly by DSRE.
Additionally, the authors reorganized the instances in Wiki20 to construct
Wiki20m, using Wiki80 as the test set. In comparison with Wiki80, both NYT10m
and Wiki20m are large-scale datasets (approximately ten times larger than
Wiki80), making it infeasible to manually check them all. Thus, DSRE-generated
train/validation sets are acceptable for them. Moreover, as discussed in
Section 3.5, we further enhanced the data quality of NYT10m, Wiki80, and
Wiki20m. Therefore, all RE datasets in KERED are of high quality and suitable
for evaluating the performance of RE methods.

As we construct KGs for KERED using external knowledge bases, it is essential
to verify their credibility, specifically Freebase and Wikidata. [29] analyzes
five widely-used knowledge bases, providing a comprehensive framework to
evaluate their characteristics and quality. According to the authors, RDF
documents and literals in both Freebase and Wikidata are syntactically
verified by editors. Furthermore, both knowledge bases provide the provenance
of each relational fact to ensure the statement’s validity. The general
accuracy of relational facts in Freebase and Wikidata exceeds 99.3% [29]. In
summary, the accuracy and trustworthiness of the knowledge sources of KGs in
KERED are robust and guaranteed.

###  3.4 Entity Linking

Entity linking involves locating entities in a knowledge base to extract the
knowledge context for each entity, enabling the construction of a KG for a
given dataset. For KERED, we applied different entity linking methods to
NYT10m [16], Wiki80 [20], and Wiki20m [16].

NYT10m. The original NYT10 [27] dataset, constructed using Freebase [18] in a
distantly-supervised manner, allows for locating all entities in NYT10 within
Freebase by their identifiers. However, Freebase was discontinued in 2014,
preventing the extraction of specific knowledge contexts. Consequently, we
utilized FB15k [30], a Freebase [18] subset, as a knowledge base to provide
knowledge context for NYT10m. Specifically, we linked all entities in NTY10m
to their counterparts in FB15k based on identifiers and extracted relation
facts containing these linked counterparts (entities) to construct our KG.

Wiki80 and Wiki20m. The original Wiki80 and Wiki20 [31] datasets were created
by aligning entities in the English Wikipedia corpus with Wikidata [17] which
provides Wiki identifiers (unique identifiers for entities in Wikidata) for
all entities in the two datasets. As a result, we directly pinpointed entities
in Wikidata using the entities’ Wiki identifiers 222Wikidata is a real-time
updated knowledge graph, and we obtained our data on 7 Mar 2022.. Then, we
extracted relational facts containing the entities in Wikidata to construct
auxiliary KGs for Wiki80 and Wiki20m.

###  3.5 Dataset Refinement

As previous datasets suffer from data quality issues and lack unified
auxiliary KGs, dataset refinement aims to enhance the data quality of each RE
dataset and construct KGs for datasets in accordance with the knowledge base.

To provide high-quality datasets, we denoised all the original datasets. We
updated identifiers of redirected entities to the latest version and removed
instances containing missing entities in Wiki80 [20] and Wiki20m [16].
Subsequently, we eliminated duplicate instances in each dataset. We discovered
that some relation facts in Wiki80 and Wiki20m had the subject and object
referring to the same entity, such as:

$(soviets[Q15180],sovietunion[Q15180],~{}country[P17]),$

and we removed instances containing these relational facts. Eventually, in
Wiki80, there were 162 redirected identifiers, and 32 entities were removed;
in Wiki20m, there were 1,092 redirected identifiers, and 269 entities were
removed.

KG for NYT10m. NYT10m [16] only provides instances with relational facts, but
no KG context for their entities. Thus, we constructed an informative KG for
this dataset, enabling it to serve as a benchmark for KG-enhanced RE tasks.
After linking all the entities to FB15k as elaborated in Section 3.4, we
extracted triplets of these entities from the knowledge base to construct the
KG.

Formally, let $\mathcal{E}$ be a set of entities containing all entities in
NYT10m and let $\mathcal{R}$ be a set of relations containing all relations in
NYT10m. Given every triplet $(h^{j},t^{j},r^{j})$ in FB15k, where $h^{j}$ and
$t^{j}$ are subject and object entities with relation $r^{j}$, we construct
the KG using every triplet that satisfies the following rules: (1) $h^{j}$ and
$t^{j}$ can be found in $\mathcal{E}$, (2) $r^{j}$ is a relation in
$\mathcal{R}$. In addition, for every instance
$r^{i}=(x^{i},e^{i}_{1},e^{i}_{2})$ in NYT10m, where $e^{i}_{1}$ and
$e^{i}_{2}$ delimit entity mentions in token sequence $x^{i}$, we can extract
a triplet such as $(e^{i}_{1},e^{i}_{2},r^{i})$. These triplets can be added
to the KG to avoid information shortage. In the final KG, the triplets
extracted from validation/test sets must be excluded.

KGs for Wiki80 and Wiki20m. Similarly, both Wiki80 and Wiki20m only provide
instances with relational facts. We also extract suitable information from
Wikidata [17] for KG construction. The KG construction method we employed for
Wiki80 and Wiki20m is the same as the one we used for NYT10m.

##  4 Data Analysis

In this section, we provide the statistics of KERED and examine the extent to
which these datasets facilitate relation extraction tasks augmented by
knowledge graphs.

###  4.1 Analysis of Instances

As discussed in Section 3.5, dataset refinement eliminates noise from the
original DSRE datasets to ensure data quality. Consequently, we obtained a
modified version of NYT10m with 475,401 instances, Wiki80 with 55,547
instances, and Wiki20m with 743,703 instances. Wiki80 is smaller than the
other datasets, while the scale of NYT10m is comparable to that of Wiki20m.
However, the number of entities and relational facts in Wiki20m is
considerably higher than those in NYT10m, and the N/A proportion of NYT10m is
higher than that in Wiki20m. The statistics of our modified datasets in KERED
are presented in Table 1.

[b]  Dataset Manual Instances Entities Facts N/A Relations KB NYT10m train No
417,893 61,112 17,137 80% 25 Freebase valid No 46,422 20,850 4,062 80% test
Part 11,086 4,554 3,899 28% Wiki80 train All 50,353 66,758 50,353 0% 80
Wikidata val All 5,194 8,662 5,194 0% Wiki20m train No 571,787 285,905 154,078
55% 81 Wikidata valid No 48,794 44,082 16,489 66% test All 123,122 89,925
53,755 23%

Table 1: Statistics of KERED datasets. The _Manual_ column indicates whether
the instances are manually labeled. _Instances_ , _Entities_ , and _Facts_
indicate the number of instances, entities, and relational facts,
respectively. The _N/A_ column shows the percentages of instances with N/A
relation in the datasets. _Relation_ indicates the number of relations in each
dataset. KB indicates the knowledge base source of datasets in KERED. Wiki80
and Wiki20m are our modified versions.

###  4.2 Analysis of Knowledge Graphs

In Section 3.5, we detailed the process of KG generation for each DSRE
dataset. For NYT10m, we used a subset of FB15k [30] as the KG, which covers
approximately 25% of entities in NYT10m. For Wiki80 and Wiki20m, we extracted
the knowledge context of entities from Wikidata [17] and generated KGs with
this context. These KGs contain a large number of relational facts and cover
almost all entities in Wiki80 and Wiki20m.

The aforementioned KGs provide abundant auxiliary information for KG-enhanced
relation extraction tasks. To enhance their performance, knowledge-enhanced RE
methods can train knowledge embeddings with our KGs or consider relational
facts in the KGs as input [19, 25, 26].

We present the statistics of KGs in KERED in Table 2, wherein the numbers of
entities and facts reflect the scales of KGs. On one hand, the scales of the
Wiki20m dataset and KG are relatively large, indicating that Wiki20m is
suitable for evaluating the performance of knowledge-enhanced RE methods. On
the other hand, the scales of the Wiki80 dataset and KG are small, suggesting
that we can quickly assess the performance of models with Wiki80. Moreover,
the scale of the KG for NYT10m is considerably smaller than that of the NYT10m
dataset, and this KG can only provide knowledge context for approximately 25%
of the entities in NYT10m. The degree of an entity is the number of triples
containing that entity in the KG. The average degree of all entities reflects
the density of the KG. For Wiki80 and Wiki20m, the KGs provide 7.98 and 8.60
direct neighbors for each entity on average, indicating that the information
in these KGs is dense. Because KG embeddings are sensitive to the density of
the KG [32], we believe that our dense KGs are beneficial to KG-enhanced RE
methods, which often rely on KG embeddings [19, 25, 26, 33]. For NYT10m, the
average degree is 2.87, which is relatively low but acceptable. According to
the number of connected components and the size of the maximal connected
components, each KG consists of a very large connected component and a few
small connected components. Relational information in large connected
components is denser than the information in small ones. As the maximal
component contains almost all entities in each KG, KG-enhanced methods can
effectively extract features of entities and relations from these KGs.

[b]

Dataset | Total En. | KG En. | Facts | Degree | Comp. | Max Comp. | KB
---|---|---|---|---|---|---|---
NYT10m | 64,890 | 16,469 | 23,643 | 2.87 | 733 | 14,730 | FB15k
Wiki80 | 72,358 | 72,353 | 288,750 | 7.98 | 513 | 70,477 | Wikidata
Wiki20m | 360,966 | 360,956 | 1,551,694 | 8.60 | 621 | 359,145 | Wikidata

Table 2: Statistics of KG for each dataset in KERED. Total En. is the total
number of entities in the dataset. KG En. and Facts show the number of
entities and relational facts in the KG (all entities and relations in the KG
are also in the dataset). Degree is the average degree of all entities in the
KG. Comp. is the number of connected components in the graph. Max Comp. is the
size of the maximal connected component. KB is the knowledge base of each KG.

We further analyze the statistics of the KGs from the perspective of
instances. As shown in Figure 2, for Wiki80 and Wiki20m, the distributions are
generally similar, and the frequency of instances reaches its peak when the
degree is approximately 16, after which the frequency tends to decrease with
the increase in the degree. For NYT10m, the frequency of instances is high
when the degree is 2, indicating that many entities only have one neighbor
entity in the KG. In general, the distribution is similar between Wiki80 and
Wiki20m. The difference is that the amount of data in Wiki80 is small, so the
overall log-scaled frequency is small. However, the distribution of NYT10m is
quite different; it is sparse and irregular.

Figure 2: Frequency distributions for degrees of instances. The X-axis
represents the number of degrees, and the Y-axis represents the log-scaled
frequency counts of instances. The degree of each instance is obtained from
the sum of degrees of subject and object entities in KG.

##  5 Experimental Settings

To assess the performance of existing RE methods, particularly knowledge-
enhanced RE methods, we carry out comprehensive experiments. In this section,
we describe our benchmarks and experimental settings.

###  5.1 Benchmarks

We assembled various current RE methods as our benchmarks to assess their
performance on our datasets. Below, we offer a brief overview of all RE
benchmarks that we evaluated on our KG-enhanced datasets, including RE methods
without KG, sentence-level methods with KG, and bag-level RE methods with KG.

All methods underwent supervised training or fine-tuning on the training set
and were evaluated on the validation/test set. For PCNN [34] and BERT [35], we
adhered to the experimental settings of [16, 20]. For the other benchmarks, we
employed their original settings and hyperparameters, except for the training
epochs.

####  5.1.1 RE Methods without KG

We chose two widely-used RE models, PCNN [34] and BERT [35], as our
benchmarks. These methods classify relations between entities without
utilizing KG information. We conducted our experiments on the original Wiki80,
Wiki20m, and NYT10m or referenced the experimental results from [16].

  * •

PCNN [34] is a DSRE method developed to automatically learn features from
evidence sentences. It employs piece-wise convolutional neural networks to
encode words in a sentence and generate sentence-level embeddings.

  * •

BERT [35] and its variants have emerged as a promising natural language
processing approach since 2019. This pretraining method achieves remarkable
performance in various tasks including relation extraction.

####  5.1.2 Sentence-level RE Methods with KG

Sentence-level RE methods leverage the auxiliary KG for RE tasks in various
ways. The most common approach is the use of embeddings derived from KGs. We
selected three methods to evaluate their sentence-level performance on our
proposed dataset.

  * •

RECON [19] employs a graph neural network to learn representations of both
relational facts in KG and the evidence sentences. Attributes such as entity
names, aliases, and types from KG entities are also utilized.

  * •

KGPool [15] dynamically selects KG context with graph pooling to enhance RE
performance. This method considers only entity attributes as side information
from the KG, rather than relational facts.

  * •

CoLAKE [33] utilizes an unlabeled data structure called word-knowledge graph,
integrating language and knowledge context to learn contextualized
representations. In the fine-tuning process, we used the label _unk_ to
represent the N/A relation in Wiki20m.

####  5.1.3 Bag-level RE Methods with KG

Bag-level RE methods classify relations between entities at the bag level
rather than the sentence level. We evaluated three bag-level methods on KERED.

  * •

CGRE [14] constructs a constraint graph with entity type information to
reflect dependencies between relations. It generates representations for
entities and relations by encoding the graph into vectors and extracting
features of the nodes to support RE tasks.

  * •

HRERE [25] is a neural framework that jointly learns knowledge and language
representations. It integrates relation extraction and KG embedding generation
in a unified manner to enhance performance.

  * •

REMAP [26] is a multimodal method for the DSRE task. It fuses knowledge graph
embeddings with deep language models to effectively extract and classify the
relations between entities.

###  5.2 Metrics

We use micro F1 and micro average precision (AP) as metrics in our experiment.
Micro F1 is the harmonic mean of global micro precision and global micro
recall. The formulas for micro F1 are:

$\textrm{F1}=\frac{2*\textrm{precision}*\textrm{recall}}{\textrm{precision}+\textrm{recall}},$
(1) $\textrm{precision}=\frac{\textrm{TP}}{\textrm{TP}+\textrm{FP}},$ (2)
$\textrm{recall}=\frac{\textrm{TP}}{\textrm{TP}+\textrm{FN}},$ (3)

where $TP$ is the global true positive rate, $FP$ is the global false positive
rate, and $FN$ is the global false negative rate. AP is the weighted mean of
precision at each threshold for all samples. The formula for micro AP is:

$\textrm{AP}=\sum_{i=2}^{n}\textrm{precision}_{i}*(\textrm{recall}_{i}-\textrm{recall}_{i-1}),$
(4)

where $\textrm{precision}{i}$ and $\textrm{recall}{i}$ represent the global
precision and recall at the $i$th threshold, and $n$ denotes the total number
of samples. The Average Precision (AP) is also referred to as the Area Under
the Curve (AUC) of the Precision-Recall (PR) curve in some articles [16, 20].

In the case of sentence-level RE, the relation for each evidence sentence is
predicted directly, and the results are reported on the test sets. In the case
of bag-level RE, instances sharing the same entity pairs were grouped into
bags, and a prediction is made for each bag in the test sets. Following the
configuration of most RE experiments [15, 16, 19, 20, 25], all instances,
including those with N/A labels, are considered for micro F1 calculation.
Conversely, instances with N/A labels are excluded from the micro AP
calculation.

##  6 Results and Discussion

In the previous section, the experimental settings for our benchmarks were
delineated. This section presents the results of our experiments, adhering to
the established configurations. Additionally, an analysis of the results is
conducted, along with a discussion on the influence of KG information on RE
tasks.

###  6.1 Experimental Results

[b]

| Model | Wiki80 | Wiki20m | NYT10m
---|---|---|---|---
| F1 | AP | F1 | AP | F1 | AP
| PCNN+AVG$\dagger$ | — | — | _71.8_ | _78.1_ | _53.6_ | _52.9_
| PCNN+ONE$\dagger$ | 77.43 | 84.68 | _70.3_ | _76.6_ | _54.8_ | _53.4_
| PCNN+ATT$\dagger$ | — | — | _71.2_ | _77.5_ | _56.5_ | _56.8_
| BERT+AVG$\dagger$ | — | — | 82.7 | 89.9 | _60.4_ | _56.7_
| BERT+ONE$\dagger$ | 86.69 | 93.41 | _81.6_ | _88.9_ | 61.9 | 58.1
RE w.o. KG | BERT+ATT$\dagger$ | — | — | _66.8_ | _70.9_ | _54.1_ | _51.2_
| RECON [19] | 77.58$\pm$0.30 | 86.86$\pm$0.17 | 74.85$\pm$1.02 | 92.58$\pm$0.68 | 52.09$\pm$1.58 | 68.30$\pm$2.30
| KGPool [15] | 78.13$\pm$0.39 | 87.51$\pm$0.23 | 78.61$\pm$0.25 | 89.67$\pm$0.23 | 50.78$\pm$0.83 | 64.18$\pm$0.18
S. RE | CoLAKE [33]${\ddagger}$ | 91.78$\pm$0.08 | 96.89$\pm$0.03 | 86.05$\pm$0.77 | 92.74$\pm$0.79 | — | —
| CGRE [14] | — | — | — | — | 51.48$\pm$0.66 | 53.51$\pm$0.40
| HRERE [25] | 80.04$\pm$0.21 | 87.42$\pm$0.17 | 77.33$\pm$0.29 | 88.38$\pm$0.24 | 30.50$\pm$0.16 | 54.40$\pm$0.88
B. RE | REMAP [26] | 88.73$\pm$0.18 | 92.16$\pm$0.22 | 84.99$\pm$0.21 | 89.92$\pm$0.19 | 52.19$\pm$0.92 | 66.39$\pm$1.19

  * $\dagger$

These experiments were conducted on original RE datasets under the settings of
[16, 20]. We conducted the experiments for Wiki80, and the results of these
models on Wiki20m and NYT10m have been cited from [16].

  * ${\ddagger}$

Because CoLake is pre-trained on Wikidata, there is a risk of information
leakage.

Table 3: Experimental results (%) of baselines. We report three-run micro F1
scores and micro average precision (AP) scores with standard deviations. _RE
w.o. KG_ indicates RE methods without using KG. _S. RE_ and _B. RE_ indicate
sentence-level and bag-level RE methods with KG. For PCNN and BERT, we only
evaluate PCNN+ONE and BERT+ONE because the other two aggregation methods do
not fit the sentence-level dataset Wiki80. CoLAKE was left untested on NYT10m
because of the lack of learned relation representations. CGRE was only
evaluated on NYT10m because the constraint graph it relies on only contains
relations from Freebase.

Table 3 shows our experimental results. RE methods without using KG are
evaluated on original Wiki80, Wiki20m, and NYT10m datasets; Sentence-level and
bag-level methods with KG are evaluated on KERED.

Average (AVG), at-least-one (ONE), and attention (ATT) represent methods
employed in bag-level Relation Extraction (RE) for aggregating sentence-level
prediction results into bag predictions [16]. AVG calculates the mean of all
sentence representations within the bag. ONE predicts relation scores for each
sentence in the bag and subsequently selects the highest score for each
relation. ATT computes a weighted average over sentence embeddings in the bag,
determining weights through attention scores between sentences and relations.
Because Wiki80 is a sentence-level RE dataset without any repetitive entity
pairs, AVG and ATT are inapplicable to experiments on this dataset.

Because CoLAKE is a pre-trained model based on Wikidata5M [36], it lacks
learned relation representations in NYT10m. Therefore, we only evaluate the
method on Wiki80 and Wiki20m. Additionally, some triplets in the test set may
appear in the training data of CoLAKE, causing information leakage. The method
CGRE is only evaluated on NYT10m because the constraint graph provided by the
author only contains relations from Freebase [18] rather than Wikidata [17].

The experimental results in Table 3 lead to the following observations: Among
RE methods without KG information, BERT+AVG exhibits the best performance on
Wiki20m, attaining 82.7% for F1 and 89.9% for AP. BERT+ONE demonstrates
superior performance on NYT10m and Wiki80. For sentence-level RE methods
incorporating KG, CoLAKE significantly outperforms other models on Wiki80 and
Wiki20m, achieving 91.78% for F1 and 96.89% for AP on Wiki80, and 86.05% for
F1 and 92.74% for AP on Wiki20m. RECON is the top sentence-level performer on
NYT10m, reaching 52.09% for F1 and 68.30% for AP. Among bag-level RE methods
with KG information, REMAP attains the best performance for both metrics
across all three datasets.

Our experimental results indicate that RE methods incorporating KG generally
surpass those without KG, particularly in AP scores. The high performance can
be attributed to the additional KG information supplying more knowledge to RE
models. A notable exception is the poor performance of all RE methods with KG
in micro F1 scores on NYT10m. This could be due to the limited number of
relational facts in the KG constructed for NYT10m. Furthermore, other KG
information such as entity attributes employed by RECON and KGPool, or
constraint graphs utilized by CGRE, may not offer the models a beneficial
predictive direction. Another observation is that methods leveraging pre-
training exhibit better performance compared to other methods.

###  6.2 Analysis and discussion

We conduct a comprehensive analysis of the experiments and then discuss the
effect of our curated KG, including basic statistics from the perspective of
instances and the effect of KG on model performance.

####  6.2.1 Analysis of Performance and Degree of Instance

In Section 4.2, we analyzed the statistics of KG and presented the frequency
distributions for degrees of instances in Table 2. Herein, We further analyze
the influence of instances’ degree on the performance of models. As shown in
Figure 3, with the increment of degree, micro F1 scores of KGPool, RECON, and
HRERE initially decline and subsequently rise in Wiki80 and Wiki20m. In
contrast, this trend is not evident in NYT10m, attributable to its limited
built KG. As highlighted in Section 6.1, the pre-training method CoLAKE
outperforms other approaches, especially when the percentile degree is lower
than 70%. Furthermore, the performance of CoLAKE and CGRE are not sensitive to
the number of degrees, as the methods do not utilize relational facts in our
curated KGs.

Figure 3: Micro F1 scores of RE models on instances with different percentiles
of degree. The degree of each instance is obtained from the sum of the degrees
of the subject and object entities in KG. We split the instances into ten
groups according to their degrees. Degrees are maximum when the percentile of
degree is 95 and minimum when it is 5.

####  6.2.2 Analysis of Performance and Relation

To investigate model performance on individual relations, we present micro F1
scores of models for each relation in Figure B.1 (see Appendix B). The results
demonstrate considerable variation in the ability of models to accurately
classify each relation. Some relations are readily identifiable, such as
$voicetype$ and $positionplayedonteam/specialty$ in Wiki80, while others are
not. Additionally, micro F1 scores can reach $1.0$ for certain relations in
Wiki80 and Wiki20m, but this observation does not extend to the NYT10m
dataset. Meanwhile, some relations in NYT10m remain unrecognizable by the
models, which may be attributed to the limited training data available for
these relations.

####  6.2.3 Discussion

Based on the analyses presented, it can be concluded that the availability of
relational facts plays a crucial role in knowledge-enhanced RE tasks. Due to
the relatively small KG of NYT10m, the performance of most methods with KG is
inferior to those without KG, indicating that these models may rely
excessively on KG information in their design, potentially causing a negative
impact when KG information is insufficient. Predicting relations between
entities can be challenging, particularly when the quantity of relational
facts in KG is limited. Thus, although knowledge-enhanced RE methods are
effective, we believe that knowledge-enhanced RE merits further investigation.
We propose the following research directions: (1) developing models that
effectively utilize KG information, particularly relational facts; and (2)
investigating approaches that capitalize on limited relational facts to
enhance RE performance.

##  7 Conclusion

We developed KERED using three knowledge-enhanced RE datasets derived from
widely-used DSRE datasets. We identified issues within the original datasets,
subsequently cleaned them to enhance data quality, and linked entities to
large-scale knowledge bases. Our curated KGs in KERED promote the study of
knowledge-enhanced RE methods in both sentence-level and bag-level settings.
Furthermore, we established a connection between auxiliary KGs and knowledge-
enhanced RE methods, demonstrating that our auxiliary KGs consistently benefit
these approaches. In conclusion, we provided three high-quality datasets for
fair comparisons between knowledge-enhanced RE methods and established
benchmarks in various settings, which will further advance RE research.

## Acknowledgements

This work was supported by National Natural Science Foundation of China
(No.72001205, 72171229, 11971504), College Stability Support Program of
Shenzhen (20200827090247001), fund for building world-class universities
(disciplines) of Renmin University of China, Chinese National Statistical
Science Research Project (2022LD06), Foundation from Ministry of Education of
China (20JZD023), Ministry of Education Focus on Humanities, and Social
Science Research Base (Major Research Plan 17JJD910001).

## Compliance with Ethical Standards

Conflict of interest The authors declared that they have no conflicts of
interest with regard to this work. We declare that we do not have any
commercial or associative interest that represents a conflict of interest in
connection with the work submitted.

##  Appendix A Access to KERED

We have posted our refined datasets Wiki80, Wiki20m, and NYT10m on Figshare:
https://figshare.com/projects/KERED/134459. Each dataset consists of four
major components: a knowledge graph, training set, validation set, and a
testing set (since the original Wiki80 has no testing set, we consider the
validation set as the testing set for our experiments). The knowledge graphs
are in CSV format. Each entry in the knowledge graphs represent a relational
fact (a subject, an object, and a relation). For training/validation/testing
sets, we follow the format of the original datasets. All instances in these
sets are in json format, providing sentences, relations, and entities by
key/value pairs.

##  Appendix B Model Performance on Relations

(a) HRERE

(b) KGPool

(c) RECON

Figure B.1: Top-10 high and top-10 low micro F1 scores of different relations
for HRERE, KGPool, and RECON.

## References

  *   * Pawar et al. [2017] Pawar, S., Palshikar, G.K., Bhattacharyya, P.: Relation extraction : A survey. CoRR abs/1712.05191 (2017) 1712.05191
  * Xu et al. [2016] Xu, K., Reddy, S., Feng, Y., Huang, S., Zhao, D.: Question answering on Freebase via relation extraction and textual evidence. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2326–2336. Association for Computational Linguistics, Berlin, Germany (2016). https://doi.org/10.18653/v1/P16-1220 . https://aclanthology.org/P16-1220
  * Chen et al. [2019] Chen, Z.-Y., Chang, C.-H., Chen, Y.-P., Nayak, J., Ku, L.-W.: UHop: An unrestricted-hop relation extraction framework for knowledge-based question answering. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 345–356. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://doi.org/10.18653/v1/N19-1031 . https://aclanthology.org/N19-1031
  * Luan et al. [2018] Luan, Y., He, L., Ostendorf, M., Hajishirzi, H.: Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3219–3232. Association for Computational Linguistics, Brussels, Belgium (2018). https://doi.org/10.18653/v1/D18-1360 . https://aclanthology.org/D18-1360
  * Bosselut et al. [2019] Bosselut, A., Rashkin, H., Sap, M., Malaviya, C., Celikyilmaz, A., Choi, Y.: COMET: Commonsense transformers for automatic knowledge graph construction. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4762–4779. Association for Computational Linguistics, Florence, Italy (2019). https://doi.org/10.18653/v1/P19-1470 . https://aclanthology.org/P19-1470
  * Qin et al. [2021] Qin, Y., Lin, Y., Takanobu, R., Liu, Z., Li, P., Ji, H., Huang, M., Sun, M., Zhou, J.: ERICA: Improving entity and relation understanding for pre-trained language models via contrastive learning. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3350–3363. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.acl-long.260 . https://aclanthology.org/2021.acl-long.260
  * Zeng et al. [2020] Zeng, C., Li, S., Li, Q., Hu, J., Hu, J.: A survey on machine reading comprehension: Tasks, evaluation metrics, and benchmark datasets. CoRR abs/2006.11880 (2020) 2006.11880
  * Ji et al. [2022] Ji, S., Pan, S., Cambria, E., Marttinen, P., Yu, P.S.: A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems 33(2), 494–514 (2022) https://doi.org/10.1109/TNNLS.2021.3070843
  * Nickel et al. [2016] Nickel, M., Murphy, K., Tresp, V., Gabrilovich, E.: A review of relational machine learning for knowledge graphs. Proceedings of the IEEE 104(1), 11–33 (2016) https://doi.org/10.1109/JPROC.2015.2483592
  * Wang et al. [2018a] Wang, Z., Chen, T., Ren, J.S.J., Yu, W., Cheng, H., Lin, L.: Deep reasoning with knowledge graph for social relationship understanding. CoRR abs/1807.00504 (2018) 1807.00504
  * Wang et al. [2018b] Wang, H., Zhang, F., Xie, X., Guo, M.: Dkn: Deep knowledge-aware network for news recommendation. In: Proceedings of the 2018 World Wide Web Conference. WWW ’18, pp. 1835–1844. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE (2018). https://doi.org/10.1145/3178876.3186175 . https://doi.org/10.1145/3178876.3186175
  * Zou [2020] Zou, X.: A survey on application of knowledge graph. Journal of Physics: Conference Series 1487(1), 012016 (2020) https://doi.org/10.1088/1742-6596/1487/1/012016
  * Zhang et al. [2019] Zhang, N., Deng, S., Sun, Z., Wang, G., Chen, X., Zhang, W., Chen, H.: Long-tail relation extraction via knowledge graph embeddings and graph convolution networks. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3016–3025. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://doi.org/10.18653/v1/N19-1306 . https://aclanthology.org/N19-1306
  * Liang et al. [2022] Liang, T., Liu, Y., Liu, X., Zhang, H., Sharma, G., Guo, M.: Distantly-supervised long-tailed relation extraction using constraint graphs. IEEE Transactions on Knowledge and Data Engineering, 1–1 (2022) https://doi.org/10.1109/TKDE.2022.3177226
  * Nadgeri et al. [2021] Nadgeri, A., Bastos, A., Singh, K., Mulang’, I.O., Hoffart, J., Shekarpour, S., Saraswat, V.: KGPool: Dynamic knowledge graph context selection for relation extraction. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 535–548. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.findings-acl.48 . https://aclanthology.org/2021.findings-acl.48
  * Gao et al. [2021] Gao, T., Han, X., Bai, Y., Qiu, K., Xie, Z., Lin, Y., Liu, Z., Li, P., Sun, M., Zhou, J.: Manual evaluation matters: Reviewing test protocols of distantly supervised relation extraction. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1306–1318. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.findings-acl.112 . https://aclanthology.org/2021.findings-acl.112
  * Vrandečić and Krötzsch [2014] Vrandečić, D., Krötzsch, M.: Wikidata: A free collaborative knowledgebase. Commun. ACM 57(10), 78–85 (2014) https://doi.org/10.1145/2629489
  * Bollacker et al. [2008] Bollacker, K., Evans, C., Paritosh, P., Sturge, T., Taylor, J.: Freebase: A collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. SIGMOD ’08, pp. 1247–1250. Association for Computing Machinery, New York, NY, USA (2008). https://doi.org/10.1145/1376616.1376746 . https://doi.org/10.1145/1376616.1376746
  * Bastos et al. [2021] Bastos, A., Nadgeri, A., Singh, K., Mulang, I.O., Shekarpour, S., Hoffart, J., Kaul, M.: Recon: Relation extraction using knowledge graph context in a graph neural network. In: Proceedings of the Web Conference 2021. WWW ’21, pp. 1673–1685. Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3442381.3449917 . https://doi.org/10.1145/3442381.3449917
  * Han et al. [2019] Han, X., Gao, T., Yao, Y., Ye, D., Liu, Z., Sun, M.: OpenNRE: An open and extensible toolkit for neural relation extraction. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pp. 169–174. Association for Computational Linguistics, Hong Kong, China (2019). https://doi.org/10.18653/v1/D19-3029 . https://aclanthology.org/D19-3029
  * Riedel et al. [2010] Riedel, S., Yao, L., McCallum, A.: Modeling relations and their mentions without labeled text. In: ECML-PKDD, pp. 148–163 (2010)
  * Han et al. [2020] Han, X., Gao, T., Lin, Y., Peng, H., Yang, Y., Xiao, C., Liu, Z., Li, P., Zhou, J., Sun, M.: More data, more relations, more context and more openness: A review and outlook for relation extraction. In: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 745–758. Association for Computational Linguistics, Suzhou, China (2020). https://aclanthology.org/2020.aacl-main.75
  * Vashishth et al. [2018] Vashishth, S., Joshi, R., Prayaga, S.S., Bhattacharyya, C., Talukdar, P.: RESIDE: Improving distantly-supervised neural relation extraction using side information. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1257–1266. Association for Computational Linguistics, Brussels, Belgium (2018). https://doi.org/10.18653/v1/D18-1157 . https://aclanthology.org/D18-1157
  * Christopoulou et al. [2021] Christopoulou, F., Miwa, M., Ananiadou, S.: Distantly supervised relation extraction with sentence reconstruction and knowledge base priors. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 11–26. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.naacl-main.2 . https://aclanthology.org/2021.naacl-main.2
  * Xu and Barbosa [2019] Xu, P., Barbosa, D.: Connecting language and knowledge with heterogeneous representations for neural relation extraction. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3201–3206. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://doi.org/10.18653/v1/N19-1323 . https://aclanthology.org/N19-1323
  * Lin et al. [2022] Lin, Y., Lu, K., Yu, S., Cai, T., Zitnik, M.: Multimodal learning on graphs for disease relation extraction. arXiv preprint arXiv:2203.08893 (2022)
  * Riedel et al. [2010] Riedel, S., Yao, L., McCallum, A.: Modeling relations and their mentions without labeled text. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 148–163 (2010). Springer
  * Han et al. [2018] Han, X., Zhu, H., Yu, P., Wang, Z., Yao, Y., Liu, Z., Sun, M.: FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4803–4809. Association for Computational Linguistics, Brussels, Belgium (2018). https://doi.org/10.18653/v1/D18-1514 . https://aclanthology.org/D18-1514
  * Färber et al. [2018] Färber, M., Bartscherer, F., Menne, C., Rettinger, A.: Linked data quality of dbpedia, freebase, opencyc, wikidata, and yago. Semantic Web 9(1), 77–129 (2018)
  * Bordes et al. [2013] Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. Advances in neural information processing systems 26 (2013)
  * Han et al. [2020] Han, X., Gao, T., Lin, Y., Peng, H., Yang, Y., Xiao, C., Liu, Z., Li, P., Zhou, J., Sun, M.: More data, more relations, more context and more openness: A review and outlook for relation extraction. In: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 745–758. Association for Computational Linguistics, Suzhou, China (2020). https://aclanthology.org/2020.aacl-main.75
  * Pujara et al. [2017] Pujara, J., Augustine, E., Getoor, L.: Sparsity and noise: Where knowledge graph embeddings fall short. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1751–1756. Association for Computational Linguistics, Copenhagen, Denmark (2017). https://doi.org/10.18653/v1/D17-1184 . https://aclanthology.org/D17-1184
  * Sun et al. [2020] Sun, T., Shao, Y., Qiu, X., Guo, Q., Hu, Y., Huang, X., Zhang, Z.: CoLAKE: Contextualized language and knowledge embedding. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 3660–3670. International Committee on Computational Linguistics, Barcelona, Spain (Online) (2020). https://doi.org/10.18653/v1/2020.coling-main.327 . https://aclanthology.org/2020.coling-main.327
  * Zeng et al. [2015] Zeng, D., Liu, K., Chen, Y., Zhao, J.: Distant supervision for relation extraction via piecewise convolutional neural networks. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1753–1762. Association for Computational Linguistics, Lisbon, Portugal (2015). https://doi.org/10.18653/v1/D15-1203 . https://aclanthology.org/D15-1203
  * Devlin et al. [2019] Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://doi.org/10.18653/v1/N19-1423 . https://aclanthology.org/N19-1423
  * Wang et al. [2021] Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., Tang, J.: KEPLER: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics 9, 176–194 (2021) https://doi.org/10.1162/tacl_a_00360