# A Comprehensive Survey of Document-level Relation Extraction (2016-2023)

Julien DELAUNAY  0009-0001-9247-5745<EMAIL_ADDRESS>Univ. La
Rochelle, L3iLa RochelleFrance ,  Hanh Thi Hong TRAN  0000-0002-5993-1630
Univ. La Rochelle, L3iLa RochelleFrance ,  Carlos-Emiliano GONZÁLEZ-GALLARDO
0000-0002-0787-2990 Univ. La Rochelle, L3iLa RochelleFrance ,  Georgeta BORDEA
0000-0001-9921-8234 Univ. La Rochelle, L3iLa RochelleFrance ,  Nicolas SIDERE
Univ. La Rochelle, L3iLa RochelleFrance and  Antoine DOUCET
0000-0001-6160-3356 Univ. La Rochelle, L3iLa RochelleFrance

###### Abstract.

Document-level relation extraction (DocRE) is an active area of research in
natural language processing (NLP) concerned with identifying and extracting
relationships between entities beyond sentence boundaries. Compared to the
more traditional sentence-level relation extraction, DocRE provides a broader
context for analysis and is more challenging because it involves identifying
relationships that may span multiple sentences or paragraphs. This task has
gained increased interest as a viable solution to build and populate knowledge
bases automatically from unstructured large-scale documents (e.g., scientific
papers, legal contracts, or news articles), in order to have a better
understanding of relationships between entities. This paper aims to provide a
comprehensive overview of recent advances in this field, highlighting its
different applications in comparison to sentence-level relation extraction.

document-level relation extraction, cross-sentence relation extraction, inter-
sentence relation extraction, DocRE, relation extraction, Information
Extraction

††ccs: Computing methodologies††ccs: Computing methodologies Natural language
processing

##  1\. Introduction

Relation extraction (RE) is a main task in natural language processing (NLP)
applications that aims to automatically identify semantic relationships
between entities. Its importance is showcased by its numerous applications in
information extraction (IE) and knowledge base (KB) construction (Pujara and
Singh, 2018; Trisedya et al., 2019; Zaporojets et al., 2021; Prieur et al.,
2023a), and in particular in the medical domain (Li et al., 2016). Given the
unit of analysis, RE can be divided into sentence-level and document-level.

The goal of sentence-level RE is to identify the relationship between two
entities that are mentioned together in the same sentence. This is a complex
task because it calls for comprehension of both the sentence’s content and the
relationships between the entities which are often already marked with special
tokens (e.g., “$<$en$>$ […] $<$/en$>$”). Moreover, a joint task of both named
entity recognition (NER) and RE can be performed, where the same model will
mark the entities and classify them, and identify the relations between each
of them (Luan et al., 2019; Wadden et al., 2019). The most common scenario is
a binary relation, that can be defined as a triplet
$t=<\varepsilon_{i},\varepsilon_{j},r>$, where $\varepsilon_{i}$ and
$\varepsilon_{j}$ are the two entities, with $\varepsilon_{i}$ the head of the
relation, $\varepsilon_{j}$ the tail of the relation, and $r$ the label of the
relation. This kind of relation is exemplified in Figure 1. When the relation
involves more than two entities, we commonly refer to it as $n$-ary with
$n\geq 2$.

$<$e1$>$“The Anomaly”$<$/e1$>$ is a novel written by $<$e2$>$Hervé Le
Tellier$<$/e2$>$.

Hervé Le TellierThe Anomalywrote

Figure 1. An example of sentence-level relation extraction. In this sentence,
we see that there is a relation between the book “The Anomaly” and the writer
Hervé Le Tellier, which reflects the fact that the latter is the author of the
first.

As shown in Figure 2, new techniques based on graph neural networks (Peng et
al., 2017; Christopoulou et al., 2018, 2019), transformers techniques (Wang et
al., 2019; Tang et al., 2020; Zhou et al., 2020a) and large-scale datasets (Li
et al., 2016; Yao et al., 2019; Zaporojets et al., 2021), made possible a
progressive shift from sentence-level to document-level relation extraction
(DocRE) beginning with the second-half of 2010s.

2016201720182019202020212022$0$$20$$40$$60$$80$$100$$13$$6$$19$$28$$54$$82$$99$YearNumber
of Publications Figure 2. Number of publications per year on DocRE according
to Scopus, for the terms “document-level relation extraction”, “cross-sentence
relation extraction”, and “inter-sentence relation extraction” in title,
abstract, or keywords. Please note that the year 2023 is not included because
it had not ended at the time of submission (it will be added in case of
acceptance).

The focus is now on identifying and classifying relations between entities
across entire documents, or at least beyond the sentence boundary. This
difference in scope implies major distinctions from one task to another (Han
and Wang, 2020):

  * •

Larger number of potential relations The number of entities contained in a
document is far greater than in a sentence, which often contains only two
entities. At the scale of a full document, we have to identify relationships
between all the different entities.

  * •

Handling coreferences An entity usually occurs only once in a sentence, while
in a document it can appear several times and in numerous guises. For example,
in Figure 3, “He” is a co-referent of “Marcus Miller”.

  * •

Transient relationships At the scope of a whole document, some relations
require logical inference across sentences. For example, Yao et al. (2019)
reported that approximately 40.7% of the relations contained in DocRED, the
largest dataset for DocRE, can only be extracted from reasoning over multiple
sentences, even if Huang et al. (2021) argued that in most cases, only three
sentences at most are required as supporting evidence to identify a relation.
In addition, this work states that importance differs widely between
sentences. Sentences that do not contain any valuable information or relation
actually hinder document comprehension.

$<$e1$>$Marcus Miller$<$/e1$>$ is an American musician, songwriter, and record
producer. He is best known for his work as a bassist. Main producer on the
famous $<$e2$>$Miles Davis$<$/e2$>$’ album $<$e3$>$Tutu$<$/e3$>$, he has also
worked with pianist $<$e6$>$Herbie Hancock$<$/e6$>$ and saxophonist
$<$e7$>$David Sanborn$<$/e7$>$, among others.

Marcus MillerMiles DavisHerbie HancockDavid
SanbornTutuworked_withworked_withworked_withproducedreleased

Figure 3. An example of document-level relation extraction.

For all these reasons DocRE is more challenging than sentence-level RE, but
there are many use cases using Knowledge Bases that require this shift such as
financial investments (Abu-Salih, 2021) and maritime security (Prieur et al.,
2023b), but also teaching and classroom resources, cybersecurity,
telecommunication, chemistry, geology. This survey aims to cover the following
aspects:

  1. (1)

A comprehensive review of resources for DocRE from the past decade, including
silver-standard as well as gold-standard annotated corpora.

  2. (2)

A systematic review of DocRE approaches, with a special focus on graph- and
transformer-based models.

  3. (3)

A detailed discussion of the main evaluation metrics used in RE.

The survey is organized in a subsequent manner. Section 2 lists relevant
previous surveys related to our problem. In Section 3, we present the problem
statement of DocRE. Section 4 delineates our methodology for selecting
articles to be included in the survey. The silver and gold annotated datasets
for DocRE are discussed in Section 5. Section 6 presents a comprehensive
overview of existing approaches to DocRE, while in Section 7, a comparative
evaluation of various systems is conducted. Finally, we discuss limitations
and perspectives for future work in Section 8 and conclude in Section 9.

##  2\. Previous surveys

Existing surveys on RE mainly focus on the more tractable task of sentence-
level relation extraction. Bach and Badaskar (2007) provide a comprehensive
overview of supervised methods based on features and kernels, along with three
semi-supervised approaches. Additionally, they present practical applications
of RE, such as question answering and biotext mining. de Abreu et al. (2013)
offer an exhaustive definition of the RE task, with a particular emphasis on
sentence-level RE datasets and systems specialized in the Portuguese language.

Shifting the focus to the biomedical domain, Cohen and Hersh (2005); Zhou et
al. (2014); Boudjellal et al. (2020) dedicate their surveys to this area. The
former presents an overview of early text mining works in the biomedical
domain, encompassing sentence-level RE. The second delivers a detailed review
of both binary and n-ary RE methods tailored to the biomedical domain. The
latter focuses on the use of distant supervision for extracting relations from
the corpus of the biological domain. Notably, it reviews a few inter-sentence
and document-level methods.

Pawar et al. (2017) conduct an exhaustive review of sentence-level RE,
covering both supervised and unsupervised approaches, alongside Open IE
methods. Notably, they describe some early convolutional neural network (CNN)
methods and mention some early approaches to cross-sentence RE. Shi et al.
(2019) propose a concise review of distant supervision, with a particular
emphasis on deep learning methods, incorporating some graph-based techniques.
However, the discussion is limited to just two distantly supervised datasets.
Bassignana and Plank (2022) present a review of scientific RE datasets,
excluding those from the biomedical domain. While they make an earnest effort
to mention document-level RE approaches and datasets and also address the
ambiguity surrounding the RE task concerning relation identification and
relation classification, the primary focus of the paper remains on sentence-
level RE.

Our work aims to bridge a significant gap in the literature, focusing
specifically on document-level relation extraction regardless of the domain of
application.

##  3\. Problem statement

In this survey, we follow the notation proposed in Zhang et al. (2020),
defining an annotated document as
$\mathcal{D}=\\{\mathcal{S}_{i}\\}^{n_{s}}_{i=1}$ with an entity set
$\mathcal{V}=\\{\mathcal{E}_{i}\\}^{n_{e}}_{{i=1}}$, where
$\mathcal{S}_{i}=\\{w_{j}\\}^{n^{i}_{w}}_{j=1}$ is the $i$-th sentence with
$n^{i}_{w}$ words $w$ and $\mathcal{E}_{i}=\\{m_{j}\\}^{n^{i}_{m}}_{j=1}$ is
the $i$-th entity with $n^{i}_{m}$ entity mentions $m$. In the sentence-level
setting, an entity is only represented by one mention, i.e., $\forall
i\in[1,n_{e}],\mathcal{E}_{i}=m_{i}$, which is not the case in the document-
level setting, where an entity can be represented by multiple mentions. To get
a fixed representation of an entity, former methods were first to make an
average of all the mention embeddings, i.e.,
$\mathcal{E}_{i}=\sum_{j=1}^{n^{i}_{m}}m_{j}/n^{i}_{m}$, or max pooling; but
then shifted progressively to a softer representation represented by
logsumexp, facilitating the accumulation of weak signals from individual
mentions: $\mathcal{E}_{i}=log\sum_{j=1}^{n^{i}_{m}}exp(m_{j})$ (Jia et al.,
2019). The goal of DocRE is thus to predict all intra- and inter-sentence
relations $\mathcal{R}^{\prime}\in\mathcal{R}=\\{r_{i}\\}^{n_{r}}_{i=1}$
between each entity pair.

##  4\. Methodology

Google Scholar111https://scholar.google.com/,
Elsevier222https://www.elsevier.com/fr-fr, ACL
Anthology333https://aclanthology.org/, and Google444https://www.google.com/
were primarily searched to identify papers relevant to the preparation of this
study. The main search progress was tracked from April to May 2023. We then
updated our survey with new methods until the date of submission. Our search
terms covered two different available DocRE resources, that is corpora and
systems. We used the following queries to collect research articles describing
methods/systems or corpora: “inter-sentence relation extraction”, “cross-
sentence relation extraction”, “document-level relation extraction”, “DocRE”,
“global-level relation extraction”, “full-abstract relation extraction”. For
datasets, we just added “corpora”, “dataset”, “corpus” and “data” to the
previous queries. Since the domain is still relatively small, an exhaustive
analysis is feasible therefore we decided not to rely on a threshold (i.e. a
minimum number of citations regarding of the year of publication) to select
articles for review.

##  5\. DocRE datasets

In this section, we review some important DocRE datasets, with a focus on
English language resources that are publicly available along with detailed
documentation regarding their structure and annotation process.

Datasets | Domain(s) | Annotation | Source | Type | Nb of documents
---|---|---|---|---|---
BC5CDR (CDR) (Li et al., 2016) | Biology | Gold | PubMed | Abstracts | 1,500
| Ternary Drug
---
Interaction (Peng et al., 2017)
Biology | Silver | - | - | -
(Mandya et al., 2018b) | General | Silver | Wikipedia | Sentence pairs | 31,970
CHR (Sahu et al., 2019) | Medicine | Silver | PubMed | Abstracts | 12,094
GDA (Wu et al., 2019), 2021 | Medicine | Silver | PubMed | Abstracts | 30,192
RENET2 (abstracts) (Su et al., 2021) | Medicine | Silver | PubMed | Abstracts | 1,000
RENET2 (full texts) (Su et al., 2021) | Medicine | Silver | PubMed | Full texts | 500
DocRED (Yao et al., 2019) | General | Gold | Wikipedia | Paragraphs | 5,053
DocRED distant (Yao et al., 2019) | General | Silver | Wikipedia | Paragraphs | 101,873
Re-DocRED (Tan et al., 2022b) | General | Gold | Wikipedia | Paragraphs | 4,053
DWIE (Zaporojets et al., 2021) | General | Gold | News | Articles | 802
BIORED (Luo et al., 2022) | Biomedicine | Gold | PubMed | Abstracts | 600

Table 1. List of the main DocRE datasets and their characteristics. A dataset
is considered to be gold-annotated when it was at least curated entirely
manually.

Datasets | Nb of entity mentions | Nb of entities |  | Nb of
---
positive relations
BC5CDR (CDR) (Li et al., 2016) | 10,227 | - | 3,116
| Ternary Drug
---
Interaction (Peng et al., 2017)
- | - | 3,452
(Mandya et al., 2018b) |  | 31,970 | 31,970
CHR (Sahu et al., 2019) | - | - | 32,406
RENET2 (full texts) (Su et al., 2021) | - | - | 51,642
DocRED (Yao et al., 2019) | - | 132,375 | 62,427
Re-DocRED (Tan et al., 2022b) | - | - |  $\approx$120,539
DWIE (Zaporojets et al., 2021) | 43,373 | 23,130 | 21,749
BIORED (Luo et al., 2022) | 20,419 | 3,869 | 6,503

Table 2. Number of entity mentions, entities and positive relations for the
main DocRE datasets. We did not put into the table datasets for which the
information were not given.

###  5.1. Medical domain datasets

####  5.1.1. Gold-standard annotations

One of the first DocRE datasets published is the BioCreative V CDR Task Corpus
(BC5CDR (CDR)) (Li et al.,
2016)555https://biocreative.bioinformatics.udel.edu/tasks/biocreative-v/track-3-cdr/.
Focused on Chemical Disease Relation (CDR) extraction, it consists of 1,500
PubMed articles, equally divided into sets of 500 articles for training,
validation, and testing. Each entity annotation includes both the mention text
spans and normalized concept identifiers, using Medical Subject Headings
(MeSH) (Lipscomb, 2000) as the controlled vocabulary. The annotators were part
of a team of four MeSH indexers for disease/chemical entity annotation and
three Comparative Toxicogenomics Database (CTD) (Davis et al., 2009) curators
for Chemical-Induced Diseases (CID) relation annotation. Entities are first
recorded individually by two annotators and subsequently annotated by
consensus. Text-mined disease and chemical results are pre-annotated using
automatic systems and displayed to the annotators to accelerate manual
annotation. Relations are of two types, either “putative mechanistic” (the
chemical influences the disease), or “biomarker” (the chemical correlates with
the disease). Even if this dataset is very restricted in its use case and
should not be used for general purposes, it still serves as a benchmark for a
lot of models, even recent ones (Nan et al., 2020; Zhang and Cheng, 2022b;
Xiao et al., 2022), given the lack of resources in DocRE datasets.

Luo et al. (2022)666https://ftp.ncbi.nlm.nih.gov/pub/lu/BioRED/ introduced
BioRED, a biomedical DocRE dataset. They annotated 600 PubMed abstracts of
11.9 sentences on average with 6 entity types (i.e., gene, disease, chemical,
variant, species, and cell line) and 8 relation types. They also
differentiated new findings and known facts in their annotations, avoiding
replica records into the automatic knowledge construction in biomedicine. Each
article was triply annotated, thus, to accelerate entity annotation, they used
previous annotations combined with automated pre-annotations. However,
relations were annotated by hand and from scratch, except for those that were
already annotated in CDR (Li et al., 2016). This dataset is still recent and
has not really been used yet to benchmark models. Nonetheless, since it is
gold-annotated and focuses on the complete abstracts rather than only a
paragraph, we think it should be highly employed.

####  5.1.2. Silver-standard annotations

Peng et al. (2017) built the Ternary Drug Interaction dataset using distant
supervision to evaluate their Graph-LSTM model, here again, because of the
lack of annotated datasets. The Gene Drug Knowledge Database (Dienstmann et
al., 2015) and the Clinical Interpretations of Variants in Cancer knowledge
bases777https://civicdb.org/ are used to automatically identify mentions of
drugs, genes, and mutations on the one million full-text articles from PubMed
(only a small subset contains drug-gene-mutation interactions). Then, co-
occurring triples with known interactions that appear in a minimal span of
text are chosen as positive examples, with a limit of up to $k\leq 3$
consecutive sentences. From 59 different drug-gene interactions, 3,452
positive instances are identified. However, such a process introduces a lot of
noise in the corpus, resulting in a relatively low adoption to evaluate other
systems.

Facing a lack of a reliable DocRE dataset, Sahu et al.
(2019)888http://nactem.ac.uk/CHR/ construct a CHemical Reactions dataset (CHR)
using distant supervision to evaluate their model. Consisting of 12,094 PubMed
abstracts (7,298 for train, 1,182 for validation, and 3,614 for test sets), it
was built using a search engine for biomedical abstracts (Soto et al., 2019)
to obtain the biomedical named entities annotations, which were then aligned
with a biomedical graph database (Swainston et al., 2017) to identify the
relationship between each. Since CHR is very specialized in biology but also
only distantly supervised with no evaluation of the accuracy of the relations
identified, it was not much used as a benchmark.

Another example of the lack of resources in DocRE is Wu et al.
(2019)999https://bitbucket.org/alexwuhkucs/gda-extraction/src/master/, where
the researchers build the Gene-Disease-Association dataset (GDA) in order to
test their model, RENET. This large dataset was constructed using distant
supervision on 30,192 abstracts from MEDLINE (1,000 for the test set, the rest
divided between the train and validation sets following a proportion of
80:20), a platform listing a large set of gene-disease associations collected
from publicly available databases. Wu et al. (2019) precise that most of the
relations were curated by hand, even if the exact proportion is unknown. The
GDA dataset was used to evaluate many methods (Christopoulou et al., 2019;
Minh Tran et al., 2020; Nan et al., 2020; Zhou et al., 2020a; Sun et al.,
2022b; Zhang et al., 2021; Xie et al., 2022), often in combination with CDR
(Li et al., 2016). Su et al. (2021)101010https://github.com/sujunhao/RENET2
improve the dataset by incorporating full texts in addition to abstracts,
creating the RENET2 dataset. They randomly sample 500 abstracts from GDA,
which were manually re-annotated by three experts with a Biology,
Bioinformatics, or Computer Science background. They introduce a new
association to represent that the relation between a gene and a disease is
ambiguous. They use these first 500 abstracts to train their RENET2 model
(discussed in Section 6), which then extracts relations from another 500
abstracts. Then, to train the model for extracting relations from the 500 full
texts, they use the distantly supervised set of annotated abstracts in the
training set while putting the manually annotated one in training and
validation. This dataset is not used much, despite the fact that it is the
only one that provides annotations for full-text articles for gene-disease
associations, even if it is still silver annotated.

###  5.2. General domain datasets

####  5.2.1. Gold-standard annotations

Eventually, given the lack of a large, general-purpose, gold-annotated DocRE
dataset, Yao et al. (2019)111111https://github.com/thunlp/DocRED decided to
build DocRED. Containing 5,053 gold-annotated documents sampled from
Wikipedia, with an average length of 196.7 words, DocRED was created to handle
more challenging scenarios in DocRE (and even document-level Information
Extraction), such as long documents and complex entity relations. The NER part
includes the classical entity labels such as PERSON, ORGANIZATION, and
LOCATION; while for RE, 96 relation types were used, covering a broad range of
subjects (i.e., science (33.3%), art (11.5%), time (8.3%), personal life
(4.2%)) including works-for, lives-in, place of death, or spouse. The dataset
was created using a hybrid approach of crowdsourcing and distant supervision,
where the crowdsourced annotators were asked to identify and classify entity
types, and the relation labels were derived from KGs. Entities were first
annotated using distant supervision and coreferenced before being manually
curated by the annotators, and then linked to their corresponding Wikidata ID
(entity linking).

However, relation annotation poses a difficulty. For a number $n$ of entities,
there is a quadratic order of possible relations (more exactly
$2\binom{n}{2}=n(n-1)$ in case of binary relations, given that the relations
are directed and that there are no self-relations for entities), and most
entities pairs do not actually have any relation between them. Thus, Yao et
al. (2019) use a recommend-revise scheme. First, distant supervision from
entity linking and RE models was used to generate the relations, and then only
human annotators were asked to review these relations and provide supporting
evidence written in the document, resulting in approximately half of the
relations reserved both for those generated with entity linking and RE models.

This dataset is particularly interesting due to its large-scale and general
purpose, but also due to its complexity. Only 38.9% of the relations can be
identified using simple pattern recognition. Of the other 61.1% of relations,
26.6% require logical reasoning, where the relation is indirectly established
by a bridge entity, 17.6% require coreference reasoning, and 16.6% require
common-sense reasoning. Moreover, Yao et al. (2019) estimate that 40.7% of the
relations can only be identified when reasoning on multiple sentences. The
researchers also included a distantly-supervised version of the dataset
(DocRED distant) with 101,873 annotated documents, far less reliable, but used
in some works for model pretraining (Tan et al., 2022a, b).

Huang et al. (2022)121212https://github.com/AndrewZhe/Revisit-DocRED point out
the fact that the recommend-revise process used in the annotation of DocRED
might cause bias and false negative issues, thus they manually re-annotate 96
articles. Tan et al. (2022b)131313https://github.com/tonytan48/Re-DocRED
equally criticize this scheme because of the Wikidata KB scarcity, which does
not provide all the relations present in the text. In addition, the RE systems
used to create the dataset and should leverage this problem are not up-to-date
anymore, and can not compete with the models that will be trained and
evaluated on DocRED. Furthermore, the inverse relations are missing in DocRED.
To overcome these limitations, they created Re-DocRED by reannotating 4,053
documents from DocRED (3,053 for train, 500 for validation, and 500 for test
sets), generating new relations with SOTA models (KD-DocRE (Tan et al.,
2022a), DocuNET (Zhang et al., 2021), and ATLOP (Zhou et al., 2020a)) and
reviewing them manually. They also added logical rules to include the inverse
and co-occurring relations. As a result, each document possesses more than two
times the number of relations compared to DocRED (12.5 vs. 28.1 on average for
training and 12.3 vs. 34.6 for validation). Nonetheless, they argue that since
the performance on Re-DocRED is slightly better than that on DocRED, the added
triples are of comparable quality to the original DocRED data. This point can
be discussed since the relations coming from logical rules are far easier for
the model to find, it is thus logical that the performance increases with it.
Also, we hold that it is acceptable to compare models given their performances
over a fixed dataset, but not the inverse. Nevertheless, Re-DocRED is a
precious resource in the domain and should be used in combination with, if not
instead of, the original DocRED for evaluating the performances of new DocRE
systems.

Zaporojets et al. (2021)141414https://github.com/klimzaporojets/DWIE released
the Deutsche Welle corpus for Information Extraction (DWIE) consisting of 802
general news articles in English, randomly selected from a corpus collected
from the German broadcaster Deutsche Welle between 2002 and 2018. It was
annotated at the document level for NER, coreference resolution, RE, and
entity linking. In contrast to the previously dominant mention-driven
approaches, the dataset takes an entity-centric approach to describe
interactions and properties of conceptual entities on the level of the
complete document. The annotation process is defined as a bottom-up approach,
involving an exploratory pass, a schema-driven pass, and an inter-rater
refinement.

During the exploratory pass, two students and one expert annotator identify
the entity mentions and freely assign them a label (labels can be overlapping,
i.e., an entity can have many labels), as well as freely annotate the
relations to reflect as closely as possible the meaning of the article. Then,
the schema-driven pass consists in sorting all the annotations into consistent
labels, both for NER (e.g., LOCATION [gpe, regio, waterbody], PERSON
[politician, sport_player, manager], EVENT [holiday, history, protest]) and RE
(e.g., “based_in”, “citizen_of”, “appears_in” or “directed_by”). These new
tags are organized using multi-label types illustrating different
granularities in the sub-types. Moreover, Zaporojets et al. (2021) define some
logical rules to guarantee the consistencies of the annotations, such as
transitivity for example. For inter-rater refinement, the researchers
reannotate 100 documents from scratch following the already defined annotation
scheme. This subset is utilized to measure the inter-annotator agreement. In
the end, the resulting dataset contains numerous annotated entities and
relations and should be being used for benchmarking DocRE, and more largely
Document-level Information Extraction models in combination with DocRED.
However, this is not the case yet, and only a small fraction of recent methods
use it to benchmark their systems (Yu et al., 2022; Han et al., 2022).

####  5.2.2. Silver-standard annotations

To address the lack of a large, general domain, annotated dataset for inter-
sentence RE, Mandya et al. (2018b) produce a dataset using distant
supervision. Taking inspiration from Mintz et al. (2009), Freebase (a large
collaborative knowledge base) is first used to provide relations based on 102
labels that are then extracted from 101,042 Wikipedia sentence pairs. To
balance the dataset, a limit of 2,000 sentence pairs containing the same
relation is set, retaining in total 31,970 sentence pairs for the 17 most
frequent relations. To evaluate the validity of the distant supervision
approach, they choose four relations, for which they verify manually 50
sentence pairs where they appear, resulting in an average of 79% of them being
correct. From this analysis, they conclude that the dataset is sufficiently
accurate for training and testing purposes. However, we argue that no
conclusion can be drawn from the particular to the general, as the reviewed
subset represents less than 1% of the total number of sentence pairs. Despite
its size, this dataset is not much used to benchmark DocRE models, probably
due to a few reasons: (1) the dataset is silver annotated when DocRED (Yao et
al., 2019), the reference dataset for DocRE, came out the year after and was
also focused on the general domain but gold-annotated, (2) the dataset is
balanced, which is not the case for practical applications of DocRE, where one
main difficulty is to extract less frequent relations.

##  6\. Supervised approaches for DocRE

for tree= grow=east, growth parent anchor=west, parent anchor=east, child
anchor=west, edge path=[edge,-¿, ¿=latex] (!u.parent anchor) – +(10pt,0pt) —-
(.child anchor) edge label; [Supervised
DOCRE, basic, l sep=6mm, [Others, xnode, l sep=6mm, [Extra training tasks,
tnode, l sep=6mm, [SAIS (Xiao et al., 2022),
SIEF (Xu et al., 2022)
, wnode] ] [Plug-and-play frameworks, tnode, l sep=6mm [LogiRE (Ru et al.,
2021), wnode] ] [Semantic segmentation, tnode, l sep=6mm, [DocuNet (Zhang et
al., 2021),
DRE-MIR (Zhang and Cheng, 2022a)
, wnode] ]] [Transformers-based approaches, xnode, l sep=6mm, [Joint learning,
tnode, l sep=6mm, [DREEAM (Ma et al., 2023), wnode] ] [Structure reasoning,
tnode, l sep=6mm, [SSAN (Xu et al., 2021), wnode] ] [Efficient use of
distantly supervised labels, tnode, l sep=6mm, [Tan et al. (2022a), wnode] ]
[Multiple relations for one pair, tnode] [Entity representation, tnode, l
sep=6mm, [DEMMT (Han and Wang, 2020),
ATLOP (Zhou et al., 2020a),
RSMAN (Yu et al., 2022)
, wnode] ] [Local to global
information aggregation, tnode, l sep=6mm, [HIN (Tang et al., 2020), wnode] ]
[Early Transformer-based systems, tnode, l sep=6mm, [BERT-Two-Step (Wang et
al., 2019),
CorefBERT (Ye et al., 2020)
, wnode] ] ] [Graph-based
approaches, xnode, l sep=6mm, [Others, tnode, l sep=6mm, [SagDRE (Wei and Li,
2022), wnode] ] [Optimal graph structure, tnode, l sep=6mm, [LSR (Nan et al.,
2020), wnode] ] [Separating intra-
and inter-sentence reasoning, tnode, l sep=6mm, [MRN (Li et al., 2021),
SIRE (Zeng et al., 2021)
, wnode] ] [Using a Heterogeneous graph, tnode, l sep=6mm, [DHG (Zhang et al.,
2020),
GLRE (Wang et al., 2020),
HeterGSAN (Xu et al., 2020),
EGCN (Sun et al., 2022b), DHGCN (Sun et al., 2022a)
, wnode] ] [Entity-centric method, tnode, l sep=6mm, [GAIN (Zeng et al.,
2020),
CorefDRE (Xue et al., 2022),
HAG (Li et al., 2022)
, wnode] ] [Information extraction
in a joint setting, tnode, l sep=6mm, [DyGIE++ (Wadden et al., 2019),
EIDER (Xie et al., 2022)
, wnode] ] [Early graph-based models, tnode, l sep=6mm, [GCNN (Sahu et al.,
2019),
EoG (Christopoulou et al., 2019),
GEDA (Li et al., 2020)
, wnode] ]] [Sequence-based
approaches, xnode, l sep=6mm, [Combining RNN and CNN, tnode, l sep=6mm, [Jia
et al. (2019), wnode] ], [Entity-centric method, tnode, l sep=6mm, [Mandya et
al. (2018a); Wu et al. (2019)
, wnode] ], [First CNN-based approaches, tnode, l sep=6mm,, [Gu et al. (2017);
Li et al. (2018)
, wnode] ] ] ]

Figure 4. Taxonomy of the main DocRE methods.

This chapter presents recent advancements in document-level relation
extraction, focusing on the use of neural models for sequence-, graph-, and
transformer-based approaches. The studies discussed in this chapter propose
novel techniques and frameworks to improve the extraction of relations between
entities across multiple sentences in a document. A taxonomy of the main DocRE
approaches can be seen in Table 4.

###  6.1. Sequence-based approaches

An approach for performing document-level relation extraction is to view the
document as an augmented sequence and apply a sequential model derived from
sentence-level relation extraction to identify relations between specific
entities.

####  6.1.1. First CNN-based approaches

Gu et al. (2017) implement two separated models for extracting Chemical-
Induced Disease relations (CID) on the CDR dataset. First, a maximum entropy
system is used for the inter-sentence level, and a CNN model for the intra-
sentence level. Then, both results are merged to obtain relations at the
document level. During the post-processing stage, if no CID can be extracted
from the abstract, some heuristic rules such as associating all chemicals
mentions in the title of the paper (or the most frequent in the abstract if
necessary) with all diseases mentioned in the abstract are defined.

Li et al. (2018) use recurrent piecewise convolutional neural networks
(RPCNN)151515https://github.com/wglassly/CID_ATTCNN for CID extraction,
combining domain knowledge, piecewise strategy, attention mechanism, and
multi-instance learning. The method first leverages a piecewise CNN to capture
the representations of individual instances belonging to a candidate. Then, a
RNN is utilized to aggregate these representations across the instances within
a record, enabling the generation of a comprehensive document-level
representation for the candidate.

####  6.1.2. Combining RNN and CNN

Mandya et al. (2018a) propose a combined model of long short-term memory
(LSTM) and CNN that exploits word embeddings and positional embeddings for
cross-sentence n-ary relation extraction. The LSTM takes as input the
transformed vector representation combining word embeddings and position
features, the output is fed into a CNN which extracts the most important
features. This approach has the advantage of being independent of intricate
syntactic features, such as dependency trees, coreference resolution, or
discourse features.

For the extraction of gene-disease association, Wu et al. (2019) also combines
a CNN and a RNN with gated recurrent unit (GRU) gates in their RENET
method161616https://bitbucket.org/alexwuhkucs/gda-extraction/src/master/.
Here, CNN first computes sentence representations from word representations
using different filters with different widths to capture the different
features. This aims to capture $n$-gram patterns surrounding the target gene
and target disease, which can potentially indicate a genuine association.
Then, the GRU transforms them into document representation. By ablation
studies, they see that even if the CNN alone might capture some important
features of the document, the extra GRU layer boosts its performance. This
study was expanded later in Su et al.
(2021)171717https://github.com/sujunhao/RENET2 where a section filtering step
is introduced prior to the RENET system, this removes paragraphs that do not
contain any gene or disease pair information and ambiguous relations.

####  6.1.3. Entity-centric method

When at the sentence level, classifying mention tuples might be sufficient,
this is not the case in the document-level setting, where any entity can be
mentioned multiple times, and applying the mention-level method would explode
the combinations of mention tuples. To tackle this problem, Jia et al. (2019)
propose to use an entity-centric multiscale formulation by making a single
relation prediction for each entity tuple. They obtain entity representation
by applying logsumexp over all the mentioned representations of the entity.

Nevertheless, these models inevitably encounter challenges when it comes to
managing context dependencies that span long distances and require multi-hop
reasoning, which is crucial for DocRE (Zhang et al., 2020).

###  6.2. Graph-based approaches

####  6.2.1. Early graph-based DocRE models

The concept of document graph was introduced by Quirk and Poon (2017). It
provides a unifying way to derive features for classifying relations by
representing words as nodes and relations (e.g., dependencies such as
adjacence, syntactic dependencies, and discourse relations) as edges, see
Figure 5. This paved the way for all the graph-based methods for DocRE. Peng
et al. (2017) decided to apply this graph representation as a backbone to
their framework, in which lies a novel graph-LSTM. In classic LSTM
architectures (Hochreiter and Schmidhuber, 1997), each unit has only one
predecessor (so only one forget gate). This is not the case for graph-LSTM,
which can have multiple antecedents including connections to the same word
through different edges. Thus, Peng et al. (2017) take inspiration from tree
LSTMs (Tai et al., 2015) and add for each unit as many forget gates as the
number of precedents. Word embeddings derived from the input text are given as
input of the graph-LSTM, which captures contextual representations for each
word. Entities’ contextual representations are combined through concatenation
and used by the relation classifiers. In the case of multi-word entities, Peng
et al. (2017) compute the average of their word representations, leaving more
sophisticated aggregation methods for future research. As for backpropagation
with graph-LSTM, they split the document graph into two directed acyclic
graphs (DAGs). One DAG contains the left-to-right linear chain as well as
other forward-pointing dependencies (forward-pass), while the other one covers
the right-to-left linear chain and the backward-pointing dependencies
(backward-pass). Notwithstanding, Song et al.
(2018)181818https://github.com/freesunshine0316/nary-grn argue that important
information could be lost in the splitting procedure of the document into the
two DAGs; thus, to keep the original graph structure as a whole, they propose
a parallel state to model each word by recurrently enriching state values via
message passing. By doing so, they remark that the information flow is more
focused on individual words, which are exchanging information with all their
graph neighbors simultaneously at each state transition. This allows the
processing of the entire contextual information for extracting features for
each word, compared to processing each direction of information flow
separately in DAG LSTM and handling arbitrary structures such as cyclic
graphs. This also permits better parallelization, speeding the computations
up. However, as pointed out by Verga et al. (2018), the model is not very
efficient due to its heavy computations, even if it paved the way for all
DocRE graph-based approaches.

Figure 5. Document graph as described by Quirk and Poon (2017). Edges
represent intra-sentential dependencies, as well as connections between the
roots of adjacent sentences. For simplicity, the edges between adjacent words
or representing discourse relations are omitted.

Christopoulou et al. (2018) first proposed a graph walk-based neural network
for sentence-level RE only, treating multiple pairs simultaneously (see Figure
6) and represented sentences as DAGs. Their walk-based algorithm is composed
of two steps. First, during the walk construction phase, two consecutive edges
in the graph are combined using a modified bilinear transformation. Then, a
walk aggregation linearly combines the initial walks and the extended walks.
They later extended their approach to document-level in Christopoulou et al.
(2019)191919https://github.com/fenchri/edge-oriented-graph, making a big leap
forward for DocRE by introducing in the Edge-oriented Graph (EoG) model a
partially-connected document graph instead of the previous fully-connected
one. This new document graph consists of heterogeneous types of nodes and
edges in comparison with the sentence-level graph that contained only entity
nodes and single edge types among them, using multi-instance learning when
mention-level annotations are available. Connections between nodes are based
on pre-defined document-level interactions. The objective is to generate
entity-to-entity edge representations using other existing edges in the graph
and consequently infer entity-to-entity relations. With this task,
Christopoulou et al. (2019) also found that the document-level information
could help to find intra-sentence relations. Early that year, the same team
proposed in Sahu et al. (2019) to apply a Graph Convolutional Network (GCN)
(Marcheggiani and Titov, 2017) enabling message passing. In this approach, a
document graph is constructed with syntactic dependency edge, adjacent
sentence and word edges, and self-node edge but also coreferences edges to
connect dependency trees of the sentence. Guo et al.
(2019)202020https://github.com/Cartus/AGGCN also proposed to extend the GCN
structure for RE by guiding them with attention (ATGGCN) to effectively focus
on the relevant sub-graphs for the task without pruning some important
information.

Figure 6. Architecture of the Christopoulou et al. (2019) approach. The model
receives a document and encodes each sentence separately. A document-level
graph is constructed and fed into an iterative algorithm to generate edge
representations between the target entity nodes. Some node connections are not
shown for brevity (from Christopoulou et al. (2019)).

Relying on the dependency graph introduced in Quirk and Poon (2017), Gupta et
al. (2019)212121https://github.com/pgcool/Cross-sentence-Relation-Extraction-
iDepNN employ a model that captures both the shortest and augmented dependency
paths between two entities, considering both intra- and inter-sentence
relations. They represent each word on a shortest-dependency path (SDP)
through a dependency subtree using a combined structure of bidirectional RNNs
and RecNNs (Socher et al., 2012). Their proposed approach, inter-sentential
Dependency-based Neural Network (iDepNN), incorporates SDP to handle inter-
sentence relations by connecting the root node of a sentence’s parse tree to
the root of the subsequent tree. Additionally, they employ an augmented
dependency path where a RecNN models the dependency subtrees for each word on
the inter-sentential SDP, followed by a bidirectional RNN to extract salient
semantic features. In most cases, the ADP model is better than the SDP one,
however, they did not test it on the benchmark datasets for DocRE. When
comparing their method with (Peng et al., 2017), they remark that their system
is much simpler and more efficient due to its non-cyclic structures
architecture.

Despite these discoveries, Minh Tran et al. (2020) claim that graph-based
edge-oriented models only focus on the graph’s edge representations, ignoring
the representations of the nodes. As a result, they miss potential important
clues contained in the entities and their mentions and prevent the models from
leveraging the relationships between node and edge representations, as well as
the similarity between representation vectors of an entity mention nodes
belonging to the same entities, which could help the system performances.
Therefore, they extend the EoG model from Christopoulou et al. (2019) by
incorporating three losses; one for Node Representation, one for Node-Edge
Representation Consistency, and one for Mention Representation Consistency
(EoGANE).

Li et al. (2020) noticed that since the same relation can be expressed in
different sentences, and inversely that one sentence can display multiple
relations, it should thus be logical to use the attention mechanism to capture
the complex many-to-many interaction. They introduce Graph Enhanced Dual
Attention network (GEDA), which uses a bi-directional attention mechanism
consisting of the attention paid by a sentence to relation instances and the
attention paid by a relation instance to sentences. By doing so, they perform
better than Christopoulou et al. (2019) on CDR and GDA.

####  6.2.2. Information extraction in a joint setting

The work of Luan et al. (2019)222222https://github.com/luanyi/DyGIE focuses
more largely on performing different IE tasks in a joint setting. They first
proposed a sentence-level IE system, using dynamic span graphs (DyGIE) to
perform NER, RE, and coreference resolution simultaneously. Since the model
can be updated as new information is added to the input text, it enables the
propagation of coreference and relation type confidences, which facilitates
the iterative refinement of span representations. This stands in contrast to
previous models, where the only interaction between tasks occurs within the
shared first-layer LSTM. A few months later, they incorporate a cross-sentence
setting into DyGIE and create DyGIE++ (Wadden et al.,
2019)232323https://github.com/dwadden/dygiepp. The difference with DyGIE is
that BERT is employed to generate token representations using a “sliding
window” technique. Spans of text are identified and constructed by
concatenating the tokens representing their left and right boundaries along
with a learned span width embedding. A graph structure is dynamically
generated based on the model’s current estimation of the relations existing
among the spans within the document. From there, equal to DyGIE, message
passing facilitates the incorporation of dependencies that span across
sentences. This improves the performances beyond what can be achieved by BERT
alone, especially on IE tasks in specialized domains.

Wang et al. (2021) use a novel rhetorical structure theory (RST)-GRAPH to
build valid semantic association, as well as a set of evidence reasoning
methods to select appropriate evidence to train their discourse-aware neural
RE model (DISCO-RE). Training DISCO-RE jointly on RE and evidence extraction
permits to alleviate the lack of ability to select and reason over evidence.

In a similar direction, Xie et al.
(2022)242424https://github.com/Veronicium/Eider propose to empower DocRE with
Efficient Evidence Extraction and Inference-stage Fusion (EIDER). During the
training step, the model is jointly trained on RE and evidence extraction
(labels for the latest task coming from both gold and silver annotation),
while during inference, the prediction results on both the original document
and the extracted evidence are combined, which encourages the model to focus
on the important sentences while reducing information loss. Forcing the model
to reason more along the sentences permits it to incorporate more knowledge on
the relations, and thus gives EIDER higher performances.

####  6.2.3. Entity-centric approaches

Zaporojets et al. (2021)252525https://github.com/klimzaporojets/DWIE further
extend DyGIE++ with Deutsche Welle corpus for Information Extraction (DWIE)
incorporating a new graph propagation technique based on attention (AttProp)
which can be used in single-task as well as joint settings. Given a desire to
make the model entity-centric, they cluster the mentions into their entities
and consider relations between the latest. As a consequence, the graph
propagation runs on the whole document. However, they only experimented on the
DWIE dataset.

Analogously, seeking an entity-centric solution to the DocRE problem, Zeng et
al. (2020)262626https://github.com/PKUnlp-icler/GAIN propose Graph
Aggregation-and-Inference Network (GAIN) which incorporates a double graph
design consisting of a heterogeneous mention- and entity-level graph. GAIN
captures complex interactions among different mentions and aggregates those
that pertain to the same entities. This is accomplished with three types of
edges: an intra-entity one to perform coreference resolution, an inter-entity
one to model interactions among entities within a sentence, and a document
edge, with which all mention nodes are connected to the document node. By
establishing these relations, the document node can effectively attend to all
mentions and facilitate interactions between the document and individual
mentions. It also addresses the challenge of long-distance dependency, since
the document node can serve as a pivot for message passing using a GCN (see
Figure 7). This method, in particular, showed a great improvement above SOTA,
but might not be very efficient, being like other graph-based models
computationally intensive. Anyway, it represents a capital step in DocRE.

Figure 7. Architecture of GAIN (Zeng et al., 2020). First, a context encoder
obtains a contextualized representation for each word. Then, the Mention-level
Graph is constructed with mention nodes and a document node. After applying
GCN, an Entity-level Graph is constructed, where the paths between entities
are identified for reasoning. Lastly, the classification module utilizes the
gathered information to make predictions about the target relations. Different
entities are in different colors. The number $i$ in the mention node denotes
that it belongs to the $i$-th sentence (from (Zeng et al., 2020)).

Xue et al. (2022) propose Coref-aware Doc-level Relation Extraction (CorefDRE)
based on Graph Inference Network by instructing the model directly to learn
the coreference by mention-pronoun clustering. They extract all the mentions-
pronouns pairs using spaCy272727https://spacy.io/ and use BERT to calculate
the affinity of each mention-pronoun pair, to reduce the noise brought by
pronouns. CorefDRE first constructs a Mention-Pronoun Affinity Graph (MPAG) by
leveraging the mention-pronoun pairs and their affinity scores, incorporating
pronoun nodes, mention nodes and various types of edges. It further applies a
GCN to MPAG to capture meaningful representations for mentions and pronouns. A
noise suppression mechanism merges pronouns into MPAG, transforming it into an
isomorphic Entity Graph. The final step involves aggregating node and edge
representations in the graph, obtaining a comprehensive representation for
effective analysis and understanding.

Similar to (Xue et al., 2022), Li et al. (2022) introduce a coreference graph
that clusters expressions based on self-supervised affinity computation. These
clusters serve as a node array, enabling the exposure of cross-sentence
reasoning clues. They are then transformed into an Heterogeneous Affinity
Graph (HAG), which combines the coreference graph with the mention graph
proposed by GAIN (Zeng et al., 2020), making it an entity graph. The HAG model
shuffles the similarity to sentences containing head or tail entities using
the affinity weight. To aggregate the information in the graph, HAG employs
the Relational Graph Convolutional Network (R-GCN) (Schlichtkrull et al.,
2018) and a noise suppression mechanism, resulting in a homogeneous entity
graph. Even if this method permits enhancing the modeling of long-distance
relations, the merging of the two graphs still introduces undesirable noise.

####  6.2.4. Using a Heterogeneous graph

To capture both sequential and structural information within documents, Zhang
et al. (2020) propose a Dual-tier Heterogeneous Graph (DHG), effectively
combining these two types of information to facilitate multi-hop reasoning and
final prediction. The model is composed notably of a structure modeling layer,
taking into account sequence, syntax, and hierarchy for encoding the inherent
structure of the document comprehensively. A relation reasoning layer
facilitates multi-hop relational reasoning by propagating relational
information among different entities.

However, most of these models construct relation information between all
entity pairs, without knowing if there is or not a relationship between them.
Thus, Xu et al. (2020)282828https://github.com/xwjim/DocRE-Rec propose to use
reconstruction to force their heterogeneous-based graph self-attention network
(HeterGSAN) to pay more attention to the learning of entity pairs with the
ground-truth relations. The reconstructor generates a sequence of node
representations along the path connecting two entity nodes, maximizing the
probability of the path when a ground-truth relationship exists between the
entity pair. This permits a further improvement in the performances of the
models.

Wang et al. (2020)292929https://github.com/nju-websoft/GLRE argue that the
graph model for which Christopoulou et al. (2019) opted involves irrelevant
information since they are integrated indiscriminately, thus harming the
system performance. Consequently, they use a Global-to-Local Neural Networks
for Relation Extraction (GLRE) for synthesizing entity global, local, and
context relation representations. BERT embeddings are utilized to capture
semantic features and common-sense knowledge, and a heterogeneous graph with
heuristic rules is built to model the complex interactions between all
mentions, entities, and sentences in the document. The entity representations
are learned more effectively by employing an R-GCN for learning global
representations and aggregation of the multiple mentions of specific entities
with multi-head attention for local representations. The context relation
representation is finally learned using self-attention.

Sun et al. (2022b) propose an Enhanced Graph Convolutional Network (EGCN)
model that uses information of node importance provided by PageRank (Page et
al., 1997). All tokens in the document are considered nodes, which reduces the
loss of semantic information. By incorporating different edge types, such as
Dependency edges, Mention edges, and Entity edges, the model is able to
leverage both syntactic dependencies and semantic connections simultaneously.
The importance of each node is then estimated using PageRank, which enhances
the model’s ability to capture topological structural information.

Later, Sun et al. (2022a) also propose to build multi-level graphs, namely
token-level, mention-level, and entity-level graphs in their Dual-Channel and
Hierarchical Graph Convolutional Networks (DHGCN). Moreover, they utilize a
GCN to aggregate interactive information from multi-level graphs to enhance
the reasoning of relations within the document. They also introduce a dual-
channel module that encodes global contextual information and incorporates it
into different levels of graph representation layers.

Drawing inspiration from previous works such as Zeng et al. (2020) and Zhang
et al. (2020), Wang et al. (2023) propose a Multi-layer Heterogeneous Graph
Neural Network (MHGNN) model. It constructs a multi-layer heterogeneous graph
comprising a word-level graph, a mention-level graph, and an entity-level
graph. The first two graphs use a node-type and edge-type oriented
heterogeneous graph attention network (NEHGAN) for message passing,
respectively. Then, the entity-level graph employs the gating-based path
inference mechanism to extract the multi-hop relations.

####  6.2.5. Capturing context information

Zhou et al. (2020b)303030https://github.com/Huiweizhou/GCGCN propose a Global
Context-enhanced Graph Convolutional Networks (GCGCN) composed of two
hierarchical inference blocks with entities as nodes and context of entity
pairs as edges between nodes to capture rich global context information. A
context-aware attention guided graph convolution (CAGGC) block establishes
connections between two entities if they co-occur in at least one sentence.
This allows CAGGC to learn a representation for an entity node by considering
the contextual representations of all its mentions in the document, as well as
the neighboring nodes, thereby encoding both local and global information. On
the other hand, a multi-head attention guided graph convolution (MAGGC) block
applies multi-head attention to generate multiple fully connected edge-
weighted graphs, improving global context representations by connecting entity
pairs from different sentences, and facilitating multi-hop relational
reasoning.

####  6.2.6. Separating intra- and inter-sentence reasoning

Kim et al. (2020) use for their Global level Relation Extractor (GREG) a
divide and conquer approach to avoid this problem entirely by using two
separate models to extract local relations and construct a knowledge graph.
GREG is composed of a LSTM-based module for sentence-level RE and a knowledge
graph constructor module responsible for generating knowledge graphs from the
local relations; both models are trained alternatively.

Li et al. (2021)313131https://github.com/ljynlp/MRN employ a locally and
globally Mention-Based Reasoning Network (MRN) to distinguish the impacts of
close and distant entity mentions. They add a new co-predictor to utilize both
local and global entity representations for jointly reasoning the relations
between close and distant entities. The model includes a two-dimensional
windowed convolution for capturing the local mention-to-mention interactions
between the subject and object arguments of relations. A co-attention module
is added for capturing the global interaction between each pair of entity
mentions, which interact with each other to provide information between local
and global contexts. The model obtained high performances, proving the
interactions between intra- and inter-sentence relations.

With similar reasoning, Zeng et al.
(2021)323232https://github.com/DreamInvoker/SIRE highlight the common practice
of treating intra- and inter-sentential relations similarly in DocRE which
leads to a lack of differentiation in the prediction of these distinct
relation types. Not all entity pairs can be connected by a path, and some
logical reasoning paths may not be fully captured in the graph, such results
in many cases of logical reasoning not being covered. Their method, SIRE,
separates intra- and inter-sentential reasoning. For an intra-sentential
relation, they employ a sentence-level encoder to represent it in every co-
occurred sentence. By aggregating the relational representations from all co-
occurred sentences, they obtain a final representation that encourages intra-
sentential entity pairs to emphasize the local patterns within their specific
context. To capture inter-sentential relations, following their prior work
(Zeng et al., 2020), they adopt a document-level encoder and a mention-level
graph to enable the model to capture both global and local context. To
selectively identify sentences that may signal cross-sentence relations, they
employ an evidence selector, which encourages inter-sentential entity pairs to
selectively focus on sentences that provide supporting evidence for their
cross-sentence relations.

Peng et al. (2022)333333https://github.com/Crysta1ovo/SGR propose a SubGraph
Reasoning (SGR) framework to integrate various paths into a much simpler
subgraph structure to perform various reasonings at once instead of building a
document-level graph first, and then focus on the overall graph structure or
the paths between the target entity pair in the graph. Specifically, they
construct a heterogeneous graph and devise a heuristic approach to generate
reasoning paths, which capture all potential reasoning skills and encompass
relevant annotated supporting sentences. This also ensures that all entity
pairs can be connected by a path. Subsequently, a subgraph is extracted based
on the generated paths, allowing the model to concentrate on critical
entities, mentions, and sentences. Finally, a R-GCN is applied to the
subgraph, enabling joint reasoning across different paths.

####  6.2.7. Optimal graph structure

Nan et al. (2020)343434https://github.com/nanguoshun/LSR discovered that
constructing a graph using co-references and rules does not always lead to the
optimal graph structure. In consequence, they propose to treat the graph
structure as a latent variable and induce it in an end-to-end fashion for
enhanced information aggregation in the whole document. Their Latent Structure
Refinement (LSR) model is comprised of a node constructor, which gives nodes
as contextual representations of each mention, and a dynamic reasoner, which
produces a document-level structure from the iteratively refined nodes.

Huang et al. (2021)353535https://github.com/AndrewZhe/Three-Sentences-Are-All-
You-Need focused on looking at how many sentences were required to identify a
relationship in popular datasets. They found that in DocRED, when paragraphs
were composed of an average of eight sentences, 96.1% instances required no
more than three sentences as supporting evidence, and 87.7% even required only
two or fewer. Similar constatations were made for CDR and GDA. One possible
explanation given by the researchers is that, given the limitations of human
working memory, the nature of the RE task is naturally constrained to a
limited number of entities and contextual information. As described by
Barreyro et al. (2012), working memory is particularly important during
reasoning tasks, but its capacity to separate information chunks is often
limited to around four sentences (Cowan, 2001). Therefore, the common pattern
of three-sentence information chunks observed in the data may be a
subconscious strategy we employ for mutual understanding. Using these
assumptions, Huang et al. (2021) designed three heuristic rules to extract
paths from each document that aim to approximate supporting evidence:
consecutive paths (head and tail entities are within three consecutive
sentences), multi-hop paths (head and tail in distant sentences, but can be
bridged via a pivot entity which co-occurs with the head and the tail in
different sentences), and default paths (all the pairs of sentences where one
contains the head and the other the tail). Even if their method seems
simplistic, they obtained better results by combining it with a BLSTM than
some SOTA methods when the paper was published (Nan et al., 2020; Zeng et al.,
2020), showing that an explicit selection of the optimal paths is better than
fully relying on the graph-based model.

Previous studies have utilized heuristic rules (Christopoulou et al., 2019;
Zeng et al., 2020) or syntactic dependency paths (Sahu et al., 2019; Guo et
al., 2019) to construct a pseudo graph structure for their GNN models,
establishing binary associations between pairs of entities based on task-
independent auxiliary information. These approaches are considered pseudo-
graph structures because they are based on task-independent auxiliary
information and are not directly learned from the data, thus raising certain
issues. Firstly, the graph construction overlooks many actual relational edges
between entities. Secondly, the simplistic information accumulation based on
the binary associative edges makes it harder to model relation-aware
structural knowledge, resulting in noisy representations. To avoid the prior
pseudo-graph structure, Qi et al. (2022) propose Constrained Transmission-
based Graph Convolutional Network (ConstGCN) to perform knowledge-based
information propagation directly between entities without prior graph
construction. Entity representations are updated in ConstGCN at each graph
convolution step by aggregating knowledge-based information transmitted from
neighboring entities across all relation spaces, enabling direct capture of
the relation-aware structural semantics. To precisely follow relational paths
for information transmission without a graph structure, Qi et al. (2022)
transmit scores that constrain the flow of information through uncertain
relational edges, allowing the model to acquire semantic representations while
preserving the original relation-aware structural information. By transmitting
scores, ConstGCN enhances the accuracy and robustness of information
propagation in the absence of a predefined graph structure.

####  6.2.8. Entity representation

In order to address the oversight of non-entity clue words in graph-based
methods for document-level relation extraction, Zhang and Cheng (2022b)
propose Non-entity Clue information for Document-level Relation Extraction
(NC-DRE). NC-DRE utilizes a variant of transformer decoder instead of GNN
models as the decoder component. This variant incorporates a Structured-Mask
Multi-head Self-Attention mechanism, which effectively models heterogeneous
document graphs and enhances the model’s reasoning ability. Additionally, it
introduces a decoder-to-encoder attention mechanism that captures more clue
information, especially non-entity clue information, to improve the model’s
reasoning capability.

In 2023, Liu et al. (2023a)363636https://github.com/UESTC-LHF/GRACR address
the representation of the entity related to its mentions. Instead of simply
averaging the mentions, they propose a graph information aggregation and
logical cross-sentence reasoning network (GRACR). After building mention- and
entity-level graphs, they use an attention mechanism to combine document
embedding, aggregation, and inference information to help identify relations.

####  6.2.9. Data imbalance

One inherent problem to DocRE is data imbalance since most entity pairs do not
possess a relation, i.e. their relation belongs to the N/A label. To alleviate
this problem, Sun et al. (2023) utilize a two-stage framework based on dynamic
graph attention networks (TDGAT). In the first stage, TDGAT employs a binary
classification model to extract links between entity pairs. During the second
stage, it employs these extracted links as prior knowledge to infer fine-
grained relations among the entities.

####  6.2.10. Others

Wei and Li (2022)373737https://github.com/IAmHedgehog/SagDRE propose a
Sequence-Aware Graph-based DocRE model (SagDRE) which learns sentence-level
directional edges to capture the information flow in the document and uses the
token-level sequential information to encode the shortest paths from one
entity to the other. This permits further consideration and captures the
original sequential information from the text. After the embedding generation
and graph construction, multiple GCN and attention layers are stacked to learn
feature representations from local and global perspectives. $k$ shortest paths
are extracted from the graph, encoded by an LSTM, and fused using an attention
layer to create a path embedding. To handle the problem of long-tailed multi-
label problems, where an entity pair in a document can have multiple relations
with a few relations being more prevalent than others, Wei and Li (2022)
utilize an adaptative margin loss, aiming to encourage separations between
positive and negative classes by employing a threshold class.

Focusing on biomedical cross-sentence RE, Zhao et al.
(2021)383838https://github.com/DaveGabbie/Cross-sentence employ a BLSTM to
model the context information, enhance long-distance context encoding through
multi-head attention, and use a GNN to propagate the neighborhood information
into nodes. Another approach is BERT-GT by Lai and Lu (2021). It integrates a
neighbor-attention mechanism into the BERT architecture in combination with a
Graph Transformer (GTN). This modified self-attention mechanism allows tokens
to focus on their neighboring tokens, reducing noise and improving
performance, particularly for long texts spanning multiple sentences. However,
experiments are limited to the CDR (Li et al., 2016) dataset. During this same
year Shi et al. (2021) created DocRE-HGNN, which after encoding a document
with a temporal convolution network to obtain token embeddings containing
long-distance dependency information, incorporates a GTN into a heterogeneous
GNN-based framework.

###  6.3. Transformer-based approaches

####  6.3.1. Early Transformers-based models

Verga et al. (2018)393939https://github.com/patverga/bran introduced a Bi-
affine Relation Attention Network (BRAN) to compute relations across mentions.
BRAN predicts both intra- and inter-sentence relations between mentions
simultaneously. To accomplish this, it first encodes token embeddings using a
transformer layer. It then produces mention-pair predictions using a bi-affine
operation of the form $W(Wx+b)+b$. Finally, it pools the mentions-level
prediction using logsumexp to obtain entity-level predictions. Verga et al.
(2018) mention that the system can be trained jointly for NER and RE, which
improves robustness to noise and lack of gold annotation at the mention level.
This system has the advantage of being more efficient at encoding long
sequences than its contemporary model based on graph-LSTM (Peng et al., 2017).

To alleviate the data imbalance issue between positive and negative relations,
Wang et al. (2019)404040https://github.com/hongwang600/DocRed adopt a two-step
finetuning process with BERT. The former identifies if a relation exists
between two mentions, and the latter identifies these entities using only the
previously identified relation facts to train the model.

To enhance coreferential reasoning in language representation models, Ye et
al. (2020)414141https://github.com/thunlp/CorefBERT tested their CorefBERT
system on various downstream tasks, notably DocRE. Instead of relying on
supervised coreference resolution data for fine-tuning, they introduce a novel
pre-training task called Mention Reference Prediction which leverages repeated
mentions in a passage and employs mention reference masking to predict the
corresponding referents. Additionally, they introduce a copy-based training
objective to encourage context-based word selection and context-sensitive
representations to facilitate coreferential reasoning.

####  6.3.2. Local to global information aggregation

To aggregate inference information from entity level to sentence level and
then to document level, Tang et al. (2020) use a Hierarchical Inference
Network (HIN). Inspired by the translation constraint given in Bordes et al.
(2013) which modeled a relation triplet $r(m_{h},m_{t})$ with $m_{h}+r\approx
m_{t}$, they represent the relation between $\varepsilon_{a}$ and
$\varepsilon_{b}$ such as $\varepsilon_{b}-\varepsilon_{a}\approx r$ for
extending the assumption to the document-level. Even if it works, we argue
about the exactitude of this assumption since in DocRE two entity pairs can
have multiple relations, and one entity can have the same relation with
multiple other entities. They employ then a bi-affine layer to get a bilinear
representation for the target entity pair. The two previous transformations
are applied in parallel in different subspaces, acquiring entity-level
inference information. Afterward, they apply a semantic matching method to
compare the entity-level inference information with each sentence vector and
use a hierarchical BiLSTM. Finally, they re-employ the attention mechanism to
identify the crucial sentence-level inference information, which contributes
to the overall representation of document-level inference in a more
comprehensive manner.

Kuang et al. (2022) argue that the relation between entity pairs can usually
be inferred just through a few keywords. To take benefit from this, they use a
Self-Attention Memory module in combination with a BERT-LSTM-based model to
capture keyword features by looking for word embeddings with high cross-
attention of entity pairs. Their model, much simple, achieves good
improvements on both specialized and non-specialized datasets and can be
applied to other ones.

####  6.3.3. Entity representation

Han and Wang (2020) utilize a document-level entity mask method with type
information (DEMMT) to provide more information about the entities in DocRE.
It introduces entity characteristics into BERT embeddings, enriching the
representations with both contextual and entity-specific information. DEMMT
first employs BERT to obtain contextualized representations of the entity
mentions. It then uses a one-pass relation prediction method to process all
the entity pairs at one time, so the text is only encoded once.

Adopting the same entity-centrism point of view as Zeng et al. (2020) and
Zaporojets et al. (2021), Yu et al.
(2022)424242https://github.com/FDUyjx/RSMAN also account for the importance
between each mention to represent an entity regarding a specific relation.
They use a Relation-Specific Mention Attention Network (RSMAN) to incorporate
essential semantics of each relation into prototype representations, and then
calculate the attention between a candidate relation’s prototype and the
representations of mentions associated with the given entity. By considering
these attentions, the model obtains a synthetic representation of the entity
by weighted aggregation of all mention representations. This enables the model
to capture information from multiple mentions in different representation
spaces, resulting in flexible and relation-specific entity representations for
different candidate relations. RSMAN can be used as a plug-in backbone to any
RE model.

As said previously, in DocRE, one document contains more than one entity pair
which can occur multiple times in the document and bear multiple possible
relations. To tackle these issues, Zhou et al.
(2020a)434343https://github.com/wzhouad/ATLOP utilize two techniques, Adaptive
Thresholding and Localized cOntext Pooling (ATLOP) over a BERT baseline. The
adaptive thresholding permits deciding the best threshold for each entity pair
by switching the global one for multi-label classification with a learnable
entities-dependent one. The localized context pooling directly transfers
attention from pre-trained language models to locate a relevant context that
is useful to decide the relation. This solves the problem of using the same
entity embedding for all entity pairs. ATLOP enriches the entity embedding
with additional context relevant to the current entity pair, enabling a more
comprehensive representation, and making a giant leap for DocRE. It is one of
the most, if not the most, popular model for this task.

####  6.3.4. Efficient use of distantly supervised labels

For its part, Xiao et al. (2020)444444https://github.com/thunlp/DSDocRE try to
reduce the inherent noise induced in distantly supervised labels via three
pre-training tasks: Mention Entity Matching, which involves intra- and inter-
document matching, the first focuses on establishing coreference connections
within a document, while the other captures entity associations across
documents; Relation Detection, which focuses on detecting the positive
relations wrongly labeled as negative; and Relational Fact Alignment, which
ensures that the model generates consistent representations for the same
entity pair, even when expressed in different ways. Xiao et al. (2020) also
use Relation Detection to conduct a pre-denoising module to filter out
negative instances before pre-training.

Tan et al. (2022a)454545https://github.com/tonytan48/KD-DocRE propose a multi-
step approach to leverage distantly supervised data. Firstly, they employ an
axial attention module as a feature extractor, allowing for attention to
elements within two-hop logical paths and capturing the interdependency among
relation triplets. Secondly, to handle class imbalance, they introduce an
Adaptive Focal Loss that focuses on the long-tail classes, making them
contribute more to the overall loss. Lastly, they utilize knowledge
distillation to bridge the gap between annotated and distantly supervised
data. It consists of a teacher model trained with a small amount of human-
annotated data that generates predictions on a large amount of distantly
supervised data, this serves as soft labels for pre-training a student model.

####  6.3.5. Structure reasoning

In their paper, Xu et al. (2021) point out one problem of graph-based
approaches for DocRE: they often separate the stages of context reasoning and
structure reasoning due to the heterogeneity between the encoding network and
graph network, leading to a lack of structure guidance for contextual
representations. Thus, they argue that structural dependencies should be
incorporated within the encoding network and throughout the overall system.
They introduce the Structured Self-Attention Network (SSAN), where the entity
structure is formulated under a unified framework. Leveraging a novel
extension of the self-attention mechanism (Vaswani et al., 2017), SSAN
effectively models mention dependencies within its building blocks and across
all network layers bottom-to-up by considering two entity structures, one for
co-occurrence in the same sentence, and one for coreference. In addition to
the dependencies between entity mentions, Xu et al. (2021) also use an
attention module to capture the relationship between entity mentions and
intra-sentential non-entity words. Parallel to the attention module, they
incorporate an additional module that models structural dependencies
conditioned on the contextualized query/key representations. This design
allows the model to benefit from the guidance of structural dependencies,
ensuring that both contextual and structural information are effectively
integrated.

####  6.3.6. Joint learning

Ma et al. (2023)464646https://github.com/YoumiMa/dreeam address two issues in
learning both DocRE and Evidence Extraction: high resource consumption and
lack of human-annotated data. For their Document-level Relation Extraction
with Evidence-guided Attention Mechanism (DREEAM), they propose to direct an
ATLOP model (Zhou et al., 2020a) on Evidence Extraction. Instead of using a
separate classifier (like in Xie et al. (2022)), DREEAM supervises the
computation of entity-pair-specific local context embeddings and incorporates
attention weights over tokens as evidence scores. This approach allows them to
introduce knowledge from evidence extraction without introducing additional
trainable parameters or expensive matrix computations. To alleviate the lack
of data they use a similar knowledge distillation technique to Tan et al.
(2022a).

###  6.4. Other approaches

####  6.4.1. Semantic segmentation

Zhang et al. (2021)474747https://github.com/zjunlp/DocuNet use a Document
U-shaped Network (DocuNet) to apply a similar approach to semantic
segmentation in computer vision, reformulating the task as an entity-level
classification problem (see Figure 8). Given relevant features between entity
pairs as an image, the model predicts the relation type for each entity pair
as a pixel-level mask. An encoder module captures the context information of
entities and a U-shaped segmentation model (Ronneberger et al., 2015) captures
global interdependency over the image-style feature map, a matrix containing
the entity-to-entity relevance estimation. To handle the imbalance relation
distribution, Zhang et al. (2021) use a balanced softmax method. This original
approach constitutes an important milestone in DocRE and is one of the most
popular in this domain.

Figure 8. Architecture of DocuNet (from (Zhang et al., 2021)).

However, Zhang and Cheng (2022a) claims that capturing correlations between
relations through convolutional neural networks might be unintuitive and
inefficient due to the intrinsic distinction between entity-pair matrices and
images. Instead, they use a Document-level Relation Extraction model based on
a Masked Image Reconstruction network (DRE-MIR), which formulates the
inference problem in DocRE as a masked image reconstruction problem. They
adopt the DocuNet method but introduce a random masking process on the entity-
pair matrix. The model then reconstructs the masked entity-pair matrix through
the inference model. This approach leverages the Masked Image Reconstruction
task to learn how to infer the masked points by utilizing the correlations
between relations. To enhance the efficiency and effectiveness of the masked
point reconstruction, they incorporate an Inference Multi-head Self-Attention
module, which significantly improves the model’s inference capabilities.

####  6.4.2. Plug-and-play frameworks

Because of the lack of transparency given by DocRE models, especially graph-
based ones which rely heavily on implicitly powerful learned representations,
Ru et al. (2021)484848https://github.com/rudongyu/LogiRE propose LogiRE to
treat logic rules as latent variables. It consists of a rule generator and a
relation extractor, trained simultaneously and enhancing each other using the
EM algorithm. The rule generator provides logic rules that guide the relation
extractor for prediction, while the relation extractor provides supervision
signals to optimize the rule generator. This approach significantly reduces
the search space and allows LogiRE to be integrated as a plug-and-play
technique with existing RE models.

####  6.4.3. Extra training tasks

Han et al. (2022) proposed to use the interdependency of relations (i.e.
relations in the real world barely appear independently, they usually
interdepend on each other and appear together), which may be rich for long-
tailed relations. Thus, transferring knowledge from data-rich relations to
data-scarce ones might improve performance in relation extraction. To exploit
this, they propose a new task, Fined-grained Relation Co-occurrence Prediction
(FRCP), which determines whether one relation cooccurs with a set of other
relations, based on both relations and entity pairs. The learned relation
embeddings which could perceive relation correlations thanks to the two
auxiliary sub-tasks, accompanied by jointly-trained document contextual
representations, are used to extract relational facts.

To tackle the problems related to ineffective supervision and uninterpretable
model predictions, Xiao et al.
(2022)494949https://github.com/xiaoyuxin1002/SAIS propose a method called
Supervising and Augmenting Intermediate Steps (SAIS). They identify
coreference resolution, NER, pooled evidence retrieval, and fine-grained
evidence retrieval to compose four intermediate steps involved in the
reasoning process. By explicitly supervising the model’s outputs in these
steps and incorporating relevant contexts and entity types, Xiao et al. (2022)
aims to improve the quality and explainability of the final outputs.

Xu et al. (2022)505050https://github.com/xwjim/SIEF focus on increasing the
robustness of DocRE models: they see that most models predict correctly when
an entire test document is fed as input, but display some errors when non-
evidence sentences are removed. Thus, they suggest a new generic framework,
Sentence Importance Estimation and Focusing (SIEF), composed of a sentence
importance score and a sentence focusing loss. The first is calculated based
on the importance of each sentence in the document when it is removed from the
text and is used by the second to encourage the DocRE model to focus on
evidence sentences by penalizing the model for predicting relations based on
non-evidence sentences.

##  7\. Comparative results

Although DocRE methods may vary according to the type of relationship to be
extracted (gene-disease associations, generic relationships, etc.), most
approaches are evaluated on DocRED, so we will use the latter to compare the
results of the main methods. Comparative results for CDR and GDA datasets are
given respectively on Tables 5 and 6. Unfortunately, since only a few methods
used DWIE (Zaporojets et al., 2021) to benchmark their model yet (Yu et al.,
2022; Han et al., 2022), we are not able to make a comparative study on this
dataset. However, we insist on the fact that it should be used in the same way
as DocRED for future articles, and give the available results in Appendice A.
The same can be said for BioRED (Luo et al., 2022), and Re-DocRED (Tan et al.,
2022b) instead of DocRED. Since using BERT in its large version does not
affect the reasoning and comparison but only leads to better scores by using a
larger model, while polluting more, all BERT-based models results are given
for its base version.

With respect to evaluation metrics used to provide these results, we will
consider mainly the F1 score and IgnF1 for DocRED. Before DocRED came out,
both Precision and Recall were usually provided in the papers, in addition to
the F1 score. However, since 2020, there is a growing tendency to only
indicate the F1 score, and the IgnF1, a computation of the F1 score which
excludes the relations that present in both the training set and the
validation and test sets.

First, we see in Table 3 the impact of GNN on model performances, and
particularly GCN, on the domain, notably with the large improvement between
the baselines (Yao et al., 2019) and the first convolutional graphs methods
(Sahu et al., 2019; Christopoulou et al., 2019). In the same way, we can
notice in Table 4, the great impact of BERT-based models, even the “simplest”
ones which can show great performances when incorporating a better
understanding of the text, via coreferences (Ye et al., 2020) or denoising
tricks (Xiao et al., 2020). We see that adopting an entity-centric point of
view greatly improves performances, which can be seen in the results of GAIN
(Zeng et al., 2020).

Even if the transformer-based methods hardly compete with the graph-based ones
on the front of performances, its adaptive thresholding and contextualized
pooling give ATLOP (Zhou et al., 2020a) great results, even better than its
contemporary graph-based models (Nan et al., 2020). Further improved with an
efficient Evidence Extraction task (Ma et al., 2023), it can rivalize and even
surpass the best graph models. This shows that, even if they perform well, the
DocRE task is not constrained to graph-based solutions. The best example to
illustrate this is DocuNet (Zhang et al., 2021), which outperformed all of the
previous models on DocRED by inspiring its solution from computer vision, only
being surpassed by SIRE (Zeng et al., 2021) in the same year. In 2022, this
approach will once again give the best results with DRE-MIR (Zhang and Cheng,
2022a).

The great performances of EIDER (Xie et al., 2022), NC-DRE (Wei and Li, 2022),
and EGCN (Sun et al., 2022b), three graph-based models which use different
ways to incorporate more knowledge in the model (EIDER by doing evidence
extraction jointly with RE during training, and the other two by improving
entities representations in the graph), show that making the model reasoning
more over the text greatly improves the results. EGCN, SIRE (Zeng et al.,
2021), and DHGCN (Sun et al., 2022a) also show the necessity of implementing
special strategies to represent both local and global information in the DocRE
setting. EGCN in particular shows a boom in performances (+6% F1). Its usage
of tokens as the graph’s nodes, then connected with dependency, mention, and
entity edges, limits the loss of information. This, combined with the great
focus on nodes (thus token) importance computed by PageRank forces a great
comprehension of the document, which explains this large step in performance.

Model | IgnF1 | F1
---|---|---
Baselines
CNN (Yao et al., 2019) | 36.44 | 42.33
LSTM (Yao et al., 2019) | 47.71 | 50.07
Bi-LSTM (Yao et al., 2019) | 48.78 | 51.06
Context-Aware (Yao et al., 2019) | 43.93 | 50.64
Graph based
GCNN (Sahu et al., 2019) * | 49.59 | 51.62
EoG (Christopoulou et al., 2019) * | 49.48 | 51.82
GAT (Veličković et al., 2018) * | 47.36 | 49.51
GEDA (Li et al., 2020) | 51.22 | 52.97
GREG (Kim et al., 2020) | - | 52.88
GCGCN-GloVe (Zhou et al., 2020b) | 50.87 | 53.13
HIN-GloVe (Tang et al., 2020) | 51.15 | 53.30
LSR+GloVE (Nan et al., 2020) | 52.15 | 54.18
GAIN-GloVe (Zeng et al., 2020) | 52.66 | 55.08
HeterGSAN (Xu et al., 2020) | 52.07 | 53.52
HeterGSAN+reconstruction (Xu et al., 2020) | 53.27 | 55.23
Paths + BLTSM (Huang et al., 2021) | - | 56.23
MRN+GloVE (Li et al., 2021) | 56.19 | 58.46
SIRE+GloVe (Zeng et al., 2021) | 54.04 | 55.96
CorefDRE+GloVe (Xue et al., 2022) | 54.37 | 56.74
SGR (Peng et al., 2022) | 57.15 | 55.12
SagDRE + GloVe (Wei and Li, 2022) | 53.90 | 56.23
BLSTM+SIEF (Xu et al., 2022) | 51.03 | 53.22
HeterGSAN+SIEF (Xu et al., 2022) | 53.94 | 55.85
GAIN+SIEF (Xu et al., 2022) | 54.72 | 56.75
HAG+GloVe (Li et al., 2022) | 54.37 | 56.74
EGCN+GloVe (Sun et al., 2022b) | 61.26 | 63.93
DHGCN+GloVe (Sun et al., 2022a) | 60.15 | 61.67
Table 3. Comparative Results on the DocRED dataset for non BERT-based methods.
All results come from their cited paper, except those with * for which it
comes from Nan et al. (2020).

Ultimately, DocRE is a very difficult task. Currently on DocRED, the best
results are only around 65% F1, and approximately 15% F1 has been gained
during the past four years. Graph models, while not perfect, are more
performant than transformer-based ones on this task, but less efficient. It
should also be pointed out that there is only one generic benchmark dataset to
assess this fact (hence the need to also use Re-DocRED and DWIE to benchmark
the models). Nonetheless, the semantic segmentation approach is very promising
and should be deeply investigated. Still, no model has been proposed that
comprehensively addresses all the inherent challenges in DocRE.

Model | IgnF1 | F1
---|---|---
Graph based
BERT-GEDA (Li et al., 2020) | 53.71 | 55.74
GCGCN-BERT (Zhou et al., 2020b) | 54.53 | 56.67
GAIN-BERT (Zeng et al., 2020) | 59.00 | 61.24
GAIN+SIEF (Xu et al., 2022) | 59.87 | 62.29
GLRE-BERT (Wang et al., 2020) | 55.40 | 57.40
LSR-BERT(Nan et al., 2020) | 56.97 | 59.05
HeterGSAN-BERT (Xu et al., 2020) | 56.21 | 58.54
HeterGSAN-BERT+reconstruction (Xu et al., 2020) | 57.12 | 59.45
HeterGSAN-BERT+SIEF (Xu et al., 2022) | 57.93 | 60.02
DISCO-RE (Wang et al., 2021) | 55.01 | 55.70
MRN-BERT (Li et al., 2021) | 59.52 | 61.74
SIRE-BERT (Zeng et al., 2021) | 60.18 | 62.05
CorefDRE-BERT (Xue et al., 2022) | 60.78 | 60.82
EIDER-BERT (Xie et al., 2022) | 60.42 | 62.47
NC-DRE (Zhang and Cheng, 2022b) | 60.59 | 62.73
SagDRE + BERT (Wei and Li, 2022) | 60.11 | 62.32
HAG-BERT (Li et al., 2022) | 60.78 | 60.82
EGCN-BERT (Sun et al., 2022b) | 65.95 | 68.72
ConstGCN-BERT (Qi et al., 2022) | 59.58 | 61.55
DHGCN-BERT (Sun et al., 2022a) | 65.21 | 66.87
GRACR (Liu et al., 2023a) | 56.47 | 58.54
TDGAT-BERT (Sun et al., 2023) | 59.35 | 61.81
MHGNN (Wang et al., 2023) | 59.93 | 62.27
Transformer based and others
BERT (Wang et al., 2019) | - | 53.20
BERT-Two-Step (Wang et al., 2019) | - | 53.92
CorefBERT (Ye et al., 2020) | 54.54 | 56.96
BERT+Denoising module (Xiao et al., 2020) | 55.53 | 57.20
BERT+Denoising module+Pretraining tasks (Xiao et al., 2020) | 56.43 | 56.68
BERT+DEMMT (Han and Wang, 2020) | 54.93 | 57.13
ATLOP-BERT (Zhou et al., 2020a) | 59.31 | 61.30
HIN-BERT (Tang et al., 2020) | 53.70 | 55.60
SSAN+BERT (Xu et al., 2021) | 55.84 | 58.16
DocuNet + BERT (Zhang et al., 2021) | 59.93 | 61.86
CorefBERT+RSMAN (Yu et al., 2022) | 55.30 | 57.53
SSAN+RSMAN (Yu et al., 2022) | 57.02 | 59.29
BERT+Correlation (Han et al., 2022) | 59.12 (+1.01) | 61.32 (+1.12)
DRE-MIR-BERT(Zhang and Cheng, 2022a) | 60.91 | 62.85
BERT-SAM (Kuang et al., 2022) | 60.68 | 62.02
(Tan et al., 2022a) (+BERT) | 62.56 | 64.76
SAIS (Xiao et al., 2022) | 60.96 | 62.77
BERT+SIEF (Xu et al., 2022) | 57.87 | 58.93
DREEAM (Ma et al., 2023) | 63.73 | 65.87
Table 4. Comparative Results on the DocRED dataset for BERT based methods. All results come from their cited paper, except those with * for which it comes from Nan et al. (2020). Model | F1
---|---
Baselines
ME-CNN (Gu et al., 2017) | 61.3
BRAN (Verga et al., 2018) | 62.1
RPCNN (Li et al., 2018) | 59.1
Non BERT embeddings
GCNN (Sahu et al., 2019) | 58.6
EoG (Christopoulou et al., 2019) | 63.6
EoGANE (Minh Tran et al., 2020) | 66.1
LSR (Nan et al., 2020) | 61.2
DHG-LSTM (Zhang et al., 2020) | 64.7
MRN+BLSTM (Li et al., 2021) | 65.9
EGCR+GloVe (Sun et al., 2022b) | 68.2
DHGCN+GloVe (Sun et al., 2022a) | 73.1
BERT embeddings
DHG-BERT (Zhang et al., 2020) | 65.9
SciBERT (Zhou et al., 2020a) | 65.1
ATLOP-SciBERT (Zhou et al., 2020a) | 69.4
GLRE-BERT (Wang et al., 2020) | 68.5
(Zhao et al., 2021) | 61.0
BERT-GT (Lai and Lu, 2021) | 65.99
DocRE-HGNN (Shi et al., 2021) | 64.4
SSAN+BERT (Xu et al., 2021) | 62.7
SSAN+SciBERT (Xu et al., 2021) | 68.7
SIRE+BioBERT (Zeng et al., 2021) | 70.8
DocuNet+SciBERT (Zhang et al., 2021) | 76.3
SagDRE+SciBERT (Wei and Li, 2022) | 71.8
EIDER+SciBERT (Xie et al., 2022) | 70.63
NC-DRE+SciBERT (Zhang and Cheng, 2022b) | 72.05
SAIS+SciBERT (Xiao et al., 2022) | 79.0
DRE-MIR-BERT(Zhang and Cheng, 2022b) | 76.8
EGCR+BERT (Sun et al., 2022b) | 72.5
DHGCN+BERT (Sun et al., 2022a) | 76.3
GRACR (Liu et al., 2023a) | 68.8
MHGNN (Wang et al., 2023) | 73.04
Table 5. Comparative Results on the CDR dataset. Model | F1
---|---
Baselines
CNN (Wu et al., 2019) | 81.3
RENET (Wu et al., 2019) | 83.5
Non BERT embeddings
EoG (Christopoulou et al., 2019) | 81.5
EoGANE (Minh Tran et al., 2020) | 82.8
LSR (Nan et al., 2020) | 79.6
DHG-LSTM (Zhang et al., 2020) | 82.2
MRN+BLSTM (Li et al., 2021) | 82.9
EGCR+GloVe (Sun et al., 2022b) | 83.1
DHGCN+GloVe (Sun et al., 2022a) | 85.6
BERT embeddings
DHG-BERT (Zhang et al., 2020) | 83.1
SciBERT (Zhou et al., 2020a) | 82.5
ATLOP-SciBERT (Zhou et al., 2020a) | 83.9
DocRE-HGNN (Shi et al., 2021) | 81.6
SSAN+BERT (Xu et al., 2021) | 82.1
SSAN+SciBERT (Xu et al., 2021) | 83.7
SIRE+BioBERT (Zeng et al., 2021) | 84.7
DocuNet+SciBERT (Zhang et al., 2021) | 85.3
EIDER+SciBERT (Xie et al., 2022) | 84.54
NC-DRE+SciBERT (Zhang and Cheng, 2022b) | 85.80
SAIS+SciBERT (Xiao et al., 2022) | 87.1
DRE-MIR-BERT(Zhang and Cheng, 2022b) | 86.4
EGCR+BERT (Sun et al., 2022b) | 87.6
DHGCN+BERT (Sun et al., 2022a) | 88.2
Table 6. Comparative Results on the GDA dataset.

##  8\. Limitations, challenges, and directions for future work

Our analysis delves into the limitations of the existing body of literature
surrounding the DocRE task, anticipates future challenges, and outlines
potential avenues for enhancement. DocRE is a challenging task that requires
more research, as evidenced by the limited number of large-scale gold-
annotated datasets and the predominant reliance on a single general-purpose
dataset, DocRED (Yao et al., 2019), alongside two domain-specific datasets,
CDR (Li et al., 2016) and GDA (Wu et al., 2019), for benchmarking purposes.
Notably, we advocate for a more extensive utilization of DWIE (Zaporojets et
al., 2021) and BioRED (Luo et al., 2022), both of which feature gold-standard
annotations, as well as Re-DocRED (Tan et al., 2022b), underscoring that while
DocRED serves a valuable purpose, it may not still the optimal benchmark for
assessing model performance.

While graph-based methods have shown promising results, they come with
substantial computational overhead compared to transformer-based counterparts.
Consequently, there is an enticing avenue for exploration, with large language
models (LLM) emerging as a novel direction, especially in light of their
cutting-edge performance across various NLP tasks (Brown et al., 2020; OpenAI,
2023; Liu et al., 2023b). Similarly, it would be interesting to apply SOTA
methods for semantic segmentation to DocRE (Thisanke et al., 2023).

In addition to these technical challenges, there are also some important
practical considerations for DocRE. As emphasized by Boudjellal et al. (2020),
the initial step of NER bears immense significance in the DocRE pipeline. An
inadequately executed NER task can severely compromise the accuracy of the
entire DocRE process. Furthermore, Zaporojets et al. (2021) have highlighted
the intricate interplay between relations and entities, suggesting that joint
training of NER and RE models could yield substantial benefits. Thus, a
concerted effort should be directed towards advancing the joint performance of
NER and RE. Moreover, the majority of existing DocRE datasets focus primarily
on named entities, overlooking the potential importance of other entity types,
such as terms and concepts.

##  9\. Conclusion

As document-level relation extraction is far more difficult than its sentence-
level counterpart, more research is needed to improve results. Indeed, as this
survey shows the domain remains relatively shallow compared to its numerous
use cases: only a few large-scale gold-annotated datasets are available and
just one dataset is used to benchmark the models, despite some inherent
problems. While the evolution of the solutions in the domain these last six
years seems promising, we note that there is still room for improvement as the
document-level setting conceals many subtleties. A possible direction for
future work is to improve the efficiency of graph-based methods since they are
computationally intensive but with competitive results.

Our survey compiles a comprehensive overview of the advancements made in the
DocRE task from 2016 to mid 2023. We first defined the DocRE task, and
provided a comprehensive list of the well-annotated datasets available in the
domain, highlighting their advantages and limitations. Then, we summarized the
key sequence-based, graph-based, and transformer-based approaches and made a
comparison of their performance in supervised DocRE. Ultimately, it brings
into focus certain constraints inherent in the proposed methodologies and puts
forward recommendations for prospective endeavors.

##  10\. Acknowledgements

The first author was partly funded by Région Nouvelle-Aquitaine. This work has
also been supported by the TERMITRAD (AAPR2020-2019-8510010) project funded by
the Nouvelle-Aquitaine Region, France.

## References

  * (1)
  * Abu-Salih (2021) Bilal Abu-Salih. 2021\.  Domain-specific knowledge graphs: A survey.  _Journal of Network and Computer Applications_ 185 (2021), 103076\.
  * Bach and Badaskar (2007) Nguyen Bach and Sameer Badaskar. 2007.  A review of relation extraction.  _Literature review for Language and Statistics II_ 2 (2007), 1–15.
  * Barreyro et al. (2012) Juan Pablo Barreyro, Jazmín Cevasco, Débora Burín, and Carlos Molinari Marotto. 2012\.  Working memory capacity and individual differences in the making of reinstatement and elaborative inferences.  _The Spanish journal of psychology_ 15, 2 (2012), 471–479.
  * Bassignana and Plank (2022) Elisa Bassignana and Barbara Plank. 2022.  What Do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_. Association for Computational Linguistics, Dublin, Ireland, 67–83.  https://doi.org/10.18653/v1/2022.acl-srw.7
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013\.  Translating Embeddings for Modeling Multi-relational Data. In _Advances in Neural Information Processing Systems_ , C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (Eds.), Vol. 26. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2013/file/1cecc7a77928ca8133fa24680a88d2f9-Paper.pdf
  * Boudjellal et al. (2020) Nada Boudjellal, Huaping Zhang, Asif Khan, and Arshad Ahmad. 2020\.  Biomedical relation extraction using distant supervision.  _Scientific Programming_ 2020 (2020), 1–9.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Christopoulou et al. (2018) Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2018\.  A Walk-based Model on Entity Graphs for Relation Extraction. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, Melbourne, Australia, 81–88.  https://doi.org/10.18653/v1/P18-2014
  * Christopoulou et al. (2019) Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019\.  Connecting the Dots: Document-level Neural Relation Extraction with Edge-oriented Graphs. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 4925–4936.  https://doi.org/10.18653/v1/D19-1498
  * Cohen and Hersh (2005) Aaron M Cohen and William R Hersh. 2005.  A survey of current work in biomedical text mining.  _Briefings in bioinformatics_ 6, 1 (2005), 57–71.
  * Cowan (2001) Nelson Cowan. 2001\.  The magical number 4 in short-term memory: A reconsideration of mental storage capacity.  _Behavioral and brain sciences_ 24, 1 (2001), 87–114.
  * Davis et al. (2009) Allan Peter Davis, Cynthia G Murphy, Cynthia A Saraceni-Richards, Michael C Rosenstein, Thomas C Wiegers, and Carolyn J Mattingly. 2009.  Comparative Toxicogenomics Database: a knowledgebase and discovery tool for chemical–gene–disease networks.  _Nucleic acids research_ 37, suppl_1 (2009), D786–D792.
  * de Abreu et al. (2013) Sandra Collovini de Abreu, Tiago Luis Bonamigo, and Renata Vieira. 2013.  A review on Relation Extraction with an eye on Portuguese.  _Journal of the Brazilian Computer Society_ 19 (2013), 553–571.
  * Dienstmann et al. (2015) Rodrigo Dienstmann, In Sock Jang, Brian Bot, Stephen Friend, and Justin Guinney. 2015\.  Database of genomic biomarkers for cancer drugs and clinical targetability in solid tumors.  _Cancer discovery_ 5, 2 (2015), 118–123.
  * Gu et al. (2017) Jinghang Gu, Fuqing Sun, Longhua Qian, and Guodong Zhou. 2017\.  Chemical-induced disease relation extraction via convolutional neural network.  _Database_ 2017 (04 2017).  https://doi.org/10.1093/database/bax024 arXiv:https://academic.oup.com/database/article-pdf/doi/10.1093/database/bax024/19233027/bax024.pdf  bax024.
  * Guo et al. (2019) Zhijiang Guo, Yan Zhang, and Wei Lu. 2019.  Attention Guided Graph Convolutional Networks for Relation Extraction. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 241–251.  https://doi.org/10.18653/v1/P19-1024
  * Gupta et al. (2019) Pankaj Gupta, Subburam Rajaram, Hinrich Schütze, and Thomas Runkler. 2019.  Neural Relation Extraction within and across Sentence Boundaries. In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence_ (Honolulu, Hawaii, USA) _(AAAI’19/IAAI’19/EAAI’19)_. AAAI Press, Article 799, 8 pages.  https://doi.org/10.1609/aaai.v33i01.33016513
  * Han et al. (2022) Ridong Han, Tao Peng, Benyou Wang, Lu Liu, and Xiang Wan. 2022.  Document-level Relation Extraction with Relation Correlations.  _arXiv preprint arXiv:2212.10171_ (2022).
  * Han and Wang (2020) Xiaoyu Han and Lei Wang. 2020.  A Novel Document-Level Relation Extraction Method Based on BERT and Entity Information.  _IEEE Access_ 8 (2020), 96912–96919.  https://doi.org/10.1109/ACCESS.2020.2996642
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Huang et al. (2022) Quzhe Huang, Shibo Hao, Yuan Ye, Shengqi Zhu, Yansong Feng, and Dongyan Zhao. 2022\.  Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Dublin, Ireland, 6241–6252.  https://doi.org/10.18653/v1/2022.acl-long.432
  * Huang et al. (2021) Quzhe Huang, Shengqi Zhu, Yansong Feng, Yuan Ye, Yuxuan Lai, and Dongyan Zhao. 2021\.  Three Sentences Are All You Need: Local Path Enhanced Document Relation Extraction. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_. Association for Computational Linguistics, Online, 998–1004.  https://doi.org/10.18653/v1/2021.acl-short.126
  * Jia et al. (2019) Robin Jia, Cliff Wong, and Hoifung Poon. 2019.  Document-Level N-ary Relation Extraction with Multiscale Representation Learning. In _Proceedings of the 2019 Conference of the North_. Association for Computational Linguistics.  https://doi.org/10.18653/v1/n19-1370
  * Kim et al. (2020) Kuekyeng Kim, Yuna Hur, Gyeongmin Kim, and Heuiseok Lim. 2020\.  GREG: A Global Level Relation Extraction with Knowledge Graph Embedding.  _Applied Sciences_ 10, 3 (2020).  https://doi.org/10.3390/app10031181
  * Kuang et al. (2022) Hailan Kuang, Haoran Chen, Xiaolin Ma, and Xinhua Liu. 2022\.  A keyword detection and context filtering method for document level relation extraction.  _Applied Sciences_ 12, 3 (2022), 1599.
  * Lai and Lu (2021) Po-Ting Lai and Zhiyong Lu. 2021.  BERT-GT: Cross-sentence n-ary relation extraction with BERT and Graph Transformer.  arXiv:2101.04158 [cs.CL]
  * Li et al. (2020) Bo Li, Wei Ye, Zhonghao Sheng, Rui Xie, Xiangyu Xi, and Shikun Zhang. 2020\.  Graph Enhanced Dual Attention Network for Document-Level Relation Extraction. In _Proceedings of the 28th International Conference on Computational Linguistics_. International Committee on Computational Linguistics, Barcelona, Spain (Online), 1551–1560.  https://doi.org/10.18653/v1/2020.coling-main.136
  * Li et al. (2018) Haodi Li, Ming Yang, Qingcai Chen, Buzhou Tang, Xiaolong Wang, and Jun Yan. 2018\.  Chemical-induced disease extraction via recurrent piecewise convolutional neural networks.  _BMC Medical Informatics and Decision Making_ 18 (07 2018).  https://doi.org/10.1186/s12911-018-0629-3
  * Li et al. (2016) Jiao Li, Yueping Sun, Robin Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn Mattingly, Thomas Wiegers, and Zhiyong lu. 2016\.  BioCreative V CDR task corpus: a resource for chemical disease relation extraction.  _Database_ 2016 (05 2016), baw068.  https://doi.org/10.1093/database/baw068
  * Li et al. (2021) Jingye Li, Kang Xu, Fei Li, Hao Fei, Yafeng Ren, and Donghong Ji. 2021.  MRN: A Locally and Globally Mention-Based Reasoning Network for Document-Level Relation Extraction. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. Association for Computational Linguistics, Online, 1359–1370.  https://doi.org/10.18653/v1/2021.findings-acl.117
  * Li et al. (2022) Rongzhen Li, Jiang Zhong, Zhongxuan Xue, Qizhu Dai, and Xue Li. 2022.  Heterogenous affinity graph inference network for document-level relation extraction.  _Knowledge-Based Systems_ 250 (2022), 109146.  https://doi.org/10.1016/j.knosys.2022.109146
  * Lipscomb (2000) Carolyn E Lipscomb. 2000\.  Medical subject headings (MeSH).  _Bulletin of the Medical Library Association_ 88, 3 (2000), 265\.
  * Liu et al. (2023a) Hongfei Liu, Zhao Kang, Lizong Zhang, Ling Tian, and Fujun Hua. 2023a.  Document-Level Relation Extraction with Cross-sentence Reasoning Graph. In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_. Springer, 316–328.
  * Liu et al. (2023b) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023b.  Pre-Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.  _ACM Comput. Surv._ 55, 9, Article 195 (jan 2023), 35 pages.  https://doi.org/10.1145/3560815
  * Luan et al. (2019) Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019.  A general framework for information extraction using dynamic span graphs. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 3036–3046.  https://doi.org/10.18653/v1/N19-1308
  * Luo et al. (2022) Ling Luo, Po-Ting Lai, Chih-Hsuan Wei, Cecilia N Arighi, and Zhiyong Lu. 2022.  BioRED: a rich biomedical relation extraction dataset.  _Briefings in Bioinformatics_ 23, 5 (jul 2022).  https://doi.org/10.1093/bib/bbac282
  * Ma et al. (2023) Youmi Ma, An Wang, and Naoaki Okazaki. 2023.  DREEAM: Guiding Attention with Evidence for Improving Document-Level Relation Extraction. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_. Association for Computational Linguistics, Dubrovnik, Croatia, 1971–1983.  https://doi.org/10.18653/v1/2023.eacl-main.145
  * Mandya et al. (2018a) Angrosh Mandya, Danushka Bollegala, Frans Coenen, and Katie Atkinson. 2018a.  Combining Long Short Term Memory and Convolutional Neural Network for Cross-Sentence n-ary Relation Extraction.  arXiv:1811.00845 [cs.IR]
  * Mandya et al. (2018b) Angrosh Mandya, Danushka Bollegala, Frans Coenen, and Katie Atkinson. 2018b.  A dataset for inter-sentence relation extraction using distant supervision. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_. European Language Resources Association (ELRA), 1559–1565.
  * Marcheggiani and Titov (2017) Diego Marcheggiani and Ivan Titov. 2017.  Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Copenhagen, Denmark, 1506–1515.  https://doi.org/10.18653/v1/D17-1159
  * Minh Tran et al. (2020) Hieu Minh Tran, Minh Trung Nguyen, and Thien Huu Nguyen. 2020\.  The Dots Have Their Values: Exploiting the Node-Edge Connections in Graph-based Neural Models for Document-level Relation Extraction. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 4561–4567.  https://doi.org/10.18653/v1/2020.findings-emnlp.409
  * Mintz et al. (2009) Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009\.  Distant supervision for relation extraction without labeled data. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_. 1003–1011.
  * Nan et al. (2020) Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. 2020\.  Reasoning with Latent Structure Refinement for Document-Level Relation Extraction. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 1546–1557.  https://doi.org/10.18653/v1/2020.acl-main.141
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Page et al. (1997) Larry Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1997\.  _PageRank: Bringing order to the web_.  Technical Report. Stanford Digital Libraries Working Paper.
  * Pawar et al. (2017) Sachin Pawar, Girish K Palshikar, and Pushpak Bhattacharyya. 2017\.  Relation extraction: A survey.  _arXiv preprint arXiv:1712.05191_ (2017).
  * Peng et al. (2017) Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017.  Cross-Sentence N-ary Relation Extraction with Graph LSTMs.  _Transactions of the Association for Computational Linguistics_ 5 (2017), 101–115.  https://doi.org/10.1162/tacl_a_00049
  * Peng et al. (2022) Xingyu Peng, Chong Zhang, and Ke Xu. 2022.  Document-level Relation Extraction via Subgraph Reasoning. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI_. 4331–4337.
  * Prieur et al. (2023a) Maxime Prieur, Cédric Du Mouza, Guillaume Gadek, and Bruno Grilheres. 2023a.  Evaluating and Improving End-to-End Systems for Knowledge Base Population. In _International Conference on Agents and Artificial Intelligence (ICAART)_.
  * Prieur et al. (2023b) Maxime Prieur, Souhir Gahbiche, Guillaume Gadek, Sylvain Gatepaille, Kilian Vasnier, and Valerian Justine. 2023b.  K-pop and fake facts: from texts to smart alerting for maritime security. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)_. 510–517.
  * Pujara and Singh (2018) Jay Pujara and Sameer Singh. 2018.  Mining Knowledge Graphs From Text. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_ (Marina Del Rey, CA, USA) _(WSDM ’18)_. Association for Computing Machinery, New York, NY, USA, 789–790.  https://doi.org/10.1145/3159652.3162011
  * Qi et al. (2022) Ji Qi, Bin Xu, Kaisheng Zeng, Jinxin Liu, Jifan Yu, Qi Gao, Juanzi Li, and Lei Hou. 2022.  ConstGCN: Constrained Transmission-based Graph Convolutional Networks for Document-level Relation Extraction.  _arXiv preprint arXiv:2210.03949_ (2022).
  * Quirk and Poon (2017) Chris Quirk and Hoifung Poon. 2017.  Distant Supervision for Relation Extraction beyond the Sentence Boundary. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_. Association for Computational Linguistics, Valencia, Spain, 1171–1182.  https://aclanthology.org/E17-1110
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015\.  U-Net: Convolutional Networks for Biomedical Image Segmentation. In _Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015_ , Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (Eds.). Springer International Publishing, Cham, 234–241.
  * Ru et al. (2021) Dongyu Ru, Changzhi Sun, Jiangtao Feng, Lin Qiu, Hao Zhou, Weinan Zhang, Yong Yu, and Lei Li. 2021\.  Learning Logic Rules for Document-Level Relation Extraction. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 1239–1250.  https://doi.org/10.18653/v1/2021.emnlp-main.95
  * Sahu et al. (2019) Sunil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019.  Inter-sentence Relation Extraction with Document-level Graph Convolutional Neural Network. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 4309–4316.  https://doi.org/10.18653/v1/P19-1423
  * Schlichtkrull et al. (2018) Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018.  Modeling Relational Data with Graph Convolutional Networks. In _The Semantic Web_ , Aldo Gangemi, Roberto Navigli, Maria-Esther Vidal, Pascal Hitzler, Raphaël Troncy, Laura Hollink, Anna Tordai, and Mehwish Alam (Eds.). Springer International Publishing, Cham, 593–607.
  * Shi et al. (2019) Yong Shi, Yang Xiao, and Lingfeng Niu. 2019.  A brief survey of relation extraction based on distant supervision. In _Computational Science–ICCS 2019: 19th International Conference, Faro, Portugal, June 12–14, 2019, Proceedings, Part III 19_. Springer, 293–303.
  * Shi et al. (2021) Yong Shi, Yang Xiao, Pei Quan, MingLong Lei, and Lingfeng Niu. 2021.  Document-level relation extraction via graph transformer networks and temporal convolutional networks.  _Pattern Recognition Letters_ 149 (2021), 150–156.  https://doi.org/10.1016/j.patrec.2021.06.012
  * Socher et al. (2012) Richard Socher, Brody Huval, Christopher D Manning, and Andrew Y Ng. 2012.  Semantic compositionality through recursive matrix-vector spaces. In _Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning_. 1201–1211.
  * Song et al. (2018) Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018\.  N-ary Relation Extraction using Graph-State LSTM. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Brussels, Belgium, 2226–2235.  https://doi.org/10.18653/v1/D18-1246
  * Soto et al. (2019) Axel J Soto, Piotr Przybyła, and Sophia Ananiadou. 2019\.  Thalia: semantic search engine for biomedical abstracts.  _Bioinformatics_ 35, 10 (2019), 1799–1801.
  * Su et al. (2021) Junhao Su, Ye Wu, Hing-Fung Ting, Tak-Wah Lam, and Ruibang Luo. 2021.  RENET2: high-performance full-text gene–disease relation extraction with iterative training data expansion.  _NAR Genomics and Bioinformatics_ 3, 3 (2021), lqab062.
  * Sun et al. (2022a) Qi Sun, Tiancheng Xu, Kun Zhang, Kun Huang, Laishui Lv, Xun Li, Ting Zhang, and Doris Dore-Natteh. 2022a.  Dual-Channel and Hierarchical Graph Convolutional Networks for document-level relation extraction.  _Expert Systems with Applications_ 205 (2022), 117678.
  * Sun et al. (2022b) Qi Sun, Kun Zhang, Kun Huang, Xun Li, Ting Zhang, and Tiancheng Xu. 2022b.  Enhanced graph convolutional network based on node importance for document-level relation extraction.  _Neural Computing and Applications_ 34, 18 (2022), 15429–15439.
  * Sun et al. (2023) Qi Sun, Kun Zhang, Kun Huang, Tiancheng Xu, Xun Li, and Yaodi Liu. 2023\.  Document-level relation extraction with two-stage dynamic graph attention networks.  _Knowledge-Based Systems_ 267 (2023), 110428.
  * Swainston et al. (2017) Neil Swainston, Riza Batista-Navarro, Pablo Carbonell, Paul D Dobson, Mark Dunstan, Adrian J Jervis, Maria Vinaixa, Alan R Williams, Sophia Ananiadou, Jean-Loup Faulon, et al. 2017\.  biochem4j: Integrated and extensible biochemical knowledge through graph databases.  _PloS one_ 12, 7 (2017), e0179130.
  * Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015\.  Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Beijing, China, 1556–1566.  https://doi.org/10.3115/v1/P15-1150
  * Tan et al. (2022a) Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022a.  Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation. In _Findings of the Association for Computational Linguistics: ACL 2022_. Association for Computational Linguistics, Dublin, Ireland, 1672–1681.  https://doi.org/10.18653/v1/2022.findings-acl.132
  * Tan et al. (2022b) Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022b.  Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 8472–8487.  https://aclanthology.org/2022.emnlp-main.580
  * Tang et al. (2020) Hengzhu Tang, Yanan Cao, Zhenyu Zhang, Jiangxia Cao, Fang Fang, Shi Wang, and Pengfei Yin. 2020.  HIN: Hierarchical Inference Network for Document-Level Relation Extraction.  arXiv:2003.12754 [cs.CL]
  * Thisanke et al. (2023) Hans Thisanke, Chamli Deshan, Kavindu Chamith, Sachith Seneviratne, Rajith Vidanaarachchi, and Damayanthi Herath. 2023.  Semantic segmentation using Vision Transformers: A survey.  _Engineering Applications of Artificial Intelligence_ 126 (2023), 106669\.  https://doi.org/10.1016/j.engappai.2023.106669
  * Trisedya et al. (2019) Bayu Distiawan Trisedya, Gerhard Weikum, Jianzhong Qi, and Rui Zhang. 2019.  Neural Relation Extraction for Knowledge Base Enrichment. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 229–240.  https://doi.org/10.18653/v1/P19-1023
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _Advances in Neural Information Processing Systems_ , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph Attention Networks.  _International Conference on Learning Representations_ (2018).  https://openreview.net/forum?id=rJXMpikCZ
  * Verga et al. (2018) Patrick Verga, Emma Strubell, and Andrew McCallum. 2018\.  Simultaneously Self-Attending to All Mentions for Full-Abstract Biological Relation Extraction. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. Association for Computational Linguistics, New Orleans, Louisiana, 872–884.  https://doi.org/10.18653/v1/N18-1080
  * Wadden et al. (2019) David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019.  Entity, Relation, and Event Extraction with Contextualized Span Representations. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 5784–5789.  https://doi.org/10.18653/v1/D19-1585
  * Wang et al. (2020) Difeng Wang, Wei Hu, Ermei Cao, and Weijian Sun. 2020\.  Global-to-Local Neural Networks for Document-Level Relation Extraction. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 3711–3721.  https://doi.org/10.18653/v1/2020.emnlp-main.303
  * Wang et al. (2019) Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, and William Wang. 2019.  Fine-tune Bert for DocRED with Two-step Process.  arXiv:1909.11898 [cs.CL]
  * Wang et al. (2021) Hailin Wang, Ke Qin, Guoming Lu, Jin Yin, Rufai Yusuf Zakari, and Jim Wilson Owusu. 2021.  Document-level relation extraction using evidence reasoning on RST- GRAPH.  _Knowledge-Based Systems_ 228 (2021), 107274.  https://doi.org/10.1016/j.knosys.2021.107274
  * Wang et al. (2023) Nianbin Wang, Tiantian Chen, Chaoqi Ren, and Hongbin Wang. 2023\.  Document-level relation extraction with multi-layer heterogeneous graph attention network.  _Engineering Applications of Artificial Intelligence_ 123 (2023), 106212\.
  * Wei and Li (2022) Ying Wei and Qi Li. 2022\.  SagDRE: Sequence-Aware Graph-Based Document-Level Relation Extraction with Adaptive Margin Loss. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ (Washington DC, USA) _(KDD ’22)_. Association for Computing Machinery, New York, NY, USA, 2000–2008.  https://doi.org/10.1145/3534678.3539304
  * Wu et al. (2019) Ye Wu, Ruibang Luo, Henry C. M. Leung, Hing-Fung Ting, and Tak-Wah Lam. 2019.  RENET: A Deep Learning Approach for Extracting Gene-Disease Associations from Literature. In _Research in Computational Molecular Biology_ , Lenore J. Cowen (Ed.). Springer International Publishing, Cham, 272–284.
  * Xiao et al. (2020) Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, and Leyu Lin. 2020\.  Denoising Relation Extraction from Document-level Distant Supervision. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 3683–3688.  https://doi.org/10.18653/v1/2020.emnlp-main.300
  * Xiao et al. (2022) Yuxin Xiao, Zecheng Zhang, Yuning Mao, Carl Yang, and Jiawei Han. 2022.  SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Seattle, United States, 2395–2409.  https://doi.org/10.18653/v1/2022.naacl-main.171
  * Xie et al. (2022) Yiqing Xie, Jiaming Shen, Sha Li, Yuning Mao, and Jiawei Han. 2022.  Eider: Empowering Document-level Relation Extraction with Efficient Evidence Extraction and Inference-stage Fusion. In _Findings of the Association for Computational Linguistics: ACL 2022_. Association for Computational Linguistics, Dublin, Ireland, 257–268.  https://doi.org/10.18653/v1/2022.findings-acl.23
  * Xu et al. (2021) Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. 2021.  Entity Structure Within and Throughout: Modeling Mention Dependencies for Document-Level Relation Extraction.  arXiv:2102.10249 [cs.CL]
  * Xu et al. (2022) Wang Xu, Kehai Chen, Lili Mou, and Tiejun Zhao. 2022\.  Document-Level Relation Extraction with Sentences Importance Estimation and Focusing. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Seattle, United States, 2920–2929.  https://doi.org/10.18653/v1/2022.naacl-main.212
  * Xu et al. (2020) Wang Xu, Kehai Chen, and Tiejun Zhao. 2020.  Document-Level Relation Extraction with Reconstruction.  arXiv:2012.11384 [cs.CL]
  * Xue et al. (2022) Zhongxuan Xue, Rongzhen Li, Qizhu Dai, and Zhong Jiang. 2022\.  CorefDRE: Document-level Relation Extraction with coreference resolution.  _arXiv preprint arXiv:2202.10744_ (2022).
  * Yao et al. (2019) Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019.  DocRED: A Large-Scale Document-Level Relation Extraction Dataset. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 764–777.  https://doi.org/10.18653/v1/P19-1074
  * Ye et al. (2020) Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020.  Coreferential Reasoning Learning for Language Representation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 7170–7186.  https://doi.org/10.18653/v1/2020.emnlp-main.582
  * Yu et al. (2022) Jiaxin Yu, Deqing Yang, and Shuyu Tian. 2022.  Relation-Specific Attentions over Entity Mentions for Enhanced Document-Level Relation Extraction. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Seattle, United States, 1523–1529.  https://doi.org/10.18653/v1/2022.naacl-main.109
  * Zaporojets et al. (2021) Klim Zaporojets, Johannes Deleu, Chris Develder, and Thomas Demeester. 2021.  DWIE: An entity-centric dataset for multi-task document-level information extraction.  _Information Processing & Management_ 58, 4 (2021), 102563\.  https://doi.org/10.1016/j.ipm.2021.102563
  * Zeng et al. (2021) Shuang Zeng, Yuting Wu, and Baobao Chang. 2021.  SIRE: Separate Intra- and Inter-sentential Reasoning for Document-level Relation Extraction. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. Association for Computational Linguistics, Online, 524–534.  https://doi.org/10.18653/v1/2021.findings-acl.47