# A Survey of Document-Level Information Extraction

First Author
Zoe / Address line 1
Affiliation / Address line 2
Affiliation / Address line 3
<EMAIL_ADDRESS>
&Second Author
Affiliation / Address line 1
Affiliation / Address line 2
Affiliation / Address line 3
email@domain
Hanwen Zheng Sijia Wang Lifu Huang
Virginia Tech
<EMAIL_ADDRESS>

###### Abstract

Document-level information extraction (IE) is a crucial task in natural
language processing (NLP). This paper conducts a systematic review of recent
document-level IE literature. In addition, we conduct a thorough error
analysis with current state-of-the-art algorithms and identify their
limitations as well as the remaining challenges for the task of document-level
IE. According to our findings, labeling noises, entity coreference resolution,
and lack of reasoning, severely affect the performance of document-level IE.
The objective of this survey paper is to provide more insights and help NLP
researchers to further enhance document-level IE performance.

##  1 Introduction

Natural language processing (NLP) triggers the present wave of artificial
intelligence (Vaswani et al., 2017; Dosovitskiy et al., 2021; Liu et al.,
2021; Zhang et al., 2021a; Zhang and Eskandarian, 2022). Information
Extraction (IE) plays a vital role in all aspects of NLP by extracting
structured information from unstructured texts Lin et al. (2020); Wang et al.
(2022). Document-level IE has witnessed significant progress, benefiting from
the enormous data resources provided by the Internet and the rapidly growing
computational power resources Yao et al. (2019a); Xu et al. (2021b); Tong et
al. (2022). However, several challenges persist within the realm of document-
level IE research, such as entity coreference resolution, reasoning across
long-span contexts, and a lack of commonsense reasoning. Furthermore, current
document-level IE research predominantly focuses on restricted domains and
languages Zheng et al. (2019a); Yang et al. (2018); Tong et al. (2022); Li et
al. (2021), which results in challenges for model comparisons and
generalization. This limitation poses difficulties in conducting model
comparisons and hampers the generalizability of findings.

To fulfill the aforementioned challenges, this survey reviews recent document-
level relation extraction (doc-RE) and document-level event extraction (doc-
EE) models and datasets to inform and encourage researchers for multilingual
and cross-domain studies. In addition, we conduct a thorough error analysis
among existing models and discuss these errors. Finally, we summarize the
current literature work and propose potential future improvements to document-
level IE research. The contributions of this survey paper include:

  * •

We systematically summarize and categorize the existing datasets and
approaches for Doc-RE and Doc-EE.

  * •

A thorough error analysis is conducted with current state-of-the-art (SOTA)
algorithms.

  * •

To identify the current model challenges and limitations, we analyze and
discuss the errors and construct error statistics.

This survey aims to contribute to the NLP community by providing valuable
insights into document-level IE tasks. Our analysis of errors encountered in
this study will serve as a foundation for future advancements in document-
level IE research, encouraging researchers to innovate and improve upon
existing methodologies. It is our hope that these findings will contribute to
a deeper understanding of document-level IE and stimulate further enhancements
in this field of study.

##  2 Tasks Definition

Figure 1: Examples of doc-EE and doc-RE.

###  2.1 Event Extraction

Event extraction Grishman (1997); Chinchor and Marsh (1998); Ahn (2006) is a
task to identify and classify event triggers and relevant participants from
natural language text. Formally, given a document consisting of a set of
sentences where each sentence consists of a sequence of words, the objective
of this task is to identify and extract the following components from a given
document: Event Mention, which refers to the phrases or sentences denoting an
event; Event Trigger, typically in the form of a verb that signals the
occurrence of an event; Event Type, indicating the predefined type of event
specified by the dataset, such as Conflict-Attack; Argument Mention,
comprising entity mentions that provide additional details on the event, such
as who, what, when, where, and how the event occurred; Argument Role,
representing the role or type of argument associated with the entity; and
finally, Event Record, the entry in an event table, containing several
arguments with argument roles.

###  2.2 Relation Extraction

The task of Relation Extraction involves predicting attributes and
relationships between entities mentioned in a given document Zhou et al.
(2005). Given a document $D$ with a set of sentences, we assume that $D$ also
contains a set of entities $V=\\{e_{i}\\}_{i=1}^{N}$. For each entity $e_{i}$,
it might contain multiple entity mentions $e_{i}=\\{m_{j}\\}_{j=1}^{M}$. The
doc-RE task is to predict the relation types between an entity pair
$(e_{s},e_{o})_{s,o\in\\{1,\cdots,N\\},s\neq o}$, where $s$ stands for the
subject and $o$ stands for the object. It is possible for an entity pair to
have multiple relations that require prediction, thereby rendering the task a
multi-label classification problem.

More specifically, Entity refers to units such as People, Geographic Entity,
Location, Organization, Date, and Number within a text. Entity Mention refers
to a phrase within a text that identifies a specific entity. For instance,
“NYC” and “the big apple” are both entity mentions for “New York City”. Intra-
sentence Relation describes the relationship between entities within a single
sentence, and the features within are often referred to as local features. On
the other hand, Inter-sentence Relation refers to the relationship between
entities across multiple sentences, and the features within are often referred
to as global features.

##  3 Datasets

Existing studies only evaluate their proposed approaches on restricted
targeted domains or languages. As a result, it is challenging to compare the
effectiveness of different methods under a more general scenario. In this
section, we list all doc-EE and doc-RE datasets, to share all possible options
with the research community.

###  3.1 Doc-RE Datasets

Dataset | Annotation | # Rel Types | # Rel Facts | # Train | # Dev | #Test
---|---|---|---|---|---|---
DGM Jia et al. (2019a) | Distant Supervision | 1 | - | 32,040 | - | -
CDR Luan et al. (2018) | Human-annotated | 1 | - | 1,500 | 500 | 500
GDA Wu et al. (2019) | Distant Supervision | 1 | - | 30,192 | 5,839 | 1,000
DocRED (Yao et al., 2019b) | Distant Supervision | 96 | 50,345 | 3,053 | 1,000 | 1,000
Re-DocRED (Tan et al., 2022b) | Combined | 96 | 120,664 | 3,053 | 500 | 500
SciREX (Jain et al., 2020) | Human-annotated | 2 | - | 438 | 131 | 131
HacRED Cheng et al. (2021) | Combined | 26 | 65,225 | 9,231 | 1,500 | 1,500
Table 1:  Statistics of Doc-RE datasets.

For biomedical domain, Drug-gene-mutation (DGM) Jia et al. (2019a) contains
4,606 PubMed articles, which are automatically labeled via distant
supervision. DGM annotations include three entity types: drugs, genes, and
mutations, and three relation types, including drug-gene-mutation, drug-
mutation, and gene-mutation relations. GDA Wu et al. (2019) gene-disease
association corpus contains 30,192 titles and abstracts from PubMed articles
that have been automatically labeled for genes, diseases, and gene-disease
associations via distant supervision. CDR Luan et al. (2018) is manually
annotated for chemicals, diseases, and chemical-induced disease (CID)
relations by domain experts. It contains the titles and abstracts of 1,500
PubMed articles and is split into training, validation, and test sets equally.

Several Doc-RE datasets are constructed for other domains or languages. DocRED
Yao et al. (2019b) is a human-annotated doc-RE dataset, that includes 132,375
entities and 56,354 relational facts annotated on 5,053 Wikipedia documents.
Doc-RED is generated by mapping Wikidata triples, originating from a
comprehensive knowledge base closely intertwined with Wikipedia, onto complete
English Wikipedia documents to get entity annotations. RE-DocRED Tan et al.
(2022b) refines 4,053 documents in the DocRED dataset targeting on resolving
the problem of false negative samples. RE-DocRED increased the relation
triples from 50,503 to 120,664 and decreased the no_relation samples by
$3.1\%$ by adding the missing relation triples back to the original DocRED.
SciREX Jain et al. (2020) is a document-level IE dataset that contains
multiple IE tasks. It mainly focuses on its doc-RE tasks, such as Binary and
N-ary relation classification. It consists of both automatic and human-
annotated articles in the computer science field. HacRED Cheng et al. (2021)
is a Chinese doc-RE dataset collected from CN-DBpedia Xu et al. (2017) that
focuses on hard cases, such as long text and long distance between argument
pairs, containing distractors or multiple homogeneous entity mentions.

###  3.2 Doc-EE Datasets

Doc-EE datasets are mainly collected from the news and financial domain. News
is a large-scale accessible source of events like social emergencies and human
life incidents, thus many datasets are created focusing on news events.
Meanwhile, exploding volumes of digital financial documents, as a byproduct of
continuous economic growth, have been created. Many datasets are created to
help extract valuable structured information to detect financial risks or
profitable opportunities. Statistics of the datasets for Doc-EE are summarized
in Table 2.

Dataset | # Docs | # Events | # Event types | # Roles | # Arguments | Ratio
---|---|---|---|---|---|---
ACE-2005111https://catalog.ldc.upenn.edu/LDC2006T06 | 599 | 4,202 | 33 | 35 | 9,590 | -
MUC-4 muc (1992) | 1,700 | 1,514 | 4 | 5 | 2,641 | 13:2:2
RAMS Ebner et al. (2020) | 9,124 | 8,823 | 139 | 65 | 21,237 | 8:1:1
WikiEvents (Li et al., 2021) | 246 | 3,951 | 50 | 59 | 5,536 | 10:1:1
DocEE Tong et al. (2022) | 27,485 | 27,485 | 59 | 356 | 180,520 | -
ChFinAnn (Zheng et al., 2019b) | 32,040 | 47,824 | 5 | 35 | 289,871 | 8:1:1
DCFEE (Yang et al., 2018) | 2,976 | 3,044 | 4 | 35 | - | 8:1:1
DuEE-Fin (Zheng et al., 2019b) | 11,699 | 15,850 | 13 | 92 | 81,632 | 6:1:3
Table 2: Statistics of Doc-EE datasets.

For the news domain, ACE-2005222https://catalog.ldc.upenn.edu/LDC2006T06 is a
sentence-level event extraction (SEE) Wang et al. (2022, 2023) dataset but has
been frequently used for comparison in doc-EE. Unlike ACE-2005 which contains
5 groups of events covering justice, life, business events, etc, MUC-4 muc
(1992) focuses on one specific event type, attack events. MUC-4 contains 1,700
human-annotated news reports of terrorist attacks in Latin America collected
by Federal Broadcast Information Services. More specifically, MUC-4 includes
six incident types: attack, kidnapping, bombing, arson, robbery, and forced
work stoppage, and four argument roles, including individual perpetrator,
organization perpetrator, physical target, and human target. WikiEvents (Li et
al., 2021) follows the ontology from the KAIROS
project333https://www.ldc.upenn.edu/collaborations/currentprojects for event
annotation, which defines 67 event types in a three-level hierarchy.
Researchers used the BRAT interface for online annotation of event mentions
(triggers and arguments) and event coreference separately. Roles Across
Multiple Sentences (RAMS) Ebner et al. (2020) is a crowd-sourced dataset with
9,124 event annotations on news articles from Reddit following the AIDA
ontology. DocEE is the largest Doc-EE dataset to date. DocEE uses historical
events and timeline events from Wikipedia as the candidate source to define 59
event types and 356 event argument roles. This dataset includes 27,485
document-level events and 180,528 event arguments that are manually labeled.

For the financial domain, DCFEE (Yang et al., 2018) comes from companies’
official finance announcements and focuses on four event types: Equity Freeze,
Equity Pledge, Equity Repurchase, and Equity Overweight. Data labeling was
done through distant supervision. ChFinAnn (Zheng et al., 2019b) contains
official disclosures such as annual reports and earnings estimates, obtained
from the Chinese Financial Announcement (CFA). The dataset has five event
types: Equity Freeze, Equity Repurchase, Equity Underweight, Equity Overweight
and Equity Pledge, with 35 different argument roles in total. In contrast to
Doc-EE with one event in each document, 29.0% of the documents in ChFinAnn
contain multiple events. DuEE-Fin (Zheng et al., 2019b) is the largest human-
labeled Chinese financial dataset. It is collected from real-world Chinese
financial news and annotated with 13 event types. 29.2% of the documents
contain multiple events and 16.8% of events consist of multiple arguments.

##  4 Evaluation Metrics

In document-level information extraction (IE), the primary evaluation metrics
are Precision (P), Recall (R), and Macro-F1 score Kowsari et al. (2019).
Additionally, for doc-RE, Ign F1 is used as an evaluation metric (Yao et al.,
2019b). Ign F1 refers to the F1 score that excludes relational facts shared by
the training and dev/test sets. This metric is important for evaluating the
generalizability of the model, as it disregards triples that are already
present in the annotated training dataset.

Task |  Main Category |  Sub Category |  Approaches
---|---|---|---
Doc-RE |  Multi-granularity-based |  Sentence-level$\to$ Paragraph-level$\to$ Document-level |  Tang et al. (2020)
Mention-level$\to$ Entity-level |  Jia et al. (2019b)
Graph-based |  Heterogeneous graph |  Quirk and Poon (2017), Peng et al. (2017), Song et al. (2018), Guo et al. (2019), Sahu et al. (2019), Christopoulou et al. (2019), Wang et al. (2020), Xu et al. (2021d), Zeng et al. (2020), Li et al. (2020), Zhang et al. (2020), Xu et al. (2023), Xu et al. (2021c)
Homogeneous graph |  Nan et al. (2020)
Sequence-based |  Neural Networks |  Xu et al. (2021a), Zhang et al. (2021b)
Attention\Transformer |  Zhou et al. (2021), Tan et al. (2022a)
Evidence-based |  Path reasoning |  Huang et al. (2021)
Evidence retrieval |  Xie et al. (2022), Xiao et al. (2022)
Doc-EE |  Multi-granularity-based |  Sentence-level$\to$ Paragraph-level$\to$ Document-level |  Yang et al. (2018), Huang and Jia (2021)
Graph-based |  Heterogeneous graph  |  Zheng et al. (2019b), Xu et al. (2021d), Zhu et al. (2022), Xu et al. (2022)
Sequence-based |  Neural Networks |  Huang and Peng (2021)
Attention\Transformer |  Yang et al. (2021), Liang et al. (2022)
Generation-based |  - |  Li et al. (2021), Zeng et al. (2022)
Memory-based |  - |  Du et al. (2022), Cui et al. (2022)

Table 3: Typology of Doc-IE methods.

##  5 Methods

The fundamental challenge in doc-RE and doc-EE is to express document content
in a concise and effective way such that key information is maintained.
Previous approaches usually resort to hierarchical, graph-based, or sequential
structures. More recently, due to the emergence of powerful generative pre-
trained language models (PLMs), generative models have also been introduced to
address doc-IE tasks. A typology of existing doc-RE and doc-EE approaches
categorized by model design is shown in Table 3.

###  5.1 Doc-RE Approaches

#### Multi-granularity-based Models

The multi-granularity-based approach aims to emphasize the use of information
from different granularities and the aggregation of global information. The
standard procedure involves concatenating features from each level to complete
the IE tasks. Jia et al. (2019b) approaches document-level N-ary relation
extraction using a multiscale representation learning method. This approach
aggregates the representations of mentions and ensembles multiple sub-
relations. The HIN (Hierarchical Inference Network) Tang et al. (2020) uses
Bi-LSTMs at the token, sentence, and document levels to extract features as
sequences and weighs the overall features with the attention mechanism to
obtain both local and global information. Multi-granularity-based designs
employ two strategies: either they address intermediate tasks using various
models, or they utilize the same model in a hierarchically ordered manner to
independently tackle each subtask of information extraction, such as from
sentence level to document level.

#### Graph-based Models

Graph-based models generally construct a graph with words, mentions, entities,
or sentences as nodes and define different types of edges across the entire
document, further predicting the relations by reasoning on the graph. The
first work done on doc-RE using a graph-based method is DISCREX Quirk and Poon
(2017), where a document graph is constructed with word nodes and edges
representing intra- and inter-sentential relations including dependency,
adjacency, and discourse relations. Peng et al. (2017) contributes a Graph-
LSTMs model with a bidirectional LSTM consisting of two directed acyclic
graphs (DAG), and edges representing relations between nodes. Song et al.
(2018) further compares bidirectional graph LSTM with bidirectional DAG LSTM,
finding that the former, which doesn’t alter the input graph structure,
exhibits superior performance. While such dependency graphs have rich
structural information, the pruning strategy does not necessarily keep the
relevant information. AGGCNs Guo et al. (2019) proposes an end-to-end neural
network that encodes the entire graph using multi-head self-attention to learn
edge weights based on paired relations and using densely connected layers to
glean global information. Sahu et al. (2019) designates words as individual
nodes and establishes five types of edges to represent inter-and intra-
sentence dependency. The model then uses an edge-oriented GCNN to retain
aggregated node representation.

EoG Christopoulou et al. (2019) is a pioneering graph-based model. It uses
entities as nodes and forms unique edge representations through the paths
between nodes to better capture the paired relations. To predict relations
between entity pairs, EoG makes iterative inferences on the path between the
entities and aggregates every edge to a direct entity-entity (EE) edge. Many
papers adapted from EoG can be divided into two main categories: homogeneous
and heterogeneous graphs. LSR Nan et al. (2020) uses graph structure as a
latent variable to form a homogeneous graph. Unlike EoG which uses a human-
constructed graph, LSR learns structured attention to refine the graph
dynamically and constructs latent structures based on the previous refinement.
For heterogeneous graphs, different types of edges are defined, representing
unique features, functions, and even dual graphs. GLRE Wang et al. (2020)
utilizes a multi-layer R-GCN to learn entity global representations which are
used as queries in the multi-headed self-attention layer to learn entity local
representations while using sentence-level information as the keys. HeterGSAN
Xu et al. (2021d) is a heterogeneous graph based on EoG that uses a GAT to
encode the graph relying more on related entity pairs’ attention.

Dual graphs are normally used to capture hierarchical information. GAIN Zeng
et al. (2020) utilized a heterogeneous mention-level graph to model
interactions between the document and all mentions. GEDA Li et al. (2020)
optimized entity representation with two attention layers and a heterogeneous
GCN layer. DHG Zhang et al. (2020) contains two heterogeneous graphs: a
structure modeling graph using words and sentences as nodes to better capture
document structure information and a relation reasoning graph using mentions
and entities as nodes to perform multi-hop relation reasoning. POR Xu et al.
(2023) is a path-retrieving method between pair entities based on the BFS
algorithm, which extracts path features through an LSTM and combines them
using the attention mechanism. DRN Xu et al. (2021c) passes encoded sentence
and entity as a heterogeneous graph to a multi-layer GCN and meanwhile, uses
the self-attention mechanism to learn a more contextual document-level
representation.

#### Sequence-based Models

Sequence-based models mostly rely on NN-based or Transformer-based
architectures, which can model complex interactions among entities by
implicitly capturing long-distance dependencies. SSAN Xu et al. (2021a)
integrates structural dependencies within and throughout the encoding stage of
the network, not only enabling simultaneous context reasoning and structure
reasoning but also efficiently modeling these dependencies in all network
layers. ATLOP Zhou et al. (2021) simply applies BERT’s own attention weights
for Localized Context Pooling as well as a dynamic adaptive thresholding
strategy, to ensure that each entity maintains the same representation and
balances the logits of positive and negative labels. DocuNet Zhang et al.
(2021b) divides model construction into three parts leveraging a u-shaped
semantic segmentation network to refine entity feature extraction. KD Tan et
al. (2022a) calculates self-attention in the vertical and horizontal
directions of an $n\\!\times\\!n$ two-hop attention entity pair table using
axial attention. The logits of paired entity relations are ranked with the
logits of the threshold classes individually instead of ranking all positive
logits together. Sequence-based approaches focus on capturing contexts and
entity information via careful designs, either an adequate neural network
structure or a novel loss function.

#### Path(Evidence)-based Models

Path-based models construct evidence paths and make relational decisions by
reasoning on crucial information between entity pairs or sentences, instead of
extracting features from the complete document. THREE Huang et al. (2021)
presents three kinds of paths to find the supporting sentences: consecutive
paths, multi-hop paths, and default paths for entity pairs. EIDER Xie et al.
(2022) defines “evidence sentences”, as a minimal number of sentences needed
to predict the relations between certain pairs of entities in a document. SAIS
Xiao et al. (2022) utilizes two intermediary phases to obtain evidence
information: pooled evidence retrieval, which distinguishes entity pairs with
and without supporting sentences, and fine-grained evidence retrieval, which
produces more interpretable evidence specific to each relation of an entity
pair. Those papers typically utilize supporting sentences from the DocRED
dataset. When humans perform relation extraction on the long span of texts, we
read through the whole document and evaluate sentences that are important for
the task. The path-based approach is consistent with human perception and
intuition, which has shown extraordinary performance.

###  5.2 Doc-EE Approaches

#### Multi-granularity-based Models

DCFEE Yang et al. (2018) first designs a SEE component to obtain the event
arguments and event trigger and splices them together to get the input for the
second component-DEE. The DEE uses a convolutional neural network to
concatenate the output of SEE and the vector representation of the current
sentence. SCDEE Huang and Jia (2021) uses Graph Attention Networks (GAT) to
transform vertex features, which are used to detect sentence communities and
then obtain event types at the sentence level.

#### Graph-based Models

Doc2EDAG Zheng et al. (2019b) first identifies all the entities in a document
and uses transformer fusing information at the document level. When an event
type is triggered, the model starts to generate an entity-based directed
acyclic graph (EDAG) and treats the Doc-EE task as an event table-filling
task. Following the order of roles in an event type, EDAG decides which entity
node to be expanded and considers a path-expanding sub-task until the EDAG is
fully recovered. GIT Xu et al. (2021d) designs a heterogeneous graph with four
types of edges between sentences and mentions. Based on detected event types,
a tracker is designed to extract corresponding arguments by expanding a
constrained event type tree while tracking and storing records in global
memory. PTPCG Zhu et al. (2022) calculates the semantic similarity between
entities to construct a pruned complete graph after event and argument
detection. Pruning is done by deciding whether entity pairs retain an edge
based on heuristics. TSAR Xu et al. (2022) leverages an AMR-guided interaction
module to generate both global and local contextualized representations. A
gate function is designed to decide the portion of global and local
representation, to predict the argument roles for potential spans.

#### Sequence-based Models

DE-PPN Yang et al. (2021) is an encoder-decoder doc-EE model which utilizes
two transformers to identify sentence-level elements as the document encoder
and a multi-granularity decoder to decode event, role, and event-role in
parallel. ReDEE Liang et al. (2022) is the first to use entity relation
information for doc-EE tasks, which utilizes SSAN to extract relation triples
and transfer them with entity and sentence dependency. DEED Huang and Peng
(2021) is an end-to-end model that utilizes Deep Value Networks (DVN), a
structured prediction algorithm that effectively bridges the disparity between
ground truth and prediction. This model directly incorporates event trigger
prediction into DVN, thereby efficiently capturing cross-event dependencies
for document-level event extraction.

#### Generative Models

The generative models are commonly found in doc-EE and joint extraction. Bart-
Gen Li et al. (2021) takes the document and event templates as input, and uses
an encoder-decoder model to generate arguments to fill in the blank in the
templates based on the previous word in the sentence. EA2E Zeng et al. (2022)
aims to achieve event-aware argument extraction by labeling arguments from
nearby events in the document to enhance the context.

#### Memory-based Models

Du et al. (2022) introduces a memory-enhanced neural generation-based
framework based on a sequence-to-sequence PLM. The memory stores gold-standard
events and previously generated events of the same document; and the decoder
retrieves event knowledge and decodes arguments dynamically based on the event
dependency constraints. HRE Cui et al. (2022) emulates the human reading
process by conducting a two-stage analysis - rough reading and elaborate
reading. The initial rough reading detects the event type and saves it as
memory tensors. Upon detection, elaborate reading extracts the complete event
record with arguments and stores them in memory while updating with previous
event type and argument memory tensors.

##  6 Discussion

We concluded seven major types of errors in three existing doc-RE works based
on the DocRED and Re-DocRED datasets, as well as in four doc-EE works based on
the WikiEvents and ChFinAnn datasets. Examples and distributions of each type
are shown in Table 4, 5, and Figure 2, 3, 4.

Figure 2: Doc-RE error distribution in DocRED and Re-DocRED

#### Entity coreference resolution

Document-level texts contain a large number of recognized entities along with
coreferential words such as them, he, which, etc. Entity coreference
resolution errors happen when the model fails to resolve all mentions in a
document that refer to the same entity.

#### Reasoning error

This type of error mainly relates to multi-hop logical reasoning. Document-
level texts contain considerable amounts of information, so models may fail to
give correct logical inferences based on the given information. Inferring from
multi-hop information requires a model to have a high level of natural
language understanding ability.

#### Long-span

Document contains multiple sentences in a long span. This error happens when
the model fails to capture the full context of a document or uses global
information for inference.

#### Commonsense knowledge

The error occurs when models fail to correctly extract relations or events or
assume the wrong semantics due to a lack of commonsense and background
knowledge, which humans are able to learn or understand instinctively. Many
datasets are specific to some domains, and in the absence of relevant
background and domain-specific knowledge models may inaccurately reason or
misinterpret information.

#### Relation transitivity error

Documents tend to have many entities appearing in the same sentence or across
sentences. Relation transitivity errors occur when a model fails to correctly
infer a relation between two entities based on their individual relations with
a third entity. Additionally, not all relations are transitive, thus the model
should correctly recognize when transitivity applies.

#### Over prediction error

This error type refers to the spurious error (as we presented in Table 4)
where there is no ground truth relation between two entities but the model
predicts a relation, and can be caused by a number of reasons. For instance,
when using large pre-trained language models to encode the documents, learned
prior can cause models to make overconfident predictions.

Figure 3: Doc-EE error distribution in ChFinAnn  Figure 4: Doc-EE error
distribution in WikiEvents

Error Type |  Text |  GT |  Prediction
---|---|---|---
Spurious error | Over prediction |  The Link River <LOC> is a short river connecting Upper Klamath Lake <LOC> to Lake Ewauna <LOC> in the city of Klamath Falls <LOC> in the U.S.<LOC> state of Oregon <LOC>. |  N/A |  Lake Ewauna <LOC>, Oregon <LOC> : located in the administrative territorial entity
Spurious error | Learned prior |  Ngoako Ramatlhodi <PER>, a senior member of the African National Congress <ORG> , was South Africa <LOC> ’s Minister |  N/A |  African National Congress <ORG>, South Africa <LOC> : country
Relation transitivity |  At the 2007 <TIME> European Indoor Athletics Championships <MISC> he won a silver medal in the 4 x 400 metres <NUM> relay , with teammates Ivan Buzolin <PER> , Maksim Dyldin <PER> and Artem Sergeyenkov <PER> |  Artem Sergeyenkov <PER>, European Indoor Athletics Championships <MISC> : participant of |  N/A

Table 4:  Corner case errors of RE models

Error Type |  Text |  GT |  Pred
---|---|---|---
ECR |  The game retains some common elements from previous Zelda<MISC> installments, such as the presence of Gorons<PER>, while introducing Kinstones <PER> and other new gameplay features. |  The Legend of Zelda <MISC>, Gorons <PER> : characters |  N/A
Multi-hop reasoning |  Parvathy<PER> married film actor Jayaram <PER> who was her co-star in many films on… She has two <NUM> children, Kalidas Jayaram<PER> and Malavika Jayaram<PER>. |  Jayaram<PER>, Kalidas Jayaram <PER>: child |  N/A
Commonsense |  Olympic Gold <MISC> is the official video game of the XXV Olympic Summer Games <MISC>, hosted by Barcelona <LOC>, Spain <LOC> in 1992 <TIME>. |  XXV Olympic Summer Games <MISC> , Spain <LOC> : country |  N/A

Table 5:  Common errors of RE models

In addition to shared error types with Doc-RE, we observe two more types of
errors based on the WikiEvents and ChFinAnn datasets.

#### Multi-events error

In Doc-EE tasks, documents contain multiple events that overlap or occur
simultaneously, which requires the model to have sufficient training or
advanced techniques to learn the inherent complexity of multi-event documents.
In an event-trigger-annotated dataset such as WikiEvents, the model can fail
at assigning arguments to the correct events or matching roles to arguments.
In a trigger-not-annotated dataset like ChFinAnn, event detection errors may
occur when models try to identify and differentiate distinct events within the
document due to the complex contextual structure of each event.

#### Other errors

Models face other error types which are mainly associated with previous tasks
like entity recognition or caused by the different linguistic features and
complexities of datasets. For example, nominal mention recognition and
argument span mismatch errors are common in many works, particularly in
generative methods.

#### Noisy data

This issue comprises natural language noises and labeling noises. Real-world
documents contain noisy, unstructured, or poorly formatted content, causing
difficulties in identifying entities and extracting relations. Natural
language can be ambiguous or vague, leading to uncertainty in model inference.
To overcome the limitations of the cost of creating annotated datasets,
researchers commonly apply automatic labeling strategies like distant
supervision to generate large-scale training data. However, this leads to
several minor problems due to noise and bias: nested entities (i.e., some
entities can be embedded within other entities), false negative labels (i.e.,
entity pairs not known to be related but getting labeled as such in the
dataset), and missing ground truth labels.

Note that Doc-EE errors vary between ChFinAnn and WikiEvents. There could be a
number of factors behind the different Doc-EE error distribution between
ChFinAnn and WikiEvents. One crucial factor is the diversity in underlying
statistics between datasets due to their distinct domains and languages.
Compared to the news dataset WikiEvents, the Chinese financial dataset
ChFinAnn requires less commonsense comprehension. Each dataset contains unique
linguistic features and complexities. WikiEvents has annotated trigger words,
and arguments tend to be near the trigger words, whereas ChFinAnn can have
events spread across the entire document and is more likely to interfere with
other events. Therefore, long-span and multi-events are major error types in
ChFinAnn. Moreover, various model designs and approaches usually aim to
address specific challenges and optimize performance on the respective
dataset.

##  7 Remaining Challenges

Current difficulties can be broadly categorized into three areas. First, a lot
of information is spread out over several sentences. Second, there might be
several mentions pointing to the same entity throughout the entire document.
Finally, some relations must be deduced from several sentences in order to be
discovered. The first two issues have been addressed by existing approaches
using attention mechanisms and graph construction, though multiple-step
reasoning techniques are less widely used. Progressively, more methods try to
use evidence sentences or evidence paths to infer complicated relations.
Nevertheless, models continue to struggle with capturing common sense and
knowledge-based reasoning as it is difficult to identify a pattern that is
extremely similar in the training set or even during pre-training.
Additionally, creating annotated datasets for this task is time-consuming and
expensive, which limits the amount of data available for training and
evaluation. Domain-specific datasets differ from main general datasets, but
are necessary for identifying relations that are specific to certain domains,
understanding domain-specific terminology, and handling the high variability
of language used in different domains. There are several promising future
directions. First, it is beneficial to incorporate entity coreference systems
into doc-IE models, which we believe will play an important role in resolving
ECR and multi-hop reasoning errors. Second, more investigations are needed to
design a model with multi-hop reasoning capability. Finally, doc-EE and doc-RE
can be supplementary tasks to each other. The information produced by these
two tasks can provide a more complete picture of the information given in the
document.

## Limitations

A thorough error analysis is conducted with current state-of-the-art
algorithms and limitations in existing approaches as well as the remaining
challenges are identified for the task of document-level IE. However, how a
system can effectively address the challenges takes appropriate action, while
we exclusively analyze existing studies mainly focused on news, financial,
biomedical, and Wikipedia datasets in English and Chinese languages, we
acknowledge that the challenges and conclusions drawn may not be generalizable
to other domains, languages, or new datasets.

## References

  * muc (1992) 1992\.  _Fourth Message Uunderstanding Conference (MUC-4): Proceedings of a Conference Held in McLean, Virginia, June 16-18, 1992_.
  * Ahn (2006) David Ahn. 2006.  The stages of event extraction.  In _Proceedings of the Workshop on Annotating and Reasoning about Time and Events_ , pages 1–8.
  * Cheng et al. (2021) Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021.  Hacred: A large-scale relation extraction dataset toward hard cases in practical applications.  In _FINDINGS_.
  * Chinchor and Marsh (1998) Nancy Chinchor and Elaine Marsh. 1998.  Muc-7 information extraction task definition.  In _Proceeding of the seventh message understanding conference (MUC-7), Appendices_ , pages 359–367.
  * Christopoulou et al. (2019) Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019.  Connecting the Dots: Document-level Neural Relation Extraction with Edge-oriented Graphs.  ArXiv:1909.00228 [cs].
  * Cui et al. (2022) Shiyao Cui, Xin Cong, Bowen Yu, Tingwen Liu, Yucheng Wang, and Jinqiao Shi. 2022\.  Document-Level Event Extraction via Human-Like Reading Process.  ArXiv:2202.03092 [cs].
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_.
  * Du and Cardie (2020) Xinya Du and Claire Cardie. 2020.  Event Extraction by Answering (Almost) Natural Questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 671–683, Online. Association for Computational Linguistics.
  * Du et al. (2022) Xinya Du, Sha Li, and Heng Ji. 2022.  Dynamic Global Memory for Document-level Argument Extraction.  ArXiv:2209.08679 [cs].
  * Ebner et al. (2020) Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020\.  Multi-sentence argument linking.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Grishman (1997) Ralph Grishman. 1997.  Information extraction: Techniques and challenges.  In _International summer school on information extraction_ , pages 10–27. Springer.
  * Guo et al. (2019) Zhijiang Guo, Yan Zhang, and Wei Lu. 2019.  Attention Guided Graph Convolutional Networks for Relation Extraction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 241–251, Florence, Italy. Association for Computational Linguistics.
  * Huang and Peng (2021) Kung-Hsiang Huang and Nanyun Peng. 2021.  Document-level Event Extraction with Efficient End-to-end Learning of Cross-event Dependencies.  In _Proceedings of the Third Workshop on Narrative Understanding_ , pages 36–47, Virtual. Association for Computational Linguistics.
  * Huang et al. (2021) Quzhe Huang, Shengqi Zhu, Yansong Feng, Yuan Ye, Yuxuan Lai, and Dongyan Zhao. 2021\.  Three Sentences Are All You Need: Local Path Enhanced Document Relation Extraction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 998–1004, Online. Association for Computational Linguistics.
  * Huang and Jia (2021) Yusheng Huang and Weijia Jia. 2021.  Exploring Sentence Community for Document-Level Event Extraction.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 340–351, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Jain et al. (2020) Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. 2020.  SciREX: A challenge dataset for document-level information extraction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7506–7516, Online. Association for Computational Linguistics.
  * Jia et al. (2019a) Robin Jia, Cliff Wong, and Hoifung Poon. 2019a.  Document-level n-ary relation extraction with multiscale representation learning.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3693–3704, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Jia et al. (2019b) Robin Jia, Cliff Wong, and Hoifung Poon. 2019b.  Document-Level N-ary Relation Extraction with Multiscale Representation Learning.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3693–3704, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Kowsari et al. (2019) Kamran Kowsari, Kiana Jafari Meimandi, Mojtaba Heidarysafa, Sanjana Mendu, Laura Barnes, and Donald Brown. 2019.  Text classification algorithms: A survey.  _Information_ , 10(4):150.
  * Li et al. (2020) Bo Li, Wei Ye, Zhonghao Sheng, Rui Xie, Xiangyu Xi, and Shikun Zhang. 2020.  Graph Enhanced Dual Attention Network for Document-Level Relation Extraction.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1551–1560, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Li et al. (2021) Sha Li, Heng Ji, and Jiawei Han. 2021.  Document-Level Event Argument Extraction by Conditional Generation.  ArXiv:2104.05919 [cs].
  * Liang et al. (2022) Yuan Liang, Zhuoxuan Jiang, Di Yin, and Bo Ren. 2022.  RAAT: Relation-Augmented Attention Transformer for Relation Modeling in Document-Level Event Extraction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4985–4997, Seattle, United States. Association for Computational Linguistics.
  * Lin et al. (2020) Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020.  A joint neural model for information extraction with global features.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7999–8009, Online. Association for Computational Linguistics.
  * Liu et al. (2021) Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. 2021.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9992–10002, Los Alamitos, CA, USA. IEEE Computer Society.
  * Luan et al. (2018) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018.  Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3219–3232, Brussels, Belgium. Association for Computational Linguistics.
  * Nan et al. (2020) Guoshun Nan, Zhijiang Guo, Ivan Sekulić, and Wei Lu. 2020.  Reasoning with Latent Structure Refinement for Document-Level Relation Extraction.  ArXiv:2005.06312 [cs].
  * Peng et al. (2017) Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017\.  Cross-Sentence N-ary Relation Extraction with Graph LSTMs.  _Transactions of the Association for Computational Linguistics_ , 5:101–115.  Place: Cambridge, MA Publisher: MIT Press.
  * Quirk and Poon (2017) Chris Quirk and Hoifung Poon. 2017.  Distant Supervision for Relation Extraction beyond the Sentence Boundary.  ArXiv:1609.04873 [cs].
  * Sahu et al. (2019) Sunil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019.  Inter-sentence Relation Extraction with Document-level Graph Convolutional Neural Network.  ArXiv:1906.04684 [cs].
  * Song et al. (2018) Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018.  N-ary Relation Extraction using Graph-State LSTM.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2226–2235, Brussels, Belgium. Association for Computational Linguistics.
  * Tan et al. (2022a) Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022a.  Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation.  ArXiv:2203.10900 [cs].
  * Tan et al. (2022b) Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022b.  Revisiting DocRED – Addressing the False Negative Problem in Relation Extraction.  ArXiv:2205.12696 [cs].
  * Tang et al. (2020) Hengzhu Tang, Yanan Cao, Zhenyu Zhang, Jiangxia Cao, Fang Fang, Shi Wang, and Pengfei Yin. 2020.  HIN: Hierarchical Inference Network for Document-Level Relation Extraction.  In _Advances in Knowledge Discovery and Data Mining_ , Lecture Notes in Computer Science, pages 197–209, Cham. Springer International Publishing.
  * Tong et al. (2022) MeiHan Tong, Bin Xu, Shuai Wang, Meihuan Han, Yixin Cao, Jiangqi Zhu, Siyu Chen, Lei Hou, and Juanzi Li. 2022.  DocEE: A Large-Scale and Fine-grained Benchmark for Document-level Event Extraction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3970–3982, Seattle, United States. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.
  * Wang et al. (2020) Difeng Wang, Wei Hu, Ermei Cao, and Weijian Sun. 2020.  Global-to-Local Neural Networks for Document-Level Relation Extraction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3711–3721, Online. Association for Computational Linguistics.
  * Wang et al. (2022) Sijia Wang, Mo Yu, Shiyu Chang, Lichao Sun, and Lifu Huang. 2022.  Query and extract: Refining event extraction as type-oriented binary decoding.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 169–182, Dublin, Ireland. Association for Computational Linguistics.
  * Wang et al. (2023) Sijia Wang, Mo Yu, and Lifu Huang. 2023.  The art of prompting: Event detection based on type specific prompts.  In _ACL 2023_. Association for Computational Linguistics.
  * Wu et al. (2019) Ye Wu, Ruibang Luo, Henry C. M. Leung, Hing-Fung Ting, and Tak Wah Lam. 2019.  Renet: A deep learning approach for extracting gene-disease associations from literature.  In _RECOMB_.
  * Xiao et al. (2022) Yuxin Xiao, Zecheng Zhang, Yuning Mao, Carl Yang, and Jiawei Han. 2022.  SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2395–2409, Seattle, United States. Association for Computational Linguistics.
  * Xie et al. (2022) Yiqing Xie, Jiaming Shen, Sha Li, Yuning Mao, and Jiawei Han. 2022.  Eider: Empowering Document-level Relation Extraction with Efficient Evidence Extraction and Inference-stage Fusion.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 257–268, Dublin, Ireland. Association for Computational Linguistics.
  * Xu et al. (2021a) Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. 2021a.  Entity Structure Within and Throughout: Modeling Mention Dependencies for Document-Level Relation Extraction.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(16):14149–14157.  Number: 16.
  * Xu et al. (2017) Bo Xu, Yong Xu, Jiaqing Liang, Chenhao Xie, Bin Liang, Wanyun Cui, and Yanghua Xiao. 2017.  Cn-dbpedia: A never-ending chinese knowledge extraction system.  In _International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems_.
  * Xu et al. (2021b) Runxin Xu, Tianyu Liu, Lei Li, and Baobao Chang. 2021b.  Document-level event extraction via heterogeneous graph-based interaction model with a tracker.
  * Xu et al. (2022) Runxin Xu, Peiyi Wang, Tianyu Liu, Shuang Zeng, Baobao Chang, and Zhifang Sui. 2022\.  A Two-Stream AMR-enhanced Model for Document-level Event Argument Extraction.  ArXiv:2205.00241 [cs].
  * Xu et al. (2021c) Wang Xu, Kehai Chen, and Tiejun Zhao. 2021c.  Discriminative Reasoning for Document-level Relation Extraction.  ArXiv:2106.01562 [cs].
  * Xu et al. (2021d) Wang Xu, Kehai Chen, and Tiejun Zhao. 2021d.  Document-Level Relation Extraction with Reconstruction.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(16):14167–14175.  Number: 16.
  * Xu et al. (2023) Wang Xu, Kehai Chen, and Tiejun Zhao. 2023.  Document-Level Relation Extraction with Path Reasoning.  _ACM Transactions on Asian and Low-Resource Language Information Processing_ , 22(4):1–14.
  * Yang et al. (2018) Hang Yang, Yubo Chen, Kang Liu, Yang Xiao, and Jun Zhao. 2018.  DCFEE: A Document-level Chinese Financial Event Extraction System based on Automatically Labeled Training Data.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 50–55, Melbourne, Australia. Association for Computational Linguistics.
  * Yang et al. (2021) Hang Yang, Dianbo Sui, Yubo Chen, Kang Liu, Jun Zhao, and Taifeng Wang. 2021.  Document-level Event Extraction via Parallel Prediction Networks.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6298–6308, Online. Association for Computational Linguistics.
  * Yao et al. (2019a) Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019a.  DocRED: A Large-Scale Document-Level Relation Extraction Dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 764–777, Florence, Italy. Association for Computational Linguistics.
  * Yao et al. (2019b) Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019b.  DocRED: A large-scale document-level relation extraction dataset.  In _Proceedings of ACL 2019_.
  * Zeng et al. (2022) Qi Zeng, Qiusi Zhan, and Heng Ji. 2022.  EA2E: Improving Consistency with Event Awareness for Document-Level Argument Extraction.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 2649–2655, Seattle, United States. Association for Computational Linguistics.
  * Zeng et al. (2020) Shuang Zeng, Runxin Xu, Baobao Chang, and Lei Li. 2020.  Double Graph Based Reasoning for Document-level Relation Extraction.  ArXiv:2009.13752 [cs].
  * Zhang and Eskandarian (2022) Ce Zhang and Azim Eskandarian. 2022.  A quality index metric and method for online self-assessment of autonomous vehicles sensory perception.  _ArXiv_ , abs/2203.02588.
  * Zhang et al. (2021a) Ce Zhang, Azim Eskandarian, and Xuelai Du. 2021a.  Attention-based neural network for driving environment complexity perception.  _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_ , pages 2781–2787.
  * Zhang et al. (2021b) Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021b.  Document-level Relation Extraction as Semantic Segmentation.  ArXiv:2106.03618 [cs] version: 2.
  * Zhang et al. (2020) Zhenyu Zhang, Bowen Yu, Xiaobo Shu, Tingwen Liu, Hengzhu Tang, Wang Yubin, and Li Guo. 2020.  Document-level Relation Extraction with Dual-tier Heterogeneous Graph.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1630–1641, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Zheng et al. (2019a) Shun Zheng, Wei Cao, Wei Xu, and Jiang Bian. 2019a.  Doc2EDAG: An End-to-End Document-level Framework for Chinese Financial Event Extraction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 337–346, Hong Kong, China. Association for Computational Linguistics.
  * Zheng et al. (2019b) Shun Zheng, Wei Cao, Wei Xu, and Jiang Bian. 2019b.  Doc2EDAG: An end-to-end document-level framework for chinese financial event extraction.  In _EMNLP_.
  * Zhou et al. (2005) GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005.  Exploring various knowledge in relation extraction.  In _Proceedings of the 43rd annual meeting of the association for computational linguistics (acl’05)_ , pages 427–434.
  * Zhou et al. (2021) Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021.  Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(16):14612–14620.  Number: 16.
  * Zhu et al. (2022) Tong Zhu, Xiaoye Qu, Wenliang Chen, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Min Zhang. 2022.  Efficient Document-level Event Extraction via Pseudo-Trigger-aware Pruned Complete Graph.  ArXiv:2112.06013 [cs].

##  Appendix A Statistics of errors for different approaches and datasets

We evaluated three Doc-RE approaches on the DocRED and Re-DocRED datasets, two
Doc-EE methods on the WikiEvents and ChFinAnn datasets. The three Doc-RE
approaches represent three state-of-the-art methods for the path-based, graph-
based, and sequence-based approaches, while the two Doc-EE studies represent
the state-of-the-art of graph-based and generative-based approaches. For each
work, we randomly selected 50 errors as provided in Tables 4 and 5, (150
errors in Figure 2, 100 errors in Figures 3 and 4) and examined the causes of
them.

##  Appendix B Performance of Existing Methods

Performance of Doc-RE Existing Methods are shown in Table 9, Table 6, and
Table 7. Performance of Doc-EE Existing Methods are shown in Table 10 and
Table 8.

Model | F1
---|---
SAIS${}^{O}_{RE+CR+ET}$-SciBERT Xiao et al. (2022) | 87.10
DocuNet-SciBERT-base Zhang et al. (2021b) | 85.30
Eider(Rule)-SciBERT-base Xie et al. (2022) | 84.54
ATLOP-SciBERT-base Zhou et al. (2021) | 83.90
SSAN-SciBERT Xu et al. (2021a) | 83.70
Table 6:  Doc-RE GDA rank  Model | F1
---|---
SAIS${}^{O}_{RE+CR+ET}$-SciBERT Xiao et al. (2022) | 79.00
DocuNet-SciBERT-base Zhang et al. (2021b) | 76.30
Eider(Rule)-SciBERT-base Xie et al. (2022) | 70.63
ATLOP-SciBERT-base Zhou et al. (2021) | 69.40
SSAN-SciBERT Xu et al. (2021a) | 68.70
Table 7:  Doc-RE CDR rank  Model | F1
---|---
ReDEE Liang et al. (2022) | 81.90
Git Xu et al. (2021d) | 80.30
PTPCG Zhu et al. (2022) | 79.40
SCDEE Huang and Jia (2021) | 78.90
DE-PPN Yang et al. (2021) | 77.90
HRE Cui et al. (2022) | 76.80
Doc2EDAG Zheng et al. (2019b) | 76.30
Table 8:  Doc-EE ChFinAnn rank  Model | F1 | Ign-F1
---|---|---
KD-Rb-l Tan et al. (2022a) | 67.28 | 65.24
SSAN-RoBERTa-large+Adaptation Xu et al. (2021a) | 65.92 | 63.78
SAIS-RoBERTa-large Xiao et al. (2022) | 65.11 | 63.44
Eider-RoBERTa-large Xie et al. (2022) | 64.79 | 62.85
DocuNet-RoBERTa-large Zhang et al. (2021b) | 64.55 | 62.40
ATLOP-RoBERTa-large Zhou et al. (2021) | 63.40 | 61.39
Table 9:  Doc-RE DocRED rank  Model | Arg Identification | Arg Classification
---|---|---
Head F1 | Coref F1 | Head F1 | Coref F1
TSARlarge Xu et al. (2022) | 76.62 | 75.52 | 69.70 | 68.79
EA2E Zeng et al. (2022) | 74.62 | 75.77 | 68.61 | 69.70
BART-GenLi et al. (2021) | 71.75 | 72.29 | 64.57 | 65.11
OneIELi et al. (2021) | 61.88 | 63.63 | 57.61 | 59.17
BERT-QADu and Cardie (2020) | 61.05 | 64.59 | 56.16 | 59.36
Table 10:  Doc-EE WikiEvent rank


# CREATING PERSONALIZED SYNTHETIC VOICES FROM ARTICULATION IMPAIRED SPEECH
USING AUGMENTED RECONSTRUCTION LOSS

###### Abstract

This research is about the creation of personalized synthetic voices for head
and neck cancer survivors. It is focused particularly on tongue cancer
patients whose speech might exhibit severe articulation impairment. Our goal
is to restore normal articulation in the synthesized speech, while maximally
preserving the target speaker’s individuality in terms of both the voice
timbre and speaking style. This is formulated as a task of learning from noisy
labels. We propose to augment the commonly used speech reconstruction loss
with two additional terms. The first term constitutes a regularization loss
that mitigates the impact of distorted articulation in the training speech.
The second term is a consistency loss that encourages correct articulation in
the generated speech. These additional loss terms are obtained from frame-
level articulation scores of original and generated speech, which are derived
using a separately trained phone classifier. Experimental results on a real
case of tongue cancer patient confirm that the synthetic voice achieves
comparable articulation quality to unimpaired natural speech, while
effectively maintaining the target speaker's individuality. Audio samples are
available at https://myspeechproject.github.io/ArticulationRepair/.

Index Terms—  Personalized speech synthesis, articulation disorder, learning
from noisy labels

##  1 Introduction

Tongue cancer is a prevalent form of head and neck cancer. Its incidence rate
has been rising in recent years [1]. For advanced or recurrent tongue cancer,
surgical intervention may involve the removal of both the tongue and larynx
[2, 3]. As a consequence, the patient would lose voice and speaking ability
permanently. Voice is not only an important means of communication, but also
an integral part of a person’s identity. Personalized text-to-speech (TTS) was
proposed to enable people with vocal disabilities to communicate using their
own voices [4]. It was shown that using personalized TTS as an alternative
communication method can significantly improve the quality of life of
laryngectomees [5].

Personalized TTS models are trained with natural speech from the target
speaker. A major challenge in creating personalized synthetic voices for
tongue cancer survivors is that the training speech is often impaired. Both
the tumor and the treatment process could cause damages to the tongue,
resulting in impaired speech production. Such impairments are most commonly at
the articulation level [6, 7]. TTS models trained with articulation-impaired
speech would generate synthetic speech that expectantly contains similar types
of impairment and may not meet the intelligibility requirement for
communication. This motivates the present study on creating personalized
synthetic speech from articulation-impaired training data. Our goal is to
restore normal articulation in the synthesized speech while maximally
maintaining the target speaker’s individuality. We consider the target
speaker’s individuality to be well maintained if both the voice timbre and
good aspects of the original speaking style are kept.

From the machine learning perspective, the above goal can be formulated as a
task of learning from noisy labels[8, 9, 10, 11]. This is justified by the
fact that the degree of articulation impairment in the speech of a tongue
cancer patient varies across different types of speech sounds [12]: some
sounds remain largely unaffected and can be viewed as having clean labels,
while those with distorted articulation are with noisy labels. Inspired by the
re-weighting approach [13, 14] and the consistency constraint approach [15]
developed in studies of learning with noisy labels, we propose to augment the
conventional speech reconstruction loss in TTS model training with two
additional terms. The first term is a regularization loss that mitigates the
negative impact of distorted articulation in training speech. The second term
is a consistency loss that promotes accurate articulation in the output
speech. Specifically, a separately trained phone classifier is incorporated
during training to provide frame-level articulation scores for both original
and generated speech. The articulation score of original speech is used as the
re-weighting criteria to derive the regularization loss. The articulation
score of generated speech quantifies the inconsistency between the phone
classifier and the TTS model, representing the consistency loss.

The proposed approach is validated on a real patient case, the same as the one
reported in [16]. A personalized synthetic voice is built for a female
Cantonese speaker, who was advised to undertake laryngectomy for recurrent
tongue cancer. The patient already underwent partial-glossectomy six years
ago, and about 3/4 of her tongue was removed by surgical operation. As a
result, she had difficulties in producing certain speech sounds. The synthetic
voice is developed from the articulation-impaired speech of this patient using
the augmented reconstruction loss. Objective and subjective evaluations are
carried out to demonstrate the effectiveness of the augmented reconstruction
loss in restoring normal articulation and maintaining the target speaker's
individuality in the generated speech.

##  2 Related work

###  2.1 Voice reconstruction from impaired speech

The problem of voice reconstruction from impaired speech was tackled in a few
previous studies focusing on dysarthric speech. In the Voicebank project [4,
17, 18], it was proposed to repair speech by substituting impaired speech
features with that of an average healthy voice for training statistical
parametric model. More recently, a two-step approach was adopted to develop
neural speech synthesizers for people with dysarthria [19, 20]. A TTS model is
first trained with unimpaired speech of a healthy speaker. Subsequently, voice
conversion is applied to transform the voice characteristics of the
synthesized speech toward the target impaired speaker. Such VC-based approach
may be sub-optimal for the specific case being considered in this study. With
voice conversion, only the voice timbre is maintained and other speech
characteristics of the target speaker are ignored, leading to a loss of
speaker individuality.

In [16], we developed an approach to alleviate impaired speech articulation in
synthesized speech with guided diffusion models. An external phone classifier
was incorporated at the inference stage to identify unaffected time frames in
the generated speech and guide the generation process to repair those affected
ones. In this way, good aspects of the target speaker's speech characteristics
are preserved. However, experimental results revealed a gap in the
effectiveness of repairing distorted articulation compared to the VC approach.
We speculate this to be related to inaccurate assessment by the phone
classifier, which has to evaluate noisy speech during the reverse diffusion
process. In contrast, in this work the phone classifier takes clean speech as
input, therefore more accurate assessment from the phone classifier is
expected.

###  2.2 Learning from noisy labels

The problem of learning from noisy labels has been extensively studied [8, 9,
10, 11]. The proposed regularization loss is related to the importance re-
weighting approach in this field [13, 14], where samples with corrupted labels
are assigned small weights based on certain criteria. We utilize the
articulation scores by the external phone classifier as the re-weighting
criteria. The proposed consistency loss is inspired by the work of [15], which
incorporates the notion of consistency into the training objective to handle
noisy labels. In our work, consistency refers to the agreement between the
phone classifier and the TTS model regarding the content of the generated
speech. A similar loss has also been employed in [21] to reduce foreign accent
in cross-lingual voice conversion .

##  3 Augmented reconstruction loss

Fig. 1: Overall framework of the TTS acoustic model.

A neural TTS system typically consists of two components: (1) an acoustic
model that converts the input tokens into compact acoustic features; and (2) a
vocoder that translates the acoustic features into a time-domain speech
waveform. The present study is focused on the acoustic model as illustrated by
the top block in Figure 1. The input tokens include phonemes in the language
concerned and a silence token. The acoustic model is trained to generate a
Mel-spectrogram $\boldsymbol{y}_{1:T}$ that is close to the ground-truth
$\boldsymbol{y}^{*}_{1:T}$, given an input token sequence
$\boldsymbol{x}_{1:N}$. The input tokens are first encoded into hidden states
$\boldsymbol{h}_{1:N}$ through the encoder, and then expanded in time to align
with $\boldsymbol{y}^{*}_{1:T}$ by replicating each element according to its
duration. The phoneme duration $\boldsymbol{d}_{1:N}$ is obtained by forced
alignment at the training phase and predicted by a separate duration model at
the inference stage. The time-expanded hidden representations
$\boldsymbol{h}^{\prime}_{1:T}$ are concatenated with the speaker embedding
$\boldsymbol{s}_{e}$ and passed to the decoder for Mel-spectrogram generation.

The acoustic model is typically trained by minimizing the $L_{1}$ and/or
$L_{2}$ loss between the ground-truth Mel-spectrogram $\boldsymbol{y}^{*}$ and
the predicted one $\boldsymbol{y}$ [22, 23]. When the training speech contains
articulation impairment, synthetic speech generated by the trained model would
contain similar types of impairment. To alleviate this problem, an augmented
reconstruction loss is introduced. It incorporates two additional loss terms,
namely regularization loss and consistency loss.

###  3.1 Regularization loss

As shown in the middle block in Figure 1, the regularization loss is derived
by explicitly modelling articulation impairment and then applying maximum
likelihood estimation.

Let $\boldsymbol{y}_{1:T}$ be the Mel-spectrogram with unimpaired articulation
generated by the acoustic model. We introduce an articulation impairment
model, which transforms the Mel-spectrogram of normal speech into that of
impaired speech:

$\displaystyle
P(\boldsymbol{y}^{*}_{1:T}|\boldsymbol{y}_{1:T})\propto\prod_{t=1}^{T}\|\boldsymbol{y}^{*}_{t}-\boldsymbol{y}_{t}\|^{\alpha_{t}}_{2}\cdot\exp\left(-\frac{\|\boldsymbol{y}^{*}_{t}-\boldsymbol{y}_{t}\|_{2}^{2}}{2\sigma^{2}}\right),$
(1)

where $\alpha_{t}>0$ controls the severity of articulation impairment at the
$t^{th}$ speech frame. The variance $\sigma^{2}$ is a constant term over time.
A larger value of $\alpha_{t}$ indicates more severe impairment, as it makes
the mode of the distribution deviate further from unimpaired speech
$\boldsymbol{y}_{t}$.

The severity of impairment can be inferred using a separately trained phone
classifier. We choose to set the values of $\boldsymbol{\alpha}_{1:T}$ as
follows:

$\displaystyle\alpha_{t}\propto\exp\left(-\lambda\cdot p^{*}_{t}\right)\text{
for }t=1\cdots T,$ (2)

where $p^{*}_{t}$ denotes the posterior for the $t^{\text{th}}$ phoneme label
given the observation $\boldsymbol{y}^{*}_{1:T}$. The parameter $\lambda>0$
controls the rate at which $\alpha_{t}$ decreases as the posterior of the
ground-truth phoneme increases.

Applying maximum likelihood estimation using the results from equations (1)
and (2), the following loss function can be obtained, where $\beta>0$ is a
hyper-parameter.

$\displaystyle\underbrace{\sum_{t=1}^{T}\|\boldsymbol{y}^{*}_{t}-\boldsymbol{y}_{t}\|_{2}^{2}}_{\mathcal{L}_{\text{rec}}}+\beta\cdot\underbrace{\sum_{t=1}^{T}-\exp\left(-\lambda\cdot
p^{*}_{t}\right)\log\|\boldsymbol{y}^{*}_{t}-\boldsymbol{y}_{t}\|}_{\mathcal{L}_{\text{reg}}}$
(3)

It consists of a conventional reconstruction loss ($\mathcal{L}_{\text{rec}}$)
and a regularization loss ($\mathcal{L}_{\text{reg}}$).
$\mathcal{L}_{\text{reg}}$ penalizes $\boldsymbol{y}$ when it closely matches
$\boldsymbol{y}^{*}$ for impaired speech frames identified by the phone
classifier, hence alleviates the impact of articulation disorder presented in
the training samples.

###  3.2 Consistency loss

The articulation impairment model described in the preceding section assumes
unimpaired articulation in the generated speech $\boldsymbol{y}_{1:T}$.
However, such assumption is not enforced explicitly in the training process.
To address the issue, we introduce another loss term inspired by the
observation-prediction consistency constraint as described in [15]. In our
research, ``observations'' refers to the input phonemes, and ``prediction'' is
the generated speech. We leverage the phone classifier to quantify the
inconsistency between input phonemes and generated speech.

Let $\boldsymbol{x}^{\prime}_{1:T}$ denote the time-expanded phoneme sequence,
aligned with the generated speech $\boldsymbol{y}_{1:T}$. From the
classifier's perspective, which only has access to the generated speech (i.e.,
the ``prediction''),
$P_{\text{cls}}(\boldsymbol{x}^{\prime}_{1:T}|\boldsymbol{y}_{1:T})=\prod_{t=1}^{T}p_{t}$,
where $p_{t}$ represents the posterior of the $t^{\text{th}}$ phoneme label
estimated by the classifier. However, according to the TTS model,
$P_{\text{tts}}(\boldsymbol{x}^{\prime}_{1:T}|\boldsymbol{y}_{1:T})=1.0$ as
$\boldsymbol{x}^{\prime}_{1:T}$ is what it observes (i.e., the
``observation"). The consistency loss is defined as the KL divergence between
the two probability distributions:

$\displaystyle\mathcal{L}_{\text{consis}}=D_{KL}\left(P_{\text{tts}}\|P_{\text{cls}}\right)=-\sum_{t=1}^{T}\log
p_{t}$ (4)

Minimizing the consistency loss enhances the consistency between input
phonemes and the generated speech, thereby promoting correct articulation in
the synthesized output.

The total loss for training the TTS acoustic model is then defined as equation
(5), where $\beta>0$ and $\gamma>0$ are hyper-parameters to be tuned.

$\displaystyle\mathcal{L}_{total}=\mathcal{L}_{\text{rec}}+\beta\cdot\mathcal{L}_{\text{reg}}+\gamma\cdot\mathcal{L}_{\text{consis}}$
(5)

##  4 Experiments

We validate the proposed approach on a real case: creating a synthetic voice
for a female Cantonese speaker from recordings of articulation-impaired
speech. The target speaker underwent partial-glossectomy six years ago and
around 3/4 of her tongue was removed surgically. Phonemes including the velar
consonants /k/, /g/, /N/, alveolar consonants /t/, /d/ and high vowels /9/,
/y/ are noticeably impacted in her speech. The quality of the synthetic voice
is evaluated in terms of intelligibility, overall impression and speaker
similarity. Experimental details are described below.

###  4.1 Systems, speech corpora and training details

We adopt the Mel-spectrogram prediction network from Tactotron 2[22] for
acoustic modelling, but replace the attention module with the duration
predictor and aligner from DurIAN [23] to improve model robustness. We use a
pre-trained HiFi-GAN [24] as the vocoder. The external phone classifier is
adapted from Jasper[25], with the convolution stride removed to enable frame-
wise prediction. We refer to the proposed system, which trains the acoustic
model with augmented reconstruction loss, as DuriTaco+AugRecLoss.

The proposed system is compared with two baselines. Both share the same
network architecture as the proposed system, but differ in terms of the
training objective and/or training data. The first baseline, DuriTaco, trains
the acoustic model with conventional $L_{2}$ Mel-spectrogram reconstruction
loss. The second baseline, DuriTaco+VC, trains the acoustic model on VC
generated speech, also with $L_{2}$ reconstruction loss. We choose NANSY [26]
as the VC model for its strong performance in cross-lingual setting. Our
assumption is that impaired speech can be viewed as a distinct language,
therefore a strong cross-lingual VC model should also be effective for voice
conversion between normal and impaired speech.

Our target speaker has contributed a total of 377 recordings, which gives
approximately 24 minutes of speech data. Speech was recorded using a TASCAM
DR-44WL at 44.1 kHz in a reasonably quiet environment. We refer to this
collection as Recording_T. We additionally collected 377 recordings of
unimpaired speech from another female Cantonese speaker. This speech corpus,
named Recording_S, contains exactly the same content as Recording_T, and
serves as the source speaker's data for voice conversion. For TTS pre-
training, we use the multi-speaker Cantonese speech corpus CUSENT[27]. It
consists of around 20-hour read speech from 80 speakers, sampled at 16 kHz.
The external phone classifier and the VC model are also trained using this
corpus.

Speech data are resampled to 22.05 kHz to be consistent with HiFi-GAN's
settings for Mel-Spectrogram computation. Acoustic models of the two baselines
are pre-trained on CUSENT for 600 epochs at batch size 32, then fine-tuned on
the target speaker's data (either original recordings or VC generated speech)
for 5000 steps at batch size 16. The acoustic model in the proposed system is
initialized with the DuriTaco baseline. It is then fine-tuned on the target
speaker's data, combined with 2000 randomly selected recordings from CUSENT,
for 750 steps at batch size 32. We empirically find that incorporating multi-
speaker speech data helps to protect the phone classifier from adversarial
attacks by the TTS model. Hyper-parameters in the loss function are set to
$\beta=0.05,\gamma=0.3,\lambda=25.0$. For all the systems, the fine-tuning
process utilized only 347 recordings from the target speaker, with the
remaining 30 sentences held out for evaluation. During fine-tuning, parameters
of the encoder are frozen.

###  4.2 Objective evaluation

For objective evaluation, we utilized a separately trained CTC-based automatic
speech recognition (ASR) model. It is implemented following the recipe from
SpeechBrain111https://github.com/speechbrain/speechbrain/tree/develop/recipes/TIMIT/ASR/CTC,
and is trained on a commercial Cantonese speech corpus, which contains 80
hours of read speech from 136 speakers.

We resynthesized all 377 sentences in Recording_T with the proposed system and
the two baselines. The Phone Error Rate (PER%) results in Table 1 show that
the DuriTaco baseline largely retains the problematic articulation style
observed in the original recordings. On the other hand, the proposed system is
able to generate highly intelligible speech, with articulation quality
comparable to the VC baseline and unimpaired natural speech. Results in the
last two rows suggest that applying both $\mathcal{L}_{\text{reg}}$ and
$\mathcal{L}_{\text{consis}}$ is the most effective, while applying
$\mathcal{L}_{\text{consis}}$ accounts for most of the improvement. This is
expected because $\mathcal{L}_{\text{consis}}$ explicitly enforces the
consistency between the input phonemes and the generated speech.

Table 1: PER(%) results on both the original recordings and the resynthesized speech. Case |  PER%$\downarrow$
---|---
Recording_S | 11.2
Recording_T | 43.1
DuriTaco | 37.6
DuriTaco + VC | 14.9
DuriTaco + AugRecLoss (w/o $\mathcal{L}_{\text{reg}}$) | 13.8
DuriTaco + AugRecLoss (w/ $\mathcal{L}_{\text{reg}}$) | 12.0

###  4.3 Subjective evaluation

The proposed TTS system and the two baselines were used to generate speech for
the 30 sentences in the held-out evaluation set. For each sentence, the three
synthetic voices along with the two real speech samples (one from the source
speaker and one from the target speaker) formed five stimuli in total.

We then carried out a web-based listening test to assess the quality of
different systems using these stimuli. The test included two parts: a mean
opinion score (MOS) test and a similarity mean opinion score (SMOS) test, each
consisting of 30 questions corresponding to the test sentences. In each
question, a stimulus was randomly chosen from the five systems, and for the
SMOS test, a reference recording sample from the target speaker was also
provided. Listeners were asked to rate the overall impression or speaker
similarity on a 5-point scale. A total of 66 native Cantonese speakers
participated in the test, and after filtering out responses with overall
impression ratings below 4.0 for natural speech recordings (Recording_S), we
obtained a total of 57 valid responses.

The MOS results in Table 2 show that all systems, including the DuriTaco
baseline, produce speech that is more natural than the original articulation
impaired speech. This general improvement might be attributed to the multi-
speaker pre-training, which incorporates the natural articulation style of
unimpaired speech into the model. Nevertheless, the VC baseline and the
proposed system exhibit the most noticeable improvement in terms of overall
impression. On the other hand, the SMOS results demonstrate the superiority of
the proposed system over the VC baseline in preserving the target speaker's
individuality, which is a desirable quality in the context of personalized
speech synthesis. Readers are encouraged to visit the demo
page222https://myspeechproject.github.io/ArticulationRepair/ to listen to
audio samples.

Table 2: MOS and SMOS evaluations with 95% confidence intervals for different systems. Case |  MOS $\uparrow$ |  SMOS $\uparrow$
---|---|---
Recording_S | $4.76\pm 0.08$ | $1.06\pm 0.04$
Recording_T | $2.38\pm 0.12$ | $4.23\pm 0.07$
DuriTaco | $2.88\pm 0.12$ | $\mathbf{4.22\pm 0.15}$
DuriTaco + VC | $\mathbf{3.60\pm 0.09}$ | $3.30\pm 0.19$
DuriTaco + AugRecLoss | $\mathbf{3.66\pm 0.08}$ | $\mathbf{4.00\pm 0.17}$

##  5 Conclusion

We introduced an augmented reconstruction loss for developing personalized
synthetic voices from articulation-impaired speech. Experimental results from
a real patient case demonstrate that when trained with the proposed loss
function, the synthetic voice is able to restore normal articulation in the
synthesized output, while effectively maintaining the target speaker's voice
identity and speaking style. Although initially validated on post-glossectomy
speech, the proposed approach should be applicable to other types impaired
speech where the speech disorder mainly affects articulation.

##  6 Acknowledgements

We would like to extend our sincere gratitude to the target speaker, who
generously agreed to showcase her synthetic voice on the demo page.

## References

  * [1] J. E. Tota, W. F. Anderson, C. Coffey, J. Califano, W. Cozen, R. L. Ferris, M. S. John, E. E. Cohen, and A. K. Chaturvedi, ``Rising incidence of oral tongue cancer among white men and women in the united states, 1973–2012,'' Oral Oncol., vol. 67, pp. 146–152, 2017.
  * [2] C. Amechi, L. Ottenstein, A. Lang, T. McClary, A. M. Avinger, A. J. Burnham, M. D. Dixon, R. D. Pentz, and N. C. Schmitt, ``Quality of life and decisional regret after total glossectomy with laryngectomy: A single-institution case series,'' Oral Oncol., vol. 142, p. 106434, 2023.
  * [3] A. J. Burnham, B. J. Boyce, A. S. Kaka, M. W. El-Deiry, M. E. Sebelik, H. M. Baddour, M. R. Patel, J. H. Gross, M. Nathan, J. M. Waller, et al., ``Survival and functional outcomes after total glossectomy with total laryngectomy: Case series from a high-volume tertiary institution,'' Oral Oncol., vol. 137, p. 106301, 2023.
  * [4] J. Yamagishi, C. Veaux, S. King, and S. Renals, ``Speech synthesis technologies for individuals with vocal disabilities: Voice banking and reconstruction,'' Acoust. Sci. Technol., vol. 33, no. 1, pp. 1–5, 2012.
  * [5] J. Mertl, E. Žáčková, and B. Řepová, ``Quality of life of patients after total laryngectomy: the struggle against stigmatization and social exclusion using speech synthesis,'' Disabil. Rehabil.: Assist. Technol., vol. 13, no. 4, pp. 342–352, 2018.
  * [6] S. Kraaijenga, I. Oskam, R. Van Son, O. Hamming-Vrieze, F. Hilgers, M. Van den Brekel, and L. Van der Molen, ``Assessment of voice, speech, and related quality of life in advanced head and neck cancer patients 10-years+ after chemoradiotherapy,'' Oral Oncol., vol. 55, pp. 24–30, 2016.
  * [7] A. Acher, P. Perrier, C. Savariaux, and C. Fougeron, ``Speech production after glossectomy: methodological aspects,'' Clin. Linguist. Phon., vol. 28, no. 4, pp. 241–256, 2014.
  * [8] X. Zhu and X. Wu, ``Class noise vs. attribute noise: A quantitative study,'' Artif. Intell. Rev., vol. 22, pp. 177–210, 2004.
  * [9] N. Natarajan, I. S. Dhillon, P. K. Ravikumar, and A. Tewari, ``Learning with noisy labels,'' in Proc. NeurIPS, vol. 26, 2013.
  * [10] W. Hu, Z. Li, and D. Yu, ``Simple and effective regularization methods for training on noisily labeled data with generalization guarantee,'' in Proc. ICLR, 2019.
  * [11] H. Song, M. Kim, D. Park, Y. Shin, and J.-G. Lee, ``Learning from noisy labels with deep neural networks: A survey,'' IEEE Trans. Neural Netw. Learn. Syst., 2022.
  * [12] B. M. Halpern, R. van Son, M. van den Brekel, and O. Scharenborg, ``Detecting and analysing spontaneous oral cancer speech in the wild,'' in Proc. INTERSPEECH, pp. 4826–4830, 2020.
  * [13] T. Liu and D. Tao, ``Classification with noisy labels by importance reweighting,'' IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 3, pp. 447–461, 2015.
  * [14] M. Ren, W. Zeng, B. Yang, and R. Urtasun, ``Learning to reweight examples for robust deep learning,'' in Proc. ICML, pp. 4334–4343, 2018.
  * [15] S. E. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich, ``Training deep neural networks on noisy labels with bootstrapping,'' in Proc. ICLR Workshop, 2015.
  * [16] Y. Tian, G. Zhang, and T. Lee, ``Creating personalized synthetic voices from post-glossectomy speech with guided diffusion models,'' in Proc. INTERSPEECH, pp. 4893–4897, 2023.
  * [17] S. Creer, S. Cunningham, P. Green, and J. Yamagishi, ``Building personalised synthetic voices for individuals with severe speech impairment,'' Comput Speech Lang, vol. 27, no. 6, pp. 1178–1193, 2013.
  * [18] C. Veaux, J. Yamagishi, and S. King, ``Using HMM-based speech synthesis to reconstruct the voice of individuals with degenerative speech disorders,'' in Proc. INTERSPEECH, pp. 967–970, 2012.
  * [19] K. Matsubara, T. Okamoto, R. Takashima, T. Takiguchi, T. Toda, Y. Shiga, and H. Kawai, ``High-intelligibility speech synthesis for dysarthric speakers with LPCNet-based TTS and CycleVAE-based VC,'' in Proc. ICASSP, pp. 7058–7062, 2021.
  * [20] R. Nanzaka and T. Takiguchi, ``Hybrid text-to-speech for articulation disorders with a small amount of non-parallel data,'' in Proc. APSIPA ASC, pp. 1761–1765, 2018.
  * [21] Y. Zhou, Z. Wu, X. Tian, and H. Li, ``Optimization of cross-lingual voice conversion with linguistics losses to reduce foreign accents,'' IEEE/ACM Trans. Audio, Speech, Language Process., vol. 31, pp. 1916–1926, 2023.
  * [22] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Ryan, R. A. Saurous, Y. Agiomyrgiannakis, and Y. Wu, ``Natural TTS synthesis by conditioning wavenet on Mel-spectrogram predictions,'' in Proc. ICASSP, pp. 4779–4783, 2018.
  * [23] C. Yu, H. Lu, N. Hu, M. Yu, C. Weng, K. Xu, P. Liu, D. Tuo, S. Kang, G. Lei, D. Su, and D. Yu, ``DurIAN: Duration informed attention network for speech synthesis,'' in Proc. INTERSPEECH, pp. 2027–2031, 2020.
  * [24] J. Kong, J. Kim, and J. Bae, ``HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,'' in Proc. NeurIPS, pp. 17022–17033, 2020.
  * [25] J. Li, V. Lavrukhin, B. Ginsburg, R. Leary, O. Kuchaiev, J. M. Cohen, H. Nguyen, and R. T. Gadde, ``Jasper: An end-to-end convolutional neural acoustic model,'' in Proc. INTERSPEECH, pp. 71–75, 2019.
  * [26] H.-S. Choi, J. Lee, W. Kim, J. Lee, H. Heo, and K. Lee, ``Neural analysis and synthesis: Reconstructing speech from self-supervised representations,'' in Proc. NeurIPS, pp. 16251–16265, 2021.
  * [27] T. Lee, W. K. Lo, P.-C. Ching, and H. Meng, ``Spoken language resources for cantonese speech processing,'' Speech Communication, vol. 36, no. 3-4, pp. 327–342, 2002.