cdevcr short = CDCR, long = cross-document event coreference resolution, tag =
main srl short = SRL, long = semantic role labeling, tag = main qa short = QA,
long = question answering, tag = main mds short = MDS, long = multi-document
summarization, tag = main iaa short = IAA, long = inter-annotator agreement,
tag = main mlp short = MLP, long = multi-layer perceptron, tag = main wsd
short = WSD, long = word sense disambiguation, tag = main cdevc short = CDC,
long = cross-document event coreference, tag = writing-aid ecb short = ECB,
long = EventCorefBank, extra = The first corpus iteration developed by Bejan
and Harabagiu (2010)., tag = corpus eecb short = EECB, long = Extended
EventCorefBank, extra = Lee et al. (2012)’s extension of the ECB corpus in
which entity coreference annotations were added., tag = corpus ecbp short =
ECB+, long = EventCorefBank+, extra = Extension of the EECB corpus in which
Cybulska and Vossen (2014b) added a second subtopic for each topic., tag =
corpus gvc short = GVC, long = Gun Violence Corpus, extra = Developed by
Vossen et al. (2018)., tag = corpus fcc short = FCC, long = Football
Coreference Corpus, extra = Sentence-level corpus developed by Bugert et al.
(2020)., tag = corpus fcct short = FCC-T, long = Football Coreference Corpus,
extra = Token-level reannotation of fcc produced in this work., tag = corpus
cr2020 short = CR2020, long = Cremisini and Finlayson (2020), tag = system
me2020 short = ME2020, long = Meged et al. (2020), tag = system ba2019 short =
BA2019, long = Barhom et al. (2019), tag = system ke2018 short = KE2018, long
= Kenyon-Dean, Cheung, and Precup (2018), tag = system mi2018 short = MI2018,
long = Mirza, Darari, and Mahendra (2018), tag = system vo2018 short = VO2018,
long = Vossen (2018), tag = system vo2016 short = VO2016, long = Vossen and
Cybulska (2016), tag = system cy2015 short = CY2015, long = Cybulska and
Vossen (2015), tag = system ya2015 short = YA2015, long = Yang, Cardie, and
Frazier (2015), tag = system le2012 short = LE2012, long = Lee et al. (2012),
tag = system

# Generalizing Cross-Document Event Coreference Resolution Across Multiple
Corpora

Michael Bugert  UKP Lab
Department of Computer Science
Technical University of Darmstadt
https://www.ukp.tu-darmstadt.de/  Nils Reimers  UKP Lab  Iryna Gurevych  UKP
Lab

###### Abstract

Cross-document event coreference resolution (CDCR) is an NLP task in which
mentions of events need to be identified and clustered throughout a collection
of documents. CDCR aims to benefit downstream multi-document applications, but
despite recent progress on corpora and system development, downstream
improvements from applying CDCR have not been shown yet. We make the
observation that every CDCR system to date was developed, trained, and tested
only on a single respective corpus. This raises strong concerns on their
generalizability — a must-have for downstream applications where the magnitude
of domains or event mentions is likely to exceed those found in a curated
corpus. To investigate this assumption, we define a uniform evaluation setup
involving three CDCR corpora: ECB+, the Gun Violence Corpus and the Football
Coreference Corpus (which we reannotate on token level to make our analysis
possible). We compare a corpus-independent, feature-based system against a
recent neural system developed for ECB+. Whilst being inferior in absolute
numbers, the feature-based system shows more consistent performance across all
corpora whereas the neural system is hit-and-miss. Via model introspection, we
find that the importance of event actions, event time, etc. for resolving
coreference in practice varies greatly between the corpora. Additional
analysis shows that several systems overfit on the structure of the ECB+
corpus. We conclude with recommendations on how to achieve generally
applicable CDCR systems in the future — the most important being that
evaluation on multiple CDCR corpora is strongly necessary. To facilitate
future research, we release our dataset, annotation guidelines, and system
implementation to the public.111https://github.com/UKPLab/cdcr-beyond-corpus-
tailored

##  1 Introduction

To move beyond interpreting documents in isolation in multi-document NLP tasks
such as mds or qa, a text understanding technique is needed to connect
statements from different documents. A strong contender for this purpose is
cdevcr. In this task, systems needs to (1) find mentions of events in a
collection of documents and (2) cluster those mentions together which refer to
the same event (see Figure 1). An event refers to an action taking place at a
certain time and location with certain participants Cybulska and Vossen
(2014b). cdevcr requires deep text understanding and depends on a multitude of
other NLP tasks such as srl, temporal inference, and spatial inference, each
of which is still being researched and not yet solved. Furthermore, cdevcr
systems need to correctly predict the coreference relation between any pair of
event mentions in a corpus. Since the number of pairs grows quadratically with
the number of mentions, achieving scalable text understanding becomes an added
challenge in cdevcr.

Figure 1: cdevcr example with excerpts of three documents from our token-level
reannotation of the fcct. The seven indicated event mentions refer to four
different events. For the "victory" event mention, three participant mentions
and one temporal mention are additionally marked.

In recent years, new cdevcr corpora such as the gvc Vossen et al. (2018) and
fcc Bugert et al. (2020) have been developed, and the state-of-the-art
performance on the most commonly used corpus ecbp Cybulska and Vossen (2014b)
has risen steadily Meged et al. (2020); Barhom et al. (2019); Kenyon-Dean,
Cheung, and Precup (2018). We believe that cdevcr can play a vital role for
downstream multi-document tasks, and so do other authors in this area Bejan
and Harabagiu (2014); Yang, Cardie, and Frazier (2015); Upadhyay et al.
(2016); Choubey and Huang (2017); Choubey, Raju, and Huang (2018); Choubey and
Huang (2018); Kenyon-Dean, Cheung, and Precup (2018); Barhom et al. (2019).
Yet, despite the progress made so far, we are not aware of a study which
demonstrates that employing a recent cdevcr system is indeed helpful
downstream. We make the key observation that all existing cdevcr systems Meged
et al. (2020); Cremisini and Finlayson (2020); Barhom et al. (2019); Kenyon-
Dean, Cheung, and Precup (2018); Mirza, Darari, and Mahendra (2018); Vossen
(2018) were designed, trained, and evaluated on a single corpus respectively.
This points to a risk of systems overspecializing on their target corpus
instead of learning to solve the overall task, rendering such systems
unsuitable for downstream applications where generality and robustness is
required. The fact that cdevcr annotation efforts annotated only a subset of
all coreference links to save costs Bugert et al. (2020) further aggravates
this situation.

We are, to the best of our knowledge, the first to investigate this risk. In
this work, we determine the state of generalizability in cdevcr with respect
to corpora and systems, identify the current issues, and formulate
recommendations on how cdevcr systems which are robustly applicable in
downstream scenarios can be achieved in the future. We divide our analysis
into five successive stages:

  1. 1.

Cross-dataset modeling of cdevcr is made difficult by annotation differences
between the ecbp, fcc, and gvc corpora. We establish compatibility by
annotating the fcct, an extension of the fcc reannotated on the token level.

  2. 2.

Analyzing generalizability across corpora is best performed with an
interpretable cdevcr system which is equally applicable on all corpora. To
fulfill this requirement, we develop a conceptually simple mention-pair cdevcr
system which uses the union of features found in related work.

  3. 3.

To compare the generalization capabilities of cdevcr system architectures, we
train and test this system and a close to state-of-the-art neural system
Barhom et al. (2019) on the ecbp, fcct and gvc corpora. We find that the
neural system does not robustly handle cdevcr on all corpora because its input
features and architecture require ecbp-like corpora.

  4. 4.

There is a lack of knowledge on how the cdevcr task manifests itself in each
corpus, especially with regard to which pieces of information (out of event
action, participants, time, and location) are the strongest signals for event
coreference. Via model introspection, we observe significant differences
between corpora, finding that decisions in ecbp are strongly driven by event
actions whereas fcct and gvc are more balanced and additionally require text
understanding of event participants and time.

  5. 5.

Finally, we evaluate our feature-based system in a cross-dataset transfer
scenario to analyze the generalization capabilities of trained cdevcr models.
We find that models trained on a single corpus do not perform well on other
unseen corpora.

Based on these findings, we conclude with recommendations for the evaluation
of cdevcr which will pave the way for more general and comparable systems in
the future. Most importantly, the results of our analysis unmistakably show
that evaluation on multiple corpora is imperative given the current set of
available cdevcr corpora.

##### Article Structure

The next section provides background information on the cdevcr task, corpora,
and systems, followed by related work on feature importance in cdevcr (Section
3). Section 4 covers the re-annotation and extension of the fcc corpus. We
explain the feature-based cdevcr system in Section 5 before moving on to a
series of experiments: we compare this system and the neural system of Barhom
et al. (2019) in Section 6. In Section 7 we analyze the signals for event
coreference in each corpus. Lastly, we test model generalizability across
corpora in Section 8. We discuss the impact of these experiments and offer
summarized recommendations on how to achieve general cdevcr systems in the
future in Sections 9 and 10. We conclude with Section 11.

##  2 Background on cdevcr

We explain the cdevcr task in greater detail, report on the most influential
cdevcr datasets and cover notable coreference resolution systems developed for
each corpus.

###  2.1 Task Definition

The cdevcr task is studied for several domains including news events in
(online) news articles, events pertaining to the treatment of patients in
physician’s notes Raghavan et al. (2014); Wright-Bettner et al. (2019) or the
identification and grouping of biomedical events in research literature Van
Landeghem et al. (2013). In this work, we restrict ourselves to the most
explored variant of cdevcr in the news domain.

We follow the task definition and terminology of Cybulska and Vossen (2014b).
Here, events consist of four event components – an action, several human or
non-human participants, a time and a location. Each of these components can be
mentioned in text, that is, an action mention would be the text span
referencing the action of an event instance. An example is shown in Figure 1,
where the rightmost document references a football match between England and
Sweden. The action mention for this event is "victory", alongside three entity
mentions "England" (the population of England), "fans" (English football
fans), and "Sweden" (the Swedish national football team) who took part in the
event. The temporal expression "on Saturday" grounds the event mention to a
certain time, which in this case depends on the date the news article was
published on.

Different definitions have been proposed for the relation of event
coreference. Efforts such as ACE Walker et al. (2006) only permit the
annotation of identity between event mentions whereas Hovy et al. (2013)
further distinguish subevent or membership relations. Definitions generally
need to find a compromise between complexity and ease of annotation,
particularly for the cross-document case (see Wright-Bettner et al. (2019) for
a detailed discussion). We follow the (comparatively simple) definition of
Cybulska and Vossen (2014b) in which two action mentions corefer if they refer
to the same real-world event, meaning their actions and their associated
participants, time, and location are semantically equivalent. Relevant
examples are shown in Figure 1 where all action mentions of the same color
refer to the same event. The two steps a cdevcr system needs to perform
therefore are (1) the detection of event actions and event components and (2)
the disambiguation of event actions to produce a cross-document event
clustering. A challenging aspect of cdevcr is the fact that finding mentions
of all four event components in the same sentence is rare, meaning that
information may have to be inferred from the document context or in some
cases, it may not be present in the document at all. The second challenge is
efficiently scaling the clustering process to large document collections with
thousands of event mentions since every possible pair of event mentions could
together form a valid cluster.

###  2.2 System Requirements

The requirements that downstream applications place on systems resolving cdevc
can be diverse. We establish high-level requirements that a system performing
cdevcr on news text should meet:

  * •

Datasets may consist of many interwoven topics. Systems should perform well on
a broad selection of event types of different properties (punctual events such
as accidents, longer-term events such as natural disasters, pre-planned events
such as galas or sports competitions).

  * •

To provide high-quality results, systems should fully support the definition
of event coreference mentioned previously, meaning they find associations
between event mentions at a level human readers would be able to by inferring
temporal and spatial clues from the document context and reasoning over event
action and participants.

  * •

Datasets may consist of a large number of documents containing many event
mentions. We expect cdevcr systems to be scalable enough to handle 100k event
mentions in a reasonable amount of time (less than one day on a single-GPU
workstation).

###  2.3 Corpora

The corpus most commonly associated with cdevcr is ecbp. Originally developed
as the ecb corpus Bejan and Harabagiu (2010), it was enriched with entity
coreference annotations by Lee et al. (2012) to form the eecb corpus. This
corpus was later extended with 500 additional documents by Cybulska and Vossen
(2014b) to create the ecbp corpus. This most recent version contains 982 news
articles on 43 topics. The topics were annotated separately, meaning there are
no coreference links across topics. For each topic (for example "bank
explosions"), there are two main events ("Bank explosion in Oregon 2008" and
"Bank explosion in Athens 2012") and several news documents which report on
either of those two events. The set of documents reporting on the same event
is commonly referred to as a subtopic. ecbp is the only corpus of those
discussed here which does not provide the publication date for each document.
It does however contain annotations for all four event components as well as
additional cross-document entity coreference annotations for participants,
time, and location mentions.

eecb

The fcc Bugert et al. (2020) contains 451 sports news articles on football
tournaments annotated with cdevc. The annotation was carried out via
crowdsourcing and focused on retrieving cross-subtopic event coreference
links. Following the nomenclature of Bugert et al. (2020), a within-subtopic
coreference link is defined by a pair of coreferring event mentions which
originate from two documents reporting about the same overall event. For
example in ecbp, two different news articles reporting about the same bank
explosion in Athens in the year 2012 may both mention the event of the
perpetrators fleeing the scene. For a cross-subtopic event coreference link,
two event mentions from articles on different events need to corefer. A sports
news article summarizing a quarter-final match of a tournament could for
example recommend watching the upcoming semifinal, whereas an article written
weeks later about the grand final may refer to the same semifinal in an
enumeration of a team’s past performances in the tournament. A concrete
example is shown in Figure 1, where the mentions "beat" and "test" corefer
while belonging to different subtopics. Cross-subtopic coreference links are a
crucial aspect of cdevcr since they connect mentions from documents with low
content overlap, forming far-reaching coreference clusters which should prove
particularly beneficial for downstream applications Bugert et al. (2020). In
fcc, event mentions are annotated only at the sentence level contrary to ecbp
and gvc which feature token level annotations.

The gvc Vossen et al. (2018) is a collection of 510 news articles covering 241
gun violence incidents. The goal was to create a challenging cdevcr corpus
with many similar event mentions. Each news article belongs to the same topic
(gun violence) and only event mentions related to gun violence were annotated
("kill", "wounded", etc.). Cross-subtopic coreference links were not
annotated.

|  ecbp |  gvc |  fcc |  fcct
---|---|---|---|---
annotation unit | token | token | sentence | token
included annotations |  |  |  |
event coreference | ✓ | ✓ | ✓ | [][c]✓
entity coreference | ✓ | [][c]  | [][c]  | [][c]
document publ. date | [][c]  | ✓ | ✓ | [][c]✓
particip., time, location | ✓ | [][c]  | [][c]  | [][c]✓
semantic roles | [][c]  | [][c]  | [][c]  | (✓)
topics | $43$ | $1$ | $1$ | $1$
subtopics per topic | $2$ | $241$ | $183$ | $183$
documents | $982$ | $510$ | $451$ | $451$
sentences | $16,314$ | $9782$ | $14,940$ | $14,940$
event mentions | $6833$ | $7298$ | $2374$ | $3563$
event clusters | $2741$ | $1411$ | $218$ | $469$
singletons | $2019$ | $365$ | $50$ | $185$
entities / event comp. |  |  |  |
participant | $12,676$ | n/a | n/a | $5937$
time | $2412$ | n/a | n/a | $1439$
location | $2205$ | n/a | n/a | $566$
event coreference links | $26,712$ | $29,398$ | $106,479$ | $145,272$
within-document | $1636$ | $14,218$ | $2344$ | $2662$
within-subtopic | $24,816$ | $15,180$ | $3972$ | $4561$
cross-subtopic | $260$ | $0$ | $100,163$ | $138,049$
cross-topic | $0$ | $0$ | $0$ | $0$
Table 1: Comparison of annotations in several cdevcr corpora. Values for fcc
refer to our cleaned version of the original corpus. fcct is our token-level
reannotation. Figure 2: Cluster size distribution in cdevcr corpora.

Table 1 presents further insights into these corpora. There, we report the
total number of event coreference links in each corpus and categorize them by
type. Note that in ecbp and gvc, nearly all cross-document links are of the
within-subtopic kind whereas fcc focused on annotating cross-subtopic links.
The stark contrast in the number of coreference links between fcc and ecbp/gvc
can be attributed to the facts that (1) the number of coreference links grows
quadratically with the number of mentions in a cluster and (2) fcc contains
clusters with more than 100 mentions, see Figure 2.

While the annotation design of each of these corpora has had different foci,
they share commonalities. The structure of each corpus can be framed as a
hierarchy with three levels: there are one or more topics/event types which
each contain subtopics/event instances which each contain multiple documents.
Both ecbp and gvc annotate event mentions on the token level in a similar
manner. Since fcc is the only cdevcr corpus missing token level event mention
annotations, we add these annotations in this work to produce the fcct corpus
(see Section 4). With this change made, it is technically and theoretically
possible to examine these cdevcr corpora jointly.

###  2.4 Systems

We here summarize the principles of cdevcr systems, followed by the state-of-
the-art systems for each cdevcr corpus.

####  2.4.1 System Principles

Given a collection of event mentions, a discrete or vectorized representation
needs to be created for each mention so that the mentions can be clustered.
Following the definition of the cdevcr task, a representation should contain
information on the action, participants, time, and location of the event
mention. This information may be scattered throughout the document and needs
to be extracted first. To do this, cdevcr may preprocess documents via srl,
temporal tagging or entity linking.

Two general strategies exist for computing the distances between mentions
which are needed for clustering: representation learning and metric learning
Hermans, Beyer, and Leibe (2017). Representation learning approaches produce a
vector representation for each event mention independently. The final event
clustering is obtained by computing the cosine distance between each vector
pair, followed by agglomerative clustering on the resulting distance matrix.
Most approaches belong to the group of conceptually simpler metric learners
which predict the semantic distance between two mentions or clusters based on
a set of features. By applying the metric on all $\binom{n}{2}$ pairs for $n$
mentions, a distance matrix is obtained which is then fed to a clustering
algorithm. Any probabilistic classifier or regression model may be used to
obtain the mention distances. Metric learning approaches can be further
divided into mention pair approaches which compute the distance between each
mention pair once and cluster pair approaches which recompute cluster
representations and distances after each cluster merge. Computing the distance
between all mention pairs can be a computationally expensive process. Some
metric learner approaches therefore perform a separate document preclustering
step to break down the task into manageable parts. The metric learning
approach is then applied on each individual cluster of documents and its
results are combined to produce the final coreference clustering.

|  |  cr2020 |  me2020 |  ba2019 |  ke2018 |  vo2016 |  cy2015 |  ya2015 |  le2012 |  mi2018 |  vo2018 |  ours
---|---|---|---|---|---|---|---|---|---|---|---|---
Prepro- | Fact KB entity linking |  |  |  |  | ✓ |  |  |  | ✓ | ✓ | ✓
cessing | Lexical KB entity linking |  |  |  |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|  srl |  | ✓ | ✓ |  | ✓ |  | ✓ | ✓ |  | ✓ | ✓
| Temporal tagging |  |  |  |  | ✓ |  |  |  |  | ✓ | ✓
Mention or | Bag of words |  |  |  | ✓ |  |  |  | ✓ |  |  |
Document | TF–IDF | ✓ |  |  | ✓ |  |  | ✓ |  |  | ✓ | ✓
Represent- | Word emb. simple | ✓ | ✓ | ✓ | ✓ |  |  | ✓ |  |  |  |
ations | Word emb. contextual |  | ✓ | ✓ |  |  |  |  |  |  |  | ✓
| Character embeddings |  | ✓ | ✓ |  |  |  |  |  |  |  |
Features | Entity Coreference |  | ✓ | ✓ |  |  | ✓ |  | ✓ |  |  |
| Ling. properties of mention | ✓ | (✓) | (✓) |  |  |  |  | ✓ |  |  |
| Paraphrase detection |  | ✓ |  |  |  |  |  |  |  |  |
| Temporal distance |  |  |  |  | ✓ |  |  |  | ✓ | ✓ | ✓
| Spatial distance |  |  |  |  |  |  |  |  |  |  | ✓
| Compare discrete reprs. |  |  |  |  | ✓ | ✓ | ✓ | ✓ | ✓ |  | ✓
| Compare vectorized reprs. | (✓) | (✓) | (✓) | ✓ |  |  | ✓ | ✓ |  | ✓ | ✓
| Compare w.r.t. lexical KB |  |  |  |  | ✓ | ✓ |  | ✓ | ✓ |  |
| Discourse-related features |  |  |  | ✓ |  | ✓ |  |  |  |  |
Table 2: Preprocessing steps, representations and features used by cdevcr
systems. We mark implictly learnt neural features with (✓).

Common types of features used by cdevcr systems are text similarity features
(string matching between event mention actions), semantic features (the
temporal distance between mentions), features using world knowledge (the
spatial distance between the locations of mentions) or discourse features (the
position of a mention in the document) as well as latent neural
features.222See Lu and Ng (2018) for more examples of common features. Table 2
shows the types of features which existing cdevcr systems rely on.

####  2.4.2 Notable cdevcr Systems

System | Target | Mention dist. | Approach | Precluster | Learning | Clustering
---|---|---|---|---|---|---
| corpus | computation |  | documents? | approach | technique
cr2020 |  ecbp | classifier | mention pair | yes |  mlp | transitive closure
me2020 |  ecbp | classifier | cluster pair | yes |  mlp | agglomerative
ba2019 |  ecbp | classifier | cluster pair | yes |  mlp | agglomerative
ke2018 |  ecbp | representation | mention pair | no |  mlp autoenc. | agglomerative
vo2016 |  ecbp | classifier | mention pair | no | rule-based | transitive closure
cy2015 |  ecbp | classifier | mention pair | yes & no | decision tree | transitive closure
ya2015 |  ecbp | classifier | mention pair | no | logistic regr. | HDDCRP
le2012 |  ecbp | classifier | cluster pair | yes | linear regr. | agglomerative
mi2018 |  gvc | classifier | document pair | n/a | rule-based | agglomerative
vo2018 |  gvc | classifier | document pair | n/a | rule-based | transitive closure
ours | n/a | classifier | mention pair | no | XGBoost | agglomerative
Table 3: Core principles of several cdevcr systems

Table 3 shows a comparison of the core principles of several cdevcr systems in
terms of their mention distance computation, learning approach and more. We
compare the systems of Cremisini and Finlayson 2020 (cr2020), Meged et al.
2020 (me2020), Barhom et al. 2019 (ba2019), Kenyon-Dean, Cheung, and Precup
2018 (ke2018), Vossen and Cybulska 2016 (vo2016), Cybulska and Vossen 2015
(cy2015), Yang, Cardie, and Frazier 2015 (ya2015), Lee et al. 2012 (le2012),
Mirza, Darari, and Mahendra 2018 (mi2018), and Vossen 2018 (vo2018). We
emphasize notable systems for each corpus.

At the time of writing, the state-of-the-art system on ecbp is me2020, a
cluster-pair approach in which a mlp is trained to jointly resolve entity and
event coreference. It is an extension of ba2019, adding paraphrasing features.
The system performs document preclustering prior to the coreference resolution
step.

gvc was used in SemEval 2018 Task 5 which featured a cdevcr subtask Postma,
Ilievski, and Vossen (2018). The best performing system was mi2018 which
clusters documents using the output of a wsd system, person and location
entities and event times. Based on the assumption that each document mentions
up to one event of each event type, the system puts all event mentions of same
event type in the same cross-document event coreference cluster. Due to the
nature of the shared task, the system is specialized on a limited number of
event types. vo2016 and vo2018 are based on the NewsReader pipeline which
contains several preprocessing stages to perform event mention detection,
entity linking, wsd and more. Using this information, one rule-based system
was defined per corpus (ecbp and gvc) which is tailored to the topics and
annotations present in the respective corpus.

The fcc is the most recently released corpus of the three. We are not aware of
any publications reporting results for this corpus.

####  2.4.3 On the Application of Event Mention Detection

With respect to the two steps a cdevcr system needs to perform (event mention
detection and event coreference resolution), several authors have decided to
omit the first step and work on gold mentions alone Cybulska and Vossen
(2015); Kenyon-Dean, Cheung, and Precup (2018); Barhom et al. (2019); Meged et
al. (2020) which simplifies the task and system development. Systems which
include a mention detection step Lee et al. (2012); Yang, Cardie, and Frazier
(2015); Vossen and Cybulska (2016); Choubey and Huang (2017); Vossen (2018);
Cremisini and Finlayson (2020) are more faithful to the task but risk
introducing another source of error. Compared to using gold event mentions,
performance drops from $20\text{\,}\mathrm{\textup{pp}}$ CoNLL F1 Vossen and
Cybulska (2016) to $40\text{\,}\mathrm{\textup{pp}}$ CoNLL F1 Cremisini and
Finlayson (2020) have been observed on ecbp. Vossen and Cybulska derive from
these results that event detection “is the most important factor for improving
event coreference” (Vossen and Cybulska, 2016, p. 518).

We think that the root cause for these losses in performance are not the event
detection approaches themselves but rather intentional limitations in the
event mention annotations of cdevcr corpora. We take the ecbp corpus as an
example. Based on the event definition stated in the annotation guidelines,
several hundred event mentions would qualify for annotation in each news
document. To keep the annotation effort manageable, only event mentions of the
document’s seminal event (the main event the article is reporting about) and
mentions of other events in the same sentence were annotated (Cybulska and
Vossen, 2014a, p. 9). Conversely, the corpus contains a large amount of valid
event mentions which were deliberately left unannotated.333In
$88\text{\,}\mathrm{\char 37\relax}$ of all sentences in ecbp, no event
actions are annotated. A mention detection system will (unaware of this fact)
predict these event mentions anyway and will be penalized for producing false
positive predictions. In the subsequent mention clustering step, coreference
chains involving these surplus mentions increase the risk of incorrect cluster
merges between valid mentions and will overall lead to lower precision. A
general purpose mention detection system may perform poorly on the fcc and gvc
corpora in similar fashion. For these corpora, affordability of the annotation
process was achieved by restricting event mentions to certain action types,
which lowers the overall number of to-be-annotated event mentions.

We therefore think that, as long as no cdevcr corpus exists in which every
single event mention is annotated, event detection and event coreference
resolution should be treated separately, meaning event coreference resolution
performance should be reported on gold event mentions. For this reason, and
because of the different approaches for limiting the number of event mentions
in each of the three corpora, we perform all experiments on gold event mention
spans in this work.

##  3 Related Work

Prior work has examined feature importance in cdevcr systems. Cybulska and
Vossen (2015) tested different combinations of features with a decision tree
classifier on ecbp. They find that system performance majorly stems from a
lemma overlap feature and that adding discourse, entity coreference and wsd
features improves BLANC F1 by only $1\text{\,}\mathrm{\textup{pp}}$. Cremisini
and Finlayson (2020) conducted a study in which they built a feature-based
mention pair approach for ecbp to gain deeper insights into the importance of
features and on the performance impact of document preclustering. Amongst four
features (fastText Bojanowski et al. (2017) word embedding similarity between
event actions, event action word distribution, sentence similarity and event
action part-of-speech comparison), the embedding similarity feature was found
to be the most important by far. The use of document preclustering caused an
improvement of $3\text{\,}\mathrm{\textup{pp}}$ CoNLL F1, leading Cremisini
and Finlayson to encourage future researchers in this field to report
experiments with and without document preclustering.

Our work significantly deepens these earlier analyses. Since research on
cdevcr systems has so far only focused on resolving cdevc in individual
corpora, we tackle the issue of generalizability across multiple corpora. We
use a broader set of features and by comparing two cdevcr approaches, while
previous work focused on the ecbp corpus using the aforementioned smaller sets
of features. We (1) develop a general feature-based cdevcr system, (2) apply
it on each of the corpora mentioned above, and (3) analyze the information
sources in each corpus which are most informative to cdevc. We thereby provide
the first comparative study of cdevcr approaches, paving the way for general
resolution of cdevc which will aid downstream multi-document tasks.

##  4 fcc Reannotation

We reannotate the fcc to improve its interoperability with ecbp and the
gvc.444We additionally conducted an annotation of the missing document
publication dates in ecbp, but found that dates could only be manually
extracted in half of the corpus documents. We therefore did not include these
annotations in our experiments. More details on this annotation effort are
reported in LABEL:sec:app_annotation_ecbp.

The fcc was recently introduced by Bugert et al. (2020) as a cdevcr corpus
with sentence-level event mention annotations (see Section 2.3). We reannotate
all event mentions on token level, add annotations of event components, and
annotate additional event mentions to produce the fcct corpus (T for token
level). The following sections cover our annotation approach, inter-annotator
agreement and the properties of the resulting corpus.

###  4.1 Annotation Task Definition

In the original fcc annotation, crowd annotators were given a predefined set
of events and sentences of news articles to work on. Each sentence had to be
marked with the subset of events referenced in the sentence. We take these
sentences and annotate the action mention of each referenced event on token
level. For each event, we additionally annotate the corresponding
participants, time, and location mentions appearing in the same sentence as
the action mention. To achieve maximum compatibility with existing corpora, we
adopted the ecbp annotation guidelines Cybulska and Vossen (2014a).555For
details and examples, please refer to the guidelines published at
https://github.com/UKPLab/cdcr-beyond-corpus-tailored We distinguish between
different subtypes of participants (person, organization, etc.), time, and
location as done by Cybulska and Vossen (2014a). We do not differentiate
between action types since all events (pre-)annotated in fcc should belong to
the OCCURRENCE type (see Cybulska and Vossen, 2014a, p.14). We do not annotate
(cross-document) entity coreference. We do annotate a rudimentary kind of
semantic roles which we found are crucially missing in ecbp: we instruct
annotators to link mentions of participants, time, and location to their
corresponding action mention.

While developing the guidelines, we noticed cases where sentence-level
mentions are evidently easier to work with than token-level mentions. For
example, enumerations or aggregated statements over events (such as
"Switzerland have won six of their seven meetings with Albania, drawing the
other.") are difficult to break down into token-level event mentions. Cases
like these are not covered by the ecbp annotation guidelines and were removed
in the conversion process. A similar issue is caused by coordinate structures
such as "Germany beat Algeria and France in the knockout stages" where two
football match events are referenced by the same verb. To handle these cases,
we annotated two separate event mentions sharing the same action mention
("beat"). Since superimposed mention spans are not supported by coreference
evaluation metrics, we additionally provide a version of the corpus in which
these mentions are removed.

In fcc, crowdworkers identified a further $1100$ sentences which mention one
or more football-related events outside of the closed set of events they were
provided with during the annotation. These event mentions were left
unidentified by Bugert et al. (2020). We instructed annotators to manually
link each event mention in this extra set of sentences to a database of 40k
international football matches666https://www.kaggle.com/martj42/international-
football-results-from-1872-to-2017/version/5 and again marked and linked the
token spans of actions, participants, times, and locations.

Annotators were given the option to mark sentences they found unclear or which
were incorrectly annotated by crowdworkers in the original dataset. We
manually resolved the affected sentences on a case-by-case basis.

###  4.2 Annotation Procedure and Results

The annotation was carried out with the INCEpTION annotation tool Klie et al.
(2018). We trained two student annotators on a set of 10 documents. The
students were given feedback on their work and afterwards annotated a second
batch of 22 documents independently. Table 4 shows the inter-annotator
agreement on this second batch. We report Krippendorff’s $\alpha_{U}$
Krippendorff (1995) which measures the agreement in span overlap on character
level as the micro average over all documents.

action mentions | 0.80
---|---
participants, time, location (spans only) | 0.67
participants, time, location (incl. subtype) | 0.57
Table 4: Inter-annotator agreement ($\alpha_{U}$)

For the annotation of action mention extents, which is the most important step
in our re-annotation effort, we reach 0.80 $\alpha_{U}$, indicating good
reliability between annotators Carletta (1996); Artstein and Poesio (2008).
The agreement for the annotation of participants, time, and location is lower
at 0.57 $\alpha_{U}$. We found that this mostly stems from the annotation of
participants: In the guidelines, we specify that annotators should only mark
an entity as a participant of an event if it plays a significant role in the
event action. The larger and more coarse an event is, the more difficult this
decision becomes for annotators. One such case is shown in Example 4.1 where
it is debatable if "Christian Teinturier" is or is not significantly involved
in the tournament event.

######  Example 4.1.

"Earlier today, French Football Federation vice-president Christian Teinturier
said if there was any basis to the reports about Anelka then he should be sent
home from the tournament immediately."

A second reason is that we do not annotate entity coreference, so only a
single entity mention is meant to be annotated for each entity participating
in an event. In case the same entity appears twice in a sentence, we instruct
annotators to choose the more specific description. If the candidates are
identical in surface form, annotators are meant to choose the one closer (in
word distance) to the event action. There remains a level of subjectivity in
these decisions, leading to disagreement.

Overall, we concluded that the annotation methodology produced annotations of
sufficient quality. The remaining 419 documents were divided among both
annotators. The corpus re-annotation required 120 working hours from
annotators (including training and the burn-in test). We fixed a number of
incorrect annotations in the crowdsourced fcc corpus. For example, we removed
several mentions of generic events ("winning a World Cup final is every
player’s dream") which were incorrectly marked as referring to a concrete
event.

Table 1 (on table 1) shows the properties of the resulting fcct corpus
alongside ecbp, gvc and our cleaned version of the sentence-level fcc corpus.
Compared to the original fcc corpus, our token-level reannotation offers
$50\text{\,}\mathrm{\char 37\relax}$ more event mentions and twice as many
annotated events. With respect to the srl annotations, we analyzed how
frequently event components of each type were attached to action mentions. We
found that $95.7\text{\,}\mathrm{\char 37\relax}$ of action mentions have at
least one participant attached, $41.6\text{\,}\mathrm{\char 37\relax}$ at
least one time mention and $15.8\text{\,}\mathrm{\char 37\relax}$ at least one
location mention. We mentioned earlier that cases exist where two or more
action mentions share the same token span. $340$ out of all $3563$ annotated
event mentions in fcct fall into this category. A further $154$ event mentions
did not have a counterpart in the event database (such as matches from
national football leagues). We jointly assigned these mentions to the
coreference cluster other_event.

By creating the fcct, a reannotation and extension of fcc on token level, we
provide the first cdevcr corpus featuring a large body of cross-subtopic event
coreference links which is compatible with the existing ecbp and gvc
corpora.777The fcct is available at https://tudatalib.ulb.tu-
darmstadt.de/handle/tudatalib/2305 This greatly expands the possibilities for
cdevcr research over multiple corpora, as we will demonstrate in Sections 6 to
8.

##  5 Defining a General cdevcr System

Recent cdevcr approaches such as neural end-to-end systems or cluster-pair
approaches were shown to offer great performance Barhom et al. (2019), yet
their black box nature and their complexity makes it difficult to analyze
their decisions. In particular, our goal is to identify which aspects of a
cdevcr corpus are the strongest signals for event coreference which cannot be
adequately investigated with recent cdevcr systems. We therefore propose a
conceptually simpler mention pair cdevcr approach which uses a broad set of
handcrafted features for resolving event coreference in different
environments. We thus focus on developing an interpretable system, whereas
reaching state-of-the-art performance is of secondary importance. This section
explains the inner workings of the proposed system.

###  5.1 Basic System Definition

We resolve cdevc by considering pairs of event mentions. At training time, we
sample a collection of training mention pairs. For each pair, we extract
handcrafted features with which we train a probabilistic binary classifier
that learns the coreference relation between a pair (coreferring or not
coreferring). The classifier is followed by an agglomerative clustering step
which uses each pair’s coreference probability as the distance matrix. At
prediction time, all $\binom{n}{2}$ mention pairs are being classified without
prior document preclustering. For the reasons outlined in Section 2.4.3, we
choose to omit the mention detection step and work with the gold event
mentions of each corpus throughout all experiments.

###  5.2 Pair Generation for Training

We explain three issues which arise when sampling training mention pairs and
how we address them in our system.

The straightforward technique for sampling training pairs is to sample a fixed
number of all possible coreferring and non-coreferring mention pairs. Due to
the sparsity of the cdevcr relation, the resulting set of pairs would mostly
consist of non-coreferring pairs when using this technique, with the majority
of coreferring pairs left unused. This issue has been partially addressed in
the past with weighted sampling to increase the ratio of coreferring pairs Lee
et al. (2012); Barhom et al. (2019).

We identified a second issue, namely the underrepresentation of mention pairs
from the long tail, which weighted sampling does not address: We previously
established that cluster sizes in corpora are imbalanced (see Figure 2). If
all $\binom{n}{2}$ coreferring pairs are generated for each cluster, the
generated pairs will largely consist of pairs from the largest clusters.888For
example, sorting all event clusters of the ecbp training split by size, the
largest cluster with 38 mentions would produce more coreferring pairs than the
smallest $50\text{\,}\mathrm{\char 37\relax}$ of clusters produce together.
Manual inspection revealed that the variation in how events are expressed is
limited, with large clusters exhibiting many action mentions with
(near-)identical surface forms.999This is particularly pronounced for fcct
where events of football tournaments are mostly mentioned by ”tournament” or
”World Cup”. Consequentially, with common pair generation approaches, there is
a high chance of generating many mention pairs which carry little information
for the classifier, while mention pairs from clusters in the long tail are
unlikely to be included.

Another issue we have not yet seen addressed in related work is the
distribution of link types in the body of sampled pairs: in terms of the
number of mention pair candidates available for sampling, the cross-topic link
candidates strongly outnumber the cross-subtopic link candidates who in turn
strongly outnumber the within-subtopic link candidates (and so on) by nature
of combinatorics. This particularly concerns the large body of non-coreferring
pairs. An underrepresentation of one of these types during training could
cause deficiencies for the affected type at test time, hence care must be
taken to achieve a balanced sampling.

We address these three issues as follows: (1) We use the distribution of
cluster sizes in the corpus to smoothly transition from generating all
$\binom{n}{2}$ coreferring pairs for the smallest clusters to generating
$(n-1)\cdot c$ pairs for the largest clusters, where $c\in\mathbb{R}^{+}$ is a
hyperparameter. (2) For each type of coreference link (within-document,
within-subtopic, etc.) we sample up to $k$ non-coreferring mention pairs for
each coreferring pair previously sampled for this type. Details on the
sampling approach are provided in LABEL:sec:app_mention_pair_generation.

###  5.3 Features and Preprocessing

Related work has demonstrated a great variety in the representations and
features used to resolve cross-document event coreference (see Table 2 on
table 2), yet it remains unclear which features contribute the most to the
coreference resolution performance on each of the three corpora. We therefore
chose to implement a series of preprocessing steps and feature extractors
which cover the majority of features used in previous systems.

####  5.3.1 Preprocessing

We perform lemmatization and temporal expression extraction with CoreNLP
Manning et al. (2014); Chang and Manning (2012), using document publication
dates to ground temporal expressions for gvc and fcct. We manually converted
complex TIMEX expressions into date and time (so that 2020-01-01TEV becomes
2020-01-01T19:00). For ecbp and gvc where participant, time, and location
mentions are not linked to the action mention, we applied the SRL system by
Shi and Lin (2019) as implemented in AllenNLP Gardner et al. (2018). We map
spans with labels ARGM-DIR or ARGM-LOC to the location, ARGM-TM to the time
and ARG0 or ARG1 to the participants of each respective event mention. For all
corpora we perform entity linking to DBPedia101010https://dbpedia.org – We
used the latest release from April 1st, 2020. via DBPedia Spotlight Mendes et
al. (2011).

####  5.3.2 Features

The list of handcrafted mention pair features includes (1) string matching on
action mention spans, (2) cosine similarity of TF–IDF vectors for various text
regions, (3) the temporal distance between mentions, (4) the spatial distance
between event actions based on DBPedia, and (5) multiple features comparing
neural mention representations. These include representations of action
mentions, embeddings of the surrounding sentence and embeddings of Wikidata
entities which we obtained via the DBPedia entity linking step. Details on
each feature are reported in Appendix C: Mention Pair Features.

###  5.4 Implementation Details

We implemented the system using Scikit-learn Pedregosa et al. (2011). To
obtain test predictions, we applied the following steps separately for each
corpus: We perform feature selection via recursive feature elimination Guyon
et al. (2002) on the respective development split. We use a random forest
classifier tasked with classifying mention pairs as “coreferring” / “not
coreferring” as an auxiliary task for this stage. We then identified the best
classification algorithm to use as the probabilistic mention classifier. We
tested logistic regression, a mlp, a probabilistic SVM and XGBoost Chen and
Guestrin (2016). We tuned the hyperparameters of each classifier via repeated
6-fold cross-validation for 24 hours on the respective training
split.111111Details on the hyperparameter optimization procedure are provided
in LABEL:sec:app_hyperopt.. Using the best classifier, we optimized the
hyperparameters of the agglomerative clustering step (i.e., the linkage
method, cluster criterion and threshold) for another 24 hours on the training
split. For each experiment, we train five models with different random seeds
to account for non-determinism. At test time we evaluate each of the five
models and report the mean of each evaluation metric.121212N.B. This applies
to every result originating from this model throughout this work. We will
therefore not point this out further.

##  6 Generalizability of cdevcr Systems

We train and test two cdevcr systems and several baselines separately on the
ecbp, fcct, and gvc corpora to evaluate how flexibly these systems can be
applied to different corpora (i.e., whether their overall design is
sufficiently general for resolving cdevc in each corpus). The two systems are
our proposed general system (see Section 5) and the system of Barhom et al.
(2019) (ba2019). We chose ba2019 because it is the best-performing ecbp system
for which an implementation is available.

In Sections 6.1 and 6.2, we define evaluation metrics and baselines. We then
establish the performance of the feature-based system (Section 6.3) on the
three corpora, including a detailed link-level error analysis which we can
only perform with this system. In Section 6.4, we explain how we apply ba2019
and compare its results to those of the feature-based system, analyzing the
impact of document preclustering on the coreference resolution performance in
the process.

###  6.1 Evaluation Metrics

Related work on cdevcr has so far only scored predictions with the CoNLL F1
Pradhan et al. (2014) metric (and its constituent parts MUC Vilain et al.
(1995), CEAFe Luo (2005) and B3 Bagga and Baldwin (1998)). We additionally
score predictions with the LEA metric Moosavi and Strube (2016). LEA is a
link-based metric which, in contrast to other metrics, takes the size of
coreference clusters into account. The metric penalizes incorrect merges
between two large clusters more than incorrect merges of mentions from two
singleton clusters. As we have shown that cluster sizes in cdevcr corpora vary
considerably (see Figure 2) this is a particularly important property. LEA was
furthermore shown to be more discriminative than the established metrics MUC,
CEAFe, B3 and CoNLL F1 Moosavi and Strube (2016).

###  6.2 Baselines

We report the two commonly chosen baselines lemma and lemma-$\delta$ as well
as a new lemma-time baseline based on temporal information:

  1. 1.

lemma: Action mentions with identical lemmas are placed in the same
coreference cluster.

  2. 2.

lemma-$\delta$: Document clusters are created by applying agglomerative
clustering with threshold $\delta$ on the TF–IDF vectors of all documents,
then lemma is applied to each document cluster. For hyperparameter $\delta$,
we choose the value which produces the best LEA F1 score on the training
split.

  3. 3.

lemma-time: A variant of lemma-delta based on document-level temporal
information. To obtain the time of the main event described by each document,
we use the first occurring temporal expression or alternatively the
publication date of each document. We create document clusters via
agglomerative clustering where the distance between two documents is defined
as the difference of their dates in hours. We then apply lemma to each
document cluster. Here, the threshold $\delta$ represents a duration which is
optimized as in lemma-delta.

###  6.3 Establishing the Feature-based System

We run in-dataset experiments to determine the performance of the feature-
based cdevcr approach on each individual corpus. Details on the splits used
for each corpus are reported in LABEL:sec:app_corpus_splits. When generating
mention pairs for training, we undersample coreferring pairs (see Section 5.2)
using hyperparameters $c=8$ and $k=8$. In experiments involving fcct, we use
$c=2$ and $k=8$ to compensate for the large clusters in this corpus. Details
on the choice of hyperparameters are provided in
LABEL:sec:app_mention_pair_generation. On all three corpora, the best mention
pair classification results were obtained with XGBoost which led us to use it
for all subsequent experiments with this system.

####  6.3.1 Mention Clustering Results

Corpus | System | CoNLL | LEA
---|---|---|---
|  | F1 | P | R | F1
ecbp | lemma | $61.9053$ | $42.8045$ | $43.4982$ | $43.1486$
| lemma-$\delta$ | $74.3987$ | $71.5428$ | $53.6916$ | $61.3449$
| feature-based | $74.8099$ | $67.8531$ | $55.0546$ | $60.7871$
fcct | lemma | $42.9238$ | $38.3587$ | $19.8697$ | $26.1788$
| lemma-$\delta$ | $42.9238$ | $38.3587$ | $19.8697$ | $26.1788$
| lemma-time | $39.7932$ | $36.8327$ | $14.2433$ | $20.5427$
| feature-based | $54.2654$ | $30.4189$ | $60.4073$ | $39.7645$
gvc | lemma | $33.7975$ | $08.805,42$ | $29.7228$ | $13.586$
| lemma-$\delta$ | $50.2972$ | $43.8354$ | $28.6605$ | $34.6598$
| lemma-time | $51.4669$ | $53.813$ | $27.2605$ | $36.1887$
| feature-based | $59.3876$ | $56.5137$ | $38.1787$ | $45.5702$
Table 5: In-dataset cdevcr results of baselines and the feature-based system.
The full set of metrics is reported in Appendix A: Full In-Dataset cdevcr
Results.

The results are shown in Table 5. For brevity, we only report cross-document
performance. It is obtained by applying the evaluation metrics on modified
gold and key files in which all documents were merged into a single meta
document Upadhyay et al. (2016).

As was initially shown by Upadhyay et al. (2016), the lemma-$\delta$ baseline
is a strong baseline on the ecbp corpus. The feature-based system performs on
par with this baseline.

For fcct, the optimal $\delta$ produces a single cluster of all documents
which leads to identical results for the lemma and lemma-$\delta$ baselines.
This is a direct consequence of the fact that in this corpus, the majority of
event coreference links connect documents from different subtopics. In
contrast to ecbp, where preclustering documents by textual content produces
document clusters which are near-identical to the gold subtopics Barhom et al.
(2019); Cremisini and Finlayson (2020), such a strategy is disadvantageous for
fcct because the majority of coreference links would be irretrievably lost
after the document clustering step. The lemma-time baseline performs worse on
fcct than lemma-$\delta$, indicating that the document publication date is
less important than the document content. The feature-based approach
outperforms the baselines on fcct, showing higher recall but lower precision
which indicates a tendency to overmerge clusters.

The lemma baselines perform worse on gvc than on ecbp in absolute numbers
which can be attributed to the fact that Vossen et al. (2018) specifically
intended to create a corpus with ambiguous event mentions. Furthermore, the
baseline results show that knowing about a document’s publication date is
worth more than knowing its textual content (at least for this corpus). The
feature-based system mostly improves over the baselines in terms of recall.

Another noteworthy aspect in Table 5 are the score differences between CoNLL
F1 and LEA F1. In the within-document entity coreference evaluations performed
by Moosavi and Strube (2016) alongside the introduction of the LEA metric, the
maximum difference observed between CoNLL F1 and LEA F1 were roughly
$10\text{\,}\mathrm{\textup{pp}}$. Our experiments exhibit differences of
$14\text{\,}\mathrm{\textup{pp}}$ for systems and up to
$20\text{\,}\mathrm{\textup{pp}}$ for baselines due to imbalanced cluster
sizes in cdevcr corpora.

####  6.3.2 Mention Pair Classifier Results

Link type |  ecbp |  fcct |  gvc
---|---|---|---
| Links | P | R | F1 | Links | P | R | F1 | Links | P | R | F1
within-document |  $10.751$ $\mathrm{k}$ | $57.633,581,802,981,96$ | $55.853,211,009,174,31$ | $56.698,234,812,845,95$ |  $6.567$ $\mathrm{k}$ | $53.495,482,645,852,89$ | $56.160,220,994,475,14$ | $54.790,474,995,984,57$ |  $7.239$ $\mathrm{k}$ | $69.135,489,240,901,89$ | $30.599,128,540,305,01$ | $42.421,888,648,205,625$
within-subtopic |  $83.191$ $\mathrm{k}$ | $64.997,204,466,710,13$ | $54.511,349,306,431,28$ | $59.294,032,508,229,01$ |  $21.481$ $\mathrm{k}$ | $51.997,852,007,069,3$ | $48.443,223,443,223,44$ | $50.155,364,142,675,15$ |  $9.650$ $\mathrm{k}$ | $70.934,702,905,947,13$ | $28.869,241,507,677,99$ | $41.034,359,968,399,56$
cross-subtopic |  $86.805$ $\mathrm{k}$ | $0.0$ | $0.0$ | $0.0$ |  $518.486$ $\mathrm{k}$ | $53.734,275,058,928,76$ | $36.275,491,690,253,29$ | $43.303,365,527,796,29$ |  $434.836$ $\mathrm{k}$ | n/a | n/a | n/a
Table 6: Mention pair classifier performance of the feature-based system for
each cross-document coreference link type. “Coreferring” is used as the
positive class. The “Links” column shows the total number of links
(coreferring and non-coreferring) per type and corpus based on which P/R/F1
were calculated.

To evaluate the probabilistic mention pair classifier in isolation for
different corpora and coreference link types, we compute binarized recall,
precision and F1 with respect to gold mention pairs.131313Note that this
approach puts higher weight on large clusters, as these produce a greater
number of mention pairs. It is nonetheless the only evaluation approach we are
aware of which permits analyzing performance per link type. Link-based
coreference metrics such as MUC Vilain et al. (1995) cannot be used as a
replacement, as these (1) require a full clustering opposed to one score per
pair and (2) by design abstract away from individual links in a system’s
response. The results are reported in Table 6. The gvc test split does not
contain coreferring cross-subtopic event coreference links, therefore these
cells are marked with “n/a”. Five links of this type are present in the ecbp
test split of which none were resolved by the in-dataset ecbp model. For fcct
and gvc, the performance in resolving within-document, within-subtopic and
cross-subtopic event coreference links decreases gradually from link type to
link type. This suggests that the greater the distance covered by an event
coreference link is in terms of the topic-subtopic-document hierarchy of a
corpus, the more difficult it becomes to resolve it correctly.

####  6.3.3 Error Analysis

Ex. |  Mention context A |  Mention context B
---|---|---
1 |  fcct, cross-subtopic link, false positive
|  Zlatko Dalic’s men will be back at the Luzhniki Stadium on Sunday to face France, who glanced a 1–0 victory over Belgium on Tuesday thanks to a Samuel Umtiti header. |  Belgium claims third place with a 2–0 win over England
2 |  gvc, cross-subtopic link, false positive
|  A 66-year-old man was shot in his leg and grazed in his arm early Monday while sitting on a park bench in Charles Village, police said. |  The victims – except an 18-year- old man who refused medical attention after a bullet grazed his left leg – were taken to UAMS Medical Center.
3 |  ecbp, cross-subtopic link, false positive
|  Tuesday, July 2, 2013. A strong earthquake struck Indonesia’s Aceh province on Tuesday, killing at least one person and leaving two others missing. |  THOUSANDS of frightened residents flooded makeshift refugee camps in Indonesia’s West Papua province today after two powerful earthquakes flattened buildings and killed at least one person.
Table 7: Notable misclassifications found during manual error analysis

To gain a better understanding of the system’s limitations, we manually
analyzed predictions of the mention pair classifier. We analyzed five false
positive and five false negative cases for each link type and corpus (roughly
90 mention pairs in total).

We found that textual similarity between action mentions accounts for a large
portion of mistakes on the ecbp and gvc corpora: unrelated but similar action
mentions caused false positive cases and vice-versa, coreferring but merely
synonymous action mentions led to false negative cases. The fcct model did not
resolve coreference well between mentions like "the tournament", "this year’s
cup" and "2018 World Cup", contributing to false negative cases. Also, the
model showed a tendency of merging event mentions prematurely when the action
mention and at least one participant matched (see example 1 in Table 7) which
would explain the high recall and low precision results seen in Table 5. For
all three models, we noticed misclassifications when a sentence contained
multiple event mentions (see example 2 in Table 7). In the given example, it
is likely that information from the unrelated "shot in his leg" mention leaked
into the representation of the "grazed" event which contributed to the
incorrect classification. For ecbp, we noticed that the lack of document
publication date information makes certain decisions considerably harder. For
example, the earthquake events seen in example 3 are unrelated and took place
four years apart. While one could come to this conclusion with geographic
knowledge alone (the provinces lie on opposite sides of Indonesia), date
information would have made this decision easier.

It is reassuring that many of the shortcomings we found would be fixable with
a cluster-level coreference resolution approach, (joint) resolution of entity
coreference, injection of corpus-specific world knowledge (a football match
must take place between exactly two teams, etc.) or with annotation-specific
knowledge (for example knowledge of Vossen et al. (2018)’s domain model for
the annotation of gvc). Our system could be improved by incorporating these
aspects, however at the cost of becoming more corpus-specific and less
interpretable.

###  6.4 Comparison to Barhom et al. (2019)

We test an established cdevcr system, the former state-of-the-art neural
cdevcr approach of Barhom et al. (2019) (ba2019), for its generalization
capabilities on the three corpora.

####  6.4.1 Experiment Setup

We trained one model of ba2019 for each corpus. ba2019 can resolve event and
entity coreference jointly. For the sake of comparability we only use the
event coreference component of this system for all experiments since fcct and
gvc do not contain entity coreference annotations. We replicate the exact data
preprocessing steps originally used for ecbp on fcct and gvc. This includes
the prediction of semantic roles with the SwiRL srl system Surdeanu et al.
(2007). The fcct corpus mainly consists of cross-subtopic event coreference
links (see Section 4.2). The trainable part of the ba2019 system (mention
representation and agglomerative mention clustering) is meant to be trained
separately on each subtopic of a corpus. This is because at prediction time,
the partitioning of documents into subtopics will already be handled by a
foregoing and separate document preclustering step. In order not to put ba2019
at a disadvantage for fcct, we train it on three large groups of documents
which correspond to the three football tournaments present in the fcct
training split instead of training it on the actual fcct subtopics. For this
corpus, we also apply undersampling with the same parameters as for the
feature-based system.

####  6.4.2 Options for Document Preclustering

Clustering documents by their content before applying mention-level event
coreference boosts performance on the ecbp corpus Cremisini and Finlayson
(2020); Barhom et al. (2019); Choubey and Huang (2017); Upadhyay et al.
(2016). As was recommended by Cremisini and Finlayson (2020), we report
results for several document preclustering strategies to better distinguish
the source of performance gains or losses. We compare (1) no preclustering,
(2) the gold document clusters, and (3) the k-means clustering approach used
in Barhom et al. (2019). The gold document clusters are defined via the
transitive closure of all event coreference links. For gvc, this gold document
clustering is identical to the corpus subtopics. For the fcct test split, the
gold clustering is a single cluster containing all documents which is
equivalent to not applying document clustering at all. For ecbp, the gold
clustering largely corresponds to the corpus subtopics with the exception of
some subtopics which are merged due to cross-subtopic event coreference links.
In the k-means approach, all $n$ input documents are represented by TF–IDF
vectors based on which all possible k-means clusterings for $k=2,\ldots,n$ are
created. From these clusterings, the one with the highest silhouette score
Rousseeuw (1987) is used.

####  6.4.3 Results

Corpus | System | Preclustering | CoNLL | LEA
---|---|---|---|---
|  |  | F1 | P | R | F1
ecbp | ours | none $\bullet$ | $74.8099$ | $67.8531$ | $55.0546$ | $60.7871$
|  | k-means | $75.3664$ | $68.9013$ | $55.6042$ | $61.5423$
|  | gold | $76.2525$ | $71.5698$ | $56.3787$ | $63.0718$
|  ba2019 | k-means | $79.21$ | $67.17$ | $68.04$ | $67.60$
|  | gold | $79.08$ | $66.67$ | $68.50$ | $67.57$
fcct | ours | none/gold $\bullet$ | $54.2654$ | $30.4189$ | $60.4073$ | $39.7645$
|  ba2019 | none/gold | $48.05$ | $39.94$ | $27.18$ | $32.35$
|  | k-means | $34.61$ | $36.93$ | $08.56$ | $13.90$
gvc | ours | none $\bullet$ | $59.3876$ | $56.5137$ | $38.1787$ | $45.5702$
|  | k-means | $60.7694$ | $63.866$ | $38.5545$ | $48.0825$
|  | gold | $62.8363$ | $66.1115$ | $41.0781$ | $50.6714$
|  ba2019 | k-means | $73.26$ | $58.68$ | $74.31$ | $65.58$
|  | gold | $79.80$ | $69.49$ | $80.89$ | $74.76$
Table 8: Comparison of cdevcr performance depending on preclustering strategy.
All results reported use gold event mentions. Results marked with $\bullet$
are taken from Table 5.

Due to long runtimes times of ba2019141414Training and optimizing one model on
the fcct corpus took 10 days on an Nvidia V100 GPU. results reported from this
system stem only from a single execution. We do not report experiments without
preclustering on ecbp and gvc using this system due to scalability issues
caused by the greater number of event mentions in these corpora (see Table 1
on table 1). The results are shown in Table 8.

We comment on the most remarkable results. The ba2019 system architecture
performs well on gvc, reaching 65.6 LEA F1. Compared to the ecbp results,
there is a notable score difference between the k-means and gold preclustering
variants on this corpus. The reason is the same one that led to the lemma-time
baseline outperforming lemma-$\delta$ on this corpus — preclustering documents
by textual content is less effective on a corpus with a single topic, and
ba2019 does not make use of document publication date annotations. For fcct,
applying ba2019 out-of-the-box with k-means preclustering performs much worse
than when the preclustering step is omitted due to the large amount of cross-
subtopic links which are being cut off.

When comparing systems against each other, ba2019 performs better than the
feature-based approach on ecbp and gvc. The opposite is the case for fcct
where the neural model shows greatly reduced recall in comparison to the
feature model. This is surprising to some extent since ba2019 is a more
powerful cluster-level approach compared to the mention pair approach. A
plausible explanation for the performance drop on fcct is the narrower set of
features in ba2019. Notably, this system lacks world knowledge on locations
and participants and does not explicitly model temporal information, all of
which would make intuitive sense to have for a corpus mentioning a variety of
football players and matches happening on specific dates. The next section
adds evidence to this intuition by analyzing in greater depth the information
necessary for resolving event coreference in each corpus.

With respect to the experiments conducted in this section, we have shown that
it cannot be taken for granted that cdevcr systems are sufficiently general to
perform equally well on different corpora. This concerns both the quality of
their results (which can fluctuate) as well as more fundamental aspects such
as their computational complexity (which may preclude their applicability). In
the concrete case of ba2019, this comes down to the choice of the input
features and the dependency on document preclustering.

##  7 Identifying the Signals for Event Coreference

According to the cdevcr task definition (see Section 2.1), coreference between
a pair of event mentions requires a match between each of their components
(action, participants, time, location). We analyze to which extent corpora
satisfy this definition in practice, namely, whether inference over all event
components is indeed required, or whether certain event components suffice as
signals for resolving event coreference. We approach this analysis in two
ways: (1) We investigate the most important features per corpus at training
time via model introspection, and (2) we mask the mentions of certain event
components in the test split and measure the impact on test performance. We
explain the two approaches and present their results (Sections 7.1 and 7.2),
then jointly discuss their outcome in Section 7.3.

###  7.1 Feature Importance

Our main reason for developing a feature-based system was that, compared to
neural systems, it allows one to directly analyze which input information a
model is making use of.

In our system architecture, the agglomerative clustering step is preceded by a
mention pair classifier which we found worked best with the decision tree
boosting framework XGBoost (see Section 6). For decision trees, feature
importance metrics can be derived from trained models. In Table 9, we report
the top features selected during feature selection for each corpus. Alongside,
we report the importance of each feature at training time according to the
gain metric of XGBoost.151515Gain refers to the gain in accuracy from
introducing a split in a decision tree using a particular feature. See
https://xgboost.readthedocs.io/en/latest/tutorials/model.html for more
details.

Gain | Feature Name | Short description
---|---|---
ecbp
$0.342,106$ | is-lemma-identical | Lemma identity between actions
$0.248,146$ | surface-form-mlipns-distance | MLIPNS distance on action surface forms
$0.130,495$ | action-mention | SpanBERT action similarity
$0.085,002,8$ | is-surface-form-identical | Action surface form identity
$0.055,557,2$ | document-similarity | TF–IDF document similarity
fcct
$0.301,784$ | surface-form-mlipns-distance | MLIPNS distance on action surface forms
$0.144,558$ | is-lemma-identical | Lemma identity between actions
$0.098,399,7$ | distance-closest-overall-level-year | Years between closest temporal expressions
$0.080,500,5$ | is-surface-form-identical | Surface form identity between actions
$0.053,452,3$ | distance-sentence-level-year | Years between temp. exprs. in same sentence
gvc
$0.254,793$ | surface-form-mlipns-distance | MLIPNS distance on action surface forms
$0.126,396$ | distance-doc.-pub.-level-week | Week distance of doc. publication dates
$0.125,259$ | distance-doc.-pub.-level-month | Month distance of doc. publication dates
$0.075,458$ | document-similarity | TF–IDF document similarity
$0.059,322,2$ | is-lemma-identical | Lemma identity between actions
Table 9: The top five selected features per corpus with feature importance
(see Section 6). See Appendix C: Mention Pair Features for detailed feature
descriptions and LABEL:sec:app_feature_importance for the full listing.

For ecbp, the selected features only cover event actions and context
representations. Few features were selected overall. For fcct, event action
and temporal information received the greatest attention. There is a notable
absence of document-level features which we attribute to the fact that
document similarity is not of prime importance for resolving cross-subtopic
coreference links. For gvc a large number of features was selected, suggesting
that diverse information is required for coreference resolution decisions. The
most prominent features cover the event action and document-level information.

###  7.2 Masking of Event Components

We want to analyze the impact of each type of event component at test time. To
do so, we create variants of the test data where the spans of certain event
mentions are masked. We then predict with the models trained in the in-dataset
scenario (Section 6) and measure the score delta.

When masking mention spans, we replace each token with a unique dummy
token.161616We use fixed-length random tokens from the set [a-zA-Z]*5 . This
is to ensure that the string similarity between two mentions is entirely
random. For action components, we replace all gold annotated mention spans.
For participant, time, and location components, we replace gold annotations as
well as any additional entities identified by DBpedia Spotlight. Masked spans
are also removed from semantic role arguments. For the fcct and gvc corpora,
we additionally mask the document publication date. This masking approach is
not without limitations. In fcct and gvc, only a subset of all events was
annotated. For participants and actions, the three corpora annotate only the
head of the phrase. Both these cases may lead to information-bearing tokens
leaking into the masked dataset. We nevertheless believe that our approach is
an effective approach for analyzing the impact of specific event components at
test time.

Table 10: Impact on test performance when masking spans of certain event
components. We report the score deltas of the feature-based system w.r.t. the
scores from Table 5.

The results are shown in Table 10. On the ecbp corpus, masking event actions
has the strongest impact on performance. This is to be expected since the
majority of features used by the model are action-related features. For fcct,
masking intensifies the issue of cluster overmerging. Between action,
participants and time, the drop in LEA F1 is comparable. When interpreting the
effects of masking on fcct, it is important to keep in mind that all events
annotated in this corpus are planned events whose time, location and (to some
extent) participants are known in advance. This increases the frequency by
which these event components are mentioned in text, and on the flipside should
cause stronger losses in performance compared to ecbp and gvc which contain a
smaller proportion of planned events. The fact that scores drop for fcct only
when the document publication date and temporal expressions are removed
indicates that the document publication date was not used for grounding
temporal expressions in text. In terms of the gvc, action and time stand out
as the event components with the highest impact whereas location and
participant information barely contribute to the results. We were surprised to
see that with regard to temporal information, temporal expressions in text
carry more information than the document publication date. Manual inspection
revealed that in sentences like "Two-year-old child shot in the chest in Palm
Harbor", the entity linker would frequently misclassify "Two-year-old" as a
temporal expression instead of a person entity. A portion of the performance
loss from masking time expressions may therefore arise from masked
participants.

###  7.3 Summary on the Signals for Event Coreference

Answering our initial question, whether established cdevcr corpora match the
cdevcr task definition in that they require inference over each event
component, we conclude that this is not the case.

Our experiments demonstrate that cdevcr decisions in ecbp are strongly driven
by action mentions. gvc, designed to overcome this shortcoming of ecbp Vossen
et al. (2018), necessitates inference over action mentions, time and to some
level participants, but does not challenge systems on spatial inference. fcct
is the most balanced of the three corpora based on the facts that at training
time, feature selection yielded a broad selection of features and at test
time, performance decreases similarly when action, location or time
information is removed. Overall, neither corpus requires inference over all
four event components which define the cdevc relation.

This indicates that cdevcr systems which focus on solving a single corpus
model only a subset of the entire cdevcr task which severely limits their
downstream use on unseen data, as this data may reflect the task differently
from what was observed at training time. The findings further raise the
question to which degree it is possible to resolve cdevc in the three corpora
with a single model, as this would be the most desirable usage scenario for
applying cdevcr downstream. We address this question in the following section.

##  8 Generalizability of Trained cdevcr Models

All preceding experiments in this work have addressed the ecbp, fcct, and gvc
corpora in isolation, training a separate model per corpus. In downstream
application scenarios, such a differentation is not possible – here, a cdevcr
system is expected to resolve cdevc in a robust manner regardless of the
selection of topics or underlying structure of a given collection of
documents. To gain insights into which performance to expect in such a
scenario, we test models of the feature-based system in a cross-dataset
transfer scenario on unseen corpora.171717We did not test ba2019 in this
scenario due to the scalability issues reported in Section 6.4.

Furthermore, recent research on the qa task has shown that training systems
jointly on multiple datasets can improve model robustness and can boost
performance Talmor and Berant (2019); Guo et al. (2021); Fisch et al. (2019).
In this work, we have established compatibility between the ecbp, fcct, and
gvc corpora and have identified the different ways in which each corpus models
event coreference. We test whether benefits similar to those observed for qa
are possible for cdevcr by training the feature-based system on multiple
cdevcr corpora.

###  8.1 Experiment Setup

We use same splits for all corpora as in previous experiments. In ecbp, a
number of topics cover sports news or news related to gun violence. We refer
to these corpus subsets as ecbp${}_{\text{sports}}$ and ecbp${}_{\text{guns}}$
respectively and treat them separately in our experiments. Their contents are
shown in Table 11. The ecbp test split remains unchanged.

Corpus Subset | Train | Dev
---|---|---
ecbp${}_{\text{sports}}$ | 5, 7, 10, 25 | 29, 31
ecbp${}_{\text{guns}}$ | 3, 8, 16, 18, 22 | 33
Table 11: Topics used as distinct subsets of ecbp.

When combining two corpora, we use the union of features previously selected
during feature selection. The increase in training data leads to an increase
in mention pairs which prolongs the training process. We therefore optimize
the hyperparameters of the mention pair classifier and the agglomerative
clustering step for three days each.

###  8.2 Results

Train + Dev Split | Test Split Performance (LEA)
---|---
ecbp |  ecbp${}_{\text{sports}}$ |  ecbp${}_{\text{guns}}$ |  fcct |  gvc |  |  |  |  |  |  |  |  |  |  |  |
ecbp |  fcct |  gvc | Harmonic Means
P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
✓ |  |  |  |  | $67.853,059,549,688,77$ | $55.054,554,949,475,48$ | $60.787,136,826,441$ | $41.420,233,976,227,81$ | $08.524,460,588,440,183$ | $14.134,440,954,440,702$ | $32.096,465,241,209,404$ | $25.261,613,010,757,49$ | $28.271,322,518,914,41$ | $42.834,747,942,761,55$ | $17.137,076,477,727,678$ | $24.475,499,500,624,8$
|  |  | ✓ |  | $15.961,317,315,693,25$ | $57.157,113,869,949,29$ | $24.907,385,103,300,675$ | $30.418,940,241,743,48$ | $60.407,334,955,685,91$ | $39.764,538,321,934,83$ | $03.552,649,244,266,014,4$ | $74.977,449,414,950,5$ | $06.781,433,642,264,527$ | $07.957,437,104,904,444$ | $63.308,155,587,517,82$ | $14.100,513,163,555,067$
|  |  |  | ✓ | $48.845,049,091,414,56$ | $50.168,338,429,825,55$ | $49.496,580,003,705,89$ | $42.164,301,661,868,27$ | $17.657,032,601,814,37$ | $24.854,341,879,942,6$ | $56.513,665,801,288,73$ | $38.178,728,669,576,83$ | $45.570,236,516,006,09$ | $48.477,417,746,963,1$ | $29.194,198,092,313,606$ | $36.415,729,595,934,2$
| ✓ |  | ✓ |  | $52.064,473,087,229,67$ | $54.279,760,939,629,45$ | $53.138,297,041,607,47$ | $42.719,187,002,918,26$ | $40.249,338,161,658,08$ | $41.311,916,123,053,15$ | $16.892,331,067,148,966$ | $26.617,969,121,567,43$ | $20.644,243,017,552,12$ | $29.465,466,606,209,06$ | $37.111,806,665,143,793$ | $32.799,587,602,450,053$
|  | ✓ |  | ✓ | $62.546,813,236,700,88$ | $55.167,194,229,236,55$ | $58.625,440,279,607,24$ | $41.478,215,050,120,704$ | $19.944,345,694,378,78$ | $26.934,653,765,287,54$ | $45.016,429,570,795,546$ | $35.448,947,061,460,14$ | $39.663,397,783,592,11$ | $48.145,454,811,491,84$ | $31.095,874,114,862,887$ | $37.785,042,985,887,357$
✓ |  |  | ✓ |  | $63.256,723,176,581,09$ | $60.701,338,697,687$ | $58.032,886,277,267,39$ | $50.921,066,762,406,23$ | $19.699,078,444,340,76$ | $28.401,396,223,288,98$ | $29.650,423,534,038,21$ | $26.674,242,869,041,537$ | $28.071,490,019,095,67$ | $43.369,419,466,606,94$ | $28.061,662,920,050,17$ | $34.065,949,123,728,94$
✓ |  |  |  | ✓ | $55.370,474,798,091,82$ | $54.774,425,460,221,91$ | $55.070,364,200,215,15$ | $38.006,099,622,530,95$ | $25.467,612,667,095,82$ | $30.493,468,263,325,83$ | $27.359,811,431,603,875$ | $32.835,165,399,860,92$ | $29.840,213,885,434,25$ | $37.072,877,743,677,98$ | $34.099,664,208,650,55$ | $35.517,956,243,209,08$
|  |  | ✓ | ✓ | $37.562,066,670,642,97$ | $47.744,658,545,293,05$ | $42.038,512,535,528,27$ | $34.101,126,962,494,51$ | $48.519,923,698,491,93$ | $40.044,002,887,171,95$ | $45.694,853,885,180,07$ | $36.558,100,435,154,43$ | $40.617,639,234,386,74$ | $38.544,830,013,277,05$ | $43.535,955,310,576,95$ | $40.883,026,960,062,213$
✓ |  |  | ✓ | ✓ | $24.284,291,773,811,88$ | $53.595,104,615,098$ | $33.411,812,635,837,485$ | $26.492,040,624,418,18$ | $58.801,776,958,887,81$ | $36.521,640,363,864,81$ | $32.313,692,176,180,625$ | $38.264,198,395,006,44$ | $35.031,913,823,262,89$ | $27.304,325,075,180,785$ | $48.544,632,572,030,2$ | $34.942,244,654,711,78$
Table 12: Performance of the feature-based system when trained on a single
respective corpus (top) vs. multiple corpora at once (bottom). No document
preclustering was applied. The rightmost set of columns shows P, R and F1
aggregated over the three corpora. The full set of metrics is reported in
Appendix B: Full Cross-Dataset cdevcr Results.

The results of our experiments, evaluated with LEA, are shown in Table 12. As
we have shown in preceding sections, the requirements for resolving cdevc vary
between the corpora which strongly influences the feature selection and model
training processes. We hence expected models trained on a single corpus to
perform poorly when evaluated on unseen corpora. This is confirmed by the top
rows of Table 12 where significant gaps between in-dataset and cross-dataset
performance can be observed.

When looking at the performance on individual corpora, models trained on
multiple corpora perform consistently worse than those trained on a single
corpus, with few exceptions (mixing fcct with ecbp${}_{\text{sports}}$ or gvc
during training leads to more balanced LEA recall and precision). However, the
best overall result in terms of the whole task (i.e., across all corpora) was
achieved with joint training: the model trained on fcct and gvc scores $40.9$
mean LEA F1 over all corpora, whereas the best single-corpus model trained on
gvc alone only reached $36.4$. In conclusion, training on multiple corpora did
not boost performance on individual corpora. Nevertheless, joint training on
multiple corpora has emerged as an important strategy for reaching general
cdevcr systems.

We have only scratched the surface of joint training for cdevcr. Further
improvements may be achieved with more sophisticated training approaches, for
example by mixing together different amounts of each corpus (potentially
aiming for certain distributions of coreference link types), testing its
effects on cdevcr systems beyond mention pair approaches or performing
training data augmentation with data from other NLP tasks.

##  9 Discussion

Despite its importance for downstream applications, the generalizability of
cdevcr systems over different corpora has not received attention in the past.

Our experiments showed that a system achieving state-of-the-art-level
performance on ecbp does not consistently produce results of the same quality
when trained and tested on other cdevcr corpora (see Section 6.4). This raises
the suspicion that similar systems which were developed for a single corpus
lack the capacity of generalizing to unseen corpora. This suspicion is
substantiated when looking at the results of our cross-dataset experiments. We
showed that training a general, feature-based cdevcr system on a single corpus
yields good results on the test split of that respective corpus, whereas
performance on other corpora falls short of these results (see Section 8).

This is due to the fact that the ecbp, fcct and gvc corpora test systems on
different, yet equally important parts of the overall task of performing CDCR
in news text (cf. our requirements posed in Section 2.2). Beyond established
knowledge, such as ecbp testing systems on a greater number of topics while
offering low variation in event instances Vossen et al. (2018), we found that:

  * •

The distribution of coreference links in each corpus varies significantly,
with gvc offering roughly the same number of within-document and within-
subtopic links, whereas fcct offers many cross-subtopic links (see Table 1).

  * •

Structural differences between corpora (such as the number of subtopics or
mentions) can pose a problem for established cdevcr techniques such as
document preclustering and lead to performance drops (see Section 6.4.2), and
expose or amplify scalability issues in systems (see Section 6.4.3).

  * •

Between the corpora, the relevance of the four event components (action,
participants, time, location) for resolving cdevc varies strongly. In
particular, ecbp stands out for requiring inference over event actions almost
exclusively (see Section 7).

This means that by designing a system against a single corpus, significant
aspects of cdevcr are disregarded. Doing so introduces a bias towards corpora
with specific properties, which severely limits a system’s usefulness for
downstream applications on data which exhibits different properties.
Therefore, when claiming that a system is capable of resolving cdevc in the
general case, it is imperative to report its performance on multiple cdevcr
corpora to certify its robustness to all aspects of cdevcr annotated therein.

Related to this finding is the recent trend of ecbp systems applying document
preclustering prior to mention-level event coreference resolution which
deserves special attention. Throughout this work, we have pointed out that by
preclustering documents via TF–IDF, one can reproduce the subtopics of a
corpus. At test time, this yields an increase in precision for the resolution
of within-document and within-subtopic links but has the downside of
precluding the resolution of cross-subtopic or cross-topic links. This
downside is negligible on ecbp because in this corpus, within-document and
within-subtopic links outnumber cross-subtopic links by a factor of 100 (see
Table 1). Many recent ecbp systems apply preclustering Cremisini and Finlayson
(2020); Meged et al. (2020); Barhom et al. (2019); Lee et al. (2012), yet
scores drop sharply when such a system is applied out-of-the-box on a corpus
with a different distribution of coreference links (see Table 8). The
performance boost from document preclustering therefore comes at the cost of
an overspecialization on the coreference link distribution in ecbp, which we
consider to be a form of overfitting. This is again a strong point for the
evaluation of cdevcr systems on multiple corpora by which this (inadvertent)
oversight in existing systems can be revealed.181818An argument could be made
at this point that reporting precision and recall separately for each type of
coreference link (as we did in Section 6.3.2) were as insightful as evaluating
a system on a variety of corpora with different properties. This is not the
case since this such analysis is only possible for the subset of cdevcr
systems which compute a degree of coreference on link-level. For
representation learning approaches or for metric learning approaches with
cluster-level features, link-level analysis is not possible.

###  9.1 Evaluation Recommendations

We summarize our findings with respect to the evaluation of cdevcr with four
recommendations for future research which pave the way for more general,
comparable and reliably evaluated cdevcr systems.

1\. cdevcr systems should be tested on more than one corpus. The ecbp, fcct,
and gvc corpora each are unique with respect to their topic structure,
selection of topics and distribution of event coreference links, all of which
can have an impact on the performance of a cdevcr system. Furthermore, the
importance of action, participants, time, and location information varies in
each corpus. For systems seeking to solve the cdevcr task in general (i.e.
where the application scenario does not necessitate the choice of one domain-
specific corpus), this prompts for joint evaluation on multiple cdevcr corpora
to reveal a system’s strengths and weaknesses. Where possible, the performance
for each link type (within-document, within-subtopic, etc.) should be
reported.

2\. The LEA evaluation metric Moosavi and Strube (2016) should be used as an
additional performance indicator for cdevcr. This metric was previously shown
to be more discriminative than previous coreference resolution metrics such as
CoNLL F1 and takes size differences between clusters into account. We showed
that cluster sizes in cdevcr corpora vary significantly and observed score
deltas between CoNLL F1 and LEA F1 of up to $20\text{\,}\mathrm{\textup{pp}}$
which motivates its use for cdevcr over CoNLL F1.

3\. In addition to a blind prediction, system performance should be reported
when using gold document clusters. Cremisini and Finlayson (2020) request that
future system development efforts should report scores with or without
document clustering. We agree with this suggestion and refine it further.
Given that in the research area of within-document coreference it is
commonplace to report separate scores for mention identification and
coreference resolution, we think distinguishing scores obtained with and
without knowledge on the gold corpus structure would only make sense for
cdevcr. Researchers must take care to define the gold document clusters based
on the transitive closure of event coreference links (see Section 6.4). Using
the topics or subtopics of a corpus for this purpose produces incorrect
results since cross-subtopic (or cross-topic) coreference links in the corpus
are not taken into account.

4\. Event detection performance should be evaluated carefully on cdevcr
corpora. From the point of view of a general-purpose event mention detection
system, the event mention annotations in the ecbp, fcct and gvc corpora are
incomplete by design (see Section 2.4.3). Care must be taken to not unfairly
penalize a systems which includes a mention detection step as it may detect
valid event mentions for which no gold annotation exists. We recommend to
compute mention detection performance only on those sentences which contain
gold event mention annotations and to report coreference resolution
performance using the gold event mention annotations as a remedy.

##  10 Future Work

Having established that evaluation on multiple of the currently available
corpora is necessary for a reliable performance assessment of cdevcr systems,
we consider the development of approaches which show consistent performance in
such a scenario as the next short to medium term goal for this task.

A key challenge will be achieving systems which scale to collections of
10k-100k documents without precluding the resolution of cross-subtopic and
cross-topic links. A foundation has already been laid by Kenyon-Dean, Cheung,
and Precup (2018) who investigated scalable representation learning approaches
for cdevcr. Since current corpora consist of less than 1k documents, this may
require the annotation of additional corpora. To keep the costs of annotating
corpora of such magnitude manageable, novel semi-automatic annotation
techniques would be required. Furthermore, the concept of cross-topic event
coreference links has not been investigated yet due to a lack of annotated
data. Once sufficient robustness and/or scalability has been achieved, use
cases for downstream applications of cdevcr could be investigated.

##  11 Conclusion

The usefulness of cdevcr for downstream multi-document NLP tasks has not been
demonstrated yet. To perform well on unseen data in general, NLP systems need
to robustly handle variations in the data they are applied on. For cdevcr,
multiple corpora with varying properties have been annotated, yet each cdevcr
system to date was developed, trained, and evaluated on only a single one of
them. Besides hurting comparability, this currently allows little conclusions
to be drawn on their robustness and generalizability, which contributes to the
initially stated problem. We addressed this situation in several ways:

We eliminated the remaining hurdles which rendered joint training and
evaluation on multiple cdevcr corpora difficult by creating fcct, a
reannotation and extension of the fcc on token level.

To identify the unique properties of each corpus for resolving event
coreference in practice, we developed a mention pair cdevcr system with a
broad set of handcrafted features and applied it on the ecbp, fcct, and gvc
corpora. Using this system, we found that only a subset of all components by
which events are commonly defined (action, participants, time, location) are
required for resolving cdevcr in each corpus in practice. In particular, ecbp
only focuses on resolving event actions whereas gvc and fcct are more balanced
and additionally demand interpretation of event dates and its participants.
Link-level analysis of this system revealed that mention distance (with
respect to the topic-subtopic-document hierarchy of a corpus) positively
correlates with difficulty in resolving cdevc links.

In the first uniform evaluation scenario involving multiple cdevcr systems and
corpora, we compared the neural ecbp system of Barhom et al. (2019) to the
feature-based system. Firstly, we found that the neural system performs well
on gvc but is outperformed by the conceptually simpler mention pair approach
on fcct. Secondly, we deduced from these experiments that systems which are
developed for ecbp and which apply document preclustering overfit to the link
distribution in this corpus.

In brief experiments with joint training on multiple corpora, we achieve a
combined LEA F1 of $40.9$ across all three corpora with the feature-based
system – over $4.5\text{\,}\mathrm{\textup{pp}}$ better than the same system
trained on either corpus in isolation.

We offered four recommendations for future research on cdevcr. Most
importantly, we advocate evaluation on multiple corpora after having provided
conclusive evidence that evaluating on a single corpus is and was
insufficient.

All in all, with our annotation effort, corpus analyses, experiments, and open
source implementation we have laid a solid foundation for future research on
robust and general cdevcr systems. Achieving such systems then constitutes a
big step forward towards cdevcr becoming an integral part of the multi-
document NLP pipeline.

###### Acknowledgements.

We thank Mohsen Mesgar, Kevin Stowe, Prasetya Ajie Utama and Mingzhu Wu for
their helpful comments. Special thanks are due to Jan-Christoph Klie and
Nafise Sadat Moosavi for the frequent exchange of ideas. This work was
supported by the German Research Foundation through the German–Israeli Project
Cooperation (DIP, grant DA 1600/1–1 and grant GU 798/17–1).

## Appendix A: Full In-Dataset cdevcr Results

Corpus | System | Preclustering | MUC |  B3 | CEAFe | CoNLL | LEA
---|---|---|---|---|---|---|---
|  |  | P | R | F1 | P | R | F1 | P | R | F1 | F1 | P | R | F1
ecbp | lemma | n/a | $59.7015$ | $69.7436$ | $64.333$ | $58.0893$ | $67.8823$ | $62.6051$ | $66.297$ | $52.7906$ | $58.7779$ | $61.9053$ | $42.8045$ | $43.4982$ | $43.1486$
|  lemma-$\delta$ | n/a | $81.4994$ | $69.1282$ | $74.8058$ | $88.4451$ | $67.6869$ | $76.6861$ | $66.1365$ | $78.2957$ | $71.7043$ | $74.3987$ | $71.5428$ | $53.6916$ | $61.3449$
| feature-based | none | $76.0833$ | $76.0205$ | $76.0518$ | $81.1882$ | $71.7779$ | $76.1934$ | $72.1489$ | $72.2205$ | $72.1845$ | $74.8099$ | $67.8531$ | $55.0546$ | $60.7871$
|  | k-means | $77.3523$ | $76.2256$ | $76.7848$ | $82.3109$ | $71.8443$ | $76.7221$ | $71.963$ | $73.2325$ | $72.5922$ | $75.3664$ | $68.9013$ | $55.6042$ | $61.5423$
|  | gold | $79.1648$ | $76.6154$ | $77.8692$ | $84.6904$ | $72.011$ | $77.8374$ | $71.6797$ | $74.4756$ | $73.0509$ | $76.2525$ | $71.5698$ | $56.3787$ | $63.0718$
|  ba2019 | k-means | $80.67$ | $83.49$ | $82.06$ | $78.25$ | $81.39$ | $79.79$ | $77.46$ | $74.19$ | $75.79$ | $79.21$ | $67.17$ | $68.04$ | $67.60$
|  | gold | $80.26$ | $83.79$ | $81.99$ | $77.48$ | $81.79$ | $79.58$ | $77.80$ | $73.65$ | $75.67$ | $79.08$ | $66.67$ | $68.50$ | $67.57$
fcct | lemma | n/a | $66.1972$ | $58.75$ | $62.2517$ | $59.2947$ | $33.269$ | $42.6231$ | $19.433$ | $31.0226$ | $23.8967$ | $42.9238$ | $38.3587$ | $19.8697$ | $26.1788$
|  lemma-$\delta$ | n/a | $66.1972$ | $58.75$ | $62.2517$ | $59.2947$ | $33.269$ | $42.6231$ | $19.433$ | $31.0226$ | $23.8967$ | $42.9238$ | $38.3587$ | $19.8697$ | $26.1788$
| lemma-time | n/a | $64.4509$ | $50.6818$ | $56.743$ | $64.6777$ | $27.6343$ | $38.7235$ | $17.5634$ | $37.4544$ | $23.9132$ | $39.7932$ | $36.8327$ | $14.2433$ | $20.5427$
| feature-based | none/gold | $78.2773$ | $82.7273$ | $80.4404$ | $38.3255$ | $70.8046$ | $49.1666$ | $40.3836$ | $28.1848$ | $33.1891$ | $54.2654$ | $30.4189$ | $60.4073$ | $39.7645$
|  ba2019 | none/gold | $84.71$ | $49.09$ | $62.16$ | $83.04$ | $36.01$ | $50.23$ | $20.79$ | $67.12$ | $31.74$ | $48.05$ | $39.94$ | $27.18$ | $32.35$
|  | k-means | $78.33$ | $34.09$ | $47.51$ | $88.25$ | $17.94$ | $29.81$ | $16.57$ | $66.18$ | $26.50$ | $34.61$ | $36.93$ | $08.56$ | $13.90$
gvc | lemma | n/a | $52.0759$ | $57.9156$ | $54.8407$ | $18.4373$ | $44.2145$ | $26.0231$ | $28.6073$ | $16.0082$ | $20.5288$ | $33.7975$ | $08.805,42$ | $29.7228$ | $13.586$
|  lemma-$\delta$ | n/a | $70.1754$ | $52.7704$ | $60.241$ | $70.226$ | $41.5697$ | $52.2252$ | $28.9452$ | $57.1406$ | $38.4256$ | $50.2972$ | $43.8354$ | $28.6605$ | $34.6598$
| lemma-time | n/a | $82.0084$ | $51.715$ | $63.4304$ | $85.4013$ | $40.5709$ | $55.0091$ | $25.3173$ | $62.047$ | $35.9612$ | $51.4669$ | $53.813$ | $27.2605$ | $36.1887$
| feature-based | none | $78.0916$ | $66.3061$ | $71.7177$ | $73.5503$ | $49.9092$ | $59.4658$ | $38.2385$ | $60.9018$ | $46.9794$ | $59.3876$ | $56.5137$ | $38.1787$ | $45.5702$
|  | k-means | $83.55$ | $66.2005$ | $73.8701$ | $81.3218$ | $49.4551$ | $61.5058$ | $36.3916$ | $66.0696$ | $46.9322$ | $60.7694$ | $63.866$ | $38.5545$ | $48.0825$
|  | gold | $85.3124$ | $68.8127$ | $76.1793$ | $82.7209$ | $51.7024$ | $63.6328$ | $38.186$ | $67.1913$ | $48.6967$ | $62.8363$ | $66.1115$ | $41.0781$ | $50.6714$
|  ba2019 | k-means | $85.23$ | $89.05$ | $87.10$ | $66.01$ | $80.95$ | $72.72$ | $66.38$ | $54.69$ | $59.97$ | $73.26$ | $58.68$ | $74.31$ | $65.58$
|  | gold | $88.90$ | $91.95$ | $90.40$ | $75.47$ | $86.21$ | $80.48$ | $73.85$ | $63.90$ | $68.52$ | $79.80$ | $69.49$ | $80.89$ | $74.76$

Table A.1: Full cdevcr results for the in-dataset scenario. We report cross-
document performance (all documents merged into one meta document before
scoring). The scores of the feature-based system are the mean of five
independent runs. Scores for ba2019 stem from a single run.

## Appendix B: Full Cross-Dataset cdevcr Results

Train + Dev | Test | Metrics
---|---|---
ecbp |  ecbp${}_{\text{sports}}$ |  ecbp${}_{\text{guns}}$ |  fcct |  gvc |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MUC |  B3 | CEAFe | CoNLL | LEA
| P | R | F1 | P | R | F1 | P | R | F1 | F1 | P | R | F1
✓ |  |  |  |  |  ecbp | $76.083,342,468,274,84$ | $76.020,512,820,512,82$ | $76.051,767,864,437,71$ | $81.188,160,051,008,72$ | $71.777,917,570,995,94$ | $76.193,374,604,072,68$ | $72.148,902,467,833,81$ | $72.220,480,566,400,46$ | $72.184,468,846,689,02$ | $74.809,870,438,399,81$ | $67.853,059,549,688,77$ | $55.054,554,949,475,48$ | $60.787,136,826,441$
fcct | $62.915,855,503,635,17$ | $44.840,909,090,909,09$ | $52.362,170,357,882,66$ | $71.009,658,118,602,09$ | $21.638,112,692,379,868$ | $33.168,289,960,301,43$ | $18.817,967,399,405,733$ | $47.477,664,580,166,52$ | $26.952,731,524,910,29$ | $37.494,397,281,031,466$ | $41.420,233,976,227,81$ | $08.524,460,588,440,183$ | $14.134,440,954,440,702$
gvc | $55.371,441,523,242,71$ | $51.688,654,353,562,01$ | $53.466,386,960,719,06$ | $50.276,067,330,544,79$ | $40.091,639,039,257,49$ | $44.608,573,085,151,95$ | $33.311,169,544,511,89$ | $42.013,925,494,023,05$ | $37.157,963,796,984,93$ | $45.077,641,280,951,97$ | $32.096,465,241,209,404$ | $25.261,613,010,757,49$ | $28.271,322,518,914,41$
|  |  | ✓ |  |  ecbp | $57.314,479,252,476,59$ | $91.138,461,538,461,56$ | $70.371,768,698,111,34$ | $22.102,907,883,145,884$ | $88.967,376,954,661,75$ | $35.367,857,846,294,576$ | $72.231,756,213,625,51$ | $20.602,364,180,457,056$ | $32.038,644,410,492,66$ | $45.926,090,318,299,54$ | $15.961,317,315,693,25$ | $57.157,113,869,949,29$ | $24.907,385,103,300,675$
fcct | $78.277,264,221,240,33$ | $82.727,272,727,272,72$ | $80.440,390,547,221,98$ | $38.325,459,638,487,27$ | $70.804,592,105,056,88$ | $49.166,629,113,737,29$ | $40.383,562,240,966,61$ | $28.184,839,487,047,55$ | $33.189,115,364,870,025$ | $54.265,378,341,943,09$ | $30.418,940,241,743,48$ | $60.407,334,955,685,91$ | $39.764,538,321,934,83$
gvc | $73.463,180,420,186,62$ | $88.970,976,253,298,15$ | $80.476,737,716,870,16$ | $06.443,801,507,038,814$ | $82.908,276,721,525,93$ | $11.955,860,613,165,73$ | $31.001,544,572,797,21$ | $05.304,055,122,736,693$ | $09.055,373,022,723,448$ | $33.829,323,784,253,11$ | $03.552,649,244,266,014,4$ | $74.977,449,414,950,5$ | $06.781,433,642,264,527$
|  |  |  | ✓ |  ecbp | $64.533,808,915,611,82$ | $82.523,076,923,076,92$ | $72.427,603,897,830,12$ | $63.516,880,474,595,09$ | $78.867,969,600,365,85$ | $70.363,015,889,124,85$ | $68.772,701,357,307,9$ | $45.553,543,748,751,51$ | $54.803,094,577,906,08$ | $65.864,571,454,953,68$ | $48.845,049,091,414,56$ | $50.168,338,429,825,55$ | $49.496,580,003,705,89$
fcct | $65.171,464,837,591,46$ | $56.659,090,909,090,91$ | $60.613,912,175,871,11$ | $62.584,679,845,520,54$ | $30.964,260,091,631,77$ | $41.391,485,381,634,4$ | $22.294,419,100,259,147$ | $37.723,725,131,897,534$ | $28.012,788,685,304,313$ | $43.339,395,414,269,95$ | $42.164,301,661,868,27$ | $17.657,032,601,814,37$ | $24.854,341,879,942,6$
gvc | $78.091,559,675,569,07$ | $66.306,068,601,583,12$ | $71.717,710,141,218,42$ | $73.550,331,949,551,58$ | $49.909,243,619,063,83$ | $59.465,826,335,008,89$ | $38.238,512,565,737,24$ | $60.901,783,719,941,02$ | $46.979,354,412,514,09$ | $59.387,630,296,247,12$ | $56.513,665,801,288,73$ | $38.178,728,669,576,83$ | $45.570,236,516,006,09$
| ✓ |  | ✓ |  |  ecbp | $67.022,378,464,783,54$ | $81.661,538,461,538,47$ | $73.618,508,315,096,93$ | $64.209,788,442,075,5$ | $77.642,333,306,162,43$ | $70.277,492,488,961,89$ | $73.619,380,089,857,62$ | $54.135,046,259,291,41$ | $62.382,106,979,251,28$ | $68.759,369,261,103,38$ | $52.064,473,087,229,67$ | $54.279,760,939,629,45$ | $53.138,297,041,607,47$
fcct | $75.702,822,175,943,07$ | $74.568,181,818,181,82$ | $75.130,836,777,367,75$ | $54.733,894,927,103,25$ | $51.495,987,498,020,07$ | $52.955,339,616,596,49$ | $33.685,016,297,199,53$ | $36.355,783,462,740,543$ | $34.965,381,570,050,17$ | $54.350,519,321,338,14$ | $42.719,187,002,918,26$ | $40.249,338,161,658,08$ | $41.311,916,123,053,15$
gvc | $52.820,071,125,776,01$ | $55.804,749,340,369,39$ | $54.270,937,965,579,69$ | $28.656,836,692,997,4$ | $41.366,587,237,769,28$ | $33.842,706,905,825,93$ | $34.586,488,528,823,56$ | $26.912,172,991,869,643$ | $30.263,129,219,062,12$ | $39.458,924,696,822,58$ | $16.892,331,067,148,966$ | $26.617,969,121,567,43$ | $20.644,243,017,552,12$
|  | ✓ |  | ✓ |  ecbp | $72.260,732,667,593,54$ | $81.005,128,205,128,22$ | $76.383,270,354,263,54$ | $76.609,753,676,227,59$ | $77.257,173,202,915,83$ | $76.931,643,764,552,26$ | $71.190,161,900,690,71$ | $60.754,604,245,548,28$ | $65.559,251,097,563,94$ | $72.958,055,072,126,58$ | $62.546,813,236,700,88$ | $55.167,194,229,236,55$ | $58.625,440,279,607,24$
fcct | $64.989,178,053,054,08$ | $59.272,727,272,727,28$ | $61.998,652,767,165,14$ | $59.308,817,954,460,77$ | $33.390,794,809,212,15$ | $42.724,368,746,981,39$ | $23.329,649,675,317,79$ | $34.210,483,084,125,44$ | $27.737,551,067,836,125$ | $44.153,524,193,994,22$ | $41.478,215,050,120,704$ | $19.944,345,694,378,78$ | $26.934,653,765,287,54$
gvc | $69.500,152,925,051,16$ | $63.614,775,725,593,66$ | $66.427,105,797,637,62$ | $60.901,552,026,307,05$ | $48.062,043,101,267,26$ | $53.724,765,720,063,48$ | $39.348,884,770,393,68$ | $52.435,445,981,754,65$ | $44.957,991,018,961,51$ | $55.036,620,845,554,21$ | $45.016,429,570,795,546$ | $35.448,947,061,460,14$ | $39.663,397,783,592,11$
✓ |  |  | ✓ |  |  ecbp | $73.203,240,117,916,46$ | $75.630,769,230,769,23$ | $74.395,886,890,345,97$ | $77.239,677,881,164,29$ | $71.838,400,212,378,04$ | $74.439,832,376,728,48$ | $72.221,954,844,907,74$ | $69.314,650,949,732,28$ | $70.736,293,245,630,44$ | $73.190,670,837,568,31$ | $63.256,723,176,581,09$ | $53.607,013,386,976,87$ | $58.032,886,277,267,39$
fcct | $71.340,745,788,456,92$ | $58.727,272,727,272,72$ | $64.420,987,388,802,01$ | $71.578,155,813,895,59$ | $31.002,911,942,246,8$ | $43.261,369,062,324,906$ | $27.952,190,219,921,924$ | $54.149,850,495,264,75$ | $36.867,580,187,192,68$ | $48.183,312,212,773,21$ | $50.921,066,762,406,23$ | $19.699,078,444,340,76$ | $28.401,396,223,288,98$
gvc | $56.957,467,964,881,43$ | $54.379,947,229,551,44$ | $55.638,006,180,158$ | $46.474,240,960,274,02$ | $41.181,921,880,428,59$ | $43.658,964,534,045,02$ | $35.058,840,263,323,18$ | $41.268,308,305,704,48$ | $37.905,240,821,140,49$ | $45.734,070,511,781,17$ | $29.650,423,534,038,21$ | $26.674,242,869,041,537$ | $28.071,490,019,095,67$
✓ |  |  |  | ✓ |  ecbp | $68.816,153,533,4$ | $81.887,179,487,179,47$ | $74.784,507,708,272,31$ | $68.521,571,979,134,22$ | $78.063,643,242,631,1$ | $72.981,115,948,084,27$ | $73.084,288,104,280,61$ | $56.269,491,626,499,29$ | $63.583,087,200,996,51$ | $70.449,570,285,784,37$ | $55.370,474,798,091,82$ | $54.774,425,460,221,91$ | $55.070,364,200,215,15$
fcct | $65.379,325,308,476,38$ | $66.136,363,636,363,64$ | $65.755,404,672,919,56$ | $50.021,796,546,219,7$ | $39.204,560,468,600,36$ | $43.952,517,145,737,68$ | $25.904,783,752,212,24$ | $24.315,127,604,172,815$ | $25.081,634,626,290,94$ | $44.929,852,148,316,05$ | $38.006,099,622,530,95$ | $25.467,612,667,095,82$ | $30.493,468,263,325,83$
gvc | $60.352,665,631,238,88$ | $61.767,810,026,385,23$ | $61.051,852,167,202,94$ | $39.999,234,184,238,68$ | $46.361,508,588,067,55$ | $42.937,979,718,816,93$ | $38.337,501,610,129,97$ | $34.809,264,709,744,203$ | $36.486,104,408,936,604$ | $46.825,312,098,318,83$ | $27.359,811,431,603,875$ | $32.835,165,399,860,92$ | $29.840,213,885,434,25$
|  |  | ✓ | ✓ |  ecbp | $59.804,167,167,595,6$ | $81.743,589,743,589,74$ | $69.071,251,141,514,4$ | $50.623,995,217,587,48$ | $78.150,166,316,494,45$ | $61.432,069,407,418,39$ | $68.433,300,014,999,14$ | $38.018,656,709,772,61$ | $48.866,818,692,542,29$ | $59.790,046,413,825,04$ | $37.562,066,670,642,97$ | $47.744,658,545,293,05$ | $42.038,512,535,528,27$
fcct | $70.276,819,095,377,67$ | $78.022,727,272,727,27$ | $73.947,289,948,992,3$ | $39.590,109,791,294,94$ | $61.019,364,236,306,57$ | $48.018,095,484,269,4$ | $38.603,228,809,591,89$ | $16.045,507,124,865,893$ | $22.658,834,601,966,965$ | $48.208,073,345,076,23$ | $34.101,126,962,494,51$ | $48.519,923,698,491,93$ | $40.044,002,887,171,95$
gvc | $71.056,533,591,968,05$ | $65.092,348,284,960,42$ | $67.942,929,485,388,51$ | $60.669,939,901,772,31$ | $48.797,035,630,435,06$ | $54.086,624,762,516,58$ | $39.519,787,363,063,97$ | $52.541,326,692,526,82$ | $45.105,420,346,014,45$ | $55.711,658,197,973,18$ | $45.694,853,885,180,07$ | $36.558,100,435,154,43$ | $40.617,639,234,386,74$
✓ |  |  | ✓ | ✓ |  ecbp | $56.804,274,004,031,49$ | $89.989,743,589,743,59$ | $69.645,508,651,759,67$ | $32.254,033,464,826,04$ | $87.165,785,352,234,4$ | $47.073,556,133,894,62$ | $65.784,528,359,514,23$ | $19.240,265,846,576,135$ | $29.767,189,919,033,72$ | $48.828,751,568,229,33$ | $24.284,291,773,811,88$ | $53.595,104,615,098$ | $33.411,812,635,837,485$
fcct | $73.579,707,005,856,82$ | $84.431,818,181,818,19$ | $78.633,017,661,414,21$ | $29.597,560,587,712,96$ | $70.689,603,429,574,64$ | $41.720,934,874,576,61$ | $35.303,879,317,654,474$ | $07.706,267,763,276,911$ | $12.644,387,441,706,034$ | $44.332,779,992,565,61$ | $26.492,040,624,418,18$ | $58.801,776,958,887,81$ | $36.521,640,363,864,81$
gvc | $63.922,877,858,828,2$ | $66.807,387,862,796,84$ | $65.333,254,167,965,32$ | $42.647,869,068,035,44$ | $50.781,187,476,363,81$ | $46.358,498,084,889,65$ | $43.032,660,696,863,06$ | $35.405,526,362,824,87$ | $38.847,449,508,725,56$ | $50.179,733,920,526,84$ | $32.313,692,176,180,625$ | $38.264,198,395,006,44$ | $35.031,913,823,262,89$
Table B.1: Full cdevcr results for the cross-dataset scenario. We report
cross-document performance (all documents merged into one meta document before
scoring). The presented scores are the mean of five independent runs.

## Appendix C: Mention Pair Features

Table H.0: Mention pair classifier performance on the ecbp development split
for different choices of $c$ and $k$. From left to right: Precision, Recall,
F1. “Coreferring” is used as the positive class.

Corpus | Gain | Feature | Type
---|---|---|---
action mention string distance |  • is-surface-form-identical • is-lemma-identical • surface-form-mlipns-distance • surface-form-levenshtein-distance |  String distance on two action mentions. We compare surface form and lemma for identity and compute Levenshtein and MLIPNS Shannaq and Alexandrov (2010) distances for lexical and phonetic distances. |
TF–IDF |  • document-similarity • surrounding-sentence-similarity • sentence-context-similarity |  We fit TF–IDF vectors on all given documents. We then compute the cosine similarity between TF-IDF vectors of text regions belonging to two mention pairs. For the regions we use (1) the full document, (2) the sentence surrounding the event mention, and (3) a sentence window of 5 sentences surrounding the mention (i.e., its context).
sentence embedding similarity |  • surrounding-sentence • doc-start |  We compute the cosine similarity between sentence representations of a sentence pair originating from a mention pair. We compare the sentences surrounding each event mention and the first sentence of a mention’s document. Sentence representions are computed with the Sentence-BERT framework Reimers and Gurevych (2019), using the pretrained distilbert-base-nli-stsb-mean-tokens model.
action mention embedding similarity |  • action-mention |  We compute a contextualized span representation of the action mention of each event mention and compute the cosine similarity of these representations for each mention pair. Span representations are created from the pretrained SpanBERT large model Joshi et al. (2020) using a window of five sentences surrounding each event mention.
spatial distance |  • distance-document-level-{geo-hierarchy-match/geodesic-distance} • distance-srl-level-{geo-hierarchy-match/geodesic-distance} • distance-sentence-level-{geo-hierarchy-match/geodesic-distance} • distance-closest-preceding-sentence-level-{geo-hierarchy-match/geodesic-distance} • distance-closest-overall-level-{geo-hierarchy-match/geodesic-distance} |  We obtain a location from each mention in five ways: (1) document-level, where we pick the first entity-linked location in a document, (2) srl-level, where we use srl to find the linked location expression attached to the mention action, (3) sentence-level, where we use the location expression closest to the mention action in the same sentence, (4) closest-sentence-level, where we use the closest preceding location expression from all previous sentences and (5), a combination which applies (2), (3), (4) in order until a location expression is found. For each location pair, we compute distances in two ways: (1) We compute the geodesic distance between the coordinates of both locations. (2) For each location, we follow the subdivision and country relations in DBpedia upwards (from more specific to less specific locations) to find a match between the two locations. The earlier a match is found, the smaller is the distance between the two locations.
temporal distance |  • distance-document-publish-level-{year/month/week/day/hour} • distance-document-level-{year/month/week/day/hour} • distance-srl-level-{year/month/week/day/hour} • distance-sentence-level-{year/month/week/day/hour} • distance-closest-preceding-sentence-level-{year/month/week/day/hour} • distance-closest-overall-level-{year/month/week/day/hour} |  Computes temporal distance between temporal expressions belonging to a mention pair on different date fields (difference ofdays, difference of hours, …). Multiple variants for finding temporal expressions exist: (1) the document publication date (where available), (2) document-level, where we pick the first temporal expression in a document, (3) srl-level, where we use srl to find the temporal expression attached to the mention action, (4) sentence-level, where we use the temporal expression closest to the mention action in the same sentence, (5) closest-sentence-level, where we use the closest preceding temporal expression from all previous sentences and (6), a combination which applies (3), (4), (5) in order until a temporal expression is found.
Wikidata embedding similarity |  • action-mention • semantic-role-args-{mean/variance/min/max} • surrounding-sentence-{mean/variance/min/max} • sentence-context-{mean/variance/min/max} • doc-start-{mean/variance/min/max} |  We obtain Wikidata QIDs for each DBpedia entity and map these to pretrained embeddings from PyTorch-BigGraph Lerer et al. (2019). For each mention in a mention pair, we look up the vectors of (1) the linked action mention (where available), (2) of all event components, (3) of all linked entities in the surrounding sentence, (4) of all linked entities in a 5-sentence window around the mention and (5) of all linked entities in the first three document sentences. Between each of these groups, we compute the pairwise cosine similarity between all vectors and retain the mean, variance, minimum and maximum similarity respectively.

## Appendix D: Corpus Splits

ecbp:

We used the official splits defined in the corpus readme file and filtered the
sentences according to the extra CSV file provided with the corpus. In Section
6.4 we only compare systems also using these splits.

fcct:

We used the tournaments 2010, 2012 and 2014 for training, 2016 for development
and 2018 for testing. We remove all mentions belonging to the other_event
cluster (see Section 4.2).

gvc:

No official splits are provided for this corpus. We compiled a list of all 241
gun violence incidents present in the corpus and the mapping from incident to
document. We shuffled the incidents and partitioned them randomly so that
train, dev and test contain 70/15/15 percent of all documents respectively.
The list of documents in each split is provided alongside our system
implementation. We remove all mentions with cluster ID 0: this cluster
contains mentions of generic events or unresolved cross-subtopic coreference
links.

## Appendix E: Feature Importance

Corpus | Gain | Feature | Type
---|---|---|---
ecbp | $0.342,106$ | is-lemma-identical | action string distance
| $0.248,146$ | surface-form-mlipns-distance | action string distance
| $0.130,495$ | action-mention | action mention embedding
| $0.085,002,8$ | is-surface-form-identical | action string distance
| $0.055,557,2$ | document-similarity | TF–IDF
| $0.051,721,8$ | context-similarity | TF–IDF
| $0.042,734,7$ | surface-form-levenshtein-distance | action string distance
| $0.027,600,8$ | surrounding-sentence-similarity | TF–IDF
| $0.016,636,1$ | doc-start | sentence embedding
fcct | $0.301,784$ | surface-form-mlipns-distance | action string distance
| $0.144,558$ | is-lemma-identical | action string distance
| $0.098,399,7$ | distance-closest-overall-level-year | temporal distance
| $0.080,500,5$ | is-surface-form-identical | action string distance
| $0.053,452,3$ | distance-sentence-level-year | temporal distance
| $0.052,424,3$ | surface-form-levenshtein-distance | action string distance
| $0.051,503,6$ | action-mention | action mention embedding
| $0.041,250,2$ | surrounding-sentence-similarity | TF–IDF
| $0.031,589,5$ | semantic-role-args-min | Wikidata embedding
| $0.024,731,3$ | distance-closest-overall-level-week | temporal distance
| $0.024,145,3$ | semantic-role-args-variance | Wikidata embedding
| $0.022,969,1$ | surrounding-sentence-min | Wikidata embedding
| $0.019,702,2$ | semantic-role-args-mean | Wikidata embedding
| $0.019,380,4$ | sentence-context-variance | Wikidata embedding
| $0.017,701,4$ | surrounding-sentence-variance | Wikidata embedding
| $0.015,908,6$ | surrounding-sentence-mean | Wikidata embedding
gvc | $0.254,793$ | surface-form-mlipns-distance | action string distance
| $0.126,396$ | distance-document-publish-level-week | temporal distance
| $0.125,259$ | distance-document-publish-level-month | temporal distance
| $0.075,458$ | document-similarity | TF–IDF
| $0.059,322,2$ | is-lemma-identical | action string distance
| $0.040,944,4$ | is-surface-form-identical | action string distance
| $0.035,529,2$ | distance-document-level-week | temporal distance
| $0.034,521,3$ | action-mention | action mention embedding
| $0.028,257,1$ | distance-document-level-month | temporal distance
| $0.023,262,8$ | surface-form-levenshtein-distance | action string distance
| $0.020,044,7$ | doc-start-min | Wikidata embedding
| $0.013,317$ | context-similarity | TF–IDF
| $0.012,053,5$ | distance-closest-overall-level-week | temporal distance
| $0.010,553,6$ | doc-start | sentence embedding
| $0.009,349,56$ | distance-closest-overall-level-year | temporal distance
| $0.009,182,12$ | distance-closest-pr-sent-level-week | temporal distance
| $0.008,627,37$ | distance-closest-overall-level-month | temporal distance
| $0.008,619,89$ | distance-closest-pr-sent-level-year | temporal distance
| $0.008,284,74$ | distance-closest-pr-sent-level-month | temporal distance
| $0.008,251,24$ | doc-start-variance | Wikidata embedding
| $0.007,491,47$ | distance-sentence-level-week | temporal distance
| $0.007,132,95$ | distance-document-publish-level-day | temporal distance
| $0.006,986,02$ | doc-start-mean | Wikidata embedding
| $0.006,920,19$ | surrounding-sentence-similarity | TF–IDF
| $0.006,746,16$ | doc-start-max | Wikidata embedding
| $0.006,421,45$ | sentence-context-max | Wikidata embedding
| $0.006,201,61$ | sentence-context-min | Wikidata embedding
| $0.006,156,17$ | distance-closest-pr-sent-level-day | temporal distance
| $0.005,889,71$ | distance-document-level-day | temporal distance
| $0.005,790,38$ | closest-pr-sent-lvl-geo-hier-match | location
| $0.005,747,41$ | surrounding-sentence-min | Wikidata embedding
| $0.005,665,89$ | surrounding-sentence | sentence embedding
| $0.005,555,67$ | sentence-context-mean | Wikidata embedding
| $0.005,268,8$ | sentence-context-variance | Wikidata embedding

## Appendix F: ECB+ Publication Date Annotation

The document publication date is an important piece of information for
grounding temporal expressions, particularly in news text. ecbp is the only
cdevcr corpus covered in this work for which document publication dates were
not annotated. Source URLs of the corpus articles, from which the publication
date could have been extracted automatically, are unfortunately only provided
for documents in the ecbp subtopics added at a later point by Cybulska and
Vossen (2014b). For the initial ecb, no URLs are present. We manually
annotated the missing date by inspecting the first few sentences of each
document. LABEL:tab:ecbp_publish_date_annotation displays the number of
documents for which the publication date could be identified.

|  ecb |  ecbp
---|---|---
Date found | 0 | 460
No date found | 480 | 42

Unfortunately, no publication dates were found in the ecb half of the corpus.
In light of these results, we decided against using these annotations in our
experiments since it may have given systems an unfair advantage in deciding
whether a document belongs to one of the ecb or ecbp subtopics. We
nevertheless release our annotations in the hopes that they will be useful for
future research.

## Appendix G: Hyperparameter Optimization Procedure

We describe our approach for optimizing the hyperparameters of the feature-
based system.

We apply repeated k-fold cross-validation to obtain reliable results. To
define folds, we first partition the documents based on their topic (ecbp) or
subtopic (fcct, gvc). From these partitioned document sets, folds are created,
based on which we generate mention pairs. Compared to the naïve approach of
creating folds from all possible mention pairs of a corpus split, this
approach guarantees that each mention in the respective test fold is unseen,
opposed to just the mention pair being unseen (with the two constituent
mentions likely having been seen at training time), which provides a more
faithful testing scenario. By using topics or subtopics for partitioning, we
ensure a high number of coreferring pairs in the folds and guide the
hyperparameter search towards models which should generalize better across
topics or subtopics.

The optimization algorithm is shown in LABEL:fig:hyperopt.

We use the optuna framework Akiba et al. (2019) for sampling increasingly
optimal sets of hyperparameters and use a configurable maximum duration as the
stopping criterion. Optimization of the agglomerative clustering step is
performed similarly, with the difference of generating a test clustering in
line 12 and using the LEA F1 metric instead of F1 for binary classification in
line 13.

## Appendix H: Mention Pair Generation at Training Time

##### Approach

We explain our approach for determining the number of coreferring mention
pairs to randomly sample for each event during training. Given a set of events
$\mathbb{E}=\\{e_{1},e_{2},\ldots\\}$ and the function
$\mathrm{m}:\mathbb{E}\rightarrow\mathbb{N}$ providing the number of mentions
for an event (i.e. the cluster size), we define
$\mathrm{pairs}_{\text{coref}}:\mathbb{E}\rightarrow\mathbb{N}$, the number of
coreferring pairs to sample for an event, as:

$\displaystyle\mathrm{pairs}_{\text{coref}}(e)$
$\displaystyle=\left\lceil\left(\mathrm{m}(e)-1\right)\cdot\min\left(\mathrm{undersample}(e),\frac{\mathrm{m}(e)}{2}\right)\right\rceil$
(H.1) $\displaystyle\mathrm{undersample}(e)$
$\displaystyle=c+\mathrm{m}(e)^{1-\mathrm{cdf}\left(\mathrm{m}(e)\right)}-1$
(H.2) where $c\in\mathbb{R}$ is a hyperparameter and
$\mathrm{cdf}:\mathbb{N}\rightarrow\mathbb{Q}$ is the percentage of all
mentions in $\mathbb{E}$ originating from clusters up to a given size $i$:
$\displaystyle\mathrm{cdf}(i)$
$\displaystyle=\frac{\sum_{\left\\{e\in\mathbb{E}\mid\mathrm{m}(e)\leq
i\right\\}}\mathrm{m}(e)}{\sum_{e\in\mathbb{E}}\mathrm{m}(e)}$ (H.3)

For the event $e_{\text{largest}}$ with the most mentions in the given data
split, this results in $\mathrm{cdf}(\mathrm{m}(e_{\text{largest}}))=1$,
therefore
$\mathrm{pairs}_{\text{coref}}(e_{\text{largest}})=\left\lceil\left(\mathrm{m}(e)-1\right)\cdot
c\right\rceil$. Hence, $c$ controls the amount of coreferring mention pairs
sampled from large clusters. The number of pairs to sample transitions
smoothly from linear to quadratic the smaller a cluster is with respect to the
overall distribution of cluster sizes in the dataset.

Having sampled coreferring pairs for each event, we determine their
coreference link type (within-document, within-subtopic, etc.). For each type
of coreference link (within-document, within-subtopic, etc.), the maximum
number of non-coreferring pairs generated is $k$ times the number of generated
coreferring pairs. We ensure that the number of non-coreferring pairs
increases per link type (so that within-document $<$ within-subtopic
$<\ldots$).

##### Impact of Hyperparameters

To evaluate how hyperparameters $c$ and $k$ affect performance, we perform
exhaustive grid search with different choices of $c,\,k$ when training the
mention pair classifier component of our feature-based approach on the ecbp
training split and evaluating it on the ecbp development split. We test
$c\in\\{2^{-3},2^{-2},\ldots,2^{5}\\}$ and $k\in\\{1,2,4,8,12,16,24,32,48\\}$.
The entire training split can produce $7.2\cdot 10^{6}$ pairs. With the
smallest choice of $(c=2^{-3},k=1)$, $0.28\cdot 10^{6}$ pairs are generated
(just $4\cdot 10^{3}$ pairs when excluding non-coreferring cross-topic pairs),
whereas the largest choice $(c=2^{5},k=48)$ yields $0.47\cdot 10^{6}$ pairs
($0.2\cdot 10^{6}$ pairs when excluding non-coreferring cross-topic pairs).

For each $(c,k)$ combination, we compute precision, recall and F1 for each
type of coreference link (using the mean of five independent trials to
mitigate noise). We further aggregate these results by computing the macro-
average over the four link types to produce one precision/recall/F1 value each
per $(c,k)$ tuple, shown in LABEL:fig:undersampling_hyperparameters.

As visualized by the plots, $k$ controls the precision/recall tradeoff, with
higher $k$ (larger proportion of non-coreferring training pairs) leading to
high precision but low recall. The choice of $c$ has a smaller impact on
performance. Overall, considering F1 scores, the amount of coreferring mention
pairs generated from large clusters can be reduced significantly (with $c$
chosen as low as $2^{-3}$) without loss in performance, unless many non-
coreferring pairs are used (high $k$). This indicates that, for ecbp, there is
little benefit in generating all possible coreferring mention pairs for
training, and that achieving a broad selection of mention pairs from many
different events is more important. Based on these results, and taking into
account the distribution of cluster sizes in each corpus (see Figure 2), we
chose $(c=8,k=8)$ for ecbp and the similarly distributed gvc in the main
experiments. For experiments involving fcct, we chose $(c=2,k=8)$ to reduce
the impact of its few large, mostly redundant clusters on training.

[include=main] [name=Corpus Acronyms, include=corpus, display=all]
[name=System Acronyms, include=system, display=all]

## References