# On Event Individuation for Document-Level Information Extraction

William Gantt1 Reno Kriz*2 Yunmo Chen*2
Siddharth Vashishtha*1 Aaron Steven White1
1 University of Rochester 2 Johns Hopkins University
{wgantt@ur.|svashis3<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

As information extraction (IE) systems have grown more capable at whole-
document extraction, the classic task of _template filling_ has seen renewed
interest as a benchmark for evaluating them. In this position paper, we call
into question the suitability of template filling for this purpose. We argue
that the task demands definitive answers to thorny questions of _event
individuation_ — the problem of distinguishing distinct events — about which
even human experts disagree. We show through annotation studies and error
analysis that this raises concerns about the usefulness of template filling
evaluation metrics, the quality of datasets for the task, and the ability of
models to learn it. Finally, we consider possible solutions.

##  1 Introduction

00footnotetext: *Equal contribution.

_Template filling_ involves extracting structured objects called _templates_
from a document, each of which describes a complex event and its
participants.111Following standard usage, we use _template_ to denote an
instance of a template _type_ (cf. Chen et al. (2022)). The task comprises
both _template detection_ — determining the number of templates present in a
document and their types — as well as _slot filling_ or _role-filler entity
extraction_ (REE) — assigning extracted entities to _slots_ in those templates
that characterize the roles entities play in the event. Since documents
routinely describe multiple events, documents in template filling datasets
regularly feature multiple templates, often of the same type. Correctly
_individuating_ events is thus a crucial and challenging component of the
task. Here, we argue that whereas there is often more than one reasonable way
to individuate events, template filling admits no such pluralism, which
undermines its value as a benchmark for document-level IE. To support our
claim, we draw evidence from several sources:

  * •

A demonstration of how template filling _evaluation metrics_ draw arbitrary
distinctions between nearly identical system predictions that differ only in
their individuation decisions.

  * •

A (re-)annotation study, whose results suggest that template filling
_datasets_ reflect questionable individuation judgments about which even human
experts with access to the annotation guidelines may disagree.

  * •

An error analysis on recent template filling _models_ , showing that they
struggle to learn the individuation rules latent in these datasets.

Finally, we conclude by outlining how the IE community might respond to these
issues.

Figure 1: An excerpted document from the MUC-4 training data. Does this
describe one bombing or two? We argue that this question is beside the point
of IE.

##  2 Background

Figure 2: Gold MUC-4 templates for the document shown in Figure 1. Plausibly,
the document describes just a single bombing (left template), yet predicting
only this template would carry a substantial penalty in score (CEAF-REE F1 =
57.14%), even though it recovers all annotated entities and assigns them the
correct roles.

##### Template Filling

Template filling is arguably _the_ foundational task of IE (Grishman, 2019),
with its origins in the Message Understanding Conferences (MUCs) of the 1980s
and 1990s (Sundheim, 1992; Grishman and Sundheim, 1996). These evaluations
produced the MUC-4 dataset, which remains a primary benchmark for both
template filling (Chambers and Jurafsky, 2011; Du et al., 2021b; Chen et al.,
2022; Das et al., 2022) and for the REE subtask (Patwardhan and Riloff, 2009;
Huang and Riloff, 2011, 2012; Huang et al., 2021; Du et al., 2021a). MUC-4
focuses on incidents of political violence in Latin America and presents an
ontology of six template types, all with the same five slots.

More recently, the IARPA BETTER program has released the BETTER Granular
dataset, which, though new, has already begun to receive attention
(Holzenberger et al., 2022; Chen et al.,
2022).222https://www.iarpa.gov/index.php/research-programs/better BETTER also
features six template types, but unlike MUC-4, each template has a distinct
set of slots, which may have other types of fillers beyond entities. We take
both datasets as case studies.333Details on each dataset can be found in
Appendix A.

##### Event Individuation

At the core of template filling is the question of event individuation:
systems must determine when one event (i.e. template) is being described or
several. This is a profoundly difficult conceptual problem that long antedates
the field of IE, with a rich history in philosophy and linguistics (Lemmon,
1967; Kim, 1973; Quine, 1985; Cleland, 1991; Davidson, 2001, i.a.). At this
level, the problem is well beyond IE’s purview, as IE systems plainly should
not be made to take a hard line on difficult metaphysical questions.

Instead, for IE purposes, we might hope to find some _rules of thumb_ for
event individuation that agree broadly with human judgments, and that can be
consistently applied during annotation. Work on template filling itself has
had surprisingly little to say on such rules. The fullest treatment of this
subject comes rather from work on event coreference by Hovy et al. (2013), who
posit conditions for full event _identity_ — the inverse of the individuation
problem. Appealing to the notion of a _discourse element_ (DE), an event or
entity that is referenced in the text, they stipulate that “two mentions fully
corefer if their activity/event/state DE is identical in all respects, as far
as one can tell from their occurrence in the text. (In particular, _their
agents, location, and time are identical or compatible_).”

Differences in times, locations, and “agents” (i.e. participants) between two
event descriptions are intuitively each sufficient conditions for
individuating events and, importantly, accord with the guidelines for
individuating templates provided for MUC-4 and for BETTER (see Appendix A).
Template _types_ offer another, coarser means of distinguishing events. Here,
we focus analysis on individuation _within type_ , since this is where most of
the difficulty lies, and consider whether these guidelines are adequate to
enable annotators and models to converge on similar individuation decisions.

##  3 Individuation and Model Evaluation

To show why individuation matters for template filling, we begin by
considering template filling evaluation. In contrast to events in _event
extraction_ , which are associated with lexical _triggers_ , templates are not
anchored to specific lexical items. Given a predicted template, there are
often multiple gold templates one could use as a reference, and one must
therefore establish an _alignment_ between predictions and references before
scoring. Historically, this is done by selecting the alignment that produces
the highest overall score, given a similarity function $\phi(T_{R},T_{P})$
between reference and predicted templates $T_{R}$ and $T_{p}$, and given the
constraint that each reference template is aligned to at most one predicted
template with matching type.444With this constraint, finding the optimal
alignment is thus a maximum bipartite matching problem.

Though they differ in their choice of $\phi$, all proposed evaluation metrics
for template filling require determining such an alignment, which is why
correct template individuation is so important for evaluation: If
individuation decisions in the reference templates are ill-grounded, this can
yield punishingly large differences in score for semantically very similar
predictions.

We illustrate this with the example document from MUC-4 shown in Figure 1 and
the gold templates in Figure 2.555Other challenging examples are shown in
Appendix B. Although arguably only one bombing is described in the document,
the gold annotations feature two templates — one for the bomb that detonated
in the restaurant and one for the bomb that was deactivated.666We note the
oddity of the missing annotation of second device as a weapon, but this is not
important to the argument. However, a system that predicts only the left
template will be heavily penalized for having “missed” the one on the right,
even though it identifies all relevant entities and assigns them the correct
roles. Under the standard alignment-based metric for MUC-4, CEAF-REE (Du et
al., 2021a), such a system would receive a score of only 57.14% F1.

Thus, because systems are tasked with reproducing _particular_ (often
idiosyncratic) individuation decisions, distinctions without much semantic
difference can reverberate as significant differences in score. Alignment-
based metrics therefore present a serious obstacle to making meaningful
comparisons between models and so to tracking progress.

##  4 Individuation Replicability

One could argue that alignment-based metrics _per se_ are not the problem, but
rather questionable individuation decisions embodied in the annotations. If
those decisions reflect real differences between events, then the difference
between predicting one template and predicting two plausibly _should_
correspond to a large difference in score. We next ask whether those decisions
_are_ well-founded in both MUC-4 and BETTER.

To investigate this, we conduct a re-annotation study of a random subset of
documents from each dataset and evaluate the level of agreement between these
new annotations and the original ones. If the original annotations reflect
sound individuation decisions, then we would expect them to be recoverable
with high accuracy by annotators versed in their ontologies.

We recruited three NLP researchers, each with multiple IE publications, to
reannotate a subset of documents from the MUC-4 and BETTER training splits for
template detection only, asking how many templates of each type are present in
each document. Documents were selected via stratified sampling, with the goal
of obtaining 10 documents annotated for each template type. Within type, we
further stratified by number of templates. This procedure yielded 42 documents
for MUC-4 and 60 for BETTER.777Study details are in Appendix C. Annotators
were provided with complete descriptions of both ontologies, as well as
relevant excerpts from the annotation guidelines. Importantly, these excerpts
contain instructions on template individuation consistent with the criteria
from Hovy et al. (2013) discussed in §2.

We consider the level of agreement among the three annotators alone
($\alpha$), and when additionally incorporating the gold annotations
($\alpha^{+}$). Table 1 presents nominal Krippendorff’s $\alpha$
(Krippendorff, 2018) in aggregate and by template type. While agreement on
certain MUC-4 template types is reasonably high among annotators alone, we
find that _aggregate_ agreement is at best moderate for MUC-4 ($\alpha=0.63$)
and consistently low for BETTER ($\alpha=0.25$). This is disheartening
considering the annotators’ qualifications and their access to the annotation
guidelines. Still worse, agreement drops further when including the gold
annotations ($\alpha^{+}=0.56$, $0.22$). If human experts cannot reliably
agree either among themselves or with the gold annotations, this suggests that
the individuation problem is ill-posed.

BETTER MUC-4 Type $\alpha$ $\alpha^{+}$ Type $\alpha$ $\alpha^{+}$ Corruption
0.24 0.27 Arson 0.58 0.35 Disaster 0.21 0.21 Attack 0.44 0.42 Epidemic 0.36
0.28 Bombing 0.82 0.66 Migration 0.20 0.14 Kidnapping 0.81 0.73 Protest 0.13
0.07 Robbery* 1.00 1.00 Terrorism 0.30 0.23 Aggregate 0.25 0.22 Aggregate 0.63
0.56

Table 1: Krippendorff’s $\alpha$ for IAA on the number of templates of each
type for the reannotated MUC-4 and BETTER documents. $\alpha=1$ indicates full
agreement; $\alpha=0$ is agreement at chance. *The sampling procedure excluded
samples of forced work stoppage templates and yielded only three robbery
templates, so robbery results should be taken with caution .

##  5 Do Models Learn to Individuate?

Given these observations, we now ask what the consequences are for our IE
models. Even if expert annotators struggle to replicate the original
individuation decisions, it is still possible that the datasets exhibit some
_internal_ consistency, and that a model could thus learn their template count
distributions.

To explore this possibility, we consider three recent top-performing models
for template filling: IterX (Chen et al., 2022), the current state-of-the-art;
GTT (Du et al., 2021b), the previous best model; and TempGen (Huang et al.,
2021), a third competitive system. We investigate the predictions of all three
systems on the MUC-4 test split, as well as the predictions of IterX on the
BETTER test split.888BETTER predictions are available only for IterX.
Additional model details and analysis in Appendix D. While some prior work has
attempted a complete taxonomy of template filling model errors (Das et al.,
2022), our concern here is with detection errors only.

First, we note that all three models make a significant number of detection
errors. Even the strongest detection model (GTT) still incorrectly predicts
the total number of templates on 32% of MUC-4 test documents (Table 2, top
row).

Second, we observe that the errors of all three models are overwhelmingly
misses (rather than spurious detections): among documents with detection
errors, IterX, GTT, and TempGen predicted fewer templates than the reference
on 72.5%, 81.3%, and 80.0% of them, respectively (Table 2, second row). These
findings are consistent not only with prior error analyses (Das et al.,
2022),999See §7.2 of Das et al. 2022: “The main source of error for both early
and modern models is missing recall due to missing templates and missing role
fillers.” but also with our observations thus far: If individuations are close
to arbitrary from the model’s perspective, it can generally do no better than
to collate all entities relevant to a particular type of event into a single
template of that type.

If this is in fact what these models are learning to do, then _collapsing_
predicted templates into a single aggregate template per type (taking a union
over slot fills) should not substantially alter overall scores. Doing this for
all model predictions and recomputing scores (Table 2, bottom row) bears this
out: across the three models, there is an average drop of just over a single
point in overall score on MUC-4 and a mere 0.3 points on BETTER. By
comparison, collapsing all _gold_ templates by type yields a drop of about
12.7 points in overall score ($\text{max score}=87.3$) on MUC-4 and 24.2
points on BETTER ($\text{max score}=75.8$).

BETTER MUC-4 IterX IterX GTT TempGen |gold| $\neq$ |pred| (%) 34.4 34.5 32.0
37.5 of which |gold| > |pred| (%) 72.7 72.5 81.3 80.0 Temp. P 89.7 79.5 81.1
82.7 Temp. R 74.5 59.7 57.7 57.2 Temp. F1 (original) 30.0 53.0 50.2 47.2 Temp.
F1 (collapsed) 29.7 51.4 48.8 46.9

Table 2: Template detection stats for template filling models. P/R/F1 scores
use CEAF-REE for MUC-4 and the official scorer for BETTER. “|gold|” and
“|pred|” denote number of gold and predicted templates.

##  6 Conclusion

The value of template filling as an IE benchmark is undermined both by
datasets that make dubious individuation decisions and by metrics that sharply
penalize model deviations from those decisions. So what can be done? One
possibility is to make standard the setting from §5 in which templates are
aggregated by type. This presents a clearer picture of whether systems are
getting the big things right (distinguishing events of different types), but
it gives up on difficult, within-type individuation.

Yet, individuation still matters. But we argue that it does so only insofar as
it is a result of other important, usually uncontroversial facts (cf. Figure
1: one bomb detonated; the other did not). _These_ facts — about things like
time, location, and irrealis status — are what matter. The trouble is that the
distribution of template counts over these variables is often too noisy for
models (or humans) to learn well.

A second possibility is thus to make these variables explicit. Recent large
language models (LLMs) have proven highly capable at extractive question
answering (QA), and much IE research has exploited this (Liu et al., 2020; Du
and Cardie, 2020; Li et al., 2020; Holzenberger et al., 2022). However, what
is missing from this work is questions beyond those about event types and
roles that target individuation-relevant properties like the ones
above.101010But see Gantt et al. (2022) for fine-grained questions about event
parthood. We envision a recasting of template filling as QA that integrates
such questions and leverages the capabilities of LLMs to answer them (Gardner
et al., 2019; He et al., 2015; White et al., 2016, 2020). This would require
recasting not only slots but also _individuations_ as sets of QA pairs about
these properties, which we believe is feasible for MUC-4 and BETTER. Notably,
this process would eliminate the need for template alignments.

Finally, whereas past work has largely treated questions independently, the
structure of templates calls for a conditional, iterative form of QA,
beginning with coarse-grained queries about event types and gradually homing
in on fine-grained ones about entities. In Appendix E, We sketch how this
could work, showing actual zero-shot generations from GPT-3 (Brown et al.,
2020). Such an approach may not only yield better results, but would be more
true to real world information needs.

##  7 Limitations

Some researchers have framed structurally similar tasks — notably, $N$-ary
relation extraction — as template filling (Huang et al., 2021; Chen et al.,
2022). However, as this paper concerns template filling for _events_ , the
issues we raise about event individuation do not obviously extend to
relations. Individuation of relations has its own challenges, but insofar as
relations are more abstract than events and not spatially or temporally
localized, they may be more readily individuable solely on the basis of the
entities they involve.

Further, as noted in §3, other document-level IE tasks that focus on lexical
triggers — notably, (multi-sentence) _argument linking_ (also called _event
argument extraction_ or _implicit semantic role labeling_ ; O’Gorman, 2019;
Ebner et al., 2020; Li et al., 2021, i.a.) — do not use the alignment-based
metrics described here and so do not run into the same evaluative
problems.111111Argument linking also goes by the names _event argument
extraction_ (EAE) and _implicit semantic role labeling_. However, this is only
because in working at the lexical level, they do not engage with any of the
important questions surrounding individuation to begin with.

## References

  * Baker et al. (1998) Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998.  The Berkeley FrameNet project.  In _36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1_ , pages 86–90, Montreal, Quebec, Canada. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chambers and Jurafsky (2011) Nathanael Chambers and Dan Jurafsky. 2011.  Template-based information extraction without the templates.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 976–986, Portland, Oregon, USA. Association for Computational Linguistics.
  * Chen et al. (2022) Yunmo Chen, William Gantt, Weiwei Gu, Tongfei Chen, Aaron Steven White, and Benjamin Van Durme. 2022.  Iterative document-level information extraction via imitation learning.  _arXiv preprint arXiv:2210.06600_.
  * Cleland (1991) Carol Cleland. 1991.  On the individuation of events.  _Synthese_ , pages 229–254.
  * Das et al. (2022) Aliva Das, Xinya Du, Barry Wang, Kejian Shi, Jiayuan Gu, Thomas Porter, and Claire Cardie. 2022.  Automatic error analysis for document-level information extraction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3960–3975, Dublin, Ireland. Association for Computational Linguistics.
  * Davidson (2001) Donald Davidson. 2001.  _Essays on Actions and Events: Philosophical Essays Volume 1_.  Clarendon Press.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Du and Cardie (2020) Xinya Du and Claire Cardie. 2020.  Event extraction by answering (almost) natural questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 671–683, Online. Association for Computational Linguistics.
  * Du et al. (2021a) Xinya Du, Alexander Rush, and Claire Cardie. 2021a.  GRIT: Generative role-filler transformers for document-level event entity extraction.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 634–644, Online. Association for Computational Linguistics.
  * Du et al. (2021b) Xinya Du, Alexander Rush, and Claire Cardie. 2021b.  Template filling with generative transformers.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 909–914, Online. Association for Computational Linguistics.
  * Ebner et al. (2020) Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020\.  Multi-sentence argument linking.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8057–8077, Online. Association for Computational Linguistics.
  * Fleiss (1971) Joseph L Fleiss. 1971.  Measuring nominal scale agreement among many raters.  _Psychological bulletin_ , 76(5):378.
  * Gantt et al. (2022) William Gantt, Lelia Glass, and Aaron Steven White. 2022.  Decomposing and recomposing event structure.  _Transactions of the Association for Computational Linguistics_ , 10:17–34.
  * Gardner et al. (2019) Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, and Sewon Min. 2019\.  Question answering is a format; when is it useful?  _arXiv preprint arXiv:1909.11291_.
  * Grishman (2019) Ralph Grishman. 2019.  Twenty-five years of information extraction.  _Natural Language Engineering_ , 25(6):677–692.
  * Grishman and Sundheim (1996) Ralph Grishman and Beth Sundheim. 1996.  Message Understanding Conference- 6: A brief history.  In _COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics_.
  * He et al. (2015) Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015.  Question-answer driven semantic role labeling: Using natural language to annotate natural language.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 643–653, Lisbon, Portugal. Association for Computational Linguistics.
  * Holzenberger et al. (2022) Nils Holzenberger, Yunmo Chen, and Benjamin Van Durme. 2022.  Asking the right questions in low resource template extraction.  _arXiv preprint arXiv:2205.12643_.
  * Hovy et al. (2013) Eduard Hovy, Teruko Mitamura, Felisa Verdejo, Jun Araki, and Andrew Philpot. 2013\.  Events are not simple: Identity, non-identity, and quasi-identity.  In _Workshop on Events: Definition, Detection, Coreference, and Representation_ , pages 21–28, Atlanta, Georgia. Association for Computational Linguistics.
  * Huang et al. (2021) Kung-Hsiang Huang, Sam Tang, and Nanyun Peng. 2021.  Document-level entity-based extraction as template generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5257–5269, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Huang and Riloff (2011) Ruihong Huang and Ellen Riloff. 2011.  Peeling back the layers: Detecting event role fillers in secondary contexts.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 1137–1147, Portland, Oregon, USA. Association for Computational Linguistics.
  * Huang and Riloff (2012) Ruihong Huang and Ellen Riloff. 2012.  Modeling textual cohesion for event extraction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 26, pages 1664–1670.
  * Kim (1973) Jaegwon Kim. 1973.  Causation, nomic subsumption, and the concept of event.  _The Journal of Philosophy_ , 70(8):217–236.
  * Krippendorff (2018) Klaus Krippendorff. 2018.  _Content analysis: An introduction to its methodology_.  Sage publications.
  * Lemmon (1967) John Lemmon. 1967.  Comments on d. davidson’s ’the logic form of action sentences’.  In Nicholas Rescher, editor, _The Logic of Decision and Action_ , pages 96–103. University of Pittsburgh Press, Pittsburgh.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2020) Fayuan Li, Weihua Peng, Yuguang Chen, Quan Wang, Lu Pan, Yajuan Lyu, and Yong Zhu. 2020.  Event extraction as multi-turn question answering.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 829–838, Online. Association for Computational Linguistics.
  * Li et al. (2021) Sha Li, Heng Ji, and Jiawei Han. 2021.  Document-level event argument extraction by conditional generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 894–908, Online. Association for Computational Linguistics.
  * Liu et al. (2020) Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020.  Event extraction as machine reading comprehension.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1641–1651, Online. Association for Computational Linguistics.
  * O’Gorman (2019) Timothy J O’Gorman. 2019.  _Bringing together computational and linguistic models of implicit role interpretation_.  Ph.D. thesis, University of Colorado at Boulder.
  * Patwardhan and Riloff (2009) Siddharth Patwardhan and Ellen Riloff. 2009.  A unified model of phrasal and sentential evidence for information extraction.  In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_ , pages 151–160, Singapore. Association for Computational Linguistics.
  * Quine (1985) Willard Van Orman Quine. 1985.  Events and reification.  _Actions and events: Perspectives on the philosophy of Donald Davidson_ , pages 162–171.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * Sundheim (1992) Beth M. Sundheim. 1992.  Overview of the fourth Message Understanding Evaluation and Conference.  In _Fourth Message Uunderstanding Conference (MUC-4): Proceedings of a Conference Held in McLean, Virginia, June 16-18, 1992_.
  * White et al. (2016) Aaron Steven White, Drew Reisinger, Keisuke Sakaguchi, Tim Vieira, Sheng Zhang, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2016.  Universal decompositional semantics on Universal Dependencies.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1713–1723, Austin, Texas. Association for Computational Linguistics.
  * White et al. (2020) Aaron Steven White, Elias Stengel-Eskin, Siddharth Vashishtha, Venkata Subrahmanyan Govindarajan, Dee Ann Reisinger, Tim Vieira, Keisuke Sakaguchi, Sheng Zhang, Francis Ferraro, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2020.  The universal decompositional semantics dataset and decomp toolkit.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 5698–5707, Marseille, France. European Language Resources Association.

##  Appendix A Dataset Details

This appendix provides more detailed information on the MUC-4 and BETTER
Granular datasets, as well as their guidelines for individuating templates.
For descriptions and discussions of the alignment-based evaluation metrics for
both datasets, we refer the reader to Appendix D in Chen et al. (2022).

###  A.1 MUC-4

####  A.1.1 Ontology and Dataset Statistics

| Train | Dev | Test
---|---|---|---
documents | 1,300 | 200 | 200
documents w/ templates | 700 | 116 | 126
total templates | 1,114 | 191 | 209
attack | 636 | 97 | 127
bombing | 298 | 50 | 55
kidnapping | 114 | 31 | 14
arson | 43 | 8 | 3
robbery | 17 | 3 | 1
forced work stoppage | 6 | 1 | 4
Table 3: Summary statistics of the MUC-4 dataset. Included in the total
templates figures are several other templates that are cross-classified as
attack / bombing or bombing / attack. There is one such template in dev and
five in test.

MUC-4 features six template types: Arson, Attack, Bombing, Forced Work
Stoppage, Kidnapping, and Robbery. All template types share the same set of
five slots: PerpInd (an individual perpetrator), PerpOrg (a group or
organizational perpetrator), Victim (sentient victims of the incident), Target
(physical objects targeted by the incident), and Weapon. All five slots accept
entity-valued fillers (i.e. sets of coreferent entity mentions). The type of
each template instance is specified as a sixth slot, incident type. Complete
dataset statistics are provided in Table 3.

####  A.1.2 Multi-Template Annotation

Below is an excerpt from the MUC-4 dataset documentation, describing the
conditions under which multiple templates are to be annotated for a
document.121212This excerpt was taken from the guidelines at the following
url:
https://github.com/xinyadu/gtt/blob/master/data/muc/raw_files/muc34/TASK/TASKDOC/templ-
doc-basic/template-doc-part1.v7 Note that the criteria for annotating multiple
instances here closely align with those presented in the main text from Hovy
et al. (2013). Note also that although the location, date, and category slots
are annotated in the original data, they are not included in the templates and
are ignored for purposes of evaluation. The location and date slots are self-
explanatory; the category slot was used to indicate whether or not the event
was an instance of “state-sponsored violence,” meaning that the perpetrator
(PerpInd / PerpOrg) is a member of the government, military, or police.

> If an article discusses more than one instance of the same relevant type of
> terrorist incident, each such incident should be captured in a separate
> template. A “separate instance” is one which has a different filler for the
> location, date, category, or perpetrator slot. If no distinctions of these
> kinds are made in the text, the instances should be incorporated into a
> single template, e.g., if the article describes “attacks” by “rebels” on
> “ten power stations” and does not give varying information on the location
> of those targets, etc. Note that the level of granularity is defined by what
> can go in a slot; thus, an article that describes two bombings on targets on
> different streets in the same neighborhood should be captured in one
> template rather than two, since the location slot cannot have a filler that
> is at the level of granularity of a street.

The bombing example discussed here is particularly interesting when compared
to the bombing described in Figure 1 and annotated in Figure 2. According to
the principle outlined here, it certainly seems as though the document in
Figure 1 should have a single bombing template annotated, rather than two, as
no distinctions between the annotated bombings can be drawn on the basis of
the slots listed.

Alongside location, date, and category, an irrealis-focused stage of execution
slot was annotated for the original data, and indicated whether an event was
accomplished, attempted, or only threatened. This is in fact the basis for the
distinction between the templates in Figure 2. The existence of these
annotations is good news in that they may help in a recasting effort of the
sort proposed in §6, but bad news in that they may form the basis for
individuation decisions (e.g. Figure 2) without actually constituting part of
the task.

###  A.2 BETTER

####  A.2.1 Ontology and Dataset Statistics

| Phase I | Phase II
---|---|---
| Train | Dev | Test | Train | Dev | Test
documents | 94 | 20 | 20 | 168 | 34 | 32
documents w/ templates | 88 | 19 | 19 | 168 | 34 | 32
templates | 252 | 56 | 47 | 255 | 57 | 57
corruption | 63 | 10 | 11 | 3 | 0 | 0
disaster* | 0 | 0 | 0 | 68 | 11 | 11
epidemic | 31 | 7 | 1 | 76 | 23 | 23
migration* | 0 | 0 | 0 | 81 | 20 | 20
protest | 109 | 37 | 25 | 12 | 2 | 2
terrorism | 49 | 2 | 10 | 15 | 1 | 1
Table 4: Summary statistics of the Phase I and Phase II BETTER Granular
datasets. * Denotes that this template type is not part of the Phase I
ontology.

The BETTER Granular Phase I and Phase II datasets collectively feature six
template types, but with a greater diversity of topics than the MUC-4
templates: Corruplate (incidents of corruption), Disasterplate (natural
disasters), Displacementplate (human migration events), Epidemiplate
(epidemics), Protestplate (protests), and Terrorplate (acts of
terrorism).131313The data is freely available for download here:
https://ir.nist.gov/better/. While some slots are common to two or more of
these template types, most slots are unique to each type, reflecting event-
specific roles, similar to those in FrameNet (Baker et al., 1998). Complete
dataset statistics are provided in Table 4.

####  A.2.2 Multi-Template Annotation

We are unable to quote from the BETTER documentation, as it has not yet been
made public, though we will attempt to summarize the relevant considerations
for multi-template annotation as we do above for MUC-4. Like in MUC-4, the
BETTER documentation indicates that multiple templates may be annotated when
the times or locations of the events described differ from each other. Some
irrealis and time annotations are provided, but these are associated with
specific fillers of event-valued slots, and are not used to distinguish whole
templates, as the stage of execution slot is for MUC-4. Multiple templates of
the same type may also be required when those templates apply to different
people (e.g. different persons charged in a corruption event). However, this
criterion is somewhat opaque, as many templates feature multiple individuals
as fillers of the same slot, and it is unclear exactly when this condition
merits individuating a given set of events.

##  Appendix B Additional Difficult Examples

Lest the example presented in Figure 1 appear cherry-picked, we present here
additional examples of documents from both datasets that present challenging
individuation decisions that also do not have a single, obviously correct
answer. Such examples are abundant.

###  B.1 MUC-4 Examples

#### Example 1

Document Text:

> Two vehicles were destroyed and an unidentified office of the Agriculture
> and Livestock Ministry was heavily damaged following the explosion of two
> bombs yesterday afternoon. The National Fire Department reported that
> following the explosions at 1830, a fire erupted that partially destroyed an
> unidentified office in the Agriculture and Livestock Ministry Building. It
> was reported that two vehicles in the area where the bombs were detonated
> were destroyed and several houses in the area were damaged. The damaged
> offices are located at 123 A Avenue in the San José neighborhood in the
> western section of San Salvador. Fortunately, there were no casualties
> reported as a result of this incident, for which the FMLN guerrillas are
> being held responsible.141414The MUC-4 source text is uncased. We have added
> casing to all quoted MUC-4 texts for readability.

Annotated Templates: 1 bombing template
Discussion: This example provides an interesting foil for the one in Figure 1
and Figure 2. Here, just as in that earlier example, two bombs are involved.
But unlike the earlier example, only a single template is annotated. From the
text, it appears that the bombs were likely planted near each other, but so
too were the pair of bombs in Figure 1 — in the same restaurant, in fact. The
only distinguishing feature between these examples is the fact that in Figure
1, only one of the bombs actually detonated. This is an important fact, but
whether one actual detonation and one non-detonation makes for two “bombings”
is dubious and is hardly the only reasonable reading, as we discuss in the
main text. Moreover, were the second bomb to actually have detonated, the
annotation guidelines (Appendix A) imply that we would instead (again) have a
single bombing, owing to the equivalence between all the relevant slots in the
two templates. This is idiosyncratic at the very least, and possibly
inconsistent.

#### Example 2

Document Text:

> Military sources reported today that fighting broke out on the night of 5
> March in the war-torn Chalatenango department to the north of San Salvador,
> where the guerrillas suffered eight casualties. The Armed Forces Press
> Committee reported that the clashes took place near San Isidro Labrador,
> where three guerrillas were killed and five others were wounded. The
> Farabundo Marti National Liberation front (FMLN), which has been fighting
> the U.S.-backed army for the last 10 years, has traditionally maintained a
> high profile in strategic areas of Chalatenango. Meanwhile, urban guerrilla
> commandos harassed troops belonging to the 1st Infantry Brigade last night
> at the Mariona Penitentiary, located on the northern outskirts of San
> Salvador, apparently leaving no casualties. Atlácatl battalion troops, which
> maintain a counterinsurgency operation at the foothills of Guazapa hill, to
> the north of San Salvador, confiscated war materiél that the guerrillas had
> hidden underground. In another turn of events, FMLN sappers who continuously
> sabotage the electrical power system destroyed several high-voltage posts in
> La Libertad and Cabanas departments in the central part of the country, and
> in Usulután department in the southeastern part of the country. There were
> widespread power outages in the three departments while technicians of the
> state-run electric power companies repaired the damage. The army and
> security corps deployed heavy patrols in San Salvador and in cities in the
> country’s interior to try and counter the wave of attacks.

Annotated Templates: 3 attack templates
Discussion: For MUC-4, attack is a catch-all type to handle terrorist acts
that are not adequately described by any of the other types, which makes it
particularly challenging to identify. Furthermore, the documentation specifies
that violent acts directed _against terrorists_ are not to be annotated. This
rule evidently excludes the eight guerrila casualties described in this
document, though it’s at least not certain from the text that the guerillas
are terrorists. The three attack templates refer rather to the FMLN attacks on
“high-voltage posts,” with one template per location (department) in which
posts were destroyed (La Libertad, Cabanas, and Usulután). This is sensible,
though one of the annotators in our study reasonably considered these to be
part of the same _coordinated_ attack on the electrical power system and so
annotated a single template.

###  B.2 BETTER Granular Examples

#### Example 1

Document Text:

> President Cyril Ramaphosa has urged South Africans to “act responsibly” as
> the second wave of the Covid-19 pandemic gripped two provinces, with fears
> that it will sweep through the rest of the country during end of year
> celebrations.
>
> His worried Health Minister, Dr Zweli Mkhize, Saturday night ‘raided’
> popular Cape Town bars and nightspots where merrymakers were breaking
> lockdown rules on social distancing, mask wearing and staying safe.
>
> What is bothering the South African government is the rapid rise in new
> cases in both the Eastern Cape Province - where the second wave of the virus
> has a full grip and is overwhelming - with as a similar pattern emerging in
> the Garden Route of the Western Cape, which borders the Eastern Cape, as
> well as in greater Cape Town, capital of the Western Cape.
>
> The fear is that the wave will prove unstoppable amid end year celebrations
> and family gatherings which are bound to drive infections up further,
> threatening to overwhelm South Africa’s already struggling medical system.
>
> Overnight, South Africa recorded another 4,932 new Covid-19 infections and
> at least 160 more deaths. The pandemic has so far claimed over 21,000 lives.
>
> Ramaphosa, speaking to the nation prior to his health minister’s on-the-
> ground inspection of potential ‘superspreader’ events and venues, had
> announced a return to tougher emergency restrictions in the major urban area
> of the Eastern Cape, the Nelson Mandela Bay metro, where conditions are
> severe and from where nearly half of all new infections are coming.
>
> But South Africa health authorities are moving away from the idea of
> sweeping new lockdowns, focusing instead on hotspots and attempting to stop
> the spread by restricting alcohol sales, imposing a tighter curfew between
> 10pm and 4am and reducing the numbers of people attending open-air and
> indoor venues.

Annotated Templates: 1 Epidemiplate
Discussion: This document describes just a single disease (Covid-19), but it’s
far from clear that it describes a single _epidemic_. In contrast to
_pandemics_ , which are global disease outbreaks, epidemics are geographically
localized. Considering the individuation criteria for BETTER presented in
Appendix A, we know we may be justified in annotating multiple epidemiplates
if there are disease outbreaks in distinct locations. The article describes
outbreaks in both the Eastern Cape and Western Cape Provinces of South Africa.
But is this one location (South Africa) or two (the two provinces)? Both seem
reasonable. One could even argue that the emergence of the virus’s second wave
in the Garden Route and in Cape Town — both within the Western Cape Province –
constitute separate epidemics of their own.

#### Example 2

Document Text:

> Uganda’s Foreign Affairs Minister Sam Kutesa and two other officials have
> denied corruption charges. Mr Kutesa, chief whip John Nasasira and junior
> labour minister Mwesigwa Rukutana appeared in court a day after they
> resigned. They are accused of abuse of office as well as financial loss over
> the 2007 Commonwealth summit in Uganda, in which scams allegedly cost some
> $150m (£95m). Former Vice-President Gilbert Bukenya was charged in July. He
> denies that he benefited from a $3.9m deal to supply cars used to transport
> dozens of heads of state during the summit. Mr Kutesa was also accused on
> Monday of taking large bribes from UK-based Tullow Oil. The minister and the
> company strongly denied the allegation. In court, prosecutor Sydney Asubo
> said the three had cost the government 14bn shillings ($4.8m; £3.1m). “The
> three irregularly convened a consultative cabinet meeting and decided that
> the government would fully fund the construction cost of driveways, parking
> areas and marina at Munyonyo Speke Resort,” he said. They face up to 13
> years in jail if convicted, reports the AFP news agency. Some MPs from Mr
> Bukenya’s Buganda ethnic group had accused the government of selective
> justice by failing to prosecute anyone else. Last week, the Inspectorate of
> Government - the body charged with fighting corruption - said Mr Kutesa, Mr
> Nasasira and Mr Rukutana would be charged. President Yoweri Museveni said on
> Wednesday that the three officials had chosen to resign. “That’s their
> decision because what we want is the truth,” he told a news conference. Mr
> Museveni sacked Mr Bukenya in May as part of a cabinet reshuffle.

Annotated Templates: 4 Corruplates
Discussion: The article describes four individuals accused of corruption: Sam
Kutesa, John Nasasira, Mwesigwa Rukutana, and Gilbert Bukenya. For this
reason, it’s understandable that the annotations would feature four corruption
templates — one per person charged. However, the first three of the four seem
to be implicated in the same corruption charges. Those charges could
reasonably be counted as a single incident of corruption involving multiple
actors, yielding two templates overall (or three if including Mr. Kutesa’s
bribery charge as a separate incident). This was a case in which there was
_complete_ disagreement among annotators in our reannotation study, who
annotated two, three, and four Corruplates.

##  Appendix C Reannotation Study: Further Details

For the reannotation study, none of the annotators had read any of the
documents prior to annotation, though they had seen examples of other
documents from each dataset. Annotators worked independently and were not
permitted to discuss any aspect of the task with each other during the
annotation.

In order to focus the task on within-type individuation, and to make it as
straightforward as possible for annotators, documents were required to have
annotated gold templates of only a single type. As discussed in §4, we aimed
to obtain 10 documents for each type, for each dataset. Given the single type
constraint, this was not always possible — as was the case for the robbery and
forced work stoppage templates in MUC-4. For these cases, we selected as many
documents as we could that satisfied the single type constraint, which
resulted in three documents for robbery, and unfortunately none for forced
work stoppage. Although it would be nice to have more documents for these
types, we do not think this detracts from the result that even experts
struggle to agree on within-type individuation decisions.

For BETTER, we selected documents for Disasterplates and Displacementplates
from the Phase II data, as the Phase I data does not contain templates of this
type. However, we selected all documents for Corruplates, Epidemiplates,
Protestplates, and Terrorplates from the Phase I data, as the Phase II data
had comparatively few documents annotated with these types.

Finally, when we say in §4 that we stratify by the number of templates within
type, we mean that we group all documents annotated exclusively with templates
of type $T$ by the number of templates annotated and then sample round robin
from each group, and uniformly at random within group.

In the main text, the (dis)agreement analysis uses the nominal version of
Krippendorff’s $\alpha$, which is equivalent to Fleiss’s $\kappa$ (Fleiss,
1971) in settings like ours in which the same set of annotators annotate all
items. This is meant to show the level agreement when considering only
_whether_ annotators agree on the number of events described, which is our
main concern. By computing the _ratio_ version of the $\alpha$ instead of the
nominal, we may also take into account the _magnitude_ of the difference in
the number of annotated events. In general, one would expect the magnitudes of
these differences to be small: although annotators clearly disagree on the
precise number of events (see Table 1), the text will usually constrain them
not to give substantially different counts. This is borne out in Table 5,
which shows the ratio form of $\alpha$. As expected, the aggregate agreement
is higher here for both datasets, though we stress that it is still far from
what one would want from expert annotators and from the data ($\alpha^{+}$).

The high agreement observed on $\alpha^{+}$ among annotators for select MUC-4
template types suggests that some types of event (e.g. kidnappings) may just
be more readily individuable than others — or perhaps merely that the
documents annotated with these types happen to present especially clear-cut
cases. We emphasize that this does not impugn the broader point, supported by
these results, that individuation is tremendously challenging even for experts
intimately familiar with the data.

BETTER | MUC-4
---|---
Type | $\alpha$ | $\alpha^{+}$ | Type | $\alpha$ | $\alpha^{+}$
Corruption | 0.23 | 0.27 | Arson | 0.48 | 0.35
Disaster | 0.26 | 0.31 | Attack | 0.39 | 0.32
Epidemic | 0.24 | 0.29 | Bombing | 0.95 | 0.68
Migration | 0.09 | 0.07 | Kidnapping | 0.99 | 0.96
Protest | 0.50 | 0.51 | Robbery* | 1.00 | 1.00
Terrorism | 0.35 | 0.40 |  |  |
Aggregate | 0.33 | 0.38 | Aggregate | 0.68 | 0.66
Table 5: The ratio version of Krippendorff’s $\alpha$, measuring IAA on the
number of templates of each type for the reannotated MUC-4 and for BETTER
documents.

##  Appendix D Model Details

The analysis in §5 is based on the GTT model released by Du et al. (2021b),
which uses BERT-base (Devlin et al., 2019) as its encoder. The IterX model is
the one from Chen et al. (2022) that uses a T5-large encoder (Raffel et al.,
2020), which showed the best results across the datasets presented in that
paper. The TempGen model is also from Chen et al. (2022), uses BART-large as
the encoder (Lewis et al., 2020), and is lightly adapted from Huang et al.
(2021) to support multi-template prediction. Although Chen et al. have not
released their code, we obtained the predictions from these models via
personal correspondence.

To drive home just how much these models struggle to learn template count
distributions (see §5), we report the level of agreement between their
template count predictions on the MUC-4 and BETTER test sets and those from
the gold annotations in Table 6, reported for the subset of documents with at
least one gold template of the corresponding type. Results for template types
with minimal test set support (indicated by *) should be interpreted with
caution, but agreement is generally very low and in many cases not much
different from chance. With the caveat that the statistics in Table 1 and
Table 6 are computed on different documents, it is noteworthy — and
disheartening — that for BETTER, the level agreement between the gold
annotations and IterX (0.25; see Table 6) actually exceeds that between the
gold annotations and the human experts (0.22; see Table 1).

BETTER | MUC-4
---|---
Type | IterX | Type | IterX | GTT | TempGen
Corruption* | 0.00 | Arson* | -0.25 | -0.11 |  -0.00
Disaster |  -0.24 | Attack |  -0.08 | -0.01 | -0.14
Epidemic |  -0.64 | Bombing |  -0.03 |  -0.03 |  -0.21
Migration | -0.06 | Kidnapping | -0.20 |  -0.14 |  -0.00
Protest* | 0.00 | Robbery* |  -0.00 |  -0.00 |  -0.00
Terrorism * | 0.00 |  |  |  |
Aggregate | 0.25 | Aggregate | -0.05 |  -0.16 | -0.02
Table 6: Nominal Krippendorff’s $\alpha$ between gold template counts and
counts as predicted by each model in §5 for all test set documents with at
least one annotated template. Agreement is poor across the board, reinforcing
our conclusion that template filling models do _not_ effectively learn
template count distributions. *Indicates that there are fewer than 10
documents in the test split containing annotated templates of this type and
that the corresponding results should be read with caution.

##  Appendix E Template Filling as QA

Here, we sketch how the recasting of template filling as QA proposed in §6
might be carried out. As the aim of this paper is merely to demonstrate the
problems with template filling as it is usually posed, we emphasize that this
is _a sketch_ , and that a complete solution along these lines is left for
future work.

To illustrate our proposal, we show how it could be implemented for the MUC-4
bombing template, using the document from Figure 1 as our example. Here, we
provide the full document as context to an autoregressive language model. We
use the text-davinci-003 version of GPT-3 (Brown et al., 2020) available on
the OpenAI playground151515https://beta.openai.com/playground as a prime
exemplar of the impressive extractive abilities of recent LLMs. We do not
perform any fine-tuning and we use the default hyperparameters for inference.
Questions are posed one at a time in the format “Q: $\langle\texttt{question
text}\rangle$ A:,” with GPT-3 generating the answer, indicated by italicized
text below. Subsequent questions are conditioned on the entire prompt so far,
including the previous QA pairs.

Given the document context, we envision question answering proceeding
_iteratively_ and in a _coarse-to-fine_ manner with respect to question
specificity. We begin with coarse-grained questions derived from the template
type ontology about what kind(s) of event the document describes (e.g.  _does
this article describe a{bombing, kidnapping, robbery, …}?_). Extraction of
templates of an event type _not_ described in the document terminates
immediately upon a negative response (e.g. “No”) to the corresponding
question. But upon a positive response, we then follow up with further
questions about the nature of the event(s) of that type, including ones about
roles as given by the slots in the ontology. Prior QA formulations of IE tasks
have suggested this much, conditioning questions about roles on responses to
questions about the (type of) event described (He et al., 2015; Du and Cardie,
2020; Liu et al., 2020; Holzenberger et al., 2022). But they generally do not
go beyond roles to consider properties of events that drive individuation:
time, location, and irrealis. Moreover, questions about roles in prior work
are usually not relativized to specific entities, which may be desirable, as
different entities may provide cues about distinct events.

The core argument is that if we care about IE models distinguishing different
templates, we should care about _why_. That requires making those reasons
explicit, and we propose that this be done in the form of questions. This is
the point at which any QA recasting effort must look at the template filling
dataset, ontology, and annotation guidelines in detail to determine what
properties of events are salient to individuation. We note two important
points about such a project:

  1. 1.

Different properties may be needed for different event types. For instance, in
order for there to be multiple bombings, there likely must be multiple
explosive devices. However, multiple _kidnappings_ may be perpetrated by a
single kidnapper. These differences are natural and perfectly acceptable:
ontologies are driven by the information needs of IE end users, which are not
one-size-fits-all.

  2. 2.

The goal is _not_ to construct a one-to-one mapping from questions back to
templates. To ask (e.g.) _what organization perpetrated the second bombing?_
is to beg the very question we want to avoid having to answer. The goal is to
have the questions cover all of the information in the gold annotations
(including all slot-filling entities) without having to deliver a verdict on
individuation. Thus, questions should be focused more granularly: e.g. _what
organization planted the second device?_

The result of this effort would be a set of questions for each document,
covering all of the information in the gold annotations, minus the within-type
individuation decisions. Standard entity precision, recall, and F1 can then be
used to evaluate answers to questions targeting entities (e.g. those about
roles), and accuracy may be used for binary questions (e.g. about types of
event). Metrics for specific slots may be computed by aggregating across
questions that target those slots. Crucially, there are no templates to align.

As in prior work, questions may be posed either independently or else
conditional on answers (gold or predicted) to relevant previous questions. We
suspect that work leveraging autoregressive LLMs may benefit from some form of
conditioning, as evidenced in the example below.

### Example

For there to be multiple bombings, there likely must be multiple explosive
devices. Once it has been established that a document describes a bombing,
_how many explosive devices were there?_ is therefore perhaps the most natural
next question, which partly encodes individuation decisions for bombings. From
there, questions about participants, time, location, and factuality — other
properties that form the basis for individuations — may be relativized to each
device. Here, we associate each MUC-4 template slot with a natural language
question, relativized to each explosive device:

  * •

PerpInd: What individuals planted the $\langle n^{th}\rangle$ device?

  * •

PerpOrg: What organization planted the $\langle n^{th}\rangle$ device?

  * •

Target: What buildings or infrastructure were targeted by the $\langle
n^{th}\rangle$ device?

  * •

Victim: Who were the victims of the $\langle n^{th}\rangle$ device?

  * •

Weapon: What was the $\langle n^{th}\rangle$ device?

However, we also augment this set of questions with questions about factuality
(did the $\langle n^{th}\rangle$ device actually explode?), time (when did the
$\langle n^{th}\rangle$ explode?), and location (where was the $\langle
n^{th}\rangle$ device?). We note that the model’s response to the factuality
question (both here and generally) could be used to condition whether
subsequent questions are asked at all. Below, we mark with an asterisk two
questions about the second device that could have been omitted in light of the
model’s response to the factuality question, which correctly indicates that
the second device did not actually explode. It is a testament to the
capabilities of GPT-3 that it answers these questions correctly even though
their presuppositions are false. As a final question — purely out of curiosity
and not as part of the proposal — we query GPT-3 for the number of bombings in
total. Perhaps unsurprisingly, we find it disagrees with the gold annotations.
This reinforces our claim that such questions are ultimately irrelevant to IE.

Article: “A bomb exploded today in a Lima restaurant, and a second device that
had been placed in the same establishment was deactivated by the Peruvian
National Police. There were no victims, and the explosion caused very little
damage to the restaurant, which is located in the commercial area of the
residential district of Miraflores. Guerrillas of the Tupac Amaru
Revolutionary Movement (MRTA) have claimed credit for the terrorist act
through pamphlets they left on the premises, according to the police.”

Q: Does this article describe a bombing?
A: Yes, this article describes a bombing.

Q: How many explosive devices were there?
A: There were two explosive devices.

Q: What was the first device?
A: The first device was a bomb that exploded.

Q: What was the second device?
A: The second device was a bomb that was deactivated by the Peruvian National
Police.

Q: Did the first device actually explode?
A: Yes, the first device actually exploded.

Q: Did the second device actually explode?
A: No, the second device was deactivated by the Peruvian National Police
before it exploded.

Q: Where was the first device?
A: The first device was located in a Lima restaurant.

Q: Where was the second device?
A: The second device was also located in the same Lima restaurant.

Q: What organization planted the first device?
A: The Tupac Amaru Revolutionary Movement (MRTA) planted the first device.

Q: What organization planted the second device?
A: The Tupac Amaru Revolutionary Movement (MRTA) also planted the second
device.

Q: What buildings or infrastructure were targeted by the first device?
A: The first device was targeted at a Lima restaurant.

Q: What buildings or infrastructure were targeted by the second device?
A: The second device was also targeted at the same Lima restaurant.

Q: Who were the victims of the first device?
A: There were no victims of the first device.

Q*: Who were the victims of the second device?
A: There were also no victims of the second device.

Q: When did the first device explode?
A: The first device exploded today.

Q*: When did the second device explode?
A: The second device did not explode; it was deactivated by the Peruvian
National Police.

Q: How many bombings were there in total?
A: In total, there was one bombing.