# Asking the Right Questions in Low Resource Template Extraction

Nils Holzenberger  Equal contributions Yunmo Chen$*$ Benjamin Van Durme
<EMAIL_ADDRESS>
Center for Language and Speech Processing
Johns Hopkins University
Baltimore, Maryland, USA

###### Abstract

Information Extraction (IE) researchers are mapping tasks to Question
Answering (QA) in order to leverage existing large QA resources, and thereby
improve data efficiency. Especially in template extraction (TE), mapping an
ontology to a set of questions can be more time-efficient than collecting
labeled examples. We ask whether end users of TE systems can design these
questions, and whether it is beneficial to involve an NLP practitioner in the
process. We compare questions to other ways of phrasing natural language
prompts for TE. We propose a novel model to perform TE with prompts, and find
it benefits from questions over other styles of prompts, and that they do not
require an NLP background to author.

##  1 Introduction

Figure 1: Top: Example template from the Granular ontology. Bottom: an example
document from the Granular dataset, with spans highlighted. Expected answers
for each slot are specified using superscripts.  (T): template name and
description, (S): slot name and description, (E): expert question, (Q): non-
expert question.

Information Extraction (IE) tasks such as event extraction (EE) and template
extraction (TE) (Du et al., 2021b, inter alia) are commonly presented as slot
filling for template-like structures, with fillers extracted from a textual
passage. Current extraction systems powered by neural networks have achieved
state-of-the-art performance but require significant amounts of annotated data
for training. Given the complexity and specialized domains of many ontologies,
annotation processes are often performed by domain experts, making high-
quality annotations hard to obtain.

One way to mitigate data scarcity is to use natural language _prompts_. In the
general sense, prompts are natural language (NL) strings which, fed to an IE
model, serve as a flexible inductive bias across many tasks (Raffel et al.,
2020). Prompts have been widely adopted in many NLP tasks (Levy et al., 2017;
McCann et al., 2018, inter alia), paving the way for formulating TE tasks as
Question Answering (QA) (Du and Cardie, 2020b; Chen et al., 2020b, inter
alia).

Previous work has focused on different ways to compose prompts (Shin et al.,
2020) and approaches to tuning prompts towards better model performance (Logan
IV et al., 2021). However, to the best of our knowledge, under the TE as QA
(TE-QA) formulation, a thorough study has not been pursued to investigate the
effectiveness of prompts formulated as NL questions, and humans’ ability to
judge the usefulness of a given prompt. In particular, we investigate whether
potential end users of a TE system can be allowed to drive task definitions.
For that, we let humans who have no background in TE write prompts in the form
of NL questions, as an efficient way of annotating for the task. Since end
users are not experts in TE modeling, and are likely more used to writing
queries used in search engines, we are concerned with whether a QA-based TE
system can perform well with their questions — a realistic setting for people
using such IE systems. With the advancement of QA models, we would like to
investigate the robustness of such systems under different types of prompts.
Furthermore, we quantify whether end users are able to intuitively distinguish
between higher- and lower-quality prompts.111 We define higher quality prompts
as prompts yielding higher model performance. The contributions of this work
are as follows:

  1. 1.

We design an analysis framework to empirically evaluate the performance of TE-
QA systems fed with different prompts: we propose a ranking-based TE-QA model
that bypasses the need to extract spans, to focus on examining the
effectiveness of various prompts.

  2. 2.

We collect prompts from potential end users, and thoroughly analyse prompt
quality.

  3. 3.

We conclude that: a) the phrasing of prompts matters; b) prompt writers
without TE modeling background can write prompts leading to high model
performance; c) humans are unable to gauge the quality of a prompt.

##  2 Background

#### TE as QA

The terms QA and MRC have both been used to describe the use of a linguistic
description to query a context in order to extract answers. Under its
formulation, prompts222 In this case, prior work has been using prompt, query,
and question interchangeably. serve as the key component that elicits
information and guides the extraction process. QA has been widely explored in
different NLP tasks (McCann et al., 2018), such as semantic role labeling
(FitzGerald et al., 2018), dialogue state tracking (Gao et al., 2019), and
fact verification (Weber et al., 2019).

Recent work adopts the formulation for various TE tasks, and achieves state-
of-the-art performance (Li et al., 2019; Wu et al., 2020; Li et al., 2020; Du
and Cardie, 2020b, inter alia). Interestingly, those works make use of
different styles of prompts: hand-written NL questions (Li et al., 2019), a
series of NL questions instantiated from a set of templates (Du and Cardie,
2020b), descriptions from annotation guidelines rewritten as bleached
definitions (Chen et al., 2020b), and mentions detected using a Conditional
Random Field along with surrounding context used as questions (Wu et al.,
2020). These forms of prompts work and, to some extent, serve their modeling
purposes well. However, there is a lack of analysis on how much the form and
the phrasing of the prompt matters for model performance. Further, it is not
clear whether end users can judge the quality of a prompt in terms of expected
model performance.

#### Prompt engineering

NL prompts contain information not captured by model parameters, hence serving
as a flexible way to inject inductive bias, and playing the role of a hyper-
parameter (Brown et al., 2020). Unlike other hyper-parameters, given the huge
discrete search space, searching for an optimal NL prompt that maximizes model
performance is an NP-hard problem. Prior work (Shin et al., 2020; Schick and
Schütze, 2021a, b, inter alia) investigates searching for improved NL prompts
with respect to model performance using gradients as signals. These approaches
usually lead to non-fluent NL prompts, e.g. a list of keywords. We are
concerned with understanding how robust a model is to prompts written by
humans, and whether performance benefits from a prompt being authored by an
NLP expert. We rely on human annotators to explore the space of NL prompts.

Another line of work (Qin and Eisner, 2021; Liu et al., 2021, inter alia)
abandons NL questions and optimizes soft prompts in a continuous space. It
allows training the prompts along with model parameters and bypasses the
difficult search problem. Although soft prompts may perform similarly or
better than explicit prompts, the tuning process is no longer interpretable
for humans. Moreover, soft prompts themselves are no longer interpretable for
humans. In low resource scenarios, where inductive biases play an important
role in compensating data scarcity, NL prompts can better capture human-
provided inductive biases.

#### Low resource

Levy et al. (2017); Chen et al. (2020b) (inter alia) conduct experiments in
zero- or few-shot learning. They provide insights on task performance and
generalization ability under such settings, but without attention to data
efficiency. Le Scao and Rush (2021) quantify the effect of providing a prompt
as opposed to more training data. Logan IV et al. (2021) contrast tuning the
prompt and tuning the model itself.

#### Templates and events

We define a _template_ as a semantic structure that consists of _slots_ and
represents an event or a conjunction of events. The definition shares
similarities with that of _events_ used throughout IE, where _slot_ is a
counterpart to _argument_ , allowing us to recast existing IE datasets to the
framework of templates and slots.

In prior work, event semantics has been investigated in different settings.
ACE is a _sentence-level_ event task (Walker et al., 2006) which requires the
detection of event trigger and arguments residing within the same sentence.
RAMS (Ebner et al., 2020) contains events with arguments spanning across
_multiple sentences_. The Granular task proposed by the BETTER program333
https://www.iarpa.gov/research-programs/better further extends the paradigm to
combine _multiple simple sentence-level events, entities, factuality, and
their temporal relationships_ to form storylines (named “templates”). Although
these datasets differ in complexity, they can fit under the same template
framework.

We are interested in how prompt phrasing affects a model’s NL understanding
ability in both high and low resource scenarios. Given the above definition of
_templates_ , we use the task of filling slots in templates (Du and Cardie,
2020a; Chen et al., 2020b; Du et al., 2021a, b, inter alia) to examine the
model, allowing us to evaluate template understanding with ontologies of
different complexities. For the aforementioned event datasets, we show
statistics in Table 1.

Split ACE 2005 RAMS Granular #Template types 33 139 4 #Slot types 22 65 63*
#Templates/#Slots train 4202/4859 7329/17026 252/1449 dev 450/605 924/2188
56/294 test 403/576 871/2023 47/269

Table 1: Dataset statistics. * indicates that we only consider slots that have
spans as fillers.

##  3 Prompt collection

In the ontologies associated with Granular, ACE and RAMS, each template comes
with an NL description. These descriptions are meant to help humans understand
the scope of the ontology, and to help when annotating documents by hand.
Slots may or may not have a linguistic description. For all three datasets, we
asked annotators to write questions that captured the information stated in
the ontology description.

###  3.1 Annotation protocol

First, one expert NLP practitioner wrote a question for each slot, referred to
as “expert”.444 For our purposes, experts are people who know how QA models
work, and have a sense of how to inject detailed information via prompts.
Second, we recruited non-expert annotators to write a question for each slot
in a given template.555 Annotators were paid undergraduate students without
background in the task nor in NLP modeling. These annotators have no
background in TE, and we view them as potential end users of the TE system. We
provided the annotators with the name of the template in the ontology666 e.g.
ArtifactExistence:ArtifactFailure:MechanicalFailure, the description of the
template, and if it exists, the description of the slot. Examples are shown in
Figure 1. Annotators were also shown some templates with the expert questions
already filled in, to help them get started. The templates, as well as the
questions within a template, were presented in a random order, mitigating any
variations in the quality of the annotations due to seeing one slot at an
earlier or later point in the annotation process. For the Granular ontology, 8
annotators were hired. Each produced questions for 3 templates. Templates were
assigned to annotators so that all 4 templates in the ontology were annotated
equally, resulting in 6 questions for each slot. Some questions were identical
across multiple annotators, resulting in less than 6 distinct questions for
some slots, as can be seen in Figure 1. Similarly, 5 annotators worked on the
ACE and RAMS ontologies, resulting in 3 and 1 questions per slot,
respectively.

###  3.2 Prompt aggregation

With $n$ questions per slot, we group questions across slots into $n$ series
(shown as Series in the following tables). Each series covers the entirety of
the dataset’s templates and contains questions that come from the smallest
number of different annotators possible. This is done by assigning an index to
each annotator. For each slot, questions are ordered by increasing index. The
$i$-th question is assigned to the $i$-th series. Variation in performance
across series is thus kept as close as possible to variations in performance
across annotators. Examples are shown in Figure 1, and Tables 5 and 9.

To measure the diversity of the questions collected, we compute pairwise
similarities of the questions filling the same slot using SBERT (Reimers and
Gurevych, 2019). We report detailed statistics in Table 2 and show examples in
Table 3. Pairwise similarity of questions is generally high. Including expert
questions does not change the similarities much, indicating a certain amount
of information being shared in the embedding space. RAMS questions show lowest
absolute similarity. We hypothesize that more fine-grained ontologies such as
used in RAMS make annotators write diverse questions to distinguish subtle
differences. For example, the AIDA ontology used for RAMS distinguishes
between one-way and two-way communication, and whether communication happens
in person or not.777 e.g. Contact:CommandOrder:Broadcast,
Contact:CommandOrder:Correspondence and Contact:CommandOrder:Meet

Manual analysis of questions reveals that annotators tended to write questions
that allow for a narrower interpretation than the expert’s, as can be seen in
Table 5. We attribute this to the expert’s effort to formulate questions close
to the meaning of the ontology, but open enough to accommodate a wide range of
answers, in an attempt to balance precision and recall. Table 4 shows the
statistics of questions written by experts and non-experts. We consider ACE
and Granular simpler than the AIDA ontology, as they have fewer template types
and slot types (shown in Table 1). For these simpler ontologies, the expert
wrote shorter, somewhat less specific questions. For the complex AIDA
ontology, the expert wrote longer questions, to highlight subtle differences
between templates. This was sometimes done at the expense of fluency.

| with expert | without expert
---|---|---
ACE | $89.1\pm 6.5$ | $88.8\pm 6.6$
Granular | $90.5\pm 4.4$ | $90.2\pm 4.4$
RAMS | $87.4\pm 8.8$ |
Table 2: Pairwise cosine similarity (in %) of questions written for the same
slot, formatted as _average_ $\pm$  _standard deviation_. “With expert”
includes the expert question in the pool of questions. ACE, similarity=0.7:
---
For what was the defendent being tried?
 _What criminal charges is the trial for?_
Granular, similarity=0.8:
How much was this person fined?
What monetary punishment is the individual facing?
RAMS, similarity=0.9:
 _Where did the money transfer take place?_
Where did the transaction occur?
Table 3: Example questions and their similarities from different datasets. Expert questions are in _italics_. | non-expert | expert
---|---|---
ACE | $5.3\pm 1.4$ | (6) | $5.1\pm 1.5$ | (6)
Granular | $7.1\pm 2.2$ | (5) | $6.5\pm 2.0$ | (7)
RAMS | $5.7\pm 1.8$ | (4) | $7.0\pm 2.4$ | (7)
Table 4: Question length in words, formatted as _average_ $\pm$  _standard
deviation_ (_median_). Life:Injure.] An Injure Event occurs whenever a Person
entity experiences physical harm. Injure Events can be accidental, intentional
or self-inflicted.
---
Instrument-Arg. The device used to inflict the harm.
_What weapon or vehicle was used to injure?_
What device was used to cause the injury?
What weapon was used to harm the person?
What device was used to inflict harm?
Movement:TransportArtifact:SmuggleExtract. The smuggling or extracting of an
artifact by an agent out of one place into another. <Transporter> transported
<Artifact> in <Vehicle> from <Origin> place to <Destination> place.
Transporter.
_Who transported the artifact that was smuggled or extracted?_
Who smuggled or extracted an artifact?
Table 5: Example questions written for the ACE (top) and RAMS (bottom)
ontologies. The expert question is in _italics_.

##  4 Model

To answer the aforementioned questions, we would like our QA model performance
to depend, as much as possible, only on prompts used, excluding the effect of
other factors such as finding candidate spans for slot filling. We propose a
learning-to-rank model that consumes a given set of candidate spans and a
prompt, and picks the most probable slot fillers.

###  4.1 Problem formulation

Let $C$ be a text passage, or context. We refer to a specific template and
slot type as $y$. The set of candidate spans $A$ in $C$ are those to be
considered when predicting $U_{y}$, the subset of $A$ that fills slot $y$. We
also assume access to a span in $C$ that triggers the template.888A trigger
span in IE is the key textual evidence that supports prediction of an event or
template. The candidate spans $A$ are given as part of the TE datasets.

We use a _prompt_ to query the model to predict a set of spans to fill the
slot $y$. We denote the prompt as ${Q_{y}=(q_{1},\cdots,q_{m})}$, and the text
passage as ${C=(c_{1},\cdots,c_{n})}$. The model predicts a set of spans
$\overline{U}_{y}$ where $\overline{U}_{y}\subset A$. Our approach is
illustrated in Figure 1.

###  4.2 Context and span encoding

We adopt an encoder-based machine reading architecture to jointly encode a
prompt $Q_{y}$ and passage $C$ (Devlin et al., 2019). The prompt $Q_{y}$ and
the text $C$ are concatenated with special delimiters999 We use BERT
delimiters in this description, but the actual delimiters might differ
depending on the encoder design. and passed through the underlying encoder:

$\textsc{Encoder}(\left[\textsc{cls},q_{1},\cdots,q_{m},\textsc{sep},c_{1},\cdots,c_{n},\textsc{sep}\right])$

where cls is a special token whose embedding represents the whole sequence,
and sep serves as the sequence separator. We denote the output encoding of
each question token ${q_{i}~{}(1\leq i\leq m)}$ as
${\mathbf{q}_{i}\in\mathbb{R}^{d}}$ and the encoding of each text token
${c_{i}~{}(1\leq i\leq n)}$ as ${\mathbf{c}_{i}\in\mathbb{R}^{d}}$. We
concatenate a positional embedding ${\mathbf{p}_{i}\in\mathbb{R}^{p}~{}(1\leq
i\leq n)}$ to each $\mathbf{c}_{i}$, marking the distance in tokens of $c_{i}$
to the tokens representing the template trigger. The distances are non-
negative integers, so that the ${\mathbf{p}_{i}}$ are learned embeddings. We
consider the set of candidate spans ${A=\left\\{a_{1},\cdots,a_{k}\right\\}}$,
where ${a_{k}=(a_{k}^{\textrm{start}},a_{k}^{\textrm{end}})}$ and ${1\leq
a_{k}^{\textrm{start}}\leq a_{k}^{\textrm{end}}\leq n}$.

###  4.3 Span ranking and dynamic threshold

As slots can have any number of filler spans, only picking the most probable
span (Devlin et al., 2019) cannot suffice. Besides, some slots might not have
any fillers. Prior work such as Devlin et al. (2019) uses the cls token to
determine whether the question is answerable. We adopt the use of cls, and
extend it with the idea of dynamic threshold from Chen et al. (2020a), to be
able to support any number of answer spans. Similar to Lee et al. (2017), we
compute a score $s(a_{k})$ for each candidate span $a_{k}$, indicating how
likely it belongs to $U_{y}$:

$s(a_{k})=\textsc{ffnn}([c_{a_{k}^{\textrm{start}}},c_{a_{k}^{\textrm{end}}}])$

where $[c_{a_{k}^{\textrm{start}}},c_{a_{k}^{\textrm{end}}}]$ is the
concatenation of the two vectors representing the start and end tokens of
$a_{k}$, and ffnn is a feed-forward neural network. We score the cls token the
same way as other candidate spans to obtain $s(\textsc{cls})$. It then serves
as a sentinel to dynamically separate relevant spans from irrelevant spans.
The underlying ranking relation to be learned is:

$u\succ\textsc{CLS}\succ v,\quad\forall u\in U_{y},\forall v\in
V_{y}=A\setminus U_{y}$

where $\succ$ means “ranks higher than”. For model training, we employ a hinge
ranking loss:

$\displaystyle J_{u\succ\textsc{cls}}$ $\displaystyle=$
$\displaystyle\displaystyle\frac{1}{|U_{y}|}\sum_{u\in U_{y}}$
$\displaystyle[$ $\displaystyle\alpha$ $\displaystyle-$ $\displaystyle(s(u)$
$\displaystyle-$ $\displaystyle s(\textsc{cls}))$ $\displaystyle]_{+}$
$\displaystyle J_{\textsc{cls}\succ v}$ $\displaystyle=$
$\displaystyle\displaystyle\frac{1}{|V_{y}|}\sum_{v\in V_{y}}$
$\displaystyle[$ $\displaystyle\beta$ $\displaystyle+$ $\displaystyle(s(v)$
$\displaystyle-$ $\displaystyle s(\textsc{cls}))$ $\displaystyle]_{+}$
$J=\lambda J_{u\succ\textsc{cls}}+(1-\lambda)J_{\textsc{cls}\succ v}$

where the hyper-parameter $\lambda$ controls the trade-off between both
losses, and hyper-parameters $\alpha$ and $\beta$ are the margins of the loss
functions. During inference, we first compute scores for all candidate spans
and cls. The predicted answer set is then decoded as
${\overline{U}_{y}=\left\\{a\mid s(a)\geq s(\textsc{cls}),\forall a\in
A\right\\}}$.

##  5 Experiments

###  5.1 Experimental settings

We run experiments on the Granular, ACE and RAMS datasets to measure the
impact of using different prompts on TE-QA. We use Unified-QA (Khashabi et
al., 2020) as the Encoder described in Section 4.2. Since our experiment
setting is inherently the same as by Ebner et al. (2020) and Chen et al.
(2020c), we follow prior work and use span-level F1 as the evaluation metric.
We take Chen et al. (2020c) as our baseline, shown as JointArg in the tables,
since it obtains previous state-of-the-art results on both ACE and RAMS using
gold mention spans. As far as we are aware, we are the first to publish
results on the Granular dataset with this experimental setup.

For each set of prompts, we use Optuna (Akiba et al., 2019) to tune hyper-
parameters, running 20 experiments per set. Each experiment consists of 3
training runs with 3 different random seeds.101010 Except for RAMS, where we
use single runs. During the hyper-parameter sweep, we pick the best performing
model on the validation set based on micro-averaged F1 scores. With the best
set of hyper-parameters, we retrain with 5 distinct random seeds and report
the test scores by taking the average of 5 runs. The process intends to
minimize the effect of hyper-parameters other than the choice of prompts,
hence illustrating the best possible performance for each set of prompts.
Therefore, we are able to make a fair comparison across different types of
prompts. Further details can be found in Table 10 in the Appendix.

###  5.2 Results and discussion

We report results in Tables 6, 7 and 8. We compare expert questions (Expert),
questions collected in Section 3 (Series), and the following different types
of prompts:

  * •

SpecialTokens, a single token per slot, providing a randomly initialized,
trainable embedding, seen as a naive soft prompt baseline in the spirit of Qin
and Eisner (2021),

  * •

the Name of the template and slot in the ontology, separated by a space,

  * •

a fluent Description of the template and slot, when available in the ontology.

Prompt | Micro-F1 | Macro-F1
---|---|---
SpecialTokens | $35.3\pm 3.0$ | $35.5\pm 2.8$
Name | $55.8\pm 0.4$ | $\mathbf{57.3}\pm 1.0$
Description | $60.6\pm 0.7$ | $53.0\pm 1.6$
Expert | $\mathbf{61.5}\pm 0.3$ | $55.6\pm 1.1$
Series-1 | $57.9\pm 0.7$ | $54.8\pm 1.3$
Series-2 | $\mathbf{61.2}\pm 0.5$ | $56.6\pm 0.9$
Series-3 | $57.5\pm 0.5$ | $49.9\pm 1.1$
Series-4 | $61.0\pm 0.4$ | $\mathbf{59.5}\pm 1.2$
Series-5 | $\mathbf{62.1}\pm 0.4$ | $\mathbf{57.9}\pm 0.9$
\- Pretraining | $\mathbf{63.8}\pm 0.3$ | $\mathbf{58.1}\pm 1.2$
Series-6 | $\mathbf{61.6}\pm 0.5$ | $56.3\pm 0.9$
Best | $59.6\pm 0.7$ | $\mathbf{58.0}\pm 1.0$
Worst | $\mathbf{61.2}\pm 1.0$ | $55.8\pm 1.4$
Table 6: Test scores on Granular dataset with standard error. Best scores and scores that would be the best by changing 1 standard error are bolded. The question set with the highest dev micro F1 is underlined. Prompt | Micro-F1 | Macro-F1
---|---|---
JointArg* | $65.6$ | -
SpecialTokens | $65.1\pm 1.5$ | $72.6\pm 0.6$
Name | $\mathbf{82.0}\pm 0.3$ | $\mathbf{86.3}\pm 0.2$
Description | $80.8\pm 0.2$ | $85.9\pm 0.2$
Expert | $\mathbf{82.1}\pm 0.2$ | $\mathbf{86.2}\pm 0.2$
Series-1 | $\mathbf{82.0}\pm 0.4$ | $\mathbf{86.4}\pm 0.5$
Series-2 | $\mathbf{82.3}\pm 0.3$ | $\mathbf{86.7}\pm 0.4$
\- Pretraining | $51.6\pm 2.9$ | $68.8\pm 2.2$
Series-3 | $\mathbf{82.1}\pm 0.4$ | $85.6\pm 0.2$
Table 7: Test scores on ACE dataset with standard error. See Table 6 for more details. * is not fully comparable given the difference of encoder used. Prompt | Micro-F1 | Macro-F1
---|---|---
JointArg | $79.9$ | -
SpecialTokens | $57.1\pm 1.7$ | $55.8\pm 2.5$
Name | $\mathbf{82.0}\pm 0.1$ | $\mathbf{81.2}\pm 0.1$
Expert | $81.2\pm 0.1$ | $81.0\pm 0.1$
Series-1 | $\mathbf{82.0}\pm 0.0$ | $\mathbf{81.3}\pm 0.1$
\- Pretraining | $78.6\pm 0.5$ | $77.7\pm 0.3$
Table 8: Test scores on RAMS dataset with standard error. See Table 6 for
further details. Figure 2: Granular, ACE and RAMS Micro F1 as a function of
slot frequency in the training data. Each data point is an average over 3
training runs with different random seeds. Bands and error bars indicate the
full range of values. The bottom right bar plot corresponds to ACE, with at
most 1 positive and 1 negative example per slot.

Overall, we find that the model is able to leverage the information contained
in NL prompts, as evidenced by the lower performance of SpecialTokens. The
best performance is consistently achieved by prompts formulated as questions,
significantly better than the fluent Description.

While the expert question achieves good performance on each dataset, results
suggest that knowledge of TE is not necessary to write questions that yield
good performance.

Surprisingly, Name performs consistently well, being comparable to the best-
performing question on 2 out of 3 datasets. Name consists of the template name
and the slot name, separated by one space. Since slot names may be shared
across templates, and since models can use the slot name as a proxy for the
kind of answer expected, this could help the model spot the correct answer,
simply based on surface patterns. Consistent with our findings, Dai and Callan
(2019) show, in the context of Information Retrieval, that whether a keyword-
based or a fluent query performs best can depend on both the dataset the
retrieval is performed on, and the retrieval method being used.

We further experiment with few-shot learning by subsampling positive and
negative examples in the training data so that they appear less than a certain
number of times (see Figure 2). The observations above extend to the few-shot
setting: the expert question outperforms other prompts for most data regimes.
In addition, we find that as above, Name consistently achieves good
performance, although with a slightly shallower learning curve.

Finally, the aforementioned Unified-QA encoder is built on T5-Base. In Tables
6 to 8, “- Pretraining” reports the results of training the model on the line
above, but using T5-Base as the encoder, thus depriving it of pretraining on
QA datasets. This ablation experiment allows us to examine the effect of QA
pretraining, and shows that a large part of the performance indeed comes from
QA-pretraining. In fact, the ACE, RAMS and Granular datasets lie, in that
order, along a spectrum stretching from sentence-level TE to document-level
TE. The type of QA pretraining we have used here involves mostly sentence-
level, factoid questions (Khashabi et al., 2020). As expected, it helps more
for sentence-level TE, and less for document-level TE. While it is always
possible to map TE to a QA problem, a QA-pretrained encoder is not necessarily
the best-performing encoder.

The datasets used here rely on general-domain ontologies, so that the
annotators possess common sense knowledge about their templates and slots. The
conclusions of this study might be different if instead, the datasets relied
on specialized ontologies, which the annotators would not be familiar with,
such as ontologies in the legal or medical domain. Such experiments are out of
scope here, and we leave it to future work.

As can be seen in Table 6, picking the right set of questions is crucial for
final performance. This is more challenging in the low-resource scenario, with
little validation data.

##  6 Analysis of human judgements

Can humans predict performance on TE-QA? Can humans reliably pick the best-
performing question for a given slot, effectively replacing the role of the
dev set? For each slot, we asked human annotators to rank questions according
to how well they describe the slot, without access to data or experimental
results.

###  6.1 Annotation protocol

For each slot of each template in the Granular ontology, we presented human
annotators with all questions written for that slot.111111Annotators come from
the same demographics as the annotators of Section 3 and were compensated in
the same way for their work. We asked them to rate each question on a scale of
1 to 4:

  1. 1.

This is the best out of all the questions

  2. 2.

This is not the best, but among the best

  3. 3.

This is not the worst, but among the worst

  4. 4.

This is the worst out of all the questions

Annotators were allowed to give the same rating to multiple questions,
especially since some questions can be quite similar. The annotators were
shown the template description as well as the slot description. There is no
overlap between the annotators writing questions and those scoring questions.
We obtain 3 ratings per question for the Granular dataset. We performed this
study on the Granular dataset only, since performance achieved by different
sets of questions was noticeably different. Examples from the annotation
process are shown in Table 9.

Human Ratings (3) — Aggregate Score — Question
---
Epidemiplate. The Epidemiplate template captures key information elements in
regards to various kinds of disease outbreaks.
Disease. Mentions of the disease that is at the heart of the disease outbreak.
1 |  1 |  1 |  2.0 |  What disease caused the outbreak?
2 |  1 |  1 |  1.5 |  What is the disease causing the outbreak?
3 |  2 |  3 |  1.0 |  _What disease broke out?_
4 |  4 |  4 |  0.8 |  What is the disease?
Protestplate. This Granular event is intended to capture the various aspects
of a protest event.
Arrested. Description or count of those arrested.
1 |  1 |  2 |  2.2 |  Who/How many were arrested at the protests?
2 |  3 |  1 |  1.4 |  Who was arrested because of the protest?
3 |  2 |  3 |  0.7 |  Who and how many were arrested?
3 |  2 |  3 |  0.7 |  Who and/or how many people have been arrested?
2 |  3 |  2 |  0.7 |  Who was arrested during or after the protest?
2 |  3 |  2 |  0.7 |  Who was arrested in the protest?
4 |  4 |  4 |  0.4 |  _Who was arrested?_
Table 9: Example question ratings from the Granular dataset. The expert
question is in _italics_.

For a given slot, the _ratings_ of each annotator implicitly established a
_ranking_ of the questions, which we use to compute an aggregated score for
each question. Using the _rankings_ instead of the _ratings_ normalizes the
annotators’ ratings. We assign each question a score of
$\sum_{i=1}^{3}\frac{1}{r_{i}}$ where $r_{i}$ is the rank attributed by the
$i$-th annotator. For questions with identical ranking, we picked the lowest
possible rank — e.g. 3 questions ranked first received each a score of
$\frac{1}{3}$. Using these scores, we can pick a Best and Worst question for
each slot, as the questions that get the highest and lowest score,
respectively. Examples of scores are shown in Table 9.

The scores provided by the 3 annotators have an average Pearson correlation of
31.4%. The scores on questions implicitly provided a scoring of the annotators
who wrote the questions. We assigned each question-writer the average score of
their questions. The scores of question-writers ranged from 1.1 to 0.8, with
the expert rated a 0.9, and ranked 5th out of 9.

Looking through the prompt set labeled Best, we find that annotators seem to
have favored questions that are grammatically correct and fluent, but with
unconventional typesetting. For instance, 6 questions contain a “/”, as in
_What did this person/network do that was corrupt?_ and _Who and/or how many
people have been injured in the protest?_. Similarly, 2 questions contain a
“(s)” appended to the end of a word, as in _What building(s) and space(s) are
the protestors occupying?_. In contrast, the expert questions contain neither
of those, and the best performing set contains 4 “/” and no “(s)”. The
encoder’s tokenizer, built using a form of aggregation of characters and sub-
word units, keeps certain frequent words whole, but breaks up less frequent
ones. Taking an example from above: _What did this person do that was
corrupt?_ is tokenized as [What, did, this, person, do, that, was, corrupt, ?]
(and similarly if replacing “person” with “network”). In contrast, _What did
this person/network do that was corrupt?_ is tokenized as [What, did, this,
person, /, net, work, do, that, was, corrupt, ?]. With conventional
typesetting, the tokenizer keeps occurrences of the word “network” as a single
token. The encoder has thus rarely, if at all, seen the token net followed by
the token work, and likely cannot compose their meaning back into that of the
token network. More generally, every time the tokenizer is forced to back off
to a sub-word representation, word-level information potentially useful for
retrieval may be lost.

###  6.2 Experiments

As shown in Table 6, performance achieved by Best and Worst questions is
within that of other question prompts, neither standing out as particularly
good or particularly bad. To quantify more precisely whether human scores
could predict F1 score performance, we ran our models trained on questions on
the test set, swapping in prompts they were not trained on. We then computed,
for each given model, the Pearson correlation between F1 performance (averaged
over 5 runs), and the score attributed by humans to the question. We found
that the correlation was less than .1 in absolute value, so that overall,
human preferences do not correlate with performance.

##  7 Conclusion

We show that NL questions written by potential end users of a TE system
consistently achieve the best performance on TE-QA on all three datasets. This
finding extends to few-shot learning. In practice, we find that a technical
background in TE is not strictly necessary to formulate good questions,
meaning that end users of the TE system can be trusted to drive the task
definition by writing questions. However, humans are unable to estimate the
usefulness of a particular question phrasing with respect to TE-QA model
performance. Our results encourage the data-driven combination of different
questions to achieve better performance on TE-QA. In addition, as TE is
shifting to a low resource scenario in which ontologies are built but not
annotated for, our results suggest that designers of such ontologies ought to
include, for each template and slot, questions that best describe the
information meant to match the template and slot.

## Acknowledgements

We thank Tongfei Chen, Zhengping Jiang, Dawn Lawrie, Doug Oard, Guanghui Qin,
Kate Sanders, Nathaniel Weir, and Patrick Xia for helpful comments and
feedback.

## References

  * Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019\.  Optuna: A next-generation hyperparameter optimization framework.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019_, pages 2623–2631. ACM.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _CoRR_ , abs/2005.14165.
  * Chen et al. (2020a) Tongfei Chen, Yunmo Chen, and Benjamin Van Durme. 2020a.  Hierarchical entity typing via multi-level learning to rank.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 8465–8475. Association for Computational Linguistics.
  * Chen et al. (2020b) Yunmo Chen, Tongfei Chen, Seth Ebner, Aaron Steven White, and Benjamin Van Durme. 2020b.  Reading the manual: Event extraction as definition comprehension.  In _Proceedings of the Fourth Workshop on Structured Prediction for NLP@EMNLP 2020, Online, November 20, 2020_ , pages 74–83. Association for Computational Linguistics.
  * Chen et al. (2020c) Yunmo Chen, Tongfei Chen, and Benjamin Van Durme. 2020c.  Joint modeling of arguments for event understanding.  pages 96–101.
  * Dai and Callan (2019) Zhuyun Dai and Jamie Callan. 2019.  Deeper text understanding for IR with contextual neural language modeling.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019_ , pages 985–988. ACM.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Du and Cardie (2020a) Xinya Du and Claire Cardie. 2020a.  Document-level event role filler extraction using multi-granularity contextualized encoding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 8010–8020. Association for Computational Linguistics.
  * Du and Cardie (2020b) Xinya Du and Claire Cardie. 2020b.  Event extraction by answering (almost) natural questions.  _CoRR_ , abs/2004.13625.
  * Du et al. (2021a) Xinya Du, Alexander M. Rush, and Claire Cardie. 2021a.  GRIT: generative role-filler transformers for document-level event entity extraction.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021_ , pages 634–644. Association for Computational Linguistics.
  * Du et al. (2021b) Xinya Du, Alexander M. Rush, and Claire Cardie. 2021b.  Template filling with generative transformers.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 909–914. Association for Computational Linguistics.
  * Ebner et al. (2020) Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020\.  Multi-sentence argument linking.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 8057–8077. Association for Computational Linguistics.
  * FitzGerald et al. (2018) Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018.  Large-scale QA-SRL parsing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , pages 2051–2060. Association for Computational Linguistics.
  * Gao et al. (2019) Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, and Dilek Hakkani-Tur. 2019.  Dialog state tracking: A neural reading comprehension approach.  In _Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 264–273, Stockholm, Sweden. Association for Computational Linguistics.
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.  Unifiedqa: Crossing format boundaries with a single QA system.  In _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ , volume EMNLP 2020 of _Findings of ACL_ , pages 1896–1907. Association for Computational Linguistics.
  * Le Scao and Rush (2021) Teven Le Scao and Alexander Rush. 2021.  How many data points is a prompt worth? In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2627–2636, Online. Association for Computational Linguistics.
  * Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017.  End-to-end neural coreference resolution.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_ , pages 188–197. Association for Computational Linguistics.
  * Levy et al. (2017) Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017.  Zero-shot relation extraction via reading comprehension.  In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Vancouver, Canada, August 3-4, 2017_ , pages 333–342. Association for Computational Linguistics.
  * Li et al. (2020) Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020\.  A unified MRC framework for named entity recognition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 5849–5859. Association for Computational Linguistics.
  * Li et al. (2019) Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019.  Entity-relation extraction as multi-turn question answering.  _CoRR_ , abs/1905.05529.
  * Liu et al. (2021) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021.  GPT understands, too.  _CoRR_ , abs/2103.10385.
  * Logan IV et al. (2021) Robert L. Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021.  Cutting down on prompts and parameters: Simple few-shot learning with language models.  _CoRR_ , abs/2106.13353.
  * McCann et al. (2018) Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018.  The natural language decathlon: Multitask learning as question answering.  _CoRR_ , abs/1806.08730.
  * Qin and Eisner (2021) Guanghui Qin and Jason Eisner. 2021.  Learning how to ask: Querying lms with mixtures of soft prompts.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 5203–5212. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Schick and Schütze (2021a) Timo Schick and Hinrich Schütze. 2021a.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021_ , pages 255–269. Association for Computational Linguistics.
  * Schick and Schütze (2021b) Timo Schick and Hinrich Schütze. 2021b.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 2339–2352. Association for Computational Linguistics.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 4222–4235. Association for Computational Linguistics.
  * Walker et al. (2006) Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006.  ACE 2005 multilingual training corpus (LDC2006T06).  _Philadelphia: Linguistic Data Consortium_.
  * Weber et al. (2019) Leon Weber, Pasquale Minervini, Jannes Münchmeyer, Ulf Leser, and Tim Rocktäschel. 2019.  NLProlog: Reasoning with weak unification for question answering in natural language.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6151–6161, Florence, Italy. Association for Computational Linguistics.
  * Wu et al. (2020) Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020.  Corefqa: Coreference resolution as query-based span prediction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 6953–6963. Association for Computational Linguistics.

##  Appendix A Appendices

part of model | hyper-parameter name |  description | value range
---|---|---|---
encoder | model |  Transformer model encoding the paragraph and prompt | allenai/unifiedqa-t5-base
| freeze |  Whether to freeze the encoder’s parameters | False
MLP input features | distance feature size |  size of the embeddings encoding the distance of each token to the template anchor | {8, 16, 32, 64,128, 256, 512}
| prior type |  the type of prior to use to compute the scores for each slot | {none, embed, logit}
MLP classifier | dropout |  feature dropout in between the layers of the MLP | [0, 0.8]
| number of units |  - | {32, 64, 128, 256, 512, 1024, 2048}
| number of layers |  - | {1, 2, 3, 4}
trainer | batch size |  - | {8, 16, 32, 64, 128, 256}
| learning rate |  - | [1e-6, 1e-3]
| learning rate scheduler |  This specific scheduler halves the learning rate when the validtion score stops improving | reduce_on_plateau
| optimizer |  - | huggingface_adamw
| number of epochs |  Maximum number of epochs to train for | 25
| patience |  - | 5
loss function | loss trade-off |  see $\lambda$ in Section 4 | [0, 1]
| negative margin |  see $\beta$ in Section 4 | [1e-2, 0.5]
| positive margin |  see $\alpha$ in Section 4 | [1e-2, 0.5]
Table 10: Hyper-parameters used for the hyper-parameter search. We use Optuna (Akiba et al., 2019) to find good hyper-parameters for each set of prompts, running 20 experiments per set. If there is a single value under “value range”, it means this parameters was set and not searched over with Optuna. Each experiment consists of 3 training runs with 3 different random seeds, using the average micro F1 score on the validation set; except for RAMS, where we use single runs. We compute the test score of the best set of hyper-parameters by running 5 training runs with distinct random seeds and taking the average score. | micro | macro
---|---|---
prompt set | P | R | F1 | P | R | F1
SpecialTokens | $27.8\pm 2.9$ | $49.9\pm 3.2$ | $35.3\pm 3.0$ | $38.6\pm 2.4$ | $41.5\pm 3.1$ | $35.5\pm 2.8$
Name | $\mathbf{68.3}\pm 1.3$ | $47.2\pm 0.5$ | $55.8\pm 0.4$ | $\mathbf{69.1}\pm 1.4$ | $53.4\pm 1.1$ | $\mathbf{57.3}\pm 1.0$
Description | $57.8\pm 2.3$ | $64.5\pm 2.1$ | $60.6\pm 0.7$ | $58.4\pm 2.5$ | $55.0\pm 1.0$ | $53.0\pm 1.6$
Expert | $57.0\pm 0.4$ | $\mathbf{66.8}\pm 0.6$ | $\mathbf{61.5}\pm 0.3$ | $59.9\pm 1.2$ | $\mathbf{58.5}\pm 1.2$ | $55.6\pm 1.1$
Series-1 | $54.9\pm 2.5$ | $62.5\pm 2.8$ | $57.9\pm 0.7$ | $59.9\pm 1.6$ | $58.0\pm 1.4$ | $54.8\pm 1.3$
Series-2 | $62.0\pm 0.4$ | $60.6\pm 1.1$ | $\mathbf{61.2}\pm 0.5$ | $64.7\pm 1.0$ | $56.5\pm 1.2$ | $56.6\pm 0.9$
Series-3 | $50.5\pm 0.8$ | $\mathbf{67.0}\pm 1.6$ | $57.5\pm 0.5$ | $54.0\pm 1.6$ | $54.1\pm 0.9$ | $49.9\pm 1.1$
Series-4 | $61.3\pm 0.6$ | $60.7\pm 0.5$ | $61.0\pm 0.4$ | $\mathbf{67.2}\pm 1.9$ | $\mathbf{59.4}\pm 1.0$ | $\mathbf{59.5}\pm 1.2$
Series-5 | $57.9\pm 1.1$ | $\mathbf{67.2}\pm 1.8$ | $\mathbf{62.1}\pm 0.4$ | $62.5\pm 1.6$ | $\mathbf{60.4}\pm 0.7$ | $\mathbf{57.9}\pm 0.9$
\- Pretraining | $59.1\pm 0.6$ | $\mathbf{69.3}\pm 0.4$ | $\mathbf{63.8}\pm 0.3$ | $62.6\pm 1.5$ | $\mathbf{60.5}\pm 1.0$ | $\mathbf{58.1}\pm 1.2$
Series-6 | $60.4\pm 1.3$ | $63.1\pm 1.1$ | $\mathbf{61.6}\pm 0.5$ | $62.2\pm 0.9$ | $56.5\pm 1.3$ | $56.3\pm 0.9$
Best | $61.7\pm 1.1$ | $58.1\pm 2.3$ | $59.6\pm 0.7$ | $\mathbf{66.6}\pm 1.3$ | $57.2\pm 1.7$ | $\mathbf{58.0}\pm 1.0$
Worst | $56.0\pm 2.1$ | $\mathbf{68.0}\pm 1.2$ | $\mathbf{61.2}\pm 1.0$ | $60.4\pm 1.8$ | $\mathbf{59.8}\pm 0.8$ | $55.8\pm 1.4$
Table 11: Micro and macro averaged scores on Granular dataset with standard error. Best scores and scores that would be the best by changing 1 standard error are bolded. The question set with the highest dev micro F1 is underlined. | micro | macro
---|---|---
prompt set | P | R | F1 | P | R | F1
SpecialTokens | $62.1\pm 1.7$ | $68.7\pm 1.6$ | $65.1\pm 1.5$ | $72.0\pm 0.6$ | $75.4\pm 0.6$ | $72.6\pm 0.6$
Name | $75.7\pm 0.4$ | $89.5\pm 0.5$ | $\mathbf{82.0}\pm 0.3$ | $83.8\pm 0.2$ | $\mathbf{91.0}\pm 0.2$ | $\mathbf{86.3}\pm 0.2$
Description | $73.7\pm 0.5$ | $89.3\pm 0.5$ | $80.8\pm 0.2$ | $83.3\pm 0.3$ | $\mathbf{91.1}\pm 0.4$ | $85.9\pm 0.2$
Expert | $\mathbf{80.3}\pm 0.5$ | $84.0\pm 0.7$ | $\mathbf{82.1}\pm 0.2$ | $\mathbf{86.1}\pm 0.3$ | $88.1\pm 0.4$ | $\mathbf{86.2}\pm 0.2$
Series-1 | $78.5\pm 1.0$ | $85.9\pm 0.9$ | $\mathbf{82.0}\pm 0.4$ | $\mathbf{85.7}\pm 0.6$ | $89.2\pm 0.7$ | $\mathbf{86.4}\pm 0.5$
Series-2 | $\mathbf{79.9}\pm 0.5$ | $84.9\pm 0.9$ | $\mathbf{82.3}\pm 0.3$ | $\mathbf{86.5}\pm 0.5$ | $88.9\pm 0.6$ | $\mathbf{86.7}\pm 0.4$
\- Pretraining | $77.7\pm 1.7$ | $38.8\pm 2.9$ | $51.6\pm 2.9$ | $77.9\pm 1.9$ | $67.0\pm 2.1$ | $68.8\pm 2.2$
Series-3 | $75.2\pm 0.8$ | $\mathbf{90.5}\pm 0.3$ | $\mathbf{82.1}\pm 0.4$ | $83.3\pm 0.2$ | $90.1\pm 0.3$ | $85.6\pm 0.2$
Table 12: Micro and macro averaged scores on ACE dataset with standard error. See Table 11 for further details. | micro | macro
---|---|---
prompt set | P | R | F1 | P | R | F1
SpecialTokens | $45.8\pm 2.4$ | $76.6\pm 1.1$ | $57.1\pm 1.7$ | $50.2\pm 3.1$ | $72.7\pm 0.9$ | $55.8\pm 2.5$
Name | $\mathbf{81.5}\pm 0.6$ | $82.5\pm 0.7$ | $\mathbf{82.0}\pm 0.1$ | $\mathbf{82.8}\pm 0.6$ | $82.4\pm 0.3$ | $\mathbf{81.2}\pm 0.1$
Expert | $73.5\pm 0.2$ | $\mathbf{90.8}\pm 0.2$ | $81.2\pm 0.1$ | $77.8\pm 0.2$ | $\mathbf{88.2}\pm 0.1$ | $81.0\pm 0.1$
Series-1 | $75.1\pm 0.2$ | $90.3\pm 0.2$ | $\mathbf{82.0}\pm 0.0$ | $78.9\pm 0.2$ | $87.5\pm 0.2$ | $\mathbf{81.3}\pm 0.1$
\- Pretraining | $69.2\pm 0.8$ | $\mathbf{91.0}\pm 0.6$ | $78.6\pm 0.5$ | $73.2\pm 0.6$ | $87.6\pm 0.5$ | $77.7\pm 0.3$
Table 13: Average micro and macro scores for the RAMS dataset, with standard
error of the mean. Statistics are computed over 5 runs of the best set of
hyperparameters, each with a different random seed. Best scores are bolded.
Scores that would be best scores if removing or adding one standard error of
the mean are bolded as well. The prompt set that would have been picked based
on the dev micro F1 score is in italics. Figure 3: F-score as a function of
the number spans in the gold answer on the Granular dataset. Each point is an
average over 5 runs. Bands indicate one standard error of the mean around the
average. Figure 4: F-score as a function of the number spans in the gold
answer on the ACE dataset. Each point is an average over 5 runs. Bands
indicate one standard error of the mean around the average. Figure 5: F-score
as a function of the number spans in the gold answer on the RAMS dataset. Each
point is an average over 5 runs. Bands indicate one standard error of the mean
around the average.