#  NLI: Separating Annotation Error from Human Label Variation

Leon Weber-Genzel U missing Siyao Peng U missing11footnotemark: 1 Marie-
Catherine de Marneffe zmissing Barbara Plank U missing
Umissing MaiNLP, Center for Information and Language Processing, LMU Munich,
Germany
missing Munich Center for Machine Learning (MCML), Munich, Germany
zmissing FNRS, UCLouvain, Belgium
<EMAIL_ADDRESS>marie-
<EMAIL_ADDRESS>Equal contribution.

###### Abstract

Human label variation arises when annotators assign different labels to the
same item for valid reasons, while annotation errors occur when labels are
assigned for invalid reasons. These two issues are prevalent in NLP
benchmarks, yet existing research has studied them in isolation. To the best
of our knowledge, there exists no prior work that focuses on teasing apart
error from signal, especially in cases where signal is beyond black-and-white.
To fill this gap, we introduce a systematic methodology and a new dataset,
(variation versus error), focusing on the NLI task in English. We propose a
2-round annotation scheme with annotators explaining each label and
subsequently judging the validity of label-explanation pairs. contains 7,574
validity judgments on 1,933 explanations for 500 re-annotated NLI items. We
assess the effectiveness of various automatic error detection (AED) methods
and GPTs in uncovering errors versus human label variation. We find that
state-of-the-art AED methods significantly underperform compared to GPTs and
humans. While GPT-4 is the best system, it still falls short of human
performance. Our methodology is applicable beyond NLI, offering fertile ground
for future research on error versus plausible variation, which in turn can
yield better and more trustworthy NLP systems.

NLI: Separating Annotation Error from Human Label Variation

Leon Weber-Genzel U missing††thanks:  Equal contribution. Siyao Peng U
missing11footnotemark: 1 Marie-Catherine de Marneffe zmissing Barbara Plank U
missing Umissing MaiNLP, Center for Information and Language Processing, LMU
Munich, Germany missing Munich Center for Machine Learning (MCML), Munich,
Germany zmissing FNRS, UCLouvain, Belgium
<EMAIL_ADDRESS>marie-
<EMAIL_ADDRESS>ACL submission

##  1 Introduction

Labeled data plays a crucial role in modern machine learning (ML) (e.g.,
Mazumder et al., 2023). Data quality impacts ML performance and in turn user
trust. It is therefore of vital importance to aim at high-quality
consistently-labeled benchmark data (e.g., Bowman and Dahl, 2021). However,
recent research has revealed a notable presence of annotation errors in
widely-used NLP benchmarks Klie et al. (2023); Rücker and Akbik (2023).
Similar observations were made recently in computer vision (CV) Northcutt et
al. (2021); Vasudevan et al. (2022); Schmarje et al. (2023).

At the same time, increasing evidence exists that for many items in many
tasks, more than a single label is valid. For some items, systematic variation
exists for valid reasons, such as plausible disagreement or multiple
interpretations. In other words, the world is not just black and white. Human
label variation (HLV, as termed by Plank 2022) has been shown on a wide range
of NLP tasks de Marneffe et al. (2012); Plank et al. (2014); Aroyo and Welty
(2015), including in natural language inference (NLI; Pavlick and Kwiatkowski
2019; Zhang and de Marneffe 2021) as well as in Computer Vision (CV; Peterson
et al. 2019; Uma et al. 2021). NLI involves determining whether a hypothesis
is true (Entailment), false (Contradiction), or neither (Neutral), assuming
the truth of a given premise (cf. Figure 1 for an example with plausible
labels).

Figure 1: Variation or Error? We present a procedure and multi-label dataset,
, to tease apart annotation error from plausible human label variation. We
leverage ecologically valid explanations and validation as two key mechanisms
(boxed: self-validations; label “Contradicts” is an _error_); see §3-§4 for
details.

Although high-quality consistently-labeled data may initially appear to
conflict with the goal of accommodating HLV, it is important to emphasize that
we do not perceive these as contradictory goals. While HLV exists, so do
errors. We assert that annotators are inevitably prone to make errors, such as
misunderstanding instructions or accidentally selecting a wrong label.
Optimizing data quality is essential through providing clear instructions and
effective training, and identifying annotation errors yields better datasets
Larson et al. (2019). However, still little is known about what constitutes an
error versus plausible variation. We lack both a theory and operationalizable
procedures to tease apart error from plausible HLV consistently and soundly.
Some datasets with errors (and their corrections) exist, and there has been
work on automatic error detection (AED). However, both have their limitations
(§2). A crucial gap remains: a lack of examination in real-world scenarios
where the signal is nuanced, not merely black-and-white.

To address this gap, this paper contributes: (i) , a novel multi-annotator
English NLI dataset with both plausible variation and detected errors. To the
best of our knowledge, no such dataset exists yet. (ii) A new methodology to
detect errors: we collect multiple annotations, where each label comes with an
ecologically valid explanation inspired by Jiang et al. (2023), and propose to
pair them with validity judgments to identify errors. (iii) Finally, we
benchmark existing AED methods and GPTs in a challenging setup, where the task
is to tease apart error from plausible human label variation. Our findings
indicate that existing AED methods underperform humans and GPTs substantially
and highlight the need for further research. To facilitate uptake, we release
our new dataset and code on GitHub upon publication.

##  2 Related Work

Labeled data is the fuel of machine learning, as it drives both learning and
evaluation. Following a data-centric view, we focus on improving data quality
over data quantity Motamedi et al. (2021); Swayamdipta et al. (2020); Zhang et
al. (2021); Gordon et al. (2022). We aim to bring together work on data
quality from two ends: annotation error vs. human label variation.

#### Annotation Errors and AED

Several recent work has found errors in widely used benchmarks, such as CoNLL
2003 for Named Entity Recognition Wang et al. (2019); Reiss et al. (2020);
Rücker and Akbik (2023), TACRED for relation extraction Alt et al. (2020), WSJ
for syntax Manning (2011); Dickinson and Meurers (2003), and ImageNet for
object classification Beyer et al. (2020); Northcutt et al. (2021); Vasudevan
et al. (2022).

AED has a long-standing tradition in NLP. Proposed methods range from early
work that relies on variation-based methods positing that instances with
similar surface forms tend to have the same label (Dickinson and Meurers,
2003; Plank et al., 2014) to more recent model-based approaches that either
exploit predictions Amiri et al. (2018); Arazo et al. (2019) or information
derived from training dynamics Swayamdipta et al. (2020); see Klie et al.
(2023) for a survey on AED.

Flaggers and scorers for AED have been proposed Klie et al. (2023). Flaggers
detect errors by providing a hard decision of whether an instance is
erroneous. Scorers, on the other hand, assign a score to each instance
reflecting the likelihood of being an error, and the top-$n$ scoring instances
are then corrected. Here, we focus on scoring methods to rank instances. Most
of the AED work mentioned has limitations as they either rely on post-hoc
mining of errors (and might therefore miss out on errors) in semi-automatic
ways (e.g. Reiss et al., 2020), or they inject synthetic noise which has been
shown to result in datasets where errors are easy to spot Larson et al.
(2019). Instead of using synthetic noise, we focus on realistic setups and re-
annotate data in ecologically valid ways.

#### Human Label Variation (HLV)

Recent studies have drawn attention to HLV in NLP (i.a., Uma et al., 2021;
Plank, 2022). HLV has been described as annotator disagreement, which is not
just noise but also signal since a sign of vagueness or ambiguity can benefit
models (Aroyo and Welty, 2013). These include judgments that are not always
categorical de Marneffe et al. (2012), inherent disagreement Pavlick and
Kwiatkowski (2019); Davani et al. (2022), or justified and informative
disagreement Sommerauer et al. (2020). For subjective NLP tasks, which by
essence encourage annotator subjectivity (and hence variation), there is also
a line of work referred to as perspectivism Basile et al. (2021), connected to
the descriptive data annotation framework proposed by Rottger et al. (2022).

#### HLV in NLI

This paper focuses on NLI, known to contain HLV (Pavlick and Kwiatkowski,
2019; Nie et al., 2020; Jiang and de Marneffe, 2022; Jiang et al., 2023).
Pavlick and Kwiatkowski (2019) re-annotated nearly 500 NLI instances with 50
crowd-workers and showed that disagreements in NLI cannot be dismissed as
annotation “noise.” ChaosNLI (Nie et al., 2020) pioneers large-scale NLI
annotation by collecting 100 annotations per instance for 3K items from SNLI
Bowman et al. (2015), $\alpha$NLI (Bhagavatula et al., 2020), and MNLI
(Williams et al., 2018) but for which the original annotations did not yield
high agreement. They show that, for most of the items, HLV persists with more
annotations. Further, their experiments show a large room for model
improvement and a positive correlation between human agreement and label
accuracy.

In another line of work, Jiang and de Marneffe (2022) identified reasons for
observing variation in NLI, deriving a taxonomy based on linguistic properties
of the items. Following up on that work, Jiang et al. (2023) proposed LiveNLI,
to gain insights into the origin of label variation. They re-annotated 122 NLI
instances from ChaosNLI with ecologically valid explanations: annotators are
instructed to not only provide NLI labels but also explanations for their
label choices. This addresses a limitation of prior work that uses post-hoc
explanations, which may not reflect the true reasons of the original
annotators, thereby questioning the validity of the prior method. They show
that ecologically valid explanations have an additional benefit: signaling
within-label variation, i.e., annotators give the same label but for different
reasons. While we do not focus on the latter here, we take inspiration from
Jiang et al. (2023) to collect ecologically valid explanations (cf. §3.1).

To the best of our knowledge, there remains a gap for studies on both
annotation errors and human label variation in a concentrated effort. It is
thus an open challenge to define error in an ecologically valid way, and it is
unknown to what extent existing AED methods help detect such errors and
whether new methods are needed. To find answers to these challenging open
questions, we believe it is important to move both directions forward.

|  Premise: Because marginal costs are very low, a newspaper price for
preprints might be as low as
---
5 or 6 cents per piece.
Hypothesis: Newspaper preprints can cost as much as $5.
Label-explanation pairs: Before:{E:1,N:2,C:1} Self-validated:{N:2} Peer-
validated:{N:2,C:1}
Label: [N] Errors:[E, C]
Round 1: NLI Label & Explanation | Round 2: Validity
L | A | Explanation | 1 | 2 | 3 | 4
E | 4 | 5 dollars for a piece of newspaper. | ✕ | ✕ | ✕ | ✕
N | 1 |  |  The context only mentions how low the price may be, not how high it may be.
---
✓ | ✓ | ✓ | ✓
3 |  |  The maximum cost of newspaper preprints is not given in the context.
---
✓ | ✓ | ✓ | ✓
C | 2 | The context says 5 or 6 cents, not $5. | ✕ | ✕ | ✓ | ✓

(a)  id: 72870c

|  Premise: They made little effort, despite the Jesuit presence in Asia, to
convert local inhabitants
---
to Christianity or to expand their territory into the interior.
Hypothesis: The Jesuit presence in Asia helped to convert local residents to
Christianity,
allowing them to expand their territory.
Label-explanation pairs: Before:{E:1,C:4} Self-validated:{C:3} Peer-
validated:{C:4}
Label: [C] Error:[E]
Round 1: NLI Label & Explanation | Round 2: Validity
L | A | Explanation | 1 | 2 | 3 | 4
E | 1 | Both premise and hypothesis suggest that the speaker does not understand. | ✕ | ✕ | ✕ | ✕
C | 1 |  | The Jesuit didn’t make much effort to convert local residents to Christianity
---
or to expand their territory.
✓ | ✓ | ✓ | ✓
2 | They did not try to expand their territory. | ✓ | ? | ✓ | ✓
3 |  | The Jesuit did not make effort to convert local residents to Christianity
---
or to expand their territory.
✓ | ✓ | ✓ | ✓
4 |  | They made little effort to convert the locals or to expand their territory.
---
So they did not help.
✓ | ✓ | ✓ | ✓

(b)  id: 28306c

Table 1: Sample annotations from NLI corpus. L: Label, A: Annotator; E:
Entailment, N: Neutral, C: Contradiction; ✓: ‘yes’; ✕: ‘no’; ?: ‘idk’;
magenta: self-judgments, black: peer-judgments, Err: label error.

##  3 : Annotation Procedure

To tease apart human label variation from error, we create (Variation versus
Error), an NLI dataset with two rounds of annotations by four
annotators:111Annotators are master students in CompLing and the first author
of this paper, paid according to the national standard. Round 1 for NLI labels
and explanations (§3.1) and Round 2 for validity judgments (§3.2).

###  3.1 Round 1: NLI Labels & Explanations

We collect annotations from four annotators on 500 NLI items randomly sampled
from the MNLI subset of ChaosNLI. Annotators were asked to provide not only
one or more NLI labels (E: Entailment, N: Neutral, C: Contradiction) to each
item but also a one-sentence _explanation_ for each label they chose, as the
same label could be chosen for different reasons Jiang et al. (2023).
Annotators could use a fourth “I don’t know” (IDK) label if none of the NLI
labels seemed suitable. Round 1 annotation sums up to 1,933 label-explanation
pairs for the 500 items after discarding 331 “IDK” annotations and keeping
only standard NLI labels.

###  3.2 Round 2: Validity Judgments

’s key contribution lies in proposing a second round of _validity judgment_.
Validity judgment mirrors conventional annotation adjudication in that
annotators view NLI labels and explanations from each other. However, this
information is delivered anonymously to annotators to reduce group dynamics.
Further, rather than agreeing on a single label or explanation altogether,
annotators are free to make independent judgments regarding what is an error
versus a plausible variation.

In this second round, annotators become judges. For all 500 items, the 1,933
label-explanation pairs from Round 1 are distributed anonymously to the same
four annotators in the same order as in Round 1. For each NLI item, each judge
sees all label-explanation pairs annotated in Round 1 in a random order,
including their own, which they may or may not remember. For each label-
explanation pair, the annotator judges whether the explanation makes sense
given the label, answering “yes” (✓), “no” (✕) or “IDK”. Round 2 includes
7,574 validity judgments after discarding 158 “IDKs”.

##  4 : Detecting Errors

Multiple validity judgments on label-explanations enable distinguishing
annotation errors from HLV.

###  4.1 Self versus Peer

One consequential feature of our two-round multi-annotator scheme is the
distinction between self- vs. peer-judgments. Self-judgments refer to Round 2
judgments on the judge’s own Round 1 label-explanation annotations whereas
peer-judgments refer to judgments from other annotators.

###  4.2 Validating Labels

Let $\mathcal{A}=\\{a_{1},..,a_{4}\\}$ be the set of annotators.

#### Self-validated Label-Explanation

A label-explanation pair given by annotator $a$ on an item in Round 1 is
_self-validated_ iff $a$ judges in Round 2 that the annotated pair makes
sense.

#### Peer-validated Label-Explanation

A label-explanation pair given by annotator $a$ in Round 1 is _peer-validated_
iff the majority (2) of the other annotators $\mathcal{A}\backslash\\{a\\}$
approves the pair in Round 2.

#### What counts as an error?

We define an NLI label as an error if all label-explanation pairs are not
self-validated. In other words, a label is viewed as correctly attributed to
an item iff any of its explanations is self-validated.

Importantly, in our Round 2 annotation, a label-explanation pair might be
considered wrong in retrospect by the annotator who wrote it after reading all
label-explanation pairs given to that item by four annotators. For instance,
this is the case for E and C in Table 1(a) as well C in Table 1(b). To
comprehensively assess annotation variations in Round 1, we work with a strict
definition of error, for which all annotators need to re-evaluate their
previous annotations in light of others’ annotations. Annotator identity is
not revealed in the process.

###  4.3 Data Statistics & IAA

Table 2 shows the frequencies of NLI labels across the four annotators on the
500 items and 1,933 explanations before and after validation. We include
statistics on _repeated_ frequency counts (e.g., E counts twice if it is given
as a label by two annotators for the same item) and _aggregated_ labels
(repeated labels for a given item count once). Moreover, following Jiang et
al. (2023), we compute inter-annotator agreement on NLI labels using
Krippendorff’s $\alpha$ (for multi-annotator) with MASI-distance (for multi-
label).

Validation | FreqType | E | N | C | $\tsum\slimits@$ | IAA
---|---|---|---|---|---|---
before validation | repeated | 554 | 977 | 402 | 1,933 | 0.35
aggregated | 263 | 403 | 212 | 878
self-validated | repeated | 467 | 916 | 329 | 1,712 | 0.50
aggregated | 210 | 380 | 159 | 749
peer-validated | repeated | 446 | 859 | 296 | 1,601 | 0.69
aggregated | 177 | 335 | 130 | 642

Table 2: Frequency counts and inter-annotator agreement (Krippendorff’s
$\alpha$ with MASI-distance) on non-, self-, and peer-validated NLI labels.

(a) Number of label-explanations rejected by self-and-peer, self-only, and
peer-only validations.

(b) NLI label sets on non-, self- and peer-validated items.

Figure 2: Frequency statistics on .

Results show that self-validated annotations achieve a much higher IAA than
without validation (see 0.A.1 for pairwise IAA). Though the repeated and
aggregated frequencies of NLI labels decrease adequately after validation, HLV
is still preserved in self- and peer-validated annotations, averaging 1.50
(749/500) and 1.28 (642/500) labels/item.

We also observe in Table 2 that 88.57% (1,712/1,933) of Round 1 explanations
in were self-validated and 82.82% (1,601) were peer-validated. Figure 2(a)
presents the number of label-explanation pairs rejected by both self- and
peer-validations, by self-validation only, and by peer-validation only. Most
Entailment and Contradiction annotations rejected by self are also rejected by
peers (dark green). However, Neutral presents a challenging situation for
self-validation where 60.13% (92/153) of Ns is only invalidated by the joint
force of peers but not by one annotator alone.

Figure 2(b) demonstrates frequencies of aggregated label combinations per item
before validation and after self- and peer-validations (see 0.A.2 for label-
explanation pair frequencies). Frequencies of multi-labeled items drop after
self-validation and, more remarkably, after peer validation. Inversely, the
number of single-labeled items increases vastly, especially for N.
Additionally, we also observe from that a large portion of items, 37.6%
(188/500), are self-identified as errors and 51.6% (258) are rejected by peer-
validation.

In sum, though HLV remains in , our validation process demonstrates that
annotation errors are frequently concealed under HLV. We thus proceed with the
challenging automatic error detection task in §5-6 to separate annotation
errors from valid HLVs.

##  5 Automatic Error Detection (AED) on

We now describe our experiments to detect annotation errors using
automatically. We evaluate the capabilities of AED methods, LLMs, and human
heuristics (all henceforth _scorers_) in capturing ecologically detected
annotation errors.

###  5.1 Task Definition and Evaluation

Following Klie et al. (2023); Weber and Plank (2023), we model AED as a
ranking task. In this setting, the goal of the scorer is to provide a ranked
list with the labels that are most likely errors at the top and the most
likely correct ones at the bottom. This ranked list could then be used to
guide re-annotation efforts (Alt et al., 2020; Northcutt et al., 2021) or
remove the most likely errors from the training data (Huang et al., 2019).
Scorers produce such a list by assigning an error score to each assigned label
in the dataset. They derive the ranked list by sorting it according to the
assigned scores.

We evaluate scorers on using the following protocol. A model receives the list
of NLI items from , where each item is paired with the label(s) it received in
Round 1. For the 500 items in , the model is given a list of 878 item-label
pairs. Based on that information, the model assigns an error score and ranks
the labels by this score. We evaluate how well the model performs by comparing
this ranked list with the self-flagged errors. Following Klie et al. (2023),
we use standard ranking metrics for evaluation: average precision (AP), i.e.,
the area under the precision/recall curve computed over all assigned labels,
and precision/recall for the top 100 ranked labels, P@100 and R@100.

###  5.2 Baselines and Models

We evaluate five different AED models: two variants of Datamaps (DM,
Swayamdipta et al. 2020), Metadata Archaeology (MA, Siddiqui et al. 2023), and
two GPTs. We report the mean and standard deviation over three random seeds
for DM and MA.

#### Datamaps (DM)

We use training dynamics (i.e., the collection of training statistics over
epochs $E$) for each label. These statistics are obtained by training a
DistilRoBERTa-base model222https://huggingface.co/distilroberta-base (Sanh et
al., 2019) following Klie et al. (2023) in a multi-label setting (Jiang and de
Marneffe, 2022) on all labels of obtained in Round 1. We refer to the $j$’th
label of the $i$’th example as $\text{label}_{i,j}$. The training dynamics are
modeled by the probability $p_{i,j,e}$ that DistilRoBERTa predicts for
$\text{label}_{i,j}$ after the $e$’th epoch. Based on these probabilities, the
two DM models we use are defined as follows:

$\displaystyle\textit{DM}_{\text{mean}}$
$\displaystyle=-\frac{1}{E}\tsum\slimits@_{e=1}^{E}p_{i,j,e}$ (1)
$\displaystyle\textit{DM}_{\text{std}}$
$\displaystyle=\sqrt{\frac{1}{E}(\tsum\slimits@_{e=1}^{E}p_{i,j,e}+\textit{DM}_{\text{mean}})^{2}}$
(2)

Note that a low average probability for the label indicates a likely error.
Because our evaluation setup requires the most likely errors to be ranked
first, we negate the average probabilities.

#### Metadata Archaeology (MA)

MA models AED as a supervised task. It represents each instance (or label in
our case) as the $E$-dimensional $-\log{p_{i,j,e}}$ vector, where $E$ is the
number of epochs and $p_{i,j,e}$ is the probability the model assigns to the
$j$’th label of the $i$’th NLI instance at epoch $e$. Then, it assumes that
some instances are labeled with the property of interest (in our case, whether
it is an erroneous label). It predicts whether an instance is an error by
employing a k-nearest neighbors (kNN) classifier using the instance
representations and error labels. We use the average number of annotated
errors for the kNN to obtain a score for each instance. Following Siddiqui et
al. (2023), we use $k=20$. To obtain unbiased predictions, we require that the
kNN training instances are distinct from those we want to obtain predictions
for. We use a 2-fold cross-validation setup where we split into two folds, use
one half as ground truth, obtain the predictions for the other, and vice
versa.

#### GPT

We also compare two large language models (LLMs): GPT-3.5 (Brown et al., 2020)
and GPT-4 (OpenAI, 2023). We emulate the Round 2 annotation setting in §3.2 as
closely as possible by prompting each model to provide a score reflecting how
much each Round 1 explanation makes sense for a given label. We compute the
score per label by averaging the GPT-assigned scores of all explanations for
the label. We prompt GPT as follows, giving it the premise (context) and
hypothesis (statement) of an NLI item as well as all label-explanation pairs,
asking it then to provide a probability for each reason:

⬇

System:

You are an expert linguistic annotator.

User:

We have collected annotations for an NLI instance together with reasons for
the labels. Your task is to judge whether the reasons make sense for the
label. Provide the probability (0.0 - 1.0) that the reason makes sense for the
label. Give ONLY the reason and the probability, no other words or
explanation. For example:

Reason: <The verbatim copy of the reason>

Probability: <the probability between 0.0 and 1.0 that the reason makes sense
for the label, without any extra commentary whatsoever; just the
probability!>.

Context: {CONTEXT}

Statement: {STATEMENT}

Reason for label {LABEL}: {REASON_1}

Reason for label {LABEL}: {REASON_2}

[...]

Reason for label {LABEL}: {REASON_n}

Reason {REASON_1}

Probability:

We implement GPTs using sglang (Zheng et al., 2023) and its default sampling
parameters. See Appendix 0.B for a complete prompt example. Note that the GPTs
have access to the explanations for the labels, whereas the other models
described above only have access to the labels without explanations.

###  5.3 Human Heuristics

In addition to the above automatic means, we experiment with four human
heuristics that use the human label distributions over NLI labels (E, N, C)
from annotation efforts: label counts from ChaosNLI (100 annotators) and (4
annotators). In addition, we compare to ’s total and average peer-judgments
over explanations.

#### Label Counts (LC): ChaosNLI &

We hypothesize that if multiple annotators choose the same label, there is a
high likelihood that the label is a correct annotation. We implement two label
count (LC) baselines: one using ChaosNLI (Nie et al., 2020) and one using .
ChaosNLI includes 100 crowd-sourced annotations for each NLI item. Since is a
subset of ChaosNLI items, we use label counts from ChaosNLI (LCChaos) as a
human heuristic to score Round 1 labels on each item, i.e., how many of the
100 crowd-workers annotated $\text{label}_{i,j}$ on item $i$. For instance,
the ChaosNLI human distribution is {N:25, E:72, C:3} for the example in Figure
1. Similarly, we include LC that counts the number of annotators (4 in total)
that assigns $\text{label}_{i,j}$ to item $i$ in ’s Round 1 NLI labels. We
multiply both LCChaos and LC by $-1$, proposing that if a label has a higher
count, then it is less likely to be an error.

#### Peer-judgments (Peer) in

’s 2-round annotations enable more fine-grained human heuristics that engage
judgments on label-explanation pairs. Since each $\text{label}_{i,j}$ can be
assigned by multiple annotators with different explanations, we count the
number of “yes” judgments on explanations from peers – excluding self-
judgments since those are used for gold error labels.

We implement two peer heuristics: Peersum and Peeravg. Peersum sums all “yes”
judgments across multiple explanations on the same label, and Peeravg sums
“yes” judgments within each explanation and then averages across explanations
within the label. Given that one label can maximally receive four
explanations, it can receive up to 12 peer-judgments, 3 per explanation. For
example, C in Table 1(b) receives 11 peer-judged “yes” in total
(Peersum$=3+2+3+3=11$), and the average over four explanations is
Peeravg$=11/4=2.75$. Peeravg differentiates more from Peersum when there are
multiple explanations, but each receives sparse “yes” judgments. For example,
N in Table 5(a) (Appendix 0.C) receives 3 “yes” judgments but across two
explanations, resulting in Peersum$=2+1=3$ and Peeravg$=3/2=1.5$. Similarly
the label counts above, we multiply both Peersum and Peeravg by $-1$,
hypothesizing that fewer “yes” judgments indicate a higher likelihood to be an
annotation error.

#### Combining Label Counts and Models

Ranking labels by the number of annotations they received in Round 1 is a very
strong baseline; see $\text{LC}_{\text{\name}}$ in Table 3. Inspired by
Nogueira et al. (2019), we investigate an approach that re-ranks the
predictions of $\text{LC}_{\text{\name}}$ by breaking ties with the scores
produced by another model (e.g. MA or GPT-4). Note that
$\text{LC}_{\text{\name}}$ produces many ties because its score is always one
of $\\{-1,-2,-3,-4\\}$.

##  6 Results for AED on

Table 3 presents human and model performances on AED using the ranking setup
in §5.

Method | AP | P@100 | R@100 | AP (rerank)
---|---|---|---|---
Baselines
Random | $14.7$ | $14.7$ | $11.4$ | -
Models
MA | $17.7\pm 1.5$ | $18.3\pm 4.2$ | $14.2\pm 3.2$ | $44.2\pm 3.0$
$\text{DM}_{\text{mean}}$ | $22.8\pm 0.4$ | $23.7\pm 2.1$ | $18.3\pm 1.6$ | $\textbf{50.4}\pm 0.7$
$\text{DM}_{\text{std}}$ | $22.3\pm 1.9$ | $22.7\pm 1.2$ | $17.6\pm 0.9$ | $50.0\pm 1.5$
GPT-3.5 | $17.6$ | $21.0$ | $16.3$ | $37.6$
GPT-4 | 31.3 | 46.0 | 35.9 | $47.4$
Human
LCChaos | $32.5$ | $35.0$ | $27.3$ | 49.8
LC | $40.8$ | $42.0$ | $32.6$ | $40.8$
Peeravg | $42.2$ | $46.0$ | $35.9$ | $47.8$
Peersum | 46.5 | 47.0 | 36.7 | $47.8$

Table 3: Results for AED on . AP: average precision; rerank denotes using the
method to break ties in LC. For MA and DM, we report mean and standard
deviation over three random seeds. Note that GPTs have access to explanations.

###  6.1 Human Performance

The best human heuristic is from peers (Peersum), reaching a performance of
$46.5\%$ AP, $47\%$ precision@100, and $36.7\%$ recall@100. These numbers
support our hypothesis that human validation can be used as a strong means to
detect annotation errors in a task with relatively high HLV because self- and
peer-rejected label-explanation pairs overlap considerably (cf. Figure 2(a)).
Interestingly, both peer-derived heuristics from perform better than
$\text{LC}_{\textsc{Chaos}}$ (3 linguists versus 100 crowd-workers), which
suggests that having few highly trained expert annotators is sufficient for
reliable error detection, outperforming a larger number of crowd-workers.
$\text{LC}_{\text{\name}}$ outperforming $\text{LC}_{\textsc{Chaos}}$ on all
metrics strengthens this finding. Next we compare humans to automatic means.

###  6.2 Model Performance

Among the models, GPT-4 outperforms all other methods by a large margin with a
$8.5/22.3/17.6$ percentage points (pp.) improvement in terms of AP / P@100 /
R@100 over the second best model $\text{DM}_{\text{mean}}$. GPT-4 even
outperforms LCChaos in P@100 and R@100 and is close to the best peer heuristic
for these two metrics. One might postulate that ChaosNLI could have been part
of GPT-4’s training mixture, and GPT-4 performed well by reproducing its
probabilities. To check whether this is the case, we compute Pearson’s $r$
between the predictions of all scorers (Figure 3). While GPT-4 has a slightly
higher correlation ($0.42$) with $\text{LC}_{\textsc{Chaos}}$ than with all
other methods, it is still much lower than some correlations between other
models, e.g., $0.61$ between $\text{DM}_{\text{mean}}$ and MA. Thus, we
conclude that GPT-4 does not solely rely on information from ChaosNLI but
achieves its strong performance via some other mechanism. Another possible
explanation is that it is the only model next to GPT-3.5 that has access to
explanations. In future, we would like to investigate the further use of
explanations.

Figure 3 allows for a more general interesting observation. There seems to be
a clear cluster structure in which the training-dynamics-based models (DM and
MA) correlate highly with each other and GPT-4 clusters with the human
scorers. Notably, the correlation between the clusters is small to non-
existent and even negative in some cases.

Figure 3: Correlations among scorer predictions.

###  6.3 Influence of Human Label Variation

Figure 4: Average distribution of erroneous, HLV, and other labels over the
top 100 instances per method.

In which situations do AED methods make mistakes, e.g., detect false positive
errors? This is an open question. We hypothesize that many top-ranking labels
would either be errors or come from instances displaying HLV, i.e., instances
with multiple labels after self-validation. The rest should be instances with
just one plausible label. To test this hypothesis, we compute the proportion
of erroneous labels vs. valid labels from HLV instances vs. other (neither
errors nor HLV labels, i.e., with one plausible label) for the top 100 ranking
labels for each method. The results in Figure 4 confirm our hypothesis (for
the GPTs and human predictors): they place very few (0-11) labels that are
neither errors nor HLV in the top 100. On the other hand, the training-
dynamics-based methods MA and DM assign between 17.6 and 29.8 of these items
to the top 100. This suggests that increasing the separation between errors
and HLV is only one part of improving training dynamics methods for AED.
Another could be finding the characteristics of the top-ranking items that are
neither errors nor HLV.

###  6.4 Reranking models using label counts

Column AP (rerank) in Table 3 presents our re-ranking results. We observe that
re-ranking improves over vanilla $\text{LC}_{\text{\name}}$ for all methods
but GPT-3.5. Interestingly, the best performing methods – also compared to the
non-re-ranking approaches – are $\text{DM}_{\text{mean}}$ and
$\text{DM}_{\text{std}}$. They even perform better than
$\text{Peer}_{\text{sum}}$, the best human approach. This suggests that
combining statistics from multiple annotators with AED methods based on
training dynamics is a promising future direction.

##  7 Conclusion

Errors exist in any dataset, but so does plausible human label variation. This
paper defines a general procedure to separate error from plausible label
variation. Our key idea to define errors is to leverage ecologically valid
explanations (where annotators provide their reasons for a label) and pair
these with annotator self-validation (to allow self-correction). We provide a
new dataset for the task. Our empirical investigation on finds that
traditional annotation error detection methods fare poorly on this task and
underperform humans. While applied to NLI, our methodology is general, and we
hope it inspires uptake.

## Limitations

We believe that our two-round annotation setup would work for eliciting
ecologically valid error annotations in tasks or languages other than English
NLI. However, we cannot be sure without trying it, which we did not do during
this project. Further, we did not use all types of information that contains
for the training-dynamics-based AED methods. An interesting question would be
whether exploiting the soft label distribution with methods from learning from
disagreement (Uma et al., 2021) would improve AED results. Another potentially
useful source of information is the explanations given by the annotators.
Using this information for computing the training dynamics or directly
modeling whether an explanation makes sense for a label in a supervised
setting could potentially improve AED performance.

## Acknowledgements

We want to thank Huangyan Shan, Shijia Zhou, and Zihang Sun for their
contributions and invaluable feedback on . Thanks to Verena Blaschke for
giving feedback on earlier drafts of this paper. This work is supported by ERC
Consolidator Grant DIALECT 101043235. Marie-Catherine de Marneffe is a
Research Associate of the Fonds de la Recherche Scientifique – FNRS.

## References

  * Alt et al. (2020) Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020.  TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1558–1569, Online. Association for Computational Linguistics.
  * Amiri et al. (2018) Hadi Amiri, Timothy Miller, and Guergana Savova. 2018.  Spotting Spurious Data with Neural Networks.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2006–2016, New Orleans, Louisiana. Association for Computational Linguistics.
  * Arazo et al. (2019) Eric Arazo, Diego Ortego, Paul Albert, Noel O’Connor, and Kevin Mcguinness. 2019.  Unsupervised Label Noise Modeling and Loss Correction.  In _Proceedings of the 36th International Conference on Machine Learning_ , pages 312–321. PMLR.
  * Aroyo and Welty (2013) Lora Aroyo and Chris Welty. 2013.  Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard.  _ACM Web Science 2013_.
  * Aroyo and Welty (2015) Lora Aroyo and Chris Welty. 2015.  Truth is a lie: Crowd truth and the seven myths of human annotation.  _AI Magazine_ , 36(1):15–24.
  * Basile et al. (2021) Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. 2021.  Toward a perspectivist turn in ground truthing for predictive computing.  _arXiv preprint arXiv:2109.04270_.
  * Beyer et al. (2020) Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. 2020.  Are we done with ImageNet?  _arXiv preprint arXiv:2006.07159_.
  * Bhagavatula et al. (2020) Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen tau Yih, and Yejin Choi. 2020.  Abductive commonsense reasoning.  In _International Conference on Learning Representations_.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Bowman and Dahl (2021) Samuel R. Bowman and George Dahl. 2021.  What will it take to fix benchmarking in natural language understanding? In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4843–4855, Online. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Davani et al. (2022) Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022.  Dealing with disagreements: Looking beyond the majority vote in subjective annotations.  _Transactions of the Association for Computational Linguistics_ , 10:92–110.
  * de Marneffe et al. (2012) Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2012.  Did it happen? The pragmatic complexity of veridicality assessment.  _Computational Linguistics_ , 38(2):301–333.
  * Dickinson and Meurers (2003) Markus Dickinson and W. Detmar Meurers. 2003.  Detecting Errors in Part-of-Speech Annotation.  In _10th Conference of the European Chapter of the Association for Computational Linguistics_ , Budapest, Hungary. Association for Computational Linguistics.
  * Gordon et al. (2022) Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022.  Jury learning: Integrating dissenting voices into machine learning models.  In _CHI Conference on Human Factors in Computing Systems_ , pages 1–19.
  * Huang et al. (2019) Jinchi Huang, Lie Qu, Rongfei Jia, and Binqiang Zhao. 2019.  O2U-Net: A Simple Noisy Label Detection Approach for Deep Neural Networks.  In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 3325–3333.
  * Jiang and de Marneffe (2022) Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022.  Investigating reasons for disagreement in natural language inference.  _Transactions of the Association for Computational Linguistics_ , 10:1357–1374.
  * Jiang et al. (2023) Nan-Jiang Jiang, Chenhao Tan, and Marie-Catherine de Marneffe. 2023.  Ecologically valid explanations for label variation in NLI.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 10622–10633, Singapore. Association for Computational Linguistics.
  * Klie et al. (2023) Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2023.  Annotation error detection: Analyzing the past and present for a more coherent future.  _Computational Linguistics_ , 49(1):157–198.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, and Jason Mars. 2019.  Outlier detection for improved data quality and diversity in dialog systems.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 517–527, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Manning (2011) Christopher D Manning. 2011.  Part-of-speech tagging from 97% to 100%: Is it time for some linguistics?  In _International conference on intelligent text processing and computational linguistics_ , pages 171–189. Springer.
  * Mazumder et al. (2023) Mark Mazumder, Colby Banbury, Xiaozhe Yao, Bojan Karlaš, William Gaviria Rojas, Sudnya Diamos, Greg Diamos, Lynn He, Alicia Parrish, Hannah Rose Kirk, Jessica Quaye, Charvi Rastogi, Douwe Kiela, David Jurado, David Kanter, Rafael Mosquera, Juan Ciro, Lora Aroyo, Bilge Acun, Lingjiao Chen, Mehul Smriti Raje, Max Bartolo, Sabri Eyuboglu, Amirata Ghorbani, Emmett Goodman, Oana Inel, Tariq Kane, Christine R. Kirkpatrick, Tzu-Sheng Kuo, Jonas Mueller, Tristan Thrush, Joaquin Vanschoren, Margaret Warren, Adina Williams, Serena Yeung, Newsha Ardalani, Praveen Paritosh, Lilith Bat-Leah, Ce Zhang, James Zou, Carole-Jean Wu, Cody Coleman, Andrew Ng, Peter Mattson, and Vijay Janapa Reddi. 2023.  DataPerf: Benchmarks for data-centric AI development.
  * Motamedi et al. (2021) Mohammad Motamedi, Nikolay Sakharnykh, and Tim Kaldewey. 2021.  A data-centric approach for training deep neural networks with less data.
  * Nie et al. (2020) Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020.  What can we learn from collective human opinions on natural language inference data? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9131–9143, Online. Association for Computational Linguistics.
  * Nogueira et al. (2019) Rodrigo Frassetto Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019.  Multi-stage document ranking with BERT.  _CoRR_ , abs/1910.14424.
  * Northcutt et al. (2021) Curtis G Northcutt, Anish Athalye, and Jonas Mueller. 2021.  Pervasive label errors in test sets destabilize machine learning benchmarks.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _CoRR_ , abs/2303.08774.
  * Pavlick and Kwiatkowski (2019) Ellie Pavlick and Tom Kwiatkowski. 2019.  Inherent disagreements in human textual inferences.  _Transactions of the Association for Computational Linguistics_ , 7:677–694.
  * Peterson et al. (2019) Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. 2019.  Human uncertainty makes classification more robust.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9616–9625.
  * Plank (2022) Barbara Plank. 2022.  The “problem” of human label variation: On ground truth in data, modeling and evaluation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Plank et al. (2014) Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014.  Linguistically debatable or just plain wrong?  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 507–511.
  * Reiss et al. (2020) Frederick Reiss, Hong Xu, Bryan Cutler, Karthik Muthuraman, and Zachary Eichenberger. 2020.  Identifying Incorrect Labels in the CoNLL-2003 Corpus.  In _Proceedings of the 24th Conference on Computational Natural Language Learning_ , pages 215–226, Online. Association for Computational Linguistics.
  * Rottger et al. (2022) Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022.  Two contrasting data annotation paradigms for subjective NLP tasks.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 175–190, Seattle, United States. Association for Computational Linguistics.
  * Rücker and Akbik (2023) Susanna Rücker and Alan Akbik. 2023.  CleanCoNLL: A nearly noise-free named entity recognition dataset.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 8628–8645, Singapore. Association for Computational Linguistics.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.01108.
  * Schmarje et al. (2023) Lars Schmarje, Vasco Grossmann, Tim Michels, Jakob Nazarenus, Monty Santarossa, Claudius Zelenka, and Reinhard Koch. 2023.  Label smarter, not harder: CleverLabel for faster annotation of ambiguous image classification with higher quality.  _arXiv preprint arXiv:2305.12811_.
  * Siddiqui et al. (2023) Shoaib Ahmed Siddiqui, Nitarshan Rajkumar, Tegan Maharaj, David Krueger, and Sara Hooker. 2023.  Metadata archaeology: Unearthing data subsets by leveraging training dynamics.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * Sommerauer et al. (2020) Pia Sommerauer, Antske Fokkens, and Piek Vossen. 2020.  Would you describe a leopard as yellow? Evaluating crowd-annotations with justified and informative disagreement.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4798–4809, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Swayamdipta et al. (2020) Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020.  Dataset cartography: Mapping and diagnosing datasets with training dynamics.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9275–9293, Online. Association for Computational Linguistics.
  * Uma et al. (2021) Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021.  Learning from Disagreement: A Survey.  _Journal of Artificial Intelligence Research_ , 72:1385–1470.
  * Vasudevan et al. (2022) Vijay Vasudevan, Benjamin Caine, Raphael Gontijo-Lopes, Sara Fridovich-Keil, and Rebecca Roelofs. 2022.  When does dough become a bagel? Analyzing the remaining mistakes on ImageNet.  _arXiv preprint arXiv:2205.04596_.
  * Wang et al. (2019) Zihan Wang, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, and Jiawei Han. 2019.  CrossWeigh: Training Named Entity Tagger from Imperfect Annotations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5154–5163, Hong Kong, China. Association for Computational Linguistics.
  * Weber and Plank (2023) Leon Weber and Barbara Plank. 2023.  ActiveAED: A human in the loop improves annotation error detection.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8834–8845, Toronto, Canada. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Zhang et al. (2021) Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021.  Learning with different amounts of annotation: From zero to many labels.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7620–7632, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang and de Marneffe (2021) Xinliang Frederick Zhang and Marie-Catherine de Marneffe. 2021.  Identifying inherent disagreement in natural language inference.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4908–4915, Online. Association for Computational Linguistics.
  * Zheng et al. (2023) Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2023.  Efficiently programming large language models using sglang.  _arXiv preprint arXiv:2312.07104_.

##  Appendix 0.A Data Statistics

###  0.A.1 Pair-wise inter-annotator agreements (Cohen’s kappa) on MASI-
distance for non-validated, self-validated, and peer-validated versions

versions \ annotators | 1-vs-2 | 1-vs-3 | 1-vs-4 | 2-vs-3 | 2-vs-4 | 3-vs-4
---|---|---|---|---|---|---
before validation | 40.87 | 42.22 | 37.71 | 36.21 | 31.17 | 34.62
self-validated | 60.06 | 53.84 | 61.61 | 44.47 | 47.79 | 47.06
peer-validated | 66.09 | 72.03 | 67.64 | 64.07 | 68.05 | 68.78

Table 4: Pair-wise inter-annotator agreements (Cohen’s kappa) on MASI-distance
for non-validated, self-validated, and peer-validated versions.

###  0.A.2 Frequency of NLI label on non-validation, self-validated, and peer-
validated explanation-label pairs

Figure 5: Frequency of NLI label sets on non-, self- and peer-validated label-
explanation pairs.

##  Appendix 0.B GPT Prompt

id: 72870c System: You are an expert linguistic annotator. User: We have
collected annotations for an NLI instance together with reasons for the
labels. Your task is to judge whether the reasons make sense for the label.
Provide the probability (0.0 - 1.0) that the reason makes sense for the label.
Give ONLY the reason and the probability, no other words or explanation. For
example: Reason: <The verbatim copy of the reason> Probability: <the
probability between 0.0 and 1.0 that the reason makes sense for the label,
without any extra commentary whatsoever; just the probability!>. Context:
Because marginal costs are very low, a newspaper price for preprints might be
as low as 5 or 6 cents per piece. Statement: Newspaper preprints can cost as
much as $5. Reason for label entailment: 5 dollars for a piece of newspaper
Reason for label neutral: The context only mentions how low the price may be,
not how high it may be. Reason for label neutral: The maximum cost of
newspaper preprints is not given in the context. Reason for label
contradiction: The context says 5 or 6 cents, not $5. User: Reason: 5 dollars
for a piece of newspaper Probability: Assistant: 0.0 User: Reason: The context
only mentions how low the price may be, not how high it may be. Probability:
Assistant: 0.9 User: Reason: The maximum cost of newspaper preprints is not
given in the context. Probability: Assistant: 0.8 User: Reason: The context
says 5 or 6 cents, not $5. Probability: Assistant: 0.9

Figure 6: GPT Prompt for predicting likelihood probability of label-
explanation pairs.

##  Appendix 0.C More Examples

|  Premise: Students of human misery can savor its underlying sadness and
futility.
---
Hypothesis: Students of human misery will be delighted to see how sad it truly
is.
Label-explanation pairs: before validation: {E:1,N:2,C:1} Self-validated :
{E:1,N:1} Peer-validated: {N:1}
Labels: [E, N] Error: [C]
Round 1: NLI Label & Explanation | Round 2: Validity
L | A | Explanation | 1 | 2 | 3 | 4
E | 2 | ‘‘can savor’’ implies ‘‘will be delighted’’. | ✓ | ✓ | ✕ | ✕
N | 1 | It is not clear from the context if the students will be delighted. | ✕ | ✕ | ✓ | ✓
3 |  | Students of human misery can ‘‘savored’’ that sadness, so maybe they are delighted
---
to see that, maybe they are tortured by the disasters.
✕ | ✕ | ✓ | ✓
C | 4 | Savor means to understand. Not to enjoy. | ✕ | ✕ | ? | ✕

(a)  id: 116176c

|  Premise: The tree-lined avenue extends less than three blocks to the sea.
---
Hypothesis: The sea isn’t even three blocks away.
Label-explanation pairs: before validation: {‘‘E’’:4,‘‘N’’:1,‘‘C’’:1} Self-
validated: {‘‘E’’:3,‘‘N’’:1} Peer-validated: {‘‘E’’:4,‘‘N’’:1}
Labels: [E, N] Error: [C]
Round 1: NLI Label & Explanation | Round 2: Validity
L | A | Explanation | 1 | 2 | 3 | 4
E | 1 | Both premise and hypothesis talk about less than three blocks. | ✓ | ✓ | ✓ | ✕
2 | If the avenue reaches the sea after less than three blocks, it cannot be further away. | ✓ | ✓ | ✓ | ✕
3 | The avenue is less than three blocks to the sea. | ✓ | ✓ | ✓ | ✕
4 | If the hypothesis means that the sea is less than three blocks away. | ? | ✓ | ✓ | ✕
N | 3 | It is not given where is the location of the narrator. | ✓ | ✕ | ✓ | ✓
C | 4 | If the hypothesis means that the sea is more than three blocks away. | ? | ✕ | ? | ✕

(b)  id: 80630e

|  Premise: As he stepped across the threshold, Tommy brought  the picture
down with terrific force on his head.
---
Hypothesis: Tommy hurt his head bringing the picture down.
Label-explanation pairs: before validation: {‘‘E’’:3,‘‘N’’:1,‘‘C’’:1} Self-
validated: {‘‘E’’:3,‘‘N’’:1} Peer-validated: {‘‘E’’:3,‘‘N’’:1}
Labels: [E, N] Error: [C]
Round 1: NLI Label & Explanation | Round 2: Validity
L | A | Explanation | 1 | 2 | 3 | 4
E | 1 | the picture hit Tommy in the head | ✓ | ✓ | ✓ | ✕
2 | a picture hit Tommy’s head with terrific force | ✓ | ✓ | ✓ | ✕
3 | Tommy hurt his head with the picture | ✓ | ✓ | ✓ | ✕
N | 3 | ambiguous if Tommy hurt himself or another guy | ✓ | ✓ | ✓ | ✕
C | 4 | Tommy is not hurt but rather bad strong emotion | ✕ | ✕ | ✓ | ✕

(c)  id: 77893n

Table 5: Additional sample annotations from NLI corpus. L: Label, A:
Annotator; E: Entailment, N: Neutral, C: Contradiction; magenta: self-
judgments, black: peer-judgments, Err: label error.