# Impact Probability Under Aleatory And Epistemic Uncertainties

Chiara Tardioli111Department of Mechanical & Aerospace Engineering, University
of Strathclyde, Glasgow, G1 1XJ, UK Davide Farnocchia222Jet Propulsion
Laboratory, California Institute of Technology, Pasadena, CA 91109, USA
Massimiliano Vasile††footnotemark:  Steve R. Chesley††footnotemark:

###### Abstract

We present an approach to estimate an upper bound for the impact probability
of a potentially hazardous asteroid when part of the force model depends on
unknown parameters whose statistical distribution needs to be assumed. As case
study we consider Apophis’ risk assessment for the 2036 and 2068 keyholes
based on information available as of 2013. Within the framework of epistemic
uncertainties, under the independence and non-correlation assumption, we
assign parametric families of distributions to the physical properties of
Apophis that define the Yarkovsky perturbation and in turn the future orbital
evolution of the asteroid. We find ${\rm IP}\leq 5\times 10^{-5}$ for the 2036
keyhole and ${\rm IP}\leq 1.6\times 10^{-5}$ for the 2068 keyhole. These upper
bounds are largely conservative choices due to the rather wide range of
statistical distributions that we explored.

##  1 Introduction

In risk analysis, uncertainties are generally classified into two categories:
aleatory and epistemic. _Aleatory uncertainty_ is an inherent variation
associated with the physical system or the environment. It may arise from
environmental randomness, variation in space, fluctuations in times, or other
system variability. Given the repetitiveness of the error, it is generally
well quantified with a known probability distribution, and, in principle, it
cannot be eliminated by further observational data, although it may be better
characterized (Kolmogorov and Bharucha-Reid,, 2018). Conversely, _epistemic
uncertainty_ is due to a lack of information about the system or phenomenon
under investigation. It may come from a lack of experimental data to
characterize new materials and processes, poor understanding of coupled
physics phenomena, or a lack of knowledge about the model formulation (see,
e.g., Helton, (1994); Oberkampf et al., (2002); Helton et al., (2004); Zio and
Pedroni, (2013)).

In the last decades there has been an increasing awareness that classical
probability theory is inadequate for the treatment of epistemic uncertainty.
Therefore, different non-probabilistic approaches have been developed:
imprecise probability, also known as interval analysis, after Walley, (1991)
and Berger et al., (1994); probability bound analysis, which combines
probability analysis and interval analysis (see, e.g., Ferson and Ginzburg,,
1996); Dempster-Shafer theory, which uses Belief and Plausibility functions,
in the two forms proposed by Dempster et al., (1968) and Shafer, (1976);
possibility theory, which is a special case of interval analysis and Dempster-
Shafer theory (see, e.g., Baudrit and Dubois,, 2006). For a review on methods
of representing uncertainty see, e.g., Zio and Pedroni, (2013).

Recently, Tardioli and Vasile, (2016) presented an approach to the design of
optimal collision avoidance and re-entry maneuvers under uncertainty. They
considered a dynamical model with six aleatory variables (the three components
of the position and velocity vectors of the spacecraft) and four epistemic
variables (some model parameters). The uncertainty was propagated through the
dynamics and the expectation of an optimal deflection maneuver was computed
with two different techniques: one using Belief and Plausibility functions and
the other using families of parametric distributions. The Belief and
Plausibility functions use no a-priori assumption on the kind of the
distribution but any distribution enclosed between an upper and lower
distribution is admissible. Whereas, in the parametric distribution approach
the assumption is that the probability density function of the uncertain
quantity belongs to a family of known distributions parametrized with unknown
parameters. Following this work, in this paper, we use a parametric
distribution approach to include epistemic uncertainties in the estimation of
impact probabilities for potentially hazardous asteroids.

Asteroid (99942) Apophis was discovered by R.A. Tucker, D.J. Tholen, and F.
Bernardi at Kitt Peak, Arizona on June 2004 (Minor Planet Supplement 109613).
With a minimum orbit intersection distance less than 0.05 au and an absolute
magnitude less than 22 (i.e., its diameter is larger than 140 m), Apophis is
classified as potentially hazardous asteroid. In December 2004, both
Sentry333https://cneos.jpl.nasa.gov/sentry. and
NEODyS444https://newton.spacedys.com/neodys. reported a probability of impact
with our planet of a few percent in April 2029 (Chesley,, 2006). Although
further observations reduced Apophis’ orbital uncertainty and ruled out any
impact possibility for 2029, the asteroid remains an interesting object worth
investigating. In fact, because of the scattering effect of the 2029
encounter, even small perturbations to the orbit of Apophis affect subsequent
impact predictions (Giorgini et al.,, 2008).

Chesley, (2006) shows that the exact encounter circumstances in 2029 depend on
the magnitude of the Yarkovsky effect (order of $10^{-15}$ au/day2). The
Yarkovsky effect is a non-gravitational perturbation due to the anisotropic
emission of thermal radiation of Solar System objects that causes a secular
drift in the semi-major axis (Bottke Jr et al.,, 2006; Vokrouhlický et al.,
2015a, ). For some near-Earth asteroids the orbital drift associated with the
Yarkovsky effect can be measured from the orbital fit to the observations
(Farnocchia et al., 2013b, ). That was the case of (101955) Bennu (Chesley et
al.,, 2014) and (29075) 1950 DA (Farnocchia and Chesley,, 2014). When the
astrometry data does not allow such a detection a Monte Carlo simulation
relying on the best physical model currently available is performed.
Nevertheless, the assumed distributions on the asteroid physical parameters
are themselves uncertain and may rely on subjective judgement. That was the
case of Apophis as analyzed by Farnocchia et al., 2013a , which we revisit
within the epistemic uncertainty framework in this paper.

##  2 Apophis risk analysis and Yarkovsky effect

The geometry of close approaches can be described projecting the relative
Earth-asteroid distance on an appropriate plane called the $b$-plane or target
plane, which is defined as the plane including the center of the Earth and
normal to the incoming asymptote of the small body trajectory with respect to
the planet during the close encounter (see, e.g., Milani et al.,, 2002;
Valsecchi et al.,, 2003). Conventionally, the coordinates on the $b$-plane are
called $(\xi,\zeta)$ and are defined so that the projection of the Earth’s
heliocentric velocity onto the $b$-plane defines the negative $\zeta$-axis.
The $\zeta$ coordinate tells if the asteroid is early or late for the minimum
possible distance encounter. The $\xi$ coordinate is the minimum distance that
can be obtained by varying the timing of the encounter. The $b$-plane
coordinates leading to a resonant impact lie on predictable circles (Valsecchi
et al.,, 2003). The intersection between the orbital uncertainty region and a
Valsecchi circle is called keyhole (Chodas,, 1999) and corresponds to a future
impact. Assigning a probability to the uncertain variables involved in the
dynamical system, a probability of impact can be estimated. For a
comprehensive review on the $b$-plane and the corresponding encounter analysis
refer to Farnocchia et al., (2019).

Farnocchia et al., 2013a  presented an impact risk analysis for asteroid
(99942) Apophis. The authors located 20 keyholes on the 2029 $b$-plane and
found that the probability density function of the $\zeta$-coordinate was
completely driven by the dispersion due to the Yarkovsky effect. The Yarkovsky
perturbation was modeled as a transverse acceleration $A_{2}/r^{2}$ where $r$
is the heliocentric distance and $A_{2}$ is a function of certain physical
quantities (diameter, Bond albedo, bulk density, thermal inertia, rotational
period, and spin orientation). Thus, different physical characteristics of the
asteroid result in a different Yarkovsky perturbation. Farnocchia et al.,
2013a  assumed a specific distribution for each of the unknown physical
parameters that define the Yarkovsky effect. Then, for each keyhole the impact
probability was computed as the product of the keyhole width and the value of
the $\zeta$ distribution, evaluated in a point of the keyhole. The keyhole
related to April 2068, situated in the core of the $\zeta$ probability density
distribution, was found to have the highest risk encounter with an impact
probability of three in a million. On the other hand, the keyhole associated
to April 2036, situated in the tail of the distribution, had the highest width
(about $600$ m) with an impact probability of seven in a billion.

These results were superseded by later astrometry and physical
characterization (Vokrouhlický et al., 2015b, ; Brozović et al.,, 2018).
However, in this work we consider Apophis’ 2013 scenario and validate our
uncertainty quantification method by computing an upper and lower limit for
the impact probabilities related to the 2036 and 2068 keyholes on the 2029
$b$-plane.

##  3 Uncertainty Quantification of the Physical Model

The Yarkovsky perturbation can be approximated as (Vokrouhlický et al., 2015b,
)

$a_{t}\simeq\dfrac{4(1-A)}{9}\Phi(1~{}{\rm au})f(\theta)\cos\gamma$ (1)

where $A$ is the Bond albedo, $\Phi(1~{}{\rm au})=3G_{S}/(2D\rho c)$ is the
standard radiation force factor at 1 au, $G_{S}=1361$ W/m2 is the solar
constant, $D$ is the asteroid’s diameter, $\rho$ is the bulk density, $\gamma$
is the spin axis obliquity, and $\theta_{rot}$ is the thermal parameter, which
depends on the rotation period, spin rate, thermal inertia, thermal
emissivity, geometric albedo, and radial distance from the Sun. These physical
quantities are in general poorly characterised, especially shortly after
discovery when there is little or no information regarding the key physical
characteristics. Therefore, the uncertainties associated to the physical
quantities are epistemic rather than aleatory. Following Tardioli and Vasile,
(2016), we employ a parametric distribution approach to treat epistemic
uncertainties.

In a parametric distribution approach, the actual probability density function
is unknown but belongs to a family of distributions parametrized with unknown
parameters. For example, a family of Gaussian probability density function
(${\rm pdf}_{\mathcal{N}}$) with mean $\mu\in[0.4,0.5]$ and standard deviation
$\sigma\in[0.5,1.5]$ is

$[{\rm pdf}_{\mathcal{N}}]=\\{{\rm
pdf}_{\mathcal{N}}(u;\mu,\sigma):0.4\leq\mu\leq 0.6\,,\ 0.05\leq\sigma\leq
0.15\,,\ \forall u\in\mathbb{R}\\}\,,$ (2)

or, using Gaussian cumulative distribution functions (${\rm
cdf}_{\mathcal{N}}$),

$[{\rm cdf}_{\mathcal{N}}]=\\{{\rm
cdf}_{\mathcal{N}}(u;\mu,\sigma):0.4\leq\mu\leq 0.6\,,\ 0.05\leq\sigma\leq
0.15\,,\ \forall u\in\mathbb{R}\\}\ .$ (3)

The families of distributions in Eq. (2) and Eq.(3), often called probability
box, or, shortly, p-box, are illustrated in Fig. 1.

Figure 1: P-box (grey region) defined by a family of Gaussian pdf’s (left) and
a family of Gaussian cdf’s (right).

A p-box of pdf’s (resp. cdf’s) is a set enclosing the minimum and maximum
values of the pdf’s (resp. cdf’s). Particularly, the p-box of cdf’s can always
be enclosed from above and from below by piece-wise curves, called,
respectively, upper expectation $E_{u}$ and lower expectation $E_{l}$ (Ferson
et al.,, 2015):

$[{\rm cdf}]=\\{{\rm cdf}\>:\>E_{l}\leq{\rm cdf}\leq E_{u}\\}\ .$ (4)

In the multivariate case, under the independence and non-correlation
assumption, the joint probability distribution belongs to a p-box whose
elements are the product of univariate pdf’s. For example, the p-box of two
families of Gaussian pdf’s as in Eq. (2) can be written as

$\big{[}{\rm pdf}_{\mathcal{N}_{1}\mathcal{N}_{2}}\big{]}=\big{\\{}{\rm
pdf}_{\mathcal{N}_{1}}\times{\rm pdf}_{\mathcal{N}_{2}}\>:\>{\rm
pdf}_{\mathcal{N}_{1}}\in\big{[}{\rm pdf}_{\mathcal{N}_{1}}\big{]}\ \wedge\
{\rm pdf}_{\mathcal{N}_{2}}\in\big{[}{\rm
pdf}_{\mathcal{N}_{2}}\big{]}\big{\\}}\ .$ (5)

A similar definition holds for the product of two families of Gaussian cdf’s.

For the Farnocchia et al., 2013a  scenario, the families of distributions
associated to Apophis’ physical quantities describing the Yarkovsky effect are
reported in Table 1. To enhance the variability in the impact probability, we
purposely considered a wide range of parameters, thus not necessarily
representing the most realistic scenario.

  * •

Diameter. Delbò et al., 2007a  estimate a diameter $D=(270\pm 60)$ m, while
Müller et al., (2014) derive greater values $D=375^{+14}_{-10}$ m. Licandro et
al., (2016) confirm a diameter in range 380–393 m. We model the diameter
uncertainty with normal distributions with mean between $270$ and $385$ m, and
standard deviation between $5$ and $60$ m.

  * •

Slope parameter. Pravec et al., (2014) assume a slope parameter $G=0.24\pm
0.11$ for S and Q types asteroids. Mommert et al., (2014) use $G=0.18\pm
0.13$, obtained averaging all $G$ measurements of asteroids from JPL Small-
Body Database Search Engine555JPL Small-Body Database Search Engine:
http://ssd.jpl.nasa.gov/sbdb_query.cgi.. Farnocchia et al., 2013a  assumed
$G=0.15\pm 0.05$. Therefore, we choose normal distributions with mean between
$0.15$ and $0.24$, and standard deviation between $0.05$ and $0.15$.

  * •

Geometric albedo. From polarimetric observations Delbò et al., 2007a  find a
geometric albedo $p_{V}=0.33\pm 0.08$. Müller et al., (2014) obtain
$p_{V}=0.30^{+0.05}_{-0.06}$ from far-infrared observations with ESA’s
Herschel space observatory. Recent thermal infrared observations put $p_{V}$
in the range 0.24–0.33 (Licandro et al.,, 2016), in accordance with radar
observations that gives $p_{V}=0.35\pm 0.10$ (Brozović et al.,, 2018).
Farnocchia et al., 2013a  used $p_{V}=0.23\pm 0.02$. Therefore, we use
lognormal distributions whose corresponding Gaussian distributions have mean
between 0.23 and 0.35, and standard deviation between 0.02 and 0.10. Then, the
Bond albedo can be derived from the slope parameter and the geometric albedo
through the formula $A=(0.29+0.684G)\,p_{V}$ (Bowell et al.,, 1989).

  * •

Bulk density. From the grain density and total porosity reported in Binzel et
al., (2009), Farnocchia et al., 2013a  obtained a lognormal distribution with
mean $2.2$ g/cm3 and variance $0.3$ g2/cm6. However, Apophis is an Sq-type
asteroid and the typical density of this taxonomic class is $2.78\pm 0.85$
g/cm3 (Carry,, 2012). Thus, we use lognormal distributions with mean between 2
and 3 g/cm3, and standard deviation between 0.5 and 0.85 g/cm3.

  * •

Thermal Inertia. According to (Delbò et al., 2007b, , Fig. 6) the thermal
inertia $\Gamma$ of near-Earth objects ranges between 100 and 1 000
Jm-2s-0.5K-1, thus the uncertainty of $\Gamma$ can be represented by
$\log_{10}(\Gamma)=2.5\pm 0.5$. Müller et al., (2014) use full range of
$\Gamma$ of 250–800 Jm-2s-0.5K-1, with best solution at $\Gamma=600$
Jm-2s-0.5K-1, giving $\log_{10}(\Gamma)=2.7\pm 0.25$. Licandro et al., (2016)
constrain the thermal inertia of Apophis to lie in the range 50–500
Jm-2s-0.5K-1, giving $\log_{10}(\Gamma)=2.2\pm 0.5$. Farnocchia et al., 2013a
used a generic relationship between the thermal inertia and the diameter which
led to $\log_{10}(\Gamma)=2.65\pm 0.08$. Hence, we consider a normal
distribution for $\log_{10}(\Gamma)$ with mean between 2.2 and 2.7, and
standard deviation between 0.08 and 0.5.

  * •

Rotational period. The uncertainty on the rotational period is very small and
it does not affect the impact probability (Farnocchia et al., 2013a, ). Pravec
et al., (2014) report a rotational period $P_{\rm rot}=(30.56\pm 0.01)$ h,
therefore we used a normal distribution with this parameter.

  * •

Obliquity. In our scenario we neglect Apophis’ complex rotation and retrograde
motion (Pravec et al.,, 2014). We consider a simple discrete model with 4 bins
in $-1\leq\cos\gamma\leq 1$, and positive frequencies
$f_{1},f_{2},f_{3},f_{4}$ such that $f_{1}+\ldots+f_{4}=1$.

Table 1: Distribution families associated to the uncertain variables defining the Yarkovsky effect for Apophis’ 2013 scenario. With $\mu$ and $\sigma$ we denote the mean and the standard deviation of the corresponding distribution. Note that the first six variables are epistemic, the last one is aleatory. For units refer to Table 2. Variable | Distribution Type | Parameter Ranges
---|---|---
Diameter | Normal | $270\leq\mu\leq 385$
$5\leq\sigma\leq 60$
Slope parameter | Normal | $0.15\leq\mu\leq 0.24$
$0.05\leq\sigma\leq 0.15$
Geometric albedo | Lognormal | $0.23\leq\mu\leq 0.35$
$0.02\leq\sigma\leq 0.10$
Bulk density | Lognormal | $2\leq\mu\leq 3$
$0.50\leq\sigma\leq 0.85$
$\log_{10}({\rm Thermal\ Inertia})$ | Normal | $2.2\leq\mu\leq 2.7$
$0.08\leq\sigma\leq 0.50$
Rotational period | Normal | $\mu=30.56$
$\sigma=0.01$
Obliquity cosine | 4-bin distr. | $0\leq f_{1}\leq 1$
$0\leq f_{2}\leq 1-f_{1}$
$0\leq f_{3}\leq 1-f_{1}-f_{2}$
$0\leq f_{4}\leq 1-f_{1}-f_{2}-f_{3}$
Table 2: Uncertainty space defined by the physical quantities involved in the Yarkovsky effect for Apophis’ 2013 scenario. Variable | Symbol | Lower Bound | Upper Bound | Unit
---|---|---|---|---
Diameter | D | 50 | 600 | m
Slope parameter | $G$ | $-0.424$ | 0.8 | -
Geometric albedo | $p_{V}$ | 0 | 1 | -
Bulk density | $\rho$ | 0.5 | 5 | g/cm3
$\log_{10}({\rm Thermal\ Inertia})$ | $\log_{10}(\Gamma)$ | 1 | 10000 | Jm-2s-0.5K-1
Rotational period | $P_{\rm rot}$ | 30.5 | 30.6 | h
Obliquity cosine | $\cos\gamma$ | $-1$ | 1 | -

In total, we have six epistemic variables and one aleatory variable. The
uncertainty analysis under aleatory and epistemic uncertainty starts from the
definition of the uncertainty space $\mathcal{U}$, which is a 7-dimensional
hyper-rectangle defined by the full ranges of Apophis’ physical parameters
involved in the Yarkovsky effect (see Table 2). The underlying assumption is
that of independence and non-correlation among the uncertain variables.

Farnocchia et al., 2013a  map the Yarkovsky-related semi-major axis drift onto
the 2029 $b$-plane and its $\zeta_{\rm 2029}$ coordinate. Denoting this map as
$f:\mathcal{U}\to\mathbb{R}$, we want to assess the probability of the event
$\mathcal{A}=\\{u\in\mathcal{U}:|f(u)-\zeta_{o}|\leq w/2\\}$, where
$\zeta_{o}$ is the center of a generic keyhole on the 2029 $b$-plane and $w$
its width. If all uncertain variables are aleatory, then the probability of
$\mathcal{A}$ coincides with the impact probability associated with the
keyhole $\zeta_{o}$: ${\rm IP}={\rm P}(\mathcal{A})$. Instead, if the
distribution parameters associated to the uncertain physical quantities vary
into intervals, then the probability of $\mathcal{A}$ is bound by a lower and
upper value given by

${\rm P}_{*}(\mathcal{A})=\min_{\mathbf{j}\in J}\int_{\mathcal{A}}{\rm
pdf}_{\mathbf{j}}(\mathbf{u})\,\,\mathrm{d}\mathbf{u}\,,\qquad{\rm
P}^{*}(\mathcal{A})=\max_{\mathbf{j}\in J}\int_{\mathcal{A}}{\rm
pdf}_{\mathbf{j}}(\mathbf{u})\,\,\mathrm{d}\mathbf{u}\,,$ (6)

where $\mathbf{u}=(D,G,p_{V},\rho,\Gamma,\cos\gamma)$ is the epistemic
variable vector,

$\mathbf{j}=(\mu_{1},\sigma_{1},\ldots,\mu_{5},\sigma_{5},f_{1},f_{2},f_{3})$
(7)

is the unknown distribution parameter vector (the parameters defining the
first six distribution families in Table 1), $J=J_{1}\times\ldots J_{13}$ is
the distribution parameter space with $J_{i}$ the range defined in Table 1 for
each $i=1,\ldots 13$, and ${\rm pdf}_{\mathbf{j}}$ is the joint probability
distribution product varying in the p-box $[{\rm
pdf}_{\mathbf{j}}:\mathbf{j}\in J]$ with rCl pdf_j&∝
pdf_N(μ_1,σ_1)×pdf_N(μ_2,σ_2)×pdf_LN(μ_3,σ_3)×pdf_LN(μ_4,σ_4)
× pdf_N(μ_5,σ_5)×pdf_N(μ_rot,σ_rot)×pdf_D_4(f_1,f_2,f_3,1-f_1-f_2-f_3) , where
$\mathcal{N}$ is the Gaussian, $\mathcal{LN}$ is the Lognormal, and
$\mathcal{D}_{4}$ the discrete 4-bin distribution. As a consequence, the
impact probability associated to the keyhole $\zeta_{o}$ is itself uncertain
and the following inequalities hold

${\rm IP}_{*}={\rm P}_{*}(\mathcal{A})\leq{\rm IP}\leq{\rm
P}^{*}(\mathcal{A})={\rm IP}^{*}\ .$ (8)

The two optimisation problems in Eq. (6) can be solved numerically by
replacing the calculation of the exact integrals with approximated forms using
a sampling scheme. In fact, given $M$ sample points
$\mathbf{u}_{1},\ldots,\mathbf{u}_{M}$, each integral in Eq. (6) can be
approximated as

$\sum_{k=1}^{M}I_{\mathcal{A}}(\mathbf{u}_{k})\,p_{k}\,,\quad p_{k}\propto{\rm
pdf}_{\mathbf{j}}(\mathbf{u}_{k})\,,\quad\sum_{k=1}^{M}p_{k}=1\,,$ (9)

where $I_{\mathcal{A}}$ is the indicator function of the set $\mathcal{A}$,
that is 1 if $\mathbf{u}_{k}$ belongs to $\mathcal{A}$, 0 otherwise. In this
work we will use a scheme with $10^{12}$ uniformly distributed Monte Carlo
points.

##  4 Results

We consider Apophis’ 2013 scenario and the keyholes found by Farnocchia et
al., 2013a  on the 2029 $b$-plane. The uncertainties for the physical
parameters are reported in Tables 1 and 2. For two keyholes, we compute the
maximum and minimum of the impact probability. One keyhole corresponds to an
impact in 2036. This is the widest keyhole and is located in the tail of the
probability distribution. The other keyhole corresponds to an impact in 2068.
It is much smaller but within the core of the distribution and yielded the
highest impact probability in Farnocchia et al., 2013a . These two keyholes
require negative values of $A_{2}$ in order to be possible. In this paper, we
are considering all possible distributions for the obliquity (which determines
the sign of $A_{2}$). Thus, the minimum value of IP is always zero and our
uncertainty analysis gives an upper bound for IP rather than a proper
interval.

Results are displayed in Table 3. To validate our method, for each keyhole, we
compute a reference impact probability corresponding to fixed distributions
derived by the ones used in Farnocchia et al., 2013a  (see Table 4). The
distribution parameters corresponding to ${\rm IP}^{*}$ are found using
multiple runs of a global optimization method proposed by Vasile et al.,
(2011) and called IDEA (Inflationary Differential Evolution Algorithm). The
values, out of all the runs of IDEA, that give the highest impact probability
are reported in Table 5 and illustrated in Figure 2. Figure 3 shows the
uncertainty region of $\zeta_{2029}$ due to the uncertainties on the physical
parameters associated to the Yarkovsky effect. The distributions associated to
the reference and to the maximum impact probability for the 2036 and 2068
keyholes are also displayed.

Table 3: Upper bound of the impact probability (IP∗) for the 2036 and 2068 keyholes relative to the distribution families in Table 1. Other columns are: width and position of the center of the keyhole on the 2029 $b$-plane, impact probability (IPref) corresponding to the reference distribution parameters. Year | Width [m] | Position [km] | IPref | IP∗
---|---|---|---|---
2036 | 616.16 | 46115.5 | $5.0\times 10^{-9}$ | $4.6\times 10^{-5}$
2068 | 2.25 | 47447.1 | $2.2\times 10^{-6}$ | $1.6\times 10^{-5}$
Table 4: Reference distribution parameters. Variable | Distribution Type | Reference Parameters
---|---|---
$D$ | Normal |  $(325\pm 15)$ m
$G$ | Normal | $(0.15\pm 0.05)$
$p_{V}$ | Lognormal | $(0.23\pm 0.02)$
$\rho$ | Lognormal |  $(2.2\pm\sqrt{0.3})$ g/cm3
$\log_{10}(\Gamma)$ | Normal | $(2.65\pm 0.08)$
$P_{rot}$ | Normal |  $(30.56\pm 0.01)$ h
$\cos\gamma$ | 4-bin distr. | 53.68%, 15.44%, 8.09%, 22.79%
Table 5: Distribution parameters corresponding to the maximum impact probability (IP∗) for the 2036 and 2068 keyholes, respectively. Variable | 2036 Max | 2068 Max
---|---|---
$D$ |  $(270\pm 60)$ m |  $(361\pm 5)$ m
$G$ | $(0.15\pm 0.05)$ | $(0.23\pm 0.05)$
$p_{V}$ | $(0.23\pm 0.02)$ | $(0.29\pm 0.02)$
$\rho$ |  $(2.0\pm 0.85)$ g/cm3 |  $(3\pm 0.5)$ g/cm3
$\log_{10}(\Gamma)$ | $(2.70\pm 0.08)$ | $(2.41\pm 0.08)$
$P_{rot}$ |  $(30.56\pm 0.01)$ h |  $(30.56\pm 0.01)$ h
$\cos\gamma$ | 100%, 0%, 0%, 0% | 100%, 0%, 0%, 0%
Figure 2: Probability boxes and probability density functions of the asteroid
physical quantities for the 2036 and 2068 keyholes corresponding to IP∗ and
IPref.

Figure 3: Probability box on the 2029 $b$-plane associated to the distribution
families in Table 1 in the pdf-space (left) and the cdf-space (right). The
origin of the $\zeta_{2029}$-coordinate is set at 47659.24 km. The curves
represent the distribution functions corresponding to ${\rm IP}^{\rm ref}$ and
IP∗ for the 2036 and 2068 keyholes.

##  5 Discussion and Conclusions

We presented an approach to include epistemic uncertainty in the physical
model of the Yarkovsky effect and to estimate an upper bound for the impact
probability of a potentially hazardous asteroid. To illustrate the method we
considered asteroid Apophis and the 2036 and 2068 keyholes as discussed by
Farnocchia et al., 2013a . The maximum impact probability required the
solution of an optimization problem with a multidimensional integral as
objective function.

Our analysis was not meant to provide the most current and realistic hazard
assessment for Apophis. Rather, for demonstration purpose, we chose a wide
range of parameters that result in very conservative upper bounds for the
impact probability. For example, if we had fixed the 4-bin distribution of the
obliquity and set the frequencies to the reference values, we would have found
a maximum impact probability of $2.69\times 10^{-6}$ for the 2036 keyhole and
of $7.13\times 10^{-6}$ for the 2068 keyhole.

As more asteroids are discovered, the number of cases where the impact hazard
assessment is affected by the Yarkovsky effect and its modeling is set to
increase. The technique presented in this paper provides a rigorous framework
to characterize the sensitivity of the impact probability to the physical
model assumptions and in turn characterize its degree of variability.

The spin state of Apophis may change as a result of the 2029 encounter
(Scheeres et al.,, 2005). As already discussed by Farnocchia et al., 2013a ,
such a change would slightly change the location of the keyholes but have a
small effect on the impact probability. Therefore, our analysis remains valid
even in that case. Eventually, we remark that that the parametric distribution
approach can be applied to more general families of distributions, for
example, defined by a weighted combination of kernels.

##  6 Acknowledgements

We would like to thank the editor AC and the reviewer GV for their
recommendations. This is a post-peer-review, pre-copyedit version of an
article published in Celestial Mechanics and Dynamical Astronomy. The final
authenticated version is available online at:
https://doi.org/10.1007/s10569-020-09991-3

## References

  * Baudrit and Dubois, (2006) Baudrit, C. and Dubois, D. (2006).  Practical representations of incomplete probabilistic knowledge.  Computational statistics & data analysis, 51(1):86–108.
  * Berger et al., (1994) Berger, J. O., Moreno, E., Pericchi, L. R., Bayarri, M. J., Bernardo, J. M., Cano, J. A., De la Horra, J., Martín, J., Ríos-Insúa, D., Betrò, B., et al. (1994).  An overview of robust bayesian analysis.  Test, 3(1):5–124.
  * Binzel et al., (2009) Binzel, R. P., Rivkin, A. S., Thomas, C. A., Vernazza, P., Burbine, T. H., DeMeo, F. E., Bus, S. J., Tokunaga, A. T., and Birlan, M. (2009).  Spectral properties and composition of potentially hazardous Asteroid (99942) Apophis.  Icarus, 200:480–485.
  * Bottke Jr et al., (2006) Bottke Jr, W. F., Vokrouhlický, D., Rubincam, D. P., and Nesvorný, D. (2006).  The yarkovsky and yorp effects: Implications for asteroid dynamics.  Annu. Rev. Earth Planet. Sci., 34:157–191.
  * Bowell et al., (1989) Bowell, E., Hapke, B., Domingue, D., Lumme, K., Peltoniemi, J., and Harris, A. W. (1989).  Application of photometric models to asteroids.  In Asteroids II, pages 524–556.
  * Brozović et al., (2018) Brozović, M., Benner, L. A. M., McMichael, J. G., Giorgini, J. D., Pravec, P., Scheirich, P., Magri, C., Busch, M. W., Jao, J. S., Lee, C. G., Snedeker, L. G., Silva, M. A., Slade, M. A., Semenov, B., Nolan, M. C., Taylor, P. A., Howell, E. S., and Lawrence, K. J. (2018).  Goldstone and Arecibo radar observations of (99942) Apophis in 2012-2013.  Icarus, 300:115–128.
  * Carry, (2012) Carry, B. (2012).  Density of asteroids.  Planetary and Space Science, 73(1):98–118.
  * Chesley, (2006) Chesley, S. R. (2006).  Potential impact detection for Near-Earth asteroids: the case of 99942 Apophis (2004 MN 4 ).  In Daniela, L., Sylvio Ferraz, M., and Angel, F. J., editors, Asteroids, Comets, Meteors, volume 229 of IAU Symposium, pages 215–228.
  * Chesley et al., (2014) Chesley, S. R., Farnocchia, D., Nolan, M. C., Vokrouhlický, D., Chodas, P. W., Milani, A., Spoto, F., Rozitis, B., Benner, L. A., Bottke, W. F., et al. (2014).  Orbit and bulk density of the osiris-rex target asteroid (101955) bennu.  Icarus, 235:5–22.
  * Chodas, (1999) Chodas, P. (1999).  Orbit uncertainties, keyholes, and collision probabilities.  In Bulletin of the American Astronomical Society, volume 31, page 1117.
  * (11) Delbò, M., Cellino, A., and Tedesco, E. F. (2007a).  Albedo and size determination of potentially hazardous asteroids: (99942) Apophis.  Icarus, 188:266–269.
  * (12) Delbò, M., Dell’Oro, A., Harris, A. W., Mottola, S., Mueller, M., et al. (2007b).  Thermal inertia of near-earth asteroids and implications for the magnitude of the yarkovsky effect.  Icarus, 190(1):236–249.
  * Dempster et al., (1968) Dempster, A. P. et al. (1968).  Upper and lower probabilities generated by a random closed interval.  The Annals of Mathematical Statistics, 39(3):957–966.
  * (14) Farnocchia, D., Chesley, S., Chodas, P., Micheli, M., Tholen, D., Milani, A., Elliott, G., and Bernardi, F. (2013a).  Yarkovsky-driven impact risk analysis for asteroid (99942) apophis.  Icarus, 224(1):192–200.
  * (15) Farnocchia, D., Chesley, S., Vokrouhlický, D., Milani, A., Spoto, F., and Bottke, W. (2013b).  Near earth asteroids with measurable yarkovsky effect.  Icarus, 224(1):1–13.
  * Farnocchia and Chesley, (2014) Farnocchia, D. and Chesley, S. R. (2014).  Assessment of the 2880 impact threat from asteroid (29075) 1950 da.  Icarus, 229:321–327.
  * Farnocchia et al., (2019) Farnocchia, D., Eggl, S., Chodas, P. W., Giorgini, J. D., and Chesley, S. R. (2019).  Planetary encounter analysis on the b-plane: a comprehensive formulation.  Celestial Mechanics and Dynamical Astronomy, 131(8):36.
  * Ferson and Ginzburg, (1996) Ferson, S. and Ginzburg, L. R. (1996).  Different methods are needed to propagate ignorance and variability.  Reliability Engineering & System Safety, 54(2-3):133–144.
  * Ferson et al., (2015) Ferson, S., Kreinovich, V., Grinzburg, L., Myers, D., and Sentz, K. (2015).  Constructing probability boxes and dempster-shafer structures.  Technical report, Sandia National Lab.(SNL-NM), Albuquerque, NM (United States).
  * Giorgini et al., (2008) Giorgini, J. D., Benner, L. A., Ostro, S. J., Nolan, M. C., and Busch, M. W. (2008).  Predicting the earth encounters of (99942) apophis.  Icarus, 193(1):1–19.
  * Helton, (1994) Helton, J. C. (1994).  Treatment of uncertainty in performance assessments for complex systems.  Risk analysis, 14(4):483–511.
  * Helton et al., (2004) Helton, J. C., Johnson, J. D., and Oberkampf, W. L. (2004).  An exploration of alternative approaches to the representation of uncertainty in model predictions.  Reliability Engineering & System Safety, 85(1-3):39–71.
  * Kolmogorov and Bharucha-Reid, (2018) Kolmogorov, A. N. and Bharucha-Reid, A. T. (2018).  Foundations of the theory of probability: Second English Edition.  Courier Dover Publications.
  * Licandro et al., (2016) Licandro, J., Müller, T., Alvarez, C., Alí-Lagoa, V., and Delbo, M. (2016).  Gtc/canaricam observations of (99942) apophis.  Astronomy & Astrophysics, 585:A10.
  * Milani et al., (2002) Milani, A., Chesley, S. R., Chodas, P. W., and Valsecchi, G. B. (2002).  Asteroid Close Approaches: Analysis and Potential Impact Detection.  In Asteroids III, pages 55–69.
  * Mommert et al., (2014) Mommert, M., Farnocchia, D., Hora, J. L., Chesley, S. R., Trilling, D. E., Chodas, P. W., Mueller, M., Harris, A. W., Smith, H. A., and Fazio, G. G. (2014).  Physical Properties of Near-Earth Asteroid 2011 MD.  The Astrophysical Journal Letters, 789:L22.
  * Müller et al., (2014) Müller, T., Kiss, C., Scheirich, P., Pravec, P., O’Rourke, L., Vilenius, E., and Altieri, B. (2014).  Thermal infrared observations of asteroid (99942) apophis with herschel.  Astronomy & Astrophysics, 566:A22.
  * Oberkampf et al., (2002) Oberkampf, W. L., DeLand, S. M., Rutherford, B. M., Diegert, K. V., and Alvin, K. F. (2002).  Error and uncertainty in modeling and simulation.  Reliability Engineering & System Safety, 75(3):333–357.
  * Pravec et al., (2014) Pravec, P., Scheirich, P., Ďurech, J., Pollock, J., Kušnirák, P., Hornoch, K., Galád, A., Vokrouhlický, D., Harris, A., Jehin, E., et al. (2014).  The tumbling spin state of (99942) apophis.  Icarus, 233:48–60.
  * Scheeres et al., (2005) Scheeres, D., Benner, L., Ostro, S., Rossi, A., Marzari, F., and Washabaugh, P. (2005).  Abrupt alteration of asteroid 2004 mn4’s spin state during its 2029 earth flyby.  Icarus, 178(1):281–283.
  * Shafer, (1976) Shafer, G. (1976).  A mathematical theory of evidence, volume 42.  Princeton university press.
  * Tardioli and Vasile, (2016) Tardioli, C. and Vasile, M. (2016).  Collision and re-entry analysis under aleatory and epistemic uncertainty.  In Majji, M., Turner, J., Wawrzyniak, G., and Cerven, W., editors, Astrodynamics 2015, volume 156 of Advances in Astronautical Sciences, pages 4205–4220. Univelt Inc.
  * Valsecchi et al., (2003) Valsecchi, G. B., Milani, A., Gronchi, G. F., and Chesley, S. R. (2003).  Resonant returns to close approaches: Analytical theory.  Astronomy & Astrophysics, 408(3):1179–1196.
  * Vasile et al., (2011) Vasile, M., Minisci, E., and Locatelli, M. (2011).  An inflationary differential evolution algorithm for space trajectory optimization.  IEEE Transactions on Evolutionary Computation, 15(2):267–281.
  * (35) Vokrouhlický, D., Bottke, W. F., Chesley, S. R., Scheeres, D. J., and Statler, T. S. (2015a).  The Yarkovsky and YORP Effects.  In Asteroids IV, pages 509–531.
  * (36) Vokrouhlický, D., Farnocchia, D., Čapek, D., Chesley, S. R., Pravec, P., Scheirich, P., and Müller, T. G. (2015b).  The yarkovsky effect for 99942 apophis.  Icarus, 252:277–283.
  * Walley, (1991) Walley, P. (1991).  Statistical reasoning with imprecise probabilities.
  * Zio and Pedroni, (2013) Zio, E. and Pedroni, N. (2013).  Literature review of methods for representing uncertainty.  FonCSI.


Classifiers commonly make use of pre-annotated datasets, wherein a model is evaluated by pre-defined metrics on a held-out test set typically made of human-annotated labels. Metrics used in these evaluations are tied to the availability of well-defined ground truth labels, and these metrics typically do not allow for inexact matches. These noisy ground truth labels and strict evaluation metrics may compromise the validity and realism of evaluation results. In the present work, we discuss these concerns and conduct a systematic posthoc verification experiment on the entity linking (EL) task. Unlike traditional methodologies, which asks annotators to provide free-form annotations, we ask annotators to verify the correctness of  annotations after the fact (, posthoc). Compared to pre-annotation evaluation, state-of-the-art EL models performed extremely well according to the posthoc evaluation methodology. Posthoc validation also permits the validation of the ground truth dataset. Surprisingly, we find predictions from EL models had a similar or higher verification rate than the ground truth. We conclude with a discussion on these findings and recommendations for future evaluations.

§ INTRODUCTION

Recent advances in machine learning are due to the wide availability of large labeled datasets. These datasets are typically curated from artifacts of digital human behavior that arise naturally, as in the case of click-through data or Wikipedia comments. But in many cases, these annotations are derived from crowd-sourced tasks where annotations are provided by amateur workers [24, 5, 37, 5, 34, 47, 9]. Because they rely on subjective human judgement, the processes used to construct these datasets are inherently error-prone [42]. Strategies to decrease the error rate have been developed [17], but errors persist. Recent work has focused on training models using noisy, subjective, human-labeled labels [46], and many have concluded that deep learning systems are robust to noisy labels – so long as the training data is not too small and not too noisy [39].

Understanding and mitigating noise in benchmark datasets remains a major focus of research [26]. For example, in natural language processing (NLP) several approaches have been developed that are robust to noise [3], logically consistent [35], explainable [23], or fair [31]. Oftentimes only a handful of annotators produce most of the dataset annotations; this is known to result in annotator bias that can severely influence model performance and generalizability [21, 43, 15, 10, 30, 48, 1, 29, 9]. And sometimes human annotations are just wrong [41].

These efforts have lead to the development of machine learning systems that are more robust to noisy labels; unfortunately, annotations that appear in the held-out test set are typically considered to be correct even though they likely have the same noise as the training set [33, 25]. Pervasive annotation errors in benchmark datasets could undermine the framework by which we validate performance and measure progress in machine learning [25].

What are label errors? Where do they come from? How do we know?
The general machine learning pipeline starts with a dataset (a collection of documents, images, medical records, etc.). When labels are not inherent to the data, they must be annotated – usually by humans. A label error occurs when an annotator provides a label that is “incorrect.” But this raises an interesting question: who gets to decide that some annotation is incorrect? One solution is to ask $k$ annotators and combine their labels somehow (, majority vote, probability distribution).

Subjectivity comes into play here. Given identical instructions and identical items, some annotators may focus on different attributes of the item or have a different interpretation of the labeling criteria. Understanding and modelling label uncertainty remains a compelling challenge in evaluating machine learning systems [45].

Soft errors and subjectivity. Tasks that require free-form, soft, or multi-class annotations present another dimension to this challenge. For example, when providing labels for an object detection task the annotator is typically asked to draw a box around some object (, identify the bicycle in this photo). These annotations are then used to train an object detection model, which will produce labels that are unlikely to exactly match the ground truth annotations. Image processing researchers have therefore developed ways to cope with inexact matches involving geometric assessments of overlap and a threshold that defines “close enough” for precision, recall, and other hard metrics [51].  But the concept of “close enough” in a geometric sense may not match what an annotator or end-user might consider to be “close enough” in a semantic sense.

Example Entity Linking task where the pre-annotated ground truth mention, represented by bounding boxes around text, and link, represented by arrows drawn to Wikipedia entries, is different from the predicted label. Standard evaluation regimes count this as a completely incorrect prediction despite being a reasonable label.

Likewise, natural language processing tasks like named entity recognition (NER) and entity linking (EL) rely heavily on datasets comprised of free-form human annotations. As in object detection, NER and EL tasks are typically evaluated against a held out portion of the already-annotated dataset. Unfortunately, NER and EL tasks produce labels that are not easily verified as “close enough” [36]. Instead, like the example in Fig. <ref>, most NER and EL evaluation metrics require exact matches against free-form annotations [44, 11]. This strict evaluation methodology may unreasonably count labels that are “close enough” as incorrect.

A similar problem is encountered in machine translation and language summarization where a translated and summarized sentences are not easily matched to pre-annotated labels. Instead, these tasks use Direct Assessment (DA) to rank and compare model predictions [13, 12]. But, as machine learning systems advance, ranking equally-valid predictions becomes difficult. Instead, if we simply ask an annotator posthoc to verify whether or not a predicted annotation was correct or useful, then the task becomes easier to annotate while still accomplishing the goal.

Correct versus Verifiable. Producing a verifiable answer is not the same as producing the correct answer. This distinction is critical. Asking a machine learning system to independently provide the same label as an annotator is a wildly different task than asking an annotator to verify the output of a predictor (posthoc verification).

Unfortunately the prevailing test and evaluation regime requires predictors to exactly match noisy, free-form, and subjective human annotations. This paradigm represents a mismatch that, if left unaddressed, threatens to undermine future progress in machine learning.

Main Contributions. In the present work we show that these distinctions matter and that they present significant consequences for how we determine the state-of-the-art in machine learning systems. Specifically, we ask three research questions:

RQ1 When evaluating machine learning systems, how do the results of posthoc verification compare to standard validation against a pre-annotated ground truth? How does this impact validation of the state of the art?

RQ2 How much agreement is there among posthoc annotators? How does it compare to the annotation agreement on the ground truth datasets?

RQ3 What do verification rates say about the ground truth data? Do some datasets perform better or worse than others?

Using a large case study of eight popular EL datasets and two state-of-the-art EL models we are able to report some surprising findings: First, state-of-the-art EL models generally had a higher verification rate than the ground truth datasets. Second, there was substantial disagreement among annotators as to what constitutes a label that is “close enough”. Third, a large proportion (between 10%-70% depending on the dataset) of verified entities were missing from the ground truth dataset.

In the remainder of this paper we will highlight some related perspectives machine learning evaluation and then describe our posthoc verification methodology. Next, each of the research questions will be addressed. Finally, we conclude the paper with a short discussion including our recommendations for future work.

§ POSTHOC VERIFICATION

In a time where hyper-tuned models measure improvement in tenths of a percent, many are beginning to recognize the importance of benchmark datasets. This realization has inspired many to wonder, given the limitations of the benchmarks, if state-of-the-art systems have reached their maximum achievable values [7, 32]. These findings have led to the development of iterative benchmarks that collect adversarial data to fool some models [18]. But others argue that this iterative/adversarial model creates a “counterproductive incentive to be different without being better” [6]. Yet, in the search for better evaluation methodologies, the annotation question has remained the same, , the annotator is asked to provide a label (or labels) for an item (or set of items).

On the contrary, posthoc verification supplies prediction to the annotator, who is then asked to verify it accuracy or not. By supplying a prediction to the annotator, this methodology asks for similar information, but in a critically different way.

In the broader literature on survey methodology and human behavior the difference between pre-annotation and posthoc verification is probably best captured by the default effect. The default effect is the tendency for an agent to generally accept the options that are presented to them [28]. In this frame, the choice of rejecting a pre-defined label is a more deliberate action than inputting a free-form annotation from scratch.

There is much to learn from this distinction. To that end, the present work explores the differences between pre-annotation and posthoc verification methodologies by focusing on the entity linking (EL) task. Despite this focus, we expect that our results generalize to any prediction task that makes use of free-form, subjective, or many-labeled benchmark datasets.

Statistics of the entity linking datasets and annotations.

3cVerified Annotations

GT     E2E     REL     GT     E2E     REL     GT     E2E     REL
AIDA-train             946     18541*     18301     21204     2801     2802     2913     18511     18274     21172

AIDA-A             216     4791     4758     5443     713     715     725     4787     4754     5439

AIDA-B             231     4485     4375     5086     636     646     654     4480     4370     5079

ACE2004           57*     257     1355     1675     114     318     334     256     1352     1672

AQUAINT           50     727     810     925     175     170     179     727     810     925

CLUEWEB           320     11154     12273     23114     3526     3678     4944     11139     12247     23056

MSNBC             20     656     629     756     164     163     171     656     629     756

WIKIPEDIA         345*     6793*     8141     11184     1348     1578     1638     6786     8136     11177

The goal of EL is to identify words or phrases that represent real-world entities and match that phrase to a listing in some knowledge base. Like most classification systems, EL models are typically trained and tested on large pre-annotated benchmark datasets. In the present work we identified eight popular benchmark datasets that are widely used throughout the EL and broader NLP communities. These datasets are listed in Table <ref>. They include three datasets from the AIDA benchmark from a 2003 CoNLL competition [16] and five datasets from the (random) Walking Named Entity Disambiguation (WNED) benchmarks curated by Guo and Barbosa [14]. Each dataset contains many individual documents. This ground truth (GT) is derived from human annotators. Specifically, the AIDA benchmark was annotated by two students (with disagreements resolved by the PI) [16], and the WNED benchmark is annotated through various means. Entries marked with * indicate statistics that are different from those reported in related work [8] because related works filter out documents that do not have any annotations that appear their pre-defined dictionary. We do not perform this filtering.

In order to better understand the effect of pre-annotated benchmarks on machine learning systems, it is necessary to test a handful of state-of-the-art EL systems. Specifically, we chose:

E2E The end-to-end entity linking model, which generates and selects span candidates associated with entity candidates. E2E is a word-level model that utilizes word and entity embeddings to compute span-level context scores. Word and entity embeddings are trained on Wikipedia, and the final model is trained and validated using AIDA-train and AIDA-A respectively [19].

REL The Radboud Entity Linker combines the Flair [2] NER system with the mulrel-nel [20] entity disambiguation system to create a holistic EL pipeline [50]. Mulrel-nel is trained on AIDA-train.

In addition to these popular EL systems, our methodology permits the evaluation of the GT as if it were a competing model. The relative performance of E2E and REL can then compared with the GT to better understand the performance of the annotations.

Pre-Annotation Metrics
Like many NLP tasks, EL evaluates model performance using standard macro and micro averaging of precision, recall, and F1 scores. These metrics typically require that predictions, , the detected mentions and associated entity-links, exactly match the ground truth. Another evaluation methodology relaxes the exact-matching requirement and allows overlapping mentions (like the example from Fig. <ref>) to be counted as correct prediction if they are linked to the same exact entity in the knowledge base (which the example from Fig. <ref> does not) [40].

Recently, the GERBIL tool has been created to give researchers an easy to use API to test EL models against many of these pre-annotated datasets [38]. As we might expect, many users have found that the GERBIL benchmarks primarily feature generic and prominent entities [49].

§.§ Data collection

We have previously argued that these evaluation metrics may not faithfully simulate in vivo performance because (1) the ground truth annotations are noisy and subjective, and (2) exact matching is too strict. We test this argument by collecting posthoc verifications of the three models, including the pre-annotated GT, over the eight datasets.

We created a simple verification system and used Amazon Mechanical Turk to solicit workers. For each document and model, we asked a single worker to verify all present entity annotations (, an entity mention and its linked entity). Large documents of more than 300 characters were broken into smaller tasks so as to not overwhelm the worker. An example task with four annotations is illustrated in Fig. <ref>; where workers are asked to click on each annotation (highlighted in yellow), open up the corresponding Wikipedia page, and select one of three possible actions in the window:

Verify The annotator determines that the current annotation (both mention and Wikipedia link) is appropriate.

Modify The annotator determines that the Wikipedia link is incorrect. In this case, they are asked to search and select a more appropriate Wikipedia link, use it to replace the existing link, and then accept the new annotation.

Remove The annotator determines that the current mention (highlighted text) is not a linkable entity. In this case, they remove the link from the mention.

We made a deliberate decision to not permit new annotation of missing entity mentions. That is, if the model did not label an entity, then there is no opportunity for the worker to add a new label. This design decision kept the worker focused on the verification task, but possibly limits the coverage of the verified dataset. We provide further comments on this decision in the Results section.

Each annotator is assigned to 20 tasks including one control task with three control annotations. We only accept and collect annotations from workers that passed the control task. We paid each worker 3 USD for each HIT. We estimate a average hourly rate of about 9 USD; and paid a total of 6,520 USD. From these, we received 167,432 annotations. The breakdown of tasks, annotations shown to workers, and verified annotations are listed in Table <ref> for each dataset and model.

Prior to launch, this experiment was reviewed and approved by an impaneled ethics review board at the University of Notre Dame. The full list of instructions and link to an example video is included in the Supplementary Material. The source code, raw results, and evaluation scripts are publicly available via the MIT license at <https://github.com/yifding/e2e_EL_evaluate>

Example posthoc verification task. A worker is asked to click on each highlighted entity mention, which displays an entity-link. The worker may then verify, edit, or remove the link.

§ METHODOLOGY AND METRICS

Using the posthoc verification dataset, we can begin to answer the three research questions. But before we directly address the specific questions, we first investigate the collected dataset and its relationship to the typical evaluation regimes. Then, in order to distinguish between traditional, pre-annotated evaluation metrics like precision and recall, we define (or rather redefine) similar metrics for the post-hoc verification scenario before discussing the posthoc verification results.

§.§ The Pre-Annotation Evaluation Regime

First, we re-tested the E2E and REL models and evaluated their micro precision and recall under the typical pre-annotation evaluation regime. These results are illustrated in Fig <ref> (left). These results are nearly identical to those reported by related works [19, 50]. In our opinion, these state-of-the-art models provide meager performance under the standard evaluation regime.

§.§ Posthoc Verification Evaluation

Our next task is to define appropriate evaluation metrics that can be used to compare the results of the posthoc verification experiment with results from the pre-annotation evaluation regime.

Verification Rate. For each combination of dataset and model providing annotations, we compute the verification rate as the percentage of annotations that were verified. Formally, let $d\in\textrm{datasets}$; $m\in\textrm{models}$; and $V_{m,d}$ be the set of verified annotations in a pairing of $d$ and $m$ (a value of a cell in the right three columns of Tab <ref>). Likewise, let $N_{d,m}$ be the pre-annotations of model $m$ on dataset $d$. We therefore define the verification rate of a dataset-model pair as $r_{m,d} = |V_{m,d}|/|N_{d,m}|$. Higher verification rates indicate that the dataset contains annotations and/or the model is capable of providing labels that pass human inspection.

Verification Union. It is important to note that each model and dataset was evaluated by only a single worker. However, we were careful to assign each worker annotations randomly drawn from model/dataset combinations. This randomization largely eliminates biases in favor or against any model or dataset. Furthermore, this methodology provides for repetitions when annotations match exactly across models – which is what models are optimized for in the first place! In this scenario the union of all non-exact, non-overlapping annotations provides a superset of annotations similar to how pooling is used in information retrieval evaluation to create a robust result set [52]. Formally, we define the verification union of a dataset $d$ as $V_d=\bigcup_{m}{V_{m,d}}$.

metric	e2eprec	e2epreclow	e2eprechigh	e2erecall	e2erecalllow	e2erecallhigh	e2ef	relprec	relpreclow	relprechigh	relrecall	relrecalllow	relrecallhigh	relf
AIDA-train	0.9295173471	0.002955018058	0.003064463172	0.9176165523	0.003457403706	0.003241315974	0.9235286122	0.7372000756	0.005148309087	0.00477045154	0.8431743288	0.004267732699	0.004375776565	0.786634075
AIDA-A	0.8737904922	0.008203618006	0.008203618006	0.8677668686	0.007520367662	0.007938165866	0.8707682633	0.7117117117	0.0102960103	0.009744438316	0.8086484228	0.009400459578	0.009400459578	0.7570897712
AIDA-B	0.8123569794	0.009839816934	0.01029748284	0.7924107143	0.01004464286	0.009598214286	0.802259887	0.7115573932	0.01063201418	0.01102579248	0.8066964286	0.009598214286	0.009375	0.7561460404
ACE2004	0.1360946746	0.01553254438	0.01553254438	0.71875	0.046875	0.05078125	0.2288557214	0.1112440191	0.01196172249	0.01255980861	0.7265625	0.046875	0.046875	0.1929460581
AQUAINT	0.3827160494	0.02839506173	0.02839506173	0.4264099037	0.02751031637	0.03026134801	0.4033832141	0.3351351351	0.02486486486	0.02378378378	0.4264099037	0.03026134801	0.03026134801	0.3753026634
CLUEWEB	0.4357801911	0.007593696415	0.007920307014	0.4791273903	0.008079719903	0.007900170572	0.4564269221	0.2749826509	0.004857737682	0.005204718945	0.5691713798	0.007451297244	0.00754107191	0.3708144466
MSNBC	0.7806041335	0.02861685215	0.02861685215	0.7484756098	0.02896341463	0.02743902439	0.7642023346	0.6917989418	0.02645502646	0.03042328042	0.7972560976	0.02591463415	0.02591463415	0.7407932011
WIKIPEDIA	0.3644296952	0.00860373648	0.009341199607	0.4369289714	0.009578544061	0.009578544061	0.3973998124	0.3044645254	0.007157555695	0.00679967791	0.5014736222	0.009578544061	0.01031535514	0.3788899404

metric	e2eprec	e2epreclow	e2eprechigh	e2erecall	e2erecalllow	e2erecallhigh	e2ef	relprec	relpreclow	relprechigh	relrecall	relrecalllow	relrecallhigh	relf	gtprec	gtpreclow	gtprechigh	gtrecall	gtrecalllow	gtrecallhigh	gtf
AIDA-train	0.9832973079	0.002947533897	0.002947533897	0.8414326551	0.008071296452	0.007734992433	0.9068503081	0.9469899666	0.005016722408	0.005016722408	0.9522448293	0.004708256264	0.004371952245	0.9496101283	0.9708999807	0.003854307188	0.004047022548	0.8471498234	0.007398688414	0.007734992433	0.9048132184
AIDA-A	0.9752906977	0.00726744186	0.00726744186	0.8366583541	0.01433915212	0.01496259352	0.9006711409	0.9514195584	0.009463722397	0.008201892744	0.9401496259	0.01059850374	0.009975062344	0.9457510191	0.962962963	0.00871459695	0.007988380537	0.8266832918	0.01620947631	0.01558603491	0.8896343509
AIDA-B	0.9802575107	0.00686695279	0.00686695279	0.7859600826	0.01858224363	0.01789401239	0.872421696	0.9471830986	0.00985915493	0.00985915493	0.9256710255	0.01169993118	0.01101169993	0.9363035155	0.9469453376	0.01045016077	0.009646302251	0.8107364074	0.0165175499	0.01789401239	0.8735632184
ACE2004	0.9728506787	0.01809954751	0.01809954751	0.7875457875	0.04395604396	0.04029304029	0.8704453441	0.9312977099	0.02671755725	0.02290076336	0.8937728938	0.03296703297	0.0293040293	0.9121495327	0.9622641509	0.02830188679	0.02830188679	0.3736263736	0.04395604396	0.04761904762	0.5382585752
AQUAINT	0.9508928571	0.02678571429	0.02232142857	0.6191860465	0.04069767442	0.04651162791	0.75	0.9196787149	0.0281124498	0.0281124498	0.6656976744	0.04069767442	0.04069767442	0.7723440135	0.9312169312	0.03174603175	0.02645502646	0.511627907	0.04360465116	0.04651162791	0.660412758
CLUEWEB	0.9632653061	0.005337519623	0.005337519623	0.5484447622	0.01018948874	0.01108330354	0.6989406538	0.9036770584	0.007194244604	0.006594724221	0.8083661065	0.008759385055	0.008580622095	0.8533685601	0.9258110014	0.007334273625	0.007334273625	0.5867000358	0.01108330354	0.0112620665	0.7182405077
MSNBC	0.966507177	0.02392344498	0.01913875598	0.7709923664	0.03816793893	0.04198473282	0.8577494692	0.9330708661	0.02755905512	0.02362204724	0.9045801527	0.03053435115	0.02671755725	0.9186046512	0.9478672986	0.02843601896	0.02369668246	0.7633587786	0.03816793893	0.04198473282	0.8456659619
WIKIPEDIA	0.9502308876	0.008209338122	0.008209338122	0.6183639399	0.01469115192	0.01469115192	0.7491909385	0.9001972387	0.009861932939	0.009467455621	0.7619365609	0.01268781302	0.01168614357	0.8253164557	0.9049217002	0.01062639821	0.01062639821	0.5402337229	0.01502504174	0.01602671119	0.6765628267

group style=
group name=my plots,
group size=2 by 2,
xlabels at=edge bottom,
ylabels at=edge left,
xticklabels at=edge bottom,
yticklabels at=edge left,
horizontal sep=9pt,
vertical sep=9pt,
/pgf/bar width=3pt,
legend style=font=,
nodes=scale=1, transform shape,
anchor=south east,
draw=none, fill=none,
legend columns=-1,
legend cell align=left,
width = 3.1in,
height = 1.3in,
ymin=0.0, ymax=1,
ylabel = ,
xlabel = Dataset,
symbolic x coords=
title style=yshift=.4cm,font=,
x tick label style=rotate=25,anchor=center,yshift=-0.25cm,xshift=-0.2cm,

[title=Pre-Annotation, ylabel=Precision]
+[error bars/.cd, y dir = both, y explicit] table[x=metric, y=e2eprec, y error minus=e2epreclow, y error plus=e2eprechigh] ;

+[error bars/.cd, y dir = both, y explicit] table[x=metric, y=relprec, y error minus=relpreclow, y error plus=relprechigh] ;

[title=Posthoc Verification, ylabel=]
+[error bars/.cd, y dir = both, y explicit] table[x=metric, y=e2eprec, y error minus=e2epreclow, y error plus=e2eprechigh] ;

+[error bars/.cd, y dir = both, y explicit] table[x=metric, y=relprec, y error minus=relpreclow, y error plus=relprechigh] ;

+[error bars/.cd, y dir = both, y explicit]  table[x=metric, y=gtprec, y error minus=gtpreclow, y error plus=gtprechigh] ;

[title=, ylabel=Recall]
+[error bars/.cd, y dir = both, y explicit]  table[x=metric, y=e2erecall, y error minus=e2erecalllow, y error plus=e2erecallhigh] ;

+[error bars/.cd, y dir = both, y explicit]  table[x=metric, y=relrecall, y error minus=e2erecalllow, y error plus=e2erecallhigh] ;

[title=, ylabel=]
+[error bars/.cd, y dir = both, y explicit]  table[x=metric, y=e2erecall, y error minus=e2erecalllow, y error plus=e2erecallhigh] ;

+[error bars/.cd, y dir = both, y explicit]  table[x=metric, y=relrecall, y error minus=relrecalllow, y error plus=relrecallhigh] ;

+[error bars/.cd, y dir = both, y explicit] table[x=metric, y=gtrecall, y error minus=gtrecalllow, y error plus=gtrecallhigh] ;

Precision and recall results from pre-annotation evaluation compared with the posthoc verification evaluation. Error bars represent 95% confidence intervals on bootstrapped samples of the data. Posthoc verification (on right) returns substantially higher scores than the pre-annotation evalauation. The ground truth (GT) data was often verified at a lower rate than EL models.

Posthoc Precision and Recall.
The precision metric is defined as the ratio of true predictions to all predictions. If we recast the concept of true predictions to be the set of verified annotations $V_{m,d}$, then it is natural to further consider $N_{d,m}$ to be the set of all predictions for some dataset and model pair, especially considering our data collection methodology restricts $V_{m,d}\subseteq N_{d,m}$. Thus the posthoc precision of a model-data pairing is simply the verification rate $r_{m,d}$.

The recall metric is defined as the ratio of true predictions to all true labels. If we keep the recasting of true positives as verified annotations $V_{m,d}$, then all that remains a definition of true labels. Like in most evaluation regimes the set of all true labels is estimated by the available labels in the dataset. Here, we do the same and estimate the set of true labels as the union of a dataset's verified annotations $V_d$. Thus posthoc recall of a model-data pairing is $|V_{m,d}|/|V_d|$.

§ POSTHOC VERIFICATION RESULTS

Using the evaluation tools introduced in the previous section, we begin to answer our original research questions. We begin with RQ1, which asks if the differences between evaluation regimes, , pre-annotation versus posthoc verification, have any affect on our perception of model performance. To answer RQ1, we compared the precision and recall metrics calculated using the pre-annotation evaluation regime against the precision and recall metrics calculated using the posthoc verification regime.

Figure <ref> compares model performance under the different evaluation regimes. Error bars represent the empirical 95% confidence internals drawn from 1000 bootstrap samples of the data. We draw two major conclusions from this comparison: (1) Pre-annotation evaluation performance is much lower performance than Posthoc verification, and (2) the EL models usually outperform the ground truth, sometimes substantially.

Pre-annotation evaluation performance is lower than Posthoc verification. The differences between the scores of the pre-annotation compared to posthoc verification are striking. Posthoc annotation shows nearly perfect precision scores across all datasets. Although the models may not exactly predict the pre-annotated label, high posthoc precision indicates that their results appear to be “close-enough” to pass a human's verification.

In other words, the exact matching against pre-annotations is too strict. Despite its intention, the pre-annotation evaluation regime does not appear to faithfully simulate a human use case.

Detailed error analysis of verification rates in Fig. <ref>(top right). The E2E model consistently outperforms the ground truth dataset.

Machine Learning models outperform the Ground Truth.
The posthoc verification methodology permits the ground truth annotations to be treated like any other model. So the right column in Fig. <ref> includes the verification precision and recall for GT. The results were unexpected. We found that labels produced by the EL models oftentimes had a higher verification rate than the pre-annotated ground truth. The recall metric also showed that the EL models were also able to identify more verified labels than GT.

What does this mean?  Higher precision performance of the EL models indicates that human annotators make more unverifiable annotations than the EL models. Even the precision of AIDA_train, which is trained and tested on GT, has a higher verification rate than the GT (although not by much). Higher recall performance of the EL models indicates that the EL models find a greater coverage of possible entities. The recall results are less surprising because human annotators may be unmotivated or inattentive during free-form annotation – qualities that tend to not affect EL models.

Error Analysis of the Ground Truth. For each linked entity, the posthoc verification methodology permitted one of three outcomes: verification, modification, or removal. Figure <ref> shows the percentage of each outcome for each model and dataset pair; it is essentially a zoomed-in, more-detailed illustration of the top-right panel in Figure <ref>, but with colors representing outcomes and patterns representing models.

Edits indicate that the named entity recognition (, mention detection) portion of the EL model was able to identify an entity, but the entity was not linked to an verifiable entity. The available dataset has an enumeration of corrected linkages, but we do not consider them further in the present work. Removal indicates an error with the mention detection. From these results we find that, when a entity mention is detected it is usually a good detection; the majority of the error comes from the linking/disambiguation subtask.

A similar error analysis of missing entities is not permitted from the data collection methodology because we only ask workers to verify pre-annotated or predicted entities, not add missing entities. Because all detected mentions are provided with some entity link, we can safely assume that missing entities is mostly (perhaps wholly) due to errors in the mention detection portion of EL models.

model	v3   v2   v1 v0
AIDA-train	0.8245187296	0.1385780805	0.03315032316	0.003752866773
AIDA-A	 0.826458037	0.1376955903	0.03214793741	0.003698435277
AIDA-B	 0.8450244698	0.1275693312	0.02349102773	0.003915171289
ACE2004     0.9440559441	0.04895104895	0.006993006993	0
AQUAINT     0.9017857143	0.08482142857	0.01339285714	0
CLUEWEB     0.8539109507	0.1135980746	0.02719614922	0.005294825511
MSNBC     0.8564356436	0.1113861386	0.01732673267	0.01485148515
WIKIPEDIA   0.8791064389	0.1024967148	0.01445466491	0.00394218134

ybar stacked,
bar width=6pt,
legend style=font=,
nodes=scale=1, transform shape,

anchor=south east,
draw=none, fill=none,
legend columns=-1,
legend cell align=left,
width = .5,
height = 1.7in,
ymin=0.75, ymax=1,
ylabel = Percent Agreement,
xlabel = ,
symbolic x coords=
x tick label style=rotate=25,anchor=center, xshift=-0.4cm, yshift=-0.2cm,

+[] table[x=model, y=v3] ;

+[] table[x=model,y=v2] ;

+[] table[x=model,y=v1] ;

+[] table[x=model,y=v0] ;

Agreement rates for three posthoc verification workers across an intersection of the results of all three models (E2E, REL, GT). $i=3$ indicates that all three workers verified the label, $i=2$ indicates that only two of the three verified the label, etc.

§.§ Posthoc Verification Agreement

Our second research question asks for an analysis of the agreement among the posthoc annotators. Just as agreement is assessed in pre-annotated ground truth data collection so as to guard against spurious or noisy data, so too is agreement useful in posthoc analysis. To answer RQ2, we first compute the intersection of all models for each dataset $I_d = \bigcap_{m}{N_{d,m}}$, that is, we only consider labels that were found in all three models. This severely restricted the analysis set in some datasets; in the most restrictive case, $I_{\scriptsize\textrm{ace2004}} = 246$ but its union contained 848. Nevertheless, from this reduced set we collected the labels $V^\prime_d = V_d\cap I_d$ that were verified by $i$ annotators.

Figure <ref> shows that most annotations had complete agreement, , all three annotators verified the same label for each of the three models. When one annotation was unverified, it still remained likely that the other annotators would still verify it. In less than 5% of instances did two or more annotators decline to verify the label. These agreement rates are drastically higher than pre-annotation agreement rates, which had 16% and 26% agreements between three and two annotators respectively [4].

We also find that the ACE2004 and AQUAINT datasets appear to have a higher agreement rate than the other datasets. This begs the question: are the ACE2004 and AQUAINT datasets better benchmarks? Or is this simply a result of these low-recall datasets restricting the intersection to only the highest-quality (or easiest) annotations? This leads us to our final research question.

§.§ Coverage of Verified Annotations in Ground Truth Datasets

We have previously seen that the pre-annotated ground truth datasets are slightly worse than some EL models. But what can these experiments tell us about RQ3? Can we evaluate the performance of the ground truth datasets relative to each other? Do some datasets contain more or fewer verifiable annotations?

metric	type_a type_b   type_c  type_d  type_e
AIDA-train	0.6811304938	0.2176360225	0.004630553671	0.002714462497	0.09388846753
AIDA-A	0.6523761081	0.2345589304	0.005377125418	0.004795814562	0.1028920215
AIDA-B	0.6461660079	0.2319367589	0.005059288538	0.006640316206	0.1101976285
ACE2004	0.2818396226	0.04245283019	0.01061320755	0.008254716981	0.6568396226
AQUAINT	0.4558599696	0.06925418569	0.02891933029	0.03729071537	0.4086757991
CLUEWEB	0.4730525736	0.1868978805	0.009193503991	0.01502890173	0.3158271401
MSNBC	0.6298342541	0.1668508287	0.02099447514	0.04751381215	0.1348066298
WIKIPEDIA	0.4283901784	0.1580714979	0.01888200612	0.04903350996	0.3456228077

xbar stacked,
/pgf/bar width=5pt,
legend style=font=,
nodes=scale=1.0, transform shape,

anchor=south east,
draw=none, fill=none,
legend columns=-1,
legend cell align=left,
width = .9,
height = 1.8in,
xmin=0.0, xmax=1,
xlabel = Outcome,
ylabel = Dataset,
symbolic y coords=

+[] table[y=metric, x=type_a] ;

+[] table[y=metric, x=type_b] ;

+[] table[y=metric, x=type_c] ;

+[] table[y=metric, x=type_d] ;

+[] table[y=metric, x=type_e] ;

Outcomes from comparing union of verified annotations to GT annotations: (A) verified mention matches GT, verified link matches GT; (B) mention matches GT, link does not match GT; (C) mention overlaps another mention in GT, link matches GT; (D) mention overlaps GT, link does not match GT; (E) mention does not overlap with any mention in GT

We answer this question by again considering, for each dataset, the union of verified annotations across all models $V_d$. Then, for each document in the dataset, we compare the annotations from $V_d$ to GT. For each annotation in $V_d$ this comparison can have five outcomes described in Fig. <ref>. Recall from Fig. <ref>(top right) that many datasets had verification rates around 90%. The ACE2004 dataset performed best with a verification rate of 96.4%. But, we can observe from Fig. <ref> that this same dataset is missing nearly 65% of verified annotations. Missing annotations (E) are generally the largest source of error in the ground truth, followed by mismatched–but still verified–entity links.

§ DISCUSSION AND RECOMMENDATIONS

The primary goal of the present work is to compare pre-annotation labels contributed by human workers against verified annotations of the same data. Using entity linking as an example task, we ultimately found that these two methodologies returned vastly different performance results. From this observation we can draw several important conclusions. First, EL models have a much higher precision than related work reports. This difference is because the standard evaluation methodology used in EL, and throughout ML generally, do not account for soft matches or the semantics of what constitutes a label that is “close enough”. Our second conclusion is that EL models, and perhaps ML models generally, sometimes perform better than ground truth annotators – at least, that is, according to other ground truth annotators.

The present work is not without its limitations. Here we only considered the entity linking task, which is only a small piece of ML research. We expect our results to generalize to any methodology that performs validation tests on subjective pre-annotation ground truth datasets, such as image labelling,

social media analysis, text classification, and others. We also recognize that we used one set of annotators to argue against the work of, basically, the same set of annotators, , turkers. Yet, our analysis is not meant to compare agreement among different sets of workers, but rather to show that the way in which we ask workers to perform their work is critical to the answer we receive. A final, rather obvious limitation of our methodology is cost. It costs money to ask workers to validate results. But in an era where millions of dollars are spent on training models, it seems reasonable to spend a few thousand on a more-thorough evaluation.

Based on the results of this paper, we recommend that ML researchers and practitioners reconsider the use of holdout test datasets based on pre-annotated data. In a time when improvement in machine learning models are measured by tenths of a percent, it is critical to remind ourselves that benchmark test sets, especially those comprised of pre-annotated labels are fallible – and oftentimes less verifiable than the results of the models themselves. Our second recommendation is to reconsider the strict matching metrics for multiclass classification tasks like entity disambiguation and entity linking. We encourage the community to develop soft evaluation metrics like the BLEU score used in machine translation [27] and the ROUGE score used in text summarization [22].

Broader Impacts and Ethical Considerations.
The present work is a call to re-evaluate the way that accuracy is evaluated in certain machine learning tasks. Implications of this work may redefine how researchers validate the state-of-the-art. We do not expect any negative broader impacts from this re-consideration of how machine learning models are evaluated. An impaneled ethical review board approved this experiment before data collection began. Human subjects were employed via Amazon Mechanical Turk to perform verification tasks. Tasks contained only pre-annotated data from existing datasets. Workers were provided an informed consent form, video, audio, and written instructions, as well as an example training task; no personal identifiable information (PII) was collected.

[1]
R. Aharoni and Y. Goldberg.
Split and rephrase: Better evaluation and stronger baselines.
In ACL 2018, pages 719–724, 2018.

[2]
A. Akbik, D. Blythe, and R. Vollgraf.
Contextual string embeddings for sequence labeling.
In COLING 2018, pages 1638–1649, 2018.

[3]
Y. Belinkov and Y. Bisk.
Synthetic and natural noise both break neural machine translation.
In ICLR 2018, 2018.

[4]
N. Botzer, Y. Ding, and T. Weninger.
Reddit entity linking dataset.
Information Processing & Management, 58(3):102479, 2021.

[5]
S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning.
A large annotated corpus for learning natural language inference.
arXiv preprint arXiv:1508.05326, 2015.

[6]
S. R. Bowman and G. E. Dahl.
What will it take to fix benchmarking in natural language
arXiv preprint arXiv:2104.02145, 2021.

[7]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.
Bert: Pre-training of deep bidirectional transformers for language
In NAACL HLT 2019, pages 4171–4186, 2019.

[8]
O.-E. Ganea and T. Hofmann.
Deep joint entity disambiguation with local neural attention.
In EMNLP 2017, pages 2619–2629, 2017.

[9]
M. Geva, Y. Goldberg, and J. Berant.
Are we modeling the task or the annotator? an investigation of
annotator bias in natural language understanding datasets.
In EMNLP, pages 1161–1166, 2019.

[10]
M. Glockner, V. Shwartz, and Y. Goldberg.
Breaking nli systems with sentences that require simple lexical
In ACL 2018, pages 650–655, 2018.

[11]
K. Goel, N. Rajani, J. Vig, S. Tan, J. Wu, S. Zheng, C. Xiong, M. Bansal, and
C. Ré.
Robustness gym: Unifying the nlp evaluation landscape.
arXiv preprint arXiv:2101.04840, 2021.

[12]
Y. Graham, T. Baldwin, A. Moffat, and J. Zobel.
Continuous measurement scales in human evaluation of machine
In Proceedings of the 7th Linguistic Annotation Workshop and
Interoperability with Discourse, pages 33–41, 2013.

[13]
Y. Graham, T. Baldwin, A. Moffat, and J. Zobel.
Is machine translation getting better over time?
In Proceedings of the 14th Conference of the European Chapter of
the Association for Computational Linguistics, pages 443–451, 2014.

[14]
Z. Guo and D. Barbosa.
Robust entity linking via random walks.
In CIKM 2014, pages 499–508, 2014.

[15]
S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. R. Bowman, and N. A.
Annotation artifacts in natural language inference data.
arXiv preprint arXiv:1803.02324, 2018.

[16]
J. Hoffart, M. A. Yosef, I. Bordino, H. Fürstenau, M. Pinkal, M. Spaniol,
B. Taneva, S. Thater, and G. Weikum.
Robust disambiguation of named entities in text.
In Proceedings of the 2011 Conference on Empirical Methods in
Natural Language Processing, pages 782–792, 2011.

[17]
N. Hynes, D. Sculley, and M. Terry.
The data linter: Lightweight, automated sanity checking for ml data
In NIPS MLSys Workshop, 2017.

[18]
D. Kiela, M. Bartolo, Y. Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen,
G. Prasad, A. Singh, P. Ringshia, et al.
Dynabench: Rethinking benchmarking in nlp.
arXiv preprint arXiv:2104.14337, 2021.

[19]
N. Kolitsas, O.-E. Ganea, and T. Hofmann.
End-to-end neural entity linking.
CoNLL 2018, 2018.

[20]
P. Le and I. Titov.
Improving entity linking by modeling latent relations between
In ACL 2018, pages 1595–1604, 2018.

[21]
O. Levy, Y. Goldberg, and I. Dagan.
Improving distributional similarity with lessons learned from word
Transactions of the Association for Computational Linguistics,
3:211–225, 2015.

[22]
C.-Y. Lin.
Rouge: A package for automatic evaluation of summaries.
In Text summarization branches out, pages 74–81, 2004.

[23]
S. M. Lundberg and S.-I. Lee.
A unified approach to interpreting model predictions.
NeurIPS 2017, 30:4765–4774, 2017.

[24]
A. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts.
Learning word vectors for sentiment analysis.
In ACL, pages 142–150, 2011.

[25]
C. G. Northcutt, A. Athalye, and J. Mueller.
Pervasive label errors in test sets destabilize machine learning
arXiv preprint arXiv:2103.14749, 2021.

[26]
Y. Oortwijn, T. Ossenkoppele, and A. Betti.
Interrater disagreement resolution: A systematic procedure to reach
consensus in annotation tasks.
In Proceedings of the Workshop on Human Evaluation of NLP
Systems (HumEval), pages 131–141, 2021.

[27]
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu.
Bleu: a method for automatic evaluation of machine translation.
In ACL 2002, pages 311–318, 2002.

[28]
C. W. Park, S. Y. Jun, and D. J. MacInnis.
Choosing what i want versus rejecting what i do not want: An
application of decision framing to product option choice decisions.
Journal of Marketing Research, 37(2):187–202, 2000.

[29]
S. Paun, B. Carpenter, J. Chamberlain, D. Hovy, U. Kruschwitz, and M. Poesio.
Comparing bayesian models of annotation.
Transactions of the Association for Computational Linguistics,
6:571–585, 2018.

[30]
A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, and B. Van Durme.
Hypothesis only baselines in natural language inference.
NAACL HLT 2018, page 180, 2018.

[31]
V. Prabhakaran, B. Hutchinson, and M. Mitchell.
Perturbation sensitivity analysis to detect unintended model biases.
In EMNLP, pages 5744–5749, 2019.

[32]
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever.
Language models are unsupervised multitask learners.
OpenAI Blog, 1(8):9, 2019.

[33]
P. Rajpurkar, R. Jia, and P. Liang.
Know what you don’t know: Unanswerable questions for squad.
In ACL 2018, pages 784–789, 2018.

[34]
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang.
Squad: 100,000+ questions for machine comprehension of text.
arXiv preprint arXiv:1606.05250, 2016.

[35]
M. T. Ribeiro, C. Guestrin, and S. Singh.
Are red roses red? evaluating consistency of question-answering
In ACL, pages 6174–6184, 2019.

[36]
M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh.
Beyond accuracy: Behavioral testing of nlp models with checklist.
In ACL 2020, pages 4902–4912, 2020.

[37]
M. Richardson, C. J. Burges, and E. Renshaw.
Mctest: A challenge dataset for the open-domain machine comprehension
of text.
In EMNLP 2013, pages 193–203, 2013.

[38]
M. Röder, R. Usbeck, and A.-C. Ngonga Ngomo.
Gerbil–benchmarking named entity recognition and linking
Semantic Web, 9(5):605–625, 2018.

[39]
D. Rolnick, A. Veit, S. Belongie, and N. Shavit.
Deep learning is robust to massive label noise.
arXiv preprint arXiv:1705.10694, 2017.

[40]
H. Rosales-Méndez, A. Hogan, and B. Poblete.
Fine-grained evaluation for entity linking.
In EMNLP-IJCNLP 2019, pages 718–727, 2019.

[41]
H. Rosales-Méndez, A. Hogan, and B. Poblete.
Nifify: Towards better quality entity linking datasets.
In WWW 2019, pages 815–818, 2019.

[42]
N. Sambasivan, S. Kapania, H. Highfill, D. Akrong, P. Paritosh, and L. M.
“everyone wants to do the model work, not the data work”: Data
cascades in high-stakes ai.
In CHI 2021, pages 1–15, 2021.

[43]
R. Schwartz, M. Sap, I. Konstas, L. Zilles, Y. Choi, and N. A. Smith.
Story cloze task: Uw nlp system.
In Proceedings of the 2nd Workshop on Linking Models of Lexical,
Sentential and Discourse-level Semantics, pages 52–55, 2017.

[44]
O. Sevgili, A. Shelmanov, M. Arkhipov, A. Panchenko, and C. Biemann.
Neural entity linking: A survey of models based on deep learning.
arXiv preprint arXiv:2006.00575, 2020.

[45]
P. Sommerauer, A. Fokkens, and P. Vossen.
Would you describe a leopard as yellow? evaluating crowd-annotations
with justified and informative disagreement.
In COLING 2020, pages 4798–4809, 2020.

[46]
H. Song, M. Kim, D. Park, Y. Shin, and J.-G. Lee.
Learning from noisy labels with deep neural networks: A survey.
arXiv preprint arXiv:2007.08199, 2020.

[47]
A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and
K. Suleman.
Newsqa: A machine comprehension dataset.
arXiv preprint arXiv:1611.09830, 2016.

[48]
M. Tsuchiya.
Performance impact caused by hidden bias of training data for
recognizing textual entailment.
In LREC 2018, 2018.

[49]
M. Van Erp, P. Mendes, H. Paulheim, F. Ilievski, J. Plu, G. Rizzo, and
J. Waitelonis.
Evaluating entity linking: An analysis of current benchmark datasets
and a roadmap for doing a better job.
In LREC 2016, pages 4373–4379, 2016.

[50]
J. M. van Hulst, F. Hasibi, K. Dercksen, K. Balog, and A. P. de Vries.
Rel: An entity linker standing on the shoulders of giants.
In SIGIR 2020, pages 2197–2200, 2020.

[51]
Z. Wang, E. Wang, and Y. Zhu.
Image segmentation evaluation: a survey of methods.
Artificial Intelligence Review, 53(8):5637–5674, 2020.

[52]
J. Zobel.
How reliable are the results of large-scale information retrieval
In SIGIR 1998, pages 307–314, 1998.

§ CHECKLIST

* For all authors...

* Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

* Did you describe the limitations of your work?
See Section <ref>

* Did you discuss any potential negative societal impacts of your work?
See Section <ref>

* Have you read the ethics review guidelines and ensured that your paper conforms to them?
This work conforms to the ethics review guidelines

* If you are including theoretical results...

* Did you state the full set of assumptions of all theoretical results?

* Did you include complete proofs of all theoretical results?

* If you ran experiments...

* Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
We provide the URL with code, data and script to get the experiment result.

* Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
We provide the instructions, code, and data in the URL. Each experiment result (tables and figures) has detailed explanation and instruction.

* Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
We report error bars.

* Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Our models were pretrained, no significant or advanced machinery was required to perform these experiments

* If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

* If your work uses existing assets, did you cite the creators?
We use two existing entity linking models, and eight existing datasets. We describe them and cite the their papers.

* Did you mention the license of the assets?
We use the MIT license. See section <ref>

* Did you include any new assets either in the supplemental material or as a URL?
We link to an anonymized code repository containing all code, data, and evaluation scripts. See section <ref>

* Did you discuss whether and how consent was obtained from people whose data you're using/curating?
Informed consent was obtained from workers. See Section <ref>

* Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Yes, see Section <ref>

* If you used crowdsourcing or conducted research with human subjects...

* Did you include the full text of instructions given to participants and screenshots, if applicable?
We include the informed consent form, instruction text, and instruction video in the supplementary material. Task instructions and an example of our task are shown in Fig <ref>

* Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Informed consent discussed potential risks. See Section<ref>

* Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
See Section <ref>