# EDIN: An End-to-end Benchmark and Pipeline for Unknown Entity Discovery and
Indexing

Nora Kassner1,2, Fabio Petroni1, Mikhail Plekhanov1, Sebastian Riedel1, Nicola
Cancedda1
1Meta AI
2Center for Information and Language Processing, LMU Munich
<EMAIL_ADDRESS>
###### Abstract

Existing work on Entity Linking mostly assumes that the reference knowledge
base is complete, and therefore all mentions can be linked. In practice this
is hardly ever the case, as knowledge bases are incomplete and because novel
concepts arise constantly. This paper created the _Unknown Entity Discovery
and Indexing_ (EDIN) benchmark where unknown entities, that is entities
without a description in the knowledge base and labeled mentions, have to be
integrated into an existing entity linking system. By contrasting EDIN with
zero-shot entity linking, we provide insight on the additional challenges it
poses. Building on dense-retrieval based entity linking, we introduce the end-
to-end EDIN pipeline that detects, clusters, and indexes mentions of unknown
entities in context. Experiments show that indexing a single embedding per
entity unifying the information of multiple mentions works better than
indexing mentions independently.

##  1 Introduction

Entity linking (EL), the task of detecting mentions of entities in context and
disambiguating them to a reference knowledge base (KB), is an essential task
for text understanding. Existing works on EL mostly assume that the reference
KB is complete, and therefore all mentions can be linked. In practice this is
hardly ever the case, as knowledge bases are incomplete and because novel
concepts arise constantly. For example, English Wikipedia, which is often used
as the reference KB for large scale linking, is growing by more than 17k
entities every
month.111https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia
(09.05.2022)

We created the EDIN benchmark and pipeline222Model and benchmark are currently
not publicly available. where _unknown entities_ , that is entities with no
available canonical names, descriptions and labeled mentions, have to be
integrated into an existing EL model in an end-to-end fashion.

The benchmark is based on Wikipedia and a subset of news pages of the common-
crawl dataset OSCAR Abadji et al. (2021), split into two parts, one preceding
time $t_{1}$ and one between $t_{1}$ and $t_{2}$. With current approaches, an
EL system created at $t_{1}$ is unable to successfully link unknown entities
that were added to Wikipedia between $t_{1}$ and a subsequent time $t_{2}$, as
it lacks the ability to represent them as part of the entity index. This sets
this task apart from zero-shot entity linking Logeswaran et al. (2019). In the
zero-shot (zs) setting, a textual description of the zs entities is assumed
available at the time of training. This allows the creation of entity
embeddings for them, their insertion in a dense entity index, and their use
during training as negative examples. In contrast, no such textual description
is initially available for unknown entities.

To adapt the model trained at $t_{1}$, the model can only make use of an
_adaptation dataset_ and unsupervised techniques. There are two parts to this
task: i) Discovery: The entity linking system needs to detect mentions of
unknown entities part of the adaptation dataset and classify them as unknown
and ii) Indexing: co-referring mentions of unknown entities need to be mapped
to a single embedding compatible with the entity index. The EDIN-pipeline
developed for this task is the first to adapt a dense-retrieval based EL
system created at $t_{1}$ such that it incorporates unknown entities in an
end-to-end fashion.

We show that distinguishing known from unknown entities, arguably a key
feature of an intelligent system, poses a major challenge to dense-retrieval
based EL systems. To successfully do so, a model has to strike the right
balance between relying on mention vs. context. On one hand, the model needs
to distinguish unknown entities carrying the same name as known entities and
co-refer different mentions of the same unknown entities. In both cases,
context reliance is necessary. E.g, the model needs to distinguish the 2019
novel ‘Underland’ from multiple other novels and fictional references carrying
the same name. On the other hand, the model needs to distinguish unknown
entities with new names but semantic similarity. Here, mention reliance is
important. E.g., the model needs to distinguishing BioNTech from other
biotechnology companies with similar names and contexts.

Another challenge in discovery is class imbalance between known and unknown
entities. With Wikipedia-size KBs, there tend to be many fewer mentions of
unknown entities than of known entities and an adaptation dataset of finite
size limits recall. In this work, we optimize for recall on the cost of
precision.

On the side of indexing, inserting unknown entities into a space of known
entities poses problems of interference with known entities in their close
proximity. Again, consider the BioNTech example from above. Semantically, we
want to place BioNTech in proximity of other biotech companies but in a way
that dense-retrieval linking can still differentiate between them. We find
that re-training the model after indexing, and therefore giving it the chance
to learn from hard negatives in the style of zs entity linking, is crucial to
overcome this challenge.

We experiment with different indexing methods. In particular, we contrast
single mention-level indexing FitzGerald et al. (2021) with indexing clusters
of mentions. We find that unifying the information of multiple mentions into a
single embedding is beneficial.

By introducing a clear-cut temporal segmentation this benchmark targets
unknown entities which are truly novel/unseen to all parts of an EL system,
specifically including the pre-trained language model (PLM). Therefore, the EL
system cannot rely on implicit knowledge captured by the PLM. This is, to the
best of our knowledge, a setting that has not been explored before in the
context of dense-retrieval based EL.

Temporal segmentation also lets us study the effect of entity encoder and PLM
degradation. We note that EL precision drops for known entities in novel
contexts which points to a large problem of PLM staleness also discussed by
Dhingra et al. (2022); Lazaridou et al. (2021). Novel entities appearing in
the context around known entities also complicate discovery, making it likely
that the known entities will be mistakenly classified as unknown.

We summarize our contributions as follows: i) We created the EDIN-benchmark, a
large scale end-to-end entity linking benchmark dataset where unknown entities
need to be discovered and integrated into an existing entity index. ii) We
contrast this task with zero-shot entity linking, and provide insight on the
additional challenges it poses. iii) We propose the EDIN-pipeline in the form
of an extension of existing dense-retrieval architectures. vi) We experiment
with different indexing methods, specifically indexing single mentions vs.
clusters of mentions.

##  2 Task definition

We formally define end-to-end EL as follows: Given a paragraph $p$ and a set
of known entities $E_{K}=\\{e_{i}\\}$ from Wikipedia, each with canonical
name, the title, $t(e_{i})$ and textual description $d(e_{i})$, our goal is to
output a list of tuples, $(e,[i,j])$, where $e\in E_{K}$ is the entity
corresponding to the mention $m_{i,j}$ spanning from the $i^{th}$ to $j^{th}$
token in $p$. We call a system that solves this task based on $d(e_{i})$ a
Description-based entity linking system L.

For EDIN-benchmark, after training a model $L_{t1}$ at time step $t_{1}$, a
set of unknown entities $E_{U}=\\{e_{i}\\}$ with $E_{U}\bigcap
E_{K}=\emptyset$ and no available canonical name, description and labeled
mentions is introduced between $t_{1}$ and $t_{2}$, with $t_{2}>t_{1}$. The
task is to adapt $L_{t1}$ such that it can successfully link mentions of the
union of $E_{U}\bigcup E_{K}$.

We use three dataset splits: the training set $D_{train}$ to train $L_{t1}$,
the adaptation dataset $D_{adapt}$ used to adapt $L_{t1}$ and the testset
$D_{test}$ to evaluate. Both $D_{adapt}$ and $D_{test}$ include mentions
between $t_{1}$ and $t_{2}$ but are disjoint, e.g., $D_{adapt}\bigcap
D_{test}=\emptyset$.

Figure 1: EDIN-pipeline: In the adaptation phase, detected mentions in
$D_{adapt}$ are mapped into a joint dense space with $E_{K}$ representations.
A clustering algorithm groups mentions and entities based on kNN-similarity.
Clusters of mentions without entity encoding are classified as $E_{U}$. To
integrate these into the index of $E_{K}$, mentions in single sentence
contexts are concatenated and mapped to a single embedding using the entity
encoder. After adaptation, the updated entity index is used for standard EL.

##  3 Model

Our UEDI-pipeline is built on top of Anonymous (2022), an end-to-end extension
of the dense-retrieval based model BLINK Ledell Wu (2020). It is composed of a
Mention Detection (MD), Entity Disambiguation (ED) and Rejection (R) head. MD
detects entity mention spans $[i,j]$ in context relying on BERT Devlin et al.
(2019). ED links these mentions to $e\in E_{K}$. It relies on bi-encoder
architecture running a k-nearest-neighbor (kNN) search between mention
encoding and candidate entity encodings (the entity index). Mention encodings
are pooled from BERT-encoded paragraph tokens $p_{1..n}$:

$\displaystyle\textbf{m}_{i,j}=FFL(BERT([CLS]p_{1}\ldots p_{n}[SEP])_{i...j})$

Entities are represented using BLINK’s frozen entity encoder:

$\displaystyle\textbf{e}=BERT_{[CLS]}([CLS]t(e)[SEP]d(e)[SEP])$

Mention-entity candidates are passed to R that controls precision-recall
trade-off by thresholding a learned candidate score.

More information about architecture and training are detailed in Appendix A.

##  4 Unknown Entity Discovery and Indexing

For $E_{U}$ without descriptions, canonical names and labeled mentions,
$L_{t1}$ is unable to successfully link mentions of the union of entities
$E_{U}\bigcup E_{K}$.

To this end, we introduce an end-to-end pipeline to encode $E_{U}$ into
$L_{t1}$’s entity index. The pipeline is depicted in Figure 1. This pipeline
is fully unsupervised and only relies on the adaptation dataset $D_{adapt}$.
It follows a two-step process: i) Discovery: The entity linking system needs
to detect mentions of unknown entities and classify them as unknown and ii)
Indexing: co-referring mentions of unknown entities need to be mapped to a
single embedding compatible with the entity index.

###  4.1 Unknown Entity Discovery

First, $L_{t1}$ detects and encodes mentions part of $D_{adapt}$. The MD head
is trained to detect mentions leveraging the context around them, and can
therefore detect mentions of both $E_{K}$ and $E_{U}$. Encoded mentions
$\textbf{M}=\\{\textbf{m}_{1},...,\textbf{m}_{|M|}\\}$ are then input to a
clustering algorithm that partitions $M$ into disjoint clusters
$C=\\{c_{1},...,c_{|C|}\\}$. We use Logan IV et al. (2021)’s greedy NN
clustering algorithm where $\textbf{m}_{i}$ is clustered with all
$\textbf{m}_{j}\in\textbf{M}$ such that
$sim(\textbf{m}_{i},\textbf{m}_{j})>\delta$.

Next, entity encodings of $e\in E_{K}$ are assigned to these clusters if
$sim(\textbf{e}_{i},\textbf{m}_{j})>\tau$ and $\textbf{e}_{i}$ is the nearest
entity of any $m_{i}\in c_{i}$. As we find this approach to result in low
precision, we add another condition, namely that
$sim(\textbf{e}_{i},\textbf{m}_{j})>\tau$ must hold for at least 70% of all
$\textbf{m}_{j}\in c_{i}$. $\delta,\tau$ are tuned on $D_{adapt}$-dev. For
more details see Appendix C. Following Agarwal et al. (2021), all clusters not
containing any entity representation are deemed to refer to entities in
$E_{U}$. We refer to this subset of automatically identified unknown entities
as $E^{\prime}_{U}$.

###  4.2 Unknown Entity Indexing

Next, clusters identified as $E^{\prime}_{U}$ are integrated into the EL index
of $L_{t1}$. We explore two different methods of entity indexing:

  * •

Cluster-based indexing: We concatenate all mentions part of the same cluster,
each with the sentence they occur in, and use the entity encoder to map to a
single entity representation. We pool over all $m_{i}\in c_{i}$ and select the
most occurring mention as $t(e)$.

  * •

Mention-based indexing: Mentions in single sentence contexts are indexed
individually using the entity encoder. Individual mentions are used as $t(e)$.

##  5 Evaluation

For _Discovery_ , we report precision and recall of $E_{U}$ classification and
clustering metrics.

To compute standard EL metrics, e.g., precision and recall, canonical names of
indexed clusters need to be consistent with the set of test labels. Our method
of assigning canonical names to clusters based on mentions is not. To resolve
this mismatch we pool on mention labels instead of the mentions itself.

Unsupervised clustering of mentions in $D_{adapt}$ naturally comes with error:
i) Clusters can be incomplete, e.g., mentions of a single entity can be split
into multiple clusters which can lead to indexing the same entity multiple
times and ii) Clusters can be impure, e.g., mentions of different entities end
in the same cluster which leads to conflation of multiple entities into one
representation.

In our evaluation we make use of the gold labels for computing standard EL
metrics by associating possibly more than one cluster to each unknown test
entity, and considering a prediction correct if a test mention is linked to
any of the clusters associated with the correct entity. Since, in a practical
setting, gold labels are not available though, EL metrics could fail to
capture shortcomings in establishing co-references between mentions.
Therefore, we report clustering metrics alongside EL metrics. We follow
Agarwal et al. (2021) and report normalized mutual information (NMI).

##  6 Datasets

| Wikipedia | OSCAR
---|---|---
Train | 100k (908k) | 100k (1.7M)
Adapt | 17k (183k) | 17k (360k)
Dev Train | 8k (78k) | 8k (142k)
Dev Adapt | - | 9k (183k)
Test | 198k (1.8M) | 569k (11M)
Table 1: Dataset Statistics:  Number of samples (number of mentions) for training, adaptation and testing. Bin | Support |  $E_{K}$ R@1 | Support |  $E_{U}$ R@1
---|---|---|---|---
[0) | 68,241 | 21.1 | 7,095 | 17.5
[1) | 59,227 | 29.1 | 3,923 | 25.9
[1, 10) | 313,232 | 45.6 | 9,939 | 40.7
[10, 100) | 901,857 | 65.7 | 7,765 | 57.3
[100, 1k) | 2,860,880 | 76.9 | 7,399 | 64.4
[1k, +) | 5,981,028 | 84.4 | 6,717 | 86.7
Table 2: Frequency effects: End-to-end EL performance of upper baseline model
$L_{t2}$ per frequency bins. Figure 2: Dataset splits:  A schema illustrating
the composition of $D_{t1}$ and $D_{t2}$. Note, that contrary to what this
plot suggests, the number of samples per datasplit is equal for $D_{t1}$ and
$D_{t2}$.

To construct the reference entity index, we download Wikipedia dumps from $t1$
and $t2$ and extract entity titles and descriptions. Setting $t_{1}$ to
September 2019 (the date when BLINK was trained) the reference KB consists of
5.9M entities, setting $t_{2}$ to March 2022 an additional set of 0.7M
entities is introduced.

Wikipedia and Oscar datasets are constructed as follows.
Wikipedia: Since usually only the first mention of an entity inside a
Wikipedia article is hyperlinked, we annotate a subset of Wikipedia. We use a
version of L that was trained at $t2$ on a labelled non-public dataset. While
noisy, these predictions are significantly better than what our best discovery
and indexing methods can achieve, therefore we adopt them as pseudo-labels for
the purpose of comparing approaches. As discovery and indexing methods
improve, manual labelling of the evaluation data will afford more accurate
measures. Wikipedia provides time stamps which enables us to separate two time
splits.

OSCAR news: This dataset is based on the common-crawl dataset OSCAR Abadji et
al. (2021). We select a subset of English language news pages which we label
automatically as described above. The dataset consists of 797k samples, which
we split based on their publication date.

For both types of datasets we created two time splits: $D_{1}$, containing
samples preceding $t1$, which is used to train model $L_{t1}$ and $D_{2}$,
with samples preceding $t2$, which is used to train an upper bound model
$L_{t2}$. To adapt $L_{t1}$, we hold out a subset of data from between $t1$
and $t2$ to construct $D_{adapt}$ ($D_{adapt}\cap D_{2}=\emptyset$). Remaining
samples are randomly split into train, dev, test. Figure 2 illustrates the
different data splits. Overall dataset statistics are listed in Table 1.

To construct $D_{adapt}$, we follow Agarwal et al. (2021), and set a ratio of
mentions of type $E_{U}$ to $E_{K}$ of 0.1. 333Naturally this ratio would lie
at 0.02. We made this artificial adjustment to reduce the strong class
imbalance and obtain more interpretable and statistically stable results. Such
adjustment could be lifted once considerably more precise unknown entity
discovery components become available.  As $D_{t2}$-test covers both known and
unknown entities, we use this dataset for EDIN-pipeline evaluation. In Oscar
$D_{t2}$-test, the average number of mentions per $E_{U}$ is 5.6 and is ten
times lower than for $E_{K}$. COVID-19 is the most occurring unknown entity
with 12k mentions. 638k of $E_{U}$ are not mentioned at all and only 733 have
a count larger 10. The distribution of number of $E_{U}$ is more skewed than
the one of $E_{K}$ (skewness 529 vs. 106).

##  7 Results and Discussion

In the following sections, we report results for OSCAR data. Results on
Wikipedia data are consistently lower than OSCAR results and shown in Appendix
F. Our main findings are summarized in Table 3 (left) where we report end-to-
end performance on OSCAR $D_{t2}$-test. As mentions of type $E_{U}$ are
significantly less frequent than mentions of type $E_{K}$, we report results
on these two types separately.

We first discuss upper ($L_{t2}$ a model trained on $D_{t2}$) and lower
($L_{t1}$ a model trained at $D_{t1}$) performance bounds. Next, we follow our
two-step pipeline in adapting $L_{t1}$ such that it can successfully link
mentions of $E_{U}$. Recall, that this pipeline involves two steps: discovery
and indexing. Running $L_{t1}$ on $D_{adapt}$, we first discover mentions of
$E_{U}$s by clustering. These clusters of mentions of $E_{U}$s are then
indexed. We first present results on these components separately and then
assemble the full end-to-end pipeline.

| Known Entities | Unknown Entities | Unknown Entities filtered
---|---|---|---
Model | R@1 | P@1 | NMI | R@1 | P@1 | NMI | R@1 | P@1 | NMI
$L_{t1}$ | 80.1 | 82.0 | 93.5 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
$L_{t2}$ | 78.7 | 79.7 | 93.1 | 49.2 | 31.8 | 93.8 | 63.1 | 26.0 | 93.4
$L_{t1}$-Descp | 80.2 | 82.6 | 93.5 | 46.5 | 32.4 | 93.8 | 58.3 | 26.2 | 90.5
$L_{t1}$-Mention-Gold | 80.6 | 81.5 | 93.3 | 24.0 | 46.6 | 87.0 | 40.7 | 46.6 | 87.0
$L_{t1}$-Mention | 80.3 | 81.9 | 93.4 | 20.5 | 43.7 | 87.6 | 34.5 | 43.5 | 88.7
$L_{t1}$-Cluster-Gold | 80.3 | 82.0 | 94.2 | 30.5 | 51.8 | 85.9 | 51.8 | 51.8 | 85.9
EDIN ($L_{t1}$-Cluster) | 80.3 | 81.9 | 93.4 | 20.8 | 43.1 | 85.9 | 35.4 | 43.1 | 85.3
Table 3: EL performance on OSCAR $D_{t2}$-test for unknown entities $E_{U}$
and known entities $E_{K}$. Left shows end-to-end performance and Right shows
filtered performance where mentions of $E_{U}$ which are not part of the
oracle Cluster-based entity index are dropped from test. Upper/Lower limit:
$L_{t1}$ uses Description-based entity representations, was trained at t1 and
constitutes the lower performance bound. It lacks representations of $E_{U}$.
$L_{t2}$ uses Description-based entity representations, was trained at t2 and
constitutes the upper performance bound. $E_{U}$ are part of the entity index
and labeled-mentions of $E_{U}$ were part of training. Adaptation: For
$L_{t1}$-Descp Description-based entity representations are added to
$L_{t1}$’s entity index. For $L_{t1}$-Mention Mention-based representations of
i) oracle $E_{U}$ and ii) discovered $E^{\prime}_{U}$ part of $D_{adapt}$ are
added to $L_{t1}$’s entity index. For $L_{t1}$-Cluster Cluster-based
representations of i) oracle $E_{U}$ and discovered $E^{\prime}_{U}$ part of
$D_{adapt}$ are added to $L_{t1}$’s entity index.

###  7.1 Lower and upper bound

Our starting point, and an obvious lower performance bound, is given by model
$L_{t1}$. This model lacks representations of $E_{U}$s and its training data
does not contain any corresponding mentions. Therefore, performance on the
subset of $E_{U}$s is 0 for all metrics.

For an upper performance bound we take model $L_{t2}$. The entities in $E_{U}$
were introduced to Wikipedia past $t_{1}$ but before $t_{2}$, meaning that to
$L_{t2}$ these entities are actually known: labeled mentions of $E_{U}$ are
part of the training data and entity representations are part of the index.

$L_{t2}$ reaches similar performance as $L_{t1}$ for $E_{K}$. We suspect
performance differences can be attributed to the difference in training data.

Performance of $L_{t2}$ on mentions of $E_{U}$ is lower than on mentions of
$E_{K}$. The performance discrepancy between $E_{U}$ and $E_{K}$ is largely
due to frequency differences, see Table 2 where we report results per
frequency band. We suspect that the remaining difference can be attributed to
the degradation of the pre-trained LM and the entity encoder. Note that while
labelled mentions of $E_{U}$ were seen during the training phase of L, BLINK’s
entity encoder was not retrained. To investigate this hypothesis further, we
test $L_{t1}$ on mentions of $E_{K}$ that meet two conditions: i) time stamps
of these samples are posterior to t1 and ii) two or more mentions of $E_{U}$
occur in their context. Thus, we target mentions of $E_{K}$ in novel contexts
to which neither BLINK nor the PLM have been exposed. We find that recall
drops only slightly from 80.1 to 79.9 but precision drops from 82.0 to 75.9.
This result points to the conclusion that $E_{U}$ are also a source of noise
when trying to link mentions of $E_{K}$.

###  7.2 Discovery

The first condition for effective discovery is the ability to reliably detect
mentions of both known and unknown entities. Recall of $L_{t1}$ on $D_{adapt}$
for Mention Detection task is 90% for $E_{K}$s and 86% for $E_{U}$s entities.
As expected, recall of mentions of $E_{K}$ is higher as no mentions of $E_{U}$
were seen during training. As a reference, running $L_{t2}$ on $D_{adapt}$, we
find that for both $E_{K}$ and $E_{U}$ 91% of mentions are recalled. Note
again, that for $L_{t2}$, $E_{U}$ are known. This indicates that mention
detection is not affected by frequency differences and PLM degradation.

Next, the clustering step to classify between mentions of $E_{U}$ and $E_{K}$
follows. We report clustering quality of 93.1% NMI. When assigning entities to
clusters as described in Section 7.2, we find that our first attempt results
in 35% recall and 5% precision on $D_{adapt}$-dev. This setting is close in
spirit to Agarwal et al. (2021) where a single entity-mention link is decisive
for discovery.

A qualitative error analysis reveals that low recall is rooted in the problem
that mention embeddings of $E_{U}$ (e.g. BioNTech) can have high similarity
with entity representations of $E_{K}$ (e.g. of other biotechnology
companies). If no mentions of these $E_{K}$ are part of $D_{adapt}$, we
wrongly assign the high similarity entity to the cluster of mentions of
$E_{U}$. We suspect that this problem is particularly pronounced in our
setting as EDIN-benchmark is a large scale EL benchmark (up to 6 times more
entities in the reference KB and up to 36 times more mentions in clustering
set compared to Agarwal et al. (2021)) with many long-tail entities (Agarwal
et al. (2021) entities were randomly selected).

We find that low precision is rooted in the issue that $E_{K}$ are indexed as
false positive when occurring in novel contexts, e.g., “blood tests” or
“vaccine” in the context of COVID form their own distinct clusters.

With the modified cluster-entity assignment method, described in 7.2,
discovery still results in low but higher precision (8%) and high recall (80%)
of $E_{U}$. 1196 of 1368 $E_{U}$ part of $D_{adapt}$ with more than three
mentions are indexed.

###  7.3 Indexing

After discovery, we need to integrate clusters of mentions of $E_{U}$ into the
existing entity index. In the following, we i) quantify the impact of re-
training the model after indexing, ii) compare Description-based with Cluster-
based indexing and iii) compare Cluster-based with Mention-based indexing.

####  7.3.1 Effect of re-training

In the zero-shot EL problem, all entities are part of the index at training
time. In the setting of EDIN-benchmark, discovery and indexing of $E_{U}$
happen after training. We run the following experiments to study the effect of
this difference:

  * •

Descp-post-train: Description-based entity representations are added to the
index _after_ training L on t1.

  * •

Descp-pre-train: Description-based entity representations are added to the
index _before_ training L on t1.

Recall and precision of $E_{U}$ in the setting where entity representations
are added before the training is 47% and 32% respectively, see Table 4. Recall
and precision in the setting where entity representations are added post
training is 26% and 17% points lower respectively.

We note that entities in $E_{U}$ can potentially be placed in close proximity
to $E_{K}$ in embedding space. When these entity encodings were present during
training, they can be picked up as hard negatives and the mention encoder can
learn to circumvent them. This hypothesis is supported by experiments showing
that the mean similarity between mentions and correct known entity embeddings
increases significantly when the mention encoder is re-trained after adding
the new entities. For details see Appendix D.

The take-away for the EDIN-pipeline is that, after adding new entity
representations to the index, another round of training is needed to adapt the
mention encoder to the updated index. We adopt this approach for the following
experiments. Besides adapting the mention encoder, retraining BLINK could have
a similar effect: in such case learning from hard negative can affect the
spacing of entity encodings. As re-training BLINK is expensive, we did not
explore this option in this work.

| Unknown Entities | Known Entities
---|---|---
| R@1 | P@1 | NMI | R@1 | P@1 | NMI
Not re-trained | 20.6 | 15.5 | 95.2 | 80.1 | 82.3 | 93.5
Re-trained | 46.5 | 32.4 | 93.8 | 80.2 | 82.6 | 93.5

Table 4: Effect of re-training: End-to-end EL performance on OSCAR
$D_{t2}$-test when adding Description-based representation of unknown entities
$E_{U}$ to the entity index before and after training of $L_{t1}$.

####  7.3.2 Description vs. Cluster-based indexing

We compared Description-based entity representations and Cluster-based
indexing. To isolate discovery and indexing performance, we first evaluate
performance using oracle clusters, where we replace the discovery method run
on $D_{adapt}$ with an oracle where mentions of $E_{U}$ are discovered with
perfect precision and recall and perfectly clustered.

In this setting, Cluster-based indexing results in 19% higher precision but
16% lower recall, compared to Description-based indexing, see Table 3 (left)
$L_{t1}$-Cluster-Gold. The drop in recall for Cluster-based indexing can
partially be attributed to the fact that a subset of $E_{U}$ part of
$D_{t2}$-test are not mentioned in $D_{adapt}$, and therefore no
representation can be built for them. When filtering mentions of $E_{U}$ not
part of $D_{adapt}$ out of the test set, the gap in recall decreases to 7%,
see Table 3 (right).

The take-away is that Cluster-based indexing relying on concatenated mentions
in context instead of manually crafted descriptions has high precision
potential but recall is challenging as a randomly selected adaptation set is
unlikely to cover long-tail entities.

####  7.3.3 Mention vs. Cluster-based indexing

In this section, we compare Mention-based and Cluster-based indexing. Again,
to isolate discovery and indexing we first report results using oracle
clusters of $E_{U}$. When indexing mentions of these clusters individually,
recall drops by 6% points and precision by 5%, see Table 3 (left),
$L_{t1}$-Mention-Gold. When reducing the testset to mentions of entities that
were actually discoverable, the difference in recall becomes even more
pronounced: 40.7% R@1 for Mention-based vs. 51.8 for Cluster-based indexing,
see Table 3 (right).

Interestingly, this means that the ability to attend over multiple mentions in
context and unify their information into a single embedding leads to superior
representations. Note that here the entity encoder was neither trained to deal
with the style of individual mentions in context nor with clusters of mentions
in context. For future work, it would be interesting to see if Cluster-based
indexing can be generally beneficial to EL, outside of the context of EDIN-
pipeline. This would require training the entity encoder to specifically adapt
to the new input style.

###  7.4 End-to-end model

The prior section presented results of the two components individually. Now,
we assemble the full end-to-end pipeline. We replace the oracle clusters of
$E_{U}$ by discovered clusters of $E_{U}^{\prime}$. Errors in discovery that
affect indexing are: i) Clusters of mentions of $E_{K}$ classified false
positively as unknown and clusters of mentions of $E_{U}$ false negatively
classified as known and ii) incomplete and impure clusters. We find that
performance of Mention-based and Cluster-based indexing in terms of R@1 and
P@1 converges and is significantly lower than their oracle counterparts.
Mention-based indexing has better clustering performance.

When reducing the testset to mentions of entities that were actually
discoverable, thus part of $D_{adapt}$, Cluster-based indexing is 1% point
better in terms of recall and 0.4 % worse in precision, Table 3 (right). When
reducing the testset further to mentions of entities that were in fact
discovered, recall of Cluster-based indexing is with 58.4% better than that of
Mention-based indexing (55.5%).

Besides end-to-end performance, we also report entity disambiguation
performance with oracle mention detection in Table 6 in Appendix E. Here, we
find that Mention-based indexing is performing worse than Cluster-based
indexing across all metrics: 1% point difference in terms of recall, 3% point
difference in terms of precision and 0.7% points difference in terms of NMI.

Overall, these results show that EDIN-benchmark is challenging. In the end-to-
end setting, errors easily propagate. Most notably, we see this manifest when
i) comparing Table 6 and 3 (right) results where the recall problem of $E_{U}$
becomes apparent and ii) comparing performance of oracle and automatic
clusters where precision drops by 10% points. Overall, we find that Cluster-
based indexing, with the advantage of attending to and unifying the
information of multiple mentions, performs better than Mention-based indexing.
We call this version the EDIN-pipeline. Besides yielding an index that scales
in memory with the number of entities rather than the number of mentions – a
significant advantage when the number of entities is already large and in view
of a streaming extension – it generates fixed-size entity embeddings as a by-
product that can have applications of their own and can be used to enhance
PLMs (e.g., Peters et al. (2019)).

In future work, we want to explore a setting where $E_{U}$ are discovered in a
streaming fashion, thus scaling up $D_{adapt}$ and dropping the crafted
90%-10% ration of $E_{K}$ vs. $E_{U}$. This would pose challenges in terms of
scale and precision in discovery. In terms of precision, a human in the loop
approach, as proposed by Hoffart et al. (2016) in the context of keeping KBs
fresh, to introduce a component of supervision into our end-to-end pipeline
might be needed.

##  8 Related work

Entity linking is an extensively studied task. Prior to the introduction of
PLMs, EL systems used frequency and typing information, alias tables, TF-IDF-
based methods and neural networks to model context, mention and entity
Cucerzan (2007); Bunescu and Paşca (2006); Milne and Witten (2008); He et al.
(2013); Sun et al. (2015a); Lazic et al. (2015); Raiman and Raiman (2018);
Kolitsas et al. (2018); Gupta et al. (2017); Ganea and Hofmann (2017); Khalife
and Vazirgiannis (2018); Onoe and Durrett (2019).

Gillick et al. (2019) present a PLM-based dual encoder architecture that
encodes mentions and entities in the same dense vector space and performs EL
via kNN search. Logeswaran et al. (2019) proposed the zero-shot EL task and
show that domain adaptive training can address the domain shift problem.
Subsequently, Wu et al. (2020) showed that pre-trained zero-shot architectures
are both highly accurate and computationally efficient at scale. None of these
works tackle the problem of unknown entities.

Recently, FitzGerald et al. (2021) model EL entirely as mappings between
mentions, where inference involves a NN search against all known mentions of
all entities in the training set. In this setting mentions need to be labeled.
They do not explore their approach in the setting of unknown entities.

Prior to dense retrieval-based EL, unknown entity discovery work includes:
Ratinov et al. (2011) train a classifier to determine whether the top ranked
EL candidate is unknown relying on local context, global Wikipedia coherence,
and additional linker features. Mentions of known entities that are
incorrectly linked are among the training examples for unknown entity
discovery thus decreasing quality.; Nakashole et al. (2013) introduce a model
for unknown entity discovery and typing leveraging incompatibilities and
correlations among entity types. This method ignores context. Therefore,
mentions not registered in the reference KB are regarded unknown; Hoffart et
al. (2014); Wu et al. (2016) study a variety of features for unknown entity
discovery: Hoffart et al. (2014) use perturbation-based confidence measures
and key-phrase representations and Wu et al. (2016) explore different feature
spaces, e.g., topical and search engine features. These features are not
readily available and incorporating them into PLM-based approaches is not
straightforward and outside the scope of this work.; Ji et al. (2015);
Derczynski et al. (2017) introduce shared tasks for discovery. These tasks do
not target end-to-end dense-retrieval based EL and are outdated, e.g., treat
iPhone as an unknown entity.; Akasaki et al. (2019) introduces a time
sensitive method of discovering emerging entities. This method relies on
training data in form of labeled emerging contexts. This data is collected by
searching only exact mentions of unknown entities and therefore neglecting the
issue of ambiguity and mention variety.

None of these works consider unknown entities in an end-to-end setting
including mention detection, unknown entity discovery and indexing. Also, we
cannot use their datasets to evaluate as these entities were part of training
the PLM.

Closely related to EL is the task of cross document entity co-reference (CDC),
where no reference KB is present Bagga and Baldwin (1998); Gooi and Allan
(2004); Singh et al. (2011); Dutta and Weikum (2015b); Barhom et al. (2019);
Cattan et al. (2021a); Caciularu et al. (2021); Cattan et al. (2021b). Most
recently, Logan IV et al. (2021) benchmark methods for streaming CDC, where
mentions are disambiguated in a scalable manner via incremental clustering.
Our work can be seen as bridging between the world of CDC and EL. We harvest
CDC to discover and cluster unknown entities but then integrate them into a
curated list of entities. Dutta and Weikum (2015a) also combine clustering-
based CDC decisions and linking but as this work is using sparse bag-of-word
representations, it is not well suited for the embedding-based representations
used in this work.

Most recently, Angell et al. (2021) introduce a new EL method using document-
level supervised graph-based clustering. Agarwal et al. (2021) extend this
work to cross-document EL and entity discovery. In this work, we adopt a more
standard bi-encoder architecture (i.e. BLINK), with better EL scalability
potential (memory linear in the number of entities and not in the number of
mentions) and an existing end-to-end extension. We use a modified version of
their discovery method.

Besides dense-retrieval based EL, De Cao et al. (2021) proposed generative EL.
We plan to study the problem of integrating unknown entities into generative
models in future work.

##  9 Conclusion

This work created EDIN benchmark and pipeline. EDIN-benchmark is a large-
scale, end-to-end entity linking benchmark with a clear cut temporal
segmentation for _Unknown Entity Discovery and Indexing_. EDIN-pipeline
detects and clusters mentions of unknown entities in context. These clusters
of unknown mentions are then collapsed into single embeddings and integrated
into the entity index of the original entity linking system.

## References

  * Abadji et al. (2021) Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2021.  Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus.  Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9) 2021. Limerick, 12 July 2021 (Online-Event), pages 1 – 9, Mannheim. Leibniz-Institut für Deutsche Sprache.
  * Agarwal et al. (2021) Dhruv Agarwal, Rico Angell, Nicholas Monath, and Andrew McCallum. 2021.  Entity linking and discovery via arborescence-based supervised clustering.
  * Akasaki et al. (2019) Satoshi Akasaki, Naoki Yoshinaga, and Masashi Toyoda. 2019.  Early discovery of emerging entities in microblogs.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_ , pages 4882–4889. ijcai.org.
  * Angell et al. (2021) Rico Angell, Nicholas Monath, Sunil Mohan, Nishant Yadav, and Andrew McCallum. 2021\.  Clustering-based inference for biomedical entity linking.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2598–2608, Online. Association for Computational Linguistics.
  * Anonymous (2022) Anonymous. 2022.  End to end entity linking in 100 languages.
  * Bagga and Baldwin (1998) Amit Bagga and Breck Baldwin. 1998.  Entity-based cross-document coreferencing using the vector space model.  In _Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics - Volume 1_ , ACL ’98/COLING ’98, page 79–85, USA. Association for Computational Linguistics.
  * Barhom et al. (2019) Shany Barhom, Vered Shwartz, Alon Eirew, Michael Bugert, Nils Reimers, and Ido Dagan. 2019.  Revisiting joint modeling of cross-document entity and event coreference resolution.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4179–4189, Florence, Italy. Association for Computational Linguistics.
  * Bunescu and Paşca (2006) Razvan Bunescu and Marius Paşca. 2006.  Using encyclopedic knowledge for named entity disambiguation.  In _11th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 9–16, Trento, Italy. Association for Computational Linguistics.
  * Caciularu et al. (2021) Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Peters, Arie Cattan, and Ido Dagan. 2021.  CDLM: Cross-document language modeling.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2648–2662, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Cao et al. (2018) Yixin Cao, Lei Hou, Juan-Zi Li, and Zhiyuan Liu. 2018.  Neural collective entity linking.  In _COLING_.
  * Cattan et al. (2021a) Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. 2021a.  Realistic evaluation principles for cross-document coreference resolution.  In _Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics_ , pages 143–151, Online. Association for Computational Linguistics.
  * Cattan et al. (2021b) Arie Cattan, Sophie Johnson, Daniel S. Weld, Ido Dagan, Iz Beltagy, Doug Downey, and Tom Hope. 2021b.  Scico: Hierarchical cross-document coreference for scientific concepts.  In _3rd Conference on Automated Knowledge Base Construction_.
  * Cucerzan (2007) Silviu Cucerzan. 2007.  Large-scale named entity disambiguation based on Wikipedia data.  In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_ , pages 708–716, Prague, Czech Republic. Association for Computational Linguistics.
  * De Cao et al. (2021) Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021.  Autoregressive entity retrieval.  In _International Conference on Learning Representations_.
  * Derczynski et al. (2017) Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017.  Results of the WNUT2017 shared task on novel and emerging entity recognition.  In _Proceedings of the 3rd Workshop on Noisy User-generated Text_ , pages 140–147, Copenhagen, Denmark. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dhingra et al. (2022) Bhuwan Dhingra, Jeremy Cole, Julian Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William Cohen. 2022.  Time-aware language models as temporal knowledge bases.  _Transactions of the Association for Computational Linguistics_ , 10(0):257–273.
  * Dutta and Weikum (2015a) Sourav Dutta and Gerhard Weikum. 2015a.  C3EL: A joint model for cross-document co-reference resolution and entity linking.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 846–856, Lisbon, Portugal. Association for Computational Linguistics.
  * Dutta and Weikum (2015b) Sourav Dutta and Gerhard Weikum. 2015b.  Cross-document co-reference resolution using sample-based clustering with knowledge enrichment.  _Transactions of the Association for Computational Linguistics_ , 3:15–28.
  * FitzGerald et al. (2021) Nicholas FitzGerald, Dan Bikel, Jan Botha, Daniel Gillick, Tom Kwiatkowski, and Andrew McCallum. 2021.  MOLEMAN: Mention-only linking of entities with a mention annotation network.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 278–285, Online. Association for Computational Linguistics.
  * Ganea and Hofmann (2017) Octavian-Eugen Ganea and Thomas Hofmann. 2017.  Deep joint entity disambiguation with local neural attention.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2619–2629, Copenhagen, Denmark. Association for Computational Linguistics.
  * Gillick et al. (2019) Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019.  Learning dense representations for entity retrieval.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 528–537, Hong Kong, China. Association for Computational Linguistics.
  * Gooi and Allan (2004) Chung Heong Gooi and James Allan. 2004.  Cross-document coreference on a large scale corpus.  In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_ , pages 9–16, Boston, Massachusetts, USA. Association for Computational Linguistics.
  * Gupta et al. (2017) Nitish Gupta, Sameer Singh, and Dan Roth. 2017.  Entity linking via joint encoding of types, descriptions, and context.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2681–2690, Copenhagen, Denmark. Association for Computational Linguistics.
  * He et al. (2013) Zhengyan He, Shujie Liu, Mu Li, Ming Zhou, Longkai Zhang, and Houfeng Wang. 2013\.  Learning entity representation for entity disambiguation.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 30–34, Sofia, Bulgaria. Association for Computational Linguistics.
  * Hoffart et al. (2014) Johannes Hoffart, Yasemin Altun, and Gerhard Weikum. 2014.  Discovering emerging entities with ambiguous names.  In _Proceedings of the 23rd International Conference on World Wide Web_ , WWW ’14, page 385–396, New York, NY, USA. Association for Computing Machinery.
  * Hoffart et al. (2016) Johannes Hoffart, Dragan Milchevski, Gerhard Weikum, Avishek Anand, and Jaspreet Singh. 2016.  The knowledge awakens: Keeping knowledge bases fresh with emerging entities.  _Proceedings of the 25th International Conference Companion on World Wide Web_.
  * Ji et al. (2015) Heng Ji, Joel Nothman, Ben Hachey, and Radu Florian. 2015.  Overview of TAC-KBP2015 tri-lingual entity discovery and linking.  In _Proceedings of the 2015 Text Analysis Conference, TAC 2015, Gaithersburg, Maryland, USA, November 16-17, 2015, 2015_. NIST.
  * Johnson et al. (2017) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017.  Billion-scale similarity search with gpus.  _arXiv preprint arXiv:1702.08734_.
  * Khalife and Vazirgiannis (2018) Sammy Khalife and Michalis Vazirgiannis. 2018.  Scalable graph-based individual named entity identification.  _CoRR_ , abs/1811.10547.
  * Kolitsas et al. (2018) Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas Hofmann. 2018.  End-to-end neural entity linking.  In _Proceedings of the 22nd Conference on Computational Natural Language Learning_ , pages 519–529, Brussels, Belgium. Association for Computational Linguistics.
  * Lazaridou et al. (2021) Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021.  Mind the gap: Assessing temporal generalization in neural language models.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 29348–29363. Curran Associates, Inc.
  * Lazic et al. (2015) Nevena Lazic, Amarnag Subramanya, Michael Ringgaard, and Fernando Pereira. 2015\.  Plato: A selective context model for entity resolution.  _Transactions of the Association for Computational Linguistics_ , 3:503–515.
  * Ledell Wu (2020) Martin Josifoski Sebastian Riedel Luke Zettlemoyer Ledell Wu, Fabio Petroni. 2020\.  Zero-shot entity linking with dense entity retrieval.  In _EMNLP_.
  * Li et al. (2020) Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih. 2020\.  Efficient one-pass end-to-end entity linking for questions.  In _EMNLP_.
  * Logan IV et al. (2021) Robert L Logan IV, Andrew McCallum, Sameer Singh, and Dan Bikel. 2021.  Benchmarking scalable methods for streaming cross document entity coreference.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4717–4731, Online. Association for Computational Linguistics.
  * Logeswaran et al. (2019) Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019.  Zero-shot entity linking by reading entity descriptions.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3449–3460, Florence, Italy. Association for Computational Linguistics.
  * Milne and Witten (2008) David N. Milne and Ian H. Witten. 2008.  Learning to link with wikipedia.  In _CIKM ’08_.
  * Nakashole et al. (2013) Ndapandula Nakashole, Tomasz Tylenda, and Gerhard Weikum. 2013.  Fine-grained semantic typing of emerging entities.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1488–1497, Sofia, Bulgaria. Association for Computational Linguistics.
  * Onoe and Durrett (2019) Yasumasa Onoe and Greg Durrett. 2019.  Fine-grained entity typing for domain independent entity linking.  _CoRR_ , abs/1909.05780.
  * Onoe and Durrett (2020) Yasumasa Onoe and Greg Durrett. 2020.  Fine-grained entity typing for domain independent entity linking.  In _AAAI_.
  * Peters et al. (2019) Matthew E Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. 2019.  Knowledge enhanced contextual word representations.  In _Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Raiman and Raiman (2018) Jonathan Raiman and Olivier Raiman. 2018.  Deeptype: Multilingual entity linking by neural type system evolution.  In _AAAI_.
  * Ratinov et al. (2011) Lev Ratinov, Dan Roth, Doug Downey, and Mike Anderson. 2011.  Local and global algorithms for disambiguation to Wikipedia.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 1375–1384, Portland, Oregon, USA. Association for Computational Linguistics.
  * Singh et al. (2011) Sameer Singh, Amarnag Subramanya, Fernando Pereira, and Andrew McCallum. 2011.  Large-scale cross-document coreference using distributed inference and hierarchical models.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 793–803, Portland, Oregon, USA. Association for Computational Linguistics.
  * Sun et al. (2015a) Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015a.  Modeling mention, context and entity with neural networks for entity disambiguation.  In _IJCAI_.
  * Sun et al. (2015b) Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015b.  Modeling mention, context and entity with neural networks for entity disambiguation.  In _Proceedings of the 24th International Conference on Artificial Intelligence_ , IJCAI’15, page 1333–1339. AAAI Press.
  * Wu et al. (2020) Ledell Yu Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020.  Scalable zero-shot entity linking with dense entity retrieval.  In _EMNLP_.
  * Wu et al. (2016) Zhaohui Wu, Yang Song, and C. Lee Giles. 2016.  Exploring multiple feature spaces for novel entity discovery.  In _AAAI_.

##  Appendix A Model

In the following sections, we explain Anonymous (2022)’s architecture in
detail.

###  A.1 Mention Detection

For every span [i,j], the MD head calculates the probability of [i,j] being
the mention of an entity by scoring whether i is the start of the mention, j
is the end of the mention, and the tokens between i and j are the insides:

$\displaystyle s_{start(i)}=\textbf{w}_{start}^{T}\textbf{p}_{i}$
$\displaystyle s_{end(i)}=\textbf{w}_{end}^{T}\textbf{p}_{j}$ $\displaystyle
s_{mention(t)}=\textbf{w}_{mention}^{T}\textbf{p}_{t}$

where $\textbf{w}_{start},\textbf{w}_{end},\textbf{w}_{mention}$ are learnable
vectors and $\textbf{p}_{i}$ paragraph token representations based on BERT:

$\displaystyle[\textbf{p}_{1}\ldots\textbf{p}_{n}]=BERT([CLS]p_{1}\ldots
p_{n}[SEP])$

Overall mention probabilities are computed as:

$\displaystyle
p([i,j])=\sigma(s_{start(i)}+s_{end(j)}+\sum_{t=i}^{j}(s_{mention(t)}))$

Top candidates are selected as mention candidates and propagate to the next
step.

###  A.2 Entity Disambiguation

The ED head receives mention spans in the text and finds the best matching
entity in the KB.

Following Wu et al. (2020), ED is based on dense retrieval. Description-based
entity representations are computed as follows:

$\displaystyle\textbf{e}=BERT_{[CLS]}([CLS]t(e)[SEP]d(e)[SEP])$

Following Li et al. (2020), mention representations are constructed with one
pass of the encoder and without mention boundary tokens by pooling mention
tokens from the encoder output:

$\displaystyle\textbf{m}_{i,j}=FFL(\textbf{p}_{i}\ldots\textbf{p}_{j})$

Similarity score $s$ between the mention candidate and an entity candidate
$e\in E$ are computed:

$\displaystyle s(e,[i,j])=\textbf{e}*\textbf{m}_{i,j}$

A likelihood distribution over all entities, conditioned on the mention
$[i,j]$ is computed:

$\displaystyle p(e|[i,j])=\frac{exp(s(e,[i,j]))}{\sum_{e^{\prime}\in
E}exp(s(e^{\prime},[i,j]))}$

$<[i,j],e^{*}>$, such that

$\displaystyle e^{*}=argmax_{e}(p([i,j],e)),$

are passed as a candidate $<$ mention span, entity $>$ tuple to the rejection
head.

###  A.3 Rejection head

MD and ED steps over-generate. R looks at an $(e^{*},[i,j])$ pair holistically
decides whether to accept it. Input features to R are the MD score $p([i,j])$,
the ED score $p(e^{*}|[i,j])$, the mention representation $\textbf{y}_{i,j}$,
top-ranked candidate representation $\textbf{x}_{e^{*}}$ as well as their
difference and Hadamard product. The concatenation of these features is fed
through a feed-forward network to output the final entity linking score
$p([i,j],e^{*})$. All $p([i,j],e^{*})>\gamma$ are accepted where $\gamma$ is a
threshold set to 0.4.

###  A.4 Training

Following prior work Sun et al. (2015b); Cao et al. (2018); Gillick et al.
(2019); Onoe and Durrett (2020), training is split into two stages. First, ED
only is trained on a Wikipedia dataset. This dataset is constructed by
extracting Wikipedia hyperlinks to labeled mention-entity pairs and consists
of 17M training samples. Then, ED, MD and R are trained jointly on the
downstream dataset (either Oscar or Wikipedia). To train the ED head, frozen
entity representations are used. As entity embeddings do not change during
training, entity embeddings can be indexed using quantization algorithms for a
fast kNN search (using FAISS Johnson et al. (2017) framework with HNSW index).
A likelihood distribution over positive and mined hard negative entities for
each mention is computed. Negative Log-Likelihood loss across all gold
mentions in the text is used.

To train MD, binary cross-entropy loss between all possible valid spans and
gold mentions in the training set is computed.

To train R, binary cross-entropy loss between retrieved mention-entity pairs
and gold mention-entity pairs is used.

##  Appendix B OSCAR-based dataset

We select the following six online news pages:

BBC: https://www.bbc.com/
CNN: https://www.cnn.com/
Deutsche Welle: https://www.dw.com/en/
Reuters: https://www.reuters.com/article/
Guardian: https://www.theguardian.com/
Associated Press: https://apnews.com/article/

##  Appendix C Hyper-parameters Adaptation phase

Using OSCAR $D_{adapt}$-dev, we optimize mention score threshold $s_{m}$,
greedy NN distance threshold $d_{m}$ and mention entity similarity threshold
$d_{e}$.

We optimize $s_{m}$ in range 0.0 to 1.0 in steps of 0.1 for $E_{U}$ discovery
recall. We optimize $d_{m}$ in range 0.5 to 1.0, in steps of 0.0001 for NMI.
We optimize $d_{e}$ for $E_{U}$ discovery recall in range 50 to 250 in steps
of 10. For results, see Table 5.

We report recall of 81% and precision of 6% for clusters referring to unknown
entities. Recall of clusters referring to known entities is 88% with precision
96%. Clustering NMI is 0.92.

Parameter | Value
---|---
$d_{m}$ | 0.8171
$s_{m}$ | 0.5
$d_{e}$ | 110
Table 5: Hyper-parameters adaptation phase

##  Appendix D Effect of re-training

We show that by re-training L after indexing, L learns to circumvent $E_{U}$:
We identify known entities part of the training set that are in close
proximity of unknown entities (confusable known entities). We compare the
average similarity between mentions and their respective linked entity when
adding unknown entities before training vs. after training. Mean similarity
when adding unknown entities before training is 93.28 for confusable known
entities and 92.57 for other known entities. A t-test shows that this
difference is significant (p-value of 0.0001 with $<$ 0.05). As a reference,
mean similarity when adding unknown entities post training is 92.65
irrespective of whether they are confusable or not.

##  Appendix E Disambiguation Results

Besides end-to-end performance, we also report entity disambiguation
performance with oracle mention detection in Table 6.

##  Appendix F Wikipedia Results

We report performance on Wikipedia $D_{t2}$-test in Table 7. Due to a smaller
$D_{adapt}$, end-to-end performance is lower. When filtering Wikipedia
$D_{t2}$-test for mentions of discovered entities, $L_{t1}$-Cluster-Gold
precision is 40.5 and $L_{t1}$-Cluster recall is 15.3.

| OSCAR |
---|---|---
| Unknown Entities | Known Entities
Model | R@1 | P@1 | NMI | R@1 | P@1 | NMI |
$L_{Dt1}$ | 0.0 | 0.0 | 0.0 | 92.2 | 92.2 | 96.0 |
$L_{Dt2}$ | 63.5 | 45.3 | 96.8 | 90.0 | 90.2 | 96.0 |
$L_{t1}$-Descp | 58.0 | 33.9 | 96.3 | 92.1 | 92.3 | 96.1 |
$L_{t1}$-Mention | 26.2 | 30.8 | 92.7 | 92.2 | 92.2 | 96.0 |
EDIN ($L_{t1}$-Cluster) | 27.9 | 34.1 | 93.4 | 92.2 | 92.2 | 96.2 |
Table 6: Entity Disambiguation performance  on OSCAR $D_{t2}$-test. | Unknown Entities | Known Entities
---|---|---
Model | R@1 | P@1 | NMI | R@1 | P@1 | NMI
$L_{t1}$ | 0.0 | 0.0 | 0.0 | 70.5 | 75.8 | 95.4
$L_{t2}$ | 33.6 | 25.0 | 98.3 | 70.6 | 75.4 | 95.3
$L_{t1}$-Descp | 33.9 | 20.0 | 98.0 | 71.2 | 74.4 | 95.3
$L_{t1}$-Cluster-Gold | 7.8 | 55.6 | 90.6 | 70.1 | 75.9 | 95.6
EDIN ($L_{t1}$-Cluster) | 1.8 | 15.4 | 93.4 | 71.1 | 74.1 | 95.3
Table 7: End-to-end EL performance on Wikipedia $D_{t2}$-test.