=2pt

# Reveal the Unknown: Out-of-Knowledge-Base Mention
Discovery with Entity Linking

Hang Dong1, Jiaoyan Chen2, Yuan He1, Yinan Liu3, Ian Horrocks1
1 University of Oxford, 2 The University of Manchester, 3 Nankai University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Discovering entity mentions that are out of a Knowledge Base (KB) from texts
plays a critical role in KB maintenance, but has not yet been fully explored.
The current methods are mostly limited to the simple threshold-based approach
and feature-based classification; the datasets for evaluation are relatively
rare. In this work, we propose BLINKout, a new BERT-based Entity Linking (EL)
method which can identify mentions that do not have a corresponding KB entity
by matching them to a special NIL entity. To this end, we integrate novel
techniques including NIL representation, NIL classification, and synonym
enhancement. We also propose Ontology Pruning and Versioning strategies to
construct out-of-KB mentions from normal, in-KB EL datasets. Results on four
datasets of clinical notes and publications show that BLINKout outperforms
existing methods to detect out-of-KB mentions for medical ontologies UMLS and
SNOMED CT.

##  1 Introduction

Knowledge Bases (KBs) are widely used for representing entities and facts
about the world with reasoning supported. KBs are inherently incomplete. New
entities are constantly emerging, for example, “Curry-Jones syndrome” Wigby et
al. (2017) was not added to SNOMED CT until 2017; and from late 2020 to early
2022, a new variant of SARS-CoV-2 emerged every few months World Health
Organization (2022). Delays in incorporating these entities into the KB may
result in failure to cataloguing, searching, and reasoning with them.
Automated discovery of mentions of new entities from common resources such as
text can support the maintenance of KBs.

Updating KBs with entities from texts is highly relevant to Entity Linking
(EL) which is to match mentions in texts to entities in a KB Rao et al.
(2013); Shen et al. (2014). Current works on EL, however, often assume all the
target mentions have corresponding entities in the KB and ignore mentions that
have no corresponding entities Wu et al. (2020); Basaldella et al. (2020). The
latter are sometimes called out-of-KB mentions and are matched with a special
NIL entity. To maintain a KB, it is required to discover out-of-KB mentions
which can be further processed as new KB entities.111In this work, we focus on
identifying out-of-KB mentions, and leave the canonicalization and placement
of the new entities in the KB for future work.

As NIL is not described in the KB, it is hard to obtain their lexical or
embedding representations, and consequently, it is hard to predict them
directly. One idea is setting a threshold towards the mention-to-entity
matching score: a mention is regarded as NIL if its scores to all the KB
entities are below the threshold Bunescu and Paşca (2006). Another idea is to
classify whether a mention is NIL or not based on a set of features regarding
the mention and its top-k entity candidates Wu et al. (2016). Some other
studies attempt to represent the NIL entity with key phrases or features with
the help of external corpora Hoffart et al. (2014) or manual effort Hoffart et
al. (2016).

Recently, neural EL has shown good performance by applying language models
(LMs), e.g. BERT Devlin et al. (2019), to represent texts and entities Wu et
al. (2020). In contrast, methods for out-of-KB mentions have been relatively
static or unchanged, as reviewed in Shen et al. (2021); Sevgili et al. (2022),
where the traditional, threshold-based approach is combined with deep learning
Ji et al. (2020); Chen et al. (2021). There is a lack of studies to seamlessly
integrate out-of-KB mention discovery with pre-trained LM-based approaches
like BLINK Wu et al. (2020).

Besides the shortage of methodology research, benchmarking datasets
considering out-of-KB mention discovery are also relatively rare compared to
in-KB EL, especially in the biomedical domain.

Also, entities in the biomedical domain are likely to have various surface
forms or synonyms D’Souza and Ng (2015); Chen et al. (2021). This entity
variety suggests enhancing synonyms for EL.

In this work, we define the out-of-KB mention discovery problem and propose a
method named BLINKout, based on the BERT-based EL method, BLINK Wu et al.
(2020), and adopt new techniques including NIL representation &
classification, and synonym enhancement. We also propose strategies to
automatically generate out-of-KB mention discovery benchmarks from an in-KB EL
dataset by pruning or using historical versions of the linked ontology. Four
datasets are constructed, including clinical notes and biomedical
publications, with the medical ontologies, UMLS and SNOMED CT. Our approach
and data settings can support establishing a life cycle for out-of-KB mention
discovery along with in-KB entities.

Experimental results on the datasets show the advantage of the BLINKout
approach for out-of-KB discovery, with comparison to rule-based, threshold-
based, and feature-based baselines and our ablation studies. Our dataset
construction scripts and system implementation will be publicly available.

##  2 Preliminaries

###  2.1 Problem Definition

Given a KB containing a set of entities $E$ and a text corpus in which a set
of mentions $M$ are identified in advance, Entity Linking (EL, a.k.a. Entity
Disambiguation or Entity Normalisation) is to map each mention $m$ in $M$ to
its corresponding entity $e$ in $E$ Shen et al. (2014). Due to the limited
knowledge coverage of a KB, it is possible that there is no entity in $E$ that
can be matched with a given mention. Thus we have the following more general
problem named out-of-KB mention discovery, which is formulated as the extended
EL task that considers both in-KB and out-of-KB mentions.

Given a text corpus with a set of identified mentions $D_{M}$ and an ontology
with entities $E$, out-of-KB mention discovery is to develop a function $f$
such that each mention $d_{m}$ in $D_{M}$ is mapped to an item $f(d_{m})\in
E\cup\\{\textit{NIL}\\}$, where NIL denotes that there are no entities in $E$
that can be matched with $d_{m}$. In this study, we focus on ontologies, a
common type of KB that is often defined as a shared, explicit specification of
a conceptualisation of a domain Gruber (1995), and we consider an ontology’s
classes (or concepts) as its entities for linking. Note that each ontology
entity may have definitions and synonyms that can be utilised for out-of-KB
mention discovery.

###  2.2 BERT-based Entity Linking

EL approaches often have two stages: candidate creation and candidate ranking
Rao et al. (2013); Ji et al. (2020); Wu et al. (2020). In candidate creation,
the approaches aim to narrow down the vast number of entities into a
manageable subset (e.g., tens or hundreds), and in candidate ranking, the
approaches aim to rank the candidate entities of each mention according to the
probability that they match the given mention.

We summarise BERT-based Entity Linking in Wu et al. (2020) below with
adaptation using a triplet loss for the bi-encoder.

#### Candidate Creation with Bi-encoder

The bi-encoder fine-tunes two BERT models $BERT_{m}$ and $BERT_{e}$ to embed
both mentions and entities, resp., into a dense embedding space, so that given
a new mention, the nearest candidates can be easily retrieved. For an entity
mention $d_{m}$ and an entity $d_{e}$, their embeddings can be accessed as

$v_{m}=\text{red}(BERT_{m}(d_{m}));\hskip
8.5359ptv_{e}=\text{red}(BERT_{e}(d_{e}))$ (1)

where $\text{red}(\cdot)$ denotes a function for extracting the vector
representation of a textual sequence from a BERT model with its last layer
representation of the [CLS] token.

Mentions and entities can be fed into the BERT models in different ways. In
the classic work Wu et al. (2020), a mention is fed into $BERT_{m}$ as [CLS]
ctxtl [Ms] mention [Me] ctxtr [SEP], where ctxtl and ctxtr are the left and
right contexts of the mention in the document; and an entity (with name and
definition) is fed into $BERT_{e}$ as [CLS] name [ENT] definition [SEP]. Note
that [Ms], [Me], and [ENT] are special tokens for separation.

With these embeddings, the score which measures the similarity between a
mention and an entity can be calculated, e.g., by their dot-product defined as
$s(m,e)=v_{m}\cdot v_{e}$. The scores are further used to generate top-$k$
candidates for each mention.

The bi-encoder is often trained using a cross-entropy loss (or a non-
temperatured contrastive loss cf. Gao et al. (2021)) which is to make each
mention close to its matched entity in the embedding space, but far away from
the other entities within the batch. An alternative loss is the max-margin
triplet loss described below, where $\alpha$ is a margin of small value (e.g.,
0.2) and $[x]_{+}$ denotes $\text{max}(x,0)$, for each mention to its gold
entity (the $i$-th) in a batch.

$L_{m_{i},e_{i}}=\sum_{j\neq i}{[\alpha-s(m_{i},e_{i})+s(m_{i},e_{j})]_{+}}$
(2)

#### Candidate Ranking with Cross-encoder

The cross-encoder is fed with information of both a mention and its top-$k$
entities from the bi-encoder which are ranked according to their scores to the
mention, and performs a multi-class classification, i.e., the entity that is
more likely to be matched with the mention is predicted with a higher score.

Each mention $m$ is concatenated with each of its top-$k$ entities following
the same input format as in the bi-encoder. The concatenation of the input for
mention and the entity $e$ (without the [SEP] and [CLS] tokens in between) is
denoted as $d_{m,e}$, i.e., [CLS] ctxtl [Ms] mention [Me] ctxtr [SEP] name
[ENT] definition [SEP]. The input $d_{m,e}$ is fed into the cross-encoder,
which is composed of a BERT model and a linear layer, for a score output:

$\begin{split}v^{(cross)}_{m,e}&=\text{red}(BERT_{cross}(d_{m,e}))\\\
s^{(cross)}_{m,e}&=v^{(cross)}_{m,e}W\end{split}$ (3)

The cross-encoder is learned with the following cross-entropy loss with
softmax activation, where $E_{cand}$ denotes the indices of the top-$k$
candidate entities of $m$, and $i$ is the index of the gold entity.

$L^{(cross)}_{m,e_{i}}=-\text{log}(\frac{\text{exp}(s^{(cross)}_{m,e_{i}})}{\sum_{j\in
E_{cand}}\text{exp}(s^{(cross)}_{m,e_{j}})})$ (4)

##  3 BLINKout

The BERT-based EL Wu et al. (2020) described previously in Section 2.2 does
not target the NIL entity. We present two approaches for out-of-KB mention
discovery that adapts the cross-encoder, including (i) threshold-based
approach, and (ii) NIL representation & classification, applicable either with
or without training. For the second approach, we further discuss the
techniques to represent the NIL entity with LMs. Then, we propose methods to
enhance both bi-encoder and cross-encoder with synonyms or variants which are
prevalent for entities in specific domains.

Figure 1: BLINKout architecture for out-of-KB mention discovery, adapting
BERT-based Entity Linking Wu et al. (2020): bi-encoder encodes separately the
mention $m$ in a context and the entities (each synonym as an entity) into a
dense embedding space; cross-encoder classifies the most relevant entity
candidate (with synonym concatenated), with NIL Representation &
Classification that appends a [NIL] special token to replace the last
candidate (if NIL was not predicted by the bi-encoder), jointly learned with
$L_{NIL}$ to classify whether the mention is out-of-KB.

###  3.1 Threshold-based Approach

The idea of the threshold approach is to determine whether a mention should be
linked to NIL, by setting a threshold for the EL system that outputs a score
for each entity candidate. Usually, the output score of the EL system is
normalized to [0,1], e.g, by a softmax function, and the threshold is thus set
between 0 and 1. The threshold is usually tuned as a hyperparameter with the
validation set Ji et al. (2011); Chen et al. (2021).

We adapt the BERT-based EL with a threshold $Th_{cross}$ for the cross-
encoder. We replace the last candidate with NIL (or the $k$-th) for $m$ if NIL
is not predicted in the top-$k$ candidates based on the bi-encoder, similar to
Rao et al. (2013). During candidate ranking, a mention is considered NIL if
all the non-NIL candidates have their scores below the threshold. We also
experimented with adding a threshold for the bi-encoder, which did not affect
the results as NIL is always included in the top-$k$ candidates.

###  3.2 Classification-based Approach

For the classification approach, we ensure that the NIL entity is within the
top-$k$ candidates by replacing the last candidate with NIL, as in the
threshold-based approach. This gives a chance for the cross-encoder with
$L^{(cross)}$ to potentially classify NIL as the top entity for the mention,
as inspired by Rao et al. (2013). The textual representation of NIL, encoded
with BERT-like LMs, will affect the performance of classification; we discuss
different NIL representations in 3.3.

#### Joint Training for NIL Classification

We add a joint loss to learn to verify whether a mention is NIL or in-KB,
conditioned on the mention representation, similar to Gu et al. (2021a). The
sigmoid function $\sigma$ with a linear layer is used to form a probability
score given the mention representation.

$\begin{split}v^{(cross)}_{m}&=\text{red}(BERT_{cross}(d_{m}))\\\
s^{isNIL}_{m}&=\sigma(v^{(cross)}_{m}W_{NIL})\end{split}$ (5)

This score, on whether the mention is NIL or in-KB, is learned with a binary
cross entropy loss.

$L^{NIL}_{m}=-y_{i}\text{log}(s^{isNIL}_{m})+(1-y_{i})\text{log}(1-s^{isNIL}_{m})$
(6)

Thus, the overall loss with joint training for the cross-encoder is described
below. The best value of $\lambda_{NIL}$ varied between 0.05 to 0.25 among the
datasets after the parameter tuning with the validation set.

$L_{m}=L^{(cross)}_{m,e_{i}}+\lambda_{NIL}*L_{NIL_{m}}$ (7)

###  3.3 NIL Entity Representations

The NIL entity does not have a native textual representation in the KB. A
better representation of NIL will help the LM to discriminate NIL from in-KB
entities. This representation can be either static, fixed (or unsupervised) or
dynamic, fine-tuned (or supervised with NIL mentions) in the LM.

We represent NIL as a special token [NIL] by taking advantage of the tokeniser
in a BERT-like LM. This assigns special semantics to the NIL entity so it is
not confused with the names of other entities in the KB. Also, the continuous
representation of [NIL] can be further fine-tuned with the LM. A more naive
representation is “NIL”, with the definition of “It is a NIL option.”, used in
a previous study Gu et al. (2021a); we also replace NIL with [NIL] in the
definition. Similar to in-KB entities, we add the [ENT] special token between
the name and the definition. A list of NIL representations and their results
are presented in Table 4. Our final approach uses the dynamic, fine-tuned
[NIL] representation that leverages the labelled out-of-KB mentions in the
training data.

###  3.4 Synonym Enhancement

The original BLINK (Wu et al., 2020) focuses on the general domain Wikipedia
and does not consider synonyms. Synonyms are prevalent for real-world entities
especially in the biomedical domain D’Souza and Ng (2015), for example, entity
C0428977 in the UMLS has the name of “Bradycardia” and some synonyms such as
“Slow heart beat” and “Heart rare slow”. When in-KB entities are better
represented with synonyms, the out-of-KB mentions can be more precisely
discriminated by the LM. We thus enhance the bi-encoder and the cross-encoder
with synonyms, with two different approaches, resp.: (i) augmentation of each
synonym as an entity in the bi-encoder, for its thorough training, and (ii)
concatenation with [SYN] special token in the cross-encoder, for its efficient
training.

Synonym Augmentation in Bi-encoder. We use synonyms for data augmentation to
enhance the training. Each synonym is treated as a separate entity to be
matched to the mention. This can significantly augment the size of the
training data. After the scoring, we aggregate the entities and synonyms into
top-$k$ unique entity candidates by setting the score w.r.t. an entity with
its highest score among all the variations.

Synonym Concatenation in Cross-encoder. We model each entity candidate $e$
with a concatenation of synonyms with the [SYN] special tokens, i.e. [CLS]
name [ENT] synonym_1 [SYN] synonym_2 ... synonym_n [SYN] definition.222We also
tested synonym concatenation for the bi-encoder, but it did not outperform
synonym augmentation. This keeps the number of entities to classify to $k$
compared to treating each synonym as an entity, and thus significantly reduces
the computation while still making full use of the synonyms.

Finally, we use BLINKout to refer to the approach that integrates BERT-based
EL with synonym enhancement, fine-tuned NIL representation for classification.
The threshold-based approach will be considered as a baseline.

##  4 Datasets

| ShARe/CLEF 2013 | MM-pruned-0.1 | MM-pruned-0.2 | MM-2014AB
---|---|---|---|---
| train / test | train / valid / test | train / valid / test | train / valid / test
# of docs | 199 / 99 | 2635 / 878 / 879 | 2635 / 878 / 879 | 2635 / 878 / 879
# of mentions | 5816-68 / 5351-24 | 6201 / 2121 / 2000 | 6201 / 2121 / 2000 | 6181 / 2112 / 1988
# of out-of-KB mentions | 1639 / 1723 | 456 / 201 / 155 | 1031 / 405 / 415 | 307 / 82 / 103
% of out-of-KB mentions | 28.2% / 32.3% | 7.3% / 9.5% / 7.8% | 16.6% / 19.1% / 20.8% | 5.0% / 3.9% / 5.2%
# of in-KB mentions | 4109 / 3604 | 5745 / 1920 / 1845 | 5170 / 1716 / 1585 | 5874 / 2030 / 1885
# of entities | 88,150 | 35,392 | 31,460 | 35,398
# of entities & synonyms | 288,490 | 126,188 | 112,097 | 124,132
Table 1: Statistics of out-of-KB linking datasets. Slashes separate the
statistics of training, validation, testing sets. “MM” denotes Medmentions;
pruned-0.1 and pruned-0.2 denote percentage of pruned concepts in the
ontology; “2014AB” refers to the previous version of the UMLS applied.

There are different strategies to construct NIL-enhanced EL datasets, which
contain out-of-KB mentions labelled with NIL (i.e., each mention is linked to
either an entity in $E$ or NIL). One straightforward way is Manual Labelling.
We adopt one existing manually NIL-labelled dataset in the medical domain,
i.e., ShARe/CLEF 2013, which consists of clinical notes (discharge summaries
and electrocardiogram, echocardiogram, and radiology reports) in the version
2.5 of the MIMIC-II dataset Suominen et al. (2013). The target ontology is
SNOMED CT (represented with their mapped UMLS CUI) which belongs to the
Disorder semantic group of 10 semantic types in the UMLS, defined in the
annotation guideline Elhadad et al. (2013). We use UMLS 2012AB following Ji et
al. (2020).333A small portion of mentions (68+24) in ShARe/CLEF 2013 were not
recognisable in UMLS 2012AB by string matching of CUIs, as shown in Table 1.
Around 30% of the mentions are out-of-KB.444The out-of-KB mentions can be out
of the Disorder semantic group but belong to another semantic type (e.g.
“[Thick] bronchial secretions” only as body substance; “soft tissue swelling”
and “vomiting coffee grounds” only as findings) Elhadad et al. (2013).

Manual labelling costs labour and there are few labelled datasets with NIL. We
thus propose two strategies, Ontology Pruning and Ontology Versioning, to
synthesise out-of-KB mentions within normal EL datasets.

#### Ontology Pruning

We randomly sample a portion (10% or 20%) of the entities in the target
ontology, remove them and link the parent and child of each removed entity to
preserve the hierarchy as in He et al. (2022). This forms a new ontology.
Mentions that are originally linked to the removed entities are labelled with
NIL.

Figure 2: Strategies for NIL-enhanced EL dataset construction: Manual
Labelling, Ontology Pruning, and Ontology Versioning. The specified UMLS
ontologies are the subset linked to SNOMED CT under the semantic type of T047,
“Disease or Syndrome”.

#### Ontology Versioning

We also consider replacing the current version of the ontology with an older
version, so that mentions linked to entities which are in the newer version
but not in the older version become out-of-KB. Using MedMentions dataset Mohan
and Li (2018) as an example (see Figure 2), UMLS2017AA was used to
exhaustively annotate entities in publication titles and abstracts in 2016; we
chose its older version of UMLS2014AB (released in Nov 2014). We narrowed the
UMLS semantic type to T047, Disease or Syndrome, and the source to those in
SNOMED CT (similar to ShARe/CLEF 2013). We gathered the entities that are in
UMLS2017AA but not in the earlier version555We further filtered out those that
were merged to a concept in the previous version of the ontology (as defined
in Retired CUI Mapping, MRCUI.RRF in the UMLS), representing around 6% of all
newly added concepts.. The mentions labelled with the newly added entities
become out-of-KB mentions666Examples include “CJS” or “Curry-Jones syndrome”
(C0795915), “CKD” or “Chronic Kidney Diseases” (C1561643), and “Pandemic
influenza” (C4304383), which were added to the UMLS between Nov 2014 and May
2017 under the semantic type T047, Disease or Syndrome..

The advantage of Ontology Pruning is that it allows the control of the process
and the percentage of out-of-KB mentions, while Ontology Versioning creates
datasets that are more close to the reality where new concepts emerge over
time. The statisics of our datasets are shown in Table 1.

##  5 Evaluation

###  5.1 Comparison Methods

The main baseline methods for out-of-KB mention discovery are: (i) Rule-based
approach, the Sieve-based approach, using carefully crafted rules designed for
biomedical texts, where NIL is detected when no entity can be linked to
D’Souza and Ng (2015). (ii) Threshold-based approach (“$Th$-based
BM25+BERT+syn”), with BM25 for candidate generation and BERT-based cross-
encoder for candidate ranking Ji et al. (2020), with synonym enhancement; we
tuned the threshold (as 0.85) and the domain-specific, SapBERT Liu et al.
(2021) setting (“$Th$-based BM25+SapBERT+syn”) based on the validation set.
(iii) Feature-based NIL classification  (“Ft-based Classifer”): including
string-matching features Rao et al. (2013); Dredze et al. (2010); McNamee et
al. (2009), entity contextual feature space and embedding-based feature space
Wu et al. (2016), with details in Appendix B; we also provide a feature-based
baseline that uses the entity candidates and dynamic features in BLINKout
(“Ft-based BLINKout $L^{(Ft)}_{NIL}$”), see Appendix B.1.

For model comparison, the default number of top-$k$ was set as 10 for the
baselines using BM25, BLINK, or BLINKout, following Ji et al. (2020). In the
“$k$ and NIL rep tuned” setting, both $k$ and NIL representations were tuned
based on the validation set to optimise $F_{1_{o}}$ (see Section 5.4 and
Appendix C for details in parameter tuning, with the optimal settings in
Appendix A).

#### Adapting BLINK without Training

We also implemented both threshold-based and the NIL representation approach
to a BLINK model trained from all in-KB entities (“$Th$-based BLINK” and “NIL-
rep-based BLINK”), either or not enhanced with synonyms (“+syn”).

###  5.2 Evaluation Metrics

In the conventional setting for EL, which does not discriminate between in-KB
and out-of-KB mentions, the overall (or in-KB$+$out-of-KB) accuracy ($A$),
precision ($P$), recall ($R$), and $F_{1}$ scores are the same, i.e.,
$A=P=R=F_{1}=\frac{TP}{|M|}$, where $TP$ and $|M|$ denote the number of
correctly linked mentions and all the target mentions, resp. Shen et al.
(2014); Sevgili et al. (2022).

In this work, we propose to use out-of-KB precision ($P_{o}$), recall
($R_{o}$), and $F_{1}$ ($F_{1_{o}}$) scores to measure how well out-of-KB
mentions are detected. Analogously, we can calculate the in-KB precision
($P_{in}$), recall ($R_{in}$), and $F_{1_{in}}$ scores.

$P_{o}=\frac{TP_{o}}{TP_{o}+FP_{o}};R_{o}=\frac{TP_{o}}{TP_{o}+FN_{o}}$ (8)

$P_{in}=\frac{TP_{in}}{TP_{in}+FP_{in}};R_{in}=\frac{TP_{in}}{TP_{in}+FN_{in}}$
(9)

where $TP_{o}$, $FP_{o}$, and $FN_{o}$ (resp. $TP_{in}$, $FP_{in}$, and
$FN_{in}$) are the number of true positive, false positive, and false negative
out-of-KB (resp. in-KB) mentions. Note that $TP_{in}$ refers to mentions that
are predicted with the correct entities in the KB instead of simply predicted
as in-KB. $F_{1}$ score is the harmonic mean of the precision and recall.

###  5.3 Results

| ShARe/CLEF 2013 | MM-pruned-0.1 | MM-2014AB
---|---|---|---
| $A$ | $P_{o}$ | $R_{o}$ | $F_{1_{o}}$ | $A$ | $P_{o}$ | $R_{o}$ | $F_{1_{o}}$ | $A$ | $P_{o}$ | $R_{o}$ | $F_{1_{o}}$
Rule-based (Sieve-based) | 90.0 | 84.7 | 92.6 | 88.5 | 83.8 | 36.3 | 94.8 | 52.5 | 82.9 | 26.6 | 98.1 | 41.9
Ft-based Classifier | 36.0 | 49.1 | 71.6 | 58.2 | 19.1 | 11.2 | 69.7 | 19.2 | 20.1 | 7.2 | 37.9 | 12.1
Ft-based BLINKout $L^{(Ft)}_{NIL}$ | 60.9 | 61.4 | 95.0 | 74.6 | 42.3 | 14.0 | 92.3 | 24.2 | 43.7 | 8.9 | 85.4 | 16.1
$Th$-based BM25+BERT+syn | 69.5 | 64.9 | 55.8 | 60.0 | 72.3 | 22.8 | 64.5 | 33.7 | 70.6 | 9.7 | 26.2 | 14.1
$Th$-based BM25+SapBERT+syn | 76.2 | 71.3 | 64.7 | 67.8 | 75.6 | 28.6 | 84.7 | 42.2 | 71.4 | 16.9 | 81.6 | 28.0
BLINK (BERT) | 53.0 | - | - | - | 67.0 | - | - | - | 62.4 | - | - | -
\+ syn | 58.1 | - | - | - | 70.4 | - | - | - | 68.8 | - | - | -
$Th$-based BLINK | 72.4 | 65.4 | 67.7 | 66.5 | 68.1 | 21.9 | 42.6 | 28.9 | 65.7 | 21.7 | 74.8 | 33.6
\+ syn | 80.1 | 74.4 | 76.2 | 75.3 | 72.7 | 25.5 | 54.8 | 34.8 | 71.1 | 22.2 | 48.5 | 30.5
NIL-rep-based BLINK | 74.8 | 67.6 | 89.2 | 76.9 | 69.7 | 24.8 | 47.1 | 32.5 | 65.7 | 19.1 | 80.6 | 30.9
\+ syn | 80.3 | 79.8 | 73.5 | 76.5 | 72.6 | 27.8 | 40.7 | 33.0 | 71.4 | 18.6 | 74.8 | 29.8
BLINKout (- syn, BERT) | 77.7 | 74.1 | 91.2 | 81.8 | 62.3 | 18.4 | 62.6 | 28.5 | 68.2 | 25.2 | 78.6 | 38.1
BLINKout (BERT) | 86.0 | 85.3 | 87.9 | 86.6 | 70.5 | 33.1 | 58.7 | 42.3 | 69.9 | 33.7 | 84.5 | 48.2
BLINKout (SapBERT) | 89.9 | 90.0 | 89.8 | 89.9 | 81.8 | 53.5 | 54.8 | 54.1 | 81.5 | 57.3 | 79.6 | 66.7
\+ $k$ and NIL rep tuned | 90.8 | 91.4 | 90.3 | 90.9 | 83.3 | 58.4 | 65.2 | 61.6 | 82.9 | 86.2 | 72.8 | 79.0
\+ $L_{NIL}$ | 91.2 | 88.5 | 92.5 | 90.5 | 83.1 | 47.0 | 75.5 | 57.9 | 82.5 | 59.3 | 83.5 | 69.4
Table 2: Comparison results for Entity Linking with out-of-KB mentions (Precision, Recall, $F_{1}$ scores w.r.t. out-of-KB mentions and overall Accuracy w.r.t. both in- and out-of-KB mentions). | MM-pruned-0.2
---|---
| A | $P_{o}$ | $R_{o}$ | $F_{1_{o}}$
Rule-based (Sieve-based) | 85.3 | 63.4 | 97.1 | 76.7
Ft-based Classifier | 25.8 | 27.5 | 63.9 | 38.4
Ft-based BLINKout $L^{(Ft)}_{NIL}$ | 48.0 | 39.3 | 95.9 | 55.7
$Th$-based BM25+BERT+syn | 70.2 | 46.2 | 60.7 | 52.5
$Th$-based BM25+SapBERT+syn | 74.7 | 51.3 | 72.5 | 60.1
BLINK (BERT) | 57.2 | - | - | -
\+ syn | 60.2 | - | - | -
$Th$-based BLINK | 71.1 | 51.6 | 77.6 | 62.0
\+ syn | 69.5 | 47.7 | 60.5 | 53.4
NIL-rep-based BLINK | 71.9 | 56.9 | 74.9 | 64.7
\+ syn | 81.0 | 54.4 | 77.1 | 61.7
BLINKout (- syn, BERT) | 77.4 | 51,1 | 80.7 | 62.6
BLINKout (BERT) | 81.4 | 57.8 | 74.0 | 64.9
BLINKout (SapBERT) | 84.1 | 72.6 | 81.0 | 76.5
\+ $k$ and NIL rep tuned | 88.9 | 72.6 | 88.7 | 79.8
\+ $L_{NIL}$ | 90.3 | 65.2 | 91.6 | 76.2
Table 3: Comparison results for Entity Linking with out-of-KB mentions on MM-
pruned-0.2 (Precision, Recall, $F_{1}$ scores w.r.t. out-of-KB mentions and
overall Accuracy w.r.t. both in- and out-of-KB mentions)

#### Out-of-KB Metrics

BLINKout (with the domain-specific SapBERT) achieved the top out-of-KB $F_{1}$
($F_{1_{o}}$) on all datasets, see main results in Table 2 and MM-pruned-0.2
results in Table 3.

Rule-based approaches were effective, given the regular syntactic structure of
the biomedical texts and concepts. Sieve-based approach D’Souza and Ng (2015)
resulted in high out-of-KB recall scores, but still lack in out-of-KB
precision and $F_{1_{o}}$ compared to BLINKout. The rule-based approach tends
to result in a large number of out-of-KB mentions using the “NIL as no
candidate” rule.

Feature-based methods, facing the challenge of imbalanced binary
classification, also tend to predict a larger number of NILs and result in a
high out-of-KB recall, but low precision and $F_{1}$ scores.

Threshold-based approaches provide higher out-of-KB precision than feature-
based, but lower metrics than the rule-based approach. The setting using
synonyms with bi-encoder (“BLINK+syn”) instead of BM25 (“BM25+BERT+syn”)
greatly improved the in-KB results and thus the out-of-KB results based on a
threshold.

NIL-rep-based approaches with fixed NIL representations performed better than
or on par with threshold-based ones. The special token [NIL], even fixed or
unsupervised, can leverage the pre-trained weights in LMs to discriminate NIL
from any in-KB entities.

BLINKout outperforms all the baselines on all the datasets. A large margin was
achieved compared to the second best, rule-based approach, especially for
MM-2014AB and MM-pruned-0.1, by nearly 40% and 10% improvement of $F_{1_{o}}$,
resp.

#### Ablation Study

The contributing components in BLINKout are: (i) fine-tuned NIL representation
with NIL-labelled training data (3-17% improvement of $F_{1_{o}}$); (ii)
synonym enhancement (2-14% improvement of $F_{1_{o}}$); (iii) domain-specific
BERT, e.g. SapBERT (3-19% improvement of $F_{1_{o}}$).

#### Overall Accuracy

For the overall accuracy on both in-KB and out-of-KB mentions ($A$), the
proposed models perform the best or competitively in 3 out of 4 datasets
(except for MM-pruned-0.1, slightly below the rule-based approach by 0.5%).
This shows that BLINKout, which focuses on out-of-KB mentions, does not
compromise the performance of in-KB entities. In-KB EL results are in Table 5
in Appendix E.

#### Qualitative Analysis

Examples in Table 7 with analysis in Appendix F on the datasets further
suggest that BLINKout can identify out-of-KB mentions with higher confidence,
compared to $Th$-based and NIL-rep-based methods on BLINK.

###  5.4 Sensitive Settings

The results of out-of-KB mention discovery with BLINKout are sensitive to the
settings below.

#### The Number of Top-k Candidates

Out-of-KB metric scores generally have a positive correlation with $k$ when it
is below a limit. A higher top-$k$ may make it harder to train a model to
discriminate out-of-KB from the remaining classes using the cross-encoder;
this especially decreases the performance when there are limited out-of-KB
samples for training (e.g., in MM-2014AB). In-KB results and bi-encoder
results positively correlate with $k$. Details and figures are in Appendix C.

#### NIL Textual Representation

NIL representation sets a prior anchor point in the embedding space for the
bi-encoder and is used in the cross-encoder for candidate ranking. Results on
ShARe/CLEF 2013 are displayed in Table 4: [NIL] representation performed the
best for all metrics, either in the fixed or the fine-tuned setting.

| ShARe/CLEF 2013
---|---
| A | $F_{1_{o}}$ | $F_{1_{in}}$
Fixed rep: |  |  |
NIL-rep-based BLINK+syn, $k$=10 |  |
NIL | 70.0 | 52.3 | 75.4
NIL [ENT] It is a NIL option. | 70.0 | 52.3 | 75.4
[NIL] | 80.3 | 76.5 | 82.0
[NIL] [ENT] It is a NIL option. | 74.5 | 64.9 | 78.0
[NIL] [ENT] It is a [NIL] option. | 76.5 | 69.2 | 79.4
Fine-tuned rep: |  |  |
BLINKout (BERT), $k$=150 |  |  |
[NIL] | 88.7 | 89.1 | 88.5
[NIL] [ENT] It is a [NIL] option. | 87.6 | 88.2 | 87.2
Table 4: Comparison results among NIL representations (“rep”), either fixed or
fine-tuned

#### Domain-Specific LMs

In-domain and ontology-pre-trained models, SapBERT Liu et al. (2021) and
PubMedBERT Gu et al. (2021b), obtained better in-KB and out-of-KB results on
the datasets. Results are in Appendix D.

##  6 Discussion

#### Results w.r.t Dataset Construction Strategies

The out-of-KB performance of BLINKout was highest on the Manual Labelling
data, ShARe/CLEF 2013, and lowest on the Ontology Versioning data, MM-2014AB.
This is partially related to the percentage of out-of-KB mentions: the lower
the percentage in the training data, the more challenging their detection. The
randomly pruned entities may be very similar to the in-KB entities, thus
difficult to discriminate among them. On the more realistic Ontology
Versioning dataset, BLINKout outperforms the rule-based approach with a larger
gap compared to datasets created with Manual Labelling or Ontology Pruning.

#### NIL Representation

The results show that NIL representation based methods perform better or on
par with the threshold-based method. BLINKout with the fine-tuned [NIL]
representation yielded a large margin of improvement. We also show that it is
safer to use non-natural language tokens, i.e., [NIL], rather than “NIL” to
avoid confusion with other vocabularies, especially in the fixed setting.

#### Joint Learning

Using $L_{NIL}$ allows to penalise the situation that a NIL mention being
wrongly classified as an in-KB entity. This tends to increase the out-of-KB
recall $R_{o}$, but may harm the precision $P_{o}$ and $F_{1_{o}}$ scores, as
shown in Tables 2 and 3. Joint learning obtained the best overall accuracy on
2 of the 4 datasets (ShARe/CLEF 2013 and MM-pruned-0.2) where NIL mentions
represented a higher percentage among all mentions. Improving jointing
learning in the highly-imbalanced setting (e.g. MM-2014AB) warrants further
studies.

#### Synonyms Enhancement

Synonyms help address the entity variant problem and further differentiate in-
KB entities from NIL after the fine-tuning. Our synonym enhancement on the bi-
encoder and cross-encoder greatly improved the out-of-KB $F_{1}$ scores across
datasets over BLINK.

##  7 Related Work

#### Entity Linking (EL) for KB Construction

EL has been extensively studied, see reviews in Shen et al. (2014, 2021);
Sevgili et al. (2022). The work in Rao et al. (2013) focuses on a KB-centric
view of EL, i.e., EL as a key step in KB construction and maintenance. Most
relevant to KB maintenance is the identification of out-of-KB or NIL mentions
so that they can be placed into the KB Rao et al. (2013). Many recent studies
in EL only considered in-KB entities Bhowmik et al. (2021); Wu et al. (2020);
Basaldella et al. (2020). Out-of-KB mention discovery is also distinct from
zero-shot EL Wu et al. (2020); Ristoski et al. (2021), as the latter targets
in-KB entities.

#### Out-of-KB Mention and Entity Discovery

Most studies apply the traditional threshold-based approach Bunescu and Paşca
(2006); Ji et al. (2020); Chen et al. (2021); Xu and Miller (2022) or NIL
classification with features Wu et al. (2016); Rao et al. (2013); Zhang et al.
(2020) and neural networks Martins et al. (2019); Gu et al. (2021a). We
proposed NIL representation & classification that utilises the embedding
representation in LMs, with BERT-based EL Wu et al. (2020).

Another group of works forms NIL clusters for the resolution of out-of-KB
mentions into potential concepts or entities, e.g. recently in Kassner et al.
(2022) and Agarwal et al. (2021). The clusters are formed mainly based on the
affinity of the out-of-KB mentions to themselves and to the in-KB entities,
and NIL clusters are those having no entities but only mentions. A threshold
is used to decide whether a mention or entity can be included in a cluster,
similar to the threshold-based baseline. NIL clustering algorithms can be
added as a part of the pipeline after out-of-KB mention detection, as
summarised in Ji et al. (2011). The study Sil and Florian (2016) also clusters
NIL mentions, after they are identified using a feature-based approach. Thus,
our methods on NIL classification and representation are independent of
clustering methods, and we leave the combination and comparison with them for
a future study.

#### Out-of-KB EL Benchmarking

In the general domain, TAC 2011 Knowledge Base Population Track Ji et al.
(2011) and NEEL challenge Rizzo et al. (2017) were created to identify new
entities from texts in news and social media texts. Recent general domain
datasets include NILK Iurshina et al. (2022) and EDIN-benchmark Kassner et al.
(2022) to enrich older versions of Wikipedia and Wikidata using text mentions
from newer Wikipedia articles or news articles.

Domain-specific datasets for out-of-KB mentions are relatively rare, where the
main dataset in the biomedical domain is ShARe/CLEF 2013 Suominen et al.
(2013). The study of Ruas and Couto (2022) uses biomedical ontologies, but the
task links an identified NIL mention to its closest parent entity, instead of
discovering the NIL mention.

Regarding evaluation metrics, overall accuracy Ji et al. (2020); Chen et al.
(2021); Gu et al. (2021a) is commonly used, which does not reflect the full
picture. As far as we know, our study is the first to apply out-of-KB
Precision, Recall, and $F_{1}$ scores.

One important contribution of our work is to provide general strategies to
construct out-of-KB EL datasets. The research endeavour that mostly resembles
ours is the ongoing work of Möller (2022) on EL for KB enrichment. This study
so far targets news events to enrich Wikipedia. Similar to the plans in Möller
(2022), our future work will canonicalise the new mentions by e.g., grouping
and naming, and place them in the KB.

##  8 Conclusion

We introduced the task of out-of-KB mention discovery from texts and proposed
BLINKout, which utilises a dynamic NIL representation & classification
approach, enhanced with synonyms, founded on BERT-based Entity Linking. We
also provided strategies, Ontology Pruning and Ontology Versioning, to
construct out-of-KB datasets. While the datasets are from the biomedical
domain, the methods are directly applicable to other domains. Future studies
will focus on the canonicalisation and placement of out-of-KB entities in the
KB.

## Limitations

In this work, we did not further canonicalise the discovered out-of-KB
mentions and place them into the KB. We mainly focused on the biomedical
domain considering the importance of biomedical ontologies and the lack of
data for the domain for out-of-KB mentions. For training bi-encoder, we used
random negatives and explored online mining of hard triplets in Musgrave et
al. (2020) instead of hard negatives. Also, our method did not leverage the
logic or graph information (e.g. parents, children, and other relations or
axioms) besides synonymy relations in the KBs or ontologies. The results were
conducted with a fixed random seed, similar to Wu et al. (2020), while the
variances were small (e.g. around or below 1<% in our experiments for BLINKout
on MM-2014AB, $k$=10). For in-KB entities, the rule-based approach, e.g.
Sieve-based D’Souza and Ng (2015), tailored for biomedical texts, achieved the
best results, and this needs to be verified with texts from other domains. We
will explore these aspects for future studies.

## Ethics Statement

Part of this work used the de-identified clinical notes in ShARe/CLEF 2013. We
completed the Collaborative Institutional Training Initiative (CITI) Program’s
“Data or Specimens Only Research”
course777https://physionet.org/content/shareclefehealth2013/view-required-
trainings/1.0/#1 and signed the data use agreement to get access to the data.
We strictly followed the guidelines to use the data.

## References

  * Agarwal et al. (2021) Dhruv Agarwal, Rico Angell, Nicholas Monath, and Andrew McCallum. 2021.  Entity linking and discovery via arborescence-based supervised clustering.  _arXiv preprint arXiv:2109.01242_.
  * Basaldella et al. (2020) Marco Basaldella, Fangyu Liu, Ehsan Shareghi, and Nigel Collier. 2020.  COMETA: A corpus for medical entity linking in the social media.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3122–3137, Online. Association for Computational Linguistics.
  * Bhowmik et al. (2021) Rajarshi Bhowmik, Karl Stratos, and Gerard de Melo. 2021.  Fast and effective biomedical entity linking using a dual encoder.  In _Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis_ , pages 28–37, online. Association for Computational Linguistics.
  * Bunescu and Paşca (2006) Razvan Bunescu and Marius Paşca. 2006.  Using encyclopedic knowledge for named entity disambiguation.  In _11th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 9–16, Trento, Italy. Association for Computational Linguistics.
  * Chen et al. (2021) Lihu Chen, Gaël Varoquaux, and Fabian M. Suchanek. 2021.  A lightweight neural model for biomedical entity linking.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(14):12657–12665.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dredze et al. (2010) Mark Dredze, Paul McNamee, Delip Rao, Adam Gerber, and Tim Finin. 2010.  Entity disambiguation for knowledge base population.  In _Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)_ , pages 277–285, Beijing, China. Coling 2010 Organizing Committee.
  * D’Souza and Ng (2015) Jennifer D’Souza and Vincent Ng. 2015.  Sieve-based entity linking for the biomedical domain.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 297–302, Beijing, China. Association for Computational Linguistics.
  * Elhadad et al. (2013) Noémie Elhadad, Wendy Chapman, and Guergana Savova. 2013.  Shareclef ehealth 2013: Natural language processing and information retrieval for clinical care 1.0.  https://physionet.org/content/shareclefehealth2013/1.0/Task1ShAReGuidelines2013.pdf.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gruber (1995) Thomas R. Gruber. 1995.  Toward principles for the design of ontologies used for knowledge sharing? _International Journal of Human-Computer Studies_ , 43(5):907–928.
  * Gu et al. (2021a) Yingjie Gu, Xiaoye Qu, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Xiaolin Gui. 2021a.  Read, retrospect, select: An mrc framework to short text entity linking.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(14):12920–12928.
  * Gu et al. (2021b) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021b.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Trans. Comput. Healthcare_ , 3(1).
  * He et al. (2022) Yuan He, Jiaoyan Chen, Hang Dong, Ernesto Jiménez-Ruiz, Ali Hadian, and Ian Horrocks. 2022.  Machine learning-friendly biomedical datasets for equivalence and subsumption ontology matching.  In _The Semantic Web – ISWC 2022_ , pages 575–591, Cham. Springer International Publishing.
  * Hoffart et al. (2014) Johannes Hoffart, Yasemin Altun, and Gerhard Weikum. 2014.  Discovering emerging entities with ambiguous names.  In _Proceedings of the 23rd International Conference on World Wide Web_ , WWW ’14, page 385–396, New York, NY, USA.
  * Hoffart et al. (2016) Johannes Hoffart, Dragan Milchevski, Gerhard Weikum, Avishek Anand, and Jaspreet Singh. 2016.  The knowledge awakens: Keeping knowledge bases fresh with emerging entities.  In _Proceedings of the 25th International Conference Companion on World Wide Web_ , WWW ’16 Companion, page 203–206, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Iurshina et al. (2022) Anastasiia Iurshina, Jiaxin Pan, Rafika Boutalbi, and Steffen Staab. 2022.  Nilk: Entity linking dataset targeting nil-linking cases.  In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_, CIKM ’22, page 4069–4073, New York, NY, USA.
  * Ji et al. (2011) Heng Ji, Ralph Grishman, Hoa Trang Dang, Kira Griffitt, and Joe Ellis. 2011.  Overview of the tac 2011 knowledge base population track.  In _Third text analysis conference (TAC 2011)_ , pages 1–33.
  * Ji et al. (2020) Zongcheng Ji, Qiang Wei, and Hua Xu. 2020.  BERT-based ranking for biomedical entity normalization.  _AMIA Summits on Translational Science Proceedings_ , 2020:269.
  * Kassner et al. (2022) Nora Kassner, Fabio Petroni, Mikhail Plekhanov, Sebastian Riedel, and Nicola Cancedda. 2022.  EDIN: An end-to-end benchmark and pipeline for unknown entity discovery and indexing.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8659–8673, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Liu et al. (2021) Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. 2021\.  Self-alignment pretraining for biomedical entity representations.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4228–4238, Online. Association for Computational Linguistics.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Martins et al. (2019) Pedro Henrique Martins, Zita Marinho, and André F. T. Martins. 2019.  Joint learning of named entity recognition and entity linking.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , pages 190–196, Florence, Italy. Association for Computational Linguistics.
  * McNamee et al. (2009) Paul McNamee, Mark Dredze, Adam Gerber, Nikesh Garera, Tim Finin, James Mayfield, Christine Piatko, Delip Rao, David Yarowsky, Markus Dreyer, et al. 2009\.  Hltcoe approaches to knowledge base population at tac 2009.  In _Proceedings of the 2009 Text Analysis Conference_.
  * Mikolov et al. (2018) Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018.  Advances in pre-training distributed word representations.  In _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Mohan and Li (2018) Sunil Mohan and Donghui Li. 2018.  Medmentions: A large biomedical corpus annotated with umls concepts.  In _Automated Knowledge Base Construction (AKBC)_.
  * Möller (2022) Cedric Möller. 2022.  Knowledge graph population with out-of-kg entities.  In _European Semantic Web Conference_ , pages 199–214. Springer.
  * Musgrave et al. (2020) Kevin Musgrave, Serge J. Belongie, and Ser-Nam Lim. 2020.  Pytorch metric learning.  _ArXiv_ , abs/2008.09164.
  * Peng et al. (2019) Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019.  Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets.  In _Proceedings of the 18th BioNLP Workshop and Shared Task_ , pages 58–65, Florence, Italy. Association for Computational Linguistics.
  * Rao et al. (2013) Delip Rao, Paul McNamee, and Mark Dredze. 2013.  Entity Linking: Finding Extracted Entities in a Knowledge Base.  In Thierry Poibeau, Horacio Saggion, Jakub Piskorski, and Roman Yangarber, editors, _Multi-source, Multilingual Information Extraction and Summarization_ , pages 93–115. Springer Berlin Heidelberg, Berlin, Heidelberg.  Series Title: Theory and Applications of Natural Language Processing.
  * Ristoski et al. (2021) Petar Ristoski, Zhizhong Lin, and Qunzhi Zhou. 2021.  Kg-zeshel: Knowledge graph-enhanced zero-shot entity linking.  In _Proceedings of the 11th on Knowledge Capture Conference_ , K-CAP ’21, page 49–56, New York, NY, USA.
  * Rizzo et al. (2017) Giuseppe Rizzo, Bianca Pereira, Andrea Varga, Marieke Van Erp, and Amparo Elizabeth Cano Basave. 2017.  Lessons learnt from the named entity recognition and linking (neel) challenge series.  _Semantic Web_ , 8(5):667–700.
  * Ruas and Couto (2022) Pedro Ruas and Francisco M. Couto. 2022.  Nilinker: Attention-based approach to nil entity linking.  _Journal of Biomedical Informatics_ , page 104137.
  * Sevgili et al. (2022) Özge Sevgili, Artem Shelmanov, Mikhail Arkhipov, Alexander Panchenko, and Chris Biemann. 2022.  Neural entity linking: A survey of models based on deep learning.  _Semantic Web_ , 13(3):527–570.  Publisher: IOS Press.
  * Shen et al. (2021) Wei Shen, Yuhan Li, Yinan Liu, Jiawei Han, Jianyong Wang, and Xiaojie Yuan. 2021\.  Entity linking meets deep learning: Techniques and solutions.  _IEEE Transactions on Knowledge and Data Engineering_.
  * Shen et al. (2014) Wei Shen, Jianyong Wang, and Jiawei Han. 2014.  Entity linking with a knowledge base: Issues, techniques, and solutions.  _IEEE Transactions on Knowledge and Data Engineering_ , 27(2):443–460.
  * Sil and Florian (2016) Avirup Sil and Radu Florian. 2016.  One for all: Towards language independent named entity linking.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2255–2264, Berlin, Germany. Association for Computational Linguistics.
  * Suominen et al. (2013) Hanna Suominen, Sanna Salanterä, Sumithra Velupillai, Wendy W Chapman, Guergana Savova, Noemie Elhadad, Sameer Pradhan, Brett R South, Danielle L Mowery, Gareth JF Jones, et al. 2013.  Overview of the share/clef ehealth evaluation lab 2013.  In _International Conference of the Cross-Language Evaluation Forum for European Languages_ , pages 212–231. Springer.
  * Wigby et al. (2017) Kristen Wigby, Stephen RF Twigg, Ryan Broderick, Katherine P Davenport, Andrew OM Wilkie, Stephen W Bickler, and Marilyn C Jones. 2017.  Gastrointestinal disorders in curry–jones syndrome: Clinical and molecular insights from an affected newborn.  _American Journal of Medical Genetics Part A_ , 173(6):1586–1592.
  * World Health Organization (2022) World Health Organization. 2022.  Tracking sars-cov-2 variants.  https://www.who.int/activities/tracking-SARS-CoV-2-variants.
  * Wu et al. (2020) Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020.  Scalable zero-shot entity linking with dense entity retrieval.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6397–6407, Online. Association for Computational Linguistics.
  * Wu et al. (2016) Zhaohui Wu, Yang Song, and C. Giles. 2016.  Exploring multiple feature spaces for novel entity discovery.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 30(1).
  * Xu and Miller (2022) Dongfang Xu and Timothy Miller. 2022.  A simple neural vector space model for medical concept normalization using concept embeddings.  _Journal of Biomedical Informatics_ , 130:104080.
  * Zhang et al. (2020) Shuo Zhang, Edgar Meij, Krisztian Balog, and Ridho Reinanda. 2020.  Novel entity discovery from web tables.  In _Proceedings of The Web Conference 2020_ , WWW ’20, page 1298–1308, New York, NY, USA.

##  Appendix A Experimental Details and Hyperparameters

BLINKout was implemented using PyTorch Huggingface
Transformers888https://huggingface.co/docs/transformers/index, based on the
original BLINK library999https://github.com/facebookresearch/BLINK. The
BM25+BERT baseline is also based on the Python library
rank_bm25101010https://pypi.org/project/rank-bm25/. Results on the Sieve-based
approach were reproduced using the original JAVA-based
implementation111111https://github.com/jennydsuza9/disorder-normalizer. All
neural network models were trained using an NVIDIA Quadro RTX 8000 GPU card.

We used the max-margin triplet loss to train the bi-encoder. We found that the
triplet loss led to a more stable convergence than the cross-entropy loss,
with a similar performance to the latter.

We tuned the hyperparameters by optimising $F_{1_{o}}$ on the validation set.
We generally tuned the best $k$ from $\\{5,10,20,50,100,150,200\\}$ for each
dataset and selected the best NIL representation. The default $k$ for model
comparison was set as 10. The best BLINKout model had $k$ as 150 for
ShARe/CLEF 2013, 50 for MM-pruned-0.1 and MM-2014AB, and 100 for MM-
pruned-0.2. The NIL representation was [NIL] for the datasets except [NIL]
with [NIL]-definition for MM-pruned-0.1.

For the threshold-based BLINK method, the threshold $Th_{cross}$ was 0.95,
0.55, 0.8, 0.95 for the datasets ShARe/CLEF 2013, MM-2014AB, MM-pruned-0.1,
and MM-pruned-0.2, resp.

We followed the optimal parameter settings in Wu et al. (2020) to use a
maximum of 32 tokens for a mention in context and 128 for an entity candidate
(including its synonyms), learning rate as 3e-05, bi-encoder training batch
size as 32, cross-encoder training batch size as 1.

We used the “large” version of LM (e.g. BERT-large, 680M parameters) in the
bi-encoder and “base” version of LM (e.g. BERT-base, 110M parameters) in the
cross-encoder. For SapBERT and PubMedBERT, we used “base” version for the bi-
encoder as no “large” version is available.

The parameter in joint learning, $\lambda_{NIL}$, was set to 0.25 (as in Gu et
al. (2021a)) for ShARe/CLEF 2013 and MM-pruned-0.2 datasets, to 0.2 for MM-
pruned-0.1 and 0.05 for MM-2014AB.

We optimised the bi-encoder and cross-encoder using AdamW Loshchilov and
Hutter (2018), with 3 and 4 epochs, resp. and trained all models with fixed
random seeds to obtain reproducible results.

#### Time Used in Training and Inference

The training of bi-encoder with SapBERT on ShARe/CLEF 2013 data took approx.
0.63 hours; training cross-encoder with SapBERT took approx. 0.88 hours when
$k$=10 and approx. 8.67 hours when $k$=150. The inferencing stage is very fast
compared to training: inferencing with a new mention took approx. 0.06 and
0.57 seconds for $k$=10 and $k$=150, resp.

##  Appendix B Baseline: Feature-based Classification

A list of features for out-of-KB mention classification is presented below,
based on previous studies:

#### Mention-Entity String Matching

Whether a mention has no, one, or several matching names in the entities
through string matching (either exact full matching, exact inclusion, or fuzzy
matching), to either the entity’s name or the entity’s name and definition,
based on Rao et al. (2013) and McNamee et al. (2009). This resulted in 7
string matching features after the selection.

#### Neural Embeddings

Neural embedding space between mentions and entities Wu et al. (2016). We used
300-dimensional FastText embedding of 2 million word vectors trained on Common
Crawl121212The FastText embedding is available at
https://fasttext.cc/docs/en/english-vectors.html. Mikolov et al. (2018) to
represent both the mention and the entity, calculated their cosine similarity,
and reported the highest cosine similarity across all entities and synonyms
for the mention as a feature.

#### Entity Contexts

Contextual entity space in (Wu et al., 2016). We averaged the embeddings of
entities in the context of a mention (with the same context length of 32),
where the entity embeddings were from FastText as in the above. The entities
were located by exact string matching. The entity contextual embeddings formed
300 features.

For the “Ft-based Classifier” baseline, we followed the original studies by
using Support Vector Machine Rao et al. (2013) or an ensemble-based classifier
Wu et al. (2016), e.g., Random Forest, to classify whether a mention is NIL or
in-KB. The exact in-KB entity is determined in the candidate generation
process with four rules: exact string matching to the name, exact string
inclusion to the name, exact string matching to synonyms, and first-letter
matching. The top in-KB entity is the one with the highest cosine similarity
of the neural embeddings between the mention and the name (or the synonym) of
the entity.

We further provided another strong feature-based baseline (“Ft-based BLINKout
$L^{(Ft)}_{NIL}$”) by integrating dynamic and static features with the
BLINKout joint learning architecture and controlling the same top-$k$
candidates as BLINKout, see Section B.1.

Figure 3: Out-of-KB $F_{1}$, in-KB $F_{1}$, and overall accuracy w.r.t the number of top-$k$ candidates using BLINKout (SapBERT) on MM-2014AB validation set Figure 4: Recall@$k$ w.r.t the number of top-$k$ candidates using BLINKout (SapBERT) on MM-2014AB validation set, after adding out-of-KB into the candidates if they were not predicted within top-$k$ | ShARe/CLEF 2013 | MM-pruned-0.1 | MM-pruned-0.2 | MM-2014AB
---|---|---|---|---
| $P_{in}$ | $R_{in}$ | $F_{1_{in}}$ | $P_{in}$ | $R_{in}$ | $F_{1_{in}}$ | $P_{in}$ | $R_{in}$ | $F_{1_{in}}$ | $P_{in}$ | $R_{in}$ | $F_{1_{in}}$
Rule-based (Sieve-based) | 93.1 | 88.8 | 90.9 | 95.9 | 82.9 | 88.9 | 95.5 | 82.2 | 88.4 | 96.1 | 82.1 | 88.6
Ft-based Classifier | 19.0 | 24.4 | 21.4 | 26.6 | 14.9 | 19.0 | 24.1 | 15.8 | 19.1 | 25.0 | 19.2 | 21.7
Ft-based BLINKout $L^{(Ft)}_{NIL}$ | 60.4 | 44.6 | 51.3 | 72.1 | 38.1 | 49.9 | 57.0 | 35.5 | 43.7 | 78.1 | 41.4 | 54.1
$Th$-based BM25+BERT+syn | 71.3 | 76.3 | 73.6 | 86.2 | 73.0 | 79.0 | 79.2 | 72.6 | 75.8 | 80.5 | 73.0 | 76.6
$Th$-based BM25+SapBERT+syn | 78.2 | 81.7 | 79.9 | 88.7 | 75.1 | 81.3 | 84.4 | 75.3 | 79.6 | 89.7 | 70.9 | 79.2
BLINK (BERT) | 53.0 | 78.4 | 63.2 | 67.0 | 72.6 | 69.7 | 57.2 | 72.2 | 63.8 | 62.4 | 65.8 | 64.1
\+ syn | 58.1 | 85.8 | 69.3 | 70.4 | 76.3 | 73.2 | 60.2 | 76.0 | 67.2 | 68.8 | 72.6 | 70.6
$Th$-based BLINK | 76.0 | 74.7 | 75.3 | 76.3 | 70.2 | 73.2 | 79.9 | 69.3 | 74.2 | 75.3 | 65.3 | 69.9
\+ syn | 82.9 | 81.9 | 82.4 | 82.2 | 74.2 | 78.0 | 77.2 | 71.8 | 74.4 | 77.3 | 72.3 | 74.7
NIL-rep-based BLINK | 80.2 | 68.0 | 73.6 | 77.4 | 71.6 | 74.4 | 77.5 | 71.0 | 74.1 | 78.8 | 64.9 | 71.2
\+ syn | 80.5 | 83.6 | 82.0 | 78.3 | 75.3 | 76.8 | 81.0 | 74.5 | 77.6 | 85.3 | 71.3 | 77.7
BLINKout (- syn, BERT) | 80.2 | 71.3 | 75.4 | 77.9 | 62.2 | 69.2 | 77.4 | 65.6 | 71.0 | 76.5 | 67.6 | 71.8
BLINKout (BERT) | 86.4 | 85.2 | 85.8 | 76.5 | 71.5 | 73.9 | 81.4 | 75.4 | 78.3 | 75.3 | 69.1 | 72.1
BLINKout (SapBERT) | 89.9 | 90.0 | 90.0 | 84.3 | 84.1 | 84.2 | 84.1 | 81.5 | 82.8 | 83.4 | 81.6 | 82.5
\+ $k$ and NIL rep tuned | 90.5 | 91.0 | 90.7 | 87.1 | 83.3 | 85.1 | 88.9 | 83.7 | 86.2 | 82.8 | 83.5 | 83.1
\+ $L_{NIL}$ | 92.5 | 90.5 | 91.5 | 88.2 | 83.7 | 85.9 | 90.3 | 80.7 | 85.2 | 84.3 | 82.4 | 83.4
Table 5: Comparison results for in-KB Entity Linking (Precision, Recall,
$F_{1}$ scores for in-KB entities)

###  B.1 Dynamic Feature-based $L^{(Ft)}_{NIL}$ Inference

We adapted BLINKout for a joint, dynamic feature-based NIL classification as a
strong baseline. This is based on using dynamic weights (fused with static
features) in the neural network in the cross-encoder to classify NILs. The
joint feature-based learning is similar to Equation 7 by replacing
$L_{NIL_{m}}$ with $L^{(Ft)}_{NIL_{m}}$, based on the prediction score with
sigmoid activation below.

$s^{isNIL{{}_{Ft}}}_{m}=\sigma([f_{dme},f_{mes},v^{(cross)}_{m}]W_{NIL_{Ft}})$
(10)

The dynamic mention entity score feature set $f_{dme}$ denotes the cross-
encoder’s mention-entity scores of the mention, concatenated with three
additional dynamic features minimum, maximum, and average pooling among the
candidate entities. This is analogous to the in-KB scores and statistics
features in Rao et al. (2013); McNamee et al. (2009). The mention-entity
string matching feature set $f_{mes}$ is similar to the one described in
Appendix B and was calculated using the WordPiece subword tokens as in the
BERT model. The vector $v^{(cross)}_{m}$ is the mention representation same as
earlier in Equation 5. The binary cross entropy loss is thus:

$L^{(Ft)}_{NIL_{m}}=-y_{i}log(s^{isNIL{{}_{Ft}}}_{m})+(1-y_{i})log(1-s^{isNIL{{}_{Ft}}}_{m})$
(11)

The final loss with joint training for cross-encoder is described in Equation
12 below.

$L_{m}=L^{(cross)}_{m,is}+\lambda_{NIL}*L^{(Ft)}_{NIL_{m}}$ (12)

The model was trained using the top-$k$ entities from the bi-encoder in
BLINKout. The inference of whether a mention is in-KB or out-of-KB is from the
dynamic feature-based classifier with the score $s^{isNIL{{}_{Ft}}}_{m}$ in
Equation 10.

##  Appendix C Out-of-KB and In-KB Performance w.r.t Top-$k$

Figure 3 displays the change of $F_{1_{o}}$, $F_{in}$, and $A$ with respect to
the number of $k$ on MM-2014AB with the BLINKout model (SapBERT), and Figure 4
displays the change of recall@$k$ of the bi-encoder under the same setting,
where top-1 is the accuracy of the bi-encoder.

##  Appendix D Impact of Domain-Specific LMs

We found that in-domain and/or ontology-pre-trained models, for example,
SapBERT (PubMedBERT further pre-trained with UMLS entities and synonyms),
PubMedBERT (pre-trained from scratch with PubMed abstracts and PMC full
texts), BlueBERT (pre-trained on PubMed abstracts only or also on MIMIC-III
clinical notes) Peng et al. (2019), and BioBERT (pre-trained on PubMed
abstracts and PMC full texts) Lee et al. (2020) provided better results than
BERT on the datasets.

$k$=10 | ShARe/CLEF 2013 | MM-pruned-0.1
---|---|---
BLINKout | A | $F_{1_{o}}$ | $F_{1_{in}}$ | A | $F_{1_{o}}$ | $F_{1_{in}}$
BERT | 86.0 | 86.6 | 85.8 | 70.5 | 42.3 | 73.9
BioBERT | 86.0 | 87.0 | 85.5 | 75.1 | 41.2 | 78.2
BlueBERT | 87.6 | 88.5 | 87.2 | 75.3 | 45.6 | 78.9
BlueBERT (P) | 88.1 | 89.5 | 87.5 | 77.0 | 46.6 | 80.9
PubMedBERT | 88.3 | 87.8 | 88.5 | 82.9 | 62.8 | 85.1
SapBERT | 89.9 | 89.9 | 90.0 | 81.8 | 54.1 | 84.2
Table 6: Comparison results among Language Models for both out-of-KB and in-KB
mention discovery, and overall Entity Linking. Second-best results underlined.
BlueBERT (P): pre-trained on PubMed Abstract only.

##  Appendix E In-KB Entity Linking Results

In-KB EL results on the four datasets are presented in Table 5.

##  Appendix F Qualitative Analysis

Data w/ row ID |  Mention in context |  Gold |  $Th$-based BLINK |  NIL-rep-based BLINK |  BLINKout (tuned) |  BLINKout (tuned)+$L_{NIL}$
---|---|---|---|---|---|---
ShARe/ CLEF 2013 test-15 |  his chest tubes were removed, but cont. to have decreased breath sounds with CXR showing cont. atelectasis and effusion. |  NIL |  Harsh breath sounds (C0231859) 1.000>$Th_{cross}$ (0.95) |  Harsh breath sounds (C0231859) 1.000 |  NIL 1.000 |  NIL 1.000
ShARe/ CLEF 2013 test-22 |  Discharge Diagnosis: PEA arrest of unclear etiology |  NIL |  Electromechanical dissociation (C0340861) 1.000>$Th_{cross}$ (0.95) |  Electromechanical dissociation (C0340861) 0.998 |  Electromechanical dissociation (C0340861) 0.938 (where NIL 0.006) |  NIL 0.882
ShARe/ CLEF 2013 test-40 |  Low dose beta blockade decreased HR to 95 in sinus and synthroid had been restarted. |  Bradycardia (C0428977) |  NIL (Cardiac murmur, intensity grade I/VI (C0232249) 0.191<$Th_{cross}$ (0.95))  (Gold not in top-$k$) |  NIL 0.973 (Gold not in top-$k$) |  Bradycardia (C0428977) 1.000 |  NIL 0.867 (where Bradycardia (C0428977) 0.132)
MM-2014AB test-89 |  …a chronic kidney disease phase after stabilization of renal function… |  NIL |  Chronic Kidney Insufficiency (C0403447) 1.000>$Th_{cross}$ (0.55) |  Chronic Kidney Insufficiency (C0403447) 1.000 |  NIL 0.998 |  NIL 0.992
MM-2014AB test-90 |  NGAL can serve a biomarker for acute kidney injury and possibly for chronic kidney disease in the tubulointerstitial rat model. |  NIL |  Chronic rejection of renal transplant (C0403592) 0.675>$Th_{cross}$ (0.55) |  NIL 0.999 |  NIL 0.836 |  NIL 0.710
MM-2014AB test-1764 |  Gastrointestinal disorders in Curry-Jones syndrome Clinical and molecular insights from an affected newborn |  NIL |  Weyers acrofacial dysostosis (C0457013) 0.994>$Th_{cross}$ (0.55) |  NIL 0.9176 |  NIL 0.264 |  NIL 0.023
MM-pruned-0.1 test-14 |  Dietary antioxidants may play an important role in the prevention of bone loss and associated fractures by reducing levels of oxidative stress. |  NIL |  NIL (Tooth eruption disorder: (C0012767) 0.627<$Th_{cross}$ (0.80)) |  Tooth eruption disorder (C0012767) 0.426 (where NIL 0.320) |  NIL 1.000 |  NIL 0.997
MM-pruned-0.1 test-35 |  … including weight and fat gain, glucose intolerance, hypertriglyceridemia, abnormal adipocytokine levels, hypertension, and adiponectin and leptin gene expression and epigenetic changes. |  NIL |  Hyperglycemia (C0020456) 1.000 |  Hyperglycemia (C0020456) 1.000>$Th_{cross}$ (0.80) |  Dyslipidemias (C0242339) 0.564 (where NIL 0.011) |  Dyslipidemias (C0242339) 0.553 (where NIL 0.205)
Table 7: Examples of out-of-KB mention discovery from clinical notes
(ShARe/CLEF 2013) and biomedical publications (MedMentions) with the top
prediction from four BERT-based Entity Linking models: Threshold-based BLINK,
NIL representation based BLINK, BLINKout (tuned), and BLINKout (tuned) with
joint learning ($L_{NIL}$). Normalised prediction score (after softmax) is
displayed after the predicted entity. Wrong predictions are marked with red.
For the three datasets, ShARe/CLEF 2013, MM-2014AB, and MM-pruned-0.1, the
threshold $Th_{cross}$ for $Th$-based BLINK was 0.95, 0.55, 0.80, and the
number of top-$k$ for BLINKout was 150, 100, 50, resp.

We selected samples from the test set in the datasets regarding erroneous
predictions for out-of-KB mention discovery, as displayed in Table 7 (on the
next page). Note that all the presented models are context-aware, producing
different predictions when the same lexical mention is in a different context
(see rows 4-5).

BLINKout models can correctly identify NIL mentions with high confidence,
while false but similar in-KB entities were predicted in $Th$\- and NIL-rep-
based methods (see rows 1 and 4). BLINKout with joint learning affects the
softmax normalised score for NILs, potentially related to the percentage of
NIL mentions in the dataset; this can result in false positive NIL predictions
(as in row 3). NIL-rep-based approach can sensitively identify NIL mentions
with a high score (if not predicted as the top), without training with NIL-
labelled mentions. The threshold-based approach may likely result in false
negative predictions for out-of-KB mentions when the incorrect but similar
entity was predicted with a very high score near 100% (e.g. in rows 1-2, 4,
6-8). Finally, the pruning strategy is likely to produce challenging “out-of-
KB” entities which are very similar to in-KB entities, so that all models
yielded a wrong prediction (see the last row).