Advance Access Publication Date: Day Month Year Application Note

∗To whom correspondence should be addressed.

Received on XXXXX; revised on XXXXX; accepted on XXXXX

Associate Editor: XXXXXXX

# Enhancing Label Consistency on Document-level Named Entity Recognition

Data and text mining

Minbyul Jeong ${}^{\text{\sfb 1}}$ and Jaewoo Kang ${}^{\text{\sfb 1,2,3,*}}$
${}^{\text{\sf 1}}$Department of Computer Science and Engineering, Korea
University, Seoul, 02841, Republic of Korea
${}^{\text{\sf 2}}$Interdisciplinary Graduate Program in Bioinformatics, Korea
University, Seoul, Republic of Korea
${}^{\text{\sf 3}}$AIGEN Sciences, Seoul, 04778, Republic of Korea

(2022; 2022)

###### Abstract

Summary: Named entity recognition (NER) is a fundamental part of extracting
information from documents in biomedical applications. A notable advantage of
NER is its consistency in extracting biomedical entities in a document
context. Although existing document NER models show consistent predictions,
they still do not meet our expectations. We investigated whether the
adjectives and prepositions within an entity cause a low label consistency,
which results in inconsistent predictions. In this paper, we present our
method, ConNER, which enhances the label dependency of modifiers (e.g.,
adjectives and prepositions) to achieve higher label agreement. ConNER refines
the draft labels of the modifiers to improve the output representations of
biomedical entities. The effectiveness of our method is demonstrated on four
popular biomedical NER datasets; in particular, its efficacy is proved on two
datasets with 7.5–8.6% absolute improvements in the F1 score. We interpret
that our ConNER method is effective on datasets that have intrinsically low
label consistency. In the qualitative analysis, we demonstrate how our
approach makes the NER model generate consistent predictions.
Availability and implementation: Our code and resources are available at
https://github.com/dmis-lab/ConNER/.
Contact<EMAIL_ADDRESS>
Supplementary information: Supplementary data are available at Bioinformatics
online.

##  1 Introduction

Named entity recognition (NER) is the task of determining entity boundaries
and classifying categories of named entities. NER is a fundamental part of
biomedical applications (Kim et al., 2019; Wei et al., 2019; Lee et al.,
2020a; Weber et al., 2021; Lewis et al., 2021; Sung et al., 2022). In the
general domain, recent studies have attempted to train and evaluate NER models
in a document-level context (Yu et al., 2020; Yamada et al., 2020; Wang et
al., 2021; Gui et al., 2021). Likewise, the biomedical domain has shifted its
focus on evaluating document contexts rather than sentence contexts (Wei et
al., 2019; Wang et al., 2021; Weber et al., 2021).

There are several advantages of using document NER models: (1) The models
suggest a better way to bridge the gap between research and application
fields. Following previous studies, several researches have leveraged sentence
NER models in biomedical domains (Cho and Lee, 2019; Perera et al., 2020;
Jeong and Kang, 2021). However, biomedical applications require an evaluation
of the document rather than the sentence context (Wei et al., 2019; Kim et
al., 2019; Weber et al., 2021; Sung et al., 2022). (2) Document NER models
provide proper and consistent predictions owing to context completeness.
Recent works (Yamada et al., 2020; Wang et al., 2021; Gui et al., 2021) have
shown that using document contexts improves the accuracy of the models.
Although the authors of (Gui et al., 2021) used powerful context
representations such as BERT (Devlin et al., 2018) or ELMo (Peters et al.,
2018), their inability to model document-level label consistency resulted in
insufficient performance. Therefore, it is challenging to understand which
factors contribute to creating document NER models that produce a consistent
prediction in the same manner.

To tackle the above challenges, a series of studies (Fu et al., 2020, 2021)
have provided an interpretable evaluation to identify the attributes of
datasets depending on their characteristics. Our intuitive motivation is that
entity-aware attributes are beneficial for achieving higher label consistency,
which can also improve the accuracy of the NER models. In this paper, our goal
is to develop a NER model that can be trained to predict an entity
consistently in a document context.

We first clarify why we need the NER model to make consistent predictions. We
provide our motivating example in Figure 1. For example, the mention
‘colorectal cancer’ is an entity of the disease type. Predicting such a
mention is challenging in a sentence context owing to context incompleteness.
As a result, the sentence model produces an error in predicting ‘non – FAP’ or
‘colorectal’. Meanwhile, the NER model trained on document contexts shows
consistent predictions because the token ‘colorectal’ occurs frequently within
the documents. Therefore, the document provides sufficient label agreement to
learn label representations of ‘colorectal’. Although the document NER model
shows consistent predictions and much better performance than the sentence NER
model (Section 1), it still falls behind our expectations. Models trained on
document contexts continue to produce 64% errors that contain modifiers (i.e.,
prepositions or adjectives), such as primary, hereditary, and congenital
(Section 5.1). Because modifiers are used as both entity and non-entity tokens
depending on the context situation, they are difficult to predict. Therefore,
the modifiers exhibit a low label consistency score and occur at short entity
lengths (Figure 2).

Figure 1:  Overview of motivating examples of entity-aware attributes. (Left)
We provide an example with a gold label, sentence prediction, and document
prediction. Most of the biomedical entities contain the adjective or
preposition within the entities. Suggesting a sufficient context improves
label agreement of those adjectives such as ‘colorectal’. (Right) It is worth
noting that the majority of biomedical datasets are derived from its golden
paragraph (PubMed).

Results on four biomedical NER benchmarks. We use BioBERT (Lee et al., 2020b)
and BioLM (Lewis et al., 2020a) to compare the predictions between different
context levels. We report the F1 score to evaluate the models.  Context Model
Evaluation (F1) NCBI-disease CDR AnatEM Gellus Avg. Sentence BioBERT 89.0 89.1
73.9 54.9 76.7 BioLM 88.3 89.5 74.9 55.9 77.2 Document BioBERT 89.1 89.9 77.1
56.3 78.1 BioLM 88.5 90.2 80.3 56.9 79.0

To avoid the aforementioned errors, we present ConNER which enhances the label
dependency of modifiers to achieve higher label agreement. An abstract of
biomedical literature is fed into the biomedical pre-trained language model,
BioBERT (Lee et al., 2020b) or BioLM (Lewis et al., 2020a), to output context
representation (Section 3.1). On top of the pre-trained language model, we
propose label refinement to improve the label representations of uncertain
tokens within entities (Section 3.2). We also suggest our loss term for
biomedical entities to resemble the label representations on two different
architectures: fully connected layers (MLP) and bidirectional long-short term
memory (BiLSTM) architecture. We use MLP as the main classification layer to
generate final output representations of raw text and use the BiLSTM
architecture to generate label representations of biomedical entities. We
adopt the notion that the MLP layer is more robust to long entities (Lafferty
et al., 2001), which is an essential property of biomedical entities. We
employ the BiLSTM architecture to improve the label dependency of biomedical
entities (Collobert et al., 2011; Lample et al., 2016). To demonstrate the
effectiveness of the proposed ConNER approach, we use four biomedical
benchmarks. On three datasets, we achieve a higher F1 score than previous
state-of-the-art models (Wang et al., 2021; Lewis et al., 2020a). In
particular, our model achieves higher label agreement and proves its efficacy
on two datasets with 7.5–8.6% absolute improvements in the F1 score (Section
4.4). We provide an interpretation of the effectiveness of our method for a
dataset that intrinsically has a low label consistency score (Section 5.2).

The contributions of this study are summarized as follows. (1) We investigate
why document NER models make inconsistent predictions in biomedical domains.
Based on our observations, we observe that modifiers (i.e., adjectives and
prepositions) exhibit a low label consistency score and produce errors by
making inconsistent predictions. (2) We present our method, ConNER, which
enhances the label dependency of modifiers to generate improved label
representations. (3) The results of the experiments show that our ConNER
approach significantly improves the accuracy of document NER models,
indicating that it can help achieve the highest level of label agreement. (4)
For different tasks related to low label consistency, we show that ConNER
outperforms various baselines, and we analyze the factors influencing its
performance.

##  2 Background

###  2.1 Named Entity Recognition

The goal of NER is to find a word or phrase that corresponds to a specific
instance, such as a person, location, organization, or any other miscellaneous
entity. The NER task primarily involves the extraction and classification of
named entities found in a corpus with pre-defined entity tags. We use BIO
tagging (Ramshaw and Marcus, 1999), where a B- prefix (Beginning) indicates
the beginning of the chunk following an I- prefix (Inside), and an O- prefix
(Outside) indicates being inside and not belonging to the chunk. The task
commonly uses two different decoding strategies: tag-independent decoding
(i.e., MLP (Collobert et al., 2011)) and tag-dependent decoding (i.e., CRF
(Lafferty et al., 2001; Lample et al., 2016)). Here, we use only the tag-
independent decoding strategy.

###  2.2 Attribute Definition

Following previous works (Fu et al., 2020, 2021), we define the term attribute
as a value that characterizes the properties of an entity that may be
correlated with performance. The authors introduced attributes bridging the
gap between the final performance (we use the F1 score) and interpretable
evaluation based on model predictions. Assuming that one attribute is given,
the test set of NER tasks naturally partitions into several interpretable
buckets, we investigate whether the attribute affects the final performance
bucket-wise.

Formally, we define notations to facilitate the definition of attributes.
Given a set of documents $D$, entity tagging aims to extract a set of entities
$\mathcal{E}$ as spans or tokens. We first denote a span set
$\mathcal{E}^{tr}$ $\subset$ $\mathcal{E}$ as an argument. Specifically, (Fu
et al., 2020) introduced a feature function $\phi(\cdot)$ to aggregate
features to interpret the properties of tagged entities $\mathcal{E}^{tr}$:

$\mathcal{F}(\textbf{x},\phi(\cdot),D(\mathcal{E}))=\frac{|\\{\varepsilon|\phi(\varepsilon)=\phi(\textbf{x}),\forall\varepsilon\in
D(\mathcal{E})\\}|}{|D(\mathcal{E})|},$ (1)

where x denotes an entity span that can also replace it as token $x$.
Similarly, we define some training set-independent attribute functions as
follows:

$\phi_{tLen}(x)=|x|:\text{token span length}$ (2)
$\phi_{eLen}(\textbf{x})=|\textbf{x}|:\text{entity span length}$ (3)
$\phi_{dLen}(\textbf{x})=|doc(\textbf{x})|:\text{document length}$ (4)
$\phi_{eDen}(\textbf{x})=|ent(doc(\textbf{x}))|/\phi_{dLen}(\textbf{x}):\text{entity
density}$ (5)
$\phi_{oDen}(\textbf{x})=|oov(doc(\textbf{x}))|/\phi_{dLen}(\textbf{x}):\text{OOV
density}$ (6)

Rather than using a sentence-level context, we provide a document-length
attribute based on the usage of the document-level context.

Figure 2:  Consistent score per entity length. The x-axis denotes the entity
length (i.e., $\phi_{eLen}$), and the y-axis denotes the consistent score
(i.e., $\phi_{eCon}$). Short lengths of entities show low consistent scores
due to the low label agreement of the modifier tokens.

We use two functions to define the density of entity $ent(\cdot)$, which
counts the number of entity words, and the density of out-of-vocabulary words
$oov(\cdot)$, which counts the number out of training set words. We also
leverage the training set-dependent attribute functions, as follows:

$\phi_{tFre}(x):=\mathcal{F}(x,\phi_{str}(\cdot),D(\mathcal{E}^{tr})):\text{token
frequency}$ (7)
$\phi_{eFre}(\textbf{x}):=\mathcal{F}(\textbf{x},\phi_{str}(\cdot),D(\mathcal{E}^{tr})):\text{entity
frequency}$ (8)
$\phi_{tCon}(x):=\mathcal{F}(x,\phi_{label}(\cdot),D(\mathcal{E}^{tr})):\text{label
consistency of token}$ (9)
$\phi_{eCon}(\textbf{x}):=\mathcal{F}(\textbf{x},\phi_{label}(\cdot),D(\mathcal{E}^{tr})):\text{label
consistency of entity}$ (10)

where $\phi_{str}(\cdot)$ and $\phi_{label}(\cdot)$ denote a string and label
of the corresponding argument, respectively, and $\phi_{tCon}(x)$ and
$\phi_{eCon}(\textbf{x})$ refer to measuring how consistently a certain token
or entity span is assigned to a pre-defined label, respectively. Here, we
define label consistency as the degree of label agreement of n tokens/entities
in the training set. We use attribute functions to interpret which dataset
attributes have an impact on performance improvement (Figure 3).

##  3 Method

In this section, we start by learning the document-level model through the
supervision of named entity recognition tasks. Our goal is to learn how to
enrich the label representations of biomedical entities by improving their
label consistency (Section 3.1). We then introduce our label refinement on
biomedical entities, which encourages label representations of uncertain
tokens within entities (Section 3.2). Finally, we discuss the use of a loss
term for biomedical entities to resemble the label representations of two
different architectures. Figure 4 depicts the overall structure of ConNER
approach.

Figure 3:  Overview of Dataset Biases. Each dot denotes an attribute score of
entities. Intuitively, these intrinsic differences in datasets explain what
factors have a significant influence on improving performance.  Figure 4:
Overview of ConNER performing label refinement on biomedical entities. The
abstract of biomedical literature is fed into the pre-trained language model
$\mathcal{M}_{LM}$ to output context representation. In the label-refinement
process, we use a mask tensor $MASK$ to find the position of biomedical
entities and feed it into the BiLSTM architecture $\mathcal{M}_{LSTM}$ to
improve label dependency of modifiers within entities that induce low label
agreement. On top of the fully connected layer, we compute an uncertainty
score $U$ to determine which tokens need to be modified for their predicted
label.

###  3.1 Document Named Entity Recognition

ConNER consists of a pre-trained language model $\mathcal{M}_{LM}$ and label-
refinement process, as shown in Figure 4. For document tagging, we use
abstracts of biomedical literature as raw input and make predictions for each
word. Let $D_{i}=\\{T_{i1},T_{i2},...,T_{iN}\\}$ represent a sequence of
tokens, where $N$ denotes the total number of tokens in a document $D_{i}$.
First, we apply a biomedical pre-trained language model (e.g., BioBERT (Lee et
al., 2020b) or BioLM (Lewis et al., 2020a)) to obtain contextualized word
representations for each token: $T_{i1},T_{i2},...,T_{iN}\in\mathbb{R}^{d}$.
Then, we represent a biomedical entity e as the concatenation of start-to-end
vector representations, as follows:

$\mathcal{M}_{LM}(e)=[T_{start}:T_{end}]\in\mathbb{R}^{\phi_{eLen}(\textbf{e})\cdot
d}$ (11)

where $T_{start}$ and $T_{end}$ denote the start and end of biomedical entity
$\textbf{e}\in\mathcal{E}$, respectively, and $d$ denotes the final hidden
dimension of language model $\mathcal{M}_{LM}$. Note that we use the notation
e as an example to help understand our method ConNER. We use tag-independent
decoding (i.e., MLP) on the model $\mathcal{M}_{LM}$ as the main
classification layer, which is useful for predicting long-named entities (Fu
et al., 2020). Formally, we can define our classification loss $L_{class}$
using cross-entropy objectives to optimize our model as follows:

$L_{class}=-\frac{1}{N}\sum_{j=1}^{N}y_{j}\cdot log(p_{j})$ (12)

where $y_{j}$ denotes the ground-truth label and $p_{j}$ denotes the
probability of the main classification layer.

#### Sentence vs. Document.

In our pilot experiments, we first observe that the label consistency of
entity $\phi_{eCon}(\textbf{x})$ is enhanced when we expand the context from
sentences to documents. We also observe that using a larger context is helpful
in achieving better performance (+1.4-1.8% in F1 score), as shown in Section
1. Based on our positive observations, we hypothesize that one key reason for
the performance gain is that biomedical entities are challenging to predict at
the sentence level owing to context incompleteness. Learning from the limited
context makes it difficult to predict a token within an entity such as
‘colorectal’ (see Figure 1).

###  3.2 Label Refinement on Biomedical Entities

Overall, we propose a label-refinement process that depends on an entity-
length attribute $\phi_{eLen}(\textbf{e})$ to enrich the label representations
of the entity spans. To encourage label representations, we add a loss term of
the entity span to make the label distribution similar between the main
classification layer and the refinement process. We use the MLP layer to
obtain a final output representation and both an uncertainty and a draft label
for each token representation. We use a BiLSTM architecture (Lafferty et al.,
2001) to improve the label dependency of tokens within biomedical entities and
to generate label representations of entities. These label representations
from the BiLSTM architecture provide a precise label to the main
classification layer by verifying the uncertainty of the draft label (see
Figure 1).

The independent decoder predicts well for longer entities, especially
biomedical entities (Fu et al., 2020, 2021; Jeong and Kang, 2021). With the
preceding observation, we find that the modifiers (i.e., prepositions or
adjectives) are both used as entity and non-entity tokens. We also find that a
low label consistency occurs at short lengths of entities (i.e.,
$\phi_{eLen}(\textbf{e})<5$), which most modifiers consist of in this scope.
As a result, we attempted to deal with long entities by employing the MLP
layer and developing a label-refinement process to target those modifiers to
be predicted consistently.

#### Uncertainty.

We ask the natural question: How can we decide what tokens should be refined?
We choose this criterion for the certainty of the draft label on token
representations (Gui et al., 2021). We calculate the uncertainty $U$ of the
probability distributions of the MLP layer as follows:

$U=H(p_{c})=-\sum_{c=1}^{C}p_{c}~{}\text{log}~{}p_{c}.$ (13)

where $C$ denotes the number of pre-defined entity types per dataset. We use
the entropy of the probability distribution $p_{c}$ to define the uncertainty
$U$. Given this uncertainty score, label refinement proposes a positive
training signal to shift the draft label.

#### Label Refinement.

We decide to improve the label dependency on modifiers (i.e., adjectives or
prepositions) used in the entities during training. Specifically, we create an
entity-aware mask tensor $MASK\in\mathcal{R}^{B\cdot N\cdot N}$ to indicate
the position of the padded indices such that our model does not attend to non-
entities, where $B$ denotes the batch size. The mask tensor is applied to the
BiLSTM layers $\mathcal{M}_{LSTM}$ to obtain a label representation
${l}(\textbf{e})$, as follows:

$l(\textbf{e})=\mathcal{M}_{LSTM}\\{MASK(\mathcal{M}_{LM}(\textbf{e}))\\}\in\mathbb{R}^{\phi_{eLen}(\textbf{e})\cdot
C}.$ (14)

Then, $\mathcal{M}_{LSTM}$ trains to softly assist the main classification
layer in performing precise predictions with improved consistency. Note that
we use the term softly assist to describe a process that assists the main
classification layer in shifting its draft label during training. That is, the
trained model $\mathcal{M}_{LSTM}$ enhances the label dependency of an
uncertain token and softly assists in refining the draft label of the
uncertain token. Formally, we add element-wisely to assist predictions before
calculating a label loss $L_{label}$, as shown below:

$p=p\oplus l,$ (15) $L_{label}=-\frac{1}{N}\sum_{j=1}^{N}y_{j}\cdot
log(l_{j})$ (16)

where $y_{j}$ and $l_{j}$ denote the ground-truth label and probability of the
$\mathcal{M}_{LSTM}$ layer, respectively, and $p$ is used to compute the
classification loss in Equation 12. Finally, we determine the criterion using
the pre-computed uncertainty score $U$ in Equation 13. We set an uncertainty
threshold $\Gamma$ to distinguish the tokens that should be refined within
entities. For example, in Figure 4, a token ‘colorectal’ of the entity
‘colorectal cancer’ is first predicted as an Outside tag. During training, the
token ‘colorectal’ is converted to a Beginning tag because of the high
uncertainty score. We found that $\Gamma=0.3$ worked well in practice. See
Figure 5 for an ablation study.

#### Distillation.

We propose improving the label representations of biomedical entities by
distilling knowledge (Hinton et al., 2015) from our decoding layers. We
minimize the Kullback-Leibler divergence between the probability distribution
from the tag-independent layer (i.e., MLP) and the tag-dependent layer (i.e.,
BiLSTM) on top of the biomedical pre-trained language model. The distillation
loss was computed as follows:

$L_{distill}=\frac{KL(p||l)+KL(l||p)}{2},$ (17)

where $p$ and $l$ denote the probability distributions of the MLP and BiLSTM
layers, respectively. Note that distillation is computed before Equation 15.

###  3.3 Training Objective

We optimize the three losses altogether to improve the label agreement of the
entity through label refinement and distilling knowledge while predicting on
the tag-independent layer. Our final loss is computed as follows:

$L=\lambda_{1}L_{class}+\lambda_{2}L_{label}+\lambda_{3}L_{distill}$ (18)

where $\lambda_{1},\lambda_{2},\lambda_{3}$ scale the importance of each loss
term. We observe that $\lambda_{1}=1$, $\lambda_{2}=1e-1$, and
$\lambda_{3}=1e-3$ exhibit the best performance in our framework. See Section
4.1 for an ablation study of the other components.

##  4 Experimental Setup

Statistics of four biomedical named entity recognition datasets. The CDR
dataset suggests two entity types that share the same text. Annotations (Ann.)
and Unique Ann. refer to the entire entity and its unique numbers
respectively.  Corpora Entity Type Documents Sentences Tokens Annotations
Unique Ann. NCBI-disease Disease 793 7,142 180,992 6,881 2,502 CDR Disease /
Chemical 1,500 14,503 346,019 12,957 / 15,837 4,477 / 3,765 AnatEM Anatomic
1,212 11,809 298,780 13,692 5,042 Gellus Cell Lines 1,212 11,809 312,584 650
243

Hyperparameters of the four biomedical named entity recognition settings. For
common hyperparameter settings, we focus on searching batch size, learning
rate, and training epoch for learning schemes. We also focus on searching for
the optimal hyperparameters of the uncertainty threshold $\Gamma$.  Context
Corpora Pretrained LM Batch Size Learning Rate Training Epoch Sequence Length
Threshold $\Gamma$ $\lambda_{1}$ $\lambda_{2}$ Document NCBI-disease BioLM 6
3e-5 50 512 0.3 1e-1 1e-3 CDR BioLM 6 3e-5 30 512 0.3 1e-1 1e-3 AnatEM BioLM 6
5e-5 30 512 0.3 1e-1 1e-3 Gellus BioLM 6 3e-5 40 512 0.3 1.0 1e-3

###  4.1 Dataset

We use four biomedical NER benchmarks across four entity types: NCBI-disease
(Doğan et al., 2014), CDR (Li et al., 2016), AnatEM (Pyysalo and Ananiadou,
2014), and Gellus (Kaewphan et al., 2016), following the standard
train/dev/test splits for biomedical NER evaluation. (1) NCBI-disease (Doğan
et al., 2014) consists of 793 PubMed abstracts with manually annotated disease
entities. (2) CDR (Li et al., 2016) contains 1,500 PubMed abstracts manually
annotated with disease and chemical entities in the same context. (3) AnatEM
(Pyysalo and Ananiadou, 2014) consists of 1,212 PubMed abstract and full-text
extracts annotated with 12 anatomical entity types. (4) Gellus (Kaewphan et
al., 2016) consists of annotating cell lines in 1,212 documents from PubMed
abstracts and PMC full-text extracts. Half of the corpora were drawn from the
AnEM corpus (Ohta et al., 2012), and the other half were drawn from the BioNLP
ST’13 Cancer Genetics (CG) task documents Pyysalo et al. (2013). Section 4
shows the dataset statistics.

Results on biomedical NER benchmarks. F1 score is reported. The best score is
displayed in bold and the second-best score is underlined. † numbers are
estimated from the figures in the original papers.  Context Model Evaluation
(F1) NCBI-disease CDR AnatEM Gellus Sentence B-MTM (Crichton et al., 2017)†
80.4 89.2 82.2 - BiLSTM-CRF (Habibi et al., 2017)† 84.6 - - 75.6 BioBERT (Lee
et al., 2020b) 89.0 89.1 73.9 54.9 BioLM (Lewis et al., 2020a) 88.3 89.5 74.9
55.9 CL-L2 (w/o context) (Wang et al., 2021)† 89.2 90.7 - - CL-KL (w/o
context) (Wang et al., 2021)† 89.2 90.7 - - Document CL-L2 (w/ context) (Wang
et al., 2021)† 89.2 91.0 - - CL-KL (w/ context) (Wang et al., 2021)† 89.0 90.9
- - ConNER (Ours) 89.9 91.3 83.5 63.4

An ablation study of ConNER components. We perform three different
experiments: removing distillation (— {$L_{distill}$}), label refinement (—
{$L_{label}$}), and both of these process (— {$L_{distill}$,$L_{label}$}). We
used our best hyperparameter setting on the BioLM (Lewis et al., 2020a) model
equally. The best scores are displayed in bold.  Model Description Evaluation
(P/R/F1) NCBI-disease CDR AnatEM Gellus ConNER 88.8 / 91.1 / 89.9 89.9 / 92.7
/ 91.3 83.2 / 83.7 / 83.5 62.6 / 64.2 / 63.4 ConNER — {$L_{distill}$} 87.5 /
89.9 / 88.7 89.2 / 92.3 / 90.7 81.6 / 81.1 / 81.3 54.3 / 60.3 / 57.1 ConNER —
{$L_{label}$} 87.4 / 90.4 / 88.8 89.4 / 92.1 / 90.7 81.0 / 81.9 / 81.5 54.2 /
61.2 / 57.5 ConNER — {$L_{distill}$,$L_{label}$} 87.4 / 89.7 / 88.5 89.8 /
90.7 / 90.2 80.3 / 80.4 / 80.3 54.8 / 59.1 / 56.9

###  4.2 Comparison Methods

We evaluate the sentence- and document-level contexts and compare ConNER with
several neural network models commonly used in biomedical domains. (1) B-MTM
(Crichton et al., 2017) developed a multi-task learning model using various
biomedical sources annotated with different entity types. (2) BiLSTM-CRF
(Habibi et al., 2017) proposed a combination of word embeddings and LSTM with
the CRF decoding strategy in the biomedical domain. (3) BioBERT (Lee et al.,
2020b) introduced a biomedical-specific language representation model pre-
trained on large-scale biomedical corpora. (4) BioLM (Lewis et al., 2020a)
suggested a biomedical-specific language representation model pre-trained on
biomedical and clinical corpora. (5) CL-KL and CL-L2 (Wang et al., 2021)
proposed a method that can retrieve and select a semantically relevant context
using a search engine to improve contextual representations, with the original
sentence as a query.

###  4.3 Implementation Details

We train ConNER using BioLM (Lewis et al., 2020b) or BioBERT (Lee et al.,
2020b) to treat biomedical entity types. These two pre-trained language models
are commonly used as backbone models in the biomedical domain, and we adopt
these models to generate contextualized representations. We set 128 tokens as
the maximum sequence length for the sentence-level context and 512 tokens for
the document-level context, and tokens with more than the maximum sequence
length were truncated. The batch size was set to 32 for the sentence level and
6 for the document level. We select a learning rate in the range {3e-5, 5e-5}.
We search for a training epoch in the range {30, 40, 50}. We suggest our total
hyperparameter settings in Section 4. We train our model with a single NVIDIA
Titan RTX (24GB) GPU for fine-tuning, and the training time took less than 2
hours.

###  4.4 Experimental Results

Section 4.1 reports the results of ConNER approach. To show the effectiveness
of ConNER approach, we evaluate our model in the named entity recognition
setting in four biomedical domains and compare its performance with that of
other methods in different contexts. Compared to previous biomedical NER
models, our approach achieves the best performance compared to all other
baselines on three datasets: NCBI-disease (Doğan et al., 2014), CDR (Li et
al., 2016), and AnatEM (Pyysalo and Ananiadou, 2014). This demonstrates that
improving the label agreement of modifiers is effective in a document context.
In addition, the performance gap between BioLM and ConNER on AnatEM (74.9 vs.
83.5) and Gellus (57.9 vs. 63.4) shows that the label refinement based on the
dataset bias, which has low label consistency, is effective. Although it still
lags behind the BiLSTM-CRF model, the gap is reduced. We provide additional
ablation studies of ConNER approach in the following sections.

##  5 Analysis

###  5.1 Ablation studies

Section 4.1 shows our ablation result on four biomedical NER benchmarks. We
evaluate our approach ConNER by removing its components: 1) distillation
(—$L_{distill}$) and 2) the label-refinement process (—$L_{label}$). The
experiments show that ConNER is effective for all four benchmarks.
Specifically, we observe that the AnatEM and Gellus datasets show significant
improvement, demonstrating that our approach ConNER is effective for datasets
with low label consistency on the entities shown in Figure 3. We also observe
that adding each component consistently improves the recall metrics. These
observations correspond to an advantage of ConNER approach, whereby it decides
which token should be refined by relying on the uncertainty threshold
$\Gamma$.

A sample prediction of ConNER on the AnatEM and CDR datasets. yellow highlight
refers to the correct prediction and red signifies to the wrong prediction.
ConNER Dataset: AnatEM (Pyysalo and Ananiadou, 2014) PMID: 10420526 Title:
[Histopathologic examination of rectal carcinoma(1)] Abstract: In patients
with rectal carcinoma, the histopathological evaluation of the surgical
specimen(2) provides pivotal prognostic and therapeutic information. Important
parameters are tumor site, depth of invasion, histological type and grade,
pattern of invasion (diffusely infiltrating versus expanding margin(3)),
degree of peritumoral lymphocytic infiltration, and tumor involvement of
surgical margins and lymph nodes(3). Evaluation of the circumferential (deep,
lateral) margin is of utmost importance. It should be labeled with ink in the
gross specimen and should be examined histologically using several tissue
blocks. The number of lymph node metastases and the total number of lymph
nodes examined should be reported. A histological evaluation of the distal
mesorectum in its entirety is recommended to detect discontinuous distal
mesorectal tumor spread. The histopathological findings should be summarized
using the TNM-classification. Dataset: CDR (Li et al., 2016) PMID: 9158667
Title: Thrombotic complications in acute promyelocytic leukemia during all-
trans-retinoic acid therapy Abstract: A case of acute renal failure, due to
occlusion of renal vessels in a patient with acute promyelocytic leukemia
(APL) treated with all-trans-retinoic acid (ATRA) and tranexamic acid has been
described recently. We report a case of acute renal failure in an APL patient
treated with ATRA alone. This case further supports the concern about
thromboembolic complications associated with ATRA therapy in APL patients. The
patients, a 43-year-old man, presented all the signs and symptoms of APL and
was included in a treatment protocol with ATRA. After 10 days of treatment, he
developed acute renal failure that was completely reversible after complete
remission of APL was achieved and therapy discontinued. We conclude that ATRA
is a valid therapeutic choice for patients with APL, although the procoagulant
tendency is not completely corrected. Thrombotic events, however, could be
avoided by using low-dose heparin.

Figure 5:  An analysis of uncertainty threshold $\Gamma$. The x-axis refers to
the threshold $\Gamma$, and the y-axis denotes the F1 score. The overall
results show that $\Gamma=0.3$ results in stable performance on the four
biomedical benchmarks.

We also provide an interpretation of the threshold $\Gamma$ (see Figure 5). We
observe that $\Gamma=0.3$ works well in our four benchmarks, except for the
Gellus dataset. When $\Gamma$ is higher than 0.6 in the NCBI-disease dataset,
the F1 score yields the same results. We analyze the situation in which a
biomedical pre-trained language model has already demonstrated stable
performance in predicting biomedical entities. We find that ConNER approach
does not have to interfere with the predictions of the main classification
layer. In contrast, in the Gellus dataset, $\Gamma=0.8$ shows the best
performance with high precision and recall metrics. The results were found to
be steady when we compute five times using different seeds. However, we
suggest using $\Gamma=0.3$ to achieve a stable performance on all benchmarks.

Case studies of inconsistent predictions in document NER models. The below
case was taken from the NCBI-disease dataset Doğan et al. (2014).
$\phi_{tCon}$ refers to the token label consistency. The predictions of BioLM
(Lewis et al., 2020a) and our ConNER approach are compared. Our ConNER
approach achieves consistent predictions on the modifier examples.  Data
Criteria Modifier Token Examples primary genetic hereditary inherited
congenital abnormal Train $\phi_{tCon}$ 0.11 0.79 0.13 0.58 0.27 0.94 Test
$\phi_{tCon}$ 0 0.81 0.08 0.55 0.25 1 BioLM 0% 50% 50% 66% 77% 44% ConNER 100%
75% 94% 100% 100% 78%

We also see if our ConNER approach can improve the label consistency of
entities containing modifier tokens. Section 5.1 shows the cases of
inconsistent predictions in document NER models. We analyze the modifier
tokens used as both entity and non-entity tokens depending on the context. For
example, in the training dataset of NCBI-disease, the ‘primary’ token is used
as an entity only 11 times out of 100 times (i.e., we denote this 0.11 in
Section 5.1). However, the corresponding token has not occurred as an entity
in the test dataset. We observe that BioLM trained on document context did not
predict consistently on these modifier tokens. Owing to the improvement of
label dependency on the tokens within entities, the ConNER approach achieves
consistent predictions on the modifier tokens.

###  5.2 Qualitative analysis

Section 5.1 shows our ConNER prediction on the AnatEM dataset. We investigate
which examples could show the advantages and disadvantages of our approach and
find three different examples: (1) Our model consistently predict ‘rectal
carcinoma’ mention as an entity. Compared to ConNER prediction, a model
without label refinement and distillation process (i.e., ConNER —
{$L_{distill}$,$L_{label}$}) predicts ‘rectal’ as an entity token in the first
appearance and as a non-entity token in the second appearance. (2) Another
surprising example is the token ‘surgical’. In the training dataset, the
‘surgical’ token achieved a consistency score of 92.3% in the entity
dictionary. In other words, it was mostly used as an entity token, accounting
for 92.3% of the total. The ConNER approach performs well in predicting the
‘surgical’ token as an entity token. As the intention was to enhance the label
dependency of adjectives within entities, it surely works well as per our
observations.

However, ConNER still has several limitations in that it cannot generalize on
out-of-density tokens such as ‘mesorectum’ or ‘margin’ that never occur in the
training dataset. ConNER approach cannot predict these tokens as biomedical
entities which is a common situation in real-world scenarios. On comparing the
predictions ‘peritumoral lymphocytic’ and ‘lymph nodes’, we can see that the
subtoken ‘lymph’ was predicted inconsistently even if they were composed of
the same subtoken. After analyzing the reason why ‘lymph’ subtoken was
mispredicted, we interpret that it has a low degree of label agreement in a
given paragraph or even in the entire dataset (0.62 of consistency score). In
our future work, we will perform an investigation on ways to make ConNER more
generalizable in out-of-density tokens and more consistent in inconsistent
subtokens.

##  6 Conclusion

In this paper, we present ConNER approach, which enhances label dependency to
construct consistent biomedical NER models in document contexts. We present a
label-refinement process and encouragement of label representation to
consistently predict biomedical entities. The ConNER approach outperforms
existing biomedical NER methods in three biomedical domains while providing a
view of connecting dataset attributes with a training framework. We also
achieve a powerful performance on low-resource datasets, showing the
possibility of adopting our approach for biomedical NER applications. In our
future work, we will attempt to handle out-of-density tokens that never occur
in the training datasets, which is a common situation in real-world scenarios.
We expect that our solution will be more sophisticated by not only using
golden paragraphs but also utilizing retrieved contexts based on out-of-
density tokens.

## Funding

## References

  * Cho and Lee (2019) Cho, H. and Lee, H. (2019).  Biomedical named entity recognition using deep neural networks with contextual information.  BMC bioinformatics, 20(1), 1–11.
  * Collobert et al. (2011) Collobert, R. et al. (2011).  Natural language processing (almost) from scratch.  Journal of machine learning research, 12(ARTICLE), 2493–2537.
  * Crichton et al. (2017) Crichton, G. et al. (2017).  A neural network multi-task learning approach to biomedical named entity recognition.  BMC bioinformatics, 18(1), 1–14.
  * Devlin et al. (2018) Devlin, J. et al. (2018).  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * Doğan et al. (2014) Doğan, R. I. et al. (2014).  Ncbi disease corpus: a resource for disease name recognition and concept normalization.  Journal of biomedical informatics.
  * Fu et al. (2020) Fu, J. et al. (2020).  Interpretable multi-dataset evaluation for named entity recognition.  arXiv preprint arXiv:2011.06854.
  * Fu et al. (2021) Fu, J. et al. (2021).  Larger-context tagging: When and why does it work?  arXiv preprint arXiv:2104.04434.
  * Gui et al. (2021) Gui, T. et al. (2021).  Leveraging document-level label consistency for named entity recognition.  In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3976–3982.
  * Habibi et al. (2017) Habibi, M. et al. (2017).  Deep learning with word embeddings improves biomedical named entity recognition.  Bioinformatics, 33(14), i37–i48.
  * Hinton et al. (2015) Hinton, G. et al. (2015).  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2(7).
  * Jeong and Kang (2021) Jeong, M. and Kang, J. (2021).  Regularization for long named entity recognition.  arXiv e-prints, pages arXiv–2104.
  * Kaewphan et al. (2016) Kaewphan, S. et al. (2016).  Cell line name recognition in support of the identification of synthetic lethality in cancer from text.  Bioinformatics, 32(2), 276–282.
  * Kim et al. (2019) Kim, D. et al. (2019).  A neural named entity recognition and multi-type normalization tool for biomedical text mining.  IEEE Access.
  * Lafferty et al. (2001) Lafferty, J. et al. (2001).  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
  * Lample et al. (2016) Lample, G. et al. (2016).  Neural architectures for named entity recognition.  arXiv preprint arXiv:1603.01360.
  * Lee et al. (2020a) Lee, J. et al. (2020a).  Answering questions on covid-19 in real-time.  arXiv preprint arXiv:2006.15830.
  * Lee et al. (2020b) Lee, J. et al. (2020b).  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics.
  * Lewis et al. (2020a) Lewis, P. et al. (2020a).  Pretrained language models for biomedical and clinical tasks: Understanding and extending the state-of-the-art.  In Proceedings of the 3rd Clinical Natural Language Processing Workshop.
  * Lewis et al. (2020b) Lewis, P. et al. (2020b).  Pretrained language models for biomedical and clinical tasks: understanding and extending the state-of-the-art.  In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 146–157.
  * Lewis et al. (2021) Lewis, P. et al. (2021).  Paq: 65 million probably-asked questions and what you can do with them.  Transactions of the Association for Computational Linguistics, 9, 1098–1115.
  * Li et al. (2016) Li, J. et al. (2016).  Biocreative v cdr task corpus: a resource for chemical disease relation extraction.  Database, 2016.
  * Ohta et al. (2012) Ohta, T. et al. (2012).  Open-domain anatomical entity mention detection.  In Proceedings of the workshop on detecting structure in scholarly discourse, pages 27–36.
  * Perera et al. (2020) Perera, N. et al. (2020).  Named entity recognition and relation detection for biomedical information extraction.  Frontiers in cell and developmental biology, page 673.
  * Peters et al. (2018) Peters, M. E. et al. (2018).  Deep contextualized word representations.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Pyysalo and Ananiadou (2014) Pyysalo, S. and Ananiadou, S. (2014).  Anatomical entity mention recognition at literature scale.  Bioinformatics, 30(6), 868–875.
  * Pyysalo et al. (2013) Pyysalo, S. et al. (2013).  Overview of the cancer genetics (cg) task of bionlp shared task 2013.  In Proceedings of the BioNLP Shared Task 2013 Workshop, pages 58–66.
  * Ramshaw and Marcus (1999) Ramshaw, L. A. and Marcus, M. P. (1999).  Text chunking using transformation-based learning.  In Natural language processing using very large corpora, pages 157–176. Springer.
  * Sung et al. (2022) Sung, M. et al. (2022).  Bern2: an advanced neural biomedical named entity recognition and normalization tool.  arXiv preprint arXiv:2201.02080.
  * Wang et al. (2021) Wang, X. et al. (2021).  Improving named entity recognition by external context retrieving and cooperative learning.  arXiv preprint arXiv:2105.03654.
  * Weber et al. (2021) Weber, L. et al. (2021).  Hunflair: an easy-to-use tool for state-of-the-art biomedical named entity recognition.  Bioinformatics.
  * Wei et al. (2019) Wei, C.-H. et al. (2019).  Pubtator central: automated concept annotation for biomedical full text articles.  Nucleic acids research.
  * Yamada et al. (2020) Yamada, I. et al. (2020).  Luke: deep contextualized entity representations with entity-aware self-attention.  arXiv preprint arXiv:2010.01057.
  * Yu et al. (2020) Yu, J. et al. (2020).  Named entity recognition as dependency parsing.  arXiv preprint arXiv:2005.07150.