# Win-Win Cooperation: Bundling Sequence and Span Models for Named Entity
Recognition

Bin Ji, Jing Yang, Shasha Li, Jie Yu, Jun Ma, Huijun Liu B. Ji, S. Li, J. Yu,
J. Ma, and H. Liu are with the College of Computer, National University of
Defense Technology, Changsha 410073, China.
E-mail: {jibin, yangjing, shashali, yj, majun<EMAIL_ADDRESS>Manuscript received July xx, 2022; revised xx xx, 2022.

###### Abstract

For Named Entity Recognition (NER), sequence labeling-based and span-based
paradigms are quite different. Previous research has demonstrated that the two
paradigms have clear complementary advantages, but few models have attempted
to leverage these advantages in a single NER model as far as we know. In our
previous work, we proposed a paradigm known as Bundling Learning (BL) to
address the above problem. The BL paradigm bundles the two NER paradigms,
enabling NER models to jointly tune their parameters by weighted summing each
paradigm’s training loss. However, three critical issues remain unresolved:
When does BL work? Why does BL work? Can BL enhance the existing state-of-the-
art (SOTA) NER models? To address the first two issues, we implement three NER
models, involving a sequence labeling-based model–SeqNER, a span-based NER
model–SpanNER, and BL-NER that bundles SeqNER and SpanNER together. We draw
two conclusions regarding the two issues based on the experimental results on
eleven NER datasets from five domains. We then apply BL to five existing SOTA
NER models to investigate the third issue, consisting of three sequence
labeling-based models and two span-based models. Experimental results indicate
that BL consistently enhances their performance, suggesting that it is
possible to construct a new SOTA NER system by incorporating BL into the
current SOTA system. Moreover, we find that BL reduces both entity boundary
and type prediction errors. In addition, we compare two commonly used labeling
tagging methods as well as three types of span semantic representations.

###### Index Terms:

bundling learning, named entity recognition, complementary advantages,
sequence labeling, span.

##  1 Introduction

Named entity recognition (NER) is a fundamental task of Natural Language
Processing (NLP) and is a precursor to many downstream NLP tasks such as
relation extraction [1] and coreference resolution [2]. The NER task currently
involves two mainstream technical routines: sequence labeling-based paradigm
[3, 4, 5, 6, 7] and span-based paradigm [8, 9, 10, 11, 12, 13, 14]. As Figure
1 shows, the sequence labeling-based paradigm formulates the problem as a
sequence labeling task, in which each text token is tagged with a label based
on the token-level representation, and entities are derived from these labels.
By contrast, the span-based paradigm formulates the problem as a span
prediction task, which considers text spans as candidate entities and
determines span types directly by carrying out type classifications on span-
level representations.

Figure 1: Comparisons of sequence labeling-based and span-based paradigms for
the NER task. The B-Weather, B-Date, I-Date, and O are BIO labels, Weather and
Date are two pre-defined entity types. None is an added type for spans that
are not entities. “rain” is a Weather entity and “this night” is a Date
entity.

Previous work [11] has proven that the two NER paradigms have clear
complementary advantages. For example, the sequence labeling-based paradigm
performs better when entities are long, while the span-based paradigm is
better at dealing with sentences with more Out-of-Vocabulary (OOV) words.
However, few studies attempt to leverage these complementary advantages within
a single NER model except for our recent study [15]. In the study, we propose
a novel Bundling Learning (BL) paradigm for the NER task. BL bundles sequence
labeling-based and span-based models by (1) making them share a common
encoding layer while keeping their decoding layers unchanged and (2) weighted
summing their training losses to jointly tune the shared layer. Consequently,
the BL paradigm permits us to model the NER task from both sequence labeling
and span prediction perspectives, allowing both token-level and span-level
semantic representations to be used simultaneously. We apply the BL paradigm
to two previously published models, namely SANER (a sequence labeling-based
NER model) [7] and SpERT (a span-based joint entity-relation extraction model)
[16].111For SANER, we bundle a span-based NER model with it. For SpERT, we
bundle a sequence labeling-based NER model with its NER module. Experimental
results show that BL consistently improves the NER performance of SANER and
SpERT. Nevertheless, we remain unexplored on three critical issues: (1) When
does BL work? (2) Why does BL work? In other words, what exhaustive
complementary advantages can BL leverage? (3) Can BL enhance the existing
state-of-the-art (SOTA) NER models?

We investigate the above issues in this paper. To be specific, for issues (1)
and (2), we first implement a span-based NER model–SpanNER and a sequence
labeling-based NER model–SeqNER, and then we bundle them together to obtain a
BL enhanced model–BL-NER. Next, We conduct detailed analyses with these models
on eleven NER datasets. For issue (3), we examine whether BL can enhance the
performance of five existing SOTA NER models. Moreover, we compare SeqNER,
SpanNER, and BL-NER from the perspective of entity prediction error, which
includes entity boundary error and entity type error. In addition, we examine
the effectiveness of CRF-based and softmax-based label tagging. As well as
this, we investigate three different methods for obtaining span-level semantic
representations, namely boundary, span-pooling, and hybrid.

Experimental results indicate that: (1) BL-NER outperforms both SeqNER and
SpanNER when the two NER models perform closely on NER datasets. When the
performance gap between the two NER models is large, BL-NER surpasses the NER
model that performs worse but has disadvantages over the other model. (2) BL
enables BL-NER to leverage the relative advantages of SeqNER and SpanNER when
it comes to four attributes: entity length, text length, entity label
consistency, and entity density. Additionally, the relative disadvantages of
the two NER models lead to BL-NER’s performance dropping on some attributes.
(3) BL consistently produces performance gains in the five models, suggesting
that BL may be used to construct a new SOTA NER system by applying it to the
current SOTA system. In addition, qualitative experiments reveal that SeqNER
generally suffers fewer entity boundary prediction errors than SpanNER, but
more entity type prediction errors. It is a good thing that BL reduces both
types of errors.

In conclusion, we summarize the contributions as follows: (1) We revisit our
proposed BL paradigm and clarify when it works. (2) We examine the exhaustive
advantages that BL can leverage in detail. (3) We suggest a possible method to
obtain new SOTA NER systems by applying BL to the existing SOTA systems.

##  2 Related Work

We roughly divide the majority of neural models for NER into two categories:
sequence labeling-based and span-based models, where the former adopts the
sequence labeling-based paradigm and the latter uses the span-based paradigm.

###  2.1 Sequence Labeling-based NER Models

Sequence labeling-based models formulate NER as a sequence labeling task.
Almost all these models adopt the encoder-decoder architecture, where the
encoder encodes a token embedding for each text token, and the decoder tags
each token with a label according to its embedding. They then obtain entities
based on token labels. Moreover, they generally use the Conditional Random
Fields (CRF) based decoder, for CRF has proven to produce higher tagging
accuracy in general [3]. Furthermore, they make attempt to use carefully
designed neural architectures as their encoders, involving Recurrent Neural
Network (RNN) [17, 18, 19, 20], Convolutional Neural Network (CNN) [21, 22,
23], Transformer Encoder [24, 25, 26], and Graph Neural Network (GNN) [27, 28,
29], with the goal of generate better token embeddings. Recently, Pre-trained
Language Models (PLMs), such as ELMo [30], BERT [31], RoBERTa [32], and ALBERT
[33], are applied to this research field. These PLMs are taken as the encoders
of a number of NER models [34, 35] directly due to their excellent encoding
ability. In addition, some models make attempts to combine PLMs and other
neural models as their encoders, such as ELMo+LSTM [30, 36, 37] and BERT+LSTM
[38]. In spite of the success of the above models, we demonstrate that they
cannot benefit from the advantages of span-based models, such as the ability
to recognize medium-length entities with higher accuracy. By contrast, the
proposed BL paradigm makes it possible to leverage these advantages in
sequence labeling-based models.

###  2.2 Span-based NER Models

It is a natural defect that sequence labeling-based models cannot deal with
overlapped entities, for they are only allowed to tag each text token with one
label. Span-based models are targeted in the scenario of recognizing
overlapped entities, for they consider text spans as candidate entities that
allow entity overlapping. Sohrab et al. [39] propose the first span-based NER
model as far as we know. The key idea of their model is to enumerate all
possible spans as potential entities and class them with deep neural networks,
where these networks first generate span representations and then use linear
classifiers to classify them. The key idea has become a standard followed by
the latter span-based models. Fu et al. [11] introduce BERT into their span-
based model, and they further use the BiLSTM to obtain better span
representations. Moreover, they investigate using their span-based model as a
NER model combiner. Li et al. [9] propose a span-based model similar to Fu et
al. [11]. The difference is that they use the Attention-Guided Graph
Convolutional Network (AGGCN) [40] rather than BiLSTM to obtain better span
representations. Moreover, their model allows double-checking overlapped
entities and detecting discontinuous entities. Tan et al. [41] demonstrate
that previous span-based models usually perform poorly in entity boundary
detection. Thus they propose to train a boundary detection model and a span-
based model jointly, where both models are built upon BERT embeddings. They
consider the boundary and span classification results when determining whether
a span is an entity. Yu et al. [42] propose a span-based model to deal with
the problem of cascading label prediction errors that exist in sequence
labeling-based models. Moreover, Ouchi et al. [10] propose an instance-based
model. At inference time, their model first enumerates all spans and then
assigns a class label to each span based on its similar spans in the training
set. Yu et al. [43] use a biaffine [44] model to score pairs of start and end
tokens in a sentence, where text spans are restricted by these start-end
pairs. Consequently, their model predicts whether these spans are entities
according to the scores. Additionally, span-based models for joint entity-
relation extraction [45, 46, 47, 48, 16, 1] have been extensively studied, and
their NER modules comply with the above span standard. Span-based models have
shown their effectiveness in the NER task. However, we demonstrate that it is
hard for these models to benefit from advantages brought by sequence labeling-
based models, such as better recognition performance for long entities.
Compared to them, the proposed BL paradigm enables span-based models to
leverage these advantages.

##  3 Models

In this section, we first implement a BL enhanced NER model–BL-NER. To this
end, we propose a sequence labeling-based NER model–SeqNER and a span-based
NER model–SpanNER, then we obtain BL-NER by bundling the two models together
(Section 3.1). Next, we present a model discussion in Section 3.2, which aims
to provide a better understanding of the BL paradigm.

###  3.1 BL-NER

As mentioned above, we obtain BL-NER by bundling SeqNER and SpanNER.
Therefore, we begin by illustrating the implementations of the two bundled
models. As shown in Figure 2, Figure 3, and Figure 4, these three models all
share a common model encoder, so we begin by illustrating the implementation
of the encoder, followed by the two bundled models, and finally, BL-NER.

####  3.1.1 The Common Encoder

We use the BERT model [31] as the common encoder, for it can produce
contextualized embeddings for input texts. Given an input text
$\mathcal{T}=t_{0}t_{1}t_{2}...t_{n}$, where $t_{0}$ is the specific [CLS]
token required by BERT and $t_{i}$ denotes the $i$-$th$ text token, we pass it
to the BERT model. For each $t_{i}$ in $\mathcal{T}$, BERT first tokenizes it
into several sub-tokens with the WordPiece vocabulary [49] to avoid the Out-
of-Vocabulary (OOV) problem. BERT therefore produces a sub-token sequence:

$\mathcal{T}^{\prime}=t_{0}t_{1}^{1}...t_{1}^{j}t_{2}^{1}...t_{2}^{j}......t_{n}^{1}...t_{n}^{j},$
(1)

where $t_{i}^{j}$ denotes the $j$-$th$ sub-token of the $i$-$th$ token and the
value of $j$ varies from token to token.

For each sub-token in $\mathcal{T}^{\prime}$, its representation is the
element-wise addition of WordPiece embedding, positional embedding, and
segment embedding. Then a list of input embeddings $\textbf{\em
H}\in\mathbb{R}^{l\ast h}$ are obtained, where l is the sequence length of
$\mathcal{T}^{\prime}$ and h is the size of hidden units. A series of pre-
trained Transformer [50] blocks then project H into a BERT embedding matrix
(denoted as $\textbf{\em E}_{\mathcal{T}^{\prime}}$):

${\textbf{\em E}_{\mathcal{T}^{\prime}}=[\textbf{\em e}_{0},\textbf{\em
e}_{1}^{1},...,\textbf{\em e}_{1}^{j},\textbf{\em e}_{2}^{1},...,\textbf{\em
e}_{2}^{j},......,\textbf{\em e}_{n}^{1},...,\textbf{\em e}_{n}^{j}]},$ (2)

where $\textbf{\em E}_{\mathcal{T}^{\prime}}\in\mathbb{R}^{(l*d)}$ and $d$ is
the BERT embedding dimension.

For each $t_{i}$ in $\mathcal{T}$, we obtain its BERT embedding $\textbf{\em
e}_{i}$ by applying the max-pooling function to the BERT embeddings of its
sub-tokens:

$\textbf{\em e}_{i}=\textrm{max-pooling}\\{\textbf{\em
e}_{i}^{1},...,\textbf{\em e}_{i}^{j}\\},$ (3)

where $\textbf{\em e}_{i}\in\mathbb{R}^{d}$.

Then we denote the BERT embedding matrix of $\mathcal{T}$ as follows:

${\textbf{\em E}_{\mathcal{T}}=[\textbf{\em e}_{0},\textbf{\em
e}_{1},\textbf{\em e}_{2},...,\textbf{\em e}_{n}]}.$ (4)

Note that $\textbf{\em e}_{0}$, the BERT embedding of the added [CLS] token,
is used as the global-level representation of $\mathcal{T}$, for it is
designed to incorporate the information of the whole text.

####  3.1.2 Implementation of SeqNER

Figure 2: The architecture of SeqNER. Given an input text, the encoder first
obtains the contextualized embeddings. Then the decoder tags each text token
with a label according to the token embedding. For example, it tags the “will”
with the label “O” and the “rain” with the label “B-Weather”. At last, the
Label Combiner obtains entities based on the token labels.

Setup. Figure 2 shows the neural architecture of the SeqNER. Given an input
text $\mathcal{T}=t_{0}t_{1}t_{2}...t_{n}$ and its label sequence
$\mathcal{Y}=y_{0}y_{1}y_{2}...y_{n}$, the encoder obtains its contextualized
embedding matrix $\textbf{\em E}_{\mathcal{T}}=[\textbf{\em e}_{0},\textbf{\em
e}_{1},\textbf{\em e}_{2},...,\textbf{\em e}_{n}]$. Then the decoder tags each
$t_{i}$ with a token label according to its representation $\textbf{\em
e}_{i}$. We use the BIO tagging scheme, which uses “B” to tag the beginning
token of an entity, “I” to tag the other tokens of an entity, and “O” to tag
tokens that do not belong to any entity, as the running example in Figure 2
shows. For the dataset containing $\mathcal{T}$, we assume it defines $m$
entity types. Thus it owns a total of ($2*m+1$) types of token labels, where
each entity type has two label types. We use the symbol $\mathbb{L}$ to denote
the token label set and the $|\mathbb{L}|=2*m+1$.

Seq Decoder. We refer to the decoder to tag token labels as Seq Decoder. We
propose two methods for the label tagging: softmax-based tagging and CRF-based
tagging.

Softmax-based Label Tagging. This method taggs each token label independently.
For each $\textbf{\em e}_{i}$ in $\textbf{\em E}_{\mathcal{T}}$, we first
reduce its dimention using a Feed Forward Netwrok (FFN):

$\hat{\textbf{\em e}}_{i}=\textbf{\em W}\textbf{\em e}_{i}+\textbf{\em b},$
(5)

where $\hat{\textbf{\em e}}_{i}\in\mathbb{R}^{|\mathbb{L}|}$, W and b are
trainable FFN parameters. We then pass $\hat{\textbf{\em e}}_{i}$ to the
softmax function, yielding probability distributions for the token $t_{i}$ on
the token label set $\mathbb{L}$:

${\hat{\boldsymbol{y}}_{i,j}=\frac{\exp(\hat{\textbf{\em
e}}_{i,j})}{\sum_{k=0}^{|\mathbb{L}|}\exp(\hat{\textbf{\em e}}_{i,k})}}.$ (6)

The highest response in $\hat{\boldsymbol{y}}_{i}$ indicates that the
corresponding label type is considered activated.

During the model training, we tune model parameters by minimizing the
following cross-entropy training loss:

${\mathcal{L}_{seqner\textrm{-}s}=-\frac{1}{N_{i}}\sum\limits_{i=1}^{N_{i}}{\boldsymbol{y}}_{{i}}\log\hat{\boldsymbol{y}}_{i}},$
(7)

where $N_{i}$ denotes the number of token instances. $\boldsymbol{y}_{i}$ is
the one-hot vector of gold label type for the token $t_{i}$.

CRF-based Label Tagging. Instead of predicting each token label independently,
the CRF-based labeling tagging considers the correlations between neighbor
labels (a.k.a. label dependencies) and jointly decodes the optimal chain of
labels by using sentence-level tag information. CRF uses a state transition
matrix A to record the sentence-level tag information. Label dependencies may
play a vital role in sequence labeling. As the running example in Figure 2
shows, it not only makes no sense but also is illegal to tag the “night” with
any other labels except for the “I-Date”.

For the input text $\mathcal{T}$, CRF describes the probability of generating
its whole label sequence as follows:

$Pr(\hat{\mathcal{Y}}|\mathcal{T})=\frac{\prod_{i=1}^{n}\psi(\hat{y}_{i-1},\hat{y}_{i},\mathcal{T})}{\sum_{\mathcal{Y}^{\prime}\in\textrm{Y}}\prod_{i=1}^{n}\psi({y}^{\prime}_{i-1},{y}^{\prime}_{i},\mathcal{T})},$
(8)

where $\hat{\mathcal{Y}}=\hat{y}_{0}\hat{y}_{1}\hat{y}_{2}...\hat{y}_{n}$ is a
generic chain of label sequence. Y is the set of all chains of label sequence.
$\psi({y}_{i-1},{y}_{i},\mathcal{T})$ calculates the probability of tagging
the token $t_{i}$ with the label $y_{i}$:

$\psi({y}_{i-1},{y}_{i},\mathcal{T})=(\textbf{\em W}\textbf{\em
e}_{i}+\textbf{\em b})+\textbf{\em A}_{y_{i-1},y_{i}},$ (9)

where we refer to the ($\textbf{\em W}\textbf{\em e}_{i}+\textbf{\em b}$) as
the emission score, which is calculated by applying an FFN to the $\textbf{\em
e}_{i}$. And we refer to the $\textbf{\em A}_{y_{i-1},y_{i}}$ as the
transition score, which is the probability of tagging the token $t_{i}$ with
the label $y_{i}$ under the condition that the token $t_{i-1}$ has been tagged
with the label $y_{i-1}$.

During the model training, we minimize the following negative loglikelihood:

$\mathcal{L}_{seqner-c}=-\log Pr(\hat{\mathcal{Y}}|\mathcal{T}).$ (10)

The training objective and its gradients can be efficiently computed by
dynamic programming. And during the model inference, we use the Viterbi
algorithm to find the optimal chain of label sequence that maximizes the
following likelihood:

$\mathcal{Y}^{*}=\textrm{arg}\max\limits_{\mathcal{Y}^{\prime}\in\textrm{Y}}Pr({\mathcal{Y}^{\prime}}|\mathcal{T}).$
(11)

Label Combiner. This module obtains entities by combining consecutive token
labels of the same entity type. For example, it obtains the Date entity–“this
night” by combining the two labels of Date type, namely the B-Date and I-Date
tagged for the “this” and “night”, respectively.

####  3.1.3 Implementation of SpanNER

Setup. Figure 3 shows the neural architecture of the SpanNER. Given an input
text $\mathcal{T}=t_{0}t_{1}t_{2}...t_{n}$, the Encoder obtains its
contextualized embedding matrix $\textbf{\em E}_{\mathcal{T}}=[\textbf{\em
e}_{0},\textbf{\em e}_{1},\textbf{\em e}_{2},...,\textbf{\em e}_{n}]$. We add
an additional None type to tag spans that are not entities. For the dataset
containing $\mathcal{T}$, we use the symbol $\mathbb{T}$ to denote the set of
the $m$ pre-defined entity types and the added None type, where
$|\mathbb{T}|=m+1$.

Figure 3: The architecture of SpanNER. Given an input text, the encoder first
obtains its contextualized embeddings. Then the Span Spliter enumerates all
text spans and obtain their semantic representations. At last, the Decoder
conducts classifications on these representations. For example, the “rain” is
classified as a “Weather” entity, the “this night” is classified as a “Date”
entity, and all the other spans are classified as the “None” type.

Span Spliter. The Span Splitter first splits the input text into text spans
and then obtains their semantic representations.

Text spans are text segments and can be overlapped. For the text
$\mathcal{T}$, we formulate the definition of text span as follows:

$\mathcal{S}=t_{i}t_{i+1},...,t_{i+j}\quad s.t.\quad 1\leq i\leq i+j\leq n,$
(12)

where the length of $\mathcal{S}$ is ($j+1$). Note that we do not take $t_{0}$
(i.e., the added [CLS] token) into consideration. Additionally, we also set a
length threshold $\epsilon$ to restrict the maximum span length, where
$(j+1)\leq\epsilon$.

We propose three methods to obtain span semantic representations, which we
refer to as boundary, span-pooling, and hybrid.

Boundary. We concatenate the BERT embeddings of span head and tail tokens, and
the span length embedding as the span semantic representation:

$\textbf{\em E}_{\mathcal{S}}=[\textbf{\em e}_{i};\textbf{\em
e}_{i+j};\textbf{\em W}_{j+1}],$ (13)

where $\textbf{\em W}_{j+1}$ is the length embeddings for any span of length
$(j+1)$. The span length embeddings are trained during model training. If a
span solely has one token, we duplicate the token embedding and take them as
the head and tail token embedding, respectively.

Span-pooling. We first apply the max-pooling function to the BERT embeddings
of all the span tokens. We then concatenate the max-pooling results, the
global-level representation of the input text, and the span length embedding
as the span semantic representation:

$\textbf{\em E}_{\mathcal{S}}=[\textrm{max-
pooling}([e_{i},e_{i+1},...,e_{i+j}]);\textbf{\em e}_{0};\textbf{\em
W}_{j+1}],$ (14)

where $\textbf{\em e}_{0}$ is taken as the global-level representation of the
text $\mathcal{T}$, as discussed in Section 3.1.1.

Hybrid. We combine the boundary and span-pooling as the hybrid method, and we
obtain the span semantic representation as follows:

$\textbf{\em E}_{\mathcal{S}}=[\textbf{\em e}_{i};\textbf{\em
e}_{i+j};\textrm{max-pooling}([e_{i},e_{i+1},...,e_{i+j}]);\textbf{\em
e}_{0};\textbf{\em W}_{j+1}].$ (15)

Span Decoder. We use a softmax-based decoder to conduct span classifications
on span semantic representations. Specifically, we first pass $\textbf{\em
E}_{\mathcal{S}}$ through an FFN:

$\hat{\textbf{\em E}}_{\mathcal{S}}=\textbf{\em W}\textbf{\em
E}_{\mathcal{S}}+\textbf{\em b},$ (16)

where $\hat{\textbf{\em E}}_{\mathcal{S}}\in\mathbb{R}^{|\mathbb{T}|}$, W and
b are trainable FFN parameters. Then we feed $\hat{\textbf{\em
E}}_{\mathcal{S}}$ to the softmax function:

${\hat{\boldsymbol{y}}_{\mathcal{S},j}=\frac{\exp(\hat{\textbf{\em
E}}_{\mathcal{S},j})}{\sum_{k=0}^{|\mathbb{T}|}\exp(\hat{\textbf{\em
E}}_{\mathcal{S},k})}},$ (17)

where $\hat{\boldsymbol{y}}_{\mathcal{S}}$ is the predicted probabilities
regarding all the types in $\mathbb{T}$. The highest response in
$\hat{\boldsymbol{y}}_{\mathcal{S}}$ indicates that the corresponding type is
considered activated.

During the model training, we tune model parameters by minimizing the
following cross-entropy loss:

${\mathcal{L}_{spanner}=-\frac{1}{N_{k}}\sum\limits_{k=1}^{N_{k}}{\boldsymbol{y}_{k}}\log\hat{\boldsymbol{y}}_{k}},$
(18)

where ${\boldsymbol{y}}_{k}$ is the one-hot vector of gold span type.
${N_{k}}$ is the number of all span instances.

During the model inference, we remain spans that are predicted as entities.

####  3.1.4 Implementation of BL-NER

Guided by the BL paradigm, we obtain BL-NER by bundling SeqNER and SpanNER
together, as shown in Figure 4.

Figure 4: The architecture of BL-NER, which is achieved by bundling SeqNER and
SpanNER together. The two bundled models share a common model encoder, and
their losses are weighted summed to jointly tune the model parameters.

Given an input text $\mathcal{T}=t_{0}t_{1}t_{2}...t_{n}$, the Encoder obtains
its contextualized embedding matrix $\textbf{\em E}_{\mathcal{T}}=[\textbf{\em
e}_{0},\textbf{\em e}_{1},\textbf{\em e}_{2},...,\textbf{\em e}_{n}]$. Then
$\textbf{\em E}_{\mathcal{T}}$ is shared by the Span Splitter (➁) and the Seq
Decoder (➃). Based on $\textbf{\em E}_{\mathcal{T}}$ and $\mathcal{T}$, the
Span Splitter and the Span Decoder (➀) conduct the same operations as they do
in SpanNER (Section 3.1.3). Similarly, the Seq Decoder (➃) and the Label
Combiner (➂) do the same thing as they do in SeqNER (Section 3.1.2).

During the model training, we weighted sum the traing losses of SeqNER and
SpanNER to jointly tune the shared common encoder:

$\mathcal{L}_{bl\textrm{-}ner}=\alpha*\mathcal{L}_{seqner}+(1-\alpha)*\mathcal{L}_{spanner},$
(19)

where $\alpha$ is a loss weight.

Note that both the Span Decoder (➀) and the Label Combiner (➂) can output
entities. During the model inference, we use only one of them to output
entities with the goal of avoiding model conflicts. And if we use the Span
Decoder, we refer to BL-NER as BL-SpanNER, which means that we use BL to
enhance SpanNER by bundling SeqNER to it. Similarly, if we use the Label
Combiner, we refer to BL-NER as BL-SeqNER, indicating that we use BL to
enhance SeqNER by bundling SpanNER to it.

###  3.2 Model Discussion

####  3.2.1 Why Bundle the Two NER models?

The sequence labeling-based and span-based paradigms are regarded as two
distinct technical routines for the NER tasks [15, 11, 51], but they exhibit
similar performance across a number of datasets, such as SciERC [46],
CoNLL2003 [52], and OntoNotes [53]. Moreover, we demonstrate that the two
paradigms depend on different granularities of semantic representation, i.e.,
the sequence labeling-based one is built upon token-level representations
while the span-based one is based upon span-level representations. It is
obvious that these two types of representations focus on different
perspectives, namely, the token-level representation pays more attention to
itself and the correlation with its neighbors, while the span-level
representation prefers to consider a sequence of representations as a whole
and is more concerned with the correlation among them. The above difference
results in quite different NER results on a number of attributes [11]. For
example, the sequence labeling-based paradigm performs better when entities
are long, while the span-based paradigm is better at dealing with sentences
with more OOV words.

The motivation of the BL paradigm is that a NER model will make use of the
complementary advantages of the two NER paradigms if it can leverage both
token-level and span-level representations simultaneously. When the two
paradigms share a common encoder, and their losses are used to jointly tune
the encoder’s parameters, contextualized embeddings will be generated that
incorporate token-level as well as span-level representations.

####  3.2.2 Why Set the Span Length Threshold?

A number of span-based NER models [9, 13, 11] enumerate all text spans during
the model training. However, we demonstrate that using all spans is not only
unnecessary but also wasteful. For example, if the length ($n$) of the text
$\mathcal{T}$ is 100, a total of 5,050 spans will be enumerated, which is
calculated by $\frac{n*(n+1)}{2}$. However, only a very few spans are with
lengths larger than 10, such as only 0.04% of the entities in OntoNotes [53].
Moreover, considering all spans needs vast GPU memory and takes a long time to
train the model. By contrast, if we set the span length threshold $\epsilon$
to 10, only 955 spans will be enumerated, which is calculated by
$\frac{\epsilon*(2n-\epsilon+1)}{2}$.

However, the majority of the 955 spans (often more than 900) are not entities,
which we refer to as negative spans and assign “None” type to them. Eberts and
Ulges [16] and Yu et al. [42] have proven that a small number of negative
spans is enough for the model training, and they propose a maximum negative
sampling strategy to randomly sample a number of negative spans. We also use
the proposed strategy. Given the text $\mathcal{T}$, we use the symbol
$\mathbb{S}$ to represent the set of all negative spans restricted by
$\epsilon$, and the symbol $\mathbb{D}$ to represent the set of sampled
negative spans:

$|\mathbb{D}|=\min\\{|\mathbb{S}|,100\\},$ (20)

where $|\cdot|$ represents the number of spans in the set.

During the model inference, we enumerate all spans restricted by $\epsilon$
and predict their types.

####  3.2.3 Heuristic Decoding

We only take the flat NER task into consideration, which assumes no overlapped
entities. We find that SpanNER and BL-SpanNER may predict overlapped entities.
For those overlapped spans, we use the heuristic decoding method proposed by
Fu et al. [11] to remove them, where we keep the span with the highest
prediction probability and drop the others.

##  4 Experiments

###  4.1 Experimental Setup

####  4.1.1 Datasets

To provide empirical evidence for the effectiveness of the proposed BL
paradigm, we conduct our experiments on eleven English datasets across the
following five domains:

General Domain.

  * •

OntoNotes 5.0 (OntoNotes) [53] dataset is a large corpus comprising various
genres of text (news, broadcast, weblogs, usenet newsgroups, conversational
telephone speech, talk shows).

News Domain.

  * •

CoNLL2003 (CoNLL03) [52] dataset is one of the widely used NER datasets. Its
annotation guideline is based on MUC Conventions [52]. This dataset has been
split into training, development, and test sets, with 14041, 3250, and 3453
sentences, respectively.

  * •

CoNLL++ [54] is a re-annotated version of the CoNLL03 dataset. Wang et al.
[54] fix annotation errors of the test set and improve the quality of the
training data through their CrossWeigh approach.

  * •

CoNLL2004 (CoNLL04) [55] dataset consists of 1,441 sentences from news
articles annotated with four entity types (Location, Organization, People, and
Other).

  * •

ACE2004 (ACE04) [56] and ACE2005 (ACE05) [57] English dataset composes of news
articles in multi-domain such as broadcast, newswire, and weblog. Each of them
contains seven entity categories. In addition, we remove the overlapped
entities from the two datasets and solely conduct experiments on those
entities not overlapped.

Medical Domain.

  * •

BC5CDR [58] dataset is proposed for the BioCreative V Chemical Disease
Relation task and is composed of 1500 PubMed articles. It contains chemical,
disease, and chemical-disease interaction entities.

  * •

NCBI [59] is a disease corpus of the National Center for Biotechnology
Information (NCBI). The dataset is a collection of 793 PubMed abstracts
annotated at the mention and concept level.

Scientific Domain.

  * •

SciERC [46] dataset is collected from 500 AI paper abstracts and defines
scientific terms and relations specially for scientific knowledge graph
construction.

Social Domain.

  * •

WNUT2016 (W16) [60] is a dataset focusing on NER over Twitter. Its training
and development data are taken from prior work on Twitter NER [61, 62], and
its test data is new and is collected from general Twitter data and domain-
specific Twitter data.

  * •

WNUT2017 (W17) [63] focuses on identifying unusual, previously unseen
entities. It covers data from multiple social media platforms. Specifically,
its training set uses tweets, its development set is based on YouTube
comments, and its testing set combines content from Reddit and StackExchange.
The cross domain nature of the dataset establishes an additional challenge to
the task.

For ACE04 and ACE05, we regard an entity mention as correct if its label and
the head region of its span are identical to the ground truth. For the other
datasets, we regard an entity mention as correct if its label and span match
the ground truth. We report more dataset details in Table I.

TABLE I: Dataset details, where the “# Train”, “# Dev”, and “#Test” columns record the number of entities in the training, development, and test sets, respectively; the “# Type” column records the number of entity types. Dataset | # Train | # Dev | # Test | # Type
---|---|---|---|---
OntoNotes | 81,828 | 11,066 | 11,257 | 18
CoNLL03 | 23,499 | 5,942 | 5,648 |  4
CoNLL++ | 23,499 | 5,942 | 5,702 |  4
CoNLL04 | 3,377 | 893 | 1,079 |  4
ACE04 | 12,055 | 1,422 | 1,618 |  7
ACE05 | 15,052 | 2,088 | 1,875 |  7
BC5CDR | 9,385 | 9,353 | 9,809 |  2
NCBI | 5,429 | 923 | 941 |  1
SciERC | 5,598 | 811 | 1,685 |  6
W16 | 2,104 | 661 | 3,473 | 10
W17 | 1,975 | 835 | 1,079 |  6

Discussion. We would like to emphasize a specific characteristic of W16 and
W17: the two datasets focus on detecting and classifying novel and emerging
named entities in noisy text. Thus their test sets include data from new
domains or social platforms, such as new cybersecurity and mass shootings
domains for W16, and new Reddit and StackExchange platforms for W17. However,
no new domains or platforms data is specifically included in the training or
development sets.

There is no doubt that the new data contained in the tests leads to a larger
OOV density [64], which is defined as follows:

$\phi_{\texttt{oDen}}=\frac{\textrm{oov}(\mathcal{X})}{|\mathcal{X}|},$ (21)

where $\mathcal{X}$ is the sentences in the test set.
$\textrm{oov}(\mathcal{X})$ is the number of words in $\mathcal{X}$ but not in
the sentences of the training set. $|\mathcal{X}|$ is the number of all words
in $\mathcal{X}$.

Fu et al. [11] have proven that span-based NER models do better in sentences
with more OOV words, which is also confirmed by our experiment, where SpanNER
significantly outperforms SeqNER on the two datasets (see Section 4.2).

####  4.1.2 Implementation Details

All the experiments are conducted on a single NVIDIA RTX 3090 GPU. We optimize
all models using the AdamW for 20 epochs with a learning rate of 5$e^{-5}$, a
drop out of 0.1, a linear scheduler with a warm-up ratio of 0.1, and a weight
decay of 1$e^{-2}$. Moreover, we report three dataset-specific parameters in
Table II. Specifically, we use the bert-large-cased [65] as the BERT-large,
the biobert-pubmed-pmc [66] as the BioBERT, and the scibert_scivocab_cased
[31] as the SciBERT.

In addition, we set the loss weight $\alpha$ to 0.1 for BL-NER and use the
softmax-based label tagging for SeqNER, and we adopt the hybrid method to
obtain span semantic representations. We will conduct detailed investigations
on the above three parameters in Section 4.5 to reduce content redundancy
here.

TABLE II: Dataset-specific parameters. We determine the span length threshold ($\epsilon$) value for each dataset based on the principle that over 99% of entities in the dataset are less than or equal to the value.  Dataset | $\epsilon$ | batch size | BERT
---|---|---|---
OntoNotes | 10 | 12 | BERT-large
CoNLL2003 |  5 | 32 | BERT-large
CoNLL++ |  5 | 32 | BERT-large
CoNLL04 | 10 |  2 | BERT-large
ACE04 | 10 |  8 | BERT-large
ACE05 | 10 |  8 | BERT-large
BC5CDR |  7 | 16 | BioBERT
NCBI |  7 | 16 | BioBERT
SciERC |  8 | 12 | SciBERT
W16 |  6 | 32 | BERT-large
W17 |  6 | 24 | BERT-large

####  4.1.3 Evaluation Metrics

We use the standard Precision (P), Recall (R), and micro-F1 to evaluate the
model performance:

P $\displaystyle=\frac{\textrm{TP}}{\textrm{TP+FP}},$ (22a) R
$\displaystyle=\frac{\textrm{TP}}{\textrm{{TP}+{FN}}},$ (22b) F1
$\displaystyle=\frac{{2*\textrm{P}*\textrm{R}}}{\textrm{{P}+{R}}},$ (22c)

where TP, FP, and FN stand for true positive, false positive, and false
negative, respectively. For all results, we report the averaged Precision,
Recall, and F1 based on five runs with different seeds.

###  4.2 Main Results

TABLE III: Performance comparisons between original NER models and their BL enhanced version. ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\uparrow}$ (${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\downarrow}$) represents the BL paradigm increases (decreases) the model performance. Dataset | Model | P | R | F1
---|---|---|---|---
OntoNotes | SeqNER | 89.56 | 88.44 | 89.00
BL-SeqNER | 89.31 | 90.14 |  89.72 $\uparrow$
SpanNER | 89.67 | 89.30 | 89.49
BL-SpanNER | 88.95 | 90.64 |  89.78 $\uparrow$
CoNLL2003 | SeqNER | 92.06 | 91.36 | 91.71
BL-SeqNER | 92.12 | 92.64 |  92.38 $\uparrow$
SpanNER | 92.70 | 91.13 | 91.91
BL-SpanNER | 92.52 | 92.50 |  92.51 $\uparrow$
CoNLL++ | SeqNER | 93.41 | 89.92 | 91.63
BL-SeqNER | 93.60 | 92.81 |  93.20 $\uparrow$
SpanNER | 93.89 | 91.33 | 92.59
BL-SpanNER | 93.80 | 92.79 |  93.29 $\uparrow$
CoNLL04 | SeqNER | 91.45 | 91.20 | 90.62
BL-SeqNER | 91.69 | 91.01 |  91.35 $\uparrow$
SpanNER | 91.88 | 89.92 | 90.89
BL-SpanNER | 92.42 | 90.87 |  91.64 $\uparrow$
ACE04 | SeqNER | 88.03 | 85.64 | 86.82
BL-SeqNER | 88.45 | 87.38 |  87.91 $\uparrow$
SpanNER | 87.12 | 88.15 | 87.63
BL-SpanNER | 89.23 | 87.96 |  88.59 $\uparrow$
ACE05 | SeqNER | 84.45 | 87.94 | 86.16
BL-SeqNER | 86.98 | 87.06 |  87.02 $\uparrow$
SpanNER | 88.96 | 86.94 | 87.94
BL-SpanNER | 88.78 | 89.26 |  89.02 $\uparrow$
BC5CDR | SeqNER | 90.47 | 89.12 | 89.79
BL-SeqNER | 89.78 | 90.91 |  90.34 $\uparrow$
SpanNER | 89.37 | 90.46 | 89.91
BL-SpanNER | 90.05 | 90.74 |  90.40 $\uparrow$
NCBI | SeqNER | 88.91 | 89.49 | 89.20
BL-SeqNER | 88.57 | 91.73 |  90.12 $\uparrow$
SpanNER | 90.30 | 89.72 | 90.01
BL-SpanNER | 89.96 | 90.99 |  90.49 $\uparrow$
SciERC | SeqNER | 67.26 | 72.05 | 69.57
BL-SeqNER | 68.98 | 71.51 |  70.22 $\uparrow$
SpanNER | 70.02 | 70.91 | 70.46
BL-SpanNER | 70.32 | 71.99 |  71.14 $\uparrow$
W16 | SeqNER | 60.23 | 55.06 | 57.53
BL-SeqNER | 59.57 | 57.78 |  58.66 $\uparrow$
SpanNER | 63.26 | 58.23 | 60.64
BL-SpanNER | 61.43 | 57.37 |  59.86 $\downarrow$
W17 | SeqNER | 44.39 | 62.24 | 51.82
BL-SeqNER | 47.26 | 59.35 |  52.62 $\uparrow$
SpanNER | 60.32 | 50.19 | 54.79
BL-SpanNER | 56.63 | 51.35 |  53.86 $\downarrow$
TABLE IV: Performance comparisons with previous strong baselines. Dataset | Model | P | R | F1
---|---|---|---|---
| Multi-turn QA [67] | 89.00 | 86.60 | 87.80
| SpERT [16] | 88.25 | 89.64 | 88.94
| Table-Sequence [68] | - | - | 90.10
| TriMF [69] | 90.26 | 90.34 | 90.30
| TablERT [70] | - | - | 91.30
CoNLL04 | BL-SpanNER | 92.42 | 90.87 | 91.64
| BERT-CRF [71] | - | - | 86.00
| RDANER [72] | - | - | 87.38
| SparkNLP [73] | - | - | 89.73
| ELECTRAMed [74] | 88.76 | 91.34 | 90.03
| BioLinkBERT [75] | - | - | 90.22
BC5CDR | BL-SpanNER | 90.05 | 90.74 | 90.40
| ELECTRAMed [74] | 85.87 | 89.29 | 87.54
| RDANER [72] | - | - | 87.89
| BioLinkBERT [75] | - | - | 88.76
| CL-KL [76] | - | - | 89.24
| SciFive-Base [77] | 88.65 | 90.14 | 89.39
| BioBERT [66] | 88.22 | 91.25 | 89.71
NCBI | BL-SpanNER | 89.96 | 90.99 | 90.49
| SciBERT [65] | - | - | 67.57
| PURE [78] | - | - | 68.90
| RDANER [72] | - | - | 68.96
| SpERT [16] | 70.87 | 69.79 | 70.33
SciERC | BL-SpanNER | 70.32 | 71.99 | 71.14
| CambridgeLTL [79] | 60.77 | 46.07 | 52.41
| InferNER [80] | - | - | 53.48
| SANER [7] | - | - | 55.01
| CL-LK [76] | - | - | 58.98
| S-NER [42] | - | - | 60.12
W16 | SpanNER | 63.26 | 58.23 | 60.64

As mentioned in Section 3.1.4, we refer to BL-NER as BL-SpanNER when using the
Span Decoder (Figure 4-➀) to output entities. Similarly, we refer to BL-NER as
BL-SeqNER when using the Label Combiner (Figure 4-➂) to output entities. In
this section, we conduct experiments to investigate the effectiveness of the
BL paradigm by comparing SeqNER to BL-SeqNER, as well as SpanNER to BL-
SpanNER. We report the comparison results in Table III, from which we observe
that:

(1) BL-SeqNER consistently outperforms SeqNER across the eleven datasets.
Specifically, BL-SeqNER brings an averaged +0.88% F1 score and a maximum of
+1.57% F1 score (on CoNLL++) compared to SeqNER. These gains reveal that the
BL paradigm always boosts the performance of SeqNER.

(2) BL-SpanNER beats SpanNER on the first-nine datasets. To be more precise,
BL-SpanNER beats SpanNER by an averaged +0.67% F1 score and a maximum of
+0.96% F1 score (on ACE04) across the nine datasets. BL-SpanNER instead
decreases -0.78% and -0.93% F1 scores on W16 and W17, respectively.

(3) SpanNER shows a consistent superiority to SeqNER across the eleven
datasets, especially on W16 and W17. Statistically, SpanNER outperforms SeqNER
by an averaged +0.70% F1 score across the first-nine datasets. We attribute
these results primarily to the fact that SpanNER uses the effective hybrid
method to obtain span representations, which is an overall consideration of
the structure, boundary, length, and context of spans. And the performance
gains are +3.11% and +2.97% on W16 and W17, respectively. We demonstrate that
the large OOV density of the two datasets has the most significant effect on
the results (see the discussion in Section 4.1.1).

Based on the above observations, we draw a conclusion regarding when the BL
paradigm works:

  * •

Conclusion #1: The BL paradigm enhances the performance of both SeqNER and
SpanNER when the two NER models perform closely on NER datasets. However, if
there is a large performance gap between the two NER models on NER datasets,
BL will boost the model that performs poorly while decreasing the other model.

The above conclusion is also consistent with human intuition. Moreover, we
demonstrate that these performance gains are due to the fact that the BL
paradigm allows the NER models to leverage their complementary advantages. We
will present detailed analyses of the exhaustive complementary advantages in
Section 4.3.

In addition, BL-SpanNER achieves competing results on CoNLL04, BC5CDR, NCBI,
and SciERC. These results also validate the effectiveness of the BL paradigm.
Moreover, SpanNER creates the current best performance on W16. We report these
performance comparisons in Table IV.

###  4.3 Why Does the BL Paradigm Work?

This section examines in detail the reasons for the success of the BL
paradigm. We select the largest dataset of each domain for the investigation,
and we actually select both W16 and W17 of the social domain due to their
specific characteristic. The selected datasets are OntoNotes (general),
CoNLL03 (news), BC5CDR (medical), SciERC (scientific), and W16 and W17
(social), and we use their dev sets to report model performance.

####  4.3.1 What Complementary Advantages Does BL Leverage?

The holistic results in Table III show the effectiveness of the BL paradigm,
but they cannot interpret the complementary advantages of SeqNER and SpanNER
that BL can leverage. Fu et al. [64] propose an interpretable evaluation idea
by breaking the holistic performance into different buckets from a number of
perspectives and using performance heatmaps to investigate relative advantages
between two systems. In this section, we make attempts to explore the
complementary advantages using their evaluation idea.

Setup. Following Fu et al. [64], we break the holistic performance into four
groups based on different attributes. To be specific, given an entity $e$ that
belongs to a text $\mathcal{T}$, we define the following attribute feature
functions:

$\displaystyle\phi_{\texttt{eLen}}$ $\displaystyle=\textrm{len}(e),$ (23a)
$\displaystyle\phi_{\texttt{tLen}}$ $\displaystyle=\textrm{len}(\mathcal{T}),$
(23b) $\displaystyle\phi_{\texttt{eCon}}$
$\displaystyle=\frac{|\\{\varepsilon\mid\operatorname{label}(\varepsilon)=\operatorname{label}(e),\forall\varepsilon\in\mathcal{E}\\}|}{|\mathcal{E}|},$
(23c) $\displaystyle\phi_{\texttt{eDen}}$
$\displaystyle=\frac{\textrm{ent}(\mathcal{T})}{\phi_{\texttt{tLen}}},$ (23d)

where $\phi_{\texttt{eLen}}$ denotes entity length; $\phi_{\texttt{tLen}}$
denotes text length; $\phi_{\texttt{eCon}}$ denotes entity label consistency
and $\mathcal{E}$ denotes all entities in the training set;
$\phi_{\texttt{eDen}}$ denotes entity density and $\textrm{ent}(\mathcal{T})$
calculates the number of entities in the text $\mathcal{T}$.

The $\phi_{\texttt{eCon}}$ attribute measures the consistency that a
particular entity is labeled with a unified label. For example, if the “rain”
in the running example of Figure 1 has a higher entity label consistency, it
means that the “rain” is frequently labeled as the “Weather” type in the
training set.

Based on the values of the four attributes, we divide test entities into four
buckets, namely extra-small (XS), small (S), large (L), and extra-large (XL).
We report the detailed attribute intervals for the four attributes in Table V.

TABLE V: The attribute intervals of bucket XS, S, L, and XL for attributes eCon, tLen, eCon, and eDen.  Bucket | eLen | tLen | eCon | eDen
---|---|---|---|---
XS | [1] | [1, 7] | [0, 0.1] | [0, 0.01]
S | [2] | [8, 16] | (0.1, 0.5] | (0.01, 0.025]
L | [3, 4] | [17, 31] | (0.5, 0.9] | (0.025, 0.05]
XL | [5, ] | [32, ] | (0.9, 1] | (0.05, 1]

We take all four models, i.e., SpanNER, BL-SpanNER, SeqNER, and BL-SeqNER,
into consideration. And we analyze the complementary advantages through three
model pairs, i.e., $<$SpanNER, SeqNER$>$, $<$SpanNER, BL-SpanNER$>$, and
$<$SeqNER, BL-SeqNER$>$. Specifically, (1) the $<$SpanNER, SeqNER$>$ enables
us to find out the relative advantages of the two models; (2) the $<$SpanNER,
BL-SpanNER$>$ allows us to conclude what advantages that BL delivers from
SeqNER to SpanNER; (3) similarly, we can be aware of the advantages that BL
transfers from SpanNER to SeqNER through the $<$SeqNER, BL-SeqNER$>$. For all
four models, we report their bucket-wise F1 scores regarding the four defined
attributes.

TABLE VI: Performance heatmaps regarding complementary advantages. For clarity, SpanNER is taken as an standard, where each value in its heatmap entry is 0. For the other models, each value in heatmap entry $(i,j)$ represents the performance gap (measured by F1 score) between the model and SpanNER on $j$-th bucket, and the green (red) area indicates that the model performs better (worse) than SpanNER. eLen, tLen, eCon, and eDen represent different attributes.  OntoNotes | CoNLL03
---|---
eLen | tLen | eCon | eDen | eLen | tLen | eCon | eDen
|  |  |  |  |  |  |
BC5CDR | SciERC
eLen | tLen | eCon | eDen | eLen | tLen | eCon | eDen
|  |  |  |  |  |  |
W16 | W17
eLen | tLen | eCon | eDen | eLen | tLen | eCon | eDen
|  |  |  |  |  |  |

Analysis. We report performance visualization results in Table VI. We can
observe that:

  * •

$<$SpanNER, SeqNER$>$. SpanNER shows clear complementary advantages with
SeqNER. On the first four datasets, (1) SpanNER generally outperforms SeqNER
when dealing with medium and long texts, short and medium entities, as well as
large entity label consistency and entity density; (2) in contrast, SeqNER
shows better performance than SpanNER when texts are short with long entities
annotated, as well as small entity label consistency and entity density. On
W16 and W17, we find that SpanNER significantly outperforms SeqNER in almost
all cases. We attribute this to the fact that the test sets of two datasets
have a larger OOV density (see the discussion of Section 4.1.1), and SpanNER
is better at dealing with this scenario.

  * •

$<$SpanNER, BL-SpanNER$>$. The BL paradigm enables SpanNER to leverage almost
all the complementary advantages of SeqNER on the first four datasets. For
example, BL-SpanNER shows obvious performance gains over SpanNER when entities
are long and texts are short. In some cases, however, BL-SpanNER shows slight
performance declines, such as for entities with medium lengths and low
densities. While on W16 and W17, we observe that BL-SpanNER performs worse
than SpanNER on almost all buckets of the four attributes. We attribute this
to the severe disadvantages of SeqNER on the two datasets, suggesting that the
BL paradigm can deliver disadvantages from one model to another.

  * •

$<$SeqNER, BL-SeqNER$>$. The BL paradigm allows SeqNER to leverage almost all
the complementary advantages of SpanNER on all six datasets. For example, BL-
SeqNER explicitly performs better than SeqNER in the case of short entities
and long texts, and large entity density on the first four datasets. On W16
and W17, we can see that BL-SeqNER obviously outperforms SeqNER in almost all
cases. Moreover, we discover that the disadvantages of SpanNER have
imperceptive effects on SeqNER across all six datasets, which is attributed to
the fact that the holistic better performance of SpanNER weakens the negative
effects of these disadvantages.

Based on the above observations, we draw the following conclusion:

  * •

Conclusion #2: The BL paradigm enables SeqNER and SpanNER to leverage their
relative advantages that cover four attributes: entity length, text length,
entity label consistency, and entity density. Additionally, the BL paradigm
can deliver disadvantages from one model to another, leading to performance
dropping on some attributes.

####  4.3.2 How Does BL Affect Entity Prediction Errors?

In this section, we investigate how the BL paradigm affects two common entity
prediction errors, which are boundary error (B.E.) and type error (T.E.). We
formalize their definitions as follows:

  * •

Boundary Error. If a predicted entity contains more or fewer tokens than a
ground truth entity, we call the unaligned entity boundary a boundary error.

  * •

Type Error. If a predicted entity and a ground truth entity have the same
boundary but different types, we call the mistakenly predicted type a type
error.

Take the running example in Figure 1 as an example. Two ground truth entities,
i.e., “rain” of the Weather type and “this night” of the Date type, are
annotated in the text “will it rain this night”. A boundary error happens if
the “it rain” is predicted as a Weather entity. The same goes for “will it
rain”, ”rain this” etc. And if the ”rain” is predicted as any other type
except for the Weather, a type error happens.

We calculate the rates of the two errors as follows:

BE-Rate $\displaystyle=\frac{\textrm{B.E.}}{\textrm{FP}+\textrm{FN}},$ (24a)
TE-Rate $\displaystyle=\frac{\textrm{T.E.}}{\textrm{FP}+\textrm{FN}},$ (24b)

where we use the B.E. and T.E. to denote the instance number of the two
errors, respectively.

We report the error analyses in Table VII. We can observe that: (1) SpanNER
shows much smaller type error rates than SeqNER, while SeqNER deals with
boundary errors much better than SpanNER. These results reveal that the two
NER models have complementary advantages regarding entity boundary and entity
type predictions. (2) BL-SpanNER generally reduces both errors compared to
SpanNER on the first four datasets. And similar results can also be seen on
BL-SeqNER compared to SeqNER. These results demonstrate that the BL paradigm
enables the two NER models to leverage their relative complementary
advantages. (3) On W16 and W17, the BL paradigm reduces the two errors in
SeqNER but increases the two errors in SpanNER. Similar phenomena have been
observed in Section 4.2 and 4.3.1, which we attribute to the specific
characteristic of the two datasets (see the discussion in Section 4.1.1).

TABLE VII: Performance heatmaps regarding entity prediction errors. SpanNER is taken as a standard. The green (red) area indicates that the error rate is higher (lower) than SpanNER. OntoNotes | CoNLL03 | BC5CDR | SciERC |  W16 | W17
---|---|---|---|---|---
|  |  |  |  |

###  4.4 Applying BL to Existing SOTA NER Models

In this section, we explore whether the BL paradigm works in existing SOTA NER
models. Specifically, for those NER models adopting the sequence labeling-
based paradigm, we add the Span Splitter and the Span Decoder module to the
top of the model’s encoder, which is similar to Figure 4-➁&➀. We regard the
combination of the two added modules and the encoder as a span-based NER
model. For those NER models adopting the span-based paradigm, we add the Seq
Decoder and the Label Combiner module to the top of the model’s encoder, which
is similar to Figure 4-➃&➂. We regard the combination of the two added modules
and the encoder as a sequence labeling-based NER model. We then sum the losses
of two bundled models to jointly train the encoder parameters. We set the loss
weight $\alpha$ to 0.1, which is the same as the one described in Section
4.1.2.

We select five existing SOTA NER models for the exploration. Three are
sequence labeling-based models, which are named RDANER [72], CL-KL [76], and
Hire-NER [81]. Two are span-based models, which are named SodNER [9] and
DyGIE++ [48]. Note that DyGIE++ is a span-based model for joint entity and
relation extraction, and we just use its NER part for the exploration.
Moreover, SodNER takes overlapped and discontinuous entities into account, and
we exclude these entities during the exploration. For each of the above NER
models, we compare it with its BL enhanced model, where the two models use the
same method to output entities for a fair comparison.

We report the model performance (measured by F1) in Table VIII, from which we
observe that the BL paradigm consistently improves the performance of the five
NER models. Specifically, it brings +0.15% to +1.73% F1 scores to the three
sequence labeling-based NER models, and it boosts the two span-based NER
models by +0.31% to +1.56% F1 scores.

Based on the above results, we conclude that given a current SOTA NER system
that adopts either the sequence labeling-based or the span-based paradigm, the
BL paradigm makes it possible to obtain a new SOTA system without requiring
additional resources, such as external data annotations.

TABLE VIII: Results of applying the BL paradigm to five existing SOTA NER models, where Seq denotes sequence labeling-based models and Span denotes span-based models. ${\dagger}$ represents we reproduce the results by removing overlapped or discontinuous entities from the datasets. ${\ddagger}$ represents we only use the NER part of the model. Seq | Model | SciERC | BC5CDR | NCBI
---|---|---|---|---
RDANER | 68.96 | 87.39 | 87.89
BL-RDANER | 69.44 | 89.12 | 88.47
Model | CoNLL03 | BC5CDR | NCBI
CLKL | 93.21 | 90.73 | 89.24
BL-CLKL | 93.86 | 90.88 | 89.94
Model | CoNLL02 | CoNLL03 | OntoNotes
HireNER | 87.08 | 93.37 | 90.30
| BL-HireNER | 88.16 | 93.68 | 90.84
Span | Model | CLEF | CADEC | ACE05
SodNER |  82.42 ${\dagger}$ |  60.38 ${\dagger}$ |  88.58 ${\dagger}$
BL-SodNER | 83.17 | 60.84 | 88.89
Model | ACE05 | SciERC | GENIA
DyGIE++ ${\ddagger}$ |  88.96 ${\dagger}$ | 67.23 |  72.24 ${\dagger}$
| BL-DyGIE++ | 89.44 | 68.79 | 73.12

###  4.5 Performance against Model variants

The purpose of this section is to examine the correlation between model
performance and three model variants. We conduct explorations on the six
datasets selected in Section 4.3.

####  4.5.1 Performance against the Loss Weight

The loss weight ($\alpha$, see Eq. 19) is a critical hyperparameter that
controls the contributions of SeqNER (the weight score is $\alpha$) and
SpanNER (the weight score is 1-$\alpha$) to BL-NER, and we demonstrate that
the larger the weight score, the larger the model contribution. When setting
$\alpha$ to 0, SeqNER contributes nothing to BL-NER. At this moment, BL-NER is
SpanNER in essence. Similarly, BL-NER is SeqNER when setting $\alpha$ to 1.

To explore the influence of various $\alpha$ acores, we run BL-NER with
setting $\alpha$ to 0.1, 0.2, 0.3,…, 0.9 and report the performance of both
BL-SeqNER and BL-SpanNER, as shown in Figure 5. We have the following
observations: (1) Performance of the two models generally decreases when the
$\alpha$ score consistently increases. (2) When setting the $\alpha$ score to
0.1, BL-SeqNER performs the best on five of the six datasets, and BL-SpanNER
shows the best performance on three of the six datasets.

We attribute the above observations to the fact that SpanNER consistently
surpasses SeqNER on the six datasets (see Table III). Thus a larger
contribution of SpanNER ensures a better model performance, and a minor
contribution of SeqNER allows the two models to leverage their complementary
advantages. For simplicity, we set $\alpha$ to 0.1 in all the other
experiments.

Figure 5: Performance against various loss weight ($\alpha$) scores.

####  4.5.2 Performance against the Label Tagging

In Section 3.1.2 we propose two label tagging methods: softmax-based and CRF-
based. In this section, we investigate their effectivenesses in both BL-SeqNER
and BL-SeqNER. Since BL-SpanNER can be affected by the bundled SeqNER model,
we also take BL-SpanNER into consideration.

We report the investigation results in Figure 6, from which we observe that:
(1) For SeqNER, the CRF-based tagging performs better than the softmax-based
tagging on two of the six datasets. And for BL-SeqNER, the CRF-based tagging
beats the softmax-based tagging on four datasets. These results indicate that
the complex CRF-based tagging does not always lead to better performance in
comparison with simple softmax-based tagging, which conforms to the conclusion
drawn by Hanh et al. [82]. As for BL-SpanNER, the CRF-based tagging
consistently performs the best across the six datasets. (2) Despite the CRF-
based tagging showing advantages in most cases, it actually delivers slight
performance gains (averaged +0.11% F1).

Moreover, we find that the CRF-based tagging takes much more time than the
softmax-based tagging. In all the other experiments, we actually use the
softmax-based tagging, which is a trade-off between the model performance and
the training efficiency.

Figure 6: Performance against various label tagging methods.

####  4.5.3 Performance against the Span Representation

In Section 3.1.3, we design three methods to obtain span semantic
representations. In this section, we conduct experiments on these methods to
explore their effectiveness. Specifically, we use SpanNER, BL-SpanNER, and BL-
SeqNER for the exploration. It is obvious that BL-SpanNER and BL-SpanNER are
directly affected by these methods. We demonstrate that BL-SeqNER can also be
affected indirectly due to its bundled SpanNER model.

We report the exploration results in Figure 7, from which we can see that: (1)
In SpanNER and BL-SpanNER, the hybrid method shows consistent superiority
across the six datasets. And in SeqNER, the hybrid method performs the best on
two of the six datasets. (2) In SpanNER, the Span-pooling method performs
better than the boundary method on five of the six datasets. And in BL-
SpanNER, the boundary and span-pooling methods show comparable performance.

We attribute the superiority of the hybrid method to the fact that it combines
the advantages of boundary and span-pooling methods, making it take the
structure, boundary, length, and context of spans into consideration.
Moreover, we find that the hybrid method does not require extra time overhead
compared to the other two methods. Based on the above facts, we use the hybrid
method to generate span representations in all the other experiments.

Figure 7: Performance against various span representations.

##  5 Conclusion

In this paper, we explore three critical issues of the BL paradigm, a win-win
NER approach achieved by bundling sequence labeling-based and span-based
models. We first design two NER models: SeqNER and SpanNER, and then we bundle
them together to form a BL enhanced model: BL-NER, to answer the first two
issues, i.e., when and why BL works. Experimental results on eleven datasets
validate the effectiveness of BL, and detailed analyses provide answers to
both issues. In order to address the third issue, i.e., whether BL enhances
existing SOTA NER models, we apply BL to five previous NER models. Extensive
results indicate that BL consistently improves their performance. Moreover, we
find that SeqNER leads to fewer entity boundary prediction errors than
SpanNER, but more entity type prediction errors. Furthermore, we make
comparisons between two label tagging methods, as well as three span
representations.

## References

  * [1] B. Ji, J. Yu, S. Li, J. Ma, Q. Wu, Y. Tan, and H. Liu, “Span-based joint entity and relation extraction with attention-based span-specific and contextual semantic representations,” in _Proceedings of the 28th International Conference on Computational Linguistics_. Barcelona, Spain (Online): International Committee on Computational Linguistics, Dec. 2020, pp. 88–99.
  * [2] W. Wu, F. Wang, A. Yuan, F. Wu, and J. Li, “CorefQA: Coreference resolution as query-based span prediction,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: Association for Computational Linguistics, Jul. 2020, pp. 6953–6963. [Online]. Available: https://aclanthology.org/2020.acl-main.622
  * [3] Z. Huang, W. Xu, and K. Yu, “Bidirectional LSTM-CRF models for sequence tagging,” _CoRR_ , vol. abs/1508.01991, 2015.
  * [4] M. Miwa and M. Bansal, “End-to-end relation extraction using LSTMs on sequences and tree structures,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 1105–1116.
  * [5] A. Katiyar and C. Cardie, “Going out on a limb: Joint extraction of entity mentions and relations without dependency trees,” in _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 917–928.
  * [6] X. Li, H. Yan, X. Qiu, and X. Huang, “FLAT: Chinese NER using flat-lattice transformer,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: Association for Computational Linguistics, Jul. 2020, pp. 6836–6842.
  * [7] Y. Nie, Y. Tian, X. Wan, Y. Song, and B. Dai, “Named entity recognition for social media texts with semantic augmentation,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 1383–1391.
  * [8] C. Tan, W. Qiu, M. Chen, R. Wang, and F. Huang, “Boundary enhanced neural span classification for nested named entity recognition,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 05, pp. 9016–9023, Apr. 2020. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/6434
  * [9] F. Li, Z. Lin, M. Zhang, and D. Ji, “A span-based model for joint overlapped and discontinuous named entity recognition,” in _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Online: Association for Computational Linguistics, Aug. 2021, pp. 4814–4828. [Online]. Available: https://aclanthology.org/2021.acl-long.372
  * [10] H. Ouchi, J. Suzuki, S. Kobayashi, S. Yokoi, T. Kuribayashi, R. Konno, and K. Inui, “Instance-based learning of span representations: A case study through named entity recognition,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: Association for Computational Linguistics, Jul. 2020, pp. 6452–6459. [Online]. Available: https://aclanthology.org/2020.acl-main.575
  * [11] J. Fu, X. Huang, and P. Liu, “SpanNER: Named entity re-/recognition as span prediction,” in _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Online: Association for Computational Linguistics, Aug. 2021, pp. 7183–7195. [Online]. Available: https://aclanthology.org/2021.acl-long.558
  * [12] M. G. Sohrab, P. M. Thang, and M. Miwa, “A generic neural exhaustive approach for entity recognition and sensitive span detection.” in _IberLEF@ SEPLN_ , 2019, pp. 735–743.
  * [13] S. Liu, Y. Sun, B. Li, W. Wang, and X. Zhao, “Hamner: Headword amplified multi-span distantly supervised method for domain specific named entity recognition,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 05, pp. 8401–8408, Apr. 2020. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/6358
  * [14] S. Toshniwal, H. Shi, B. Shi, L. Gao, K. Livescu, and K. Gimpel, “A cross-task analysis of text span representations,” in _Proceedings of the 5th Workshop on Representation Learning for NLP_. Online: Association for Computational Linguistics, Jul. 2020, pp. 166–176. [Online]. Available: https://aclanthology.org/2020.repl4nlp-1.20
  * [15] J. Bin, X. Yalong, Y. Jie, L. Shasha, M. Jun, J. Yun, and L. Huijun, “A novel bundling paradigm for named entity recognition,” _Knowledge-based Systems_ , 2022.
  * [16] M. Eberts and A. Ulges, “Span-based joint entity and relation extraction with transformer pre-training,” _arXiv preprint arXiv:1909.07755_ , 2019.
  * [17] B. Y. Lin, F. Xu, Z. Luo, and K. Zhu, “Multi-channel BiLSTM-CRF model for emerging named entity recognition in social media,” in _Proceedings of the 3rd Workshop on Noisy User-generated Text_. Copenhagen, Denmark: Association for Computational Linguistics, Sep. 2017, pp. 160–165.
  * [18] S. Zheng, F. Wang, H. Bao, Y. Hao, P. Zhou, and B. Xu, “Joint extraction of entities and relations based on a novel tagging scheme,” in _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 1227–1236.
  * [19] Y. Xin, E. Hart, V. Mahajan, and J. D. Ruvini, “Learning better internal structure of words for sequence labeling,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2018, pp. 2584–2593.
  * [20] P. H. Martins, Z. Marinho, and A. F. Martins, “Joint learning of named entity recognition and entity linking,” _ACL 2019_ , p. 190, 2019.
  * [21] J. Chen, C. Yuan, X. Wang, and Z. Bai, “MrMep: Joint extraction of multiple relations and multiple entity pairs based on triplet attention,” in _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 593–602.
  * [22] P. Zhou, S. Zheng, J. Xu, Z. Qi, H. Bao, and B. Xu, “Joint extraction of multiple relations and entities by using a hybrid neural network,” in _Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data_ , M. Sun, X. Wang, B. Chang, and D. Xiong, Eds. Cham: Springer International Publishing, 2017, pp. 135–146.
  * [23] E. Strubell, P. Verga, D. Belanger, and A. McCallum, “Fast and accurate entity recognition with iterated dilated convolutions,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2017, pp. 2670–2680.
  * [24] H. Yan, B. Deng, X. Li, and X. Qiu, “TENER: adapting transformer encoder for named entity recognition,” _CoRR_ , vol. abs/1911.04474, 2019.
  * [25] H. Ye, N. Zhang, S. Deng, M. Chen, C. Tan, F. Huang, and H. Chen, “Contrastive triple extraction with generative transformer,” _CoRR_ , vol. abs/2009.06207, 2020.
  * [26] A. Ushio and J. Camacho-Collados, “T-ner: An all-round python library for transformer-based named entity recognition,” in _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_ , 2021, pp. 53–62.
  * [27] Y. Sui, F. Bu, Y. Hu, W. Yan, and L. Zhang, “Trigger-gnn: A trigger-based graph neural network for nested named entity recognition,” 2022.
  * [28] M. Carbonell, P. Riba, M. Villegas, A. Fornés, and J. Lladós, “Named entity recognition and relation extraction with graph neural networks in semi structured documents,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 9622–9627.
  * [29] L. Zhou, J. Li, Z. Gu, J. Qiu, B. B. Gupta, and Z. Tian, “Panner: Pos-aware nested named entity recognition through heterogeneous graph neural network,” _IEEE Transactions on Computational Social Systems_ , 2022.
  * [30] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 2227–2237. [Online]. Available: https://aclanthology.org/N18-1202
  * [31] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186.
  * [32] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” _CoRR_ , vol. abs/1907.11692, 2019.
  * [33] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A lite BERT for self-supervised learning of language representations,” _CoRR_ , vol. abs/1909.11942, 2019.
  * [34] F. Souza, R. F. Nogueira, and R. de Alencar Lotufo, “Portuguese named entity recognition using BERT-CRF,” _CoRR_ , vol. abs/1909.10649, 2019. [Online]. Available: http://arxiv.org/abs/1909.10649
  * [35] A. O. B. Sapci, O. Tastan, and R. Yeniterzi, “Focusing on possible named entities in active named entity label acquisition,” _arXiv preprint arXiv:2111.03837_ , 2021.
  * [36] Z. Jie and W. Lu, “Dependency-guided lstm-crf for named entity recognition,” _arXiv preprint arXiv:1909.10148_ , 2019.
  * [37] X. Dai, S. Karimi, B. Hachey, and C. Paris, “Using similarity measures to select pretraining data for ner,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2019, pp. 1460–1470.
  * [38] L. Xu, Z. Jie, W. Lu, and L. Bing, “Better feature integration for named entity recognition,” in _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2021, pp. 3457–3469.
  * [39] M. G. Sohrab, P. M. Thang, and M. Miwa, “A generic neural exhaustive approach for entity recognition and sensitive span detection.” in _IberLEF@ SEPLN_ , 2019, pp. 735–743.
  * [40] Z. Guo, Y. Zhang, and W. Lu, “Attention guided graph convolutional networks for relation extraction,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , Florence, Italy, Jul. 2019, pp. 241–251. [Online]. Available: https://aclanthology.org/P19-1024
  * [41] C. Tan, W. Qiu, M. Chen, R. Wang, and F. Huang, “Boundary enhanced neural span classification for nested named entity recognition,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 05, 2020, pp. 9016–9023.
  * [42] J. Yu, B. Ji, S. Li, J. Ma, H. Liu, and H. Xu, “S-ner: A concise and efficient span-based model for named entity recognition,” _Sensors_ , vol. 22, no. 8, p. 2852, 2022.
  * [43] J. Yu, B. Bohnet, and M. Poesio, “Named entity recognition as dependency parsing,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , Online, Jul. 2020, pp. 6470–6476. [Online]. Available: https://aclanthology.org/2020.acl-main.577
  * [44] D. Q. Nguyen and K. Verspoor, “End-to-end neural relation extraction using deep biaffine attention,” in _European Conference on Information Retrieval_. Springer, 2019, pp. 729–738.
  * [45] K. Dixit and Y. Al-Onaizan, “Span-level model for relation extraction,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 5308–5314.
  * [46] Y. Luan, L. He, M. Ostendorf, and H. Hajishirzi, “Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 3219–3232.
  * [47] Y. Luan, D. Wadden, L. He, A. Shah, M. Ostendorf, and H. Hajishirzi, “A general framework for information extraction using dynamic span graphs,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 3036–3046.
  * [48] D. Wadden, U. Wennberg, Y. Luan, and H. Hajishirzi, “Entity, relation, and event extraction with contextualized span representations,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 5784–5789.
  * [49] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, and M. Norouzi, “Google’s neural machine translation system: Bridging the gap between human and machine translation,” _CoRR_ , vol. abs/1609.08144, 2016.
  * [50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [51] T. Ma, H. Jiang, Q. Wu, T. Zhao, and C.-Y. Lin, “Decomposed meta-learning for few-shot named entity recognition,” in _Findings of the Association for Computational Linguistics: ACL 2022_ , 2022, pp. 1584–1596.
  * [52] D. Roth and W.-t. Yih, “A linear programming formulation for global inference in natural language tasks,” in _Proc. of CoNLL_ , 2004.
  * [53] W. Ralph, M. Palmer, M. Marcus, E. Hovy, S. Pradhan, L. Ramshaw, and N. Xue, “Ontonotes release 5.0 ldc2013t19,” in _Linguistic Data Consortium_ , 2013\.
  * [54] Z. Wang, J. Shang, L. Liu, L. Lu, J. Liu, and J. Han, “Crossweigh: Training named entity tagger from imperfect annotations,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2019, pp. 5154–5163.
  * [55] D. Roth and W.-t. Yih, “A linear programming formulation for global inference in natural language tasks,” in _Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004_. Boston, Massachusetts, USA: Association for Computational Linguistics, May 6 - May 7 2004, pp. 1–8.
  * [56] G. Doddington, A. Mitchell, M. Przybocki, L. Ramshaw, S. Strassel, and R. Weischedel, “The automatic content extraction (ace) program tasks, data, and evaluation.”
  * [57] C. Walker, S. Strassel, J. Medero, and K. Maeda, “Ace 2005 multilingual training corpus-linguistic data consortium,” _URL: https://catalog. ldc. upenn. edu/LDC2006T06_ , 2005.
  * [58] J. Li, Y. Sun, R. Johnson, D. Sciaky, C. Wei, R. Leaman, A. Davis, C. Mattingly, T. Wiegers, and Z. Lu, “Biocreative v cdr task corpus: a resource for chemical disease relation extraction.” _Database: the Journal of Biological Databases and Curation_ , vol. 2016, pp. baw068–baw068, 2016\.
  * [59] R. Dogan, R. Leaman, and Z. lu, “Ncbi disease corpus: A resource for disease name recognition and concept normalization,” _Journal of biomedical informatics_ , vol. 47, 01 2014.
  * [60] B. Strauss, B. Toma, A. Ritter, M.-C. de Marneffe, and W. Xu, “Results of the WNUT16 named entity recognition shared task,” in _Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT)_. Osaka, Japan: The COLING 2016 Organizing Committee, Dec. 2016, pp. 138–144.
  * [61] A. Ritter, E. Wright, W. Casey, and T. Mitchell, “Weakly supervised extraction of computer security events from twitter,” in _Proceedings of the 24th international conference on world wide web_ , 2015, pp. 896–905.
  * [62] T. Baldwin, M.-C. De Marneffe, B. Han, Y.-B. Kim, A. Ritter, and W. Xu, “Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normalization and named entity recognition,” in _Proceedings of the Workshop on Noisy User-generated Text_ , 2015, pp. 126–135.
  * [63] L. Derczynski, E. Nichols, M. van Erp, and N. Limsopatham, “Results of the WNUT2017 shared task on novel and emerging entity recognition,” in _Proceedings of the 3rd Workshop on Noisy User-generated Text_. Copenhagen, Denmark: Association for Computational Linguistics, Sep. 2017, pp. 140–147.
  * [64] J. Fu, P. Liu, and G. Neubig, “Interpretable multi-dataset evaluation for named entity recognition,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2020, pp. 6058–6069.
  * [65] I. Beltagy, K. Lo, and A. Cohan, “Scibert: A pretrained language model for scientific text,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2019, pp. 3615–3620.
  * [66] J. Lee, W. Yoon, S. Kim, D. Kim, C. So, and J. Kang, “Biobert: a pre-trained biomedical language representation model for biomedical text mining.” _Bioinformatics (Oxford, England)_ , vol. 36, no. 4, pp. 1234–1240, 2020\.
  * [67] X. Li, F. Yin, Z. Sun, X. Li, A. Yuan, D. Chai, M. Zhou, and J. Li, “Entity-relation extraction as multi-turn question answering,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 1340–1350. [Online]. Available: https://aclanthology.org/P19-1129
  * [68] J. Wang and W. Lu, “Two are better than one: Joint entity and relation extraction with table-sequence encoders,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 1706–1721.
  * [69] Y. Shen, X. Ma, Y. Tang, and W. Lu, “A trigger-sense memory flow framework for joint entity and relation extraction,” in _Proceedings of the web conference 2021_ , 2021, pp. 1704–1715.
  * [70] Y. Ma, T. Hiraoka, and N. Okazaki, “Named entity recognition and relation extraction using enhanced table filling by contextualized representations,” _Journal of Natural Language Processing_ , vol. 29, no. 1, pp. 187–223, 2022\.
  * [71] A. O. B. Sapci, O. Tastan, and R. Yeniterzi, “Focusing on possible named entities in active named entity label acquisition,” _arXiv preprint arXiv:2111.03837_ , 2021.
  * [72] H. Yu, X.-L. Mao, Z. Chi, W. Wei, and H. Huang, “A robust and domain-adaptive approach for low-resource named entity recognition,” in _2020 IEEE International Conference on Knowledge Graph (ICKG)_. IEEE, 2020, pp. 297–304.
  * [73] V. Kocaman and D. Talby, “Biomedical named entity recognition at scale,” in _International Conference on Pattern Recognition_. Springer, 2021, pp. 635–646.
  * [74] G. Miolo, G. Mantoan, and C. Orsenigo, “Electramed: a new pre-trained language representation model for biomedical nlp,” _arXiv preprint arXiv:2104.09585_ , 2021.
  * [75] M. Yasunaga, J. Leskovec, and P. Liang, “Linkbert: Pretraining language models with document links,” _arXiv preprint arXiv:2203.15827_ , 2022.
  * [76] X. Wang, Y. Jiang, N. Bach, T. Wang, Z. Huang, F. Huang, and K. Tu, “Improving named entity recognition by external context retrieving and cooperative learning,” in _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 2021, pp. 1800–1812.
  * [77] L. N. Phan, J. T. Anibal, H. Tran, S. Chanana, E. Bahadroglu, A. Peltekian, and G. Altan-Bonnet, “Scifive: a text-to-text transformer model for biomedical literature,” _arXiv preprint arXiv:2106.03598_ , 2021.
  * [78] Z. Zhong and D. Chen, “A frustratingly easy approach for entity and relation extraction,” in _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2021, pp. 50–61.
  * [79] N. Limsopatham and N. Collier, “Bidirectional lstm for named entity recognition in twitter messages,” in _Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT)_ , 2016, pp. 145–152.
  * [80] M. Shahzad, A. Amin, D. Esteves, and A.-C. N. Ngomo, “Inferner: An attentive model leveraging the sentence-level information for named entity recognition in microblogs,” in _The International FLAIRS Conference Proceedings_ , vol. 34, 2021.
  * [81] Y. Luo, F. Xiao, and H. Zhao, “Hierarchical contextualized representation for named entity recognition,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 34, no. 05, 2020, pp. 8441–8448.
  * [82] T. T. H. Hanh, A. Doucet, N. Sidere, J. G. Moreno, and S. Pollak, “Named entity recognition architecture combining contextual and global features,” in _International Conference on Asian Digital Libraries_. Springer, 2021, pp. 264–276.


# Open-Source GEMM Hardware Kernels Generator: Toward Numerically-Tailored
Computations

Louis Ledoux12, Marc Casas12 E-mail: {louis.ledoux<EMAIL_ADDRESS>
1Barcelona Supercomputing Center, Barcelona, Spain  2Universitat Politècnica
de Catalunya, Barcelona, Spain

###### Index Terms:

GEMMs, matrix-matrix-multiply, full stack framework, automated pipeline,
flopoco, OpenCAPI, OpenBLAS, High Performance Computing,
approximate/trans/extended precision.

##  I Extended Abstract

Many scientific computing problems can be reduced to Matrix-Matrix
Multiplications (MMM), making the General Matrix Multiply (GEMM) kernels in
the Basic Linear Algebra Subroutine (BLAS) of interest to the high-performance
computing community. However, these workloads have a wide range of numerical
requirements. Ill-conditioned linear systems require high-precision arithmetic
to ensure correct and reproducible results [1]. In contrast, emerging
workloads such as deep neural networks, which can have millions up to billions
of parameters, have shown resilience to arithmetic tinkering [2] and precision
lowering [3].

Figure 1: Overview of the 2 phases framework. Left is Runtime execution flow
and right is Hardware generation flow.

General purpose arithmetic units and computer formats such as the IEEE754
standard naturally underperform in this vast land of scenarios. We propose the
generation of numerically tailored circuits where the necessary and sufficient
internal precision is generated to target the computations requirements in
terms of numerical quality while improving the energy cost.

###  I-A Open Source SW/HW co-designed framework for numerically tailored MMMs

As depicted by Fig. 1, our framework [4] is composed of two distinct phases,
the a priori Hardware generation flow and the runtime execution flow. Because
MMMs are basically made of arbitrary long dot products, we design a custom
Fused Dot Product (FDP) operator that is agnostic to the computer format and
supports posit, IEEE754, and bfloat16 variations, while never rounding between
two accumulations. The intermediate precision of the fixed-point accumulator
used in the dot-product is a key aspect of this work, and is configurable
through the length of the scratchpad delimited by the parameters MSB (Most
Significant Bit) and LSB (Least Significant Bit). We leverage the automated
pipeline feature of _flopoco_ [5], which is an effective tool for efficiently
exploring the wide range of functional specifications along with performance
specifications, to produce MMM kernels with the necessary basic elements
(LUTs, FFs, Carry chains, DSPs) for a targeted $(chip,frequency)$ couple (see
Fig. 1\- \raisebox{-.9pt} {B}⃝ ).

The essence of this work is to make intermediate precision tweakings from the
hardware accessible to high-end software code as transparent as possible. We
achieve that by taking into account that many HPC codes rely on BLAS libraries
to perform MMM operations. Such libraries receive the function call to perform
a GEMM and dispatch adequately to the underlying hardware at their disposal.

Figure 2: Sea Surface Height computation comparing IEEE-754 double-, quad-
precision FMAs and a 91-bit FDP wrt numerical quality and power consumption.
Figure 3: Top1 Accuracy vs validation dataset inference energy cost for
various combinations of datasets, models, computer formats, and accumulators.

###  I-B HPC workloads results

We experiment with two families of real HPC workloads with contrasting
numerical requirements, namely Artificial Intelligence (AI) and Sea Surface
Height (SSH), whose respective results can be observed in Fig. 3 and Fig. 2.

For the SSH computation, the results obtained with 64-bit and 128-bit FPUs
exhibit decreasing reproducibility as the vector size increases. In contrast,
our 91-bit $\left\langle ovf:30,msb:30,lsb:30\right\rangle$ FDP maintains
reproducibility for all vector sizes without deviation. Our proposed FDP
consistently exhibits 52 correct bits, which is at least 5$\times$ and
27.7$\times$ more than quad-precision and double-precision. Our measurements
on VU3P-2 FPGA at $200MHz$ show that the units power consumption are 0.266,
0.549, and 0.491 watts for double-precision FMA, quad-precision FMA, and the
91-bit FDP, respectively. For all evaluated sizes, the 91-bit FDP yields at
least $5.6\times$ and $15.1\times$ more correct bits for the same wattage as
quad-precision and double-precision FMAs, respectively.

For AI workloads, we employ Pytorch as a base framework and link it to our
modified OpenBLAS. We use popular neural network models such as ResNet18,
ResNet34, ResNet50, DenseNet121, DenseNet161, DenseNet169, and VGG11 with
batch normalization, and evaluate them on the CIFAR-10 and ImageNet datasets.
To measure power consumption and accuracy, we use the BrainFloat16 and
IEEE-754 32-bit formats for our computations with a large variety of
accumulators varying their $OVF$, $MSB$, and $LSB$ parameters. Fig. 3 shows
the relationship between power consumption and accuracy for different
accumulator and arithmetic combinations. For example, if $84\%$ Top1 accuracy
is satisfying for Imagenet with Resnet50, the most suited
arithmetic/accumulator combination is IEEE-754 32-bit/$\left\langle
ovf:9,msb:6,lsb:-20\right\rangle$ represented by a light purple hexagon as all
other markers are either on the right or below.

###  I-C Conclusion

Overall, our work highlights the importance of numerically tailored
accumulators for reproducibility in scientific computing applications. Our
results provide valuable insights into the trade-offs between power
consumption and accuracy, and we believe that our results have the potential
to inform the design of future AI and scientific computing systems. In light
of this, we encourage other researchers to explore the possibilities of low
precision accumulators using our open-source framework.

##  II Acknowledgment

Marc Casas is supported by Grant RYC-2017-23269 funded by MCIN/AEI/
10.13039/501100011033 and by “ESF Investing in your future”.

## References

  * [1] D. Bailey and J. M. Borwein, “High-Precision Computation and Mathematical Physics,” in _Proceedings of XII Advanced Computing and Analysis Techniques in Physics Research — PoS(ACAT08)_. Erice, Italy: Sissa Medialab, Oct. 2009, p. 014\. [Online]. Available: https://pos.sissa.it/070/014
  * [2] J. Johnson, “Rethinking floating point for deep learning,” _arXiv:1811.01721 [cs]_ , Nov. 2018, arXiv: 1811.01721. [Online]. Available: http://arxiv.org/abs/1811.01721
  * [3] M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1,” _arXiv:1602.02830 [cs]_ , Feb. 2016, arXiv: 1602.02830. [Online]. Available: http://arxiv.org/abs/1602.02830
  * [4] L. Ledoux and M. Casas, “A Generator of Numerically-Tailored and High-Throughput Accelerators for Batched GEMMs,” in _2022 IEEE 30th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM)_ , May 2022, pp. 1–10, iSSN: 2576-2621.
  * [5] M. Istoan and F. de Dinechin, “Automating the pipeline of arithmetic datapaths,” in _Design, Automation & Test in Europe Conference & Exhibition (DATE 2017)_, Lausanne, Switzerland, Mar. 2017. [Online]. Available: https://hal.inria.fr/hal-01373937

|  Louis Ledoux received his BSc degree in 2016 in Computer Science from
Université de Rennes1, France. The following years, he pursued his MSc degree
in parallel with an Engineer diploma from École Supérieure d’Ingénieurs de
Rennes (ESIR). He concluded in 2018 his studies in Rennes with a position of
Hardware Engineer at b<>com, a national research laboratory. This position
allowed him to experiment with the first FPGAs in the cloud and their
virtualizations. Since 2018, he has been a PhD candidate at the Computer
Architecture departments of Barcelona Supercomputing Center (BSC) and
Universitat Politècnica de Catalunya (UPC), Spain.
---|---