# Comparing and combining some popular NER approaches on Biomedical tasks

Harsh Verma, Sabine Bergler, Narjesossadat Tahaei
CLaC Labs, Concordia University
{h_ver, bergler, n_tahaei} @cse.concordia.ca

###### Abstract

We compare three simple and popular approaches for NER: 1) SEQ (sequence-
labeling with a linear token classifier) 2) SeqCRF (sequence-labeling with
Conditional Random Fields), and 3) SpanPred (span-prediction with boundary
token embeddings). We compare the approaches on 4 biomedical NER tasks: GENIA,
NCBI-Disease, LivingNER (Spanish), SocialDisNER (Spanish). The SpanPred model
demonstrates state-of-the-art performance on LivingNER and SocialDisNER,
improving F1 by 1.3 and 0.6 F1 respectively. The SeqCRF model also
demonstrates state-of-the-art performance on LivingNER and SocialDisNER,
improving F1 by 0.2 F1 and 0.7 respectively. The SEQ model is competitive with
the state-of-the-art on the LivingNER dataset. We explore some simple ways of
combining the three approaches. We find that majority voting consistently
gives high precision and high F1 across all 4 datasets. Lastly, we implement a
system that learns to combine the predictions of SEQ and SpanPred, generating
systems that consistently give high recall and high F1 across all 4 datasets.
On the GENIA dataset, we find that our learned combiner system significantly
boosts F1(+1.2) and recall(+2.1) over the systems being combined. We release
all the well-documented code necessary to reproduce all systems at this Github
repository.

##  Introduction

NER has frequently been formulated as a sequence-labeling problem Chiu and
Nichols (2016); Ma and Hovy (2016); Wang et al. (2022) in which a model learns
to label each token using a labeling scheme such as BIO(beginning, inside,
outside). However, in recent years people have also formulated the NER task as
a span-prediction problem Jiang et al. (2020); Li et al. (2020); Fu et al.
(2021); Zhang et al. (2023) where spans of text are represented and labeled
with entity types.

Let SEQ be the simplest sequence-labeling model which represents each token
using a language model and then classifies each token-representation with a
linear layer. Let SeqCRF be another popular sequence-labeling model which is
identical to SEQ model except that the token representations from the language
model are fed into a linear-chain conditional random field layerLafferty et
al. (2001); Lample et al. (2016). Let SpanPredLee et al. (2017); Jiang et al.
(2020) be a model that represents every possible span of text using two token-
embeddings located at the its boundary, and then classifies every span-
representation using a linear layer. We describe all three models in detail in
section 4. We evaluate SEQ, SeqCRF, and SpanPred models on four biomedical NER
tasks: GENIAKim et al. (2003), NCBI-DiseaseDoğan et al. (2014),
LivingNER(Spanish)Miranda-Escalada et al. (2022), and
SocialDisNER(Spanish)Gasco Sánchez et al. (2022). Despite being simple, the
SpanPred and CRF models improve the state-of-the-art on the LivingNER and
SocialDisNER tasks.

Fu et al. (2021) show that the sequence-labeling approaches(eg. Seq and
SeqCRF) and span-prediction approaches(eg. SpanPred) have different strengths
and weaknesses while having similar(F1) performance. This motivated us to try
and combine Seq, SeqCRF, and SpanPred models using two simple methods and
study the results. We refer to the two simple methods as Union and MajVote.
Union is inspired by the set(mathematical) union operation and it simply
involves "unioning" the sets of predictions made by the models. MajVote is the
classic majority voting method. We find that MajVote can yield systems that
have both high precision and high F1.

Inspired by the boost in recall(and the corresponding drop in precision)
resulting from the Union method, we implemented a combiner system (which we
refer to as Meta) that aims to combat the drop in precision as a result of the
Union method. We find that Meta shows very promising signs of increasing
precision while preserving high recall and high F1. Meta borrows ideas from
work on generating span representations using "solid markers"Baldini Soares et
al. (2019); Xiao et al. (2020); Ye et al. (2022), work on using prompts Li et
al. (2020), and work by Fu et al. (2021) to combine the span-prediction and
sequence-labeling approaches using the span-prediction approach.

##  Preliminaries

Let every prediction $p$ for NER be a tuple of the form

$p=(\text{SampleId},\text{EntityType},\text{BeginOffset},\text{EndOffset})$

which consists of the identifier of the sample/text in which the entity is
found, the type of the entity, and the beginning and ending offsets for the
entity.

##  Preprocessing

For GENIA and NCBI-Disease, each sample is an English sentence. For
SocialDisNER, each sample is an entire Spanish tweet. For LivingNER, we use
the FLERTSchweter and Akbik (2020) approach for document-level NER, in which
each Spanish sentence is surrounded by a context of 100 characters to the left
and 100 characters to the right.

Dataset | SocialDisNER | LivingNER | Genia | NCBI-Disease
---|---|---|---|---
SOTA | Fu et al. (2022) | Zotova et al. (2022) | Shen et al. (2022) | Tian et al. (2020)
| 89.1, 90.6, 87.6 | 95.1, 95.8, 94.3 | 81.7, -, - | 90.08, -, -
SpanPred | 90.4, 90.5, 90.4 | 95.7, 95.4, 96.0 | 77.1, 77.0, 77.1 | 89.0, 88.1, 89.9
SEQ | 88.7, 88.3, 89.1 | 95.0, 94.7, 95.3 | 76.1, 79.8, 72.7 | 88.7, 87.8, 89.5
SeqCRF | 89.8, 89.6, 90.0 | 95.3, 95.6, 95.0 | 75.7, 79.7, 72.1 | 87.9, 86.2, 89.6
SpanPred $\cup$ SEQ | 89.0, 86.0, 92.2 | 95.2, 93.4, 97.1 | 77.2, 73.5, 81.4 | 88.2, 84.6, 92.2
SpanPred x SEQ | 90.2, 93.3, 87.3 | 95.5, 96.9, 94.2 | 75.8, 85.0, 68.5 |  89.6, 91.9, 87.4
SpanPred $\cup$ SEQ $\cup$ SeqCRF | 88.3, 84.1, 93.0 | 94.9, 92.5, 97.4 | 76.4, 71.3, 82.3 | 87.1, 81.4, 93.8
SpanPred x SEQ x SeqCRF |  90.8, 91.2, 90.4 | 95.7, 96.1, 95.4 | 77.1, 81.9, 72.9 | 89.5, 88.8, 90.1
Meta(SpanPred $\cup$ SEQ) |  90.5, 89.7, 91.3 |  95.7, 94.6, 96.9 |  78.3, 77.4, 79.2 |  89.1, 86.3, 92.2
Table 1: Performance of all systems on test set on all 4 biomedical datasets.
$\cup$ represents the Union combiner and x represents the MajVote combiner.

##  Models

###  Seq model

#### Token Representation Step

Given a sentence $\mathbf{x}=[w_{1},w_{2},...,w_{n}]$ with $n$ tokens, we
generate for each token $w_{i}$ a contextualized embedding
$\mathbf{u}_{i}\in\mathbb{R}^{d}$ that corresponds to the last-hidden-layer
representation of the language model. Here, $d$ represents the size of the
token embedding. Importantly, special tokens like [CLS] and [SEP] are also
represented. We find that the performance can drop significantly(especially
for SEQ) if they are not incorporated in the learning process.

XLM-RoBERTa largeConneau et al. (2020) is the multilingual language model that
we use for the LivingNER and SocialDisNER spanish tasks. Inspired by its high
performance on the BLURBGu et al. (2021) biomedical benchmark, we use
BioLinkBert largeYasunaga et al. (2022) for the NCBI-Disease and GENIA
datasets.

#### Token Classification Step

In this layer, we classify every token representation into a set of named
entity types corresponding to the BIO(beginning, inside, outside) tagging
scheme. Assuming $\mathbf{\Theta}$ is the set of all named entity types, then
the set of all BIO tags $\mathbf{B}$ is of size
$(2\times|\mathbf{\Theta}|)+1$. In other words, a linear layer maps each token
representation $\mathbf{u}_{i}\in\mathbb{R}^{d}$ to a prediction
$\mathbf{p}_{i}\in\mathbb{R}^{|\bf{B}|}$, where $d$ is the length of the token
embedding. Finally, the predictions are used to calculate loss of given
sentence $\mathbf{x}$ with $n$ tokens as follows:

$\text{Loss}(\mathbf{x})=\frac{-1}{n}\sum_{i=1}^{n}\text{log}(\text{Softmax}(\mathbf{p}_{i})_{y_{i}})$
(1)

Here $y_{i}$ represents the index of the gold BIO label of the $i^{th}$ token.

###  SeqCRF Model

This model is identical to the Seq model except that we pass the
contextualized token representation $\mathbf{U}$ through a a Linear Chain
CRFLafferty et al. (2001) layer. The CRF layer computes the probabilities of
labeling the sequence using the Viterbi algorithmForney (1973). A loss suited
to the CRF layer’s predictions is then used to train the model. We directly
use the CRF implementation available in the FLAIRAkbik et al. (2019)
framework. The BIO scheme is used for token classification.

###  Span Model

#### Token Representation Layer

Same as the token representation layer of the Seq model.

#### Span Representation Layer

Let a span $\mathbf{s}$ be a tuple $\mathbf{s}=(b,e)$ where $b$ and $e$ are
the beggining and ending token indices, and $\mathbf{s}$ represents the text
segment $[w_{b},w_{b+1},...,w_{e}]$ where $w_{i}$ is the $i^{th}$ token. In
this layer, we enumerate all possible spans and then represent each span using
two token embeddings located at its boundary. More precisely, given embeddings
$[\mathbf{u}_{1},\mathbf{u}_{2},...,\mathbf{u}_{n}]$ of $n$ tokens, there are
$\binom{n}{2}=\frac{n^{2}}{2}$ possible spans, which can be enumerated and
represented as the list
$[(0,0),(0,1),...,(0,n),(1,1),(1,2)...(1,n),...(n,n)]$. Then we removed all
spans that have a length longer than 32 tokens – this was important to fit the
model in GPU memory with a batch size of 4. Finally, as in Lee et al. (2017),
each span $s_{i}$ will be represented by
$\mathbf{v}_{i}=[\mathbf{u}_{b_{i}};\mathbf{u}_{e_{i}}]$, a concatenation of
the beginning and ending token embeddings. Hence, the output of this layer is
$\textbf{V}\in\mathbb{R}^{k\times(2\times d)}$ where $k=\frac{n^{2}}{2}$ and
$d$ is length of the token embedding vector.

#### Span Classification Layer

In this layer, we classify each span representation with a named entity type.
We introduce an additional label `Neg_Span` which represents the absence of a
named entity. Precisely, a linear layer maps each span representation
$\mathbf{v}_{i}\in\mathbb{R}^{(2\times d)}$ to a prediction
$\mathbf{p}_{i}\in\mathbb{R}^{|\Omega|}$, where $\Omega$ is the set of all
named entity types(including `Neg_Span`) and $d$ is the size of the token
embedding. Finally, the predictions are used to calculate loss of given
sentence $\mathbf{x}$ with $l$ possible spans as follows:

$\text{Loss}(\mathbf{x})=\frac{-1}{l}\sum_{i=1}^{l}\text{log}(\text{Softmax}(\mathbf{p}_{i})_{y_{i}})$
(2)

Here $y_{i}$ represents the index of the gold label of the $i^{th}$ span.

###  Union combiner model

This model doesn’t learn weights. For a given list $P_{0},P_{1},...,P_{n}$
where $P_{i}$ is the set of predictions(as defined in section 2) made by the
$i^{th}$ NER model and $n$ is the total number of models, it returns the set
$P_{1}\cup P_{2}\cup...P_{n}$.

###  MajVote combiner model

This model doesn’t learn weights. This is the classic majority voting combiner
model. Precisely, when given a list $P_{0},P_{1},...,P_{n}$ where $P_{i}$ is
the set of predictions(as defined in section 2) made by the $i^{th}$ NER model
and $n$ is the total number of models, it returns a set which only includes
predictions in $P_{1}\cup P_{2}\cup...P_{n}$ that have been predicted by more
that $\lfloor\frac{n}{2}\rfloor$ models.

###  Meta combiner model

The job of meta is simple : "Learn to tell if a prediction made by SEQ or
SpanPred is a mistake or not". In other words, Meta looks at a prediction made
by SEQ or SpanPred on the validation set and learns to classify the prediction
as being either "correct" or "incorrect". "correct" means that the prediction
is a good prediction, and that it should not be removed. "incorrect" means
that the prediction should be removed. In other words, if $P_{\text{SEQ}}$ is
the set of all predictions of the SEQ and $P_{\text{Span}}$ is the set of all
predictions of SpanPred, then Meta acts as (and learns to be) a filter for
$P_{\text{Span}}\cup P_{\text{SEQ}}$. During evaluation, Meta filters
$P_{\text{Span}}\cup P_{\text{SEQ}}$, generating a final set of predictions.

We borrow the idea of using markers made with special tokens Baldini Soares et
al. (2019); Xiao et al. (2020); Ye et al. (2022) which, intuitively, help
models "focus their attention on the span-of-interest". In other words, by
introducing special tokens(which act as markers) like [e] and [/e] in the
language model’s vocabulary, and then surrounding the span-of-interest with
them, one can help the model "focus" of the span of interest while making some
prediction. In Meta’s case, the markers are supposed to help locate/identify
the entities predicted by SEQ or SpanPred in raw text. See subsection 4.7 for
an example input prediction with markers highlighting the entity.

We also borrow the idea of promptingLi et al. (2020), which involves pre-
pending some text(prompt) to the original input text with the goal of
priming(or aiding) a model’s decision making with a useful bias. In
particular, every input to Meta includes the type of the predicted entity as
prompt. Intuitively, this helps Meta recognize the type of the entity it is
dealing with. See subsection 4.7 for an example of prompting with the entity
type "disease".

Note that prompting and special markers are only used to prepare the training
data for Meta using the predictions of SEQ and SpanPred on the validation set.
Meta itself is a simple binary classification neural model. Just like SEQ,
SeqCRF and SpanPred, it first creates contextualized token representations
from raw input using the appropriate language model(XLM-RoBERTa or
BioLinkBERT) and then classifies the pooler token([CLS] or [s]) representation
using a linear layer. As in SpanPred and SEQ, cross-entropy loss is used to
train the model.

Because META acts as a "filter"(it allows certain predictions and disallows
others), it cannot improve recall – it can only improve precision. Ideally,
Meta will learn the true nature of the mistakes that SEQ and SpanPred make and
remove all false positives, resulting in a perfect precision score of 100 and
no drop in recall.

#### Preparing the training data for Meta:

all predictions(with "correct" and "incorrect" labels) on the validation set
for all 20 epochs by both SEQ and SpanPRED, and all gold predictions(that only
have "correct" labels) from the original training data make up the training
set for Meta. We hold out 15 percent of Meta’s training set for validation.
Note that we incorporate the predictions of SpanPred and SEQ from earlier
epochs because the fully trained high-performing models don’t make that many
mistakes(which META needs for its learning). As expected, the test set is not
touched while training Meta. During evaluation, Meta filters the predictions
made by SEQ and SpanPred on the test set.

###  Meta input example

Assume the example sentence "Bob has HIV and flu." and the task of identifying
diseases. Now assume that SEQ predicted
(id, disease, 8, 11) (see section 2 for the definition of prediction) and
correctly identified the disease "HIV" in the input. Then, the input to meta
will be the the text "disease Bob has [e] HIV [/e] and flu" and the associated
gold label of correct. Prompting with disease informs Meta that it is dealing
with a prediction representing a disease. Meta has to make a judgement on
whether the prediction is correct or not.

###  Training and Optimization

Both XLM RoBERTa largeConneau et al. (2020) and BioLinkBERT largeYasunaga et
al. (2022) are fine-tuned on the training data using the AdafactorShazeer and
Stern (2018) optimizer with a learning rate of `1e-5`(see code) and a batch
size of 4 for all 4 datasets. Specifically, we used the implementation of
Adafactor available on HuggingFaceWolf et al. (2019). It was not possible for
us to use the same learning rate and batch size for every dataset with
AdamKingma and Ba (2015) because we noticed it was prone to over-fitting(and
then collapsing) mid-training on LivingNER, NCBI-Disease, and GENIA – batch-
size had to be increased to avoid over-fitting. Moreover, we found that SEQ,
SeqCRF, and SpanPred converged to better solutions with Adafactor on all
datasets. However, we found that Meta consistently converged to better
solutions on the NCBI disease dataset using Adam.

The best model is selected using early stopping with a patience(in terms of
epochs) of 5.

##  Evaluation Methodology

All tasks evaluate systems using the strict(no partial matching) Micro F1,
Precision and Recall. For SocialDisNER, all systems were submitted to the
corresponding CodaLabPavao et al. (2022) competition website for evaluation.
For LivingNER, all our systems have been evaluated using the official
evaluation script that the organizers made available. For Genia and NCBI-
Disease, we unfortunately couldn’t find official CodaLab websites, so we had
to use our own script, which can be inspected here.

##  Analysis of Results

Note that among the 3 models, SpanPred consistently outperforms the other two
on all datasets. This is anticipated on tasks with overlapping entities like
LivingNER and GENIA(because SEQ and SeqCRF cannot represent them), but not on
"flat" NER tasks like SocialDisNER and NCBI-Disease.

Note that any system resulting from a Union combination should have higher
recall than any of the involved systems because a set union operation is
incapable of removing a correct prediction (the set of false negatives can
only shrink with more systems). Also, the resulting system’s precision cannot
be higher than the highest precision observed in any sub-system. Table 1
adheres to both of these expectations. On the other hand, a system resulting
from a MajVote combiner is inclined to have higher precision when the systems
being combined are diverse and comparable because – intuitively – MajVote can
be a more "picky" system (only allowing a prediction if it has been voted on
by several). In Table 1, note that both SpanPredxSEQ and SpanPredxSEQxCRF
consistently boost precision across all datasets. Also note that the best
MajVote systems significantly outperform all other systems on precision while
maintaining the highest F1 on all datasets except Genia, where Meta
outperforms all other systems on F1 for the first(and last) time. Also on
Genia is the only time when a Union model
($\texttt{SpanPred}\cup\texttt{SEQ}$) outperforms the MajVote models due to a
significant boost in recall. Finally, note how Meta, across all datasets,
outperforms SpanPred, SEQ, and SeqCRF models on Recall and delivers an F1 that
is at least as high as any of the three models.

##  Conclusion

Our implementation(code available) of CRF and SpanPred, two simple models,
improves the state of the art on LivingNER and SocialDisNER datasets. We used
two simple approaches called Union and MajVote to combine the NER models’
predictions and studied the results. MajVote on the three NER models seems to
be effective at generating systems with high precision and high F1. While
Union can generate systems with higher recall, it is only at the cost of F1
due to a significant drop in precision. Meta seems to be effective at
alleviating Union’s issue, generating systems with both high recall and high
F1.

## References

  * Akbik et al. (2019) Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019.  FLAIR: An easy-to-use framework for state-of-the-art NLP.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 54–59, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Baldini Soares et al. (2019) Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019\.  Matching the blanks: Distributional similarity for relation learning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2895–2905, Florence, Italy. Association for Computational Linguistics.
  * Chiu and Nichols (2016) Jason P.C. Chiu and Eric Nichols. 2016.  Named entity recognition with bidirectional LSTM-CNNs.  _Transactions of the Association for Computational Linguistics_ , 4:357–370.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Doğan et al. (2014) Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014.  Ncbi disease corpus: a resource for disease name recognition and concept normalization.  _Journal of biomedical informatics_ , 47:1–10.
  * Forney (1973) G David Forney. 1973.  The viterbi algorithm.  _Proceedings of the IEEE_ , 61(3):268–278.
  * Fu et al. (2022) Jia Fu, Sirui Li, Hui Ming Yuan, Zhucong Li, Zhen Gan, Yubo Chen, Kang Liu, Jun Zhao, and Shengping Liu. 2022.  CASIA@SMM4H’22: A uniform health information mining system for multilingual social media texts.  In _Proceedings of The Seventh Workshop on Social Media Mining for Health Applications, Workshop & Shared Task_, pages 143–147, Gyeongju, Republic of Korea. Association for Computational Linguistics.
  * Fu et al. (2021) Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021.  SpanNER: Named entity re-/recognition as span prediction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7183–7195, Online. Association for Computational Linguistics.
  * Gasco Sánchez et al. (2022) Luis Gasco Sánchez, Darryl Estrada Zavala, Eulàlia Farré-Maduell, Salvador Lima-López, Antonio Miranda-Escalada, and Martin Krallinger. 2022\.  The SocialDisNER shared task on detection of disease mentions in health-relevant content from social media: methods, evaluation, guidelines and corpora.  In _Proceedings of The Seventh Workshop on Social Media Mining for Health Applications, Workshop & Shared Task_, pages 182–189, Gyeongju, Republic of Korea. Association for Computational Linguistics.
  * Gu et al. (2021) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare (HEALTH)_ , 3(1):1–23.
  * Jiang et al. (2020) Zhengbao Jiang, Wei Xu, Jun Araki, and Graham Neubig. 2020.  Generalizing natural language analysis through span-relation representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2120–2133, Online. Association for Computational Linguistics.
  * Kim et al. (2003) J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. 2003.  Genia corpus—a semantically annotated corpus for bio-textmining.  _Bioinformatics_ , 19(suppl_1):i180–i182.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _Proceedings of the 3rd International Conference on Learning Representations, ICLR’15_.
  * Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.  Neural architectures for named entity recognition.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 260–270, San Diego, California. Association for Computational Linguistics.
  * Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017.  End-to-end neural coreference resolution.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics.
  * Li et al. (2020) Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020\.  A unified MRC framework for named entity recognition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5849–5859, Online. Association for Computational Linguistics.
  * Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016.  End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1064–1074, Berlin, Germany. Association for Computational Linguistics.
  * Miranda-Escalada et al. (2022) Antonio Miranda-Escalada, Eulàlia Farré-Maduell, Salvador Lima-López, Darryl Estrada, Luis Gascó, and Martin Krallinger. 2022.  Mention detection, normalization & classification of species, pathogens, humans and food in clinical documents: Overview of the livingner shared task and resources.  _Procesamiento del Lenguaje Natural_ , 69:241–253.
  * Pavao et al. (2022) Adrien Pavao, Isabelle Guyon, Anne-Catherine Letournel, Xavier Baró, Hugo Escalante, Sergio Escalera, Tyler Thomas, and Zhen Xu. 2022.  _CodaLab Competitions: An open source platform to organize scientific challenges_.  Ph.D. thesis, Université Paris-Saclay, FRA.
  * Schweter and Akbik (2020) Stefan Schweter and Alan Akbik. 2020.  Flert: Document-level features for named entity recognition.  _arXiv preprint arXiv:2011.06993_.
  * Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _International Conference on Machine Learning_ , pages 4596–4604. PMLR.
  * Shen et al. (2022) Yongliang Shen, Xiaobin Wang, Zeqi Tan, Guangwei Xu, Pengjun Xie, Fei Huang, Weiming Lu, and Yueting Zhuang. 2022.  Parallel instance query network for named entity recognition.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 947–961, Dublin, Ireland. Association for Computational Linguistics.
  * Tian et al. (2020) Yuanhe Tian, Wang Shen, Yan Song, Fei Xia, Min He, and Kenli Li. 2020.  Improving biomedical named entity recognition with syntactic information.  _BMC bioinformatics_ , 21(1):1–17.
  * Wang et al. (2022) Xinyu Wang, Yongliang Shen, Jiong Cai, Tao Wang, Xiaobin Wang, Pengjun Xie, Fei Huang, Weiming Lu, Yueting Zhuang, Kewei Tu, Wei Lu, and Yong Jiang. 2022.  DAMO-NLP at SemEval-2022 task 11: A knowledge-based system for multilingual named entity recognition.  In _Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)_ , pages 1457–1468, Seattle, United States. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Xiao et al. (2020) Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, and Leyu Lin. 2020.  Denoising relation extraction from document-level distant supervision.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3683–3688, Online. Association for Computational Linguistics.
  * Yasunaga et al. (2022) Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022.  LinkBERT: Pretraining language models with document links.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8003–8016, Dublin, Ireland. Association for Computational Linguistics.
  * Ye et al. (2022) Deming Ye, Yankai Lin, Peng Li, and Maosong Sun. 2022.  Packed levitated marker for entity and relation extraction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4904–4917, Dublin, Ireland. Association for Computational Linguistics.
  * Zhang et al. (2023) Sheng Zhang, Hao Cheng, Jianfeng Gao, and Hoifung Poon. 2023.  Optimizing bi-encoder for named entity recognition via contrastive learning.  In _The Eleventh International Conference on Learning Representations_.
  * Zotova et al. (2022) Elena Zotova, Aitor García-Pablos, Naiara Perez, Pablo Turón, and Montse Cuadros. 2022.  Vicomtech at livingner 2022.  In _CEUR workshop proceedings_.