# A Computational Model for the Assessment of Mutual Intelligibility Among
Closely Related Languages

Jessica Nieder
MCL Chair
University of Passau
Passau, Germany
<EMAIL_ADDRESS>Johann-Mattis List
MCL Chair / DLCE
University of Passau / MPI-EVA
Passau / Leipzig, Germany
<EMAIL_ADDRESS>
###### Abstract

Closely related languages show linguistic similarities that allow speakers of
one language to understand speakers of another language without having
actively learned it. Mutual intelligibility varies in degree and is typically
tested in psycholinguistic experiments. To study mutual intelligibility
computationally, we propose a computer-assisted method using the Linear
Discriminative Learner, a computational model developed to approximate the
cognitive processes by which humans learn languages, which we expand with
multilingual semantic vectors and multilingual sound classes. We test the
model on cognate data from German, Dutch, and English, three closely related
Germanic languages. We find that our model’s comprehension accuracy depends on
1) the automatic trimming of inflections and 2) the language pair for which
comprehension is tested. Our multilingual modelling approach does not only
offer new methodological findings for automatic testing of mutual
intelligibility across languages but also extends the use of Linear
Discriminative Learning to multilingual settings.

A Computational Model for the Assessment of Mutual Intelligibility Among
Closely Related Languages

Jessica Nieder MCL Chair University of Passau Passau, Germany
<EMAIL_ADDRESS>Johann-Mattis List MCL Chair / DLCE University of
Passau / MPI-EVA Passau / Leipzig, Germany<EMAIL_ADDRESS>

##  1 Introduction

Speakers of a given language can often partially comprehend other languages in
the same language family. This mutual intelligibility has been demonstrated to
be dependent on several linguistic variables, such as phonological,
orthographic or lexical similarity, and extralinguistic factors, such as the
amount of previous exposure to or the attitude towards the other language
(Gooskens and Swarte, 2017; Gooskens et al., 2015; Heeringa et al., 2014). In
addition, the phonetic similarity between words expressing similar meanings
has been shown to be a major factor driving cross-linguistic mutual
intelligibility (Gooskens et al., 2018). Phonetically and semantically similar
words are often called _cognates_ in studies on mutual intelligibility,
foreign language learning, and bilingualism (Squires et al., 2020).
Originally, however, the term denotes words inherited from the same ancestral
language in genetically related languages (List, 2016). Although cognates in
the original sense often exhibit phonetic and semantic similarity across
related languages, they do not necessarily do so, and words can also be
similar in pronunciation and meaning due to other factors, including – most
importantly – intensive borrowing, and – to a much lower degree – different
kinds of sound symbolism (see Casad 1987, 87 for more details on the
difference between mutual intelligibility and genetic relationship).

Due to a focus on the abilities of language users, research on mutual
intelligibility often involves experimental studies with different groups and
numbers of participants. Experiments are diverse, usually consisting of
certain comprehension tasks. Experimental studies show some general
limitations, in so far as uniform methods are rarely used (1), finding
participants with a minimum or no exposure to the test language is difficult
(2), and comparing several languages simultaneously is a time- and resource-
consuming effort (3) (Gooskens and Swarte, 2017; Tang and van Heuven, 2009).
Gooskens and Swarte (2017) present a large-scale study on mutual
intelligibility of five Germanic languages using a _Cloze Test_ , i.e. a
written or audibly presented text in the target language with gaps that need
to be filled in. However, they report a substantial loss in the number of
participants when testing _inherent intelligibility_ , the ability to
comprehend the target language with no or little previous exposure (Gooskens
and Swarte, 2017). In an ideal setting with zero exposure to the target
language, inherent intelligibility captures how comprehensible the target
language is based on structural similarities only. This, in turn, would offer
insights into what linguistic structures give rise to mutual intelligibility
without extralinguistic or other language exposure-based interference. In
reality, the goal of finding participants with no or a minimum of exposure to
certain languages is an almost impossible requirement to fulfill due to the
status of some languages of being a common _lingua franca_ (Gooskens and
Swarte, 2017; Hongyan, 2017).

Figure 1: Distribution of cosine similarity scores between language pairs for
all cognate triplets. Note that smoothing of the distribution results in
values exceeding 1.0.

In this study we propose a computer-assisted method couched in the
discriminative lexicon framework by Baayen et al. (2019) to assess mutual
intelligibility in Germanic languages. By focusing on computational methods
instead of human subjects we can overcome the mentioned limitations. Our
proposed model does not involve the recruitment of participants, there are no
extralinguistic factors nor target language exposure involved in training. We
offer a uniform method that can be adapted to various language families and
lead to new insights into intelligibility based on a careful selection of
linguistic factors that are involved in language comprehension.

##  2 Linear Discriminative Learning

With the discriminative lexicon framework (DL), Baayen et al. (2019) propose a
model of language processing that explores the cognitive mapping mechanisms
involved in language learning. Language comprehension is understood as a
mapping of phonological forms onto meaning (Baayen et al., 2019).
Mathematically, it is implemented as multivariate multiple regression in the
Linear Discriminative Learner (LDL) model. Given a phonological matrix _$C$_
and a semantic matrix _$S$_ , the comprehension matrix _$F$_ is obtained by
post-multiplying _$C$_ with _$F$_ : _$CF=S$_. The _$F$_ matrix then specifies
the associaton weights between all phonological cues and all semantic
dimensions (Chuang et al., 2023). Multiplying _$C$_ with _$F$_ finally
predicts the semantic vector _$\hat{S}$_ for all input word forms that can be
used for evaluating comprehension accuracy of the model. Computationally, the
LDL model conceptualizes language comprehension as a simple artificial neural
network directly connecting phonological and semantic vectors without any
hidden layers (Nieder et al., 2023; Chuang et al., 2023). In this study, we
make use of LDL to explore the mutual comprehension of the Germanic languages
Dutch, German and English based on a cross-language learning setting (see also
Chuang et al., 2018, for another multilingual approach using LDL). As
phonological input we use cognate sets from all languages. For the semantic
matrix, we opted for the multilingual ConceptNet Numberbatch word embeddings
_version 19.08_ from Speer et al. (2017) that offer the possibility to
directly compare the meaning of cognate concepts.

##  3 Materials and Methods

###  3.1 Dataset of German Cognates

We use cognate sets derived from Kluge’s etymological dictionary in a rather
recent, updated edition (Kluge, 2002). From the etymological dictionary of
German, we hand-selected 340 entries that had reflexes in Dutch, German, and
English with their proto-forms in Proto-Germanic, added phonetic
transcriptions, and provided phonetic alignments by annotating the data with
the help of the _EDICTOR_ tool (List, 2023).

In order to ease data sharing and reuse, the etymological dataset was shared
in the formats recommended by the Cross-Linguistic Data Formats initiative
(Forkel et al., 2018) using the workflow developed for the construction of the
Lexibank repository (https://lexibank.clld.org, List et al. 2022). This means
in this specific case that languages are linked to Glottolog
(https://glottolog.org, Hammarström et al. 2023) and that the individual
speech sounds employed in the phonetic transcription we provide follows the
Cross-Linguistic Transcription Systems (CLTS, https://clts.clld.org, List et
al. 2021). CLTS is a reference catalog for speech sounds which provides a
standard transcription system that defines a subset of the International
Phonetic Alphabet (IPA, 1999) as a standard (Anderson et al., 2018), which has
by now been mapped to several datasets providing phoneme inventory data
(Anderson et al., 2023) and also underlies most data in Lexibank.

###  3.2 Multilingual Semantic Vectors

For semantic vectors, we used the multilingual ConceptNet Numberbatch word
embeddings _version 19.08_ from Speer et al. (2017). The ConceptNet
Numberbatch word embeddings did not provide any data for the Dutch word form
_beukeboom_ ‘beech’, thus we deleted the German and English counterparts from
the data resulting in a set of 339 cognates in total. To ensure that the
embeddings capture semantic similarites of the cognate dataset, we computed
the cosine similarity for each word triplet across the languages. Figure 1
shows the distribution of cosine similarity values between language pairs.
While the peaks for all language pairs are located at around 0.9, indicating
an overall high semantic similarity of the word embeddings for cognate
triplets, some of the German-English data and Dutch-English data is
distributed over a lower cosine similarity range (green and red curve). This
results in less concentrated peaks for these language pairs. From this we can
conclude that German vs. Dutch cognates are semantically more similar than
German vs. English or Dutch vs. English cognates.

| English | German
---|---|---
Word | drink | trinken
IPA |  d R I N k |  t R I N k @ n
IPA (trimmed) |  d R I N k |  t R I N K @
Sound Classes | T R V N K | T R V N K V N
Sound Cl. (trimmed) | T R V N K | T R V N K V

Table 1: Exemplary data representation for English and German with full forms
vs. trimmed forms and sound class representations.

###  3.3 Multilingual Sound Classes

Scholars have proposed to test mutual intelligibility by representing word
forms in phonetic transcriptions and measuring string similarity for words
that express the same meaning (Tang and van Heuven, 2007). This approach to
intelligibility has, however, the disadvantage of not being able to test for
_asymmetric forms_ of intelligibility by which speakers of one language can
understand speakers of another language more properly than vice versa. For our
model-based approach, we need a more abstract – phonetically broader –
representation of speech sounds that allows us to capture broad phonetic
similarities in a multilingual setting. Taking inspiration from computational
approaches in historical linguistics, we decided to represent word forms with
_sound class models_. Sound classes have been first introduced by Dolgopolsky
(1986), who proposed 9 broad classes by which all possible consonants can be
represented, searching for cognates across distantly related languages. While
this is a really crude reduction of phonetic detail, Dolgopolsky sound classes
have been shown to work very well for comparative tasks (Turchin et al.,
2010). In our approach, we use Dolgopolsky’s original consonant classes and
represent vowels by an additional symbol.

The fact that our original data are provided in CLDF with standardized
phonetic transcriptions is a great advantage when it comes to the conversion
of phonetic strings to sound classes. Since sound class conversion routines
are readily available for phonetic transcriptions that conform to the standard
for IPA proposed in CLTS, converting the cognate sets in German, Dutch, and
English to sound classes requires very few preprocessing operations.

###  3.4 Trimming Word Forms

We experiment with two different representations of word forms, full forms and
trimmed forms, where we automatically exclude endings. Full forms reflect the
word forms as they are typically encountered in dictionaries (with nominative
case for nouns in German and infinitive endings for verbs). Bare stems are
typically used in historical language comparison in order to show how words
were historically related before they were modified in the respective
descendant languages by various morphological processes. In order to obtain
bare stems from our cognate sets in German, English, and Dutch, we make use of
the recently introduced technique for the _trimming_ of phonetic alignments
(Blum and List, 2023). With this technique, those sites (columns) in a
multiple phonetic alignment that show an exceeding amount of gaps (sounds that
do not have counterparts across all languages in the sample) are excluded from
the alignment. Although not identical with manually prepared word stem
representations, we find that applying this technique drastically reduces the
amount of gaps in the multiple alignments, while at the same time successfully
removing verb endings in our sample. Table 1 displays the representation of
our data with full forms vs. bare stems sound class representation.

###  3.5 Linear Discriminative Learning Model

In a first step, we evaluated the LDL model on the cognate data of each
language separately. The model is trained and tested on all 339 word forms.
Phonological input cues are 4-gram, 3-gram and 2-gram chunks of sound classes,
while multilingual word embeddings are representing the semantic vectors. In a
second step, we train the model on a single language, i.e. creating a naive
speaker of a language with zero exposure to other languages, and subsequently
test the model on the cognate data from the target language. In doing so, we
are replicating the setting of psycholinguistic studies but overcome the
limitations of previous language exposure to exclusively focus on the
predictiveness of historical sound classes as cues to mutual intelligibility.

###  3.6 Implementation

The experiments are implemented in the form of Python and Julia scripts. For
sound class conversion, we used the LingPy Python package (List and Forkel,
2023a). For the extraction of bare word stems through trimming, the LingRex
package was used (List and Forkel, 2023b). For the implementation of the LDL
models, the Linear Discriminative Learner from the _JudiLing_ package (an
implementation of DL in the _Julia_ programming language) was used (Luo et
al., 2021). Data and code needed to replicate the experiments from this study
are curated on GitHub (https://github.com/digling/intelligibility) and
archived with Zenodo (https://doi.org/10.5281/zenodo.10609356). Detailed
instructions on how to run the code are given in the repository.

|  | 4-grams | 3-grams | 2-grams
---|---|---|---
German | 0.99 | 0.93 | 0.51
Dutch | 1.0 | 0.93 | 0.52
English | 1.0 | 0.95 | 0.54
(a) Training data (full words)
|  | 4-grams | 3-grams | 2-grams
---|---|---|---
German | 0.99 | 0.92 | 0.50
Dutch | 1.0 | 0.89 | 0.48
English | 1.0 | 0.99 | 0.57
(b) Training data (trimmed words)

Table 2: Comprehension accuracies on full (a) and trimmed (b) training data.
Top-1 candidate is taken into account to compute accuracies.

Language Pair | (a) Full Word Forms | (b) Trimmed Word Forms
---|---|---
| 4-grams | 3-grams | 2-grams | 4-grams | 3-grams | 2-grams
GER-DUT | 0.57 (0.71) | 0.51 (0.68) | 0.28 (0.52) | 0.81 (0.86) | 0.75 (0.86) | 0.39 (0.65)
DUT-GER | 0.51 (0.67) | 0.48 (0.61) | 0.25 (0.48) | 0.82 (0.83) | 0.75 (0.83) | 0.40 (0.67)
GER-ENG | 0.68 (0.75) | 0.62 (0.73) | 0.29 (0.53) | 0.79 (0.85) | 0.75 (0.84) | 0.33 (0.59)
ENG-GER | 0.48 (0.59) | 0.46 (0.55) | 0.23 (0.45) | 0.60 (0.66) | 0.59 (0.64) | 0.32 (0.53)
DUT-ENG | 0.68 (0.75) | 0.6313 (0.72) | 0.31 (0.55) | 0.77 (0.84) | 0.71 (0.81) | 0.30 (0.60)
ENG-DUT | 0.53 (0.64) | 0.50 (0.59) | 0.29 (0.49) | 0.60 (0.67) | 0.59 (0.64) | 0.35 (0.54)

Table 3: Comprehension accuracies of multilingual models for comprehension for
(a) full word forms and (b) trimmed word forms. Values without brackets
indicate results when the top-1 candidate is considered to compute accuracies,
values in brackets indidcate results when top-5 candidates are considered.

##  4 Evaluation

###  4.1 Evaluation on Individual Languages

Table 2(a) displays the comprehension accuracies on the training data for full
word forms. For the evaluation process only the predicted meaning, the top-1
candidate, was considered. The evaluation results suggest a good comprehension
memory of the model when Dolgopolsky sound classes are provided as 4-gram or
3-gram chunks. If sound classes are fed into the model as 2-gram chunks we
observe a substantial drop in accuracy, indicating a reduced discriminative
power to predict a semantic vector _$\hat{S}$_ that is similar to the gold
standard vector _$S$_ of the training language. Table 2(b) displays the
evaluation results after trimming word forms. Comprehension accuracy remains
high for 4-gram and 3-gram chunks. Again, the accuracy drops substantially
when 2-gram chunks are taken into account.

###  4.2 Evaluation Across Languages

Table 3(a) illustrates the result of the multilingual models for full word
forms. The first column contains the training-test language pairs. Values
without brackets indicate accuracies when the top-1 candidate was taken into
account for evaluation, values in brackets indicate accuracies when the
correct meaning among top-5 candidates was considered. Allowing the model to
evaluate comprehension accuracy based on a set of top-5 candidates accounts
for possible confusion of the target word form with similar word forms, giving
the model room for multiple answers. The cross-linguistic comprehension
results in Table 3(a) unsurprisingly replicate the chunk size effect we have
seen in our training models, with 4-gram chunks providing the best
comprehension results. We observe the best comprehension results for the
language pair Dutch-English with an accuracy of 68% (75% for an evaluation on
top-5 candidates), followed by German-English and German-Dutch. The worst
comprehension results are given for a training on English and a test on German
cognates (see row 4 of Table 3(a)). Gooskens and Swarte (2017) report a
similar result for human participants, indicating that our LDL models show a
human-like performance when assessing comprehension abilities across
languages.

Table 3(b) displays the comprehension accuracies after applying the trimming
procedure. Trimming phonetic alignments results in a substantial rise of
prediction accuracies with Dutch-German, German-Dutch and German-English
providing the best comprehension results. Again, the language pair English-
German shows the lowest comprehension accuracy, similar to human results
(Gooskens and Swarte, 2017).

##  5 Discussion and Conclusion

In this study we presented a computer-assisted method to mutual
intelligibility based on a model that captures the cognitive processes by
which humans comprehend languages. We expanded the model with multilingual
semantic vectors and multilingual sound classes. Our multilingual sound
classes were predictive when a combination of at least 3 sound classes is
given, indicating that knowing the order of sound classes allows the model to
comprehend languages from the same language family. However, we observe an
effect of the training language, with English being the least advantageous
language in our setting and in the data of Gooskens and Swarte (2017) with
human participants. We report a higher accuracy for German-English than
German-Dutch, again in line with the human data of Gooskens and Swarte (2017).
If sound classes are trimmed, we find the opposite effect. The pair Dutch-
English shows better comprehension accuracies than Dutch-German, again with
the opposite picture for the trimmed version. From a language learning
perspective, the change of direction, i.e. the better prediction for German-
Dutch and Dutch-German after trimming would imply a certain morphological
knowledge of speakers. Speakers of German or Dutch knowing verb endings and
ignoring them purposefully have an advantage in comprehending English. Our
proposed model does not only offer a new method for automatic testing of
mutual intelligibility but shows clear similarities to data obtained from
human participants, making it a useful cognitive tool for research on language
comprehension.

## Supplementary Material

All data and code needed to replicate the experiments discussed in this study
are curated on GitHub (https://github.com/digling/intelligibility) and
archived with Zenodo (https://doi.org/10.5281/zenodo.10609356). The German
cognate dataset is also curated on GitHub
(https://github.com/lexibank/germancognates) and archived with Zenodo
(https://doi.org/10.5281/zenodo.10609476).

## Limitations

While our model offers some fruitful results for further investigation of
mutual intelligibility, the dataset we provided contains a limited amount of
carefully selected historical cognates. It remains to be seen how the model
would deal with a much larger set of random words. Moreover, we cannot account
for other language families or other languages than German, Dutch and English.
However, we see our modeling procedure as a starting point for assessing
mutual intelligibility computationally. For that reason, limiting our data to
historical cognates and three languages only is a necessary step. For a
complete picture, more languages from the Germanic language family need to be
tested and the results need to be compared with comprehension results for
other language families.

## Ethics Statement

This research does not involve human or animal data. No potential ethical
conflict or conflict of interest was reported by the authors.

## Acknowledgements

This research was supported by the Max Planck Society Research Grant _CALC³_
(JML, https://digling.org) and the ERC Consolidator Grant _ProduSemy_ (JML,
Grant No. 101044282, see https://doi.org/10.3030/101044282). Views and
opinions expressed are however those of the authors only and do not
necessarily reflect those of the European Union or the European Research
Council Executive Agency (nor any other funding agencies involved). Neither
the European Union nor the granting authority can be held responsible for
them. We thank Maria Heitmeier and Harald Baayen for their valuable input
regarding the computational models used in this study.

## References

  * Anderson et al. (2018) Cormac Anderson, Tiago Tresoldi, Thiago Costa Chacon, Anne-Maria Fehn, Mary Walworth, Robert Forkel, and Johann-Mattis List. 2018.  A Cross-Linguistic Database of Phonetic Transcription Systems.  _Yearbook of the Poznań Linguistic Meeting_ , 4(1):21–53.
  * Anderson et al. (2023) Cormac Anderson, Tiago Tresoldi, Simon J. Greenhill, Robert Forkel, Russell D. Gray, and Johann-Mattis List. 2023.  Variation in phoneme inventories: quantifying the problem and improving comparability.  _Journal of Language Evolution_ , 0(0):1–20.
  * Baayen et al. (2019) R. Harald Baayen, Yu-Ying Chuang, Elnaz Shafaei-Bajestan, and James P. Blevins. 2019.  The discriminative lexicon: A unified computational model for the lexicon and lexical processing in comprehension and production grounded not in (de)composition but in linear discriminative learning.  _Complexity_ , 2019.
  * Blum and List (2023) Frederic Blum and Johann-Mattis List. 2023.  Trimming phonetic alignments improves the inference of sound correspondence patterns from multilingual wordlists.  In _Proceedings of the 5th Workshop on Computational Typology and Multilingual NLP_ , pages 52–64. Association for Computational Linguistics.
  * Casad (1987) Eugene H. Casad. 1987.  _Dialect intelligibility testing_.  Summer Institute of Linguistics, Dallas.
  * Chuang et al. (2018) Yu-Ying Chuang, Melanie J. Bell, Isabelle Banke, and R. Harald Baayen. 2018.  Bilingual and multilingual mental lexicon: A modeling study with linear discriminative learning.  _Language Learning_ , 71(S1):219–292.
  * Chuang et al. (2023) Yu-Ying Chuang, Mihi Kang, Luo Xuefeng, and R. Harald Baayen. 2023.  Vector space morphology with linear discriminative learning.  In Davide Crepaldi, editor, _Linguistic Morphology in the Mind and Brain_ , 1st edition, pages 17–248. Routledge, London.
  * Dolgopolsky (1986) Aharon B. Dolgopolsky. 1986.  A probabilistic hypothesis concerning the oldest relationships among the language families of northern Eurasia.  In Vitalij V. Shevoroshkin, editor, _Typology, Relationship and Time_ , pages 27–50. Karoma Publisher, Ann Arbor.
  * Forkel et al. (2018) Robert Forkel, Johann-Mattis List, Simon J. Greenhill, Christoph Rzymski, Sebastian Bank, Michael Cysouw, Harald Hammarström, Martin Haspelmath, Gereon A. Kaiping, and Russell D. Gray. 2018.  Cross-Linguistic Data Formats, advancing data sharing and re-use in comparative linguistics.  _Scientific Data_ , 5(180205):1–10.
  * Gooskens and Swarte (2017) Charlotte Gooskens and Femke Swarte. 2017.  Linguistic and extra-linguistic predictors of mutual intelligibility between Germanic languages.  _Nordic Journal of Linguistics_ , 40(2):123–147.
  * Gooskens et al. (2015) Charlotte Gooskens, Renée van Bezooijen, and Vincent J. van Heuven. 2015.  Mutual intelligibility of Dutch-German cognates by children: The devil is in the detail.  _Linguistics_ , 53(2):255–283.
  * Gooskens et al. (2018) Charlotte Gooskens, Vincent J. van Heuven, Jelena Golubović, Anja Schüppert, Femke Swarte, and Stefanie Voigt. 2018.  Mutual Intelligibility between Closely Related Languages in Europe.  _International Journal of Multilingualism_ , 15(2):169–193.
  * Hammarström et al. (2023) Harald Hammarström, Martin Haspelmath, Robert Forkel, and Sebastian Bank. 2023.  _Glottolog. Version 4.8_.  Max Planck Institute for Evolutionary Anthropology, Leipzig.
  * Heeringa et al. (2014) Wilbert Heeringa, Femke Swarte, Anja Schüppert, and Charlotte Gooskens. 2014.  Modeling Intelligibility of Written Germanic Languages: Do We Need to Distinguish Between Orthographic Stem and Affix Variation? _Journal of Germanic Linguistics_ , 26(4):361–394.
  * Hongyan (2017) Wang Hongyan. 2017.  _English as a lingua franca: Mutual intelligibility of Chinese, Dutch and American speakers of English_.  Ph.D. thesis, University of Utrecht.
  * IPA (1999) International Phonetic Organisation IPA. 1999.  _Handbook of the International Phonetic Association_.  Cambridge University Press, Cambridge.
  * Kluge (2002) Friedrich Kluge. 2002.  _Etymologisches Wörterbuch der deutschen Sprache_ , 24 edition.  de Gruyter, Berlin.
  * List (2016) Johann-Mattis List. 2016.  Beyond cognacy: Historical relations between words and their implication for phylogenetic reconstruction.  _Journal of Language Evolution_ , 1(2):119–136.
  * List (2023) Johann-Mattis List. 2023.  _EDICTOR. A web-based tool for creating, editing, and publishing etymological datasets [Software Tool, Version 2.1.0]_.  MCL Chair at the University of Passau, Passau.
  * List et al. (2021) Johann-Mattis List, Cormac Anderson, Tiago Tresoldi, and Robert Forkel. 2021.  _Cross-Linguistic Transcription Systems. Version 2.1.0_.  Max Planck Institute for the Science of Human History, Jena.
  * List and Forkel (2023a) Johann-Mattis List and Robert Forkel. 2023a.  _LingPy. A Python library for quantitative tasks in historical linguistics [Software Library, Version 2.6.13]_.  MCL Chair at the University of Passau, Passau.
  * List and Forkel (2023b) Johann-Mattis List and Robert Forkel. 2023b.  _LingRex: Linguistic reconstruction with LingPy_.  Max Planck Institute for Evolutionary Anthropology, Leipzig.
  * List et al. (2022) Johann-Mattis List, Robert Forkel, Simon J. Greenhill, Christoph Rzymski, Johannes Englisch, and Russell D. Gray. 2022.  Lexibank, a public repository of standardized wordlists with computed phonological and lexical features.  _Scientific Data_ , 9(316):1–31.
  * Luo et al. (2021) Xuefeng Luo, Yu-Ying Chuang, and R. Harald Baayen. 2021.  JudiLing: An implementation in Julia of linear discriminative learning algorithms for language modeling.
  * Nieder et al. (2023) Jessica Nieder, Yu-Ying Chuang, Ruben van de Vijver, and R. Harald Baayen. 2023.  A discriminative lexicon approach to word comprehension, production, and processing: Maltese plurals.  _Language_ , 99(2):242–274.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  ConceptNet 5.5: An open multilingual graph of general knowledge.  In _Proceedings of the AAAI Conference on Artificial Intelligence 2017_ , pages 4444–4451.
  * Squires et al. (2020) Lindsey R. Squires, Sara J. Ohlfest, Kristen E. Santoro, and Jennifer L. Roberts. 2020.  Factors influencing cognate performance for young multilingual children’s vocabulary: A research synthesis.  _American Journal of Speech-Language Pathology_ , 29(4):2170–2188.
  * Tang and van Heuven (2007) Chaoju Tang and Vincent J. van Heuven. 2007.  Mutual intelligibility and similarity of Chinese dialects. Predicting judgments from objective measures.  In Bettelou Los and Marjo van Koppen, editors, _Linguistics in the Netherlands 2007_ , pages 223–234. John Benjamins Publishing Company.
  * Tang and van Heuven (2009) Chaoju Tang and Vincent J. van Heuven. 2009.  Mutual intelligibility of Chinese dialects experimentally tested.  _Lingua_ , 119(5):709–732.
  * Turchin et al. (2010) Peter Turchin, Ilja Peiros, and Murray Gell-Mann. 2010.  Analyzing genetic connections between languages by matching consonant classes.  _Journal of Language Relationship_ , 3:117–126.


# Rethinking Negative Instances for Generative Named Entity Recognition

Yuyang Ding1, Juntao Li1, Pinzheng Wang1, Zecheng Tang1, Bowen Yan2, Min
Zhang1
1Soochow University 2Tsinghua University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author

###### Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities for
generalizing in unseen tasks. In the Named Entity Recognition (NER) task,
recent advancements have seen the remarkable improvement of LLMs in a broad
range of entity domains via instruction tuning, by adopting entity-centric
schema. In this work, we explore the potential enhancement of the existing
methods by incorporating negative instances into training. Our experiments
reveal that negative instances contribute to remarkable improvements by (1)
introducing contextual information, and (2) clearly delineating label
boundaries. Furthermore, we introduce a novel and efficient algorithm named
Hierarchical Matching, which is tailored to transform unstructured predictions
into structured entities. By integrating these components, we present GNER, a
Generative NER system that shows improved zero-shot performance across unseen
entity domains. Our comprehensive evaluation illustrates our system’s
superiority, surpassing state-of-the-art (SoTA) methods by 11 $F_{1}$ score in
zero-shot evaluation.111Code, datasets and models will be released at
https://github.com/yyDing1/GNER.

##  1 Introduction

Named Entity Recognition (NER) is a critical and challenging task in the field
of Natural Language Processing (NLP). Previous NER models are constrained by a
pre-defined label set and require extensive human annotations, which limits
their flexibility and adaptability to unseen entity domains. Recent advantages
in LLMs have enabled the models to be capable of generalizing to unseen tasks
Ouyang et al. (2022); Achiam et al. (2023) in an auto-regressive generation
manner, making it possible to construct powerful NER systems. However, despite
these advancements, recent studies Wei et al. (2023); Li et al. (2023) show
that the zero-shot performance of LLMs still falls behind the supervised
training state-of-the-art (SoTA) methods, as LLMs train with limited NER data.

Figure 1: Zero-shot performance of our models. Our models GNER-LLaMA and
GNER-T5 both outperform the SoTA Sainz et al. (2023) in zero-shot settings.
GPT results are from Zhou et al. (2023).

To bridge this gap, recent works have fine-tuned open-sourced LLMs on diverse
NER datasets, enhancing their domain adaptability for NER tasks. They utilize
varied task schemas to handle NER tasks across multiple domains. Specifically,
InstructUIE Wang et al. (2023) is fine-tuned on a wide range of IE datasets
using a single-round conversation manner. Meanwhile, UniversalNER Zhou et al.
(2023) found that querying all entities at once is less effective than making
multiple inquiries, with each inquiry focusing on one entity type at a time.
Additionally, GoLLIE Sainz et al. (2023) enhances zero-shot performance with
well-crafted code-style guidelines. However, these approaches primarily adopt
an entity-centric training strategy, focusing on recognizing entities while
overlooking the non-entity text, which is crucial as negative instances.
Actually, negative instances play an important role in traditional
classification models like BERT Tagging Devlin et al. (2018). For generative
models, the role of negative instances in the training process has not yet
been fully explored.

To calibrate the potential enhancement of including negative instances in
training, we first conduct a preliminary study. We choose Flan-T5-large Chung
et al. (2022) as the generative backbone model and design strategies for
training with negative instances. Through experiments, we show that negative
instances can significantly boost the model’s performance by (1) incorporating
the contextual information, and (2) enhancing the label boundary between
entities and non-entities. The possible drawback of introducing the negative
instances is the increase of the prediction length, leading to inaccurate
predictions, reflected by the word addition, omission and substitution. To
tackle the inaccuracy drawbacks, we aim to design a more accurate and
efficient algorithm to convert unstructured text into structured entities.

Inspired by the above observations, we design an effective and efficient
Generative NER framework named GNER. We first design a proper task schema
integrating negative instances into the instruction tuning process.
Additionally, we design a Hierarchical Matching algorithm to tackle the issues
in the structuring process efficiently. This innovation ensures accurate
categorization and alignment of extracted entities. We also demonstrate that
zero-shot performance can be enhanced with beam search through a self-
correction mechanism. These strategic developments collectively advance the
GNER framework, setting a new standard for accuracy and efficiency in the
field of NER.

We conduct experiments on two representative generative models, Flan-T5 and
LLaMA. The resulting models, GNER-T5 and GNER-LLaMA, outperform SoTA by a
large margin. As stated in Fig. 1, GNER-LLaMA-7B outperforms the GoLLIE Sainz
et al. (2023) trained on Code-LLaMA-7B by 8 $F_{1}$ score. Furthermore,
compared to the similarly configured model UniversalNER, GNER-LLaMA-7B shows
an improvement of 12.7 $F_{1}$ score, with a 2.5$\times$ boost in inference
speed. Our GNER-T5-11B model also achieves SoTA performance in both zero-shot
and supervised settings.

##  2 Related Work

#### Named Entity Recognition

Early works format Named Entity Recognition (NER) as a sequence labeling
problem Chiu and Nichols (2016); Huang et al. (2015); Akbik et al. (2018); Qin
et al. (2019). Among these, BERT Tagging Devlin et al. (2018) is the most
representative one. Then, different methods are proposed to address more
complex scenarios, i.e., nested and discontinuous NER. These methods regard
NER as question answering Li et al. (2020a); Mengge et al. (2020), span
classification Fu et al. (2021); Li et al. (2020b), dependency parsing Yu et
al. (2020), word-level relation classification Li et al. (2022a), and so on.
In most of these approaches, negative instances have played a crucial role in
the training process, either by integrating all negative instances or
employing sampling methods to select part of them Li et al. (2022b). However,
the performance of the above-mentioned supervised models significantly
decreases in zero-shot settings Liu et al. (2021), especially when the data
and domain distribution significantly diverge from those seen of training.

#### Zero-shot NER

Figure 2: A simplified example of instructions in InstructUIE Wang et al.
(2023), UniversalNER Zhou et al. (2023) and GoLLIE Sainz et al. (2023).

Instruction tuning Wei et al. (2021); Chung et al. (2022), also known as
multi-task fine-tuning, has emerged as a leading method to achieve
generalization to unseen tasks by fine-tuning pre-trained LLMs on a diverse
collection of tasks phrased as text-to-text problems Longpre et al. (2023). In
NER, numerous works have explored the potential of LLMs across diverse
domains. For instance, InstructUIE Wang et al. (2023) is fine-tuned on a wide
range of IE datasets and achieves impressive results in both zero-shot and
supervised settings. UniversalNER Zhou et al. (2023) explores the
effectiveness of knowledge distillation and multi-round conversational
training paradigms in enhancing model generalization, achieving superior
results. GoLLIE Sainz et al. (2023) introduces an innovative strategy by
integrating well-crafted code-style guidelines into instructions, which has
been found to further improve the model’s zero-shot performance. A simplified
version of the task schema of the mentioned methods above is shown in Fig. 2.
It can be concluded that the task schema plays an important role in
determining the learning paradigm of models, significantly influencing their
performance. We observe that these methods are entity-centric, meaning only
the entity portions are involved in training and used for backpropagation to
train the model.

##  3 Preliminary Study

To explore possible improvements of recent entity-centric schema, we launch a
preliminary exploration from two perspectives: (1) incorporating the context
of entities into the training process and (2) enhancing the label boundary
between entities and non-entities. We then discuss the accompanying longer
sequence prediction issues to LLMs when presenting context and label
boundaries.

###  3.1 Definitions

In this part, we give the formal definitions of the entity-centric method and
negative instances. The Named Entity Recognition (NER) Task can be formally
defined as a function mapping of input tokens
$X=\\{x_{1},x_{2},\ldots,x_{n}\\}$ and a pre-defined set of entity types
$L=\\{l_{1},l_{2},\ldots l_{m}\\}$ to entity labels
$Y=\\{y_{1},y_{2},\ldots,y_{n}\\}$. The positive and negative instances can be
formulated as follows:

$\displaystyle P$ $\displaystyle=\left\\{(x_{i},y_{i})\mid
i\in\\{1,\ldots,n\\},y_{i}\in L\right\\},$ (1) $\displaystyle N$
$\displaystyle=\left\\{(x_{i},y_{i})\mid
i\in\\{1,\ldots,n\\},y_{i}=O\right\\},$

where O represents non-entity text. Entity-centric generative methods train
the model using $P$ as training instances. It is worth noting that the
training instances mentioned here refer to tokens that directly influence the
model’s weights through the backpropagation algorithm during training.

An example in the Preliminary Study Definition (section 3.1) Token inputs
($X$): John explored Tokyo , sampling its famed sushi , and flew back to New
York . Entity type ($L$): [Person, Location] Training prompt (section 3.2 and
3.3) w/o context: [John](Person) [Tokyo](Location) [New York](Location). w/
context length 2: [John](Person) explored [Tokyo](Location) , sampling …… back
to [New York](Location) . w/ full context: [John](Person) explored
[Tokyo](Location) , sampling its famed sushi , and flew back to [New
York](Location). w/ full context and label boundary: John(B-Person)
explored(O) Tokyo(B-Location) ,(O) sampling(O) its(O) famed(O) sushi(O) ,(O)
and(O) flew(O) back(O) to(O) New(B-Location) York(I-Location) .(O) Perdiction
Issues (section 3.4) Omission and Addition: John(B-Person) first(O)
explored(O) Tokyo(B-Location) ……(omitted text) and(O) then(O) flew(O) back(O)
to(O) New(B-Location) York(I-Location) .(O) Substitution: John(B-Person)
explored(O) Tokyo(B-Location) ,(O) sampling(O) its(O) famed(O) sushi(O) ,(O)
and(O) returned(O) to(O) New(B-Location) York(I-Location) .(O) Figure 3:
Prompts that are used in our preliminary study.

###  3.2 Learning with Entity Context

The context surrounding entities plays a significant role in determining their
categories. For example, phrases like “go to” and “travel to” are often
followed by a Location entity. We integrate the contextual information before
and after an entity into our training process to explicitly enable the model
to recognize entities based on their surrounding context. To achieve this, we
introduce negative instances that are closest to the entity, extending up to a
length $L$, until encountering the boundary of the sentence, as part of our
training instances. The remaining negative instances that are not selected as
contextual information are represented by ellipses. An example of our
constructed prompt used for training is shown in Fig. 3. From the example, we
can also glimpse the importance of context, such as “flew to” guiding the
model to pay more attention to the subsequent entity “New York”, rather than
just remembering “New York” as a location in the training process. We conduct
experiments to explore the impact of contextual length. For the experimental
setup, we choose Flan-T5-large, a model with 783M parameters, as our backbone.
Additionally, we sample 10K samples from the Pile-NER Zhou et al. (2023)
dataset as our training set and 200 samples for each subtask of CrossNER Liu
et al. (2021) validation set to evaluate the model’s zero-shot performance.
Results are shown in Fig. 4. The improvement from a contextual length from 0
to 1 is particularly notable, suggesting that the closest negative instance to
the entity plays a significant role in identifying the current entity. As the
contextual length increases, performance progressively improves, showing that
the model benefits from the context. Yet, beyond a certain point, the speed of
improvement slows down, indicating the amount of contextual information
available in the entity context is approaching saturation.

Figure 4: Zero-shot performance of training with entity context and enhanced
boundary strategies. A contextual length of 0 indicates no context is
included, while a length of $N$ signifies that the entire sentence is
included.

###  3.3 Entity Boundary of Generative Model

Method | $\text{Strict }F_{1}$ | $\text{Partial }F_{1}$ | $\delta$
---|---|---|---
UniNER-7B | 53.4 | 56.7 | 3.3
UniNER-7B w/ sup | 61.8 | 64.8 | 3.0
Table 1: Zero-shot performance of UniversalNER Zhou et al. (2023). $\delta$
denotes the difference between strict match $F_{1}$ and partial match $F_{1}$,
indicating the model’s ability to distinguish entity boundaries.

The above analysis has demonstrated the effectiveness of entity contexts. We
then introduce label information of contexts to enhance the label boundary
between entities and non-entities. To some extent, the results of Zhou et al.
(2023) expose the problem of vague entity boundaries in model predictions, as
shown by the significantly higher $F_{1}$ score for partial match compared to
strict match. Specifically, strict match requires the boundary to exactly
match the ground truth, while partial match considers predicted entities that
overlap with but do not strictly match the ground truth as half correct,
counting them as 0.5 in true positive. We list their results in Table 1.
Motivated by this, we further enhance the label boundary between entities and
non-entities by the following strategies: (1) for the non-entity part, we
label the context surrounding an entity with the negative label “O”, (2) for
the entity part, the beginning of an entity is marked as “B-”, and the inside
of the entity is marked as “I-”. The training prompt and results are shown in
Fig. 3 and Fig. 4, respectively. Compared to training with contexts alone,
incorporating our enhanced label boundary strategies results in consistent
improvements across various contextual lengths.

###  3.4 Long Sequence Prediction Issues

The prediction length of the model increases with the integration of entity
contexts and label boundaries. A longer generation sequence might bring
challenges to the popular generative LLMs, as illustrated by the example in
Fig. 3. The model’s output may include omissions, additions, and substitutions
of words. For instance, it may add conjunctions “first” and “then” to
sentences, omit irrelevant non-entity text, such as “sampling its framed
sushi”, or replace “flew back” with “returned”. We launch a detailed case
study and find the potential causes of these issues: (1) noise in the original
text, (2) missing words in the vocabulary, and (3) accumulative exposure bias.
The representative examples, issue proportion, along with detailed analysis,
are documented in Appendix C.

##  4 Method

In this section, we present our GNER framework. We start by describing our
task schema, which integrates negative instances into the training process for
better usage of contextual information (section 3.2) and sensitivity to the
entity boundaries (section 3.3), followed by the correlated tuning strategies.
In response to the issues in section 3.4, we propose an effective algorithm,
named Hierarchical Matching, to convert the model’s unstructured text outputs
into structured data, thereby enhancing the accuracy of our system.

###  4.1 Task Schema

Integrating negative instances, specifically those parts of the sentence
labeled as “O” to indicate non-entity text, enhances the generative process by
including contextual information and the discrimination of entity boundaries,
thereby boosting the model’s performance, as detailed in section 3. Due to the
token-by-token generation paradigm of generative models, we design a token-by-
token prediction task schema, where the model predicts the category of each
token as it generates them, either entities or non-entities. This schema
offers a more direct and focused way, where each token is annotated
individually and assigned a specific entity label based on its context within
the sequence.

Model | Backbone | AI | Literature | Music | Politics | Science | Movie | Restaurant | Avg
---|---|---|---|---|---|---|---|---|---
ChatGPT | - | 52.4 | 39.8 | 66.6 | 68.5 | 67.0 | 5.3 | 32.8 | 47.5
InstructUIE† | flan-t5-xxl (11B) | 49.0 | 47.2 | 53.2 | 48.2 | 49.3 | 63.0 | 21.0 | 47.3
GoLLIE-7B† | Code-LLaMA-7B | 59.1 | 62.7 | 67.8 | 57.2 | 55.5 | 63.0 | 43.4 | 58.4
GoLLIE-13B† | Code-LLaMA-13B | 56.7 | 59.7 | 65.5 | 54.4 | 56.2 | 62.5 | 49.8 | 57.8
UniNER-7B‡ | LLaMA-7B | 53.5 | 59.4 | 65.0 | 60.8 | 61.1 | 42.4 | 31.7 | 53.4
UniNER-13B‡ | LLaMA-13B | 54.2 | 60.9 | 64.5 | 61.4 | 63.5 | 48.7 | 36.2 | 55.6
GLiNER-L‡ | DeBERTa-v3-304M | 57.2 | 64.4 | 69.6 | 72.6 | 62.6 | 57.2 | 42.9 | 60.9
GNER-T5‡ | flan-t5-base (248M) | 56.8 | 58.7 | 72.3 | 64.5 | 68.0 | 54.5 | 41.4 | 59.5
flan-t5-large (783M) | 62.6 | 58.2 | 76.7 | 67.0 | 72.6 | 58.6 | 48.6 | 63.5
flan-t5-xl (3B) | 62.1 | 64.9 | 80.6 | 73.7 | 68.7 | 63.0 | 49.8 | 66.1
flan-t5-xxl (11B) | 68.2 | 68.7 | 81.2 | 75.1 | 76.7 | 62.5 | 51.0 | 69.1
GNER-LLaMA‡ | LLaMA-7B | 63.1 | 68.2 | 75.7 | 69.4 | 69.9 | 68.6 | 47.5 | 66.1
Table 2: Zero-shot evaluation results, where $\dagger$ denotes IE Models and
$\ddagger$ denotes NER Models. Results for ChatGPT and UniNER are from Zhou et
al. (2023); InstructUIE are from Wang et al. (2023); GoLLIE are from Sainz et
al. (2023); GLiNER-L are from Zaratiana et al. (2023). We bold the best
results and underline the second-best results. More details about the
performance including error bars are shown in Appendix E.

###  4.2 Instrucion Tuning

Instruction Tuning Prompt Task Description: Please analyze the sentence
provided, identifying the entity type for each word on a token-by-token basis.
Output format is: word_1(label_1), word_2(label_2), … Guideline: We’ll use the
BIO-format to label the entities, where: 1\. B- (Begin) indicates the start of
a named entity. 2\. I- (Inside) is used for words within a named entity but
are not the first word. 3\. O (Outside) denotes words not part of a named
entity. Use the specific entity tags: $l_{1},l_{2},\ldots$, $l_{m}$ and O.
Input: $x_{1}\quad x_{2}\quad\ldots\quad x_{n}$ Output:
$x_{1}(\hat{y}_{1})\quad x_{2}(\hat{y}_{2})\quad\ldots\quad
x_{n}(\hat{y}_{n})$ Figure 5: Prompt that are used for instruction tuning.

#### Instruction Format

As shown in Fig. 5, our designed instruction prompt includes four parts: task
description, guideline, input, and output. To enhance our model’s ability to
generalize across diverse labels and effectively handle real-world data, we
implement some regularization strategies: (1) class order shuffling, where the
order of entity classes is randomly shuffled, and (2) external entity
sampling222Zhou et al. (2023) refers to this as negative entity sampling,
which is different from the negative instances discussed in this work. We term
it “external” to differentiate it., involving the entity types that are absent
in the given sentence in the training prompt.

#### Task Adaption & Supervised fine-tuning

Zero-shot capabilities of LLMs in NER are limited due to their exposure to
relatively little NER data during training. To equip the model with
capabilities specific to NER tasks, we first perform task adaptation on NER
data spanning various domains. Subsequently, to assess the model’s zero-shot
capabilities, we evaluate it against unseen entity types. We proceed to
extensively fine-tune our models on a wide range of publicly available NER
data, aiming to enhance our model’s effectiveness in supervised settings,
followed by supervised evaluations.

###  4.3 Hierarchical Matching Algorithm

To handle the omission, addition, and substitution problems outlined in the
generated predictions in our task schema, as detailed in section 3.4, we
develop a Hierarchical Matching algorithm that provides a straightforward and
effective solution to these challenges. Formally, given a sentence
$X=\\{x_{1},x_{2},\ldots,x_{n}\\}$, the generated outputs can be formatted as
$``\tilde{x}_{1}(\tilde{y}_{1})\quad\tilde{x}_{2}(\tilde{y}_{2})\quad\ldots\quad\tilde{x}_{m}(\tilde{y}_{m})"$.
Firstly, we utilize regular expression matching to obtain the predicted
sequence $\tilde{X}=\\{\tilde{x}_{1},\tilde{x}_{2},\ldots,\tilde{x}_{m}\\}$
and the corresponding answers
$\tilde{Y}=\\{\tilde{y}_{1},\tilde{y}_{2},\ldots,\tilde{y}_{m}\\}$. Due to the
inherent uncertainties in generation, $\tilde{X}$ often differs from $X$.
Next, we establish a one-to-one correspondence between the words in the
original sequence $X$ and the generated sequence $\tilde{X}$, then map the
labels of the corresponding words $\tilde{Y}$ back to obtain the final
prediction results $\hat{Y}$. A common method involves calculating the Longest
Common Subsequence (LCS) between $X$ and $\tilde{X}$ to identify the
correspondence between words, using the classic dynamic programming algorithm
with time complexity of $O(N^{2})$. Combined with the actual NER task
scenarios and our task schema, we make the following two optimizations in the
matching algorithm and condition:

#### LCS Matching Algorithm

We optimize the complexity of the LCS algorithm using a hierarchical divide-
and-conquer approach through the following steps: (1) If the sequence does not
have the above problem, i.e., $\tilde{X}=X$, it is obvious that
$\hat{Y}=\tilde{Y}$. The time complexity is $O(N)$, (2) For the omission case,
where $\tilde{X}$ is a subsequence of $X$, the matching process can be
accomplished in $O(N)$ through greedy matching. (3) In other cases, we have
implemented a fast version of the LCS algorithm Hunt and Szymanski (1977)
within $O(N\log N)$, based on the nature of the small number of duplicate
words in $\tilde{X}$.

#### Back Tokenization

One notable problem in the matching process is the missing words in the
vocabulary, as detailed in our case study in section C. For example,
“antropología” in the original text becomes ‘antropologa’ in the model’s
predictions, resulting in an inaccurate match in the matching process. To
address this, we employ back tokenization, which involves tokenizing each word
in the original text and then detokenizing it to match a word in the model’s
vocabulary, thereby creating a more resilient matching condition.

##  5 Experiments

###  5.1 Settings

#### Datasets

The datasets used in our experiments include: (1) Task Adaptation Datasets:
Following the setting of Zhou et al. (2023), we first train our model with
Pile-NER, which consists of approximately 240K entities across 13K distinct
entity categories. These passages are sampled from the Pile Corpus Gao et al.
(2020) and subsequently processed using ChatGPT to generate the inherent
entities openly. To evaluate the model’s zero-shot performance in unseen
entity types, we adopt two widely-used datasets, i.e., CrossNER Liu et al.
(2021) and MIT Liu et al. (2013). (2) Supervised Datasets: Following the task
adaptation phase, the performance of the model can be further enhanced by
training across a wide range of well-annotated NER datasets Zhou et al.
(2023). To achieve this, we compile 18 public NER datasets in the BIO format
for additional training, subsequently assessing performance on the test splits
of these 18 datasets. From the 20 datasets used in Wang et al. (2023), we
exclude two nested NER datasets, ACE2005 and GENIA, due to their
incompatibility with the BIO format. Following the settings of Wang et al.
(2023), we randomly select 10K data points from each dataset to create a mixed
set. In cases where a dataset contains fewer than 10K samples, we incorporate
its entire dataset. Additional information regarding the datasets is available
in Appendix A.

#### Compared Baselines

Our main point of comparison is UniversalNER Zhou et al. (2023) as it is the
approach closest to our system, with similar data and training procedures.
Another baseline considered for comparison is GLiNER Zaratiana et al. (2023),
which utilizes bi-directional models to match entity types with textual spans
in a latent space. We also include some strong Information Extraction (IE)
systems like InstructUIE Wang et al. (2023), which is based on Flan-T5-xxl
Chung et al. (2022) and fine-tuned on diverse information extraction datasets,
and GoLLIE Sainz et al. (2023), which is based on Code-LLaMA Roziere et al.
(2023), and use guidelines to improve model’s zero-shot performance. We use
strict entity-level micro-$F_{1}$ as the evaluation metric for comparison.

#### Backbones & Implementation

Generative models typically consist of two types of architectures, i.e., the
encoder-decoder architecture and the decoder-only architecture. We conduct
experiments on both of these architectures. Specifically, we select Flan-T5
(encoder-decoder) and LLaMA (decoder-only) as our backbone models. To ensure a
fair comparison, our training settings for GNER-T5 align with those of
InstructUIE Wang et al. (2023), and those for GNER-LLaMA are consistent with
UniversalNER Zhou et al. (2023). Due to our model producing longer output
sequences, we implement longer length limits for both input and output. More
details can be found in Appendix B.

Method | ChatGPT | InstructUIE | GNER-T5 | $\Delta$ | UniNER | GNER-LLaMA | $\Delta$
---|---|---|---|---|---|---|---
Backbone | - | flan-t5-xxl (11B) | LLaMA-7B
AnatEM | 30.7 | 88.52 | 90.30 | +1.78 | 88.65 | 90.24 | +1.59
bc2gm | 40.2 | 80.69 | 84.29 | +3.60 | 82.42 | 83.18 | +0.76
bc4chemd | 35.5 | 87.62 | 90.04 | +2.42 | 89.21 | 89.40 | +0.19
bc5cdr | 52.4 | 89.02 | 89.95 | +0.93 | 89.34 | 90.27 | +0.93
Broad Twitter | 61.8 | 80.27 | 84.56 | +4.29 | 81.25 | 83.74 | +2.49
CoNLL2003 | 52.5 | 91.53 | 93.28 | +1.75 | 93.30 | 93.60 | +0.30
FabNER | 15.3 | 78.38 | 83.20 | +4.82 | 81.87 | 85.39 | +3.52
FindVehicle | 10.5 | 87.56 | 97.37 | +9.81 | 98.30 | 98.62 | +0.32
HarveyNER | 11.6 | 74.69 | 76.33 | +1.64 | 74.21 | 74.73 | +0.52
Movie | 5.3 | 89.58 | 89.28 | -0.30 | 90.17 | 90.23 | +0.06
Restaurant | 32.8 | 82.59 | 83.84 | +1.25 | 82.35 | 81.73 | -0.62
MultiNERD | 58.1 | 90.26 | 94.35 | +4.09 | 93.73 | 94.30 | +0.57
ncbi | 42.1 | 86.21 | 87.27 | +1.06 | 86.96 | 89.27 | +2.31
Ontonotes | 29.7 | 88.64 | 91.83 | +3.19 | 89.91 | 90.69 | +0.78
PolyglotNER | 33.6 | 53.31 | 66.90 | +13.59 | 65.67 | 67.52 | +1.85
TweetNER7 | 40.1 | 65.95 | 67.97 | +2.02 | 65.77 | 66.87 | +1.10
WikiANN | 52.0 | 64.47 | 85.19 | +20.72 | 84.91 | 86.87 | +1.96
wikiNeural | 57.7 | 88.27 | 93.71 | +5.44 | 93.28 | 93.71 | +0.43
Avg | 34.9 | 81.53 | 86.15 | +4.62 | 85.07 | 86.09 | +1.02
Table 3: Supervised evaluation results. $\Delta$ indicates the improvement over the corresponding baseline. Results for InstructUIE and UniNER are derived from Wang et al. (2023) and Zhou et al. (2023), respectively. Model | #Params. | 0-shot | Sup. | Instance/s
---|---|---|---|---
InstructUIE | 11B | 47.3 | 81.53 | 3.4
UniNER-7B | 7B | 53.4 | 85.07 | 1.6
GNER-T5-small | 77M | 48.2 | 77.43 | 32.5
GNER-T5-base | 248M | 59.5 | 83.21 | 20.2
GNER-T5-large | 783M | 63.5 | 85.45 | 11.5
GNER-T5-xl | 3B | 66.1 | 85.94 | 4.6
GNER-T5-xxl | 11B | 69.1 | 86.15 | 3.0
GNER-LLaMA | 7B | 66.1 | 86.09 | 4.0
Table 4: Model’s performance and inference speed in zero-shot and supervised
settings. The inference speed is tested in a single A100 node with batch size
4 per device. More details are outlined in Appendix E.

###  5.2 Zero-shot Evaluation

We evaluate the zero-shot performance of our models after the domain
adaptation phrase. Table 2 summarizes the results. Our model demonstrates
significant improvements compared to other models. Significantly, although our
GNER-LLaMA model shares the same backbone model (LLaMA-7B) and dataset (Pile-
NER) with UniNER Zhou et al. (2023), it demonstrates a notable improvement.
Our results show that our 7B model outperforms the UniNER model of the same
scale by approximately 12.7 $F_{1}$ score points on average, and exhibits
improvements across every dataset. Remarkably, our 7B model surpasses the
UniNER 13B model by 10.5 points. When considering smaller backbone models such
as GNER-T5-base and GNER-T5-large, it’s noteworthy that they also outperform
all the aforementioned strong baselines.

###  5.3 Supervised Evaluation

To test our model’s performance on supervised data, we conduct supervised
multi-task fine-tuning based on the NER-specialized model. The results are
summarized in Table 3 and 4. We first compare our approach with two closely
related baselines, InstructUIE and UniNER, as we share the same backbone model
and train with similar data. As a result, our method demonstrates significant
improvements over these baselines: GNER-T5 achieves a 4.6-point increase in
the $F_{1}$ score, while GNER-LLaMA shows a 1-point $F_{1}$ score improvement.
Moreover, we observe consistent enhancements across almost all datasets. We
also experiment with smaller models, considering both effectiveness and
inference efficiency. As shown in Table 4, our GNER-T5-large model, with only
10% the parameter size of UniNER, achieves superior performance and boasts
10$\times$ the inference efficiency.

Method | GNER-T5-large | GNER-LLaMA
---|---|---
0-shot | Sup. | 0-shot | Sup.
Ours | 63.47 | 85.45 | 66.07 | 86.09
w/o BT | 63.16 | 85.09 | 66.07 | 86.09
w/o LCS+BT | 62.31 | 84.91 | 65.77 | 85.99
Table 5: Ablation study of Hierarchical Matching.

###  5.4 Ablation Results

We have demonstrated the effectiveness of negative instances in our
preliminary study. In this part, we conduct the ablation study in Table 5 to
evaluate the performance of our Hierarchical Matching Algorithm. Removing Back
Tokenization and the LCS algorithm results in a decrease in effectiveness,
underscoring the efficacy of our Hierarchical Matching algorithm.

##  6 Analysis

Beam size | 1 | 2 | 3 | 4
---|---|---|---|---
UniNER-7B | 53.46 | 52.87 | - | -
GNER-T5-base | 59.46 | 60.32 | 60.40 | 60.44
GNER-T5-large | 63.47 | 64.13 | 64.27 | 64.31
GNER-T5-xl | 66.12 | 66.81 | 66.86 | 66.88
GNER-T5-xxl | 69.06 | 69.20 | 69.33 | 69.33
LLaMA-7B | 66.07 | 66.87 | 67.00 | 67.08
Table 6: Zero-shot performance of UniNER and our model GNER via beam search. A
Self-correction Example with beam size 2 Token inputs: What was the fog rated
? Ground Truth: What(O) was(O) the(B-title) fog(I-title) rated(O) ?(O) Medium
prediction results highest beam score: What(O) was(O) the(O) fog(O) second-
highest beam score: What(O) was(O) the(B-title) fog(I-title) Final prediction
results: What(O) was(O) the(B-title) fog(I-title) rated(O) ?(O) Figure 6: An
example of the self-correction mechanism when using beam search.

#### Scaling Law of Generative NER Models

Our experiments show that even smaller models like Flan-T5-large possess
significant potential. We investigate the scaling law of Generative NER tasks
in both zero-shot and supervised settings. The results are illustrated in Fig.
7. In the zero-shot setting, our methods scale well with model size. As the
model size increases, the zero-shot capability of the model continues to rise,
showing ample potential for further improvement with even larger models. In
the supervised setting, our 783M model already demonstrates strong multi-task
generalization abilities, and as the model size increases further, the
improvements tend to converge.

Figure 7: Scaling behavior of zero-shot and supervised performance with
respect to model size (# parameters).

#### Self-Correction Mechanism via Beam Search

Beam search can enhance the performance of generative models by expanding the
search space to include multiple hypotheses at each generation step. Previous
research Yan et al. (2021) has demonstrated that applying beam search in an
entity-centric generation does not improve the model’s performance or even
degrade it. We conduct experiments on UniNER and our model, the results of
which are shown in Table 6. We discover that as the beam size increases, the
performance of the UniNER model decreases. In contrast, we observe a
consistent improvement with beam search under our task schema. Upon a detailed
case study of the model’s generated results, we found that our task schema
possesses a self-correction mechanism. The model retains some other hypotheses
while generating subsequent results. In the decoding process that follows, the
model can correct earlier mistakes. As demonstrated in Fig. 6, the model
revises its previous incorrect prediction of “O” for “the fog” upon
encountering the subsequent token “rated”. This token is crucial for
identifying the entity type associated with “the fog”. A detailed analysis is
provided in Appendix D.

##  7 Conclusion

This paper explores the potential of a strong Generative Named Entity
Recognition (NER) system based on pre-trained LLMs by integrating the negative
instances into training. Through experiments, we have demonstrated significant
advancements. Our approach, which combines the inclusion of contextual
information and a clear definition of entity boundaries through negative
instances, has proven to be highly effective in improving the model’s
performance, especially in zero-shot scenarios where prediction uncertainty is
high. The introduction of a Hierarchical Matching algorithm further addresses
the challenges of converting unstructured text into structured entities,
ensuring accurate categorization and alignment. These findings highlight the
crucial role of negative instances in NER tasks and the potential of
generative models to revolutionize the field.

##  8 Limitation

Despite our system achieving impressive results, there remain limitations and
space for improvement. In task settings, our approach focuses on the main-
stream Flat-NER settings, where entities appear as continuous text segments,
without addressing the discontinuous forms, i.e., discontinuous NER. Actually,
it has always been challenging for generative models to adopt a unified
paradigm to resolve all the complex settings. Previous entity-centric methods
can address the discontinuous settings but fail to manage polysemy, where a
phrase corresponds to different entity types in different sentence parts. The
primary focus of this paper is to explore the impact of negative instances in
the training process, and we will explore a unified framework for generative
models in future work.

## Ethics Statement

In this paper, we utilize the pre-trained large language models, i.e., Flan-T5
and LLaMA, as the foundational models. It’s important to acknowledge that
these models may contain inherent biases resulting from their pre-training
processes. However, this issue is mitigated through our fine-tuning process,
which refines the models to specifically concentrate on the Named Entity
Recognition (NER) task, thereby reducing potential biases. Moreover, we
strictly use all datasets and corpora in our study for scientific research
purposes only.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018.  Contextual string embeddings for sequence labeling.  In _Proceedings of the 27th international conference on computational linguistics_ , pages 1638–1649.
  * Al-Rfou et al. (2015) Rami Al-Rfou, Vivek Kulkarni, Bryan Perozzi, and Steven Skiena. 2015.  Polyglot-ner: Massive multilingual named entity recognition.  In _Proceedings of the 2015 SIAM International Conference on Data Mining_ , pages 586–594. SIAM.
  * Chen et al. (2022) Pei Chen, Haotian Xu, Cheng Zhang, and Ruihong Huang. 2022.  Crossroads, buildings and neighborhoods: A dataset for fine-grained location recognition.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3329–3339, Seattle, United States. Association for Computational Linguistics.
  * Chiu and Nichols (2016) Jason PC Chiu and Eric Nichols. 2016.  Named entity recognition with bidirectional lstm-cnns.  _Transactions of the Association for Computational Linguistics_ , 4:357–370.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Derczynski et al. (2016) Leon Derczynski, Kalina Bontcheva, and Ian Roberts. 2016.  Broad Twitter corpus: A diverse named entity recognition resource.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 1169–1179, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Doğan et al. (2014) Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014.  Ncbi disease corpus: a resource for disease name recognition and concept normalization.  _Journal of biomedical informatics_ , 47:1–10.
  * Fu et al. (2021) Jinlan Fu, Xuan-Jing Huang, and Pengfei Liu. 2021.  Spanner: Named entity re-/recognition as span prediction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7183–7195.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Guan et al. (2023) Runwei Guan, Ka Lok Man, Feifan Chen, Shanliang Yao, Rongsheng Hu, Xiaohui Zhu, Jeremy Smith, Eng Gee Lim, and Yutao Yue. 2023.  Findvehicle and vehiclefinder: A ner dataset for natural language-based vehicle retrieval and a keyword-based cross-modal vehicle retrieval system.  _arXiv preprint arXiv:2304.10893_.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional lstm-crf models for sequence tagging.  _arXiv preprint arXiv:1508.01991_.
  * Hunt and Szymanski (1977) James W Hunt and Thomas G Szymanski. 1977.  A fast algorithm for computing longest common subsequences.  _Communications of the ACM_ , 20(5):350–353.
  * Krallinger et al. (2015) Martin Krallinger, Obdulia Rabal, Florian Leitner, Miguel Vazquez, David Salgado, Zhiyong Lu, Robert Leaman, Yanan Lu, Donghong Ji, Daniel M Lowe, et al. 2015.  The chemdner corpus of chemicals and drugs and its annotation principles.  _Journal of cheminformatics_ , 7(1):1–17.
  * Kumar and Starly (2022) Aman Kumar and Binil Starly. 2022.  “fabner”: information extraction from manufacturing process science domain literature using named entity recognition.  _Journal of Intelligent Manufacturing_ , 33(8):2393–2407.
  * Li et al. (2023) Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023.  Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness.  _arXiv preprint arXiv:2304.11633_.
  * Li et al. (2016) Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016.  Biocreative v cdr task corpus: a resource for chemical disease relation extraction.  _Database_ , 2016.
  * Li et al. (2022a) Jingye Li, Hao Fei, Jiang Liu, Shengqiong Wu, Meishan Zhang, Chong Teng, Donghong Ji, and Fei Li. 2022a.  Unified named entity recognition as word-word relation classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 10965–10973.
  * Li et al. (2020a) Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020a.  A unified mrc framework for named entity recognition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5849–5859.
  * Li et al. (2022b) Yangming Li, Lemao Liu, and Shuming Shi. 2022b.  Rethinking negative sampling for handling missing entity annotations.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7188–7197.
  * Li et al. (2020b) Yangming Li, Shuming Shi, et al. 2020b.  Empirical analysis of unlabeled entity problem in named entity recognition.  In _International Conference on Learning Representations_.
  * Liu et al. (2013) Jingjing Liu, Panupong Pasupat, Scott Cyphers, and Jim Glass. 2013.  Asgard: A portable architecture for multilingual dialogue systems.  In _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pages 8386–8390. IEEE.
  * Liu et al. (2021) Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2021.  Crossner: Evaluating cross-domain named entity recognition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 13452–13460.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023.  The flan collection: Designing data and methods for effective instruction tuning.  _arXiv preprint arXiv:2301.13688_.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Mengge et al. (2020) Xue Mengge, Bowen Yu, Zhenyu Zhang, Tingwen Liu, Yue Zhang, and Bin Wang. 2020.  Coarse-to-fine pre-training for named entity recognition.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6345–6354.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pan et al. (2017) Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.  Cross-lingual name tagging and linking for 282 languages.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
  * Pyysalo and Ananiadou (2014) Sampo Pyysalo and Sophia Ananiadou. 2014.  Anatomical entity mention recognition at literature scale.  _Bioinformatics_ , 30(6):868–875.
  * Qin et al. (2019) Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. 2019.  A stack-propagation framework with token-level intent detection for spoken language understanding.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2078–2087.
  * Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_.
  * Sainz et al. (2023) Oscar Sainz, Iker García-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. 2023.  Gollie: Annotation guidelines improve zero-shot information-extraction.  _arXiv preprint arXiv:2310.03668_.
  * Sang and De Meulder (2003) Erik Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Smith et al. (2008) Larry Smith, Lorraine K Tanabe, Cheng-Ju Kuo, I Chung, Chun-Nan Hsu, Yu-Shi Lin, Roman Klinger, Christoph M Friedrich, Kuzman Ganchev, Manabu Torii, et al. 2008.  Overview of biocreative ii gene mention recognition.  _Genome biology_ , 9(2):1–19.
  * Tedeschi et al. (2021) Simone Tedeschi, Valentino Maiorca, Niccolò Campolungo, Francesco Cecconi, and Roberto Navigli. 2021.  WikiNEuRal: Combined neural and knowledge-based silver data creation for multilingual NER.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2521–2533, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tedeschi and Navigli (2022) Simone Tedeschi and Roberto Navigli. 2022.  MultiNERD: A multilingual, multi-genre and fine-grained dataset for named entity recognition (and disambiguation).  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 801–812, Seattle, United States. Association for Computational Linguistics.
  * Ushio et al. (2022) Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco. Barbieri, and Jose Camacho-Collados. 2022.  Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts.  In _The 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing_ , Online. Association for Computational Linguistics.
  * Wang et al. (2023) Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, et al. 2023.  Instructuie: Multi-task instruction tuning for unified information extraction.  _arXiv preprint arXiv:2304.08085_.
  * Wang et al. (2019) Zihan Wang, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, and Jiawei Han. 2019.  Crossweigh: Training named entity tagger from imperfect annotations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5157–5166.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Wei et al. (2023) Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023.  Zero-shot information extraction via chatting with chatgpt.  _arXiv preprint arXiv:2302.10205_.
  * Weischedel et al. (2013) Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013.  Ontonotes release 5.0 ldc2013t19.  _Linguistic Data Consortium, Philadelphia, PA_ , 23:170.
  * Wu et al. (2022) Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. 2022.  Sustainable ai: Environmental implications, challenges and opportunities.  _Proceedings of Machine Learning and Systems_ , 4:795–813.
  * Yan et al. (2021) Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021.  A unified generative framework for various ner subtasks.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5808–5822.
  * Yu et al. (2020) Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020.  Named entity recognition as dependency parsing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6470–6476.
  * Zaratiana et al. (2023) Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. 2023.  Gliner: Generalist model for named entity recognition using bidirectional transformer.  _arXiv preprint arXiv:2311.08526_.
  * Zhou et al. (2023) Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2023.  Universalner: Targeted distillation from large language models for open named entity recognition.  _arXiv preprint arXiv:2308.03279_.

##  Appendix A Data Statistics and Pre-processing

We show the full dataset statistics in Table 7, including the domain of
datasets, the number of instances in train/valid/test data, and their download
address. In particular, we have to pre-process the Pile-NER Zhou et al. (2023)
dataset to fit in our task schema. We observe nuances between our compiled
datasets and those referenced by Wang et al. (2023) and Zhou et al. (2023).
Specifically, their MultiNERD and PolyglotNER datasets omit the last 10,000
training samples. Furthermore, they miss the last sample in some datasets for
the validation and test sets, such as CrossNER politics, MIT Movie, and MIT
Restaurant. We have included these omitted instances in our dataset, adhering
to the original dataset compositions. The modifications have a negligible
impact on our results. This is because our sampling approach aligns with those
used in the referenced studies, ensuring that the number of data instances
sampled from each training set, up to 10,000 samples, is consistent. Moreover,
adding a single extra sample in the test sets hardly affects the final
results.

Dataset | Domain | Types | #Train | #Valid | #Test | Download Link
---|---|---|---|---|---|---
Pile-NER Zhou et al. (2023) | General | 13,020 | 45,889 | 0 | 0 | link
CoNLL2003 Sang and De Meulder (2003) | 4 | 14,041 | 3,250 | 3,453 | link
conllpp Wang et al. (2019) | 4 | 14,041 | 3,250 | 3,453 | link
CrossNER AI Liu et al. (2021) | 13 | 100 | 350 | 431 | link
CrossNER literature Liu et al. (2021) | 11 | 100 | 400 | 416 | link
CrossNER music Liu et al. (2021) | 12 | 100 | 380 | 465 | link
CrossNER politics Liu et al. (2021) | 8 | 200 | 541 | 651 | link
CrossNER science Liu et al. (2021) | 16 | 200 | 450 | 543 | link
MultiNERD Tedeschi and Navigli (2022) | 16 | 144,144 | 10,000 | 10,000 | link
Ontonotes Weischedel et al. (2013) | 18 | 59,924 | 8,528 | 8,262 | link
PolyglotNER Al-Rfou et al. (2015) | 3 | 403,982 | 10,000 | 10,000 | link
WikiANN en Pan et al. (2017) | 3 | 20,000 | 10,000 | 10,000 | link
WikiNeural Tedeschi et al. (2021) |  | 3 | 92,720 | 11,590 | 11,597 | link
AnatEM Pyysalo and Ananiadou (2014) | Biomed | 1 | 5,861 | 2,118 | 3,830 | link
bc2gm Smith et al. (2008) | 1 | 12,500 | 2,500 | 5,000 | link
bc4chemd Krallinger et al. (2015) | 1 | 30,682 | 30,639 | 26,364 | link
bc5cdr Li et al. (2016) | 2 | 4,560 | 4,581 | 4,797 | link
ncbi Doğan et al. (2014) | 1 | 5,432 | 923 | 940 | link
HarveyNER Chen et al. (2022) | Social media | 4 | 3,967 | 1,301 | 1,303 | link
Broad Tweet Corpus Derczynski et al. (2016) | 3 | 6,338 | 1,001 | 2,001 | link
TweetNER7 Ushio et al. (2022) | 7 | 7,111 | 886 | 576 | link
mit-movie Liu et al. (2013) | 12 | 9,775 | 2,443 | 2,443 | link
mit-restaurant Liu et al. (2013) | 8 | 7,660 | 1,521 | 1,521 | link
FabNER Kumar and Starly (2022) | STEM | 12 | 9,435 | 2,183 | 2,064 | link
FindVehicle Guan et al. (2023) | Transportation | 21 | 21,565 | 20,777 | 20,777 | link
Table 7: Statistics of datasets in our collected datasets.

##  Appendix B Hyper-parameters settings

Model | Issue | Case | Type |  Prediction
---|---|---|---|---
GNER LLaMA |  Omission (39%) | 1 | raw |  who directed the film the lorax
pred. |  who directed the lorax
2 | raw |  any reasonably priced indian restaurants in the theater district
pred. |  any reasonably priced indian restaurants in theater district
Addition (3%) | 3 | raw |  the conservative regionalist navarra suma finished first and …
pred. |  the conservative regionalist regionalist navarra suma finished first and …
Substitution (58%) | 4 | raw |  which five star italian restaurants in manattan have the best reviews
pred. |  which five star italian restaurants in manhattan have the best reviews
5 | raw |  polyethylene terephthalate ( pet ) bottles are made from ethylene and p-xylene .
pred. |  polyethylene terephthalate ( p e t) bottles are made from ethylene and p-xylene .
GNER T5 |  Omission (23%) | 6 | raw |  … whose debut album tol cormpt norz norz norz rock hard journalist wolf-rüdiger mühlmann considers a part of war metal ’s roots .
pred. |  … whose debut album tol cormpt norz norz rock hard journalist wolf-rüdiger mühlmann considers a part of war metal ’s roots .
7 | raw |  jennifer lien starred in this action film of the the last six years that received a really good rating
pred. |  jennifer lien starred in this action film of the last six years that received a really good rating
Addition (2%) | 8 | raw |  … performed by wet wet wet that remained at number 1 …
pred. |  … performed by wet wet wet wet that remained at number 1 …
9 | raw |  … liked by many people that starred william forsythe
pred. |  … liked by many people that starred william forsythe the
Substitution (75%) | 10 | raw |  four more children followed : charlotte brontë , ( 1816-1855 ) , branwell brontë ( 1817-1848 ) , emily brontë , ( 1818-1848 ) and anne ( 1820-1849 ) .
pred. |  four more children followed : charlotte bront , ( 1816-1855 ) , branwell bront ( 1817-1848 ) , emily bront , ( 1818-1848 ) and anne ( 1820-1849 ) .
Table 8: Representative examples concerning the word addition, omission, and
substitution problems in the zero-shot evaluation. We remove the label
information in the predictions for a clear comparison with the raw texts.

In our experiments, we train all models using a batch size of 256, employing
the AdamW optimizer Loshchilov and Hutter (2018) for optimization. For the T5
model, we set a constant learning rate of $5\times 10^{-5}$ and impose a
length limitation of 640 tokens for both the encoder and decoder. For the
LLaMA model, we adopt a cosine learning rate schedule, initiating with a warm-
up phase that covers 4% of the training steps, ramping up to a learning rate
of $2\times 10^{-5}$, followed by a decay phase for the remainder of the
training steps. The length limitation is set to 1280. Due to our prediction
sequences being longer, more training steps are required. The number of
training epochs for our models varies by size: 20 epochs for both the small
and base models, 10 epochs for the large and xl models, and 6 epochs for the
xxl model. For the LLaMA model, we set the number of epochs to 3. We observe
an interesting phenomenon that the T5 model often requires more training steps
to converge. A possible explanation is that the backbone model, Flan-T5, an
instruction-tuned model without any Named Entity Recognition (NER) related
data in the instruction-tuning process, requires more training steps to adapt
to the NER task.

##  Appendix C Problems in long sequence

In response to the issues in long predictions mentioned in section 3.4, we
conduct a detailed case study. The representative examples are presented in
Table 8. The problems can primarily be categorized into word omission,
addition, and substitution, with omission and substitution accounting for the
majority. We can conclude the following causes:

#### Noise in the original text

Some of the issues can be attributed to noise in the original text. For
example, in case 4, the model corrects “manattan” to “manhattan”, and in case
7, it corrects the misuse of “the”. However, we also observe that the model
can introduce errors, as seen in cases 6 and 8, where the entities with
repeated words, “norz norz norz” and “wet wet wet”, confuse the model.

#### Missing words in the vocabulary

Furthermore, we find that a certain proportion of issues can be derived from
missing words in the model’s vocabulary. As a result, these words naturally do
not appear in the model’s output. For instance, in case 10, “brontë” was
replaced with “bront” because “brontë” does not exist in the vocabulary. We
also discovered that several special characters do not exist in the T5
vocabulary, leading to more occurrences of omission and substitution.

#### Accumulative exposure bias

The issue of repetitive generation of words and phrases is common in long text
generation (LTG) due to the accumulative exposure bias as the prediction
length increases. As illustrated by cases 3 and 9, the model produces
meaningless and repetitive information.

##  Appendix D Self-correction Mechanism

In this section, we conduct a case study to explore (1) the reasons behind the
reduced effectiveness of entity-centric methods like UniNER Zhou et al. (2023)
when beam search is applied, and (2) the specific enhancements of the self-
correction mechanism in our task schema. Upon comparing UniNER’s performance
with and without beam search, we observe that beam search leads to the model
responding with the same answers across a variety of entity-type queries. For
our models, we provide representative examples in Table 9 to illustrate the
self-correction mechanism’s impact, showcasing (1) enhanced precision in
determining entity boundaries (cases 1 and 2), (2) the use of contextual clues
to recognize inherent entities (cases 3, 6 and 7), and correct mistakes (cases
4 and 5).

##  Appendix E Detailed Evaluation Results

We detail the performance of our models across all datasets in Table 10,
including error bars for zero-shot performance derived from the variance of
five separate runs. For the supervised settings, we do not conduct multiple
runs due to the extensive size of the datasets, where the training and
inference process can be very time-consuming. Our trials with smaller models
indicate that the variability, or error bars, for models in the supervised
settings is minimal, approximately around 0.15.

Model | Case | Type |  Text Generations
---|---|---|---
GNER LLaMA | 1 | w/o beam search |  who(O) is(O) directing(O) the(O) hobbit(B-title)
w/ beam search |  who(O) is(O) directing(O) the(B-title) hobbit(I-title)
2 | w/o beam search |  what(O) is(O) the(O) plot(O) of(O) the(O) wild(B-title) bunch(I-title)
w/ beam search |  what(O) is(O) the(O) plot(O) of(O) the(B-title) wild(I-title) bunch(I-title)
3 | w/o beam search |  was(O) there(O) a(O) romantic(O) film(O) noir(O)
w/ beam search |  was(O) there(O) a(O) romantic(B-genre) film(I-genre) noir(I-genre)
4 | w/o beam search |  does(O) paymon(B-Restaurant Name) serves(O) white(B-Cuisine) wine(I-Cuisine)
w/ beam search |  does(O) paymon(B-Restaurant Name) serves(O) white(B-Dish) wine(I-Dish)
GNER T5 | 5 | w/o beam search |  … some(O) batman(B-character) movies(O) from(O) the(O) 1990s(B-year)
w/ beam search |  … some(O) batman(B-title) movies(I-title) from(O) the(O) 1990s(B-year)
6 | w/o beam search |  where(O) was(O) the(O) presidio(B-title) filmed(O)
w/ beam search |  where(O) was(O) the(B-title) presidio(I-title) filmed(O)
7 | w/o beam search |  … the(O) third(O) harry(O) potter(O) movie(O) called(O)
w/ beam search |  … the(O) third(B-title) harry(I-title) potter(I-title) movie(I-title) called(O)
Table 9: Representative examples in the self-correction mechanism via beam
search.

##  Appendix F Environmental Impact

Training huge models can have a negative impact on the environment. All our
models are trained on the hardware of a single A100 node (8$\times$
Nvidia-A100-80G-SXM4) with approximately 800 GPU hours in total. The carbon
footprint estimation is 135.3 kg $\text{CO}_{2}$eq according to Wu et al.
(2022).

Method | GNER-T5 | GNER-T5 | GNER-T5 | GNER-T5 | GNER-T5 | GNER-LLaMA
---|---|---|---|---|---|---
Backbone | Flan-T5-small | Flan-T5-base | Flan-T5-large | Flan-T5-xl | Flan-T5-xxl | LLaMA-7B
# Params. | 77M | 248M | 783M | 3B | 11B | 7B
Zero-shot Performance
AI | 50.18$\pm$0.9 | 56.83$\pm$0.4 | 62.56$\pm$0.2 | 62.09$\pm$0.3 | 68.19$\pm$0.3 | 63.11$\pm$0.2
Literature | 49.78$\pm$1.5 | 58.68$\pm$0.8 | 58.20$\pm$0.4 | 64.94$\pm$1.1 | 68.66$\pm$0.2 | 68.20$\pm$0.3
Music | 65.83$\pm$1.3 | 72.29$\pm$0.3 | 76.73$\pm$0.7 | 80.59$\pm$0.6 | 81.24$\pm$0.4 | 75.72$\pm$0.8
Politics | 57.28$\pm$1.1 | 64.50$\pm$1.1 | 66.99$\pm$0.8 | 73.73$\pm$0.6 | 75.11$\pm$0.9 | 69.38$\pm$1.2
Science | 62.68$\pm$1.9 | 68.00$\pm$1.2 | 72.60$\pm$0.2 | 68.74$\pm$1.2 | 76.70$\pm$1.0 | 69.93$\pm$0.4
Movie | 37.38$\pm$1.8 | 54.52$\pm$0.2 | 58.59$\pm$0.1 | 62.96$\pm$0.4 | 62.52$\pm$0.5 | 68.63$\pm$0.5
Restaurant | 14.30$\pm$1.4 | 41.41$\pm$1.2 | 48.61$\pm$0.5 | 49.82$\pm$0.2 | 51.04$\pm$0.4 | 47.49$\pm$1.1
Avg. | 48.20$\pm$1.1 | 59.46$\pm$0.8 | 63.47$\pm$0.2 | 66.12$\pm$0.2 | 69.06$\pm$0.3 | 66.07$\pm$0.3
Supervised Performance
AnatEM | 81.02 | 86.99 | 90.22 | 90.29 | 90.30 | 90.24
bc2gm | 69.02 | 79.11 | 83.10 | 84.25 | 84.29 | 83.18
bc4chemd | 76.33 | 85.19 | 88.51 | 90.22 | 90.04 | 89.40
bc5cdr | 82.02 | 87.16 | 88.81 | 89.83 | 89.95 | 90.27
Broad Twitter | 80.09 | 81.59 | 82.61 | 84.34 | 84.56 | 83.74
CoNLL2003 | 89.12 | 91.82 | 93.14 | 93.14 | 93.28 | 93.60
FabNER | 68.20 | 77.34 | 81.89 | 81.54 | 83.20 | 85.39
FindVehicle | 90.64 | 93.61 | 95.71 | 95.97 | 97.37 | 98.62
HarveyNER | 60.27 | 70.77 | 75.24 | 74.00 | 76.33 | 74.73
Movie | 85.03 | 88.08 | 89.39 | 89.31 | 89.28 | 90.23
Restaurant | 78.98 | 82.21 | 83.72 | 83.06 | 83.84 | 81.73
MultiNERD | 90.94 | 93.17 | 94.24 | 94.51 | 94.35 | 94.30
ncbi | 82.06 | 87.14 | 88.46 | 89.58 | 88.27 | 88.55
Ontonotes | 86.36 | 89.33 | 90.54 | 91.63 | 91.83 | 90.69
PolyglotNER | 45.27 | 62.13 | 66.16 | 67.15 | 66.90 | 67.52
TweetNER7 | 62.92 | 67.36 | 67.50 | 68.07 | 67.97 | 66.87
WikiANN | 76.58 | 82.56 | 85.32 | 86.09 | 85.19 | 86.87
wikiNeural | 88.97 | 92.24 | 93.56 | 93.85 | 93.71 | 93.71
Avg. | 77.43 | 83.21 | 85.45 | 85.94 | 86.15 | 86.09
Table 10: Zero-shot and supervised evaluation results.