# PromptNER : Prompting For Named Entity Recognition

Dhananjay Ashok
Carnegie Mellon University
<EMAIL_ADDRESS>
Zachary C. Lipton
Carnegie Mellon University
<EMAIL_ADDRESS>

###### Abstract

In a surprising turn, Large Language Models (LLMs) together with a growing
arsenal of prompt-based heuristics now offer powerful off-the-shelf approaches
providing few-shot solutions to myriad classic NLP problems. However, despite
promising early results, these LLM-based few-shot methods remain far from the
state of the art in Named Entity Recognition (NER), where prevailing methods
include learning representations via end-to-end structural understanding and
fine-tuning on standard labeled corpora. In this paper, we introduce PromptNER
, a new state-of-the-art algorithm for few-Shot and cross-domain NER. To adapt
to any new NER task PromptNER requires _a set of entity definitions_ in
addition to the standard few-shot examples. Given a sentence, PromptNER
prompts an LLM to produce a list of potential entities along with
corresponding explanations justifying their compatibility with the provided
entity type definitions. PromptNER achieves state-of-the-art performance on
few-shot NER, achieving a 4% (absolute) improvement in F1 score on the ConLL
dataset, a 9% (absolute) improvement on the GENIA dataset, and a 4% (absolute)
improvement on the FewNERD dataset. PromptNER also moves the state of the art
on Cross Domain NER, outperforming prior methods (including those not limited
to the few-shot setting), setting a new mark on $3/5$ CrossNER target domains,
with an average F1 gain of $3\%$, despite using less than $2\%$ of the
available data.

##  1 Introduction

Named Entity Recognition (Chinchor, 1995) is often a vital component in text
processing pipelines for information extraction and semantic understanding
(Sharma et al., 2022; Ali et al., 2022). Current methods perform well when
training data is plentiful (Wang et al., 2022b; Yu et al., 2020; Li et al.,
2022; Wang et al., 2020). However, their applicability to many real-world
problems is hindered by their reliance on fixed entity definitions and large
amounts of in-domain training data for the specific NER formulation and
population of interest. Unfortunately, commitments about what constitute the
relevant entities vary wildly across use cases, a fact that is reflected in
the diversity of academic datasets (contrast, e.g., medical NER datasets with
CoNLL or OntoNotes). Ultimately, these differing commitments stem from
differences in the envisioned use cases. Should we categorize the phrase
‘Theory of General Relativity’ as an entity? A media company tasked with
extracting information from political articles might not designate physical
laws as a relevant class of entities but a scientific journal might. Given the
diversity of use cases and underlying documents that characterize different
deployment settings, we might hope ideally for a system to adapt to new
settings flexibly, requiring minimal labeled data, human effort, and
computational cost.

With the emergence of LLMs, the NLP community has developed a repertoire of
in-context learning strategies that have rapidly advanced the state of few-
shot learning for myriad tasks (Brown et al., 2020; Wei et al., 2022; Liu et
al., 2023). However, such prompting-based approaches have yet to show
comparable impact in NER, where current methods typically cast few-shot
learning as a domain transfer problem, training on large amounts of source
data and fine-tuning on exemplars from the target domain (Huang et al., 2022a;
Yang et al., 2022). Moreover, a significant gap remains between the best few-
shot NER methods and the performance of end-to-end trained models (Wang et
al., 2022b; Xu et al., 2022). These few-shot methods struggle when the source
and target domains differ with respect to what constitutes an entity (Yang et
al., 2022; Das et al., 2022). A separate class of adaptation methods have
shown promise when the source and target vary considerably, but they tend to
require hundreds of data points to be effective (Hu et al., 2022a; Chen et
al., 2023; Hu et al., 2022b; Chen et al., 2022).

In this paper, we introduce PromptNER , a prompting-based NER method that
achieves state-of-the-art results on FewShot NER and CrossDomain NER. Our
method consists of 4 key components—a backbone LLM, a modular definition (a
document defining the set of entity types), a few examples from the target
domain, and a precise format for outputing the extracted entities, which is
communicated to the model via the formatting of the few-shot examples. To
adapt to a new domain, our method requires modifying only the definition and
the provided examples. This makes the method flexible and easy to apply across
domains. PromptNER achieves 83.48% F1 score on the CoNLL dataset (Sang and De
Meulder, 2003) in a few-shot setting, improving over the best previous few-
shot methods by 4% (absolute). PromptNER outperforms the best-competing
methods by 9% (absolute) on the GENIA (Kim et al., 2003) dataset and 4%
(absolute) on the FewNERD-Intra (Ding et al., 2021) setting and sets a new
state of the art on three out of five of the CrossNER (Liu et al., 2021)
target domains, despite using only $2\%$ of the available training data. In
ablations, we show that PromptNER outperforms standard Few-Shot Prompting
(Brown et al., 2020) and Chain-of-Thought Prompting (Wei et al., 2022).

Figure 1: Example of prompt to pre-trained language model. Definition is in
blue

, question and task in red, example answer and chain of thought format in
green

##  2 Background and Related Works

Named Entity Recognition has been well studied since the formulation of the
problem in the mid-90s (Chinchor, 1995), with early methods including rule-
based systems (Eftimov et al., 2017; Farmakiotou et al., 2000; Mikheev et al.,
1999), statistical learning based methods (Borthwick et al., 1998; Borthwick,
1999; Zhou and Su, 2002), and the use of traditional machine learning methods
like SVMs (Makino et al., 2002; Mayfield et al., 2003). With the shift towards
deep learning, RNN-based methods and then transformer-based methods came to
dominate on most NER tasks (De Mulder et al., 2015; Huang et al., 2015;
Limsopatham and Collier, 2016). Most recently, methods leveraging large pre-
trained Transformers have advanced the state-of-the-art further (Lu et al.,
2022b; Wang et al., 2022b; Yu et al., 2020; Tong et al., 2022; Lu et al.,
2022a; Li et al., 2022).

Recent methods include DeepStruct (Wang et al., 2022b), which modify the
typical language modelling procedure to make the language model more aware of
logical structure in the corpus, and then use this trained model for
downstream NER. Other methods introduce specialized architectures with
features of the NER problem in mind (Yu et al., 2020; Li et al., 2022). These
methods typically require full dataset access and require significant
computational resources for training. Motivated by settings where training on
full datasets is not possible or practical, some researchers have turned their
attention to few-shot NER (Church et al., 2021; Das et al., 2022; Huang et
al., 2022a). Leading few-shot NER methods include approaches which seek to
create a pretrained LLM that can then be adapted for NER (Wang et al., 2022b;
Xu et al., 2022) and approaches which view NER as a metric learning problem
and use prompts to guide the learning process.(Huang et al., 2022b; Chen et
al., 2022). The final class of methods we discuss tackles the cross domain
generalization of NER methods, where the training and testing tasks are from
different domains, these include CP-NER (Chen et al., 2023), which uses
collaborative prefix tuning to learn domain-specific prefixes that can be
swapped flexibly to perform NER; FactMix (Yang et al., 2022), which uses a
model-agnostic data augmentation strategy to improve generalization; and LANER
(Hu et al., 2022a), which tries to improve transferability of learnt label
information. A few methods try to directly apply in context learning (Wei et
al., 2021) for NER, often interpreting the problem as a templated cloze
statement infilling problem (Lee et al., 2021; Cui et al., 2021).

Prompting and Chain-of-Thought Prompting (Brown et al., 2020; Wei et al.,
2022) represent new ways to use LLMs. It has been shown that LLMs can achieve
high performance on Few-Shot tasks with a few examples in the context window
of the input. Chain-of-Thought prompting improves on this approach by
providing examples in the prompt which not only contain question answer pairs,
but also some stated reasoning for the provided answer. Key Innovations:  Our
method, PromptNER , leverages the power of Chain-of-Thought Prompting to
provide an accurate, flexible, and accessible way to perform few-shot NER
without requiring any parameter updates. Our key innovations include the use
of a swappable entity dictionary and a new output format. Together these lead
to significant gains over the previous state of the art in few-shot NER and
cross domain NER.

Method | CoNLL
---|---
COPNERHuang et al. (2022a) | $75.8\pm 2.7$
EntLMMa et al. (2021) | $51.32\pm 7.67$
FactMixYang et al. (2022) | $60.8$
ProMLChen et al. (2022) | $79.16\pm 4.49$
UIELu et al. (2022b) | $67.09$
CONTaiNERDas et al. (2022) | $75.8\pm 2.7$
PMRXu et al. (2022) | $65.7\pm 4.5$
PromptNER T5XXL (Us) | $45.66\pm{12.43}$
PromptNER GPT3.5 (Us) | $78.62\pm 4.62$
PromptNER GPT4 (Us) | $\textbf{83.48}\pm\textbf{5.66}$
Table 1: FewShot Learning ($0<k<5$) on CoNLL dataset. Results show micro-F1 averages and associated standard deviation over 5 runs when available.  Method | k | Politics | Literature | Music | AI | Sciences
---|---|---|---|---|---|---
FactMixYang et al. (2022) | 100 | 44.66 | 28.89 | 23.75 | 32.09 | 34.13
LANERHu et al. (2022a) | 100-200 | 74.06 | 71.11 | 78.78 | 65.79 | 71.83
CPNERChen et al. (2023) | 100-200 | 76.35 | 72.17 | 80.28 | 66.39 | 76.83
EnTDAHu et al. (2022b) | 100 | 72.98 | 68.04 | 76.55 | 62.31 | 72.55
PromptNER T5XXL (Us) | 2 | 39.43 | 36.55 | 41.93 | 30.67 | 46.32
PromptNER GPT3.5 (Us) | 2 | 71.74 | 64.15 | 77.78 | 59.35 | 64.83
PromptNER GPT4 (Us) | 2 | 78.61 | 74.44 | 84.26 | 64.83 | 72.59
Table 2: Cross Domain results on CrossNER dataset with CoNLL as source domain. $k$ is the number of target domain datapoints used by each method. Results show micro-F1 scores. Despite using only $1\%-2\%$ of the data our method achieves state-of-the-art performance on three of the five datasets Method | GENIA
---|---
CONTaiNERDas et al. (2022) | $44.77\pm 1.06$
BCLMing et al. (2022) | $46.06\pm 1.02$
SpanProtoShen et al. (2021) | $41.84\pm 2.66$
PACL | $49.58\pm 1.82$
PromptNER T5XXL (Us) | $25.13\pm 3.22$
PromptNER GPT3.5 (Us) | $\textbf{52.80}\pm\textbf{5.15}$
PromptNER GPT4 (Us) | $\textbf{58.44}\pm\textbf{6.82}$
Table 3: FewShot Learning ($0<k<5$) on GENIA dataset. Results show micro-F1
averages and associated standard deviation over 5 runs when available.

##  3 Method

In NER, we are given a sentence and asked to predict the set of entities that
it contains. We are sometimes given a natural language description of what we
would like to consider an entity, or a list of possible entity types.

Our method consists of 3 key components 1:

Conditional Generation Language Model: we leverage the power of pretrained
LLMs which have been trained on conditional generation tasks. This is a
departure from several works in the field that consider the NER problem as a
sequence to label (or discriminative) problem (Ali et al., 2022; Sharma et
al., 2022), or the prompting-based approaches that consider NER to be a cloze
statement prediction problem (Cui et al., 2021; Lee et al., 2021). We believe
that a sequence-to-sequence formulation of NER has an advantage over these
other formulations because it resolves the combinatorial explosion which
occurs when entities can be more than a single word long (e.g "United
Nations"). Discriminative and in-filling approaches would have to iterate over
all possible combinations of words, which includes either uncomfortable manual
thresholding of the maximum token length of an entity, or requires a search
over all n-grams over the sentence. Additionally, using general Seq-to-Seq
models opens up modeling possibilities, allowing us to pursue strategies based
on chain-of-thought-like reasoning. These possibilities are not readily
available in the more traditional aligned sequence tagging architectures
traditionally used for NER.

Modular Definitions: the implicit commitments about what constitute instances
of the relevant entity types can vary wildly across different NER settings.
The difficulty of capturing these subtleties in just a few examples can hinder
typical few-shot approaches to NER. In our approach to few-shot, NER, each
problem is defined not only by a small set of exemplars (the few-shot
examples) but also by a per-domain definition. Here, the modular definition
consists of a natural language description of what does (and does not)
constitute an entity. This can be useful in instances where the typical
natural language connotation of the word ‘entity’ may include concepts the
specific NER task would want to exclude. Since this document is composed in
natural language, it can easily be composed by an end user with no technical
knowledge.

Potential Entity Output Template: recent work on Chain-of-Thought Prompting
has shown that incentivizing the model to output its answer in a structure
which facilitates reasoning like steps along with the final answer can
significantly boost the performance of large language models (Wang et al.,
2022a; Wei et al., 2022). Motivated by this, we create a template structure
for the output of the LLM which allows it to emulate reasoning to decide
whether a given phrase is classified as an entity or not as well as what type
of entity it is. The exact structure is one where each line of the output
mentions a distinct candidate entity, a decision on whether or not the
candidate should be considered an entity and an explanation for why or why not
along with what entity type it belongs to. In the ablation section, we show
that the inclusion of examples and a clear definition are the most important
parts of this NER system.

Seen as a whole, this pipeline provides flexibility with little cost. The only
components that need to change to adapt the method to new domains are the
definition and examples, both of which have virtually no computational cost
and do not require an entire data collection initiative to gather. There are
many situations where such a flexible solution is preferred over methods that
require fine-tuning of parameters (Ding et al., 2023; Wei et al., 2021), Since
the method makes no assumptions on the architecture or specifics of the
backbone LLM, it can quickly be upgraded if a new generation of more powerful
LLMs were to be released (Liu et al., 2023; Kaplan et al., 2020).

In the following sections we demonstrate the predictive power of this method
across a variety of settings. 111Code available at
https://anonymous.4open.science/r/PromptNER-2BB0/

Method | k | FewNERD
---|---|---
ProMLChen et al. (2022) | 5 | 68.1 $\pm$ 0.35
CONTaiNERDas et al. (2022) | 5 | 47.51
Meta LearningMa et al. (2022) | 5 | 56.8 $\pm$ 0.14
PromptNER T5XXL (Us) | 2 | 55.7 $\pm$ 1.09
PromptNER GPT3.5 (Us) | 2 | ${62.33}\pm{6.30}$
PromptNER GPT 4 (Us) | 2 | $\textbf{72.63}\pm\textbf{5.48}$
Table 4: Few Shot results on the FewNERD dataset on the INTRA 10-way task, $k$
is the number of datapoints used by each method. Results show micro-F1 scores.
Our method manages to outperform other methods by $4\%$ on F1-score

##  4 Experiments and Results

Resource and Model Description

For all the experiments below we present the result of our method when using a
Pretrained Conditional Generation Language Model of T5-Flan (11B) (Chung et
al., 2022), GPT-3.5 (Brown et al., 2020) (specifically the text-davinci-003
model) and GPT4 (OpenAI, 2023). The results for all competing methods are
taken from the tables reported in their respective publications and papers. We
use the standard metric in NER (De Mulder et al., 2015; Wang et al., 2022b)
Micro-F1 score and report the mean and variance over 5 runs on a random sample
of 500 examples on the test set.

Experiment 1: Standard Low Resource NER

In this experiment we used the most common NER dataset: CoNLL (Sang and De
Meulder, 2003). This dataset is one where the standard methods are capable of
reaching F1-score performance in the range of $91\%-94\%$, (Wang et al.,
2022b) however, when performed in the low resource settings, these methods are
significantly less powerful. We show results for Few Shot Learning with $k=5$
points from CoNLL for all the competing methods in table 1 and report the
standard averaged micro-F1 scores of the models. The table shows that our
method outperforms all competing methods when evaluated in the low resource
regime - with GPT4 achieving absolute performance gains of around $4\%$ in F1
score.

Experiment 2: Cross Domain NER

In this experiment we use the CrossNER (Liu et al., 2021) dataset. The
training set is a subset of the CoNLL dataset (Sang and De Meulder, 2003), but
the evaluation domains are from five domains with different entity types -
Politics, Literature, Music, AI, and Natural Sciences. The dataset does not
have any explicitly contradictory examples across splits (i.e. there is no
phrase that is considered an entity in one domain that is explicitly marked as
not an entity in another domain), however often the conception of what is an
entity can vary significantly. For example, in the AI split abstract
mathematical algorithms and methods in Machine Learning like the phrase ‘deep
learning’ are considered entities, while in Politics abstract political
ideologies and methods in Political Science like ‘polling’ are not entities.
On this dataset, few shot methods are not typically able to perform well with
only around 5-10 examples, and hence all other successful methods use a
significant portion of the training and dev splits of CrossNER (100-200
examples). We show our results in table 2, and despite using only $1\%-2\%$ of
the data as the other methods, we are able to achieve state-of-the-art
performance in $3/5$ of the domains with GPT4, outperforming other methods by
an absolute F1 of $3\%$ on average.The systems seem to perform worse on the AI
and Science domain, however it is difficult to make any inferences on the
reason why this occurs.

Experiment 3: Biomedical Domain NER

We next use the GENIA dataset (Kim et al., 2003), a biomedical dataset taken
from medical database. This presents a different domain with a significant
shift in vocabulary in the corpus when compared to the previous datasets. This
dataset also has more entity types (32 vs 17) and is more technical than the
CrossNER Natural Sciences domain. We show our results on the 32-way 5 shot
setting in table 3. Once again PromptNER outperforms all competing methods,
with GPT3.5 outperforming the best competitor by 3% absolute F1 and GPT4 doing
so by 9%. This shows our method is flexible, and can perform well across a
variety of domains with little overhead.

Experiment 4: Contradictory Domain NER

In this experiment we use the Intra split of the FewNERD dataset (Ding et al.,
2021), using the test split and compiling results for the 10-way problem
setting. This dataset is contradictory, in that the sets marked train, dev and
test all have non-overlapping entity types which are labeled as entities. For
example the train split considers people to be entities, but does not consider
events or buildings to be entities, while the dev set considers only buildings
or events to be entities. This is a difficult benchmark as the labels of the
training and dev sets are actively misleading when the goal is to perform well
on the test set. Table 4 shows how our method outperforms all competing
methods in this setting, doing so by an average (absolute) percentage increase
of over $4\%$ F1. This shows the flexibility of our method in even the most
pathological of cases: where the requirement for entities to be extracted is
actively in conflict with the common understanding of the word ‘entity’, and
the existing data sources. Changing the definition to explicitly rule out
objects which would normally be considered an entity is particularly useful
here, as shown in later ablations.

Data Contamination Concerns:  Since we are using Large Language Models which
have been trained on data from undisclosed sources (Brown et al., 2020;
OpenAI, 2023), we consider the possibility that parts of the evaluation sets
from the above experiments has been seen by the model during the language
modeling period, and the implications of this on the results of the
experiments. Recent results (Chowdhery et al., 2022) suggest that when using
Large Language Models for Few-Shot Learning, the performance on contaminated
data (which has been seen during training) is not significantly different from
the performance on clean, unseen data.

The underlying text corpora for all these datasets are sourced from easily
accessible collections of corpora online (e.g. Reuters articles) and so is
quite likely to have been seen during the training period. This is not too
problematic for the NER task in specific, as seeing the sentences with a
language modeling objective does not by itself provide any signal on which
words are named entities, hence we are more concerned with the possibility of
the label information being seen during language modeling. Since we cannot
reliably say that there was not some explicit entity-aware training segment
during the GPT3 to GPT4 series, we rely on the published dates on which GPT3.5
stops updating its information. The ConLL and GENIA datasets have been
available for over 10 years and have been extremely popular datasets for a
long time, implying that the label set may have been seen during Language
Modelling. There is still valuable insights to obtain from the experiment,
however, as GPT4 has significantly better performance than GPT3.5 on both
datasets, implying that even if both methods have seen the label information,
there are benefits to scaling and using a more powerful backbone Language
Model. CrossNER was released in Dec 2020, and was published as Liu et al.
(2021) only 6 months before the official GPT3.5 knowledge cutoff date. It is
also a much less popular NER dataset, being used only by methods seeking to
show domain transferability of their systems, as such we argue it is unlikely
that explicit label information from this dataset was used during GPT3.5
training. Finally FewNERD was released only a few months before the official
cut-off date for GPT3.5 data of September 2021. This makes it highly unlikely
that FewNERD label data was seen in the training process of GPT3.5. Overall we
cannot make any strong claims about data contamination, however we can say
that for the CrossNER and especially the FewNERD experiments, it is unlikely
that data contamination is the primary driving force behind the improvement in
the results on these datasets.

Model | Model Size | ConLL | Genia | Politics | Literature | Music | AI | Science | FewNERD
---|---|---|---|---|---|---|---|---|---
GPT4 | ? | 83.48 | 58.44 | 78.61 | 74.44 | 84.26 | 64.83 | 72.59 | 72.63
GPT3 | 175 Billion | 78.62 | 52.8 | 71.74 | 64.15 | 77.78 | 59.35 | 64.83 | 62.33
T5XXL | 11 Billion | 45.66 | 19.34 | 39.43 | 36.55 | 41.93 | 30.67 | 46.32 | 23.2
T5XL | 3 Billion | 24.12 | 10.5 | 18.45 | 18.62 | 25.79 | 10.58 | 26.39 | 8.35
Table 5: Model performance over various model sizes, there are clear benefits to scaling the backbone Large Language Model Def | FS | CoT | Cand | ConLL | Genia | Pol | Lit | Mus | AI | Sci | FewNERD | Avg Rank
---|---|---|---|---|---|---|---|---|---|---|---|---
✓ | ✓ | ✓ | ✓ | 78.6 | 52.8 | 71.7 | 64.1 | 77.7 | 59.3 | 64.8 | 62.3 | 1
✓ | ✓ | ✓ | ✗ | 71.6 | 38.5 | 61.3 | 46.3 | 60.2 | 34.2 | 46.8 | 57.3 | 3.5
✓ | ✓ | ✗ | ✓ | 75.1 | 49.2 | 70.4 | 54.9 | 70.6 | 53.6 | 60.5 | 42.4 | 2.1
✓ | ✗ | ✓ | ✓ | 68.1 | 23.2 | 20.3 | 21.3 | 24.5 | 40.7 | 40.6 | 34.6 | 5.6
✗ | ✓ | ✓ | ✓ | 63.3 | 46.2 | 57.7 | 49.6 | 50 | 29 | 50.8 | 34.8 | 4
✗ | ✓ | ✓ | ✗ | 54.8 | 37.2 | 49.8 | 37.3 | 54.7 | 27.8 | 21.7 | 18.8 | 5.6
✗ | ✓ | ✗ | ✗ | 49.7 | 39.3 | 42.5 | 40.3 | 48.6 | 24.5 | 35.9 | 16.1 | 6.1
Table 6: Ablation over components of PromptNER on GPT3.5. Def: Definitions,
FS: Few Shot Examples, CoT: Explanations required, Cand: Candidate entities in
predicted list. Every component improves performance of the method in general,
with the setting of all components vastly outperforming the traditional Few
Shot Prompting and Chain-of-Thought Prompting methods

##  5 Ablations and Variations

In this section, we set out to investigate the effects of the different
aspects of this pipeline and answer some questions on what really matters to
the success of the method: In all the tables for this section we refer to the
CrossNER Domains by their domain name alone and we refer to the FewNERD Test
Intra 10 way setting as FewNERD alone.

Pretrained Language Model: We hold all other parts of the pipeline constant
i.e the definitions, examples and chain of thought structure of the output and
vary the baseline model that we use to compute our predictions, we hope to
understand whether there is any trend over which models perform better. Table
5 shows the results that we might expect, there are significant gains to
scaling the size of the Large Language Model. A qualitative analysis of the
results suggests that T5XL is barely able to perform the instruction provided
in the prompt, sometimes breaking the output format structure, often
predicting a single entity multiple times, etc. T5XXL is much better, it is
able to follow the output format consistently, however, it is unable to use
the definition of entities properly, often labeling dates, numbers, and months
as entities despite being explicitly provided a definition that excludes these
types of words. This gives us reason to believe that this method is likely to
improve as LLMs get better at following instructions more exactly.

Components of PromptNER : We can consider PromptNER as having 4 different
components that can sequentially be turned off - the provision of a definition
(Defn), the provision of few-shot examples (FS), the requirement to explain
the reasoning on why a candidate is or is not an entity (CoT) and finally
whether or not the list should contain only entities, or also contain
candidate entities that may be declared as not meeting the definition of an
entity (Cand). In tables 6 we show how the performance changes for GPT3.5 as
we remove only one component of the system and specifically check
configurations of Chain-of-thought Prompting and FewShot Prompting. For the
sake of brevity, we only show the mean of the Micro-F1, however, a complete
table with standard deviations can be found in the appendix 9. The results
consistently show that every part of the pipeline is useful, with the
definition being more important in tasks where the common conception of an
entity is quite different from that of the domain (FewNERD, AI) as opposed to
the more standard setting (ConLL, Politics). Notably, the row that corresponds
to classic Few Shot Learning has an average rank of $6.1$, with classic Chain
of Thought Prompting having an average rank of $5.6$, both of these are
greatly outperformed by the setting with all components included. This shows
that the inclusion of definitions and candidates offers benefits over
classical prompting-based approaches. The results for GPT4 7 show very similar
trends, with again the row corresponding to all components being included
having a much better average rank than all other settings. We also see that of
all components of the system, the removal of the definitions and the examples
are the most damaging across all the datasets.

Def | FS | CoT | Cand | ConLL | Genia | Pol | Lit | Mus | AI | Sci | FewNERD | Avg Rank
---|---|---|---|---|---|---|---|---|---|---|---|---
✓ | ✓ | ✓ | ✓ | 83.4 | 58.4 | 78.6 | 74.4 | 84.2 | 64.8 | 72.5 | 72.6 | 1.2
✓ | ✓ | ✓ | ✗ | 78.5 | 51.8 | 70.3 | 69.6 | 73.8 | 66.1 | 67.6 | 59.5 | 2.5
✓ | ✓ | ✗ | ✓ | 67.4 | 49.2 | 72.4 | 63.5 | 80.5 | 60.5 | 59.7 | 62.8 | 3
✓ | ✗ | ✓ | ✓ | 84.3 | 30.2 | 62.8 | 53.2 | 63.5 | 42.7 | 41.4 | 30.2 | 4.7
✗ | ✓ | ✓ | ✓ | 70.7 | 53.3 | 65.6 | 55 | 53 | 45.1 | 53.7 | 43.2 | 4.1
✗ | ✓ | ✓ | ✗ | 63 | 38.5 | 60.2 | 54.6 | 58.9 | 35.9 | 46.3 | 27.1 | 5.6
✗ | ✓ | ✗ | ✗ | 66.7 | 27.4 | 58.3 | 46.4 | 54.7 | 27.7 | 36.2 | 21.9 | 6.2
Table 7: Ablation over components of CoTNER on GPT4. Def: Definitions, FS: Few Shot Examples, CoT: Explanations required, Cand: Candidate entities in predicted list. Every component improves performance of the method in general, with the setting of all components vastly outperforming traditional Few Shot Prompting and Chain-of-Thought Prompting. Candidate inclusion is the single most important component Survey | All | P | L | M | A | S
---|---|---|---|---|---|---
PromptNER Acc | 44 | 60 | 30 | 65 | 35 | 30
Labels Acc | 52 | 35 | 80 | 65 | 40 | 40
PromptNER better | 33 | 60 | 20 | 20 | 30 | 35
PromptNER worse | 56 | 15 | 60 | 60 | 65 | 50
Table 8: Survey Results with the CrossNER datasets as columns, all numbers are
percentages. PromptNER Acc and Labels Acc show the fraction of these lists
marked as having a correct answer. Better and worse rows show the percentage
of times annotators selected PromptNER as the better or worse list. Results
show there are many cases where the ‘incorrect predictions’ of PromptNER
should be considered as good as the ground truth.

##  6 Human Survey of Errors

An inspection of the errors of PromptNER shows many cases where the ground
truth and the prediction are in fact different, however, it could be argued
that the difference is either meaningless or that given the under-specified
nature of NER both sets of entities are equally valid. To quantify the extent
to which this is true we randomly picked 20 examples from each CrossNER
dataset where the prediction and ground truth entity lists differ (the set of
entities identified in a sentence, ignoring type) and asked 10 human
annotators (each example is seen by 3 distinct annotators) to comment on the
lists. Specifically, the annotators are given a dataset dependent definition
of the NER problem and are shown two lists of entities. One of these lists is
the set of entities identified by our GPT4 method, the other is the ground
truth, however, evaluators are not given this information. They identify if
the lists have correctly identified all the named entities and only named
entities in a sentence, and if not are asked to provide the phrases that are
incorrectly identified. Finally, they are asked for an opinion on which list
(if any) can be considered better at identifying which words or phrases are
named entities.

The results of the survey are summarized in Table 8. In all of the domains, at
least 20% of the examples are ones where PromptNER has the better entity list
and PromptNER is worse than the ground truth label no more than 40% of the
time. In 4/5 of the domains, the difference between the accuracy of the ground
truth labels and PromptNER does not exceed 10%. There is interestingly a
domain (Politics) where PromptNER is considerably better than the ground truth
labels. This survey confirms that there are many legitimate errors in the
PromptNER predictions, with the ground truth label consistently providing a
better option. The survey also shows, however, that even before we consider
the entity type annotations, a very considerable percentage of the disagreeing
predictions made by PromptNER are not ‘mistakes’ but rather equivalent and
roughly equally acceptable solutions to the NER instance. The results suggest
that the F1 metric scores reported in Table 2 are estimates with a downward
bias, especially in domains like Politics and Sciences, where the annotators
were not likely to say that the PromptNER prediction was worse than the ground
truth labels. More generally the survey exposes a need for human evaluation
and a measure of inter-annotator disagreement on NER datasets, as on datasets
with high disagreement it may be less fruitful to expect models to approach
the 100% F1 mark.

##  7 Limitations

Due to its reliance on prompting an LLM, PromptNER is unable to preserve span
information and represent the spans of the words or phrases in the LLM output.
This then necessitates a hand-crafted parsing strategy to extract the
predicted entity spans and types from the LLM output, which leads to
suboptimal solutions when there are many repeated words or phrases in a
sentence. The experiments in this paper are not able to explicitly control for
data contamination, unfortunately, this is unavoidable when using pre-trained
LMs from sources that do not reveal the data they trained on. There is also a
danger of interpreting parts of this pipeline incorrectly for the sake of
interpretability — the explanations provided by the system can be logically
inconsistent and should not be considered a way to make the NER system
interpretable, the candidate lists can completely miss entities from a
sentence and so should not be considered as a safe shortlist which will
contain all entities with high probability.

##  8 Conclusion

In this paper, we introduce PromptNER , an NER system that outperforms
competing few-shot methods on the ConLL, GENIA and FewNERD datasets,
outperforms cross-domain methods on 3/5 splits of CrossNER, and has superior
performance than few-shot and chain-of-thought prompting on GPT4. We also
conduct a human study of the disagreements between the ground truth of the
CrossNER dataset and our model output, finding that a notable percentage of
the disagreements are not ones that are considered ‘mistakes’ by a human
annotator. Overall we provide an alternate way to approach the few-shot NER
problem that requires no specialized pretraining, is easy to adjust to
different domains, and maintains performance despite using very little
training data.

## References

  * Ali et al. (2022) Sajid Ali, Khalid Masood, Anas Riaz, and Amna Saud. 2022.  Named entity recognition using deep learning: A review.  In _2022 International Conference on Business Analytics for Technology and Security (ICBATS)_ , pages 1–7. IEEE.
  * Borthwick et al. (1998) Andrew Borthwick, John Sterling, Eugene Agichtein, and Ralph Grishman. 1998.  Exploiting diverse knowledge sources via maximum entropy in named entity recognition.  In _Sixth Workshop on Very Large Corpora_.
  * Borthwick (1999) Andrew Eliot Borthwick. 1999.  _A maximum entropy approach to named entity recognition_.  New York University.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2023) Xiang Chen, Lei Li, Qiaoshuo Fei, Ningyu Zhang, Chuanqi Tan, Yong Jiang, Fei Huang, and Huajun Chen. 2023.  One model for all domains: Collaborative domain-prefix tuning for cross-domain ner.  _arXiv preprint arXiv:2301.10410_.
  * Chen et al. (2022) Yanru Chen, Yanan Zheng, and Zhilin Yang. 2022.  Prompt-based metric learning for few-shot ner.  _arXiv preprint arXiv:2211.04337_.
  * Chinchor (1995) Nancy Chinchor. 1995.  Muc-6 named entity task definition (version 2.1).  In _6th Message Understanding Conference, Columbia, Maryland_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Church et al. (2021) Kenneth Ward Church, Zeyu Chen, and Yanjun Ma. 2021.  Emerging trends: A gentle introduction to fine-tuning.  _Natural Language Engineering_ , 27(6):763–778.
  * Cui et al. (2021) Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021.  Template-based named entity recognition using bart.  _arXiv preprint arXiv:2106.01760_.
  * Das et al. (2022) Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca J Passonneau, and Rui Zhang. 2022\.  Container: Few-shot named entity recognition via contrastive learning.  In _ACL_.
  * De Mulder et al. (2015) Wim De Mulder, Steven Bethard, and Marie-Francine Moens. 2015.  A survey on the application of recurrent neural networks to statistical language modeling.  _Computer Speech & Language_, 30(1):61–98.
  * Ding et al. (2023) Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023.  Parameter-efficient fine-tuning of large-scale pre-trained language models.  _Nature Machine Intelligence_ , pages 1–16.
  * Ding et al. (2021) Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Hai-Tao Zheng, and Zhiyuan Liu. 2021.  Few-nerd: A few-shot named entity recognition dataset.  _arXiv preprint arXiv:2105.07464_.
  * Eftimov et al. (2017) Tome Eftimov, Barbara Koroušić Seljak, and Peter Korošec. 2017.  A rule-based named-entity recognition method for knowledge extraction of evidence-based dietary recommendations.  _PloS one_ , 12(6):e0179488.
  * Farmakiotou et al. (2000) Dimitra Farmakiotou, Vangelis Karkaletsis, John Koutsias, George Sigletos, Constantine D Spyropoulos, and Panagiotis Stamatopoulos. 2000.  Rule-based named entity recognition for greek financial texts.  In _Proceedings of the Workshop on Computational lexicography and Multimedia Dictionaries (COMLEX 2000)_ , pages 75–78.
  * Hu et al. (2022a) Jinpeng Hu, He Zhao, Dan Guo, Xiang Wan, and Tsung-Hui Chang. 2022a.  A label-aware autoregressive framework for cross-domain ner.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 2222–2232.
  * Hu et al. (2022b) Xuming Hu, Yong Jiang, Aiwei Liu, Zhongqiang Huang, Pengjun Xie, Fei Huang, Lijie Wen, and Philip S Yu. 2022b.  Entda: Entity-to-text based data augmentation approach for named entity recognition tasks.  _arXiv preprint arXiv:2210.10343_.
  * Huang et al. (2022a) Yucheng Huang, Kai He, Yige Wang, Xianli Zhang, Tieliang Gong, Rui Mao, and Chen Li. 2022a.  Copner: Contrastive learning with prompt guiding for few-shot named entity recognition.  In _Proceedings of the 29th International conference on computational linguistics_ , pages 2515–2527.
  * Huang et al. (2022b) Yucheng Huang, Kai He, Yige Wang, Xianli Zhang, Tieliang Gong, Rui Mao, and Chen Li. 2022b.  COPNER: Contrastive learning with prompt guiding for few-shot named entity recognition.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 2515–2527, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional lstm-crf models for sequence tagging.  _arXiv preprint arXiv:1508.01991_.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_.
  * Kim et al. (2003) J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. 2003.  Genia corpus—a semantically annotated corpus for bio-textmining.  _Bioinformatics_ , 19(suppl_1):i180–i182.
  * Lee et al. (2021) Dong-Ho Lee, Akshen Kadakia, Kangmin Tan, Mahak Agarwal, Xinyu Feng, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, and Xiang Ren. 2021.  Good examples make a faster learner: Simple demonstration-based learning for low-resource ner.  _arXiv preprint arXiv:2110.08454_.
  * Li et al. (2022) Jingye Li, Hao Fei, Jiang Liu, Shengqiong Wu, Meishan Zhang, Chong Teng, Donghong Ji, and Fei Li. 2022.  Unified named entity recognition as word-word relation classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 10965–10973.
  * Limsopatham and Collier (2016) Nut Limsopatham and Nigel Henry Collier. 2016.  Bidirectional lstm for named entity recognition in twitter messages.
  * Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Computing Surveys_ , 55(9):1–35.
  * Liu et al. (2021) Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2021.  Crossner: Evaluating cross-domain named entity recognition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 13452–13460.
  * Lu et al. (2022a) Jinghui Lu, Rui Zhao, Brian Mac Namee, and Fei Tan. 2022a.  Punifiedner: a prompting-based unified ner system for diverse datasets.  _arXiv preprint arXiv:2211.14838_.
  * Lu et al. (2022b) Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022b.  Unified structure generation for universal information extraction.  _arXiv preprint arXiv:2203.12277_.
  * Ma et al. (2021) Ruotian Ma, Xin Zhou, Tao Gui, Yiding Tan, Qi Zhang, and Xuanjing Huang. 2021.  Template-free prompt tuning for few-shot ner.  _arXiv preprint arXiv:2109.13532_.
  * Ma et al. (2022) Tingting Ma, Huiqiang Jiang, Qianhui Wu, Tiejun Zhao, and Chin-Yew Lin. 2022.  Decomposed meta-learning for few-shot named entity recognition.  _arXiv preprint arXiv:2204.05751_.
  * Makino et al. (2002) Takaki Makino, Yoshihiro Ohta, Jun’ichi Tsujii, et al. 2002.  Tuning support vector machines for biomedical named entity recognition.  In _Proceedings of the ACL-02 workshop on Natural language processing in the biomedical domain_ , pages 1–8.
  * Mayfield et al. (2003) James Mayfield, Paul McNamee, and Christine Piatko. 2003.  Named entity recognition using hundreds of thousands of features.  In _Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003_ , pages 184–187.
  * Mikheev et al. (1999) Andrei Mikheev, Marc Moens, and Claire Grover. 1999.  Named entity recognition without gazetteers.  In _Ninth Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1–8.
  * Ming et al. (2022) Hong Ming, Jiaoyun Yang, Lili Jiang, Yan Pan, and Ning An. 2022.  Few-shot nested named entity recognition.  _arXiv preprint arXiv:2212.00953_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Sang and De Meulder (2003) Erik F Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  _arXiv preprint cs/0306050_.
  * Sharma et al. (2022) Abhishek Sharma, Sudeshna Chakraborty, and Shivam Kumar. 2022.  Named entity recognition in natural language processing: A systematic review.  In _Proceedings of Second Doctoral Symposium on Computational Intelligence: DoSCI 2021_ , pages 817–828. Springer.
  * Shen et al. (2021) Yongliang Shen, Xinyin Ma, Zeqi Tan, Shuai Zhang, Wen Wang, and Weiming Lu. 2021\.  Locate and label: A two-stage identifier for nested named entity recognition.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2782–2794, Online. Association for Computational Linguistics.
  * Tong et al. (2022) Yiqi Tong, Fuzhen Zhuang, Deqing Wang, Haochao Ying, and Binling Wang. 2022.  Improving biomedical named entity recognition with a unified multi-task mrc framework.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 8332–8336. IEEE.
  * Wang et al. (2022a) Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2022a.  Towards understanding chain-of-thought prompting: An empirical study of what matters.  _arXiv preprint arXiv:2212.10001_.
  * Wang et al. (2022b) Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. 2022b.  Deepstruct: Pretraining of language models for structure prediction.  _arXiv preprint arXiv:2205.10475_.
  * Wang et al. (2020) Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2020.  Automated concatenation of embeddings for structured prediction.  _arXiv preprint arXiv:2010.05006_.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_.
  * Xu et al. (2022) Weiwen Xu, Xin Li, Wenxuan Zhang, Meng Zhou, Lidong Bing, Wai Lam, and Luo Si. 2022\.  From clozing to comprehending: Retrofitting pre-trained language model to pre-trained machine reader.  _arXiv preprint arXiv:2212.04755_.
  * Yang et al. (2022) Linyi Yang, Lifan Yuan, Leyang Cui, Wenyang Gao, and Yue Zhang. 2022.  Factmix: Using a few labeled in-domain examples to generalize to cross-domain named entity recognition.  _arXiv preprint arXiv:2208.11464_.
  * Yu et al. (2020) Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020.  Named entity recognition as dependency parsing.  _arXiv preprint arXiv:2005.07150_.
  * Zhou and Su (2002) GuoDong Zhou and Jian Su. 2002.  Named entity recognition using an hmm-based chunk tagger.  In _Proceedings of the 40th annual meeting of the association for computational linguistics_ , pages 473–480.

##  Appendix A Appendix

Definitions | Few Shot | CoT | Candidates | ConLL | Genia | Politics | Literature | Music | AI | Science | FewNERD
---|---|---|---|---|---|---|---|---|---|---|---
✓ | ✓ | ✓ | ✓ | 83.4 $\pm$ 5.6 |  58.4  $\pm$ 6.8 |  78.6  $\pm$ 4.7 |  74.4  $\pm$ 6.2 |  84.2  $\pm$ 5.1 | 64.8 $\pm$ 6.8 |  72.5  $\pm$ 4.1 |  72.6  $\pm$ 5.4
✓ | ✓ | ✓ | ✗ | 78.5 $\pm$ 5.2 | 51.8 $\pm$ 5.2 | 70.3 $\pm$ 3.1 | 69.6 $\pm$ 4.1 | 73.8 $\pm$ 2.5 |  66.1  $\pm$ 4.6 | 67.6 $\pm$ 4.1 | 59.5 $\pm$ 3.6
✓ | ✓ | ✗ | ✓ | 67.4 $\pm$ 3.4 | 49.2 $\pm$ 3.6 | 72.4 $\pm$ 5.1 | 63.5 $\pm$ 5.6 | 80.5 $\pm$ 6.4 | 60.5 $\pm$ 4.7 | 59.7 $\pm$ 3.5 | 62.8 $\pm$ 4.7
✓ | ✗ | ✓ | ✓ | 84.3 $\pm$ 2.4 | 30.2 $\pm$ 7.9 | 62.8 $\pm$ 5.2 | 53.2 $\pm$ 6.7 | 63.5 $\pm$ 5.6 | 42.7 $\pm$ 3.8 | 30.2 $\pm$ 7.9 | 30.2 $\pm$ 7.9
✗ | ✓ | ✓ | ✓ | 70.7 $\pm$ 5.8 | 53.3 $\pm$ 6.1 | 65.6 $\pm$ 4.3 | 55 $\pm$ 2.35 | 53 $\pm$ 5.8 | 45.1 $\pm$ 4.2 | 53.3 $\pm$ 6.1 | 43.2 $\pm$ 6.3
✗ | ✓ | ✓ | ✗ | 63 $\pm$ 4.72 | 38.5 $\pm$ 6.2 | 60.2 $\pm$ 6.1 | 54.6 $\pm$ 5.7 | 58.9 $\pm$ 5.3 | 35.9 $\pm$ 4.6 | 46.3 $\pm$ 7.5 | 27.1 $\pm$ 5.3
✗ | ✓ | ✗ | ✗ | 66.7 $\pm$ 7.0 | 27.4 $\pm$ 5.2 | 58.3 $\pm$ 5.2 | 46.4 $\pm$ 5.1 | 54.7 $\pm$ 2.4 | 27.7 $\pm$ 6.3 | 36.2 $\pm$ 5.3 | 21.9 $\pm$ 4.5
Table 9: Complete Ablation Results for GPT4

###  A.1 Exact Prompts used for each dataset

Figure 2: AI Input Prompt Figure 3: Conll Input Prompt Figure 4: FewNERD Input
Prompt Figure 5: GENIA Input Prompt Figure 6: Literature Input Prompt Figure
7: Music Input Prompt Figure 8: Science Input Prompt Figure 9: Politics Input
Prompt