# P-ICL: Point In-Context Learning for Named Entity Recognition
with Large Language Models

Guochao Jiang†, Zepeng Ding†, Yuchen Shi†, Deqing
Yang${}^{{\dagger}{\ddagger}\textrm{\Letter}}$
†School of Data Science, Fudan University, Shanghai, China
‡Shanghai Key Laboratory of Data Science, Shanghai, China
†{gcjiang22, zpding22<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

In recent years, the rise of large language models (LLMs) has made it possible
to directly achieve named entity recognition (NER) without any demonstration
samples or only using a few samples through in-context learning (ICL).
However, standard ICL only helps LLMs understand task instructions, format and
input-label mapping, but neglects the particularity of the NER task itself. In
this paper, we propose a new prompting framework P-ICL to better achieve NER
with LLMs, in which some point entities are leveraged as the auxiliary
information to recognize each entity type. With such significant information,
the LLM can achieve entity classification more precisely. To obtain optimal
point entities for prompting LLMs, we also proposed a point entity selection
method based on K-Means clustering. Our extensive experiments on some
representative NER benchmarks verify the effectiveness of our proposed
strategies in P-ICL and point entity selection.

P-ICL: Point In-Context Learning for Named Entity Recognition
with Large Language Models

Guochao Jiang†, Zepeng Ding†, Yuchen Shi†, Deqing
Yang${}^{{\dagger}{\ddagger}\textrm{\Letter}}$ †School of Data Science, Fudan
University, Shanghai, China ‡Shanghai Key Laboratory of Data Science,
Shanghai, China †{gcjiang22, zpding22<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

(a) Standard In-Context Learning (b) Point In-Context Learning (P-ICL)

Figure 1: The overview of LLM-based NER using the standard ICL and the P-ICL
proposed in this paper. Standard ICL cannot provide LLM with sufficient entity
type information and entity classification. Comparatively, P-ICL can make up
for this shortcoming by providing some point entities per entity type.

Named Entity Recognition (NER) Li et al. (2022) is a critical task of
information extraction and plays a vital role in various downstream
applications, including knowledge graph Xu et al. (2017) construction,
improving information retrieval systems Banerjee et al. (2019) and question-
answering systems Aliod et al. (2006). The primary goal of NER is to precisely
identify and classify the spans of entity mentions within a sentence into
their corresponding entity types.

Large Language Models (LLMs), such as
ChatGPT111https://openai.com/blog/chatgpt and GPT-4 OpenAI (2023), have
introduced new research directions in various natural language processing
tasks. This shift is largely due to the in-context learning (ICL) abilities of
LLMs, allowing them to perform tasks with only a few examples. With the rapid
progress of LLMs, exploring their direct use or applying few-shot learning
techniques to NER tasks is highly valuable. This strategy, which utilizes LLMs
without extensive fine-tuning, offers a potential way to significantly lower
the training costs associated with NER tasks.

Existing research has often neglected the unique aspects of using LLMs for NER
through ICL, especially when compared to their usage in other natural language
processing (NLP) tasks. Previous study Min et al. (2022) has shown that
improvements in LLMs’ performance through ICL mainly come from input-label
mapping and format following. In fact, for NER, the information of entity
type, along with entity classification, are the key factors to achieve overall
task success. Unfortunately, such information has not yet been emphasized in
previous ICL-based approaches.

To tackle this issue, we propose a new framework for NER in this paper, named
P-ICL (Point In-Context Learning). Our framework equips the LLM with some
representative entity instances for each entity type in the prompt, which
provides significant information on entity type and entity classification.
Compared with the demonstration examples in standard ICL’s prompt, the
representative entity instances added in P-ICL’s prompt can be regarded as the
points in entity space, which are denoted as Point entities in this paper. As
shown in Figure 1, providing these point entities for each entity type in the
prompt greatly helps the LLM accurately understand the semantic distinctions
between different entity types and their related domains. This enhancement is
crucial for improving the NER performance of LLMs.

Our main contributions in this paper are summarized as follows.

1\. We propose a novel framework P-ICL with an LLM to achieve NER. Our
framework leverages point entities to provide the LLM with the significant
information of entity type and entity classification, that is overlooked by
standard ICL.

2\. Our results of using three LLMs upon four representative NER benchmarks
demonstrate the advantages of our proposed P-ICL over the standard ICL method.

3\. The point entity selection method we proposed achieves better performance
in the experiments compared with random selection, illustrating the
significance of selecting more representative point entities for enhancing
P-ICL’s capability of NER.

##  2 Related Work

NER is an important task in the field of information extraction, which focuses
on identifying named entities of interest from sentence-level or document-
level text Sang and Meulder (2003). The existing NER solutions can be divided
into three main categories: sequence labeling, span classification and
generative methods. Using the sequence labeling approaches Ratinov and Roth
(2009); Straková et al. (2019); Dai et al. (2020) split the text into tokens
and the model assigns a label like BIO tagging to each token. Methods that use
span classification Wang et al. (2020); Yu et al. (2020) take the text into
account in the form of the basic elements of a span, and then for the span
decide whether it is an entity. Generative methods, on the other hand, rely on
the overall framework of generative modeling to output the results of the
final NER in the form of text generation Cui et al. (2021); Yan et al. (2021).

In recent years, LLMs have shown promising results on many NLP tasks including
NER. Wei et al. (2023b) explored the direct use of ChatGPT for zero-shot named
entity recognition, relation extraction and event extraction, and proposed a
framework named ChatIE to interact with LLMs to accomplish the task of
information extraction. Similarly, Ji (2023) explores the performance of the
open-source model Vicuna Zheng et al. (2023) on NER in the zero-shot and few-
shot scenarios. Hu et al. (2023) explores the NER capabilities of ChatGPT
within a specific clinical domain, revealing the possibilities of applying
generalized language models in specialized domains. Xie et al. (2023a) present
a systematic empirical investigation of zero-shot NER with LLM and adapt
prevalent reasoning methods to NER. Xie et al. (2023b) explored the
possibility of boosting zero-shot NER with LLMs via self-improving.

In-context learning (ICL), as an emergent capability of LLMs, allows LLMs to
quickly have the ability to adapt to downstream tasks given a few task
examples in the prompt. The work that has been done focuses on the working
mechanism of ICL and the factors that influence it Min et al. (2022); Lyu et
al. (2023); Wei et al. (2023a). Min et al. (2022) found that the label space
of the examples, the distribution of the input text and the overall form of
the sequence were factors that influenced ICL performance. Wang et al. (2023)
delves into the mechanism of label words in ICL from the information flow
perspective and introduces a re-weighting method to enhance the ICL
performance of LLM based on this finding.

##  3 Preliminaries

NER aims to extract entities from the given text and assign the correct entity
type to the entities. Formally, we denote the input token sequence as
$x=[x_{1},x_{2},\cdots,x_{m}]$. With LLM $f_{\text{LLM}}$, we denote the
output token sequence as
$y=f_{\text{LLM}}(x|\mathcal{I})=[y_{1},y_{2},\cdots,y_{n}]$ under the
instruction $\mathcal{I}$. Instruction $\mathcal{I}$ usually includes the
entity types that the task focuses on such as PER, LOC, ORG and MISC in the
CoNLL2003 dataset Sang and Meulder (2003).

LLMs can learn from demonstrations to improve the downstream performance with
ICL ability. Given some demonstrations
$(i_{1},o_{1}),(i_{2},o_{2}),\cdots,(i_{k},o_{k})$, LLMs can output results
that are more consistent with the task instructions and these examples. For
NER, $i_{t},t=1,\cdots,k$ represents the input text, $o_{t},t=1,\cdots,k$
represents the extraction result, and $k$ represents the number of
demonstrations. Therefore, in the case of ICL, the extraction result of model
$f_{\text{LLM}}$ for text $x$ is

$\displaystyle
y=f_{\text{LLM}}(x|\mathcal{I},(i_{1},o_{1}),(i_{2},o_{2}),\cdots,(i_{k},o_{k}))$
(1)

To facilitate the analysis of the generated results, we stipulate that $o_{t}$
must be a parsable JSON result, where the key represents the entity type and
the value is the entity list of the entity type corresponding to the key. For
example:

Input: He said further scientific study was required and if it was found that
action was needed it should be taken by the European Union.

Output: {"ORG": ["European Union"]}

Input: Japan then laid siege to the Syrian penalty area and had a goal
disallowed for offside in the 16th minute.

Output: {"LOC": ["Japan"], "MISC": ["Syrian"]}

##  4 Methodology

Figure 2: The overview of P-ICL with point entity selection. These entities
are first given a vector representation using a BERT-like encoder. Then, the
K-Means clustering method and nearest neighbor decoding strategy are used to
identify the point entities for each entity type. The word cloud figure on the
right displays the outcomes of this selection process. For instance, London,
China, and California emerge as central entities in their clusters and are
identified as point entities for LOC.

In this section, we initially discuss the standard ICL and point entity.
Subsequently, we introduce our P-ICL and the method for selecting point
entities. Figure 2 illustrates the overview of our P-ICL pipeline, including
the point entity selection process.

###  4.1 Standard ICL

In the previous discussion, we introduced a JSON schema aimed at facilitating
the generation of structured and easily parsable outputs by LLM. Nonetheless,
it is crucial to emphasize that without employing prompt engineering
techniques, such as ICL, the outputs from LLM may not consistently adhere to
the specified requirements.

ICL has been shown through empirical research to help LLMs learn the necessary
input-output mapping relevant to specific downstream tasks. This strategy
plays a key role in minimizing instances where LLM outputs fail to meet task
requirements. In this study, which concentrates on the NER task, we employ the
CoNLL2003 dataset as a case in point. We detail and apply particular prompts
to carry out zero-shot and few-shot experiments under the ICL framework.

Zero-shot:
Please list all named entities of the following entity types in the input
sentence:
\- PER
\- ORG
\- LOC
\- MISC
You should output your results in the format {"type": [entity]} as a json.
Input:  $x$

====================================

Few-shot with ICL:
Please list all named entities of the following entity types in the input
sentence:
\- PER
\- ORG
\- LOC
\- MISC
Here are some examples:
Input:  $i_{1}$
Output:  $o_{1}$
...
Input:  $i_{k}$
Output:  $o_{k}$
You should output your results in the format {"type": [entity]} as a json.
Input:  $x$

###  4.2 Point Entity

In the approach described, the standard ICL method requires a certain number
of examples to understand the details of downstream tasks. However, it does
not fully capture the unique features of the NER task. As we know, two main
factors affect NER performance: contextual text and entity mention. Typically,
the contextual text provides extensive semantic information about a sentence,
while the entity mention clearly identifies the potential entity. For LLMs,
pre-training on a wide corpus gives the model a comprehensive understanding of
natural language. As a result, many high-quality LLMs have strong text
comprehension abilities. This is why LLMs perform well in various NLP tasks,
even without specific training or in a few-shot ICL scenario. Thus, contextual
text would not be a significant limitation for LLMs’ performance in NER.

Entity mention, referring to how natural language identifies actual named
entities, should be a key focus of NER for LLMs aiming for advanced text
understanding. From a human intuition standpoint, entity mentions are usually
quite specific. A major challenge in NER involves effectively defining the
scope of each entity type in the downstream tasks. For LLMs, the strategies to
address this challenge generally fall into two categories: providing detailed
and clear definitions or presenting examples of entities.

For the methods that utilize neural networks, particularly those focusing on
sequence labeling and span classification, model training relies heavily on
the training data. This strategy aligns more closely with the previously
mentioned latter solution. Unlike other approaches, these methods do not
necessitate predefined definitions of entity types and they instead learn
patterns directly from the training data.

The inclusion of entity mention examples can greatly enhance LLMs’ capability
of identifying the boundaries of various entity types. Often, these models
cannot undergo fine-tuning in zero-shot and few-shot scenarios due to the
prohibitive computational resource requirements, which exceed the capacities
of most users. Standard ICL can empower general LLMs to acquire the knowledge
about downstream tasks through the demonstration examples (about the task) in
prompts. Similarly, embedding a specific number of entity instances for each
entity type in the prompt can help LLMs capability discriminate the boundaries
of relevant entity types, thereby boosting LLMs’ performance on NER. Since
this method only requires the presentations of entity mentions, which aligns
well with the concept of a data point, we denote it as Point entity.

###  4.3 P-ICL: Point ICL

The concept of a point entity offers a novel approach to prompting LLM for
NER. By specifying the entities of interest through sufficient point entities,
we can effectively communicate the types of entities to the LLM. This method,
when compared to the standard ICL technique, allows for the inclusion of more
training data for the LLM. Unlike standard ICL, which necessitates the full
context, the point entity approach only requires the entity mention itself.

The ICL-like method that provides data information to LLM based on point
entities is called P-ICL. Formally, assume that the entity type label set of
the NER task is $\mathcal{T}$, and the number of entity types considered is
$|\mathcal{T}|$. For the entity type $t\in\mathcal{T}$, there are $a_{t}$
point entities, respectively $p_{t,1},p_{t,2},\cdots,p_{t,a_{t}}$. In this
paper, for the NER task, taking the CoNLL2003 dataset as an example, we
consider the following prompts to conduct experiments under P-ICL:

P-ICL:
Please list all named entities of the following entity types in the input
sentence:
\- PER: e.g.  $p_{\texttt{PER},1}$,
$p_{\texttt{PER},2}$,$\cdots$,$p_{\texttt{PER},a_{\texttt{PER}}}$
\- ORG: e.g.  $p_{\texttt{ORG},1}$,
$p_{\texttt{ORG},2}$,$\cdots$,$p_{\texttt{ORG},a_{\texttt{ORG}}}$
\- LOC: e.g.  $p_{\texttt{LOC},1}$,
$p_{\texttt{LOC},2}$,$\cdots$,$p_{\texttt{LOC},a_{\texttt{LOC}}}$
\- MISC: e.g.  $p_{\texttt{MISC},1}$,
$p_{\texttt{MISC},2}$,$\cdots$,$p_{\texttt{MISC},a_{\texttt{MISC}}}$
Here are some examples:
Input:  $i_{1}$
Output:  $o_{1}$
...
Input:  $i_{k}$
Output:  $o_{k}$
You should output your results in the format {"type": [entity]} as a json.
Input:  $x$

###  4.4 Point Entity Selection

Building on the previous discussion, it’s clear that the role of point
entities is to provide LLMs with insights about different types of entities
through mentions. Therefore, it makes sense to carefully select the right
point entity. When point entities effectively represent the distribution of
the entire dataset, LLMs can use their advanced natural language understanding
to gain more information from inputs that are more representative, compared to
when point entities are chosen randomly. This highlights the importance of
careful selection of point entities to improve the performance of LLMs in
processing and interpreting data, thus maximizing the information obtained
from the inputs.

Recent advancements in Pre-trained Language Models (PLMs) like BERT Devlin et
al. (2019) have provided many solutions for NLP tasks through representation
learning. This approach mainly uses PLMs to extract relevant features from
text for use in various tasks. In a given dataset, all the data in the
training set can be used to help select representative entities. Since each
entity can have a corresponding representation, choosing some representative
entities aligns with unsupervised learning methods. In this scenario, the use
of clustering algorithms Ezugwu et al. (2022) proves to be an extremely
effective technique.

By using PLMs alongside common clustering algorithms, it’s possible to encode
training entities related to a specific entity type to obtain their
representations. Then, by clustering these representations, one can pinpoint
central entities. The selection of a central entity, based on its position in
the representation space, naturally reflects its representativeness. This
method highlights the benefit of combining PLMs with clustering algorithms for
a refined identification and selection of central entities. This enhances the
model’s capacity to generalize from representative examples in unsupervised
settings.

Formally, let
$\mathcal{E}_{t}=\left\\{e_{t,1},e_{t,2},\cdots,e_{t,|\mathcal{E}_{t}|}\right\\}$
be the entity set for entity type $t\in\mathcal{T}$. For each entity $e_{t}$
of entity type $t$, we can use the following method to obtain its
corresponding representation:

$\displaystyle\mathbf{e}_{t}=\text{Encoder}(e_{t})\in\mathbb{R}^{d},$ (2)

where $d$ is the representation dimension, which generally depends on the
encoder used. Let
$\mathbf{E}_{t}=\left\\{\mathbf{e}_{t,1},\mathbf{e}_{t,2},\cdots,\mathbf{e}_{t,|\mathcal{E}_{t}|}\right\\}$
be the corresponding entity representation set. Consistent with the above
discussion, for entity type $t$, we use the following form to obtain $a_{t}$
entity centers with classic K-Means clustering algorithm:

$\displaystyle(\mathbf{c}_{t,1},\mathbf{c}_{t,2},\cdots,\mathbf{c}_{t,a_{t}})=\text{K-Means}(\mathbf{E}_{t},a_{t}).$
(3)

Then we need to convert the corresponding representation in the representation
space into a string entity that can be used in prompt. For each entity center
$\mathbf{c}_{t}$, we obtain the corresponding point entity $c_{t}$ by
maximizing the similarity between the representation of the real entity and
its entity center:

$\displaystyle
c_{t}=\operatorname*{arg\,max}_{e\in\mathcal{E}_{t}}S\left(\text{Encoder}(e),\mathbf{c}_{t}\right),$
(4)

where the function $S$ measures the degree of similarity between two given
elements, and is generally chosen as cosine similarity in most NLP tasks.

##  5 Experiments

In this section, we will evaluate the results of our proposed P-ICL method on
some NER datasets, using different closed-source and open-source models.

###  5.1 Datasets

Dataset | Training | Training | Test | Test
---|---|---|---|---
Name | Sen. # | Ent. # | Sen. # | Ent. #
CoNLL2003 | 14,041 | 8,082 | 3,453 | 2,637
WNUT2017 | 3,394 | 703 | 1,287 | 344
ACE2004 | 6,202 | 10,090 | 812 | 1,692
ACE2005 | 7,299 | 9,873 | 1,060 | 1,573

Table 1: The datasets’ statistics of sentence number and unique entity number
in the train and test sets. In the zero-shot and few-shot scenarios, all the
in-context examples and point entities are from the training set.

We conducted our experiments on the following four NER benchmarks.

CoNLL2003 CoNLL2003 Sang and Meulder (2003) is a collection of news wire
articles from the Reuters Corpus, which contains 4 entity types including LOC,
ORG, PER and MISC.

WNUT2017 WNUT2017 Derczynski et al. (2017) focuses on unusual and previously-
unseen entities in the context of emerging discussions, which contains 6
entity types including person, location, corporation, product, creative-work
and group.

ACE2004 and ACE2005 ACE2004 Doddington et al. (2004) and ACE2005 Walker et al.
(2006) are two nested named entity recognition datasets, which contain 7
entity types including person, location, organization, geographical social
political, weapon, facility and vehicle. We followed the same dataset split
setup as the previous work Katiyar and Cardie (2018); Lin et al. (2019).

These datasets’ statistics are listed in Table 1.

###  5.2 Implementation Details

We report the entity-level micro F1 scores in the following result tables and
figures to compare the NER performance. To more comprehensively compare the
P-ICL performance of existing LLMs, we selected three representative models at
this stage, including GPT-3.5 Turbo, LLaMA-2-70B Touvron et al. (2023) and
Mixtral 8x7B Jiang et al. (2024). For standard ICL and P-ICL experiments, due
to a certain degree of randomness, the corresponding experimental results are
the average of the F1 scores of 5 experiments. For experiments on open-source
models, we use the vLLM Kwon et al. (2023) framework for inference. In order
to maintain the stability and consistency of model generation, we use the
greedy generation to control the outputs of LLMs. We conducted our experiments
on eight NVIDIA A800 80GB PCIe GPUs.

###  5.3 Overall Performance

We experimented on the four datasets with a variety of backbone LLMs,
including Mixtral 8x7B, GPT-3.5 Turbo, and LLaMA-2-70B. The results for
Mixtral 8x7B are documented in Table 2, while the outcomes for the other two
models are detailed in Appendix A. For each model, we employed different
approaches, ranging from a zero-shot version without demonstrations, referred
to as Vanilla, to few-shot versions, labeled ICL, which incorporate a range of
example samples. For the P-ICL method proposed in this paper, we tested
various selection techniques, including random selection and K-Means
clustering-based selection, designated as P-ICL${}_{\text{random}}$ and
P-ICL${}_{\text{bert}}$, respectively. For the ICL configurations, we
evaluated the model’s performance with different numbers of samples $b$,
specifically $b=3,5,10,20$. Moreover, in the P-ICL models, we experimented
with the number of clusters $a$ per type, exploring $a=5,10$. For instance, in
the 5+3-shot experiment for P-ICL${}_{\text{bert}}$, this denotes the
selection of five entity examples per type and three NER samples for few-shot
demonstrations. Based on the experimental results, we have the following
observations:

  1. 1.

In the case of Standard ICL, we observed that the model’s results improve with
an increasing number of demonstrations until a certain threshold is reached.
For example, there is no significant performance enhancement when scaling from
10-shot to 20-shot.

  2. 2.

When comparing Standard ICL with P-ICL${}_{\text{random}}$ at equal numbers of
demonstrations, the performance of P-ICL is consistently superior. This
finding underscores the significance of introducing point entities for the NER
task.

  3. 3.

Comparing P-ICL${}_{\text{random}}$ with P-ICL${}_{\text{bert}}$ under the
same setting, the latter outperforms the former. This result validates the
effectiveness of our proposed point entity selection method based on K-Means
clustering in enhancing the performance of P-ICL.

Furthermore, a comparison of experimental outcomes with $a=5$ and $a=10$
reveals minimal differences, hence, in the following section, we delve further
into the analysis of the impact of point entity quantity.

Method | CoNLL2003 | WNUT2017 | ACE2004 | ACE2005 | Avg.
---|---|---|---|---|---
Vanilla | 37.38 | 6.31 | 19.01 | 17.26 | 19.99
Standard ICL | 3-shot | 45.93 | 8.76 | 28.22 | 25.42 | 27.08
5-shot | 48.10 | 10.11 | 30.54 | 27.90 | 29.16
10-shot | 51.00 | 12.56 | 32.42 | 30.94 | 31.73
20-shot | 51.12 | 11.37 | 32.93 | 29.78 | 31.30
P-ICL${}_{\text{random}}$ | 5+3-shot | 46.37 | 10.62 | 31.09 | 28.79 | 29.22
10+3-shot | 45.85 | 11.00 | 30.74 | 28.91 | 29.13
5+5-shot | 48.87 | 11.67 | 31.13 | 31.33 | 30.75
10+5-shot | 48.53 | 12.64 | 33.26 | 31.06 | 31.37
5+10-shot | 53.09 | 13.49 | 35.92 | 34.57 | 34.27
10+10-shot | 53.07 | 13.09 | 35.37 | 35.04 | 34.14
5+20-shot | 56.50 | 14.85 | 37.02 | 36.69 | 36.27
10+20-shot | 56.67 | 14.26 | 36.01 | 35.22 | 35.54
P-ICL${}_{\text{bert}}$ | 5+3-shot | 50.59 | 12.54 | 32.18 | 31.27 | 31.66
10+3-shot | 48.59 | 13.00 | 33.67 | 30.69 | 31.49
5+5-shot | 51.81 | 14.55 | 36.08 | 34.03 | 34.12
10+5-shot | 50.02 | 13.75 | 37.12 | 33.59 | 33.62
5+10-shot | 56.18 | 15.36 | 36.77 | 36.78 | 36.27
10+10-shot | 54.42 | 15.06 | 37.82 | 36.14 | 35.86
5+20-shot | 59.12 | 16.92 | 39.38 | 38.60 | 38.50
10+20-shot | 58.70 | 16.19 | 41.18 | 38.34 | 38.60
Table 2: Overall Experimental Results. Performance of Mixtral 8x7B across four
datasets, with the best results highlighted in bold and the second-best
underlined in each group.

###  5.4 Effect of Point Entity Number

(a) CoNLL2003 (b) WNUT2017

Figure 3: The impact of varying point entity numbers on model performance.
Performance of Mixtral 8x7B on the CoNLL2003 and WNUT2017 datasets.

To delve deeper into how the quantity of point entities influences the
performance of P-ICL, we conducted experiments on the CoNLL2003 and WNUT2017
datasets, with the results displayed in Figure 3. For a fixed sample size
$b=5,10,20$, we varied the number of point entities randomly selected from
each category, $a$, ranging from 1 to 10. The findings indicate that: (1)
P-ICL outperforms standard ICL when comparing results under an equal number of
ICL demonstrations. (2) With the same number of ICL demonstrations, P-ICL
performance initially increases as the number of point entities grows.
However, when the quantity of point entities per type is higher, the model’s
performance shows fluctuations. This variability could be attributed to
redundancy among the randomly selected point entities, which may inadvertently
compromise the LLM’s effectiveness.

###  5.5 Effect of Point Entity Selection

| BERT-large | RoBERTa-large | BGE-large | mxbai-embed-large
---|---|---|---|---
# Param. | 336M | 355M | 335M | 335M
5+3-shot | 50.59 | 49.95 | 52.01 | 51.74
10+3-shot | 48.59 | 48.37 | 50.63 | 49.46
5+5-shot | 51.81 | 50.30 | 54.04 | 53.02
10+5-shot | 50.02 | 50.20 | 52.64 | 51.96
5+10-shot | 56.18 | 55.43 | 57.76 | 57.76
10+10-shot | 54.42 | 55.65 | 57.74 | 57.29
5+20-shot | 59.12 | 60.05 | 60.19 | 60.60
10+20-shot | 58.70 | 59.65 | 60.48 | 60.59
Avg. | 53.68 | 53.70 | 55.69 | 55.30
Table 3: Impact of different encoders on point entity selection. Results on
the CoNLL2003 dataset, with the best outcomes in bold and the second-best
underlined.

To investigate the influence of point entity selection on performance
outcomes, this section presents two sets of experiments. Firstly, we explore
the impact of text encoders on results by utilizing different textual encoders
for entity representation. Secondly, we examine the effects of varying the
number of selected point entities for each entity center.

To examine the effects of different encoders, we selected four representative
models: two classic BERT-like models (BERT-large Devlin et al. (2019) and
RoBERTa-large Liu et al. (2019)), and two recent retrieval embedding models
(BGE-large Xiao et al. (2023) and mxbai-embed-
large222https://www.mixedbread.ai/blog/mxbai-embed-large-v1). All of these
models have a similar number of parameters. Employing these encoders for
entity representation and performing point entity selection based on the
methodology described in Section 4.4, we conducted experiments on the
CoNLL2003 dataset. The results, as depicted in Table 3, indicate that: (1)
Utilizing BGE-large and mxbai-embed-large as encoder consistently outperforms
BERT-like models, demonstrating that retrieval embedding models can yield more
effective point entity selection results. (2) The outcomes for both BERT-like
models are comparable, as are the results for the two retrieval embedding
models. Overall, BGE-large exhibits superior performance to mxbai-embed-large,
hence we have selected BGE-large as the encoder for our final model.

$\tau$ | P-ICLτ | P-ICL${}_{\text{bert}}$ \+ Extend
---|---|---
1 | 56.18 | 56.18
2 | 54.93 | 55.11
3 | 55.62 | 56.23
4 | 55.40 | 55.44
5 | 55.43 | 54.47
Avg. | 55.51 | 55.49
Table 4: Impact of different $\tau$ values on performance, with 5+10-shot
setting on CoNLL2003. P-ICLτ refers to the selection of $\tau$ most similar
entities as point entities for each entity center, while
P-ICL${}_{\text{bert}}$ \+ Extend denotes the model extending
P-ICL${}_{\text{bert}}$ with randomly selected entities.

In Equation 4, we select point entities by maximizing the similarity between
the real entity’s representation and the entity center. This approach is
predicated on the notion that each entity center requires only one most
representative entity. To validate this concept, we experimented with
selecting a varying number of point entities for the same entity center,
referred to as $\tau$. When $\tau>1$, a single entity center corresponds to
multiple point entities. Additionally, to eliminate the influence of the
number of point entities in comparative experiments, we expanded the number of
point entities by randomly selecting $(\tau-1)\times a$ entities post-
selection based on Equation 4.

The experimental outcomes for a 5+10-shot setting on the CoNLL2003 dataset are
presented in Table 4, where P-ICLτ denotes the model selecting $\tau$ most
similar entities as point entities for each entity center, and the
corresponding P-ICL${}_{\text{bert}}$ \+ Extend represents the model extending
P-ICL${}_{\text{bert}}$ with additional entities. The results reveal that: (1)
Comparing P-ICLτ for $\tau=1$ and $\tau>1$, selecting a larger $\tau$ does not
lead to better outcomes and may even result in a decline, suggesting that each
entity center needs only the most representative entity, and selecting more
similar entities could lead to redundancy due to excessive similarity. (2)
There is no significant difference between the results of P-ICLτ and
P-ICL${}_{\text{bert}}$ \+ Extend, implying that compared to point entities
obtained through K-Means clustering, selecting more entities for each entity
center or employing random extension does not yield better results.

##  6 Conclusion

In this paper, we introduce P-ICL, a framework that employs point entities to
improve LLM in performing NER through ICL. By incorporating point entities, we
equip LLMs with specific entity type information and entity classification,
addressing the limitations of the standard ICL approach for NER. Our findings,
based on four widely used NER datasets and experiments conducted with three
renowned LLMs, confirm the efficacy of P-ICL.

## Ethics Statement

We hereby declare that all authors of this article are aware of and adhere to
the provided ACL Code of Ethics and honor the code of conduct.

#### Use of Human Annotations

Human annotations are only used in methodological research at the beginning of
the work, to assist in analyzing the feasibility of the proposed solution.
Annotators consented to the use of data for research purposes. We ensure that
the privacy of all annotators is protected throughout the annotation process,
and all of them are adequately paid according to local standards. Human
annotations are not applied during the evaluation of our method.

#### Risks

In this paper, all datasets are obtained from official sources. The datasets
adopted have been anonymized and do not contain offensive information.
However, we cannot guarantee that the datasets do not contain socially harmful
or toxic language.

## Limitations

We focus solely on evaluating the P-ICL framework’s performance with fixed
point entities. For LLMs, having fixed point entities may not always be ideal
for varying inputs. This limitation highlights the potential for future work
on dynamically choosing point entities depending on the input sentence.

## References

  * Aliod et al. (2006) Diego Mollá Aliod, Menno van Zaanen, and Daniel Smith. 2006.  Named entity recognition for question answering.  In _Proceedings of the Australasian Language Technology Workshop, ALTA 2006, Sydney, Australia, November 30-December 1, 2006_ , pages 51–58. Australasian Language Technology Association.
  * Banerjee et al. (2019) Partha Sarathy Banerjee, Baisakhi Chakraborty, Deepak Tripathi, Hardik Gupta, and Sourabh S. Kumar. 2019.  A information retrieval based on question and answering and NER for unstructured information without using SQL.  _Wirel. Pers. Commun._ , 108(3):1909–1931.
  * Cui et al. (2021) Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021.  Template-based named entity recognition using BART.  In _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021_ , volume ACL/IJCNLP 2021 of _Findings of ACL_ , pages 1835–1845. Association for Computational Linguistics.
  * Dai et al. (2020) Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cécile Paris. 2020.  An effective transition-based model for discontinuous NER.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 5860–5870. Association for Computational Linguistics.
  * Derczynski et al. (2017) Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017.  Results of the WNUT2017 shared task on novel and emerging entity recognition.  In _Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@EMNLP 2017, Copenhagen, Denmark, September 7, 2017_ , pages 140–147. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Doddington et al. (2004) George R. Doddington, Alexis Mitchell, Mark A. Przybocki, Lance A. Ramshaw, Stephanie M. Strassel, and Ralph M. Weischedel. 2004.  The automatic content extraction (ACE) program - tasks, data, and evaluation.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation, LREC 2004, May 26-28, 2004, Lisbon, Portugal_. European Language Resources Association.
  * Ezugwu et al. (2022) Absalom E. Ezugwu, Abiodun M. Ikotun, Olaide Nathaniel Oyelade, Laith Mohammad Abualigah, Jeffrey O. Agushaka, Christopher Ifeanyi Eke, and Andronicus Ayobami Akinyelu. 2022.  A comprehensive survey of clustering algorithms: State-of-the-art machine learning applications, taxonomy, challenges, and future research prospects.  _Eng. Appl. Artif. Intell._ , 110:104743.
  * Hu et al. (2023) Yan Hu, Iqra Ameer, Xu Zuo, Xueqing Peng, Yujia Zhou, Zehan Li, Yiming Li, Jianfu Li, Xiaoqian Jiang, and Hua Xu. 2023.  Zero-shot clinical entity recognition using chatgpt.  _CoRR_ , abs/2303.16416.
  * Ji (2023) Bin Ji. 2023.  Vicunaner: Zero/few-shot named entity recognition using vicuna.  _CoRR_ , abs/2305.03253.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024.  Mixtral of experts.  _CoRR_ , abs/2401.04088.
  * Katiyar and Cardie (2018) Arzoo Katiyar and Claire Cardie. 2018.  Nested named entity recognition revisited.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_ , pages 861–871. Association for Computational Linguistics.
  * Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023.  Efficient memory management for large language model serving with pagedattention.  In _Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023_ , pages 611–626. ACM.
  * Li et al. (2022) Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2022.  A survey on deep learning for named entity recognition.  _IEEE Trans. Knowl. Data Eng._ , 34(1):50–70.
  * Lin et al. (2019) Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. 2019.  Sequence-to-nuggets: Nested entity mention detection via anchor-region networks.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 5182–5192. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Lyu et al. (2023) Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.  Z-ICL: zero-shot in-context learning with pseudo-demonstrations.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 2304–2317. Association for Computational Linguistics.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , pages 11048–11064. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _CoRR_ , abs/2303.08774.
  * Ratinov and Roth (2009) Lev-Arie Ratinov and Dan Roth. 2009.  Design challenges and misconceptions in named entity recognition.  In _Proceedings of the Thirteenth Conference on Computational Natural Language Learning, CoNLL 2009, Boulder, Colorado, USA, June 4-5, 2009_ , pages 147–155. ACL.
  * Sang and Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NAACL 2003, Edmonton, Canada, May 31 - June 1, 2003_ , pages 142–147. ACL.
  * Straková et al. (2019) Jana Straková, Milan Straka, and Jan Hajic. 2019.  Neural architectures for nested NER through linearization.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 5326–5331. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _CoRR_ , abs/2307.09288.
  * Walker et al. (2006) Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006.  Ace 2005 multilingual training corpus.  _Linguistic Data Consortium, Philadelphia 57_.
  * Wang et al. (2020) Jue Wang, Lidan Shou, Ke Chen, and Gang Chen. 2020.  Pyramid: A layered model for nested named entity recognition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 5918–5928. Association for Computational Linguistics.
  * Wang et al. (2023) Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023.  Label words are anchors: An information flow perspective for understanding in-context learning.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , pages 9840–9855. Association for Computational Linguistics.
  * Wei et al. (2023a) Jerry W. Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023a.  Larger language models do in-context learning differently.  _CoRR_ , abs/2303.03846.
  * Wei et al. (2023b) Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wenjuan Han. 2023b.  Zero-shot information extraction via chatting with chatgpt.  _CoRR_ , abs/2302.10205.
  * Xiao et al. (2023) Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof. 2023.  C-pack: Packaged resources to advance general chinese embedding.  _CoRR_ , abs/2309.07597.
  * Xie et al. (2023a) Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu, and Hongwei Wang. 2023a.  Empirical study of zero-shot NER with chatgpt.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , pages 7935–7956. Association for Computational Linguistics.
  * Xie et al. (2023b) Tingyu Xie, Qi Li, Yan Zhang, Zuozhu Liu, and Hongwei Wang. 2023b.  Self-improving for zero-shot named entity recognition with large language models.  _CoRR_ , abs/2311.08921.
  * Xu et al. (2017) Bo Xu, Yong Xu, Jiaqing Liang, Chenhao Xie, Bin Liang, Wanyun Cui, and Yanghua Xiao. 2017.  Cn-dbpedia: A never-ending chinese knowledge extraction system.  In _Advances in Artificial Intelligence: From Theory to Practice - 30th International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2017, Arras, France, June 27-30, 2017, Proceedings, Part II_ , volume 10351 of _Lecture Notes in Computer Science_ , pages 428–438. Springer.
  * Yan et al. (2021) Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021.  A unified generative framework for various NER subtasks.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 5808–5822. Association for Computational Linguistics.
  * Yu et al. (2020) Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020.  Named entity recognition as dependency parsing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 6470–6476. Association for Computational Linguistics.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.  In _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_.

##  Appendix A Overall Performance

The results for GPT-3.5 Turbo and LLaMA-2-70B are documented in Table 5 and
Table 6.

Method | CoNLL2003 | WNUT2017 | ACE2004 | ACE2005 | Avg.
---|---|---|---|---|---
Vanilla | 62.49 | 8.70 | 17.99 | 14.35 | 25.88
Standard ICL | 3-shot | 64.95 | 10.55 | 20.30 | 19.84 | 28.91
5-shot | 66.47 | 10.92 | 22.48 | 22.32 | 30.55
10-shot | 67.65 | 11.51 | 23.94 | 25.42 | 32.13
20-shot | 66.35 | 10.39 | 23.90 | 27.05 | 31.92
P-ICL${}_{\text{random}}$ | 5+3-shot | 66.80 | 16.61 | 24.31 | 24.57 | 33.07
10+3-shot | 65.70 | 14.93 | 23.59 | 24.19 | 32.10
5+5-shot | 67.56 | 16.95 | 25.49 | 27.01 | 34.25
10+5-shot | 66.58 | 16.10 | 25.85 | 26.51 | 33.76
5+10-shot | 67.66 | 16.83 | 28.18 | 28.95 | 35.40
10+10-shot | 68.27 | 18.51 | 26.64 | 30.02 | 35.86
5+20-shot | 69.31 | 19.36 | 29.10 | 32.05 | 37.46
10+20-shot | 69.58 | 18.40 | 28.31 | 31.99 | 37.07
P-ICL${}_{\text{bert}}$ | 5+3-shot | 68.87 | 19.10 | 26.34 | 25.57 | 34.97
10+3-shot | 69.47 | 17.99 | 26.88 | 25.36 | 34.93
5+5-shot | 70.33 | 16.70 | 27.23 | 29.12 | 35.85
10+5-shot | 70.83 | 19.41 | 26.74 | 28.09 | 36.27
5+10-shot | 71.04 | 18.18 | 29.66 | 31.03 | 37.48
10+10-shot | 71.12 | 20.98 | 29.11 | 30.45 | 37.92
5+20-shot | 72.58 | 21.09 | 31.17 | 32.69 | 39.38
10+20-shot | 71.80 | 20.23 | 30.53 | 32.19 | 38.69
Table 5: Overall Experimental Results. Performance of GPT-3.5 Turbo across four datasets, with the best results highlighted in bold and the second-best underlined in each group. Method | CoNLL2003 | WNUT2017 | ACE2004 | ACE2005 | Avg.
---|---|---|---|---|---
Vanilla | 32.12 | 6.83 | 22.07 | 18.77 | 19.95
Standard ICL | 3-shot | 43.65 | 6.34 | 31.17 | 27.97 | 27.28
5-shot | 46.54 | 6.95 | 34.25 | 30.95 | 29.67
10-shot | 50.88 | 8.02 | 37.86 | 36.43 | 33.30
20-shot | 50.13 | 7.88 | 37.57 | 36.18 | 32.94
P-ICL${}_{\text{random}}$ | 5+3-shot | 47.10 | 7.96 | 36.10 | 32.25 | 30.85
10+3-shot | 46.94 | 7.51 | 34.81 | 32.68 | 30.49
5+5-shot | 50.41 | 8.94 | 37.81 | 34.84 | 33.00
10+5-shot | 49.31 | 7.51 | 35.69 | 35.89 | 32.10
5+10-shot | 54.32 | 8.62 | 39.69 | 36.05 | 34.67
10+10-shot | 54.92 | 7.77 | 39.03 | 38.15 | 34.97
5+20-shot | 58.63 | 8.53 | 40.44 | 39.66 | 36.82
10+20-shot | 58.44 | 8.29 | 40.35 | 39.60 | 36.67
P-ICL${}_{\text{bert}}$ | 5+3-shot | 49.96 | 9.45 | 36.04 | 32.45 | 31.97
10+3-shot | 49.02 | 8.83 | 36.88 | 32.59 | 31.83
5+5-shot | 52.11 | 9.92 | 39.68 | 35.54 | 34.31
10+5-shot | 51.37 | 9.92 | 39.38 | 36.46 | 34.28
5+10-shot | 57.62 | 9.35 | 41.05 | 38.14 | 36.54
10+10-shot | 56.41 | 10.30 | 40.98 | 39.12 | 36.70
5+20-shot | 61.70 | 8.40 | 42.32 | 40.67 | 38.27
10+20-shot | 60.40 | 10.74 | 42.51 | 40.99 | 38.66
Table 6: Overall Experimental Results. Performance of LLaMA-2-70B across four
datasets, with the best results highlighted in bold and the second-best
underlined in each group.