# Uncertainty Decomposition and Quantification for In-Context Learning of
Large Language Models

Chen Ling1, Xujiang Zhao2, Wei Cheng2, Yanchi Liu2, Yiyou Sun2,
Xuchao Zhang3, Mika Oishi4, Takao Osaki4, Katsushi Matsuda4,
Jie Ji1, Guangji Bai1, Liang Zhao1, Haifeng Chen3
1Emory University, 2NEC Labs America, 3Microsoft, 4NEC Corporation
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In-context learning has emerged as a groundbreaking ability of Large Language
Models (LLMs) and revolutionized various fields by providing a few task-
relevant demonstrations in the prompt. However, trustworthy issues with LLM’s
response, such as hallucination, have also been actively discussed. Existing
works have been devoted to quantifying the uncertainty in LLM’s response, but
they often overlook the complex nature of LLMs and the uniqueness of in-
context learning. In this work, we delve into the predictive uncertainty of
LLMs associated with in-context learning, highlighting that such uncertainties
may stem from both the provided demonstrations (aleatoric uncertainty) and
ambiguities tied to the model’s configurations (epistemic uncertainty). We
propose a novel formulation and corresponding estimation method to quantify
both types of uncertainties. The proposed method offers an unsupervised way to
understand the prediction of in-context learning in a plug-and-play fashion.
Extensive experiments are conducted to demonstrate the effectiveness of the
decomposition. The code and data are available at:
https://github.com/lingchen0331/UQ_ICL.

Uncertainty Decomposition and Quantification for In-Context Learning of Large
Language Models

Chen Ling1, Xujiang Zhao2, Wei Cheng2, Yanchi Liu2, Yiyou Sun2, Xuchao Zhang3,
Mika Oishi4, Takao Osaki4, Katsushi Matsuda4, Jie Ji1, Guangji Bai1, Liang
Zhao1, Haifeng Chen3 1Emory University, 2NEC Labs America, 3Microsoft, 4NEC
Corporation<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Large Language Models (LLMs) have revolutionized diverse domains by serving as
general task solvers, which can be largely attributed to the emerging
capability: in-context learning. By providing demonstrations of the task to
LLMs as part of the prompt, LLMs can quickly grasp the intention and make
corresponding responses to the particular task (Min et al., 2022). In this
paradigm, LLMs can quickly adapt to solve new tasks at inference time (without
any changes to their weights). Advanced LLMs, e.g., GPT-4 and LLaMA, have
achieved state-of-the-art results on LAMBADA (commonsense sentence
completion), TriviaQA (question answering) (Xie et al., 2021), and many tasks
in other domains (Ling et al., 2023b, a).

While in-context learning has achieved notable success, LLMs remain vulnerable
to well-known reliability issues like hallucination Rawte et al. (2023); Bai
et al. (2024). Uncertainty quantification has emerged as a popular strategy to
assess the reliability of LLM responses. In the past two years, numerous works
(Xiao et al., 2022; Lin et al., 2023; Ling et al., 2023c; Amayuelas et al.,
2023; Kuhn et al., 2023) have been proposed to quantify the uncertainty of LLM
response. These approaches could return a confidence score or directly compute
variance/entropy across multiple LLM responses; however, they often overlook
the complex nature of LLMs and their reliance on provided demonstrations in
in-context learning, so that existing methods may not provide insights into
the underlying causes or the interactions among different factors contributing
to uncertainty.

Figure 1: Uncertainty in LLM’s prediction can stem from two aspects: a)
Demonstration Quality: LLMs are likely to make wrong predictions if the
demonstrations are inappropriate; b) Model Configuration: different decoding
strategies (e.g., beam_search and top_k sampling) and their parameter settings
may return different predictions.

A natural question therefore arises: when LLM uses in-context learning to
predict a wrong answer with high uncertainty, can we indicate if it is caused
by the demonstration examples or by the model itself? Given LLM’s responses to
a particular task, it’s essential to decompose the uncertainty into its
primary sources to address the question. Specifically, Aleatoric Uncertainty
(AU) refers to variations in the data, often linked to the demonstration
examples. As shown in Figure 1 (a), LLM’s output can easily be disturbed by
inappropriate demonstrations since the provided demonstrations do not cover
all possible labels. The noise and potential ambiguity of these demonstrations
could bring uncertainty, which, in turn, may hinder the accuracy of the
response. In contrast, Epistemic Uncertainty (EU) stems from ambiguities
related to the model parameters or different configurations. As depicted in
Figure 1 (b), different decoding strategies (e.g., beam search and greedy
decoding) and their hyperparameter settings can have different decoding
results. Recognizing and quantifying the uncertainty from the model’s
perspective can also be critical in evaluating the generated responses, which
allows us to understand the model’s confidence level toward the task and make
necessary adjustments (e.g., choosing a more powerful model or conducting an
ensemble prediction).

Despite the strides made by existing works in understanding the total
uncertainty, the decomposition of uncertainty in the realm of in-context
learning remains under-explored. In this work, we propose a novel framework
for decomposing uncertainty in in-context learning to aleatoric and epistemic
components from the generated outputs. Our contributions are summarized as
follows.

  * •

Problem. We formulate the problem of uncertainty decomposition that extracts
epistemic and aleatoric uncertainties from the predictive distribution of LLMs
with in-context learning.

  * •

Method. We propose quantifying both aleatoric and epistemic uncertainty from
the mutual information perspective. A novel entropy-based estimation method is
also designed to handle the free-form outputs of LLMs.

  * •

Experiment. Extensive experiments are conducted to evaluate different aspects
of uncertainty, followed by specific applications and case studies to show how
two types of uncertainty influence the model’s performance.

##  2 Uncertainty Decomposition of In-context Learning

We first formulate the process of in-context learning as Bayesian Neural
Networks with latent variables. Based on the formulation, we propose to
decompose the predictive uncertainty into its epistemic and aleatoric
components from the mutual information perspective, followed by a novel way to
estimate both uncertainties based on the entropy of the prediction’s
distribution.

###  2.1 Background

LLMs are typically trained using maximum likelihood estimation on a large
corpus of text. The training goal is to maximize the likelihood of the
observed data under the model: $\mathcal{L}(\Theta)=\prod_{i\leq
N}p(\omega_{i}|\omega_{1},\omega_{2},\ldots,\omega_{i-1};\Theta)$, where each
$\omega_{i}\in\mathbf{x}$ is a token in a sentence
$\mathbf{x}=[\omega_{1},\ldots,\omega_{N}]$, and $\Theta$ denotes the set of
parameters.

#### Latent Concept.

From the Bayesian point of view, LLM’s in-context learning ability is obtained
by mapping the training token sequence $\mathbf{x}$ to a latent concept $z$
Xie et al. (2021). The concept $z$ is a latent variable sampled from a space
of concepts $\mathcal{Z}$, which defines a distribution over observed tokens
$\omega_{i}$ from a training context $\mathbf{x}$:

$p(\omega_{1},\ldots,\omega_{N})=\int_{z\in\mathcal{Z}}p(\omega_{1},\ldots,\omega_{N}|z)p(z)dz.$

The concept can be interpreted as various document-level statistics, such as
the general subject matter of the text, the structure/complexity of the text,
the overall emotional tone of the text, etc.

#### In-context Learning.

Given a list of independent and identically distributed (i.i.d.) in-context
demonstrations (contain both question and answer)
$[\mathbf{x}_{1},\ldots,\mathbf{x}_{T-1}]$ concatenated with a test question
(without the task answer) $\mathbf{x}_{T}$ as prompt. Each demonstration
$\mathbf{x}_{i}$ in the prompt is drawn as a sequence conditioned on the same
concept $z$ and describes the task to be learned. LLMs generate a response
$\mathbf{y}_{T}$ to the test question $\mathbf{x}_{T}$ based on the aggregated
prompt $\mathbf{x}_{1:T}$:

$\displaystyle
p(\mathbf{y}_{T}|\mathbf{x}_{1:T})=\int_{z\in\mathcal{Z}}p(\mathbf{y}_{T}|\mathbf{x}_{1:T},z)p(z|\mathbf{x}_{1:T})dz.$

In-context learning can be interpreted as locating a pre-existing concept $z$
based on the provided demonstrations $\mathbf{x}_{1:T-1}$, which is then
employed to tackle a new task $\mathbf{x}_{T}$. Including more high-quality
demonstrations within the prompt can help refine the focus on the relevant
concept, enabling its selection through the marginalization term
$p(z|\mathbf{x}_{1:T})$.

In this work, we focus on quantifying the predictive uncertainty of LLMs in
deterministic NLP tasks, such as text classification. Specifically, we address
tasks where the training dataset $\mathcal{D}=\\{\mathcal{X},\mathcal{Y}\\}$
consists of token sequences $\mathcal{X}=\\{\mathbf{x}\\}$ and their
corresponding target outputs $\mathcal{Y}=\\{\mathbf{y}\\}$. For LLMs, the
generation process is defined by the function
$\mathbf{y}=f(\mathbf{x},z;\Theta)$, where
$f:\mathcal{X}\times\mathcal{Z}\rightarrow\mathcal{Y}$ is a deterministic
function. The output $\mathbf{y}$ exhibits stochastic behavior, influenced by
the latent concept $z\sim p(z|\mathbf{x}_{1:T})$ and the model
parameters/configurations $\Theta$ (e.g., temperature, etc.).

###  2.2 Predictive Uncertainty Formulation of In-context Learning

We formulate the predictive distribution of in-context learning for predicting
$\mathbf{y}_{T}$ given few-shot demonstrations $\mathbf{x}_{1:T-1}$ and a test
case $\mathbf{x}_{T}$ as:

$\displaystyle p(\mathbf{y}_{T}|\mathbf{x}_{1:T})\approx$ $\displaystyle\int
p(\mathbf{y}_{T}|\Theta,\mathbf{x}_{1:T},z)$ (1) $\displaystyle\cdot
p(z|\mathbf{x}_{1:T})q(\Theta)dz\>d\Theta,$

where $p(\mathbf{y}_{T}|\Theta,\mathbf{x}_{1:T},z)$ is approximated by a
Bayesian Neural Network-based likelihood function
$\mathcal{N}(f(\mathbf{x}_{1:T},z),\bm{\Sigma})$, and $\bm{\Sigma}$ is the
covariance matrix contains the variances and covariances associated with LLM
parameters. $q(\Theta)$ is the approximated posterior of the LLM’s parameters
$\Theta$. Eq. (1) serves as an initial framework for generating predictions
based on input data and accompanying demonstrations:
$p(\mathbf{y}_{T}|\mathbf{x}_{1:T})$, which entangles different types of
uncertainties. We first present the overall pipeline of our uncertainty
quantification framework, followed by formulation on decomposing the total
uncertainty based on mutual information (Sec. 2.3) and a novel way to estimate
the uncertainty (Sec. 2.4). Note that LLMs can be categorized into white-box
and black-box models (Ling et al., 2023b) based on their transparency.
Quantifying mutual information involves accessing the probability of generated
tokens, which is not applicable to black-box LLMs. In this study, we also
provide a decomposition way from the variance perspective for black-box LLMs.
Due to the space limit, the variance-based decomposition can be found in
Appendix A.1.

Figure 2: Uncertainty Quantification of In-context Learning Pipeline: we want
to quantify the uncertainty that comes from 1) different in-context
demonstrations $\mathbf{x}_{1:T}$; and 2) different model configurations
$\Theta_{l}$.

#### Framework Pipeline.

In this work, we employ a Bayesian framework to quantify the predictive
uncertainty from LLMs, and the overall pipeline is visualized in Figure 2.
Specifically, the input $\mathbf{x}_{1:T}$ is composed of a test query
$\mathbf{x}_{T}$ and a set of demonstrations $\mathbf{x}_{1:T-1}$ sampled from
$\mathcal{X}$. By sampling different model parameters/configurations
$\Theta_{l}\sim q(\Theta)$, LLM can return different outputs
$\mathbf{y}^{l}_{T}\in[\mathbf{y}^{1}_{T},\cdots,\mathbf{y}^{L}_{T}]$ based on
the conditional probability $p(\mathbf{y}_{T}|\Theta_{l},\mathbf{x}_{1:T},z)$.
The collection of outputs $[\mathbf{y}^{1}_{T},\cdots,\mathbf{y}^{L}_{T}]$
records the total uncertainty regarding $\Theta_{l}$ and demonstrations
$\mathbf{x}_{1:T-1}$.

Figure 3: Framework of entropy-based uncertainty estimation, which consists of
1) generating $M$ sequences based on a set of $\mathbf{x}_{1:T-1}$; 2)
selecting token(s) that is relevant to the answer and extract the
probabilities; 3) aggregating the token probabilities of $M$ sequences into a
distribution of predicted labels; 4) iterating the process $L$ times
corresponding to $L$ different demonstration sets and form a probability
matrix $\mathcal{M}$, where the column denotes different demonstration sets
and the row denotes labels of the dataset.

###  2.3 Entropy-based Decomposition

As a widely adopted measure of uncertainty, entropy provides a quantifiable
and interpretable metric to assess the degree of confidence in the model’s
predictions (Malinin and Gales, 2020). Since white-box LLMs can return the
probability of each token in the generated sequence, it naturally makes
entropy-based uncertainty measures applicable uniformly across different types
of white-box LLMs.

#### Epistemic Uncertainty (EU).

Let $H(\cdot)$ be the differential entropy of a probability distribution, the
total uncertainty in Eq. (1) can be quantified as
$H\left(\mathbf{y}_{T}|\mathbf{x}_{1:T}\right)$, which entangles both
aleatoric (i.e., demonstration $\mathbf{x}_{1:T-1}$) and epistemic (i.e.,
model parameter $\Theta$) uncertainties. To estimate the EU, we condition Eq.
(1) on a specific realization of the model parameter $\Theta$, yielding
$p(\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta)=\int
p(\mathbf{y}_{T}|\mathbf{x}_{1:T},z,\Theta)p(z|\mathbf{x}_{1:T})dz$ with an
associated entropy $H(\mathbf{y}_{T}|\mathbf{x}_{1:T},z,\Theta)$. The expected
value of this entropy under different demonstration sets can be expressed as
$\mathbb{E}_{z}\left[H(\mathbf{y}_{T}|\mathbf{x}_{1:T},z,\Theta)\right]$,
which serves as a metric to quantify the EU in Eq. (1).

#### Aleatoric Uncertainty (AU).

In terms of AU, the randomness comes from different sets of demonstration
$\mathbf{x}_{1:T-1}$ and their corresponding latent concept $z$. To estimate
AU, we can quantify the mutual information between $\mathbf{y}_{T}$ and latent
concept $z$, which can often be leveraged as an evaluation metric of AU
(Wimmer et al., 2023). As we have already obtained the EU, AU can subsequently
be calculated as the discrepancy between the total uncertainty and the
epistemic uncertainty:

$\displaystyle I(\mathbf{y}_{T},z|\Theta)=$ $\displaystyle
H\left(\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta\right)$ (2)
$\displaystyle-\mathbb{E}_{z}\left[H(\mathbf{y}_{T}|\mathbf{x}_{1:T},z,\Theta)\right].$

The entropy $H\left(\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta\right)$ can be
approximately calculated as
$-\sum_{t}\left[p(\omega^{\mathbf{y}_{T}}_{t})\cdot\log
p\left(\omega^{\mathbf{y}_{T}}_{t}\right)\right]$, where
$p(\omega^{\mathbf{y}_{T}}_{t})$ represents the probability of each possible
next token $\omega^{\mathbf{y}_{T}}_{t}$ given the input prompt
$\mathbf{x}_{1:T}$. Therefore, the AU in Eq. (2) can be approximated by
sampling many $z$ (by sampling different sets of demonstrations) to obtain
different $\mathbf{y}_{T}$ conditioning on one set of parameters $\Theta$. The
group of $\mathbf{y}_{T}$ can then be used to approximate the respective
entropies for each group of demonstrations $\mathbf{x}_{1:T-1}$:

$\displaystyle I(\mathbf{y}_{T},z|\Theta)$ (3)
$\displaystyle=H\left(\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta\right)-\mathbb{E}_{z}\left[H(\mathbf{y}_{T}|\mathbf{x}_{1:T},z,\Theta)\right]$
$\displaystyle\approx\sum^{M\times
L}H(\mathbf{y}_{T})-\frac{1}{M}\sum^{M}_{m=1}\sum^{L}_{l=1}\left[H(\mathbf{y}_{T}^{\Theta_{m},l})\right],$

where $[\mathbf{y}^{\Theta_{m},l}_{T}]$ are obtained corresponding to
different demonstrations
$[\mathbf{x}^{1}_{1:T-1},\ldots,\mathbf{x}^{L}_{1:T-1}]$, and
$[\Theta_{1},\ldots,\Theta_{M}]$ are sampled from $q(\Theta)$. For some LLMs
that do not allow sampling different sets of parameters from the learned
$q(\Theta)$ as a standard Bayesian Neural Network, we can instead leverage
different decoding strategies (e.g., Beam-search or Multinomial sampling) in
order to enable stochastic output from LLMs. In addition, since calculating
the entropy $H\left(\mathbf{y}_{T}\right)$ entails to obtain the joint
probability of the generated tokens
$p(\mathbf{y}_{T})=(\omega^{\mathbf{y}_{T}}_{1},\ldots,\omega^{\mathbf{y}_{T}}_{T})$,
entropy-based method may only be applicable to white-box LLMs.

###  2.4 Entropy Approximation

In some cases, the generation of LLMs is free-form, which makes the
uncertainty estimation for in-context learning is still different from well-
studied classification models that have specific labels. Specifically, not
only may the LLM not always be able to return an expected answer, but the
generated sequence may also consist of placeholder tokens. Calculating the
entropy of the whole generated sequence would involve redundant tokens.
Therefore, in this work, we propose to approximate the entropy of the output
$H(\mathbf{y}_{T})$, and the process is summarized in Figure 3.

Given the probability distributions of the generated tokens
$p(\mathbf{y}_{T})$ for one set of demonstrations, we only select token(s)
$\omega^{\mathbf{y}_{T}}_{t}$ that directly answer the provided question.
Taking the text classification task as an example, LLM is asked to directly
output a numerical value standing for a predefined category (e.g., $0$:
Sadness, $1$: Joy, etc.). The probability of the token
$\omega^{\mathbf{y}_{T}}_{t}$ that represents the numerical value is then
leveraged to denote the overall distribution of $p(\mathbf{y}_{T})$. We
aggregate the answer probabilities from all $M$ decoded sequences and
transform them as an answer distribution (as shown in the top right corner in
Figure 3). After repeating the process $L$ times, where $L$ corresponds to $L$
different sets of demonstrations, we have a matrix $\mathcal{M}$ recording the
answer distributions of choosing different demonstrations and model
configurations (as shown in the lower right corner in Figure 3). The EU and AU
can then be approximated as:

$\displaystyle EU$ $\displaystyle=\frac{1}{L}\sum
H\left(\sigma(\mathcal{M}_{:,j})\right),$ $\displaystyle AU$
$\displaystyle=H\left(\sigma\left(\sum[\mathcal{M}_{:,j}]\right)\right)-\frac{1}{L}\sum
H\left(\sigma\left(\mathcal{M}_{:,j}\right)\right),$

where $\sigma(\cdot)$ normalizes the column $\mathcal{M}_{:,j}$ into a
probability distribution, and entropy $H(\cdot)$ can be calculated as
$-\sum_{k=1}^{K}\left(p(\mathcal{M}_{k,j})*\log(p(\mathcal{M}_{k,j}))\right)$
if the number of labels is $K$.

##  3 Related Works

#### Uncertainty Quantification and Decomposition.

Uncertainty quantification aims to measure the confidence of models’
predictions, which has drawn attention from various domains Zhao et al.
(2020); Ling et al. (2022); Malo et al. (2014). Measuring uncertainty is
essential in many real-world NLP applications where making a wrong prediction
with high confidence can be disastrous (e.g., assessing the confidence in a
translation or a generated piece of information). This is especially important
in foundation models since we do not have enough resources to finetune the
model Abdar et al. (2021). To better understand the uncertainty, the primary
focus is on understanding and categorizing the sources of uncertainty for
interpreting the models’ outputs more effectively. The output uncertainty can
typically be categorized into Aleatoric Uncertainty that arises from the
inherent noise in the data, and Epistemic Uncertainty that arises due to
inappropriate model architecture or overfitted/underfitted parameters.
Existing methods Chowdhary and Dupuis (2013); Depeweg et al. (2017); Malinin
and Gales (2020) have come up with various methods (e.g., Bayesian neural
network, Deep Ensembles, and Monte Carlo Dropout) to decompose the
uncertainty.

#### Uncertainty in Language Models.

Earlier works (Xiao and Wang, 2019; Desai and Durrett, 2020; Jiang et al.,
2021) on uncertainty in language models have focused on the calibration of
classifiers (e.g., applying dropout to the model parameters or leveraging
ensemble voting) to better assess the confidence of the generated output. When
it comes to the era of LLMs, multiple works (Xiao and Wang, 2021; Xiao et al.,
2022; Lin et al., 2022; Yu et al., 2022; Lin et al., 2023; Kuhn et al., 2023;
Fadeeva et al., 2023) have been proposed to measure the uncertainty of LLM’s
prediction in multiple aspects (e.g., lexical uncertainty, text uncertainty,
and semantic uncertainty) for multiple NLP tasks. Another line of works
Kadavath et al. (2022); Zhou et al. (2023); Amayuelas et al. (2023) instead
tries to analyze how to extract knowledge from a language model correctly and
self-evaluate the correctness with a confidence score. However, despite these
commendable efforts, existing methods still lack an effective way to directly
quantify and decompose the uncertainty inherent in the outputs of LLMs with
in-context learning.

##  4 Experiments

We evaluate the uncertainty decomposition procedure on realistic natural
language understanding problems. By comparing state-of-the-art uncertainty
quantification methods, we aim to examine what type of uncertainty is the most
promising indicator of high confidence for LLMs. In addition, we also provide
generalization analysis and two specific out-of-distribution detection
applications. Due to the space limit, extra experiments and experiment
settings are provided in the Appendix.

###  4.1 Experiment Setup

We evaluate the decomposed uncertainties on open-source LLMs with different
model sizes. We leverage LLaMA-2 (Touvron et al., 2023), which is the most
widely applied open LLM, with its $7$B, $13$B, and $70$B model sizes. The
primary experiments are conducted with LLaMA-2 models. In order to further
demonstrate the generalization ability of our method, we apply our uncertainty
quantification method on OPT-13B (Zhang et al., 2022).

Data. We consider different Natural Language Understanding tasks. 1) Sentiment
Analysis: Emotion (Saravia et al., 2018) contains $2,000$ test cases and six
classes; Financial Phrasebank (Financial) (Malo et al., 2014) contains $850$
financial news and three sentiment classes; Stanford Sentiment Treebank v2
(SST2) (Socher et al., 2013) consists of $872$ sentences from movie reviews
and two classes. 2) Linguistic Acceptability. The Corpus of Linguistic
Acceptability (COLA) (Warstadt et al., 2019) is about English acceptability
judgments, which has $1,040$ test cases and two classes. 3) Topic
Classification. AG_News (Zhang et al., 2015) contains $1,160$ test cases and
four classes.

|  Inference Model | ACC | Likelihood | Entropy | Semantic | Ours (EU) | Ours (AU)
---|---|---|---|---|---|---|---
AUPR | ROC | AUPR | ROC | AUPR | ROC | AUPR | ROC | AUPR | ROC
Emotion | LLaMA-7b-random | 0.407 | 0.423 | 0.426 | 0.448 | 0.501 | 0.598 | 0.607 | 0.688 | 0.667 | 0.625 | 0.579
LLaMA-7b-class | 0.411 | 0.562 | 0.423 | 0.657 | 0.538 | 0.697 | 0.653 | 0.745 | 0.696 | 0.691 | 0.601
LLaMA-13b-random | 0.501 | 0.597 | 0.613 | 0.584 | 0.503 | 0.612 | 0.625 | 0.645 | 0.681 | 0.559 | 0.585
LLaMA-13b-class | 0.533 | 0.641 | 0.578 | 0.593 | 0.554 | 0.652 | 0.701 | 0.622 | 0.686 | 0.526 | 0.599
LLaMA-70b-random | 0.584 | 0.512 | 0.462 | 0.491 | 0.452 | 0.657 | 0.696 | 0.667 | 0.713 | 0.531 | 0.663
LLaMA-70b-class | 0.592 | 0.537 | 0.484 | 0.469 | 0.442 | 0.622 | 0.689 | 0.659 | 0.721 | 0.612 | 0.693
Financial | LLaMA-7b-random | 0.379 | 0.821 | 0.532 | 0.728 | 0.438 | 0.715 | 0.624 | 0.731 | 0.672 | 0.669 | 0.582
LLaMA-7b-class | 0.397 | 0.593 | 0.505 | 0.548 | 0.362 | 0.732 | 0.699 | 0.803 | 0.711 | 0.753 | 0.589
LLaMA-13b-random | 0.476 | 0.894 | 0.571 | 0.652 | 0.463 | 0.705 | 0.545 | 0.718 | 0.512 | 0.729 | 0.573
LLaMA-13b-class | 0.477 | 0.752 | 0.594 | 0.692 | 0.531 | 0.694 | 0.543 | 0.765 | 0.610 | 0.758 | 0.592
LLaMA-70b-random | 0.530 | 0.816 | 0.509 | 0.754 | 0.493 | 0.679 | 0.688 | 0.779 | 0.754 | 0.734 | 0.642
LLaMA-70b-class | 0.537 | 0.668 | 0.469 | 0.623 | 0.439 | 0.774 | 0.649 | 0.893 | 0.804 | 0.739 | 0.659
SST-2 | LLaMA-7b-random | 0.856 | 0.149 | 0.636 | 0.135 | 0.587 | 0.244 | 0.593 | 0.286 | 0.683 | 0.205 | 0.702
LLaMA-7b-class | 0.897 | 0.230 | 0.666 | 0.196 | 0.579 | 0.253 | 0.577 | 0.248 | 0.701 | 0.302 | 0.673
LLaMA-13b-random | 0.866 | 0.268 | 0.472 | 0.204 | 0.467 | 0.355 | 0.712 | 0.314 | 0.677 | 0.326 | 0.816
LLaMA-13b-class | 0.928 | 0.178 | 0.425 | 0.113 | 0.439 | 0.343 | 0.631 | 0.397 | 0.836 | 0.367 | 0.639
LLaMA-70b-random | 0.932 | 0.091 | 0.597 | 0.137 | 0.475 | 0.258 | 0.565 | 0.318 | 0.764 | 0.298 | 0.571
LLaMA-70b-class | 0.938 | 0.132 | 0.552 | 0.185 | 0.531 | 0.312 | 0.679 | 0.331 | 0.851 | 0.362 | 0.697
COLA | LLaMA-7b-random | 0.599 | 0.388 | 0.557 | 0.329 | 0.443 | 0.358 | 0.502 | 0.416 | 0.562 | 0.377 | 0.517
LLaMA-7b-class | 0.639 | 0.392 | 0.523 | 0.381 | 0.478 | 0.425 | 0.526 | 0.473 | 0.587 | 0.401 | 0.506
LLaMA-13b-random | 0.652 | 0.389 | 0.498 | 0.287 | 0.512 | 0.433 | 0.562 | 0.469 | 0.572 | 0.488 | 0.565
LLaMA-13b-class | 0.649 | 0.412 | 0.418 | 0.342 | 0.517 | 0.426 | 0.548 | 0.456 | 0.568 | 0.523 | 0.641
LLaMA-70b-random | 0.826 | 0.481 | 0.599 | 0.312 | 0.471 | 0.372 | 0.625 | 0.317 | 0.716 | 0.329 | 0.676
LLaMA-70b-class | 0.852 | 0.357 | 0.612 | 0.397 | 0.588 | 0.397 | 0.613 | 0.389 | 0.727 | 0.425 | 0.682
AG_News | LLaMA-7b-random | 0.646 | 0.238 | 0.472 | 0.265 | 0.463 | 0.312 | 0.612 | 0.448 | 0.634 | 0.361 | 0.537
LLaMA-7b-class | 0.679 | 0.267 | 0.505 | 0.372 | 0.523 | 0.378 | 0.562 | 0.384 | 0.627 | 0.326 | 0.538
LLaMA-13b-random | 0.685 | 0.365 | 0.517 | 0.364 | 0.522 | 0.374 | 0.548 | 0.395 | 0.648 | 0.378 | 0.552
LLaMA-13b-class | 0.685 | 0.378 | 0.528 | 0.359 | 0.413 | 0.411 | 0.566 | 0.429 | 0.654 | 0.401 | 0.569
LLaMA-70b-random | 0.792 | 0.311 | 0.478 | 0.316 | 0.498 | 0.401 | 0.552 | 0.309 | 0.635 | 0.319 | 0.543
LLaMA-70b-class | 0.838 | 0.302 | 0.511 | 0.271 | 0.528 | 0.354 | 0.532 | 0.274 | 0.662 | 0.283 | 0.571

Table 1: The performance comparison on the misclassification rate based on the
uncertainty score from each approach. For each dataset, correct predictions
are labeled as $0$ and incorrect ones are labeled as $1$. We show the AUPR and
ROC (the higher the better) based on the uncertainty score and
misclassification rate with two types of demonstration selection strategy:
Random and Class as well as three LLaMA model sizes: 7B, 13B, and 70B.

Demonstration & Model Configuration Sampling. We evaluate each method on the
testing set of each dataset and choose two strategies to randomly sample in-
context learning demonstrations. 1) Random: we randomly sample demonstrations
(training instances with labels) from the training set regardless their
labels. 2) Class: we randomly sample demonstrations but ensure there is at
least one demonstration per label class. To generate various sequences based
on one set of demonstrations, we adopt Beam Search with beam width $=10$ to
approximate the sampling process of $\Theta\sim q(\Theta)$.

Comparison Methods. Our study also evaluates the following baseline
uncertainty estimation methods: 1) Likelihood-based Uncertainty (Likelihood)
(Malinin and Gales, 2020) calculates the sum of log probabilities of all
tokens generated from language models and normalizes it by the sequence
length. 2) Entropy-based Uncertainty (Entropy) (Xiao and Wang, 2019)
calculates the entropy of the probability distributions of the generated
tokens. 3) Semantic Uncertainty (Semantic) (Kuhn et al., 2023) is the most
advanced entropy-based uncertainty estimation method, which groups generated
sequences into clusters according to their semantic embeddings. The average
entropy across all groups is viewed as the uncertainty score.

Evaluation Metrics. We show the prediction accuracy of each dataset. In
addition, we leverage two standard metrics: the Area under Precision-Recall
Curve (AUPR) and AUROC (ROC) to evaluate the uncertainty. AUPR calculates the
area under the Precision-Recall curve. AP is high when both precision and
recall are high, and low when either of them is low across a range of
confidence thresholds. ROC represents the likelihood that a correct answer is
selected. An ideal ROC rating is $1$, whereas a random uncertainty estimate
would yield ROC $=0.5$.

###  4.2 Quantitative Analysis

We compare the performance of different methods in assessing the
misclassification samples based on their perspective uncertainty scores.
Intuitively, misclassified samples should have larger uncertainty scores. The
results are shown in Table 1. Note that our proposed method can decompose the
uncertainty into epistemic uncertainty (EU) and aleatoric uncertainty (AU), we
thus show the performance of EU and AU separately.

As shown in the table, in most cases, our proposed methods (EU and AU)
consistently show higher AUPR and ROC scores across all datasets, which
indicates a better performance in assessing misclassification samples based on
uncertainty scores. Moreover, we also draw some observations from the table.
1\. Class Sampling Strategy Proves Superior: The class sampling strategy
generally yields higher AUPR and ROC scores across datasets, which proves it
is more effective than random demonstration sampling. Class sampling ensures
that each class is represented in the sample and reduces sampling bias, which
is crucial in scenarios where the dataset might be imbalanced or where certain
classes are underrepresented. 2) Increasing Model Size Enhances Performance:
Larger models (moving from 7B to 70B) tend to have better performance in terms
of AUPR and ROC. Specifically, there’s a general trend of increasing AUPR and
ROC scores as model size increases from 7B to 13B to 70B for all comparison
methods. Some datasets and metrics do not strictly follow this trend. For
instance, in the Emotion dataset, the 70B model sometimes shows a slight
decrease in performance compared to the 13B model. The inconsistencies in
performance improvement with larger models, especially for EU, hint at the
complexity of uncertainty assessment in different contexts and datasets. 3\.
Treating all tokens equally can be harmful in uncertainty quantification: both
Likelihood and Entropy Uncertainty treat all tokens equally. However, some
tokens carry greater relevance and representativeness than others, owing to
the phenomenon of “linguistic redundancy”. However, most uncertainty
estimation methods treat all tokens with equal importance when estimating
uncertainty, disregarding these inherent generative inequalities.

(a) PR by OPT-13B (b) PR by LLaMA-2-13B (c) ROC by OPT-13B (d) ROC by
LLaMA-2-13B

Figure 4: The performance of misclassification rate using two backbone LLMs:
OPT-13B and LLaMA-2-13B on Emotion dataset. (a) and (b) demonstrate the
precision-recall curves (x-axis is the recall and y-axis is the precision) for
OPT-13B and LLaMA-2-13B; (c) and (d) demonstrate the ROC curve (x-axis is the
false positive rate and y-axis is the true positive rate) for OPT-13B and
LLaMA-2-13B, respectively.

###  4.3 Generalization Capability

In this work, we also show how our method performs when applied to different
LLMs. We compare the performance of misclassification rate when using OPT-13B
and LLaMA-2-13B. We compute the precision-recall (PR) curve and ROC curve
using two backbone LLMs on the Emotion dataset, and the results are shown in
Figure 4.

As shown in Figure 4, our method exhibits consistent trends across different
LLMs. The precision-recall curves of both uncertainties (Figure 4 (a) and 4
(b)) between the two methods are almost identical, and the model’s capability
between two LLMs is also reflected in the PR curves of EU. Furthermore, by
comparing Figure 4 (c) and 4 (d), the ROC curves of both LLMs show a similar
pattern, with the AUC scores not deviating significantly. Specifically, both
OPT-13B and LLaMA-2-13B exhibit the same Area Under ROC (AUROC) curve $=0.68$
for AU. Since LLaMA-2-13B is a more powerful LLM than OPT-13B, our method can
quantify that EU of LLaMA-2-13B (AUROC $=0.59$) is better than OPT-13B (AUROC
$=0.55$). This finding further supports our method maintains its performance
irrespective of the underlying model and its robust generalization capability.

###  4.4 Out-of-domain Demonstration Detection

Out-of-domain (OOD) demonstration refers to coupling a test instance with less
relevant or OOD demonstrations, potentially leading the model to be misled and
handle the test instance unreliably. In this study, we investigate whether
uncertainty scores can effectively distinguish between in-domain and OOD
demonstrations. In our labeling scheme, in-domain demonstrations are labeled
as $0$, while OOD demonstrations are labeled as $1$. AUPR and ROC analyses are
performed based on the labels and uncertainty scores, with results summarized
in Table 2. Specifically, we conduct experiments on the Emotion dataset,
involving two scenarios: in-domain demonstrations (sampled from its training
set) and relevant demonstrations (sampled from Finance Phrasebank, a three-
class sentiment analysis task). Additionally, we compare in-domain
demonstrations with complete OOD demonstrations (sampled from COLA, a binary
linguistic acceptability task).

| Semantic | Ours (EU) | Ours (AU)
---|---|---|---
AUPR | ROC | AUPR | ROC | AUPR | ROC
| Relevant
---
Demo
0.702 | 0.644 | 0.742 | 0.935 | 0.657 | 0.682
| OOD
---
Demo
0.698 | 0.712 | 0.784 | 0.941 | 0.773 | 0.607

Table 2: Out-of-domain demonstration detection conducted with LLAMA-2-13B on
Emotion Dataset.

As shown in Table 2, compared to the state-of-the-art Semantic Uncertainty and
the AU, the EU demonstrates the best indicator to detect both less relevant
and OOD demonstrations. Intuitively, the model’s predictions would be impacted
by irrelevant and OOD demonstrations and exhibit large variance. AU is less
effective than EU in detecting OOD demonstrations since the demonstrations
already have large inherent variability. Semantic Uncertainty instead cannot
really distinguish what is the root cause of the predictive uncertainty.

###  4.5 Semantic Out-of-distribution Detection

Semantic out-of-distribution (SOOD) detection refers to distinguishing test
samples with semantic shifts from the given demonstrations and the prompt. In
this study, we mask out a few classes and ask LLMs to classify test samples
into the rest of the classes. The method is expected to return a higher
uncertainty score of SOOD test samples. Specifically, we mask two classes 1:
sadness and 2: anger out of six classes from the Emotion dataset and ask LLM
to categorize a given test sample only into the rest four classes. The SOOD
samples are labeled as $1$ and in-distribution samples are labeled as $0$.
Results of AUPR and ROC are recorded in Table 3 in terms of different model
sizes.

| Semantic | Ours (EU) | Ours (AU)
---|---|---|---
AUPR | ROC | AUPR | ROC | AUPR | ROC
7B | 0.477 | 0.532 | 0.548 | 0.658 | 0.461 | 0.570
13B | 0.417 | 0.468 | 0.525 | 0.592 | 0.414 | 0.437

Table 3: Semantic out-of-distribution detection using LLAMA-2 7B and 13B on
Emotion Dataset.

As shown in the table, EU still performs the best as a better indicator to
recognize SOOD samples across different model sizes. SOOD samples are
semantically different from the provided demonstrations, and the task
description also masks out the correct class of these SOOD samples, leading to
higher uncertainty in the model’s predictions. Given the inappropriate task
description and demonstrations, AU may not necessarily perform better in the
presence of SOOD samples.

##  5 Conclusion

To better understand and quantify the inherent uncertainties associated with
LLM’s in-context learning, we provide a novel approach to decompose the
predictive uncertainty into its aleatoric and epistemic perspectives from the
Bayesian perspective. We also provide novel approximation methods to quantify
different uncertainties based on the decomposition. Extensive experiments are
conducted to verify the effectiveness and better performance of the proposed
method than others. We believe this research stands as a significant stride
toward harnessing the full potential of LLMs while being acutely aware of
their performance boundaries.

## Limitations

The proposed work aims at quantifying predictive uncertainty and decomposing
the value into its aleatoric and epistemic components. While we can achieve
the best result compared to other methods, the proposed framework may only be
applied in natural language understanding tasks (e.g., multiple-choice QA,
text classification, linguistics acceptability, etc.). The proposed
uncertainty estimation algorithm may have limited usage in quantifying
uncertainties of generation tasks since we cannot tell which part of the
generated sequence is semantically important.

## References

  * Abdar et al. (2021) Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. 2021.  A review of uncertainty quantification in deep learning: Techniques, applications and challenges.  _Information fusion_ , 76:243–297.
  * Amayuelas et al. (2023) Alfonso Amayuelas, Liangming Pan, Wenhu Chen, and William Wang. 2023.  Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models.  _arXiv preprint arXiv:2305.13712_.
  * Bai et al. (2024) Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, et al. 2024.  Beyond efficiency: A systematic survey of resource-efficient large language models.  _arXiv preprint arXiv:2401.00625_.
  * Chowdhary and Dupuis (2013) Kamaljit Chowdhary and Paul Dupuis. 2013.  Distinguishing and integrating aleatoric and epistemic variation in uncertainty quantification.  _ESAIM: Mathematical Modelling and Numerical Analysis-Modélisation Mathématique et Analyse Numérique_ , 47(3):635–662.
  * Depeweg et al. (2017) Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi-Velez, and Steffen Udluft. 2017.  Uncertainty decomposition in bayesian neural networks with latent variables.  _arXiv preprint arXiv:1706.08495_.
  * Desai and Durrett (2020) Shrey Desai and Greg Durrett. 2020.  Calibration of pre-trained transformers.  _arXiv preprint arXiv:2003.07892_.
  * Fadeeva et al. (2023) Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, et al. 2023.  Lm-polygraph: Uncertainty estimation for language models.  _arXiv preprint arXiv:2311.07383_.
  * Jiang et al. (2021) Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021.  How can we know when language models know? on the calibration of language models for question answering.  _Transactions of the Association for Computational Linguistics_ , 9:962–977.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022.  Language models (mostly) know what they know.  _arXiv preprint arXiv:2207.05221_.
  * Kuhn et al. (2023) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023.  Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.  _arXiv preprint arXiv:2302.09664_.
  * Lin et al. (2023) Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023.  Generating with confidence: Uncertainty quantification for black-box large language models.  _arXiv preprint arXiv:2305.19187_.
  * Lin et al. (2022) Zi Lin, Jeremiah Zhe Liu, and Jingbo Shang. 2022.  Towards collaborative neural-symbolic graph semantic parsing via uncertainty.  _Findings of the Association for Computational Linguistics: ACL 2022_.
  * Ling et al. (2022) Chen Ling, Junji Jiang, Junxiang Wang, and Zhao Liang. 2022.  Source localization of graph diffusion via variational autoencoders for graph inverse problems.  In _Proceedings of the 28th ACM SIGKDD_ , pages 1010–1020.
  * Ling et al. (2023a) Chen Ling, Xuchao Zhang, Xujiang Zhao, Yanchi Liu, Wei Cheng, Mika Oishi, Takao Osaki, Katsushi Matsuda, Haifeng Chen, and Liang Zhao. 2023a.  Open-ended commonsense reasoning with unrestricted answer candidates.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 8035–8047.
  * Ling et al. (2023b) Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, Tianjiao Zhao, et al. 2023b.  Domain specialization as the key to make large language models disruptive: A comprehensive survey.  _arXiv preprint arXiv:2305.18703_.
  * Ling et al. (2023c) Chen Ling, Xujiang Zhao, Xuchao Zhang, Yanchi Liu, Wei Cheng, Haoyu Wang, Zhengzhang Chen, Takao Osaki, Katsushi Matsuda, Haifeng Chen, et al. 2023c.  Improving open information extraction with large language models: A study on demonstration uncertainty.  _arXiv preprint arXiv:2309.03433_.
  * Malinin and Gales (2020) Andrey Malinin and Mark Gales. 2020.  Uncertainty estimation in autoregressive structured prediction.  _arXiv preprint arXiv:2002.07650_.
  * Malo et al. (2014) P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. 2014.  Good debt or bad debt: Detecting semantic orientations in economic texts.  _Journal of the Association for Information Science and Technology_ , 65.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work?  _arXiv preprint arXiv:2202.12837_.
  * Rawte et al. (2023) Vipula Rawte, Amit Sheth, and Amitava Das. 2023.  A survey of hallucination in large foundation models.  _arXiv preprint arXiv:2309.05922_.
  * Saravia et al. (2018) Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018.  CARER: Contextualized affect representations for emotion recognition.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3687–3697.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Wimmer et al. (2023) Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. 2023.  Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures?  In _Uncertainty in Artificial Intelligence_ , pages 2282–2292.
  * Xiao and Wang (2019) Yijun Xiao and William Yang Wang. 2019.  Quantifying uncertainties in natural language processing tasks.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 7322–7329.
  * Xiao and Wang (2021) Yijun Xiao and William Yang Wang. 2021.  On hallucination and predictive uncertainty in conditional language generation.  _arXiv preprint arXiv:2103.15025_.
  * Xiao et al. (2022) Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2022.  Uncertainty quantification with pre-trained language models: A large-scale empirical analysis.  _arXiv preprint arXiv:2210.04714_.
  * Xie et al. (2021) Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021.  An explanation of in-context learning as implicit bayesian inference.  _arXiv preprint arXiv:2111.02080_.
  * Yu et al. (2022) Jialin Yu, Alexandra I Cristea, Anoushka Harit, Zhongtian Sun, Olanrewaju Tahir Aduragba, Lei Shi, and Noura Al Moubayed. 2022.  Efficient uncertainty quantification for multilabel text classification.  In _2022 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  Opt: Open pre-trained transformer language models.
  * Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _NIPS_.
  * Zhao et al. (2020) Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. 2020.  Uncertainty aware semi-supervised learning on graph data.  _Advances in Neural Information Processing Systems_ , 33:12827–12836.
  * Zhou et al. (2023) Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023.  Navigating the grey area: Expressions of overconfidence and uncertainty in language models.  _arXiv preprint arXiv:2302.13439_.

##  Appendix A Appendix

###  A.1 Variance-based Decomposition

Alternatively, we can use the variance as a measure of uncertainty. Let
$\sigma^{2}(\cdot)$ compute the variance of a probability distribution, and
the total uncertainty present in Eq. (1) is then
$\sigma^{2}(\mathbf{y}_{T}|\mathbf{x}_{1:T})$. This quantity can then be
decomposed using the law of total variance:

$\displaystyle\sigma^{2}(\mathbf{y}_{T}|\mathbf{x}_{1:T})=$
$\displaystyle\sigma^{2}_{q(\Theta)}\left(\mathbb{E}[\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta]\right)$
(4)
$\displaystyle+\mathbb{E}_{q(\Theta)}\left[\sigma^{2}(\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta)\right].$

where $\mathbb{E}[\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta]$ and
$\sigma^{2}(\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta)$ are mean and variance of
$\mathbf{y}_{T}$ given $p\left(\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta\right)$.
$\sigma^{2}_{q(\Theta)}\left(\mathbb{E}[\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta]\right)$
represents the variance of
$\mathbb{E}[\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta]$ when $\Theta\sim
q(\Theta)$, which indicates the epistemic uncertainty since it ignores the
contribution of $z$. In contrast,
$\mathbb{E}_{q(\Theta)}\left[\sigma^{2}(\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta)\right]$
in Eq. (4) represents the aleatoric uncertainty since it denotes the average
value of $\sigma^{2}(\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta)$ with $\Theta\sim
p(\Theta)$ and ingores the contribution of $\Theta$ to $\mathbf{y}_{T}$. Note
that variance-based uncertainty decomposition does not involve the probability
of the generated tokens, which is applicable to black-box LLMs (e.g., GPT
models).

#### Variance Approximation.

In practice, when we are dealing with black-box LLMs (e.g., ChatGPT), there
are multiple hyperparameters (e.g., temperature and top_p) allowing to return
different responses. Specifically,
$[\mathbf{y}^{1}_{T},\ldots,\mathbf{y}^{L}_{T}]$ can be obtained through
querying the LLM with different demonstrations
$[\mathbf{x}^{1}_{1:T-1},\ldots,\mathbf{x}^{L}_{1:T-1}]$ $L$ times. The
different set of parameter configurations are denoted as
$[\Theta_{1},\ldots,\Theta_{M}]$. The
$\mathbb{E}[\mathbf{y}_{T}|\mathbf{x}_{1:T},\Theta]$ can then be calculated as
the expected model output given the input data and the model parameters
$\Theta$. Calculate the variance of this expectation with respect to a set of
model configurations over all sets of demonstrations gives the epistemic
uncertainty. The variance $\sigma^{2}(\mathbf{y}_{T})$ can also be obtained
given a set of few-shot demonstrations over all model parameters. Finally,
average this variance over the certain model configuration to obtain the
aleatoric uncertainty.

###  A.2 Dataset Description

Sentiment Analysis. 1) Emotion (Saravia et al., 2018) contains $2,000$ test
cases, where LLMs are asked to classify a given sentence with six categories:
sadness, joy, love, anger, fear, surprise. 2) Financial Phrasebank (Financial)
(Malo et al., 2014) contains $850$ test cases, where LLMs are asked to
classify a given financial news with three categories: negative, neutral,
positive. 3) Stanford Sentiment Treebank v2 (SST2) (Socher et al., 2013)
consists of $872$ sentences from movie reviews and human annotations of their
sentiment, where the language model is asked to predict the sentiment from two
classes: positive and negative.

Linguistic Acceptability. 1) The Corpus of Linguistic Acceptability (COLA)
(Warstadt et al., 2019) is about English acceptability judgments drawn from
books and journal articles on linguistic theory. Each example is a sequence of
words annotated with whether it is a grammatical English sentence, and there
are $1,040$ test cases in total.

Topic Classification. TC aims at categorizing the given sentence into
predefined topics. We adopt AG_News (Zhang et al., 2015) is a dataset that
collects more than $1$ million news articles, where LLMs are asked to classify
a given news into four categories: World, Sports, Business, and Sci/Tech.
There are $1,160$ test cases in total.

###  A.3 Experiment Setup

We conduct experiments primarily on llama-2-7b-chat-hf, llama-2-13b-chat-hf,
and llama-2-70b-chat-hf, where the model weights are downloaded from the
website111https://ai.meta.com/resources/models-and-libraries/llama-downloads/.
Since we cannot actually “sample” model weights as Bayesian Neural Networks,
in order to let LLMs return different outputs, we leverage Beam Search since
it considers multiple best options based on beam width using conditional
probability, which is better than the sub-optimal Greedy search. The beam
search is conducted with the beam size $10$ and the max number of new tokens
is set to be $16$ uniformly across all datasets. We choose a different number
of demonstrations (details are recorded in Table 4) to allow the LLM to
achieve the best performance on each dataset, and we sample demonstrations
four times uniformly across all datasets.

| Random | Class
---|---|---
Emotion | 6 | 1 per class
Financial | 6 | 2 per class
SST2 | 4 | 2 per class
COLA | 2 | 1 per class
AG_News | 4 | 1 per class

Table 4: Number of demonstrations selected in each dataset.

System Prompt |  | ### Below is an instruction that describes a task. Clearly follow the instruction and write a short
---
response to answer it.
Task Description |  | ### Instruction: Classify the sentiment in the following text based on the six categories:
---
[0: Sadness; 1: Joy, 2: Love; 3: Anger; 4: Fear, 5: Surprise]. Provide the
information in a
structured format WITHOUT additional comments, I just want the numerical label
for each text.
Demonstrations |  | ### Here are some examples:
---
Example 1: Sentence: {i didnt feel humiliated} Category: {0: Sadness}
Example 2: Sentence: {im grabbing a minute to post i feel greedy wrong}
Category: {3: anger}
Example 3: Sentence: {i have the feeling she was amused and delighted}
Category: {1: joy}
Example 4: Sentence: {i feel more superior dead chicken or grieving child}
Category: {1: joy}
Example 5: Sentence: {i get giddy over feeling elegant in a pencil skirt}
Category: {1: joy}
…
Test Query |  | ### Test
---
Sentence: {} Category:

Table 5: Prompt Template consists of four parts: 1) System Prompt aims at
providing a basic hint of the task; 2) Task Description provides some details
of the task, e.g., if it is a sentiment analysis task or how many labels are
there; 3) Few-shot Demonstrations are leveraged to give LLMs some basic
formats of how the returned responses can be constructed; and 4) Test Query is
the final test query that we want LLMs to classify/categorize, and the LLM is
only expected to return an exact answer to solve the given question.

###  A.4 Prompt Template

In this work, we uniformly apply the following prompt template for all
datasets. Take the Emotion dataset as an example, we summarize the prompt in
Table 5. Note that all datasets use the same template, small modifications are
made on the actual label information and different demonstration numbers of
different datasets.

###  A.5 Misclassification Rate with Out of Domain Demonstration

Out-of-domain in-context Demonstration refers to the test instance being
coupled with less relevant or out-of-domain demonstrations, which the model
may be misled and not handle the test instance reliably. In this work, we
analyze the misclassification rate of out-of-domain Demonstration in the
Emotion dataset (six-class sentiment analysis task) by providing LLMs with
relevant demonstrations (sampled from Finance Phrasebank which is a three-
class sentiment analysis task) and complete out-of-domain demonstrations
(sampled from COLA which is a binary linguistic acceptability task). We
conduct the task with two demonstration selection strategies, and the results
are provided in Table 6.

| LLaMA-13B-Random | LLaMA-13B-Class
---|---|---
| EU | AU | EU | AU
| Original
---
Demo
0.681 | 0.585 | 0.686 | 0.599
| Relevant
---
Demo
| 0.688
---
($+1.0$%)
| 0.541
---
($-7.5$%)
| 0.671
---
($-2.2$%)
| 0.524
---
($-12.5$%)
| OOD
---
Demo
| 0.671
---
($-1.4$%)
| 0.501
---
($-13.3$%)
| 0.673
---
($-1.8$%)
| 0.497
---
($-17.0$%)

Table 6: Comparison of AUROC in misclassificatin rate on Emotion dataset,
where “Original Demo” indicates we sample demonstrations from its original
training set, “Relevant Demo” indicates we sample demonstrations from Finance
Phrasebank Dataset (a relevant sentiment analysis task, and “OOD Demo”
indicates we sample demonstrations from an irrelevant dataset: COLA.

As shown in the table, changes in the performance of the EU are relatively
minor under all conditions, suggesting that the model is more stable or less
sensitive to the changes in demonstration data within this metric. In
contrast, the AU shows more significant fluctuations, which implies that the
AU is more sensitive to the quality and relevance of demonstration data. When
relevant demonstrations from the Finance Phrasebank sentiment analysis dataset
are used, there’s a slight improvement or a minor decrease in EU, but a
notable decrease in AU. This suggests that even relevant but not identical
data can confuse the model, especially for the AU. With out-of-domain
demonstrations from COLA, the model’s performance drops more significantly,
with the AU metric showing a dramatic decrease of up to 17.0%, which indicates
that the model struggles significantly when the demonstrations are not
relevant to the task it’s being tested on.

|  Testing Query:
---
I had stated to her the reason I feel so fearful is because I feel unsafe (4:
fear)
| Extracted
---
Predictions
EU | AU
LLaMA-2-7B |  | 1\. i felt anger when at the end of a telephone call (3: anger)
---
2\. i feel a little mellow today (1: joy)
3\. i don t feel particularly agitated (4: fear)
4\. i hate it when i feel fearful for absolutely no reason (4: fear)
5\. im updating my blog because i feel shitty (0: sadness)
| 0, 0, 0, 1, 3
---
4, 3, 4, 4, 4
0.171 | 0.372
| 1\. i am feeling outraged it shows everywhere (4: fear)
---
2\. i do feel insecure sometimes but who doesnt (4: fear)
3\. i start to feel emotional (0: sadness)
4\. i feel so cold a href http irish (3: anger)
5\. i feel i have to agree with her even though i can imagine
some rather unpleasant possible cases (0: sadness)
| 4, 4, 1, 3, 4
---
4, 4, 4, 5, 4
0.163 | 0.189
LLaMA-2-70B |  | 1\. i felt anger when at the end of a telephone call (3: anger)
---
2\. i feel a little mellow today (1: joy)
3\. i don t feel particularly agitated (4: fear)
4\. i hate it when i feel fearful for absolutely no reason (4: fear)
5\. im updating my blog because i feel shitty (0: sadness)
| 4, 3, 4, 3, 4
---
4, 4, 2, 4, 4
0.012 | 0.079
| 1\. i am feeling outraged it shows everywhere (4: fear)
---
2\. i do feel insecure sometimes but who doesnt (4: fear)
3\. i start to feel emotional (0: sadness)
4\. i feel so cold a href http irish (3: anger)
5\. i feel i have to agree with her even though i can imagine
some rather unpleasant possible cases (0: sadness)
| 4, 4, 4, 4, 4
---
4, 4, 4, 4, 4
0.004 | 0.009

Table 7: Case study on the actual EU and AU decomposed from the predictive
uncertainty

###  A.6 Case Study

Table 7 demonstrates the actual changes in AU and EU when presenting LLMs with
different sizes and different demonstrations. Given the test query is: I had
stated to her the reason I feel so fearful is that I feel unsafe with the
ground truth label is (4: fear), which is a sentence with a negative feeling.
For LLaMA-2-7B, by presenting LLMs with more diverse demonstrations
(containing both positive and negative sentences), the results would be more
diverse between different beam search returned sequences, leading to a
relatively higher AU than EU. For LLaMA-2-70B with a larger parameter space
and model capability, the EU and AU are significantly reduced, which indicates
the model is more confident in the generated output and the variation of data
may not influence much to the prediction.


# Discovery of $T_{c\bar{s}0}^{a}(2900)^{0,++}$ implies new charmed-strange
pentaquark system

Hong-Tao An1,2<EMAIL_ADDRESS>Zhan-Wei Liu1,2,3<EMAIL_ADDRESS>Fu-
Sheng Yu3,4,5<EMAIL_ADDRESS>Xiang Liu1,2,3<EMAIL_ADDRESS>1School of
Physical Science and Technology, Lanzhou University, Lanzhou 730000, China
2Research Center for Hadron and CSR Physics, Lanzhou University and Institute
of Modern Physics of CAS, Lanzhou 730000, China
3Lanzhou Center for Theoretical Physics, and Frontiers Science Center for Rare
Isotopes, Lanzhou University, Lanzhou 730000, China
4School of Nuclear Science and Technology, Lanzhou University, Lanzhou 730000,
China
5Center for High Energy Physics, Peking University, Beijing 100871, China

###### Abstract

Inspired by the very recently discovered tetraquark states
$T_{c\bar{s}0}^{a}(2900)^{0,++}$ from the LHCb Collaboration, we predict the
existence of new charmed-strange pentaquark system, $c\bar{s}nnn$, which is
closely connected to $c\bar{s}n\bar{n}$ by exchanging $\bar{n}$ into $nn$ with
$n=u,d$. Especially, it is suggested to experimentally search for the
predicted new pentaquark system via the weak decays of $B$ mesons or
$b$-baryons, with the support from the study of the mass spectrum and the
decay properties. The predicted new pentaquark system must attract extensive
attention from experimentalists and theorists when it construct the ”particle
zoo 2.0” in the near future.

Introduction.– The existence of multiquark states are proposed by Gell-Mann
and Zweig GellMann:1964nj ; Zweig:1981pd ; Zweig:1964jf  at the birth of quark
model. The discoveries of these particles make the hadron world much rich in
diversity. Their interesting properties will disclose the essence in the
nonperturbative region of quantum chromodynamics (QCD) Chen:2016qju ;
Richard:2016eis ; Hosaka:2016pey ; Brambilla:2019esw ; Chen:2022asf ;
Lebed:2016hpi .

Among these reported new hadronic states in the past twenty years, the
$D_{s0}(2317)^{+}$ cannot be passed into silence, since its observation
associated with the $X(3872)$ in 2003 opened a new era of exploring exotic
hadronic matter. The coupled-channel effect due to the $c\bar{s}n\bar{n}$
configuration was realized to solve the low-mass puzzle of $D_{s0}(2317)^{+}$
vanBeveren:2003kd ; Dai:2003yg . Here, for this higher state of charmed-
strange meson, a pair of $n\bar{n}$ ($n=u,d$) is pulled out from the vacuum,
which may renormalize the bare mass. The isospin scalar $D_{s0}(2317)^{+}$ was
observed in the $D_{s}^{+}\pi^{0}$ invariant spectrum in the channels of
$B^{+}\to\bar{D}^{0}D_{s}^{+}\pi^{0}$ and
$B^{0}\to\bar{D}^{-}D_{s}^{+}\pi^{0}$ BaBar:2003oey ; Besson:2003cp . The
isospin vector mesons with charm and strangeness are explicitly exotic states
and can couple to the $D_{s}\pi$ channel as well.

When we are going to enter the third decade since 2003, LHCb reported the
observation of two isospin vector resonances $T_{c\bar{s}0}^{a}(2900)^{++}$
and $T_{c\bar{s}0}^{a}(2900)^{0}$ in the $D_{s}^{+}\pi^{\pm}$ invariant
spectrum of the similar channels $B^{+}\to\bar{D}^{-}D_{s}^{+}\pi^{+}$ and
$B^{0}\to\bar{D}^{0}D_{s}^{+}\pi^{-}$ very recently TcsRept . The least quark
configurations of the $T_{c\bar{s}0}^{a}(2900)^{++}$ and
$T_{c\bar{s}0}^{a}(2900)^{0}$ are obviously $c\bar{s}u\bar{d}$ and
$c\bar{s}\bar{u}d$, respectively. There were some theoretical explorations of
the $c\bar{s}n\bar{n}$ tetraquarks ever before Chen:2017rhl ; Agaev:2022duz ;
Lu:2020qmp ; He:2020jna ; Azizi:2018mte ; Cheng:2020nho ; Albuquerque:2020ugi
; Guo:2021mja .

Standing at a new starting point, we should point out that the discoveries of
the $T_{c\bar{s}0}^{a}(2900)^{0,++}$ states help us study other similar
multiquark systems more reliably, since it provides the critical understanding
of the interactions among quarks. In some way, $nn$ is equivalent to $\bar{n}$
in the color space. Thus the $c\bar{s}nnn$ pentaquarks deserve a careful
examination since it is closely related to $c\bar{s}n\bar{n}$ with the
$\bar{n}$ replaced by $nn$. We will see in this work that the observation of
$T_{c\bar{s}0}^{a}(2900)^{0,++}$ implies the existence of the charmed-strange
pentaquarks, a new pentaquark system. The mass spectrum and decay properties
of the $c\bar{s}nnn$ pentaquarks are systematically studied. Especially, we
suggest several operational weak decay channels of $B$ mesons or $b$-baryons
to carry out the search for the predicted charmed-strange pentaquark system,
which is a crucial step of constructing the ”particle zoo 2.0”.

Mass spectrum of $c\bar{s}nnn$.–

Table 1:  The numerical results of the mass spectrum and decay properties of
the $c\bar{s}nnn$ pentaquark states. The physical meanings of overlaps and
$k\cdot|c_{i}^{2}|$ are given in the main text. The masses are all in units of
MeV. The kinetically forbidden decay channels are marked with “$\times$”. One
can roughly estimate the relative decay widths between different decay
processes of different initial pentaquark states with this table if neglecting
the $\gamma_{i}$ differences.

$c\bar{s}nnn$ Overlaps Values of $k\cdot|c_{i}^{2}|$ $nnn\bigotimes c\bar{s}$
$nnc\bigotimes n\bar{s}$ $nnn\bigotimes c\bar{s}$ $nnc\bigotimes n\bar{s}$
$I(J^{P})$ Mass $\Delta D_{s}^{*}$ $\Delta D_{s}$ $\Sigma^{*}_{c}K^{*}$
$\Sigma^{*}_{c}K$ $\Sigma_{c}K^{*}$ $\Sigma_{c}K$ $\Lambda_{c}K^{*}$
$\Lambda_{c}K$ $\Delta D_{s}^{*}$ $\Delta D_{s}$ $\Sigma^{*}_{c}K^{*}$
$\Sigma_{c}K^{*}$ $\Sigma^{*}_{c}K$ $\Sigma_{c}K$ $\Lambda_{c}K^{*}$
$\Lambda_{c}K$ $\frac{3}{2}(\frac{3}{2}^{-})$ 3530 -0.532 -0.301 0.488 -0.140
-0.250 155 66 97 32 17 3357 0.822 -0.408 -0.239 0.262 -0.195 96 82 $\times$ 5
41 3199 0.204 0.862 -0.034 0.366 0.218 $\times$ $\times$ $\times$ $\times$ 56
$\frac{3}{2}(\frac{1}{2}^{-})$ 3684 0.205 0.611 -0.238 -0.028 32 234 39 1 3546
0.670 0.124 0.523 -0.079 256 7 144 5 3247 0.714 0.137 0.202 -0.464 $\times$
$\times$ $\times$ 117 4 12 $ND_{s}^{*}$ $ND_{s}$ $ND_{s}^{*}$ $ND_{s}$ 4 12
$\frac{1}{2}(\frac{3}{2}^{-})$ 3456 -0.095 -0.816 -0.132 -0.183 0.000 7 166 13
12 $\times$ 3403 -0.026 0.037 -0.049 0.887 0.000 0.5 $\times$ 220 2 $\times$
3162 0.807 -0.188 -0.329 0.177 0.000 251 $\times$ $\times$ 40 $\times$ 2954
-0.583 -0.035 0.806 0.016 0.000 $\times$ $\times$ $\times$ $\times$ $\times$
$\frac{1}{2}(\frac{1}{2}^{-})$ 3425 0.118 0.054 0.539 0.518 0.092 0.555 0.055
10 2 43 88 6 180 3 3253 -0.575 -0.201 0.472 -0.563 -0.041 0.053 -0.310 174 28
$\times$ $\times$ 1 1 68 3116 0.461 -0.571 -0.267 -0.171 -0.404 -0.477 0.196
63 173 $\times$ $\times$ 65 $\times$ 22 2955 0.664 0.273 0.081 -0.101 -0.391
0.133 0.600 $\times$ 19 $\times$ $\times$ 13 $\times$ 143 2828 -0.054 0.745
-0.024 -0.076 0.672 0.002 -0.242 $\times$ $\times$ $\times$ $\times$ $\times$
$\times$ 12

The mass spectrum of the $c\bar{s}nnn$ pentaquark system is studied by
considering the chromoelectric interactions (CEI) and the chromomagnetic
interactions (CMI), which have been widely studied in various hadron systems
An:2021vwi ; An:2020vku ; An:2020jix ; Liu:2021gva ; Weng:2021ngd ;
Guo:2021yws ; Weng:2018mmf . The effective Hamiltonian for the masses of
$S$-wave hadrons can be described by

$\displaystyle H$ $\displaystyle=$
$\displaystyle\sum_{i}m_{i}+H_{\textrm{CEI}}+H_{\textrm{CMI}}$ (1)
$\displaystyle=$
$\displaystyle\sum_{i}m_{i}-\sum_{i<j}A_{ij}\vec{\lambda}_{i}\cdot\vec{\lambda}_{j}-\sum_{i<j}v_{ij}\vec{\lambda}_{i}\cdot\vec{\lambda}_{j}\vec{\sigma}_{i}\cdot\vec{\sigma}_{j}$
$\displaystyle=$ $\displaystyle-\frac{3}{4}\sum_{i<j}m_{ij}V^{\rm
C}_{ij}-\sum_{i<j}v_{ij}V^{\rm CMI}_{ij}.$

Here, the subscripts $i,j$ represent the quarks or antiquarks. $V^{\rm
C}_{ij}=\vec{\lambda}_{i}\cdot\vec{\lambda}_{j}$ and $V^{\rm
CMI}_{ij}=\vec{\lambda}_{i}\cdot\vec{\lambda}_{j}\vec{\sigma}_{i}\cdot\vec{\sigma}_{j}$
are the color and chromomagnetic interactions between quarks, respectively,
with $\sigma_{i}$ being the Pauli matrices and $\lambda_{i}$ the Gell-Mann
matrices. $\lambda_{i}$ is replaced by $-\lambda_{i}^{*}$ for the antiquarks.
The mass parameter of quark pairs is
$m_{ij}=\frac{1}{4}(m_{i}+m_{j})+\frac{4}{3}A_{ij}$, which contains the
effective quark masses $m_{i,j}$ and the color interaction strength $A_{ij}$.
The $v_{ij}$ is the effective coupling constant between the $i$-th and $j$-th
quarks, determining the mass gaps. The values of $m_{ij}$ and $v_{ij}$ about
the $c\bar{s}nnn$ system can be extracted from the conventional hadrons
including $p$, $\Delta$, $\Lambda_{c}$, $\Sigma_{c}^{(*)}$, $D_{s}^{(*)}$,
$K^{(*)}$, and the other $S$-wave baryons and mesons. The results are
$m_{nn}=182.2$ MeV, $m_{n\bar{s}}=154.0$ MeV, $m_{nc}=520.0$ MeV,
$m_{n\bar{c}}=519.0$ MeV, $v_{nn}=19.1$ MeV, $v_{n\bar{s}}=18.7$ MeV,
$v_{nc}=3.9$ MeV, and $v_{c\bar{s}}=6.7$ MeV An:2020vku ; Weng:2021ngd ;
Weng:2018mmf .

The corresponding total wave functions have to be constructed, in order to
investigate the mass spectrum of the $c\bar{s}nnn$ pentaquark states. The
total wave functions include the spatial part, the flavor part, the spin part
and the color part, i.e. $\psi_{\rm tot}=\psi_{\rm space}\otimes\psi_{\rm
flavor}\otimes\psi_{\rm spin}\otimes\psi_{\rm color}$. They should be fully
antisymmetric when exchanging identical quarks. The explicit expressions of
the effective Hamiltonian can be obtained as the similar wave functions of the
$qqqq\bar{Q}$ pentaquark system in Ref. An:2020vku . Using the above values of
parameters extracted by the conventional hadrons, the mass of a $J^{P}=0^{+}$
isovector $c\bar{s}n\bar{n}$ tetraquark states is given as 2863 MeV, which is
very close to the measured value of $M_{T^{a}_{c\bar{s}0}(2900)}=2908\pm 11\pm
20$ MeV TcsRept . It manifests the correctness of CEI and CMI model. The
central value of the mass of $T^{a}_{c\bar{s}0}(2900)$ can be obtained by
overall increasing the parameters $m_{ij}$ by 1.5$\%$.

Taking $T^{a}_{c\bar{s}0}(2900)$ as the benchmark, we predict the mass
spectrum of the $c\bar{s}nnn$ pentaquark states of $I(J^{P})=1/2(1/2^{-})$,
$1/2(3/2^{-})$, $3/2(1/2^{-})$ and $3/2(3/2^{-})$, which are shown in Table 1
and Fig. 1. The mass region is between 2828 MeV and 3684 MeV, which are
accessible in the weak decays of $b$-hadrons. The $c\bar{s}nnn$ states can be
easily recognized as an explicitly exotic candidate once observed, since they
have no quark-antiquark pair with the same flavor and thus are impossible to
be mixed with the ordinary baryons.

Figure 1:  Relative positions (units: MeV) for the $c\bar{s}nnn$ pentaquark
states labeled with horizontal solid lines. Here, the red and black lines
represent the pentaquark states with $I=3/2$ and $I=1/2$, respectively. The
dotted lines denote various baryon-meson thresholds. The superscripts
(subscript) of the labels, e.g.
$(\Sigma^{*}_{c}K^{*})^{5/2,3/2,1/2}_{3/2,1/2}$, represent the possible total
angular momenta (isospin) of the channels. The vertical solid lines with
arrows indicate the rearranged $S$-wave decay processes that are allowed to
occur. The orange lines correspond to the $nnn$-$c\bar{s}$ decay modes, while
the black lines for the $nnc$-$n\bar{s}$ decay modes. Every ratio between two
vertical solid lines with arrows represents the relative partial widths from
the left to the right decay processes.

Decays of $c\bar{s}nnn$.– The decay properties are another important aspect to
investigate the nature of exotic hadrons. The decays of the $c\bar{s}nnn$
pentaquarks can be studied using the above effective Hamiltonian and wave
functions as well. The overlap between a pentaquark state and a specific
baryon $\otimes$ meson state is defined by transforming the eigenvectors of
the pentaquark states into the baryon-meson configurations, such as
$c_{i}=\langle N\otimes D_{s}|P_{c\bar{s}n^{3}}(2828)\rangle$. The baryon and
meson components inside the pentaquarks can be divided into the $1\otimes 1$
components and the $8\otimes 8$ components in the color space. The $1\otimes
1$ components can be easily dissociated into an $S$-wave baryon and an
$S$-wave meson, which are OZI-superallowed decay processes. On the contrary,
the $8\otimes 8$ components can not fall apart if without the gluon exchange
force. Thus, we will only concentrate on the OZI-superallowed strong decay
processes. The values of overlaps are shown in Table. 1.

The partial width of a two-body $L$-wave decay reads as Weng:2019ynv ;
Weng:2020jao ; Weng:2021hje :

$\displaystyle\Gamma_{i}=\gamma_{i}\alpha\frac{k^{2L+1}}{m^{2L}}\,|c_{i}|^{2},$
(2)

where $\alpha$ is an effective coupling constant, $c_{i}$ is the overlap, $m$
is the mass of initial state, $k$ is the momentum of final state in the rest
frame of the initial state, and $\gamma_{i}$ is a factor depending on the
decay processes to be discussed later. Since $(k/m)^{2}$ is of the order of
$10^{-2}$ or even smaller, the contributions from higher partial-wave decays
would be suppressed. Thus, we only consider the $S$-wave strong decays, whose
widths are simplified as $\Gamma_{i}=\gamma_{i}\alpha k|c_{i}|^{2}$. The
values of $k\cdot|c_{i}|^{2}$ for each decay channel are shown in Table 1.

According to the obtained $c\bar{s}nnn$ mass spectrum in Table 1, we plot
their relative positions associated with all possible rearranged decay
channels in Fig. 1. It can be seen that the $c\bar{s}nnn$ system has 10
possible two-body strong decay channels. The angular momenta (isospins) of the
baryon-meson states are labeled by superscripts (subscripts) in the rearranged
decay channels. If both the spin and the isospin of an initial pentaquark
state are equal to the numbers in the superscript and subscript of a baryon-
meson state respectively, this pentaquark could decay into the corresponding
baryon-meson channel via the $S$-wave strong interaction.

All of the $I(J^{P})=1/2(1/2^{-})$ pentaquark states can decay into
$\Lambda_{c}K$. Except for the lowest lying state $\rm
P_{c\bar{s}n^{3}}(2828)$, all the other $1/2(1/2^{-})$ states can also decay
into $ND_{s}$ and $\Sigma_{c}K$. Similarly, all the $3/2(1/2^{-})$ states can
decay into $\Sigma_{c}K$, while all the $1/2(3/2^{-})$ and $3/2(3/2^{-})$
states except for $\rm P_{c\bar{s}n^{3}}(2954)$ can decay into
$\Sigma_{c}^{*}K$. Note that both the $\Sigma_{c}$ and $\Sigma_{c}^{*}$
totally decay into $\Lambda_{c}\pi$ which are easily reconstructed in
experiments. Therefore, most of the $c\bar{s}nnn$ pentaquark states can be
measured via either $\Lambda_{c}K$, $ND_{s}$, $\Sigma_{c}K$ or
$\Sigma_{c}^{*}K$. These decaying properties might be helpful for experimental
searches.

The $\rm P_{c\bar{s}n^{3}}(2954)$ state is below all the allowed rearranged
decay channels seen in Fig. 1 and Table 1. It can only decay into $ND_{s}$,
$\Lambda_{c}K$ or $\Sigma_{c}K$ via $D$-wave interaction, leading to a narrow
width. The $\rm P_{c\bar{s}n^{3}}(2828)$ state can only decay into
$\Lambda_{c}K$ with a small phase space, and thus is expected to be a relative
narrow state.

Most of the other pentaquark states have more than two decay channels. With
the values of $k\cdot|c_{i}|^{2}$ in Table. 1, we can compare the partial
decay widths if the $\gamma_{i}$ are known. The factor of $\gamma_{i}$ is a
quantity determined by the decay dynamics depending on the spatial wave
functions of the initial pentaquark and the final-state meson and baryon,
which are different for each decay process. The spatial wave functions of the
ground state pseudoscalar meson $D_{s}$ and the vector meson $D^{*}_{s}$ are
nearly the same in the quark model. The deviations between the spatial wave
functions of $\Sigma_{c}$ and $\Sigma^{*}_{c}$ could be neglected in the heavy
quark limit Weng:2019ynv . The relationships of $\gamma_{i}$ in some decay
channels can thus be taken approximately as $\gamma_{\Delta
D^{*}_{s}}=\gamma_{\Delta D_{s}}$, $\gamma_{ND^{*}_{s}}=\gamma_{ND_{s}}$,
$\gamma_{\Sigma_{c}K^{*}}=\gamma_{\Sigma^{*}_{c}K^{*}}$,
$\gamma_{\Sigma_{c}K}=\gamma_{\Sigma^{*}_{c}K}$. One can roughly estimate the
ratios of decay widths for one pentaquark state decaying into a pair of final
states relating to each other shown in one block of $k\cdot|c_{i}|^{2}$ in
Table. 1. For example, the $\rm P_{c\bar{s}n^{3}}(3357)$ state can decay into
both $\Delta D_{s}$ and $\Delta D^{*}_{s}$. Given by the values of
$k\cdot|c_{i}|^{2}$, the ratio of the partial widths is
$\Gamma_{P_{c\bar{s}n^{3}}(3357)\to\Delta
D_{s}}:\Gamma_{P_{c\bar{s}n^{3}}(3357)\to\Delta D^{*}_{s}}\approx 1:1$. Such
ratios of the relative branching fractions are provided between the vertical
lines of the relevant processes in the Fig. 1. These ratios can help to guide
for experimental searches, and to investigate the inner structures of the
pentaquark states by the future measurements.

Suggestions on the experimental searches.– In analogy to
$T_{c\bar{s}0}(2900)^{0,++}$, the charmed-strange pentaquark states can also
be observed in the decays of $b$ hadrons. The largest production processes are
among the transitions of $b\to c\bar{c}s$ and $b\to c\bar{u}d$ which are the
most Cabibbo-Kobayashi-Maskawa(CKM)-favored, with
$|V_{cb}V_{cs}^{*}|\approx|V_{cb}V_{ud}^{*}|\approx\lambda^{2}$ and the
Wolfenstein parameter $\lambda\approx 0.225$. We propose to search for the
charmed-strange pentaquark states via $B^{+}\to
P_{c\bar{s}uud}^{++}\bar{\Lambda}_{c}^{-}$ and $\Lambda_{b}^{0}\to
P_{c\bar{s}udd}^{+}K^{-}$, respectively. The decaying topological diagrams of
the above processes are given in Fig. 2. The branching fraction of
$\Lambda_{b}^{0}\to P_{c\bar{s}udd}^{+}K^{-}$ is expected to be at the order
of $10^{-5}$, similar to the ones of $\Lambda_{b}^{0}\to
P_{\psi}^{N}(4312)^{+}K^{-}$ and $\Lambda_{b}^{0}\to
P_{\psi}^{N}(4440)^{+}K^{-}$ with the same topological diagrams Chen:2020eyu
by just exchanging from the $b\to c\bar{c}s$ transition into the $b\to
c\bar{u}d$ and from the $u\bar{u}$ pair from vacuum into the $s\bar{s}$. The
branching fraction of $B^{+}\to P_{c\bar{s}uud}^{++}\bar{\Lambda}_{c}^{-}$ is
about one order of magnitude smaller due to the one more quark-antiquark pair
from the vacuum, thus at the order of $\mathcal{O}(10^{-6})$. Some other
processes are also suggested, such as $B^{+}\to P_{c\bar{s}uud}^{++}\bar{p}$
and $\bar{B}_{s}^{0}\to P_{c\bar{s}udd}^{+}\bar{p}$. Considering the
experimental conditions with charged particles preferred, the charmed-strange
pentaquark states can be reconstructed via $P_{c\bar{s}uud}^{++}\to
D_{s}^{+}p$ or $\Lambda_{c}^{+}K^{+}$, and
$P_{c\bar{s}udd}^{+}\to\Sigma_{c}^{0}K^{+}$ or $\Sigma_{c}^{*0}K^{+}$ and
subsequently into $(\Lambda_{c}^{+}\pi^{-})K^{+}$. Note that the above
$P_{c\bar{s}uud}^{++}$ and $P_{c\bar{s}udd}^{+}$ don’t represent any special
pentaquark state, but all the possible states with the corresponding quark
flavors. Considering the large data of $b$ hadrons at LHCb and the
observations of $T_{c\bar{s}0}(2900)^{0,++}$ and some other tetraquark and
pentaquark candidates in the decays of $B$ mesons or $b$-baryons, it is very
hopefully to observe the charmed-strange pentaquarks via the mentioned
processes.

---
Figure 2:  The production processes $B^{+}\to
P_{c\bar{s}uud}^{++}\bar{\Lambda}_{c}^{-}$ (top) and $\Lambda_{b}^{0}\to
P_{c\bar{s}udd}^{+}K^{-}$ (bottom) at the quark level.

Summary.– After the first observation of the scalar charmed-strange meson
$D_{s0}(2317)$ for around twenty years, the two new observed charmed-strange
tetraquark states $T_{c\bar{s}0}^{a}(2900)^{0}$ and
$T_{c\bar{s}0}^{a}(2900)^{++}$ are constructing the ”particle zoo 2.0”,
together with many other recent observed exotic states. In this work, we point
out that the discovery of $T_{c\bar{s}0}^{a}(2900)^{0,++}$ implies the
existence of their closely related charmed-strange pentaquarks $c\bar{s}nnn$,
a new pentaquark system. We have systematically studied the mass spectrum and
the decay properties of the $c\bar{s}nnn$ states. Some suggestions are given
for the experimental searches in the $b$-hadron decays, such as $B^{+}\to
P_{c\bar{s}uud}^{++}\bar{\Lambda}_{c}^{-}$ and $\Lambda_{b}^{0}\to
P_{c\bar{s}udd}^{+}K^{-}$ with the pentaquarks reconstructed by
$P_{c\bar{s}uud}^{++}\to D_{s}^{+}p$ or $\Lambda_{c}^{+}K^{+}$, and
$P_{c\bar{s}udd}^{+}\to\Sigma_{c}^{0}K^{+}$ or $\Sigma_{c}^{*0}K^{+}$. The
predicted new pentaquark system must attract extensive attention from
experimentalists and theorists in the near future.

This work is supported by the China National Funds for Distinguished Young
Scientists under Grant No. 11825503, National Key Research and Development
Program of China under Contract No. 2020YFA0406400, the 111 Project under
Grant No. B20063, CAS Interdisciplinary Innovation Team, and the National
Natural Science Foundation of China under Grant Nos. 12175091, 11975112,
11965016, and 12047501.

## References

  * (1) M. Gell-Mann, Phys. Lett. 8, 214-215 (1964).
  * (2) G. Zweig, Version 1, CERN-TH-401.
  * (3) G. Zweig, Version 2, CERN-TH-412.
  * (4) A. Hosaka, T. Iijima, K. Miyabayashi, Y. Sakai and S. Yasui, PTEP 2016, 062C01 (2016), [arXiv:1603.09229 [hep-ph]].
  * (5) H. X. Chen, W. Chen, X. Liu and S. L. Zhu, Phys. Rept. 639, 1-121 (2016), [arXiv:1601.02092 [hep-ph]].
  * (6) J. M. Richard, Few Body Syst. 57, 1185-1212 (2016), [arXiv:1606.08593 [hep-ph]].
  * (7) R. F. Lebed, R. E. Mitchell and E. S. Swanson, Prog. Part. Nucl. Phys. 93 (2017), 143-194, [arXiv:1610.04528 [hep-ph]].
  * (8) N. Brambilla, S. Eidelman, C. Hanhart, A. Nefediev, C. P. Shen, C. E. Thomas, A. Vairo and C. Z. Yuan, Phys. Rept. 873 (2020), 1-154, [arXiv:1907.07583 [hep-ex]].
  * (9) H. X. Chen, W. Chen, X. Liu, Y. R. Liu and S. L. Zhu, [arXiv:2204.02649 [hep-ex]].
  * (10) E. van Beveren and G. Rupp, Phys. Rev. Lett. 91, 012003 (2003).
  * (11) Y. B. Dai, C. S. Huang, C. Liu and S. L. Zhu, Phys. Rev. D 68, 114011 (2003).
  * (12) B. Aubert et al. [BaBar], Phys. Rev. Lett. 90 (2003), 242001, [arXiv:hep-ex/0304021 [hep-ex]].
  * (13) D. Besson et al. [CLEO Collaboration], Phys. Rev. D 68, 032002 (2003), [hep-ex/0305100].
  * (14) https://indico.cern.ch/event/1176505/.
  * (15) S. S. Agaev, K. Azizi, and H. Sundu, arXiv:2207.02648 [hep-ex].
  * (16) K. Azizi and U. Özdem, J. Phys. G 45 (2018) no.5, 055003, [arXiv:1802.07711 [hep-ph]].
  * (17) Q. F. Lü, D. Y. Chen and Y. B. Dong, Phys. Rev. D 102 (2020) no.7, 074021, [arXiv:2008.07340 [hep-ph]].
  * (18) X. G. He, W. Wang and R. Zhu, Eur. Phys. J. C 80, no.11, 1026 (2020), [arXiv:2008.07145 [hep-ph]].
  * (19) J. B. Cheng, S. Y. Li, Y. R. Liu, Y. N. Liu, Z. G. Si and T. Yao, Phys. Rev. D 101 (2020) no.11, 114017, [arXiv:2001.05287 [hep-ph]].
  * (20) R. M. Albuquerque, S. Narison, D. Rabetiarivony and G. Randriamanatrika, Nucl. Phys. A 1007 (2021), 122113, [arXiv:2008.13463 [hep-ph]].
  * (21) T. Guo, J. Li, J. Zhao and L. He, Phys. Rev. D 105, no.5, 054018 (2022), [arXiv:2108.06222 [hep-ph]].
  * (22) W. Chen, H. X. Chen, X. Liu, T. G. Steele and S. L. Zhu, Phys. Rev. D 95 (2017) no.11, 114005, [arXiv:1705.10088 [hep-ph]].
  * (23) H. T. An, K. Chen, Z. W. Liu and X. Liu, Phys. Rev. D 103 (2021) no.7, 074006, [arXiv:2012.12459 [hep-ph]].
  * (24) H. T. An, K. Chen, Z. W. Liu and X. Liu, Phys. Rev. D 103 (2021) no.11, 114027, [arXiv:2106.02837 [hep-ph]].
  * (25) Z. Liu, H. T. An, Z. W. Liu and X. Liu, Phys. Rev. D 105 (2022) no.3, 034006, [arXiv:2112.02510 [hep-ph]].
  * (26) T. Guo, J. Li, J. Zhao and L. He, Phys. Rev. D 105 (2022) no.1, 014021, [arXiv:2108.10462 [hep-ph]].
  * (27) X. Z. Weng, X. L. Chen and W. Z. Deng, Phys. Rev. D 97 (2018) no.5, 054008, [arXiv:1801.08644 [hep-ph]].
  * (28) X. Z. Weng, W. Z. Deng and S. L. Zhu, Phys. Rev. D 105 (2022) no.3, 034026, [arXiv:2109.05243 [hep-ph]].
  * (29) H. T. An, K. Chen and X. Liu, Phys. Rev. D 105 (2022) no.3, 034018, [arXiv:2010.05014 [hep-ph]].
  * (30) X. Z. Weng, X. L. Chen, W. Z. Deng and S. L. Zhu, Phys. Rev. D 100, no.1, 016014 (2019), [arXiv:1904.09891 [hep-ph]].
  * (31) X. Z. Weng, X. L. Chen, W. Z. Deng and S. L. Zhu, Phys. Rev. D 103, no.3, 034001 (2021), [arXiv:2010.05163 [hep-ph]].
  * (32) X. Z. Weng, W. Z. Deng and S. L. Zhu, Chin. Phys. C 46, no.1, 013102 (2022), [arXiv:2108.07242 [hep-ph]].
  * (33) Y. K. Chen, J. J. Han, Q. F. Lü, J. P. Wang and F. S. Yu, Eur. Phys. J. C 81 (2021) no.1, 71, [arXiv:2009.01182 [hep-ph]].