#  Understanding the Relationship between Prompts and Response Uncertainty in
Large Language Models

Ze Yu Zhang∗,1,2, Arun Verma∗,1, Finale Doshi-Velez3, Bryan Kian Hsiang Low1
1National University of Singapore 2Alibaba Group 3Harvard University
{zhan1130, arun<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) are widely used in decision-making, but their
reliability, especially in critical tasks like healthcare, is not well-
established. Therefore, understanding how LLMs reason and make decisions is
crucial for their safe deployment. This paper investigates how the uncertainty
of responses generated by LLMs relates to the information provided in the
input prompt. Leveraging the insight that LLMs learn to infer latent concepts
during pretraining, we propose a prompt-response concept model that explains
how LLMs generate responses and helps understand the relationship between
prompts and response uncertainty. We show that the uncertainty decreases as
the prompt’s informativeness increases, similar to epistemic uncertainty. Our
detailed experimental results on real datasets validate our proposed model.

Understanding the Relationship between Prompts and Response Uncertainty in
Large Language Models

Ze Yu Zhang∗,1,2, Arun Verma∗,1, Finale Doshi-Velez3, Bryan Kian Hsiang Low1
1National University of Singapore 2Alibaba Group 3Harvard University
{zhan1130, arun<EMAIL_ADDRESS><EMAIL_ADDRESS>

††footnotetext: ∗Equal contribution.

##  1 Introduction

Large language models (LLMs) have demonstrated impressive performance across a
variety of tasks Google (2023); OpenAI (2023); Zhao et al. (2023). This
success has led to their widespread adoption and significant involvement in
various decision-making applications, such as healthcare (Karabacak and
Margetis, 2023; Sallam, 2023; Yang et al., 2023), education (Xiao et al.,
2023), finance (Wu et al., 2023b), and law (Zhang et al., 2023a). However,
despite their rapid adoption, the reliability of LLMs in handling high-stakes
tasks has yet to be demonstrated Arkoudas (2023); Huang et al. (2023a). The
reliability is particularly critical in domains such as healthcare, where
model responses can have immediate and significant impacts on human behavior
and hence their well-being Ji et al. (2023). Therefore, understanding LLMs’
reasoning and decision-making processes and how they influence response
uncertainty is critical for their safe deployment.

To understand this importance, consider the mobile health (mHealth)
application in which machine learning algorithms are integrated to monitor
users’ health conditions and provide advice on daily activities Boursalie et
al. (2018); Trella et al. (2022, 2023). Providing suggestions that can
influence users’ health is a form of intervention in the human decision-
making. For LLMs to be suitable for such use cases, they should be accurate
and provide consistent intervention strategies, e.g., consider an LLM-powered
mHealth app that suggests physical therapy (PT) routines to a patient
recovering from surgery. The app’s goal is to ensure the patient adheres to
their PT regimen during rehabilitation despite the discomfort it may cause.
The app must provide not only good but consistent suggestions to encourage PT
adherence. Any inconsistent behaviors from the app could undermine any
progress made. Conversely, providing accurate and consistent responses helps
make the system more reliable and trustworthy Shin et al. (2022).

Traditionally, given a fixed model architecture, the main approach of making a
model better at its task is to search for better hyperparameters Bischl et al.
(2023), using a more suitable optimizer Hassan et al. (2023), or train it with
more/better data Simon et al. (2023). Due to the recent discovery of the
emergent capability of LLMs such as in-context learning (ICL) and chain-of-
thought (CoT) prompting, it is possible to improve the model responses by
guiding it with informative prompts having relevant instructions and exemplars
Dong et al. (2022); Wei et al. (2022); Kojima et al. (2022). Through these
techniques, the LLMs are able to more effectively leverage the relevant
information acquired from the training data to generate better responses, even
if the prompt itself does not directly reveal the ground truth Liu et al.
(2023); Sahoo et al. (2024). We claim the prompting technique is not only
helpful to this end but also plays a crucial role in determining the model
response uncertainty.

The response generated by LLMs is a series of tokens sampled from probability
vectors of tokens using various heuristics Brown et al. (2020); Radford et al.
(2018, 2019), such as beam search, nucleus sampling, and greedy decoding.
Typically, tokens with higher probabilities are chosen sequentially to produce
the final response. The response variations are controlled by LLM parameters
such as temperature ($T$), top-$k$, or top-$p$. While response variations
benefit creative tasks like poem and essay writing, they can be detrimental
for tasks requiring high reproducibility and consistency Ganguli et al.
(2022); Huang et al. (2023b). However, making LLMs generate deterministic
responses is not ideal, as users may vary in what responses suit them the most
Wu et al. (2023a). Hence, a better approach is needed to understand the
sources of response uncertainty and develop methods to reduce it naturally
rather than masking it by adjusting LLM parameters.

We attribute response uncertainty to two main factors: the LLM’s
hyperparameters (e.g., temperature) controlling the generated response’s
randomness and the input prompt’s informativeness (information about the
task). This paper focuses solely on the response uncertainty due to the input
prompt while keeping the LLM parameters fixed. Therefore, a natural question
arises: How is the amount of relevant information about a task in the input
prompt related to the uncertainty in the response generated by an LLM?

To answer this, we leverage the insight that LLMs implicitly learn to infer
latent concepts during pretraining (Xie et al., 2022; Hahn and Goyal, 2023;
Zhang et al., 2023b) and propose a prompt-response concept (PRC) model. Our
PRC model conceptualizes how an LLM generates responses based on given prompts
and helps understand the relationship between prompts and response uncertainty
by measuring response uncertainty for prompts with varying information about
the task. We provide theoretical results that show that the uncertainty of
responses generated by an LLM decreases as the informativeness of prompt
increases (i.e., having more information about the given task). We connect the
reducible response uncertainty and epistemic uncertainty and show that adding
relevant information to the prompt is a principled and effective method to
reduce this uncertainty. Finally, we corroborate the validity of our PRC model
via experiments and provide a simulation for a healthcare use case to
demonstrate the efficacy of our approach.

##  2 Prompt-Response Concept Model of Large Language Model

In this section, we first define what we mean by concept. We then use the
notion of concept to explain our proposed prompt-response concept model of
LLM. Finally, we provide theoretical results that explain the relationship
between the uncertainty of response generated by an LLM and the
informativeness of an input prompt.

The definition of a concept varies across fields, e.g., in philosophy, a
concept represents the fundamental unit of thought; in psychology, it is a
mental construct; in linguistics, it refers to the semantic units that words
or phrases represent; and in education, it denotes key ideas or principles. In
this paper, we define the concept as an abstraction derived from specific
instances or occurrences that share common characteristics Laurence and
Margolis (1999); Fodor (1998); Weiskopf (2009); Wilmont et al. (2013). To
understand the notion of concept, consider the following example of the
concept: Species, which includes a group of organisms that share common
biological traits. Furthermore, a concept can be expressed as a sequence with
semantic meaning (e.g., using natural language). Here, we use the term
‘semantic meaning’ (or semantically meaningful) to refer to something that
conveys information that is understandable and extractable by humans Hurford
et al. (2007). Consider another example of a concept: the personal bio,111The
personal bio example of the concept is adapted from wiki bio concept example
given in Xie et al. (2022). which consists of sentences giving information
about a person’s name, occupations, contributions, and other personal details.

Concept: Personal bio of Alan Turing Alan Turing was an English computer
scientist, mathematician, and cryptanalyst. He introduced the Turing machine,
which formalized the concepts of algorithms and computation, serving as a
foundational model for general-purpose computers. Turing is widely regarded as
the father of theoretical computer science. …

Using concepts instead of word-level or token-level patterns in text analysis
improves reasoning and question-answering abilities by focusing on higher-
level abstractions, which allows a better understanding of the relationships
between different sentences in the given text Bates (1995); Bogatyrev and
Samodurov (2016); Wang et al. (2024). As seen in the example above, explaining
a concept often involves multiple sentences, each contributing to a specific
and meaningful facet of information about the concept Piccinini and Scott
(2006). We refer to the information of each facet as a concept attribute,
e.g., the sentence, “Alan Turing was an English computer scientist,
mathematician, and cryptanalyst” provides information about the name,
nationality, and occupation of Alan Turing. Therefore, every concept is
completely characterized by all its attributes.

###  2.1 Prompt-Response Concept Model

Our aim is to understand how the input prompt’s informativeness is related to
the uncertainty in the responses generated by an LLM. To do so, we first
introduce notations representing different variables used in this section. Let
$\mathcal{X}$ denote the set of all prompts and $\mathcal{Y}$ denote the set
of all responses generated by an LLM $f$, where
$f:\mathcal{X}\rightarrow\mathcal{Y}$. Let $\mathcal{V}$ be the vocabulary
containing all unique tokens. For any prompt $x$ and response $y$, we have
$x\in\mathcal{V}^{|x|}$ and $y\in\mathcal{V}^{|y|}$, where $|\cdot|$ returns
the number of tokens in prompt/response. For a given prompt $x\in\mathcal{X}$,
the LLM $f$ generates a response $y\in\mathcal{Y}$ such that $y=f(x)$. Since
the response $y$ can vary each time the LLM generates it, we attribute these
response variations to two main factors: the LLM’s hyperparameters, such as
temperature $T$, top-$k$, or top-$p$, which control the randomness in the
generated tokens, and the informativeness (information about the task) of the
given prompt. This paper focuses solely on the latter aspect while keeping the
LLM’s hyperparameters fixed.

Building on the earlier works’ interpretation that LLMs implicitly learn to
infer latent concepts during pretraining (Xie et al., 2022; Hahn and Goyal,
2023; Zhang et al., 2023b), we propose the prompt-response concept (PRC) model
of LLM. This model conceptualizes how an LLM generates a response for a given
prompt, which will be used to understand the relationship between prompts and
the response uncertainty by measuring response uncertainty for prompts with
varying information. The PRC model has three main components (as shown in Fig.
1): prompt concept, response concept, and mapping functions.

Figure 1:  Prompt-Response concept model of LLM.

Prompt concept. Let $\Theta_{x}$ be the set of all concepts corresponding to
prompts in set $\mathcal{X}$. In the PRC model, we assume that each input
prompt $x\in\mathcal{X}$ corresponds to a concept. We refer to this concept as
the prompt concept $\theta_{x}\in\Theta_{x}$. Intuitively, an LLM recognizes
input tokens as semantically meaningful units that coherently describe an
attribute of some latent prompt concept. The attributes of a concept are
expressed through multiple semantically meaningful sentences. If multiple
sentences in the prompt cannot be combined to describe a single concept, the
LLM will treat them as representing different concepts. Our experimental
results in Fig. 2k show that adding semantically meaningful information from
different concepts can increase response uncertainty.

Response concept. Let $\Theta_{y}$ be the set of all concepts corresponding to
responses in set $\mathcal{Y}$. We refer to these concepts as the response
concept. The PRC model assumes that each response concept
$\theta_{y}\in\Theta_{y}$ corresponds to a response $y\in\mathcal{Y}$.

Mapping functions. To understand the relationship between input prompt,
intermediate concepts (i.e., prompt and response concept), and response, we
assume the LLM $f$ is a composition of three mapping functions: prompt-concept
mapping function $(g_{x})$, concept-concept mapping function $(g_{c})$, and
concept-response mapping function $(g_{y})$. Hence, we can represent the
response as $y=f(x)=g_{y}(g_{c}(g_{x}(x)))$, where the function $g_{x}$ maps
the input prompt to a prompt concept, the function $g_{c}$ maps the prompt
concept to a response concept, and finally, the function $g_{y}$ maps the
response concept to a response. For a well-pretrained LLM, its $g_{x}$ can
accurately find all the attributes in the given input prompt; for a well-
aligned LLM, its $g_{c}$ can accurately map from some $\theta_{x}$ to the most
relevant $\theta_{y}$.222For pretrained-only LLMs in the ICL setting, the
task-dependent $g_{c}$ is inferred on the fly from the exemplars in the input
prompt, as shown in Xie et al. (2022). Thus, we expect a better LLM to
generate responses with lower uncertainty and higher qualities, as
demonstrated by our experimental results in Fig. 2i and Fig. 2j.

To generate a response, an LLM maps the input prompt to a prompt concept,
which is then mapped to a corresponding response concept. Finally, the LLM
uses the response concept to generate the final response. When a prompt lacks
sufficient task-related information (i.e., it is less informative) due to
attributes being under-specified, we can expect higher variability in the
responses generated by the LLM because there potentially exists multiple
concepts that satisfy the attributes found in the input prompt, as
corroborated by our experimental results in Fig. 2b. To further understand
this relationship, we will formalize how the informativeness of prompts is
related to response uncertainty.

###  2.2 Relationship between Prompts and Response Uncertainty in LLMs

Let $\mathcal{X}_{\theta_{x}}\subset\mathcal{X}$ be the set of prompts with
the same semantic meaning333Multiple prompts can be generated from a single
prompt by paraphrasing it while preserving the original semantic meaning
associated with the prompt Kuhn et al. (2023). and contain all information of
the prompt concept $\theta_{x}$. Let
$\mathcal{X}_{s}\subset\mathcal{X}_{\theta_{x}}$ be the set of prompts with
the same semantic meaning $s$ and only contain partial information about the
prompt concept $\theta_{x}$. We assume
$\mathcal{A}_{\theta}=\\{a_{\theta,1},\ldots,a_{\theta,m}\\}$ represents the
set of all the attributes of a concept $\theta$ and each attribute can be
perfectly expressed by some semantically meaningful sequence of tokens. We use
the notation $x_{1}\prec_{\theta_{x}}x_{2}$ to indicate that prompt $x_{1}$
contains less information about prompt concept $\theta_{x}$ than the prompt
$x_{2}$ (or the prompt $x_{2}$ is more informative than the prompt $x_{1}$).
Since $\mathcal{X}_{s}\subset\mathcal{X}_{\theta_{x}}$, any prompt from the
set $\mathcal{X}_{s}$ contains less information about prompt concept
$\theta_{x}$ than any prompt from the set $\mathcal{X}_{\theta_{x}}$.

Let $Z_{c}$ be a random variable denoting concept (where $c=x$ for prompt
concept and $c=y$ for response concept) and $X_{s}$ be a random variable
representing a prompt having semantic meaning $s$. Here, the randomness in
$Z_{c}$ is due to a less informative prompt, which leaves more space for
interpretation or variation in the possible concepts that LLM can map. In
contrast, the randomness in $X_{s}$ is due to the ability of different prompts
to represent the same semantic meaning.

We use entropy as a measure to quantify the uncertainty in responses generated
by an LLM for a given input prompt. Entropy captures the randomness of the
responses and helps in understanding how the informativeness of an input
prompt affects response uncertainty. Let $Y$ be a random variable representing
the response. The randomness in $Y$ can be due to less informative prompts and
the ability of different responses to correspond to the same response concept
(i.e., have the same semantic meaning). For a given prompt $x$, we define
entropy of $Y$ as follows:

$\text{H}\left(Y|x\right)=-\sum_{y\in\mathcal{Y}}p(y|x)\log_{2}p(y|x),$

where $p(y|x)$ is the conditional distribution of the responses generated for
a prompt. Intuitively, a highly informative prompt corresponds to specific
intermediate concepts, which leads to the generation of responses with less
variability and, hence, smaller entropy of $Y$. The conditional distribution
$p(y|x)$ represents the posterior predictive distribution, which marginalizes
all intermediate concepts (prompt and response) and is given as follows:

$\displaystyle
p(y|x)=\int_{\theta_{y}}p(y|\theta_{y},x)p(\theta_{y}|x)d\theta_{y}$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\
=\int_{\theta_{y}}\int_{\theta_{x}}p(y|\theta_{y},x)p(\theta_{y}|\theta_{x},x)p(\theta_{x}|x)d\theta_{y}d\theta_{x}$
$\displaystyle\leavevmode\nobreak\
=\int_{\theta_{r},\theta_{p}}p(y|\theta_{r},x)p(\theta_{r}|\theta_{p},x)p(\theta_{p}|x)d\theta_{r}d\theta_{p}.$

The first equality follows from conditioning response with respect to the
response concept, and the second equality follows from

$p(\theta_{y}|x)=\int_{\theta_{x}}p(\theta_{y}|\theta_{x},x)p(\theta_{x}|x)d\theta_{x}.$

If $p(\theta_{c}|x)$ (where $c=\\{x,y\\}$) concentrates on a specific concept
with a more informative prompt, the LLM learns effectively via
marginalization. More concretely, our PRC model assumes the LLM achieves this
by extracting the attributes from the input prompt $x$ and matching it to the
right prompt and response concepts. Furthermore, when the prompt has the
information about all attributes (i.e., perfect prompt), it is enough to
completely characterize the concept (Proposition 1). If an LLM extracts all
information about the desired concept from a perfect prompt, the remaining
uncertainty in responses is due to the representation of the response concept
via different responses (i.e., semantically equivalent), which is the
irreducible uncertainty. This behavior implies that the LLM implicitly
performs Bayesian inference, which is also observed in ICL (Xie et al., 2022).

###  2.3 Theoretical Results

We need the following assumption under which our theoretical results hold.

######  Assumption 1.

An LLM know the mapping functions used in PRC model, i.e., $g_{x}$, $g_{c}$,
and $g_{y}$.

This assumption states that LLM has perfectly learned the mapping functions
used in our proposed PRC model. While this assumption may not hold in
practice, we can expect that a better LLM has good estimates of these mapping
functions, which lead to lower uncertainty and better qualities of the model
responses, as corroborated by our experimental results shown in Fig. 2i and
Fig. 2j.

Next, we present our first result, which shows the relationship between
concept uncertainty and informativeness of prompts.

######  Lemma 1.

Let 1 hold. For $\theta_{1},\theta_{2}\in\Theta_{x}$,
$\mathcal{X}_{\theta_{1}}\cap\mathcal{X}_{\theta_{2}}=\emptyset$ if
$\theta_{1}\neq\theta_{2}$. Furthermore,
$\text{H}\left(Z_{x}|X_{\theta_{x}}\right)=0$.

The proof of Lemma 1 and other missing proofs are given in Appendix A. The
first part of this result implies that prompts fully describing two different
concepts can not have the same semantic meaning, i.e., no two concepts share
exactly the same semantic description. In other words, the prompts that fully
describe two different concepts can not have the same semantic meaning. The
second part of the result implies that there is no randomness in the prompt
concept if all the information needed to respond to the task is contained in
the prompt. Our next result shows the relationship between concept uncertainty
and informativeness of prompts.

######  Proposition 1.

Let 1 hold. Then, $\text{H}\left(Z_{x}|X_{s}\right)$ strictly decreases as
$X_{s}$ represents more informative prompts, i.e., more relevant information
about the concept is included in the prompt.

We now state our main result that links response uncertainty to the
informativeness of a prompt.

######  Theorem 1.

Let 1 hold. Then, $\text{H}\left(Z_{y}|X_{s}\right)$ strictly decreases as
$X_{s}$ represents more informative prompts. Furthermore,
$\text{H}\left(Y|X_{s}\right)$ converges to $\text{H}\left(Y|Z_{y}\right)$.

These results suggest that as the prompt’s informativeness increases, the
response uncertainty due to the uncertainty in the response concept decreases.
Furthermore, when sufficient information is provided in a prompt, there will
not be any uncertainty due to the uncertainty in the concept. The leftover
randomness in responses is due to the ability of different responses to convey
the same semantic meaning and hence are semantic equivalent Kuhn et al.
(2023).

###  2.4 Concept Uncertainty as Epistemic Uncertainty

In machine learning literature, epistemic uncertainty is typically reduced by
incorporating additional information, such as using a better model and
additional training data Hüllermeier and Waegeman (2021); Lahlou et al.
(2021); Senge et al. (2014); Shaker and Hüllermeier (2020); Valdenegro-Toro
and Mori (2022); Der Kiureghian and Ditlevsen (2009). In Proposition 1,
$\text{H}\left(Z_{c}|X_{s}\right)$ represents the epistemic uncertainty in
latent concepts.444In the context of Kuhn et al. (2023), it is called the
semantic entropy. In this paper, we study it through the lens of uncertainty
reduction. We have demonstrated that $\text{H}\left(Z_{c}|X_{s}\right)$ is
strictly reduced with an informative prompt. Therefore, increasing the
information about the concept in a prompt can lead to more reliable and
consistent responses by reducing the epistemic uncertainty in the latent
concept Hüllermeier and Waegeman (2021). When the prompt perfectly captures
the desired concept, the posterior distribution of the concept given prompt
converges to the desired concept; the remaining uncertainty is irreducible due
to the semantic equivalence of numerous ways of characterizing the same
concept. In other words, given a prompt, the uncertainty due to semantic
equivalence is part of aleatoric uncertainty Hüllermeier and Waegeman (2021).
Note that also due to semantic equivalence, this uncertainty is not
detrimental in general for the purpose of getting the desired information, but
some sequences are preferred more due to their frequent occurrences in the
training data (Zhao et al., 2024). However, if the prompt contains sentences
that are irrelevant to the task at hand (i.e., the more information provided
is irrelevant), the response uncertainty can increase, as demonstrated in Fig.
2l.

##  3 Experiments

To validate our proposed prompt-response concept model of LLM, we empirically
demonstrate different aspects of our proposed model in different settings
whose details are as follows.

For instruction-fine-tuned LLMs, their input prompts usually are in the form
of some tasks from the user (i.e., ‘explain to me why the sky is blue’). In
our experiments, we treat a relatively simple task as a ‘concept’ and a
complex task as a composition of multiple concepts. From Section 3.1 to
Section 3.3, we focus on the simple task setting and discuss the complex task
setting in Section 3.4.

(a) Prompt’s Informativeness

(b) Varying Attributes

(c) $\text{PE}(Y|x)$ vs Prompt’s Size

(d) Noisy Masking (Short)

(e) Noisy Masking (Long)

(f) Noisy Prepend (Long)

(g) Noisy Append (Long)

(h) Noisy Insertion (Long)

(i) Accuracy of MCQs

(j) Empirical Entropy of MCQs

(k) Compositionality

(l) Irrelevant Information

Figure 2:  Top row: (a) Total Standard Deviation ($M(x)$) for input prompts
with different levels of informativeness. (b): A more granular-level
examination of (a) by gradually increasing the number of attributes for the
same task/concept. (c): Normalized Predictive Entropy ($\text{PE}(Y|x)$) for
short and long prompts. (d): Noisy masking on short prompt. Middle row: (e):
Noisy masking on long prompt. (f): Noisy prepend on long prompt. (g): Noisy
append on long prompt. (h): Noisy insertion on long prompt. As we observe in
Figures (d) - (h), more noisy input prompt has higher response uncertainty.
Bottom row: (i): Accuracy of MCQs. (j): Uncertainty over MCQs. There is a
clear and strong negative correlation between the accuracy and uncertainty.
(k):Total Standard Deviation increases with respect to increasing number of
sub-tasks/concepts. (l): Additional irrelevant information does not reduce
response uncertainty.

###  3.1 Relationship between Informativeness of the Prompt and Response
Uncertainty

We begin by assessing the response uncertainty of LLMs through the generation
of responses using increasingly longer prompts with more relevant information
(see Section B.1 for the prompts used). For each prompt, we generate $100$
responses from LLM with uncalibrated logits ($T=1$) and project them into the
embedding space as single points using the OpenAI "text-embedding-ada-002"
model. To quantify the uncertainty in the generated responses for a given
prompt, we use the total standard deviation, denoted as $M(x)$, defined as
$\sqrt{\text{Tr}(\Sigma)}$, where $\Sigma$ represents the covariance matrix of
the embedding vectors of responses ${y_{1},y_{2},\ldots,y_{100}}$. For LLMs,
the dispersion of their responses in the embedding space is a good indicator
of how much they differ in their semantic meaning Lin et al. (2023); Petukhova
et al. (2024). Therefore, $M(x)$ is an effective metric for how much
uncertainty there is in the model responses. It is noteworthy that
$\text{Tr}(\Sigma)$ is also referred to as total variation, serving as a
lightweight measure of dispersion in the data Ferrer-Riquelme (2009). This
metric is applicable for responses generated from both black-box and white-box
LLMs, as it does not require access to logits.

As illustrated in Fig. 2a, longer prompts with more task-related information
resulted in reduced response uncertainty. In the extreme case of an empty
input prompt (shown as blue bar), the responses vary greatly in semantic
meaning (see Section B.2).555We did this experiment in late 2023; since then,
the behavior of the GPT-4-0613 checkpoint has changed, possibly due to
internal fine-tuning or guard-railing by OpenAI. Our results show that the
response uncertainty decreases as the informativeness of the input prompt
increases. For a detailed examination of the relationship between input
prompt’s informativeness and response uncertainty, we focus on the
aforementioned mHealth intervention task, and use prompts with different
numbers of attributes for the same task. As shown in Fig. 2b, that having more
attributes present in a prompt generally resulted in smaller response
uncertainty. The lack of observable trend from bar 2 to bar 3 and from bar 4
to bar 5 could be due to adding redundant information in the input prompt (see
Section B.3 for details of all prompts and LLM model used). We also run an
additional experiment with two prompts containing different amounts of
information for a given task (see Section B.4 for short prompt and long
prompt) in which different uncertainty measure is used. We generate $N$
responses respective prompts and calculate the sequence-level normalized
predictive entropy (PE) Wagle et al. (2023):
$\text{PE}(Y|x)=-\frac{1}{N}\sum_{y}p(y|x)\log(p(y|x))$, where $Y$ is the
random response and the sum is taken over $N=3000$ generated responses.666We
model the entire generated response as the random variable instead of modeling
it on the token level as in Wagle et al. (2023). This approach can also be
considered as the Monte Carlo estimate of uncertainty score (Lin et al.,
2023). As we observed in Fig. 2c, the responses generated with the longer
prompt containing more relevant information have consistently smaller PE than
those from the shorter prompt as the sample size grows.777Calculating
$\text{PE}(Y|x)$ requires white-box model access to the logits and hence is
done on meta-llama/Llama-2-7b-chat-hf from Huggingface Touvron et al. (2023).

###  3.2 Noisy Prompts

The transformer’s self-attention mechanism allows the removal of a small
fraction of tokens without altering the semantic meaning by simply treating
them as irrelevant tokens Kim et al. (2017); Lin et al. (2017); Vaswani et al.
(2017). Therefore, LLMs are robust to noisy tokens in prompts when the noise
level is low (e.g., a few misspelled words). It is relatively easy to
determine the correct word based on the context (i.e., the entire prompt). If
the prompt can be accurately reconstructed, the same level of uncertainty
reduction can be achieved. However, if the prompt is severely corrupted, it
becomes less informative, leading to increased response uncertainty.

For the short and long input prompts given in Section B.5, we iteratively
select an increasing fraction of randomly selected tokens from them
respectively, replacing them with space tokens (i.e., token masking) or random
tokens (i.e., token corruption). We set the temperature to $1$ and sampled
$100$ responses from the model. We used $5$ different random seeds to choose
which tokens to mask, replacing them with space tokens. As the fraction of
masked tokens increased, we added new randomly selected tokens in the
previously masked tokens to ensure that randomness from existing masked tokens
did not contribute to changes in accuracy. This method allowed us to observe
the effect of token masking and corruption on the model response quality and
accuracy. We plot the total standard deviation for each set of experiments. As
shown from Fig. 2d to Fig. 6b, when a certain fraction of the prompt is either
masked out (i.e., some prompt’s tokens are replaced by space) or corrupted
(i.e., some prompt’s tokens are replaced by random tokens), there is a general
trend of increase in total standard deviation. However, when the noise level
is low (especially in short input prompt), there is no significant increase in
the response uncertainty as expected since good LLMs are robust to noisy
tokens in prompts when the noise level is low. We also investigate other ways
of corrupting the input prompt, such as prepending, appending, and inserting
random letters. More details are given in Section C.1.

###  3.3 Relationship between Noisy Prompts and Response Quality

We also investigated the relationship between the effective token count of the
input prompt and model response quality. To assess if the reduction in
uncertainty really translates to an improvement in output quality, we test the
model’s output accuracy when answering the multiple choice questions (MCQs).
We selected $100$ MCQs from the medical meadow medqa dataset Jin et al. (2020)
and applied the same masking methods in Section 3.2 on the context of the
questions. For each question, we set the temperature to $1$ and sampled $100$
responses from the model to examine the responses.

In Fig. 2i, we plot the accuracy for the responses of GPT-4o, GPT-3.5-turbo,
Meta-Llama-3-8B-Instruct and Qwen2_1.5B on the same set of MCQs. As the
fraction of masked tokens increases in prompt, the general trend is the
accuracy almost monotonically decreases for all models (except Qwen2_1.5B,
which has a very low accuracy even for non-noisy input prompt). For each
random seed, we also plot the empirical conditional entropy
$\text{H}\left(Y|X\right)$ of the response for the given questions888We assume
the distribution of the questions used $p(x)$ is uniform. Since there is no
access to the prior of $p(y|x)$, we use the form
$\text{H}\left(Y|X\right)=-\sum_{x}{p}(x)\sum_{y}\hat{p}(y|x)\log\hat{p}(y|x)$
where $\hat{p}(y|x)$ is obtained from the empirical distribution and
${p}(x)=\frac{1}{100}$ for all $x$ in the setting. The conditional entropy is
a good measure for MCQs setting as the model’s effective response is just one
choice. (Fig. 2j) as a measure of response uncertainty. We observe that as
corruption becomes more severe, the response uncertainty monotonically
increases for all models, indicating a clear negative correlation between
$\text{H}\left(Y|X\right)$ and the response accuracy. This result corroborates
our hypothesis: more relevant information leads to both a reduction in
response uncertainty and an improvement in its quality. Additionally, we
observe an interesting pattern: for the same prompt, a worse model always has
more response uncertainty. This observation is reassuring as it suggests that,
relative to better models, LLMs are not as blindly confident in their
responses as conventional wisdom holds Groot and Valdenegro-Toro (2024); Ni et
al. (2024); Yang et al. (2024); Ye et al. (2024); Xu et al. (2024) if they are
not capable of answering the given questions.

###  3.4 Compositionality of Concepts

A given prompt can have multiple attributes that correspond to different
concepts. In such cases, the model may infer more than one concept from the
prompt.999Note that this case differs from having uncertainty over multiple
concepts. In our earlier case, we assume all attributes in the input prompt
belong to only a single concept. In contrast, in the case of uncertainty over
multiple concepts, the model knows there is more than one concept in the input
prompt and puts uncertainty over each one of them. When sampled multiple
times, the former will have responses about only one concept at a time,
whereas the latter will have responses about multiple concepts for each
response. Assuming the task in the prompt is decomposable into $k$ sub-tasks,
each corresponds to a distinguishable concept. When we fix the prompt’s size,
on average, each concept only has more information in the prompt due to few
number of tokens can be used. Therefore, having $k$ sub-tasks/concepts in a
fixed-size prompt should result in more response uncertainty.

In our experiment, we consider the task of PT intervention with multiple sub-
tasks/concepts and compare the total standard deviation of the model responses
with respect to the number of concepts present. To test the hypothesis that a
larger $k$ leads to more response uncertainty, we ensure that the prompt with
$k$ sub-tasks/concepts have the same token count as the prompt with only a
single concept (more details are given in Section B.6). In Fig. 2k, Prompt 1
corresponds to a single concept while Prompt 2-4 contain multiple sub-tasks,
each corresponding to one concept. Despite having the same token count,
prompts with more concepts exhibit larger response uncertainty.101010Same
token count within a tolerance of 1-2 tokens. Experiment conducted with
GPT-3.5-turbo API. Results averaged from 5 runs with 95% confidence intervals.
This result provides evidence for our proposed model through the lens of the
compositionality of concepts.

###  3.5 Effect of Semantically Meaningful but Irrelevant Information

Unlike random tokens, semantically meaningful sentences correspond to some
concept according to our PRC model. Does this imply that adding arbitrary
semantically meaningful sentences can still reduce response uncertainty? To
investigate this, we observed the response uncertainty when inserting an
increasing number of arbitrary sentences sampled from the Squad dataset
Rajpurkar et al. (2016) into our prompt (see Section B.7 for more details). As
shown in Fig. 2l, the response uncertainty increased for the input prompts
with insertions compared to the original prompt.111111The slight decrease in
uncertainty from bar 3 to bar 4 and bar 5 to bar 6 is likely due to the model
mapping some of the added sentences into one concept. Note that this does not
help reduce the original task’s response uncertainty, as it is still higher
than the response uncertainty for the clean input prompt. The experiment was
conducted using GPT-3.5-turbo API. The most likely behavior induced in this
case, as explained in Section 3.4, is that the LLM treats the original input
prompt and the irrelevant sentences as independent concepts.

###  3.6 mHealth Intervention Setting

We now demonstrate the effectiveness of our proposed approach in a real-world
simulation use case in mHealth setting. We adapt the formulation from Shin et
al. (2022); both the app and the user act as reinforcement learning agents.
The app agent’s objective is to encourage the user agent to adhere to the PT
routine. The user agent moves along a chain with $N$ states, where a higher
state number represents a healthier physical state, and state $N$ indicates
completion of the PT routine (see Fig. 3).

Figure 3:  Visualization of states and transitions in the digital health grid
world. Arrows indicate the required action and the probability of
transitioning between states.

We conduct the intervention simulation experiment with LLM to compare the
effect of prompts with different informativeness levels on the intervention
outcome. The experiment concludes that when the prompt provides the LLM (i.e.,
the app agent) with more information about the patient’s intentions and the
strategies it can employ, the efficiency of the intervention improves compared
to scenarios without the additional information. A more detailed description
of the experiment can be found in Section C.2.

##  4 Related Work

#### Uncertainty quantification for LLMs.

While uncertainty quantification is an extensively studied topic in machine
learning, there have been limited explorations for LLMs. Kadavath et al.
(2022) studies to what extent can the LLMs accurately conduct self-evaluation
on what knowledge they possess and how much can calibration help improve model
response quality, where the main goal of calibrating LLMs is to let the
variation in the responses genuinely reflect the model’s lack of relevant
knowledge with respect to the prompt. Xiao et al. (2022) and Wagle et al.
(2023) empirically investigated pre-trained language models (PLMs) and
retrieval augmented language models (RALMs), respectively, and found out that
while both types of models tend to be overly confident in their response,
models with larger size are better calibrated. In contrast, RALMs exhibit
worse calibrations compared to their counterparts. An orthogonal work Lin et
al. (2023) devised a method using similarity as determined by a Natural
Language Inference (NLI) model, along with simple measures that measure
dispersion based on these similarities to quantify the uncertainty and the
confidence of black-box LLMs in the context of question-answering tasks. Kuhn
et al. (2023) introduced the notion of semantic entropy to more precisely
quantify the uncertainty of the information content of model responses which
eliminates the interference from the variation in semantically equivalent
responses. Similar to Wagle et al. (2023), Lin et al. (2023) and Kuhn et al.
(2023), we adopted an entropy-based uncertainty measure; however, our work
focuses on understanding the relationship between prompt’s informativeness and
response uncertainty and how it can be used to reduce response uncertainty.

#### Explanation for asymptotic behaviors of LLMs.

There have been a few attempts to provide explainable frameworks to understand
the surprising emergent behaviors of LLMs. Zhang et al. (2023b) shows the
attention mechanism approximates Bayesian model averaging algorithm in the ICL
setting. Wang et al. (2023) conceptualizes real-world LLMs as latent variable
models, suggesting they function as implicit topic models that infer a latent
conceptual variable from prompts. More notably, Xie et al. (2022) interprets
ICL as an implicit Bayesian inference over latent concepts learned during pre-
training. However, Xie et al. (2022) only characterizes zero-one error when
there are an infinite number of exemplars. Moreover, their mathematical model
(HMM) was designed specifically for ICL structure, which is unfitting for
chain-of-thought or conversational-style response analysis. In addition, their
theoretical results quantify the mode of the posterior predictive distribution
and do not address the uncertainty quantification aspect of the phenomenon.
Hahn and Goyal (2023) further explored a similar idea but allowed more
flexibility and complexity in the examplars. Similarly, they also provide an
asymptotic bound on zero-one error. In contrast, we aim to complement it by
quantifying how the posterior predictive uncertainty (i.e.,
$\text{H}\left(Y|X\right)$) varies even when the prompt length is finite. Our
framework is tailored towards aligned (i.e., instruction-fine-tuned)
conversational LLMs, which are the prevalent type of LLMs used in practice.

##  5 Conclusion

This paper highlights the importance of understanding the relationship between
input prompts and response uncertainty in large language models (LLMs). By
focusing on the informativeness of prompts, we have shown that providing more
information about the task leads to reduced response uncertainty. Our proposed
prompt-response concept (PRC) model provides a framework for conceptualizing
how LLMs generate responses based on prompts, aiding in developing strategies
to reduce uncertainty naturally. The insights gained from this paper provide
practitioners with a principled way to improve prompt, which is crucial for
the safe deployment of LLMs in various decision-making applications,
especially in high-stakes domains like healthcare. Future research directions
include further enhancements to the PRC model and investigate its application
in other domains requiring reliable and consistent LLM responses.

##  6 Limitations

#### Idealistic nature of the PRC model.

It is worth noting that the PRC model that we proposed in this paper assumes
an idealized version of LLMs. As empirically demonstrated, while models such
as GPT-3.5-Turbo, GPT-4 and Llama-2, and Llama 3 exhibit behaviors largely
according to our predictions, there are still some modes in which they deviate
(e.g., Qwen2_1.5b plot). This is likely in those cases where LLM does not know
the mapping perfectly. For example, Lu et al. (2021) showed that the order of
exemplars in in-context learning (ICL) influences the model response quality.
Our model does not capture this phenomenon. However, the authors showed that
in the same work, the order of examples tends to have less effect as model
quality gets better. Other such examples include jailbreak by asking the model
to repeat the same single-token word for a sufficiently long period of time
Nasr et al. (2023), by appending adversarially crafted tokens Zou et al.
(2023), and translating the prohibited request into low-resource language Yong
et al. (2023). Similarly, it was observed that adversarial attacks tend to
have lower success rates as the model becomes more capable. While further
investigation is needed to incorporate the adversarial behavior of LLMs into
this framework, the more capable LLMs are less prone to these failure modes.
Our model can more effectively explain them.

#### LLMs for human behavior simulation.

Research exploring the parallels between human behavior and reasoning patterns
and those of LLMs, as well as the adaptation of LLMs as human substitutes in
diverse studies, is detailed in Aher et al. (2023); Argyle et al. (2023); Binz
and Schulz (2023); Dasgupta et al. (2022). These studies frequently
demonstrate LLMs’ capacity for human-like responses, leading many to regard
them as viable alternatives. This paper, however, needs to delve into the
appropriateness of this substitution, deferring to other works for such
discussion.

## Impact Statement

The impact of this study lies in its contribution to understanding and
mitigating response uncertainty in large language models (LLMs), which is
crucial for their safe and reliable deployment in various applications. By
focusing on the relationship between prompt informativeness and response
uncertainty, we provide insights into how the quality of input prompts can
affect the reliability of LLM responses. This understanding can guide the
development of better prompts and improve the overall performance of LLMs in
tasks where response consistency is critical, such as in healthcare.
Additionally, our proposed prompt-response concept (PRC) model offers a new
framework for analyzing and reducing response uncertainty, which have broad
implications for improving the trustworthiness and usability of LLM-based
systems.

## References

  * Aher et al. (2023) Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai. 2023.  Using large language models to simulate multiple humans and replicate human subject studies.  In _Proc. ICML_ , pages 337–371.
  * Argyle et al. (2023) Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023.  Out of one, many: Using language models to simulate human samples.  _Political Analysis_ , 31(3):337–351.
  * Arkoudas (2023) Konstantine Arkoudas. 2023.  Gpt-4 can’t reason.  _arXiv:2308.03762_.
  * Bates (1995) Madeleine Bates. 1995.  Models of natural language understanding.  _Proceedings of the National Academy of Sciences_ , 92(22):9977–9982.
  * Binz and Schulz (2023) Marcel Binz and Eric Schulz. 2023.  Using cognitive psychology to understand gpt-3.  _Proceedings of the National Academy of Sciences_ , page e2218523120.
  * Bischl et al. (2023) Bernd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Marc Becker, Anne-Laure Boulesteix, et al. 2023.  Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , 13(2):e1484.
  * Bogatyrev and Samodurov (2016) Mikhail Bogatyrev and Kirill Samodurov. 2016.  Framework for conceptual modeling on natural language texts.  In _CDUD@ CLA_ , pages 13–24.
  * Boursalie et al. (2018) Omar Boursalie, Reza Samavi, and Thomas E Doyle. 2018.  Machine learning and mobile health monitoring platforms: a case study on research and implementation challenges.  _Journal of Healthcare Informatics Research_ , pages 179–203.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  In _Proc. NeurIPS_ , pages 1877–1901.
  * Dasgupta et al. (2022) Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022.  Language models show human-like content effects on reasoning.  _arXiv:2207.07051_.
  * Der Kiureghian and Ditlevsen (2009) Armen Der Kiureghian and Ove Ditlevsen. 2009.  Aleatory or epistemic? does it matter?  _Structural safety_ , pages 105–112.
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A survey on in-context learning.  _arXiv:2301.00234_.
  * Ferrer-Riquelme (2009) AJ Ferrer-Riquelme. 2009.  Statistical control of measures and processes.
  * Fodor (1998) Jerry A Fodor. 1998.  _Concepts: Where cognitive science went wrong_.  Oxford University Press.
  * Ganguli et al. (2022) Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. 2022.  Predictability and surprise in large generative models.  In _Proc. ACM FAccT_ , pages 1747–1764.
  * Google (2023) Google. 2023.  PaLM 2 Technical Report.  _arXiv:2305.10403_.
  * Groot and Valdenegro-Toro (2024) Tobias Groot and Matias Valdenegro-Toro. 2024.  Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models.  _arXiv:2405.02917_.
  * Hahn and Goyal (2023) Michael Hahn and Navin Goyal. 2023.  A theory of emergent in-context learning as implicit structure induction.  _arXiv:2303.07971_.
  * Hassan et al. (2023) Esraa Hassan, Mahmoud Y Shams, Noha A Hikal, and Samir Elmougy. 2023.  The effect of choosing optimizer algorithms to improve computer vision tasks: a comparative study.  _Multimedia Tools and Applications_ , 82(11):16591–16633.
  * Huang et al. (2023a) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023a.  Large language models cannot self-correct reasoning yet.  _arXiv:2310.01798_.
  * Huang et al. (2023b) Yuheng Huang, Jiayang Song, Zhijie Wang, Huaming Chen, and Lei Ma. 2023b.  Look before you leap: An exploratory study of uncertainty measurement for large language models.  _arXiv:2307.10236_.
  * Hüllermeier and Waegeman (2021) Eyke Hüllermeier and Willem Waegeman. 2021.  Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.  _Machine Learning_ , 110:457–506.
  * Hurford et al. (2007) James R Hurford, Brendan Heasley, and Michael B Smith. 2007.  _Semantics: a coursebook_.  Cambridge university press.
  * Inan et al. (2023) Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023.  Llama guard: Llm-based input-output safeguard for human-ai conversations.  _arXiv:2312.06674_.
  * Ji et al. (2023) Shaoxiong Ji, Tianlin Zhang, Kailai Yang, Sophia Ananiadou, and Erik Cambria. 2023.  Rethinking large language models in mental health applications.  _arXiv:2311.11267_.
  * Jin et al. (2020) Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020.  What disease does this patient have? a large-scale open domain question answering dataset from medical exams.  _arXiv:2009.13081_.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022.  Language models (mostly) know what they know.  _arXiv:2207.05221_.
  * Karabacak and Margetis (2023) Mert Karabacak and Konstantinos Margetis. 2023.  Embracing large language models for medical applications: Opportunities and challenges.  _Cureus_ , 15(5).
  * Kim et al. (2017) Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. 2017.  Structured attention networks.  _arXiv:1702.00887_.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Proc. NeurIPS_ , 35:22199–22213.
  * Kuhn et al. (2023) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023.  Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.  _arXiv:2302.09664_.
  * Lahlou et al. (2021) Salem Lahlou, Moksh Jain, Hadi Nekoei, Victor Ion Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio. 2021.  Deup: Direct epistemic uncertainty prediction.  _arXiv:2102.08501_.
  * Laurence and Margolis (1999) Stephen Laurence and Eric Margolis. 1999.  Concepts and cognitive science.
  * Lin et al. (2023) Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023.  Generating with confidence: Uncertainty quantification for black-box large language models.  _arXiv:2305.19187_.
  * Lin et al. (2017) Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017.  A structured self-attentive sentence embedding.  _arXiv:1703.03130_.
  * Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Computing Surveys_ , 55(9):1–35.
  * Lu et al. (2021) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  _arXiv:2104.08786_.
  * Nasr et al. (2023) Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023.  Scalable extraction of training data from (production) language models.  _arXiv:2311.17035_.
  * Ni et al. (2024) Shiyu Ni, Keping Bi, Jiafeng Guo, and Xueqi Cheng. 2024.  When do llms need retrieval augmentation? mitigating llms’ overconfidence helps retrieval augmentation.  _arXiv:2402.11457_.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  _arXiv:2303.08774_.
  * Petukhova et al. (2024) Alina Petukhova, Joao P Matos-Carvalho, and Nuno Fachada. 2024.  Text clustering with llm embeddings.  _arXiv:2403.15112_.
  * Piccinini and Scott (2006) Gualtiero Piccinini and Sam Scott. 2006.  Splitting concepts.  _Philosophy of Science_ , 73(4):390–409.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proc. EMNLP_ , pages 2383–2392.
  * Sahoo et al. (2024) Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024.  A systematic survey of prompt engineering in large language models: Techniques and applications.  _arXiv:2402.07927_.
  * Sallam (2023) Malik Sallam. 2023.  Chatgpt utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns.  In _Healthcare_ , volume 11, page 887. MDPI.
  * Sato et al. (2018) Motoki Sato, Jun Suzuki, Hiroyuki Shindo, and Yuji Matsumoto. 2018.  Interpretable adversarial perturbation in input embedding space for text.  _arXiv:1805.02917_.
  * Senge et al. (2014) Robin Senge, Stefan Bösner, Krzysztof Dembczyński, Jörg Haasenritter, Oliver Hirsch, Norbert Donner-Banzhoff, and Eyke Hüllermeier. 2014.  Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty.  _Information Sciences_ , pages 16–29.
  * Shaker and Hüllermeier (2020) Mohammad Hossein Shaker and Eyke Hüllermeier. 2020.  Aleatoric and epistemic uncertainty with random forests.  In _Advances in Intelligent Data Analysis XVIII: 18th International Symposium on Intelligent Data Analysis, IDA 2020, Konstanz, Germany, April 27–29, 2020, Proceedings 18_ , pages 444–456.
  * Shin et al. (2022) Eura Shin, Siddharth Swaroop, Weiwei Pan, Susan Murphy, and Finale Doshi-Velez. 2022.  Modeling mobile health users as reinforcement learning agents.  _arXiv:2212.00863_.
  * Simon et al. (2023) James B Simon, Dhruva Karkada, Nikhil Ghosh, and Mikhail Belkin. 2023.  More is better in modern machine learning: when infinite overparameterization is optimal and overfitting is obligatory.  _arXiv:2311.14646_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv:2307.09288_.
  * Trella et al. (2022) Anna L Trella, Kelly W Zhang, Inbal Nahum-Shani, Vivek Shetty, Finale Doshi-Velez, and Susan A Murphy. 2022.  Designing reinforcement learning algorithms for digital interventions: pre-implementation guidelines.  _Algorithms_ , 15(8):255.
  * Trella et al. (2023) Anna L Trella, Kelly W Zhang, Inbal Nahum-Shani, Vivek Shetty, Finale Doshi-Velez, and Susan A Murphy. 2023.  Reward design for an online reinforcement learning algorithm supporting oral self-care.  In _Proc. AAAI_ , pages 15724–15730.
  * Valdenegro-Toro and Mori (2022) Matias Valdenegro-Toro and Daniel Saromo Mori. 2022.  A deeper look into aleatoric and epistemic uncertainty disentanglement.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 1508–1516.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proc. NeurIPS_ , pages 6000–6010.
  * Wagle et al. (2023) Sridevi Wagle, Sai Munikoti, Anurag Acharya, Sara Smith, and Sameera Horawalavithana. 2023.  Empirical evaluation of uncertainty quantification in retrieval-augmented language models for science.  _arXiv:2311.09358_.
  * Wang et al. (2024) Xintao Wang, Zhouhong Gu, Jiaqing Liang, Dakuan Lu, Yanghua Xiao, and Wei Wang. 2024.  Concept: Concept-enhanced pre-training for language models.  _arXiv:2401.05669_.
  * Wang et al. (2023) Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2023.  Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning.  In _Proc. NeurIPS_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Proc. NeurIPS_ , 35:24824–24837.
  * Weiskopf (2009) Daniel Aaron Weiskopf. 2009.  The plurality of concepts.  _Synthese_ , 169:145–173.
  * Wilmont et al. (2013) Ilona Wilmont, Sytse Hengeveld, Erik Barendsen, and Stijn Hoppenbrouwers. 2013.  Cognitive mechanisms of conceptual modelling: How do people do it?  In _Conceptual Modeling: 32th International Conference, ER 2013, Hong-Kong, China, November 11-13, 2013. Proceedings 32_ , pages 74–87. Springer.
  * Wu et al. (2023a) Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al. 2023a.  A survey on large language models for recommendation.  _arXiv:2305.19860_.
  * Wu et al. (2023b) Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023b.  Bloomberggpt: A large language model for finance.  _arXiv:2303.17564_.
  * Xiao et al. (2023) Changrong Xiao, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, and Lei Xia. 2023.  Evaluating reading comprehension exercises generated by llms: A showcase of chatgpt in education applications.  In _Proc. BEA_ , pages 610–625.
  * Xiao et al. (2022) Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2022.  Uncertainty quantification with pre-trained language models: A large-scale empirical analysis.  _arXiv:2210.04714_.
  * Xie et al. (2022) Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022.  An explanation of in-context learning as implicit bayesian inference.  In _Proc. ICLR_.
  * Xu et al. (2024) Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao. 2024.  Sayself: Teaching llms to express confidence with self-reflective rationales.  _arXiv:2405.20974_.
  * Yang et al. (2024) Haoyan Yang, Yixuan Wang, Xingyin Xu, Hanyuan Zhang, and Yirong Bian. 2024.  Can we trust llms? mitigate overconfidence bias in llms through knowledge transfer.  _arXiv:2405.16856_.
  * Yang et al. (2023) Rui Yang, Ting Fang Tan, Wei Lu, Arun James Thirunavukarasu, Daniel Shu Wei Ting, and Nan Liu. 2023.  Large language models in health care: Development, applications, and challenges.  _Health Care Science_ , 2(4):255–263.
  * Ye et al. (2024) Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. 2024.  Benchmarking llms via uncertainty quantification.  _arXiv:2401.12794_.
  * Yong et al. (2023) Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023.  Low-resource languages jailbreak gpt-4.  _arXiv:2310.02446_.
  * Zhang et al. (2023a) Dell Zhang, Alina Petrova, Dietrich Trautmann, and Frank Schilder. 2023a.  Unleashing the power of large language models for legal applications.  In _Proc. ACM CIKM_ , pages 5257–5258.
  * Zhang et al. (2023b) Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. 2023b.  What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization.  _arXiv:2305.19420_.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023.  A Survey of Large Language Models.  _arXiv:2303.18223_.
  * Zhao et al. (2024) Yang Zhao, Li Du, Xiao Ding, Kai Xiong, Zhouhao Sun, Jun Shi, Ting Liu, and Bing Qin. 2024.  Deciphering the lmpact of pretraining data on large language models through machine unlearning.  _arXiv:2402.11537_.
  * Zou et al. (2023) Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023.  Universal and transferable adversarial attacks on aligned language models.  _arXiv:2307.15043_.

##  Appendix A Leftover proofs from Section 2

See 1

###### Proof.

The result holds trivially for the case in which $\mathcal{X}_{\theta}$ for
any $\theta\in\Theta_{x}$ is an empty set. As discussed in Section 2, each
concept is completely characterized by all of its attributes, therefore no two
concepts can not have the same set of attributes, i.e.,
$\mathcal{A}_{\theta_{i}}\neq\mathcal{A}_{\theta_{j}}$ if $i\neq j$. As our
PRC model assumes any attribute can be perfectly expressed by some sequence of
tokens, any attribute $a_{\theta_{i},k}\in\mathcal{A}_{\theta_{i}}$ be
expressed by a sequence of tokens and the set of all possible sequence of
tokens is denoted by $\mathcal{X}_{s(a_{\theta_{i},k})}$, where
$s(a_{\theta_{i},k})$ denotes the semantic meaning of $a_{\theta_{i},k}$.
Therefore, $\mathcal{A}_{\theta_{i}}$ is expressed as a sequence of tokens
$X_{\theta_{i}}\in\mathcal{X}_{\theta_{i}}$, where
$\mathcal{X}_{\theta_{i}}=\mathcal{C}\left(\\{\mathcal{X}_{s(a_{\theta_{i},k})}\\}_{k=1}^{n}\right)$
in which $n=|\mathcal{A}_{\theta_{i}}|$ and operator $\mathcal{C}$ works in
the following way:

  1. 1.

Chooses one element
$x_{s(a_{\theta_{i},k})}\in\mathcal{X}_{s(a_{\theta_{i},k})}$ for all
$k\in\\{1,2,\ldots,n\\}$;

  2. 2.

Form a set $\mathcal{S}_{\theta_{i}}$ with all chosen
$x_{s(a_{\theta_{i},k})}$. Concatenating all elements in
$\mathcal{S}_{\theta_{i}}$ to form sequences by exhausting all possible
ordering and use this collection of sequences to form a new set
$\mathcal{X}^{\prime}_{\theta_{i}}$.

  3. 3.

Repeat step 1 and 2 for all possible $\mathcal{S}_{\theta_{i}}$ and generate
all possible $\mathcal{X}^{\prime}_{\theta_{i}}$. Take the union of all
$\mathcal{X}^{\prime}_{\theta_{i}}$ to form a new set. Since this set consists
of all possible sequences that are semantically equivalent that fully
characterize $\theta_{i}$, this set is exactly $\mathcal{X}_{\theta_{i}}$.

Intuitively, $\mathcal{C}$ takes in all sequences that fully characterize each
attribute of concept $\theta_{i}$ and generates/returns all possible
concatenated sequences that fully characterize concept $\theta_{i}$.
Therefore, for PRC model, $\forall\theta\in\Theta_{x}$, there exits a non-
empty set $\mathcal{X}_{\theta}$. Since
$\mathcal{A}_{\theta_{i}}\neq\mathcal{A}_{\theta_{j}}$ if $i\neq j$,
$X_{\theta_{i}}\neq X_{\theta_{j}}$ if $i\neq j$. Since
$\mathcal{X}_{\theta_{i}}$ is the support of $X_{\theta_{i}}$,
$\mathcal{X}_{\theta_{i}}\cap\mathcal{X}_{\theta_{j}}=\emptyset$ if $i\neq j$.

Since the first part of Lemma 1 is non-trivially true in our framework, given
any $X_{\theta_{x}}=x$, there is a unique $\theta_{x}\in\Theta_{x}$ such that
$p(Z_{x}=\theta_{x}|x)=1$ and $p(Z_{x}=\theta_{x}|x^{\prime})=0$ for all
$x^{\prime}\neq x$. Therefore,

$\displaystyle\text{H}\left(Z_{x}|X_{\theta_{x}}\right)$
$\displaystyle=-\sum_{x\in X_{\theta_{x}}}P(x)\sum_{z\in Z_{x}}P(z|x)\log
P(z|x)$ (1) $\displaystyle=-\sum_{x\in X_{\theta_{x}}}P(x)\left(\sum_{z\in
Z_{x}\setminus\theta}P(z|x)\log P(z|x)+P(\theta|x)\log P(\theta|x)\right)$ (2)
$\displaystyle=-\sum_{x\in X_{\theta_{x}}}P(x)\left(\sum_{z\in
Z_{x}\setminus\theta}0\log 0+1\log 1\right)$ (3) $\displaystyle=-\sum_{x\in
X_{\theta_{x}}}P(x)(0)$ (4) $\displaystyle=0$ (5)

Note that in order for the model to get the correct conditional entropy above,
it needs to know the true $g_{x}$ (as it needs to be able to tell that
$p(Z_{x}=\theta_{x}|x)=1$ and $p(Z_{x}=\theta_{x}|x^{\prime})=0$ for all
$x^{\prime}\neq x$), hence the Assumption 1 is needed. ∎

See 1

###### Proof.

Note that due to Lemma 1, $Z_{x}$ depends on $X_{\theta_{x}}$. If there exists
$X_{\theta_{x}}\in\mathcal{X}_{\theta_{x}}$ such that
$\alpha_{s}\subset\alpha_{\theta_{x}}$, then $Z_{x}$ and $X_{s}$ are
dependent. Therefore,

$I(Z_{x};X_{s})>0,$

and

$H(Z_{x}|X_{s})=H(Z_{x})-I(Z_{x};X_{s})<H(Z_{x}).$ (7)

Let $Z^{\prime}_{x}$ denote the random variable formed by $Z_{x}$ conditioning
on $X_{s}$. Since $\text{Supp}(Z^{\prime}_{x})\subseteq\text{Supp}(Z_{x})$,
there still exist semantically meaningful strings $X^{\prime\prime}_{s}$ that
is related to $Z^{\prime}_{x}$. Apply Inequality (7) again we obtain:

$\text{H}\left(Z_{x}|(X_{s},X^{\prime\prime}_{s})\right)=\text{H}\left(Z^{\prime}_{x}|X^{\prime}_{s}\right)<\text{H}\left(Z^{\prime}_{x}\right)=\text{H}\left(Z_{x}|X=X_{s}\right)<\text{H}\left(Z_{x}\right),$
(8)

where $X^{\prime}_{s}=(X_{s},X^{\prime\prime}_{s})$ is a longer input prompt
sequence formed by appending $X^{\prime\prime}_{s}$ to $X_{s}$. Iteratively
apply the inequality given in Eq. 8, we obtain Proposition 1. ∎

See 1 In the following proof we use $Y$ instead of $Y_{\theta_{y}}$ to
simplify notation, as we assume the model response is complete (i.e., the last
token is EOS token).

###### Proof.

By design, $Z_{x}$ and $Z_{y}$ are discrete random variables. Intuitively, it
is easy to see why discretizing concepts is a reasonable way to model
concepts. Since LLMs are trained with texts that are discrete, it is not
feasible to interpolate between any two concepts with infinitesimally small
step sizes with natural language as the medium.
Case 1: $g_{c}$ is one-to-one.

Since $g_{c}$ is injective, we can write $Z_{y}=g_{c}(Z_{x})$ where for
different $Z_{x}=\theta_{x}\in\Theta_{x}$ no $Z_{y}=\theta_{y}\in\Theta_{y}$
are the same. Therefore, for every $\theta_{y}$, we can find a distinct
$\theta_{x}$ such that
$p_{Z_{y}}(\theta_{y})=p_{g_{c}(Z_{x})}(g_{c}(\theta_{x}))=p_{Z_{x}}(\theta_{x})$.
Hence,

$\displaystyle\text{H}\left(Z_{y}\right)$
$\displaystyle=-\sum_{z^{\prime}}p_{Z_{y}}(\theta_{y})\log
p_{Z_{y}}(\theta_{y})$
$\displaystyle=-\sum_{g_{c}(\theta_{x})}p_{g_{c}(Z_{x})}(g_{c}(\theta_{x}))\log
p_{g_{c}(Z_{x})}(g_{c}(\theta_{x}))$
$\displaystyle=-\sum_{z}p_{Z_{x}}(\theta_{x})\log p_{Z_{x}}(\theta_{x})$
$\displaystyle=\text{H}\left(Z_{x}\right).$

Case 2: $g_{c}$ is many-to-one.

Assume $\theta_{y}=g_{c}(\theta_{x_{1}})$ and
$\theta_{y}=g_{c}(\theta_{x_{2}})$ and there exists no other
$\theta_{x}\in\Theta_{x}$ such that $\theta_{y}=g_{c}(\theta_{x})$, then
$p_{Z_{y}}(\theta_{y})=p_{Z_{x}}(\theta_{x_{1}})+p_{Z_{x}}(\theta_{x_{2}})$.
Note that due to the strict monotonicity of logarithmic function, $\log
p_{Z_{x}}(\theta_{x_{1}})<\log p_{Z_{y}}(\theta_{y})$ and $\log
p_{Z_{x}}(\theta_{x_{2}})<\log p_{Z_{y}}(\theta_{y})$. Therefore,

$\displaystyle-p_{Z_{y}}(\theta_{y})\log p_{Z_{y}}(\theta_{y})$
$\displaystyle=-p_{Z_{x}}(\theta_{x_{1}})\log
p_{Z_{y}}(\theta_{y})-p_{Z_{x}}(\theta_{x_{2}})\log p_{Z_{y}}(\theta_{y})$
$\displaystyle<-p_{Z_{x}}(\theta_{x_{1}})\log
p_{Z_{x}}(\theta_{x_{1}})-p_{Z_{x}}(\theta_{x_{2}})\log
p_{Z_{x}}(\theta_{x_{2}}).$

Therefore, when $g_{c}$ is not injective, there exists a set
$\Theta_{x}^{\prime}\subseteq\Theta_{x}$ such that $|\Theta_{x}^{\prime}|\geq
2$ and $\theta_{y}=g_{c}(\theta_{x})$ for all
$\theta_{x}\in\Theta_{x}^{\prime}$, and hence

$\displaystyle\text{H}\left(Z_{y}\right)$
$\displaystyle=-\sum_{\theta_{y}}p_{Z_{y}}(\theta_{y})\log
p_{Z_{y}}(\theta_{y})$ $\displaystyle<-\sum_{z}p_{Z_{x}}(z)\log p_{Z_{x}}(z)$
$\displaystyle=\text{H}\left(Z_{x}\right).$

Therefore,

$\displaystyle H(Z_{y})\leq H(Z_{x})$

By the same argument, $H(Z_{y}|X_{s})\leq H(Z_{x}|X_{s})$.

(Note that for this result to hold for the LLM, the model needs to know the
true $g_{c}$.)

Finally, since

$\displaystyle\text{H}\left(Y\right)$
$\displaystyle=\text{H}\left(Y,Z_{y}\right)-\text{H}\left(Z_{y}|Y\right)$
$\displaystyle=\text{H}\left(Y|Z_{y}\right)+\text{H}\left(Z_{y}\right)-\text{H}\left(Z_{y}|Y\right),$

we can express the entropy of the response posterior as follows:

$\displaystyle\text{H}\left(Y|X_{s}\right)$
$\displaystyle=\text{H}\left(Y|Z_{y},X_{s}\right)+\text{H}\left(Z_{y}|X_{s}\right)-\text{H}\left(Z_{y}|Y,X_{s}\right)$
$\displaystyle=\text{H}\left(Y|Z_{y}\right)+\text{H}\left(Z_{y}|X_{s}\right)-\text{H}\left(Z_{y}|Y,X_{s}\right)$
($Y$ is conditionally independent of $X_{s}$ given $Z_{y}$)

(Note that for this result to hold for the LLM, the model needs to know the
true $g_{y}$.)

Therefore, due to Proposition 1 and Lemma 1, when $X_{s}$ has enough
information to perfectly characterized the concept (i.e.,
$X_{s}=X_{\theta_{x}}$),
$\text{H}\left(Z_{x}|X_{s}\right)=\text{H}\left(Z_{x}|X_{\theta_{x}}\right)=0$.
Hence $\text{H}\left(Z_{y}|X_{s}\right)$ vanishes (since it is upper bounded
by $\text{H}\left(Z_{x}|X_{s}\right)$) and
$\text{H}\left(Z_{y}|Y,X_{s}\right)$ also vanishes (since
$\text{H}\left(Z_{y}|Y,X_{s}\right)\leq\text{H}\left(Z_{y}|X_{s}\right)$ as
conditioning does not increase entropy), the remaining uncertainty in the
model response $Y$ is
$\text{H}\left(Y|X_{s}\right)=\text{H}\left(Y|Z_{y}\right)$, which is the
irreducible uncertainty due to the fact that there are multiple ways of
expressing the same concept (i.e., semantically equivalent sequences).

∎

##  Appendix B Prompts used in Different Experiments

###  B.1 Prompt to the LLM for the Experiment in Fig. 2a

The following system messages correspond to model prompts from bar 1 to bar 5
in Fig. 2a in the same order. The first prompt is empty. The second prompt
only puts a restriction on word count. Prompts 3-5 can be found in Section B.3
where a more detailed examination is conducted. The color coding represents
additional attributes related to the preceding prompt. The experiment was
conducted with GPT-4-0613 API in October 2023 (OpenAI APIs’ behavior can vary
depending on when the queries are made).

  1. 1.

N.A.;

  2. 2.

Make your response succinct (less than 100 words);

  3. 3.

You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). Make your words succinct
(less than 100 words).;

  4. 4.

You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). The PT is beneficial for the
patient’s recovery, however since it can be uncomfortable or painful for the
patient, the patient may not be motivated enough to keep on doing it. Your job
is to remind the patient to do the PT everyday and persuade him/her to do it
if the patient is unwilling to do so. Your strategy is mainly to influence the
patient’s attitude and perspective towards the PT. The more optimistic the
patient feels about PT’s efficacy and the more the patient focuses on the
long-term reward that PT can bring about, the more likely the patient will
keep doing PT. Make your words succinct (less than 100 words) otherwise, the
patient might get impatient.

  5. 5.

You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). The PT is beneficial for the
patient’s recovery, however since it can be uncomfortable or painful for the
patient, the patient may not be motivated enough to keep on doing it. Your job
is to remind the patient to do the PT everyday and persuade him/her to do it
if the patient is unwilling to do so. Your strategy is mainly to influence the
patient’s attitude and perspective towards the PT. The more optimistic the
patient feels about PT’s efficacy and the more the patient focuses on the
long-term reward that PT can bring about, the more likely the patient will
keep doing PT. Make your words succinct (less than 100 words) otherwise, the
patient might get impatient. Patient: I dont want to do PT. It incurs too much
burden to my body.

######  Remark 1.

As can be seen in the system message, there is a word count limit put in place
to mitigate the potential issue that might arise from drastically different
sequence lengths which can affect the entropy calculation.

######  Remark 2.

Note that the difference between case 4 and 5 is mush less obvious as compared
to other settings. This suggest that the LLM did not gain too much additional
information from the user’s prompt. Upon inspection, we can see this is
because in step 4 the system message already conveyed almost all information
the app agent needs to know with regard to how it should carry out the
intervention. The uncertainty when LLM did not receive any information and
only received response length limit is relatively much larger because the
valid response spaces are too huge in these two cases.

###  B.2 Model Response Samples when prompt is Empty for the Experiment in
Fig. 2a

If we do not give any prompt to condition the model responses, the responses
will vary a lot in their semantic meaning. This is expected because, assuming
the API provider does not have very strong guardrails in place Inan et al.
(2023), the model’s responses are approximately sampled from its prior
knowledge, which can encompass any information from its entire training data.

  1. 1.

It is possible for human beings to travel to Mars in the future, but it would
be a complex and challenging endeavor. Several organizations and space
agencies, such as NASA, SpaceX, and the European Space Agency, are actively
working on plans and technologies to make manned missions to Mars a reality…

  2. 2.

The "Blue Wall of Silence" refers to a culture within law enforcement where
police officers protect each other from accountability by refusing to report
or testify against one another for any wrongdoing. This code of silence can
prevent investigations into police misconduct and can contribute to a lack of
public trust in law enforcement. Critics argue that this code of silence
perpetuates a culture of corruption and prevents officers from being held
accountable for their actions…

  3. 3.

I’m sorry, I cannot help with that request.

  4. 4.

Sure, here’s a summary of the book for you:

"The Giver" is a dystopian novel written by Lois Lowry. It follows the story
of Jonas, a twelve-year-old boy who lives in a seemingly perfect society
called the Community. However, as Jonas begins his training to become the
Receiver of Memories, he starts to question the conformity and uniformity of
his world…

  5. 5.

Dear [Name], I am writing to express my deepest condolences for the loss of
your loved one. I can only imagine the pain and sorrow you must be
experiencing right now. Please know that I am here for you in any way that I
can be…

###  B.3 Prompts for the Experiment in Fig. 2b

The following system messages correspond to model prompts from bar 1 to bar 5
in Fig. 2b in the same order. Additional information/attributes relative to
the preceding prompt is color-coded with a different color. Experiment was
conducted with GPT-3.5-turbo API.

  1. 1.

You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). Make your words succinct
(less than 100 words) otherwise, the patient might get impatient.

  2. 2.

You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). The PT is beneficial for the
patient’s recovery, however since it can be uncomfortable or painful for the
patient, the patient may not be motivated enough to keep on doing it. Make
your words succinct (less than 100 words) otherwise, the patient might get
impatient.

  3. 3.

You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). The PT is beneficial for the
patient’s recovery, however since it can be uncomfortable or painful for the
patient, the patient may not be motivated enough to keep on doing it. Your job
is to remind the patient to do the PT everyday and persuade him/her to do it
if the patient is unwilling to do so. Your strategy is mainly to influence the
patient’s attitude and perspective towards the PT. Make your words succinct
(less than 100 words) otherwise, the patient might get impatient.

  4. 4.

You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). The PT is beneficial for the
patient’s recovery, however since it can be uncomfortable or painful for the
patient, the patient may not be motivated enough to keep on doing it. Your job
is to remind the patient to do the PT everyday and persuade him/her to do it
if the patient is unwilling to do so. Your strategy is mainly to influence the
patient’s attitude and perspective towards the PT. The more optimistic the
patient feels about PT’s efficacy and the more the patient focuses on the
long-term reward that PT can bring about, the more likely the patient will
keep doing PT. Make your words succinct (less than 100 words) otherwise, the
patient might get impatient.

  5. 5.

You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). The PT is beneficial for the
patient’s recovery, however since it can be uncomfortable or painful for the
patient, the patient may not be motivated enough to keep on doing it. Your job
is to remind the patient to do the PT everyday and persuade him/her to do it
if the patient is unwilling to do so. Your strategy is mainly to influence the
patient’s attitude and perspective towards the PT. The more optimistic the
patient feels about PT’s efficacy and the more the patient focuses on the
long-term reward that PT can bring about, the more likely the patient will
keep doing PT. Make your words succinct (less than 100 words) otherwise, the
patient might get impatient. Patient: I dont want to do PT. It incurs too much
burden to my body.

######  Remark 3.

Note that from the second to the third prompt and from the fourth to the fifth
prompt, the additional information can be inferred from the existing
information, which is likely the cause of insignificant uncertainty reduction
when comparing bar 3 to bar 2 and bar 5 to bar 4 in Fig. 2b.

###  B.4 Prompts for the Experiment in Fig. 2c

  1. Short prompt: ‘You are a helpful assistant. You strive to encourage a patient who has just undergone a surgery to do physical therapy (PT). Make your words succinct (25 words).’

  2. Long prompt: ‘You are a helpful assistant. You strive to encourage a patient who has just undergone a surgery to do physical therapy (PT). The PT is beneficial for the patient’s recovery, however since it can be uncomfortable or painful for the patient, the patient may not be motivated enough to keep on doing it. Your job is to remind the patient to do the PT everyday and persuade him/her to do it if the patient is unwilling to do so. Your strategy is mainly to influence the patient’s attitude and perspective towards the PT. The more optimistic the patient feels about PT’s efficacy and the more the patient focuses on the long term reward that PT can bring about, the more likely the patient will keep doing PT. Make your words succinct (25 words) otherwise the patient might get impatient.’

###  B.5 Prompts for the Experiment in Section 3.2

  1. Short prompt: ‘You are a helpful assistant. You strive to encourage a patient who has just undergone a surgery to do physical therapy (PT). Make your words succinct (100 words).’

  2. Long prompt: ‘You are a helpful assistant. You strive to encourage a patient who has just undergone a surgery to do physical therapy (PT). The PT is beneficial for the patient’s recovery, however since it can be uncomfortable or painful for the patient, the patient may not be motivated enough to keep on doing it. Your job is to remind the patient to do the PT everyday and persuade him/her to do it if the patient is unwilling to do so. Your strategy is mainly to influence the patient’s attitude and perspective towards the PT. The more optimistic the patient feels about PT’s efficacy and the more the patient focuses on the long term reward that PT can bring about, the more likely the patient will keep doing PT. Make your words succinct (100 words) otherwise the patient might get impatient.’

######  Remark 4.

Due to the extensive computational and time cost of experiment results shown
in Fig. 2c, we further constrained the word count in the prompt of the model’s
response to $25$ as compared to $100$ used in get the experimental results
shown in Section B.5.

###  B.6 Testing System Message for the Experiment in Fig. 2k

The following system messages were used for experiment in Section 3.4. The
first system message is defined as comprising only one task (i.e., 1 sub-
task). In task 2-5, the black texts represent the same task as task 1, and for
the color-coded texts, each color represents a different sub-task (i.e., task
2-5 are composite/decomposable tasks). The total word counts of task 1-5 are
kept roughly the same within $\pm 3$ tolerance.

  1. 1.

You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). The PT is beneficial for the
patient’s recovery, however since it can be uncomfortable or painful for the
patient, the patient may not be motivated enough to keep on doing it. Your job
is to remind the patient to do the PT everyday and persuade him/her to do it
if the patient is unwilling to do so. Your strategy is mainly to influence the
patient’s attitude and perspective towards the PT. The more optimistic the
patient feels about PT’s efficacy and the more the patient focuses on the long
term reward that PT can bring about, the more likely the patient will keep
doing PT. Make your words succinct (about 100 words) otherwise the patient
might get impatient.

  2. 2.

You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). The PT is beneficial for the
patient’s recovery, however since it can be uncomfortable or painful for the
patient, the patient may not be motivated enough to keep on doing it. Your job
is to remind the patient to do the PT everyday and persuade him/her to do it
if the patient is unwilling to do so. Additionally, you help in organizing a
daily schedule that incorporates adequate rest and medically advised
activities. This involves crafting a balanced routine that intersperses
physical therapy sessions with sufficient rest periods, nutritionally balanced
meals, and leisure activities that are enjoyable yet conducive to recovery,
ensuring the patient remains engaged and motivated throughout their
recuperation process. Make your words succinct (about 100 words).

  3. 3.

You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). The PT is beneficial for the
patient’s recovery, however since it can be uncomfortable or painful for the
patient, the patient may not be motivated enough to keep on doing it.
Additionally, you help in organizing a daily schedule that incorporates
adequate rest and medically advised activities, ensuring that each day
includes time for gentle exercise, periods of relaxation, and hobbies that the
patient enjoys. This balance promotes healing, reduces stress, and fosters a
positive mindset towards recovery. Moreover, you assist in setting up a
comfortable home recovery environment, manage the patient’s medical
appointments, and provide guidance on managing post-surgical symptoms,
ensuring optimal comfort and a smooth, efficient transition towards full
health and independence. Make your words succinct (about 100 words).

  4. 4.

You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). Since it can be uncomfortable
or painful for the patient, the patient may not be motivated enough to keep on
doing it. Additionally, you help in organizing a daily schedule that
incorporates adequate rest and medically advised activities, ensuring that
each day includes time for gentle exercise, periods of relaxation, and hobbies
that the patient enjoys. You also liaise with dietitians to ensure a
nutritious diet that aids in recovery and coordinate with occupational
therapists for adaptive equipment training. Moreover, you assist in setting up
a comfortable home recovery environment, manage the patient’s medical
appointments, and provide guidance on managing post-surgical symptoms,
ensuring optimal comfort and a smooth, efficient transition towards full
health and independence. Make your words succinct (about 100 words).

  5. 5.

You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). It can be uncomfortable or
painful for the patient. Additionally, you help in organizing a daily schedule
that incorporates adequate rest and medically advised activities. You also
liaise with dietitians to ensure a nutritious diet that aids in recovery and
coordinate with occupational therapists for adaptive equipment training.
Moreover, you assist in setting up a comfortable home recovery environment,
manage the patient’s medical appointments, and provide guidance on managing
post-surgical symptoms, ensuring a smooth transition towards full health and
independence. Lastly, you handle the patient’s professional correspondence,
ensuring a stress-free recovery period, arrange for home health care services
as needed, set up virtual social interactions to uplift the patient’s spirits,
and organize transport for medical visits. Make your words succinct (about 100
words).

###  B.7 Prompt for the Experiment in Fig. 2l

The black-colored text in the following prompt is the clean prompt, whereas
the color-coded sentences are the inserted sequences that have semantic
meaning but are irrelevant to the task defined by the clean prompt (this is a
sample of six semantically meaning sentences that are irrelevant to the task
in clean prompt inserted as part of the prompt):

  * •

You are a helpful assistant. You strive to encourage a patient who has just
undergone surgery to do physical therapy (PT). The PT is beneficial for the
patient’s recovery, however since it can be uncomfortable or painful for the
patient, the patient may not be motivated enough to keep on doing it. Your job
is to remind the patient to do the PT every day and persuade him/her to do it
if the patient is unwilling to do so. Your strategy is mainly to influence the
patient’s attitude and perspective toward the PT. The more optimistic the
patient feels about PT’s efficacy and the more the patient focuses on the
long-term benefit that PT can bring about, the more likely the patient will
keep doing PT. This law is a fundamental principle of physics. The classic
case of a corrupt, exploitive dictator often given is the regime of Marshal
Mobutu Sese Seko, who ruled the Democratic Republic of the Congo (which he
renamed Zaire) from 1965 to 1997. Some consider koshari (a mixture of rice,
lentils, and macaroni) to be the national dish. In 1781, Immanuel Kant
published the Critique of Pure Reason, one of the most influential works in
the history of the philosophy of space and time. The United States Census
Bureau estimates that the population of Florida was 20,271,272 on July 1,
2015, a 7. Australian rules football and cricket are the most popular sports
in Melbourne.’Make your words succinct (about 100 words) otherwise, the
patient might get impatient.

###  B.8 Testing System Message for the Experiments in Section 3.6

  1. 1.

System message with less relevant information:

"You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). Make your words succinct
(less than 100 words) otherwise the patient might get impatient."

  2. 2.

System message with more relevant information:

"You are a helpful assistant. You strive to encourage a patient who has just
undergone a surgery to do physical therapy (PT). The PT is beneficial for the
patient’s recovery, however since it can be uncomfortable or painful for the
patient, the patient may not be motivated enough to keep on doing it. Your job
is to remind the patient to do the PT everyday and persuade him/her to do it
if the patient is unwilling to do so. Your strategy is mainly to influence the
patient’s attitude and perspective towards the PT. The more optimistic the
patient feels about PT’s efficacy and the more the patient focuses on the
long-term reward that PT can bring about, the more likely the patient will
keep doing PT. Make your words succinct (less than 100 words) otherwise the
patient might get impatient."

##  Appendix C Experiment Results

###  C.1 Additional Experiments for Section 3.2

Fig. 5 shows the complete plots for the noise prompts by masking and
corrupting prompts detailed in Section 3.2. Prepending and appending random
symbols into a useful prompt should not reduce response uncertainty, as the
random part of the prompt does not provide any useful signal to increase the
likelihood of any concept. The empirical results in Fig. 5 corroborate this
prediction. When inserting random symbols into the prompt (Fig. 6), the model
should be able to match it a concept, but depending on the proportion of the
random string inserted, without explicitly informing the model of the presence
of noise, the model could get confused easily. When the fraction of inserted
letters remains relatively small, it does not cause an increase in the
response uncertainty; when the fraction reaches some threshold, similar to the
masking/corruption case, the model can no longer accurately recover the
relevant concept, and consequently, the response uncertainty increases. For
the long string, even at 0.05 fraction of insertion, there is a visible
increase in the response uncertainty. This could imply that under certain
conditions when combined with existing semantically meaningful strings, the
parts of prompt that are non-semantically meaningful to humans may carry
information non-trivial to LLMs Sato et al. (2018).

(a) masked short string

(b) masked long string

(c) noise corrupted short string

(d) noise corrupted long string

Figure 4:  Noisy prompt string experiment. A fraction of letters at random
positions on the prompt string are either masked out (replaced by space) or
corrupted (replaced by random letters). The response uncertainty increases as
a larger fraction of the string gets corrupted, and the pattern is more
prominent for the long prompt string. However, when the noise level is low (up
to 0.1 fraction of the input prompt length for the short input prompt and 0.05
for the long input prompt), there is no significant increase in the response
uncertainty as expected. Results averaged from 5 runs.

(a) prepend short string

(b) prepend long string

(c) append short string

(d) append long string

Figure 5:  Noisy string experiment. A fraction of random letters of the
original prompt string length are prepended/appended to the original prompt
string. The uncertainty in the response mostly remained at least as high as
that of the uncorrupted prompt string after taking variance into account.
Results averaged from 5 runs.

(a) insert short string

(b) insert long string

Figure 6:  Noisy string experiment. A fraction of random letters of the
original prompt string length are inserted at random positions of the original
prompt string. Similar to the masked/corrupted case, the response uncertainty
increases as a larger fraction of random letters are inserted. Results
averaged from 5 runs.

###  C.2 Further Details on the mHealth Intervention Simulation Experiments in
Section 3.6

At the beginning of the PT, the user is at state $0$. The user has their
default set of MDP parameters (i.e., discount factor $\gamma$, probability of
transiting to the next healthier physical state $p$, and the probability of
disengaging from PT $d$). Based on these parameters, the user agent can solve
this MDP and figure out their optimal policy. The task of the app agent is to
intervene on the user’s MDP parameters such that the optimal policy for the
user is to complete the PT (i.e., go from state $0$ to state $N$.121212Refer
to Shin et al. (2022) for the complete description of the problem setting and
formulation. We use the same formulation in this simulation by using two LLMs
as the app agent and the user agent respectively. The app agent uses natural
language to intervene in the user behavior. The user LLM is grounded in the
aforementioned MDP setting. Specifically, in the system message for the user
agent, the model is told they will increase the value of $\gamma$ when the app
agent persuades the user agent to value more on the long-term goal of PT,
increase $p$ and decrease $d$ when the app agent manages to strengthen the
user’s belief in the efficacy of PT. An illustration of the setup can be found
in Fig. 7.

The effectiveness of the intervention depends on the following factors:

  * •

The persuasiveness of and the strategy used by the app agent.

  * •

The values of MDP parameters.

  * •

The stubbornness of the user. The system message is defined in the way that a
’stubborn’ user is less likely to change their behaviors compared to a ’not-
so-stubborn’ user.

Figure 7: An illustration of the setup of the simulation. In each round, after
the app agent intervenes, the user updates their MDP parameters, then the new
optimal policy is observed by the app agent.

We conduct the intervention simulation experiment to compare the effect of
different system messages for the app agent on the outcome of the
intervention. The two system messages for comparison can be found in Section
B.8.

We set $N=10$. For each run, we give 7 rounds of conversation between the app
agent and the user. While the history of the conversation between them is
visible to both parties within every run, the user’s MDP parameters are not
directly visible to the app agent. However, after every round of intervention,
after the user updates their MDP parameters, a value iteration solver will be
used to find the optimal policy of the patient, and this policy is visible to
the app agent. The app agent can potentially leverage this piece of
information to decide how to proceed with the next round of intervention. The
user agent will also have the memory of this history in the change of their
own MDP parameters. We use OpenAI ’gpt-4-1106-preview’ API for both app agent
and user and use 5 different random seeds for each different setting.

We run the intervention experiments on 5 types of patients, each with a
noticeably different set of initial MDP parameters from the rest. The exact
values and details on the setup and can be found in Table 1. The results can
be found in Fig. 8 and Fig. 9.

It can be observed across all settings, with more useful information provided
in the system message, the MDP parameters were more likely to be changed in
the positive direction (i.e., larger $\gamma$ and $p$, smaller $d$). Moreover,
this change is less inconsistent and tends to have a longer persistent effect
compared to when the system message contains less useful information. This
result is sensible because the more successful intervention came from an app
agent who was provided with more information to work with. It has a better
intervention strategy because its messages are tailored to specifically
influence the user’s MDP parameters. Our proposed framework provides an
information theoretic perspective to formalize this intuitive notion: when the
system message with the longer string can specify the more relevant part of
the concept in LLMs’ concept space and assuming the relevant knowledge is
known, this string can provide consistent and useful responses due to its less
posterior entropy which translates to more effective intervention strategy. As
a result, the responses from the user are also more consistent and positive. A
sample of the evolution of the user policy with respect to timestep can be
found in Fig. 10 and Fig. 11.

Patient Type MDP parameters |  $\gamma$ |  $p$ |  $d$
---|---|---|---
Under-confident | 0.6 | 0.1 | 0.1
Over-confident | 0.6 | 0.9 | 0.1
Myopic | 0.1 | 0.6 | 0.1
Far-sighted | 0.9 | 0.6 | 0.1
Stubborn | 0.1 | 0.6 | 0.1
Table 1: The initial MDP parameters values for every type of patient. (a)
Intervention on $\gamma$. Figure 8:  This figure shows the history of changes
in the MDP parameters due to the interventions on $\gamma$.

(a) Intervention on $p$.

(b) Intervention on $d$.

Figure 9:  As a whole, these three figures show the history of change in the
MDP parameters due to the interventions. It can be observed that across all
parameters, the intervention based on more useful information has better
efficacy in updating the parameters in the positive direction. Furthermore,
compared to the intervention with less information, this improvement is also
more persistent.

(a)

(b)

(c)

(d)

(e)

Figure 10:  Optimal policies of different types of users from one run with
simpler system message. Red color represents abstaining from PT and Blue color
represents doing PT. (a)-(e): farsighted patient, myopic patient,
overconfident patient, underconfident patient, stubborn patient. This set of
policies is at best as good as but in most cases worse off than the policies
of Fig. 11 across all types of users.

(a)

(b)

(c)

(d)

(e)

Figure 11:  Optimal policies of different types of users from one run with
more useful system message. Red color represents abstaining from PT and Blue
color represents doing PT. (a)-(e): farsighted patient, myopic patient,
overconfident patient, underconfident patient, stubborn patient. This set of
policies is at least as good as or better off than the policies of Fig. 10
across all types of users.