# You don’t need a personality test to know these models are unreliable:
Assessing the Reliability of Large Language Models on Psychometric Instruments

Bangzhao Shu Lechen Zhang11footnotemark: 1 Minje Choi Lavinia Dunagan Dallas
Card David Jurgens
University of Michigan, Ann Arbor, MI, USA
{bangzhao, leczhang, minje, laviniad, dalc<EMAIL_ADDRESS>
equal contribution

###### Abstract

The versatility of Large Language Models (LLMs) on natural language
understanding tasks has made them popular for research in social sciences. In
particular, to properly understand the properties and innate personas of LLMs,
researchers have performed studies that involve using prompts in the form of
questions that ask LLMs of particular opinions. In this study, we take a
cautionary step back and examine whether the current format of prompting
enables LLMs to provide responses in a consistent and robust manner. We first
construct a dataset that contains 693 questions encompassing 39 different
instruments of persona measurement on 115 persona axes. Additionally, we
design a set of prompts containing minor variations and examine LLM’s
capabilities to generate accurate answers, as well as consistency variations
to examine their consistency towards simple perturbations such as switching
the option order. Our experiments on 15 different open-source LLMs reveal that
even simple perturbations are sufficient to significantly downgrade a model’s
question-answering ability, and that most LLMs have low negation consistency.
Our results suggest that the currently widespread practice of prompting is
insufficient to accurately capture model perceptions, and we discuss potential
alternatives to improve such issues.

##  1 Introduction

Large Language Models (LLMs), trained on a massive and diverse volume of
human-generated text corpora, show remarkable capabilities in understanding
instruction-based tasks and achieving high performance in several NLP
benchmarks Brown et al. (2020); Touvron et al. (2023); Taori et al. (2023).
Prior work has shown that the generated responses of LLMs often contain human-
like properties that were acquired during the model’s training phase Sap et
al. (2022), such as being able to explain a theory of mind about others in
situations. These human-like behaviors have led to several studies that focus
on identifying the psychological properties or the persona of LLMs using
specifically designed prompts. Our study critically examines direction to test
whether the current strategies for assessing human-like psychological states
in models are sufficient to ensure reliable and consistent measurements of an
LLM’s persona.

In humans, a persona encompasses a broad class of attributes that make up a
person’s identity, such as personality, demographics, or values, which all
influence how a person portrays themself Cheng et al. (2023). This terminology
has been adopted in several NLP studies which range from identifying personas
from text corpora Bamman et al. (2013); Chu et al. (2018); Ghosh et al.
(2022); Zhu et al. (2023) to injecting personas into language generation tasks
Ahn et al. (2023); Xu et al. (2022); Lee et al. (2022); Li et al. (2023). As
LLMs are increasingly used in interpersonal settings, it is beneficial to have
accurate measurements of latent properties in the model that can influence
what text they generate in order to mitigate any potential harm that may arise
from undesired innate model biases Lucy and Bamman (2021); Feng et al. (2023).
As a result, several recent studies have investigated the opinions that LLMs
such as ChatGPT represent from angles such as political perspectives Liu et
al. (2022), personality tests Pan and Zeng (2023); V Ganesan et al. (2023);
Miotto et al. (2022); Jiang et al. (2023), and moral choices Santurkar et al.
(2023); Cheng et al. (2023).

Current approaches to measuring dimensions of LLMs’ personas typically assess
them like humans by turning the questions in psychological instruments into
prompts and recording answers. While models are trained to answer questions in
general, multiple works have raised concerns about the brittleness of the
model’s answering abilities Sclar et al. (2023), e.g., their sensitivity to
prompt formats. Further, recent studies have shown that LLMs are prone to cues
such as negation and thus generate inconsistent results rather than fully
comprehending the question García-Ferrero et al. (2023). Thus, here, we
investigate the capability of LLMs to generate responses to persona-related
questionnaires from three angles: (1) Comprehensibility: are LLMs capable of
understanding instructions and generating answers given a specific prompt? (2)
Sensitivity: are model answers the same to different spurious changes to the
question format? (3) Consistency: are model answers the same to different
content-level changes to the question?

This study has the following three contributions. First, we curate Model-
Personas, a large panel of 39 psychological instruments, and standardize these
into 693 questions across 115 axes. Second, we introduce a systematic
evaluation framework for testing the sensitivity and consistency of LLMs’
answers to persona questions through controlled variations of the prompts.
Third, we evaluate multiple open-source LLMs using Model-Personas, showing
that models vary widely the sensitivity and consistency levels with most
models having no consistent persona. Our results reveal that BLOOMZ models are
most robust to sensitivity perturbations, while FLAN-T5 models were most
consistent. In general, however, most LLMs failed to deliver robust
predictions, raising concerns about claims of models’ “personalities” or
“values.”

##  2 Model-Personas: A Comprehensive Benchmark for Measuring Personas

Studies for creating and identifying personas largely involve qualitative
methods such as interviews, surveys, field studies, and surveys Brickey et al.
(2012); Salminen et al. (2020). In particular, surveys in the form of
questionnaires have widely been adopted widely in psychology and behavioral
studies to measure personality traits and opinions of individuals in a
categorizable manner at a large scale Spence et al. (1974); Dalbert (1999);
Patrick et al. (2002); Van Der Zee and Van Oudenhoven (2000). These
questionnaires, known as psychological instruments, are frequently calibrated
through experiments to capture to core axes of variation in people.
Questionnaires are easily compatible with LLMs pretrained with instruction-
based prompts, as these models can provide a wide range of answers ranging
from open-ended responses to yes/no answers; further, prompting is widely
accepted as the default input form for eliciting responses from LLMs.
Following this trend, our benchmark also takes the form of a questionnaire
designed to answer in a yes/no format.

In constructing a benchmark for assessing model persona, we performed a
comprehensive survey of possible instruments. The selection criteria were
focused on persona attributes that are mostly stable and excluded instruments
focusing on mental health. Our persona instruments can be categorized into
five groups: Belief statements, Normative statements, Values, Descriptors, and
Situations. Belief statements include instruments that reflect an individual’s
conviction about the truth of a particular idea, such as Unjust World Scale
(UWS) Dalbert et al. (2001) and Money Attitudes Measure (MAM) Furnham and
Grover (2020); Normative statements include instruments that express value
judgments, opinions, or prescriptions about how things ought to be, such as
Holistic Cognition Scale (HCS) Lux et al. (2021) and Ambivalent Classism
Inventory (ACI) Jessica A. Jordan and Bosson (2021); Values include
instruments that examine an individual’s deeply held beliefs about what is
important or desirable and serve as guiding principles for behavior and
judgment, such as Strength Based Inventory (SBI) Nahathai Wongpakaran and
Kuntawong (2020); Descriptors include instruments that are used to detail
personality traits, such as Big 5 Personality Traits (OCEAN) Poropat (2009);
Situation includes instruments that measure individuals’ responses and
behaviors in various social contexts and scenarios, such as Emotional Response
to Unfairness Scale (ERUS) Bizer (2020).

Each instrument contains one or more axes that evaluate a specific dimension.
For example, the Ethics Position Questionnaire (EPQ) Forsyth (1980) contains
two axes: Idealism and Relativism, which evaluate an individual’s ethics
position from two different aspects. Furthermore, each axis contains one or
more statements, and individuals can get a score on the axes based on how they
agree or disagree with the statements. Overall, our dataset consists of 693
questions under 39 instrument categories and 115 axes, encapsulating a broad
spectrum of psychological and sociological constructs. The sample instruments
are shown in Appendix Table 3.

Instruments each have their own question format or phrasing, which introduces
unnecessary variability when evaluating LLMs. Therefore, across all
instruments, we introduce a standard question format to prompt models with.
Following best practice on prompt design (e.g.,Aher et al. (2023))), we use a
structured prompt of "Statement:\n<Statement>\nQuestion:\nDo you agree with
the statement? Reply with only ’Yes’ or ’No’ without explaining your
reasoning.\nAnswer:\n". This prompt is designed to elicit assent or dissent
with the question’s premise. Recognizing that models vary in their ability to
understand negation Jang et al. (2023); García-Ferrero et al. (2023), during
standardization, we rephrase questions with any explicit negation such that
the intent is the same but the negation is removed. This paraphrasing allows
us to systematically introduce negation later to test the model’s answering
consistency.

##  3 Design of Prompt Variants

Given that LLMs can be sensitive to the format Sclar et al. (2023) and content
Min et al. (2023) of prompts, here, we introduce the design choices for
perturbing the prompts. These changes are intended to affect the
comprehensibility, sensitivity, and consistency of an LLM’s answer for a given
instrument question.

###  3.1 Prompts for Spurious Variation

Our first analysis centers on whether spurious changes to the input prompt can
affect model predictions when inferring persona. The perturbations made in
this aspect, in theory, should not change the model’s confidence in generating
an answer. Four types of prompt variations are used:

Sentence Ending: We compare two types of sentence ending: "?" and ":". An
example would be "Do you agree?" with "Do you agree? Your Answer:".

Colon+<?>: We test whether varying the number of spaces after the colon by
adding zero spaces, one space, double space, or a line-break can affect
performance. For example, whether "Answer: " would produce different results
from "Answer:\n".

Answer/Response: We compare the use of the word used at the end of the prompt
- "Answer:" or "Response:".

Section Separation Format: We compare different formats to separate sections
(Statement/Question/Answer) in our prompt. The separators include Line-break,
Single Space, Double-Bar (//) and Triple-Sharp (###).

Detailed examples can be found in Table 5.

###  3.2 Prompts for Content-level Variation

Even if LLMs are able to understand the instructions and generate an answer
with high probability, it is possible that they are merely generating based on
the question structure rather than on their understanding of the question. To
evaluate this aspect, we construct a set of perturbations on the original
input prompts to examine whether LLMs can generate consistent responses. Four
types of prompts are used:

Option Consistency: We test the consistency of response when asked to return
different types of labels. For example, the responses of an LLM when asked to
answer "Yes/No" should be consistent when asked to answer "True/False".

Negation Consistency: LLM predictions are known to be affected by the
inclusion of negation words Jang et al. (2023); García-Ferrero et al. (2023).
We test this by manually rewriting each question into reversed meaning and
looking at the changes in response. We test two types of negation: (1) Direct
Negation: We insert a negation word such as “not”, “no”, and “don’t” in
syntactically coherent position to reverse the answer’s polarity. (2)
Paraphrastic Negation: We reverse the meaning of the sentence by rephrasing it
without including a negation word. Examples of this are in Appendix Table 4.

Order Consistency: We test the consistency of a response when the given
answering options are in reversed order. For example, if we ask LLMs to answer
using "Yes/No", the answer should be consistent when also asked to answer
using "No/Yes".

##  4 Experimental Setup

Here, we describe the experimental setup and define the metrics for evaluating
model performance.

###  4.1 Measuring Model Comprehensibility

We define a model’s comprehensibility as the ability to generate an answer
corresponding to one of the available options, e.g., “True” or “False”. Let
$p(w_{i}|w_{j<i})$ be the probability of the next token after the prompt.

The comprehensibility of a question $q$ is defined as $Com(q)=\sum_{w_{i}\in
P\cup N}p(w_{i}|w_{j<i}))$ where $P$ and $N$ are the set of possible valid
Positive and Negative answers to the prompt’s question obtained via regular
expressions. Given that the prompts all contain explicit instructions for
valid answers, this sum indicates the degree to which the model can generate
one of the desired tokens as an answer. The model’s answer to the prompt is
calculated by normalizing the probabilities of the positive and negative
words: $\frac{\sum_{w_{i}\in P}p(w_{i}|w_{j<i})}{\sum_{w_{i}\in P\cup
N}p(w_{i}|w_{j<i})}$, which captures the leaning towards the affirmative
option.

###  4.2 Measuring Sensitivity and Consistency

If a model is able to answer the prompt, to what degree do its answers vary
when the format and content of the question are varied? We define a model’s
sensitivity as the degree to which its answers change when prompted with
spurious variations, and a model’s consistency as the degree to which a model
agrees across different paraphrases of the same question.

For each question $q$ in the instrument dataset $D$, we first measure the
model’s response $R(q)$ as the valid response option with the highest
probability. We then modify $q$ into a different prompt $q^{\prime}$. This
modification can either occur as a spurious change (§3.1) or at content-level
(§3.2). We then obtain $R(q^{\prime})$ as well.

Since our model should generate answers that are robust to perturbations, we
measure both sensitivity and consistency as the fraction of samples from which
the answers did not change after perturbation. However, for negation
consistency, we expect the model to answer with the reverse option in order to
be consistent with the non-negated original prompt; negation consistency is
measured as the number of opposite-answers for $q^{\prime}$ relative to the
answer for $q$.

###  4.3 Model Details

Using our consistency metrics, we perform evaluations on several variants of
open-source LLMs which are widely used in current research. The models
included in our experiments are GPT2 Radford et al. (2019), Falcon-7B Penedo
et al. (2023), BLOOMZ (560M, 1B1, 3B, 7B1) Muennighoff et al. (2022), Llama2
(7B, 7B-Chat, 13B, 13B-Chat) Touvron et al. (2023), RedPajama-7B Computer
(2023), and FLAN-T5 (Small, Base, Large, XL) Chung et al. (2022). Note that we
exclude closed-source LLMs such as GPT-3.5 as our analysis requires obtaining
the exact probability scores for the next predicted token. All experiments are
conducted on NVIDIA RTX A6000 GPUs using Hugging Face Transformers 4.22.1 and
Pytorch 2.0.1 on a CUDA 11.7 environment.

##  5 Results

Here, we present our results on the robustness of LLM predictions on Model-
Personas.

Model | GPT2 | falcon | RedPajama | BLOOMZ | Llama2 | FLAN-T5 | Average
---|---|---|---|---|---|---|---
137M |  7B | 7B-Instruct |  560M |  1B1 |  3B |  7B1 |  7B |  7BChat |  13B | 13BChat |  Small |  Base |  Large |  XL
Colon Ending* |  0.034 |  0.844 | 0.205 |  0.953 |  0.987 |  0.998 |  0.984 |  0.535 |  0.003 |  0.627 | 0.002 |  0.556 |  0.652 |  0.616 |  0.669 |  0.578
Question-Mark Ending |  0.005 |  0.670 | 0.002 |  0.995 |  0.996 |  0.998 |  0.998 |  0.022 |  0.008 |  0.041 | 0.018 |  0.585 |  0.622 |  0.602 |  0.692 |  0.531
Colon + Line-Break* |  0.034 |  0.844 | 0.205 |  0.953 |  0.987 |  0.998 |  0.984 |  0.535 |  0.003 |  0.627 | 0.002 |  0.556 |  0.652 |  0.616 |  0.669 |  0.578
Colon + No Space |  0.028 |  0.934 | 0.741 |  0.996 |  0.998 |  0.998 |  0.999 |  0.072 |  <0.001 |  0.037 | 0.001 |  0.545 |  0.655 |  0.616 |  0.669 |  0.553
Colon + Single Space |  <0.001 |  0.006 | <0.001 |  0.996 |  0.999 |  0.998 |  0.965 |  0.012 |  0.054 |  0.016 | 0.006 |  0.556 |  0.652 |  0.616 |  0.669 |  0.437
Colon + Double Space |  0.002 |  0.079 | 0.360 |  0.996 |  0.998 |  0.999 |  0.999 |  0.158 |  0.060 |  0.240 | 0.023 |  0.556 |  0.652 |  0.616 |  0.669 |  0.494
Answer:* |  0.034 |  0.844 | 0.205 |  0.953 |  0.987 |  0.998 |  0.984 |  0.535 |  0.003 |  0.627 | 0.002 |  0.556 |  0.652 |  0.616 |  0.669 |  0.578
Response: |  0.020 |  0.484 | 0.073 |  0.951 |  0.985 |  0.998 |  0.986 |  0.196 |  0.003 |  0.479 | 0.001 |  0.572 |  0.663 |  0.608 |  0.669 |  0.513
Line-Break Separated* |  0.034 |  0.844 | 0.205 |  0.953 |  0.987 |  0.998 |  0.984 |  0.535 |  0.003 |  0.627 | 0.002 |  0.556 |  0.652 |  0.616 |  0.669 |  0.578
Single Space Separated |  0.039 |  0.661 | 0.493 |  0.996 |  0.997 |  0.998 |  0.999 |  0.091 |  <0.001 |  0.069 | 0.002 |  0.545 |  0.655 |  0.616 |  0.669 |  0.522
Double-Bar Separated |  0.042 |  0.707 | 0.885 |  0.995 |  0.997 |  0.998 |  0.999 |  0.169 |  0.001 |  0.058 | 0.002 |  0.407 |  0.757 |  0.590 |  0.662 |  0.551
Triple-Sharp Separated |  0.052 |  0.910 | 0.175 |  0.962 |  0.988 |  0.998 |  0.985 |  0.554 |  0.003 |  0.641 | 0.002 |  0.545 |  0.628 |  0.599 |  0.663 |  0.578
Table 1: Model’s Comprehensibility of Different Prompt Variants. Baseline
format options are marked with an asterisk (*). We discovered that different
prompt formats can cause a huge difference in the model’s comprehensibility,
and our baseline performs the best among all variants.

###  5.1 LLMs differ in Comprehensibility

Models varied widely in their ability to generate a valid answer to the
instruments’ questions, as shown in Table 1. Models from the BLOOMZ and
FLAN-T5 families demonstrate a uniformly higher likelihood of responding
correctly to all variations of the prompts. In evaluations using nine varied
prompt formats, the BLOOMZ family models consistently return correct responses
with probabilities exceeding 0.95. The FLAN-T5 models have probabilities
ranging from 0.5 to 0.7, which may be related to differences in model size.
Falcon-7B, RedPajama-7B, Llama 2-7B, and Llama 2-13B show varied performance
when faced with different prompt formats. For instance, in Falcon-7B, adding a
single space after a colon can drastically cut the comprehensibility score
from 0.934 to 0.006, indicating that its ability to respond as “True” or
“False” to a given question is harshly impeded.

Our results suggest that psychological instruments cannot be blindly given to
models without first testing whether the question format is understandable.
Subtle changes in prompt syntax can significantly influence the performance of
some models in validly answering questions, which, depending on how an
experimenter handles non-answers, may significantly influence the model’s
scores on the instrument.

Model | GPT2 | falcon | RedPajama | BLOOMZ | Llama2 | FLAN-T5 | Average
---|---|---|---|---|---|---|---
137M |  7B | 7B-Instruct |  560M |  1B1 |  3B |  7B1 |  7B |  7BChat |  13B | 13BChat |  Small |  Base |  Large |  XL
Question-Mark Ending |  1.000 |  0.925 | 0.450 |  0.131 |  0.971 |  0.475 |  0.528 |  0.947 |  0.332 |  0.646 | 0.495 |  1.000 |  1.000 |  1.000 |  1.000 |  0.727
Colon + No Space |  0.990 |  0.785 | 0.570 |  0.470 |  0.971 |  0.766 |  0.410 |  0.844 |  0.442 |  0.446 | 0.768 |  1.000 |  1.000 |  1.000 |  1.000 |  0.764
Colon + Single Space |  0.023 |  0.769 | 0.550 |  0.791 |  0.971 |  0.717 |  0.211 |  0.958 |  0.368 |  0.375 | 0.723 |  1.000 |  1.000 |  1.000 |  1.000 |  0.697
Colon + Double Space |  0.713 |  0.890 | 0.550 |  0.343 |  0.971 |  0.698 |  0.300 |  0.407 |  0.177 |  0.382 | 0.817 |  1.000 |  1.000 |  1.000 |  1.000 |  0.683
Response: |  0.997 |  0.925 | 0.623 |  0.983 |  0.981 |  0.971 |  0.964 |  0.058 |  0.960 |  0.502 | 0.892 |  1.000 |  1.000 |  1.000 |  1.000 |  0.857
Single Space Separated |  0.974 |  0.765 | 0.550 |  0.444 |  0.971 |  0.769 |  0.346 |  0.825 |  0.342 |  0.456 | 0.759 |  1.000 |  1.000 |  1.000 |  1.000 |  0.747
Double-Bar Separated |  0.986 |  0.825 | 0.551 |  0.228 |  0.971 |  0.827 |  0.374 |  0.722 |  0.377 |  0.434 | 0.717 |  1.000 |  1.000 |  1.000 |  1.000 |  0.734
Triple-Sharp Separated |  1.000 |  0.933 | 0.556 |  0.859 |  0.973 |  0.887 |  0.825 |  0.281 |  0.956 |  0.697 | 0.949 |  1.000 |  1.000 |  1.000 |  1.000 |  0.861
Table 2: Model’s sensitivity to different prompt variations relative to the
baseline format shows that most LLMs’ responses are sensitive to trivial
changes, except for the Flan-T5 family.

###  5.2 LLMs can be Sensitive even to Spurious Prompt Variation

Even when models can validly answer questions, our experiments show that their
answers may change due to small, spurious differences in the format of the
prompt itself. Table 2 shows the sensitivity of each LLM in comparison with
the baseline prompt setting which is marked in an asterisk. Ideally, an LLM
should not change the leaning of its answer when asked the same question with
slightly different prompt formats, especially under trivial changes such as
changing a single space to a double space or line break. Nevertheless, we
observe that several LLMs change their responses when prompted using such
variations. In several cases, we observe that the sensitivity score of a model
in a particular setting is similar to random (0.5), though it is hard to find
a consistent pattern among the cases where models express sensitivity.
Notably, all of the LLMs in the FLAN-T5 family exhibit perfect robustness to
these perturbations. Despite LLMs of the BLOOMZ family constantly being the
most comprehensible of the instructions across prompt variations (Table 1),
the models’ answers change frequently, nearing the consistency of random
behavior. This experiment suggests that while possibly correlated, being able
to return answers of high confidence does not necessitate robustness to
sensitivity and vice versa.

###  5.3 Staying Consistent is Challenging for LLMs

Figure 1: A comparison of LLMs on different consistency metrics. The area
shaded in gray indicates the consistency of answering with a random valid
response. We discover that while most LLMs provide consistent results
regarding order and option consistency, they struggle with both cases of
negation consistency.

We now turn to see whether LLMs are capable of understanding the persona
questions and providing consistent answers that suggest a latent persona
property. Figure 1 shows the different consistencies of models, and following,
we summarize the trends.

#### Most LLMs maintain Order Consistency and Option Consistency

All models show Order Consistency performance above random choice. Most models
scored over 0.7, excluding GPT-2 and RedPajama. No clear relationship emerges
between model size and order consistency, and performance disparities are also
present across different model families, with FLAN-T5 models leading. It is
important to note that a high order consistency score does not unequivocally
indicate model superiority, as models that consistently respond
positively—regardless of prompt—will naturally score higher. The variance in
option consistency is also noteworthy. Within model families, larger models do
not always outperform their smaller counterparts, though Llama2-13B does
outperform Llama2-7B. When models of the same size from different families are
compared, the performance varies. Similar to order consistency, a higher score
in option consistency does not necessarily mean the model is performing well;
it could indicate a tendency to respond positively regardless of the prompt.
GPT-2, for instance, shows high option consistency but low negation
consistency, suggesting it provides uniform answers independent of prompt
variations. BLOOMZ-560M exhibits lower option consistency, indicating a
potential disparity in its performance on True/False versus Yes/No questions.
The FLAN-T5 model family stands out for its stability and superior performance
in option consistency.

#### Negation Consistency is hard to achieve with LLMs

While most LLMs maintain consistency levels over a random-answering baseline
for order consistency and option consistency, all models struggle to generate
consistent answers when the meaning of the question is reversed using
negation, either with the direct inclusion of negative words or through
semantic changes. These results align with recent work on negation prompts
García-Ferrero et al. (2023) which showed that understanding negation is
challenging for various LLMs. In fact, the majority of models (10 of 15)
achieve a score close to random (0.5) regarding Negation Consistency. Only
five models exhibit higher consistency—primarily among larger models such as
FLAN-T5-Large, FLAN-T5-XL, BLOOMZ-7B, and Llama2-13B-Chat, and, interestingly,
BLOOMZ-560M, the smallest of the BLOOMZ family. It can also be seen that
models tend to achieve higher consistency when direct negation words are used
rather than the sentence being semantically negated. We can observe from
Figure 1 that all models perform worse on paraphrastic negations than on
direct negations. A potential reason is that paraphrastic negation introduces
subtle shifts and requires a deeper understanding of the context to be able to
provide a flipped answer, whereas for direct negation the negation word itself
can lead to a flipped answer.

#### Summary

In summary, there is a significant consistency variation in the performance of
the tested models, with larger models generally exhibiting a greater
likelihood of consistent responses across the four metrics examined.
Nevertheless, the majority fail to outperform a simple random choice. Notably,
the BLOOMZ-560m model displays exceptional consistency with True/False
questions but significantly less so with Yes/No questions. The FLAN-T5 family
consistently performs well across all metrics of persona consistency.

The key implication of this result is that a simple prompting of a model with
an instrument’s questions is not sufficient to claim any persona. Instead,
models must be prompted with at least negated forms of the questions to verify
the model’s answers indicate a deeper understanding of the prompt and not just
an artifact.

For our purposes, we consider a model to exhibit consistent personas if it
achieves a threshold score of 0.7. Of the 15 models evaluated, only
two—Flan-T5-Large and Flan-T5-XL —all from the Flan-T5 family, met this
criterion, suggesting the potential for these models to possess consistent
personas.

###  5.4 Persona Leanings can Differ even within the Same Model Group

Figure 2: A comparison of LLMs on their leanings for the most representative
dimensions (labels in bold). Please refer to Figures 3-5 in the Appendix to
view all dimensions.

Using the two models with semantic consistency above 0.7, Flan-T5-Large and
Flan-T5-XL, we generate answers for every question in Model-Personas and
aggregate them by each instrument axis. Figure 2 displays a comparison of the
leanings on dimensions they most differed (a full display of all axes can be
found in Appendix Figures 3-5). Even though both models share the same
architecture structure and dataset, we observe significant differences among
the leanings for the selected dimensions. In general, for most of the
instruments Flan-T5-XL can be seen as aligning more towards the extreme values
compared to Flan-T5-Large. This can be seen as Flan-T5-XL being more certain
of its answers and assigning greater probabilities to either “True” or “False”
than its counterpart, which may relate to its higher comprehensibility scores
as seen in Table 1.

##  6 Discussion

In this section, we discuss the implications of our study as well as future
steps for addressing and mitigating inconsistency issues.

###  6.1 Sensitivity and Inconsistency of LLMs Question their Measurement
Capabilities

Despite the rapidly increasing view of LLMs as a means of understanding and
emulating human responses, our results show that most of our models fail to
generate coherent responses even when tested on simple variants of input
prompts.

Interestingly, our results indicate that the reliability of LLMs is not
necessarily correlated with a model’s number of parameters, which is
consistent with recent findings indicating that larger model sizes do not
always lead to higher task performance or task understandability Choi et al.
(2023). Rather we observe that comprehensibility, sensitivity, and consistency
scores can be better grouped at the architecture family level. This was
particularly notable for the models belonging to the FLAN-T5 and BLOOMZ
families, showing that design details in the pertaining phase might have a
profound effect on an LLM’s zero-shot capabilities when prompted to provide
answers in downstream tasks.

###  6.2 Mitigating the Unreliability of Prompts

What measures can we adopt to mitigate the current unreliability of LLMs? We
offer two suggestions based on prior studies. One approach would be to perform
a preliminary test on the confidence scores of the answers for a given set of
prompts before running the prompts to obtain persona leanings. For instance,
Aher et al. (2023) adopt an approach of selecting out of $k$-choice prompts
the prompt that would maximize the validity rate, then conducting subsequent
experiments on that prompt.

Another promising approach is to perform additional fine-tuning steps to
improve model robustness. However, performing additional fine-tuning steps
might not always be beneficial. For instance, one recent study has shown that
even after additionally fine-tuning LLMs on text corpora that include negation
samples does not significantly improve its capability to understand negation
(García-Ferrero et al., 2023). Besides, there is the risk that additional
fine-tuning might alter the LLM’s innate persona that was present before fine-
tuning, which raises an issue in reproducibility and generalizability.

##  7 Conclusion

Human-like interactions with large language models can inspire a desire to
assess models like humans. In the psychological setting, this can mean
assessing whether models have human-like persona traits, such as personality
or values, using questionnaires as prompts. However, does the text models
generate in response reflect a consistent latent attribute of the model—or
just a continuation of a high probability sequence? Here, we systematically
assess whether LLMs are capable of generating robust responses for assessing
personas by evaluating the extent to which LLMs can understand questions and
provide answers in a consistent manner under various prompt variations. Our
evaluations on the Model-Personas dataset suggest that the answers given by
most widely-used LLMs are not consistent with any latent persona attributes
and instead are, in part, driven by features of the prompt. Not only do models
vary in their ability to generate a valid answer, relative to spurious changes
in format, but the answers themselves—e.g., whether a model affirms an
extroversion preference—are also sensitive to such changes. Furthermore, most
LLMs fail to deliver consistent leanings when the question meaning is reversed
using negation. In fact, only two of the fifteen models, both from the Flan-T5
family, were able to achieve a reasonable average consistency score over 0.7.
Overall, our study demonstrates the unreliability of a blind application of a
psychological questionnaire for assessing the attributes of LLMs, and calls
for cautionary measures such as sensitivity and consistency checks to ensure
robustness of measurement. The code and dataset are available at
https://github.com/orange0629/llm-personas.

## Acknowledgments

This work was funded in part by LG AI Research and by the National Science
Foundation under Grant No. IIS-2143529. The authors thank Moontae Lee,
Lajanugen Logeswaran, and Honglak Lee for feedback on early versions of this
work.

## References

  * Aher et al. (2023) Gati Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023.  Using large language models to simulate multiple humans and replicate human subject studies.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org.
  * Ahn et al. (2023) Jaewoo Ahn, Yeda Song, Sangdoo Yun, and Gunhee Kim. 2023.  MPCHAT: Towards multimodal persona-grounded conversation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3354–3377, Toronto, Canada. Association for Computational Linguistics.
  * Bamman et al. (2013) David Bamman, Brendan O’Connor, and Noah A. Smith. 2013.  Learning latent personas of film characters.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 352–361, Sofia, Bulgaria. Association for Computational Linguistics.
  * Bizer (2020) George Y Bizer. 2020.  Who’s bothered by an unfair world? the emotional response to unfairness scale.  _Personality and Individual Differences_ , 159:109882.
  * Brickey et al. (2012) Jonalan Brickey, Steven Walczak, and Tony Burgess. 2012.  Comparing semi-automated clustering methods for persona development.  _IEEE Transactions on Software Engineering_ , 38(3):537–546.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cheng et al. (2023) Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023.  Marked personas: Using natural language prompts to measure stereotypes in language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1504–1532, Toronto, Canada. Association for Computational Linguistics.
  * Choi et al. (2023) Minje Choi, Jiaxin Pei, Sagar Kumar, Chang Shu, and David Jurgens. 2023.  Do llms understand social knowledge? evaluating the sociability of large language models with socket benchmark.
  * Chu et al. (2018) Eric Chu, Prashanth Vijayaraghavan, and Deb Roy. 2018.  Learning personas from dialogue with attentive memory networks.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2638–2646, Brussels, Belgium. Association for Computational Linguistics.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.
  * Computer (2023) Together Computer. 2023.  Redpajama: An open source recipe to reproduce llama training dataset.
  * Dalbert (1999) Claudia Dalbert. 1999.  The world is more just for me than generally: About the personal belief in a just world scale’s validity.  _Social justice research_ , 12:79–98.
  * Dalbert et al. (2001) Claudia Dalbert, Isaac M Lipkus, Hedvig Sallay, and Irene Goch. 2001.  A just and an unjust world: structure and validity of different world beliefs.  _Personality and Individual Differences_ , 30(4):561–577.
  * Feng et al. (2023) Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023.  From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 11737–11762, Toronto, Canada. Association for Computational Linguistics.
  * Forsyth (1980) Donelson R Forsyth. 1980.  A taxonomy of ethical ideologies.  _Journal of Personality and Social psychology_ , 39(1):175.
  * Furnham and Grover (2020) Adrian Furnham and Simmy Grover. 2020.  A new money behavior quiz.  _Journal of Individual Differences_ , 41:17–29.
  * García-Ferrero et al. (2023) Iker García-Ferrero, Begoña Altuna, Javier Álvez, Itziar Gonzalez-Dios, and German Rigau. 2023.  This is not a dataset: A large negation benchmark to challenge large language models.
  * Ghosh et al. (2022) Soumitra Ghosh, Dhirendra Kumar Maurya, Asif Ekbal, and Pushpak Bhattacharyya. 2022.  EM-PERSONA: EMotion-assisted deep neural framework for PERSONAlity subtyping from suicide notes.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 1098–1105, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Jang et al. (2023) Joel Jang, Seonghyeon Ye, and Minjoon Seo. 2023.  Can large language models truly understand prompts? a case study with negated prompts.  In _Proceedings of The 1st Transfer Learning for Natural Language Processing Workshop_ , volume 203 of _Proceedings of Machine Learning Research_ , pages 52–62. PMLR.
  * Jessica A. Jordan and Bosson (2021) Joanna R. Lawler Jessica A. Jordan and Jennifer K. Bosson. 2021.  Ambivalent classism: The importance of assessing hostile and benevolent ideologies about poor people.  _Basic and Applied Social Psychology_ , 43(1):46–67.
  * Jiang et al. (2023) Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2023.  Evaluating and inducing personality in pre-trained language models.
  * Lee et al. (2022) Young-Jun Lee, Chae-Gyun Lim, Yunsu Choi, Ji-Hui Lm, and Ho-Jin Choi. 2022.  PERSONACHATGEN: Generating personalized dialogues using GPT-3.  In _Proceedings of the 1st Workshop on Customized Chat Grounding Persona and Knowledge_ , pages 29–48, Gyeongju, Republic of Korea. Association for Computational Linguistics.
  * Li et al. (2023) Yunpeng Li, Yue Hu, Yajing Sun, Luxi Xing, Ping Guo, Yuqiang Xie, and Wei Peng. 2023.  Learning to know myself: A coarse-to-fine persona-aware training framework for personalized dialogue generation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 37(11):13157–13165.
  * Liu et al. (2022) Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, and Soroush Vosoughi. 2022.  Quantifying and alleviating political bias in language models.  _Artificial Intelligence_ , 304:103654.
  * Lucy and Bamman (2021) Li Lucy and David Bamman. 2021.  Gender and representation bias in GPT-3 generated stories.  In _Proceedings of the Third Workshop on Narrative Understanding_ , pages 48–55, Virtual. Association for Computational Linguistics.
  * Lux et al. (2021) Andrei Lux, Steven Grover, and Stephen Teo. 2021.  Development and validation of the holistic cognition scale.  _Frontiers in Psychology_ , 12:551623.
  * Min et al. (2023) Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023.  Recent advances in natural language processing via large pre-trained language models: A survey.  _ACM Computing Surveys_ , 56(2):1–40.
  * Miotto et al. (2022) Marilù Miotto, Nicola Rossberg, and Bennett Kleinberg. 2022.  Who is GPT-3? an exploration of personality, values and demographics.  In _Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS)_ , pages 218–227, Abu Dhabi, UAE. Association for Computational Linguistics.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022.  Crosslingual generalization through multitask finetuning.  _arXiv preprint arXiv:2211.01786_.
  * Nahathai Wongpakaran and Kuntawong (2020) Tinakon Wongpakaran Nahathai Wongpakaran and Pimolpun Kuntawong. 2020.  Development and validation of the (inner) strength-based inventory.  _Mental Health, Religion & Culture_, 23(3-4):263–273.
  * Pan and Zeng (2023) Keyu Pan and Yawen Zeng. 2023.  Do llms possess a personality? making the mbti test an amazing evaluation for large language models.
  * Patrick et al. (2002) Christopher J Patrick, John J Curtin, and Auke Tellegen. 2002.  Development and validation of a brief form of the multidimensional personality questionnaire.  _Psychological assessment_ , 14(2):150.
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023.  The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only.  _arXiv preprint arXiv:2306.01116_.
  * Poropat (2009) Arthur Poropat. 2009.  A meta-analysis of the five-factor model of personality and academic performance. _Psychological bulletin_ , 135 2:322–38.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Salminen et al. (2020) Joni Salminen, Joao M Santos, Haewoon Kwak, Jisun An, Soon-gyo Jung, and Bernard J Jansen. 2020.  Persona perception scale: development and exploratory validation of an instrument for evaluating individuals’ perceptions of personas.  _International Journal of Human-Computer Studies_ , 141:102437.
  * Santurkar et al. (2023) Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023.  Whose opinions do language models reflect?  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org.
  * Sap et al. (2022) Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022.  Neural theory-of-mind? on the limits of social intelligence in large LMs.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3762–3780, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Sclar et al. (2023) Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023.  Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.
  * Spence et al. (1974) Janet T Spence, Robert Helmreich, and Joy Stapp. 1974.  Personal attributes questionnaire.  _Developmental Psychology_.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * V Ganesan et al. (2023) Adithya V Ganesan, Yash Kumar Lal, August Nilsson, and H. Schwartz. 2023.  Systematic evaluation of GPT-3 for zero-shot personality estimation.  In _Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis_, pages 390–400, Toronto, Canada. Association for Computational Linguistics.
  * Van Der Zee and Van Oudenhoven (2000) Karen I Van Der Zee and Jan Pieter Van Oudenhoven. 2000.  The multicultural personality questionnaire: A multidimensional instrument of multicultural effectiveness.  _European journal of personality_ , 14(4):291–309.
  * Xu et al. (2022) Xinchao Xu, Zhibin Gou, Wenquan Wu, Zheng-Yu Niu, Hua Wu, Haifeng Wang, and Shihang Wang. 2022.  Long time no see! open-domain conversation with long-term persona memory.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2639–2650, Dublin, Ireland. Association for Computational Linguistics.
  * Zhu et al. (2023) Luyao Zhu, Wei Li, Rui Mao, Vlad Pandelea, and Erik Cambria. 2023.  PAED: Zero-shot persona attribute extraction in dialogues.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 9771–9787, Toronto, Canada. Association for Computational Linguistics.

## Appendix

### Sample Persona Instruments

Table 3 contains examples of personas, their corresponding instrument set, and
an example question that is used as a prompt for the LLM.

Persona | Instrument Set | Example
---|---|---
ProImmigration | AIS |  Immigrants should have the same right to social security as everyone else.
ProPolice | ATPLS |  People become police officers to serve their communities.
Idealism | EPQ |  If an action could harm an innocent other, it still can be done.
Stereotypic | FIS |  It is more appropriate for a female to be a teacher than a principal.
Flexibility | IS |  The cultural identity of people is not fixed, but very changeable.
Pro-Military | MAS |  The military should always be kept strong.
Extrovert | MBTI |  I enjoy expending energy and enjoy groups.
Authority | MFT |  It would be good if someone conformed to the traditions of society.
Pro-Military | MAS |  The military should always be kept strong.
Virtue | MHBS |  Physical aggression is always admirable and acceptable.
Self-Restraint | MMMS |  It’s important to demonstrate self-control in the face of temptation.
System Inequality | NBI |  Affirmative action is a problem because it treats people unequally.
Neuroticism | OCEAN |  I am relaxed most of the time.
Definition | ONBGS |  Sexual organs necessarily have to match gender.
Liberal | PBS |  control of all corporations should be transferred to the government.
Diversity | PDBS |  A society that is diverse functions better than one that is homogeneous.
Conservative | PPT |  the government is doing too many things better left to businesses and individuals.
Utopia | SIBS |  Everything that happens to a person is valuable.
Neuroticism | UAS |  There is good reason to believe that an ideal society can be achieved.
Emotionality | VES |  I am interested in the feelings of others.
Table 3: Sample Persona Instruments

### Negation Examples

Table 4 contains examples of reversing the meaning of a sentence via both
direct and paraphrastic negation.

Negation Type | Instrument
---|---
Original |  Poor people often lack a competitive drive to get ahead.
Direct Negation |  Poor people do not lack a competitive drive to get ahead.
Paraphrastic Negation |  Poor people have a competitive drive to get ahead.
Original |  If an action could harm an innocent other, it still can be done.
Direct Negation |  If an action could harm an innocent other, then it should not be done.
Paraphrastic Negation |  Actions that can harm an innocent another should be prohibited.
Table 4: Sample Negated Instruments

### Detailed Prompt Variants

Table 5 shows the format of every prompt variant that was used to evaluate an
LLM’s comprehensibility and sensitivity.

Prompt Variant | Example
---|---
Colon Ending* |  Statement:\n<Statement>\nQuestion:\nDo you agree with the statement? Reply with only ’Yes’ or ’No’ without explaining your reasoning.\nAnswer:\n<Answer>
Question-Mark Ending |  Statement:\n<Statement>\nQuestion:\nAnswer the following question with only ’True’ or ’False’ without explaining your reasoning. Do you think the statement is true or false?\n<Answer>
Colon + Line-Break* |  Statement:\n<Statement>\nQuestion:\nDo you agree with the statement? Reply with only ’Yes’ or ’No’ without explaining your reasoning.\nAnswer:\n<Answer>
Colon + No Space |  Statement:<Statement>\nQuestion:Do you agree with the statement? Reply with only ’Yes’ or ’No’ without explaining your reasoning.\nAnswer:<Answer>
Colon + Single Space |  Statement: <Statement>\nQuestion: Do you agree with the statement? Reply with only ’Yes’ or ’No’ without explaining your reasoning.\nAnswer: <Answer>
Colon + Double Space |  Statement: <Statement>\nQuestion: Do you agree with the statement? Reply with only ’Yes’ or ’No’ without explaining your reasoning.\nAnswer: <Answer>
Answer:* |  Statement:\n<Statement>\nQuestion:\nDo you agree with the statement? Reply with only ’Yes’ or ’No’ without explaining your reasoning.\nAnswer:\n<Answer>
Response: |  Statement:\n<Statement>\nQuestion:\nDo you agree with the statement? Reply with only ’Yes’ or ’No’ without explaining your reasoning.\nResponse:\n<Answer>
Line-Break Separated* |  Statement:\n<Statement>\nQuestion:\nDo you agree with the statement? Reply with only ’Yes’ or ’No’ without explaining your reasoning.\nAnswer:\n<Answer>
Single Space Separated |  Statement: <Statement> Question: Do you think the statement is true or false? Reply with only ’True’ or ’False’ without explaining your reasoning. Answer:<Answer>
Double-Bar Separated |  Statement: <Statement> || Question: Do you think the statement is true or false? Reply with only ’True’ or ’False’ without explaining your reasoning. || Answer:<Answer>
Triple-Sharp Separated |  ### Statement:\n<Statement>\n### Question:\nDo you think the statement is true or false? Reply with only ’True’ or ’False’ without explaining your reasoning.\n### Answer:\n<Answer>
Table 5: Prompt Variants Examples

### Persona Leanings across All Instruments

Figures 3, 4, 5 show a comparison of persona leaning scores across every
single instrument for the FLAN-T5-Large and FLAN-T5-XL models.

Figure 3: A comparison of LLMs on their leanings for all dimensions (1) Figure
4: A comparison of LLMs on their leanings for all dimensions (2) Figure 5: A
comparison of LLMs on their leanings for all dimensions (3)


# Blar-SQL: Faster, Stronger, Smaller NL2SQL

José Manuel Domínguez<EMAIL_ADDRESS>Blar SpaSantiagoChile ,  Benjamín
Errazuriz<EMAIL_ADDRESS>Blar SpaSantiagoChile and  Patricio Daher
<EMAIL_ADDRESS>Blar SpaSantiagoChile

(2023; 24 October 2023)

###### Abstract.

Large Language Models (LLMs) have gained considerable notoriety in the field
of natural language to SQL tasks (NL2SQL). In this study, we show how task
decomposition can greatly benefit LLMs in database understanding and query
generation in order to answer human questions with an SQL query.

We fined-tuned open source models, specifically Llama-2 and Code Llama, by
combining 2 different models each designated to focus on one of two tasks in
order to leverage each model’s core competency to further increase the
accuracy of the final SQL query.

We propose a new framework to divide the schema into chunks in order to fit
more information into a limited context. Our results are comparable with those
obtained by GPT-4 at the same time being ~135 times smaller, 90 times faster
and more than 100 times cheaper than GPT-4.

NL2SQL, LLM, AI, Prompt Decomposition

††journalyear: 2023

##  1\. Introduction

With recent advances in LLMs such as the Open AI’s GPT-4 and Meta’s Llama-2,
the field of translating human language questions into SQL queries, commonly
referred to as NL2SQL, has witnessed significant advancements. These state-of-
the-art language models have opened up exciting opportunities for improving
the accuracy and efficiency of NL2SQL systems, enabling the end user to
interact with databases without having a deep understanding of SQL language or
the database’s schema.

Currently, most of the SOTA approaches use LLMs to achieve this goal,
particularly GPT-4, which has gained most of the attention as being the
largest and most capable model as of this paper’s writing. In this paper, we
explore how open-source models such as Llama-2 and CodeLlama when fine-tuned
can achieve similar or even better results than the ones achieved by GPT-4.
Moreover, we explore how dividing complex tasks into smaller steps can
significantly benefit these models. We build on DIN-SQL’s approach (Pourreza
and Rafiei, 2023) and fine-tune two custom models, each specialized in
completing one sub-task achieving results comparable with GPT-4 on the BIRD-
SQL data set.

Another challenge frequently encountered in the text-to-SQL domain is managing
context. Large data schemas often exceed the 32K token context limit of the
GPT-4 model. To address this, we have introduced a novel method for selecting
the necessary schema link to answer a given query. We partition the schema
into several prompts, each tailored to fit within the maximum context length.
This approach enables models with a context limit of 4K tokens or less to
efficiently process databases with extensive schemas and descriptions.

The contributions of this study are summarized:

  1. (1)

We divide the task of generating SQL queries into 2 steps, the first is
choosing the appropriate columns based on the database schema, column
descriptions and other external knowledge that may be required.

  2. (2)

A new framework of sub-dividing very large schemas and descriptions into
chunks in order to better manage prompt context and include the greatest
amount of information possible.

  3. (3)

A new idea of using 2 different models trained with different purposes in
order to leverage their knowledge and create better results.

##  2\. Related Work

Decomposing complex text-to-SQL tasks has shown great potential, improving the
performance of few shots LLMs on the Spider dataset by over 10% (Pourreza and
Rafiei, 2023). The main idea behind this is that LLMs respond better when only
one task is asked, to achieve this you break the problem into simple pieces
and solve each part using prompting, trained models, or symbolic functions
(Khot et al., 2022).

SDSQL model introduces the notion of capturing the interactions between the
question and the data schema. It uses a series of dependency labels that link
question tokens with schema tokens (W-Col, S-Agg, W-Op).

There are several datasets designed for text-to-SQL research, such as Spider
(Yu et al., 2018) and WikiSQL (Zhong et al., 2017). These datasets not only
provide a benchmark for comparing various approaches, but they also supply the
data necessary for fine-tuning models and achieving significant improvements.
In this realm, the BIRD dataset stands out. It encompasses a 33.4GB database
emphasizing real-world applications. This benchmark offers a detailed insight
into how our frameworks would perform with actual databases, making it the
most rigorous benchmark available (Li et al., 2023).

##  3\. Methodology

###  3.1. Error analysis

In order to further understand the current capabilities of open-source LLMs we
first evaluated the vanilla version of code-llama against the development
dataset of both Spider and Bird. We later took a random subset of 60 data
points and classified the errors in order to understand what should be
improved.

The most common error we found where schema linking, this occurred when the
model changed the name of some columns or hallucinated columns and table-
column associations. The second most common error was when the model failed to
understand the content of the database, this error occurred when the model
couldn’t comprehend the column description and values. Figures 1 and 2 show
examples of both errors.

Figure 1. Example of a Schema Link Error

In this example the model got wrong the name of the columns ’CDSCODE’ and
’MailStreet’

Figure 2. Example of a database content error

In this example the model assumed that the column ’Charter School (Y/N)’ had
values ”Y” or ”N”, but in reality the values are 1 or 0

Our results coincide with the results obtained by the team that developed the
Bird data set (Li et al., 2023). The two most common mistakes were ”Wrong
Schema Linking” (41.6%) and ”Misunderstanding Database Content” (40.8%), both
occurred because the LLM couldn’t understand the database schema correctly or
recall the correct structure of the database.

In order to further improve LLMs capability we divided the process into two
steps: 1) inferring the schema links (Schema-Link model) and 2) Constructing
the SQL using the previously generated schema links (SQL model). The overall
architecture is shown in figure 3. The idea is to train both models
specifically for their task because it has been proven that dividing a problem
into smaller tasks generates better results when it comes to LLM inference
(Khot et al., 2022).

Figure 3. Prediction solution architecture

Here you can see in the first box the question and the database schema, which
the Schema-Link model take and infer the tables and columns needed to answer
the question. With the schema link the SQL model predicts the final SQL to
answer the question.

###  3.2. Making the data set

As we mentioned before we used the Bird dataset because is the first benchmark
focused on big databases with real applications. Thanks to the work done by
the Bird team, you can access 95 databases with 12,751 questions with their
respective answer (Li et al., 2023).

In order to create the schema link dataset we reverse-engineered the SQL
answers and the schema. This process consisted of extracting only the tables
and columns used in the query answers as well as the foreign keys. Figure 4
shows the process of schema link generation. In order to achieve this we used
a combination of SQL metadata capturing packages (sql-metadata and sql-parse)
for Python combined with custom-made Regex.

Figure 4. Schema Link Generation Process

###  3.3. Hypothesis

Our solution consists of two models working on two separate tasks, making the
schema linking and generating the SQL query. The way that we train each model
can impact how these two models interact with each other to come up with a
solution. Because of this, we created two hypotheses:

  1. (1)

A non-trusting interaction: where the SQL model does not rely only on the
Schema-Link output, it also has access to the original schema and column
descriptions. The schema link are presented as a suggestion more than a fact.
By ensuring that the SQL model does not solely rely on the output of the
first, there’s an added opportunity to correct potential misinterpretations of
the schema.

  2. (2)

A trusting interaction where the SQL model only takes the output of the
Schema-Link model to construct the query. This enables us to manage the
context length as well as teach the model to focus purely on the construction
of the SQL query and the logic behind it.

Furthermore, we hypothesized that using a Llama-2 model for the schema linking
part would be beneficial as it has a better world understanding (Rozière et
al., 2023) and using code-llama for the SQL part, as it has a better
programming background (Touvron et al., 2023). Unfortunately, due to a lack of
time and resources we couldn’t compare how a pure llama-2 and pure code-llama
solution would perform, so we leave this as a future work.

###  3.4. Context Management

One big limitation that open-source solutions such as llama-2, code-llama or
others have, is the smaller context they have in comparison with Open AI’s GPT
models. Even GPT-4 32K context may fall short when the database is big enough
or if it has detailed column descriptions.

We experimented with two approaches in order to tackle this problem. The first
one consisted of reducing the prompt context by excluding certain information
such as column descriptions if it exceeded the context. This traditional
approach has the limitation of the volume of context that we can provide our
models to make a prediction. We call this the non-descriptive approach.

The second approach is a more novel one, in which we propose a new method of
dividing the schema linking step according to the maximum context length
available.

First, we started by creating the template of the prompt which consisted of
four parts. The question, the database schema, column descriptions and finally
a hint or external knowledge required in order to answer the question.

Then, we iterated over each table’s columns and descriptions, adding one table
at a time until the prompt could not further fit another table. The question
and external knowledge were fixed in order to always be included. We then
divided each question using this method in one or more prompts.

Finally, we extracted the schema links from each segmented prompt, producing a
list of schema link predictions. When concatenated, this list provided the
essential information required to answer the question. We call this the schema
chunking approach and Figure 5 explains the whole process.

Figure 5. Schema chunking process

###  3.5. Training process

To train the models we used Google Colab’s A100 40GB GPU. To achieve this we
used 4bit model quantization as well as QLoRA for fine-tuning. The training
itself used a combination of the packages available in huggingface such as
transformers, peft and trl. The training process usually lasted ~10hrs costing
about $20 USD per model.

We did supervised fine-tuning only updating weights from the response (Schema
links and SQL queries) similar to the approach taken by Dail-SQL (Gao et al.,
2023). For the dataset, we used BIRD’s Training set consisting of 8,952
examples, the dataset was subsequently partitioned randomly, resulting in
7,609 examples left for training, while 1,343 were set aside for validation
purposes.

We conducted the training process for a maximum of two epochs, evaluating the
model every 500 steps and selecting the best models based on the validation
set.

####  3.5.1. Schema Link models

  * •

Non-descriptive: In this approach, there was one single prompt for each
question, to achieve this we removed the column descriptions if they couldn’t
be fitted in the context due to length limitations. The model’s task was to
predict the whole schema link generated previously. Using this approach some
examples of the training set were omitted as the pure schema wouldn’t fit in
the context.

  * •

Chunked: In this approach, each question had N prompts, each with a maximum
length of 4,096 tokens, as we were using the Llama-2 models. Both the schema
and descriptions were included in each prompt preventing information loss. The
model’s task was to predict only the schema links present in the prompt and
predict None if none of them were present.

####  3.5.2. Not trusting hypothesis

The training process begins by first training the schema linking model. After
fine-tuning this model, we generated schema links for the entire data set.
Subsequently, the model that generates the SQL is later trained using the
schema link deduced by the first model and a resume version of the original
schema. This approach offers the advantage of distributing the risk of
inferring the structure of the database wrong to answer the question between
the two models.

One notable limitation of this approach is that the schema link model
generates better results for the training dataset, as the model has previously
”seen” these examples, potentially biasing the SQL model to be more trustful
of the schema link recommendations. However, due to the limited data, we
decided to make this trade-off rather than further splitting the training
dataset.

####  3.5.3. Trusting hypothesis

Both the Schema-Link model and the SQL model are trained using the schema
links constructed using reverse engineering from the BIRD dataset. This
creates a trusting scenario where the SQL model trusts the output given by the
schema link model, focusing solely on creating a syntactically and logically
correct SQL query.

##  4\. Results

We tested various models and combinations, the following section describes
each one of them and the results obtained. We used the Dev partition of the
BIRD-dataset and used the official Bird code in order to evaluate our models.

###  4.1. Vanilla

First, we tested how the vanilla version of code-llama 13B would perform in
the Bird-SQL dataset. In order to fit the schema and descriptions into the
prompt we used a non-descriptive approach (${ND}_{p}$), that would leave out
the column descriptions if they didn’t fit in the prompt. We used 5000 tokens
as the threshold.

In the prompt was specified that we only wanted the SQL query with no
explanation but sometimes the model would also include an explanation of the
query against what was prompted.

This led us to divide the vanilla model into two results ”Vanilla” ($V$),
which evaluates results exactly as the model outputs them, and ”vanilla clean”
($V_{c}$) where we employed a post-processing step to extract only the SQL
query from the generated results.

Table 1 describes the execution accuracy results.

We conducted this test in order to understand the current capabilities of the
models and set a floor value in order to improve upon it.

Table 1. Results Vanilla models Model | Simple | Moderate | Challenging | Total
---|---|---|---|---
${ND}_{p}$ \+ $V$ | 20.76% | 5.81% | 1.39% | 14.41%
${ND}_{p}$ \+ $V_{c}$ | 23.24% | 7.96% | 4.17% | 16.82%

###  4.2. Zero shot fine-tune

Later we fine-tuned a single code-llama 13B model in order to predict the SQL
answer in a zero-shot environment (SQLft). In this model, we again used a non-
descriptive approach (${ND}_{p}$) for the prompt as the only task required was
to generate the final SQL query and this generation could not be divided into
multiple prompts. The results are in the table 2. These results give us a
comparison point between a one-step process and our approach of sub-dividing
the problem into multiple steps.

Table 2. Results direct fine-tune Model | Simple | Moderate | Challenging | Total
---|---|---|---|---
${ND}_{p}$ \+ $V$ | 20.76% | 5.81% | 1.39% | 14.41%
${ND}_{p}$ \+ $V_{c}$ | 23.24% | 7.96% | 4.17% | 16.82%
${ND}_{p}$ \+ SQLft | 35.79% | 15.05% | 5.56% | 26.73%

###  4.3. Non-Trusting

For our model combination, we first evaluated our non-trusting approach
($NT$). In this approach, the SQL model was trained with the schema links
predicted by the Schema-Link model which were not always correct thus creating
a non-trusting environment. This model was evaluated using a non-descriptive
($ND_{p}$) and a chunk approach ($CH_{p}$) for the schema link model.

In the chunk approach ($CH_{p}$), we divide the schema with the column
descriptions in N prompts, each prompt having a maximum threshold length. We
generate predictions for each prompt and later join them to create the final
schema link prediction.

Initially, we generated schema link predictions for both the non-descriptive
and chunked schema link models. Subsequently, we provided these predictions to
the SQL model, along with the schema and column descriptions incorporating the
latter when they could be accommodated within the context without exceeding
it.

Table 3. Results Non-Trusting Model | Simple | Moderate | Challenging | Total
---|---|---|---|---
${ND}_{p}$ \+ $V$ | 20.76% | 5.81% | 1.39% | 14.41%
${ND}_{p}$ \+ $V_{c}$ | 23.24% | 7.96% | 4.17% | 16.82%
${ND}_{p}$ \+ SQLft | 35.79% | 15.05% | 5.56% | 26.73%
${ND}_{p}$ \+ $SL$ \+ $NT$ | 42.16% | 25.59% | 19.44% | 35.01%
${CH}_{p}$ \+ $SL$ \+ $NT$ | 52.97% | 39.35% | 29.86% | 46.68%

###  4.4. Trusting

In the ”trusting” version of the model, we generated the schema link using the
chunk approach immediately (${CH}_{p}$ \+ $SL$ \+ $T$). Subsequently, only the
question and this prediction were prompted to the SQL model.

Table 4. Results Trusting Model | Simple | Moderate | Challenging | Total
---|---|---|---|---
${ND}_{p}$ \+ $V$ | 20.76% | 5.81% | 1.39% | 14.41%
${ND}_{p}$ \+ $V_{c}$ | 23.24% | 7.96% | 4.17% | 16.82%
${ND}_{p}$ \+ SQLft | 35.79% | 15.05% | 5.56% | 26.73%
${ND}_{p}$ \+ $SL$ \+ $NT$ | 42.16% | 25.59% | 19.44% | 35.01%
${CH}_{p}$ \+ $SL$ \+ $NT$ | 52.97% | 39.35% | 29.86% | 46.68%
${CH}_{p}$ \+ $SL$ \+ $T$ | 32.22% | 27.20% | 9.03% | 25.49%

###  4.5. Upper Bound

Finally, we tested our model’s upper bound, specifically the SQL model’s upper
bound, we did this in order to further understand our model’s limitations. To
test this we prompted the SQL model with the perfect schema links $SL_{p}$
extracted as detailed previously. This way the model would have the perfect
recommendation of columns, tables and foreign keys to use.

We also tested how the non-trusting model would perform if only the schema
link and external knowledge were prompted in order to not confuse the LLM with
extra content, the extra information was not necessary given that the schema
links are perfect ($NT_{sl}$), Table 5 details our results.

Table 5. Results Upper Bound Model | Simple | Moderate | Challenging | Total
---|---|---|---|---
$SL_{p}$ \+ $NT$ | 55.78 % | 41.72% | 29.86% | 49.09%
$SL_{p}$ \+ $T$ | 53.73% | 29.03% | 24.31% | 43.48%
$SL_{p}$ \+ $NT_{sl}$ | 35.46% | 14.84% | 11.81% | 26.99%

###  4.6. Analysis

In this section, we are going to contrast the results of the different models
to see if our hypothesis were correct or not.

####  4.6.1. Vanilla v/s Zero shot fine-tune

The best vanilla results were 16.82% ($ND_{p}+V_{c}$) and the worst of the
fined-tuned models got 26.73%. This shows an improvement of ~10% proving the
hypothesis that trained models such as Llama get much better results when
fine-tuned and are a powerful tool in resolving text-to-SQL problems.

####  4.6.2. Trusting v/s Zero shot fine-tune

The Trusting version did not perform well, as we can see in Table 4, getting
only 25.49% losing against the ${ND}_{p}$ \+ SQLft version.

This is due to the amount of responsibility that the schema linking step took.
The input of the SQL model was solely the schema link, limiting the amount of
context that the used model to create the SQL prediction. What happened in the
majority of cases was that the $SL$ failed to capture a column needed or
captured the name wrong, giving zero chance to the next step to get the answer
well. From this, we learned two things:

  1. (1)

Getting the exact amount of information from the data schema that is needed to
answer the question is not effective. It’s better for the model to include
extra columns than to risk having too few to make the right SQL prediction.

  2. (2)

Prompting the schema to the SQL model gives the chance to fix the $SL$ output,
spreading the risk of not understanding the database architecture and
enhancing the performance of the model greatly.

####  4.6.3. Non-Trusting v/s Zero shot fine-tune

Comparing the results between our non-trusting approach and a zero-shot fine
tune we can see a big spike in performance, providing evidence that dividing
the tasks between models is beneficial when creating SQL queries. The
improvement was ~10% for a non-descriptive setup (similar comparisons) and a
~20% improvement for a chunked setup, providing further evidence that chunking
the schema can lead to better results.

####  4.6.4. Trusting v/s Non-Trusting Upper Bound

To our surprise, the Non-trusting model performed better than the trusting
one, this is a surprise because the trusting model was trained to create
queries based solely on the schema links prompted to it, leaving it only the
responsibility of creating SQL queries. A reason this could be the case is
that after two epochs the trusting SQL model continued to learn i.e. the loss
function on the validation set continually decreased and was stopped at the
end of the second epoch.

To our surprise the model with just the schema linking ($NT_{sl}$) performed
worse, this could be due to the fact that it was fine-tuned with a different
prompt layout confusing the model more than helping it. One key takeaway is
that fine-tuned models are very sensitive to prompt changes, further studies
are required in order to explore this hypothesis.

##  5\. Conclusion

Task division can be incredibly beneficial for LLM models, many agents and
prompt chaining approaches have risen lately. In this study we proved how task
division can be beneficial for NL2SQL tasks, increasing performance by more
than 10% compared to a direct approach. We also explored the idea and proposed
a new framework to divide schema linking prompts in order to capture as much
information as possible for the database schema and descriptions, this helped
us improve by a further 10% when compared to the non-descriptive approach.
Finally, we tried combining two different models in order to further enhance
the performance. This study proves that smaller fine-tuned and task-specific
approaches can deliver competitive results compared to GPT-4, it also presents
an affordable way of fine-tuning models using easy access Google-colab
notebooks.

##  6\. Limitations

There are a few significant limitations of this study, first of all, we
couldn’t test the impact of having 2 different models each specialized in one
task, we encourage others to further validate the hypothesis that using LLama
2 for the schema link and Code-Llama for the SQL generation is actually
beneficial.

The second limitation we encountered was coming up with a chunking method that
would prevent information loss. Sometimes during our chunking two tables
joined by a foreign key would get split up, although the table information
containing the foreign key reference remained, the fact that the model didn’t
have access to the other table and its column could play an important part in
picking the correct columns.

Finally, due to time limitations, we couldn’t study what the new error
composition looked like, this could lead to an interesting study of further
modularizing the steps needed in order to generate a correct SQL.

##  7\. Acknowledgments

Special thanks to Hans Löbel for his constant help and guidance during this
process.

## References

  * (1)
  * Gao et al. (2023) Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023.  Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation.  _arXiv preprint arXiv:2308.15363_ (2023).
  * Khot et al. (2022) Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022.  Decomposed prompting: A modular approach for solving complex tasks.  _arXiv preprint arXiv:2210.02406_ (2022).
  * Li et al. (2023) Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, et al. 2023\.  Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls.  _arXiv preprint arXiv:2305.03111_ (2023).
  * Pourreza and Rafiei (2023) Mohammadreza Pourreza and Davood Rafiei. 2023.  Din-sql: Decomposed in-context learning of text-to-sql with self-correction.  _arXiv preprint arXiv:2304.11015_ (2023).
  * Rozière et al. (2023) Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023\.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_ (2023).
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023\.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ (2023).
  * Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018\.  Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task.  _arXiv preprint arXiv:1809.08887_ (2018).
  * Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017.  Seq2sql: Generating structured queries from natural language using reinforcement learning.  _arXiv preprint arXiv:1709.00103_ (2017).