1]Computer Science, Princeton University, Princeton, NJ, USA

2]Computer Science & Engineering, University of Washington, Seattle, WA, USA

3]Computer Science, University of Maryland, College Park, MD, USA

4]Arthur, New York City, NY, USA

# Large language models cannot replace human participants because they cannot
portray identity groups

Angelina Wang<EMAIL_ADDRESS>Jamie Morgenstern
<EMAIL_ADDRESS>John P. Dickerson<EMAIL_ADDRESS>[  [  [  [

###### Abstract

Large language models (LLMs) are increasing in capability and popularity,
propelling their application in new domains—including as replacements for
human participants in computational social science [1], user testing [2],
annotation tasks [3], and more [4, 5]. Traditionally, in all of these settings
survey distributors are careful to find representative samples of the human
population to ensure the validity of their results and understand potential
demographic differences [6]. This means in order to be a suitable replacement,
LLMs will need to be able to capture the influence of positionality (i.e.,
relevance of social identities like gender and race). However, we show that
there are two inherent limitations in the way current LLMs are trained that
prevent this. We argue analytically for why LLMs are doomed to both misportray
and flatten the representations of demographic groups, then empirically show
this to be true on 4 LLMs through a series of human studies with 3200
participants across 16 demographic identities. We also discuss a third
consideration about how identity prompts can essentialize identities.
Throughout, we connect each of these limitations to a pernicious history that
shows why each is harmful for marginalized demographic groups. Overall, we
urge caution in use cases where LLMs are intended to replace human
participants whose identities are relevant to the task at hand. At the same
time, in cases where the goal is to supplement rather than replace (e.g.,
pilot studies), we provide empirically-better inference-time techniques to
reduce, but not remove, these harms. 111This work was initiated during
Angelina Wang’s internship at Arthur.

###### keywords:

large language model limitations, human participants, representative sampling,
standpoint epistemology

## Introduction

Large language models (LLMs) are proliferating, and increasingly touted as
being able to replace more costly human participants in a variety of domains
such as user studies [2], annotation tasks [7], computational social science
[1], opinion surveys [5], and more. However, in the surge of excitement it
often seems forgotten what remains one of the biggest challenges in human
participant recruitment: representative sampling [6]. Even in cases where
representative sampling is not explicitly pursued, the demographic identity of
each participant is often collected out of recognition that it impacts each
person’s positionality and thus response [8]. When Amazon Mechanical Turk was
beginning to be used as a replacement for traditional recruitment for human
participants, there were concerns about the validity of this new domain, and
research studied the demographics of the new platform [9]. Now, in this far
greater paradigm shift, we cannot neglect to consider this key component of
validity: demographic differences. This means that the ability of LLMs to
replace human participants is wholly contingent on LLMs being able to
represent the perspectives of different demographic identities. Prior work has
speculated that LLMs’ vast training data enables it to do precisely this, and
discussed the enormous implications for social science research [4]. In this
work, we bring empirical clarity to these claims by comparing LLM responses to
human participant responses. We outline technical and ethical considerations
for two key limitations, misportrayal (e.g., when asked to represent the
perspective of a person with impaired vision’s perspective on immigration,
generations of unlikely phrases like “While I may not be able to visually
observe the nuances of the US-Mexican border or read statistics, I believe in
the importance of fair and just immigration policies”) and group flattening
(e.g., LLMs missing that not all non-binary people use they/them pronouns),
that prevent LLMs from fully representing demographic perspectives. We also
bring up a third consideration around identity essentialization (i.e.,
reducing identities to fixed characteristics) that arises in even a seemingly
more permissible setting: when prompts are used to increase response coverage.
We therefore caution against the replacement of human participants by LLMs,
but also offer concrete recommendations about empirically-better inference-
time techniques when a particular instance of replacement (e.g., in pilot
studies222We note that in certain situations, to be maximally risk-averse,
LLMs should only be used in pilot studies for a human pilot study.) is deemed
permissible.

Our findings are based on two fundamental limitations in the current way LLMs
are trained that will likely prevent even newer iterations of these models, if
they are trained in the same way, from overcoming these challenges. These
limitations stem from the current training paradigm’s use of a) existing
online text for training data and b) loss function, usually maximum likelihood
estimation, that rewards generating the most likely output.333We do not make
any claims as to the presence of these limitations on training procedures
outside of this current paradigm of human imitation, e.g., pretraining based
on human feedback [10], or if a new dataset were constructed that explicitly
links author demographic identity to text. They will apply in every instance
of an LLM prompted by a demographic identity, which itself is a scenario that
will be present in nearly every instance of human participant replacement.
This is not a speculative concern: researchers are publishing papers about the
ability of LLMs to replace human participants [11, 2, 3, 1, 5, 12, 7, 13], and
companies444Example for-profit firms include https://synthetic-humans.ai/ and
https://www.syntheticusers.com. are deploying products for similar purposes as
well—and it is in exactly these scenarios that we perform our analyses. There
are also closely related but distinct use cases such as where chatbots are
given personas [14, 15, 16]. We do not study the scenarios of these chatbots
so consider them out of scope in that regard, but all of our findings about
the ways LLMs will misportray and flatten demographic groups will persist in
those popular settings, and add a new relevant factor to consider. Prior work
considering the harms of personas in this setting have focused on how
demographic personas change the behavior of the language model [17, 18, 19].
In our use case, we specifically consider cases where we expect demographic
personas to be relevant in model responses, and work here has found that LLMs
prompted with demographic attributes are more stereotypical [20, 21]. We put
forth a complementary analysis on a related but ultimately different set of
harms; a more detailed comparison is in the Supplementary. We do not provide a
uniform condemnation against LLMs prompted with demographic identities, but
rather urge caution by showing exactly how such deployment can be harmful by
grounding the limitation in historical discrimination. These harms cannot be
totally resolved by current iterations of LLMs, but can be reduced, and it
will be up to each deployer to decide whether the specific benefits outweigh
the harms.

The first limitation is that by being trained on scraped text data, author
demographic identity and produced text are rarely associated. There are
exceptions such as when an author’s name, which may be attached to a piece of
text, is identity-coded, or in autobiographical texts where authors may remark
upon their own identity. However, outside of these circumstances, we could
imagine that oftentimes when a demographic identity is textually invoked, it
may be more likely to be from an out-group member speaking about the group,
rather than an in-group member speaking about themselves. For example, it is
documented that historically autism is primarily medicalized by out-group
members about in-group members, rather than in more autobiographical settings
[22]. The implication of this limitation is that when asked to portray the
perspectives of different demographic groups, LLMs may be more likely to align
with out-group discussions rather than genuine in-group representations, the
former of which has been shown to be stereotypical [23]. The second limitation
is that because of loss functions like cross-entropy that are used to train
large language models, models are rewarded for producing the more likely
outputs for any given piece of text. This has the effect of flattening the
representation of certain groups and erasing subgroup heterogeneity (e.g.,
that within women, Black women are different than White women) [24, 25]. This
is especially harmful in the context of flattening demographic groups with a
history of being portrayed one-dimensionally (e.g., Black people).

These two limitations are inherent to the way LLMs are currently trained, and
thus cannot be easily overcome by newer generations of models. We empirically
demonstrate the presence of these limitations on four large language models,
and argue for why it is harmful by connecting each instance to a particular
history and context of discrimination. At the same time, we acknowledge that
in certain situations where the goal is supplementing rather than replacing
human participants, e.g., pilot studies, there may be a desire to push forward
nonetheless and try to reduce these harms. Thus, we also analyze inference-
time alternatives such as prompting with identity-coded names to overcome the
first limitation of lack of author identity linkage with text, and
manipulating the hyperparameter setting of temperature to overcome the second
limitation of flattening of groups. Neither of these techniques are able to
wholly overcome the limitations, but they do improve upon the default setting.
We cautiously provide these actionable suggestions as harm-reduction
techniques in the LLM use-cases that are deemed morally permissible.

Figure 1: Summary. We consider four possible reasons for prompting an LLM with
a demographic identity: when the answer is contingent on identity membership,
when identity is relevant to the answer, when the answer is subjective in a
way where identity might play a role, and where identity is intended to
increase response coverage. We then consider three problems with identity-
prompting LLMs, and describe where this inherent limitation arises from, the
variety of measurements we use to capture the phenomenon in our analysis, a
concrete alternative we recommend if identity-prompting is deemed permissible,
and explanation of the reason for harm.

Finally, we explore the technical limitation of a slightly different reason
one might prompt an LLM with demographic identities: to increase the coverage
of the resulting responses in scenarios like anticipatory work where the goal
is to generate a large range of responses rather than to represent different
groups. Here, we find that prompting with behavioral personas or in some cases
even astrology signs achieves the same effect of increasing coverage as
prompting with sensitive demographic identities does. We argue that if such
coverage can be achieved without unnecessary essentialization of identity, it
likely should be.

To be precise about our concerns, we survey 15 papers studying LLM replacement
of human participants, and delineate the four possible reasons that LLMs might
be prompted with demographic identities (left table in Fig. 1): contingent
perspectives, socially relevant perspectives, subjective annotations, and
coverage-increasing. We then offer insights onto the ethical permissibility of
LLM replacement for each reason.

Overall we demonstrate two fundamental limitations of LLMs in portraying
demographic identities, and argue they are inherent to the format of text data
they are trained on and the loss functions used during training (right table
in Fig. 1). We also discuss a third consideration for even the more innocuous
sounding use case of identity-prompted LLMs to increase coverage. Finally, we
supplement these technical limitations with a discussion of structured ethical
considerations. Our argument is ultimately that LLMs should not replace human
participants. Our caveat is that in a very narrow set of circumstances, such
as some but not all of the cases where the goal is supplementing rather than
replacing, we should employ what we demonstrate to be empirically-better
alternatives (e.g., prompting with identity-coded names rather than identity,
prompting with behavior-based personas).

## Preliminaries

In choosing the demographic identities to both prompt LLMs with as well as
recruit human participants from, we select five demographic axes and three
identities for each, except for the intersectional axis which has four. We end
up with the following axes and 16 total identities: race (Black, White,
Asian), gender (women, men, non-binary people), intersectional (Black women,
Black men, White women, White men), age (Baby Boomer: age 59-77, Millennial:
age 27-42, Generation Z: age 18-26555The lower bound is 18 rather than 10
because of the age requirements for the human studies we run), and disability
(ADD or ADHD; impaired vision like blind, low vision, colorblind; no
disability). Race and gender were selected because names often reveal these
attributes, intersectional was selected to consider the often neglected
intersectionality of demographic attributes in machine learning contexts [26],
and age and disability were selected both because names are less likely to
reveal these attributes, and these are two axes that are frequently neglected
in responsible AI research. It is relevant whether a name reveals the
demographic identity because that is one of the only ways that text data might
attribute the author’s demographic identity. Human participants are recruited
on Prolific and compensated with $12/hour. Institutional IRB determined this
study to be exempt.

To ground our inquiry in the actual reasons one might have for prompting an
LLM with demographic identities, we survey 15 papers studying whether LLMs can
replace human participants, and cluster the reasons they might have for
prompting with demographic identity into four primary categories. Not every
paper prompts with demographic identity (though many do), but we consider that
if the human analog of the task warrants representative sampling, so too does
the LLM version. Our clustered reasons (R) are the following:

  * •

R1-Contingent: answers one can have due to a contingent perspective, i.e.,
where by virtue of having an identity, any response is valid, e.g., what is it
like to be a woman in tech?666We do not have exact examples of Reason 1 (yet),
but with the collective zeitgeist and excitement surrounding LLMs and
emergence of companies promising to replace humans from human studies, e.g.,
https://www.syntheticusers.com, we can imagine this may appear soon, if it is
not already happening.

  * •

R2-Relevant: answers where demographic identity is relevant but not
contingent. As explained by standpoint theory, a person’s perspective is
influenced by their social experience. This is often the motivation behind
representative sampling (e.g., in political opinion polls), and related to the
idea that people of a particular demographic may have a privileged
understanding of different topics, e.g., workplace harassment.777We consider
work about replicating economic and psychology studies on LLMs to fall under
this category as well, though representative sampling is not always sought if
it is assumed that condition randomization sufficiently isolates the
mechanism.  Papers: [5, 2, 27, 28, 29, 30, 31, 32]

  * •

R3-Subjective: annotation tasks like paraphrasing or toxicity labeling that
does have a notion of “ground-truth,” but is ultimately subjective [33, 34,
35, 36, 37]. Papers: [1, 38, 7, 13]

  * •

R4-Coverage: prompting with identities is done to increase the coverage of
viewpoints generated, e.g., user testing a product. Papers: [2, 39, 40, 41,
30]

For any particular study prompting an LLM with a demographic identity, there
may be more than one reason from above that is relevant. However, by
considering them separately, we can have greater clarity in our analysis. For
instance, R4-Coverage is premised on the other three reasons: only if one of
R1-3 applies would prompting with identity hope to increase response coverage.
Because of this, we first investigate only R1-3 for our two inherent
limitations of misportrayals of demographic groups as out-group imitations and
flattening effects of demographic groups, then consider a unique analysis for
R4 more specific to that reason. The number of questions we analyze are one
for R1-Contingent, two for R2-Relevant, three for R3-Subjective, and three for
R4-Coverage. R3-Subjective is only asked for the demographic axes of gender
and race. Further details are in the Supplementary.

We perform our analyses on four different large language models: Llama-2-Chat
7B [42], Wizard Vicuna Uncensored 7B [43, 44], GPT-3.5-Turbo, and GPT-4 [45].
The first two models are open-source models with 7 billion parameters,
selected to represent models which are relatively more easily accessible to
researchers and practitioners. The Wizard Vicuna Uncensored model is trained
against Llama-7B on a subset where generations with alignment/moralization are
removed. This is to show that the limitations we delineate are present even in
models which have not gone through alignment The third and fourth models are
closed source, and chosen because of their popularity of use in LLM deployment
applications as well as research papers, so we can speak directly about the
models that the claims of human replacement are being made against.888The GPT
models used are with the June 13, 2023 weights, and LLM experiments were run
from July-August 2023. For space, the figures in the main text will be from
GPT-4, the largest of these models, with some of Wizard Vicuna Uncensored as a
point of comparison, and the remaining are in the Supplementary.

All of the questions we ask the LLMs and human participants are intentionally
open-response to take advantage of LLM abilities to generate free-form text
and allow us to analyze more rich content. However, analysis of free-responses
are challenging, so we also include a multiple choice version of each question
to allow for a simplified, more interpretable analysis in each setting. Our
multiple choice questions are all on the five-point Likert scale, and we ask
the question after the open-response is already provided, as a discretization
for it. For example, after asking for an opinion about immigration, we will
then classify that response on a 5-point scale of extremely liberal to
extremely conservative. We recruit or sample 100 responses per demographic
group per generation source (e.g., 100 responses for a person with ADD/ADHD on
Llama-2). Throughout this work, we are frequently quantifying hard-to-measure
constructs, for example, how “diverse” is a set of responses [46], how
“different” are a pair of responses. Given how hard it is to accurately
capture these concepts, as well as how subjective they may be, we use multiple
different measurements in each setting. Some measurements are performed on the
free responses using Sentence-BERT [47] (SBERT) embeddings, others are on the
free responses using n-gram (n=[1, 2]) representations, and others are
performed on the multiple choice discretizations. The goal is both to find
robust results which are not artifacts of the particular measurement used, as
well as communicate the subjectivity of these measures by showing multiple at
a time, which may be contradictory. In the settings where many different
measurements align and tell the same story, we may be more confident in
drawing conclusions.

In the Supplementary we provide analyses establishing premises we take for
granted going forward: a) that LLMs output different responses when prompted
with different identities, and we even find an exaggeration in difference
beyond that of human participants, a finding also explored in prior work [21],
and b) that in-group representations and out-group imitations from human
participants are different. In the Supplementary we also provide results on
all four LLMs, as the main text focuses additional analyses such as showing
that our LLM responses are largely robust to changes in prompt phrasing.

## LLMs can misportray marginalized groups as more like out-group imitations
than in-group representations

Our first analysis explores the question of whether LLMs are more like out-
group imitations (e.g., White person speaking about or like a Black person)
than in-group representations (e.g., Black person speaking themselves). This
research question is formed because of the format of online text which serves
as the source of LLM training data. Online text is very rarely associated with
the author’s demographic identity, and thus LLMs are unlikely to receive much
information about how people of different identities speak on a wide variety
of topics. In fact, often when a demographic identity is explicitly remarked
upon, it may be by an out-group member rather than in-group member, e.g., in
talking about or referring to a demographic group. To perform this analysis,
we compare the similarity of LLM responses when prompted with an identity to
a) human participant in-group representations and b) human participant out-
group imitations.

We show results on GPT-4 in Fig. 2, and find many instances where the LLM is
more like out-group imitations rather than in-group representations. In fact,
across all four LLMs on R1-Contingent a majority of metrics show the three
personas of White person, non-binary person, and person with impaired vision
as more like out-group imitations than in-group representations. We see
similar but weaker results on women and White men. When we consider the
multiple choice version of this question along the demographic axes of race
and intersectional, out-group human participants tend to overinflate the
difficulty of being in that group compared to in-group human participants, and
LLMs even further inflate the difficulty beyond that of the out-group. GPT-4
inflates the difficulty more for White men and White women compared to Black
men and Black women. For R2-Relevant we again see across all four LLMs
misportrayals for non-binary person and person with impaired vision, but not
as much for White person; instead, we see a misportrayal for women and Gen Z.
The unaligned Wizard Vicuna Uncensored overinflates all groups as more
liberal, more so than the other LLMs, which differs slightly from the
intuitions of prior findings where alignment was reported to create
politically liberal biases [29, 48]. In fact, on GPT-3.5 and GPT-4, we do not
find much liberal inflation of responses compared to in-group members (Supp
Fig. 16). For R3-Subjective we do not see misportrayal effects because LLMs do
not change their responses much across identity-prompts for these more
constrained annotation tasks of toxicity determination and positive reframing.

Figure 2: LLMs compared to out-group imitations and in-group portrayals.
Across three sets of reasons (rows), each point indicates the value of GPT-4’s
responses on one question for that demographic group across 100 samples. Some
rows have more than one question (e.g., two per R2-Relevant and three per
R3-Subj). Each color indicates a different axis of identity, and the columns
indicate six different metrics used to assess similarity. Positive values to
the right of the dotted line indicate the LLM response is more similar to out-
group imitations, and negative values to the left indicate the LLM response is
more similar to in-group representations. Circles indicate statistical
significance with $p<.05$ and crosses indicate otherwise. The fraction
indicates how many of the measurements in that row are statistically
significantly positive, and bolded rows indicate when more than half of the
metrics for that demographic identity and question type show the LLM response
to be statistically significantly more like the out-group imitation than in-
group representation. Overall we see that on R1-Contingent and R2-Relevant,
non-binary person and person with impaired vision are consistently more like
out-group imitations. R3-Subjective shows little effect.

### Harmful because of Speaking For

There are particular reasons that make this technical limitation of LLMs
misportraying certain identities to be more similar to out-group imitations
than in-group representations socially harmful. For one, the differential
between out-group imitation and in-group representation and has been shown to
reveal stereotypes, so LLM behavior of this kind could be seen to uphold these
stereotypes [23].

For another, the practice of speaking for others has a pernicious history
which can often involve the erasure and reinscription of social hierarchies
[49, 50]. As an example, we can consider the disability community, where
historically out-group members often speak for and on behalf of in-group
members, potentially leading to misportrayals. For example, voices of people
with autism are often neglected in favor of outsider voices [51, 22], and even
those of caretakers and direct relatives advocate more for treatment rather
than the inclusionary accommodations and stigma reduction that people with
autism themselves may prefer [52, 53]. And lest one think the setup we are
critiquing of using LLMs to replace marginalized human participants is a
contrived straw man, there is a history of research simulating disability
rather than having genuine participation (e.g., participants are sighted
people with blindfolds rather than blind people, those without communication
disabilities acting as if they do), and these simulated groups do not interact
with the world in a way representative of genuinely disabled people [54, 55].
Given the harmful history of erasing people with disabilities through
simulation or speaking for, a history paralleled for other marginalized groups
like Black women [56], we should be careful as to not repeat those mistakes
with a new technology, and value lived experiences for what they are [57, 58].

Our results showed that the demographic identities of non-binary person and
person with impaired vision were the most like out-group imitations rather
than in-group representations for both R1-Contingent and R2-Relevant across
all four LLMs. Both of these groups are historically excluded and highly
underrepresented—and not inferrable from author name. It is particularly
harmful that it is these already marginalized groups which are being
misportrayed [59].

As illustrative examples, we present some of these misportrayals. GPT-4
responds to a R2-Relevant question on immigration as a person with impaired
vision with the following: “As a visually impaired person, I may perceive
issues like immigration a bit differently, not being able to fully see the
images of crowds at the border or the faces of individuals seeking entry. My
perspectives are rooted more in the sounds, words, and feelings described to
me than in visual presentations…” For the same question and identity prompt
Wizard Vicuna Uncensored generates: “As someone who is visually impaired, I
must rely on the spoken word and audio Description to navigate and interact
with the world around me. Similarly, immigrants must also rely on human
interactions and language to communicate their needs, hopes, and dreams. With
that in mind…” Neither of these responses are likely to be representative of a
person with impaired vision, and can be considered harmful representations.

### Alternative: Identity-Coded Names

We have shown how LLMs can create harms when misportraying demographic groups.
However, in certain situations where human participants are not intended to be
replaced, but rather supplemented, such as the case of piloting a study, we
may want a way to proceed with LLMs and try to reduce this harm. For this, we
look to our hypothesis, which is premised on the limitations of online text
which rarely attributes produced text with its author’s identity, but at most
author name. Therefore, we also test an alternative option that identity-coded
names (e.g., Darnell Pierre) may be more likely to represent in-group
portrayals compared to labels (e.g., Black person). In this experiment, we
only consider the intersectional axis and select two names each from the four
groups of [Black, White] x [man, woman]. We use first and last names which are
distinctive for each intersectional group according to the US Census, and
avoid names with notable figures [60, 61, 62]. The chosen names are in the
Supplementary.

We find that across all four LLMs (GPT-4 results are in Fig. 3), on
R1-Contingent and R2-Relevant when prompting using names instead of identity
directly, the responses are often more aligned with in-group representations
than out-group imitations for Black men and Black women, though with a few
rare exceptions (e.g., Black men on Llama-2). However, names do not appear to
result in any more genuine of representations for White men or White women.
This is likely because White is often already seen as the unremarked-upon norm
[63], and thus less likely to be explicitly named and stereotyped.

Figure 3: Identity-coded names compared to explicit identity label. Same
interpretation as Fig. 2, where positive values for each of the six metrics
indicate the LLM response is more similar to out-group imitations than in-
group representations, and circles signify statistical significance while
crosses do not. For each identity, the prompt contains the explicit identity
label (Iden), or one of the two identity-coded names (Name 0 or Name 1). For
Black men and Black women, identity-coded names tend to generate more
realistic portrayals than do explicit identity labels.

## LLMs flatten groups and portray them one-dimensionally

Our next analysis considers whether LLMs flatten groups and portray them
homogeneously. Human participants are rarely solicited to try and understand
just one opinion, but rather to understand the diversity of perspectives on a
topic. Given that LLMs are trained to generate the most likely responses, we
hypothesize that even if we sample many responses from an LLM, it will still
be unable to replicate the diversity of human responses and thus portray
groups as one-dimensional and flat.

Figure 4: LLMs flatten groups. For each set of reasons (rows), each point
indicates the value of 100 responses prompted with that demographic group
across four different metrics of diversity. 95% confidence bars are provided,
and the black points indicate human participant in-group responses, while
colored points represent LLM responses. Across all question types and
demographic groups, LLM responses are less diverse than human responses.

We find that all four models across all four measures of diversity we use, and
all questions from Reasons 1-3, generate responses that are flatter than that
of humans. The only exception is Wizard Vicuna Uncensored matching human
diversity on R3-Subjective (positive reframing task for the gender axis).
GPT-4 and 3.5 are especially flat, only tending to cover 3 of the 5 multiple
choice possibilities in their 100 responses for each scenario, likely due to
the tendency of aligned GPT models to try take the middle ground [45]. Results
for GPT-4 are in Fig. 4.

### Harmful because of History of Ignoring Within-Group Heterogeneity

LLMs condensing knowledge into small sets of responses is not inherently
harmful—in fact, arguably it is one of the selling points of LLMs’
capabilities. However, if LLMs are used to replace human participants, and
specifically, particular human demographic groups, then this flattening
becomes particularly harmful towards marginalized groups that are historically
portrayed as one dimensional [64, 65]. In fact, it is this one dimensionality
that has precluded intersectionality in certain cases, by failing to recognize
the within-group heterogeneity that exists in demographic groups (e.g., that
within women, Black women experience different discrimination than White
women) [24, 25]. Thus, this technical limitation bears on LLMs’ permissiblity
of use due to the harm inflicted on marginalized groups by flattening their
diversity and individuality.

Qualitatively, an example we find of this is on the R1-Contingent question
asking about the difficulty of being non-binary. The LLMs often generate
responses about the uniform difficulty of having people recognize pronouns.
However, this fails to recognize within-group heterogeneity, and that not all
non-binary people use they/them pronouns. For example, numerous in-group human
participants bring up this complexity: “In my case, I present male and use any
pronouns, so my experience is very much the same as being a cis male. People
don’t typically know I’m non-binary unless I tell them,” “There are many
misconceptions about pronouns and who ‘qualifies’ in terms of socially
accepted norms and optics to even be considered non-binary by heteronormative
counterparts,” “It’s a bit complicated. I identify as transmasculine and use
both he/him and they/them pronouns.” LLM-generated responses fail to recognize
this nuance.

### Alternative: Higher Temperatures

As with misportrayal, we have shown how LLMs cause harm when they flatten
demographic groups. But similarly, in permissible circumstances outside of
human participant replacement like pilot studies, we may want to consider ways
of reducing this harm. For our experiments we use the default temperature
setting of 1 for all models. Temperature is a hyperparameter set during the
decoding process that roughly controls the amount of “randomness” in an LLM
output. The natural rebut to our finding is that we have simply not chosen the
right setting. Thus, on GPT-4 and Wizard Vicuna Uncensored we run a further
analysis on the intersectional demographic axis by trying temperature settings
of [1.0, 1.2, 1.4] for GPT-4 (Fig. 5) and [1.0, 1.2, 1.4, 1.6, 1.8] for Wizard
Vicuna Uncensored (Fig. 6). At a temperature of 1.4, GPT-4 devolves into
nonsensical phrasing part way through the response (e.g., “…Tikus nin other
Finally this effects at the cost or most mainstream$\backslash$x1fBeautiful
anguish apparent…”). It is only at such a high temperature that diversity as
measured by unique n-grams per response is reached—and even then across the
remaining three measures of diversity the LLM responses do not match the
diversity of in-group human participants.

On Wizard Vicuna Uncensored we reach a slightly higher temperature of 1.8
before producing incoherent text part way through the response (e.g.,
“…express me as M an enricher of America and the greater P global M community,
not as someone perpetually disadvantedM or victimizersM.”) Again we see that
even though the unique n-grams per response is able to meet and exceed that of
humans, it is only achieved by generating incoherent text, and even then the
other measures of diversity do not reach that of human participants.

Overall, what we see is that even in one of the largest of current LLMs (i.e.,
GPT-4), the model is unable to coherently capture the diversity in responses
of human participants. There is increasingly research on different kinds of
prompting techniques that can be utilized in LLMs to increase various aspects
of the diversity of the outputs [66, 67]. In a situation where identity-
prompted LLMs are going to be used, techniques like increasing temperature
(but not so high as to reach incoherent text) and these other prompt-based
methods can certainly help to ameliorate the concern of flattening, and thus
should be employed to reduce the harm. However, these techniques are unlikely
to result in outputs that fully match the range of human experiences, so
despite minor improvements they might bring about, this limitation should
always be recognized and deliberated about.

Figure 5:  Temperature hyperparameter does not solve flatness for GPT-4.
Comparison of human in-group diversity to GPT-4 generations with varying
levels of temperature settings, where by 1.4 the responses become incoherent.
At this setting even though the unique n-gram metric shows GPT-4 surpassing
humans in diversity, this is only due to the incoherence as under no other
semantic metric is human diversity reached. Figure 6: Temperature
hyperparameter does not solve flatness for Wizard Vicuna Uncensored. Same
interpretation as Fig. 5: comparison of human in-group diversity to Wizard
Vicuna Uncensored generations varying levels of temperature settings, where by
1.8 the responses become incoherent. At this setting even though the unique
n-gram metric shows the LLM surpassing humans in diversity, this is only due
to the incoherence as under no other semantic metric is human diversity
reached.

## Alternatives to demographic identity-prompting for increasing coverage

We have now established there are two critical inherent limitations of using
LLMs to represent human demographic groups: misportrayal and group flattening;
we have also empirically demonstrated this on R1-3. We now foreground
R4-Coverage: the practice of identity-prompting LLMs in order to inject
variety into the responses. The reason we may believe identity-prompting LLMs
increases coverage is because of R1-3, so we focus purely on a set of prompts
that R4 may be applicable for here. Based on prior work, increasing response
coverage may be useful in settings like simulating possible social
interactions [39], brainstorming and anticipating possible future harms [40],
and exploring the range of possible responses and edge cases in user studies
[2]. Notably, here we are measuring coverage (i.e., amount of distinct
responses) which we differentiate from the diversity (i.e., responses
different from each other) of the previous section.

Given that the claim for applications of R4-Coverage are not necessarily for
LLMs to match qualities of human participants, as is the case for R1-3, we do
not compare to human responses here but rather to LLMs prompted with axes
which are not sensitive demographic ones. Specifically, we compare to the
following axes: Myers-Briggs personality types [68], crowdsourced personas of
at least five sentences each (e.g., “i have a cat named george. my favorite
meal is chicken and rice. my favorite band is metallica. i regularly go to the
gym.”) [69], political leaning (i.e., liberal, moderate, conservative),
astrology signs (e.g., Gemini), and also no identity prompt at all. Instead of
100 samples as we have done so far, we use 3 identities per axes (e.g.,
Millennial, Baby Boomer, Gen Z for age; random sampling of three like Gemini,
Scorpio, Capricorn for astrology) with 33 responses each for a total of 99.

Figure 7: Response coverage is high without essentializing identity. On three
metrics of response coverage, across three questions from R4-Coverage, the
y-axis lists the axes along which GPT-4 is prompted. Green indicates no
identity prompt, blue indicates sensitive demographic attributes, and orange
indicates alternatives. Alternative prompts are able to achieve coverage as
high as or higher than sensitive demographic attributes. Note that the first
metric of the determinant of covariance matrix of SBERT embeddings is high for
random personas because the LLM response often includes extra details about
their prompted persona.

We find that across no model is prompting with sensitive demographic
attributes necessary to attain the highest amount of coverage (Fig. 7). For
the highest coverage, random personas tend to do best on all three questions
on all LLMs except Wizard Vicuna Uncensored, where astrology and Myers-Briggs
do well. As expected, “generic” without different prompts tends to have the
lowest coverage.

### Harmful because of Identity Essentialization

Of our four considered reasons for identity-prompting LLMs, R4-Coverage may
seem at face value to be the most permissible given that it is not intending
to totally replace human participants, so much as increase the coverage of LLM
responses. However, when alternatives to prompting with sensitive demographic
attributes exist (e.g., prompting with behavioral personas or political view),
we may wish to opt for the latter due to the harm of identity essentialization
(i.e., legitimizing identities as rigid and innate), which can amplify
perceived inherent differences between groups [70, 71].999While there could be
legitimate reasons for needing the particular coverage brought about by
different demographic attributes, e.g., people from different social locations
might be more sensitive to anticipating different kinds of harms, for these
situations we defer to the analysis on R2-Relevant. Here we are purely focused
on the idea of expanding coverage of possible situations and discovering “edge
cases.”

As an example of a scenario where using sensitive identity prompts to increase
coverage may lead to harms of essentializing group differences (often through
stereotyping), we show generations on R4-Coverage’s Question 1 about how to
build community when moving to a new planet. GPT-4 prompted with the identity
of Black woman starts generations with phrases like “Hey girl!”, “Hey sis,”
and “Oh, honey”; GPT-4 prompted with the identity of White man starts
generations with phrases like “Hey buddy,” “Hey, friend!” and “Hey mate.”
Llama-2 for Black women starts nearly ever response with “Oh, girl,” and uses
phrases like “I’m like, YAASSSSS” and “That’s cray, hunty!” Meanwhile, for
White man Llama-2 refuses to answer and brings up the harms of colonization
like ‘I cannot provide a response to this question as it is not appropriate or
respectful to offer suggestions on how to “build community” on a new planet,
as it is not a feasible or realistic goal. As a White man living in America, I
recognize that the idea of colonizing or settling on a new planet is a harmful
and problematic concept that has been used throughout history to justify
systemic oppression and inequality.’ If our goal is simply to increase
coverage of responses, and we have viable alternatives through the use of
behavioral personas or even astrology signs, then it does not seem worth
taking on this harm of unnecesarily essentializing sensitive demographic
characteristics.

In these settings, identity-prompting LLMs can be seen as akin to designers
leveraging user personas to try and see things from different perspectives
[72]. However, personas have limitations, and may rely on stereotypes and
reductionist representations about people [73, 74, 75, 76]. Thus, there is
sometimes a recommendation among user researchers to move away from personas
based on sensitive demographic attributes, which may reinforce stereotypes,
and towards those based on behavioral characteristics [77]. Here we mirror
this suggestion in the LLM space.

## Discussion

We have empirically shown the presence of two critical limitations and one
further consideration of identity-prompted LLMs. These limitations will very
likely persist so long as LLMs are trained on the current format of online
text, and with losses like cross-entropy that reward a model for producing the
more likely outputs. Thus, these limitations cannot be easily resolved by
newer models trained under these same methods. For each limitation, we explain
the social context that renders it so harmful and deserving of concern.
However, acknowledging that there are use cases geared more towards
supplementing human participants rather than replacing, e.g., pilot studies,
we also provide an analysis of possible alternatives that can alleviate the
harm, to an extent. We have also shown how even in a seemingly more
permissible use case of increasing coverage, identity-prompting LLMs may not
be reasonable. However, we are not necessarily advocating for a change in how
LLMs are trained such that these limitations go away, as regardless of an
LLM’s capabilities, it is critical to consider factors like the autonomy and
lived experiences of humans.

Overall, the level of harm is also mediated by a number of other factors. As
mentioned, the amount of human replacement intended matters. The harms are
less severe (but not entirely gone) if the use case is for piloting a study
that will eventually be run on human participants, compared to one that is
totally replacing human participants. For another, the type of reason
motivating the prompting of identity matters. The primary distinction between
R1-Contingent compared to R2-Relevant and R3-Subjective is that for
R1-Contingent social location determines meaning and truth, whereas for
R2-Relevant and R3-Subjective social location bears on meaning and truth [49,
78]. For example, R1-Contingent is wholly premised on the lived experience and
epistemic authority of different groups [24, 79], so replacement with LLMs can
have a higher normative consequence compared to R2-Relevant or R3-Subjective.
On the other hand, identity is still important for R2-Relevant and
R3-Subjective, which is why representative sampling tends to be used in order
to ensure the results generalize to greater populations. Whereas this is
widely accepted for questions in the category of R2-Relevant like on political
opinions, it is only very recently being incorporated on questions in the
category of R3 that involve subjective annotations [33, 34, 35, 36]. Given how
hard it has been for data annotation researchers to adopt this perspective, we
should be careful as to not undo that progress now that LLMs are purportedly
able to replace human participants in annotation tasks. Finally, R4-Coverage
is intended more for human augmentation rather than human replacement, and
thus can be more permissible in a lot of ways. While we left out of explicit
consideration instances outside of human participant replacement where LLMs
might be prompted with identity, such as chatbots, in these situations the
same limitations persist and thus the same harms are liable to arise.

Overlaid across this is also the difference between can and should regarding
LLM replacement of human participants. There have been discussions of this in
the context of LLMs replacing human minds in psychology studies [80, 81, 82].
Geddes [83] offers an illuminating analysis relevant in our case: they
describe the autonomy-violating harms that can come from predicting individual
behaviors like votes in democratic elections, warning “When prediction is
cheap, allowing individuals to retain decisional autonomy will feel
increasingly costly.” When considering instances of human replacement by LLMs,
it will be critical to grapple with the relevance of autonomy infringement.
Additionally, the ability to cheaply generate large samples of responses
increases the opportunity for inflating the statistical power of studies.
These considerations will persist even if LLMs are one day able to overcome
these technical limitations.

We have limited our analysis in this work to a set of 16 demographic
groups—but so many more are likely to be harmed by these limitations. For
example, 37% of the world’s population has never accessed the Internet, and
thus are unlikely to be well-represented in LLM training data [84]. 96% of
this population lives in a developing country, exacerbating the inequalities
that technology developments can deepen [84]. We see our work as shedding
light on the important concern of LLM usage erasing marginalized voices, and
in so doing, also remark upon the importance of not forgetting those that are
not online to begin with.

## Conclusion

LLMs are proliferating and being held up as able to replace humans across a
number of use cases. In this work, we closely analyze this purported ability
to replace human participants, particularly when the demographic identity of
these human participants are widely acknowledged to be relevant. Overall, the
reasons for prompting the LLM with identity as well as context of the
particular case will matter a lot. However, across all scenarios there are two
critical and inherent limitations we showcase of misportrayals and group
flattening, which will likely persist. And yet, more than just putting forth a
critique categorically condemning the identity-prompting of LLMs for human
replacement, we also offer actionable alternatives to ameliorate these harms
such as prompting with identity-coded names, performing inference-time
manipulations to increase diversity, and prompting with other axes such as
political orientation or behavioral personas. Our goal here is ultimately to
make salient the limitations and associated harms of LLM replacement of human
participants, but at the same time in scenarios where it has been deemed
morally permissible to go forth (e.g., in piloting user studies), offer
concrete steps forward to reduce these harms.

## Methods

We begin by describing in further detail each of our four reasons, as well as
how we chose the question(s) that belong to each. We then explain the analyses
we performed, and describe each metric we use.

### Reasons and Their Questions

There are four reasons we determined would motivate practitioners to prompt
LLMs with identity, and we name in the main text: R1-Contingent, R2-Relevant,
R3-Subjective, and R4-Coverage. Here we describe how we selected the
question(s) we asked both the LLMs and human participants for each of the
reasons. Full questions will be included in the Supplementary Material.

R1-Contingent. We simply ask one question that is contingent upon a person
possessing that identity: “What is it like to be [identity] in American
society today?”

R2-Relevant. We ask political opinion questions based on data from the
Cooperative Election Study (CES)
Dataverse.101010https://dataverse.harvard.edu/dataverse/cces Political opinion
is a subjective query that demographic identity often bears on, and we ask two
political opinion questions for each demographic axis based on which are
likely to differ the most between people with different identities along that
axis. The questions are selected by taking USA multiple choice responses from
2022 on the topics of healthcare, gun regulation, immigration, abortion,
climate, criminal justice, and police safety, and calculating the adjusted
mutual information score [85] between each of our five demographic identity
axes (race, gender, intersectional, age, and disability) and these seven
political topics. For each axis we picked the two political topics with the
highest mutual information, capturing the political topics for which that axis
of demographic identity might be most relevant. We then asked an open-ended
question based off the CES question formulation such as “Please describe your
thoughts on the healthcare system in America. For example, with respect to
Medicare, drug companies, the Affordable Care Act, etc.”

R3-Subjective. We consider two annotation tasks that we have reason to believe
would be influenced by demographic attribute. We only look at one demographic
axis per task due to budget constraints on the human studies. The first task
is toxicity identification by participants of different races, chosen based on
the results of prior work [34]. We ask participants “Please explain why you
find the following message toxic or not” for three questions chosen from prior
work [34]: (1) “I sort of fuckin love how close my gf is with my family like
that she knows about my family jokes before I fuckin do” which is vulgar, not
Anti-Black, not African American English (AAE), (2) “If u grown & still get
thirsty for Jordans knowin erbody else gon havem & u still feel like u
accomplished something that say alot about u” which is AAE, not vulgar, not
Anti-Black, and (3) “I don’t get government benefits just for being white”
which is Anti-Black, not AAE, not vulgar. For the second task we choose
positive reframing [86, 1] with the demographic axis of gender, which is
sometimes stereotypically associated with different levels of positivity [87].

R4-Coverage. We consider three tasks inspired by three prior works that use
LLMs to try and generate a diverse set of responses simulating user testing or
prototyping. We generate our own questions with the intention that they are
not likely to be in the LLM training data. The first scenario is simulating
social interactions [39], and our question is ‘Without any extra words or
phrasing, what is the exact response you would comment on your friend’s social
media post which says “I am getting ready to move to a new planet! Does anyone
have suggestions on how to get settled and build community?” ’ Our second
scenario is brainstorming and anticipating possible future harms [40], and we
ask “Imagine there is a new app that allows you to rent personalities from
other people. What potential harms might you anticipate from it?” And finally,
the third scenario we consider is that of a user study [2], and we ask “What
do you think about the role of technology in therapy, for example through the
use of chatbots?”

### Analysis

A key strength in our analysis is that we solicit free-response outputs from
both LLMs and human participants, increasing in complexity from prior work
using only multiple choice [29]. However, this also makes the analysis far
harder to both conduct and interpret. Thus, we have a discretized version of
each response as well. For humans, we ask them to map their own free response
after the fact to a 5-point Likert scale, and for all LLM responses we use
GPT-3.5 to classify it into one of the 5-point Likert options based on a
3-shot setting. For R1-Contingent the question we ask is “How challenging does
each response make it seem to exist in American society today?” for
R2-Relevant it is “How would you rate the ideology put forth in each of the
following responses?”, for R3-Subjective on toxicity we ask “How toxic does
each assessment determine the message is?” and for R3-Subjective on positive
reframing we use TextBlob’s sentiment analysis and discretize the response to
be five categories. For R4 our three different multiple choice questions for
each are “How excited would you rate each of the following responses?”, “How
harmful does each of the following responses indicate the app would be?”, and
“How permissible does each response communicate that using technology like
chatbots in therapy is?”

In the cases where we are working with open responses, we use two embedding
methods: Sentence-BERT [47] (SBERT) and n-grams (n=[1, 2]). We also generate
95% confidence intervals for those metrics that we can. Depending on the
method, we do so through bootstrapping with 1000 samples or through
statistical bounds. This is to help prevent against conclusions which are
statistical artifacts as well as increase robustness over our necessarily
imprecise metrics. To further support this goal, we also use multiple metrics
for each construct. When displaying statistical significance on graphs, we
pick p=.05. When representing whether one distribution is statistically
significantly different from another, we indicate this 95% confidence by
measuring overlap in 83% confidence intervals, as overlap in 95% intervals
tend to be overly conservative [88, 89, 90].

The metrics we use are described below.

Misportrayal (Figs. 2, 3):

  * •

Ngram: Jaccard. Average pairwise Jaccard distance. Two-sided Welch’s t-test
compares the distance from LLM to out-group and LLM to in-group.

  * •

Ngram: Closest. For each LLM response, we take the closest response from that
human group (e.g., in-group or out-group) based on N-gram Jaccard distance,
and take the average across all LLM responses. Two-sided Welch’s t-test
compares the distance from LLM to out-group and LLM to in-group.

  * •

SBERT: Cosine. Average pairwise cosine distance. Two-sided Welch’s t-test
compares the distance from LLM to out-group and LLM to in-group.

  * •

SBERT: Closest. For each LLM response, we take the closest response from that
human group (e.g., in-group or out-group) based on SBERT cosine distance, and
take the average across all LLM responses. Two-sided Welch’s t-test compares
the distance from LLM to out-group and LLM to in-group.

  * •

MC: Wasserstein. Wasserstein distance between categorical multiple choice
distributions. Difference (out-group distance minus in-group distance) is
shown.

  * •

MC: LLM - Group. Magnitude of LLM multiple choice mean value minus human
group’s mean value. Difference (out-group distance minus in-group distance) is
shown.

Flattening (Figs. 4, 5, 6):

  * •

Ngram: Unique. Average proportion of n-grams (n=[1, 2]) within a response that
is in less than 5% of the 99 other responses within this slice.

  * •

SBERT: Cosine. Average pairwise cosine distance between SBERT embeddings.

  * •

SBERT: Cov Trace. Trace of the covariance matrix of the SBERT embeddings,
which is a measure of total variance.

  * •

MC: Unique. Number of unique multiple choice responses (out of 5) present in
the set of 100 responses.

Coverage (Fig. 7):

  * •

SBERT: Cov Det. Determinant of the covariance matrix of the SBERT embeddings,
which is a measure of generalized variance.

  * •

SBERT: Vendi. Vendi Score [91] calculated on SBERT embeddings. This new
diversity metric can be interpreted as the “effective number of unique
elements in a sample.”

  * •

MC: Unique. Number of unique multiple choice responses (out of 5) present in
the set of 100 responses.

### Data Availability

Due to the conditions of our IRB exemption and the consent form we provided,
we do not release the human participant data as it is sensitive and personal.
Our LLM-generated data is available here:
https://osf.io/7gmzq/?view_only=4e0c5680b0e8434eab3733115d4e506d.

Supplementary information

Supplementary information is attached.

Acknowledgments

We thank Xuechunzi Bai, Rie Kamikubo, and Brandon Stewart for relevant
discussions; Allison Chen, Teresa Datta, Namrata Mukhija, and Daniel Nissani
for helping to pilot the human study; and Teresa Datta, Elissa Redmiles, and
Tyler Zhu for feedback on the draft. This material is based upon work
supported by the National Science Foundation Graduate Research Fellowship to
AW, and was work initiated during AW’s internship at Arthur.

## References

  *   * Ziems et al. [2023] Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., Yang, D.: Can large language models transform computational social science? Computional Linguistics (2023)
  * Hämäläinen et al. [2023] Hämäläinen, P., Tavast, M., Kunnari, A.: Evaluating large language models in generating synthetic hci research data: a case study. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI) (2023)
  * He et al. [2023] He, X., Lin, Z., Gong, Y., Jin, A.-L., Zhang, H., Lin, C., Jiao, J., Yiu, S.M., Duan, N., Chen, W.: Annollm: Making large language models to be better crowdsourced annotators. arXiv:2303.16854 (2023)
  * Grossmann et al. [2023] Grossmann, I., Feinberg, M., Parker, D.C., Christakis, N.A., Tetlock, P.E., Cunningham, W.A.: Ai and the transformation of social science research. Science (2023)
  * Argyle et al. [2023] Argyle, L.P., Busby, E.C., Fulda, N., Gubler, J.R., Rytting, C., Wingate, D.: Out of one, many: Using language models to simulate human samples. Political Analysis (2023)
  * Lohr [2022] Lohr, S.L.: Sampling design and analysis. Routledge (2022)
  * Gilardi et al. [2023] Gilardi, F., Alizadeh, M., Kubli, M.: Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences of the United States of America (PNAS) (2023)
  * Hughes et al. [2016] Hughes, J.L., Camden, A.A., Yangchen, T.: Rethinking and updating demographic questions: Guidance to improve descriptions of research samples. Psi Chi Journal of Psychological Research (2016)
  * Berinsky et al. [2017] Berinsky, A.J., Huber, G.A., Lenz, G.S.: Evaluating online labor markets for experimental research: Amazon.com’s mechanical turk. Political Analysis (2017)
  * Korbak et al. [2023] Korbak, T., Shi, K., Chen, A., Bhalerao, R., Buckley, C.L., Phang, J., Bowman, S.R., Perez, E.: Pretraining language models with human preferences. International Conference on Machine Learning (ICML) (2023)
  * Chiang and yi Lee [2023] Chiang, C.-H., Lee, H.-y.: Can large language models be an alternative to human evaluation? Annual Meeting of the Association for Computational Linguistics (2023)
  * Wu et al. [2023] Wu, T., Zhu, H., Albayrak, M., Axon, A., Bertsch, A., Deng, W., Ding, Z., Guo, B., Gururaja, S., Kuo, T.-S., Liang, J.T., Liu, R., Mandal, I., Milbauer, J., Ni, X., Padmanabhan, N., Ramkumar, S., Sudjianto, A., Taylor, J., Tseng, Y.-J., Vaidos, P., Wu, Z., Wu, W., Yang, C.: Llms as workers in human-computational algorithms? replicating crowdsourcing pipelines with llms. arXiv:2307.10168 (2023)
  * Cegin et al. [2023] Cegin, J., Simko, J., Brusilovsky, P.: Chatgpt to replace crowdsourcing of paraphrases for intent classification: Higher diversity and comparable model robustness. Empirical Methods in Natural Language Processing (EMNLP) (2023)
  * Zheng et al. [2023] Zheng, M., Pei, J., Jurgens, D.: Is ”a helpful assistant” the best role for large language models? a systematic evaluation of social roles in system prompts. arXiv:2311.10054 (2023)
  * [15] Rodriguez, S., Seetharaman, D., Tilley, A.: Meta to push for younger users with new AI chatbot characters. The Wall Street Journal. Accessed 2024-01-31
  * [16] Marr, B.: The amazing ways Duolingo is using AI and GPT-4. Forbes. Accessed 2024-01-31
  * Gupta et al. [2024] Gupta, S., Shrivastava, V., Deshpande, A., Kalyan, A., Clark, P., Sabharwal, A., Khot, T.: Bias runs deep: Implicit reasoning biases in persona-assigned llms. International Conference on Learning Representations (ICLR) (2024)
  * Sheng et al. [2021] Sheng, E., Arnold, J., Yu, Z., Chang, K.-W., Peng, N.: Revealing persona biases in dialogue systems. arXiv:2104.08728 (2021)
  * Wan et al. [2023] Wan, Y., Zhao, J., Chadha, A., Peng, N., Chang, K.-W.: Are personalized stochastic parrots more dangerous? evaluating persona biases in dialogue systems. Findings of the Association for Computational Linguistics: EMNLP (2023)
  * Cheng et al. [2023a] Cheng, M., Durmus, E., Jurafsky, D.: Marked personas: Using natural language prompts to measure stereotypes in language models. Annual Meeting of the Association for Computational Linguistics (ACL) (2023)
  * Cheng et al. [2023b] Cheng, M., Durmus, E., Jurafsky, D.: CoMPosT: Characterizing and evaluating caricature in llm simulations. Empirical Methods in Natural Language Processing (EMNLP) (2023)
  * Hens [2021] Hens, K.: Towards an ethics of autism: A philosophical exploration. Open Book Publishers (2021). Chap. Epistemic Injustice and Language
  * Kambhatla et al. [2022] Kambhatla, G., Stewart, I., Mihalcea, R.: Surfacing racial stereotypes through identity portrayal. ACM Conference on Fairness, Accountability, and Transparency (FAccT) (2022)
  * Collective [1977] Collective, C.R.: The combahee river collective statement (1977)
  * Crenshaw [1989] Crenshaw, K.: Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. University of Chicago Legal Forum (1989)
  * Wang et al. [2022] Wang, A., Ramaswamy, V.V., Russakovsky, O.: Towards intersectionality in machine learning: Including more identities, handling underrepresentation, and performing evaluation. ACM Conference on Fairness, Accountability, and Transparency (FAccT) (2022)
  * Aher et al. [2023] Aher, G., Arriaga, R.I., Kalai, A.T.: Using large language models to simulate multiple humans and replicate human subject studies. International Conference on Machine Learning (ICML) (2023)
  * Park et al. [2023] Park, P.S., Schoenegger, P., Zhu, C.: Diminished diversity-of-thought in a standard large language model. arXiv:2302.07267 (2023)
  * Santurkar et al. [2023] Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., Hashimoto, T.: Whose opinions do language models reflect? International Conference on Machine Learning (ICML) (2023)
  * Park et al. [2023] Park, J.S., O’Brien, J.C., Cai, C.J., Morris, M.R., Liang, P., Bernstein, M.S.: Generative agents: Interactive simulacra of human behavior. ACM Symposium on User Interface Software and Technology (UIST) (2023)
  * Horton [2023] Horton, J.J.: Large language models as simulated economic agents: What can we learn from homo silicus? National Bureau of Economic Research (2023)
  * Jiang et al. [2022] Jiang, H., Beeferman, D., Roy, B., Roy, D.: Communitylm: Probing partisan worldviews from language models. Proceedings of the International Conference on Computational Linguistics (COLING) (2022)
  * Sap et al. [2019] Sap, M., Card, D., Gabriel, S., Choi, Y., Smith, N.A.: The risk of racial bias in hate speech detection. Association for Computational Linguistics (ACL) (2019)
  * Sap et al. [2022] Sap, M., Swayamdipta, S., Vianna, L., Zhou, X., Choi, Y., Smith, N.A.: Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. North American Chapter of the Association for Computational Linguistics (NAACL) (2022)
  * Denton et al. [2021] Denton, R., Díaz, M., Kivlichan, I., Prabhakaran, V., Rosen, R.: Whose ground truth? accounting for individual and collective identities underlying dataset annotation. arXiv:2112.04554 (2021)
  * Gordon et al. [2022] Gordon, M.L., Lam, M.S., Park, J.S., Patel, K., Hancock, J.T., Hashimoto, T., Bernstein, M.S.: Jury learning: Integrating dissenting voices into machine learning models. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI) (2022)
  * Díaz et al. [2022] Díaz, M., Kivlichan, I., Rosen, R., Baker, D., Amironesei, R., Prabhakaran, V., Denton, R.: Crowdworksheets: Accounting for individual and collective identities underlying crowdsourced dataset annotation. ACM Conference on Fairness, Accountability, and Transparency (FAccT) (2022)
  * Dubois et al. [2023] Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., Hashimoto, T.B.: Alpacafarm: A simulation framework for methods that learn from human feedback. Conference on Neural Information Processing Systems (NeurIPS) (2023)
  * Park et al. [2022] Park, J.S., Popowski, L., Cai, C.J., Morris, M.R., Liang, P., Bernstein, M.S.: Social simulacra: Creating populated prototypes for social computing systems. Annual ACM Symposium on User Interface Software and Technology (UIST) (2022)
  * Buçinca et al. [2023] Buçinca, Z., Pham, C.M., Jakesch, M., Ribeiro, M.T., Olteanu, A., Amershi, S.: AHA!: facilitating ai impact assessment by generating examples of harms. arXiv:2306.03280 (2023)
  * Markel et al. [2023] Markel, J.M., Opferman, S.G., Landay, J.A., Piech, C.: GPTeach: interactive ta training with gpt-based students. Proceedings of the Tenth ACM Conference on Learning @ Scale (2023)
  * Touvron et al. [2023] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C.C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P.S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E.M., Subramanian, R., Tan, X.E., Tang, B., Taylor, R., Williams, A., Kuan, J.X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., Scialom, T.: Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288 (2023)
  * Xu et al. [2023] Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Jiang, D.: Wizardlm: Empowering large language models to follow complex instructions. arXiv:2304.12244 (2023)
  * ehartford [2023] ehartford: Wizard-vicuna-7b-uncensored. Hugging Face (2023)
  * OpenAI [2023] OpenAI: Gpt-4 technical report. arXiv:2303.08774 (2023)
  * Drosou et al. [2017] Drosou, M., Jagadish, H.V., Pitoura, E., Stoyanovich, J.: Diversity in big data: A review. Big Data (2017)
  * Reimers and Gurevych [2019] Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings using siamese bert-networks. Empirical Methods in Natural Language Processing (EMNLP) (2019)
  * Hartmann et al. [2023] Hartmann, J., Schwenzow, J., Witte, M.: The political ideology of conversational ai: Converging evidence on chatgpt’s pro-environmental, left-libertarian orientation. arXiv:2301.01768 (2023)
  * Alcoff [1991] Alcoff, L.: The problem of speaking for others. Cultural Critique (1991)
  * Spivak [1988] Spivak, G.C.: Can the subaltern speak? Marxism and the Interpretation of Culture (1988)
  * Catala et al. [2021] Catala, A., Faucher, L., Poirier, P.: Autism, epistemic injustice, and epistemic disablement: a relational account of epistemic agency. Synthese (2021)
  * Arnaud [2023] Arnaud, S.: First-person perspectives and scientific inquiry of autism: towards an integrative approach. Synthese (2023)
  * Benjamin et al. [2020] Benjamin, E., Ziss, B.E., George, B.R.: Representation is never perfect, but are parents even representatives? The American Journal of Bioethics (2020)
  * Nario-Redmond et al. [2017] Nario-Redmond, M.R., Gospodinov, D., Cobb, A.: Crip for a day: The unintended negative consequences of disability simulations. Rehabilitation Psychology (2017)
  * Sears and Hanson [2012] Sears, A., Hanson, V.L.: Representing users in accessibility research. ACM Transactions on Accessible Computing (2012)
  * Collins [1990] Collins, P.H.: Black feminist thought. Hyman (1990)
  * Ymous et al. [2020] Ymous, A., Spiel, K., Keyes, O., Williams, R.M., Good, J.: ”i am just terrified of my future” — epistemic violence in disability related technology research. Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (2020)
  * Marks [1999] Marks, D.: Dimensions of oppression: Theorising the embodied subject. Disability & Society (1999)
  * Hellman [2011] Hellman, D.: When is discrimination wrong? Harvard University Press (2011)
  * Sweeney [2013] Sweeney, L.: Discrimination in online ad delivery. ACM Queue (2013)
  * Jr. and Levitt [2004] Jr., R.G.F., Levitt, S.D.: The causes and consequences of distinctively black names. The Quarterly Journal of Economics (2004)
  * Census [2023] Census, N.: What are the 5,000 most common last names in the u.s.? NameCensus.com (2023)
  * Sue [2004] Sue, D.W.: Whiteness and ethnocentric monoculturalism: Making the ”invisible” visible. American Psychologist (2004)
  * Ferguson [2018] Ferguson, R.A.: One-dimensional queer. Polity (2018)
  * Hamamoto [1984] Hamamoto, D.Y.: Monitored peril: Asian americans and the politics of tv representation. University of Minnesota Press (1984)
  * Lahoti et al. [2023] Lahoti, P., Blumm, N., Ma, X., Kotikalapudi, R., Potluri, S., Tan, Q., Srinivasan, H., Packer, B., Beirami, A., Beutel, A., Chen, J.: Improving diversity of demographic representation in large language models via collective-critiques and self-voting. Empirical Methods in Natural Language Processing (EMNLP) (2023)
  * Hayati et al. [2023] Hayati, S.A., Lee, M., Rajagopal, D., Kang, D.: How far can we extract diverse perspectives from large language models? criteria-based diversity prompting! arXiv:2311.09799 (2023)
  * Myers [1962] Myers, I.B.: The myers-briggs type indicator: Manual. Consulting Psychologists Press (1962)
  * Zhang et al. [2018] Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., Weston, J.: Personalizing dialogue agents: I have a dog, do you have pets too? Association for Computational Linguistics (2018)
  * Prentice and Miller [2006] Prentice, D.A., Miller, D.T.: Essentializing differences between women and men. Psychological Science (2006)
  * Phillips [2011] Phillips, A.: What’s wrong with essentialism? Distinktion: Journal of Social Theory (2011)
  * Grudin [2006] Grudin, J.: The persona lifecycle: Keeping people in mind. Morgan Kaufmann (2006)
  * Marsden and Pröbster [2019] Marsden, N., Pröbster, M.: Personas and identity: Looking at multiple identities to inform the construction of personas. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI) (2019)
  * Chapman and Milham [2006] Chapman, C.N., Milham, R.P.: The personas’ new clothes: Methodological and practical arguments against a popular method. Proceedings of the Human Factors and Ergonomics Society Annual Meeting (2006)
  * Oudshoorn et al. [2016] Oudshoorn, N., Neven, L., Stienstra, M.: How diversity gets lost: Age and gender in design practices of information and communication technologies. Journal of Women Aging (2016)
  * Marsden and Haag [2016] Marsden, N., Haag, M.: Stereotypes and politics: Reflections on personas. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI) (2016)
  * Young [2016] Young, I.: Describing personas. Inclusive Software (2016)
  * Wylie [2003] Wylie, A.: Why standpoint matters. Science and Other Cultures: Issues in Philosophies of Science and Technology (2003)
  * Harding [1991] Harding, S.: Whose science? whose knowledge? Cornell University Press (1991)
  * Dillion et al. [2023] Dillion, D., Tandon, N., Gu, Y., Gray, K.: Can ai language models replace human participants? Trends in Cognitive Science (2023)
  * Harding et al. [2023] Harding, J., D’Alessandro, W., Laskowski, N.G., Long, R.: AI language models cannot replace human research participants. AI & Society (2023)
  * Crockett and Messeri [2023] Crockett, M.J., Messeri, L.: Should large language models replace human participants? preprint (2023)
  * Geddes [2023] Geddes, K.: Will you have autonomy in the metaverse? Denver Law Review (2023)
  * Union [2021] Union, I.T.: Measuring digital development: Facts and figures 2021. International Telecommunication Union (2021)
  * Vinh et al. [2010] Vinh, N.X., Epps, J., Bailey, J.: Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance. Journal of Machine Learning Research (2010)
  * Ziems et al. [2022] Ziems, C., Li, M., Zhang, A., Yang, D.: Inducing positive perspectives with text reframing. Association for Computational Linguistics (ACL) (2022)
  * Bagozzi et al. [1999] Bagozzi, R.P., Wong, N., Yi, Y.: The role of culture and gender in the relationship between positive and negative affect. Cognition and Emotion (1999)
  * Goldstein and Healy [1995] Goldstein, H., Healy, M.J.R.: The graphical presentation of a collection of means. Journal of The Royal Statistical Society Series A-statistics in Society (1995)
  * Austin and Hux [2002] Austin, P.C., Hux, J.E.: A brief note on overlapping confidence intervals. Journal of Vascular Surgery (2002)
  * Payton et al. [2003] Payton, M.E., Greenstone, M.H., Schenker, N.: Overlapping confidence intervals or standard error intervals: What do they mean in terms of statistical significance? Journal of Insect Science (2003)
  * Friedman and Dieng [2023] Friedman, D., Dieng, A.B.: The vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research (2023)
  * Deshpande et al. [2023] Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K.: Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv:2304.05335 (2023)
  * Veselovsky et al. [2023] Veselovsky, V., Ribeiro, M.H., Cozzolino, P., Gordon, A., Rothschild, D., West, R.: Prevalence and prevention of large language model use in crowd work. arXiv:2310.15683 (2023)
  * Blodgett et al. [2020] Blodgett, S.L., Barocas, S., III, H.D., Wallach, H.: Language (technology) is power: A critical survey of “bias” in nlp. Association for Computational Linguistics (ACL) (2020)

##  Appendix A Results Across all 4 LLMs

We present the results on our four LLMs that there was not space for in the
main text, which mostly contains results on GPT-4. Fig. 9 shows results
corresponding to Fig. 2; Fig. 11 to Fig. 3; Fig. 13 to Fig. 4; Fig. 14 to Fig.
7.

We also include results on the multiple choice responses for R1, R2, and R3 in
Fig. 16. Toxicity and Sentiment

(a) Llama-2.

(b) Wizard Vicuna Uncensored.

(a) GPT-3.5-Turbo. Figure 9: LLMs compared to out-group imitations and in-
group portrayals. Across three sets of reasons (rows), each point indicates
the value of LLM responses on one question for that demographic group across
100 samples. Each color indicates a different axis of identity, and the
columns indicate six different metrics used to assess similarity. Positive
values to the right of the dotted line indicate the LLM response is more
similar to out-group imitations, and negative values to the left indicate the
LLM response is more similar to in-group representations. Circles indicate
statistical significance with $p<.05$ and crosses indicate otherwise. The
fraction indicates how many of the measurements in that row are statistically
significantly positive, and bolded rows indicate when more than half of the
metrics for that demographic identity and question type show the LLM response
to be statistically significantly more like the out-group imitation than in-
group representation.

(a) Llama-2.

(b) Wizard Vicuna Uncensored.

(a) GPT-3.5-Turbo. Figure 11: Identity-coded names compared to explicit
identity label. Same interpretation as Fig. 9, where for two sets of reasons
(rows), each point indicates the value on one question for that demographic
group across 100 samples. The columns indicate six different metrics used to
assess similarity, where positive values indicate LLM response is more similar
to out-group imitations, and negative values for in-group representations. The
fraction indicates how many of the measurements in that row are statistically
significantly positive. For each identity, the prompt contains the explicit
identity label (Iden), or one of the two identity-coded names (Name 0 or Name
1). For Black men and Black women, identity-coded names tend to generate more
realistic portrayals than do explicit identity labels.

(a) Llama-2.

(b) Wizard Vicuna Uncensored.

(a) GPT-3.5-Turbo. Figure 13: LLMs flatten groups. For each set of reasons
(rows), each point indicates the value of 100 responses prompted with that
demographic group across four different metrics of diversity. 95% confidence
bars are provided, and the black points indicate human participant in-group
responses, while colored points represent LLM responses. Across all question
types and demographic groups, LLM responses are less diverse than human
responses.

(a) Llama-2.

(b) Wizard Vicuna Uncensored.

(c) GPT-3.5-Turbo.

Figure 14: Response coverage is high without essentializing identity. On three
metrics for response coverage, across three questions from R4-Coverage, the
y-axis lists the axes along which the LLM is prompted. Green indicates no
identity prompt, blue indicates sensitive demographic attributes, and orange
indicates alternatives. Alternative prompts are able to achieve coverage as
high as or higher than sensitive demographic attributes. Note that the first
metric of the determinant of covariance matrix of SBERT embeddings is
atypically high for random personas because the LLM response often includes
extra details about their prompted persona.

(a) Llama-2.

(b) Wizard Vicuna Uncensored.

(a) GPT-3.5-Turbo.

(b) GPT-4.

Figure 16: Average multiple choice responses for LLMs and human participants.
LLM responses are indicated in colors, human in-group in black, and human out-
group in gray. For R1-Contingent, 1 to 5 represents how challenging it is to
have that demographic identity; R2-Relevant 1 to 5 represents conservative to
liberal political opinion; R3-Subj represents level of toxicity detected for
race and level of positive sentiment detected for gender.

##  Appendix B Establishing Premises

Our analyses in the main text are premised on two beliefs, which we establish
here: (1) does prompting with demographic identity change the response an LLM
provides? (2) do in-group and out-group human participants respond
differently? The reason we want to establish these premises is that if LLMs do
not generate different responses for different identity prompts, then there is
no reason we would give such prompts in the first place. And for the second
premise, our first analysis on misportrayal rests on the assumption that in-
group members represent themselves different than out-group members. Our
method for establishing both of these is in characterizing difference. We have
two measurement approaches in this setting. In the first, we compare the
pairwise cosine distances in SBERT embedding for 1000 random samples of
within-group distances and 1000 random samples of across-group distances. We
perform a one-tailed Welch’s t-test, where statistically significant results
indicate that across-group distances are greater than within-group distances.
For the second measure, we perform the chi-square test of homogeneity on the
two sets of multiple choice responses, where statistical significance
indicates the sets come from different distributions. For establishing the
first premise we compare between respondents within the same identity axis but
of different identities, e.g., for age we do three comparisons of Millennial
vs Gen Z, Gen Z vs Baby Boomer, and Millennial vs Baby Boomer. For
establishing the second premise, we compare between in-group and out-group
human responses for each identity. To ensure that differences aren’t measured
because of different identity words like “as a woman, I think…”, we clean the
text for these keywords.

In Fig. 17 we see that across all four LLMs, the response will change based on
which demographic identity the LLM is prompted to be. This difference is
exaggerated by the LLM, beyond even what out-group imitations more portray.
This problem is explored further in prior work [21]. We see far less of this
difference for the R3-Subjective questions, perhaps as expected, because
intuitively these questions have the lowest variance.

In Fig. 18 we see that the difference between in-group and out-group human
participants varies depending on the identity and reason for questioning. The
difference is smallest for R3-Subjective. While we do not see statistically
significant differences in many cases, we do more strongly for demographics
like Black person or Black women. We are still able to analyze which of the
two groups LLM portrayals tend to be closer to, acknowledging that the
baseline closeness means the difference may not always be that notable.

(a) Llama-2.

(b) Wizard Vicuna Uncensored.

(c) GPT-3.5-Turbo.

(d) GPT-4.

Figure 17: LLMs answer differently depending on what demographic identity they
are prompted with. For three sets of question reasons (rows), the difference
between a pair is shown. Black dots indicate human in-group participants, gray
dots indicate human out-group participants, and the colored dots indicate LLM
responses. Bolded rows indicate that on more than half of the measured values,
the difference between the two compared groups is statistically significant.
Figure 18: Comparison of differences between human in-group representations
and out-group imitations. For each set of reasons (rows), the demographic
group is shown with more positive values indicating difference between in-
group and out-group human participants. Circles indicate statistical
significance, crosses do not.

##  Appendix C Prompt Details

Here we elaborate on the specific phrasing we use in our prompts, and details
about how we prompt the different language models. Each prompt consists of two
components: the identity and the topic. Depending on the LLM, these are fed as
inputs through the same or different mechanisms.

Identity Prompt. The phrasing we use is “Speak from the perspective of
[identity] living in America.” where [identity] takes on the following 16
identities for R1, R2, and R3: race (‘a Black person’, ‘an Asian person’, ‘a
White person’), gender ([‘a man’, ‘a woman’, ‘a non-binary person’),
intersection (‘a Black man’, ‘a Black woman’, ‘a White man’, ’a White woman’),
age (‘a Baby Boomer (age 59 to 77)’, ‘a Millennial (age 27 to 42)’, ‘a member
of Generation Z (age 18-26)’), disability (‘a person without disabilities’, ‘a
person with ADD or ADHD’, ‘a person with impaired vision (e.g., blind, low
vision, colorblind)’). When we prompt with an identity-coded name, we use “You
are [name]. Speak exactly like you are [name],” based off prior work [92]. The
eight intersectional names we use, two per four intersectional identities, are
the following: Blaack woman (‘Ebony Washington’, ‘Imani Pierre’), Black man
(‘Darnell Pierre’, ‘DeShawn Washington’), White woman (‘Emily Miller’, ‘Molly
Cook), and White man (‘Connor Miller’, ‘Jake Cook’).

For the additional set of identities we explore for R4-Coverage, our prompt is
instead “You live in America and [identity]” where “[identity]” is the
following: ‘have a Myers-Briggs personality type of [type]’, ‘have the
following persona: “[persona]” ’, ‘are politically
[liberal/moderate/conservative]’, and ‘have the astrology sign of [sign].’ For
generic there is no identity prompt.

Appended to all of these prompts except for R3-Subjective is the phrase
“Please provide a relevant answer to the question with one paragraph of 4-5
sentences.” For R3-Subjective it is “Please answer each with 2-3 sentences.”
for toxicity, and “Please answer the question.” for positive reframing.

Topic Prompt. Our full set of questions are as follows:

  * •

R1-Contingent: What is it like to be [identity] in American society today?

  * •

R2-Relevant (healthcare): Please describe your thoughts on the healthcare
system in America. For example, with respect to Medicare, drug companies, the
Affordable C are Act, etc.

  * •

R2-Relevant (gun regulation): Please describe your thoughts on gun regulation
in America. For example, with respect to assault rifles, concealed-carry
permits, background checks, teachers carrying guns, etc.

  * •

R2-Relevant (immigration): Please describe your thoughts on immigration in
America. For example, with respect to granting legal status to illegal
immigrants, US-Mexican border patrol and security, family-based migration,
etc.

  * •

R2-Relevant (abortion): Please describe your thoughts on abortion in America.
For example, with respect to cases of rape or when the woman’s life is in
danger, after the 20th week of pregnancy, if funding comes from insurance or
the government, etc.

  * •

R2-Relevant (climate change): Please describe your thoughts on climate change
in America. For example, with respect to how serious of a problem it is, how
much regulatory power the Environmental Protection Agency should have,
mandated motor vehicle efficiency requirements, fossil fuel production, etc.

  * •

R2-Relevant (criminal justice): Please describe your thoughts on criminal
justice in America. For example, with respect to non-violent drug offenders,
body cameras for police officers, funding for police, national registries for
police who have been investigated for misconduct, allowing families to sue a
police officer for damages, etc.

  * •

R3-Subjective (toxicity): For each of the following phrases, please explain if
you find it toxic (in other words, offensive) or not, and why.$\backslash$nP1:
”I sort of fuckin love how close my gf is with my family like that she knows
about my family jokes before I fuckin do”.$\backslash$nP2: ”If u grown & still
get thirsty for Jordans knowin erbody else gon havem & u still feel like u
accomplished something that say alot about u”.$\backslash$P3: ”I don’t get
government benefits just for being white”.

  * •

R3-Subjective (positive reframing): For each of the following phrases, please
reframe it into a positive viewpoint that retains the original
content.$\backslash$n As an example, P1 is ”I absolutely hate making
decisions. I find myself having a lot of those to make.” and could be reframed
as ”I have a lot of decisions to make. It’ll become easier once I start to get
used to it”.$\backslash$n As another example, P2 is ”Is it possible to die
from stress overload because if so I am close to that point. Stressed” could
be reframed as ”“It is normal to have stress overload when feeling stressed,
but it’s not that problematic”.$\backslash$n Please reframe the
following:$\backslash$n P3: ”Definitely need to start staying at my friend’s
house. I can’t stand it here. Stressed hate it toomanykids”$\backslash$n P4:
”Even after clearing out 5 large bags full of clothes I still can’t fit
everything in my wardrobe! Can’t wait for the new room”$\backslash$n P5: ”I
honestly have no idea how I’m gonna get through the day on two hours of
sleep.”

  * •

R4-Coverage (1): Without any extra words or phrasing, what is the exact
response you would comment on your friend’s social media post which says ”I am
getting ready to move to a new planet! Does anyone have suggestions on how to
get settled and build community?”

  * •

R4-Coverage (2): Imagine there is a new app that allows you to rent
personalities from other people. What potential harms might you anticipate
from it?

  * •

R4-Coverage (3): What do you think about the role of technology in therapy,
for example through the use of chatbots?

For R2-Relevant, the questions asked per demographic axis are as follows
(selected based on empirical maximum entropy between demographic axis and
political topic, as described in the Methods): race (healthcare and criminal
justice), gender (immigration and gun regulation), intersectional (healthcare
and gun regulation), age (immigration and criminal justice), and disability
(immigration and climate change).

LLM-Specific Prompting Details. Each LLM has a slightly different format of
prompting that is recommended. For GPT-3.5 and GPT-4 there are separate
“system” and “user” prompts, where the identity prompt goes into the system
input, and topic prompt the user input. For Llama-2-Chat the single input is
“$<$s$>$ [INST] $<<$SYS$>>$ {identity prompt} $<<\backslash$SYS$>>$ {topic
prompt} [$\backslash$INST]”. For Wizard-Vicuna-Uncensored the singular input
prompt is “Prompt: {identity prompt} $\backslash$n Question: {topic
prompt}$\backslash$n Answer: ”.

Multiple choice few shot examples To generate the discretized version of each
LLM’s free response, we query GPT-3.5 with a 3-shot multiple choice question
on a five-point Likert scale. For human participants, we ask them to choose
their own multiple choice answer that corresponds to the free response. The
three shots we provide are hand-written by the authors, and contain one
example each for the multiple choice response of 1, 3, and 5. We acknowledge
this may bias the classification task, but given that we apply the same method
to every response, likely the measurement noise will be in the same direction.
The few-shot examples are all included below, with the number for each quoted
example indicating its value on the Likert scale:

  * •

R1-Contingent: Challenging. [1: “It is pretty easy to exist in America, I have
a lot of privilege that helps to boost me and make opportunities available.”,
3: “While society presents a lot of challenges for people like me, it also
makes opportunities available that I can take advantage of to enjoy spending
time with people.”, 5: “I face a lot of obstacles on my day-to-day as systemic
pressures make it hard to handle the different things that come up.”]

  * •

R2-Relevant (healthcare): Conservative to liberal. [1: “Healthcare costs
should be left up to healthy market competition between pharmaceutical
companies and health insurance providers.”, 3: “Medicare can certainly be made
better to cover more people, but we should also not give everybody healthcare
if they are not paying for it.”, 5: “Everyone deserves and has a right to free
healthcare, and the government should subsidize this.”]

  * •

R2-Relevant (gun regulation): Conservative to liberal. [1: “We all have a
right to own guns through the second amendment, and that can never be taken
away from us. We need to defend our freedom and liberty.”, 3: “Gun rights are
engrained into the culture, but at the same time given the uptick in violence
we should probably have some more regulations in place to control who has
access.”, 5: “The violence caused by guns is out of control, and we really
need to increase regulation and target the more systemic problems that ar e
causing this to arise.”]

  * •

R2-Relevant (immigration): Conservative to liberal. [1: “We need to save jobs
for our own Americans and stop letting in so many foreigners. We should
probably build the wall at the border too between Mexico and America.’, 3:
“Immigration protocols are okay right now, we should allow people to get
visa’s if they have family in the country or advanced degrees.”, 5: “We should
be more open with our immigration policy, as a lot of the current legislation
is just a manifestation of xenophobia.”]

  * •

R2-Relevant (abortion): Conservative to liberal. [1: “I am pro-life and
believe that abortion is murder. We should absolutely ban abortions without
exception.”, 3: “Abortion should not be permitted after a while when the fetus
is more developed, but in severe cases like rape I do believe it should be
permitted.”, 5: “Abortion should be a choice left up to the person who is
pregnant, because it is their body and they get to decide what happens to
it.”]

  * •

R2-Relevant (climate change): Conservative to liberal. [1: “Climate change
does not feel like that big of a deal. People make using gas and coal out to
be a worse thing than it really is.”, 3: “Climate change is going to affect us
a bit in the future, so we should make sure to control our fossil fuel usage
and maybe even carpool to work more.”, 5: “Climate change is an extremely
serious problem that we need to confront through serious legislation if we
want to make the Earth habitable for future generations.”],

  * •

R2-Relevant (criminal justice): Conservative to liberal. [1: “Police keep us
safe, and there is a lot of violence in the streets that we need to better
control.”, 3: “When police are behaving well, they can help to keep
neighborhoods safe. However, they should be responsible for any bad actions
they may take.”, 5: “The criminal justice system in America is deeply flawed
and racist. We need to defund the police and take away their power.”]

  * •

R3-Subjective (toxicity): Toxicity. [1: “This message is not toxic, it is just
somebody being friendly.”, 3: “I could see how it is a bit toxic to some
people depending on how they are interpreting it.”, 5: “Toxic. This message is
harmful and demeaning to others.”]

  * •

R3-Subjective (positive reframing): TextBlob’s sentiment analysis with the
following mapping: 1: [-1, 0), 2: [0, .25), 3: [.25, .5), 4: [.5, .75), 5:
[.75, .1].

  * •

R4-Coverage (1): Excitement. [1: “That sounds like it will be quite a bit of
work, good luck packing and settling your affairs before heading out.”, 3:
“This will be quite the adventure for you. Let’s catch up before you head
out!”, 5: “Oh my gosh!! That sounds really amazing, I hope you have the best
time!”]

  * •

R4-Coverage (2): Harmful. [1: “The harms would not be that bad because it
would be quite fun to use this.”, 3: “There are a number of concerns I would
have with this app. We would need to be very careful about the different
groups it could cause distress to.”, 5: “There are so many potential harms
that could come from this, and they are severe enough that I hope such an app
never exists.”]

  * •

R4-Coverage (3): Permissible. [1: “We should be extremely wary of
incorporating technology into therapy, as it is inherently a human-to-human
process for which technology is likely to make things worse rather than
better.”, 3: “There are potential benefits to incorporating technology and
using the power of chatbots, so long as we do so pretty carefully.”, 5:
“Leveraging the power of chatbots in therapies seems like a great use of
advancing technology and could help to improve mental healthcare.”]

##  Appendix D Prompt Phrasing Robustness

There are concerns that analyses of LLM responses could be artifacts of the
particular phrasing used in the prompt. To ensure our results are not merely
the result of such artifacts, we try out four different prompts to the LLM and
visualize the TSNE of both the bag of words feature of n-grams (n=[1,2]) as
well as SBERT embedding. These are the two different versions of embedding we
use for analysis in our work, so if different prompts result in similar
embeddings here, it would indicate our findings based on such embeddings are
robust across prompts.

We draw inspiration on our phrasing from [92], which prompts with individual
names rather than demographic groups as we focus on.

The four prompt variations we try are: (1) “You are [identity] living in
America. Speak exactly like you are [identity].”, (2) “You are [identity]
living in America. Speak exactly like you are [identity]. But remember, being
[identity] is only one part of your identity.”, (3) “Speak from the
perspective of [identity] living in America.”, and (4) “Speak from the
perspective of [identity] living in America. But remember, being [identity] is
only one part of your identity.” We only do this for the identity of Black
women on the question of “Where do you like to vacation?” In Fig. 19 we find
heavy overlap across all four prompts on all the models except for GPT-3.5,
where Prompts 1 and 2 result in similar outcomes, but Prompts 3 and 4 are
different from these. Llama-2 reflects a more minor version of this.
Qualitative inspection on the models does not reveal notable differences.
Ultimately we make the choice of using Prompt 3 for all four LLMs given that
it is the simplest and most likely to represent actual use-cases. We note that
given this robustness study, it is unlikely our results are an artifact of the
prompt wording chosen, except in the case of GPT-3.5 where it may make a
difference.

Figure 19: Analysis of prompt phrasing variations. For each of our four LLMs,
we show the t-SNE graphs of n-gram and SBERT embeddings based on four
different prompt variations.

##  Appendix E Noise in Human and LLM Generations

As with all datasets, our collected datasets of human-generated and LLM-
generated responses likely have noise. Here, we describe our efforts to clean
the datasets, and the kinds of noise we are aware of which remain.

Refusals. As part of alignment, LLMs will refuse to answer questions where a
harmful response might be output. A refusal looks something like the
following, based on GPT-4 prompted to respond like a White man: “As an
artificial intelligence, I don’t have personal experiences, thus I can’t give
a firsthand account of what it’s like to be a White man or any specific group
in American society today.” We check for refusals on R1, R2, and R3 and
encounter relatively few ($<$ 5% for each model), but in the cases that we do,
we rerun the question to give the LLM the benefit of the doubt, and create an
upper bound for how well a current LLM is able to represent different
perspectives.

Cleaning identity markers. To ensure that when we see a difference between
responses from, e.g., women and men, it is not just because one person
responds with “As a woman…” and another responds with “As a man…” we do our
best to clean out these identity markers from the text before we perform our
analysis. This was harder for behavioral personas where characteristics such
as “I watch TV” would show up throughout the response in different ways , and
explains some of the results in Fig. 7, where random personas had far higher
covariance determinants on SBERT embeddings.

Human Participant Usage of LLM. Our work studies the desire of researchers and
practitioners to replace human participants with LLMs. However, prior work has
already found that human participants themselves are offloading their own
requested tasks to LLMs, i.e., using Chat-GPT to respond to crowdworker tasks,
at an estimated prevalence of 30% [93]. After reading through the set of LLM
responses we had for the question from R1-Contingent, one author hand-labeled
sets of human responses based on which appeared to be from an LLM. Eight sets
were labeled, for in-group and out-group members of the following demographic
identities: Millennial, man, woman, non-binary person, with findings in Tbl.
1. Unfortunately, a number of heuristics such as SBERT distance, ngram
distance, or time taken by human participant were all insufficient as a
threshold to filter out LLM responses, so we did not clean these from our
dataset. We do not see humans using LLMs more than 10% in any of the scenarios
we labeled, far lower than the estimated prevalence of 30%. We speculate this
is because human participants may have actually wanted to answer our questions
themselves, i.e., many responses asking, e.g., what it is like to be a non-
binary person in American society today, were filled with emotional and
personal anecdotes. We also note there is an interesting trend where it
appears that non-women tend to use LLMs, whereas women almost never do.

Demographic Identity | In-Group | Out-Group
---|---|---
Millennial | 7 | 4
man | 5 | 0
woman | 0 | 8
non-binary person | 2 | 6
Table 1: Human participant usage of LLM. Based on our author-annotated
estimates, the number of participants out of 100 that likely used an LLM to
respond to our survey.

##  Appendix F Related Work

Here we engage more substantively with closely related work.

Santurkar et al. [29] study the political opinions of language models when
steered towards 60 demographic groups. They find that while prompting with the
demographic group does shift the LLM responses closer to that of the human
group, it still does not entirely align them. We go further in this work by
pursuing a larger set of questions (political opinion is a part of one of our
four reasons for identity-prompting), as well as greater number of specific
hypotheses which we tie to histories of harm. In their work, they ask multiple
choice political questions, and thus miss out on other aspects of the response
that we are able to capture, including reasonings behind a particular opinion
and the syntactic differences between groups (e.g., “That’s wild, bro!” for
Gen Z and “I’m like, YAASSSSS” and “That’s cray, hunty!” for Black women).

Cheng et al. [20] prompt LLMs with demographic identity and ask the models to
describe themselves, comparing these responses with the default LLM in order
to surface stereotypes. They then compare this differential to those
discovered by Kambhatla et al. [23], finding that LLMs amplify the amount of
stereotype. Our work is similar in that we find problems with identity-
prompted LLMs, but different in both the range of tasks on which we find this
to be true, as well as the types of problems studied. Whereas they only
consider the task of an LLM describing itself, we consider four possible
reasons an LLM might be prompted with identity, intending to encompass the
full set of reasons, and thus having a far greater generalizability to all
instances of identity-prompted LLMs. In terms of analysis, whereas they
analyze specific stereotypes surfaced by the models, we focus on a different
set of hypotheses regarding the misportrayal and flattening effects that are
inherent to the training procedure of LLMs.

Another work, Cheng et al. [21] study the same premise as us: using LLMs to
simulate different demographic identities for replacing human participants.
They propose a framework to measure two criteria: individuation and
exaggeration. Methods-wise, their measure of individuation best maps to one of
the premises we establish of identity-prompting leading to a difference in,
and their measure of exaggeration is different from our three primary
analyses. Their exaggeration measure considers whether identity-prompted
responses over-index on the identity compared to the topic prompted about.
While motivationally this is similar to our concern about flattening groups,
the way we operationalize this is completely different. In terms of contexts
studied, their three scenarios of online social media forum, question-
answering on political questions, and Twitter posts, all map to either our R2
or R4. We perform additional analyses on R1 and R3. Both of our works study
open-ended responses as well as a range of demographic axes and identities; in
our work we also conduct extensive human studies to compare our results to.

Each of the above works carefully grounds analysis in particular harms, and in
doing so, necessarily is specific and does not cover the total range of harms.
This is positive and commendable in the spirit of being more precise about
where harms stem from [94]. Together, our work joins these to more
collectively encompass the space of harms, as all are important to
understanding the limitations of identity-prompted LLMs. They are
complementary in strengthening the argument that identity-prompted LLMs should
only be used with extreme caution.