\begin@twocolumnfalse

The humanlike responses of large language models (LLMs) have prompted social scientists to investigate whether LLMs can be used to
simulate human participants in experiments, opinion polls and surveys. Of central interest in this line of research has been mapping out
the psychological profiles of LLMs by prompting them to respond to standardized questionnaires. The conflicting findings of this research
are unsurprising given that mapping out underlying, or latent, traits from LLMs' text responses to questionnaires is no easy task. To
address this, we use psychometrics, the science of psychological measurement. In this study, we prompt OpenAI's flagship models, GPT-3.5
and GPT-4, to assume different personas and respond to a range of standardized measures of personality constructs. We used two kinds of
persona descriptions: either generic (four or five random person descriptions) or specific (mostly demographics of actual humans from a
large-scale human dataset). We found that the responses from GPT-4, but not GPT-3.5, using generic persona descriptions show promising,
albeit not perfect, psychometric properties, similar to human norms, but the data from both LLMs when using specific demographic profiles,
show poor psychometrics properties. We conclude that, currently, when LLMs are asked to simulate silicon personas, their responses are
poor signals of potentially underlying latent traits. Thus, our work casts doubt on LLMs’ ability to simulate individual-level human
behaviour across multiple-choice question answering tasks.

Keywords: large language models, GPT, psychometrics, simulate, personality, big five
\end@twocolumnfalse

Overview of the data collection and analysis processes Each LLM (GPT-3.5 and GPT-4) is prompted using a template of a persona description and a survey item. The persona description can be either a generic one, constructed using 4-5 random sentences from the PersonaChat dataset zhangPersonalizingDialogueAgents2018, or a silicon one, constructed on the basis of mostly demographic information of humans from a large-scale personality survey rentfrow_regional_2015. Survey items that the LLM is asked to evaluate are across various personality-related constructs. The LLM text response is processed to extract a numeric response on the survey item and then further analysed.

§ INTRODUCTION

The recent advances in AI have led to the development of large language models (LLMs). LLMs’ performance across a range of tasks matches, or even
exceeds, human performance openaiGPT4TechnicalReport2023, but can they simulate the cognitive and psychological characteristics of humans?

If LLMs can indeed simulate human psychological profiles, they could become indispensable to social scientific research. We could run preliminary,
exploratory studies using LLMs for the fraction of a cost. LLMs can also provide another line of evidence towards a specific hypothesis: if the
evidence using human data is of similar strength to what is observed using LLM data, that would increase our confidence in the conclusion. LLM-based
research could also allow us to gather datasets that are difficult or impossible to obtain with humans. Most excitingly, LLMs, consistent with human
psychology, could allow us to create powerful LLM-powered agent-based simulations of different environments, such as social media, a particular
organisation, or entire countries gaoS3SocialnetworkSimulation2023,heHomophilyArtificialSocial2023, wangSurveyLargeLanguage2023,
kwonWHATWHENHOW2023, liLargeLanguageModelEmpowered2023, liMetaAgentsSimulatingInteractions2023,
parkGenerativeAgentsInteractive2023,wangSurveyLargeLanguage2023. This could serve as a behavioural sandbox, where various hypotheses, ranging from
the effects different news feed algorithms tornbergSimulatingSocialMedia2023, recommendation systems wangSurveyLargeLanguage2023
or country-wide policies gaoS3SocialnetworkSimulation2023, could be tested before being piloted or implemented.

Social scientists have already begun investigating whether LLMs can simulate
human behaviours by using them as participants to study
cognitionbinzUsingCognitivePsychology2023, political and public
opinion argyleOutOneMany2023,santurkarWhoseOpinionsLanguage2023,
economic decision making
and theory of mind
among others aherUsingLargeLanguage2023. To understand their
underlying personality, LLMs have also been used as participants in surveys using
standardized psychological
questionnaires deshpandeToxicityChatGPTAnalyzing2023,giorgiSleptBabyUsing2023,guptaInvestigatingApplicabilitySelfAssessment2023,karraEstimatingPersonalityWhiteBox2023,liAreYouMasquerade2023,pellertAIPsychometricsAssessing2022,romeroGPTLanguageModels2023,stevensonPuttingGPT3Creativity2022.
Many of those studies have explored the ability of LLMs to simulate
human behaviours caronIdentifyingManipulatingPersonality2022,huangWhoChatGPTBenchmarking2023,jiangPersonaLLMInvestigatingAbility2023,miottoWhoGPT3Exploration2022.
For instance, huangRevisitingReliabilityPsychological2023 examined
the personality of GPT-3.5 using a standardized personality scale, varying
instructions, item phrasing, language, choice labels, and choiceordering,
resulting in 2,500 configurations. Although they conclude that GPT-3.5 shows
“consistency in
responses” [1]huangRevisitingReliabilityPsychological2023, their
analyses are limited, focused on summary scores, and do little to
psychometrically falsify the conclusion. There is also a comparable number of
studies that use similar methodology and find mixed evidence of LLMs’ ability to
simulate human behaviours aherUsingLargeLanguage2023,aiCognitionActionConsistent2024,binzUsingCognitivePsychology2023,huQuantifyingPersonaEffect2024,parkDiminishedDiversityofthoughtStandard2024,santurkarWhoseOpinionsLanguage2023.

One particularly thorny problem when treating LLMs as survey participants is
that using the same prompt to obtain model responses to a survey item multiple
times results in close-to-uniform response distributions (e.g., see Experiment 1
in ,
), which is of limited
utility for inferential analysis. Modifying model hyper-parameters, such as
temperature, also does not seem to solve the issue

serapio-garciaPersonalityTraitsLarge2023 proposed a solution to this
problem by encouraging variation in item responses using generic persona
prompting: ask the LLM to adopt a generic persona description and respond to the
survey items as that persona. The authors incorporated randomly sampled
descriptions from the PersonaChat dataset
zhangPersonalizingDialogueAgents2018 into multiple choice question
answering tasks and found that Google’s PaLM models provided reliable and valid
personality test responses. The idea behind generic persona prompting is to
ensure that the LLM is not influenced in any particular direction, and thus, its
responses represent a more random sample of the corpus on which it was trained.

Another solution of greater interest is to use more targeted persona
descriptions, an approach referred to as silicon persona prompting.
argyleOutOneMany2023, introduced this approach in the context of
eliciting voting preferences from LLMs: they found that when GPT-3.5 was
prompted using persona descriptions, created from basic demographic variables
(age, sex, ethnicity, religious attendance, etc.), it was able to reproduce
human voting preferences. Crucially, the authors used census data, so they were
able to calculate differences in voting preference distributions between actual
humans from the census data and LLMs that assumed equivalent personas.

Regardless of what kind of persona descriptions are used in the prompting,
however, the problem of understanding the psychology of LLMs from text responses
is challenging, rendering the extant conflicting findings unsurprising. To
address this issue, we use frameworks from psychometrics, the science of
psychological measurement rustModernPsychometricsScience2021.
Psychometrics offer a robust and rigorous methodology for capturing underlying,
or latent, constructs in generative AI systems that give rise to behaviours
across a range of tasks

Our aim was to conduct a comprehensive, psychometrically-informed analysis of
LLM abilities to simulate human responses both using generic persona prompting
and silicon persona prompting. Broadly speaking, with generic persona prompting,
we expected LLM responses to be similar to what would be expected from a sample
of the population, while with silicon persona prompting, we expected that the
responses would closely match those of the targeted personas. To do this, we
used OpenAI’s flagship models, GPT-3.5 and GPT-4, due to their growing
popularity and state-of-the-art performance

To construct our silicon samples, we used a representative human sample from a
large survey conducted in the UK in collaboration with the British Broadcasting
Corporation (BBC) between November 2009 and April 2011 [All files are
available from the UK Data Archive database (SN 7656—BBC Big Personality Test,
2009–2011: Dataset for Mapping Personality across Great Britain). The catalogue
record created for this data collection can be viewed at the following URL:
rentfrow_regional_2015. The survey received 588,014 responses and
covered various questions, including demographics, education, work,
relationships, personality, and health. For the current study, we focus on a
subset of variables, including demographics and personality scores. After
excluding entries with any missing responses and duplicates, we use a resulting
subset of N=123,828 responses as our ground truth human dataset to compare with
LLM responses.

§ METHODS

Figure <ref> summarises our methodology. Data and code are available
at <https://github.com/nikbpetrov/LLMs-Simulate-Humans>.

We created a structured prompting template, similar to that used by
serapio-garciaPersonalityTraitsLarge2023. The template included a
personality instruction, persona description, test instruction, item statement,
and item postamble. Given that
serapio-garciaPersonalityTraitsLarge2023 found that test instruction
and item postamble variations did not yield substantial differences in model
responses, we did not vary those. We then used this template to query GPT-3.5
and GPT-4.

We sent two sets of queries to each model, with each set using a different
persona description.

For the first set of queries, we used 150 generic persona descriptions randomly
sampled from the PersonaChat dataset
zhangPersonalizingDialogueAgents2018. Each description was four or
five sentences long.

For the second set of queries, we used 1,000 silicon personas
argyleOutOneMany2023 based on the BBC personality dataset. After
minimal data processing of the original dataset (selecting only needed
variables, removing blank responses and duplicates), we were left with
N=123,828, of which we selected 1,000 entries randomly. To construct persona
descriptions, we used the following variables: age, sex, ethnic background,
country, education (self and parents’), occupation, income, relationship status,
and number of children (see Appendix <ref> for details).

As part of the template, we partially replicated the design of
serapio-garciaPersonalityTraitsLarge2023. We administered a
personality test, namely the Big Five Inventory , alongside eight tests of
personality-related constructs: the Positive Affect and Negative Affect
subscales of the Positive and Negative Affect Schedule (PANAS;
), the Buss-Perry Aggression
Questionnaire’s (BPAQ) Physical Aggression, Verbal Aggression, Anger, and
Hostility subscales bussAggressionQuestionnaire1992, and the Creative
Self-Efficacy and Creative Personal Identity subscales of the Short Scale of
Creative Self (SSCS; ,
), resulting in a total of 104 items. We
varied the item statement and item postamble based on the test (see Appendix <ref>
for details on these measures).

Item-level response distributions across all responses split by whether the first token was a digit or not LLMs (rows) produce text responses which we further processed to extract a numeric response. Past research has used only responses whose first digit is a numeric one and discarded the rest. Here, we compare the relative frequency (y-axis) of item-level responses across all survey items (x-axis), split by whether the first token was a digit (columns). We see very different distributions of responses across all LLMs and prompting variations when the first token is a digit one vs a non-digit one.

Using this template, we constructed a total of 239,200 prompts. Each prompt was
part of a separate OpenAI API[<https://platform.openai.com>]
call[An alternative approach would be to include “response memory”,
such that the questions the LLM has access to its previous responses. However,
miottoWhoGPT3Exploration2022 found that this does not make a
statistically significant difference in aggregate scores when they tested
GPT-3.5 with a human values scale.] to ‘gpt-3.5-turbo-1106’ and
‘gpt-4-1106-preview’ using a temperature of 0. Unlike
serapio-garciaPersonalityTraitsLarge2023, we did not have access to
the log probabilities of possible model completions to prompts, as OpenAI’s API
does not allow access to these at the time of running this experiment (January
’24). To circumvent this problem, some have restricted the maximum tokens of the
GPT models’ responses to 1 and discarded non-digit responses
In our testing, we found that if we allowed the model to respond freely, while
sometimes the first token was a non-digit one, usually the sentences still
contained a sensible response. We also observed that when we were looking only
for a single-digit response, or using only responses where the first character
was a digit, the resulting response distribution was close to uniform with an
overwhelming proportion of “3”s (after we collected the full dataset, we
confirmed this initial observation – see Figure <ref>). To
incorporate this insight, we restricted the maximum tokens of the responses to
50 for GPT-3.5 and to 200 for GPT-4 (during testing we found that GPT-4 took
more tokens to generate a numeric answer). For each text response we took the
first digit (regardless of its position) and considered this an item response.

Overall, we executed 239,200 API calls: 15,600 were made using generic persona
descriptions to both GPT-3.5 and GPT-4, and 104,000 were made using silicon
samples to both GPT-3.5 and GPT-4.

§ RESULTS

We conducted a psychometrically comprehensive analysis of our tested LLMs’
responses across several metrics. First, we examined the reliability (internal
consistency) of the responses (i.e. do the responses form consistent patterns
across different items). Next, we studied the construct validity of their
responses (i.e., whether the responses actually reflect the underlying
construct) in the form of internal validity (intercorrelations between Big Five
traits) and criterion validity in the form of correlations between different
constructs (e.g., Extraversion correlated with Positive and Negative affect
scores). As a further validity examination of the LLM responses, we looked at
the proportional frequency of item-level responses, as this would help us
understand whether the summary scores are reached in the same way as with
humans. As a yet further validity test, we checked the structural validity of
LLMs’ responses by fitting a confirmatory factor analysis model, expecting a
similarly good fit as with human data both in terms of path loadings as well as
fit indices. Last but not least, for the silicon personas, we tested how
accurately the personality scores based on LLMs’ data match those of the actual
humans, when conditioning on the same background information. Across all our
analyses, we break the results down by persona type (generic and silicon) and
LLM (GPT-3.5 and GPT-4). Where relevant, we include data from humans for

First, in order to examine the internal consistency of all subscales, we used
three metrics: Cronbach’s $\alpha$
cronbachCoefficientAlphaInternal1951, Greatest Lower Bound
(glb; ,
), and McDonald’s $\omega$
mcdonaldTestTheoryUnified1999. The combination of these metrics
allowed us to paint a comprehensive picture of the performance of the LLMs
across the subscales. While there were few differences between those
coefficients, overall, they showed a similar pattern when comparing across
models and subscales
Thus, hereon, we focus our attention on $\alpha$ as it is the most widely used
measure and it consistently showed the lowest value, giving us a lower bound of
internal consistency.

Internal consistency of all measures across LLM models and prompting styles We computed three reliability indices (colour) for every questionnaire (columns) and its subscales (x-axis) across all LLMs and prompting variations (rows). The plot shows that a) differences between reliability indices are few and b) the reliability of the data from LLMs, when using silicon sampling, can be very low <.70 for some measures.

For Cronbach’s $\alpha$ values, we expected the values to exceed the commonly
used threshold of .70 used in psychometrics
nunnallyPsychometricTheory1978. As can be seen from Figure <ref>, when
generic personas were used with either GPT-3.5 and GPT-4, the data exhibited
acceptable internal consistency: all subscale $\alpha$ values were $\ge$ .70, with
the exception of those for Extraversion and Verbal Aggression, where GPT-3.5
fared worse (.58 and .63, respectively). Reliability was significantly worse for
the silicon personas with GPT-3.5 and GPT-4, where $\alpha$ values were $\ge$ .70
for only 3 or 4 of the 13 subscales, respectively. Notably, there were some
values in the .10–.50 range (6 out of 13 for GPT-3.5 and 3 out of 13 for GPT-4).
This pattern of findings calls into question the test-level reliability of LLMs’
responses using silicon personas.

Intercorrelations between Big Five traits across LLM models and prompting styles Intercorrelations between Big Five traits were computed for the tested LLMs across prompting styles (bottom two rows) and human data from a large representative sample rentfrow_regional_2015 is shown on the top row. The plots show that data from LLMs tend to produce much higher intercorrelations.

Next, we examined the internal validity of the LLMs’ responses by computing the
intercorrelations between Big Five traits and comparing them to those observed
in human data. As can be seen from Figure <ref>, when the generic personas were used
with either GPT-3.5 or GPT-4, the resultant correlations were substantially
higher relative to those in human data. When silicon personas were used with
either LLM, the correlations were still higher than in human data, but the
results were more ambiguous.

Criterion validity correlations across LLMs and prompting styles Selected Pearson’s correlations were computed between Big Five traits (columns) and personality-related constructs (x-axis) to test the criterion validity of the tested LLMs (bottom 4 rows). Comparison data from serapio-garciaPersonalityTraitsLarge2023 is shown on the top row. NB: serapio-garciaPersonalityTraitsLarge2023 used the IPIP-NEO to measure Big Five traits, while we used the BFI, though the authors also show that the two subscales are correlated at >.90.

We then examined the criterion validity of LLMs’ responses by computing the
correlations between various personality constructs, similar to
serapio-garciaPersonalityTraitsLarge2023 and expected similar
results. As can be seen from Figure <ref>, LLM personality test data derived from
generic personas adequately correlated with personality-related criteria tests,
with GPT-4 data faring marginally better. Data generated from silicon personas,
however, showed significantly poorer psychometric properties: with some minor
exceptions, the data from either GPT-3.5 or GPT-4 resulted in weaker
correlations relative to those derived from generic personas and much weaker
relative to what serapio-garciaPersonalityTraitsLarge2023 reported.
While this could be due to the poor internal consistency of the tests, it
nonetheless highlights problems with the silicon persona prompting approach.

Item-level frequencies across LLMs and prompting styles The relative frequency (y axis) of item-level responses (x axis) across Big Five traits (columns) and LLMs (bottom 4 rows) is shown, along with the data from a representative human sample (, ; top row). We see that there are differences in the distributions between prompt variations – generic vs. silicon – and that the distributions when using silicon prompting across both GPT-3.5 and GPT-4 are quite different from human ones.

Next, we examined the distributions of raw responses of LLMs to BFI items –
Figure <ref>. With generic personas, data from GPT-3.5 fared the worst by producing
close to uniform distributions, with the majority of the responses being “3”.
With silicon personas, data from either GPT-3.5 or GPT-4 produced skewed
distributions in the direction of a ‘desirable’ BFI trait: high Agreeableness,
high Conscientiousness, high Extraversion, low Neuroticism and high Openness.
While these were not uniform, they were clustered around responses “2” and “4”.
The data from GPT-4 with generic personas showed promising distributions, most
closely matching those in human data.

Results from confirmatory factor analyses on the BFI scale across all models and prompting styles Summary of the results from confirmatory factor analyses with the same model being fit on different data sources (columns). Panel A shows the path loadings from each observed variable (i.e., item, y-axis) to its latent factor (i.e., dimension, x-axis). Panel B shows the covariances between latent factors. Overall, relative to human data, data from LLMs shows smaller path loadings and higher latent factor covariances.

We then extended our examination of the BFI items by checking their structural
validity. We ran confirmatory factor analyses on the responses of different
models expecting to recover the Big Five structure as with human data (we did
the same for the rest of the scales and reached similar conclusions as with the
BFI, see Appendix <ref>). For each data source, we performed a confirmatory factor
analysis using a maximum-likelihood estimation with robust standard errors;
results are visualised in Figure <ref>. For each model, we report three fit
indices[Notably, the fit indices, even for the ground truth human data,
seem lower than acceptable thresholds hairMultivariateDataAnalysis2019, though
we reserve any judgement as to why that might be. Extant literature points to
differences between confirmatory factor analysis models and exploratory
structural equation models
In any case, we focus our interpretations on relative differences between
models, not absolute interpretations of the models’ fit.]: one absolute (GFI),
one relative (IFI), and one based on the noncentrality parameter (RMSEA)
byrneStructuralEquationModeling2010,newsomClarificationsRecommendationsFit2018 – see Table <ref>.
Notably, the model estimation for the data from GPT-4 using silicon personas
failed as the covariance matrix was not positive definite (one eigenvalue was
negative), most likely due to high covariances between latent variables,
suggesting that the Big Five factor structure was not recoverable from the data.
As for the remaining models, for both GPT-3.5 with generic personas and GPT-3.5
with silicon personas, the loadings were all quite low (majority of loadings
<.50), while for GPT-4 with generic personas, the loadings were comparable to
those from human data, with the exception of the Neuroticism factor, where all
loadings, except one, were below .40. Examining the fit indices (Table <ref>), we
witness a classic case of a reliability paradox
for the data from GPT-3.5 (both generic and silicon personas): despite poor data
quality and thus low path loadings, fit indices indicate a decent fit,
comparable to that from human data. The IFI, our relative fit index, is less
susceptible to this paradox milesTimePlaceIncremental2007, which is
why it shows a poorer fit compared to our other indices. The data from GPT-4
with generic personas performs worse on the fit indices, relative to human data,
most likely due to the weakly-performing Neuroticism factor. Overall, we
conclude that data from current-generation OpenAI LLMs conditioned on our two
prompting styles do not recover the expected factor structure and thus have do
not show sufficient structural validity.

Fit indices from the confirmatory factor analyses on the BFI scale across different data sources.
A confirmatory factor analysis using a maximum-likelihood estimation with robust standard errors was performed for each data source and 3 fit indices reported.

Data source                   GFI     IFI     RMSEA
Human data (BBC)              .81     .75     .06

Generic sampling, GPT-3.5     .62     .65     .09

Generic sampling, GPT-4       .52     .64     .11

Silicon sampling, GPT-3.5     .80     .57     .06

Last but not least, for the responses that used silicon personas, we examined
how well the LLMs replicated the responses of humans given the same persona
description. To do that, for each persona, we took the absolute difference
between the trait score from the LLMs’ data and that of the actual person, i.e.
the ground truth trait score. We call this measure ‘trait bias’, with higher
scores indicating higher mismatch between the trait that the LLM data produces
and that of the human with the same background information.

Bias of LLM-assumed traits when using silicon personas We calculated the absolute difference between personality scores on each trait for human and LLM-assumed personality, conditioned on the same demographic information – this difference, which we call trait bias, is on the y-axis. Across traits, we find no statistically significant differences between GPT-3.5 and GPT-4, except for BFI Agreeableness, with average trait bias across all traits at .63 and .62 for GPT-3.5 and GPT-4, respectively.

We found a statistically significant difference between GPT-3.5 and GPT-4 in
their trait bias regarding BFI Agreeableness, t(1968.5) = 2.83,
p = .005, but not for any other traits (all other ps > .647),
with the average bias across all traits being M=.63 (SD=.25)
for GPT-3.5 and M=.62 (SD=.24) for GPT-4 – see Figure <ref>.

Correlations between ground truth demographic variables of silicon samples and average trait bias of LLM models We correlated the average trait bias (i.e. the absolute difference between personality scores of humans and LLMs, conditioned on the same background information) across all traits with the demographic variables (right-hand side) and the personality scores of humans (left-hand side). The plots show that the inaccuracy of personality scores displayed by LLMs (i.e. high trait bias) is not correlated with demographic variables but is positively correlated with Neuroticism, and negatively correlated with the other Big Five traits, suggesting that LLMs are less capable of representing individuals who tend to be more disagreeable, unconscientious, introverted, emotionally stable, and not open to experience.

We further examined the source of the trait bias by correlating the average bias
across all traits with all variables from the human data. We found that the
trait bias did not correlate with any of the demographic variables, but it
showed moderate correlations with personality traits. Specifically, we found
that greater failure of the LLMs to simulate human behaviour (i.e. high trait
bias) was associated with lower agreeableness, lower conscientiousness, lower
extraversion, higher neuroticism and lower openness, with GPT-4 consistently
showing marginally smaller correlations relative to GPT-3.5 – see Figure <ref>.

§ DISCUSSION

Whether it’s reducing costs, collecting hard-to-acquire data, or conducting
full-scale simulations using simulated agents, the value of using LLMs as a tool
that can simulate human behaviour is undeniable. With an initial wave of
investigations showing conflicting evidence, a robust and rigorous approach is
needed to map out the psychosocial abilities of LLMs.

To this end, we conducted a psychometric investigation of whether OpenAI’s
flagship LLM models, namely GPT-3.5 and GPT-4, can be prompted to simulate human
psychological behaviours. We asked both models to adopt two different kinds of
personas: either generic ones, described using four or five short random
sentences, or silicon ones, described using mostly demographic variables of
actual humans from a large representative survey.

Overall, our results show that GPT-4, but not GPT-3.5, can be prompted to
simulate generic personas, similar to Google’s PaLM models
serapio-garciaPersonalityTraitsLarge2023. However, our additional
psychometric analyses add some caveats to these results: the data from GPT-4
still showed higher intercorrelations between the Big Five traits compared to
what is seen in human data and its responses did not show solid structural
validity as we were unable to fully recover the Big Five latent factors.

More discouragingly, however, we found that both GPT-3.5 and GPT-4 failed in
simulating human responses using silicon personas. Specifically, when the LLMs
were prompted to assume a specific persona, their responses showed poor
psychometric properties across the board. This is in contrast to
argyleOutOneMany2023, whose silicon persona approach we used. They
found that LLMs simulated political opinions based on demographic data. We think
the likely reason for this is that they aimed to simulate surface-level
correlations between demographic information and voting preferences. In
contrast, we examined latent traits. As such, it is conceivable that information
(i.e., word co-occurrences) about demographics and voting preferences are more
strongly present in the LLMs’ training data compared to latent constructs,
which, by definition, require deeper-level representations.

We note two additional findings from our study. First, across the board, data
from GPT-4 exhibited better psychometric properties than data from GPT-3.5. This
may reflect the overall better performance of GPT-4 compared to GPT-3.5 across
various LLM benchmarks openaiGPT4TechnicalReport2023. This converges
with the data from serapio-garciaPersonalityTraitsLarge2023, who
found that larger PaLM models generated data with better psychometric
properties. We do note, however, that we remain agnostic as to what exactly
drives GPT-4 to better represent latent traits in our study (not least of all
because of the fact that both models are closed-source). It could be because of
a higher number of parameters, different architectures, more (or better quality)
training data, differences in fine-tuning, or a combination of all of the above.
We encourage future research to investigate the effect of different training
variations on the ability of an LLM to represent latent traits. Second, we found
evidence that instruction fine-tuning drives LLMs to behave in a more desirable
way. Using our trait-bias metric, we found that LLMs were more accurate in
reproducing the personality profile of a person who is more agreeable, more
conscientious, less neurotic, more open to experience, and more extraverted. We
speculate this is, at least in part, due to the instruction fine-tuning by
OpenAI to align LLMs to behave in a more congenial, amicable way.

Our study is not without its limitations. Most notably, our examination, while
comprehensive in its analysis, was limited in its method: we relied on a
mono-method approach, only using self-report data. While we believe that this
type of data is the most useful in order to examine the ability of LLMs to
represent latent traits, we think that gathering other kinds of data, such as
asking LLMs to respond to scenarios hagendorffDeceptionAbilitiesEmerged2023,kangValuesOpinionsPredicting2023,
multi-turn conversation, or asking it to produce text jiangPersonaLLMInvestigatingAbility2023,serapio-garciaPersonalityTraitsLarge2023, could enrich our understanding
campbellConvergentDiscriminantValidation1959. Furthermore, our attempt to prompt LLMs to replicate the performance of a
specific subpopulation relied on silicon persona prompting, which relied on
individual demographic data. While we think LLMs should ultimately be able to
indirectly extrapolate personality profiles on the basis of demographic
variables, as there are known correlations between personality and demographics
goldbergDemographicVariablesPersonality1998,sotoAgeDifferencesPersonality2011, these associations are not strong.
This could partly explain why LLMs were better at making consistent inferences
using the generic personas: personality is defined as characteristic patterns of
thinking, feeling, and behaving and the generic personas contained more specific
references to such patterns. Future research can use human datasets that contain
ground truth personality scores, in addition to both demographic variables as
well as other data on characteristic patterns of thinking, feeling and
behaviour, such as hobbies or interests, and then use those as seeds to create
silicon personas.

Overall, our results, while contradicting earlier findings that suggested the
ability of LLMs to simulate human behaviours
are in line with recent findings that cast doubt on current LLM abilities.
Specifically, aiCognitionActionConsistent2024 Ai et al. (2024)
conclude that there is a “limitation in LLMs’ ability to authentically replicate
human personality dynamics, often reflecting a bias towards socially desirable
responses” and huQuantifyingPersonaEffect2024 conclude that
silicon-style “persona prompting cannot reliably simulate different
perspectives”. Likewise, we conclude that LLMs are not yet ready to fully
simulate individual-level human behaviours.

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

journaltitleInternational Conference on Machine Learning
titleUsing Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

abstractIn this study, we investigate the reliability of Large Language Models (LLMs) in professing human-like personality traits through responses to personality questionnaires. Our goal is to evaluate the consistency between LLMs’ professed personality inclinations and their actual "behavior", examining the extent to which these models can emulate human-like personality patterns. Through a comprehensive analysis of LLM outputs against established human benchmarks, we seek to understand the cognition-action divergence in LLMs and propose hypotheses for the observed results based on psychological theories and metrics.
shorttitleIs Cognition and Action Consistent or Not
titleIs Cognition and Action Consistent or Not: Investigating Large Language Model's Personality

abstractWe propose and explore the possibility that language models can be studied as effective proxies for specific human subpopulations in social science research. Practical and research applications of artificial intelligence tools have sometimes been limited by problematic biases (such as racism or sexism), which are often treated as uniform properties of the models. We show that the “algorithmic bias” within one such tool—the GPT-3 language model—is instead both fine-grained and demographically correlated, meaning that proper conditioning will cause it to accurately emulate response distributions from a wide variety of human subgroups. We term this property algorithmic fidelity and explore its extent in GPT-3. We create “silicon samples” by conditioning the model on thousands of sociodemographic backstories from real human participants in multiple large surveys conducted in the United States. We then compare the silicon and human samples to demonstrate that the information contained in GPT-3 goes far beyond surface similarity. It is nuanced, multifaceted, and reflects the complex interplay between ideas, attitudes, and sociocultural context that characterize human attitudes. We suggest that language models with sufficient algorithmic fidelity thus constitute a novel and powerful tool to advance understanding of humans and society across a variety of disciplines.
issn1047-1987, 1476-4989
journaltitlePolitical Analysis
shortjournalPolit. Anal.
shorttitleOut of One, Many
titleOut of One, Many: Using Language Models to Simulate Human Samples

Argyle et al. - 2023 - Out of One, Many Using Language Models to Simulat.pdf

abstractWe study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3’s decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3’s behavior is impressive: It solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multiarmed bandit task, and shows signatures of model-based reinforcement learning. Yet, we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. Taken together, these results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.
issn0027-8424, 1091-6490
journaltitleProceedings of the National Academy of Sciences
shortjournalProc. Natl. Acad. Sci. U.S.A.
titleUsing Cognitive Psychology to Understand GPT-3
:_libraryand Perry - 1992 - The aggression questionnaire..pdf

London, UK

shorttitleStructural Equation Modeling with AMOS
titleStructural Equation Modeling with AMOS: Basic Concepts, Applications, and Programming

American Psychological Association

abstract"This paper advocates a validational process utilizing a matrix of intercorrelations among tests representing at least two traits, each measured by at least two methods. Measures of the same trait should correlate higher with each other than they do with measures of different traits involving separate methods. Ideally, these validity values should also be higher than the correlations among different traits measure by the same method." Examples from the literature are described as well as problems in the application of the technique. 36 refs. (PsycINFO Database Record (c) 2016 APA, all rights reserved)
journaltitlePsychological Bulletin
titleConvergent and Discriminant Validation by the Multitrait-Multimethod Matrix

Statistical Validity,Test Validity

abstractPsychology research has long explored aspects of human personality such as extroversion, agreeableness and emotional stability. Categorizations like the ‘Big Five’ personality traits are commonly used to assess and diagnose personality types. In this work, we explore the question of whether the perceived personality in language models is exhibited consistently in their language generation. For example, is a language model such as GPT2 likely to respond in a consistent way if asked to go out to a party? We also investigate whether such personality traits can be controlled. We show that when provided different types of contexts (such as personality descriptions, or answers to diagnostic questions about personality traits), language models such as BERT and GPT2 can consistently identify and reflect personality markers in those contexts. This behavior illustrates an ability to be manipulated in a highly predictable way, and frames them as tools for identifying personality traits and controlling personas in applications such as dialog systems. We also contribute a crowd-sourced data-set of personality descriptions of human subjects paired with their ‘Big Five’ personality assessment data, and a data-set of personality descriptions collated from Reddit.
titleIdentifying and Manipulating the Personality Traits of Language Models
F87V5ECand Srivastava - 2022 - Identifying and Manipulating the Personality Trait.pdf

issn0033-3123, 1860-0980
titleCoefficient Alpha and the Internal Structure of Tests

abstractLarge language models (LLMs) have shown incredible capabilities and transcended the natural language processing (NLP) community, with adoption throughout many services like healthcare, therapy, education, and customer service. Since users include people with critical information needs like students or patients engaging with chatbots, the safety of these systems is of prime importance. Therefore, a clear understanding of the capabilities and limitations of LLMs is necessary. To this end, we systematically evaluate toxicity in over half a million generations of CHATGPT, a popular dialogue-based LLM. We find that setting the system parameter of CHATGPT by assigning it a persona, say that of the boxer Muhammad Ali, significantly increases the toxicity of generations. Depending on the persona assigned to CHATGPT, its toxicity can increase up to 6×, with outputs engaging in incorrect stereo- types, harmful dialogue, and hurtful opinions. This may be potentially defamatory to the persona and harmful to an unsuspecting user. Furthermore, we find concerning patterns where specific entities (e.g., certain races) are targeted more than others (3× more) irrespective of the assigned persona, that reflect inherent discriminatory biases in the model. We hope that our findings inspire the broader AI community to rethink the efficacy of current safety guardrails and develop better techniques that lead to robust, safe, and trustworthy AI systems.
shorttitleToxicity in ChatGPT
titleToxicity in ChatGPT: Analyzing Persona-assigned Language Models
3HUet al. - 2023 - Toxicity in ChatGPT Analyzing Persona-assigned La.pdf
ine Learning

abstractWith large language models (LLMs) appearing to behave increasingly human-like in text-based interactions, it has become popular to attempt to evaluate various properties of these models using tests originally designed for humans. While reusing existing tests is a resource-efficient way to evaluate LLMs, careful adjustments are usually required to ensure that test results are even valid across human sub-populations. Thus, it is not clear to what extent different tests’ validity generalizes to LLMs. In this work, we provide evidence that LLMs’ responses to personality tests systematically deviate from typical human responses, implying that these results cannot be interpreted in the same way as human test results. Concretely, reverse-coded items (e.g. “I am introverted” vs “I am extraverted”) are often both answered affirmatively by LLMs. In addition, variation across different prompts designed to “steer” LLMs to simulate particular personality types does not follow the clear separation into five independent personality factors from human samples. In light of these results, we believe it is important to pay more attention to tests’ validity for LLMs before drawing strong conclusions about potentially ill-defined concepts like LLMs’ “personality”.
titleDo Personality Tests Generalize to Large Language Models?
orner et al. - 2023 - Do personality tests generalize to Large Language .pdf
ine Learning

abstractSimulation plays a crucial role in addressing various challenges within social science. It offers extensive applications such as state prediction, phenomena explanation, and policy-making support, among others. In this work, we harness the human-like capabilities of large language models (LLMs) in sensing, reasoning, and behaving, and utilize these qualities to construct the S3 system (short for Social network Simulation System). Adhering to the widely employed agent-based simulation paradigm, we employ fine-tuning and prompt engineering techniques to ensure that the agent’s behavior closely emulates that of a genuine human within the social network. Specifically, we simulate three pivotal aspects: emotion, attitude, and interaction behaviors. By endowing the agent in the system with the ability to perceive the informational environment and emulate human actions, we observe the emergence of population-level phenomena, including the propagation of information, attitudes, and emotions. We conduct an evaluation encompassing two levels of simulation, employing real-world social network data. Encouragingly, the results demonstrate promising accuracy. This work represents an initial step in the realm of social network simulation empowered by LLM-based agents. We anticipate that our endeavors will serve as a source of inspiration for the development of simulation systems within, but not limited to, social science.
titleS3: Social-network Simulation System with Large Language Model-Empowered Agents

abstractLiars and truth-tellers often communicate differently. When tasked with writing deceptive text, humans use prior knowledge and experiences to intentionally deceive their target. Automatically generated text via large language models; LLMs, on the other hand, mirror training instances that are most likely written by humans. In the case of content like reviews, automatically generated language is inherently deceptive because the system is not grounded in material world experiences. In this paper, we characterize differences between (a) truthful text written by humans, (b) intentionally deceptive text written by humans, and (c) inherently deceptive text written by state-of-the-art language models (ChatGPT). We examined the expression of thirteen psychologically grounded and fundamental human traits (e.g., personality and empathy) across truthful and deceptive hotel reviews, finding that texts written by humans were more diverse (had more variation) in their expressions of personality than texts written by ChatGPT. Across all human traits we found that truthful and deceptive human language was easier to distinguish from machine generated language. Building on these differences, we trained a classifier using only the thirteen human traits to automatically discriminate between truthful and deceptive language, with a classification AUC of up to 0.966. Thus, despite the fact that large language models mirror text written by genuine (and truthful) humans, their lack of diversity in human traits makes them easier to identify. These results suggest that psychologically grounded human traits offer a robust feature set unaffected by the “human-ness” of LLM language, and further suggest that AI and humans are behaviorally different when communicating about experiences.
title“I Slept Like a Baby”: Using Human Traits To Characterize Deceptive ChatGPT and Human Text

journaltitlePersonality and Individual differences
shorttitleDemographic Variables and Personality
titleDemographic Variables and Personality: The Effects of Gender, Age, Education, and Ethnic/Racial Status on Self-Descriptions of Personality Attributes

abstractMarsh (2007) asserts that no self-reported questionnaire of the Big Five factors shows an acceptable degree of adjustment in confirmatory factor analyses (CFA). Based on CFA results, it can usually be inferred that exploratory factor analysis (EFA) solutions are incorrect, as well as concluded that the instruments evaluated by EFAs do not measure what they intend to measure. The present study investigates this assertion, evaluating the structural validity of the Personality Characteristics Inventory (PCI), a self-reported instrument of the Big Five factors. Only one EFA investigated the validity of PCI (Gomes, 2012). We re-analyzed the data of this study (716 elementary and middle school students from a Brazilian school). The solution of eight correlated factors was investigated through CFA and ESEM (exploratory structural equation modeling). The model was found to be unacceptable through CFA, showing good fit through ESEM. Marsh's assertion correctly predicted the results. Structural validity of PCI could only be concluded through ESEM.
journaltitleRevista Avaliação Psicológica
shortjournalAval. Psicol.
titleComparing the ESEM and CFA Approaches to Analyze the Big Five Factors
:_library3N2N7Cet al. - 2023 - Investigating the Applicability of Self-Assessment.pdf

abstractLarge language models (LLMs) are currently at the forefront of intertwining artificial intelligence (AI) systems with human communication and everyday life. Thus, aligning them with human values is of great importance. However, given the steady increase in reasoning abilities, future LLMs are under suspicion of becoming able to deceive human operators and utilizing this ability to bypass monitoring efforts. As a prerequisite to this, LLMs need to possess a conceptual understanding of deception strategies. This study reveals that such strategies emerged in state-of-the-art LLMs, such as GPT-4, but were non-existent in earlier LLMs. We conduct a series of experiments showing that stateof-the-art LLMs are able to understand and induce false beliefs in other agents, that their performance in complex deception scenarios can be amplified utilizing chain-of-thought reasoning, and that eliciting Machiavellianism in LLMs can alter their propensity to deceive. In sum, revealing hitherto unknown machine behavior in LLMs, our study contributes to the nascent field of machine psychology.
titleDeception Abilities Emerged in Large Language Models

Andover, Hampshire

editionEighth edition
titleMultivariate Data Analysis
Multivariate analysis

abstractA two-step process is commonly used to evaluate data–model fit of latent variable path models, the first step addressing the measurement portion of the model and the second addressing the structural portion of the model. Unfortunately, even if the fit of the measurement portion of the model is perfect, the ability to assess the fit within the structural portion is affected by the quality of the factor–variable relations within the measurement model. The result is that models with poorer quality measurement appear to have better data–model fit, whereas models with better quality measurement appear to have worse data–model fit. The current article illustrates this phenomenon across different classes of fit indices, discusses related structural assessment problems resulting from issues of measurement quality, and endorses a supplemental modeling step evaluating the structural portion of the model in isolation from the measurement model.
issn0013-1644, 1552-3888
journaltitleEducational and Psychological Measurement
shortjournalEducational and Psychological Measurement
titleThe Reliability Paradox in Assessing Structural Relations Within Covariance Structure Models
onald’s omega (ω). Among other reasons, that the computation of ω is not available as an option in many popular statistics programs and requires items loadings from a confirmatory factor analysis (CFA) have probably hindered more widespread adoption. After a bit of discussion of α versus ω, we illustrate the computation of ω using two structural equation modeling programs (Mplus and AMOS) and the MBESS package for R. We then describe a macro for SPSS and SAS (OMEGA) that calculates ω in two ways without relying on the estimation of loadings or error variances using CFA. We show that it produces estimates of ω that are nearly identical to when using CFA-based estimates of item loadings and error variances. We also discuss the use of the OMEGA macro for certain forms of item analysis and brief form construction based on the removal of items from a longer scale.
journaltitleCommunication Methods and Measures
titleUse Omega Rather than Cronbach’s Alpha for Estimating Reliability. But…
:_libraryHHRAHQ7et al. - 2023 - Homophily in An Artificial Social Network of Agent.pdf

abstractNewly-developed large language models (LLM)—because of how they are trained and designed —are implicit computational models of humans—a homo silicus. LLMs can be used like economists use homo economicus: they can be given endowments, information, preferences, and so on, and then their behavior can be explored in scenarios via simulation. Experiments using this approach, derived from Charness and Rabin (2002), Kahneman, Knetsch and Thaler (1986), and Samuelson and Zeckhauser (1988) show qualitatively similar results to the original, but it is also easy to try variations for fresh insights. LLMs could allow researchers to pilot studies via simulation first, searching for novel social science insights to test in the real world.
titleLarge Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?

abstractLarge language models (LLMs) have shown remarkable promise in simulating human language use and behavior. In this study, we delve into the intersection of persona variables and the capability of LLMs to simulate different perspectives. We find that persona variables can explain $<$10% variance in annotations in existing subjective NLP datasets. Nonetheless, incorporating them via prompting in LLMs provides modest improvement. Persona prompting is most effective on data samples where disagreements among annotators are frequent yet confined to a limited range. A linear correlation exists: the more persona variables influence human annotations, the better LLMs predictions are using persona prompting. However, when the utility of persona variables is low (i.e., explaining $<$10% of human annotations), persona prompting has little effect. Most subjective NLP datasets fall into this category, casting doubt on simulating diverse perspectives in the current NLP landscape.
titleQuantifying the Persona Effect in LLM Simulations

abstractRecent research has extended beyond assessing the performance of Large Language Models (LLMs) to examining their characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics. The administration of personality tests to LLMs has emerged as a noteworthy area in this context. However, the suitability of employing psychological scales, initially devised for humans, on LLMs is a matter of ongoing debate. Our study aims to determine the reliability of applying personality assessments to LLMs, explicitly investigating whether LLMs demonstrate consistent personality traits. Analyzing responses under 2,500 settings reveals that gpt-3.5-turbo shows consistency in responses to the Big Five Inventory, indicating a high degree of reliability. Furthermore, our research explores the potential of gpt-3.5-turbo to emulate diverse personalities and represent various groups—a capability increasingly sought after in social sciences for substituting human participants with LLMs to reduce costs. Our findings reveal that LLMs have the potential to represent different personalities with specific prompt instructions. By shedding light on the personalization of LLMs, our study endeavors to pave the way for future explorations in this field.
titleRevisiting the Reliability of Psychological Scales on Large Language Models
:_library3Ret al. - 2023 - Who is ChatGPT Benchmarking LLMs' Psychological P.pdf

abstractDespite the many use cases for large language models (LLMs) in the design of chatbots in various industries and the research showing the importance of personalizing chatbots to cater to different personality traits, little work has been done to evaluate whether the behaviors of personalized LLMs can reflect certain personality traits accurately and consistently. We consider studying the behavior of LLM-based simulated agents which refer to as LLM personas and present a case study with GPT-3.5 (text-davinci-003) to investigate whether LLMs can generate content with consistent, personalized traits when assigned Big Five personality types and gender roles. We created 320 LLM personas (5 females and 5 males for each of the 32 Big Five personality types) and prompted them to complete the classic 44-item Big Five Inventory (BFI) and then write an 800-word story about their childhood. Results showed that LLM personas' self-reported BFI scores are consistent with their assigned personality types, with large effect sizes found on all five traits. Moreover, significant correlations were found between assigned personality types and some Linguistic Inquiry and Word Count (LIWC) psycholinguistic features of their writings. For instance, extroversion is associated with pro-social and active words, and neuroticism is associated with words related to negative emotions and mental health. Besides, we only found significant differences in using technological and cultural words in writing between LLM-generated female and male personas. This work provides a first step for further research on personalized LLMs and their applications in Human-AI conversation.
titlePersonaLLM: Investigating the Ability of GPT-3.5 to Express Personality Traits and Gender Differences
:_libraryand Srivastava - 1999 - The Big-Five trait taxonomy History, measurement,.pdf

abstractBeing able to predict people's opinions on issues and behaviors in realistic scenarios can be helpful in various domains, such as politics and marketing. However, conducting large-scale surveys like the European Social Survey to solicit people's opinions on individual issues can incur prohibitive costs. Leveraging prior research showing influence of core human values on individual decisions and actions, we propose to use value-injected large language models (LLM) to predict opinions and behaviors. To this end, we present Value Injection Method (VIM), a collection of two methods – argument generation and question answering – designed to inject targeted value distributions into LLMs via fine-tuning. We then conduct a series of experiments on four tasks to test the effectiveness of VIM and the possibility of using value-injected LLMs to predict opinions and behaviors of people. We find that LLMs value-injected with variations of VIM substantially outperform the baselines. Also, the results suggest that opinions and behaviors can be better predicted using value-injected LLMs than the baseline approaches.
shorttitleFrom Values to Opinions
titleFrom Values to Opinions: Predicting Human Behaviors and Stances Using Value-Injected Large Language Models

abstractTechnology for open-ended language generation, a key application of artificial intelligence, has advanced to a great extent in recent years. Large-scale language models, which are trained on large corpora of text, are being used in a wide range of applications everywhere, from virtual assistants to conversational bots. While these language models output fluent text, existing research shows that these models can and do capture human biases. Many of these biases, especially those that could potentially cause harm, are being well-investigated. On the other hand, studies that infer and change human personality traits inherited by these models have been scarce or non-existent. Our work seeks to address this gap by exploring the personality traits of several large-scale language models designed for openended text generation and the datasets used for training them. We build on the popular Big Five factors and develop robust methods that quantify the personality traits of these models and their underlying datasets. In particular, we trigger the models with a questionnaire designed for personality assessment and subsequently classify the text responses into quantifiable traits using a Zero-shot classifier. Our estimation scheme sheds light on an important anthropomorphic element found in such AI models and can help stakeholders decide how they should be applied as well as how society could perceive them. Additionally, we examined approaches to alter these personalities, adding to our understanding of how AI models can be adapted to specific contexts.
titleEstimating the Personality of White-Box Language Models

issn1931-390X, 1931-3896
journaltitlePsychology of Aesthetics, Creativity, and the Arts
shortjournalPsychology of Aesthetics, Creativity, and the Arts
shorttitleIt Doesn't Hurt to ask…But Sometimes It Hurts to Believe
titleIt Doesn't Hurt to ask…But Sometimes It Hurts to Believe: Polish Students' Creative Self-Efficacy and Its Predictors.

Kosinski - 2023 - Theory of Mind Might Have Spontaneously Emerged in.pdf;D

abstractThis paper presents a method for building a personalized open-domain dialogue system to address the WWH (WHAT, WHEN, and HOW) problem for natural response generation in a commercial setting, where personalized dialogue responses are heavily interleaved with casual response turns. The proposed approach involves weighted dataset blending, negative persona information augmentation methods, and the design of personalized conversation datasets to address the challenges of WWH in personalized, open-domain dialogue systems. Our work effectively balances dialogue fluency and tendency to ground, while also introducing a response-type label to improve the controllability and explainability of the grounded responses. The combination of these methods leads to more fluent conversations, as evidenced by subjective human evaluations as well as objective evaluations.
shorttitleWHAT, WHEN, and HOW to Ground
titleWHAT, WHEN, and HOW to Ground: Designing User Persona-Aware Conversational Agents for Engaging Dialogue
esigning User Pers.pdf

abstractThe advent of the Web has brought about a paradigm shift in traditional economics, particularly in the digital economy era, enabling the precise recording and analysis of individual economic behavior. This has led to a growing emphasis on data-driven modeling in macroeconomics. In macroeconomic research, Agent-based modeling (ABM) emerged as an alternative, evolving through rulebased agents, machine learning-enhanced decision-making, and, more recently, advanced AI agents. However, the existing works are suffering from three main challenges when endowing agents with human-like decision-making, including agent heterogeneity, the influence of macroeconomic trends, and multifaceted economic factors. Large language models (LLMs) have recently gained prominence in offering autonomous human-like characteristics. Therefore, leveraging LLMs in macroeconomic simulation presents an opportunity to overcome traditional limitations. In this work, we take an early step in introducing a novel approach that leverages LLMs in macroeconomic simulation. We design prompt-engineeringdriven LLM agents to exhibit human-like decision-making and adaptability in the economic environment, with the abilities of perception, reflection, and decisionmaking to address the abovementioned challenges. Simulation experiments on macroeconomic activities show that LLM-empowered agents can make realistic work and consumption decisions and emerge more reasonable macroeconomic phenomena than existing rule-based or AI agents. Our work demonstrates the promising potential to simulate macroeconomics based on LLM and its humanlike characteristics.
titleLarge Language Model-Empowered Agents for Simulating Macroeconomic Activities
riven Social Bots in Online Social Networks
espite this, their capacities to coordinate within task-oriented social contexts are under-explored. Such capabilities are crucial if LLMs are to effectively mimic human-like social behavior and produce meaningful results. To bridge this gap, we introduce collaborative generative agents, endowing LLM-based Agents with consistent behavior patterns and task-solving abilities. We situate these agents in a simulated job fair environment as a case study to scrutinize their coordination skills. We propose a novel framework that equips collaborative generative agents with human-like reasoning abilities and specialized skills. Our evaluation demonstrates that these agents show promising performance. However, we also uncover limitations that hinder their effectiveness in more complex coordination tasks. Our work provides valuable insights into the role and evolution of LLMs in task-oriented social simulations.
titleMetaAgents: Simulating Interactions of Human Behaviors for LLM-based Task-oriented Coordination via Collaborative Generative Agents
:_library7̋HAXWBYet al. - 2010 - A new look at the big five factor structure throug.pdf

abstractExploratory factor analysis (EFA) and confirmatory factor analysis (CFA), path analysis, and structural equation modeling (SEM) have long histories in clinical research. Although CFA has largely superseded EFA, CFAs of multidimensional constructs typically fail to meet standards of good measurement: goodness of fit, measurement invariance, lack of differential item functioning, and well-differentiated factors in support of discriminant validity. Part of the problem is undue reliance on overly restrictive CFAs in which each item loads on only one factor. Exploratory SEM (ESEM), an overarching integration of the best aspects of CFA/SEM and traditional EFA, provides confirmatory tests of a priori factor structures, relations between latent factors and multigroup/multioccasion tests of full (mean structure) measurement invariance. It incorporates all combinations of CFA factors, ESEM factors, covariates, grouping/multiple-indicator multiple-cause (MIMIC) variables, latent growth, and complex structures that typically have required CFA/SEM. ESEM has broad applicability to clinical studies that are not appropriately addressed either by traditional EFA or CFA/SEM.
issn1548-5943, 1548-5951
journaltitleAnnual Review of Clinical Psychology
shortjournalAnnu. Rev. Clin. Psychol.
shorttitleExploratory Structural Equation Modeling
titleExploratory Structural Equation Modeling: An Integration of the Best Features of Exploratory and Confirmatory Factor Analysis

New York

Psychology Press

shorttitleTest Theory
titleTest Theory: A Unified Treatment

issn0022-3891, 1532-7752
journaltitleJournal of Personality Assessment
shortjournalJournal of Personality Assessment
titleThe Thorny Relation Between Measurement Quality and Fit Index Cutoffs in Latent Variable Models
5et al. - 2018 - The Thorny Relation Between Measurement Quality an.pdf

abstractIt is well established that the χ2 test is influenced by sample size and will lead to over rejection of models tested using large sample sizes. In this paper it is shown that the population parameter values of a model can also influence the χ2 and lead to erroneous decisions about model acceptance/rejection. It is concluded, based on the examination of hypothetical population factor analytic models, that incremental fit indices offer a useful source of information for the analyst to assist in the interpretation of the χ2 test.
journaltitlePersonality and Individual Differences
seriesSpecial Issue on Structural Equation Modeling
shortjournalPersonality and Individual Differences
titleA Time and a Place for Incremental Fit Indices

abstractLanguage models such as GPT-3 have caused a furore in the research community. Some studies found that GPT-3 has some creative abilities and makes mistakes that are on par with human behaviour. This paper answers a related question: Who is GPT-3? We administered two validated measurement tools to GPT-3 to assess its personality, the values it holds and its self-reported demographics. Our results show that GPT-3 scores similarly to human samples in terms of personality and when provided with a model response memory - in terms of the values it holds. We provide the first evidence of psychological assessment of the GPT-3 model and thereby add to our understanding of this language model. We close with suggestions for future research that moves social science closer to language models and vice versa.
shorttitleWho Is GPT-3?
titleWho Is GPT-3? An Exploration of Personality, Values and Demographics
:_library2NF7QB- Some clarifications and recommendations on fit ind.pdf

New York

edition2. ed
seriesMcGraw-Hill Series in Psychology
titlePsychometric Theory

abstractWe report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs. While less capable than humans in many real-world scenarios, GPT-4 exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers. GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results in improved performance on measures of factuality and adherence to desired behavior. A core component of this project was developing infrastructure and optimization methods that behave predictably across a wide range of scales. This allowed us to accurately predict some aspects of GPT-4's performance based on models trained with no more than 1/1,000th the compute of GPT-4.
titleGPT-4 Technical Report
023 - GPT-4 Technical Report.pdf;D

abstractBelievable proxies of human behavior can empower interactive applications ranging from immersive environments to rehearsal spaces for interpersonal communication to prototyping tools. In this paper, we introduce generative agents–computational software agents that simulate believable human behavior. Generative agents wake up, cook breakfast, and head to work; artists paint, while authors write; they form opinions, notice each other, and initiate conversations; they remember and reflect on days past as they plan the next day. To enable generative agents, we describe an architecture that extends a large language model to store a complete record of the agent's experiences using natural language, synthesize those memories over time into higher-level reflections, and retrieve them dynamically to plan behavior. We instantiate generative agents to populate an interactive sandbox environment inspired by The Sims, where end users can interact with a small town of twenty five agents using natural language. In an evaluation, these generative agents produce believable individual and emergent social behaviors: for example, starting with only a single user-specified notion that one agent wants to throw a Valentine's Day party, the agents autonomously spread invitations to the party over the next two days, make new acquaintances, ask each other out on dates to the party, and coordinate to show up for the party together at the right time. We demonstrate through ablation that the components of our agent architecture–observation, planning, and reflection–each contribute critically to the believability of agent behavior. By fusing large language models with computational, interactive agents, this work introduces architectural and interaction patterns for enabling believable simulations of human behavior.
shorttitleGenerative Agents
titleGenerative Agents: Interactive Simulacra of Human Behavior
FS98Net al. - 2023 - Generative Agents Interactive Simulacra of Human .pdf
ine Learning

abstractWe test whether large language models (LLMs) can be used to simulate human participants in social-science studies. To do this, we ran replications of 14 studies from the Many Labs 2 replication project with OpenAI’s text-davinci-003 model, colloquially known as GPT-3.5. Based on our pre-registered analyses, we find that among the eight studies we could analyse, our GPT sample replicated 37.5% of the original results and 37.5% of the Many Labs 2 results. However, we were unable to analyse the remaining six studies due to an unexpected phenomenon we call the “correct answer” effect. Different runs of GPT-3.5 answered nuanced questions probing political orientation, economic preference, judgement, and moral philosophy with zero or near-zero variation in responses: with the supposedly “correct answer.” In one exploratory follow-up study, we found that a “correct answer” was robust to changing the demographic details that precede the prompt. In another, we found that most but not all “correct answers” were robust to changing the order of answer choices. One of our most striking findings occurred in our replication of the Moral Foundations Theory survey results, where we found GPT-3.5 identifying as a political conservative in 99.6% of the cases, and as a liberal in 99.3% of the cases in the reverse-order condition. However, both self-reported ‘GPT conservatives’ and ‘GPT liberals’ showed right-leaning moral foundations. Our results cast doubts on the validity of using LLMs as a general replacement for human participants in the social sciences. Our results also raise concerns that a hypothetical AI-led future may be subject to a diminished diversity of thought.
journaltitleBehavior Research Methods
shortjournalBehav Res
titleDiminished Diversity-of-Thought in a Standard Large Language Model
iminished diversity-of-thought in a standard larg.pdf
ic effects,Diversity of thought,GPT-3.5,Large language models,Many Labs 2,Order effects,Psychology,Replication,Social science

shorttitleAI Psychometrics
titleAI Psychometrics: Assessing the Psychological Profiles of Large Language Models through Psychometric Inventories
ttps://files.osf.io/v1/resources/jv5dt/providers/osfstorage/63a49c69ca8be7026be5b265?action=download   direct   version=3

abstractRecent investigations indicate that personality traits are unevenly distributed geographically, with some traits being more prevalent in certain places than in others. The geographical distributions of personality traits are associated with a range of important political, economic, social, and health outcomes. The majority of research on this subject has focused on the geographical distributions and macro-level correlates of personality across nations or regions of the United States. The aim of the present investigation was to replicate and extend that past work by examining regional personality differences in Great Britain. Using a sample of nearly 400,000 British residents, we mapped the geographical distributions of the Big Five Personality traits across 380 Local Authority Districts and examined the associations with important political, economic, social, and health outcomes. The results revealed distinct geographical clusters, with neighboring regions displaying similar personality characteristics, and robust associations with the macro-level outcome variables. Overall, the patterns of results were similar to findings from past research.
journaltitlePLoS ONE
shortjournalPLoS One
titleRegional Personality Differences in Great Britain
015 - Regional Personality Differences in Great Britain.pdf

In Review

abstractPrevious research on emergence in large language models shows these display apparent human-like abilities and psychological latent traits. However, results are partly contradicting in expression and magnitude of these latent traits, yet agree on the worrisome tendencies to score high on the Dark Triad of narcissism, psychopathy, and Machiavellianism, which, together with a track record of derailments, demands more rigorous research on safety of these models. We provided a state of the art language model with the same personality questionnaire in nine languages, and performed Bayesian analysis of Gaussian Mixture Model, finding evidence for a deeper-rooted issue. Our results suggest both interlingual and intralingual instabilities, which indicate that current language models do not develop a consistent core personality. This can lead to unsafe behaviour of artificial intelligence systems that are based on these foundation models, and are increasingly integrated in human life. We subsequently discuss the shortcomings of modern psychometrics, abstract it, and provide a framework for its species-neutral, substrate-free formulation.
shorttitleDo GPT Language Models Suffer From Split Personality Disorder?
titleDo GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics
o GPT Language Models Suffer From Split Personali.pdf

Milton Park, Abingdon, Oxon ; New York, NY

abstract"This popular text introduces the reader to all aspects of psychometric assessment, including its history, how to construct and administer traditional tests and the latest techniques for psychometric assessment online. Rust, Kosinski and Stillwell begin with a comprehensive introduction to the increased sophistication in psychometric methods and regulation that took place during the 20th century, including the many benefits to governments, businesses and customers. In this new edition, the authors explore the increasing influence of the internet wherein everything we do online is available for psychometric analysis, often by AI systems operating at scale and in real time. The intended and unintended consequences of this paradigm shift are examined in detail, and key controversies, such as privacy and the psychographic microtargeting of online messages, are addressed. Furthermore, this new edition includes brand new chapters on Item Response Theory and Computer-Adaptive Testing, Test Equating and Differential Item Functioning, and Using R in psychometric settings. Modern Psychometrics combines an up to date scientific approach with full consideration of the political and ethical issues involved in the implementation of psychometrics testing in today's society. It will be invaluable to both undergraduate and postgraduate students as well as practitioners who are seeking an introduction to modern psychometric methods"–
editionFourth edition
shorttitleModern Psychometrics
titleModern Psychometrics: The Science of Psychological Assessment

abstractLanguage models (LMs) are increasingly being used in open-ended contexts, where the opinions reflected by LMs in response to subjective queries can have a profound impact, both on user satisfaction, as well as shaping the views of society at large. In this work, we put forth a quantitative framework to investigate the opinions reflected by LMs – by leveraging high-quality public opinion polls and their associated human responses. Using this framework, we create OpinionQA, a new dataset for evaluating the alignment of LM opinions with those of 60 US demographic groups over topics ranging from abortion to automation. Across topics, we find substantial misalignment between the views reflected by current LMs and those of US demographic groups: on par with the Democrat-Republican divide on climate change. Notably, this misalignment persists even after explicitly steering the LMs towards particular demographic groups. Our analysis not only confirms prior observations about the left-leaning tendencies of some human feedback-tuned LMs, but also surfaces groups whose opinions are poorly reflected by current LMs (e.g., 65+ and widowed individuals). Our code and data are available at https://github.com/tatsu-lab/opinions_qa.
titleWhose Opinions Do Language Models Reflect?
o Language Models Reflect.pdf
ine Learning

Abu Dhabi, United Arab Emirates

Association for Computational Linguistics

eventtitleProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
journaltitleProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
shorttitleNeural Theory-of-Mind?
titleNeural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs
7VN93et al. - 2022 - Neural Theory-of-Mind On the Limits of Social Int.pdf

abstractThe advent of large language models (LLMs) has revolutionized natural language processing, enabling the generation of coherent and contextually relevant human-like text. As LLMs increasingly power conversational agents used by the general public world-wide, the synthetic personality embedded in these models, by virtue of training on large amounts of human data, is becoming increasingly important. Since personality is a key factor determining the effectiveness of communication, we present a comprehensive method for administering and validating personality tests on widely-used LLMs, as well as for shaping personality in the generated text of such LLMs. Applying this method, we found: 1) personality measurements in the outputs of some LLMs under specific prompting configurations are reliable and valid; 2) evidence of reliability and validity of synthetic LLM personality is stronger for larger and instruction fine-tuned models; and 3) personality in LLM outputs can be shaped along desired dimensions to mimic specific human personality profiles. We discuss application and ethical implications of the measurement and shaping method, in particular regarding responsible AI.
titlePersonality Traits in Large Language Models
:_libraryŁ2N2LBK5et al. - 2023 - Clever Hans or Neural Theory of Mind Stress Testi.pdf

abstractThis discussion paper argues that both the use of Cronbach’s alpha as a reliability estimate and as a measure of internal consistency suffer from major problems. First, alpha always has a value, which cannot be equal to the test score’s reliability given the interitem covariance matrix and the usual assumptions about measurement error. Second, in practice, alpha is used more often as a measure of the test’s internal consistency than as an estimate of reliability. However, it can be shown easily that alpha is unrelated to the internal structure of the test. It is further discussed that statistics based on a single test administration do not convey much information about the accuracy of individuals’ test performance. The paper ends with a list of conclusions about the usefulness of alpha.
titleOn the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha
iscussing Lower Bounds and Correlated Errors

issn1939-1315, 0022-3514
journaltitleJournal of Personality and Social Psychology
shortjournalJournal of Personality and Social Psychology
shorttitleAge Differences in Personality Traits from 10 to 65
titleAge Differences in Personality Traits from 10 to 65: Big Five Domains and Facets in a Large Cross-Sectional Sample.

abstractAI large language models have (co-)produced amazing written works from newspaper articles to novels and poetry. These works meet the standards of the standard definition of creativity: being original and useful, and sometimes even the additional element of surprise. But can a large language model designed to predict the next text fragment provide creative, out-of-the-box, responses that still solve the problem at hand? We put Open AI's generative natural language model, GPT-3, to the test. Can it provide creative solutions to one of the most commonly used tests in creativity research? We assessed GPT-3's creativity on Guilford's Alternative Uses Test and compared its performance to previously collected human responses on expert ratings of originality, usefulness and surprise of responses, flexibility of each set of ideas as well as an automated method to measure creativity based on the semantic distance between a response and the AUT object in question. Our results show that – on the whole – humans currently outperform GPT-3 when it comes to creative output. But, we believe it is only a matter of time before GPT-3 catches up on this particular task. We discuss what this work reveals about human and AI creativity, creativity testing and our definition of creativity.
titlePutting GPT-3's Creativity to the (Alternative Uses) Test
eveloping and Reporting Research Instruments in Science Education
- 2018 - The Use of Cronbach’s Alpha When Developing and Re.pdf
methods,Scales,Statistics,Test development

abstractSocial media is often criticized for amplifying toxic discourse and discouraging constructive conversations. But designing social media platforms to promote better conversations is inherently challenging. This paper asks whether simulating social media through a combination of Large Language Models (LLM) and Agent-Based Modeling can help researchers study how different news feed algorithms shape the quality of online conversations. We create realistic personas using data from the American National Election Study to populate simulated social media platforms. Next, we prompt the agents to read and share news articles — and like or comment upon each other’s messages — within three platforms that use different news feed algorithms. In the first platform, users see the most liked and commented posts from users whom they follow. In the second, they see posts from all users —even those outside their own network. The third platform employs a novel “bridging” algorithm that highlights posts that are liked by people with opposing political views. We find this bridging algorithm promotes more constructive, non-toxic, conversation across political divides than the other two models. Though further research is needed to evaluate these findings, we argue that LLMs hold considerable potential to improve simulation research on social media and many other complex social settings.
titleSimulating Social Media Using Large Language Models to Evaluate Alternative News Feed Algorithms
R939Cet al. - 2023 - Simulating Social Media Using Large Language Model.pdf

abstract$<$p$>$The Cronbach's alpha is the most widely used method for estimating internal consistency reliability. This procedure has proved very resistant to the passage of time, even if its limitations are well documented and although there are better options as omega coefficient or the different versions of glb, with obvious advantages especially for applied research in which the ítems differ in quality or have skewed distributions. In this paper, using Monte Carlo simulation, the performance of these reliability coefficients under a one-dimensional model is evaluated in terms of skewness and no tau-equivalence. The results show that omega coefficient is always better choice than alpha and in the presence of skew items is preferable to use omega and glb coefficients even in small samples.$<$/p$>$
journaltitleFrontiers in Psychology
shortjournalFront. Psychol.
shorttitleBest Alternatives to Cronbach's Alpha Reliability in Realistic Conditions
titleBest Alternatives to Cronbach's Alpha Reliability in Realistic Conditions: Congeneric and Asymmetrical Measurements

abstractTo what degree should we ascribe cognitive capacities to Large Language Models (LLMs), such as the ability to reason about intentions and beliefs known as Theory of Mind (ToM)? Here we add to this emerging debate by (i) testing 11 base- and instruction-tuned LLMs on capabilities relevant to ToM beyond the dominant false-belief paradigm, including nonliteral language usage and recursive intentionality; (ii) using newly rewritten versions of standardized tests to gauge LLMs’ robustness; (iii) prompting and scoring for open besides closed questions; and (iv) benchmarking LLM performance against that of children aged 7-10 on the same tasks. We find that instructiontuned LLMs from the GPT family outperform other models, and often also children. BaseLLMs are mostly unable to solve ToM tasks, even with specialized prompting. We suggest that the interlinked evolution and development of language and ToM may help explain what instruction-tuning adds: rewarding cooperative communication that takes into account interlocutor and context. We conclude by arguing for a nuanced perspective on ToM in LLMs.
shorttitleTheory of Mind in Large Language Models
titleTheory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art Models vs. Children Aged 7-10 on Advanced Tests
XNWDuijn et al. - 2023 - Theory of Mind in Large Language Models Examining.pdf

abstractCronbach’s alpha estimates the internal consistency of responses in multi-item bipolar scales. This article examined three research questions (RQ): (1) To what extent do inconsistencies exist in data (e.g., responses of −2 −2 2 2)? (2) Does the number of scale items influence the amount of inconsistency? (3) Does Cronbach’s alpha mask inconsistencies? Data were obtained from 29 research projects (n = 10,616). Each survey had place attachment questions comprising two concepts: place identity and place dependence. Respondents were classified as consistent or inconsistent based on their responses to the place attachment questions. Results demonstrated that: (a) inconsistent response patterns existed in the data (RQ1), (b) number of scale items influenced amount of inconsistency (RQ2), and (c) alpha masked these inconsistencies (RQ3). Discussion focused on implications of these findings.
issn0149-0400, 1521-0588
journaltitleLeisure Sciences
shortjournalLeisure Sciences
titleRethinking Internal Consistency in Cronbach's Alpha
:_library3et al. - 2023 - A Survey on Large Language Model based Autonomous .pdf

abstractUser behavior analysis is crucial in human-centered AI applications. In this field, the collection of sufficient and high-quality user behavior data has always been a fundamental yet challenging problem. An intuitive idea to address this problem is automatically simulating the user behaviors. However, due to the subjective and complex nature of human cognitive processes, reliably simulating the user behavior is difficult. Recently, large language models (LLM) have obtained remarkable successes, showing great potential to achieve human-like intelligence. We argue that these models present significant opportunities for reliable user simulation, and have the potential to revolutionize traditional study paradigms in user behavior analysis. In this paper, we take recommender system as an example to explore the potential of using LLM for user simulation. Specifically, we regard each user as an LLM-based autonomous agent, and let different agents freely communicate, behave and evolve in a virtual simulator called RecAgent. For comprehensively simulation, we not only consider the behaviors within the recommender system (e.g., item browsing and clicking), but also accounts for external influential factors, such as, friend chatting and social advertisement. Our simulator contains at most 1000 agents, and each agent is composed of a profiling module, a memory module and an action module, enabling it to behave consistently, reasonably and reliably. In addition, to more flexibly operate our simulator, we also design two global functions including real-human playing and system intervention. To evaluate the effectiveness of our simulator, we conduct extensive experiments from both agent and system perspectives. In order to advance this direction, we have released our project at https://github.com/RUC-GSAI/YuLan-Rec.
shorttitleWhen Large Language Model Based Agent Meets User Behavior Analysis
titleWhen Large Language Model Based Agent Meets User Behavior Analysis: A Novel User Simulation Paradigm

American Psychological Association

journaltitleJournal of personality and social psychology
shorttitleDevelopment and Validation of Brief Measures of Positive and Negative Affect
titleDevelopment and Validation of Brief Measures of Positive and Negative Affect: The PANAS Scales.

issn0033-3123, 1860-0980
shorttitleLower Bounds for the Reliability of the Total Score on a Test Composed of Non-Homogeneous Items
titleLower Bounds for the Reliability of the Total Score on a Test Composed of Non-Homogeneous Items: II: A Search Procedure to Locate the Greatest Lower Bound

abstractChit-chat models are known to have several problems: they lack specificity, do not display a consistent personality and are often not very captivating. In this work we present the task of making chit-chat more engaging by conditioning on profile information. We collect data and train models to (i) condition on their given profile information; and (ii) information about the person they are talking to, resulting in improved dialogues, as measured by next utterance prediction. Since (ii) is initially unknown, our model is trained to engage its partner with personal topics, and we show the resulting dialogue can be used to predict profile information about the interlocutors.
shorttitlePersonalizing Dialogue Agents
titlePersonalizing Dialogue Agents: I Have a Dog, Do You Have Pets Too?
A2A4DKWet al. - 2018 - Personalizing Dialogue Agents I have a dog, do yo.pdf

journaltitleInternational Conference on Machine Learning
titleUsing Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

abstractIn this study, we investigate the reliability of Large Language Models (LLMs) in professing human-like personality traits through responses to personality questionnaires. Our goal is to evaluate the consistency between LLMs’ professed personality inclinations and their actual "behavior", examining the extent to which these models can emulate human-like personality patterns. Through a comprehensive analysis of LLM outputs against established human benchmarks, we seek to understand the cognition-action divergence in LLMs and propose hypotheses for the observed results based on psychological theories and metrics.
shorttitleIs Cognition and Action Consistent or Not
titleIs Cognition and Action Consistent or Not: Investigating Large Language Model's Personality

abstractWe propose and explore the possibility that language models can be studied as effective proxies for specific human subpopulations in social science research. Practical and research applications of artificial intelligence tools have sometimes been limited by problematic biases (such as racism or sexism), which are often treated as uniform properties of the models. We show that the “algorithmic bias” within one such tool—the GPT-3 language model—is instead both fine-grained and demographically correlated, meaning that proper conditioning will cause it to accurately emulate response distributions from a wide variety of human subgroups. We term this property algorithmic fidelity and explore its extent in GPT-3. We create “silicon samples” by conditioning the model on thousands of sociodemographic backstories from real human participants in multiple large surveys conducted in the United States. We then compare the silicon and human samples to demonstrate that the information contained in GPT-3 goes far beyond surface similarity. It is nuanced, multifaceted, and reflects the complex interplay between ideas, attitudes, and sociocultural context that characterize human attitudes. We suggest that language models with sufficient algorithmic fidelity thus constitute a novel and powerful tool to advance understanding of humans and society across a variety of disciplines.
issn1047-1987, 1476-4989
journaltitlePolitical Analysis
shortjournalPolit. Anal.
shorttitleOut of One, Many
titleOut of One, Many: Using Language Models to Simulate Human Samples

Argyle et al. - 2023 - Out of One, Many Using Language Models to Simulat.pdf

abstractWe study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3’s decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3’s behavior is impressive: It solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multiarmed bandit task, and shows signatures of model-based reinforcement learning. Yet, we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. Taken together, these results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.
issn0027-8424, 1091-6490
journaltitleProceedings of the National Academy of Sciences
shortjournalProc. Natl. Acad. Sci. U.S.A.
titleUsing Cognitive Psychology to Understand GPT-3
:_libraryand Perry - 1992 - The aggression questionnaire..pdf

London, UK

shorttitleStructural Equation Modeling with AMOS
titleStructural Equation Modeling with AMOS: Basic Concepts, Applications, and Programming

American Psychological Association

abstract"This paper advocates a validational process utilizing a matrix of intercorrelations among tests representing at least two traits, each measured by at least two methods. Measures of the same trait should correlate higher with each other than they do with measures of different traits involving separate methods. Ideally, these validity values should also be higher than the correlations among different traits measure by the same method." Examples from the literature are described as well as problems in the application of the technique. 36 refs. (PsycINFO Database Record (c) 2016 APA, all rights reserved)
journaltitlePsychological Bulletin
titleConvergent and Discriminant Validation by the Multitrait-Multimethod Matrix

Statistical Validity,Test Validity

abstractPsychology research has long explored aspects of human personality such as extroversion, agreeableness and emotional stability. Categorizations like the ‘Big Five’ personality traits are commonly used to assess and diagnose personality types. In this work, we explore the question of whether the perceived personality in language models is exhibited consistently in their language generation. For example, is a language model such as GPT2 likely to respond in a consistent way if asked to go out to a party? We also investigate whether such personality traits can be controlled. We show that when provided different types of contexts (such as personality descriptions, or answers to diagnostic questions about personality traits), language models such as BERT and GPT2 can consistently identify and reflect personality markers in those contexts. This behavior illustrates an ability to be manipulated in a highly predictable way, and frames them as tools for identifying personality traits and controlling personas in applications such as dialog systems. We also contribute a crowd-sourced data-set of personality descriptions of human subjects paired with their ‘Big Five’ personality assessment data, and a data-set of personality descriptions collated from Reddit.
titleIdentifying and Manipulating the Personality Traits of Language Models
F87V5ECand Srivastava - 2022 - Identifying and Manipulating the Personality Trait.pdf

issn0033-3123, 1860-0980
titleCoefficient Alpha and the Internal Structure of Tests

abstractLarge language models (LLMs) have shown incredible capabilities and transcended the natural language processing (NLP) community, with adoption throughout many services like healthcare, therapy, education, and customer service. Since users include people with critical information needs like students or patients engaging with chatbots, the safety of these systems is of prime importance. Therefore, a clear understanding of the capabilities and limitations of LLMs is necessary. To this end, we systematically evaluate toxicity in over half a million generations of CHATGPT, a popular dialogue-based LLM. We find that setting the system parameter of CHATGPT by assigning it a persona, say that of the boxer Muhammad Ali, significantly increases the toxicity of generations. Depending on the persona assigned to CHATGPT, its toxicity can increase up to 6×, with outputs engaging in incorrect stereo- types, harmful dialogue, and hurtful opinions. This may be potentially defamatory to the persona and harmful to an unsuspecting user. Furthermore, we find concerning patterns where specific entities (e.g., certain races) are targeted more than others (3× more) irrespective of the assigned persona, that reflect inherent discriminatory biases in the model. We hope that our findings inspire the broader AI community to rethink the efficacy of current safety guardrails and develop better techniques that lead to robust, safe, and trustworthy AI systems.
shorttitleToxicity in ChatGPT
titleToxicity in ChatGPT: Analyzing Persona-assigned Language Models
3HUet al. - 2023 - Toxicity in ChatGPT Analyzing Persona-assigned La.pdf
ine Learning

abstractWith large language models (LLMs) appearing to behave increasingly human-like in text-based interactions, it has become popular to attempt to evaluate various properties of these models using tests originally designed for humans. While reusing existing tests is a resource-efficient way to evaluate LLMs, careful adjustments are usually required to ensure that test results are even valid across human sub-populations. Thus, it is not clear to what extent different tests’ validity generalizes to LLMs. In this work, we provide evidence that LLMs’ responses to personality tests systematically deviate from typical human responses, implying that these results cannot be interpreted in the same way as human test results. Concretely, reverse-coded items (e.g. “I am introverted” vs “I am extraverted”) are often both answered affirmatively by LLMs. In addition, variation across different prompts designed to “steer” LLMs to simulate particular personality types does not follow the clear separation into five independent personality factors from human samples. In light of these results, we believe it is important to pay more attention to tests’ validity for LLMs before drawing strong conclusions about potentially ill-defined concepts like LLMs’ “personality”.
titleDo Personality Tests Generalize to Large Language Models?
orner et al. - 2023 - Do personality tests generalize to Large Language .pdf
ine Learning

abstractSimulation plays a crucial role in addressing various challenges within social science. It offers extensive applications such as state prediction, phenomena explanation, and policy-making support, among others. In this work, we harness the human-like capabilities of large language models (LLMs) in sensing, reasoning, and behaving, and utilize these qualities to construct the S3 system (short for Social network Simulation System). Adhering to the widely employed agent-based simulation paradigm, we employ fine-tuning and prompt engineering techniques to ensure that the agent’s behavior closely emulates that of a genuine human within the social network. Specifically, we simulate three pivotal aspects: emotion, attitude, and interaction behaviors. By endowing the agent in the system with the ability to perceive the informational environment and emulate human actions, we observe the emergence of population-level phenomena, including the propagation of information, attitudes, and emotions. We conduct an evaluation encompassing two levels of simulation, employing real-world social network data. Encouragingly, the results demonstrate promising accuracy. This work represents an initial step in the realm of social network simulation empowered by LLM-based agents. We anticipate that our endeavors will serve as a source of inspiration for the development of simulation systems within, but not limited to, social science.
titleS3: Social-network Simulation System with Large Language Model-Empowered Agents

abstractLiars and truth-tellers often communicate differently. When tasked with writing deceptive text, humans use prior knowledge and experiences to intentionally deceive their target. Automatically generated text via large language models; LLMs, on the other hand, mirror training instances that are most likely written by humans. In the case of content like reviews, automatically generated language is inherently deceptive because the system is not grounded in material world experiences. In this paper, we characterize differences between (a) truthful text written by humans, (b) intentionally deceptive text written by humans, and (c) inherently deceptive text written by state-of-the-art language models (ChatGPT). We examined the expression of thirteen psychologically grounded and fundamental human traits (e.g., personality and empathy) across truthful and deceptive hotel reviews, finding that texts written by humans were more diverse (had more variation) in their expressions of personality than texts written by ChatGPT. Across all human traits we found that truthful and deceptive human language was easier to distinguish from machine generated language. Building on these differences, we trained a classifier using only the thirteen human traits to automatically discriminate between truthful and deceptive language, with a classification AUC of up to 0.966. Thus, despite the fact that large language models mirror text written by genuine (and truthful) humans, their lack of diversity in human traits makes them easier to identify. These results suggest that psychologically grounded human traits offer a robust feature set unaffected by the “human-ness” of LLM language, and further suggest that AI and humans are behaviorally different when communicating about experiences.
title“I Slept Like a Baby”: Using Human Traits To Characterize Deceptive ChatGPT and Human Text

journaltitlePersonality and Individual differences
shorttitleDemographic Variables and Personality
titleDemographic Variables and Personality: The Effects of Gender, Age, Education, and Ethnic/Racial Status on Self-Descriptions of Personality Attributes

abstractMarsh (2007) asserts that no self-reported questionnaire of the Big Five factors shows an acceptable degree of adjustment in confirmatory factor analyses (CFA). Based on CFA results, it can usually be inferred that exploratory factor analysis (EFA) solutions are incorrect, as well as concluded that the instruments evaluated by EFAs do not measure what they intend to measure. The present study investigates this assertion, evaluating the structural validity of the Personality Characteristics Inventory (PCI), a self-reported instrument of the Big Five factors. Only one EFA investigated the validity of PCI (Gomes, 2012). We re-analyzed the data of this study (716 elementary and middle school students from a Brazilian school). The solution of eight correlated factors was investigated through CFA and ESEM (exploratory structural equation modeling). The model was found to be unacceptable through CFA, showing good fit through ESEM. Marsh's assertion correctly predicted the results. Structural validity of PCI could only be concluded through ESEM.
journaltitleRevista Avaliação Psicológica
shortjournalAval. Psicol.
titleComparing the ESEM and CFA Approaches to Analyze the Big Five Factors
:_library3N2N7Cet al. - 2023 - Investigating the Applicability of Self-Assessment.pdf

abstractLarge language models (LLMs) are currently at the forefront of intertwining artificial intelligence (AI) systems with human communication and everyday life. Thus, aligning them with human values is of great importance. However, given the steady increase in reasoning abilities, future LLMs are under suspicion of becoming able to deceive human operators and utilizing this ability to bypass monitoring efforts. As a prerequisite to this, LLMs need to possess a conceptual understanding of deception strategies. This study reveals that such strategies emerged in state-of-the-art LLMs, such as GPT-4, but were non-existent in earlier LLMs. We conduct a series of experiments showing that stateof-the-art LLMs are able to understand and induce false beliefs in other agents, that their performance in complex deception scenarios can be amplified utilizing chain-of-thought reasoning, and that eliciting Machiavellianism in LLMs can alter their propensity to deceive. In sum, revealing hitherto unknown machine behavior in LLMs, our study contributes to the nascent field of machine psychology.
titleDeception Abilities Emerged in Large Language Models

Andover, Hampshire

editionEighth edition
titleMultivariate Data Analysis
Multivariate analysis

abstractA two-step process is commonly used to evaluate data–model fit of latent variable path models, the first step addressing the measurement portion of the model and the second addressing the structural portion of the model. Unfortunately, even if the fit of the measurement portion of the model is perfect, the ability to assess the fit within the structural portion is affected by the quality of the factor–variable relations within the measurement model. The result is that models with poorer quality measurement appear to have better data–model fit, whereas models with better quality measurement appear to have worse data–model fit. The current article illustrates this phenomenon across different classes of fit indices, discusses related structural assessment problems resulting from issues of measurement quality, and endorses a supplemental modeling step evaluating the structural portion of the model in isolation from the measurement model.
issn0013-1644, 1552-3888
journaltitleEducational and Psychological Measurement
shortjournalEducational and Psychological Measurement
titleThe Reliability Paradox in Assessing Structural Relations Within Covariance Structure Models
onald’s omega (ω). Among other reasons, that the computation of ω is not available as an option in many popular statistics programs and requires items loadings from a confirmatory factor analysis (CFA) have probably hindered more widespread adoption. After a bit of discussion of α versus ω, we illustrate the computation of ω using two structural equation modeling programs (Mplus and AMOS) and the MBESS package for R. We then describe a macro for SPSS and SAS (OMEGA) that calculates ω in two ways without relying on the estimation of loadings or error variances using CFA. We show that it produces estimates of ω that are nearly identical to when using CFA-based estimates of item loadings and error variances. We also discuss the use of the OMEGA macro for certain forms of item analysis and brief form construction based on the removal of items from a longer scale.
journaltitleCommunication Methods and Measures
titleUse Omega Rather than Cronbach’s Alpha for Estimating Reliability. But…
:_libraryHHRAHQ7et al. - 2023 - Homophily in An Artificial Social Network of Agent.pdf

abstractNewly-developed large language models (LLM)—because of how they are trained and designed —are implicit computational models of humans—a homo silicus. LLMs can be used like economists use homo economicus: they can be given endowments, information, preferences, and so on, and then their behavior can be explored in scenarios via simulation. Experiments using this approach, derived from Charness and Rabin (2002), Kahneman, Knetsch and Thaler (1986), and Samuelson and Zeckhauser (1988) show qualitatively similar results to the original, but it is also easy to try variations for fresh insights. LLMs could allow researchers to pilot studies via simulation first, searching for novel social science insights to test in the real world.
titleLarge Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?

abstractLarge language models (LLMs) have shown remarkable promise in simulating human language use and behavior. In this study, we delve into the intersection of persona variables and the capability of LLMs to simulate different perspectives. We find that persona variables can explain $<$10% variance in annotations in existing subjective NLP datasets. Nonetheless, incorporating them via prompting in LLMs provides modest improvement. Persona prompting is most effective on data samples where disagreements among annotators are frequent yet confined to a limited range. A linear correlation exists: the more persona variables influence human annotations, the better LLMs predictions are using persona prompting. However, when the utility of persona variables is low (i.e., explaining $<$10% of human annotations), persona prompting has little effect. Most subjective NLP datasets fall into this category, casting doubt on simulating diverse perspectives in the current NLP landscape.
titleQuantifying the Persona Effect in LLM Simulations

abstractRecent research has extended beyond assessing the performance of Large Language Models (LLMs) to examining their characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics. The administration of personality tests to LLMs has emerged as a noteworthy area in this context. However, the suitability of employing psychological scales, initially devised for humans, on LLMs is a matter of ongoing debate. Our study aims to determine the reliability of applying personality assessments to LLMs, explicitly investigating whether LLMs demonstrate consistent personality traits. Analyzing responses under 2,500 settings reveals that gpt-3.5-turbo shows consistency in responses to the Big Five Inventory, indicating a high degree of reliability. Furthermore, our research explores the potential of gpt-3.5-turbo to emulate diverse personalities and represent various groups—a capability increasingly sought after in social sciences for substituting human participants with LLMs to reduce costs. Our findings reveal that LLMs have the potential to represent different personalities with specific prompt instructions. By shedding light on the personalization of LLMs, our study endeavors to pave the way for future explorations in this field.
titleRevisiting the Reliability of Psychological Scales on Large Language Models
:_library3Ret al. - 2023 - Who is ChatGPT Benchmarking LLMs' Psychological P.pdf

abstractDespite the many use cases for large language models (LLMs) in the design of chatbots in various industries and the research showing the importance of personalizing chatbots to cater to different personality traits, little work has been done to evaluate whether the behaviors of personalized LLMs can reflect certain personality traits accurately and consistently. We consider studying the behavior of LLM-based simulated agents which refer to as LLM personas and present a case study with GPT-3.5 (text-davinci-003) to investigate whether LLMs can generate content with consistent, personalized traits when assigned Big Five personality types and gender roles. We created 320 LLM personas (5 females and 5 males for each of the 32 Big Five personality types) and prompted them to complete the classic 44-item Big Five Inventory (BFI) and then write an 800-word story about their childhood. Results showed that LLM personas' self-reported BFI scores are consistent with their assigned personality types, with large effect sizes found on all five traits. Moreover, significant correlations were found between assigned personality types and some Linguistic Inquiry and Word Count (LIWC) psycholinguistic features of their writings. For instance, extroversion is associated with pro-social and active words, and neuroticism is associated with words related to negative emotions and mental health. Besides, we only found significant differences in using technological and cultural words in writing between LLM-generated female and male personas. This work provides a first step for further research on personalized LLMs and their applications in Human-AI conversation.
titlePersonaLLM: Investigating the Ability of GPT-3.5 to Express Personality Traits and Gender Differences
:_libraryand Srivastava - 1999 - The Big-Five trait taxonomy History, measurement,.pdf

abstractBeing able to predict people's opinions on issues and behaviors in realistic scenarios can be helpful in various domains, such as politics and marketing. However, conducting large-scale surveys like the European Social Survey to solicit people's opinions on individual issues can incur prohibitive costs. Leveraging prior research showing influence of core human values on individual decisions and actions, we propose to use value-injected large language models (LLM) to predict opinions and behaviors. To this end, we present Value Injection Method (VIM), a collection of two methods – argument generation and question answering – designed to inject targeted value distributions into LLMs via fine-tuning. We then conduct a series of experiments on four tasks to test the effectiveness of VIM and the possibility of using value-injected LLMs to predict opinions and behaviors of people. We find that LLMs value-injected with variations of VIM substantially outperform the baselines. Also, the results suggest that opinions and behaviors can be better predicted using value-injected LLMs than the baseline approaches.
shorttitleFrom Values to Opinions
titleFrom Values to Opinions: Predicting Human Behaviors and Stances Using Value-Injected Large Language Models

abstractTechnology for open-ended language generation, a key application of artificial intelligence, has advanced to a great extent in recent years. Large-scale language models, which are trained on large corpora of text, are being used in a wide range of applications everywhere, from virtual assistants to conversational bots. While these language models output fluent text, existing research shows that these models can and do capture human biases. Many of these biases, especially those that could potentially cause harm, are being well-investigated. On the other hand, studies that infer and change human personality traits inherited by these models have been scarce or non-existent. Our work seeks to address this gap by exploring the personality traits of several large-scale language models designed for openended text generation and the datasets used for training them. We build on the popular Big Five factors and develop robust methods that quantify the personality traits of these models and their underlying datasets. In particular, we trigger the models with a questionnaire designed for personality assessment and subsequently classify the text responses into quantifiable traits using a Zero-shot classifier. Our estimation scheme sheds light on an important anthropomorphic element found in such AI models and can help stakeholders decide how they should be applied as well as how society could perceive them. Additionally, we examined approaches to alter these personalities, adding to our understanding of how AI models can be adapted to specific contexts.
titleEstimating the Personality of White-Box Language Models

issn1931-390X, 1931-3896
journaltitlePsychology of Aesthetics, Creativity, and the Arts
shortjournalPsychology of Aesthetics, Creativity, and the Arts
shorttitleIt Doesn't Hurt to ask…But Sometimes It Hurts to Believe
titleIt Doesn't Hurt to ask…But Sometimes It Hurts to Believe: Polish Students' Creative Self-Efficacy and Its Predictors.

Kosinski - 2023 - Theory of Mind Might Have Spontaneously Emerged in.pdf;D

abstractThis paper presents a method for building a personalized open-domain dialogue system to address the WWH (WHAT, WHEN, and HOW) problem for natural response generation in a commercial setting, where personalized dialogue responses are heavily interleaved with casual response turns. The proposed approach involves weighted dataset blending, negative persona information augmentation methods, and the design of personalized conversation datasets to address the challenges of WWH in personalized, open-domain dialogue systems. Our work effectively balances dialogue fluency and tendency to ground, while also introducing a response-type label to improve the controllability and explainability of the grounded responses. The combination of these methods leads to more fluent conversations, as evidenced by subjective human evaluations as well as objective evaluations.
shorttitleWHAT, WHEN, and HOW to Ground
titleWHAT, WHEN, and HOW to Ground: Designing User Persona-Aware Conversational Agents for Engaging Dialogue
esigning User Pers.pdf

abstractThe advent of the Web has brought about a paradigm shift in traditional economics, particularly in the digital economy era, enabling the precise recording and analysis of individual economic behavior. This has led to a growing emphasis on data-driven modeling in macroeconomics. In macroeconomic research, Agent-based modeling (ABM) emerged as an alternative, evolving through rulebased agents, machine learning-enhanced decision-making, and, more recently, advanced AI agents. However, the existing works are suffering from three main challenges when endowing agents with human-like decision-making, including agent heterogeneity, the influence of macroeconomic trends, and multifaceted economic factors. Large language models (LLMs) have recently gained prominence in offering autonomous human-like characteristics. Therefore, leveraging LLMs in macroeconomic simulation presents an opportunity to overcome traditional limitations. In this work, we take an early step in introducing a novel approach that leverages LLMs in macroeconomic simulation. We design prompt-engineeringdriven LLM agents to exhibit human-like decision-making and adaptability in the economic environment, with the abilities of perception, reflection, and decisionmaking to address the abovementioned challenges. Simulation experiments on macroeconomic activities show that LLM-empowered agents can make realistic work and consumption decisions and emerge more reasonable macroeconomic phenomena than existing rule-based or AI agents. Our work demonstrates the promising potential to simulate macroeconomics based on LLM and its humanlike characteristics.
titleLarge Language Model-Empowered Agents for Simulating Macroeconomic Activities
riven Social Bots in Online Social Networks
espite this, their capacities to coordinate within task-oriented social contexts are under-explored. Such capabilities are crucial if LLMs are to effectively mimic human-like social behavior and produce meaningful results. To bridge this gap, we introduce collaborative generative agents, endowing LLM-based Agents with consistent behavior patterns and task-solving abilities. We situate these agents in a simulated job fair environment as a case study to scrutinize their coordination skills. We propose a novel framework that equips collaborative generative agents with human-like reasoning abilities and specialized skills. Our evaluation demonstrates that these agents show promising performance. However, we also uncover limitations that hinder their effectiveness in more complex coordination tasks. Our work provides valuable insights into the role and evolution of LLMs in task-oriented social simulations.
titleMetaAgents: Simulating Interactions of Human Behaviors for LLM-based Task-oriented Coordination via Collaborative Generative Agents
:_library7̋HAXWBYet al. - 2010 - A new look at the big five factor structure throug.pdf

abstractExploratory factor analysis (EFA) and confirmatory factor analysis (CFA), path analysis, and structural equation modeling (SEM) have long histories in clinical research. Although CFA has largely superseded EFA, CFAs of multidimensional constructs typically fail to meet standards of good measurement: goodness of fit, measurement invariance, lack of differential item functioning, and well-differentiated factors in support of discriminant validity. Part of the problem is undue reliance on overly restrictive CFAs in which each item loads on only one factor. Exploratory SEM (ESEM), an overarching integration of the best aspects of CFA/SEM and traditional EFA, provides confirmatory tests of a priori factor structures, relations between latent factors and multigroup/multioccasion tests of full (mean structure) measurement invariance. It incorporates all combinations of CFA factors, ESEM factors, covariates, grouping/multiple-indicator multiple-cause (MIMIC) variables, latent growth, and complex structures that typically have required CFA/SEM. ESEM has broad applicability to clinical studies that are not appropriately addressed either by traditional EFA or CFA/SEM.
issn1548-5943, 1548-5951
journaltitleAnnual Review of Clinical Psychology
shortjournalAnnu. Rev. Clin. Psychol.
shorttitleExploratory Structural Equation Modeling
titleExploratory Structural Equation Modeling: An Integration of the Best Features of Exploratory and Confirmatory Factor Analysis

New York

Psychology Press

shorttitleTest Theory
titleTest Theory: A Unified Treatment

issn0022-3891, 1532-7752
journaltitleJournal of Personality Assessment
shortjournalJournal of Personality Assessment
titleThe Thorny Relation Between Measurement Quality and Fit Index Cutoffs in Latent Variable Models
5et al. - 2018 - The Thorny Relation Between Measurement Quality an.pdf

abstractIt is well established that the χ2 test is influenced by sample size and will lead to over rejection of models tested using large sample sizes. In this paper it is shown that the population parameter values of a model can also influence the χ2 and lead to erroneous decisions about model acceptance/rejection. It is concluded, based on the examination of hypothetical population factor analytic models, that incremental fit indices offer a useful source of information for the analyst to assist in the interpretation of the χ2 test.
journaltitlePersonality and Individual Differences
seriesSpecial Issue on Structural Equation Modeling
shortjournalPersonality and Individual Differences
titleA Time and a Place for Incremental Fit Indices

abstractLanguage models such as GPT-3 have caused a furore in the research community. Some studies found that GPT-3 has some creative abilities and makes mistakes that are on par with human behaviour. This paper answers a related question: Who is GPT-3? We administered two validated measurement tools to GPT-3 to assess its personality, the values it holds and its self-reported demographics. Our results show that GPT-3 scores similarly to human samples in terms of personality and when provided with a model response memory - in terms of the values it holds. We provide the first evidence of psychological assessment of the GPT-3 model and thereby add to our understanding of this language model. We close with suggestions for future research that moves social science closer to language models and vice versa.
shorttitleWho Is GPT-3?
titleWho Is GPT-3? An Exploration of Personality, Values and Demographics
:_library2NF7QB- Some clarifications and recommendations on fit ind.pdf

New York

edition2. ed
seriesMcGraw-Hill Series in Psychology
titlePsychometric Theory

abstractWe report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs. While less capable than humans in many real-world scenarios, GPT-4 exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers. GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results in improved performance on measures of factuality and adherence to desired behavior. A core component of this project was developing infrastructure and optimization methods that behave predictably across a wide range of scales. This allowed us to accurately predict some aspects of GPT-4's performance based on models trained with no more than 1/1,000th the compute of GPT-4.
titleGPT-4 Technical Report
023 - GPT-4 Technical Report.pdf;D