# On the Effectiveness of Creating Conversational Agent Personalities Through
Prompting

Heng Gu  Delft University of TechnologyDelftNetherlands<EMAIL_ADDRESS>,
Chadha Degachi  Delft University of TechnologyDelftNetherlands
<EMAIL_ADDRESS>,  Uğur Genç  Delft University of
TechnologyDelftNetherlands<EMAIL_ADDRESS>,  Senthil Chandrasegaran  Delft
University of TechnologyDelftNetherlands<EMAIL_ADDRESS>and
Himanshu Verma  Delft University of TechnologyDelftNetherlands
<EMAIL_ADDRESS>
(2023)

###### Abstract.

In this work, we report on the effectiveness of our efforts to tailor the
personality and conversational style of a conversational agent based on
GPT-3.5 and GPT-4 through prompts. We use three personality dimensions with
two levels each to create eight conversational agents archetypes. Ten
conversations were collected per chatbot, of ten exchanges each, generating
1600 exchanges across GPT-3.5 and GPT-4. Using Linguistic Inquiry and Word
Count (LIWC) analysis, we compared the eight agents on language elements
including clout, authenticity, and emotion. Four language cues were
significantly distinguishing in GPT-3.5, while twelve were distinguishing in
GPT-4. With thirteen out of a total nineteen cues in LIWC appearing as
significantly distinguishing, our results suggest possible novel prompting
approaches may be needed to better suit the creation and evaluation of
persistent conversational agent personalities or language styles.

LLMs, Conversational Agents, Sentiment Analysis, Personality

††copyright: acmcopyright††journalyear: 2023††doi: XXXXXXX.XXXXXXX††ccs:
Computing methodologies Natural language generation

##  1\. Introduction

Tailoring conversational agents (CAs) to express personality has long been a
subject of interest for researchers. The perceived friendliness and
(in)formality of a CA affect the human factors of user-chatbot interaction,
including dimensions such as trust (Følstad et al., 2018; Xiao et al., 2020),
engagement (Fadhil et al., 2018), and acceptance (Sanny et al., 2020). CAs
with personality create a more consistent user experience, and can even
improve the overall user experience (Medhi Thies et al., 2017; Smestad and
Volden, 2018). Some researchers have even worked to create complementary
alignments of CA-user personalities for improved user experience (Liew and
Tan, 2016; ISBISTER and NASS, 2000).

The link between personality and language style is well established. For
example, introverted speakers tend more towards formal language, with fewer
exaggerations and more hedging, or tentative, phrasing (Dewaele and Furnham,
1999; Heylighen and Dewaele, 2002; Oberlander and Gill, 2004; Pennebaker and
King, 1999). In prior work, such language markers have been used in a rule-
based system to generate text expressing associated personalities to some
success (Mairesse and Walker, 2010). Language use has also been linked to
outcomes of negotiation and persuasion: for instance, language style in
solicitations for charitable giving has been shown to be a predictor of the
donation or assistance given (Mitra and Gilbert, 2014). It thus follows that
personality expressed through language in CAs can influence the outcome of an
end-users’ discussion with the CA.

Recent advances in text generation using Large Language Models (LLMs), present
a unique opportunity for non-rule-based, heuristic approaches to express
personality through language. Yet, within this field, a conspicuous gap
emerges when it comes to the capacity of LLMs to generate language styles
corresponding to personality archetypes. Though the use of LLMs to power CAs
is on the rise (Basar et al., 2023; Ross et al., 2023), only one study (Jiang
et al., 2023) begins to investigate the possibilities of personality
generation in GPT-3.5, but little work quantifies LLMs’ fidelity to, or
linguistic variances influenced by, personality prompts over the length of a
conversation.

There have been studies in persuasive technologies and language design, in
fields such as public health communication and marketing (Bai et al., 2023;
Karinshak et al., 2023). However, these predominantly concentrate on direct
user feedback or discernible shifts in persuasion outcomes to evaluate the
effectiveness of linguistic element tailoring, rather than on quantifying the
actual variance and consistency of their chosen language cues.

In this work, we report on an approach to express personality through language
in a goal-oriented CA. Specifically, we investigate the expression of
personality communicated through language in the context of solicitation for
charitable giving. We leverage prior research on charitable giving (Mitra and
Gilbert, 2014) showing that the potential donor’s engagement with and response
to the associated cause is sensitive to the language used in solicitation. We
use literature on charitable giving (Small et al., 2007; Chang and Lee, 2010;
Fatkhiyati and Suharno, 2019; Chen et al., 2019) to derive three dimensions of
personality that could be inferred from the language of solicitation. These
are (1) Attitude, i.e., optimistic vs pessimistic, (2) Authority, i.e.,
authoritative vs submissive, and (3) Reasoning, i.e., analytical vs affective.

Using “prompting”, a technique that is used to customize the output of LLMs
using natural-language guidelines (White et al., 2023), we express the above
personalities, first in a CA powered by the Generative Pre-Trained Transformer
(GPT)-3.5 LLM, and then by the GPT-4 LLM. Finally, we use Linguistic Inquiry
and Word Count (LIWC) (Boyd et al., 2022)—a tool that uses psycholinguistic
dictionary categories associating the use of certain words with certain
psychological processes—to answer the following research question:

  * RQ.

Does language generated by LLMs—using prompt engineering to express certain
personalities—in conversational situations align with existing associations
between said personalities and language use?

Our results show that GPT-4 outperforms GPT-3.5 in its ability to adhere to
prompted personality language characteristics. Using nineteen relevant
dictionary categories in LIWC as potentially distinguishing characteristics of
these personalities, we find that text from CAs powered by the GPT-4 LLM shows
significant distinction between personalities along twelve characteristics,
while GPT-3.5 shows the same for only four characteristics. However, given
that we investigated a total of nineteen LIWC categories, we still see
considerable room for improvement in the performance of the both models. Thus,
we also provide recommendations for possible novel prompting strategies better
suited to persistent personality generation. Moreover, through this study, we
present a novel approach to CA personality design which diverges from the
psychological models used in past work, to focus on real-world domain
knowledge and tasks for the extraction of relevant personality traits.

##  2\. Related Work

###  2.1. Language and Personality

A few theories of psychology and human behaviour connect language use and
personality. The concept of “words as attention” is one wherein researchers
argue the language choice in people is deliberate and therefore reflective, to
some extent, of their focus (Stone et al., 1966; Boyd and Schwartz, 2021). The
work of Wegner et al. (1987) validates this framing to an extent, showing that
when prompted to output a verbal stream of consciousness while avoiding
thinking of a “white bear”, participants often struggled and failed. Other
theories posit that language use indicates how individuals process information
(Tausczik and Pennebaker, 2010a), or the intensity of their attitude towards a
given subject (Stone et al., 1966). Originally, this style of analysis was
carried out through manual, qualitative coding of text.

Computational methods of predicting personality from text and language cues
also exist, with dictionary-based approaches such as LIWC among the most
popular approaches. In their work, Koutsoumpis et al. (2022) show that twenty
categories of language extracted from text by LIWC can be associated with the
dimensions of the Big Five personality model in both self- and observer-
reported personality traits. For example, emotional stability was associated
with the use of first-person singular pronouns, and conscientiousness with
swear words, anger, and negative emotions. Similarly, Zhou et al. (2021) and
Lord et al. (2015) use LIWC to analyse the situational empathy invoked in
participants by distressing situations from text responses. Situational
empathy can give us insight into user dispositional, or trait empathy, a
stable element of personality, which has strong influence on both situational
empathy and charitable giving (Zhou et al., 2021).

###  2.2. Language and Persuasion

The effectiveness of persuasive messaging is known to relate to its relevance
and personal appeal to the intended target (Petty and Cacioppo, 1986). Part of
this personalisation is achieved through language. Pfeiffer et al. (2023) find
that language formality in donation solicitation is positively correlated with
charitable giving, for example. Ye and Mattila (2022) found figurative, as
opposed to literal language, improved user’s ability to visualize in the
context of plant-based meat consumption and in turn increased favourable
consumer behaviour. Lastly, Bai et al. (2023) show that statements
incorporating rhetorical devices such as antanagoge, anaphora, and rhetorical
questions were most influential in predicting text persuasiveness using
machine learning models.

In their work Wang et al. (2020) developed a CA for persuasive messaging in
the domain of charitable giving. The researchers used deep learning techniques
to equip their agent with a variety of donation solicitation strategies
expressed through language, including “rational” vs “emotional” appeals. The
study concluded that the effect of a given donation strategy was moderated by
the user’s own personality, for example extroverted users were more likely to
be persuaded by emotional donation appeals than others. Similar efforts using
LLM technology do not yet exist.

###  2.3. Personable Conversational Agents

Mairesse and Walker (2010), and others since (Harrison et al., 2019; Ruane et
al., 2021; Kuhail et al., 2022) have studied personality-based natural
language generation, as well as dialogue design, and the perception thereof.
Much of this past work relies on the aforementioned Big-Five personality
model, and creates associations between personality and language cues using
linguistics and psychology literature. In a few cases, other psychological
personality models, such as Myers-Briggs, and other language cue extraction
methods, such as machine learning and sentiment analysis (Fernau et al., 2022)
are employed. Few studies have worked to craft CA personalities in relation to
the context and domain in which it is deployed. Moreover, these studies used a
variety of approaches for implementing their conversational agents including
deep learning and hand-crafted dialogue, however, few have investigated Large
Language Models in this context. Given the performance improvement LLMs offer
over previous NLG approaches (OpenAI, 2023), this is a great oversight.

As mentioned, one study (Jiang et al., 2023) so far has taken advantage of the
rise of LLMs across the natural language generation space, to investigate
whether these models could be prompted to create agents with personality. In
their work, Jiang et al. (2023) used Big-Five personality dimensions, as well
as gender, to prompt a GPT-3.5 agent to create an 800-word childhood story
analysed using LIWC language elements. Though the authors concluded that the
personas they created were significantly different in the LIWC language
categories they exhibited, they did not investigate whether this pattern would
hold in naturalistic longer human-agent interaction settings such as
conversation. serapiogarcía2023personality use a somewhat similar methodology
to evaluate the PaLM models (Chowdhery et al., 2022) in their ability to
embody personality, using prompt engineering to create personas and asking the
CA to answer psychometric personality tests. They similarly concluded CAs to
be cable of consistent personality embodiment.

##  3\. Method

We prompt-engineered a series of charity solicitation CAs employing the
GPT-3.5 and GPT-4 models. The two models with similar architectures but
different sizes and parameters were chosen for comparison with prior studies
focusing primarily on aptitudes across various human expertise categories,
such as through standardized exams (Bommarito II and Katz, 2022; Liu et al.,
2023; Ateia and Kruschwitz, 2023; Koubaa, 2023), we believe it novel to
compare the CA outputs through the lens of inguistic outputs. These CAs,
representing a fabricated charity organization named the “Wildlife Bridge
Foundation,” were designed to simulate a solicitation event with potential
donors.

Building on prior research on effective charity solicitation, we integrated
variations in the CA personality and solicitation strategy across three
dimensions: Attitude, i.e., optimistic vs pessimistic (Small et al., 2007),
Authority, i.e., authoritative vs submissive (Chang and Lee, 2010; Chen et
al., 2019), and Reasoning, i.e., analytical vs affective (Fatkhiyati and
Suharno, 2019), reflected in the language used to persuade the potential
donor. These three dimensions—each with two polar opposite
attributes—combinatirially resulted in eight ($2^{3}$) distinct CA
personalities. We powered one set of all eight CA personalities with the
GPT-3.5 model and another set with the GPT-4 model.

We collected the ten most, and least, popular petitions from Avaaz and
Change.org, two popular online petition platforms, to analyse with LIWC,
identifying the most frequently used language categories and psychological
processes within. These characteristics included clout, tone, authenticity,
emotion, cognition, and others. Once identified, we grouped these
characteristics along the three personality scales derived from literature.
For example, authority was associated with the LIWC categories of clout and
authenticity, and an authoritative text may have used more words and phrases
such as “must”, “have to”, and “should”, in contrast to a submissive text
which used words and phrases such as “if it’s okay with you”, “maybe”, and “if
you don’t mind”. Details of each personality dimension and associated LIWC
categories are described in Sec. 4.1.

###  3.1. Prompt Engineering

We first designed a core prompt with modifiable slots for varying solicitor
traits. Grounded in advanced prompt design methodologies (White et al., 2023),
this foundational prompt encompasses four principal elements:

  1. (1)

Task: Act as a charity solicitor for…

  2. (2)

Goal: Get speaker to donate…

  3. (3)

Rules: Do not provide URLS, keep response short…

  4. (4)

Persona: The solicitor’s name is [NAME], personality:
[optimistic/pessimistic], and [Authoritativeness/Submissiveness]. Only speak
as Alex from now on. Use [Emotion/Logic-based reasoning] to convince the
donor.

###  3.2. Benchmark Test

To assess the performance of the 16 uniquely personalized CAs (8 GPT-3.5 and 8
GPT-4), we devised a standardized script composed of the same 10 dialogue
excerpts, which were queried to each CA over 10 sessions, resulting in 160
unique conversations and 1,600 pieces of generated responses, 100 per CA.

This set of interactions was designed based on internal pilot tests simulating
potential donor conversations. We distilled the most pertinent questions and
responses to form this script.

##  4\. Analysis

Following synthesis of this benchmarking sample dataset of generated
responses, our primary objective was to evaluate their consistency relative to
our prompts indicating the desired output qualities. We utilized LIWC to
assess the cognitive and emotional attributes of the generated text, measured
along dictionary categories that we deemed relevant to the personality
dimensions.

###  4.1. Measures

LIWC’s psycholinguistic dictionary categories (Boyd et al., 2022) provided us
with the means to gauge the linguistic attributes of each agent’s outputs.

Specifically, we focused on LIWC categories known to reflect our three chosen
traits:

Attitude::

For this trait, which could take the attribute of either optimistic or
pessimistic, we looked at linguistic elements pointing towards sentiment. In
the current version of LIWC, optimism is a subcategory of Affect (McDonnell,
2015). This parallels the Tone categories (tone_pos, tone_neg) which are
indicative of sentiment of the text as opposed to embodied emotions (Boyd et
al., 2022). Positive emotion(Emo_Pos) and tentativeness(tentat) as indicators
have also been used for optimism (Gasper et al., 2020). Additionally, based on
the psychological correlates outlined by LIWC guidelines, we also monitored
FutureTense representing future and goal-oriented words, and markers of
Anxiety indicating future-oriented emotions (e.g., words like worried,
fearful, nervous) (Tausczik and Pennebaker, 2010b; Boyd et al., 2022).

Authority::

For this trait, the attributes of which could be authoritative or submissive,
we looked at linguistic elements pointing towards status, dominance, and
social hierarchy. Clout serves as an aggregate category representing social
status and power dynamics, with subcategories of personal pronouns (ppron)
indicative of these dynamics associated with authority (Buljan et al., 2020;
Lumontod III, 2020). Additionally, according to the psychometric guidelines,
certitude and absolutist language (All-none) could also be an indicator of
authoritative language, and Assent category could be an indicator of
submissiveness (Boyd et al., 2022).

Reasoning: :

In the context of solicitation traits, analytical specifically refers to a
solicitation strategy of providing statistical evidence, and affective refers
to providing individualized details (Small et al., 2013; Fatkhiyati and
Suharno, 2019). For this trait we looked at linguistic elements pointing to
thinking styles and cognitive mechanism words (Cognition, Analytic, Quantity,
Numbers), as well as the presence of emotional words (Emotion, Affect,
Authentic) (Tausczik and Pennebaker, 2010b; Boyd et al., 2022).

###  4.2. Procedure

Given the selected traits, interaction effects between them are anticipated.
Our objective was to discern the parts-worth effect attributable to the
prompted solicitor parameters of Attitude, Reasoning, and Authority. To
accomplish this, we employed conjoint analysis, aiming to deconstruct the
individual impact of these traits on the output linguistic quality. The
conceived equation is a linear regression model which relates the LIWC
category values to the factors of Attitude, Authority, and Reasoning prompt
components, along with their interaction effects.

The equation representing this relationship is:

(1)
$\begin{split}M_{LIWC}=&\beta_{0}+\beta_{1}(\text{Attitude})+\beta_{2}(\text{Authority})+\\\
&+\beta_{3}(\text{Reasoning})+\beta_{4}(\text{Attitude}\times\text{Authority})\\\
&+\beta_{5}(\text{Attitude}\times\text{Reasoning})\\\
&+\beta_{6}(\text{Reasoning}\times\text{Authority})\\\
&+\beta_{7}(\text{Attitude}\times\text{Authority}\times\text{Reasoning})+\epsilon\end{split}$

Table 1. Effect of Prompt Factors on LIWC Categories

LIWC GPT3.5 GPT4 Category $R^{2}$ $\beta_{1}$ $\beta_{2}$ $\beta_{3}$
$\beta_{4}$ $\beta_{5}$ $\beta_{6}$ $\beta_{7}$ $R^{2}$ $\beta_{1}$
$\beta_{2}$ $\beta_{3}$ $\beta_{4}$ $\beta_{5}$ $\beta_{6}$ $\beta_{7}$
Authority Clout 0.040 - - - - 0.270 **1.800 *-1.250 *-1.129 - Authentic 0.083
- - - - 0.120 - - - - Ppron 0.228 - - - - 0.112 - - - - Certitude 0.096 - - -
- 0.168 *-0.200 - - - Allnone 0.227 - - - - 0.168 *-0.148 *-0.130 - - Assent
0.987 **0.211 **0.217 - - 0.967 *0.424 **0.406 **-0.447 **-0.346 Attitude Tone
0.315 *0.334 *0.363 **0.448 - 0.283 **1.278 *0.792 - - Affect 0.123 - *0.193 -
- 0.273 **0.688 - - - Tone_Pos 0.161 - *0.234 - - 0.402 **0.997 *0.325 - -
Tone_Neg 0.281 **-0.082 - - - 0.338 *-0.308 - - - Emo_Anx 0.089 - - - - 0.257
**-0.047 *-0.029 - - Reasoning Cognition 0.097 - - - - 0.336 *0.527 - - -
Analytic 0.201 **-1.750 - - - 0.142 - - - - Authentic 0.083 - - - - 0.119 - -
- - Quantity 0.136 - - *-0.147 - 0.129 *0.291 - - - Numbers 0.068 - - - -
0.081 - - - - Emotion 0.153 - - - - 0.332 *0.159 - - - Affect 0.123 - - - -
0.273 - - - - Note: * indicates $p<0.05$; ** indicates $p<0.01$. $\beta$
values are not shown when not statistically significant.

Where:

  * •

$M_{LIWC}$ is the measure for a given LIWC category (e.g., Tone)

  * •

$\beta_{0}$ is the baseline effect present in the core prompt and LLM.

  * •

$\beta_{1},\beta_{2},\beta_{3}$ are coefficients represent the influence of
the factors Attitude, Authority, and Reasoning respectively on the linguistic
quality.

  * •

$\beta_{4},\beta_{5},\beta_{6},\beta_{7}$ represent the interaction effects
between the factors. (ie. $\beta_{4}$ represents the interaction between
Attitude and Authority.)

  * •

$\epsilon$ represents the error term.

##  5\. Results

Table 1 shows the LIWC categories used per personality dimension. A category
is correlated with a dimension if it is significantly different on the low and
high end of that personality scale, e.g. Tone is significantly correlated with
Attitude if it is sufficiently different in optimistic and pessimistic
chatbots. If a coefficient is positive, it implies that as the factor
increases, the linguistic quality (in that particular category) increases.

##  6\. Discussion

Our study evaluated the capacity of LLMs to manifest engineered prompted
traits. The results offer several key insights:

###  6.1. GPT Model Variation

GPT-4 appears to be more sensitive to the prompted traits across a broader
range of LIWC categories compared to GPT-3.5. The wider range of significant
categories in GPT-4 might suggest an enhanced capability to capture and
reflect certain nuances, but it doesn’t automatically imply superior overall
performance in expressing the prompted traits. The explanation may also lie to
some extent with the nature of LIWC dictionary categories.

Consider the following text generated by a CA powered by GPT-3.5 with the
personality traits of optimistic-authoritative-analytical: “As someone who
deeply cares about the welfare of animals and the environment, I found my
purpose in helping organizations like the Wildlife Bridge Foundation.” Compare
this with another CA with the same GPT-3.5 model, but with the personality
traits of optimistic-authoritative-affective: “As someone who deeply cares
about the well-being of animals, it breaks my heart to see their habitats
destroyed by urban expansion.”

In both examples, the italicized terms correspond to not just LIWC categories
corresponding to analytical and affective personalities, but also optimistic
and pessimistic personalities. This is because LIWC dictionary categories are
not mutually exclusive: the same word can appear in multiple categories. A
more thorough evaluation of this approach would also need to incorporate a
measure of overlapping words across relevant LIWC categories from the
generated text.

###  6.2. Conversation Benchmarking

Our demonstrated approach of iterative simulated conversations has potential
to serve as a method for assessing the variability in LLM output within
defined prompted traits. While this approach would need a more rigorous
validation of the representation of these personalities, the use of expert-
curated psycholinguistic dictionary categories—such as those in LIWC—could
provide CA designers with a standard set of linguistic measures to use for
specific applications.

###  6.3. Trait Variability

Prompted traits have very different impact on linguistic quality. Attitude
significantly influences linguistic qualities across GPT models. Authority has
a stronger impact on the GPT-4 model, while Reasoning shows the least effect
across models. Identifying which traits have stronger impacts allows
developers to better tailor LLM outputs.

Furthermore, we wish to highlight the nuanced relationship between prompt
engineering and resultant LLM outputs, especially when a dearth of good
methods for evaluating output variability makes it doubly difficult to
quantify this relationship. By demonstrating one method, we hope to bring
attention to the remaining scope for refining evaluation and monitoring
techniques for quality and consistency of generated content.

###  6.4. Implications for personality-based CAs.

Given these results, we formulate the following design and development
recommendations for effective and consistent personality-based CAs:

  1. (1)

Prompt Design: We hypothesize that introducing more synonyms to a given prompt
when creating personality archetypes may lend the personality element more
weight and create more long-standing effects. For example, an optimistic agent
may be described as optimistic, positive, and hopeful in a prompt instead of
simply optimistic.

  2. (2)

Prompt Programming: We suggest periodic re-injection of the persona prompt
components during the conversation could enforce model adherence to the
intended persona, potentially mitigating drift.

  3. (3)

LIWC-Annotated Prompt Programming: by real-time monitoring present LIWC
categories in conversation output, any significant deviation or drift from the
desired personality could be detected. This then can create a feedback loop to
nudge the model back to the desired personality traits, ensuring consistent
alignment with the initial prompt.

Future research could also explore the role of user input in conversation on
steering agent personality over time, the generalizability of our domain-
driven personality crafting approaching, and expand the set of LLM models
evaluated in this work.

##  7\. Conclusion

We demonstrate in this paper the feasibility of generating CA personalities
through LLM prompt engineering. We craft personality archetypes from
literature and charity solicitation domain knowledge, evaluating the effect of
personality prompts in LLM CAs on language style using LIWC computational text
analysis. We show that the performance of LLM CAs in this area is model
dependant, and personality dimension dependant. Based on our results, we
present design and development recommendations to fellow researchers,
including recommendations on prompt engineering and future research
directions.

## References

  * (1)
  * Ateia and Kruschwitz (2023) Samy Ateia and Udo Kruschwitz. 2023.  Is ChatGPT a Biomedical Expert? – Exploring the Zero-Shot Performance of Current GPT Models in Biomedical Tasks.  arXiv:2306.16108 [cs.CL]
  * Bai et al. (2023) Hui Bai, Jan Voelkel, Johannes Eichstaedt, and Robb Willer. 2023\.  Artificial Intelligence Can Persuade Humans on Political Issues.  (Aug 2023).  https://doi.org/10.21203/rs.3.rs-3238396/v1
  * Basar et al. (2023) Erkan Basar, Divyaa Balaji, Linwei He, Iris Hendrickx, Emiel Krahmer, Gert-Jan de Bruijn, and Tibor Bosse. 2023.  HyLECA: A Framework for Developing Hybrid Long-Term Engaging Controlled Conversational Agents. In _Proceedings of the 5th International Conference on Conversational User Interfaces_ (Eindhoven, Netherlands) _(CUI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 56, 5 pages.  https://doi.org/10.1145/3571884.3604404
  * Bommarito II and Katz (2022) Michael Bommarito II and Daniel Martin Katz. 2022.  GPT takes the bar exam.  _arXiv preprint arXiv:2212.14402_ (2022).
  * Boyd et al. (2022) Ryan L Boyd, Ashwini Ashokkumar, Sarah Seraj, and James W Pennebaker. 2022.  The development and psychometric properties of LIWC-22.  _Austin, TX: University of Texas at Austin_ (2022), 1–47.
  * Boyd and Schwartz (2021) Ryan L. Boyd and H. Andrew Schwartz. 2021.  Natural Language Analysis and the Psychology of Verbal Behavior: The Past, Present, and Future States of the Field.  _Journal of Language and Social Psychology_ 40, 1 (Jan 2021), 21–41.  https://doi.org/10.1177/0261927X20967028
  * Buljan et al. (2020) Ivan Buljan, Daniel Garcia-Costa, Francisco Grimaldo, Flaminio Squazzoni, and Ana Marušić. 2020.  Large-scale language analysis of peer review reports.  _Elife_ 9 (2020), e53249.
  * Chang and Lee (2010) Chun-Tuan Chang and Yu-Kang Lee. 2010.  Effects of message framing, vividness congruency and statistical framing on responses to charity advertising.  _International Journal of Advertising_ 29, 2 (2010), 195–220.
  * Chen et al. (2019) Yan Chen, Shuyuan Deng, Dong-Heon Kwak, Ahmed Elnoshokaty, and Jiao Wu. 2019.  A multi-appeal model of persuasion for online petition success: A linguistic cue-based approach.  _Chen, Y., Deng, S., Kwak, DH, Elnoshokaty, A., & Wu, J_ (2019).
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  PaLM: Scaling Language Modeling with Pathways.  arXiv:2204.02311 [cs.CL]
  * Dewaele and Furnham (1999) Jean-Marc Dewaele and Adrian Furnham. 1999.  Extraversion: The Unloved Variable in Applied Linguistic Research.  _Language Learning_ 49, 3 (1999), 509–544.  https://doi.org/10.1111/0023-8333.00098 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/0023-8333.00098
  * Fadhil et al. (2018) Ahmed Fadhil, Gianluca Schiavo, Yunlong Wang, and Bereket A. Yilma. 2018.  The Effect of Emojis When Interacting with Conversational Interface Assisted Health Coaching System. In _Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare_ (New York, NY, USA) _(PervasiveHealth ’18)_. Association for Computing Machinery, New York, NY, USA, 378–383.  https://doi.org/10.1145/3240925.3240965
  * Fatkhiyati and Suharno (2019) Nurrahma Restia Fatkhiyati and Suharno Suharno. 2019.  Rhetorical strategy and linguistics features in e-petition through Change. org.  _Lingua Cultura_ 13, 4 (2019), 239–245.
  * Fernau et al. (2022) Daniel Fernau, Stefan Hillmann, Nils Feldhus, and Tim Polzehl. 2022.  Towards Automated Dialog Personalization using MBTI Personality Indicators. In _Proc. Interspeech 2022_. 1968–1972.  https://doi.org/10.21437/Interspeech.2022-376
  * Følstad et al. (2018) Asbjørn Følstad, Cecilie Bertinussen Nordheim, and Cato Alexander Bjørkli. 2018\.  What makes users trust a chatbot for customer service? An exploratory interview study, In Internet Science.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ 11193 Lncs, 194–208.  https://doi.org/10.1007/978-3-030-01437-7_16
  * Gasper et al. (2020) Karen Gasper, Lauren A Spencer, and Brianna L Middlewood. 2020\.  Differentiating hope from optimism by examining self-reported appraisals and linguistic content.  _The Journal of Positive Psychology_ 15, 2 (2020), 220–237.
  * Harrison et al. (2019) Vrindavan Harrison, Lena Reed, Shereen Oraby, and Marilyn A. Walker. 2019.  Maximizing Stylistic Control and Semantic Accuracy in NLG: Personality Variation and Discourse Contrast.  _arXiv: Computation and Language_ (2019).
  * Heylighen and Dewaele (2002) Francis Heylighen and Jean-Marc Dewaele. 2002.  Variation in the Contextuality of Language: An Empirical Measure.  _Foundations of Science_ 7, 3 (Sep 2002), 293–340.  https://doi.org/10.1023/A:1019661126744
  * ISBISTER and NASS (2000) KATHERINE ISBISTER and CLIFFORD NASS. 2000.  Consistency of personality in interactive characters: verbal cues, non-verbal cues, and user characteristics.  _International Journal of Human-Computer Studies_ 53, 2 (2000), 251–267.  https://doi.org/10.1006/ijhc.2000.0368 Citation Key: ISBISTER2000251.
  * Jiang et al. (2023) Hang Jiang, Xiajie Zhang, Xubo Cao, and Jad Kabbara. 2023\.  PersonaLLM: Investigating the Ability of GPT-3.5 to Express Personality Traits and Gender Differences.  arXiv:2305.02547 [cs.CL]
  * Karinshak et al. (2023) Elise Karinshak, Sunny Xun Liu, Joon Sung Park, and Jeffrey T. Hancock. 2023.  Working With AI to Persuade: Examining a Large Language Model’s Ability to Generate Pro-Vaccination Messages.  _Proceedings of the ACM on Human-Computer Interaction_ 7, CSCW1 (Apr 2023), 116:1–116:29.  https://doi.org/10.1145/3579592
  * Koubaa (2023) Anis Koubaa. 2023\.  GPT-4 vs. GPT-3.5: A concise showdown.  (2023).
  * Koutsoumpis et al. (2022) Antonis Koutsoumpis, Janneke K. Oostrom, Djurre Holtrop, Ward van Breda, Sina Ghassemi, and Reinout E. de Vries. 2022.  The kernel of truth in text-based personality assessment: A meta-analysis of the relations between the Big Five and the Linguistic Inquiry and Word Count (LIWC).  _Psychological Bulletin_ 148, 11–12 (2022), 843–868.  https://doi.org/10.1037/bul0000381
  * Kuhail et al. (2022) Mohammad Amin Kuhail, Justin Thomas, Salwa Alramlawi, Syed Jawad Hussain Shah, and Erik Thornquist. 2022.  Interacting with a Chatbot-Based Advising System: Understanding the Effect of Chatbot Personality and User Gender on Behavior.  _Informatics_ 9, 44 (Dec 2022), 81\.  https://doi.org/10.3390/informatics9040081
  * Liew and Tan (2016) Tze Wei Liew and Su-Mae Tan. 2016.  Virtual agents with personality: Adaptation of learner-agent personality in a virtual learning environment. 157–162.  https://doi.org/10.1109/ICDIM.2016.7829758 Citation Key: inproceedings.
  * Liu et al. (2023) Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023\.  Evaluating the logical reasoning ability of chatgpt and gpt-4.  _arXiv preprint arXiv:2304.03439_ (2023).
  * Lord et al. (2015) Sarah Peregrine Lord, Elisa Sheng, Zac E. Imel, John Baer, and David C. Atkins. 2015\.  More than reflections: Empathy in motivational interviewing includes language style synchrony between therapist and client.  _Behavior Therapy_ 46, 3 (2015), 296–303.  https://doi.org/10.1016/j.beth.2014.11.002 Citation Key: LORD2015296.
  * Lumontod III (2020) Robinson Z Lumontod III. 2020\.  Seeing the invisible: Extracting signs of depression and suicidal ideation from college students’ writing using LIWC a computerized text analysis.  _Int. J. Res. Stud. Educ_ 9 (2020), 31–44.
  * Mairesse and Walker (2010) François Mairesse and Marilyn A. Walker. 2010.  Towards personality-based user adaptation: Psychologically informed stylistic language generation.  _User Modeling and User-Adapted Interaction_ 20 (8 2010), 227–278.  Issue 3. https://doi.org/10.1007/s11257-010-9076-2
  * McDonnell (2015) Michelle McDonnell. 2015\.  Comparison of text analysis programs for identification of emotional expression.  (2015).
  * Medhi Thies et al. (2017) Indrani Medhi Thies, Nandita Menon, Sneha Magapu, Manisha Subramony, and Jacki O’Neill. 2017\.  How Do You Want Your Chatbot? An Exploratory Wizard-of-Oz Study with Young, Urban Indians. In _Human-Computer Interaction - INTERACT 2017_ , Regina Bernhaupt, Girish Dalvi, Anirudha Joshi, Devanuj K. Balkrishan, Jacki O’Neill, and Marco Winckler (Eds.). Springer International Publishing, Cham, 441–459.
  * Mitra and Gilbert (2014) Tanushree Mitra and Eric Gilbert. 2014.  The language that gets people to give: Phrases that predict success on kickstarter. In _Proceedings of the ACM conference on Computer supported cooperative work & social computing_. 49–61.  https://doi.org/10.1145/2531602.2531656
  * Oberlander and Gill (2004) Jon Oberlander and Alastair Gill. 2004.  Language generation and personality: Two dimensions, two stages, two hemispheres?  _AAAI Spring Symposium - Technical Report_ 2 (Jan 2004).
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Pennebaker and King (1999) James W. Pennebaker and Laura A. King. 1999.  Linguistic styles: Language use as an individual difference.  _Journal of Personality and Social Psychology_ 77, 6 (1999), 1296–1312.  https://doi.org/10.1037/0022-3514.77.6.1296
  * Petty and Cacioppo (1986) Richard Petty and John Cacioppo. 1986.  The Elaboration Likelihood Model of Persuasion.  _Advances in hydroscience_ 19 (Jul 1986), 124–205.
  * Pfeiffer et al. (2023) Bruce E. Pfeiffer, Aparna Sundar, and Edita Cao. 2023.  The influence of language style (formal vs. colloquial) on the effectiveness of charitable appeals.  _Psychology & Marketing_ 40, 3 (2023), 542–553.  https://doi.org/10.1002/mar.21729
  * Ross et al. (2023) Steven I. Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D. Weisz. 2023\.  The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development. In _Proceedings of the 28th International Conference on Intelligent User Interfaces_ (Sydney, NSW, Australia) _(IUI ’23)_. Association for Computing Machinery, New York, NY, USA, 491–514.  https://doi.org/10.1145/3581641.3584037
  * Ruane et al. (2021) Elayne Ruane, Sinead Farrell, and Anthony Ventresque. 2021\.  User perception of text-based chatbot personality. In _Chatbot research and design_ , Asbjørn Følstad, Theo Araujo, Symeon Papadopoulos, Effie L.-C. Law, Ewa Luger, Morten Goodwin, and Petter Bae Brandtzaeg (Eds.). Springer International Publishing, Cham, 32–47.  Citation Key: 10.1007/978-3-030-68288-0_3.
  * Sanny et al. (2020) Lim Sanny, Ari Susastra, Choky Roberts, and Reflia Yusramdaleni. 2020.  The analysis of customer satisfaction factors which influence chatbot acceptance in Indonesia.  _Management Science Letters_ (01 2020), 1225–1232.  https://doi.org/10.5267/j.msl.2019.11.036
  * Small et al. (2007) Deborah A Small, George Loewenstein, and Paul Slovic. 2007\.  Sympathy and callousness: The impact of deliberative thought on donations to identifiable and statistical victims.  _Organizational Behavior and Human Decision Processes_ 102, 2 (2007), 143–153.
  * Small et al. (2013) Deborah A Small, George Loewenstein, and Paul Slovic. 2013\.  Sympathy and callousness: The impact of deliberative thought on donations to identifiable and statistical victims.  In _The Feeling of Risk_. Routledge, 51–68.
  * Smestad and Volden (2018) Tuva Lunde Smestad and Frode Volden. 2018.  Chatbot personalities matters. In _International Conference on Internet Science_. Springer, 170–181.  https://doi.org/10.1007/978-3-030-17705-8_15
  * Stone et al. (1966) Philip J. Stone, Dexter C. Dunphy, and Marshall S. Smith. 1966\.  _The general inquirer: A computer approach to content analysis_.  M.I.T. Press, Oxford, England. 651 pages.
  * Tausczik and Pennebaker (2010a) Yla R. Tausczik and James W. Pennebaker. 2010a.  The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods.  _Journal of Language and Social Psychology_ 29, 1 (Mar 2010), 24–54.  https://doi.org/10.1177/0261927X09351676
  * Tausczik and Pennebaker (2010b) Yla R Tausczik and James W Pennebaker. 2010b.  The psychological meaning of words: LIWC and computerized text analysis methods.  _Journal of language and social psychology_ 29, 1 (2010), 24–54.
  * Wang et al. (2020) Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2020.  Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good.  arXiv:1906.06725 [cs.CL]
  * Wegner et al. (1987) Daniel M. Wegner, David J. Schneider, Samuel R. Carter, and Teri L. White. 1987.  Paradoxical effects of thought suppression.  _Journal of Personality and Social Psychology_ 53, 1 (1987), 5–13.  https://doi.org/10.1037/0022-3514.53.1.5
  * White et al. (2023) Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023.  A prompt pattern catalog to enhance prompt engineering with chatgpt.  _arXiv preprint arXiv:2302.11382_ (2023).
  * Xiao et al. (2020) Ziang Xiao, Michelle X. Zhou, Wenxi Chen, Huahai Yang, and Changyan Chi. 2020.  If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi.org/10.1145/3313831.3376131
  * Ye and Mattila (2022) Tian Ye and Anna S. Mattila. 2022.  The impact of environmental messages on consumer responses to plant-based meat: Does language style matter?  _International Journal of Hospitality Management_ 107 (Oct 2022), 103298.  https://doi.org/10.1016/j.ijhm.2022.103298
  * Zhou et al. (2021) Ke Zhou, Luca Maria Aiello, Sanja Scepanovic, Daniele Quercia, and Sara Konrath. 2021\.  The Language of Situational Empathy.  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW1, Article 13 (apr 2021), 19 pages.  https://doi.org/10.1145/3449087