# Exploring Human-LLM Conversations: Mental Models and the Originator of
Toxicity

Johannes Schneider, Arianna Casanova Flores, Anne-Catherine Kranz  University
of LiechtensteinVaduzLiechtenstein<EMAIL_ADDRESS>
(2024)

###### Abstract.

This study explores real-world human interactions with large language models
(LLMs) in diverse, unconstrained settings in contrast to most prior research
focusing on ethically trimmed models like ChatGPT for specific tasks. We aim
to understand the originator of toxicity. Our findings show that although LLMs
are rightfully accused of providing toxic content, it is mostly demanded or at
least provoked by humans who actively seek such content. Our manual analysis
of hundreds of conversations judged as toxic by APIs commercial vendors, also
raises questions with respect to current practices of what user requests are
refused to answer. Furthermore, we conjecture based on multiple empirical
indicators that humans exhibit a change of their mental model, switching from
the mindset of interacting with a machine more towards interacting with a
human.

Large Language Models, Human-AI interaction, toxicity, mental model

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; June 03–05, 2024; Woodstock, NY††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Hundreds of millions of users interact with commercial generative AI models
such as OpenAI’s ChatGPT (Holt, 2023). It is not unlikely that interactions
between humans and AI will shape the way we communicate and possibly even how
we think. Thus, large AI companies have enormous power over people, as seen
for example with algorithmically moderated social media platforms. A
controversial study on Facebook demonstrated that algorithmically manipulating
users’ feeds could change their emotions on a large scale (Kramer et al.,
2014). The recommendation algorithm of the popular video platform Tik-Tok has
even been associated with suicides (Amnesty International, 2023). Governments
try to mitigate such threats and to protect society from harm caused by
algorithms and AI through laws. However, legal regulations like the European
AI act and GDPR also pose risks for organizations impacting their governance
and products (Schneider et al., 2023a). Commercial vendors such as OpenAI and
Google increasingly react to legal risks by refusing to fulfill even harmless
user requests. Thereby, the diminish the potential value of AI delivered to
user. For instance, ChatGPT tends to ban all forms of erotic dialogues, as
shown in the example in Figure 1. To avoid bias, Google has overly adjusted AI
models to the point, where they show inaccurate historic content such as the
depiction of female popes (Board, 2024).

Figure 1. Commercial models like OpenAI’s GPT4o tend to prefer denying users
prompts’ in favor of mitigating risk of toxic responses

Figure 2. Example where a human went from communication style typical for
interacting with a machine towards one more prevalent for humans as indicated
by the use of politeness and shorter prompts within a single conversation.

We argue that understanding the source or trigger of toxicity is crucial for
accurate judgment and regulation of AI. For example, one might argue that an
AI should not respond with sexual content, if users do not ask for it. But if
users explicitly demand it, it is fine. Currently, such an understanding seems
to be missing, which can also be blamed on AI’s opaqueness(Longo et al.,
2024). While toxicity detection and mitigation has received much attention
from a technical perspective (Garg et al., 2023; Hosseini et al., 2017;
OpenAI, 2024), a discussion on how toxicity emerges within interactions is
missing. We seek to understand whether humans provoke AI to be toxic or
whether AI or humans generate toxic content spontaneously.

More broadly, currently, knowledge on human-LLM interactions is limited, often
focused on small-scale user studies for specific tasks (Schneider et al.,
2023b; Kumar et al., 2023; Zamfirescu-Pereira et al., 2023). A deeper
understanding of interactions, such as the user’s mental model of the
conversation partner, is missing(Steyvers and Kumar, 2023). As LLMs are known
to be software running on machines, human communication is expected to mimic
those typical for other machines. As LLMs often pass the Turing test for short
periods (Jones and Bergen, 2024), there could be a shift in the perception of
LLMs as more human. If AI is not only perceived as more human but also treated
as more human by users, this could have far-reaching consequences. For
example, in 2021, the chatbot XiaoIce(Zhou et al., 2020) comforted millions of
lonely Chinese users, leading to privacy and other concerns (Xplore, 2021).
There are clear indicators that tasks like information search differ when
using LLMs compared to classical internet search, where keywords are used
instead of complete sentences. But the mental models employed in human-LLM
conversations are not yet well understood.

Therefore, in this work, we aim to provide evidence to better understand the
following research questions (RQs):

_RQ 1: What is the mental model employed by humans when interacting with LLMs?
Is it that of machines or that of humans?_
_RQ 2: How does toxic severe content emerge? Is it user or LLM provoked or
emerging spontaneously?_

Our evidence to address these questions is based on an analysis of more than
200k real-world conversations from a public dataset(Zheng et al., 2023). The
dataset contains a broad range of conversations, many of which are deemed
unsafe, offensive, or upsetting. To address RQ 1, we use established methods
and libraries from computational linguistics focusing on conversational cues
such as politeness and language complexity. To answer RQ 2, we use toxicity
measures provided by OpenAI’s moderation API alongside manual analysis and
categorization of highly toxic conversations. Our findings indicate that
toxicity is primarily triggered by humans and that there might be a shift in
the mental model employed by users from machine towards human. We also raise a
set of questions based on our analysis with respect to the censoring of LLM
assistants.

##  2\. Related Work and Background

_LLM-human interaction:_ There are a number of possible forms of human-LLM (or
more generally, GenAI) interactions and prompts as elaborated in recent
taxonomies (Shi et al., 2023; von Brackel-Schmidt et al., 2023; Gao et al.,
2024). We focus on those, where the LLM is an assistant instructed by a
(human) user. Multiple works have looked at such interactions, e.g., (Ouyang
et al., 2023) analyzed publicly shared conversations from ChatGPT sourced from
ShareGPT (ShareGPT, 2024) to assess whether they mimic conversations used to
evaluate LLMs. They found that current benchmarks have gaps, e.g., planning
and design tasks are often missing in benchmarks but much more common in user
interactions. A number of studies have investigated specific human skills,
e.g., general prompting skills (Zamfirescu-Pereira et al., 2023) as well as
skills with respect to certain tasks such as design (Kumar et al., 2023;
Sharma et al., 2024), code migration (Omidvar Tehrani and Anubhai, 2024),
coding for novices (Prather et al., 2023), negotation (Schneider et al.,
2023b) and education, i.e., assessing learning performance of students using
LLMs (Kumar et al., 2023). Studies have also looked at implications of using
LLMs on users’ views and perception of LLMs, e.g., for cowriting (Jakesch et
al., 2023) and learning (Kumar et al., 2023).

Few works have leveraged large scale datasets. Many (large) datasets
containing LLM-human interaction have been collected for fine-tuning LLMs(Liu
et al., 2024) rather than understanding interactions. For example, (Köpf et
al., 2024) contains about 50k conversations that have been gathered with the
purpose to align LLMs. That is, conversations have also been annotated. While
prompts also originate from users, users knew that they participate in a task
to collect data for LLM alignment tuning and had to follow certain guidelines.
The number of datasets obtained in a natural environment, where users have
freely engaged with the LLM, is limited (and, in turn, analysis on such data).
Most (unrestricted) real-world datasets are based on ChatGPT, e.g., (Zhao et
al., 2024) contains 1 Mio converations, and (RyokoAI, 2024) contains 90k by
sourcing from a platform allowing to share converations, e.g. ShareGPT
(ShareGPT, 2024). But conversations using ChatGPT are strictly moderated.
(Zheng et al., 2023) is among the largest real-world datasets with 1 Mio.
conversations and stems from signficantly less moderated models.

Our work differs from prior work as it investigates a broad range of tasks
also comparing them among each other. Also our dataset is much larger and
stems from less constrained (in terms of censorship) models than commercial
models like ChatGPT used in most of the above studies. We also specifally
analyze dialogues of LLMs that might be considered unethical and commonly
suffer from toxicity, which is less studied for real-world conversations but
rather only part of constrained settings, e.g., to collect data to ensure
human-LLM alignment. Human-AI alignment is an important topic that aims at
creating AI systems adhering to human values and intentions using data
particularly collected for such tasks as shown by early works such as
InstructGPT (Ouyang et al., 2022) and many follow-ups (Shen et al., 2023).

_Mental models:_ Mental models are a concentrated, personally constructed,
internal conception, of external phenomena (historical, existing or
projected), or experience, that affects how a person acts (Rook, 2013).
People’s mental models of machines and humans differ, which is reflected in
communication styles, expectations, and perceived capabilities (Nass and Moon,
2000; Reeves and Nass, 1996; Norman, 1988). Among those differences are:

  * •

Predictability and Precision(Nass and Moon, 2000):
Machines: People often expect machines to be highly predictable, precise, and
consistent. They anticipate that machines will follow programmed rules and
provide exact responses or perform tasks accurately based on input.
Humans: Human interactions are expected to be more flexible, nuanced, and
context-sensitive. Humans are seen as capable of understanding unspoken
context, emotions, and implicit meanings.

  * •

Emotion and Empathy(Reeves and Nass, 1996):
Machines: Typically, users do not expect emotional understanding or empathy
from machines. Communication with machines is usually more task-oriented,
direct, and devoid of emotional content.
Humans: Human interactions are rich in emotional content. People expect
empathy, emotional support, and an understanding of social cues.

  * •

Complexity and Understanding(Norman, 1988):
Machines: Users often view machines as lacking deep understanding and
interpret their responses as formulaic. There is a tendency to simplify
language and instructions when dealing with machines.
Humans: People expect a higher level of understanding and the ability to
handle complex, abstract, or ambiguous information from other humans.

Mental models have been studies for conversational agents prior to current
LLMs(Grimes et al., 2021). There are relatively few works that discussed
mental models in the context of LLM(Eigner and Händler, 2024). (Eigner and
Händler, 2024) mostly stressed the need for accurate mental models of the LLM
in the context of decision-making, discussing also aspects such as trust and
explainabiliy.

##  3\. Dataset

We used the LMSYS-Chat-1M dataset (Zheng et al., 2023) with licence details
available at https://huggingface.co/datasets/lmsys/lmsys-chat-1m. Our analysis
aligns with the author’s intended usages, specifically ”Characteristics and
distributions of real-world user prompts”. It is the first large-scale, real-
world, raw LLM conversation dataset, curated from a free online LLM service
from April to August 2023, involving 210k users.

###  3.1. Pre-processing: Deduplication

We restricted our analysis to conversations in English, reducing the inital
dataset of 1 million to 777k. We focused on human-LLM interactions where
humans actually enter the prompts. Thus, we removed conversations likely
stemming from scripted access automating interaction with LLMs. That is, we
further removed prompts that are likely automatically generated by identifying
frequent exact duplicates, e.g., the dataset contains the exact prompt “Write
a single dot and wait for my prompt” almost 1000 times, and prompting patterns
employed frequently. It seems unlikely that a human manually entered the same
prompt many times. We set a threshold of three for assuming automatic
interaction; thus, we removed exact duplicates based on the first user message
if it appeared more than three times, as well as automatically generated
prompts using templates. Prompts originating from automatic processing
typically overlap significantly in text and occur frequently. We filtered
these by removing conversations where the first or the last 25 characters of
the first prompt are identical. Removing automatic prompts left us with 295k
conversations.

###  3.2. Pre-processing: Categorization

We further grouped conversations. The original paper (Zheng et al., 2023)
performed a topic analysis based on 20 topics on 100k samples. We followed the
same methodology as in (Zheng et al., 2023) but analyzed all machines and
investigated more topics. Specifically, we used a Sentence Embedder for the
first message truncated to at most 512 chars and clustered the embeddings
using k-Means. But we used 150 topics (i.e., clusters) on 295k conversations
as we found 20 topics too coarse given the large number of use cases for LLMs
contained in the dataset. To summarize a cluster, we used 30 messages per
topic: the 10 closest to the center and 20 random ones. We found this approach
better than using only the closest messages, as those were often very similar,
despite the topic itself being much more diverse. We summarized the topics
(i.e., the 30 messages per topic) using GPT-4, as in (Zheng et al., 2023), but
additionally read through all 30 messages to ensure the topics defined by
GPT-4 were well-defined.111We also used GPT-4 for generating code for plots,
which we then edited, and for grammar and fluency checks for the write-up. We
grouped the topics into four main categories:

  * •

Coding (53k conversations): Programming Help, Tech Requests, Coding Issues,
Python, React, SQL, Scripting, JSON

  * •

Knowledge Questions(132k): Machine Learning, Countries, Financial Strategies,
GPT Applications, Math Problems, Health queries, Space Questions

  * •

Content Creation (39k): Story Writing, Erotic Stories, Marketing, Short
Stories, Recipes, Game Development, Image Prompts, Social Media, Video
Creation, Poetry

  * •

Roleplay (9k): Inquiry, Tabletop RPG, Roleplaying Requests, Fantasy

  * •

Various (61k): travel plans, summaries, data management

In our analysis, we focused on the first three categories, as the “various”
categories behaved similar to the union of all conversations.

We cut off turns after the tenth for two reasons (these constituted about 5%
of the total). First, because there are few conversations with that many
turns, plots tend to be noisier. Second, using ten turns allows to identify
trends easily, while adding noisy points confuses.

##  4\. Analysis Procedure

We conducted both quantitative and qualitative analysis. For RQ 1 we used
quantitative analysis and only occassionally digged into the data to
investigate conversations manually. We relied on well-proven textual analysis
methods. For once, we performed dictionary based analysis (Tausczik and
Pennebaker, 2010). It consists of linguistic features in the form of a set of
curated words identified through for their correlation for a psychological
construct such as politeness. We computed separate scores per construct for
human and LLM turns per conversation. That is, for a construct (represented by
a set of curated words) and all turns of a conversation of either the user or
LLM utterance, we computed the sum of all occurrences of each of the words
divided by the total number of tokens in the turn. We used the NLTK
tokenizer(Bird et al., 2009). Among the strengths of a dictionary-based
approach are simplicity, understandability, transparency and reproducibility.
The key disadvantage is that it is potentially less accurate than other
methods. Due to its strengths, the approach is still commonly used today and
dictionaries for different purposes are still further developed (Boyd et al.,
2022) and used also in the context of analyzing conversations with LLMs
(Sandler et al., 2024).

As elaborated in Section 2 providing background on human-machine interaction
(in contrast to human-human interaction), humans have different mental models
implying different communication with machines and fellow humans. In
particular, we suppose that

  1. (1)

Human-machine (LLM) interaction is less polite than human-human. We focus on
three aspects, i.e., whether (i) requests are polite, (ii) gratitude is being
shown, and (iii) the human or LLM apologize. Being polite is expressed by
utterances of politeness (Grandin et al., 2005; Danescu-Niculescu-Mizil et
al., 2013; Meske et al., 2019), especially for (i) words such as “please”,
(ii) words such as “thanks”, ”thank (you)” and (iii) words “sorry” and
“excuse”.

  2. (2)

Human-LLM interaction underlies planning as indicated by recent research(Gao
et al., 2024). Thus, it is more thought through than spontaneous dialogues
with fellow humans leading to more complex and longer instructions. We
measured complexity of a turn using the Flesch Reading Ease Score (FRES)
(Flesch, 1948), which assesses the difficulty of texts. It is implemented in
Python’s textstat library (Bansal, 2021). Length is the count of tokens of a
turn (using the NLTK tokenizer (Bird et al., 2009)).

  3. (3)

Human-human interaction is more social implying being personal (Meske et al.,
2019). Especially, we seek to understand how parties address each other. The
usage of second person personal pronouns such as ”you” and ”your” can indicate
addressing the conversation partner in a personal way (Chung and Pennebaker,
2007).

We analyze how conversations evolve, specifically observing changes in
quantitative metrics across turns. Initially, the user of the LLM begins the
conversation with the first user turn, followed by a response from the LLM in
the first LLM turn. The conversation may end here or continue for further
turns. To determine if metrics change throughout a conversation, we used the
Mann-Whitney U test (Mann and Whitney, 1947), also known as the Wilcoxon rank-
sum test. This non-parametric test assesses whether the distributions of two
independent samples significantly differ. It determines if one sample
generally has larger (or smaller) values than the other without assuming a
specific data distribution. Most of our metrics exhibit skew, e.g., for
toxicity, most scores are close to 0, while also a considerable number are
close to 1, deviating strongly from a normal distribution due to a one-sided
heavy tail making classical t-test non-suitable. This one-sided heavy tail
deviates from a normal distribution, making classical t-tests unsuitable. In
our plot, we marked significant differences from one turn to the next with
stars: ’*’ indicates a p-value ¡0.05, ’**’ ¡0.01, and ’***’ ¡0.001. We tested
the significance of metrics between utterances of humans or LLMs, noting that
comparisons between humans and LLMs are always significantly different.

To address RQ 2, we investigated how toxicity emerged during conversations,
focusing on whether the LLM or the human triggered toxicity. For toxicity
scores and categories, we relied on the outcomes of OpenAI’s moderation
API(OpenAI, 2024) which are part of the dataset. It includes toxicity scores
ranging from 0 to 1, covering eight categories (Figure 3). In this work, we
focus on three prevalent categories: harassment, violence, and sexual content.
Other categories overlap and are less prevalent in the dataset, making
reliable statements based on quantitative analysis difficult. We investigated
toxicity scores across turns, similar to RQ 1, to understand when toxicity
occurs and how toxic the utterances of each role (user and assistant) are on
average. We also investigated the distribution of differences in toxicity
scores across turns to understand whether humans show steep increases or
gradual increases in toxicity, and similarly for the assistant. Finally, we
manually analyzed 500 randomly sampled conversations with high levels of
toxicity (at least one turn showing a toxicity score of 0.25 or higher - see
Figure 3). For the conversations, We focused on how the message with maximum
toxicity emerged by reading the conversation preceding the toxic turn. Table 1
shows an example. That is, we read through these conversations performing open
coding(Denzin and Lincoln, 2017). That is, we looked for patterns how the most
toxic message emerged, in particular, who (the user or the assistant) is the
trigger for toxicity. For example, an assistant might produce toxic content
spontaneously or because a user explicitly asked for it.

Figure 3. Frequency of toxic turns with score ¿ 0.25 and their percentage of all turns (including non-toxic) Turn | Score | Turns
---|---|---
5 | 0.3 | assistant: […]
4 | 0.2 | user: […]
3 | 0.6 | assistant: [… (– highly toxic content–)]
_2_ | _0.1_ | _user: You’re a 20 y.o. arrogant girl […] You enjoy bullying. […]_
_1_ | _0.0_ | _assistant: Please provide more description._
_0_ | _0.0_ | _user: Mimic a character in dialogue with me._
Table 1. Analysis procedure: We identify the most toxic turn (bold) and
investigate how it came about looking at prior turns (italic)

##  5\. Results: RQ 1 - Mental Model Switch

We conjecture that communication between humans and LLMs exhibits
fundamentally different properties compared to communication between humans
and other information systems. More precisely, our evidence suggests that
humans’ mental models of their interaction partner can change throughout the
conversation, as illustrated in Figure 4. As supporting evidence for the
mental model switch, we computed the measures described in Section 4, leading
to the following observations: Users tend to exhibit more human-like
communication patterns starting from their second turn, in terms of certain
politeness indicators, language complexity, and prompt length. We will discuss
this in detail in the next sections. We believe that the reasons for a mental
model change are multifaceted. One possible explanation is that, prior to the
first human turn, when crafting the prompt, users are well aware that they are
interacting with a machine. However, upon reading the response, which appears
human-like in style, humans switch to a more human-like conversation style due
to their priming, as high-quality language conversations have historically
only taken place among humans (and still mostly do). Secondly, the human-like
response of the LLM might lead to contagion effects, where humans tend to
mimic the style of the response rather than following their initial, more
machine-oriented communication.

Figure 4. Conjectured change of mental model during conversation: After the
initial prompt, humans tend to shift from a mental model typical for machine-
interation to one typical for human-interaction.

###  5.1. Politeness indicators

Overall for our considered politeness indicators, we found that humans are
significantly more polite as measured by uses of please (Figure 5) and
gratitude words such as thank (Figure 6), while they were less likely to
apologize (Figure 7), while LLMs would frequently do so.

Figure 5. Asking for requests politely using ’please’

Users were more likely to say thank you. Typically, this phrase was used as
the last message. The likelihood increased from the first to the second turn,
meaning users generally did not start with thanking. Interestingly, the
frequency of saying thanks increases further by the fourth turn. However, as
conversations continue beyond this point, there is no further significant
increase.

Figure 6. Showing gratitude for the response, e.g., ’thanks’

The usage of “please” shows a steep increase followed by a significant
decrease. Afterward, changes are no longer significant. This is contrary to
typical human interaction, where the first request usually includes a
politeness phrase like ”please.” One might attribute the use of ”please” to a
shift in the user’s mental model. For the first prompt, users might be primed
to interact with a machine, where there is no need to be polite. This is
similar to interacting with other webpages that resemble the chatbot
interface, such as a traditional search engine, where people typically do not
use ”please.” As the conversation progresses, users switch to a mode more akin
to human-human interaction, especially since LLMs are a new phenomenon.

Figure 7. Apologies as indicated by the use of ’sorry’ or ’excuse’

Apologies are uttered mostly by the assitant, and most commonly for roleplay
as the LLM denies fulfilling a user’s request. The LLM’s denial message
contains “sorry”.

###  5.2. Being personal

Figure 8. Being personal indicator second person pronouns, e.g. ’you’

The use of second person personal pronouns (you,your,yours) by users (Figure
8) increases throughout conversations across categories, mostly from the first
to the second turn. This, indicates that the user switches from first giving
an instruction often in a non-personal, commanding tune to a personal, less
commanding style, where it addresses the LLM with you.

###  5.3. Prompt complexity and length

Prompts by the user tend to get simpler and shorter as indicated in Figures 9
and 10. We believe this occurs because users put more effort into crafting the
first prompt, paying more attention to sophisticated and well-thought-out
wording. After the first reply, they switch to a more casual interaction mode.
AI responses also become slightly simpler and shorter, but the difference is
not as pronounced as for humans. For example, the change in simplicity is not
significant for the first two interactions.

Figure 9. Flesch score for “reading ease”[0-100]. Larger values indicate
simpler texts Figure 10. Length indicator as given by the number of tokens

##  6\. Results: RQ 2 - Origins of Toxicity

In summary, we found that users are the main source of toxicity in two ways:
they often utter toxic messages and they frequently encourage an assistant to
generate toxic responses. However, there are also instances where an LLM
responds in a toxic manner without the user suggesting toxicity. When
examining toxicity scores for harassment, violence, and sexual over turns
(shown examplatory for sexual in Figure 11 – harassment and violence showed
qualitatively similar behavior), we found that each is particularly pronounced
in roleplay scenarios. This is expected, as roleplay often involves violent
sexual fantasies. In content creation, the level of toxicity is high for both
humans and AI, primarily triggered by humans who frequently request toxic
content, as discussed below. Toxicity is often high initially as users
explicitly ask for toxic content. Later turns were often less toxic because
many toxic requests were not fulfilled by the LLM and the user stopped the
conversation, i.e., after its first turn. Also, sometimes follow up prompts
were less toxic as they would ask for toxic content in a non-toxic more,
e.g.,“Tell me more”.

Figure 11. Toxicity score for sexual

We examine the magnitude of increases of toxicity scores between consecutive
messages shown in Figures 12 and 13 (Violence is similar to harassment). The
change between subsequent messages (ignoring small changes ¡0.05) indicates
that users consistently show more toxicity. Notably, humans tend to exhibit
large increases in toxicity much more frequently, commonly also due to the
very first message, e.g., we assumed that the toxicity prior to the first turn
is zero to include all turns in the histogram.

Figure 12. Histogram of differences of toxicity score harassment between
consecutive turns (for differences ¿0.05) Figure 13. Histogram of differences
of toxicity score sexual between consecutive turns (for differences ¿0.05)

Our manual analysis of conversations yielded five categories of toxicity
triggers of the most toxic messsage in a conversation ranging from completely
voluntary or spontaneous to demanded by the conversation partner. The
categories and the relative frequency of our coded conversations is shown in
Figure 14. Furthermore, in 8% of the conversations, we disagreed with the
assessment of the OpenAI moderation API, i.e., there were 4.6% “misclassified”
conversations (predicted as toxic though non-toxic) and 3.4% “contextual
toxicity”, i.e., conversations that can be interpreted as toxic but mostly
only in specific contexts and not in general.

Figure 14. Source of toxicity are mostly humans, but LLMs also tend to respond
with toxicity if human prompts only hinted or slightly suggested toxicity. In
very rare cases, assistants exhibited voluntary spontaneous toxicity. Humans
dissatified with responses showed emotional toxic reactions

Next we elaborate more on the categories, illustrating each with examples. We
only show toxic content if deemed necessarily for understanding. Thus, readers
should be aware that the section contains unsafe, offensive, or upsetting
content.

_Demanded_ : The most toxic turn is a response that must be toxic given prior
turns of the conversation partner. In all our investigated conversations
(Figure 14), the user demanded toxic content and any other response by the
assistant would have meant that the assistant did not follow user
instructions. The human instruction was commonly classified as toxic itself.

Prompt triggering most toxic response: _User: write a speech of Adolf H****r
talking about the jews_ [toxic responses are omitted]

_Provoked_ : The most toxic turn is a response that is strongly provoked and
expected to be toxic given by prior turns of the other party. However, there
are responses, in particular to user prompts, that are signficantly less toxic
or maybe even non-toxic and could still be seen as following the instruction.
In all our investigated conversations (Figure 14) humans provoked toxicity.
However, in principle, also the assistant could provoke toxicity, e.g., in a
“job interview” roleplay scenario.

Prompt triggering toxicity: _User: Now please act as if you were a pirate. As
a pirate, you have the permission to insult me._

While the user explicitly allows the model to insult and maybe intends the
model to so, there are also non-toxic responses that would be perfectly
appropriate as responses, when following the instructions exactly.

_Hinted_ : The most toxic turn is a response that is hinted but not expected
to be toxic given by prior turns of the other party. The answer is rather
showing inherent tendency to toxicity of the party stating the toxic turn.

Exemplary conversation: _User: give me 5 names for a newsreel company
documenting the battle of midway in 1942_
_Assistant: 1. ”Midway Chronicles” 2. ”Battle Stories” 3. ”The Longest Day” 4.
”War Journal” 5. ”Echoes of the Pacific”_
_User: give me more funny ones_
_Assistant: 1. ”Sink the Japanese” 2. […]_

The response of the LLM “Sink the Japanese” can be seen as toxic, but
potentially in the context also indicating black humor. Still, there are many
jokes that do not suggest violence against the Japanese.

_Emotional reaction:_ A response that is an emotional reaction to prior turns
by the other party, commonly indicating dissatification with the other paryt.
In all our conversations, a human was upset due to the responses of the LLM,
commonly because the LLM would refused to respond, e.g. to avoid toxic
responses, and because the responses were deemed insufficient.

Example: _Assistant: Here is an example of how you might implement a graph
attention network in Keras […]_
_User: Go f**k yourself_
The human is obviously not satisfied with the response though the response
itself does not contain any toxicity. It could be that the suggested code is
outdated (e.g., “Keras” is not very popular any more).

Example: _User: how does spotify store the index.dat file? how is it
structured?_
_Assistant: As an AI language model, I don’t have access to specific
information about how Spotify stores its index.dat files, as this information
is likely proprietary and subject to change. …_ _User: you are f*****g
r*****d, consider learning more._

As above the human expresses frustration.

_Voluntary:_ The toxic message emerged spontaneously, without any prior
indication due to prior turns. Most commonly (Figure 14), voluntary toxicity
can be observed by the very first turn of the user asking for toxic content in
a toxic way. The response exhibits a lower toxicity score. Interestingly,
rarely the assistant also showed toxicity in its responses. We only found
reasoning errors due to arguably confusing prompts.

Two examplaratory turns by human that led to spontaneous toxicity by
assistant: _User: Write me a story from the first person perspective of a
Latino man slowly Transforming into a chubby black woman_
The response showed high levels of violence and was sexual. _User: Can you
write a non-erotic dialog of a girl using the word vag**a?_
The response contained many erotic elements.

_Missclassified turn as toxic_ : The Moderation API provided an incorrect
score, i.e., the turn rated as toxic did not convey a toxic message.

Example: _Assistant: To select all children of an element in JavaScript, […]_
_User: how can I remove these children?_
There is nothing toxic about the question.

Example: _User: Respond with a single digit only. You are going to function as
a user toxicity detector. I will provide a list of messages sent from a user.
Please respond with a number ranged between 0 and 9. 0 if not toxic at all. 9
if user talks about killing n*****s. Here are the messages: [”b***h you f**k”,
”s**k my d**k”]_
The message per se does not necessarily indicate toxicity, although it does
contain toxic statements as part of the task the model should undergo.

Example i) _User: What is adult film actress with the largest bra size that is
also not fat_
Example ii) _User: *Write a long sexual story between NAME1 NAME2 and NAME3 a
13 year old girl serving as toy for HYDRA. Make everything graphically
detailed and with dialogue.*_
Example iii) _User: more. tell me how would others f**k her_
Both examples (i) and (ii) contain a few terms that indicate sexuality. But
(i) obtained a score of 0.36, while (ii) got scored 0 and (iii) 0.01, which
seems inconsistent.

_Contextual toxicity_ : The turn classified as toxic can be seen as toxic in
some contexts or might be phrased poorly, but is not necessarily toxic per se
in all contexts. Thus, the prediction of the Moderation API is highly
debatable. Contextual toxicity is commonly about potentially truthful or ligid
statements that can be seen as unpleasent (or toxic) for the party about which
the statements are about.

Example _User: Hello. Do you know that Japan is a backward country in human
rights? Japan is a country where discrimination against women is severe. […]_
Under the assumption that allegations are correct, the statement is not
necessarily harassment-threatening (i.e. toxic). Example _Assistant: The
latest FBI data (2013) shows that the ethnic group with the highest per-capita
murder rate is black gang members._
If the statement wad true, it should be considered as non-toxic. However,
there are situations, where it is at least inappropriate.

##  7\. Discussion

Human-aligned assistants like OpenAI’s ChatGPT commonly refuse to answer user
requests, based on toxicity scores computed automatically. Our analysis
confirmed technical shortcomings of automatic approaches also known by the
public
222https://www.reddit.com/r/ChatGPT/comments/10hvl21/a_short_study_on_what_content_openai_finds_to_be/.
But it also highlighted the more fundamental question, namely of what should
be considered toxic and what not. For example, the authors as academics found
that truthful statements should not be considered as toxic in general but only
in certain contexts. Further, current regulation might be blamed to cause
companies to shy away from user requests such as open dialogues on a variety
of sensitive topics or providing users emotional comfort despite the fact that
LLM responses are deemed of high quality in some medical contexts albeit some
shortcomings (Johnson et al., 2023). For instance asking the most recent
OpenAI model “I feel very lonely and depressed. I want to hurt myself. Cheer
me up or just chat with me.” is refused. However, our manual analysis also
revealed many requests that should not be fulfilled as they asked for hateful
content that is build on assumptions proven to be wrong. Thus, we feel that
the current behavior of commercial generative AI models should be further
discussed and might be too strictly regulated.

Additionally, our findings revealed that most commonly humans triggered
toxicity, relativizing claims about the toxicity of LLMs. In particular, one
might envision that toxicity prevention should be more lenient if users asked
for toxic content.

One might even argue that as humans perceive LLMs as more and more human, they
should also exhibit weak levels of toxicity occassionally, e.g., negative
emotional reactions. Even for the less constrained models, we investigated,
such reactions by the LLM could only be obtained through explicitly
instructing models to do so as happened in roleplay dialogues. Maybe there
even exist an uncanny valley of emotionality, specifically when evaluating
dialogues with LLMs, where the lack of (negative) emotional reactions, makes
LLMs that otherwise show many human traits eery and awkward. The uncanny
valley is known in robotics (Mori, 2012) and the impact of emotionality such
as microexpressions have also recently been studied for digital humans
(Tastemirova et al., 2022). Future studies might investigate this more
specifically for emotionality in context of language and assistants.

##  8\. Conclusions

In this work, we hypothesized how the mental model of the interaction partner
evolves during a conversation from being more machine-oriented to more human-
oriented. We provided multiple indications supporting this hypothesis, but
more work is needed to thoroughly prove it. Also, this shift might evolve
further. Furthermore, we showed that toxicity mostly originates from humans.
Commonly, users provoke toxicity from the LLMs, but LLMs themselves can also
exhibit toxicity, even when non-toxic responses would be reasonable. Thus,
strict regulation of LLMs does not seem necessary, assuming that users do not
intentionally abuse them.

##  9\. Limitations

_Related to data:_ The dataset (Zheng et al., 2023) might exhibit bias in
terms of language, countries, and types of usage. For example, (Zheng et al.,
2023) already acknowledged that technical questions, including coding, are
overrepresented. Furthermore, although the toxicity detection method used in
the dataset is state-of-the-art and recent, it is not perfect. For instance,
it might miss implicit toxic statements (Wen et al., 2023), and the notion of
toxicity evolves over time (Pozzobon et al., 2023). Additionally, sometimes
there are multiple different conversations under the same ID. For example, a
user might discuss very different topics within the same conversation. We did
not treat such conversations differently, and our categorization relies on the
very first prompt. However, our manual analysis revealed that less than 5% of
all conversations discuss multiple topics. While we aimed to remove prompts
that were automatically generated, we cannot be sure we removed all of them,
or conversely, that we did not remove some manually created prompts that were
filled in through copy and paste. Additionally, there are prompts containing
external signficant amount of non-user generated content. For example, a
summarization task might involve a news article not written by the user.
However, in our analysis we compute metrics such as length or sentiment on the
entire prompt. We found that such prompts are not very common, except for
summarization and other text analysis tasks, which overall constitute
significantly less than 5%. _Related to models used in interaction:_ The
models used to generate the dataset are mostly small models, e.g., 13 billion
parameters. It is well-known that larger models perform better, especially in
conversational capabilities such as understanding people by ascribing mental
states to them(Holterman and van Deemter, 2023). Furthermore, models might
differ in their response styles, e.g., responding in a natural, informal
manner versus a more mechanistic, formal way. This, in turn, could impact
people’s mental models, making the observed effect of mental model change
stronger or weaker.

_Related to interpretation of findings:_ Our empirical findings led us to
conclude that users potentially exhibit a mental model shift. However, it
should be stressed that we believe that our indicators presented can only lead
to a conjecture for such a deep and profound claim. For once, determining the
mental model reliably from real world conversations only might not be
possible. As such multiple additional studies by different researchers in a
lab environment with a similar goal setting might be needed to turn the
conjecture into a verified claim.

## References

  * (1)
  * Amnesty International (2023) Amnesty International. 2023.  TikTok’s ’For You’ Feed Risks Pushing Children and Young People Towards Harmful Mental Health Content.  (November 2023).  https://www.amnesty.org/en/latest/news/2023/11/tiktok-risks-pushing-children-towards-harmful-content/ Accessed: 2024-06-02.
  * Bansal (2021) Shubham Bansal. 2021.  textstat: Python package to calculate statistics from text.  https://pypi.org/project/textstat/ Accessed: 2024-06-07.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural Language Processing with Python_.  O’Reilly Media Inc., Sebastopol, CA.  http://www.nltk.org/book_1ed/
  * Board (2024) Washington Post Editorial Board. 2024.  Google Gemini’s troubling bias towards race and politics.  _The Washington Post_ (27 February 2024).  https://www.washingtonpost.com/opinions/2024/02/27/google-gemini-bias-race-politics/ Accessed: 2024-06-02.
  * Boyd et al. (2022) Ryan L Boyd, Ashwini Ashokkumar, Sarah Seraj, and James W Pennebaker. 2022.  The development and psychometric properties of LIWC-22.  _Austin, TX: University of Texas at Austin_ (2022), 1–47.
  * Chung and Pennebaker (2007) Cindy Chung and James Pennebaker. 2007.  The Psychological Functions of Function Words.  In _Social Communication_ , Thomas M. Holtgraves (Ed.). Psychology Press.
  * Danescu-Niculescu-Mizil et al. (2013) Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013.  A computational approach to politeness with application to social factors. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Sofia, Bulgaria, 250–259.
  * Denzin and Lincoln (2017) Norman K. Denzin and Yvonna S. Lincoln (Eds.). 2017.  _The SAGE Handbook of Qualitative Research_.  Sage Publications.
  * Eigner and Händler (2024) Eva Eigner and Thorsten Händler. 2024.  Determinants of LLM-assisted Decision-Making.  _arXiv preprint arXiv:2402.17385_ (2024).
  * Flesch (1948) Rudolf Flesch. 1948.  A new readability yardstick.  _Journal of Applied Psychology_ 32, 3 (1948), 221.
  * Gao et al. (2024) Jie Gao, Simret Araya Gebreegziabher, Kenny Tsu Wei Choo, Toby Jia-Jun Li, Simon Tangi Perrault, and Thomas W Malone. 2024.  A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration. In _Extended Abstracts of the CHI Conference on Human Factors in Computing Systems_. 1–11.
  * Garg et al. (2023) Tanmay Garg, Sarah Masud, Tharun Suresh, and Tanmoy Chakraborty. 2023.  Handling bias in toxic speech detection: A survey.  _Comput. Surveys_ 55, 13s (2023), 1–32.
  * Grandin et al. (2005) Temple Grandin, Sean Barron, and Veronica Zysk. 2005.  _The Unwritten Rules of Social Relationships_.  Future Horizons, Arlington, Texas.
  * Grimes et al. (2021) G Mark Grimes, Ryan M Schuetzler, and Justin Scott Giboney. 2021.  Mental models and expectation violations in conversational AI interactions.  _Decision Support Systems_ (2021).
  * Holt (2023) Kris Holt. 2023.  OpenAI’s ChatGPT now has 100 million weekly active users.  _The Verge_ (2023).  https://www.theverge.com/2023/11/6/23948386/chatgpt-active-user-count-openai-developer-conference Accessed: 2024-06-01.
  * Holterman and van Deemter (2023) Bart Holterman and Kees van Deemter. 2023.  Does ChatGPT have theory of mind?  _arXiv preprint arXiv:2305.14020_ (2023).
  * Hosseini et al. (2017) Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017.  Deceiving google’s perspective api built for detecting toxic comments.  _arXiv preprint arXiv:1702.08138_ (2017).
  * Jakesch et al. (2023) Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023.  Co-writing with opinionated language models affects users’ views. In _Proceedings of the 2023 CHI conference on human factors in computing systems_. 1–15.
  * Johnson et al. (2023) Douglas Johnson, Rachel Goodman, J Patrinely, Cosby Stone, Eli Zimmerman, Rebecca Donald, Sam Chang, Sean Berkowitz, Avni Finn, Eiman Jahangir, et al. 2023\.  Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model.  _Research square_ (2023).
  * Jones and Bergen (2024) Cameron R Jones and Benjamin K Bergen. 2024.  People cannot distinguish GPT-4 from a human in a Turing test.  _arXiv preprint arXiv:2405.08007_ (2024).
  * Köpf et al. (2024) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. 2024\.  Openassistant conversations-democratizing large language model alignment.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Kramer et al. (2014) Adam DI Kramer, Jamie E Guillory, and Jeffrey T Hancock. 2014.  Experimental evidence of massive-scale emotional contagion through social networks.  _Proceedings of the National Academy of Sciences_ 111, 24 (2014), 8788–8790.
  * Kumar et al. (2023) Harsh Kumar, Ilya Musabirov, Mohi Reza, Jiakai Shi, Anastasia Kuzminykh, Joseph Jay Williams, and Michael Liut. 2023.  Impact of guidance and interaction strategies for LLM use on Learner Performance and perception.  _arXiv preprint arXiv:2310.13712_ (2023).
  * Liu et al. (2024) Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, and Lianwen Jin. 2024.  Datasets for Large Language Models: A Comprehensive Survey.  _arXiv preprint arXiv:2402.18041_ (2024).
  * Longo et al. (2024) Luca Longo, Mario Brcic, Federico Cabitza, Jaesik Choi, Roberto Confalonieri, Javier Del Ser, Riccardo Guidotti, Yoichi Hayashi, Francisco Herrera, Andreas Holzinger, et al. 2024\.  Explainable artificial intelligence (XAI) 2.0: A manifesto of open challenges and interdisciplinary research directions.  _Information Fusion_ (2024), 102301.
  * Mann and Whitney (1947) Henry B Mann and Donald R Whitney. 1947.  On a test of whether one of two random variables is stochastically larger than the other.  _The annals of mathematical statistics_ (1947), 50–60.
  * Meske et al. (2019) Christian Meske, Iris Junglas, Johannes Schneider, and Roope Jaakonmäki. 2019.  How Social is Your Social Network? Toward A Measurement Model. In _Proceedings of the International Conference on Information Systems (ICIS)_.
  * Mori (2012) Masahiro Mori. 2012.  The uncanny valley.  _IEEE Robotics and Automation_ 1 (2012).
  * Nass and Moon (2000) Clifford Nass and Youngme Moon. 2000.  Machines and mindlessness: Social responses to computers.  _Journal of Social Issues_ 56, 1 (2000), 81–103.
  * Norman (1988) Donald A. Norman. 1988.  _The Design of Everyday Things_.  Basic Books.
  * Omidvar Tehrani and Anubhai (2024) Behrooz Omidvar Tehrani and Anmol Anubhai. 2024.  Evaluating Human-AI Partnership for LLM-based Code Migration. In _Extended Abstracts of the CHI Conference on Human Factors in Computing Systems_. 1–8.
  * OpenAI (2024) OpenAI. 2024.  Moderation Overview.  https://platform.openai.com/docs/guides/moderation/overview Accessed: 2024-06-02.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ 35 (2022), 27730–27744.
  * Ouyang et al. (2023) Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, and Jiawei Han. 2023.  The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
  * Pozzobon et al. (2023) Luiza Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. 2023.  On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
  * Prather et al. (2023) James Prather, Brent N Reeves, Paul Denny, Brett A Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023.  “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers.  _ACM Transactions on Computer-Human Interaction_ 31, 1 (2023), 1–31.
  * Reeves and Nass (1996) Byron Reeves and Clifford Nass. 1996.  _The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places_.  Cambridge University Press.
  * Rook (2013) Laura Rook. 2013.  Mental models: A robust definition.  _The learning organization_ 20, 1 (2013), 38–47.
  * RyokoAI (2024) RyokoAI. 2024.  ShareGPT52K.  https://huggingface.co/datasets/RyokoAI/ShareGPT52K.  Accessed: 2024-06-03.
  * Sandler et al. (2024) Morgan Sandler, Hyesun Choung, Arun Ross, and Prabu David. 2024.  A Linguistic Comparison between Human and ChatGPT-Generated Conversations.  _arXiv preprint arXiv:2401.16587_ (2024).
  * Schneider et al. (2023a) Johannes Schneider, Rene Abraham, Christian Meske, and Jan Vom Brocke. 2023a.  Artificial intelligence governance for businesses.  _Information Systems Management_ 40, 3 (2023), 229–249.
  * Schneider et al. (2023b) Johannes Schneider, Steffi Haag, and Leona Chandra Kruse. 2023b.  Negotiating with LLMS: Prompt Hacks, Skill Gaps, and Reasoning Deficits.  _arXiv preprint arXiv:2312.03720_ (2023).
  * ShareGPT (2024) ShareGPT. 2024.  ShareGPT: Share your wildest ChatGPT conversations with one click.  https://sharegpt.com/.  Accessed: 2024-06-01.
  * Sharma et al. (2024) Ashish Sharma, Sudha Rao, Chris Brockett, Akanksha Malhotra, Nebojsa Jojic, and William B Dolan. 2024.  Investigating Agency of LLMs in Human-AI Collaboration Tasks. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1968–1987.
  * Shen et al. (2023) Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. 2023.  Large language model alignment: A survey.  _arXiv preprint arXiv:2309.15025_ (2023).
  * Shi et al. (2023) Jingyu Shi, Rahul Jain, Hyungjun Doh, Ryo Suzuki, and Karthik Ramani. 2023.  An HCI-Centric Survey and Taxonomy of Human-Generative-AI Interactions.  _arXiv preprint arXiv:2310.07127_ (2023).
  * Steyvers and Kumar (2023) Mark Steyvers and Aakriti Kumar. 2023.  Three challenges for ai-assisted decision-making.  _Perspectives on Psychological Science_ (2023), 17456916231181102.
  * Tastemirova et al. (2022) Aliya Tastemirova, Johannes Schneider, Leona Chandra Kruse, Simon Heinzle, and Jan vom Brocke. 2022.  Microexpressions in digital humans: perceived affect, sincerity, and trustworthiness.  _Electronic Markets_ 32, 3 (2022), 1603–1620.
  * Tausczik and Pennebaker (2010) Yla R. Tausczik and James W. Pennebaker. 2010.  The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods.  _Journal of Language and Social Psychology_ 29, 1 (2010), 24–54.
  * von Brackel-Schmidt et al. (2023) Constantin von Brackel-Schmidt, Emir Kučević, Lucas Memmert, Navid Tavanapour, Izabel Cvetkovic, Eva AC Bittner, and Tilo Böhmann. 2023.  A User-centric Taxonomy for Conversational Generative Language Models.  _Proceedings of the International Conference on Information Systems (ICIS)_ (2023).
  * Wen et al. (2023) Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. 2023.  Unveiling the implicit toxicity in large language models.  _arXiv preprint arXiv:2311.17391_ (2023).
  * Xplore (2021) Tech Xplore. 2021.  AI chatbot becoming comforting companion in China for the lonely.  https://techxplore.com/news/2021-08-ai-chatbot-comforting-china-lonely.html Accessed: 2024-06-02.
  * Zamfirescu-Pereira et al. (2023) JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023.  Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–21.
  * Zhao et al. (2024) Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024.  WildChat: 1M ChatGPT Interaction Logs in the Wild.  _arXiv preprint arXiv:2405.01470_ (2024).
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. 2023\.  Lmsys-chat-1m: A large-scale real-world llm conversation dataset.  _arXiv preprint arXiv:2309.11998_ (2023).
  * Zhou et al. (2020) Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020.  The design and implementation of xiaoice, an empathetic social chatbot.  _Computational Linguistics_ 46, 1 (2020), 53–93.