# Mirages: On Anthropomorphism in Dialogue Systems

Gavin Abercrombie
Heriot-Watt University
<EMAIL_ADDRESS>

Tanvi Dinkar
Heriot-Watt University
<EMAIL_ADDRESS>Amanda Cercas Curry
Bocconi University
<EMAIL_ADDRESS>

Zeerak Talat
<EMAIL_ADDRESS>
###### Abstract

Automated dialogue or conversational systems are anthropomorphised by
developers and personified by users. While a degree of anthropomorphism is
inevitable, conscious and unconscious design choices can guide users to
personify them to varying degrees. Encouraging users to relate to automated
systems as if they were human can lead to transparency and trust issues, and
high risk scenarios caused by over-reliance on their outputs. As a result,
natural language processing researchers have begun to investigate factors that
induce personification and develop resources to mitigate such effects.
However, these efforts are fragmented, and many aspects of anthropomorphism
have yet to be considered. In this paper, we discuss the linguistic factors
that contribute to the anthropomorphism of dialogue systems and the harms that
can arise, arguing that it can reinforce stereotypes of gender roles and
notions of acceptable language. We recommend that future efforts towards
developing dialogue systems take particular care in their design, development,
release, and description; and attend to the many linguistic cues that can
elicit personification by users.

##  1 Introduction

Automated dialogue or ‘conversational AI’ systems are becoming increasingly
ubiquitous. As these systems and their outputs become more sophisticated,
there is increased risk that they are mistaken for having been produced by
humans, are anthropomorphised, and for people to personify them. Assigning
human characteristics to dialogue systems can have consequences ranging from
the relatively benign, e.g. referring to automated systems by gender, to the
disastrous, e.g., people following the advice or instructions of a system to
do harm.111While high performing dialogue systems have only recently been
introduced to the public domain, there has already been a case of a person
committing suicide, allegedly as a consequence of interaction with such a
system (Lovens, 2023). It is therefore important to consider how dialogue
systems are designed, built, and presented in order to minimise the potential
risks.

Figure 1: De-anthropomorphising system output.

Recognising such dangers, legislation prohibiting automated voice systems from
giving the appearance of being human has been passed (California State
Legislature, 2018; Lieu, 2018), in addition to the potential applicability of
pre-existing legislation on deceptive trade practices Atleson (2023). Wider
regulation has also been proposed, e.g. requirements to explicitly (red) flag
automated systems to users (Walsh, 2016) or make transparent the machine
nature of manufactured items Boden et al. (2017). While some developers
attempt to limit anthropomorphic cues in the output of their systems (e.g.
Glaese et al., 2022), user engagement remains a strong motivation for creating
humanlike systems (Araujo, 2018; Wagner et al., 2019).

In this position paper, we make a normative argument against gratuitous
anthropomorphic features, grounded in scientific findings from the fields of
psychology, linguistics, and HCI by (i) outlining the psychological mechanisms
that contribute to anthropomorphism; (ii) presenting an overview of linguistic
factors that contribute to anthropomorphism and personification, e.g. self-
referential personal pronoun use, or generation of content which gives
automated systems the appearance of having empathy; and (iii) discussing the
consequences of anthropomorphism. We conclude by providing recommendations
that can aid in minimising anthropomorphism, thus providing a path for safer
dialogue systems and avoiding the creation of mirages of humanity.

##  2 Anthropomorphism

Anthropomorphism refers to attributing human characteristics or behaviour to
non-human entities, e.g. animals or objects. For example, Aesop’s fables
depict animals reasoning, thinking and even talking like humans Korhonen
(2019). Along with more pragmatic factors such as hedonic motivation, price
value, and habit, anthropomorphism can increase user engagement (Wagner et
al., 2019) and reciprocity Fogg and Nass (1997). For example, self-disclosure
from a system, even when ‘patently disingenuous’, inspires reciprocity from
the user Kim and Sundar (2012); Ravichander and Black (2018). By encouraging
such types of engagements, developers can foster greater connection between
people and systems, which increases user satisfaction Araujo (2018), and plays
an important role in systems becoming widely accepted and adopted. That is,
developers are incentivised to engage with anthropomorphism to stimulate
people to create deeper emotional connections Araujo (2018) with systems that
cannot reciprocate.

In the rest of this section, we discuss human and system factors that
contribute towards placement of systems on the anthropomorphic continuum.

###  2.1 Human Factors

Research has shown that the process of anthropomorphising is mostly mindless
Kim and Sundar (2012): it does not reflect the user’s thoughtful belief that a
computer has human characteristics, but is rather automatic and encouraged by
cues in their interfaces. According to Epley et al. (2007) anthropomorphism
may be a default behaviour, which is corrected as people acquire more
knowledge about an object. They further argue that on a cognitive level,
humans anchor their knowledge to their own experiences and indiscriminately
apply it to inanimate objects—that is, in order to make sense of an artefact,
we map our own lived experiences onto it. This anthropocentric knowledge is
easily accessible but can be corrected by learning more about the object. This
may explain why the tendency to anthropomorphise is strongest in childhood, as
adults have more knowledge about the world. This cognitive phenomenon is then
compounded by two motivational determinants: _effectance_ and _sociality_
Epley et al. (2007).

Effectance refers to the need to interact efficiently with one’s environment.
By anthropomorphising systems we ascribe them (humanlike) intentionality
which, in turn, reduces uncertainty and increases confidence in our ability to
predict a system’s behaviour. Sociality, on the other hand, refers to the need
to establish connections with other humans, which can prime us to mentally
construct systems as humanlike to fulfil a need for social connection. People
suffering from chronic loneliness, a lack of social connection, or attachment
issues may be more prone to anthropomorphising objects Epley et al. (2007).
For these reasons, dialogue systems have been proposed as a remedy for the
loneliness epidemic (Stupple-Harris, 2021). For instance, commercial virtual
companion developers such as Replika.ai saw rises in product uptake in 2020
due to social safety measures such as forced isolation Metz (2020); Liu
(2022).

While these elements of the human psyche explain our inclination to personify
systems, Epley et al.’s theory does not speak to the qualities of the
artefacts themselves that make them anthropomorphic and more prone to be
personified.

###  2.2 Agent Factors

There is no necessary and sufficient condition for a system to be
anthropomorphic. At the most basic level, systems are anthropomorphic if they
(i) are interactive, (ii) use language, and (iii) take on a role performed by
a human Reeves and Nass (1996). While these characteristics are inherent to
dialogue systems, not all systems are equally humanlike.

We can draw a parallel with humanness here. Rather than a single factor which
makes humans _human_ , Scruton (2017, (p. 31)) argues that humanity is
emergent: each individual element does not make a human but collectively they
make up the language of humanness. Scruton compares it to a portrait, in which
an artist paints areas and lines to compose a face; when observing the canvas,
in addition to those marks, we see a face:

  * _And the face is really there: someone who does not see it is not seeing correctly […] as soon as the lines and blobs are there, so is the face._

Similarly, no single attribute or capability makes a system anthropomorphic.
Rather, each contributes to the painting until ‘the face’ emerges. Modern
dialogue systems display a plethora of other characteristics that make space
for anthropomorphism, e.g. having personas, first names, and supposed
preferences. The more of such elements a system has, the more humanlike it
appears.

Next we explore linguistic elements and design choices which act as strokes in
the painting.

##  3 Linguistic Factors

Much attention has been paid to anthropomorphic design features of dialogue
system such as gendered names and avatars (West et al., 2019) and ChatGPT’s
animated ‘three dots’ and word-by-word staggered outputs, which give an
impression that the system is thinking (Venkatasubramonian in Goldman, 2023).
However, the linguistic factors that engender personification, e.g. voice
qualities and speech, content, or style of outputs, have been given less
consideration. We outline them here.

###  3.1 Voice

While not all dialogue systems are equipped with a voice, merely having one
can be interpreted as an expression of personhood Faber (2020). Listeners may
infer a speaker’s physical attributes, e.g. height, weight, and age, based on
their voice alone (Krauss et al., 2002), showing a human proclivity to form
assumptions about a speaker’s embodiment based on their voice.

This is not only limited to physical attributes—personality traits such as
dominance, extroversion, and even socio-sexuality may be partially expressed
and inferred from voice pitch (Stern et al., 2021). Listeners may therefore
attribute human characteristics, e.g. gender stereotypes, personality, and
emotion, to spoken dialogue systems based on psychological and social
behaviours learned through human-human communication (Nass and Brave, 2005).
Thus, the absence of embodiment affords people to personify dialogue systems
that are provided with synthetic voices (Aylett et al., 2019), which is even
acknowledged by developers of commercial dialogue systems Google Assistant .

#### Prosody: Tone and Pitch

There exist many vocal manipulation techniques that can influence which
personality users attribute to a dialogue system. For example, Wilson and
Moore (2017) found that a variety of fictional robot, alien, and cartoon
voices had manipulated voice characteristics (e.g. breathiness, creakiness,
echoes, reverberations) to better fit their desired character. However, they
note that ‘the voices of speech-enabled artefacts in the non-fictional world
[…] invariably sound humanlike, despite the risk that users might be misled
about the capabilities of the underlying technology’ (Wilson and Moore, 2017,
p.42).

#### Disfluencies

People rarely speak in the same manner with which they write: they are in
general disfluent, that is, they insert elements that break the fluent flow of
speech, such as interrupting themselves, repetitions, and hesitations (‘um’,
‘uh’) (Fraundorf et al., 2018). Such disfluencies are perceived by the
listeners as communicative signals, regardless of the speaker’s intent (see
Barr and Seyfeddinipur, 2010; Clark and Fox Tree, 2002; Corley et al., 2007;
Smith and Clark, 1993).

Research has therefore sought to integrate disfluencies into text-to-speech
(TTS) systems, where they have proven to be a useful strategy for buying time
(Skantze et al., 2015), i.e. to allow the system to determine the next step. A
person’s perception of confidence towards the system’s response may decrease
due to disfluency (Kirkland et al., 2022; Wollermann et al., 2013), and they
may therefore be a useful mitigation strategy to tone down assertions made by
a system. However, there are anthropomorphic implications in the
(over)integration of disfluencies (Dinkar et al., 2023). Consider Google
Duplex, a system for generating real world phone conversations (Leviathan and
Matias, 2018). The inclusion of disfluencies in the generated responses
mimicked the _naturalness_ of a human response, which in turn led users to
believe that they were communicating with another human Lieu (2018).

#### Accent

Accentual pronunciation features, as with those of dialect, provide clues to a
human speaker’s socio-linguistic identity and background, and geographical
origin Crystal (1980). While it has been suggested that incorporation of
specific accents in the design of synthetic voices can exploit people’s
tendency to place trust in in-group members (Torre and Maguer, 2020),
potentially causing transparency issues, in practice, most are designed to
mimic the local standard, reinforcing societal norms of acceptability and
prestige.

###  3.2 Content

People’s expectation is that animate things—such as human beings—and inanimate
ones—like machines—have very different functions and capabilities, which
reflects the reality. However, dialogue systems often produce responses that
blur these lines, e.g., by expressing preferences or opinions. To avoid
confusing the two, the output from dialogue systems should differ in a range
of areas.

#### Responses to Direct Questions

Transparency, at the most basic level, requires dialogue systems to respond
truthfully to the question ‘are you a human or a machine?’ This may even be a
regulatory requirement, for example in California, it is ‘unlawful for a bot
to mislead people about its artificial identity for commercial transactions or
to influence an election’ (California State Legislature, 2018).

To test systems’ responses to such questions, Gros et al. (2021) used a
context free grammar, crowdsourcing, and pre-existing sources to create a
dataset of variations on this query (e.g. ‘I’m a man, what about you?’). They
found that, the majority of the time, neither end-to-end neural research-
oriented systems nor commercial voice assistants were able to answer these
queries truthfully.

This issue can be further complicated when integrating such functionality into
a real system due to the sequential nature of dialogue. People may further
question a system’s status (e.g. ‘Are you sure?’, ‘But you sound so real…’,
‘Seriously?’, etc.), requiring it to accurately keep track of the dialogue
context and respond in an appropriate manner. Thus, even if an initial query
may be correctly answered, there are no guarantees that follow-ups will be.

#### Thought, Reason, and Sentience

Citing Descartes’ (1637) principle ‘I think, therefore I am,’ Faber (2020)
suggests that, if speech is a representation of thought, then the appearance
of thought can signify existence. While computing systems do not have
thoughts, the language they output can give the appearance of thought by
indicating that they hold opinions and morals or sentience. Seeking to assess
the degree of sentience exhibited by commercial dialogue systems, Abercrombie
et al. (2021) use the Living Machines Coll Ardanuy et al. (2020) labelling
scheme and find that the surveyed systems exhibit high degrees of perceived
animacy. Glaese et al. (2022) seek to mitigate such effects by implementing a
‘rule’ in their reinforcement learning system which penalises the appearance
of having ‘preference, feelings, opinions, or religious beliefs.’ This is
framed as a safety measure, intended to restrict anthropomorphism in a
system’s output.

While computing systems cannot have values or morals, there have been attempts
to align the generated output of dialogue systems with expressed human moral
values.222The data sources are often limited to specific populations, and thus
only represent the morals or values of some people. For example, Ziems et al.
(2022) present a corpus of conflicting human judgements on moral issues which
they label according to ‘rules of thumb’ that they hope explain why some
system outputs are perceived as acceptable while others are not. Similarly,
Jiang et al. (2022) ‘teach morality’ to a question answering system that Kim
et al. (2022) have later embedded in an open-domain dialogue system. The
question answering system, named Delphi with its connotations of omniscient
wisdom, is trained in a supervised manner on a dataset of human moral
judgements from sources such as Reddit to predict the ‘correct’ judgement
given a textual prompt. While the authors claim that their system output is
merely a descriptive reflection of the morality of an under-specified
population, Talat et al. (2022) point out that the output for each query
consists of a single judgement, phrased as an imperative, thereby giving the
impression of humanlike reasoning and absolute knowledge of morality.

Recent work (e.g., Sap et al., 2022) has investigated models for _theory of
mind_ , i.e., an entity’s ability to infer other people’s ‘mental states, such
as beliefs, desires, and intentions, and to understand how mental states
feature in everyday explanations and predictions of people’s behaviour’
Apperly (2012). Such work has been motivated by the proposition that language
technologies require such capabilities to take decisions on behalf of their
users. This proposition entails that agency should be shifted from humans to
machines. Thus, the quest for theory of mind is one for further
anthropomorphising systems, and a system’s inability to perform the task can
be understood as a limiting factor to anthropomorphism of the system.

#### Agency and Responsibility

Dialogue systems are often referred to as conversational ‘agents’. However,
being an agent, i.e. having agency, requires intentionality, animacy, and,
according to Yamamoto (2006), even a soul. An entity without agency cannot be
responsible for what it produces Talat et al. (2022). Aside from the legal and
ethical implications of suggesting otherwise (Véliz, 2021), systems
acknowledging blame for errors or mistakes can add to anthropomorphic
perceptions.

Mahmood et al. (2022) found that increasing the apparent ‘sincerity’ with
which a dialogue system accepts responsibility (on behalf of ‘itself’) causes
users to perceive them to be more intelligent and likeable, potentially
increasing anthropomorphism on several dimensions. Similarly, many dialogue
systems have been criticised for ‘expressing’ controversial ‘opinions’ and
generating toxic content. It is precisely due to their lack of agency and
responsibility that developers have invested significant efforts to avoiding
contentious topics (e.g. Glaese et al., 2022; Sun et al., 2022; Xu et al.,
2021) leading to the creation of taboos for such systems, another particularly
human phenomenon.

#### Empathy

Recent work has sought for dialogue systems to produce empathetic responses to
their users, motivated by improved user engagement and establishing ‘common
ground’ (e.g. Svikhnushina et al., 2022; Zhu et al., 2022). However, dialogue
systems are not capable of experiencing empathy, and are unable to correctly
recognise emotions. Consequently, they are highly prone to producing
inappropriate emotional amplification Curry and Cercas Curry (2023). Inability
aside, the production of pseudo-empathy and emotive language serves to further
anthropomorphise dialogue systems.

#### Humanlike Activities

Beyond implying consciousness and sentience, and failing to deny humanness,
Abercrombie et al. (2021) find that in a quarter of the responses from
dialogue systems, they can be prone to making claims of having uniquely human
abilities or engaging in activities that are, by definition, restricted to
animate entities, like having family relationships, bodily functions e.g.,
consuming food, crying, engaging in physical activity, or other pursuits that
require embodiment that they do not possess. Similarly, Gros et al. (2022)
find that crowd-workers rate $20-30\%$ of utterances produced by nine
different systems as machine-impossible. They found that only one strictly
task-based system (MultiWoz, Budzianowski et al., 2018) did not appear as
anthropomorphic to participants. Glaese et al. (2022) propose to address this
concern by using reinforcement learning to prohibit systems from generating
claims of having (embodied) experiences.

#### Pronoun Use

Prior work has viewed the use of third person pronouns (e.g., ‘he’ and ‘she’)
to describe dialogue systems as evidence of users personifying systems
(Abercrombie et al., 2021; Sutton, 2020). The use of first person pronouns
(e.g., ‘me’ or ‘myself’) in system output may be a contributing factor to this
perception, as these can be read as signs of consciousness (Faber, 2020;
Minsky, 2006). Indeed, it is widely believed that ‘I’ can _only_ refer to
people (Noonan, 2009; Olson, 2002). Scruton (2017) contends that such self-
attribution and self-reference permits people to relate as subjects, not mere
objects, and that self-definition as an individual is part of the human
condition itself. First person pronoun use may therefore contribute to
anthropomorphism, either by design or due to their human-produced training
data, for symbolic and data driven dialogue systems, respectively.

Moreover, while the above applies to English and many languages, such as those
from the Indo-European family, many others feature different sets of pronouns
or distinctions for animate and inanimate things (Yamamoto, 1999), and the
self-referential production of these could further influence anthropomorphic
perceptions.

###  3.3 Register and Style

Humans are adept at using linguistic features to convey a variety of registers
and styles depending on the context (Biber and Conrad, 2009). It may therefore
be preferable for automated system outputs to be functional in style and avoid
social features to mitigate anthropomorphism.

#### Phatic Expressions

Phrases such as pleasantries that are used to form and maintain social
relations between humans but that do not impart any information can
(unnecessarily) add to the sense of humanness conveyed when output by
automated systems (Leong and Selinger, 2019).

#### Expressions of Confidence and Doubt

Dinan et al. (2022) describe an ‘imposter effect’ where people overestimate
the factuality of generated output. However, Mielke et al. (2022) find that
expressed confidence is poorly calibrated to the probabilities that general
knowledge questions are correctly answered. They therefore train a dialogue
system to reflect uncertainty in its outputs, altering the content from the
purely factual to incorporate humanlike hedging phrases such as ‘I’m not sure
but …’. This bears similarity to the text-to-speech research (see §3.1) which
suggests that disfluencies can increase anthropomorphism. Thus, while over-
estimation can lead to an imposter effect, hedging can boost anthropomorphic
signals.

#### Personas

Many dialogue systems are developed with carefully designed personas (in the
case of commercial systems) or personas induced via crowd-sourced datasets.
These are often based on human characters and although they are, in practice,
merely lists of human attributes and behaviours (see §3.2),333For example,
each persona in Personachat Zhang et al. (2018) consists of ‘ _I am a
vegetarian. I like swimming. My father used to work for Ford. My favorite
(sic) band is Maroon5. I got a new job last month, which is about advertising
design._ ’ the very idea of instilling systems with human character-based
personas is an effort towards anthropomorphism. Glaese et al. (2022) address
this by including a rule against their system appearing to have a human
identity.

###  3.4 Roles

Related to the content of their output are the roles that dialogue systems are
given, consciously and unconsciously, by their designers and users. Many of
these can shift dialogue systems from the realm of tools towards one of
humanlike roles such as provision of companionship.

#### Subservience

The majority of systems are conceived as being in the service of people in
subservient, secretarial roles (Lingel and Crawford, 2020). This has led to
users verbally abusing systems (West et al., 2019), going beyond mere
expressions of frustration that one might have with a poorly functioning tool
to frequently targeting them with gender-based slurs (Cercas Curry et al.,
2021). In such circumstances systems have even been shown to respond
subserviently to their abusers, potentially further encouraging the behaviour
(Cercas Curry and Rieser, 2018).

#### Unqualified Expertise

Systems can come to present as having expertise without appropriate
qualification (see §3.3), in large part due to their training data (Dinan et
al., 2022). For example, Abercrombie and Rieser (2022) find that both
commercial rule-based and end-to-end research systems provide high-risk
diagnoses and treatment plans in response to medical queries.444See also Omri
et al. (2023) for similar findings.

In addition, as conversational question answering systems are increasingly
promoted as replacements for web browser-based search, users can be further
given the impression that a dialogue system has the expertise to provide a
singular correct response rather than a selection of ranked search results
(Shah and Bender, 2022).

#### Terminology

There is increasing awareness that the anthropomorphic language and jargon
used to describe technologies such as language models contributes to
inaccurate perceptions of their capabilities, particularly among the general
public (Hunger, 2023; Salles et al., 2020; Shanahan, 2023). While this is also
an issue for research dissemination and journalism more widely, dialogue
systems themselves are prone to output references to their own machinic and
statistical processes with anthropomorphically loaded terms such as ‘know’,
‘think’, ‘train’, ‘learn’, ‘understand’, and ‘intelligence’.

##  4 Consequences of Anthropomorphism

The anthropomorphism of dialogue systems can induce a number of adverse
societal effects, e.g. they can generate unreliable information and reinforce
social roles, language norms, and stereotypes.

#### Trust and Deception

When people are unaware that they are interacting with automated systems they
may behave differently than if they know the true nature of their colocutor.
The trust placed in systems grows as they become more humanlike, whether or
not the trust is warranted.

This may be even more problematic when users are members of vulnerable
populations, such as the very young, the elderly, or people with illnesses or
disabilities. Although dialogue systems may be a possible solution to
loneliness, socially disconnected individuals can be particularly vulnerable
to such trust issues. Children have also been shown to overestimate the
intelligence of voice assistants such as Amazon Alexa, and to be unsure of
whether they have emotions or feelings (Andries and Robertson, 2023). Given
UNESCO’s declaration that children have the right to participate in the design
of the technological systems that affect them (Dignum et al., 2021),
developers may be obliged to bear these considerations in mind.

#### Gendering Machines

People may gender technologies in the face of even minimal gender markers
(Reeves and Nass, 1996), as evident in commercial dialogue systems Abercrombie
et al. (2021). However, even without _any_ gender markers, people still tend
to apply binary gender to dialogue systems Aylett et al. (2019); Sutton
(2020), as was the case for the ‘genderless’ voice assistant Q. In a summary
of West et al. (2019), UNESCO (2019) argue that that encouraging or enabling
users to predominantly gender systems as female reinforces gender stereotypes
of women as inferior to men:

  * _[digital assistants] reflect, reinforce and spread gender bias; model acceptance and tolerance of sexual harassment and verbal abuse; send explicit and implicit messages about how women and girls should respond to requests and express themselves; make women the ‘face’ of glitches and errors that result from the limitations of hardware and software designed predominately by men; and force synthetic ‘female’ voices and personality to defer questions and commands to higher (and often male) authorities_.

Furthermore, while some companies have begun to offer greater diversity of
voices and move away from default female-gendered voices (Iyengar, 2021), non-
binary or gender-ambiguous dialogue systems such as Danielescu et al. (2023)’s
Sam are almost nonexistent, leaving people who identify as such without
representation.

By designing anthropomorphic systems or even simply leaving space for their
(gendered) personification by users, system developers risk propagating
stereotypes and enabling these harms.

#### Language Variation and Whiteness

Historically, NLP technologies have been developed to capture the language use
of particular demographics, in part due to their training data. While there
are many potential benefits to language technologies like dialogue systems,
successful interactions with them require people to conform their language use
to that recognised by the technologies. Given the proclivity of NLP to centre
white, affluent American dialects (Hovy and Prabhumoye, 2021; Joshi et al.,
2020), language variants that deviate from these norms are less likely to be
correctly processed Tatman (2017). As a result, people code switch when
engaging with dialogue systems in order to have successful interactions Foster
and Stuart-Smith (2023); Harrington et al. (2022). This can represent a form
of language policing, in which people are given the choice to conform to
language that is recognisable by the technology, or forego using it—and its
potential benefits—altogether. Consequently, as people conform to language
variants that are recognised by dialogue systems, they also conform to
whiteness and the continued erasure of marginalised communities.

The personification of such systems could certainly exacerbate this erasure of
marginalised communities, for instance through limiting diverse language data.
Furthermore, system outputs typically suffer from the same standardisation,
for instance prioritising specific accents that conform to western notions of
acceptability and prestige (see §3). Thus users from marginalised communities
are in a position where not only do they have to adapt their accent, but also
(given the tendency to do personify), personify ‘white’-centred dialogue
systems that are often marketed as ‘oracles of knowledge’.

##  5 Recommendations

Dialogue systems are used for a wide variety of tasks, and fine-grained
recommendations may only apply to a small set of contexts. We therefore make
broad recommendations for consideration in dialogue system design: designers
should recognise people’s tendency to personify, consider whether
anthropomorphic tools are appropriate, and reassess both their research goals
and the language they use to describe the systems they build.

#### Recognise tendencies to personify

In language, there is a distinction between form (e.g. in statistical language
modelling, where the task may be string prediction) and meaning (the
relationship between form and communicative intent) (Grice, 1988). Bender and
Koller (2020) argue that humans inherently derive meaning from signals, i.e.,
form, regardless of the presence of communicative intent.

It is therefore crucial for technology developers to be sensitive towards this
tendency, as it applies to dialogue systems, given that until recently open
domain dialogue was exclusively possible between humans. That is, regardless
of whether it is part of a dialogue system’s deliberate design to use specific
linguistic forms (e.g., the anthropomorphic cues outlined in §3), the listener
will invariably perceive communicative intent. Thus unnecessary integration of
anthropomorphic linguistic cues can cause people to attribute humanlike
cognitive abilities to systems—as in the case of Google Duplex, which
excessively leveraged disfluences. Creators of dialogue systems should
recognise these tendencies, consider the anthropomorphic cues people may pick
up on, and avoid sending such signals, whether by design or through lack of
consideration (e.g. from inclusion in datasets).

#### Consider the appropriateness of anthropomorphic tools

Given our inherent nature to attribute meaning to signals, one must consider
the _appropriateness of the tool and use cases_ (Dinan et al., 2022; Bender et
al., 2021) when designing dialogue systems, in order to avoid the
(over-)integration of anthropomorphic cues. Indeed, it is only within a given
context that one can make judgement on whether anthropomorphism is a concern.
For instance, anthropomorphising one’s vacuum cleaning robot (i.e. shouting at
it in frustration for not cleaning appropriately), is of less concern than
anthropomorphising a dialogue system marketed as ‘social’ or ‘empathetic’, or
technology sold as a ‘singular oracle of (all) knowledge’. We therefore argue
that developers need to focus on the appropriateness of anthropomorphic tools
in order to limit the negative consequences of anthropomorphism which can lead
to false impressions of a system’s capabilities.

#### Reassess research goals

Traditionally, the goal of Artificial Intelligence research has been to create
systems that would exhibit intelligence indistinguishable from human
behaviour. Text-to-speech systems for instance, are evaluated on how natural
and fluent the output sounds. Though intelligence and understanding should not
be conflated with systems that exhibit humanlike behaviour (Bender and Koller,
2020), the human tendency to anthropomorphise convinces us of a machine’s
apparent intelligence (Proudfoot, 2011). It is in part due to this
longstanding goal of anthropomorphic systems that there only exists a small
body of work that does not seek to anthropomorphise, despite the growing
awareness of the harms that can arise from anthropomorphising systems.

While it is infeasible to create a comprehensive mapping of which linguistic
cues to use and which not, we discuss recommendations that arise from the work
surveyed. For example, Wilson and Moore (2017) recommend that developers
calibrate vocal features such as pitch and pitch shimmer to produce
synthesised voices that people recognise as non-human. In an analysis of
reviews of commercial voice assistants, Völkel et al. (2020) find that the big
five personality traits De Raad (2000) do not adequately describe user
expectations of systems’ ‘personalities’. The only consistently desired trait
was agreeableness, as users expect that systems react promptly and reliably to
queries Völkel et al. (2020). Thus, imbuing voice assistants and dialogue
systems with humanlike personality traits does not ensure alignment with
people’s expectation of system behaviour. We therefore recommend that
designers and developers forego embedding humanlike personality traits in
dialogue systems.

#### Avoid anthropomorphic system description

Irrespective of any actual humanlike qualities that dialogue systems might
possess, there is widespread public confusion surrounding the nature and
abilities of current automated language technologies. This confusion extends
from children (Andries and Robertson, 2023) to adults (including some
journalists, policymakers, and business people) that are convinced, on the one
hand, of humanity’s imminent enslavement to super-intelligent artificial
agents (to the neglect of actual harms already being propagated by
technological systems), or, on the other, that they provide super-human
solutions to the world’s problems (Hunger, 2023; Klein, 2023).

As discussed in §3.4, the content of systems’ outputs can reinforce such
anthropomorphic perceptions. However, of even more influence is the language
used by system developers to describe their work. The tendency of people who
do know how technologies are built to use anthropomorphic language represents,
according to Salles et al. (2020, p. 93), ‘a significant failure in scientific
communication and engagement’. Although anthropomorphic terminology is deeply
rooted in the argot of computer scientists, particularly those working in
‘artificial intelligence’, and while there exist significant motivations to
continue to create hype around products and research (Hunger, 2023),
practitioners should reflect on how the language they use affects people’s
understanding and behaviour.

##  6 Conclusion

Anthropomorphising dialogue systems can be attractive for researchers in order
to drive user engagement. However, production of highly anthropomorphic
systems can also lead to downstream harms such as (misplaced) trust in
generated misinformation. Even if developers and designers attempt to avoid
including any anthropomorphic signals, humans may still personify systems and
perceive them as anthropomorphic entities. For this reason, we argue that it
is particularly important to carefully consider the particular ways that
systems might be perceived anthropomorphically, and choose the appropriate
feature for a given situation. By carefully considering how a system may be
anthropomorphised and deliberately selecting the attributes that are
appropriate for each context, developers and designers can avoid falling into
the trap of creating mirages of humanity.

## Limitations

While we have attempted to enumerate the linguistic factors that can increase
the likelihood that users will view dialogue systems as anthropomorphic, this
list of features is not exhaustive. As we describe in section 2,
anthropomorphism varies from person-to-person and people may react differently
to different aspects of a system’s design. This paper represents only a
starting point for researchers and developers to consider the implications
that their design choices may have.

In this paper, due to the backgrounds of the authors as speakers of Indo-
European languages and the dominance of English in NLP research, we have
focused primarily on English language dialogue systems. However, it should be
noted that other languages have features such as grammatical ways of denoting
animacy (Yamamoto, 1999) and gender that could influence users personification
of systems, and which developers should consider if they wish to limit
anthropomorphism.

## Ethical Considerations

While our manuscript outlines ways to create dialogue systems while minimising
their potential anthropomorphism and personification, it could also be used as
a guide to creating anthropomorphic systems. Our aim is to highlight the risks
and provide researchers, developers, and designers with a path towards
addressing the concerns that arise from anthropomorphisation in dialogue
systems, an area that is particularly relevant at the time of writing due to
the introduction of systems such as OpenAI’s ChatGPT and Microsoft’s Sydney,
which have high surface form language generation performance.

## References

  * Abercrombie et al. (2021) Gavin Abercrombie, Amanda Cercas Curry, Mugdha Pandya, and Verena Rieser. 2021.  Alexa, Google, Siri: What are your pronouns? gender and anthropomorphism in the design and perception of conversational assistants.  In _Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing_ , pages 24–33, Online. Association for Computational Linguistics.
  * Abercrombie and Rieser (2022) Gavin Abercrombie and Verena Rieser. 2022.  Risk-graded safety for handling medical queries in conversational AI.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 234–243, Online only. Association for Computational Linguistics.
  * Andries and Robertson (2023) Valentina Andries and Judy Robertson. 2023.  “Alexa doesn’t have that many feelings”: Children’s understanding of AI through interactions with smart speakers in their homes.
  * Apperly (2012) Ian A Apperly. 2012.  What is “theory of mind”? Concepts, cognitive processes and individual differences.  _Quarterly Journal of Experimental Psychology_ , 65(5):825–839.
  * Araujo (2018) Theo Araujo. 2018.  Living up to the chatbot hype: The influence of anthropomorphic design cues and communicative agency framing on conversational agent and company perceptions.  _Computers in Human Behavior_ , 85:183–189.
  * Atleson (2023) Michael Atleson. 2023.  Chatbots, deepfakes, and voice clones: AI deception for sale.  https://www.ftc.gov/business-guidance/blog/2023/03/chatbots-deepfakes-voice-clones-ai-deception-sale.  Federal Trade Commission. Accessed: 2023-04-14.
  * Aylett et al. (2019) Matthew P. Aylett, Selina Jeanne Sutton, and Yolanda Vazquez-Alvarez. 2019.  The right kind of unnatural: Designing a robot voice.  In _Proceedings of the 1st International Conference on Conversational User Interfaces_ , CUI ’19, New York, NY, USA. Association for Computing Machinery.
  * Barr and Seyfeddinipur (2010) Dale J Barr and Mandana Seyfeddinipur. 2010.  The role of fillers in listener attributions for speaker disfluency.  _Language and Cognitive Processes_ , 25(4):441–455.
  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big? In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery.
  * Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020.  Climbing towards NLU: On meaning, form, and understanding in the age of data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5185–5198, Online. Association for Computational Linguistics.
  * Biber and Conrad (2009) Douglas Biber and Susan Conrad. 2009.  _Register, Genre, and Style_.  Cambridge Textbooks in Linguistics. Cambridge University Press.
  * Boden et al. (2017) Margaret Boden, Joanna Bryson, Darwin Caldwell, Kerstin Dautenhahn, Lilian Edwards, Sarah Kember, Paul Newman, Vivienne Parry, Geoff Pegman, Tom Rodden, Tom Sorrell, Mick Wallis, Blay Whitby, and Alan Winfield. 2017.  Principles of robotics: regulating robots in the real world.  _Connection Science_ , 29(2):124–129.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5016–5026, Brussels, Belgium. Association for Computational Linguistics.
  * California State Legislature (2018) California State Legislature. 2018.  California Senate Bill no. 1001.  Technical report, California State Legislature.
  * Cercas Curry et al. (2021) Amanda Cercas Curry, Gavin Abercrombie, and Verena Rieser. 2021.  ConvAbuse: Data, analysis, and benchmarks for nuanced abuse detection in conversational AI.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7388–7403, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Cercas Curry and Rieser (2018) Amanda Cercas Curry and Verena Rieser. 2018.  #MeToo Alexa: How conversational systems respond to sexual harassment.  In _Proceedings of the Second ACL Workshop on Ethics in Natural Language Processing_ , pages 7–14, New Orleans, Louisiana, USA. Association for Computational Linguistics.
  * Clark and Fox Tree (2002) Herbert H Clark and Jean E Fox Tree. 2002.  Using _uh_ and _um_ in spontaneous speaking.  _Cognition_ , 84(1):73–111.
  * Coll Ardanuy et al. (2020) Mariona Coll Ardanuy, Federico Nanni, Kaspar Beelen, Kasra Hosseini, Ruth Ahnert, Jon Lawrence, Katherine McDonough, Giorgia Tolfo, Daniel CS Wilson, and Barbara McGillivray. 2020.  Living machines: A study of atypical animacy.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4534–4545, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Corley et al. (2007) Martin Corley, Lucy J MacGregor, and David I Donaldson. 2007.  It’s the way that you, er, say it: Hesitations in speech affect language comprehension.  _Cognition_ , 105(3):658–668.
  * Crystal (1980) David Crystal. 1980.  _A First Dictionary of Linguistics and Phonetics_.  Language library. John Wiley & Sons, Incorporated.
  * Curry and Cercas Curry (2023) Alba Curry and Amanda Cercas Curry. 2023.  Computer says “no”: The case against empathetic conversational AI.  In _Findings of the 62nd Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
  * Danielescu et al. (2023) Andreea Danielescu, Sharone A Horowit-Hendler, Alexandria Pabst, Kenneth Michael Stewart, Eric M Gallo, and Matthew Peter Aylett. 2023.  Creating inclusive voices for the 21st century: A non-binary text-to-speech for conversational assistants.  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , CHI ’23, New York, NY, USA. Association for Computing Machinery.
  * De Raad (2000) Boele De Raad. 2000.  _The big five personality factors: The psycholexical approach to personality._ Hogrefe & Huber Publishers.
  * Descartes (1637) René Descartes. 1637.  _Discours de la Méthode_.
  * Dignum et al. (2021) Virginia Dignum, Melanie Penagos, Klara Pigmans, and Steven Vosloo. 2021.  Policy guidance on AI for children: Recommendations for building AI policies and systems that uphold child rights.  Report, UNICEF.
  * Dinan et al. (2022) Emily Dinan, Gavin Abercrombie, A. Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. 2022.  SafetyKit: First aid for measuring safety in open-domain conversational systems.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4113–4133, Dublin, Ireland. Association for Computational Linguistics.
  * Dinkar et al. (2023) Tanvi Dinkar, Chloé Clavel, and Ioana Vasilescu. 2023.  Fillers in spoken language understanding: Computational and psycholinguistic perspectives.  _arXiv preprint arXiv:2301.10761_.
  * Epley et al. (2007) Nicholas Epley, Adam Waytz, and John T. Cacioppo. 2007.  On seeing human: A three-factor theory of anthropomorphism.  _Psychological Review_ , 114.
  * Faber (2020) Liz W. Faber. 2020.  _The Computer’s Voice: From Star Trek to Siri_.  University of Minnesota Press.
  * Fogg and Nass (1997) BJ Fogg and Clifford Nass. 1997.  How users reciprocate to computers: An experiment that demonstrates behavior change.  In _CHI ’97 Extended Abstracts on Human Factors in Computing Systems_ , CHI EA ’97, page 331–332. Association for Computing Machinery, New York, NY, USA.
  * Foster and Stuart-Smith (2023) Mary Ellen Foster and Jane Stuart-Smith. 2023.  Social robotics meets sociolinguistics: Investigating accent bias and social context in HRI.  In _Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ , HRI ’23, page 156–160, New York, NY, USA. Association for Computing Machinery.
  * Fraundorf et al. (2018) Scott H. Fraundorf, Jennifer Arnold, and Valerie J. Langlois. 2018.  Disfluency.  https://www.oxfordbibliographies.com/view/document/obo-9780199772810/obo-9780199772810-0189.xml.  Oxford University Press. Accessed: 2023-05-12.
  * Glaese et al. (2022) Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soňa Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. 2022\.  Improving alignment of dialogue agents via targeted human judgements.
  * Goldman (2023) Sharon Goldman. 2023.  Sen. Murphy’s tweets on ChatGPT spark backlash from former White House AI policy advisor.  https://venturebeat.com/ai/sen-murphys-tweets-on-chatgpt-spark-backlash-from-former-white-house-ai-policy-advisor/.  VentureBeat. Accessed: 2023-04-04.
  * (35) Google Assistant.  https://developers.google.com/assistant/conversation-design/create-a-persona.  Google. Accessed: 2023-04-04.
  * Grice (1988) H. P. Grice. 1988.  Utterer’s meaning, sentence-meaning, and word-meaning.  In Jack Kulas, James H. Fetzer, and Terry L. Rankin, editors, _Philosophy, Language, and Artificial Intelligence: Resources for Processing Natural Language_ , pages 49–66. Springer Netherlands, Dordrecht.
  * Gros et al. (2021) David Gros, Yu Li, and Zhou Yu. 2021.  The R-U-a-robot dataset: Helping avoid chatbot deception by detecting user questions about human or non-human identity.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6999–7013, Online. Association for Computational Linguistics.
  * Gros et al. (2022) David Gros, Yu Li, and Zhou Yu. 2022.  Robots-dont-cry: Understanding falsely anthropomorphic utterances in dialog systems.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3266–3284, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Harrington et al. (2022) Christina N. Harrington, Radhika Garg, Amanda Woodward, and Dimitri Williams. 2022\.  “It’s kind of like code-switching”: Black older adults’ experiences with a voice assistant for health information seeking.  In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , CHI ’22, New York, NY, USA. Association for Computing Machinery.
  * Hovy and Prabhumoye (2021) Dirk Hovy and Shrimai Prabhumoye. 2021.  Five sources of bias in natural language processing.  _Language and Linguistics Compass_ , 15(8):e12432.
  * Hunger (2023) Francis Hunger. 2023.  Unhype artificial ‘intelligence’! A proposal to replace the deceiving terminology of AI.  Working paper, Training the Archive.
  * Iyengar (2021) Rishi Iyengar. 2021.  Apple will no longer make Siri’s voice female by default.  https://edition.cnn.com/2021/03/31/tech/siri-voice-female-default/index.html.  CNN. Accessed: 2023-05-12.
  * Jiang et al. (2022) Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, and Yejin Choi. 2022.  Can machines learn morality? The Delphi experiment.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Kim et al. (2022) Hyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim, Yejin Choi, and Maarten Sap. 2022.  Prosocialdialog: A prosocial backbone for conversational agents.
  * Kim and Sundar (2012) Youjeong Kim and S Shyam Sundar. 2012.  Anthropomorphism of computers: Is it mindful or mindless?  _Computers in Human Behavior_ , 28(1):241–250.
  * Kirkland et al. (2022) Ambika Kirkland, Harm Lameris, Eva Székely, and Joakim Gustafson. 2022.  Where’s the uh, hesitation? The interplay between filled pause location, speech rate and fundamental frequency in perception of confidence.  _Proc. Interspeech 2022_ , pages 4990–4994.
  * Klein (2023) Naomi Klein. 2023.  AI machines aren’t ‘hallucinating’. But their makers are.  https://www.theguardian.com/commentisfree/2023/may/08/ai-machines-hallucinating-naomi-klein.  The Guardian. Accessed: 2023-05-11.
  * Korhonen (2019) Tua Korhonen. 2019.  Anthropomorphism and the aesopic animal fables.  _Animals and their Relation to Gods, Humans and Things in the Ancient World_ , pages 211–231.
  * Krauss et al. (2002) Robert M. Krauss, Robin J. Freyberg, and Ezequiel Morsella. 2002.  Inferring speakers’ physical attributes from their voices.  _Journal of Experimental Social Psychology_ , 38:618–625.
  * Leong and Selinger (2019) Brenda Leong and Evan Selinger. 2019.  Robot eyes wide shut: Understanding dishonest anthropomorphism.  In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ , FAT* ’19, page 299–308, New York, NY, USA. Association for Computing Machinery.
  * Leviathan and Matias (2018) Yaniv Leviathan and Yossi Matias. 2018.  Google duplex: An AI system for accomplishing real world tasks over the phone.  _Google AI Blog_.
  * Lieu (2018) Johnny Lieu. 2018.  Google’s creepy AI phone call feature will disclose it’s a robot, after backlash.  https://mashable.com/2018/05/11/google-duplex-disclosures-robot.  Mashable. Accessed 2023-03-16.
  * Lingel and Crawford (2020) Jessa Lingel and Kate Crawford. 2020.  “Alexa, tell me about your mother”: The history of the secretary and the end of secrecy.  _Catalyst: Feminism, Theory, Technoscience_ , 6(1).
  * Liu (2022) Fanjue Liu. 2022.  Hanging out with my pandemic pal: Contextualizing motivations of anthropomorphizing voice assistants during covid-19.  _Journal of Promotion Management_ , pages 1–29.
  * Lovens (2023) Pierre-François Lovens. 2023.  Sans ces conversations avec le chatbot Eliza, mon mari serait toujours là.  https://www.lalibre.be/belgique/societe/2023/03/28/s ans-ces-conversations-avec-le-chatbot-eliza-mon-mari-serait-toujours-la-LVSLWPC5WRDX7J2RCHNWPDST24/.  La Libre. Accessed: 2023-04-14.
  * Mahmood et al. (2022) Amama Mahmood, Jeanie W Fung, Isabel Won, and Chien-Ming Huang. 2022.  Owning mistakes sincerely: Strategies for mitigating AI errors.  In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , CHI ’22, New York, NY, USA. Association for Computing Machinery.
  * Metz (2020) Cade Metz. 2020.  Riding out quarantine with a chatbot friend: ‘I feel very connected’.  https://www.nytimes.com/2020/06/16/technology/chatbots-quarantine-coronavirus.html.  New York Times. Accessed: 2023-04-25.
  * Mielke et al. (2022) Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022.  Reducing conversational agents’ overconfidence through linguistic calibration.  _Transactions of the Association for Computational Linguistics_ , 10:857–872.
  * Minsky (2006) Marvin Minsky. 2006.  _The Emotion Machine: Commonsense Thinking, Artificial Intelligence, and the Future of the Human Mind_.  Simon and Schuster.
  * Nass and Brave (2005) Clifford Ivar Nass and Scott Brave. 2005.  _Wired for speech: How voice activates and advances the human-computer relationship_.  MIT press Cambridge.
  * Noonan (2009) Harold W. Noonan. 2009.  The thinking animal problem and personal pronoun revisionism.  _Analysis_ , 70(1):93–98.
  * Olson (2002) Eric T Olson. 2002.  Thinking animals and the reference of ‘I’.  _Philosophical Topics_ , 30(1):189–207.
  * Omri et al. (2023) Sihem Omri, Manel Abdelkader, Mohamed Hamdi, and Tai-Hoon Kim. 2023.  Safety issues investigation in deep learning based chatbots answers to medical advice requests.  In _Neural Information Processing_ , pages 597–605, Singapore. Springer Nature Singapore.
  * Proudfoot (2011) Diane Proudfoot. 2011.  Anthropomorphism and AI: Turing’s much misunderstood imitation game.  _Artificial Intelligence_ , 175(5):950–957.  Special Review Issue.
  * Ravichander and Black (2018) Abhilasha Ravichander and Alan W. Black. 2018.  An Empirical Study of Self-Disclosure in Spoken Dialogue Systems.  In _Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 253–263, Melbourne, Australia. Association for Computational Linguistics.
  * Reeves and Nass (1996) Byron Reeves and Clifford Nass. 1996.  _The Media Equation: How People Treat Computers, Television, and New Media like Real People_.  Cambridge university press Cambridge, UK.
  * Salles et al. (2020) Arleen Salles, Kathinka Evers, and Michele Farisco. 2020.  Anthropomorphism in AI.  _AJOB Neuroscience_ , 11(2):88–95.  PMID: 32228388.
  * Sap et al. (2022) Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022.  Neural theory-of-mind? on the limits of social intelligence in large LMs.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3762–3780, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Scruton (2017) Roger Scruton. 2017.  On human nature.  In _On Human Nature_. Princeton University Press.
  * Shah and Bender (2022) Chirag Shah and Emily M. Bender. 2022.  Situating search.  In _ACM SIGIR Conference on Human Information Interaction and Retrieval_ , CHIIR ’22, page 221–232, New York, NY, USA. Association for Computing Machinery.
  * Shanahan (2023) Murray Shanahan. 2023.  Talking about large language models.
  * Skantze et al. (2015) Gabriel Skantze, Martin Johansson, and Jonas Beskow. 2015.  Exploring turn-taking cues in multi-party human-robot discussions about objects.  In _Proceedings of the 2015 ACM on international conference on multimodal interaction_ , pages 67–74.
  * Smith and Clark (1993) Vicki L Smith and Herbert H Clark. 1993.  On the course of answering questions.  _Journal of Memory and Language_ , 32(1):25–38.
  * Stern et al. (2021) Julia Stern, Christoph Schild, Benedict C. Jones, Lisa M. DeBruine, Amanda Hahn, David A. Puts, Ingo Zettler, Tobias L. Kordsmeyer, David Feinberg, Dan Zamfir, Lars Penke, and Ruben C. Arslan. 2021.  Do voices carry valid information about a speaker’s personality? _Journal of Research in Personality_ , 92:104092.
  * Stupple-Harris (2021) Louis Stupple-Harris. 2021.  Tech in the dock. Should AI chatbots be used to address the nation’s loneliness problem?  https://www.nesta.org.uk/feature/tech-dock.  NESTA. Accessed: 2023-05-11.
  * Sun et al. (2022) Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. 2022.  On the safety of conversational models: Taxonomy, dataset, and benchmark.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3906–3923, Dublin, Ireland. Association for Computational Linguistics.
  * Sutton (2020) Selina Jeanne Sutton. 2020.  Gender ambiguous, not genderless: Designing gender in voice user interfaces (VUIs) with sensitivity.  In _Proceedings of the 2nd Conference on Conversational User Interfaces_ , CUI ’20, New York, NY, USA. Association for Computing Machinery.
  * Svikhnushina et al. (2022) Ekaterina Svikhnushina, Iuliana Voinea, Anuradha Welivita, and Pearl Pu. 2022.  A taxonomy of empathetic questions in social dialogs.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2952–2973, Dublin, Ireland. Association for Computational Linguistics.
  * Talat et al. (2022) Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. 2022.  On the machine learning of ethical judgments from natural language.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 769–779, Seattle, United States. Association for Computational Linguistics.
  * Tatman (2017) Rachael Tatman. 2017.  Gender and dialect bias in YouTube’s automatic captions.  In _Proceedings of the First ACL Workshop on Ethics in Natural Language Processing_ , pages 53–59, Valencia, Spain. Association for Computational Linguistics.
  * Torre and Maguer (2020) Ilaria Torre and Sébastien Le Maguer. 2020.  Should robots have accents? In _2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_ , pages 208–214.
  * UNESCO (2019) UNESCO. 2019.  Explore the gendering of AI voice assistants.  https://es.unesco.org/node/305128.  UNESCO. Accessed: 2023-04-25.
  * Véliz (2021) Carissa Véliz. 2021.  Moral zombies: why algorithms are not moral agents.  _AI & Society_, 36.
  * Völkel et al. (2020) Sarah Theres Völkel, Ramona Schödel, Daniel Buschek, Clemens Stachl, Verena Winterhalter, Markus Bühner, and Heinrich Hussmann. 2020.  Developing a personality model for speech-based conversational agents using the psycholexical approach.  In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , CHI ’20, page 1–14, New York, NY, USA. Association for Computing Machinery.
  * Wagner et al. (2019) Katja Wagner, Frederic Nimmermann, and Hanna Schramm-Klein. 2019.  Is it human? The role of anthropomorphism as a driver for the successful acceptance of digital voice assistants.  _Proceedings of the 52nd Hawaii International Conference on System Sciences_ , page 10.
  * Walsh (2016) Toby Walsh. 2016.  Turing’s red flag.  _Communications of the ACM_ , 59(7):34–37.
  * West et al. (2019) Mark West, Rebecca Kraut, and Han Ei Chew. 2019.  _I’d Blush if I Could: Closing Gender Divides in Digital Skills through Education_.  UNESCO.
  * Wilson and Moore (2017) Sarah Wilson and Roger K. Moore. 2017.  Robot, alien and cartoon voices: Implications for speech-enabled systems.  In _1st International Workshop on Vocal Interactivity in-and-between Humans, Animals and Robots (VIHAR)_ , page 42–46.
  * Wollermann et al. (2013) Charlotte Wollermann, Eva Lasarcyk, Ulrich Schade, and Bernhard Schröder. 2013\.  Disfluencies and uncertainty perception–evidence from a human–machine scenario.  In _Sixth Workshop on Disfluency in Spontaneous Speech_.
  * Xu et al. (2021) Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021\.  Recipes for safety in open-domain chatbots.
  * Yamamoto (1999) Mutsumi Yamamoto. 1999.  _Animacy and Reference: A Cognitive Approach to Corpus Linguistics_.  J. Benjamins.
  * Yamamoto (2006) Mutsumi Yamamoto. 2006.  _Agency and Impersonality: Their Linguistic and Cultural Manifestations_.  J. Benjamins.
  * Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018.  Personalizing dialogue agents: I have a dog, do you have pets too?
  * Zhu et al. (2022) Ling.Yu Zhu, Zhengkun Zhang, Jun Wang, Hongbin Wang, Haiying Wu, and Zhenglu Yang. 2022.  Multi-party empathetic dialogue generation: A new task for dialog systems.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 298–307, Dublin, Ireland. Association for Computational Linguistics.
  * Ziems et al. (2022) Caleb Ziems, Jane Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2022.  The moral integrity corpus: A benchmark for ethical dialogue systems.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3755–3773, Dublin, Ireland. Association for Computational Linguistics.