# Survey of Design Paradigms for Social Robots

Rita Frieske, Xiaoyu Mo, Yini Fang, Jay Nieles, Bertram E. Shi

###### Abstract

The demand for social robots in fields like healthcare, education, and
entertainment increases due to their emotional adaptation features. These
robots leverage multimodal communication, incorporating speech, facial
expressions, and gestures to enhance user engagement and emotional support.
The understanding of design paradigms of social robots is obstructed by the
complexity of the system and the necessity to tune it to a specific task. This
article provides a structured review of social robot design paradigms,
categorizing them into cognitive architectures, role design models, linguistic
models, communication flow, activity system models, and integrated design
models. By breaking down the articles on social robot design and application
based on these paradigms, we highlight the strengths and areas for improvement
in current approaches. We further propose our original integrated design model
that combines the most important aspects of the design of social robots. Our
approach shows the importance of integrating operational, communicational, and
emotional dimensions to create more adaptive and empathetic interactions
between robots and humans.

###### Index Terms:

Social robots, robot design, survey, dialogue.

##  I Introduction

Social robots are interactive machines designed to engage with humans in a
socially meaningful way. These robots are capable of processing and responding
to human emotions, intentions, and actions through various modalities, such as
speech, facial expressions, gestures, and other forms of communication. With
the increased demand for technological support in fields such as healthcare,
education, entertainment, and customer service, a social robot’s ability to
simulate empathetic social interaction enhances user engagement, emotional
support, and task efficiency [Fong et al., 2003, Dautenhahn, 2007].

During our work on designing health care robot assistant Grace, we discovered
a discrepancy between advancement and a body of literature on traditional
chatbots and the coverage of the design and application of social robots. In
their survey on dialogue management in HRI [Reimann et al., 2023] point out
that the reason behind, it is that most of the dialogue management in HRI is
handcrafted based on how the robot is used. We believe that the complexity of
the integration of hardware components with the software. Through our design
experience, we discovered that the creation of a plausible design required
combining knowledge from various fields stretching far beyond engineering.
Subsequently, we decided that there remains a need for a comprehensive
analysis of the various design paradigms of social robots, which could be
potentially helpful for both researchers and engineers to clarify the scope of
their work.

This article contributes to the field by providing a structured review of
design paradigms for social robots, categorized into cognitive architecture,
role design model, linguistic model, communication flow, activity system
model, and integrated design model. By breaking down the articles on social
robot design and application based on these paradigms, this study highlights
the strengths and areas for improvement in current approaches. Furthermore,
our article associates different design paradigms with specific fields of
study such as linguistics, psychology, communication theory, neuroscience, and
engineering.

Additionally, we propose a novel integrated design model. We discovered that
the analyzed designs concentrate either on interactive communicational aspects
or engineering aspects when creating the design, while the emotional responses
are often added as an additional feature to the model. We believe that since
the emotional response is crucial to the social robot operation it has to be
considered from the very beginning of the design process. Our integrated
design model combines operational, emotional, and communicational dimensions
that are crucial for the social robot design.

HRI | NLP
---|---
Multimodal (audio+vision+haptic+others) | Mostly bi-modal (text+image)
Pipeline systems | End-to-end deep learning models
Specific domain | Open domain or specific domain
Active side: robot or user | Active side: user
Wide possibility of output expression | Limited possibility of output expression
Wide context | Narrow context (mostly semantic)
Real-time interaction | Temporal sequence interaction

TABLE I: Comparison between HRI and NLP multimodal dialogue systems.

##  II Features of Dialogue in HRI

Although communication with robots with natural language as a medium was
introduced early on in the robotics field the back and forth communication or
more complex interaction is still an underdeveloped task. There is a great
discrepancy between the development of NLP dialogue systems and their
successful application in robotic agents.

Modern NLP systems are mostly created with computer users in mind, and as
such, they work within restrictions of computer communication, namely mostly
operating with written text, with no time limits, with only linguistic context
and output expression the same as an input. Multimodal dialogue systems in NLP
are majority bi-modal, consisting of text and images. Table I shows the
breakdown of major differences in dialogue in NLP and HRI.

#### Real-time Interaction

Time is a more important concept in HRI than in most NLP-powered dialogue
systems. Sequential progression is an important concept in natural language,
but for robots being immersed in reality the timely and accurate response is
crucial for successful operation. This is reflected in different approaches
towards state tracking.

Time is a much more important concept in HRI than in most NLP-powered dialogue
systems. Sequential progression is an important concept in natural language,
but for robots being immersed in reality a timely and accurate response is
crucial for successful operation. This is reflected in different approaches
towards state tracking. In the NLP case, the state represents the progression
of communication, focusing primarily on the sequence of conversational turns
and the context of the dialogue. The system tracks the history of the
conversation to manage and generate appropriate responses, ensuring coherence
and relevance in the interaction.

In contrast, for robots, the state is not limited to verbal exchange but also
includes real-time sensory inputs and situational awareness. This information
may be concentrated on different aspects such as the task at hand, the human
interlocutor, or the surrounding environment. For instance, a robot assistant
in a healthcare setting needs to process and integrate data from multiple
sources—such as patient vitals, environmental conditions, and spoken
instructions—to provide timely and contextually appropriate actions.

Moreover, the temporal aspect of HRI necessitates that robots continuously
update their state based on immediate inputs and feedback. This includes
recognizing and responding to non-verbal cues like gestures, facial
expressions, and body language, which can change rapidly and require
instantaneous interpretation. This is essential for maintaining the flow of
interaction and ensuring the robot’s actions are relevant and effective.

This complexity in state tracking for robots also involves managing and
prioritizing multiple streams of information simultaneously. For example, a
robot may need to balance focusing on a conversational task with monitoring
environmental changes that could impact its operation or the safety of humans
around it. The integration of multimodal data thus requires sophisticated
algorithms capable of real-time processing and decision-making, far beyond the
linear progression typically managed in traditional NLP dialogue systems.

#### Wide Communication Context

One of the distinct features of the application of multimodal pathways in
robots is a widening of the communication context. In the case of dialogue
systems trained on text, the context is limited to the linguistic cues in a
sentence. In the case of ASR systems context consists of acoustic and
linguistic cues. HRI widens context that can take additional information from
the environment including face tracking, emotion recognition, environment
tracking, pose estimation, and pressure cues. This expansion allows robots to
gather and interpret a rich array of data, providing a more comprehensive
understanding of the interaction and enabling more nuanced responses.

Communication context is not limited to receiving information from the
environment. Embodied robots possess more means to mirror and respond to
humans with their behaviors and actions. This is especially striking in social
robots, due to the need to use gaze, mirroring, and other social cues so that
the robot becomes more responsive to users’ emotional needs. For instance, a
social robot can maintain eye contact, mimic facial expressions, and adjust
its posture to align with the emotional state of the human interlocutor,
thereby enhancing the sense of connection and empathy in the interaction.

Figure 1: The communication context of text chatbot consists of textual
information, whereas for robots that operate in the physical environment, the
communication context is grounded in its direct environment and limited only
by the number of its sensors.

#### Symmetrical Communication

The aspect very specific to social-robot interaction is their active role in
the communication process. Personal assistants such as Apple Siri or Amazon
Alexa are passively waiting for the user’s command, which is clear since the
user needs to use a ”wake up phrase” to start the interaction. In the case of
a social robot, both humans and robots can be agents, since the task of some
social robots is engaging humans to interact (such as in the case of humanoid
elderly companions). This ability to initiate interaction allows social robots
to participate in a more symmetrical communication pattern, where the flow of
dialogue is more dynamic and can be influenced by both parties.

Social robots are designed to exhibit social behaviors and cues that
facilitate a bidirectional flow of communication. Robots can initiate
interactions by moving closer to a person, gesturing to attract attention, or
even using tactile feedback, such as a gentle touch, to provide comfort or
reassurance. These behaviors are not only crucial for making the interaction
feel more human-like but also for ensuring that the robot can effectively
engage and maintain the user’s interest. In contrast, virtual chatbots
typically lack these physical attributes and rely solely on textual or vocal
exchanges, which can limit their ability to create a symmetrical experience
and put them in a more passive role.

#### Multimodality

The concept of multimodality is natural to HRI due to different sensory
inputs. Recently this idea was transplanted into deep learning frameworks
including multimodal dialogue in NLP. The variety of fields where the concept
is used caused it to become blurry. For this survey, we assume the definition
of the modality as pathways of sensory experience for the perceiving entity.

Figure 2: Diagram of auditory, haptic, and visual modalities with examples of
processed information.

In human-robot interaction, different modalities can be used to convey
different types of information. For example, speech can be used to communicate
complex ideas and instructions, while gesture can be used to indicate simple
commands or express emotions. Touch can be used to provide feedback or convey
a sense of physical presence, while facial expressions can be used to
communicate emotions and attitudes. In their survey [Su et al., 2023] compile
the commonly used modalities in Human Robot interactions divided by input and
output. They order main modalities understood as information pathways into
visual, auditory, haptic, kinesthetic, and proprioceptive. We describe the
three former ones since they have a direct impact on the multimodal dialogue
systems. Kinesthetic ability is the ability of a robot to sense and decode
motion and movement and proprioception is the ability of an organism to sense
the position and orientation of its body parts. These two modalities are
currently more relevant to industrial robots and exoskeletons rather than
dialogue systems thus we do not analyze them in this survey.

There is a growing body of work on multi-modal dialogue systems, however, the
majority of the cases concern visual dialogue such as in [Sun et al., 2022, He
et al., 2020], which is the task that takes in images and text as inputs and
produces text or both text and images in the response turn, the most famous
example of such system nowadays is GPT4 [OpenAI, 2023]. Several works mention
dialogue incorporating modalities different from static images and text such
as from [Frieske and Shi, 2024a]. After text and image, the next popular
modality is audio and speech. The works on video and dialogue resulted in a
visual dialogue dataset that incorporates gaze tracking to dialogue systems
[Kamezawa et al., 2020], the feature crucial for social robot interaction.

##  III Design

The design of multimodal dialogue systems in robots largely adjusts itself to
the task of the robot, and so the majority of the implementation falls into
the NLP category of task-oriented dialogue. As of now, almost all robotic
systems use a kind of framework that allows for the integration of multiple
sensors and adjustment to the performed task rather than training it end-to-
end. Since the design of a social robot is an interdisciplinary undertaking we
propose a framework based on analysis of multiple design approaches that
allows us to treat the dialogue design process as a path through different
paradigms. We propose looking at the dialogue design not only as an
engineering task, but also include communication, psychological, and
linguistic perspectives. Our approach is based on the survey of the literature
as well as our experience in the design process of the social robot Grace.

###  III-A Cognitive Architectures

is a group of models that try to create a robot cognition framework based on
neuroscience. Some of the examples of cognition models are the standard model
of the mind (or standard model) proposed first by Laird et al. in 2013 [Laird
et al., 2017], ACT-R proposed by [Ritter et al., 2018], SOAR [Laird, 2012],
CLARION or DUAL. Cognitive architectures provide a structured approach to
developing intelligent systems by mimicking human cognitive processes. They
enable robots to perceive, reason, and act in complex environments, supporting
advanced dialogue capabilities.

Standard Model of the Mind: This model aims to capture general cognitive
functions such as perception, memory, learning, and decision-making, creating
a comprehensive framework for understanding and simulating human cognition.
ACT-R (Adaptive Control of Thought-Rational): ACT-R is a cognitive
architecture that simulates human thought processes in a structured and
hierarchical manner, focusing on the interaction between declarative and
procedural knowledge. SOAR: SOAR is designed to model general intelligence
through goal-directed behavior, using a problem-space computational model to
simulate human reasoning and learning. CLARION (Connectionist Learning with
Adaptive Rule Induction ON-line): CLARION integrates symbolic and subsymbolic
processes to model human cognition, emphasizing the role of implicit and
explicit knowledge in learning and decision-making. DUAL: DUAL is a cognitive
architecture that combines rule-based and connectionist approaches, aiming to
capture the parallel and interactive nature of cognitive processes.

During face-to-face interactions with robots personalised approaches and
remembering user data and new words account for fluent and natural
interaction. [Amiri et al., 2019] explore possibilities of statistical updates
of knowledge databases in dialogue systems that allow integrating user
information and new terms, exploring the

###  III-B Role Design Model

is the design paradigm that designs actions of both humans and robots in the
interaction. Conversely, to certain systems that treat humans as a ’black box’
that can create every possible scenario, this model looks into the task that
the robot has to perform and tries to create a finite number of interaction
paths for both humans and robots. It takes inspiration from the communication
theory and the idea that interaction with the robot can be symmetrical and
asymmetrical [Frijns et al., 2021]. However the idea comes naturally from the
need to design the script of the interaction itself before getting into the
details of the architecture, it accepts the user as an integral part of the
dialogue design rather than treating him only as a creator of an input.

[Mehlmann et al., 2016] look at the dialogue system as a series of actions
assigned to humans and to robots rather than only potential robotic responses.
By looking at the dialogue act from both human and robot perspective they
assign certain roles to both parties in the same way drawing the limits of the
interaction.

In their 2022 survey [Berzuk and Young, 2022] analyzed 75 articles on robotic
dialogue systems and categorized dialogue design along 3 axes: interlocutor
composition, dialogue structure, and discourse genres. Interlocutor
composition means the number of robots and humans interacting. Dialogue
structure informs whether dialogue is scripted and linear, can be more
flexible and branch out, or finally depends on user input and be open domain
and unstructured. Discourse genres finally are their linguistic approach that
divides dialogue intentions into commanding, questioning, informing, and
entertaining.

###  III-C Linguistic Model

This approach views speech as an action rather than just conveying
information. It accommodates the initiative for communication from both the
user and the robot, categorizing actions into giving and receiving
information. The Linguistic Model focuses on the pragmatic aspects of language
use, such as speech acts, conversational implicatures, and context-dependent
meaning.

Speech Acts: Speech acts theory categorizes utterances based on their function
in communication, such as asserting, questioning, requesting, or commanding.
Conversational Implicatures: This concept refers to the implied meanings and
intentions behind spoken language, which are often inferred from context and
shared knowledge. Context-Dependent Meaning: The meaning of an utterance can
vary based on the situational context, requiring the system to interpret
language dynamically and adaptively. [Fernández-Rodicio et al., 2020]
introduce so-called communicative acts (CA) and complex communicative acts
(CCA) as basic interaction units modeling dialogue between the robot and the
user. Their approach concentrates on linguistic understanding of speech as an
action rather than just a form of conveying information. CAs accommodate the
fact that initiative for communication can come both from the user and from
the robot, which differentiates social robots from assistants such as Google
or Alexa that do not take initiative. Furthermore, CAs are divided into those
giving and receiving information.

Design Paradigm | Examples | Approach | Articles
---|---|---|---
Cognitive Architectures |  | Standard Model, Soar,
---
ACT-R, DUAL, CLARION
| Neuroscience,
---
Psychology, Robotics
|  [Laird et al., 2017], [Ritter et al., 2018],
---
[Laird, 2012]
Role Design Model |  | Communication Theory |  |  [Frijns et al., 2021], [Berzuk and Young, 2022] ,
---
[Mehlmann et al., 2016]
Linguistic Model |  | Communicative Acts,
---
Acyclic Graphs
Linguistics | [Fernández-Rodicio et al., 2020]
Communication Flow |  | Constructive Dialogue Model,
---
Dialogue Manager
Computing, Communication Theory |  |  [Jokinen, 2018], [Jokinen et al., 2018],
---
[Reimann et al., 2023], [Abbasi et al., 2019],
[Lala et al., 2021], [Ishii et al., 2021],
[Amiri et al., 2019], [Milhorat et al., 2018]
Activity System Model |  | Psychology |  |  [Leontiev, 1978], [Tewari and Lindgren, 2022],
---
[Abbasi et al., 2019] , [Amiri et al., 2019]
Integrated Design Model |  |  | Psychology, Communication Theory,
---
Robotics
|  [Shi et al., 2019], [Chen et al., 2013],
---
[Wang et al., 2022], [Wang et al., 2023],
[Baghaei et al., 2021], [Sakakibara et al., 2017],
[Bickmore et al., 2005], [Shibata and Wada, 2011],
[Wright, 2023], [Jean et al., 2008],
[Liu et al., 2016], [Johnson et al., 2016],
[Foster et al., 2016], [Liang and Hwang, 2023],
[Majgaard, 2015], [van Ewijk et al., 2020]

TABLE II: Breakdown of the articles on social robot design and application
based on the categorized design paradigms. This table categorizes cited
authors from the article according to their respective design paradigms,
highlighting key contributions to the development and application of
multimodal dialogue systems.

###  III-D Communication Flow

Communication flow provides a framework for managing the exchange of
information in a dialogue, ensuring that interactions are coherent,
contextually relevant, and goal-oriented.

Awareness and Perception: Both the human and the robot need to be aware of
each other’s communication attempts and perceive multimodal signals
accurately. Understanding and Interpretation: The system must be capable of
interpreting the meaning of signals and utterances based on context, previous
interactions, and multimodal cues. Response and Reaction: The robot’s
responses should be timely, relevant, and appropriate to the ongoing dialogue,
maintaining the flow and coherence of the interaction.

[Jokinen, 2018, Jokinen et al., 2018] look at the dialogue from the
perspective of roles and capabilities of robot and user, i.e. what human and
robot should be able to do within the interaction. Building on this notion
they use Constructive Dialogue Model (CDM). In this model, conversation
progresses cyclically with encouragement to continue conversations. Both
parties need to be in contact and aware of the partner’s attempt to
communicate. They need to perceive all the multimodal signals, understand
their meaning, and react appropriately. Therefore modules of such a dialogue
system are divided into contact, perception, understanding, and reaction
modules. This system was specifically designed for humanoid Kristina, based on
work with another humanoid, ERICA, [Milhorat et al., 2018] propose a question-
answering system with a fallback to generate more naturally sounding
dialogues.

In their survey [Reimann et al., 2023] look at the human-robot dialogue
specifically from the perspective of dialogue management. Dialogue management
controls turn-taking between users and robots, that is controls communication
exchange between them.

###  III-E Activity System Model

Derived from activity theory, this model focuses on the purposeful activity
defined by its object. It emphasizes the use of mediating tools to affect and
transform the activity’s object, providing a deeper understanding of human-
robot interaction. The Activity System Model highlights the dynamic and
situated nature of dialogues, considering the broader context of activities
and social interactions.

Activity Theory: Activity theory, first proposed by [Leontiev, 1978], provides
a framework for analyzing human actions as purposeful and goal-directed,
influenced by social, cultural, and material factors. The goal of the activity
is to fulfill an individual’s needs. Mediating Tools: These tools, such as
language, gestures, and artifacts, mediate the interaction between the human
and the robot, facilitating communication and task completion. Object of
Activity: The object of the activity defines the purpose and direction of the
interaction, shaping the roles, actions, and outcomes of the dialogue. This
interplay between the three core components of human activity: subject,
object, and tool is often visualized in the form of a triangle. Through their
analysis of older and middle-aged adults’ interaction with robots [Tewari and
Lindgren, 2022] discovered four themes influencing HRI: expecting,
understanding, relating, and interacting. The themes span complex human
activity at different complementary levels and provide a further developed
understanding of breakdown situations in human-robot interaction (HRI). In
their experiments, older and middle-aged adults emphasized emphatic behavior
and adherence to social norms, while younger adults focused on functional
aspects such as gaze, response time, and length of utterances.

[Abbasi et al., 2019] integrates pointing information with utterance and
passes it into the mediation module that defines actions that the robot needs
to take, which falls into the category of activity theory.

In their work with humanoid ERICA [Lala et al., 2021] analyzes dialogue system
performance for ERICA in several social roles to establish the extent to which
the robot could be used formally.

###  III-F Our Novel Paradigm: Integrated Design Model

Our novel paradigm combines the operational dimension, emotional dimension,
and communicational dimension of design, based on knowledge distilled from the
analyzed articles. This integrated approach ensures a holistic design of
multimodal dialogue systems for social robots, balancing functionality with
user experience.

#### Operational Dimension

includes the technical and functional components required for the robot to
perform tasks efficiently. It encompasses sensor integration, real-time
processing, and task-specific adaptations to ensure the system’s operational
efficiency and reliability. The most important question asked while designing
a robot is what is the task it is going to perform. Hence, task adaptation
customizes the system’s behavior and responses based on the specific tasks and
contexts in which the robot operates, enhancing its effectiveness and
relevance. The second important element of the operational dimension is module
integration. It is responsible for combining inputs from various sensors such
as cameras, microphones, and touch sensors to provide a comprehensive
understanding of the environment and user interactions. The real-time
processing aspect of operation ensures that the system can process multimodal
inputs quickly and accurately to respond in real-time, which is essential for
maintaining natural and fluid interactions.

#### Emotional Dimension

focuses on the robot’s ability to recognize, interpret, and respond to human
emotions, making interactions more engaging and empathetic. Emotion
Recognition: Utilizing facial expressions, voice tone, and body language to
detect and interpret the user’s emotional state. One of the most important
design aspects in this dimension is designing the robot’s responses to
acknowledge and appropriately address the user’s emotions, fostering a sense
of understanding and connection. By modifying the robot’s behavior and
interaction style based on the user’s emotional state, we can ensure a
supportive and positive interaction experience. Conceptualizing mirroring
strategies is essential for a social robot to showcase its empathetic
capabilities.

#### Communicational Dimension

includes the design of the dialogue flow, linguistic processing, and
multimodal interaction management to ensure effective and natural
communication. Design of the dialog flow means structuring the conversation to
be coherent, contextually relevant, and goal-oriented, maintaining the user’s
engagement and satisfaction. Turn-taking and barging in are elements of
communication that need to be conceptualized on this level of the design.
Handling natural language understanding and generation to interpret user input
accurately and generate appropriate responses depends on linguistic
processing, this consists of elements such as semantic parsing. Finally, by
coordinating inputs and outputs across different modalities (e.g., speech,
gestures, touch), we create a seamless and intuitive interaction experience.
Hence multimodal fusion is an element that appears in this design level.

##  IV Evaluation of Social Robot Designs

Just as with robotic design, a comprehensive evaluation of robotic dialogue
systems is a complex task due to the software and hardware involved, as well
as depending on the task that the robot is dealing with. There exist several
evaluation frameworks designed for robots but they refer to the interaction of
operators with factory robotic interfaces [Scholtz, 2003, Apraiz et al.,
2023]. Due to the system complexity, there is a prevalence of qualitative
evaluation standards in robotics, where the human subjects evaluate what is
their impression of the particular robot [Sirithunge et al., 2021]. The
quantitative approach mostly assesses metrics from particular submodules.

###  IV-A Quantitative Evaluation

According to [Deriu et al., 2021] automatic evaluation of the dialogue system
should be repeatable, correlated to human judgments, should differentiate
between different dialogue systems, and should be explainable. Some evaluation
metrics aim to evaluate task-oriented dialogue systems, hence the main
objective is evaluating the success rate of the task i.e., successful non-
interrupted interaction. Such metrics include the number of turns needed for
the successful interaction, including information about the backchanneling
turns that result from barging in. Finally, it combines the error rates from
modules processing different modalities, such as ASR or CV to evaluate their
impact on the overall system.

[Abbasi et al., 2019] in their work concerning task-oriented dialogue with
pointing and object recognition combine qualitative and quantitative
evaluation. In their work they evaluated qualitatively system as an entity
using the rate of successful interactions as well as particular modules, e.g.
they used word error rate (WER) for speech recognition as well as their
measure of serious speech recognition errors (SSRE) that was defined as speech
errors that are alter the meaning of the sentence so that task cannot be
completed. Pointing errors were shown by the percentage of not-recognized
gestures or the ones that were not tagged with the correct location. The
performance of the multimodal fusion module was evaluated by comparing
generated dialogue acts tags with manually labeled dialogue acts of all
utterances.

For the dialogue module alone the most standard metrics are usually F1,
accuracy, or precision. [Ni et al., 2022] differentiate the dialogue
evaluation between open domain and task-oriented dialogue. The former type is
mostly evaluated with BLEU, Rouge, and METEOR. The latter can be evaluated by
the Task Completion Rate or Task Completion Cost. Task-oriented dialogue is
the dialogue type mostly implemented in social robots, and the Task Completion
Rate can be easily applicable to most of the use cases. [Yeh et al., 2021]
test multiple automatic dialogue metrics, however apart from BLEU, Rouge, and
METEOR, many of them are model dependent i.e. they are designed to evaluate
BERT-based models. There is a growing body of work on multi-modal dialogue
systems, however, the majority of the cases concern visual dialogue such as in
[Sun et al., 2022, He et al., 2020], who use F1, BLEU, NIST, Rouge,
perplexity, and image specific Frechet Inception Distance (FID) and Inception
Score (IS).

The combination of many subsystems creates a danger of error propagation in
social robots, that is the possibility that errors from one component will
influence the performance of others. Furthermore, with the common deployment
of deep learning models, errors such as hallucinations might become a real
threat [Ji et al., 2023], even more if they are not detectable by the common
metrics such as hallucination is ASR [Frieske and Shi, 2024b].

###  IV-B Qualitative Evaluation

For qualitative robot evaluation, the most widely used questionnaire is based
on the Godspeed questionnaire [Bartneck, 2023]. Often unrelated questionnaires
are used to evaluate aspects not mentioned in Godspeed. For instance, one
might add aspects of trust evaluation [Zhu et al., 2022], since this is the
element that is not used in the Godspeed questionnaire.

Human evaluation encompasses the subjective assessment of dialogue systems
from the perspective of users. Irrespective of the specific approach employed,
all human evaluations involve soliciting input from external annotators who
respond to inquiries regarding dialogues.

Previous studies have exhibited considerable variation in the dimensions used
to evaluate dialogue systems through human evaluations. For instance, certain
studies [Tian et al., 2019, Zhou et al., 2019] focus on assessing the overall
quality of dialogue system responses. [Moghe et al., 2018] examines the
”Humanness” of responses, capturing the extent to which they appear human-
like. Emotion [Li and Sun, 2018] and Empathy [Lin et al., 2019] dimensions
encompass the evaluation of both the understanding and generation of emotional
responses. Furthermore, [Zhang et al., 2018] directs attention to
”engagingness”, evaluating whether the response includes interesting content.
Proactivity, used in [Wu et al., 2019], pertains to whether the response
introduces new topics while maintaining coherence. [Liu et al., 2018, Wang et
al., 2020] examined accuracy, which measures the correctness of responses
based on real-world knowledge, and, knowledge relevance [Liu et al., 2018,
Wang et al., 2020], which investigates whether the knowledge conveyed in the
response aligns appropriately with the context.

##  V Integrated Design for Social Robot Applications

The application of social robots spans fields that benefit from empathetic
mirroring in technology. In this section, we list current and possible
deployments of functioning social robots and analyze them through the
perspective of our integrated design model to showcase their most important
elements: functionality, emotional responsiveness, and communicational
agility.

###  V-A Healthcare

Social robots can be used in the field of healthcare, for example, to provide
emotional support for patients. A robot that can respond to a patient’s
emotional cues, such as facial expressions and gestures, can provide comfort
and companionship for the patient, which can be especially important for
patients who are isolated or lonely. Wright’s anthropological analysis
describes how robots influence elderly care in Japan, noting that the
operation of these robots can be time-consuming for nurses [Wright, 2023].

In the contemporary healthcare landscape, technology plays an increasingly
pivotal role, exemplified by innovations such as exoskeletons in
rehabilitation [Shi et al., 2019, Chen et al., 2013] and the application of
Virtual Reality (VR) in psychological care [Wang et al., 2022, Wang et al.,
2023, Baghaei et al., 2021]. Furthermore, social robots, which integrate
various forms of communication, are gaining traction in healthcare.

#### Operational Dimension

Social robots in healthcare integrate technical and functional components
necessary for task efficiency, such as sensor integration, real-time
processing, and task-specific adaptations. These systems enhance user
engagement by interpreting cues from various user groups, including the
elderly and individuals with dementia, and responding through multiple
modalities. For instance, exoskeletons and rehabilitation tools are rooted in
the physical embodiment of the robot, enhancing the operational capacity in
therapeutic settings [Shi et al., 2019, Chen et al., 2013]. Sakakibara et al.
developed a humanoid virtual caregiver capable of interacting with dementia
patients using voice, incorporating life history and linked open data to
facilitate person-centered conversations [Sakakibara et al., 2017].

#### Emotional Dimension

Social robots excel in recognizing, interpreting, and responding to human
emotions, providing emotional support, cognitive training, and soothing
companionship to users. Bickmore et al. observe that with the decline of
mental capabilities and sensory functions in the elderly, one skill that
people often retain is multimodal face-to-face communication [Bickmore et al.,
2005]. Shibata et al. introduced a unique robot with a seal-like appearance,
designed to interact with and provide companionship to the elderly, featuring
soft artificial fur that evokes a sense of warmth and comfort [Shibata and
Wada, 2011].

#### Communicational Dimension

These systems design dialogue flow, linguistic processing, and multimodal
interaction management for effective and natural communication. In their work
[Ishii et al., 2021] explored humanoid ERICA as a potential well-being
assistant with emotion recognition capacity for maintaining communication with
patients during COVID-19.

###  V-B Entertainment

In the domain of entertainment applications, we primarily observe two distinct
scenarios.

#### Operational Dimension

Technical and functional components include the integration of sensors and
real-time processing for interactive play with robots. For instance, Jean et
al. investigated a robot designed to exhibit behaviors such as singing,
crying, laughing, and looking around, focusing on user interactions for
general entertainment purposes [Jean et al., 2008]. Liu’s study on playing
finger guessing with a robot also highlights the operational capabilities [Liu
et al., 2016].

#### Emotional Dimension

Social robots in entertainment recognize and respond to human emotions to
enhance user engagement. Johnson et al. demonstrate that entertainment
experiences are enriched when robots provide expected multimodal feedback
during games, adding a new dimension to the entertainment value [Johnson et
al., 2016].

#### Communicational Dimension

Effective and natural communication is achieved through the design of dialogue
flow, linguistic processing, and multimodal interaction management. This is
exemplified by the project MuMMER (MultiModal Mall Entertainment Robot), led
by Foster et al., where a humanoid robot engages with customers in a shopping
mall through a combination of speech-based interaction, non-verbal
communication, and human-aware navigation [Foster et al., 2016].

###  V-C Education

Another application of multimodal dialogue systems is in the field of
education.

#### Operational Dimension

Robots can be used as teaching assistants, integrating sensor inputs, real-
time processing, and task-specific adaptations to help students learn new
concepts and skills. Liang et al. demonstrated that a robot digital
storytelling approach based on multimodal dialogue systems can facilitate
students’ speaking and storytelling abilities better than traditional one-way
videos [Liang and Hwang, 2023].

#### Emotional Dimension

These systems recognize and interpret student emotions to make learning
interactions engaging and empathetic. The NAO robot, used in a workshop led by
Majgaard Gunver, provided an engaging learning experience for students aged 11
to 16, teaching various subjects such as programming, language, ethics,
technology, and mathematics. Students enjoyed the interactive experience,
which enhanced their knowledge beyond the core curriculum [Majgaard, 2015].

#### Communicational Dimension

The design of dialogue flow, linguistic processing, and multimodal interaction
management ensures effective and natural communication. Gijs van Ewijk and
colleagues examined the ethical considerations of social robots in educational
settings, discussing privacy, security, psychological well-being, and user-
friendliness through targeted interviews with teachers. This research provides
an in-depth analysis of both the challenges and opportunities in the
development and design of educational robots [van Ewijk et al., 2020].

##  VI Challenges and Conclusion

The primary challenges in multimodal dialogue systems revolve around
effectively managing complex interactions across diverse domains, including
semantic parsing, turn-taking, and facial expression recognition. One key
difficulty is switching between task-oriented and open-domain interactions,
which requires a careful balance to avoid the rigidity of task-specific
systems and the potential biases of open-domain models. Semantic parsing
presents its challenges due to the variability and ambiguity of natural
language inputs, which can be compounded by multimodal inputs that may
conflict [Zhang et al., 2020]. Turn-taking involves accurately determining
when and how to shift communication between humans and robots, especially when
multiple modalities are involved [Liu et al., 2016]. Facial expression
recognition faces hurdles in achieving real-time accuracy in dynamic, real-
world settings and integrating subtle micro-expressions that reveal genuine
emotional states [Fang et al., 2023].

In addressing these challenges, incorporating various design approaches can
enhance the effectiveness and robustness of social robots. Balancing between
task-oriented and open-domain interactions involves integrating both rigid and
flexible paradigms to optimize performance and safety. Design approaches such
as hybrid models that blend neural and rule-based methods can address this
challenge by combining the structured reliability of task-oriented systems
with the adaptability of open-domain models. Data augmentation techniques, as
suggested by [Zhang et al., 2020], can further support this balance by
generating diverse training scenarios that help systems handle multiple
correct responses and minimize biases. By leveraging these design approaches,
systems can navigate the complexities of human interactions while maintaining
accuracy and relevance.

Semantic parsing, turn-taking, and facial expression recognition all benefit
from integrating specific design paradigms. For semantic parsing, employing
linguistic models and cognitive architectures can enhance the system’s ability
to interpret and integrate multimodal inputs accurately. Techniques such as
dependency parsing and semantic role labeling, combined with multimodal data
fusion, improve the system’s understanding of complex user inputs. Turn-taking
challenges can be addressed by analyzing communication flow models that ensure
smooth transitions and natural interactions. In terms of facial expression
recognition, taking an integrated design model as a starting point allows for
better planning of real-time processing and interpretation of expressions,
leading to more empathetic interactions. These design-focused strategies help
tackle the inherent difficulties in multimodal dialogue systems, creating more
effective and engaging human-robot interactions.

In conclusion, the design of social robots must integrate a variety of
paradigms to effectively address the complex challenges of multimodal
communication. By focusing on operational, emotional, and communicational
dimensions, designers can enhance the functionality and user experience of
robots. The operational design ensures that robots perform tasks efficiently
through sophisticated sensor integration and real-time processing. Emotional
design contributes to more engaging and empathetic interactions by enabling
robots to recognize and respond to human emotions. The communicational design
facilitates natural and effective dialogue by managing the flow of
conversation, linguistic processing, and multimodal interactions. Addressing
these dimensions holistically allows for the creation of social robots that
are not only task-oriented but also capable of delivering nuanced and
intuitive interactions. As research and technology continue to evolve,
incorporating these diverse design approaches will be essential in advancing
social robots that connect more deeply with users, making human-robot
interactions both more functional and more human-centered.

## References

  * [Abbasi et al., 2019] Abbasi, B., Monaikul, N., Rysbek, Z., Eugenio, B. D., and Žefran, M. (2019).  A multimodal human-robot interaction manager for assistive robots.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6756–6762.
  * [Amiri et al., 2019] Amiri, S., Bajracharya, S., Goktolgal, C., Thomason, J., and Zhang, S. (2019).  Augmenting knowledge through statistical, goal-oriented human-robot dialog.  IEEE International Conference on Intelligent Robots and Systems, pages 744–750.
  * [Apraiz et al., 2023] Apraiz, A., Mulet Alberola, J. A., Lasa, G., Mazmela, M., and Nguyen, H. N. (2023).  Development of a new set of heuristics for the evaluation of human-robot interaction in industrial settings: Heuristics robots experience (heurobox).  Frontiers in Robotics and AI, 10.
  * [Baghaei et al., 2021] Baghaei, N., Chitale, V., Hlasnik, A., Stemmet, L., Liang, H.-N., and Porter, R. (2021).  Virtual reality for supporting the treatment of depression and anxiety: Scoping review.  JMIR mental health, 8(9):e29681.
  * [Bartneck, 2023] Bartneck, C. (2023).  Godspeed Questionnaire Series: Translations and Usage, pages 1–35.
  * [Berzuk and Young, 2022] Berzuk, J. M. and Young, J. E. (2022).  More than words: A framework for describing human-robot dialog designs.  In Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’22, page 393–401. IEEE Press.
  * [Bickmore et al., 2005] Bickmore, T. W., Caruso, L., and Clough-Gorr, K. (2005).  Acceptance and usability of a relational agent interface by urban older adults.  In CHI ’05 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’05, page 1212–1215, New York, NY, USA. Association for Computing Machinery.
  * [Chen et al., 2013] Chen, G., Chan, C. K., Guo, Z., and Yu, H. (2013).  A review of lower extremity assistive robotic exoskeletons in rehabilitation therapy.  Critical Reviews™ in Biomedical Engineering, 41(4-5).
  * [Dautenhahn, 2007] Dautenhahn, K. (2007).  Socially intelligent robots: Dimensions of human–robot interaction.  Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1480):679–704.
  * [Deriu et al., 2021] Deriu, J., Rodrigo, A., Otegi, A., Echegoyen, G., Rosset, S., Agirre, E., and Cieliebak, M. (2021).  Survey on evaluation methods for dialogue systems.  Artificial Intelligence Review, 54:755–810.
  * [Fang et al., 2023] Fang, Y., Deng, D., Wu, L., Jumelle, F., and Shi, B. (2023).  Rmes: Real-time micro-expression spotting using phase from riesz pyramid.  arXiv preprint arXiv:2305.05523.
  * [Fernández-Rodicio et al., 2020] Fernández-Rodicio, E., González, Á. C., Alonso-Martín, F., Maroto-Gómez, M., and Salichs, M. A. (2020).  Modelling multimodal dialogues for social robots using communicative acts.  Sensors (Basel, Switzerland), 20.
  * [Fong et al., 2003] Fong, T., Nourbakhsh, I., and Dautenhahn, K. (2003).  A survey of socially interactive robots.  Robotics and Autonomous Systems, 42(3-4):143–166.
  * [Foster et al., 2016] Foster, M. E., Alami, R., Gestranius, O., Lemon, O., Niemelä, M., Odobez, J.-M., and Pandey, A. K. (2016).  The mummer project: Engaging human-robot interaction in real-world public spaces.  In Social Robotics: 8th International Conference, ICSR 2016, Kansas City, MO, USA, November 1-3, 2016 Proceedings 8, pages 753–763. Springer.
  * [Frieske and Shi, 2024a] Frieske, R. and Shi, B. E. (2024a).  Erit lightweight multimodal dataset for elderly emotion recognition and multimodal fusion evaluation.  arXiv preprint arXiv:2407.17772.
  * [Frieske and Shi, 2024b] Frieske, R. and Shi, B. E. (2024b).  Hallucinations in neural automatic speech recognition: Identifying errors and hallucinatory models.
  * [Frijns et al., 2021] Frijns, H. A., Schürer, O., and Koeszegi, S. T. (2021).  Communication models in human–robot interaction: An asymmetric model of alterity in human–robot interaction (amodal-hri).  International Journal of Social Robotics, 15(3):473–500.
  * [He et al., 2020] He, W., Li, Z., Lu, D., Chen, E., Xu, T., Huai, B., and Yuan, J. (2020).  Multimodal dialogue systems via capturing context-aware dependencies of semantic elements.  In Proceedings of the 28th ACM International Conference on Multimedia, MM ’20, page 2755–2764, New York, NY, USA. Association for Computing Machinery.
  * [Ishii et al., 2021] Ishii, E., Winata, G. I., Cahyawijaya, S., Lala, D., Kawahara, T., and Fung, P. (2021).  ERICA: An empathetic android companion for covid-19 quarantine.  In Li, H., Levow, G.-A., Yu, Z., Gupta, C., Sisman, B., Cai, S., Vandyke, D., Dethlefs, N., Wu, Y., and Li, J. J., editors, Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 257–260, Singapore and Online. Association for Computational Linguistics.
  * [Jean et al., 2008] Jean, J.-H., Chen, K.-T., Shih, K.-Y., and Lin, H.-L. (2008).  Design of an entertainment robot with multimodal human-robot interactions.  In 2008 SICE Annual Conference, pages 1437–1441. IEEE.
  * [Ji et al., 2023] Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. (2023).  Survey of hallucination in natural language generation.  ACM Comput. Surv., 55(12).
  * [Johnson et al., 2016] Johnson, D. O., Cuijpers, R. H., Pollmann, K., and van de Ven, A. A. (2016).  Exploring the entertainment value of playing games with a humanoid robot.  International Journal of Social Robotics, 8:247–269.
  * [Jokinen, 2018] Jokinen, K. (2018).  Dialogue models for socially intelligent robots.  undefined.
  * [Jokinen et al., 2018] Jokinen, K., Nishimura, S., Watanabe, K., and Nishimura, T. (2018).  Human-robot dialogues for explaining activities.  In International Workshop on Spoken Dialogue Systems Technology.
  * [Kamezawa et al., 2020] Kamezawa, H., Nishida, N., Shimizu, N., Miyazaki, T., and Nakayama, H. (2020).  A visually-grounded first-person dialogue dataset with verbal and non-verbal responses.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3299–3310, Online. Association for Computational Linguistics.
  * [Laird, 2012] Laird, J. E. (2012).  The Soar Cognitive Architecture.  The MIT Press.
  * [Laird et al., 2017] Laird, J. E., Lebiere, C., and Rosenbloom, P. S. (2017).  A standard model of the mind: Toward a common computational framework across artificial intelligence, cognitive science, neuroscience, and robotics.  Ai Magazine, 38(4):13–26.
  * [Lala et al., 2021] Lala, D., Inoue, K., Yamamoto, K., and Kawahara, T. (2021).  Findings from human-android dialogue research with erica.  1st RobotDial Workshop on Dialogue Models for Human-Robot Interaction.
  * [Leontiev, 1978] Leontiev, A. N. (1978).  Activity, consciousness, and personality.
  * [Li and Sun, 2018] Li, J. and Sun, X. (2018).  A syntactically constrained bidirectional-asynchronous approach for emotional conversation generation.  arXiv preprint arXiv:1806.07000.
  * [Liang and Hwang, 2023] Liang, J.-C. and Hwang, G.-J. (2023).  A robot-based digital storytelling approach to enhancing efl learners’ multimodal storytelling ability and narrative engagement.  Computers & Education, 201:104827.
  * [Lin et al., 2019] Lin, Z., Madotto, A., Shin, J., Xu, P., and Fung, P. (2019).  Moel: Mixture of empathetic listeners.  arXiv preprint arXiv:1908.07687.
  * [Liu et al., 2018] Liu, S., Chen, H., Ren, Z., Feng, Y., Liu, Q., and Yin, D. (2018).  Knowledge diffusion for neural dialogue generation.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489–1498.
  * [Liu et al., 2016] Liu, Z.-T., Pan, F.-F., Wu, M., Cao, W.-H., Chen, L.-F., Xu, J.-P., Zhang, R., and Zhou, M.-T. (2016).  A multimodal emotional communication based humans-robots interaction system.  In 2016 35th Chinese Control Conference (CCC), pages 6363–6368. IEEE.
  * [Majgaard, 2015] Majgaard, G. (2015).  Multimodal robots as educational tools in primary and lower secondary education.  In I The 9th International Conference on Interfaces and Human Computer Interaction, volume 22, page 24.
  * [Mehlmann et al., 2016] Mehlmann, G., Janowski, K., and André, E. (2016).  Modeling grounding for interactive social companions.  KI - Kunstliche Intelligenz, 30:45–52.
  * [Milhorat et al., 2018] Milhorat, P., Lala, D., Inoue, K., Zhao, T., Ishida, M., Takanashi, K., Nakamura, S., and Kawahara, T. (2018).  A conversational dialogue manager for the humanoid robot erica.  In International Workshop on Spoken Dialogue Systems Technology.
  * [Moghe et al., 2018] Moghe, N., Arora, S., Banerjee, S., and Khapra, M. M. (2018).  Towards exploiting background knowledge for building conversation systems.  arXiv preprint arXiv:1809.08205.
  * [Ni et al., 2022] Ni, J., Young, T., Pandelea, V., Xue, F., and Cambria, E. (2022).  Recent advances in deep learning based dialogue systems: A systematic survey.  Artif. Intell. Rev., 56(4):3055–3155.
  * [OpenAI, 2023] OpenAI (2023).  Gpt-4 technical report.
  * [Reimann et al., 2023] Reimann, M. M., Kunneman, F. A., Oertel, C., and Hindriks, K. V. (2023).  A survey on dialogue management in human-robot interaction.  arXiv preprint arXiv:2307.10897.
  * [Ritter et al., 2018] Ritter, F. E., Tehranchi, F., and Oury, J. D. (2018).  Act‐r: A cognitive architecture for modeling cognition.  WIREs Cognitive Science, 10(3).
  * [Sakakibara et al., 2017] Sakakibara, S., Saiki, S., Nakamura, M., and Yasuda, K. (2017).  Generating personalized dialogue towards daily counseling system for home dementia care.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 10287 LNCS:161–172.
  * [Scholtz, 2003] Scholtz, J. (2003).  Theory and evaluation of human robot interactions.  In 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the, pages 10 pp.–.
  * [Shi et al., 2019] Shi, D., Zhang, W., Zhang, W., and Ding, X. (2019).  A review on lower limb rehabilitation exoskeleton robots.  Chinese Journal of Mechanical Engineering, 32(1):1–11.
  * [Shibata and Wada, 2011] Shibata, T. and Wada, K. (2011).  Robot therapy: a new approach for mental healthcare of the elderly - a mini-review.  Gerontology, 57:378–386.
  * [Sirithunge et al., 2021] Sirithunge, C., Jayasekara, B., and D. Pathirana, C. (2021).  An evaluation of human conversational preferences in social human-robot interaction.  Applied Bionics and Biomechanics, 2021:1–13.
  * [Su et al., 2023] Su, H., Qi, W., Chen, J., Yang, C., Sandoval, J., and Laribi, M. A. (2023).  Recent advancements in multimodal human–robot interaction.  Frontiers in Neurorobotics, 17.
  * [Sun et al., 2022] Sun, Q., Wang, Y., Xu, C., Zheng, K., Yang, Y., Hu, H., Xu, F., Zhang, J., Geng, X., and Jiang, D. (2022).  Multimodal dialogue response generation.  In Muresan, S., Nakov, P., and Villavicencio, A., editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2854–2866, Dublin, Ireland. Association for Computational Linguistics.
  * [Tewari and Lindgren, 2022] Tewari, M. and Lindgren, H. (2022).  Expecting, understanding, relating, and interacting-older, middle-aged and younger adults’ perspectives on breakdown situations in human–robot dialogues.  Frontiers in Robotics and AI, 9.
  * [Tian et al., 2019] Tian, Z., Bi, W., Li, X., and Zhang, N. L. (2019).  Learning to abstract for memory-augmented conversational response generation.  In Proceedings of the 57th annual meeting of the Association for Computational Linguistics, pages 3816–3825.
  * [van Ewijk et al., 2020] van Ewijk, G., Smakman, M., and Konijn, E. A. (2020).  Teachers’ perspectives on social robots in education: an exploratory case study.  In Proceedings of the interaction design and children conference, pages 273–280.
  * [Wang et al., 2020] Wang, J., Liu, J., Bi, W., Liu, X., He, K., Xu, R., and Yang, M. (2020).  Improving knowledge-aware dialogue generation via knowledge base question answering.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9169–9176.
  * [Wang et al., 2022] Wang, X., Mo, X., Fan, M., Lee, L.-H., Shi, B. E., and Hui, P. (2022).  Reducing stress and anxiety in the metaverse: A systematic review of meditation, mindfulness and virtual reality.  In Proceedings of the 10th International Symposium of Chinese CHI (Chinese CHI 2022), Guangzhou, China.
  * [Wang et al., 2023] Wang, X., Mo, X., Lee, L.-H., Wei, X., Jin, X., Fan, M., and Hui, P. (2023).  Designing loving-kindness meditation in virtual reality for long-distance romantic relationships.  In Proceedings of the 31st ACM International Conference on Multimedia, pages 7608–7617.
  * [Wright, 2023] Wright, J. (2023).  Robots Won’t Save Japan An Ethnography of Eldercare Automation.  Cornell University Press, Ithaca, NY.
  * [Wu et al., 2019] Wu, W., Guo, Z., Zhou, X., Wu, H., Zhang, X., Lian, R., and Wang, H. (2019).  Proactive human-machine conversation with explicit conversation goals.  arXiv preprint arXiv:1906.05572.
  * [Yeh et al., 2021] Yeh, Y.-T., Eskenazi, M., and Mehri, S. (2021).  A comprehensive assessment of dialog evaluation metrics.  In Wei, W., Dai, B., Zhao, T., Li, L., Yang, D., Chen, Y.-N., Boureau, Y.-L., Celikyilmaz, A., Geramifard, A., Ahuja, A., and Jiang, H., editors, The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pages 15–33, Online. Association for Computational Linguistics.
  * [Zhang et al., 2018] Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., and Weston, J. (2018).  Personalizing dialogue agents: I have a dog, do you have pets too?  arXiv preprint arXiv:1801.07243.
  * [Zhang et al., 2020] Zhang, Y., Ou, Z., and Yu, Z. (2020).  Task-oriented dialog systems that consider multiple appropriate responses under the same context.  Proceedings of the AAAI Conference on Artificial Intelligence, 34:9604–9611.
  * [Zhou et al., 2019] Zhou, K., Zhang, K., Wu, Y., Liu, S., and Yu, J. (2019).  Unsupervised context rewriting for open domain conversation.  arXiv preprint arXiv:1910.08282.
  * [Zhu et al., 2022] Zhu, X., Zhang, X., Chen, Z., Dong, Z., Gu, Z., and Chang, D. (2022).  The trusted listener: The influence of anthropomorphic eye design of social robots on user’s perception of trustworthiness.  In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA. Association for Computing Machinery.


11institutetext: National Institute of Chemical Physics and Biophysics, Rävala
10, 15042 Tallinn, Estonia 22institutetext: IFISC, Instituto de Física
Interdisciplinar y Sistemas Complejos (CSIC-UIB), E-07122 Palma de Mallorca,
Spain 33institutetext: Dipartimento di Fisica, Università di Camerino, I-62032
Camerino, Italy

# Stochastic resonance in bistable confining potentials

On the role of confinement

Els Heinsalu 1122 Marco Patriarca 11 Fabio Marchesoni 33

(Received: date / Revised version: date)

###### Abstract

We study the effects of the confining conditions on the occurrence of
stochastic resonance (SR) in continuous bistable systems. We model such
systems by means of double-well potentials that diverge like $|x|^{q}$ for
$|x|\to\infty$. For super-harmonic (hard) potentials with $q>2$ the SR peak
sharpens with increasing $q$, whereas for sub-harmonic (soft) potentials,
$q<2$, it gets suppressed.

###### pacs:

05.40.-aFluctuation phenomena, random processes, noise, and Brownian motion
and 02.50.EyStochastic processes

##  1 Introduction

The simplest dynamical system displaying stochastic resonance (SR) is a
Brownian particle bound into a one-dimensional double well under the action of
a time oscillating tilt and subjected to fluctuating forces (noise)
Gammaitoni1998a ; Benzi1981a . The SR mechanism can be revealed as a maximum
in the amplitude of the periodic component of the average particle position as
a function of the noise intensity (temperature). Due to fluctuations, the
particle randomly jumps between the two potential wells with Kramers rate
borkovec  that depends on the double well potential and temperature. When the
average escape time of the particle out of the potential minima (i.e., the
inverse of the Kramers rate) approximately equals the half time-period of the
applied perturbation, the noise induced interwell jumps and the periodic force
synchronize, thus leading to SR.

When studying the problem of a Brownian particle in a symmetric double well
periodically tilted in time, the corresponding potential $U(x)$ is usually
assumed to diverge like $U(x)\sim x^{4}$ at large $x$ Gammaitoni1998a ;
borkovec , so as to ensure a robust confining action. However, the divergence
of the potential for $|x|\to\infty$ strongly affects the response of the
system to an external time-periodic forcing. The goal of the present paper is
to investigate how the Brownian motion in a double well changes with the
confining strength of the one-dimensional potential $U(x)$. For simplicity we
assume that $U(x)\sim|x|^{q}$ for $x\to\pm\infty$. By studying the dependence
of a SR spectral quantifier on $q$, we conclude that bistability is a
necessary, but not sufficient condition for a one-dimensional system to
exhibit SR.

##  2 Model

The model discussed in the following represents an overdamped Brownian
particle with coordinate $x$. Its dynamics is described by the Langevin
equation,

$\eta\dot{x}=-U^{\prime}(x)+A(t)+\xi(t)\,,$ (1)

where $(\dots)^{\prime}\equiv\mathrm{d}(\dots)/\mathrm{d}x$. The confining
potential,

$U(x)=U_{0}\exp\left(-x^{2}/L_{0}^{2}\right)+k|x|^{q}/q\,,$ (2)

is obtained by superimposing a Gaussian repulsive barrier of height $U_{0}$
and width $L_{0}$, to a power-law potential well. To ensure confinement, our
analysis is restricted to $q>1$. The total potential is mirror symmetric at
$x=0$, i.e. $U(x)=U(-x)$. Depending on $q$ a potential $U(x)$ is called hard
(super-harmonic) for $q>2$, or soft (sub-harmonic) for $q<2$ zannetti . The
periodic drive $A(t)$ is chosen as

$A(t)=A_{0}\cos(\Omega t)\,,$ (3)

with amplitude, $A_{0}$, angular frequency, $\Omega\equiv 2\pi\nu$, and time
origin arbitrarily set to zero. The fluctuating force $\xi(t)$ is modeled as a
stationary zero-mean Gaussian noise with auto-correlation function
$\langle\xi(t)\xi(t^{\prime})\rangle=2\eta
k_{\mathrm{B}}T\delta(t-t^{\prime})$. Here $T$ is the temperature and $\eta$
the friction coefficient.

For numerical purposes it is convenient to choose $U_{0}$, $L_{0}$, and
$\tau\equiv\eta L_{0}^{2}/U_{0}$ as the new units respectively of energy,
space and time. Correspondingly, the variables and the parameters appearing in
Eq. (1) can be replaced by the dimensionless quantities $\tilde{x}=x/L_{0}$,
$\tilde{t}=t/\tau$, $\tilde{k}=L_{0}^{q}k/U_{0}$,
$\tilde{A}_{0}=A_{0}L_{0}/U_{0}$, $\tilde{\Omega}=\Omega\tau$, and
$\tilde{T}=k_{B}T/U_{0}$. To avoid a cumbersome notation, in the following we
omit all the tildes. In dimensionless notation the potential (2) reads,

$U(x)=\exp(-x^{2})+k|x|^{q}/q\,,$ (4)

and the Langevin equation (1) can be rewritten as

$\displaystyle\dot{x}=2{x}\exp(-x^{2})-k|{x}|^{q}/x+{A}_{0}\cos({\Omega}{t})+\sqrt{{T}}\xi(t)\,,$
(5)

after the Gaussian noise $\xi(t)$ has been further rescaled so that
$\langle\xi({t})\rangle=0$ and
$\langle\xi({t})\xi({t}^{\prime})\rangle=2\delta({t}-{t}^{\prime})$. In the
following we study how changing $q$ influences the response of the particle to
the periodic forcing signal $A(t)$. As a result of rescaling, the height,
$U_{0}$, and the width, $L_{0}$, of the potential barrier, as well as the
friction coefficient, $\eta$, have been set to one. The remaining tunable
parameter $k$ of the potential (4) will be kept fixed to $k=0.2$ throughout
the present paper. Due to the Gaussian nature of the potential barrier, the
barrier height, $\Delta U$, and the potential minima, $\pm x_{m}$, weakly
depend on $q$ (see Fig. 1); therefore, the observed residual SR dependence on
$q$ is mostly an effect of the varying confining strength of the potential.

We have simulated the behavior of the system by numerically integrating the
rescaled Langevin equation (5) through a Milshtein algorithm Kloeden1999a ;
Milstein2004a . Stochastic trajectories were simulated for different time
lengths $t_{\mathrm{max}}$ and time steps $\Delta t$, so as to ensure
appropriate numerical accuracy and transient effects subtraction. Average
quantities have been obtained as ensemble averages over at least $10^{4}$
trajectories.

Figure 1: Rescaled potential (4) for $k=0.2$ and $q$ ranging between $1.5$ and
$8$. The barrier height is approximately constant, $\Delta U\simeq 0.66$, and
the minima $\pm x_{m}$ slowly shrink with $q$ from $x_{m}\simeq 1.59$ down to
$x_{m}\simeq 1.17$.

##  3 Results

In the long time regime, after transient effects subsided, the response
$\langle x(t)\rangle$ of a particle moving in a symmetric bistable potential
$U(x)$ under the action of the signal (3) with small-amplitude,
$A_{0}x_{m}\ll\Delta U$, and low-frequency, $\Omega\ll
U^{\prime\prime}(x_{m})$, results from the interplay of inter- and the
intrawell dynamics Gammaitoni1998a . On ignoring for the time being the
intrawell dynamics, the system response at low temperatures is dominated by
its harmonic component Gammaitoni1998a ; McNamara1989a ; Presilla1989a ;
Hu1990a ; Jung1991a

$\langle x(t\\!\to\\!\infty)\rangle=\bar{x}(T)\cos[\Omega t-\bar{\phi}(T)]\,,$
(6)

with amplitude, $\bar{x}(T)$, and phase, $\bar{\phi}(T)$, approximated by

$\displaystyle\bar{x}(T)$ $\displaystyle=$ $\displaystyle\frac{A_{0}\langle
x^{2}\rangle_{0}}{T}\frac{2r}{\sqrt{4r^{2}+\Omega^{2}}},$ (7)
$\displaystyle\bar{\phi}(T)$ $\displaystyle=$
$\displaystyle\arctan(\Omega/2r)\,.$ (8)

Here $r\propto\exp(-\Delta U/T)$ is the Kramers rate and $\langle
x^{2}\rangle_{0}$ the variance of the stationary unperturbed process $x(t)$
($A_{0}=0$), both temperature dependent quantities. The amplitude $\bar{x}(T)$
can be manipulated by tuning the noise level. Note that Eqs. (6)-(8) hold in
the linear response theory limit, only, i.e., for $A_{0}x_{m}\ll T$ and
$\Omega>r$ Jung1993 ; schneidman .

According to Eq. (7), in the limit $T\to 0$ the amplitude $\bar{x}(T)$
vanishes due to the potential barrier. The rate $r$ for the particle to
overcome the potential barrier decreases to zero exponentially when lowering
the temperature, that is $r\ll\Omega$. The interwell jumps are thus inhibited
and the particle gets locked in either minima with probability $1/2$; hence
$\lim_{T\to 0}\langle{x}\rangle=0$. In contrast, for high temperatures,
$T\gg\Delta U$, $r$ may grow much larger than $\Omega$ and, consequently,
$\bar{x}(T)\simeq\langle x^{2}\rangle_{0}/T$. For a hard potential with $q>2$
we show below that $\langle x^{2}\rangle_{0}\sim T^{2/q}$, so that, again,
$\lim_{T\to\infty}\bar{x}(T)=0$. The occurrence of these limits for $T\to 0$
and $T\to\infty$ implies the existence of a maximum of $\bar{x}(T)$ for some
optimal $T\sim\Delta U$. This is the so-called spectral characterization of SR
Gammaitoni1998a .

Figure 2: Rescaled amplitude $\bar{x}(T)/A_{0}$, defined in Eq. (6), versus
$T$ for the potential (4) with $k=0.2$, $q=2$. The dashed lines represent the
intrawell oscillations, Eq. (9), with $\kappa=1/|2k\ln(k/2)|$ for $T\to 0$,
and $\kappa=k$ in the limit $T\to\infty$.

###  3.1 Harmonic confining potentials

However, even if the approximate results (6)-(8) describe correctly the
occurrence of SR in most bistable systems, Figs. 2 and 3 ($q>1$) clearly show
that for $T\to 0$ the amplitude $\bar{x}(T)$ approaches a non-zero limit
$\bar{x}(0)>0$. This is a characteristic signature of the intrawell dynamics
Jung1993 ; schneidman . Moreover, for (and only for) $q=2$ a similar behavior
occurs also in the opposite limit $T\to\infty$: the curves $\bar{x}(T)$ attain
an horizontal asymptote, see Fig. 2. The coexistence of these two asymptotes,
peculiar to $q=2$, strongly suppresses the SR peak.

The nonzero $\bar{x}(T)$ limits for $T\to 0$ and $T\to\infty$ can be explained
by noticing that an overdamped Brownian particle bound to a generic harmonic
potential well, $U(x)=\kappa(x-x_{0})^{2}/2$, responds to the signal (3) with
amplitude

$\bar{x}=A_{0}/\sqrt{\Omega^{2}+\kappa^{2}}.$ (9)

[Note also that its variance in the absence of forcing ($A_{0}=0$) is $\langle
x^{2}\rangle_{0}=T/\kappa$.]

In the low temperature limit, $T\to 0$, the particle described by the Langevin
equation (5) is locked in either the right or left potential well, where it
executes additional harmonic oscillations around the corresponding minima
$x_{0}=\pm x_{m}$ Gammaitoni1998a ; Jung1993 ; schneidman ; lowD . Such
intrawell oscillations should not be mistaken for the interwell dynamics
described by Eq. (6) Hu1990a . Their amplitude is well reproduced by Eq. (9)
with $\kappa\equiv U^{\prime\prime}(\pm x_{m})=|2k\ln(k/2)|$.

In the high temperature limit, $T\to\infty$, the fluctuations $\xi(t)$ may
grow so intense that the barrier of the bistable potential (4) becomes
ineffective; the particle is thus effectively confined into a parabolic
potential with $\kappa=k$ and centered at $x_{0}=0$. The amplitude of the
periodic component of the particle response to the external force is then
described again by Eq. (9) but with $\kappa=k$.

For small frequencies the rescaled amplitude ${\bar{x}}/A_{0}$ only depends on
the curvature of the bistable potential at $x_{0}=\pm x_{m}$ for $T\to 0$,
${\bar{x}}/A_{0}=1/|2k\ln(k/2)|$, and at $x_{0}=0$ for $T\to\infty$,
${\bar{x}}/A_{0}=1/k$.

The argument above can be easily generalized for any value of $q$ at low
temperatures, but it becomes untenable in the limit $T\to\infty$, where
nonlinearity comes into play.

Figure 3: Rescaled amplitude $\bar{x}(T)/A_{0}$, defined by Eq. (6), versus
$T$ for the potential (4) with $k=0.2$ and different $q>2$ (hard potentials).
The dashed lines are the decay power law $T^{2/q-1}$.

###  3.2 Hard confining potentials

As anticipated above, at high temperatures the presence of the central barrier
can be ignored. This implies that for $T\to\infty$ Eq. (7) simplifies to

$\frac{{\bar{x}}(T)}{A_{0}}=\frac{\langle
x^{2}\rangle_{0}}{T}=\frac{1}{T}\frac{\int_{0}^{\infty}dx~{}x^{2}~{}\exp{(-kx^{q}/qT)}}{\int_{0}^{\infty}dx~{}\exp{(-kx^{q}/qT)}}.$
(10)

In Eq. (10) we made use of the inequality $r\gg\Omega$ and of the approximate
expression $P_{0}(x)={\cal N}\exp(-kx^{q}/qT)$ for the stationary probability
density of the unperturbed process (5); ${\cal N}$ is an appropriate
normalization constant. Note that for sufficiently low $\Omega$, the condition
$r\gg\Omega$ can be consistent with the approximations in Eq. (7), whereas
suppressing the potential barrier makes the very definition of $r$
meaningless.

An explicit calculation yields

$\frac{{\bar{x}}(T)}{A_{0}}=\left(\frac{q}{k}\right)^{2/q}\frac{\Gamma(3/q)}{\Gamma(1/q)}~{}\frac{1}{T^{1-2/q}}.$
(11)

Ignoring the algebraic factors we conclude that

$\displaystyle\lim_{T\to\infty}\bar{x}(T)\sim T^{\,2/q-1}\,.$ (12)

From here one can see that $\bar{x}$ decreases with increasing $T$ only for
hard confining potentials with $q>2$. In particular, for the prototypical case
of a quartic potential, $q=4$ Gammaitoni1998a , one finds $\bar{x}(T)\sim
1/\sqrt{T}$, as confirmed by the simulation results (see Fig. 3). For $q=2$,
one recovers the harmonic limit discussed in the foregoing subsection.

The decay law of $\bar{x}(T)$, Eq. (12), is clearly a consequence of the
nonlinearity of the potential. Indeed, the same power law can be recovered by
implementing the stochastic linearization scheme of Ref. bulsara : In Gaussian
approximation, for $q$ an integer, $\lim_{|x|\to\infty}U(x)=\kappa x^{2}/2$
with $\kappa=(q-1)!!k\langle x^{2}\rangle_{0}^{q/2-1}$; from the relation
$\langle x^{2}\rangle_{0}=T/\kappa$, holding for harmonic potentials, Eq. (12)
follows.

Moreover, $\bar{x}(T)$ cannot decrease faster than $T^{-1}$, which happens for
$q\to\infty$. It should be noticed that $\bar{x}(T)\sim T^{-1}$ is the decay
law predicted in two-state model approximation McNamara1989a , where $\langle
x^{2}\rangle_{0}$ is replaced by $x_{m}^{2}$ (i.e., a constant).

Figure 4: Rescaled amplitude $\bar{x}(T)/A_{0}$ versus $T$ for the potential
(4) with $k=0.2$ and $q=1.5$ (soft potential). The dashed lines represent the
horizontal asymptotes $1/\Omega$ (see text). In place of the SR peak an
inflexion point is detectable for low $\Omega=2\pi\nu$.

###  3.3 Soft confining potentials

Equation (12) for $q<2$ suggests that $\bar{x}(T)$ may diverge at high
temperatures. However, when dealing with soft potentials, the linear theory
approximations (6)-(8) must be used with caution. In the limit $T\to 0$ the
interwell oscillation amplitude (7) is known to apply only for very small
perturbation amplitudes zannetti : This explains the residual $A_{0}$
dependence of the low $T$ plateaus reported in Fig. 4.

More importantly, in the high $T$ limit, although the barrier of a soft
potential is awash with noise, confinement gets so weak that the particle is
driven up and down the potential walls primarily by the deterministic force
$A(t)$, rather than by the noise. [For a comparison, we remind that a particle
falls from $\pm\infty$ down to $\pm x_{m}$ in a finite time for $q>2$ and in
an infinite time for $q<2$.] In conclusion, on assuming that the Brownian
particle oscillates as if it were (almost) free, its amplitude would read

$\lim_{T\to\infty}\bar{x}(T)\sim A_{0}/\Omega\,.$ (13)

$\bar{x}(T)$ is then expected to develop high $T$ plateaus also for $q<2$,
but, in contrast with the cases discussed in Sec. 3.1, such plateaus are
inverse proportional to the drive frequency (also for low frequency drives,
see Fig. 4).

In the case of sub-harmonic bistable potentials the hallmark of SR is thus the
monotonic increase of the response amplitude with T, as opposed to the
occurrence of a maximum often detected in the super-harmonic potentials. Such
a behavior resembles the phenomenon of ”SR without tuning” discussed in Ref.
Collins1995 , with the important difference that here it has been observed in
a single unit, rather than in a summing network of $N$ excitable units.

##  4 Conclusions

We conclude this note with two important remarks:

(i) The coexistence of two locally stable minima separated by a potential
barrier is commonly advocated to explain the occurrence of a SR peak in a
continuous bistable dynamics. Here we have shown that this keeps being true as
long as the confining action exerted by the potential is super-harmonic. Most
notably, for harmonic and sub-harmonic potentials the periodic component of
the system response may increase monotonically with the noise level.

(ii) In many experimental reports (see, for a review, Ref. JSP ), the authors
tried to characterize the SR peak by means of Eq. (6), without paying much
attention to the $T$ dependence of the quantity $\langle x^{2}\rangle_{0}$. In
some cases they adopted an outright two-state model with $\langle
x^{2}\rangle_{0}=x_{m}^{2}$. This led to a poor fit of the decaying tail of
$\bar{x}(T)$, whereas a more accurate fit could have given a valuable clue to
better model the system at hand EPR .

## Acknowledgments

This work has been supported by the Estonian Science Foundation through Grant
No. 7466 (M.P., E.H.), Spanish MEC and FEDER through project FISICOS
(FIS2007-60327), and ESF STOCHDYN project (E.H.).

## References

  * (1) L. Gammaitoni, P. Hänggi, P. Jung, and F. Marchesoni, Rev. Mod. Phys. 70, (1998) 223.
  * (2) R. Benzi, A. Sutera, and A. Vulpiani, J. Phys. A 14, (1981) L453.
  * (3) P. Hänggi, P. Talkner, and M. Borkovec, Rev. Mod. Phys. 62, (1990) 251.
  * (4) F. Marchesoni, P. Sodano, and M. Zannetti, Phys. Rev. Lett. 61, (1988) 1143.
  * (5) P. Kloeden and E. Platen, Numerical Solution of Stochastic Differential Equations (Springer, Berlin, 1999).
  * (6) G. Milsten and M. Tretyakov, Stochastic Numerics for Mathematical Physics (Springer, Berlin, 2004).
  * (7) B. McNamara and K. Wiesenfeld, Phys. Rev. A 39, (1989) 4854.
  * (8) C. Presilla, F. Marchesoni, and L. Gammaitoni, Phys. Rev. A 40, (1989) 2105.
  * (9) Gang Hu, H. Haken, and C. Z. Ning, Phys. Lett. A 172 (1992) 21.
  * (10) P. Jung and P. Hänggi, Phys. Rev. A 44, (1991) 8032.
  * (11) P. Jung and P. Hänggi, Z. Phys. B 90, (1993) 255; J. Casado-Pascual, J. Gomez-Ordonez, M. Morillo, and P. Hänggi, Europhys. Lett. 58, (2002) 342.
  * (12) V. A. Shneidman, P. Jung, and P. Hänggi, Phys. Rev. Lett. 72, (1994) 2682.
  * (13) L. Gammaitoni, F. Marchesoni, E. Menichella-Saetta, and S. Santucci, Phys. Rev. E 51, (1995) R3799.
  * (14) A. R. Bulsara, K. Lindenberg, and K. E. Shuler, J. Stat. Phys. 27, (1982) 787.
  * (15) J. J. Collins, C. C. Chow, and T. T. Imhoff, Nature 376, (1995) 236.
  * (16) A. Bulsara, P. Hänggi, F. Marchesoni, F. Moss, and M. Shlesinger, Proceedings of the NATO ARW on Stochastic Resonance in Physics and Biology, J. Stat. Phys. 70, (1993) 1.
  * (17) L. Gammaitoni, F. Marchesoni, M. Martinelli, L. Pardi, and S. Santucci, Phys. Lett. A 158, (1991) 449.