[figure]style=plain,subcapbesideposition=top

# Data-driven Emotional Body Language Generation for Social Robotics

Mina Marmpena
University of Plymouth
Plymouth, UK
<EMAIL_ADDRESS>
&Fernando Garcia
ANYbotics AG
Zurich, Switzerland
<EMAIL_ADDRESS>
&Angelica Lim
Simon Fraser University
British Columbia, Canada
<EMAIL_ADDRESS>
&Nikolas Hemion
dSPACE GmbH
Paderborn, Germany
<EMAIL_ADDRESS>
&Thomas Wennekers
University of Plymouth
Plymouth, UK
<EMAIL_ADDRESS>

###### Abstract

In social robotics, endowing humanoid robots with the ability to generate
bodily expressions of affect can improve human-robot interaction and
collaboration, since humans attribute, and perhaps subconsciously anticipate,
such traces to perceive an agent as engaging, trustworthy, and socially
present. Robotic emotional body language needs to be believable, nuanced and
relevant to the context. We implemented a deep learning data-driven framework
that learns from a few hand-designed robotic bodily expressions and can
generate numerous new ones of similar believability and lifelikeness. The
framework uses the Conditional Variational Autoencoder model and a sampling
approach based on the geometric properties of the model’s latent space to
condition the generative process on targeted levels of valence and arousal.
The evaluation study found that the anthropomorphism and animacy of the
generated expressions are not perceived differently from the hand-designed
ones, and the emotional conditioning was adequately differentiable between
most levels except the pairs of neutral-positive valence and low-medium
arousal. Furthermore, an exploratory analysis of the results reveals a
possible impact of the conditioning on the perceived dominance of the robot,
as well as on the participants’ attention.

##  1 Introduction

In the near future more robots are expected to be deployed in social
environments such as schools, care homes and shops. Their ability to interact
socially with humans is becoming increasingly important [1] and as a
consequence the field of social robotics is receiving increasing attention.
Bartneck and Forlizzi [2] maintain that a social robot must adhere to “the
behavioural norms expected by the people with whom the robot is intended to
interact”, while Breazeal et al. [3] point out that social robots are expected
to communicate with humans in an interpersonal manner, to engage them as
partners, to collaborate or coordinate with them in order to accomplish
positive outcomes. Among the characteristics of socially interactive robots
identified by Fong et al. [4], we distinguish the ability to express and/or
perceive emotions, establish/maintain social relationships, and exhibit
personality and character. With our current work, we aim to facilitate these
aspects of a social robot by endowing humanoid robots with the capability to
display believable, diverse and appropriate emotional body language (EBL).

Body language is a subset of a broader group of nonverbal signals involved in
human communication. Other such signals include facial expression or
vocalizations. Facial expression is by far the most widely studied nonverbal
signal of affect, with affective bodily expression lagging behind [5],
although it constitutes a powerful communication channel among humans [6, 7,
8]. From a developmental point of view, body configurations are perceived from
a very early age [9], and they comprise the most primitive behavioural channel
for humans to express emotion [10]. Compared to facial expression, EBL can
reveal more about the actual affective state [11, 12, 10, 13], even when it is
contradicted by facial expression and verbal communication [11]. Gestures and
body postures evoke more trust [14], and they are less susceptible to social
editing compared to facial expression [15]. Changes in a communicator’s body
orientation, posture configuration, proxemics, etc. can influence the overall
likeability, interest, and openness between individuals [11, 12, 16, 13].
Bodily expression of emotion has also been found more effective than facial
expression in discriminating between intense positive and negative emotions
[17] and in communicating emotion from a distance [18, 19].

These insights on the effects of human EBL, taken together with the human
tendency to attribute anthropomorphic traits to non-human agents [20, 21, 22],
support the hypothesis that the integration of robotic EBL in human-robot
interaction (HRI) can enhance the user experience and make it more appealing
and engaging [23]. Evidence from studies in HRI show that bodily expressions
of affect can draw human’s attention [24], they are essential in naturalistic
social interaction for robots that lack expressive faces (appearance-
constrained) [25], and can make the interaction more enjoyable [26, 27].

Believable robotic EBL needs to be lifelike, adjusted to the robot morphology,
and granular to sustain attention in long-term interaction. Most importantly,
it needs to be targeted, in the sense that the robot must be able to chose
particular classes of emotions for different situations. In this work, we
implemented a Conditional Variational Autoencoder (CVAE), a deep learning
framework, to learn a latent, low-dimensional probabilistic representation of
EBL animations for a Pepper robot. The model was trained with a small set of
hand-designed expressions and its latent space can be sampled to generate
numerous new animations of targeted valence and arousal. We present the
extensive results of a user study designed to evaluate the interpretability of
the generated animations, how they compare with designed animations in terms
of anthropomorphism and animacy, and to what extent they are considered as
emotional and they draw users’ attention.

The paper is organized as follows: Section 2 briefly discusses related work
and Section 3 describes the theoretical aspects of the CVAE. In Section 4 we
outline the methodologies we used for structuring the training set and
sampling the CVAE, and we describe the design of the user study and the
statistical methods for the data analysis. The results are presented in
Section 5.

##  2 Related work

Robotic EBL design has attracted much attention in recent years since for many
robots, body motion is the main channel of non-linguistic expression.
Nonetheless, comparing the proposed methodologies can be challenging due to
differences in the design principles, the robot’s level of embodiment, the
modalities involved in the expressions, the type of the expressions (static
postures or dynamic animations), and the evaluation methodologies. Another
important aspect of variability is the emotion representation model adopted by
the researchers both in the development and the evaluation phase of the
synthetic EBL expressions. The most broadly used model is the categorical
representation, which is based on the discrete basic emotions theories.
According to it, there is a core set of distinct emotion categories, the basic
emotions, each corresponding to distinct brain locations or networks, each
manifesting with a specific feeling and neuro-physiological signature [28].
Proponents of the theory maintain that this multimodular emotion system is
universal within the human species and perhaps also shared by other primates
[29, 30]. They also consider the basic emotions as prewired responses to
different stimuli, an “affect program” written by the evolutionary process
through natural selection, that can be however somewhat influenced
epigenetically by learning. Arguably, the most frequently used emotion
categories in the robotic EBL research is Ekman’s list of six basic emotions:
anger, disgust, fear, happiness, sadness and surprise [31].

Another popular emotion representation in robotic EBL research is based on the
dimensional models of emotion, in which affect states are represented by one
or more continuous dimensions, each delimited by two polar states, such as
pleased-displeased for the dimension of valence, activated-deactivated for the
dimension of arousal, dominant-submissive for the dimension of control,
approach-avoidance for the dimension of motivation. The most broadly adopted
dimensional model is the circumplex model of emotion [32, 33], implemented as
a two-dimensional space defined by the valence dimension ranging on a
continuum from pleasure to displeasure on the horizontal axis, and the
dimension of arousal ranging from activation to deactivation on the vertical
axis. Russel and Barrett [32] suggest that these two affective dimensions may
be sufficient to capture the core affect, which becomes a full-blown emotion
when it is assessed within a situational framework. This representation allows
us to depict core affect as a data point on this 2D space, which is visually
intuitive and readily informative, and makes clear how different emotions are
not independent from each other, but instead, they are variations of the same
two variables. Another popular dimensional model is the PAD emotional state
model [34], which uses an additional dimension, dominance, to support more
nuanced emotions, and discern states like anger and fear which are both of low
valence and high arousal (i.e., fear is presumably of low dominance, while
anger is of high).

In the following brief review, we summarize related work in robotic EBL
synthesis under four main approaches: 1) Direct human imitation, 2) Feature-
based design, 3) Creative design, and 4) Deep learning generative models. In
many cases, researchers fuse elements from more than one approaches. The rest
of this section discusses briefly each approach and provides some
characteristic examples with a main focus on methods applied on humanoid
robots and studies that include a user evaluation in terms of emotion
recognition. For a broader systematic review on robotic animation techniques,
the interested reader is directed to Schulz et al. [35].

###  2.1 Direct human imitation

In the first approach, the robot joints are configured to match the human
joints in a single posture or motion setting. To accomplish this, computer
vision techniques, markers, or sensors are employed to track key joint
positions in human body motion, and then map them to the robot joint space
manually by observation [36, 37, 38], or by way of a transfer function [39].
Regarding the second method, although it has an advantage as an end-to-end
process, constructing good transfer functions can be very challenging because
of the extensive differences in the morphology and kinematics. To the best of
our knowledge, there are no studies so far applying it specifically for
robotic EBL synthesis.

###  2.2 Feature-based design

A prominent approach in robotic EBL design is to use features extracted from
human EBL. Such features can be found in studies coding postures or patterns
of movement from recordings of actors performing emotional expressions [40,
41, 14, 42, 43, 44], computer-generated mannequin figures [8], or human motion
capture data [45]. These studies provide lists of validated features, such as
body orientation, the symmetry of limbs, joint positions, force, velocity,
which can be employed for the design of robotic EBL expressions.

Regarding the emotion representation in humanoid robots, most of the studies
use the categorical model (e.g., Ekman’s basic emotions) to design their
robotic EBL expressions [46, 47, 48, 49, 50, 51], but some adopted the
dimensional representation in the evaluation [46]. The robotic platforms used
in these studies were Nao [46, 48], Daryl [47], Pepper and Hobbit [49], Brian
2.0 [50], and WABIAN-2R [52]. In the evaluation, some of the studies displayed
the designed expressions on a physical robot [46, 47], while others on a
virtual robot, videos or images of the physical robot [48, 50, 52]. The
evaluation results in these studies vary a lot, something that comes as no
surprise considering the differences in the platforms and the design
principles. Häring et al. [46] found that all the body movements were rated in
the right octant of the PAD model except sadness which was rated with positive
arousal. Embgen et al. [47] reported that participants were able to identify
the emotions from the EBL and Erden [48] that anger was recognized with 45%,
happiness with 72.5%, and sadness with 62.5%. Tsiourti et al. [49] found that
happiness was recognized more accurately, while sadness and surprise were
poorly recognized from body motion alone. McColl and Nezat [50] found that
sadness and surprise had the highest recognition rate ($>80\%$), while fear
and happiness had the lowest recognition rate ($<30\%$). Destephe et al. [52]
reported a high recognition rate for all the emotions (72.32% average).

Another feature-based paradigm for robotic EBL design is inspired by the Laban
Motion framework [53], based on kinesiology, anatomy, and psychological
analysis of human motion. Often, researchers use only a subset of the
framework, the Laban Effort System, which provides four motion parameters:
space, weight, time, and flow. These parameters have been used to handcraft
features and generate locomotion trajectories that express affect in non-
humanoid robots with low degrees of freedom [54, 55, 56, 57, 58], with
promising results in terms of readability by humans. For example, Sharma et
al. [55] used the four parameters to configure a set of flying patterns for a
Parrot AR.Drone, and they found in their dimensional evaluation that space and
time were significant predictors of valence, while all four Laban parameters
could predict arousal. Takahashi et al. [58] designed expressions to convey
the six basic emotions for a teddy bear robot using the Laban principles and
they reported that although fear and disgust had a low recognition rate, the
rest of the emotions were well recognised. The Laban Motion framework has also
been used to design EBL for humanoids [59, 60, 61, 62, 63]. Masuda et al. [59]
used Laban features (space, time, weight, inclination, height, area) to modify
three basic movements for the humanoid robot HFR to express emotion. They
found that sadness had the highest modulating effect.

Another interesting approach is to extract features from other human signals,
such as voice, and encode them into robotic motion. The SIRE model [64] was
proposed for generating robotic EBL based on four voice features: speed,
intensity, regularity and extent. The model was evaluated on a Nao robot using
the dimensional PAD model, and the results showed successful recognition of
happiness and sadness.

Feature-based design can also be inspired by animal models instead of human
EBL, an approach which can be potentially more effective with pet-like or
animal-like robots [65]. Lakatos et al. [66] designed emotions of fear and joy
for MogiRobi, a dog-like mechanical robot and they reported high recognition
rates for joy. Other interesting studies on emotion modelling for animal-like
robots include the seal-robot Paro [67, 68], the dog-robot AIBO [69], the cat-
robot NeCoRo [70]. Although these studies have not directly evaluated emotion
recognition and interpretability, they provide some indirect evaluation by
testing the effects of using these emotionally expressive robots on robot-
assisted therapy.

###  2.3 Creative design

Another influential approach in the design of robotic EBL emerges from
applying the artistic principles and practices used in cartoon animation. The
animator, conceives several expressive key postures for a given robot
morphology, configures the robot like a puppet according to these postures to
record the joints’ positions, and then interpolates to derive intermediary
positions so that the overall sequence appears continuous and smooth. This is
the pose-to-pose animation technique. The approach is more robot-centric, in
the sense that human EBL is not a direct prototype, and is often influenced by
Disney’s twelve basic principles of animation [71] producing very lifelike
expressions. In some studies, puppeteers [72, 73] or random participants [74,
75] are enlisted to configure robot EBL postures or sequences of postures
according to their subjective perception.

Monceaux et al. [76] describe their methodology of creating a big library of
dynamic multimodal emotion expressions as an iterative process involving
imitation from direct observation, and abstraction of key positions to adjust
them to the Nao robot morphology. Ribeiro and Paiva [77] adapted Disney’s
twelve basic principles of animation [71] for an EMYS head robot to express
the six basic emotions, on three different intensities. Their user study
evaluation found that anger, sadness and fear were very well recognized. The
principles of animation have also been used to design affect expression for
non-humanoid robots. Yohanan and MacLean [78] followed a pose-to-pose design
for the Haptic Creature, an animal-like robot. The modalities involved ears,
lungs and purr and the dimensional representation was adopted both in the
design and the evaluation. The design was found effective in conveying arousal
but ambiguous in the communication of valence.

###  2.4 Deep learning approaches

Lately, there have been some efforts to generate robotic EBL using deep
learning. Suguitan et al. [79] used human EBL for training three CycleGANs to
generate animations expressing happiness, sadness, and anger for a Blossom
robot with four degrees-of-freedom. They reported that the intended emotions
were fairly discernible. Marmpena et. al. [80] trained a Variational
Autoencoder (VAE) with a small set of robotic EBL animations created from
professional animators for a Pepper robot to generate numerous new ones. The
model was agnostic to the emotion class of the EBL, but after inspecting how
the sampled trajectories from the learned 3D spherical latent space are
decoded to the robot’s joints’ space, the authors propose that the latent
space radius can be used as a sampling parameter to modulate the arousal
content of the generated animations. Suguitan et al. [81] also proposed a
Variational Autoencoder, but they coupled it with a classifier to map the
latent space motion representations to an emotion class. Their setup allows to
modify a movement’s emotion class by using latent space arithmetic. The user
evaluation study showed that to some extent the modified movements are
comparable to the original in terms of recognizability and legibility.

###  2.5 Contribution

Previous research has been mainly focused on hand-coded methods which can be
tedious and expensive, thus resulting in a limited number of expressions,
characterised by less granularity in their expressivity. Consequently, the
expressions might appear repetitive and predictable which might be disengaging
in long-term human-robot interaction. Furthermore, many of the previous
studies used human EBL as a prototype, but scaling down the human motion range
or excluding joints to match the robot’s simpler morphology might end up
loosing information, or preserving redundant or trivial information with a
negative impact on the lifelikeness of the robot character.

Our current work proposes a data-driven deep learning methodology for the
automatic generation of numerous new robotic EBL animations, which appear
smooth, natural, and exhibit increased granularity. Instead of human EBL, the
prototype used is based on creative design which is directly adapted to the
robot morphology and kinematics to ensure the illusion of life effect.
Importantly, the proposed methodology seeks to generate animations of specific
emotion class, so that the robot can select appropriate responses to given
signals. Our work also seeks to leverage the dimensional representation of
emotion, and more specifically the circumplex model of core affect [32, 33]
with the dimensions of valence and arousal. We use the representation as an
integral component of the generative process.

Concretely, we build on the methodology proposed in [80], where a VAE model
was trained with a small set of hand-designed motion animations which were
created with direct adaptation to the Pepper robot morphology and motion
capabilities. From the same work, we adopt the proposed method to modulate
arousal by exploiting the geometry of the model’s latent space. For valence,
we expand the VAE model to a conditional VAE (CVAE) which takes as input a
label of valence as a continuous scalar value, and learns to adapt the
generated animations based on it. Furthermore, to improve the expressiveness
of the generated animations, we augmented the motion training set with
sequences of eye LEDs’ colourful patterns.

##  3 Conditional Variational Autoencoders

The Variational Autoencoder (VAE) framework [82, 83, 84] can be used to learn
a posterior probability distribution that represents the unknown underlying
process that generates the observed data. Subsequently, this distribution can
be sampled to generate purely novel content which is similar to the original.
The VAE is composed of an encoder and a decoder, which can be implemented with
any neural network architecture. During training, the vectorized input
$\bm{x}$ is passed through the encoder which compresses it into a lower-
dimension stochastic representation $\bm{z}$, which retains all the sufficient
information for a faithful reconstruction of $\bm{x}$. The distribution of all
$\bm{z}$ encodings defines a latent space, i.e., a continuous and structured
manifold in which similar datapoints are close to each other. Then, the
encoding $\bm{z}$ is passed though the decoder which uses the information
compressed in it to reconstruct the input.

In probabilistic graphical model formulation, the encoder is an inference
model $q_{\phi}(\bm{z}\mid\bm{x})$ which approximates the true but intractable
posterior. It infers the parameters of a predefined distribution, that is
$\bm{\mu}$ and $\bm{\sigma}$ in the case of a multivariate Gaussian with a
diagonal covariance matrix, which is the usual choice for real-valued output.
Subsequently, the parameters are used to sample a latent datapoint $\bm{z}$,
which is then passed through the decoder, a generative model
$p_{\theta}(\bm{x}\mid\bm{z})$. Both $\phi$ and $\theta$, the parameters of
the two models, are learned jointly during training. This is accomplished by
minimizing the following loss function:

$\mathcal{L}_{\theta,\phi}(\bm{x})=\underbrace{-\operatorname{\mathbb{E}}_{\bm{z}\sim
q_{\phi}(\bm{z}\mid\bm{x})}[\log{p_{\theta}(\bm{x}\mid\bm{z})}]}_{reconstruction~{}error}+\underbrace{D_{KL}[q_{\phi}(\bm{z}\mid\bm{x})\parallel
p_{\theta}(\bm{z})]}_{regularization~{}term},$ (1)

In Eq. 1, the reconstruction error is the negative expectation of the log-
likelihood taken over the latent parameters $\bm{z}$. It encourages the
$\bm{z}$ values that contribute to the faithful reconstruction of $\bm{x}$ by
the generative model. The regularization term (also called variational loss)
is the Kullback-Leibler divergence which estimates the disparity between the
latent distribution and a prior which is usually set to be a centered
isotropic multivariate Gaussian. It incentivizes the inference model to learn
$\bm{z}$ values that are close to the prior distribution, and this way it
applies a constraint on the latent space structure, resulting in a continuous
manifold from which we can sample and generate new realistic samples.

The VAE is a powerful generative model, but in its standard form, there is no
control on the class of the generated output. Controlling the generative
process so that the decoded output matches a given class can be very important
in certain applications. For example, in the present work, we want to be able
to generate animations of targeted valence. The Conditional Variational
Autoencoder (CVAE) [85], an extension of the VAE framework, does exactly what
we need by learning to make a discriminative prediction additionally to
reconstructing the input. This is accomplished by using an auxiliary variable
$\bm{c}$, essentially a continuous or discrete label, to condition the
encoding and decoding processed. The loss function of the CVAE modifies Eq. 1
as follows:

$\mathcal{L}_{\theta,\phi}(\bm{x},\bm{c})=-\operatorname{\mathbb{E}}_{z\sim
q_{\phi}(\bm{z}\mid\bm{x},\bm{c})}[\log{p_{\theta}(\bm{x}\mid\bm{z},\bm{c})}]+D_{KL}[q_{\phi}(\bm{z}\mid\bm{x},\bm{c})\parallel
p_{\theta}(\bm{z})].$ (2)

Fig. 1, illustrates the structure and computational flow of the CVAE. During
training, at each step, we concatenate the input $\bm{x}$ and the label
$\bm{c}$, and we feed them as one vector to the encoder. The output of the
encoder is reparameterized with some random noise $\bm{\epsilon}$ to produce
$\bm{z}$, which is concatenated with $\bm{c}$ again and fed to the decoder,
which outputs the reconstructed $\bm{x}$. After training, generating new
output with label $\bm{c}$ is accomplished by sampling a $\bm{z}$ from the
multivariate Gaussian prior, concatenation with $\bm{c}$ and decoding.

Figure 1: The structure and computational flow of the CVAE. Compared to the
standard VAE, there is an additional input $\bm{c}$ (green block) that holds a
label or an attribute on which we want to condition the generative process.

##  4 Methods and materials

This section presents the training dataset and the implementation of the CVAE
framework for robotic EBL generation, as well as the design of a user
evaluation study and the statistical methods applied for the analysis of the
collected data.

###  4.1 Training dataset

The dataset we used contains 36 robotic animations designed with the pose-to-
pose method by professional animators for the Pepper robot. Initially, each
animation consisted of three different modalities: motion, eye LEDs colour
patterns, and non-linguistic sounds. Motion is represented by a series of
keyframes, with each keyframe describing a posture defined by 17 values, i.e.,
the angles of the robot’s joints in radians (Fig. 2A). Each keyframe also has
a timestamp indicating after how many frames (starting from 0) the posture
should be executed. At runtime, the intermediary frame postures are obtained
with cubic Bezier interpolation applied locally at each joint. The eye LEDs
sequences are somewhat different in that their values can be set at any frame
for a specified interval and they remain static until the interval is over.
Pepper has two eyes with 8 LEDs each (Fig. 2B). Each LED is described by 3
values representing an RGB colour. Thus, in total, there are 48 LED values
within a range of $[0,1]$. For the current study, we excluded the audio
modality.

The particular set of animations has been previously annotated with valence
and arousal ratings in a user study with 20 participants [86]. In the present
study, we only use the valence ratings as labels to condition the CVAE as
described in Subsection 3. Each animation is annotated with a single real-
valued label of valence in $[0,1]$, where 0 indicates very negative valence
and 1 very positive. For arousal, we used a different method of conditioning,
based on the topology of the latent space (more details are provided in the
next subsection).

Figure 2: A) A Pepper robot has 17 joints that control the body configuration.
The drawing shows the StandInit posture. B) The eye LEDs of Pepper. The robot
has 8 RGB LEDs around each eye.

Our goal for the CVAE training was to use postures as input at each timestep
to allow the network to learn a continuous and structured manifold of
postures, which can be interpolated to generate smooth sequences of motion.
The initial keyframe representation was unsuitable for that, due to the abrupt
changes between subsequent keyframe postures, and also the small size of the
dataset. Furthermore, since the eye LEDs changes did not have the same onset
with the keyframe postures, a dataset with the initial representation would be
very sparse. To address these issues, we had the animations executed by the
real robot and we recorded the angles of the joints and the eye LEDs values at
a constant sampling rate of 25 frames per second, thus capturing both
keyframes and intermediary frames. Before the recording, we made a small
modification in the keyframe representation by adding an extra keyframe
describing the StandInit posture (see posture in Fig. 2) at the beginning and
the end of each animation (whenever it was missing), to obtain a clear onset
and offset of an animation. In total, the initial keyframe representation
dataset contained 697 keyframes, and with the recording we obtained 5074
frames. At this stage, each frame was a vector of 65 values: 17 values for the
angles of the joints and 48 for the eye LEDs state.

The next issue to address was related to the prevalence of possible body
orientations. For example, if many postures assume a right-side orientation,
the training could be heavily biased and the network might not learn enough
postures with left side orientation. We made the assumption that the
interpretation of the bodily expression of emotion is independent of left or
right side orientation with respect to the body’s vertical axis. This
assumption allowed us to augment the dataset by having every posture mirrored
from left to right or the opposite. We avoided any mirroring with respect to
the horizontal axis of the body, e.g., changing upward motion to downward or
vice versa, since such bias could affect the emotion content interpretation.
More specifically, to create a mirrored posture we swapped the values of the
entire chain of arm joints between the left and right arm, i.e shoulder pitch
and roll, elbow roll and yaw, wrist yaw, and hand. For the roll and yaw joints
of the arm, we had to additionally invert the signs of their values because
the ranges of these joints had inverse signs between left and right.
Furthermore, we also mirrored the head yaw and hip roll joints since these
also move right and left with respect to the vertical axis of the body. For
this mirroring, only the signs needed to be inverted. This augmentation
doubled the dataset to 10148 frames.

Regarding the valence attribute, we repeated the valence rating related to
each animation for all the frames belonging to it. Therefore, each training
example consists of 66 elements: 17 for the joints’ angles describing the
posture, 48 for the eye LEDs state, and one value indicating the valence
rating of the animation to which the frame belongs.

The last step in the preprocessing pipeline was the data normalization. The
ranges of plausible joint values differ among the joints, thus we rescaled all
the values in [0, 1] with Min-Max Normalization. This was not necessary for
the LEDs or the valence label since their range was already in $[0,1]$. Since
the implemented CVAE is agnostic to sequential dynamics of the dataset, we
shuffled the dataset and used 80% for the training, and the rest for the
validation.

###  4.2 CVAE network implementation

The CVAE framework for robotic EBL generation was implemented with Multilayer
Perceptrons (MLPs). The decoder MLP has three dense layers with 128, 512, and
128 units respectively, as well as an additional dense layer after the output,
serving as a reconstruction layer, with 66 units to match the input
dimensions. The capacity of the encoder was increased by one extra fully-
connected hidden layer with 512 units. This was decided after several
empirical tests that showed that this configuration was more efficient to
avoid posterior collapse111Posterior collapse is a well documented problem
arising in VAE training. It appears when the the model instead of learning
meaningful latent features from the input, learns to ignore the latent
variables resulting in a posterior distribution that collapses to the prior
(usually selected to be an uninformative prior, e.g., a unit Gaussian). More
concretely, posterior collapse indicates that the training is trapped into a
trivial local optimum where $q_{\phi}(\bm{z}\mid\bm{x})\simeq p(\bm{z})$ for
all $\bm{x}$. Empirically, the problem can be detected during training when
the variational loss $D_{KL}[q_{\phi}(\bm{z}\mid\bm{x})\parallel
p_{\theta}(\bm{z})]$ becomes almost zero.. After each hidden layer, we used a
layer of rectified linear activation functions (ReLU), while the output layers
were followed by linear activations. The reconstruction layer of the decoder
uses a sigmoid function to rescale the output in the range [0,1]. After each
hidden layer, we used a dropout layer, which randomly sets half of the units
to zero [87]. This was not necessary in terms of regularization, but it
improved the speed of the training.

For the latent layer, we used a strong bottleneck with only 3 dimensions. More
dimensions did not decrease the validation error significantly, while less
dimensions increased it. As we discussed in Section 3, the latent layer infers
the parameters of a multivariate Gaussian distribution (a 3D Gaussian in our
case), so it consists of two sub-layers of 3 units each, one for the latent
mean $\mu$ and another one for the latent standard deviation $\sigma$ used in
the diagonal covariance matrix. In terms of the loss function implementation,
the reconstruction error is equivalent to the Mean Squared Error (MSE) between
the input and the reconstructed output when we assume a Gaussian model with
real-valued output for the decoder. The regularization term can be computed
analytically when we assume Gaussian distributions for both the prior and the
approximate posterior [82]. Furthermore, the regularization term was weighted
with a $\beta$ coefficient [88, 89]. In general, the larger this
hyperparameter is, the stronger the pressure on the approximate posterior
$q_{\phi}(\boldsymbol{z}\mid\boldsymbol{x})$ to match the prior
$p_{\theta}(\boldsymbol{z})$. However, a large $\beta$ might also have a cost
on the reconstruction fidelity. After several empirical tests, we tuned the
hyperparameter to $\beta=0.001$. This value was found to allow a low
reconstruction error and also a well structured latent space.

The network weights were initialized with a Xavier uniform initializer [90],
i.e., the initial weight of each unit was sampled from a uniform distribution
with limits normalized by the sum of its input and output units. For the
optimization, we used an Adam optimizer [91] with a learning rate set to 1e-4.
Finally, we trained for 250 epochs.

###  4.3 Conditional sampling of the latent space

The simplest way to generate a new animation with the trained CVAE is to
randomly sample the prior distribution (centered isotropic 3D Gaussian) as a
proxy for the learned latent space, interpolate the samples, concatenate the
target valence label at each datapoint, and finally decode them into the joint
space. However, a more systematic sampling method proposed in previous work
[80] uses concentric spherical grids projected conceptually on the latent
distribution and samples latent trajectories along their longitudes. This work
showed that trajectories sampled from the inner spheres, closer to the core
(which is also the center of the latent zero-centered distribution), decode
into animations with lower amplitude and variance compared to animations
decoded from trajectories sampled from the outer spheres. Based on this
finding, it was suggested that the radius of the projected spheres can be used
as a topological feature that modulates the arousal content of a generated
animation. Therefore, to generate low arousal animations, we use a small
radius as standard deviation to sample the prior distribution, and when we
want to increase the arousal we increase the radius.

In the current work, we test this hypothesis about the arousal conditioning
but we modify the conceptual tool of the spherical grid into a torus. A
shortcoming of the spheres was that the decoded animations appeared to begin
and end abruptly, and always in the same robotic posture. This is due to the
fact that all the longitudes of a sphere end at the same point, the pole of
the sphere. Especially in larger spheres, this point is far away from the
origin where all three latent dimensions (LD1, LD2, LD3) are equal to zero,
i.e., the encoding that decodes into the StandInit position, the most neutral
and symmetric posture. By changing the spherical grid topology into a torus
grid, and more specifically, a horn torus (see Fig. 3), which is a torus
without a hole, with all the circles forming its tube touching each other at
the centre of the 3D latent space where $LD1=LD2=LD3=0$, we get a clear
advantage. Although all the longitudinal trajectories begin and end at the
same point, and unlike the poles of the sphere, this point is now located in
the core of the latent space, and thus, it decodes into the StandInit posture.
This way, we obtain a clear onset and offset of the animation and we still
keep the radius feature with which we will modulate the arousal dimension of
the generated animations.

Figure 3: An example of a horn torus grid used as a template for the
systematic sampling of latent trajectories. In this example, the radius is
equal to 3, and the axis is in parallel with the latent dimension LD3. All
torus grids have their center in the core of the latent space where
$LD1=LD2=LD3=0$. The latent trajectories are sampled along the colored
longitude lines.

To condition the generated animations on the arousal as hypothesized to be
modulated by the radius of the latent space, we defined three levels of
arousal (low, medium, and high), which were obtained by sampling the 3D latent
space with a radius of 3, 4, and 5 respectively. More concretely, we sampled 8
circular latent trajectories from torus grids rotated in three ways (so that
the central axis is parallel to one of three latent dimensions), and radiuses
of 3, 4 and 5. For example, Fig. 3 illustrates 8 circular trajectories from a
torus grid of radius equal to 3, and its central axis in parallel with LD3.
Note that in this approach we define radius as the distance from the centre of
the latent space to the outer point of the torus surface as if the whole torus
is enclosed in a sphere with the same centre. Each latent circle was defined
by 20 points. Subsequently, the 20 points of each latent trajectory were
interpolated using B-spline interpolation of 15, 20, 25 steps per segment for
radius 3, 4, and 5 respectively. We gradually increased the interpolation
steps as the radius increased to keep similar distances between the points of
each interpolant. In total, we derived 72 latent trajectories (8 longitudes *
3 radius values * 3 latent dimensions) with 20 points each.

Subsequently, to condition the generative process with the valence label, we
concatenated each of the 72 latent interpolants with 3 different values
representing levels of valence: 0 for negative, 0.5 for neutral, and 1 for
positive. Thus we obtain 216 latent trajectories which were passed through the
decoder of the CVAE to get the final library of 216 generated animations.
Essentially, the valence conditioning is achieved via the CVAE model
structure, while the arousal conditioning is accomplished explicitly based on
the topological features of the spherical latent space.

###  4.4 Evaluation study

This section presents the design of a user study aimed to evaluate the
interpretability of the CVAE generated animations with respect to the valence
and arousal conditioning. The statistical methods applied for the analysis of
the collected data are also presented.

####  4.4.1 Participant’s profile

A total of 20 volunteers were recruited for this study (9 female and 11 male).
The mean age was 26.4 years ($SD=5.36$, $min=21$, $max=44$). All participants
were employees of SoftBank Robotics Europe, in various professional roles,
e.g., engineers, administrative staff, marketing, etc. They self-reported
their experience with the robot’s animation capabilities on a scale ranging
from 0 (no experience at all) to 10 (extremely familiar). The mean self-
reported experience was equal to 6.35 ($SD=1.81$, $min=3$, $max=9$). The
experiment was carried out at the company’s premises in Paris, France. The
experimental protocol was granted ethical approval by the University of
Plymouth, Faculty of Science and Engineering Research Ethics Committee.

####  4.4.2 General setup

The experimenter briefly explained that the aim of this experiment was to
evaluate a set of body language animations displayed by Pepper. This would
require to observe the robot located in front of the participant,
approximately 2.5 meters away, and reply to the questions appearing on a
screen right in front of the participant. The session was split into three
parts, all of which were completed by each participant. In general, when
deciding on the number of trials, as well as the length of questionnaires, our
main concern was to keep participant’s workload as low as possible, so that
they could retain their concentration and motivation. The interface was
implemented as a website running on a local server and it was projected on the
screen. The participant would click on a play button to give the command for
the robot to execute an animation, and afterwards, at her own pace, she would
insert her responses.

####  4.4.3 Stimulus, interface, and questionnaires

The stimuli used for this experiment were animations displayed on a physical
Pepper robot. More specifically, we had two kinds of animations: designed
animations, designed by professional animators with the pose-to-pose method,
and generated animations, generated with the CVAE and the sampling method we
described in section 4.3.

Part A: This part aimed to compare the designed animations to the generated
ones in terms of the anthropomorphism and animacy degree attributed to the
robot. It consisted of two trials, in which participants watched the robot
executing 9 designed, and 9 generated animations. In both groups, the
animations were displayed continuously without a pause. The animations were
randomly selected in each group from the respective broader libraries of
designed and generated animations. The two trials appeared in randomized
order. After each trial, participants had to fill in a questionnaire comprised
of two scales, Anthropomorphism and Animacy, selected from the Godspeed
Questionnaire Series (GQS) [92]. Anthropomorphism is a measure of how
humanlike the robot appears to be. The scale comprises of five 5-point
semantic differentials (Fake/Natural, Machinelike/Humanlike,
Unconscious/Conscious, Artificial/Lifelike, and Moving rigidly/Moving
elegantly). Animacy is a measure of how lifelike the robot appears to be. The
scale comprises of six 5-point semantic differentials (Dead/Alive,
Stagnant/Lively, Mechanical/Organic, Artificial/Lifelike, Inert/Interactive,
Apathetic/Responsive).

Part B: This part involved only generated animations, which were evaluated on
three dimensions of emotion: valence, arousal and dominance. It consisted of
18 randomized trials. In each trial, a single generated animation was
displayed on the robot when the participant clicked on a digital play button.
After the animation was over, the participant would enter the dimensional
ratings on three different sliders as depicted in Fig. 4. The sliders were
titled with the two extremes of each measured emotion dimension: for valence
the title is Displeased - Pleased, for arousal Inactive - Active, and for
dominance Submissive - Dominant. We selected this verbal description as more
intuitive. The interface uses visual aids to illustrate the extremes in a more
intuitive way. The whole idea of this view is based on the Affective Slider
[93], a digital scale for the self-assessment of emotion, which we modified by
adding the extra dimension of dominance (Submissive - Dominant) and also by
making small changes to the original icons for valence and arousal to avoid
misreadings between valence and arousal. The participants were allowed to
click the play button and watch the animation for a second time if needed.
After watching the animation they could click on the sliders to enter their
ratings. Each slider was constructed with a range in $[0,1]$ and 100 points of
discretization. However, the numerical representation was hidden.

Figure 4: The view we used to collect valence, arousal and dominance ratings.
The participant clicks on the play button and watches the real robot executing
a generated animation. Subsequently, the user clicks on the three sliders to
enter the perceived valence, arousal and dominance scores. The concept is
adopted from the Affective Slider [93].

After submitting the ratings, a second view appeared in the same trial. It
contained two 5-point Likert scales and the play button. The two declarations
on the Likert scales were “The robot’s behaviour draws my attention” and “The
robot’s expression was emotional”. The responses for each of declaration were:
Strongly disagree, Disagree, Neither agree nor disagree, Agree, Strongly
agree. The participant could watch the animation once more if it was necessary
to recall it in the context of the new set of questions. The first Likert
declaration is adapted from [94], a study that proposes several dimensions and
related metrics to evaluate the believability of an artificial agent. The
authors of this study, propose a Likert scale with the template phrase, “<X>’s
behaviour draws my attention.” as a quantifiable metric of the visual impact
dimension of believability. The second Likert scale was constructed by us as a
manipulation check, to obtain a secondary, more direct and general assessment
on whether participants perceived a generated animation as an emotional
expression. The 18 generated animations used in Part B were selected randomly
from the broader set of generated animations, obtained as we described in
Section 4.3. We aimed to have 2 animations for each combination of valence and
arousal levels. The task was solicited with the question ‘How does the robot
feel?’. A practice period of 3 trials preceded it so that the participants
could get accustomed to the use of the sliders and the emotion dimension
concepts.

Part C: This part was identical to Part A in terms of structure but used
different designed and generated animations to minimize the chances of
confounds due to the choice of animations. Furthermore, the generated
animations used in Part A and C were excluded. Part C, was added as a pretest-
post-test design to examine if the results of the comparison between designed
and generated samples (on anthropomorphism and animacy) persist or change
after the participants become more familiar with the robot range of motion.

###  4.5 Statistical analysis

The statistical analysis of the collected data examines: 1) How valence and
arousal conditioning of the generative process influences participants in
their evaluation of the generated animations’ valence, arousal and dominance,
2) How the participants compare the generated vs the hand-designed animations
in terms of anthropomorphism and animacy, and 3) How valence and arousal
conditioning of the generative process influences the participants’ attention
and their belief that the animations express emotions.

####  4.5.1 Valence, arousal and dominance ratings

This analysis aims to explore how valence and arousal conditioning of the
generated animations affected the ratings of valence, arousal and dominance
assigned by the participants. Our two independent variables are the following:
1) v_cond with levels negative, neutral, and positive refers to the valence
conditioning of the CVAE generative process using an attribute $c$ equal to 0,
0.5, and 1 respectively, and 2) a_cond with levels low, medium, and high
represents the arousal sampling of the CVAE’s latent space using a radius $r$
equal to 3, 4, and 5 respectively.

There are three dependent variables, valence, arousal and dominance, which are
based on the corresponding slider ratings given by the participants. Since we
had multiple responses from a participant within the levels of the independent
variables (e.g., a participant rated 6 animations with v_cond = neutral), we
aggregated the ratings for each participant within each level of the two
independent variables v_cond and a_cond. The aggregation was based on the mean
within each level.

The main research question is whether the valence and arousal ratings given by
the participants are affected by the valence and arousal conditioning we used
to generate the animations. Specifically, we had two hypotheses:

  * •

H1: Valence ratings for animations generated with v_cond = negative will be
lower than those of animations generated with v_cond = neutral or v_cond =
positive. Valence ratings for animations generated with v_cond = neutral will
be lower than those of animations generated with v_cond = positive.

  * •

H2: Arousal ratings for animations generated with a_cond = low will be lower
than those of animations generated with a_cond = medium or a_cond = high.
Arousal ratings for animations generated with a_cond = medium will be lower
than those of animations generated with a_cond = high.

We also conducted an exploratory analysis to examine whether v_cond had an
effect on the arousal or dominance ratings, and similarly, if a_cond had an
effect on valence or dominance ratings. Since the experimental design was
within-subjects, i.e., each participant evaluated animations in each level of
the independent variables, we used repeated-measures one-way analysis of
variance (ANOVA). The test was decided to be one-way (with a single
independent variable at a time) because the data aggregation applied on the
levels of each independent variable could not be applied concurrently for both
independent variables without inflating the number of appearances of each
animation and inserting multiple missing values. In total, we applied 6 such
tests and we conducted post hoc comparisons between the levels of the
independent variables with paired samples t-tests, only for those
relationships that were found significant. The p values were adjusted with
Bonferroni correction for multiple comparisons error. Before proceeding with
the tests, we checked whether the necessary ANOVA assumptions hold. We checked
for outliers using boxplot methods. The normality assumption was tested with
the Shapiro-Wilk test and visual inspection of the QQ plots within each level
of the independent variables. The assumption of sphericity was checked with
Mauchly’s test and Greenhouse-Geisser sphericity correction was applied when
the assumption was violated.

####  4.5.2 Comparison of designed and generated animations

This analysis aims to examine the comparison between generated and designed
animations in terms of anthropomorphism and animacy. It also examines for
possible effects arising from the pretest-posttest experimental design (Part A
vs Part C as described in 4.4.3) to determine if the attribution of
anthropomorphism and animacy is altered after the participants become more
familiar with the robot’s body language. The main hypothesis for this analysis
is the following:

  * •

H3: The attribution of anthropomorphism and animacy to the robot is not
significantly different between the designed and the generated animations,
either in the pretest trials (Part A) or the posttest trials (Part C).

Initially, we computed Cronbach’s alpha to estimate the internal consistency
of the responses as a measure of reliability. An acceptable reliability would
permit to collapse the items of each scale to a single response per
participant and per scale using a measure of central tendency. The decision
was based on a threshold $\alpha>0.7$ which is widely considered as an
indication of acceptable internal consistency. Given that result, we proceeded
with taking the median of the items of each scale and each participant, i.e.,
one score for Anthropomorphism and another one for Animacy for each
participant. Since Likert scales provide ordinal data, the mean is not
considered a valid parameter to aggregate their items, and similarly,
parametric tests cannot be applied due to the normality assumption, thus we
used ordered logistic regression [95] to model the effects. For pairwise
differences of the estimated marginal medians, the p values of the contrasts
were corrected with Tukey’s multiple comparisons adjustment.

We fitted 3 models for each scale. The first one aimed to predict the Likert
scores using the group of the animation set (designed_pre, generated_pre,
designed_post, generated_post) as a predictor, and to examine if there are
differences between the generated and the designed animations, either in the
pretest phase or the posttest, according to H3. For the second model, we used
as a predictor only two groups (pre, post), each containing both designed and
generated animations, in order to explore if there are pretest-posttest
differences in general. The third model aimed to examine potential gender
differences affecting the overall scores of anthropomorphism and animacy.
Finally, we applied a likelihood ratio test to each model in order to test if
the proportional odds assumption of the ordered logistic regression holds.
This is a crucial assumption for ordinal regression analyses, and it asserts
that the independent variables have the same effect on the odds irrespective
of the splits between each pair of levels of the ordinal outcome variable. The
null hypothesis of the test upholds the proportionality of the odds, thus for
the assumption to be justifiable, it must not be rejected.

####  4.5.3 Attention and emotional content

This analysis examines the effect of the CVAE conditioning variables v_cond
(levels Negative, Neutral, Positive) and a_cond (levels Low, Medium, High) on
the 5-point Likert scores of two items: Attention (“The robot’s behaviour
draws my attention”) and Emotion (“The robot’s expression was emotional”). For
each of the Likert items, we fit an ordered logistic regression model and
checked for pairwise differences with Tukey’s adjustment of p values. The
proportional odds assumption is checked again with likelihood ratio tests.

###  4.6 Software

For recording and executing robotic animations with a Pepper robot we used
NAOqi 2.5 SDK222NAOqi 2.5: http://doc.aldebaran.com/2-5/home_pepper.html. For
the robot simulations we used the Choregraphe Suite333Choregraphe Suite 2.5:
http://doc.aldebaran.com/2-5/software/choregraphe/index.html. The code for the
CVAE model was adapted from the TFModelLib collection of neural
networks444TFModelLib repository: https://github.com/nhemion/tfmodellib. For
the implementation of B-splines we used Splipy555Splipy v1.3.1:
https://sintefmath.github.io/Splipy/. The rest of the code for the CVAE
implementation (data preprocessing, sampling, torus grids, etc.) was written
in Python 3 with packages such as NumPy [96], SciPy [97], pandas [98], scikit-
learn [99], Matplotlib [100]. The interface for the user study data collection
was written in Python 2 with Django v1.11.10666Django web framework:
https://www.djangoproject.com/. The statistical analysis of the collected data
was carried out in R [101, 102, 103, 104, 105, 106, 107, 108].

##  5 Results

###  5.1 Valence, arousal and dominance ratings

In Table 1, we present the descriptive statistics of the aggregated ratings
(valence, arousal, and dominance), with respect to the levels of the
explanatory variables v_cond and a_cond.

Table 1: Summary statistics for valence, arousal and dominance ratings Aggregated ratings | v_cond level | Mean | SD | a_cond level | Mean | SD
---|---|---|---|---|---|---
valence | Negative | 0.46 | 0.12 | Low | 0.47 | 0.1
Neutral | 0.54 | 0.09 | Medium | 0.54 | 0.11
Positive | 0.57 | 0.12 | High | 0.55 | 0.13
arousal | Negative | 0.55 | 0.12 | Low | 0.5 | 0.12
Neutral | 0.55 | 0.12 | Medium | 0.55 | 0.12
Positive | 0.58 | 0.11 | High | 0.63 | 0.14
dominance | Negative | 0.4 | 0.12 | Low | 0.42 | 0.11
Neutral | 0.46 | 0.1 | Medium | 0.44 | 0.09
Positive | 0.52 | 0.13 | High | 0.52 | 0.12

Regarding the normality assumption, only two extreme outliers were detected,
one in the relationship where v_cond is used to predict arousal, and a second
one where a_cond is used to predict valence. Since these relationships are not
substantial for our main hypotheses and we only have an exploratory interest
in them, we decided to proceed. All three dependent variables—valence, arousal
and dominance—were found normally distributed at each level of v_cond as
assessed by Shapiro-Wilk’s test ($p>0.05$). The same was true for arousal and
dominance with respect to the levels of a_cond, but valence was found to
violate the assumption for the a_cond level medium ($p=0.029$). Again, taking
into account that it is not in our main hypothesis to predict valence ratings
with the arousal conditioning of the CVAE, this violation of the normality
assumption was not of concern. Furthermore, one-way ANOVA is considered a
robust test against the normality assumption.

First, we will discuss the results for the tests in which we used the valence
conditioning of the CVAE (v_cond), to predict valence, arousal and dominance
ratings. The results revealed a significant effect of v_cond on the aggregated
valence ratings ($F(2,38)=13.5,~{}p<0.001,~{}\eta_{g}^{2}=0.16$) and on the
aggregated dominance ratings ($F(2,38)=12.6,~{}p<0.001,~{}\eta_{g}^{2}=0.16$).
No significant effect was detected on arousal
($F(2,38)=1,~{}p=0.38,~{}\eta_{g}^{2}=0.01$).

With regard to the arousal conditioning of the CVAE (a_cond), the one-way
repeated-measures ANOVAs detected significant effects on the aggregated
valence ratings ($F(2,38)=6.47,~{}p=0.004,~{}\eta_{g}^{2}=0.09$), the
aggregated arousal ratings
($F(1.5,28.42)=10.19,~{}p=0.001,~{}\eta_{g}^{2}=0.15$, with Greenhouse-Geisser
correction for the degrees of freedom due to sphericity assumption violation),
and the aggregated dominance ratings
($F(2,38)=16.159,~{}p<0.0001,~{}\eta_{g}^{2}=0.166$).

Table 2: Post hoc tests for valence, arousal and dominance ratings

DV | IV | Group 1 | Group 2 | t(df) | p | p (adjusted)
---|---|---|---|---|---|---
valence | v_cond | Negative | Neutral | $t(19)=-3.65$ | 0.002 | $0.005^{**}$
Negative | Positive | $t(19)=-4.47$ | $<0.001$ | $<0.001^{***}$
Neutral | Positive | $t(19)=-1.63$ | 0.12 | $0.36$
dominance | v_cond | Negative | Neutral | $t(19)=-2.06$ | 0.05 | $0.16$
Negative | Positive | $t(19)=-4.69$ | $<0.001$ | $<0.001^{***}$
Neutral | Positive | $t(19)=-3.46$ | 0.003 | $0.008^{**}$
arousal | a_cond | Low | Medium | $t(19)=-2.27$ | 0.04 | $0.11$
Low | High | $t(19)=-3.62$ | 0.002 | $0.005^{**}$
Medium | High | $t(19)=-2.88$ | 0.01 | $0.029^{*}$
dominance | a_cond | Low | Medium | $t(19)=-0.89$ | 0.381 | $1$
Low | High | $t(19)=-4.37$ | $<0.001$ | $<0.001^{***}$
Medium | High | $t(19)=-5.24$ | $<0.001$ | $<0.001^{***}$

Note: DV = Dependent Variable, IV = Independent Variable, t(df) = t statistic
and degrees of freedom, p = p values, p (adjusted) = p values with Bonferroni
correction.

Pairwise differences between the levels of conditioning were examined with
post hoc tests and are summarized in Table 2. We excluded the relationships
for which ANOVAs did not reveal significant effects (arousal with v_cond as a
predictor), and also those for which the normality assumption was violated
(valence with a_cond as a predictor).

For the effect of v_cond on valence ratings (Fig. 5A), paired samples t-tests
with Bonferroni correction revealed statistically significant differences for
Negative-Neutral ($t=-3.65$, $p=0.005$) and Negative-Positive ($t=-4.47$,
$p<0.001$). Thus, the results partially support the hypothesis H1, in that
animations generated with negative valence conditioning received lower valence
ratings compared to animations generated with neutral or positive valence
conditioning.

For the effect of a_cond on the arousal ratings (Fig. 5B), paired samples
t-tests with Bonferroni correction detected statistically significant
differences for Low-High ($t=-3.62$, $p=0.005$) and Medium-High ($t=-2.88$,
$p=0.029$). For Low-Medium a statistically significant difference was detected
($t=-2.27$, $p=0.01$), but it did not survive the Bonferroni adjustment. Thus,
the results partially support the hypothesis H2, in that animations generated
with high arousal conditioning received higher arousal ratings compared to
animations generated with low or medium arousal conditioning.

Figure 5: Post hoc t-tests with Bonferroni adjustment. A) Valence conditioning
impact on valence ratings. B) Arousal conditioning impact on arousal ratings.
The graph was originally published in [109]

.

Furthermore, we conducted post hoc tests to examine the pairwise differences
in the levels of v_cond and a_cond regarding their effect on the aggregated
dominance ratings. For v_cond, statistically significant differences were
detected for Negative-Positive ($t=-4.69$, $p<0.001$) and Neutral-Positive
($t=-3.46$, $p=0.008$) (Fig. 6A). For a_cond, statistically significant
differences were detected for Low-High ($t=-4.37$, $p<0.001$) and Medium-High
($t=-5.24$, $p<0.001$) (Fig. 6B). These results suggest that the CVAE
conditioning with v_cond and a_cond can potentially influence how people
perceive the level of dominance in the robot’s expression, with positive
valence or high arousal conditioning making the robot to appear more dominant
in terms of the simulated emotion.

Figure 6: Post hoc t-tests with Bonferroni adjustment for multiple
comparisons. A) Differences among the valence conditioning levels with respect
to the participants’ aggregated ratings of dominance. B) Differences among the
arousal conditioning levels with respect to the participants’ aggregated
ratings of dominance.

###  5.2 Comparison of designed and generated animations

We begin the presentation of the results with the descriptive statistics
presented as bar plots in Fig. 7 for the Anthropomorphism and Animacy scores.
The responses are from 1 to 5 (with 5 indicating that the participant
attributes a higher degree of Anthropomorphism or Animacy to the robot), and
the percentages represent the portion of participants in each response. The
basic contrast is designed vs generated animations presented in the pretest
phase (Part A of the session) and then again in the posttest phase (Part C).

For each scale (Anthropomorphism and Animacy), the internal consistency tested
with Cronbach’s alpha was found above our chosen threshold $\alpha\leq 0.85$.
Thus, we proceeded with collapsing the semantic differentials of each scale (5
items for Anthropomorphism and 6 for Animacy) in one score per participant and
per group, by taking the median value.

Figure 7: Anthropomorphism (A) and Animacy (B) bar plots for designed vs
generated animations in pretest and posttest phase.

In terms of the hypothesis H3 testing with ordered logistic regression, none
of the groups (designed_pre, generated_pre, designed_post, generated_post) was
found to have a statistically significant effect for Anthropomorphism or
Animacy (results are summarized in Table 3). This result supports our
hypothesis that participants do not attribute different Anthropomorphism or
Animacy levels to the designed animations compared to the generated ones,
either in the pretest or posttest phase. In Fig. 8, we plot the probabilities
derived from the predictions of the two ordered logit models of
Anthropomorphism and Animacy scores for each group of animations.

Table 3: Ordered logistic regression results for Anthropomorphism and Animacy

DV | IV | Coef. | SE | t | p | 95% CI
---|---|---|---|---|---|---
Anthropomorphism | generated_pre | -0.20 | 0.58 | -0.35 | 0.730 | (-1.34, 0.93)
designed_post | 0.57 | 0.58 | 0.98 | 0.328 | (-0.57, 1.73)
generated_post | 0.46 | 0.57 | 0.81 | 0.420 | (-0.66, 1.60)
Animacy | generated_pre | 0.38 | 0.67 | 0.57 | 0.572 | (-0.94, 1.71)
designed_post | 1.15 | 0.66 | 1.75 | 0.080 | (-0.13, 2.46)
generated_post | 0.81 | 0.69 | 1.18 | 0.240 | -0.54, 2.19)
Anthropomorphism | posttest | 0.61 | 0.41 | 1.49 | 0.135 | (-0.19, 1.42)
Animacy | posttest | 0.83 | 0.49 | 1.69 | 0.091 | (-0.12, 1.81)
Anthropomorphism | Male | -0.86 | 0.42 | -2.06 | $0.039^{*}$ | (-1.69, -0.05)
Animacy | Male | -0.45 | 0.49 | -0.92 | 0.359 | (-1.42, 0.5)

Note: DV = Dependent Variable, IV = Independent Variable, Coef. = Coefficient
of ordered logistic regression model, SE = Standard Error, t = t statistic, p
= p value, CI = Confidence Interval.

The two models of ordered logistic regression for Anthropomorphism and Animacy
with all the pretest and all the posttest animations as a two-level predictor
revealed some trends for Animacy (Fig. 9B), but no significant effects were
detected for either scale, suggesting that the interval in between the two
evaluations did not impact significantly the perception of Anthropomorphism
and Animacy. The detailed results are presented in Table 3.

Finally, regarding the two models that tested for gender effects, female
participants gave higher scores than male ($p=0.039$) on the Anthropomorphism
scale, but no statistically significant differences were detected for Animacy
($p=0.35$). The predictions’ probabilities are plotted in Fig. 10 and the
detailed results are included in Table 3.

The likelihood ratio tests checking for violations of the proportional odds
assumption did not obtain statistically significant p values for any of the
six models (all p values $\leq 0.08$ ), thus the assumption was considered
tenable.

Figure 8: Anthropomorphism and Animacy scores for designed vs generated
animations. The pre and post suffixes indicate the evaluation that was
conducted during Part A and Part C of the experimental session respectively.
Figure 9: Anthropomorphism and Animacy scores for all the animations evaluated
in Part A and Part C. A slight increase in the posttest phase, is not
confirmed with statistically significant results in pairwise comparisons.
Figure 10: Anthropomorphism and Animacy scores between male and female
participants. Female participants gave significantly higher Anthropomorphism
ratings.

###  5.3 Attention and emotional content

The descriptive statistics of the Attention and Emotion Likert scales with
respect to the valence and arousal conditioning levels are presented in Fig.
11 as bar plots with the exact percentage for each response. The two ordered
logistic regression models that predicted the Attention Likert scores (“The
robot’s behaviour draws my attention”) with the valence and arousal
conditioning of the CVAE as a predictor respectively, detected statistically
significant effects. The results are summarised in Table 4.

Figure 11: Bar plots for Attention and Emotion Likert scores with respect to
the valence (Negative, Neutral, Positive) and arousal (Low, Medium, High)
conditioning levels.

Post hoc tests on the levels of the valence conditioning (v_cond) determined
that the contrast Negative-Neutral was statistically significant ($z=2.77$,
$p=0.02$), suggesting that animations generated with negative valence
conditioning might attract more attention from participants compared to
animations generated with neutral valence. This result is illustrated in Fig.
12A, where we present the probabilities of predictions obtained by the model.
In the same figure, we observe that the same appears to be true for animations
generated with positive conditioning, however, this difference (Neutral-
Positive) did not survive the Tukey adjustment of the p value. The results are
summarized in Table 5.

Table 4: Ordered logistic regression results for Attention and Emotion

DV | IV | Coef. | SE | t | p | 95% CI
---|---|---|---|---|---|---
Attention | v_cond:Neutral | -0.11 | 0.17 | -0.64 | 0.52 | (-0.44, 0.22)
v_cond:Negative | 0.48 | 0.17 | 2.85 | $<0.001^{***}$ | (0.15, 0.81)
Attention | a_cond:Medium | 0.95 | 0.18 | 5.33 | $<0.001^{***}$ | (0.60, 1.30)
a_cond:High | 0.29 | 0.17 | 1.73 | 0.08 | (-0.04, 0.62)
Emotion | v_cond:Neutral | -0.36 | 0.17 | -2.10 | $0.04^{*}$ | (-0.69, -0.02)
v_cond:Negative | 0.74 | 0.17 | 4.39 | $<0.001^{***}$ | (0.41, 1.07)
Emotion | a_cond:Medium | 0.63 | 0.17 | 3.71 | $<0.001^{***}$ | (0.30, 0.97)
a_cond:High | 0.19 | 0.17 | 1.18 | 0.24 | (-0.13, 0.52)

Note: DV = Dependent Variable, IV = Independent Variable, Coef. = Coefficient
of ordered logistic regression model, SE = Standard Error, t = t statistic, p
= p value, CI = Confidence Interval.

Table 5: Post hoc tests for Attention and Emotion

DV | IV | Group 1 | Group 2 | z | p (adjusted)
---|---|---|---|---|---
Attention | v_cond | Negative | Neutral | 2.77 | $0.02^{*}$
Negative | Positive | 0.64 | $0.8$
Neutral | Positive | -2.16 | $0.08$
Attention | a_cond | Low | Medium | -1.34 | 0.37
Low | High | -5.33 | $<.001^{***}$
Medium | High | -4.17 | $<.001^{***}$
Emotion | v_cond | Negative | Neutral | 4.75 | $<.001^{***}$
Negative | Positive | 2.1 | $0.09$
Neutral | Positive | -2.81 | $0.01^{**}$
Emotion | a_cond | Low | Medium | -0.89 | 0.64
Low | High | -3.71 | $<.001^{***}$
Medium | High | -2.85 | $0.01^{**}$

Note: DV = Dependent Variable, IV = Independent Variable, z = z statistic, p
(adjusted) = p value with Tukey correction

Figure 12: Attention scores (“The robot’s behavior draws my attention”) for
different levels of valence (A) and arousal (B) conditioning. A) Pairwise
comparisons revealed significantly higher scores for negative conditioning
compared to neutral. B) Pairwise comparisons revealed significantly higher
scores for high arousal conditioning compared to both medium and low.

Post hoc tests with Tukey’s p value adjustment for the arousal conditioning
(a_cond) revealed statistically significant differences for the contrasts Low-
High ($z=-5.33$, $p<0.001$) and Medium-High ($z=-4.17$, $p<0.001$). The result
suggests that potentially animations sampled with larger radius from the
latent space tend to draw participants’ attention more. The differences are
presented in the model’s predictions in Fig. 12B and the results are
summarized in Table 5.

The two ordered logistic regression models predicting the Emotion Likert
scores (“The robot’s expression is emotional”) with the valence and arousal
conditioning of the CVAE as a predictor respectively, detected statistically
significant effects. The results are summarised in Table 4. Post hoc tests
with Tukey’s p value adjustment on the levels of the the valence conditioning
(v_cond) obtained statistically significant results for the contrasts
Negative-Neutral ($z=4.75$, $p<0.001$) and Neutral-Positive ($z=-2.81$,
$p=0.01$). The results imply (Fig. 13A) that animations generated with
negative or positive valence conditioning are perceived more as emotional
compared to the ones generated with neutral valence conditioning. In the
figure, a similar trend can perhaps be noted for Positive vs Negative, but
this difference did not remain significant after the correction for multiple
comparisons. The summary of the results is presented in Table 5.

For a_cond (arousal conditioning), statistically significant differences were
detected for the contrasts Low-High ($z=-3.71$, $p<0.001$) and Medium-High
($z=-2.85$, $p=0.01$). This result suggests that animations sampled with a
larger radius are perceived as more emotional. See also Fig. 13B, and Table5
for a summary of the results.

Regarding the proportional odds assumption, the likelihood ratio tests of the
model terms produced p values greater than 0.05 for all four models, thus the
null hypothesis cannot be rejected and the assumption is tenable.

Figure 13: Emotion scores (“The robot’s expression is emotional”) for
different levels of valence (A) and arousal (B) conditioning. A) Pairwise
comparisons revealed significantly higher scores for negative and positive
conditioning compared to neutral. B) Pairwise comparisons revealed
significantly higher scores for high arousal conditioning compared to both
medium and low.

##  6 Conclusion

In social robotics, human-robot interaction can be enhanced by endowing a
robot with the ability to communicate using emotional body language (EBL). For
robotic EBL to be engaging, it has to appear expressive, granular and
interpretable in terms of the emotional class. To this aim, we implemented a
Conditional Variational Autoencoder for the generation of multi-modal robotic
EBL of targeted valence and arousal. The model was trained with a small set of
robotic EBL animations designed for a Pepper robot by professional animators,
including motion and eye LEDs colour sequences. The valence content of the
generated animations was controlled by explicitly conditioning the training
samples with scalar valence labels, while arousal conditioning was achieved by
exploiting the geometry of the model’s latent space to guide the sampling
process.

The interpretability of the generated animations was tested with a user
evaluation study. The results provide support for valence conditioning which
appears effective in differentiating positive or neutral from negative
expressions. Similarly, arousal conditioning appears effective for
differentiating high arousal from medium or low. However, only weak evidence
were obtained for the differentiation between positive and neutral valence or
between medium and low arousal. This limitation might be related to the
composition of the training set, in which the representation of high valence
and low arousal animations was lower than the rest of the classes [86]. Such
imbalance could suggest a sample bias within the dataset, i.e., the model was
not trained with enough examples from these two categories and did not learn
to differentiate them well. This is an issue which can be resolved by
collecting more labelled animations of high valence and low arousal.
Nevertheless, this is consistent with previous studies using a NAO and a
Robovie [110, 111] also report limitations in creating EBL expressions that
are interpreted as of low arousal and high valence. Hence, it might be the
case that this affect subspace is inherently difficult to represent, or
perhaps emotional states from this affect subspace can be more successfully
conveyed through other modalities, such as facial expression.

The analysis also showed that the generated animations do not appear to be
less appealing compared to the designed animations in terms of
Anthropomorphism and Animacy, either in the pretest phase or in the posttest
phase. With regards to the impact of the robot’s behaviour on the user’s
attention (presumably an aspect of believability), the results demonstrate
that animations of more extreme levels of valence, or of high arousal, seem to
draw more attention. Similarly, these animations are more strongly classified
as emotional. A video with the physical robot executing several CVAE generated
animations is available online777CVAE generated animation set video:
https://youtu.be/wmLT8FARSk0. The animations used for training, as well as the
generated ones presented in the user study in this study can also be found
online888REBL-Pepper Dataset: https://github.com/minamar/rebl-pepper-data.

## Acknowledgments

This project received funding from the European Union’s Horizon 2020 research
and innovation program under the Marie Skłodowska Curie grant agreement No
674868 (APRIL).

## References

  * [1] Cynthia Breazeal.  Role of expressive behaviour for robots that learn from people.  Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535):3527–3538, December 2009.
  * [2] C. Bartneck and J. Forlizzi.  A design-centred framework for social human-robot interaction.  In 13th IEEE International Workshop on Robot and Human Interactive Communication, pages 591–594, 2004.
  * [3] Cynthia Breazeal, Kerstin Dautenhahn, and Takayuki Kanda.  Social robotics.  In Bruno Siciliano and Oussama Khatib, editors, Springer Handbook of Robotics, Springer Handbooks, pages 1935–1972. Springer International Publishing, Cham, 2016.
  * [4] Terrence Fong, Illah Nourbakhsh, and Kerstin Dautenhahn.  A survey of socially interactive robots.  Robotics and Autonomous Systems, 42:143–166, 03 2003.
  * [5] M. Karg, A. Samadani, R. Gorbet, K. Kühnlenz, J. Hoey, and D. Kulić.  Body movements for affective expression: A survey of automatic recognition and generation.  IEEE Transactions on Affective Computing, 4(4):341–359, 2013.
  * [6] A. Kleinsmith and N. Bianchi-Berthouze.  Affective body expression perception and recognition: A survey.  IEEE Transactions on Affective Computing, 4(1):15–33, 2013.
  * [7] Zachary Witkower and Jessica L. Tracy.  Bodily communication of emotion: Evidence for extrafacial behavioral expressions and available coding systems.  Emotion Review, 11(2):184–193, 2019.
  * [8] Mark Coulson.  Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence.  Journal of Nonverbal Behavior, 28(2):117–139, 2004-06-01.
  * [9] Teodora Gliga and Ghislaine Dehaene-Lambertz.  Structural encoding of body and face in human infants and adults.  Journal of Cognitive Neuroscience, 17(8):1328–1340, 2005.
  * [10] Julius Fast.  Body Language.  Henry Holt and Company, 1970.
  * [11] Michael Argyle.  Bodily communication.  Methuen, London, UK, 1975.
  * [12] Peter E. Bull.  Posture and gesture.  Pergamon Press, Oxford, UK, 1987.
  * [13] Spiegel John and Machotka Pavel.  Messages of the body.  Free Press New York, 1974.
  * [14] Nele Dael, Marcello Mortillaro, and Klaus R. Scherer.  Emotion expression in body action and posture.  Emotion, 12(5):1085–1101, October 2012.
  * [15] Paul Ekman and Wallace V Friesen.  Detecting deception from the body or face.  Journal of personality and Social Psychology, 29(3):288, 1974.
  * [16] Albert Mehrabian.  Nonverbal communication.  Aldine Transaction, New Brunswick, NJ, 2007.
  * [17] Hillel Aviezer, Yaacov Trope, and Alexander Todorov.  Body cues, not facial expressions, discriminate between intense positive and negative emotions.  Science, 338(6111):1225–1229, 2012.
  * [18] Beatrice de Gelder, Jan Van den Stock, Hanneke K.M. Meeren, Charlotte B.A. Sinke, Mariska E. Kret, and Marco Tamietto.  Standing up for the body. Recent progress in uncovering the networks involved in the perception of bodies and bodily expressions.  Neuroscience & Biobehavioral Reviews, 34(4):513 – 527, 2010.
  * [19] R. Walk and K. L. Walters.  Perception of the smile and other emotions of the body and face at different distances.  Bulletin of the Psychonomic Society, 26(6):510–510, 1988.
  * [20] Fritz Heider and Marianne Simmel.  An experimental study of apparent behavior.  The American Journal of Psychology, 57(2):243–259, 1944.
  * [21] Luisa Damiano and Paul Dumouchel.  Anthropomorphism in human–robot co-evolution.  Frontiers in Psychology, 9:468, 2018.
  * [22] Markus Koppensteiner.  Perceiving personality in simple motion cues.  Journal of Research in Personality, 45(4):358–363, 2011.
  * [23] R. Hortensius, F. Hekele, and E. S. Cross.  The perception of emotion in artificial agents.  IEEE Transactions on Cognitive and Developmental Systems, 10(4):852–864, 2018.
  * [24] E. Saad, J. Broekens, M. A. Neerincx, and K. V. Hindriks.  Enthusiastic robots make better contact.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1094–1100, 2019.
  * [25] C. L. Bethel and R. R. Murphy.  Survey of non-facial/non-verbal affective expressions for appearance-constrained robots.  IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(1):83–92, 2008.
  * [26] K. Lee, W. Peng, S. Jin, and C. Yan.  Can robots manifest personality? An empirical test of personality recognition, social responses, and social presence in human-robot interaction.  Journal of Communication, 56:754–772, 2006.
  * [27] L. Moshkina and R. C. Arkin.  Human perspective on affective robotic behavior: a longitudinal study.  In 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1444–1451, 2005.
  * [28] Rainer Reisenzein.  A short history of psychological perspectives on emotion.  In Rafael Calvo, Sidney D’Mello, Jonathan Gratch, and Arvid Kappas, editors, The Oxford Handbook of Affective Computing, chapter 2, pages 21–37. Oxford University Press, Inc., USA, 1st edition, 2015.
  * [29] Paul Ekman and Daniel Cordaro.  What is meant by calling emotions basic.  Emotion Review, 3(4):364–370, 2011.
  * [30] Jaak Panksepp and Douglas Watt.  What is basic about basic emotions? Lasting lessons from affective neuroscience.  Emotion Review, 3(4):387–396, 2011.
  * [31] Paul Ekman.  Universals and cultural differences in facial expressions of emotions.  In Nebraska Symposium on Motivation, volume 19, pages 207–283, 1971\.
  * [32] James A Russell and Lisa Feldman Barrett.  Core affect, prototypical emotional episodes, and other things called emotion: Dissecting the elephant.  Journal of Personality and Social Psychology, 76 5:805–19, 1999\.
  * [33] James A Russell.  Core affect and the psychological construction of emotion.  Psychological Review, 110(1):145, 2003.
  * [34] Albert Mehrabian and James A. Russell.  An approach to environmental psychology.  The MIT Press, 1974.  Pages: xii, 266.
  * [35] Trenton Schulz, Jim Torresen, and Jo Herstad.  Animation techniques in human-robot interaction user studies: A systematic literature review.  ACM Transactions on Human-Robot Interaction, 8(2):12:1–12:22, June 2019.
  * [36] Andrea Kleinsmith, Issam Rebai, Nadia Berthouze, and Jean-Claude Martin.  Postural expressions of emotion in a motion captured database and in a humanoid robot.  In Proceedings of the International Workshop on Affective-Aware Virtual Agents and Social Robots, AFFINE ’09, New York, NY, USA, 2009. Association for Computing Machinery.
  * [37] Aryel Beck, Brett Stevens, Kim A. Bard, and Lola Cañamero.  Emotional body language displayed by artificial agents.  ACM Transactions on Interactive Intelligent Systems, 2(1):Article 2, March 2012.
  * [38] Aryel Beck, Lola Cañamero, and K. A. Bard.  Towards an affect space for robots to display emotional body language.  In 19th International Symposium in Robot and Human Interactive Communication, pages 464–469, 2010.
  * [39] Daisuke Matsui, Takashi Minato, Karl F. MacDorman, and Hiroshi Ishiguro.  Generating natural motion in an android by mapping human motion.  In Hiroshi Ishiguro and Fabio Dalla Libera, editors, Geminoid Studies: Science and Technologies for Humanlike Teleoperated Androids, pages 57–73. Springer Singapore, Singapore, 2018.
  * [40] Charles Darwin.  The expression of the emotions in man and animals.  John Murray, London, England, 1872.
  * [41] Marco de Meijer.  The contribution of general features of body movement to the attribution of emotions.  Journal of Nonverbal Behavior, 13(4):247–268, December 1989.
  * [42] H.G. Wallbott.  Bodily expression of emotion.  European Journal of Social Psychology, 28:879–896, 1998.
  * [43] Kossinna Wasala, Rafael Gomez, Jared Donovan, and Marianella Chamorro-Koc.  Emotion specific body movements: Studying humans to augment robots’ bodily expressions.  In Proceedings of the 31st Australian Conference on Human-Computer-Interaction, OZCHI’19, page 503–507, New York, NY, USA, 2019\. Association for Computing Machinery.
  * [44] Hatice Gunes and Massimo Piccardi.  Bi-modal emotion recognition from expressive face and body gestures.  Journal of Network and Computer Applications, 30:1334–1345, 11 2007\.
  * [45] Andrea Kleinsmith and Nadia Bianchi-Berthouze.  Recognizing affective dimensions from body posture.  In A. Paiva, R. Prada, and R. W. Picard, editors, Affective Computing and Intelligent Interaction, pages 48–58. Springer Berlin Heidelberg, 2007.
  * [46] Markus Häring, Nikolaus Bee, and Elisabeth Andre.  Creation and evaluation of emotion expression with body movement, sound and eye color for humanoid robots.  In IEEE International Workshop on Robot and Human Interactive Communication, 07 2011.
  * [47] S. Embgen, M. Luber, C. Becker-Asano, M. Ragni, V. Evers, and K. O. Arras.  Robot-specific social cues in emotional body language.  In IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication, pages 1019–1025, September 2012.
  * [48] Mustafa Suphi Erden.  Emotional postures for the humanoid-robot Nao.  International Journal of Social Robotics, 5(4):441–456, November 2013.
  * [49] Christiana Tsiourti, Astrid Weiss, Katarzyna Wac, and Markus Vincze.  Designing emotionally expressive robots: A comparative study on the perception of communication modalities.  In Proceedings of the 5th International Conference on Human Agent Interaction, HAI ’17, page 213–222, New York, NY, USA, 2017. Association for Computing Machinery.
  * [50] Derek McColl and Goldie Nejat.  Recognizing emotional body language displayed by a human-like social robot.  International Journal of Social Robotics, 6(2):261–280, 2014.
  * [51] M. Destephe, T. Maruyama, M. Zecca, K. Hashimoto, and A. Takanishi.  Improving the human-robot interaction through emotive movements A special case: Walking.  In 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 115–116, March 2013.
  * [52] M. Destephe, A. Henning, M. Zecca, K. Hashimoto, and A. Takanishi.  Perception of emotion and emotional intensity in humanoid robots gait.  In IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 1276–1281, December 2013.
  * [53] R. Laban.  Modern educational dance.  Macdonald & Evans Ltd, 1964.
  * [54] H. Knight and R. Simmons.  Expressive motion with x, y and theta: Laban Effort Features for mobile robots.  In The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pages 267–273, August 2014.
  * [55] M. Sharma, D. Hildebrandt, G. Newman, J. E. Young, and R. Eskicioglu.  Communicating affect via flight path: Exploring use of the Laban Effort System for designing affective locomotion paths.  In 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 293–300, March 2013.
  * [56] Julian M. Angel-Fernandez and Andrea Bonarini.  Robots showing emotions.  Interaction Studies, 17(3):408–437, January 2016.
  * [57] Jekaterina Novikova and Leon Watts.  A design model of emotional body expressions in non-humanoid robots.  In Proceedings of the Second International Conference on Human-agent Interaction, HAI ’14, pages 353–360, New York, NY, USA, 2014\. ACM.
  * [58] K. Takahashi, M. Hosokawa, and M. Hashimoto.  Remarks on designing of emotional movement for simple communication robot.  In 2010 IEEE International Conference on Industrial Technology, pages 585–590, 2010.
  * [59] Megumi Masuda, Shohei Kato, and Hidenori Itoh.  Laban-based motion rendering for emotional expression of human form robots.  In Byeong-Ho Kang and Debbie Richards, editors, Knowledge Management and Acquisition for Smart Systems and Services, pages 49–60, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg.
  * [60] Megumi Masuda, Shohei Kato, and Hidenori Itoh.  A Laban-based approach to emotional motion rendering for human-robot interaction.  In Hyun Seung Yang, Rainer Malaka, Junichi Hoshino, and Jung-Hyun Han, editors, Entertainment Computing - ICEC 2010, 9th International Conference, ICEC 2010, Seoul, Korea, September 8-11, 2010. Proceedings, volume 6243 of Lecture Notes in Computer Science, pages 372–380. Springer, 2010.
  * [61] Megumi Masuda and Shohei Kato.  Motion rendering system for emotion expression of human form robots based on laban movement analysis.  In Carlo Alberto Avizzano and Emanuele Ruffaldi, editors, 19th IEEE International Conference on Robot and Human Interactive Communication, Viareggio, Italy, RO-MAN, 2010, September 13-15, 2010, pages 324–329. IEEE, 2010.
  * [62] Tatsuya Nomura and Akira Nakao.  Comparison on identification of affective body motions by robots between elder people and university students: A case study in Japan.  International Journal of Social Robotics, 2(2):147–157, 2010.
  * [63] Toru Nakata, Taketoshi Mori, and Tomomasa Sato.  Quantitative analysis of impression of robot bodily expression based on Laban movement theory.  Journal of the Robotics Society of Japan, 19(2):252–259, 2001.  In Japanese.
  * [64] Angelica Lim and Hiroshi G. Okuno.  A recipe for empathy.  International Journal of Social Robotics, 7(1):35–49, November 2014\.
  * [65] Ádám Miklósi and Márta Gácsi.  On the utilization of social animals as a model for social robotics.  Frontiers in psychology, 3:75, 2012.
  * [66] G. Lakatos, M. Gácsi, Veronika Konok, Ildikó Brúder, B. Bereczky, P. Korondi, and Á. Miklósi.  Emotion attribution to a non-humanoid robot in different social situations.  PLoS ONE, 9, 2014.
  * [67] T. Shibata, T. Mitsui, K. Wada, A. Touda, T. Kumasaka, K. Tagami, and K. Tanie.  Mental commit robot and its application to therapy of children.  In 2001 IEEE/ASME International Conference on Advanced Intelligent Mechatronics., volume 2, pages 1053–1058 vol.2, 2001.
  * [68] K. Wada, T. Shibata, T. Saito, and K. Tanie.  Effects of robot-assisted activity for elderly people and nurses at a day service center.  Proceedings of the IEEE, 92(11):1780–1788, 2004.
  * [69] Toshiyo Tamura, Satomi Yonemitsu, Akiko Itoh, Daisuke Oikawa, Akiko Kawakami, Yuji Higashi, Toshiro Fujimooto, and Kazuki Nakajima.  Is an entertainment robot useful in the care of elderly people with severe dementia?  The Journals of Gerontology Series A: Biological Sciences and Medical Sciences, 59(1):M83–M85, 2004.
  * [70] Alexander V Libin and Elena V Libin.  Person-robot interactions from the robopsychologists’ point of view: The robotic psychology and robotherapy approach.  Proceedings of the IEEE, 92(11):1789–1803, 2004.
  * [71] Ollie Johnston Frank Thomas.  The illusion of life: Disney animation.  Disney Editions, 1995.
  * [72] Jamy Li and Mark Chignell.  Communication of emotion in social robots through simple head and arm movements.  International Journal of Social Robotics, 3(2):125–142, April 2011\.
  * [73] Zane Thimmesch-Gill, Kathleen A. Harder, and Wilma Koutstaal.  Perceiving emotions in robot body language: Acute stress heightens sensitivity to negativity while attenuating sensitivity to arousal.  Computers in Human Behavior, 76:59 – 67, 2017.
  * [74] Junchao Xu, Joost Broekens, Koen Hindriks, and Mark A. Neerincx.  Mood contagion of robot body language in human robot interaction.  Autonomous Agents and Multi-Agent Systems, 29(6):1216–1248, November 2015.
  * [75] K. Itoh, H. Miwa, M. Matsumoto, M. Zecca, H. Takanobu, S. Roccella, M. C. Carrozza, P. Dario, and A. Takanishi.  Various emotional expressions with emotion expression humanoid robot WE-4RII.  In IEEE Conference on Robotics and Automation, 2004. TExCRA Technical Exhibition Based., pages 35–36, 2004.
  * [76] Jérôme Monceaux, Joffrey Becker, Céline Boudier, and Alexandre Mazel.  Demonstration: First steps in emotional expression of the humanoid robot Nao.  In Proceedings of the 2009 International Conference on Multimodal Interfaces, ICMI-MLMI ’09, pages 235–236, New York, NY, USA, 2009. ACM.
  * [77] T. Ribeiro and A. Paiva.  The illusion of robotic life: principles and practices of animation for robots.  In 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 383–390, March 2012.
  * [78] S. Yohanan and K. E. MacLean.  Design and assessment of the Haptic Creature’s affect display.  In 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 473–480, 2011.
  * [79] Michael Suguitan, Mason Bretan, and Guy Hoffman.  Affective robot movement generation using CycleGANs.  In 14th ACM/IEEE International Conference on Human-Robot Interaction, HRI 2019, Daegu, South Korea, March 11-14, 2019, pages 534–535, 2019.
  * [80] M. Marmpena, A. Lim, T. S. Dahl, and N. Hemion.  Generating robotic emotional body language with variational autoencoders.  In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 545–551, 2019.
  * [81] Michael Suguitan, Randy Gomez, and Guy Hoffman.  Moveae: Modifying affective robot movements using classifying variational autoencoders.  In Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’20, page 481–489, New York, NY, USA, 2020. Association for Computing Machinery.
  * [82] Diederik P Kingma and Max Welling.  Auto-encoding variational Bayes.  arXiv e-prints, page arXiv:1312.6114, December 2013.
  * [83] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models.  arXiv e-prints, page arXiv:1401.4082, January 2014.
  * [84] Diederik P. Kingma and Max Welling.  An Introduction to Variational Autoencoders.  arXiv e-prints, page arXiv:1906.02691, June 2019.
  * [85] Kihyuk Sohn, Honglak Lee, and Xinchen Yan.  Learning structured output representation using deep conditional generative models.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 3483–3491. Curran Associates, Inc., 2015.
  * [86] Mina Marmpena, Angelica Lim, and Torbjørn S. Dahl.  How does the robot feel? Perception of valence and arousal in emotional body language.  Paladyn, 9(1):168–182, 2018.
  * [87] Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov.  Improving neural networks by preventing co-adaptation of feature detectors.  arXiv e-prints, page arXiv:1207.0580, 2012.
  * [88] Irina Higgins, Loïc Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  $\beta$-vae: Learning basic visual concepts with a constrained variational framework.  In 5th International Conference on Learning Representations, 2017\.
  * [89] Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner.  Understanding disentangling in $\beta$-VAE.  arXiv e-prints, page arXiv:1804.03599, April 2018.
  * [90] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249–256, 2010.
  * [91] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv e-prints, page arXiv:1412.6980, December 2014.
  * [92] Christoph Bartneck, Elizabeth Croft, and Dana Kulic.  Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots.  International Journal of Social Robotics, 1(1):71–81, 2009.
  * [93] Alberto Betella and Paul F. M. J. Verschure.  The affective slider: A digital self-assessment scale for the measurement of human emotions.  PLoS ONE, 11(2):e0148037, Feb 2016.
  * [94] Paulo Gomes, Ana Paiva, Carlos Martinho, and Arnav Jhala.  Metrics for character believability in interactive narrative.  In Proceedings of the 6th International Conference on Interactive Storytelling - Volume 8230, ICIDS 2013, page 223–228, Berlin, Heidelberg, 2013. Springer-Verlag.
  * [95] Torrin M. Liddell and John K. Kruschke.  Analyzing ordinal data with metric models: What could possibly go wrong?  Journal of Experimental Social Psychology, 79:328 – 348, 2018.
  * [96] Travis E Oliphant.  A guide to NumPy, volume 1.  Trelgol Publishing USA, 2006.
  * [97] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake Vand erPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1. 0 Contributors.  Scipy 1.0: Fundamental algorithms for scientific computing in Python.  Nature Methods, 17:261–272, 2020.
  * [98] Wes McKinney et al.  Data structures for statistical computing in Python.  In Proceedings of the 9th Python in Science Conference, volume 445, pages 51–56. Austin, TX, 2010.
  * [99] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12(Oct):2825–2830, 2011.
  * [100] John D Hunter.  Matplotlib: A 2D graphics environment.  Computing in Science & Engineering, 9(3):90–95, 2007.
  * [101] R Core Team.  R: A language and environment for statistical computing.  R Foundation for Statistical Computing, Vienna, Austria, 2019.
  * [102] Hadley Wickham.  ggplot2: Elegant graphics for data analysis.  Springer-Verlag New York, 2016.  https://ggplot2.tidyverse.org.
  * [103] Alboukadel Kassambara.  rstatix: Pipe-friendly framework for basic statistical tests, 2019\.  R package version 0.3.0. https://CRAN.R-project.org/package=rstatix.
  * [104] W. N. Venables and B. D. Ripley.  Modern applied statistics with S.  Springer, New York, fourth edition, 2002.
  * [105] R. H. B. Christensen.  ordinal: Regression models for ordinal data, 2019.  R package version 2019.12-10. https://CRAN.R-project.org/package=ordinal.
  * [106] Russell Lenth.  emmeans: Estimated marginal means, aka Least-Squares Means, 2019\.  R package version 1.4.3.01. https://CRAN.R-project.org/package=emmeans.
  * [107] Hadley Wickham, Romain François, Lionel Henry, and Kirill Müller.  dplyr: A grammar of data manipulation, 2019.  R package version 0.8.3. https://CRAN.R-project.org/package=dplyr.
  * [108] Jason Bryer and Kimberly Speerschneider.  likert: Analysis and visualization Likert items, 2016.  R package version 1.3.5. https://CRAN.R-project.org/package=likert.
  * [109] Mina Marmpena, Fernando Garcia, and Angelica Lim.  Generating robotic emotional body language of targeted valence and arousal with conditional variational autoencoders.  In Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’20, page 357–359, New York, NY, USA, 2020. Association for Computing Machinery.
  * [110] Aryel Beck, Antoine Hiolle, Alexandre Mazel, and Lola Cañamero.  Interpretation of emotional body language displayed by robots.  In Proceedings of the 3rd International Workshop on Affective Interaction in Natural Environments, pages 37–42. ACM, 2010.
  * [111] K. Nakagawa, K. Shinozawa, H. Ishiguro, T. Akimoto, and N. Hagita.  Motion modification method to control affective nuances for robots.  In 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5003–5008, 2009.