# Weakly-Supervised Emotion Transition Learning for
Diverse 3D Co-speech Gesture Generation

Xingqun Qi1, Jiahao Pan1, Peng Li1, Ruibin Yuan1, Xiaowei Chi1, Mengfei Li1
Wenhan Luo2, Wei Xue1, Shanghang Zhang3, Qifeng Liu${}^{1,\textrm{{\char
0\relax}}}$, Yike Guo${}^{1,\textrm{{\char 0\relax}}}$
1 The Hong Kong University of Science and Technology
2 Sun Yat-sen University 3 Peking University
<EMAIL_ADDRESS>{liuqifeng<EMAIL_ADDRESS>

###### Abstract

Generating vivid and emotional 3D co-speech gestures is crucial for virtual
avatar animation in human-machine interaction applications. While the existing
methods enable generating the gestures to follow a single emotion label, they
overlook that long gesture sequence modeling with emotion transition is more
practical in real scenes. In addition, the lack of large-scale available
datasets with emotional transition speech and corresponding 3D human gestures
also limits the addressing of this task. To fulfill this goal, we first
incorporate the ChatGPT-4 and an audio inpainting approach to construct the
high-fidelity emotion transition human speeches. Considering obtaining the
realistic 3D pose annotations corresponding to the dynamically inpainted
emotion transition audio is extremely difficult, we propose a novel weakly
supervised training strategy to encourage authority gesture transitions.
Specifically, to enhance the coordination of transition gestures w.r.t.
different emotional ones, we model the temporal association representation
between two different emotional gesture sequences as style guidance and infuse
it into the transition generation. We further devise an emotion mixture
mechanism that provides weak supervision based on a learnable mixed emotion
label for transition gestures. Last, we present a keyframe sampler to supply
effective initial posture cues in long sequences, enabling us to generate
diverse gestures. Extensive experiments demonstrate that our method
outperforms the state-of-the-art models constructed by adapting single
emotion-conditioned counterparts on our newly defined emotion transition task
and datasets.

Figure 1: Diverse exemplary clips sampled by our method from our newly
collected BEAT Emotion Transition Dataset. The vital frames are visualized to
demonstrate that the upper body gestures change with the emotion transition of
human speech, synchronously. From top to bottom: the input speech audio, the
corresponding transcript, and two sampled clips. Best view on screen.

††footnotetext: ✉ Corresponding authors.

##  1 Introduction

Co-speech gesture generation aims to synthesize vivid and emotional human
postures coordinated with the audio input. These non-verbal behaviors serve as
a key factor during human conversations that significantly facilitates the
delivery of speech content [3, 12, 14]. Meanwhile, modeling co-speech gestures
has a wide range of embodied AI applications in human-machine interaction [20,
25], robot assistants [7], and virtual/augmented reality (AR/VR) [9].
Conventionally, many researchers usually focus on synthesizing human upper
body gestures consistent with speech audio [29, 49, 46, 44].

Nevertheless, except for a few recent works that generate the co-speech
gestures of a single emotion category [26, 48, 2], previous works mostly focus
on emotion-agnostic generation [46, 44]. Most of them overlook synthesizing
the long sequence co-speech gestures with the emotion transitions, which are
more practical in real-world scenes. For example, a person may not maintain a
single emotion forever when communicating with others or in speech talking. In
this work, we therefore introduce the task of _speech-driven emotion
transition_ for generating vivid and diverse 3D co-speech gestures, displayed
in Figure 1. There are two main challenges in this task: 1) Datasets of 3D co-
speech gestures synchronized with emotion transition speech audios are scarce.
Creating such a dataset containing 3D human postures is difficult due to the
lack of guidance from emotional experts and complex motion capture systems. 2)
Modeling the plausible and temporal coherent co-speech gestures from one
emotion to another in long sequences is difficult, especially in the
transition duration.

To overcome the issue of data scarcity, we newly present two datasets
containing emotion-transition human speech that are built upon previous
single-emotion ones [29, 26]. In particular, thanks to the developed language
model ChatGPT-4 [31], we first leverage it to generate text transcripts of the
transition based on speech context. Then, by employing the audioLDM2 technique
[27, 28] for audio inpainting, we ensure the inpainted transition’s timbral
consistency with its adjacent contexts and a smooth emotional transition
throughout. Afterwards, to support our insight on modeling co-speech gestures
coherent with emotion-transition speech, corresponding 3D human postures of
transition are required.

However, due to the dynamically generated transition transcripts, it is
infeasible to construct the aligned realistic 3D pose annotation of human
bodies. Hence, we solve the challenges of vivid co-speech gesture generation
in a novel weakly supervised pattern, containing a motion transition infusion
mechanism and an emotion mixture strategy. Specifically, in the motion
transition infusion mechanism, we model the temporal correlation between the
generated head and tail gesture features as style guidance representation. The
style guidance representation provides motion transition cues that are infused
into the transition embedding via an adaptive instance normalization (AdaIN)
layer [16]. In this manner, we can effectively enhance the coordination of
transition gestures w.r.t. two different emotional ones.

Moreover, to alleviate the lack of supervision during the transition between
two emotions, the emotion mixture strategy is built to provide weak emotional
supervision of the generated transition gestures. Concretely, we learn a joint
embedding of two different emotional gesture sequences using a temporal
aggregation encoder. Then, we pre-train an emotion classifier based on the
annotated human 3D poses with single emotion labels in the dataset. Here, this
joint embedding is leveraged as an emotion mixture weight for the pre-trained
classifier to facilitate high-fidelity transition gesture synthesis with
desirable properties. Finally, considering the generated 3D postures should be
non-deterministic given the human speech, we devise a keyframe sampler to
produce diverse initial poses as reference. In this fashion, our method
enables diverse co-speech gesture generation with emotion transitions.
Extensive experiments conducted on our newly constructed two datasets verify
the effectiveness of our methods, displaying vivid and emotional 3D co-speech
gestures.

Overall, our contributions are summarized as follows:

  * •

We introduce a new task of emotion transition co-speech gesture generation
cooperating with two newly constructed datasets named BEAT Emotion Transition
(BEAT-ETrans) and TED Emotion Transition (TED-ETrans), significantly
facilitating research on 3D human motion modeling.

  * •

We design a motion transition infusion mechanism to ensure the temporal
coordination of transition gestures w.r.t. two different emotional ones and a
weakly supervised emotion mixture strategy to enable high-fidelity transition
gesture synthesis with desirable properties.

  * •

Extensive experiments show that our method outperforms state-of-the-art
counterparts on both datasets, displaying realistic and vivid co-speech
gestures given emotion transition human audios.

##  2 Related Work

###  2.1 Co-speech Gesture Synthesis

Synthesizing human co-speech gestures plays a significant role in various
applications [20, 15, 43, 34]. Numerous studies have been proposed to address
these issues that are roughly divided into rule-based approaches [18, 19],
machine-learning-based approaches [21, 38], and deep learning-based ones [29,
49, 46, 44, 26, 33, 48, 2]. Traditional researchers follow the rule-based
workflow, leveraging the speech-gesture pairs as guidance to generate co-
speech gestures pre-defined by linguistic experts. Other early works integrate
the manually defined gesture features with machine learning techniques to
synthesize the co-speech gestures. In the aforementioned two manners, the
researchers usually focus on the optimization of the matching process between
human speech and pre-defined gestures. It may require experts much expensive
effort in the speech-gesture pair construction.

Recently works focus on building the mapping directly from the input human
speech and sequential gestures by exploiting the deep neural networks. They
usually leverage multi-modality cues to facilitate the generation of co-speech
gestures, associating with the speech audio [23, 49, 44], speaker identity
[46], emotion [26, 2], and text transcript [29]. However, they overlook that
emotion transition of the long sequential co-speech gesture modeling is much
more practical in the real scenes. Moreover, since the lack of annotated 3D
gestures corresponding to dynamically constructed transition speech, few of
the above methods could be directly adapted to this new thought.

###  2.2 3D Human Motion Modeling

Human motion modeling aims to generate realistic and smooth human motions with
various multi-modality conditions [17, 30, 6], including co-speech gesture
generation as a sub-task. One of the hottest topics is synthesizing human
motion from text prompts with a few past postures as the seed [32, 47, 42].
These methods usually engage in forcing human motion to represent the semantic
expression aligned with the text. Literally, the task most closely related to
ours is AI choreographer [22, 24, 39] which generates the motion from music
signals. However, the AI choreographer works mainly focus on the rhythmic-
coherent motion of the whole human body but without subtle finger gestures.
While sharing a similar goal with the approaches mentioned above, our newly
defined work differs from them significantly. We take the emotion-transition
long sequence co-speech gesture without corresponding 3D human pose annotation
into consideration, thus motivating us to utilize the motion transition of two
annotated emotional gestures and coherent the overall sequence.

Figure 2: The overview of our proposed method. The middle part (blue) displays
the overall pipeline for 3D co-speech gesture generation from emotion
transition human speech. The left part (green) depicts our proposed Motion
Transition Infusion Mechanism that enhances the coordination of transition
gestures w.r.t. head/tail ones. The right part (orange) shows the designed
Emotion Mixture Strategy to provide weak supervision of the generated
transition gestures, thereby achieving authority producing.

##  3 Proposed Method

###  3.1 Emotion Transition Dataset Construction

We aim to address the emotion transition co-speech gesture generation in a
weakly supervised manner. Due to the existing paired speech-gesture datasets
[26, 33], we could focus on synthesizing the high-fidelity transition human
audios. Synthesizing datasets conducive to our task focuses on ensuring
semantic coherence, smooth emotional audio transitions, consistent timbre, and
audio fidelity.

Preliminary: Considering our key insight to modeling the co-speech gesture
with emotion transition, we first split the existing aligned speech-gesture
pairs [26, 33] into four-second clips. Then we randomly splice two clips from
the same speaker to construct an emotion-transition candidate pair. The clip
with neutral emotion is leveraged as head speech, and the other with various
emotions is represented as tail speech. We leverage the dynamically
synthesized two-second audio as a transition to combine the head and tail
speeches. In the following, we briefly summarize our efforts in constructing
the dynamic transition speech audio.

Textual Inpainting of Transition: To ensure the semantic coherence of
transition w.r.t. head/tail speeches, we exploit the advanced language
generation model ChatGPT-4 [31] to complete the transcript according to the
context. Literally, we follow the conventional estimation that people usually
talk 30 phonemes [37] in a two-second speech as the prompt to guide transcript
generation by ChatGPT-4.

Synthesis of Transition Audio: Once we obtain the transcript of transitions,
we employ a superior text-to-speech model, audioLDM2 [27, 28], to generate
corresponding speech audio. Here, we leverage the speaker embeddings extracted
via SpeechBrain’s ECAPA-TDNN [36, 5] as prior guidance to maintain the
identity consistency of the generated transition audio. Then, we adopt Whisper
[35] to restrict the word error rate, thereby ensuring the accuracy and
clarity of the speech content. In this fashion, the synthesized transition
audio realizes controlled duration while the natural smooth tonality is well
preserved. The datasets will be released in the future. More details are
provided in the Section 4.1.

###  3.2 Problem Formulation

Given a sequence of audio signal $A=\left\\{a_{1},...,a_{N}\right\\}$ as input
to model $\mathcal{G}$, our goal is to generate vivid and emotional 3D co-
speech gestures $P=\left\\{p_{1},...,p_{N}\right\\}$ of the upper human body.
$N$ represents the number of synthesized postures corresponding to the audio
$A$. Each $p_{i}$ is denoted as $J$ joints with 3D representation. In
particular, we define the audio signal as consisting of a head speech, a
dynamically inpainted transition speech, and a tail speech. Here, the head and
tail speeches are randomly selected from previous co-speech gesture datasets
[29, 26] with a single emotion label, respectively. Following the conventions
of previous works [46, 29, 49, 26], we invoke $M$ frame poses as the initial
seed to guide generation. The overall objective is expressed as

$\underset{\mathcal{G}}{\operatorname*{arg\,min}}\left\|P-\mathcal{G}\left(A,\left\\{p_{1},...,p_{M}\right\\}\right)\right\|.$
(1)

Note that only the generated gestures with head and tail speeches are
supervised with ground truth coming from existing datasets [29, 26]. The
transition gestures with $L$ frames, where $L\ll N$, will be weakly supervised
through the following processes within a motion transition infusion mechanism
and an emotion mixture strategy. The audio signal is fed into an audio encoder
for feature extraction. Our overall workflow is shown in Figure 2.

###  3.3 Weakly-supervised Emotion Transition

Motion Transition Infusion Mechanism: To ensure the temporal coherence of the
transition gestures w.r.t. head/tail ones, we propose a motion transition
infusion mechanism to model the temporal association representation between
different emotional gestures. As depicted in Figure 2 (left), the total
sequential features $\mathbf{f}^{total}$ consist of the features of head
$\mathbf{f}^{head}$, transition $\mathbf{f}^{tran}$, and tail
$\mathbf{f}^{tail}$. Inspired by [30], we nominate a head chunk
$\mathbf{f}_{chunk}^{head}$ composed of the last $L$ frames of head gesture
embeddings and a tail chunk $\mathbf{f}_{chunk}^{tail}$ consisting of the
first $L$ ones of the tail gesture embeddings. Here, the dimension of each
frame representing gesture embedding is $\mathbb{R}^{1\times D}$.

In particular, we first calculate the temporal correlation matrix
$\mathbf{C}_{trans}^{head}\in\mathbb{R}^{L\times L}$ between the head chunk
embedding and transition embedding. Here, the temporal correlation matrix
represents the temporal variations in the gestures from the head to the
transition. Similarly, we obtain the correlation matrix
$\mathbf{C}_{trans}^{tail}$ from tail chunk embedding and transition
embedding. Then, the global temporal dependency from head to tail is computed
via matrix multiplication between $\mathbf{C}_{trans}^{head}$ and
$\mathbf{C}_{trans}^{tail}$. Further, we develop a motion encoder to acquire
the sequence-aware style guidance representation based on the global temporal
dependency. The style guidance representation is exploited to boost the
transition embeddings via an adaptive instance normalization (AdaIN) layer
[16]. By doing so, we derive the transition $\mathbf{f}^{tran}$ as

$\mathbf{f}^{tran}\\!=\\!\text{AdaIN}\left\\{\mathbf{f}^{tran},\text{Enc}(\mathbf{C}_{trans}^{head}\otimes\mathbf{C}_{trans}^{tail})\right\\},$
(2)

where $\otimes$ indicates matrix multiplication, and Enc denotes the motion
encoder.

Emotion Mixture Strategy: Considering obtaining the realistic 3D pose
annotations corresponding to dynamically inpainted transition audio is quite
difficult, we design an emotion mixture strategy to provide weak supervision
of the generated gestures. Our key insight is built upon the fact that
different emotions in the head/tail would lead to different gesture motions,
thereby the emotion represented by transition gestures would be a mixture of
the head and tail ones. As shown in Figure 2 (right), we utilize two learnable
parameters as soft emotion labels of the transition gestures.

Specifically, through computing the correlation matrix between the embeddings
of head gestures and tail gestures, we obtain the motion deformation
representation $S$ from one emotion to another (_e.g_., neutral-to-anger).
This motion deformation is formulated as:

$\mathbf{S}_{head}^{tail}\\!=\\!\text{Softmax}(\mathbf{f}^{head}\otimes{\mathbf{f}^{tail}}^{{}^{\prime}}),$
(3)

where ′ indicates the transpose operation. With the help of the motion
deformation representation, we obtain the two learnable emotion embeddings
$(\mathbf{e}^{head},\mathbf{e}^{tail})$ from the head and tail, respectively,
as

$\mathbf{e}^{head}=\text{MaxPool}\left\\{\mathbf{W}_{linear}(\mathbf{f}^{head})\odot\mathbf{S}_{head}^{tail}\right\\},$
(4)

where $\mathbf{W}_{linear}$ denotes linear matrix, and $\odot$ means dot
product. Maxpool indicates the AdaptiveMaxPooling operation.
$\mathbf{e}^{tail}$ is calculated in a similar way. Then, we obtain the two
learnable parameters represented by emotion weights of head and tail gestures,
as follows

$(\sigma,1-\sigma)=\text{Softmax}(\mathbf{e}^{head},\mathbf{e}^{tail}).$ (5)

Once we acquire the learnable emotion weights, we leverage a pre-trained pose-
based emotion classifier to provide weak supervision. In this fashion, the
authority of generated transition gestures is well-preserved. The training
details of the pre-trained pose-based emotion classifier are provided in the
supplementary material.

Keyframe Posture Sampling: Conventionally, researchers [29, 49, 46] directly
leverage the padded initial poses as conditional seeds to guide the co-speech
gesture generation. In long-sequence modeling, an intuitive manner is to
extend the initial pose length $M$ for adapting the overall synthesized
gestures’ length. However, this would lead to the poor generalizability of the
network (_i.e_., the performance gains degradation from the reduction of
initial poses). Therefore, we propose a simple yet effective VAE-based [40]
keyframe sampler to provide high-fidelity posture prior conditions while
enabling diverse results. Evenly, we split the annotated sequence into several
chunks keeping the same length with the transition of length $L$. The sampler
is trained with the keyframe randomly selected to reconstruct the
corresponding chunk. In the inference phase, the keyframe sampler samples
diverse chunks, blending as the initial postures for gesture generation. Since
the transition sequence lacks the pose annotation, we randomly select the
frame from head or tail sequences for posture sampling.

To further enhance the sequence-aware correspondence of the generated co-
speech gestures, we leverage the diverse initial postures as the query $Q$ to
match the key features $K$ and value features $V$ in the transformer-based
backbone [41]. Similar to [34], we adopt a motion discriminator to ensure the
temporal smoothness of the generated results. For more details about network
architecture please refer to supplementary material.

###  3.4 Objective Functions

Reconstruction Loss: We leverage the ground truth 3D pose annotation of the
head and tail to constrain the generated co-speech gestures as:

$\displaystyle\mathcal{L}_{rec}\\!$
$\displaystyle=\\!\left\|P_{\left\\{head,tail\right\\}}\\!-\\!\widehat{P_{\left\\{head,tail\right\\}}}\right\|_{1},$
(6)

where $\widehat{P_{\left\\{head,tail\right\\}}}$ denotes generated gestures of
head and tail speeches.

Adversarial Learning Loss: To ensure the realism of the generated gestures, we
further exploit the adversarial training loss, expressed as:

$\displaystyle\mathcal{L}_{adv}=\mathbb{E}_{P}\left[\log\mathcal{D}(P)\right]+\mathbb{E}_{A}\left[\log(1-(\mathcal{G}\left(A,\left\\{p_{1},...,p_{M}\right\\}\right))\right],$
(7)

where $\mathcal{D}$ denotes the motion discriminator and $\mathcal{G}$ means
gesture generator.

Weakly Supervision Loss: We leverage the pre-trained pose-based emotion
classifier to provide weak supervision of the transition gestures upon the
learnable emotion weights:

$\displaystyle\mathcal{L}_{emotion}=-y\log\mathcal{F}(P_{trans}),$ (8)

where $y$ is the learnable emotion label, $\mathcal{F}$ is the emotion
classifier, $\widehat{P_{trans}}$ is the generated transition gestures.

Finally, the overall objective is:

$\displaystyle\\!\\!\min_{G}\max_{D}\mathcal{L}_{total}\\!=\\!\lambda_{r}\mathcal{L}_{rec}+\lambda_{adv}\mathcal{L}_{adv}+\mathcal{L}_{emotion}.$
(9)

The $\lambda_{r}$, and $\lambda_{adv}$ are weight coefficients.

##  4 Experiments

Table 1: Statistics comparison of existing 3D co-speech gesture datasets with
ours. Our BEAT-ETrans and TED-ETrans are built upon the existing BEAT [26] and
TED-Expressive [29], respectively. To the best of our knowledge, we are the
first to present two large datasets with emotion transition human speech.

|  | Modality |
---|---|---|---
Dataset |  Joint Annotation | Body | Hand | Audio | Text | Speakers | Single Emotion | Emotion Transition | Duration (hours)
TED [46] | pseudo label | 9 | ✗ | ✓ | ✓ | 1,766 | ✗ | ✗ | 106.1
SCG [13] | pseudo label | 14 | 24 | ✓ | ✗ | 6 | ✗ | ✗ | 33
Trinity [8] | mo-cap | 24 | 38 | ✓ | ✓ | 1 | ✗ | ✗ | 4
ZeroEGGS [10] | mo-cap | 27 | 48 | ✓ | ✓ | 1 | ✗ | ✗ | 2
BEAT [26] | mo-cap | 27 | 48 | ✓ | ✓ | 30 | ✓ | ✗ | 35
TED-Expressive [29] | pseudo label | 13 | 30 | ✓ | ✓ | 1,764 | ✗ | ✗ | 100.8
BEAT-ETrans (ours) | mo-cap | 27 | 48 | ✓ | ✓ | 30 | 8 | ✓ | 161.3
TED-ETrans (ours) | pseudo label | 13 | 30 | ✓ | ✓ | 1,764 | 6 | ✓ | 59.8

Table 2: Comparison with the start-of-the-art methods on our newly collected BEAT-ETrans and TED-ETrans datasets. $\uparrow$ denotes the higher the better, and $\downarrow$ indicates the lower the better. $\pm$ means 95% confidence interval.  Models | BEAT-ETrans | TED-ETrans
---|---|---
FGDh+t $\downarrow$ |  FGDtrans $\downarrow$ |  BC $\uparrow$ |  Diversity $\uparrow$ |  FGDh+t $\downarrow$ |  FGDtrans $\downarrow$ |  BC $\uparrow$ |  Diversity $\uparrow$
Seq2Seq [45]${}_{ICRA^{\prime}19}$ | 40.95 | 47.93 | 0.141 |  96.66±2.16 | 29.60 | 49.47 | 0.265 |  72.81±1.99
S2G [11]${}_{CVPR^{\prime}19}$ | 25.56 | 37.04 | 0.671 |  98.26±2.04 | 18.16 | 41.63 | 0.824 |  76.82±2.32
Trimodal [46]${}_{TOG^{\prime}20}$ | 14.09 | 42.50 | 0.764 |  100.87±2.12 | 21.06 | 33.20 | 0.758 |  82.87±1.86
CAMN [26]${}_{ECCV^{\prime}22}$ | 9.03 | 27.53 | 0.794 |  118.46±2.33 | 19.28 | 41.04 | 0.785 |  79.03±1.49
HA2G [29]${}_{CVPR^{\prime}22}$ | 7.28 | 25.79 | 0.779 |  121.77±2.31 | 16.72 | 40.38 | 0.787 |  80.14±1.65
DiffGesture [49]${}_{CVPR^{\prime}23}$ | 6.68 | 25.03 | 0.788 |  122.29±2.01 | 18.69 | 25.13 | 0.818 |  92.01±2.07
Ours | 4.42 | 18.84 | 0.881 | 124.93±2.10 | 12.19 | 23.54 | 0.906 | 93.79±2.53

###  4.1 Datasets and Experimental Setting

BEAT Emotion Transition Dataset (BEAT-ETrans): Since there are only _single
emotion_ labels of aligned speech-gesture corpus in the original BEAT dataset
[26], to satisfy our insight on emotion transition co-speech gesture
generation modeling, we recollect a BEAT Emotion Transition Dataset (dubbed
BEAT-ETrans). In particular, we resample the motion FPS as $15$ and intercept
the continuous $60$ frames with stride $30$ as the head/tail clips. Here, the
head clips are all annotated as _neutral_ , and tail clips are denoted with
the other seven emotions: _anger, happiness, fear, disgust, sadness, contempt,
and surprise_. As for one head speech, we randomly select two or three tails
with different emotions to construct the head-tail pairs. By leveraging the
two-second (_i.e_., corresponding $30$-frame postures) transition to blend the
heads and tails, we obtain the $10$-second human speech clips in our BEAT-
ETrans. We obtain $58,077$ clips, including a total of $161.3$ hours reported
in Table 1. Then the clip numbers of training/validation/testing sets are
randomly split as $41,908$/$4,077$/$12,092$. In all of our experiments, we
utilize the upper body with $71$ joints.

TED Emotion Transition Dataset (TED-ETrans): Inspired by [29, 33], we further
newly collect a TED Emotion Transition Dataset (dubbed TED-ETrans) based on
more than 1.7K speakers from in-the-wild TED talk show videos, demonstrated in
Table 1. Due to the lack of emotional labels, we first leverage the annotated
BEAT dataset to pre-train an audio-based emotion classifier for labeling TED
audios. To ensure the authority of emotion labels, we set the classification
threshold as $\geq 0.95$, and the two uncommon emotions (_i.e_. _fear,
disgust_) are dropped. Then we maintain the same data pre-processing strategy
with our BEAT-ETrans to obtain a total of $21,515$ clips with $59.8$ hours.
The final clip division criteria of the TED-ETrans dataset are
training/validation/testing with 15,061/2,152/4,302 respectively. In practice,
the $43$ upper body joints are leveraged in the experiments.

Implementation Details: We set the total generated co-speech gesture length as
$N=150$, and the transition and chunk lengths are $L=30$. Conventionally, we
leverage $M=4$ frames as the reference initial poses. The feature dimension
$D=512$ in practice. The raw audio of human speech is converted to mel-
spectrograms with FFT window size $1024$, and hop length $512$. The audio
encoder takes the ResNetSE34 [4] as the backbone. Empirically, we set
$\lambda_{r}=20$, and $\lambda_{adv}=2$. Our models are implemented on the
Pytorch platform with a single NVIDIA Tesla V100 GPU. The initial learning
rate is set to $0.0003$ by utilizing Adam Optimizer. The whole training takes
$100$ epochs with a batch size of $96$.

Table 3: Ablation study on different components of the proposed method. ✓ indicates the employment of a certain module. $\uparrow$ denotes the higher the better, and $\downarrow$ indicates the lower the better. $\pm$ means 95% confidence interval. MTIM: Motion Transition Infusion Mechanism; EMS: Emotion Mixture Strategy; FKS: Keyframe Sampler. Model Variations | BEAT-ETrans | TED-ETrans
---|---|---
Baseline | MTIM | EMS | KFS |  FGDh+t $\downarrow$ |  FGDtrans $\downarrow$ |  BC $\uparrow$ |  Diversity $\uparrow$ |  FGDh+t $\downarrow$ |  FGDtrans $\downarrow$ |  BC $\uparrow$ |  Diversity $\uparrow$
✓ |  |  |  | 21.21 | 56.33 | 0.701 |  88.89±1.58 | 26.44 | 47.87 | 0.712 |  74.19±2.75
✓ | ✓ |  |  | 10.75 | 27.94 | 0.866 |  108.69±2.63 | 20.95 | 36.38 | 0.827 |  82.42±2.24
✓ | ✓ | ✓ |  | 5.69 | 21.70 | 0.878 |  112.80±1.95 | 14.02 | 26.21 | 0.900 |  87.21±2.12
✓ | ✓ | ✓ | ✓ | 4.42 | 18.84 | 0.881 | 124.93±2.10 | 12.19 | 23.54 | 0.906 | 93.79±2.53

Evaluation Metrics: To fully evaluate the realism and diversity of the
generated co-speech gestures, we introduce various metrics***More details are
in the supplementary material.:

  * •

FGD: Fréchet Gesture Distance (FGD) [46] is utilized to measure the
distribution distance between the realistic sequential gestures and generated
ones. Since we only have the 3D joint annotation of the head/tail, we take the
network architecture provided in [46, 29] to train the auto-encoder for
distance computing on the two datasets, respectively. The FGD of transition
gestures is calculated as the average value between the distribution distance
of transition and head/tail, indicated as FGDtrans. Similarly, FGDh+t means
the distance between the generated head/tail gestures and ground truth.

  * •

BC: Beat Consistency Score (BC) [26, 29] measures the speech audio alignment
degree with the generated co-speech gestures.

  * •

Diversity: Similar to [29, 49], we exploit the same feature extractor in FGD
to obtain the feature embeddings of the generated gestures. The diversity
reflects the average distance between $500$ random combination pairs in the
testing set of $60$ speech audios.

###  4.2 Quantitative Evaluation

Comparisons with SOTA Methods: To the best of our knowledge, we are the first
to explore the co-speech gesture generation with emotion transition human
audios. To fully verify the superiority of our method, we implement various
state-of-the-art single-emotion-based counterparts: Seq2Seq [45], S2G [11],
Trimodal [46], CAMN [26], HA2G [29], and DiffGesture [49]. For a fair
comparison, all the models are implemented by the source code released by the
authors. Note that since we only have the ground truth of head and tail
gestures, several recent VAE-based works [2, 48, 44, 1] cannot be directly
applied in the experiments (or they have not released codes so far).

As reported in Table 2, we adopt the FGDh+t, FGDtrans, BC, and Diversity for a
well-rounded view of comparisons. Our method outperforms all the competitors
by a large margin on both two datasets. Remarkably, on the TED-ETrans dataset,
our method even achieves 34.8% (_i.e_., $(18.69-12.19)/18.69\approx 34.8\%$)
improvement over the sub-optimal counterparts in FGDh+t. We observe both the
DiffGesture [49] and ours synthesize the high-fidelity gestures of head/tail
speech with much lower FGDh+t than others. However, the DiffGesture shows
worse performance on FGDtrans due to lack of supervision. In terms of
diversity, our simple yet effective keyframe sampler provides authority and
diverse initial postures as the reference, thus enabling us to demonstrate
diverse gesture styles compared to other counterparts. Moreover, we find that
the diversity scores on BEAT-ETrans are much higher than those on TED-ETrans
dataset. This can be attributed to the more complex human joints in the BEAT-
ETrans dataset.

Ablation Study: To further verify the effectiveness of our proposed methods,
we conduct the ablation study of different components as variations, reported
in Table 3. The baseline model is implemented by a simple transformer-based
pipeline with stacking three times blocks in the encoder-decoder. Obviously,
all the combinations of our proposed components have positive impacts on the
generated results. Specifically, by adding the motion transition infusion
mechanism to the baseline, the indicator BC has achieved significant
improvement (_e.g_., $0.701$ $\to$ $0.866$ in the BEAT-Etrans). This result
verifies that our motion transition infusion mechanism effectively models the
temporal correlation between the transitions w.r.t. head/tail gestures, thus
leading to the generated results preserving rhythm coherency with given
speech, globally.

Moreover, adopting the emotion mixture strategy ideally improves the
performance of FGDtrans on both two datasets. This indicates that the
learnable emotion mixture wights can provide effective weak supervision by
leveraging the pre-trained pose-based emotion classifier. Besides, we have
observed significant improvement in the performance of FGDh+t during this
phase compared to the previous version. This aligns with our transformer
backbone’s emphasis on modeling the sequential temporal correlations as a
whole. The better transition gestures encourage our model to maintain better
temporal consistency, thereby the head/tail gestures achieve better results.

Finally, after additionally employing the keyframe sampler to produce
authority initial postures as the reference, our method obtains the best
performance. Although the properties of BC and FGD on both datasets just have
slightly improvement, the diversity realizes a noticeably better achievement
(_e.g_., 112.80 $\to$ 124.93 in the BEAT-ETrans dataset). This highly supports
our insight into keyframe-driven diversification strategy.

Figure 3: Visualization of our generated 3D co-speech gestures against various
state-of-the-art methods. The samples of the left part are from our newly
collected TED-ETrans dataset, and the samples of the right part are from our
BEAT-ETrans dataset. Best view on screen.

###  4.3 Qualitative Evaluation

Visualization: To fully demonstrate the performance of our method, we show the
visualized keyframes generated from ours compared with counterparts on our
newly collected TED-ETrans and BEAT-ETrans datasets, respectively. As depicted
in Figure 3, our method displays vivid and diverse results against others. In
particular, we observe that Seq2Seq and Trimodal tend to synthesize
unreasonable and stiff results (_e.g_., the blue rectangle in the right BEAT-
ETrans dataset). Although CAMN and HA2G can generate natural upper-body
postures, we find that they sometimes produce unreliable subtle fingers
(_e.g_., the red rectangle in the left TED-ETrans dataset). Both our method
and DiffGesture create reasonable gestures. However, the results synthesized
by DiffuGesture are mismatched with the emotion transitions. In contrast, our
method can synthesize the synchronous motions (_e.g_., in the BEAT-ETrans, the
arms become droopy as emotion turns to sadness). Meanwhile, we further verify
the diversification results as shown in Figure 1. Given the same input audio,
our method generates diverse and vivid co-speech gestures. Please refer to the
supplementary material for more visualization results.

User Study: To further analyze the quality of results by various counterparts
and ours, we conduct a user study by visualizing the results and inviting $15$
subjects. In particular, the subjects are required to rate the generated co-
speech gestures from $0$ to $5$ (the higher, the better) in terms of
naturalness, motion smoothness, and speech-gesture coherency. The results are
demonstrated in Figure 4. Our method showcases the best performance compared
with all the competitors. Especially in terms of motion smoothness, our method
achieves noticeable advantages, indicating the effectiveness of our proposed
motion transition infusion mechanism and the emotion mixture strategy.

Figure 4: User study on gesture naturalness, motion smoothness, and speech-
gesture coherency.

##  5 Conclusion

In this paper, we introduce a new task of 3D co-speech gesture generation
given emotion transition human speech. We therefore newly collected two
datasets named the BEAT-ETrans and the TED-ETrans to fulfill this goal while
significantly facilitating the research on 3D human motion modeling. Then, we
fully take advantage of the sequential temporal correlation via a motion
transition infusion mechanism to ensure the generated gestures preserve
temporal coherence. Furthermore, we design an emotion mixture strategy to
supply emotional weak supervision of the synthesized transitions. Extensive
experiments conducted on our two newly collected datasets show the superiority
of the method. As our method intends to generate diverse and vivid emotion
transition gestures, we will investigate diversifying the 3D gesture with
temporal smooth sampling, instead of the keyframe-wise manner.

Supplementary Material

Figure 5: The pipeline of dataset construction. Head and tail audios as well
as the corresponding transcripts are fed into the pipeline to generate a
smooth and high quality transition.

##  6 Overview

To demonstrate the effectiveness of our data construction techniques and the
proposed method of emotion transition co-speech gesture generation, we further
elaborate on the detailed data synthesis and vision perception in the
supplementary material. The additional content is illustrated in the following
folds:

  * •

Dataset Construction

  * •

Architecture Details

  * •

Additional Experiments

##  7 Dataset Construction

_We will release our newly collected the TED-ETrans and BEAT-ETrans datasets
in the future._ The overall pipeline of our approach to constructing the
dataset is displayed in Figure 5. The details involve the following steps:

Figure 6: Details of emotion transition distribution of our newly collected
TED-ETrans and BEAT-ETrans datasets. All the transitions start from the
neutral emotional speeches.

#### Segmentation and Emotion Labeling

: We first divide the previously aligned single emotion co-speech gesture
datasets [26, 29] into head and tail segments by splitting the original audio
into 4-second clips. Heads are identified as clips with neutral emotions,
while tails contain various emotions. This segmentation was achieved using
either the pre-annotated dataset’s emotion labels or an emotion classifier.
Both head and tail segments originated from the same speaker, ensuring vocal
tone consistency.

#### Emotion Transition

: The head segments consistently exhibit neutral emotions, while the tails
display a variety of emotional states. In our approach, we intentionally
avoided pairing segments with extreme emotional shifts (_e.g_., happiness-to-
anger, happiness-to-sadness). Such drastic transitions are infrequent in
natural speech and not only result in less smooth transitions in both speech
and textual contexts but also risk introducing a long-tail phenomenon in the
dataset. By avoiding these extremes, we aimed to maintain a more balanced and
realistic dataset distribution as shown in Figure 6.

#### Transcript Generation with GPT-4

: We engage GPT-4 to generate transitional text between the head and tail
clips. The GPT-4 is instructed to create a smooth transition in both content
and emotion, producing about 5-10 words. For each data sample, GPT-4 generated
three candidate transitions, each accompanied by a confidence score, returned
in JSON format. We finally discard samples with low confidence or excessive
length.

#### Synthesis of Transition Speech

: We employ the AudioLDM2 [27, 28] model for audio inpainting, ensuring
natural and time-controlled speech synthesis. Speaker embeddings are extracted
using SpeechBrain’s ECAPA-TDNN to measure the consistency of the transition
speech with the head and tail segments. Samples with significant speaker
embedding discrepancies are excluded. We ensure the head, tail, and
synthesized parts share the same speaker’s tone, maintaining consistency.

#### Quality Control through ASR

: We utilize Whisper [35] for automatic speech recognition (ASR) on transition
speech. ASR transcripts are compared to ground truth, and samples with the
word error rate of over 0.125 are re-synthesized for better accuracy and
clarity.

#### Final Note

: We observe that GPT-3.5 often produces similar candidates, lacking
diversity, thus our preference for GPT-4. Our final prompt structure, designed
to guide the model in generating contextually and emotionally coherent
transitions, is presented below:

Prompt ⬇ As a skilled playwright, you’ve been assigned a task to fill in the
blanks. You will be given two sentences (in a talk) with distinct emotions,
and your job is to provide a transition of **10 words** to ensure a natural
emotional and semantic flow between them. For each blank, you should return
three potential options along with your confidence level in your responses in
JSON format. **DO NOT** return anything else. JSON template: { ”opt1”: option
1, ”opt2”: option 2, ”opt3”: option 3, ”confi”: confidence score scale from 1
to 5, } Input:\n

##  8 Architecture Details

Figure 7: Details of our proposed keyframe sampling strategy. Once we obtain
the pre-trained keyframe encoder, we leverage it to model the conditional
distribution, producing diverse initial postures as the reference.

#### Audio Encoder.

Inspired by [4, 29, 49], the backbone of our audio encoder $E_{a}$ is
constructed as ResNetSE34. We adopt three stacking blocks and leverage the
2D-convolution-based header to map the dimension of audio features to be
$N\times 512$, where $N$ is the temporal dimension.

#### Transformer-based backbone.

We leverage the diversified authority initial postures to interact with the
extracted audio features. In particular, we leverage the pose reference $Q$ to
match the key features $K$ and value features $V$ in the transformer-based
encoder via three times Multi-Head Attention (MHA) [41], expressed as:

$\displaystyle MultiHead(Q,K,V)=softmax(\frac{QK}{\sqrt{d}})V,$ (10)

where $d$ is a normalization constant.

#### Pose-based Emotion Classifier.

In our emotion mixture strategy, we pre-train a pose-based emotion classifier
for providing emotional weak supervision on the generated transition gestures.
Specifically, the emotion classifier directly leverages the transformer
backbone, the same as the pipeline encoder, to extract the sequential pose
features. Then, we utilize an MLP-based classifier header on the pose gestures
to produce the final emotion categories. In the BEAT-ETrans dataset, our pre-
trained emotion classifier achieves 99.92% accuracy. In the TED-ETrans
dataset, the accuracy is 99.26%.

#### Keyframe Sampler.

We design a simple but effective VAE-based keyframe sampler to produce
authority initial postures as motion cues, thereby facilitating the
diversification of the generated 3D co-speech gestures. As shown in Figure 7,
the keyframe sampler aims to model the conditional distribution upon the given
randomly selected postures. In the pre-training phase, the posterior
distribution is denoted as the latent variable from the encoded chunk-wise
gestures. The prior distribution of this latent variable is modeled by the
keyframe encoder. The training goal in this phase is to minimize the distance
between the posterior distribution and the prior one via KL divergence
represented as $KL(\cdot\parallel\cdot)$ in Figure 7. Meanwhile, we exploit
the $L_{1}$ loss to constrain the reconstructed chunk-wise gestures.

##  9 Additional Experiments

###  9.1 Metric Calculation Details

Inspired by [29, 46], we take FGD to evaluate whether the generated gestures
maintain realism with the ground truth ones in the perceptive of distribution.
Conventionally, the feature extractor of FGD is calculated to embed overall
sequential gestures into latent space and then utilize a decoder for
reconstruction. However, since we do not have the ground truth of the
transition gestures, we newly pre-train the feature extractor with the
transition length $L$. In the inference stage, FGDh+t is calculated by
averaging the distances between five randomly selected chunks of length $L$
from the head/tail and GT, respectively. Similarly, FGDtrans is computed as
the average value between the distance of transition and five randomly
selected chunks of head/tail. _We will release the code of our pipeline and
evaluation metrics in the future._

###  9.2 Additional Visualization Results

Here, we provide more visual results of our methods compared with other
counterparts in the _demo video._ Meanwhile, to fully demonstrate the
effectiveness of our proposed components in the ablation study, we visualize
vital frames of the synthesized gestures. As illustrated in Figure 8 and
Figure 9, we can clearly observe that all the combinations of our proposed
components have positive impacts on the generated results.

Figure 8: Visual comparisons of ablation study on our newly collected TED-
ETrans dataset. We show the key frames of the generated motions given the
emotion transition of human speech. Best view on screen.  Figure 9: Visual
comparisons of ablation study on our newly collected BEAT-ETrans dataset. We
show the key frames of the generated motions given the emotion transition of
human speech. Best view on screen.

## References

  * Ao et al. [2022] Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu.  Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings.  _ACM Transactions on Graphics (TOG)_ , 41(6):1–19, 2022.
  * Ao et al. [2023] Tenglong Ao, Zeyi Zhang, and Libin Liu.  Gesturediffuclip: Gesture diffusion model with clip latents.  _arXiv preprint arXiv:2303.14613_ , 2023.
  * Cassell et al. [1999] Justine Cassell, David McNeill, and Karl-Erik McCullough.  Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information.  _Pragmatics & cognition_, 7(1):1–34, 1999.
  * Chung et al. [2020] Joon Son Chung, Jaesung Huh, Seongkyu Mun, Minjae Lee, Hee Soo Heo, Soyeon Choe, Chiheon Ham, Sunghwan Jung, Bong-Jin Lee, and Icksang Han.  In defence of metric learning for speaker recognition.  _arXiv preprint arXiv:2003.11982_ , 2020.
  * Desplanques et al. [2020] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck.  Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification.  _arXiv preprint arXiv:2005.07143_ , 2020.
  * Endo et al. [2023] Mark Endo, Joy Hsu, Jiaman Li, and Jiajun Wu.  Motion question answering via modular motion programs.  _arXiv preprint arXiv:2305.08953_ , 2023.
  * Farouk [2022] Maged Farouk.  Studying human robot interaction and its characteristics.  _International Journal of Computations, Information and Manufacturing (IJCIM)_ , 2(1), 2022.
  * Ferstl and McDonnell [2018] Ylva Ferstl and Rachel McDonnell.  Investigating the use of recurrent motion modelling for speech gesture generation.  In _Proceedings of the 18th International Conference on Intelligent Virtual Agents_ , pages 93–98, 2018.
  * Fu et al. [2022] Yu Fu, Yan Hu, and Veronica Sundstedt.  A systematic literature review of virtual, augmented, and mixed reality game applications in healthcare.  _ACM Transactions on Computing for Healthcare (HEALTH)_ , 3(2):1–27, 2022.
  * Ghorbani et al. [2023] Saeed Ghorbani, Ylva Ferstl, Daniel Holden, Nikolaus F Troje, and Marc-André Carbonneau.  Zeroeggs: Zero-shot example-based gesture generation from speech.  In _Computer Graphics Forum_ , pages 206–216. Wiley Online Library, 2023.
  * Ginosar et al. [2019] Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik.  Learning individual styles of conversational gesture.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3497–3506, 2019.
  * Goldin-Meadow and Alibali [2013] Susan Goldin-Meadow and Martha Wagner Alibali.  Gesture’s role in speaking, learning, and creating language.  _Annual review of psychology_ , 64:257–283, 2013.
  * Habibie et al. [2021] Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed Elgharib, and Christian Theobalt.  Learning speech-driven 3d conversational gestures from video.  In _Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents_ , pages 101–108, 2021.
  * Hostetter and Alibali [2008] Autumn B Hostetter and Martha W Alibali.  Visible embodiment: Gestures as simulated action.  _Psychonomic bulletin & review_, 15:495–514, 2008.
  * Huang et al. [2022] Ann Huang, Pascal Knierim, Francesco Chiossi, Lewis L Chuang, and Robin Welsch.  Proxemics for human-agent interaction in augmented reality.  In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , pages 1–13, 2022.
  * Huang and Belongie [2017] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1501–1510, 2017.
  * Jiang et al. [2023] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen.  Motiongpt: Human motion as a foreign language.  _arXiv preprint arXiv:2306.14795_ , 2023.
  * Kipp [2005] Michael Kipp.  _Gesture generation by imitation: From human behavior to computer character animation_.  Universal-Publishers, 2005.
  * Kopp et al. [2006] Stefan Kopp, Brigitte Krenn, Stacy Marsella, Andrew N Marshall, Catherine Pelachaud, Hannes Pirker, Kristinn R Thórisson, and Hannes Vilhjálmsson.  Towards a common framework for multimodal generation: The behavior markup language.  In _Intelligent Virtual Agents: 6th International Conference, IVA 2006, Marina Del Rey, CA, USA, August 21-23, 2006. Proceedings 6_ , pages 205–217. Springer, 2006.
  * Koppula and Saxena [2013] Hema Swetha Koppula and Ashutosh Saxena.  Anticipating human activities for reactive robotic response.  In _IROS_ , page 2071. Tokyo, 2013.
  * Levine et al. [2010] Sergey Levine, Philipp Krähenbühl, Sebastian Thrun, and Vladlen Koltun.  Gesture controllers.  In _Acm siggraph 2010 papers_ , pages 1–11. 2010.
  * Li et al. [2020] Jiaman Li, Yihang Yin, Hang Chu, Yi Zhou, Tingwu Wang, Sanja Fidler, and Hao Li.  Learning to generate diverse dance motions with transformer.  _arXiv preprint arXiv:2008.08171_ , 2020.
  * Li et al. [2021a] Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao.  Audio2gestures: Generating diverse gestures from speech audio with conditional variational autoencoders.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 11293–11302, 2021a.
  * Li et al. [2021b] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa.  Ai choreographer: Music conditioned 3d dance generation with aist++.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13401–13412, 2021b.
  * Liu et al. [2023a] Chen Liu, Peike Patrick Li, Xingqun Qi, Hu Zhang, Lincheng Li, Dadong Wang, and Xin Yu.  Audio-visual segmentation by exploring cross-modal mutual semantics.  In _Proceedings of the 31st ACM International Conference on Multimedia_ , pages 7590–7598, 2023a.
  * Liu et al. [2022a] Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng.  Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesis.  In _European Conference on Computer Vision_ , pages 612–630. Springer, 2022a.
  * Liu et al. [2023b] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley.  AudioLDM: Text-to-audio generation with latent diffusion models.  _Proceedings of the International Conference on Machine Learning_ , 2023b.
  * Liu et al. [2023c] Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley.  AudioLDM 2: Learning holistic audio generation with self-supervised pretraining.  _arXiv preprint arXiv:2308.05734_ , 2023c.
  * Liu et al. [2022b] Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou.  Learning hierarchical cross-modal association for co-speech gesture generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10462–10472, 2022b.
  * Mao et al. [2022] Wei Mao, Miaomiao Liu, and Mathieu Salzmann.  Weakly-supervised action transition learning for stochastic human motion prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8151–8160, 2022.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Petrovich et al. [2021] Mathis Petrovich, Michael J Black, and Gül Varol.  Action-conditioned 3d human motion synthesis with transformer vae.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10985–10995, 2021.
  * Qi et al. [2023a] Xingqun Qi, Chen Liu, Lincheng Li, Jie Hou, Haoran Xin, and Xin Yu.  Emotiongesture: Audio-driven diverse emotional co-speech 3d gesture generation.  _arXiv preprint arXiv:2305.18891_ , 2023a.
  * Qi et al. [2023b] Xingqun Qi, Chen Liu, Muyi Sun, Lincheng Li, Changjie Fan, and Xin Yu.  Diverse 3d hand gesture prediction from body dynamics by bilateral hand disentanglement.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4616–4626, 2023b.
  * Radford et al. [2023] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision.  In _International Conference on Machine Learning_ , pages 28492–28518. PMLR, 2023.
  * Ravanelli et al. [2021] Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, et al.  Speechbrain: A general-purpose speech toolkit.  _arXiv preprint arXiv:2106.04624_ , 2021.
  * Roach [1998] Peter Roach.  _Some languages are spoken more quickly than others_.  1998\.
  * Sargin et al. [2008] Mehmet E Sargin, Yucel Yemez, Engin Erzin, and Ahmet M Tekalp.  Analysis of head gesture and prosody patterns for prosody-driven head-gesture animation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 30(8):1330–1345, 2008.
  * Siyao et al. [2022] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu.  Bailando: 3d dance generation by actor-critic gpt with choreographic memory.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11050–11059, 2022.
  * Sohn et al. [2015] Kihyuk Sohn, Honglak Lee, and Xinchen Yan.  Learning structured output representation using deep conditional generative models.  _Advances in neural information processing systems_ , 28, 2015.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Voas et al. [2023] Jordan Voas, Yili Wang, Qixing Huang, and Raymond Mooney.  What is the best automated metric for text to motion generation?  _arXiv preprint arXiv:2309.10248_ , 2023.
  * Wang et al. [2022] Zijian Wang, Xingqun Qi, Kun Yuan, and Muyi Sun.  Self-supervised correlation mining network for person image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7703–7712, 2022.
  * Yi et al. [2023] Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J Black.  Generating holistic 3d human motion from speech.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 469–480, 2023.
  * Yoon et al. [2019] Youngwoo Yoon, Woo-Ri Ko, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee.  Robots learn social skills: End-to-end learning of co-speech gesture generation for humanoid robots.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 4303–4309. IEEE, 2019.
  * Yoon et al. [2020] Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee.  Speech gesture generation from the trimodal context of text, audio, and speaker identity.  _ACM Transactions on Graphics (TOG)_ , 39(6):1–16, 2020.
  * Zhao et al. [2023] Weiyu Zhao, Liangxiao Hu, and Shengping Zhang.  Diffugesture: Generating human gesture from two-person dialogue with diffusion models.  In _International Cconference on Multimodal Interaction_ , pages 179–185. 2023.
  * Zhi et al. [2023] Yihao Zhi, Xiaodong Cun, Xuelin Chen, Xi Shen, Wen Guo, Shaoli Huang, and Shenghua Gao.  Livelyspeaker: Towards semantic-aware co-speech gesture generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 20807–20817, 2023.
  * Zhu et al. [2023] Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu.  Taming diffusion models for audio-driven co-speech gesture generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10544–10553, 2023.


# Learning Crowd Behaviors in Navigation with Attention-based Spatial-Temporal
Graphs

Yanying Zhou1 and Jochen Garcke1 1 All the authours are from Bonn University,
Germany.

###### Abstract

Safe and efficient navigation in dynamic environments shared with humans
remains an open and challenging task for mobile robots. Previous works have
shown the efficacy of using reinforcement learning frameworks to train
policies for efficient navigation. However, their performance deteriorates
when crowd configurations change, i.e. become larger or more complex. Thus, it
is crucial to fully understand the complex, dynamic, and sophisticated
interactions of the crowd resulting in proactive and foresighted behaviors for
robot navigation. In this paper, a novel deep graph learning architecture
based on attention mechanisms is proposed, which leverages the spatial-
temporal graph to enhance robot navigation. We employ spatial graphs to
capture the current spatial interactions, and through the integration with
RNN, the temporal graphs utilize past trajectory information to infer the
future intentions of each agent. The spatial-temporal graph reasoning ability
allows the robot to better understand and interpret the relationships between
agents over time and space, thereby making more informed decisions. Compared
to previous state-of-the-art methods, our method demonstrates superior
robustness in terms of safety, efficiency, and generalization in various
challenging scenarios.

##  I INTRODUCTION

As artificial intelligence advances, robots are becoming essential in modern
human life. However, ensuring the safety and efficiency of social-aware robots
navigating in crowds is a considerable challenge [1, 2, 3]. These robots must
not only avoid collisions and reach their destinations promptly but also act
in a socially compliant manner and maintain a pleasant respectful interaction
with pedestrians. Therefore, we aim to develop a proactive collision avoidance
scheme that is both adaptive and generalizable.

Social-aware robot navigation tasks have been extensively studied and achieved
many successful implementations. Early rule-based methods [4, 5, 6, 7, 8] such
as Optimal Reciprocal Collision Avoidance (ORCA) [5] and Social Forces (SFs)
[6, 7, 8] use artificially set geometric or physical rules, where the next
action of the robot is taken depending on one-step rules and the current
state. This makes these methods short-sighted and highly dependent on the
accuracy of the model. Further, trajectory-based methods [9, 10, 11] plan an
appropriate path for the robot after predicting the future trajectories of
other agents, which results in far-sighted decisions. However, both methods
are prone to the freezing robot problem (FRP), which means that the planner
determines that all feasible routes are unsafe and the robot gets stuck [12,
13].

Figure 1: Illustration of our work. Our model uses spatial-temporal graphs to
capture complex crowd dynamics, which aggregates both spatial and temporal
attention maps from each agent.

Recently, the use of Deep Reinforcement Learning (DRL) [14, 15, 16, 17, 18,
19] has enabled the development of learning-based methods for effective
navigation strategies. These methods train a value network using state
transitions of agents to estimate the value of each state. An optimal policy
is then derived from the value network, guiding the robot to make decisions
and choose the action that leads to the state with the highest value. They
implicitly encode the interactions between agents, capturing the collective
impact of the crowd by incorporating pairwise interactions through LSTMs [20]
or the maximin operator. However, interactions vary significantly across
different environments, rendering navigation policies trained in one
environment unsuitable for another. This is due to uncertainties in the
dynamic crowd, which affects the way and impact of interactions. Specifically,
as the density of crowds increases, it leads to increased mutual influence and
interference among individuals, which in turn increases the difficulty and
complexity of human-robot interaction in such environments. Therefore, robots
need to timely adjust their behavior strategies according to changes in crowd
behavior, to better adapt to different scenarios and improve the quality of
human-robot interaction.

In this paper, we propose an attention-based spatial-temporal graph learning
framework for crowd navigation trained with DRL, which is called ”ASTG”.
First, we introduce graph attention networks to fully model both spatial and
temporal relations of the crowd navigation scenario separately. In addition,
we incorporate an RNN [21] to encode the historical trajectory information of
the agents, before modeling temporal interactions for implicit reasoning about
the intentions of all humans. Then, we jointly aggregate the pairwise spatial-
temporal interactions into a social attention mechanism to capture the
relative importance of each human. Together, a comprehensive understanding of
crowd behaviors enables efficient robot navigation policies, adaptable to
various crowded scenarios.

We present the following contributions: (1) We introduce a novel deep graph
neural network ASTG, which fully models the spatial and temporal relations
efficiently in crowd navigation. (2) We combine the above with a social-
attention mechanism to encode the collective influence of neighbors. (3) We
demonstrate the improved performance of our approach in comparison to previous
methods, exhibiting strong robustness and generalization in both simple and
complex scenarios with varying numbers of humans.

##  II RELATED WORK

###  II-A Social-aware Robot Navigation

Early works model agent interactions through rule-based methods [4, 5, 6, 7,
8] or trajectory-based methods [9, 10, 11]. Rule-based methods have a fast
response, but are prone to encounter the freezing robot problem. This is
because they take actions that only consider the current state without a long-
term view ahead. Trajectory-based methods generate a path for the robot to
follow after predicting other agents’ future trajectories, which can be
computationally intensive, especially when the environment has many dynamic
agents.

Figure 2: Network architecture from Section IV. (a) The spatial graph utilizes
the GAT to encode direct and indirect spatial interactions between agents. (b)
The temporal graph incorporates an RNN to reason about the temporal
interactions based on historical information. (c) The social attention module
jointly aggregates the pairwise spatial-temporal interactions to capture the
crowd representation in the crowd feature, which is then used to estimate the
action values.

Recently, combined with deep learning algorithms and RL, learning-based
methods [22, 23, 24, 25, 26] learn how to achieve good navigation performance
in an environment. A self-attention module is built to compute the relative
importance in order to take actions, called SARL [22]. [23] propose a
structural recurrent neural network to reason about spatial and temporal
relationships for planning.

All of the above methods, however, may not generalize well to new or unseen
environments and may require additional training data and fine-tuning for each
new environment. Significantly, deep reinforcement learning models can suffer
from overfitting to the training data, leading to poor generalization
performance in new states. To tackle the problem, we introduce the graph
attention network to adaptively weigh the importance of different nodes in the
graph structure, providing a more informative and discriminative
representation of the data. In addition, the attention mechanisms in Graph
Attention Network (GAT) can also provide additional regularization, helping to
prevent overfitting to the training data and improve generalization
performance.

###  II-B Graph-based Learning

Currently, graph neural networks (GNNs) are widely utilized in various
applications, including social network analysis, to learn the relationships
between entities [27, 28, 29, 30, 31, 32, 33, 34]. Combining the robust
representation capabilities of graphs with the powerful end-to-end learning of
neural networks, GNNs represent components as nodes and their relationships as
edges. Through local message passing on graphs, these networks efficiently
encapsulate relations between nodes, proving effective in a wide array of
structured tasks.

GNNs can be used to extract efficient representations in crowd navigation.
Graph Convolutional Network (GCN) [28, 29, 30] and GAT [34, 33, 35] are the
typical types of GNNs. Chen et al. [28] present a relational graph model to
learn the agents’ interaction based on the two-layer GCN. G-GCNRL [29] learns
attention weights and aggregates the crowd information with two GCNs. Unlike
GCNs that use fixed weights in convolution operations, GAT uses learnable
attention coefficients to weigh the contribution of each neighbor when
aggregating information. In [34], Zhou et al. introduce GAT to extract a nice
graph representation. However, it only refers to spatial interaction without
the temporal dynamics, which also are useful for planning paths in crowd
navigation. Thus, our method applies GAT on the spatial graph and temporal
graph separately to encode the environment relationships for dealing with
uncertain dynamic environments. In addition, for better decision-making, we
incorporate RNN into encoding temporal features to reason about the future
intentions of each agent based on the current and past observations of each
human. We show that combing spatial-temporal GAT with RNN improves performance
in dynamic social navigation environments.

##  III PROBLEM FORMULATION

###  III-A Crowd Navigation Modeling

We consider the crowd robot navigation task, where a holonomic robot needs to
move towards a goal position through a crowd of $n$ humans. The task can be
formulated as a sequential decision-making problem. Assume that all agents
move in a two-dimensional space with Euclidean geometry. The current position
$\mathbf{p}=[p_{x},p_{y}]$, velocity $\mathbf{v}=[v_{x},v_{y}]$ and radius $r$
of each agent can be observed by the rest. The goal position
$\mathbf{g}=[g_{x},g_{y}]$ and the preferred velocity $v_{pref}$ are the
hidden parts that the robot is aware of only for itself and one cannot observe
the hidden states of humans. Let $\mathbf{s}_{t}^{r}$ be the current state of
the robot and
$\mathbf{s}_{t}^{h}=[\mathbf{s}_{t}^{1},\mathbf{s}_{t}^{2},...,\mathbf{s}_{t}^{n}]$
be the observable state of humans. Thus, the joint state for robot navigation
is $\mathbf{s}_{t}^{jn}=[\mathbf{s}_{t}^{r},\mathbf{s}_{t}^{h}]$.

We transform the coordinate system with robot-centric coordinates, where the
robot is set as the center and the $x$-axis is the direction towards the goal.
Thereby, the robot state and the $i$-th human state at time $t$ are defined
as:

$\begin{split}&s^{r}=[d_{g},v_{x}^{r},v_{y}^{r},v_{pref},r],\\\
&s^{i}=[p_{x}^{i},p_{y}^{i},v_{x}^{i},v_{y}^{i},r^{i},d^{i},r^{i}+r],\end{split}$
(1)

where $d_{g}$ is the $L_{2}$ distance between the robot and its goal. $d^{i}$
is the $L_{2}$ distance between the robot and the $i$-th-human.

###  III-B Reinforcement Learning for Crowd Navigation

In this work, a Deep V-learning approach [29] is employed to evaluate the
state values and choose the best action. The robot starts from an initial
state $\mathbf{s}^{jn}_{0}$ at the beginning of each episode. Then, based on
the learned policy $\pi(\mathbf{a}_{t}|\mathbf{s}^{jn}_{t})$, after taking an
action $\mathbf{a}_{t}\in\mathcal{A}$ at each timestep $t$, the robot receives
a reward $r_{t}$ and transitions to the successor state
$\mathbf{s}^{jn}_{t+1}$. For humans, based on their policies, they take
actions and transition to their successor states. For each episode, the
process continues until the robot reaches the destination, collides with other
humans, or the navigation time $t$ exceeds the episode length $T$. The
objective is to maximize the cumulative reward for each state and thereby to
find the optimal policy $\pi^{*}(\mathbf{s}^{jn}_{t})$:

$\begin{split}\pi^{*}&(\mathbf{s}^{jn}_{t})=\mathop{\arg\\!\max}_{\mathbf{a}_{t}}R(\mathbf{s}^{jn}_{t},\mathbf{a}_{t})+\\\
&\gamma^{\Delta t\cdot v_{pref}}\int_{\mathbf{s}^{jn}_{t+\Delta
t}}{P(\mathbf{s}^{jn}_{t},\mathbf{a}_{t},\mathbf{s}^{jn}_{t+\Delta
t})V^{\ast}(\mathbf{s}^{jn}_{t+\Delta t})\,d\mathbf{s}^{jn}_{t+\Delta
t}},\end{split}$
$V^{*}(s_{t}^{jn})=\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}\cdot
v_{pref}}R(\mathbf{s}^{jn}_{t^{\prime}},\pi^{*}(\mathbf{s}^{jn}_{t^{\prime}})),$
(2)

where $\Delta t$ is the time step and
$r_{t}=R(\mathbf{s}^{jn}_{t},\mathbf{a}_{t})$ denotes the received reward at
time $t$. $P(\mathbf{s}^{jn}_{t},\mathbf{a}_{t},\mathbf{s}^{jn}_{t+\Delta t})$
is the transition probability between $t$ and $t+\Delta t$. $\gamma\in(0,1)$
is a discount factor that is normalized by the preferred velocity $v_{pref}$.
We utilize the reward function formulated as in [22], which provides rewards
for task accomplishment while simultaneously imposing penalties for collisions
or uncomfortable distances.

##  IV METHODOLOGY

We propose a novel framework for social-aware robot navigation, see Fig. 2.
This framework leverages the superior flexibility of graph structures over
feed-forward neural networks, which are limited to a fixed number of agents.

###  IV-A Spatial Graph Representation

The spatial-temporal graph network calculates the spatial attention and
temporal attention separately from the environment state matrix, which is used
to represent the edges of the spatio-temporal graph. In the spatial graph, we
input rough pairwise spatial interactions to calculate agents’ spatial
dependencies. For $i$-th human, we initially aggregate its state at the
current moment with that of the robot to compute rough spatial interactions
$(\mathbf{s}^{r}\oplus\mathbf{s}^{i})$. Then, we calculate spatial attention
maps through graph attention network layers, representing the importance of
adjacent interactions, and encode spatial features such as distance and
relative direction between agents in pairwise interactions. For instance,
interaction at the $i$-th node is influenced by its connection with other
nodes, with a more substantial influence reflected by a higher attention
weight. Simultaneously, this node aggregates spatial information from adjacent
interactions through the influence of spatial edges, representing the
collective impact on the node from the crowd dynamics. This impact not only
considers explicitly modeled pairwise human-robot interactions but also
implicitly captures indirect human-human interactions and crowd-robot
interactions through edge weights.

As shown in Fig. 2, in the spatial graph, we use the states of the robot and
the $i$-th human at time $t$ to derive a node feature via a multilayer
perceptron ($\text{MLP}_{S}$):

$e^{i}_{t}=f_{spatial}(\mathbf{s}^{r}_{t},\mathbf{s}^{i}_{t};W_{e}),$ (3)

where $W_{e}$ are the network weights and $f_{spatial}(\cdot)$ is an MLP with
a ReLU activation function. The rectified linear unit (ReLU) activation
function used in the framework aims to capture non-linear features in the
feedforward network.

We then use a graph attention layer to model spatial interactions in a crowd.
The input is denoted by $E=[e^{1}_{t},...,e^{n}_{t}]$ and $e^{i}=e^{i}_{t}$
for short. The normalized spatial coefficients $\alpha_{ij}^{S}$ in the
attention mechanism can be computed by:

$\alpha^{ij}_{S}=\frac{{\textrm{exp}}({\textrm{LeakyRelu}}(a[\mathbf{W}e^{i}||\mathbf{W}e^{j}]))}{\sum_{k\in{\mathcal{N}}^{i}}{\textrm{exp}}({\textrm{LeakyRelu}}(a[\mathbf{W}e^{i}||\mathbf{W}e^{k}]))},$
(4)

where $\alpha^{ij}_{S}$ represents the importance of node $j$ to node $i$.
$\mathbf{W}(\cdot)$ is the weight matrix and $||$ is the concatenation
operation, while ${\mathcal{N}}^{i}$ indicates the neighborhood of node $i$ in
the graph. After passing through a fully connected network (FCN) $a(\cdot)$, a
LeakyReLU activation follows. Finally, a softmax function is used to normalize
the spatial attention coefficients $\alpha_{ij}^{S}$. Thus, the output node
features of the graph attention layer are:

${\tilde{e}}^{\,i}=\sigma(\sum_{j\in{\mathcal{N}}^{i}}\alpha^{ij}_{S}~{}\mathbf{W}e^{\,j}),\quad\tilde{E}=[{\tilde{e}}^{\,1},...,\tilde{e}^{\,n}],$
(5)

In this work, through the graph attention layer, the $i$-th interaction
feature is transformed into a higher-level spatial feature that models
indirect robot-human and human-human interactions in a crowd. Additionally,
residual connections are employed in the spatial and temporal graph networks
to accelerate convergence and stabilize the framework [36]. Thus, the final
spatial features $H_{spatial}$ is described by:

$H_{spatial}=E+{\tilde{E}},\quad H_{spatial}=[H_{S}^{1},...,H_{S}^{n}].$ (6)

###  IV-B Temporal Graph Representation

Due to the high motion-dependency of the temporal dimension, we introduce an
RNN for each agent to capture the dynamics of its trajectory. Then, in the
temporal graph, we treat the motion feature of each human as a node of the
graph, and the edges signify the relationships between humans’ motions, whose
weights can indicate the importance level. Since motion is continuous, we can
predict future actions based on the current motion information. To achieve
this, we utilize graph attention network layers to compute a temporal
attention map, estimating the mutual influence of neighboring agents’ motion
behaviors and aggregating these influences to form temporal features. These
temporal features encompass both the current trajectory information of humans
and predictions of future trajectories, allowing the robot to better
understand human behavior and intentions.

First, we embed $i$-th human’s state through an $\text{MLP}_{T}$ into a fixed
length vector $g^{i}_{t}$:

$g^{i}_{t}=f_{temporal}(\mathbf{s}^{i}_{t};W_{g}),$ (7)

where $W_{g}$ are the embedding weights and $f_{temporal}(\cdot)$ is an MLP
with ReLU activation function. Then, we process $g_{i}^{t}$ with an RNN cell:

$h^{i}_{t}={\textrm{RNN}}(h^{i}_{t-1},g^{i}_{t}),$ (8)

where $h^{i}_{t}$ is the hidden state at time $t$, which changes over time,
reflecting the evolution of $i$-th agent’s state.

The hidden states $h^{i}_{t}$ are then fed into the graph attention layer to
model temporal interactions. GAT learns the attention distribution between
nodes adaptively, enabling it to weigh the information from different nodes
based on their relationship. This implies that GAT facilitates the propagation
of temporal interaction information between nodes. The input is
$H=[h^{1}_{t},...,h^{n}_{t}]$ and $h^{i}=h^{i}_{t}$ for short. The normalized
temporal coefficients $\alpha_{ij}^{T}$ is described by:

$\alpha^{ij}_{T}=\frac{{\textrm{exp}}({\textrm{LeakyRelu}}(\alpha[\mathbf{W}h^{i}||\mathbf{W}h^{j}]))}{\sum_{k\in{\mathcal{N}}^{i}}{\textrm{exp}}({\textrm{LeakyRelu}}(\alpha[\mathbf{W}h^{i}||\mathbf{W}h^{k}]))},$
(9)

Combining RNN and GAT allows for a more accurate capture of the evolving
states and interactive changes of agents in a social environment over time.
While RNN captures the temporal variations in agents’ states, GAT, through its
attention mechanism, considers interactions with other nodes, leading to a
finer-grained temporal dynamic modeling. By comprehensively considering the
evolution of each agent’s state alongside their social relationships, the
robot can better predict future human behaviors and intentions, and even
assist in analyzing collective behavioral patterns within a group. Hence, the
final temporal graph feature with a one-layer GAT can be described as follows:

$\tilde{h}^{i}=\sigma(\sum_{j\in{\mathcal{N}}^{i}}\alpha^{ij}_{T}~{}\mathbf{W}h^{j}),\quad\tilde{H}=[\tilde{h}^{1},...,\tilde{h}^{n}],$
(10)

After deploying a residual connection structure, the final temporal features
$H^{temporal}$ is:

$H_{temporal}=H+\tilde{H},\quad H_{temporal}=[H_{T}^{1},...,H_{T}^{n}].$ (11)

###  IV-C Social Attention Mechanism

To capture the uncertainty of crowd movements, we build a social attention
module that fuses the spatial and temporal features of each agent and captures
dependencies among agents. Inspired by [22], we build a self-attention network
to assign attention weights $w^{i}$ for each agent’s aggregated spatial-
temporal pairwise feature $ST^{\,i}=[H_{S}^{i},H_{T}^{i}]$ and encode the
collective impact of a crowd.

First, $ST^{\,i}$ is transformed into an attention score $w^{i}$:

$ST^{\,m}=\frac{1}{n}\sum_{k=1}^{n}ST^{\,k},\quad
w^{i}=f_{\alpha}(ST^{\,i},ST^{\,m};W_{\alpha}),$ (12)

where $W_{\alpha}$ is the weight and $f_{\alpha}$ is nonlinear transformation.
We then obtain the final crowd representation $c$ by the product sum of each
spatial-temporal pairwise feature $ST^{\,i}$ and attention weights $w_{i}$:

$c=\sum_{i=1}^{n}softmax(w^{i})ST^{\,i}$ (13)

###  IV-D Graph-Based Planning

The crowd representation $c$ is the input to $\text{MLP}_{V}$ to estimate the
state value $V$ for planning:

$V=f_{v}(\mathbf{s}^{r},c;W_{v})$ (14)

where $W_{v}$ denotes the weights.

Figure 3: Quantitative evaluation on scenarios with different numbers of
humans.

Methods | different numbers of dynamic humans | 5 dynamic humans and different numbers of static humans
---|---|---
Disc.(%) | $t_{succ.nav}$ | $t_{weighted.nav}$ | Disc.(%) | $t_{succ.nav}$ | $t_{weighted.nav}$
5 | 10 | 15 | 5 | 10 | 15 | 5 | 10 | 15 | 2 | 4 | 8 | 2 | 4 | 8 | 2 | 4 | 8
SARL [22] | 0.02 | 0.11 | 0.22 | 10.58 | 11.64 | 12.33 | 10.91 | 14.18 | 17.80 | 0.05 | 0.09 | 0.15 | 10.51 | 10.64 | 11.16 | 11.10 | 11.71 | 13.52
RGL [28] | 0.21 | 0.42 | 0.53 | 10.23 | 11.43 | 12.51 | 12.57 | 17.65 | 21.60 | 0.34 | 0.43 | 0.57 | 10.59 | 10.97 | 12.06 | 13.88 | 15.53 | 18.41
SG-DQN [34] | 0.05 | 0.08 | 0.09 | 10.03 | 11.24 | 13.41 | 10.63 | 13.71 | 16.56 | 0.04 | 0.06 | 0.09 | 10.07 | 11.24 | 12.07 | 10.42 | 11.79 | 13.62
Ours | 0.01 | 0.04 | 0.08 | 11.40 | 12.65 | 13.28 | 11.45 | 12.97 | 14.13 | 0.03 | 0.06 | 0.10 | 11.09 | 11.22 | 11.97 | 11.25 | 11.56 | 12.83

Table I: Evaluation performance comparison in the simple scenarios. ”Disc.
(%)” is the average frequency of duration that the robot invades the humans’
comfort area. ”$t_{succ.nav}$” is the average navigation time of success
cases. ”$t_{weighted.nav}$” (Eq.(18)) is a novel weighted navigation time
metric considering the impact of the discomfort steps and collision cases.

##  V EXPERIMENTS

###  V-A Simulation Environment

We build our 2D simulation environment for crowd robot navigation following
[22]. We keep the invisible setting for the robot to prevent humans from
overreacting to the robot and thus avoid learning an aggressive navigation
policy that moves too directly to the goal position and gets too close to
humans. We use circle crossing scenarios with a radius of 4m in our
simulation. All humans are controlled by the ORCA policy. Dynamic humans are
randomly positioned on the circle and will move to opposite positions on the
same circle, while static humans are randomly located in the same circle and
do not move. To fully analyse the effectiveness of the proposed model, we
evaluate all models for 1000 random test cases in both simple and complex
scenarios.

(a) DS (distributed)

(b) RO (row3and2)

(c) CO (concave5)

Figure 4: Simple and complex scenarios. They are all with 5 dynamic humans and
5 static humans with different combinations, such as (a) Completely dispersed.
(b) Divided into two rows with 3 and 2 static humans separately. (c) Formed as
concave.

###  V-B Implementation Details

Baselines for comparison: We compare the performance of our model with three
state-of-the-art methods: SARL [22], RGL [28] and SG-DQN [34]. SARL is a
baseline for Deep V-learning. In addition, RGL and SG-DQN are used as
baselines for graph-based learning methods.

Training: We train all methods in an environment that consists of 5 dynamic
and 2 scattered static humans using data from 10k episodes. For a fair
comparison, the network architectures of all baselines are trained as stated
in the original papers.

Evaluation: We evaluate the generalization capability of the policy of each
method, learned in the above training scenario, in both simple and complex
scenarios with 1k episodes. To vary the start and end positions of agents over
the episodes we use different random seeds, where each (random) episode
configuration is evaluated for all methods. As evaluation metrics, we use the
percentage of success, collision, and timeout cases, respectively, as well as
navigation time-related metrics.

Simple scenarios are set up with different numbers of distributed dynamic
and/or static humans (like Fig. 4(a)), such as scenarios with only $n=5,10,15$
dynamic humans, and scenarios with 5 dynamic humans or $m=2,4,8$ scattered
static humans. In addition, based on 5 dynamic humans, the complex scenario
has 5 static humans with different group combinations (like Fig. 4(b),(c)),
whose positions are randomly set in the circle.

###  V-C Quantitative Evaluation

To show the effectiveness of our method, we compare our method with SARL [22],
RGL [28] and SG-DQN [34]. Fig.3, 5, Tab.I, II show the results of all methods
under different environments, where ASTG is the best one with the highest
success rate and the lowest comfort rates in all scenarios.

In Fig. 3 and 5, we observe that the SARL method [22] demonstrates restricted
performance and social compliance due to its inability to fully capture
intricate social interactions. Moreover, despite utilizing GNN for modeling
human-robot interactions, RGL [28] has not demonstrated strong generalization
performance. This limitation stems from the GCN’s simple convolution
operations applied on features dictated by fixed rule-based weights, which
weaken adaptability to varying environments, leading to a sharp increase in
collision rates as the obstacles’ amount grows. In contrast, GAT can assign
different weights based on the importance of these features, thereby better
capturing critical environmental characteristics. This advantage is visible in
SG-DQN’s [34] competitive stance alongside our ASTG. However, SG-DQN [34]
lacks the ability to capture time-varying motion features, thus reducing
predictive accuracy. In comparison, our ASTG exhibits significantly slower
increases in collision rates and navigation times in pure dynamic environments
and environments containing both dynamic and static humans, demonstrating
excellent generalization performance.

Figure 5: Quantitative evaluation under complex scenarios.

Group statics | Disc.(%) | $t_{succ.nav}$ | $t_{weighted.nav}$
---|---|---|---
DS | RO | CO | DS | RO | CO | DS | RO | CO
SARL [22] | 0.11 | 0.10 | 0.05 | 10.83 | 11.08 | 11.54 | 12.18 | 12.31 | 13.21
RGL [28] | 0.44 | 0.56 | 0.59 | 11.25 | 11.37 | 11.27 | 16.22 | 16.50 | 18.58
SG-DQN [34] | 0.07 | 0.07 | 0.06 | 10.96 | 11.25 | 11.15 | 11.93 | 12.20 | 13.06
Ours | 0.06 | 0.06 | 0.03 | 11.35 | 11.34 | 11.05 | 11.75 | 11.76 | 12.21

Table II: Evaluation performance comparison in the scenarios with 5 dynamic
humans and different static groups (DS (distributed), RO (row3and2), and CO
(concave)).

From Tab.I, II, it shows that the baselines enjoy a better performance in
$t_{succ.nav}$, which is defined as the average navigation time of the
successful cases in seconds. However, they have either a higher discomfort
frequency or higher collision rate, which means, to achieve a shorter
navigation time, the robot takes a lot of aggressive actions, badly affecting
human comfort or even having collisions. To better measure and analyse the
behavior, we propose a weighted navigation time metric that penalizes
aggressive policies:

$t_{weighted.nav}=\frac{t_{dnav}+N_{coll}*t_{limit}}{N_{succ}+N_{coll}},$ (15)

where $N_{succ}$ and $N_{coll}$ are the number of success and collision cases,
respectively. $t_{limit}$ is the maximum navigation time, which set as 25s in
our simulations. $t_{dnav}$ is extended from the original navigation time
$t_{nav}$ to penalize the impact of discomfort. Specifically, if a step is
causing discomfort, we add half of a time step to $t_{succ.nav}$, which equals
$t_{dnav}=t_{succ.nav}+N_{disc}*0.5*{\Delta}t$ and $N_{disc}$ is the number of
discomfort steps in success cases. Thus, when considering collision cases and
the discomfort frequency, our method shows a shorter weighted navigation time
$t_{weighted.nav}$.

(a) Only spatial graph

dyn

(b) Only temporal graph

(c) Full ASTG

Figure 6: Simulation trajectories on the same case.

(a) SARL

(b) SG-DQN

(c) Ours

(d) SARL

(e) RGL

(f) Ours

Figure 7: Trajectory comparisons under simple and complex scenarios.

###  V-D Ablation Study

To assess the individual contributions of the different components in our
model, we conduct an ablation study where we independently train and test the
spatial and temporal branches with the same conditions as before. Fig. 6 shows
that the spatial graph and ASTG learn to navigate a safer side away, while the
spatial graph has a longer navigation time. Furthermore, the temporal graph
and ASTG are good at reasoning about the dynamics of the crowd, while ASTG
demonstrates better social norm compliance. These findings suggest that
incorporating the spatial and temporal graph can significantly improve the
performance and better adapt to dense scenarios with dynamic varying numbers
of humans.

###  V-E Qualitative Evaluation

To qualitatively evaluate the effectiveness of our method, we show in Fig.7
the robot’s trajectories for different methods under simple and complex
scenarios. In the simple scenario with 10 dynamic humans, Fig. 7(top), SARL
collides with humans, while SG-DQN detours largely to achieve the goal. In
contrast, our method slightly detours at first and then directly reaches the
goal, resulting in a shorter path. When the scenarios are extended from simple
to complex, Fig.7(bottom) shows SARL and RGL have timeouts in the static group
scenario. As shown in Fig.7(f), our method outperforms these two methods by
slowing down at first and then going through the crowd. Spatial-temporal graph
reasoning helps to understand relationships between agents over time and space
and use that to make more informed decisions. Our method thus is able to
better adapt to dense and complex situations in uncertain environments.

##  VI CONCLUSIONS

In this work, we proposed a novel attention-based spatial-temporal graph
learning model for crowd robot navigation. By formulating spatial and temporal
graphs with a GAT, we implicitly compute both direct and indirect spatial
interactions and temporal crowd features so that our model can reason for
decision-making under changing scenarios. Integrating a RNN, our model
incorporates past trajectories into the temporal graph to also reason about
the future intentions of each agent. In the experiments, we showed our model
outperforms baseline methods and can handle both uncertain simple and complex
environments. Based on the promising results, we intend in future work to
further analyse and extend the GAT-module, for example using multiple graph
layers, which can help to extract deeper information from the crowd.

## References

  * [1] C. Mavrogiannis, F. Baldini, A. Wang, D. Zhao, P. Trautman, A. Steinfeld, and J. Oh, “Core challenges of social robot navigation: A survey,” _arXiv preprint arXiv:2103.05668_ , 2021.
  * [2] R. Möller, A. Furnari, S. Battiato, A. Härmä, and G. M. Farinella, “A survey on human-aware robot navigation,” _Robotics and Autonomous Systems_ , vol. 145, p. 103837, 2021.
  * [3] X. Xiao, B. Liu, G. Warnell, and P. Stone, “Motion planning and control for mobile robot navigation using machine learning: a survey,” _Autonomous Robots_ , vol. 46, no. 5, pp. 569–597, 2022.
  * [4] J. Van den Berg, M. Lin, and D. Manocha, “Reciprocal velocity obstacles for real-time multi-agent navigation,” in _2008 IEEE ICRA_ , 2008, pp. 1928–1935.
  * [5] J. Van Den Berg, S. J. Guy, M. Lin, and D. Manocha, “Reciprocal n-body collision avoidance,” in _Robotics Research: The 14th International Symposium ISRR_. Springer, 2011, pp. 3–19.
  * [6] G. Ferrer, A. Garrell, and A. Sanfeliu, “Social-aware robot navigation in urban environments,” in _2013 European Conference on Mobile Robots_ , 2013, pp. 331–336.
  * [7] G. Ferrer, A. G. Zulueta, F. H. Cotarelo, and A. Sanfeliu, “Robot social-aware navigation framework to accompany people walking side-by-side,” _Autonomous robots_ , vol. 41, no. 4, pp. 775–793, 2017.
  * [8] X.-T. Truong and T. D. Ngo, “Toward socially aware robot navigation in dynamic and crowded environments: A proactive social motion model,” _IEEE Transactions on Automation Science and Engineering_ , vol. 14, no. 4, pp. 1743–1760, 2017.
  * [9] K. Li, M. Shan, K. Narula, S. Worrall, and E. Nebot, “Socially aware crowd navigation with multimodal pedestrian trajectory prediction for autonomous vehicles,” in _2020 IEEE ITSC_ , 2020, pp. 1–8.
  * [10] G. S. Aoude, B. D. Luders, J. M. Joseph, N. Roy, and J. P. How, “Probabilistically safe motion planning to avoid dynamic obstacles with uncertain motion patterns,” _Autonomous Robots_ , vol. 35, no. 1, pp. 51–76, 2013.
  * [11] Y. Chen, F. Zhao, and Y. Lou, “Interactive model predictive control for robot navigation in dense crowds,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 52, no. 4, pp. 2289–2301, 2021.
  * [12] P. Trautman and A. Krause, “Unfreezing the robot: Navigation in dense, interacting crowds,” in _IEEE/RSJ IROS_ , 2010, pp. 797–803.
  * [13] A. J. Sathyamoorthy, U. Patel, T. Guan, and D. Manocha, “Frozone: Freezing-free, pedestrian-friendly navigation in human crowds,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 3, pp. 4352–4359, 2020.
  * [14] Y. Li, “Deep reinforcement learning: An overview,” _arXiv preprint arXiv:1701.07274_ , 2017.
  * [15] K. Zhu and T. Zhang, “Deep reinforcement learning based mobile robot navigation: A review,” _Tsinghua Science and Technology_ , vol. 26, no. 5, pp. 674–691, 2021.
  * [16] W. Zhao, J. P. Queralta, and T. Westerlund, “Sim-to-real transfer in deep reinforcement learning for robotics: A survey,” in _2020 IEEE Symp. Series on Comput. Intelligence (SSCI)_ , 2020, pp. 737–744.
  * [17] R. Wang, W. Wang, and B.-C. Min, “Feedback-efficient active preference learning for socially aware robot navigation,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 11 336–11 343.
  * [18] S. S. Samsani, H. Mutahira, and M. S. Muhammad, “Memory-based crowd-aware robot navigation using deep reinforcement learning,” _Complex & Intelligent Systems_, vol. 9, no. 2, pp. 2147–2158, 2023.
  * [19] H. Zeng, R. Hu, X. Huang, and Z. Peng, “Robot navigation in crowd based on dual social attention deep reinforcement learning,” _Mathematical Problems in Engineering_ , vol. 2021, pp. 1–11, 2021.
  * [20] A. Graves and A. Graves, “Long short-term memory,” _Supervised sequence labelling with recurrent neural networks_ , pp. 37–45, 2012.
  * [21] L. R. Medsker and L. Jain, “Recurrent neural networks,” _Design and Applications_ , vol. 5, pp. 64–67, 2001.
  * [22] C. Chen, Y. Liu, S. Kreiss, and A. Alahi, “Crowd-robot interaction: Crowd-aware robot navigation with attention-based deep reinforcement learning,” in _2019 IEEE ICRA_ , 2019, pp. 6015–6022.
  * [23] S. Liu, P. Chang, W. Liang, N. Chakraborty, and K. Driggs-Campbell, “Decentralized structural-rnn for robot crowd navigation with deep reinforcement learning,” in _2021 IEEE ICRA_ , 2021, pp. 3517–3524.
  * [24] Y. F. Chen, M. Everett, M. Liu, and J. P. How, “Socially aware motion planning with deep reinforcement learning,” in _2017 IEEE/RSJ IROS_ , 2017, pp. 1343–1350.
  * [25] W. Wang, R. Wang, L. Mao, and B.-C. Min, “Navistar: Socially aware robot navigation with hybrid spatio-temporal graph transformer and preference learning,” _arXiv preprint arXiv:2304.05979_ , 2023.
  * [26] S. Wang, R. Gao, R. Han, S. Chen, C. Li, and Q. Hao, “Adaptive environment modeling based reinforcement learning for collision avoidance in complex scenes,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 9011–9018.
  * [27] H. He, H. Fu, Q. Wang, S. Zhou, and W. Liu, “Spatio-temporal transformer-based reinforcement learning for robot crowd navigation,” _arXiv preprint arXiv:2305.16612_ , 2023.
  * [28] C. Chen, S. Hu, P. Nikdel, G. Mori, and M. Savva, “Relational graph learning for crowd navigation,” in _IROS_ , 2020, pp. 10 007–10 013.
  * [29] Y. Chen, C. Liu, B. E. Shi, and M. Liu, “Robot navigation in crowds by graph convolutional networks with attention learned from human gaze,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 2754–2761, 2020.
  * [30] Y. Chen, C. Liu, X. Mei, B. E. Shi, and M. Liu, “HGCN-GJS: Hierarchical graph convolutional network with groupwise joint sampling for trajectory prediction,” in _IEEE/RSJ IROS_ , 2022, pp. 13 400–13 405.
  * [31] Z. Zhou, Z. Zeng, L. Lang, W. Yao, H. Lu, Z. Zheng, and Z. Zhou, “Navigating robots in dynamic environment with deep reinforcement learning,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 12, pp. 25 201–25 211, 2022.
  * [32] S. Liu, P. Chang, Z. Huang, N. Chakraborty, W. Liang, J. Geng, and K. Driggs-Campbell, “Socially aware robot crowd navigation with interaction graphs and human trajectory prediction,” _arXiv preprint arXiv:2203.01821_ , 2022.
  * [33] T. Zhang, T. Qiu, Z. Pu, Z. Liu, and J. Yi, “Robot navigation among external autonomous agents through deep reinforcement learning using graph attention network,” _IFAC-PapersOnLine_ , vol. 53, no. 2, pp. 9465–9470, 2020.
  * [34] Z. Zhou, P. Zhu, Z. Zeng, J. Xiao, H. Lu, and Z. Zhou, “Robot navigation in a crowd by integrating deep reinforcement learning and online planning,” _Applied Intelligence_ , pp. 1–17, 2022.
  * [35] T. Zhang, Z. Liu, Z. Pu, J. Yi, Y. Liang, and D. Zhang, “Robot subgoal-guided navigation in dynamic crowded environments with hierarchical deep reinforcement learning,” _International Journal of Control, Automation and Systems_ , pp. 1–13, 2023.
  * [36] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.