# Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative
Adversarial Affective Expression Learning

Uttaran Bhattacharya  University of Maryland
College Park, MD, USA
<EMAIL_ADDRESS>Elizabeth Childs  University of Maryland
College Park, MD, USA
<EMAIL_ADDRESS>Nicholas Rewkowski  University of Maryland
College Park, MD, USA
<EMAIL_ADDRESS>Dinesh Manocha  University of Maryland
College Park, MD, USA
<EMAIL_ADDRESS>
###### Abstract

We present a generative adversarial network to synthesize 3D pose sequences of
co-speech upper-body gestures with appropriate affective expressions. Our
network consists of two components: a generator to synthesize gestures from a
joint embedding space of features encoded from the input speech and the seed
poses, and a discriminator to distinguish between the synthesized pose
sequences and real 3D pose sequences. We leverage the Mel-frequency cepstral
coefficients and the text transcript computed from the input speech in
separate encoders in our generator to learn the desired sentiments and the
associated affective cues. We design an affective encoder using multi-scale
spatial-temporal graph convolutions to transform 3D pose sequences into
latent, pose-based affective features. We use our affective encoder in both
our generator, where it learns affective features from the seed poses to guide
the gesture synthesis, and our discriminator, where it enforces the
synthesized gestures to contain the appropriate affective expressions. We
perform extensive evaluations on two benchmark datasets for gesture synthesis
from the speech, the TED Gesture Dataset and the GENEA Challenge 2020 Dataset.
Compared to the best baselines, we improve the mean absolute joint error by
10–33%, the mean acceleration difference by 8–58%, and the Fréchet Gesture
Distance by 21–34%. We also conduct a user study and observe that compared to
the best current baselines, around 15.28% of participants indicated our
synthesized gestures appear more plausible, and around 16.32% of participants
felt the gestures had more appropriate affective expressions aligned with the
speech.

Figure 1: We synthesize 3D pose sequences of co-speech upper-body gestures
with appropriate affective expressions. We extract the affective cues from the
speech, the sentiments from the corresponding text transcripts, the individual
speaker styles, and the joint-based affective expressions from the seed poses
(shown on the left). We train a generative adversarial network to synthesize
gestures aligned with the speech by leveraging the affective information in
both the generation and the discrimination phases. We show two such affective
gestures on the right, with the affects furious and appalled denoted in
italics.

##  1 Introduction

Co-speech gestures are bodily expressions associated with a person’s speech
[1]. They help underline the subject matter and the context of the speech,
particularly in the form of beat, deictic, iconic, or metaphoric expressions
[2]. Beat gestures are rhythmic movements following the speech, and deictic
gestures point to an entity. Iconic gestures describe physical concepts, e.g.,
spreading and contracting the arms to denote “large” and “small”, and
metaphoric gestures describe abstract concepts, e.g., putting a hand to the
heart to denote “love”. Synthesizing co-speech gestures is an important task
in creating socially engaging characters and virtual agents. These are useful
in a variety of multimedia application such as online learning [3, 4, 5],
interviewing and counseling [6, 7], robot assistants [1], character designs
and game development [8, 9], and visualizing stories and scripts [10].

In our work, we focus on synthesizing the upper-body gestures associated with
speech. We consider the joints at the root, spine, head, and the two arms as
part of the upper body, which are the joints most commonly used in co-speech
gestures [11, 9]. Current state-of-the-art methods for co-speech upper-body
gesture synthesis are based on an end-to-end learning approach [12, 11, 9].
These methods train deep neural networks using gestures (available as videos
or motion-captured datasets), raw speech waveforms and the corresponding text
transcripts, and individual speaker styles. While these methods can generate
different beat, deictic, iconic, and metaphoric co-speech gestures and adapt
to speaker-specific styles, they do not have any mechanism to reliably
incorporate affective expressions in the gestures.

Affective expressions are the modulations in gestures resulting from the
emotions experienced by the speakers [13, 14]. Even for a given speaker, the
style of gesture expressions can change depending on the emotional context,
and human observers are keenly alert to these changes [14]. The combined
understanding of the content of the speech and the speaker’s gesture-based
affective expressions are crucial to human-human interactions [15, 16].
Therefore, it is essential to incorporate affective expressions in co-speech
gestures of animated characters and virtual agents to improve their
plausibility in human-machine interactions.

In human-human interactions, we can break the gesture-based affective
expressions down into a set of biomechanical features known as affective
features, such as body postures, head positions, and arm motions [13]. Each
affective expression is a combination of one or more affective features, e.g.,
rapid arm swings and head jerks are often used as expressions of anger or
excitement [14]. A multitude of macroscopic and microscopic factors influence
the affective features in a given context, including the social setting and
the speaker’s idiosyncrasies, making an exhaustive enumeration of affective
features tedious and challenging [17]. Nevertheless, it is essential to learn
these affective features to understand and synthesize the desired affective
expressions.

Moreover, co-speech affective gesture synthesis also requires aligning the
gestures with the affective cues obtained from the speech. To this end, prior
methods have either learned to map the raw speech waveforms to gestures via
latent embeddings [11] or utilized the log-Mel spectrograms to obtain a richer
understanding of the affective cues, including the prosody and the intonations
in the speech [12, 9]. However, these are high-dimensional representations of
speech that require significant computation overhead to be downscaled into
convenient latent embedding spaces.

Main Contributions. We present an end-to-end learning approach for generating
3D pose sequences of co-speech gestures with appropriate affective expressions
while maintaining the speakers’ individual styles and following a short
sequence of seed poses.111Code and additional materials available at
https://gamma.umd.edu/s2ag. We leverage the Mel-frequency cepstral
coefficients (MFCCs) from the speeches obtained by performing DCT on the log-
Mel spectrograms. MFCCs are highly compressible representations containing
sufficient information for speaker identification and also encode affective
cues such as prosody and intonation for speech-based emotion recognition. We
use separate encoders to encode the MFCCs from the raw speeches, the text
transcripts obtained from the speeches, the speakers’ styles, and the seed
poses. We use available text- and speaker-encoders proposed by Yoon et al.
[11] to learn latent features from the text transcript and a latent style
embedding space using a variational encoding of the speaker styles. We propose
an encoder for the MFCCs that captures the affective cues in the speech. We
also develop an “affective encoder” that transforms the 3D pose sequences to
latent affective features using multi-scale spatial-temporal graph
convolutions (STGCNs). We design our multi-scale STGCNs to expand attention
from the local joints to the macroscopic body parts in a bottom-up manner. We
use our affective encoder both in the generator to learn affective features
from the seed poses to guide the gesture synthesis and in our discriminator to
differentiate between the real and the synthesized gestures based on the
affective expressions. To the best of our knowledge, we are the first to learn
affective features directly from the gesture data to synthesize gestures with
affective expressions. Our main contributions include:

  * •

Synthesizing co-speech affective gestures. We synthesize 3D pose sequences of
gestures with appropriate affective expressions given a speaker’s speech,
maintaining the speakers’ individual styles of gesticulation and following a
short sequence of seed poses.

  * •

Affective encoder for learning latent affective features. Our affective
encoder leverages the localized joint movements and the macroscopic body
movements in the 3D pose sequences to learn latent affective features that are
used for synthesizing the future poses from the seed poses and adversarially
guiding the synthesis as per affective expressions.

  * •

MFCC encoder for leveraging the affective cues from the speech. Our MFCC
encoder takes in low-dimensional MFCCs containing information on the affective
cues from the speech, including prosody and intonations, and transforms them
into latent embeddings for affective gesture synthesis.

We evaluate the quantitative performance of our network on two benchmark
datasets, the TED Gesture Dataset [1] and the GENEA Challenge 2020 Dataset
[19]. We observe an improvement of 10–33% on the mean absolute joint error,
8–58% on the mean acceleration difference, and 21–34% on the Fréchet Gesture
Distance (FGD) [11] for our network compared to the current state-of-the-art
baselines. We also conduct a user study to evaluate the plausibility of our
synthesized gestures and the consistency between the affective expressions in
the gestures and the speech. Around 15.28% participants indicated that our
synthesized gestures are more plausible than the best current baseline of Yoon
et al. [11], and around 16.32% participants felt the gestures had more
appropriate affective expressions aligned with the speech compared to the same
baseline.

##  2 Related Work

We briefly summarize related prior work on how humans perceive affective body
expressions and how these studies were leveraged to synthesize emotionally
expressive characters. We also summarize works on synthesizing body motions,
especially those aligned with a speech, a text transcript, or both.

###  2.1 Perceiving Affective Body Expressions

Affect is traditionally expressed in psychology in terms of its valence,
arousal, and dominance (VAD) [20]. Valence measures the level of pleasantness
(e.g., happy vs. sad), arousal measures how animated the person is (e.g.,
angry vs. bored), and dominance measures the level of control over the affect
(e.g., admiration vs. fear). Studies in both psychology and affective
computing indicate the existence of biomechanical affective features that
provide cues to a person’s perceived affect to human observers [13, 14, 21,
22, 23]. These affective features can be observed at different scales: they
can be localized joint movements such as rapid arm swings and head jerks,
indicating excitement or anger, as well as macroscopic body movements such as
the upper body being expanded, indicating pride or confidence, or collapsed,
indicating shame or nervousness. Subsequently, there has been work on
detecting perceived emotions by leveraging known affective features either as
input to a neural network [24] or to constrain the embedding space [17]. In
contrast, we design our neural network to explicitly attend to the body
movements at these multiple scales to learn latent affective features directly
from the input gesture samples.

###  2.2 Synthesizing Affective Body Expressions

There has been substantial work on synthesizing affective expressions for
embodied conversation agents [25, 26] and other social virtual agents to
interact via facial expressions [27, 28] or gaits [29]. Furthermore, the
synthesis of affective facial expressions has been aligned with a character’s
speech using data-driven techniques [30]. While synthesizing speech-aligned
affective facial expressions has been relatively well-studied, aligning the
speech with affective body expressions has been more challenging. Some of the
widely used approaches are rule-based systems such as that of DeVault et al.
[7], which has a virtual human counselor expressing appropriate affective hand
and body gestures following known mappings between the emotional states and
the stored animations. Recent methods utilize gait datasets annotated with
categorical emotions such as happy, sad, and angry to generate emotive gaits
[31, 32]. Other techniques have extended to the VAD space of affect, where
body gestures are generated given the text transcripts of speech and the
corresponding intended emotion as a point in the VAD space [33]. Our approach
is based on designing an end-to-end system that can synthesize body
expressions by automatically understanding the affective content in the input
speech.

###  2.3 Synthesizing Gestures

There is a rich body of work gesture synthesis using rule-based systems, as
surveyed comprehensively by Wagner et al. [34]. However, scalability to novel
scenarios remains a challenge for rule-based systems on account of manually
designing new rules. Instead, we focus on a summary of the recent data-driven
approaches of automated gesture synthesis in novel scenarios [35], which are
in line with our learning-based approach. Existing techniques have utilized
hidden Markov models [36], recurrent neural network variants [37, 1], and
autoencoders [38] to learn robust latent features that encode the input
speech, available as either an audio or a text transcript, and can be used to
decode the output gestures. Other approaches have opted to learn stochastic
generation processes using tools such as invertible sub-transformations [39]
to map between the speech and the gesture spaces. To improve the realism of
the generated speech-driven gestures, more recent works incorporate the speech
semantics into the training process [9], and even combined the synthesized
gestures with rule-based head nods and hand waves for embodied conversation
agents [40].

Our approach is complementary to these approaches in that we learn mappings
from the text transcripts of speech to gestures. It eliminates the noise in
speech signals and helps us focus only on the relevant content and context.
Learning from the text also enables us to focus on a broader range of
gestures, including iconic, deictic, and metaphoric gestures [2]. Our work is
most closely related to that of Yoon et al. [1]. They learn upper body
gestures as PCA-based, low-dimensional pose features, corresponding to text
transcripts from a dataset of TED-talk videos, then map these 3D gestures to
an NAO robot. They have also followed up this work by generating upper-body
gestures aligned with the three modalities of speech, text transcripts, and
person identity [11]. On the other hand, we learn to map text transcripts to
3D pose sequences corresponding to semantic-aware, full-body gestures of more
human-like virtual agents using an end-to-end trainable transformer network
and blend in emotional expressiveness.

###  2.4 Incorporating Speaker Styles

Co-speech gesture generation is intrinsically related to stylized gesture
generation. There has been considerable progress on stylized generation of
head motions [41, 42], facial motions [43, 28] as well as locomotions [44, 45,
46]. At the same time, many techniques have been proposed to generate
appropriately styled body motions from textual descriptions of the actions
[47, 48]. Other approaches have developed separate gesture generation networks
for individual speakers to adapt to their individual styles [12], together
with adversarial losses to improve the fidelity of the generation [49].
Recently, Yoon et al. [11] proposed a unified architecture that considers the
speech, its text transcript, and the speaker identity to generate co-speech
gestures with continuously varying speaker styles. We extend such speaker-
aware gesture synthesis to further incorporate the appropriate affective body
expressions that align with the affective content in the speech.

Figure 2: Our network consists of a generator (pale-green box) and a
discriminator (pale-blue box). Our generator takes in the MFCC from the
speech, the text transcript, the speaker ID, and a sequence of 3D seed poses.
We use four encoders: the MFCC encoder (Sec. 3.1.1), the text encoder (Sec.
3.1.2), the speaker encoder (Sec. 3.1.3), and the affective encoder (Sec.
3.1.4). We feed the concatenation of these latent features into our Bi-GRU
followed by a set of FC layers to synthesize the gestures aligned with the
speech. Our discriminator learns to discriminate between the real and the
synthesized gestures based on the latent affective features from the affective
encoder, constraining the generator to synthesize appropriate affective
expressions.

##  3 Approach

Our goal is to generate 3D pose sequences of co-speech upper-body gestures
with appropriate affective expressions and speaker styles, given the raw
speech waveform, the speaker identity, and a short sequence of seed poses. We
consider affective expressions to be specific sequences of joint movements,
generally as a combination of the affective features [33]. We learn these
affective expressions both at the localized joint neighborhoods and the
macroscopic body movements, and use them to condition the training of a
generative adversarial network. We show our overall network architecture in
Fig. 2.

###  3.1 Synthesizing Co-Speech Gestures

Our generative network takes in the raw speech waveform as a 1D array, the
corresponding text transcript as a sequence of words, the speaker identity as
a unique number, and the seed poses as a 3D pose sequence. Similar to Yoon et
al. [11], we encode the speech waveform, the text transcript, and the speaker
identities using separate encoders. However, unlike Yoon et al., we convert
the speech waveform to Mel-Frequency Cepstral Coefficients (MFCCs) to guide
the encoding process based on the affective cues from speech. We also propose
an affective encoder to encode the pose-based affective expressions into
latent features for both gesture generation and discrimination. In the
generation process, we combine the latent embeddings learned from the four
encoders, speech, text, speaker, and affective, into a joint embedding for
learning the upper-body gestures.

####  3.1.1 MFCC Encoder

MFCCs are known to encode signal frequencies consistent with how humans
perceive sound, and are therefore particularly useful for tasks such as speech
recognition [50], speaker identification [51] and speech-based emotion
recognition [52]. In our case, we design our MFCC encoder to embed the speech-
based affective cues such as prosody and intonations captured by the MFCCs and
incorporate them in gesture synthesis. Given a raw waveform as a 1D array, we
transform it to its top 14 MFCCs. These include the log-energy spectrum and 13
coefficients containing sufficient information on the speaker’s pitch,
intonation, prosody, and other relevant parameters [53, 43]. We also append
the first- and the second-order discrete forward differences of the 13
coefficients, obtaining a total of 37 values. Using a window size $W$ on a
input waveform of length $L$, we obtain individual MFCCs of shape $\left\lceil
L/W\right\rceil$, leading to a combined feature tensor
$f_{m}\in\mathbb{R}^{37\times\left\lceil L/W\right\rceil}$. We pass these
features through a series of 1D temporal convolutions, followed by a single
fully-connected (FC) layer, to obtain a latent feature sequence
$\hat{f}_{m}\in\mathbb{R}^{D_{m}\times T}$ of sequence length $T$ equal to 3D
pose sequence length of the seed poses, as

$\hat{f}_{m}=\textrm{Conv}\circ\textrm{FC}_{mfcc}\left(f_{m};W_{mfcc}\right),$
(1)

where $D_{m}$ is the dimension of the latent features,
$\textrm{Conv}\circ\textrm{FC}_{mfcc}$ denotes the series of 1D convolutions
followed by the FC layer, and $W_{mfcc}$ its set of trainable parameters.

####  3.1.2 Text Encoder

Given the text transcript corresponding to the speech, we first pad the
transcript with padding tokens following the approach of Yoon et al., to
ensure that the text transcript has the same sequence length $T$ as the seed
poses. We then use the pre-trained FastText [54] word embedding model to
transform the word sequence into 300-dimensional features, leading to a
feature tensor $f_{x}\in\mathbb{R}^{300\times T}$. We use FastText for its
memory efficiency and its usefulness in sentiment analysis [55], which is
important in understanding text-based affect. We pass the FastText features
through a series of temporal 1D convolutions to obtain a latent feature
sequence $\hat{f}_{x}\in\mathbb{R}^{D_{x}\times T}$, as

$\hat{f}_{x}=\textrm{Conv}_{text}\left(f_{x};W_{text}\right),$ (2)

where $D_{x}$ is the dimension of the latent features, $\textrm{Conv}_{text}$
denotes the series of 1D convolutions with trainable parameters $W_{text}$.

####  3.1.3 Speaker Encoder

For the speaker IDs, we use one-hot vectors
$f_{s}\in\left\\{0,1\right\\}^{\mathcal{S}}$, assuming $\mathcal{S}$ is the
number of available speakers. Following Yoon et al. [11], we use two sets of
FC layers to learn an embedding space capturing the mean
$\mu_{s}\in\mathbb{R}^{D_{s}}$ and the variance
$\Sigma_{s}\in\mathbb{R}^{D_{s}\times D_{s}}_{+}$ of the latent distribution
of the speaker styles as

$\displaystyle\mu_{s}$
$\displaystyle=\textrm{FC}_{\mu}\left(f_{s};w_{\mu}\right)$ (3)
$\displaystyle\log\Sigma_{s}$
$\displaystyle=\textrm{FC}_{\Sigma}\left(f_{s};w_{\Sigma}\right),$ (4)

where $D_{s}$ is the dimension of the latent distribution space,
$\textrm{FC}_{\mu}$ and $\textrm{FC}_{\Sigma}$ denote the two sets of FC
layers, and $W_{\mu}$ and $W_{\Sigma}$ denote the corresponding sets of
trainable parameters. Intuitively, this latent distribution space consists of
all the available speakers, plus speakers that can be “constructed” by linear
combinations of those speakers in the latent space. As a result, we can pick a
random point from the latent space to use in the synthesis, resulting in some
variability in the synthesized gestures even when the speech remains the same.
We term this variability as having “speaker-aware” styles. Given the
parameters $\mu_{s}$ and $\Sigma_{s}$ of latent distribution space, we use the
re-parametrization trick [56] to generate a random speaker-aware style sample
$\hat{f}_{s}\in\mathbb{R}^{D_{s}}$ and repeat it for all the $T$ time steps of
the input pose sequence.

####  3.1.4 Affective Encoder

We propose an encoding mechanism that transforms the pose-based affective
expressions into a latent embedding. Since gestures typically consist of
movements in the trunk, arms, and head, we only consider ten joints
corresponding to these parts of the body: root, spine, neck, head, left and
right shoulders, left and right elbows, and left and right wrists. We consider
a directed graph for the pose, where the joints are the vertices, and the
edges are directed from the root towards the extremities. We assume the edge
lengths are known for each input and train our encoder only on the directions
of the edges. We consider nine unit-vector sequences
$U=\left[u_{1};\dots;u_{9}\right]$, each of sequence length $T$, to denote the
edge directions at the corresponding $T$ time steps of the input pose
sequence.

We employ a hierarchical encoding strategy using spatial-temporal graph
convolutions (STGCNs) [57]. STGCNs are adapted to leverage localized
dependencies in generalized graph-structured data, and are therefore suitable
for our pose graph sequences. We use two levels of hierarchy, the first at the
level of individual bones and the second at the level of the three body parts,
the trunk and the two arms. At the first level, our unweighted adjacency
matrix $A_{1}\in\left\\{0,1\right\\}^{9\times 9\times T}$ captures the
temporal counterparts of each edge at the four nearest time steps (past two
and future two), and spatially adjacent edges with a maximum hop of two, i.e.,
we consider two edges to be spatially adjacent if they either share a vertex
or are connected to the two ends of a third edge. This size of the adjacent
neighborhood sufficiently groups the edges influenced by typical affective
expressions such as arm swings, head jerks, and upper-body collapse.
Consequently, the convolution filters can learn a latent feature sequence
$\hat{f}_{a_{1}}\in\mathbb{R}^{D_{a_{1}}\times 9\times T}$ from the edges
based on the variations in the affective expressions, obtained as

$\hat{f}_{a_{1}}=\textrm{STGCN}_{1}\left(U,A_{1};W_{a_{1}}\right),$ (5)

where $D_{a_{1}}$ is the dimension of the per-edge latent features,
$\textrm{STGCN}_{1}$ denotes the first-level STGCN with trainable parameters
$W_{a_{1}}$. At the second level, the three body parts, the trunk and the two
arms, capture the macroscopic body movements such as raising or crossing the
arms, and bending or straightening the trunk. In the second-level adjacency
matrix $A_{2}\in\left\\{0,1\right\\}^{3\times 3\times T}$, we assume both the
arms to be adjacent to the torso but not to each other, since the movements on
one arm need not influence the other. We again consider the temporal
counterparts of each body part in the four nearest time steps in the temporal
adjacency. We reshape the latent features $\hat{f}_{a_{1}}$ to
$3D_{a_{1}}\times 3\times T$, to collect the per-edge features corresponding
to the three body parts in the feature dimension. Our second-level STGCN then
operates on these reshaped features to produce the second-level latent
features $\hat{f}_{a_{2}}\in\mathbb{R}^{D_{a_{2}}\times 3\times T}$ as

$\hat{f}_{a_{2}}=\textrm{STGCN}_{2}\left(\hat{f}_{a_{1}},A_{2};W_{a_{2}}\right),$
(6)

where $D_{a_{2}}$ is the dimension of the per-edge latent features,
$\textrm{STGCN}_{2}$ denotes the second-level STGCN with trainable parameters
$W_{a_{2}}$. We then apply a series of 1D convolutions on the reshaped second-
level features $\hat{f}_{a_{2}}\in\mathbb{R}^{3D_{a_{2}}\times T}$ to obtain
the latent affective feature sequence $\hat{f}_{a}\in\mathbb{R}^{D_{a}\times
T}$, as

$\hat{f}_{a}=\textrm{Conv}_{aff}\left(\hat{f}_{a_{2}};W_{a}\right),$ (7)

where $D_{a}$ is the dimension of the latent affective features, and
$\textrm{Conv}_{aff}$ denotes the series of 1D convolutions with trainable
parameters $W_{a}$.

####  3.1.5 Gesture Generator

Given the latent feature sequences $\hat{f}_{m}$, $\hat{f}_{x}$,
$\hat{f}_{s}$, and $\hat{f}_{a}$, we concatenate them, pass them through a
bidirectional gated recurrent unit (Bi-GRU), and sum the bidirectional outputs
to obtain the predicted edge embeddings sequence
$\hat{u}_{e}\in\mathbb{R}^{D_{e}\times T}$, as

$\displaystyle out_{frw},out_{bkw}$
$\displaystyle=\textrm{GRU}_{e}\left(\left[\hat{f}_{m};\hat{f}_{x};\hat{f}_{s};\hat{f}_{a}\right];W_{e}\right),$
(8) $\displaystyle\hat{u}_{e}$ $\displaystyle=out_{frw}+out_{bkw},$ (9)

where $D_{e}$ is the dimension of the predicted edge embeddings,
$\textrm{GRU}_{e}$ denotes the bidirectional GRU with the corresponding set of
trainable parameters $W_{e}$, and $out_{frw}$ and $out_{bkw}$ respectively
denote the outputs of the forward and the backward channels of the GRU. As in
Yoon et al., we then transform the predicted edge embeddings to predicted edge
vector sequences $\hat{U}=\left[\hat{u}_{1};\dots;\hat{u}_{9}\right]$, each of
sequence length $T$, using a set of FC layers as

$\hat{U}=\textrm{FC}_{gen}\left(\hat{u}_{e};W_{gen}\right),$ (10)

where $\textrm{FC}_{gen}$ denotes the set of FC layers with the trainable
parameters $W_{gen}$. Thus, our generator is designed to take in a sequence of
seed poses of length $T$ and predicts a pose sequence of gestures for the next
$T$ time steps. Finally, we scale each predicted edge vector $\hat{u}_{i}$ to
have the corresponding bone length $b_{i}$, $i=1,\dots,9$. We add it to the 3D
position $pos_{s\left(i\right)}$ of the source joint $s\left(i\right)$ of that
edge vector to obtain 3D position $pos_{d\left(i\right)}$ of the destination
joint $d\left(i\right)$ of the same edge vector, as

$pos_{d\left(i\right)}=pos_{s\left(i\right)}+b_{i}\cdot\frac{\hat{u}_{i}}{\left\|\hat{u}_{i}\right\|}.$
(11)

###  3.2 Discriminating Gestures

Our discriminator takes in a gesture of sequence length $T$ and computes its
latent affective feature sequence $\hat{f}_{a}\in\mathbb{R}^{D_{a}\times T}$
using our affective encoder (Sec. 3.1.4). We pass this feature sequence
through another bidirectional GRU, and sum the bidirectional outputs to obtain
the discriminator embeddings sequence $\hat{d}\in\mathbb{R}^{h\times T}$, as

$\displaystyle out_{frw},out_{bkw}$
$\displaystyle=\textrm{GRU}_{disc}\left(\hat{f}_{a};W_{GRU,disc}\right),$ (12)
$\displaystyle\hat{d}$ $\displaystyle=out_{frw}+out_{bkw},$ (13)

where $D_{d}$ is the dimension of the predicted discriminator embeddings,
$\textrm{GRU}_{disc}$ denotes the bidirectional GRU with trainable parameters
$W_{GRU_{d}isc}$, and $out_{frw}$ and $out_{bkw}$ respectively denote the
outputs of the forward and the backward channels of the GRU. We then transform
the discriminator embeddings to a probability vector $c\in\left[0,1\right]$
using a set of FC layers as

$c=\textrm{FC}_{disc}\left(\hat{d};W_{FCdisc}\right),$ (14)

where $\textrm{FC}_{disc}$ denotes the set of FC layers with trainable
parameters $W_{FCdisc}$, and $c$ is such that $c\geq 0.5$ implies the
discriminator predicts the input gesture to be real, and generated otherwise.

TABLE I: Hyperparameters (HPs) for our network. We chose all the values via
empirical search.

HP | Description | Value
---|---|---
$D_{m}$ | Latent feature from the MFCC encoder | 32
$D_{x}$ | Latent feature from the text encoder | 32
$D_{s}$ | Latent distribution space of speaker styles | 16
$D_{a_{1}}$ | Per-edge latent features after $\textrm{STGCN}_{1}$ in the affective encoder | 16
$D_{a_{2}}$ | Per-edge latent features after $\textrm{STGCN}_{2}$ in the affective encoder | 16
$D_{a}$ | Latent affective features from the affective encoder | 16
$D_{e}$ | Predicted edge embeddings from the GRU in the generator | 150
$D_{d}$ | Predicted embeddings from the GRU in the discriminator | 150

##  4 Dataset and Training

We train our network on the TED Gesture Dataset [1], which consists of videos
of English-language speakers at TED Talks. It provides 3D pose sequences of
the upper-body gestures of the speakers, their speech audio, and the
associated text transcripts. Each data sample has a sequence length of $T=34$
time steps at a rate of 15 fps. There are 200,038 training samples in total,
constituting around 80% of the dataset. The evaluation set consists of 26,903
samples or around 10% of the dataset. The test set consists of 26,245 samples,
making up the remaining 10% of the dataset.

We use loss functions $\mathcal{L}_{G}$ and $\mathcal{L}_{D}$ identical to
Yoon et al. [11] to train our generator and discriminator respectively:

$\displaystyle\mathcal{L}_{G}$
$\displaystyle=\lambda_{Hub}\mathcal{L}_{Hub}+\lambda_{gen}\mathcal{L}_{gen}+\lambda_{stl}\mathcal{L}_{stl}+\lambda_{KLD}\mathcal{L}_{KLD},$
(15) $\displaystyle\mathcal{L}_{D}$
$\displaystyle=-\mathbb{E}\left[\log\left(\textrm{Disc}\left(U\right)\right)\right]-\mathbb{E}\left[\log\left(1-\textrm{Disc}\left(\hat{U}\right)\right)\right],$
(16)

where Disc denotes the discriminator network (Sec. 3.2), $\lambda_{*}$ are the
weights of the corresponding loss terms with the same values as in Yoon et al.
[11], and the individual loss terms of the generator are:

  * •

Huber loss [58] between the ground truth and predicted edge vectors,

  * •

generative adversarial loss on the output of the discriminator,

$\mathcal{L}_{gen}=-\mathbb{E}\left[\log\left(\textrm{Disc}\left(\hat{U}\right)\right)\right],$
(17)

  * •

diversity regularization between the synthesized gestures and other gestures
in the dataset to ensure that the styles of different speakers appear visually
different,

  * •

Kullback-Leibler (KL) divergence between the latent distribution space of the
styles defined by $\mu_{s}$ and $\Sigma_{s}$, and the normal distribution
$\mathcal{N}\left(0,I\right)$.

Table I lists the latent dimensions we use for training our network. We use
the Adam optimizer [59] with $\beta_{1}=0.5$, $\beta_{2}=0.999$, batch size of
$512$, and learning rate of $5\textsc{e}^{-4}$ for the generator and
$1\textsc{e}^{-4}$ for the discriminator with no warm-up epochs (i.e.,
$\lambda_{gen}>0$ starting from the first epoch). We train our network for
$300$ epochs, which took close to $45$ hours on an NVIDIA GeForce GTX 1080 Ti
GPU.

TABLE II: Evaluation of our method with baselines and ablated versions of our
method on two benchmark dataset, using the objective metrics of mean absolute
joint error (MAJE), mean acceleration difference (MAD), and the Fréchet
Gesture Distance (FGD). Bold indicates best.

Dataset | Method | MAJE (mm) | MAD (mm/s2) | FGD
---|---|---|---|---
TED Gesture [1] | Seq2Seq [1] | 45.62 | 6.33 | 6.62
S2G-IS [12] | 45.11 | 7.22 | 6.73
JEM [60] | 48.56 | 4.31 | 5.88
GTC [11] | 27.30 | 3.20 | 4.49
Ours w/o MFCC Enc. | 27.84 | 3.02 | 4.21
| Ours w/o Aff. Enc. | 25.38 | 3.51 | 4.84
| Ours | 24.49 | 2.93 | 3.54
GENEA Challenge 2020 [19] | Gesticulator [9] | 82.41 | 3.62 | 31.04
Ours w/o MFCC Enc. | 105.71 | 1.57 | 23.03
| Ours w/o Aff. Enc. | 92.90 | 2.81 | 24.28
| Ours | 54.93 | 1.49 | 20.36

Figure 3: Qualitative results on the gestures synthesized by our method for
two sample speech excerpts from the TED Gesture Dataset [1]. The italicized
words very excited and bored indicate the primary affect in the corresponding
speeches. We compare with the corresponding gestures of the original speakers,
the output of GTC [11], and that of the two ablated versions of our network
(Sec. 5.3). See Sec. 5.4 for a detailed discussion of the results.

##  5 Experiments

We describe the objective evaluation of our method compared to current
baseline methods. We highlight the benefit of our proposed components via
ablation studies on the objective evaluation metrics. We also show the
qualitative performance of our method on selected samples from the TED Gesture
Dataset [1] and the perceived quality of our synthesized gestures through a
user study.

###  5.1 Baseline Methods

We compare our method with the baseline methods on two benchmark datasets, the
TED Gesture Dataset [1], and the GENEA Challenge 2020 Dataset [18, 19].

On the TED Gesture Dataset, we compare with the methods of Seq2Seq [1], Speech
to Gestures with Individual Styles (S2G-IS) [12], Joint Embedding Model (JEM)
[60], and Gestures from Trimodal Context (GTC) [11]. Seq2Seq and JEM generate
gestures based only on the text transcript of the speech, whereas S2G-IS uses
only the speech to generate the gestures. GTC uses the speech, the
corresponding text transcript, and the speaker styles to generate gestures.
Seq2Seq follows an encoder-decoder architecture, where the authors transform
the text to latent features and predict gestures based on both the latent
features and a short gesture history. The authors of S2G-IS employ a
generative adversarial network that generates gestures from a latent space
obtained from the input log-Mel spectrograms. JEM maps both the text and the
target gesture into a common latent embedding space and uses a decoder to
reconstruct the gestures from the embedding space. The authors train the model
to learn to align the text-based and the gesture-based embeddings for the same
input and decode gestures from only the text-based embeddings. For Seq2Seq,
S2G-IS, and JEM, we follow the training routine and the hyperparameters used
by Yoon et al. [11]. For GTC, we directly use the pre-trained model provided
by Yoon et al. [11].

The GENEA Challenge 2020 Dataset is the publicly available version of the
Trinity Gesture Dataset [18, 19]. It consists of the speech and the full-body
motion capture of a male actor talking unrestrained about various topics over
multiple recording sessions. The full dataset is about 242 minutes long, of
which 221 minutes are used as training data, and the remaining 21 minutes are
kept for testing. We do not fine-tune our network on this dataset and evaluate
our network on the test partition. Since we consider only upper-body gestures,
we consider the ten relevant upper-body joints at the root, the spine, the
head, and the two arms for evaluating our performance. On this dataset, we
compare with the method of Gesticulator [9], which leverages the acoustics and
the semantics of the speech to generate semantically consistent beat, deictic,
metaphoric, and iconic gestures. For a fair comparison, we use the pre-trained
model provided by the authors and compare the performance on the same ten
joints that we use for our method.

###  5.2 Objective Evaluation

While evaluation metrics for gesture synthesis are not standardized, we
evaluate on the commonly used metrics of mean absolute joint error (MAJE),
mean acceleration difference (MAD), and the Fréchet Gesture Distance (FGD)
proposed by Yoon et al. [11]. MAJE measures the mean of the absolute
differences between the ground truth and the predicted joint positions over
all the time steps, joints, and samples. MAD measures the mean $\ell_{2}$-norm
error between the ground truth and predicted joint accelerations over all the
time steps, joints, and samples. FGD measures the difference between the
distributions of the latent features of the ground truth and the predicted
gestures. The latent features are computed from an autoencoder network trained
on the well-known Human 3.6M dataset [61] of human motions using the ten
joints in the TED Gesture Dataset [1]. MAJE indicates how closely the
predicted joint positions follow the ground truth joint positions. MAD
indicates how closely the ground truth and predicted joint movements match.
Since affective expressions are based on joint movements, a lower MAD is
especially desirable for our stated aim of generating gestures with
appropriate affective expressions. FGD is shown to align well with the
perceived plausibility of the synthesized gestures to human users [11];
therefore, a lower FGD is equally desirable to gauge the quality of our
synthesized gestures.

Table II summarizes the performance of all the methods on all these evaluation
metrics. Our method consistently has the lowest MAJE, MAD, and FGD on both the
benchmark datasets. On the TED Gesture Dataset, we observe improvements of
10.29%, 8.44%, and 21.16% on MAJE, MAD, and FGD, respectively, over the best
current baseline of GTC. On the GENEA Challenge 2020 Dataset, we observe
improvements of 33.34%, 58.84%, and 34.41% on MAJE, MAD, and FGD,
respectively, over the baseline of Gesticulator. We note that the absolute FGD
values are significantly higher on the GENEA Challenge 2020 Dataset than on
the TED Gesture Dataset. We hypothesize that this is because the gestures in
the GENEA Challenge 2020 Dataset are more abstract and unscripted compared to
the well-defined actions in the Human 3.6M Dataset or polished speeches in the
TED Gesture Dataset. As a result, the pre-trained latent embeddings used for
FGD are not as good at reconstructing the joints movements in the GENEA
Challenge 2020 Dataset.

###  5.3 Ablation Studies

We perform ablation studies on our two proposed components: the MFCC encoder
(Sec. 3.1.1) and the affective encoder (Sec. 3.1.4). In one study, we replace
only our MFCC encoder with an encoder for the raw audio waveform, identical to
that of GTC [11], and train the resultant network. In the other study, we
remove the affective encoder from our network. Our generator takes in the raw
seed poses instead of the latent affective features. Our discriminator uses a
convolution filter identical to GTC [11] to transform the input gestures to
latent features for the bidirectional GRU.

Without the MFCC encoder, our generator cannot take the speech-based affective
cues into account. It results in a degradation of the synthesis, leading to
higher MAJE, MAD, and FGD. However, without the affective encoder, our network
is severely limited in understanding both the affective expressions in the
seed poses and the affective expressions of the synthesized poses. It results
in more severe performance degradation on all the evaluation metrics, and the
synthesized gestures appear less diverse and plausible.

(a) Plausibility of the different types of gestures.

(b) Synchronization of the movements and the affective expressions of the
different types of gestures with the speech.

Figure 4: Mean fraction of participant responses on each point of the Likert
scales across the 12 speech excerpts from the TED Gesture Dataset [1] and the
corresponding gestures in our user study. See Sec. 5.5 for details.

###  5.4 Qualitative Results

We show qualitative results on two sample speech excerpts from the TED Gesture
Dataset [1] in Fig. 3. It has five rows of gestures: those of the original
speakers’, those synthesized by GTC [11] ( the current state-of-the-art),
those by the two ablated versions of our network: one without our MFCC encoder
(Sec. 3.1.1) and the other without our affective encoder (Sec. 3.1.4), and
those by our proposed network with all the encoders. We observe a diversity of
speaker styles in the synthesized gestures compared to the original speaker,
which results from using a variational embedding of speaker styles using the
speaker encoder (Sec. 3.1.3). GTC, however, cannot generate affective
expressions except for a few words with strong intonations in the speech, such
as “excited” (second row, left column). Without our MFCC encoder, our network
can still match the speech content but cannot align the gestures with the
affective cues from the speech. For example, it can match the words “I was, I
believe” with a deictic gesture pointing to the speaker himself (third row,
right column) but cannot generate any expressions for “bored”. Without our
affective encoder, we observe only slight body movements but no appreciable
affective expressions in the synthesized gestures. With all our encoders in
place, we observe appropriate affective expressions that align well with the
speech. For example, we observe rapid arm movements when saying “excited”
(fifth row, left column) and dropping of the arms and shoulders when saying
“bored” (fifth row, right column).

###  5.5 User Study

We conducted a user study to evaluate the perceptual quality of our
synthesized gestures in terms of how plausible they appear and how well-
aligned are their affective expressions with the corresponding speeches. 24
participants took part in our study, of which 20 were male, and 4 were female.
10 participants were between 18 and 24 years of age, 13 were between 25 and
34, and one was above 35. Each participant observed gestures corresponding to
the same 12 speech excerpts, each taken from a different TED Talk in the TED
Gesture Dataset [1]. For each speech excerpt, the participants observed three
different types of gestures: that of the original speaker as a 3D pose
sequence (provided in the dataset), those synthesized by GTC [11], the current
state-of-the-art, and those synthesized by our network. The order of the
gestures was unknown to and randomized for each participant. We then asked the
participants to answer two questions. The first question was how plausible the
gestures appeared on a five-point Likert scale ranging from “very unnatural”
(1) to “look like they could be from a real person” (5). The second question
was how well the gestures synchronized with the corresponding speeches on a
five-point Likert scale, ranging from “no or arbitrary gestures” (1) to “well-
synchronized with the speech, and are appropriately emotionally expressive”
(5). Intuitively, our Likert-scale points for both questions reflect the
participants’ individual assessments of quality, with 1 being the worst, 3
being average, and 5 being the best. The entire study took around 20 minutes
on average for each participant.

We summarize the participants’ responses in Fig. 4. When adjudging the
plausibility of the gestures (Fig. 4(a)), we observe that 15.28% more
participants marked our synthesized gestures either 4 or 5 compared to the
gestures synthesized by GTC [11]. Further, 3.82% more participants marked our
synthesized gestures 4 or 5 than the original speakers’ gestures, indicating
that the participants found our synthesized gestures to have visual quality
comparable to that of the original data. When adjudging the synchronization of
the movements and the affective expressions of different types of gestures
with speech (Fig. 4(b)), we observed that 16.32% more participants marked our
synchronization quality either 4 or 5 compared to that of GTC [11]. Also,
4.86% more participants marked out synchronization quality 4 or 5 than that of
the original speakers, indicating that the participants perceived our
synthesized gestures to be as well-synchronized and expressive as the original
data.

##  6 Conclusion, Limitations and Future Work

We have presented an end-to-end learning approach to generate 3D pose
sequences of co-speech gestures with appropriate affective expressions. Our
contributions include an MFCC encoder to guide the gesture synthesis based on
the speech-based affective cues such as prosody and intonation, and an
affective encoder to learn joint-based affective features from the gesture
data. Using these encoders in a generative adversarial learning framework, we
have synthesized affective gestures that advance the state-of-the-art on co-
speech gesture synthesis on multiple evaluation metrics. Our synthesized
gestures also appeared more plausible and well-synced with the corresponding
speeches to participants in a user study.

Our work has some limitations. First, we do not build a mechanism to control
the affective expressions; we compute them automatically from the input
modalities. We plan to investigate the interface between affective expressions
from the speech and the gestures, especially when expressing contradictory
cues such as sarcasm and irony. Second, we plan to use a finer representation
of poses in the future since affective expressions in the gestures are often
associated with subtle movements not captured by our current representation.
Lastly, our network uses only upper-body gestures with a fixed root. We plan
to expand to affect-aware gestures of the whole body, incorporating locomotion
in the global coordinate space. We also plan to align the body gestures with
the corresponding facial expressions, leading to fully emotive characters.

## Acknowledgment

This work has been supported in part by ARO Grants W911NF1910069 and
W911NF1910315, and Intel.

## References

  * [1] Y. Yoon, W.-R. Ko, M. Jang, J. Lee, J. Kim, and G. Lee, “Robots learn social skills: End-to-end learning of co-speech gesture generation for humanoid robots,” in _Proc. of The International Conference in Robotics and Automation (ICRA)_ , 2019. [Online]. Available: https://github.com/youngwoo-yoon/Co-Speech\\_Gesture\\_Generation
  * [2] D. McNeill, _Hand and mind: What gestures reveal about thought_. University of Chicago press, 1992.
  * [3] J. Li, R. Kizilcec, J. Bailenson, and W. Ju, “Social robots and virtual agents as lecturers for video instruction,” _Computers in Human Behavior_ , vol. 55, pp. 1222 – 1230, 2016. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0747563215002885
  * [4] M. Liao, C. Sung, H. Wang, and W. Lin, “Virtual classmates: Embodying historical learners’ messages as learning companions in a vr classroom through comment mapping,” in _2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)_ , 2019, pp. 163–171.
  * [5] A. L. Simeone, M. Speicher, A. Molnar, A. Wilde, and F. Daiber, “Live: The human role in learning in immersive virtual environments,” in _Symposium on Spatial User Interaction_ , ser. SUI ’19. New York, NY, USA: Association for Computing Machinery, 2019. [Online]. Available: https://doi.org/10.1145/3357251.3357590
  * [6] T. Baur, I. Damian, P. Gebhard, K. Porayska-Pomsta, and E. André, “A job interview simulation: Social cue-based interaction with a virtual character,” in _2013 International Conference on Social Computing_ , 2013, pp. 220–227.
  * [7] D. DeVault, R. Artstein, G. Benn, T. Dey, E. Fast, A. Gainer, K. Georgila, J. Gratch, A. Hartholt, M. Lhommet _et al._ , “Simsensei kiosk: A virtual human interviewer for healthcare decision support,” in _Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems_ , 2014, pp. 1061–1068.
  * [8] S. Mascarenhas, M. Guimarães, R. Prada, J. Dias, P. A. Santos, K. Star, B. Hirsh, E. Spice, and R. Kommeren, “A virtual agent toolkit for serious games developers,” in _2018 IEEE Conference on Computational Intelligence and Games (CIG)_ , 2018, pp. 1–7.
  * [9] T. Kucherenko, P. Jonell, S. van Waveren, G. E. Henter, S. Alexandersson, I. Leite, and H. Kjellström, “Gesticulator: A framework for semantically-aware speech-driven gesture generation,” ser. ICMI ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 242–250. [Online]. Available: https://doi.org/10.1145/3382507.3418815
  * [10] K. Watson, S. S. Sohn, S. Schriber, M. Gross, C. M. Muniz, and M. Kapadia, “Storyprint: An interactive visualization of stories,” in _Proceedings of the 24th International Conference on Intelligent User Interfaces_ , ser. IUI ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 303–311. [Online]. Available: https://doi.org/10.1145/3301275.3302302
  * [11] Y. Yoon, B. Cha, J.-H. Lee, M. Jang, J. Lee, J. Kim, and G. Lee, “Speech gesture generation from the trimodal context of text, audio, and speaker identity,” _ACM Transactions on Graphics_ , vol. 39, no. 6, 2020.
  * [12] S. Ginosar, A. Bar, G. Kohavi, C. Chan, A. Owens, and J. Malik, “Learning individual styles of conversational gesture,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019\.
  * [13] A. Kleinsmith and N. Bianchi-Berthouze, “Affective body expression perception and recognition: A survey,” _IEEE Transactions on Affective Computing_ , vol. 4, no. 1, pp. 15–33, 2013.
  * [14] M. Karg, A. Samadani, R. Gorbet, K. Kühnlenz, J. Hoey, and D. Kulić, “Body movements for affective expression: A survey of automatic recognition and generation,” _IEEE Transactions on Affective Computing_ , vol. 4, no. 4, pp. 341–359, 2013.
  * [15] D. Matsumoto, M. G. Frank, and H. S. Hwang, _Nonverbal communication: Science and applications_. Sage Publications, 2012.
  * [16] M. L. Knapp, J. A. Hall, and T. G. Horgan, _Nonverbal communication in human interaction_. Cengage Learning, 2013\.
  * [17] U. Bhattacharya, C. Roncal, T. Mittal, R. Chandra, K. Kapsaskis, K. Gray, A. Bera, and D. Manocha, “Take an emotion walk: Perceiving emotions from gaits using hierarchical attention pooling and affective mapping,” in _Computer Vision – ECCV 2020_ , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 145–163.
  * [18] Y. Ferstl and R. McDonnell, “Iva: Investigating the use of recurrent motion modelling for speech gesture generation,” in _IVA ’18 Proceedings of the 18th International Conference on Intelligent Virtual Agents_ , Nov 2018. [Online]. Available: https://trinityspeechgesture.scss.tcd.ie
  * [19] T. Kucherenko, P. Jonell, Y. Yoon, P. Wolfert, and G. E. Henter, “A large, crowdsourced evaluation of gesture generation systems on common data: The GENEA Challenge 2020,” in _26th International Conference on Intelligent User Interfaces_ , ser. IUI ’21. New York, NY, USA: Association for Computing Machinery, 2021, pp. 11–21. [Online]. Available: https://doi.org/10.1145/3397481.3450692
  * [20] A. Mehrabian and J. A. Russell, _An approach to environmental psychology._ the MIT Press, 1974.
  * [21] G. Castillo and M. Neff, “What do we express without knowing? emotion in gesture,” in _Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems_ , ser. AAMAS ’19. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems, 2019, p. 702–710.
  * [22] A. Banerjee, U. Bhattacharya, and A. Bera, “Learning unseen emotions from gestures via semantically-conditioned zero-shot perception with adversarial autoencoders,” _arXiv preprint arXiv:2009.08906_ , 2020.
  * [23] S. Kim, S. J. Guy, W. Liu, D. Wilkie, R. W. Lau, M. C. Lin, and D. Manocha, “Brvo: Predicting pedestrian trajectories using velocity-space reasoning,” _The International Journal of Robotics Research_ , vol. 34, no. 2, pp. 201–217, 2015.
  * [24] U. Bhattacharya, T. Mittal, R. Chandra, T. Randhavane, A. Bera, and D. Manocha, “Step: Spatial temporal graph convolutional networks for emotion perception from gaits,” in _Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence_ , ser. AAAI’20. AAAI Press, 2020, p. 1342–1350.
  * [25] A. Chowanda, P. Blanchfield, M. Flintham, and M. Valstar, “Computational models of emotion, personality, and social relationships for interactions in games: (extended abstract),” in _Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent Systems_ , ser. AAMAS ’16. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems, 2016, p. 1343–1344.
  * [26] S. S. Sohn, X. Zhang, F. Geraci, and M. Kapadia, “An emotionally aware embodied conversational agent,” in _Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems_ , ser. AAMAS ’18. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems, 2018, p. 2250–2252.
  * [27] T. Karras, T. Aila, S. Laine, A. Herva, and J. Lehtinen, “Audio-driven facial animation by joint end-to-end learning of pose and emotion,” _ACM Trans. Graph._ , vol. 36, no. 4, Jul. 2017. [Online]. Available: https://doi.org/10.1145/3072959.3073658
  * [28] Y. Ferstl and R. McDonnell, “A perceptual study on the manipulation of facial features for trait portrayal in virtual agents,” in _Proceedings of the 18th International Conference on Intelligent Virtual Agents_ , ser. IVA ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 281–288. [Online]. Available: https://doi.org/10.1145/3267851.3267891
  * [29] T. Randhavane, A. Bera, K. Kapsaskis, R. Sheth, K. Gray, and D. Manocha, “Eva: Generating emotional behavior of virtual agents using expressive features of gait and gaze,” in _ACM Symposium on Applied Perception 2019_. ACM, 2019, p. 6.
  * [30] J. H. Chuah, B. Rossen, and B. Lok, “Automated generation of emotive virtual humans,” in _Intelligent Virtual Agents_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, pp. 490–491.
  * [31] T. Randhavane, A. Bera, K. Kapsaskis, U. Bhattacharya, K. Gray, and D. Manocha, “Identifying emotions from walking using affective and deep features,” _arXiv preprint arXiv:1906.11884_ , 2019.
  * [32] U. Bhattacharya, N. Rewkowski, P. Guhan, N. L. Williams, T. Mittal, A. Bera, and D. Manocha, “Generating emotive gaits for virtual agents using affect-based autoregression,” in _2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)_ , 2020, pp. 24–35.
  * [33] U. Bhattacharya, N. Rewkowski, A. Banerjee, P. Guhan, A. Bera, and D. Manocha, “Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents,” in _2021 IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR)_. IEEE, 2021.
  * [34] P. Wagner, Z. Malisz, and S. Kopp, “Gesture and speech in interaction: An overview,” _Speech Communication_ , vol. 57, pp. 209 – 232, 2014. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0167639313001295
  * [35] C.-C. Chiu, L.-P. Morency, and S. Marsella, “Predicting co-verbal gestures: A deep and temporal modeling approach,” in _Intelligent Virtual Agents_. Cham: Springer International Publishing, 2015, pp. 152–166.
  * [36] S. Levine, P. Krähenbühl, S. Thrun, and V. Koltun, “Gesture controllers,” in _ACM SIGGRAPH 2010 Papers_ , ser. SIGGRAPH ’10. New York, NY, USA: Association for Computing Machinery, 2010. [Online]. Available: https://doi.org/10.1145/1833349.1778861
  * [37] D. Hasegawa, N. Kaneko, S. Shirakawa, H. Sakuta, and K. Sumi, “Evaluation of speech-to-gesture generation using bi-directional lstm network,” in _Proceedings of the 18th International Conference on Intelligent Virtual Agents_ , ser. IVA ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 79–86. [Online]. Available: https://doi.org/10.1145/3267851.3267878
  * [38] T. Kucherenko, D. Hasegawa, G. E. Henter, N. Kaneko, and H. Kjellström, “Analyzing input and output representations for speech-driven gesture generation,” in _Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents_ , ser. IVA ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 97–104. [Online]. Available: https://doi.org/10.1145/3308532.3329472
  * [39] S. Alexanderson, G. E. Henter, T. Kucherenko, and J. Beskow, “Style-controllable speech-driven gesture synthesis using normalising flows,” _Computer Graphics Forum_ , vol. 39, no. 2, pp. 487–496, 2020. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.13946
  * [40] N. Sadoughi and C. Busso, “Speech-driven animation with meaningful behaviors,” _Speech Communication_ , vol. 110, pp. 90 – 100, 2019. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0167639318300013
  * [41] D. Greenwood, S. Laycock, and I. Matthews, “Predicting head pose from speech with a conditional variational autoencoder.” ISCA, 2017.
  * [42] N. Sadoughi and C. Busso, “Novel realizations of speech-driven head movements with generative adversarial networks,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 6169–6173.
  * [43] S. Suwajanakorn, S. M. Seitz, and I. Kemelmacher-Shlizerman, “Synthesizing obama: Learning lip sync from audio,” _ACM Trans. Graph._ , vol. 36, no. 4, Jul. 2017. [Online]. Available: https://doi.org/10.1145/3072959.3073640
  * [44] D. Holden, J. Saito, and T. Komura, “A deep learning framework for character motion synthesis and editing,” _ACM Trans. Graph._ , vol. 35, no. 4, Jul. 2016. [Online]. Available: https://doi.org/10.1145/2897824.2925975
  * [45] D. Holden, T. Komura, and J. Saito, “Phase-functioned neural networks for character control,” _ACM Transactions on Graphics (TOG)_ , vol. 36, no. 4, p. 42, 2017.
  * [46] S. Starke, H. Zhang, T. Komura, and J. Saito, “Neural state machine for character-scene interactions,” _ACM Transactions on Graphics (TOG)_ , vol. 38, no. 6, p. 209, 2019.
  * [47] Y. Li, M. R. Min, D. Shen, D. E. Carlson, and L. Carin, “Video generation from text,” in _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_ , ser. AAAI’18. AAAI Press, pp. 7065–7072.
  * [48] X. Zhou, S. Huang, B. Li, Y. Li, J. Li, and Z. Zhang, “Text guided person image synthesis,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [49] Y. Ferstl, M. Neff, and R. McDonnell, “Multi-objective adversarial gesture generation,” in _Motion, Interaction and Games_ , ser. MIG ’19. New York, NY, USA: Association for Computing Machinery, 2019. [Online]. Available: https://doi.org/10.1145/3359566.3360053
  * [50] R. Vergin, D. O’Shaughnessy, and A. Farhat, “Generalized mel frequency cepstral coefficients for large-vocabulary speaker-independent continuous-speech recognition,” _IEEE Transactions on Speech and Audio Processing_ , vol. 7, no. 5, pp. 525–532, 1999.
  * [51] K. S. R. Murty and B. Yegnanarayana, “Combining evidence from residual phase and mfcc features for speaker recognition,” _IEEE Signal Processing Letters_ , vol. 13, no. 1, pp. 52–55, 2006.
  * [52] D. Neiberg, K. Elenius, and K. Laskowski, “Emotion recognition in spontaneous speech using gmms,” in _Ninth international conference on spoken language processing_ , 2006.
  * [53] A. Bagher Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 2236–2246. [Online]. Available: https://www.aclweb.org/anthology/P18-1208
  * [54] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” _Transactions of the Association for Computational Linguistics_ , vol. 5, pp. 135–146, 2017.
  * [55] I. Santos, N. Nedjah, and L. de Macedo Mourelle, “Sentiment analysis using convolutional neural network with fasttext embeddings,” in _2017 IEEE Latin American Conference on Computational Intelligence (LA-CCI)_ , 2017, pp. 1–5.
  * [56] D. P. Kingma and M. Welling, “An introduction to variational autoencoders,” _arXiv preprint arXiv:1906.02691_ , 2019.
  * [57] S. Yan, Y. Xiong, and D. Lin, “Spatial temporal graph convolutional networks for skeleton-based action recognition,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, April 2018. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/12328
  * [58] P. J. Huber, “A robust version of the probability ratio test,” _The Annals of Mathematical Statistics_ , pp. 1753–1758, 1965.
  * [59] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014. [Online]. Available: https://arxiv.org/pdf/1412.6980v9.pdf
  * [60] C. Ahuja and L. Morency, “Language2pose: Natural language grounded pose forecasting,” in _2019 International Conference on 3D Vision (3DV)_ , 2019, pp. 719–728.
  * [61] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, “Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 36, no. 7, pp. 1325–1339, 2014.


Here, we recall that $T^{*\vee}\in\mathcal{C}(\bar{U},\bar{V})$ is the
morphism characterized by

$(\mathrm{id}_{U}\otimes
T^{*\vee})\bar{R}_{U}=(T^{*}\otimes\mathrm{id}_{\bar{V}})\bar{R}_{V}.$

Such a structure induces a dagger category structure on the associated module
category $\mathcal{M}_{A}$. In particular,
$A(U\otimes\bar{U})\simeq\mathcal{M}_{A}(U_{A},U_{A})$ is a $*$-algebra for
every $U\in\mathcal{C}$. First note that we have $*$-algebra embeddings

$j_{U,V}\colon A(U\otimes\bar{U})\to A(U\otimes
V\otimes\bar{V}\otimes\bar{U}),\quad a\mapsto
A(\mathrm{id}_{U}\otimes\bar{R}_{V}^{*}\otimes\mathrm{id}_{\bar{U}})(a),$

corresponding to

$\mathcal{M}_{A}(U_{A},U_{A})\to\mathcal{M}_{A}(U_{A}\stmryolt
V,U_{A}\stmryolt V),\quad T\mapsto T\stmryolt\mathrm{id}_{V}.$

This has a left inverse $a\mapsto
d_{V}^{-1}A(\mathrm{id}_{U}\otimes\bar{R}_{V}\otimes\mathrm{id}_{\bar{U}})(a)$.
Moreover,

$\tensor[_{A_{0}}]{\langle a,b\rangle}{}=A(R_{U})(a\cdot b^{\natural})$ (A.2)

defines an $A_{0}$-valued inner product on the left $A_{0}$-module $A(U)$.

######  Definition A.1.

A _pre-C ∗-algebra object_ in $\operatorname{Vec}(\mathcal{C})$ is a
$*$-algebra object $A$ such that $A_{0}$ is a C∗-algebra and the inner product
(A.2) is Hermitian, i.e., $\tensor[_{A_{0}}]{\langle a,a\rangle}{}$ is a
positive element of $A_{0}$ for any $a\in A(U)$.

Let us now assume that $A$ is a pre-C∗-algebra object. By taking fiberwise
completion, we may assume that $A(U)$ is a left Hilbert module over $A_{0}$.
Our goal is to see that this is a C∗-algebra object.

######  Proposition A.2.

Let $A$ be a pre-C∗-algebra object in $\operatorname{Vec}(\mathcal{C})$. Then
we have

$\left\|A(T)(a)\right\|\leq\left\|T\right\|\left\|a\right\|\quad(T\in\mathcal{C}(V,U),a\in
A(U))$

for the norm of left Hilbert $A_{0}$-modules. Similarly, we have

$\left\|a\cdot b\right\|\leq\left\|a\right\|\left\|b\right\|\quad(a\in
A(U),b\in A(V)).$

###### Proof.

We have

$\tensor[_{A_{0}}]{\langle A(T)(a),A(T)(a)\rangle}{}=\tensor[_{A_{0}}]{\langle
A(TT^{*})(a),a\rangle}{}$

by (A.1) and the way we define inner product, (A.2). From
$TT^{*}\leq\left\|T\right\|^{2}$, there is $S$ such that
$\left\|T\right\|^{2}-TT^{*}=SS^{*}$. Then we get

$\left\|T\right\|^{2}\tensor[_{A_{0}}]{\langle
a,a\rangle}{}-\tensor[_{A_{0}}]{\langle
A(T)(a),A(T)(a)\rangle}{}=\tensor[_{A_{0}}]{\langle
A(S)(a),A(S)(a)\rangle}{}\geq 0,$

hence the first claim.

As for the second claim, we can start from $\tensor[_{A_{0}}]{\langle
a,a\rangle}{}\leq\left\|a\right\|^{2}$ and use a similar argument. ∎

######  Theorem A.3.

Let $A$ be a pre-C∗-algebra object in $\operatorname{Vec}(\mathcal{C})$. Then
the fiberwise completion of $A$ with respect to the Hilbert module structure
over $A_{0}$ is a C∗-algebra object.

###### Proof.

The natural action of $A(U\otimes\bar{U})$ on $A(U)$ is by bounded and
adjointable $A_{0}$-homomorphisms by Proposition A.2. We can thus assume that
each $A(U)$ is already a Hilbert $A_{0}$-module.

From the associativity of product $\cdot$ and the antimultiplicativity of
involution $\natural$, we see that the natural action of
$A(U\otimes\bar{U}\otimes U\otimes\bar{U})$ on $A(U\otimes\bar{U})$ is by
adjointable homomorphisms. Thus, the norm

$\left\|a\right\|^{\prime}=\left\|j_{U,\bar{U}}(a)\right\|_{\mathcal{B}_{A_{0}}(A(U\otimes\bar{U}))}$

on $A(U\otimes\bar{U})$ is a pre-C∗-norm. We are going to show that this norm
is equivalent to the Hilbert module norm on $A(U\otimes\bar{U})$. Then the
fiberwise completion of $A$ gives a C∗-module category, hence the completion
will be a C∗-algebra object in the sense of Definition 2.7.

On one hand, the action of $j(a)=j_{U,\bar{U}}(a)$ on
$\eta_{U}=A(\bar{R}_{U}^{*})(1_{A_{0}})$ (the unit of $A(U\otimes\bar{U})$) is
$a$, hence we have

$\left\|a\right\|^{\prime}\geq\left\|a\right\|_{A(U\otimes\bar{U})}\left\|\eta_{U}\right\|_{A(U\otimes\bar{U})}^{-1}.$

On the other, Proposition A.2 implies

$\left\|R_{U}\right\|\left\|a\right\|_{A(U\otimes\bar{U})}\geq\left\|a\right\|^{\prime},$

and we obtain the claim. ∎

## References


# Human-Machine Collaborative Video Coding Through Cuboidal Partitioning

###### Abstract

Video coding algorithms encode and decode an entire video frame while feature
coding techniques only preserve and communicate the most critical information
needed for a given application. This is because video coding targets human
perception, while feature coding aims for machine vision tasks. Recently,
attempts are being made to bridge the gap between these two domains. In this
work, we propose a video coding framework by leveraging on to the commonality
that exists between human vision and machine vision applications using
cuboids. This is because cuboids, estimated rectangular regions over a video
frame, are computationally efficient, has a compact representation and object
centric. Such properties are already shown to add value to traditional video
coding systems. Herein cuboidal feature descriptors are extracted from the
current frame and then employed for accomplishing a machine vision task in the
form of object detection. Experimental results show that a trained classifier
yields superior average precision when equipped with cuboidal features
oriented representation of the current test frame. Additionally, this
representation costs $7\%$ less in bit rate if the captured frames are need be
communicated to a receiver.

Index Terms—  Cuboid, HEVC, VCM, Object detection

##  1 Introduction

Many video analysis systems employ a client-server architecture where video
signals are captured at the front-end devices and the analyses are carried out
at cloud-end servers. In such systems, video data need be communicated from
the front end to the server. Prior to this transmission, captured video signal
is encoded using a video coding paradigm and at the receiving end the obtained
signal is decoded before performing any analysis task. An alternative approach
is to extract features from the video data at the client end and then
communicate the encoded features rather than the video signal itself. Since
capturing of the signal and feature extraction are conducted at the front end,
the server becomes less clumsy pertaining to large scale applications and can
dedicate most of its resources for the analysis work. However, as the
estimated features are devised for specific tasks only, they can be difficult
to generalize to a broad spectrum of machine vision tasks in the cloud end.

Traditional video coding standards like H.264/AVC [1] and HEVC [2] are block-
based. They are also pixel and frame centric. In order to model the
commonality that exists within a video sequence, the frame that need be coded,
known as the current frame, is artificially partitioned into square or
rectangular shape blocks. These blocks are formed by grouping together
neighboring pixels. After this partitioning, for each current frame block,
commonality modeling is employed to form a prediction of it either using the
already coded neighboring blocks belonging to the current frame (intra-
prediction) or using motion estimation and compensation within a co-located
neighborhood region in the set of already coded reference frame(s) (inter-
prediction), by minimizing a rate-distortion criterion.

Due to the requirements for communicating feature descriptors, coding
standards like compact descriptors for visual search (CDVS) [3] and compact
descriptors for video analysis (CDVA) [4] were developed. In CDVS, to
represent the visual characteristics of images, local and global descriptors
are designed. Deep learning features are employed in CDVA to further augment
the video analysis performance. Although these features showed excellent
performance for machine vision tasks, it is not possible to reconstruct full
resolution videos for human vision from such features. This results into two
successive stages of analysis and compression for machine and human vision
[5].

As modern video coding standards like HEVC focus on human vision by
reconstructing full resolution frames from the coded bitstream and standards
like CDVS and CDVA focus on machine vision tasks and incapable of
reconstructing full resolution pictures from the coded feature descriptors;
there is a need for human-machine collaborative coding that can leverage on
the advances made on the frontiers of video coding and feature coding as well
as bridge the gap between these two technologies. For example, in autonomous
driving use case, some decisions can be made by the machine itself and for
some other decisions humans would interact with machine. Such collaborative
coding problem is known as video coding for machines (VCM) [5]. In this
direction of work, it was proposed in [6] to extract features in the form of
edge maps from a given image which is used to perform machine vision tasks.
After that a generative model was employed to perform image reconstruction
based on the edge maps and additional reference pixels.

Although the objectives of human vision and machine vision are different,
human-machine collaborative coding can benefit from exploring the significant
overlap that exists between these two domains. One important similarity is
that human vision is mostly hierarchical. For example, an image of a study
room can be described by humans using annotations like - over the floor carpet
(global context) there is a desk (local detail) and a laptop (finer detail) is
on the desk; on the wall (global context) a clock (local detail) is mounted,
etc. Machine vision tasks like object detection can also be modeled in this
kind of hierarchical way. For instance, in [7] a method is proposed for
performing hierarchical object detection in images through deep reinforcement
learning [8]. Using scale-space theory [9] and deep reinforcement learning,
another work [10] explored optimal search strategies for locating anatomical
structures, based on image information at multiple scales.

Murshed _et al._ proposed the hierarchical cuboidal partitioning of image data
(CuPID) algorithm in [11]. In CuPID framework, an entire image is initially
partitioned into two rectangular shaped regions, known as cuboids, by finding
a hyperplane orthogonal to one of the axes. A greedy optimization heuristic,
equipped with sum of the information entropy in the split-pair cuboids as the
objective function, is employed to find the minimizing hyperplane. This
ensures that the obtained split-pair cuboids are maximum dissimilar in terms
of image pixel intensity distribution. Next, each cuboid is recursively
partitioned by solving the same greedy optimization problem. It was shown that
cuboids have excellent global commonality modeling capabilities, are object
centric and computationally efficient [12, 13, 14, 15].

The aforementioned properties of cuboids were leveraged on to traditional
video coding in the work of [16, 17]; wherein a reference frame is generated
using cuboidal approximation of the target frame and incorporating this
additional reference frame into a modified HEVC encoder outperformed the rate-
distortion performance of a standalone HEVC encoder. The obtained gain in
delta PSNR and savings in bit rate can be attributed to the fact that HEVC
employs a partitioning scheme that begins at a rigid fixed size of $64\times
64$ pixels level and during this process does not take into account the
structural properties of the scene need be coded. This sub-optimal
partitioning was improved by incorporating cuboids, which are estimated
considering the entire frame’s pixel intensity distribution and therefore more
object-centric.

Building on the work [16], in this paper we investigate the applicability of
cuboidal features in video coding targeting machines. Based on the CuPID
algorithm, _(i)_ features are extracted from a given video frame for the
machine vision task of object detection. These features costs less bits to
encode. Addition to this, _(ii)_ it is possible to reconstruct a full-
resolution frame from the obtained feature descriptors alone; unlike the
approach in [6] that require additional reference pixels along with edge based
feature descriptors in this regard. The reconstructed frame is a coarse
representation of the original frame and is capable of preserving important
structural properties of the original frame. Finally, _(iii)_ the object
detection performance of the cuboidal descriptors set are put to test across
varying number of cuboids.

The rest of the paper is organized as follows: in section II, we briefly
describe the cuboidal feature extraction process from a given video frame and
the reconstruction process of a full-resolution frame from those estimated
feature descriptors. Section III, describes the performance of the cuboidal
features over an object detection task. Finally, in section IV, we present our
conclusions.

Fig. 1: The estimated cuboid map over the current frame, $C$. This frame is
part of the vehicle detection dataset in [18]. The cuboid map, consisting of
$70$ cuboids, is determined by the CuPID algorithm [16]. Each cuboid’s
coverage is depicted with white boundary pixels.

(a) $R_{\text{\emph{co}}}$ frame ($n=100$): $R_{\text{\emph{co}}}^{(n=100)}$

(b) $R_{\text{\emph{co}}}$ frame ($n=200$): $R_{\text{\emph{co}}}^{(n=200)}$

(c) $R_{\text{\emph{co}}}$ frame ($n=300$): $R_{\text{\emph{co}}}^{(n=300)}$

Fig. 2: Different coarse representations of the current frame $C$. Each one is
a $R_{\text{\emph{co}}}$ frame, obtained by varying the number of cuboids,
$n$.

##  2 Determination of Cuboidal Features from the Current Frame using CuPID

Given the original uncompressed current frame $C$, of resolution $X\times Y$
pixels, the CuPID algorithm hierarchically partitions it into $n$ cuboids
$C^{(1)},C^{(2)},\ldots,C^{(n)}$, where $n$ is a user-defined number of tiles.
The first partitioning can produce two half-cuboids $C^{(1)}_{i}$ and
$C^{(2)}_{i}$ of size $i\times Y$ and $(X-i)\times Y$ pixels respectively.
This is achieved by selecting a vertical line $x=i+0.5$ over $C$ from the set
of lines $i\in\\{1,2,\ldots,X-1\\}$. Alternatively, $C$ could be split into
two half-cuboids $C^{(1)}_{x-1+j}$ and $C^{(2)}_{x-1+j}$ of size $X\times j$
and $X\times(Y-j)$ pixels, respectively by selecting a horizontal line
$y=j+0.5$ from the set of lines $j\in\\{1,2,\ldots,Y-1\\}$.

The first optimal split $s^{*}$ of $C$ from the possible number of splits
$(X-1)+(Y-1)=X+Y-2$ is obtained by solving a greedy optimization [19] problem
whose objective function is taken to be a measure of dissimilarity between the
candidate cuboid split-pair [16]. After that each obtained half-cuboid is
further split recursively until the total number of cuboids obtained from $C$
becomes equal to the user input $n$. Fig. 1 shows an example cuboid map for
the frame $C$. It can be observed that the estimated cuboids are fairly
homogeneous in terms of pixel intensity distribution and they occupy arbitrary
shaped rectangular regions. Classes like road, sky and trees are represented
using separate larger cuboids and important cues for vehicle detection are
also preserved.

The estimated cuboid map can be reconstructed at a decoder side from the
optimal partitioning indices $\\{s^{*}\\}_{i=1}^{n-1}$ alone. That means, for
example, if the first optimal partitioning, which split the entire image into
$2$ cuboids, took place at height (or width) $h$, the number $h$ need be
encoded and so on. These indices are encoded and augmented into a bitstream in
the way described in [16]. Next, for each obtained cuboid, $C^{(i)}$, from the
cuboid map, a feature descriptor, $\mathbf{m}^{(i)}$, is computed. This value
is taken to be the mean pixel intensity (for each image channel of $C$),
considering all the pixels $p$ and their intensities $p(x,y)$ within the
coverage of the corresponding cuboid.

$m_{j}^{(i)}=\frac{\sum_{p\in C^{(i)}}p_{j}(x,y)}{|C^{(i)}|}$ (1)

where $j$ represents different color component channels of the frame $C$. The
feature descriptors $\mathbf{m}^{(i)}\in{\mathbb{Z}}^{n}_{*}$ are encoded in
the way described in [16] and are also part of the bitstream.

Having decoded the optimal partitioning indices and the feature descriptors
from the communicated bitstream, it is possible to generate a full-resolution
frame, $R_{\text{\emph{co}}}$. This frame $R_{\text{\emph{co}}}$ provides a
coarse representation of the current frame, $C$ since it is created by
replacing every encompassing pixel $p\in C^{(i)}$ intensity by the associated
cuboidal feature descriptor $\mathbf{m}^{(i)}$. Fig. 3 shows examples of this
coarse representation frame $R_{\text{\emph{co}}}$. It can be observed that
this low-frequency representation $R_{\text{\emph{co}}}$ attempt to capture
important structure information about the scene and the degree of detailed
information it can communicate tends to increase with growing number of
cuboids, $n$. Table 1 reports the bits requirements, computational complexity
and quality (in terms of PSNR) of these $R_{\text{\emph{co}}}$ frames. Here,
$b_{c}$ and $t_{c}$ stand for bits and computational time (in seconds)
required for an HEVC encoder [20] to intra-code (with QP $47$) the frame $C$.
The employed system configuration is: Intel Core i7-8650U<EMAIL_ADDRESS>GHz, 32.0
GB RAM. The maximum PSNR is achieved from the frame
$R_{\text{\emph{co}}}^{(n=300)}$ and this frame can be encoded at less than
$4\%$ bits requirements that of HEVC. Moreover, the encoding process is
$3.45\%$ times faster compared to the HEVC reference software [20].

Table 1: Savings in bits and complexity requirements from the cuboidal feature descriptors at different scale (varying number of cuboids, $n$) over HEVC reference. It also reports the corresponding reconstructed $R_{\text{\emph{co}}}$ frames PSNR with respect to the current frame $C$. n | 100 | 200 | 300
---|---|---|---
Savings in bits:$\frac{(b_{c}-b_{co})}{b_{c}}$ | $71\%$ | $38\%$ | $4.05\%$
Savings in comp. time:$\frac{(t_{c}-t_{co})}{t_{c}}$ | $72.41\%$ | $41.38\%$ | $3.45\%$
Y-PSNR (in dB) | $32.20$ | $32.69$ | $33.16$

##  3 Object Detection using the Cuboidal Feature Descriptors

The cuboidal feature descriptors attempt to capture important structural
properties of the image at comparatively lower bit and computational
complexity requirements. In this work, we propose to utilize them for a
machine vision task, in particular the $R_{\text{\emph{co}}}$ frames are
employed in a vehicle detection problem. For the addressed vehicle detection
problem, the object detector You only look once (YOLO) v2 [21] is used. YOLO
v2 is a deep learning object detection framework that uses a convolutional
neural network (CNN) [22] for detection task. Using unoccluded RGB images of
the front, rear, left, and right sides of cars on a highway scene, the
detector is trained. The CNN used with the vehicle detector, uses a modified
version of the MobileNet-v2 network architecture [23]. $60\%$ of the data is
used for training, $10\%$ for validation, and the rest $30\%$ is used for
testing the trained detector.

(a) HEVC coded test frame with detection confidence scores:
$\\{0.79,0.80,0.73,0.81\\}$ respectively.

(b) $R_{\text{\emph{co}}}^{(n=300)}$ test frame with detection confidence
scores: $\\{0.78,0.68,0.78,0.86\\}$ respectively.

Fig. 3: The trained vehicle detector’s performance over different test frame. (From left to right) HEVC intra-coded version and coarse representation $R_{\text{\emph{co}}}$ of $C$ respectively. The detector managed to detect same number of vehicles in the $R_{\text{\emph{co}}}$ frame with various confidence scores. Table 2: Average precision (AP) values of the trained vehicle detector over different representations of the same test set [18]. It also reports the bit rate required to code the test set frames. Testset | AP | Bit rate (Kbps)
---|---|---
HEVC coded | $0.9366$ | $110.63$
$R_{\text{\emph{co}}}^{(n=100)}$ | $0.8158$ | $31.91$
$R_{\text{\emph{co}}}^{(n=200)}$ | $0.9083$ | $67.53$
$R_{\text{\emph{co}}}^{(n=300)}$ | 0.9462 | $102.74$

Once the detector has been trained, it is used to detect vehicles over _(i)_
HEVC coded versions of the original test video frames (refers to Fig. 3(a))
and _(ii)_ cuboidal descriptor oriented coarse representation frames
($R_{\text{\emph{co}}}$) of the original test frames (refers to Fig. 3(b)). As
can be seen, the detector managed to detect same number of vehicles in the
$R_{\text{\emph{co}}}^{(n=300)}$ frame like its HEVC coded counterpart.
Moreover, for $2$ vehicles the detector’s detection performance (in terms of
higher confidence score) improved when compared to the HEVC coded frame.
However, for the second vehicle, the confidence score is lower in the case of
$R_{\text{\emph{co}}}^{(n=300)}$ frame. This is because that vehicle in the
scene is partly occluded by another vehicle and therefore its cuboidal
representation lacks in some structural information.

Table 2 reports the performance of the detector over these different test data
in the form of average precision. It can be noted that the detector produced
the maximum average precision of $0.9462$ over
$R_{\text{\emph{co}}}^{(n=300)}$ test frames. This result is achieved with a
bit rate requirements that of HEVC; in particular a bit rate savings of around
$7\%$ is obtained. Bit rate savings improve further (a savings of $39\%$) if
the $R_{\text{\emph{co}}}^{(n=200)}$ test frames are used instead. However, in
this case the average precision decreases ($-0.0283$) compared to the HEVC
coded test set. Fig. 4 shows the precision-recall curves for the employed test
sets. At almost $80\%$ recall, the precision is more over the test sets
$R_{\text{\emph{co}}}^{(n=200)}$ and $R_{\text{\emph{co}}}^{(n=300)}$,
compared to the HEVC coded test set. And these cuboidal descriptor oriented
test sets take less bits to encode than HEVC.

Fig. 4: Precision recall performance of the trained vehicle detector over
different representations of the same test set.

##  4 Conclusion

Despite the difference in objectives, there exists similarities between human
vision and machine vision. A collaborative coding paradigm is required for
many use cases. The cuboidal descriptors oriented representation of the
current frames were shown to be beneficial in traditional video coding.
Leveraging on their properties like object oriented, scene structural
awareness, compact representation and computational simplicity, in this paper,
cuboidal descriptors are employed in a machine vision task. Experimental
results show that the addressed vehicle detection problem can be solved more
accurately (average precision improved by $0.01$) if cuboidal descriptors are
used. Along with this a savings in bit rate of around $7\%$ is reported over a
HEVC reference.

## References

  * [1] T. Wiegand, G.J. Sullivan, G. Bjøntegaard, and A. Luthra,  “Overview of the H.264/AVC video coding standard,”  IEEE Trans. on CSVT, vol. 13, no. 7, pp. 560–576, July 2003.
  * [2] G.J. Sullivan, J. Ohm, W. Han, and T. Wiegand,  “Overview of the high efficiency video coding (HEVC) standard,”  IEEE Trans. on CSVT, vol. 12, no. 22, pp. 1649–1668, 2012.
  * [3] L. Duan, T. Huang, and W. Gao,  “Overview of the MPEG CDVS standard,”  in 2015 Data Compression Conference, 2015, pp. 323–332.
  * [4] L. Duan, Y. Lou, Y. Bai, T. Huang, W. Gao, V. Chandrasekhar, J. Lin, S. Wang, and A. C. Kot,  “Compact descriptors for video analysis: The emerging MPEG standard,”  IEEE MultiMedia, vol. 26, no. 2, pp. 44–54, 2019.
  * [5] L. Duan, J. Liu, W. Yang, T. Huang, and W. Gao,  “Video coding for machines: A paradigm of collaborative compression and intelligent analytics,”  IEEE Trans. on Image Processing, vol. 29, pp. 8680–8695, 2020.
  * [6] Y. Hu, S. Yang, W. Yang, L. Y. Duan, and J. Liu,  “Towards coding for human and machine vision: A scalable image coding approach,”  in IEEE ICME, 2020, pp. 1–6.
  * [7] M. Bellver, X. Giró-i-Nieto, F. Marqués, and J. Torres,  “Hierarchical object detection with deep reinforcement learning,”  CoRR, vol. abs/1611.03718, 2016.
  * [8] V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, M. Bellemare, A. Graves, M. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis,  “Human-level control through deep reinforcement learning,”  Nature, vol. 518, no. 7540, pp. 529–533, Feb. 2015.
  * [9] T. Lindeberg,  “Scale-space for discrete signals,”  IEEE Trans. on PAMI, vol. 12, no. 3, pp. 234–254, 1990.
  * [10] F. Ghesu, B. Georgescu, Y. Zheng, S. Grbic, A. Maier, J. Hornegger, and D. Comaniciu,  “Multi-scale deep reinforcement learning for real-time 3D-landmark detection in ct scans,”  IEEE Trans. on PAMI, vol. 41, no. 1, pp. 176–189, 2019.
  * [11] M. Murshed, S. W. Teng, and G. Lu,  “Cuboid segmentation for effective image retrieval,”  in DICTA, Nov 2017, pp. 1–8.
  * [12] M. Murshed, P. Karmakar, S. W. Teng, and G. Lu,  “Enhanced colour image retrieval with cuboid segmentation,”  in DICTA, Dec 2018, pp. 1–8.
  * [13] S. Shahriyar, M. Murshed, M. Ali, and M. Paul,  “Depth sequence coding with hierarchical partitioning and spatial-domain quantisation,”  IEEE Trans. on CSVT, pp. 1–1, 2019.
  * [14] A. Ahmmed, M. Murshed, and M. Paul,  “Leveraging cuboids for better motion modeling in high efficiency video coding,”  in ICASSP, 2020, pp. 2188–2192.
  * [15] A. Ahmmed, M. Paul, M. Murshed, and D. Taubman,  “Dynamic point cloud compression using a cuboid oriented discrete cosine based motion model,”  in ICASSP, 2021, pp. 1935–1939.
  * [16] A. Ahmmed, M. Paul, M. Murshed, and D. Taubman,  “A coarse representation of frames oriented video coding by leveraging cuboidal partitioning of image data,”  in MMSP, 2020, pp. 1–6.
  * [17] A. Ahmmed, M. Paul, M. Murshed, and D. Taubman,  “Dynamic point cloud geometry compression using cuboid based commonality modeling framework,”  in ICIP, 2021.
  * [18] “Detect vehicles using YOLO v2 Network,” https://www.mathworks.com/help/driving/ref/vehicledetectoryolov2.html.
  * [19] T. Cormen, C. Leiserson, R. Rivest, and C. Stein,  Introduction to Algorithms, Third Edition,  The MIT Press, 3rd edition, 2009.
  * [20] “HM Reference Software for HEVC (version 16.20),” https://hevc.hhi.fraunhofer.de/.
  * [21] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi,  “You only look once: Unified, real-time object detection,”  CoRR, vol. abs/1506.02640, 2015.
  * [22] S. Albawi, T. A. Mohammed, and S. Al-Zawi,  “Understanding of a convolutional neural network,”  in ICET, 2017, pp. 1–6.
  * [23] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. Chen,  “Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation,”  CoRR, vol. abs/1801.04381, 2018.